Kristen I lucked up on finding my size. My fiancé gave me my first pair because she had lost some weight and the didn’t fit her any more. Those were a large.
As I started buying my own I would just pull them out and look at them to assess if they were going to be big or small as a large. Keep in mind they are quite stretchy and rather forgiving in the size department and all makers sizes are different of course.
The athletic compression style ones aren’t as forgiving as all the others. You have to be careful with them. I recently got some of that variety in my usual large and they fit good through the legs but it takes a little wiggling to get them up over my hips but once they’re up they’re fine. That particular pair is supposed to give you a little lift in all the right places and they do just that lol.
But basically you’re right. Stick to panty size.