Yeah, a good rule of thumb is that if the game feels like the right difficulty for you, as the designer, it's way too hard. It's a combination of the fact that you know your own game inside and out, you've played it repeatedly in testing it, and also the fact that you probably designed it to feel good to you, and act exactly how you expect, while other people need to get a feel for the game and get the hang of it.
Is this always true though? I mean, I've found that my playtesters are usually
better at my game than I am, probably because I'm shit at games. I guess the point still stands though, playtest, then playtest some more.
Of course anyone's mileage may vary. But very often I've seen indie developers designing the game to be a satisfying difficulty for themselves, and having it be much too difficult for most players. And, yeah, like you said, no way to know how other people are going to react to your game until you get other people to try it.
Additionally, an important thing I've been told about playtesting: try to avoid talking to your tester while they play, aside from asking for their thoughts. Try not to tell them what they should be doing, or help them if they get stuck, or tell them useful techniques (I mean, unless not doing so grinds testing to a halt, you take a note of the problem area, and give them just enough information to move on). Players getting stuck on things or not figuring out how to perform certain actions are exactly the things you should be testing for.