Aside from anecdotal evidence of having shown it to people who seemed interested or there being games out there with a similar theme/genre is there a more scientific/mathematical way of actually figuring out who might be interested in the game you're making?
It's important to remember that even large companies often rely on customer feedback and basically showing their potential product to people, it's just that in their case, they may use focus groups and show it to a whole lot of people.
I think the simple answer to your question is that you just have to try to show it to as many people as you can and get as much feedback as you can (which of course can be very challenging). I'd tend to think that even an indie developer can get quite a lot of feedback about their potential game project if they're willing to put that much work into the marketing end of it; it's just that most Indies aren't, and from what I can tell many really don't like that sort of thing.
In the case of video games, it's really tricky because games are a creative product and therefore no one "needs" them in the same way that they might find use for a more "practical product" that is designed to solve a specific problem in their lives. Because most games are about entertainment or something similar to that, that's where the value of them has to come from and this means that personal tastes factor into it a lot.
It's difficult to try to quantify the potential demand for something like this in a more "scientific/mathematical way", because when most people play games they are looking for some sort of entertainment value and it's not always obvious what people are going to enjoy or not. The best approach is probably just to ask them, or even better yet, have a prototype and see what people do when they play. In my opinion watching people play a game is even more informative than just having them tell you their opinions about it. I've also heard that if you don't enjoy playing your own game as you're developing it, that's probably not a good sign.
I'll also mention that another tricky thing about this is that in many cases the details matter and whether people will like your game or not could be dependent on just changing one little thing about it. Video games are complex things which have both a creative/artistic element and an engineering one. The creative concept has to be compelling or at least somehow interesting and engaging, but the thing also has to function correctly from a technical standpoint and these two things are very much intertwined in many cases. Although sometimes the whole basic premise of the game might be flawed and therefore unmarketable, I think that in many cases there are smaller details of a game that can be changed in order to make it more compelling for people. But unfortunately it's hard to tell what those might be until you have people look at it and give you feedback about it.
Of course from a marketing standpoint, it also helps if the game just sounds cool, before people even get their hands on it.