Nice! I love me a nice masochistic rhythm game! At least I assume that's what you're going for with Chaoz Airflow there. ;-)
Also, they disappear sooner because it's much too easy if you hit the notes with your side
Just out of curiosity, does it have to have a binary hit/miss response for notes? For example, DDR has a wide range of responses for almost-hits, ranging from half-points to nothing at all. Also, the penalties for missing a note are much too evil. Like, I understand if flubbing a stream causes serious penalties, but I shouldn't be completely boned for missing just one note. (But that's just a side effect of the combo multiplier system, which I never liked for that reason)
Also, like Blackcorn mentioned, feedback is definitely an issue. Most rhythm games don't really need to concern themselves with it, because any action is accompanied with a firm stamp of the foot or some other "heavy" action. However, here you're just swinging your mouse left and right. I don't really know any way to fix this, but you could maybe alleviate some of the pain by adding a bit of eye candy. Maybe give the orbs something else to do besides just disappear. Add some nice feedback whenever the combo meter goes up. Add some SFX, a little auditory feedback goes a long way, and since you're syncing everything to the beat anyway the SFX will line up nicely.