These has been a couple threads on the subject, such as
this one.
I'll just copypaste my reply (with some additions) from there to here:
I use FRAPS wherever possible since it provides the most solid recording - plus the lossless recordings are easy to work with afterwards.
I've also tried Camtasia but it tends to be quite terribly slow and not really well suited for in-game recording. Sadly since FRAPS is pretty much limited to capturing DirectX games, you must resort to other means like Camtasia for some things. For a free alternative, you could try CamStudio.
For processing the videos I just generally use AviSynth and encode the video with x264 using something like x264 --preset slow --crf 22. I also convert audio to AAC with neroaacenc and then finally mux to MKV which can then be uploaded to YouTube or whatever.
Also, free tip: If your game footage has lots of stuff going on and you're going to upload the video to YouTube, I recommend blurring the video beforehand (if you know any AviSynth, the function you're looking for is simply Blur(1.0) ). It increases the quality on YouTube quite notably. For an example, you can compare these two videos:
(not blurred beforehand)
(blurred beforehand)
As you can see, the 480p quality difference is quite drastic.
Other things to note about YouTube is that it automatically decimates videos to 30fps, so if you record your game footage at 60fps it's a good idea to decimate it beforehand to save some bandwidth and encoding time.
Community
