I just Googled and found this, dunno if it's any good:
http://www.codeplex.com/q3libxna/The BSP file format is also readily available if you want to roll your own loader. I've done it for Quake 1 maps, and my understanding is that it's somewhat easier for Quake 3. Trying to make sense of the uncompiled .map would be extremely slow and difficult. It stores brushes as sets of planes, as you pointed out, so you'd have to clip them and build the faces yourself. That's a lot of work and a big hit at runtime--there's a reason maps are compiled offline and shipped as .bsp.
I understand your argument for just loading a poly soup--the complexity of a BSP tree is significant and mostly unnecessary if you're making simple boxy worlds. On the other hand, it will be a huge benefit to the performance of both your rendering and collision code if you use
some form of spatial subdivision, especially as you build larger and more complex maps. There's no reason you can't just load up a .bsp file and render it all at once, either (the structure of a BSP tree and the precomputed visibility set can improve rendering performance, but that doesn't preclude rendering the whole thing as a poly soup if you want to).