The nintendo 64 was the playstation 3 of that era. A complicated powerful piece of hardward, except nintendo is an ass for third party.
Interestingly, I use the exact same analogy when I talk about the PS3 to people, even though it's a bit off because the PS3 has lots of multiplatform games, whereas hardly any PS1 games were ported to the N64 or vice versa. Oh and by the way, I prefer both the N64 over the PS1 and the PS3 over the 360.

Do people the playsation was actually the idea of nintendo? It was an extension of the snes at the time, it was also in the continuity of the super fx dsp and partenario with sony with their music chip.
Also, the original PS1 controller had more or less the same button layout as the SNES (so has the 360 BTW), plus two additional shoulder buttons. The Dualshock added sticks to it. I guess the PS1 was what a lot of people were expecting Nintendo to do at the time, which explains why so many "jumped ship".
Cartridge was not a good move but 8mm disc does not change anything since any disc had his own burn format. If they incompatible with everything else what's the deal of a more convenient sized one?
Two reasons: Piracy prevention (you had to jump through quite a few hoops to get a pirated GC game actually running) and not having to pay money to the DVD board. The Wii also uses its own format of "Not-DVDs", for the same reasons, even though it's a lot easier to pirate a Wii game.