Oh bugger, I just saw I forgot to answer to my own topic after reading the posts back then. Well...
Quest 64!
What's so special about it? Wikipedia isn't helpful regarding that!
[..Death Gate book series..]
Aaah, these books were awesome. Somehow I doubt that they could catch the spirit of their magic system inside a game, but I will try it out - just found it on Abandonia.
Deep Magic 2's seemed pretty cool - the game is unfinished, but it's worth checking out.
I tried it - it's kinda nice, especially since in its simplicity. Though somewhat short, and could've been a bit more spells. Anyway, I enjoyed it!
King's Quest III [...]
Sounds like an interesting story, though the magic system doesn't look too unusual/special for me. Am I overlooking something?
Except this:
Spellcaster. The idea is that there are two dueling wizards; each round they each make a hand signal. Sequences of hand signals form spells. So do you want to go for short spells, or really long ones? If you recognize what spell your opponent's trying to cast, can you try to mess with it?
Awesome! I was thinking about creating a simplified multiplayer game like this before, only in real-time. Has lots of design problems, though'. Gotta look through the rules of this one!
By the way, the great thing about Eternal Darkness is that it's Lovecraftian horror. Whenever you cast a spell, you're calling on the power of an ancient evil god; you actually hear the deity's voice reading off the runes as the spell gets cast. "TIER! PARGON! ARETAK! PARGON! ULYAOTH!" It's completely awesome.
That sounds totally badass! Pity I don't like horror games, I want stuff like that included in other RPGs! :D
I'm currently playing through Treasure of the Rudras, an old SNES RPG by Square that never saw stateside release (but there's an english fan translation).
You have a spell book that has 32 slots for spells. To make a spell, you open up a screen identical to the screen you use to name your characters, and you punch in a word. Every word makes a spell, but a lot of them are useless. To learn good spells you: [...]
Put it down on my list to try out, sounds nice indeed.
Also, Tactics Ogre had a magic system where you started with 0 MP and you charged magic as the fight dragged on, so instead of using your best spells off the bat, you instead had to wait and save up to launch battle-changing spells midway to late through the fight.
So, charging while being hit or charging just over time? Both ways sound nice and introduce a whole new tactical aspect.
There happens to be a mud (multiplayer text game), God Wars 2, based on this system -- not just for spells, but all combat, but you can just check out the spell system easily enough.
Pity that MUDs are bit too retro for me. Thanks for pointing it out, anyway!
There also is a neat IF game, Savoir-Faire, that uses an interesting magic linking system. Mostly (all?) non-combat stuff.
Game's IFDB page for more info.
The linking system looks interesting indeed. Reminds me of the book
The Name of the Wind - they have an entire magic branch for this linking-thing, called "sympathy". (It's awesome, by the way.)
Arx Fatalis was this first person action rpg thing, and to cast spells you held down the magic button and drew runes you learned before into the air with your mouse. It was pretty fun

My brother had it and didn't like it... guess I have to try it now?

I'm not sure what you mean by unusual, but I've never seen a game quite like
MagiWoah, Magi is awesome. Bought it! (To be correct, bought it after reading your post months ago.

)
I don't suppose the Merlin's Revenge series count do they? I don't know how "unusual" it is, but it is very well executed and great fun.
Also, while the Final Fantasy series are about as far from "unusual" as you can get, they each have an interesting way of implementing magic, often with it being integral to the story. Gotta count for something, right?
Nah, Merlin's Revenge is ok, but not THAT great magic wise - at least what I played of it so far.
And yes, Final Fantasy sure has some interesting ways of doing magic spells

I want a game that is to spellcasting what Toribash is to fighting.
Y'know, you have to manually craft your movements in karoshi instead of using button combinations to choose the premade, preanimated attack you wanted to use from a list. I want a game where instead of picking "fireball" from a list, you have to use tools like "increase/decrease heat", "increase/decrease air pressure" and other stuff like that to make your fireball by yourself.
Hum. I would love to have something like your premise "spellcasting like Toribash to fighting" too, but "increase/decrease heat" or "increase/decrease air pressure" sounds more like a physics simulation, doesn't it?