I've been thinking for a long time about ways to flesh out the way dialogue can be treated in games. First, the complexity of dialogue mechanics in your game should be proportional to the amount of dialogue in your game. If dialogue is as important in your game as combat, its mechanics should be just as detailed. While dialogue doesn't play a big enough role in my current project to warrant the inclusion of most of the features I list here, some side- projects in the design docs folder could make great use of them.
1. The most important thing in dialogue is that it actually
be dialogue. Games are unique because they are interactive--take advantage of this dimension whenever possible. If you never give your player (at least the illusion of) a voice, he might as well be reading a book during the wordy bits of your game.
In the past, dialogue interactivity has taken the form of menu-driven dialogue trees in most wRPGs, text parsers in interactive fiction, and keyword wikis in a few older RPGs. Though my favorite (and the most popular) of these is the menu-driven approach, each of these systems has its own failings (which I won't go into detail about here, but they're fairly obvious if you think about them).
2. An idea for expanding dialogue mechanics in one of my future projects involves a fusion of the menu and wiki approaches and is based off an indie adventure game called Rorschach. The player's character would have a skill called, say, "small-talk" or "memory" that would govern the number of topics he could store in his memor at a time. When talking with an NPC, he could start a conversation about one of these topics, which would open a menu-based dialogue tree with the NPC (in which other dialogue-based skills might be employed). In such conversations, he could swap topics he encounters while talking with NPCs for topics already in his memory. "Topics" could also be acquired by reading books and newspapers. Perhaps these topics could also be added to a long-term topic bank which would allow the player to swap active and inactive topics around at, say, the beginning of every day.
3. I mentioned other possible dialogue skills earlier, one of which I will describe here. In social interaction, some people are better able to act against their nature than others. This could be modeled in the player character by a "self-control" stat (though that isn't quite the right phrase). This stat would determine the number of options available to a character in each dialogue menu, with the responses most in line with your character's personality appearing first. Thus, a naturally bloodthirsty character with high self-control might be able to reign himself in and have access to responses which would be more natural for more peaceful individuals. Over time, doing so might even have an effect on his personality. Additionally, a character with low self control may opt to spend a "turn" of dialogue trying to think of something to say (increasing the number of options available). However, the NPC might decide to fill your silence by continuing on, and in any case will likely perceive your character as socially awkward.
4. Too often, NPCs are treated merely as information dispensers. This becomes painfully obvious in instances such as the "investigate" dialogue branches in some of Bioware's more modern games. Most NPCs should tire of conversation with your character by the tenth time you've asked them the same question. A lot of work needs to be done to make NPCs more lifelike in this regard. This is much easier said than done, however, which brings me to a way to possibly go about remedying this...
5. Procedural dialogue. I've had vague ideas concerning this, but it seems to be the only way to tackle a large-scale game in which dialogue is crucial. This fellow captures the gist of it over here:
http://groups.google.com/group/rec.games.roguelike.development/browse_thread/thread/823d1607ec9d98d6.