When you define a particle in GameMaker, you may make it use one of the default shapes, or just assign a sprite to it using
part_type_sprite function. Making your own sprites for flares, smoke, etc. gives you much more control over the style of your particles and should look much better right away. For example - instead of using
pt_shape_sphere you could make a nice pixel art blob.
This is my flame sprite for example:
Another important thing is to use several different particles and sprites per effect. Otherwise it looks artificial and quickly burns in to pure white. For example - let's say you want to make a fire effect. A typical newbie way to do it would be to spawn several yellow and red flares or smoke puffs. A better way is to make a custom flame sprite that fits your game style, and then break the effect like this:
- Create the main flame particle using the said sprite.
- Make another flame particle using the same sprite. Make it smaller, faster, and longer lasting. It will make the effect's pattern harder to notice and overall more natural.
- Have small flares/sparks shoot upwards with some gravity to simulate burning matter and ash.
- Have larger flares with small alpha pop up from time to time to simulate flashes of light.
- Add a slight smoke above it all (optional).
This way you can use less particles per second, and create much more believable and natural looking effects. People are very good at discerning patterns and will see through an effect consisting of only one particle. Using more than two is usually enough to trick the brain into thinking it's something actually natural and unpredictable.
A bit old, but a good example:
http://youtu.be/hwtawrNs5IM?t=37s. Notice how the meteor effect is done. It uses two particles with custom flame sprite for the main trail, a flare on the tip to make it glow naturally, and then there are those delicate shockwaves on the sides. Looks much better than fireballs in Magi.