next »

[muku]

Hey all. After two days of hacking away at it (two nights, more like), may I present to you my craptacular procedural improviser getting funky:

http://www.box.net/shared/b2kvacxgco

It's almost 5am here, so... more info tomorrow, if anyone wants it.



UPDATE: Here's a link to a flash widget where you can directly play (without downloading) all mp3s I've created so far.
http://www.box.net/static/flash/box_explorer.swf?widgetHash=yiivqi0w0o&cl=0

[imaginationac]

Hmm...interesting. Has a jazzy feel too it.

I'll be waiting to hear how you made this

[J]

Procedurally generated music, that's something i haven't heard a lot about (actually, haven't heard about it period outside of Spore), but a very interesting concept. Song actually sounds good too (I came in here with the assumption that the notes would be all over the place), and like imaginationac said, it does hold a jazzy feel to it.
Very well done, interested to know how it was created.

[muku]

Ah, a good 7 hours worth of sleep, nothing quite like it

Well, I've been working on a very procedural-content-heavy concept (main inspirations: Dwarf Fortress and Spore) for months, on and off, and I've always had vague fantasies how complementing that with a procedural soundtrack would be very cool. Well, a few days ago I finally got curious enough to actually give the idea a try; I figured I had the music theoretical background basically down (I dabble in jazz guitar, you see), so I got hacking.

First things first, choice of language. I played around in puredata for a while, and it's a really cool environment with tremendous possibilities, but me, I'm simply much quicker writing good old-fashioned code instead of dragging boxes around. So, a "real" language. While my main project is in D (which I can only recommend, it's such a boost of productivity over C++), I figured for this extremely explorative experiment something even more dynamic would be a good match, so I settled for Python. The immediate edit-run cycle turned out to be invaluable for testing, as compared to the edit-compile-run of statically compiled languages. Of course, should this ever get to a really usable state, I would reimplement it in a more static language.

Second, I needed a way to actually coax notes out of the speakers. Perhaps there are free softsynth libraries which can do this easily (I'm curious, are there?), but the easiest way to go seemed to be MIDI. In reality, it turned out there's a surprising scarcity of non-bitrotted MIDI libraries in Python (for a language which otherwise seems to have libraries for everything and the kitchen sink), so in the end I settled for a combination of the C portmidi library (which I had to compile myself too, what a hassle) and a crappy ctypes wrapper called portmidizero around that for the Python side. I actually had to fix bugs in that wrapper as I went as it would sometimes crash randomly, but eventually I was able to output notes conveniently to the soundcard and I was good to go. Of course, the basic cheap-soundcard-MIDI sound is awful, but hey, it's an experiment.

So next, the really interesting stuff, the music. I started teaching Python about scales, so my program knows about the major and minor scale as well as the modes of the major scale, i.e. Dorian, Mixolydian, that sort of stuff (if you know a bit about music theory, that should sound familiar).
Then I taught it to form simple seventh chords over a scale by just doing basic thirds stacking, and finally I implemented chord progressions so I could do stuff like a I-IV-V-I progression. I actually did some experiments with that particular progression, but it just sounded awfully stilted and formulaic, so for now I've settled with a more modal approach where I mostly harp on one root chord.

Somewhere along the way I implemented a simple pseudo-walking bass which would simply play chord notes on every beat as well as sometimes chord extensions in between, which actually sounds pretty nice for a first experiment.

As for rhythm, I started with a class which would accept patterns in a form like 'XOXO' and translate that into proper timings. That turned out to be a great concept as I could then very easily write algorithms to generate patterns or create variations on them as they are simply strings.

There are no drumloops so far, but with this rhythm system in place (which is used for all instruments so far), it shouldn't be hard to do something basic.

The last thing I've done so far is the solo-type melody you hear in the mp3, and that is basically a very simply Markov process for the notes with some more or less randomly chosen note durations, with care taken that it realigns with the beat every once in a while so that it doesn't sound too random. The jazzy sound of it might on one hand come from this whole jazz theory thing I've put into it; on the other hand, it actually seems simpler to me to randomly create the kind of jagged, syncopated lines characteristic of jazz/funk improvisation as compared to simpler, more catchy melodies. Though I definitely want to try the latter. The current form of "melody" would get on the player's nerves within minutes, so I need something more subdued as well.

At the moment, the structure of it all is very hardcoded. There probably is a wide range of stylistic variations possible already (I haven't even played with the parameters all that much), but it's not readily accessible. The ultimate goal would be to make this into a (open-source) library which everyone could integrate into their game; ideally, you would simply give it parameters like levels of "danger", "calmness", "excitement" etc and it would generate a soundtrack in realtime for you. That's of course a long way off. But it's nice to dream, isn't it?

Well, that turned out much longer than I hoped. Sorry, I've got an annoying tendency to ramble.

[J]

Now that was quite music jargon heavy, but i understand the basic idea of what's going on. Great to see how much thought and effort you've actually put into this thus far.

The ultimate goal would be to make this into a (open-source) library which everyone could integrate into their game; ideally, you would simply give it parameters like levels of "danger", "calmness", "excitement" etc and it would generate a soundtrack in realtime for you. That's of course a long way off. But it's nice to dream, isn't it?

That would have to be one of the best idea's i've heard in a long time, i'm sure you just got the support of every indie developer, especially those who've had trouble in the past finding music composers for their games.

Well, that turned out much longer than I hoped. Sorry, I've got an annoying tendency to ramble.

'Contrary, was quite an interesting and enjoyable read

[muku]

Turns out this thing is insanely fun to play around with once you start tweaking stuff. I've just written a new melody generation algorithm which produces more laid-back, ambient type melodies. The result is still a bit too random for my tastes, but I think this has potential. So, listen here:

http://www.box.net/shared/r6j1jtla80

I went for a more peaceful ambiance this time around. Also, shamisen FTW.

EDIT: Also, the first post now has a link to a flash widget with all the mp3s.

[muku]

Heh. How 'bout some good old-fashioned true procedural metal? Perfect for those blood-thumping boss fights, I reckon. See Flash link in first post to listen.

It has much the same problems as the last one, being just a bit too random, plus the solo notes sometimes get a bit too high. But that's okay, at the moment I'm just sketching out different ways in which this could go, I can finetune later.

EDIT: Bonus points for anyone who can identify in which modes the musical snippets so far were

[Lord Ash]

muku, that is awesome !!

and all that just from some maths huh, way cool !

[PaulMorel]

Good stuff, ash!  I have always wanted to implement "real" procedural music (as opposed to the pseudo-procedural music in my PGC entry, Last Colonists), but I have never really had the time.

There is a nice piece of professional software devoted to procedural music that I thought you might like.  It's called Noatikl, and it can be found here: http://www.intermorphic.com/tools/noatikl/

I have been very close to buying it for about a year, but I have never been able to really justify it as anything other than a toy.

I also wanted to note that there are a number of algorithms for generative music that you might find interesting.  If you already have the framework set up, then implementing the algorithms shouldn't be too tough.  I wish wikipedia had better entries on these...
http://en.wikipedia.org/wiki/Algorithmic_composition
http://en.wikipedia.org/wiki/Generative_music

Specifically, Chomsky's linguistic models have been used very effectively.  Those models give the computer a knowledge of music syntax.  The computer knows what it is doing now and what it has done in the past, so it can make an educated guess about what to do next based on a sort of musical grammar.

And also the mathematical models have been used to great effect.  As opposed to Chomsky's style of model, mathematical approaches rely less on an understanding of the rules of music (a dominant harmony usually follows a subdominant), and more on numbers-based, probabalistic approaches.

Obviously, that's a vast oversimplification, but hopefully that's useful to someone...

[muku]

Good stuff, ash!

You got the names mixed up 

There is a nice piece of professional software devoted to procedural music that I thought you might like.  It's called Noatikl, and it can be found here: http://www.intermorphic.com/tools/noatikl/

Might have to look at it, perhaps there's a demo...

I also wanted to note that there are a number of algorithms for generative music that you might find interesting.

Definitely. So far, I just went off my intuition for algorithms, but it would certainly be nice to benefit from the research others have done in that area. You wouldn't happen to have any links to papers which describe relevant algorithms? I have just started scouring Citeseer, that might turn up some interesting stuff.

On the other hand, I want to keep computational effort down so that generating music in realtime in an actual game would still be feasible, but it certainly can't hurt to look at the state of the art in algorithmic music.

One "problem" is that much of the generative music community seems to be concerned with generating rather abstract soundscapes instead of more traditional interpretations of music. That's of course a valid artistic choice, and it certainly makes sense to go for new directions which only algorithmic music generation has made possible, but I here am going for more conventional stuff that you might find in your average indie game soundtrack.

[Alex May]

I'm extremely impressed with this. Good work! Look forward to seeing how it comes along.

[muku]

Thanks for all the encouragement, I appreciate it. I've just skimmed an interesting paper off Citeseer which described an approach to create jazz solos via a genetic algorithm. I'm itching to put some of those ideas into code (AI buzzword alarm notwithstanding), though I'm a bit wary about performance issues. But we'll see.

For now, though, I have to put some hours into real-life stuff. Curse you, diploma thesis! *shakes fist*

[J]

Heh. How 'bout some good old-fashioned true procedural metal? Perfect for those blood-thumping boss fights, I reckon. See Flash link in first post to listen.

It has much the same problems as the last one, being just a bit too random, plus the solo notes sometimes get a bit too high.

Despite that one instance of insanely high tones compared with the rest of the piece, i'm still astonished that these are procedurally generated, very well done, loved the metal.

[PaulMorel]

Sorry for mixing up the names.  To be more correct: Good stuff, Muku!

On the other hand, I want to keep computational effort down so that generating music in realtime in an actual game would still be feasible, but it certainly can't hurt to look at the state of the art in algorithmic music.

One "problem" is that much of the generative music community seems to be concerned with generating rather abstract soundscapes instead of more traditional interpretations of music. That's of course a valid artistic choice, and it certainly makes sense to go for new directions which only algorithmic music generation has made possible, but I here am going for more conventional stuff that you might find in your average indie game soundtrack.

The first problem there is definitely a problem.  Computational complexity can get quite high, and when you need to spend most of your time painting the graphics, it's tough to devote any extra time to the music.  This is compounded when you consider the memory required for realistic sampling.  If you want patches that sound better than general midi, then you are stuck devoting tons of memory to audio files.  The extra CPU and memory use is why no mainstream games really use generative music.  AFAIK Spore will be the first to do truly generative music, and even then, I bet that they simplify it in a bunch of ways (using synths instead of sampled instruments, simplifying the math to make it quicker .. etc).

That second problem is only illusory.  There is a TON of research on generative music and the algorithms to do it.  I wish I had some of them handy to show you.   

One relatively new music model that I absolutely LOVE is being peddled by a guy named Dmitri Tymoczko.  His model is amazing, not only in its completeness, but also in its complexity.  It has massive implications for generative music, but is probably far beyond the scope of your project.  I wrote about it on my blog last week: http://www.thisisnotalabel.com/talk.php?bp=524

[Cymon]

Okay, my 2 cents. When I was younger I played a game called Star Control and I thought it would be awesome if when I got older and could write a new Star Control if the melee music would shift depending on who was winning. I devised a method where by two songs that each had similar hits could be faded back and forth. Then I thought that would be boring and fiddled with something like this, where the music would actually be generated by the variables of the game. This would be perfect for that since in Star Control Melee the music really just sets the mood.

Also, didn't Douglas Adams write about something like this in "Dirk Gently's Holistic Detective Agency"?

Now cool as it is that you have music that was created san human hands, it's not very good music. It's still kinda random and it lack a "hook". However, can these things be added without destroying (or limiting) the procedural nature of the output? How would using variables decided at the beginning to set a beat box and keep it going the whole time to give it continuity?

[increpare]

Those are pretty effective dude!  The pieces sound quite structured, but nonetheless, you didn't mention anything about imposing any rigid form on them? Have you thought much about that? (even if just at the level of chords instead of melodies).  The markov-stuff approach produces some really strong melodies.

Dmitri Tymoczko

I've read a number of things by him.  His papers are v. interesting though, at least when I was following his publications (I've not been keeping up for the past three years or so), he said to me that he wasn't personally interested in generative procedures, but rather in musical analysis of existing works, and this is where his strongest contributions for the moment seem to lie.  His use of maths is solid (though some might question the purpose of some of his axiometizations).   That's not to say it doesn't hold a promise for these sorts of approaches.

There's a guy called Fred Lerdahl who's published a lot of (I think) quite influential influuntial stuff about Chomskyian-style musical analysis, which might work quite well for generative music stuff.  It's been sitting beside my bed for a while, but I haven't yet been able to fully get into it and figure out what makes it less arbitrary than Schenkerian analysis.  This approach would also probably be quite good for dealing with chord progressions.

(My readings have been generally more about musical analysis than generative stuff).

I've written a few programs to help me out with composition stuff, but mostly these have been focused around the theories of Guerino Mazzola (which far outstrip anything else I know of in terms of their mathematical complexity, but I also feel they're a lot less trustworthy/obfuscatory; even after a couple of years dealing with them, I still don't really know what's going on with much of his theory).  He has an interesting model of modulation that I found diverting to code, but yeah.  (His work overlaps stylistically in many ways with DT's, but overall I find the latter's stuff to be much more theoretically sound, incorporating as it does many, many times more detailed musical examples than GM's).

[muku]

Okay, my 2 cents. When I was younger I played a game called Star Control and I thought it would be awesome if when I got older and could write a new Star Control if the melee music would shift depending on who was winning. I devised a method where by two songs that each had similar hits could be faded back and forth.

My grand scheme is actually that, once I have the generation of basic music building blocks down, to introduce something like this, where the mood could shift smoothly from, say, calm to danger within a few bars. But, one thing after another.

Also, didn't Douglas Adams write about something like this in "Dirk Gently's Holistic Detective Agency"?

No idea. Only thing I ever read by him were two or three from the Hitchhiker series. Loved them though.

Now cool as it is that you have music that was created san human hands, it's not very good music. It's still kinda random and it lack a "hook".

Of course. Keep in mind these are the results of two or three days of hacking; basically I only have the framework down and built some very crude algorithms on top. This is the reason I'm currently looking into more sophisticated algorithms, the research of people who have spent many years thinking about this stuff.

However, can these things be added without destroying (or limiting) the procedural nature of the output? How would using variables decided at the beginning to set a beat box and keep it going the whole time to give it continuity?

The chords do that pretty much, actually. I usually randomly set up some rhythmic accents like 3-3-2 or whatever at the start of each section and then introduce minor variations into that via a concept I called micropatterns. I think the results are okay so far.

The major problem of course are the melodies. I've tried several approaches (in fact the melody of each of the three samples was generated by a different algorithm), but none of them was completely satisfying so far (I mentioned that). Well in fact I still quite enjoy the first one for what it does, as a sort of one-trick pony, but it doesn't easily generalize to other styles.

The thing is that the problem is hard. Getting a program to introduce themes, making variations on it, writing a hook, all these without sounding neither too random nor too formulaic; that basically means teaching the computer creativity, and it's crazy difficult. I'm not sure yet how far serious research has progressed on that front, so it might even be infeasible at this time. But, I'm reading into it.

[muku]

The first problem there is definitely a problem.  Computational complexity can get quite high, and when you need to spend most of your time painting the graphics, it's tough to devote any extra time to the music.  This is compounded when you consider the memory required for realistic sampling.  If you want patches that sound better than general midi, then you are stuck devoting tons of memory to audio files.  The extra CPU and memory use is why no mainstream games really use generative music.  AFAIK Spore will be the first to do truly generative music, and even then, I bet that they simplify it in a bunch of ways (using synths instead of sampled instruments, simplifying the math to make it quicker .. etc).

I've been thinking about the synthesis part. Sooner or later, some kind of synthesis will of course be required, and one really has to watch CPU and RAM usage there. I think the most simple, feasible approach would be something in chiptune style. I've played around with FamiTracker yesterday night, just to get an idea what the capabilities of such a thing are, and I think it could work; you can't have chords, obviously, but you can just arpeggiate them. It might work.

So, if someone feels like taking one of the chip emulator libraries at http://www.slack.net/~ant/libs/audio.html and writing a thin MIDI-like layer over the top of it (basically, I just need note on and note off events), I'd be so over it Shouldn't even be that much work, I suspect.

One relatively new music model that I absolutely LOVE is being peddled by a guy named Dmitri Tymoczko.  His model is amazing, not only in its completeness, but also in its complexity.  It has massive implications for generative music, but is probably far beyond the scope of your project.  I wrote about it on my blog last week: http://www.thisisnotalabel.com/talk.php?bp=524

Thanks for the article, I read it. The math isn't all as deep as it's made out to be, and I agree with increpare that it seems to be more a formulism for analyzing existing work instead of something that would help a lot with generative music; at least that was my first impression. Still, very interesting read.

[muku]

Those are pretty effective dude!  The pieces sound quite structured, but nonetheless, you didn't mention anything about imposing any rigid form on them? Have you thought much about that? (even if just at the level of chords instead of melodies).  The markov-stuff approach produces some really strong melodies.

So far, the macro-structure of the examples I gave was hardcoded. That is, I usually specified some 4 bar chord progression and looped over that several times; only the structures at a smaller level (rhythm patterns, melody etc.) were procedurally generated. I'm going for a bottom up approach here: first get the basic building blocks working, then scale upwards and give the computer another higher level of structure to work with.

It's true that the Markov process melodies do sometimes produce interesting stuff, but I mostly agree with guesst that by and large they still sound too random. I think the problem is that these simple Markov chains are very local processes; you have one note and going from that you decide on the next one. Of course you can give them more context, look at the last two, three notes, whatever, but computation skyrockets and at its core it's still a local thing. From what I've read, that lack of melodic structure is an inherent limitation with "note-by-note" composition algorithms. That's why I'm reading up on some more global algorithms where you try to create an entire one-bar or even four-bar melody at once while taking some kind of "fitness" into account. Basically, it's a discrete optimization problem in some rather high-dimensional space. Some people seem to use genetic algorithms, but I'm sure there are other approaches as well.



Thanks for the other pointers, I'll check out those Lerdahl and Mazzola guys if I get the chance. There really seems to be a hefty amount of literature in the area.

[ravuya]

Jeff Minter experimented with fractal music in his 90s Camels game for the Konix. I've always been really partial to that approach, using the natural beauty of a fractal to make stuff for humans. I also like the idea of a background bassline and other song-global settings like tempo and key.