Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length

 
Advanced search

1362001 Posts in 63514 Topics- by 55393 Members - Latest Member: sherylbutler

June 25, 2019, 12:48:24 PM

Need hosting? Check out Digital Ocean
(more details in this thread)
TIGSource ForumsCommunityTownhallCosyne Synthesis Engine
Pages: 1 [2]
Print
Author Topic: Cosyne Synthesis Engine  (Read 17226 times)
0rel
Level 4
****


View Profile WWW
« Reply #20 on: April 08, 2010, 07:38:31 AM »

Great work muku! Coffee

I look what I can do with this new modulation feature, testing it probably today more in depth. The way you implemented the parameters seems really flexible. The documentation looks is also very useful, and I like the 1 page layout... So compact and streamlined, but still covering everything it needs. Same as with the library itself... Smiley

A small description of each interface function could also help a little maybe, to make it easier to get started, although all is quite self explanatory... The only thing I didn't get somehow was how the velocity parameter of Cosyne_PlayNote() really affects the instrument... Is it clamped to [0,255] or only 127? Also I thought, maybe it would be good to add it the same way to the instrument description language as LFOs and parameters can be used in expressions, by using a symbol like 'vel' or something, so that one could do thing like 'shape({ 0.1+vel*0.8 })'? As float parameter in that case... Just an idea though.
Logged
muku
Level 10
*****



View Profile WWW
« Reply #21 on: April 08, 2010, 08:12:46 AM »

A small description of each interface function could also help a little maybe, to make it easier to get started, although all is quite self explanatory...

Yup, it's on my mental todo list Wink

Quote
The only thing I didn't get somehow was how the velocity parameter of Cosyne_PlayNote() really affects the instrument... Is it clamped to [0,255] or only 127?

Range 0..127, for compatibility with MIDI where it's the same. Right now this just linearly scales the gain of the instrument, but I might think of something more physical here...

Quote
Also I thought, maybe it would be good to add it the same way to the instrument description language as LFOs and parameters can be used in expressions, by using a symbol like 'vel' or something, so that one could do thing like 'shape({ 0.1+vel*0.8 })'? As float parameter in that case... Just an idea though.

That's already possible! It's not yet in the docs though. Use the variable f for frequency and v for velocity. f is in Hz, v should already be scaled to 0..1.
Logged

The Cosyne Synthesis Engine - realtime music synthesis for games
muku
Level 10
*****



View Profile WWW
« Reply #22 on: April 10, 2010, 12:16:40 PM »

I sort of forgot/postponed to reply to this point.

Another solution that came to mind was to handle the interaction at a different rate than the audio API asks for updates, which can be unpredictable at times (OpenAL uses somewhat mysterious buffer queues for example). I often use a Move() function in my game code separated from Render() to handle all interactions/animations/state changes, which should be independent of the framerate. So, Cosyne_Render() could be called there at steady intervals, to render a small number of samples to a temporary soundbuffer, which would be cleared at the next audio API callback... That way the interaction (PlayNote/ChangeParam events) would affect the audio at a much finer resolution, and one wouldn't directly depend on the audio buffer size. Maybe I'll try that out this way... It isn't really a issue of your library though, more about how to use it.

It's already possible to schedule PlayNote/ChangeParam events at a quite fine resolution by just specifying a suitable time value when you call the function; no need to call Render() several times per sample.

The issue I see with your suggestion though is that, as far as the game logic is concerned, everything that happens during one frame happens conceptually at the same time. Just because you detect, say, one collision before another in your Move() function doesn't mean that they necessarily happened in this order. So I don't see how it would be very useful to do this. Or are you talking about a separate thread which is not in lockstep with the main game logic? I'm haven't used OpenAL at all, only just read up on how it does audio streaming, so I may be missing something.

I think the problems you saw with high latencies stemmed solely from the size of your buffers. How many buffers did you queue on your source, and how large were they? (The SDL way with a callback which is called when samples are needed definitely seems easier here...)


Quote
Hm, that volume issue was annoying me too by the way. I also don't have a perfect solution for this though...

I just implemented something which should help a bit with that. Maybe the following information is useful to you.

I realized a while back that the linear mapping of velocity to amplitude of the generated waveform isn't a very good choice. The ear doesn't work in a linear way, so if you halve the amplitude of a signal, it doesn't seem half as loud, but typically quite a bit louder than half.

Decibel, a logarithmic scale, were invented just for this kind of thing. It turns out that a gain of -10dB (which corresponds to a gain factor of about 0.31622) sounds about half as loud as the original. So I did the math and came up with the formula v ^ 1.661 for the amplitude scaling, where v is the velocity scaled to the range 0..1. With this formula, every time you halve the velocity, you will get a relative gain of -10dB and thus an approximate halving of the perceived loudness, which I think is a reasonable assumption to make.

Here's a simple example to compare the two methods: The notes in these audio samples increase/decrease in velocity by 16 from one to the next. The first sample is with the old linear velocity mapping, the second with the new nonlinear one. I think the second one sounds quite a bit more natural. The new method also gives you more headroom for mixing together multiple voices.

* linear velocity
* nonlinear velocity

So this will go out with the next update.
Logged

The Cosyne Synthesis Engine - realtime music synthesis for games
0rel
Level 4
****


View Profile WWW
« Reply #23 on: April 10, 2010, 05:34:01 PM »

Sorry I didn't reply yet, muku. Just wanted to finish the new experiment first, but it's still not ready yet...
But if you want to try it out, here's the current state. I've tried to combine my lib with Cosyne this time, and technically it worked fine (needs lots of CPU though). The sound is not that much interactive though. Clicking and dragging changes the filter frequency of a simple cosyne instrument, the background is done with es. Initially I wanted to bind some interactive sound function to the moving objects... Probably I try to do that next. But technically the ChangeParam-feature worked immediately. Nice!

Quote from: muku
Range 0..127, for compatibility with MIDI where it's the same. Right now this just linearly scales the gain of the instrument, but I might think of something more physical here...
I already expected that somehow. MIDIII alert!

Quote
Use the variable f for frequency and v for velocity. f is in Hz, v should already be scaled to 0..1.
Ah that's pretty useful I think. Per-note-modulations can help a lot to achieve interesting sound variations...

Quote
It's already possible to schedule PlayNote/ChangeParam events at a quite fine resolution by just specifying a suitable time value when you call the function; no need to call Render() several times per sample.

The issue I see with your suggestion though is that, as far as the game logic is concerned, everything that happens during one frame happens conceptually at the same time. Just because you detect, say, one collision before another in your Move() function doesn't mean that they necessarily happened in this order. So I don't see how it would be very useful to do this. Or are you talking about a separate thread which is not in lockstep with the main game logic? I'm haven't used OpenAL at all, only just read up on how it does audio streaming, so I may be missing something.
Hm, maybe I'm worng about this... But I currently render the samples in one go in my drawing function (whenever OpenAL needs a buffer update, what is not the case for every frame... I sometimes make the buffer as big as 8192 samples or even larger (to keep it glitch free on mostly all machines), which leads to only ~5.4 updates per second. For OpenAL streaming I fixed the buffer count to 3. That sadly leads to lots of latency :/... But I just like OpenAL because it also provides cross-platform 3d sound, otherwise, another audio library (like RtAudio) might be a little faster maybe...).

During a buffer update I only know the current state of the game, so it's difficult to be accurate about the timing. But I see what you mean about that collision example... When I want to sync the collision sound to a global rhythm, I need to know the time offset to the next beat (Cosyne_GetCurrentTime() + offset). To solve this, I should probably store the current time after each Cosyne_Render() call, and then just increase it in every Move() call accordingly (as a double float)... - Although everything happens with a certain delay, it's still important to keep the timing as accurate as possible (44100 vs 25 FPS, we can hear so highres "temporal subtleties" in sound)... Otherwise I would be triggering all notes with the same time stamp Cosyne_PlayNote( instr, -1...), which necessarily leads to inconsistent timing...

Quote
It's already possible to schedule PlayNote/ChangeParam events at a quite fine resolution by just specifying a suitable time value when you call the function
So, yes, that's the solution for this, I guess. (But this point was really only about how to use the lib, not about the functionality it provides.)

Quote
Decibel, a logarithmic scale, were invented just for this kind of thing. It turns out that a gain of -10dB (which corresponds to a gain factor of about 0.31622) sounds about half as loud as the original. So I did the math and came up with the formula v ^ 1.661 for the amplitude scaling, where v is the velocity scaled to the range 0..1. With this formula, every time you halve the velocity, you will get a relative gain of -10dB and thus an approximate halving of the perceived loudness, which I think is a reasonable assumption to make.
Oh, that' actually really useful! Thanks for clearing that up. I always had troubles with dB scales and such... And in my synth, I also always used linear scaling, just to keep it simple. But it actually always felt unnatural to me... I surly will try your method now Smiley. - I also have the impression that the non-linear relationship between frequency and perceived loudness is a similar problem like this (keyword Psychoacoustics). But this remains an open issue in synth as well... I know that many soft-synths often have a key/pitch parameter to cope with that sort of thing, but I never really understood the maths behind it... Maybe I just need to scale the amplitude by some scale factor like (new_amp = amp * freq ^ x)? Didn't try that yet. But similar to the velocity scaling that might lead to a more natural sounding instrument on the whole range of the keyboard...

Oh and just keep up the good work! I think it's really great. I guess people here are not that much into interactive audio, to play around with it just for fun...
For me, it's still a main interest though, although my skills are still very limited when it comes to making actual music... But I want to learn more about music theory next, and do more "games" in that direction probably.
« Last Edit: April 10, 2010, 05:45:51 PM by 0rel » Logged
muku
Level 10
*****



View Profile WWW
« Reply #24 on: April 11, 2010, 03:02:22 AM »

Sorry I didn't reply yet, muku. Just wanted to finish the new experiment first, but it's still not ready yet...
But if you want to try it out, here's the current state.

Unfortunately that doesn't run for me Sad Just sits and eats 100% CPU, no window, no sound. No error messages on the console either.

Quote
Quote from: muku
Range 0..127, for compatibility with MIDI where it's the same. Right now this just linearly scales the gain of the instrument, but I might think of something more physical here...
I already expected that somehow. MIDIII alert!

In fact it shouldn't be too hard to implement a MIDI interface so that you can actually play Cosyne from a real MIDI keyboard. I might do that sometime, but right now it's not such a high priority. I have even more grandiose dreams with producing an actual VSTi version of Cosyne so that you can compose music for it in some sequencer and then import it into your game as a simple MIDI file... but just a dream at the moment.


Quote
Hm, maybe I'm worng about this... But I currently render the samples in one go in my drawing function (whenever OpenAL needs a buffer update, what is not the case for every frame... I sometimes make the buffer as big as 8192 samples or even larger (to keep it glitch free on mostly all machines), which leads to only ~5.4 updates per second. For OpenAL streaming I fixed the buffer count to 3. That sadly leads to lots of latency :/...

So you effectively have 3*8192 samples worth of buffering, which at 44.1kHz corresponds to a latency of 0.56s... that can't be right though, while your first example lagged quite a bit, I don't think it lagged that bad? Or maybe you don't always write into the "last" buffer, so that it's sometimes better than that?

I produced a little graphic to maybe elucidate the situation:

Time moves from left to right. Let's just view time as segmented into an endless stream of equal-sized buffers (the black boxes), which you essentially emulate in OpenAL by unqueuing fully played buffers and re-enqueuing rendered buffers.

The blue line marks the current playback position of the sound card, and the red line marks the sample position at which Cosyne is currently rendering. Of course you always have to keep the render position in front of the playback position, or skipping will occur. The render position is advanced in chunks by calling Cosyne_Render(), and the playback position advances more or less continuously with time.

Cosyne effectively allows you to place events freely in the gray (empty) area at a fine resolution (64 samples) by specifying a time when calling PlayNote() and the other event functions.

It seems that OpenAL lets you determine the playback position by querying the AL_SAMPLE_OFFSET source property. So maybe you could, whenever you add a new note event, take that position, add a constant offset to it and use that as your note time, completely ignoring the Cosyne render position? That should ensure a more or less constant latency for all notes, no matter their position within a buffer.

Of course you still have to ensure that there is always a filled buffer available for the sound card, but yet the render position doesn't advance beyond your fixed latency offset. It's a bit of a balance act. Maybe you can always render just enough samples so that the render position is, say, 1.5 buffers ahead of the playback position? In other words, don't always render one full buffer in one frame, but just enough samples to ensure this, and when a buffer is filled enqueue it.

Do I make sense? I'm not sure.


Quote
Oh, that' actually really useful! Thanks for clearing that up. I always had troubles with dB scales and such... And in my synth, I also always used linear scaling, just to keep it simple. But it actually always felt unnatural to me... I surly will try your method now Smiley. - I also have the impression that the non-linear relationship between frequency and perceived loudness is a similar problem like this (keyword Psychoacoustics). But this remains an open issue in synth as well... I know that many soft-synths often have a key/pitch parameter to cope with that sort of thing, but I never really understood the maths behind it... Maybe I just need to scale the amplitude by some scale factor like (new_amp = amp * freq ^ x)? Didn't try that yet. But similar to the velocity scaling that might lead to a more natural sounding instrument on the whole range of the keyboard...

Nice that it was useful. For the frequency/amplitude thing, I'm afraid it's more complicated than that. Basically you'd have to implement something like equal-loudness contours (Fletcher-Munson or somesuch). The problem is that these only hold for pure sine waves at a given frequency, while a typical synth voice will have a spectrum which extends far into the overtones. So I think there is no way to solve this at the note level. The real way to solve it would probably be to do equalization as a post-processing step: Once you have your final output all mixed together, do a Fourier transform, scale each frequency band according to some loudness contour, and then do the inverse Fourier transform. That's of course way expensive, and I'd rather stay out of that.


Quote
Oh and just keep up the good work! I think it's really great. I guess people here are not that much into interactive audio, to play around with it just for fun...

Yeah, of course I can't expect everyone to jump on this immediately Smiley But I don't mind, I'm mainly doing this for my own interest (and to have it available for some of my game experiments in the future). That at least one other person (you) is interested in this stuff is great already. Smiley
« Last Edit: April 11, 2010, 03:05:23 AM by muku » Logged

The Cosyne Synthesis Engine - realtime music synthesis for games
increpare
Guest
« Reply #25 on: April 11, 2010, 03:21:43 AM »

Sorry I didn't reply yet, muku. Just wanted to finish the new experiment first, but it's still not ready yet...
But if you want to try it out, here's the current state. I've tried to combine my lib with Cosyne this time, and technically it worked fine (needs lots of CPU though). The sound is not that much interactive though. Clicking and dragging changes the filter frequency of a simple cosyne instrument, the background is done with es. Initially I wanted to bind some interactive sound function to the moving objects... Probably I try to do that next. But technically the ChangeParam-feature worked immediately. Nice!
Works okay for me, fwiw.  I had thought you had added in ambient sound effects, of someone typing in the background, but it turns out my work computer is streaming in the flapping of the blinds from the server room for some strange reason... .  The music I found to be a little monotonous for my tastes.
Logged
muku
Level 10
*****



View Profile WWW
« Reply #26 on: April 11, 2010, 08:11:27 AM »

Haha, am I the only one who is reminded of old LucasArts adventures by this here? I love it Corny Laugh
Logged

The Cosyne Synthesis Engine - realtime music synthesis for games
0rel
Level 4
****


View Profile WWW
« Reply #27 on: April 11, 2010, 06:22:03 PM »

Quote from: muku
Unfortunately that doesn't run for me Sad Just sits and eats 100% CPU, no window, no sound. No error messages on the console either.
Maybe we'll be able to fix that over PM...

Quote
In fact it shouldn't be too hard to implement a MIDI interface so that you can actually play Cosyne from a real MIDI keyboard. I might do that sometime, but right now it's not such a high priority. I have even more grandiose dreams with producing an actual VSTi version of Cosyne so that you can compose music for it in some sequencer and then import it into your game as a simple MIDI file... but just a dream at the moment.
The method I used in 'es' was to implement MIDI over the standard MIDI interface from the OS (but RtMidi cloud also be helpful here), and to play with it over a software sequencer, I use LoopBe1 to connect them virtually (only needed on Windows). That works pretty well. But I also heard about some advantages of Open Sound Control over MIDI... I don't have any experience with that yet, though.

Quote
So you effectively have 3*8192 samples worth of buffering, which at 44.1kHz corresponds to a latency of 0.56s... that can't be right though, while your first example lagged quite a bit, I don't think it lagged that bad? Or maybe you don't always write into the "last" buffer, so that it's sometimes better than that?
I'm also not sure about this, since OpenAL handles this internally. I'm just updating buffers  whenever they have been marked as fully processed, so when there's only one at a time, then OpenAL might (theoretically) only uses 2 buffers actively.. One currently playing and the other updated/enqueued (since it doesn't use a typical ringbuffer with read/write position). But I've read somewhere, that's this is actually not the case, it just goes from one buffer to the next all the time. Confusing... Actually I don't want to care too much about this buffer queuing issue, since it's quite specific. I like to hide that stuff in the wrapper class and have just a callback, which requests a certain number of samples (one buffer in this case) at a time...

At the end I'm just feeding OpenAL with chunks of audio, one after another, which are played back with a certain lag then. I think the queuing is just another way to handle that interface.

In DirectSound for example, there's just one single ringbuffer, which allowed write and read access to the user all the time, which can be quite tricky to handle. OpenAL's queuing lets you only write in offline buffers, and only reads from locked buffers in the queue, and finally also writes to a hardware ringbuffer internally... Which is a somewhat cleaner interface than the one of DirectSound, although it adds some additional overhead...

Quote
It seems that OpenAL lets you determine the playback position by querying the AL_SAMPLE_OFFSET source property. So maybe you could, whenever you add a new note event, take that position, add a constant offset to it and use that as your note time, completely ignoring the Cosyne render position? That should ensure a more or less constant latency for all notes, no matter their position within a buffer.
I haven't used AL_SAMPLE_OFFSET with streaming buffers yet. But I think it could get rather difficult... I'd rather like to advance a time variable independently of OpenAL according to the fixed interval Move() gets called, and then use that as a start time for all Cosyne events. I'm almost sure this could work well, with a fixed lag of course.

Quote
For the frequency/amplitude thing, I'm afraid it's more complicated than that. Basically you'd have to implement something like  (Fletcher-Munson or somesuch). The problem is that these only hold for pure sine waves at a given frequency, while a typical synth voice will have a spectrum which extends far into the overtones. So I think there is no way to solve this at the note level. The real way to solve it would probably be to do equalization as a post-processing step: Once you have your final output all mixed together, do a Fourier transform, scale each frequency band according to some loudness contour, and then do the inverse Fourier transform. That's of course way expensive, and I'd rather stay out of that.
Thanks again! That sounds interesting... Also quite complicated. I think it could also work with some simple hand-tweaked,  key factor lookup-table, or some simplified function maybe, instead of doing it analytically on the whole spectrum in frequency domain. I find it just annoying that some instruments, a simple rectangle wave for example, sound much louder in the higher octaves, than in the lower... - It seems to me that up to ~1000 Hz those diagrams almost look like an exponential curve of some sorts. And I found that this roughly corresponds to [C1,~C6] (MIDI Note 84). Maybe there's a way to model such a ReAmp( freq ) function in a much cheaper way, which also gives somewhat "natural" results... I'll try to find out more about that soon.

Quote from: increpare
Works okay for me
Thanks for testing it!
Quote
I had thought you had added in ambient sound effects, of someone typing in the background, but it turns out my work computer is streaming in the flapping of the blinds from the server room for some strange reason... .
Please record that somehow next time Smiley
« Last Edit: April 11, 2010, 06:28:05 PM by 0rel » Logged
curby
Level 0
**

bacon grills


View Profile WWW
« Reply #28 on: April 13, 2010, 06:34:27 AM »

Great stuff.  I'm a big fan of interactive audio, will check this out when I get home!   Smiley Hand Thumbs Up Left
Logged

curby: Twitter

beatnik:  Plain Sight
0rel
Level 4
****


View Profile WWW
« Reply #29 on: April 13, 2010, 08:49:37 AM »

@Buffers again:
I was mostly talking non-sense up there... This whole buffer/latency was confusing me I guess. After testing it with the method from above (calling Cosyne_Render() to render small chunks repeatedly in Move() to a temporary buffer) failed for me. To keep the "offline rendering" in sync with the audio API timing wasn't working properly after some testing... But I found a simple workaround about that: Instead of having 3 large buffers (3 x 6144 samples), I just switched the OpenAL streaming to many smaller buffers (12 x 1024 samples). That works okay, and gives enough resolution of interactive use (without having to worry about these start time offsets).

@Key-Follow / Key-Tracking Feature:
After some read up on that, it seems like it most often not really refers to that "psychoacoustic loudness issue" from above. Most often it is used to control filter frequencies according to current pitch, to add some additional expressiveness to the instruments, and to make them sound better on the whole keyboard. Other parameters, like envelopes also often get modulated by the current key. I also checked out how a synth in NI Reaktor I remembered implemented that function in form of a key-track knob... And it turns out that it really only modulates the filters, and envelope durations. The overall gain of the note keeps unchanged... Also worth mentioning is that they always use the key numbers, not the note frequency, so that the influence keeps linear over the whole range of the keyboard (so maybe 'n' besides 'f' could also be useful in the Cosyne instrument language, I thought).

And hm, that problem with these louder higher tones of rough sounding waveforms probably stems mainly from the aliasing, I think. They often sound very screechy, not effectively louder... (Psychoacoustic effects are subtler, I guess). Probably a low-pass filter can solve that easily. And if really needed, a linear ( ( midi_note - 60 ) * key_track_factor + current_gain ) could already show some sufficient attenuation to the higher octaves (untested though).

But these are all minor techy details after all... Wink
Logged
muku
Level 10
*****



View Profile WWW
« Reply #30 on: April 13, 2010, 03:29:20 PM »

Just got portamento implemented, and I think it's jolly good fun:

http://www.box.net/shared/ryv9ke1nhx (Sopwith Camel anyone? Corny Laugh)
http://www.box.net/shared/ch4j9gz2i1 (yeah sorry, it's not in time and annoying)

You get the choice between either a constant-time or a constant-rate portamento, both configurable.

0rel, I'll reply to your post tomorrow; it's late...
Logged

The Cosyne Synthesis Engine - realtime music synthesis for games
increpare
Guest
« Reply #31 on: April 13, 2010, 03:35:08 PM »

Quote
http://www.box.net/shared/ryv9ke1nhx (Sopwith Camel anyone? )
The low registration in the early part of that really got inside my ... ears, I think ... creepy...
Logged
muku
Level 10
*****



View Profile WWW
« Reply #32 on: April 14, 2010, 02:29:35 PM »

So I started feeling a bit bad about spamming the announcements forum. Let's move over to the devlog, people.
Logged

The Cosyne Synthesis Engine - realtime music synthesis for games
Nagnazul
Level 0
**


View Profile
« Reply #33 on: September 29, 2010, 12:33:41 AM »

I think this could be useful to me, I'll look into it.

Thanks for making this!
Logged
Pages: 1 [2]
Print
Jump to:  

Theme orange-lt created by panic