Blunt procedural audio library

• Source code repository
• Repository with ready-to-use examples explained below

Blunt is a procedural and generative audio library, written in C# for Unity3D. It was written specifically for the game Surfing the Wave. Blunt focuses on high-performance synthesis and a modular audio graph that, together with a algorithmic sequencing system, makes it possible to create everything from synces effects to whole generative music compositions.

Much more to come, on this article.

Contents

• Intro
• Features
• Mixer & effects
• Simple Example
• Making a simple polyphonic synthesizer sound
• Parameter modulations
• Creating a basic sequencing system
• Generative Music
• Troubleshooting
• Development Status
• License

Features

• Fast envelopes with linear or exponential behaviour
• N-operator/modulator FM/AM/additive/subtractive synthesizer
• Additive resynthesizing synthesizer
• Piano component allowing to play the synthesizer on the keyboard
• Visual mixer with measurements
• White-noise generators
• SVF resonant 12 dB bandpass, highpass and lowpass filters
• Reverb, phaser, limiter and valve overdrive effects (last 3 also as separate components)
• Linear and smooth-step parameter thread-safe modulations
• Resonators and oscillators
• Generic synth, voice and voice buffer interfaces (allowing simple voice design, stealing and allocation)
• Central place for tempo and pitch calculations, allowing global changes
• Optional SIMD codepaths for synthesizers, filters and envelopes
• General model for thread-safe programing
• Sequencing system with beat and distance calculations
• Call-back sequencing system
• Probability-based rhytmic pattern-banks sequencing
• Instrument, melody and rhytm collections into pieces/compositions that can be stopped/started/paused
• FFT's and some image signal processing algorithms

I'll begin a technical reference now, that explains how the audio system works by example. Any references to Blunt code objects lives inside the master Blunt namespace.

Mixer & effects

The basic unit of the system is the Sound.Mixer. The Sound.Mixer interconnects the Unity audio system and the Blunt system. The reason for the difference (and need) is firstly because Unity 4.0 doesn't actually have a mixer, and any object interacting with Unity needs to inherit MonoBehaviour, to override OnAudioFilterRead. This eliminates any further inheritance in your object system, since C# doesn't support multiple inheritance. This is an awkward limitation, which is also why most of Blunt's functionality is achieved through interfaces.

The Sound.Mixer's main job is dispatching sound to Blunt elements. You can have multiple mixers, but need at least one. To get started, create an empty Game Object in the Unity Editor. In the GameObject, add a Audio Source - this is a static requirement to get sound. After this, add a Blunt.Mixer component. Then, add a new script called AudioSystem. This is where you'll add your audio scripting code. Lastly, add a Limiter Effect at the end of the chain. You'll usually want to have a limiter, to avoid blowing up your speaker when working with unprotected sound (like, resonating filters and such). Your hierachy should now look like this, from the Unity Editor:

Your basic code in a top-level audio script (AudioSystem.cs) should start out like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

using UnityEngine;
using Blunt;
using Blunt.Sound;
using System.Collections;

public class AudioSystem : MonoBehaviour
{
    Mixer mixer;

    void Awake()
    {
        mixer = GetComponent<Mixer>();
    }

}

Next, you'll have to indicate the sound system is fully loaded (should be done after all components have been initialized) using mixer.enableSoundSystem();. Every audio element in Blunt (synthesizers, effects, sequencers, listeners) implements the Sound.SoundStage interface (which is just a wrapper around a process() call). The Sound.Mixer stores Sound.Mixer.MixerChannels which is a container of Sound.SoundStages. Sound.Mixer.MixerChannel processes its Sound.SoundStages serially, that is, the first Sound.SoundStage feeds into the next. Sound.Mixer.MixerChannels, on the other hand, are processed in parallel and are associative entries of the Sound.Mixer. They are summed additively to the output of the mixer.

Note that Sound.Mixer.MixerChannel itself implements Sound.SoundStage, so you can nest mixer channels inside mixer channels. The Sound.Mixer also allows listeners on channels.

Simple Examples

Now, we'll create a simple object making some sound. For this purpose we will just generate some white-noise. If you're not confident with DSP code, don't worry, you won't actually be making stuff like this - it's just to create some example sound.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

class Noise : SoundStage
{
    Utils.CheapRandom r = new Utils.CheapRandom();

    public void process(float[] data, int nSampleFrames, int channels, bool channelIsEmpty)
    {
        for (int i = 0; i < nSampleFrames * channels; ++i)
        {
            // generate a random float between -1 and 1, and scale the volume a bit.
            data[i] = r.random11() * 0.25f;
        }
    }
}

Notice the channels are interleaved in the audio buffer. Now, we'll add the audio object to our audio system object. In summary, your audio script should look something like this, non-explained steps have comments next to them.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

using UnityEngine;
using Blunt;
using Blunt.Sound;
using System.Collections;


public class AudioSystem : MonoBehaviour
{
	Mixer mixer;
	Mixer.MixerChannel channel;
	Noise noise;

	class Noise : SoundStage
	{
		Utils.CheapRandom r = new Utils.CheapRandom();

		public void process(float[] data, int nSampleFrames, int channels, bool channelIsEmpty)
		{
			for (int i = 0; i < nSampleFrames * channels; ++i)
			{
				data[i] += r.random11() * 5;

			}
		}
	}

	void Awake()
	{
		mixer = GetComponent<Mixer>();
		noise = new Noise();
		// get the channel called noise (created if it doesn't exist)
		channel = mixer.getChannel("noise");
		// add our noise object to the chain.
		channel.addStage(noise);
		// enable the channel:
		channel.setIsEnabled(true);
		// and, enable the mixer:
		mixer.setIsEnabled(true);
	}

	void Start()
	{
		// start runs after awake. Enable the mixers sound system, and 
		mixer.enableSoundSystem();
	}

}

You can now procede to start the project inside the Unity Editor. If everything goes well, it should look something like this:

As the project plays, the mixer and limiter will display diagnostics, like level meters, status, cpu usage etc. Notice that the output of the Mixer is subject to Unity's Audio Source transformation and modulation. This means the sound is spatialized in the 3D space according to its position relative to the listener (usually attached to the Main Camera or your Player). If you want to create non-spatialized sound, you can either disable it in the Audio Source, or translate the position of our AudioSystem game object to the Camera - or simply, make it a child of the Camera.

Simple polyphonic synthesizer sound

In this example, we'll design a polyphonic sound for the Synthesis.FMSynth synthesizer. Any classes from now on lives inside the Blunt.Synthesis namespace. We'll start out with the audio system object again. Besides the FMSynth, we are going to need a VoiceBuffer<Voice>. Substituting the Synthesis.FMSynth in (as Synthesis.FMSynth has of course a Sound.SoundStage interface), our code should look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

using UnityEngine;
using Blunt;
using Blunt.Sound;
using Blunt.Synthesis;
using System.Collections;


public class AudioSystem : MonoBehaviour
{
	Mixer mixer;
	Mixer.MixerChannel channel;
	FMSynth synth;
    VoiceBuffer<FMSynth.Voice> voiceBuffer;

	void Awake()
	{
		mixer = GetComponent<Mixer>();
		synth = new FMSynth();
        voiceBuffer = new VoiceBuffer<FMSynth.Voice>(synth);
		channel = mixer.getChannel("fm synthesis");
		channel.addStage(synth);
		channel.setIsEnabled(true);
		mixer.setIsEnabled(true);
	}

	void Start()
	{
		// start runs after awake. Enable the mixers sound system, and 
		mixer.enableSoundSystem();
	}

}

There is a base class called Voice and a generic interface called Synthesizer. The Voice base allow to create, play, stop, pitch and release sounds, control volume etc.; in general basic MIDI-stuff. The Synthesizer interface adds support for adding and removing voices, along with playhead/position info. In general, synthesizers are designed such that they render all the properties in the Voice. The Synthesizer instead define a more complex voice as a class member. The Voice, then, carries all state information which allows the Synthesizer to render any supported voice, and more importantly, any amount.

This practically means, that you don't do sound design on a specifc synthesizer, you design a group of voices that (may) sound exactly similar. This allows you to only have one synth for all your sound designs. This is where the VoiceBuffer<T> comes in to play. It does resource management for you, and allow to do operations on all voices at once. The voice buffer then, is able to decay to a generic polyphonic playable device / sound source linked to the actual synthesizer. This encapsulation allows generic sequencers to play a sound source, with automatic voice-stealing.

The interface we need to use is VoiceBuffer<T>.initialize. It allocates N (five in the following example) voices, and runs a lambda on each of them, that configurates the voice (thus, also the sound). This code should be a part of the Awake() function, in the end.

1
2
3
4
5
6

voiceBuffer.initialize(5,
    voice => 
    {
        // design the voice sound in here.
    }
);

Now, I don't (yet) document everything like this, so I'll advice you to have a tool with a decent kind of intellisense, since a lot of the documentation is written in doxygen-style comments (if the naming isn't descriptive enough). That aside, the FMSynth works with operators. A operator is basically a modulatable sound creating/modifying device, like a filter, oscillator, value/parameter etc. (full list can be seen in FMSynth.Voice.Operator.Kind). An operator has two destination targets/pipelines, the modulator pipeline or the sound pipeline. Modulators only have access to the modulator pipeline, while sound modifiers has access to both. At the end of both pipelines, the operator's operation - whether it is added accumulatively to the sum of the chain, or if it is multiplied with the rest of the sum (along with a mix parameter). This system allows anything to audio-rate modulate any other arbitrary modulator - adding parameters, you can change mostly anything on the fly. With that in mind, let's design a sound.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

voiceBuffer.initialize(5,
    voice => 
    {
        FMSynth.Voice.Operator op;

        // Let's setup some FM. First we create a modulator. They are sine waves by default, and has additive operation.
        op = voice.createAndAddOperator();
        // Set the envelope to be constant, and have a long release.
        op.envelope.setADSR(0, 0, 0, 100000);
        // Patch sound through the modulator pipeline.
        op.isModulator = true;
        // Set the output volume of the modulator. This directly affects the FM-index (and sidebands).
        op.volume = 0.3f;

        // Create the carrier. Again, it's just a additive sine wave by default.
        op = voice.createAndAddOperator();
        // Set the envelope to a 2 ms sharp attack, followed by a medium decay down to -64 dB.
        op.envelope.setADSR(2, 1000, -64, 1000);

        // Now we create a LP filter modulated by a LFO.
        // First setup the LFO, so it feeds into the filter.
        op = voice.createAndAddOperator();
        // Set a long attack on the LFO, and make it stay there (sustain is 0 dB)
        op.envelope.setADSR(1000, 0, 0, 10000);
        // Make it a modulator, again.
        op.isModulator = true;
        // Any operator has a harmonic relationship to the base pitch (which the synth decides).
        // This controlled through op.pitchRelation, and is 1 by default.
        // However, if we set op.isFixed to true, the pitch relation is ignored and assumes a fixed value.
        op.isFixed = true;
        // Since no relation is specified, we have to set the frequency (set through omega) ourselves.
        // The EnvironmentTuning specifies a set of variables related to tuning and tempo.
        // This one in particular is tied to the BPM, so we scale it to create a beating tempo-synced frequency.
        op.omega = EnvironmentTuning.bpmFreq * 2;
        // Modulations to filters is equal to cf * 2^(mod), so since the filters base frequency is 50,
        // the filter cutoff will modulate between 12.5 and 200 (volume scales modulations, of course)
        op.volume = 4f;
        // alter the starting phase of the modulator by 3/4.
        op.phaseOffset = Mathf.PI * 1.5f;

        // create the filter that recieves the modulation, with a cutoff of 50 hz, and fairly resonant (5)
        op = voice.createAndAddFilter(FMSynth.Voice.Operator.FilterTypes.LP, 50, 5);
        voice.setPitch(220);
    }
);

This should sound like some LFO-filtered FM. It is quite some code, but it demonstrates some of the possible concepts. You can check out some sound designs I did for Surfing the Wave in this file. Now we only need a mechanism to trigger the sounds, maybe a keypress?

1
2
3
4
5
6
7
8

void Update()
{
    if(Input.GetKeyDown(KeyCode.A))
    {
        // steal the oldest voice, and play it!
        voiceBuffer.stealOldestVoice().play();
    }
}

The synth should now generate the sound polyphonically, when you press the 'a' button and the Unity project is running. If you want a more advanced 'piano player', check out this file. Note the previous file has now been integrated in the examples project here.

Parameter modulation

In this example, we'll see how we can alter the sound of the synthesizer in real-time, by using parameters. Parameters are basically modulators (though they don't need be), outputting linear interpolations from current to next value, over some time interval. Firstly, we'll design a sound where we will add a control to modulate the FM-index using a random device, added in the class:

1

Utils.CheapRandom rnd = new Utils.CheapRandom();

Parameters are created through FMSynth.Voice.createAndAddParameter, and functions exactly like the other operators - that is, they generate a sound. Through modulations, you can control filters, offsets, phases and using multiplicative operation, you can control the amount of modulations based on external inputs. Without further ado, lets get to the sound:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

voiceBuffer.initialize(10,
	voice =>
	{
		FMSynth.Voice.Operator op;

		// Create the sine modulator, again, that we will scale using a parameter.
		op = voice.createAndAddOperator();
		op.envelope.setADSR(0, 0, 0, 100000);
		// lower the pitch and detune it slightly, to get a flanging effect.
		op.pitchRelation = 0.499f;
		op.isModulator = true;
		op.volume = 1f;

		// add a modulating parameter now, that will scale the volume of the modulator.
		// take note, this is the second operator we add - we need to index it later.
		op = voice.createAndAddParameter(0.0f);
		op.isModulator = true;
		// and make a wide range:
		op.volume = 2;
		// using multiplicative operation, we scale the previous modulator's volume.
		op.op = FMSynth.Voice.Operator.Operation.Multiplicative;

		// Create the carrier. Again, it's just a additive sine wave by default.
		op = voice.createAndAddOperator();
		// Set the envelope to a 2 ms sharp attack, followed by a medium decay down to -120 dB.
		op.envelope.setADSR(2, 7000, -120, 1000);


		// Add a filter.
		op = voice.createAndAddFilter(FMSynth.Voice.Operator.FilterTypes.LP, 200, 4);

		// Set the pitch of every voice to A2.
		voice.setPitch(EnvironmentTuning.pitchFromComposite(EnvironmentTuning.Pitch.A, 2));
	}
);

Now we just need to alter the parameter somewhere - for example the 's' key. Thus, we will update the Update function:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

void Update()
{
	if (Input.GetKeyDown(KeyCode.A))
	{
		// steal the oldest voice, and play it!
		voiceBuffer.stealOldestVoice().play();
	}
	else if (Input.GetKeyDown(KeyCode.S))
	{
		// generate a random target for the FM-index
		float rndTarget = rnd.random01();

		// set the parameter of each voice
		voiceBuffer.foreachVoice(
			voice =>
			{
				if (voice.enabled)
				{
					// interpolate the current paramater to the new target over 1500 miliseconds.
					voice.operators[1].parameterSetGoal(rndTarget, 1500);
				}
				else
				{
					// voices not currently playing is just set to the target directly.
					voice.operators[1].parameterSetValue(rndTarget);
				}
			}
		);
	}
}

Pressing the 's' key (after playing a sound with 'a') should now noticably alter the FM modulator's index, thus making a sweeping sound. Now that we've actually done all the work, I can safely reveal all this functionality is present already through the FMSynth.Voice.Kind.RandomOffset, created through FMSynth.Voice.createAndAddRandomOffset(). The RandomOffset will create a random offset (which, again, can be used as a sound or modulator source) each time the note is played, which is slightly different use-cases, though.

Sequencer Example

In this example, we'll look at sequencing sounds in Blunt. At the basic level, each Sound.Synthesizer renders a block of N samples of audio. The block size, N, can be controlled by setting the latency of each synthesizer, and this directly affects timings and precision for sequencing. As always, lower latency means lower performance. The interface Sound.SequencingSynthesizer<Synth> maintains a Synthesis.PlayHead and an interface for getting a callback on each render block. You will rarely use this interface yourself, as we will now investigate the GenerativeMusic.CallbackSequencer<Synth>.

The CallbackSequencer is basically a wrapper around the callback interface, that allows sequenced/timed callbacks using CallbackSequencer<Synth>.Tokens. Tokens are representative of a unique callback relationship between the owner, callback, sequencer and synthesizer. Tokens are created through CallbackSequencer.createCallbackToken. It has a basic function called Token.callbackIn(PlayHead.TimeOffset t), which does just that - calls you back in a certain time offset, and it is inside this callback you will sequence sounds.

The PlayHead contains all information about the current timing, "song position" etc. and it is able to calculate distances, that is, time offsets, to quantizable points in time. If you're at any arbitrary point in time, PlayHead.getDistanceToNextBar() or PlayHead.getDistanceToNextBeat() will calculate the distance to the next N bar, N beat with N subdivisions and a fractional offset as well. This is returned as a PlayHead.TimeOffset, which the CallbackSequencer happily eats. This system ensures that you of course can sequence completely unrelated to any timeline, but it is also brings the facilities to sequence on-beat/on-divisions, emulating a timeline based off settings in the Synthesis.EnvironmentTuning.

With that in mind, lets use this system to sequence our synth dynamically, generating some algorithmic music. The CallbackSequencer lives inside the Blunt.Synthesis.GenerativeMusic namespace. We are also going to modulate the pitch of the voice, so from the start we'll also create a harmonic index table that makes randomization a bit easier. The start of our script will now look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

using UnityEngine;
using Blunt;
using Blunt.Sound;
using Blunt.Synthesis;
using Blunt.Sound.GenerativeMusic;
using System.Collections;

public class AudioSystem : MonoBehaviour
{
	Mixer mixer;
	Mixer.MixerChannel channel;
	FMSynth synth;
    VoiceBuffer<FMSynth.Voice> voiceBuffer;
	Utils.CheapRandom rnd = new Utils.CheapRandom();

	int[] majorScale = {0, 2, 4, 5, 7, 9, 11};

	// our callback sequencer, and token.
	CallbackSequencer<FMSynth> cbs;
	CallbackSequencer<FMSynth>.Token token;

Now, we need to do some more setup'ing in the Awake() function. The sound design / initialization of the voice stays the same as in the previous example.

1
2
3
4

		// initiate the callback sequencer
		cbs = new CallbackSequencer<FMSynth>(synth);
		token = cbs.createCallbackToken(onSequencerCallback);
		token.callbackIn(PlayHead.TimeOffset.zero);

As we can see, we create an instant callback, such that we will get sequenced at 0.0.0.0 when the 'timeline' starts. The callback function, now, is where the magic will happen:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

// our callback function.
void onSequencerCallback(CallbackSequencer<FMSynth>.Token token, PlayHead ph)
{
	// the base pitch of our sound (D3)
	int baseNote = EnvironmentTuning.A4 - 7;
	// as said, we intent to alter the pitch each time a new note is played (which is what we're currently doing).
	// basically, we just select a random index in the majorScale array, which means we will get a random deviation distributed in the major scale of D.
	int transposition = majorScale[(int)(rnd.random01() * (majorScale.Length - 1))];
	// since we know we're at a quantizable beat (we started at zero), we don't have to quantize now - we can just select a 
	// random perfect beat offset between 1 and 4.
	PlayHead.TimeOffset offset = PlayHead.createTimeOffset(0, Mathf.RoundToInt(rnd.random01() * 3) + 1, 0, 0.0f);
	// we will also modulate the timbre of the sound using the previous method (random FM index)
	float rndTarget = rnd.random01();
	// collect the voice we will modulate
	FMSynth.Voice currentVoice;
	voiceBuffer.stealOldestVoice(out currentVoice);
	// alter the pitch
	currentVoice.setPitch(baseNote + transposition, 0.0f);
	// alter the timbre
	currentVoice.operators[1].parameterSetValue(rndTarget);
	// slightly alter the volume (in decibels, -24 to -12 dB) and play it.
	currentVoice.play(-18 + rnd.random11() * 6);
	// some debug info.
	print("Sequencing a sound " + transposition + " semitones away. Will be back in " + offset + ", currently at position: " + ph);
	// this is important - we will sequence the next callback dynamically between 1 and 4 beats away.
	token.callbackIn(offset);
}

The script should now play sequenced and modulated music when you start the project, ever-changing. Just to make sure we're on the same page, it should sound something like this:

Generative Music

Development Status

While not actively being worked on, it is scheduled for a rework - especially since it was developed for Unity 4. Unity 5 includes a lot of new built-in real-time audio utilities, even including a full-blown mixer. A list of the relevant work can be found here. I believe Blunt should utilize these new features, and perhaps focus more on synthesis and sequencing.

License

Blunt is licnsed under GPL v3.

Permanent link ◦ return to top ◦ return to home...