Cookies help us deliver our services. By using our services, you agree to our use of cookies. Learn more

Tech

Recording And Streaming Audio

Posted by Mark Alexander on 31 October 2014

In the latest updates to the Early Access version of GameMaker: Studio there have been some major additions to the audio functions:

  • New getter functions to get information about the state of a sound or the audio system
  • New listener functions to specifically target a given output like headphones or gamepad (on some targets)
  • New functions for streaming and recording audio using an audio buffer.

The first two new features are pretty self explanatory and easy enough to use (just check the manual beforehand), but the recording audio and using audio buffers functions may be confusing at first to some users, and so in this tech blog we are going to cover this new functionality and create a basic test app so you can get some "hands-on" experience with them. 

Create A New project

Since just reading about something is not the same as actually doing it, in this tech blog we are going to create a simple test app to illustrate a little bit of each of the new features related to recording and streaming audio. This app is not meant to represent real-world uses of the functions, but rather it is to introduce you to them and show how they can all work togther. So, to start with, you should open up GameMaker: Studio (the latest EA version) and create a new project called "audio_test_app" or something similar.

In this new object, we are going to initialise recording from a given device, then play it back again when the recording is finished using an audio queue. However before we get onto that, we need to set up the recording devices and a buffer to stream the audio from. So, add a Create Event to the object now, with the following code:

var count = audio_get_recorder_count();
switch(count)
{
case 0:
    show_message("You have no recording devices!");
    game_end();
    break;
case 1:
    audio_recorder = 0;
    break;
default:
    audio_recorder = get_integer("You have " + string(count) + " recording devices. Enter index (0-" + string(count - 1) + "):", 0);
    break;
}

Here we use the new function audio_get_recorder_count() to find out how many devices are available for recording from, and then we check this value to either exit the app (no devices available) or to set the variable audio_recorder to hold the integer Channel ID of the device that we want to record from.

Now we need to add some control variables for playing and recording as well as set up an "info" variable to show what the properties of the recording device are:

var map = audio_get_recorder_info(audio_recorder);             // Get information about the recording device
var device = string(audio_recorder);                           // The channel ID value for the device
var name = ":#Name: " + map[? "name"];                         // The device name as returned by the OS
var str_rate = "#Sample rate: " + string(map[? "sample_rate"]);// The sample rate that the device uses

info = "##Using recorder device " + device + name + str_rate;  // Information string to display to the user
playing = false;                                               // Control variable for playing
recording = false;                                             // Control variable for recording
audio_recorder_channel = -1;                                   // Control variable which will hold the recording channel ID

Here we use another new function, audio_get_recorder_info() to get some data on the recording device being used. The function will return a ds_map with a number of keys related to the device and the audio quality it will give, and this data is then used to create an information string to display later. Note that we don't destroy the ds_map returned just yet, as we also need it to set up the audio queue.

Audio Queues

When creating audio from a buffer, you need to add it into an audio queue before you can play it. This audio queue should have the same format as the audio that is in the buffer used to create it, and in this case we get the information from the recording device (as shown above) using the audio_get_recorder_info() function.

NOTE: You are not limited to queueing recorded audio only. You can create queues for audio files that have been loaded into a buffer too, as long as the audio is in an uncompressed format like*.wav. In this way you can create your own playlists from external files, for example. However queues are most useful for "double buffering" input audio, You can add the audio as it is received from the source (which could be over the internet, not just from a local microphone) to a buffer, and then queue the buffer for playing.

An individual queue can be thought of as a new sound made up of various bits of audio taken from a buffer, so you can, for example, add various music tracks to a single buffer then queue the buffer up and have them play one after the other streamed from the buffer through the queue. However, for our test app, we are simply going to create a buffer, add the recorded audio from a device to that buffer, then queue and play it when the user presses a key.

To set this up we need to add the following code into the Create Event of our controller, after the previous code given above:

audio_buff = buffer_create(256, buffer_grow, 1);                // We create a "grow" type buffer to hold the recorded audio
format = map[? "data_format"];                                  // We get the device format for creating the queue...
rate = map[? "sample_rate"];                                    // ...and thee sample rate
channels = map[? "channels"];                                   // ...and the number of channels
audio_queue = audio_create_play_queue(format, rate, channels);  // Here we create the audio queue
queue_len = 0;                                                  // Control variable for getting the queue "size"
ds_map_destroy(map);                                            // Destroy the map with the device info

All this does is create our "grow" type buffer, initialise the various variables required, and then create the audio queue that we are going to use. Note that the function to create the audio queue returns a unique ID value for the queue which iwhat you will need to use to play it or clear it later.

Playing A Queue

We need to close the code for the Create Event now and add a new Step Event. Here we will detect a couple of key presses to either record to or playback from the queue. Add a code block now with the following:

record = keyboard_check(vk_alt);
play = keyboard_check_pressed(vk_shift);

if (play && (queue_len > 0))
{
if (!recording && !playing)
    {
// No sound playing, no sound being recorded, so play the queue audio_play_sound(audio_queue, 10, false); playing = true; } }

This first section of code checks the keys and if the <SHIFT> key is pressed it checks the length of the queue (as stored in the variable), and then checks to make sure there is nothing currently being played or recorded. If all this is true, then it plays the audio queue using the regular audio function. When you play an audio queue in this way, it will trigger a new Asynchronous Event called the Audio Playback Event.

The Audio Playback Event

The asynchrnonous event catagory now contains the Audio Playback Event. This event will be triggered whenever a "chunk" of audio from an audio queue is finished. In this test app, we are going to use it to count the number of queued adio chunks and then show a message when they have all been played, so close the Step Event code for now, and add this new event with the following:

if (--queue_len <= 0)
{
playing = false;
show_message("Queue playback ended!");
}

Since we are counting how many audio chunks have been queued using the queue_len variable, we can use this to count down as each queued sound plays until the last one is played at which point the message will be shown. This event is very handy, as not only does it trigger at the end of each queued sound, but it returns a ds_map in the variable async_load which contains a bit of information about what exactly has triggered the event. The keys are:

  • "queue_id" - the queue index for the queue that has finished playing, as returned by the function audio_create_play_queue().
  • "buffer_id" - the buffer ID for the buffer that is no longer being played from
  • "queue_shutdown" - this is set to 0 during normal playback and 1 when the event is received because audio_free_play_queue() has been called. When this is set to 1, you don't want to queue up any further data.

We don't need to use this map in our test app, but you should be aware that it exists and the information it holds as it can be very handy for controlling more complex applications of the audio queues.

Recording And Queueing Audio

We now need to check for the <ALT> key as that is what will trigger recording - in this case, we have it polled every step so that the app will record as long as it is held down - so close the Audio Playback Event, and open the Step Event again, as we need to add the following else to that first if:

else
{
if (record && !recording && !playing)
    {
if (queue_len <= 0)
{
// Free the current audio queue, reset the audio buffer and create a new queue audio_free_play_queue(audio_queue); buffer_delete(audio_buff); audio_queue = audio_create_play_queue(format, rate, channels); audio_buff = buffer_create(256, buffer_grow, 1);
// Store the channel ID of the channel we start recording on audio_recorder_channel = audio_start_recording(audio_recorder); recording = true; playing = false; queue_len = 0;
} } }

As you can see, the first thing we do is free the audio queue from memory. This is essential otherwise you will face issues with memory leaks, and you will also not be able to delete the buffer associated with it - once a buffer has been assigned to an audio queue, it is essentially "locked" until the queue is freed.

After that we create a new queue and a new buffer ready to record to, then use the audio_start_recording() function (along with the stored channel ID to tell GameMaker: Studio which input source to record from) to start recording. This function returns a unique ID for the recorder channel being used, which can then be used in the new Audio Recording Event to identify which channel is triggering the event. In this way you can record multiple audio sources then parse the ID in the event and decide what to do with the incoming audio.

Audio Recording Event

To detect incoming audio on any channel as it is being recorded, there has been a new event added to the Asynchronous Event category - The Audio Recording Event. Add this event to your controller object now. It will be triggered by any device that is currently recording, and will return a pre-populated ds_map in the variable async_load. This map will contain the following keys:

  • "buffer_id" - the ID of the temporary buffer you can use to retrieve the audio data
  • "channel_index" - the recording channel index as returned by the calling function that this data came from
  • "data_len" - the length of data (in bytes) you've received

The buffer ID is the identifier for the temporary buffer that is storing the chunk of audio that has been received. This buffer only exists for the duration of the event, so you should copy the data it contains into a permanent buffer if you don't want to lose it (we'll show how in a moment). The channel index is the ID of the recorder channel that is being used, as returned by the function audio_start_recording() in the code above, and the length value is the length of the audio data (in bytes) that has been received.

So, how can we use this information in our test app? Like this:

var channel = async_load[? "channel_index"];              // Get the channel ID that triggered this event
if (audio_recorder_channel == channel)                    // Check the channel and device channel ID are the same
{
var t_buff = async_load[? "buffer_id"]; // Get the index into the temporary buffer with the audio var length = async_load[? "data_len"]; // Get the length of the data returned buffer_seek(audio_buff, buffer_seek_end, 0); // Move to the end of the audio buffer var pos = buffer_tell(audio_buff); // Get the position of the end of the audio buffer buffer_copy(t_buff, 0, length, audio_buff, pos); // Copy the temp buffer to the audio buffer at the correct place audio_queue_sound(audio_queue, audio_buff, pos, length); // Now add this new audio chunk onto the audio queue ++queue_len; // Increase the queue length variable if (!record) {
// If the <ALT> key is no longer held down, stop recording audio_stop_recording(audio_recorder_channel); recording = false; } }

Hopefully you can see from the comments and the code that it's all fairly straightforward. We get the contents of the temporary buffer and copy them on to the end of the buffer we created for this. We then queue this section of audio along with the previous ones (if any) ready for playback. We also check to see if the <ALT> key is held down, and if not, then we stop recording.

The Draw GUI Event

The final task for us to complete with our test app is to draw the information about our recording to the screen. So, add a Draw GUI Event with the following:

var s = info + "##Recording: " + string(recording) + "##<Alt> to record.#<Shift> to play back.";
draw_text(10, 10, s);
draw_text(10, 180, "Queue Length: " + string(queue_len));
draw_text(10, 200, "Buffer size: " + string(buffer_get_size(audio_buff)));

Summary

If all has gone correctly your app is ready to run. It should now record audio from a given source to a buffer when you press <ALT> and then play it back when you press <SHIFT>. That ends our introduction to the new audio functions, and we hope that you take some time to play with them and use them in your games!

You can find a copy of the test app GMZ from the following link: https://www.yoyogames.com/uploads/RecordAudioQueue.gmz

Back to Top