|
| 1 | +.. _wasm_audio_worklets: |
| 2 | + |
| 3 | +======================= |
| 4 | +Wasm Audio Worklets API |
| 5 | +======================= |
| 6 | + |
| 7 | +The AudioWorklet extension to the `Web Audio API specification |
| 8 | +<https://webaudio.github.io/web-audio-api/#AudioWorklet>`_ enables web sites |
| 9 | +to implement custom AudioWorkletProcessor Web Audio graph node types. |
| 10 | + |
| 11 | +These custom processor nodes process audio data in real-time as part of the |
| 12 | +audio graph processing flow, and enable developers to write low latency |
| 13 | +sensitive audio processing code in JavaScript. |
| 14 | + |
| 15 | +The Emscripten Wasm Audio Worklets API is an Emscripten-specific integration |
| 16 | +of these AudioWorklet nodes to WebAssembly. Wasm Audio Worklets enables |
| 17 | +developers to implement AudioWorklet processing nodes in C/C++ code that |
| 18 | +compile down to WebAssembly, rather than using JavaScript for the task. |
| 19 | + |
| 20 | +Developing AudioWorkletProcessors in WebAssembly provides the benefit of |
| 21 | +improved performance compared to JavaScript, and the Emscripten |
| 22 | +Wasm Audio Worklets system runtime has been carefully developed to guarantee |
| 23 | +that no temporary JavaScript level VM garbage will be generated, eliminating |
| 24 | +the possibility of GC pauses from impacting audio synthesis performance. |
| 25 | + |
| 26 | +Audio Worklets API is based on the Wasm Workers feature. It is possible to |
| 27 | +also enable the `-pthread` option while targeting Audio Worklets, but the |
| 28 | +audio worklets will always run in a Wasm Worker, and not in a Pthread. |
| 29 | + |
| 30 | +Development Overview |
| 31 | +==================== |
| 32 | + |
| 33 | +Authoring Wasm Audio Worklets is similar to developing Audio Worklets |
| 34 | +API based applications in JS (see `MDN: Using AudioWorklets <https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API/Using_AudioWorklet>`_), with the exception that users will not manually implement |
| 35 | +the JS code for the ScriptProcessorNode files in the AudioWorkletGlobalScope. |
| 36 | +This is managed automatically by the Emscripten Wasm AudioWorklets runtime. |
| 37 | + |
| 38 | +Instead, application developers will need to implement a small amount of JS <-> Wasm |
| 39 | +(C/C++) interop to interact with the AudioContext and AudioNodes from Wasm. |
| 40 | + |
| 41 | +Audio Worklets operate on a two layer "class type & its instance" design: |
| 42 | +first one defines one or more node types (or classes) called AudioWorkletProcessors, |
| 43 | +and then, these processors are instantiated one or more times in the audio |
| 44 | +processing graph as AudioWorkletNodes. |
| 45 | + |
| 46 | +Once a class type is instantiated on the Web Audio graph and the graph is |
| 47 | +running, a C/C++ function pointer callback will be invoked for each 128 |
| 48 | +samples of the processed audio stream that flows through the node. |
| 49 | + |
| 50 | +This callback will be executed on a dedicated separate audio processing |
| 51 | +thread with real-time processing priority. Each Web Audio context will |
| 52 | +utilize only a single audio processing thread. That is, even if there are |
| 53 | +multiple audio node instances (maybe from multiple different audio processors), |
| 54 | +these will all share the same dedicated audio thread on the AudioContext, |
| 55 | +and will not run in a separate thread of their own each. |
| 56 | + |
| 57 | +Note: the audio worklet node processing is pull-mode callback based. Audio |
| 58 | +Worklets do not allow the creation of general purpose real-time prioritized |
| 59 | +threads. The audio callback code should execute as quickly as possible and |
| 60 | +be non-blocking. In other words, spinning a custom `for(;;)` loop is not |
| 61 | +possible. |
| 62 | + |
| 63 | +Programming Example |
| 64 | +=================== |
| 65 | + |
| 66 | +To get hands-on experience with programming Wasm Audio Worklets, let's create a |
| 67 | +simple audio node that outputs random noise through its output channels. |
| 68 | + |
| 69 | +1. First, we will create a Web Audio context in C/C++ code. This is achieved |
| 70 | +via the ``emscripten_create_audio_context()`` function. In a larger application |
| 71 | +that integrates existing Web Audio libraries, you may already have an |
| 72 | +``AudioContext`` created via some other library, in which case you would instead |
| 73 | +register that context to be visible to WebAssembly by calling the function |
| 74 | +``emscriptenRegisterAudioObject()``. |
| 75 | + |
| 76 | +Then, we will instruct the Emscripten runtime to initialize a Wasm Audio Worklet |
| 77 | +thread scope on this context. The code to achieve these tasks looks like: |
| 78 | + |
| 79 | +.. code-block:: cpp |
| 80 | +
|
| 81 | + #include <emscripten/webaudio.h> |
| 82 | +
|
| 83 | + uint8_t audioThreadStack[4096]; |
| 84 | +
|
| 85 | + int main() |
| 86 | + { |
| 87 | + EMSCRIPTEN_WEBAUDIO_T context = emscripten_create_audio_context(0); |
| 88 | +
|
| 89 | + emscripten_start_wasm_audio_worklet_thread_async(context, audioThreadStack, sizeof(audioThreadStack), |
| 90 | + &AudioThreadInitialized, 0); |
| 91 | + } |
| 92 | +
|
| 93 | +2. When the worklet thread context has been initialized, we are ready to define our |
| 94 | +own noise generator AudioWorkletProcessor node type: |
| 95 | + |
| 96 | +.. code-block:: cpp |
| 97 | +
|
| 98 | + void AudioThreadInitialized(EMSCRIPTEN_WEBAUDIO_T audioContext, EM_BOOL success, void *userData) |
| 99 | + { |
| 100 | + if (!success) return; // Check browser console in a debug build for detailed errors |
| 101 | + WebAudioWorkletProcessorCreateOptions opts = { |
| 102 | + .name = "noise-generator", |
| 103 | + }; |
| 104 | + emscripten_create_wasm_audio_worklet_processor_async(audioContext, &opts, &AudioWorkletProcessorCreated, 0); |
| 105 | + } |
| 106 | +
|
| 107 | +3. After the processor has initialized, we can now instantiate and connect it as a node on the graph. Since on |
| 108 | +web pages audio playback can only be initiated as a response to user input, we will also register an event handler |
| 109 | +which resumes the audio context when the user clicks on the DOM Canvas element that exists on the page. |
| 110 | + |
| 111 | +.. code-block:: cpp |
| 112 | +
|
| 113 | + void AudioWorkletProcessorCreated(EMSCRIPTEN_WEBAUDIO_T audioContext, EM_BOOL success, void *userData) |
| 114 | + { |
| 115 | + if (!success) return; // Check browser console in a debug build for detailed errors |
| 116 | +
|
| 117 | + int outputChannelCounts[1] = { 1 }; |
| 118 | + EmscriptenAudioWorkletNodeCreateOptions options = { |
| 119 | + .numberOfInputs = 0, |
| 120 | + .numberOfOutputs = 1, |
| 121 | + .outputChannelCounts = outputChannelCounts |
| 122 | + }; |
| 123 | +
|
| 124 | + // Create node |
| 125 | + EMSCRIPTEN_AUDIO_WORKLET_NODE_T wasmAudioWorklet = emscripten_create_wasm_audio_worklet_node(audioContext, |
| 126 | + "noise-generator", &options, &GenerateNoise, 0); |
| 127 | +
|
| 128 | + // Connect it to audio context destination |
| 129 | + EM_ASM({emscriptenGetAudioObject($0).connect(emscriptenGetAudioObject($1).destination)}, |
| 130 | + wasmAudioWorklet, audioContext); |
| 131 | +
|
| 132 | + // Resume context on mouse click |
| 133 | + emscripten_set_click_callback("canvas", (void*)audioContext, 0, OnCanvasClick); |
| 134 | + } |
| 135 | +
|
| 136 | +4. The code to resume the audio context on click looks like this: |
| 137 | + |
| 138 | +.. code-block:: cpp |
| 139 | +
|
| 140 | + EM_BOOL OnCanvasClick(int eventType, const EmscriptenMouseEvent *mouseEvent, void *userData) |
| 141 | + { |
| 142 | + EMSCRIPTEN_WEBAUDIO_T audioContext = (EMSCRIPTEN_WEBAUDIO_T)userData; |
| 143 | + if (emscripten_audio_context_state(audioContext) != AUDIO_CONTEXT_STATE_RUNNING) { |
| 144 | + emscripten_resume_audio_context_sync(audioContext); |
| 145 | + } |
| 146 | + return EM_FALSE; |
| 147 | + } |
| 148 | +
|
| 149 | +5. Finally we can implement the audio callback that is to generate the noise: |
| 150 | + |
| 151 | +.. code-block:: cpp |
| 152 | +
|
| 153 | + #include <emscripten/em_math.h> |
| 154 | +
|
| 155 | + EM_BOOL GenerateNoise(int numInputs, const AudioSampleFrame *inputs, |
| 156 | + int numOutputs, AudioSampleFrame *outputs, |
| 157 | + int numParams, const AudioParamFrame *params, |
| 158 | + void *userData) |
| 159 | + { |
| 160 | + for(int i = 0; i < numOutputs; ++i) |
| 161 | + for(int j = 0; j < 128*outputs[i].numberOfChannels; ++j) |
| 162 | + outputs[i].data[j] = emscripten_random() * 0.2 - 0.1; // Warning: scale down audio volume by factor of 0.2, raw noise can be really loud otherwise |
| 163 | +
|
| 164 | + return EM_TRUE; // Keep the graph output going |
| 165 | + } |
| 166 | +
|
| 167 | +And that's it! Compile the code with the linker flags ``-sAUDIO_WORKLET=1 -sWASM_WORKERS=1`` to enable targeting AudioWorklets. |
| 168 | + |
| 169 | +Synchronizing audio thread with the main thread |
| 170 | +=============================================== |
| 171 | + |
| 172 | +Wasm Audio Worklets API builds on top of the Emscripten Wasm Workers feature. This means that the Wasm Audio Worklet thread is modeled as if it was a Wasm Worker thread. |
| 173 | + |
| 174 | +To synchronize information between an Audio Worklet Node and other threads in the application, there are two options: |
| 175 | + |
| 176 | +1. Leverage the Web Audio "AudioParams" model. Each Audio Worklet Processor type is instantiated with a custom defined set of audio parameters that can affect the audio computation at sample precise accuracy. These parameters are passed in the ``params`` array into the audio processing function. |
| 177 | + |
| 178 | +The main browser thread that created the Web Audio context can adjust the values of these parameters whenever desired. See `MDN function: setValueAtTime <https://developer.mozilla.org/en-US/docs/Web/API/AudioParam/setValueAtTime>`_ . |
| 179 | + |
| 180 | +2. Data can be shared with the Audio Worklet thread using GCC/Clang lock-free atomics operations, Emscripten atomics operations and the Wasm Worker API thread synchronization primitives. See :ref:`wasm_workers` for more information. |
| 181 | + |
| 182 | +3. Utilize the ``emscripten_audio_worklet_post_function_*()`` family of event passing functions. These functions operate similar to how the function family emscripten_wasm_worker_post_function_*()`` does. Posting functions enables a ``postMessage()`` style of communication, where the audio worklet thread and the main browser thread can send messages (function call dispatches) to each others. |
| 183 | + |
| 184 | + |
| 185 | +More Examples |
| 186 | +============= |
| 187 | + |
| 188 | +See the directory tests/webaudio/ for more code examples on Web Audio API and Wasm AudioWorklets. |
0 commit comments