Skip to content

Latest commit

 

History

History
86 lines (55 loc) · 8.71 KB

orenshoham_gsoc_2019.md

File metadata and controls

86 lines (55 loc) · 8.71 KB

AudioWorklet Support in p5.js-sound

For my Google Summer of Code 2019 project, I worked with my mentor Jason Sigal to add AudioWorklet support to p5.js-sound, allowing certain parts of the library to run more efficiently by moving custom audio processing to a separate audio thread. I also helped Jason integrate Webpack and Babel into the p5.js-sound Grunt build pipeline, allowing the library's developers to use ES6 JavaScript features and laying the groundwork for modernizing the codebase and examples.

AudioWorklet

The AudioWorklet API consists of two classes: AudioWorkletProcessor and AudioWorkletNode. AudioWorkletProcessors contain audio processing code that runs in a separate thread and are meant to be loaded from separate files, while AudioWorkletNodes run on the main thread and are used to connect to other Web Audio nodes.

AudioWorklet replaces ScriptProcessorNode, a now-deprecated Web Audio node that runs audio code in the browser's main thread. p5.js-sound used ScriptProcessorNode internally in three classes:

  • p5.SoundFile, which used a ScriptProcessorNode to keep track of a SoundFile's current playback position.
  • p5.Amplitude, which used a ScriptProcessorNode to perform amplitude analysis.
  • p5.SoundRecorder, which used a ScriptProcessorNode to concatenate audio buffers together during the recording process.

For each of these classes, I created new AudioWorkletProcessors for p5.SoundFile, p5.Amplitude, and p5.SoundRecorder that replicated the corresponding ScriptProcessorNode's onaudioprocess function.

AudioWorkletProcessors and Async Loading

Before these AudioWorkletProcessors could be used as AudioWorkletNodes, they needed be loaded asynchronously via a call to audioWorklet.addModule(), which expects a URL pointing to the file containing the processor definition.

This presented a problem: p5.js-sound is typically included in sketches as a single file, so I couldn't load the worklet processors as separate files. The solution was to convert the source for each AudioWorkletProcessor into a string using Webpack's raw-loader (see Webpack and ES6 below for more details), then construct a Blob from that string and generate an object URL for the Blob which could be passed into audioWorklet.addModule().

function loadAudioWorkletModules() {
  return Promise.all(moduleSources.map(function(moduleSrc) {
    const blob = new Blob([moduleSrc], { type: 'application/javascript' });
    const objectURL = URL.createObjectURL(blob);
    return ac.audioWorklet.addModule(objectURL);
  }));
}

To play nicely with the rest of p5.js, the AudioWorkletProcessors needed to be loaded before a sketch's setup() function, during preload(). However, I didn't want to force users to write a preload() function every time they made a sketch with p5.js-sound, as that would break existing sketches and make the library harder to use.

The (slightly-hacky) solution that I arrived at was to register a function with the p5.js "init" hook that first ensured that a preload() function was defined on the sketch, then incremented p5.js's internal preload counter to get preload() to wait for all of the processor modules to load:

p5.prototype.registerMethod('init', function() {
  // ensure that a preload function exists so that p5 will wait for preloads to finish
  if (!this.preload && !window.preload) {
    this.preload = function() {};
  }
  // use p5's preload system to load necessary AudioWorklet modules before setup()
  this._preloadCount++;
  const onWorkletModulesLoad = function() {
    this._decrementPreload();
  }.bind(this);
  loadAudioWorkletModules().then(onWorkletModulesLoad);
});

This approach is probably a little too dependent on p5.js's internal workings, but it might be useful for other p5.js library developers running into similar asynchronous loading issues in the future.

Polyfill

To avoid breaking p5.js-sound in browsers like Firefox and Safari that haven't yet implemented the AudioWorklet API, I added a polyfill to the library that falls back to ScriptProcessorNode in unsupported browsers.

Ring Buffers

AudioWorklet processors use a fixed buffer size of 128 frames, unlike ScriptProcessorNodes which take a buffer size as an argument. A lower buffer size will generally result in lower latency but also higher CPU usage.

To allow for more control over the buffer sizes of the various AudioWorkletNodes in p5.js-sound, I added a ring buffer (adapted from some of Google Chrome Labs' sample code) to each AudioWorkletProcessor which accumulates frames of audio until the desired buffer size has been reached. For more details on this approach, see the "Handling Buffer Size Mismatch" section of the article Audio Worklet Design Patterns.

Webpack and ES6

As mentioned above, loading AudioWorkletProcessor modules as strings was made much easier by including Webpack in the p5.js-sound build pipeline. Since ES6 JavaScript support had been on the p5.js-sound to-do list for a while, Jason and I decided to pair on integrating Webpack and Babel into the p5.js-sound Grunt build configuration. This involved replacing requirejs with grunt-webpack and adding a Webpack config. This change allows p5.js-sound developers to use ES6 features throughout the codebase, along with source maps for easier debugging of compiled ES6 code in the browser.

Unfortunately, the additional comments inserted by Webpack into the compiled bundle had the side-effect of breaking the p5.js-sound documentation generated from YUIDoc comments. Jason is currently in the process of fixing this issue by extracting the YUIDoc comments into a separate file during the build. Progress on this can be followed in PR #377.

Contributions

Acknowledgements

I'm extremely grateful to the Processing Foundation for giving me the opportunity to contribute to p5.js, as well as to my mentor Jason Sigal for all of his support over the course of this project.