GStreamer LV2 Surround

drobilla — Wed, 12 Aug 2009 15:06:00 -0400

My last post was mostly about adding multi-channel support to GStreamer.

That was only partially implemented though - many elements in GStreamer (and pretty much all of the ones that support more than stereo) require each channel to be explicitly labeled with its position. This information wasn't bridged until now.

The two sides of this situation are:

The LV2 Port Groups extension: http://lv2plug.in/ns/dev/port-groups.lv2/port-groups.html
GStreamer multi-channel: http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-libs/html/gst-plugins-base-libs-gstmultichannel.html

LV2 “roles” correspond to GstAudioChannelPosition. Unfortunately, the mapping was a bit strained with the Port Groups extension as it was at midterm time. Fortunately, that extension is not finalized yet, and in the process I found some other problems with it, so I decided to bring it closer to the way GStreamer does things, which seems to be common in other software that does this sort of stuff. Formerly, the port groups extension used the channel names from the relevant surround spec (see the Wikipedia page on Surround Sound). This resulted in a lot of roles that were really the same thing, e.g. the “Surround Left” channel in 5.1 is really the same as the “Left Back” channel in 7.1. This was pretty confusing in practice. GStreamer instead always uses positional names like “Left Back”, which is unambiguous regardless of context (you don't have to know if it's 17.2 or whatever to know what “Left Back” probably means...). So, I changed the LV2 port groups extension to be similar (though not identical; munging an LV2 extension to match GStreamer API wouldn't be appropriate).

This information is now queried out from LV2 plugins that have it, mapped to GStreamer channel positions, and set as capabilities on mulit-channel pads. This means multi-channel surround LV2 plugins should now work fine with surround sound GStreamer set-ups, for example playing a Theora video through some audio processing and out through Alsa.

I updated swh-plugins for the minor changes, and added port group information to Invada Studio Plugins.

More to come...

GStreamer SoC Midterm Summary

drobilla — Sun, 05 Jul 2009 19:30:00 -0400

Apologies for not blogging along the way for those who are interested in this project. I'm more into actually doing things rather than blogging about doing things ;)

So, LV2 in Gst, where are we? Well, I've had LV2 plugins working with the same functionality as LADSPA plugins for a while. The main downside was shared by LADSPA: no support for multi-channel streams.

Some background for those unfamiliar with LADSPA or LV2: LADSPA and LV2 plugins are very simple in terms of I/O. A plugin has a collection of “ports” which contain either a single floating point value (control) or an array of floating point values (audio). LV2 extends this to any types but that's not important here. This is simple but causes problems in the face of stereo and other multi-channel streams: if a plugin has 3 audio inputs and 2 audio outputs, for example, what is what? Some kind of 3-channel audio input with stereo output? 5 completely unrelated ports? A stereo input and stereo output with a “sidechain” (common with e.g. compressors)? This information is not available in LADSPA, the best you can do is guess (which is practice means it just doesn't work).

GStreamer works with multi-channel streams as a single interleaved stream, so this is a problem. Thankfully, LV2 makes it trivial to add whatever information you like about plugins without having to touch a line of code since plugins are described in RDF (see the LV2 site for more information). All that needs to be decided is how to actually model that information. A specification for this is called an “ontology” in general/theory, and an “extension” in the LV2 community.

So, the problem is we need(ed) a good multi-channel ontology for LV2 plugins to work well inside Gst, since most things in Gst are at least stereo. The difficult thing with creating ontologies is making sure anything anyone might want to describe in the future is accounted for. Here's my best shot at this so far: LV2 Port Groups, based largely on earlier work by Lars Luthman and some input from the LV2 community. This extension isn't final, but expresses all the information needed by Gst for multi-channel (and more). (I also wrote the documentation generator that created the aforementioned specification page in the hopes that more user friendly documentation will encourage adoption by plugin and host authors).

I've created patches for the popular SWH and Calf plugin packages to add this information. When the extension goes final they will be included in these projects, but in the mean time are included in my git branch of gst-plugins-bad (see http://git.drobilla.net).

This is, as far as I know, the first time coherent multi-channel information has been available about plugins from the “LAD” community (e.g. LADSPA, DSSI, LV2). Coincidentally, this information is required for recent work on Ardour, among other things. In hindsight, this was a pretty glaring hole in the general state of LAD plugins, but back to GStreamer...

I've rewritten quite a bit of the GstSignalProcessor class (used by the LADSPA and now LV2 wrapper elements) to support multi-channel plugins. In code terms this means creating a set of buffers for non-interleaved data used by the plugin, and interleaving/deinterleaving buffers to/from Gst, respectively.

In summary: many LV2 plugins now exist with useful multi-channel information, and stereo LV2 plugins now work correctly in GStreamer. It should now be simple to add support for other audio plugin interfaces (VST? AudioUnits?) that also works with multi-channel streams.

Next up:

Get the “role” information from the plugin data and use it to support surround streams correctly
Finalize and publish the LV2 Port Groups extension, and contribute patches for all major plugin collections

Where to go from there is pretty open-ended. Unlike LADSPA, LV2 can theoretically support any kind of data, or any feature (non-realtime and non-audio things included). Extensions just need to be made to bridge the gap. What sort of functionality would you like to see bridge the GStreamer/LV2 gap?

drobilla.net - GStreamer

GStreamer LV2 Surround

GStreamer SoC Midterm Summary