GStreamer SoC Midterm Summary
Apologies for not blogging along the way for those who are interested in this project. I'm more into actually doing things rather than blogging about doing things ;)
So, LV2 in Gst, where are we? Well, I've had LV2 plugins working with the same functionality as LADSPA plugins for a while. The main downside was shared by LADSPA: no support for multi-channel streams.
Some background for those unfamiliar with LADSPA or LV2: LADSPA and LV2 plugins are very simple in terms of I/O. A plugin has a collection of "ports" which contain either a single floating point value (control) or an array of floating point values (audio). LV2 extends this to any types but that's not important here. This is simple but causes problems in the face of stereo and other multi-channel streams: if a plugin has 3 audio inputs and 2 audio outputs, for example, what is what? Some kind of 3-channel audio input with stereo output? 5 completely unrelated ports? A stereo input and stereo output with a "sidechain" (common with e.g. compressors)? This information is not available in LADSPA, the best you can do is guess (which is practice means it just doesn't work).
GStreamer works with multi-channel streams as a single interleaved stream, so this is a problem. Thankfully, LV2 makes it trivial to add whatever information you like about plugins without having to touch a line of code since plugins are described in RDF (see the LV2 site for more information). All that needs to be decided is how to actually model that information. A specification for this is called an "ontology" in general/theory, and an "extension" in the LV2 community.
So, the problem is we need(ed) a good multi-channel ontology for LV2 plugins to work well inside Gst, since most things in Gst are at least stereo. The difficult thing with creating ontologies is making sure anything anyone might want to describe in the future is accounted for. Here's my best shot at this so far: LV2 Port Groups, based largely on earlier work by Lars Luthman and some input from the LV2 community. This extension isn't final, but expresses all the information needed by Gst for multi-channel (and more). (I also wrote the documentation generator that created the aforementioned specification page in the hopes that more user friendly documentation will encourage adoption by plugin and host authors).
I've created patches for the popular SWH and Calf plugin packages to add this information. When the extension goes final they will be included in these projects, but in the mean time are included in my git branch of gst-plugins-bad (see http://git.drobilla.net).
This is, as far as I know, the first time coherent multi-channel information has been available about plugins from the "LAD" community (e.g. LADSPA, DSSI, LV2). Coincidentally, this information is required for recent work on Ardour, among other things. In hindsight, this was a pretty glaring hole in the general state of LAD plugins, but back to GStreamer...
I've rewritten quite a bit of the GstSignalProcessor class (used by the LADSPA and now LV2 wrapper elements) to support multi-channel plugins. In code terms this means creating a set of buffers for non-interleaved data used by the plugin, and interleaving/deinterleaving buffers to/from Gst, respectively.
In summary: many LV2 plugins now exist with useful multi-channel information, and stereo LV2 plugins now work correctly in GStreamer. It should now be simple to add support for other audio plugin interfaces (VST? AudioUnits?) that also works with multi-channel streams.
- Get the "role" information from the plugin data and use it to support surround streams correctly
- Finalize and publish the LV2 Port Groups extension, and contribute patches for all major plugin collections
Where to go from there is pretty open-ended. Unlike LADSPA, LV2 can theoretically support any kind of data, or any feature (non-realtime and non-audio things included). Extensions just need to be made to bridge the gap. What sort of functionality would you like to see bridge the GStreamer/LV2 gap?
Opinionated dichotomies in science
Computing is the only real science. The rest can be simulated.
patchage 0.4.2 has been released. Patchage is a modular patch bay for Jack and ALSA based audio/MIDI systems. For more information, see http://drobilla.net/software/patchage. -- David Robillard firstname.lastname@example.org
- LASH support via D-BUS from ladi-patchage branch
- Remove old LASH support via liblash
raul 0.5.1 has been released. Raul (Realtime Audio Utility Library) is a lightweight C++ convenience library for realtime programming, with a bias towards audio applications. For more information, see http://drobilla.net/software/raul.
- OSC blob fixes
- Use true atomic integers for Queue indexes
Iteration in Scheme
Thoughts brought on by using SWIG to make Scheme bindings for C/C++ code:
In Scheme, being a LISP, pretty much everything is usually a list. This is great and makes for elegant code, but as a data structures guy I cringe to see the ridiculous amount of O(n) searching done in LISP code. Why is the underlying structure (a list) a dependency of the code using it? Lack of abstraction: car and cdr only work on cons cells, and lists are made of cons cells - period. This sucks for plenty of reasons (the one that encouraged this rant is making Scheme bindings for C++ code).
Say you want to expose a C or C++ list in Scheme. To be really useful in Scheme, it will have to work with car and cdr. Luckily any list has the concept of 'get the value at this position' (car) and 'get the next position' (cdr). For C++ the position is an iterator, car is *, and cdr is ++. Unfortunately making 'nice' (ie typical and un-weird) Scheme bindings for these means you literally have to duplicate the entire C++ list and build a Scheme list out of it, because there's no functional abstraction going on here, and the type system is nonexistant.
I guess this boils down to a rant about needing iterators in Scheme. I cringe at sounding so much like an OO "patterns" kind of guy, but there it is :)
Lets say our Scheme dialect has some a type system with polymorphism. Scheme code is generally something like:
(define (do-something list) (something (car list)) (if (not (null? (cdr list))) (do-something (cdr list)))) (define a-list '(1 2 3 4 5)) (do-something a-list)
Rename car "get" and cdr "next" to avoid confusion, and use an iterator:
(define (do-something iter) (something (get iter)) (if (valid? (next iter)) (do-something (next iter)))) (define a-collection (whatever)) (do-something (start a-collection))
The idiom for do-something (the fundamental LISP idiom if there ever was one) is the same. The only difference is the call to do-something must pass an iterator, here with the start function. Slightly unfortunate maybe, but the benefits are huge:
- Common list-like iteration for all data structures (e.g. trees)
- Different styles of iteration (use reverse-end instead of start and have do-something work in reverse without modification)
I suppose since I'm assuming a stronger/better/faster Scheme dialect with a good type system, a collection could just automatically convert to the 'start' iterator, and the calling code could even be the same if you just want classic LISP style code. I'm a huge fan of LISP style languages (mostly because the code is a simple list structure itself), but there really needs to be an abstraction between the underlying data structure and the code that uses it.
DOAP replacing AUTHORS/MAINTAINERS/etc
There's been a bit of talk in the GNOME camp lately about using DOAP instead of the unstructured text files that are the current norm for source packages. On the one hand, people want the benefits of having machine readable data in projects, OTOH, RDF/XML is a nightmare ("I'll never maintain such bloat!" - "That is one hell of an ugly file.").
This is how RDF/XML hurts RDF. The original loathed file:
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl"?> <rdf:RDF xml:lang="en" xmlns="http://usefulinc.com/ns/doap#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:asfext="http://projects.apache.org/ns/asfext#"> <Project rdf:about="http://ant.apache.org/"> <created>2006-02-17</created> <license rdf:resource="http://usefulinc.com/doap/licenses/asl20" /> <name>Apache Ant</name> <homepage rdf:resource="http://ant.apache.org" /> <asfext:pmc rdf:resource="http://ant.apache.org" /> <shortdesc>Java-based build tool</shortdesc> <description>Apache Ant is a Java-based build tool. In theory, it is kind of like Make, but without Make\'s wrinkles.</description> <bug-database rdf:resource="http://issues.apache.org/bugzilla/buglist.cgi?product=Ant" /> <mailing-list rdf:resource="http://ant.apache.org/mail.html" /> <download-page rdf:resource="http://ant.apache.org/bindownload.cgi" /> <programming-language>Java</programming-language> <category rdf:resource="http://projects.apache.org/category/build-management" /> <release> <Version> <name>Apache Ant 1.7.0</name> <created>2006-12-13</created> <revision>1.7.0</revision> </Version> </release> <repository> <SVNRepository> <location rdf:resource="http://svn.apache.org/repos/asf/ant"/> <browse rdf:resource="http://svn.apache.org/viewcvs.cgi/ant"/> </SVNRepository> </repository> </Project> </rdf:RDF>
and the equivalent in Turtle (a subset of N3) (automatically generated with rapper -o turtle doap_Ant.rdf):
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> . @prefix : <http://usefulinc.com/ns/doap#> . @prefix asfext: <http://projects.apache.org/ns/asfext#> . a :Project; :created "2006-02-17"@en; :license ; :name "Apache Ant"@en; :homepage <http://ant.apache.org>; asfext:pmc <http://ant.apache.org>; :shortdesc "Java-based build tool"@en; :description "Apache Ant is a Java-based build tool. In theory, it is kind of like Make, but without Make's wrinkles."@en; :bug-database <http://issues.apache.org/bugzilla/buglist.cgi?product=Ant>; :mailing-list <http://ant.apache.org/mail.html>; :download-page <http://ant.apache.org/bindownload.cgi>; :programming-language "Java"@en; :category <http://projects.apache.org/category/build-management>; :release [ a :Version; :name "Apache Ant 1.7.0"@en; :created "2006-12-13"@en; :revision "1.7.0"@en ]; :repository [ a :SVNRepository; :location <http://svn.apache.org/repos/asf/ant> :browse <http://svn.apache.org/viewcvs.cgi/ant> ] .
I wouldn't want to hand maintain for RDF/XML version either, but the Turtle version? Sure. It's the exact same information, far more human readable, and about as terse as it could be while representing the same information.
The best thing about a syntax independent model like RDF is.. well, it's syntax independent. Choose one that doesn't suck :)
Ardour: back on track. Maybe.
Well, I've been working on Ardour but not directly on my SoC project.. finally got around to fixing that whole mixer-strip-element thing that's been on the table since last year. Things are more open to extensibility now (and a unified bus implementation should appear soon, so we'll have MIDI busses or even MIDI/audio (instrument) busses). Maybe we'll see things like this actually happen some time soonish...
It would be nice to figure out a really good MIDI meter (something more clever than just an audio peak meter abused), and a MIDI fader that can work in various ways, or even instrument plugin support, but it's probably time to get on the actual piano roll editing part of the project and leave that stuff until afterwards.
Kinda ruins my flow though, you know?
Back to the Hack
Well.. finally fully moved; I guess it's time to stop screwing around and get back into hacking. Specifically Ardour, since I'm being payed to do so and all.
Looks like we don't have a fancy new cairo canvas to play with. Maybe it will be worth my time to do that first, but I'm going to do some fiddling with displaying MIDI data with gnomecanvas first to get a grasp on things. Wouldn't hurt to have some visually obvious clue that I am, in fact, actually doing something. This summer's project should be more rewarding than last in that sense; most of what I need to do is visual stuff which tends to be more fun since you have something nice and tangible at the end of the day.
Of course, displaying data is one thing. One relatively easy thing. Actually editing it on the other hand.......... that's where the "fun" (i.e. hard) part comes in.
I think a top-down (GUI->implementation) strategy is best here. The Grand Battle Plan(TM) goes something like this:
- Get MIDI data displayed in regions, notes visible as one canvas item (rect) per note
- Attach event signals from note canvas items (move, click, etc) to a set of stub methods that encompass all the editing operations
- Now there's a nice centralized area where all the editing operations need to be implemented
- Figure out how the hell to implement them (ie ???????????)
- Implement them (ie Profit!)
Unix Modular Synth
I've decided to use my blog to archive random ideas until I figure something better out (on that note, anyone know of any good RDF based mind-map software, or something generic that can easily be used as one?)
<drobilla> crazy thought of the day: write an LV2 extension for text stream input/output ports, and you can write a trivial unix cmd line utility wrapper plugin <drobilla> anyone want to convert midi to text then run it through sed and grep and convert back to midi in a modular? :) <gordonjcp> ahahaha <gordonjcp> you sick sick man <drobilla> unix modular, best thing ever
Ingix? Really wouldn't be that hard. Could even make it parallelize forkey pipelines, if the LV2 contexts extension is powerful enough. More on that later...
« Page 12 / 12