Beautiful C and C++ Documentation with Sphinx
Like many, I've long suffered under the antiquated and inflexible HTML documentation generated by Doxygen. Having recently worked on some Python documentation using Sphinx, though, I found it powerful and pleasant enough to use. It also has a way of encouraging actually writing documentation, rather than just generating a dump of glorified comments, which is a good thing. Though I'm not at all a fan of ReStructuredText syntax (which at times seems like it's trying to be cryptic on purpose), Sphinx is undeniably powerful, and I like the "assemble a bunch of plainish text files" approach in general. The support for multiple languages is also very appealing, though not without its problems, as we'll get to.
So, is it possible to use Sphinx to generate documentation for C and C++ libraries? Yes! As explained somewhat recently in a post by Sy Brand, there is a project called Breathe that integrates Doxygen (for extracting documentation) with Sphinx (for generating output). That sounded promising, so I attempted to migrate a library to using Breathe instead of Doxygen's HTML support. Unfortunately, though, I encountered quite a few roadblocks where I couldn't quite get output that I was happy with. Worse, the project itself is very complicated, and as I poked around in swaths of originally generated but manually modified code, I decided that Breathe was not for me. That would feel like just exchanging one inflexible and unhackable system for another.
What, then, to do? Though I realize that deep integration via modules like Breathe is usually the way things are done with Sphinx, I am a KISS sort of person, so I like to think of it as something more like a Static Site Generator: it reads a bunch of plainish text input files, and outputs HTML (or whatever other presentation format). How do we describe C and C++ things in Sphinx? It turns out that recent versions have built-in support for these "domains" now, which define markup for describing everything in these languages. This means that everything to do with nicely formatting and cross-referencing C and C++ is already dealt with out of the box. Excellent.
So, taking a step back and assessing the situation: we have some XML files that describe the documentation, and we have a tool that reads text files and produces nice documentation. This strikes me as a relatively straightforward task for a nice and simple "files in, files out" script, not somewhere a Goldbergian contraption that mashes Doxygen into Sphinx is required. So, after investigating any other promising options (no such luck), I resigned myself to trying to write such a thing, at the very least to see if it's feasible. I certainly have no time or interest in writing and maintaining a Documentation System, but a self-contained script to convert one thing to another seems reasonable enough.
As it turns out, I wouldn't call it trivial, but it's certainly feasible. I ended up with a ~700 line Python script that does everything I need (though this is of course not the same as everything possible). It's a bit "gluey" and makes some assumptions about the structure and so on, but it does the job and is something I feel I can maintain as necessary. I won't be publishing or supporting this as an independent project any time soon, and make no claims about it being general purpose, but feel free to steal it if any of this sounds appealing.
With this, I was able to get around some long-standing gripes I have with Doxygen, and easily make whatever I wanted to happen a reality, so I'm pretty happy with this approach. Everything is nicely decoupled, so I don't feel over-invested in any of the tools involved. If, for example, someone finally writes a good clang-based extractor that gains traction (JSON please, I did not enjoy this revisitation of the horrors of XML at all), I should be able to switch to using that easily enough. I've actually found this somewhat crude and UNIXey approach quite convenient: you can simply look at the ReST files to understand what is happening, or tweak them a bit and run Sphinx to test what you're aiming for, and so on. Text files are good.
The slightly cumbersome links are an artifact of the one problem I encountered
using Sphinx domains: you can't really document C and C++ APIs nicely in the
same documentation set. If you use the
cpp domain everywhere, you get name
mangling in links even for C symbols, which is really unfortunate, and you
can't really mix them. To take a contrived example, if you have a
MylibThing in C, then a type alias in C++ like
using Thing = MylibThing,
Sphinx isn't clever enough to figure out that
MylibThing is from C, and will
generate warnings and not link correctly. Perhaps someday it will, which would
be nice, but for now I opted to simply generate completely separate
documentation sets. This means the C documentation is duplicated in the C++
documentation so that things can be hyperlinked, which isn't ideal, but I can
live with it. A certain amount of redundancy is inherent in multi-language
As I add Python bindings to most libraries, having a unified documentation system for all of these languages will be very nice. There is one additional thing I'll need at some point for the LV2 documentation in particular: a domain for RDF properties and classes. The LV2 documentation really suffers from an unnatural code (via Doxygen) and data (via lv2specgen) documentation split, and my hope is that Sphinx can provide a nice environment for writing documentation that refers to both worlds freely. That, unfortunately, will be much more work, but hopefully writing a custom Sphinx domain isn't too hard...
I have finally migrated all of my software to git. This was not a very fun process due to the nested nature of my "drobillad" repository, but now all projects live in their own git repositories with history and tags from SVN preserved.
It is still possible to build all my audio software in one step, the top-level repository http://git.drobilla.net/drobillad.git is now a skeleton with git submodules for each project. Anyone using SVN should switch immediately, the SVN repositories will remain in their current state for the foreseeable future but all development activity will move to git.
LV2 Plugin Control Units in Ardour
LV2 has had a "units" extension since the beginning, which allows plugins to specify units like Hz or dB for their controls. To date this information has not been used very widely by hosts, so I've done some work in Ardour to use this information for better UI generation and intelligent plugin control.
Units can specify a format string which describes how to print a value in that unit. This is now used to draw the text on control sliders:
The same controls are used in automation lane headers. If the control is a note number, then right-clicking will present a menu with an option to pop up a note selector dialog where the value can be chosen on a piano keyboard:
Similarly, numeric frequency controls have a context menu which can set the value to a specific number of beats in the current tempo, if the frequency range is low:
If the frequency range is high, then numeric frequency ports can be set with the note selector dialog just like note numbers:
In the future, it would be nice to have this idea applied more extensively so automation lanes can "pretend" a port is in the desired unit, for example allowing the user to automate an LFO frequency in beats, or a cutoff frequency in notes. Until then, being able to at least easily set controls to musically sensible values makes many production tasks easier, particularly in electronic music where it's often desirable to set plugin controls based on key or tempo.
Up next is setting time ports based on tempo, for cases like setting delay lines to a certain number of beats, but many plugins are missing the unit information required to make this possible. Hopefully better and more widespread host support will provide some incentive for plugin authors to specify the units of their ports. It is very simple to do so, see the LV2 units documentation for examples.
Pretty names in Patchage via Jack Metadata
The much-awaited (by me, at least) Jack metadata API has arrived. This will allow us to easily achieve many new things with minimal/nonexistent API friction. One of the simplest and most obvious is pretty names for Jack clients and ports, so I've chosen this as the first thing to tackle (as part of a drive to get Patchage polished up for a much overdue release).
Unfortunately, as far as I can tell, there is a chicken & egg scenario here since nothing is setting this metadata yet. So, I've made Jalv and Ingen both set pretty names for their ports. In the case of Jalv, the "pretty name" is the label given in the plugin data (distinct from the LV2 "symbol" which is restricted and unique).
Firing up Jalv with the Tal Dub III plugin (LV2 port courtesy KXStudio), we can see the port symbols, which are a bit awkward for end users with their prefixes and underscores. Conflating strong identifiers with human-readable labels is a serious design error I learned of the hard way, but that's a discussion for another time...
Enable "Human Names" in the view menu, or press C-h, and voilà, we see the pretty names set in Jack metadata (if present) instead.
The metadata API is very simple to use for ports, though there seems to be a hole in the API which makes it difficult to get the UUID for your client to set metadata (I want a simple jack_client_uuid, like jack_port_uuid, but it seems you have to get a string UUID and parse it to a jack_uuid_t, clunky enough that I just didn't bother). In any case, I am happy to see a low-friction mechanism in Jack which apps can use to share metadata towards making a better user experience.
It will be interesting to see what sort of information proves useful and becomes established/standard. For those of us of a mad scientist bent who live in a nest of patch cables, a CV tag seems like another obvious simple step, but for everyone, a big step is finally having meaningful port grouping and channel roles. I have always liked to joke that Jack (like LADSPA) doesn't really even do stereo, but with metadata, we can mark up stereo, 5.1, Ambisonics, etc., and other clients will be able to make sense of the channel assignments without resorting to dirty kludges based on guessing from names. All without changing the ABI one bit. Good stuff.
Labeled MIDI controller values in Ardour
A while ago, I implemented MIDI patch, controller, and note name support for Ardour.
There was one thing missing from that work: labelled controller values. This is particularly useful for controllers that don't represent a continuous numeric parameter, but instead have a set of specific values. For example, the Moog Minitaur supports CC #86 to control the LFO sync clock division, but it's not obvious how a 0-127 number maps to a clock division. From the manual, we can learn that 61-67 is "1/4", 68-73 is "1/8 Dot", and so on.
Translate that information into standard MIDINameDocument format, and a few hours of hacking later... voila, Ardour displays something sensible when hovering over an automation point instead of a meaningless number.
If you want to add support for your MIDI device or program, my .midnam documents for the Moog Minitaur, MF-104M, and MF-108M, included in the Ardour distribution all contain examples of labelled values. It should be relatively obvious how to copy and modify these for other devices.
Interactive force-directed graph layout in Ingen
GraphViz has, unfortunately, broken reverse compatibility of their library API, which has been nagging as a maintenance headache for FlowCanvas and its successor Ganv for a while now. It would be nice to just get rid of that dependency altogether...
Meanwhile, I had to write some force-directed graph layout (FDGL) code for another project...
... You can probably see where this is going. In the spirit of it theoretically being a holiday (there are no holidays in practice in grad school), I spent some time implementing live FDGL in Ganv, and recorded a video of interactive layout in Ingen to show progress.
One nice thing about using live FDGL in a program like Ingen is that the user can customize the layout in an intuitive tactile way, since objects behave physical laws. This video demonstrates some manual rearranging of a patch, and a total reconfiguration by switching the signal flow direction. Manual tweaking of the layout like this was not possible with the previous GraphViz-based layout code.
It took quite a bit of tweaking to get the physics working well, so layouts looked good but the patch didn't explode if there are disconnected components. It's still not quite ideal, but usable.
This is implemented at the Ganv level and thus works in Ingen (as
shown), Patchage, and Machina. I'm still not confident enough in this
implementation to enable it by default and/or drop Graphviz quite yet,
but the adventurous who follow svn,
./waf configure --debug --fdgl. If
you're hit by the graphviz breakage (and I still haven't fixed the
--no-graphviz to work around the problem.
Better stock LV2 plugin UIs
Today I implemented better UI generation for LV2 plugin controls in Jalv, particularly grouping controls under headings which makes a big difference. Unfortunately few plugins group their controls currently, but hopefully more host support will provide the incentive to do so.
- Added a spinbutton to slider controls for precisely setting values numerically
- Much more efficient layout to pack more controls on the screen at once
- Support for named scale points ("ticks") on non-enumeration ports
Here is the UI generated for Amsynth (after I added the necessary metadata):
Still pretty utilitarian, but much more usable, which is the important thing. Of course, this plugin is quite large, and has a pretty good custom UI, but I happened to be polishing up its metadata anyway and the controls group nicely so I used it as my test case.
Maybe some day this code will get smarter and evolve into a library that can be used by other Gtk hosts. Better group layout is the obvious "next level" advancement, a flat linear list of controls is pretty limited. Unfortunately there's no stock Gtk container which can do text-like reflow, which would be ideal. Perhaps a simpler scheme based on a maximum reasonable width for a controller would do, where the table will expand to have more columns if the window is wide enough for several.
Most of the metadata required to generate a good UI is also useful for other purposes, for example groups make building a decent menu of many controls feasible (e.g. to add automation lanes in Ardour), and some types of host UIs like a modular patcher inherently need to generate their own "UI" of sorts.
Good metadata is not at odds with custom UIs, they each have their uses, but it is important to remove the burden of custom UIs for simple plugins with just a few controls that really don't need them. It's a waste of that most precious of all resources - developer time - to have to code UIs for a few sliders and toggles. Hopefully better host generated UIs and support for more advanced controls like file selectors free up developers to spend more time making useful plugins and less time wrestling with GUI toolkits.
It is, after all, all about the sound.
Ardour MIDI Patch, Controller, and Note Names
Ardour has supported displaying MIDI patch names loaded from a MIDINameDocument (or "midnam" file) for a while, though only patches, and even that was pretty flaky. Since this has become an itch for me and it's time for Ardour MIDI to reach release quality, I took some time to give it a serious overhaul. In this case, a picture or two is indeed worth a thousand words:
The note names are mainly useful for doing percussion. Several of the bundled midnam files have note names defined, and I have added one for General MIDI drums, which more or less corresponds to a lot of instruments, including Hydrogen.
Controller names, to my surprise, were not present in any of the existing midnam files at all. I wanted to sequence a hardware synth (the Moog Minitaur) without constantly referring to the manual and trying to remember which CC is which, so I wrote a new midnam file for the device and implemented support in Ardour. I am very disappointed to learn that there is no ability to group controllers, which really is the best way to do things, so I may embrace and extend (in a completely backwards compatible way) the format in the future to provide a better interface.
The quality of midnam files scattered around the 'net is atrocious. The MMA actually hosts a DTD for it, but most files use the wrong DOCTYPE (usually with a broken link), among other problems. I fixed and cleaned up all the bundled ones in Ardour, validated them against the DTD, and left a README in that directory about how to do so. Hopefully this, along with incentive added by these features, encourages the community to grow a nice set of quality midnam files.
So, MIDI name support in Ardour is now in pretty good shape. Now we just need to add the corresponding LV2 support...
In the past few years of hammering on my repository while moulding Ingen into the modular of my dreams, my poor little generative sequencer Machina was often left to rot. That's a shame, because it's a fun tool to play with if you're into generative gadgets and pretty graphs.
So, I spent the weekend working on turning Machina back into a usable app. It's come a very long way in two days, from a mess that doesn't even compile to most functionality working pretty well:
Aside from rewriting a lot of the code, testing, and fixing things that were broken, I've changed the conceptual model a bit. There is now always just one start node for a machine. This tends to create more manageable structures, especially when recording several takes into the same machine.
Step recording has also been significantly improved. You can build machines entirely in the UI, but this is pretty tedious. It's much more effective to record, or step record if you want regular timing and no delays in-between notes. The above pictured machine was created by step recording MIDI, though I have tweaked the edge probabilities with the mouse. You can also import a MIDI file, which is an easy way to get a big machine to work with.
Hopefully I can find the time to turn this into an LV2 plugin and release it, but for now it is an unreleased Jack application. Give it a try if you're feeling adventurous, I would appreciate any feedback.
Host-generated LV2 file choosers
LV2 has been slowly but surely moving towards message-based control to overcome the limitations of static ports. The required low level facilities have been available for a while now, but the lack of a standard way to document message-based parameters and host UI support has been a sticking point. One particularly pressing need is file parameters. The current sample example has a stop-gap UI to enable sample loading, but parameters really should be host controllable and simple plugins should not need a custom UI to be usable. Today I did some work towards resolving that situation.
Note this is the built-in UI generated by Jalv and not a custom plugin UI. Unlike traditional LADSPA style controls, the impulse parameter is not a port, but is set by sending a message to an event port. The plugin lists parameters with the new patch:writable property, so the host knows which controls to display (or which parameters it can automatically set). Parameters are described as normal RDF properties, so any existing property can be used, or plugins can define them as needed in their data files (don't let the "RDF" scare you, the description of a Property is as simple as LV2 ports). One nice advantage of this over ports is that a project can describe a parameter once and use it in many plugins, without having to describe the parameter every time. This really adds up when you have hundreds of plugins with, say, an identical gain parameter.
These conventions will be established in the next LV2 release (1.4.0), and the sampler example will serve as a test case. When host support becomes established, we can also move towards using messages for numeric parameters, which will finally allow for dynamic parameters, control ramps, and so on.
Page 1 / 3 »