Well, I did the Redland modularization thing to tear those dependencies out. Luckily for me, Dave Becket wrote the storage module stuff in anticipation of modularization, so it was actually really easy (what’s with people named Dave and being absolutely brilliant eh?) So, pending whatever autohell stuff needs doing to make it possible/easy to package into separate bits, that problem’s solved.
Armed with my new anti-whining-user ammo of future dependency dissapearance, I pretty much rewrote all of SLV2 to use one big model of all LV2 related RDF on the system. I figured with an in memory model, querying the whole thing should be quite fast (it’s not that big), and there’s some fun to be had with a big single web of data containing everything plugin/extension/whatever related. Happily, going through and parsing all the files is nearly instantaneous (Raptor has to be the fastest RDF parser in the west by a long shot)…
… and non-trivial (but necessary) queries take so long and chew 100% CPU that they’re unusable.
Shit.
Long story short, debugging and/or rewriting half of Rasqal isn’t exactly high on my Things I Feel Like Doing Right Now list, so I rewrote SLV2 again. Now each plugin has it’s own in-memory model, which works quite well. Lost that big web of data, which I thought was a pretty cool idea, but hopefully whatever’s wrong with Redland can get sorted out by the time anyone has a real use for that anyway. The plugin model can still load plugin extension bundles etc. into the same model, and that’s the important part anyway (which makes LV2 plugins extendable without having to modify the plugin’s bundle itself). Not really sure how the main ontology and LV2 extensions like categories and other system stuff fit in to this just yet, but that’s a train of thought for another day.
The performance problems Zynjacku was having are much better now. The loading guage no longer pauses on plugins with large numbers of ports, it zips along at a constant speed and is quite a bit faster overall.
It could be faster still if I added an API to get plugins with a custom query, but that will probably have to wait until the Redland query lock-up thing is solved. Anyway, it’s much, much better now, and the API has been redone in a better more abstractioney way that should allow any kind of redesign behind the scenes (as far as cacheing models and whatnot) without breaking the API. Also the API is now reentrant (well, if Redland is anyway?) which I figure is probably going to be an issue at some point once some plugin and it’s host both use SLV2.
Now to just get the thing fully working again…
Steve Harris | 19-Apr-07 at 5:38 am | Permalink
The slow query in rasqal thing should be solvable - probably a bug. In memory stores should be queryable in sub-millisecond times.
Dave Robillard | 19-Apr-07 at 4:13 pm | Permalink
It’s present in the last 2 versions up through and including SVN. Pretty glaring show-stopper of a bug, which surprises me, but oh well.
It would certainly be nice to have one big model, but debugging Rasqal would probably be way too huge of a digression and it’s in flux at the moment anyway. For now I’ll take the path of abstracting it away fully so it can change in the future without breaking the API at all.
A forced dependency on a very recent Redland (once it even got released with the fixes, which would take a while) wouldn’t be the greatest thing for LV2 anyway.