Posts

Meson Features and Packaging

Recently, I have been porting all of the software I maintain to meson. For the most part, this has been a nice improvement. Meson strives to be a generally sensible and coherent build system that addresses the needs of both upstream developers and packagers, and it shows. It isn't perfect (like all software), but it's a pretty big improvement over what came before in many ways.

One disagreement has come up, however, and I think it - at best - betrays a misunderstanding of meson feature options, and - at worst - threatens to effectively destroy their utility entirely for no good reason. This disagreement originates from arch-meson, which adds --auto-features=enabled by default. This isn't technically wrong, but it's problematic at scale. To explain why, I first need to explain how meson feature options work.

Meson feature options are a better solution to the --enable-this and --disable-that style options of autoconf and similar. They make all the possible states of a feature clear and explicit. Those states are:

  • disabled, which unconditionally disables the feature.
  • auto, which enables the feature if possible (typically if dependencies are found).
  • enabled, which unconditionally enables the feature.

The default value for an option can be any of the three, and is given by the developer in the option definition. This is a better system because there are three states with clear meanings. They allow the builder to:

  • Ensure that a feature is definitely not included.
  • Ask that a feature be included if possible.
  • Ensure that a feature is definitely included (failing to configure otherwise).

Each of these states (and desires) are distinct and, importantly, their definitions are not conditional. There is no "unless" in the definition of disabled or enabled, and adding one would destroy their utility. It's not "enabled unless someone feels like it's fine to disable", it's just "enabled".

Concretely, for enabled in particular, this means that, as a builder of whatever meson project, you can:

meson setup build -Dsomething=enabled

Then, you know that if configuration succeeded, the something feature is enabled. This is important. It is also the only way to reliably/mechanically ensure that a build contains a feature. "Destroying" it (as something meson users can generally expect) would therefore mean that there there is no way to ensure that a build contains a feature in general. Surely everyone - especially packagers - can agree that's bad.

Hopefully that's a decent enough explanation of why meson feature options are a good, coherent, and reliable system.

What's the problem then? Well, there is that convenience option, --auto-features=enabled, which enables all features for convenience. This allows the builder to override the developer-chosen defaults, and force the build script to be explicit about all non-enabled features. This is, again, not technically wrong, and is indeed convenient at times.

The problem is more of a social one. Because Arch chose to make this the default (unlike most other distributions, as far as I can tell), people have apparently adopted the mentality that --auto-features=enabled not building is an upstream bug. It isn't. The shorthand --auto-features is just that - a shorthand for enabling all features. It is equivalent to explicitly enabling every feature independently, so doing so is quite literally asking for the build to fail if any feature is not present.

Where this gets weird is when features seem unlikely, or even impossible, to build on a given system. The bug filed against my project was about Cocoa on Linux. I get it, Cocoa is mostly a MacOS thing, and is almost certainly unavailable elsewhere, but my argument is that this doesn't matter at all. Even if it were literally impossible for Cocoa to exist on top of, say, FreeBSD somewhere (which it isn't, and cross-compiling is a thing anyway), it still wouldn't matter. A similar case can be made for optional dependencies that are not publicly available at all (something I have done in the past). Feature options working as expected in general is much more important than any of this subjective case-specific thinking.

"This doesn't work for me, so the package should not build when this feature is enabled" is not a good argument, regardless of why it doesn't work. It doesn't matter if you imagine a feature "can't" be built on some system (anyone who has been in this game long enough should be able to see where that slippery slope leads...). The power of feature options depend on the states meaning just what they say, and making enabled sometimes effectively equivalent to auto destroys that power. There's not really any point in using features at that point, you might as well just use a boolean flag.

Put another way, the proposed "fix" here would make it so that:

meson setup build -Dsomething=enabled

Can successfully configure, but without the something feature being included. Et voila, we've made the feature states effectively meaningless. That would be a bug, and letting this mentality take over would resign us back to manually wading through logs to see if what we asked for actually happened.

It's also worth noting that feature options are deeply integrated all over the place in meson, for example, you can pass them to as "required" parameters and the right thing will happen. Diluting the meaning of enabled would cause deeper problems than the builder-visible ones explained here. What does it even mean for a feature to be enabled, except not actually enabled, when that feature status is being used to dictate that another dependency is required? Everything falls apart. The feature option system, when used as intended, is designed well, but messing around with the meaning of enabled like this ruins it. I don't think it's hyperbole to say it "destroys" the whole thing: it turns a sensible and coherent system into some confused boolean, which is both objectively less powerful, and, well... confusing.

This may seem trivial when you're zoomed in to a single project or handful of options, but globally it is a much bigger deal than it might seem at first. Even in my relatively tidy third-party source directory, there's nearly a thousand meson options. I can't imagine how many there are globally. The basic way that features work can't depend on the builder understanding details about every single option, that would be a complete disaster. It is crucial that disabled and enabled mean exactly what they say, always.

I have never seen this problem come up from a user configuring the project themselves. Presumably, people realize that explicitly requesting a potentially failed build, then complaining that it did what they ask, is... rather silly. This gets obscured when a tool (arch-meson) does it automatically, but remains just as silly. It makes no difference from the perspective of a meson package or its maintainer: by enabling all features, the builder has explicitly requested ignoring the defaults and failing to build if any feature was not present.

Yet, here we are, with upstream developers getting "bug" reports for feature options working exactly how they are supposed to work. This, unfortunately, means that Arch is putting toxic pressure on the meson ecosystem, pushing upstream developers to make things objectively worse.

I don't know what the solution to that is, and I'm not an Arch developer (though I'm typing this on an Arch-derived distribution and have nothing in particular against it). Breaking all the existing packages that use arch-meson is probably not going to happen, but perhaps improving the documentation would be good enough.

All I know is that --auto-features=enabled not building is very definitely not a bug. Quite the opposite, that is literally the whole point of feature options. If there's something out there that's leading people to thinking that enabled features not building is a bug, then it is misleading people in a bad way, and needs to be fixed.

Beautiful C and C++ Documentation with Sphinx

Like many, I've long suffered under the antiquated and inflexible HTML documentation generated by Doxygen. Having recently worked on some Python documentation using Sphinx, though, I found it powerful and pleasant enough to use. It also has a way of encouraging actually writing documentation, rather than just generating a dump of glorified comments, which is a good thing. Though I'm not at all a fan of ReStructuredText syntax (which at times seems like it's trying to be cryptic on purpose), Sphinx is undeniably powerful, and I like the "assemble a bunch of plainish text files" approach in general. The support for multiple languages is also very appealing, though not without its problems, as we'll get to.

So, is it possible to use Sphinx to generate documentation for C and C++ libraries? Yes! As explained somewhat recently in a post by Sy Brand, there is a project called Breathe that integrates Doxygen (for extracting documentation) with Sphinx (for generating output). That sounded promising, so I attempted to migrate a library to using Breathe instead of Doxygen's HTML support. Unfortunately, though, I encountered quite a few roadblocks where I couldn't quite get output that I was happy with. Worse, the project itself is very complicated, and as I poked around in swaths of originally generated but manually modified code, I decided that Breathe was not for me. That would feel like just exchanging one inflexible and unhackable system for another.

What, then, to do? Though I realize that deep integration via modules like Breathe is usually the way things are done with Sphinx, I am a KISS sort of person, so I like to think of it as something more like a Static Site Generator: it reads a bunch of plainish text input files, and outputs HTML (or whatever other presentation format). How do we describe C and C++ things in Sphinx? It turns out that recent versions have built-in support for these "domains" now, which define markup for describing everything in these languages. This means that everything to do with nicely formatting and cross-referencing C and C++ is already dealt with out of the box. Excellent.

So, taking a step back and assessing the situation: we have some XML files that describe the documentation, and we have a tool that reads text files and produces nice documentation. This strikes me as a relatively straightforward task for a nice and simple "files in, files out" script, not somewhere a Goldbergian contraption that mashes Doxygen into Sphinx is required. So, after investigating any other promising options (no such luck), I resigned myself to trying to write such a thing, at the very least to see if it's feasible. I certainly have no time or interest in writing and maintaining a Documentation System, but a self-contained script to convert one thing to another seems reasonable enough.

As it turns out, I wouldn't call it trivial, but it's certainly feasible. I ended up with a ~700 line Python script that does everything I need (though this is of course not the same as everything possible). It's a bit "gluey" and makes some assumptions about the structure and so on, but it does the job and is something I feel I can maintain as necessary. I won't be publishing or supporting this as an independent project any time soon, and make no claims about it being general purpose, but feel free to steal it if any of this sounds appealing.

With this, I was able to get around some long-standing gripes I have with Doxygen, and easily make whatever I wanted to happen a reality, so I'm pretty happy with this approach. Everything is nicely decoupled, so I don't feel over-invested in any of the tools involved. If, for example, someone finally writes a good clang-based extractor that gains traction (JSON please, I did not enjoy this revisitation of the horrors of XML at all), I should be able to switch to using that easily enough. I've actually found this somewhat crude and UNIXey approach quite convenient: you can simply look at the ReST files to understand what is happening, or tweak them a bit and run Sphinx to test what you're aiming for, and so on. Text files are good.

So, after however many years, I think I've found an approach to documentation I'm actually quite happy with, that can support all of the languages that I use, and in general doesn't seem to get in my way. Hooray. For starters, I did my window system portability layer, Pugl. The generated documentation for the C API can be seen at https://lv2.gitlab.io/pugl/c/singlehtml/, and the C++ at https://lv2.gitlab.io/pugl/cpp/singlehtml/. This is more or less the standard Alabaster theme with a few tweaks, which I'm not sure feels appropriate for API documentation (and is much more bloated with a bunch of Javascript than I'd like), but it's pretty enough, at least. I'll tinker with themes later when I feel like jumping down that rabbit hole.

The slightly cumbersome links are an artifact of the one problem I encountered using Sphinx domains: you can't really document C and C++ APIs nicely in the same documentation set. If you use the cpp domain everywhere, you get name mangling in links even for C symbols, which is really unfortunate, and you can't really mix them. To take a contrived example, if you have a struct MylibThing in C, then a type alias in C++ like using Thing = MylibThing, Sphinx isn't clever enough to figure out that MylibThing is from C, and will generate warnings and not link correctly. Perhaps someday it will, which would be nice, but for now I opted to simply generate completely separate documentation sets. This means the C documentation is duplicated in the C++ documentation so that things can be hyperlinked, which isn't ideal, but I can live with it. A certain amount of redundancy is inherent in multi-language documentation anyway.

As I add Python bindings to most libraries, having a unified documentation system for all of these languages will be very nice. There is one additional thing I'll need at some point for the LV2 documentation in particular: a domain for RDF properties and classes. The LV2 documentation really suffers from an unnatural code (via Doxygen) and data (via lv2specgen) documentation split, and my hope is that Sphinx can provide a nice environment for writing documentation that refers to both worlds freely. That, unfortunately, will be much more work, but hopefully writing a custom Sphinx domain isn't too hard...

Git migration

I have finally migrated all of my software to git. This was not a very fun process due to the nested nature of my "drobillad" repository, but now all projects live in their own git repositories with history and tags from SVN preserved.

It is still possible to build all my audio software in one step, the top-level repository http://git.drobilla.net/drobillad.git is now a skeleton with git submodules for each project. Anyone using SVN should switch immediately, the SVN repositories will remain in their current state for the foreseeable future but all development activity will move to git.

All the repositories are available in cgit, but changes and tickets and so on are in Trac as always. Happy hacking.

LV2 Plugin Control Units in Ardour

LV2 has had a "units" extension since the beginning, which allows plugins to specify units like Hz or dB for their controls. To date this information has not been used very widely by hosts, so I've done some work in Ardour to use this information for better UI generation and intelligent plugin control.

Units can specify a format string which describes how to print a value in that unit. This is now used to draw the text on control sliders:

An Ardour control dialog for an LV2 plugin.

The same controls are used in automation lane headers. If the control is a note number, then right-clicking will present a menu with an option to pop up a note selector dialog where the value can be chosen on a piano keyboard:

The select note menu on a note number port.

The Ardour note selector dialog.

Similarly, numeric frequency controls have a context menu which can set the value to a specific number of beats in the current tempo, if the frequency range is low:

Setting a low frequency port in beats.

If the frequency range is high, then numeric frequency ports can be set with the note selector dialog just like note numbers:

Setting an audible frequency port by note.

In the future, it would be nice to have this idea applied more extensively so automation lanes can "pretend" a port is in the desired unit, for example allowing the user to automate an LFO frequency in beats, or a cutoff frequency in notes. Until then, being able to at least easily set controls to musically sensible values makes many production tasks easier, particularly in electronic music where it's often desirable to set plugin controls based on key or tempo.

Up next is setting time ports based on tempo, for cases like setting delay lines to a certain number of beats, but many plugins are missing the unit information required to make this possible. Hopefully better and more widespread host support will provide some incentive for plugin authors to specify the units of their ports. It is very simple to do so, see the LV2 units documentation for examples.

Pretty names in Patchage via Jack Metadata

The much-awaited (by me, at least) Jack metadata API has arrived. This will allow us to easily achieve many new things with minimal/nonexistent API friction. One of the simplest and most obvious is pretty names for Jack clients and ports, so I've chosen this as the first thing to tackle (as part of a drive to get Patchage polished up for a much overdue release).

Unfortunately, as far as I can tell, there is a chicken & egg scenario here since nothing is setting this metadata yet. So, I've made Jalv and Ingen both set pretty names for their ports. In the case of Jalv, the "pretty name" is the label given in the plugin data (distinct from the LV2 "symbol" which is restricted and unique).

Firing up Jalv with the Tal Dub III plugin (LV2 port courtesy KXStudio), we can see the port symbols, which are a bit awkward for end users with their prefixes and underscores. Conflating strong identifiers with human-readable labels is a serious design error I learned of the hard way, but that's a discussion for another time...

Enable "Human Names" in the view menu, or press C-h, and voilĂ , we see the pretty names set in Jack metadata (if present) instead.

Tal Dub III in Jalv as shown by Patchage with human names off.

Tal Dub III in Jalv as shown by Patchage with human names enabled.

The metadata API is very simple to use for ports, though there seems to be a hole in the API which makes it difficult to get the UUID for your client to set metadata (I want a simple jack_client_uuid, like jack_port_uuid, but it seems you have to get a string UUID and parse it to a jack_uuid_t, clunky enough that I just didn't bother). In any case, I am happy to see a low-friction mechanism in Jack which apps can use to share metadata towards making a better user experience.

It will be interesting to see what sort of information proves useful and becomes established/standard. For those of us of a mad scientist bent who live in a nest of patch cables, a CV tag seems like another obvious simple step, but for everyone, a big step is finally having meaningful port grouping and channel roles. I have always liked to joke that Jack (like LADSPA) doesn't really even do stereo, but with metadata, we can mark up stereo, 5.1, Ambisonics, etc., and other clients will be able to make sense of the channel assignments without resorting to dirty kludges based on guessing from names. All without changing the ABI one bit. Good stuff.

Labeled MIDI controller values in Ardour

A while ago, I implemented MIDI patch, controller, and note name support for Ardour.

There was one thing missing from that work: labelled controller values. This is particularly useful for controllers that don't represent a continuous numeric parameter, but instead have a set of specific values. For example, the Moog Minitaur supports CC #86 to control the LFO sync clock division, but it's not obvious how a 0-127 number maps to a clock division. From the manual, we can learn that 61-67 is "1/4", 68-73 is "1/8 Dot", and so on.

Translate that information into standard MIDINameDocument format, and a few hours of hacking later... voila, Ardour displays something sensible when hovering over an automation point instead of a meaningless number.

Ardour displaying a labelled MIDI controller value when hovering over an automation point.

If you want to add support for your MIDI device or program, my .midnam documents for the Moog Minitaur, MF-104M, and MF-108M, included in the Ardour distribution all contain examples of labelled values. It should be relatively obvious how to copy and modify these for other devices.

Interactive force-directed graph layout in Ingen

GraphViz has, unfortunately, broken reverse compatibility of their library API, which has been nagging as a maintenance headache for FlowCanvas and its successor Ganv for a while now. It would be nice to just get rid of that dependency altogether...

Meanwhile, I had to write some force-directed graph layout (FDGL) code for another project...

... You can probably see where this is going. In the spirit of it theoretically being a holiday (there are no holidays in practice in grad school), I spent some time implementing live FDGL in Ganv, and recorded a video of interactive layout in Ingen to show progress.

One nice thing about using live FDGL in a program like Ingen is that the user can customize the layout in an intuitive tactile way, since objects behave physical laws. This video demonstrates some manual rearranging of a patch, and a total reconfiguration by switching the signal flow direction. Manual tweaking of the layout like this was not possible with the previous GraphViz-based layout code.

It took quite a bit of tweaking to get the physics working well, so layouts looked good but the patch didn't explode if there are disconnected components. It's still not quite ideal, but usable.

This is implemented at the Ganv level and thus works in Ingen (as shown), Patchage, and Machina. I'm still not confident enough in this implementation to enable it by default and/or drop Graphviz quite yet, but the adventurous who follow svn, ./waf configure --debug --fdgl. If you're hit by the graphviz breakage (and I still haven't fixed the check...), add --no-graphviz to work around the problem.

Better stock LV2 plugin UIs

Today I implemented better UI generation for LV2 plugin controls in Jalv, particularly grouping controls under headings which makes a big difference. Unfortunately few plugins group their controls currently, but hopefully more host support will provide the incentive to do so.

Other improvements:

  • Added a spinbutton to slider controls for precisely setting values numerically
  • Much more efficient layout to pack more controls on the screen at once
  • Support for named scale points ("ticks") on non-enumeration ports

Here is the UI generated for Amsynth (after I added the necessary metadata):

Generic Jalv UI for Amsynth LV2

Still pretty utilitarian, but much more usable, which is the important thing. Of course, this plugin is quite large, and has a pretty good custom UI, but I happened to be polishing up its metadata anyway and the controls group nicely so I used it as my test case.

Maybe some day this code will get smarter and evolve into a library that can be used by other Gtk hosts. Better group layout is the obvious "next level" advancement, a flat linear list of controls is pretty limited. Unfortunately there's no stock Gtk container which can do text-like reflow, which would be ideal. Perhaps a simpler scheme based on a maximum reasonable width for a controller would do, where the table will expand to have more columns if the window is wide enough for several.

Most of the metadata required to generate a good UI is also useful for other purposes, for example groups make building a decent menu of many controls feasible (e.g. to add automation lanes in Ardour), and some types of host UIs like a modular patcher inherently need to generate their own "UI" of sorts.

Good metadata is not at odds with custom UIs, they each have their uses, but it is important to remove the burden of custom UIs for simple plugins with just a few controls that really don't need them. It's a waste of that most precious of all resources - developer time - to have to code UIs for a few sliders and toggles. Hopefully better host generated UIs and support for more advanced controls like file selectors free up developers to spend more time making useful plugins and less time wrestling with GUI toolkits.

It is, after all, all about the sound.

Ardour MIDI Patch, Controller, and Note Names

Ardour has supported displaying MIDI patch names loaded from a MIDINameDocument (or "midnam" file) for a while, though only patches, and even that was pretty flaky. Since this has become an itch for me and it's time for Ardour MIDI to reach release quality, I took some time to give it a serious overhaul. In this case, a picture or two is indeed worth a thousand words:

Human readable MIDI note names in Ardour 3

Human readable MIDI controller menu in Ardour

The note names are mainly useful for doing percussion. Several of the bundled midnam files have note names defined, and I have added one for General MIDI drums, which more or less corresponds to a lot of instruments, including Hydrogen.

Controller names, to my surprise, were not present in any of the existing midnam files at all. I wanted to sequence a hardware synth (the Moog Minitaur) without constantly referring to the manual and trying to remember which CC is which, so I wrote a new midnam file for the device and implemented support in Ardour. I am very disappointed to learn that there is no ability to group controllers, which really is the best way to do things, so I may embrace and extend (in a completely backwards compatible way) the format in the future to provide a better interface.

The quality of midnam files scattered around the 'net is atrocious. The MMA actually hosts a DTD for it, but most files use the wrong DOCTYPE (usually with a broken link), among other problems. I fixed and cleaned up all the bundled ones in Ardour, validated them against the DTD, and left a README in that directory about how to do so. Hopefully this, along with incentive added by these features, encourages the community to grow a nice set of quality midnam files.

So, MIDI name support in Ardour is now in pretty good shape. Now we just need to add the corresponding LV2 support...

Reviving Machina

In the past few years of hammering on my repository while moulding Ingen into the modular of my dreams, my poor little generative sequencer Machina was often left to rot. That's a shame, because it's a fun tool to play with if you're into generative gadgets and pretty graphs.

So, I spent the weekend working on turning Machina back into a usable app. It's come a very long way in two days, from a mess that doesn't even compile to most functionality working pretty well:

A small Machina machine created by step recording

Aside from rewriting a lot of the code, testing, and fixing things that were broken, I've changed the conceptual model a bit. There is now always just one start node for a machine. This tends to create more manageable structures, especially when recording several takes into the same machine.

Step recording has also been significantly improved. You can build machines entirely in the UI, but this is pretty tedious. It's much more effective to record, or step record if you want regular timing and no delays in-between notes. The above pictured machine was created by step recording MIDI, though I have tweaked the edge probabilities with the mouse. You can also import a MIDI file, which is an easy way to get a big machine to work with.

Hopefully I can find the time to turn this into an LV2 plugin and release it, but for now it is an unreleased Jack application. Give it a try if you're feeling adventurous, I would appreciate any feedback.

Page 1 / 3 »