Author Archives: mosu

Running Buildbot on IPv6

As I’ve struggled a bit here I thought I’d make this short write-up for others to find. The following writeup was done against Buildbot 3.6.1 on Ubuntu 22.04 & Arch Linux.

There are two well-known components to Buildbot: one master process (purposefully ignoring multi-master setup) & one or more workers. The master is the one providing the web interface, scheduling the work, keeping track of results & sending notifications. The workers do the work (well, duh!).

This means that we’re basically dealing with two different scenarios here:

  1. The master process must listen on IPv6 addresses for connections, both for the web interface & the workers which connect on a dedicated port.
  2. The workers must connect via IPv6 to the master.

Using the default configuration or the examples from the website neither of the two components use IPv6, for different reasons.

Both use the twistd Python framework for connection handling.

Making the master listen on IPv6

The configuration for the master is done solely via master.cfg, not in buildbot.tac.

Basically for every connection the master listens on a ConnectionString is used which contains both the port number & the address to listen on, e.g. "tcp:8010:interface=127.0.0.1" One might attempt to simply replace the IPv4 address 127.0.0.1 with an IPv6 address, but that doesn’t work as colons are used to separate key/value pairs within the ConnectionString. Therefore the colons must be escaped. Note that the syntax often used in other programs such as web browser to escape IPv6 addresses, placing them in square brackets such as [::], doesn’t work here either.

How are those colons escaped, though? With backslashes. However, these are Python strings, and Python uses backslashes for escaping in strings as well. Therefore in the source code we have to use two backslashes per colon. The resulting connection string might look like this: "tcp:8010:interface=\\:\\:" (which means “listen on all IPv6 & IPv4 addresses, regardless of the interface, on port 8010”).

Web interface handler

For the web interface listener, which is usually referred to as c["www"] (with c being the an instance of BuildmasterConfig), a dict is used. The port key can be a port number, or a full-blown connection string. Example:

c['www'] = {
    'port': "tcp:8010:interface=\\:\\:",
    'plugins': { 'waterfall_view': {}, 'console_view': {}, 'grid_view': {} },
    'auth': util.UserPasswordAuth({'user': 'supersecret'}),
    'authz': authz,
    'change_hook_auth': [strcred.makeChecker("file:changehook.passwd")],
    'change_hook_dialects': { 'gitlab': { 'secret': 'reallysecret'  } }
}

As hinted in the example the web interface handles both requests from us humans as well as change hooks from version control systems.

Worker handler

The worker handler is configured via c["protocols"]["pb"], and again a dict with a key called port is used which can be set to a connection string. For example:

c['protocols'] = {'pb': {'port': "tcp:interface=\\:\\::port=9989"}}

Making the workers connect via IPv6

The configuration is done in buildbot.tac.

The worker creates an instance of the Worker class. The first two parameters are the host address & the port number of the worker handler on the master’s side. Unlike the master, the parameters cannot seem to be ConnectionStrings.

Furthermore, the twistd client framework used here cannot resolve DNS AAAA records at all. If you provide a host name instead of an IPv6 or IPv4 address as the first argument, the library will only try to resolve it to an A record. For AAAA-only DNS records this means that name resolution fails, and the worker will not start. For DNS entries with both AAAA and A records, only the A record will be used, and the connection will be attempted via IPv4.

Luckily you can use an IPv6 address directly as the first argument. As there are no ConnectionStrings involved, colons can be left as-is. For example (all following arguments left out as they aren’t important):

s = Worker('2001:db8:1234::2', 9989, …)

Making the master connect via IPv6

We have to go back to the master once more as it can also make certain outbound connections that you might want to run via IPv6. These include but aren’t limited to:

  • email notifications
  • version control system status updates (Gitlab, Github etc.)

Email notifications

The email notifier also uses a pair of parameters specifying the SMTP server’s hostname & its port. Unfortunately it uses the same twistd client components that the worker uses, too, with the same DNS resolution issues. Sure, you can specify an IPv6 address directly instead of a host name, but that’ll be problematic if you want to enable TLS as certificate validation will now fail. I don’t have a good solution for IPv6-only setups in this case.

An example without authentication & without TLS:

mailer = reporters.MailNotifier(
    fromaddr="buildbot-master@example.com",
    sendToInterestedUsers=False,
    extraRecipients=["buildbot-failure-recipient@example.com"],
    relayhost='2001:db8:fefe::9876',
    generators=[generator_failing])
c["services"].append(mailer)

Version control status handlers

Handlers that signal the build status to version control systems such as Gitlab & Github don’t use the twistd framework for communication, it seems. They probably use something like urllib2. For their configuration they actually expect a base URL such as https://gitlab.com, and luckily for us, if that address resolves to an AAAA record then the status is indeed sent via IPv6. Nothing for us to do here.

Other considerations

Last but not least: the build instructions you create must also take IPv6 connectivity into account. If you configure git repositories on IPv4-only websites such as Github (shame!), you won’t be able to use them in an IPv6-only environment. Dual-stack setups will be fine, though.

On the other hand: external tools such as git that do not use the same Python libraries will resolve DNS entries to AAAA & A records & prefer AAAA = IPv6 over A = IPv4 — meaning configuring a git repo on sites such as Gitlab will indeed be retrieved via IPv6.

MKVToolNix v72.0 released

Heyo!

here’s MKVToolNix release v72. It fixes quite a lot of bugs, several one of them in the component driving the in-place file modification in mkvpropedit & the GUI’s chapter & header editors. Therefore I urge everyone to update.

You can download the source code or one of the binaries. The Windows and macOS binaries as well as the Linux AppImage are available already. The other Linux binaries are stil being built and will be available over the course of the next couple of hours.

Here are the NEWS since the previous release:

New features and enhancements

  • mkvmerge: AV1 parser: the variable-width OBU size field will be re-written with minimal length if it’s encoded longer than necessary.
  • mkvmerge: when splitting is active the program will output the timestamps actually used for making the decision when to split. If GUI mode is active, a specially formatted line #GUI#splitting_before_timestamp <timestamp> is output as well. Lines prefixed with#GUI# are suitable for machine parsing, won’t be translated and are guaranteed not to change in format. Implements #3421.
  • MKVToolNix GUI: multiplexer: when dragging & dropping directories to the “attachments” tab, the files contained in those directories will be attached. Implements #3410.
  • MKVToolNix GUI: info tool: added information about the file (directory, size, modification timestamp) at the top of each tab. Implements #3407.

Bug fixes

  • mkvmerge: AV1 parser: fixed the parser completely aborting when parsing the OBU size field fails due to there not being enough data to parse. Instead the parser will remember the last known-good position & restart from there after more data is available. Fixes #3431.
  • mkvmerge: HDMV PGS subtitles: reverted the change that implemented a heuristic for detecting bogus timestamps & attempting to fix them. This was done to fix #3268. Unfortunately this affected valid subtitle files with intentional huge gaps in timestamps, e.g. forced subtitle tracks. The heuristic has simply been removed, fixing #3392.
  • mkvmerge: Matroska reader: fixed reading files with EBML Void elements before the Matroska Segment element.
  • mkvmerge: fixed reversed attachment selection: --attachments !4 would not copy any attachment instead of all attachments but the one with ID 4. Fixes #3427.
  • mkvextract: IETF BCP 47/RFC 5646 language tags: mkvextract will now use & prefer IETF BCP 47 track language elements if they’re present. Only affects the VobSub & USF subtitle extraction.
  • mkvpropedit, MKVToolNix GUI’s chapter & header editors: updated the list of deprecated Matroska elements. The applications will no longer try to write those elements, even if they’re found in the file to be modified. The programs will no longer abort with error messages such as assertion "false" failed. Fixes #3416.
  • mkvpropedit, MKVToolNix GUI’s chapter & header editors: when the Matroska version numbers stored in the EBML Head element are updated, the updated EBML Head element might be smaller than the existing one. In that case the programs used to shrink the EBML Head & write a small EBML Void element between the updated EBML Head & the following element, usually a Matroska Segment element. This isn’t widely supported by programs including MKVToolNix itself, causing them to declare such files as invalid. The programs will now create the EBML Void element inside the EBML Head element, making them a level 1 element instead of a level 0 element. Fixes #3355.
  • mkvpropedit, MKVToolNix GUI’s chapter & header editors: often the programs have to relocate the Master elements in which the modifications were done. In that case the Seek Head elements must also be updated to reflect to the Master elements’ new positions. If a file contained a Seek Head element at the start already and if that Seek Head was too small to contain the updated positions, the programs would end up in an endless loop trying to write data to the end, creating ever-growing files. This is now handled properly by voiding this too-small Seek Head & finding a proper space for a new one instead. Fixes #3338.
  • MKVToolNix GUI: header editor: fixed pixelated icons on higher display scaling values. Fixes #3420.

Have fun 😁

MKVToolNix v71.1 released

Heyo.

I’ve just released a new minor update, v71.1. It solely fixes the “configure” script not having the correct requirements wrt. to libEBML & libMatroska. It also fixes several issues in the XML DTDs, but those aren’t actually used in MKVToolNix itself. Functionality wise v71.1 is the same as v71.0. If you aren’t a Linux package maintainer, feel free to skip this release.

You can download the source code or one of the binaries. The Windows and macOS binaries as well as the Linux AppImage are available already. The other Linux binaries are stil being built and will be available over the course of the next couple of hours.

Here are the NEWS since the previous release:

Bug fixes

  • Chapters DTD: added the new edition & chapter elements from Matroska v5.
  • Chapters DTD: fixed EditionUID being mandatory. mkvmerge can auto-generate it if it’s missing from the XML file.
  • Chapters DTD: fixed the example-chapters-1.xml not validating against the DTD.
  • Chapters DTD: corrected the old name EditionManaged to EditionFlagOrdered.
  • Chapters DTD: fixed ChapLanguageIETF missing its element declaration & ChapterLanguage being required.
  • XML DTDs: fixed elements representing binary data not allowing the format attribute.

Build system changes

  • bug fix: configure accidentally tested for libEBML ≥ 2.0.0 & libMatroska ≥ 2.0.0, even though the actual requirements are 1.4.4 & 1.7.1 respectively.

Have fun 😁

MKVToolNix v71.0 released

Hey everyone!

MKVToolNix v71 is out, bringing some more maintenance, but fixes, partial support for the latest Matroska elements. Partial in the sense that the GUI’s chapter editor still needs to be updated for the new elements used for naming/titling editions (similar to how atoms can already be named). This will come in the next release.

A couple of notes for package maintainers: the latest releases of libEBML (v1.4.4) & libMatroska (v1.7.1) are required. They’re also bundled, as usual. Those two releases were made on the same day as this MKVToolNix release: 2022-10-08. They should restore ABI compatibility with libEBML v1.4.2 & libMatroska v1.6.3 respectively, which we accidentally broke with last week’s releases (the bad ones: libEBML v1.4.3 & libMatroska v1.7.0).

You can download the source code or one of the binaries. The Windows binaries as well as the Linux AppImage are available already. The macOS binaries & the other Linux binaries are stil being built and will be available over the course of the next couple of hours.

Here are the NEWS since the previous release:

New features and enhancements

  • mkvmerge, MKVToolNix GUI’s multiplexer: added support for the Emphasis audio track header element via the --audio-emphasis option. It is also read from Matroska files.
  • mkvmerge: the options --list-stereo-modes and --list-audio-emphasis have been added to list all support numerical & textual values for the --stereo-mode & --audio-emphasis options.
  • mkvpropedit, MKVToolNix GUI’s header editor: added support for the Emphasis audio track header element via the emphasis property.
  • MKVToolNix GUI: multiplexer: when the track properties are configured to be displayed in a tab widget below the files & tracks in the “Input” tab, the “Color information” tab will be laid out in two columns. This reduces the minimum height the whole tab widget requires. Implements #3401.
  • MKVToolNix GUI: greatly sped up closing the preferences dialog by only re-running certain initialization functions when their corresponding settings have changed. Also fixes issues such as #3406.
  • mkvinfo, MKVToolNix GUI’s info tool: added support for the following new elements from Matroska v5: Emphasis audio track header element; Chapter Skip Type; Edition Display; Edition String; Edition Language IETF.
  • mkvmerge, mkvpropedit: added support for the following new chapter elements from Matroska v5: Chapter Skip Type; Edition Display; Edition String; Edition Language IETF. The GUI’s chapter editor will receive support for them in the next release.

Bug fixes

  • mkvmerge: Ogg/OGM reader: mkvmerge will now correctly skip tracks of unsupported types as well as tracks for which all track header pages haven’t been found. Fixes #3394.

Build system changes

  • libEBML 1.4.4 & libMatroska 1.7.1 are now required.
  • The bundled libEBML & libMatroska libraries were updated to v1.4.4 & v1.7.1 respectively.
  • The bundled fmt library was updated to v9.1.0.
  • The bundled nlohmann-json library was updated to v3.11.2.

Have fun 😁