Category Archives: Uncategorized

Running Buildbot on IPv6

As I’ve struggled a bit here I thought I’d make this short write-up for others to find. The following writeup was done against Buildbot 3.6.1 on Ubuntu 22.04 & Arch Linux.

There are two well-known components to Buildbot: one master process (purposefully ignoring multi-master setup) & one or more workers. The master is the one providing the web interface, scheduling the work, keeping track of results & sending notifications. The workers do the work (well, duh!).

This means that we’re basically dealing with two different scenarios here:

  1. The master process must listen on IPv6 addresses for connections, both for the web interface & the workers which connect on a dedicated port.
  2. The workers must connect via IPv6 to the master.

Using the default configuration or the examples from the website neither of the two components use IPv6, for different reasons.

Both use the twistd Python framework for connection handling.

Making the master listen on IPv6

The configuration for the master is done solely via master.cfg, not in buildbot.tac.

Basically for every connection the master listens on a ConnectionString is used which contains both the port number & the address to listen on, e.g. "tcp:8010:interface=127.0.0.1" One might attempt to simply replace the IPv4 address 127.0.0.1 with an IPv6 address, but that doesn’t work as colons are used to separate key/value pairs within the ConnectionString. Therefore the colons must be escaped. Note that the syntax often used in other programs such as web browser to escape IPv6 addresses, placing them in square brackets such as [::], doesn’t work here either.

How are those colons escaped, though? With backslashes. However, these are Python strings, and Python uses backslashes for escaping in strings as well. Therefore in the source code we have to use two backslashes per colon. The resulting connection string might look like this: "tcp:8010:interface=\\:\\:" (which means “listen on all IPv6 & IPv4 addresses, regardless of the interface, on port 8010”).

Web interface handler

For the web interface listener, which is usually referred to as c["www"] (with c being the an instance of BuildmasterConfig), a dict is used. The port key can be a port number, or a full-blown connection string. Example:

c['www'] = {
    'port': "tcp:8010:interface=\\:\\:",
    'plugins': { 'waterfall_view': {}, 'console_view': {}, 'grid_view': {} },
    'auth': util.UserPasswordAuth({'user': 'supersecret'}),
    'authz': authz,
    'change_hook_auth': [strcred.makeChecker("file:changehook.passwd")],
    'change_hook_dialects': { 'gitlab': { 'secret': 'reallysecret'  } }
}

As hinted in the example the web interface handles both requests from us humans as well as change hooks from version control systems.

Worker handler

The worker handler is configured via c["protocols"]["pb"], and again a dict with a key called port is used which can be set to a connection string. For example:

c['protocols'] = {'pb': {'port': "tcp:interface=\\:\\::port=9989"}}

Making the workers connect via IPv6

The configuration is done in buildbot.tac.

The worker creates an instance of the Worker class. The first two parameters are the host address & the port number of the worker handler on the master’s side. Unlike the master, the parameters cannot seem to be ConnectionStrings.

Furthermore, the twistd client framework used here cannot resolve DNS AAAA records at all. If you provide a host name instead of an IPv6 or IPv4 address as the first argument, the library will only try to resolve it to an A record. For AAAA-only DNS records this means that name resolution fails, and the worker will not start. For DNS entries with both AAAA and A records, only the A record will be used, and the connection will be attempted via IPv4.

Luckily you can use an IPv6 address directly as the first argument. As there are no ConnectionStrings involved, colons can be left as-is. For example (all following arguments left out as they aren’t important):

s = Worker('2001:db8:1234::2', 9989, …)

Making the master connect via IPv6

We have to go back to the master once more as it can also make certain outbound connections that you might want to run via IPv6. These include but aren’t limited to:

  • email notifications
  • version control system status updates (Gitlab, Github etc.)

Email notifications

The email notifier also uses a pair of parameters specifying the SMTP server’s hostname & its port. Unfortunately it uses the same twistd client components that the worker uses, too, with the same DNS resolution issues. Sure, you can specify an IPv6 address directly instead of a host name, but that’ll be problematic if you want to enable TLS as certificate validation will now fail. I don’t have a good solution for IPv6-only setups in this case.

An example without authentication & without TLS:

mailer = reporters.MailNotifier(
    fromaddr="buildbot-master@example.com",
    sendToInterestedUsers=False,
    extraRecipients=["buildbot-failure-recipient@example.com"],
    relayhost='2001:db8:fefe::9876',
    generators=[generator_failing])
c["services"].append(mailer)

Version control status handlers

Handlers that signal the build status to version control systems such as Gitlab & Github don’t use the twistd framework for communication, it seems. They probably use something like urllib2. For their configuration they actually expect a base URL such as https://gitlab.com, and luckily for us, if that address resolves to an AAAA record then the status is indeed sent via IPv6. Nothing for us to do here.

Other considerations

Last but not least: the build instructions you create must also take IPv6 connectivity into account. If you configure git repositories on IPv4-only websites such as Github (shame!), you won’t be able to use them in an IPv6-only environment. Dual-stack setups will be fine, though.

On the other hand: external tools such as git that do not use the same Python libraries will resolve DNS entries to AAAA & A records & prefer AAAA = IPv6 over A = IPv4 — meaning configuring a git repo on sites such as Gitlab will indeed be retrieved via IPv6.

MKVToolNix v51.0.0 released

Hello everyone,

I’ve just pushed out MKVToolNix v51. As expected the new IETF BCP 47 language tag functionality introduced in v50 had a couple of bugs, which I’ve fixed in v51. Apart from that I’ve also started fuzzying mkvmerge, which turned up several issues that have also been fixed. Fuzzying will continue in the future and I fully expect a lot more bugs to be unearthed by it.

Important news for package managers is that `configure` can now detect and use a system-wide installed copy of the JPCRE2 header library. If it isn’t found, the bundled version will be used (just like in v50).

You can download the source code or one of the binaries. The Windows and macOS binaries as well as the Linux AppImage are available already. The other Linux binaries are stil being built and will be available over the course of the next couple of hours.

Here are the NEWS since the previous release:

New features and enhancements

  • mkvmerge, mkvpropedit, MKVToolNix GUI: IETF BCP 47 language tags: added
    missing support for UN M.49 regions for which there are no corresponding ISO
    3166 alpha codes. This enables language tags such as es-419 = Spanish in
    Latin America and the Caribbean. Implements #2919.
  • MKVToolNix GUI: IETF BCP 47 language tags: the text in the widget displaying
    the current language & the corresponding edit button is now displayed like a
    link (depending on the theme: different color & underlined), making it
    clearer that not just the edit button can be clicked but the text, too.

Bug fixes

  • mkvmerge: AV1 parser: fixed mkvmerge crashing after uncaught exceptions due
    to certain data conditions. Found by fuzzying.
  • mkvmerge: AV1 packetizer: the duration of frames wasn’t set properly. When
    appending AV1 IVF or OBU files this meant that the last frame of the Nth
    file and the first frame of file N+1 had the same timestamp. Fixes #2937.
  • mkvmerge: DTS reader: fixed handling of buffers with an odd length when
    byte-swapped DTS is detected so that mkvmerge doesn’t abort with a failed
    assertion. Found by fuzzying.
  • mkvmerge: h.264/AVC and h.265/HEVC elementary stream readers: mkvmerge will
    no longer claim to recognize data that looks like AVC or HEVC but with
    invalid video width/height values as that lead to failed assertions in
    libEBML later. Found by fuzzying.
  • mkvmerge: h.265/HEVC parser: fixed mkvmerge aborting after uncaught
    exceptions due to certain data constellations found by fuzzying.
  • mkvmerge: IVF reader: fixed mkvmerge crashing with a "division by zero"
    error due to certain data conditions. Found by fuzzying.
  • mkvmerge: fixed mkvmerge aborting with a "division by zero" error due to the
    timestamp scaling factor being 0 after handling certain data constellations
    found by fuzzying.
  • mkvmerge: track statistics: fixed a "division by zero" error when the
    content’s shorter than 1ms. Found by fuzzying.
  • mkvmerge, mkvinfo, mkvextract, mkvpropedit: Matroska access class: fixed an
    invalid memory access under certain data conditions. Found by fuzzying.
  • MKVToolNix GIU: IETF BCP 47/RFC 5646 language tags: the GUI will no longer
    open the language dialog when clicking on a disabled language display
    widget.
  • MKVToolNix GUI: IETF BCP 47 language tags: the "variants" combo-boxes were
    not populated even when the language tag was valid and contained at a
    variant. Fixes #2923.
  • MKVToolNix GUI: IETF BCP 47 language tags: when no language is selected, at
    least one of the other components (extended subtags, region, or variants)
    has something selected and "private use" is not empty, the GUI would claim
    this to be a valid tag, which it isn’t. Fixes #2924.
  • MKVToolNix GUI: multiplexer: when the tracks/chapters/tags selection changes
    from "at least one entry selected" to "no entry selected", the input
    controls will be reset to their default state and not just disabled. Fixes
    #2927.

Build system changes

  • configure now checks for the presence of the a system-wide installed copy
    of the JPCRE2 C++ wrapper library for the PCRE2 library and uses that if
    it’s new enough (at least v10.32.1). If not, the bundled version will be
    used as a fallback. Implements #2929.

Have fun!

Debian/Ubuntu APT repository changes

In the upcoming release of Ubuntu 18.04 APT repositories without signed “Release” files aren’t supported out of the box anymore. I’ve therefore changed my Debian & Ubuntu APT repositories to a new layout that includes proper “Release” files. This also means that you have to update your APT repository definition.

Here’s what such a change would look like:

Before:

deb https://mkvtoolnix.download/ubuntu/artful/ ./
deb-src https://mkvtoolnix.download/ubuntu/artful/ ./

After:

deb https://mkvtoolnix.download/ubuntu/ artful main
deb-src https://mkvtoolnix.download/ubuntu/ artful main

Head over to the downloads page where you can copy & pate the appropriate entries from.

MKVToolNix v18.0.0 released

Welcome to release v18.0.0 of MKVToolNix. This is just a smallish bug fix release which also contains a couple of performance improvements.

There were no changes for package maintainers.

You can download the source code or one of the binaries. The Windows and macOS binaries are available already. The Linux binaries are stil being built and will be available of the course of the next couple of hours.

Here are the NEWS since the previous release:

New features and enhancements

  • build system: when building with clang v3.8.0 or newer, configure will no longer restrict optimization flags to -O1 and use -O3 again (older versions of clang suffered from excessive memory usage with higher optimization levels).
  • build system: when building with mingw 7.2.0 or newer, configure will no longer restrict optimization flags to -O2 and use -O3 again (older versions of mingw suffered from bugs such as segmentation faults with higher optimization levels).
  • build system: stack protection is enabled when building with clang 3.5.0 or newer on all platforms.
  • mkvmerge: AVC & HEVC ES parsers: performance improvements by copying much less memory around.
  • mkvmerge: tags: reintroduced a workaround for non-compliant files with tags that do not contain the mandatory SimpleTag element. This workaround was removed during code refactoring in release v15.0.0.
  • GUI: multiplexer: the "AAC is SBR/HE-AAC/AAC+" checkbox in the "audio properties" section will be disabled if the functionality is not implemented for the selected track’s codec & container.
  • GUI: multiplexer: the "reduce to core" checkbox in the "audio properties" section will be disabled if the functionality is not implemented for the selected track’s codec. See #2134.

Bug fixes

  • mkvmerge: AAC ADTS parser: fixed interpretation of the channel_configuration header element for ADTS files that do not contain a program configuration element: value 7 means 7.1 channels. Fixes #2151.
  • mkvmerge: Matroska identification: the date_local and date_utc attributes will only be output if the identified Matroska file actually contains the "date" header field.
  • mkvmerge: WebVTT: mkvmerge did not recognize timestamp lines if the hours components were absent. Fixes #2139.
  • mkvpropedit, GUI’s header editor: the date header field won’t be added automatically anymore whenever the segment info section is edited and the date element is either deleted or not present in the first place. Fixes #2143.

Have fun :)