Speeding up MKVToolNix compilation speed with zapcc

Compiling MKVToolNix can take quite a bit of time. It’s a C++ application, it uses a lot of template code, and it doesn’t make use of the pimpl idiom as much as it could. For years I’ve been using the usual several techniques trying to keep the time down: parallel compilation and pre-compiled headers. However, it still takes quite a lot of time, and that’s a bother during development.[1]

That’s why I was instantly stoked when reading about an announcement earlier this weak: zapcc, a clang-based C/C++ compiler heavily tuned towards performance, is being open-sourced. Having more Open Source options in the compiler world is great, having someone working on speed is even better.

zapcc’s web site has some outrageous numbers, toting 40x speedup during re-compilation. That sure sounds like marketing numbers. So how much better is it for my use case, compiling MKVToolNix? Well, look at this:

chart of compilation time of different compilers with different options

This sure looks nice! Here are the actual numbers:

Compiler drake -j1 drake -j5
gcc 8.1.1, no precompiled headers 39:49 15:21
gcc 8.1.1, with precompiled headers 24:34 09:23
clang 6.0.0, no precompiled headers 29:11 12:27
clang 6.0.0, with precompiled headers 18:27 07:05
zapcc revision c2b11ba7 07:16 03:05

Now those numbers explore the whole range between “no help at all” (no pre-compiled headers, no parallelism during compilation, slowest compiler) and “bells and whistles” (maximum parallelism, fastest compiler). A realistic comparison is between my usual setup and the fastest zapcc variant. Those two numbers are the bold ones above: clang using pre-compiled headers with five running compilers in parallel vs. zapcc with five running compilers in parallel.[2] And this is quite remarkable: from seven minutes down to three, down to 43% of the original time. Yes, this does make quite a difference during development.

So if you’re looking for a way to speed up your C++ compilation time, take a look at zapcc. I’m just glad people focus on different aspects of compilers, and us users can profit from them thanks to the compilers being Open Source. A big thanks to all compiler developers!

All tests were done on my Arch Linux installation:

  • MKVToolNix revision 7008661ed951e79c9cc6b7dc167137e84bed8805
  • CFLAGS=-fno-omit-frame-pointer CXXFLAGS=-fno-omit-frame-pointer ./configure --enable-debug --enable-optimization
  • gcc & clang from Arch’s repositories
  • zapcc compiled from git
  • Intel i5-4690K (four real cores, no hyperthreading)
  • 32 GB RAM DDR-3 1600 MHz
  • Samsung EVO 850 SSD
  • no swap space
  • all compiled binaries pass my test suite
  1. [1]I’m also caching compilation results using ccache so that re-compiling is super fast, but that doesn’t help with initial compilation times or if something in one of the central header files that’s included all over the place changes.
  2. [2]zapcc doesn’t support pre-compiled headers, hence only one line for zapcc

MKVToolNix v24.0.0 released

Hey.

time for MKVToolNix v24.0.0, a release that’s on the smaller side regarding the number of changes: nothing major, a number of smaller bugs squashed, a couple of enhancements.

AV1 support hasn’t changed as the bitstream format still hasn’t been finalized.

To my users on Debian and Ubuntu: just a friendly reminder that the layout of my APT repositories was changed in April. If you haven’t done so, you’ll have to update your sources.list entry accordingly. Read more about that in this blog post.

You can download the source code or one of the binaries. The Windows and macOS binaries are available already. The Linux binaries are stil being built and will be available of the course of the next couple of hours.

Here are the NEWS since the previous release:

New features and enhancements

  • mkvmerge: MP4 reader: improved the detection of edit lists consisting of two
    identical entries, each spanning the file’s duration as given in the movie
    header atom. The second entry is ignored in such cases. See #2306.
  • mkvmerge: JSON identification: the "display unit" video track property is
    now reported as display_unit. The JSON schema has been bumped to v11 for
    this change.
  • mkvmerge, mkvextract: AVC/h.264: empty NALUs will now be removed.
  • mkvextract: VobSub extraction: empty SPU packets will now be dropped during
    extraction as other tools such as MP4Box cannot handle them
    correctly. Implements #2293.

Bug fixes

  • mkvmerge: E-AC-3 parser: fixed determining the number of channels for
    streams that contain an AC-3 core with dependent E-AC-3 frames. Fixes #2283.
  • mkvmerge: Matroska reader: fixed mkvmerge buffering the whole file if a
    video track is multiplexed that consists of only one or a few frames. Fixes
    #2304.
  • mkvmerge: the "display unit" video track property will now be kept if it is
    set in the source file. Fixes #2317.
  • MKVToolNix GUI: multiplexer: when scanning playlists, all playlists were
    offered for selection regardless of the value of the "minimum playlist
    duration" setting. Fixes #2299.
  • MKVToolNix GUI: multiplexer: deriving track languages from file names: the
    regular sub-expressions for ISO 639-1 codes could match on empty strings,
    too, causing matches in wrong places and hence no language being recognized
    in certain situations. Fixes #2298.
  • MKVToolNix GUI: header editor: fixed a crash when saving the file fails
    (e.g. because it isn’t writable). Fixes #2319.
  • MKVToolNix GUI: header editor: the editor was wrongfully claiming that
    mandatory elements with default values cannot be removed in the "status"
    text. Fixes #2320.
  • MKVToolNix GUI: preferences: on macOS & Linux the setting "enable copying
    tracks by their type" wasn’t restored on program start. Fixes #2297.

Other changes

  • Niels Lohmann’s JSON library: the bundled version has been updated from
    v1.1.0 (git revision 54d3cab) to v3.1.1 (git revision g183390c1).
  • pugixml library: the bundled version has been updated from v1.8 to v1.9 (git
    revision e584ea3).

Have fun :)

MKVToolNix v23.0.0 released

Hey.

MKVToolNix v23.0.0 is out containing a moderate number of enhancements and bug fixes.

About AV1 support: even though the previous release was the first to support AV1, keep in mind that neither the bitstream format nor the method of storage in MP4, WebM and Matroska has been finalized yet. I’ve therefore decided to disable AV1 support by default. You have to enable it manually by passing --engage enable_av1 to mkvmerge. mkvmerge can still identify AV1 without that option, but it’ll refuse to multiplex it.

Note further that only supported bitstream format is the one that was active on 2018-05-02.

To my users on Debian and Ubuntu: the layout of my APT repositories was changed in April. You’ll have to update your sources.list entry accordingly. Read more about that in this blog post.

You can download the source code or one of the binaries. The Windows and macOS binaries are available already. The Linux binaries are stil being built and will be available of the course of the next couple of hours.

Here are the NEWS since the previous release:

New features and enhancements

  • mkvmerge: input: format detection uses file-extension to improve performance
    and to give preference when several formats match.
  • mkvmerge: AV1: added support for reading AV1 video from Open Bitstream Unit
    files.
  • mkvmerge: AV1: adjusted the code for the AV1 bitstream format changes made
    up to 2018-05-02 (git revision d14e878).
  • mkvmerge: MP4 reader: if a track has an edit list with two identical
    entries, each spanning the file’s duration as given in the movie header
    atom, then the second entry will now be ignored. Improves the handling of
    files with bogus data; see #2196 and #2270.
  • MKVToolNix GUI: multiplexer: added options to only enable tracks of certain
    types by default. Implements #2271.
  • MKVToolNix GUI: multiplexer: added an option to enable dialog normalization
    gain removal by default for all audio tracks for which the operation is
    supported. Implements #2272.
  • MKVToolNix GUI: multiplexer: when deriving track languages from the file
    names is active and the file name contains the usual season/episode pattern
    (e.g. "S02E14"), then only the part after the season/episode pattern will be
    used for detecting the language. Part of the improvements for #2267.
  • MKVToolNix GUI: multiplexer: the regular expression used for deriving track
    languages from the file names can now be customized in the preferences. Part
    of the improvements for #2267.
  • MKVToolNix GUI: multiplexer: the user can now customize the list of track
    languages the GUI recognizes in file names. This list defaults to a handful
    of common languages instead of the full list of supported languages. Part of
    the improvements for #2267.

Bug fixes

  • mkvmerge: MP3 packetizer: removed a memory leak growing linearly with the
    track’s size.
  • mkvmerge: VobSub packetizer: whenever a VobSub packet doesn’t contain a
    duration on the container level, mkvmerge will now set it from the duration
    in the SPU packets. Before it was accidentally setting the SPU-level
    duration to 0 instead. Fixes #2260.
  • mkvmerge: track statistics tags: if writing the Date element is
    deactivated via --no-date, the _STATISTICS_WRITING_DATE_UTC isn’t
    written either anymore. Fixes #2286.
  • mkvmerge, mkvextract, mkvpropedit: removed several small, constant-size
    memory leaks.
  • mkvextract: fixed a crash when mkvextract with a non-Matroska file as the
    source file. Fixes #2281.
  • MKVToolNix GUI: the central area is now scrollable, allowing the GUI to be
    resized to almost arbitrary sizes. Fixes #2265.
  • MKVToolNix GUI: multiplexer: the "copy file title to destination file name"
    functionality will now replace everything in the destination file name up to
    the last period instead of only up to the first period. Fixes #2276.

Build system changes

  • build system: MKVToolNix now requires a compiler that supports the following
    features of the C++14 standard: "user-defined literals for
    std::string". For the GNU Compiler Collection (gcc) this means v5.x or
    newer; for clang it means v3.4 or newer.
  • Windows: linking against and installing shared version of the libraries with
    MXE is now supported by setting configure‘s host triplet accordingly,
    e.g. --host=x86_64-w64-mingw32.shared.

Other changes

  • mkvmerge: AV1: support for AV1 must be activated manually by adding
    --engage enable_av1 as the AV1 bitstream specification hasn’t been
    finalized yet.

Have fun :)

Debian/Ubuntu APT repository changes

In the upcoming release of Ubuntu 18.04 APT repositories without signed “Release” files aren’t supported out of the box anymore. I’ve therefore changed my Debian & Ubuntu APT repositories to a new layout that includes proper “Release” files. This also means that you have to update your APT repository definition.

Here’s what such a change would look like:

Before:

deb https://mkvtoolnix.download/ubuntu/artful/ ./
deb-src https://mkvtoolnix.download/ubuntu/artful/ ./

After:

deb https://mkvtoolnix.download/ubuntu/ artful main
deb-src https://mkvtoolnix.download/ubuntu/ artful main

Head over to the downloads page where you can copy & pate the appropriate entries from.