Category Archives: Programming

Speeding up MKVToolNix compilation speed with zapcc

Compiling MKVToolNix can take quite a bit of time. It’s a C++ application, it uses a lot of template code, and it doesn’t make use of the pimpl idiom as much as it could. For years I’ve been using the usual several techniques trying to keep the time down: parallel compilation and pre-compiled headers. However, it still takes quite a lot of time, and that’s a bother during development.[1]

That’s why I was instantly stoked when reading about an announcement earlier this weak: zapcc, a clang-based C/C++ compiler heavily tuned towards performance, is being open-sourced. Having more Open Source options in the compiler world is great, having someone working on speed is even better.

zapcc’s web site has some outrageous numbers, toting 40x speedup during re-compilation. That sure sounds like marketing numbers. So how much better is it for my use case, compiling MKVToolNix? Well, look at this:

chart of compilation time of different compilers with different options

This sure looks nice! Here are the actual numbers:

Compiler drake -j1 drake -j5
gcc 8.1.1, no precompiled headers 39:49 15:21
gcc 8.1.1, with precompiled headers 24:34 09:23
clang 6.0.0, no precompiled headers 29:11 12:27
clang 6.0.0, with precompiled headers 18:27 07:05
zapcc revision c2b11ba7 07:16 03:05

Now those numbers explore the whole range between “no help at all” (no pre-compiled headers, no parallelism during compilation, slowest compiler) and “bells and whistles” (maximum parallelism, fastest compiler). A realistic comparison is between my usual setup and the fastest zapcc variant. Those two numbers are the bold ones above: clang using pre-compiled headers with five running compilers in parallel vs. zapcc with five running compilers in parallel.[2] And this is quite remarkable: from seven minutes down to three, down to 43% of the original time. Yes, this does make quite a difference during development.

So if you’re looking for a way to speed up your C++ compilation time, take a look at zapcc. I’m just glad people focus on different aspects of compilers, and us users can profit from them thanks to the compilers being Open Source. A big thanks to all compiler developers!

All tests were done on my Arch Linux installation:

  • MKVToolNix revision 7008661ed951e79c9cc6b7dc167137e84bed8805
  • CFLAGS=-fno-omit-frame-pointer CXXFLAGS=-fno-omit-frame-pointer ./configure --enable-debug --enable-optimization
  • gcc & clang from Arch’s repositories
  • zapcc compiled from git
  • Intel i5-4690K (four real cores, no hyperthreading)
  • 32 GB RAM DDR-3 1600 MHz
  • Samsung EVO 850 SSD
  • no swap space
  • all compiled binaries pass my test suite
  1. [1]I’m also caching compilation results using ccache so that re-compiling is super fast, but that doesn’t help with initial compilation times or if something in one of the central header files that’s included all over the place changes.
  2. [2]zapcc doesn’t support pre-compiled headers, hence only one line for zapcc

MKVToolNix 5.6.0 released

Hey,

I’ve released v5.6.0. It fixes a couple of important issues with the new --split parts: functionality. Other bugs were fixed as well. A Polish translation of the programs has been added as well as a Spanish translation of mmg’s guide.

Here are the usual links: the home page, the source code and the Windows installer and 7zip archive.

All of the binaries that I provide myself are already available.

Here’s the full ChangeLog since release 5.5.0:

  • 2012-05-27 Moritz Bunkus <moritz@bunkus.org>
    • Released v5.6.0.
    • documentation: Added Spanish translation of mmg’s guide by Israel Lucas Torrijos (see AUTHORS).
  • 2012-05-20 Moritz Bunkus <moritz@bunkus.org>
    • mkvmerge: bug fix: SRT subtitle entries with colons as the decimal separator are accepted. Fix for issue 754.
  • 2012-05-13 Moritz Bunkus <moritz@bunkus.org>
    • mkvmerge: bug fix: XML tag files with <Simple> tags that only contained a name and nested <Simple> were wrongfully rejected as invalid. Fixes issue 752.
    • mkvmerge: enhancement: mkvmerge was optimizied to keep cluster timecodes strictly increasing in most situations.
  • 2012-04-24 Moritz Bunkus <moritz@bunkus.org>
    • all: Added a translation to Polish by Daniel (see AUTHORS).
  • 2012-04-16 Moritz Bunkus <moritz@bunkus.org>
    • mkvextract: bug fix: Extraction of AVC/h.264 was completely broken after 2012-04-09 resulting in files with a length of 0 bytes.
  • 2012-04-09 Moritz Bunkus <moritz@bunkus.org>
    • mmg: new feature: When adding a Matroska file that has either the "previous segment UID" or the "next segment UID" set then mmg will copy those two and the source file’s segment UID into the corresponding controls on the "globla" tab if they haven’t been set before. Implements ticket 733.
    • mkvmerge: new feature: The verbose identification mode for Matroska files will now includes the "segment UID", the "next segment UID" and "previous segment UID" elements.
    • mkvmerge: enhancement: In "–split parts:" mode mkvmerge will use the output file name as it is instead of adding a running number to it if all the ranges to be kept are to be written into a single output file. Implements ticket 743.
    • mkvextract: bug fix: mkvextract will no longer abort extracing h.264 tracks if it encounters a NAL smaller than its size field. Instead it will warn about it and drop the NAL.
  • 2012-04-08 Moritz Bunkus <moritz@bunkus.org>
    • mkvmerge: bug fix: Writing more than two parts into the same file with "–split parts:" resulted in the timecodes of the third and all following parts to be wrong. Fixes ticket 740.
    • mkvmerge: bug fix: The "–split parts:" functionality was not taking dropped ranges into account when calculating the segment duration for files that more than one range was written to. Fixes ticket 738.
    • mkvmerge: bug fix: The "–split parts:" functionality was producing a small gap between the first part’s last packet’s timecode and the second part’s first packet’s timecode if two parts are written to the same file. Fixes ticket 742.
  • 2012-04-07 Moritz Bunkus <moritz@bunkus.org>
    • mkvmerge: bug fix: The "–split parts:" functionality was writing a superfluous and empty first part if the first range starts at 00:00:00. Fixes ticket 737.
  • 2012-04-07 Moritz Bunkus <moritz@bunkus.org>
    • mmg, build system: Fixed building with wxWidgets 2.9.3.

MKVToolNix v5.0.1 released

Hey,

I’ve released mkvtoolnix v5.0.1. It’s a release with a few improvements/bug fixes and one important fix for a regression introduced in v5.0.0 regarding PGS subtitles.

There were no changes that concern package maintainers.

Here are the usual links: the home page, the source code and the Windows installer and 7zip archive.

All of the binaries that I provide myself are already available.

Here’s the full ChangeLog since release 5.0.0:

  • 2011-10-09 Moritz Bunkus <moritz@bunkus.org>
    • Released v5.0.1.
  • 2011-10-08 Moritz Bunkus <moritz@bunkus.org>
    • build system: Updated the Debian/Ubuntu files to debhelper v7/quilt 3.0 format.
    • mkvmerge: enhancement: Implemented support for yet another way of storing EAC3 and DTS in MPEG transport streams.
  • 2011-10-05 Moritz Bunkus <moritz@bunkus.org>
    • mkvinfo: bug fix: Track information was not reset when opening more than one file in the GUI.
  • 2011-10-03 Moritz Bunkus <moritz@bunkus.org>
    • mkvmerge: bug fix: The PGS subtitle output module was not outputting any packet in certain cases due to uninitialized variables.
  • 2011-09-27 Moritz Bunkus <moritz@bunkus.org>
    • mkvmerge: bug fix: Fixed mkvmerge not finding any track in TS streams whose first PMT packet could not be parsed (e.g. invalid CRC).
    • mkvmerge: bug fix: Fixed detection of TS streams that only contain one PAT or PMT packet within the first few KB but no others within the first 10 MB.

Have fun.

MKVToolNix v5.0.0 released

Hey,

I’ve released mkvtoolnix v5.0.0. It’s a release with a lot of small bug fixes, but it also features support for MPEG transport streams.

Change for package maintainers: Building against external versions of libEBML and libMatroska is possible again. libEBML 1.2.2 and libMatroska 1.3.0 are required. If they’re not found or too old then the internal versions will be used and linked statically.

Here are the usual links: the home page, the source code and the Windows installer and 7zip archive.

All of the binaries that I provide myself are already available.

Here’s the full ChangeLog since release 4.9.1:

  • 2011-09-24 Moritz Bunkus <moritz@bunkus.org>
    • Released v5.0.0.
    • build system: libEBML 1.2.2 and libMatroska 1.3.0 are required for building. If external versions are not found or if they’re too old then the included versions will be used as a fallback.
  • 2011-09-21 Moritz Bunkus <moritz@bunkus.org>
    • mkvmerge: bug fix: The "writing application" element will not be localized but always be written in English.
  • 2011-09-20 Moritz Bunkus <moritz@bunkus.org>
    • mkvmerge: new feature: MPEG TS: mkvmerge will extract the track languages from a corresponding clpi (clip info) file. That file is searched for in the same directory and in ../CLIPINF and must have the same base name but with the ".clpi" extension.
    • mkvmerge: enhancement: Added new stereo mode options to match the current specs. The new options are "anaglyph_green_magenta" (12), "both_eyes_laced_left_first" (13) and "both_eyes_laced_right_first" (14).
    • mkvmerge: The –stereo-mode named option "anaglyph" was renamed to "anaglyph_cyan_red" to match the specs. The numerical value (10) remains unchanged.
  • 2011-09-18 Moritz Bunkus <moritz@bunkus.org>
    • mkvextract: bug fix: Fixed attachment number displayed during extraction. Fix for bug 663.
    • mkvmerge: enhancement: MPEG TS: Added support for HDMV PGS subtitles.
    • mkvmerge: enhancement: MPEG TS: Added support for DTS HD Master Audio tracks.
  • 2011-09-17 Moritz Bunkus <moritz@bunkus.org>
    • mkvmerge: enhancement: MPEG TS: Streams that are mentioned in the PMT but do not actually contain data are neither reported during identification nor muxed.
  • 2011-09-14 Moritz Bunkus <moritz@bunkus.org>
    • mkvmerge: new feature: MPEG TS: Added support for reading the language code.
  • 2011-09-13 Moritz Bunkus <moritz@bunkus.org>
    • mmg: enhancement: Added MPEG transport streams to the "add file" dialog file selector.
    • mkvmerge: new feature: MPEG TS: Added support for normal DTS tracks.
    • mkvmerge: Tons of fixes and additions to the MPEG transport stream demuxer.
  • 2011-09-10 Moritz Bunkus <moritz@bunkus.org>
    • build system: configure will accept external versions of libEBML and libMatroska again. Minimum required versions are libEBML 1.2.1 and libMatroska 1.1.0.
  • 2011-09-07 DenB <denb10@free.fr>
    • All: Updated the French translation with a complete set by DenB (see AUTHORS).
  • 2011-09-05 Cosme Domínguez <cosme.ddiaz@gmail.com>
    • mmg: mmg respects the XDG Base Directory Specification regarding its configuration files (environment variable $XDG_CONFIG_HOME/mkvtoolnix if set, otherwise ~/.config/mkvtoolnix).
  • 2011-08-24 Moritz Bunkus <moritz@bunkus.org>
    • all: Added an Lithuanian translation by Mindaugas Baranauskas (see AUTHORS).
  • 2011-08-14 Massimo Callegari <massimocallegari@yahoo.it>
    • mkvmerge: new feature: Implemented a MPEG transport stream demuxer.
  • 2011-08-02 Moritz Bunkus <moritz@bunkus.org>
    • mkvmerge: enhancement: When looking for MPEG files with the same base name as a source file mkvmerge will be stricter what it accepts. The file name must consist of at least one char followed by "-" or "_" followed by a number. That will match VTS_01_1.VOB but not e.g. "some_series_s03e10.mpg".
  • 2011-07-31 Moritz Bunkus <moritz@bunkus.org>
    • mkvmerge: bug fix: Opening MPEG files with numbers in their name from folders with e.g. Cyrillic names failed on Windows.
    • mkvmerge: bug fix: Several elements are not written when creating WebM compliant files. In the segment headers: SegmentUID, SegmentFamily, ChapterTranslate, PreviousSegmentUID, NextSegmentUID. In the track headers: MinCache, MaxCache and MaxBlockAdditionID.
  • 2011-07-19 Moritz Bunkus <moritz@bunkus.org>
    • mkvmerge: enhancement: Sped up file identification by caching read operations.
    • mkvmerge: bug fix: Fixed identifying QuickTime/MP4 files that start with a ‘skip’ atom.
  • 2011-07-13 Moritz Bunkus <moritz@bunkus.org>
    • mkvmerge: bug fix: Fixed a crash when reading AVI files with DTS audio tracks that do not contain valid headers in the first couple of packets. Fix for bug 646.

Have fun.