Author Archives: mosu

MKVToolNix v25.0.0

Hello everyone,

July’s release of MKVToolNix is here: v25.0.0. After two smaller ones, this one packs quite a number of bug fixes and enhancements, including fixing a regression in the GUI’s header editor preventing elements from being removed that was introduced in v24.0.0. Sorry about that.

AV1 support hasn’t changed. While the bitstream format’s been finalized in the meantime, the mapping to Matroska & WebM hasn’t. It’s currently being discussed on the CELLAR mailing list.

You can download the source code or one of the binaries. The Windows and macOS binaries as well as the Linux AppImage are available already. The other Linux binaries are stil being built and will be available of the course of the next couple of hours.

Here are the NEWS since the previous release:

New features and enhancements

  • mkvmerge: SRT/ASS/SSA text subtitles: for files for which no encoding has
    been specified, mkvmerge will try UTF-8 first before falling back to the
    system’s default encoding. Part of the implementation of #2246.
  • mkvmerge: SRT/ASS/SSA/WebVTT text subtitles: a warning is now emitted if
    invalid 8-bit characters are encountered outside valid multi-byte UTF-8
    sequences. Part of the implementation of #2246.
  • mkvmerge: Matroska & MPEG transport stream readers: the encoding of text
    subtitles read from Matroska files can now be changed with the
    --sub-charset parameter.
  • Linux: starting with release 25 an AppImage will be provided which should
    run on any Linux distribution released around the time of CentOS 7/Ubuntu
    14.04 or later.
  • macOS: translations: updated the build.sh script to build libiconv and a
    complete gettext. Together with an additional fix to how translation files
    are located, MKVToolNix can now use all interface languages on macOS,
    too. Fixes #2110, #2307, #2323.

Bug fixes

  • mkvmerge: AVC/h.264: fixed file identification failing for certain
    elementary streams due to internal buffers not being cleared properly. Fixes
    #2325.
  • mkvmerge: HEVC/h.265: fixed file identification failing for certain
    elementary streams due to internal buffers not being cleared properly. This
    is the HEVC analog to what was fixed for AVC in #2325.
  • mkvmerge: MLP code: fixed various issues preventing MLP from being parsed
    correctly. Fixes #2326.
  • mkvmerge: TrueHD/MLP packetizer; dialog volume normalization removal isn’t
    attempted if the track is an MLP track as the operation is only supported
    for TrueHD, not MLP.
  • mkvmerge: MPEG TS reader: when reading MPLS mkvmerge will now compare the
    MPLS’s start and end timestamps against the transport stream’s PTS instead
    of its DTS. Otherwise the first key frame of a video track might be dropped
    if it isn’t the first in presentation order. Fixes #2321.
  • mkvmerge: JSON identification: mkvmerge will ensure that all strings passed
    to the JSON output modules are valid UTF-8 encoded strings by replacing
    invalid bytes with placeholder characters. This avoids the JSON library
    throwing an exception and mkvmerge aborting on such data. Fixes #2327.
  • mkvmerge: audio packetizers: mkvmerge will now keep discard padding values
    if they’re present for packets read from Matroska files. Fixes #2296.
  • mkvmerge: Ogg Opus reader: packet timestamps aren’t calculated by summing up
    the duration of all packets starting with timestamp 0 anymore. Instead the
    algorithm is based on the Ogg page’s granule position and which packet
    number is currently timestamped (special handling for the first and last
    packets in the stream).

    • This fixes the first timestamp if the first Ogg packet’s granule position
      is larger than the number of samples in the first packet (= if the first
      sample’s timestamp is bigger than 0). mkvmerge will keep those offsets now
      and inserts "discard padding" only where it’s actually needed.
    • It also improves handling of invalid files where the first Ogg packet’s
      granule position is smaller than the number of samples in the first packet
      (= the first sample’s timestamp is smaller than 0). mkvmerge will now
      shift all timestamps up to 0 in such a case instead of inserting "discard
      padding" elements all over the place.
    • mkvmerge will no longer insert "discard padding" elements if the
      difference between a) the calculated number of samples in the packet
      according to the granule position and b) the actual number of samples as
      calculated from the bitstream is one sample or less and if the packet
      isn’t the last one in the stream. This circumvents certain rounding
      errors.
    • The timestamp of the first packet after a gap in the middle of the stream
      is now calculated based on the Ogg page the packet belongs to, and not
      based on the timestamps before the gap.
      Fixes #2280.
  • mkvmerge: complete rewrite of the progress handling. It’s now based upon the
    total size of all source files and the current position within them instead
    of the number of frames/blocks to be processed. This simplifies calculation
    when appending files and fixes rare cases of when progress report was
    obvious wrong (e.g. stuck at 0% right until the end). Fixes #2150 and #2330.
  • MKVToolNix GUI: header editor: non-mandatory elements couldn’t be removed
    anymore due to a regression while fixing #2320. They can now be removed
    again. Fixes #2322.

Have fun :)

Speeding up MKVToolNix compilation speed with zapcc

Compiling MKVToolNix can take quite a bit of time. It’s a C++ application, it uses a lot of template code, and it doesn’t make use of the pimpl idiom as much as it could. For years I’ve been using the usual several techniques trying to keep the time down: parallel compilation and pre-compiled headers. However, it still takes quite a lot of time, and that’s a bother during development.[1]

That’s why I was instantly stoked when reading about an announcement earlier this weak: zapcc, a clang-based C/C++ compiler heavily tuned towards performance, is being open-sourced. Having more Open Source options in the compiler world is great, having someone working on speed is even better.

zapcc’s web site has some outrageous numbers, toting 40x speedup during re-compilation. That sure sounds like marketing numbers. So how much better is it for my use case, compiling MKVToolNix? Well, look at this:

chart of compilation time of different compilers with different options

This sure looks nice! Here are the actual numbers:

Compiler drake -j1 drake -j5
gcc 8.1.1, no precompiled headers 39:49 15:21
gcc 8.1.1, with precompiled headers 24:34 09:23
clang 6.0.0, no precompiled headers 29:11 12:27
clang 6.0.0, with precompiled headers 18:27 07:05
zapcc revision c2b11ba7 07:16 03:05

Now those numbers explore the whole range between “no help at all” (no pre-compiled headers, no parallelism during compilation, slowest compiler) and “bells and whistles” (maximum parallelism, fastest compiler). A realistic comparison is between my usual setup and the fastest zapcc variant. Those two numbers are the bold ones above: clang using pre-compiled headers with five running compilers in parallel vs. zapcc with five running compilers in parallel.[2] And this is quite remarkable: from seven minutes down to three, down to 43% of the original time. Yes, this does make quite a difference during development.

So if you’re looking for a way to speed up your C++ compilation time, take a look at zapcc. I’m just glad people focus on different aspects of compilers, and us users can profit from them thanks to the compilers being Open Source. A big thanks to all compiler developers!

All tests were done on my Arch Linux installation:

  • MKVToolNix revision 7008661ed951e79c9cc6b7dc167137e84bed8805
  • CFLAGS=-fno-omit-frame-pointer CXXFLAGS=-fno-omit-frame-pointer ./configure --enable-debug --enable-optimization
  • gcc & clang from Arch’s repositories
  • zapcc compiled from git
  • Intel i5-4690K (four real cores, no hyperthreading)
  • 32 GB RAM DDR-3 1600 MHz
  • Samsung EVO 850 SSD
  • no swap space
  • all compiled binaries pass my test suite
  1. [1]I’m also caching compilation results using ccache so that re-compiling is super fast, but that doesn’t help with initial compilation times or if something in one of the central header files that’s included all over the place changes.
  2. [2]zapcc doesn’t support pre-compiled headers, hence only one line for zapcc

MKVToolNix v24.0.0 released

Hey.

time for MKVToolNix v24.0.0, a release that’s on the smaller side regarding the number of changes: nothing major, a number of smaller bugs squashed, a couple of enhancements.

AV1 support hasn’t changed as the bitstream format still hasn’t been finalized.

To my users on Debian and Ubuntu: just a friendly reminder that the layout of my APT repositories was changed in April. If you haven’t done so, you’ll have to update your sources.list entry accordingly. Read more about that in this blog post.

You can download the source code or one of the binaries. The Windows and macOS binaries are available already. The Linux binaries are stil being built and will be available of the course of the next couple of hours.

Here are the NEWS since the previous release:

New features and enhancements

  • mkvmerge: MP4 reader: improved the detection of edit lists consisting of two
    identical entries, each spanning the file’s duration as given in the movie
    header atom. The second entry is ignored in such cases. See #2306.
  • mkvmerge: JSON identification: the "display unit" video track property is
    now reported as display_unit. The JSON schema has been bumped to v11 for
    this change.
  • mkvmerge, mkvextract: AVC/h.264: empty NALUs will now be removed.
  • mkvextract: VobSub extraction: empty SPU packets will now be dropped during
    extraction as other tools such as MP4Box cannot handle them
    correctly. Implements #2293.

Bug fixes

  • mkvmerge: E-AC-3 parser: fixed determining the number of channels for
    streams that contain an AC-3 core with dependent E-AC-3 frames. Fixes #2283.
  • mkvmerge: Matroska reader: fixed mkvmerge buffering the whole file if a
    video track is multiplexed that consists of only one or a few frames. Fixes
    #2304.
  • mkvmerge: the "display unit" video track property will now be kept if it is
    set in the source file. Fixes #2317.
  • MKVToolNix GUI: multiplexer: when scanning playlists, all playlists were
    offered for selection regardless of the value of the "minimum playlist
    duration" setting. Fixes #2299.
  • MKVToolNix GUI: multiplexer: deriving track languages from file names: the
    regular sub-expressions for ISO 639-1 codes could match on empty strings,
    too, causing matches in wrong places and hence no language being recognized
    in certain situations. Fixes #2298.
  • MKVToolNix GUI: header editor: fixed a crash when saving the file fails
    (e.g. because it isn’t writable). Fixes #2319.
  • MKVToolNix GUI: header editor: the editor was wrongfully claiming that
    mandatory elements with default values cannot be removed in the "status"
    text. Fixes #2320.
  • MKVToolNix GUI: preferences: on macOS & Linux the setting "enable copying
    tracks by their type" wasn’t restored on program start. Fixes #2297.

Other changes

  • Niels Lohmann’s JSON library: the bundled version has been updated from
    v1.1.0 (git revision 54d3cab) to v3.1.1 (git revision g183390c1).
  • pugixml library: the bundled version has been updated from v1.8 to v1.9 (git
    revision e584ea3).

Have fun :)

MKVToolNix v23.0.0 released

Hey.

MKVToolNix v23.0.0 is out containing a moderate number of enhancements and bug fixes.

About AV1 support: even though the previous release was the first to support AV1, keep in mind that neither the bitstream format nor the method of storage in MP4, WebM and Matroska has been finalized yet. I’ve therefore decided to disable AV1 support by default. You have to enable it manually by passing --engage enable_av1 to mkvmerge. mkvmerge can still identify AV1 without that option, but it’ll refuse to multiplex it.

Note further that only supported bitstream format is the one that was active on 2018-05-02.

To my users on Debian and Ubuntu: the layout of my APT repositories was changed in April. You’ll have to update your sources.list entry accordingly. Read more about that in this blog post.

You can download the source code or one of the binaries. The Windows and macOS binaries are available already. The Linux binaries are stil being built and will be available of the course of the next couple of hours.

Here are the NEWS since the previous release:

New features and enhancements

  • mkvmerge: input: format detection uses file-extension to improve performance
    and to give preference when several formats match.
  • mkvmerge: AV1: added support for reading AV1 video from Open Bitstream Unit
    files.
  • mkvmerge: AV1: adjusted the code for the AV1 bitstream format changes made
    up to 2018-05-02 (git revision d14e878).
  • mkvmerge: MP4 reader: if a track has an edit list with two identical
    entries, each spanning the file’s duration as given in the movie header
    atom, then the second entry will now be ignored. Improves the handling of
    files with bogus data; see #2196 and #2270.
  • MKVToolNix GUI: multiplexer: added options to only enable tracks of certain
    types by default. Implements #2271.
  • MKVToolNix GUI: multiplexer: added an option to enable dialog normalization
    gain removal by default for all audio tracks for which the operation is
    supported. Implements #2272.
  • MKVToolNix GUI: multiplexer: when deriving track languages from the file
    names is active and the file name contains the usual season/episode pattern
    (e.g. "S02E14"), then only the part after the season/episode pattern will be
    used for detecting the language. Part of the improvements for #2267.
  • MKVToolNix GUI: multiplexer: the regular expression used for deriving track
    languages from the file names can now be customized in the preferences. Part
    of the improvements for #2267.
  • MKVToolNix GUI: multiplexer: the user can now customize the list of track
    languages the GUI recognizes in file names. This list defaults to a handful
    of common languages instead of the full list of supported languages. Part of
    the improvements for #2267.

Bug fixes

  • mkvmerge: MP3 packetizer: removed a memory leak growing linearly with the
    track’s size.
  • mkvmerge: VobSub packetizer: whenever a VobSub packet doesn’t contain a
    duration on the container level, mkvmerge will now set it from the duration
    in the SPU packets. Before it was accidentally setting the SPU-level
    duration to 0 instead. Fixes #2260.
  • mkvmerge: track statistics tags: if writing the Date element is
    deactivated via --no-date, the _STATISTICS_WRITING_DATE_UTC isn’t
    written either anymore. Fixes #2286.
  • mkvmerge, mkvextract, mkvpropedit: removed several small, constant-size
    memory leaks.
  • mkvextract: fixed a crash when mkvextract with a non-Matroska file as the
    source file. Fixes #2281.
  • MKVToolNix GUI: the central area is now scrollable, allowing the GUI to be
    resized to almost arbitrary sizes. Fixes #2265.
  • MKVToolNix GUI: multiplexer: the "copy file title to destination file name"
    functionality will now replace everything in the destination file name up to
    the last period instead of only up to the first period. Fixes #2276.

Build system changes

  • build system: MKVToolNix now requires a compiler that supports the following
    features of the C++14 standard: "user-defined literals for
    std::string". For the GNU Compiler Collection (gcc) this means v5.x or
    newer; for clang it means v3.4 or newer.
  • Windows: linking against and installing shared version of the libraries with
    MXE is now supported by setting configure‘s host triplet accordingly,
    e.g. --host=x86_64-w64-mingw32.shared.

Other changes

  • mkvmerge: AV1: support for AV1 must be activated manually by adding
    --engage enable_av1 as the AV1 bitstream specification hasn’t been
    finalized yet.

Have fun :)