Creating files with long names/paths
Windows
The Windows APIs that take file/path name arguments have always had a built-in length limit of 260 characters. Therefore long paths are problematic. Applications cannot simply circumvent that limit. However, there are two ways how users can circumvent it.
Method 1: enabling long paths in Windows 10, version 1607 or later
With Windows 10, version 1607, Microsoft has changed all APIs receiving filen ames and removed that limitation of 260 characters. However, in order to be backwards compatible that feature must be turned on in the registry. This article describes how to enable it both via editing the registry directly and how to create a group policy for the same effect.
In addition to that change the applications must signal that they support long path names. MKVToolNix does so starting with version 50.0.0.
After enabling the aforementioned registry key you can use long, regular path names:
mkvmerge.exe -o D:\very\long\path\and\filename.mkv C:\even\longer\name\PLAYLIST\00001.mpls
Method 2: using the \\?\ prefix
The APIs receiving file name arguments will accept long path names if the file name argument starts with the prefix \\?\. Here's how that would look:
mkvmerge.exe -o \\?\D:\very\long\path\and\filename.mkv \\?\C:\even\longer\name\PLAYLIST\00001.mpls
The advantage of this method is that it works on both older Windows versions as well as on Windows 10 versions that don't have long paths enabled in the registry (see method 1).
Linux, BSD, macOS etc.
For most file systems Linux has a maximum limit of 255 characters for file names and 4096 characters for the whole path.
FreeBSD and macOS seem to have a limit of 1023 characters. Other BSDs seem to have similar limits.
MKVToolNix
MKVToolNix itself is agnostic to path name lengths, meaning it can deal with arbitrarily long names. It's just the OS-level APIs that impose limits.