This page details the XFD decompression system, and the "slaves" I have written for XFD that decompress files packed with various compression methods.
You might also be interested in the DeliTracker XFD/XAD Decrunch genie, the PowerPacker encryption cracker or other formats which do not have a regular identifier and thus cannot have XFD slaves written for them. These are on my main compression page.
- Introduction to file crunchers and XFD
- AMOS Pac.Pic. banks
- CPic images
- Kenny Grant
- MMCMP
- Microsoft SZDD
- Resident De/Compressor
- SPacker v3
- Zulu & Grey
Introduction to file crunchers and XFD
download XFD
XFD home page
Ronald van Dijk's Amiga cruncher collection
For a long time, since the days of the C64, game and demo programmers have used compression, known by them as "crunching", with realtime decompressors used in their software, known as "decrunchers". This way, they can fit more on their disks and make bigger and better demos or games. Raster graphics take a lot of space and can be compressed very well. Also, it is fashionable to write small demos called "intros". These compete at demo parties with a given a size limitation - 64 kilobytes (65536 bytes or less), 40kb, 32kb or even 4kb. With an executable file compressor, it is possible to fit over 200kb of data in a 64kb file, thus allowing for more graphics, more effects and more sound within the limitations.
Technically, most Amiga compression formats are based on a mix of delta coding, run length encoding, Lempel-Ziv style dictionary compression and Huffman style entropy compression. These methods were the best known by regular programmers at the time.
Once programmers had developed compression and decompression routines for their own use, they often released their "cruncher" tools to the public, so all other programmers could use compression in their software. This is not the same as programmers releasing archivers, although often programmers released both crunchers and archivers. For example, Stefan Boberg released both the Defjam Packer and wrote the leading Amiga implementation of the LhA archiver. Archivers are designed for the regular computer user to compress and maintain their files. Crunchers are designed to compress executable programs that automatically decompress and run as normal. Crunchers can also pack individual data files.
As all these compression tools were released, rogue programmers often used them to hide viruses. A virus scanner would not see the virus code an executable if it was compressed, but the virus code would execute once unpacked. Georg Hörmann, an anti-virus programmer, thought it a good idea to collect all the decompression methods, so he could automatically decompress any crunched executable or data file, and then look for viruses. He created decrunch.library, which later became xfdmaster.library. The library is now maintained by Dirk Stöcker.
The XFD library operates by using what it calls "slaves". Each slave can decompress a particular type of file. The most common slaves are included directly in the XFD library itself, others are stored externally and are loaded when the library starts up. Anyone can write slaves for XFD. The central functions of the XFD library give a simple interface to use all the slaves at once to identify and decompress any kind of crunched file.
AMOS Pac.Pic. banks
download version 1.2 and source code
AMOS The Creator is an amazing Amiga version of the BASIC programming language, written by François Lionet. One of the many standard extensions to the language is the Picture Compactor. The Picture Compactor tries to compress AMOS graphics screens, and stores them in what AMOS calls "memory banks". This allows a programmer to store more images in memory or on disk than would be possible if the pictures were simply uncompressed.
The Compactor operates like so. First, the AMOS programmer draws something on an AMOS graphics screen. Then, the SPACK command is used to run the picture compactor. For example, the command SPACK 0 TO 12 compresses graphics screen 0 into memory bank 12. The compactor first tries to compress the graphics screen at each compression level without storing the result. The method yielding the smallest file size is chosen. The compactor then allocates memory for a memory bank and compresses the graphics screen once more, storing the result in the memory bank. The memory bank's name is always "Pac.Pic.". The bank can be saved to disk with a command like SAVE "picture.abk",12.
The format of Pac.Pic. uses run length compression, but in a strange way. First, the order of data in the screen is reordered for better compression. Being an Amiga graphics compressor, the picture is handled in a planar format. This is how all Amiga graphics screens are stored. So, the least significant bit of each colour pen on the screen is read, then the next significant bit, and so on for every bitplane. Next, each bitplane is split into horizontal strips, or "lumps" as I call them. Each lump is handled in turn, from the top of the bitplane to the bottom of the bitplane. Each lump is encoded as a stream of 8 pixel bytes, from top to bottom, working left to right. The height of lumps varies -- as detailed previously, the compactor tries different lump sizes, and finally settles on the lump size that gives the best compression.
The result of picture re-ordering is a continuous stream of bytes. These are run length encoded in an interesting way. The picture bytes are written to a stream called PICDATA. For each picture byte, one RLE bit is written to a stream called RLEDATA. If a picture byte is the same as the previous picture byte, nothing is written to PICDATA, and a 0 bit is written to RLEDATA. If the picture byte is different from the previous byte, it is written to PICDATA and a 1 bit is written to RLEDATA.
The RLEDATA stream is also run length encoded. After every 8 bits are written to the RLEDATA stream, a comparison is made. If the most recent 8 bits are the same as the previous 8 bits, then those 8 bits are not written to the RLEDATA stream. A 0 bit is written to a data stream called POINTS. If the 8 bits are different from the previous 8 bits, they are written to the RLEDATA stream and a 1 bit is written to the POINTS stream.
For simplicity, and Amiga-centricity, this XFD slave always decodes the Pac.Pic. file into the uncompressed, Amiga IFF ILBM image format. Unlike Pac.Pic., this is a very well known graphics format, and is simple to convert into any other format using graphics software like NetPBM.
The worrying thing is that AMOS has always been bundled with the source code of its extensions (like the Music extension, and the Picture Compactor extension) since it was released in 1990. For all intents and purposes, the format of the Picture Compressor has been public knowledge for over a decade. Yet, nobody until now has released a decompressor for the format, despite there being demand for a decompressor.
CPic images
download version 1.0 and source code
CPic is an old picture compression system, written in 1990 by Thomas Zipproth, using the Modula-2 language. You can find CPic on AMOK disk 42. CPic offered better compression than IFF ILBM images. However, CPic's compression is still only run length encoding, so it only performs well on images with a lot of blank space, a photorealistic CPic image is just as large as the ILBM would be, if not bigger because of mandatory "no compression used" markers every 96 bytes.
Like the AMOS Pac.Pic. XFD slave, the CPic XFD slave decompresses to the well known Amiga IFF ILBM file format.
Kenny Grant
download version 1.1 and source code
This XFD slave decompresses data and executables that were packed onto Amiga Power and CU Amiga coverdisks by a programmer named Kenny Grant.
As I understand, Kenny was the archetypal coder; attending school or university, skilled at disassembly and cracking, and under the legal employ of magazine publishers back in the days when you got floppy disks on the front cover. At the time, games publishers would send game demos to magazine publishers for use on their coverdisk. Occasionally, they were nice, and sent a single executable file that could just be run to play the game. This meant to the magazine buyer that they could play games straight from the disk, without having to de-archive their coverdisk onto two or more blank disks first. However, many games publishers didn't like sharing the coverdisk with other games, so they used raw disk loading systems, or worse still, the protected loading systems that they would use in the full games. These disks could only be copied by disk distributors and software pirates, not normal magazine buyers.
In order to reclaim the multi-game immediate-run coverdisk days of yore, Amiga Power hired the services of Coverdisk Compiler Of Champions, Kenny Grant. Although it seems he was a one-man outfit, his trading name was Grants Computing. Kenny would turn these game demos into one file, or at least reorganize the disk so track-loading games could share the disk, and the disk was in the standard copyable Amiga format. For more about this, read Jonathan Nash's comments on AP coverdisk hi-jinks. Although I can't confirm this (Kenny? Are you out there?), he developed or stole two LZH compressors - GC!2 and GC!3. The "GC" stands either for "Grants Computing" or (allegedly) "Graftgold Cruncher".
Kenny could hack his decompressor into the loading code, and therefore could compress external data files as well as single executables. He also developed his own "A1200 compatible", auto-degrading, system killing, absolute-address decompressor. You may have seen this in action - it appears as the word DECRUNCHING! in italic capitals on a black background, flashing all sorts of colours, before the game appears. Kenny went one step further, and created an overlayed executable compressor, in a similar style to (or possibly copied from) the Titanics Cruncher. This meant that that games could be loaded directly from normal files on disk, and decompressed without needing any more memory than the game itself required. This meant that very large game demos, like Alien Breed 3D, could still be packed onto coverdisks.
MMCMP
download version 1.0 and source code
MMCMP is an MS-DOS program by Emmanuel Giasson (Zirconia). It watches for you saving MOD and Impulse Tracker files to disk, intercepts this, compresses them, and writes them to disk itself. It also does the reverse when you load these files.
I have not reverse-engineered the MMCMP program or compression format, but thankfully Olivier Lapicque wrote a partial decruncher for his MODPlug program with the blessing of the original authors. This seems to work with most MMCMP compressed files. This code then appeared in Claudio Matsuoka's XMP player, where I took the code and translated it into an XFD slave.
Microsoft SZDD
download version 1.4 and source code
Microsoft used to supply MS-DOS and other software in a compressed form where each file had an underscore at the end of the filename, e.g. VBRUN666.DL_ or MSWORD.EX_. The tools used to create and extract these files are called COMPRESS.EXE and EXPAND.EXE.
The original COMPRESS.EXE, and the system library LZEXPAND.DLL, create and unpack files in the "SZDD" format. SZ stands for Steven Zeck. The compression method used by SZDD is Haruhiko Okumura's implementation of the LZSS algorithm, although Microsoft call this the "Steven Zeck algorithm". It is also identified as method "A" by the SZDD file format, although no other SZDD methods have been created.
Before being replaced by the Cabinet file format, the SZDD file format evolved to the rare KWAJ file format. This is not yet supported by the SZDD XFD slave. The KWAJ format supports three compression methods. Method 2 is identical to SZDD's LZSS method, but using a different window start offset. Method 3 is LZSS but uses four Huffman trees to encode matches. Method 4 is identical to the ZIP deflate/inflate method. For more on the history of Microsoft compression, read my cabextract page.
Resident De/Compressor
download version 1.1 and source code
Resident De/Compressor is a program on the Atari ST, written by Simon Gibbs, which causes all file operations to transparently read and write compressed files. The compression scheme used by Resident De/Compressor is simply Huffman coding.
SPacker v3
download version 1.1 and source code
This is a popular format in the Atari ST scene. It has the ID SPv3. Mr. Larmer has already written a slave for these files, but it failed to decrunch some files and crashed on some others, so I wrote a new one based on debugging some source code I found on an Atari disk. Like most popular Atari ST packers, SPv3 is an inbred hybrid of Pack-Ice, Automation Packer and Atomik Packer. One of the nice things about the SPacker application itself is that it could decrunch files from rival packers, using an external slave system similar to XFD.
Zulu & Grey
download version 1.1 and source code
This is a hacked version of StoneCracker 4.03, that was used by Zulu and Grey in their Rebels productions "Switchback" and "Whammer Slammer". These files have the signature Z&G! or ZULU.
This XFD slave is no longer needed. As of XFD 1.37, is is part of XFD's StoneCracker slave.