How the optvgm-tools work

Got a new set? Wanna point something out with an old set? Wanna update an existing set? Here's your place to do it!
Post Reply
ValleyBell
Board Regular
Posts: 128
Joined: Mon Aug 30, 2010 7:10 am

How the optvgm-tools work

Post by ValleyBell »

Here I'll describe how my vgm-optimizing tools work (from a technical view).

VGM Compressor (vgm_cmp)
This tool "emulates" parts of the chips and evaluates, if the a command is needed (if neccessary by requesting following commands). Else it's deleted.
Example: The YM2612 received a channel 0 frequency write A4 -> 04 and A0 -> 25 and saved that. (Freq0 25, 04)
Then it receives A4 -> 04 (LSB part of frequency) and will request the next command until it gets an A0 (Freq MSB) or A4 (another Freq LSB).
An A4 command will cause the earlier one to be deleted, because only the A0 command applies the values of both A0 and A4.
If the command is A0 -> 30 both commands are kept and the new frequency (Freq0 30, 04) is saved.
If it is A0 -> 25 both will be deleted, because the channel's frequency already has this value.

That was one of the more complicated examples. Most commands just need a comparison between the current and the last value without being affected by other commands.

When I wrote this tool, I didn't just want "the same sound" but 100% identical output. So a vgm2wav of an uncompressed vgm and a compressed one match on a byte-by-byte-comparison.

Effectivity (% is the size reduction without data blocks):
- mostly good (from 5 to 90%, 60% average), heavily depends on the game's music engine


VGM RF-PCM Optimizer (optvgmrf)
The RF5Cxxx chips (like the Sega CD PCM chip) have some memory (64KB) to store their samples/waveform data. This isn't very much, so games have two ways of making sounds with such a chip.
1. They store low quality (or only a few) samples at the initialization phase and don't touch the memory for the rest of the game (or just small areas for voices and FMVs).
This is preferred by quite many games, especially if they use the PCM chip to play only drums and the YM2612 for the rest. (vgm_sro deals with that sort of vgms)
2. They use high quality samples and stream them so that they can use more than 64 KB of sample data. That means that the game has to do write as much data to the chip's memory as the chip plays, so that generates quite huge vgms. optvgmrf deals with that sort of vgms.

Step 1: At first the program walks through the whole vgm and looks for RAM writes and merges consecutive ones into larger blocks. (e.g.: There are writes to address 00, 01, 02, ... 29, then 80, ..., 85, they will be merged to one block with data for 00-29 and another one with data for 80-85.)
This is a little lossy, as you lose the accurate position of the writes. (All memory writes are moved to the sample of the first write - that's where the block starts.)
A comparision of two vgm2wavs shows small differences, each only one sample long and not audible.

Step 2: Then it will run through the vgm again to read all the data blocks and copy them to the block database. (the file offset is noted to localize the block later)
Now all blocks are compared with each other. Blocks that exist multiple times get a reference to the first block with this data. There're also some statistic counters that count, how often one block of used.

Step 3: Finally the new vgm is written with the database placed at the start of the vgm. All memory writes are handled via 0x68 commands, with exception of the blocks that are used only once. These are stored as a normal memory write data block at their actual position to reduce the size of the database (and the initial loading time).

Effectivity (% is size reduction of data block):
- very good (always around 90%)


VGM Sample-ROM Optimizer (vgm_sro)
This tool goes a completely other way than optvgmrf. It doesn't reorganize the data, it consequently deletes it.
At first the complete sample ROM/RAM data is saved, as well as a data mask where used bytes are marked. When there is a command that sets a start or end position of a sample, it's saved the same way the actual chip (emulator) does it. When a key on/channel on command is received, some bytes are marked as used using the saved start/end position.
Now the "ROM Region List" is created by simply counting how many consecutive bytes are used or unused. All used areas are noted in a Memory Region List. (but both used and unused areas are shown)
Finally the vgm is written. At the point where is finds the first sample ROM/RAM data block, multiple data blocks (of the same type) are written instead, one for each memory region. All following data blocks of this type (if there are any) are ignored to avoid bugs because a user wants to optimize an already optimized file. All other data is simply copied.

Effectivity (% is size reduction of data block):
- SegaCD PCM: little (64 KB don't leave much room for improvement)
- others: very good (mostly around 90%)


VGM Data Block Compressor (vgm_dbc)
This tool tries to compress data blocks (e.g. YM2612 DAC or PWM data) with some fast and lossless algorithms.
At first it searches the whole file for uncompressed data blocks (type 00 to 3F) and stores them into a buffer.
Then it creates a compression table for each block. That table contains the bit count of the decompressed and compressed data, as well as the used compression (with algorithm type and attributes).
Finally the file is rewritten with value tables (type 7F, explained below) and compressed data blocks (now with type 40-7E).

Compression Algorithm Types:
- 00 - n-Bit compression (attributes: sub type and subtracted value)
- others - reserved for future use

n-Bit compression:
This compression is quite simple. Let's say we have some numbers [00, 04, 01, 0D, 0F, 08] with each one takes up exactly 1 byte (8 bits). As you can see, no number uses the first hex-digit. Because we noticed that, we can strip the first digit of each number. Now we have [04 1D F8] and great - we compressed the data by 50% without losing any information, simply by using just 4 bits instead of 8.

Additionally the n-bit compression has these sub-types:
- 00 - add and copy
Before compression, a constant number ("subtracted value"-attribute) is subtracted from all values in the block in order to make the compression more effective.
Example: [80, 87, 84] - 80 -> [00, 07, 04]
- 01 - add and shift right
Like add and copy, but additionally all bits are shifted right by (Bits Decompressed - Bits Compressed).
Example: [F5, A5, 95] - 95 -> [60, 10, 00] >> 4 -> [06, 01, 00]
- 02 - use table
A table of values is created, similar to the palettes in indexed picture files.
Example: [09, A5, 84] -> table [09, 84, A5] -> [00, 02, 01]
Value tables are stored seperately from the data block (in the vgm file the table precedes the data block), so that multiple data blocks can use the same table.
Note: The value tables of consecutive data blocks get combined as long as all values fit into the compressed value without having it increase its size.

Effectivity (% is the size reduction of data block):
- YM2612 DAC: Bad (mostly none because all 8 bits are fully used, but sometimes 40-50%)
- PWM: quite good (always >= 25%, often 40% - 16 bits are stored, only 12 can be used, 10 are commonly used)


Tools currently in developement:
- VGM PWM Optimizer (optvgm32)
Last edited by ValleyBell on Fri Apr 22, 2011 10:03 am, edited 3 times in total.
SmartOne
Posts: 196
Joined: Fri Jul 20, 2007 2:33 pm
Location: Your mom. HA!

Re: How the optvgm-tools work

Post by SmartOne »

I agree.
ValleyBell
Board Regular
Posts: 128
Joined: Mon Aug 30, 2010 7:10 am

Post by ValleyBell »

Thanks.

When I'm looking at the tools I wrote (I'm counting 15 without the WIP tools, players and emulator mods), I must admit that I was quite productive during the last year.

When I made my first vgm mods for NeoPop and MAME, I had never imagined that I'd change the VGM format that much. But seeing that the vgm format wasn't touched for years, I wanted to make some suggestions for improvements and became the format's new developer because no one stopped me from adding new stuff.
And now it's undoubtable that I brought the vgm format to a new level. :)

Btw: I'm almost 20 years old. And I've got some talent when it comes to read and understand others' code. I'm sure that helped me a lot, because I'm reading MAME's sound cores everytime I need to know something about how the chips work. The rest is mostly trial and error. (And maybe the fact that I'm programming 80% of my free time.)
ValleyBell
Board Regular
Posts: 128
Joined: Mon Aug 30, 2010 7:10 am

Post by ValleyBell »

(Sorry for the late response. I always tend to delay things.)

I can't say that I really know how to control the YM2612.
The PSG - yes, but that's easy. The OPL3 chip on my SoundBlaster 16 - yes, I made my own MIDI Player for it.
But the YM2612 - not really. I learned all I know about the chip when I wrote vgm2txt (MAME cores are quite well commented, so I could use the comments).
I just know some very basic things (e.g. how the Note On/Off and Stereo commands work), but I can't even say how it really calculates a note's frequency.
On the other side I haven't yet bothered to learn the YM2612.

About reading code: I always try to understand only the parts of the code I need to read/modify, but nothing more. So I e.g. can log vgms in MAME without having to bother how that thing loads roms/executes something. The rest is just trial and error.

I started to learn coding with a very old BASIC language that's older than me.
ValleyBell
Board Regular
Posts: 128
Joined: Mon Aug 30, 2010 7:10 am

Post by ValleyBell »

I wish you good luck. And take some time.
If you get to a point where you think "programming is fun", that's more than good and surely the right way.

I was nice to do some normal talk. (My posts are too often just a collection of too much information.)

Btw: I just added the description of the Data Block Compressor I finished a week ago. I haven't relased it yet, because it's quite useless without an optvgm for PWM data.

EDIT: It would be nice if someone could update the Wikipedia article for the VGM file format. Thanks.
ValleyBell
Board Regular
Posts: 128
Joined: Mon Aug 30, 2010 7:10 am

Post by ValleyBell »

I don't think you need to know a lot about the sound cores. Wikipedia doesn't have ANY sound core related information in the article. (Despite of the fact that the PCM optimization is mentioned.)
And if you have any questions, I'm willing to tell you all you want to know. (Btw: The "new" things start with v1.51.)

Here's a small list of sound cores and their usage. Maybe you can include it in some form.
    SN76489 - all Sega consoles already listed
    YM2413 - 8-bit Sega consoles (already listed), MSX ("MSX Music" or "FM PAC")
    YM2612 - MegaDrive, of course
    YM2151 - MSX ("Yamaha SFG")
    YM2610 - NeoGeo Consoles
    YM3826 ("OPL2") - PC (AdLib cards)
    YMF262 ("OPL3") - PC (e.g. SoundBlaster 16), MSX
    YMF278 - MSX ("Moonsound")
    Y8950 - MSX ("MSX Audio")
    RF5C164 - Sega CD PCM
    PWM - Sega 32x
    AY8910 - MSX ("MSX PSG")
All chips not listed are (AFAIK) only used in arcade machines. And most listed ones, too, btw.

The thread's first post convers almost all optimizations but again - I don't know if a Wikipedia article needs that lot of detail. An approximate description would be surely more than enough.
The only thing it doesn't conver is the DAC Streaming Driver, which is quite easy. It simply sends data bytes (from the PCM data bank) at a changeable sample rate (independent from the VGM's one) to a chip (chip type and command are set in the VGM). It also supports interleaved streams.

I hope that helpled. I would be great if you'd update the Wiki page.
ValleyBell
Board Regular
Posts: 128
Joined: Mon Aug 30, 2010 7:10 am

Post by ValleyBell »

Monkeymook wrote:I'm looking for new capabilities of the format, or capabilities that have been updated. Your jargon is kind of hard to interpret, and there doesn't seem to be much 'to-the-point' information on what has been done from version to version, or from update to update.
I'm sorry for that and I'll try to summarize all changes here:
v1.50
- last offical version by Maxim

v1.51
- added veery many chips here (including SegaCD PCM and PWM). Chip means header values, commands and data blocks for ROM data (RAM writes for SegaCD PCM).
The ROM blocks already supported the optimization vgm_sro does.
- added dual-chip-support for almost all chips. (This was initially implemented as some sort of "cheat"-feature and still isn't documented properly because of that.)
I'm currently adding some things (this and others) I forgot to note when I wrote the vgm spec updates.
(Note: v1.51 files are technically fully compatible to v1.50, but in_vgm had problems.)

v1.60
This is where I started to invent special optimizations impossible with existing commands (and defined command ranges). (optvgmrf and optvgm32)
- added RF5Cxx (Arcade + SegaCD PCM) and PWM stream data blocks and a command for optimized RF5Cxx data streaming
- added the thing I call "DAC Stream Control"
- no new chips
All in all this completely broke compatibility with v1.50 players, but it was similar to the version jump from 1.10 to 1.50.

Hope this helps more.
(Another note: As you could saw recently, I don't do most updates on one day, but over rather long periods of time. So I already had started with v1.60 when I added the last chip to v1.51. That's why I need to update the players soo often. But this list should still be very accurate.)

Monkeymook wrote:Also, the VGM spec sheet and info on this page needs to be updated. I haven't an account there, therefore I am unable to update it.
I'll take care of this.
I hadn't updated the page in the beginning because I just made "unoffical updates".
ValleyBell
Board Regular
Posts: 128
Joined: Mon Aug 30, 2010 7:10 am

Post by ValleyBell »

Thanks. You did a nice work.

I've just updated the VGM Format page on SMSPower.
I made very many changes to the last vgm spec (e.g. documentation of dual-chip commands). btw: the v1.60 beta was left by accident

EDIT: Just noticed that I don't need an account to edit the wiki page, so I'll do that tomorrow.
EDIT2: I edited the VGM Format Wikipedia page and could remove my notes from this post. I think it looks cleaner now without losing important information.
Monkeymook's update was still a nice base to work on.
learnerofskills
New Around Here
Posts: 2
Joined: Tue Mar 08, 2011 7:26 pm

Re: How the optvgm-tools work

Post by learnerofskills »

ValleyBell. Thanks so much for making these tools. I was waiting years for someone to make tools like these and I could not have made a Sega CD VGM set without them.

I have one question however. After making my first VGM set, I'd have to say that the most difficult (and most time consuming) part is finding the loops in the VGMs. Do you have any plans for updating vgmlpfnd.exe to work with sega cd VGMs?
ValleyBell
Board Regular
Posts: 128
Joined: Mon Aug 30, 2010 7:10 am

Post by ValleyBell »

vgmlpfnd actually supports SegaCD vgms (I used it to trim Popful Mail), but maybe I could improve the way it finds the loops.
One or two untrimmed vgms would be nice to test.

Btw: I know how time consuming it is to find the right loops. (SegaSonic Arcade was pure horror - it took 15 minutes and more to get some of the loops right. But that's another story.)
If you have problems to find the exact loop sample, you should optimize the vgm (keep the original) and make a vgm2txt of that. (Then you can delete the optimized vgm.)
Trim at some "Channel Enable" command and your chance for a good loop is a lot higher.
Post Reply