Bitrate reduction from lossless encoding in MP3 encoded audio files

Csaba Horváth,
tonestack.net

June 20, 2025

Lossy audio compression is a result of lossy compression (quantization) and lossless compression (transform gain, Huffman coding, differential encoding, etc.). The efficiency of a lossy audio format depends largely on the efficiency of the lossless encoding stages. Furthermore, the efficiency of lossless encoding can be measured by disabling all quantization except rounding.

The ratio of lossy and lossless compression varies from signal to signal. For signals with low harmonic content (four tone multitone, triangle) bitrate reduction is mainly the result of lossless encoding. On the contrary, for signals with high harmonic content bitrate reduction is largely derived from quantization. Although the efficiency of lossless compression is low for signals with many harmonics, the greatly increased masking threshold can hide a large amount of quantization noise.

The ratio of lossy and lossless compression also varies with signal level: the contribution of lossless compression to the entire compression is greater at low signal levels. In fact, perceptual coding is secondary at very low signal levels. (Lossless encoding of low-level signals requires very few bits, mainly because low-level signals don't use all the bits in a 16-bit WAV file, so unused bits can be simply discarded.)

There are two lossless stages in MP3 encoding: MDCT transformation (with subbands) and Huffman-coding. Although MDCT transformation is lossless, the implementation in MP3 (and other lossy encoders) is not completely lossless, since MDCT uses non-integer arithmetic, whereas the input for Huffman-coding are integers. At least a rounding has to be performed before Huffman-coding of frequency values. Fortunately, quantization can be configured to produce a fixed 16-bit noise floor. The nonlinear quantizer (x^0.75) can be either included or excluded from the encoding process.

1. The modified encoder

Description of the encoder used for generating the tables:

• MP3 type transform blocks (overlapped Hann-windows, 384 or 1152 PCM data in);
• MP3 type bit allocation (code regions and Huffman code tables);
• global gain is fixed at 210, this means that global quantizer is 1;
• scalefactors are not used;
• Fs = 44.1 kHz, bins between 523-575 are zero;
• one channel.

Encoder noise floor, linear vs. nonlinear quantizer

Since lossless encoding of low-level signals requires very few bits and perceptual coding is secondary at low signal levels (ATH is important, masking is not) we can ignore the analysis of low-level signals and focus on high-level signals. By ignoring low-level signals, we can weaken the requirements of the analysis.

Two types of quantizer setting were tested, which has a minor effect on the noise floor.

When power law quantization is enabled and global gain is 210 (slightly nonlinear quantizer), the encoder noise floor for high level signals is equivalent with a noise floor of a 14 bit system. The dynamic range is equivalent with the dynamic range of a 16-bit system. Not lossless compression but way beyond transparency.

Quantization for this setting:

ix(i) = int ( |xr(i)|^0.75 - 0.0946 ),

where ix(i) is integer, xr(i) is float or double.

When power law quantization is disabled and global gain is 210 (linear quantizer), the encoder noise floor is equivalent to a noise floor of a 16-bit system. The actual signal-to-noise ratio and dynamic range depend on how dither is applied at the output and input. In our case dynamic range and SNR are not so important, as we are mainly interested in signal compression above -48 dBFS. (Note: the dynamic range of an MP3 file can be much larger, as the global gain varies from block to block in a real encoder.)

Quantization for this setting:

ix(i) = int ( |xr(i)| )

In this case, quantization is rounding a floating point frequency value to the nearest integer.

2. Periodic signals

Power law quantization is enabled (nonlinear quantizer)

Details:

• power law quantization ( y = int ( |x (i)|^0.75) ) is the only quantization
• global gain = 210, global quantizer =1
• long blocks
• The result is average of 9 frames

Signal	Bitrate (kbps)
	0 dBFS	-6 dBFS	-12 dBFS	-30 dBFS	-48 dBFS
sine, f=100Hz	22	20	18	11	6
sine, f=1kHz	36	32	28	19	11
sine, f=10kHz	42	36	34	21	13
Triangle, f=100Hz	145	110	77	25	8
Triangle, f=400Hz	220	180	150	70	-
Sawtooth, f=100Hz	393	352	300	190	83
Sawtooth, f=200Hz	410	370	320	200	95
Sawtooth, f=400Hz	440	400	350	210	110
100Hz+1k+5k+15k	175	160	140	85	40
Multitone	315	-	240	160	-

Notes:

• "-" means not tested.
• Multitone: third-octave multitone with a base frequency of 40 Hz.
• For comparison: a 16-bit / 44.1 kHz mono wav file has a bitrate of 705.6 kbps.
• Sawtooth @ 100Hz: this signal doesn't sound as a continuous signal, but as a series of pulses ('pulse train').
• Correct encoding of triangle signal without scalefactor quantization requires a global gain of 205 (quantizer = 0.5). Global gain of 210 introduces some unmasked HF noise.

Power law quantization is disabled (rounding only)

Details:

• no power law quantization, quantization is rounding float MDCT value to integer;
• long blocks;
• The result is average of 9 frames,

Signal	Bitrate (kbps)
	0 dBFS	-6 dBFS	-12 dBFS	-30 dBFS	-48 dBFS
sine, f=1kHz	no table	40	-	-	-
sine, f=10kHz	no table	40	-	-	-
Triangle, f=100Hz	no table	no table	80	-	-
Triangle, f=400Hz	no table	no table	160	-	-
Sawtooth, f=200Hz	no table	no table	410	-	-
Sawtooth, f=400Hz	no table	no table	430	-	100
Multitone	410	350	320	180	75

3. Bitrates for pulse signals

Two extremes should be analyzed:

• Best case: pulse is located in the middle of a transform window.
• Worst case: pulse is located at the meeting point of two transform windows.

In the following simulations bitrates are average of two granulas. Global gain is set to 210, quantizer is 1.

For the empty granula (best case) 50 kbps for background noise is allocated. (Granula: one long block or three short blocks.)

Short windows

Quantizer	Signal	Bitrate (kbps)
		0 dBFS	-6 dBFS	-12 dBFS	-30 dBFS
Non-linear	Pulse best case	170	150	120	-
	Pulse worst case	250	235	200	-
Rounding	Pulse best case	215	190	160	-
	Pulse worst case	290	270	240	-

Long windows

Quantizer	Signal	Bitrate (kbps)
		0 dBFS	-6 dBFS	-12 dBFS	-30 dBFS
Non-linear	Pulse best case	180	150	120	52
	Pulse worst case	310	270	230	120
Rounding	Pulse best case	225	180	130	60
	Pulse worst case	390	310	260	130

4. Start of periodic signals

Signal level is 0 dBFS, power law quantization is enabled. Start is at the meeting point of two granulas (worst case). Short blocks.

Signal	Granules (bitrates, kbps)
	#0	#1	#2	#3
Sine, f = 10 kHz	280	280	30	30
Triangle, f = 400 Hz	262	322	250	254
Multitone	254	354	291	294

5. Contribution of lossless coding and perceptual coding to bitrate reduction

The chart shows the bitrates after lossless compression (rounding is the only quantization) and lossy compression with a non-modified MP3 encoder (standard perceptual coding). The bitrate for the lossy compression is set for "transparent" quality (errors are not audible).

Blue: lossless compression, red: lossy compression,
signal level is -12 dBFS; results are in kbps

The difference between lossless and lossy compression is the bitrate reduction from perceptual coding.

Unfortunately, this method has a minor flaw: Huffman-code tables are optimized for encoding quantized values less than or equal to 4 bits, or for bitstreams where values greater than 4 bits are rare (4 bit is meant without sign). When quantization is disabled, Huffman-coding becomes inefficient at high signal levels. It's likely that the compression resulting from lossless stages in standard MP3 files is somewhat greater than the previous graph suggests.