Kara-Moon Forum

Looking for the "best" way to represent a quality recording in compressed "lossy" format has been a bit of a project for me over the last couple of years, and after some limited success (and repeated failures, accompanied by depression and feelings of self-doubt :D), here's the situation, as I see it....

Adjusting the source file - applying compression and equalization to provide the most advantageous combination of dynamics and tonal characteristics for your compression codec (Ogg Vorbis or MP3) is critical for best results. In general, lower the bass frequencies, gently boost the frequencies above 10,000 Hz, and compress the file as much as possible without loosing too much dynamic range. The idea is to pack as much useful audio information as possible into your original full-size audio file, while removing those ingredients which don't really contribute to the listening experience in the resulting compressed file.


Ogg Vorbis, by its nature, is variable bit-rate format. By selecting from the various quality settings, you can determine the relative audio quality and resulting file size. The higher quality settings in Ogg Vorbis are commonly perceived by the average listener as indistinguishable from the original audio file. Ogg Vorbis development is ongoing, with updates periodically available from the open-source community.

MP3 as represented by the Fraunhofer codecs, is a reliable, high quality audio compression format. It is generally applied in a "constant bit-rate" method of compression, and is considered the industry standard for MP3 encoding. Quality settings are determined by selecting a bit-rate number; 128kilobits/second being the most common setting. The Fraunhofer codecs are a proprietary format.

MP3 as represented by the LAME codecs, is the open-source version of this audio compression format, and the one I've been working with since adopting Audacity and Ardour/JAMin as my main recording/mixing/mastering tools. Originally developed as an offshoot of current MP3 technologies, it has since been gradually re-written so that none of the original code remains, and thus approaches audio compression in a fundamentally different way than Fraunhofer, while remaining compatible with all recent MP3 players. Unlike Fraunhofer, it is now most advantageously applied as variable bit-rate compression, with a choice of "variable bit-rate" and "average bit-rate" formats, depending on your application. LAME development, like Ogg Vorbis, is ongoing.

My most recent concerns regarding a curious "warbling" effect in Will's(Folderol) "Empires of Dust" was a result of forcing the LAME codec to encode the compressed file in "constant bit-rate" mode. With the recent upgrades to LAME, the constant bit-rate format is optional, and not recommended. Now I know why...

Live and learn... ;D

Interest Oren, thanks for posting this.

Can you explain what you mean by "compress the file as much as possible without loosing too much dynamic range"
Do you mean use a compressor as much as possible without loosing dynamics ?

k

Cool stuff..I never thought of it that way, and I have a good example of a song that mp3 trashes to work with.

Thanks for posting this.

8)

Wyatt

You really have got your scholars hat on these days, haven't you :D

This rather explains something. I tried encoding this myself at a bit rate of only 128k (the default for LAME) and didn't seem to get this effect anywhere near so badly. I didn't pursue it (shame on me) but I'm now wondering if LAME also defaults to variable bit rate.

It's funny how MP3 can trash a sound from one system, and have a different effect on another.  Oren has put a lot of time into learning these compression schemes; heed his advice.  He does it very well. 
 MP3 is pretty much a brick wall at 16kHz, and has a slight curve just ahead of that frequency in which the signal declines.
 By boosting (and cutting) the areas Oren mentioned you are trying to make the most out of what the format allows you to keep.  Push it before the MP3 format cuts it, so the end result is a close approximation of the original.
 Also by using compression (carefully) you can give the codec 'more info' to force it to handle it as you want.  Cymbals are an example.  Sometimes the extra ring the compressor gives to the signal by boosting that fading cymbal crash can make a big difference on how the MP3 format handles it..
 I had started studying this before Oren and I met, and now he has surpassed me for knowing what to do to a mix ahead of time to get the desired result.  He's good at it; I use a little trial and error..still..

 I just wish LAME would make a simple installer that installs the codec on your system so other softwares can use it.  It seems now that you have to go the inf route to install it into the system, and sometimes that is a little 'iffy'.  I have codecs that show up under the Lame MP3 listing for certain bitrates and frequencies that never work here, and other bitrates and frequencies from Lame do work here..and I know people that can't get it to work at all that way; they have to use a seperate program to convert to MP# (like Audacity)

That's exactly what I mean :-

Yes. MP3, as represented by the LAME codec, is a variable bit-rate compression method. Similar to Ogg Vorbis in this respect.
Constant bit-rate is an available option, but must be specifically selected from a drop-down menu.

Me too!
As it stands, I use Audacity as a G.U.I. "front end" for LAME. It tends to present the application and its various options in the most comprehensive and useable package.

guys .. guys .. drop the mp3's. FLAC is the way to go!
if you disagree with my input, please give me some constructive feedback on why the MP3 Lame encoder and Ogg Vorbis encoders are better than FLAC .. :)

FLAC is superior in every way.
The file size is a bit large for posting on websites, and some people's download speed is very slow, making the MP3 and Ogg Vorbis more practical for casual listening.
As musicians, however, we should be using FLAC wherever possible.... O0

On the few collabs I've done here I've tended to chuck out flac files to my co-musicians for precisely this reason.

For my own personal listening on limited space devices, I use ogg at quality 5, and 128k lame encoded mp3 stuff for the unwashed masses :)

I second that. If i'm to do a collab i want no less than .wav tracks to work with. There's only 1 person in the whole wide world i give permission to send me mp3 or ogg vorbis compressed mixdowns. a dear friend on the jungle with a 56k dial-up modem(!)  :o hahaha
For the masses I'm most likely to do a 192kbps VBR, but a minimum 160kbps mp3. 160kbps CBR is the same compression you get on any cd you buy in the recordstore. 128kbps is fucking (sorry language) rubbish. Drums, espacially cymbals tend to just wash into jittering. as do snares and hi-hats.  ::)

Shame on you compressing > 128kbps mp3's !!  >:D

IMHO Lame 160 is equal to Fraunhofer Pro codec 128.  Both have their little idiosynchratic issues..one will make an electric piano just intolerable, the other may ruin a ride cymbal...

 Of course wav is the way to go, and ape files have the ability to make wav files very manageable in size with absolutely no loss.  I still think FLAC somehow has an effect when decompressed.  Just my opinion from experimentation.  I'm sure others will point to their claim as lossless compression, but use it and run the files thru an analyzer.. or better yet, use it and run the original and the uncompressed file thru a set of effects using the exact same settings.. see if they sound exactly the same  still..  There have been times when I've heard the difference pop up.

I had no idea :o A sobering thought....

It gets worse!

From an engineering point of view I couldn't accept the idea of a lossless compression system that was not, well, lossless. For this to happen there would have to be some defect in the methodology (or its implementation) and with the amount of exposure, an open-source codec like flac gets, surely this would have been commented on by now ... widely and loudly!

I decided to do some experiments. The source material was one of my own tracks created in Audacity. I then used Audacity to create a Flac, killed and restarted Audacity reloading the file and converting it back to Wav.

I then used md5sum to do a quick integrity check. Lo and behold, the second Wav had a different signature to the first. Loading both files into Audacity I could see absolutely no difference in the waveform - even massively stretching them to see individual cycles. Thinking that maybe some tag data might be different (timestamp etc) I then examined the files with a Hex editor, and at lots of places through the files, compared them side by side. The data was quite different and it wasn't just a time shift - i.e. the same pattern wasn't just slightly later in the file.

As a double blind, I then used the original Wav file and got Audacity to just save a copy. To my amazement this was also different. Thinking that maybe there was some quirk in the original file (even though it was created with exactly the same copy of Audacity) I made a second direct copy. This was yet again different from all the previous ones, as was a third copy from the original file.

This now means that  I have no way of discovering whether Flac is truly lossless or not, as I can't create a stable, repeatable Wav file anyway. I would add, that in spite of the binary differences, I could hear no difference at all, and could see no difference in Audacity. However if the source files are changing then obviously any translation code will do different things each time.
My flabber is ghasted :o

This is an interesting experiment Will
So, this means that everytime Audacity is saving a wav file, the file is different  ???
I wonder what happens there....

k

My first thought in this regard is: How many other applications alter an audio file at some tiny, but fundamental, level?

Next: Are any "lossless" codecs actually completely lossless?

And finally: How many of these technical nuances are of practical concern to the recording musician?

Some of these technicalities are beginning to surface in a most comical manner, for instance: 24bit audio files are a must for those who want extended headroom and maximum dynamic range, but...a 24bit signal will not "fit" through an audio cable - the cable effectively "compresses" the signal as it passes from one component to another... so what's the friggin' point? :o :D ::)

Well, I've set the cat among the pigeons over on the linux audio users list  ::)

I've given them the URLs of the test tracks and the md5sum signatures of each of them. One guy who is, shall we say, 'interested' is the person that wrote the code Audacity uses in it's export function :o

I will keep you informed of developments.

BTW Oren, if my mathematics is correct, 24bit corresponds to a dynamic range of 144dB. Maybe if you helium cooled your cables you could get the noise floor down that far ;D

Hey guys. New business model! Super-cooled audio transmission cables :;

Or, errrr, SCAT cables  :P

SCAT.......love it! O0

The more techno-gibberish I hear from those who would like us to believe "more and bigger" is better, the more I'm inclined to SCOOT back to 16/44.1 audio and 128kbit MP3 and Ogg Vorbis :o :D 8). Some of what I hear represented in these formats is superior in every way to much of what gets the 24bit/96KHz/256Kbit treatment.

LAME MP3 and FLAC are difficult formats to compare with traditional MP3 and lossless compression because, represented in their complete form, each has multiple levels of encoding quality. Depending on the GUI you choose to access the features of LAME and FLAC, you may be working with a relatively low-functioning version of the codec, chosen by the GUI designer because of it's encoding speed.

LAME MP3 can be encoded as CBR (constant bit rate), TBR (target bit rate), ABR (average bit rate), or VBR (variable bit rate).
CBR, TBR, and ABR are actually "crippled" versions of the codec that force it to perform more like traditional compression formats.
VBR is the version of LAME in which it performs at full capacity.
Players, although able to decode and play LAME files, cannot accurately measure the bit rate of the file, so a VBR file averaging 192Kbits/sec will show up on the player as a 320Kbit/sec file. File size, however, will be approximately the same as a CBR file encoded at 192Kbit/sec.

Currently, FLAC has 8 levels of encoding accuracy... the faster the encoding speed, the lower the accuracy. If your GUI only offers one level of encoding accuracy, it is likely one of the mid-range selections, and thus not the best FLAC can do.

Generally, open-source applications are developed by enthusiasts for enthusiasts, and a certain level of competence and familiarity is assumed. When represented by a "user-friendly" GUI, many of the features and options are rendered unavailable, for the sake of convenience. It is up to the user to find the GUI which most comprehensively represents the application, and to explore the various performance options to determine how best to configure it for a particular job.

This tread is becoming mighty interesting  8)

k

Right then!

{rolls up sleeves}

I have some answers - not at all what I expected.

This is what Erik (the programmer of libsndfile) wrote after looking at the sample files I gave.

The differences are small, about 2 least significant bits, which suggests
that the audio has been dithered.

My guess is that Audacity has a default internal format of normalized
floats (ie range [-1.0, 1.0]) and that by default it dithers when saving
internal floating point data to a 16 bit WAV files.

In general, when converting from float to 16 bit int, dither should be
applied. However, one case where it makes sense not to dither is when
you load 16 bit integer audio data into a float internal format and then
save it back to a 16 bit integer format. In other words, the int16_t ->
float -> int16_t chain should be lossless.

If you are interested in lossless conversion from integer format to
integer format, you should try sndfile-convert which (obviously)
handles all the formats supported by libsndfile. The sndfile-convert
program by default does all conversions in 32 bit integer format
(hence int16_t -> int16_t is lossless) and only uses floating
point if either the source or destination format is float.


And this is what Fons (another respected audio programmer) wrote, after a detailed analysis of the files.

I had a closer look. As posted before the difference is
noise, numerically generated (not analog) and strangely
lowpass filtered: -3dB at around 8 Khz going down to
-20dB at Fs/2.

The noise power is -81dB FS, which is at least 14 dB
higher than would be required for any form of dithering
of 16-bit PCM. At least 2.5 bits are lost. No idea what
Audacity is trying to do, but it looks quite wrong.

Clearly now there is a problem with Audacity itself, and I have thrown it back to the list to ask where we should go from here. I don't know how serious this is in reality, or what, if any solutions there are.

However there is good news. Using sndfile-convert with its default settings from a terminal window, as Erik suggests, gives repeatable identical files, as confirmed by md5sum signatures and binary comparisons. What is more, still using default settings going wav -> flac -> wav also gives repeatable identical files.

sndfile-convert can handle a wide range of file lossless formats but not, unfortunately, the lossy ones, but I shall certainly use it in future for any wav or flac work.

As usual for Linux stuff for more info open a terminal window and type:
  sndfile-convert --help

I think the moral to be drawn here (if any) is that where you suspect 'odd' conversion behaviour, suspect the the upper layers of programming first, rather than the nuts & bolts stuff.

A bone-head tutorial on how to re-configure Audacity to employ sndfile-convert as it's default program would be a great boon to hosers who have not yet learned how to work with the Linux terminal. Have mercy... ::)