Amiga.org
Operating System Specific Discussions => Amiga OS => Amiga OS -- Development => Topic started by: Thorham on June 04, 2010, 07:17:48 PM
-
Hi,
If I take a 16 bit audio file, and chop of the bottom two bits from each sample, it seems that the sound quality (when playing back in 14 bit on the miggy) isn't reduced. What I want to know is simple: Is there really no audible difference, or is there still some?
Any thoughts are appreciated :)
-
I used to use my Amiga 1200 for ALL audio, and I've had it connected to very nice, very accurate speakers and a very clean amplifier at a time when one of my roommates was an audiophile. After explaining to him that bits are bits (why, for instance, the $5000 CD players which do 50x oversampling and "bathe the underside of the disc in blue laser light" and so on give you the exact same bits as a $39 CD player, and that it's just the digital to analog convertors which matter), he began to focus more on the quality and merits of various DACs. He then proceeded to help me test various types of DACs, including the ones in my Amiga.
14 bit on the Amiga really doesn't make a very audible difference. If anything, it slightly softens the sound, but this could also be due to the Amiga audio hardware (note I always had the filters off). But that doesn't mean you don't get accurate treble; rather, you get good treble that doesn't sound artificial as it sometimes does when all of your gear is clean and you put in a test CD.
Note that Amiga hardware with the filters off and good quality sound will always exceed the quality you can get from most audio formats. mp3s, for instance, make me cringe - the quality difference is tremendous. Most people can't tell because they're used to mp3s, but I can easily hear the difference. Also, most compressed (filesize compression) formats also compress (audio compression) the audio levels, so a track from a CD compressed (file and audio) would often have all of the levels evened out, and this completely ruins certain kinds of music. Classical, for instance, will die from audio compression.
So if you have good quality, clean 16 bit audio files without audio or filesize compression and you convert them to 14 bit and play them on the Amiga with the filters off, you'll have audio quality which exceeds most consumer device outputs, in my opinion.
There are other factors - make sure your power supply is clean, and make sure you have good grounds so you don't need a ground loop isolator (which affects the quality of the sound), et cetera. An Amiga can sound quite good!
-
So if you have good quality, clean 16 bit audio files without audio or filesize compression and you convert them to 14 bit and play them on the Amiga with the filters off, you'll have audio quality which exceeds most consumer device outputs, in my opinion.
No conversion to 14-bit is necessary. The 14-bit replay routines do all that anyway.
-
I used to use my Amiga 1200 for ALL audio, and I've had it connected to very nice, very accurate speakers and a very clean amplifier at a time when one of my roommates was an audiophile.
That's cool, that will certainly sound great :)
14 bit on the Amiga really doesn't make a very audible difference.
Sorry, but I've asked the question in an unclear way :o What I'm really asking is if a 14 bit player actually chops off bits 0 and 1 of each sample. I should've been more clear here :o
Classical, for instance, will die from audio compression.
Yeah, I know, just try Classical internet radio stations. The music itself is still very enjoyable, but the quality certainly isn't optimal.
So if you have good quality, clean 16 bit audio files without audio or filesize compression and you convert them to 14 bit and play them on the Amiga with the filters off, you'll have audio quality which exceeds most consumer device outputs, in my opinion.
Cool :) This is about a little project of mine. I'm using 16 bit, 48Khz stereo WAVs which I've down sampled to 28KHz using Sox, so the quality is quite good.
There are other factors - make sure your power supply is clean, and make sure you have good grounds so you don't need a ground loop isolator (which affects the quality of the sound), et cetera. An Amiga can sound quite good!
Oh man, everything here is ungrounded :(
No conversion to 14-bit is necessary. The 14-bit replay routines do all that anyway.
Ah, yes, that's what I wanted to hear, thanks :) The bit chopping I'm doing is for a lossless compression algorithm I'm writing. Of course bit chopping isn't lossless, but the resulting data is meant to be replayed on an Amiga in 14 bit, so effectively it should be lossless.
Nice to have this and some other things cleared up, thanks guys :)
-
What I'm really asking is if a 14 bit player actually chops off bits 0 and 1 of each sample.
No, not if the output uses the calibration table. In this case, the conversion is not linear and all the 16 bits are used for mapping the input to the 2 x 8 bit output channels (one at full volume, the other at volume 1).
-
Yeah, I know, just try Classical internet radio stations. The music itself is still very enjoyable, but the quality certainly isn't optimal.
I think jonklos was reffering more to so called "loudness war" in this case - i.e. severely reducing (compressing) dynamics by evening out the volume levels of different sections of a track (http://www.youtube.com/watch?v=xkkqsN69Jac&feature=fvst). Not that classical music would be the usual victim...
-
Reducing bit resolution doesn't do anything to the frequency range, means bass and trebble are not affacted.
2 bits less raise the quantisation noise about 4 times. Nothing more, but nothing less. Depending on the audio material, this can get audible of course, especially in very silent passages. For "loud" regular pop music, this won't be much audible though. If the dynamic range is well used, quantisation noise starts to be audible roughly at 12 bits.
On the other hand, if you want to compress the audio data, removing 2 bits of 16 is ridiculous. Better use 8 or 4 bit ADPCM, the 8bit variant will sound almost as 16bits, and halves the data size. 4 bit is good for voices, but can get problematic for complex music with a lot of high frequencies, then it starts to crackle a bit.
-
No, not if the output uses the calibration table. In this case, the conversion is not linear and all the 16 bits are used for mapping the input to the 2 x 8 bit output channels (one at full volume, the other at volume 1).
Perhaps the table includes the bit reduction?
I think jonklos was reffering more to so called "loudness war" in this case - i.e. severely reducing (compressing) dynamics by evening out the volume levels of different sections of a track (http://www.youtube.com/watch?v=xkkqsN69Jac&feature=fvst). Not that classical music would be the usual victim...
Ugh, that's a pretty bad practice (and I'm not even an audiophile) :(
Reducing bit resolution doesn't do anything to the frequency range, means bass and trebble are not affacted.
2 bits less raise the quantisation noise about 4 times. Nothing more, but nothing less. Depending on the audio material, this can get audible of course, especially in very silent passages. For "loud" regular pop music, this won't be much audible though. If the dynamic range is well used, quantisation noise starts to be audible roughly at 12 bits.
Very interesting :)
On the other hand, if you want to compress the audio data, removing 2 bits of 16 is ridiculous.
It's for an audio compressor I'm writing for a little project of mine, the bit chopping isn't used by itself.
Better use 8 or 4 bit ADPCM, the 8bit variant will sound almost as 16bits, and halves the data size. 4 bit is good for voices, but can get problematic for complex music with a lot of high frequencies, then it starts to crackle a bit.
ADPCM is a last resort in case I can't fit all the music on a single CD. I really want lossless 14 bit compression.
-
Just to clarify: by "compressing" you mean data compression, not dynamic range compression as a DSP effect?
Looseless:
If you compress looseless, you can not guarantee any compression rate. If the data to compress doesn't show any redundancy, no compression is possible. (like zipping a zip file)
In reality, true looseless compression achives about 2 times compression rate on average complex audio data.
Loosless compression almost always works with a predictor (a function that predicts the next sample from the previous samples), calculates the error of the prediction (the delta between the predicted and real value) and zipps the result. E.g. like Flac for audio or PNG for images.
DPCM or ADPCM is not looseless. ADPCM8 uses 8 bits to store the error of the prediction and unpacks into 16bit samples. This is loossy, but almost not audible. ADPCM4 uses 4 bit per 16bit sample, and is audible, depending on the material of course. Music will sound a bit harsh in high frequencies. MP3 uses 1-2bit per 16bit sample and designed for music. So if you want to pack music, this should be your first choise. If you use 192 or 256 kbps and a decend encoder like LAME, the compression is almost not audible. Better than 12bit or ADPCM4. ADPCM8 is much faster to encode/decode than MP3, but packs only 1:2.
-
Just to clarify: by "compressing" you mean data compression, not dynamic range compression as a DSP effect?
Yes, I mean data compression. Adding audio effects is absolutely out of the question, even if it improves compressibility.
Loosless compression almost always works with a predictor
Yes, but I've been unable to come up with a reasonable predictor (it's really easy to make a bad one :lol:), so I'm experimenting whith the deltas of the samples, or, to be more accurate, the deltas between the deltas. I'll try those predictors again, but I just wanted to try something else after failing initially.
DPCM or ADPCM is not looseless.
Yep, I know, hence the 'last resort' remark ;) This might be a last resort because I'm trying to fit 1.67 gigs of audio on a CD with a few megs to spare (for code and graphics) :lol:
Anyway, I want to store the audio without any loss, and while mp3 could be useful, it's to heavy for the target hardware (as low as possible).
-
Yes, I mean data compression.
http://flac.sourceforge.net/
You're unlikely to beat flac with your custom "predictors".
-
http://flac.sourceforge.net/
You're unlikely to beat flac with your custom "predictors".
Sorry, but FLAC isn't an option, because it's to heavy. Someone is doing a FLAC decoder for Amiga, and while it's quite optimized, it still uses about 90% CPU time on a 50Mhz '030. This is sadly much too heavy. Not a problem for me personally, but I want this to work on lesser CPUs than said CPU.
Also, I'm not using predictors at all right now (tried something quick, and failed :lol:), just deltas ;)
-
I wrote a low CPU use audio compression format. Encoding was relatively expensive though. Essentially, it split 16-bit audio into frames, found the best N delta values (depending on the # of bits to encode) for that frame (the expensive bit).
It supported, if I recall correctly, between 2-5 bit encoding. It was lossy but didn't sound too awful.
The idea was that each frame could be independently replayed.
Each compressed frame contained a 16-bit word that had some encoding flags, followed by a 16-bit signed sample (2 for stereo and so on) that comes unmodified from the input frame and serves as the starting point. After that, the next 4/8/16/32 16-bit words contain the best 4/8/16/32 delta values as determined by the encoder.
The remainder of the compressed frame was simply bitfield lookups into that table. Stereo data was interleaved IIRC. For 2-bit encoding, each 16-bit word contained 8 entries, 3-bit encoding stored 5 entries (LSB aligned), 4-bit encoding stored 4 entries, 5-bit stored 3 entries (LSB aligned).
The replay algorithm simply takes the start value and then extracts each field value, looks up the delta value from the table and adds it to the current value to recreate the next sample.
I am sure I still have the sources somewhere.
-
^ I never implemented it, but the replay routines were aimed at keeping the sample data compressed in memory and simply generating output directly to a mixing stage. In order to support volume, each frame would have it's start sample and delta table attenuated to the current volume, resulting in an output stream at the correct level of attenuation.
This meant that the mixing routine only ever had to add the output sample values. Overall the aim was to avoid excessive multiplication or table lookup.
-
Loosless compression will be always too expensive for your purpose, since it always involves some kind of zip compression.
MP3 is too expensive as well, otherwise this would be the best choise.
What's left is only DPCM or ADPCM. Those are relatively cheap to decode in linear complexity.
If you are calculating deltas, you are already using a predictor. The predictor says "the next sample will have the same value as the current one". Which isn't a bad predictor at all. This gives you already 80% of the quality.
ADPCM is significantly better than DPCM, since the value range is adaptive to the current audio data. But I would say roughly 4 times more expensive to calculate.
If this is a game project, I would also consider using a lower samplingrate and maybe mono, because nobody would recognize your nearly-looseless afforts anyway. 44100Hz/16bit/stereo is already quite expensive to sqeeze through the Zorro Slot without any decoding involved.
In 16kHz/8bit/mono you could also consider to use mp3, and decode it "offline" into a ram buffer. If the songs are not soo long, this should be affordable. 8bit isn't too bad either if you do proper dithering. (same effect as with pictures, when doing dithering even 12bit color looks acceptable)
-
It supported, if I recall correctly, between 2-5 bit encoding. It was lossy but didn't sound too awful.
Cool, but I really want lossless encoding. Lossy encoding is simply not an option.
Each compressed frame contained a 16-bit word that had some encoding flags, followed by a 16-bit signed sample (2 for stereo and so on) that comes unmodified from the input frame and serves as the starting point. After that, the next 4/8/16/32 16-bit words contain the best 4/8/16/32 delta values as determined by the encoder.
The remainder of the compressed frame was simply bitfield lookups into that table. Stereo data was interleaved IIRC. For 2-bit encoding, each 16-bit word contained 8 entries, 3-bit encoding stored 5 entries (LSB aligned), 4-bit encoding stored 4 entries, 5-bit stored 3 entries (LSB aligned).
The replay algorithm simply takes the start value and then extracts each field value, looks up the delta value from the table and adds it to the current value to recreate the next sample.
I am sure I still have the sources somewhere.
There are some interesting ideas in here that I may be able to use in my compressor, thanks :)
Loosless compression will be always too expensive for your purpose, since it always involves some kind of zip compression.
Not necessarily. I'm first going to try the simplest of Huffmann implementations. This is very cheap. Basically, when I calculate the deltas, and the deltas of the deltas, I simply store the sign sepperately, and make negative values positive.
This has the effect of creating easily compressed delta values. They're currently stored in the following way: I add two bits per delta. These bits tell how many bits the delta contains. The bit lengths are simply 4, 8, 12 and 16. This could be improved by setting those ranges in a better way. After this, a very simple implementation of Huffmann encoding can be used.
The sign data is stored in 'bit toggled' form (each time a bit in the data is different than the previous bit, a one is written out, for repeating bits a zero is written out).
I'm hoping that those signs can be reasonably well compressed with a simple Huffmann encoder.
All this is very cheap to decode, and should give reasonable compression rates.
If you are calculating deltas, you are already using a predictor. The predictor says "the next sample will have the same value as the current one". Which isn't a bad predictor at all. This gives you already 80% of the quality.
Do you mean sound quality? If that's the case, than that isn't correct. The process I use is lossless. In case it's not, than I don't understand what you mean :D
If this is a game project, I would also consider using a lower samplingrate and maybe mono, because nobody would recognize your nearly-looseless afforts anyway. 44100Hz/16bit/stereo is already quite expensive to sqeeze through the Zorro Slot without any decoding involved.
In 16kHz/8bit/mono you could also consider to use mp3, and decode it "offline" into a ram buffer. If the songs are not soo long, this should be affordable. 8bit isn't too bad either if you do proper dithering. (same effect as with pictures, when doing dithering even 12bit color looks acceptable)
No, it'a not a game project. It's for music CD, something like an old-school music disk, but a lot more extensive.
That's the main reason for me wanting maximum quality. Also, there are already two lossy factors: The downsampling from 48Khz to 28Khz and playing back on a miggy so that two bits are lost. I really hope I don't have to use lossy compression :(
-
Again, loosless is expensive, and gives you in average 1:2 compression. If you even consider a huffman decoder and some nifty bit ticks, then you can affort ADPCM, this is cheaper. ADPCM8 can guarantee 1:2 compression and sounds almost as good as PCM16. Given your low-fi conditions (Paula14/28kHz), the quality loss is absolutely neglegtible.
You could also encode the stereo channel with 4bit, then you end up in 12 bits per stereo sample instead of 32bit, not too bad.
You should also consider 24kHz, because of the integer ratio of downsampling. The downsampling in your case has the biggest quality impact, much more than ADPCM8 would harm to your data. All this of course depends on the actual audio data.
If this is sampled MOD music, produced with 8kHz samples in 8bit, all those assumptions might be wrong. I assume high-fidelity random pop music as you can hear in the radio.
-
Cool, but I really want lossless encoding. Lossy encoding is simply not an option.
Also, there are already two lossy factors: The downsampling from 48Khz to 28Khz and playing back on a miggy so that two bits are lost. I really hope I don't have to use lossy compression
Depending on your algorithm, your loss error might be limited to the bits you can't replay anyway. Also, consider how human hearing works. For example, you can't perceive the same degree of error in a quiet sound immediately after a loud one.
Experiment, I say.
-
Anyway, if you are curious, I have dug around my old HD and found it.
I've uploaded the codec binary for you to play with. It only imports/exports AIFF 16-bit (mono and stereo supported).
I've encoded a short section of 44.1kHz stereo music, provided the compressed version and the decoded version for your appraisal. The default encode options were used which IIRC are 4-bit, frame length 256. This gives a compression of about 3.5:1.
Speech, with properly gated silences can compress much better, since an entire silence frame can be encoded as a single word, more or less.
An interesting side effect of the codec is that it is "first time lossy" only. If you re-encode the decoded output, except in very rare cases, will output the same compressed interpretation as the first pass of the original did.
If you do a waveform subtraction of the decoded from the original, you'll see what has been thrown away (and it is quite noticeable), yet it's a lot harder to perceive when just listening.
http://extropia.co.uk/_temp/xdac_codec.zip (http://extropia.co.uk/_temp/xdac_codec.zip)
-edit-
I think the codec has been compiled with FPU support, which isn't used in the codec but may be used when interpreting the AIFF sample rate (which is stored as an 80-bit long float)
-
Again, loosless is expensive, and gives you in average 1:2 compression. If you even consider a huffman decoder and some nifty bit ticks, then you can affort ADPCM, this is cheaper. ADPCM8 can guarantee 1:2 compression and sounds almost as good as PCM16. Given your low-fi conditions (Paula14/28kHz), the quality loss is absolutely neglegtible.
You sure like your lossy encoding :lol: Anyway, I've already said that my lossless encoder isn't very heavy, certainly fast enough to decode on an A1200 with some fastmem in the trapdoor slot. But enough of that ;) Karlos has uploaded a nice archive for me, and I must say that the lossy encoding he uses sounds quite good, actually :) In other words, lossy encoding is now a serious option, rather than just a last resort.
You could also encode the stereo channel with 4bit, then you end up in 12 bits per stereo sample instead of 32bit, not too bad.
I'll try that.
You should also consider 24kHz, because of the integer ratio of downsampling. The downsampling in your case has the biggest quality impact, much more than ADPCM8 would harm to your data.
I'm using a high quality algorithm from Sox on the peecee. Even when halving the sample rate, just taking the average may not be enough. I've used cheap methods, and they're bad :)
All this of course depends on the actual audio data.
If this is sampled MOD music, produced with 8kHz samples in 8bit, all those assumptions might be wrong. I assume high-fidelity random pop music as you can hear in the radio.
The music is all the music from Final Fantasy 10, ripped to PSF format. This is the original, tracked audio data, and includes the player code from the game (!). PSF players 'simply' emulate Playstation 1 and 2 audio hardware and CPU (and various other bits, of course), producing the original sound.
Depending on your algorithm, your loss error might be limited to the bits you can't replay anyway. Also, consider how human hearing works. For example, you can't perceive the same degree of error in a quiet sound immediately after a loud one.
I didn't know that. Very interesting :)
Experiment, I say.
Absolutely, and I'm also not even remotely done with my lossless experiments, yet.
Anyway, if you are curious, I have dug around my old HD and found it.
I've uploaded the codec binary for you to play with. It only imports/exports AIFF 16-bit (mono and stereo supported).
Thanks :)
I've encoded a short section of 44.1kHz stereo music, provided the compressed version and the decoded version for your appraisal. The default encode options were used which IIRC are 4-bit, frame length 256. This gives a compression of about 3.5:1.
Again, thanks :) Sounds good! I expected a lot worse, to be honest, and now that I've heard this kind of lossy compression, I must sat that it has definitively become a serious option to consider for me. However, I do hear the difference, unfortunately, and that's without high end equipment, so I would need a solution for that.
-edit-
I think the codec has been compiled with FPU support, which isn't used in the codec but may be used when interpreting the AIFF sample rate (which is stored as an 80-bit long float)
Oh, good to know, I don't have an FPU on my Blizzard '030. Guess I'll use WinUae, then, no problem. If I'm going to use this, then I have 1.67 gigabytes to encode, and this would take forever on my miggy anyway.
-
The codec tool is very old. Not sure if I even implemented proper streaming to/from disk with it. When I find the source code (alas it wasn't in the same place as the bin), I'll put it up.
Run the codec without any parameters to see what options it takes.
-snd is to specify the aiff source for compression, target for decompression
-xdac is to specify xdac target for compression, source for decompression
-encode - pretty obvious, encodes the aiff to the xdac target (default is to decode)
-fsize to set the framesize. Default is 256 IIRC, think it maxes out at 1024. Longer frames give better compression, at the expense of quality.
-brate to set the maximum bitrate for encoding. This is not really a bitrate value in the mp3 sense but the maximum number of bits (thus delta table size) per frame to use.
Note that the compressor detects those cases in which there are less delta values to store than the current bit rate specifies and reduces those frames accordingly, with silence being compressed out all together. Doesn't happen in music much, but is common in speech.
However, I do hear the difference, unfortunately, and that's without high end equipment, so I would need a solution for that.
Try encoding with -brate 5 and -fsize 128. That should produce better quality, at the expense of file size.
-
I think you should first define how much memory you want to spend to store the music, and from that estimate the compression ratio that you need.
Given the compression ratio and the CPU power, we can evaluate what your options are.
A fact is: on avarage music, looseless compression will give you approximatly 2:1. You cannot break this barrier, otherwise you would be a good candiate for the Nobelprize in natural sience. ;-)
An experience: loosless doesn't necessarily mean the result sounds worse than the original. Looseless just says that the data is not reproduced bit-identical, like one needs for exact data like exacutables. Many people like a moderate mpeg compression on audio, because a lot of "garbadge" gets filtered out and the result is somewhat easier and more transparent to listen.
I would always prefere ADPCM8 over looseless, because the difference is not audible to humans and th compression ratio is predictable fix 2:1.
-
@Karlos: Thanks for the explanation.
Try encoding with -brate 5 and -fsize 128. That should produce better quality, at the expense of file size.
I'll try that.
I think you should first define how much memory you want to spend to store the music, and from that estimate the compression ratio that you need.
Given the compression ratio and the CPU power, we can evaluate what your options are.
Okay, here goes:
There's 93 WAVs which use up 1.67 gigabytes. Many of them (80+) are looped, and thus have repeating data. The repeats may take up 25% to 50% of the data. It's probably less than 50%. The problem with this that although the loopings can be chopped off and done in software easily enough, it has to be done by hand, and for so many tracks this is a downright pain in the backside, and it's certainly something I don't want to have to do if it's avoidable.
I want to store them on a CD with a couple of megabytes to spare for code an graphics (one megabyte will probably be more than enough).
The CPU I'm working on is a 50 mhz '030, but the lowest target should be something like an A1200 with some fastmem in the trap door. Or, at max, a 28 Mhz '020 board (a Blizzard, I believe).
It would be great if I could find the loop times somewhere, because then this would be a done deal.
A fact is: on avarage music, looseless compression will give you approximatly 2:1. You cannot break this barrier, otherwise you would be a good candiate for the Nobelprize in natural sience. ;-)
Don't you mean computer science ;) Somehow I doubt nature has set this ratio to 2:1, though, and I like to believe it can be done, but that's just me :lol:
An experience: loosless doesn't necessarily mean the result sounds worse than the original. Looseless just says that the data is not reproduced bit-identical, like one needs for exact data like exacutables.
That's a good point, I never considered that.
I would always prefere ADPCM8 over looseless, because the difference is not audible to humans and th compression ratio is predictable fix 2:1.
While ADPCM and similar lossy techniques are certainly an option now, ADPCM8's 2:1 ratio still isn't good enough, I'm afraid :( Karlos's method, however, might be the solution to this problem.
-
I don't currently have the source code handy (I'll have to dig through a lot of backup cd's), but I do remember the technique well enough:
1) Choose a frame length and bit rate (say 256 samples / 4-bit for example)
2) For one complete frame of audio, transform the samples into a sequence of delta values, leaving the first sample as is (ie, in a mono stream with frame length 256, you now have 1 sample and 255 subsequent delta values). Another way of looking at it is that you have 256 delta values from 257 samples, where sample 0 had the value 0.
Note that for a stereo stream, remember that the source samples are usually interleaved so remember that when performing this step. Unless you plan to do some mid + side encoding, treat them separately.
3) Now find all the unique delta values for your frame and the popularity of each one. Don't include the first one here. My method simply did a qsort() and then walked through them counting duplicates as it went. Not particularly fast, but for encoding, who cares?
4) Use a reduction algorithm (I tried several) to find the best fit 2^N delta values for the above set, where N is your "bit rate".
5) Store the first delta value (which is the same as the first sample in the original frame) exactly (or pair of samples for a stereo stream) as 16-bit signed data.
6) Store these best fit delta values as 16-bit signed data. This is now your delta table with which to encode the rest of the frame.
7) Starting with your unblemished "start" sample, for each successive sample in the original frame, choose the delta value from your table that gets you nearest to that sample without clipping. Store the index of the used delta value as a bitfield, packing successive bitfields into 16-bit words.
8) Repeat from (7) until you've encoded the entire frame.
If I remember correctly, my compressed frame, now looks something like this, assuming a mono source with 4-bit encoding
word
000: [ frame header word ]
001: [ start sample ]
002: [ best fit delta 0 ]
003: [ best fit delta 1 ]
004: [ best fit delta 2 ]
...
016: [ best fit delta 14 ]
017: [ best fit delta 15 ]
018: [ev 004][ev 003][ev 002][ev 001]
019: [ev 008][ev 007][ev 006][ev 005]
...
081: [ empty ][ev 255][ev 254][ev 253]
ev N: encoded delta value for original sample N. Note we don't bother encoding the first (zeroth) sample as we already have it. Thus the last bitfield is always empty in a word aligned stream such as above. For 3/5-bit encoding, this may or may not always be true.
A stereo stream with the same frame length is encoded as follows:
word
000: [ frame header word ]
001: [ start sample R ]
002: [ start sample L ]
003: [ best fit delta 0 ]
004: [ best fit delta 1 ]
005: [ best fit delta 2 ]
...
017: [ best fit delta 14 ]
018: [ best fit delta 15 ]
019: [evL 002][evR 002][evL 001][evR 001]
020: [evL 004][evR 004][evL 003][evR 003]
...
021: [ empty ][ empty ][evR 127][evL 127]
Notice that the encoder regards frame length as total number of samples, it doesn't consider a stereo frame of length 256 as having 256 sample pairs.
Decoding the above data is so easy that even a vanilla 68000 can do it. Assuming you have a compressed frame in memory, you simply:
1) set a pointer into the best fit area
2) set your current sample value to the start value
3) write your current value to the output
4) extract the next ev bitfield from the compressed block
5) look up the delta value indexed by your value from (4)
6) add it to the current sample
7) repeat from 3 until the entire frame has been decoded.
-
@Karlos: Thanks for the explanation.
It would be great if I could find the loop times somewhere, because then this would be a done deal.
You could write a tool that tries to find the loop points, but that would probably take longer than editing them manually.
Don't you mean computer science ;) Somehow I doubt nature has set this ratio to 2:1, though, and I like to believe it can be done, but that's just me :lol:
No, I do mean natural sience. And unfortunately yes, nature has set this to 2:1. Without extra World-knowledge, the entropy of an average music signal in time domain is roughly 0.5, means 1bit gives 0.5bit of information.
You will never ever be able to compress better than 2:1. The sooner you accept this, the better for your precious spare time.
Check out this page:
http://wiki.hydrogenaudio.org/index.php?title=Lossless_comparison
Many wise men have worked on looseless codec. Here some examples:
FLAC 58.70%
WavPack 58.0%
TAK 57.0%
Monkey's 55.50%
OptimFROG 54.70%
ALAC 58.50%
WMA 56.30%
So dont fool yourself by thinking you can beat this.
While ADPCM and similar lossy techniques are certainly an option now, ADPCM8's 2:1 ratio still isn't good enough, I'm afraid :( Karlos's method, however, might be the solution to this problem.
If you need more than 2:1, loosless is out of the game anyway. Lossy is your only option.
The best is mpeg, e.g. mp3 can easily reach 10:1 without significant degradation. With ADPCM, you could get ~3:1 I'd say (ADPCM8 for the mid channel, and ADPCM4 for the stereo channel). ADPCM is fast and easy to implement compared to mpeg.
-
I don't currently have the source code handy (I'll have to dig through a lot of backup cd's)
Oh no, don't search for it, I might not use it, and I much prefer a good explanation anyway. Usually, even when I don't end up using something, an explanation always contains interesting and useful ideas, and is thus much more enlightening than source code (where the source code is basically stripped to what's needed, and that's then used as is).
1) Choose a frame length and bit rate (say 256 samples / 4-bit for example)
2) For one complete frame of audio, transform the samples into a sequence of delta values, leaving the first sample as is (ie, in a mono stream with frame length 256, you now have 1 sample and 255 subsequent delta values). Another way of looking at it is that you have 256 delta values from 257 samples, where sample 0 had the value 0.
Note that for a stereo stream, remember that the source samples are usually interleaved so remember that when performing this step. Unless you plan to do some mid + side encoding, treat them separately.
3) Now find all the unique delta values for your frame and the popularity of each one. Don't include the first one here. My method simply did a qsort() and then walked through them counting duplicates as it went. Not particularly fast, but for encoding, who cares?
4) Use a reduction algorithm (I tried several) to find the best fit 2^N delta values for the above set, where N is your "bit rate".
5) Store the first delta value (which is the same as the first sample in the original frame) exactly (or pair of samples for a stereo stream) as 16-bit signed data.
6) Store these best fit delta values as 16-bit signed data. This is now your delta table with which to encode the rest of the frame.
7) Starting with your unblemished "start" sample, for each successive sample in the original frame, choose the delta value from your table that gets you nearest to that sample without clipping. Store the index of the used delta value as a bitfield, packing successive bitfields into 16-bit words.
8) Repeat from (7) until you've encoded the entire frame.
If I remember correctly, my compressed frame, now looks something like this, assuming a mono source with 4-bit encoding
word
000: [ frame header word ]
001: [ start sample ]
002: [ best fit delta 0 ]
003: [ best fit delta 1 ]
004: [ best fit delta 2 ]
...
016: [ best fit delta 14 ]
017: [ best fit delta 15 ]
018: [ev 004][ev 003][ev 002][ev 001]
019: [ev 008][ev 007][ev 006][ev 005]
...
081: [ empty ][ev 255][ev 254][ev 253]
ev N: encoded delta value for original sample N. Note we don't bother encoding the first (zeroth) sample as we already have it. Thus the last bitfield is always empty in a word aligned stream such as above. For 3/5-bit encoding, this may or may not always be true.
A stereo stream with the same frame length is encoded as follows:
word
000: [ frame header word ]
001: [ start sample R ]
002: [ start sample L ]
003: [ best fit delta 0 ]
004: [ best fit delta 1 ]
005: [ best fit delta 2 ]
...
017: [ best fit delta 14 ]
018: [ best fit delta 15 ]
019: [evL 002][evR 002][evL 001][evR 001]
020: [evL 004][evR 004][evL 003][evR 003]
...
021: [ empty ][ empty ][evR 127][evL 127]
Notice that the encoder regards frame length as total number of samples, it doesn't consider a stereo frame of length 256 as having 256 sample pairs.
Decoding the above data is so easy that even a vanilla 68000 can do it. Assuming you have a compressed frame in memory, you simply:
1) set a pointer into the best fit area
2) set your current sample value to the start value
3) write your current value to the output
4) extract the next ev bitfield from the compressed block
5) look up the delta value indexed by your value from (4)
6) add it to the current sample
7) repeat from 3 until the entire frame has been decoded.
That's quite clear, and very interesting, thanks a tonne, much appreciated :)
You could write a tool that tries to find the loop points, but that would probably take longer than editing them manually.
Yes, it would, even a quick and dirty one. But at least it would be much less boring, though ;)
No, I do mean natural sience. And unfortunately yes, nature has set this to 2:1. Without extra World-knowledge, the entropy of an average music signal in time domain is roughly 0.5, means 1bit gives 0.5bit of information.
Rreally? But doesn't entropy mostly apply to entropy coders?
You will never ever be able to compress better than 2:1. The sooner you accept this, the better for your precious spare time.
I can never accept these things. And always have to challenge them :lol: And don't worry about my precious time, because I like spending my free time on things like this ;) Even when this sort of thing fails (which tends to happen most of the time, of course :lol:), I've still learned something. Going through these kinds of failures is better than simply taking someones word for it (too easy) ;)
So dont fool yourself by thinking you can beat this.
The point in trying is that people may have missed things. It happens. Also, if no one challenges existing methods, then in my opinion there's no progress. Although I'm certainly not kidding myself in believing that I can beat these ratios, I also have to say that I won't know until I try. While I probably won't beat them, half the fun is in trying :)
If you need more than 2:1, loosless is out of the game anyway.
Probably ;)
The best is mpeg, e.g. mp3 can easily reach 1:10 without significant degradation. With ADPCM, you could get 3:1 I'd say. ADPCM is fast and easy to implement compared to mpeg.
If I'm forced to use lossy compression, I think I'll first try Karlos's method. Seem easy enough to implement, so I'll experiment with that first, or better yet, compare it to ADPCM and see wich produces the best quality at the right compression rate.
-
Regarding my old code versus ADPCM, ADPCM is probably better but is more expensive to decode and less fault tolerant.
I designed the above codec for a very specific purpose. I wanted a sound format, that would allow a mixing engine to decode N streams of compressed audio straight from memory with as little CPU usage as possible. Using a frame based mechanism helped in the following ways:
1) Mixing generally works on taking a "packet" of sound data and mixing it into a buffer. Having your sound in discrete chunks already facilitates this.
2) It's relatively cheap to apply a volume to the compressed data. In essence, you only do as many multiplications as you have start samples / delta values. By pre-multiplying (a copy of) these data by the desired volume, you save having to calculate the volume of every output sample.
On my 68040, hand optimised decode routines for mono sound were arguably faster than replaying uncompressed audio. As silly as it sounds, it's true. The reason being that for all the extra shift/add work we are doing (which is a tiny loop in reality and fits cache even on 020), we are doing far less memory reading for the amount of data we are spitting back out.
-
If I'm forced to use lossy compression, I think I'll first try Karlos's method. Seem easy enough to implement, so I'll experiment with that first, or better yet, compare it to ADPCM and see wich produces the best quality at the right compression rate.
One thing to consider with the method I used is the choice of reduction algorithm for finding the "best fit" delta table. For any given frame length/bit-depth, how you choose the values for this table will have the most obvious effect on quantization error and thus the overall quality.
-
> Rreally? But doesn't entropy mostly apply to entropy coders?
Everybody has to obey the laws of nature. If you want to or not.
> The point in trying is that people may have missed things.
You didn't get the point. It is a law of nature. Unless you are Q from Startrek, you won't be able to change this. It is just not that intuitive like the apple that falls down from the tree, but it is the same thing.
Plus, it is quite pathetic to think that what hundreds of PhD-Level researchers achieved over decades can be wiped away by a hobbyist in a few afternoon-sessions without even undestanding the fundamentals of information theory.
I'll post you the code for my ADPCM4 implementation soon. It should be fast enough for a vanilla A1200, and doesn't need a lot of stuff around it, just one function to encode and one to decode.
You can use my Tool "AudioConverter" or Samplemanager to generate ADPCM files and listen to the result.
I am currently tuning some parameters to minimize the error, and adding stereo support.
Right now, ADPCM4 gives me an average error of ~300 quantisation steps of a 16bit sample. This is roughly like 8bit PCM, but the distribution of the errors is better.
If the material contains a lof of high frequencies, the errors go up, but are less audible.
If the material contains more low frequencies or is quieter, the error goes down. E.g. if your music fades out, there will be no audible noise like with 8bit PCM.
Karlos' algorithm follows a so called "code-book based" approach.
From what I can see it has several drawbacks:
1. On 256/4bit is has more than 25% data overhead because he stores individual codebooks for each frame. Means instead of 4:1, you will get ~3:1.
2. How to find the "best" delta representants is not well defined and might need a lot of experimenting to find the optimal algoritm.
3. The encoder has a very high complexity because it needs to do vector quantisation. Luckily only for the encoder.
4. Doesn't make use of the assumption that the left and right channel of a stereo signal are correlated.
5. "Wastes" precious 4 bits in the 256/4bit case ;-)
6. The choosen Deltas may cause an error of up to 4096 quantisation steps (=reduces to 4bit PCM quality) in the worst case. However, very unlikely of course, but unlike in ADPCM, the error is not correlated with the high frequencies, so the error is not "masked".
> Regarding my old code versus ADPCM, ADPCM is probably better but is more expensive to decode and less fault tolerant.
Decoding is cheaper than ADPCM yes, but why less fault tolerent? Because you have a "sync" Sample at the beginning of a block? ADPCM can be "resetted" every N samples too. It wouldn't even need an explicit sync value.
-
Karlos' algorithm follows a so called "code-book based" approach.
From what I can see it has several drawbacks:
1. On 256/4bit is has more than 25% data overhead because he stores individual codebooks for each frame. Means instead of 4:1, you will get ~3:1.
It's actually ~3.5:1. If you download the zip file, you'll see for yourself.
2. How to find the "best" delta representants is not well defined and might need a lot of experimenting to find the optimal algoritm.
Absolutely, which is why I wanted to dig out the source.
3. The encoder has a very high complexity because it needs to do vector quantisation. Luckily only for the encoder.
It isn't massively complex and it could certainly be implemented more simply than I did. Once it was working satisfactorily for my needs, I didn't bother improving it, since playback was my main concern.
4. Doesn't make use of the assumption that the left and right channel of a stereo signal are correlated.
That's not actually true. In the stereo case, there is still only one delta table derived from both channels. The independent variation of left and right will produce similar spread of delta values when there is a strong correlation between them. An advantage here is that the correlation of delta value spread isn't really affected by phase differences between the channels.
Experimentation with mid and side band encoding did not produce any real difference from a QSNR perspective.
5. "Wastes" precious 4 bits in the 256/4bit case ;-)
Yeah, you got me there. Of course, a modified algorithm would simply pack an extra source sample into that and live with odd sized frames. I just happened to require an arrangement that decompressed an even number of samples per frame.
6. The choosen Deltas may cause an error of up to 4096 quantisation steps (=reduces to 4bit PCM quality) in the worst case. However, very unlikely of course, but unlike in ADPCM, the error is not correlated with the high frequencies, so the error is not "masked".
> Regarding my old code versus ADPCM, ADPCM is probably better but is more expensive to decode and less fault tolerant.
Decoding is cheaper than ADPCM yes, but why less fault tolerent? Because you have a "sync" Sample at the beginning of a block? ADPCM can be "resetted" every N samples too. It wouldn't even need an explicit sync value.
My experience with ADPCM decode was that corrupt data in the compressed stream can (but won't necessarily) knock the decode out permanently from that point onwards. Explicit audio frames mean that at most only the remaining samples in the current frame will be corrupted.
-
Perhaps the title should be changed to "(Not quite so) Simple..." :)
-
It's actually ~3.5:1. If you download the zip file, you'll see for yourself.
original datasize =
samples*sizeof(word) =
256 * 2 =
512 Bytes
compressed datasize =
samples*sizeof(4bit) + table*sizeof(word) + startsample*sizeof(word) + header =
(256*0.5) + (16*2) + 2 + 2 =
164 Bytes
=> ratio = 512/164 = 3.1219...
It isn't massively complex and it could certainly be implemented more simply than I did.
Complexity in sense of information theory, not implemtation wise. Implementing VQ (at high complexity) is easy.
That's not actually true. In the stereo case, there is still only one delta table derived from both channels. The independent variation of left and right will produce similar spread of
Yes (phase) and no (correlation).
If you would encode both independently, you would need 2 tables. But you use only one which does not resolve the tiny differences between left and right. Actually it is very likely that the stereo difference is killed completely, because it needs to compete with the other deltas. So you are actually worse than 2 independent channels with 2 tables, which is again worse than 2 joint stereo channels.
> the correlation of delta value spread isn't really affected by phase differences between the channels.
This is cool, didn't think of the phase difference. You got me here ;-)
(but only if the phase is shifted consistantly accross all sinus waves, which is almost never the case in reality)
Experimentation with mid and side band encoding did not produce any real difference from a QSNR perspective.
Yes, not with this algorithm, because the sideband suffers to much from competing against the mid band channel. You would need 2 Tables. (can be used together, but "trained" seperately).
My experience with ADPCM decode was that corrupt data in the compressed stream can (but won't necessarily) knock the decode out permanently from that point onwards. Explicit audio frames mean that at most only the remaining samples in the current frame will be corrupted.
Nobody stopps you from putting ADPCM into chunks of N samples. If this should be a stream, you need to do this anyway, because otherwise you cannot join the stream at any position you want, which is the main point of being a "stream".
But putting ADPCM into chunks has zero overhead. You just reset the adaptation factor to something average all N samples. Of course, setting the factor to the best value is better and can be achieved by adding 1 extra byte each frame.
-
@Karlos
What about a competition? ;-)
Let Thorham choose a short sample as uncompressed 16bit .wav (lets say 10 secs).
(maybe with some music, fading, voice, sound effect etc.)
We can both compress and uncompress it again, and let him decide what sounds better?
(as a proof, we both must provide the compressed file too, of course).
-
@Karlos
What about a competition? ;-)
Let Thorham choose a short sample as uncompressed 16bit .wav (lets say 10 secs).
(maybe with some music, fading, voice, sound effect etc.)
We can both compress and uncompress it again, and let him decide what sounds better?
(as a proof, we both must provide the compressed file too, of course).
LOL!
I don't care enough really. I am sure ADPCM is capable of better compression/quality than my method, but you are welcome to try if it amuses you to do so :D
The only reason I mentioned it is that he was asking for a low CPU usage playback routine and I happened to have one I'd made earlier that will run on a 7MHz 68000 (though the codec application is foolishly compiled for 020/fpu) :)
-
C'mon, it is just for fun and curiosity. I implemented my own version of ADPCM4, which is much more lightweight than e.g. the G.711 standard implementation of ADPCM4. I'd like to see how it compares to other home-brewn codecs.
We could even messure encoding/decoding time of e.g. 10min of audio on a Classic Amiga.
-
original datasize =
samples*sizeof(word) =
256 * 2 =
512 Bytes
compressed datasize =
samples*sizeof(4bit) + table*sizeof(word) + startsample*sizeof(word) + header =
(256*0.5) + (16*2) + 2 + 2 =
164 Bytes
=> ratio = 512/164 = 3.1219...
I might have lied about stereo streams. Sample pairs may actually be treated as single entities, in which case, 256 sample pairs are compressed per frame, in which case it's 256*2*2 bytes of sample data. I wrote it in 1996 or so, so my memory is slightly hazy on the details. However:
-rwxr-xr-x 1 karlos karlos 1763382 2010-06-06 15:36 test_decoded.aiff
-rwxr-xr-x 1 karlos karlos 506328 2010-06-06 15:36 test.xdac
That's ~3.48:1 based on filesize alone
Also consider that the algorithm will happily use less than N bit per frame if it finds it can. Doesn't happen much in music, but speech is a different matter.
Until I dig out the source, it's hard to say.
-
> Rreally? But doesn't entropy mostly apply to entropy coders?
Everybody has to obey the laws of nature. If you want to or not.
Sure, but that doesn't mean I can't play with this.
Plus, it is quite pathetic to think that what hundreds of PhD-Level researchers achieved over decades can be wiped away by a hobbyist in a few afternoon-sessions without even undestanding the fundamentals of information theory.
Yes, but I don't think that. I know that I don't know much at all about this, and because of this, I also know it's likely that I'll fail to do it better. However, this doesn't mean that I have to take what all the PhD's say at face value, without challenging it. Like I said, trying is half the fun, and if I stop doing that, then I might as well throw my whole programming hobby in to the bin.
I'll post you the code for my ADPCM4 implementation soon. It should be fast enough for a vanilla A1200, and doesn't need a lot of stuff around it, just one function to encode and one to decode.
Sounds good, thanks :)
You can use my Tool "AudioConverter" or Samplemanager to generate ADPCM files and listen to the result.
I'll certainly do that.
Karlos' algorithm follows a so called "code-book based" approach.
From what I can see it has several drawbacks:
1. On 256/4bit is has more than 25% data overhead because he stores individual codebooks for each frame. Means instead of 4:1, you will get ~3:1.
2. How to find the "best" delta representants is not well defined and might need a lot of experimenting to find the optimal algoritm.
3. The encoder has a very high complexity because it needs to do vector quantisation. Luckily only for the encoder.
4. Doesn't make use of the assumption that the left and right channel of a stereo signal are correlated.
5. "Wastes" precious 4 bits in the 256/4bit case ;-)
6. The choosen Deltas may cause an error of up to 4096 quantisation steps (=reduces to 4bit PCM quality) in the worst case. However, very unlikely of course, but unlike in ADPCM, the error is not correlated with the high frequencies, so the error is not "masked".
Those are interesting considerations, although it wouldn't matter much, because of the 3:1 compression, which is already enough. It's a different story if the audio quality of ADPCM is better.
Perhaps the title should be changed to "(Not quite so) Simple..." :)
Yes, it has become less than simple, hasn't it?
@Karlos
What about a competition? ;-)
Let Thorham choose a short sample as uncompressed 16bit .wav (lets say 10 secs).
(maybe with some music, fading, voice, sound effect etc.)
We can both compress and uncompress it again, and let him decide what sounds better?
(as a proof, we both must provide the compressed file too, of course).
Sounds interesting. Perhaps I'll do that.
Classic Amiga.
I have one here, so it's no problem to try this.
-
Here is the C-port of the simpliest version of the ADPCM4 decoder for 16bit/mono. It packs exactly 1:4.
Check if this is easy and fast enough for your purposes. The code can be optimized, this here is tuned to be a clean example.
The audio quality degrades of course, but how much depends on the actual material.
There are also better versions, but the computation power is higher, and the result is only slightly better.
/* C-port from "file format audio" include, Amiblitz3 */
/* ADPCM4 defines */
#define FFA_ADPCM4_MAXFB 12
#define FFA_ADPCM4_MINFB 0
#define FFA_ADPCM4_LOW 0x0
#define FFA_ADPCM4_HI 0xF -5
/* context struct for all ADPCM/ADDPCM codecs */
typedef struct ffa_ADPCM_ctx {
int cSampM;
int cSampS;
int cDeltaM;
int cDeltaS;
int fbM;
int fbS;
int fpos;
};
/* ADPCM4 lookup table (Fibonacci) */
int ffa_ADPCM4_LUT[0xF];
ffa_ADPCM4_LUT[0x00] = 0;
ffa_ADPCM4_LUT[0x01] = -1;
ffa_ADPCM4_LUT[0x02] = 1;
ffa_ADPCM4_LUT[0x03] = -2;
ffa_ADPCM4_LUT[0x04] = 2;
ffa_ADPCM4_LUT[0x05] = -3;
ffa_ADPCM4_LUT[0x06] = 3;
ffa_ADPCM4_LUT[0x07] = -5;
ffa_ADPCM4_LUT[0x08] = 5;
ffa_ADPCM4_LUT[0x09] = -8;
ffa_ADPCM4_LUT[0x0a] = 8;
ffa_ADPCM4_LUT[0x0b] = -13;
ffa_ADPCM4_LUT[0x0c] = 13;
ffa_ADPCM4_LUT[0x0d] = -21;
ffa_ADPCM4_LUT[0x0e] = 21;
ffa_ADPCM4_LUT[0x0f] = -34;
/* ADPCM4 decoder function */
void ffa_DecodeADPCM4(BYTE *srcP, short* dstP, int flength, ffa_ADPCM_ctx *ctx) {
/* declare/init variables */
int cSamp = ctx->cSampM // previous sample
int fb = ctx->fbM // current bit shift
int n, cValue;
/* decoder loop */
for (n=0; n<flength; n++) { // for all samples
if (n&1) cValue = (*srcP++ & 0xF0) >> 4; // get the upper 4bit and go to next byte
else cValue = (*srcP & 0x0F); // get the lower 4bit
cSamp += (ffa_ADPCM4_LUT[cValue] << fb); // uncompress the sample
*dstP++ = (short)cSamp; // store in destination PCM 16bit buffer
if (cValue>=FFA_ADPCM4_HI && fb<FFA_ADPCM4_MAXFB) fb++; // raise the bit shifter
else if (cValue<=FFA_ADPCM4_LOW && fb>FFA_ADPCM4_MINFB) fb--; // lower the bit shifter
}
/* rescue decoder context for next call */
ctx->cSampM = cSamp // previous sample
ctx->fbM = fb // current bit shift
ctx->fpos + flength
}
-
Here is the C-port of the simpliest version of the ADPCM4 decoder for 16bit/mono. It packs exactly 1:4.
Thanks for the code :)
Check if this is easy and fast enough for your purposes. The code can be optimized, this here is tuned to be a clean example.
This should be fine, and can easily be rewritten in assembler.
The audio quality degrades of course, but how much depends on the actual material.
There are also better versions, but the computation power is higher, and the result is only slightly better.
So far I've only tried IMA ADPCM, and I must say that the quality isn't good enough. Will your encoder do better? Normally I'm not much of an audiophile, but in this case quality is everything (which is why I'm still trying to do this with some sort of lossless encoder as well).
I've tried Samplemanager, but this fails on my WinUae setup (which is the same as my Amiga setup). I've also checked Aminet for AudioConverter, but it doesn't seem to be on there. Should I try/search a little harder?
-
> I've tried Samplemanager, but this fails on my WinUae setup
Can you specify "fails"? Normaly it should work. Maybe a lib is missing that I forgot to ask at the begining?
(it needs minimum 68020+FPU, and ~16MB RAM or more, and I guess 24bit datatypes)
Neighter the released Samplemanager nor AudioConverter offer this codec. I just added it now.
I haven't compared it to IMA, but my codec is surely not better (but faster), otherwise you should break those IMA guys all 10 fingers and not let them touch an computer again.
But it is probably close to and it depends on the parameters you have used.
You may give me a wav and I can create a preview for you. I didnt release the ADPCM codec yet, because I still tweak it here and there, and files get immediately incompatible of course. So in case of codec you have to be sure that it doesn't need an update when you release it.
If 1:3 is enough for you, or even 1:2, one could easily adjust the codec from 4 bits to 6 or 8 bits. In 8bit I expect a quality you wont hear any difference. 6bit should be still much better than 4.
I tried 5 bit when I wrote the stereo version (5+3bits), and 5 bit truly halves the artefacts from 4 bits.
Another thing that can be observed that the artefacts are at very high frequencies. So if you use 44kH, you barley here them, if you use 16kHz it is clearly crackling. The same effect should be observable with IMA or any other ADPCM codec.
If you use looseless, 1:2 is the maximum you can get, ever.
Some files may get smaller, but some will remain larger, 1:2 is the average maximum of "expensive" codecs like FLAC etc.).
-
(it needs minimum 68020+FPU, and ~16MB RAM or more, and I guess 24bit datatypes)
Hehe, the readme file in the archive doesn't state any CPU requirements, and I had FPU emulation turned off, so it works now, although the GUI looks incomplete. Probably because of missing 24bit data types. Not a problem.
Neighter the released Samplemanager nor AudioConverter offer this codec. I just added it now.
Oh, okay.
I haven't compared it to IMA, but my codec is surely not better (but faster), otherwise you should break those IMA guys all 10 fingers and not let them touch an computer again.
Perhaps, but who says the designers of IMA have an optimal routine? Perhaps yours is better. You won't know until you try (although trying with Sox under Winblows in the command line is somewhat sucky).
You may give me a wav and I can create a preview for you.
Is Zshare and a 32 megabyte stereo wav a problem? It will be 16bit, 28Khz stereo.
So in case of codec you have to be sure that it doesn't need an update when you release it.
Probably won't be a problem, because I'll do my own assembly language versions :)
If 1:3 is enough for you, or even 1:2, one could easily adjust the codec from 4 bits to 6 or 8 bits.
Although I need more than 1:2 (not much, less than 1:3), I can certainly play around with it.
If you use looseless, 1:2 is the maximum you can get, ever.
'Perhaps' (I'll try to find some hard proof, or explanation, of this so that I can understand why this is), but because I can chop two bits off for each sample to get 14 bit lossless (14 bit is basically the format needed, not 16 bit), it seems that better than 1:2 is possible, and I actually don't need much more than 1:2, certainly not as much as 1:3. About 1:2.4 is enough to fit everything on a CD. Perhaps that is possible?
-
A dowload of 32mb is no problem.
> Probably won't be a problem, because I'll do my own assembly language versions
Yes, but you need the encoder too.
>> If you use looseless, 1:2 is the maximum you can get, ever
> 'Perhaps'
No.
> I'll try to find some hard proof
This is hard because in theory, the compression can be anything from 0 to 99.9% percent, because it is not defined how audio data actually has to look like.
The only thing you can do is messure the entropy of a large amount of real-world audio data (or the songs you plan to compress) and calculate the entropy. This will give you an idea of the loosless packing potential.
Only the extremes can be proofen hard. E.g. a random walk cannot be packed at all. Otherwise the "walk" would not be random. Entropy is 1.0.
If the audio data is completely perfect silence, it can be reduced completely, the entropy is 0.
Real world audio data is somewere inbetween usually ~0.5. That's why if you try hard to predict the signal, you end up in 50% packing rate. BTW, none of the currently existing codecs packs lower than 50%, all are higher. And some of them are really really complex and sophisticated. But doesnt help. You cannot overcome laws of nature.
Just zipping audio data gives you usually 75% of the size. Using a good predictor gives you 50-60%. But this will get very expensive to decode. Try to run Flac on a Vanilla A1200. It will be far slower than realtime and gives you 54% in average.
-
A dowload of 32mb is no problem.
Didn't think so ;) Here's the file: 15_Final Fantasy 10 - Besaid Island.wav (http://www.zshare.net/audio/77056440b7f4bb2a/). Hope it works well, because it's the first time I've used this kind of sharing service :o
Yes, but you need the encoder too.
I'll probably just write one in FreeBasic under winblows.
This is hard because in theory, the compression can be anything from 0 to 99.9% percent, because it is not defined how audio data actually has to look like.
...
...
That's certainly interesting. The reason I question these things is because computer science is such a young branch of science, whether it has to do with laws of nature or not.
Another reason is that scientists don't now everything about nature yet, not by a long shot and therefore it seems perfectly reasonable to me to at least try these things (it's not exactly quantum physics :)), knowing full well that I will probably fail.
It's never about trying to prove people wrong, or rebelling against the establishment. It's about trying, and learning something from the experience. Nothing more.
BTW, none of the currently existing codecs packs lower than 50%, all are higher. And some of them are really really complex and sophisticated.
Again, it can't hurt to try, because I enjoy trying (of course, I'll probably fail, but I'll have learned something in the process, and that's valuable to me).
In the end I don't mind having to use a lossy method (as long as I don't have to soft-loop all those tracks), but I just want to try out something myself first.
-
Here is an example of my implementation of ADPCM4.
All files are converted back to .wav so you can easily play them.
If you decide to use this codec, I can pass you the source along.
I personally find 4bit quite good, although with extreme material you can hear a slight crackling. In case of 8bit I dont hear any difference, whatever I encode.
The 2bit is more experimental how low you can go. I find it amazing what 2 bits per sample still can do, however, you can hear a significant degradation in quality.
Probably 6bit is quite hi-fi too, but it's a little hazzle to implement since 6 is not a fraction of 8, so you need to decode always 4 samples at once. But if 4 samples turns out to be too poor, 6bit would be doable too.
FF10 (original 16bit PCM) (http://www.hd-rec.de/Archive/FF10_PCM16.wav) - File size ~5MB
FF10 (8bit ADPCM) (http://www.hd-rec.de/Archive/FF10_ADPCM8.wav) - File size ~ 2.5MB
FF10 (4bit ADPCM) (http://www.hd-rec.de/Archive/FF10_ADPCM4.wav) - File Size ~ 1.3MB
FF10 (2bit ADPCM) (http://www.hd-rec.de/Archive/FF10_ADPCM2.wav) - File size ~ 0.8 MB
-
Here is an example of my implementation of ADPCM4.
All files are converted back to .wav so you can easily play them.
Thank you very much :) Much better than I expected, to be honest, especially 4 bit (and also better than IMA, it seems that IMA is for speech). Sounds good :)
If you decide to use this codec, I can pass you the source along.
Although I'm still playing around with lossless encoding ideas, I have a feeling I'm going to get stuck and need lossy anyway, so sharing your source code will certainly be appreciated a lot.
Probably 6bit is quite hi-fi too, but it's a little hazzle to implement since 6 is not a fraction of 8, so you need to decode always 4 samples at once. But if 4 samples turns out to be too poor, 6bit would be doable too.
Decoding 4 samples at once shouldn't be much of a problem to implement here.
-
Haven't been busy with this for a while now... I think I'll just write an ADPCM encoder and decoder myself. I'll try lossless encoding once more, and I'm also tempted to soft loop the tracks, although it's quite a chore (sigh).
Anyway, thanks to all who've posted their suggestions and comments :)
One more thought: If one could do software channel separation and sound recognition/reconstruction, I'm sure the entropy of music would be better than 0.5, but just try to write it :roflmao:
-
One more thought: If one could do software channel separation and sound recognition/reconstruction, I'm sure the entropy of music would be better than 0.5, but just try to write it :roflmao:
If by channel you mean left/right, then many encoders already do take advantage of the correlation between left and right to increase compression*.
If by channel you mean frequency band, then basically this is what mp3 and similar systems use.
*the encoder I wrote doesn't do mid/side band analysis. The reason for this is that the method of encoding tries to pick the best n deltas per frame of audio, using both channels anyway. As a consequence, when the channels are correlated; that is to say they have a similar range of delta values (regardless of overall phase difference between the left/right signals) then the encoder picks values equally suitable for each channel. When one channel has much more variation than the other (for instance, a loud sound in one and quiet in the other), then the louder channel dominates the pool of chosen delta values. The result being that the channel with most going on is always given a bigger share of the available "bits" than the other.
-
What I mean is this: If you have a tracked piece of music (all music is tracked), then each of the channels will play part of the music, one instrument at a time. The brain can easily separate the channels and recognize the instruments as independent samples, meaning it can convert a piece of music to a tracked format.
With a tracked format you can easily beat 1:2 lossless compression. A MOD recorded to WAV and compressed with, say, FLAC would be much bigger than the original MOD.
This is why I don't believe that the 1:2 ratio is dictated by nature, as bubblebobble claims. People simply don't know how to do it yet, because WAV to tracked format conversion isn't exactly easy to write, especially not without neural algorithms.
-
No need to separate into instruments/voices.
Since any possible sequence of numbers can be found in Pi (can be mathmatically proven) you just have to search Pi's digits for the sequence of your PCM stream and store the index where the sequence starts.
For decompression you just calculate the digits of Pi from that index on and voilà: your original stream will play back.
-
With a tracked format you can easily beat 1:2 lossless compression. A MOD recorded to WAV and compressed with, say, FLAC would be much bigger than the original MOD.
This is why I don't believe that the 1:2 ratio is dictated by nature, as bubblebobble claims. People simply don't know how to do it yet, because WAV to tracked format conversion isn't exactly easy to write, especially not without neural algorithms.
Ok, now I get you. Theoretically, assuming you could perform the required convolutions to identify the individual components, this might work for a piece of music where each "channel" is played by some easily identified oscillator function (square wave, sine wave, trinagle wave). You could then translate to a format that describes how those oscillators are played over time (pitch, volume, pan) in order to be able to reproduce it using those same oscillator functions. What you have there is something like your SMF (standard MIDI file) format.
However, real music produced on real instruments does not follow this paradigm. An individual instrument can produce a near infinite variation in tonal quality depending on how it is played, never mind the complexities added by production effects.
There are only two real analysis you can do with a recorded waveform in PCM format:
1) Analyse the sample data in the time domain.
2) Perform discrete FFT to convert into the frequency domain and analyse that.
In either case, you can perform an additional analysis based on the known effects of human audio perception and head related transfer function.
-
@Thorham
This won't work. A MOD is smaller because it uses samples, usually at a low quality (short, low samplerate, bitresolution, mono etc.). In a regular music production (if done using samples), they could easily exceed the memory needed to store the final mixdown. As soon as you start to use analog things like voice recordings, real instruments etc. the amount of information needed to describe this goes immediately up.
Apart from that, mixing the channels is a lossy process and cannot be reverted. What we hear is extemely lossy, even worse than an mp3.
What the human ear is doing is tracking instruments/noises by harmonic frequencies or other frequency distributions that have been learned before. Doing that, they miss a lot of stuff that is actually in the music. You might hear it the next time you listen, but then ignore something else. The ear uses a lot of information available prior to listening to the music.
E.g. put it to the extreme, you could write a codec that stores the entire music that will ever be encoded/decoded using this codec in a database. Then, encoding/decoding comes down to an index lookup, every music score is compressed to an integer.
But, at the end, you have to store the information, in the encoded data or outside from meta knowledge.
The same with predictors in looseles codecs. They represent prior knowledge that is assumed and put into the decoder. Once the data doesnt obey this knowledge, your codec screws up, e.g. on white noise.
The best predictors achieve roughly 1:2. Just accept this. Anything else is lossy.
mp3 or ogg are pretty good by compressing 1:12. They make assumptions how the human ear works.
-
The best predictors achieve roughly 1:2. Just accept this. Anything else is lossy.
I can only accept this if there's hard mathematical proof for this. If this 1:2 ratio is the result of the current best implementations, than it's circumstantial evidence, and not hard proof. If there is hard proof, then it's a fact, until than it's theory.
Anyway, the idea I presented is probably too hard to implement in this day and age anyway, and I just wanted to hear peoples thoughts about it :)
-
I've been away from the Amiga more than I would have liked. In the past, Karlos or somebody explained to me the process of cutting the first two bits off a WAV file in order to make it friendly with the Amiga. Since then I forget what I was told, would someone mind refreshing my memory?
Thank you!
-
Please?
-
Please?
I can't remember. What was the question?
-
This:
I've been away from the Amiga more than I would have liked. In the past, Karlos or somebody explained to me the process of cutting the first two bits off a WAV file in order to make it friendly with the Amiga. Since then I forget what I was told, would someone mind refreshing my memory?
Thank you!
-
This:
Hmm. I'm not sure it was me that told you that, but well, you should be able to load wav files straight into Soundstudio.
The RIFF WAV specification is a bit of a mess as it tried to become a multiformat container for different audio codecs and sort of failed. However, basic 8 and 16-bit linear PCM encodings are fine in Soundstudio.
You can't load 16-bit WAV data as raw because it's byteswapped and basically it will sound awful. 8-bit WAV data is, for legacy reasons, also unsigned, I believe (centred at 128, rather than 0).
-
Don't know if it's useful, but here goes ;)
I've been away from the Amiga more than I would have liked. In the past, Karlos or somebody explained to me the process of cutting the first two bits off a WAV file in order to make it friendly with the Amiga. Since then I forget what I was told, would someone mind refreshing my memory?
Thank you!
What you have to do is endian convert each sample in the WAV file, because the bytes are swapped:
move.w (a0)+,d0 ;a0 points to sample.
rol.w #8,d0 ;swap sample bytes.
In case the samples are stored as unsigned integers you can convert them to signed like this:
sub.w #32768,d0 ;d0 contains the sample.
After that you can convert the sample from 16 to 14 bit like this:
asr.w #2,d0 ;d0 contains the sample.
I sure hope I didn't make any stupid mistakes ;)
-
WOW, OK, maybe nobody did explain this one to me before, seeing as all that looks...
...a bit over my head.
Are these command lines for a program or something? I'm sure if I start typing that stuff in CLI that nothing will happen but errors, so what's this for?
Thankx! :)
-
@XDelusion
Why do you need to convert 16-bit WAV's to 14-bit anyway? Tools like Play16 etc use AHI which does the conversion from 16-bit data to calibrated 14-bit as it plays.
If you just want to use 16-bit sample data in OctaMED, just go right a head and load the WAV into the sample editor. It should be fine.
@Thoram
I sure hope I didn't make any stupid mistakes ;)
Just the one. The conversion from unsigned to signed should have subtracted 32768, not 128. You'd subtract 128 for 8-bit unsigned sample data.
-
Are these command lines for a program or something? I'm sure if I start typing that stuff in CLI that nothing will happen but errors, so what's this for?
Oops! It's assembly language (language of the computer in readable text format). Seeing how this is the programming section of the forum, I thought it might be useful to you. Sorry :)
Why do you need to convert 16-bit WAV's to 14-bit anyway? Tools like Play16 etc use AHI which does the conversion from 16-bit data to calibrated 14-bit as it plays.
Playing WAVs is better done with programs that have their own 14 bit calibrated player routine, because AHI sucks for Paula, and isn't needed for this anyway. Play16 uses it's own replay routine, and Hippo player, which can also stream big WAVs, also has it's own replay routine (but I'm sure you already knew that ;)).
Just the one. The conversion from unsigned to signed should have subtracted 32768, not 128. You'd subtract 128 for 8-bit unsigned sample data.
Yeah, you're completely right, of course. Guess I was typing too fast :o
-
Oops! It's assembly language (language of the computer in readable text format). Seeing how this is the programming section of the forum, I thought it might be useful to you. Sorry :)
Playing WAVs is better done with programs that have their own 14 bit calibrated player routine, because AHI sucks for Paula, and isn't needed for this anyway. Play16 uses it's own replay routine, and Hippo player, which can also stream big WAVs, also has it's own replay routine (but I'm sure you already knew that ;)).
Yeah, you're completely right, of course. Guess I was typing too fast :o
Actually yeah, Play16 talks directly to the calibrated "cybersound" driver. However, if you have AHI, far from sucking, it performs quite well. Later versions, however, are far too cpu hungry it seems.
-
Hmmmm.
So in other words, I better just stick with OctaMED and stop dreaming of using 16 bit samples with other software. This thread is confusing. :)
-
Actually yeah, Play16 talks directly to the calibrated "cybersound" driver. However, if you have AHI, far from sucking, it performs quite well. Later versions, however, are far too cpu hungry it seems.
The point is that AHI isn't needed at all for 14 bit+calibration. AHI might be cool for audio cards, but for Paula it isn't needed in my opinion (yes, that might have some small back draws, but I still don't use it) ;)
This thread is confusing. :)
That's because it's the wrong thread to post your question in. This thread is about what a 14 bit driver does to 16 bit samples, and about audio compression. Also, it's programming related, and doesn't have anything to do with audio software such as Octamed. Note that this thread is in the development (programming) section of the board. It's better if you make your own thread in an appropriate part of the forum :)
-
The point is that AHI isn't needed at all for 14 bit+calibration. AHI might be cool for audio cards, but for Paula it isn't needed in my opinion (yes, that might have some small back draws, but I still don't use it) ;)
One reason to use AHI on Paula is if you have multiple apps that need to play back audio concurrently. Think of it as a simple mixer interface to Paula.