Anyone for Tea?

I am gooood!

OK, as no-one else (perhaps) is going to be interested in me spouting off about my victory over all the combined forces of technical nastiness I thought I just post it here.

Basically - right - we've just finished a sermon series at church (based on the 10 commandments, using some of J John's 'Ten' series as source material) - as part of this, the standard deal is that we put together a box set of CDs to sell onto people so that they can listen to our pastor's wisdom to their hearts' content.

Now, a couple of weeks back - our Philips MP3 Hard-Disk recorder decided to stop recording stuff - which was annoying, but thankfully I had my iRiver Flash recorder with me and we managed to get the sermon.... fine.... but now we had a no-worky Philips.

Jen-from-the-office (as is her officially recognised work name) got on the phone to Philips, and as part of the process of trying to trace the problem, she was advised to format it. Unfortunately, we then come to this week where we discover that actually one of the MP3's that we thought was Sermon Number 4 ("How to keep the sabbath holy") was actually just the worship from the same service and the sermon was no more.

So, step 1 was to try out a bit of 'undelete' software. This initially looked like it was going to do the trick, lots of MP3 files recovered - however, unfortunately not the exact ones we were looking for (Sermons #3 and #5, but no #4!). At this point, normal practice would have encouraged us to give up - it was pretty clear that the FAT table on the drive had taken the sermon to heart and had decided to have a sabbath of it's own!

So - there we were, it seemed all hope was lost, or at least all the recordings were lost. But I thought - weeeell, lets see whether we can find some sort of useful information by just bypassing the FAT and going straight to the disk data itself. So, here's what I did:

1. I got hold of a copy of 'dd' for Windows, and used this to rip as much as possible of the raw data on the drive to a file on my PC. I got about 4GB of the stuff, before it all got a bit confused and died, but I had enough data to start with.


2. I searched and found a detailed description of the MP3 file format.
   
   I'd picked up from my work at work that MP3 was primarily a 'frame' based format - that if you were able to identify a certain block of data as 'MP3', you could stitch them together to make a full MP3 file - ie. that the loss of a few frames at the start/end of an MP3 file would not be fatal to the sound file as a whole.
   
   
3. So, after searching the 'net and finding nothing suitable, I just went and wrote my own little utility (in C) which would read through a stream of data looking for the MP3 frame header markers (0xFF 0xF?) and then read the block of MP3 data that followed.
   
   
4. This was tested extensively on lots of other 'basic' MP3s, and knowing some details (192kbs 'normal' 44khz MP3, Joint Stereo, etc...) of the recording format used by the Philips unit I was able to be fairly sure that a run across the data I'd "farmed" from the unit would turn something up.
   
   
5. After a bit of investigation, and dealing with output files that sounded like very much like tongues (or Afrikaans, take your pick!), I also extended the utility to stitch together any sequential MP3 frames that it found into individual files, and output all the files it found to a given directory.
   
   
6. I then ran this on my data, and was presented with 500-odd MP3 files, ranging in size from ~700 bytes to 200MB.
   
   
7. The end result? After listening through all of the reasonably sized files, I turned up two MP3s which - appended to each other - turned out to be the recording of #4 from the morning service that day - and another single MP3 that turned out to be the complete evening service from the same day. Yayyyy!!! Open the champagne!

So, what did we learn from this? We learned:

1. Keep backups of everything, from as soon as possible until you absolutely totally completely have verifiably made sure that you have exactly what you wanted. And then take one more backup for posterity anyway.
   
   
2. A 'complete format' thankfully (on this occasion) doesn't neccesarily mean that you've destroyed the data.
   
   
3. We hadn't fragmented the filesystem onboard as much as i'd feared and the recorder had kept most of the data together in contiguious lumps.
   
   
4. MP3 rocks as a file format that can deal with marginal corruption of this sort.
   
   
5. The 'last resort' sometimes actually works!
   
   
6. I rock! Although - to be fair - if "I shall not boast in anything, save in Jesus Christ" - then it'll be to His glory that these CD's can make a real difference in someone's walk with Him.... and he gave me a brain and put me in the right places to be a servant in His house.

But yeah... I was rather happy about that. I'll post the source code for the MP3 reader up here in the New Year anyway, so others can make use of it... or if you're really interested in how good my code quality actually is...

Anyway. Church tommorrow (yes, Saturday, Christmas Eve) so I'd best get off to bed. Later...

posted by RichB @ Saturday, December 24, 2005