Fast audio format converter

For PC type software that runs under some PC opsys.

Fast audio format converter

Postby Doug Coulter » Sat Apr 20, 2013 5:52 pm

Well, it turns out that I have a rather huge collection of CD's. Some bought, some inherited and so on - all legit, no piracy. I ripped them all to full quality .flac files over the course of a couple weeks (we're talking huge here), but while those play fine on the computer and are "only" about 150gb total, I wanted pretty-good mp3's of it all for use in automobiles and so on. Well, there is a nice script out there you can put in .gnome2/nautilus scripts that converts things, but this problem was well past its speed, so to speak. It just takes too much human intervention, can't do a whole slew of subdirectories, handle mixed input formats (some things I only had as mp3, like some very old music I'd done myself) and so on. Soooo....we find that sometimes it's quicker to write code to automate something than to do it manually, even if you're only going to do that "something" one time - convert something like 7k files of mixed input types to a decent quality (average 192kbs) mp3's in this case.
Here's what I began with:
audio-convert.zip
This is what I began with - not bad, considering, but all in bash and well, ugh..too much user input needed, and with this my job will take days or weeks.
(8.1 KiB) Downloaded 300 times


Soooo, bash ain't my cup of tea, really. Perl is more to my liking. So, I wrote a perl script to do the subset of this that I want/need, and optimized it in a couple of ways.
Rather than fool with threading or forking, I simply divided my main CDRips directory in two, and hardcoded two different versions of source dir into a couple copies of my perl script - allowing two to run at once on a two-core machine I have here that gives best mips/watt - and has a SSD I can use for temp files; I am willing to beat it up for this one job - using it for temp .wav files converted from .flac so lame can do the .wav to mp3 on them.

I worked it out based on some initial tests. A single thread version of this was going to take well over 40 hours - at a 40-50x speed faster than real time(!). Well, on solar power with uncertain spring weather, and not wanting to make it so I could stop and restart the program, I decided to go for raw speed. I divided my source CDRips directory into CDRipsA and CDRipsN (alphabetically, they are about equal in size) and made two versions of the perl script to use the two hardcoded (yuck, I know, but adding a gui etc would have been more work and I didn't need it - this is a one-shot) to convert it all into another directory called "MP3s".

And here they are. No differences other than a different source dir name and temp file name for each.
They are screaming along right now (6:30 on 4/20/2013) and ought to be finished by tomorrow afternoon. Did I say huge? This is pegging out both CPU's in a core duo running 3.12 ghz...using nearly no memory, a SSD for the two temp files and a spinner for input and output. Zowie...I used most of the tricks to make this go fast, even if it cut only a second per conversion, I did it. I'm using variable bit rate here, with an average of 192kbs or so, the latest/greatest lame settings and all that. I could have made it maybe .5 sec per conversion quicker if I eliminated all the status messages, but...at that point I wanted to just be able to see it working, so I just slowed those down to 1/second.

And here is the result. Turns out I could have done it just a little better, as commented in there - lame now supports tagging better so I didn't have to copy the old way, but frankly, when it takes a bunch of seconds to convert, and an un-percievable time to re-tag (copy the tags from the original .flac into variables, and then into the resulting mp3 file) - it wasn't worth experimenting any further.
flac2mp3.zip
The "solution" to my problem. Took a few hours to write, but will save me sitting and inputting maybe for 2-3 days - good gain.
(3.42 KiB) Downloaded 300 times


So, sometime tomorrow afternoon, I'll have ~~ 150gb compressed roughly 8::1 to burn on USB sticks for my cars, or onto CD's for my truck. That's a pretty nice compression ratio and will make my life a lot simpler for on-road entertainment. Heck, that's only 18.5 gb estimated - if I ditch a few of the less worthwhile titles, I should be able to fit this all on one stick - or about 4 dvds....that's real progress and will unclutter my vehicles a good deal.

Motto - do something cool every day. Then at some point you find you're surrounded by uber-coolness. Try it.
No pix, no attention span. So here's a screen shot of this maxing out my main server (not my fastest machine, just the most efficient one so it can be on more often - on solar power).
AudioConvert.png
Make those cpus get hot!
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: Fast audio format converter

Postby Doug Coulter » Sun Apr 21, 2013 2:17 pm

Well, this ran overnight and converted 6550 and change files, with one burp when it encountered a non-flac file and one of the two instances died. I added some code to prevent re-occurrence of that, but the job is done now, and I'd posted this code to show how, not so much as a product. I'll get the newer code (basically just ignores non-flac and also now won't reconvert something it's already done so it didn't have to really start over from the beginning) up here at some point (it's on another machine here, one that has my best mips/watt ratio).

In this case, it saved me a couple weeks spare time, as the existing tool would only do one folder at a time, and required a lot of intervention for each new folder (as in asking quality, info tag transferral and so on). Then you waited half a hour, and had to point it at the next folder and do all that again. This way it was a one shot deal.

In other words - and this is what perl is good for - it took much less time to make the tool than it saved me in a single run. And next time I need to deal with all those music transcoding libraries - I have a live, working example, so if I have to do it again, I've got a great jump-start.

~~150gb became about 33 gb. This means that if I remove just a few things, it will now all fit into my car's audio storage. I don't spend much time listening to the Feynman lectures or hitchikers guide to the galaxy (about 12 hours between those two) so...job done!
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA


Return to PC

Who is online

Users browsing this forum: No registered users and 2 guests