Anyway, usenet/ newsgroups were around well before the world wide web, and were the thing, back in the day, for universites and special-interest groups to stay in touch.

Even today, there are tens of thousands of active newsgroups, still plugging away on just about any topic you care to name (and plenty you wouldn’t want to, given their adult nature).

Newsgroups are text based, and therefore not suited to sending or receiving binary files. Until some other smart puppies figured out a way to take binary files, convert (encode) them into a series of text messages, send them via usenet, and then convert it back (decode it) into the original file.

This was the start of filesharing via usenet.

Where’s all this going? Well, it so happens that there’s a vast amount of content, including mp3 content, just waiting for someone to find it, select what they want, and upload for their listening pleasure. This content is not publicly accessible, so there’s three things you need. I’ll mention them here.

#1 – A premium newsgroup provider

All these files exist on servers with massive storage, and you (generally) need to pay to get access. Like peer-to-peer, these files exist on a conglomeration of servers around the world but unlike P2P, you can’t just tap into them. A premium newsgroup provider runs an array of servers that hold complete copies of files, and will serve them up to you at speeds that will max out your broadband connection.

The one I use and recommend is Giganews – specifically, their Diamond Plan. This one lets you download as much as you want each month, plus it gives you access to VyprVPN (for added security) and Giganews Accelerator (for slower connections).

#2 – A Newsreader

This is a program you  install on your PC or Mac to connect to your selected newsgroup provider. For Windows, I’ve used Newsleecher (which is good and bad – it has an inbuilt search engine but it’s not free) but my favourite is the oddly-named SABnzbd, which is available in Mac, Windows and Linux flavours. And it’s free.

There are many others, these are simply the ones I’ve used over the years. A good newsreader will handle all the unarchiving and decoding work for you so, at the end of the process, you have a neat package of files assembled just as they should be (it wasn’t always this easy, in fact the decoding process took some knowledge and time).

#3 – Finally, a usenet search engine

You won’t find these files in Google. The way to find them is to use a usenet search engine (if you purchased Newsleecher your’e in luck,  there’s a search engine built in). One easy one is newzbin but my favourite is nzbmatrix. There are many choices here and unfortunately most require you to be a member (free for some, paid for others) to get the full benefit.

Basically, when you find (say) an album you’d like to download, the usenet search engine will provide with an nzb file. You load your nzb file into your newsreader and the download starts automatically. Similar, I guess, to a torrent file and a P2P client – except that a usenet download is “one to one” (you connect to the usenet server, not to a group of peers) and therefore anonymous.

Cdex

A little while ago I wrote an eBook on how to install, configure and use CDex. It’s been updated since, and it’s my pleasure to offer this to you as a free, no-strings-attached download. (18 pages, PDF format).

Hunchy’s Insider Guide to CDex

Enjoy, and please let me know of you found this eBook useful (or if you found any mistakes).

photo by RW PhotoBug (CC)

I’m not downplaying the importance of the case, the liner notes or anything else that comes with the CD itself. These all add to the music experience, but the music is THE deal.

Before We Start – What Is Ripping?

When you rip a CD you take each audio track and turn it into a file on your PC. There’s software that does this (I’ll cover this in a later post) that makes the process as easy or hard as you want.

Why would it be hard? The reason is that it’s actually quite difficult to get a perfect rip of any CD. The CDs themselves are mass-produced items, as are CD drives. Read errors are common, and although most errors are captured and corrected within the CD drive, some are missed. If you’re a perfectionist I’ll shortly be writing about a couple of products that you’ll want to know about – Exact Audio Copy (EAC) and dBpoweramp – that are capable of producing perfect rips when set up correctly.

Why would it be easy? There’s many CD ripping products that are so simple to use it’s mad. Just slip in your CD, hit the “rip” button, and in a few minutes you have your files. However, ease of use like this usually means crap quality. I’ve written a a free eBook about one I’ve used for many years that does produce good results – it’s called cDex – and it’s very easy to use.

I rip my files to the MP3 format. I do this because it means they are relatively compact, they can be tagged with all the track information (title, album name, artist, etc) I want, they can be played on just about any portable music player produced, and they give me high quality playback.

Let’s talk about the six reasons to rip your CD collection.

#1 Wear And Tear

When CDs first hit the retail shelves they were touted as being pretty much indestructible. Compared to vinyl records, they are… but experience has proven they aren’t really that tough and its easy to turn a brand new CD into a drinks coaster with a bit of mishandling and bad luck.

When a CD becomes unplayable because of scratches, warping or cracks, you’ve lost the music forever. Your only choice is to go and buy another one, or – if you were smart enough to rip it first – you just fire up your PC and burn a copy. You still have the case and liners, you certainly have the right to play your music because you licensed it already with your first purchase, so as far as I am concerned you have the right to burn another CD for your listening pleasure.

That’s reason #1, and maybe it’s the only reason you need. It’s cheap insurance against the deterioration that any favourite and often-played CD will suffer from over time.

#2 Parties

CDs get damaged big time at parties. They get used as coasters, they get sat on and stepped on, they get scratched, and they get stolen.

Before the party, decide the music you want to play and burn compilation CDs. Then round up all your originals and lock them away. If the copies get damaged or go missing, who cares? A CDR disk only costs a few cents, and it takes only a few minutes to burn a copy. Great insurance.

#3 Cars

The interior of your car is a bad place for CDs. They roll round on the floor when you take a corner, they get scratched when you try to change CDs and drive at the same time, they end up on the floor and get trodden on, and the sun warps them when you leave them on the dashboard.

If you car get stolen or broken in to, you’ll likely lose the lot. And if you have more than a few CDs stolen, your insurance company won’t cover their full replacement value.

Rip your CDs and only take copies in your car. When they become unplayable, throw them out and burn new copies. If your car gets stolen, at least you haven’t lost $3,000 worth of CDs as well.

#4 Travel

This is much the same as for cars. When you travel, you’ll probably throw some music into your luggage to occupy all the lazy hours at airports and bus stations. Now it may come as a surprise… but CD collections (and iPods) don’t last long in a backpacker hostel.

So rip your CDs and only take copies with you. You’ll be glad you did.

#5 Portability

Most of us have multiple devices we use to listen to music. Your PC, your CD player, your iPod, your home theatre system, and so on.

You can’t play a CD on your iPod, so it’s clear that you need to rip your CDs and load up iTunes, or you need to buy the same music again from ITMS. I prefer to rip then upload, as I already have the moral right to listen to my music by virtue of my CD purchase. In Australia I have the legal right as well, as it’s legal to “format shift” in this country.

If your home is set up for it, you can play your ripped music collection, stored on your PC, in other parts of your house. There are network-attached players available that are audio-only, or even audio plus video, that can be integrated into your home theatre system. This is a big topic and I’ll be covering this in a future post. There is some outstanding equipment hitting the retail shelves as I write this in January 2008, and it’s well worth some investigation.

The baseline is that your CD collection needs to be ripped into music file format before you can do any of this stuff. So if you’re thinking about enabling your music (and/ or video) around the house, you need to get ripping now. Ripping isn’t the fastest process on the planet when you aim for perfect quality, so get a head start and start ripping now.

#6 CDs Aren’t Perfect

Retail CDs are mass produced. And like all mass-produced things, quality can be variable.

I’ll try to keep non-technical, so bear with me. Audio CDs and data CDs may look the same but the data is formatted differently on each. For this discussion, the important difference is that audio CDs have less error correction information than data CDs. Both have sectors of 2,352 bytes – all of this is audio data for audio CDs yet only 2,048 bytes of this is data in data CDs. The difference – 304 bytes, or 13% – is error correction data.

Sidenote: there’s still lots of error correction information on an audio CD, but as users we don’t get to ever see it. The point I’m trying to make is that data CDs have an extra level or error correction, and are less susceptible to damage than audio CDs.

What this means is that a damaged audio CD will sound like crap, whereas a similarly-damaged data CD will still work fine. And because none of this error information is accessible to users, we don’t know how close an audio CD is to failure.

I’ve had brand-new CDs that won’t play, even on a top-end Bose. And I’ve had CDs that play fine on the Bose, but won’t play on the piece-of-crap player I have in my car. You’ve probably had the same experience.

When you buy a CD, the first time you play it should be in your PC so you can rip it to hard disk. Then if it deteriorates to the point where it won’t play in your player of choice, you can just burn another copy.

In fact what I do is rip every new CD, then put the original away and only ever play burned copies. That keeps my investment safe from damage.

– oOo –

So there you have six solid reasons to turn your physical CDs into music files. As you know I prefer the MP3 format over all others because it’s the defacto standard worldwide. And being so popular, it’s got the best chance of being playable 50 years from now.

In future posts I want to cover the ripping process in detail, as it’s surprisingly hard to do it right. But when you take your time to set things up you’ll only have to do it the one time for every CD.

photo by ehnmark (CC)

It also means I need to drag my sorry arse into action and finish all the half-written content I’ve got lying around. So can you guys tell me what it is you want to hear about? Leave a comment, tell me what’s on your mind (mp3-wise), and what you’d like to know first so I can prioritize what gets published.

It also means I had to increase my bandwidth limit on this hosting account, which is no problem as I have one of those master reseller accounts that gives me 50 full cpanel accounts under it, of which this is one. So sorry if you visited & got an error page earlier.

How MP3 Got Its Name

MP3 is an acronym for MPEG Level 1, Layer III. MPEG is itself an acronym for Motion Picture Experts Group, established in 1987 as a working group of the International Standards Organization (ISO). MPEG’s job was to establish standards for compressed digital video and audio signals.

In 1991 MPEG called for submissions for audio coding and got 14 responses. A group of researchers from Germany submitted their work, a codec named ASPEC, and with some tweaking this became the basis of MPEG Layer 1, Level III.

As you guessed, two other submissions got the nod as well. These became Level I and Level II – lower complexity, lower quality, simpler to implement. However as computers got more powerful and capable of handling the complicated but more efficient Layer III standard, the others have fallen into disuse.

The filename extension chosen for MPEG Layer 1 Level III was .mp3. And this is what gave MP3 its name.

MP3 is Proprietary

The original research was performed by the Erlangen-Nuremberg University and sponsored by a German company Fraunhofer IIS, and further work was in part sponsored by French company Thomson SA. Between them Fraunhofer and Thomson control the licensing of MP3 technology throughout the world.

However Fraunhofer and Thomson have not been too diligent about enforcing their rights, at least as far as software is concerned. Freeware encoders and decoders are easy to locate on the Internet, although many shareware and all commercial products do indeed license their MP3 technology from these two companies (as do all portable MP3 players).

Psychoacoustics

The basis of MP3’s effectiveness is psychoacoustics, which is the study of how humans perceive sound. The human ear is not a perfect instrument, and the algorithms used in MP3 compression take advantage of this.

Auditory masking is a phenomenon whereby we can’t hear weaker audio signals in a particular frequency range in the presence of a stronger audio signal at a single, similar frequency. At low frequencies, the range around one of these dominant frequencies may only be 100 Hz. At the upper end of the scale, it’s around 4 kHz.

Psychoacoustics and auditory masking are the guts of MP3 encoding – take samples of the audio signal across 32 predefined frequency bands, and break them into single tones of varying volume. It’s worth pointing out that MP3 encoding is a whole lot more complicated than this. There are subtleties in the way the signal is analysed that help minimise the effect of sound “artifacts”, like echoes or ringing, and the signal-noise ratio comes in for special treatment as well.

MP3 is Lossy

Lossy is the opposite of lossless. It means that some of the original signal is lost irretrievably, and can’t be recreated. But that’s the point of using psychoacoustic theory – the audio that’s lost is the stuff we can’t hear (at least, not at high bit rates).

Frames

MP3 files are encoded into frames – chunks of data that represent the audio signal. The time represented by a frame varies according to the bit rate specified during encoding (which can range from 32-320 kbit/sec), the tonal complexity of the audio, and (sometimes) the sample bit depth (which can be 8 bit or 16 bit).

Each frame holds 1,152 samples, organised as a group of 36 sets of 32 frequency samples (from each frequency band). So frames are based on the number of samples, not time. A single frame doesn’t cover a fixed number of milliseconds, it covers a fixed number of samples.

Compression

Those frames aren’t fixed length. What happens is that some frames that represent complex-toned audio can use space that’s left over from shorter frames earlier in the file. If that extra space remains unused, then the file is shorter and compression is higher. If it is used, then sound quality is better than it would be otherwise.

Also, there’s a reasonable amount of empty space in each frame. MP3 uses a form of compression called Huffman coding to squeeze them down to a minimal size without losing frame data. This is a bit like zipping a file with winzip.

Sample Rates

MP3 files usually have a specified sample rate of 44.1 kHz, which is the same as a retail CD. Other sample rates allowed for are 32 kHz and 48 kHz, and there are plenty of encoders that use non-standard values down to 8 kHz.

The standard 44.1 kHz is good for human hearing, theoretically it can represent frequencies up to 20, 500 kHz (beyond what we can hear). That’s the one I use. If I was a dog I’d use 48 kHz, but I’m not.

For speech, lo-fi or radio 32 kHz could be the choice to get less dynamic range and smaller file sizes, and 48 kHz is really only there because it was used on digital tapes and , more recently on DVDs.

Sample Depth

Only two choices are available, 8 bit and 16 bit. This is different to bit rate, which is the bits per second used to represent the audio. 320 kbit/sec is the highest quality, and 32 kbit/sec is the lowest.

Sample depth means the number of bits used to represent a sample. If it’s 8 bit, it means the sample can have one of 256 different values. If it’s 16 bit, the sample can have one of 65,536 different values. So 16 bit is much higher resolution and the one to use. It doesn’t make much of a difference to the file size anyway.

Sample depth only matters when you’re decoding the MP3 file (playing it back), as the MP3 encoding process uses the bits it needs to represent the audio signal. It fits it to the required bit rate (a predefined value between 32-320 kbit/sec) and adjusts the file size by using more frames as required.

Tags

MP3 tags aren’t a part of the MPEG standard. They were an afterthought and even now, aren’t covered by an ISO standard. ID3v2.3 tags are used almost universally these days and support for ID3v2.4 is growing. Tag data can be located at any frame boundary but usually gets located at the beginning of the MP3 file. The older ID3v1.1 tags are positioned at the end of the file.

More Information?

I wanted to keep this post simple, without too much techo detail. But if you’re hankering for some in depth information, here’s a few links that can take you further:

Wikipedia

MP3 (entry)

Psychoacoustics (entry)

Audio data compression (entry)

Others

Yen Pan, D., A tutorial on MPEG/ audio compression (1995)

Yen Pan, D., Digital audio compression (1993)

Here’s why.

Audio CDs have error correction. Technically it’s called Cross-Interleaved Reed-Solomon Code, or CIRC for short.

There are two levels of error correction, C1 and C2. C1 corrects bit errors, and C2 corrects frame errors. An audio CD frame hold 192 bits (24 bytes), and there are 98 frames per sector.

The audio data and the error correction data are interleaved, which means they are mixed up in the data stream read by your CD player. The theory is that a surface defect will affect data in a sequence; when your player does its de-interleaving thing on playback, the affected data is out of sequence and there’s a much better chance it can be recreated.

On playback, the CD player is smart enough to de-interleave the data, separate out the audio data and error correction data and, if it finds errors, will use the error correction data to recreate the original audio. If not, it will interpolate between known good samples to get a reasonable value to insert into the audio stream.

If you clean your CD in a circular motion you run the risk of making both the audio and error correction information unreadable because you may damage too much data in a single track. This will give you clicks, pops, or may even make the player skip a second or two of audio.

If you clean from the centre out, any scratches you make will cut across multiple tracks. The error correction can do its thing, and you’ll get perfect playback even with those scratches.

Note that some computer CD drives are able to report back the number of uncorrected C2 errors. They may be so small that you can’t hear them, yet your ripping software can use this information to figure out whether you’ve achieved a perfect rip or not.

SprookjesBeelden aan Zee

What do you do when there’s a knock at the door at 3am? And the authorities, god bless ‘em, want to take your PC for ‘analysis’?

Plausible Deniability

Google the phrase ‘plausible deniability’. I am not a lawyer (IANAL), but to my layman’s mind it means that I can assert that something ain’t so. And anyone pulling my PC apart, bit by bit, will have a hard time proving me wrong.

It’s like a hypothesis, if you’ve studied research at college or uni. The authorities hypothesize there’s illegal content on your PC. You tell them no. They have to prove their hypothesis is true, otherwise they must accept their hypothesis is wrong (or at best unproven).

Introducing Truecrypt

There’s a neat piece of free software called Truecrypt. When I dicovered it, it blew the ole’ socks off. Right into the next zipcode.

Here’s the link (this is freeware, so no affiliate link):

Truecrypt Disk Encryption Software (non-aff link)

Truecrypt lets you set up an encrypted volume on your PC hard disk. The volumes it creates are indistinguishable from random data on your disk, yet with the appropriate key will open themselves up and reveal all their ill-gotten contents.

But you need the key. Theoretically you could crack the key, but that would take some effort spread over a gazillion years or so.

The authorities could also put you in front of the magistrate, who would demand you reveal the key.

It’s against the law to diss the judge, right? So you give them the key.

But here’s the fun bit. Truecrypt allows encrypted volumes within encrypted volumes, so the key you give them unlocks the first layer. This holds a small quantity of ripped MP3s you already legally own, or material that’s so banal you wouldn’t bother downloading it anyway. Like Britney, or the Pet Shop Boys (shudders hard, spills beer).

The real stuff is encrypted at the next level, and there’s no damn way they can prove it even exists.

Gets even better.

(Almost) Zero Footprint

The Truecrypt software can be installed on a flash drive. Admittedly there’s a bit of extra tomfoolery involved to initially mount the encrypted drive, but the benefit is there’s almost a zero footprint.

Once the drive is mounted at boot time, you take the flash drive out and stick it in your pocket. Because there’s no Trucrypt footprint on your hard drive, noone can say for sure what’s there. In fact, the drive just looks like it’s random bits.

That’s the way you run it. Your PC’s data volume, or your network attached storage, is encrypted to two levels using Truecrypt. While it’s powered up you have access to everything there. Power it down, and it looks empty until you mount it with your pocketed flash drive.

Here’s the Gotcha

The whole system depends on a minimal footprint being left by Truecrypt. Just by itself, Truecrypt does this pretty well. It’s not perfect – someone could see evidence of Truecrypt activity in the Windows registry, but they can’t tell what it was or how it was used.

Here’s the issue. If you use a Media Player that relies on a database of some sort, you’re (potentially) exposed.

The player database holds all details of your media library – even when the source files aren’t available. So if you use Microsoft’s various media products (Media Player, Media Centre, Vista Home Premium), Apple iTunes, or many other database-dependent media players – you’ve unwittingly created a footprint beyond Truecrypt’s ability to control.

How do you protect yourself? The obvious answer, and the only correct one, is you don’t use a database-dependent media player. Ever.

Sounds like a subject for a future post?

Know what they all have in common?

You’d be right if you guessed they ALL play MP3 files.

An MP3 is a file that can be copied, deleted, moved and renamed just like any other file on your PC. But it’s a format that’s exclusively used for audio content, and it’s the out-and-out leader, everywhere in the world, of all other audio file formats. MP3 first, daylight second.

This blog is all about MP3s. I’ll tell you what they are, how to get them, how to organize your MP3 collection on or off your PC, how to play them (wherever you are), and how to archive them for safe keeping and future enjoyment.

MP3 files will be with us for many, many years. There’s a huge volume of material, both legal and illegal, that use the MP3 format, and every portable player supports it. Equally important is the number of software tools to rip CDs to MP3 files, to correctly tag and organize MP3 collections, and to play MP3 files on just about every hifi system, from low-end micro systems to network-aware jukebox systems.

My plan over the next few months is to post in these broad topic areas:

The MP3 Format

I want to cover some background information about the MP3 file format, including a very brief history and some good detail about quality and digital rights management (DRM). I’ll also talk about the alternatives to MP3 – what they are, why they’re better (or worse), when you’d bother to use them, and where you get them from.

Sources of MP3 Files

Where do MP3 files come from? I’ll touch on the legal and illegal sources of MP3 files (note – I am not a lawyer), and the benefits and drawbacks of each. You’ll get good information about how to get the content you want, at the price you want to pay, and the risk you want to bear.

Organizing Your MP3 Collection

It won’t take long for your collection to grow out of control (and that’s a big understatement). I’ll go over details about keeping your MP3 collection correctly tagged and named so you can always find and play exactly what you want… even if your collection is hundreds of gigabytes and thousands of CDs in size.

Playing Your MP3s

It’s not much fun having to sit at your PC to enjoy that massive collection you’ve built. I’ll show you how to burn custom CDs and DVDs to take your music anywhere you want, how to use and organize your music on portable MP3 players, and how to connect your home stereo so you can stream music all over the house.

Archiving Your Collection (and any other digital data you value)

Before long you’ll need to think real hard about how to protect your collection against hard disk failure, theft and honest accidents (like erasure). There’s some really good tools to safely archive your collection to DVDs in a way that pretty much guarantees readability decades from now.

The Toolbox

Throughout this blog you’ll see references to software that makes the MP3 experience a whole lot easier. Some is freeware, and some is commercial software. Links and descriptions will gathered together in this section for easy reference, and there’s a link at the top of every page so you can find it fast.

Glossary of MP3 Terms

Confused about an MP3 term or acronym? I’ll put up a section, also accessible at the top of every page, that pulls them together for convenience and reference.

So that’s the plan. These posts will be forming the guts of an eBook – read about that project here – and I’ve put together a sweet offer that I hope you check out.

Sorta disruptive, wouldn’t you say? Maybe I’m a closet anarchist.

But there’s one other thing I want this blog to do. And that’s to pay its way, and to pay for my MP3 addiction along the way. There’s three ways this will happen.

Advertising

First up, on the blog pages there will be adverts that take you to various (relevant) offers on the web. If you click on these and end up buying something, I get a cut. If that bugs you, don’t click on the advert. All the information on this blog is still available to you, for free.

Affiliate Links

Second, in some posts (not many) there will be affiliate links to some product or another. Here’s an example:

Ipower Web Hosting (aff link)

Affiliate links have a tag that identifies me as the referrer; if you click the link and buy something, I get a cut. I will clearly identify an affiliate link, plus I will give an alternative non-affiliate link to the same product. Here’s the alternative to the same offer:

Ipower Web Hosting (non-aff link)

You make the choice. Obviously I would prefer to make money, and the cost is the same to you; but it’s your choice and I won’t try to hide the true nature of the link.

The eBook

Finally, all these posts are being collected, expanded, massaged and formatted into the basis of an eBook on MP3. Hell, may even end up as one of those paper things you get at Borders. But it will contain a lot of information about the legal and illegal (in most countries) use of MP3s – finding, downloading and uploading, organizing, playing and archiving them.

You probably guessed the eBook will cost money. Well, yes, it will. But here’s the kicker. I am actively seeking input from readers of this blog. I want the eBook to be the best it can be, and I figure it’s a good idea to solicit suggestions about content and format from you. To encourage this, I’ve put a deal together that may interest you, and it’s detailed on this page.