A Day In The Life
IntroductionWhen I bought myself an Archos Jukebox (Recorder 20) MP3 player I quickly found out that its 20Gb disk would not be big enough to hold all my MP3 files. So the need arose to bring more structure in my collection of MP3 files.
For convenience, all MP3 files are maintained on my workstation, running RedHat GNU/Linux. A subset of the MP3 files are synchronised with the Jukebox MP3 Player. No maintenance takes place on the player itself.
The MP3 Player is running Rockbox, a superior replacement of the standard Archos firmware.
The Data ModelThe basic entity for maintenance is the album, that is an ordered set of MP3 data. In terms of a file system, the album is a directory with MP3 files and, as we will see later, other information.
The albums are organised in a four level hierarchy:
GroupThe Group indicates the status and/or origin of the MP3 files.
Currently implemented groups are:
CategoryThe Category defines the kind of music.
Putting music into categories is a hard and ungrateful task. Most of all, it depends on one's personal taste and mood. I use the following categories:
ArtistThe Artist is the name of the artist, performer, or whatever appropriate for this album.
I use the names as much as they appear on the albums. So Miles Davis is stored as "Miles Davis", although "Davis, Miles" would improve the sorting order. Prefixes like "The" and suffixes like "Trio" and "Quartet" are left out in most cases, depending on the common use. This results in sometimes inconsequent namings like "The Alan Parsons Project" and "The Beatles".
The words of the name are all spelled with title caps.
CollectionsFor collections, albums with tracks of different artists, the artist is set to "Various".
AlbumFor the name of the Album the same rules are used as with Artist.
In addition, when an album is part of a multi-disc set, the title is suffixed by ", Disc N", where N is the disc number.
TracksFor the title of the Tracks the same rules are used as with Artist.
For multi-title tracks a slash is used to separate the titles.
For multi-track titles, a suffix ", Part N" is
For collections the ID3 tags for artist and track title are set to the "real" artist and track title.
Making File NamesWhen forming file names from artist names, album names, and track titles, special care is taken to prevent sensitive characters in file names. Since the files are eventually stored on the Jukebox, that has a VFAT file system, the following characters are left out from file names: spaces (replaced by an underscore), / (slash, replaced by a semicolon), \ (back-slash, same), : (colon, replaced by an underscore), " (quote), ` (back-quote), ? (question mark), ! (exclamation mark), | (vertical bar) and trailing periods.
Note that the ' (apostrophe) is allowed, since it occurs very often in song titles. The character set used is ISO-8859-1 (Latin-1).
The resultant file name can always be protected against Unix command line programs by putting " " around it.
For artist names and album titles, the file name correponds to the name and title. "John Mayall" becomes John_Mayall.
File names for tracks have a two-digit track number prepended, separated with an underscore. In the case of a collection album, the name of the artist is added to the track title.
Some examples of file names and the correponding ID3 information:
John_Mayall/The_Collection/10_The_Super-Natural.mp3 artist: John Mayall album: The Collection track: The Super-Natural number: 10 Various/Acoustic_Blues_Instrumentals/09_Etta_Baker__Carolina_Breakdown.mp3 artist: Etta Baker album: Acoustic Blues Instrumentals track: Carolina Breakdown number: 9
The Internet CD DatabasesThere are several CD databases available on the internet. These sites provide information for a gigantic collection of CDs. The two most popular are GraceNote's CDDB.com and its free counterpart, FreeDB.org.
The canonical "unit of information" is the so-called CDDB entry. This is special formatted data that contains information about the CD, like artist, album, tracks, comments, and some details about the layout of the CD.
The CDDB entries play an important part in managing the collections of MP3 files.
An example of CDDB entry data:
# xmcd CD database file generated by Grip 3.0.0 # # Track frame offsets: # 150 # 90165 # # Disc length: 2822 seconds # # Revision: 11 # Processed by: cddbd v1.4PL0 Copyright (c) Steve Scherf et al. # Submitted via: Grip 3.0.0 # DISCID=060b0402,070b0402 DTITLE=Miles Davis / Bitches Brew, Disc 1 TTITLE0=Pharaoh's Dance TTITLE1=Bitches Brew EXTD=ID3G: 8 Prepared for cd by Teo Macero. Engineered by Ray Moore\nCol EXTD=umbia Records G2K 40577 EXTT0= EXTT1= PLAYORDER=Sometimes it is not possible to get CDDB information. For example, when creating a custom CD. For this purpose I developed a fallback 'pseudo' CDDB format. It is just a header line and a list of tracks.
An example of CDDB fallback data:
Miles Davis / Bitches Brew, Disc 1 1. Pharaoh's Dance 2. Bitches Brew Prepared for cd by Teo Macero. Engineered by Ray Moore Columbia Records G2K 40577
The RepositoryAs described earlier, the Group, category, Artist and Album form the path to the actual collection of MP3 data files for a specific album.
Besides the MP3 files, the following meta-data is stored here, in the form of hidden files:
Note: the playlist files refer to a local HTTP server to serve the MP3 data.
Classification IconsEach entry in the main index page has an icon that designates its status. A hand with thumb up indicates an approved entry. Thumb down indicates a rejected entry, and no thumbs indicates a new, yet unclassified entry. If an album is not complete (one or more tracks are missing) then the hand misses a finger, e.g., . If the sleeve is red, e.g., the entry is currently on the MP3 player.
SynchronisationWhen synchronisation with the MP3 player is desired, a shadow hierarchy of all MP3 files is constructed by traversing the groups (New, Archive, Rips and Rejects) and symlink all the directories that have an .onPlayer file present. This forms a pseudo-group Player that can be synchronised using the rsync program.
I automount the Archos Jukebox on /misc/usd1, so the rsync command needed for synchronisation is:
rsync -av --delete --modify-window=1 --copy-links \ --exclude='.??*' --exclude='*~' \ --exclude=\*.ajz --exclude=\*.ucl --exclude=.rockbox \ Player/ /misc/usd1/
Note that excluding *.ajz, *.ucl and .rockbox is mandatory. Leaving these out would cause the Rockbox information the be removed from the player.
File ManagementFile management is implemented by the mp3fm tool. It supports the following commands:
ID3 Tag ManagementID3 Tag management is implemented by the id3tag tool. It can be run in interaction mode or verification mode.
In interaction mode, id3tag requires the name of a directory that contains the MP3 files and meta data. It load the CDDB data (or the fallback data if no .cddb file is present) and examines all the MP3 files. For each of the files it verifies whether the artist, album, track title and number correspond to the CDDB data and offers to change it. It also offers to rename the MP3 file to its correct form.
Example of a id3tag run (with comments added):
$ ls New/jazz/Miles_Davis/Bitches_Brew,_Disc_1 01_Pharaoh's_Dance.mp3 02_Bitches_Brew.mp3 $ id3tag New/jazz/Miles_Davis/Bitches_Brew,_Disc_1 CD: Miles Davis / Bitches Brew, Disc 1; 2 tracksThis is the information derived from the CDDB data.
1: Pharaoh's DanceThe first track, according to the CDDB data. Since the name of the MP3 file is 01_Faraoh's Dance, it is not automatically found.
2: Bitches Brew File: 02_Bitches_Brew.mp3 Change track title "Bitches Brew" to Bitches Brew? Change track number "2" to 2?The second track is found, and the program offers to change the track name and number. The suggestions given are presented using the readline routines and can easily be edited on-line.
--: File: 01_Faraoh's_Dance.mp3 Change track title "Faraoh's Dance" to Faraoh's Dance? 1This is a file the program could not find a track title for. So it offers a replacement. By answering 1 we indicate that this is track 1, and the program corrects the question:
Change track title "Faraoh's Dance" to Pharaoh's Dance? Change track number "1" to 1? Updating 01_Faraoh's_Dance.mp3The ID3 information is updated, and it now asks to rename the file:
Rename 01_Faraoh's_Dance.mp3 to 01_Pharaoh's_Dance.mp3 ?
In verification mode, no questions are asked but all the MP3 data files, their ID3 tags and the CDDB data are verified for consistency. If the validation succeeds, id3tag adds a file .verified to the meta data.
Combining CDsWhen using an MP3 player, why not combine multi-CD albums into one?
This is, indeed, very handy and easy to accomplish. In particular, if you have a system that supports links, you can link the original tracks into the new location so it doesn't cost you any additional disk space.
The utility mp3combine can be helpful in combining two or more directories into one. Files from the first directory will be prefixed with 1, the second with 2, and so on.
$ cd Pink_Floyd $ ls The_Wall,_Disc_1 01_In_The_Flesh.mp3 02_The_Thin_Ice.mp3 03_Another_Brick_In_The_Wall_(Part_1).mp3 ... $ ls The_Wall,_Disc_2 01_Hey_You.mp3 02_Is_There_Anybody_Out_There.mp3 03_Nobody_Home.mp3 ... $ mkdir The_Wall $ cd The_Wall $ mp3combine --verbose ../The_Wall,_Disc_1 ../The_Wall,_Disc_2 Disc 1: Pink Floyd / The Wall, Disc 1 Disc 2: Pink Floyd / The Wall, Disc 2 $ ls 101_In_The_Flesh.mp3 102_The_Thin_Ice.mp3 103_Another_Brick_In_The_Wall_(Part_1).mp3 ... 201_Hey_You.mp3 202_Is_There_Anybody_Out_There.mp3 203_Nobody_Home.mp3 ...Additionally, the program will create fake .nocddb and .verified files that reflect the new structure.
Getting The MP3 DataSeveral programs exist that can extract the audio data from CDs. Most of them are capable of directly producing MP3 data. Later we'll see why it can be advantageous to extract the audio data in 'raw' WAV format, and compress it explicitly.
Sometimes the extraction program can query the CDDB internet databases and store the audio data in files with names that reflect the artist, album, and track title. If not, you'll end up with a series of files like Track01.mp3, Track02.mp3 and so on.
If the new MP3 files have reasonable ID3 tag values, the mp3rename program can be used to rename the files into a Artist/Album/NN_Track_Title.mp3 form.
The getcddb program examines a directory with MP3 files and queries the CDDB servers. It then stores all CDDB entry data in the directory, with file names .cddb00, .cddb01 and so on.
The next step is to select one of the CDDB data files, rename it to .cddb and edit it. I change the artist, album and track title information to match the conventions described above. Then the id3tag program can be used to change the ID3 data and file names.
In case no CDDB entry can be found, it is worth trying Google or
any other search engine to find the track titles, or type then in
manually from the CD cover, in CDDB fallback format
There are three ways to produce MP3 files that can be played without interruptions.
MP3Gain: Smoothing the volumeIn real life situations, the volume of MP3 rips can differ significantly. Although this may not impose a real ploblem, it is quite annoying at least. Every time you start playing a CD you need to adjust the volume.
There is a nifty tool called
I grew the habit of applying MP3Gain to all my rips. I have it normalise the volume per album, not per track, to retain the relative volume of the tracks of the album.
In Real LifeThese are the steps as I execute them:
Generating PlaylistsPlaylists, as used on the Archos Jukebox, are just a list of mp3 file names. For example, if you store the MP3 files in de top directory "music", and generate a playlist, the file wil contain names that all start with "/music/...".
Since I maintain all the MP3 data on the hard disk of the PC, and copy selected portions to the Archos MP3 player, it is straighforward to create playlists on the PC. For example, I likt to have big playlists (with over 1000 entries) that contain all kinds of tracks in random order, so I can play it in the car. With the new Rockbox Bookmarks, I can stop and resume multiple playlists simultaneously.
I wrote a small tool
All my Rockbox
files reside in a directory
perl playlist.pl --strip /sp3/Player /sp3/Player/music/*/* > playlist.m3u
will write the desired playlist to the named file. The
Playlist contains 450 tracks from 39 directories
The function of the
It is also possible to manually create a list of files and directories, and feed them to the standard input:
perl playlist.pl < filelist.dat > playlist.m3u
I use this to create preselected sets of albums that I want to generate the playlist from, or to exclude some albums from a large list.
FutureCurrently, I'm working on a database application to maintain all the information. Take a look at the Domain model and the DBMS model if you're interested in these things. The pictures are generated by OptimalJ from Compuware.
The database I use is PostgreSQL, a powerful, industrial strength and Open Source database system.
ToolsAll tools discussed above are either self-developed or otherwise free (as in speech) and available on the Internet. I advise against the use of proprietary software tools.
Contact the author for details.
Last modified: Sat Nov 18 21:27:24 CET 2006
© Copyright 2003-2018 Johan Vromans. All Rights Reserved.