Efficiently Reading ID3v2 Tags Over HTTP?

Paul Crowley asks: "Given an HTTP URL for an MP3 file, what's the best way to read its ID3 tags on a GNU/Linux system? It shouldn't be necessary to fetch the whole file: HTTP byteranges should make it possible to fetch only the tiny fraction that's needed, for a big saving in network bandwidth. However, existing ID3v2 libraries are designed to read local files. Extending these libraries for this purpose, or implementing a new one, would be a big job. What's the clean solution - is FUSE the best way, or is there a simpler way that doesn't require root privs? Can I do it using the existing id3lib binary?"

Re:Not really answering your question..

[Ashish Kulkarni]

ID3v2 data *IS* at the start of the file, normally takes the first 4kB or so (depends on the padding settings).

ID3v2 Sucks

[DeadSea]

As somebody who has tried to write libraries that read ID3v2 tags, I'd have to say I hate them. The standard is clear and well documented, but the chosen format is horrible. It is very hard to write a parser correctly. It would have been so much better to embed an XML document at the front of the MP3 file. Instead they decided to make each field in a special binary format prepended by a length field.

The number of checks you have to do is phenominal. The biggest worry is buffer overflow where the length given is greater than the actual length of the tag and you read more than is in the file. There are just hundreds of such edge cases. Libraries for ID3v2 are likely to be buggy, crashy, and just no fun.

Re:Er...

[cjpez]

Well, *actually* from what I understand, the id3v2 tags *can* be put anywhere in the file, so you could have (for instance) an hour-long mp3 of a radio stream and have the tags change with each song. I'm not sure if anything actually supports that, though. Regardless, if you tag with id3v2, it'll be right at the beginning.

Re:ID3v2 Sucks

[foofboy]

I hear you on that. I wrote a python module [sourceforge.net] for id3v2 tags. It reminded me of nothing so much as ASN.1/BER/DER.

It does, however, support arbitrary character sets and arbitrary binary formats, though. Not sure there's another way to do it. Vorbis-comments are ASCII only, right?

I look forward to your reply.

Re:HTTP 499

[eyeball]

From the ID3v2 FAQ [id3.org]:

Q: Where is an ID3v2 tag located in an MP3 file?

It is most likely located at the beginning of the file. Look for the marker "ID3" in the first 3 bytes of the file.

If it's not there, it could be at the end of the file (if the tag is ID3v2.4). Look for the marker "3DI" 10 bytes from the end of the file, or 10 bytes before the beginning of an ID3v1 tag.

That's the problem -- it could be at the end, requiring you to spin through all x bytes (most likely megs) until you get to the end.

MP3::Info Perl module

[extra88]

Look at the MP3::Info [cpan.org] Perl module, you might recognize the author's handle. It reads (and writes) tag info. It's used by the "jukebox" module Apache::MP3 [cpan.org] (sample site [modperl.com]) to generate pages with track info.

Basically every web jukebox out there does something like this so I'm sure there's plenty of other code available to work from. The mod_perl way is to put SetHandler perl-script then PerlHandler [name of module] in your httpd.conf file so when a URL request falls within that Location or Directory, the perl module handles returning whatever you want it to return.

Re:HTTP 499

[cryptor3]

That's the problem -- it could be at the end, requiring you to spin through all x bytes (most likely megs) until you get to the end.

Yeah, that could be true, but if it's not within say, the first 100KB, then the smart thing to do is to stop trying to find it and just return an error.

If it's not at the beginning, you could then use byte ranges to try to fast forward to the end and guess that it will be within the last say, 50 KB of the end.

Re:ID3v2 Sucks

[Fweeky]

foobar2000 [foobar2000.org] uses APEv2 tags on MP3's by default; the standard's just as flexible (well, as much as anyone wants anyway), but, well, you just need to compare filesizes for their handlers; an ID3v2 reader/writer I saw was ~150k of code -- the APEv2 one was 15k. They're always at the end, but obviously since fb2k is the only player I'm aware of which supports it the appeal may be limited. You can at least mix them with ID3v1, which should be good enough for portables.

And before anyone goes off on one because it's non-standard, I'll point out that MP3 has *no* provision for metadata. ID3v1 and 2's are just as arbitary addons as APEv2; they're just older (and lamer, either in big limitations or extreme overcomplication).

I believe the recommended *standard* way of attaching metadata to an MP3 now is to put it in an MP4 container, which has it's own more sensible format. Again, I'm pretty sure foobar2000 (maybe with some plugin in the Special Installer) can put them in, and I think they should play on anything which knows about MP4. Fully reversable too.

CDDB: Feel the Pain

[cpeterso]

Here is Netscape's JWZ hilariously sad-but-true rant about the ID3 header format:

CDDB: Feel the Pain [jwz.org]

In case you didn't know, the file format that CDDB (and FreeDB) use is complete garbage. In addition to random idiotic crap like it being impossible to unambiguously represent a song title that has a slash in it, it's rocket science to figure out how long a song is supposed to be. I need this info not only to display it in Gronk (my MP3 jukebox software), but also for some error-checking that my CD-ripping scripts do, so that I don't end up with truncated files if there was a crash or a full disk or something.

So get this. CDDB files contain junk like this:

# Track frame offsets:
# 150
# 18265
# 32945
# 49812 ...
# Disc length: 3603 seconds
#
DISCID=...
DTITLE=...

(You'd think that the fact that it's in a comment would mean something, but no: you have to parse both comments and non-comments, begging the question of what they thought "comment" means.)

Those numbers are the starting sectors of each track on the disc. There are 75 sectors per second. So you convert those to seconds by dividing, and then find the length of each track by subtracting each from the previous. Oh, but wait, they don't give you the sector address of the end of the last track: for that one, it's expressed in seconds instead of sectors, for no sensible reason. Still, the info is there, right?

Uh, almost.

It turns out that if the last track on a CD is a data track (an ISO9660 file system) then there is a gap between the last track (the data track) and the second-to-last track (the last audio track.) This gap is exactly 11400 sectors (152 seconds, 2:32.) On some discs, you can actually see this track, it's a differently-shiny ring. Why's it there? I don't know. Why's it that size? I don't know. What if the data track is not the last track on the CD? (Does that even work?) I don't know.

So what this means is, when computing the length that a track should be, you have to subtract 152 seconds from the length of the second-to-last track, only if the last track is a data track.

How do you tell whether the last track is a data track, without having the CD in question physically in your computer? By hoping that the CDDB file contains the words "data track" in the title of that track, I guess. Yeah, that's reliable.

And, just to keep things interesting, it turns out that older versions of grip and cdparanoia didn't skip over this gap when ripping: instead, they would append 152 seconds of silence onto the end of the second-to-last track. So now my script that sanity-checks the lengths of the files has to consider two different lengths to be "right", since I now have CDs that were ripped both ways.

Whee. I love love love supporting standards invented by 12-year-olds.

Of course the reason that I use CDDB files at all in Gronk is because of the mind-blowing worthlessness of ID3 tags (32 character limits on titles, etc.) Yay more standards invented by 12-year-olds. (Please don't even mention ID3v2 or Ogg. I laugh at you, you silly person. Those are universally-unsupported fantasies that simply trade one set of problems for a whole new set of problems.)

And as if CDDB wasn't bad enough, FreeDB has taken the CDDB braindeadness and layered even more braindeadness on top of it: it is truly a thing of wonder.

For example, go ahead and try to ever have the "genre" field be something approaching reality -- oops! The first person who ripped this CD said it was "folk" because that's genre number zero! So fix it and resubmit it to the database? Sorry! You can't ever change the genre of an entry in the database after creation, since the genre dictates what directory the file goes in on their server. And so on.

It's a wonder anything works at all.

Re:Silly Question - NOT

[Anonymous Coward]

You know from the mp3 spec where the tag will be.

If you had any knowledge of ID3x2, you'd be aware that you DON'T know where the tag will be. It can be placed pretty much anywhere within the file.

Vorbis comments UTF-8

[rillian]

Vorbis-comments are ASCII only, right?

No. The field names are ACSII only (actually a printable subset minus '=') but the contents of the fields are specified [xiph.org] as UTF-8.

The intention was you could put arbitrary binary data in there too, but there's no general mechanism for marking it as anything else. So any non-UTF-8 use would be application specific.

Re:ID3v2 Sucks

[pdh11]

The standard is clear and well documented

...and never followed. In particular the bit about text being either ISO Latin 1 or UTF-16 (or, in later versions of ID3v2, UTF-8), which is a very sensible idea, is always completely ignored; the overwhelming majority of tag writers, both on Windows and Linux, write text in arbitrary 8-bit encodings (shift-JIS, GBK, whatever) and then mark them as being Latin 1. There's nothing a tag reader can do about that, as there's no way to work out what the writer's locale was. Taglib can write Unicode tags correctly, but no front-end for it that I've seen does the Right Thing: use Latin 1 tags if all the characters used are available in Latin 1 (or, given the problems above, US-ASCII) and UTF-16 tags otherwise.

The problem isn't the standard. It's the implementors.

Peter

You still don't get the fundamental problem?

[danielsfca2]

This is where you tell us how you expect to find out which sections of the files contain the tag. The "libraries that read ID3v2 tags know exactly where and how much to read" because of the fact that they have access to the whole file! If you want that kind of perfect efficiency, then you'll have to download the whole file!

If you want a solution that will allow you to escape downloading the whole file, just check for ID3 in the first 3 bytes and 3DI @ 10 bytes back from the end. Download a couple K in the appropriate direction (more if you expect more than a minority to have images). Neither means no tag, move on. Then parse the tags with your libraries, catch errors, and try to grab more on the few that will throw errors, in case someone put a huge image or War and Peace in a tag.

It's not that hard, and I'm not sure what sorcerer's magic you expected Ask Slashdot to come up with to help you with this.

I think we can all agree on one thing though. Whatever asshole who decided that a tag at the end of the file is a good idea needs to be smacked in the head.