Music Filesystems? 89
Cutriss asks: "I suspect the odds are fairly good that a high percentage of Slashdot readers have a folder, disk partition, network share, or even a hard drive dedicated to storing Oggs or MP3s. Given that this is the case, wouldn't it be beneficial to develop a filesystem optimized for such a use? For instance, in Windows, Microsoft has special folder layouts for folders declared to have digital music or images in them. The digital music folders don't bother with file permissions or any unnecessary data, but instead display your songs with the appropriate ID3 tag information instead. If a filesystem were developed that intrinsically indexed its files by ID3 tags (or similar mechanisms), wouldn't this be an ideal structure for storing and searching through your album collection? And, more to the point, are there any such efforts under way to develop such a beast?"
Metadata (Score:3, Insightful)
Either way you're going to need a custom tool to do searching and whatnot, the custom tool might as well just index the stuff in the ID3 tag and work with an existing filesystem.
- Steve
Re:Metadata (Score:2)
Re:Metadata (Score:4, Informative)
There are a lot of reasons why filesystem-level metadata is harder than it sounds. (XFS supports it, incidentally, though extended attributes. There are command-line tools, and also a C API. Dead simple.)
Filesystem-level metadata is usually stored in the inode itself. If it's not stored literally inside the inode, then it's tightly coupled to the inode. This is good; it makes it fast. Reading a big chunk of metadata (say, 128K) is about as fast as doing a stat(). So that's good.
But most applications do something tricky with their data files. Say, Photoshop for example. When you're working on a file (foo.tiff) in Photoshop and you tell it to "save," Photoshop doesn't truncate and write to the file foo.tiff. Instead it writes to a temporary file. When the write is complete, it unlinks foo.tiff and renames the temporary file "foo.tiff." This is also good, because it makes it very unlikely for a program or system crash to destroy foo.tiff, even if saving the file takes a really long time. A crash or other interruption during the save process leaves foo.tiff just as it was; at worst, it leaves a dangling temporary file laying around.
Here's the catch: when Photoshop saves that new temporary file it gets a new inode. That inode has no metadata in it. All the metadata associated with foo.tiff is in foo.tiff's inode. When the temp file has been written and Photoshop unlinks foo.tiff, that wonderful rich metadata disappears in a tiny puff of blue smoke.
Sure, there are ways to work around this. We could implement a system call like clone() or something that allows an application to get a new, empty file open for writing that has all the filesystem metadata from another file automatically copied to it. Or we could do any number of other clever things. But all of them-- at least to my knowledge-- would involve changing end-user applications. That makes it tough.
ID3 tags are different, because they're stored as part of the MP3 file itself. Copying the file's data, using any method you might choose, will also copy the ID3 tags. But inode-based filesystem-level metadata is something else entirely.
Re:Metadata (Score:1)
There is no technical reason why you can't have temporary files [or their modern equivalent] in a failsafe environment that persists even when the computer crashes.
I agree with the new metadata file approach, it's sort of like the mac HFS. But, I would also use a local metadata database to backup metadata transfers when the excrement hits the air-cooling system.
It's possible to store these ID3 tags as part of the XML MIME info, then you can search for music at filesystem access speed rather than read the guts of the file for the ID3.
And before you ask: yes, this is a part of the Kaos operating system I'm working on.
Re:Metadata (Score:2)
Re:Metadata (Score:1)
As for me being an idiot, a common method of implementing AI programs is to use insects as the template.
I am merely going one step further by defining an evolutionary basis for individual agents that behave in an emotional manner.
It is possible that you are one of the geeks who is insulted by the use of game theory, a method widely used to anticipate change.
And if you are one of the herd of americans who views the internet as a place to insult foreigners, I don't care.
I'm lucky enough to live 1 minute from the beach, in the capital city of a country that is home to Lord of The Rings.
I really don't have time for petty personal disputes whenever there is a procedural expert or 'ideas person' around, I'm just trying to get things done & I have no time to waste on trivial matters raised during consultation.
Re:Metadata (Score:2)
But the fact that you live in a nice place doesn't make you any less of an idiot. I don't mean that in a perjorative sense; I just mean it literally. Your posts and journal entries reveal you to be a person of less-than-average insight. Nothing personal.
Please, by all means, prove me wrong. Share some piece of wisdom that makes us all stop and think. Say something to make me reply, "Yeah, that's a good point." Throw me a bone here. My opinion will change quickly in the face of new information.
I really don't have time for petty personal disputes whenever there is a procedural expert or 'ideas person' around, I'm just trying to get things done & I have no time to waste on trivial matters raised during consultation.
Well, good luck with that. Let us all know how it works out for you.
Re:Metadata (Score:1)
If I was an idiot, I would give you the full picture on certain technology which is banned from development in the USA.
Nothing personal, but the NSA aren't the best people to deal with new and interesting information.
I would be glad to give you new information I have, but you would find out that Big Brother doesn't respect the laws or the constitution of your country at all.
If I am successful in my current project, the technology will find it's way to freedom. If I fail, it will never be heard of again. [and nor will I]
Metadata in a database file system (Score:1)
By using MIME, your files will always open in an app that can handle those files regardless of which app made it.
Of course, this is something I've been doing for a few months now as a vital part of the Kaos operating system.
Not a specialized one, no (Score:3, Informative)
That all sounds like BeFS to me. -- which, lo and behold, was discussed on Slashdot [slashdot.org] just the other day!
What other general features might be good for the specific application of music?
gosh (Score:4, Funny)
Usually this sort of presentation is up to the application, not the filesystem, and Windows is no exception here. The thing presenting the "My Music" folder with ID3 tags and whatnot is the shell (explorer.exe); it's a graphical file browser. There are many of them available for Linux as well.
And personally, I don't really care if my file browser supports this, as long as my MP3 player can read ID3 tags...
Please generalize that idea (Score:3, Insightful)
And please, please, at least store the file's mime type, don't play the extension guessing game.
Web server: let's guess what the file's mime type is based on the extension, and send it.
Browser: Let's guess what this file is, given the extension AND the mime type.
Stupid computers!
Re:Please generalize that idea (Score:1)
Re:Object Oriented Filesystem (Score:1)
AmigaOS got it right. (Again.) (Score:1)
We have countless examples of how hard this sucks. It ranges from an annoyance for Mac users dealing with files that came through a non-Mac system all the way up to lost time and money when users spread
The better way is plain: use an intrinsic property of the data in a file. Put another way: Decide what data a file contains by looking at what data the file contains. That's exactly how the Unix 'file' command works. It matches bit patterns in the file against the "magic" database to determine a file type.
Of course, there are file types not easily identified by simple bit strings. Starting with v3.0, AmigaOS had it right with the DataTypes system. Part of DataTypes was the 'descriptor' which provided the system with three ways to identify types of data: a bit string, a callback function, and a filename pattern. These three can be combined, as well. Thus, it's possible to identify a JPEG image file by looking at the first four bytes, to tell the difference between an MPEG video stream and an MP3 with the callback function, and to fall back to identifying unstructured files, such as a raw dump of PCM sound data, by filename.
Why oh why has no other system adopted this cool system?
More proofreading goodness from Cliff! (Score:1, Redundant)
I thought that it was "introducing [dictionary.com]", but what do I know? Cliff may be using a word that I've not been endtroduced to yet!
Re:More proofreading goodness from Cliff! (Score:1)
Endtroducing (Score:1)
Stupid Windows My Music directory (Score:2)
Mr. Spey
Re:Stupid Windows My Music directory (Score:1)
Actually, you can sort by date, it just takes a bit of work.
First, you have to view the My Music folder using details, as opposed to small icon view or whatever you are using. Right click on one of the column headers (such as name, artist, etc.). This will bring up a menu in which you can check date modified and/or date created. Each selection you check will then be added as a detail you can view for each file. Then just click on the date modified or date created column (whichever one you added), and the folder will sort itself accordingly.
Voila.
Re:Stupid Windows My Music directory (Score:1)
Re:Stupid Windows My Music directory (Score:2)
PHB: Yes, I do look thinner today. It must be the situp I did yesterday.
Mr. Spey
metadata (Score:3, Interesting)
A better solution is to take advantage of the flexibility of a modern Unix filesystem, where any charater (excepting '/') can bu used in a file name, including the newline. There is no reason why you couldnt name a file:
Title: Symphony 2
.
.
. .mp3
Composer: Bach, Johann Sebastian
Composition Date: a long time ago
Performing Artists: St. Louis Symphony Orchestra
Performance Date: not very long ago
Genre: Classical
Format:
The Unix filesystem is adequate for the job, metadata is a curse disguised as a blessing.
Re:metadata (Score:1)
the parser would be easy on such a plan. Just split the filename by newlines for each metadata item, and then by colons for the datatype:value sets. Searching would be easy as can be, just use the good old-fashioned locate and find commands to do the searching. As for readability, what is more readable than this? Not much. This seems like a non-issue to me.
Re:metadata (Score:2)
Re:metadata (Score:1)
Re:metadata (Score:2)
ID3v2 is still a standard written by people who don't know how classical music works, and don't really seem to care. Hell, they have a place to list the people in pop music's album credits, but they don't have a way to note that a track may be a single movement of a larger composition, the closest you can come is noting the original track number on the cd, which isn't what you really care about anyway.
Re:metadata (Score:1)
So here we are. Id3v2 allows you to define your own fields, including year of composition, name of the piece that the track belongs to, and what your mother bad for breakfast. Applications can choose which fields they show, and which fields they let you define. So now I will let you complain that id3 sucks because your mp3-application doesn't allow you to specify the year of composition or your mothers breakfast habits.
Re:metadata (Score:2)
Instead of designing a new FS, writing drivers and apps, we could use your kludge.
A short script could extract the idv3 from the mp3s in folder A, and copy them accross to partition B with a stupid filename. The stupid filename is therefore all the metadata you could ever want to contain.
Then all you need is a few apps to do whatever you might want with this metadata...... erm, ideas people?
Re:metadata (Score:1)
http://ben.ursux.com/?func=code
Re:metadata (Score:1)
Re:metadata (Score:1)
I think you are also confused about what people mean when they are saying "metadata" in this context. The more common term for it is probably "extended attributes." It provides a trivial mechanism for storing key/value pairs associated with an inode. There's no reason you couldn't have a "Composer" key, or any other kind you wanted.
A truly intelligent file browser could be configured to display metadata selectively, sort on it, and so on. That may become a reality in Linux as a metadata-capable API emerges.
I tend to agree that metadata-in-files is a messy approach. It is better to have a distinct source for it, and filesystem-based metadata might be the solution. I guess we'll have to wait and see if it really is.
Re:metadata (Score:1)
Metadata is a better way of doing things, it is possible to use XML for MIME filetyping so that the browser is capable of displaying data in the way you want.
As for reality of doing it in Linux, you'd need major support by a few major distributers or the main programmers to get the critical support that Linux needs for a projecty such as this to survive.
I know this is not the party line, but BSD has a better chance of getting this supported as it is a decentralised group of programmers.
Linus Torvalds is on record as saying that he needs a full time deputy to take care of non-core duties, yet a new filesystem would require the full cooperation of the kernel programmer.
Re:metadata (Score:1)
Again, let me emphasize: XFS and JFS already can do this. XFS did it first, and the JFS people were primarily waiting for de facto standardization of the API (they will be using the same one as XFS uses). All we need now is for programs to start taking advantage of the existing functionality. In this case, we need programs that provide ID3-like functionality and an mp3/ogg browser that does the same.
I will concede, however, that we need more people to use EA-capable filesystems for this really to take hold. Changing the "default" filesystem on the major distros would sure do that. But implementing the EA stubs (which may already have been done in 2.5 kernels) for e[23]fs would also do the trick, and pretty much leave Reiser as the only modern Linux FS that (last I checked) doesn't support EAs.
Re:metadata (Score:1)
I actually consider extensions in the filename superior to storing the MIME type, or keeping a seperate extension like in DOS. If you think about it, keeping the MIME type seperate from the filename is exactly the same as the DOS 8.3 format, just a lot longer, say 255.32. The perfect example as to why this is a bad example is file.tar.gz. Making a seperate MIME type for *.tar, *.gz, and *.tar.gz is just plain stupid. What if I want source.c.gz, should we invent a new MIME type? Can different files have the same name if they have different MIME types? If not, how can I have files like paper.LaTeX, paper.dvi, paper.ps and paper.pdf, a situation that exists often with me. If so, how would I tell it to copy one of those files somewhere and not the rest, without using some bad kludge like the name.ext in DOS (probably something like index:text/html).
I know this is not the party line, but BSD has a better chance of getting this supported as it is a decentralised group of programmers.
I have no trouble believing that, if this was a good idea, it would show up in BSD before Linux. On a related note, I use FreeBSD, not Linux.
Nautilus (Score:2, Informative)
Use a database (Score:2, Insightful)
Re:Use a database (Score:1)
The only problem is, this project appears dead in the water. If anyone knows of an equivelant system, for this or another database, I'd like to know. It makes a neat way of being able to do queries, and still access the files directly through the FS. You can even end up with dirs containing all the files in your query, then just use that directory to input to whatever program you like - nice.
Re:Use a database (Score:1)
Re:Use a database (Score:1)
Linux Extended Attributes (Score:1)
Storing metadata about files in the file system? Could be useful...
And of course it's already been done... [bestbits.at]
Not exactly a good idea (Score:2)
Also you need to store
Anyhow, my main point was this: Can I be the first person to point out that BeFS was/is superb at this?
Re:Not exactly a good idea (Score:1)
pr0n partition full, huh?
Re:Not exactly a good idea (Score:1)
No [slashdot.org], no you cannot.
Re:Not exactly a good idea (Score:1)
I have 6 days & 6 hours of MP3's, there has to be tons of songs that suit porn. Perhaps a slogan for this sort of site would be "come here for MP3 & cum there for porno"?
Higher Level Solution (Score:1)
If you do it at the FS level, you would also have to rewrite things like ls and change the way the stat ioctl works, which would break the posix compliance OS. In short, you'd have to modify a ton of stuff just to get id3 tags to display, not to mention that once you got it all working, you'd have to track the development of all the newest FS features like journaling because people are going to want all that stuff to store their mp3s too.
It would be much easier just to have your filebrowser do it for you, and let the filesystem do its job: store files.
Re:Higher Level Solution (Score:1)
For example: The file system has all the usual info about an MP3 and when a legacy application wants to read a file from it it can generate a filename on the fly like: <artist>-<song>.mp3 or whatever. And directories could also be created on the fly for
Personally I don't see any advantages of a music filesystem over a regular file system, but I think the idea is an interesting one. Maybe there are other applications out there that would benifit with a unique file system, but instead are using a kludge on a regular file system. What if music was stored in a CDDA format allowing a simple copying of the data to a CD when you want to create a CD of your music? What if your regular file system was steganographed on top of your music collection? Twice the space for half the price!
We must not let our old ideals stifle innovation. Backwards compatability can usually be achived and a new format might have added benifits that have not even been explored yet.
Metadata directories (Score:5, Interesting)
NeXT had a great solution to this: instead of just having a file "mysong.mp3", have a directory "mysong" that includes files like data.mp3, title.txt, etc. The file browser presented this to the user in a simple UI (e.g. the directory would have the "mp3" icon in the file browser, double-clicking it would play the song in the mp3 player, displaying the title.txt, etc). But traditional file transfer and backup tools worked just fine, and you could use standard tools (your regular editor, copy program, etc) instead of crufty special purpose ones (regedit, or binhex, or whatever) to work with the metadata.
Think of it this way: you don't want to just have 2-3 defined metadata fields, you would like to be able to add arbitrary new ones in a flexible way. And you want the contents of each to be flexible, without size limits or such. So you want a way of associating names with values--wait, we have that, it's called a filesystem. And you want a way of grouping a bunch of name-value pairs together--wait, we have that, it's called a directory. You'll either end up reimplementing a filesystem, badly, as your metadata, but need special tools to view/edit the metadata or move it around (a la resource forks or the registry), or you'll end up going with a NeXT-like solution and use the filesystem itself (which deals quite nicely with things like this).
Sumner
Re:Metadata directories (Score:2)
Re:Metadata directories (Score:1)
Re:Metadata directories (Score:4)
Not if your filesystem has decent support for small files, like e.g. ReiserFS or some of the recent ext2 devel branches. This is one thing ReiserFS definitely got right (though I'm less sure about the direction they're taking for ReiserFSv4).
As I said, it's a workable approach for grafting extensible metadata onto an existing FS, but it's not how you would design an ideal from-scratch FS.
It's definitely how I would design an ideal from-scratch FS. Metadata in the FS is a really horrible hack that goes against all the namespace unification efforts that OS designers have been making in the last 10-15 years.
Sumner
Re:Metadata directories (Score:2)
Well, the trend seems to be to make the whole FS a relational database anyway. It will be interesting to watch FS developments over the next five years. Of course, the initial efforts seem to be towards a relational DB FS on top of a legacy FS (seems that's what MS is doing with their SQL IFS), but long-term I guess you would see the FS store the DB straight to the metal.
Re:Metadata directories (Score:1)
Re:Metadata directories (Score:2)
It's not an issue of getting beat to it, it's a really old idea. Heck, ReiserFS was hardly the first and they've done it for over 5 years. ext2 hasn't, but there is some movement in that direction.
Sumner
Re:Metadata directories (Score:1)
Similar mood/sound/etc.. (Score:1)
I like the idea of saying "Play Happy/Sad/Fast/Uplifting songs like <song>"
Yeah, I'm dreaming... so what?
Been there, done that with BeOS (Score:2, Informative)
BeOS (god rest its soul) uses the Be Filesystem, or BFS. It is a 64-bit journalling and indexing filesystem. It is essentially a filesystem that works much like a database. You can attach any number of arbitrary attributes to any file. Filetyping is handled by MIME filetypes, which are also stored as attributes. The default BeOS installation includes all of the usual attributes for music files and it is very easy to add many more. The BeOS "find file" tool is not unlike a database query tool.
In terms of speed, file searches are done almost instantaneously, even when searching through several gigs of data.
Re:Been there, done that with BeOS (Score:1)
It really works. And it really works fast.
Viva Be.
Which version of windows? (Score:2)
This sounds like it's a Windows XP thing, and we all know that XP is the devil.
Re:Which version of windows? (Score:1)
Nautilus (Score:2, Informative)
I don't think this has anything to do with the Windows file system. Its just how explorer presents data in certain folders. Nautilus does the exact same thing, for instance. It displays bitrate, time, artist, and presents a mini-player at the bottom of the folder view. I assume it gets the artist info from ID3 tags.
WORM Advantages (Score:1)
A dedicated music partition would probably have files dumped on to it in large chunks, without much deletion or other write activities being performed on it otherwise.
So, just making sure files aren't fragmented when they're dumped on the partition, and maybe caching things like ID3 tags would be useful.
Is there really a problem to address, though?
Re:WORM Advantages (Score:2)
My way of doing it (Score:1)
I organize them into subdirectories, etc. with a standard naming convention. Nothing new here; everybody does this.
The problem I found is that my playlists would break whenever a renamed or moved a file.
Solution:
NB the relative, not absolute, symlink: this allows me to serve up the lists over NFS, SMB, etc.
Altogether a very easy, powerful and flexible system. If something goes wrong, it's just standard files and directories, so I can fix it without editing inodes or dropping into an SQL shell.
A trick you should also know is that just taking the MD5 of an mp3 does work very well since the ID3 is also part of the mp3 and it might change. So I take the MD5 of the last megabyte of mp3 files (the ID3 tag is near the beginning of the file).
One could easily extend the perl script to use ID3 tags to generate directories and playlists. Oh, and the reason I do it this way is because Konqueror is a very nice visual file manager and it allows creating symlinks easily (albeit absolute ones, but my script fixes that).
Anyway, hope these ideas help. Taking the problem down to the filesystem level introduces all sorts of complexity while the problem can easily be solved by a couple hundred lines of perl (writing perl scripts is much easier than kernel work, trust me). Right tool for the right job and all that. This only took me one night to throw together (this includes organizing the actual playlists, which took the most time).
I also have lots of pr0n, and I'm working on ways to automate pr0n management. Let me know and I can get back to you once I've solved the problem.
Re:My way of doing it (Score:1)
sounds like a job for plan9 (Score:2)
But we're stuck with 23 char filenames until the eagerly awaited 9p2000 as I discovered the day I statrted to move my mp3s over, doh!
Store music separately (Score:2)
The filesystem would then transparently re-create the MP3 file when copied out to move it across the network, zip file, etc... by attaching the meta-data back onto the music stream as it is ready from disk.
Thoughts?
Mounting a database as a filesystem??? (Score:1)
Then again... what the hell do I know..
Re:Mounting a database as a filesystem??? (Score:1)
And yes, you're right it is useful to store the ID3 tags in the database FS because you get faster and better search results for files.
What Im thinking (Score:1)
Re:What Im thinking (Score:1)
ID3 tags are in most modern MP3 players, the catagory you want is Religious.
[which covers diet rock, evangelism, boring crap, stuff that sucks and blows at the same time & annoying tunes you hear in elevators]
You could also put mood music in Unclassifiable, which is useful if you are currently listening to stuff you'd hate to hear when you grow up.
Uniform attributes (Score:2)
One idea would be to do this on a directory by directory basis (like 4dos descriptions), this would overcome the mapping problem. If each file type has its own header, then you could store current and future formats in this.
One can have a utility to go through and look at files by type, eg like a "viewer" dll.
Backing up such a file would allow the support of this feature on cdroms under different operating systems.