Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Why a New Filesystem Matters

By ttfkam in Technology
Sat Aug 09, 2003 at 10:52:50 PM EST
Tags: Internet (all tags)
Internet

Depending on how much you care for writing new versions of old programs, Reiser4 could be either one of the greater boons to software engineers or one of the greater banes. Either way, programming on Linux will likely never be the same. It's not too remote a possibility that other popular platforms will be influenced as well.


Why should a new filesystem be considered a watershed event for programming? Filesystems are a known quantity aren't they?

  • Allow the creation, modification, and deletion of directories.
  • Create, modify, and delete files in directories.
  • Make a file or directory available from multiple locations (symbolic and hard links).
  • Record when the file is created.
  • Record the last time a file is modified.
  • Record the size of entries.
  • Keep track of which users are allowed to access and write to a file or directory.
  • Store and retrieve files as quickly as possible.

As long as they serve the files quickly and avoid losing data, what more could anyone ask for? Sure, there can be variation in the implementation. For example, some filesystems support ACLs in addition to standard, single block, UNIX-style 32-bit permissions. But no matter how one implements a filesystem, it's a static bit repository. According to the dinosaurs such as the Minix filesystem, the FAT filesystem, the ext filesystem and its descendants, this is the case. It has even been the case for the newcomers JFS, XFS, ReiserFS, and their ilk. Speeds change. The amount that can be stored has increased dramatically. Reliability in adverse conditions has improved. No fundamental shifts in usage however. This is the limited role of a filesystem isn't it? Isn't it?

WHY FILESYSTEMS ARE THE WAY THEY ARE TODAY

All filesystems contain meta information -- data about data: a file's size, a directory's creation timestamp, etc. This list of meta information is fixed, however. If I wanted to add a "usefulness" attribute to every filesystem entry for example, I would have to patch the filesystem and quite likely the kernel to allow access to this attribute. In addition, I might break other applications. At the very least, I would have to recompile every application that called the program/routine stat.

When confronted with issues of custom meta information, filesystem architects have largely told application developers to shove off. And for good reason. Very few would assert that the album, artist, and song information in an MP3 file is unimportant. But this information is customarily so small that a great deal of space on the hard disc is wasted. Filesystems are organized into "blocks" of data and these blocks usually contain only one file or directory entry. In order to effectively access these blocks, they are made a uniform size. If your meta information is only 400 bytes, allocating a 4 kilobyte block of storage is gratuitous. Even in today's climate where 80GB drives are normal and larger are commonplace, no one wants to run out of space when the disc is really only half full. In some situations, where the block size is 32 kilobytes for example, the situation is even worse. ReiserFS had tackled the problem with tail packing, but the common usage of the notail mount option is testament to the space vs. performance tradeoff.

Thus ID3 tags were born. Rather than save independent files that are less than 100 bytes in length, they are appended to the MP3 file in a custom data structure, a miniature filesystem if you will. This has caused a few problems over time. People wanted more information than could be provided by ID3, so ID3v2 and its ilk were born. Unfortunately, there were already quite a few programs that understood ID3v1 meta information. In order to add more information, the additional information had to be added in front of the ID3v1 information but, again, after the audio data. Every revision of the ID3 spec has had to do this. To make matters worse, once Internet access became fast enough to stream audio data, online jukeboxes and news streams started to proliferate. Matters are worse because ID3 meta information is at the end of a MP3 audio file. An MP3 file couldn't just be shipped over the wire unchanged as it wouldn't do to find out what was playing until after the song was over and the next one was just starting. Most of us want to know while the song is playing. Thus the spec was further massaged to allow sending these attributes at the beginning of the file...as long as you were streaming.

The same balkanization can be found in image files and the way they store copyright, authorship, comments, and other such meta information. The way comments are stored in a GIF image is markedly different from storage in a JPEG, TIFF, or PNG image ad thus requires a completely different codebase to handle each.

WHY DON'T FILESYSTEMS JUST HANDLE SMALL FILES BETTER?

Some filesystems have tried. It has been a major topic of discussion in academia and among commercial operating system developers for almost as long as there have been filesystems. All the testing, the research, and the math has pointed to a marked loss of performance -- many times a drastic loss in performance. It hardly seemed worth it when existing files were at least 4 kilobytes in length more than 90% of the time. (Of course, if all meta information were stored as independent files, it would be far less than 90% of the time.)

Another stumbling block in the quest for the handling of smaller files is one of data management. After all is said and done, even if filesystems could efficiently handle very small files for metadata, a programmer would run into association issues. Specifically, the directory of hundreds or thousands of MP3s would now contain hundreds or thousands of files that describe the MP3 files. If an MP3 file were deleted or renamed, the corresponding meta information would suddenly describe a file that didn't exist. If one were ever to copy a song, all the associated files would have to be remembered as well. What a pain! Better to attach the info somehow. Once again, ID3 tags seem sensible.

WHAT IS A FILE ANYWAY?

This is an easy one. A file is a sequence of bytes. A directory is a collection of files. Simple, right?

But why is this so? Why must there be a strict distinction? Why cannot a resource be both a file and a directory at the same time? Wait a second! That's crazy talk! How would that work?

From a developer perspective, it's quite simple. Let's say you have a file called foo.mp3 in your home directory. If I try to access ~/foo.mp3, I get the file as is expected. If however I try to access ~/foo.mp3/, I get a directory entry; I get a list of file attributes, meta data. Now our MP3 file could have information that would normally be included in an ID3 tag but accessible through normal file access utilities like Emacs, vi, gedit, etc.

Sounds great, so why hasn't anyone done this. Surely I'm not the first person to mention this. In fact, some people have done this in the past. It never quite caught on due to efficiency reasons on the filesystem that were cited above.

NO! ME FIRST!

Another major limitation of existing filesystems is one of atomicity. One of the most common security issues in software today is the race condition. A quick glance at Bugtraq would show you the scale of this problem. Many system administrators are familiar with the security issues for setuid scripts. UNIX-like systems open executable files and check to see how they are to be executed. A file may be an ELF binary, an a.out binary, a Perl script, etcetera. If a file begins with the characters '#!', the file is passed to an interpreter for processing. For example, if a file begins with the following:

#!/bin/csh

a UNIX-like system will open the file, find that it has what is called a "magic line", and run the script through the program /bin/csh, the C-shell. A file is read to find the interpreter to be used, the permissions on that file are checked, and the interpreter is invoked with the script as input. Seems innocent enough unless a script kiddie has good timing. Imagine that after the permissions were read but before the script is read in and passed to the interpreter, the contents of the file were changed? Suddenly you have arbitrary code running.

Another problem is the use of temporary files. A reasonably careful program will check to see if a file exists prior to creating a file. But let's say that program has setuid root rights. It checks the temp directory for the existence of a file. It doesn't exist. A malicious gremlin makes a symbolic link of the filename to /etc/passwd just in the nick of time. Then the setuid program creates and writes to the temporary file. Oops! Say goodbye to your user manifest and password file.

Currently, tricks like the use of fsync and file renaming are used to simulate atomic file operations in current filesystems. In the end, they can be effective, but they impart an undue burden on the developer to handle correctly. This is assuming of course that the developer even knows how to use these tricks...or that the tricks are even necessary.

These attacks can be difficult -- although some programs have made them entirely too easy -- but if a filesystem had support for transactions similar to the functionality of advanced databases, they could be avoided. Imagine being able to link multiple operations into a single atomic component. The programmer could conceivably make the check for file existence, open, and write and be assured that the outside world was isolated from the process. The fix for race conditions becomes only a couple of lines of filesystem transaction code instead of POSIX hacks that rely on side effects to function properly.

WAIT! I'M NOT DONE YET!

Atomicity also rears its head when performing multiple, related operations. Let's say that you have a queue where you want to delete one file, create another, and update a log. With transactions, it would be possible to enforce that all operations succeed or none succeed. No more checking for dangling files when the program crashed in the middle of an operation. If the replacement file creation or the log addition fails, the source file would not be deleted.

So why don't filesystems do this already if these are such good ideas? Once again, speed concerns raise their ugly heads. So far, the speed loss for what are a minority of filesystem accesses has outweighed the development cost passed on to application developers.

THE TIMES THEY ARE A CHANGING

But what would happen if the efficiency issues were no longer an obstacle? What if a filesystem could be made that could store files of arbitrarily small size without wasting 90% of available storage? What would happen if that filesystem was called ReiserFS v4 (hereafter referred to as Reiser4)?

Recently on the Linux kernel mailing list, Grant Miner posted some speed comparisons on ext3, JFS, XFS, and Reiser4. CPU usage for Reiser4 is higher than the others but so is the performance. Even with its superior handling of very small files, treatment of entries simultaneously as files and directories, transaction support, and coupled with the fact that this is pre-release software that is very much still in development, it is faster than any other filesystem on Linux!

It doesn't take a huge leap of logic to assume that if these speed advances are corroborated by others independently and the filesystem proves itself stable enough for production use with vital data, it will supplant ReiserFS v3 and ext3 as the dominant filesystem used on Linux. If this comes to pass, many programs will be written to take advantage of Reiser4's advances.

Take /etc/passwd for example. As keeping passwords out in the open is a bad idea for security -- even if they are encrypted -- many distributions use shadow password files. The original file which contains the user IDs, home directories, default shells, etc. is left publicly readable. The password hashes however are placed in the shadow file which is not publicly readable. So far, I haven't told anyone who's spent any appreciable amount of time on a UNIX-like system anything new.

Keeping in mind everything you've read about the capabilities of Reiser4, the ability to efficiently manage small files, if /etc/passwd were to be split up into multiple files, one for each user, each file could have permissions set that would match the user. A shadow file would no longer be necessary as the user's password hash does not need to be protected from the user; They already presumably know their password.

But this, of course, breaks older programs that read and edit the password file. It is for this reason (among others) that Reiser4 supports filesystem plugins. One such plugin is a directory aggregator. Whereas before it was mentioned that a file could also be a directory, a directory can appear as a file. In this case, the directory would appear as an aggregation of files separated by a delimiting character (such as '\n'). Taking this example to its logical conclusion, each field could be a separate file. So to find a user's preferred shell (in this example, user ID is 107), one would access the file /etc/passwd/107/shell. Changing the shell would be a simple matter of:

echo '/bin/bash' > /etc/passwd/107/shell

Since user ID 107 owns the /etc/passwd/107/ directory and all of its descendants, a setuid program/script isn't necessary. Seeing possibilities yet?

A convenient side effect of a modular plugin-based filesystem and the blurring of what is a file (versus an attribute or a directory) is that Reiser4 helps to fully realize the UNIX goal of making everything a file: even the directories are files.

I CAN'T BELIEVE NO ONE HAS DONE THIS YET

Some have done similar things in there filesystems or supplementary utilities. The functionality is similar to fsattr on Solaris, XFS's Extended Attribute, and others that aren't as widely used. The primary difference between Reiser4's implementation and these others is that an attribute is not fundamentally different from a "normal" file; No changes need to be made to existing programs and libraries in order to take advantage. There is no limit to name or content that does not already exist for directories and files. However, there is an interface (not yet finalized) that allows access to ReiserFS attributes with a single file descriptor and call to fopen in a manner similar to the Solaris and XFS. Without this interface, a file descriptor and a call to fopen would be necessary to retrieve each attribute -- a certain performance killer for multiple attributes.

BeFS made great strides toward implementing the filesystem as a database but ultimately scaled back their ambitions to attributes and indexed access.

MacOS has had extended file attributes as well, but not truly at the filesystem level. An application layer was added to the system that tracked attributes. Work at Microsoft for the next version of their operating system similarly has scaled back their initially ambitious plans for the filesystem to function at a higher level.

However all of these efforts underscore an industry-wide trend. Metadata belongs with the data, not with the application.

WHY WOULDN'T IT BE USED?

Inertia. It can't be ignored. MS Outlook has been one of the most exploited vector for viruses ever seen (if not the most exploited) and yet a large number of people still use it. Assuming another program came along with all of MS Outlook's functionality but none of the security holes, it would still take some time for people to make the transition.

The fact that Resier4 would still be POSIX compliant -- would still support last modified timestamps, file permissions, et al. -- makes the transition easier. But systems would have to be upgraded and some of those systems have been up for years and are still running kernel version 2.2 or earlier.

Assumptions with regard to future application development will have to be revisited. Software developers are no different from most other segments of the population. When they find something that works for them and has worked for years, they will not likely give those practices up easily.

How do you convince a mail server author that each message can be a separate file and the mail headers can be attributes of that file when conventional wisdom for years has stated quite the opposite? But imagine how much simpler the daemon would be if it didn't have to spend so much time managing data or parsing through a mailbox separated by ^G control characters.

All audio files could have the same interface for attributes without regard to whether they have the extension *.mp3, *.ogg, *.wav, or anything else under the sun. ID3 tags and their equivalents in other file formats would be redundant and unnecessary code complexity.

Suppose you could embed the MIME type in the file attributes directly instead of relying on a three-letter extension when serving files to the web; The web server would just know.

Discard excessive usage of fsync and weird file renaming hacks while increasing security.

Depending on how much you care for writing new versions of old programs, Reiser4 could be either one of the greater boons to software engineers or one of the greater banes. Either way, programming on Linux will likely never be the same. Reiser4 may be the first, but I doubt it will be the last. Rather, I think the ante just got upped for filesystems and Reiser4 is raising the table.

You can read more at the ReiserFS homepage. There you can find more information about Reiser4, its design goals, filesystem transactions, and blurring the line between files and directories.

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
Will the features of Reiser make a difference?
o Not at all: totally useless 7%
o Not at all: it'll never work as advertised 8%
o Eventually in twenty years 40%
o Damn Reiser4: It changes everything 1%
o Hail Reiser4: It changes everything 40%

Votes: 103
Results | Other Polls

Related Links
o ACLs
o ID3 tags
o fsync
o some speed comparisons on ext3, JFS, XFS, and Reiser4
o fsattr
o fopen
o ReiserFS homepage
o Reiser4
o its design goals
o filesystem transactions
o blurring the line between files and directories
o Also by ttfkam


Display: Sort:
Why a New Filesystem Matters | 286 comments (260 topical, 26 editorial, 1 hidden)
it doesn't matter to me (1.08 / 48) (#3)
by DJ Glock on Sat Aug 09, 2003 at 05:49:43 PM EST

and i'm sure it doesn't matter to the millions of children that U.S. troops are mercilessly slaughtering overseas in your name.

the world doesn't revolve around you and your linux computer.

*** ANONYMIZED ***

Then read something else schmuck (3.33 / 9) (#8)
by Simowen on Sat Aug 09, 2003 at 05:59:02 PM EST

This is a well researched article. It contains plenty of information without reading like a technical manual.

[ Parent ]
you suck (2.88 / 9) (#34)
by Battle Troll on Sat Aug 09, 2003 at 08:59:56 PM EST

What a lame ripoff of elenchos's intellectual property. Here's a nickel, kid, go buy yourself half a wit.
--
Skarphedinn was carrying the axe with which he had killed Thrainn Sigfusson and which he called 'Battle Troll.'
Njal's Saga, ca 1280 AD
[ Parent ]
looks to me like those kids are alive [nt] (2.00 / 2) (#75)
by durkie on Sun Aug 10, 2003 at 02:35:32 AM EST



[ Parent ]
yeah but... (none / 1) (#95)
by Fuzzwah on Sun Aug 10, 2003 at 06:22:13 AM EST

Saddam gassed his own people....

--
The best a human can do is to pick a delusion that helps him get through the day. - God's Debris
[ Parent ]

Extended filesystem attributes? (3.75 / 4) (#7)
by komadori on Sat Aug 09, 2003 at 05:53:00 PM EST

It sounds like Solaris extended filesystem attributes but without openat(2) and friends.

See fsattr(5), http://docs.sun.com/db/doc/816-0220/6m6nkorp9?a=view

"When we are victorious on a world scale I think we shall use gold for the purpose of building public lavatories in the streets of some of the largest cities of the world." Vladimir Illich Lenin


cross-platform interactions (4.92 / 13) (#18)
by anon 17753 on Sat Aug 09, 2003 at 07:08:27 PM EST

It sounds like a good thing, but I'm wondering what can be done to help play nice with other OSes/filesystems, that is, getting the expected, user-friendly results? If I copy an MP3 from my Mac to my Linux/Reiser4 there should be a trigger to extract the ID3 tags into the Reiser4 meta tags. Likewise, when copying a file from Reiser4 to another machine on the network, the meta data should be automatically rolled into the appropriate in-file meta storage for that particular file type.

Play nice with the old dumb OSes/filesystems and you'll be much more attractive to people who work in a heterogenous computing environment.

As a matter of fact (4.66 / 6) (#37)
by ttfkam on Sat Aug 09, 2003 at 10:38:31 PM EST

That is possible through the use of plugins.


If I'm made in God's image then God needs to lay off the corn chips and onion dip. Get some exercise, God! - Tatarigami
[ Parent ]

What services are provided by a file system? (3.75 / 8) (#23)
by HidingMyName on Sat Aug 09, 2003 at 07:42:52 PM EST

You don't really address the fundamental issues, which include
  • Persistence = Data objects need to have lifetimes that exceed the lifetimes of the process creating and accessing these objects.
  • Naming - How to uniquely identify a persistent object without a handle.
  • Coherence - All users see the same data
  • Consistency - When do updates become visible
Efficiency and performance concerns and small file handling are important, but some file systems (e.g. the BeOS file system) have SQL support, and others such as OceanStore, and CODA/Intermezzo handle issues of mobility and distributed data management, while the EROS operating system has some interesting persistence semantics.

Answers (4.00 / 1) (#80)
by lambda on Sun Aug 10, 2003 at 03:05:34 AM EST

SQL support, or whatevery querying system you want, can be added through plugins. Or Namesys might write it for Reiser 5. Intermezzo and CODA are layered on top of any regular filesystem, so can be used with Reiser 4.

[ Parent ]
In other words (none / 0) (#129)
by HidingMyName on Sun Aug 10, 2003 at 04:19:44 PM EST

SQL support, or whatevery querying system you want, can be added through plugins. Or Namesys might write it for Reiser 5. Intermezzo and CODA are layered on top of any regular filesystem, so can be used with Reiser 4.
Namesys doesn't yet have an answer to the fundamental problems and you aren't prepared to discuss progress by any other file system designers or groups.

[ Parent ]
Another comment by omghax (3.62 / 8) (#26)
by omghax on Sat Aug 09, 2003 at 08:05:10 PM EST

"But this, of course, breaks older programs that read and edit the password file. It is for this reason (among others) that Reiser4 supports filesystem plugins. One such plugin is a directory aggregator. Whereas before it was mentioned that a file could also be a directory, a directory can appear as a file. In this case, the directory would appear as an aggregation of files separated by a delimiting character (such as '\n'). Taking this example to its logical conclusion, each field could be a separate file."

Or you could write a program that allows a user to modify his line of /etc/passwd...

You could also have /etc/passwd/userid/info in an existing, conventional filesystem. Perhaps it wouldn't perform as well, but you could tweak it so that it would be suitable for most purposes.

Nope (4.85 / 7) (#45)
by ttfkam on Sat Aug 09, 2003 at 10:54:40 PM EST

Or you could write a program that allows a user to modify his line of /etc/passwd...
Except that the program would have to be setuid root in order to write to /etc/passwd. This is precisely what splitting the file up is meant to avoid.

You could also have /etc/passwd/userid/info in an existing, conventional filesystem. Perhaps it wouldn't perform as well, but you could tweak it so that it would be suitable for most purposes.
  1. The point of Reiser4 is that you can do it and it performs well.
  2. Splitting up the file on conventional filesystems would break existing programs.

If I'm made in God's image then God needs to lay off the corn chips and onion dip. Get some exercise, God! - Tatarigami
[ Parent ]
You have good points (none / 0) (#140)
by omghax on Sun Aug 10, 2003 at 05:53:22 PM EST

A setuid program on a properly set up system doesn't have to be a danger.

But yes, you are right about breaking existing programs. However, you break existing filesystem conventions :D

[ Parent ]

Not breaking; Extending (none / 0) (#183)
by ttfkam on Mon Aug 11, 2003 at 01:50:21 AM EST

Programs using existing APIs still work on Reiser4.  Developers simply have more options than before.  It's not even like some Redmond companies that screw the standard and make their own widgets.  It implements the standard in addition to features not currently available in the standard.

If folks really want to keep to the POSIX religion, it's not for me to question.

If I'm made in God's image then God needs to lay off the corn chips and onion dip. Get some exercise, God! - Tatarigami
[ Parent ]

Will filesystems matter in 10 years? (4.09 / 11) (#28)
by i on Sat Aug 09, 2003 at 08:11:50 PM EST

Old Unix Way: everything is a file.
New ???? Way: everything is a memory block.

Filesyatems are ugly. Transparent persistency (TP) is so much cleaner and simpler, and any filesystem functionality (no matter how advanced or weird) is easy to provide on top of TP in user-space. Just you wait. Sooner or later 32-bit systems will become obsolete. On a 64-bit computer, TP of single flat address space is the most natural way to implement persistent storage. Files and filesystems will kick and scream for a while, but eventually they must go the way of the dinosaurs.

For the interested: currently you may replace ???? with EROS, but many other systems will probably come.

and we have a contradicton according to our assumptions and the factor theorem

Transparent persistency has its own problems (3.75 / 4) (#30)
by grout on Sat Aug 09, 2003 at 08:39:07 PM EST

It's been technically feasible since 32-bit VM systems, at least for small purposes, but where is it? Implement /usr/bin/find for a TP system....
--
Chip Salzenberg, Free-Floating Agent of Chaos

[ Parent ]
What's the problem? (4.50 / 2) (#33)
by i on Sat Aug 09, 2003 at 08:59:49 PM EST

As I said, you can layer any filesystem functionality you wish on top of your address space. Naming, directories, permissions, attributes, access/modification times, metadata of any kind, you name it. But I don't think most of this functionality is needed for a TP systems. Hierarchical names, yes. You can happily implement find with them.

I think TP is not here because (a) everybody has legacy apps that depend on the filesystem and (b) in 32-bit space it's not that convenient. Possible, yes, but not great.

and we have a contradicton according to our assumptions and the factor theorem

[ Parent ]

A filesystem, by any other name (5.00 / 7) (#46)
by grout on Sat Aug 09, 2003 at 11:03:02 PM EST

You can layer any filesystem functionality you wish on top of your address space.

Kind of makes "getting rid of the filesystem" an imaginary accomplishment, doesn't it?

PS: Sucks to get a SIGSEGV from following a filename pointer, doesn't it?
--
Chip Salzenberg, Free-Floating Agent of Chaos

[ Parent ]

Not really. (none / 0) (#90)
by i on Sun Aug 10, 2003 at 04:33:56 AM EST

A name service for in-memory objects is not really a filesystem as traditionally understood. For one, there's no distinction between data and metadata. This alone can greatly simplify things. For two, it runs in user space, so you can have exactly the functionality you need, without messing with the kernel.

I don't understand your remark about SIGSEGV.

and we have a contradicton according to our assumptions and the factor theorem

[ Parent ]

Let's see if I understand (4.00 / 1) (#125)
by vadim on Sun Aug 10, 2003 at 03:52:05 PM EST

We have this TP thing that pretends everything is a linear address space where stuff can be stored. Ok. So we've got this giant block of memory and want to find something in it. We now must create an index of the information in it. And here you go, we arrive at the filesystem. An indexed block of data.

Making it all happen in a linear address space doesn't make it any different. There's tmpfs too. A file system, accessed by reading blocks or with mmap is still the same thing, IMO
--
<@chani> I *cannot* remember names. but I did memorize 214 digits of pi once.
[ Parent ]

Not exactly. (none / 0) (#153)
by i on Sun Aug 10, 2003 at 08:26:00 PM EST

Files in a norml filesystem and data in your program exist in different namespaces. You can't just dump a complex data structure with pointers to a file and expect to be able to read it later, much less by a different application. You call mmap and it makes all these pointers invalid. It's much simpler to forget about the whole mapping thing.

and we have a contradicton according to our assumptions and the factor theorem

[ Parent ]
Still didn't get a good idea of it (none / 0) (#155)
by vadim on Sun Aug 10, 2003 at 08:35:13 PM EST

I googled a bit for it, and didn't find anything very concrete. Maybe you could give me a link with some info?

The idea I got it's that it's more or less the equivalence of Data::Dumper, which in Perl can dump a complex structure. While that is a pretty cool thing to have, I wouldn't say it's a perfect solution. It records the internal organization of data, which can change very often and would be very inadequate for exchange of information with other programs.
--
<@chani> I *cannot* remember names. but I did memorize 214 digits of pi once.
[ Parent ]

It is true (none / 0) (#185)
by i on Mon Aug 11, 2003 at 04:13:25 AM EST

that this method of storing things is non-portable in space and time. Its advantages are simplicity and efficiency. In the normal course of operation, the applications do nothing special. The data is not "recorded" or "stored". It just is. Other programs running on the same machine can access it just fine (presumably they all use the same headers/libraries/whatever). When you need to export your document in a portable format, your apps do that, but normally they just leave their data sitting there.

Transparent persistency is implemented in EROS. There are some articles on the l4ka site too.

and we have a contradicton according to our assumptions and the factor theorem

[ Parent ]

Something that always got me about TP... (4.80 / 5) (#41)
by Kyle on Sat Aug 09, 2003 at 10:47:17 PM EST

What about the physical disc(s)?

When a disc fails in my system, I know what was on it. I know what to get from backup. If I pull a drive from my system and put it in another, I know what I've transplated.

Do TP systems give you any control or knowledge about what's on the physical discs? Can I back up parts of it without backing up the whole thing? How do I email some piece of data without it pretending to be a file, one way or another?

Maybe my head is too stuck in filesystems. I think of TP as treating the disc as the same as physical memory. I can't imagine what the OS does when a piece of its memory goes bye-bye, so I don't know what happens when some USB memory stick gets yanked off the system.

Basically, how do you treat all the storage in the system as "one linear address space" when it just isn't?

[ Parent ]

It's an interesting topic. (none / 0) (#93)
by i on Sun Aug 10, 2003 at 05:43:01 AM EST

I don't know how TP treats removable media. I don't have much practical experience. I guess each physical device or volume or whatever lives in its own separate namespace. It shouldn't make much difference to the application.

For backup and other purposes, your data needs to be exported via some sort of user-mode name service.

It is one linear address space in the following sense. Every chunk of storage has its own unique address in RAM. All apps see it at the same address, so any valid pointer is valid throughout the system (subject to access restrictions of course).
When a storage device is yanked from the system, corresponding chunks of memory get unmapped and apps no longer see them. AFAICT here lies the problem: what if some app has a pointer to unmapped piece of storage? Again, I don't know much about treatment of removable media.

and we have a contradicton according to our assumptions and the factor theorem

[ Parent ]

TP versus Evolutionary Approach (none / 0) (#225)
by frankwork on Mon Aug 11, 2003 at 06:24:11 PM EST

"Every chunk of storage has its own unique address in RAM."

In a UNIX-like system (in the sense that you have many small apps interacting to make a useful environment), something much more portable than a 64-bit number is needed. Something much more human-readable is very desireable. IIRC, Hans Reiser expanded Metcalfe's law from networks to systems: the utility of a system goes up with the square of the number of connections between pieces.

So you can use some kind of name service to map path names to a 64-bit number, and you have something very much like a 64-bit filesystem, except that every time you write to it, you can consider your changes permanent. So instead you work on a copy. But this copy (which is most often in an inconsistent state) is backed up on disk. Which leaves you with something practically the same as the current state of the art.

Now it would be super if the underlying OS was transparently persistent (quicker recovery), or if applications could allocate chunks of transparently persistent memory (a memory-mapped file, as it were). But taking the purist approach (as usual) has some practical disadvantages. Taking the non-purist approach leaves you with something very much like the current evolutionary approaches.

(Also if you have per-process namespaces (like Plan9 and unstable Linuxes), you can implement something very much like a capability-based system).

I've been thinking about putting together an article on the current assaults on traditional filesystems: solid state systems like PalmOS, database-like systems like BeOS's BFS, and transparently persistent systems like EROS. But I've decided the title would be "The Filesystem is Dead, Long Live the Filesystem."

[ Parent ]

Filesystems are not ugly (3.50 / 2) (#49)
by vrt3 on Sat Aug 09, 2003 at 11:21:30 PM EST

I don't see what's so ugly about filesystems.

If I understand everything correctly, filesystems and transparent persistence solve different problems.

Transparent persistence is meant to save the state of a program. The central point is the application; applications happen to need data, and transparent persistence is an attempt at an easy solution to deal with that. Traditionally the filesystem is used for that, but that's not the only purpose.

The purpose of a filesystem is to store pieces of information in a way that make it universally accessible. The central point here is the data, which is put in a namespace so we can refer to it from any appllication we wish.

In my opinion, for general use the data-centric model is more useful. It also fits better in the Unix-approach of many small specialized tools.

When a man wants to murder a tiger, it's called sport; when the tiger wants to murder him it's called ferocity. -- George Bernard Shaw
[ Parent ]

You're somewhat right. (none / 0) (#91)
by i on Sun Aug 10, 2003 at 04:45:05 AM EST

But you don't need all data universally accessible, only some of it. For that part of data you can run a name service (or three) that will put the data in a namespace (or three).

Take a word processor. You may or may not need to refer to its data from other applications; the thing is, the word processor itself doesn't care. All it has is chunks of in-memory data. Some of them may be exported to the world via some name service, but it doesn't make any difference as to how they are handled in the word processor. All it has is data in memory.

and we have a contradicton according to our assumptions and the factor theorem

[ Parent ]

worlds largest windows registry? (none / 0) (#109)
by joto on Sun Aug 10, 2003 at 12:05:11 PM EST

But wouldn't this sorta be like the windows registry, except, you could never read it or edit it yourself?

While I somewhat like the idea of not having to expose every file to everybody, there are plenty of advantages for doing it too.

First: resource management. You need to be able to find out what it is that consumes all the space, which disk it lives on, move it to another disk, backup, delete, etc.

Second: stability. I tend to trust file-system code. But applications often crash. Add persistence, no global filesystem to take care of data after an application crashes, and your average buggy user-space applications, and you sure as hell are going to loose data, unless you do something really smart (which I can't think of).

Third: organization. Today we can use the same file-managing software for every kind of document. Should every application need a built-in document manager instead? Do you think it would be better?

Fourth: Interoperability. Say I need to email someone something made in foopaint. I save it to a file, and add it as an attachment in my email-program. What would you use in your persistent world? The clipboard? Say you want to mail them 27 of your latest digital camera pictures instead? Still prefer the clipboard? Oh, by the way, before you send those pictures, you might want to edit a few of them a bit in foopaint.

Fifth: more organization. How do you organize documents that logically belong together, but are created/viewed/organized/managed/edited in totally different applications?

Sixth: user-friendlyness. We have been more and more accustomed to a document-centered computing experience. We no longer start word to open a half-written letter. We click on the letter instead. This is productive, and user-friendly.

As you can tell, I am certainly not all against the idea of replacing the file-system with persistence. But as a general solution for general computers, it really shouldn't happen. However, for appliances, and things with more limited functionality, it certainly makes a lot of sense. But then, if you are not aiming for generality, you could just as well hide the intricacies of the file system in user-space.

[ Parent ]

Some answers. (none / 0) (#201)
by i on Mon Aug 11, 2003 at 12:08:15 PM EST

  1. Buggy apps. Of course any responsible app will work with a copy of your data (just mark the corresponding pages copy-on-write and off we go).
  2. Interoperability. Of course some kind of interchange format is needed.
  3. Organisation. In general, you need some kind of name service for your in-memory objects. This looks like a filesystem, but such service runs in user space, you can have any number of domain-specific services running simultaneously, it is decoupled from the data itself, etc. A lot of advantages over your regular fs.


and we have a contradicton according to our assumptions and the factor theorem

[ Parent ]
EROS (4.00 / 1) (#160)
by baka_boy on Sun Aug 10, 2003 at 09:47:32 PM EST

I remember checking out EROS about a year and a half ago, and being very impressed by the work they had done: not only did the OS provide orthogonal persistence and extremely fine-grained access controls via a capability-based security model, but it supported IPC (including limited continuations!) and actually managed to perform reasonably well. Unfortunately, the whole project seems to have more or less died down -- the last update to their website was last May, and even that was simply announcing a paper one of the designers had written being accepted for USENIX.

Unfortunately, it seems like getting a sufficient number of developers to join up with any revolutionary, rather than evolutionary, improvement in something as esoteric as operating system kernels is just too hard. Linux attracts and keeps developers because it's "just like UNIX," supports the standard POSIX environment via the GNU tools, and has decend compilers, email clients, etc., which allow the developers to spend most of their time acutally using the system.

Hell, look at GNU/Hurd: it's basically a working system, and should have the enthusiastic backing of a good portion of the OSS crowd, but instead, it moves about as quickly as the average SourceForge OS project (read: not at all). Why? Mainly because it's based on a microkernel architecture, which is unfamiliar to most old-school UNIX hackers, and doesn't have the application support that Linux does, which puts it out of reach for the script kiddies and your average programmer/hobbyist.

[ Parent ]

HURD's slow pace... (none / 0) (#207)
by tkatchev on Mon Aug 11, 2003 at 12:47:04 PM EST

...is due solely to Stallman being involved in it.

There are a billion and a half non-innovative OS's being written out there, and 99.9% of them are in an even worse state than GNU/HURD.

   -- Signed, Lev Andropoff, cosmonaut.
[ Parent ]

the Hurd (4.00 / 1) (#222)
by unknownlamer on Mon Aug 11, 2003 at 06:07:13 PM EST

The Hurd isn't dead. Most of the work is happening behind the scenes now with a port to the L4 microkernel. The port is a fairly large endeavor because L4 uses synchronous IPC (whereas Mach uses asychronous IPC) and there is no device driver support in L4 (so a user space device driver framework is being designed). The port to L4 will free the Hurd from the slow and buggy GNUMach (GNUMach is basically the worst part of the GNU system; the Hurd itself is actually working for the most part).

As for software support, the Hurd has had POSIX Threads support for several months now. A lot of software that wouldn't work before now works fine, but stuff like GNOME2 still doesn't work because a few features of pthreads haven't been implemented yet (and may never be for the current Hurd running on top of GNUMach). Driver support right now is also really really poor since only Linux 2.0 drivers can be used in GNUMach 1.3 and GNUMach 2.0 only improves that to Linux 2.2 (because it uses the Flux OSKit DriverKit). I haven't been able to get X running yet except in plain VGA mode because 3.3 doesn't support the Radeon and XFree86 4.x doesn't build on the Hurd because the compile uses more disk space than the ext2fs translator can handle. That's being fixed too...



--
<vladl> I am reading the making of the atomic bong - modern science
[ Parent ]
Well, it's gotta be said, (1.50 / 2) (#167)
by buck on Sun Aug 10, 2003 at 10:16:29 PM EST

I need TP for me bunghole!

-----
“You, on the other hand, just spew forth your mental phlegmwads all over the place and don't have the goddamned courtesy to throw us a tissue afterwards.” -- kitten
[ Parent ]
Performance, transactions, distribution (4.66 / 3) (#169)
by greenrd on Sun Aug 10, 2003 at 10:50:16 PM EST

Persistence shouldn't be entirely transparent, for the following reasons:

1. Performance. There's a reason why Mozilla and IE both have a seperate disc cache and memory cache. If they just used a very large memory cache for everything, the swap overhead would be terrible. This is because, under transparent persistence, the application has no idea what the kernel is doing, and the kernel has no idea what the appplication is doing, so neither can apply any intelligent optimisations which take into account the core fact that transparent persistence advocates tend to shove under the carpet: disc is several orders of magnitude slower than memory, and even more so when lots of seeking is involved.

2. Correctness. Transactions involve what are externally supposed to be atomic operations, but internally are a series of steps. Committing a database transaction involves turning a transient state into a persistent state, logically all at once (or a close approximation). This sounds to be like persistence-awareness.

3. Distributed systems. I need to save my data to the network. What happens if there is an hour-long network outage right in the middle of transferring data to the network server? If an application is fs-aware it can popup an error dialog and continue working. If it treats the entire network as a giant address space, this is utter madness because you will have severe stability problems.


"Capitalism is the absurd belief that the worst of men, for the worst of reasons, will somehow work for the benefit of us all." -- John Maynard Keynes
[ Parent ]

Please enter a subject for your comment. (none / 0) (#200)
by i on Mon Aug 11, 2003 at 11:51:40 AM EST

1. No, Mozilla and IE have disk cache because cache must persust between sessions (doh!) and also because browser's cache could become quite large and clog the swap space.

under transparent persistence, the application has no idea what the kernel is doing, and the kernel has no idea what the appplication is doing

This is not entirely true. The application still needs to explicitly request objects to be mapped to its address space, and can unmap them when they are not needed. This is not very different from today's mmap() from kernel's perspective.

  1. I don't entirely understand this. If you need transactions, you have to program them in, no matter whether in memory or on disc.
  2. Of course you don't treat the network as one giant address space. Not unless you are prepared to have faulty memory. No question about it.


and we have a contradicton according to our assumptions and the factor theorem

[ Parent ]
Oops (5.00 / 1) (#226)
by greenrd on Mon Aug 11, 2003 at 06:26:36 PM EST

The application still needs to explicitly request objects to be mapped to its address space,

Sounds like we're talking at cross-purposes. Googling, it seems your definition is more common. I can't locate the project right now which led me to these conclusions, but I was thinking of a system more like Grasshopper or this one.


"Capitalism is the absurd belief that the worst of men, for the worst of reasons, will somehow work for the benefit of us all." -- John Maynard Keynes
[ Parent ]

You mean... (none / 0) (#206)
by tkatchev on Mon Aug 11, 2003 at 12:40:11 PM EST

...Palm OS?

   -- Signed, Lev Andropoff, cosmonaut.
[ Parent ]

Kind of. (none / 0) (#216)
by i on Mon Aug 11, 2003 at 05:02:36 PM EST

But not entirely. Something with a real hardware MMU and secure access to individual objects and a disk that backs main memory. Hm, not very similar to PalmOS after all.

and we have a contradicton according to our assumptions and the factor theorem

[ Parent ]
+1, excellent troll. (2.04 / 21) (#29)
by RobotSlave on Sat Aug 09, 2003 at 08:38:45 PM EST

I had no idea there might still be some life left in that hoary ReiserFS bitchfest.

Hats off to you, sir, and let me be the first to say:

"BUY AN AD!!1!"

Reiser's pissed that Linus passed on Reiser4 (2.00 / 6) (#48)
by grout on Sat Aug 09, 2003 at 11:06:31 PM EST

Linus didn't accept Reiser4 for the 2.6 kernel. Reiser's pissed about that. He makes his money from conslutting, and you get lots more visibility and lots more work when your code is in millions of kernels, rather than just thousands.

Indeed: "Buy an ad."
--
Chip Salzenberg, Free-Floating Agent of Chaos

[ Parent ]

Yah. (2.33 / 3) (#86)
by valar on Sun Aug 10, 2003 at 03:54:11 AM EST

makes his money from conslutting

Maybe that was intentional, maybe not. Either way, it's funny and somehow appropriate. Either way, I think Linus had good reason for passing on Reiser4. It has serious performance issues when tails are on, and it has serious stability/data loss issues when tails are off. If you want a journaling fs right now, there are lots of good, mature options. That said, Reiser is getting more stability and more performance rapidly.

[ Parent ]
doesn't appear so... (2.00 / 1) (#110)
by bobzibub on Sun Aug 10, 2003 at 12:06:31 PM EST

If I recall, Reiser 4 is compiled as default in 2.6.0 test 3, which I'm about to boot.

Cheers,
-b


[ Parent ]

Well, it's not in the test3 I downloaded... (none / 0) (#172)
by grout on Sun Aug 10, 2003 at 10:57:49 PM EST

... so I can't figure where you got it.
--
Chip Salzenberg, Free-Floating Agent of Chaos

[ Parent ]
Wrong; reiser 4 will be in 2.6.0 (none / 0) (#176)
by lambda on Mon Aug 11, 2003 at 12:25:32 AM EST

Read this. Scroll down to the "before 2.6.0" section. Notice Reiserfs v4.

[ Parent ]
That's not Linus's list; don't trust it. (none / 0) (#196)
by grout on Mon Aug 11, 2003 at 09:59:21 AM EST

You're far too trusting in random lists. Linus keeps his own counsel. He may have suggested at one point that 2.6 would be a good time for introducing reiser4 if it works. Or, more likely, the reiser4 team set inclusion in 2.6 as a goal of their own. But no matter who says what, it's not in 2.6 until it's in 2.6. And you know what? It's not in 2.6.

Of course, they could get a last-second reprieve from the governor. But I'd bet money against it. Reiser's personality and the quality of his team's work have both made Linus skeptical, and rightly so.
--
Chip Salzenberg, Free-Floating Agent of Chaos

[ Parent ]

Why a New OS Matters (1.29 / 24) (#31)
by tofubar on Sat Aug 09, 2003 at 08:42:22 PM EST

Because Linux is an archaic piece of shit.

Not quite (3.44 / 9) (#55)
by synik on Sun Aug 10, 2003 at 12:11:09 AM EST

Unix is an archaic piece of shit.

Linux imitates an archaic piece of shit.

[ Parent ]

Yea? (1.50 / 2) (#121)
by eleusis on Sun Aug 10, 2003 at 02:36:50 PM EST

Of course it doesn't have to be that way... There's a massive amount of code that can be found in most Linux systems that is freely modifiable. Why not modify all of that and make your own non-archaic-piece-of-shit?

Or better still, write your own thing from scratch?

:)
[ Parent ]
Verily, indeed. (5.00 / 1) (#127)
by tkatchev on Sun Aug 10, 2003 at 04:03:12 PM EST

Do you have any more bycicles you want me to re-invent?

   -- Signed, Lev Andropoff, cosmonaut.
[ Parent ]

The Hurd (5.00 / 2) (#168)
by unknownlamer on Sun Aug 10, 2003 at 10:18:59 PM EST

The Hurd is doing just that, and has been for a very long time. The reason that Linux was able to get to where it is so quickly while the Hurd has lagged behind in features is a combination of two factors: The Hurd lacks resources, and the Hurd is doing something new.

The Hurd runs on Mach. GNUMach in particular. GNUMach sucks. GNUMach 2.0 uses the OSKit for drivers; did I mention that the OSKit is now abandoned software? GNUMach is basically the big weak spot of the Hurd. All hope is not lost because the Hurd is being ported to L4. L4 is a much nicer microkernel than Mach. This still leaves a problem with the number of developers; there are far too few. Most of the higher profile developers are working on getting the Hurd run on L4 right now (which is mainly focusing on writing a bridge between GRUB and L4 named laden and getting a Device Driver Environment designed and written), which is a good thing.

And yes, the Hurd is doing something original. I looked at some stuff on plan9 and things like translators are very similar to what plan9 has, but the Hurd is still doing a lot more innovation than Linux is. Linux is a very nice traditional monolithic kernel which really isn't such a bad thing for now. Monolithic kernels are arguably faster than microkernels can be on current hardware and the problem of maintaining one huge kernel with many modules all running in the same address space isn't so bad. But, in the long run, having everything running in its own address space is a better idea. If the network stack has a bug and dies, the system shouldn't go down. If the file system driver has a bug and it dies the system should make some reasonable attempt at not dying (e.g. if the iso9660 driver on a cdrom dies, whatever; but if the ext2 driver on / dies then it is entirely reasonable for the system to stop functioning).

When you think about it, file system plugins in ReiserFS 4 do basically the job of translators. E.g. the aggregate plugin could be implemented as a simple translator that forwarded all requests for the directory it was installed on to the file system translator if the file was opened as a directory and service the requests itself if the file was opened as a file. Otherwise, it appears that plugins merely make it easy to store metadata that may as well just be made available using a generic interface. The Hurd has limited support in the ext2 translator for extended attribute bits (which are used for permissions now IIRC), but it really doesn't seem like it would be terribly difficult to add generic meta data support to many file systems.

This brings up the issue of using something like SQL on a file system; one could write an sqlfs translator for the Hurd that was backed by a real filesystem; the sqlfs translator would get all calls to the real file system and update them in its database and then forward the message onto the real filesystem. An sqlfs_initializedb program could be provided to generate a file system database on the underlying file system before it was used with sqlfs. One could access a file the "normal" way with a path or by doing something like "SELECT filename FROM files WHERE (owner == 'clinton' AND mime_type == 'application/x-ogg' AND contains (filename, "foo"));" (where "contains" is a mythical function that finds a string inside of a table field). Or one could simply use the file system as an easy way to provide a backingstore for the database and not allow any access to it directly (so that the author of sqlfs wouldn't have to worry about designing his own low level on-disk layout for stuff).



--
<vladl> I am reading the making of the atomic bong - modern science
[ Parent ]
Reply within. (none / 0) (#205)
by tkatchev on Mon Aug 11, 2003 at 12:37:10 PM EST

...the problem of maintaining one huge kernel with many modules all running in the same address space isn't so bad.

Actually, yes it is. It is a gigantic problem. So gigantic, in fact, that Linux (after 12 years) still doesn't have a functional graphics subsystem.

Look, writing an OS is a relatively simple task. All you need is are decent interrupt, memory and process handling subsystems, and then a working and safe way of loading kernel modules. A very good and modern OS could be written in about 50k lines of code; all the remaining kernel modules will be adapted quite quickly, provided the base micro-kernel's design isn't braindead.

Sigh.

Perhaps, when I retire, I could make my contribution to the world and finally write an OS that works. Something tells me we still won't have one in 40 years.

   -- Signed, Lev Andropoff, cosmonaut.
[ Parent ]

A problem, yes (none / 0) (#220)
by unknownlamer on Mon Aug 11, 2003 at 05:49:00 PM EST

I know that it is a large problem. But I didn't want to get a bunch of "nuh uh! Everything running in one address space isn't a problem!" replies. However, although the problem is a huge one, it is (at the moment) a manageable problem. Now, in five years, it probably won't be.

I like the idea of micro kernels myself. Which is why I have a copy of the Hurd sitting on my second hard drive. I am really excited by the port to L4 which promises to make the Hurd usable for daily work. The cool stuff is the new Device Driver Environment (DDE) that should allow for the usage of unmodified Linux drivers, each running inside of a little encapsulating server and completely separate from each other. Transporting drivers meant to run in kernel mode to user mode will be interesting.

The nicest part of l4hurd is that the core does exactly what your design would; there is a task server, a memory server, an interrupts server, and then the kernel which handles IPC and address space allocation/deallocation/transfer. I can't wait to start playing with it.



--
<vladl> I am reading the making of the atomic bong - modern science
[ Parent ]
Is it that easy? (none / 0) (#239)
by squigly on Tue Aug 12, 2003 at 06:38:54 AM EST

Look, writing an OS is a relatively simple task. All you need is are decent interrupt, memory and process handling subsystems, and then a working and safe way of loading kernel modules. A very good and modern OS could be written in about 50k lines of code;

That's what I've always assumed.  But if it is that easy, why don't we have several dozen competing free microkernels, and why does using L4 rather than Mach for Hurd make such a difference?

Okay, I I've probably proved I know next to nothing about kernel design.  Where is the real complexity?  Surely there has been enough research in the matter to make sure that anyone who does a little research isn't going to remake any totally braindead mistakes.

[ Parent ]

There isn't any complexity. (none / 0) (#240)
by tkatchev on Tue Aug 12, 2003 at 07:23:56 AM EST

The problem is that an OS, by itself, is totally useless.

A sucessful OS is made by the applications and popular support for it, not by good software design.

In short, making OS's is mostly about proper marketing. Look at Linux, for example; an inferior OS, (compared, to, say, FreeBSD) yet a good advertising campaign has made Linux the second-best known OS in the world.

   -- Signed, Lev Andropoff, cosmonaut.
[ Parent ]

And the best known OS (none / 0) (#263)
by Cro Magnon on Wed Aug 13, 2003 at 10:28:03 AM EST

is far inferior to Linux.
Information wants to be beer.
[ Parent ]
The second-best known? (none / 0) (#271)
by Zerotime on Thu Aug 14, 2003 at 12:42:49 AM EST

What about the Mac and its very own proprietry OS(s)?

[ Parent ]
Simplicity (4.00 / 1) (#270)
by unknownlamer on Thu Aug 14, 2003 at 12:12:06 AM EST

That's what I've always assumed. But if it is that easy, why don't we have several dozen competing free microkernels, and why does using L4 rather than Mach for Hurd make such a difference?

Mach is a first generation microkernel and is very slow and bloated. L4, however, is a much more simple microkernel that implements a gigantic seven syscalls. Mach has hardware support in the kernel (like Linux does), whereas L4 doesn't deal with hardware at all. Mach has a kernel level memory pager (well, in some implementations you can push that out to user space) whereas L4 doesn't deal with paging at all (it allows privileged servers to take ownership of physical blocks of memory and nothing more). Basically, Mach is sort of a halfway between a monolithic and microkernel (I know someone is going to have a problem with my saying that; I do know that Mach is a microkernel, but what I'm trying to say is that Mach isn't exactly the best micro kernel from the make-the-os-as-small-as-possible perspective) and L4 learned from the mistakes that Mach made and is a "true" micro kernel (although some people think that micro kernels can be made to do even less).

So L4 provides a much better environment for writing OS servers than Mach does, thus allowing for an OS to be implemented in a much more simple way. It also has the benefit of being potentially much faster than Mach could ever hope to be.



--
<vladl> I am reading the making of the atomic bong - modern science
[ Parent ]
What's your fucking problem, (2.25 / 8) (#272)
by butfuk on Thu Aug 14, 2003 at 01:44:26 AM EST

dickhead?



Unix and C are the ultimate computer viruses.
[ Parent ]
Why write my own? (none / 0) (#195)
by synik on Mon Aug 11, 2003 at 08:59:32 AM EST

Windows XP and OSX are both excellent desktop operating systems.

I also have a linux gateway. Just because it's copying an archaic piece of shit doesn't mean it doesn't have it's uses (while nice, Windows 2003 server would be massive overkill).

[ Parent ]

not for me, thanks (4.36 / 11) (#32)
by martingale on Sat Aug 09, 2003 at 08:44:01 PM EST

The idea of arbitrary attributes is way overrated. You're missing the point that this is a question of data design.

Say, you want to add ID tag support in the file system. Now why would you want to do that? Is it really logical to have the author and title as part of the file? I don't think so.

To me, author and title information belongs in a genre database. Suppose you download a collection of mp3s. Each filename is just a standardized hash, kind of like the Dewey Decimal System. By glancing at the hash, you know that the author and title is kept in a corresponding genre database, and by using a modified "ls" program, you can display the author and title automatically.

Now because you've got a genre database instead of the info directly in the file, you can do so much more. Like get automatic suggestions for similar genre music files, for example. You can look up the alphabetically immediately preceding author and see if you've got any mp3 files from him/her. You can "surf" your music like you surf the 'net.

All you get from file attributes is lots of small bits of information strewn all over the place. It's useless, because it's *raw* information. Information is much more useful once it's processed, and that's what you gain from real compiled databases. Moreover, you get completion, because a real database will point to things that don't exist in your filesystem. Like that mp3 file from that author you saw in your author/title database and which you haven't yet checked out but will download the next time you're on the net.

The kind of attributes that belong with files are those the filesystem needs to function. Everything else is unnecessary, and probably better kept somewhere else. Depends on the application, and the design skills of the application programmers. You'll notice also that by keeping things elsewhere, you don't have the red herring issue of lots of microfiles all over the place. Content aggregation is not just for websites.

Well... (4.75 / 8) (#47)
by ttfkam on Sat Aug 09, 2003 at 11:03:18 PM EST

A lot of people seem to think associating the author and title information belongs with the file is important. If not, why does ID3 exist?

Leaving aside the point that a filesystem is a database, a genre (I'm assuming you mean relational) database suffers from the same problems that metainfo files suffer from: synchronization with the filesystem. How do you solve this with a genre database? Be careful. If you suggest that the file be kept in the database as well, you've simply moved the filesystem up a layer rather than replace the filesystem.

There is nothing stopping anyone from running an indexing algorithm on the filesystem for quick and easy cross-file lookups and associations.

And technically, the little bits of information are not strewn all over the place; The bits are strongly associated with the files they describe.


If I'm made in God's image then God needs to lay off the corn chips and onion dip. Get some exercise, God! - Tatarigami
[ Parent ]

let's see (4.75 / 4) (#65)
by martingale on Sun Aug 10, 2003 at 01:24:46 AM EST

A lot of people seem to think associating the author and title information belongs with the file is important. If not, why does ID3 exist?
Just because it exists doesn't mean it's a good idea. I tried to suggest some advantages of doing it differently (with an external index). In the end, it's up to the programmers implementing their systems.

Leaving aside the point that a filesystem is a database, a genre (I'm assuming you mean relational) database suffers from the same problems that metainfo files suffer from: synchronization with the filesystem. How do you solve this with a genre database? Be careful. If you suggest that the file be kept in the database as well, you've simply moved the filesystem up a layer rather than replace the filesystem.
I used genre to mean music genres here, but you're right that all this is properly a discussion of relational database functionality. The ID3 example was discussed briefly in the story, so I thought it would make sense to use it to illustrate my comment.

You'll note that in that illustration, desynchronization is not in fact a problem, but a strength. The author/title index refers to mp3 files which do not exist on the user's filesystem. If the index was perfectly synchronized with the user's data, then it wouldn't be possible to discover new authors/titles easily (ie we would need an external mechanism, such as word of mouth, advertizing, etc.). This way however, the user can discover mp3 files which do not reside on his/her disk, and he or an application can take the appropriate steps to synchronize (ie download in this case) the index entry if he so chooses.

You note that implementing an arbitrary attribute system essentially recreates a database layer. I agree and ask why? Why recreate an incomplete database system in the filesystem, when advanced database engines exist, which already have far more functionality and performance advantages?

A database system like Oracle already reads and writes the physical disk directly to gain the last bit of performance, bypassing the suboptimal (for its purposes) filesystem. In fact, this is what many performance critical applications do, and for good reason. A generic filesystem can never optimize its services as well as the application can, since only the application knows what it really wants to do and how to cleverly optimize its requirements. Do you expect Google to use an off-the-shelf database for storing its web pages and indexes? Laughable.

There is nothing stopping anyone from running an indexing algorithm on the filesystem for quick and easy cross-file lookups and associations.
Exactly. Moreover, it they do so, the programmers will be forced to think through why they are doing it, and maybe design their data tables instead of appending attributes everywhere.

And technically, the little bits of information are not strewn all over the place; The bits are strongly associated with the files they describe.
That's exactly what can be bad. You mentioned synchronization problems. What if last year, you associated all your mp3 files with "owned by Smit", but this year you use "owned by Smith"? Is ReiserFS going to recognize that you want "owned by Smit" to be changed on all those old files? Where's the logic that will do this?

If you have an external index, at least you'll have an external application which created it in the first place, and that's the natural place where you can expect the logic to be (if the program author implemented it).

For the filesystem-as-database, the attributes have no special meaning, so there is no support for special logic to help manage the information *intelligently*.

The attributes are simply an alphabet soup. They are strongly associated with the file, but it's a stupid way to design an information retrieval system. It's like suddenly, every house owner gets an attic to put their old stuff in any way they want, and you get a master key to open them all. Now go find a copy of Spiderman #34, with a coffee stain on p.5 :-P.

[ Parent ]

well reiserfs4, (4.40 / 5) (#56)
by werner on Sun Aug 10, 2003 at 12:16:31 AM EST

as far as i can tell, just draws these 2 things together - the meta-database and the fs. for anyone who has written a program which stores data, this can only be a huge boon. the possibility of being able to store arbitrary attributes with your files eliminates any amount of work. instead of 2 interfaces - one to the fs, one to the db - as you seem to prefer, a programmer can deal with this all at once. any number of data-consistency and security problems are solved in one fell swoop here, and with them a huge amount of code. i think you misunderstand reiserfs4. you remain free to build databases of your files elsewhere, you don't have to use the extra functionality if you don't want it. but the underlying fs just made building that database server 300% easier.

[ Parent ]
different interfaces (4.75 / 4) (#72)
by martingale on Sun Aug 10, 2003 at 02:00:27 AM EST

Why throw away existing databases and start from scratch with a database/filesystem hybrid? Too much functionality?

What I like about the separation between the filesystem and a database is that the latter forces the programmer to think just a little about data structures and how best to use them.

The arbitrary file attributes are a disaster waiting to happen. Perhaps ReiserFSx can manage all this data efficiently, but people will be drowning in a sea of unstructured information. We'll always need dedicated, domain specific application programs to manage the information *intelligently*, but we don't need every application and its brother spamming their custom attributes all over everyone's files. This is simply going to increase the difficulty in managing your files in the end. Imagine the web on your hard drive. Without Google.

[ Parent ]

actually (3.00 / 1) (#78)
by QuantumG on Sun Aug 10, 2003 at 02:52:07 AM EST

Maybe that's the master plan of Mr Reiser. He has stated openly that he is working towards putting full relational semantics into the file system. Namesys only implements new features when someone is paying for them. So maybe that idea is to drown you in meta data until you demand some relational way to deal with it.

Gun fire is the sound of freedom.
[ Parent ]
Oh, no! (none / 0) (#282)
by Omnifarious on Fri Aug 22, 2003 at 06:07:48 AM EST

Databases might become so easy to use that any clod will be able to make a badly designed database. We can't let this happen!


--
Copyright is dead, so copy this post at will since you would anyway.
[ Parent ]
I have a much better idea. (1.50 / 2) (#128)
by tkatchev on Sun Aug 10, 2003 at 04:05:35 PM EST

For ghod's sake, somebody please invent a music format that doesn't suck!!1

   -- Signed, Lev Andropoff, cosmonaut.
[ Parent ]

Reiser6 (5.00 / 1) (#180)
by uhoreg on Mon Aug 11, 2003 at 01:11:33 AM EST

Reiser6 (I believe) will allow semi-structured data queries. So you would be able to do something like "ls music/[style=Jazz and date<1980]" (assuming you had properly tagged files). See Hans' "Future Vision" paper. Yeah, it's still quite a ways off, but it will be very cool when it arrives. Hans, and the rest of Namesys, are just taking things one step at a time. You can't really do semi-structured queries until the filesystem understands metadata.

[ Parent ]
So, since you want a feature reiser4 doesn't have (none / 0) (#281)
by Omnifarious on Fri Aug 22, 2003 at 06:05:02 AM EST

That makes all the features it does have not worth trying it? Seems like flawed logic to me.

I prefer using this cracked unglazed earthenware cup without handles because your new glass cup doesn't have handles.


--
Copyright is dead, so copy this post at will since you would anyway.
[ Parent ]
The HFS experience, and lessons it could teach... (4.60 / 15) (#35)
by DLWormwood on Sat Aug 09, 2003 at 09:47:14 PM EST

This article briefly mentions the Mac OS's support for enhanced metadata, but dismisses it as "not part of the file system."

Actually, much of it is. Specifically, file type, creating application, additional dates, and a few status bits like file visability are stored at the disk catalog level. The writer's confusion is understandable, however, since additonal stuff (like icons and file type names) are stored elsewhere in a "Desktop Database."

The Mac has always had this philosophy of using the file system to store exchangable data, eventually leading to the use of "resource forks" to store metadata like IPTC tags and other strings used by archiving and imaging apps.

However, looking over the history of Mac usage, it's clear than in dealing with file exchange between PC users and the Internet, most developers expect file systems to be a "lowest common denominator" and refuse to write applications that try to cope with added value at this level. I personally have difficulty explaining to PC/Unix developers how HFS knows what a file's type is without resorting to file extensions or how to make a file invisible without putting a period in front of the name. The file system on Macs was designed to be a user's organizational tool, while most programmers treat it as an architechual structure or foundation that users aren't supposed to get too involved with.

It's like there's some sort of mental block that prevents most developers from seeing file systems as being more than they traditionally have been. "Be" tried to go so far as implement the file system of the BeOS on a database model, only to get a cold shoulder from most of the developer community when they tried to get support for it. Apple, throwing in the towel, has informally adopted a policy of dumbing down file system access under OS X, and discouraging the use file metadata and resource forks. (Wonder why all the iApps now provide a front end to file management? They are handing the file system over to developers, and pigeonholing users into a different organizing principle.)

It seems the computer industry is settling into a pattern much like that which has crippled the auto industry. Pandemic inertia is starting to curtail new developments in technical methodology. Even the Linux movement is more about politics and economic issues than technical progress; Unix variants have been around for decades. Just as the auto industry is having difficulty getting over a dependance over the internal combustion engine, we are having issues getting beyond current file systems and common platforms like x86.
--
Those who complain about affect & effect on k5 should be disemvoweled

'having difficulty' more like 'making money' (2.25 / 4) (#40)
by turmeric on Sat Aug 09, 2003 at 10:46:22 PM EST

the oil/auto/rubber people are all in league with each other. look at their interlocking boards of directorates, investors, incestuous relationships all over the place. they work together. they replace democracy with their own profits. 'whats good for general motors is good for america'.

the only salvation we had was capitalism, japanese competition. but even that is just making different cars to fit on the road structure we have. you want to revolutionize transport? yet you limit your conversation to autos. when transport is a holistic thing encompassing city planning politics environmentalism pedestrian studies sociology and business and economics.

yes and what are our computers? are they tools used by the elitse to control the people? or are they tools used by the masses to control their own lives?

only if OSS comes down on the second side will it produce anything new. because everything else, is old.

[ Parent ]

Reiser Extended Attributes (3.87 / 8) (#36)
by Bad Harmony on Sat Aug 09, 2003 at 09:53:46 PM EST

How do they differ from the extended attributes supported by HPFS (OS/2) and the streams in NTFS?

5440' or Fight!

streams (3.50 / 2) (#79)
by lambda on Sun Aug 10, 2003 at 02:57:28 AM EST

NTFS has a limit of 255 streams per file.

[ Parent ]
Streams (4.00 / 2) (#81)
by matthead on Sun Aug 10, 2003 at 03:07:06 AM EST

Streams are different 'cause they work just like files. You can open, seek, read, write, and close streams. With EAs, you can only read and write them - using a single API call, with a really restricted length (like 255 bytes or something). On NTFS, your normal file data is just the default data stream.

I'll also link to my filesystem terminology page. It's incomplete, but I try to describe these things.


--
- Matt
I'm at (-3.1, -5.0). Where are you?
[ Parent ]
EAs (none / 0) (#182)
by uhoreg on Mon Aug 11, 2003 at 01:25:48 AM EST

I don't know about streams, since I don't use NTFS. Reiser's EAs are accessed just like normal files, so you can do stuff like "cat <file>/<attribute>", whereas in OS/2, you need to use a special system call. Under Reiser4, you should also be able to do other normal filesystem stuff with EAs, like using symlinks or hard links, if you ever feel the need to do that. You could also set different permissions on attributes, or create a hierarchy of attributes (say, <image>/thumbnail/jpeg, <image>/thumbnail/png).

[ Parent ]
Completely irrelevant (1.55 / 27) (#44)
by STFUYHBT on Sat Aug 09, 2003 at 10:53:33 PM EST

This will of course be made completely irrelevant when Microsoft releases their upcoming WFS. You should already know the reason so I will not explain this point - do your own homework.

-
"Of all the myriad forms of life here, the 'troll-diagnostic' is surely the lowest, yes?" -medham
What the fuck are you talking about? [nt] (3.00 / 1) (#53)
by Edward Carter on Sun Aug 10, 2003 at 12:06:29 AM EST



[ Parent ]
WFS, aka Windows Future Storage [NT] (3.00 / 1) (#60)
by STFUYHBT on Sun Aug 10, 2003 at 12:47:51 AM EST



-
"Of all the myriad forms of life here, the 'troll-diagnostic' is surely the lowest, yes?" -medham
[ Parent ]
what I meant was... (none / 0) (#123)
by Edward Carter on Sun Aug 10, 2003 at 03:27:18 PM EST

How will some new filesystem from Microsoft make this irrelevant?

[ Parent ]
As I said, (none / 0) (#138)
by STFUYHBT on Sun Aug 10, 2003 at 05:23:46 PM EST

If you consider yourself the least bit involved in the computer industry you will already know the reason; I'm a busy girl and I can't be bothered to do your homework for you. Ta ta for now!

-
"Of all the myriad forms of life here, the 'troll-diagnostic' is surely the lowest, yes?" -medham
[ Parent ]
Heh, MS is not THAT powerful. (none / 0) (#139)
by vadim on Sun Aug 10, 2003 at 05:44:43 PM EST

MS will of course hype the new FS and all that. But MS already has NTFS and I don't see people stopping to use Linux just because of that.

I even think that it will have the reverse effect. Those people who'd find those features useful can already try them on Linux.
--
<@chani> I *cannot* remember names. but I did memorize 214 digits of pi once.
[ Parent ]

Erm... (none / 0) (#192)
by synaesthesia on Mon Aug 11, 2003 at 08:28:40 AM EST

...it appears you are mistakenly interpretting the username "STFUYHBT" as something other than "Shut The Fuck Up You Have Been Trolled".


Sausages or cheese?
[ Parent ]
WFS (4.00 / 2) (#82)
by lambda on Sun Aug 10, 2003 at 03:08:32 AM EST

As far as I know, Microsoft is not releasing a Linux version. So this will not "of course be made completely irrelevant."

[ Parent ]
What he means. (none / 0) (#94)
by i on Sun Aug 10, 2003 at 05:56:46 AM EST

Anything incompatible with WFS will be made completely irrelevant. You may or may not agree.

and we have a contradicton according to our assumptions and the factor theorem

[ Parent ]
After BFS; I just didnt get off anymore [n/t] (1.11 / 9) (#50)
by Gornauth on Sat Aug 09, 2003 at 11:24:07 PM EST



*yawn* (1.30 / 23) (#51)
by Nigga on Sun Aug 10, 2003 at 12:05:11 AM EST

Like i give a shit if small files take up more than their share.... I find it hard enough as it is to fucking fill up my gig after gig after gig of cheap storage. I mean running out of space isn't an issue.. fucking managing, organizing, finding my thousands if not millions of files IS... fuckin that's why this Reiser 4 bullshit is being designed by linux nerds cuz it's impractical and doesn't solve any normal user's problems... WinFS on the other hand will fucking solve my organization problems... FUCK directories man... long live the relational database file systems... Fuckin if linux dweebs knew what the fuck was best for their users they'd be fuckin striving for WinFS compatability..

--------
The fuck happened to Nigga?

That's not likely (3.00 / 1) (#54)
by synik on Sun Aug 10, 2003 at 12:09:06 AM EST

NTFS support doesn't even work properly yet....

[ Parent ]
actually (4.33 / 3) (#59)
by QuantumG on Sun Aug 10, 2003 at 12:37:22 AM EST

Reiser is a big relational dude too, and you can bet that WFS is gunna suck.

Gun fire is the sound of freedom.
[ Parent ]
heh (3.00 / 3) (#116)
by eleusis on Sun Aug 10, 2003 at 02:17:23 PM EST

*ahem* Linux nerds? normal user's problems? Linux dweebs' users?!

lmao

Dude, feel free to look the other way if you don't like what you see here. :) Linux nerds/dweebs don't have any 'users'. They do not write stuff for 'users' - that is the whole point, they write the stuff for themselves. (lamers sometimes do try and use the stuff, and sometimes do complain loudly afterwards...) Fair enough if you wanna use the stuff, but the license usually does say that there is no guarantee that the 'user' will be able to use the 'stuff' to fit a square peg in a round hole...

Suggestion: Read up on what the heck 'metadata' actually is, and what it is used for. That is, if you can be bothered...

:)
[ Parent ]
tell this shit to lindows ceo nt (1.50 / 2) (#118)
by Nigga on Sun Aug 10, 2003 at 02:20:38 PM EST


--------
The fuck happened to Nigga?
[ Parent ]

umm yea... (4.00 / 1) (#120)
by eleusis on Sun Aug 10, 2003 at 02:27:10 PM EST

Dude, I don't buy Lindows stuff. I don't make stuff for Lindows. And I don't sell Lindows stuff.

In short: I got nothing to do with Lindows. :)
You wanna use Lindows? Fine, go ahead. :P

:)
[ Parent ]
Oh ok so at least now i hear you (1.50 / 2) (#135)
by Nigga on Sun Aug 10, 2003 at 04:53:15 PM EST

loud and clear... You're saying that what you get out of linux is all that matters? And that peops trying to make it accessible to non-nerds like lindows and ximian is not really linux? Ok... So at least now we know your words come out of arrogance rather than ignorance :) :P

--------
The fuck happened to Nigga?
[ Parent ]

Linux is GPLed (2.00 / 1) (#164)
by /dev/trash on Sun Aug 10, 2003 at 09:58:25 PM EST

You have done things for Lindows.

---
Updated 07/20/2003!!
Summer Tour!
[ Parent ]
Great (1.57 / 14) (#52)
by Talez on Sun Aug 10, 2003 at 12:05:32 AM EST

We can implement metadata in an utterly stupid fashion that only a l00nix nut could appreciate.

I got news for you. Someone already went and implemented metadata properly and without Paul Reiser's help. They called it BeFS and all was right in the world once again.

Si in Googlis non est, ergo non est

BeOS... (2.50 / 2) (#88)
by darqchild on Sun Aug 10, 2003 at 04:23:03 AM EST

it was a wonderful OS..
it was amazingly well designed.
it had all the features that anybody could ever want.
the look and feel was natural,
the file system was very efficient, and useful
it was efficient, stable, and everything else that you would drool for in an operating system.

it was the batmobile of operating systems!
and that is why it failed miserably :o\

~~~
Death is God's way of telling you not to be such a smartass.
[ Parent ]

Reiser's talk about /etc/passwd is a good example (4.00 / 8) (#57)
by QuantumG on Sun Aug 10, 2003 at 12:25:14 AM EST

For those who don't know, on a unix machine your /etc/password looks something like this:
root:x:0:0:Super-User:/root:/sbin/sh

xpeters:x:60:666:Peter Sutton:/server/home/xpeters:/bin/bash

xcadman:x:61:666:Jamie Cadman:/server/home/xcadman:/bin/bash

xsimonc:x:62:666:Simon Connelly:/server/home/xsimonc:/bin/bash

xdoug:x:63:666:Douglas Kosovic:/server/home/xdoug:/bin/bash

Of course, that's a "shadowed" password file (notice the x's.. that's where a hash of your password would be in a non-shadowed password file). The part where people's names are, that's called the GECOS or "real name" field. Once upon a time you could change this field to whatever you wanted, but that capability has disappeared in recent years (for no reason other than lack of use). On the other hand, the final field is the "shell" program that is executed when you login. You can change this with the 'chsh' command and you can make it anything you want. 'chsh' is a suid-root command. This means that when you run it, you temporarily become root, the only user on the system that has rights to modify the /etc/passwd file. These suid-root commands are a security risk and have to be written very carefully. Seems kind of silly doesn't it? Why is /etc/passwd a file, shouldn't it be a directory?

/etc/passwd/root/hash

/etc/passwd/root/uid

/etc/passwd/root/gid

/etc/passwd/root/gecos

/etc/passwd/root/homedir -> /root

/etc/passwd/root/shell -> /sbin/sh

/etc/passwd/xpeters/hash

/etc/passwd/xpeters/uid

/etc/passwd/xpeters/gid

/etc/passwd/xpeters/gecos

/etc/passwd/xpeters/homedir -> /root

/etc/passwd/xpeters/shell -> /sbin/sh

/etc/passwd/xcadman/hash

/etc/passwd/xcadman/uid

/etc/passwd/xcadman/gid

/etc/passwd/xcadman/gecos

/etc/passwd/xcadman/homedir -> /root

/etc/passwd/xcadman/shell -> /sbin/sh

That way, we can use normal file security to say who can and can't change or modify the particulars of the passwd file. If I want to change my shell, all I need do is ln -sf /bin/bash /etc/passwd/myname/shell. I don't have to know what fields of the passwd file are which, and I don't have to learn how to use chsh (or have a suid-root program on my system that is a potential threat to system security).

Of course, the reason we don't do it is because all those little files take up too much space. Well, with ReiserFS, that is no longer the case.

Gun fire is the sound of freedom.

Bad idea (3.00 / 1) (#68)
by drsmithy on Sun Aug 10, 2003 at 01:40:31 AM EST

Because then people can set their shell to be whatever exectuable they damn well please, either opening a gaping security hole, or making it exceptionally easy for users to lock themselves out.

[ Parent ]
they can do that now (2.50 / 2) (#71)
by QuantumG on Sun Aug 10, 2003 at 01:59:35 AM EST

man chsh fool

Gun fire is the sound of freedom.
[ Parent ]
the difference is... (4.00 / 1) (#73)
by durkie on Sun Aug 10, 2003 at 02:27:24 AM EST

...that chsh checks the user's desired shell against /etc/shells to make sure that it's a valid shell. how do you implement that with symlinks?

[ Parent ]
use a plugin (4.00 / 2) (#77)
by QuantumG on Sun Aug 10, 2003 at 02:47:04 AM EST

ReiserFS4 allows arbitary security restrictions to be implemented by an administrator. So you can not only say that the user can change a file, but you can also say what possible values the user can change the file to.

Gun fire is the sound of freedom.
[ Parent ]
They shouldn't be able to (4.00 / 2) (#74)
by drsmithy on Sun Aug 10, 2003 at 02:30:57 AM EST

From a Redhat box:
chsh will accept the full pathname of any executable file on the system. However, it will issue a warning if the shell is not listed in the /etc/shells file. On the other hand, it can also be configured such that it will only accept shells listed in this file, unless you are root.
From a FreeBSD box:
The shell field is the command interpreter the user prefers. If the shell field is empty, the Bourne shell, /bin/sh, is assumed. When altering a login shell, and not the superuser, the user may not change from a non-standard shell or to a non-standard shell. Non-standard is defined as a shell not found in /etc/shells.
As usual the BSD setup is better, but the simple fact is any sysadmin who allows his users to change their shell arbitrarily is negligent (not to mention just plain stupid).

[ Parent ]
The actual way to implement this (4.33 / 3) (#76)
by QuantumG on Sun Aug 10, 2003 at 02:45:22 AM EST

is with a plugin that restricts the possible values that the user can change their shell to.. not a suid.

Gun fire is the sound of freedom.
[ Parent ]
well then (5.00 / 1) (#89)
by darqchild on Sun Aug 10, 2003 at 04:27:19 AM EST

the answer is very simple..
if you don't trust your users to correctly set their own shell, then don't give them write permissions to  "/etc/passwd/lusername/shell".  users can already change their shell with chsh, which lets users choose from a predetermined set of authorized shells

~~~
Death is God's way of telling you not to be such a smartass.
[ Parent ]
So... (none / 0) (#158)
by drsmithy on Sun Aug 10, 2003 at 09:19:14 PM EST

if you don't trust your users to correctly set their own shell, then don't give them write permissions to "/etc/passwd/lusername/shell". users can already change their shell with chsh, which lets users choose from a predetermined set of authorized shells

The allegedly better solution actually *reduces* functionality. Why is it better, again ?

[ Parent ]

use a plugin (none / 0) (#181)
by QuantumG on Mon Aug 11, 2003 at 01:22:13 AM EST

to restrict the possible values of the file to the list in /etc/shells. This is a better solution than writing an arbitary suid program because:
  1. You can now do that with any file you like; and
  2. There's no chance that a suid is going to violate your security model


Gun fire is the sound of freedom.
[ Parent ]
suid (none / 0) (#184)
by binaryalchemy on Mon Aug 11, 2003 at 03:18:53 AM EST

How exactly is having a bunch of little plugins running in the kernel better than having a bunch of little programs running suid? Who's enforcing security on them? In a micro-kernel you could get away with this, but in a monolithic kernel (a.k.a. Linux) these components still need to be given God level permissions to do what they need to do so they're just as much of a risk as suid programs.
------
Defending the GPL from a commercial perspective is like defending the Microsft E
[
Parent ]
actually no (none / 0) (#187)
by QuantumG on Mon Aug 11, 2003 at 04:58:35 AM EST

they run as the user.. but, in our example, only the administrator can set which one to use.

Gun fire is the sound of freedom.
[ Parent ]
SUID (none / 0) (#232)
by drsmithy on Mon Aug 11, 2003 at 11:00:14 PM EST

This is a better solution than writing an arbitary suid program because [...]

Just about anything is a better solution than writing an arbitrary SUID program, because SUID is nothing more than a broken-by-design hack to get around unix's primitive security semantics - the same primitive semantics that make SUID programs such a security nightmare in the first place.

However, the solution on offer here is not much better. Firstly, the potential for exploit simply moves from a program to a filesystem module (running in kernel space, no less - not that there's a huge difference between than and root-privilege code).
Secondly, it doesn't get past the fundamental problem that the owner/group/other security model of unix sucks, so trying to do everything in the filesystem is pretty silly. It is somewhat ironic in light of unix's mostly-but-not-quite-completely-pervasive "everything is a file" philosophy, that its permission capabilities are so limited.
And finally, it requires re-engineering an already working solution, re-educating everyone using the old method and then rolling it out onto millions of systems around the world. All for practically no short-term and very questionable long-term gains.

The "proper" way to do it, from a security perspective, is an ACL-controlled database (which Reiser4, in and of itself, could probably emulate - but most unixes couldn't take advantage of it). Of course, suggesting that tends to put the everything-should-be-in-a-text-file crowd into such a state they often have trouble breathing. Nevertheless, hacking, twisting and distorting a filesystem around the inherent limitations of both it and the system(s) it runs under, is in no way a "proper" solution.

There are undoubtedly places where the advantages of Reiser4 will make it a must have. I don't think trying to force everything to represent itself through a filesystem is one of them.

[ Parent ]

Why is this a security hole? (none / 0) (#134)
by roystgnr on Sun Aug 10, 2003 at 04:52:51 PM EST

And if it is, why does restricting the user to a fixed set of shells remove the security problem, when every one of those shells will probably have a user-editable startup script (like .bash_profile) and a command to replace itself with another arbitrary executable (like "exec /home/lusr/whatever") which could be placed in that startup script?

[ Parent ]
too hard to migrate (4.60 / 5) (#58)
by zarqman on Sun Aug 10, 2003 at 12:36:18 AM EST

this is an interesting idea.  and thanks to the author for taking the time to explain reiser4 well.  all that said, i think reiser4 is very likely to remain a fringe feature.

as addressed, mac os has dealt with meta-data for a long time.  the biggest problems that creates are interoperability issues.  transferring files to other people creates regular problems.  if i attach a file to an email, i have to strip the metadata if it's going to a non-mac recipient.  if i am compressing files, i have to use .sit or else lose the metadata.  if i'm transferring files using software which doesn't recognize metadata (basically everything except my email client) i'm back to .sit files.  

so, to pull off this migration, we need to extend http, ftp, other network transfer protocols, define and implement some mime extension for email.  we also will need updates to popular compression programs, backup programs, and more.  some of this can possibly be worked around using compression programs (as .sit is used on mac), so http, ftp, and the like don't have to deal with said metadata.  but then we'll have to retrain all our users that everything they do has to be compressed first--i don't think so.

the majority of computers are desktops and are used by non-geeks.  these people can barely handle .zips and email attachments.  a system like reiser4 will just make things more complicated for them--especially because the things that are making it more difficult, this metadata, is hidden, non-obvious, non-tangible.  

i fail to see a compelling reason why embedded data, such as id3 tags, are a bad enough to solution to warrant this mess.  

from a practical point of view, i can think of two major technical migrations that need to happen more than this.  and i'm not optimistic on the time frame of either:  moving from ipv4 to ipv6 and revamping the smtp system to deal with spam.  ipv6 will probably happen eventually.  a replacement to smtp probably won't happen at all.  why should this happen?  

to summarize:  a good idea, but much too difficult to implement widely enough to be useful mostly because the collateral damage will be staggering.  

# cat .sig
cat: .sig: No such file or directory

you mean like file permissions (4.00 / 4) (#61)
by QuantumG on Sun Aug 10, 2003 at 01:03:43 AM EST

and every other piece of meta data is just ignored now when you email it to someone. If you *want* the receiver to get the meta data you'll send it with the file.

Gun fire is the sound of freedom.
[ Parent ]
more metadata (none / 0) (#146)
by zarqman on Sun Aug 10, 2003 at 07:08:06 PM EST

correct, if you want them to get it you have to send it. now, how, using present standards, does one send that metadata? you don't. we'll have to have new standards -- either new protocols, extensions to existing protocols, new mime-types, etc. important focus here is something must change in the various transport mechanisms to move said metadata. the need for that change creates enough inertia that the whole concept has a high chance of failure. and at the very least, a long acceptance/roll-out period. at this point, metadata like file permissions and owners isn't sent because it isn't useful. frankly, most additional metadata we're likely to think up and use _would_ be useful to someone else, eg: id3 tag type information, and would therein need to be transferred.

# cat .sig
cat: .sig: No such file or directory

[ Parent ]

not at all (none / 0) (#179)
by QuantumG on Mon Aug 11, 2003 at 12:57:05 AM EST

file attachments already have paths. You would simply send an email with attachments like:

any_given_song
any_given_song/type
any_given_song/artist
any_given_Song/homepage

and when I save them they just work(tm).

Gun fire is the sound of freedom.
[ Parent ]

HTTP/FTP protocols (1.50 / 2) (#67)
by MfA on Sun Aug 10, 2003 at 01:39:09 AM EST

Are the protocols really an issue? I doubt it would be impossible to use the existing semantics to access files as directories, only the servers&clients need changing AFAICS.

As I see it non embedded metadata is not always a choice, it is sometimes plain necessary ... the problems with metadata you mention are inherent to the concept, storing it as files under the original file/directory is the most obvious and tangible way.

[ Parent ]

protocols and metadata (5.00 / 1) (#148)
by zarqman on Sun Aug 10, 2003 at 07:25:08 PM EST

the directory portion of it is doable, yes.  however, from my understanding of the article, a directory implementation of metadata would be a plugin addition to reiser4.  and it would seem that we'd need different plugins for different sets of metadata, eg: one for /etc/passwd, one /etc/aliases, etc....  a generic one would be possible, but then you'd have to determine which files it was supposed to extract into such directory trees and which ones it's not supposed to -- more maintenance headaches.  

of course, a huge collection of plugins also would create maintenance headaches.  everyone who enjoys browsing the web and happening upon a site that requires a new plugin you don't have, raise your hand.  all three of you?  good.  that's what i expected.

i've very rarely seen the need for non-embeddable metadata, although perhaps in whatever field you're most experienced in, you've seen it more (if so, please share -- i'm interested.)  however, when that's the case, storing it as another generic file, without the filesystem handling it specially for you seems appropriate, and is what's done presently.  why do we need to move this seemingly rare case scenario into the filesystem code rather than leave it alone in the applications?  embedded metadata really should be the same way.  it doesn't appear to be that big of a deal for mp3 players to deal with id3.  

back to the plugins:  if we restructure /etc/passwd into metadata, then we either have to have a plugin to make it look like the old single passwd file or we have to modify all programs that access passwd to use the metadata -- the same is true for any other program where we monkey with all this.  we are going to have to change a LOT of programs to make all this work:  locally run programs, protocols, servers, clients, and so on.  yes, it is doable -- and there might even be some small benefits when it's finished.  but, will the benefits outweigh the difficulty to getting the majority of computers online today upgraded to deal with this?  i say no.  if we cannot get a large number of systems to be compatible with this, then the lowest common denominator kicks in (or remains, really) and we are never able to take advantage of it.  at least not without yet more plugins.  

if we, as a computer-using society, are unable to get people to close their mail relays and open proxies, then we have no hope of upgrading everything necessary to make this metadata viable to a large number of people.  the potential benefits do not outweigh the costs.

# cat .sig
cat: .sig: No such file or directory

[ Parent ]

Passwords&aliases are not metadata (5.00 / 1) (#173)
by MfA on Sun Aug 10, 2003 at 11:01:20 PM EST

Strictly speaking the passwd and aliases examples do not actually concern metadata ... they are transparant ways of exposing data captured in multiple files through a pseudo-file for backwards compatibility. It is a seperate issue.

Non embedded metadata concerns things such as user defined attributes and the summary information you can add to files under windows. With metadata everything will always still look like a plain file so I dont see why it should effect the transfer protocols.

As for an example ... I mentioned summary information under windows before, there are several windows utilities which can use the file summary information for answering search queries. The least of which windows own indexing service.

This kind of non embedded metadata is used for indexing images/pdfs/etc. It can be stored externally in a database, or internally in NTFS or Reiser4 ... no additional problems are created by storing it in the filesystem, actually having a direct link between the file and its metadata maintained by the filesystem while moving/copying/deleting files removes some problems.

[ Parent ]

Simple stop-gap solution (none / 0) (#97)
by p3d0 on Sun Aug 10, 2003 at 07:21:20 AM EST

Why not write a utility that converts all file contents+metadata into a simple stream of bytes and back? (Kind of like how tar converts a directory tree to a stream of bytes and back.)

Then any app that doesn't know about metadata could (1) ignore it, just reading/writing the file's actual data (eg. text editors could do this), or (2) use the aforementioned utility and read/preserve all the metadata (eg. compression, transmission, backup, revision control systems could do this).

(As a matter of fact, if I understand the description of reiser4 above, metadata just looks like a directory structure, so tar would almost already do the right thing.)

It's not a perfect solution, as for instance, legacy apps would create files devoid of metadata. However, it's probably good enough for most uses until apps evolve to handle metadata natively.
--
Patrick Doyle
My comments do not reflect the opinions of my employer.
[ Parent ]

Sorry, (5.00 / 3) (#100)
by it certainly is on Sun Aug 10, 2003 at 07:45:32 AM EST

the mac tried this. They realised that simple copies or FTPs of files didn't work on binaries that required the resource fork, so they invented the MacBinary format, the AppleSingle format and the AppleDouble format. They also invented their own UUEncode, called BinHex.

It's an absolute pain in the arse, and every client on every other operating system has to explicitly write code to understand it. Even then, most of the useful information (= ESSENTIAL information for a Mac) is lost when you unpack it to a non HFS/HFS+ filesystem.

kur0shin.org -- it certainly is

Godwin's law [...] is impossible to violate except with an infinitely long thread that doesn't mention nazis.
[ Parent ]

Network protocols (none / 0) (#159)
by baka_boy on Sun Aug 10, 2003 at 09:27:31 PM EST

HTTP, SMTP, POP, and IMAP all support MIME types natively, both for sending and receiving files. In fact, a single email message (or HTTP POST request, with the 'multipart-form' encoding) may "package" multiple text or binary-format files, with each one's MIME type fully specified, and the file safely encoded as 7-bit ASCII for portability.

Most of the popular Internet protocols are actually designed to preserve such metadata, and rely on stable, well-specified standards for how to encode it in a portable way. It's the client machines (notably, pretty much every version of Windows) that fail to maintain this metadata, and therefore create the mess we have today.

[ Parent ]

Metadata in filesystems can't work. (4.00 / 3) (#62)
by acceleriter on Sun Aug 10, 2003 at 01:08:09 AM EST

There is too much of a need to ship files across platforms. This works best when a file is just a stream of bits. Consider the problems with:
  • Mac resource fork headaches
  • OS/2 extended attribute pain
  • The joy of mangled binaries shipped from one VMS system to another via another platform
Let me revise this--metadata in the filesystem can work iff practically everyone agrees on a filesystem. IOW, if Microsoft does it, and doesn't encumber it with patents, it could be widely adopted.

Metadata not crucial (3.80 / 5) (#84)
by lambda on Sun Aug 10, 2003 at 03:22:36 AM EST

This is really not as huge of a problem as everyone makes it out to be. Practically all metadata that exists (right now) you do not get: file owner, group owner, permissions, timestamps, ACL's, etc, because most stuff is system dependent and not important for the receiver to get; most metadata is not crucial. If I get an MP3 without an ID3 tag, I can still play it.

The nice thing is, if the transition to files as directories is made on most OS's, then metadata storage is permanently solved; i.e., no repatching tar every time someone invents a new attribute. In sum, if your platform supports the metadata, you gets it, otherwise, no big loss.

[ Parent ]
Metadata does matter (4.50 / 2) (#147)
by acceleriter on Sun Aug 10, 2003 at 07:17:19 PM EST

For example, a document that's perfectly serviceable on a PC, when copied to a Mac, would be unopenable (without going through the mojo to tell the Finder what file type it is, i.e. specifying the metadata manually) even if the file was in exactly the same format (e.g. PSD).

From this user's point of view, it's more trouble than it's worth to try to store all that mess in the filesystem, when extensions/magic headers work well enough, and more extensive tagging needs (e.g. ID3) take care of themselves.

[ Parent ]

Good point (none / 0) (#149)
by lambda on Sun Aug 10, 2003 at 07:32:54 PM EST

I forgot about file type, and it is probably the most important datum. One overlooked solution would be for Finder (and other file managers) to use heuristics like the 'file' utility, to determine what kind of files unknowns are. That capability would still be very useful even in a perfect world.

[ Parent ]
atomicity: the push, the pop (3.00 / 1) (#63)
by xs euriah on Sun Aug 10, 2003 at 01:17:40 AM EST

a UNIX-like system will open the file, find that it has what is called a "magic line", and run the script through the program /bin/csh, the C-shell. A file is read to find the interpreter to be used, the permissions on that file are checked, and the interpreter is invoked with the script as input. Seems innocent enough unless a script kiddie has good timing. Imagine that after the permissions were read but before the script is read in and passed to the interpreter, the contents of the file were changed? Suddenly you have arbitrary code running.

Why not push it into a read only stack: arbitrary code would be impossible to run.



Typically... (4.00 / 1) (#85)
by valar on Sun Aug 10, 2003 at 03:48:21 AM EST

Typically, it is all put on a stack, then fed to the interpreter. This way, there is no appreciable time for one to modify the code. Additionally, arbitrary code is only possible if you somehow have the perms to modify the file. If you don't trust what a user would do with your scripts, he or she shouldn't have access to modify them.

[ Parent ]
You're solving the wrong problem (none / 0) (#96)
by p3d0 on Sun Aug 10, 2003 at 07:12:30 AM EST

Look again. This is not a stack overflow attack that can be solved with read-only pages. It's a script being executed by an interpreter.
--
Patrick Doyle
My comments do not reflect the opinions of my employer.
[ Parent ]
Look again (none / 0) (#106)
by mikpos on Sun Aug 10, 2003 at 11:26:15 AM EST

Of course it's a script being executed by an interpreter. The original exploit was to modify the script in the brief window between perm checking and execution. If the script is held in a read-only portion of memory, it makes it quite difficult to modify, don't you think?

[ Parent ]
Nope (1.00 / 1) (#193)
by p3d0 on Mon Aug 11, 2003 at 08:33:31 AM EST

Are we reading the same exploit? You change the file, not the memory. That's the attack.
--
Patrick Doyle
My comments do not reflect the opinions of my employer.
[ Parent ]
Yeah, but PERL reads files... (none / 0) (#252)
by Alhazred on Tue Aug 12, 2003 at 10:57:29 AM EST

Not memory. All the kernel does is essentially FORK the interpreter and direct its standard input to the file in question, now you would have to have a whole different way of getting a file, and EVERY interpreter would have to be patched/written to understand that. Not only that but what if I want to run my 12 megabyte perl script? You really want to buffer that in memory a 2nd time?
That is not dead which may eternal lie And with strange aeons death itself may die.
[ Parent ]
I'm too late (1.57 / 7) (#64)
by RyoCokey on Sun Aug 10, 2003 at 01:23:20 AM EST

Everyone has already trolled the linux users. :(



farmers don't break into our houses at night, steal our DVDs and piss on the floor. No
The Internet & the problem of interoperability (4.90 / 11) (#66)
by MichaelCrawford on Sun Aug 10, 2003 at 01:37:04 AM EST

Two of the nicest things that the Mac OS had from the earliest days was that the file type and creator code of a file were embedded in the filesystem metadata.

For example, where Windows might call a file ReadMe.txt and it would always open in NotePad, on the Mac you could call it just ReadMe and the file would open into whatever application created it. The type and creator are each 32-bit values usually spelled out as their four-character ASCII equivalents, so our ReadMe would have type 'TEXT' and would open in TeachText if the creator were 'ttxt'.

While you can change the file association in Windows, you change it for all files with the given extension; on the Mac, each file of a given format can open into a different editor, and it is possible to change the creator if you want to.

BFS (NOT BeFS - that's not the correct name for it, just what it's called on Linux) went further with this by giving each file a mimetype string as its filetype, rather than a 32-bit value.

I personally feel that embeddeding the type and creator code in the metadata was one of the more significant advances that came from the creation of the Mac OS. It's right up there with having a GUI at all - and apple didn't invent the GUI. (I don't know who invented the type and creator code though, but the Mac OS was the first to use it commercially as far as I know.)

One nice thing about it is that the user can name the file anything they want to, with no ugly extensions, and there are no issues with being unable to open the file if the file gets renamed the wrong way. It is absolutely wrong to indicate the format of a file in its name - the entire name should be under the user's control.

Unfortunately, that's not the way the Mac is today, not under Mac OS X. While the HFS and HFS+ filesystems still support types and creator codes under OS X, their use is deprecated, and they are not supported at all under the UFS (BSD) filesystem that is also available for OS X.

Apple has been so adamant about discouraging the use of types and creator codes that many applications don't set them anymore, notably most of Apple's own applications. The header files that are found in /System/Library/Frameworks don't have a 'TEXT' filetype, for example.

The reason Apple gives for this is that the type and creator metadata are incompatible with the Internet. File exchange with other operating systems has always been troublesome, but it is doubly so in today's highly networked age, especially with heterogenous office LANs. Apple also discourages the use of resource forks, a second bytestream in the file which is structured as a simple sort of database.

However, I feel that this is just the wrong attitude to take. Rather than dumbing down one of the nicest features of one of the nicest operating system so that it works like the more primitive systems that happen to be more widely used because of the more effective marketing of Apple's competitors, I feel that Apple should keep the old metadata, and even to add even richer metadata as this article suggests, and to work with the IETF, the W3C and other standards bodies to work out ways to allow file transmission over the internet with the metadata being included.

And I don't mean emailing around interchange formats that have to be converted to use, like MacBinary files. I should be able to post a file with a type, creator code and resource fork unaltered on my web page, which is hosted on a linux box, and have a windows user be able to download it and put it to immediate use.

That's what I'd like to see the Linux community do too, once Reiser4 is ready for production use. You need to do a lot more than rewrite your applications. You also need to show up at standards committee meetings so that network protocols are revised (or new ones created) to deal with files that have metadata.

Windows and OS X both make the attempt to deal with the problem that filetypes present us by hiding from from the user in the UI. That way the user can't mess up the extension when editing the filename. But it creates the problem that more than one file can seem to have the same filename - ReadMe.txt and ReadMe.mp3 will present the same names even though one is text and the other is music.

This is discussed in Hiding Extensions in Mac OS 10.1 -- why it's a BAD idea.

Really weird things happen when extensions are hidden and you try to create a file with an extension on purpose. For example, it often happens that when I try to write a web page in Mac OS X' TextEdit application that the name of the file on disk ends up being called either "homepage.html.txt" or "homepage.html.rtf".

A while back Slashdot covered an Ars Technica article called Metadata, The Mac, and You which discussed the issue in some detail and similarly concluded that for Mac OS X to denote the file format by the filename extension is a really bad idea.


--

Live your fucking life. Sue someone on the Internet. Write a fucking music player. Like the great man Michael David Crawford has shown us all: Hard work, a strong will to stalk, and a few fries short of a happy meal goes a long way. -- bride of spidy


MIME types as creator codes (5.00 / 3) (#70)
by QuantumG on Sun Aug 10, 2003 at 01:54:35 AM EST

What's funny is that if you try to email something on MacOS X it has to look at the extension to figure out what MIME type to put in the email.. whereas if the MIME type were stored as an attribute of the file then it would be trivial to send a file with the correct MIME type.

Gun fire is the sound of freedom.
[ Parent ]
Interoperable solution (4.00 / 1) (#101)
by Julian Morrison on Sun Aug 10, 2003 at 08:24:05 AM EST

I expect there will be some hack that will let you tar-up a file "in directory mode". So when you expand foo.mp3 on a less well-endowed operating system, you'd get a directory called "foo.mp3" containing the files "foo.mp3" "foo.id3" "foo.permissions" etc etc.

[ Parent ]
Filesystem "plugins" mentioned (5.00 / 1) (#119)
by BeeDee on Sun Aug 10, 2003 at 02:24:16 PM EST

ttfkam mentioned in another comment that Reiser4 supported "filesystem plugins" that could handle this kind of thing.

What I imagine that to mean is that when a file with a name ending in "MP3" is copied into a Reiser4 filesystem with an approriate plugin, it will recognize that this is a file whose format it understands and will "unpack" the contents of the ID3 tag into metadata files as described in the article. Then, when the Reiser4 filesystem is later called upon to produce a copy of the MP3 file for export to some other filesystem, the plugin would once again be called upon to recompile the metadata files into an ID3 tag to append to the data file. If the plugin is written well, then nothing should be lost in the process and the MP3 that comes out the other side should be byte-for-byte identical to the one which went in in the first place.

However, that's just my interpretation of a one-line comment, and I haven't actually read up on the subject. So I could be wrong. :)

[ Parent ]

Filesystem "plugins" mentioned (none / 0) (#254)
by ArsonSmith on Tue Aug 12, 2003 at 12:18:39 PM EST

I don't think just copying the file to the file system will do this.  And I also don't think there will be any unpacking actually done.  I think the plugin would only be used when someone does:

cd file.mp3
ls

and even this wouldn't actually unpack anything.  if someone cats file.mp3/id3 it would filter the file and drop out just the id3 tags, and possibly store them in the files meta data for future performance speedup, or if you cat file.mp3/raw it might run the entire mp3 through a super efficient mp3 to au converter and give raw audio output.  This I would guess wouldn't be cashed in meta data but in file system buffers due to the size but would be accessible from the plugin at any time.

This way the plugins could be made to filter any file type so your music player could play any file type you have file system plugins for.  These file system plugins could be used and optimized by everyone that has ever wanted to make an audio/movie/picture/etc viewer.  Then making the viewer you only have to support a raw format and let the file system take care of everything else.

[ Parent ]

What, how is that useful? (4.14 / 7) (#105)
by joto on Sun Aug 10, 2003 at 11:25:07 AM EST

For example, where Windows might call a file ReadMe.txt and it would always open in NotePad, on the Mac you could call it just ReadMe and the file would open into whatever application created it.

So, if some looser wrote it in vi, I couldn't read it in emacs? You realize that usually there are more than one program that can handle a file-type, don't you? Except for limiting metadata to three letters, windows actually did it right. You can have several actions associated with each file-type, and also set one of them as default.

The type and creator are each 32-bit values usually spelled out as their four-character ASCII equivalents, so our ReadMe would have type 'TEXT' and would open in TeachText if the creator were 'ttxt'.

Yeah, I can imagine this must be much better than the three-letter file-extensions used in windows. After all, on the mac you could use four letters!

While you can change the file association in Windows, you change it for all files with the given extension; on the Mac, each file of a given format can open into a different editor, and it is possible to change the creator if you want to.

Ok, I admit this could be useful. But could you also change it for all files?

One nice thing about it is that the user can name the file anything they want to, with no ugly extensions, and there are no issues with being unable to open the file if the file gets renamed the wrong way. It is absolutely wrong to indicate the format of a file in its name - the entire name should be under the user's control.

No, it's not absolutely wrong. There are good arguments on both sides. For example, I find it mighty convenient that I can simply give a file a .c, .java, or .html extension to differentiate them from a normal .txt file (which is what it really is). Being forced to do this via more cumbersome file-attributes doesn't really strike me as helpful.

Furthermore, even if it is more aesthetically pleasing to separate name and type, humans often like a bit of redundancy. Being able to quickly spot a .doc file, instead of reading it's mime-type, or looking at the resource fork, or the icon (of which there are far too many to remember which is which), is sometimes useful.

Note that I am not saying that file-extensions are the right thing. All I am saying is that they are certainly not "absolutely wrong". There are good arguments on both sides.

The header files that are found in /System/Library/Frameworks don't have a 'TEXT' filetype, for example.

Ahh, another great argument against universal file-type schemes. They are not simply 'TEXT' files. They are header-files. A good metadata facility should be able to tell you, depending on the amount of detail you need, that they are either text-files, or header-files. MIME, with it's limited two-level hierarchy is not able to do this. And a more complex system, would have other disadvantages, such as being more complex.

[ Parent ]

Type/Creater codes. (5.00 / 1) (#151)
by FyreFiend on Sun Aug 10, 2003 at 07:46:00 PM EST

So, if some looser wrote it in vi, I couldn't read it in emacs? You realize that usually there are more than one program that can handle a file-type, don't you? Except for limiting metadata to three letters, windows actually did it right. You can have several actions associated with each file-type, and also set one of them as default.

Not quite how it works. On a Mac you create a text file with, say SimpleText. You can still open the file with BBEdit, emacs, vi, whatever. Just when you double click on that file it will fire up in SimpleText.



--
Only kings, presidents, editors, and people with tapeworms have the right to use the editorial "we".
-- Mark Twain


[ Parent ]
The OS/2 way ;-) (none / 0) (#177)
by uhoreg on Mon Aug 11, 2003 at 12:41:06 AM EST

The Macintosh creator attribute seems useless to me. Under OS/2 (at least under the Workplace Shell), you could assign a default application to open files of a specific extension (yeah, unfortunately, they still used file extensions), but for specific files, you could select to open it in a specific application. (Nautilus (the file manager in GNOME) also does similar things.)

[ Parent ]
Recommended Reading (4.85 / 7) (#69)
by MichaelCrawford on Sun Aug 10, 2003 at 01:53:30 AM EST

Few people are qualified to discuss filesystem issues until they have read:

Besides explaining the BFS in some detail, it gives an overview of a number of popular filesystems, discussing their data structure, and relative strengths and weaknesses. The other filesystems covered include:

  • BSD Fast Filesystem
  • Linux ext2
  • Macintosh HFS
  • Irix XFS
  • Windows NTFS (or what was publicly known about it at the time anyway)
He also explores some of the issues and tradeoffs that influence the design of a new filesystem. It is to Dominic's credit that he thoroughly investigated a number of other filesystems before designing and implementing the BFS.

It's also to his credit that as far as I know he wrote the feature-rich, journaling (fault-tolerant), multithreaded BFS pretty much by himself. Be, Inc. never had that many engineers working for it.

You may also be interested to read an interview of Dominic Giampaolo.


--

Live your fucking life. Sue someone on the Internet. Write a fucking music player. Like the great man Michael David Crawford has shown us all: Hard work, a strong will to stalk, and a few fries short of a happy meal goes a long way. -- bride of spidy


this started as a great article... (2.00 / 3) (#83)
by j0s)( on Sun Aug 10, 2003 at 03:17:36 AM EST

... and turned into a bitchy flame-war. its sad really.

-- j0sh



security mistake (4.40 / 5) (#87)
by Alt SysRq B on Sun Aug 10, 2003 at 03:58:55 AM EST

A shadow file would no longer be necessary as the user's password hash does not need to be protected from the user; They already presumably know their password.

Wrong. A shadow file is also useful if an intruder gets access to a terminal that's already logged in. Your idea (let the users read their own password hashes) would let the intruder read the hash for that account. Which would be impossible with a shadow file.



Write-only (none / 0) (#92)
by Wouter Coene on Sun Aug 10, 2003 at 05:34:28 AM EST

A shadow file is also useful if an intruder gets access to a terminal that's already logged in. Your idea (let the users read their own password hashes) would let the intruder read the hash for that account. Which would be impossible with a shadow file.

Why not make the file that contains the password hash write-only for the user then? That wouldn't solve another problem though: the just outright changing of a user's password by an intruder.

[ Parent ]

Can't read or change... (3.66 / 3) (#98)
by WWWWolf on Sun Aug 10, 2003 at 07:32:16 AM EST

Why not make the file that contains the password hash write-only for the user then? That wouldn't solve another problem though: the just outright changing of a user's password by an intruder.
But /bin/passwd asks for old password before letting the user to change it (to protect the user in case the intruder gets to an open terminal, probably), which in turn either involves reading the password hash, or authenticating some other way (PAM). So there's no point in making it write-only... Allowing the user to read or write the hash directly is dangerous...

-- Weyfour WWWWolf, a lupine technomancer from the cold north...


[ Parent ]
Shadow files offer other benefits (none / 0) (#157)
by baka_boy on Sun Aug 10, 2003 at 09:19:13 PM EST

Most shadowed-password systems I've used in recent memory use at least MD5 to hash the password before storing it on disk, rather than simply hiding it via file permissions. That means that even if an attacker gained read access to the 'shadow' file, they couldn't get your password by any means other than brute-forcing a strong (not unbreakable, but strong regardless) hash algorithm.

Of course, if someone has physical access to a machine, especially with a terminal already logged-in active, they pretty much have unlimited options for how to "0wn" you. Even under current POSIX-like systems, most users have the ability to change their own login password, or to alter settings such as the search path for executables, so an attacker could always lock you out or install a keystroke logger to collect remote logins and terminal activity.

If you want real fine-grained access control and security, something like Reiser4 will help by allowing you to specify precise access semantics for not only entire files and directories, but individual metadata attributes of each file. Also, with the ability to make directories masquerade as single files, you could allow the appearance of range-specific access: i.e., a user or group could be allowed to access all but the first kilobyte of a "file", which makes implementing things like signed binaries trivial.


[ Parent ]

Great idea. (4.66 / 6) (#99)
by megid on Sun Aug 10, 2003 at 07:38:51 AM EST

One wonders a bit why many around here are flaming as if a file system was a topic for alt.religion.emacs; no trace of a positive comment to be found (on top-level, that is).

I find the idea of metadata/"inside files" novel and useful. There has never been a standard access to arbitrary metadata before, and I welcome every effort to design a good one.

This "POV of files" thing is easy to comprehend -- after all, the "standard" of using files is so deeply ingrained in our brain that most of us dont ever doubt the concept of files anymore. To me, this programming approach seems certainly useful for future projects.

I salute the ReiserFS team. Good work.

--
"think first, write second, speak third."

Wrong. (4.50 / 4) (#102)
by Antiorganic on Sun Aug 10, 2003 at 10:34:00 AM EST

"Unfortunately, there were already quite a few programs that understood ID3v1 meta information. In order to add more information, the additional information had to be added in front of the ID3v1 information but, again, after the audio data."

No. ID3v2 data is at the beginning of the file, before the audio data.

Link (3.00 / 3) (#113)
by Detritus on Sun Aug 10, 2003 at 01:10:36 PM EST

ID3 is not nearly as inadequate or badly patched up as the author suggests. It definitely is painful to implement; Xmms, for example, still only supports ID3v1. Here you go: ID3v2 made easy.

Kings and lords come and go and leave nothing but statues in a desert, while a couple of young men tinkering in a workshop change the way the world works — Havelock Vetinari
[ Parent ]
Thanks for the link (5.00 / 1) (#115)
by ttfkam on Sun Aug 10, 2003 at 01:40:22 PM EST

I stand by my assertion that apps would be simpler if they wrote to file attributes instead of having ID3 code.  But then, I don't gather you disagree.  ;-)

If I'm made in God's image then God needs to lay off the corn chips and onion dip. Get some exercise, God! - Tatarigami
[ Parent ]
Portability (none / 0) (#212)
by Cheetah on Mon Aug 11, 2003 at 01:33:05 PM EST

Even if all computers eventually support metadata attributes, they won't support it in a sufficiently same way for many years to come, I expect.  So, if you want to be able to preserve metadata when moving between systems, you have to integrate it into the data stream of the file.

Remember BinHex?  Eeeeevil!

[ Parent ]

You're right (5.00 / 1) (#114)
by ttfkam on Sun Aug 10, 2003 at 01:38:09 PM EST

My mistake.  A question: How do older clients that don't support v2 know to skip the data they don't understand and get to the audio data?

Am I correct in assuming that the motivation for info at the front of the file was for streaming purposes?

If I'm made in God's image then God needs to lay off the corn chips and onion dip. Get some exercise, God! - Tatarigami
[ Parent ]

sync bits (5.00 / 1) (#133)
by sesquiped on Sun Aug 10, 2003 at 04:36:23 PM EST

The MPEG spec says that any valid frame must start with a sequence of 12 consecutive one bits. All compliant MPEG players scan forward in the stream until they hit such a sequence. The ID3v2 spec has provisions for escaping any sequence of 12 on bits that might appear in the ID3v2 data.

As far as I know, the primary motivation was streaming. Putting data at the front causes some new problems, like trying to add a piece of metadata would mean pushing the whole rest of the file up a few bytes, which would be extremely inefficient on any standard filesystem. To deal with that, the spec mentions that any application that creates an ID3v2 tag should leave lots of empty padding space so that you can add a reasonable amount of new data without pushing the real data up.


[ Parent ]

Security of #! (3.00 / 1) (#103)
by Skapare on Sun Aug 10, 2003 at 10:46:09 AM EST

Shell scripts run via #! are insecure because the original design was broken. But this is not the only "interpreter" used by exec. Even binary executables are "interpreted". What I'm really referring to is the run-time dynamic linker which links in dynamic shared libraries. But this is actually done securely because the kernel maps the executable file as well as the dynamic linker before anything in the process space is run. Thus, there is no window of opportunity to switch the executable with an exploit. Shell scripts, instead, are run by passing the file NAME to the shell interpreter. This opens the opportunity because now the shell interpreter has to open the file, which by now might be a different file. A better design would have been for the kernel to leave the original shell script open, and pass the open descriptor to the shell interpreter (for instance it could have been standardized to always used descriptor 3 for #! style scripts, and indicate this to the interpreter by some standard argument such as "-3"). Then the interpreter will be certain to be reading the very same file that had the associated SUID bit and owner. We might be able to fudge that even now on some systems by a kernel patch to leave the script file open and pass a /proc/self/fd/3 or /dev/fd/3 filename (assuming the search sequence for opening those isn't expoitable).</p?

So don't do that! (5.00 / 1) (#137)
by derobert on Sun Aug 10, 2003 at 05:06:45 PM EST

Why, exactly, are you executing files that someone else can write to or replace? That's your real security problem.

[ Parent ]
Missing word (5.00 / 1) (#166)
by greenrd on Sun Aug 10, 2003 at 10:07:58 PM EST

Shell scripts run via #! are insecure because the original design was broken.

I didn't understand your argument at first. You forgot to mention the word "setuid" in this sentence.

In fact, Linux ignores the setuid bit on scripts, so it is not a security issue on Linux. It's a lack-of-functionality issue.

Otherwise I agree with your comment.


"Capitalism is the absurd belief that the worst of men, for the worst of reasons, will somehow work for the benefit of us all." -- John Maynard Keynes
[ Parent ]

Plan 9 (3.00 / 1) (#104)
by danny on Sun Aug 10, 2003 at 10:57:05 AM EST

Doesn't Plan 9 do some of these things? It certainly takes the "everything is a file" idea further than Unix.

Danny.
[900 book reviews and other stuff]

Yes (5.00 / 1) (#142)
by lambda on Sun Aug 10, 2003 at 06:03:50 PM EST

Plan 9 has two things: per-process namespace (my filesystem tree can be different from another user's) and unions (multiple directories unioned into one). Linux added the first feature in 2.5.2, and Al Viro plans to add the second after 2.6.0 is out.

[ Parent ]
Yes (4.00 / 1) (#163)
by greenrd on Sun Aug 10, 2003 at 09:55:14 PM EST

It looks like Linux - and currently-third-party projects like LUFS - are slowly subsuming some of the innovative ideas of Plan 9, Hurd, and other alternatives... so that you won't have to choose between fancy filesystems and poor hardware support, or stodgy old filesystems and medium hardware support. :)


"Capitalism is the absurd belief that the worst of men, for the worst of reasons, will somehow work for the benefit of us all." -- John Maynard Keynes
[ Parent ]

*NIX (5.00 / 4) (#107)
by UnConeD on Sun Aug 10, 2003 at 11:32:26 AM EST

Whenever inertia is mentioned, people seem to think of stupid managers who love their MS Outlook so much, but it's equally valid for the *NIX admin who is so used to having his Apache logs at /var/www/apache/whatever that he protests at any attempts to change it. Regular people are used to capitalisation, writing words out in full rather than using cryptic abbreviations and sorting things in a way that actually benefits them rather than the programmer who designed it.

Perhaps ReiserFS4 can't fix all of that, but it sure is a step in the right direction.

Perhaps being knowledgable about a *nix will no longer require a God-like status, but anyone who doesn't consider that a good thing needs to get out more.

Extendable attributes and NCP (5.00 / 1) (#108)
by sphealey on Sun Aug 10, 2003 at 11:36:57 AM EST

All filesystems contain meta information -- data about data: a file's size, a directory's creation timestamp, etc. This list of meta information is fixed, however. If I wanted to add a "usefulness" attribute to every filesystem entry for example, I would have to patch the filesystem and quite likely the kernel to allow access to this attribute.
Correct me if I am wrong, but Novell Netware (NCP) has supported user-definable file attributes since Version 4, released in 1994 and very widely deployed.

sPh

Salvage is the best damn feature ever (none / 0) (#189)
by matthead on Mon Aug 11, 2003 at 06:35:30 AM EST

Are you sure? I never knew about that (I've worked with NetWare 4 and 5) - but then, everything I learned about Novell was from using it. I almost never read any docs. Ceertainly no books, and only a few TIDs and such.

The best two things about NetWare's filesystem was the salvage feature (quite the lifesaver! why the hell they never tried to sell their fs is beyond me), and the elevator code. I still think Novell had the fastest multi-user filesystem ever when under load.


--
- Matt
I'm at (-3.1, -5.0). Where are you?
[ Parent ]
BFS did all this, and then some... (5.00 / 4) (#111)
by jonr on Sun Aug 10, 2003 at 12:27:44 PM EST

BFS is what I miss most about BeOS. Very fast, stable and with nice features. Live queries were a lifesaver. Instead of opening mail application, I just kept one query on my desktop: "File of type E-Mail and Status is New". Custom tags were easy to add, and everything was visible in the file browser. Same program used for viewing email, addresses, mp3 files, etc. I want THAT in Linux, and I would gladly sacrifice IMatch, games and Photoshop. J.

It's coming (none / 0) (#175)
by uhoreg on Mon Aug 11, 2003 at 12:24:34 AM EST

Reiser6, I believe, will add semi-structured queries, so that you could do your email query under Linux. (Reiser5 I think is going to be a distributed FS.) But, of course, FS support for that stuff won't matter much until we get application support as well.

[ Parent ]
The joy of different operating systems. (5.00 / 4) (#112)
by it certainly is on Sun Aug 10, 2003 at 12:54:14 PM EST

I'm all for having fun things like extended metadata on a local system. However, I never kid myself that these things will be portable to different platforms. They never are.

What is portable? Surely, plain text would be portable? Alas, no. MacOS uses CR bytes to mark line endings. UNIX uses LF bytes to mark line endings. MS-DOS uses CR/LF bytes to mark line endings, and a ^Z to mark the end of file. While each OS uses ASCII (thank goodness for that), each OS uses a different character set, and might even use Unicode or UTF8. Then there are thousands of file formats for documents, image files, sound, everything. These are all incompatible with each other, they all have different features, and we still need converter programs that may drop some information when converting from one format to another. The only way to fix this is to use human intervention to take as best advantage of lesser formats as is possible.

We have to be brutally honest. Nothing will be perfect. Nothing will preserve all metadata all the time. We simply have to realise this, and avoid relying on metadata -- we can certainly use metadata when we know it's available, and we can certainly generate metadata from our own sources. For example, all my MP3s have their ID3 tags metadata stored in their filename: "/san/mp3/albums/Artist - Album/NN Track.mp3" or "/san/mp3/albums/Various Artists - Album/NN Artist - Track.mp3". A perl script makes ID3s based on this. However, it's not reasonable to actually store all mail headers in metadata -- instead, mail headers should be stored in the actual file data, and replicated in the metadata. We can now do filesystem searches based on metadata, but our mail headers are still there, with no special coding needed, when we insert the file into a text editor.

Does a file have to be a directory too? This really is just a gimmick, and it's possible in all filesystems. ARCHandler on the Amiga would give you an LHA file if you asked for "ARC:ram/temp.lha", but would give you a directory listing of the contents of that lha file if you asked for "ARC:ram/temp.lha/". A filesystem's number one feature is the key(pathname) -> data translation. This can be as esoteric as wanted. For example, zerofs on the Amiga would give you a file with exactly 500 zero bytes in it if you opened the file "ZERO:500". Linux has the procfs, which translates pathname accesses into operating system queries. Nobody would think it sensible to extract a tar file to /proc, because its contents aren't real files. The same goes for accessing a file's metadata by accessing it as a directory. This is just user interface to the filesystem, not filing itself. On the disk, an object is either a directory or not. It's a special type of inode, just like files, FIFO pipes, block special files and char special files.

On a related note, how does one access the metadata of a directory? You couldn't just add a slash to the directory name, that always has the semantics of "list/access files in that directory" under UNIX, no matter how many slashes you use.

kur0shin.org -- it certainly is

Godwin's law [...] is impossible to violate except with an infinitely long thread that doesn't mention nazis.

SoftUpdates! (4.25 / 4) (#117)
by redelm on Sun Aug 10, 2003 at 02:20:07 PM EST

New FS? Journalling? Why? For crash recovery? I prefer Kirk McKusick's SoftUpdates found in *BSD. Transparent, no new filesys. Just order the writes so data hits the disk before metadata points to it. I pulled the plug on 4 FreeBSD kernel compiles near the end, and `make` picked up without a hitch in three of the four. On the fourth, `make clean` was necessary, but the fs wasn't scrambled, and the power-on fsck was always clean. Don't try this on Linux! It happened to me once, and I had to reinstall the /usr partition.

Don't try this on Linux? (none / 0) (#124)
by loucura on Sun Aug 10, 2003 at 03:27:48 PM EST

I think you mean, don't try this on linux without a good journalled file-system. Sure, if you used ext2 and pulled the plug you'd lose data, but with a good journalled file-system, you generally don't have that problem.

My evidence, of course, is as anecdotal as yours but since all my Linux machines run ext3 or ReiserFS, I've never lost data as a result of unexpected power loss.

[ Parent ]

I think this is a misunderstanding.... (5.00 / 1) (#150)
by mindstrm on Sun Aug 10, 2003 at 07:42:20 PM EST

Journalling filesystems do not prevent you from losing data.. they prevent the need for a long filesystem check. Data still gets lost.. the only difference is, the filesystem knows what's what without having to analyze it. It's just as dangerous to yank the power.

[ Parent ]
NTFS (none / 0) (#191)
by Kruador on Mon Aug 11, 2003 at 06:41:00 AM EST

I know a little of NTFS internals (basically, what I've learned from Inside Windows 2000 by Solomon and Russinovich).

NTFS journalling basically works by recording what it's about to do to the file system structure before it does it. Periodically it writes a checkpoint indicating which transactions have been completed (this prevents having to serialize operations on the file system). When the computer reboots, it looks backwards through the log for the last checkpoint and either rolls forward any operations that are fully recorded but not necessarily fully completed, and rolls back any operations that aren't fully recorded.

The records in the transaction log are written in such a way that they have no adverse effect if an operation is redone in either direction (e.g. the record says 'set this bit/clear this bit' rather than 'toggle this bit').

This is only done for file system structures such as the master file table, directory entries and the allocation bitmap. It isn't done for regular files. Important files like registry hives have their own journalling system for data on Windows NT-based systems (this also offers journalled registry on FAT file systems). You can still lose data, but you shouldn't lose the whole file system - assuming that the underlying disk hardware and drivers are working correctly, i.e. writing the data that the file system passes faithfully, and reading it back correctly. If your disk subsystem is failing, all bets are off.

I've personally not had a problem with CHKDSK on Windows 2000, but then I've not had to use it very often.

[ Parent ]

Ah, thanks for the clarification. (none / 0) (#231)
by loucura on Mon Aug 11, 2003 at 08:52:57 PM EST

I was under the impression that journalled filesystems mitigated data loss by maintaing a journal of the data being written, so that it would know what made it to the disk and what didn't, thereby minimising data-loss to that which wasn't written to the disk before an unexpected crash or power-out.

Thanks for clarifying that for me.

[ Parent ]

It depends (none / 0) (#250)
by Alhazred on Tue Aug 12, 2003 at 10:47:33 AM EST

Both ReiserFS (at least V4) and ext3 provide both metadata journalling AND optionally data journalling. Most people only journal metadata because data journaling is expensive. The result is that if you do only metadata journaling your file SYSTEM should always be consistent, but the contents of files are anyone's bet. So if you pull the plug on a system running Reiser3 its possible that any open file might end up corrupt, but unlike with ext2 you won't suddenly not HAVE the file anymore, nor will some random trash show up in the middle of your file.

There are some applications where full data journaling might be appropriate, but it WILL create a pretty big performance hit (like around 50%) on the filesystem.
That is not dead which may eternal lie And with strange aeons death itself may die.
[ Parent ]

data=ordered (none / 0) (#257)
by uhoreg on Tue Aug 12, 2003 at 03:02:24 PM EST

Reiser4 and ext3 (and ReiserFS 3 with a patch) also have a data=ordered mode, which does exactly what redelm's SoftUpdates comments suggested: it ensures that data is written before metadata, so you get the same result as data journaling, without the performance hit.

I'm sure SoftUpdates is nice, and will save your butt, but I doubt it will guarantee filesystem consistency the way jounaling does. AFAICT, you can still run into problems if you pull the plug just as the OS tries to sync the metadata. SoftUpdates just makes sure that the metadata isn't synced very often.

[ Parent ]

There is always a chance of inconsistency (none / 0) (#278)
by Alhazred on Wed Aug 20, 2003 at 11:37:04 AM EST

Logically NOTHING can GUARANTEE consistency. The best you can do is say 'Consistency is guaranteed as long as the hardware works right'. That will always be so.

Ordered writes do as you say, though if you peruse some of Hans Reiser's papers carefully you will come to the conclusion that in practical reality ordered writes don't come free after all, though their additional costs may be defrayed.
That is not dead which may eternal lie And with strange aeons death itself may die.
[ Parent ]

Ganger and Patt's Soft Updates (none / 0) (#132)
by HidingMyName on Sun Aug 10, 2003 at 04:25:23 PM EST

Was implemented by McKusick, but was invented by Greg Ganger while he was studying under Yale Patt for his Ph.D. Your description is a bit simplistic, soft updates caches some updates so that many local updates can be aggregated, the tricks are pretty clever.

[ Parent ]
features and fsck (4.80 / 5) (#122)
by TRS-80 on Sun Aug 10, 2003 at 03:15:26 PM EST

To quote someone else, "Hans Reiser would do well to spend his time making good filesystem recovery tools than trying to get experimental code into the kernel" (or words to that effect, after the recent Reiser 4 announcement calling for testers). While slightly unfair, I have heard quite a few horror stories about people losing their entire disk to filesystem corruption that could not be repaired by fsck.reiser or other recovery tools.

Because of this, reiserfs rapidly fell out of favour as the preferred journalled fs, with ext3 taking over (espcially for upgrades: what could be easier than tune2fs -j /dev/hd?# compared to a dump and restore?) and some preferring XFS (particularly for large filesystems and ACL support). Both of these are well tested formats (XFS has been developed by SGI for years, ext[23] follows a quite conservative development model which has plenty of regression tests for fsck). While fancy features are fine in mail client or web browser, when it's my filesystem and all my data at stake, I'll take a well-tested, reliable filesystem over one that's featureful, fast and prone to losing data, any day of the week.

ReiserFS is not supposed to have a fsck (5.00 / 2) (#126)
by vadim on Sun Aug 10, 2003 at 03:57:05 PM EST

The whole idea of a journalled filesystem is that metadata is always correct, so fsck is not needed. If it happens otherwise it's the filesystem code what needs to be fixed.

So it's more or less logical that ReiserFS doesn't have a really good fsck, since it's only intended to fix errors made by buggy filesystem code, and perhaps hardware failure.

I have used ReiserFS starting at 2.4.1 (rather dangerous, I know). I've never had a serious loss of data with it. I did have some problems with the early versions, but reiserfsck always solved them all. Make sure you have the latest version of it, btw.
--
<@chani> I *cannot* remember names. but I did memorize 214 digits of pi once.
[ Parent ]

That's stupid (5.00 / 3) (#141)
by krogoth on Sun Aug 10, 2003 at 05:53:58 PM EST

Software isn't supposed to have bugs, but it does. It doesn't even have to be bugs in the filesystem; what if I'm running in some other OS and it decides to mess with partitions it shouldn't be touching? It could be a bug in the user too; if I accidentally write a small ISO image to the wrong partition, I'd want to save as much data as possible.

Even with ext3 regular fscks occasionally find and fix an error, and having used reiserfs before it will be a long time before I can trust it to be that reliable.
--
"If you've never removed your pants and climbed into a tree to swear drunkenly at stuck-up rich kids, I highly recommend it."
:wq
[ Parent ]

That's why it does exist (5.00 / 2) (#144)
by vadim on Sun Aug 10, 2003 at 06:31:09 PM EST

To fix mistakes made by buggy versions.

My point was that fsck is pretty much needed by design with ext2. It's a filesystem that can't maintain consistency in case of a failure, so it needs fsck, just like FAT and FAT32 need scandisk or another program to get them back to a proper state.

However, in ReiserFS, NTFS and other journalling and logging filesystems fsck is more of an useful tool than a necessity, so it may not be so good. I've heard that the chkdsk tool in Win2K can be pretty stupid and break stuff instead of fixing it.

Besides, the structure of a logging or journalling filesystem is much more complicated that such simple things as FAT, which of course makes it much harder to fix.
--
<@chani> I *cannot* remember names. but I did memorize 214 digits of pi once.
[ Parent ]

Shit happens (Horror story) (none / 0) (#198)
by JAM on Mon Aug 11, 2003 at 10:38:29 AM EST

I have been using ReiserFS since the 2.2 kernel time (patched) without a problem. About two months ago, on a boot (with kernel 2.4.20 or 2.4.21, I don't remember), the kernel stopped booting and ReiserFS told me to run fsck.reiserfs to fix something, so that's what I did, I booted with my rescue CD, ran fsck.reiserfs --redebuilt-tree /dev/hda3 (as the message told me to do) and about at half the process it segfaulted, leaving me with a totally corrupted and unrecoverable (at least cheaply) filesystem (and with a backup two months old, but of course that's not ReiserFS fault...).

Now I'm using Ext3. I'm not a fs-troll or something like that, I was a happy user of ReiserFS until this happened, and I even wrote a short guide (in spanish) about installing reiserfs on a patched 2.2 kernel.

So, people saying that H.R. team should focus a little on having good recovery tools instead of adding more and more features is not, IMHO, wrong at all.
-- Sorry for my engRish (TM)
[ Parent ]
Backups, backups, backups (none / 0) (#214)
by vadim on Mon Aug 11, 2003 at 02:23:37 PM EST

Shit happens indeed. However, not every time that you can't mount reiserfs, or get reiserfs messages in the log it's actually what's not working well. There are other things that will cause filesystem corruption, including bad RAM, overheated CPUs, bad power supplies (voltages out of tolerance), bad hard drives, other buggy code in the kernel...

Ext3, XFS, or anything else won't help you much if your problem isn't in reiserfs. Also, repair tools aren't magical. Many times bad data can't be fixed, and all they can do is just throw it out.

BTW, when a programmer says a program is experimental it IS experimental. reiserfsck says to backup your data before trying --rebuild-tree, IIRC. If you don't follow that useful advice, well, you have to deal with the consequences.

I needed reiserfsck twice: Once due to a bug in an early 2.4 kernel. But early 2.4 kernels were quite unstable without reiserfs anyway. The second time was when a kernel crashed quite spectacularly on me. After I forgot about the idea of applying 5 experimental patches to it and just went with what was on kernel.org I stopped having problems.

In both cases, I remembered to dd the partition to another one, and reiserfsck that with the latest version of the tools, then dd back. If that didn't work I would have mailed the list and could have easily tried to repair the partition several times.
--
<@chani> I *cannot* remember names. but I did memorize 214 digits of pi once.
[ Parent ]

You are mistaken. (none / 0) (#209)
by komet on Mon Aug 11, 2003 at 01:22:29 PM EST

Journalling filesystems are not guaranteed to always have correct data. They merely guarantee consistent data in the face of system crashes, power outages or unexpected disconnects. That doesn't mean the hard drive can't develop defects in the middle of your directory, or that a script kiddie can't gain root and randomly write shit onto your hard drive.

And that's why all filesystems need to have check tools.
--
Any technology which is distinguishable from magic is not sufficiently advanced.
[ Parent ]

New hardware is far from perfect (none / 0) (#238)
by voblia on Tue Aug 12, 2003 at 06:19:02 AM EST

Why do you think you don't need a good fsck to fix hardware related bugs ?

I don't want to lose my data if my computer gets shocked when someone turns off the power main. Yes ofcourse losing some of my informations is inevitable in such case, but believe me I was angry when i lost 14 GB partition without any chances of restoring (no tool i could find, even bare eye examination using a text editor couldn't tell it was reiserFS, or some data) because of some hardware inconsistency that shows up only when my computer is heavily loaded (ReiserFS does this real good, copying lot's of files makes the faults vissible more often than compiling the kernel). I had no problems with Ext3 yet (one year allready), but i wouldn's put reiserFS into any of my new machines, unless i am 100% sure that the ram is 100% good, the processor is never overheated and etc.

[ Parent ]

Not in my experience. (5.00 / 1) (#197)
by LukeyBoy on Mon Aug 11, 2003 at 10:05:46 AM EST

I've got a massive ReiserFS3 partition running inside an LVM space, totalling nearly a terrabyte of data.  One day when I rebooted the system, DMA wasn't enabled for one of the Promise controller cards (I never got around to setting that up on boot) and I massively corrupted the drive, with scary messages about ReiserFS needing to "rebuild the tree completely".

So I read the resierfsck man page, and it of course tells me to backup the entire disk before rebuilding the tree... Which for me was a logistical impossibility.  So I said "Fuck it", and ran the command.  Four or five hours later (slow machine) the entire tree on the system was rebuilt with not a single file corrupted (which was verified with MD4 hashing).  True story.

[ Parent ]

I disagree... (none / 0) (#247)
by sbalea on Tue Aug 12, 2003 at 10:36:13 AM EST

I have been using ReiserFS3 for several years (2-3 at least). Never had a FS problem (although I had several disk crashes -- damn you Maxtor!).
Right now we use it at work on our desktop machines and partially on our development servers. In fact I have two servers in a mirror, one with ext3 and the other with Reiser (RH and Mandrake). They get synchronized via rsync. Even though ext3 sits on a better machine (2x1 Ghz CPUs, fast SCSI drives) than the reiser server (2x 850Mhz CPUs, IDE drive), the reiser server is much faster, and it's all due to the filesystem.
I'm not that worried about FS corruption, that's why we have a mirrored server configuration, and the important stuff is backed up regularly. In these circumstances, I think the speed advantages of Reiser are well worth it...

[ Parent ]
Alternative view... (none / 0) (#248)
by Alhazred on Tue Aug 12, 2003 at 10:36:48 AM EST

I've been running ONLY ReiserFS filesystems since MDK8.0 was released (I think it was Kernel 2.4.9 that was included in that).

I've never had a single problem of any kind whatsoever on more than 10 systems, even with a number of upgrades etc. I think it works great!

Personally I have the feeling that Hans Reiser comes off as being a bit of a 'Boy Wonder' and everyone feels like they have to put a pin in him, on top of which is the fact that everyone remembers only the horrible things that happen to them.

I mean come on guys, I can show you a dd of an ext3 filesystem that was TOTALLY trashed, and running fsck on it just made it worse... Does that mean ext3 is kaka? No, it just means that you can kill most any file system.

The whole issue with ReiserFS being 'too complex' is kind of a red herring as well. Lets face it guys, as systems become larger and hardware becomes more sophisticated and we demand more from our systems things HAVE to get more complex. Its only a matter of answering the question 'Will we put the complexity over here, or over there?' and the answer is 'Put it where it does the most good.' That IS the essence of everything Hans Reiser is talking about.

And yah, sure, EVERY filesystem can use a better 'fsck'... duh!
That is not dead which may eternal lie And with strange aeons death itself may die.
[ Parent ]

Database ? (5.00 / 1) (#130)
by bugmaster on Sun Aug 10, 2003 at 04:23:28 PM EST

I'm probably wrong about this, but doesn't the next Windows release (Longhorn, I think) support the notion of using a SQL database as a filesystem ? This approach seems like it will solve most of the problems mentioned in the article, since databases were made to manipulate metadata to begin with. Furthermore, databases are fairly well understood by now, and fairly well optimized, so the performance shouldn't be a huge issue.
>|<*:=
Reiser 4 is a database (5.00 / 1) (#136)
by lambda on Sun Aug 10, 2003 at 04:57:40 PM EST

For what it's worth, I have heard that Longhorn uses/will use a Microsoft Access database in addition to a NTFS filesystem. Reiser developers said why force people to change their habits from hierarchical to an SQL database; let them organize their data still as they want. If you want to do SQL queries, you will be able to with plugins.

Reiser 4 is a database (it is hierarchical, not relational). Reiser 4 is a fast database--LivingXML.com says Reiserfs outperforms all relational databases! Reiser 4's algorithms are fairly well understood by now, and fairly well optimized, and the filesystem was designed to manipulate small files (i.e., metadata) to begin with.

[ Parent ]
File system complexity, reliability (1.18 / 11) (#131)
by Rolley on Sun Aug 10, 2003 at 04:24:19 PM EST

I expect that pushing the handling metadata of hundreds of file formats into the filesystem would make the file system code

1. too complex
2. unreliable
3. eternally out of date

Stability, security and low overhead are the foremost desirables od a filesystem.

Another interesting idea would be to map an SQL database onto the file system:

/.../database/query or view/subquery

You could read and modify the data with your favourite file editor or include the file into an html document.

SQL is fairly standard so the project would not be chasing an everchanging goal.

 


--
Rolley
To the filesystem it isnt metadata (5.00 / 1) (#143)
by MfA on Sun Aug 10, 2003 at 06:06:19 PM EST

To it it is all just files, that is the beauty of it.

[ Parent ]
Metadata maintenance, complexity (4.00 / 2) (#156)
by baka_boy on Sun Aug 10, 2003 at 09:01:35 PM EST

The whole point of adding these features to Reiser4 is to reduce the complexity of file metadata code, not increase it. Right now, even at the OS level, (ignoring such common but esoteric examples as ID3 tags) much of the work in filesystem and security code is mapping internal data structures (i.e., file permissions or ACLs) into packed binary file headers or external indices.

By allowing any file to hold both an opaque stream of data, and any number of named sub-entries, you give developers a much easier system by which to associate metadata with its target. In addition, you offer an excellent method for packaging applications with associated libraries, config files, and resources such as icons: simply include them in the 'subdirectories' the binary file supports, and while the file will appear to be a single monolithic object to a user, (meaning it can be moved, copied, deleted, transferred to another user, etc.) the OS can treat it as an entire directory tree.

Of course, all of this comes with a similar caveat to the resource-fork woes that plagued MacOS users for so long: as soon as one of these structured files is moved from a system (or even disk or partition!) supporting Reiser4 to a "traditional" one, you must either strip all the attached resources away, or encode or package them somehow in a platform-independent way, potentially rendering the file unusable on the target system. Presumably, though, these issues will have to be mostly addressed well before widespread adoption of the Reiser4 model.

[ Parent ]

Doesn't have to be complex and bloated (4.00 / 1) (#262)
by mirkwood on Wed Aug 13, 2003 at 06:48:25 AM EST

I’m neither an expert for Reiser4 nor for databases, but I think the intent of Reiser4 and the whole metadata handling is something different. I assume Reiser4 won’t try to provide metadata structures for every file type (which would indeed make it complex and permanently out of date). It’s more like pulling the layer of metadata out of the applications, and putting it between the file system and the application.
Presumably this is similar to databases: you have the DBMS layer (mySQL, Postgre, whatever), the DB layer (your particular tables/relations structure) and the content (data populating your tables). So the idea is not that the mySQL maintainers include colossal libraries of tables so that every database admin can only choose from the presets, but to include nothing and provide a means for everyone to create their own database (=tables structure). I haven’t read much about Reiser4 besides this article, but I imagine that’s what they’re really trying to do.

[ Parent ]
What's your fucking problem, (2.66 / 6) (#269)
by butfuk on Wed Aug 13, 2003 at 08:51:48 PM EST

dude?



Unix and C are the ultimate computer viruses.
[ Parent ]
I would like to point out for the record another (3.00 / 1) (#145)
by danni on Sun Aug 10, 2003 at 06:55:21 PM EST

little known file system, ADFS where each file has a file type in the form of a 24 bit number stored in the filesystem entry for the file (the number can be associated with a textual name in the operating system for the benefit of the user), if you were to do something kooky with the file i.e. email it or put it on a web page, there is a MIME map module which simply maps the file types onto MIME types.

How will it affect code? (none / 0) (#152)
by vadim on Sun Aug 10, 2003 at 08:15:13 PM EST

Say, for writing a simple function that calculates the size of a directory with all its files and subdirectories.

So, in Perl we have something like this:

sub dir_size {
    my $directory = shift;

    opendir(DIR, $directory);
    while (my $entry = readdir(DIR)) {
        if ( -d $entry ) {
            $size += dir_size($entry);
        } elsif ( -f $entry ) {
            $size += -s $entry;
        }
    }
    closedir(DIR);
    return $size;
}

Depending on how we write this code we might have some problems. The way above would probably be correct. A version with two if's instead of an else would count space twice. A program that checked if it's a file first would never descend into the directory, which might not be good if we were going a recursive search looking for file names that are inside. A program checking if it's a directory first might be also bad. For example, it's more efficent to compress one large file than many small ones, unless you're using tar to combine them.
--
<@chani> I *cannot* remember names. but I did memorize 214 digits of pi once.

It won't. (5.00 / 2) (#178)
by it certainly is on Mon Aug 11, 2003 at 12:50:50 AM EST

On the disk, a file is a file and a directory is a directory. That's one of the inode attributes. However, you can use the user interface that the FS code provides you with. Reiser lets you attempt to open a file as a directory, and it will manufacture an on-the-fly directory listing containing metadata labels as files.

While unix appears to enforce strict UNIXy filesystem rules, the reality is that a FS can do anything it wants. For example, I could mount "webfs" on /www, and then access the file "/www/http://www.blah.com/fish/wibble.asp?hello=3&test=2" - the entire part after /www/ is passed to the webfs code, and it decides how to deal with it.

kur0shin.org -- it certainly is

Godwin's law [...] is impossible to violate except with an infinitely long thread that doesn't mention nazis.
[ Parent ]

Sir, your code is broken. (1.80 / 5) (#203)
by tkatchev on Mon Aug 11, 2003 at 12:24:36 PM EST

It is written in Perl.

   -- Signed, Lev Andropoff, cosmonaut.
[ Parent ]

Grrr (4.00 / 2) (#154)
by enderwiggin99 on Sun Aug 10, 2003 at 08:29:16 PM EST

Reading some of the replies posted below on how this or an SQL-based filesystem would reduce complexity and increase reliability, I shudder to think of that wonderful Registry in Windows and how similar Reiser4 seems. And we all know that basically whatever happens to Windows is made near-impossible to fix due to sheer complexity. Where is the equivalent THAT WORKS of backing up /etc and reinstalling, then copying back over just what you need? File-locks are a fucking nightmare. Then again, the registry is not really a logical extension of what it holds information on; a system like it but built directly as the filesystem ALA Reiser4 might be a godsend.

__Ender__
Reverse-engineering the Universe from life until Zen.
Registry vs. config files (none / 0) (#165)
by baka_boy on Sun Aug 10, 2003 at 10:03:46 PM EST

While I, like any good UNIX geek, absolutely abhor and fear the registry, it has a few good things going for it: ubiquity, standard data format, and remote accessibility.

Also, while I certainly like the fact that I can just replace the system configuration piece-by-piece using the files in the etc directory, I also know that I've wasted entire man-months dealing with the idiosyncracies of each different subsystem's config file format and placement. Even the BSDs, IMHO, do a better job, since most of the core configuration files are simply shell scripts that call command-line configuration tools.

Reiser4 could help the situation, but only if a distro abandons the file-per-daemon and format-per-file model for configuration, which has the unfortunate downside of breaking compatibility with just about every piece of UNIX software in existence, forcing them to patch or rewrite every piece of system software the kernel didn't provide.

[ Parent ]

Not necessarily (?) (none / 0) (#223)
by scruffyMark on Mon Aug 11, 2003 at 06:10:38 PM EST

If I understand the article right (and I'll admit I haven't read through the links or anything), a program that expects to find a single flat file could find one, even though the info could also be available as a bunch of subfiles in a directory, if you try to look at it that way.  This would be possible, similar to the /etc/passwd example.

e.g. I could
$ cat /etc/httpd/httpd.conf/PidFile
"/var/run/httpd.pid"
or
$ grep PidFile /etc/httpd/httpd.conf
PidFile "/var/run/httpd.pid"

Looking at it the first way is about equivalent to the second for finding info (provided you know exactly what you're looking for, including capitalization etc.), but easier for making changes. (cat > /etc/httpd/httpd.conf/PidFile)

[ Parent ]

EXACTLY! (none / 0) (#246)
by Alhazred on Tue Aug 12, 2003 at 10:02:21 AM EST

This is the entire point! Reiser4's way of doing things allows the OLD semantics and the NEW semantics to coexist as peacefully as possible. Thats why the example of /etc/password. You would still be able to edit it with 'vipw' just like you do now and yet you would be able to apply seperate permissions to each record in the 'file' because of aggregation.
That is not dead which may eternal lie And with strange aeons death itself may die.
[ Parent ]
Windows Registry (none / 0) (#174)
by lambda on Mon Aug 11, 2003 at 12:02:51 AM EST

...I shudder to think of that wonderful Registry in Windows and how similar Reiser4 seems.

Reiser4 is not stored as a database in two large, hidden files. You can use cp, ls, rm, and friends on its contents, unlike Windows registry. Any filesystem can be made as unorganized as the Windows registry. The Windows registry just happens to be horribly organized.

[ Parent ]
Windows Registry (none / 0) (#275)
by dknj on Sat Aug 16, 2003 at 12:51:41 PM EST

You can use cp ls rm if you wanted on the Windows Registry. Microsoft details all of the registry function calls in their sdk. All that has to be done is write a set of tools or a psuedo filesystem that gives you access to the Windows Registry in the manner you describe.
-dk
[ Parent ]
Retard. (1.00 / 3) (#202)
by tkatchev on Mon Aug 11, 2003 at 12:23:23 PM EST

Right, if it reminds you of Windows, then it must be bad?

Never mind the fact that /etc is horribly crufty, outdated, incompatible with every single version of every single distribution out there, impossibly unintuitive to use, filled with brain-dead application-imposed idiocy and frequently broken?


   -- Signed, Lev Andropoff, cosmonaut.
[ Parent ]

I don't believe this (2.00 / 4) (#161)
by Pinkerton FIoyd on Sun Aug 10, 2003 at 09:50:38 PM EST

REally, I don't.
Remember when you were young? You shone like the sun.
Nice. (2.00 / 5) (#171)
by Pinkerton Floyd on Sun Aug 10, 2003 at 10:56:18 PM EST

I wonder how many will be fooled by this imposter? You just raised the price for me to actually leave to $1000.

Remember when you were young? You shone like the sun.
[ Parent ]

Cruft (4.00 / 1) (#170)
by solarsail on Sun Aug 10, 2003 at 10:51:11 PM EST

While I believe meta-data is a good thing overall, I am concerned about its abuse by applications. If we are talking about standard meta-data for a file type, then it is a good thing. If, on the other hand, we are talking about meta-data on a per-application basis, then we are talking more about a major headache. Just imagine every mp3 program with its own meta-data. Application designers typically feel that they want to do things the "right way" instead of the way other have done before them, so we will see a lot of duplicated meta-data that is only accessible to the application that wrote it. Just think about address books in Linux. Short of setting up a LDAP server for your own local use, there is no way to get any 2 programs to read each other's address books (finding two applications that can talk to an LDAP server is another story). So the end result? I can't be bothered to take the time to enter any contact info into my computer. The other annoying aspect about this type of application-specific meta-data is that it is likely to be strewn about associated with random files. I don't want the KDE or Gnome file browser associating various odd info like desktop position along with the file. How do I clean this cruft up when I am tired of experimenting with Gnome? I can't just rm -r ~/.gnome* The article worries about meta-data staying in synch with the file's existence, but what about with the application that created it? And how do multiple users store conflicting meta-data? The other problem is that applications that take advantage of these features will have also implement the lowest common denominator as well. It seems faced with this reality, most applications will not support the new features in a more than superficial way. Plugins don't add too much if everyone defaults to the lowest common denominator anyway, it just adds another layer of shuffling to the process. I don't really understand the obsession with being like the Mac either. Unix users understand the utility of data in and of itself. All this feuding about creator codes are irrelevant. We are more interested in text-based formats that are useful to multiple programs. If you read the http://www.namesys.com you will see that the motivation for all of this is to increase the interaction between programs. Windows and Mac users live in a world where writing a letter means MS Word or looking at a webpage means Internet Explorer. Unix users are more advanced than that and see a world where we work with various types of data. I know many think attracting the less intelligent members of our population is a worthy goal. We shouldn't sink to their level, we should raise them to ours.

I think you might be a little extreme there. (none / 0) (#188)
by kesuari on Mon Aug 11, 2003 at 05:31:46 AM EST

I think it's nonsense to suggest that, when provided with the option to (a) use the title EA to encode the title or (b) create a new EA called oogabooga to encode the title, most people who write software I want to use will go for option (a).

All this feuding about creator codes are irrelevant. We are more interested in text-based formats that are useful to multiple programs. ... Unix users are more advanced than that and see a world where we work with various types of data.

Here, you're entirely missing the advantage. I have some HTML files on my hard drive from my own website, and some that are other people's documentation and the like. I want to open my website files in gvim to edit them, and I want to open others' in Galeon to view them. This seems reasonable enough. Creator codes, or, better, file-based associations (rather than type-based ones) give me the power to easily do this. Same with Python scripts: mine I probably want to edit (I'll be running them in a console while debugging to get information about what I've stuffed up because I'm still learning); others I probably want to run. Under the current regime, without metadata for each file telling how I want to open the file, I have to use right-click menus and use the Open as Text option for my website.

Both files are the same type, but I'm doing different things with them. In a file-centric desktop, it only makes sense to do it this way. I can't remember the last time I used an open dialog box that I wouldn't need to use if we had creator codes. (And though drag-and-drop mostly solves that, that means I have to already have the application running or feel like navigating through my folders while dragging.)

And anyway, no-one said you had to use the metadata. If it makes it easier for me and doesn't change it for you, why does it matter if I choose what you view as the cop-out option?

[ Parent ]

But... (none / 0) (#227)
by beergut on Mon Aug 11, 2003 at 06:30:27 PM EST

What happens when you want to peruse the source code of someone else's Python program, or to run your own? What kind of hoops do you have to jump through when you have creator codes?

Inverting the situation isn't the answer. Creator codes are a stupid hack.

i don't see any nanorobots or jet engines or laser holography or orbiting death satellites.
i just see some orangutan throwing code-feces at a computer screen.

-- indubitable
[ Parent ]

So what do you suggest? (none / 0) (#245)
by Alhazred on Tue Aug 12, 2003 at 09:57:09 AM EST

Its all well and good to call something a 'stupid hack' and maybe it is, but lets hear a better idea!
That is not dead which may eternal lie And with strange aeons death itself may die.
[ Parent ]
Sadly... (none / 0) (#267)
by beergut on Wed Aug 13, 2003 at 05:16:24 PM EST

I don't (yet) have a better idea.

I don't know what is a good idea, but I can spot a dumb one. ;-)

i don't see any nanorobots or jet engines or laser holography or orbiting death satellites.
i just see some orangutan throwing code-feces at a computer screen.

-- indubitable
[ Parent ]

3 Syllables... Namespaces (none / 0) (#236)
by SuperSheep on Tue Aug 12, 2003 at 02:18:08 AM EST

Really the idea is that people the meta data that already exists in all these files will move into the filesystem.  So if they could agree with MP3's (more or less), hopefully they can agree here.

SuperSheep

[ Parent ]

Big deal. (2.00 / 2) (#186)
by Mr Hogan on Mon Aug 11, 2003 at 04:29:33 AM EST

The Amiga had all this 15 years ago.

--
Life is food and rape, then tilt.

Welcome to our universe (4.00 / 1) (#199)
by Grimmtooth on Mon Aug 11, 2003 at 11:31:10 AM EST

Excuse me, but Amiga had *none* of this 15 years ago, nor does it now in OS 3.9 using FFS 40 or 41 or whatever it is up to these days.

Amiga FFS had file comments for meta data, no different than MSDOS did at the time. Amiga dirs and files behave EXACTLY the way described above for ext3 et al. Amgia FFS was subject to file size versus file block size issues just the same as everybody else (in fact I spent quite some time fine tuning the block sizes of my BBS machine to get peak performance).

Oh ... this was a joke, right? You got me. I fell for it. :-)
// Worst. Comment. Ever.
[ Parent ]

The Amiga did it all, with style and panache! (5.00 / 1) (#210)
by porkchop_d_clown on Mon Aug 11, 2003 at 01:31:15 PM EST

Well, I think it did. I'm not real sure what "panache" is. :-P

But, I have to agree with you. The Amiga file systems were pretty SOP for that time. What I find interesting is that Apple's finally given up on metadata just in time for other people to start pushing it....


--
His men will follow him anywhere, but only out of morbid curiousity.


[ Parent ]
Technology is stagnant (3.25 / 4) (#190)
by SwampGas on Mon Aug 11, 2003 at 06:38:25 AM EST

...not because of lack of invention or implementation.  Lack of reason.

Why are we still using PCI?  IDE?  Or basically ANYTHING that old?  Why is it that the 8086, 8088, 286, 386, 486, Pentium, et al caused people to FLOCK to the next step up?  One minute 30 meg MFM is all craze, then suddenly I have a 486DX2/66 with 540 meg IDE drive...and the AMAZING SB16...with 8, yes EIGHT, meg of EDO ram.  Wow...EDO?  8 meg?  Awesome.  Wait.  Pentium?  GOTTA HAVE IT!  No!  AMD is better!  512 meg of DDR?  Firewire!  YES!

But when we look at the time line, we're using the same old junk for years.  i386 architecture, PCI, IDE, etc.  Back during the technology wave of the 80s and early 90s it was insane.

Why?  It's not strictly geeks anymore.  

When the AMAZING i386 came out with Windows 3, it was HUGE.  EVERYONE in the community went CRAZY and dropped $3000 on that amazing new machine.

When the next big thing to hit computers (WinXP) came out with machines that have Firewire and USB 2.0 compliant ports no one cared.  The general public was the target, not the geeks.  The general public doesn't care.

Likewise, they don't care about performance enhancements and benchmarks.  All they care about is getting their email, IMs and burning CDs.  That's why we're still using FAT32, NTFS, PCI, IDE, broken ass Windows, etc.  The geeks are nobodies now...outcasts again amongst the millions and millions of clueless sheeple who stifle invention and setting new standards.

Why bother setting new standards and raising the bar when the people who will spend all the cash don't care?  They have no reason.

I'm sure I'm not the only one who misses the innovation and technology rush of the past.

Incremental tech has a huge advantage (5.00 / 1) (#208)
by grout on Mon Aug 11, 2003 at 01:21:31 PM EST

It's not a computer phenomenon, it's true of all technology. Why the hell do we have translation technology instead of a common language? Because of the installed base of people speaking natural languages, of course.
--
Chip Salzenberg, Free-Floating Agent of Chaos

[ Parent ]
Except most everything you listed... (none / 0) (#235)
by SuperSheep on Tue Aug 12, 2003 at 02:14:10 AM EST

is either out or on it's way out.

Fat32 hasn't been the primary file system since 2000 (WinME) and isn't supported in WinXP.  NTFS in it's original incarnation died with NT4.  Windows 2000 replaced it with something called NTFS 2000 which was renamed to NTFS by the marketing people.  PCI is going to be replaced with PCI Express and was replaced by AGP for graphics cards.  Parallel IDE is on it's way out.  It's being replaced by Serial IDE.  Times are changing.  Sometimes people refuse to notice, sometimes the marketing people fool us into thinking it's the same thing and sometimes these things just sneak up on us.

SuperSheep

I try to keep at least one sig per voice in my head.  It's been murder trying to keep up.

[ Parent ]

Um...what? (none / 0) (#279)
by Control Group on Wed Aug 20, 2003 at 04:02:35 PM EST

Fat32...isn't supported in WinXP.

Er...excuse me? It isn't? I'm fairly certain that I'm running XP on one of my home machines, and I'm equally certain that it's sitting on one HDD, split amongst four FAT32 partitions.

I suppose it's possible my system is actually running some other FS, and just telling me it's FAT32, but I've got no reason to think that's the case.

I'm also curious as to what the primary FS was for WinME—that's an incarnation of Windows that I never touched (and, come to think of it, it might be the only incarnation of Windows I've never touched...and I do include Windows 1.0. I'm not sure if I should be geekily proud, or horribly depressed to realize this), and I suppose I just assumed it was a FAT system. Did that use NTFS, too?

***
"Oh, nothing. It just looks like a simple Kung-Fu Swedish Rastafarian Helldemon."
[ Parent ]

Directories are a terrible concept (5.00 / 2) (#194)
by synaesthesia on Mon Aug 11, 2003 at 08:41:18 AM EST

By trying to emulate paper files and folders, we have exposed ourselves to their limitations.

I'd like a filesystem which is conceptually flat, but always viewed through powerful filters.

Why do I use a '~/Music' directory? All I want is a filter which says "Show me all files which are music".


Sausages or cheese?

bingo (none / 0) (#241)
by speek on Tue Aug 12, 2003 at 09:06:08 AM EST

Directory names are just arbitrary search criteria.

--
al queda is kicking themsleves for not knowing about the levees
[ Parent ]

And how would you manage your system.... (none / 0) (#244)
by Alhazred on Tue Aug 12, 2003 at 09:47:58 AM EST

Remember, there are many issues besides just how you want to search the filesystem! Basically if the filesystem was fast enough then 'find / -name '*.mp3' would be fine for your purposes...

Now what about the sysadmin who needs to restrict the FTP server to not be able to download an uploaded file before someone approves it? Either you need a hierarchical namespace which is agnostic to file metadata or you would need some other system like arbitrary file attributes.

Actually if you read Hans Reiser's various musings on the whole subject I think you will get the point that the hierarchical structure of the ReiserFS V4 filesystem is just a transitional step. He does talk about the semantic layer gaining advanced query capabilities and he's even given it the label 'ReiserFS V6'.
That is not dead which may eternal lie And with strange aeons death itself may die.
[ Parent ]

Like so (none / 0) (#249)
by synaesthesia on Tue Aug 12, 2003 at 10:40:21 AM EST

Remember, there are many issues besides just how you want to search the filesystem! Basically if the filesystem was fast enough then 'find / -name '*.mp3' would be fine for your purposes...

Quite.

Now what about the sysadmin who needs to restrict the FTP server to not be able to download an uploaded file before someone approves it? Either you need a hierarchical namespace which is agnostic to file metadata or you would need some other system like arbitrary file attributes.

Arbitrary file attributes is exactly what I want, so that I can mark those files 'Approved'.

Actually if you read Hans Reiser's various musings on the whole subject I think you will get the point that the hierarchical structure of the ReiserFS V4 filesystem is just a transitional step. He does talk about the semantic layer gaining advanced query capabilities and he's even given it the label 'ReiserFS V6'.

Excellent, I shall put those on my reading list!


Sausages or cheese?
[ Parent ]

I still have this feeling... (none / 0) (#253)
by Alhazred on Tue Aug 12, 2003 at 11:16:43 AM EST

That a 'default' hierarchical file system organization is NOT going away anytime soon. Not only is it ingrained in all existing software, its also ingrained in the whole way we approach organizing our systems.

For example, how do you know which partition a file is on? Well, you simply know that everything under the mount point /usr is a partition. Granted that could show up as metadata, but I'm going to get real tired of typing

tar cfzv backup.tgz 'all files where partion=hda3'

instead of

tar cfzv backup.tgz /usr

real fast...

I think both are useful. Containers certainly seem to be something humans understand intuitively.
That is not dead which may eternal lie And with strange aeons death itself may die.
[ Parent ]

I'm sure you're right... (4.00 / 1) (#255)
by synaesthesia on Tue Aug 12, 2003 at 12:23:15 PM EST

...that a 'default' hierarchical file system organization is NOT going away anytime soon.

However, it should be possible to make the transition relatively smooth, even with something as simple as a "legacy directory path" attribute.

I also agree that it's ingrained in the way we approach organising our systems, but hopefully getting less and less so (for instance, have you ever used iTunes?)

Your example about mount points is a little one-sided, because /usr essentially reads a shortcut from /etc/fstab so that you don't have to type

tar cfzv backup.tgz '/dev/hda3,ext2,rw,1,1'

every time. You would have named filters to achieve much more powerful results:

tar cfzv backup.tgz @"important AND recent"

or whatever, and replace concepts like "current working directory" with "current working filter", which is again potentially much more powerful.

Sausages or cheese?
[ Parent ]

I'm not saying your wrong (none / 0) (#277)
by Alhazred on Wed Aug 20, 2003 at 11:32:31 AM EST

What I'm saying is that inertia is a HUGE factor in IT. Cobol is still being used heavily and its been obsolete for 40 YEARS.

Maybe 40 years from now people will be about done with file system hierarchy as their primary way of dealing with organizing storage.

There are other issues as well, such as levels of indirection.
That is not dead which may eternal lie And with strange aeons death itself may die.
[ Parent ]

I take your point, but. (none / 0) (#280)
by synaesthesia on Thu Aug 21, 2003 at 08:20:59 AM EST

I'm sure that the notion of a heirarchical filesystem is deeply-enough ingrained into existing code that programs would still be using a compatibility layer for many years to come. (BTW are you really trying to claim that COBOL was obsolete by 1963?)

But forty years for the primary filesystem organizational method to move away from heirarchical? That's like saying that most new programs are being written in COBOL. Not so.

Levels of indirection would be no more of an issue for the compatibility layer than they are for any current filesystem. Some piece of code somewhere has to interpret "../..", either way.


Sausages or cheese?
[ Parent ]

Cobol? LOL (none / 0) (#284)
by Alhazred on Sun Sep 07, 2003 at 03:48:13 PM EST

Cobol was obsolete BEFORE 1963, I'm being generous to it...
That is not dead which may eternal lie And with strange aeons death itself may die.
[ Parent ]
It's old and clunky (none / 0) (#285)
by Cro Magnon on Tue Sep 09, 2003 at 09:28:00 AM EST

But it STILL does what it's meant to do. I sure wouldn't write an OS in COBOL, but I wouldn't write a large accounting program in C either.
Information wants to be beer.
[ Parent ]
They're working on it (none / 0) (#259)
by seraph93 on Tue Aug 12, 2003 at 09:43:00 PM EST

I'd like a filesystem which is conceptually flat, but always viewed through powerful filters.

Then you might be interested in Microsoft's new FS, which will be based on a relational database instead of the traditional hierarchical format. Just google "longhorn fs" or something to that effect, I'm too lazy to do it and post links. It seems like a pretty neat idea, but I doubt it'll be running on Linux anytime soon.

Why do I use a '~/Music' directory? All I want is a filter which says "Show me all files which are music".

Not only could you do that, you could have filters like "show me all music files that are hip hop from 1983-1985" * or "show me all image files that feature redheads involved in bondage and/or watersports" **. That would be pretty damn sweet, IMO.

* Disclaimer: This is just a random example that I made up on the spot. I don't particularly enjoy 80s hip hop, in case you were wondering.
** Further disclaimer: This one's just another random example, I swear!
--
Ph-nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn.
[ Parent ]
How does the metadata get in there? (none / 0) (#261)
by simon farnz on Wed Aug 13, 2003 at 05:37:21 AM EST

My experience of systems with rich metadata is that someone has to be paid to enter all the data about a file; given that most people I've encountered can't be bothered with sane filenames, how many are going to enter serious, consistent descriptions?
--
If guns are outlawed, only outlaws have guns
[ Parent ]
The metadata? How does the *data* get in there? (none / 0) (#265)
by seraph93 on Wed Aug 13, 2003 at 04:07:51 PM EST

Somebody's going to be doing some typing in any case. I was thinking more of personal uses than business ones, but in a business environment, you get serious, consistent descriptions the same way you get sane filenames: Implement a sane and consistent procedure for file creation. Define a limited set of concise metadata fields which are allowable. Document everything. Train people.

Even better: Don't let the data entry folks mess with the metadata directly. Give them a few checkboxes in whatever they're using to enter the data. Then the computer's entering all the metadata and there's less room for error.

If doing all this sounds like a pain in the ass, that's only because it is. But that's just life in Bidness Land. You can take your time to make sure everyone's on the same page, or you can deal with the chaos that results from everyone doing whatever they please. While personally I'm a fan of chaos, not too many investors are.
--
Ph-nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn.
[ Parent ]
I was thinking purely of home users (none / 0) (#273)
by simon farnz on Thu Aug 14, 2003 at 06:19:56 AM EST

I can see that business users are different; if there's a policy on metadata, you follow it or risk your job. It's just that in my experience of home users and filesharers, very few people even bother with sane and sensible filenames, and those who do don't bother about current metadata.

In addition, many people have atrocious spelling; when it comes to shared files (P2P etc), many files are obviously misspelt, and some of the worst offenders even "correct" filenames to match their own personal misspellings. If they do the same with metadata, then it's useless to me until I've cleaned it up.

I guess what I'm saying is that the ability to run database queries on your filesystem ("All music by the Counting Crows lasting between 3 and 6 minutes, and not released before 1999") sounds good, until you start thinking about the amount of effort needed to get that information there to query. How many people even bother now to organise their video collection or CD collection into a carefully indexed order? There are some, but most just rely on their ability to find it on the shelf again later.
--
If guns are outlawed, only outlaws have guns
[ Parent ]

Aaah, the home users (5.00 / 1) (#276)
by seraph93 on Sun Aug 17, 2003 at 05:15:51 PM EST

Oh, I see. It *would* suck trying to run queries on Joe Sixpack's computer, but my computer would be a different story entirely. I'd be willing to take the time to enter the metadata. It's just a few seconds per new file, really, and you get much more than that few seconds back once you run those queries on your filesystem.

I suppose it's just a matter of how organised you are in general. A lot of people don't organise their CD collections, but mine's in alphabetical order, first by artist then by album name. My mp3s and ogg files are all organised: ~/tune/<artist name>/<album name>/<song title>.ogg

It'll be really sweet to get a new level of organisation to play with, IMO. And generating playlists on the spot with nothing more than grep will be oh so neat. I don't think the average person will get as much out of it, but I'd rather that Joe Sixpack kept his hands off my computer anyway.
--
Ph-nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn.
[ Parent ]
Look at CDDB (none / 0) (#266)
by Merc on Wed Aug 13, 2003 at 04:21:04 PM EST

If there's a way to enter metadata about something in a shared way so it benefits lots of people, you'll generally find someone willing to do it once or twice, and in the end everybody benefits.

Besides, if someone *really* cares about their porn, (not mentioning any names of course), they may well categorize it and load it with metadata.



[ Parent ]
in Linux (none / 0) (#268)
by uhoreg on Wed Aug 13, 2003 at 07:20:43 PM EST

It seems like a pretty neat idea, but I doubt it'll be running on Linux anytime soon.

It's actually going to be what Reiser6 will do. Reiser6 is actually more of a semi-structured system, because the over-structured filing that you get from a relational database is pretty bad for casual data storage. You should read Reiser's "Future Vision" paper, which is actually linked from the article, under a bad description (the "its design goals").

[ Parent ]

WinFS (none / 0) (#274)
by synaesthesia on Thu Aug 14, 2003 at 11:30:08 AM EST

As described here, WinFS will be built on top of NTFS. Does this mean I won't be able to write a file's metadata if someone else is reading it? ;)

Sausages or cheese?
[ Parent ]
Show me all files which... (none / 0) (#283)
by jasonditz on Tue Sep 02, 2003 at 08:42:23 PM EST

That's basically how the Zaurus file manager does things (at least how they did things before 3.0), and believe me, its an unbelievable drag on resources.

Why force the system to keep all your stuff sorted all the time when its just easier to do it yourself?


[ Parent ]

hmm (none / 0) (#204)
by Samiti on Mon Aug 11, 2003 at 12:36:13 PM EST

You mention, as an example, ID3 tags on mp3, and how they would be no longer necessary under such a file system. However, as mp3s are traditionally cross playform, creating a miniture level of organization in the file is necessary. If you swapped these traits over to Reiser4, then people useing MacOS or Windows would not get any id information.

You may have to rewrite some programs to make use of these attributes on Linux, but what happens if you want to move a file cross platform or to a partition that doesn't support these tags? Not usually a problem because ID tags aren't really a NECESSARY part of a file (you don't NEED to know what song that mp3 is).

But if, say, email was trated in this manner, and you lose the attribute that is the header to attribute tags unique to a file system, then you have a problem.



#/bin/bash script.kiddies

RTFA NT (2.50 / 2) (#218)
by synaesthesia on Mon Aug 11, 2003 at 05:12:31 PM EST



Sausages or cheese?
[ Parent ]
incorrect (none / 0) (#237)
by dumky on Tue Aug 12, 2003 at 02:34:02 AM EST

Even though Reiser allows you to access these attributes as files in a sub-directory, they are still stored as part of the file. Which means you can send the file to a friend using another platform without his mp3 player to be broken.

Reiser just offers a more uniform way to access these attributes. And since there is a wide variety of file formats with their own attributes, Reiser needs plugins to understand them. For ex, you might have a ID3v2 plugin for Reiser.

[ Parent ]

Not quite so (none / 0) (#286)
by typo on Mon Oct 20, 2003 at 06:23:53 PM EST

Even though Reiser allows you to access these attributes as files in a sub-directory, they are still stored as part of the file.

I think this is not quite so.

Say you have example.mp3 and example.mp3/author. Reiser4 will only store example.mp3/author inside example.mp3 if the author file is actually a view of the id3 tags inside the file generated by a filesystem plugin.

The idea with reiser4 is that you store metadata for a file, not with special atributes but with other files that are "inside" the file only because they're accessed by using the file as a directory. So to store the metadata for a mp3 file you wouldn't use id3 at all. Instead simple files "inside" example.mp3 would describe each of the atributes. To copy this around you'd need to copy all of the files. Between reiser4 filesystems the regular copy should work, but to work with other systems you'd need some other way to store this stuff.

Note that I've never used reiser4, this is just my understanding from what I've read



[ Parent ]
Isn't this the sort of thing BeOS did? (5.00 / 1) (#211)
by porkchop_d_clown on Mon Aug 11, 2003 at 01:32:38 PM EST

Or was it NeXT? I remember someone boasting how you could write an e-mail database engine using shell scripts and the file system...


--
His men will follow him anywhere, but only out of morbid curiousity.


BeMail (5.00 / 1) (#217)
by frankwork on Mon Aug 11, 2003 at 05:08:43 PM EST

BeOS's mail system bascially dumped all the email downloaded from the POP server into one directory.

It then poked into the files to see the subject, from, to, etc., and flagged the message as new.

You could write "canned queries" like all new mail from bob, and save it as a sort of "virtual folder" on your desktop that always contained new mail from bob. Pretty slick.

[ Parent ]

More BeMail (none / 0) (#230)
by ewhac on Mon Aug 11, 2003 at 07:52:46 PM EST

What's more, the queries and virtual folders were all managed within the standard Tracker desktop. No special application needed. The Tracker could be instructed to display and sort on any metadata field. So your email index was, in fact, a Tracker window.

And yes, it was fairly slick.

Schwab
---
Editor, A1-AAA AmeriCaptions. Priest, Internet Oracle.
[ Parent ]

My problem with Reiser (none / 0) (#213)
by Mr.Surly on Mon Aug 11, 2003 at 01:54:43 PM EST

Is that some rather incredible claims are made. For instance, if you're compiling your kernel with ReiserFS, and you get compile errors, then evidently you have bad hardware, but it's normal not to get errors when compiling other code with the bad hardware.

FAQ Notice how it says "compiling".
Q: Why do I get a signal 11 when compiling the kernel using ReiserFS and not ext2?

A: Your CPU is overheating or you have bad RAM.
FAQ:
Q: I am having extensive problems using ReiserFS; it seems to have bugs all over the place. I'm not compiling with a buggy compiler. What is happening? How can this be stable?

A: You have hardware problems. Really, you do. Even if the bugs don't show up with ext2, you have hardware problems. (See FAQ question about ReiserFS running 3C hotter than ext2.) Most SuSE users use ReiserFS. Obscure bugs probably still exist; but if you find bugs as easily as using Windows, you have bad RAM, bad CPU, bad cable, bad cooling, VIA chipset with PCI quirks turned on, or other hardware or other software layer bugs. ReiserFS is stable. You can be sure that if the bugs are encountered easily and commonly with normal usage patterns, it is not us. This does not mean that the next release won't somehow break something though :-/..... Real bug reports are at the time of writing outnumbered 10 to 1 by hardware bugs that trigger error messages. We are working on making our error messages better at catching hardware bugs and identifying them as such. There is only so far we can go though in runtime consistency checking without serious speed reductions. We don't release software unless it goes through extensive testing; so if you don't think that our testing could have missed the bug, it is probably hardware.


So is your problem with ReiserFS or the FAQ? (none / 0) (#219)
by Anonymous American on Mon Aug 11, 2003 at 05:43:07 PM EST

Lots of open source developers get surly when they get a request for support, especially when they are not able to reproduce the so called problem.  

If you don't like the faq pay 20$ to open a support incident.

[ Parent ]

Read it again ... (none / 0) (#221)
by Mr.Surly on Mon Aug 11, 2003 at 05:58:03 PM EST

It's not about the FAQ, Reiser itself, or support. I've use Reiser, works fine, compiles fine.

The Reiser people are actually claiming that if you cannot compile Reiser that you have a hardware problem, even if Reiser is the only part of the kernel that won't compile.

I just find it incredulous that anyone would claim that their source code can expose hardware flaws during it's compilation.

[ Parent ]
Uhh... (4.66 / 3) (#224)
by beergut on Mon Aug 11, 2003 at 06:21:44 PM EST

I'll assume that English is not your native language.

What that question/answer is asking/answering is, "Why is it that when I compile the kernel (or something else large, with lots of files) with its source located on a ReiserFS filesystem, I get SIGSEGV and other errors?"

This isn't about compiling the ReiserFS source code itself, but about the increased CPU utilization of managing the ReiseerFS vs. EXT2. More CPU cycles for a given operation means more heat, and more heat can push normally-fine-but-in-reality-iffy hardware over the edge.

i don't see any nanorobots or jet engines or laser holography or orbiting death satellites.
i just see some orangutan throwing code-feces at a computer screen.

-- indubitable
[ Parent ]

Ambiguous, but you're right. (none / 0) (#234)
by Mr.Surly on Tue Aug 12, 2003 at 01:41:22 AM EST

Okay, when I read this, it totally seemed like it meant compiling Reiser into the kernel, not compiling the kernel source located on reiserfs. I've had this in my mind for over a year after reading the FAQ, and it read the same after re-reading recently, but after reading your post a few times, and quite smug in my correctness, I noticed my error. At which point, I re-wrote this post =)

Here:
Q: Why do I get a signal 11 when compiling the kernel using ReiserFS and not ext2?

A: Your CPU is overheating or you have bad RAM.


[ Parent ]
How is that ambiguous !? (4.50 / 2) (#251)
by MythMoth on Tue Aug 12, 2003 at 10:47:58 AM EST

"Why do I get a signal 11 when compiling the kernel using ReiserFS and not ext2?"

This would be ambiguous:
"Why do I get a signal 11 when compiling the kernel with ReiserFS and not ext2?"

This would be what you thought the original meant:
"Why do I get a signal 11 when compiling ReiserFS into the kernel and not ext2?"

But it's not ambiguous, you just read it wrong.

[ Parent ]

Sounds right to me... (5.00 / 3) (#229)
by seraph93 on Mon Aug 11, 2003 at 06:55:48 PM EST

...if you're compiling your kernel with ReiserFS, and you get compile errors, then evidently you have bad hardware, but it's normal not to get errors when compiling other code with the bad hardware.

Actually, compiling a kernel is one of the best ways to test if your heatsink is up to snuff or not. A typical kernel compile fills up your memory with pointers to everything, and keeps your CPU working at 99-100% for a good long time. If the CPU or the RAM overheats and misplaces a 1 or a 0 somewhere, it's quite likely to leave one of these pointers pointing to something outside of gcc's memory space, and then what do you get? That's right, kids, signal 11, segmentation violation. Other code is typically not as complicated as the Linux kernel and usually doesn't take as long to compile, which is why signal 11 is seen less often outside the realm of kernel compiles.

The reason that ReiserFS brings out these kind of errors more readily than, say, ext2, is because Reiser is more CPU-intensive. The more numbers your CPU crunches, the hotter it gets. The hotter it gets, the more it fucks up.

They're not lying to you. ReiserFS really is stable. It really is a problem with your hardware.
--
Ph-nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn.
[ Parent ]
But NTFS has alternate data streams! (1.00 / 2) (#215)
by gr00vey on Mon Aug 11, 2003 at 04:50:57 PM EST

Why would anyone need another filesystem? /sarcasm....

XML directory aggregator (none / 0) (#228)
by frankwork on Mon Aug 11, 2003 at 06:31:35 PM EST

This would solve a lot of the interoperability issues, assuming XML becomes and stays as much of a standard as everyone says.

XPath already exists as a template to do the mapping, and file names could be translated to elements, with attributes getting an '@' prefix.

Yeah, its wanted badly (none / 0) (#243)
by Alhazred on Tue Aug 12, 2003 at 09:34:36 AM EST


That is not dead which may eternal lie And with strange aeons death itself may die.
[ Parent ]
You don't really need a d (3.50 / 2) (#233)
by Pig Hogger on Mon Aug 11, 2003 at 11:31:30 PM EST

Keeping in mind everything you've read about the capabilities of Reiser4, the ability to efficiently manage small files, if /etc/passwd were to be split up into multiple files, one for each user, each file could have permissions set that would match the user. A shadow file would no longer be necessary as the user's password hash does not need to be protected from the user; They already presumably know their password.

But this, of course, breaks older programs that read and edit the password file. It is for this reason (among others) that Reiser4 supports filesystem plugins. One such plugin is a directory aggregator. Whereas before it was mentioned that a file could also be a directory, a directory can appear as a file. In this case, the directory would appear as an aggregation of files separated by a delimiting character (such as '\n'). Taking this example to its logical conclusion, each field could be a separate file. So to find a user's preferred shell (in this example, user ID is 107), one would access the file /etc/passwd/107/shell. Changing the shell would be a simple matter of:

<pre>echo '/bin/bash' > /etc/passwd/107/shell </pre>

Since user ID 107 owns the /etc/passwd/107/ directory and all of its descendants, a setuid program/script isn't necessary. Seeing possibilities yet?

You don't need a new filesystem to do that.

A mere patch to the file parser portion of the OS could do. A special character (let's say, for now, "±") within the filename would mark where the data branches-off, à la subdirectory.

When you refer to the filename up to the special character, you'll get back everything below. One could presumably encode different separator strings for each level to separate records; those could be a simple file attribute that is set with the describing filename.

To take your passwd example, let's say our special caracter is ± :

your directory enties would be in /etc/passwd. However, it would be subdivided several ways:

(file) = contents

passwd±100±login = root
passwd±100±passwd = @³²¤@¼³£²¢¤¼@³&s up2;
passwd±100±uid = 100
passwd±100±gid = 100
passwd±100±home = /root
passwd±100±shell = /bin/bash
passwd±100±fullname = root
passwd±200±login = cneal
passwd±200±passwd = @³²¤@¼³£²¢¤¼@³&s up2;
passwd±200±uid = 200
passwd±200±gid = 200
passwd±200±home = /root
passwd±200±shell = /bin/bash
passwd±200±fullname = Cowboy Neal

The separation string attributes ("\n" for the first level, and ":" for the second level) could be set this way:

chsep passwd -d "\n"
chsep passwd±* -d ":"

What could be a default separator is left as an exercise to the reader. Heck, one could also specify inside the separator string the "file" name itself, to allow for named fields.

So, whenever you do à "cat /etc/passwd", you'll get back the whole file:

root:@³²¤@¼³£²¢¤¼@&su p3;²:100:100:/root:/bin/bash:root
cneal:@³²¤@¼³£²¢¤¼@&s up3;²:200:200:/root:/bin/bash:Cowboy Neal

Likewise, if you ask for "/etc/passwd±200", you'd get only the line for user 200:

cneal:@³²¤@¼³£²¢¤¼@&s up3;²:200:200:/root:/bin/bash:Cowboy Neal

This neat hack could be made to work on any existing filesystem without to redefine the whole thing.

And, with a little imagination, the possibilities are endless; XML files could be decoded easily into such a scheme and parsed/modified effortlessly.


--

Somewhere in Texas, a village is missing it's idiot

Thats what ReiserFS V4 IS!!! (none / 0) (#242)
by Alhazred on Tue Aug 12, 2003 at 09:31:54 AM EST

Its just a filesystem with a plugin architecture so you can do exactly what you're talking about, except your concept of seperator characters is flawed in the sense that your namespace lacks orthogonality and that you would have to parse the file INSIDE the filesystem layer.

If you take your proposal at total face value you're going to have THE OS PARSING FILES???!!! I don't think parsing file structures belongs at the OS level AT ALL! The end result of that is going to be a nightmare.

Maybe a special plug-in that understands XML and disagregates that MIGHT be useful, but at that point is there really a distinction between your approach and Reiser's? Not from a user perspective I suspect.
That is not dead which may eternal lie And with strange aeons death itself may die.
[ Parent ]

Yes, that would be possible (none / 0) (#256)
by ttfkam on Tue Aug 12, 2003 at 02:02:33 PM EST

And of course you would run into the cited problems that (a) it would be slow on many filesystems and (b) tons of 50 byte files would be grossly inefficient in space and/or time on most filesystems.

This is why few have done anything like this.  This is why Reiser4 seems significant to me (and others).  This is possible, transparent to standard programming techniques, and now fast enough to be practical.

If I'm made in God's image then God needs to lay off the corn chips and onion dip. Get some exercise, God! - Tatarigami
[ Parent ]

reiser is very good (4.33 / 3) (#260)
by Nomak on Wed Aug 13, 2003 at 12:54:11 AM EST

At factory we use reiser mandrel and press for stamping aluminum and steel. Before using Mitsubishi stamping and not suiting heavy gauge needed for larger application. Also russain-make mandrel but no parts available for spare and problems. If you send SMS or ICQ I have information of representative for Pakistan or local to you.

Reiser4 as new platform (none / 0) (#264)
by pbreton on Wed Aug 13, 2003 at 12:06:37 PM EST

Reiserfs attempts to change the role of filesystems; if it is successful (ie, it both works and becomes widely used), then it will essentially become a new platform.

You'll know that this is happening if applications which only work on reiserfs (because the cost of development is lower) become common. Other signs are a distro which requires reiserfs, and reiserfs clones.



Why a New Filesystem Matters | 286 comments (260 topical, 26 editorial, 1 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!