Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
A-Patching we will go...

By jd in Op-Ed
Wed May 23, 2001 at 03:17:41 PM EST
Tags: Software (all tags)
Software

One of the things that's always seriously bugged me about most of the "Open Source"/"Free Software" operating systems out there is that extensions are supplied as "diffs"...


To those not familar with programming, a "diff" is a file containing a series of "differences" between some original version of a program and the extended version.

Now, this system works extremely well, when the number of people supplying "diffs" is small. Unfortunately, it totally breaks-down when this isn't the case.

The reason for this is simple. When supplying a "diff" file, you assume that the original is -the- original. Unfortunately, everyone else makes the same assumption.

Result -- lots of very useful extensions, none of which will work together.

Some examples might help to illustrate this. Here are some "diffs" for the latest version of Linux:

  • MOSIX - an extension which automagically spreads the workload across a network of computers. You don't have to do anything to achieve this, it's all done for you.
  • SE-Linux - an extension which protects -everything-. Nobody - not even the system administrator - can overstep their authority.
  • Alan Cox' patches - a whole host of bug-fixes and extensions. Almost essential.
  • Usagi IPv6 - a replacement IPv6 stack for Linux. A definite must-have, if you want to use IPv6.
  • Real-Time Scheduler - an alternative task scheduler for Linux, for better handling of heavy loads.
  • Global File System - fast, robust networked filing system.
  • FreeS/WAN - Secure Internet Communication.
You want to have secure distributed computing? Or the most bug-fixed kernel possible? Or do real-time work, using a network file system? Or maybe just have a good, robust system that handles encrypted networking?

You do?

Well, isn't that nice. Well you can't!

Because all of these extremely handy bits of code are designed to plug directly into the kernel source code, rather than run as pluggable modules, they clash with each other horribly.

As a result, single-tasked, single-minded geeks can function beautifully. So long as they never desire to be anything but single.

Working together is, IMHO, one of the greatest potentials "Open Source" can provide. But where is it???

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
I use a...
o ...vanilla 2.2.x kernel 15%
o ...vanilla 2.4.x kernel 43%
o ...strawberry and apricot 3.2.1 kernel 3%
o ...kernel with 1-3 extensions/patches 13%
o ...kernel with 4-6 extensions/patches 3%
o ...kernel with none of the original code left in it 1%
o ...closed-source kernel, and have no idea what's in it 12%
o ...Tektronix terminal, connected to my digital toaster-oven 6%

Votes: 115
Results | Other Polls

Related Links
o MOSIX
o SE-Linux
o Alan Cox' patches
o Usagi IPv6
o Real-Time Scheduler
o Global File System
o FreeS/WAN
o Also by jd


Display: Sort:
A-Patching we will go... | 25 comments (24 topical, 1 editorial, 0 hidden)
patch-only licenses (4.14 / 7) (#2)
by Delirium on Tue May 22, 2001 at 05:04:05 PM EST

And that is exactly why licenses that only allow modifications as patches to the original source (like the original QT license and Dan Bernstein's qmail/djbdns license) are a Bad Thing.

I thought I was alone... (3.00 / 2) (#3)
by Rift on Tue May 22, 2001 at 05:04:38 PM EST

Being not too linux savvy, I just figured that there was some magic switch to patch, or a usefull tool that I didn't know about to fix this very problem.

I have a SMP motherboard with a built in UDMA66 IDE controller that was not supported by a stock linux kernel. I search and search until !bingo!, I find a 'driver' for it. However, this driver is a patch to the kernel. after making sure I get the patch that matches the kernel I use, I find another problem - the IDE tape drive I use is also not supported by the kernel. However, after just a few google searches, a 'driver' emerges. However, when applying this patch, I get the dreaded rejects.

Now, I can choose - the secondary controller on my mobo, or the IDE tape drive I picked up cheap. I'm sure that a person willing to go look at the reject files could have sucessfully merged in both patches, but why make it that difficult?

That's all behind me now that I'm on 2.4, but when will the next patch incompatibility bite me? I can't really complain too much - I'm not helping by coding a smart patch, but I do know that this is yet another issue to be resolved before linux (and lots of other large OSS projects) is ready for prime time.

--Rift
A pen is to a car what a meteor is to a _____
would your mobo by any chance be a BP6? (3.75 / 4) (#4)
by mikael_j on Tue May 22, 2001 at 05:10:34 PM EST

If so, just switch to freebsd, afaik there is support for the hpt366 controller in freebsd

/Mikael Jacobson (OT)
We give a bad name to the internet in general. - Rusty
[ Parent ]
You get a cookie! (2.66 / 3) (#5)
by Rift on Tue May 22, 2001 at 05:19:59 PM EST

It is a BP6, in fact! And, I do dual boot with FreeBSD and am very glad to have the support there. But, since I can now use it with 2.4, the problem is not so urgent.

But very good ID!

--Rift
A pen is to a car what a meteor is to a _____
[ Parent ]
why? because its good for ya (none / 0) (#21)
by h2odragon on Wed May 23, 2001 at 04:01:23 PM EST

Go look at the reject files in your sample case, are there any that are actually a problem? Possibly not. If there are, go ahead and have a stab at fixing it, however ignorant you might be. Account it as time spent learning new stuff and you can't lose; you might even make it work in the end.

I don't write C, but I've only found one or two occasions where I couldn't beat together multiple patches to an official kernel when I wanted to. I haven't tried something like the example presented in this article, which I think is extreme in any case. Both Mosix and SE-Linux are hacking some of the same code to acheive different goals; likewise the two network patches. If you find a version of diff that can reconcile project intentions like that, well, I want a copy too.

[ Parent ]

Once again, I say... (4.37 / 8) (#6)
by trhurler on Tue May 22, 2001 at 05:24:52 PM EST

Interesting, if true. But not.

You see, many of those extensions cannot be kernel modules, short of extensively rewriting the base code to support them, which just means more patches that will clash. This results in multiple changes to the same code. There is no method of distributing multiple independent mods to the same piece of code that will allow them to merge neatly without the intervention of a programmer. Your argument is not with diff, but rather with reality.

The solution, to the extent that it is a solution, is to get all the code that practically can be incorporated into the main tree in there, and to get multiple machines for other stuff. For instance, MOSIX really has no place in an ordinary system anyway, and if this other IPv6 implementation is so much better, then get them to put it in the main kernel sources.

Personally, I don't have this problem, because development of my OS of choice is managed in a sane manner by people of professional caliber, instead of anarchistically thrown together and glued up by the heroic efforts of a few heroic men with names like Torvalds:)

--
And when you consider that Siggy is second only to trhurler as far as posters whose name at the top of a comment fill me with forboding, that's sayin
Exactly (4.66 / 3) (#7)
by rusty on Tue May 22, 2001 at 06:02:12 PM EST

There are two situations under discussion here. The first is for things that radically alter the system behavior, and which are only needed by a very few people, like SELinux. These will likely always be patches, against the current stable source.

The second situation is patches that implement early or experimental support for a feature that may or may not eventually be in the official kernel. If you want or need to try these out, it's best if you're not able to apply lots of other experimental patches, because then when something goes wrong, how will you know what caused it? Basically, if you want to run experimental stuff, you are a tester, and shouldn't expect it to be easy or convenient. Right now, for example, I'm trying to get Gabor Kuti's software-suspend patch to work, because APM appears to be badly broken on my picturebook. No, it's not easy or convenient, and I have to actually do some hacking and jury-rigging to even get the patch to apply. But if it works, it'll have been worth it. :-)

Finally, we have Linus' well known attitude toward kernel hacking. That is, that is should be hard, because if it's not, you'll get a kernel developed by people who can't handle the hard stuff, and thus a worse kernel. I have to agree with him there.

All that aside, I also would like to point out that the ac-* patches are not necessarily "bugfixes" or "almost essential". I can't even get 2.4.4 to compile with ac-12 applied. The ac series is considered development, and may or may not work. Unless I'm wrong, in which case someone with more kernel experience please say so and I'll shut up. :-)

____
Not the real rusty
[ Parent ]

Fred Brooks All Over Again (4.00 / 1) (#15)
by lower triangular on Wed May 23, 2001 at 07:09:23 AM EST

heh. You're right there ...

This results in multiple changes to the same code. There is no method of distributing multiple independent mods to the same piece of code that will allow them to merge neatly without the intervention of a programmer. Your argument is not with diff, but rather with reality

Exactly ... or to put it another way, the number of potential conflicts and communication problems increases as the square of the number of programmers, which leads us to Brooks' Law, and having a coordination mechanism "at least as good as the Internet" doesn't make much difference.

Personally, I don't have this problem, because development of my OS of choice is managed in a sane manner by people of professional caliber, instead of anarchistically thrown together and glued up by the heroic efforts of a few heroic men with names like Torvalds:)

Or to put it another way, you don't have this problem because your OS of choice has a pace of development an order of magnitude slower due to the personality problems of a few megalomaniac oddballs with names like de Raadt .... :) .... which is actually really saying the same thing in a different way.

Linux - the ultimate Windows Service Pack!
Windows - the ultimate Linux Productivity Suite!

[ Parent ]

Heh (4.50 / 2) (#19)
by trhurler on Wed May 23, 2001 at 11:55:36 AM EST

Or to put it another way, you don't have this problem because your OS of choice has a pace of development an order of magnitude slower due to the personality problems of a few megalomaniac oddballs with names like de Raadt .... :) .... which is actually really saying the same thing in a different way.
Bah. I've had beers with the man. He's a great guy. No tolerance for fools whatsoever, but a great guy. As for the pace of development, OpenBSD already has serious IPv6 committed. It had stateful firewalling years before the Linuxers even started talking about such a thing. It doesn't need crap like nonexecutable stack patches, because of the heroic efforts of a few megalomaniac oddballs with names like de Raadt. It has one of the best ipsec implementations you can buy for love or money. The things that are actually being worked on are accomplished with a greater degree of quality and in less time than Linux ever manages anything, but in fairness, OpenBSD is not an OS for boneheads who think enlightenment is the best thing ever to happen to unix, because the target audience is not point and drool fools.

--
And when you consider that Siggy is second only to trhurler as far as posters whose name at the top of a comment fill me with forboding, that's sayin
[ Parent ]
Forks are not patches (4.33 / 6) (#8)
by Carnage4Life on Tue May 22, 2001 at 06:58:58 PM EST

I don't know about the others but SE Linux is not a mere patch (i.e. quick bugfix) but is a sizable restructuring of the internal workings of the Linux kernel. It would be impossible to make SE Linux a module for Linux even if Linux was a microkernel if it didn't have some built in support for capabilities and priviledges. I am sure the other patches you mention are of similar types.

That said, it is true that a more sophisticated means of providing extensions to Open Source software is desirable besides diffs would be desirable but I am unsure as to the best way to tackle this. I do know however that expecting that all patches should be kernel modules does not consider the entire picture.

Functional Replacement? (4.80 / 5) (#9)
by delmoi on Tue May 22, 2001 at 07:03:32 PM EST

Well, one solution would be to try to isolate changes by function, or even nested statement rather then by file/line (as I assume diff works)

That way you'll only have to wory about code that clashes semantically rather then code that clashes syntactically.

Doing this, you'd also have to worry about data dependency problems (IE, a global variable that's used by more then one mod)

Anyway, I don't think that there's really any theoretical reason why we would need to be exclusionary about these patches (unless they are patches that would naturally conflict, like two different process distribution patches).
--
"'argumentation' is not a word, idiot." -- thelizman
*cough* Microkernel *cough* (4.80 / 5) (#10)
by Pseudonym on Tue May 22, 2001 at 11:49:24 PM EST

Someone had to bring it up, so it might as well be me. This is yet one more reason why I'd rather be running a microkernel-based OS. (No, for the fifteenth time, I'm not talking about dinosaurs like Mach or NT. I'm talking about modern microkernels like BeOS, QNX, ChorusOS or L4/Fiasco.)

If you had a kernel with minimal functionality (only that which was required for system security) and functionally complete, stable internal APIs, we would not be in patch hell now. There would be no need to patch the kernel except in case of bug fixes, and new functionality would only be a new server away. Most of the time you wouldn't even need to stop the kernel running to do it.



sub f{($f)=@_;print"$f(q{$f});";}f(q{sub f{($f)=@_;print"$f(q{$f});";}f});
oh get reeeallll (3.75 / 4) (#17)
by lower triangular on Wed May 23, 2001 at 09:22:45 AM EST

... man ... hey I'd love to visit this place where conflicts only happen in kernel space (they don't) and where system resources are so plentiful that everything can have its own process (they aren't) and where microkernels solve all our problems (they don't). But I seem to be stuck on this little blue planet, third from the sun ...

and what yr describing isn't really a microkernel anyway ... it's a small monolithic kernel with a lot of (hypotheticomagically) well designed and stable drivers and APIs ... each running as a separate process, natch. A true microkernel approach would handle internal kernel tasks with a single, threaded process, with the conflicts auto-hypothetico-magically sorted out and it would be truly wonderful

... the basic problem is that theoretical, in-principle microkernel designs are always superb, but all actually existing microkernels suck and this isn't a coincidence ... until the progress of hardware outstrips the demand for features, there's never going to be enough spare capacity to handle microkernel scheduling, and that doesn't look like happening soon ... or at all.

Linux - the ultimate Windows Service Pack!
Windows - the ultimate Linux Productivity Suite!

[ Parent ]

Monolithic argument? (5.00 / 2) (#24)
by Pseudonym on Thu May 24, 2001 at 04:05:14 AM EST

I'd love to visit this place where conflicts only happen in kernel space (they don't)
You're right. They don't. However, a conflict outside the kernel is a bit easier to fix. A clean modular design is easier to patch because patches are not as likely to interfere.

and where system resources are so plentiful that everything can have its own process (they aren't)
When you used the word "process" there you meant "Unix process". Unix processes are indeed quite heavyweight. Microkernel servers are different beasts from Unix processes.

and where microkernels solve all our problems (they don't).
I never said that. I do believe that they would solve a lot of problems that we have nowadays, though, and one of them is "patch hell".

but all actually existing microkernels suck and this isn't a coincidence
"They all suck" is about the most dismissive and evasive argument I can think of. It is true that about ten years ago, pretty much all microkernels sucked, except for research purposes.

I named a number of microkernels which I believe suck (Mach and Windows NT), and a number which I believe don't suck (BeOS, QNX, ChorusOS and L4/Fiasco). There are a number of others that I didn't name which, IMO, had the opportunity not to suck, but for various reasons missed that opportunity (e.g. Plan 9, Windows CE). Perhaps rather than using a monolithic argument, you'd like to get a little more fine-grained and explain why the kernels which I said didn't suck do in fact suck?



sub f{($f)=@_;print"$f(q{$f});";}f(q{sub f{($f)=@_;print"$f(q{$f});";}f});
[ Parent ]
bugfix modules (4.00 / 2) (#11)
by Pink Daisy on Wed May 23, 2001 at 01:15:16 AM EST

I guess this is pretty obvious, but it's hard to implement a bug fix as a module.

This coming from a guy who runs a few versions old vanilla 2.4 kernel. Trust me, I don't care too much about the latest bugfixes and hardware. I did track closely for a while, though, as support for USB devices trickled in.

*Read* and *think*, dammit. (3.50 / 2) (#13)
by Estanislao Martínez on Wed May 23, 2001 at 03:27:40 AM EST

I guess this is pretty obvious, but it's hard to implement a bug fix as a module.

What I'm about to say *is* obvious: the examples the author gave were for the most part *extensions*, not bug fixes.

--em
[ Parent ]

You're right, but the author did mention it. (5.00 / 1) (#16)
by Pink Daisy on Wed May 23, 2001 at 09:09:55 AM EST

From the article:

You want to have secure distributed computing? Or the most bug-fixed kernel possible? Or do real-time work, using a network file system? Or maybe just have a good, robust system that handles encrypted networking?

While you're right that most of the stuff was for extensions, the author mentioned both bug fixes and robustness. I'm not going to quibble with the parts of the article that make sense to me. (=

[ Parent ]

The point of a diff.. (4.33 / 3) (#12)
by enterfornone on Wed May 23, 2001 at 01:31:17 AM EST

The point of a diff is that it is human readable. If you have lots of them then you just merge the changes in by hand.

Yeah modules are nice, but sometimes you really have to change the core. In that case the diffs let people see what has been changed. The fact that you can run the diff through patch to automate the change is just a secondary feature.

--
efn 26/m/syd
Will sponsor new accounts for porn.
Human readable (4.00 / 2) (#22)
by ghjm on Wed May 23, 2001 at 04:46:05 PM EST

Only for a suitable definition of "human." You and I can read diffs and patch kernels. Most people can't, even among people who work professionally as sysadmins. Yes, it's possible to put in sufficient effort to reconcile multiple patches. But that doesn't make it an ideal distribution method.


[ Parent ]
ideal distribution method (3.50 / 2) (#23)
by enterfornone on Thu May 24, 2001 at 01:53:01 AM EST

The ideal distribution method for most people is deb or rpm. Most diffs are there for developers to look at, not for end users. Certainly the ac patches aren't essential, they are bleeding edge dev stuff that Linus won't even put into unstable kernels.

--
efn 26/m/syd
Will sponsor new accounts for porn.
[ Parent ]
Correcting a Slight Error (4.60 / 5) (#14)
by moshez on Wed May 23, 2001 at 04:22:24 AM EST

Patch and the "good diff" formats (context and unified) are actually pretty smart about patching with a diff from the non-original. The correct way to think of diff is as a second-tier (in the first of the Linux kernel, the first-tier) source control mechanism. Someone changes their copy of the code, and instead of "cvs commit" (which would try to resolve conflicts) he mails it to a developer (or to the development list), and the developer then patches *his* copy of the tree. Most of the patchers are trying to get their modifications included in the main tree. Putting it in a loadable module where it plain does not make sense would hurt that. One good example: the MOSIX guys, who are actually working a lot with Linus on integrating the MOSIX stuff. MOSIX needs a *lot* of stuff Linux does not have, so they add it. Eventually, I guess it will be folded into the tree. Do note that usually, unless the patches touch areas close by, then patching with two different diffs actually works flawlessly -- justl like "cvs update" on a modified tree works. When it does not, it is usually not very hard to reconcile the differences. If it is, then usually the problem is an inherent difficulty, not one caused by the patch/diff methodology.

[T]he k5 troll HOWTO has been updated ... This update is dedicated to moshez, and other bitter anti-trolls.
Some furher thoughts from the loony bin (4.50 / 2) (#18)
by jd on Wed May 23, 2001 at 11:32:27 AM EST

The argument that you can't really use modules to handle bug-fixes is very valid. However, correctly-written code is self-contained, and bug-fixes should not extend out of the scope containing the bug.

(Ok, well, duh, yeah, if the code was correctly-written, there wouldn't BE any damn bugs. Out-of-scope effects should still be rare to non-existant, though.)

The worst area for clashes are lists. The list of directories/modules in the primary makefile, for example. arch/i386/kernel/entry.S is another culprit. But why have static lists, in the first place?

If there was some means for each module to register itself in all appropriate files, at configure time, you'd end up with MUCH smaller, simpler files, no surplus entries, and NO BLOODY REJ'S!!!

Modifying the core code is slightly tricker. What, exactly, IS the core code? The scheduler? Not according to RED, which allows you to load and unload schedulers at will. The processor code, surely! But CPU hot-swapping is all the rage. To work, this has to be outside of the "core", too.

In the end, the only conclusion you can really reach is that the module loader/unloader IS the core. The rest is all replacable, and very lkely is, in some patch or other.

If everything - absolutely everything - were defined as modules, it wouldn't matter if you were using MOSIX, SE-Linux, RT-Linux, or some combination of all three. Each affects a different element of the kernel, and therefore all should be independently pluggable, without generating side-effects.

Java-OS (3.50 / 2) (#20)
by weirdling on Wed May 23, 2001 at 02:02:53 PM EST

I'm beginning to like the idea of a JVM running on top of a lightweight Linux kernel. There are some people working on these. If properly designed, the kernel could become very static, and fixes could be simple .class changes. Of course, this won't deal with two people changing the same class for different uses, but the only solution to that is ownership.

I'm not doing this again; last time no one believed it.
That's not really what you want, though... (4.00 / 1) (#25)
by Pseudonym on Fri May 25, 2001 at 12:19:17 AM EST

If you're really writing a Java-OS, you really want to write as much of it in Java as possible. Wouldn't you like to have plug-in modules run in a protected Java sandbox, for example?



sub f{($f)=@_;print"$f(q{$f});";}f(q{sub f{($f)=@_;print"$f(q{$f});";}f});
[ Parent ]
A-Patching we will go... | 25 comments (24 topical, 1 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest © 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!