Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Don't Make Make Make That Makefile

By ramses0 in News
Sun Mar 05, 2000 at 09:00:47 AM EST
Tags: Software (all tags)
Software

Peter Miller wrote an interesting paper titled Recursive Make Considered Harmful. Anyone who has compiled a large software project has probably seen recursive "make"s, but Miller states that this is a sub-optimal method.


This is good reading for anyone who works in software. The points and methods he suggests are clear and lucid. Having never worked on a large project involving makefiles, I can't judge what he says, but his explanations (don't recurse, include, building nothing takes 10 seconds for 1000 source files, etc...) seem reasonable.

Some here develop software, what do you think?

Oh, and apologies for the story title... it's been a long week ;^)=

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Related Links
o Peter Miller
o Recursive Make Considered Harmful
o Also by ramses0


Display: Sort:
Don't Make Make Make That Makefile | 20 comments (20 topical, editorial, 0 hidden)
Ok, I'm totally down with this. I *... (none / 0) (#1)
by rusty on Sat Mar 04, 2000 at 11:07:24 PM EST

rusty voted 1 on this story.

Ok, I'm totally down with this. I *hate* the stages of a make when make is just rolling through all the directories, even though all it's going to do is copy a couple files. 'make install' is usually a stage like this. And for large source trees, he's right, it takes forever to do nothing much. There's gotta be a better way...

____
Not the real rusty

It's quite old, but worthwhile (and... (1.00 / 1) (#3)
by fvw on Sat Mar 04, 2000 at 11:56:22 PM EST

fvw voted 1 on this story.

It's quite old, but worthwhile (and true) nonetheless.

... (1.00 / 1) (#7)
by Velian on Sun Mar 05, 2000 at 12:25:54 AM EST

Velian voted -1 on this story.



might be interesting to some (I thi... (1.00 / 1) (#2)
by rongen on Sun Mar 05, 2000 at 01:26:03 AM EST

rongen voted 1 on this story.

might be interesting to some (I think it's cool, anyway)
read/write http://www.prosebush.com

Interesting article, but I can't pa... (1.00 / 1) (#5)
by kmself on Sun Mar 05, 2000 at 02:14:48 AM EST

kmself voted 1 on this story.

Interesting article, but I can't pass judgement on it myself. Are there any make mavens who can argue the pros and cons of what's being suggested here?

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.

Interesting article, but it's old.... (1.00 / 1) (#4)
by Wil Mahan on Sun Mar 05, 2000 at 02:55:15 AM EST

Wil Mahan voted -1 on this story.

Interesting article, but it's old.

Definitely useful. More articles o... (3.00 / 1) (#6)
by fluffy grue on Sun Mar 05, 2000 at 03:08:15 AM EST

fluffy grue voted 1 on this story.

Definitely useful. More articles on programming would definitely be nice to see, as well.
--
"Is not a quine" is not a quine.
I have a master's degree in science!

[ Hug Your Trikuare ]

Re: Definitely useful. More articles o... (none / 0) (#9)
by henrik on Sun Mar 05, 2000 at 09:51:18 AM EST

Yes. More programming. more more more! :)
How about book reviews? I'd be happy to write a couple.. :)


Akademiska Intresseklubben antecknar!
[ Parent ]
Book reviews (none / 0) (#10)
by rusty on Sun Mar 05, 2000 at 01:41:53 PM EST

Please do! I love reading good book reviews.

____
Not the real rusty
[ Parent ]
Re: Book reviews (none / 0) (#14)
by henrik on Sun Mar 05, 2000 at 03:33:56 PM EST

oh no - he took me up on it. Now i'm going to have to do something.. :) Alright - stay tuned. I've got this week off school, so i should have time to write something up. :)

BTW, no promise of the reviews beeing good. *grin*

Akademiska Intresseklubben antecknar!
[ Parent ]

Re: Definitely useful. More articles o... (none / 0) (#15)
by joeyo on Sun Mar 05, 2000 at 04:23:54 PM EST

I've got some book reviews in the works.. I have to finish the books first though... :)

--
"Give me enough variables to work with, and I can probably do away with the notion of human free will." -- demi
[ Parent ]

Re: Don't Make Make Make That Makefile (3.00 / 1) (#8)
by henrik on Sun Mar 05, 2000 at 09:28:09 AM EST

Another way of doing this would be to split modules into separate shared libraries, and only have a small executable loading the .so's and calling everything from there. In that case, a developer would only need to build his shared lib, and could ignore everything else. He wouldn't even need to recompile the application to see the new changes.

It wouldn't reduce the time to build the whole package from scratch (but how often do you do that?), but it would make the compile-edit-link cycle when developing a lot faster.

Of couse, this not all projects can be divided like this.

-henrik

Akademiska Intresseklubben antecknar!
Re: Don't Make Make Make That Makefile (none / 0) (#11)
by johnmeacham on Sun Mar 05, 2000 at 02:53:43 PM EST

But that does not really help with the original problem, which was that the code must be re-compiled whenever the INTERFACE changes... note that shared libraries dont make it any easier than building a bunch of .a static libraries in different directories, you just pass different flags on the command line. in any case this still only works if your interfaces to the shared libraries are set in stone to begin with, the whole point of the recursive make was that a single top level call to make would rebuild anything necisarry, shared library, static or just plain object files dont make a differene... the fundamental problem of an incomplete DAG (directed graph) over multiple instantiations of the make program still exist... (and this doesnt solve mutually recursive dependencies between directories)

now, there actually are relatively clean (compared to what is being done now) ways to solve this problem, the trick i like best is to keep a file i call module.make in each directory which only lists the files in the current directory and the subdirectories off of the current directory which have their own module.make files... then the global top level make recursivly 'includes' each subdir's moudle.make and so-forth... that way you have a complete DAG, it works with standard make and each directory only has to keep track of local information.... and the difference is amazing i have had projects that took a minute or two to figure out they needed to do nothing suddenly just zap! done! and never having to second guess make with spurious make clean's is a huge bonus....

[ Parent ]

Re: Don't Make Make Make That Makefile (none / 0) (#12)
by rusty on Sun Mar 05, 2000 at 03:06:14 PM EST

then the global top level make recursivly 'includes' each subdir's moudle.make and so-forth

Which is basically what the paper recommends you do, right? I.e. use include to make one great big Makefile out of the smaller ones. So, all the performance, without any maintaining of the One Great Big Makefile.

My question is, the paper was written in 1997. Why the hell aren't more people *doing* this yet? AFAIK, the linux kernel uses recursive make. Has anyone tried to create a global kernel makefile, and see how much time it saves?

____
Not the real rusty
[ Parent ]

Re: Don't Make Make Make That Makefile (none / 0) (#19)
by CodeWright on Thu Mar 09, 2000 at 10:44:28 AM EST

In the past, after running into some of the dependency problems described by the author as inherent to "recursive" makes, I long ago started using a single "monolithic" make for each given project.

Most of the time, though, I have found that my various and sundry co-workers over the years have been unwilling to understand even _simple_ makefiles, much less one which addresses the various dependencies and targets of a large project.

In short, to address the rhetorical question:
"My question is, the paper was written in 1997. Why the hell aren't more people *doing* this yet?"
my answer would be: "because they don'tcare"

--
A: Because it destroys the flow of conversation.
Q: Why is top posting dumb? --clover_kicker

[ Parent ]
Re: Don't Make Make Make That Makefile (none / 0) (#20)
by rusty on Thu Mar 09, 2000 at 11:03:43 AM EST

I suspected that might be the answer. That's a shame. I think if I worked with a compiled language, and builds took 15 minutes to change *one file* I'd probably start caring in a hurry. Of course, maybe I'm just impatient. :-)

BTW-- I recognize your .sig. I always like that one. :-)

____
Not the real rusty
[ Parent ]

Re: Don't Make Make Make That Makefile (none / 0) (#13)
by henrik on Sun Mar 05, 2000 at 03:26:48 PM EST

But that does not really help with the original problem, which was that the code must be re-compiled whenever the INTERFACE changes... note that shared libraries dont make it any easier than building a bunch of .a static libraries in different directories, you just pass different flags on the command line.

That's true. But how often do you change the interface of major app? Once every 100th build? If you can save the time with the normal compile-edit link cycle it's worth a lot more than the special cases where you change the interface or build from scratch.

Also there's one major advantage of shared libraries. You dont have to relink the executable, just make your lib and restart the app. Links can be very memory/time consuming for a large project.

Hmm.. but on the other hand, recursivly including module.make's seems like a great idea - and it is a lot easier than defining and implementing shared libs for everything. :)
-henrik

Akademiska Intresseklubben antecknar!
[ Parent ]

Re: Don't Make Make Make That Makefile (4.00 / 1) (#16)
by mihalis on Sun Mar 05, 2000 at 05:07:37 PM EST

I think this article was very good and although it may be old it's new to me.

I work on some graphics components for a large financial information system. Luckily for me the architecture of my components is a sort of homegrown COM where each is built as a shared object (Windows DLL or Solaris .so) and dynamically loaded at run-time. Each has a standard interface which is very small with richer functionality being "discovered" at run-time by a query mechanism. This makes life so simply on the make front that for about 10 or 11 months I got by on hand-written makefiles with a rule for each file (yes, no suffix rules even!).

The guys who make the main program that handles the network traffic and a lot of the local business logic processing have something like the traditional recursive make described in the article. I assumed that was the way to go and so I started writing makefiles more like theirs and I have to say I rapidly ran into the problems the author describes. I'm now convinced to go for the single project makefile instead. Thanks guys!

Chris Morgan
-- Chris Morgan <see em at mihalis dot net>

Re: Don't Make Make Make That Makefile (none / 0) (#17)
by faniz on Tue Mar 07, 2000 at 05:46:39 PM EST

First, the headline of this article is plain wrong: it's not at all about remaking makefiles with make! It's about recursively entering directories.

Besides that, great article! I wish I would have read it a few weeks ago, because I just had to rewrite the makefiles for a large project with many subdirs - and I'm just finished. :-(

Needless to say that I used recursive make invocations. It "only" takes 14 seconds if everything is already built, i. e. 14 seconds to check the dependencies. But with only one Makefile and many module includes it would certainly be faster, and without all that annoying "entering/leaving directory" output. Another annoyance is that a parallel make (GNU make option -j) doesn't work right now. This would take more effort to fix the dependencies between the directories.

And I have the problem that in some directories shared and static libraries are build, with include files which are needed in other directrories where those libraries are linked together. If you change something in the libraries (or includes), the executables won't be linked again at the moment. I didn't think about that one-makefile solution because I hadn't seen it before and didn't know it was possible with reasonable effort.

But still there is one point which makes me wonder: if different executables or libraries or objects need different linker or compiler flags, what is the best way to handle that? This is not mentioned in the article and I can't think of an easy solution at the moment.

At least there was one think I did "right": the dependency checking and inclusion. But I took that from the GNU make documentation. :-)

Ad for something completely offtopic: what is this "rate" thing next to the comments about? Is "5" a good rate or "1"? I didn't find this in the FAQ. That FAQ even tells me that there is no "moderation or karma". So what is a rate then?

- Stephan.
--
Carpe diem!
Re: Don't Make Make Make That Makefile (none / 0) (#18)
by rusty on Tue Mar 07, 2000 at 07:04:46 PM EST

Err... the FAQ should be ignored in cases where it conflicts with observed reality. :-)

There are a couple of things that need to be added to the FAQ. Check back tomorrow. It'll be in there. And thanks for pointing it out.

____
Not the real rusty
[ Parent ]

Don't Make Make Make That Makefile | 20 comments (20 topical, 0 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!