Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

Documenting the Beast

By semis in Technology
Wed Oct 04, 2000 at 02:50:02 AM EST
Tags: Help! (Ask Kuro5hin) (all tags)
Help! (Ask Kuro5hin)

Recently I had the [mis?]fortune of rebuilding a server. Not just a simple mail server, but the whole works -- apache, netatalk, ftp, samba, local user accounts, postfix, mailman -- the list goes on.

One of the problems that we faced when rebuilding the machine was a severe lack of documentation. Aside from scrounging around in the old /etc, it was really a case of "guess and check" with regards to making sure that services behaved in the same manner as before.

We certainly do not want to leave the next person in the dark when the root password is handed over (this is a university server, so we (hopefully) inevitably graduate and move on).

So, how does one document the beast? What techniques/methods are best to ensuring that configurations and policies are properly documented? I always leave a little #note when I edit configuration files - but is that enough? Shouldn't we be writing down our policies and changelogs into some kind of database - tracking bugs/problems and managing users?

Does there exist any specific software for the task of documenting a system, and if so - is anybody using it?


Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure


Related Links
o Also by semis

Display: Sort:
Documenting the Beast | 25 comments (22 topical, 3 editorial, 0 hidden)
CVS (4.07 / 13) (#1)
by mattdm on Tue Oct 03, 2000 at 11:54:23 PM EST

Use CVS for your config files. You get backups *and* documentation.

Re: CVS (3.00 / 6) (#5)
by h2odragon on Wed Oct 04, 2000 at 01:32:36 AM EST

Excellent. I've been wondering about CVS for exactly that purpose myself, but I haven't been able to take the time to read enough of the docs to get an idea what its strengths and weaknesses are.

Your brilliant, informative advocacy for CVS as opposed to other like systems has converted me, though, so I'll be putting all my servers' /etc and other configs under CVS right away. Might as well do my web sites, personal writings, and stock book too, while I'm at it.

Seriously, though, why CVS? What makes you a fan of it and what else have you used? How much extra effort is required for its use? If I need to change one character in one file; am I required to got through a half hour of set up and tear down ritual to keep the system happy? Does it puke if I just make the change and ignore the rituals?

I'll have to research the answers to these questions all by meself sooner or later, but perhaps somebody else has asked those questions already and would be willing to spend a bit of time dishing out that knowledge?

[ Parent ]

Re: CVS (4.50 / 4) (#10)
by cysgod on Wed Oct 04, 2000 at 04:08:47 AM EST

I'm going to try to directly answer your questions...

Your brilliant, informative advocacy for CVS as opposed to other like systems has converted me, though, so I'll be putting all my servers' /etc and other configs under CVS right away. Might as well do my web sites, personal writings, and stock book too, while I'm at it.
So, beautiful sarcasm aside here, there is a very important set of files that belong in CVS. And the set is defined as this:
Any file which I want to remember how and why I changed it without having to commit that information to memory.

That's it.

And that is the big win with CVS. I don't have to wonder why I made changes to my .vimrc and .procmailrc because I'll have recorded why in my CVS log message.

Whether that change added the whole file, changed the whole file, or changed one character, with one simple 'cvs commit' on the command line (or 'c' from PCL-CVS in [X]?Emacs) (or a mouse click in jcvs or tkcvs) all that information is safely stored in a repository.

Granted, I'm a little biased, because I'm implementing CVS for a moderate sized corporation right now, but with the hooks provided, you can do some neat stuff. Think instant email notification on commits. Think to all those files.

It's really nice to be able to look back and see when I changed some line in my MUA's config file, and see what I was thinking when I did it. Particularly when it causes it to fail when I upgrade, or when the upgrade overwrites it I can reapply the patches of what I did.

I highly recommend the Corolios Press book on CVS,
Open Source Development with CVS. It'll help you when you are just getting your feet wet.

CVS is good for even the dot files in your home directory, and is indispensible on a large project. <rant>Why the Linux kernel/Perl developers aren't using some sort of version control? My guess is that they were dropped on their heads when young, that's why I use FreeBSD, OpenBSD, NetBSD, and Python</rant>

CVS makes development on big projects simpler, CVS makes managing many files easier, and keeps you from having to remember why you put that space into the end of that shell variable in your .zshrc. As usual, feel free to follow up and I can answer more specific questions, or someone else with reasonable levels of CVS clue can.

[ Parent ]
Re: CVS (3.00 / 1) (#24)
by h2odragon on Wed Oct 04, 2000 at 04:43:37 PM EST

Good answer, thank you and thanks to everybody else who took the time. Terran answered my next specific question which would have been about multiple users modifying the same tree. Ya'll have just about got me convinced that I need to take the time set up CVS and play with it first hand, 'cuz it sounds like fun without undue extra effort.

Less to the point, I can maybe help illuminate the lack of version control in the Linux kernel by pointing you to Kernel Traffic #83, where you can read ESR's take on why Linus doesn't like shared code, version control, and so on.

[ Parent ]

Re: Open Source Development with CVS (4.00 / 1) (#25)
by Garc on Wed Oct 04, 2000 at 05:13:51 PM EST

The author of "Open Source Development with CVS" has released some chapters under a free liscense. The Chapters released are a great tutorial on how to use and administer CVS. Enjoy!

Tomorrow is going to be wonderful because tonight I do not understand anything. -- Niels Bohr
[ Parent ]
Re: CVS (4.00 / 1) (#12)
by jcamp on Wed Oct 04, 2000 at 05:42:10 AM EST

We run CVS on our DNS servers, so when we update zone files, we can go back and see what other people have done. CVS adds a few steps, but I consider them minimal for the type of information I can get out of it:

co -l somedomain.com.db
vi somedomain.com.db
ci somedomain.com.db (then CVS prompts for comments)
co somedomain.com.db

It's become 2nd nature do just do that now...plus CVS will also make the files read only when you check out at the end, so you don't accidentally bypass the CVS versioning. I think it works well, and when dealing with small config files, doesn't really take away from HD space. The only problem with CVS is user error, it has to be something that you actively use, or else it's pointless...but you could have a machine up for a year and have every change you've ever made documented right on the box itself.


[ Parent ]
Re: CVS (3.00 / 1) (#15)
by bscanl on Wed Oct 04, 2000 at 08:21:09 AM EST

What's the advantage of CVS over RCS in terms of simply keeping track of who done what and where on a small machine? I've used RCS in a "students running a machine for students" environment, it works well, but CVS doesn't seem to offer any more than RCS here... co -l file vi file ci -u file Between RCS comments, and comments in config files, it's very manageable.

[ Parent ]
Re: CVS over RCS in general (4.00 / 1) (#22)
by terran on Wed Oct 04, 2000 at 01:41:49 PM EST

The main advantage of CVS over RCS is when you have a collection of files which need to be tracked *collectively* - like source code for a project. If foo.c uses functions from bar.c, you need versions of both of them that go together.

With RCS, you can check in bar.c 12 times, and be on versision 1.12, while you've checked in foo.c 3 times, and are on version 1.3. You have to note somewhere that foo.c version 1.3 goes with bar.c version 1.12, or tag every checkin, or use some other kluge. CVS does this for you, automatically.

The other big plus of CVS is that it doesn't do file locking like RCS. It allows multiple people to edit files at the same time, and uses diff and patch (or the conceptual equivalent thereof) to apply changes when you do check in; if you haven't edited the same section of the file, this is automatic; otherwise you have to resolve the conflict yourself.

There are additional miscellaneous benefits, but those are the two big ones. CVS is also somewhat harder to learn (simply because it's larger and more complex), and requires more steps to do the same thing as RCS, so there's no point in using it where you don't actually need it. I'm a firm believer in CVS for source, where its advantages are the most applicable. I'm not convinced it's necessarily better than RCS for web pages or configuration files.

[ Parent ]
DocBook (3.16 / 6) (#2)
by acidos on Wed Oct 04, 2000 at 12:01:17 AM EST

I work for a college that uses an IP_MASQ server for giving residents access to the internet. This server was created by myself and a fellow student (we both work for the school), and is a new project. We currently use DocBook to write a "Guide" to the server, in which we put in things about what services the server offers, what programs and daemons we use to implement those services, how they should be set up, considerations to keep in mind, etc. Its not substitute for commenting your config files, but gives your next-of-kin a more readable form of what you were thinking at the time.

hobbes.nmsu.edu (3.60 / 5) (#3)
by fluffy grue on Wed Oct 04, 2000 at 01:08:17 AM EST

I have been the archiver/maintainer/webmaster/etc. of Hobbes twice now.

The first time I left (because I graduated), I didn't know who was going to replace me. As it turned out, a friend of mine got the job. I had left behind some docs on how to maintain the archive, and he followed them more or less, but because I was unavailable to help in person when bits hit the fan (and he never emailed me when problems came up), Hobbes got all screwy.

The second time I left (because I had gone back to Hobbes as a summer job), I had the opportunity to train the replacement for a couple weeks before leaving. I found that even with all the docs, nothing could help more than just walking him through the procedure. It also helps that he occasionally emails me for help whenever he has a problem with something, and that I'm still around to keep tabs on him (which I do on occasion). It also helps that I was able to scrutinizingly screen him and make sure that he was up to the task (although the first replacement was a friend of mine, I don't think I'd have recommended him as a hire).

So yeah, documentation's all well and good, but even the most thorough docs can't really replace hands-on training experience and continuing support.
"Is not a quine" is not a quine.
I have a master's degree in science!

[ Hug Your Trikuare ]

Use a package manager (2.33 / 6) (#4)
by khym on Wed Oct 04, 2000 at 01:21:34 AM EST

Use some sort of package manager (like RPM and the Debian thingy); if you don't trust downloaded packages, try to make a package from the tarball you've just compiled. This has two advantages:

  1. The packages database acts as another form of documentation, for what versions of what software you have installed.
  2. You can save all of the currently used packages somewhere. When trying to duplicate the setup on a new system, just take that archive of packages, install them all, and you're halfway there.

Give a man a match, and he'll be warm for a minute, but set him on fire, and he'll be warm for the rest of his life.
Re: Use a package manager (1.50 / 2) (#9)
by RadiantMatrix on Wed Oct 04, 2000 at 04:01:39 AM EST

Use some sort of package manager (like... the Debian thingy)
"The Debian thingy" you are referring to is apt-get, for your information. As for a blatant pitch, it really is one of the nicer automatic pakage transfer systems.
I'm not going out with a "meh". I plan to live, dammit. [ZorbaTHut]

[ Parent ]
Re: Use a package manager (3.00 / 2) (#11)
by cysgod on Wed Oct 04, 2000 at 04:15:51 AM EST

A package manager is a good start, but don't forget to keep track of your /etc files in some sort of version control system as well. A majority of the work in recovering a workstation or server, isn't installing all the big programs, it's the weeks you spent tweaking those conf files just right so everything worked and all the blinky light illuminated.

Having to redo that work is annoying and a royal pain in the ass.

See post 10 on this subject for my encouragement of CVS (or any version control that works [i.e. not Visual SourceUnsafe]) to store this stuff and keep change history.

<aside>No, I don't hate all MS products, it's just that they don't trust VSS for their own development, so why on earth would I?</aside>

[ Parent ]
Re: Use a package manager (4.00 / 1) (#13)
by Nickus on Wed Oct 04, 2000 at 06:02:43 AM EST

A package manager is good but not for this. I think most people what kind of software runs on their servers and it easy to install. It is the configuring and tweaking that takes time and if you don't write it down you don't remember it. And a package manager doesn't help you with that.

Document, write down.. and write down even more. The only solution

Due to budget cuts, light at end of tunnel will be out. --Unknown
[ Parent ]
Document the differences (3.33 / 3) (#14)
by stuartf on Wed Oct 04, 2000 at 07:46:15 AM EST

When documenting a new server, I always just do a straight list of all the software versions that are installed, and then just list where the config differs from the default. You can always get bogged down with too much documentation, so you need to document what's relevant

Good Documentation Takes Time (3.60 / 5) (#16)
by Doscher on Wed Oct 04, 2000 at 08:42:54 AM EST

The biggest problem I have seen for any network/server team is not the unwillingness to document, but the unrealization of exactly how much time it is going to take.

The best documentation assumes nothing. This means that you assume your DHCP configuration will be handled by neither an idiot or a genius. The point of the documentation is to cover all the bases, so that a successor can easily read your docs and pick up where you left off.

Good documentation takes time, and you are not only weighing against how much time it takes you to figure it out, but also whether your successor figures it out at all. Comments are not enough. For each install of software, cover the process from start to finish, making sure your reader will know what to do to get from step A to step B. Word or Wordperfect, or even html docs with screenshots are good, as pictures speak a thousand words and can easily show the reader if they are 'in the right place' when configuring something.

In the projects I am invloved with, the documentation often takes more time than the actual development, or close to the same. This makes sure that the next person doesn't have to do your work all over again. And that is why good documentation is worth it.

Re: Good Documentation Takes Time (4.00 / 1) (#21)
by terran on Wed Oct 04, 2000 at 12:34:51 PM EST

This makes sure that the next person doesn't have to do your work all over again.
I thoroughly agree with your goal. However, I am not convinced that your procedure actually leads to this goal:
For each install of software, cover the process from start to finish, making sure your reader will know what to do to get from step A to step B. Word or Wordperfect, or even html docs with screenshots are good, as pictures speak a thousand words and can easily show the reader if they are 'in the right place' when configuring something.
This sounds to me like you're documenting a procedure which can subsequently be followed by rote to reproduce the same result. This is all well and good for the sake of getting the software installed, but I do not believe that it will cause your successor to actually understand what you learned when you figured it out the first time. You learned all the theory behind the choices that you made to the install. The person reading the instructions will know how to do the install by rote. The person who understands the theory can find problems later. The person who pushed the buttons in the specified order can't. In my opinion, part of the "work" of administering something is understanding it well enough that you can fix problems. Thus, someone who follows the instructions may duplicate everything you've done so far, but they won't have the same ability to deal with future problems that they would have if they'd figured it out themselves.

It's possible that you meant that one should document the theory and the exact procedure, in which case I agree with you except for the necessity of documenting the exact procedure. ;) I'm all in favor of notes like "When it asks you foo, what it really means is bar" if something is counterintuitive, but I think things like "select the third option, then press m" are overkill.

[ Parent ]

Re: Good Documentation Takes Time (3.00 / 1) (#23)
by Doscher on Wed Oct 04, 2000 at 01:52:11 PM EST

Again, you cannot assume the technical ability of the person reading your documentation.

a procedure which can subsequently be followed by rote.. You're assuming the person reading the doc has to follow it step by step. If I am writing a doc on how to add gasoline to your car, you can use the documentation to just find the gas cap (behind the license plate, etc) if you have experience filling cars with gasoline. But if you need to know what side of the car it's on, and I assumed you were technical enough to know, you may have pulled into the wrong pump station. See my analogy?

Again, from my previous post pictures speak a thousand words and can easily show the reader if they are 'in the right place'

"select the third option, then press m"... I never said 'perform step by step instructions telling the user what button to click', but rather to make sure the user or admin knows they are in the right place. You are absolutely correct that you cannot predict the questions, but even by looking at old good documentation, you can get an idea of what information will be required.

[ Parent ]
The Old Fashioned Approach (3.00 / 3) (#17)
by Ruidh on Wed Oct 04, 2000 at 09:12:12 AM EST

Keep a notebook for each server. Log in all the packages that are installed and upgraded. List all the site specific configurations. Dump off hard copies of important files: /etc/fstab, the sendmail m4 file, /etc/inetd.conf, directory listings of the rc.d directories.

Imagine what you would need to recreate the server if it went up in flames and you had nothing at all to refer to.

It's tedious. It's easy to forget to keep up, but if you do, it's worth it.
"Laissez-faire is a French term commonly interpreted by Conservatives to mean 'lazy fairy,' which is the belief that if governments are lazy enough, the Good Fairy will come down from heaven and do all their work for them."
My docs are a web site (3.50 / 4) (#19)
by mrr on Wed Oct 04, 2000 at 11:37:41 AM EST

My server docs are web pages on the internal net. They are written on the assumption that a knowledgeable admin could review the web page and quickly spin up on how things are put together. There is also some information to help a new admin come up to speed if s/he doesn't know the technology.

Each page includes:

  • The version of the software,
  • location of the config files and when feasible a link to a web page containing their contents,
  • variations from standard practices,
  • configuration overview where I describe the background thinking and expand on the basic config,
  • potential problems, where I list what may go bad or what unforeseen problems have bitten us in the past, an example would be in the NTP configuration a note saying that our GPS clock receiver occasionally loses sync with the satellites and won't resync until aftar a power cycle.
  • topic resources where I point to other references available on the topic.
All this information can be easily written up at install time. It's proven helpful to remind me of why we've taken one approach over another.

Having a web page set up also allows me to point new people in the company to a store of information they can access easily form their desks.

Document intent (3.50 / 2) (#20)
by terran on Wed Oct 04, 2000 at 12:24:20 PM EST

In my opinion, the important thing to document is your intent when you make a change - why you did it, and what it's supposed to do. This is because intent is the one thing that you *can't* get from the configuration files on the old machine.

If I want a list of packages that were installed, I'll look in the package manager database on the old machine (or backup). Same for configuration files. What I want to have written down somewhere by a human is something that diff won't tell me - *why* things were done.

I don't really see the point (perhaps someone who does can explain it) of writing instructions which can be followed by rote - anything that can be redone by rote can probably just be reinstalled from the backup, too.

Documenting the Beast | 25 comments (22 topical, 3 editorial, 0 hidden)
Display: Sort:


All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!