Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

Unreliable scripts

By MSBob in Op-Ed
Sat Sep 29, 2001 at 12:55:36 AM EST
Tags: Software (all tags)

This week most of my time in the office was spent covering for our build process sysadmin. He had to take a week off and it's been my responsibility to carry out the duty of building and deploying our application. I can't describe the experience as a pleasant one.

My woes are mostly to do with the quality (or lack of thereof) of our scripts. Our build environment is fairly typical and makes heavy use of shell scripts together with Ant to provide for a "seamless" build environment. In my view the word "seamless" next to the word "shell script" should be considered an oxymoron. It's not that our scripts are any worse than any other out there in the industry if my previous experience is anything to go by. They grab files from perforce, compile stuff, deploy it to our testing environment, patch configuration settings etc. Problems begin however, the moment you try to do something that is slightly out of the daily routine like pulling a different version of a specific source file from the SCM system. Well, I tried that and ran into a number of weird problems and multiple places where assumptions have been made about the way our build process should work. Going through the mess of undebuggable scripts together with uninformative Ant logs left me with a desire to just write my own little script to do the job at hand. That's exactly what I ended up doing. Rewriting some of our build process functionality from scratch ot accomplish several simple tweaks that were required by the development team.

Now the week is coming to a close and I'm glad my colleague will be back on Monday. While I never was a fan of scripting languages after this week I came to the conclusion that scripting is the Achilles heel of the software industry.

Scripts are horrible to debug. Oftentimes they don't abstract things like system responses well enough to allow for truly platform independent scripting. In particular handling OS level exceptions such as lack of file access permissions seems to behave different on various platforms. Additionally scripts often tend to make many assumptions about the environment they operate in. In particular they ususally assume a particular directory and file layout which makes them error prone and inflexible. Now I realise that these problems can be overcome with extra code in the script that prevents it from running if things aren't set up the way they're supposed to be. Unfortunately there are very few scripts that go the extra mile. Oftentimes you run scripts at your own peril. Sometimes they work other times they leave random files scattered around your file system. Often scripts move files and if they don't succeed at first running them for a second time is worthless as they often leave things in an inconsistent state (by moving files for instance) regardless of the success or failure of a script.

So what can we do to improve matters? I'm not sure but there are certain things that can be done to improve the quality of scripting. A decent debugger would be a good start. I believe there are debuggers out there for things like Perl or Python. However, korn scripts can only be debugged by print statements. Ant scripts can only be traced via logging the output. Lack of a debugger means that at least some of the functionality of your script has to be taken on faith.

The one piece of functionality that seems like a particularly glaring ommission in scripting languages is a transactional API. Transactions help immensly in keeping things in a consistent state. It would be invaluable to be able to "rollback" any file moving activity if a file is not found or some environment setting is missing. This way scripts would be able to leave things clean when they terminate unsuccessfully. Assert statements would also add to robustness if scripting languages provided them. Being able to assert that a requested operation was successful would make for scripts that are much easier to trace in case of problems rather than having to go into directories affected and visually inspecting what went wrong.

While scripting languages are becoming powerful and new build tools try to make things much simpler it seems that there are several things that we learned in other areas of computing which should be explored in scripting languages to bring them to the level of robustness of compiled code.


Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure


Related Links
o Also by MSBob

Display: Sort:
Unreliable scripts | 20 comments (20 topical, editorial, 0 hidden)
assert sucks (4.20 / 5) (#1)
by Arkady on Fri Sep 28, 2001 at 01:20:51 PM EST

Oh, please don't go asking for assert as an upgrade; there are much better mechanisms available.

Assert, as practiced by every C++ environment I've used, basically means "check this condition and blow chuncks if it's not met". Yuk; it leaves you with an error message and a program that's already died, so you can't check why your condition failed. This is particularly egregious if you're doing something with serious state, since the assert's just dumped your whole state when it killed your app.

Live debugging, as practiced by Lisp systems, gives you a chance to examine the running program when a condition triggers an exception by dropping you into a debug Listener. Rather than a dead app, you have a paused one and access to fiddle with it without having to restart it from scratch. Your full state is intact and available for examination. You can even tweak it and resume its execution where it stopped.

Assert, on the other hand, treats your app like a sick horse: it shoots it for you and lets you figure out why by looking at the corpse.

Again: yuk.


Turning and turning in the widening gyre
The falcon cannot hear the falconer;
Things fall apart; the centre cannot hold;
Mere Anarchy is loosed upon the world.

C assert does (4.00 / 2) (#2)
by MSBob on Fri Sep 28, 2001 at 01:25:40 PM EST

I didn't necessarily mean a C style assert. It's true that most of the time C style asserts just terminate the running thread leaving you with little to go on. What I wanted (but didn't describe well enough) is an assert where you can customise the handler (a bit like java/C++ exception handling but not quite). I believe that java's assert allows you to customise assertion failure handling.
I don't mind paying taxes, they buy me civilization.

[ Parent ]
You can do that in C++ too, but... (none / 0) (#17)
by djelovic on Sun Sep 30, 2001 at 03:15:46 AM EST

What I wanted (but didn't describe well enough) is an assert where you can customise the handler (a bit like java/C++ exception handling but not quite). I believe that java's assert allows you to customise assertion failure handling.

You can also do that in C++, but why?

If an assert fails, something has gone wrong in your program. An unexpected condition, something that you didn't prepare for.

There is no known way of handling an unexpected problem correctly.

When an assert fails what you want is to have the debugger pop up and let you examine the call stack, variables and data structures. And that's what all popular Windows C++ compilers do.

On the other hand, if the problem is something you have prepared for and know how to handle, then you shouldn't assert, but instead throw an exception that will be handled at the place where the transaction has originated.


[ Parent ]
Assert (none / 0) (#5)
by ucblockhead on Fri Sep 28, 2001 at 01:45:24 PM EST

In Microsoft Visual Studio, _ASSERTE() gives you three options when the condition is false. You can terminate the program, go into the debugger at that line or simply ignore the assertion and continue. In "release" mode, all _ASSERTE()s are stripped out.

It is quite nice.
This is k5. We're all tools - duxup
[ Parent ]

Assert on Visual C (none / 0) (#6)
by squigly on Fri Sep 28, 2001 at 01:46:46 PM EST

There's no need for assert to be so icky. Visual C gives you a dialogue box which lets you run the debugger, or continue. There's no reason why other environments can't do the same.

Granted, its a pretty rubbishy thing to use unless you already know what the error is, and I'd recommend proper debug output for scripts, but it is easy to add, and can be useful at times.

[ Parent ]

Better scripts (3.00 / 1) (#3)
by wiredog on Fri Sep 28, 2001 at 01:33:19 PM EST

First, what are the ones you use written in? Bash? Csh? If so, rewrite in perl or python and add comments. It's the lack of comments in some scripts (and source files) that makes them un-debuggable. Using a Real Language would allow it to test for failure of an operation, and ask for input from the user in order to retry the operation.

If there's a choice between performance and ease of use, Linux will go for performance every time. -- Jerry Pournelle
Ksh (4.00 / 1) (#4)
by MSBob on Fri Sep 28, 2001 at 01:40:44 PM EST

The thing is based on Korn shell scripts and Ant scripts working together to accomplish the build. I believe it's Ksh simply because that was the only scripting language that the guy who wrote it knew well. Now this isn't my regular job. I was thrown into it because all our test/production folks decided to go on holidays all at the same time. Myself I'm a developer who prefers compiled stuff with the compiler and linker doing much of the verification for me. I was just making some observations about which direction I think scripting languages should go in order to make them more reliable. However, given the experience of this week I'll probably avoid any work with scripts as much as I can :).
I don't mind paying taxes, they buy me civilization.

[ Parent ]
...only scripting language... (3.80 / 5) (#7)
by johnjtrammell on Fri Sep 28, 2001 at 02:59:55 PM EST

I believe it's Ksh simply because that was the only scripting language that the guy who wrote it knew well.
WOOP WOOP WOOP! Danger! Danger!

If the guy only knew this one scripting language well enough, you likely [a] didn't get the right scripting language, and [b] didn't get someone with the expertise to do the job.


[ Parent ]

Live in the real world (2.50 / 2) (#15)
by frabcus on Sat Sep 29, 2001 at 02:31:43 AM EST

Are you seriously telling me that

a) If you need to do a job, you shouldn't use a tool that you already know that is suitable? Even if you only know one such tool, if it's suitable it's suitable.

b) You can always find staff with the right expertise to do every job?

[ Parent ]

The problem isn't scripting; it's process (4.62 / 8) (#8)
by tmoertel on Fri Sep 28, 2001 at 04:20:57 PM EST

It sounds like your problem isn't so much scripting as much as a poorly defined build process. Ideally, a build process should be:
  • well defined;
  • documented on paper;
  • divided into logically related chunks (e.g., checkout, build, regression, labeling, deployment, ...);
  • automated chunk-by-chunk in scripts or makefiles; and
  • automated as a whole by a master script or makefile that invokes each chunk in turn.
The first two are by far the most important, and it sounds like your environment lacks them both. With them, it would have been easy to handle your out-of-the-norm build. Even if you had to do the build by hand, you would have been assured of a correct result, and you wouldn't have been subject to trial-and-error discovery of the build process.
While I never was a fan of scripting languages after this week I came to the conclusion that scripting is the Achilles heel of the software industry.
The software industry's weakest link isn't scripting languages. It's the rampant refusal to adopt sound development processes.

To be clear: Using a "real" language wouldn't have solved your problems, but establishing a solid build process would have.

My blog | LectroTest

[ Disagree? Reply. ]

It's documentend, modularised... (3.00 / 3) (#10)
by MSBob on Fri Sep 28, 2001 at 05:44:51 PM EST

and still a royal pain in the butt. The problem is that writing scripts as well as testing them against paranoid conditions is not as easy as it should be. Scripting languages are tailored towards quick hack jobs rather than proper software engineering. That is the main problem I was trying to emphasise in the article
I don't mind paying taxes, they buy me civilization.

[ Parent ]
Are you sure it's not the process? ;-) (4.66 / 3) (#14)
by tmoertel on Sat Sep 29, 2001 at 01:20:22 AM EST

Scripting languages are tailored towards quick hack jobs rather than proper software engineering.
We're going to disagree on this one. Scripting languages aren't tailored for "hack jobs"; they're tailored for their intended purposes. Shell scripting languages like bash are tailored for job control, embeddable scripting languages like siod and Guile are tailored for integrating with other software packages, and heavier scripting languages like Python and Perl are intended for a wide range of tasks, including general-purpose programming. They're tools. Nothing more, nothing less.

How the tools are used -- not the tools themselves -- determines whether the resulting work is a hack job. Nobody blames the chisel when a sculptor destroys beautiful piece of marble rather than revealing the sculpture within it. Likewise, if the sculptor chooses to use an axe instead of a chisel, nobody blames that mistake on the axe or the chisel.

Do you see what I'm saying? Scripting languages are tools. If you use a scripting language on a job and the result isn't up to snuff, it is not because of the scripting language. It's because the scripting language was used improperly (in which case there is a problem with your process) or because the wrong tool was used for the job (in which case there is a problem with your process). Either way, fix the process, don't blame the tool.


My blog | LectroTest

[ Disagree? Reply. ]

[ Parent ]
Quick universal fix (3.00 / 3) (#9)
by ubu on Fri Sep 28, 2001 at 05:20:27 PM EST

Everything should be written in Jython.


This signature is a magical vanity summoner. (streetlawyer,Inoshiro,spiralx,alprazolam,eLuddite)
I agree (2.50 / 2) (#16)
by sjmurdoch on Sat Sep 29, 2001 at 08:34:36 AM EST

I was about to say the same thing (except I use normal Python, but I think my comments will still apply to Jython).

Firstly I should say that you can still write poor quality scripts in Python, and using Python is no subsitute to good comments and documentation but it helps a lot.

Python is trully cross platform. I wrote an app of around 1,000 lines of code. I was told it would run only on RedHat Linux 6.2. I made no effort to make it cross platform, but someone copied over the program to a Windows 2000 box and double clicked on the icon, and it worked perfectly. This won't be true for every application but it is better than pretty much every other language apart from Java. It doesn't need lots of external programs to do everything and this imporves the readability too.

The syntax is great. Despite it having the the power of Perl it is very easy to read and write. You may dislike whitespace being used to define blocks at the start (I did) but you'll probably learn to love it. I found it often did exactly what I wanted without me thinking about it. I only noticed the lack of "missing semicolon" errors.

It's object oriented, and not a hack like C++. This might be yet anotehr buzzword and isn't the answer to all problems but in some cases, particularly absraction it makes life much easier.

It has exception handling. This is superb and makes the code much much easier to read if used properly. It also has a debugger, which is apparently good, but for all the above reasons I have never had to use it.

In summary writing the scripts in Python will probably help (all other things being equal).

P.S. When did JPython change to Jython?
Steven Murdoch.
web: My Home Page
[ Parent ]

This problem is not technical. (4.75 / 4) (#11)
by RobotSlave on Fri Sep 28, 2001 at 08:23:21 PM EST

The build system for any large software project typically works ok, but could really use rewriting.

This is an organizational or management problem.

The project starts small, one of the engineers whips up a build system in a day, everyone can understand it because it's small, and life is groovy. Development proceeds apace, various engineers tack extra bits onto the build process as they go, and it bloats. At some point, someone says, "hey, we really need one person to just look after the build as a full time job." Since none of the existing engnieers wants to do it, some poor slob is brought in from outside to do the job. It takes the new person a while to get up to speed on the tangled mess, and it's clear that a rewrite is called for, but this person can't work on a new design, because his or her day is totally interrupt driven (hey, could you tweak the build in such and such a manner? Hey! My build died! Why? etc). After a year of this, the poor slob finally qualifies for vacation, and leaves, hoping that in his or her absence, someone will realize that he or she is in a rotten position, and do something about it. Looks to me like you reached the first conclusion, but then went off into left field and decided that the real problem here is the technical nature of scripting languages.

If you haven't encountered the "lousy build system at large company" problem before, then you're lucky. If you've been the poor slob in charge of the build, then you've probably considered leaving the software development world entirely.

Job security (none / 0) (#18)
by schlouse on Mon Oct 01, 2001 at 04:43:39 AM EST

I agree with most of what you've said, but you're missing something: Knowing exactly how the build system works, down to the ground, can be a very good thing to have from a job security standpoint (if that's something you care about ;-)

A couple of months ago, I started a small campaign to "revise" our build process. As soon as I got the go ahead from management, I promptly got to work and came up with the best damn build process I could. Of course there's always some prerequsite knowledge needed; but learn all of that before you make the offer to management, and then act like you've known it all along.

The project was hell, sure, but it turned out to be one of the best things I've done, from a professional standpoint. If you're relatively untested at your new job, then it's even better because it automatically gives you a bit of street cred, which can be useful. When you walk in off the street and make people's lives easier, you gain quite a bit of allies that you perhaps normally would not make. This is usually more valuable than most people believe.

Mark S.

[ Parent ]

Sure, a little bit of (none / 0) (#19)
by RobotSlave on Mon Oct 01, 2001 at 06:12:25 AM EST

street cred will make your job easier. But if you do too good a job, you'll make enemies (think of the engineers and managers who were in charge of whatever you rewrote). Since you don't know what the hell the political structure of the place is like when you walk in, you're just as likely to make more (or more powerful) enemies than friends if you overachieve. If you just stick with the system that's in place, and don't rock the boat, your job security is better on average.

Does it suck? Oh yes. But it's a lot smarter to hold your grand reforms in reserve until you figure out exactly what sort of situation you've been dropped into.

[ Parent ]
Hmm (none / 0) (#20)
by schlouse on Mon Oct 01, 2001 at 02:02:02 PM EST

I certainly wasn't advocating just jumping in as soon as you get hired. It was more along the lines of "So you need some job security or more street cred..."

As far as build systems/makefiles go, I've found that there's often nobody that really wants to deal with it, and they're really happy when you come along and make the desired changes. However, as you point out, this will not always be the case.

Mark S.

[ Parent ]

Don't blame the medium (5.00 / 3) (#12)
by malikcoates on Fri Sep 28, 2001 at 08:45:27 PM EST

If someone handed me a pile of bad ksh code for a week, I'd probably be just as upset as you are. I just think you should place blame on the whoever wrote and uses the scripts.

This article sounds like someone complaining that there isn't anything good on TV, therefore we should upgrade TV's capabilities. I don't buy it. Give ksh a builtin debugger and transaction capability and I bet your scripts will still suck.

It's a lot more likely that the guys who use the scripts didn't give you enough info to do what you want, or the scripts were not designed to do what you want, or the scripts were not tested to do what you want.

Personal Observations (4.66 / 6) (#13)
by Damien Vryce on Fri Sep 28, 2001 at 10:53:11 PM EST

My personal experience is that all forms of coding, and all forms of documentation share a common weakness: the humans who create them.

My first shell scripting effort happened because there wasn't anyone else to do the job, so I had to learn sh scripting on the fly. The result was a set of sh scripts that I wince to think of now. I rarely was aware of the best solution, and was working under time pressure. However, I documented exactly what was going on in every part of those scripts. Both the logic behind the flow of the process, and what I was intending for the various lines of code to do.

While I did write the scripts under time pressure, with a focus on getting things to work, I also took the time later to make the scripts do everything I knew how to do to proactively detect and handle just about any error condition I could think of and that I could find a way to handle.

The end result is that when I handed that particular build system off to those who came after me, they were able to understand what was going on and update them when needed.

Since I handed them off, that set of scripts has been in regular use for several years with only cosmetic enhancements.

What's the moral of this story? I poured months of my life into doing everything I knew how to do to build those scripts to be stable, flexible and friendly to learn. I've run into other script like the ones mentioned, which are a nightmare to work with. My opinion is that it all comes down to the element I find missing in so many things: quality.

Unreliable scripts | 20 comments (20 topical, 0 editorial, 0 hidden)
Display: Sort:


All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!