maynard voted 1 on this story.
I'm in the last days with my current employer (BBN) and am about to switch jobs to MIT. However, we currently have a 300+ Linux/Solaris/IRIX cluster which handles batch jobs for training our speech recognition engine.
Given this I'd like to point out that you're putting the cluster before the application, which is bassackwards (so to speak). The first question you should ask is "what kind of computational resources does your department need?" instead of "how can I use this cluster?" We can't help you with that simply because it's different from site to site.
Chances are you may find running a batch manager much easier at first than a traditional Beowulf cluster, simply because programs need to be tailored to use either their MPI or PVM message passing libraries. These are somewhat like a userspace threading library such as pthreads only instead of spawning threads within a process, an application calls one of these libraries to request CPU from another node in the cluster. A batch system is much easier to implement if you already have your application written and don't want to port to one of these libraries... with the caveat that you lose some fine grained load balancing.
You might also want to consider checking out the MOSIX kernel patches which allow for dynamic load balancing and a shared process table space across a cluster. It's quite amazing... I set a six node mosix cluster up for a login server at work and it's run like a charm ever since. The system will dynamically checkpoints a process and move it to the least active node without a user ever knowing... it forwards file descriptors and network sockets to the home node, has a cute memory system which moves dirty pages first and "leaks" inactive pages slowly so it doesn't saturate the network... all sorts of cools stuff.
I'm running out of time so I gotta go...
Read The Proxies, a short crime thriller.