First, a brief overview of what this project was all about. It's name was the VDP, for Virtual Dental Patient. The idea is simple: use a high resolution 3D scanner to digitize stone replicas of a patient's teeth (these replicas are made from those bite impressions you may have had made at the dentist's office at some point). Dentists make these replicas to allow them to make reliable comparisons of a persons teeth and gums as they might change over periods of many years. This analysis is high qualitative, and is concerned with things like "is this tooth wearing" or "are these teeth moving". However, by using 3D models of scanned replicas analysis can be done quantitatively, we can measure how much volume a tooth has lost due to wear, or how far a specific tooth has moved. Sophisticated visualization was also available. I can't say a lot more about what a dentist can do with such tools in terms of specific diagnosis or treatment planning because I'm no dentist myself, but suffice it to say that such tools would definitely be useful. The VDP project in a nutshell was concerned with developing these tools, and equally important, proving their reliability and accuracy in making measurements.
The project was developed by a small team ranging from about three to eight people at a time, mostly undergrad and graduate students, but also a couple non-student developers (including myself after graduation, hence the five years working on the project). Of the two other undergrads that worked long term on the project, one stay on for about four years total, getting his Masters degree after graduation, the other guy who handled more of the hardware and networking for the department was there for about three years I believe (I think he had a double major) but didn't participate as much in direct VDP software development or project design. We developed on Windows using Visual C++/Visual Studio 6 and the Microsoft Foundation Classes (MFC) for the interface. 3D graphics used OpenGL, though Windows GDI (Graphics Device Interface) were used for limited 2D graphics capabilities. The code we wrote was mostly spread across two major applications and totaled at least 100,000 lines total (probably not more than twice that), which I would guess is medium sized by industrial standards. The two major applications were Stratus, which integrated multiple 3D scans of an object into a single model, and Cumulus, which was used to build and analyze virtual dental patients by assembling multiple models representing upper and lower jaws, as well as various bite records.
One of the biggest problems from the get go was that the original grad students developing the project were skilled coders but didn't seem to believe in comments, and there were no design docs to aid us. This eventually lead to much code duplication in Stratus because we didn't know how to make use of most of what was in there. Cumulus hadn't been started yet, so we made bad design decisions from scratch there. Eventually a complete replacement for Stratus was developed using a custom built VDP class library. An upgrade for Cumulus (which was functional, but much larger than Stratus and ugly in terms of architecture and design) was started in the hopes of improving future research and development, but never really got very far. In the end our class library, which was built from the ground up using lessons learned from early mistakes in Cumulus was only used to replace the less significant application of the two. By the time the new Stratus was complete the development team was a shell of itself, I would soon be let go as the grant funding my position expired, and so the rewrite ended as a definite net loss of productivity.
Though the project in the end did not really fulfill all of its research goals, the software that was built did fulfill most of the original goals purely in terms of capabilities. The problem is that it took the better part of five years, and I freely admit that much of this was not due to lack of programming ability but to project design decisions for which I was in large part responsible. This process has made me a better developer in the end, and I'd like to share a few nuggets of wisdom that I've learned to help present and future young developers who get put in charge of development projects for which they lack the experience to guide effectively.
Don't reinvent (big) wheels. If you have a need for some general utility package (math, GUI, etc.) someone out there has already written what you need, except theirs is mature, complete, and optimized while yours will be buggy and slow (though maybe a bit leaner). There's probably an open source wheel too, and under a license favorable to even proprietary projects.
In the VDP we had two clear instances of this behavior that I can think of, one that was sort of bad, and another that was really bad. The sort of bad example was our math and geometric primitives library. In any 3D graphics application there are going to be a lot of vectors and matrices tossed around, and significant use of lines, line segments, rays, planes, and other primitives. Probably 80% of the operations used (remember the 80/20 rule?) are just vector arithmetic, dot products, cross products, rotations, translations, and scaling, but the operations in the other 20% can keep you busy for a long time in planning, coding, and debugging. Moreover, these operations are so fundamental to 3D applications and might be repeated so many times that performance is essential.
Such libraries are fairly ubiquitous as they are essential components of games, simulations, and visualization systems, but for the VDP we wrote our own. The process was made more tedious by the fact that we decided template-based primitives were important even though all data storage and the majority of the calculations were done using single precision floats. In a few cases double precision may have been warranted (mainly transformation matrices), but in the vast majority of cases single precision was completely sufficient given the precision of our input data.
The really bad instance of wheel reinvention was creating our own string type. This might have been justified for a word processing application, but remember this is a 3D modeling and visualization app. The entire justification for this approach was the primitive string features in the standard C library, which we felt were insufficient for our relatively modest parsing needs, and the desire to make the software portable (e.g. not rely on Win32 or MFC string types). I'm not sure what string libraries are out there, but surely the number dwarfs the number of geometry libraries. In any case I'm fairly certain at this point that a few customized parsing commands to augment the basics in the C/C++ libraries would have been sufficient for our needs. Either a 3rd party library or a small set of custom commands (or even a very simple custom string class) would have been a much better choice than what we actually did, which was to implement in C++ the unholy child of the union of the Java String and StringBuffer classes. No, I did not participate much in the coding of this beast, yes, at some point I actually thought this was an at least okay idea.
Runner up wheel reinvention: implementing clones of Java style nestable stream classes. Yes we, had the ability to write a 3D model object to a binary output stream, which passed it to a text writer stream, which in turn could encode this in UTF-8. That is, if anyone had ever wanted to do such a thing, which I'm pretty sure no one ever did.
Don't write your own scripting language. Whatever you make, it will take a long time and be buggy and primitive compared to anything else you can use. The headaches of getting a scripting engine inserted into your code only seem big, they will be small compared to rolling your own.
For the original Cumulus we wrote our own language (we called it a macro language instead of a scripting language, but same idea). It was a simple object oriented language which reflected the types of objects available in the software: 3D surfaces and transformation matrices were the most important components, but other objects were also available. The idea was nice enough, but it took a fair amount of time and the syntax was primitive (designed for parseability to ease implementation, instead of writeability). Because it wasn't very effective it wasn't used enough to justify the work that went into creating it. The problem was compounded by the fact that it was bolted on a little late in the game, so many aspects of the application were not available to its interface. Many algorithm options were available only in the dialogs designed as the original interface to those algorithms. On the plus side, the realization that this was a problem did lead to much better design in the new VDP framework to ensure that all elements were scriptable, and that interface was kept separate from core logic.
In the rewrite we switched to TCL as our scripting language. This might have been a success had we ever rewritten Cumulus to use the new libraries since Cumulus is really where scripting might have shined, but since only Stratus (which had little need for scripting) was rewritten there was never much incentive to make any use of scripting features. I guess the lesson here is to build an application in a scripting friendly way, but don't actually add a scripting engine until you have a need that justifies the headaches involved with incorporating it into your software.
Don't create large platform/toolkit independent abstraction layers. Yours will be limited, buggy and slow. See the Dos for better alternatives.
To future proof our software against new platforms (which the three of us software development leaders all secretly wished for, because none of us had any great love for Windows) we decided to ensure that all platform dependent code was safely isolated behind platform independent abstraction layers. The main culprits we determined were GUI, 2D drawing, threads and locking, and application initialization. Fortunately we came to our senses before implementing our own cross-platform GUI toolkit, though this was a serious consideration for a while. On that front we decided that only non-GUI portions of our code had to be truly platform independent (well, sort of, we ended up with platform independent code for setting up menus and toolbars) and that the rest could be wrapped fairly easily.
This actually didn't turn out too bad, but we really would have saved a lot of time if we had just picked out a platform independent toolkit and been done with it. If we would have used Qt all these issues would have been taken care of, with much greater flexibility and completeness than the very simplistic, though functional wrapper code we came up with. Plus, we would have almost certainly had large boosts in productivity using a modern GUI toolkit like Qt over the awful MFC (itself a crummy object oriented wrapper for the decidedly not object oriented Win32 GUI libraries). Alas, our projects were not open source, and it's difficult convincing your boss to pony up a few grand a year for the Qt developers license.
Don't create your own handles for objects unless you have an actual, absolute need for object memory relocation, and then only use it for objects that actually need it. Big, professional toolkits might have reasons for doing these things universally, you almost certainly do not.
MFC and Win32 use handles to reference pretty much all objects. We didn't really understand how object relocation was used in Windows (heck, I still only have a vague notion) but that didn't stop us from implementing out own handle table to track every persistent object. We never moved these objects around in memory, so these handles created an absolutely worthless layer of indirection that infested a large amount of our code, mostly in the nebulous area where interfaces and algorithms came together. If we had just mapped object name strings (which facilitated scripting capabilities) to object pointers we would have had all the benefits of our actual implementation, minus a lot of boilerplate code needed to actually penetrate the abstractions and reach an object.
Don't mix 2D and 3D graphics toolkits. If you need mixed 2D and 3D graphics use OpenGL and try to do the 2D stuff in OpenGL. If you absolutely need to overlay toolkit specific 2D graphics over 3D (e.g., OpenGL does not support fonts in a nice way, and you need good fonts to overlay 3D graphics), find a toolkit that is meant to work within your 3D system, or failing that use a cross platform GUI toolkit with good integration with OpenGL. I suppose a lot of this holds for Direct3D as well as OpenGL, but with OpenGL you get a library that is supported on every major platform as opposed to only one platform.
This one is pretty application specific, but there are probably similar examples from other problem domains as well. If you need to mix libraries/toolkits with related functionality, first make sure that neither library is actually capable of doing everything, and failing that try to find libraries that are designed to work together. We used Win32 GDI commands to overlay 3D rendering with 2D lines and text, and compounded the matter by actually using our own custom wrapper for 2D graphics in the name of portability. Since OpenGL was already the most portable part of our software (other than the C and C++ libraries themselves) doing the 2D stuff in OpenGL would have simplified things greatly, eliminating both abstract platform independent wrapper classes and the derived, platform dependent implementation classes. For our simple 2D drawing needs OpenGL was certainly up to the task, and I'm sure we could have found a 3rd party font library for the few cases where we actually wanted to use text in a 3D window.
Don't create interesting but ultimately irrelevant side projects that lead your team astray from the true goals of the project.
I think that this may be a particularly large danger in a University research setting, where projects have the dual goals of educating students as well as developing worthwhile software. In our case the side project was a small cluster intended to quickly perform processor intensive algorithms on 1999's hardware that would be more reasonably executed on a high end desktop 3-5 years later. I didn't have much to do with this particular endeavor, and I'm not really sure whose idea it was, but in hindsight it is obvious to me that even if the cluster had ended up working perfectly it never would have been worth the effort that was put into it. Most of our most intensive algorithms ran just fine on a desktop machine, though some experimental algorithms might take an hour or so (a few brain dead algorithms that one or two developers cooked up took over a day to run, but that was because the algorithms were naively implemented, not because the problem was inherently that difficult). Even if we really did have a large number of algorithms that really required days to execute the sensible solution would be to set aside a few powerful desktops for this task, and if necessary make modifications to the algorithms to work on small chunks of the data one at a time (i.e. by swapping the rest out to disk). This would have been much simpler than building a cluster, developing software to communicate with the cluster, and developing fully parallel versions of algorithms.
Suffice it to say the most useful thing the cluster did for us was teach our hardware/networking guy about building clusters and parallel programming. And it meant that when we hired more developers we had some dual CPU workstations ready to go after a simple video card upgrade.
If the program will definitely (or at least very likely) be cross platform start with a good cross platform GUI toolkit.
I've already mentioned Qt, though there are plenty of others. This isn't an easy decision, as feature set, maturity, support, stability, cost, and licensing must all be taken into consideration. But considering that huge parts of your application will depend on this decision, parts that might take months to code, a few weeks spent carefully evaluating the alternatives and doing some costs/benefits analysis is worth it.
Whether or not the program will be made cross platform in the future, isolate core logic from interface and other platform dependent code sections, i.e., use a Model-View-Controller (MVC) architecture. Deal with the porting obstacle when (and if) you come to it.
An oldie, but a goodie. When you know exactly what the finished product will look like any design will probably do as long as all the details are thought out, but for the other 99.9% of projects that can change directions several times during development, modular design is crucial. Using a MVC architecture is the fundamental step, but further isolation of components is a good idea if the project is big enough.
Use standard GUI elements. This will help future proof your application against difficult porting issues and prevent developers spending a lot of time coming up with "cute" solutions to simple problems.
This is something we did well for the most part. Standard menus, toolbars and dialogs are generally easy to implement in any toolkit. If your application does get ported to multiple platforms this means it will probably have a different feel on each platform, but it will be that platform's native feel, which many users will appreciate. Some people would prefer that the app is consistent across all platforms (a.k.a. the Mozilla approach), but there is no consensus as to which approach is better, and using native interfaces will be less work except in the largest projects. As long as you keep the distinction between interface and logic clear, porting the interface should be relatively painless.
Create a string mapping table to organize persistent objects. This gives a lot of freedom to change program structure in the future, and allows global access to objects without global namespace pollution.
We did this, and other than using handles as an unnecessary layer of indirection between the string identifier and the object it worked out well. By encoding object hierarchy directly into the string identifiers it was easy to assemble important objects belonging to classes spanning multiple VDP applications in organizations customized to each application without incurring additional class overhead. This also made it easy to reorganize the object hierarchy without much code modification. It should be noted that the benefits of this method of organization would have come virtually for free if this high level application structuring was done in a language with dynamic binding and built in hash table/dictionary types rather than in C++.
Create platform/toolkit abstractions for small, essential components that must necessarily interact with core logic. Threads which must support any level of communication are a prime example, as thread interaction is tightly coupled with algorithm implementation (the C in MVC).
Our thread wrapper class was nice because it was a fairly simple class which allowed a clean way for 100% portable algorithmic code to interface with platform dependent threads. If we ever did port the code to another platform it would have been easily to get basic thread support working, and even for a single platform it kept the messy parts of threading out of the algorithmic code.
I want to repeat that in spite of all these issues we did develop some pretty good software. The problem was not with quality but with development time, and I believe that if we knew then what we know now the project could have been done just as well in half the time, maybe less. While the anecdotes I've provided touch on some fairly specific issues I believe they can be distilled into a few key principles: 1) know the critical goals for the project, and 2) for each decision made ask "is this the most effective way of achieving the project's goals?" If we had asked ourselves this question early and often (and been honest with ourselves in our answers) most of our big, time consuming mistakes could have been avoided.