OK. I'm a senior software developer/engineer/programmer/artist/whatever, and my last job was senior client/server architect in-house with about a thousand users (well, 6000 if you count the time-clock users), so it isn't too dissimilar. When I came on board, there was some crap code I rewrote. There was a lot that I didn't. You have to know the difference.
Young grassahoppa, you must learn The Tao of Programming.
In order to fix some of the large and nasty problems with the software some large parts have to be redesigned and/or pulled out. How do I do this when I must maintain a degree of stability in the project?
The following may seem insultingly simple, but it is not; even many experienced programmers do not understand it. Your job is not to engineer software. Your job is to facilitate the jobs of the users. Sometimes this means engineering software. Other times it means SCPing a text file once a week. Other times it means doing nothing. It could be worse. You could be working for a shrink-wrapped software company, in which case you would regularly be required to write ugly software for marketing reasons. At least, with a user base that has to take what you give them, you have the freedom to write good code. But that is not your job.
You will at times feel a strong disgust at the code you work with. You will see atonishingly bad programming. You will feel an overwhelming impulse to rewrite large chunks. Resist this temptation. When you are no longer filled with disgust, when you can look at the code with a calm and accepting eye, then and only then should you even consider rewriting it.
Regardless of what you might find wrong with the code, there is one thing that is right. It works. Its value is that it works and has worked. Find a way to respect that value before you do anything.
Also, respect your users. Listen very carefully to what they say, but don't do what they say. You must make a synthesis, finding out what they need, which is often very different from what they think they need. If they say, for example, "This screen takes too long. I have to go through fifty of these things a day. A couple of extra seconds means a lot," do not simply make the screen faster. Find out what they need to do and figure out why they need to go through fifty screens to do it.
The stability is not a degree. The stability is what is important. You must maintain stability, while also maintaining a degree of the design of the code.
Let me tell you a parable. When I was training my replacement for my last job, I gave him our Book of Knowlege, a 250-page document I had been accumulating for about a year. He came upon a section I had written about how to produce spreadsheets using the SYLK format. He came excitedly to me to ask me if we had used any of the Perl modules. I nodded and said that, yes, we had, and it had used OLE. And then Ram had to produce a spreadsheet and spent three days trying to get the old module to work even given changes in OLE. And so, when I needed to produce a complex report and didn't have three days, I just wrote it out in SYLK. It may not be pretty or modern or even good, but it got the job done. He then asked about modules that didn't use OLE. I said, fine, use what you like. But make sure that some poor sucker three years from now isn't going to have an experience like Ram did.
I've already tried to make a large change twice and had to can it because it would have made the software too unstable to be able to do the constant small bug fixes and feature releases.
These are lessons, not problems. Many people have suggested refactoring as a way of making large changes to code. Others have suggested that you understand the code completely. Both are right. However, refactoring only tells you how to do it, not whether or why.
Any change requires testing (sometimes known ironically as Quality Assurance, ironically, because it is more about assuring rather than ensuring quality). Respect your testers. If you don't have any, go out there and get you some. Especially respect the testers who know nothing about the system. In any group of users, there will be some who will be willing to help.
Any major change requires an initial, limited release to a limited set of users and an overlap period where the old system is still entirely in place for backup. You should go out there and watch them use the new system.
I was always taught how to plan a project from scratch, how to prevent it becoming out of control and how to spot the signs of a project that is close to being a disaster. Unfortunately no-one ever explained how to rescue such a disaster.
So, you have been taught all the easy and fun stuff. This is one of the problems with education. In the old apprenticeship system, there was a lot of "wax on, wax off" before the student was taught the theory. Now, you get the theory straight away, but the thing about theory is that it's the same in practice, but only in theory.
I assume that you already know all about the nature of programs, how some small changes require big rewrites, how some large changes require small updates, how cruft accumulates and increases the difficulty and cost of bug fixes. What you haven't learned is that, sometimes, you just have to eat it.
The truth may be out there, but lies are inside your head.--Terry Pratchett