David Stein started this current blog meme. He passed the buck on to Brent Ozar, who shared a horrifying tale of his time in the trenches as a developer. Brent then thoughtfully pointed at me and demanded that I carry on the blog meme torch.
Blog memes are great. They give me an opportunity to pretend to be inventive and creative, take someone else’s great idea and spin it in a centrifuge to extract the good stuff and then pass the detritus along to someone else to deal with. As a career developer turned developer berating apparatus, I have seen my fair share of cobbled together dung heaps. In fact, I would be proud to have someone call some of my solutions a cobbled together dung heap.
Bad Process? Just Add Software
Many years ago I was asked to put together a simple proof of concept application showing code movement using VSS automation. The phrase “simple proof of concept” should have sent me running. Of course, I’m assuming that the me of 5 years ago had any sense in his head.
I’ll back up a minute and let you see the whole situation. This organization had a complex development lifecycle. Developers were frequently developing several releases into the future in a combination of major and minor releases. Hot fixes would go out in production and when the next major release went out the icon be cornflower blue but the hot fix would be nowhere to be found. The hot fix would be hot fixed into production and the icon would be back to alice blue. The situation was, in short, a nightmare. What makes the nightmare worse was that it was a process nightmare: development practices in this organization dictated that multiple releases be developed simultaneously by separate teams.
Think about complex, line of business, software. Important software. Software that is core to your business. Imagine that three separate teams of developers are actively developing new features and fixes at the same time in three separate source trees. This is exactly what was going on at this organization. My job was to automate the problem away with software.
Bad Software? Just Add More Software
This organization was heavily invested in using Visual Source Safe. For those of you young enough to have worked with modern version control systems (such as a team of scribes working around the clock), let me tell you about VSS. VSS exists entirely on the client computer. VSS’s source control mojo operates through a network share using a combination of black magic and pure luck. Like all things that work with luck, it fails. It is safer to print your source code, shred it, and burn the paper shreds that to keep your code in VSS.
Despite the pain of VSS, VSS was the platform. How do you automate a platform when there is no server-side component? You install a VSS client on a server and you begin automating it. Thankfully VSS supplied an API – a poorly documented COM+ API.
I created a proof of concept application that showed code motion through various stages of development in VSS. The client was happy. The client signed a statement of work. We took their money and strode purposefully towards what appeared to be a set of rotating blades.
As the project unfolded it became very apparent that no amount of software could solve this problem. The problem was process, not software. Valiantly, foolishly, 26ishly, I soldiered forward. I worked 14 hour days. I build a cobbled together GUI that attempted to model a software development process that was beyond complicated. The GUI was cumbersome to use and brittle to code. A small change in one line of code meant that 17 other bugs showed up in various parts of the program. There were easily 100,000 lines of code in what should have been a simple management application. Edge cases lead to other edge cases. After one marathon 24 hour programming session (after four 18 hour days) the GUI tool finally worked. The users could configure a process. A process that potentially didn’t work.
Throwing Worse Code After Bad
The actual automation was a supposed to be easy. The users could apply VSS labels and the automated software would, in theory, pick up the changes within 15 minutes. The automated software would pick up the changes on a regular basis but it frequently exploded in a fiery error when confronted with real world problems. When two teams are actively developing against the same code base, code merges don’t go very well.
The process was intrinsically human but the client wanted automation. Being young and stupid, I didn’t know enough to say no. I cobbled together a working knowledge of text parsing and attempted out outfox the people who wrote the VSS Merge Tool. Armed with a fistful of academic papers, articles, and APIs I attempted to build a better merge tool using existing libraries.
At the end of the day, the tool built a log of errors using the unified diff format (which no human can read), collected the log and emailed a copy of it to the developers who committed the conflicting files. Or, it would have had anyone used the software.
Didn’t You Read the Manual
I actually wrote a software manual. It was actually a good manual. Want to know how I know? I gave the manual to the project manager and had him create instructional videos. They worked.
The project sponsor was promoted before the project was completed. The primary users transferred to another department. The new users were too busy to read the manual and expected the software to work easily and obviously. Who wouldn’t, right?
The GUI tool looked like the directory structure in VSS, but it didn’t reflect the changes in VSS, it simply let the users configure how the tool should behave. Sound confusing? It was.
At the end of the day, I had written a complicated manual for a complex piece of software that actually worked. It was a cobbled together solution built out of several hundred thousand lines of buggy .NET 1.1 code sitting on top of a buggy COM+ API intended to solve a buggy process. It’s a miracle that this software abomination wasn’t fueled by the shattered dreams of orphans.
Silent Rage Would Be Better Than This
I said that was the end of the day. It wasn’t. I got one support call about the product about 3 months after it had been delivered, signed off, and paid for. I went back to the client’s location and sat down with the current team to learn their problems. None of them knew how to use the software. No original team members were around who had been involved in the original project. All of the original documentation had been lost in some kind of bit rot tar pit.
My beautiful, cobbled together, duct tape and baling wire solution would never see the light of day. It may have been ugly, but it worked. There’s nothing like feeling the triumph of having saved the day and then having that day get flushed right down the toilet.
Calling Other MacGyvers
Since Brent Ozar stole my default victim, Matt Nowack, I’m going to have to find more victims. I’m going to point the finger of shame at TJay Belt, Grant Fritchey, and Alex Moore.
