July 2006

Who's Driving This Thing, Anyway?
Or, How Marketing and Engineering Buried the Hatchet
(Warning: Contains a Curse Word)
by Brian Dunning

digg this article | del.icio.us this article

Dramatists may consider the "Eternal Conflict" to be the battle between good and evil, but those in the software business know better. It is the ageless conflict of interest between Marketing and Engineering: Marketing wants a product that can do everything, and Engineering wants a product that can be delivered within practical development limits.

Usually the engineers are cast in the role of "Evil", since their input is inevitably interpreted as lazy foot dragging and a lack of sensitivity to the needs of the customer. This makes the "Good" Marketing folks into quixotic Robin Hoods, obsessed with pleasing the customers at all costs, without regard for their own health or bottom line.

Not that this problem is restricted to companies large enough to have different people or even departments dedicated to these tasks. Even the independent consulting developer must balance his customer's wishes with what can be practically created in the proper sequence. Over promise like our "good" salesman and the customer is bound for disappointment down the road; express too much caution like our "evil" engineer and the customer's disappointment is immediate.

So it's a universal problem - it affects so many projects, not only in software development but in manufacturing, ditch digging, movie production, and space shuttle launching. And if it is so ubiquitous and pervasive a problem, surely there must be a well established solution. There have, without a doubt, been innumerable attempts to find one.

I'm happy to report that there is indeed a solution, but it might not be the fast and easy one you're hoping for. But first let's look at these easy and obvious ways to create harmony.

Ironically, the first effort is usually to apply tyranny. Warm and fuzzy concepts like "working together" and "finding the middle ground" are hardly practical solutions by themselves, especially when the fundamental differences that create the rift genuinely have valid positions.

On one project, we had severe performance problems with a large web application. I had written some specifications for a new feature which the programmers had just completed, and now presented for approval. With each page load, the new feature retrieved a set of records from the database. In light of the performance problems, I felt that the records should have been cached instead. In a fit of frustration, I bellowed "That's just lazy architecture, like you don't even give a damn."

My Engineering manager, placing himself in front of the two programmers, quite accurately countered "No, you wrote lazy specifications."

Rather than get into a pissing contest, I simply asserted my authority and told them to redo it the way I wanted. They did. It took much longer than I wanted, and the cache ultimately grew into a kludge that we all regretted for a long, long time.

This was only one example. Collectively, these experiences effectively illustrated that authoritative decision making, while prompt, is rarely the best way to get the best product built in the most efficient way.

And this brings us to the second most often attempted way to solve the Eternal Conflict, which is just as destined to failure: Decision by committee.

The way it works is this. All the stakeholders are brought in and seated about the round table. Marketing makes its pitch for why the product must include everything, Engineering makes its pitch to explain why the product can't include everything, and everyone else weighs in somewhere in the middle. Usually the sticking points come down to some combination of voting and executive authority to form consensus decisions. Predictably, this means that some features will be officially annointed as indispensable even though they may truly be show stoppers from an Engineering standpoint, and others will be dismissed as expendable when a single engineer could have knocked them out in ten minutes.

Probably the committee method will be reasonably successful more often than not, but even at its best, its batting average is still unacceptable.

Dismayed with the lack of success found with either the Tyranny or Committee methods, I once created a sort of Switzerland/United Nations of a department, "Product Management", dedicated to the handling of the Eternal Conflict. I had two talented product managers and an assistant. The four of us met regularly with all the department heads (always one on one, never in committee), and gradually built a huge database of features. The database included all the pros and cons of each feature, and all the significant Marketing and Engineering points. We acted mainly as filters and censors, deciding what should go into the database and what was noise. We included dependencies. We wrote preliminary business requirements, to which the VP Engineering appended brief development requirements. This allowed us to include estimates of time and cost for each feature in the database. Finally, we developed a scoring system that allowed us to easily see which features were most important and cost the least; in effect a cost/benefit ratio.

Soon we discovered that we were spending an inordinate amount of our time answering questions and explaining prioritizations. So, with no more motivation than a little self-interest, we made the decision that turned out to be among the most significant in our nascent department's petite history: We opened up the system, so that everyone in the company had total access to all features and the status of everything. The result was that everyone understood their pet feature's ramifications, its relation to other features in the pipeline, and why it was prioritized where it was.

The other result was that the product was always being built in the most intelligent sequence. At any given moment, it included the most important features that were feasible up to that point. If it ever became necessary to stop where we were and deliver a version, the product was constantly in a state of being as good as it could be, for the money we'd spent.

It had been my good fortune to independently discover the three tenets of good product development. Following these three rules converts the "conflict between Marketing and Engineering" into "helpful input from both Marketing and Engineering".

1. Prioritization.

I've seen many feature lists and product specifications that have degenerated to the point of simply a disorganized stack of papers, or even just a categorized but unprioritized list. Keep it organized by priority. Use a method similar to that described above to keep your list prioritized by its cost/benefit ratio.

There are certain fundamental features that the product is useless without. These should be prioritized in their own group at the top of the list. Next are features that some stakeholders consider essential. In many cases, these can be broken out separately into different versions of the product serving different market segments.

A corollary of prioritization is to keep a "wish list" for a future version, and to use it liberally. When you have a designated place to drop neat ideas and last-minute clever bells and whistles that are not part of the core functionality, it goes a long way toward preventing your main feature list from becoming swamped, overpopulated, and unmanageable. Keep it lean and clean by exising all the unnecessaries to the wish list.

2. Transparent Project Management.

Good project management means that, among other things, you keep a detailed schedule that shows the timing and current status of every feature or phase. It must be clear at a glance what any undeveloped feature is waiting for, and for a feature that appears disappointingly low on the list of priorities, what the cost/benefit ratio or other rationale that put it there is.

Making your project management transparent - that means building mutual trust by keeping no secrets, and having everything visible to everyone - is the key. This self-motivates Marketing to explore different product choices that will work better in Engineering, and self-motivates Engineering to work as best they can with all stakeholders to ensure that every feature request encounters a minimum number of technical stumbling blocks.

3. Willingness to Kill Your Babies.

Everyone must understand that the product is the priority, separate and distinct from any given feature, no matter how seemingly essential or even just "cool". A product that's on the market now, making money, is a bird in the hand, despite its lack of some feature unknown to the customer. Moreover, that feature can now be developed for the next version, which you can now pay for because you had the foresight to ship the earlier one.

Be handy with your red pen, and get used to the term "redline". Redline any feature that stands in the way of progress. When you do this, it's a good thing, even when it hurts. Remember you always have that wish list, and no good idea ever need be killed for good.

None of the above is easy or trivial to implement, and it will often encounter resistance from parties who don't want procedural change. But once it's in place, the benefits and fruits will be appreciated by all. Marriages are always beautiful, but when it's Marketing and Engineering, it's also extraordinary.