Why Predictability is Bad and Surprises are Good

“Too many mangers think that the key problem with product development is the surprises. They try to eliminate all variability from the process. By now you should understand that it is the uncertainty that creates the information and the information that crates the value of product development. This means that it is foolish to try to drive out variability from the development process.

“We would propose an alternative solution. We need to create processes that continue to function in a world with variation. Fortunately, there are abundant tools to do this. The primary obstacle to using them is the belief that product development is or should be deterministic. It is time to discard this notion and use the right tools. It is time to recognize that the emperor has no clothes on, and that he never will. We need to treat development as a process with inherent variability [and] approach development process design with the objective of making the process tolerant of variability as a key design objective.”

These are the words of words of product development expert Donald G. Reinertsen in his excellent book, Managing the Design Factory, Free Press, 1997. A co-author Developing Products in Half the Time, John Wiley and Sons, 1991, Second Edition 1997, Reinertsen developed simple economic models for trading off development cost, unit cost, product performance and development delay. In Managing the Design Factory, he sets out to show how to apply the principles of Lean Manufacturing to Product Development, and how not to apply them.

Try-it-fix-it gives better quality faster
“There are two schools of thought as to how we might get to [a] good design. One school holds that we should strive as developers to reduce the error rates. If we keep analyzing the design to minimize the number of errors, we will get a better design on the first try.... The other school of thought says: do it, try it, fix it. This school lacks the moral high ground of the other approach, but is well-grounded in the practical observation of what works for successful companies in the real world.” says Reinertsen in Managing the Design Factory. He goes on to show that a reduced cycle time for iterations produces lower defects in less time. “On the surface, this seems too good to be true. The try-it-fix-it approach is faster and higher quality.” Reinertsen says. However, as long as iterations do not contain significant fixed costs, the try-it-fix-it approach dose in fact produce better quality faster.

The reason for this is that design processes must create information if they are to add value. Creating information involves finding failures, especially unexpected (or low probability) failures. “The fallacy in thinking that high first-pass success optimizes the design process lies in underestimating the importance of information generation that occurs with failure.”

Do It Right The Second Time
Reinertsen notes that once we learn from a failure, it is a waste to learn the same lesson again. So we need to learn how to create failure and how to avoid the same failure a second time.

Screen Beans Art, © A Bit Better Corporation

XP in a Safety-Critical Environment

Recently I chanced to meet a gentleman on a plane who audits the software used in medical and pharmaceutical instruments. During our long and interesting conversation, he cited several instances where defects in software had resulted in deaths. One that comes to mind is a machine which mixed a lethal dosage of radiation [1]. We discussed how such deaths could be prevented, and he was adamant – it is a well-known fact that when developing safety-critical software, all requirements must be documented up front and all code must be traced to the requirements. I asked how one could be sure that the requirements themselves would not cause of a problem. He paused and admitted that indeed, the integrity of the requirements is a critical issue, but one which is difficult to regulate. The best hope is that if a shop is disciplined shop in other areas, it will not make mistakes in documenting requirements.

One of the people this auditor might be checking up on is Ron Morsicato. Ron is a veteran developer who writes software for computers that control how devices respond to people. The device might be a weapon or a medical instrument, but often if Ron’s software goes astray, it can kill people. Last year Ron started using Extreme Programming (XP) for a pharmaceutical instrument, and found it quite compatible with a highly regulated and safety-critical environment. In fact, a representative of a worldwide pharmaceutical customer audited his process. This seasoned auditor concluded that Ron’s development team had implemented practices sufficiently good to be on a par with the expected good practices in the field. This was a strong affirmation of the practices used by the only XP team in the company.

However, Ron’s team did not pass the audit. The auditor was disturbed that the team had been allowed to unilaterally implement XP. He noted that the organization did not have policies concerning which processes must be used, and no process, even one which was quite acceptable, should be independently implemented by a development team.

The message that Ron’s team heard was that they had done an excellent job using XP when audited against a pharmaceutical standard. What their management heard was that the XP process had failed the audit. This company probably won’t be using XP again, which is too bad, because Ron thinks it is an important step forward in designing better safety-critical systems.

The Ying and Yang of Safety-Critical Software
Ron points out that there are two key issues with safety-critical systems. First, you have to understand all the situations in which a hazardous condition might occur. The way to discover all of the safety issues in a system is to get a lot of knowledgeable people in a room and have them imagine scenarios that could lead to a breach of safety. In weapons development programs, there is a Systems Safety Working Group that provides a useful forum for this process. Once a dangerous scenario is identified, it’s relatively easy to build into the system a control that will keep it from happening. The hard part is thinking of everything that could go wrong in the first place. Software rarely causes problems that were anticipated, but the literature is loaded with accounts of accidents whose root causes stem from a completely unexpected set of circumstances. Causes of accidents include not only the physical design of the object, but also its operational practices [2]. Therefore, the most important aspect of software safety is making sure that all operational possibilities are considered.

The second issue with safety is to be sure that once dangerous scenarios are identified and controls are designed to keep them from happening, future changes to the system take this prior knowledge into account. The lethal examples my friend on the airplane cited were cases in which a new programming team was suspected of making a change without realizing that the change defeated a safety control. The point is, once a hazard has been identified, it probably will be contained initially, but it may be forgotten in the future. For this reason, it is felt that all changes must be traced to the initial design and requirements.

Ron has noticed an inherent conflict in these two goals. He is convinced that best way to identify all possible hazard scenarios is to continually refactor the design and re-evaluate the safety issues. Yet the traditional way to avoid forgetting about previously identified failure modes is to freeze the design and trace all code back to the original requirements.

Ron notes that up until now there were two approaches: the ‘ad hoc’ approach and the ‘freeze up front’ approach. The ‘ad hoc’ approach might identify more hazards, but it will not insure that they will continue to be addressed through the product lifecycle. The ‘freeze up front’ approach insures that identified failure modes have controls, but it is not good at finding all the failure modes. Theoretically, a good safety program employs both approaches, but when a new hazard is identified there is a strong impetus to pigeonhole a fix into the current design so as not to disturb the audit trails spawned by policing a static design. XP is a third option – one that is much better at finding all the failure modes, yet can contain the discipline to protect existing controls.

Requirements Traceability
My encounter on the plane told me that those who inspected Ron’s XP process would be looking for traceability of code to requirements. Since his XP processes fared well under review, I wondered how he satisfied inspectors that his code was traceable to requirements. Did he trace code to requirements after it was written?

“Just because you’re doing XP doesn’t mean you abandon good software engineering practices,” Ron says. “It means that you don’t have to pretend that you know everything there is to know about the system in the beginning.” In fact, XP is quite explicit about not writing code until there is a user story calling for it to be written. And Ron points out that the user stories are the requirements.

The important thing about requirements, according to Ron, is that they must reflect the customer’s perspective of how the device will be used. In a DOD contract, requirements stem from a document aptly named the Operational Requirements Document, or ORD. In a medical device development, requirements would be customer scenarios about how the instrument will be used. Sometimes initial requirements are broken down into more detail, but that process results in derived requirements, which are actually the start of the design. When developing safety-critical systems, it is necessary to develop a good understanding of how the device will be used, so derived requirements are not the place to start. The ORD or customer scenarios, along with any derived requirements that have crept in, should be broken down into story cards.

In order to use XP in a safety environment, the customers representatives working on story cards should 1) be aware of the ORD and/or needs of system users and able to distinguish between originating and derived requirements, 2) have a firm understanding of system safety engineering, preferably as a member of the System Safety Working Group, and 3) have the ear and confidence of whatever change authority exits. Using XP practices with this kind of customer team puts in place the framework for a process that maintains a system’s fitness for use during its development, continually reassesses the risk inherent in the system, and facilitates the adaptation of risk reduction measures.

Refactoring
Ron finds that refactoring a design is extremely valuable for discovering failure scenarios in embedded software. It is especially important because you never know how the device will work at the beginning of software development. Ron notes that many new weapons systems will be built with bleeding edge technology, and any new pharmaceutical instrument will be subject to the whims of the marketplace. So things change, and there is no way to get a complete picture of all the failure modes of a device at the beginning of the project. There is a subtler but equally important advantage to refactoring. The quality of a safety control will be improved because of the opportunities to simplify its design and deal with the inevitable design flaws that will be discovered.

“It’s all about feedback. You try something, see how it works, refactor it, improve it.” In fact, the positive assessment of the auditor notwithstanding, if there were one thing that Ron’s team would do differently the next time: they would do more refactoring. Often they knew they should refactor, but forged ahead without doing it. “We did a root-cause analysis of our bugs, and concluded that when we think refactoring might be useful, we should go ahead and do it.”

It is dangerous to think that all the safety issues will be exposed during an initial design, according to Ron Morsicato. It is far better to review safety issues on a regular basis, taking into account what has been learned as development proceeds. Ron is convinced that when the team regularly thinks through failure scenarios, invariably new ones will be discovered as time goes on.

Refactoring activities must be made visible to the business side of the planning game, for it is from there that the impetus to reevaluate the new design from a systems safety aspect needs to occur. Ron believes that if the system safety designers feel that impetus and take on an “XP attitude,” then the benefits of both the “ad hoc” and “freeze” approaches can be realized. The customer-developer team will achieve a safer design by keeping the system as simple as possible, helping them to achieve a heightened focus on the safety of its users.

Testing
The most important XP discipline is unit testing, according to Ron Morsicato. He noted that too many managers ignore the discipline of thorough testing during development, which tends to create a ‘hacker’s’ environment. When presented with a ton of untested code, developers are presented with an impossible task. Random fixes are often applied, the overall design gets lost, and the code base becomes increasingly messy.

Ron feels that no code should be submitted to a developing system unless it is completely unit tested, so the systems debuggers need only look at the interfaces for causes of defects. Instead of emphasizing sequential steps in development and thorough documentation, emphasizing rigorous in-line testing will result in better code. When coupled with effective use of the planning game and regular refactoring, on-going testing is the best way to develop safe software.

The XP testing discipline provides a further benefit for safety-critical systems. By assuring that all safety controls have tests that run every time the system is changed, it is easier to be sure that safety controls cannot be broken as the software undergoes inevitable future changes.

Us vs. Them
I asked Ron what single thing was the most important trait of a good manager. He replied without hesitation, “Managers who give you the big picture of what you are supposed to achieve, rather than telling you what to do, are far and away the best managers. Developers really do not like managers telling them how to do their job, but they don’t appreciate a hacking environment either.” One thing Ron has observed in his experiences is that the increasing pressure on developers to conform to a specific process has created an “us” vs. “them” mentality. The word “process” has become tainted among developers; it means something imposed by people who have lost touch with the realities of code development. Developers accuse ‘them’ of imposing processes because they sound good in theory, and are a foolproof way of passing an auditor’s comparison of the practice to a particular standard. Developers find themselves overloaded with work that they feel they don’t have to do in order to produce good code. The unfortunate consequence of this is that anything said by the “process camp” tends to be disregarded by the “developer camp.” This leads to an unwillingness to adopt a good practice just because the process people support it.

According to Ron, XP is a process that doesn’t feel like a process. It’s presented as a set of practices that directly address the problems developers continually run into from their own perspective. When engaged in any of the XP practices, a developer has a sense of incrementally contributing to the quality of the product. This reinforces developers’ commitment to quality and strengthens their confidence that they are doing the right thing. If I were getting some medical treatment from a device that looked like it could kill me if someone made the wrong move, I’d certainly hope that the engineers who developed the gizmo had that confidence and commitment.

Software developers will deliver high quality code if they clearly understand what quality means to their customer, if they can constantly test their code against that understanding, and if they regularly refactor. Keeping up with change, whether the emerging insights into the system design, the inevitable improvements in device technology, or the evolving customer values, is critical to their success. With good management, they will look upon themselves as members of a safety team immersed in a culture of safety.

The Audit
Ron’s team implemented XP practices with confidence and dedication, met deadlines that would have been impossible with any other approach, and delivered a solid product, while adhering to practices that met pharmaceutical standards. And yet even though these practices were praised by a veteran auditor, the organization failed the audit at the policy level. What went wrong?

The theory of punctuated equilibrium holds that biological species are not likely to change over a long period of time because mutations are usually swamped by the genes of the existing population. If a mutation occurs in an isolated spot away from the main population, it has a greater chance of surviving. This is like saying that it is easier for a strange new fish to grow large in a small pond. Similarly, disruptive technologies [3] (new species of technologies) do not prosper in companies selling similar older technologies, nor are they initially aimed at the markets served by the older technologies. Disruptive technologies are strange little fish, so they only grow big in a small pond.

Ron’s project was being run under a military policy, even though it was a commercial project. If the company policy had segmented off a commercial area for software development and explicitly allowed the team to develop its own process in that segment, then the auditor would have been satisfied. He would not have seen a strange little fish swimming around in a big pond, looking different from all the other fish. Instead he would have seen a new little fish swimming in its own, ‘official’ small pond. There XP practices could have thrived and grown mature, at which time they might have invaded the larger pond of traditional practices.

But it was not to be. The project was canceled, a victim not of the audit but of the economy and a distant corporate merger. Today the thing managers remember about the project is that XP did not pass the audit. The little fish did not survive in the big pond.
__________________
Footnotes:

[1] What he was probably was referring to was the Therac-25 series of accidents, where indeed they had suspect software practices, including after-the-fact requirements traceability.

[2] For a comprehensive account of accidents in software based systems, see Safeware: System Safety and Computers, Nancy G. Leveson, Addison-Wesley, 1995

[3] See The Innovator’s Dilemma, by Clayton M. Christensen, Harper-Business edition, 2000.

Published in Cutter IT Journal – September, 2002 

Screen Beans Art, © A Bit Better Corporation

Lean Contracts

Tool and Die Contracts
The cost of developing dies which stamp out body panels for a new model car can account for half of the model’s capital investment. Consequently, a great deal of time is spent in all automotive companies working to minimize the cost of these dies. The approach in Japan is distinctly different from that in the U.S., and dramatically more effective. The best Japanese companies develop these dies for half the cost and in half the time as their counterparts in the West. The resulting Japanese dies average five shots per panel, while U.S. dies average seven shots per panel, significantly reducing manufacturing costs as well.

From the classic book Product Development Performance by Clark and Fujimoto, Harvard Business School Press, 1991:

Japan firms use an ‘early design, early cut’ approach, while U.S. practice is essentially ‘wait to design, wait to cut.’

Because it entails making resource commitments while the body design is still subject to frequent changes, the Japanese early design, early cut approach entails significant risks of waste and duplication of resources…. Many engineering changes occur after final release of blueprints. At peak, hundreds of changes are ordered per month.

Behind the wait to design, wait to cut approach in U.S. projects is a desire to avoid expensive die rework and scrappage, which we would expect to be an inevitable consequence of the bold overlapping that characterizes the Japanese projects. However, our study revealed a quite different reality. U.S. firms, despite their conservative approach to overlapping, were spending more on engineering changes than Japanese firms. U.S. car makers reported spending as much as 30-50 percent of original die cost on rework due to engineering changes, compared to a 10-20 percent margin allowed for engineering changes by Japanese products.

The Japanese cost advantage comes not from lower wages or lower material prices, but from fundamental differences in the attitudes of designers and tool and die makers toward changes and the way changes are implemented…. In Japan, when a die is expected to exceed its cost target, die engineers and tool makers work to find ways to compensate in other areas…. Die shops in high-performing companies develop know-how techniques for absorbing engineering changes at minimum cost…. In the United States, by contrast, engineering changes have been viewed as profit opportunities by tool makers….

Suppose a body engineer decides to change the design of a panel to strengthen body-shell rigidity. The high performers tend to move quickly. The body designer immediately instructs the die shop to stop cutting the die on the milling machine. Without paperwork or formal approval, the body designer goes directly to the die shop, discusses modifications with the die engineers, checks production feasibility, and makes the agree-upon changes on the spot. Unless the changes are major, decisions are made at the working level. Traditionally, the die shop simply resumes working on the same die. Paperwork is completed after the change has been made and submitted to supervisors for approval. The cost incurred by the change is also negotiated after the fact. The attitude is “change now, negotiate later.”

In companies in which die development takes a long time and changes are expensive, the engineering change process is quite different. Consider the context in which changes occur. In extreme versions of the traditional U.S. system, tool and die makers are selected in a competitive bidding process that treats “outside” tool shops as providers of a commodity service. The relationship with the die maker is managed by the purchasing department, with communication taking place through intermediaries and drawings. The individuals who design the dies and body panels never interact directly whit the people who make the dies.

You would think that tool and die makers in Japan must be a department inside the automotive company. How else could it be possible for a designer to walk into a tool and die shop, stop the milling, make changes, and start up the milling again, leaving approvals and cost negotiations for later? But this is not the case. Tool and die makers are supplier companies in Japan, just as they are in the U.S. The difference lies in the attitudes of the different countries toward supplier contracts.

For Toyota in particular, a supplier is a partner. The basis of this partnership is a target cost for each area of the car. This translates into target costs for all development activities, including dies. Of course, U.S. companies have target costs for each component also, but they tend to impose the cost on the supplier without regard to feasibility. This has a tendency to create a win-loose relationship, leaving the supplier no option but to recoup costs through the change process.

In contrast, Toyota does not impose cost targets on suppliers that it does not know how to meet, and it works with suppliers to help them meet their targets. If something goes wrong and the targets cannot be met, Toyota shares the problem in an equitable manner. In this win-win environment, arms-length exchange of information through written documentation and an extensive change approval processes is unnecessary.

Software Contracts
It can be argued that the emphasis on written requirements coupled with an extensive change approval process comes from a win-loose atmosphere in contracting for software development. Even when development is done inside a company, the influence of traditional requirements management and change approval processes is difficult to avoid.

Let’s face it, traditional project management practices are an outgrowth of the military contracting environment of the 1980’s. In software development today, everyone knows that the waterfall approach is ineffective, but no one seems to know how to escape it. Management tends to value written requirements that do not need human interpretation and rigorous change approval procedures. After all, these seem to be necessary if a contract is to be enforceable. Even for development done without a contract, the same approach is frequently used, despite the fact that it is a sub-optimizing outgrowth of a win-loose contracting environment.

There is a better way, and the results of Toyota’s partnership relationship with their tool and die suppliers indicates that the better way can dramatically improve operating results. Efficient information flow between development teams and users is far too important to be committed exclusively to writing or passed through a third party. A ‘wait to design, wait to code’ approach may seem to offer significant risk reduction, but the truth is quite the opposite. In a environment where change is the norm, it is far more effective to develop techniques and practices which allow changes to be rapidly implemented.

Lean Contracts
There are two ways of looking at a contracting relationship. One view is that a contract is a way for a company to shed responsibility. The other way is for a company to share responsibility. If a company wants to achieve a certain result in a set timeframe for a fixed amount of money, it might consider a contract with a vendor as an method to shed the responsibility for achieving this goal. In theory, the company no longer has to worry about achieving the goal, responsibility has been transferred through a contract.

In practice, if a contract sets up a win-loose situation, the long term result cannot be good for either company. There is rarely a winning party to such deal. A vendor that loses money will not be around to support the system, and often cannot even complete the contract satisfactorily. In fact, a company that makes a practice of negotiating contracts without regard for the vendor’s ability to deliver and make a profit has no one to blame but themselves when the results do not match the agreements.

I used to be responsible for negotiating contracts with suppliers for a division which then re-sold these products at a good profit. I quickly learned that if the contract was not win-win, it was not worth the paper it was written on. We depended on our suppliers over the long run, and they depended on us. The contract was a method to share, not shed responsibility. These contracts were partnership contracts, the kind that Toyota has with its tool and die makers.

Requiring a software vendor – or even an internal organization – to deliver a pre-defined set of functionality over a set timeframe for a fixed price in an environment where stakeholders and technologies change is almost never a win-win situation. It is an attempt to shed responsibility, and the only way the vendor can win is to significantly overcharge or make their profit in the change orders.

Why do we impose such an inefficient contracting system on our companies and our vendors? It doesn’t have to be that way. Look to the way your business deals with other suppliers in its supply chain and you will probably find good models for win-win contracts.

Screen Beans Art, © A Bit Better Corporation

Righteous Contracts

Right"eous a. Doing that which is right; yielding to all their due; just; equitable.
[Webster’s Revised Unabridged Dictionary, 1913]

Righteous contracts. A good name for contracts whose purpose is to assure that the parties act in a just and equitable manner and yield to the other their due. Righteous contracts are those governing investments in specialized assets – assets which are very important to a business, but have no value anywhere else. For example, software developed specifically for a single company is a specialized asset, since it is useful only by the company for which it was developed. Agreements to develop specialized assets create a bilateral monopoly; that is, once the parties start working together, they have little option but to continue working together. This bilateral monopoly provides an ideal environment for opportunistic behavior on the part of both supplier and customer.

Thus the purpose of righteous contracts is to prevent opportunistic behavior, to keep one party from taking advantage of another when market forces are not in a position to do so. In a free market where there are several competent competitors, market forces control opportunism. This works for standard or commodity components, but not for specialized assets.

The traditional way to develop specialized assets has been to keep the work inside a vertical organization, where opportunism is controlled by administration. Inside a company, local optimization would presumably be prevented by someone positioned to arbitrate between departments for the overall good of the enterprise. Vertical integration allows a company to deal with uncertainty and change in a rapid and adaptive manner.

Outsourcing
Recently, however, outsourcing has become common in many companies, for very good reasons. An outside company may have lower labor costs or more specialized experience in an area that is not one of the firms core competencies. The cost of producing a service or asset can be considerably lower in an outside company. Of course, there are transaction costs associated with outsourcing, and the total cost (production costs plus transaction costs) must be lower, or vertical integration would make more sense.

Transaction costs associated with outsourcing include the cost of selecting potential suppliers, negotiating and renegotiating agreements, monitoring and enforcing the agreement, billing and tracking payments. Transaction costs also include inventory and transportation above that needed for vertical integration. In addition, there are risks associated with outsourcing, which may result in additional costs. One cost would be that of diminished communication. For example, development of any kind usually requires intense communication between various technical specialties and target users. If distance or intellectual property issues reduce the communication, it will cost more to develop the asset and the results may suffer as well. In addition, moving a specialized skill outside the company may incur lost opportunity costs.

There are two types of contracts which are used for developing specialized assets – Contracts which are executed before the development is done by the supplier, and contracts which are executed after the supplier does the work. A contract executed before work is done is known as a before-the-fact (or ex ante) contract. There are two types of before-the-fact contracts – fixed price contracts and flexible (time-and-materials) contracts. Once these contracts are executed, they set up a bilateral monopoly, fraught with opportunities for exploitation on one side or the other. Therefore, the purpose of these contract is to set up control systems to prevent exploitation.

A contract executed after work is done is called an after-the-fact (or ex post) contract. Suppose a supplier develops a system that it thinks a customer will find valuable and then tries to sell the system. In this case, control comes after the fact; the supplier makes its own decisions, and it’s reward is based on the results. Of course this is a risky proposition, so the supplier has to hedge its bets. One way to do this is to sell the system to multiple customers, basically making it into a commodity product. But this doesn’t help a company that wants suppliers to develop proprietary components for them. In order to entice suppliers to develop specialized assets prior to a contract, a company usually sets up a sole source or favored source program. If a company treats its favored suppliers well, the suppliers develop confidence that their investments will be rewarded and continue to make investments.

On the surface, after-the-fact contracts may seem implausible for software development, but in fact, they are the best solution for contracting a development project. Moreover, the best kind of development processes to use inside a company are those that mimic after-the-fact contracts. How can this be? The explanation starts by understanding why before-the-fact contracts provide poor governance for development projects.

Fixed-Price Contracts
Let’s examine the most commonly used before-the-fact contract, the fixed price contract. A key motivator for fixed price contracts is the desire of a customer to transfer risk to the supplier. This may work for simple, well-defined problems, but it is inappropriate for wicked problems.[1] If the project is complex or uncertain, a fixed price contract transfers a very high risk to the supplier. If the supplier is not equipped to deal with this risk, it will come back to haunt the customer.

Risk should be born by the party best able to manage it. If a problem is technically complex, then the supplier is most likely to be in a position to manage it. If a problem is uncertain or changing, then the customer is in the best position to manage it. Transferring the risk for such problems to the supplier is not only unfair, it is also unwise. There is no such thing as a win-loose contract. If a supplier is trapped on the wrong side of a win-loose contract, the bilateral monopoly which has been formed will trap the customer as well. Both sides loose in the end.

Fixed price contracts do not usually lower cost, because there is always at least some risk in estimating the cost. If the supplier is competent, it will include this risk in the bid. If the supplier does not understand the complexity of the problem, it is likely to underbid. The process of selecting a supplier for a fixed-price contract has a tendency to favor the most optimistic (or the most desperate) supplier. Consequently, the supplier least likely to understand the project’s complexity is most likely to be selected. Thus fixed price contracts tend to select the supplier most likely to get in trouble.

Therefore it is quite common for the customer find a supplier unable to deliver on a fixed price contract. Because the customer no longer has the option to choose another supplier, they must often come to the rescue of the supplier. Alternatively, the supplier might be able to cover its loss, but most likely it will attempt to make the loss up through change orders which add more revenue to the contract. This leads the customer to aggressively avoid any change to the contract. Faced with no other way to recoup the loss, a supplier will be motivated to find ways to deliver less than the customer really wants, either by lowering the quality or reducing the features.

The customer using fixed price contracts to transfer responsibility and risk will often find both back on their plates in the end, and if so, they will be worse off because of it.

Flexible Contracts
“Customers should prefer flexible-price contracts to fixed-price contracts where it is cheaper for the customer to deal with uncertainty than it is for the contractor to do so or where the customer is more concerned with the ability of the contractor to provide a product that works than with price,” writes Fred Thompson in the Handbook of Public Administration. (Second Edition), Rabin, Hildreth, Miller, editors, New York: Marcel Dekker, Inc., 1998.

The flexible-price contract is designed to deal with uncertainty and complexity, but it does not do away with risk, it simply shifts it from the supplier to the customer. For example, after the DOD (U.S. Department of Defense) experienced some very high profile bailouts on fixed price contracts, it began to use more flexible-price contracts is situations where the government was better able to manage the risk. Of course, with the risk transferred to the customer, the supplier has little incentive to contain costs in a flexible-price contract, a point that did not escape contract negotiators at DOD. In order to protect the public interest, DOD perfected controls imposed on the supplier.

Controlling suppliers of flexible-price contracts evolved into a discipline called project management. The waterfall lifecycle grew out of military contracts, and an early focus of PMI (Project Management Institute) was DOD contracts. Companies with DOD contracts not only hire administrators to oversee compliance with contract requirements, they also add accountants to sort out allowable and unallowable costs. Flexible-price contracts invariably have high transaction costs, due to the high cost of control.

Controls Do Not Add Value
High transaction costs would be reasonable if they added value, but in fact, transaction costs are by definition non-value-adding costs. Fred Thompson (Ibid.) notes, “Controls contribute nothing of positive value; their singular purpose lies in helping us to avoid waste. To the extent that they do what they are supposed to do, they can generate substantial savings. But it must be recognized that controls are themselves very costly.”

One way to avoid the high cost of control in flexible-price contracts is not to use them. It may be better to do development internally, where it is easier to deal with uncertainty and respond to change. The question is, on what basis should an outsourcing decision be made? Thompson (Ibid.) counsels, “The choice of institutional design should depend upon minimizing the sum of production costs and transactions costs.” He also notes, “Vertical integration occurs because it permits transaction or control costs to be minimized.”

An interesting problem with this equation is that vertical integration does not always work to minimize control costs. In fact, many organizations find themselves using DOD-like project management controls internally. It seems incongruous that control mechanisms which add cost but not value, and which were invented to prevent opportunistic behavior, would come to dominate development in the very place where they should not be needed. If the reason to develop internally is to provide flexibility in the face of uncertainty, then costly, change-resistant control practices are inappropriate. Traditional project control practices (that freeze requirements, require approval for changes, and track tasks instead of features) have a tendency to create waste, not value, when used inside a company.

After-the-fact Contracts
Let’s assume for the sake of argument that the choice has been made to outsource a complex, specialized development effort. The next question is, how can transaction costs be reduced? In the manufacturing industry, this is done with after-the-fact contracts.

Despite the obvious risks, is not uncommon for suppliers to develop specialized components for a manufacturer prior to obtaining a contract. For example, 3M Optical Systems Division used to develop optically precise lenses for specialized automotive taillights. The reward was a one year contract for a specific model. Unfortunately, after the first year, the automotive company would invariably find a cheaper way to make a similar lens, and Optical Systems would loose the business before it had recovered its investment. The division eventually decided that after-the-fact contracts with Detroit automakers were not profitable and left the business.

There are ways to make after-the-fact contracts work better. Toyota awards contracts for an entire run of a model, and uses target costing to manage costs. Thus a supplier knows that if it wins the business, it can recover its investment, while the customer is confident that the supplier will work reduce costs in line with its economic requirements. In addition, the supplier understands that it will receive favored consideration for similar components in the future.

After-the-fact contracts require two elements to work: shared risk and trust. Toyota shares the risk with a component supplier by guaranteeing the business over the life of an model. Both parties agree to work together to try to meet a target cost profile over the life of the agreement. Note that meeting future target costs is neither guaranteed nor is it the sole responsibility of the supplier. In the best relationships, technical personnel from each company work freely together without worrying about proprietary information, both to meet target costs and to develop new components not yet subject to a contract.

If both parties are pleased with the results of the first contract, they develop trust and a good working relationship, and are more likely continue to do business together. The supplier is inclined to risk more in developing new components when it has developed confidence that the investment will pay off. This kind of relationship can achieve all of the benefits of both outsourcing and vertical integration combined.

But Software is Different…
You might be saying to yourself, this is fine if there is something to be manufactured and sold many times over, like a taillight, but in software we develop a system only once, it is complex and expensive, it is subject to many changes, and if it is not properly designed and executed, huge waste might result. Where is the parallel to THIS in manufacturing?

Consider the large and expensive metal dies which stamp out vehicle body panels. The cost of developing dies can account for half of a new model’s capital investment. Consequently, a great deal of time is spent in all automotive companies working to minimize the cost of these dies. The approach in Japan is distinctly different from that in the U.S., and dramatically more effective. The best Japanese companies develop stamping dies for half the cost and in half the time as their counterparts in the West. The resulting Japanese dies will be able to stamp out a body panel in 70% of the time needed by U.S. stamping operations.

From the classic book Product Development Performance by Clark and Fujimoto, Harvard Business School Press, 1991:

Japan firms use an ‘early design, early cut’ approach, while U.S. practice is essentially “wait to design,, wait to cut.”

Because it entails making resource commitments while the body design is still subject to frequent changes, the Japanese early design, early cut approach entails significant risks of waste and duplication of resources…. Many engineering changes occur after final release of blueprints. At peak, hundreds of changes are ordered per month.

Behind the wait to design, wait to cut approach in U.S. projects is a desire to avoid expensive die rework and scrappage, which we would expect to be an inevitable consequence of the bold overlapping that characterizes the Japanese projects. However, our study revealed a quite different reality. U.S. firms, despite their conservative approach to overlapping, were spending more on engineering changes than Japanese firms. U.S. car makers reported spending as much as 30-50 percent of original die cost on rework due to engineering changes, compared to a 10-20 percent margin allowed for engineering changes by Japanese products.

The Japanese cost advantage comes not from lower wages or lower material prices, but from fundamental differences in the attitudes of designers and tool and die makers toward changes and the way changes are implemented…. In Japan, when a die is expected to exceed its cost target, die engineers and tool makers work to find ways to compensate in other areas…. Die shops in high-performing companies develop know-how techniques for absorbing engineering changes at minimum cost…. In the United States, by contrast, engineering changes have been viewed as profit opportunities by tool makers….

Suppose a body engineer decides to change the design of a panel to strengthen body-shell rigidity. The high performers tend to move quickly. The body designer immediately instructs the die shop to stop cutting the die on the milling machine. Without paperwork or formal approval, the body designer goes directly to the die shop, discusses modifications with the die engineers, checks production feasibility, and makes the agree-upon changes on the spot. Unless the changes are major, decisions are made at the working level. Traditionally, the die shop simply resumes working on the same die. Paperwork is completed after the change has been made and submitted to supervisors for approval. The cost incurred by the change is also negotiated after the fact. The attitude is “change now, negotiate later.

In companies in which die development takes a long time and changes are expensive, the engineering change process is quite different. Consider the context in which changes occur. In extreme versions of the traditional U.S. system, tool and die makers are selected in a competitive bidding process that treats ‘outside’ tool shops as providers of a commodity service. The relationship with the die maker is managed by the purchasing department, with communication taking place through intermediaries and drawings. The individuals who design the dies and body panels never interact directly whit the people who make the dies.

You would think that tool and die makers in Japan must be a department inside the automotive company. How else could it be possible for a designer to walk into a tool and die shop, stop the milling, make changes, and start up the milling again, leaving approvals and cost negotiations for later? But this is not the case. Tool and die makers are supplier companies in Japan, just as they are in the U.S. The difference lies in the attitudes of the different countries toward supplier contracts.

For Toyota in particular, a supplier is a partner. The basis of this partnership is a target cost for each area of the car. This translates into target costs for all development activities, including dies. Of course, U.S. companies have target costs for each component also, but they tend to impose the cost on the supplier without regard to feasibility. This has a tendency to create a win-loose relationship, leaving the supplier no option but to recoup costs through the change process.

In contrast, Toyota does not impose cost targets on suppliers that it does not know how to meet, and engineers from both companies work together to meet target costs. If something goes wrong and the targets cannot be met, Toyota shares the problem in an equitable manner. In this win-win environment, arms-length exchange of information through written documentation and an extensive change approval processes is unnecessary.

The Toyota Production System is founded on the premise that superior results come from eliminating anything which does not add value. Since control systems do not add value, they must be minimized, just like inventory and set-up times. Therefore supplier partnerships based on shared risk and trust are the preferred relationship. The hallmarks of these partnerships are worker-level responsibility for meeting business goals, intense communication at the technical level, a stop-the-line and fix-it-immediately attitude, and an emphasis on speed. Even for large, one-of-a-kind development projects which require highly specialized design, this approach produces dramatically superior results.

Can this work for Software Development?
Developing specialized dies is not that much different than developing specialized software. The key is to establish a partnership relationship which allows true development to take place. Development is done using a repeated cycle of design-build-test, allowing the solution to emerge. The question is, how can a contract be written to support the emergent nature of development?

Neither fixed-price nor flexible-price contracts support the nature of software development. Development always involves tradeoffs, and an organization which facilitates the best tradeoff decisions will produce the best result. Before-the-fact contracts do not support the give-and-take between developers and customers necessary to make the best tradeoffs. A developer should not have to worry about dealing with problems as they arise, but with before-the-fact contracts, this activity has to be paid for by one company or the other. Since every hour must be accounted for, the give-and-take necessary for trade-off decisions is discouraged.

What is needed is a contract approach which allows developers and customers work closely together to develop a business value for a target cost. Examples of how to do this in a vertical organization abound. There many successful examples of using Scrum for product development. Microsoft’s approach to product development is documented by Michael Cusumano in Microsoft Secrets, Simon and Schuster, 1998. The general approach is to set a clear business goal, fix resources, prioritize features, deliver working software in short cycles, and stop working on features when time runs out. This approach has a track record of delivering systems, even large ones, in a predictable timeframe for a predicable cost.

The question is, how can a contract be written to support the same approach? The answer is to move to after-the-fact contracts in which a supplier is paid for the value of the work they do. It works like this: A customer has a clearly defined business value and a target cost in mind for achieving that value. This target cost includes payments to a supplier for their contributions. The customer comes to an agreement with a supplying partner that the business value and the target cost are achievable, including the target cost for the supplier’s participation. Work proceeds without contractual guarantees that the value will be delivered or the target cost will be achieved, but both partners are committed to meet these goals.

Workers at each company use adaptive processes[2] to develop the system as a single team. They communicate intensely at the developer-customer level to make the necessary tradeoffs to achieve the value within the target cost. As working software is delivered, both supplier and customer work together using velocity charts to monitor development progress. If adjustments to the business value or the target cost structure are required, these become apparent early, when they can be addressed by limiting the feature list or extending the schedule. If this changes the business value or target cost, the parties negotiate an equitable way to share the burden or benefit.

Conclusion
Trusted-based partnerships are the first requirement to make after-the-fact contracts work. Partnerships are necessary to facilitate worker-level responsibility for meeting business goals, intense communication between developers and users to make optimal tradeoffs, daily builds and automated testing to facilitate a fix-it-immediately attitude, and a focus on early delivery of working software to create the feedback system critical to good development.

Companies that develop contracts allowing these values to flourish can expect to produce the same dramatically superior results in software development that these values produce in product development.

Lessons for Outsourcers
If your company outsources software development, consider the following:

1. Fixed Price Contracts
Fixed price contracts are risky. There is both a technical risk that the job can’t be done for the allotted cost, and the very real risk that the selection process favors less knowledgeable suppliers. If you assure that you get a competent supplier, then you can be sure the supplier will add a good margin to the cost to cover their risk. Remember that risk should be born by the party most able to manage it, so if the project is complex and changes are likely, you should assume the risk. If the project is a wicked project, you should not even consider a fixed price contract.

If you are considering a fixed price contract, you are probably interested in transferring all responsibility to your supplier. But remember, if this results in a win-loose situation, you will not win. You are going to be committed to the supplier before the cracks begin to show, and if things go wrong, you will suffer as much, if not more, than your supplier. You may have to bail them out. They will no doubt be looking to make up for loses through change orders, so you will have to control these aggressively. That means if your project is prone to uncertainty or change, you really don’t want a fixed price contract.

And finally, it is much more difficult to get what you really need under a fixed price contract. If you got the low bidder, you probably did not get the supplier most familiar with your domain. If the bid was too low, your supplier will want to cut corners. This may mean less testing, fewer features, a clumsy user interface, out-of-date technology. You are going to need to carefully limit user feedback to control changes and keep the price in line, which will make it more difficult to get what your users really want.

Traditional Control Processes
Traditional project management processes tend to emphasize scope management using requirements traceability and an authorization-based change control system. Typically cost control is provided with some variation of an earned value measurement. The first thing to realize is that all of these techniques are expensive and do not add any value to the resulting software. These practices are meant to control opportunism, and if you are concerned that your supplier might take advantage of you, they might make sense. (But try partnerships first.)

You most likely do not want to be using these practices inside your own company; they get in the way of good software development. It’s pretty well known that an iterative approach to software development, with regular user communication and feedback, is far better than the waterfall approach. However, those pesky project management practices tend to favor waterfalls. It’s a good bet that your project will be subject to change (users change their preferences, technology changes, someone forgot a critical requirement), so you want to be using adaptive processes.

2. Trust-based Partnerships
For starters, all internal development should be based on trust-based partnerships – after all, that’s why you are doing inside development in the first place! If you can’t trust someone in your own company, who can you trust?

The fastest, cheapest way to develop software with a supplier is to let their technical people make decisions based on close interaction with and regular guidance from your users. You get the best results and the happiest users this way too. This kind of relationship requires risk sharing and excellent on-going communications. In exchange for this investment, trust-based partnerships adapt well to change and uncertainty and are most likely to yield faster, better, cheaper results.

Lessons for Contractors
If your company supplies software development, consider the following:

1. Fixed Price Contracts
You owe it to your customers to educate them on the pitfalls of fixed price contracts. Make sure they understand that this will make it more difficult for you to deliver the best business value.

2. Traditional Control Processes
Don’t accept traditional control mechanisms; there are better ways. Instead, use prioritized feature sets, rapid iterations and velocity charts to monitor projects.

Never allow the customer to fix cost, schedule and features simultaneously. Preferably, you want to agree to meet cost, schedule and overall business value targets, and provide a variable feature set. If the detailed feature set is not negotiable, then at least one of the other two must be flexible.

Find out what is REALLY important to your customer in terms of business value and deliver that.

3. Trust-based Partnerships
Your top priority when negotiating the relationship is to assure that your development team will have constant user involvement and feedback. You can negotiate what this means and who represents the user, but if you don’t have access to users or a user proxy, you will have a difficult time delivering business value. And delivering business value must be your main objective.
____________________
Footnotes:

[1] A Wicked Problem is one in which each attempt at creating a solution changes the understanding of the problem. See “Wicked Projects” by Mary Poppendieck, Software Development Magazine, May, 2002, posted on this site under the title “Wicked Problems.”

[2] For a discussion of adaptive processes, see “Wicked Projects” by Mary Poppendieck, Software Development Magazine, May, 2002, posted on this site under the title “Wicked Problems.”

Screen Beans Art, © A Bit Better Corporation

Lean Design

For over a decade, a manufacturing metaphor has been used to bring about improvements in software development practices.  But even the originators of the metaphor recognize that it’s time for a change.  From SEI’s COTS-Based Systems (CBS) Initiative we hear:[1]

“Indeed, to many people software engineering and software process are one and the same thing. An entire industry has emerged to support the adoption of CMM or ISO-9000 models, and process improvement incentives have played a dominant role in defining roles and behavior within software development organizations. The resulting roles and behaviors constitute what we refer to as the process regime.

“The process regime was born of the software crisis at a time when even large software systems were built one line of code at a time. With some logic it established roles and behaviors rooted in a manufacturing metaphor, where software processes are analogous to manufacturing processes, programmers are analogous to assembly-line workers, and the ultimate product is lines of code. When viewed in terms of software manufacturing, improvements in software engineering practice are equated with process improvement, which itself is centered on improving programmer productivity and reducing product defects. Indeed, the manufacturing metaphor is so strong that the term software factory is still used to denote the ideal software development organization.

“The process regime might have proven adequate to meet the software crisis, or at least mitigate its worst effects, but for one thing: the unexpected emergence of the microprocessor and its first (but not last!) offspring, the personal computer (PC). The PC generated overwhelming new demand for software far beyond the  capacity of the conventional software factory to produce.

“The response to the growing gap between supply and demand spawned an impressive range of research efforts to find a technological “silver bullet.” The US government funded several large-scale software research efforts totaling hundreds of millions of dollars with the objective of building software systems “better, faster and cheaper.” While the focused genius of software researchers chipped away at the productivity gap, the chaotic genius of the free market found its own way to meet this demand—through commercial software components.

“The evidence of a burgeoning market in software components is irrefutable and overwhelming. Today it is inconceivable to contemplate building enterprise systems without a substantial amount of the functionality of the system provided by commercial software components such as operating systems, databases, message brokers, Web browsers and servers, spreadsheets, decision aids, transaction monitors, report writers, and system managers.

“As many organizations are discovering, the traditional software factory is ill equipped to build systems that are dominated by commercial software components. The stock and trade of the software factory—control over production variables to achieve predictability and then gradual improvement in quality and productivity—is no longer possible. The software engineer who deals with component-based systems no longer has complete control over how a system is partitioned, the interfaces are between these partitions, or how threads of control are passed or shared among these partitions. Traditional software development processes espoused by the process regime and software factory that assume control over these variables are no longer valid. The process regime has been overthrown, but by what?

“Control has passed from the process regime to the market regime. The market regime consists of component producers and consumers, each behaving, in the aggregate, according to the laws of the marketplace.

“The organizations that have the most difficulty adapting to the component revolution are those that have failed to recognize the shift from the process to the market regime and the loss of control that is attendant in this shift. Or, having recognized the shift, they are at a loss for how to accommodate it.

So it’s official, the manufacturing metaphor for software development improvement is needs to be replaced, but with what?  Let’s look to Lean Thinking for a suggestion.

How Programmers Work
A fundamental principle of Lean Thinking is that the starting point for improvement is to understand, in detail, how people actually do their work.   If we look closely at how software developers spend their time, we see that they do these things in sequence:  {analyze–code–build–test}.   First they figure out how they are going to address a particular problem, then they write code, then they do a build and run the code to see if it indeed solves the problem, and finally, they repeat the cycle.  Many times.  This is how programmers work.

An interesting thing about software development is that this cycle:  {analyze–code–build–test}, occurs both in the large and in the small.  Every large section of software will pass through this cycle (many times), but so will every small section of code.  A developer may go through these steps several times a day, or even, many times per hour.  Generally there is no particular effort, nor any good reason, to get the code exactly right the first time.  Try it, test it, fix it is a far more efficient approach to programming than perfection in the first draft.  Just as writers go through several drafts to create a finished piece of work, so do software developers.

The Wrong Metaphor
Because the software development cycle occurs both in the large and in the small, there have been attempts to divide the software development cycle and give each piece of the cycle to a different person.  So for instance, someone does the analysis, another person does the design, someone else writes code, a clerk does an occasional build, and QC people run tests.  This ‘assembly line’ approach to software development comes from the manufacturing metaphor, and basically, it just doesn’t work.

The reason a manufacturing metaphor does not work for development is because development is not sequential, it is a cycle of discovery. The {analyze–code–build–test} cycle is meant to be repeated, not to happen only once.  Further, as ideas and information move through this cycle, two things must be assured.  First,  information must not be lost through handoffs, and second, feedback from the cycle must be as short as possible.

The manufacturing metaphor violates both of these requirements.  First of all, handing off information in a written format will convey at best half of the information known to those who write the documents.  The tacit knowledge buried in the minds of the writers simply does not make it into written reports.  To make matters worse, writing down information to pass along to the next step in the cycle introduces enormous waste and dramatically delays feedback from one cycle to the next. 

This second point is critically important.  The cycle time for feedback from the test phase of the cycle should be in minutes or hours; a day or two at the outside.  Dividing the development cycle among different functions with written communication between them stretches the cycle out to the point of making feedback difficult, if not impossible.

Some may argue with the premise that software development is best done by using a discovery cycle.  They feel that developers should be able to write code and ‘Get it right the First Time”.  This might make sense in manufacturing, where people make the same thing repeatedly.  Software development, however, is a creative activity.  You would never want a developer to be writing the same code over and over again.  That’s what computers are for.

The Difference Between Designing and Making
Glenn Ballard of the Lean Construction Institute (LCI) sheds some light on this topic in a paper called “Positive vs Negative Iteration in Design”.  He draws a clear distinction between the two activities of designing and making.  He points out, “This is the ancient distinction between thinking and acting, planning and doing.  One operates in the world of thought; the other in the material world.”  Ballard summarizes the difference between designing and making in this manner:


The important thing to notice is that the goals of ‘designing’ and ‘making’ are quite different.  Designing an artifact involves understanding and interpreting the purpose of the artifact.  Making an artifact involves conforming to the requirements expressed in the design, on the assumption that the design accurately realizes the purpose.

A striking difference between designing and making is the fact that variability of outcomes is desirable during design, but not while making.  In fact, design is a process of finding and evaluating multiple solutions to a problem, and if there were no variability, the design process would not be adding much value.  As a corollary, Ballard suggests that iteration creates value in design, while it creates waste (rework) in making.  To put it another way, the slogan “Do it Right the First Time” applies to making something after the design is complete, but it should not be applied to the design process.

In the {analyze–code–build–test} cycle, notice that both analyzing and coding are design work.  There are many ways to create a line of code; individual developers are making decisions every minute they are writing code.  There is no recipe to tell them exactly how to do things.  They are writing the recipe for the computer to follow.  It is not until we get to the ‘build’ stage of the cycle that we find ‘making’ activity.  And indeed, all of the rules of ‘making’ apply to a software build:  No one should break the build, and every build with the same inputs, had better get the same outputs.

Cycles
This brings us to the last step of the software development cycle:  test.  Is testing ‘designing’ or ‘making’ or yet a third element?  In fact, designing tests is a creative activity, often part of the design.  Further, the results of tests are continually fed back into the design to improve it.  So in a very real sense, the test step is ‘designing’, not ‘making’.  Further, the ‘test’ step is what causes the cycle to loop back and repeat, it is what makes development work into a cycle in the first place.  In making, it is not desirable to test and rework; however, in development, repeating the cycle is the essence of doing work.  Development is basically an experimental activity.

There are other well-known cycles that bear mentioning here, and all of them end with a step which causes the cycle to repeat.  Some examples are:

1. The Scientific Method:  {Observe – Create a Theory – Predict from the Theory – Test the Predictions}  Graduate students know this well.


2. The Development Approach:  {Discover – Assemble – Assess} This bears a striking (and not accidental) resemblance to the software development cycle.


3. The Demming Cycle:  {Plan – Do – Check} – Act.  This is a three-step cycle {Plan – Do – Check}, followed by – Act once the cycle yields results.   A more complete definition of the Demming Cycle is: {Identify Root Causes of Problems – Develop and Try a Solution – Measure the Results}  Repeat Until a Solution is Proven, then – Standardize the Solution.  Demming taught that all manufacturing processes should be continually improved using this cycle.

In Search of Another Metaphor
If software developers spend their days in a continual {cycle of design–code–build–test}, we might gain insight if we find other workers who use a similar cycle.  In this quest we might eliminate workers in manufacturing, who are not involved in designing the product they produce.  On the other hand, in Lean Manufacturing, workers are continually involved in redesigning their work processes.  Despite this, it seems that software developers more closely resemble product designers than product makers, because a large portion of software development time involves designing the final product, both in the large and in the small.  But unlike product developers, software developers not only design, but also produce and test their product.

We might compare software developers to the skilled workers in construction, who often do a lot of on-site design before they actually produce work.  An electrician, for instance, must understand the use of the room to locate outlets, and must take framing, HVAC and plumbing into account when routing wires.   Software developers might also be thought of as artists and craftsmen, who routinely extend the design process right into the making process.

Learning Lessons from Metaphors
New Product Development, Skilled Construction Workers, Artists and Craftsmen – as we attempt to learn from these metaphors we must also take care not to go too far, as happened with the manufacturing metaphor.  The careful use of a metaphor involves abstracting to a common base between disciplines, and then applying the abstraction to the new discipline (software development) in a manner appropriate to the way work actually occurs in that discipline.   

Three useful abstractions come immediately to mind as we apply design and development metaphors to software development:

Abstraction 1:  Emphasize ‘Designing’ Values, not ‘Making’ Values
Code should not be expected to “Conform to Requirements” or be “Right the First Time”.  These are ‘making’ values.  Instead, software should be expected to be “Fit for Use” and “Realize the Purpose” of those who will be using it.   Disparaging software changes as ‘rework’ exemplifies the misuse of a ‘making’ value.  Since software development is mostly about designing, not making, the correct value for software development is precisely the opposite.  Iterations are good, not bad.  They lead to a better design.

Ballard states that:  “Designing can be likened to a good conversation, from which everyone leaves with a better understanding than anyone brought with them…  Design development makes successively better approaches on the whole design, likegrinding a gem, until it gets to the desired point….”

Abstraction 2:  Compress the {Design–Code–Build–Test} Cycle Time
Once we recognize that the {design–code–build–test cycle} is the fundamental element of work in software development, then principles of lean thinking suggest that compressing this cycle will generate the best results.  Compressing cycle time makes feedback immediate, and thus allows for a system to rapidly respond to both defects and change.

Based on this hypothesis, we may predict that the effectiveness of Extreme Programming comes from its dramatic compression of the {design–code–build–test cycle}.  Pair programming works to shorten design time because design reviews occur continuously, just as design occurs continuously.  Writing test code before production code radically reduces the time from coding to testing, since tests are run immediately after code is written.   The short feedback loop of the {design–code–build–test cycle} in all agile practices is a key reason why they produce working code very quickly.

Abstraction 3:  Use Lean Design Practices to Reduce Waste
Not all design iteration is good; iterations must add value and lead to convergence.  Many times a design will pass from one function to another, each adding comments and changes, causing more comments and changes, causing another round of comments and changes, in a never-ending cycle.  This kind of iteration does not produce value, and is thus ‘negative iteration’ or ‘waste’.

Ballard suggests the following ‘Lean Design’ techniques to reduce negative iteration, or in other words, obtain design convergence:

1. Design Structure Matrix.   Steven Eppinger’s article “Innovation at the Speed of Information” in the January 2001 issue of Harvard Business Review suggests that design management should focus on information flows, not task completions, to achieve the most effective results. The Design Structure Matrix is a tool that answers the question: “What information do I need from other tasks before I can complete this one?”


2. Cross Functional Teams.  Cross-functional teams which collaborate and solve problems are today’s standard approach for rapid and robust design with all interested parties contributing to decisions.  One thing to remember is to ‘let the team manage the team’.


3. Concurrent Design / Shared Incomplete Information.    Sequential processing results in part from the assumption that only complete information should be shared.  Sharing incomplete information allows concurrent design to take place.  This both shortens the feedback loop and allows others to start earlier on their tasks.


4. Reduced Batch Sizes.   Releasing small batches of work allows downstream work to begin earlier batches and provides for more level staffing.  It also is the best mechanism for finding and fixing problems early, while they are small, rather than after they have multiplied across a large batch.


5. Pull Scheduling.  Ballard notes:  “The Lean Construction Institute recommends producing such a work sequence by having the team responsible for the work being planned to work backwards from a desired goal; i.e., by creating a 'pull schedule'. Doing so avoids incorporation of customary but unnecessary work, and yields tasks defined in terms of what releases work and thus contributes to project completion.”


6. Design Redundancy. When it is necessary to make a design decision in order to proceed, but the task sequencing cannot be structured to avoid future changes, then the best strategy may be to choose a design to handle a range of options, rather than wait for precise quantification.  For example, when I was a young process control engineer, I used to specify all process control computers with maximum memory and disk space, on the theory that you could never have enough.  In construction, when structural loads are not known precisely, the most flexible approach is often to design for maximum load.


7. Deferred Commitment / Least Commitment.  Ballard writes:  “Deferred commitment is a strategy for avoiding premature decisions and for generating greater value in design. It can reduce negative iteration by simply not initiating the iterative loop. A related but more extreme strategy is that of least commitment; i.e., to systematically defer decisions until the last responsible moment; i.e., until the point at which failing to make the decision eliminates an alternative. Knowledge of the lead times required for realizing design alternatives is necessary in order to determine last responsible moment.


8. Shared Range of Acceptable Solutions  / Set-Based Design.   The most rapid approach to arriving at a solution to a design problem is for all parties to share the range of acceptable solutions and look for an overlap.  This is also called set-based design, and is widely credited for speeding up development at Toyota, decreasing the need for communication, and increasing the quality of the final products.

These eight Lean Construction techniques, particularly set-based design, is being tested in construction and expected to result in dramatic improvements in design time (~50%) and construction time (~30%).  In addition, work can be leveled throughout the project, better, more objective decisions are expected.

The following two additional Lean Design techniques are particularly applicable to software development:

9. Frequent Synchronization.  It is widely recognized in software development that daily (or more frequent) builds with automated testing is the best way to build a robust system rapidly.


10. The Simplest ‘Spanning Application’ Possible.  This is a software development technique  particularly good for testing component ensembles and legacy system upgrades. The idea is not to implement module-by-module, but implement a single thread across the entire system, so as to test the interactions of all parts of a system along a narrow path.

____________________

[1] From draft version of Chapter 1 of Building Systems from Commercial Components,  [Addison-Wesley, 2001] by Kurt Wallnau, Scott Hissam, and Robert Seacord; downloaded from SEI COTS-Based Initiative website.


Screen Beans Art, © A Bit Better Corporation

Is Agile Software Development Sustainable?

Agile software development practices are often criticized as being suitable only for small, co-located teams of experts working on modest sized projects. If agile development is truly limited to these perceived boundaries, then it is probably not sustainable. Software development practices must address large projects, multiple geographies, and a general population of developers if they are to become the basis of a thriving new paradigm.

On the other hand, in The Innovator’s Dilemma, [HarperBusiness, 2000], Clayton Christensen notes that all disruptive technologies start by addressing a market which is considered small and “down-market” from existing technologies. So if agile practices are a “disruptive technology” compared to traditional software development processes, then it would be quite in character for them to start by addressing small systems. The questions is, can agile processes grow to address the needs of large systems?

The answer lies in recognizing that the needs of large systems are changing in a way that is best addressed by agile practices. Over the last decade, corporations have tended to use commercially available systems to address more and more information needs. Initially, these systems tended to be monolithic and proprietary, but this is changing. Corporations have found that any monolithic system, even one which is commercially supplied, rapidly becomes a legacy system as new technologies combine with mergers to obsolete any system that cannot adapt to change.

The result has been a tendency for developers of large systems to prefer the use of commercially supplied components, from API’s to Web Services, for a sizeable portion of any application. In Building Systems from Commercial Components, [Addison-Wesley, 2001] Kurt Wallnau and coauthors point out that large systems cannot be built from software components using traditional development processes. The marketplace in which component vendors compete dictates that components are complex, they change frequently, and customers pretty much have to take whatever is offered.

System development in the component marketplace is distinctly different from traditional software development. To begin with, one does not start by capturing user requirements, but by understanding the capabilities of available components. The challenge is to bring the user requirements into line with available software, rather than the other way around. System architecture is generally dictated by the available components and the key architectural task is to create an ‘ensemble’ of components that work well together. Finally, designing a system to be able to deal with unpredictable change is a fundamental skill when working with commercial components. If nothing else is certain, the fact that these components will constantly change can be guaranteed.

The bottom line is that the problems that used to be addressed by traditional software processes have changed, and those processes are no longer up to the task of addressing large project development. Meanwhile, the agile practices being honed in small projects are just the ones needed in the a large project environment which deals with legacy systems and commercial components. So don’t be surprised to see agile practices move up-market, as disruptive technologies always do, and take over the large projects as well.

A quote from a version of Chapter 1 of Building Systems from Commercial Components, [Addison-Wesley, 2001] by Kurt Wallnau, Scott Hissam, and Robert Seacord.

Whatever position one takes regarding the CMM versus ISO-9000 one thing is clear: these software management standards have, for over a decade, established the context for improving the practice of software development. Indeed, to many people software engineering and software process are one and the same thing. An entire industry has emerged to support the adoption of CMM or ISO-9000 models, and process improvement incentives have played a dominant role in defining roles and behavior within software development organizations. The resulting roles and behaviors constitute what we refer to as the process regime.

The process regime… established roles and behaviors rooted in a manufacturing metaphor, where software processes are analogous to manufacturing processes, programmers are analogous to assembly-line workers, and the ultimate product is lines of code. When viewed in terms of software manufacturing, improvements in software engineering practice are equated with process improvement, which itself is centered on improving programmer productivity and reducing product defects. Indeed, the manufacturing metaphor is so strong that the term software factory is still used to denote the ideal software development organization….

Today it is inconceivable to contemplate building enterprise systems without a substantial amount of the functionality of the system provided by commercial software components such as operating systems, databases, message brokers, Web browsers and servers, spreadsheets, decision aids, transaction monitors, report writers, and system managers.

As many organizations are discovering, the traditional software factory is ill equipped to build systems that are dominated by commercial software components. The stock and trade of the software factory—control over production variables to achieve predictability and then gradual improvement in quality and productivity—is no longer possible. The software engineer who deals with component-based systems no longer has complete control over how a system is partitioned, the interfaces are between these partitions, or how threads of control are passed or shared among these partitions. Traditional software development processes espoused by the process regime and software factory that assume control over these variables are no longer valid. The process regime has been overthrown….

Screen Beans Art, © A Bit Better Corporation

Lean Construction

“What are you doing here?”  they asked.

They were construction foremen, superintendents and project managers attending a course in construction planning from the Lean Construction Institute (LCI).  Indeed, what was I doing there?

I started to explain:  “In software development, we are told we should manage our projects like construction projects, where a building is designed at the start, cost and schedule are predictable, and customers get what they expect.”

Silence.  “You’re kidding, right?”  “No, honest, that’s what we’re told.”

Incredulity turns to laughter.  The idea that programmers would want to manage projects like the construction industry strikes my classmates as ludicrous.

They struggle every day with a master schedule which bears little relationship to reality, with materials that should be on site but are not, or materials that need to be stored because they arrived before they were needed.  The never know when the crew that precedes them will be ready to turn an area over to them, so they never know how to staff their crews.  They are plagued constantly by the two biggest forms of construction waste – people waiting for materials and work waiting for people.

Construction might be thought of as a long series of handoffs between trades.  When building a house, there maybe 165 handoffs.  Every handoff introduces an element of variability.  Add to this the variability introduced by weather, staging material, finding tools, sharing equipment, and wide variation from the master schedule is a simple matter of statistics.

In manufacturing, MRP systems are known to produce wide swings in plans when adjusting for small variations in production, which makes them relatively useless for detailed shop floor planning.   For the same reason, the construction industry has found a master schedule to be a relatively useless technique for detailed construction planning.  These are open loop control systems which are being incorrectly applied to planning a variable system, which needs to be controlled by a closed loop system.

The Lean Construction Institute teaches construction superintendents, foremen, and project managers to plan work in three windows:  a phase window of three months or so, a six week look-ahead window which rolls weekly, and a detailed plan for the next week.

The phase plan is the point at which major plans for how work is done will be devised.  The level of pre-fabrication of building elements, the sequence of construction events, and the need for long lead time items are addressed.  The phase is planned backward in a ‘pull’ method; that is, the plan starts from the completed set of work and moves backward to lay out what needs to be done to get there.

Each week a new week rolls off the phase plan and onto the six week (or thereabouts)  look-ahead plan.  Each week the construction superintendents and foremen (called ‘Last Planners’ by LCI) review the look-ahead plan to assure that as work becomes due, the necessary materials are at hand and all pre-work is completed.  Every effort is made to assure that most material ordering and preparation work can be done within the look-ahead window, so keeping an eye on the next six weeks gives adequate time to be sure everything is in place when it comes time to do the work.

Each week the ‘Last Planner’ team commits to the plan for the following week.   The most important thing that they learn in the LCI class is to commit only to what should be done and can be done.  Once the commitment is made, each crew is expected to meet its commitment, and success is the measured in terms of meeting the weekly promises.

The ‘Last Planner’ system taught by LCI greatly reduces variability in construction planning, because it is a closed loop control system.  Work is not planned by the master schedule, but by real people (‘Last Planners’) who make detailed short term plans based on what material is actually available, what the near term weather forecasts say, whether the previous crew can be trusted to be done, what crew members are available for the job, etc.  The reliability of these plans has resulted in enormous productivity gains and a significant reduction in construction site problems.

What Does This Have To Do With Software?
Software developers don’t think of their work as a series of handoffs between trades, because often projects are broken into small segments and a single team is assigned to develop each segment.  In this environment, daily builds and automated tests are often used to integrate new code, assuring that teams do not interfere with each other’s work.

However, there are a lot of handoffs in software development.  Trace the path of requirements as they move from customer to developer, and count the handoffs.

As I sat in the construction class, I was surprised at how much design work goes on during construction.  You might think that once construction drawings and specs are approved, the design would be complete.  You would be wrong.  Here are some typical examples of things that happen every day in construction:

Example One:  “What’s that cloud in the basement drawing?  Oh!  It’s an elevator shaft that hasn’t been specified.  But we are about to pour concrete!  Call the architect!”

Example Two:  “How is this conference room going to be furnished?  The electrician has to decide how to lay out the lighting and where to put the outlets.  Someone needs to let us know where the phone and Ethernet terminations go.  Call the customer.”


Example Three:  “The hospital has hired a new head surgeon and he wants the surgical area to be laid out differently.  He says technology has changed in the two years since the drawings were approved and the layout in the drawings is completely out of date.”

In my construction class, the idea of “freeze the design” meets the same fate as “follow the master schedule.”  Laughter.  Not only is freezing the design impossible, but given the long building times in construction, attempting to do so is sure to make the real customer – the people who move into the building – unhappy with the result.

Instead, design is just another element of the ‘Last Planner’ system.  During the look-ahead period, incomplete designs are identified and arrangements are made to fill in the blanks.  In the same way that lack of materials is the biggest cause of construction delay, lack of requirements is the biggest cause of design delay.  The ‘Last Planner’ system pulls in requirements just as it pulls in materials, and LCI recognizes that designs done as late as possible are often the best.  In fact, they recommend that design decisions be made at the ‘Last Responsible Moment’ even as materials arrive ‘Just in Time’. 

The ‘Last Planner’ system creates commitments between workers on what will happen next week, while looking out six weeks to assure that everything will be in place for work that should happen in the near future.  Most planning systems are directive, but this planning system is collaborative.  In a weekly meeting that lasts less than an hour, the ‘Last Planners’ – foremen, crew chief, superintendents, designers – commit to each other and then make good on those commitments.  The system adapts each week for variations such as delayed material or bad weather or changing customer requirements.

The Problem With Task-based Planning
At a construction site, various trades typically work separately on tasks specified in and coordinated by the master schedule.  But this doesn’t give them any incentive to work together, nor does it provide for much planning on the best way to deliver a feature.  At the LCI class I learned that significant gains can be achieved by looking at a construction project as delivering a set of features, rather than accomplishing a set of isolated tasks.

Case One:  The hallway wall of a prison cell is typically pre-fabricated and set in place, then cement is poured, and much later doors are added.  The problem is, if the cement is just a bit too high, the doors don't close.  In one prison, fully a third of the doors had to be ground to fit. On a recent prison project, the management firm (one of LCI’s best customers) suggested that the wall pre-fabricator add the door to the pre-fabrication process.  This was unheard-of, because it involved the coordination of two trades that did work in very different phases of the project.  Upon investigation, the idea was found to be not only possible, but in the end it saved a large chunk of money.  Better still, the new approach is saving more money on each new project.

Case Two:  In building a parking structure, materials for beams were lifted by the crane onto each floor and assembled in place.  Then the crew assembling the beams was released while the next floor was prepared for beams.  The boom and bust cycle of employment created a problem in retaining good crews to work on the beams.  LCI suggested that the beams be assembled on the ground and lifted into place by the cranes, creating steady work for a smaller crew which would assemble the beams as well as for the crew adding the next floor.  It was tricky to work out the crane’s schedule, because typically it is released to only one crew at a time and the fact that it was used only one third of the time was not immediately apparent.  However, the change was made and site productivity increased dramatically thereafter.

Case Three:  One of LCI’s best customers has moved to feature-based planning from the beginning of a project.  It recently divided a new field house at a university into about a dozen main ‘features’:  practice fields, swimming pool, basketball courts, etc.  Each feature was given an appropriate portion of the budget, and a cross-functional team (including users) is assigned to each feature.  The teams in turn broke their features down into sub-features and decided how to spend their allotted money.  Sometimes they negotiated with other feature teams for more of the budget.  As the building went up, the feature teams made decisions which kept their portion under budget.  The result was a smooth construction cycle with a minimum of changes, and a very satisfied customer.

The lesson for software development is that planning in terms of features rather than tasks can yield tremendous advantages.  The Work Breakdown Structure (WBS) method of project planning, with its emphasis on managing individual tasks, leads to sub-optimized thinking which does not correct itself, even when a more effective approach is begging to be discovered.

The Contract Environment
Greg Howell of the Lean Construction Institute told me that construction companies used to be vertically integrated.  But as time went on, tasks became segregated and associated costs identified, leading construction companies to sub-contract the cost and risk of various activities.  In a sub-contracting environment, Greg pointed out, responsibility is shifted to the sub-contractor.  The project manager assigns work, but is not responsible for it. Integrating across the trades is not really anyone’s job.

Typically there is a design firm and a construction firm under contract for any job; however, this has led to countless disagreements, finger-pointing, and even lawsuits.  Recently a practice called design-build has become popular, where the design firm is also responsible for construction.  This has created an environment where more responsibility lies with a single firm, which often leads to greater speed and productivity.  However, not all design-build arrangements guarantee that work will be coordinated across sub-contractors.  The key, Greg says, is to view construction as an integrated flow, rather than a collection of independent tasks. Companies with this mindset are the ones who come up with the ‘innovative’ ideas in the case studies above.

Lessons For Software Development
If construction projects have predictable costs, schedules and results, there is probably a sizable contingency fund for covering surprises. There is a lot of wasted time and productivity in a typical construction project. This is caused by two things:

1. An open-loop plan which does not address variation and thus magnifies it.
2. Fragmentation of responsibility, giving each trade incentives to optimize their individual performance.

Software developers can avoid the open loop planning problem through short cycle, closed loop planning. Various agile practices recommend adopting an iteration cycle of two to six weeks, augmented with daily planning meetings. The important capability is to deliver code which is fully tested and ready for release on a regular, short cycle. This is the essence of closed loop control in software development.

Sub-optimization is caused by rewarding people based on measurements of performance in a narrow area, rather than rewarding people for achieving broader objectives. The fact is, emphasizing cost and schedule control during software development is a contractor mentality, which tends to de-emphasize the importance of achieving overall business objectives. In practice, contracts which are meant to reduce risk often end up reducing responsibility for achieving the business goals instead.

In summary, every metaphor has its limitations, and the construction metaphor is no exception. Metaphors usually suffer when people have an incomplete understanding of the field upon which the metaphor is based. Digging deeper into construction, for example, we find that master schedules are useless for planning work, contracting practices create islands of optimization, and there are large opportunities for productivity improvement. The feature-based planning and short, closed loop cycles of agile software development are similar to the Lean Construction Institute's practices, which have been the source of significant improvements in the construction industry.

Screen Beans Art, © A Bit Better Corporation