Thursday, 10 January 2002

Wicked Problems

“Our infrastructure is essentially developed.  The easy problems have been solved.  Designing systems today is difficult because there is no consensus on what the problems are, let alone how to resolve them.”   The year is 1973 and the topic is public policy.  In the landmark article ‘Dilemmas in a General Theory of Planning’[1], Horst Rittel and Melvin Webber observed that there are a set of problems that cannot be resolved with traditional analytical approaches.  They labeled such problems ‘Wicked Problems’.

It seems that wicked problems are not unique to public policy.  In a wonderful book published in 1990 called “Wicked Problems, Righteous Solutions,”[2]  Peter DeGrace and Leslie Hulet Stahl pointed out that many of the systems problems facing software developers have all the characteristics of wicked problems.  Judge for yourself.

Wicked Problems
Wicked problems, according to Horst and Webber, have ten characteristics:[1]
  1. There is no definitive formulation of a wicked problem.     Formulating the problem and the solution are essentially the same thing.  Each attempt at creating a solution changes the understanding of the problem.
  2. Wicked problems have no stopping rule.  Since you cannot define the problem, it is difficult to tell when it is resolved.  The problem solving process ends when resources are depleted, stakeholders loose interest or political realities change.
  3. Solutions to wicked problems are not true-or-false but good-or-bad.  Since there are no unambiguous criteria for deciding if the problem is resolved, getting all stakeholders to agree that a resolution is ‘good enough’ can be a challenge.  
  4. There is no immediate and no ultimate test of a solution to a wicked problem.  Solutions to wicked problems generate waves of consequences, and it is impossible to know how all of the consequences will eventually play out.
  5. Every implemented solution to a wicked problem has consequences.  Once the web site is published or the new customer service package goes live, you can’t take back what was on-line or revert to the former customer database.
  6. Wicked problems do not have a well-described set of potential solutions.  Various stakeholders will have differing views of acceptable solutions.  It is a matter of judgment as to when enough potential solutions have emerged and which should be pursued.
  7. Every wicked problem is essentially unique.  There are no ‘classes’ of solutions that can be applied to a specific case.  “Part of the art of dealing with wicked problems is the art of not knowing too early what type of solution to apply.”[1]
  8. Every wicked problem can be considered a symptom of another problem.  A wicked problem is a set of interlocking issues and constraints which change over time, embedded in a dynamic social context.
  9. The causes of a wicked problem can be explained in numerous ways.  There are many stakeholders who will have various and changing ideas about what might be a problem, what might be causing it, and how to resolve it.
  10. The planner (designer) has no right to be wrong.  A scientist is expected to formulate hypothesis, which may or may not be supportable by evidence.  A designer doesn’t have such a luxury, they are expected to get things right.
Recognizing Wicked Problems
What kind of problems are wicked problems?  Here are some examples:
  1. Locating a new freeway or homeless shelter.
  2. Optimizing all the features on a new model car.    
  3. Deciding on the best way to re-engineer a business process.
Wicked problems arise when an organization must deal with something new, with change, and when multiple stakeholders have different ideas about how the change should take place.

How might you identify a wicked problem?   The thing to look for is divergence.  If requirements are volatile, constraints keep changing, stakeholders can’t agree and the target is constantly moving, in all likelihood, you are dealing with a wicked problem.  If considerable time and effort has been spent, but there isn’t much to show for it, there is probably a wicked problem lurking somewhere.   If business process reengineering is involved, there is a good chance of encountering a wicked problem.

Tame Problems
According to Rittel and Webber, the opposite of a wicked problem is a ‘tame’ problem.  Tame problems may be quite complex, but the lend themselves to analysis and solution by known techniques.  A traditional linear processes is sufficient to produce a workable solution to a tame problem in an acceptable period time, and it is clear when a solution has been reached.

It is possible, but not advisable, to pretend that a wicked problem is a tame problem.  This makes it easy to address the well-formulated problem with standard techniques.  In time, however, the wicked problem will surface as changed constraints, volatile requirements, or stakeholder resistance.  If a problem was truly a wicked problem in the first place, treating it like a tame problem before it is actually tamed is a recipe for disaster.

How to Handle Wicked Problems
The most fundamental rule for handling wicked problems is that they must not be treated like tame problems.  To quote Rittel and Webber:  “The classical systems approach … is based on the assumption that a … project can be organized into distinct phases:  ‘understand the problems’, ‘gather information,’ ‘synthesize information…,’ ‘work out solutions’ and the like.  For wicked problems, however, this type of scheme does not work.  One cannot understand the problem without knowing about its context; one cannot meaningfully search for information without the orientation of a solution concept, one cannot first understand, then solve.”[1]

The appropriate way to tackle wicked problems is to discuss them. Consensus emerges through the process of laying out alternative understandings of the problem, competing interests, priorities and constraints. The application of more formal analysis tools is impossible before the problem can be articulated in a concise, agreed upon, well-bounded manner.  In other words, the problem must first be tamed.

Wicked Projects
Wicked projects arise when a project is organized to tackle a wicked problems as if it were a tame problem.  Another candidate for a wicked project is a ‘Death March Project’ (as defined by Yourdon in [3], this means a mission-critical project with less than half the time and resources necessary to do the job).  Think about it.  Death March Projects are often caused when volatile requirements, changing constraints and stakeholder disagreements meet up against immovable deadlines.   They are frequently a symptoms of management's unwillingness to tackle wicked problems.

Failure to recognize wicked projects has given Software Development a bad name.  A 1994 Standish Group Report[4] found, for example, that about a third of software development projects get canceled and half do not meet their original cost projections.  Some have taken this to indicate that the state of software development is in disarray.  However, it can also be read as strong evidence that there are a large number of wicked software development projects out there, trying to address wicked problems with the wrong approach.

A typical management reaction to a failed software development project is to conclude that the organization is immature and to aim for more maturity.  This usually means imposing more requirements documentation, more analysis, more planning and tracking against the plan.  Managers feel that more use of the classic project management processes will avert future disasters.  If the failed project was addressing a tame problem, this approach will probably be beneficial.

However, classic project management practices simply do not work for wicked projects.  In fact, referring again to Rittel and Webber, attempting to baseline requirements and then use an analytical approach to reach a solution is a recipe for disaster with wicked problems.  These problems are resolved through discussion, consensus, iterations, and accepting change as a normal part of the process.

How to Deal with a Wicked Project
Step 1:  Recognize that this is a Wicked Project.
The first step to combating most systemic problems is to recognize them – admit that they are there.  If the project has to satisfy stakeholders who do not agree on fundamental issues surrounding the project, then it is definitely a wicked project.  It’s fine to say that there should be an executive sponsor or that the customers should speak with one voice.  What if they don’t?  What if they pretend to agree, but deep down, they really don’t?   This is a wicked project and everyone would be better off if it was recognized as such.

Step 2:  See if the Project can be Tamed.
In recent Harvard Business Review[5], Robert Herbold gave an excellent example of taming a wicked project.  As new COO of Microsoft in 1994, he realized that he had to impose centralized financial, purchasing and human resource systems on staunchly independent Microsoft business units.  Here’s how he tamed the problem:
  1. Executive Support.  Bill Gates made it clear to all divisions that anyone not reporting results through the new system would be deemed to have no results at all.    Steve Ballmer told balking general managers:  “We define the measures.  You put all your creative juices into growing them.”
  2. Clear problem Definition. All central systems were implemented with off-the-shelf software which dumped data into a data warehouse.  There were few system decisions to be made beyond which package to use and how to configure it.  This was accomplished by a small group of technical and business experts because, according to Herbold, “you can get the job done in the time it takes a cross-functional committee to decide on its goals.”
  3. Separation of Concerns / Reduction of Stakeholders. Business units had web-based access to the data warehouse for each system, and they could get at the data however they wanted to, but they did not have access to the packaged software.  Thus business unit managers had no stake in the packaged software, and where they did have an interest, which was data access, they had a high degree of control and flexibility.  This had the effect of reducing the number of stakeholders for the central system to a very few.
A project is not easily tamed at a low level, although a component-based architecture helps to reduce stakeholders and thus tame the problem.

The main approach for taming a wicked project is to obtain appropriate executive support.  This means that the sponsor must recognize and resolve wicked problems.  If the sponsor cannot resolve conflicts and the customer does not speak with one voice, the problem has not been tamed.

Step 3.  Use Adaptive Processes
Wicked problems are resolved through discussion, consensus, iterations, and accepting change as a normal part of the process.  There is nothing that difficult about an adaptive process.  It has been used successfully in World Class new product development organizations for decades.  Most public policy is developed this way.  Most business policy is developed this way.  Most marketing is done with adaptive processes. All sophisticated process control systems use adaptive control techniques.  

There is nothing wrong with using adaptive processes to solve wicked software development problems.  In fact, it if the problem cannot be tamed, this is the only ‘good’ choice.

There is intense pressure not to use adaptive processes for software development, because they are not considered ‘mature’ or ‘predictable’ or ‘controllable’.  In fact, empirical control techniques are just a successful at controlling processes as defined control techniques, when used for the right kind of problems.  Since classic processes have such a poor track record in controlling wicked projects, it is time to give adaptive processes a chance.  If the project is a wicked one, the chances of success will be significantly improved.

Scrum
Perhaps the best adaptive project management approach for wicked software development projects is Scrum.   This Rugby term was first used by Takeuchi and Nonaka in 1986[6] to describe the way in which world class companies develop new products.  Considering that a new product development project is by its nature a wicked problem, the best new product development processes present a good model for solving other wicked problems.

Ken Schwaber and Mike Beedle describe the use of Scrum for software development projects in their book Agile Software Development with Scurm[7].  The important thing about Scrum is that it forces incremental action which creates a basis for stakeholder dialog and project feedback.  A scrum project collects stakeholder input in a feature list called the Backlog.  Each month, the development team starts at the top of the Backlog and selects as many of the top priority features as they think can develop in a month.  The team is then left alone for a month, when the result is demonstrated to the stakeholders.  This provides a basis for rethinking the backlog features and priorities.  The stakeholders are allowed to modify and reprioritize the backlog, after which the team selects it’s next month’s work from the top of the list .

Scrum provides a way for the development team to make regular progress even if the problem is not well understood.  At the same time, it provides a method for stake holders to discuss the problem and reach consensus.  It provides a way for solutions to emerge, as is necessary for the resolution of wicked problems.

At the same time, the Scrum process provides a high degree of predictability.  Each line on the Backlog is estimated, and the estimates are added to create an overall project completion estimate. After three months, a graph of the Backlog estimate against time is a highly reliable indicator of the project’s progress and estimated completion date.   Features may be added or subtracted from the Backlog monthly to adjust for constraints as well as changing stakeholder interests.  The trend in the Backlog estimate is a reliable indication of whether the project is converging or diverging.

Conclusion
If software development projects have not responded well to traditional project management practices, the fault may lie in the attempt to apply inappropriate methodologies to software development projects.  When projects must deal with conflicts in stakeholder requirements and changes in management constraints, an adaptive process is far more likely to succeed than traditional methodologies.  At minimum, adaptive project management practices will quickly determine if a project will converge on a solution, and thus provides actionable data for project portfolio management.

_____________________________
Footnotes:

1.    Rittel, H., and M. Webber; “Dilemmas in a General Theory of Planning” pp 155-169, Policy Sciences, Vol. 4, Elsevier Scientific Publishing Company, Inc., Amsterdam, 1973.

2.    DeGrace, P., and L. H. Stahl. Wicked Problems, Righteous Solutions. Prentice Hall, Yourdon Press, 1990.

3.    Yourdon, Edward; Death March Projects, the Complete Software Developer's Guide to Surviving ‘Mission Impossible’ Projects.  Prentice Hall, 1999, 1997

4.    The Standish Group.  “Charting the Sea of Information Technology – Chaos.”  The Standish Group International, 1994.

5.    Herbold, Robert J.; “Inside Microsoft:  Balancing Creativity and Discipline”, Harvard Business Review, January 2002.

6.    Takeuchi, H., and I. Nonaka. “The New New Product Development Game” Harvard Business Review, January-February 1986

7.    Schwaber, Ken, and Mike Beedle. Agile Software Development with SCRUM. Prentice Hall, 2001

Screen Beans Art, © A Bit Better Corporation

Tuesday, 18 December 2001

Reflections on Development

Software Development is just that ‘Development’. Writing software involves creating something new. (After all, if developing software were repeatable, a computer could be programmed to do it.)

If we cast around for other uses of the word ‘Development’, we find that it means:
  1. Fund raising for a non-profit corporation

  2. Turning vacant land into a subdivision or shopping center

  3. Commercializing new products

Bypassing definitions (1) and (2) for the moment, let’s consider the third. If we determine good New Product Development practices, might we come up with some good ideas for Software Development? It seems that Software Development might learn a lesson or two from New Product Development.

New Product Development has been around for a long time. Some companies do a great job of it, and we can look at corporate track records and find out which ones do it well. In fact, some companies do New Product Development so well that they are considered benchmarks of excellent Product Development practices. Two companies which clearly qualify for this honor are Toyota and 3M.

Toyota has the fastest product development times in the automotive industry, is a consistent leader in quality, has a large variety or products designed by a lean engineering staff, and has maintained US market share despite a strong yen.

Almost half of all 3M products are only 5 years old, and its innovation machine continues to roll out an astounding number high quality of new products every year. New product development practices at 3M have been studied and imitated for over a decade.

Development methods used by 3M and Toyota may be distilled into to four key practices:
  1. Use Entrepreneurial Teams

  2. Focus on Business Value

  3. Overlap Development Phases

  4. Converge on Solutions

Use Entrepreneurial Teams
3M has a 15% rule: All scientists may use 15% of their time working on any project they wish. In addition, the company expects all new product teams to be led by a ‘Product Champion’. The champion typically has an entrepreneurial vision of a new product, and recruits scientists to use their 15% time to help develop the vision. New product teams can make great progress on company time with no official approval, and once they have developed their idea far enough, they appeal to the company for funding to bring their venture to market.

A new product development team at 3M is cross-functional, collaborative, autonomous and self-organizing. It deals well with ambiguity, accepts change, takes initiative and assumes risks. If the team is making progress toward a new product, it will be left alone.

Teams at Toyota are led by a chief engineer who is expected to understand the market and whose primary job is vehicle system design. The Chief Engineer is totally responsible for vehicle development, similar to the product champion at 3M. This creates an entrepreneurial environment in which the second practice can be accomplished:

Focus on Business Value
3M new product development teams have a wide measure of freedom and extremely challenging goals. But these goals are not cost, schedule and scope. At each review, the key hurdle the team must pass is proving that the proposed product has a good margin potential. One reason why 3M gets hundreds of high margin new products each year is because senior management decision-making focuses teams on this broad strategic goal.

Toyota was the birthplace of Lean Production, originally known as the Toyota Production System (TPS). The fundamental concept of the is to eliminate waste, which means eliminating anything which does not add business value. Toyota was the first company to eliminate sub-optimized measurements and allow overall business value to drive decisions at every level. This tends to knock down barriers between departments and facilitates the third practice:

Overlap Development Phases
At 3M, product development appears to move through a set of phases: concept, feasibility, product and process development, pilot production, scale-up. In fact, with a single cross-functional team handling the product from start to finish, a new product will be manufactured on a pilot line and test marketed as early as possible. Changes will be made to the product spec, to the manufacturing process and to the product design, based on feedback from the manufacturing line and from customers. The major problem with this approach is that when a pilot product is successful, the team may not be able to scale up fast enough to meet demand. It’s a nice problem to have.

Toyota uses compressed overlapping phases to rapidly bring new products to market. Traditional serial engineering in the automotive industry involves moving a design from styling to marketing to body to chassis to manufacturing. At Toyota, concurrent engineering takes place: styling, body, chassis and manufacturing designs are done simultaneously, even as market investigation continues. This works because each function develops multiple options, and the Chief Engineer integrates the product by looking for intersections between feasible regions of functional designs. Which leads to the fourth practice:

Converge on Solutions
Toyota makes more clay models than other companies, freezes the nominal dimensions far later than US manufacturers, and obtains detailed input on process capability at the earliest stages of design. Instead of optimizing from a single starting point, tradeoffs are made by rigorously evaluating sets of alternatives, bringing forward multiple options, making tradeoffs based on data, and deciding as late as possible in the design process.

Similarly, 3M teams would not be expected to know at the beginning of a new product development project exactly what they will commercialize. After all, Art Fry was trying to make a sticky bookmark when he invented Post-it® Notes. When there are inventions to be made and markets to be tested, the product development team tries out many ideas, abandoning the ones that fail and pursuing those that work. Failure is not a problem at 3M – less than half of new product development projects are expected to succeed. But from an abundance of options comes an large number of innovative products, including a regular stream of blockbuster successes.

Conclusion
Software Development, in as much as it is really ‘Development’, might well borrow a few ideas from companies that have mature, high performance development practices. Truly innovative development comes from entrepreneurial teams expected to deliver business value, where all functions work in parallel to converge on an optimal solution.

Screen Beans Art, © A Bit Better Corporation

Tuesday, 1 May 2001

Lean Programming

About the time of the 1980 NBC documentary ‘If Japan Can, Why Can’t We?’, I was the System Manager in a video cassette manufacturing plant, and our management team was asking this question every day.  Our Japanese competition was selling superior products at much lower prices, and we couldn’t figure out how they did it.  We knew we needed to make dramatic changes or close up shop, but we didn’t know what to change.

As far as we could tell, we were doing everything right.  We relied on optimized forecasting methods to determine economic lot sizes, and we used the latest MRP (Manufacturing Requirements Planning) software to launch daily schedules into the plant.  We had a sophisticated computer system that analyzed QC results and process parameters, to pinpoint the causes of defects.

We had some quality problems, and it took a month to fill most orders.  In any given week, we were able to pack out about 60% of the planned line items for the week.  But this was okay, because the other 40% of the week’s packout went into finished goods inventory.  Usually we had plenty of on-hand inventory for shipping standard orders.  Special orders were another matter, however.  The division vice president would often call to expedite special orders for important customers.

We moved in-process video cassettes from one workstation to another on carts, and we had a lot of carts.  There was never enough room to store all the carts at the next workstation, so carts full of inventory would get misplaced.   Sometimes cassettes were stacked on top of carts, and occasionally they would spill on the floor.  Video cassettes piled up in front of testing stations, so whenever a process drifted out of control, it took a while to discover that we were producing marginal product.  We had plenty of rework stations, to be absolutely sure that everything we shipped was good product.

All-in-all, we had about a month’s worth of work-in-process inventory. At the time, we blamed our inability to rapidly fill orders on bad forecasts from marketing. Later we were surprised to learn that the real culprit was our in-process inventory.  Today it is well known that the average shipping time of most supply chains is about the same as the average level of inventory in the supply chain.

Lean Manufacturing
At the end of World War II, Sakichi Toyoda, founder of Toyoda Spinning and Weaving company, dreamed of providing cars for the general public, much like Henry Ford’s dream thirty years earlier.  He chartered Taiichi Ohno to put in place an efficient production system to produce high quality automobiles.  Over the next three decades, Ohno developed the Toyota Production System, now known world-wide as Lean Manufacturing[1].  The foundation Ohno’s system was the absolute elimination of waste.

Ohno studied US manufacturing techniques, and learned a lot from Henry Ford’s pioneer work in assembly line flow.  However, the assembly line produced large lots of identical cars.  Ohno didn’t have the customer base to imitate the US practice of manufacturing in ‘economic’ (ie. large) lot sizes.  He was captivated by US supermarkets, however, where a small quantity of every product was placed on shelves, and as shoppers removed  products, the shelves were rapidly replenished.  He decided to place inventory ‘supermarkets’ throughout his plant, and found that this technique dramatically lowered the ‘waste’ of in-process inventory.  He named these inventory supermarkets ‘kanban’.

Because Ohno was converting a spinning and weaving company to an automobile manufacturer, he already knew how to avoid making bad product.  Founder Toyoda Sakichi had invented an automatic shut-off mechanism that stopped a weaving machine the minute a flaw such as a broken thread was detected.  Ohno moved this concept to car manufacturing, where he insisted that each part be examined immediately after it was processed, and the line stopped immediately if a defect was found.

To maximize product flow, standard work sheets were created, but these were not developed at a desk by engineers.  They were developed on the shop floor by the workers who know the process.  Standard cycle times and kanban shelf space for each item was determined and workflow was leveled.  Production workers were like a relay team, handing off the baton (product) to the next person.  The handoff required 100% quality and tight timing.  If things got delayed, teammates were expected to help each other set up a machine or recover from a malfunction.

Ohno’s aggressive elimination of waste led him to the twin values of rapid product flow and built-in quality.  Over time, Ohno discovered that these two values led to the highest quality, lowest cost, shortest lead time products possible.

Total Quality Management
About the same time, Dr. W. Edwards Deming was teaching Quality Management in Japan.  In fact, the Total Quality Management (TQM) movement cannot be separated from Lean Manufacturing.  Demming’s photo is in the lobby of  Toyota’s headquarters, bigger than the photo of founder Toyoda Sakichi. Demming didn’t find an audience in the US after WW II, because managers at the time thought that poor quality was caused by people who just didn’t want to do a good job.  They didn’t think there was much managers could do to improve quality except exhort employees to do a better job.

Demming’s basic message was that quality is a management responsibility, and poor quality was almost always the result of systems imposed on workers which thwarted people’s desire to do high quality work.  He taught the Japanese managers how to empower production workers to investigate problems and systematically improve processes.  He taught that teamwork and long term, trust-based relationships with suppliers were far better than adversarial relationships.  He emphasized a culture of continuous improvement of both processes and products. 

In the 1980’s, Demming’s fourteen points (See Appendix 1) were studied by virtually every manufacturing manager.  Among these fourteen points are the well known mantra’s:
  • Don’t Inspect Quality In.
  • Constantly Improve the System.
  • Break Down Barriers Between Departments.
But a few of Demming’s fourteen points might seem revolutionary even today, such as:
  • Drive Out Fear.
  • Eliminate Quotas, Numerical Goals and Merit Ratings.
  • Don’t Award Business Based on Price; Minimize Total Cost.
Paradigm Shift
When we first heard about Lean Manufacturing, we thought it was a hoax.  Get rid of safety stock, don’t run machines at full capacity, have suppliers deliver small lots on a daily basis?  This was so counter-intuitive, so against the paradigm of the day, that the Japanese manufacturing techniques were widely discounted.  TQM concepts were more intuitive, but alone they were not enough to lift us out of our dire situation.  Desperate for a change, we decided to give Lean Manufacturing a try, and in the end, it saved our plant.

The critical step in implementing Lean Manufacturing in our plant was a carefully planned changeover from push scheduling to pull scheduling.  We decided that we could not do it part way, we had to switch plant-wide, cold turkey, over a weekend.  We devised a simple simulation which we taught to every one in the plant – managers, shift supervisors, and operators.  Using the simulation, teams of workers designed the layout and flow in their areas, including the kanban cards and rapid changeover methods.  The entire plant held its collective breath as the pull system went into effect, but the workers knew what to do – they had developed the methods themselves.  The first week packout accuracy was 92%, and it got better from there.  We were able to fill special orders in two weeks, so the vice president could stop expediting orders.  In a short time we were down to one week of inventory and could fill any order in the same amount of time.  We had lots of extra space, and quality had never been better.

The most difficult part of implementing Lean Manufacturing was the paradigm shift it required.  Everyone ‘knew’ that large lot sizes were necessary to keep expensive machines running at full capacity.   They also ‘knew’ that machine changeovers took a long time, and every minute a machine was idle its burden rate went up.  In addition, large warehouse inventories were necessary to make sure that when a customer ordered a product it could be shipped immediately.  After all, customers didn’t want to wait the month it took us to produce the product.

One of the reasons why Lean Manufacturing has been so difficult to implement is because people must question established, known truths, and this is not easy.  Another reason is that practices which create local optimization at the expense of the overall system are difficult to recognize, let alone change.  Local optimization points provide attractive points of measurement, and inevitably, what is measured is optimized.

Simple Rules
In a January 2001 article in Harvard Business Review titled ‘Strategy as Simple Rules’, Kathleen Eisenhardt describes how smart companies thrive in a complex business environment by establishing a set of simple rules which define direction without confining it.[2]  She suggests that instead of following complex processes, using simple rules to communicate strategy is the best way to empower people to seize fleeting opportunities in rapidly changing markets.

The 1980’s were a time of profound change in US manufacturing, and the change was guided by a set of simple rules.  Simple rules gave guidance to every level of the organization, and got everyone on the same sheet of music.  They empowered people at all levels of the organization, because the provided guidance for making day-to-day decisions.  With simple rules, work teams were able to continuously improve the processes and products without detailed guidance or complex processes.  

The basic practices of Lean Manufacturing and TQM in the 1980’s might be summed up in these ten simple rules:

   1. Eliminate Waste
   2. Minimize Inventory
   3. Maximize Flow
   4. Pull From Demand
   5. Empower Workers
   6. Meet Customer Requirements
   7. Do it Right the First Time
   8. Abolish Local Optimization
   9. Partner With Suppliers
  10. Create a Culture of Continuous Improvement

These Lean Manufacturing rules have been tested and proven over the last two decades.  They have been adapted to logistics, customer service, health care, finance, and even construction.  The application of the rules may change slightly from one industry to the next, but the underlying principles have stood the test of time in many sectors of the economy.

Lean Programming

Recent work in Agile Methodologies, Adaptive Software Development, and Extreme Programming have in effect applied the simple rules of Lean Manufacturing to software development.  The results, which we call Lean Programming, are as dramatic as the improvements in manufacturing brought on by the Just-in-Time and Total Quality Management movements of the 1980’s.

Lean Rule #1:  Eliminate Waste
The first rule of Lean Programming is:  Eliminate waste.  That is, eliminate anything which does not add value to the final product.  In Lean Manufacturing, waste is identified through a value stream analysis, a process which identifies all activities in the value stream and identifies the specific value they add to the final product. The value analysis process then attempts to find a different, more efficient way to add the same value,

The documents, diagrams, and models produced as part of a software development project are often consumables, aids used to produce the system, but not necessarily a part of the final product.  Once a working system is delivered, the user may care little about the intermediate consumables.  Lean principles suggest that every consumable is a candidate for scrutiny.  The burden is on the artifact to prove not only that it adds value to the final product, but also that it is the most efficient way of achieving that value.

Lean Rule #2:  Minimize Inventory (Minimize Intermediate Artifacts)
In our manufacturing plant, we communicated this message:  Inventory is waste.  Why?  Inventory consumes resources.  Inventory slows down response time.   Inventory hides quality problems.  Inventory gets lost.  Inventory degrades and becomes obsolete.  The ‘benefits’ of inventory are oversold.  The ‘cost’ of inventory almost always outweighs such ‘benefits’.

The inventory of software development is documentation that is not a part of the final program.  As inventory, this documentation should be subject to value analysis.  Take requirements and design documents, for example. How much value do the really add?  How important are they to the final product?  If you compare requirements and design documents to in-process inventory, then it is striking to note that the time it takes to produce these documents probably determines the cycle time of the project.  Just as inventory must be minimized to maximize manufacturing flow, so too requirements and design documents must be kept to a minimum to maximize development flow.

There are many wastes associated with this excess documentation:  The waste of time producing the documents, waste of time reviewing the documents, and the work that goes into change requests and associated evaluations, priority setting, and system changes.  But the biggest waste of all is the waste of building the wrong system if the documentation does not correctly and completely capture the user requirements.

The best approach for minimizing intermediate artifacts is to raise the level of abstraction of documentation.  Instead of a 100 page detailed specification, write a 10 page set of rules and guidelines, and document only the exceptions.  Instead of a few inches of specifics, produce a concise 25 page matrix which summarizes the effort.

We know that users are relatively poor at envisioning the details of a system from most documents, and are even less likely to correctly perceive how it should operate in their environment until they actually use it.  Even if users could predict exactly how the system should operate at the present time, it is unlikely that the way the system is supposed to work months before it is delivered will be exactly the way users need it to work for the rest of its useful life.  All of this must be taken into account when we determine how much value these documents actually add to the final product.

Lean Rule #3:  Maximize Flow (Drive Down Development Time)
During the 1980’s we learned how to make products in hours which used to take days or weeks.  We learned that very rapid product flow resulted in very short cycle times, often one or two orders of magnitude lower than before.  During the 1990’s, e-commerce projects were often able to accomplish in weeks what used to take months or years in the traditional software development world.  Yes, in some sense they cheated.  But the bottom line is, huge amounts of useful software was deployed in the last five years with extremely short cycle times by traditional standards.

In a recent paper titled ‘Reducing Cycle Time,’[3] Dennis Frailey  proposes reducing software development cycle time using the same techniques employed to reduce manufacturing cycle time.  He suggests looking for and reducing accumulations of WIP (Work in Process).  Just as in manufacturing, if WIP is reduced, and the cycle time will be reduced.  To reduce WIP, Frailey recommends using the ‘Small Batch’ principle and the ‘Smooth Flow Principle’, concepts straight from Lean Manufacturing.

Iterative development is basically the application of these principles to programming.  The basic premise of iterative development is that small but complete portions of a system are designed and delivered throughout the development cycle, with each iteration adding an additional set of features.  The cycle time from start to finish of any iteration varies from a couple of weeks to a couple of months, and each iteration engages the entire development process from gathering requirements to acceptance testing.

Lean Rule #4:  Pull from Demand (Decide as Late as Possible)
In our video cassette manufacturing plant, we used to think that it would be ideal if our marketing department could forecast exact market requirements.  A lot of work went into sophisticated forecasting techniques to more accurately predict the future.  Then  one day we realized that we were trying to do the wrong thing.  It would not be ideal if we had a perfect forecast.  Instead, it would be ideal if we could reduce our reliance on forecasts by reducing the system response time so dramatically that the system could respond to change rather than predict it.

In a market where volatile technology requires constant product upgrades, Dell Computer has a huge advantage over it’s keenest competitors because it doesn’t forecast demand, it responds to it by making-to-order in an average of six days.  While Dell holds about six days of inventory, it’s competitors maintain six weeks of inventory.  Dell’s ability to make decisions as late as possible gives Dell a significant competitive advantage in a fast-moving market.

Software development practices which keep requirements flexible as close to system delivery as possible can provide a significant competitive advantage in a changing market.  In a volatile business environment, users are not able forecast their future needs accurately.  Freezing the design early in a software development project is just as speculative as forecasting.  Software systems should be designed to respond to change, not predict it.  In software development, as in building computers, the ability to make decisions as late as possible provides a competitive advantage.

Lean Rule #5: Empower Workers (Decide as Low as Possible)
A basic principle of Lean Manufacturing is to drive decisions down to the lowest possible level, providing both the tools and the authority for people “on the floor” to make decisions. When Toyota took over GM’s manufacturing plant in Fremont, California in 1983, it inherited workers with the worst productivity and absenteeism record in the industry. Those same workers doubled their quality and productivity record in two years. This was accomplished through formation of teams that were trained in work measurement and improvement techniques and expected to develop and continually improve their own work standards and practices.[4]

One of the problems with heavyweight intermediate documentation is that it attempts to make all of the decisions for developers, rather than giving them a set of guidelines.  In general, raising the level of abstraction of intermediate artifacts will give guidance as well as freedom to the developers as they make the detailed design and programming decisions.  It is always better to tell developers what needs to be done, not how to do it.

Developers need to understand the goal of their work and how it fits into the overall flow, what it means to meet customer requirements, and the architectural structure and GUI standards of the system.  They also need to know what they must accomplish, by when, and how to tell when it is complete.  Finally, their work needs to be made visible in short iterative cycles to provide the feedback necessary for continual improvement.

Lean Rule #6:  Meet Customer Requirements (Now and in the Future)
In his 1979 book ‘Quality is Free’, Philip Crosby defines quality as ‘conformance to requirements’.   The Standish Group study of 1994[5] noted that the most common cause of failed projects was missing, incomplete, or incorrect requirements.   The software development world has responded to this risk by amplifying the practice of gathering detailed user requirements and getting user sign-off prior to proceeding with system design.   However, this approach to defining user requirements is deeply flawed.

I  worked on one project in which the customer wanted a complex system delivered in ten months.  Time was of the essence – 10 months or bust.  And yet, being a government agency, the contract required sign-off on an external design document before internal design and coding could begin.  Several users were involved, and they dragged their feet on signing the documents.  Why?  They were concerned that they might approve something which would prove to be a mistake later on.  Since there was no easy way to change things after the design documents were signed, they took two months to approve the design.  An who can blame them?  Their jobs depended on them getting it right.  So half way into a very tight schedule, over two months of time and a lot of paper was wasted producing and getting user sign-off on design documents. 

Instead of encouraging user involvement, user sign-off tends to create an adversarial relationship between the developers and the users.  Users are required to make decisions early in the development process, and are not allowed to change their minds, even when they do not have a full concept of how the system will work or how their business situation may develop in the future.  Users are understandably reluctant to make these commitments, and they will instinctively delay decisions to as late in the process as possible.  Note that this instinct on the part of the users is in line with Lean Rule #4.

The most effective way to accurately capture user requirements is found in the iterative approach to system development.  By developing core features early and obtaining customer feedback in a focus-group demonstration of each iteration, a far more correct definition of customer requirements can be obtained.  In addition, if we accept that the requirements will necessarily change over time, we must start with the essential requirement that the system must be designed to easily adapt to changes over its lifecycle.

Lean Rule #7:  Do it Right The First Time (Incorporate Feedback)
Before Lean Manufacturing arrived at our plant in the early 1980’s, we occasionally had output of marginal quality. We would test to find the good product and rework the bad product. After understanding the “Do it Right the First Time” rule, we closed down rework stations and stopped trying to test quality into the product. Instead, we assured that each component was good at every handoff.  This involved having tests and controls at every point of manufacture to detect a drift toward out-of-spec product and stop production before any bad product was made.

“Do It Right the First Time” did not mean “Freeze the Spec”.  On the contrary, product specs changed constantly, and lean discipline meant being able to flawlessly adapt to changing market conditions.  This was accomplished through a product architecture which facilitated manufacturing change, monitoring techniques which detected errors before they happened, and tests which were designed before manufacturing began.

In 1987 Barry Boehm observed that it costs 100 times more to find and fix a problem after software delivery than to find and fix it in early design phases.[6] This observation and the “Do it Right the First Time” rule have been widely used to justify the overhead of developing a detailed system design before code is written.

The problem lies in the assumption that it is possible to generate a detailed set of documents that correctly define customer requirements, and that those requirements will not change. The fact is that requirements do change, and frequently, over the life of most systems. “Do it Right” has also been misinterpreted to mean “don’t allow changes.” In fact, once we acknowledge that change is a fundamental customer requirement, it becomes clear that what “Do it Right” requires that we provide for change.

If we want to meet customer requirements, and we acknowledge that customers don’t really know what they want at the beginning of development, then we need to incorporate a method of obtaining customer feedback during development.  Instead, most software development practices include a “Change Control Process” which makes it so difficult to respond to user feedback that developers are discouraged from asking for it.  Far from insuring a quality result, these change-resistant practices actually get in the way of “Doing it Right”.

Lean Programming employs two key techniques that make change easy. Just as Lean Manufacturing builds tests into process so as to detect when the process is broken, Lean Programming builds tests into the development process in order to ensure that when changes don’t inadvertently break the code. In fact, the best approach is to write the tests first, and then write the code. An excellent unit and regression testing capability is the best way to encourage change late in the development process.

The second technique for allowing change to happen late in development is refactoring, or improving the design of existing software in a controlled and rapid manner. When refactoring is an accepted practice, early designs can focus on the issue at hand rather than speculate as to what additional design elements will be needed. As the additional features are actually added, refactoring provides a new, simplified design to handle the new reality. When refactoring is a part of the process, we reduce speculation as to what will be needed in the future by making it easy to accommodate the future if and when it becomes the present.

Lean Rule #8: Abolish Local Optimization (Sub-Optimized Measurements are the Enemy)
In the 1980s, the biggest enemy of Lean Manufacturing was often the accounting department. We had big, expensive machines in our plant, and the idea that they should not be run at full capacity was radical, to put it mildly. We compiled daily reports of work-in-process inventory, and the accountants didn’t want these reports abandoned just because there was virtually no WIP to report.

A generation of accountants had to retire before it was “OK” to run machines below their full capacity. Designing machines for rapid changeover rather than highest throughput remains a tough sell even today. After 20 years, Lean Manufacturing is still counter-intuitive to those who lack a broad view of the enterprise.

In this context, let’s examine the role of managing scope in a software development project. Project managers have been trained to focus on managing scope, just as we in manufacturing were trained to focus on maximizing machine productivity. However, Lean Programming is fundamentally driven by time and feedback. In the same way that localized productivity optimization creates a sub-optimized overall process, so too, does focus on managing scope create a sub-optimized overall project management process.

Think about it – holding the scope to exactly what was envisioned at the beginning of a project has little value for the user whose world is changing.  In fact, it adds anxiety and paralyzes decision-making.  It doesn’t add much to the ultimate system, which will be outdated by the time it is delivered. Managing to a scope that is no longer valid wastes a lot of time and takes up a lot of space on issue lists, trade-off negotiations and ultimate fixes to the system to get things right in the end. However, as long as keeping a project within its original scope is a key project management goal, this measurement will continue to be optimized—at the expense of the overall value delivered by the project.

Scope will take care of itself if the domain is well understood and there is a well-crafted, high-level agreement on what the system will do in the domain. Scope will take care of itself if the project is driven in time buckets that are not allowed to slip. Scope will take care of itself if both parties focus on rapid development and solving the user’s problem, and adopt waste-free methods of achieving these goals.

Lean Rule #9: Partner With Suppliers (Use Evolutionary Procurement)
Lean Manufacturing did not remain in the manufacturing plant. Once the idea of partnering with suppliers was combined with an understanding of the value of rapid product flow, Supply Chain Management was born. People began to realize that it took tons of paperwork to move material between companies, and this did not add value to the product. Moreover, the paperwork cost more than one might expect, not to mention the delay in product flow that it caused. Even today, predictions of billions of dollars of savings resulting from business-to-business Web portals are based on cutting the cost of transactions required to move goods between companies.

Supply Chain Management caused companies to take a close look at their business-to-business contracts. All too often, these contracts were focused on keeping the companies from cheating each other. In addition, it was common to pit one vendor against another to assure supply and obtain the lowest cost. Again, Lean Manufacturing changed this paradigm. Deming taught that trusting relationships with single suppliers create an environment that allows optimizing the overall value to both companies.

Throughout the 1980s, companies achieved the highest quality and lowest costs in their supply chains by reducing the number of suppliers and working with the remaining suppliers as partners. The quality and creativity which resulted from collaborating supply chains was demonstrated to far outweigh the apparent (and sub-optimal) benefits that came from competitive bids and rapid turnover of suppliers. Partnering companies helped each other improve product designs and product flows. They linked systems to allow just-in-time movement of goods across several suppliers with little or no paperwork. The long-term advantages of a collaborative supply chain relationships are well documented.

Wise companies realize that traditional software development contract practices generate hidden wastes. As manufacturers discovered in the 1980s, trusted relationships with a limited set of suppliers can yield dramatic advantages. Without the adversarial relationship created by a constant focus on controlling scope and cost, software development vendors can focus on providing the best possible software for customers, fixing requirements as late as possible in the development process and providing the most value for the available money.

Lean Rule #10: Create a Culture of Continuous Improvement
When software development seems to be out of control, one response has been to increase the level of “software maturity” of the organization. This might seem to be in line with good manufacturing practice, where ISO 9000 certification and Malcom Baldridge awards are sometimes equated with excellence. However, these process documentation programs indicate excellence only when the documented process is excellent in the context of it’s use.

In many current software development projects, excellence means the ability to adapt to fast moving, rapidly changing environments.  Process-intensive approaches such as the higher levels of Software Engineering Institute's (SEI) Capability Maturity Model (CMM) may lack the flexibility to respond rapidly to change.  In a recent e-mail advisor from Cutter Consortium, Jim Highsmith highlights the tension between such heavyweight methodologies and lightweight methodologies such as Lean Programming.[7]

The question becomes, do process documentation certification programs stifle, rather than foster, a culture of continuous improvement?  Deming would probably turn over in his grave at the thought of tomes of written processes substituting for his simple Plan-Do-Check-Act approach:
  • Plan:     Choose a problem. Analyze it to find a probable cause.
  • Do:        Run an experiment to investigate the probable cause.
  • Check:   Analyze the data from the experiment to validate the cause.
  • Act:     Refine and standardize based on the results.
Iterative development allows the use of the Plan-Do-Check-Act approach within a project. During the first iteration, the hand-off from design to programming or programming to testing may be a bit rough. It’s okay if the first iteration provides a learning experience for the project team, because there are more iterations to come, so the team can improve its process. In a sense, an iterative project environment becomes an operational environment, because processes are repeated and Deming’s techniques of process improvement can be applied from one iteration to the next.

Product improvement is also possible with iterations, particularly if refactoring is used. In fact, refactoring provides a tremendous vehicle to apply the principle of continuous improvement to the programming environment.

However, we need improvements that span more than a single project. We must improve future project performance by learning from existing ones. Here again, Lean Manufacturing can point the way. During the 1980s, a set of practices summarized in the ten rules of Lean Manufacturing were adopted widely across most manufacturing plants in the West. These practices then spread to service organizations, to logistics organizations, to supply chains, and beyond. They have withstood the test of time across multiple domains.

Following the simple rules of Lean Manufacturing has brought dramatic improvements to every industry in which they have been applied. These same rules can and should be applied to software development projects. The resulting Lean Programming practices will lead to the highest quality, lowest cost, shortest lead time software development possible.
_______________

Appendix 1:  Summary of W. Edwards Demming’s 14 points

   1. Create consistency of purpose.
   2. Adopt a win-win philosophy.
   3. Don’t depend on mass inspection; build quality in.
   4. Don’t award business based on price; minimize total cost; build long-term relationships of loyalty and trust with a single suppliers.
   5. Constantly improve the system of production, service, planning, etc.
   6. Train for skills.
   7. Provide leadership:  help people do a better job.
   8. Drive out fear and build trust so everyone can do a better job.
   9. Break down barriers between departments; abolish competition and build a win-win system of cooperation.
  10. Eliminate slogans, exhortations and zero defect targets; the cause of the bulk of problems lie in the system, and are beyond the power of workers to correct.
  11. Eliminate quotas, numerical goals and Management by Objectives; substitute leadership.
  12. Remove barriers that rob people of joy in their work; abolish the annual rating or merit system.
  13. Educate and improve individuals.
  14. Involve the entire organization.

There are many summaries of Demming’s 14 points, which he modified throughout the years, in the spirit of continuous improvement.  The above summary is based on his last version of the 14 points.[8].
_________________

References

[1] The Machine That Changed the World : The Story of Lean Production, by Womack, James P., Daniel T. Jones, and Daniel Roos, New York: Rawson and Associates; 1990

[2] Strategy as Simple Rules, by Eisenhardt, Kathleen M and Donald N. Sull, Harvard Business Review, Volume 79, Number 1, January 2001, pp 107- 116

[3] Reducing Cycle Time, by Frailey, Dennis, Software Development Magazine, August, 2000

[4] Time-and-Motion Regained, by Paul Adler, Harvard Business Review, January-February 1993 pp 97-108

[5]Charting the Seas of Information Technology – Chaos, by The Standish Group International, 1994

[6] Industrial Software Metrics Top 10 List, by Boehm, Barry, IEEE Software, Volume 4 Number 5, September, 1987, pp 84-85

[7] E-Projects in India, by Jim Highsmith, e-Project Advisor, Cutter Consortium's e-Project Management Advisory Service, March 1, 2001.

[8] Gone But Never Forgotten, by Brad Stratton, editor, Quality Progress Magazine, March 1994
______________

Annotated Bibliography (Chronological)

Quality Control Handbook, Joseph M. Juran, originally published in 1951, now in its forth edition
Considered the standard reference in the field of quality. Like Demming, Juran consulted mainly in Japan during the 1950’s and 60’s.
Managerial Breakthrough, Joseph M. Juran, originally published in 1964
Widely ignored when it was first published, this book is now considered a landmark treatise on continuous improvement.
Quality is Free, by Philip Crosby, New York:  McGraw-Hill, Inc. 1979
This is the book we used to launch the TQM program in our plant.  Demming never liked the zero defects approach advocated in this book, and it contains little about statistical quality control.  However, many corporate executives got the quality message from Phil Corsby.
Study of ‘Toyota’ Production System from Industrial Engineering Viewpoint, by Shigeo Shingo, Osaka, Japan, Shinsei Printing Co. Ltd.  1981
The title of this book is an indication of the quality of the translation, but that was not a barrier to our plant.  We studied the book cover-to-cover and got our introduction to Lean Manufacturing from this book.
Toyota Production System, Practical Approach to Production Management, by Yasuhiro Monden, Norcross, Georgia, Industrial Engineering and Management Press.  1983
This is the classic book on Lean Manufacturing.
Zero Inventories, by Robert W. Hall, Homewood, IL, Dow Jones-Irwin 1983.
Robert Hall, a professor at Indiana University, understood the concept of Just-in-Time earlier than most academics.  This book is still considered the definitive work on JIT.
A Revolution in Manufacturing, the SMED System, by Shigeo Shingo, Cambridge, MA, Productivity, Inc.; Originally published as Shinguru Dandori in 1983, English translation 1985.
The title of this book is an indication of the quality of the translation, but that was not a barrier to our plant.  We studied the book cover-to-cover and got our introduction to Lean Manufacturing from this book.
The Goal, by Eliyahu M. Goldratt, First Edition Published in 1984, Second Revised Edition Published in Great Barrington, MA, 1992
This book is a business novel. It is the easiest book to read for an introduction to Lean Manufacturing and is definitely a classic.  You will find this book on the reading list of most Operations Management courses.  Goldratt went on to develop the ‘Theory of Constraints’ and write several related business novels.
Out of the Crisis, by W. Edwards Demming; 1986
This is the book in which Demming outlined his famous 14 points.
The Demming Management Method, by Mary Walton and W. Edwards Demming; 1988
Probably the best book there is on Demming and his management approach.
Toyota Production System – Beyond Large Scale Production, Taiichi Ohno, published in Japanese in 1978 and in English in 1988 by Productivity, Inc.
This is an explanation of JIT by the inventor. The English version arrived after JIT had become widespread in the US.
The Machine That Changed the World : The Story of Lean Production, by Womack, James P., Daniel T. Jones, and Daniel Roos, New York: Rawson and Associates; 1990
This landmark book about the Toyota Production System was the first to associate the term ‘Lean’ with manufacturing.
Lean Thinking, by Womack, James P., and Daniel T. Jones, Simon & Schuster; 1996
A follow-up on the story of Lean Production, this book extends the concept of Lean throughout the enterprise.  It shows how the "lean principles" of value (as defined by the customer), value stream, flow, pull, and perfection can be applied to all areas of the enterprise.  (Software development is implicitly included.)


Screen Beans Art, © A Bit Better Corporation

Monday, 26 March 2001

Component-Based Software Development



Around 1800, Eli Whitney proposed manufacturing rifles with interchangeable parts, instead of crafting each rifle individually. Widely regarded at the beginning of mass production, the concept of interchangeable parts led to a dramatic increase in rifle production capacity while delivering the additional benefits of consistent operation and easy field maintenance of weapons.

Component-based systems represent a paradigm shift in software development similar to that of using interchangeable parts in manufacturing. A component-based system is built of standard, reusable parts that become the fundamental building blocks of future software. Component-based systems promise numerous benefits, including flexibility, scalability, and maintainability.

The promise of component-based systems has been difficult to realize. The reality is that even through the last decade, monolithic systems have dominated corporate IT. And after a few years, monolithic system become legacy systems, because they are not flexible, changeable, or easy to connect to other systems. A company will get acquired or acquire other companies or spin off divisions, and all of a sudden, the monolithic systems get in the way of progress.

During the 1990's more software by far was developed for the Internet that was developed for all the corporations in the world. But Internet software was typically developed vary rapidly, with expediency taking over where architecture once reigned. And guess what - Internet software is largely component-based. It may seem like heresy to 'true' software developers, but an Internet startup could get a shopping cart and a search module and a merchant account and a product display package from other Internet startups. There is Mapping software and travel planning software and collaborative filtering software, and when you look at it closely, these are true components. Web sites can be rapidly manufactured with 85% components, 15% glue.

Now it's time to bring components to the corporate world, and they are headed our way under the name of Web Services. Here's a quote from the January 3, 2002 CBDi forum newsletter:
Web services provide formalized separation of concerns at a usable level of abstraction and you can implement web services right now with most of the current versions of platforms and software tools. Web services are fundamentally a low-cost technology investment that easily extend what you already have to provide real business value. First look for internal applications, there are many opportunities to simplify internal integration. Second look externally at public (non secure) or private B2B applications, and make sure those security and privacy budgets address the needs of Web services. Look at the Web services that IBM and Microsoft’s customers are already delivering.

Theory of Constraints

Two decades ago, Japanese car-makers developed several new manufacturing methods, including 'Just-in-Time'. Based on the theory that most in-process inventory spends most of its time in a buffer, waiting to be processed, Just-in-Time scheduling dramatically reduced inventory, which coincidentally reduced the amount of time it took product to move through a manufacturing plant.

In the 1980's, Dr. Eli Goldratt publicized the Theory of Constraints, which improved upon Just-in-Time methodologies, and in the 1990's, Goldratt adapted the Theory of Constraints to project management in his book "Critical Chain". Based on the idea that individual estimates of project time must be padded to allow for contingencies, Goldratt demonstrated that padded project time operates in the same manner as inventory buffers to delay projects.

It's Okay to be Late
The fundamental concept behind the Theory of Constraints is that elements of a project must be scheduled based on the average time they will take to accomplish, not the maximum time they might take to accomplish. A project buffer is placed at the end of the project, which allows for some activities to be late.

Of course, it's human nature to estimate time based on the worst case scenario, not average time to complete. As soon as people are penalized for estimating average time and then not meeting the estimate, they will revert to estimating the maximum time, and project schedules will again fill up with padded time estimates. Thus, in a project using Theory of Constraints, it is important that it is 'okay' to be late.

It may be hard to buy at first, but even though it's okay for individual elements to be late, Theory of Constraints project management makes it far easier to assure that the overall project is completed on time.

Last Planner

The Lean Construction Institute has developed an effective project management technique for construction called 'Last Planner System'. This is a deceptively simple system which involves having foremen (last planners) planning activities for the immediate future, using only two rules:
  1. If it can't be done, then don't plan to do it.
  2. If you plan to do it, then get it done.
These two simple rules, along with a similar set of rules for the period after the immediate future, have led to enormous productivity increases in construction. For more on this visit the Lean Construction Institute.

Screen Beans Art, © A Bit Better Corporation

Tuesday, 9 May 2000

The Impact of Logistics Innovations on Project Management

A project is a “temporary endeavor undertaken to create a unique product or service” (Project Management Institute, 1996). Projects include building a structure, developing a product, and executing a contract. Logistics “plans, implements, and controls the efficient, effective flow and storage of goods, services, and related information from the point of origin to the point of consumption” (Council of Logistics Management, 2000). Logistics is usually thought of in connection with military or manufacturing operations. However, there can be projects involved in logistics (eg. building a bridge to move troops) and logistics involved in projects (eg. supplying material to a construction site). It is not surprising, then, that these two disciplines interact and learn from each other. In this paper, great events in logistics are examined to uncover their impact on project management.

Great Logistics Feats of Antiquity
In May of 218 BC, 29 year Hannibal led about 40,000 troops, thousands of horses, and 38 elephants over the Pyrenees and Alps from Spain to Italy, a feat which had been considered impossible. During this 15 day ‘project’, his troops built jetties and rafts for the elephants to cross the Rhone River, raided towns for provisions, and struggled through avalanche-blocked mountain passes while under attack. Many of the troops and most of the elephants were lost, but even in this weakened conditioned, Hannibal defeated the Roman army that awaited his arrival. (Encyclopedia Britannica: Hannibal)

Hannibal has been singled out as a forerunner of grand strategy in military campaigns. His superb logistics while crossing the Alps is a precursor of military logistics today, a discipline in which the US military is considered the world leader. The routine logistics of military operations are interspersed with unique, temporary ‘projects’, of which Hannibal’s crossing is a prime example. The project management challenges of crossing the Alps included fixed resources, limited time, physical obstacles and troops with mixed allegiances who were ill-suited to the task. The techniques Hannibal used to meet these challenges (planning, field engineering, rapid movement, periodic re-supply, and attentiveness to troops and animals) are standard military practice today. (Encyclopedia Britannica: Strategy)

There are other great feats of antiquity which are forerunners of today’s project management practices. Building the pyramids comes immediately to mind. The Great Pyramid in Egypt was built around 2500 BC. It took about two decades and somewhere between 20,000 and 100,000 laborers to erect. Stone was cut with crude hand tools, transported without wheeled vehicles, and put into place without lifting machinery. This “masterpiece of technical skill and engineering ability” remains “perhaps the most colossal single building ever erected on the planet”. No taller building was built until the 19th century. (Encyclopedia Britannica: Giza, Pyramids of; Building Construction)

Construction projects today are not significantly different than those which built the pyramids. An architect designed the building, materials were obtained from quarries and transported to the site, experts in various ‘trades’ prepared the materials and constructed the pyramid. Built to a fixed plan, time and resources were more or less unlimited for the pyramids (unlike today).

There is another impressive feat that has been executed from the most ancient of times and continues to this day: The Event. Perhaps it is a wedding or a funeral or a coming-of-age ceremony – just about any large gathering has always required serving a lot of food to a lot of people in a very short time. Perhaps several clans gathered once a year, and there was a woman who was particularly good at organizing such events. Planning, gathering food, attending to living quarters, cooking, serving, might be a tradition or a ritual, but the event always came off best if someone behind the scenes was coordinating it all. In some sense, the most seasoned project managers throughout various cultures and times might be the event organizers who really knew how to throw a good party on a inflexible schedule with limited resources.

From Logistics to Project Management
All of the knowledge areas of project management were are developed in these three examples of logistics from antiquity: staging a celebration, managing construction, and executing a military movement. A celebration required excellent procurement, scheduling, quality and resource management. Construction projects required superior planning, managing a fixed scope over a long time, procurement and human resource management. Military movements needed excellent execution, careful management of fixed resources, precise timing, superior human resource management and continual risk assessment.

Viewed from a different perspective, advances in logistics might be thought of as starting with project management. For instance, military logistics may start with unique projects, like ferrying elephants over a river, but once the technique is developed, it can be reused to ferry other large items over a river. Feeding a large number of relatives may be a project the first time it is done, but feeding hundreds of people a sit-down dinner in fifteen minutes is routine logistics at conventions held every day. Even the pyramids were more or less standard burial structures for a period of over 500 years. (Encyclopedia Britannica: Pyramid)

The disciplines of logistics and project management overlap in such a way that it is not always possible, or even necessary to tell them apart. Projects that are repeated and standardized may become exercises in logistics, while the first implementation of a logistical solution might be considered a project. Since both disciplines often address similar problems, there is likely to be a significant transfer of ideas from one discipline to the other.

Traditionally, logistics has been the focus of much study and innovation. Since logistics is usually applied to on-going operations, improvements in logistics usually result in increasing benefits over time. If, for example, the cost of a single car can be reduced by a dollar, then in a factory making 250,000 cars annually, a quarter million dollars will be saved each year.

The author has observed that major advances in logistics usually make their way into project management in ten or fifteen years, since the same forces that spurred the paradigm shifts in logistics are usually at work in the broader economy. Below are two examples of logistics innovations moving from manufacturing to construction in one or two decades.

Standardized Parts
Two hundred years ago, the US military wished to procure 40,000 rifles. At the time, rifles were made by skilled workers who fashioned one rifle at a time. Each rifle was different from the next, so maintenance required individually fashioned parts. Unfortunately, there were not enough skilled workers in the country to make the necessary rifles, let alone maintain them.

In 1798, Eli Whitney, the inventor of the cotton gin, proposed that he could make an incredible 10,000 rifles in two years by designing machinery (templates and fixtures) to make standardized, interchangeable parts. This was a major paradigm shift for manufacturing. It took Whitney over ten years to perfect his manufacturing system, which is regarded as the birth of the tool and die industry. Ultimately the system was successful, and the rifles in the war of 1812 were produced much faster, had higher quality, could be more easily maintained, and cost a great deal less than previous rifles. (Encyclopedia Britannica: Eli Whitney; Tool and Die Making) (Taylor, 1990)

In the 1820’s, US sawmills began producing standard dimension lumber in quantity, and in the 1830’s cheap machine-made nails became available. The first ‘balloon’ frame building is thought to be a warehouse in Chicago built in 1832. Standard construction techniques rapidly evolved: 2x4 studs placed 16 inches on center, 2x10 joists spanning up to 20 feet, stability provided by ¾” sheathing through which windows and doors were cut. Machine-made nails were easily driven into the soft wood and a building could be rapidly assembled from manufactured materials by (relatively) unskilled workers. Houses are still built this way, almost two centuries later. (Encyclopedia Britannica: Building Construction)

It’s easy to visualize how standardized, interchangeable manufacturing parts influenced the development of standardized, interchangeable building parts. The entire direction of the construction industry was heavily influenced by Eli Whitney’s rifles. Although a construction project was still a unique and temporary event, it took on many of the features of standardized manufacturing.

Mass Production
In 1908, Henry Ford introduced the Model T Ford. It was so successful that Ford had to continually invent faster, cheaper ways to manufacture the car. Over the two decades of the Model T’s life, Ford perfected the assembly process, introducing the first moving assembly line in 1913. In 1927, the River Rouge plant received just enough iron ore each morning to make a day’s worth of cars; 28 hours later the ore had become the steel in a finished car. In two decades, Ford produced almost 17 million Model T’s, displacing many horses in the process. The Model T precipitated one of the most rapid and pervasive changes of the lifestyle of common people in history. (Encyclopedia Britannica: Henry Ford)

The steel coming into Ford’s plant also moved into the building industry. During the 1920’s between World War I and the Depression, steel framed high-rise buildings came to the cities of America. As these buildings went up, specialized trades became more important, since framing, sheathing, plumbing, heating, elevators, etc. each required full time specialists. (Encyclopedia Britannica: Building Construction) The various trades required someone to sequence their activities, supply materials, assure safety standards were met, and generally manage the project. Thus the construction project manager emerged to coordinate a complex series of specialized but interrelated construction tasks, each of which contributed to the completion of the building. A well-managed construction project bears quite a few similarities to an assembly line.

Just In Time
Henry Ford’s River Rouge plant was an excellent example of Just-in-Time manufacturing, but his ideas on inventory were not widely held. After World War II, operations research developed theories on inventory management which determined optimized lot sizes, reorder points, and distribution stocking levels. In the 1970’s, a radically different theory was developed in Japan, spearheaded by Toyota Motor Corporation. (Shingo, 1981) The concept that inventory should be kept at a minimum, lots should be very small, and products should be built ‘on demand’ rather than stocked, ran contrary to the current manufacturing ‘wisdom’ in the US. However, Just-in-Time models proved to have significant advantages in capital reduction, plant throughput, quality assurance and market responsiveness.

The benefits of Japanese manufacturing techniques began to dawn on the US manufacturing community in the 1980’s. Eliyahu Goldratt’s 1984 novel, The Goal, popularized Just-In-Time concepts and introduced the ‘Theory of Constraints’. (Goldratt, 1984) This theory suggested that finding and ‘feeding’ the bottleneck workstation in a manufacturing plant would allow all other processes to hold to a steady and predictable pace. Pacing manufacturing with the ‘constraint’ workstation required the manufacturing logistics community to abandon many long-held beliefs about inventory management. But the new logistics paradigm worked so well that it rapidly became standard practice in manufacturing and distribution logistics.

If the author’s thesis that logistics innovations make their way into project management in about a decade is true, then Just-In-Time concepts or their derivatives should have impacted project management in the mid 1990’s. Sure enough, in 1997, Goldratt published Critical Chain, a novel in which he applies the Theory of Constraints to Project Management. (Goldratt, 1997)

The Theory of Constraints is useful in project management when multiple projects are competing for the same resources. Similar to its application in manufacturing, Theory of Constraints project management finds and ‘feeds’ critical path tasks and bottleneck resources. Most of the ‘common wisdom’ embodied in project scheduling techniques must be re-thought for this to work in the project environment. For instance, task durations are planned at 50% probability of completion. Thus it is expected that half of the time, tasks will be late. Indeed, it has proven to be difficult to abandon the perception that it is ‘good’ for all tasks to be completed within their estimated time.

Just-In-Time and Theory of Constraints focus on optimizing the entire flow of material or work, instead of sub-optimizing individual areas. The organizational structure and reward system in the US is often set up to reward individual effort, without much regard to (or understanding of) the impact of that effort on the ‘big picture’. Since World War II, the US economy has became increasingly globalized, and countries with different cultures and reward systems are selling products in the US. Unconstrained by ‘standard practice’, these countries consider and optimize manufacturing from a different, often broader perspective. This globalization of the economy has played a large role in bringing Just-In-Time and the Theory of Constraints to logistics.

Globalization has influenced project management also. In the electronics industry, very rapid time-to-market with ‘cast-in-stone’ release dates are required for new products, and Theory of Constraints project management is being widely applied to meet these challenges.

Lessons for the Information Industry
One of the fundamental driving forces behind paradigm shifts in logistics is an overwhelming need for lower cost and less specialized labor, which leads to standardization. This is often followed by an opposing force, the desire for customization.

Standardization
In 1800, the lack of enough skilled workers to make rifles in the face of an impending war was an unacceptable situation that demanded the invention of standardized, interchangeable parts. In the early 1900’s, the demand for inexpensive transportation that allowed ordinary people to travel great distances was a key forcing function in the emergence of mass production.

It’s easy to see this drive for standardized, interchangeable parts at work in the information industry. In 1983, the IBM PC took the world by force, rapidly becoming the standard of the industry and soon outstripping IBM’s ability to maintain control. (One can imagine that Henry Ford had to invent the assembly line to avoid the same fate as IBM.)

Standardized Internet access, embodied in standard web browser capability, has driven the on-line revolution that is sweeping the country. A single cell phone frequency in Europe has spurred European cell phone use to a far greater level than in the US, which has multiple cell phone frequencies. In the face of a huge demand for a standardized document format, Microsoft jeopardized its position by releasing incompatible versions of Microsoft® Word, while Adobe® Acrobat® filled the vacuum with a standardized ‘Portable Document Format’ (PDF).

Standardization is driven by an insatiable need which cannot be satisfied at an acceptable price by the available techniques or skilled people. Today the software industry has an enormous gap between the need for programming and the available programmers. It should be obvious to students of economic history and that the software development environment is ripe for a massive switch to standardized, interchangeable, mass-produced components.

Mass Customization
As brilliant as Henry Ford was, he missed out on a major trend that overtook the success of his Model T. After two decades, people began to get tired of black (the only color of the Model T). Copying Ford’s methods, other manufacturers figured out how to make inexpensive cars, and they started adding new features. Ford’s failure to recognize the shift in customer desires caused his company to loose its overwhelmingly dominant position in the industry. (Encyclopedia Britannica: Henry Ford) In the 1970’s the same thing happened to the US car makers – their failure to respond rapidly to the huge demand for fuel economy and high quality allowed Japanese car-makers to gain a large market share.

It is not surprising that one of the latest logistics innovations is mass customization, enabled by the information revolution. Dell Computer builds computers only after receiving an order, and ships the customized computer within days. Anderson Windows custom-builds windows to fit individual residential houses. Mass customization, a key advance in new product development and logistics, requires modular architectures and standardized components that are not assembled until individual customers place orders. Sophisticated information system support, especially in order management, is necessary for successful mass customization. (Gooley, 1998)

Products are not the only things that can be customized; the Internet has abundant examples of customized marketing, customer service, and information delivery. ASPs (Application Service Providers) are rapidly positioning themselves to ‘rent’ applications through the web. ASPs sell common applications that can be rapidly configured to meet individual customer requirements. In fact, the appearance of ASPs is probably a direct result of the pressure to produce ‘standard’ software in the face of a severe programmer shortage. ASPs success will depend on how they manage to retain the advantages of standardization while moving toward mass customization.

Software project management is also being influenced by mass customization. Instead of finalizing system requirements at the beginning of a project, it is considered good practice these days to allow users and customers to modify the requirements on-going basis as the software is being developed. The concept of continuing user feedback during software development started with the spiral life cycle, originally proposed in 1988 (Boehm, 1988), and evolved to The Unified Software Development Process (Jacobson, 1999). Common wisdom today holds that IT projects should iterate toward an increasingly competent set of capabilities through incremental builds which allow continued customization of the software during its development lifecycle.

Summary and Conclusion
Logistics innovations are not accidents. They are driven by economic forces which demand a paradigm shift to keep an undesirable situation from overwhelming the economy. Once the paradigm shift occurs, the forces that caused it continue to exert pressure on other areas. Over time, the new paradigm will make its way into project management practice.

Often the economic driving force behind innovation is an overwhelming need for something in far greater quantities at far cheaper prices than current practices allow. This drives innovation in the direction of standardization. However, once standardization becomes the common paradigm, a shift in expectations usually occurs. Customers begin to expect tailored products, without increasing the cost, of course. This drives innovation in the direction of mass customization.

Another driving force behind innovation is globalization, which brings unique perspectives to logistics and project management. A fresh look at a problem by someone who doesn’t know the ‘right’ answer can lead to out-of-the-box thinking and new approaches.

By looking for trends in economic driving forces and noting innovations in logistics, project managers can predict the forces which will drive project management practices in the future.
____________________
References
Barry W. Boehm, “A Spiral Model of Software Development and Enhancement”, IEEE Computer, Volume 21, Number 5, May, 1988, Pages 61-72

Council of Logistics Management, “Definition of Logistics”, retrieved March 29, 2000 from http://www.clm1.org/

Goldratt, Eliyahu, Critical Chain, North River Press, 1997

Goldratt, Eliyahu, The Goal, North River Press, 1984

Gooley, Toby B., “Mass Customization: How logistics makes it happen”, Logistics Management and Distribution Report, April, 1998, retrieved on March 29, 2000, from ‘Logistics Online’

Jacobson, Ivar, Booch, Grady and Rumbaugh, James; The Unified Software Development Process, Addison-Wesley, 1999

Project Management Institute, Guide to the Project Management Body of Knowledge, 1996, The definition of ‘Project’ is in Section 1.2, Page 4.

Shingo, Shigeo, Study of ‘Toyota’ Production System form Industrial Engineering Viewpoint, Japan Management Association, 1981

Taylor, David, Object-Oriented Technology, A Manager’s Guide, Addison-Wesley, 1990, Reference to Eli Whitney, Pages 85-87

Screen Beans Art, © A Bit Better Corporation