Bringing Lean to Software Development

Software is not about making something, it’s about making something work. Take software out of a cell phone, a car, a factory, a business, and it won’t work anymore. Software is the brains behind just about everything complicated that we try to do.

In 1960, J.C.R. Licklider, the visionary behind the Internet, wrote: “The main aims [of software] are 1) to let computers facilitate formulative thinking as they now facilitate the solution of formulated problems, and 2) to enable men and computers to cooperate in making decisions and controlling complex situations without inflexible dependence on predetermined programs.”[1] In 1962, Licklider articulated his vision of an “intergalactic network” connecting everyone in the world.[2] Throughout his life, Licklider could be found in a leadership role in one of the organizations that worked to convert his vision into reality: Bolt Beranek and Newman(BBN), ARPA, IBM Research, MIT. He died in 1990, just as the ideas he championed finally grew into the Internet.

If we look at the Internet through Licklider’s eyes, we see the real purpose of software – it does the routine thinking for us so that we can turn our minds to more complex problems. If we look at the evolution of the Internet, we see what developing software is really all about. At its core, software development is the process of gradually finding ways to turn over more and more of what we know to computers so that we have more space left in our minds to discover ever more interesting things.

The Truck-Driving Problem
Imagine that you are responsible for driving a truck across America, along highways, through cities and around detours, dealing with whatever idiosyncrasies that weather and traffic might throw at you. Now imagine that your job is not to drive the truck, but program a computer to drive the truck for you. How would you go about this task?

Many people – especially those who have never built software – would break this problem into a series of steps: first design the logic of the system; second, translate that logic into code; and finally, add the code to the truck and test that it works. But you can imagine that a problem as complex as the truck-driving problem is far too complex to solve with this approach. Instead, the problem is being tackled over time with a lot of experimentation and testing, one module at a time. Today we have laser following cruise control and tomorrow we will have lane following capability. Soon we might expect congestion detectors and GPS guidance. Someday the truck-driving problem might be solved, but by the time it is, the system will no doubt be a lot different than what we imagine today.

Conventional Wisdom
Unfortunately, it has been conventional wisdom in software development to separate design from coding, coding from testing, and software from the system it supports. This gives us development processes that ignore the learning loops necessary to discover the logic of a complex system, create concrete representations of the system, make sure that these work together, and over time, grow a system that becomes capable of accomplishing the complete job. In software development, we have been asked to solve too many truck-driving problems. And when it turns out that we have been handed an impossible problem, it’s usually the developers – not the process or the scale of the problem – that are held responsible for the failure.

When I was an IT manager in a manufacturing plant in the early 1980’s, we discovered that the conventional wisdom about the way that manufacturing should be done had become obsolete. We figured this out because our competitors in Japan started doing things that produced far better results than we were able to achieve. In response to this serious competitive threat, we threw out conventional wisdom and decided to try the counterintuitive approach being used in Japan, called “Just-in-Time.” As a management team, we came to agree on a few basic principles and enlisted the wisdom of everyone in the plant to figure out how to make these principles work in practice. Indeed, it took some time and a lot of hard work to implement Just-in-Time in our plant. As part of the effort, we completely changed the way we thought about quality and used statistics to gain a deeper insight into how our equipment worked. Slowly, it seemed to us, we got better. In retrospect, the improvement was dramatic, but it certainly didn’t feel like it at the time.

Today, the name “Just-in-Time” has been replaced with “Lean,” but to me it remains largely about the same thing: abandoning conventional wisdom that has become obsolete and moving on to a new way of thinking. The transformation will be multifaceted, principle-based, and entail a lot of very hard work by everyone, workers and managers alike. And – probably – it will only happen if there is something that shakes people up enough to cause them to abandon conventional wisdom and look for a new way of thinking about how to do their job.

Analogies
For those who look upon Lean as a set of operational practices, it doesn’t make sense to apply Lean to software development. Inventory, motion, variation, overproduction – the wastes of an operational process – don’t translate easily to the job of creating the brains of a system and making it smarter it over time. In fact, software development has long been plagued by inappropriate comparisons with other fields. Too often, analogies are created by someone who has only a passing acquaintance with a field. For example, someone who knows little about construction might say: “Buildings are completely designed before construction starts – if it works for buildings, why can’t it work for software?” I once discussed this analogy with a group of construction superintendants, who just about laughed me out of the room. They live in a world of incomplete and conflicting drawings, master schedules that bear no resemblance to reality, owners who regularly change their minds, and the constant threat of lawsuits.

Manufacturing analogies have been foisted upon software development by those who think that “cutting code” is pretty much the same thing as cutting metal. It’s painful to imagine the damage that has been done to software development by the manufacturing slogan “Do it Right the First Time.” I often wonder how this slogan came to mean that code should never have to be changed once it is committed to the code base. In our plant, product specs and manufacturing processes changed all the time; we just tested a product after each tiny step to be sure that no defects had appeared. Declaring that we should not have to change code once it is written is like decreeing that once we start our truck out across the country we should never have to change the planned route, no matter what detours or bad weather or flat tires we might encounter along the way.

I was only vaguely aware of the dangers of analogies when I set out to compare Lean software development to Lean manufacturing. I like using analogies – consider the truck-driving problem for example. I think of analogies as tools to help us understand and agree upon principles; but they should never be used as implementation guidelines. Copying specific practices from another field – or even from another company in our field – is a lazy way to implement change. First class execution is possible only when teams carefully think through the principles behind new ideas, understand how these principles apply to their world, and carefully adapt specific practices so that they make sense in the new context.

Paradigm Shift
I didn’t run into the term “waterfall” until 1999, and when I learned what it meant, I did not understand how it could possibly work. Therefore I was not surprised to discover that “waterfall” processes really don’t work – what surprised me was the fact that conventional wisdom held that they should work. I decided that the time had come for a paradigm shift in software development, similar to the paradigm shift that manufacturing experienced during the 1980’s and early 90’s.

Paradigm shifts require that someone with a clear understanding of a core problem in a particular field demonstrate a new and significantly better way to address this core problem. There’s no question in my mind that in manufacturing, the core problem was push scheduling – at least it certainly was in our plant. Simply implementing an effective pull system drove all sorts of other good behavior, and led eventually to significant cost reductions. We decommissioned a lot of software when we implemented our pull system, because the logic of our existing scheduling system was fundamentally flawed.

In that 1999 waterfall project, the core problem I experienced was the idea that development should be a series of handoffs: Analysis, Design, Coding, Testing, Deployment. The problem wasn’t that these steps were unimportant, but that each step was supposed to be completed before the next step began – and since the next step would be done by a different group of people, the handoff had to be done through documentation. The counterintuitive concept in Lean software development is to turn this process on its side and deploy completely done software in small increments. Let’s not even try to develop a system that will drive a truck, let’s see if it is possible to add some intelligence to cruise control and get that working while there is still a driver behind the wheel. Of course this means we are not sure how a fully automated truck will work, when it might be ready or how much it might cost. But if we keep the ultimate goal of an automated truck in mind and design our laser following cruise control wisely, we will be a step closer to a self-driving truck.

It might seem obvious that a completely automated truck is far too ambitious an undertaking, so splitting it up into incremental steps is the best way to make progress. What is counterintuitive is that this approach should scale downward. But it does. And if you look at any really successful complex system, this is the way it was developed. Lean development requires a visionary leader and a sponsor willing to fund the vision rather than a promise that is unlikely to be kept. However, such a sponsor can be hard to find; sponsors always seem to want a “plan.”
 
The “Plan”
The core problem in software development can be boiled down to the intense pressure for a “plan” that lays out what the software will be able to do, how long it will take, and what it will cost – and the subsequent expectation that development will follow this “plan”. The problem is magnified when the software is expected to be developed independently from the system for which the software will supply the brains. Why is planning a problem? Isn’t a plan essential? As Dwight Eisenhower once said: planning is indispensable, but plans are useless.

The problem with software is that the logic behind any system is inherently complex, and developing that logic independently from the system increases the complexity by at least an order of magnitude. As Fred Brooks once noted, “The complexity of software is an essential property… Many of the classical problems of developing software products derive from this essential complexity and its nonlinear increase with size.”[3] Trying to plan a large software development effort is not much different than trying to plan the development of a software package to drive a truck across America – without access to the truck.

You might notice that the core problem of software development is beginning to bear some resemblance to the core problem of manufacturing – we suffer from push scheduling in manufacturing and push planning in software development. But the antidote is somewhat different. In manufacturing, switching from push to pull scheduling gives dramatic results. In software development, switching from forecast-driven to feedback-driven development gives equally dramatic results.

The Antidote
Embedded in every development “plan” is a forecast – a guess really – of what will happen in the future. Plans that separate the logic of a system from the system itself are based on an additional layer of guessing – ahem – forecasting. If we base the development of a complex system on forecasts, we are doing forecast-driven development. If we develop and deploy small useful feature sets and get them working incrementally, then we are doing feedback-driven development. For simple systems, either approach will work. For large, complex or evolving systems, the only feasible approach is feedback-driven development.

There is a great appetite for predictability in any business that has to answer to shareholders or any organization that is under public scrutiny. This is entirely appropriate. However, the idea that predictability comes from creating a plan and then following the plan is fundamentally flawed when complex systems are involved. Following a plan will create predictability only to the extent that the plan is based on an accurate forecast of the future and a thorough understanding of the complexity of the system to be developed. Since I am a process control engineer, I would like to call to your attention something we learned a long time ago: When you really need predictability, you create a feedback system that can respond to change as it occurs, rather than a predictive system that is incapable of dealing with variation.

Many people are concerned about feedback-driven development – it seems to imply a lack of control. But when we are trying to create the brains behind a system, control is hardly what it is all about. The vision behind the Internet had nothing to do with control – and everything to do with finding ways for people to communicate and think more effectively. It took decades of detailed work engaging great minds from around the world for the Internet to become a reality, and it will continue to emerge for decades to come. This great network of software started with a vision and grew with careful guidance, detailed technical work, and extensive collaboration. If you want to understand how Lean software development works, you need look no further.

Originally Published in Lean Manufacturing 2007, a Supplement to Manufacturing Engineering

References and Notes
[1]Licklider, L.C.R. (1960). "Man-Computer Symbiosis" IRE Transactions on Human Factors in Electronics, v.HFE-1. p.4-11.

[2] This was described in a series of memos by Licklider.

[3] The Mythical Man Month, 20th Anniversary Edition, by Fred P. Brooks, Jr., Addison Wesley,1995.

How Does Toyota Do it? (Book Review)

When Matthew May was asked to help translate the Toyota Production System for the knowledge worker, he thought: “Huh? It doesn’t make any sense. Everyone knows factory work isn’t creative, right?”

How wrong he was. He found that factory workers were more engaged and creative than their corporate counterparts. Their jobs weren’t creative, their job were to be creative. He found that the Toyota organization implements a million new ideas a year – three thousand ideas a day. May notes that these new ideas are the real reason Toyota makes over twice as much money as any other carmaker, with under 15% of the market. These thousands upon thousands of implemented ideas are the engine of Toyota’s innovation.

Who needs another book on innovation? May asks in the introduction to his book, The Elegant Solution: Toyota’s Formula for Mastering Innovation. You do. This book is a great read – short lessons, lots and lots of case studies and examples both in the text and in the sidebars liberally sprinkled throughout the book. This is a book unlike any other on innovation; it is the result of May’s deep thinking and experience in trying to identify for Toyota University what innovation in Toyota actually means.

The book outlines three principles and ten practices for driving innovation in knowledge work, and concludes with a couple of brief chapters on how to get started down this path.

Principles:

1. The Art of Ingenuity. The pressure to innovate falls on the individual – every single individual in the company. First, ingenuity means connecting with your work, whatever it is, understanding why it is important, and making sure that it is a good fit with your skills and interests. If it isn’t create a job that is. Second, ingenuity means constantly experimenting to figure out how to do that job better. Third and most important, everybody is expected to use their ingenuity – everybody – all of the time.
2. The Pursuit of Perfection.  There is no such thing as perfection – but aspiring to achieve perfection is nevertheless the goal. A good example of this principle can be found in the Fast Company article No Satisfaction at Toyota, an article about constant innovation at the Georgetown Kentucky factory. Read the article, it does a fantastic job at explaining the pursuit of perfection.
3. The Rhythm of Fit. Innovation is always obvious – after the fact. But discovering the obvious is not such an easy task. It requires that you are grounded in today, yet have a clear vision of tomorrow. It requires systems thinking, rather than program thinking. And it requires the right social context – one that inspires rather than suppresses creativity.

Practices:

1. Let Learning Lead. Real learning is not about books or lectures or workshops. It is about constantly trying things and finding out what works. Learning involves asking the right questions much more than finding the right answers. Learning means moving the Scientific Method from PhD programs in Universities to the shop floor and the knowledge worker. Learning comes first.
2. Learn to See. Walk in the shoes of the front line worker. Live the life of the customer. Data is important, but it’s interpretation that converts data into information. Learn to walk around and observe what is really happening. Watch your customer, become your customer, involve your customer.
3. Design for Today. As good as Toyota is at anticipating the future, their innovations are always grounded in clear and present needs – demographic shifts, energy shortages, safety concerns. The idea is, as hockey great Wayne Gretzky once said: “Skate to where the puck is going to be, not where it has been.” Market leaders don’t so much create markets as they understand where the market is going to go, and skate there.
4. Think in Pictures.

5. Capture the Intangible. The most compelling solutions are often perceptual and emotional.” May says. For example, when the Lexus team was designing the car, the entire team spend three months living in luxury in southern California, just to feel what their customers felt. They learned that luxury cars were not transportation, they were a safe sanctuary and quiet escape. Provide an ‘experience’ instead of a ‘product’.
6. Leverage the Limits. This chapter centers on a wonderful story about Toyota’s North American Parts Operations (NAPO) stretch goals. In the year 2000, Jane Beseda, newly appointed vice president and general manager, challenged NAPO with ten audacious and mutually competing goals, including inventory reduction, response time reduction, and waste reduction. The goals were such that they could only be achieved by departments working together across what had been organizational boundaries, and the results were nothing short of amazing.
7. Master the Tension. A McKinsey study on global productivity in the late 1990’s found that the companies which improved productivity the most were invariably companies in highly competitive industries. These companies had to find a better way of doing things – it was a matter of survival – but could not afford to throw money at the problem. We agree with May – constraints coupled with challenge provide the breeding ground for innovation.
8. Run the Numbers.There is a place for instinct, but temper it with facts. Think hard about what the important numbers are – and they are often not the obvious ones. With insight into what numbers are important, capturing and analyzing data will confirm instinct (or not) and uncover patterns that are otherwise invisible. 
9. Make Kaizen Mandatory. Standards exist to be challenged and changed. They are the current best-known way of doing things, and they are documented and followed by everyone. But they objective is to change the standard, to keep on improving the way things are done. Taiichi Ohno once said “Something is wrong if workers do not look around each day, find things that are tedious or boring, and then rewrite the procedures. Even last month’s manual should be out of date.”
10. Keep it Lean. Scale it back, keep it simple, make it flawless, let it flow. May notes “We are hardwired to hunt, gather, and horde. To think more. So lean runs counter to human nature. Getting lean requires fighting the basic instinct to add, accumulate, store. Lean requires a precise understanding of value: the who, what, when, where, how, and why of the customer’s need. It means getting that value to them without complexity creeping in.

Get this book. Read it. It will change the way you think about innovation.

Cause and Effect

Is low cost achieved by focusing on cutting costs?
Is high utilization achieved by trying to utilize resources full time?
Does standardized work mean that work processes are followed without challenge?
Does everyone in your organization agree on the answers to these questions?

Cost
When Jane Beseda took over Toyota’s North American Parts Operation (NAPO), she knew that really dramatic results required breaking down the barriers between departments [1].  So she set three year Stretch Goals that were all but impossible – a 50% reduction in inventory, 25% increase in throughput, 25% reduction in freight costs, 50% reduction in packaging expense, 25% increase in space utilization, 50% decrease in backorders.  After a year of effort, department managers began to realize that that they were not going to achieve the Stretch Goal targets unless they changed their focus to cross-organizational projects.  Project teams had been struggling to coordinate their work through the traditional functional department planning approach. This was changed; the departments started to consider potential impact of their plans on other functions and areas of the broader organization. Only then were truly significant gains in eliminating waste and reducing cost realized. After three years, the results at NAPO were truly amazing; almost all of the Stretch Goals were achieved. But this would never have happened if everyone in the organization had not focused on overall system waste rather than individual department costs.

Low overall costs rarely come from lowering costs in individual departments; they come from lowering system costs. Lean companies have learned that this requires a keen understanding of underlying cost drivers and a system-wide effort to eliminate waste.  Consider Zara, a huge women’s fashion clothing chain headquartered in western Spain. It fills retail store orders twice a week, shipping garments around the world in two days – not folded compactly in boxes – but ironed and hanging on hangers ready to display. Zara realizes that it lowers overall costs by getting the clothing that women are asking for on shelves very rapidly.  Zara has fewer markdowns and unsold clothes, and it drives more business to stores with less advertising than its competitors. These tremendous savings would disappear if Zara focused on reducing shipping costs. This is systems thinking at work.

Over time, individual departments will find ways to drive out costs, but rarely do organizations attack the larger costs that occur between departments, divisions, functions, and companies.  A quick and easy cost reduction approach for a department might be to outsource work to obtain lower labor costs.  However, in most cases, outsourcing simply adds more boundaries to cross, and boundaries typically add 20 to 30% to overall costs [2]. One company I know outsourced the process of engaging contractors.  Since training is included in this outsourcing arrangement, all trainers must work through the outsourced contractor management company.  The amount of time and paperwork necessary to set up a training engagement at this company is about an order of magnitude greater than at any other company I have dealt with. However, the arrangement appears to be a cost reduction for the company, because contractors pay the contractor management firm through a fee imposed on their payment. This financial arrangement hides the tremendous waste in the engagement process, which includes a great deal of problem resolution by employees in the company receiving the training. But because of organizational barriers, these same employees have no way to change or improve a very burdensome process.

Standardized Work
In studying Toyota, many companies notice that standardized work processes are a key element of the company’s success.  What they generally miss is that at Toyota, standards exist to be challenged and changed by the front line employees doing the work.  Toyota actively encourages all employees to question any part of their job that is annoying or gets in the way of doing a good job. Employees are expected to look for a better way, prove with experiments that the new way is better, and then implement a new standard.  Taiichi Ohno, father of the Toyota Production System, wrote: “Something is wrong if workers do not look around each day, find things that are tedious or boring, and then rewrite the procedures. Even last month’s manual should be out of date.”

There are those who believe that standardized processes should be imposed from the outside and followed without question. However, companies with a Lean perspective believe that the essence of eliminating waste is to encourage everyone to attack and change the things that annoy them about their jobs. Only by engaging people in doing their job better on a daily basis can you get the sustainable gains that companies like Toyota enjoy. Work standards in a lean company spell out the current best known way of doing things, and they are a baseline for change. New standards are developed through constant experimentation by work teams, using the current work standards as the baseline against which improvement is measured.

At Toyota’s Georgetown, Kentucky plant the paint department has always been the biggest bottleneck preventing one-piece-flow of vehicles through the plant.  Over the past few years, however, things began to change [3].  First the people in the department devised a way to paint any color in any order – instead of piping paint to a robot through a tube that needs cleaning between colors, they now pipe each paint color to a canister just the right size to paint a car.  The robot picks up the right canister and Viola! the cost and time needed to clean out paint lines is eliminated.  The bottom line:  30% less paint used, a huge drop in the use of cleaning solvents, and cars can now be painted any color in any order. The throughput of the shop has increased from 33 to 50 an hour while the space required for painting was reduced by a third and employees were freed up for other work.

At most auto companies, this achievement would be cause for celebration – but at Toyota, constant improvement is the day-in-day-out job of the paint manager. There are no black belts, no outside process experts – just the people doing the work led by their first line managers. Everyone’s job is to keep on improving their work processes, every day, every week, every month. All workers are engaged in the relentless pursuit of perfection – the elusive goal which keeps everyone looking forward rather than patting themselves on the back.

In software development, we have been led to believe that “maturity” involves documenting best practices and making sure that they are followed. On the contrary, a truly mature organization expects development teams to constantly challenge and improve their processes.  The day an organization settles into believing that it is perfect is the day it invites its people to stop thinking. Real, sustainable improvement on all fronts comes only when all workers are expected to challenge and fix anything about their job that annoys them or keeps them from taking pride in their work.

Utilization
Managers in manufacturing plants, supervisors of computer operations, and highway engineers agree on one cause and effect relationship:  if your resources approach full utilization, response time slows to a crawl. Queuing theory is widely applied to servers and highways and manufacturing equipment.  However, in a development environment, managers seem to be unaware that attempting to achieve full utilization slows response time to a crawl and decreases actual utilization. Development organizations are no more immune from the laws of queuing theory than airlines. We see how constantly full airlines with no slack cause cascading problems whenever a slight perturbation is introduced into the airline system. And weather being what it is, perturbations happen all the time.

Similarly, when a slight perturbation is introduced into a fully scheduled development organization, cascading problems are inevitable. Knowledge work being what it is, there will always be perturbations to the most carefully laid out schedules. The only way to deal effectively with these perturbations is to follow the advice of queuing theory:  work in small batches, minimize the length of queues of work to be done, and never, ever schedule an organization beyond its capacity to deliver.  Do this and utilization will increase.  Focus directly on utilization and it is guaranteed to be sub-optimal.

Many organizations queue up requests for software development into large batches of work to be done, presumably because better decisions can be made if the whole picture is visible before work begins.  Challenge this assumption.  Why is it better to respond very slowly to a pile of accumulated requests rather than respond very quickly to the current most important outstanding request? What good, really, does it do to have a long list of work to do?  Looked at closely, it's hard to make a case for piling up lists of work-to-do that are far longer than you have a hope of accomplishing. You're pretty much wasting your time keeping track of stuff you can never get around to, and you're probably setting incorrect expectations on the part of the requestors. Just say no.

Unfortunately, once software has been around for a while, we often see that it takes longer and longer to run the regression tests as each new feature is added, since every feature increases the regression test load.  We call this increasing regression load the regression deficit, and unless it is systematically tackled and reduced, the code base will become increasingly difficult to change.  You will be tempted to release software in larger and larger batches, because regression testing takes so long. Similar to the paint shop manager, a software development manager’s job should be to help development teams chip away at the regression deficit every day, every week, every month, until throughput is increased and one-piece-flow of small feature sets becomes practical.

Cause & Effect
In order for organizations to perform brilliantly, there are two prerequisites:  First, everyone has to agree on what they want, and second everyone has to agree on cause and effect [4].  Let’s assume that everyone in your organization agrees on the results they want, their values and priorities, and the trade-offs they are willing to make in order to achieve those results.  The question to ask is – does everyone agree on cause and effect?

Is there general agreement on what actions will result in system-wide cost reduction?  Are individual departments expected to reduce costs independently, or is it clear that departments must work together to reduce overall system costs, even at the expense of costs in individual departments? Are the measurements in place to reduce the costs of crossing boundaries?

Is there general agreement on the best way to achieve consistent results through standardization?  Are standards always followed?  If not, do you really understand the root cause – are the standards irrelevant or inaccurate, too complex, or too far from the actual work being done? Are standards maintained by a central process group, or are they continually improved by the people doing the work?  Are front line people expected and encouraged to continually challenge and change standards?

Do you have a project scheduling system aimed at optimizing utilization?  Does the management team believe that the system is in fact optimizing utilization? Do people believe that full utilization is the right measurement to emphasize? Is time-to-market an important factor in your business? Does everyone agree on the relationship between time-to-market and utilization?

Lean thinking is counterintuitive; it flies in the face of conventional wisdom concerning cause and effect. No matter how well proven the results, no matter how intellectually sound the arguments of systems thinking are, it is almost impossible for those who have been successful with conventional wisdom to change their habits. It is very difficult for leaders to address the system costs that flourish between department boundaries if they believe that lowering costs in each department will add up to lower overall costs. When leaders believe that standardized processes are an end rather than a beginning, they have a difficult time leveraging the wisdom of front line employees. For those leaders who believe that focusing on full utilization will increase productivity, the outstanding productivity gains that come from focusing instead on throughput will not be available.  It all comes down to measures – small measures cause small results; global measures promote globally optimized results.

Those who have seen the dramatic results of Lean Thinking have come to believe that concentrating on throughput, making front line employees the center of focus of the company, and eliminating waste across the system are the tools of choice in a competitive environment.  Most management teams that have been threatened by fierce competition and survived have changed their habits and adopted the counterintuitive tenets of Lean Thinking. Virtually all companies facing fierce competition from Lean companies that have failed to adopt similar thinking have failed to thrive in the long run.
_________________
Footnotes:

[1] This story is from The Elegant Solution: Toyota’s Formula for Mastering Innovation, by Matthew E. May, Free Press, 2007, p. 139.

[2] Management Challenges for the 21st Century, by Peter Drucker, HarperBusiness, 2001, p. 33.

[3] From “No Satisfaction” by Charles Fishman, Fast Company, Dec 2006 / Jan 2007

[4] See “The Tools of Cooperation and Change,” by Clayton Christensen and co authors, Harvard Business Review, October 2006.


Screen Beans Art, © A Bit Better Corporation

Managing the Pipeline

“I used to have everyone I needed in my business unit,” she told me. “But then we reorganized in the interest of efficiency, and all of my software people were moved into a central group. Before the reorganization, I had a great software architect who helped design all of our new products. Today we’re supposed to figure out exactly what we want before we get a project approved. Then the IT people assigned to my projects don’t know anything about my business, so the team spins its wheels for a long time before it gets traction. Efficiency? What a joke.”

A couple years earlier, this rapidly growing young company (call it XYZ) saw its revenue flattening and had regulators questioning its processes. In an attempt to impose discipline and cut costs, XYZ centralized software development. Executives felt that this would give them more visibility into the development portfolio and ensure that a standard process was used throughout the organization. They also hoped to even out the software development workload, increase utilization of resources, and eventually be able to outsource a good deal of development.

The problem was, XYZ’s products were largely based on software. When the software developers left the business units, the company's track record for fielding successful new products plummeted. Time-to-market stretched out for even the most important initiatives, and market share dropped. What went wrong?

People or Resources?
Company XYZ did very well when new products were designed and implemented inside of a business unit. When a key part of the product development team, those developing the software, were removed from the business unit, many important features of product development were lost. For example, funding was no longer incremental, based on stages and gates. Complete funding for product development had to be justified and allocated in order to get software development resources assigned. In turn, a complete specification and estimate of spending was required before work could begin. Software architects were no longer involved at the fuzzy front end of new product development, and the discovery loops that used to be part of product development were no longer acceptable.

Even as feedback loops were removed from product development, they became more important than ever, because the software development people on the product team were usually not familiar with the business, and worse, they were not even familiar with each other. In places where tacit knowledge and team cohesiveness are important, treating people as interchangeable resources just doesn’t work.

Scheduling
Prior to the reorganization, when software people were embedded in divisions, some people were regularly assigned to several projects at a time, while others appeared to be less than fully utilized. The company decided that it would be more effective to assign individuals to only one project at a time, and if possible, the projects should be small. Unfortunately, this created a scheduling nightmare, so XYZ invested in a computerized resource scheduling system to help sort out the complex resource assignments.

Computerized scheduling systems have a well known problem: they do not accommodate variation. XYZ discovered that if projects didn’t end when they were scheduled to end, the system’s assumptions were invalid, so the system’s resource assignments were often out of touch with reality. The company tried to fix such problems by keeping some teams intact and by holding weekly management meetings to arbitrate the conflicts between the computer’s schedule and reality. In practice, the overhead of management intervention and idle workers waiting for teams to assemble outweighed any efficiencies the system generated.

Company XYZ also tried to reduce the variability of project completion by urging teams to make reliable estimates and rewarding project managers who delivered on schedule. Unfortunately, such attempts to reduce variability generally don’t work. The reason for this becomes clear with a quick look at the theory of variation.

Variation
W. Edwards Deming[1] first popularized the theory of variation, which is now a cornerstone of Six Sigma programs. Deming taught that there are two kinds of variation: common variation and special variation. Common variation is inherent in the system, and special variation is something that can be discovered and corrected. Common variation can be measured and control charts can be used to keep the system within the predicted tolerances. But it is not possible for even the most dedicated workers to reduce common variation; the only way to reduce common variation is to change the system. And here’s the important point: Deming felt that most variation, (95%+)[2] is common variation, especially in systems where people are involved.

The other kind of variation is special variation, which is variation that can be attributed to a cause. Once the cause is determined, action can be taken to remove it. But there is danger here: “tampering” is taking action to remove common variation based on the mistaken belief that it is special variation. Deming insisted that tampering creates more problems that it fixes.

In summary: The overwhelming majority of variation is inherent in a system, and trying to remove that variation without changing the system only makes things worse. We can assume that most of the variation in project completion dates is common variation, but since computerized scheduling systems are deterministic, they can’t really deal with any variation. The bottom line: a computerized scheduling system will almost never work at the level of detail that XYZ was trying to use it. Exhorting workers to estimate more carefully and project mangers to be more diligent in meeting deadlines is not going to remove variation from projects. We need to change the rules of the game.

We know that estimates for large systems and for distant timeframes have a wide margin of uncertainty, made wider if the development team is an unknown. We should stop trying to change that; it is inherent in the system. If we want reliable estimates, we need to reduce the size of the work package being estimated and limit the estimate to the near future. Furthermore, estimates will be more accurate if the team implementing the system already exists, is familiar with the domain and technology, makes its own estimates covering a short period of time, and updates these estimates based on feedback. The good news is, once such a team establishes a track record, its variability can be measured and predicted.

Utilization
Unfortunately, Company XYZ believed that efficiency would be improved by increasing resource utilization. Trying to maximize utilization can have serious unexpected side effects, not the least of which is decreased efficiency and reduced utilization. If this seems odd, think about how efficient our highways are during rush hour. Most systems behave like traffic systems; as utilization of resources passes a critical point, non-linear effects take over, and everything slows to a crawl. Even the most brilliant scheduling system cannot prevent delays if you insist on 100% utilization.

When a computer operations manager looks at the utilization history of her equipment, she would never say: “Look at that – we’re only using 80% of our server capacity and 85% of our SAN’s. Let’s use them more efficiently!” She knows that such high utilization is a warning that the systems are operating on the edge of their capacity, and even now response times are probably slowing down.

But when a development manager takes a look at the utilization history of his department, he will often say: “Look at that – we are only using 95% of our available hours. We have enough free time to add another project!” At the same time, he is probably asking himself, “I wonder why I’m getting all these complaints about our response time?” And all too often his solution is, “We’ll just have to set more aggressive deadlines.”

Response Time
Consider the release manager of a software product. Assume she has service level agreement which calls for critical defects to be found and patched in four hours, serious defects to be found and patched in 48 hours, and normal defects to be fixed in the next monthly release. You can be sure that her primary measurement is response time and she adjusts staffing until the service level is achieved. Because of this, there will always be people available to attack defects, and occasionally people may have a bit of spare time.

In one of my classes, two teams did value stream maps for almost the same problem – deliver on a feature request which would take about 2 hours of coding. One team documented an average response time of 9 hours to deployment, the other team documented an average response time of 32 days. In the first case, the policy was: “When a request is approved, there will always be someone available to work on it.” In the second case, the request got stuck twice in two-week-long queues waiting for resources. The interesting thing is that the first organization actually did more work with fewer people, because they did not have to manage queues, customer queries, change requests and the like. They were more efficient despite, or perhaps because of, a focus on response time rather than resource utilization.

The bottom line is that managing response time, or time-to-market, is more efficient and more profitable than managing utilization. You need some slack to keep development and innovation flowing. As any good operations manager already knows, when work flows rapidly and reliably through an organization, its efficiency and utilization will be higher than in a organization jammed up with too much work.

Rules of the Game
Queuing theory gives us six rules for reducing software development cycle time:

1. Limit work to capacity
2. Even out the arrival of work
3. Minimize the number of Things-in-Process
4. Minimize the size of the Things-in-Process
5. Establish a regular cadence
6. Use pull scheduling

1. Limit work to capacity
The biggest favor you can do for your organization is not to accept any more work than it can handle. Of course, to do this, you have to know the capacity of your organization. One way to estimate the capacity is to look at output. If you currently complete one large system a year, deliver about three services a quarter, and respond to about seven change requests per week, this is a rough approximation of your capacity, and a good limit on the amount of new work you should accept.

Next you might calculate how much work you have already accepted. In one of my classes, an executive did the math and discovered that he had seven years worth of work in a queue that was reviewed every week. He decided that he could toss out all but a few months of work; the rest would never get done, but it was consuming a lot of time.

2. Even out the arrival of work
At Company XYZ, one of the scheduling headaches was caused by a huge workload during the first six months of the year, and a relatively low demand for the second half of the year. At first this puzzled me, because the company’s business was not seasonal, so there seemed to be no reason for the uneven demand. I suspected that there was a sub-optimizing measurement somewhere that might be the cause.; When I asked if the annual budgeting cycle or executive performance measurement system might be driving uneven demand, my suspicions were confirmed. I recommended that the organization work to change the measurement system, rather than accommodate it.

3. Minimize the number of Things-in-Process
One of the basic laws of queuing theory is Little’s Law[3]:

According to Little’s Law there are two ways to improve response time: you can spend money to improve the Average Completion Rate, or you can apply intellectual fortitude to reduce the number of Things-in-Process. For example, assume you can respond to about six feature requests per month. If twelve requests are released for work, they will take an average of two months to complete. If, however, only three requests are released at a time, it will take an average of two weeks to respond to a feature request.

4. Minimize the size of the Things-in-Process
We’ve already noted the effect of high utilization on cycle time; we should also note that as batch size increases the effect is much more pronounced.[4] This is shown in the graph:

So if you want high utilization, you should develop in very small batches. For example, you will get much faster throughput and higher utilization if you develop ten services one at a time, rather than developing all ten at the same time.

5. Establish a regular cadence
In a lean factory, every process is runs at a regular cadence called ‘tact time.’ If you want to produce 80 cars in 8 hours, you produce 10 cars per hour, so one car rolls of the line every 6 minutes. In software development the recommended practice is to establish an iteration cadence of perhaps two weeks or a month, and deliver small batches of deployment-ready software every iteration.

A regular cadence, or ‘heartbeat,’ establishes the capability of a team to reliably deliver working software at a dependable velocity. An organization that delivers at a regular cadence has established its process capability and can easily measure its capacity.

A regular cadence also gives inter-dependent teams synchronization points that they can depend on. Synchronization points are good places to get customer feedback, they are useful for coordinating the work across multiple feature teams, and they can help decouple hardware development from software development.

6. Use pull scheduling
Once both batch and queue size have been reduced and a cadence has been established, pull scheduling is the best method to compensate for variation and limit work to capacity. At the beginning of an iteration, the team ‘pulls’ work from a prioritized queue. They pull only the amount of work that they have demonstrated they can complete in an iteration. When a team is first formed or the project is new, it may take a couple of iterations for the team to establish its ‘velocity’ (the amount of work it can complete in an iteration). But once the team hits its stride, it can reliably estimate how much work can be done in an iteration and that is the amount of work it pulls from the queue.

There are other points where queues might be established: there could be a queue of proposed work that needs a ROM (Rough Order of Magnitude estimate). There may be a queue of work for a preliminary architecture assessment. (See figure below.) Note that these queues should be short, and a team should not pull work from a queue until it has available time to do the work.

A pull system assures that everyone always has something to do (unless a queue is empty), but no one is overloaded. The development process is managed by managing the queues. Management intervention is accomplished by changing the priority or contents of the queues. The cadence should be fast enough that changes can wait until the next iteration; in which case, changes are accommodated at the cadence of the process.

Conclusion
Development teams can do a lot to control their own destiny. They can make sure they have the right information, the necessary skills, and the appropriate processes to do a good job. But some things that impact the performance of the development team are outside of their control. Managing the pipeline is one of those things. If a development organization is swamped with work, no amount of good intentions or good process can overcome the laws of physics. If deterministic rules are applied to an inherently variable system, no amount of exhortation, reward, or punishment can make the system work. When the rules of the game have to change, the six rules for reducing cycle time are a good place to start.

References
[1]W. Edwards Deming – 1900-1993. Thought leader of the Quality Movement in Japan and the US.

[2] W. Edwards Deming, The New Economics, Second Edition, MIT Press, 2000, p 38.

[3] Usually the numerator is WIP (Work-in-Process). The term Things-in-Process comes from Michael L. George and Stephen A. Wilson, Conquering Complexity in Your Business, McGraw-Hill, 2004, p.37.

[4] See Factory Physics by Wallace Hopp and Mark Spearman, McGraw-Hill, 2000

Disclaimer: XYZ Company is not a real company, it is an amalgamation of companies I have worked with.

Breaking The Quality–Speed Compromise

In the hotly contested commodity business of assembling computers, Dell enjoys a 50% cost advantage over its competitors.[1] This commanding advantage comes from Dell’s exceptional responsiveness to customers, flawless operations, and remarkable speed of execution. Conventional wisdom once held that the low cost producer could not provide customized high quality products. But Dell decided that its customers could have it all – low cost computers with the latest technology, built to order and delivered in a week.

In the book Hardball, George Stalk notes that when an industry imposes a compromise on its customers, the company that breaks the compromise stands to gain a significant competitive advantage.[2] For example, the airline industry imposes a big compromise on travelers: if you want low cost tickets, you have to make your plans early and pay a stiff penalty to change them. Southwest Airlines breaks this compromise: its customers can apply the cost of unused tickets to a future flight without a change fee.

In the software development industry, we impose many compromises on our customers. We tell them that high quality software takes a lot of time; we ask them to decide exactly what they want when they don’t really know; we make it clear that changes late in the development process will be very expensive. There’s a significant competitive advantage waiting for companies that can break these compromises. In particular, I’d like to focus on breaking the compromise between quality and speed, because many companies have achieved great leverage by competing on the basis of time.

When I teach classes on Lean Software Development, the first thing we do is draw value stream maps of existing software development processes. Starting with a customer request, the class draws each step that the request goes through as it is turned into deployed software which solves the customer’s problem. The average time for each step is noted, as well as the time between steps, giving a picture of the total time it takes to respond to a customer.

Next the class determines how much of the time between request and deployment is spent actually working on the problem. Typically, less than 20% of the total time is spent doing work on the request; for 80+% of the time the request is waiting in some queue. For starters, driving down this queue-time will let us deliver software much faster without compromising quality.

But reducing wait time is not the only opportunity for faster software development. Typically the value stream maps in my classes show a big delay just before deployment, at a step which is usually called ‘verification’. Now, I don’t have any problem with verification just before deployment, but when I ask, “Do you find any defects during verification?” the answer is always “Yes.” Therein lies the problem. When a computer hits the end of Dell’s assembly line, it is powered on and it is expected to work. The verification step is not the time to find defects; by the time software hits verification, it should work.

The way to get rid of the big delay at verification is to move testing closer to coding – much closer. In fact, testing should happen immediately upon coding; if possible the test should have been written before the code. New code should be integrated into the overall system several times a day, with a suite of automated unit tests run each time. Acceptance tests for a feature should pass as soon as the feature is complete, and regression testing should be run on the integrated code daily or perhaps weekly.

Of course, this testing regime is not feasible with manual testing, automated unit and acceptance tests are required. While this may have been impractical a few years ago, the tools exist today to make automated testing practical. Obviously not all tests can be automated and not all automated test suites are fast enough to run frequently. But there are many ways to make automated testing more effective; for example, each layer is usually tested separately – ie. the business rules are tested below the GUI with most database calls mocked out.

In most of the value stream maps I see in my classes, there is a huge opportunity to move tests far forward in the process and catch defects at their source. Many companies spend a great deal of time tracking, prioritizing, and fixing a long queue of defects. Far better to never let a defect into the queue in the first place.

There is another area of my classes’ value stream maps that raises a flag. Toward the beginning of the map there is usually a step called ‘requirements’ which often interacts with a queue of change requests. Dealing with change requests takes a lot of time and approved changes create significant churn. There has been a feeling that if only we could get the requirements right, this ‘change churn’ would go away. But I generally find that the real problem is that the requirements were specified too early, when it was not really clear what was needed. The way to reduce requirements churn is to delay the detailed clarification of requirements, moving this step much closer to coding. This greatly reduces the change request queue, because you don’t need to change a decision that has not yet been made!

Toward the end of my classes, we draw a future value stream map, and invariably the new value stream maps show a dramatically shortened cycle time, the result of eliminating wait time, moving tests forward, and delaying detailed specification of requirements. We usually end up with a process in which cross-functional teams produce small, integrated, tested, deployment-ready packages of software at a regular cadence.

This kind of software development process exposes another compromise: conventional wisdom says that changes late in the development cycle are costly. If we are developing small bits of code without full knowledge of everything that the system will require, then we are going to have to be able to add new features late in the development process at about the same cost as incorporating them earlier.

The cost of adding or changing features depends on three things: the size of the change, the number of dependencies in the code, and whether or not the change is structural. Since we just agreed to keep development chunks small, let’s also agree to keep changes small. Then let’s agree that we are going to get the structural stuff right – including proper layering, modularization that fits the domain, appropriate scalability, etc.

We are left to conclude that the cost of non-structural change depends on the complexity of the code. There are several measurements of complexity, including the number of repetitions (the target is zero), the use of patterns (which reduce complexity), and McCabe scores (the number of decisions in a module). It has been shown that code with low complexity scores has the fewest defects, so a good measure of complexity is the number of defects.

Which brings us back to our testing regime. The most important thing we can do to break the compromises we impose on customers is to move testing forward and put it in-line with (or prior to) coding. Build suites of automated unit and acceptance tests, integrate code frequently, run the tests as often as possible. In other words, find and fix the defects before they even count as defects.

Companies that respond to customers a lot faster than their industry average can expect to grow three times faster and enjoy twice the profits of their competitors.[3] So there is a lot of competitive advantage available for the software development organization that can break the speed–quality compromise, and compete on the basis of time.

References

[1] Michael L. George and Stephen A. Wilson, Conquering Complexity in Your Business, McGraw-Hill, 2004, p.48.

[2] George Stalk and Rob Lachenauer, Hardball: Are you Playing to Play or Playing to Win, Harvard Business School Press, 2004

[3] George Stalk, Competing against Time: How time-based Competition is Reshaping Global Markets, Free Press, 2003, originally published in 1990. p.4.

Screen Beans Art, © A Bit Better Corporation

Team Compensation

The New Venture team had done an incredible job, and they knew it. Increment by increment they had built a new software product, and when the deadline came, everything that had to be operational was working flawlessly. The division vice president thanked everyone who had contributed to the effort at an afternoon celebration, and the team members congratulated each other as they relived some of the more harrowing moments of the last six months.

The next day the team’s Scrum Master was catching up on long-ignored e-mail when Dave, the development manager, called. “Say Sue,” he said, “Great job your team did! I’ve been waiting for the product launch before I bothered you with this, but the appraisal deadline is next week. I need your evaluation of each team member. And if you could, I’d like you to rank the team, you know, who contributed the most down to who contributed the least.”

Sue almost heard the air escaping as her world deflated. “I can’t do that,” she said. “Everyone pitched in 100%. We could not have done it otherwise.” “But Sue,” Dave said. “certainly there must have been a MVP. And a runner-up. And so on.” “No, not really,” Sue replied. “But what I can do is evaluate everyone’s contribution to the effort.”

Sue filled out an appraisal input form for each team member. She rated everyone’s performance, but found that she had to check the ‘far exceeded expectations’ box for each team member. After all, getting out the product on time was a spectacular feat, one that far exceeded everyone’s expectations.

Two days later Sue got a call from Janice in human resources. “Sue,” she said, “Great job your team did! And thanks for filling out all of those appraisal input forms. But really, you can’t give everyone a top rating. Your average rating should be ‘meets expectations’. You can only have one or two people who ‘far exceeded expectations’. Oh and by the way, since you didn’t rank the team members, would you please plan on coming to our ranking meeting next week? We are going to need your input on that. After all, at this company we pay for performance, and we need to evaluate everyone carefully so that our fairness cannot be questioned.”

Sue felt like a flat tire. In the past, when she had a particularly difficult problem she always consulted the team, and she decided to consult them once again. At 10:00 the next morning, the entire team listened as Sue explained her problem. The had always come up with a creative solutions to problems she presented before, and she could only hope they would be able to do it again. She thought she might convince them to elect an MVP or two, to help her put some variation into the evaluations.

Sue suspected that Dave and Janice might not approve of her approach, but she didn’t realize that when the team members heard her dilemma, they would deflate just as quickly as she had. The best they could do was insist that everyone had given 200% effort, they had all helped each other, and they had thought that every single person had done a truly outstanding job. They were not interested in electing a MVP, but they were willing to choose a LVP. It would the unnamed manager who was asking Sue to decide amongst them.

Now Sue really had a problem. She had no idea how to respond to Dave and Janice, and the New Venture team had turned angry and suspicious. Tomorrow they would have to start working together on the next release. How could something that was supposed to boost performance do such a through job of crushing the team’s spirit?

Deming’s View
Sue is not the only one who had trouble with merit pay evaluation and ranking systems. One of the greatest thought leaders of the 20th century, W Edwards Deming, wrote that un-measurable damage is created by ranking people, merit systems, and incentive pay. Deming believed that every business is a system and the performance of individuals is largely the result of the way the system operates. In his view, the system causes 80% of the problems in a business, and the system is management’s responsibility. He wrote that using exhortations and incentives to get individuals to solve management problems simply doesn’t work. Deming opposed ranking because it destroys pride in workmanship, and merit raises because they address the symptoms, rather than the causes, of problems.

It’s a bit difficult to take Deming at face value on this; after all, companies have been using merit pay systems for decades, and their use is increasing. Moreover Deming was mainly involved in manufacturing, so possibly his thinking does not apply directly to knowledge work like software development. Still, someone as wise as Deming is not to be ignored, so let’s take a deeper look into employee evaluation and reward systems, and explore what causes them to become dysfunctional.

Dysfunction #1: Competition
As the New Venture team instinctively realized, evaluation systems which rank people for purposes of merit raises pit individual employees against each other and strongly discourage collaboration. Even when the rankings are not made public, the fact that they happen does not remain a secret. Sometimes ranking systems are used as a basis for dismissing the lowest performers, making them even more threatening. When team members are in competition with each other for their livelihood, teamwork quickly evaporates.

Competition between teams, rather than individuals, may seem like a good idea, but it can be equally damaging. Once I worked in a division in which there were two separate teams developing software products that were targeting similar markets. The members of the team which attracted the largest market share were likely have more secure jobs and enhanced career opportunities. So each team expanded the capability of its product to attract a broader market. The teams ended up competing fiercely with each other for the same customer base as well as for division resources. In the end, both products failed. A single product would have had a much better chance at success.

Dysfunction #2: The Perception of Unfairness
There is no greater de-motivator than a reward system which is perceived to be unfair. It doesn’t matter if the system is fair or not, if there is a perception of unfairness, then those who think that they have been treated unfairly will rapidly loose their motivation.

People perceive unfairness when they miss out on rewards they think they should have shared. What if the vice president had given Sue a big reward? Even if Sue had had acknowledged the hard work of the team, they would probably have felt that she was profiting at their expense. You can be sure that Sue would have had a difficult time generating enthusiasm for work on the next release, even if the evaluation issues had not surfaced.

Here’s another scenario: What would have happened if the New Venture team was asked out to dinner with the VP and each member given a good sized bonus? The next day the operations people who worked late nights and weekends to help get the product out on time would have found out and felt cheated. The developers who took over maintenance tasks so their colleagues could work full time on the product would have also felt slighted. Other teams might have felt that they could have been equally successful, except that they got assigned to the wrong product.

Dysfunction #3: The Perception of Impossibility
The New Venture Team met their deadline by following the Scrum practice of releasing a high quality product containing only the highest priority functionality. But let’s try a different scenario: Let’s assume that the team was given a non-negotiable list of features that had to be done by a non-negotiable deadline, and let’s further speculate that the team was 100% positive that the deadline was impossible. (Remember this is hypothetical; surely this would never happen in real life.) Finally, let’s pretend that the team was promised a big bonus if they met the deadline.

There are two things that could happen in this scenario. Financial incentives are powerful motivators, so there is a chance that the team might have found a way to do the impossible. However, the more likely case is that the promise of a bonus that was impossible to achieve would make the team cynical, and the team would be even less motivated to meet the deadline than before the incentive was offered. When people find management exhorting them to do what is clearly impossible rather than helping to make the task possible, they are likely to be insulted by the offer of a reward and give up without half trying.

Dysfunction #4: Sub-Optimization
I recently heard of a business owner who offered testers $5 for every defect they could find in a product about to go into beta release. She thought this would encourage the testers to work harder, but the result was quite different. The good working relationship between developers and testers deteriorated as testers lost their incentive to help developers quickly find and fix defects before they propagated into multiple problems. After all, the more problems the testers found, the more money they made.

When we optimize a part of a chain, we invariably sub-optimize overall performance. One of the most obvious examples of sub-optimization is the separation of software development from support and maintenance. If developers are rewarded for meeting a schedule even if they deliver brittle code without automated test suites or an installation process, then support and maintenance of the system will cost far more than was saved during development.

Dysfunction #5: Destroying Intrinsic Motivation
There are two approaches to giving children allowances. Theory A says that children should earn their allowances, so money is exchanged for work. Theory B says that children should contribute to the household without being paid, so allowances are not considered exchange for work. I know one father who was raised with Theory B, but switched to Theory A for his children. He put a price on each job and paid the children weekly for the jobs they had done. This worked for a while, but then the kids discovered that they could choose amongst the jobs and avoid doing the ones they disliked. When the children were old enough to earn a paycheck, they stopped doing household chores altogether, and the father found himself mowing the lawn along side the teenage children of his neighbors. Were he to do it again, this father says he would not tie allowance to work.

In the same way, once employees get used to receiving financial rewards for meeting goals, they begin to work for the rewards, not the intrinsic motivation that comes from doing a good job and helping their company be successful. Many studies have shown that extrinsic rewards like grades and pay will, over time, destroy the intrinsic reward that comes from the work itself.

One Week Later
Sue was nervous as she entered the room for the ranking meeting. She had talked over her problem with Wayne, her boss, and although he didn’t have any easy solutions, he suggested that she present her problem to the management team. Shortly after the meeting started, Janice asked Sue how she would rank her team members. Sue took a deep breath, got a smile of encouragement from Wayne, and explained how the whole idea of ranking made no sense for a team effort. She explained how she had asked for advice from the team and ended up with an angry and suspicious team.

“You should never have talked to the team about this,” said Janice. “Hold on a minute,” Wayne jumped in. “I thought our goal in this company is to be fair. How can we keep our evaluation policies secret and expect people to consider them fair? It doesn’t matter if we think they are fair, it matters if employees think they are fair. If we think we can keep what we are doing a secret, we’re kidding ourselves. We need to be transparent about how we operate; we can’t make decisions behind closed doors and then try to tell people ‘don’t worry, we’re being fair’”

Sue was amazed at how fast the nature of the discussion changed after Wayne jumped to her defense. Apparently she wasn’t the only one who thought this ranking business was a bad idea. Everyone agreed that the New Venture team had done an excellent job, and the new product was key to their business. No one had thought that it could be done, and indeed the team as a whole had far exceeded everyone’s expectations. It became apparent that there wasn’t a person in the room who was willing to sort out who had contributed more or less to the effort, so Sue’s top evaluation for every team member was accepted. More importantly, the group was concerned that a de-motivated New Venture team was a serious problem. Eventually the vice president agreed to go to the next meeting of the New Venture team and discuss the company’s evaluation policies. Sue was sure that this would go a long way to revitalize the team spirit.

Now the management team had a problem of its own. The knew that they had to live within a merit pay system, but they suspected they needed to rethink the way it was implemented. Since changes like that don’t happen overnight, they formed a committee to look into various evaluation and pay systems.
The committee started by agreeing that evaluation systems should not be used to surprise employees with unexpected feedback about their performance. Performance feedback loops must be far shorter than an annual, or even a quarterly, evaluation cycles. Appraisals are good times to review and update development plans for an employee, but if this is the only time an employee finds out how they are doing, a lot more needs fixing than the appraisal system.

With this disclaimer in mind, the committee developed some guidelines for dealing with various forms of differential pay systems.

Guideline #1: Make Sure The Promotion System Is Unassailable
In most organizations, significant salary gains come from promotions which move people to a higher salary grade, not merit increases. Where promotions are not available, as is the case for many teachers, merit pay systems have a tendency to become contentious, because merit increases are the only way to make more money. When promotions are available, employees tend to ignore the merit pay system and focus on the promotion system. Of course this system of promotions tends to encourage people to move into management as they run out of promotional opportunities in technical areas. Companies address this problem with ‘dual ladders’ that offer management-level pay scales to technical gurus.

The foundation of any promotion system is a series of job grades, each with a salary range in line with industry standards and regional averages. People must be placed correctly in a grade so that their skills and responsibilities match the job requirements of their level. Initial placements and promotion decisions should be carefully made and reviewed by a management team.

Usually job grades are embedded in titles, and promotions make the new job grade public through a new title. Thus a person’s job grade is generally considered public information. If employees are fairly placed in their job grade, and promoted only when they are clearly performing at the new job grade, then salary differences based on job grade are generally perceived to be fair. Thus a team can have both senior and junior people, generalists and highly skilled specialists, all making different amounts of money. As long the system of determining job grades and promotions is transparent and perceived to be fair, this kind of differential pay is rarely a problem.

The management team at Sue’s company decided to focus on a promotion process that did not use either a ranking or a quota system. Instead, clear promotion criteria would be established for each level, and when someone had met the criteria, they would be eligible for promotion. A management committee would review each promotion proposal and gain a consensus that the promotion criteria were met. This would be similar to existing committees that reviewed promotions to fill open supervisor or management positions.

Guideline #2: De-emphasize The Merit Pay System
When the primary tool for significant salary increases is promotion, then it’s important to focus as much attention as possible on making sure the promotion system is fair. When it comes to the evaluation system that drives merit pay, it’s best not try too hard to sort people out. Studies show that when information sharing and coordination are necessary, organizations that reduce pay differences between the highest and the lowest paid employees tend to perform better over time.

Use evaluations mainly to keep everyone at an appropriate level in their salary grade. Evaluations might flag those who are ready for promotion and those who need attention, but that should trigger a separate promotion or corrective action process. About four evaluation grades are sufficient, and a competent supervisor with good evaluation criteria and input from appropriate sources can make fair evaluations that accomplish these purposes.

Even when annual raises are loosely coupled to merit, evaluations will always be a big deal for employees, so attention should be paid to making them fair and balanced. Over the last decade, balanced scorecards have become popular for management evaluations; at least in theory, balanced scorecards ensure that the multiple aspects of a manager’s job all receive attention. A simple version of a balanced scorecard might also be used for a merit pay evaluations, to emphasize the fact that people must perform well on many dimensions to be effective. A supervisor might develop a scorecard with each employee that takes into account team results, new competencies, leadership, and so on. It is important that employees perceive that the input to a scorecard is valid and fairly covers the multiple aspects of their job. It is important to keep things simple, because too much complexity will unduly inflate the attention paid to a pay system which works better when it is understated. Finally, scorecards should not be used to feed a ranking system.

Guideline #3: Tie Profit Sharing To Economic Drivers
Nucor Steel decided to get into the steel business in 1968, and thirty years later it was the biggest steel company in the US. When Nucor started up, Bethlehem Steel considered it a mere gnat, but 35 years later Bethlehem Steel was not only bankrupt, but sold off for assets. So Nucor Steel is one very successful company that has done a lot of things right in a tough industry. Quite surprisingly, Nucor has a decades-old tradition of paying for performance. How does the company avoid the dysfunctions of rewards?
Nucor Steel started with the realization that profit per ton of finished steel as its key economic driver, and based its profit sharing plan on the contribution a team makes to improving this number. So for example, a team that successfully develops a new steel-making process or starts up a new plant on schedule will not get see an increase in pay until the process or plant has improved the company’s profit per ton of steel. Thus Nucor avoids sub-optimization by tying its differential pay system as close to the economic driver of its business as possible.

Guideline # 4: Reward Based on Span of Influence, Not Span of Control
Conventional wisdom says that people should be evaluated based on results that are under their control. However, this kind of evaluation creates competition rather than collaboration. Nucor makes sure that its profit sharing formula rewards relatively large teams, not just the individuals or small groups who have direct responsibility for an area. Following this principle, if a software program creates a significant profit increase, everyone from those who brought the idea into the company to developers and testers to operations and support people to the end users should share in any reward.

Nucor Steel works hard to create a learning environment, where experts move from one plant to another, machine operators play a significant role in selecting and deploying new technology, and tacit knowledge spreads rapidly throughout the company. It’s reward system encourages knowledge sharing by rewarding people for influencing the success of areas they do not control.

How, exactly, can rewards be based on span of influence rather than span of control? I recommend a technique called ‘Measure UP’. No matter how hard you try to evaluate knowledge work or how good a scorecard you create, something will go unmeasured. Over time, the unmeasured area will be de-emphasized and problems will arise. We have a tendency to add more measurements to the scorecard to draw attention to the neglected areas.

However, it is a lot easier to catch everything that falls between the cracks by reducing the number of measurements and raising them to a higher level. For instance, instead of measuring software development with cost and schedule and earned value, try creating a P&L or ROI for the project, and help the team use these tools to drive tradeoff decisions.

Guideline #5: Find Better Motivators Than Money
While monetary rewards can be a powerful driver of behavior, the motivation they provide is not sustainable. Once people have an adequate income, motivation comes from things such as achievement, growth, control over one’s work, recognition, advancement, and a friendly working environment. No matter how good your evaluation and reward system may be, don’t expect it to do much to drive stellar performance over the long term.

In the book “Hidden Value” , Charles O’Reilly and Jeffrey Pfeffer present several case studies of companies that obtain superb performance from ordinary people. These companies have people-centered values which are aligned with actions at all levels. They invest in people, share information broadly, and rely on teams, and emphasize leadership rather than management. Finally, they do not use money as a primary motivator; they emphasize the intrinsic rewards of fun, growth, teamwork, challenge and accomplishment.

Treat monetary rewards like explosives because they have will a powerful impact whether you intend it or not. So use them lightly and with caution. They can get you in trouble much faster than they can solve your problems. Once you go down the path of monetary rewards, you may never be able to go back, even when they cease to be effective, as they inevitably will. Make sure that people are fairly and adequately compensated, and then move on to more effective ways to improve performance.

Six Months Later
The New Venture Team is having another celebration. They had all been surprised when the VP came to their team meeting six months earlier. But they quickly recovered and told her that they each wanted to be the best, they wanted to work with the best, and they did not appreciate the implication that some of them were better than others. When the VP left, the team cheered Sue for sticking up for them, and then got down to work with renewed enthusiasm. Now, two releases later, the customers were showing their appreciation with their pocketbooks.

There haven’t been any dramatic pay increases and only the occasional, well-deserved promotion. But the company has expanded its training budget and New Venture team members have found themselves mentoring other teams. Sue is rather proud of them all as she fills out the newly revised appraisal input forms that have more team-friendly evaluation criteria. This time Sue is confident that her judgment will not be questioned.

Published in Better Software Magazine July, 2004 

Screen Beans Art, © A Bit Better Corporation

An Introduction to Lean Software Development

Gustaf Brandberg, CEO and co-founder of Citerus, a software consulting firm in Uppsala, Sweden, conducted this interview, featured in PNEHM! #2 2004, a newsletter on software development in Swedish, produced and distributed by Citerus.

Gustaf Brandberg: What is Lean Software Development?

Mary Poppendieck: Lean Software Development is the application of Lean Thinking to the software development process. Organizations that are truly lean have a strong competitive advantage because they respond very rapidly and in a highly disciplined manner to market demand, rather than try to predict the future. Similarly, Lean Software Development is the discipline of creating software that readily adapts to changes in its domain. There is a pendulum that swings from one extreme to the other – first there was ad-hoc software development, which could not scale to large complex systems. Then the pendulum swung far to the other side and heavy process discipline was favored, but this proved unresponsive to change in a world where change is constant. Lean Software Development provides a middle ground: high discipline along with high responsiveness to change.

Gustaf: When did you first start working according to Lean principles? From where did you get the inspiration?

Mary: I was working in a manufacturing plant making video tapes in the mid 1980’s, and our Japanese competitors were selling video tapes at half of what it cost us to make them. We needed to understand how they could do this, and we discovered Just-in-Time production. We did two things: first, we provided every single worker in the plant with training in how a Just-in-Time flow works and had the workers design the details of a pull system. Then we stopped scheduling each workstation and instead sent a weekly schedule to the packing station. It worked like magic; inventory disappeared, quality improved, costs dropped, and customer response time was a week instead of a month.

Gustaf: You believe that there is no such thing as a 'best' practice. Why is that?

Mary: Frederick Winslow Taylor wrote The Principles of Scientific Management in 1911. In it, he proposed that manufacturing should be broken down into very small steps, and then industrial engineers should determine the ‘one best way’ to do each step. This ushered in the era of mass production, with ‘experts’ telling workers the ‘one best way’ to do their jobs.

The Toyota Production System is founded on the principles of the Scientific Method, instead of Scientific Management. The idea is that no matter how good a process is, it can always be improved, and that the workers doing the job are the best people to figure out how to do it better. In Lean Production, workers learn how to create a hypothesis, test it, analyze the results, and – if the data supports the hypothesis – make the change permanent.

Software development covers a lot of territory, and no matter how good a practice may be, it will not apply universally across all software development environments. Moreover, even where a practice does apply, it can and should always be improved upon. There are certainly underlying principles that do not change. These principles will develop into different practices in different domains, driven by the economic reality of each environment.

Gustaf: What are these core principles in Lean Software Development you are referring to?

Mary:

• Eliminate Waste – Do only what adds value for a customer, and do it without delay.


• Amplify Learning – Use frequent iterations and regular releases to provide feedback.


• Delay Commitment – Make decisions at the last responsible moment.


• Deliver Fast – The measure of the maturity of an organization is the speed at which it can repeatedly and reliably respond to customer need.


• Empower the Team – Assemble an expert workforce, provide technical leadership and delegate the responsibility to the workers.


• Build Integrity In – Have the disciplines in place to assure that a system will delight customers both upon initial delivery and over the long term.


• See the Whole – Use measurements and incentives focused on achieving the overall goal.

Gustaf: OK. You say I need to eliminate waste, but how do I recognize it? Where should I start looking?

Mary: Something like 45% of the features in a typical software system are never used, another 19% are rarely used. That means two thirds of the software in a typical system is waste. Eliminating this waste is the first place to look for reducing software waste. We need to focus on creating more value with less effort, and make sure that the resulting systems do not turn into legacy software.

Gustaf: If what you are saying is true, all that needless functionality is certainly a waste. How come so many features are never or rarely used?

Mary: Quite often at the beginning of a software development project, we ask customers what they want, even though they don’t really know. We make it clear to customers that they need to tell us about everything they might possibly want, and quite often we record their wish list without question. This is what we call ‘Scope’. Later, if customers want to add or change items in the ‘Scope’ we challenge them with a ‘Change Review Process’. So we reward customers for coming up with a long initial list of features, and punish them if they want to modify the list at a later stage. Is it any wonder that two thirds of the features in a system developed playing this game are rarely or never used?

Gustaf: So, I realize putting in extra features is a big source of waste. In your book, you write that another way of discovering waste is mapping your value stream. What is a value stream?

Mary: In the physical product world, a value stream is the flow of a product from raw material to final use. For instance, a Cola can starts out as bauxite, is reduced to alumina, smelted into aluminum ingots, rolled into sheets, rolled again into thin strips, stamped into disks, formed into cans, cleaned and painted, filled with Cola and sealed, packaged and put on a pallet, warehoused at a distributor, sent to a retail store, put on a shelf, purchased by a consumer, stored in a refrigerator, and finally the Cola is consumed. At every step, a huge pile of inventory builds up waiting for the next step, which is days or weeks away. This value stream takes 319 days, of which only 3 hours (less than .04%) are spent actually making the product.

In the software world, a value stream starts when a business unit decides that better information would help it increase revenue or reduce cost. This is the start of the value stream. The value stream ends when the deployed software starts generating the extra revenue or reducing costs. Inventory in the software development value stream is partially done work: requirements that are not analyzed and designed, designs that are not coded, code that is not tested and integrated, features that are not deployed, and deployed features that are not saving money or reducing costs. When the software value stream has as little of this partially done work as possible, risks are reduced and productivity is greatly
improved.

Gustaf: Another principle is delaying commitment. Isn't the project schedule in danger of slipping when no one dares make a decision?

Mary: A military officer who was about to retire once said: ‘The most important thing I did in my career was to teach young leaders that whenever they saw a threat, their first job was to determine the timebox for their response. Their second job was to hold off making a decision until the end of the timebox, so that they could make it based on the best possible data.’

Our natural tendency is to make decisions and get them over with. However, it is far better to determine the timebox for every decision, and then make the decision at the end of the timebox, because then we can make decisions based on the best possible data. In Lean Software Development, decisions are not avoided; they are scheduled and made at the last responsible moment. This assures that all decisions are made in a timely manner, yet they are made with as much information as possible to help make the best decision possible.

Gustaf: What do you mean by ‘integrity’? Why is it so important to maintain it?

Mary: Integrity is a level above quality; it is the thing that makes people want to use a product. For instance, there are many high quality search engines, but only one Google. There is something about Google that attracts people to use it every day. That is integrity. Software is never an end in itself, it is always a means to an end. It gets used when it provides the best means to the end. To maintain integrity, software needs regular updates to make sure that it continues to provide the best way for users to achieve their goals, even as the goals and the technology changes. Otherwise the software becomes irrelevant at best, or legacy at worst.

Gustaf: Why have not all organizations been successful in applying Lean Thinking? What are the most frequent pitfalls?

Mary: At its core, Lean Thinking means creating a learning environment for workers in order to increase the flow of value. All too often, practices from successful Lean companies are adopted, but the heart of Lean Thinking is lost. If people, learning and value are not the central focus of a Lean initiative, it will not be particularly successful.

Last week I explained to a president of a small software company how to implement responsibility-based planning and control: schedule releases and iterations, make sure that the development team agrees at the start on what features will be included, and then leave it to the team to meet their commitment. This made the president of the company uneasy. He was already using iterations, but he wasn’t quite ready to give up the responsibility for planning and tracking tasks to the development team.

Think of an emergency response team, firefighters or paramedics for example. They are trained in emergency scenarios that establish patterns for responding to anything they are likely to encounter. When an emergency occurs, there is no time for decisions to go up the chain of command and back down; emergency responders are expected to use their own judgment to deal with an emergency as they confront it. Similarly in a Lean organization, everyone is trained and equipped to do their job, and the organization is structured so that it is clear what needs to be accomplished. But it is the workers who make decisions about what to do, track their own work, and assume the responsibility for meeting their goals.

Gustaf: This sounds a lot like Scrum to me. In Scrum, the team is responsible for managing itself. How are Agile methodologies such as Scrum and Extreme Programming related to Lean Software Development?

Mary: Both Scrum and Extreme Programming (XP) are examples of Lean Thinking applied to developing software. Scrum is an excellent approach to responsibility-based planning and control. XP is a tremendous set of disciplines that enable rapid, repeatable, reliable delivery of code. I particularly like XP’s focus on testing, continuous integration and refactoring. I think of refactoring as constantly improving the code base – it’s sort of like applying kaizen (the Japanese word for continuous improvement) to a software system.

Gustaf: Where can I learn more about Lean Software Development?

Mary: A good start would be my book: Lean Software Development: An Agile Toolkit. Other recommended reading is Lean Thinking: Banish Waste and Create Wealth in Your Corporation, 2nd edition (by Jones & Womack), Slack: Getting Past Burnout, Busywork, and the Myth of Total Efficiency (DeMarco) and Product Development for the Lean Enterprise (Kennedy). Also,check my own website and articles on Lean Development at Agile Alliance.

Screen Beans Art, © A Bit Better Corporation