An Introduction to Lean Software Development

Gustaf Brandberg, CEO and co-founder of Citerus, a software consulting firm in Uppsala, Sweden, conducted this interview, featured in PNEHM! #2 2004, a newsletter on software development in Swedish, produced and distributed by Citerus.

Gustaf Brandberg: What is Lean Software Development?

Mary Poppendieck: Lean Software Development is the application of Lean Thinking to the software development process. Organizations that are truly lean have a strong competitive advantage because they respond very rapidly and in a highly disciplined manner to market demand, rather than try to predict the future. Similarly, Lean Software Development is the discipline of creating software that readily adapts to changes in its domain. There is a pendulum that swings from one extreme to the other – first there was ad-hoc software development, which could not scale to large complex systems. Then the pendulum swung far to the other side and heavy process discipline was favored, but this proved unresponsive to change in a world where change is constant. Lean Software Development provides a middle ground: high discipline along with high responsiveness to change.

Gustaf: When did you first start working according to Lean principles? From where did you get the inspiration?

Mary: I was working in a manufacturing plant making video tapes in the mid 1980’s, and our Japanese competitors were selling video tapes at half of what it cost us to make them. We needed to understand how they could do this, and we discovered Just-in-Time production. We did two things: first, we provided every single worker in the plant with training in how a Just-in-Time flow works and had the workers design the details of a pull system. Then we stopped scheduling each workstation and instead sent a weekly schedule to the packing station. It worked like magic; inventory disappeared, quality improved, costs dropped, and customer response time was a week instead of a month.

Gustaf: You believe that there is no such thing as a 'best' practice. Why is that?

Mary: Frederick Winslow Taylor wrote The Principles of Scientific Management in 1911. In it, he proposed that manufacturing should be broken down into very small steps, and then industrial engineers should determine the ‘one best way’ to do each step. This ushered in the era of mass production, with ‘experts’ telling workers the ‘one best way’ to do their jobs.

The Toyota Production System is founded on the principles of the Scientific Method, instead of Scientific Management. The idea is that no matter how good a process is, it can always be improved, and that the workers doing the job are the best people to figure out how to do it better. In Lean Production, workers learn how to create a hypothesis, test it, analyze the results, and – if the data supports the hypothesis – make the change permanent.

Software development covers a lot of territory, and no matter how good a practice may be, it will not apply universally across all software development environments. Moreover, even where a practice does apply, it can and should always be improved upon. There are certainly underlying principles that do not change. These principles will develop into different practices in different domains, driven by the economic reality of each environment.

Gustaf: What are these core principles in Lean Software Development you are referring to?

Mary:

• Eliminate Waste – Do only what adds value for a customer, and do it without delay.


• Amplify Learning – Use frequent iterations and regular releases to provide feedback.


• Delay Commitment – Make decisions at the last responsible moment.


• Deliver Fast – The measure of the maturity of an organization is the speed at which it can repeatedly and reliably respond to customer need.


• Empower the Team – Assemble an expert workforce, provide technical leadership and delegate the responsibility to the workers.


• Build Integrity In – Have the disciplines in place to assure that a system will delight customers both upon initial delivery and over the long term.


• See the Whole – Use measurements and incentives focused on achieving the overall goal.

Gustaf: OK. You say I need to eliminate waste, but how do I recognize it? Where should I start looking?

Mary: Something like 45% of the features in a typical software system are never used, another 19% are rarely used. That means two thirds of the software in a typical system is waste. Eliminating this waste is the first place to look for reducing software waste. We need to focus on creating more value with less effort, and make sure that the resulting systems do not turn into legacy software.

Gustaf: If what you are saying is true, all that needless functionality is certainly a waste. How come so many features are never or rarely used?

Mary: Quite often at the beginning of a software development project, we ask customers what they want, even though they don’t really know. We make it clear to customers that they need to tell us about everything they might possibly want, and quite often we record their wish list without question. This is what we call ‘Scope’. Later, if customers want to add or change items in the ‘Scope’ we challenge them with a ‘Change Review Process’. So we reward customers for coming up with a long initial list of features, and punish them if they want to modify the list at a later stage. Is it any wonder that two thirds of the features in a system developed playing this game are rarely or never used?

Gustaf: So, I realize putting in extra features is a big source of waste. In your book, you write that another way of discovering waste is mapping your value stream. What is a value stream?

Mary: In the physical product world, a value stream is the flow of a product from raw material to final use. For instance, a Cola can starts out as bauxite, is reduced to alumina, smelted into aluminum ingots, rolled into sheets, rolled again into thin strips, stamped into disks, formed into cans, cleaned and painted, filled with Cola and sealed, packaged and put on a pallet, warehoused at a distributor, sent to a retail store, put on a shelf, purchased by a consumer, stored in a refrigerator, and finally the Cola is consumed. At every step, a huge pile of inventory builds up waiting for the next step, which is days or weeks away. This value stream takes 319 days, of which only 3 hours (less than .04%) are spent actually making the product.

In the software world, a value stream starts when a business unit decides that better information would help it increase revenue or reduce cost. This is the start of the value stream. The value stream ends when the deployed software starts generating the extra revenue or reducing costs. Inventory in the software development value stream is partially done work: requirements that are not analyzed and designed, designs that are not coded, code that is not tested and integrated, features that are not deployed, and deployed features that are not saving money or reducing costs. When the software value stream has as little of this partially done work as possible, risks are reduced and productivity is greatly
improved.

Gustaf: Another principle is delaying commitment. Isn't the project schedule in danger of slipping when no one dares make a decision?

Mary: A military officer who was about to retire once said: ‘The most important thing I did in my career was to teach young leaders that whenever they saw a threat, their first job was to determine the timebox for their response. Their second job was to hold off making a decision until the end of the timebox, so that they could make it based on the best possible data.’

Our natural tendency is to make decisions and get them over with. However, it is far better to determine the timebox for every decision, and then make the decision at the end of the timebox, because then we can make decisions based on the best possible data. In Lean Software Development, decisions are not avoided; they are scheduled and made at the last responsible moment. This assures that all decisions are made in a timely manner, yet they are made with as much information as possible to help make the best decision possible.

Gustaf: What do you mean by ‘integrity’? Why is it so important to maintain it?

Mary: Integrity is a level above quality; it is the thing that makes people want to use a product. For instance, there are many high quality search engines, but only one Google. There is something about Google that attracts people to use it every day. That is integrity. Software is never an end in itself, it is always a means to an end. It gets used when it provides the best means to the end. To maintain integrity, software needs regular updates to make sure that it continues to provide the best way for users to achieve their goals, even as the goals and the technology changes. Otherwise the software becomes irrelevant at best, or legacy at worst.

Gustaf: Why have not all organizations been successful in applying Lean Thinking? What are the most frequent pitfalls?

Mary: At its core, Lean Thinking means creating a learning environment for workers in order to increase the flow of value. All too often, practices from successful Lean companies are adopted, but the heart of Lean Thinking is lost. If people, learning and value are not the central focus of a Lean initiative, it will not be particularly successful.

Last week I explained to a president of a small software company how to implement responsibility-based planning and control: schedule releases and iterations, make sure that the development team agrees at the start on what features will be included, and then leave it to the team to meet their commitment. This made the president of the company uneasy. He was already using iterations, but he wasn’t quite ready to give up the responsibility for planning and tracking tasks to the development team.

Think of an emergency response team, firefighters or paramedics for example. They are trained in emergency scenarios that establish patterns for responding to anything they are likely to encounter. When an emergency occurs, there is no time for decisions to go up the chain of command and back down; emergency responders are expected to use their own judgment to deal with an emergency as they confront it. Similarly in a Lean organization, everyone is trained and equipped to do their job, and the organization is structured so that it is clear what needs to be accomplished. But it is the workers who make decisions about what to do, track their own work, and assume the responsibility for meeting their goals.

Gustaf: This sounds a lot like Scrum to me. In Scrum, the team is responsible for managing itself. How are Agile methodologies such as Scrum and Extreme Programming related to Lean Software Development?

Mary: Both Scrum and Extreme Programming (XP) are examples of Lean Thinking applied to developing software. Scrum is an excellent approach to responsibility-based planning and control. XP is a tremendous set of disciplines that enable rapid, repeatable, reliable delivery of code. I particularly like XP’s focus on testing, continuous integration and refactoring. I think of refactoring as constantly improving the code base – it’s sort of like applying kaizen (the Japanese word for continuous improvement) to a software system.

Gustaf: Where can I learn more about Lean Software Development?

Mary: A good start would be my book: Lean Software Development: An Agile Toolkit. Other recommended reading is Lean Thinking: Banish Waste and Create Wealth in Your Corporation, 2nd edition (by Jones & Womack), Slack: Getting Past Burnout, Busywork, and the Myth of Total Efficiency (DeMarco) and Product Development for the Lean Enterprise (Kennedy). Also,check my own website and articles on Lean Development at Agile Alliance.

Screen Beans Art, © A Bit Better Corporation

Why the Lean in Lean Six Sigma?

A free-wheeling mid-sized company ran smack up against the Sarbanes-Oxley Act a few months ago and found that it had to pause and take stock.[1] The company commissioned a small team to map out every place that financial data moved, and to no one’s surprise, it uncovered a lot of un-coordinated, manual processes. Next the team put into place some simple automation that managed inventory and financial data, an online order entry system, and even some automated human resources capabilities. The company found that with these more streamlined processes, days were cut out of order processing, inventories were reduced, and productivity was significantly improved.

The irony is that the fast-moving company had to slow down to speed up; more discipline led to higher speed. This is not an isolated case, there is a direct relationship between speed and discipline. Adding discipline should streamline a process, and streamlined processes don’t work without discipline.

When I first heard the term Lean Six Sigma, I wondered what Lean added to Six Sigma. I found that the answer is speed. The first principle of Lean Six Sigma is: Delight your customers with speed and quality. The second principle says: Improve process flow and speed. Lean Six Sigma emphasizes that speed is directly tied to excellence.

Speed is not the same thing as schedule. Schedule is about when something is supposed to get done, speed is about how fast it gets done. Speed has a bad reputation; it is often equated with hasty, undisciplined work. But if Lean Six Sigma has anything to teach us, it is that we should be looking for opportunities to streamline our core processes. This does not mean we should be compressing already tight schedules. It means that we first determine what our core processes are, and then focus on making them flow smoothly.

For example, core processes in software development would be naming conventions and coding standards, a configuration management system, an automated build process, a suite of automated unit tests that are built and maintained as part of the code, daily build/integrate/test cycles, acceptance testing integrated into the development process, and usability testing immediately after the features are implemented. Assuring that these disciplines are in place is fundamental to the smooth flow of any software development process.

However, the most important process to streamline in a development project is the knowledge-creating process. Whether we are developing a new product or a new software system, the fundamental thing we are doing is discovering what needs to be in the system in order to delight the customer. Lean thinking supports two basic disciplines for speeding up the knowledge creation process: short, frequent learning cycles and delayed commitment.

Short, frequent learning cycles are the antithesis of thorough front-end planning, but they are the best approach to processes which have the word ‘development’ in their title. In product development, for example, the typical process is to define requirements, choose a solution, refine the solution, and implement the solution. Although this process is common, it is not the best way to generate knowledge.

Toyota has a different approach, one that is much faster and delivers products of superior quality that consistently outsell the competition.[2] Toyota builds sets of possibilities to satisfy customer needs and then, through a series of combining and narrowing, the new product emerges. The combining and narrowing process is paced by timeboxed milestones that define stages of the narrowing process. Milestones are always met, despite the fact that there are no task breakouts or tracking. Decisions are delayed as long as possible, so that they can be based on the maximum amount of information.

When a project involves knowledge creation, rather than just knowledge replication, speed and quality come from improving the flow of creating knowledge. Many of our project management practices have a tendency to impede knowledge creation by forcing early choices and reducing the number of possibilities explored. This often leads us down bind alleys: after a lot of work has gone into a solution, we learn that it has a fatal flaw and we have to retrace our steps and repeat a whole lot of work.

The key to streamlining a development process is to clearly distinguish between true knowledge creating iterations and iterations that lead down blind alleys. Knowledge creating iterations explore multiple options and leave as many possibilities open as possible, delaying decisions until the last responsible moment.

Returning to our software development example, lean practices promote speed and flexibility by implementing core disciplines that promote change tolerance and allow decisions to be delayed as long as possible. Lean software development changes the focus from gathering requirements to encoding all requirements in tests. It introduces the concept of refactoring, that is, creating a simple design at the beginning of development to handle early requirements, and then improving the design later as more requirements are discovered. Finally, lean software development requires full testing and integration of code as soon as it is developed, on a daily basis at minimum. The result is easily maintainable code that has been designed to be flexible and built to be rapidly changed.

In lean software development, scope is not set at the beginning; small features sets are added based on priority determined by their ROI.[3] This tends to lead to a significant increase in both speed and productivity for a simple reason: most of the features we put into software systems are never going to be used. How can this be? When we freeze scope early, we encourage our customers, who don’t really know what they want, to ask for everything they can imagine. When we delay commitment on scope until we are well into the knowledge generating process, we end up reducing scope down to the minimum set that is really going to pay off.

If we’re not careful, Six Sigma might lead us to apply practices aimed at improving replication processes to knowledge generating processes. This often leads to slow, unresponsive, change-intolerant practices that are not appropriate for knowledge creation. Quite often the slow, deliberate nature of these practices is mistaken to be a sign of good discipline. But when we add Lean to Six Sigma, we discover that we need to re-think these slow processes, because we have come to understand that speed, discipline, and excellence go hand-in-hand.

References
[1] “Better Safe Than Sorry,” by Jeffrey Rothfeder, CIO Insight, February 2004

[2] Product Development for the Lean Enterprise, by Michael N. Kennedy, Oaklea Press, 2003.

[3] Software by Numbers, by Mark Denne and Jane Cleland-Huang, Prentice Hall, 2004

Screen Beans Art, © A Bit Better Corporation

Product Development for the Lean Enterprise (Book Review)

“How could a business book keep me up until 2:30 in the morning?” I wondered as I collapsed into bed. True, it was a business novel, so it had engaging characters, a hint of a plot, and actual villains. But that wasn’t what kept me reading into the wee hours of the morning. All through the book I kept applauding Michael Kennedy for doing such an excellent job of showing how to apply lean thinking to product development. “He gets it!” I kept saying to myself. “He understands that product development is a whole different ballgame than manufacturing.”

The book’s narrator is three weeks from retirement and coasting. His boss has just been asked to take over a flagging engineering department, and our narrator gets a week to assemble a swat team and figure out how to revamp the product development process. Buried deep in the organization is an engineer who has studied how Toyota does product development. We follow her as she first convinces the swat team and then the executives that they need to change the way they think about product development.

We learn that product development is a knowledge-creating process, and learn what that means: Entrepreneurial leadership, responsibility-based planning, expert workforce, and set-based concurrent engineering. Before you fall asleep, note that the novel format helps make these concepts come alive. The swat team has to digest a new paradigm for product development, and then sell it to an executive who has the mistaken notion that lean product development should be pretty much like lean manufacturing.

After the book convinces you that you can’t treat development like production, it goes on to describe in detail what does work for development in an understandable and practical way. I learned a lot about set-based design, and I really liked the description of responsibility-based planning and control.

The book ends with a ‘where do we go from here’ section, offering large group interventions as a way to trigger participative change. The book didn’t offer a lot of guidance on exactly what to do, but it does draw an interesting parallel between the desired new product development process and the change process itself.

Business novels, by their nature, provide ideas and examples, rather than specifics of how proceed in situations that differ from the ideal presented in the novel. But a well-written business novels such as this one provide good way to clarify important ideas in a quick-to-read, entertaining style. It’s easier to attack sacred cows in fiction, and easier help the reader visualize what might happen if a real paradigm shift takes place. If you liked Goldratt’s novel “The Goal” then you might stay up most of the night reading this book, just like I did.

Reference
Product Development for the Lean Enterprise by Michael N. Kennedy, Oakela Press, 2003.

Incremental Funding (Book Review)

The customer wanted an on-line currency exchange capability added to their on-line financial service offerings. They figured it would take several months to implement. But the development team suggested a different approach: start by hiring a dozen telephone operators and implement the necessary software for these folks to execute currency trades. The company gave it a try, and in six weeks the first iteration was ready. With no more than an 800 number on their web site and a rudimentary interface to the currency market, new business was being transacted and profits being made.

Over the course of the next several months, the on-line trading capability was implemented around the core module originally used by the telephone operators. Not only did the company see early revenue, but the risk of failure disappeared once trading started. In addition, system requirements were defined by observing real trades.

In the book Software by Numbers – Low Risk, High Return Development (Prentice Hall, 2004) Mark Denne and Jane Cleland-Huang make the case for incremental delivery of software. Mark Denne developed the Incremental Funding Methodology (IFM) in the 1990’s to help land a large software development contract. In an attempt to distinguish his bid from the pack, he reorganized the deliveries into units of value, and adjusted the development sequence so that the customer would realize revenue faster and in the end, receive a greater return on their investment. The customer discovered that this approach dramatically reduced their need to borrow money and gave them earlier product release with lowered risk. In what seemed like hotly competitive bidding process, Mark’s company won the bid by emphasizing “time to value” instead of development efficiency.

Software by Numbers recommends dividing a project into Minimum Marketable Features (MMF’s). These are small feature sets which deliver some identifiable value to the customer. Each MMF should have its own return on investment (ROI). By laying out the potential ROI’s of various feature sets, an optimal development sequence for the MMF’s can be determined. Early deployment of key MMF’s reduces risk while generating revenue to help fund the remainder of the project.

When business people and software developers focus on identifying and valuing marketable features, their conversation is changed. Developers are exposed to ROI and stakeholders are confronted with the realities of software development. The entire team is focused on achieving business ROI early in the development cycle. Management sees continuously measurable progress, and the team benefits from the early and compelling feedback generated by real software being used in production.

With so many benefits, why wouldn’t IFM be the preferred approach for developing software? Denne and Cleland-Huang note that many practitioners view software architecture as a monolithic whole, requiring early definition because of the extensive impact that architectural changes can have on a system. On the other hand, they argue, it is not until the details of an architecture are implemented that one can tell if the architecture is viable. Thus architecture presents us with a chicken and egg problem.

The book recommends that architectural elements which support each set of MMF’s should be developed with their respective MMF’s. In other words, architectural development should be sequenced using the same financially driven priorities as feature development. While there may be times where architectural coherency may dictate early development of features not immediately related to the current feature set, in general it is not only possible but also preferable to defer implementation of architectural elements until the features requiring these elements is being developed.

As we discover that architecture not only can evolve, but in fact, in any deployed system, the architecture will evolve over time, the view that architecture must be fixed early in the development process becomes a liability. This is a view that creates architectures that are not tolerant of the change they inevitably must undergo. Once we accept that software architectures must be designed to be change-tolerant, the barrier to early deployment of high value features is lowered.

Returning to the currency trading example, we see that early deployment of marketable features is a compelling strategy for increasing return and reducing risk. At the same time the stage is set for a better understanding of the real system requirements and improved collaboration between developers and their customers.

Reference
Software by Numbers: Low-Risk, High-Return Development by Mark Denne, Jane Cleland-Huang, Prentice Hall, 2004

Screen Beans Art, © A Bit Better Corporation

Toward A New Definition of Maturity

In the mid 90's, a company named Zeos assembled PC's in my home state of Minnesota. I was impressed when Zeos was named as a finalist for the Malcolm Baldrige Award, because competing for this quality award is more or less the equivalent of a software company trying to reach CMM Level 5. At the same time that Zeos was focusing on the Malcolm Baldrige Award, a similar company in Austin, Texas, called Dell Computer, was focusing all of its energy on two rather different objectives: 1) keep inventory as low as possible, because it is the biggest risk in the PC assembly business, and 2) deliver computers just as fast as possible after customers decide what they want. Dell could not possibly meet these goals unless it had the key Malcolm Baldrige capabilities in place, but that was not its focus. On the other hand, in order to be a finalist, Zeos had to spend huge amounts of executive and management attention on Malcolm Baldrige activities. So which of these two companies would you call the most mature?

A company does not rise to the top of its industry by perfecting its normative procedures. While General Motors was busy refining it’s four phase process for product development, Toyota and Honda were developing cars in something approaching half the time, for half the cost. The resulting cars were less expensive to produce and captured a much larger market share. Yet when looked at through Detroit’s eyes, the development approach of the Japanese companies was decidedly immature – why, they even let die makers start cutting dies long before the final drawings were released!

The problem with maturity models is that they foster a mental model of ‘good practice’ that can block out paradigm shifts that are destined to reshape the industry. This has happened in manufacturing. It has happened in product development. Can it be happening in software development today?

Why Do We Do This To Ourselves?
We human beings a tendency to decompose complex problems into smaller pieces, and then focus on the individual pieces, often, unwittingly, at the expense of the whole. One reason for this is that people can only hold a few chunks of information in short term memory at once; Miller’s law claims that this number is seven plus or minus two. Thus we have a strong tendency to take concepts which we cannot get our minds around and decompose them into parts, so we can deal with one part at a time. There is nothing wrong with our very natural tendency to decompose problems, except that we often forget to reconfigure the parts into a whole and take their interactions into account.

The problem is, decomposition only works if the whole is indeed equal to the sum of its parts, and interactions between the parts can be ignored. In practice, this is rarely the case; in fact, optimization of the parts has a tendency to sub-optimize the whole. For example, a manager might think that the best way to run a testing department is to make sure that every single person is working all of the time. In order to guarantee maximum productivity of each individual, he makes sure there is always a pile of testing waiting to be done. Down the hall, the operations manager knows better. She knows that if she runs the servers at high utilization, the entire system bogs down, just like rush hour traffic. If only the testing manager would realize that his policy of full utilization of testing resources creates the same kind of traffic jam in software development!

Decomposition of a problem into pieces is a standard problem solving heuristic, but it needs to be accompanied by regular aggregation of the parts into a whole. It is here that iterative development shines, because it forces us to develop, test, integrate, and release complete threads of the system early and often. Thus we reset our view of the big picture every iteration. However, iterative development requires that we decompose the problem space along dimensions that are orthogonal to those traditionally recommended. Instead of decomposing the problem into requirements, analysis, design, programming, testing, integration, deployment, we decompose the problem into features along the natural fault lines of the domain.

Decomposition vs. Abstraction
There is an alternative to the decomposition of a problem into components, and that is to abstract the problem to a higher level, rather than decompose it to a lower level. The result is the same in both cases – you have reduced the number of things you need to think about to 7 +/- 2. But when you move the problem up to a higher level of abstraction, the interaction of the parts is maintained, so abstractions would seem to be the better approach to solving complex problems.

There is a catch: people can’t create abstractions without understanding the domain, because correct abstractions require that the abstractor knows what is important, what fits together naturally, and where to find the natural joints or fault lines of the domain. For this reason, experts in the domain have a greater tendency to abstract to a higher level, while those who are not familiar with the domain tend to decompose the problem, and that decomposition will probably not be along the natural fault lines of the domain.

It is when we decompose a problem in a manner which does not fit the domain, and then drill down to the details too fast, that important things get missed. We would like to think that a lot of early detail will help us find all the hidden gremlins that might bite us later. This is only true if those gremlins are actually inside the areas we investigate – but the tough problems usually lurk between the cracks in our thinking. Unfortunately we only discover those problems when we integrate the pieces together at the end, and by that time we have invested so much in the details that change is very difficult.

It is safer to work from a higher level or abstraction, but this means we don’t drill down to detail right away. We take a breadth-first approach, keep options open and gradually fill in the details. With this approach we are more likely to find those gremlins while we still can do something about them, but some people might think that we don’t have a plan, aren’t tracking the project, and can’t manage requirements.

Assessment Rather than Certification
Quite frequently CMM is implemented as a certification process, decomposing ‘mature’ into separately verifiable capabilities. The danger is that we may lose sight of forest for the trees. For example, focusing on requirements management can be detrimental to giving users what they really want. Focusing on quality assurance separate from development can destroy the integrating effect of developers and testers working side-by-side every day. Focusing on planning as a predictive process can keep us from using planning as an organizing process.

A better approach to discovering the underlying competence of an organization is to use an assessment process rather than a certification process. Assessments present challenging situations that cannot be dealt with unless a host of capabilities are in place; if the challenge is properly handled, the presence of these capabilities can be inferred. For example, a pilot’s ability to fly a plane can be assessed by observing how the pilot lands a plane in a stiff cross wind.

Consider your hiring process. When a candidate lists Microsoft or PMI certifications, you take that into account, but you are really looking for a track record of success which demonstrates that the certifications were put to good use. One software company I know administers a one hour logic tests to job applicants, a test without a line of code in it. Yet the test does an admirable job of assessing the capability of the candidate to think like a good developer.

Measure UP
If we accept that we get what we measure then, as Rob Austin points out in “Measuring and Managing Performance in Organizations,” we do not get what we don’t measure. If we are not measuring everything, if something important just didn’t make it into our measurement system, then we are not going to get it. To counter this problem, we have a tendency to pile one measurement on top of another, every time we discover that we forgot to measure something. In CMM, for example, each KPA addresses something that caused a failure of some software project somewhere. As people discovered new ways for software projects to fail, a KPA’s was added to address the new failure mode. Nevertheless, all of those KPA’s still don’t cover everything that could go wrong, although they certainly try.

Once we admit that we just can’t measure everything, then we are ready to move to assessment-style measurements rather than a certification-style measurements. For example, in project management, we decompose the measurement system into cost, schedule, scope and defects, and we try hard to make these measurements work because we think they are the measurements we should use. But no matter how hard we try, we are often unsuccessful in measuring true business value by totaling up the yardsticks of cost, schedule, scope and defects.

What if we just measured business value instead of cost, schedule, scope, and defects? When I was developing new products at 3M, we did not pay much attention to cost, schedule and scope. Instead we developed a P&L which we would use to check out the impact of a late introduction date or a lower unit cost. The development team tried to optimize the overall P&L, not any one dimension. It may seem strange that a project team would concern itself with the financial model of the business, but I assure you it is far better decision-support tool than cost, schedule and scope.

The Measure of Maturity
I believe that our industry could use a simpler measurement of software development maturity, one that is not subject to the dangers of decomposition and does not attempt to defy Miller’s Law with a long lists of things to think about. I propose that we use a measurement that has been successfully used in countless organizations, and has proven to be a true indicator of the presence of the kind of capabilities measured by CMM. To introduce this measurement, let’s go back to Dell and see what it measured:

1. The level of inventory throughout the entire system, and
2. The speed with which the organization can repeatedly and reliably respond to a customer request.

Okay, you say, fine for manufacturing, but software development is different.

True, I counter, but these measurements still work.

The level of inventory in your system is the amount of stuff you have under development. The more inventory of unfinished development work you have sitting around, the more you are at risk of it growing obsolete, getting lost, and hiding defects. If you capitalized it, you also bear the risk of having to write it off if it doesn’t work. The less of that kind of stuff you have on hand, the better off you will be.

The speed with which you can responds to a customer is directly proportional to the amount of unfinished development work you have clogging up your system. In truth, the two measurements above are one and the same – you can deliver faster if your system is not clogged with unfinished work. But more to the point, you cannot reliably and repeatedly deliver fast if you do not have a mature organization. You need version control, built-in quality, ways to gather requirements fast and routinely translate them correctly into code. You need everything you measure in CMM, and you need all of those capabilities working together as an integrated whole. In short, the speed with which you can repeatedly and reliably deliver on customers requests is a better measure of the maturity of your software development organization.

Published as “Measuring Maturity” in Software Development Magazine, February 2004.

Screen Beans Art, © A Bit Better Corporation

Software Development Productivity

Productivity is the prime determinant of our standard of living. If the revenue generated per work hour goes up, income levels rise. If productivity lags, wages also decline. When all else is equal in a market, the more productive company will enjoy greater profits. Thus the key to sustaining and increasing wages in the software development industry is year-to-year improvements in software development productivity.

Over the decade of the 1990’s, productivity increases in the technology sector resulted largely from increased capability of the underlying technology and intense customer demand for new technology. In recent years, however, software development productivity has stagnated as the demand for newer, faster technology has flattened. Thus it should come as no surprise that wages in the software development profession have flattened or declined. In order for the software development industry to see rising incomes, year-to-year increases in software development productivity must be generated.

As the technology sector matures, it can no longer depend on increasing growth in the underlying technology to fuel productivity increases. The time has come for serious efforts to increase productivity through more efficient use of labor and more effective value propositions for customers. This is how more mature economic sectors have been increasing productivity for decades.

But first we need to define what we mean by productivity improvement in the software development industry. Traditionally, we have measured productivity as thousand lines of code (kloc) per labor hour. However, the key process of a development activity is the transformation ideas of into products. To measure the real productivity of software development, we need look at how efficiently and effectively we turn ideas into software. So perhaps we should start with a new definition of software development productivity:

• For companies that develop and sell software as a product, productivity may be defined as the revenue generated per employee.
• For internal IT organizations, productivity may be defined as increased revenue realized by the supported business per dollar spent by the IT organization [1]

There are three basic approaches to productivity improvement:

1. Reduce product costs by eliminating investments in product features that customers do not find valuable.
2. Reduce indirect costs by streamlining processes and eliminating inefficiencies in development, delivery and support.
3. Increase revenue by adding more value to a product so customers will pay more for it.

Let’s explore each of these approaches to improving software development productivity.

1. Reduce Software Development Effort.
About two-thirds of the features of a typical software system are seldom or never used. Only twenty percent of the features are used frequently. [2]  Eliminating extra features that no one really wants represents the single largest opportunity for increasing software development productivity in most organizations.

Extra features are generated by a software development process that attempts to nail down the features in a system at the beginning of the development process. Typically, customers are asked to decide at the start of a project what features they want. Often they have little incentive to keep the features list short, but they are penalized if the forget to include something. Can there be any wonder that a feature list generated with such incentives contains far more features than are really necessary?

The biggest opportunity for reducing software development effort is to limit overproduction of features by developing features on an as-needed basis. For many companies, this may require a paradigm shift in architecture and design. However, it is becoming increasingly apparent that there are many disciplined approaches to software development that provide for an emergent architecture. Key approaches include:

• Automated test harnesses developed at the same time as the underlying code
• Refactoring the code on an on-going basis to keep its design simple and clean

2. Streamline the Development Processes.
The measure of maturity of an organization is the speed at which it can reliably and repeatedly execute it’s key processes. A mature software development organization is one that can rapidly, reliably and reliably translate customer needs into deployed code. Too often, rapid software development has been equated with sloppy work, so attempts to streamline the development process are often looked upon with suspicion. However, in industry after industry, when sequential development processes are replaced with concurrent development processes, costs are slashed, quality is improved, and development time is dramatically reduced.

Organizations which record year-on-year productivity improvements spend a lot of time focusing on streamlining key processes while increasing their reliability. They do this by focusing on the flow of value through the process. There are three main techniques used to do this:

• Value stream mapping is a tried and true approach to streamlining value-creating processes. A value stream map of the current state is invaluable for spotting waste; a value stream map of the desired future state is a roadmap for process improvement. Poppendieck.LLC helps organizations use value stream mapping to improve software development productivity.
• Kaizen events are a typical implementation vehicle for streamlining operational processes. In software development, a modification of Kaizen events is usually required, because software improvement efforts usually take more time than is generally allocated for Kaizen events. We facilitate effective Kaizen events for software development.
• An Integrated Product Team (IPT) is frequently used to facilitate information flow across an entire development team. Software development IPT’s involve not only architects, designers and developers, but also those responsible for deployment, operations, customer support, and maintenance.

3. Increase Customer Value.
Understanding customer value well enough to obtain additional revenue is the third key to increasing software development productivity. In general, customers cannot be relied upon to tell a software development organization how to increase the value of its offerings. In fact, customers generally think of software as a tool that should adapt to their needs over time. This makes understanding customer value elusive not only during a software development project, but even after deployment. And yet, the price that can be charged for software is directly related to understating how to increase its value proposition.

We are not likely to increase software’s value proposition unless we increase our understanding how customers might use our software to create value for themselves. There are three steps to increasing customer value:

• Iterative development with frequent releases creates a short feedback loop between customers and developers. The key to successful iterative development is to prioritize the feature list, implement features in order of business value, and deploy them as soon as possible. The short feedback loop created by iterative development and early deployment not only limits the development of unnecessary code, it brings to light innovative new uses of technology that can significantly improve business results.
• The next step to understanding customer value is to understand how our customers create value for their customers. Increasing the support of key value creating processes for customers is the most likely source of increased revenue for software development. One method of discovering how our customers create value is to focus on their key processes and map their value stream.
• The final step to providing customer value is to link organizations through partnerships that focus on the overall productivity of the combined enterprise. Peter Drucker noted in Management Challenge for the 21st Century [3] that “In every single case…the integration into one management system of enterprises that are linked economically rather than controlled legally, has given a cost advantage of at least 25 percent and more often 30 percent.” The bottom line is that when organizations work together for their mutual benefit rather than optimizing the results of the individual organizations, a large increase in overall productivity can be realized.

Software development organizations have always emphasized process; however, the focus has been on predictable delivery of pre-defined scope rather than increased productivity. Not surprisingly, processes which did not value productivity did not deliver productivity increases; quite often they decreased productivity instead. We need to move from processes which define scope early to processes which allow only the most valuable scope to be addressed. We need to move from slow processes with many wasteful steps to streamlined processes which eliminate non-value-adding activities. We need to focus on increasing revenue potential over cutting costs by discovering how we can help our customers increase their productivity.
____________________
References and Notes
[1] Productivity for internal IT organizations may be also defined as increased revenue realized by the supported business per employee in the IT organization, but such a definition would ignore make-vs-buy decisions, not to mention outsourcing.

[2] Johnson, Jim, Chairman of The Standish Group, ‘ROI, It’s Your Job,’ Keynote: Third International Conference on Extreme Programming, Alghero, Italy, May, 26-29, 2002

[3] Management Challenge for the 21st Century, Peter Drucker, Harper Business, 2001, p 33.

Concurrent Development

When sheet metal is formed into a car body, a massive machine called a stamping machine presses the metal into shape. The stamping machine has a huge metal tool called a die which makes contact with the sheet metal and presses it into the shape of a fender or a door or a hood. Designing and cutting these dies to the proper shape accounts for half of the capital investment of a new car development program, and drives the critical path. If a mistake ruins a die, the entire development program suffers a huge set-back. So if there is one thing that automakers want to do right, it is the die design and cutting.

The problem is, as the car development progresses, engineers keep making changes to the car, and these find their way to the die design. No matter how hard the engineers try to freeze the design, they are not able to do so. In Detroit in the 1980’s the cost of changes to the design was 30 – 50% of the total die cost, while in Japan it was 10 – 20% of the total die cost. These numbers seem to indicate the Japanese companies must have been much better at preventing change after the die specs were released to the tool and die shop. But such was not the case.

The US strategy for making a die was to wait until the design specs were frozen, and then send the final design to the tool and die maker, which triggered the process of ordering the block of steel and cutting it. Any changes went through a arduous change approval process. It took about two years from ordering the steel to the time that die would be used in production. In Japan, however, the tool and die makers order up the steel blocks and start rough cutting at the same time the car design is starting. This is called concurrent development. How can it possibly work?

The die engineers in Japan are expected to know a lot about what a die for a front door panel will involve, and they are in constant communication with the body engineer. They anticipate the final solution and they are also skilled in techniques to make minor changes late in development, such as leaving more material where changes are likely. Most of the time die engineers are able to accommodate the engineering design as it evolves. In the rare case of a mistake, a new die can be cut much faster because the whole process is streamlined.

Japanese automakers do not freeze design points until late in the development process, allowing most changes occur while the window for change is still open. When compared to the early design freeze practices in the US in the 1980’s, Japanese die makers spent perhaps a third as much money on changes, and produced better die designs. Japanese dies tended to require fewer stamping cycles per part, creating significant production savings.

The significant difference in time-to-market and increasing market success of Japanese automakers prompted US automotive companies to adopt concurrent development practices in the 1990’s, and today the product development performance gap has narrowed significantly.

Concurrent Software Development
Programming is a lot like die cutting. The stakes are often high and mistakes can be costly, so sequential development, that is, establishing requirements before development begins, is commonly thought of as a way to protect against serious errors. The problem with sequential development is that it forces designers to take a depth-first rather than a breadth-first approach to design. Depth-first forces making low level dependant decisions before experiencing the consequences of the high level decisions. The most costly mistakes are made by forgetting to consider something important at the beginning. The easiest way to make such a big mistake is to drill down to detail too fast. Once you set down the detailed path, you can’t back up, and aren’t likely to realize that you should. When big mistakes can be made, it is best to survey the landscape and delay the detailed decisions.

Concurrent development of software usually takes the form of iterative development. It is the preferred approach when the stakes are high and the understanding of the problem is evolving. Concurrent development allows you to take a breadth-first approach and discover those big, costly problems before it’s too late. Moving from sequential development to concurrent development means starting programming the highest value features as soon as a high level conceptual design is determined, even while detailed requirements are being investigated. This may sound counterintuitive, but think of it as an exploratory approach which permits you to learn by trying a variety of options before you lock in on a direction that constrains implementation of less important features.

In addition to providing insurance against costly mistakes, concurrent development is the best way to deal with changing requirements, because not only are the big decisions deferred while you consider all the options, but the little decisions are deferred as well. When change is inevitable, concurrent development reduces delivery time and overall cost, while improving the performance of the final product.

If this sounds like magic – or hacking – it would be if nothing else changed. Just starting programming earlier, without the associated expertise and collaboration found in Japanese die cutting, is unlikely to lead to improved results. There are some critical skills that must be in place in order for concurrent development to work.

Under sequential development, US automakers considered die engineers to be quite remote from the automotive engineers, and so too, programmers in a sequential development process often have little contact with the customers and users who have requirements and the analysts who collect requirements. Concurrent development in die cutting required US automakers to make two critical changes – the die engineer needed the expertise to anticipate what the emerging design would need in the cut steel, and had to collaborate closely with the body engineer.

Similarly, concurrent software development requires developers with enough expertise in the domain to anticipate where the emerging design is likely to lead, and close collaboration with the customers and analysts who are designing how the system will solve the business problem at hand.

The Last Responsible Moment
Concurrent software development means starting developing when only partial requirements are known and developing in short iterations which provide the feedback that causes the system to emerge. Concurrent development makes it possible to delay commitment until the Last Responsible Moment, that is, the moment at which failing to make a decision eliminates an important alternative. If commitments are delayed beyond the Last Responsible Moment, then decisions are made by default, which is generally not a good approach to making decisions.

Procrastinating is not the same as making decisions at the Last Responsible Moment; in fact, delaying decisions is hard work. Here are some tactics for making decisions at the Last Responsible Moment:

• Share partially complete design information. The notion that a design must be complete before it is released is the biggest enemy of concurrent development. Requiring complete information before releasing a design increases the length of the feedback loop in the design process and causes irreversible decisions to be made far sooner than necessary. Good design is a discovery process, done through short, repeated exploratory cycles.


• Organize for direct, worker-to-worker collaboration.  Early release of incomplete information means that the design will be refined as development proceeds. This requires that upstream people who understand the details of what the system must do to provide value must communicate directly with downstream people who understand the details of how the code works.


• Develop a sense of how to absorb changes.  In ‘Delaying Commitment,’ IEEE Software (1988), Harold Thimbleby observes that the difference between amateurs and experts is that experts know how to delay commitments and how to conceal their errors for as long as possible. Experts repair their errors before they cause problems. Amateurs try to get everything right the first time and so overload their problem solving capacity that they end up committing early to wrong decisions. Thimbleby recommends some tactics for delaying commitment in software development, which could be summarized as an endorsement of object-oriented design and component-based development:

• Use Modules.  Information hiding, or more generally behavior hiding, is the foundation of object-oriented approaches. Delay commitment to the internal design of the module until the requirements of the clients on the interfaces stabilize. 

• Use Interfaces. Separate interfaces from implementations. Clients should not de-pend on implementation decisions.

• Use Parameters.Make magic numbers – constants that have meaning – into parameters. Make magic capabilities like databases and third party middleware into parameters. By passing capabilities into modules wrapped in simple interfaces, your dependence on specific implementations is eliminated and testing becomes much easier.

• Use Abstractions.  Abstraction and commitment are inverse processes. Defer commitment to specific representations as long as the abstract will serve immediate design needs.

• Avoid Sequential Programming.  Use declarative programming rather than procedural programming, trading off performance for flexibility. Define algorithms in a way that does not depend on a particular order of execution.

• Beware of custom tool building.  Investment in frameworks and other tooling frequently requires committing too early to implementation details that end up adding needless complexity and seldom pay back. Frameworks should be extracted from a collection of successful implementations, not built on speculation.

Additional tactics for delaying commitment include:

• Avoid Repetition. This is variously known as the Don’t Repeat Yourself (DRY) or Once And Only Once (OAOO) principle. If every capability is expressed in only one place in the code, there will be only one place to change when that capability needs to evolve and there will be no inconsistencies.

• Separate Concerns.  Each module should have a single well defined responsibility. This means that a class will have only one reason to change. 

• Encapsulate Variation. What is likely to change should be inside, the interfaces should be stable. Changes should not cascade to other modules. This strategy, of course, depends on a deep understanding of the domain to know which aspects will be stable and which variable. By application of appropriate patterns, it should be possible to extend the encapsulated behavior without modifying the code itself. 

• Defer Implementation of Future Capabilities. Implement only the simplest code that will satisfy immediate needs rather than putting in capabilities you ‘know’ you will need in the future. You will know better in the future what you really need then and simple code will be easier to extend then if necessary.

• Avoid extra features. If you defer adding features you ‘know’ you will need, then you certainly want to avoid adding extra features ‘just-in-case’ they are needed. Extra features add an extra burden of code to be tested and maintained, and understood by programmers and users alike. Extra features add complexity, not flexibility.

Much has been written on these delaying tactics, so they will not be covered in detail in this book.

• Develop a sense of what is critically important in the domain.  Forgetting some critical feature of the system until too late is the fear which drives sequential development. If security, or response time, or fail safe operation are critically important in the domain, these issues need to be considered from the start; if they are ignored until too late, it will indeed be costly. However, the assumption that sequential development is the best way to discover these critical features is flawed. In practice, early commitments are more likely to overlook such critical elements than late commitments, because early commitments rapidly narrow the field of view.


• Develop a sense of when decisions must be made.  You do not want to make decisions by default, or you have not delayed them. Certain architectural concepts such as usability design, layering and component packaging are best made early, so as to facilitate emergence in the rest of the design. A bias toward late commitment must not degenerate into a bias toward no commitment. You need to develop a keen sense of timing and a mechanism to cause decisions to be made when their time has come.


• Develop a quick response capability. The slower you respond, the earlier you have to make decisions. Dell, for instance, can assemble computers in less than a week, so they can decide what to make less than a week before shipping. Most other computer manufacturers take a lot longer to assemble computers, so they have to decide what to make much sooner. If you can change your software quickly, you can wait to make a change until customers’ know what they want.

Cost Escalation
Software is different from most products in that software systems are expected to be upgraded on a regular basis. On the average, more than half of the development work that occurs on a software system occurs after it is first sold or placed into production. In addition to internal changes, software systems are subject to a changing environment – a new operating system, a change in the underlying database, a change in the client used by the GUI, a new application using the same database, etc. Most software is expected to change regularly over its lifetime, and in fact once upgrades are stopped, software is often nearing the end of its useful life. This presents us with a new category of waste, that is, waste caused by software that is difficult to change.

In 1987 Barry Boehm wrote, “Finding and fixing a software problem after delivery costs 100 times more than finding and fixing the problem in early design phases”. This observation became been the rational behind thorough up front requirements analysis and design, even though Boehm himself encouraged incremental development over “single-shot, full product development.” In 2001, Boehm noted that for small systems the escalation factor can be more like 5:1 than 100:1; and even on large systems, good architectural practices can significantly reduce the cost of change by confining features that are likely to change to small, well-encapsulated areas.

There used to be a similar, but more dramatic, cost escalation factor for product development. It was once estimated that a change after production began could cost 1000 times more than if the change had been made in the original design. The belief that the cost of change escalates as development proceeds contributed greatly to the standardizing the sequential development process in the US. No one seemed to recognize that the sequential process could actually be the cause of the high escalation ratio. However, as concurrent development replaced sequential development in the US in the 1990’s, the cost escalation discussion was forever altered. The discussion was no longer how much a change might cost later in development; the discussion centered on how to reduce the need for change through concurrent engineering.

Not all change is equal. There are a few basic architectural decisions that you need to get right at the beginning of development, because they fix the constraints of the system for its life. Examples of these may be choice of language, architectural layering decisions, or the choice to interact with an existing database also used by other applications. These kinds of decisions might have the 100:1 cost escalation ratio. Because these decisions are so crucial, you should focus on minimizing the number of these high stakes constraints. You also want to take a breadth-first approach to these high stakes decisions.

The bulk of the change in a system does not have to have a high cost escalation factor; it is the sequential approach that causes the cost of most changes to escalate exponentially as you move through development. Sequential development emphasizes getting all the decisions made as early as possible, so the cost of all changes is the same – very high. Concurrent design defers decisions as late as possible. This has four effects:

• Reduces the number of high-stake constraints.


• Gives a breadth-first approach to high-stakes decisions, making it more likely that they will be made correctly.


• Defers the bulk of the decisions, significantly reducing the need for change.


• Dramatically decreases the cost escalation factor for most changes.

A single cost escalation factor or curve is misleading. Instead of a chart showing a single trend for all changes, a more appropriate graph has at least two cost escalation curves, as show in Figure 3-1. The agile development objective is to move as many changes as possible from the top curve to the bottom curve.

Figure 3-1. Two Cost Escalation Curves

Returning for a moment to the Toyota die cutting example, the die engineer sees the conceptual design of the car and knows roughly the size of door panel is necessary. With that information, a big enough steel block can be ordered. If the concept of the car changes from a small, sporty car to a mid-size family car, the block of steel may be too small, and that would be a costly mistake. But the die engineer knows that once the overall concept is approved, it won’t change, so the steel can be safely ordered, long before the details of the door emerge. Concurrent design is a robust design process because the die adapts to whatever design emerges.

Lean software development delays freezing all design decisions as long as possible, because it is easier to change a decision that hasn’t been made. Lean software development emphasizes developing a robust, change-tolerant design, one that accepts the inevitability of change and structures the system so that it can be readily adapted to the most likely kinds of changes.

The main reason why software changes throughout its lifecycle is that the business process in which it is used evolves over time. Some domains evolve faster than others, and some domains may be essentially stable. It is not possible to build in flexibility to accommodate arbitrary changes cheaply. The idea is to build tolerance for change into the system along domain dimensions that are likely to change. Observing where the changes occur during iterative development gives a good indication of where the system is likely to need flexibility in the future. If changes of certain types are frequent during development, you can expect that these types of changes will not end when the product is released. The secret is to know enough about the domain to maintain flexibility, yet avoid making things any more complex than they must be.

If a system is developed by allowing the design to emerge through iterations, the design will be robust, adapting more readily to types of changes that occur during development. More importantly, the ability to adapt will be built-in to the system, so that as more changes occur after its release, they can be readily incorporated. On the other hand, if systems are built with a focus on getting everything right at the beginning in order to reduce the cost of later changes, their design is likely to be brittle and not accept changes readily. Worse, the chance of making a major mistake in the key structural decisions is increased with a depth-first, rather than a breadth-first approach.

Book Excerpt: Lean Software Development: An Agile Toolkit

Published in Software Development Magazine, August 2003