Measure Up

Getting measurements right can be devilishly difficult, but getting them wrong can be downright dangerous. If you look underneath most self-defeating behavior in organizations, you will often find a well-intentioned measurement which has gone wrong. Consider the rather innocent-sounding measurement of productivity, and it’s close cousin, utilization. One of the biggest impediments to adopting Just-in-Time manufacturing was the time-honored practice of trying to extract maximum productivity out of every machine. The inevitable result was that mounds of inventory collected to feed machines and additional piles of inventory stacked up at the output side of the machines. The long queues of material slowed everything down, as always queues do. Quality problems often took days to surface, and customer orders often took weeks to fill. Eventually manufacturing people learned that running machines for maximum productivity was a sub-optimizing practice, but it was a difficult lesson.

As software development organizations search for productivity on today’s tight economy, we see the same lesson being learned again. Consider the testing department which is expected to run at 100% utilization. Mounds of code tend to accumulate at the input side of the testing department, and piles completed tests stack up at the output side of the testing department. Many defects lurk in the mountain of code, and more are being created by developers who do not have immediate feedback on their work. When a testing department is expected to run at full utilization, the likely result will be an increased defect level, resulting in more work for the testing department.

Nucor Steel grew from a startup in 1968 into a $4 billion giant, attributing much of its success to an incentive pay system based on productivity. Productivity? How did Nucor keep their productivity measurement robust and honest throughout all of that growth? How did they avoid the sub-optimization so common most productivity measurements?

The secret is that Nucor measures productivity at a team level, not at an individual level. For example, a plant manager is not rewarded on the productivity of his or her plant, but on the productivity of all plants. The genius of Nucor’s productivity measurement is that it avoids sub-optimization by measuring results at one level higher than one would expect, thus encouraging knowledge sharing and system-wide optimization.

How can this be fair? How can plant managers be rewarded based on productivity of plants over which they have no control? The problem is, if we measure people solely on results over which they have full control, they have little incentive to collaborate beyond their own sphere of influence to optimize the overall business. While local measurements may seem fair to individuals, they are hardly fair to the organization as a whole.

Measure-UP, the practice of measuring results at the team rather than the individual level, keeps measurements honest and robust. The simple act of raising a measurement one level up from the level over which an individual has control changes its dynamic from a personal performance measurement to a system effectiveness indicator.

In the book “Measuring and Managing Performance in Organizations”, Dorset House 1996, Robert Austin discusses the dangers of performance measurements. The beauty of performance measurements is that “You get what you measure.” The problem with performance measurements is that “You get only what you measure, nothing else.” You tend to loose the things that you can’t measure: insight, collaboration, creativity, dedication to customer satisfaction.

Austin recommends aggregating individual performance measurements into higher level informational measures that hide individual results in favor of group results. As radical as this may sound, it is not unfamiliar. Edward Demming, the noted quality expert, insisted that most quality defects are not caused by individuals, but by management systems that make error-free performance all but impossible. Attributing defects to individuals does little to address the systemic causes of defects, and placing blame on individuals when the problem is systematic perpetuates the problem.

Software defect measurements are frequently attributed to individual developers, but the development environment often conspires against individual developers and makes it impossible to write defect-free code. Instead of charting errors by developer, a systematic effort to provide developers with immediate testing feedback, along with a root cause analysis of remaining defects, is much more effective at reducing the overall software defect rate.

By aggregating defect counts into an informational measurement, and hiding individual performance measurements, it becomes easier to address the root causes of defects. If an entire development team, testers and developers alike, feel responsible for the defect count, then testers will tend to become involved earlier and provide more timely and useful feedback to developers. Defects caused by code integration will become everyone’s problem, not just the unlucky person who wrote the last bit of code.

It flies in the face of conventional wisdom to suggest that the most effective way to avoid the pitfalls of measurements is to use measurements that are outside the personal control of the individual being measured. But conventional wisdom is misleading. Instead of making sure that people are measured within their span of control, it is more effective to measure people one level above their span of control. This is the best way to encourage teamwork, collaboration, and global, rather than local, optimization.

Screen Beans Art, © A Bit Better Corporation

Lessons from Planned Economies

Just as a market economy which relies on the collective actions of intelligent agents gives superior performance in a complex and changing economic environment, so too an agile project leadership approach which leverages the collective intelligence of a development team will give superior performance in a complex and changing business environment. However, conventional project management training focuses on using a plan as the program for action; it does not teach project leaders how to create a software development environment that fosters self-organization and learning. Since very few courses with such a focus are available today, this paper proposes a curriculum for agile software development leaders.

Planned Economies
In the middle of the 20th century, dozens of countries and millions of people believed that central planning was the best way to run their economies. Even today there are many people who can’t quite understand why market economies invariably outperform planned economies; it would seem that at least some of the planned economies should have flourished. After all, there are advantages to centralizing economic decisions: virtually full employment is possible; income can be distributed more equally; central coordination should be more efficient; directing resources into investment should spur growth. So why did planned economies fail?

There are two fundamental problems with planned economies: First, in a complex and changing economic system, it is impossible to plan for everything, so a lot of things fall between the cracks. For instance, planned economies usually suffer a shortage of spare parts, because no one plans for machines to break down. Secondary effects such as environmental impact are often ignored. Furthermore, planners do not have control of the purchase of goods, so they have to guess what consumers really want. Inaccurate forecasts are amplified by a long planning cycle, causing chronic shortages and surpluses.

The second problem with planned economies is diminished incentives for individuals. When compensation does not depend on contribution, there is little to gain from working hard. When incentives are tied to meeting targets, risk adverse managers focus on routine production to meet goals. The stronger that rewards are tied to meeting targets, the more disincentive there is for being creative or catching the things that fall between the cracks.

If we look at conventional software project management, we see practices similar to those used in planned economies, and we also see the similar results. Among projects over $3 million, less than 10% meet the conventional definition of success: on time, on budget, on scope. For projects over $6 million the number drops below 1% [1]. Interestingly, the underlying causes of failure of planned economies are the same things that cause failure in software projects, and further, the remedy is similar in both cases.

The difference between a planned and a market economy is rooted in two different management philosophies: management-as-planning/adhering and management-as-organizing/learning. Management-as-planning/adhering focuses on creating a plan that becomes a blueprint for action, then managing implementation by measuring adherence to the plan. Management-as-organizing/learning focuses on organizing work so that intelligent agents know what to do by looking at the work itself, and improve upon their implementation through a learning process.

The Planning/Adhering Model Of Project Management
Conventional wisdom holds that managing software projects is equivalent to meeting pre-planned cost, schedule and scope targets. The unquestioned dominance of cost, schedule and scope – often to the exclusion of less tangible factors such as usability or realization of purpose – draws heavily on the contract administration roots of project management. Therefore project management training and certification programs tend to focus on the management-as-planning/adherence philosophy. This philosophy has become entrenched because it seems to address two fears: a fear of scope-creep, and a fear that the cost of changes escalates significantly as development progresses.

However, management-as-planning/adherence leads to the same problems with software projects that planned economies suffered: in a complex and changing environment, it is virtually impossible for the plan to cover everything, and measuring adherence to the plan diminishes incentives for creativity and catching the things that fall between the cracks.

In the classic article ‘Managing by Whose Objectives?,’[4] Harry Levinson suggests that the biggest problem with management-by-objectives is that important intangibles which are not measurable fail to get addressed, because they are not in the action plan of any managers. Often these are secondary ‘hygiene’ factors, similar to environmental considerations in a planned economy.

In the book ‘Measuring and Managing Performance in Organizations’[5], Robert Austin makes the same point: over time, people will optimize what is measured and rewarded. Anything which is not part of the measurement plan will fade from importance. Austin points out that managers are often uncomfortable with the idea of not being able to measure everything, so they compensate through one of three techniques:

• Standardization. By creating standards for each step in a development process, it is hoped that all steps in the project can be planned and measured, and nothing will be missed.


• Specification. Specification involves constructing a detailed model of the product and/or process and planning every step in detail.


• Subdivision, functional decomposition. The Work Breakdown Structure (WBS) is the classic example of attempts to decompose a project into steps so that all steps can be planned. 

Conventional project management practices have emphasized all of these techniques to help a project manager be certain that everything is covered in the project plan. However, just as in a planned economy, these techniques are insufficient to catch everything in all but the simplest of projects. In fact, drilling down to detail early in the project has the opposite effect – it tends to create blind spots, not resolve them. By taking a depth-first rather than a breadth-first approach to planning, mistakes and omissions become more likely,[6] and these tend to be more costly because of early investment in details. Thus an early drill-down approach tends to amplify, not reduce, the cost of change.

A management-as-planning/adherence approach also tends to amplify, not reduce, scope-creep. In many software development projects, a majority of the features are seldom or never used.[1] Part of the reason for this is that asking clients at the beginning of a project what features they want, and then preventing them from changing their minds later, creates strong incentives to increase the number of features requested, just in case they are needed. While limiting scope usually provides the best opportunity for reducing software development costs, fixing scope early and controlling it rigidly tends to expand, not reduce scope.

Just as in planned economies, management-as-planning/adhering tends to have unintended consequences that produce precisely the undesirable results that the plans were supposed to prevent. The problem lies not in the planning, which is very useful, but in using the plan as a roadmap for action and measuring performance against the plan.

The Organizing/Learning Model Of Project Management
Market economies deal with the problems of planned economies by depending upon collaborating intelligent agents to make decisions within an economic framework. In market economies, it is the job of the government to organize the economic framework with such things as anti-trust laws and social safety nets. Economic activity is conducted by intelligent agents who learn from experience what is needed and how to fill the needs.

Of course, the economies of individual countries dwarf most software projects, so we might look further to find examples of management-as-organizing/learning. We will explore two domains: manufacturing and product development.

Throughout most of the 20th century, mass production in the US focused on getting things done through central planning and control, reflecting the strong influence of Frederick Taylor’s Scientific Management. The climax came when computer systems made it possible to plan the exact movement of materials and work throughout a plant. Material Requirements Planning (MRP) systems were widely expected to increase manufacturing efficiency in the 1980’s, but in fact, most MRP systems were a failure at detailed production planning. They failed for the same reasons that planned economies failed: the systems could not readily adapt to slight variations in demand or productivity. Thus they created unworkable schedules, which had to be ignored, causing the systems to become ever more unrealistic.

As the centralized MRP planning systems were failing, Just-in-Time systems appeared as a counterpoint to Scientific Management. Just-in-Time forsakes central planning in favor of collaborating teams (intelligent agents). The environment is organized in such a way that the work itself and the neighboring teams signal what needs to be done; rather than a central plan. When problems occur, the root cause is sought out and eliminated, creating an environment in which intelligent agents continually improve the overall system. In almost all manufacturing environments, implementing Just-in-Time trumps any attempt to plan detailed production activities using a MRP system. These systems succeed for the same reason a market economy succeeds: intelligent agents are better at filling in the gaps and adapting to variation than a centrally planned system.

An argument can be made that manufacturing analogies are not appropriate for software development, because manufacturing is repetitive, while projects deal with unique situations. Because of this uniqueness, the argument goes, management as planning/ adhering is the only way to maintain control a design and development environment. A look at product development practices shows that the opposite is true: creating a detailed plan and measuring adherence to that plan is actually a rather ineffective approach a complex product development project.

In the late 1980’s Detroit was shocked to discover that a typical Japanese automotive company could develop a new car in 2/3’s the time for half the cost as a typical US automaker.[7] The difference was that product development in Japan used a concurrent development process, which allows for learning cycles during the design process as well as on-going communication and negotiation among intelligent agents as design proceeds.

Just as market economies routinely outperform planned economies, concurrent development routinely outperforms sequential development. Replacing sequential (plan-up-front) engineering with concurrent (plan-as-you-go) engineering has been credited with reducing product development time by 30-70%, engineering changes by 65-90%, and time to market by 20-90%, while improving quality by 200-600%, and productivity by 20-110%.[8]

Based on experience from other domains, management-as-organizing/learning would appear to have a better chance of resulting successful software projects than the prevailing management-as-planning/adhering approach. An examination of the Agile Manifesto shows that agile software development approaches favor the management-as-organizing/learning philosophy. (See Table 1.) Therefore, we can expect that in the long run, agile software development might significantly outperform traditional software development practices. In fact, evidence is mounting that agile approaches can be very effective. [9]

Table 1. Mapping Values from the Agile Manifesto to Management Philosophies

A Curriculum For Agile Software Project Leadership
Existing training for project management appears to be largely focused on the management-as-planning/adhering philosophy. Courses seem to be aimed at obtaining certification in an approach to project management developed for other domains, such as facilities construction or military procurement. Even courses aimed specifically at software development tend to focus on work breakdown and a front-end-loaded approach to managing scope. As we have seen, this is not a good match for concurrent development or agile software development.

It would seem that at a curriculum should be available for leaders of agile software development projects, given the dismal track record of current approaches and the potential of agile software development. Project managers who know how to develop work breakdown structures and measure earned value often wonder what to do with an agile software development project. Senior managers often wonder how to improve the skills of project leaders. Courses on management-as-organizing/learning are needed to fill this void, but there seem to be few project management courses with this focus. To help make agile project leadership training more widely available, this article outlines a possible curriculum for such courses.

Change the Name: Project Leadership
Since this is new territory, we may as well start with a new name, and move away from the administrative context of the word management. All projects, agile or otherwise, benefit from good leaders; that is, people who set direction, align people, and create a motivating environment. By using the term leadership we distinguish this course from one which focuses on the management tasks of planning, budgeting, staffing, tracking, and controlling.

Setting Direction
Planning is a good thing; the ultimate success of any project depends upon the people who implement a project understanding what constitutes success. Planning becomes brittle with it decomposes the problem too fast and moves too quickly to solutions. The best approach to early planning is to move up one notch and take a broader view, rather than decompose the problem and commit to detail too early.[6] A project leader starts by understanding the purpose of the project and keeping that purpose in front of the development team at all times.

Organizing Through Iterations
The idea of management-as-organizing/learning is to structure the work so that developers can figure out what to do from the work itself. The fundamental tool for doing this is short iterations which develop working software delivering business value. Project leaders need to know how to organize iteration planning meetings and how to structure the development environment and workday so that people know what to do when they come in to work every day, without being told.

This part of the curriculum must cover such concepts as story cards, backlog lists, daily meetings, pair programming, and information radiators. It should also stress the importance of organizing worker-to-worker collaboration between those who understand what the system must do to provide value and those who understand what the code does.

Concurrent Development
Strategies for concurrent development are an important tool for project leaders, especially in organizations which are used to sequential development. General strategies include:

• sharing partially complete design information


• communicating design constraints instead of proposed solutions


• maintaining multiple options


• avoiding extra features


• developing a sense of how to absorb changes


• developing a sense of what is critically important in the domain


• developing a sense of when decisions must be made


• delaying decisions until they must be made


• developing a quick response capability

System Integrity
Project leaders must assure that the software developed under their guidance has integrity. This starts with assuring that the basic tools are in place for good software development: version control, a build process, automated testing, naming conventions, software standards, etc. Leaders must assure that the development team is guided by the true voice of the customer, so that the resulting system delivers value both initially and over time. They must assure that technical leadership establishes the basis of a sound architecture and an effective interface. They must make sure that comprehensive tests are used to provide immediate feedback, as well a framework so that refactoring can safely take place.

Leading Teams
People with experience in traditional project management who are about to lead agile software development projects might need some coaching in how to encourage a team make its own commitments, estimate its own work, and self-organize around iteration goals. Project leaders attend the daily meetings, listen to and solve team problems, serve as an intermediary between management and the team, secure necessary resources and technical expertise, resolve conflicts and keep everyone working together effectively; but they do not tell developers how to do their job. Project leaders coordinate the end-of iteration demonstration and the beginning of iteration planning, making sure that work is properly prioritized and all stakeholder interests are served.

Measurements
Feature lists or release plans, along with associated effort estimates, are often developed early in an agile project. The difference from traditional project management occurs when actual measurements begin to vary from these plans; for agile development, it is assumed that the plan is in error, and needs to be revised. By measuring the actual velocity or burndown rate, a far better picture of project health can be obtained than measuring variance from a guesstimate. Creating more accurate estimates becomes easier as developers gain experience in a domain and customers see working software. Leaders should learn how to combine reports of actual progress with increasingly accurate estimates into a tool for negotiating the scope of a project; this can be far more effective at limiting scope that the traditional method of fixing scope and controlling it with change approval mechanisms.

A useful technique for project leaders is to aggregate all measurements one level higher than normal.[5] This encourages collaboration and knowledge sharing because issues are called to the attention of a larger group of people. It also helps to avoid local optimization and the dangers of individual performance measurements. This technique is useful, for instance, for defect measurements.

Large Projects
Various techniques for synchronizing agile development across multiple teams are important for project managers to understand. Some of the techniques that might be covered are: divisible architectures, daily build and smoke test, spanning applications, loosely coupled teams that develop interfaces before modules.

Conclusion
We have often heard that the sequential, or waterfall, approach to software development projects is widely known to be ineffective, but it seems to be quite difficult to do things differently. We also heard that many, many features in most systems, possibly even a majority, are not needed or used; yet limiting scope to what is necessary seems to be an intractable problem. One solution to these problems can be found in concurrent engineering, which is widely used in product development as an alternative to sequential development. Concurrent engineering works more or less like a market economy: both depend on collaborating intelligent agents, both allow the end result emerge through communication, and both provide incentives for individuals to be creative and do everything necessary to achieve success.
_______________
References:
[1] Johnson, Jim, Chairman of The Standish Group, Keynote “ROI, It’s Your Job,” Third International Conference on Extreme Programming, Alghero, Italy, May, 26-29, 2002.

[2] Johnston, R B, and Brennan, M, “Planning or Organizing: the Implications of Theories of Activity for Management of Operations”, Omega: The International Journal of Management Science, Vol 24 no. 4 pp. 367-384, Elsevier Science, 1996.

[3] Koskela1, Lauri, “On New Footnotes to Shingo”, 9th International Group for
Lean Construction Conference, Singapore, August, 6-8 2001.

[4] Levinson, Harry, “Management by Whose Objectives?” Harvard Business Review, Vol 81, no 1, January, 2003, Reprint of 1970 article.

[5] Austin, Robert D., Measuring and Managing Performance in Organizations, 1996, Dorset Publishing House

[6] Thimbleby, Harold, “Delaying Commitment”, IEEE Software, vol 5, no 3, May, 1988

[7] Clark, Kim B, Fujimoto, Takahiro, Product Development Performance; Strategy, Organization, and Management in the World Auto Industry, Harvard Business School Press, Boston, 1991.

[8] Thomas Group Inc., National Institute of Standards & Technology Institute for Defense Analyses, from Business Week, April 30, 1990, pp 111

[9] Weber Morales, Alexandra, “Extreme Quality”, Software Development, Volume 11, No 2, February 2003.

[10] Kotter, John P, “What Leaders Really Do,” Harvard Business Review, Vol 79, no 11, December, 2001, Reprint of 1990 article.

Screen Beans Art, © A Bit Better Corporation