Outsource this!

In theory, organizations should focus on their core competencies and outsource everything else. The problem with that theory is the difficulty in deciding what is a core competency and what isn't. If I run a business making concrete, I want to focus on making concrete and selling concrete. Those are my core competencies and everything else should be outsourced. Typically accounting and IT can be done by specialty organizations. Let a company whose core competency is accounting or IT do my it for me.

Generally speaking, concrete is a classic commodity business. Nothing fancy or complicated. Like any commodity business, the product and its manufacturing process have not changed in decades. Unlike the music industry where iTunes revolutionized the distribution channel, concrete is not likely to see technology radically change its physical shipping methods.

In this type of business (and arguably any type of business) accounting techniques are legally required to be identical in every organization.  There is no point in making it a core competency. In the concrete business the final product is so similar among manufacturers that there is no competitive advantage in accounting. Accounting is the classic outsourcing target.

But is accounting just like IT? They both may not look like core competencies, but IT warrants further scrutiny. First of all, it is not legally required to operate the same way across every organization. That gives IT some room for creativity and innovation. Second, IT can have a multiplier effect in global economies and within corporate cultures. The multiplier effect of technology means the right IT investments can have disproportionately large impact on profits.

How does innovation occur in your organization? In a culture that respects new ideas, innovation can take hold. Even in the example of a commodity business like concrete, there is always room for new process ideas and creativity. Maybe the final product is boring and simple concrete. But there is an unlimited amount of room for improving internal methods.

Any organization can create sustainable competitive advantage by continuously, rigorously, and ruthlessly examining their internal processes. All process improvements are driven by information changes and new knowledge. Inevitably, new information systems technologies are demanded to support and often drive these changes. An internal IT organization that understands the intimate details of the business model is infinitely better suited to drive such innovation than any outsourced IT organization. That makes IT a core competency.

What you outsource should never be cast in stone ... or concrete.

~

Never trust the word trust

Perhaps the most abused word in management is trust. "I trust you to get this done" is simply an interesting way to ask someone to get some work done. "I don't trust our competitors" is pretty normal. But managers fail when they use trust to distinguish, assess, and evaluate staff. The notion of "I trust you, but I don't trust you" might make the manager feel like they have drawn the line between good and bad.

But trust is an abstract noun. Any abstract noun's definition is so vague, that it is essentially meaningless. Vague language is a crutch. Bad managers cling to such vague language in performance appraisals and team assessments. An abstract statement such as "I trust you" really means "I don't have anything constructive to say about you." Saying "I don't trust you" simply means "I don't want to tell you what I really don't like about you."

Maybe the managers who abuse the word trust should be forced to use the word in only one context: "I trust you had a good lunch."

~

Don’t Separate Design from Implementation

I was a programmer for about fifteen years. Then I managed a factory IT department for a few years, and managed vendors delivering software for yet more years.  In all of those years (with one exception), software was delivered on time and customers were happy. Yet I never used a list of detailed requirements, let alone a backlog of stories, to figure out what should be done – not for myself, not for my department, not even for vendors.

In fact, I couldn’t imagine how one could look at a piece of paper – words – and decipher what to program. I felt that if the work to be done could be adequately written down in a detailed enough manner that code could be written from it, well, it pretty much had to be pseudocode. And if someone was going to write pseudocode, why not just write the code? It would be equally difficult, less error-prone, and much more efficient.

Software Without Stories
So if I didn’t use detailed requirements – how did I know what to code? Actually, everything had requirements, it’s just that they were high level goals and constraints, not low level directives. For example, when I was developing process control systems, the requirements were clear: the system had to control whatever process equipment the guys two floors up were designing, the product made by the process had to be consistently high quality, the operator had to find the control system convenient to use, and the plant engineer had to be able to maintain it. In addition, there was a deadline to meet and it would be career-threatening to be late. Of course there was a rough budget based on history, but when a control system was going to be used for some decades, one was never penny wise and pound foolish. With these high level goals and constraints, a small team of us proceeded to design, develop, install, and start up a sophisticated control system, with guidance from senior engineers who had been doing this kind of work for decades.

One day, after I had some experience myself, an engineering manager from upstairs came to ask me for help. He had decided to have an outside firm develop and install a process monitoring system for a plant. There was a sophisticated software system involved – the kind I could have written, except that it was too large a job for the limited number of engineers who were experienced programmers. He had chosen to contract with the outside firm on a time-and-materials basis even though his boss thought time-and-materials was a mistake. The engineering manager didn’t believe that it was possible to pre-specify the details of what was needed, but if a working system wasn’t delivered on time and on budget, he would be in deep trouble. So he gave me this job: “Keep me out of trouble by making sure that the system is delivered on time and on budget, and make sure that it does what Harold Stressman wants it to do.”

Harold was a very senior plant product engineer who wanted to capture real time process information in a database. He already had quality results in a database, and he wanted to do statistical analysis to determine which process settings gave the best results. Harold didn’t really care how the system would work, he just wanted the data. My job was to keep the engineering manager out of trouble by making sure that the firm delivered the system Harold envisioned within strict cost and schedule constraints.

The engineering manager suggested that I visit the vendor every few weeks to monitor their work. So every month for eighteen months I flew to Salt Lake City with a small group of people.  Sometimes Harold came, sometimes the engineers responsible for the sensors joined us, sometimes the plant programmers were there. We did not deliver “requirements;” we were there to review the vendor’s design and implementation. Every visit I spent the first evening pouring over the current listings to be sure I believed that the code would do what the vendor claimed it would do. During the next day and a half we covered two topics: 1) What could the system actually do today (and was this a reasonable step toward getting the data Harold needed)? and 2) Exactly how did the vendor plan to get the system done on time (and was the plan believable)?

This story has a happy ending: I kept the engineering manager out of trouble, the system paid for half of its cost in the first month, and Harold was so pleased with the system that he convinced to plant manager to hire me as IT manager.

At the plant, just about everything we did was aimed at improving plant capacity, quality, or throughput, and since we were keepers of those numbers, we could see the impact of changes immediately. The programmers in my department lived in the same small town as their customers in the warehouse and on the manufacturing floor. They played softball together at night, met in town stores and at church, had kids in the same scout troop. Believe me, we didn’t need a customer proxy to design a system. If we ever got even a small detail of any system wrong, the programmers heard about it overnight and fixed it the next day.

Bad Amateur Design
The theme running through all of my experience is that the long list of things we have come to call requirements – and the large backlog of things we have come to call stories – are actually the design of the system. Even a list of features and functions is design. And in my experience, design is the responsibility of the technical team developing the system. For example, even though I was perfectly capable of designing and developing Harold’s process monitoring system myself, I never presumed to tell the vendor’s team what features and functions the system should have. Designing the system was their job; my job was to review their designs to be sure they would solve Harold’s problem and be delivered on time.

If detailed requirements are actually design, if features and functions are design, if stories are design, then perhaps we should re-think who is responsible for this design. In most software development processes I have encountered, a business analyst or product owner has been assigned the job of writing the requirements or stories or use cases which constitute the design of the system. Quite frankly, people in these roles often lack the training and experience to do good system design, to propose alternative designs and weigh their trade-offs, to examine implementation details and modify the design as the system is being developed. All too often, detailed requirements lists and backlogs of stories are actually bad system design done by amateurs.

I suggest we might get better results if we skip writing lists of requirements and building backlogs of stories. Instead, expect the experienced designers, architects, and engineers on the development team to design the system against a set of high-level goals and constraints – with input from and review by business analysts and product managers, as well as users, maintainers, and other stakeholders.

A couple of my “old school” colleagues agree with me on this point. Fred Brooks, author of the software engineering classic “The Mythical Man Month” wrote in his recent book “The Design of Design” [1]:

“One of the most striking 20th century developments in the design disciplines is the progressive divorce of the designer from both the implementer and the user. … [As a result] instances of disastrous, costly, or embarrassing miscommunication abound.”

Tom Gilb, author of the very popular books “Principles of Software Engineering Management” and “Competitive Engineering” recently wrote [2]:

“The worst scenario I can imagine is when we allow real customers, users, and our own salespeople to dictate ‘functions and features’ to the developers, carefully disguised as ‘customer requirements’. Maybe conveyed by our product owners. If you go slightly below the surface of these false ‘requirements’ (‘means’, not ‘ends’), you will immediately find that they are not really requirements. They are really bad amateur design for the ‘real’ requirements…. 

"Let developers engineer technical solutions to meet the quantified requirements. This gets the right job (design) done by the right people (developers) towards the right requirements (higher level views of the qualities of the application).”

Separating design from implementation amounts to outsourcing the responsibility for the suitability of the resulting system to people outside the development team. The team members are then in a position of simply doing what they are told to do, rather than being full partners collaborating to create great solutions to problems that they care about.

_________________________________
Footnotes:
[1] “The Design of Design” by Fred Brooks, pp 176-77. Pearson Education, 2010
[2] "Value-Driven Development Principles and Values;" by Tom Gilb, July 2010 Issue 3, Agile Record 2010 (www.AgileRecord.com)

How Cadence Predicts Process

If you want to learn a lot about a software development organization very quickly, there are a few simple questions you might ask. You might find out if the organization focuses on projects or products. You might look into what development process it uses. But perhaps most the revealing question is this: How far apart are the software releases?

It is rare that new software is developed from scratch; typically existing software is expanded and modified, usually on a regular basis. As a result, most software development shops that we run into are focused on the next release, and very often releases are spaced out at regular intervals. We have discovered that a significant differentiator between development organizations is the length of that interval – the length of the software release cycle.

Organizations with release cycles of six months to a year (or more) tend to work like this: Before a release cycle begins, time is spent deciding what will be delivered in the next release. Estimates are made. Managers commit. Promises are made to customers. And then the development team is left to make good on all of those promises. As the code complete date gets closer, emergencies arise and changes have to be made, and yet, those initial promises are difficult to ignore. Pressure increases.

If all goes according to plan, about two-thirds of the way through the release cycle, code will be frozen for system integration testing (SIT) and user acceptance testing (UAT). Then the fun begins, because no one really knows what sort of unintended interactions will be exposed or how serious the consequences of those interactions will be. It goes without saying that there will be defects; the real question is, can all of the critical defects be found and fixed before the promised release date?

Releases are so time-consuming and risky that organizations tend to extend the length of their release cycle so as not to have to deal with this pain too often. Extending the release cycle invariably increases the pain, but at least the pain occurs less frequently. Counteracting the tendency to extend release cycles is the rapid pace of change in business environments that depend on software, because the longer release cycles become a constraint on business flexibility. This clash of cadences results in an intense pressure to cram as many features as possible into each release. As lengthy release cycles progress, pressure mounts to add more features, and yet the development organization is expected to meet the release date at all costs.

Into this intensely difficult environment a new idea often emerges – why not shorten the release cycle, rather than lengthen it? This seems like an excellent way to break the death spiral, but it isn’t as simple as it seems. The problem, as Kent Beck points out in his talk “Software G Forces: the Effects of Acceleration,” is that shorter release cycles demand different processes, different sales strategies, different behavior on the part of customers, and different governance systems. These kinds of changes are notoriously difficult to implement.

Quick and Dirty Value Stream Map
I’m standing in front of a large audience. I ask the question: “Who here has a release cycle longer than three months?” Many hands go up. I ask someone whose hand is up, “How long is your release cycle?” She may answer, “Six months.” “Let me guess how much time you reserve for final integration, testing, hardening, and UAT,” I say. “Maybe two months?” If she had said a year, I would have guessed four months. If she had said 18 months, I would have guessed 6 months. And my guess would be very close, every time. It seems quite acceptable to spend two-thirds of a release cycle building buggy software and the last third of the cycle finding and fixing as many of those bugs as possible.

The next question I ask is: When do you decide what features should go into the release? Invariably when the release cycle is six months or longer, the answer is: “Just before we start the cycle.” Think about a six month release cycle: For the half year prior to the start of the cycle, demand for new or changed features has been accumulating – presumably at a steady pace. So the average wait of a feature to even be considered for development is three months – half of the six month cycle time. Thus – on the average – it will take a feature three months of waiting before the cycle begins, plus six months of being in development and test before that feature is released to customers; nine months in all.

Finally I ask, “About how many features might you develop during a six month release cycle?” Answers to this vary widely from one domain to another, but let’s say I am told that that about 25 features are developed in a six month release, which averages out to about one feature per week.

This leaves us with a quick and dirty vision of the value stream: a feature takes a week to develop and best case it takes nine months (38 weeks) to make it through the system. So the process efficiency is 1÷38, or about 2.6%. A lot of this low efficiency can be attributed to batching up 25 features in a single release. A lot more can be attributed to the fact that only 4 months of the 9 total months are actually spent developing software – the rest of the time is spent waiting for a release cycle to start or waiting for integration testing to finish.

Why not Quarterly Releases?
With such dismal process efficiency, let’s revisit to the brilliant idea of shortening release cycles. The first problem we encounter is that at a six month cadence, integration testing generally takes about two months; if releases are annual, integration testing probably takes three or four months. This makes quarterly releases quite a challenge.

For starters, the bulk of the integration testing is going to have to be automated. However, most people rapidly discover that their code base is very difficult to test automatically, because it wasn’t designed or written to be tested automatically. If this sounds like your situation, I recommend that you read Gojko Adzic’s book “Specification by Example.” You will learn to think of automated tests as executable specifications that become living documentation. You will not be surprised to discover that automating integration tests is technically challenging, but the detailed case studies of successful teams will give you guidance on both the benefits and the pitfalls of creating a well architected integration test harness.

Once you have the beginnings of an automated integration test harness in place, you may as well start using it frequently, because its real value is to expose problems as soon as possible. But you will find that code needs to “done” in order to be tested in this harness, otherwise you will get a lot of false negatives. Thus all teams contributing to the release would do well to work in 2-4 week iterations and bring their code to a state that can be checked by the integration test harness at the end of every iteration. Once you can reasonably begin early, frequent integration testing, you will greatly reduce final integration time, making quarterly releases practical.

Be careful, however, not to move to quarterly releases without thinking through all of the implications. As Kent Beck noted in his Software G Forces talk, sales and support models at many companies are based on annual maintenance releases. If you move from an annual to a quarterly release, your support model will have to change for two reasons: 1) customers will not want to purchase a new release every quarter, and 2) you will not be able to support every single release over a long period of time. You might consider quarterly private releases with a single annual public release, or you might want to move to a subscription model for software support. In either case, you would be wise not to guarantee long term support for more than one release per year, or support will rapidly become very expensive.

From Quarterly to Monthly Releases
Organizations that have adjusted their processes and business models to deal with a quarterly release cycle begin to see the advantages of shorter release cycles. They see more stability, more predictability, less pressure, and they can be more responsive to their customers. The question then becomes, why not increase the pace and release monthly? They quickly discover that an additional level of process and business change will be necessary to achieve the faster cycle time because four weeks – twenty days – from design to deployment is not a whole lot of time.

At this cadence, as Kent Beck notes, there isn’t time for a lot of information to move back and forth between different departments; you need a development team that includes analysts and testers, developers and build specialists. This cross-functional team synchronizes via short daily meetings and visualization techniques such as cards and charts on the wall – because there simply isn’t time for paper-based communication. The team adopts processes to ensure that the code base always remains defect-free, because there isn’t time to insert defects and then remove them later. Both TDD (Test Driven Development) and SBE (Specification by Example) become essential disciplines.

From a business standpoint, monthly releases tend to work best with software-as-a-service (SaaS). First of all, pushing monthly releases to users for them to install creates huge support headaches and takes far too much time. Secondly, it is easy to instrument a service to see how useful any new feature might be, giving the development team immediate and valuable feedback.

Weekly / Daily Releases
There are many organizations that consider monthly releases a glacial pace, so they adopt weekly or even daily releases. At a weekly or daily cadence, iterations become largely irrelevant, as does estimating and commitment. Instead, a flow approach is used; features flow from design to done without pause, and at the end of the day or week, everything that is ready to be deployed is pushed to production. This rapid deployment is supported by a great deal of automation and requires a great deal of discipline, and it is usually limited to internal or SaaS environments.

There are a lot of companies doing daily releases; for example, one of our customers with a very large web-based business has been doing daily releases for five years. The developers at this company don’t really relate to the concept of iterations. They work on something, push it to systems test, and if it passes it is deployed at the end of the day. Features that are not complete are hidden from view until a keystone is put in place to expose the feature, but code is deployed daily, as it is written. Occasionally a roll-back is necessary, but this is becoming increasingly rare as the test suites improve. Managers at the company cannot imagine working in at a slower cadence; they believe that daily deployment increases predictability, stability, and responsiveness – all at the same time.

Continuous Delivery
Jez Humble and David Farley wrote “Continuous Delivery” to share with the software development community techniques they have developed to push code to the production environment as soon as it is developed and tested. But continuous delivery is not just a matter of automation. As noted above, sales models, pricing, organizational structure and the governance system all merit thoughtful consideration.

Every step of your software delivery process should operate at the same cadence. For example, with continuous delivery, portfolio management becomes a thing of the past; instead people make frequent decisions about what should be done next. Continuous design is necessary to keep pace with the downstream development, validation and verification flow. And finally, measurements of the “success” of software development have to be based on delivered value and improved business performance, because there is nothing else left to measure.

Angry Management

An icon in the computer industry recently lost his temper and fired all his staff in a particular product line because the product was not a success. I'm pretty sure most of my business school textbooks said he did the exact right thing. Management is holding staff accountable for a failure. But if you draw that logic to its ultimate conclusion, if all his staff failed then why didn't he fire himself too?

I agree that managers have to hold staff accountable for failures. I disagree with letting emotions interfere with good judgment. The problem isn't with holding staff accountable for a product failure. The problem isn't with firing some folks who may have been incompetent and should not have been in the roles they held. The problem is with losing your temper.

As a manager, how can you possibly make a rationale business decision when you are angry? Every rational road of reason in your brain is blocked in a traffic gridlock of anger. You are going to make mistakes. Making arbitrary decisions is the result of letting emotions shutdown alternative routes of logic. Seeing only one road to follow is the result of letting emotions limit broader thinking. Anger leads to simplified logic: if the product is a failure, then team is a failure. If the team is a failure, then fire them all.

Much like throwing the baby out with the bathwater, the icon of the computer industry threw the whole team out on the street. A less emotional decision may have called for a post-mortem review. Create an opportunity to let emotions dissipate before making a decision. Let rush hour to fade and discover new roads of logic. Maybe some members of the team truly needed to leave the organization. But it is inconceivable to think all of them were a problem.

No one comes into work with the intention of doing a bad job. As a manager you need to take some time to understand the full reasons for the failure. Before you fire the whole team, think about how much support and guidance you gave your staff. Did you hire the right people? Did you set the appropriate strategy? Did you put the right support processes in place? When you ask these questions you realize accountability for a failure of this magnitude moves up the organization. You need to think carefully about the true origins of the failure.

Responding to a crisis on an emotional level allows the manager to hide the truth behind a wall of anger.

~

Change Challenges

Sometimes I wonder why it is so hard to introduce technology change in organizations. While planning a new enterprise system implementation several years ago I discovered that everyone said the old system was bad and had to be replaced. After we installed the new system, everyone blissfully reminisced about how wonderful the old system was. If this reaction happened once for one technology change I would understand. But seems to be a consistent reaction to technology change.

Is the reason human nature, or how we approach technology change? The very nature of the word technology might have a small part to play in the problem. The word technology is derived from "knowledge of technique." Not "cool gadgets are fun" or "my new iPad will change the world." I would suggest that the etymology of the word of technology reveals an interesting perspective on how to introduce technical change to an organization.

What if we paid more attention to understanding the technique needed to use new technology, rather than the technology itself? To address technique we naturally focus on process improvement. Let's try to improve our processes at the same time we implement new technology to ensure the technology is optimally used and the organization is appropriately prepared to apply the new technology.  Makes sense, but I've seen serious resistance to this approach too. Changing process concurrently with new technology implementation isn't enough.

I suspect we implement change with blinders on. We have a narrow focus on planting new technology using advanced processes but ignore the fertility of the ground. What about looking at the rate of technology change relative to the rate of culture change in your organization? How fast does your organization's culture change relative to its outside environment? How fast are you planning to implement new technology internally? I suggest that anytime you introduce technological change faster than the natural pace of change in your organization, you are going to have change challenges. Conversely, anytime you introduce technology slower than the internal change rate you get stagnation.

Is first prize matching the rate of technology change with the acceptable rate of organizational change? Is that the ideal synchronization? Nope. How do you compete or derive a competitive advantage if you simply plant the same crops the same way every year? The challenge is how do we increase our rate of technology change without the resistance caused by being faster than our natural organizational rhythm of change?

I suggest you change the culture first, and then introduce the new technology. When the rate of technology change has to be faster than the rate of organization change, the culture has to change first. Make sure the field has all the nutrients needed to grow a new crop and you will reap what you sow. But changing an organizational culture is a lot harder than buying a new technology. I'm afraid that too often we take the path of least resistance and try to use technology to drive the organization change. In this field of endeavour, maybe we have put the plough before the horse.

~

Some thoughts on privacy legislation

Attached is my speech on privacy legislation from a CIO perspective.

~