Over the last two decades, the software technology stack has undergone a rapid evolution, as this diagram from Docker.io lays out.
The evolution continues. Today’s world of smart phones is giving way to tomorrow’s world of smart devices with sensors and actuators and not much more. The app layer will only get thinner.
If you think this trend will not affect your organization, think again. Tony Scott, CIO of the US federal government, advised CIO’s throughout the country to move to the cloud as fast as possible. Why? Because the large cloud providers can provide more secure, less expensive, and more reliable infrastructure than most organizations can provide for themselves. Major industries, from banking to health care, are discovering the benefits of moving to the cloud. Thin apps and assembled services running on off-premises hardware will soon become the norm for most organizations, probably even yours.
What does the cloud have to do with software development? Quite a bit, it turns out. In the cloud:
1. The development team is responsible for product design.
Assembling services is a dynamic process, not a one-time affair.
The thin app is often the only differentiator in the stack.
2. The development team is responsible for its own infrastructure.
When infrastructure is code, one team does it all:
design/code/test/deploy/monitor/maintain.
Keeping things running is a new challenge for many software engineers.
3. Apps must be immune to infrastructure and service failure.
Stateless designs replace object-oriented designs.
Distributed, immutable data sets replace databases.
Things get done through producer/consumer chains.
So here’s the point: Practices designed for the problems of 1995 are not going to work for the problems of 2020. We need to frame today’s and tomorrow’s problems in a way that helps us to identify and tackle them effectively; we need to use fundamental principles to help us ask the right questions. [1]
What are the right questions? Consider this guidance from Taiichi Ohno, the father of Lean:
The new technology stack produces its own friction, a different kind of friction than was typically found in the old stack. This friction is particularly strong in organizations moving from the old to the new technology stack because the transition brings a lot of change to software development. Unfortunately, that change is not always well supported by the organization or welcomed by the software engineers.
To assess the current situation, take a look at the value stream – the stream of activities that deliver value to customers – and identify areas of friction. In the modern technology stack, friction generators tend to be either deeply technical or highly organizational in nature, as you can see from the discussion above. Unfortunately, these are not usually the problems that companies tackle when they move to modern software development. Why? Quite often the organizational structure is so entrenched that changing it is not considered. Or perhaps the people leading the transition do not understand the underlying technology and the problems presented by the new stack. In either case, the underlying problem becomes an elephant in the room that everyone ignores, while easier challenges - like adopting agile processes - are taken up.
It is important to confront the deep-seated friction generators that people would rather ignore. Start by talking about the elephant, and then actively imagine what your world would be like without that elephant. Once you have a clear vision of the future, you can work out how to move constantly toward that vision by eliminating the most pernicious friction generators, one step at a time. This approach has helped teams and organizations around the world make steady progress in the right direction, and eventually the steady progress adds up to amazing accomplishments.
Identifying, addressing, and overcoming challenging problems is one of the most engaging activities there is. People thrive when their day-to-day work involves getting good at conquering meaningful challenges. Companies do much better when they wake up the sleeping giant in each employee by encouraging them to reduce the friction that gets in the way of delivering value to customers.
If your company is not the highly successful leader-in-its-field that you hoped it would be (and no company ever is), then waiting around for things to change is not likely to make the situation better. Round up your colleagues and assess the situation. Find the elephant in the room and imagine what things would be like if it were gone. And then – since you are smart engineers – you need to engineer a way to get that elephant out of the room. Quit waiting for someone else to do this for you. You’re on.
______________________
Footnote:
1. One proven set of principles for tackling tough technology problems are the Lean principles: Focus on Customers, Energize Workers, Reduce Friction, Enhance Learning, Increase Flow, Build Quality In, Keep Getting Better.
What are the right questions? Consider this guidance from Taiichi Ohno, the father of Lean:
All we are doing is looking at the time line, from the moment the customer gives us an order to the point when we collect the cash. And we are reducing the time line by reducing the non-value adding wastes.In the product development world, our timeline starts with a consumer problem instead of a customer order:
We look at the time line from the moment our consumers experience a problem until that problem is resolved. And we reduce the time line by reducing the non-value adding friction.The technology stack of 1995 generated different kinds of friction than you will find in a modern technology stack. When banks moved to mobile apps a few years ago, they discovered that app development requires an agile approach because the underlying platforms change all the time. While the old technology stack resisted agile practices, the cloud demands them. There is no place for large projects or long release cycles in the new technology stack; agile development is simply table stakes - you need it to play the cloud game.
The new technology stack produces its own friction, a different kind of friction than was typically found in the old stack. This friction is particularly strong in organizations moving from the old to the new technology stack because the transition brings a lot of change to software development. Unfortunately, that change is not always well supported by the organization or welcomed by the software engineers.
Friction Generator #1: Since the new technology stack virtually requires small deployments, the development team can - and should - become deeply involved in designing differentiated products using tight feedback loops. In short, the development team becomes a product team. But frequently this product team does not have the right people (designers, for example), the authority, or the process to make dynamic product decisions. Too often development teams are told what to develop, rather than being asked to move business measures in the right direction. A lot of friction can occur if the organizational structure does not support the concept of fully responsible product teams.
Friction Generator #2: The development team must engineer solutions to quality, reliability and resilience issues that arise after deployment. This requires a different mindset than was common with the old technology stack, when the development team sent their code to the ops department, whose job it was to keep the system running. In the cloud, a team procures and releases to its own infrastructure, and there is no one else to deal with the inevitable problems that occur. Product teams must have the capability, the charter, and the mindset to accept 24/7 responsibility for their deployed code.
Friction Generator #3: The new technology stack is designed to be fault tolerant, not failure proof. This means that any service or app must be able to fail and get restarted at any time, and not produce problems due to these interruptions. But writing "restartable" code [idempotent modules with immutable data sets] is new to most software engineers and is rarely taught in schools. Software engineers skilled at writing code for the new technology stack are in short supply and demand is intense. Good leadership, training, and support are required to help interested software engineers transition to the new languages and paradigms needed to thrive in the cloud.
Friction Generator #4: The old technology stack and associated batch processes encouraged extensive outsourcing, leaving many IT departments without software engineers or even data centers. Today, as software drives differentiation, many firms are attempting to bring software technology back in-house. But they often lack the management experience, organizational structure and personnel policies necessary to attract and retain the skilled software and reliability engineers they need for the new technology stack.Today, almost every business has to face the fact that their most serious competition is likely to come from companies living in the new technology stack, unencumbered by the old way of doing things. Governments and non-profits must realize that the people they serve have their expectations set by experiences with the cloud. If your organization is living in the old paradigm, it’s time to move on; big back end systems are rapidly becoming the COBOL of the 21st century.
To assess the current situation, take a look at the value stream – the stream of activities that deliver value to customers – and identify areas of friction. In the modern technology stack, friction generators tend to be either deeply technical or highly organizational in nature, as you can see from the discussion above. Unfortunately, these are not usually the problems that companies tackle when they move to modern software development. Why? Quite often the organizational structure is so entrenched that changing it is not considered. Or perhaps the people leading the transition do not understand the underlying technology and the problems presented by the new stack. In either case, the underlying problem becomes an elephant in the room that everyone ignores, while easier challenges - like adopting agile processes - are taken up.
It is important to confront the deep-seated friction generators that people would rather ignore. Start by talking about the elephant, and then actively imagine what your world would be like without that elephant. Once you have a clear vision of the future, you can work out how to move constantly toward that vision by eliminating the most pernicious friction generators, one step at a time. This approach has helped teams and organizations around the world make steady progress in the right direction, and eventually the steady progress adds up to amazing accomplishments.
Identifying, addressing, and overcoming challenging problems is one of the most engaging activities there is. People thrive when their day-to-day work involves getting good at conquering meaningful challenges. Companies do much better when they wake up the sleeping giant in each employee by encouraging them to reduce the friction that gets in the way of delivering value to customers.
If your company is not the highly successful leader-in-its-field that you hoped it would be (and no company ever is), then waiting around for things to change is not likely to make the situation better. Round up your colleagues and assess the situation. Find the elephant in the room and imagine what things would be like if it were gone. And then – since you are smart engineers – you need to engineer a way to get that elephant out of the room. Quit waiting for someone else to do this for you. You’re on.
______________________
Footnote:
1. One proven set of principles for tackling tough technology problems are the Lean principles: Focus on Customers, Energize Workers, Reduce Friction, Enhance Learning, Increase Flow, Build Quality In, Keep Getting Better.
