Balancing innovation in software engineering teams

Software engineering as a field of constant change #

Software engineering is a relatively new discipline and for that reason alone we should question its fundamental axioms on a regular basis. We do not seem to be in a place where we have good, fairly future-proof answers for most problems like older disciplines do. Practices such as extreme programming or micro-services created massive disruptions in the field not so long ago and things do not seem to have slowed down, with the advancements in machine learning creating another major shift being just one example.

As software engineers, we need to periodically review and question assumptions left over by previous eras. On a more practical level, this means maintaining a healthy discipline in encouraging experimentation and change. There's nothing quite like the fresh eyes of a newcomer in the moment of seeing a project for the first time. That's why rotating developers between projects can be a very valuable strategy for keeping things in check.

The inherent value of experimentation #

We should also recognise the value of experimentation in and of itself. All mammals have the play circuit in their nervous system which is often the birthplace of new behaviours, biologically speaking. In humans, the playful pursuit of curiosity allows the discovery of ideas and solutions that could not be found otherwise. In software engineering teams this can take the form of regular hackathons on side-projects - but even better if it's allowed within the main product to a certain degree as it's sometimes difficult to translate knowledge from hackathons to real projects.

Experimentation is particularly important in the current era of obsession with productivity. The narrow field of view that's a result of such a mindset certainly has its place at work but can lead to pathology if not kept in check and combined with a broad field of view that's associated with play.

To allow and encourage play in your team you need to ensure their psychological safety. Nobody will want to play if they're worried about a predator that might be lurking around. That's why it's important to keep a healthy attitude towards mistakes made by the team. With true innovation, mistakes will happen.

The risks of stifling innovation #

We need to recognise experimentation as something that has value in and of itself and a requirement and a sign of healthy software teams.

Doing things as they used to be done is easy. It's low-risk. If a project fails, it's less likely it will be blamed on the old way of doing things. They did work for so long, and it's not likely attention would be drawn to them as a cause of failure. Often something else gets the blame, like the lack of time ("we did it this way because we didn't have time for major changes") or the need for consistency.

Teams are often afraid of bringing too much chaos into the system as it makes things less predictable. That's the inherent nature of change. Things destabilise for a while until a new state of equilibrium is reached. This is required for growth.

The exact amount of change versus stability can be a subject of debate but it's safe to assume software engineering will require more innovation than other, more established fields as it still lives on the edge. Not allowing for a degree of innovation can not only lead to boredom and or/burnout of certain team members but ultimately failure of the product.

It's a balancing act #

Innovation is great, but you can have too much of the good thing, and both the lack of innovation and the lack of rules can be a sign of dysfunction. Too much innovation can cause a lot of stress, especially for team members that are more resistant to change. It should be approached with caution and be kept under constant review to make sure the right balance has been reached.

Balancing innovation brought on by new ideas and the stability of established ways of doing things is a complicated dance. And although it's hard to say what's the right balance between the two and will most likely depend on the circumstances, one thing is for certain - you need them both.

• Previous: Enzyme encourages bad testing practices