Several of the companies that I work with build very complex systems that are hard to break down into largely independent parts. Instead the components in these architectures are internally complex and have elaborate dependencies between them. Although this is an architectural challenge, it also leads to an organizational challenge.
Traditional R&D organizations have a preference for component teams where the team owns a component and coordinates with other teams over the interfaces between this component and other components. Assuming the BAPO model (see one of my earlier posts), this is a model where the organization follows the architecture which generally should be considered a good thing. A component team has a solid understanding of the component that it is responsible for and consequently can limit design erosion inside the component.
The disadvantage of organizing R&D as component teams is threefold. The first is that it leads to coordination overhead as most features and other R&D work items affect multiple components. With component teams, the functionality needs to be assigned to the different components and the teams need to agree on how to extend or change the interfaces between them in response to the new functionality. This requires teams to spend significant amounts of time together as the initial division and interface specification never works as intended. Second, the architecture of the overall system is difficult to protect as each component team will optimize locally, but feel little responsibility for the overall system. Finally, component teams have a tendency to ensure that they have enough work. So, teams and their product owners/managers will fill their backlog to ensure that they have work, rather than prioritize the R&D effort based on what is most important for the company overall.
In recent years, I have been a proponent of feature teams. A feature team is responsible for realizing a feature and can make changes in any component that is involved in the feature. The major advantage is twofold. First, the inter-team coordination overhead is removed and the team only needs to align internally. Second, the realization of the feature tends to be much more homogeneous as all functionality is built by one team.
The disadvantages of feature teams are mostly concerned with protecting the architecture, ensuring quality and competence. In organizations successfully using the feature team model, these disadvantages are counteracted by several means. First, these organizations appoint architects responsibility for protecting and explaining the part of the architecture they are responsible for. Second, continuous integration allows for rapid and frequent feedback on work performed by feature teams, ensuring that teams breaking functionality get immediate feedback. The main challenge that I see in companies that I work with, though, concerns competence: these systems have parts and interactions that are so complicated and that take so long to become proficient in that it is unreasonable to expect that a feature team can cover this all by itself. Even when using feature teams in this context, these teams spend enormous amounts of time with other teams, experts and architects in order to understand what needs to be build and how it is best realized.
The above leaves us with a bit of a conundrum because both models lead to significant coordination overhead in the context of complex systems. When basic models fail, as a leader you need to return to first principles. For me, there are four basic principles that should be followed when organizing R&D in complex systems.
First, autonomy is king: To the largest extent possible, teams should be organized such that their autonomy if the highest possible. Coordination cost, as expressed in meetings, documents, emails back and forth, etc. are simply the death of productivity. Coordination should be managed through architecture and automation, such as the continuous integration infrastructure, and not through human processes.
Second, less is more: Many R&D leaders always seek to add more resources to the organization, leading to more and larger teams. In my experience, adding resources can easily lead to so much additional effort that the productivity of the organization sinks below that of the earlier, smaller organization. The aim should always be minimize amount of R&D resources involved as productivity asymptotically drops when growing the organization.
Third, hybrid works: In several companies that I have worked with, homogeneity is the highest goal. Everywhere in the R&D organization, the same team setup, processes, ways of working and tooling are used. The underlying assumption is that it simplifies things for everyone involved and it allows for greater mobility of staff. Especially in complex systems, however, this approach breaks the Einstein principle of making everything as simple as possible, but not simpler. In reality, it is often much better to be flexible and allow for different models within the R&D organization responsible for a system.
Finally, break architectural complacency: Due to their complexity, the companies building these systems tend to have been around for a long time. Due to resource constraints and limits in computing power years or decades back, the system architecture frequently became so complex as deep cross-architecture dependencies were required to meet challenging quality attributes. The organization then easily develops a shadow belief that this highly complex architecture is a necessity and a fact of life, resulting in architecture complacency. The final principle is that the R&D organization needs to frequently reevaluate the need for the elaborate dependencies between different parts of the architecture. Simplifying dependencies and adopting highly decoupled approaches such as service-oriented and micro-service architectures not only decouple components, but also increase the autonomy of teams. The miracle of Moore’s law helps immensely address these issues.
Concluding, as an R&D leader, it is tempting to grasp for models that have worked in the past or that work well at other companies. It’s so easy and comforting to have a template and recipe to bring to the meeting and to use as an anchor. Over the years, however, I have become increasingly convinced that each organization is unique; similar, but not the same. Consequently, every organization needs to find its own path and go through its own journey to achieve the desired outcomes. This requires leaders to start from first principles and reason through and experiment with the implications of these principles. To paraphrase the Chinese saying, don’t be the imbecile that looks at the finger, but rather look at the moon the finger is pointing at. Teams follow leaders that set high bars for themselves and their organization, so start from principles and reach for the stars!
