Does Agile need architecture to be successful?

At a recent Agile conference, one of the speakers opened his speech with the words: „Agile without a plan is only chaos“. Agile without effective architecture inn place will eventually lead to chaos, especially when organizations try to scale their Agile practice without any reference framework.

As architects, we all try to operate within partially given constraints, which in their nature can be customer oriented, financial, regulatory, technical or competitive. While agile practices have traditionally been limited to software development, companies, especially large enterprises, are under considerable pressure to use agile practices to manage traditional business functions. This new trend is euphemistically referred to as „New Ways of Working“. The benefits of using agile practices are many, with the fundamental advantage that companies see agile practices as a way to deliver better results to their customers and stakeholders – faster, more efficiently, continuously and more consistently. This can only work if basic enterprise architecture elements are defined and a framework is provided that supports agile procedures.

The use of an architectural framework that is perceived as old-fashioned, such as TOGAF®, seems to stand in direct contrast to the agile and slim view of the world. It is extremely detailed with numerous frameworks, techniques, and templates described in a fairly sterile manual covering nearly 700 pages. This does not fit in well with the agile focus on favouring working software over documentation.

There’s a cultural difference here, like everything else. Agile is committed to unrestricted teams and promotes direct collaboration, while the world perceives TOGAF as architecture boards and carefully indexed document repositories.

It is important to note that TOGAF captures delivery objects that are created and needed in a large organization anyway, regardless of whether they are executed as part of a framework. Enterprises should all have a kind of architectural vision that is aligned with a business strategy. They need to track their capabilities and applications portfolio, perform costing calculations, make architectural decisions, reduce unnecessary repetitions, document processes, enable common agreement on how to proceed, and ensure that decisions are fair and transparent. At the same time, TOGAF artifacts such as lists, matrices and models can be reused in other organizations such as quality management, business process management, ISO certification documentation, etc. New approaches in low-code application development such as APPIAN allow the direct reuse of BPMN 2.0 models – created e.g. during architectural work – for the agile development of applications with minimal programming effort.

TOGAF is not a rigid frame that must be strictly followed. Rather, it is intended to actively encourage practitioners to adapt and modify the framework to the circumstances. This means that the different parts of TOGAF can be adapted and adopted to agile development. Many believe that TOGAF implies a waterfall approach – and that’s not exactly true.

SAFe

Most agile frameworks that try to implement Scrum for large organizations always have a placeholder for architecture. The main difference is that they tend to emphasise cooperation, simplicity and decentralised governance. This can take place in its full beauty in the capability architecture within the framework of strategic and segment architecture.

The SAFe (Scaled Agile Framework) framework has contributed most to identifying the role of architecture in an agile environment. As with all things around Agile, it’s about creating a consistent value, and architecture is no different.

The Architecture Domains in Safe are the same as the Architecture Domains in TOGAF, which simplifies the assignment very much.

SAFe defines two different elements of architecture.

Emergent Design

Emergent Design provides the technical foundation for the development and gradual implementation of initiatives. It helps designers and architects respond to changing customer and stakeholder needs to ensure that the initiative continues to create value. At this level, SAFe practitioners see architecture as a collaborative and interactive exercise that can create the design element.

Intentional Architecture

Intentional architecture is a more structured approach and is more aligned with what many would call traditional architecture, i.e. a set of defined and planned architectural initiatives designed to support and improve the performance and usability of the initiative. Indeed, Intentional Architecture is a clear realization that we all must operate within certain constraints, such as the choice of architectural guidelines, technology platform, financial budget, etc. If these constraints can be identified and integrated into the initiative, the likelihood that the initiative will be successful and deliver value increases, or the results will trigger architectural adjustments as part of Architecture Change Management.

SAFe practitioners claim that by balancing emerging design and intentionality, agile practices can be scaled to deliver enterprise-level solutions. In Safe, this combination is related to the Architectural Runway, which provides the technical foundation for creating business value, which is fully consistent with traditional views of architecture.

The key to the success of this approach lies in the level of abstraction at which the balance between Enterprise Architecture, Emergent Design and Intentional Architecture takes place.

Even in an agile world, the role of the architect remains a key one. Training and, if necessary, certification in one of the frameworks, such as TOGAF, is sensible and effective. Some architects must also have knowledge, skills and experience in Agile, SAFe and SCRUM. All team members must understand how enterprise architecture and Agile fit together.

The ability to work together is one of the key skills needed here. Architects must be able to productively collaborate with Agile teams to provide rapid and local support in the application of Emergent Design, while helping Agile teams identify and manage the constraints defined by Intentional Architecture. One of the most important features of Agile Practices is the fact that Agile teams are encouraged to provide continuous feedback to their stakeholders. When developing new designs, architects can use this information to adapt and develop Intentional Architecture to ensure that the overall architecture of the organization evolves with the organization in the medium to long term.

Agile and ADM

The core of the TOGAF is the Architecture Development Method, ADM. Looks like a waterfall approach, but it’s not necessarily. The individual phases and groups of phases can be completed in an agile manner and in sprints.

One of the primary goals of Phase A is a consensus on the „vision“. One could argue that a phase A defines an „epic“. Alternatively, another phase A could define a „User Story“. Phase A can define the „Program Increment“ for an „Agile Release Train“. Each of these things can be mapped at different levels on Phase A: Architecture Vision. Transferred to SAFe Agile, Phase A could also map and simplify the definition of „strategic topics at the corporate level“ or „value streams at the program portfolio management level“. TOGAF’s „Capability-Based-Planning“ aids here.

From an enterprise architecture and IT perspective, capability-based planning is a powerful way to ensure that the strategic business plan leads the enterprise from top to bottom. It can also be adapted using capability engineering to take advantage of emerging bottom-up innovations. Regardless of how the organization itself is structured, it must manage the delivery of business capabilities whose implementation requires coordination and alignment across all business units. The functions are ideally business oriented. One of the main challenges is that the benefits are often achieved at company level and not at divisional or departmental level. As a result, projects in sector- or department-oriented portfolios tend to work more from their own perspective than from the company’s perspective. Managing the delivery of a capability is a challenge but anchoring a capability-based perspective in a company is a powerful mechanism to achieve synergistically derived business value that translates into profitability and stock value.

Every agile iteration at every level – and every ADM iteration at every level – must involve stakeholders and their requirements. Some stakeholders can be considered „Epic Owners“ or „Persona“ in a „User Story“.

Stakeholder management is of crucial importance in TOGAF. Anyone who fails in stakeholder management risks not achieving the goals that have been set. 80% of transformation projects fail precisely because of this – and at Agile everything revolves around stakeholder management. In an agile context, Epic owners are directly involved throughout the agile design process. Agile says that and TOGAF says exactly the same thing.

Requirements and design arise when we iterate in agile sprints and ADM cycles. Always directly and continuously connected with our stakeholders – shoulder to shoulder, in the same room, in the same Scrum teams.

It must be decided to what extent phases B, C and D are relevant and what needs to be done in relation to these architectural domains or „concerns“ (Concerns). Ideally, the result of phase D is already defined, and it can be assumed that the technical architecture is rather durable, already given and reusable. So, the teams in your sprints focus only on phase B and C – the business architecture, application architecture and data architecture. Of course, it doesn’t have to be that way. You might as well do agile design and iteration for the provision of an infrastructure technology or platform, but I don’t think Agile was really designed for that purpose.

It’s likely that phase D: technology architecture is a kind of „architecture epic“ that we will realize separately. I don’t think Phase D is very often designed or realized in a sprint or program step. Rather, I am convinced that an agile sprint will focus on business architecture, application architecture and information/data architecture. Phases B and C.

It is necessary to determine what is available and given (baseline) and where they want to go (target). In fact, development is rarely carried out on greenfield sites. Therefore, it may be good to know what we’re assuming. There may be restrictions or requirements that need to be known, there may be principles that have already been adopted that need to be adhered to, Phases B and C can be mapped to Agile/Scrum practices. One could say that phases B and C – performed in parallel – can define a „sprint“ and that phases B and C – performed in parallel – can define an iterative „agile release train“ (ART), i.e. a „program increment“.

It can be decided that Phase B – Business Architecture-  is only executed in agile iterations. This may be the case, in particular, at the level where strategic business themes are defined or where consideration is given to changing the major value-added flows at the ‚programme portfolio management‘ level. At this level, perhaps the strategic „what“ is more interesting than the tactical „how“.

I think whoever changes an aspect of the business architecture will almost always change an aspect of the application and information/data architecture. And this is exactly where agility is required. Most of the flexible release trains I see combine – and focus almost exclusively on – business architecture, application architecture (presentation, business rules, workflow) and information and data. So they constantly iterate about phases B, C and rarely D.

After some incremental iterations of phases B, C, and possibly D have been performed, it is necessary to consider a number of alternative options in phase E for implementing the changes made in an agile release train. This may include checking a number of different options for assembling and reusing architectural building blocks from the TOGAF Architecture repository. Which parts best realize the „minimum viable product“, which parts should be omitted or removed and reinserted into the Kanban-Backlog? Sometimes phase E may not be required and can be combined with phase F. In an Agile + DevOps environment, phase E and phase F become the same.

What is planned in phase F can be developed immediately and completely by a DevOps team, implemented, tested and made available as separate iterations parallel to the ADM cycle. It is important that the information required in ADM phases F and G is always provided  in order to ensure any adaptations in the architecture and/or architectural framework. Thus, one can imagine phases F and G as „DevOps iteration“ of the ADM. And DevOps is agile. If you really want to be agile, you should try to reach DevOps.

In Phase H: Architecture Change Management, the Kanban backlog can be managed. I regard Phase H as the place where the „architectural debts“ are recorded. Architecture Debt is not even mentioned in TOGAF. But it is a general term used by architects all over the world. Sometimes they call it „technical debt“ or „corporate debt“. Once increments are implemented in the production environment, the team in Phase H immediately reviews the Kanban backlog and begins planning and executing the next agile release train, the next program increment, the next epic and the next sprint, the next Phase A.

Agile and TOGAF: No contradiction

In this sense, there is no conflict between architectural frameworks and agility, as both try to solve different problems at different levels. At the level of Strategic Architecture and Segment Architecture, enterprise architecture deals with the rather long-lived definition of business principles and standards, application building blocks, infrastructure and the standards within which agile teams work. Agile methods are more about the most efficient organization of a software development team and are therefore best suited at the capability architecture level for fast iterations.

Enforcement and governance

Despite this separation, there is a point at which the two approaches can collide, mainly when it comes to enforcing the strategic and segment architecture and governance.

Although we want agile teams to be autonomous and self-governing, they are still part of a broader organization. Agile teams in large organizations can never be completely autonomous because they operate within the boundaries of data, conventions, technologies, rules, and strategies set for the entire organization. TOGAF tries to define a structural process to make this boundary transparent and inclusive.

Compliance checks ensure that best practices are being applied, standards are being met, and a common infrastructure is being used. This can be a very detailed process – a full compliance check includes a checklist of more than a hundred elements, including hardware, information management, security and technical procedures.

TOGAF plans to adapt the standard checklist to focus on high-risk areas with the intention of bringing an external panel of architects to a very detailed level of backlog of agile development teams. Architectural governance is necessary and okay when it comes to the wider context that teams cannot recognize. It is less appropriate where development aspects are concerned where the teams themselves should be the recognised experts.

Does Agile need architecture to be successful? „Yes!“ I would say, and extend the question: „What kind of architecture does Agile need to be successful?“ Agile requires a stable architecture that supports the way Agile practice delivers results and added value. The type of architecture that will do this must be a combination of an agile, reactive architectural style supported by a more traditional, structured approach to architecture. The challenge, as with many things, is to find the right mix.