We introduced the topic "Beyond Agile" in "Going Beyond Agile", and in that I had promised to breakout the practice of the three-level agile transformation. This article is first of that series, takes a look at what most organizations do at Level 1 (Team or Execution Level), and adds some more perspectives on approach.
Many organizations go for Scrum as if that were the only Agile methodology and are surprised when they fail or achieve only partial success. So what went wrong?
They focused on the wrong thing!
Firstly, Scrum is a prescriptive methodology that lays down exactly what is to be done, when and in how much time. It defines the roles, ceremonies and artifacts. It is fairly simple to understand and most fall prey to that superficial simplicity and elegance. They then take it as gospel and attempt to ram it through the organization as THE organizational standard! "Pure agile or bust" is the clarion call!
So what's wrong with pure agile?
So what is the solution?
Before that, let's discuss the problem!
We are all aware that 80% of the issues impacting "team cadence" is beyond its control. It's sources lie elsewhere, with Management, with HR, with Sales and Marketing, with customer care and often even with the Executive. That means, everywhere the organization except within the team. So what can the team do about it.
It could be that management calls more people than is desirable for unplanned periodic reviews at the drop of a hat, the talent acquisition teams take 120 days or more to find a replacement for critical resources, IT support is not forthcoming in an acceptable timeframe, architects and systems engineering teams have a large backlog from other teams besides evaluating and implementing the technology roadmap elements, sales has the mobile number of individual product owners and call them directly to log customer issues, instead of going through customer care, and marketing appears to be not addressing the real issues in the funnel and have instead taken off on a tangent on their own priorities.
The problem may be as simple as not having a standardized way of prioritizing issues across the various functions of the organization, to unavailability of a critical resource or data on test or staging servers. Unless these team "environmental issues" are resolved, how to you get the team to a consistent agile cadence?
In other words, there is an extended period of transition and evolution that needs to be stage managed in a manner as to keep interest alive and not overburden the teams with "pure agile". Let things roll as they explore the agile framework and adapt what seems appropriate for that stage of evolution. Let the team decide on cycle parameters. Any attempt to force a transition when the team is not ready will be met with resistance.
But let's get back to whether Scrum is the most appropriate methodology for the organization simply because it appears to be the most popular. How do you determine which methodology is suitable for each team in the organization?
To determine the most appropriate agile method for a team, we need to first understand the characteristics of its workflow. For startups not used to the discipline and levels of accountability of large organizations any attempt to evaluate workflow will seem a huge non-value adding imposition. So getting the idea of an evaluation past the management itself is a challenge.
Here it is opportune to use the analogy of the role of scaffolding in construction.
All construction work is done after a suitable temporary framework - a scaffold - is built. It is a familiar sight - the lattice framework on the outside of buildings, made of wooden planks and metal poles, and used by workers while building, repairing or cleaning a building. It has no purpose other than for the specific and temporary purpose of construction. Once construction is completed, the scaffolding is removed.
Similarly, all organizational change needs scaffolding. And this can take on a variety of forms. At its very simplest, is a constraint like a release window (time constraint), a process for which compliance is made mandatory, a periodic event, or even just a data collection mechanism used to, say, collect data regarding a workflow.
Let us consider a data collection mechanism constraint imposed on the system. This collection mechanism could be in the form of an existing (legacy) system, a temporary open source application or even a simple spreadsheet.
Why is data collection mechanism considered a constraint?
Because all resources in scope of the activity have to mandatorily log requisite data at the prescribed interval, every period (usually daily or twice a day).
And what data is collected?
This is dependent on the objective. For example, if the objective is to understand the characteristics of team workflow, we would need to collect data about requirements, its sources and sinks, tasks, dependencies, incidence and duration of work, to start.
How will this data be used?
Data so collected drives the Agile program. Data about workflow characteristics above, for example, would be used, among other things, to benchmark workloads and team processes, map out dependencies and the interconnectedness of team work nodes and roles.
And why would we impose a constraint such as a data collection mechanism on the team that could possibly be already overwhelmed with the workload?
Okay, so let us consider the phrase "overwhelmed with the workload", as a case in point. What exactly is the nature of the workload? From where did it originate? How long would it take? What will be the outcome? What is the downstream processor for such outcomes? How often does such work come to the team? Who supports it, in case a need for support arises? If resources are pooled, who allocates resources? If there is resource contention, who arbitrates?
There are further uses of constraints. This is better understood by using a complexity management framework like the sense making Cynefin framework.
The figure, the four domains - Obvious, complicated, complex and chaotic - are represented. Most mature firms attempt to create processes in quadrant A where events are simple or obvious, and their cause-effect relations can be pre-determined and set into institutionalized event handler processes. This is the domain of best practices. However, many events that face a firm exist in a more complicated (quadrant B), or a complex and volatile environment (quadrant C), where cause-effect relationships are not easily discernible - where events need to be analysed and interpreted before an appropriate response can be developed or more often than not, probed further through the use of experimentation for it to make sense, and then new practices and associated roles and processes need to be formulated to enable the organization to respond to such events in the future.
The clockwise dynamic (in blue) represents shifting event handlers gradually from quadrant C thru quadrant B to quadrant A. This is achieved by increasing the level of constraints in the system. At some point in the attempted transitioning, the process begins to break, i.e., it can no longer handle the event. In other words, the level of expertise available in the handler processes is no longer adequate to handle such events. These events are of particular interest in the transformation process, however, more on that later. Suffice to say, these constraints that move event handlers clockwise are our scaffolding. Scaffolding is also used to isolate a team from influence of other parts of the organization that are not in scope of the transformation program, for the duration of change.
That said, let's develop the discussion from the point where we said, teams were generally overwhelmed by workloads.
In any organization, the common refrain is that no one has the time as everyone is overwhelmed with work. Really?
Okay, so what exactly are they doing? And how will it impact the customer and what, if any, are the implications on gross revenue?
It is amazing that very few organizations really know what exactly is happening except in very generic terms. You need to be a fly on the wall at the water cooler to understand the underlying narrative. And somewhere between the perceived workloads and the narratives is the truth.
Let us change tack and take another perspective on things.
Consider a simple definition of an agile team.
Voice and leadership style are but just two of the nuances of implementing agile in an organization. As teams explore the agile frameworks, more such nuances are exposed and assimilated. It is these nuances that make agile so difficult to implement, not the methodology itself. Nuances are also organization specific, so program events regularly occur in our Cynefin quadrant C, thus needing careful probing and sense making before a course of action can be determined. Such events usually result in emergent practices.
This article - about Level 1 transformation - was only to highlight the fact that no matter which Agile method you select for adoption, you will merely be scratching the surface in terms of the significant quantifiable benefits that can potentially accrue to the organization. And true benefits can only be achieved if there is significant movement on the nuances of agile, that is, in creating a truly enabling environment for Agile. And until this is achieved, it was all be just so much of pointless thrashing about with very little movement.
In the first two phases of Level 1 transition, the event handler processes and functions increase the level of constraints so as to attempt to move these into quadrant A on the Cynefin framework. Remember, Cynefin, unlike so many other frameworks, is a sense making framework, and we are attempting to setup processes in quadrant A (Simple/Obvious) to deal "all events". Some will succeed. Some will throw exceptions because these events need to be analyzed before they can be handled. Others will need further probing to determine cause-effect. Often there will be no correlation between cause and effect, so typical in market shifts or competitor actions. In each of these cases, a "team" has to determine a course of action.
Once the range of events over the course of Agile transition are "handled", constraints imposed need to be released and simultaneously, new structures need to be defined and institutionalized, new processes have to be setup to handle the remaining events that cannot be handled by "systems & handler processes" in quadrants A or B. More on this later. The end result will look like this.
Note: The Orange box-arrows are event handlers.
While event handlers can be refined to a fine level of granularity, it is adequate to release constraints and have one generic event handler at the junction of Complicated and Complex, a cross-functional "intervention team" that has the mandate to act across functions in the business. After this, the "intervention team" will set up a series of fail-safe experiments through multiple dynamics to verify and validate the readiness of the organization to handle events.
Ideally, we would like to have everyone to have the capacities of the intervention team. But it needs a lot of experience and grooming to develop intervention resources, so organizations start with a handful of well regarded people and gradually expand the team. Intervention team is supported by a capacity to "sense" events well beyond the organizational boundary. This does not mean that the capacity cannot be extended beyond the intervention team. One of the earliest stages of the grooming processes for intervention resources is building this capacity to sense events. The "sensors" feed event queues and the interventionists respond and take action.
All these transitions have to be evolutionary. One cannot go from zero to end-state in one leap. It is precisely why such programs must necessarily take time to evolve at a pace that carries everyone along. Those that cannot adapt, are counselled - multiple times if needed - and if they still fail to adapt, will necessarily have to be let off gracefully. And this is always traumatic. So give it time.
Finally, how do we create an environment for Agile. In our second part of this series we will take a look at how nuances are enabled and also on what needs to be done at the level of portfolio.
Many organizations go for Scrum as if that were the only Agile methodology and are surprised when they fail or achieve only partial success. So what went wrong?
They focused on the wrong thing!
Firstly, Scrum is a prescriptive methodology that lays down exactly what is to be done, when and in how much time. It defines the roles, ceremonies and artifacts. It is fairly simple to understand and most fall prey to that superficial simplicity and elegance. They then take it as gospel and attempt to ram it through the organization as THE organizational standard! "Pure agile or bust" is the clarion call!
So what's wrong with pure agile?
Nothing, if it can be achieved in the timeframe envisaged. That would be brilliant, of course! However, what is most often misunderstood is that the methodology needs a specific set of organisational conditions for it to work. And that, achieving those conditions will take time.
Until the team's external environment becomes agile orientated, nothing other than the mechanics of the method appear to work and with the spirit of Agile missing, it doesn't make much sense to most. After the initial excitement, it begins to appear burdensome and largely pointless.
For example, if "urgent requirements" are pushed to the team mid-cycle, simply because it came from the boss, or, backlog grooming, planning, and retrospectives are given the short shrift due to "lack of time and other more urgent or higher priority considerations", or if, in the context of Scrum, progress is not being tackled using the BDC, and if product owners are more often than not, missing from the daily standups!
So why are all these pointless? Surely you have to trust the team and the larger organisation to know what they are doing, right?
Wrong!
This kind of adhocism, is a waste of valuable time and resources. And more often than not, that "urgent requirement" that came from the COO was just a suggestion to the team for further consideration "possibly in the roadmap for the following year". It is usually some over zealous manager down the hierarchy that assigns it a priority high enough for all other work to be halted.
This kind of adhocism, is a waste of valuable time and resources. And more often than not, that "urgent requirement" that came from the COO was just a suggestion to the team for further consideration "possibly in the roadmap for the following year". It is usually some over zealous manager down the hierarchy that assigns it a priority high enough for all other work to be halted.
So what is the solution?
Before that, let's discuss the problem!
We are all aware that 80% of the issues impacting "team cadence" is beyond its control. It's sources lie elsewhere, with Management, with HR, with Sales and Marketing, with customer care and often even with the Executive. That means, everywhere the organization except within the team. So what can the team do about it.
It could be that management calls more people than is desirable for unplanned periodic reviews at the drop of a hat, the talent acquisition teams take 120 days or more to find a replacement for critical resources, IT support is not forthcoming in an acceptable timeframe, architects and systems engineering teams have a large backlog from other teams besides evaluating and implementing the technology roadmap elements, sales has the mobile number of individual product owners and call them directly to log customer issues, instead of going through customer care, and marketing appears to be not addressing the real issues in the funnel and have instead taken off on a tangent on their own priorities.
The problem may be as simple as not having a standardized way of prioritizing issues across the various functions of the organization, to unavailability of a critical resource or data on test or staging servers. Unless these team "environmental issues" are resolved, how to you get the team to a consistent agile cadence?
In other words, there is an extended period of transition and evolution that needs to be stage managed in a manner as to keep interest alive and not overburden the teams with "pure agile". Let things roll as they explore the agile framework and adapt what seems appropriate for that stage of evolution. Let the team decide on cycle parameters. Any attempt to force a transition when the team is not ready will be met with resistance.
But let's get back to whether Scrum is the most appropriate methodology for the organization simply because it appears to be the most popular. How do you determine which methodology is suitable for each team in the organization?
To determine the most appropriate agile method for a team, we need to first understand the characteristics of its workflow. For startups not used to the discipline and levels of accountability of large organizations any attempt to evaluate workflow will seem a huge non-value adding imposition. So getting the idea of an evaluation past the management itself is a challenge.
Here it is opportune to use the analogy of the role of scaffolding in construction.
All construction work is done after a suitable temporary framework - a scaffold - is built. It is a familiar sight - the lattice framework on the outside of buildings, made of wooden planks and metal poles, and used by workers while building, repairing or cleaning a building. It has no purpose other than for the specific and temporary purpose of construction. Once construction is completed, the scaffolding is removed.
Similarly, all organizational change needs scaffolding. And this can take on a variety of forms. At its very simplest, is a constraint like a release window (time constraint), a process for which compliance is made mandatory, a periodic event, or even just a data collection mechanism used to, say, collect data regarding a workflow.
Let us consider a data collection mechanism constraint imposed on the system. This collection mechanism could be in the form of an existing (legacy) system, a temporary open source application or even a simple spreadsheet.
Why is data collection mechanism considered a constraint?
Because all resources in scope of the activity have to mandatorily log requisite data at the prescribed interval, every period (usually daily or twice a day).
And what data is collected?
This is dependent on the objective. For example, if the objective is to understand the characteristics of team workflow, we would need to collect data about requirements, its sources and sinks, tasks, dependencies, incidence and duration of work, to start.
How will this data be used?
Data so collected drives the Agile program. Data about workflow characteristics above, for example, would be used, among other things, to benchmark workloads and team processes, map out dependencies and the interconnectedness of team work nodes and roles.
And why would we impose a constraint such as a data collection mechanism on the team that could possibly be already overwhelmed with the workload?
Okay, so let us consider the phrase "overwhelmed with the workload", as a case in point. What exactly is the nature of the workload? From where did it originate? How long would it take? What will be the outcome? What is the downstream processor for such outcomes? How often does such work come to the team? Who supports it, in case a need for support arises? If resources are pooled, who allocates resources? If there is resource contention, who arbitrates?
There are further uses of constraints. This is better understood by using a complexity management framework like the sense making Cynefin framework.
The figure, the four domains - Obvious, complicated, complex and chaotic - are represented. Most mature firms attempt to create processes in quadrant A where events are simple or obvious, and their cause-effect relations can be pre-determined and set into institutionalized event handler processes. This is the domain of best practices. However, many events that face a firm exist in a more complicated (quadrant B), or a complex and volatile environment (quadrant C), where cause-effect relationships are not easily discernible - where events need to be analysed and interpreted before an appropriate response can be developed or more often than not, probed further through the use of experimentation for it to make sense, and then new practices and associated roles and processes need to be formulated to enable the organization to respond to such events in the future.
The clockwise dynamic (in blue) represents shifting event handlers gradually from quadrant C thru quadrant B to quadrant A. This is achieved by increasing the level of constraints in the system. At some point in the attempted transitioning, the process begins to break, i.e., it can no longer handle the event. In other words, the level of expertise available in the handler processes is no longer adequate to handle such events. These events are of particular interest in the transformation process, however, more on that later. Suffice to say, these constraints that move event handlers clockwise are our scaffolding. Scaffolding is also used to isolate a team from influence of other parts of the organization that are not in scope of the transformation program, for the duration of change.
That said, let's develop the discussion from the point where we said, teams were generally overwhelmed by workloads.
In any organization, the common refrain is that no one has the time as everyone is overwhelmed with work. Really?
Okay, so what exactly are they doing? And how will it impact the customer and what, if any, are the implications on gross revenue?
It is amazing that very few organizations really know what exactly is happening except in very generic terms. You need to be a fly on the wall at the water cooler to understand the underlying narrative. And somewhere between the perceived workloads and the narratives is the truth.
Let us change tack and take another perspective on things.
Consider a simple definition of an agile team.
The Agile Team is an autonomous,self-organizing, cross-functional group of individuals that has the ability and authority to deliver value incrementally.Let us take the key words one by one.
Autonomous - acting independently or having the freedom to act independently, collectively, democratically, as a group, rather than by external or internal diktat;
Self-Organizing - by itself, arrange into a structured whole; order; coordinate; make arrangements or preparations for; self-organizing teams choose how best to accomplish their work, rather than being directed by others outside the team
Cross-functional - appropriate mix of specialties for a purpose, task or activity; relating to the way in which something works or operates
Value - the importance, worth or usefulness of something; estimate of monetary worth; important or beneficial; advantage; gain; profit; good; help; merit
Incrementally - increasing gradually by regular degrees or additions; regular, measurable movements that are usually smallThe definition also immediately suggests a "safe team environment" to enable equal "voice" to all team members, irrespective of authority by seniority or designation, while at the same time, hinting at a laissez-faire leadership style for its management (of letting things take their own course, without interfering).
Voice and leadership style are but just two of the nuances of implementing agile in an organization. As teams explore the agile frameworks, more such nuances are exposed and assimilated. It is these nuances that make agile so difficult to implement, not the methodology itself. Nuances are also organization specific, so program events regularly occur in our Cynefin quadrant C, thus needing careful probing and sense making before a course of action can be determined. Such events usually result in emergent practices.
This article - about Level 1 transformation - was only to highlight the fact that no matter which Agile method you select for adoption, you will merely be scratching the surface in terms of the significant quantifiable benefits that can potentially accrue to the organization. And true benefits can only be achieved if there is significant movement on the nuances of agile, that is, in creating a truly enabling environment for Agile. And until this is achieved, it was all be just so much of pointless thrashing about with very little movement.
In the first two phases of Level 1 transition, the event handler processes and functions increase the level of constraints so as to attempt to move these into quadrant A on the Cynefin framework. Remember, Cynefin, unlike so many other frameworks, is a sense making framework, and we are attempting to setup processes in quadrant A (Simple/Obvious) to deal "all events". Some will succeed. Some will throw exceptions because these events need to be analyzed before they can be handled. Others will need further probing to determine cause-effect. Often there will be no correlation between cause and effect, so typical in market shifts or competitor actions. In each of these cases, a "team" has to determine a course of action.
Once the range of events over the course of Agile transition are "handled", constraints imposed need to be released and simultaneously, new structures need to be defined and institutionalized, new processes have to be setup to handle the remaining events that cannot be handled by "systems & handler processes" in quadrants A or B. More on this later. The end result will look like this.
Note: The Orange box-arrows are event handlers.
While event handlers can be refined to a fine level of granularity, it is adequate to release constraints and have one generic event handler at the junction of Complicated and Complex, a cross-functional "intervention team" that has the mandate to act across functions in the business. After this, the "intervention team" will set up a series of fail-safe experiments through multiple dynamics to verify and validate the readiness of the organization to handle events.
Ideally, we would like to have everyone to have the capacities of the intervention team. But it needs a lot of experience and grooming to develop intervention resources, so organizations start with a handful of well regarded people and gradually expand the team. Intervention team is supported by a capacity to "sense" events well beyond the organizational boundary. This does not mean that the capacity cannot be extended beyond the intervention team. One of the earliest stages of the grooming processes for intervention resources is building this capacity to sense events. The "sensors" feed event queues and the interventionists respond and take action.
All these transitions have to be evolutionary. One cannot go from zero to end-state in one leap. It is precisely why such programs must necessarily take time to evolve at a pace that carries everyone along. Those that cannot adapt, are counselled - multiple times if needed - and if they still fail to adapt, will necessarily have to be let off gracefully. And this is always traumatic. So give it time.
Finally, how do we create an environment for Agile. In our second part of this series we will take a look at how nuances are enabled and also on what needs to be done at the level of portfolio.
