I have coached many agile teams and my goal has been to help them towards high performing teams. I see the following main markers for high performance around teamwork, continuous value delivery of product(s) and relentless focus on improvement.
In the presence of multiple metrics, organizational structure and KPIs/OKRs, if the foundation element of the team is not set right, everything else collapses at the first sound of escalation from the top.
When it comes to data teams, I have found varied flavors of the organizational structures that I have elaborated later in this blog. And there has been one common denominator for successful team. It’s the way they have structured their user stories. It was never vertical from the user interface to the backend. It was always horizontal starting from the user’s need to all the way to the source of the data.
This blog is about critically looking at how agile teams in data can move the needle with ease with one change in the absence of everything else.
Before we go to the crux of the horizontal vs vertical, we need to look at what are the common patterns and differentiators of web and data teams. I have used web teams to generalise the edge teams. Data teams to generalise the analytics team
Web and Data Teams are same in a few areas
1. Dedicated
Dedicated team members would ensure that the work is going to the team rather than team members going to the work
2. Architects and Modeler shared
Just like architects and UX modelers in web development, data modelers can be shared.
3. Team Goals set
Entire team gets the same set of KPIs rather than having different goals based on the type of work they perform. This enables the team to collaborate across their role definition.
4. Faster Product Feedback
Institutionalizing feedback mechanism at the end of every sprint creates a cadence towards developing for user needs rather than being futuristic
5. Test Data setup
Clear test data for the user story enables having acceptance criteria and boundary conditions for validations.
Important differences between web and data team
1) Horizontal instead of vertical user stories
In traditional rest services or web development, vertical stories are better understood from the front end to the back end. However, in the data space, the stories need to be looked at Horizontal from the point of arrival to the point of intelligence
This is an extremely complicated terminology as horizontal stories are shunned in the agile world, while horizontal stories are welcomed in the data space. For adding clarity, when we say horizon story we are referring to how the data is flowing from the staging to the transformation to reaching the data lake/target system
2) Cross functional team is hard
Number of skills are composed within Agile data space from ETL developer, Data modeler, Tableau developer, Data analyst, Data scientist, Data Architect and so on. In traditional web development roles would be limited to Architect, Developer and Analyst. In many teams that I am coaching, I see the role of Analyst getting encompassed within the developer role.
With such a wide variety of skills required to form a team and traditionally the way organizations are setup, the cross functional teams are harder to come by. There is a sense of efficiency over effectiveness that we are seeing in many data teams.
Hence the examples of analysts, designers, modelers being separated completely. More examples of complexities can be found here – http://datavaluemap.com/a-map-of-data-related-roles/.
Horizontal story based on user need
With agile data – few steps can be take to bridge the skill gap and cross functional gap
- Cross skilling one or two members in another skill. I have had great success when one ETL developer got skilled in Tableau reports. This increased the productivity within the team (due to lesser hand offs and waiting) as well as reduced late surprises (easier to find issues earlier)
- Dedicated and Traveling members can be identified. In one of the teams I was coaching, data modeler was identified as a traveler and all other members were dedicated in nature. Hence the traveler was shared between three teams. While the modeler was involved in the quarterly planning, they became aware of upcoming modeling needs. This got the team to have a sense of modeler being part of the team during the time of need at the same time modeler was also not stretched beyond his limits.
3) Tools and devops pipelines
There are number of tools that are available which have a lock in period. Tools need to be standardized right at the unit test level till the end to end automation. The tool selection and standardization of the foundational practices is a huge challenge that I have observed in the Data space.
This leads to couple of unintended consequences
- Longer release cycles
- Multiple silos created
DevOps pipeline creation is more complex in an Agile data scenario due to a few reasons such as multiple integration points, lack of unified tools, lack of automation and longer data setups.
Examples of horizontal stories are illustrated below
Story 1 – Middle tier system created stories based on each module or technology area. After the transformation, the team ended up creating stories moved from right to left with user needs taking the centre stage
Story 2 – Reconciliation needed many manual tasks to be done. The data team’s goal was to move the accuracy closer to 100% to reduce manual tasks. After the transformation, the teams started delivering smaller stories based on manual tasks and made progress with help of data.
Some of the instances and observations on agile data teams are as follows
One flavor of the Agile Data team is as below
Analysis team: This team creates detailed requirements after talking to the requesting team and uses this document to get acceptance
Design team: High level modeling including checking the FACT and DIM tables are taken care by this team and acceptance and walkthrough by the entire team is taken up
Scrum team: This is the team that comprises of some of the designers, developers and testers. Each scrum team accepts the epics that have been detailed by the previous teams and produces the software. Some of the automation, pipeline readiness gets ready in this group
PMO: PMO office takes care of work in take care of the orchestration between each team
Delivery managers: In addition to team management, the delivery managers also work with the team to ensure that there is work for the quarter for each scrum team. Standards of coding and testing is maintained.
Another flavor of agile team
Scrum team: This team comprises of designers and developers. They work in understanding some of the requirements wherever appropriate. They work with the downstream and upstream teams to understand the requirements
CoE Modelers: Modelers are part of a separate team and involved when required
Delivery managers: in addition to team management, the delivery managers are closely working with the teams and requesters to create a release date, design reviews and provide technical oversight for the team
One more flavor of agile data team
Scrum Team: This team has designers and developers along with data modelers who are like travelers to the team. Whenever there is a requirement they are involved in the planning and designing. Else they have other priorities
Product Management: Primarily working with the stakeholders to create demand as well as enable the training for the systems that are being created. They also are involved in getting the users to the sprint review as and when there is a sizable chunk of work ready.
Delivery managers: In addition to team management, they are accountable for training and capability improvement. Creating an awareness of the standards that rest of the organization is working on
If we look at all three flavors of the agile teams in data, my observation is that agile data works differently from software development teams.
There is a lot of commonality between web and data teams. However, the biggest gap arises on how teams are organized. The teams have been presented to provide a glimpse of various stages of evolution that agile data teams are set.
In addition to the experiences with data, some of my inspirations and references are here
1) Agile Analytics by Ken Collier
2) Lean Data modeling from AgileData.org
3) Datavaluemap.org
4) Visualbi.org
