How Engineering Leaders Can Map Developer Productivity To Business Goals?
Talk about the “Why” to gain momentum on developer metrics
During the last decade the tech industry has seen a number of new products that were launched to improve developer experience and developer productivity. There are still murmurs in the industry that nobody has cracked the perfect way to measure, iterate, and improve developer efficiency.
If there is one question that Engineering Managers dread being asked, it is how they are measuring the efficiency of their teams.
One of the biggest challenges Engineering leaders encounter is identifying which metrics are important for their teams, and how those metrics translate to helping the company achieve its goals. Between DORA, SPACE, and a multitude of Developer Experience SaaS products, there are hundreds of metrics that are presented as valuable to the engineering organization. Whoever tells you to build dashboards and present all the metrics recommended in DORA and SPACE is sending you for a wild ride, and you should question its value.
At any given time, engineering teams are actively engaged in one of the following, seldom both at the same time:
- Building a new product or feature to capture a new market segment
- Refining an existing product or feature with the intent to improve user experience and increase market adoption
These two business-focussed outcomes are mapped to two KPIs that are invariably relevant for product organizations: Time to Market, and Time to Value.
Aligning engineering metrics with business goals and KPIs clearly explains the “Why” behind each metric. It is a better way to build momentum among engineers and also get buy-in from the business to invest in improving the metrics.
Engineering Metrics associated to Time to Market:
Time to Market is measured by the length of time from the conception of a product idea, until the product is released to the market. Time to Market can be measured from the time an idea is conceived until the time the product is launched for an alpha or a beta release. The goal is to release the product in a way that creates enough opportunities to gather customer feedback.
If engineers are working towards a product launch that is associated with a Time to Market KPI, the following metrics will be invaluable to measure, iterate, and improve.
Cycle time — The time it takes for a ticket (unit of work) to move from InPrg to Deployed. Cycle time reflects the efficiency of the development practices followed by the team, and directly impacts the Time to Market of a new product. A cycle time that is trending high is affected by many factors: unclear functional requirements, developers learning to program in a new tech stack, lack of test and deployment automation, rework due to code review feedback, and low code review velocity. When cycle time trends up, it’s time for the team to dive deeper and figure out why. Moving the needle on the Cycle time early on in the development lifecycle gives a sure shot success on achieving a faster Time to Market.
Deployment frequency — How often the team successfully deploys releases to production. When teams are focussed on bringing a new product or a feature to market, they do not pay attention to how often the feature branch is merged and deployed to production. This is because the new features are not launched as and when new lines of code are added to the codebase. Frequent deployments (I won’t say how frequent, because each team should use their contextual judgment) mean that you are improving the team’s confidence to move faster, because the code for the yet-to-be-launched feature is live in production, and hasn’t caused a lot of havoc. And even if it has caused havoc, the team has already figured the root cause, and has learned ways to eliminate it in future deployments.
Code review velocity — Speed at which code changes are accepted and reviewed during the development process. A high code review velocity may indicate that the team is able to accept and review code changes quickly, which translates to better cycle time and faster Time to Market. A team that is built on strong development principles will have a code review velocity that trends high or plateaus at its peak.
Code review acceptance rate — Measures how often code changes are accepted during the code review process. This is a useful metric to track the team’s progress in meeting deadlines. A low acceptance rate means that the teams are in a rush to get the product or feature out, and as a result are not paying enough attention to the quality of their work. Rejection of PR during code review leads to rework, longer cycle time, and slower Time to Market.
Number of commits — Measures the efficiency and productivity of the team. It is obvious that the more code commits, the faster the flow of work, and hence the team can achieve faster Time to Market. I would emphasize not to use this as a singular metric, but one that is used in conjunction with other metrics.
Engineering metrics associated to Time to Value:
Time to value (TTV) is a critical metric that measures the length of time it takes for a product to provide value to the user. It is used to measure adoption rate for newly launched products.
Products are launched with an expectation to have faster Time to Value. However, in reality, most products go through a longer adoption curve. After launching an alpha or beta release of the product, engineering teams often spend a few sprints to decrease the Time to Value, or increase the adoption rate of newly launched features. Here are some metrics to measure, iterate, and refine, ultimately making features valuable to the customers in the fastest possible time.
Lead time for changes — How long it takes a PR to go from code commit to successfully running in production. When the users provide feedback, teams should aim to make the necessary changes to the code base and ship it to customers in a short span of time. Shorter lead time for changes keeps your alpha and beta customers engaged in the process, and helps to improve the adoption rate.
Change failure rate (CFR) — The percentage of deployments causing a failure in production. If the CFR is trending high, it means that the feature is not thoroughly tested, and the deployment process is not foolproof. High CFRs are an indication that the team should focus on strengthening the test and deployment automation, which will also reduce lead time for changes.
Bug to Code ratio — The number of bugs reported per line of code. The normal way to measure the Bug to Code ratio is the number of bugs created per 1000 lines of code. A high Bug to Code ratio is an indication of two things: developers not having the bandwidth to deliver high quality code, and tests not being good enough to catch these bugs prior to deploying the code to production. An increased number of bugs per change leads to customer frustration and results in customer churn.
Mean Time to Recover (MTTR) — Measures the time it takes to restore a system to its usual functionality. If it’s an early alpha release, and the product is being tested by trusted customers, the team has leverage to extend their time to recover. The team should use such early failures as an opportunity to learn about the unknowns that will come in handy when the product goes for a wider release. The goal is for the MTTR to trend down as the number of customers and the adoption rate increases. On the contrary, if the MTTR continues to trend up, it is a recipe for unhappy customers.
Service Uptime vs downtime — Uptime SLAs are not very demanding during the early days of a product release, when it’s only used by a handful of customers. However, as the adoption rate of the product increases, the team should closely track the Service Uptime, and aim to deliver the maximum SLA that similar products in the industry offer. Every potential incident shrinks the error budget, leaving less room to experiment changes in Production that cannot be simulated in a pre-prod environment. A 99.9% SLO has an 0.1% error budget.
Developer satisfaction metrics that indirectly contribute to Time to Market and Time to Value
If I told you that metrics should only be measured as a function of Time to Market and Time to Value, you’d be on a path to lose trust from your developers. The way you measure, iterate, and share feedback becomes transactional in this scenario. Developer satisfaction has an exponential impact on meeting Time to Market and Time to Value KPIs. Organizations run annual employee surveys to determine the Employee Satisfaction Index. However, the surveys cducted for an entire organization do not provide enough insight into the challenges and opportunities specific to developers.
Below are some metrics that help derive developer satisfaction.
Developer satisfaction — The degree of satisfaction among developers, and whether they would recommend their team to others. This is generally measured through bi-annual surveys (both anonymous and identifiable), 1:1s with managers, and feedback loops..
Developer efficacy — Assesses whether developers have the tools, processes, and resources they need to get work done. It is generally measured by conducting surveys and micro check ins.
Burnout — Burnout is a chronic workplace stress that is caused by a stressful and overworked culture. This is constantly prevalent in workplaces due to aggressive deadlines and lack of context and priority on the work. The way to measure burnout is by asking employees if they are striking a good balance between work, life, and passion projects.
Most of the datasets gathered to assess developer satisfaction are subjective and qualitative in nature. Engineering leadership team should meet every quarter to go through this data and assess the overall health of their teams. The outcome of these conversations (measurable actions) will lead to change in the way the teams work. Being empathetic to the needs of your teams and staying conscious about your teams’ health is key to reaching organizational goals.
Some other critical by-products of assessing developer satisfaction is understanding work breakdown and distribution, understanding the availability of focus time during the day, and creating a collaborative environment overall. Measuring and improving the developer experience will have positive impacts on reducing Time to Market and Time to Value, thus providing a positive throughput to the organization.
Conclusion
Channeling the engineering team’s focus towards specific metrics helps managers guide them to achieve critical results. A team can choose to actively monitor specific metrics depending on whether it is contributing to a product launch, or refining a product to increase adoption. If the team parallelly contributes to launching a new product while also refining another existing product (this approach is not recommended as it makes the team’s cognitive load unmanageable), the team can choose to classify the metrics as “active” and “passive”. Active metrics are the ones that are critical to the success of the team, and the passive metrics are considered secondary to the team’s success during that specific quarter.
One of the advantages of mapping metrics to the wider KPIs of the organization, is that it makes their work more relatable to the goals and priorities of the organization. Product, Revenue, and Customer Experience teams can relate to the Engineering metrics if linked to KPIs, and are more likely to support Engineering leadership in the endeavor to improve Time to Market and Time to Value. This helps engineering managers to not only gain momentum within their own teams, but also bring organization-wide momentum for Engineering metrics.
