Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.
Goodhart’s law (source Wikipedia)
Or, in other words, once you tell people “improve that KPI or else”, that KPI will be abused to the point of becoming useless: the KPI will improve, but the overall quality/speed/cost (pick a metric you really care about) will not necessarily improve – you might even notice a deterioration because people spend more effort on keeping the KPI happy instead of doing their work. Reduce regressions by increasing test coverage? Developers will produce trivial tests that just run through code and don’t verify anything. Increase team productivity by incentivising velocity? The team will assign higher story points to the same tasks. Improve air quality through reducing CO2 emissions? Car companies work around that instead of building better cars.
Another example is service providers who are incentivised for matching SLAs. Time to completion is almost always part of an SLA, so a sly service provider might eagerly close tickets without waiting for the customer’s acknowledgement or they might try to convince the customer to reopen the ticket, which essentially resets the countdown. Once reopen counts are tracked in the SLAs, the friendly help desk representative might prompt us to open a new ticket instead of keeping the current one open.
The issue with KPIs is dishonesty
The KPI-quality disassociation is a basic, symmetric dishonesty on both the regulator and the producer side of the bargain. From the producer’s point of view it is easier to optimise for a KPI than for the quality it is intended to measure: if people buy vegetables by weight, I’ll just soak them in water first. From a regulator’s point of view it is easier to monitor a few metrics half-arsedly than engaging in complete quality control: it is easier to measure a cucumber’s convexity than savouring every batch in a salad. KPIs become a legal cat and mouse game between regulators and producers because both want to save time and money and because neither are the consumer who cares about affordable quality.
Tighten KPIs until it becomes cheaper to build good quality than faking it
The conundrum’s solution is picking KPIs that are easier to monitor and to get right with a quality product than to fake, which means that KPIs must evolve with the state of technology. For example, it’s easy to produce high test coverage when unit tests touch a lot of code (easy) but don’t verify the code’s behaviour (hard) – the problem is that those tests produce excellent KPIs but assert very little quality. A simple solution is mutation testing which actually tests the tests we write and is nearly impossible to fool.
One should note that KPI ≠ metric. A metric is something measured, it’s an FYI you don’t necessarily act on. A KPI is a composite indicator which consists of one or more metrics that are bound to the performance of a product. Your product should have many metrics which you might not even look at but just record for future use while KPIs are actionable indicators of how we are doing.
A good KPI is customer satisfaction – after all, good products make happy consumers, but that KPI doesn’t come without issues: it takes a while to compute and can be measured only after the product or service hits the market. Customer satisfaction also obscures flaws; eg. drivers are unlikely to complain about emissions unless the government taxes them for their wasteful cars.
Some recommendations for managing KPIs:
- Pick KPIs that are meaningful, honest and that you care about regressions and epic completion
- De-prioritise artificial KPIs like code complexity
- Don’t pay for better numbers: a KPI is either sufficient or it isn’t
- Tie KPIs to deliverables, eg. get paid for unit tests and documentation
- Assert a KPI’s effectiveness: if good numbers aren’t making your product better then remove the KPI
- Don’t conflate product and production KPIs: if you work for an CMMI certified organisation then code quality is a KPI your boss rewards you for, not the customer
- Monitor all dimensions: structure, processes, testing, resource consumption, consistency
If you are interested in more about quality, KPIs and managing service providers I recommend the accordionist and the value of specifications.
This article appeared first on George’s Techblog.
George's Techblog 