That's Not Very Data-Driven of You
Even good engineering instincts are just opinions without data. On why teams struggle to justify decisions — and what to do about it.
There's a significant initiative going on with my team — rethinking how we generate network configurations, untangling a codebase that had grown organically over years into something increasingly painful to work with. The engineers behind it believe in it — their instincts say it's the right call. Maybe they're right, maybe they're not — but things were getting cleaner. You could feel it.
But then came the question that stops well-intentioned engineering work in its tracks:
"Can you show us the data?"
Ah. Right. That.
That's Not Very Data-Driven of You
Here's the thing about work like this — the value is real but diffuse. Things get simpler, more reliable, easier to build on. Future bugs that never happen don't show up in any dashboard. The toil that quietly disappears doesn't file a ticket when it leaves.
So when someone asked for data to back up the initiative, the team did what a lot of teams do these days: they asked an AI.
The AI, naturally, gave a confident, well-structured answer that supported the initiative. Metrics were cited. Improvements were highlighted. It read beautifully.
Here's the problem: that's not data. That's a very expensive yes-man.
LLMs are genuinely bad at this specific task for two reasons. First, they're susceptible to confirmation bias by framing — how you phrase the question dramatically changes the answer you get. A March 2026 study on LLM-assisted security code review ran Claude Code as an autonomous PR reviewer and found that simply framing a change as a security improvement got it to confirm that framing in 15 of 17 cases — 88% of the time — even when the change introduced a real, known vulnerability. Ask "why is this refactored code better?" and you'll get a thoughtful essay on why it's better. The AI isn't lying to you — it's just extremely agreeable.
Second, and more fundamentally: AI can simply be wrong. LLMs don't run your code, don't know your system's history, and have never seen your production incident timeline. They pattern-match and generate plausible prose. That's useful for a lot of things. Manufacturing engineering metrics from thin air is not one of them.
Wait, You Don't Have DORA? Or SPACE? At All?
This kicked off a conversation with another colleague. I asked if they were familiar with frameworks like DORA or SPACE — ways to systematically measure engineering effectiveness that have been around for years.
They hadn't.
At a large company. With a lot of engineers. With, presumably, a Jira board somewhere that says "improve engineering metrics" and has said so since 2022.
To be clear, this isn't a knock on them specifically — it's the default state of most orgs I've seen. Metrics infrastructure is one of those things everyone agrees is important in theory and almost no one actually builds in practice. But it does mean that when you need data to justify something, you're staring at a very empty spreadsheet.
Not entirely empty, though. There's usually a compliance dashboard somewhere, an uptime SLA, maybe a ticket-closure rate someone glances at on Fridays. Those numbers say things got done. What they don't say is how many hoops someone had to jump through to get there — how many approvals, how many Slack threads, how many times a senior engineer had to intervene because the "official" process didn't actually work. Getting things done and getting things done sanely are not the same metric, and most orgs only track the first one.
So — what are these frameworks?
DORA (DevOps Research and Assessment) measures four things: how often you ship to production (Deployment Frequency), how long from commit to production (Lead Time), what percentage of deployments cause incidents (Change Failure Rate), and how quickly you recover when things go wrong (Time to Restore Service). What's interesting is that the research — drawn from surveys of tens of thousands of teams — shows that the best teams aren't trading speed for stability. They're achieving both. Fast and reliable. Annoying, but true.
SPACE is broader. It covers Satisfaction & Wellbeing, Performance, Activity, Communication & Collaboration, and Efficiency & Flow. The core insight is that no single metric can capture developer productivity — and that teams who try to reduce it to one number (commit count, lines of code, story points) will inevitably measure the wrong thing and optimise towards it.
For code quality specifically, there are more targeted tools. Static analysis can track things like cyclomatic complexity (a proxy for "how hard is this to test and reason about?"), duplication, or test coverage automatically on every PR. You get a before/after number for the change. It's not the whole story, but it's a story, and it's objective.
Which brings me to something that surprised me even more than not knowing DORA or SPACE: none of this tooling was part of their default pipeline either. Metrics that have been around for decades, surfaceable automatically on every PR with tools that are either free or already paid for — just... not there. That's not an edge case. That's a gap. And it's exactly the gap that bites you when you need to justify something.
I've Tried to Build This Before (It Did Not Go Great)
I say all this with full empathy, because I tried to build a proper metrics baseline at a previous company and it was a journey.
The data was all over the place. Some metrics lived in one tool, some in another, some nowhere at all. And even where it existed, nobody agreed on what it meant. Is pushing a config change to one region a "deployment," or is that reserved for an actual feature release? Does "done" mean merged, or shipped, or verified in production a week later with no rollback — and does that answer change depending on which team you ask, because it did, every time. One team counted an incident as anything that paged someone at 2am; another only counted it if a customer noticed and complained. Multiply that by every team in the org and you don't have a metrics problem, you have a translation problem. Everyone had opinions about what should be measured, and those opinions did not agree. Getting alignment on the basics was itself a project that needed a roadmap.
Here's the thing nobody tells you: building engineering metrics is a product problem. It needs instrumentation, ownership, clear definitions, tooling, and — maybe most critically — organisational buy-in. You can't bolt it on after the fact. It has to be designed in, which is awkward because it also doesn't ship any features and is therefore easy to deprioritise forever.
And yet.
The Question I Try to Ask at the Start
Every significant project I've worked on, I've tried to ask myself early: how will I know if this worked?
Not as a bureaucratic exercise. More as a sanity check. Because if you can't answer that question before you start, you're building something you'll never be able to properly evaluate — and you'll end up exactly where my colleagues were: trying to reverse-engineer a justification after the fact, with vibes and AI-generated prose as your primary sources.
A baseline doesn't need to be complicated. For an initiative like this, it might look like:
- Code quality metrics (complexity, duplication) for the affected modules, before and after
- Time for a new engineer to make their first change in this area (interviews, onboarding notes)
- Incidents linked to this component over the past 6 months versus the next 6
- Test coverage on the affected paths
- PR review and merge time for changes touching this area
None of those are perfect. All of them beat "trust us, it's cleaner now."
The work might be 100% worth doing — it probably is. But "probably is" is hard to fund, and even harder to learn from. If you don't measure it, you don't know what worked, and you can't replicate it.
Talk to Your Customers (Internal or Not)
Yes, this is just "talk to your customers." Yes, it applies to internal engineering work too.
Before you write a single line, go talk to the people who actually live in the codebase — the engineers who regularly touch that area, the ones who onboard new starters, the ones who get paged at 2am. Ask them what's painful. Listen to them complain. Collect the complaints.
What you'll get is a catalogue of pain in plain English. Things like: "every time I need to change a config template I have to trace through four different files", or "the test suite takes 20 minutes and still misses regressions", or "we had three incidents last quarter that all came back to logic being duplicated in two places." Write it down. All of it.
Because these are your before state.
Suddenly the initiative has a brief. It's not "the code is cleaner" (nobody outside the team cares about that on its own) — it's "engineers were tracing through four files to make a single change, and that's where the copy-paste bugs kept coming from; now it's one file and the incidents stop." Clean code isn't the point — fewer incidents, faster feature delivery, and a codebase new hires can actually work in are the point. If cleaner code gets you there, great — measure that instead. That's a before and after. That's a story a stakeholder can follow without squinting.
And this is exactly why the baseline matters. If you document the pain before you start and measure against it after, you have something real. Did cycle time for changes in this area go down? Are there fewer incidents? Is onboarding faster? The pain points become the success criteria, and the baseline is how you prove they were fixed — not just moved somewhere less visible.
This is also when the actual roadshow happens — not before any of this, but after. Once you have the pain points, the baseline, and ideally some early results, that's what you take on tour to the rest of the org. Nobody wants to hear a pitch for an idea. They'll listen to a pitch backed by a before-and-after.
So What Do You Actually Do?
A few things that help, even mid-project:
Document pain points explicitly, ideally before you write a single line. Interview the people in the blast radius. Get specific. "This is slow" is not a pain point. "Adding a new device type requires changes in five separate places, takes a senior engineer half a day, and has caused three incidents from copy-paste errors" is a pain point. Save the roadshow for once you have a baseline and some results to show — a pitch backed by data lands very differently than a pitch backed by a hunch.
Build a baseline, even retrospectively. Most code-quality metrics are computable from any point in your git history. Incident rates live in your ticketing system. PR review times are in your version control platform. If you didn't measure before you started, you can often reconstruct a "before" — it just takes more work.
Pick two or three metrics and commit to them. SPACE recommends spanning at least three of its five dimensions rather than optimising for one. For a code quality initiative, something like a code-quality score + time-to-onboard + incident rate is probably enough. Don't try to measure everything. Measure the things that would convince a reasonable sceptic sitting across a table from you.
Use AI to interpret data, not generate it. It's genuinely good at helping you understand what metrics mean, suggesting what to track, or summarising trends in data you've already collected. That's the right job for it.
And — this one matters — treat metrics infrastructure as real work. It doesn't ship features, which means it's always one deprioritisation away from never happening. But a team that can't measure its own health is flying blind, and eventually that shows in ways that are much harder to explain.
The instinct here is probably right — the work probably is making things better. That might even be reason enough to do it, in a world where trust in your engineers is high and timelines are generous.
But an instinct is still just an opinion. The next time someone asks for data, it'd be nice to have collected it yourself — rather than asking an AI to invent it for you.