What to Measure to Know Whether the Design Work Worked

The most common failure in a design engagement is that nobody can say afterwards whether it worked.

Not because the work was poor. Because the metric was never decided, so at the end there is nothing to compare against. What fills the vacuum is opinion. It feels better than before. Actually I preferred the old one. And then there is no second budget.

The metric has to be decided before the work starts, for a mechanical reason: you need the number from before. Once the work has begun, the before is gone.

Three metrics, minimum

Completion rate. Of the people who started the thing on that screen or in that flow, what share finished it. Application, purchase, sign up, enquiry, configuration. Whatever it is, the ratio of started to finished.

This is the one that carries the most weight. Almost all design work reduces to making fewer people give up partway. And it means something to whoever holds the budget.

Drop off by step. Completion rate alone tells you the number moved without telling you where. Count the steps separately. Three screens, three numbers.

This is not for measuring improvement. It is for deciding what to do next. If one of four steps loses 70% of people, the next budget has a destination and there is nothing left to argue about in the meeting.

Enquiry volume and content. How many calls and emails ask something the screen was supposed to answer.

This one converts into cost, which makes it the easiest number to get approved on. And a reduction in it is unambiguous.

Take the baseline first

The order matters more than the sophistication.

Measure the current state, over several weeks. Numbers move with the day of the week and the end of the month, so one week is not enough. Four weeks shows you the shape.

Then start the work.

Then measure again over the same window. Four weeks against four weeks. Two weeks against four proves nothing.

I have watched projects skip this and the ending is always identical. The people who built it say it worked, the people watching are unconvinced, and it stops there.

What not to measure

Internal reaction. Whether an executive liked it is not a metric. Make it one and the decisions drift towards the preferences of whoever speaks loudest. I have seen exactly that happen.

Time on page. Whether up is good or bad flips depending on the screen. Longer is better for an article and worse for a checkout. A metric with no fixed direction can be interpreted favourably after the fact, which is the same as having no metric.

Page views. Design work does not move this. It is an acquisition number. Mixing it in means acquisition activity gets credited to, or blamed on, the design.

Satisfaction surveys. Not useless, but not usable alone. The people who answer are biased towards people still using the product. The ones who gave up do not respond. The group you most want to hear from is structurally absent.

How to pick the window

I usually propose three months.

One month cannot separate a seasonal effect from a real one. Six months lets in so many other changes that attribution becomes guesswork: reorganisations, price changes, a new ad campaign. Three months usually means one or two other variables rather than ten.

Three months also matches the cycle on which most companies revisit a budget. If the numbers moved you continue, and if they did not you look at why. Either way there is a decision available.

When the numbers do not move

This happens. It has happened to me.

The first thing to check is whether the change actually reached anyone. Was the new screen served to everyone. Was any part of the old one still live. Was it broken on a particular device or browser.

A surprisingly large share of flat results are delivery failures rather than design failures, and they have completely different fixes.

If it did reach everyone and the number still did not move, the cause was somewhere else. That is also information. It narrows where to look next.

Honestly, that is not a good result, but it is far more valuable than the alternative. A project with no measurement cannot even have this conversation.

The parts that will not become numbers

Not everything does, and I would rather say so plainly.

Trust, brand impression, the fact that your own staff can now explain the product more easily. These are real effects and three months will not measure them.

How I handle it: before the engagement, separate what will be measured from what will not, in writing. Then do not claim the unmeasured half as a result afterwards.

Scope the work so that the measurable part alone justifies the cost. I think that is the honest structure, because reaching for the unmeasurable half is mostly what people do when the measurable half did not deliver.

Write one page before you commission anything

It does not need to be long.

Three metrics. The current number for each. The number you expect in three months. Who pulls the numbers.

If that page fills in, the engagement is much more likely to succeed. If a line will not fill in, that line is the actual thing to discuss.