Measuring the right thing

In digital marketing there is always a number on the screen. Reach, clicks, cost per click, conversion rate. After a while you start to see the world through them, and that is both the strength and the danger of the job.

The strength is clarity. A number makes a discussion concrete. The danger is that the number slowly replaces the thing it was supposed to represent. A post with high reach is not the same as a brand people trust. A low cost per click is not the same as a good customer.

Research has made this problem harder to ignore. Much of my work uses Data Envelopment Analysis, or DEA, a method for comparing the relative efficiency of units such as firms, branches or hospitals. Each unit uses several inputs to produce several outputs, and DEA builds a frontier from the best performers in the sample. Every other unit is measured by its distance from that frontier.

It is an elegant idea, and it has no opinion about what you feed it. If you choose inputs and outputs carelessly, the model will still give you a tidy ranking. It will simply be a ranking of the wrong thing. A firm can look efficient because it spends little on training, or inefficient because it invests in services that only pay off years later.

So the most important step happens before any calculation: deciding what efficiency should mean in this context, which resources really matter, and which results the firm is actually trying to produce. That is a conceptual question, not a technical one.

I find this oddly comforting. It means good measurement is not only about better software or bigger datasets. It is about thinking clearly, being honest about what a number can and cannot say, and writing those limits down where readers can see them.

The question I try to ask now, in research and anywhere else, is simple: if this number goes up, what has actually improved, and for whom?

---

Photos: nicolagiordano, danielkirsch on Pixabay, used under the Pixabay Content License. These are not my photographs.