Who rates the raters?

There is a question that quietly decides whether you can trust a rating, and almost no rating answers it. Who rates the rater?

We answered this question properly once, for one industry, and it took a few disasters to learn. After enough companies collapsed while holding clean audit opinions, the world built an entire apparatus around a single idea: the people checking the books cannot be paid by, owned by, or otherwise tangled up with the people whose books they check. Independence requirements. Rotation rules. A prohibition on auditing a company you also consult for. None of it assumes auditors are dishonest people. All of it assumes that incentives, left alone, bend judgment, and that the only reliable fix is structural separation, not good character.

Then, in market after market, we forgot the lesson and rebuilt the conflict from scratch.

The pattern is always the same. A market gets confusing enough that people need a guide. Someone builds a guide. The guide gets popular. And then the guide discovers that the straightforward way to make money is to charge the very companies it is supposed to judge — for placement, for partnership, for a "premium profile," for the privilege of being ranked favourably. At that point the guide stops being a guide and becomes a marketplace for favourable judgment, while keeping the appearance of a guide. Nobody had to decide to become dishonest. The incentive did the work slowly, and then the result looked normal.

I know this most directly from gambling, where I have spent 16 years, including building affiliate and rating businesses myself. Most consumer "ratings" of betting and casino operators are produced by affiliates who earn commissions from the operators they rank. The grader is on the operator's payroll, paid more when the player they sent loses more. The outcome is exactly what you would predict: rankings that follow commercial terms more closely than they follow how players are actually treated, with a confident number on top to make it look objective. It is not a few bad actors. It is the default state of any rating that can be monetised by the thing it rates, and I have been part of that default.

But this is not really a gambling story. It is a structural one, and it shows up anywhere ratings became a business. Wherever a "rating" can be monetised by the thing being rated, the rating drifts toward whoever pays. The specifics change from market to market. The shape does not.

I want to be careful, because the care is the point. I am not accusing any particular company of acting in bad faith, and you should be suspicious of anyone who does, because intent is exactly the thing you cannot prove and do not need. The argument does not require a villain. It requires only ownership and incentive, both of which are usually a matter of public record. When the business producing a ranking also profits from the things being ranked, you do not have to allege a conspiracy. You only have to point at the structure and ask the auditor's question: where is the independence here? Most of the time there is no answer, because there is no independence.

The standard defence is disclosure. "We disclose our relationships." Disclosure does not fix a conflict of interest. It only records one. A note at the bottom of the page does nothing against the incentive sitting at the top of it. Finance learned this the expensive way: knowing the agency was paid by the issuer did not make the rating sound. It only meant the failure was announced in advance.

So what does fix it? The same thing that fixed it for auditors. Structure, not promises.

A rating earns trust when the channel through which money could move the number has been physically removed, not merely disclosed. That means the rating cannot depend on revenue from the things it rates. It means the method is public and inspectable, so anyone can check whether the score follows the evidence or the money. It means any advertising that funds operations is labelled and kept structurally separate from the judgment. And it means the rating is visibly willing to say something inconvenient about the most prominent operator in its market, because a rating that never bites the hand that could feed it has already told you who it works for.

I am building toward exactly this, and I will not pretend to be a neutral observer. I still run affiliate businesses, so I have a direct interest in this argument landing. But the test I am proposing applies to me too, and I want it to. Ask of my work the same thing I am asking of everyone else's: who rates the rater, and what stops the money from moving the number? If I cannot answer that with structure instead of a promise, do not trust me either.

That is the whole point, and it is also a tool you can use today, on any rating you read. Do not go looking for a more trustworthy rater. Ask who pays them, and whether anything but their word stops that payment from reaching the number. If the only answer is "trust us," you already have your answer.