The ratings game: New research by MIT Sloan School of Management’s Cynthia Rudin provides tools for companies to reverse-engineer product quality rankings

Findings have implications for how and where firms make R&D investments

CAMBRIDGE, Mass., February 2, 2012—A good or bad quality rating can make or break a product. The success of a new flat screen TV depends on its scores in CNET and PCMag; the reputation of a mutual fund relies on its Morningstar rating; and mortgages hinge on the models of credit rating agencies such as Experion, Equifax, and TransUnion. But considering most independent rating companies rank product quality using a secret formula, how can companies and organizations better compete?

The answer, according to new research* by MIT Sloan School of Management’s Cynthia Rudin, is by reverse-engineering that secret formula. “Product quality rankings are important for companies because they directly impact sales—particularly if a company’s products are in the top tier. It’s understandable. For consumers, sometimes rankings are really the best way to decide what to buy: you can't just buy all the products and compare them, so you might as well trust someone who has,” she says.

“The problem is that many rating companies—even the big ones—don’t make their ranking system completely transparent, so consumers and companies don’t have a clear sense of how quality is defined. By reverse-engineering the formula used for product rankings, organizations can better understand how their products are being judged. This will help them design better products, and, by extension, give consumers access to those products.”

Combining ten year’s worth of data from a major rating company with information about the way these ranking models are commonly built, Rudin and her colleagues—Allison Chang, a graduate student at MIT’s Operations Research Center, and Michael Cavaretta, Robert Thomas, and Gloria Chou, executives at Ford Motor Company—designed a mathematical algorithm to reconstruct the formulas used in quality ratings.

Rudin says companies that reverse-engineer quality rankings can easily determine the most efficient ways to spend their research and development budgets. Take, for instance, a camera maker aiming to get its newest digital model into the top four of a particular rating in the most cost-effective way.

“There are a bunch of possible modifications you could make to your camera—put in a larger battery, for example, add megapixels, or give it a better lens—and of course each of these changes costs a certain amount of money,” she explains. “After you’ve reverse-engineered the rating system with the first algorithm, another algorithm finds a combination of these changes that will get you into the top four at the lowest possible cost. Or, let's say you only have a certain budget to make changes to the camera. We can find a combination of these changes that would give you the highest rank possible, staying within your budget.”

Rudin says the algorithms may motivate rating companies to become more transparent, and to re-evaluate their formulas to ensure their rankings fairly represent quality.

“The goal is to change the power structure of the ratings game,” she says. “Many companies are, in some sense, at the mercy of these rankings because it isn’t entirely clear how their products are being evaluated. But hopefully the tools we’ve designed can empower them to achieve better rankings for their products. The bigger goal is to encourage rating companies to make their formulas more transparent, which will make high rankings truly represent high quality. That alone would help provide better products to customers.”

*The paper How to Reverse-Engineer Quality Rankings by Allison Chang, Cynthia Rudin, Michael Cavaretta, Robert Thomas, and Gloria Chou can be downloaded here: http://dspace.mit.edu/handle/1721.1/67838