How We Rate Vehicles

LOS ANGELES, CA — Coming to you from the car capital of the USA (sorry Detroit!), we are stoked to bring you our expert vehicle ratings, which use a 10-point scale and power our class-leading Rankings content. If close to 20 years in automotive journalism have taught us anything, it's that rating cars accurately and helpfully is a whole lot harder than it seems. Let's talk about what it takes to do it right.

What's a Rating? What are Rankings?!

Good questions. Let's actually back up a step and start there.

Ratings

The MD Rating consists of seven evaluation categories, each with its own score on a 10-point scale. To calculate the overall score, we take the average of those category scores. It's a weighted average, by the way, in two areas: We count Driving Performance twice, since driving is probably what you're going to spend most of your time doing, and we give the X Factor extra weight, too, because sometimes a car's standard evaluation fails to capture its true appeal. That overall score is what represents the vehicle on our Rankings pages.

Rankings

MotorDonkey's Rankings are simply grouped lists of vehicles ordered by overall score. We do have a tiebreaker feature behind the scenes that can add or subtract a hundredth of a point, in case we feel that one of the tied cars is clearly better than the other. But other than that, the score is the score, and we let the chips fall where they may. A key part of our process, though, is always scrutinizing the scores of rival vehicles in each rating category before rating the next vehicle. If you rate vehicles in a vacuum then you end up with chaos, as we'll see in this article a bit farther down.

Market Segments

In order to make Rankings useful, you have to establish the right competitive sets, or market segments, which put the ratings in context. Otherwise, you'll be asking the average car shopper to sift through a bunch of apples and oranges when all they wanted was a little help narrowing things down. Hardly any publication gets the segments right; what you typically see instead is a hodgepodge of sizes, configurations and capabilities within each list category. That's not helpful for shoppers who are trying to make good decisions.

At MotorDonkey, our segments reflect the reality of how automakers position rival models against each other. If you're not sold on a particular model, you'd probably like to know more about its direct competitors, and that's precisely what our segments give you.

The Rank-Order Laugh Test

Getting the segments sorted out is hard enough, but now the process needs to pump out Rankings that make sense. The point is to quantify how good each car is relative to its rivals—again, easier said than done.

Not to pick on a publication better known for college rankings, but it's impossible that the Acura Integra—a mildly dressed-up Honda Civic that is one of the slowest and least refined cars in its class—is the best small luxury car. Yet there it was in pole position as of January 24, 2024, lording it over luminaries like the BMW 3 Series (tied for third place) and the ancient but still highly competent Lexus IS (a very distant 14th). How did we get here? Well, 2023 was evidently a slow year over at NACTOY (North American Car, Utility and Truck of the Year) headquarters, because the Integra won Car of the Year...and the publication depicted above openly relies on other automotive outlets' verdicts when creating their own...so you can begin to understand the outcome. But it goes to show that it's easy to end up with some laughers if you don't have a rigorous process in place.

Indeed, this remains an occupational hazard even for true experts. For example, if you asked us right now to name our top midsize luxury SUVs, we'd give you the BMW X5 and the Lexus GX, for sure, but the Mercedes-Benz GLE has to be in that conversation. Why? Its engines are diverse and superb, its build quality is excellent, it tracks like a train on the highway, and it generally lets Mercedes' world-class engineering shine through. You'd be hard-pressed to find any disagreement on those points. It offers a wee third-row seat, too, which many rivals don't. And yet, here's where the generally excellent vehicle testers at Car and Driver rank it:

See? Even the most esteemed publication in the business isn't immune. There's an easy solution, though. Just ask the question: "Is the GLE really a worse midsize luxury SUV than the Genesis GV80 and the Lincoln Nautilus and the Land Rover Range Rover Sport and thirteen others? Is that what we want to be saying?" We're betting none of our friends at C/D would answer in the affirmative. How do you guard against this kind of outcome? You always look at the ranking order when you add a new rating, and if it doesn't make sense, you take a closer look and make adjustments if warranted. We think about this stuff all the time, which is honestly the only way you can hope to maintain accurate rankings in a perpetually evolving industry.

The "Do You Use AI?" Test

AI, artificial intelligence—perhaps you've heard of it. It's going to drive profits for publishers, you see, because machines can generate passable content instantaneously for a fraction of the human cost. That's true, but at what cost to the brand, right? It's disappointing, if not surprising, that so few people seem to be asking that question amid the rush to publish AI-produced content for short-term gain.

To state the hopefully obvious, humans don't seek out expert advice because they want a computer-generated guess; they're looking for other humans with genuine expertise who can guide them through difficult and expensive decisions. If a survey finds that people prefer AI-generated content to the real thing, it doesn't follow that they must want machines to replace humans—it just means that the experts need to do a better job of being helpful. There's no question that authentically human content is everyone's preference as long as it meets their needs. That's why we give you our word that no artificial intelligence is used to generate our content, period. If you see it on this site, you can rest assured that it came straight from the fingertips of a real live human with no AI influence whatsoever.

How many car sites can say the same? How many publications in general? Yet everyone knows that you don't build trust by passing off automated summaries as real advice; you build it by, you know, giving real advice! So why isn't there more resistance within the publishing world to the siren call of AI? At MotorDonkey, we eschew AI-generated content on principle. It's inherently inauthentic and untrustworthy, and we're not into that.

Who Do Ya Trust?

The fundamental problem here is that automotive content generally tends to get commodified and treated like widgets in a factory. Which widgets are the most profitable? How do we make more of those? It becomes less about producing the best content for car shoppers who are making some of the biggest decisions of their lives, and more about cranking out products that boost the bottom line.

Of course, there's a bottom line at MotorDonkey, too—we're not a nonprofit (though we tip our cap to Consumer Reports for remaining one). We need to make a living along the way. But we're committed to doing so by producing original, AI-free, data-driven expert content that makes car shopping easier for the 15 to 20 million Americans every year who need that help. That is how we rate cars: with uncompromising integrity and complete independence from external agendas and whims. We also have a lot of fun with it, and we hope you do, too. ⛐ md