This January, U.S. News & World Report released their 2018 Best Online Program Rankings. The publication has become an authority on college rankings around the world, and they offer some pretty detailed reports in numerous categories. Guidance counselors, students, and parents of future undergrads have come to rely on their rankings to discern which colleges they should apply to and, when they receive their acceptance letters in the spring, which program to attend.

But despite the trust placed in these reports, it seems a good wager to bet that most who rely on them have no idea how they calculate their college rankings. The truth of the matter is that most, if not each, report goes through a unique process to transform raw data into informative statistics. It would be madness to compile each of those inconsistencies in one post. But as an example of the nuanced—and at times, surprising—ways that U.S. News & World Report compiles their data, we’re going to look at how they generated their Best Online Bachelor’s Degree rankings.

The Data

To begin, U.S. News & World Report relies on institutions voluntarily reporting information in dozens if not hundreds of different categories along with a voluntary peer reputation survey. Some might raise an eyebrow at this data source, but the fact of the matter is that no one holds a candle to the publication’s reputation. Schools that fail to report their data face an all-but-certain low ranking.

With this raw data, U.S. News divides it into four categories, each of which receive a different weight. The categories and resulting weights are as follows:

• Student engagement – 35%: Engagement is one of the greatest challenges with which online programs still struggle. It’s no wonder it is rated so highly.
• Student services and technology – 25%: Following engagement, student support services are notoriously lacking with online degrees. Some research shows that support services can be as important as a strong academic program for graduation rates.
• Faculty credentials and training – 20%: This is a no-brainer for any prestigious program—online or in-person.
• Peer reputation – 20%: Peer reputation comes from voluntary surveys sent to “high-ranking academic officials” and seeks to account for “intangible factors affecting program quality that are not captured by statistics,” according to U.S. News.

Each of these main categories are made up of individual subcategories, each of which have their own unique means of collection and calculation. Digging down to this level, the casual observer might be surprised for various reasons. To highlight this, we’ll look at two main categories: Student engagement and Student services and technology.

The Fine Print of College Rankings

Student engagement is far and away the most complex category. It is broken into five sub-categories: Best practices, Graduation rates, Class size, Assessments, and Retention rates. Each of these is weighted differently and comprised of still more sub-categories. For example, “Best practices” is comprised of “An index based on 16 equally weighted factors” which range from protections against plagiarism to the systems for ensuring student participation to the institution’s observance of the Americans with Disabilities Act. In other words, it incorporates factors that many wouldn’t even consider when researching an online bachelor’s degree, and that goes a long way to lending some peace of mind.

But Student services and technology kind of disrupts that trust. It is based on three unequally rated subcategories: Technological infrastructure, Support services, and Student indebtedness. Can you guess which one received the highest weight? Student indebtedness makes up 50% of this category while the other two sub-categories (the two that happen to be mentioned in the category title) get only 25% a piece. How does that even work?

To be clear: each of these categories should receive a huge amount of consideration in every college rankings list. Technological infrastructure is about student access to 10 equally weighted technologies. Student support services forms an index of student access to 10 equally weighted support services from help with IT to mentoring and career placement. Each of these is probably more important to a given student than an institution’s policy on plagiarism.

Here’s the takeaway: U.S. News & World Report is the most respected authority when it comes to college rankings. But they likely have different priorities in mind from a given future student. Before you take their word, check out how they calculated their findings. We didn’t even begin to touch on their peer reporting. The information on their methodology for a given ranking can be found on their site. The information related above can be found here.