This study examines 23 prominent healthcare comparison websites that U.S. consumers, and other healthcare participants, can use to find and evaluate healthcare providers in terms of, for example, their quality, patient experiences, and cost. Our emphasis is on free website tools that offer side-by-side comparison of two or more providers, in a way that informs consumer choice but also provides value to other users within the large and complex landscape of healthcare.
Our goals in carrying out this empirical study are these:
- Inform potential users about the landscape of website tools, analogous to how each tool informs users about the healthcare landscape.
- Inform website-tool designers about the scope of existing features that could be offered by their tools, and perhaps prompt thoughts about non-existent features that should be invented and added.
- Data on healthcare providers contributes to transparency and helps drive improvement via several alternative pathways. Similarly, data on comparison websites enhances tool transparency and can drive improvement among comparison tools.
In science it is understood that when there exist numerous similar entities, it helps to overlay a structure based on identifying aspects on which they differ; such is our intent here. Often such structure leads further to devising categories that help make sense of the landscape, much as how medical science forms categories out of individual diseases.
Finally, readers should note that the author is affiliated with one of the tools evaluated here (benchmine.com) and welcomes commentary or corrective feedback on this study and its conclusions.
Also, the websites in this study were evaluated in February 2019 and may have added or removed features since then.
Methods
How the websites were selected
The original intent was to survey website tools that leverage the comprehensive healthcare datasets published by the Centers for Medicare & Medicaid Services (aka, CMS). We then opted to broaden the study upon finding fewer such public tools than expected, perhaps because the CMS data is somewhat complex for consumer audiences to comprehend and factor into their healthcare decisions. A 2016 National Academy of Medicine article pointed out that “Research has demonstrated that many of the current public reports make it cognitively burdensome for the audience to understand the data.”
To help compile a broader list, we browsed discussion forums, social media, and general web articles, and concluded with the 23 representative website tools show in the Figure below and listed in this Appendix. We may enlarge this list in the future.
How the features were selected and evaluated
The chosen website tools target a variety of healthcare sectors, such as hospitals, nursing homes, physicians, and others. We sought to analyze user-experience features that are not specific to a sector, but instead are common to the general task of comparing healthcare providers.
The websites address a diversity of audiences, roles, and tasks. The audiences include certainly consumers, patients, and families, but also providers themselves, patient referrers, government overseers, policy makers, journalists, performance improvement professionals, and others.
We followed this approach: Examine each website to figure out what it offers and how, noting the pros and cons of each offering. Then compile an overall list of notable features. Next, re-examine each site and check each feature’s availability, with details on its implementation.
The resulting 16 features are shown in the Figure below and are described in this Appendix.
How the website features were graded
Given the numerous websites and features, together with the varying details on each pair, we decided to present the results in the form of a visual heatmap, which implies that each website/feature grade needs a color. We used these five colors in the heatmap that appears later:
- GREEN: Very good
- Sites that offer the best implementation of the feature.
- YELLOW: Good
- Sites that offer some level of implementation of the feature, but were not the best.
- PURPLE: Average
- Sites that offer a very basic implementation of the feature.
- RED: Not featured
- Sites that didn’t offer a visible implementation of the feature.
- GRAY: Unknown or N/A
- Sites whose feature availability could not be determined, or was not applicable to its service.
Admittedly, judging whether a feature’s implementation is very good, good, or average, is somewhat subjective, but also transparently open to evaluation by readers themselves upon visiting the sites at issue.
Figure 1
Results
Fig 1 shows a heatmap of the comparative results of our empirical evaluation of 23 healthcare comparison websites. The rows (features) are ordered from the most frequently observed (17 green grades for the feature “enables access to the address of the provider being compared”) down to the least frequent (1 green for the feature “uses a single page to display all the available measures”). Users clearly have many options, depending on their roles and the tasks they seek to accomplish.
Several observations emerge from this heatmap.
- No single site dominates all the others.
- Comparison means different things. Some tools compare quality measures and other attributes. Other tools show what providers are nearby or within some geography.
- The use of fresh (i.e., up-to-date) CMS data is not common.
- Performing a deeper comparative assessment of an individual provider, in contrast to a side-by-side comparison of multiple providers, is also uncommon.
- Usually tools focus on either quality or cost factors, not both.
- Some features involve technology that is proprietary and isn’t broadly available, hence appears few times.
Conclusions
Comparative data on healthcare providers is made public in the service of performance transparency. Many website tools make use of such quality, cost, and reviews data, to both empower consumer choice as well as drive performance improvement by providers. Likewise, healthcare comparison tools can benefit from greater transparency, again to empower user choice of tools and drive improvement by their developers. We have performed an observational study of 23 free comparison tools evaluated on 16 feature dimensions. The results have been compiled into a heatmap to enable visual comparison of tools and evaluation of the frequency of feature availability.
– André Lessa, CTO and co-founder
Appendix
I. The complete list of features
Here are the 16 features that we evaluated at each website.
1. Uses CMS data
Does a site make use of CMS data? The possible values for this feature are simply YES (green) or NO (red).
2. Uses most recent CMS data
CMS updates its provider datasets every one or several months, so comparison sites should likewise update their data promptly. Whenever possible we evaluated how fresh was the site’s data and gave higher grades for freshness.
3. Handles multiple healthcare sectors
CMS publishes datasets for multiple healthcare sectors: hospitals, nursing homes, home health agencies, etc. We gave higher grades to sites that compare more sectors, although most examined sites handle just one sector, usually hospitals, physicians, or nursing homes. Other sectors have fewer or no evaluation options beyond Medicare.gov’s Compare sites.
4. Offers comparative insights instead of just lists
Most sites do compare providers side-by-side, but some sites only yield directory listings, which sometimes are not even ranked. Such sites offer a search-based front-end but with little or no comparative insights. We gave the best grades to sites that perform insightful peer comparison.
5. Does deeper comparisons of individual providers
Different audiences call for different types of comparison, with different reasoning algorithms underlying those types. A consumer may want to compare and contrast N providers in order to choose one, whereas a provider may want to comparatively assess just itself, also known as benchmarking. We graded sites based on whether they also perform a deeper comparative evaluation akin to benchmarking, and hence can serve a broader audience.
6. Compares many providers side-by-side
We agree with and followed the recommendations of this design article: “The basic comparison table uses columns for the products or services, and rows for the attributes. It allows for quick and easy comparison between each offering’s features and characteristics. […] Stick to the standard table layout: options as columns, attributes as rows, with row labels on the left and column labels above. Use consistent text alignment in each column.” A site that lets users compare more options on a single page reduces the cognitive burden of assembling insights from multiple pages. So we graded higher those sites that enable more than the usual maximum of three columns of provider options.
7. Easily differentiates between better and worse provider values
Ideally, comparisons should cue who are the best and worst among the selected providers instead of burdening users with figuring it out. For this feature we evaluated whether sites signal such comparative outcomes, e.g., via the use of icons or color highlighting. Sites with more cues get higher grades.
8. Offers options for hiding certain values, to reduce data overload
Many sites offer much detailed information about all available measures, which suits some users but not others. To accommodate diverse user preferences, sites can offer features that reduce data overload by the user’s choice. There are various reasons to hide measure comparisons: (1) all values are identical; (2) the values are not identical, but they differ by small amounts, which the user has the option to specify; (3) there is no objective preference for larger or smaller values of a measure (e.g., number of beds); and (4) they form a category of measures in which the user isn’t interested. The more options we observed for dealing with data overload, the higher we graded the site.
9. Uses a single page/tab to display all the available measures
Most sites force users to click on multiple tabs to explore all measures. We graded higher those cases where all measures were available on a single page or tab.
10. Uses graphs to better visualize the insights
Sometimes graphs can add value to textual insights so we graded higher the sites that offered some level of graph-based visualizations in their comparisons. The possible values for this feature are simply YES or NO.
11. Enables access to the address of the provider being compared
Many users value seeing a provider’s street address and contact info. We graded higher those sites that offer that information. The possible values for this feature are simply YES or NO.
12. Compares providers to their State and National averages
While benchmarking a U.S. healthcare provider, some sites perform comparisons to its respective State and National averages. We scored higher those sites that offer those comparisons. The possible values for this feature are simply YES or NO.
13. Uses advanced Search/Filter query options
Most sites offer some fields/attributes (e.g., CMS measures) that can be used to build search or filtering queries. Commonly, users can search on names and locations, but often features go little beyond that. For example, it’s rare to see sites leveraging the complete set of CMS measures available. Some sites go further by creating derived fields that users can leverage in their search queries. The more query/filter attributes that are supported by a site, the higher we graded it.
14. Includes computed, derived, or proprietary data or reviews
Some sites are driven by non-CMS data while others augment CMS data with their own data, such as reviews, proprietary data, derivations from CMS data, etc. So this feature goes beyond the directly sourced CMS measures. The possible values for this feature are simply YES or NO.
15. Offers Price/Cost information
Do sites include cost/price of healthcare procedures or services? The possible values for this valuable feature are simply YES or NO.
16. Uses maps to run search queries
Do sites let users employ maps to help formulate queries that specify the geography of matching providers? The possible values for this feature are simply YES or NO.
II. Selected tools
The healthcare-comparison websites evaluated for this study are these:
- aplaceformom.com
- benchmine.com
- carechex.com
- carepathways.com
- caring.com
- gohealthshare.com
- health.usnews.com
- healthcarebluebook.com
- healthcost.com
- healthgrades.com
- healthinsight.org
- hospitalcostcompare.com
- hospitalinspections.org
- leapfroggroup.org
- medicare.gov
- mpirica.com
- newchoicehealth.com
- propublica.org
- qcor.cms.gov
- qualitycheck.org
- rand.org
- senioradvisor.com
- whynotthebest.org
OnlyBoth Blog 
this was amazing to read! I’d love to ask you a few questions about this.