Once again, healthcare costs for employers and employees are rising sharply, with no end in sight. Promising to stem the tide, health plans and third-party vendors offer a wide variety of programs that aim to decrease medical costs, supported by a multitude of proof points for their claims. Nonetheless, in a recent survey, only 7% of employers indicated that they look at results validated by a third party or published in a peer-reviewed journal when selecting a new vendor partner.
Here are considerations for HR and benefits professionals as they evaluate vendor claims of future cost savings.
1. Consult peer-reviewed articles in medical journals.
The gold standard to evaluate whether an intervention works—whether it is a new drug, medical device or medical management program—is research published in a peer-reviewed medical journal. Such articles are vetted by academic experts, and statistical methodology is likely to be rigorous. High-impact journals tend to be the most selective and most vigorous in their evaluation of submissions. They usually reject articles that have small numbers, short follow-up or that compare two groups that are not well-matched.
But medical journals are not perfect. Publication of medical journal articles often takes a year or more, and many companies with highly effective interventions don’t have the time or resources to shepherd their data through this review process. Medical journals prefer randomized controlled trials, but these are often impractical in the commercial world. Nimble vendors tweak programs all the time; the program that was evaluated in an article featured in a prominent medical journal might have morphed considerably since that article was written. Journals tend to publish papers that show positive impact, and authors are less likely to submit papers that show their intervention has failed.
Some vendors might offer to share data that they are preparing to publish but that is not yet accepted for publication. Prominent medical journals are unwilling to publish articles if the data has been widely circulated, so companies are limited in how much data they can share. Such data has not yet been evaluated by unbiased reviewers outside of the sponsoring organization, and the data provided could be selective, excluding populations where no benefit was found. Much data that is offered as “pre-publication data” never finds its way into peer-reviewed journals.
2. Use caution with independent, third-party white papers.
Some vendors will share their data with an independent third party that will publish a white paper demonstrating the impact of the intervention. Third parties can be less biased than data scientists employed directly by the vendor, although payment for their services can influence the findings through subconscious bias. Many third parties jealously guard their reputation for independence, which increases credibility of such reports. However, the third party can report only on the data shared. Be cautious that the data evaluated represents a reasonable cross-section of the vendor’s business and review the methodology section to evaluate whether the third party had access to all relevant information.
See also: Global healthcare benefits costs expected to rise sharply in 2023
3. Compare the methodology in published studies done by others.
Some vendors will present data published by others to demonstrate that their program is likely to be successful at lowering costs. For instance, there are a multitude of studies showing efficacy of diabetes-prevention programs, so some vendors cite earlier studies completed by others. Yet, how similar is the vendor’s intervention to the published work, and are the populations equivalent? A study of in-person therapy in Europe, for instance, should not be used as a “proof point” for a digital cognitive behavioral therapy intervention in the United States.
4. Scrutinize marketing claims.
Vendors collect and include information about the effectiveness of their programs in marketing slides, presentations, white papers and other collateral. The problem of bias is more pronounced with internal data presentations, which often restrict reporting to highly selected groups with better outcomes than the overall population. Internal marketing material often offers only highly selective and non-representative data and requires more thorough review.
HR and benefits professionals should ask the following questions when evaluating marketing data:
- Is the reporting on a substantial majority of the firm’s business? Vendors can sometimes mislead by carefully selecting which clients or which data to share. Small data sets are more likely to show success based on random variation alone.
- What is the duration of the intervention and the measurement? Some interventions that appear successful after a few months wane in effectiveness later.
- Is the vendor making fair and consistent comparisons of their outcomes against the market or against appropriate benchmarks?
* * *
Employers can feel more confident that a vendor will be able to have a substantial impact on their population by assessing the influence of these factors on vendor savings claims:
- Monitor for selection bias
Those who participate in a program are likely to be more engaged than those who refuse; they might also have higher educational levels, higher incomes and lower smoking rates. Propensity matching can help reduce such bias, but it is not possible to adjust for metrics such as motivation that are not measured. - Assess adequacy of sample size
Larger sample sizes are more credible, and exceptionally good (or bad) results in small sample sizes can be due to randomness rather than vendor performance. - Confirm how participation is defined
Attribute vendor success only for those programs that achieve meaningful interactions with members, and properly account for members who drop out after initiating the program. For example, evaluation of weight loss programs has historically been skewed because those who don’t lose weight drop out and their results are excluded. - Consider employee turnover
Many programs could lead to medical claims savings over multiple years, but this will be of less value to companies with high levels of employee turnover. - Evaluate the net financial impact of changes in services
Companies project savings based on their assessment of the value of averted services, but often overstate the cost of avoided care or omit substitution costs, including their own fees. - Review treatment of “catastrophic” claims
Claims above certain thresholds are often removed from study results because the vendors state that these claims are beyond their control and may skew the results of the analysis. However, vendors should have impact on some portion of these catastrophic claims. Regardless, high-cost claims should be treated similarly when the treatment is compared to other groups or benchmarks. - Appraise the full scope of metrics
To get a full picture, look across all key success metrics to see impacts on claims cost, quality and member experience. If a vendor has promised process measures like wait times, these should be included in reporting.
In some instances, aggressive performance guarantees can guard against disappointment if the data review does not definitively predict substantial cost savings. Performance guarantees should be methodologically clear and simple to reconcile, and employers should be wary of high built-in inflation rates.
Remember that not every valuable program lowers medical costs. Employers offer many programs to improve health, delight employees and help with recruitment and retention, regardless of impact on medical costs.
Employers face an imperative to carefully address rising medical costs, and interventions that lower total healthcare cost can lower premiums for the employer and lower cost-sharing for employees. The right solutions are ones that will deliver on their promises. Careful evaluation of the evidence offered by health plans and vendors will help employers have confidence in solutions that can help manage their medical costs.
