[Skip to Content]
[Skip to Content Landing]
Table 1.  Baseline Characteristics of SNFs, 2014-2018
Baseline Characteristics of SNFs, 2014-2018
Table 2.  Risk-Standardized 30-Day Hospital Readmission Rates From SNFa
Risk-Standardized 30-Day Hospital Readmission Rates From SNFa
Table 3.  Common Themes From Reviews of SNFs With High vs Low Rehospitalization Rates
Common Themes From Reviews of SNFs With High vs Low Rehospitalization Rates
1.
Medicare Payment Advisory Commission. Report to Congress. Chapter 8: skilled nursing facility services. Published March 2019. Accessed February 26, 2020. http://www.medpac.gov/docs/default-source/reports/mar19_medpac_ch8_sec.pdf?sfvrsn=0
2.
Neuman  MD, Wirtalla  C, Werner  RM.  Association between skilled nursing facility quality indicators and hospital readmissions.   JAMA. 2014;312(15):1542-1551. doi:10.1001/jama.2014.13513 PubMedGoogle Scholar
3.
Jencks  SF, Williams  MV, Coleman  EA.  Rehospitalizations among patients in the Medicare fee-for-service program.   N Engl J Med. 2009;360(14):1418-1428. doi:10.1056/NEJMsa0803563 PubMedGoogle Scholar
4.
Kleinpell  RM, Fletcher  K, Jennings  BM. Reducing functional decline in hospitalized elderly. In: Hughes  RG, ed.  Patient Safety and Quality: An Evidence-Based Handbook for Nurses. Agency for Healthcare Research and Quality; 2008.
5.
Candlish  P, Watts  P, Redman  S, Whyte  P, Lowe  J.  Elderly patients with heart failure: a study of satisfaction with care and quality of life.   Int J Qual Health Care. 1998;10(2):141-146. doi:10.1093/intqhc/10.2.141 PubMedGoogle Scholar
6.
Zuckerman  RB, Sheingold  SH, Epstein  AM.  The Hospital Readmissions Reduction program.   N Engl J Med. 2016;375(5):494.PubMedGoogle Scholar
7.
Rahman  M, McHugh  J, Gozalo  PL, Ackerly  DC, Mor  V.  The contribution of skilled nursing facilities to hospitals’ readmission rate.   Health Serv Res. 2017;52(2):656-675. doi:10.1111/1475-6773.12507 PubMedGoogle Scholar
8.
Ryskina  KL, Konetzka  RT, Werner  RM.  Association between 5-star nursing home report card ratings and potentially preventable hospitalizations.   Inquiry. 2018;55:46958018787323. doi:10.1177/0046958018787323PubMedGoogle Scholar
9.
Cornell  PY, Grabowski  DC, Norton  EC, Rahman  M.  Do report cards predict future quality? the case of skilled nursing facilities.   J Health Econ. 2019;66:208-221. doi:10.1016/j.jhealeco.2019.05.008 PubMedGoogle Scholar
10.
Arrington  M. Yelp throws down on CitySearch. Accessed September 12, 2019. http://social.techcrunch.com/2008/11/19/yelp-throws-down-on-citysearch/
11.
Stoppelman  J. Yelp's consumer protection initiative: ProPublica partnership brings medical info to Yelp. Accessed September 12, 2019. https://www.yelpblog.com/2015/08/yelps-consumer-protection-initiative-propublica-partnership-brings-medical-info-to-yelp
12.
Merchant  RM, Volpp  KG, Asch  DA.  Learning by listening-improving health care in the era of Yelp.   JAMA. 2016;316(23):2483-2484. doi:10.1001/jama.2016.16754 PubMedGoogle Scholar
13.
Werner  RM, Konetzka  RT, Kruse  GB.  Impact of public reporting on unreported quality of care.   Health Serv Res. 2009;44(2 Pt 1):379-398. doi:10.1111/j.1475-6773.2008.00915.x PubMedGoogle Scholar
14.
Konetzka  RT, Polsky  D, Werner  RM.  Shipping out instead of shaping up: rehospitalization from nursing homes as an unintended effect of public reporting.   J Health Econ. 2013;32(2):341-352. doi:10.1016/j.jhealeco.2012.11.008 PubMedGoogle Scholar
15.
Werner  RM, Konetzka  RT, Stuart  EA, Polsky  D.  Changes in patient sorting to nursing homes under public reporting: improved patient matching or provider gaming?   Health Serv Res. 2011;46(2):555-571. doi:10.1111/j.1475-6773.2010.01205.x PubMedGoogle Scholar
16.
Verhoef  LM, Van de Belt  TH, Engelen  LJ, Schoonhoven  L, Kool  RB.  Social media and rating sites as tools to understanding quality of care: a scoping review.   J Med internet Res. 2014;16(2):e56. doi:10.2196/jmir.3024 PubMedGoogle Scholar
17.
Jacobsen  JML, Schnelle  JF, Saraf  AA,  et al.  Preventability of hospital readmissions from skilled nursing facilities: a consumer perspective.   Gerontologist. 2017;57(6):1123-1132.PubMedGoogle Scholar
18.
von Elm  E, Altman  DG, Egger  M, Pocock  SJ, Gøtzsche  PC, Vandenbroucke  JP; STROBE Initiative.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.   Int J Surg. 2014;12(12):1495-1499. doi:10.1016/j.ijsu.2014.07.013 PubMedGoogle Scholar
19.
O’Brien  BC, Harris  IB, Beckman  TJ, Reed  DA, Cook  DA.  Standards for reporting qualitative research: a synthesis of recommendations.   Acad Med. 2014;89(9):1245-1251. doi:10.1097/ACM.0000000000000388 PubMedGoogle Scholar
20.
Yelp Inc. Recommended reviews. Accessed February 26, 2020. https://www.yelp-support.com/Recommended_Reviews?l=en_US
21.
Centers for Medicare and Medicaid Services. Design for Nursing Home Compare five-star quality rating system: Technical users' guide. Accessed February 26, 2020. https://www.cms.gov/Medicare/Provider-Enrollment-and-Certification/CertificationandComplianc/downloads/usersguide.pdf
22.
Abt Associates. Nursing Home Compare Claims-based Quality Measure Technical Specifications. Accessed September 16, 2019. https://www.cms.gov/Medicare/Provider-Enrollment-and-Certification/CertificationandComplianc/Downloads/Nursing-Home-Compare-Claims-based-Measures-Technical-Specifications.pdf
23.
Hoel  PG.  Introduction to Mathematical Statistics. 5th ed. Wiley; 1984.
24.
White  H.  A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity.   Econometrica. 1980;48:817-830. doi:10.2307/1912934 Google Scholar
25.
Blei  DM, Ng  AY, Jordan  MI.  Latent Dirichlet allocation.   J Mach Learn Res. 2003;3(4-5):993-1022. doi:10.1162/jmlr.2003.3.4-5.993Google Scholar
26.
Mor  V, Intrator  O, Feng  Z, Grabowski  DC.  The revolving door of rehospitalization from skilled nursing facilities.   Health Aff (Millwood). 2010;29(1):57-64. doi:10.1377/hlthaff.2009.0629 PubMedGoogle Scholar
27.
Thornblade  LW, Arbabi  S, Flum  DR, Qiu  Q, Fawcett  VJ, Davidson  GH.  Facility-level factors and outcomes after skilled nursing facility admission for trauma and surgical patients.   J Am Med Dir Assoc. 2018;19(1):70-76.e1. doi:10.1016/j.jamda.2017.08.007PubMedGoogle Scholar
28.
You  K, Li  Y, Intrator  O,  et al.  Do nursing home chain size and proprietary status affect experiences with care?   Med Care. 2016;54(3):229-234. doi:10.1097/MLR.0000000000000479 PubMedGoogle Scholar
29.
Intrator  O, Castle  NG, Mor  V.  Facility characteristics associated with hospitalization of nursing home residents: results of a national study.   Med Care. 1999;37(3):228-237. doi:10.1097/00005650-199903000-00003 PubMedGoogle Scholar
30.
Rahman  M, Norton  EC, Grabowski  DC.  Do hospital-owned skilled nursing facilities provide better post-acute care quality?   J Health Econ. 2016;50:36-46. doi:10.1016/j.jhealeco.2016.08.004 PubMedGoogle Scholar
31.
Palinkas  LA, Horwitz  SM, Green  CA, Wisdom  JP, Duan  N, Hoagwood  K.  Purposeful sampling for qualitative data collection and analysis in mixed method implementation research.   Adm Policy Ment Health. 2015;42(5):533-544. doi:10.1007/s10488-013-0528-y PubMedGoogle Scholar
32.
Schwartz  HAGS, Sap  M, Crutchley  P, Eichstaedt  JC, Ungar  L. DLATK: Differential Language Analysis ToolKit. Proceedings of the 2017 EMNLP System Demonstrations. 2017:55-60.
33.
Zhu  JM, Patel  V, Shea  JA, Neuman  MD, Werner  RM.  Hospitals using bundled payment report reducing skilled nursing facility use and improving care integration.   Health Aff (Millwood). 2018;37(8):1282-1289. doi:10.1377/hlthaff.2018.0257 PubMedGoogle Scholar
34.
Ranard  BL, Werner  RM, Antanavicius  T,  et al.  Yelp reviews of hospital care can supplement and inform traditional surveys of the patient experience of care.   Health Aff (Millwood). 2016;35(4):697-705. doi:10.1377/hlthaff.2015.1030 PubMedGoogle Scholar
35.
Kellogg  C, Zhu  Y, Cardenas  V,  et al.  What consumers say about nursing homes in online reviews.   Gerontologist. 2018;58(4):e273-e280. doi:10.1093/geront/gny025 PubMedGoogle Scholar
36.
Williams  A, Straker  JK, Applebaum  R.  The nursing home five star rating: how does it compare to resident and family views of care?   Gerontologist. 2016;56(2):234-242. doi:10.1093/geront/gnu043 PubMedGoogle Scholar
37.
Shugarman  LR, Brown  JA. RAND Corporation. Nursing home selection: findings from focus groups of consumers and information intermediaries. Accessed February 26, 2020. https://aspe.hhs.gov/basic-report/nursing-home-selection-how-do-consumers-choose-volume-i-findings-focus-groups-consumers-and-information-intermediaries
38.
Schapira  MM, Shea  JA, Duey  KA, Kleiman  C, Werner  RM.  The Nursing Home Compare report card: perceptions of residents and caregivers regarding quality ratings and nursing home choice.   Health Serv Res. 2016;51(2)(suppl 2):1212-1228. doi:10.1111/1475-6773.12458 PubMedGoogle Scholar
39.
Konetzka  RT, Perraillon  MC.  Use of Nursing Home Compare website appears limited by lack of awareness and initial mistrust of the data.   Health Aff (Millwood). 2016;35(4):706-713. doi:10.1377/hlthaff.2015.1377 PubMedGoogle Scholar
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    Health Policy
    May 14, 2020

    消费者对专业护理机构的网上评价与患者再住院率的关联性

    Author Affiliations
    • 1Perelman School of Medicine, Division of General Internal Medicine, Department of Medicine, University of Pennsylvania, Philadelphia
    • 2Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia
    • 3Center for Digital Health, University of Pennsylvania Health System, Philadelphia
    • 4University of Pennsylvania, Philadelphia
    • 5University of Pennsylvania School of Nursing, Philadelphia
    • 6Corporal Michael J. Crescenz VA Medical Center, Philadelphia, Pennsylvania
    • 7Perelman School of Medicine, Department of Emergency Medicine, Department of Medicine, University of Pennsylvania, Philadelphia
    JAMA Netw Open. 2020;3(5):e204682. doi:10.1001/jamanetworkopen.2020.4682
    关键点 español English

    问题  在评论网站上发表的专业护理机构评价是否与患者再住院率有关?

    结果  这项横断面研究涉及 1536 个专业护理机构。研究发现,2 个评论网站上评分最低的专业护理机构的再入院率要比评分最高的机构低 2.0%。定性意见反映了消费者在其他方面对专业护理机构体验的看法,(例如,物理基础设施和设备的质量、工作人员的态度以及与护理人员的沟通等)。

    意义  这些调查结果表明,主动提供的在线评价可能会提供重要的信息,为选择专业护理机构提供依据。

    Abstract

    Importance  There are areas of skilled nursing facility (SNF) experience of importance to the public that are not currently included in public reporting initiatives on SNF quality. Whether patients, hospitals, and payers can leverage the information available from unsolicited online reviews to reduce avoidable rehospitalizations from SNFs is unknown.

    Objectives  To assess the association between rehospitalization rates and online ratings of SNFs; to compare the association of rehospitalization with ratings from a review website vs Medicare Nursing Home Compare (NHC) ratings; and to identify specific topics consistently reported in reviews of SNFs with the highest vs lowest rehospitalization rates using natural language processing.

    Design, Setting, and Participants  A retrospective cross-sectional study of 1536 SNFs with online reviews on Yelp (a website that allows consumers to rate and review businesses and services, scored on a 1- to 5-star rating scale, with 1 star indicating the lowest rating and 5 stars indicating the highest rating) posted between January 1, 2014, and December 31, 2018. The combined data set included 1536 SNFs with 8548 online reviews, NHC ratings, and readmission rates.

    Main Outcomes and Measures  A mean rating from the review website was calculated through the end of each year. Risk-standardized rehospitalization rates were obtained from NHC. Linear regression was used to measure the association between the rehospitalization rate of a SNF and the online ratings. Natural language processing was used to identify topics associated with reviews of SNFs in the top and bottom quintiles of rehospitalization rates.

    Results  The 1536 SNFs in the sample had a median of 6 reviews (interquartile range, 3-13 reviews), with a mean (SD) review website rating of 2.7 (1.1). The SNFs with the highest rating on both the review website and NHC had 2.0% lower rehospitalization rates compared with the SNFs with the lowest rating on both websites (21.3%; 95% CI, 20.7%-21.8%; vs 23.3%; 95% CI, 22.7%-24.0%; P = .04). Compared with the NHC ratings alone, review website ratings were associated with an additional 0.4% of the variation in rehospitalization rates across SNFs (adjusted R2 = 0.009 vs adjusted R2 = 0.013; P = .003). Thematic analysis of qualitative comments on the review website for SNFs with high vs low rehospitalization rates identified several areas of importance to the reviewers, such as the quality of physical infrastructure and equipment, staff attitudes and communication with caregivers.

    Conclusions and Relevance  Skilled nursing facilities with the best rating on both a review website and NHC had slightly lower rehospitalization rates than SNFs with the best rating on NHC alone. However, there was marked variation in the volume of reviews, and many SNF characteristics were underrepresented. Further refinement of the review process is warranted.

    Introduction

    More than 40% of patients who require postacute care after hospitalization are discharged to skilled nursing facilities (SNFs), which provide short-term skilled nursing or rehabilitation services after hospital discharge.1 One in 4 patients discharged to SNFs for postacute care are rehospitalized within a month.2 Rehospitalizations represent a particularly important outcome in this patient population that has been shown to estimate future hospital use,3 have negative associations with function4 and quality of life.5 Hospitals have been subjected to financial pressure to reduce readmissions after facing penalties from Medicare for excessive readmission rates.6 In contrast with other postacute care settings, rehospitalizations from SNFs represent an opportune target for hospitals given the hospitals’ ability to influence the choice of SNF. Rehospitalizations from SNFs play a substantial role in the discharging hospital’s readmission rate,7 suggesting that hospitals could reduce readmissions by diverting their discharges to SNFs with low readmission rates.

    One challenge hospitals and patients face in selecting a SNF is the limited information available on SNF quality and outcomes. Current publicly available ratings of SNFs are not associated with hospital readmissions. Evaluations of the Medicare Nursing Home Compare (NHC) ratings found that a patient’s likelihood of rehospitalization differed by less than 1 percentage point between SNFs with the lowest vs highest star rating.2,8 Furthermore, NHC ratings could be used to match more patients with clinically complex conditions to higher-quality SNFs.9 Whether higher rehospitalization rates represent an influx of patients who are sicker or worse care quality at a SNF is not observable to consumers. Considering the need for more detailed information on SNF care quality, we sought to assess the association between SNF rehospitalization rates and SNF ratings reported by a large online platform (Yelp, which allows consumers to rate and review businesses and services on a 1- to 5-star rating scale, with 1 star indicating the lowest rating and 5 stars indicating the highest rating).

    Unsolicited health-related reviews have been posted on this review website for more than a decade,10 with a substantial increase in the past 5 years.11 Two advantages of the review website are the short lag time between review submission and posting and the ability to comment on aspects of SNF care that reviewers find salient.12 In contrast, NHC ratings are reported with a considerable time lag. Furthermore, HNC does not include a mechanism to relay open-ended information that may be relevant to consumers. The measures are vetted for facilities to know what they are measured on, but this opens possibility for gaming the measures, teaching to the test, and negative spillover on unmeasured aspects of quality.13-15 Disadvantages of unsolicited online reviews include a lack of transparency regarding the reviewers’ experience with the facility (ie, timing and relationship to the facility), unmeasured selection bias (ie, who posts a review), and intrinsically subjective nature of the ratings.12,16

    We conducted this national study to evaluate reviews of SNFs on the review website as a potential source of information about SNF quality. We hypothesized that excess rehospitalizations from SNFs may be associated with aspects of quality covered in online reviews. For instance, caregiver perceptions of care may be particularly important in rehospitalizations from SNFs, given that some avoidable rehospitalizations are associated with caregiver demands.17 Because the review website reviews are available within days of submission, the review website ratings may more closely track SNF care quality over time. For these reasons, we hypothesized that the review website ratings of SNFs are associated with rehospitalization rates. Our objectives were to: (1) assess the association between rehospitalization rates and the review website ratings of SNFs; (2) compare how well the review website vs NHC ratings explain the variation in rehospitalizations from the SNFs; and (3) identify specific topics consistently reported in reviews of SNFs with the highest vs lowest rehospitalization rates using natural language processing.

    Methods

    The study was deemed exempt from review by the University of Pennsylvania Institutional Review Board as not meeting the regulatory definition of human research. The reporting of this study follows the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for cross-sectional studies,18 and the Standards for Reporting Qualitative Research (SRQR) guideline for qualitative studies.19

    Data and Sample

    The review website publishes free-text comments submitted online by consumers as well as a cumulative average 5-point rating of all submissions. There is no expiration time frame for older reviews, but more recent reviews appear first on screen. The comments are unedited. The review website requires firsthand experiences from contributors, using proprietary algorithms to highlight reviews that “best reflect the opinions of the Yelp community” and exclude “fraudulent submissions”.11 We used Yelp in this study because it provides fraud detection mechanisms that allowed us to remove reviews that were likely fraudulent.20

    To merge the review website ratings to NHC data, we manually extracted all 16 191 reviews in the nursing home category from the review website, consistent with the terms of service of the platform. The reviews represented 2467 unique business names and addresses and were submitted between August 27, 2005, and January 17, 2019. We collected all available rehospitalization rates on NHC, which were measured between the first quarter of 2014 and last quarter of 2018. There were 14 787 SNFs with available overall rating and rehospitalization rates in NHC. These data were linked to the Medicare Provider of Services files containing SNF characteristics (eg, number of beds and ownership type).

    The review website facilities were matched to NHC identifiers via 2 rounds of matching. First, NHC name and the review website business name were considered matched if there was an exact string match between the names and zip codes. Second, the review website facilities not matched in the first round were assigned to an unmatched NHC facility located in the same zip code if the names varied by at most 3 characters. All facilities matched in the second round of matching were checked manually by one of us (K.L.R.) using a prespecified protocol. A match was considered acceptable if facility names differed by an article (ie, a/an/the), space (eg, “Willowbrook” vs “Willow Brook”), abbreviation (“Rehabilitation & Healthcare” vs “Rehabilitation and healthcare”), or generic business designation (eg, “LLC” or “SNF”). Otherwise, a match was considered ambiguous and was deleted (eg, “St Mary” vs “St Martha”). The combined data set included 1536 SNFs with 8548 review website reviews, NHC ratings, and readmission rates measured between January 1, 2014, and December 31, 2018 (10.4% of SNFs with rehospitalization rates and ratings on NHC) (eFigure in the Supplement).

    Variables

    The NHC reports an annual overall 5-star rating calculated based on an algorithm that weighs health inspections and staffing relative to quality measures.21 In 2016, NHC began to report measures of hospital readmissions as part of the quality measures. We used the risk-standardized 30-day rehospitalization rate for short-term SNF stays calculated using the NHC Quality Measures specifications.22 The rate is adjusted for patient demographics, clinical conditions, and functional status using Medicare claims and SNF clinical assessments. The risk-standardized rehospitalization rates are measured over a 12-month period and updated quarterly (lag time of 9 months).22 We used the rates measured over the 12-month period from January to December of each year in the study. To match to the corresponding NHC measurement interval for rehospitalization rates, we calculated a mean review website rating through the end of each calendar year of the study.

    Statistical Analysis

    The analyses were conducted between January 1 and September 30, 2019. We compared SNF characteristics in our sample with the characteristics of all NHC SNFs during the study period using 1 sample test of proportion and unpaired t test for means.23 P values less than .05 were considered statistically significant; all hypothesis tests were 2-sided.

    Analyses were performed at the SNF-year level. To measure the association between rehospitalization rates, NHC overall rating, and review website rating, we estimated 3 sets of models. First, we estimated the risk-standardized rehospitalization rate as a function of the overall NHC star rating. Second, the same models were estimated with the review website ratings instead of NHC ratings. Third, the models were estimated with both the review website rating and NHC rating. Linear probability models were used to estimate the rates. The difference in the R2 statistic between the first, second, and third set of models, adjusted for the number of independent variables in each model, was used to measure the proportion of the variance in readmission rates associated with the review website variables. To do so, we first calculated the adjusted R2 statistic for each model, which measures the proportion of variance associated with variables in the model. Next, we subtracted the adjusted R2 for the NHC only model from the R2 for the combined NHC and review website ratings model. The difference represents the proportion of variance associated with review website ratings, in addition to the proportion of variance associated with the NHC ratings alone. The Huber-White sandwich estimator was used in all regressions to account for clustering of observations within SNFs.24 Statistical analyses were performed using Stata, version 15 (StataCorp LP).

    To generate topics (word clouds), which consist of clusters of co-occurring words, we extracted the following features from each review: the relative frequency of single words and phrases with at most 3 words, and the distribution of 20 latent Dirichlet allocation (LDA) topics associated with the reviews for SNFs in the top vs bottom quintiles of rehospitalization rates. LDA is a statistical generative model commonly used in natural language processing that identifies topics in text based on word co-occurrence.25

    We performed a number of additional analyses. First, we re-estimated the rehospitalization models on a subsample of SNFs with at least 3 reviews. Second, we re-estimated the models including SNF characteristics that were selected a priori because they were previously found to be associated with SNF quality (region,26 rural vs urban location,27 size (<100, 100-199, or 200+ beds),2 ownership (for-profit vs nonprofit),28 presence of advanced practitioners,29 whether the facility was part of a chain,28 and whether it was hospital based30). Third, we performed qualitative thematic analyses of the reviews for SNFs in the top vs bottom quintiles of rehospitalization rates and triangulated the results with the LDA generated word clouds.31 Two hundred reviews were randomly selected and independently reviewed by 2 trained qualitative researchers with prior clinical experience working in SNFs (K.M. and K.F.). We used an inductive approach to identify common themes across the reviews. First, the 2 qualitative researchers independently reviewed all comments and coded recurrent topics and ideas. Next, the researchers conducted a series of group discussions to go over the topics their identified. Through these iterative discussions, a consensus list of most salient topics was developed. Lastly, the researchers triangulated the findings with the content of the word clouds. We used the DLATK package for the LDA analysis and the word clouds.32

    Results
    Study Sample

    The characteristics of SNFs in the main sample, in the subsample of SNFs with at least 3 reviews, and all SNFs with rehospitalization rates reported during the study period are shown in Table 1. The SNFs in the main sample had a median of 6 reviews (interquartile range 3-13), with a mean (SD) review website rating of 2.7 (1.1). The mean (SD) rehospitalization rate was 22.5% (5.1). Compared with all SNFs on NHC, SNFs located in the West, in urban areas, large in size, and with for-profit ownership were over-represented in the main study sample. The subsample of 1011 SNFs with at least 3 reviews was similar to the main sample (Table 1).

    Review Website Ratings, NHC Ratings, and Rehospitalization Rates

    Of the 3614 SNF ratings during 2014-2018, over a third (34.3%) of NHC ratings were 5-star and 7.6% were 1-star (Table 2). In contrast, 12.1% of review website ratings were in the highest (>4.4) and 22.1% were in the lowest category (<1.5). The 2 sets of ratings matched (eg, 1-star rating on NHC and <1.5 points on the review website) in 20.7% of cases.

    SNFs with the highest NHC rating (5-star) had 1.6% lower rehospitalization rate compared with SNFs with the lowest rating (1-star) (21.8%; 95% CI, 21.5% to 22.1% for 5-star vs 23.4%; 95% CI, 22.8% to 23.9% for 1-star; P < .001) (Table 2). SNFs with the highest review website rating (≥4.5 out of 5) had 0.9% lower rehospitalization rate compared with SNFs with the lowest review website rating (<1.5 out of 5) (21.9%; 95% CI, 21.4% to 22.4% for the review website rating >4.4 stars vs 22.8%; 95% CI, 22.4% to 23.2% for the review website rating <1.5 stars; P = .008). Facilities with the best rating on both the review website and NHC had 2.0% lower rehospitalization rates compared with the SNFs with the worst rating on both websites (21.3%; 95% CI, 20.7% to 21.8% for SNFs with the highest ratings on both websites vs 23.3%; 95% CI, 22.7% to 24.0% for SNFs with the lowest ratings on both websites, P = .04).

    Assessment of Review Website vs NHC Ratings

    NHC ratings alone and review website ratings alone were associated with approximately between 0.7% and 0.9% of variance across rehospitalization rates from SNFs (adjusted R2 = 0.009 for the NHC only model and adjusted R2 = 0.007 for the review website only model). Both the review website and NHC ratings together increased the proportion of variance in rehospitalization rates from SNFs by 0.4% from 0.9% to 1.3% (adjusted R2 = 0.009 vs adjusted R2 = 0.013; P = .003) (Table 2).

    We identified 5 topics representing differences in reviews between SNFs with high vs low rehospitalization rates: environment (eg, facility atmosphere and experiences in the physical space), staff attitudes and behaviors (eg, affect, professionalism, and caring behaviors), management (eg, billing, customer service, and organizational culture), communication (eg, effectiveness of transferring information among staff and caregivers), and care quality (eg, delivery of safe care, managing pain, and performing activities of daily living) (Table 3).

    The findings for the subsample of SNFs with at least 3 reviews were generally consistent with the main results (eTable 1 in the Supplement). The association between online ratings and rehospitalization rates were not sensitive to adjustment for SNF characteristics (eTable 2 in the Supplement).

    Discussion

    Online consumer ratings of SNFs published on the review website provided novel information associated with a small albeit significant proportion of variation in SNF rehospitalization rates. The use of the review website and NHC ratings together identified SNFs with slightly lower rehospitalization rates compared with NHC ratings alone. The review website reviews identified several areas of importance to the consumers, such as the quality of physical infrastructure and equipment, staff attitudes, and communication with caregivers.

    In a recent survey, hospitals reported that forming preferred provider networks of SNFs in their area represented a leading strategy for adjusting to alternative purchasing reforms in postacute care.33 Little information is available to date about how hospitals select SNFs into their network. Since 2016, NHC reported rehospitalization rates as part of its SNF quality measures, but the rates are reported with considerable lag time. Hospitals may be able to track internal readmission rates in a more timely manner, but those rates would not reflect readmissions to other hospitals. Our findings suggest that online reviews could inform the selection process by providing additional information to help identify SNFs with the lowest readmission rates. However, absolute differences in readmission rates between the highest vs lowest-rated SNFs were small, and we did not observe significant differences in readmissions between SNFs in the middle categories.

    Consistent with prior studies,34-36 several topics regularly mentioned in the review website reviews are not currently covered by NHC, including staff attitudes and communication as well as physical environment (cleanliness and comfort). Specifically, reviewers commented about caring, helpful, and professional staff and fresh food, nice meals, and private rooms for SNFs with low readmission rates. For SNFs with high readmission rates, reviewers commented about waiting for call back, lack of choice and autonomy, and poor administration and billing. NHC may consider incorporating systematic measures of these aspects of patient and caregiver experience into SNF ratings.

    Our findings also highlight previously noted concerns with online information about health care facilities. We were unable to obtain demographic information about the review website reviewers or the nature of their experience with the facilities (eg, timing and duration of stay, relationship to the patient). Sugarman et al37 found that online information about SNFs is not well accessible (eg, written in college-level language), and does not contain information of interest to likely consumers (eg, quality of meals). Recent evaluations of consumer experience with NHC reported limited awareness of the website and suspicion about the quality of information available.38,39 Whether unsolicited online reviews are associated with consumer decisions or their likelihood to be satisfied with their SNF choice depends on the success of efforts to address these concerns, such as generating ratings in way that is more representative, protects privacy, and remains timely.

    Limitations

    This study has several limitations. First, only 10% of the SNFs had any reviews and 34% of those had fewer than 3 reviews. However, the findings from the subsample of SNFs with 3 or more reviews were consistent with the main results. Second, we were not able to distinguish between appropriate and excessive rehospitalizations, or rehospitalizations specifically because of poor quality of care in the SNF. For example, higher readmission rates to a hospital from a SNF may be due to the hospital discharging patients prematurely and may not reflect the quality of care in the SNF. Third, the SNFs with the review website reviews were not representative of all US SNFs. For example, almost all the SNFs in the sample were located in urban areas and half were located in the west. Our findings suggest that certain biases persist in online reviews (eg, computer access in urban vs rural areas, age). Efforts to address these concerns could start with improving transparency of fraud detection algorithms and sharing characteristics of online reviewers.

    Conclusions

    To our knowledge, this is the largest evaluation of unsolicited online reviews of SNFs to date. Consistent with prior work, we identified several domains of SNF experience that are not currently covered by NHC. Furthermore, SNFs with the best rating on both the review website and NHC had slightly lower rehospitalization rates than SNFs with the best rating on NHC alone. However, there was marked variation in the volume of reviews and many SNF characteristics were under-represented. These findings suggest that, if key limitations are addressed, unsolicited online reviews could provide salient information to inform SNF selection.

    Back to top
    Article Information

    Accepted for Publication: March 1, 2020.

    Published: May 14, 2020. doi:10.1001/jamanetworkopen.2020.4682

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2020 Ryskina KL et al. JAMA Network Open.

    Corresponding Author: Kira L. Ryskina, MD, MSHP, Perelman School of Medicine, Division of General Internal Medicine, Department of Medicine, University of Pennsylvania, 423 Guardian Dr, 12-30 Blockley Hall, Philadelphia, PA 19104 (ryskina@pennmedicine.upenn.edu).

    Author Contributions: Dr Ryskina had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Ryskina, Merchant.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: Ryskina, Andy, Manges, Merchant.

    Critical revision of the manuscript for important intellectual content: Ryskina, Manges, Foley, Werner.

    Statistical analysis: Ryskina, Andy, Merchant.

    Administrative, technical, or material support: Ryskina, Manges.

    Supervision: Ryskina, Foley, Werner, Merchant.

    Conflict of Interest Disclosures: Dr Werner reported receiving grants from National Institute of Aging during the conduct of the study; and personal fees from CarePort Health outside the submitted work. Dr Merchant reported receiving grants from the National Heart, Lung, and Blood Institute during the conduct of the study. No other disclosures were reported.

    Funding/Support: This study was supported by the National Institutes on Aging (NIA) Career Development Award (K08-AG052572) (Dr Ryskina), by the Agency for Healthcare Research and Quality (T32HS026116-02) (Dr Manges), by the NIA (K24-AG047908) (Dr Werner), and by the National Heart, Lung, and Blood Institute (R01-HL141844-01A1) (Dr Merchant).

    Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the Agency for Healthcare Research and Quality.

    References
    1.
    Medicare Payment Advisory Commission. Report to Congress. Chapter 8: skilled nursing facility services. Published March 2019. Accessed February 26, 2020. http://www.medpac.gov/docs/default-source/reports/mar19_medpac_ch8_sec.pdf?sfvrsn=0
    2.
    Neuman  MD, Wirtalla  C, Werner  RM.  Association between skilled nursing facility quality indicators and hospital readmissions.   JAMA. 2014;312(15):1542-1551. doi:10.1001/jama.2014.13513 PubMedGoogle Scholar
    3.
    Jencks  SF, Williams  MV, Coleman  EA.  Rehospitalizations among patients in the Medicare fee-for-service program.   N Engl J Med. 2009;360(14):1418-1428. doi:10.1056/NEJMsa0803563 PubMedGoogle Scholar
    4.
    Kleinpell  RM, Fletcher  K, Jennings  BM. Reducing functional decline in hospitalized elderly. In: Hughes  RG, ed.  Patient Safety and Quality: An Evidence-Based Handbook for Nurses. Agency for Healthcare Research and Quality; 2008.
    5.
    Candlish  P, Watts  P, Redman  S, Whyte  P, Lowe  J.  Elderly patients with heart failure: a study of satisfaction with care and quality of life.   Int J Qual Health Care. 1998;10(2):141-146. doi:10.1093/intqhc/10.2.141 PubMedGoogle Scholar
    6.
    Zuckerman  RB, Sheingold  SH, Epstein  AM.  The Hospital Readmissions Reduction program.   N Engl J Med. 2016;375(5):494.PubMedGoogle Scholar
    7.
    Rahman  M, McHugh  J, Gozalo  PL, Ackerly  DC, Mor  V.  The contribution of skilled nursing facilities to hospitals’ readmission rate.   Health Serv Res. 2017;52(2):656-675. doi:10.1111/1475-6773.12507 PubMedGoogle Scholar
    8.
    Ryskina  KL, Konetzka  RT, Werner  RM.  Association between 5-star nursing home report card ratings and potentially preventable hospitalizations.   Inquiry. 2018;55:46958018787323. doi:10.1177/0046958018787323PubMedGoogle Scholar
    9.
    Cornell  PY, Grabowski  DC, Norton  EC, Rahman  M.  Do report cards predict future quality? the case of skilled nursing facilities.   J Health Econ. 2019;66:208-221. doi:10.1016/j.jhealeco.2019.05.008 PubMedGoogle Scholar
    10.
    Arrington  M. Yelp throws down on CitySearch. Accessed September 12, 2019. http://social.techcrunch.com/2008/11/19/yelp-throws-down-on-citysearch/
    11.
    Stoppelman  J. Yelp's consumer protection initiative: ProPublica partnership brings medical info to Yelp. Accessed September 12, 2019. https://www.yelpblog.com/2015/08/yelps-consumer-protection-initiative-propublica-partnership-brings-medical-info-to-yelp
    12.
    Merchant  RM, Volpp  KG, Asch  DA.  Learning by listening-improving health care in the era of Yelp.   JAMA. 2016;316(23):2483-2484. doi:10.1001/jama.2016.16754 PubMedGoogle Scholar
    13.
    Werner  RM, Konetzka  RT, Kruse  GB.  Impact of public reporting on unreported quality of care.   Health Serv Res. 2009;44(2 Pt 1):379-398. doi:10.1111/j.1475-6773.2008.00915.x PubMedGoogle Scholar
    14.
    Konetzka  RT, Polsky  D, Werner  RM.  Shipping out instead of shaping up: rehospitalization from nursing homes as an unintended effect of public reporting.   J Health Econ. 2013;32(2):341-352. doi:10.1016/j.jhealeco.2012.11.008 PubMedGoogle Scholar
    15.
    Werner  RM, Konetzka  RT, Stuart  EA, Polsky  D.  Changes in patient sorting to nursing homes under public reporting: improved patient matching or provider gaming?   Health Serv Res. 2011;46(2):555-571. doi:10.1111/j.1475-6773.2010.01205.x PubMedGoogle Scholar
    16.
    Verhoef  LM, Van de Belt  TH, Engelen  LJ, Schoonhoven  L, Kool  RB.  Social media and rating sites as tools to understanding quality of care: a scoping review.   J Med internet Res. 2014;16(2):e56. doi:10.2196/jmir.3024 PubMedGoogle Scholar
    17.
    Jacobsen  JML, Schnelle  JF, Saraf  AA,  et al.  Preventability of hospital readmissions from skilled nursing facilities: a consumer perspective.   Gerontologist. 2017;57(6):1123-1132.PubMedGoogle Scholar
    18.
    von Elm  E, Altman  DG, Egger  M, Pocock  SJ, Gøtzsche  PC, Vandenbroucke  JP; STROBE Initiative.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.   Int J Surg. 2014;12(12):1495-1499. doi:10.1016/j.ijsu.2014.07.013 PubMedGoogle Scholar
    19.
    O’Brien  BC, Harris  IB, Beckman  TJ, Reed  DA, Cook  DA.  Standards for reporting qualitative research: a synthesis of recommendations.   Acad Med. 2014;89(9):1245-1251. doi:10.1097/ACM.0000000000000388 PubMedGoogle Scholar
    20.
    Yelp Inc. Recommended reviews. Accessed February 26, 2020. https://www.yelp-support.com/Recommended_Reviews?l=en_US
    21.
    Centers for Medicare and Medicaid Services. Design for Nursing Home Compare five-star quality rating system: Technical users' guide. Accessed February 26, 2020. https://www.cms.gov/Medicare/Provider-Enrollment-and-Certification/CertificationandComplianc/downloads/usersguide.pdf
    22.
    Abt Associates. Nursing Home Compare Claims-based Quality Measure Technical Specifications. Accessed September 16, 2019. https://www.cms.gov/Medicare/Provider-Enrollment-and-Certification/CertificationandComplianc/Downloads/Nursing-Home-Compare-Claims-based-Measures-Technical-Specifications.pdf
    23.
    Hoel  PG.  Introduction to Mathematical Statistics. 5th ed. Wiley; 1984.
    24.
    White  H.  A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity.   Econometrica. 1980;48:817-830. doi:10.2307/1912934 Google Scholar
    25.
    Blei  DM, Ng  AY, Jordan  MI.  Latent Dirichlet allocation.   J Mach Learn Res. 2003;3(4-5):993-1022. doi:10.1162/jmlr.2003.3.4-5.993Google Scholar
    26.
    Mor  V, Intrator  O, Feng  Z, Grabowski  DC.  The revolving door of rehospitalization from skilled nursing facilities.   Health Aff (Millwood). 2010;29(1):57-64. doi:10.1377/hlthaff.2009.0629 PubMedGoogle Scholar
    27.
    Thornblade  LW, Arbabi  S, Flum  DR, Qiu  Q, Fawcett  VJ, Davidson  GH.  Facility-level factors and outcomes after skilled nursing facility admission for trauma and surgical patients.   J Am Med Dir Assoc. 2018;19(1):70-76.e1. doi:10.1016/j.jamda.2017.08.007PubMedGoogle Scholar
    28.
    You  K, Li  Y, Intrator  O,  et al.  Do nursing home chain size and proprietary status affect experiences with care?   Med Care. 2016;54(3):229-234. doi:10.1097/MLR.0000000000000479 PubMedGoogle Scholar
    29.
    Intrator  O, Castle  NG, Mor  V.  Facility characteristics associated with hospitalization of nursing home residents: results of a national study.   Med Care. 1999;37(3):228-237. doi:10.1097/00005650-199903000-00003 PubMedGoogle Scholar
    30.
    Rahman  M, Norton  EC, Grabowski  DC.  Do hospital-owned skilled nursing facilities provide better post-acute care quality?   J Health Econ. 2016;50:36-46. doi:10.1016/j.jhealeco.2016.08.004 PubMedGoogle Scholar
    31.
    Palinkas  LA, Horwitz  SM, Green  CA, Wisdom  JP, Duan  N, Hoagwood  K.  Purposeful sampling for qualitative data collection and analysis in mixed method implementation research.   Adm Policy Ment Health. 2015;42(5):533-544. doi:10.1007/s10488-013-0528-y PubMedGoogle Scholar
    32.
    Schwartz  HAGS, Sap  M, Crutchley  P, Eichstaedt  JC, Ungar  L. DLATK: Differential Language Analysis ToolKit. Proceedings of the 2017 EMNLP System Demonstrations. 2017:55-60.
    33.
    Zhu  JM, Patel  V, Shea  JA, Neuman  MD, Werner  RM.  Hospitals using bundled payment report reducing skilled nursing facility use and improving care integration.   Health Aff (Millwood). 2018;37(8):1282-1289. doi:10.1377/hlthaff.2018.0257 PubMedGoogle Scholar
    34.
    Ranard  BL, Werner  RM, Antanavicius  T,  et al.  Yelp reviews of hospital care can supplement and inform traditional surveys of the patient experience of care.   Health Aff (Millwood). 2016;35(4):697-705. doi:10.1377/hlthaff.2015.1030 PubMedGoogle Scholar
    35.
    Kellogg  C, Zhu  Y, Cardenas  V,  et al.  What consumers say about nursing homes in online reviews.   Gerontologist. 2018;58(4):e273-e280. doi:10.1093/geront/gny025 PubMedGoogle Scholar
    36.
    Williams  A, Straker  JK, Applebaum  R.  The nursing home five star rating: how does it compare to resident and family views of care?   Gerontologist. 2016;56(2):234-242. doi:10.1093/geront/gnu043 PubMedGoogle Scholar
    37.
    Shugarman  LR, Brown  JA. RAND Corporation. Nursing home selection: findings from focus groups of consumers and information intermediaries. Accessed February 26, 2020. https://aspe.hhs.gov/basic-report/nursing-home-selection-how-do-consumers-choose-volume-i-findings-focus-groups-consumers-and-information-intermediaries
    38.
    Schapira  MM, Shea  JA, Duey  KA, Kleiman  C, Werner  RM.  The Nursing Home Compare report card: perceptions of residents and caregivers regarding quality ratings and nursing home choice.   Health Serv Res. 2016;51(2)(suppl 2):1212-1228. doi:10.1111/1475-6773.12458 PubMedGoogle Scholar
    39.
    Konetzka  RT, Perraillon  MC.  Use of Nursing Home Compare website appears limited by lack of awareness and initial mistrust of the data.   Health Aff (Millwood). 2016;35(4):706-713. doi:10.1377/hlthaff.2015.1377 PubMedGoogle Scholar
    ×