At the end of 2013 Professor Sir Bruce Keogh asked for an independent view from the Academy on behalf of the medical profession on how they see the data on UK and USA hospital death rates based on HSMRs produced by Professor Sir Brian Jarman.
The Academy brought together a distinguished group of clinicians from Colleges with an
understanding of these issues along with expert advisors to consider and discuss the data.
Professor Jarman was engaged throughout the process and his input sought although the report is the product of the Academy group alone.
The full report of the working group is attached as Annex A and the membership of the group as Annex B.
The question asked of the Academy was whether HSMR can be relied upon to make comparisons between hospitals in the NHS and between hospitals in other countries.
Following the detailed consideration and discussion it was the clear view that HSMR in its own right is subject to a number of limitations.
Furthermore, without the full data set that Sir Brian Jarman has utilised in his analysis being subject to independent peer review to see how much these variables affect the HSMR in making comparisons, it cannot be concluded that the quality of care in an American hospital is superior to that of an NHS hospital.
Differences in US and UK healthcare
We identified a number of differences that represent considerable challenges to compare US hospital and UK Hospital mortality rates. They are summarised here but a more detailed analysis is covered in the full report.
— Differences in diagnostic practices: use of lower risk diagnoses in England patients are
less likely to be assigned high risk in England as opposed to the USA. The most striking
example is the use of ‘septicaemia’ as the primary diagnosis: in England it is 4.5 per
10 000 admissions but in the USA it is about 8-fold higher at 35 per 10000.
This trend in the USA is thought to be driven by reimbursement considerations where
there are greater financial incentives to place patients in a higher risk which would skew
the HSMR as lower in the USA.
The impact of differences in diagnostic practice will have a dramatic effect on the HSMR
— Difference in number of secondary diagnoses: fewer recorded in England. There are
again discrepancies in the recording of secondary diagnoses. There has always been
a serious under recording of secondary diagnoses (co-morbidity) in HES in England.
In contrast, in the USA where there are financial incentives to include co-morbidities
in the hospital information system, a third of admissions have 10 or more recorded.
As a result, patients in England appear to be at much lower risk of death than those in
the USA which again will skew the results towards a worse HSMR in the UK.
Differences in severity: patients in England may be sicker. Comparisons based upon
administrative data (HES and NIS) are not able to provide any information on the severity
of patients’ primary condition. But there is some evidence that on admission patients
in England are more severely ill than in the USA and thus at higher risk of death.
— Proportion of population deaths in acute hospitals: The proportion of patients who
die in acute hospitals is higher in England and Wales (56%) than in the USA (45%).
This matters because higher proportions of deaths occurring in hospital are known to
be associated, albeit weakly, with higher HSMRs.10
Choice of statistical methods
In addition to the aforementioned differences in the data used to compare England and the USA,
further uncertainty arises from the choice of statistical methods used for estimating HSMRs from
these data. The relative HSMRs for England and for the USA will depend on the risk adjustment
model used. There are four concerns:
— Choice of factors included in the risk adjustment model.
— Choice of data used to derive the model
— Choice of hospitals included from each country
— Choice of assuming same level of underlying risk in both countries
These are explained in detail in the full report.
How might valid comparisons be made?
That is not to say that HSMR cannot ever be useful. Where there are outliers and particularly high mortality rates in hospital it could serve as a useful tool for further investigation and evaluation i.e. HSMR should be used as a “smoke detector” to generate a signal, not as the definitive answer.
There are also a number of other areas that were brought to the attention of the group from the existing literature on HSMR during the review which merit consideration moving forward. There is little correlation between how well a hospital performs on one standard of safe effective care and how well it performs on another. This is reviewed in a 2010 BMJ analysis paper “differences in quality of care within hospitals themselves are much greater than differences between hospitals”.1
Hospitals are more likely to fail on specifics – pathology in Liverpool, paediatric cardiac surgery in Bristol for example. 2,3
Sir Brian has cooperated fully with the review in responding to all the questions posed. Although he agrees that numerous variables will influence his HSMR calculations, he feels this will be of a small magnitude that wouldn’t make a major difference to his conclusions. There are two limitations to this. The first is that only Sir Brian Jarman has access to the raw data which he is obliged to keep confidential which hinders independent scrutiny. We recognise that he has signed confidentiality agreements and are not expecting him to breach those. The second is that there is already in existence extensive literature that highlights that HSMR is only one marker of health care quality and does not have primacy.
To move forward, we would recommend that studies are commissioned to establish whether there are systematic differences in the quality of hospital care between the two countries. Perhaps the focus should shift towards outcomes (other than hospital mortality) and clinical processes. This would require detailed comparisons of matched sets of patients with the same primary diagnosis in both the UK (or England) and the USA (or another first world comparator country). This would be more useful to NHS England than broad HSMR comparisons because of the methodological concerns set out in more detail in the appendix.
Comparison of HSMRs for England and USA
On 11 September 2013, Professor Sir Brian Jarman stated in the media that the likelihood of dying in NHS hospitals in England was higher than in hospitals in the USA. Specifically, he stated that risk-adjusted mortality was 45% higher in 2012. Based on the data he made publicly available, augmented by methodological information he provided in response to our requests, we have carefully considered the scientific basis of his claims.
The comparisons he made were based on routine hospital data publicly available in each country: in England the Hospital Episode Statistics (HES)1 and for the USA the Nationwide Inpatient Sample (NIS)2 – a 20% stratified sample of community hospitals (short-term non-federal hospitals, accounting for 97% of all acute admissions in the USA).
In a meeting with Mike Campbell and Terence Stephenson, Sir Brian Jarman acknowledged the problem arising from his confidentiality agreements which precluded scrutiny of the data and peer review.
We have considered two key aspects of the comparisons: the accuracy and comparability of the two datasets; and the choice of statistical method (the risk adjustment model).