Outlier policy: SSNAP stroke mortality reporting

1. Purpose

This document outlines the identification, communication and management of outlying organisations (both stroke teams and ICBs/LHBs) in the SSNAP 30-day casemix adjusted mortality funnel based on annual financial year data. This policy is based on the HQIP guidance “HQIP outlier guidance”, HQIP, [3 January 2024]”.

Mortality is reported annually as to report it more frequently does not yield high quality data. The next outlier notification round will cover those admitted with confirmed stroke between 1 April 2021 and 31 March 2023. Data for the two years will be reported together. 

2. 30-day casemix adjusted mortality

The chosen performance indicator subject to outlier analysis for SSNAP is 30-day casemix adjusted mortality. This indicator has been chosen as higher than expected mortality rates may signal an issue with a particular stroke service. Mortality within 30 days of hospital admission for stroke was also one of the six stroke measures within the CCG Outcomes Indicator Set (CCG OIS 1.5) and will continue to be reported for ICBs. 

For each team, the Standardised Mortality Ratio (SMR) is reported alongside the number of observed deaths, number of expected deaths (calculated using a model adjusted for casemix and detailed in section 4), and the crude mortality rate (number of deaths divided by stroke admissions). Where the SMR falls outside the set control limits (see section 5), the stroke team or ICB/LHB will be determined either an alarm or alert outlier for mortality.

3. Data Quality

a) Data Accuracy

SSNAP employs a proactive approach to the management of data quality in order to ensure that all reporting can be undertaken with a very fast turnaround. In order to achieve this, the lead clinician at every stroke team must have clinical oversight and sign off the data as correct before each data deadline. This clinical sign off ensures that the data have been checked and approved for analysis ahead of time and is enacted by a process of “locking” the record. Only locked records are therefore analysed. This eliminates the need for any discussion or debate with teams after the deadline about whether the records need amending. No amendments are possible after the data deadline, and this process is now well established and understood by teams. In addition to this, SSNAP publicly reports the breakdown of each team’s casemix, and comparisons to the national casemix, every 3 months. Teams and clinicians are encouraged to review this information using a specifically designed “casemix tool” in order to identify any issues with casemix data quality early and correct this information before the annual data deadline. In addition, the casemix models used in the SSNAP mortality reporting are peer-reviewed models published in Stroke, which has been well publicised to all teams. Teams are therefore aware of the need to accurately report these data items.

b) Case Ascertainment

Case ascertainment is determined by comparison with estimates from previous HES/PEDW figures, updated by teams supported by documentation of clinical coding where necessary. Teams are penalised for low case ascertainment in their regular reporting; this has led to very high case ascertainment – for example in 2022-2023 98% of routinely admitting teams in England and Wales submitted over 90% of their stroke cases to SSNAP. The case ascertainment for each team is always reported alongside mortality results. If an individual stroke team or ICB/LHB has a case ascertainment of less than 60% in the relevant annual period, their mortality data is not reported.

c) Data Completeness

A core minimum dataset must be completed for every confirmed stroke added to SSNAP and only complete and locked data is analysed. High data quality and completeness is ensured through the use of built in validations which prevents illogical data being entered.

The only casemix variable used which is non-mandatory is the full National Institutes of Health Stroke Scale (NIHSS) on arrival. All other variables are fully complete for all patients. The SSNAP mortality report uses two models; one for patients with a fully completed NIHSS, and another for patients where the NIHSS is incomplete – where it is incomplete, the model utilises the level of consciousness (mandatory variable) instead. The NIHSS was fully completed for 93.7% of patients nationally, and a range of initiatives are in place to encourage teams to complete this for even more patients. The percentage of patients at each team with a fully completed NIHSS score is also reported alongside the mortality results. 

For the outcome variable, death within 30 days of hospital arrival (or onset if onset in hospital), linkage to the Office of National Statistics (ONS) register of deaths is used to ensure completeness, as well as in-hospital deaths reported on SSNAP.

Further information on data analysis and methodology is available here: https://www.strokeaudit.org/Audits/Clinical-audit-information/Data-analysis-and-methodology.aspx 

4. Casemix Adjustment

The SSNAP mortality models adjust for important casemix variables; patient age, stroke type, diagnosis of Atrial Fibrillation (AF) prior to stroke, and stroke severity (either NIHSS on arrival or, if incomplete, level of consciousness on arrival). The variables in the model were determined by backwards elimination of factors, and validated both internally and externally using a population-based study (the South London Stroke Register), which demonstrated that the models are very reliable. This methodology for the derivation of casemix variables was published in the peer-reviewed journal Stroke (Bray BD, Campbell J, Cloud GC, Hoffman A, James M, Tyrrell PJ, Wolfe CD, Rudd AG. Derivation and External Validation of a Case Mix Model for the Standardized Reporting of 30-Day Stroke Mortality Rates. Stroke. 2014; 45: 3374-3380). The models are recalibrated each year, and the ROC statistic, predictiveness curve and coefficients are reported in the technical appendix for each model. The possibility of “residual confounding” from variables not included in the model is raised in the letters sent to alert and alarm teams.

5. Detection of Outliers

Alarms are defined as teams or ICBs/LHBs outside the 99.8% control limit. Alerts are defined as teams outside the 99% control limit. Byar’s approximation is used to calculate the control limits to determine whether teams or ICBs/LHBs fall outside the alarm or alert levels.

6. Presentation of Outliers

Two different funnel plots are used for comparison between different teams, and between ICBs/LHBs, as they clearly demonstrate how the limits of acceptable performance depend on the number of cases. In addition, public tables of mortality present the data in tabular form, alongside the information on case ascertainment and data completeness.

7. Policy for Management of Outliers

The provisional timescales outlined below may be changed due to issues arising, for example, when public holidays fall, the analytical processes, or if there are no outliers in any one of the categories of alerts or alarms are identified (e.g., at ICB level outliers), or if one stage takes longer (in which case all subsequent timings are adjusted). However, results for trusts will never be put into the public domain any earlier than 8 weeks after team alarms are contacted, in order for teams to discuss the findings with SSNAP, to be supported to understand the data, to conduct mortality reviews, and to investigate potential causes.

Step	Process	Owner
1	Lead contact for stroke at each hospital approves the data for analysis by locking records by the data deadline. After this deadline, no further changes to the data are possible (see “data accuracy” above, for further explanation of the pre-deadline process).	Lead Clinician at team
2	Financial year data downloaded from the webtool.	SSNAP
3	Patient data file sent securely to NHS England for linkage with ONS mortality (at least 4 months after data collection to allow time for deaths to be recorded).	SSNAP
4	Mortality data analysed. Identify the team outliers (alarms), and their positions on the funnel plot.	SSNAP
5	Identify the alert (borderline) teams and their position on the funnel plot.	SSNAP
6	Identify the ICB/LHB alarm outliers and their position on the ICB/LHB funnel plot. Identify any ICBs/LHBs who are not outliers themselves but whose patients primarily go to alarm teams.	SSNAP
7	Contact healthcare provider Lead Clinician by telephone, prior to sending written notification of confirmed ‘alarm’ status to healthcare provider CEO and copied to healthcare provider Lead Clinician and Medical Director. All relevant data and statistical analyses made available to healthcare provider Medical Director and CEO. Letters are sent by SSNAP Clinical Director with a view to giving the Trust an opportunity to investigate the findings and to receive support from SSNAP. The letter informs the trust that the CQC1 and NHS England2/Welsh Government3, HQIP4 and National Clinical Director for Stroke in England5/National Clinical Lead for Stroke in Wales6 will be informed. The letter states that the trusts have 10 working days to respond, copying in the CQC/Welsh Government, and a further 10 working days to finalise action plans. The letter offers the stroke team help to put together a plan for service improvement if it does look as if there are issues that need to be addressed which may include taking advantage of the peer review visits offered by the Stroke Peer Review Scheme. Healthcare provider CEO informed that SSNAP will publish information of comparative performance which will identify healthcare providers.	SSNAP Clinical Lead/ team
8	Notify CQC and NHS England/Welsh Government, HQIP and National Clinical Director for Stroke in England/National Clinical Lead for Stroke in Wales of ‘alarm’ status.	SSNAP
9	Contact healthcare provider Lead Clinician by telephone, prior to sending written notification of confirmed ‘alert’ status Lead Clinician and copied to Medical Director to healthcare provider Lead Clinician. All relevant data and statistical analyses made available. Letters are sent by SSNAP Clinical Director and inform the trust that the CQC and NHS England/Welsh Government, HQIP and National Clinical Director for Stroke in England/National Clinical Lead for Stroke in Wales will be informed for information only, and that trusts do not have to formally respond.	SSNAP Clinical Lead/ team
10	Notify CQC and NHS England/Welsh Government, HQIP and National Clinical Director for Stroke in England/National Clinical Lead for Stroke in Wales of ‘alert’ status.	SSNAP
11	Arrange phone calls with alarm and alert teams as required.	SSNAP Clinical Lead/ team
12	Within 10 working days of notification, acknowledge receipt of the written notification confirming that a local investigation will be undertaken with independent assurance of the investigation’s validity for ‘alarm’ level outliers, copying in the CQC/Welsh Government.	Healthcare provider CEO
13	If no acknowledgement received within 10 working days, a reminder letter should be sent to the healthcare provider CEO. For outliers in Wales, the Welsh Government, HQIP and National Clinical Lead for Stroke in Wales are copied in. For outliers in England, the National Clinical Director for Stroke in England is copied in.	SSNAP
14	If reminder letter is not acknowledged within 15 working days of reminder letter, Welsh Government and National Clinical Lead for Stroke in Wales (for outliers in Wales) and National Clinical Director for Stroke in England (for outliers in England) notified of non-compliance in consultation with HQIP.	SSNAP
15	Within 20 working days of notification, finalise action plan.	Healthcare provider CEO
16	Individual team mortality files are uploaded to the webtool for each team. SSNAP informs users by sending a standard email.	SSNAP
17	The all-teams public table of mortality is uploaded for logged in users, and users are informed by a standard email. Programme Manager emails this table of mortality to HQIP for sign-off.	SSNAP
18	ICB/LHB alarm letters and letters informing ICBs/LHBs whose team are outliers are sent. No formal response is required.	SSNAP
19	The all-ICB/LHB public table of mortality is uploaded for logged in users and users are informed by a standard email. Programme Manager emails this table of mortality to HQIP for sign-off.	SSNAP
20	Phase public table to public. This is subject to sign-off from funders but will be no earlier than 8 weeks after team alarms are contacted.	SSNAP

¹Via [e-mail] for England only

²Via [e-mail] for England only

³Via [e-mail] for Wales only

⁴ Via HQIP PM and AD, see the HQIP website for contact details: https://www.hqip.org.uk/about-us/our-team/ 

⁵Via email address for National Clinical Director for Stroke in England for England only

⁶Via email address for National Clinical Lead for Stroke in Wales for Wales only

Table 1.1: Summary of mortality outlier phased reporting process (steps 7-20 above)	Stage 0	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5	Stage 6	Stage 7	Stage 8
Alarm trust contact details obtained. Phone calls made. Letters sent.
CQC and NHS England/Welsh Government, HQIP and National Clinical Leads for Stroke in England and Wales informed of alarm trusts.
Alert trust contact details obtained. Phone calls made, Letters sent.
CQC and NHS England/Welsh Government, HQIP and National Clinical Leads for Stroke in England and Wales informed alert trusts.
Alarm trust – time for response
Phone calls with teams
Alarm trust – time to finalise plan
Team mortality files uploaded
ICB/LHB alarm letters sent
All-teams public tables uploaded – all logged in users
Programme Manager sends all-teams table to HQIP
ICB/LHB public tables uploaded – all logged in users
Programme Manager sends ICB/LHB table to HQIP
Public table made public (subject to sign-off from funders).

8. Who to contact for further information

If you have any further queries about the mortality outlier process, or about SSNAP in general, please feel free to contact the SSNAP helpdesk at [e-mail].

9. Mortality results

Mortality results for stroke teams and ICBs/LHBs are published online on the SSNAP website: www.strokeaudit.org in excel spreadsheet format.

Results for 2021/23 are available here:

Download Public Table of Mortality for Teams             Download CCG and LHB Public Table of Mortality

SSNAP Outlier Policy 2024 - External Version 3.3