Background: Cath lab activation based on ST-elevation myocardial infarction (STEMI) criteria is founded on aging data and requires evolution. In the “Occlusive Myocardial Infarction (OMI) Manifesto,” emergency physicians Dr. Steve Smith, Dr. Pendell Meyers, and Dr. Scott Weingart introduced a new paradigm —OMI vs. non-occlusive myocardial infarction (NOMI).

The OMI/NOMI paradigm focuses on the presence of coronary occlusion, while STEMI/NSTEMI categorizes myocardial infarctions based on electrocardiogram (ECG) findings. Patients with OMI exhibit higher mortality and worse left ventricular function compared to those with NOMI.^{1, 2, 3} Detecting OMI is more difficult and necessitates scrutiny of the ECG, which is challenging in a busy emergency department where ED clinicians are interrupted more than ten times per hour.^{4, 5} Some OMI ECG signs include ST elevation in only one lead, subtle ST elevation with minimal reciprocal changes, isolated ST depressions, and hyperacute T waves.

To meet this challenge, Dr. Steve Smith, Dr. Pendell Meyers (Dr. Smith’s ECG Blog), and their team developed The Queen of Hearts, a machine-learning AI model that has the potential to aid in the early detection of subtle OMI ECG changes. Accurately identifying OMI changes in ECG that STEMI criteria might otherwise miss would allow for more timely intervention, potentially salvaging more myocardium. An AI model that is highly sensitive in detecting OMI while maintaining a high degree of specificity would be an ideal tool to support emergency physicians’ clinical decision-making. The performance of this tool is unknown.

Paper: Herman R, Meyers HP, Smith SW, et al. International evaluation of an artificial intelligence-powered electrocardiogram model detecting acute coronary occlusion myocardial infarction. Eur Heart J Digit Health. 2023;5(2):123-133. Published 2023 Nov 28. PMID: 38505483

Clinical question: “Can an AI model detect an OMI lesion using a single 12-lead ECG?”

What They Did:

• Investigators performed a retrospective derivation study followed by validation on an internal data set from the same Acute Coronary Syndrome (ACS) database.
• Cases eligible for inclusion were randomly assigned to a model development training set (derivation set) and testing set (validation set).  
  • The training set included ECG feature extraction and classification
    • Feature extraction used 60,000 parameters
    • The classification component combined all extracted features and used an additional 150,000 parameters.
  • The validation data set was used for hyperparameter tuning and threshold selection. 
• Investigators then tested the AI model on two data sets
  • An internal European data set (internal validation set)
  • A separate US data set (external validation set) from the DOMI ARIGATO database.
• They compared the AI model with the existing criteria for detecting OMI on 12-lead ECGs and analyzed the AI model in various subgroups.

Population:

• Derivation Set: Random selection of ACS patients from the Cardiovascular Centre Aalst in Belgium and ACS patients from an international image database patient.
• EU Internal Test Set: Random Selection of ACS patients from the Cardiovascular Centre Aalst in Belgium and ACS patients from an international image database patient.
• US External Test Set: Patients from the DOMI ARIGATO database.

Exclusion:

• • ECGs >24 h before CAG and post-CAG
  • ECGs with poor signal quality 
  • ECGs with missing Expert Annotation, undigitizable ECGs, Baseline ECGs (additionally excluded from the US External Database)

Intervention:

AI-powered ECG model implemented on ECGs from the internal EU and external US datasets.

Comparator:

• Blinded physician annotations of the standard ‘STEMI criteria’ on ECG
• Blinded subjective ECG expert annotations of OMI
• Angiographic clinical outcome data

Outcomes:

Primary Outcome: AI model’s ability to identify patients with angiographically confirmed OMI using only the 12-lead ECG.

Secondary Outcomes:

• OMI AI model performance across demographic and ECG subgroups
• A comparison of the AI model performance against the existing STEMI criteria for detecting acute coronary occlusion from 12-lead ECGs
• A sensitivity analysis of AI model performance using various angiographic and laboratory cut-offs of OMI
• An evaluation of misclassified cases

Results:

• The derivation set used in the AI model development included 18,616 ECGs from 10,543 patients with clinically validated outcomes.
• The overall test set included 3254 ECGs from 2222 patients 
  •  The internal EU testing cohort 2016 ECGs from 1630 patients
  •  The US testing cohort 1238 ECGs from 633 patients 
• The prevalence of OMI differed between the internal EU and the external US test sets, 16% compared with 36.2%, respectively ( < 0.001).
• The patients in the US test set were younger, had more ECGs recorded before catheterization, and were more likely to present with a STEMI-positive ECG.

AI Model Performance:

• Achieved an Area Under the ROC Curve (AUC) of 0.938 [95% CI: 0.924–0.951].
• Accuracy: 90.9% [95% CI: 89.7–92.0].
• Sensitivity: 80.6% [95% CI: 76.8–84.0].
• Specificity: 93.7% [95% CI: 92.6–94.8].

STEMI Criteria Performance:

• STEMI criteria accuracy: 83.6% [95% CI: 82.1–85.1].
• Sensitivity: 32.5% [95% CI: 28.4–36.6].
• Specificity: 97.7% [95% CI: 97.0–98.3].

ECG Experts Performance:

• Accuracy of ECG experts was 90.8% [95% CI: 89.5–91.9].
• Sensitivity: 73.0% [95% CI: 68.7–77.0].
• Specificity: 95.7% [95% CI: 94.7–96.6].

OMI AI Model vs. STEMI Criteria:

• The OMI AI model performs significantly better than the STEMI criteria in sensitivity, Negative Predictive Value (NPV), Matthews correlation coefficient (MCC), and AUC.
• However, it has lower specificity and Positive Predictive Value (PPV) compared to the STEMI criteria.

OMI AI Model vs. ECG Experts:

• The OMI AI model has higher sensitivity and NPV than ECG experts.
• It shows equal performance in AUC and is adjudicated as equal overall to ECG experts.
• Specificity and PPV are lower than ECG experts, and MCC is neutral.

ECG Experts vs. STEMI Criteria:

• ECG experts have higher sensitivity, NPV, MCC, and AUC than STEMI criteria.
• They perform the same in specificity and PPV compared to STEMI criteria, leading to significantly better adjudication.

Strengths:

• Rigorous Methodological Approach: The study follows a comprehensive methodological approach, encompassing stages of development, validation, and comparison.
• Large and Diverse Dataset: The model was trained and tested on a substantial dataset of 18,616 ECGs from 10,543 patients with ACS across multiple international cohorts. This diversity enhances the model’s generalizability and robustness.
• Expert Interpretation and Validation: All cases in the derivation set included expert ECG interpretations alongside clinically validated angiographic outcome data, ensuring high accuracy in the model’s training process.
• High Agreement Among Experts: Two authors, serving as ECG experts, annotated all tracings for the presence of OMI while being blinded to all clinical data. They achieved a 94% agreement (kappa = 0.849), demonstrating the reliability of the expert annotations.
• Independent Review: Blinded independent clinical reviewers adjudicated all angiographic data in the EU internal testing set, adding an extra layer of objectivity and reliability to the validation process.
• Comprehensive Performance Comparison: The study compares the AI model’s performance with existing STEMI criteria and expert ECG interpretations. This sets a quantifiable humanistic standard, highlighting the AI model’s enhanced performance.

Limitations:

• Applicability Limited to ACS Patients: The AI model was developed using patients and ECGs exclusively from ACS databases, restricting its applicability to only those with ACS.
• Disease-Oriented Outcomes: The outcomes in this study are disease-oriented. While facilitating the diagnosis of OMI may lead to improved patient-oriented outcomes, this was not directly studied.
• Limited Generalizability to Asymptomatic Patients: The study is not generalizable to a broader population of asymptomatic patients and was not designed to quantify other relevant clinical endpoints such as mortality, in-hospital complications, or major adverse cardiovascular events (MACE).
• Lack of Prospective Validation: The validation set was analyzed retrospectively, lacking prospective validation to confirm the model’s effectiveness in real-world clinical settings.
• Randomization Process Not Mentioned: The randomization process used to allocate cases to the derivation or validation set is not mentioned, which may impact the robustness of the findings.
• Comparison Limited to 12-Lead ECG: The AI model was compared to the 12-lead ECG alone. Some patients undergo emergency angiography without clear STEMI criteria based on the full clinical picture. Therefore, the interpretation of the overall gain is limited without a pragmatic comparison to real-world clinical practices and patient-oriented outcomes.
• Limited Generalizability to Younger Population and Women: Approximately 10% of ECGs were from patients ≤45 years of age, and three-quarters of the cases were from males, limiting the generalizability to younger populations and women.
• Inappropriate Use of P-Values: The inclusion of p-values in Tables 1 and 2 is puzzling, as this is not a randomized controlled trial (RCT). Demographic differences between validation sets are expected and desired for external validity.
• Variability in Care Standards: Significant differences in clinical presentation and management between patients in Europe and the USA (e.g., the USA had younger patients and more ECGs before catheterization) could affect the model’s performance across different healthcare systems.
• Subjective Outcome Verification: The detection of OMI relied on visual verification of TIMI flow on angiograms, which may be subjective.
• Conflict of Interest: The lead author disclosed a conflict of interest as the co-founder and Chief Medical Officer of Powerful Medical. Other authors have disclosed employee or shareholder status in Powerful Medical.

Discussion:

Inside the Numbers: The data for this AI model is impressive, showcasing a remarkable capability in early and accurate detection of OMI on ECGs, demonstrating a sensitivity of 80.6% (76.8–84.0) and specificity of 93.7% (92.6–94.8). The AI model not only surpassed the standard STEMI ECG criteria [sensitivity 32.5% (28.4–36.6) and specificity 97.7% (97.0–98.3)] but also matched the performance of Dr. Steve Smith and Dr. Pendell Meyers, two well known ECG experts [sensitivity 73.0% (68.7–77.0) and specificity 95.7% (94.7–96.6)]. Additionally, when considering the existing evidence, the AI model would likely outperform ED physicians’ and cardiologists’ ability to detect ischemia on ECG, who achieved sensitivities of approximately 65% and specificities ranging from 65–79% in multiple studies.^{7, 8, 9} This high accuracy demonstrates AI’s potential to improve diagnostic processes and patient outcomes in emergency settings.

The AI model’s PPV in this study was 0.780 (0.742–0.816), and the NPV was 0.946 (0.935–0.957) for the primary outcome. PPV and NPV are heavily influenced by disease prevalence, and a high prevalence increases the PPV, indicating that a positive test result is more likely to be a true positive. The 16% and 36.2% prevalence of OMI in the internal and external validation sets are likely much higher than expected from a random group of patients assessed for ACS in the ED on any given day. Consequently, the PPV and NPV are likely lower in a less risky population with a lower prevalence for ACS.  

The AI model’s AUC for detecting OMI was 0.938 (0.924–0.951), with an optimal threshold of 0.1106. The optimal threshold refers to the chosen point that maximizes the AI model’s accuracy. The point is a probability that ranges from 0–1. However, investigators must choose the value (optimal threshold) at which the model determines whether the ECG is positive or negative. Therefore, the optimal threshold converts a continuous variable (probability) into a binary decision, such as distinguishing between the presence or absence of OMI on ECG. If the threshold is set too low, it might result in high sensitivity but low specificity, leading to many false positives. The ROC curve is a graphical plot that illustrates the diagnostic ability of a binary classifier as its discrimination threshold is varied. In this instance, a ROC curve with an AUC of 0.938 is outstanding and highlights the potential of the AI model to optimize clinical decision-making processes.

Critical Biases and Considerations: The primary flaw in this paper is selection bias. All patients included in the derivation and validation sets were selected from ACS databases. As mentioned, the prevalence of OMI in the internal and external validation sets is very high. Physicians should exercise caution when applying this data more broadly (i.e., all patients with an ECG in the ED). 

The AI model detected OMI in 979 cases total, 267 of which also met the STEMI criteria on ECG. Therefore, 27% of the OMIs detected by the AI model might have been more obvious and less noteworthy to an emergency physician aiming to improve their diagnostic capabilities. However, the remaining 73% of AI-detected OMIs are particularly interesting because they require meticulous ECG scrutiny for accurate diagnosis. While not all these AI-detected OMI cases met the primary outcome criteria, technology can fill a void in identifying patients who may benefit from emergent intervention despite the lack of STEMI-specific criteria on ECG.

“Time is myocardium,” and the primary goal in ACS treatment is to detect OMI on ECG as early as possible to prevent myocardial necrosis. Utilization of STEMI criteria missed 330 OMI patients —false negatives. Among these, 133 had a median revascularization time of 9.3 hours but were correctly identified by the AI model on the first ECG. Early detection can potentially improve patient outcomes, especially in cases with real-world median angiography time of 9 hours. While this data is compelling, it highlights the need for prospective evaluation of the AI model compared to the performance of the average emergency physician to fully assess its clinical effectiveness.

The Future and Transformative Potential of AI: This AI model’s development and validation process mirrors that of a clinical decision instrument, beginning with retrospective derivation followed by internal and external validation. Before widespread implementation, prospective validation in various clinical settings with diverse populations is necessary. Additionally, utilization studies should confirm that the AI model achieves its intended goals, such as earlier detection of OMI and improved patient-oriented outcomes.

While the idea of AI taking over the world might be an exaggeration, its transformative impact cannot be overstated. The continuous advancement and integration of AI technologies can lead to more efficient, accurate, and personalized solutions. Moreover, AI’s continuous refinement through machine learning suggests its performance will only improve over time. As the AI model is exposed to more data and varied cases, it can refine its algorithms, enhance its accuracy, and adapt to new patterns, making it an invaluable tool in the medical field. And, unlike human counterparts, AI will not fatigue and will maintain high accuracy levels, even after the 12th hour on duty and dozens of ECG interpretations. The possibilities for AI applications in healthcare are virtually limitless. 

Author’s conclusion: “AI model outperformed gold-standard STEMI criteria in the diagnosis of OMI, but further prospective clinical studies are needed to define the role of the OMI AI model in guiding ACS triage and the timely referral of patients benefiting from immediate revascularization.”

Clinical Bottom Line:

The Queen of Hearts AI model demonstrates impressive accuracy, surpassing STEMI criteria and matching expert interpretation for detecting OMI on ECG. However, the high prevalence of OMI in the study’s datasets may overestimate AI’s ability to detect OMI in a general ED population with a lower disease prevalence. Ultimately, the model requires prospective validation in diverse clinical settings before widespread adoption— but this could be a winning hand.

Follow Me: 

 

 

 

 

References:

1. Wang TY, Zhang M, Fu Y, et al. Incidence, distribution, and prognostic impact of occluded culprit arteries among patients with non-ST-elevation acute coronary syndromes undergoing diagnostic angiography. Am Heart J. 2009;157(4):716-723. PMID: 19332201
2. Pride YB, Tung P, Mohanavelu S, et al. Angiographic and clinical outcomes among patients with acute coronary syndromes presenting with isolated anterior ST-segment depression: a TRITON-TIMI 38 (Trial to Assess Improvement in Therapeutic Outcomes by Optimizing Platelet Inhibition With Prasugrel-Thrombolysis In Myocardial Infarction 38) substudy. JACC Cardiovasc Interv. PMID: 20723851
3. Khan AR, Golwala H, Tripathi A, et al. Impact of total occlusion of culprit artery in acute non-ST elevation myocardial infarction: a systematic review and meta-analysis. Eur Heart J. 2017;38(41):3082-3089. PMID: 29020244
4. Ratwani RM, Fong A, Puthumana JS, Hettinger AZ. Emergency Physician Use of Cognitive Strategies to Manage Interruptions. Ann Emerg Med. 2017;70(5):683-687. PMID: 28601266
5. Chisholm CD, Dornfeld AM, Nelson DR, Cordell WH. Work interrupted: a comparison of workplace interruptions in emergency departments and primary care offices. Ann Emerg Med. 2001;38(2):146-151. PMID: 11468609
6. Herman R, Meyers HP, Smith SW, et al. International evaluation of an artificial intelligence-powered electrocardiogram model detecting acute coronary occlusion myocardial infarction. Eur Heart J Digit Health. 2023;5(2):123-133. Published 2023 Nov 28. PMID: 38505483
7. Veronese G, Germini F, Ingrassia S, et al. Emergency physician accuracy in interpreting electrocardiograms with potential ST-segment elevation myocardial infarction: Is it enough?. Acute Card Care. 2016;18(1):7-10. PMID: 27759433
8. McCabe JM, Armstrong EJ, Ku I, et al. Physician accuracy in interpreting potential ST-segment elevation myocardial infarction electrocardiograms. J Am Heart Assoc. 2013;2(5):e000268. Published 2013 Oct 4. PMID: 24096575
9. Paez Perez Y, Rimm S, Bove J, et al. Does the Electrocardiogram Machine Interpretation Affect the Ability to Accurately Diagnose ST-Elevation Myocardial Infarction by Emergency Physicians?. Crit Pathw Cardiol. 2023;22(1):8-12. PMID: 36812338

Guest Post By:

Marco Propersi, DO FAAEM
Vice-Chair, Emergency Medicine
Assistant Emergency Medicine Residency Program Director
Vassar Brothers Hospital, Poughkeepsie, New York
Twitter/X: @marco_propersi

Joseph Bove, DO FAAEM
Associate Director Emergency Medicine
Co-Director of the EM Residency Clerkship
St. Joseph’s University Medical Center
Email: bovej@sjhmc.org

Post Peer Reviewed By: Anand Swaminathan, MD (Twitter/X: @EMSwami)