Testing and Reporting Performance Results of ANSI/AAMI EC57 Heart Rate and ST Segment Measurement Algorithms

The "ANSI/AAMI EC57 Testing and Reporting Performance Results of Heart Rhythm and ST Segment Measurement Algorithms" standard, jointly published by the American National Standards Institute (ANSI) and the Association for Medical Device Improvement (AAMI), describes a method used to test and report the performance of algorithms used to detect heart rhythm abnormalities, including the ST segment.

The availability of annotated arrhythmia and ST databases has allowed testing of different automated arrhythmia detection algorithms on the same data. This proposed application provides a protocol for a reproducible test with realistic clinical requirements and emphasizes the record-based presentation of results reflecting an algorithm's ability to detect clinically significant events. This standard uses beat-by-beat comparisons to measure performance in QRS, ventricular ectopic beat (VEB), and supraventricular ectopic beat detection. Additionally, study-by-study comparisons are used to measure an algorithm's ability to detect consecutive VEB and supraventricular ectopic beats.

Ventricular flutter, atrial flutter, ventricular fibrillation, and atrial fibrillation detection are addressed. The evaluation of heart rate variability measurement algorithms and ST segment measurement algorithms is also examined. While this standard aims to establish clinically significant performance metrics for comparing algorithms, it must be acknowledged that some clinical concerns cannot be addressed within the context of this proposed application. Existing databases do not yet include a representative sample of non-ventricular arrhythmias occurring in a clinical setting, pacemaker-implanted patients, or artifacts specific to a significant portion of ECG signals. Furthermore, these databases have limited bandwidth and should be used with caution when testing algorithms designed for devices with full ECG diagnostic bandwidth. Therefore, the clinical implications of a test are limited by the size, scope, and characteristics of the databases used for the test. Performance metrics derived from such tests should be evaluated as uncertain performance indicators in clinical settings.

This proposed application is designed for testing algorithms, not entire systems. It is not a performance standard, but rather a set of recommendations for testing cardiac rhythm and ST measurements and reporting the results of these tests. The aim of this proposed application is to ensure the reproducibility of automated testing methods.

This proposed practice applies to algorithms implemented in devices or systems that use automated methods to analyze ECGs. This standard is applicable to human-operated, standalone devices that use automated methods to analyze recorded ECGs, and to real-time event recorders that use automated methods to record abnormal events.

Beat-by-beat techniques and test methodologies other than the specified rhythm analysis and ST segment analysis are outside the scope of this standard. Evaluation of systems relying heavily on user interaction is also outside the scope of this standard. However, if beat-by-beat evaluations are performed, the results of these tests must be consistent with this proposed practice.

In medical diagnostics, electrocardiogram (ECG) analysis plays a crucial role in detecting cardiac arrhythmias and ischemic events. As healthcare technology advances, particularly with the integration of artificial intelligence and automated algorithms, ensuring the reliability and accuracy of these tools is becoming increasingly important. This is where the ANSI/AAMI EC57 standard comes in, providing a comprehensive framework for testing and reporting the performance of algorithms designed to detect cardiac rhythm disturbances and measure ST segment deviations in ECG signals. This standard was developed to address the increasing complexity of ECG interpretation software, making it an indispensable benchmark for manufacturers, researchers, and clinicians.

The ANSI/AAMI EC57 standard is not a descriptive design standard, but rather a methodological guideline for performance evaluation. The fundamental objectives of this standard are twofold:

• Standardized test protocols: These outline procedures for evaluating algorithm accuracy using reference ECG databases, minimizing subjective interpretation.
• Transparent reporting: Ensuring informed comparisons between systems requires detailed descriptions of test conditions, failure rates, and limitations.

This standard applies to both standalone ECG analysis software and integrated systems in devices such as Holter monitors, exercise stress testers, and implantable cardioverter defibrillators. Specifically, it focuses on black-box testing, which evaluates inputs and outputs without resorting to internal algorithmic processes to ensure reproducibility in laboratories worldwide.

At the heart of the ANSI/AAMI EC57 standard is a classification scheme for ECG beats that divides arrhythmia detection into clinically significant categories. This standard divides 23 original beat types into five overarching classes:

• Normal or bundle branch block beats: Examples include normal sinus rhythm, left/right bundle branch block. Clinical significance includes basic healthy rhythms; in some cases, accelerated beats.
• Supraventricular ectopic beats: Examples include atrial prematurity and nodal prematurity. Clinical significance: Premature beats originating from the top of the ventricles; usually benign, but indicative of atrial problems.
• V. Ventricular ectopic beats: Examples include premature ventricular contractions (PVC) and ventricular escape. Clinically significant, ectopic foci in the ventricles may indicate an increased risk of arrhythmias.
• F. Fusion beats: Examples include the fusion of ventricular and normal beats. Clinical significance: Hybrid beats resulting from competing pacemakers are rare but important for assessing rhythm stability.
• Q Unclassifiable or accelerated pulses: Examples include accelerated, fusion, or unknown. Clinical significance encompasses everything for ambiguous signals, providing comprehensive coverage.

This classification has become common in research and development, facilitating apples-to-apples comparisons in studies on ECG machine learning models.

This standard outlines a multi-step testing process:

• Database selection: Algorithms should be evaluated on at least two independent datasets. These contain thousands of pulses from different demographic characteristics.
• Performance metrics: Sensitivity (true positive rate), positive predictive power (accuracy), and overall accuracy are calculated per class. Mean absolute error and standard deviation are required for ST segment measurement.
• Cross-validation: Tests should include both within-sample (training data) and out-of-sample (unseen data) assessments to identify overfitting.
• Extreme cases: Algorithms for simulating clinical variability are challenged with noise, artifacts, and variable derivation configurations.

Reporting should include tabulated confusion matrices, shot-by-shot descriptions, and statistical confidence intervals to ensure clarity.

For many years, our organization, with its competent and experienced team, has been understanding the needs of its clients, providing the advanced testing services they require, and assisting in the establishment, implementation, and improvement of management systems. Within this framework, we also provide testing services to businesses in accordance with the "ANSI/AAMI EC57 Testing of Heart Rate and ST Segment Measurement Algorithms and Reporting of Performance Results" standard.