Verification of a novel point-of-care HbA1c device in real world clinical practice by comparison to three high performance liquid chromatography instruments

Introduction A real world clinical study was designed and conducted to evaluate the performance of a novel point-of-care device for determination of glycated haemoglobin A1c (HbA1c), A1C EZ 2.0, in daily clinical practice. Materials and methods Five hundred and fourteen subjects were included in this study, and divided into three groups. HbA1c was measured by A1C EZ 2.0 and three different high performance liquid chromatography (HPLC) devices: Bio-Rad Variant II Turbo, Tosoh HLC-723 G8 and Premier Hb9210 separately. Precision of A1C EZ 2.0 was also evaluated. Results Results obtained from A1C EZ 2.0 and all HPLC devices are correlated. Passing-Bablok regression analysis shows the equation of A1C EZ 2.0 results against the mean of HPLC devices with corresponding 95% confidence intervals (95% CI) for the intercept and slope is y = 0.10 (- 0.17 to 0.10) + 1.00 (1.00 to 1.04) x. Bland-Altman difference plot shows that the mean relative difference between A1C EZ 2.0 and Variant II Turbo, G8, Hb9210 and all HPLC results is 2.5%, 0.6%, 0.4% and 1.1%, respectively. In addition, 121 pairs of results determined by using both venous and capillary blood prove that the difference of two kinds of blood sample causes no notable variation when measured by A1C EZ 2.0. Precision study gives 2.3% and 1.9% of total coefficient of variation for normal and abnormal HbA1c sample in A1C EZ 2.0. Conclusions HbA1c values measured by A1C EZ 2.0 were in good accordance with the results obtained with the reference HPLC devices.


Introduction
Diabetes mellitus (DM) has become a cause of serious concern worldwide, as well as in China. The prevalence of DM and pre-diabetes among adult Chinese population in 2010 was estimated to be 11.6% and 50.1%, respectively, representing 113. 9 and 493.4 million people. However, only 28.4% of DM patients have been diagnosed till now in China (1). This implies that over eighty million patients with undiagnosed DM still need to be screened and diagnosed, which can be a major challenge for the public health system of China. Glycated haemoglobin A 1c (HbA 1c ) values, which can reflect the average blood glucose concentration over the past 90-120 days, is now not only a standard index for evaluating the effectiveness of treatment, but also a widely acknowledged parameter for screening and diagnosing DM (2)(3)(4). High performance liquid chromatography (HPLC) is currently widely used for HbA 1c measurement in clinical practice, but it is generally used in the hospital central laboratory, where physician and patients have to wait for 1-3 days for getting the test results. Point-of-Wang Y. et al.
Performance assessment of A1C EZ 2.0 care (POC) HbA 1c devices, which can provide test results within few minutes, is fast becoming a tool for the screening and diagnosis of DM at a relatively lower cost. Clinical studies have shown that POC devices help doctors in making timely adjustments in treatment plan, which further brings additional benefits to long-term glycaemic control (5)(6)(7). On the other hand, there were common concerns about the accuracy and precision of POC HbA 1c devices. Lenters-Westra et al. have studied the performance of eight POC HbA 1c devices in 2009, and only two of them met the generally accepted performance criteria (8). Most of POC HbA 1c devices in the study failed to achieve the total CV of 3%, and show notable differences of analytical performance between different reagent lots. A similar investigation of seven POC HbA 1c devices has been performed by the same authors in 2014 (9). Generally, significant improvement has been found in precision and accuracy of these devices, four of seven devices met the generally accepted performance criteria, though some specific problems, such as interferences or choice of reference device for calibration has been found in this study (9).

Study design
We performed an analytical verification study of the POC HbA 1c device A1C EZ 2.0 by comparison with three other IFCC certified secondary reference method (SRM) HPLC instruments. Blood samples were collected from subjects in hospital, and the study lasted for seven months (detail information of instruments and subjects are given below).
Both venous and fingertip blood samples were used in the comparison, to investigate the test results difference between A1C EZ 2.0 and three HPLC instruments. Potential difference of test results caused by sample types was also studied by testing both venous and fingertip blood from the same subjects in the same instrument in this study.
We also evaluated the precision of A1C EZ 2.0 following CLSI EP5-A2 (Evaluation Of Precision Performance Of Quantitative Measurement Methods. Approved Guideline -Second Edition). Two potassium-ethylenediaminetetraacetic acid (K 2 -EDTA) anticoagulant whole blood samples (one within normal range and the other in the higher range of HbA 1c values) were tested in two runs per day in duplicate on twenty consecutive days. Controls provided by manufacturers were used before and after every day's experiments to ensure that performance of all instruments meets the required standards by manufacturers. Otherwise calibration was carried out following manufacturers' manual.

Subjects
After approval from the institutional review committee of Shanghai Jiao Tong University Affiliated Sixth People's Hospital (Shanghai, China), and in accordance with the principles of the Helsinki Declaration II, we randomly recruited 514 subjects from the Outpatient Departments of the Endocrinology and Metabolism of Shanghai Jiao Tong University Affiliated Sixth People's Hospital. We included both diabetes and non-diabetes cases aged fourteen and older. We excluded specimens from subjects with anaemia, haemolytic disease, red cell cycle abnormalities, nephropathy and other known comorbidi-ties, whose diseases were reported through a questionnaire survey or determined by relevant laboratory results. Subjects aged 60 (15 -93) were enrolled from April to October 2016. Median age of male and female subjects was 61 and 60, respectively.
All subjects have signed informed consent before their blood sample being collected. Subjects were divided into three groups, venous blood samples were collected from 197 subjects (subject group I, SGI), fingertip capillary whole blood samples were collected from other 196 subjects (subject group II, SGII), and both venous blood samples and fingertip capillary whole blood samples were collected from the rest 121 subjects (subject group III, SGIII).

Sample collection
All venous blood samples were collected in the standard tube (3mL, WeiGao Group, China) containing K 2 -EDTA anticoagulant, stored at 4 °C and then tested within 24 hours of collection. One tube of venous blood sample was collected from each subject in SG I and SG III. For subjects in SG II, the fingertip capillary whole blood samples were directly used for the tests by A1C EZ 2.0, fingertip capillary whole blood samples were also collected in one mini tube containing K 2 -EDTA anticoagulant and tested with three the HPLC devices within 24 hours.

Methods
The

Statistical analysis
Linear regression analysis Passing-Bablok linear regression analysis of HbA 1c values obtained by A1C EZ 2.0 versus three HPLCs was further conducted to assess the accuracy of A1C EZ 2.0 while considering the above mentioned HPLC devices as references (11). The linear regression analysis in each subject group was performed separately. Six hundred and thirty-five pairs of data from all 514 subjects (121 subjects tested both venous blood and fingertip capillary whole blood) were also combined for linear regression analysis for each reference HPLC device. Finally, to check overall agreement independent of chosen reference HPLC device, the mean HbA 1c values of three HPLC devices (referred as "mean SRM" below) were used to perform the linear regression against results given by A1C EZ 2.0.
Bias plot, absolute value of relative difference and mean absolute value of relative difference Bland-Altman difference plot was performed to evaluate the difference between the HbA 1c values given by A1C EZ 2.0 and HPLC devices (12). Mean difference as well as the mean relative difference between A1C EZ 2.0 results and each HPLC results were calculated. The limits of agreement for the difference between A1C EZ 2.0 and each HPLC device were also calculated to reflect the distribution of these differences. In addition, the absolute value of relative difference (ARD) analysis and mean absolute value of relative difference (MARD) analysis was performed. Absolute value of relative difference and MARD were calculated by using the following equations: The distribution of the difference was further analysed based on the data of ARD and MARD. Six hundred and thirty-five pairs of data from all 514 subjects were combined and used for bias plot, ARD and MARD analysis. Precision Precision of A1C EZ 2.0 was evaluated using CLSI EP5-A2. The calculation of within-run, betweenrun, within-day and total coefficient of variance was followed by the standard method provided in CLSI EP5-A2.

Comparison of venous blood results and fingertip capillary whole blood results
Values of 121 pairs of venous blood and fingertip capillary whole blood on A1C EZ 2.0 were compared by Passing-Bablok linear regression analysis, to see if there is a significant difference between the two kinds of blood samples.

Sensitivity and specificity analysis
Sensitivity and specificity were calculated by using HbA 1c clinical decision limit for DM diagnosis of 6.5% and mean SRM or Premier Hb9210 as reference value. MedCalc (Ostend, Belgium) statistical software (version 16.4) was used to perform statistical analyses.

Overall results
A total of 514 subjects, including 266 men and 248 women, were randomly recruited in this study. The venous and fingertip capillary whole blood were tested in the SGI (N = 197) and SGII (N = 196) group, respectively. SGIII (N = 121) subjects had evaluations of both venous blood and fingertip capillary whole blood.
The frequency distribution patterns of HbA 1c values for all four instruments were similar ( Figure 1).

Linear regression
Linear regression analysis indicates that test results of A1C EZ 2.0 show good agreement with the results obtained by reference HPLC devices ( Table  2). The linear regression equation of A1C EZ 2.0 values against mean SRM is presented in Figure 2. A1C EZ 2.0 is most closely related to Premier Hb9210 of all the three reference HPLC devices. The slope and intercept given by linear regression between A1C EZ 2.0 and Premier Hb9210 with all of the combined data is 1.00 (1.00 to 1.00) and 0.00 (-0.00 to 0.00), respectively, which coincides with the ideal y = x function.

Bias analysis
The mean difference between A1C EZ 2.0 values versus mean SRM is 0.08% HbA 1c (mean relative difference of 1.1%) ( Figure 3 and Table 2). Respective mean difference for each reference HPLC device was 0.17% HbA 1c for Bio-Rad Variant II Turbo, 0.04% HbA 1c for Tosoh HLC-723 G8 and 0.02% HbA 1c for Premier Hb9210 (mean relative difference of 2.5%, 0.6% and 0.4%, respectively). All detailed statistics for each subject group are listed in Table 2. Bland-Altman difference plot of A1C EZ 2.0 results vs mean SRM shows that 96.5% of results have relative differences from -6.9% to 9.2% (Figure 3). In addition, 84.6% of data has ARD lower than 6%, and 94.2% of data has ARD lower than 8%. MARD of 3.26% of A1C EZ 2.0 results compared with mean SRM was found.

Differences in HbA 1c values depending upon types of blood sample
In this study, no notable difference in HbA 1c values between venous and fingertip capillary whole blood of SGIII subjects was seen as measured by A1C EZ 2.0. Passing-Bablok regression analysis towards paired data from venous and fingertip capillary whole blood yielded the equation y = -0.10 + 1.00 x; with 95%CI of intercept -0.10 to 0.08, and slope: 0.98 to 1.00, which indicates that there is no significant difference between HbA 1c values on measurement with different kinds of blood sample. Also, there was no significant difference in  HbA 1c values between venous and capillary blood on assessment with reference HPLC devices (Bio-Rad Variant II Turbo: y = 0.00 (0.00 to 0.00) + 1.00 (1.00 to 1.00) x; Tosoh HLC-723 G8: y = 0.0 (0.00 to 0.00) + 1.0 (1.00 to 1.00) x; Premier Hb9210: y = 0.0 (0.00 to 0.00) + 1.00 (1.00 to 1.00) x.     Table 3). The total CV is 2.3% for low HbA 1c sample (5.2% HbA 1c ) and 1.9% for high HbA 1c sample (11.6% HbA 1c ). In addition, the within-run CV, between-run CV and betweenday CV all fell in the acceptable region (less than 2%, Table 3), implying that A1C EZ 2.0 has consistent performance over a certain period of time under different circumstances.

Sensitivity and specificity of test by A1C EZ 2.0
We further evaluated the sensitivity and specificity of A1C EZ 2.0. While considering Premier Hb9210 results as reference and HbA 1c of 6.5% as clinical decision level for DM diagnosis, the sensitivity and specificity of A1C EZ 2.0 were both 96%. When mean SRM was used as reference, sensitivity and specificity of A1C EZ 2.0 were 96% and 93%, respectively (Table 4).

Discussion
A large scale clinical study in the hospital setting was designed to evaluate the performance of a novel POC HbA 1c device A1C EZ 2.0. Overall performance of A1C EZ 2.0 was fairly good in our study, and was similar to the previously reported performance by another study (7). What is different is that in this study we used the clinical specimens including the fingertip capillary whole blood, venous blood samples and both venous blood samples and fingertip capillary whole blood samples from the same participant. Analysis of our results indicates that A1C EZ 2.0 has smallest mean difference (0.02% HbA 1c ) with Premier Hb9210, and slightly less consistence with Tosoh HLC-723 G8 and Bio-Rad Variant II Turbo. This might be as both A1C EZ 2.0 and Premier Hb9210 use boronate affinity method. Boronate affinity method was proven to have good tolerance to some common interference factors such as haemoglobin variants and certain drugs, which usually affect the HbA 1c results determined by cation-exchange method (13,14). In this study, sample from one patient (data was excluded in our statistics) was found to have haemoglobin abnormality and both Bio-Rad Variant II Turbo and Tosoh HLC-723 G8 failed to give reasonable results, while Premier Hb9210 and A1C EZ 2.0 gave similar HbA 1c value (7.6% and 7.8% HbA 1c , respectively). For several other samples where the relative differences between Premier Hb9210 and Tosoh HLC-723 G8 or Bio-Rad Variant II Turbo were more than 10%, we inferred that some interference factors might have existed in those samples and Premier Hb9210 results were considered to be more reliable. Although venous blood may differ from capillary blood to some extent, in many aspects, such as total protein, blood cell concentration and ion concentration, the HbA 1c value usually shows no notable difference when measured using different type of blood sample in different instruments (15)(16)(17)(18)(19). There was no significant difference in HbA 1c values between venous blood and fingertip capillary whole blood of SGIII subjects as seen on measurement by A1C EZ 2.0.
Sensitivity and specificity analysis of A1C EZ 2.0 indicates that the risk of false positive and negative results is very low with evaluation by A1C EZ 2.0. Several former studies have shown that the ideal cut-off HbA 1c values for diagnosis of DM varies among ethnic groups, and the best clinical decision level for Chinese population might be 6.3% (20)(21)(22)(23). We also used 6.3% as cut-off value, and found that the sensitivity and specificity of A1C EZ 2.0 was 96% and 89% as compared to Premier Hb9210 as reference, and 97% and 89% as compared to mean SRM as reference. In this analysis, when clinical decision level was changed from 6.5% to 6.3%, high sensitivity was maintained, whereas specificity dropped as compared to both references (Table 4). In conclusion, the sensitivity and specificity of A1C EZ 2.0 meets the general clinical requirement for HbA 1c testing, and can be used for clinical diabetes diagnosis.
As the POC HbA 1c devices can provide results in short time and at relatively lower cost, these can help in improving the doctor-patient interaction, appropriate modification of the treatment if required and/or diagnosis/screening of new cases of DM from large population with higher efficiency. On the other hand, doubts towards POC HbA 1c devices have been raised in some studies showing that some POC HbA 1c devices may not be suitable for clinical use due to their unsatisfactory precision and/or variation in different lots (5,6). Thus, more precise and accurate POC HbA 1c device is in urgent need, especially in countries like China, with an increasing incidence and prevalence of DM, as expensive HPLC HbA 1c instruments are usually inaccessible and unaffordable to many grassroots medical institutions.
While it is widely accepted that HPLC devices represent one of the most accurate way to measure HbA 1c , we found that performance of POC HbA 1c device is not significantly different from the HPLC devices in this study. The 84.7% of Bio-Rad Variant II Turbo results have ARD lower than 6%, and 94.6% have lower than 8% as compared with Premier Hb9210 results. Both values are slightly lower than A1C EZ 2.0 87.6% and 96.5%, respectively (data not shown). The controls were tested before and after every day's experiments and routine calibrations were carried out in all HPLC devices to confirm that performance of all instruments were in the claimed acceptable region by manufacturers. However, Bio-Rad Variant II Turbo results show consistent negative bias (about -0.2% HbA 1c ) as compared with Tosoh HLC-723 G8, Premier Hb9210, A1C EZ 2.0 in our study. This negative bias was observed in all three subject groups and through seven months period of this study. Such kind of bias might have occurred due to the change of condition of cation-exchange column during use, or the individual calibration process we performed, as both are commonly seen in clinical applications. On the other hand, use of POC HbA 1c device like A1C EZ 2.0, which usually uses single cartridge or strip for one test can decrease the risk of systematic bias and might achieve similar performance as HPLC in clinical practice while having slightly higher CV than HPLC device at the same time.
In conclusion, we verified a novel POC HbA 1c device A1C EZ 2.0, and found its performance met the general clinical requirement in this study. However, there are still some limitations in our study. Only one lot of strip of A1C EZ 2.0 was used in this study, thus lot-to-lot variation of strips has not been evaluated; we also did not evaluate the stability of test strip of A1C EZ 2.0 in this work. Tests by A1C EZ 2.0 and HPLC instruments were not performed at exactly the same point of time. All these limitations merit further investigation in future.