Featured Publication

Using simulation to assess the sensitivity and specificity of a signal detection tool for multidimensional public health surveillance data

Henry Rolka 1 *, Dwayne Bracy 2, Channing Russell 3, David Fram 3, Robert Ball 4

1Centers for Disease Control and Prevention (CDC), Epidemiology Program Office, Division of Public Health Surveillance and Informatics,
2DynTel Corporation
3Lincoln Technologies, Inc.
4Food and Drug Administration (FDA), Center for Biologics Evaluation and Research

E-mail: Henry Rolka (hrolka@cdc.gov)
*Correspondence to Henry Rolka, Centers for Disease Control and Prevention (CDC), National Center for Public Health Informatics (NCPHI), Mailstop E06, Clifton Road, NE, Atlanta, GA 30333, U.S.A.
This article is a U.S. Government work and is in the public domain in the U.S.A.

Funded by:
 Data management Division of the National Immunization Program

ABSTRACT
The objective of the work described in this paper is to develop a means for characterizing the validity of an empirical methodology for detecting signals potentially related to complicated adverse event (AE) coding terms in multidimensional public health surveillance data. The signal detection tool under evaluation is the multi-item gamma Poisson shrinkage (MGPS) estimation program. We were interested in its potential application to passive surveillance system monitoring, to screen for signals of complicated adverse event coding terms (AE terms) in complex and noisy data. The research was to design and produce a flexible and user-friendly utility for probabilistically defining complicated signals in a database, iterating large numbers of applications of the MGPS detection algorithm and establishing proportions of correct detection events. We sought to establish the specificity of the MGPS by developing a random background using a gradient that ranged from rigorous (but not very relevant) to relevant (but noisy).

To establish the sensitivity, signals were defined based on recognized public health issues of interest (such as the introduction of a new vaccine into the population). Methods of representing a signal included a simple pair-wise association consisting of a new vaccine and one AE term, as well as a more realistic complex of multiple AE terms comprising a syndrome. A web application has been developed to create and insert signals with user-defined probabilities in multiple iterations of simulated random background data. Three forms of simulated data based on the vaccine adverse event reporting system (VAERS) cumulative spontaneous database were defined to serve as background noise against which to contrast introduced vaccine adverse event signals: (1) completely random associations between vaccines and AE terms, (2) random associations of vaccine sets and AE term sets preserving naturally observed vaccine co-occurrences and AE term co-occurrences and (3) samples from the actual VAERS data as reported. Rates of detection by the MGPS algorithm can be established for specific signal patterns at varying probabilistic intensities in a choice of random background data forms. Knowing these rates is important for determining the degree of response to an MGPS signal detection event in live data. Published in 2005 by John Wiley & Sons, Ltd.

(p 551-562) Statistics in Medicine 2005 Feb 28;24(4).

Read More