Comparison of Video and Audio Rating Modalities for Assessment of Provider Fidelity to a Family-Centered, Evidence-Based Program

The current study assessed whether the interrater reliability and predictive validity of fidelity ratings differed significantly across the modalities of audio and video recordings. As empirically supported programs are moving to scale, attention to fidelity, the extent to which a program is delivered as intended, is essential because high fidelity is needed for positive program effects. Consequently, an important issue for prevention science is the development of feasible and acceptable methods for assessing fidelity. Currently, fidelity monitoring is rarely practiced, as the typical way of measuring fidelity, which uses video of sessions, is expensive, time-consuming, and intrusive. Audio recording has multiple advantages over video recording: 1) it is less intrusive; 2) equipment is less expensive; 3) recording procedures are simpler; 4) files are smaller so it takes less time to upload data and storage is less expensive; 5) recordings contain less identifying information; and 6) both clients and providers may be more willing to have sensitive interactions recorded with audio only. For these reasons, the use of audio recording may facilitate the monitoring of fidelity and increase the acceptability of both the intervention and implementation models, which may serve to broaden the scope of the families reached and improve the quality of the services provided. The current study compared the reliability and validity of fidelity ratings across audio and video rating modalities using 77 feedback sessions drawn from a larger randomized controlled trial of the Family Check-Up (FCU). Coders rated fidelity and caregiver in-session engagement at the age 2 feedback session. The composite fidelity and caregiver engagement scores were tested using path analysis to examine whether they predicted parenting behavior at age 3. Twenty percent of the sessions were double coded to assess interrater reliability. The interrater reliability and predictive validity of fidelity scores and caregiver engagement did not significantly differ across rating modality. However, caution must be used in interpreting these results because the interrater reliabilities in both conditions were low. Possible explanations for the low reliability, limitations of the current study, and directions for future research are discussed.