Continuous Improvement of Inter-rater Reliability in Transition Compliance at a State Agency

A mixed methods action research study was designed to answer three research questions based on inter-rater reliability (IRR) in compliance calls for transition at a state education agency, perceived confidence levels in making and discussing compliance calls, and perceived confidence in sharing transition resources. An innovation based on andragogy and frame of reference training (FOR) was designed and implemented with twelve participants to answer these questions. To measure the effects of the innovation, participants completed a pre-and post-innovation review of five student files, analyzing the IRR for the group as compared with a gold standard (GS) both before and after the innovation. Additionally, a smaller group sample for the same five files post-innovation was collected to compare group results for IRR with the GS to the combined individual results. A retrospective survey was also utilized in which participants rated their confidence in each component pre- and post-innovation. Based upon analyses of these data, several key findings were identified. Higher inter-rater reliability was noted when participants reviewed files within small groups and in the area of annual Individualized Education Program (IEP) goals aligned with measurable post-secondary goals. Lower IRR was reported in nuanced files, files for students with low-incidence disabilities, and files with more instances of non-compliance. Results indicated that participant confidence in making and discussing transition files in the field improved post-innovation. Lastly, participants indicated higher confidence in sharing best practices in transition with the field post-innovation. Implications for this research include training suggestions, additional practice with low-incidence and nuanced files at the state agency, and group review of files in other state monitoring systems.