Research Projects

​A key component of air carrier advanced qualification programs is the calibration and training of instructors and evaluators and assurance of reliable and valid data in support of such programs. A significant amount of research is available concerning the calibration of air carrier evaluators, but no research exists regarding the calibration of pilot school check instructors. This study was designed to determine if pilot school check instructors can be calibrated against a gold standard to perform reliable and accurate evaluations.

Calibration followed the principles and theories of andragogy and adult learning and teaching, including an emphasis on the cognitive domain of learning, learner-centered instruction, and human resource development. These in combination with methods commonly used in aviation instruction aimed to increase the effectiveness of the calibration. Discussion of these combinations is included. A specific method for delivery of the calibration was provided along with a complete lesson plan. This study used a one-group pretest-posttest design. A group of 10 pilot school check instructors was measured before and after receiving rater calibration training. Statistical measures included raw inter- and referent-rater agreement percentages, Cohen’s kappa and kappa-like statistics for inter- and referent-rater reliability, Pearson product-moment correlations for sensitivity to true changes in pilot performance, and a standardized mean absolute difference for grading accuracy. Improvement in all the measurements from pretest to posttest was expected, but actual results were mixed. However, a holistic interpretation of the results combined with feedback from the check instructors showed promise in calibration training for pilot school check instructors. A thorough discussion of the limitations and lessons learned from the study, recommendations for pilot schools, and recommendations for future research is included.