Introduction and Aim: Final medical examinations should provide reliable evidence that graduating students are prepared for clinical practice. However, evidence on the psychometric performance of large-scale computer-based examinations involving local and international cohorts remains limited. This study aimed to evaluate five-year trends in student performance and the psychometric characteristics of the Final Medical Examination at the large medical university in Lithuania.

Materials and Methods: This retrospective observational study included anonymised data from all Lithuanian and International medical students who completed the examination between 2021 and 2025. The examination consisted of 200 computer-based, scenario-based multiple-choice questions administered in two parts. Student performance and psychometric indicators, including facility index, discrimination index, internal consistency, and standard error of measurement, were analysed separately for both cohorts. Differences in categorical distributions across years were assessed using the Chi-square test and pairwise z-tests with Bonferroni correction. Differences were regarded as statistically significant when p < 0.05.

Results: Annual cohort sizes ranged from 263 to 285 Lithuanian students and from 74 to 169 international students. Mean examination scores varied over the five-year period in both cohorts, with the highest mean scores observed in 2023. Internal consistency ranged from 0.747 to 0.875 in the Lithuanian cohort and from 0.684 to 0.813 in the International cohort. Question difficulty distributions varied significantly over time in both cohorts (p = 0.004 and p < 0.001, respectively), with a marked increase in very easy questions in 2023. The greatest improvement in question discrimination was observed in 2022 among international students and in 2023 among Lithuanian students.

Conclusions: The examination demonstrated generally reliable psychometric performance and improved question quality, particularly during 2022-2023. Continuous psychometric monitoring and systematic question-bank review are important for maintaining examination quality across student cohorts.