Context  We wished to determine which factors are important in ensuring interviewers are able to make reliable and valid decisions about the non‐cognitive characteristics of candidates when selecting candidates for entry into a graduate‐entry medical programme using the multiple mini‐interview (MMI).

Methods  Data came from a high‐stakes admissions procedure. Content validity was assured by using a framework based on international criteria for sampling the behaviours expected of entry‐level students. A variance components analysis was used to estimate the reliability and sources of measurement error. Further modelling was used to estimate the optimal configurations for future MMI iterations.

Results  This study refers to 485 candidates, 155 interviewers and 21 questions taken from a pre‐ prepared bank. For a single MMI question and 1 assessor, 22% of the variance between scores reflected candidate‐to‐candidate variation. The reliability for an 8‐question MMI was 0.7; to achieve 0.8 would require 14 questions. Typical inter‐question correlations ranged from 0.08 to 0.38. A disattenuated correlation with the Graduate Australian Medical School Admissions Test (GAMSAT) subsection ‘Reasoning in Humanities and Social Sciences’ was 0.26.

Conclusions  The MMI is a moderately reliable method of assessment. The largest source of error relates to aspects of interviewer subjectivity, suggesting interviewer training would be beneficial. Candidate performance on 1 question does not correlate strongly with performance on another question, demonstrating the importance of context specificity. The MMI needs to be sufficiently long for precise comparison for ranking purposes. We supported the validity of the MMI by showing a small positive correlation with GAMSAT section scores.