Construct validity is essential to evaluate the generalizability of findings on literacy and dyslexia. Operational definitions of reading literacy determine the measurement method, yielding territory or country-wide literacy rates. This practice echoes the norm in diagnosis and prevalence estimates of dyslexia. International Large-Scale Assessments (ILSA) of literacy such as the Programme for International Student Assessment (PISA) compare countries’ performances in relation to how well their students are reading. In this paper, we reexamine the validity claims and evidence using the examples of countries in Southeast Asia—Indonesia, Malaysia and Thailand, purported to have high proportions of poor readers. The challenge of characterizing reading performance and designing suitable measures for valid international comparisons is similar across phases of reading development and proficiency. The importance of the specificity of scripts and languages for reading abilities and impairments is highlighted. We suggest ways in which researchers can approach the assessment of reading proficiency from a cross-cultural and an interdisciplinary perspective. These can foster contextual caveats for generating and interpreting evidence.