Are Cognitive Test Scores Comparable Across Countries and Generations?

The question: what does it take to compare two scores?

Three of the most quoted quantitative claims in the social sciences rest on the same hidden premise.

The first is a country ranking. Every few years, the Programme for International Student Assessment (PISA) publishes scores that let a minister say that her country's fifteen-year-olds are "one year of schooling behind" a neighbor's. The second is a dataset: several widely used compilations assign every nation a single average "national IQ," and these numbers have been fed into hundreds of studies of economic growth, institutions, and health. The third is a trend: for a century, measured intelligence rose in every country that kept records — the Flynn effect — and recently the rise stalled or reversed, prompting headlines about a generation getting less intelligent.

Each claim is a comparison. The country ranking compares people in different countries; the national-IQ dataset compares people in different countries and samples; the trend compares people born at different times. And every comparison quietly assumes something about the instrument. It assumes that the test measures the same underlying ability in both groups, on the same scale, with the same difficulty, so that a given score means the same thing on both sides of the comparison.

Psychometrics has a name and a test for that assumption: measurement invariance. This brief asks a narrow question about it: when is a cognitive-test score comparison across countries or generations actually supported by evidence, and when is it not? The answer is not "never" or "always." It is that comparability is a claim requiring demonstration for each population and each time point — and that the most sweeping comparisons routinely made in public debate are exactly the ones that overstep what has been demonstrated.

Why it matters

The stakes are concrete on four fronts.

Policy. PISA rankings drive real decisions — curriculum reform, teacher policy, budget priorities — and the political force of a ranking comes from the belief that the numbers are commensurable. If a two-place gap is within the uncertainty of the scaling model rather than a fact about learning, the policy inference changes.

Economic inference. National-IQ datasets are used as explanatory variables in cross-country regressions. If those numbers are not comparable measurements, the regressions are correlating incomparable quantities with income, and the causal stories built on them inherit the flaw.

Clinical decisions. Intelligence tests carry nation-level norms. Whether a child, an immigrant, or a minority-group member is described as functioning at a particular level depends on whether the norm applies to them, which is a measurement-invariance question, not a clinical judgment.

Science. Cognitive aging, education, and development research all compare scores over time. A trend reported as a change in ability can be a change in the test — differential item functioning over the years — and the two cannot be separated without an invariance analysis.

What "comparable" requires: invariance in plain terms

Think of a test as a ruler with a printed scale. Two groups can be compared only if the ruler's markings mean the same distance in both. Measurement invariance formalizes this in levels:

• Configural invariance: the same broad factor structure holds in each group (the test measures the same constructs).
• Metric (weak) invariance: the items load on those factors with the same strength, so the units are comparable.
• Scalar (strong) invariance: the items also have the same intercepts, so group means can be compared.
• Strict invariance: residual variances also match, which is what comparing spread, inequality, or variance requires.

A test that fails scalar invariance can still be ordered within each group, but a difference between group means is no longer a clean difference in ability — it absorbs differential item functioning (DIF), the tendency of particular questions to behave differently in different populations for reasons unrelated to the trait. Every substantive comparison — means, gaps, trends, dispersions — maps onto a specific rung that must be established first.

There is a second, subtler requirement. Statistical invariance asks whether the model's parameters hold. It does not ask whether the construct itself is stable. The general factor g is the shared variance among diverse tests; the "positive manifold" is that shared variance. If the strength of the manifold changes across cohorts, then a battery can be simultaneously measurement-invariant and drifting in what it measures. A comparison can therefore pass the formal test and still mislead.

Evidence 1: cross-country comparisons in large international assessments

PISA is engineered for comparability. Its scores are calibrated so that successive cycles are directly comparable, and its design explicitly tests for measurement non-invariance — items are rotated across positions within test forms precisely to detect position and adaptation effects, and non-invariance analyses are run on field-trial data each cycle [21]. The relevant question is whether that engineering survives scrutiny.

The strongest defense comes from a robustness study. Researchers re-scaled PISA 2015 science data under a series of technical alterations — varying the item-response model, allowing or removing item-by-country interactions, and changing how "not reached" items were treated — and found that cross-country comparisons of central tendency within wealthy OECD countries were, in their words, largely unaffected; the substantive conclusions barely moved [8]. In plain terms: for rich countries, the ranking is not an artifact of the statistician's choices.

The strongest critique comes from a reanalysis of PISA 2006 reading data that found "strong evidence of misfit" of the scaling model and "very strong evidence" of DIF, and concluded that the country rankings were not robust [9]. The two results are not contradictory. They target different things: the robustness study examined central tendency in the OECD; the critique examined whether the model itself fits and whether a ranking can be defended question by question. A separate comparison of four international surveys found that the cross-country pattern of central tendency was robust to the choice of item-response model, but that the pattern of dispersion was not — and that results for less-developed countries were much less robust than for rich ones [11]. So the same data can support a ranking of average scores among similar, wealthy systems while failing to support a comparison of inequality, or a ranking that spans widely different development levels.

This is why PISA's own technical work treats invariance as an ongoing empirical question rather than a settled property. A 2019 OECD working paper on large-scale surveys notes that conventional invariance criteria behave poorly when many countries are compared at once and that the step from metric to scalar invariance frequently fails, motivating approximate-invariance methods that allow some parameters to vary [20]. Item-fit thresholds for PISA's cognitive items have been recalibrated specifically to judge cross-country comparability [10], and even PISA's non-cognitive scales are not automatically invariant across immigrant and non-migrant respondents [18]. The honest summary: comparability in large assessments is real but bounded — strongest for average performance among similar systems, weakest for dispersion and for the extremes of the development range.

Evidence 2: clinical batteries, norms, and "national IQ"

Clinical intelligence batteries pose the question more sharply because their norms are applied to individuals.

A widely cited review of invariance in neurocognitive testing found that in over half of the studies it examined, IQ batteries were not invariant across subgroups defined by ethnicity, gender, educational background, and other variables — which means population-based norms cannot be assumed valid for every subgroup [12]. Yet invariance is clearly achievable under the right conditions. A systematic review of 57 studies concluded that the factor structure of cognitive-ability measures generalizes well across cultures, and a cross-national study of the Wechsler Intelligence Scale for Children, Fifth Edition (WISC-V) found strict invariance across France, Spain, and the United States — the strongest rung, permitting direct comparison of latent means and variances [13][14].

The decisive evidence is not a single verdict but the variation. Tests of four archival Wechsler datasets against US norms found that the Ghanaian and Kenyan samples achieved strict invariance, while the Pakistani and Sudanese datasets failed to reach scalar or strict invariance — meaning, in the authors' words, that the global IQ scores from the latter two cannot be compared to American scores on the same tests [15]. A 2025 study of a general cognitive-performance composite in the Survey of Health, Ageing and Retirement in Europe — 55,569 adults aged 60–102 across 27 countries and Israel — found substantial non-invariance in both factor loadings and item intercepts and concluded that researchers should abstain from cross-country comparisons of the composite [16].

At the extreme sits the "national IQ" enterprise, which assembles convenience samples, often children, sometimes deliberately unrepresentative, spanning decades, and transforms them into one number per country through a chain of adjustments. A detailed re-analysis of the most recent compilation argues it is not fit for purpose: the sources are unrepresentative of national populations, the transformations embed the compilers' prior beliefs, and — the deeper point — comparable measures of cognitive ability cannot be generated from culturally diverse populations without proving the very invariance the enterprise assumes [19]. And even a non-verbal test cannot be assumed culture-free: a review of visuo-spatial reasoning tests finds cultural assumptions embedded in the "culture-fair" formats themselves, which disqualifies clean interpretation of group mean differences on their own [17]. Taken together: invariance is a live, per-study empirical property, not a nameplate any battery carries.

Evidence 3: cross-generational comparisons and the Flynn effect

For most of the twentieth century, measured intelligence rose roughly two to three IQ points per decade — the Flynn effect, one of the most durable regularities in the behavioral sciences [7]. Since the 1990s it has flattened or reversed in several high-income countries. Both the rise and the reversal are claims about comparing people born in different decades, and so both depend on cross-temporal invariance.

A recent wave of studies takes that requirement seriously and mostly finds the changes are real. In two population-representative samples tested with measurement-invariant subtests, researchers documented meaningful gains across all six intelligence domains between 2005 and 2024 — while the strength of the general factor decreased [3]. An Austrian standardization study found positive Flynn effects across all domains alongside a trend toward declining g saturation [4]. A study of German secondary-school samples found a decline in measurement-invariant fluid reasoning between 2012 and 2022 — an "anti-Flynn" effect in the domain most often assumed to be culturally stable [5]. In other words, when drift is ruled out, the changes persist; but the changes are domain-specific and accompanied by a weakening of the very common factor that makes a single "IQ" number meaningful.

Two designs sharpen the interpretation. A Norwegian study using military-conscription data with family relationships recovered the entire Flynn effect — the rise, the turning point, and the reversal — from within-family variation, meaning the trend is not produced by changing population composition but by environmental change acting on siblings [1]. And a 2026 analysis of 579,379 Norwegian men across 25 birth cohorts found that the reversal began earlier among sons of high-income fathers, was masked for a time by rising educational attainment among lower-income groups, and became visible across all strata only after educational expansion plateaued [2]. A single national trend line, it turns out, was an average of strata whose trajectories diverged — which is to say the "national IQ decline" was not one phenomenon at all.

Even here the evidence is not unanimous. A study of Austrian Air Force pilots and air traffic controllers spanning up to 25 years found positive Flynn effects but no systematic decline in the positive manifold — a direct counterexample to the weakening-g story in a specialized sample [6]. The honest reading is that cross-temporal comparability can be earned in well-invariant batteries, but that the results are heterogeneous by domain, stratum, and sample, and that a single headline number hides more than it reports.

What each comparison actually identifies: mechanisms that need different designs

The most common error in this literature is treating four different problems as one.

Differential item functioning and adaptation error are measurement problems, identifiable by invariance testing — configural, metric, scalar, strict — using multi-group factor models or item-response models with anchor items. This is what the PISA item-fit work and the clinical invariance studies directly estimate.

Construct drift — a changing positive manifold or a shifting composition of what a battery measures — is not an item defect. It requires longitudinal anchor-item designs and latent-variable modeling to separate genuine domain change from item drift, as the cross-temporal studies explicitly attempt [3][5].

Test familiarity, schooling, and motivation are contextual confounds. They can be probed with samples that vary in schooling, with effort manipulations, or by examining whether an effect replicates in low-familiarity populations — a different design from any invariance test, and the reason a task can be invariant in form yet unfamiliar in practice.

Sampling and representativeness are design problems, addressed by weights, coverage checks, and comparisons of sample quality, not by psychometrics. The national-IQ critique is largely a sampling critique [19].

Only after these four are handled can a true ability difference be estimated. A study that establishes scalar invariance but uses convenience samples has tested the ruler, not the population; a study with flawless sampling but an unadapted translation has done the reverse. Confusing the mechanisms is how a defensible measurement result becomes an indefensible inference about people.

Competing explanations and counterevidence

The skeptical case is strong, but it can be overstated, and the strongest counterevidence deserves stating plainly.

First, cross-cultural generalizability is not a null result. The 57-study systematic review found the underlying cognitive model generalizes well, and the WISC-V result reached strict invariance across three countries [13][14]. The failure cases tend to involve unadapted translations, non-comparable samples, or norms transplanted without local validation. Second, the PISA robustness study found the central rankings stable [8], so claims that PISA rankings are meaningless overreach. Third, the positive-manifold decline is contested: at least one well-measured sample shows no such decline [6].

What survives the counterevidence is not "scores can never be compared" but "a comparison of scores is an evidential claim whose strength varies enormously." The same test can be strictly invariant against one national norm and non-invariant against another [15]. That is the central fact, and it is why blanket claims — in either direction — are the wrong form of answer.

Best current answer, confidence, and reversal conditions

Best current answer. Cognitive test scores are comparable only conditionally and locally. A comparison is supported when (a) the populations are defined, (b) the test was designed or adapted for both, and (c) measurement invariance at the required rung is demonstrated on actual data for those populations and that time point. Within those bounds — a well-adapted battery compared across similar, wealthy, schooled populations, or a well-invariant battery compared across nearby cohorts — comparison is defensible. Beyond them — a single "national IQ" per country, a full cross-national ranking spanning development levels, a comparison of dispersion, or a single number describing a generational trend — the comparison exceeds what the measurement supports. And even a formally invariant test can drift in meaning, so statistical invariance is necessary but not sufficient.

Calibrated confidence.

• ~90% that measurement invariance is a genuine precondition for score comparison and is frequently untested or shown to fail.
• ~85% that "national IQ" datasets do not provide comparable, unbiased measures of national cognitive ability.
• ~80% that cross-country composite comparisons (of the SHARE type) are unsupported without demonstrated invariance.
• ~75% that a single national generational trend misrepresents the underlying data, because strata with different trajectories are averaged together.
• ~60% that the positive manifold is weakening over time; the evidence points that way but is contested across samples.

Reversal conditions. I would substantially raise confidence in broad cross-national comparability if a preregistered, multi-country study established scalar or strict invariance for a common battery and showed that the resulting national means predict external outcomes (schooling, income, health) comparably to within-country estimates. I would lower confidence in the generational-trend story if the positive-manifold decline failed to replicate in large, independent, population-representative cohorts. And I would treat any single headline ranking or trend as revisable whenever the invariance analysis behind it is not reported.

The practical upshot is a rule that is easy to state and hard to fake: before comparing two cognitive-test scores, name the populations, name the comparison rung, and show the invariance. If any of the three is missing, the comparison is a hypothesis, not a result.

References

1. Bratsberg, B., & Rogeberg, O. (2018). Flynn effect and its reversal are both environmentally caused. Proceedings of the National Academy of Sciences, 115(26), 6674–6678. https://doi.org/10.1073/pnas.1718793115
2. Bratsberg, B., Fjell, A. M., Rogeberg, O., & Walhovd, K. B. (2026). Educational expansion and the hidden reversal of the Flynn effect—Differential trends across socioeconomic strata. Proceedings of the National Academy of Sciences, 123(28), e2536089123. https://doi.org/10.1073/pnas.2536089123
3. Oberleiter, S., Fries, J., Dejardin, F., Heller, J., Schaible, C., Vetter, M., Voracek, M., & Pietschnig, J. (2024). Inconsistent Flynn effect patterns may be due to a decreasing positive manifold: Cohort-based measurement-invariant IQ test score changes from 2005 to 2024. Intelligence, 107, 101867. https://doi.org/10.1016/j.intell.2024.101867
4. Andrzejewski, D., Oberleiter, S., Vetter, M., & Pietschnig, J. (2024). Increasing IQ test scores and decreasing g: The Flynn effect and decreasing positive manifold strengths in Austria (2005–2018). Journal of Intelligence, 12(12), 130. https://doi.org/10.3390/jintelligence12120130
5. Oberleiter, S., Patzl, S., Fries, J., Diedrich, J., Voracek, M., & Pietschnig, J. (2024). Measurement-invariant fluid anti-Flynn effects in population-representative German student samples (2012–2022). Journal of Intelligence, 12(1), 9. https://doi.org/10.3390/jintelligence12010009
6. Oberleiter, S., Wurzer, J., Mikas, M., Held, M., Wieland, B., Zeilinger, E. L., Voracek, M., & Pietschnig, J. (2025). Generational IQ test score changes and the positive manifold of intelligence: Evidence from Austrian Air Force pilots and air traffic controllers (1992–2016). Frontiers in Psychology, 16, 1547520. https://doi.org/10.3389/fpsyg.2025.1547520
7. Pietschnig, J., & Voracek, M. (2015). One century of global IQ gains: A formal meta-analysis of the Flynn effect (1909–2013). Perspectives on Psychological Science, 10(3), 282–306. https://doi.org/10.1177/1745691615577701
8. Jerrim, J., Parker, P., Choi, A., Chmielewski, A. K., Sälzer, C., & Shure, N. (2018). How robust are cross-country comparisons of PISA scores to the scaling model used? Educational Measurement: Issues and Practice, 37(4), 28–39. https://doi.org/10.1111/emip.12211
9. Kreiner, S., & Christensen, K. B. (2014). Analyses of model fit and robustness: A new look at the PISA scaling model underlying ranking of countries according to reading literacy. Psychometrika, 79(2), 210–231. https://doi.org/10.1007/s11336-013-9347-z
10. Joo, S.-H., Khorramdel, L., Yamamoto, K., Shin, H. J., & Robin, F. (2021). Evaluating item fit statistic thresholds in PISA: Analysis of cross-country comparability of cognitive items. Educational Measurement: Issues and Practice, 40(2), 37–48. https://doi.org/10.1111/emip.12404
11. Brown, G., Micklewright, J., Schnepf, S. V., & Waldmann, R. (2007). International surveys of educational achievement: How robust are the findings? Journal of the Royal Statistical Society: Series A, 170(3), 623–646. https://doi.org/10.1111/j.1467-985x.2006.00439.x
12. Wicherts, J. M. (2016). The importance of measurement invariance in neurocognitive ability testing. The Clinical Neuropsychologist, 30(7), 1006–1016. https://doi.org/10.1080/13854046.2016.1205136
13. Wilson, C. J., Bowden, S. C., Byrne, L. K., Joshua, N. R., Marx, W., & Weiss, L. G. (2023). The cross-cultural generalizability of cognitive ability measures: A systematic literature review. Intelligence, 98, 101751. https://doi.org/10.1016/j.intell.2023.101751
14. Wilson, C. J., Bowden, S. C., Byrne, L. K., Vannier, L.-C., Hernandez, A., & Weiss, L. G. (2023). Cross-national generalizability of WISC-V and CHC broad ability constructs across France, Spain, and the US. Journal of Intelligence, 11(8), 159. https://doi.org/10.3390/jintelligence11080159
15. Warne, R. T. (2023). Tests of measurement invariance of three Wechsler intelligence tests in economically developing nations in South Asia and Sub-Saharan Africa. Gifted and Talented International, 38(2), 122–138. https://doi.org/10.1080/15332276.2023.2245007
16. García-Mollá, A., Fernández, I., Oliver, A., & Tomás, J. M. (2025). Measurement invariance of a general cognitive performance measure across 27 European countries and Israel. European Journal of Ageing, 22(1), 37. https://doi.org/10.1007/s10433-025-00872-y
17. Gonthier, C. (2022). Cross-cultural differences in visuo-spatial processing and the culture-fairness of visuo-spatial intelligence tests: An integrative review and a model for matrices tasks. Cognitive Research: Principles and Implications, 7(1), 11. https://doi.org/10.1186/s41235-021-00350-w
18. Casas, M., & Sireci, S. G. (2024). Measurement invariance across immigrant and nonimmigrant populations on PISA non-cognitive scales. International Journal of Testing, 25(2), 111–126. https://doi.org/10.1080/15305058.2024.2426022
19. Sear, R. (2022/2026). 'National IQ' datasets do not provide accurate, unbiased or comparable measures of cognitive ability worldwide [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/26vfb
20. Organisation for Economic Co-operation and Development. (2019). Invariance analyses in large-scale studies (OECD Education Working Papers). OECD Publishing. https://doi.org/10.1787/254738dd-en
21. Organisation for Economic Co-operation and Development. (2026). PISA 2025 assessment and analytical framework. OECD Publishing. https://www.oecd.org/content/dam/oecd/en/publications/reports/2026/05/pisa-2025-assessment-and-analytical-framework_3abce6d6/86c36975-en.pdf