Psychometrics: Measurement, Test Construction and Standardization

This chapter answers Section XH5.2 of the Humanities and Social Sciences (XH) paper, “Psychometrics”, which is Section XH5.2 of XH5 Psychology, one of the six optional sections chosen at the time of application. It teaches the foundations of psychological measurement and the levels of measurement; scales and the construction and analysis of items (intelligence, performance, ability and aptitude items, and personality questionnaires); the methods of test construction; the three pillars of standardization — reliability, validity and norms, including the standard error of measurement; and the applications of testing in educational, counselling, clinical, organizational and developmental settings. The numericals are reliability from the Spearman–Brown formula, the standard error of measurement SEM = SD√(1−r), the item-difficulty index and the variance explained by a validity coefficient.

1. Foundations of measurement and levels of measurement

Measurement is the assignment of numbers to characteristics according to a rule. Psychological attributes such as intelligence or anxiety are latent constructs — they cannot be observed directly, so a test infers them from a sample of behaviour. Because the inference is indirect, every score carries error, and psychometrics is the discipline of quantifying and reducing that error. Stevens’ four levels of measurement fix which arithmetic a score permits.

The four levels of measurement
LevelWhat it preservesExample
NominalIdentity only (unordered categories)Diagnostic category, gender code
OrdinalRank order, but unequal gapsRank in class, ordered grade bands
IntervalEqual intervals, arbitrary zeroIQ scores, temperature in Celsius
RatioEqual intervals and a true zeroReaction time, number of errors

Beyond the level of measurement, attitude and rating scales are built by recognised methods. A Likert (summated-rating) scale presents statements answered on an agree–disagree continuum and sums the item responses into one score, dropping items that fail to correlate with the total. A Thurstone (equal-appearing intervals) scale has judges place a large pool of statements along a favourability continuum, and each surviving statement carries the median scale value they gave it. A Guttman (cumulative) scale orders items so that endorsing a strong item implies endorsing all weaker ones, yielding a reproducible response pattern. A semantic differential rates a concept on a series of bipolar adjective pairs such as good–bad.

⚠️ An arbitrary zero forbids ratios
On an interval scale the zero point is conventional, not an absence of the attribute, so ratio statements are false: an IQ of 140 is not ‘twice as intelligent’ as 70, and 20°C is not twice as hot as 10°C. Ratio claims are valid only on a ratio scale, where zero means none of the attribute.

2. Items, test types and item analysis

  • Intelligence and ability items sample reasoning — analogies, number and letter series, matrices, classification; performance tests use non-verbal tasks and apparatus (block design, mazes) and suit people with language or literacy barriers.
  • Aptitude tests predict future learning or performance, while achievement tests measure past learning; the items look alike but differ in purpose and in the inference drawn.
  • Personality questionnaires are self-report instruments, often Likert-scaled; objective (structured) tests have fixed scoring, while projective techniques present ambiguous stimuli and infer personality from the response.

Item analysis screens items after a tryout. The item-difficulty index p is the proportion of examinees who answer correctly; a high p means an easy item. An item that everyone or no one passes carries no information, so for a maximum-performance test items near p = 0.5 discriminate best (adjusted upward for guessing). The item-discrimination index D = (proportion correct in the high-scoring group) − (proportion correct in the low-scoring group); a good item has high positive D, and a negative D is a warning that the item or its key is wrong. Distractor analysis checks that each wrong option draws some low scorers.

Two further points complete item analysis. The correlation of a single item with the total test score — a point-biserial item-total correlation — is a continuous discrimination measure: items that correlate weakly with the total are candidates for revision. The analysis also depends on the test’s purpose: a power test gives ample time so that difficulty, not speed, limits the score, whereas a speed test uses easy items under strict time pressure, and the two call for different item statistics.

3. Methods of test construction

Building a test moves from a clear definition of the construct and its domain to a test blueprint (a table of specifications fixing how many items cover each content area), then a large item pool, a tryout on a representative sample, item analysis to retain the best items, and finally standardization on a norm group. The way items are selected defines the strategy.

  • Rational (content/theory-based): items are written and kept because their content logically fits the construct; depends on expert judgement.
  • Empirical (criterion-keying): an item is kept only if it actually separates known groups, regardless of its apparent content — the strategy behind inventories like the MMPI.
  • Factor-analytic: items are grouped by the latent factors that the correlations reveal, so each scale measures one dimension (as in the 16PF).
  • Construct-based: items are retained through an ongoing programme of evidence that the scores behave as the theory of the construct predicts.

4. Reliability

Reliability is the consistency of a measurement — the proportion of observed-score variance that is true-score rather than error variance, reported as a coefficient from 0 to 1. Test–retest reliability (coefficient of stability) correlates the same test given twice; parallel (alternate) forms reliability (coefficient of equivalence) correlates two equivalent forms; internal consistency measures how well the items hang together, by the split-half method, Cronbach’s alpha, or KR-20 for dichotomous items; and inter-rater reliability (e.g., Cohen’s kappa) measures agreement between scorers.

The split-half method correlates two halves of one test, but each half is only half as long, so the raw correlation underestimates the whole test’s reliability. The Spearman–Brown prophecy formula corrects it: for doubling a test the full-length reliability is r_full = 2r / (1 + r), where r is the half-test correlation. More generally, lengthening a test by a factor k gives r′ = kr / [1 + (k − 1)r]. Reliability rises with test length, with more homogeneous items, and with a more heterogeneous sample.

Cronbach’s alpha is the average of all possible split-half coefficients and is the usual index of internal consistency for scales with graded responses; KR-20 is its equivalent for right/wrong items. Alpha rises when the items are more homogeneous and when the test is longer, which means a high alpha reflects both internal consistency and sheer length, and a very high alpha can even signal redundant items. Reliability is also inflated by a heterogeneous sample, because a wider spread of true scores enlarges the true-score share of the variance.

🎯 Why longer tests are more reliable
Each item adds a little true-score signal and a little random error. Across more items the random errors tend to cancel while the true signal accumulates, so the signal-to-error ratio improves — which is exactly what Spearman–Brown quantifies, and why a ten-item scale is usually less reliable than a well-built forty-item one.

5. Validity, norms and the standard error of measurement

  • Content validity: the items cover the whole domain they claim to (judged by experts; mere surface plausibility is only face validity).
  • Criterion-related validity: the test correlates with an outside criterion — concurrent (criterion measured now) or predictive (criterion measured later); the correlation is the validity coefficient.
  • Construct validity: the test behaves as the theory predicts, shown by convergent evidence (agrees with measures of the same construct) and discriminant evidence (does not agree with measures of different ones).

Reliability sets a ceiling on validity: the validity coefficient cannot exceed the square root of the test’s reliability (taking the criterion as perfectly reliable), so error limits how well a test can predict. Norms turn a raw score into meaning by comparing it to a standardization sample: percentile ranks, standard scores (z, or T = 50 + 10z, stanines, sten), deviation IQs (mean 100, SD 15 on the Wechsler convention) and age or grade norms. Norms are only as good as the representativeness of the sample. The standard error of measurement, SEM = SD√(1 − r), where SD is the test’s standard deviation and r its reliability, gives the band of error around an observed score — a confidence interval for the true score — and shrinks as reliability rises.

ℹ️ Norm-referenced versus criterion-referenced
A norm-referenced interpretation compares a person to others (a percentile or standard score answers ‘how does this person rank?’), while a criterion-referenced interpretation compares the score to a fixed standard of mastery (‘has this person met the benchmark?’). A licensing or minimum-competency decision is criterion-referenced; selecting the top few applicants is norm-referenced. The same raw score can be read either way depending on the decision being made.

6. Applications of testing across settings

  • Educational: achievement and diagnostic tests, placement and selection, and the identification of learning difficulties and giftedness.
  • Counselling and guidance: interest inventories, aptitude and values measures that help a person match education and career to their profile.
  • Clinical: intelligence scales, objective inventories and projective techniques used in assessment, diagnosis and treatment planning.
  • Organizational: personnel selection, assessment centres, and performance and leadership appraisal, where predictive validity against job performance is central.
  • Developmental: developmental and screening scales that track growth across the life span and flag delays early.
ℹ️ The same chain in every setting
Whatever the setting, a defensible decision rests on a test that is reliable, valid for that specific use, and interpreted against appropriate norms. A test valid for one purpose (say, diagnosing a disorder) is not automatically valid for another (say, selecting employees), because validity is a property of an inference, not of the test alone.

Key takeaways

  • Measurement assigns numbers by a rule; psychological constructs are latent, so scores carry error. Stevens’ levels — nominal, ordinal, interval, ratio — fix the permissible arithmetic, and an interval scale’s arbitrary zero forbids ratio statements.
  • The item-difficulty index p is the proportion correct (near 0.5 discriminates best); the discrimination index D is high-group minus low-group proportion correct, and a negative D flags a faulty item or key. Aptitude predicts future learning, achievement measures past learning.
  • Test construction runs from construct and blueprint to item pool, tryout, item analysis and standardization. The selection strategy may be rational, empirical (criterion-keying, as in the MMPI), factor-analytic or construct-based.
  • Reliability is consistency: test–retest, parallel forms, internal consistency (split-half, Cronbach’s alpha, KR-20) and inter-rater. Spearman–Brown corrects a split-half to full length by r_full = 2r/(1+r); reliability rises with test length and item homogeneity.
  • Validity is content, criterion-related (concurrent, predictive) and construct (convergent, discriminant); maximum validity is √reliability. Norms (percentiles, z and T scores, deviation IQ) need a representative sample, and SEM = SD√(1−r) bands the true score.
  • Tests are applied in educational, counselling, clinical, organizational and developmental settings; in every one a sound decision needs reliability, validity for that specific use, and appropriate norms, because validity belongs to an inference, not to the test alone.

Practice questions (16)

Attempt each one before opening the answer. Every explanation names the tempting wrong option as well as the right one, because that is where marks are lost.

  1. IQ scores are measured on a scale with equal intervals but no true zero point. On such a scale, which statement is NOT justified?

    1. A person with an IQ of 140 is exactly twice as intelligent as a person with an IQ of 70
    2. The difference between IQ 100 and IQ 110 equals the difference between IQ 120 and IQ 130
    3. A person with an IQ of 120 has a higher measured score than a person with an IQ of 100
    4. Two people with the same IQ score are treated as equivalent on the attribute the test measures
    Show answer

    Answer: A — A person with an IQ of 140 is exactly twice as intelligent as a person with an IQ of 70

    An interval scale supports order and equal differences but not ratios, because its zero is arbitrary — so the ‘twice as intelligent’ claim is not justified. Equal differences, a higher score meaning more, and equal scores meaning equivalence are all valid on an interval scale.

  2. In a multiple-choice achievement test, one item has an item-difficulty index of p = 0.97. For a test meant to spread examinees out, what does this most likely indicate about the item?

    1. The item is extremely difficult and should be removed because almost nobody answered it correctly
    2. The item has ideal difficulty for a maximum-performance test and should always be retained unchanged
    3. The item is very easy and contributes little to discriminating between stronger and weaker examinees
    4. The item’s discrimination index is necessarily very high because so many examinees succeed on it
    Show answer

    Answer: C — The item is very easy and contributes little to discriminating between stronger and weaker examinees

    A difficulty index of 0.97 means 97% answered correctly, so the item is very easy and separates almost nobody; an item passed by nearly everyone carries little information. A high p means easy, not difficult; it is far from the ideal of about 0.5; and near-universal success actually drives discrimination toward zero, not up.

  3. During item analysis, one item shows a strongly negative discrimination index: low-scoring examinees pass it more often than high-scoring ones. What is the most reasonable conclusion?

    1. The item is of ideal quality, since any non-zero discrimination index shows the item is working as intended
    2. The item is likely flawed or mis-keyed, because the better examinees should do better, not worse, on a sound item
    3. The item is simply very difficult, and difficulty alone always produces a negative discrimination index
    4. The item should be kept because negative discrimination makes the overall test score more reliable
    Show answer

    Answer: B — The item is likely flawed or mis-keyed, because the better examinees should do better, not worse, on a sound item

    Negative discrimination means the item works against the total score — weaker examinees pass it more than stronger ones — which usually signals a flawed item or a wrong key. A negative index is a danger sign, not ideal; difficulty by itself does not force negative discrimination; and a negatively discriminating item lowers, not raises, reliability.

  4. A test developer keeps an item on a depression scale solely because, in trials, it reliably separated a clinically depressed group from a non-depressed group — even though the item’s content seems unrelated to mood. Which test-construction strategy is this?

    1. Content sampling from a blueprint, in which the table of specifications fixes how many items cover each area
    2. Rational content-based construction, in which items are kept because their meaning logically fits the construct
    3. Factor-analytic construction, in which items are grouped by the latent factors revealed by their inter-correlations
    4. Empirical criterion-keying, in which an item is kept only if it actually distinguishes the relevant groups
    Show answer

    Answer: D — Empirical criterion-keying, in which an item is kept only if it actually distinguishes the relevant groups

    Retaining an item purely for its power to separate known groups, regardless of its apparent meaning, is empirical criterion-keying — the logic behind the MMPI. Rational construction keeps items for their logical content; factor-analytic construction groups items by latent factors; and blueprint sampling allocates items by content coverage, none of which ignores content the way criterion-keying does.

  5. A researcher gives the same anxiety questionnaire to the same group twice, four weeks apart, and correlates the two sets of scores. Which type of reliability is being estimated?

    1. Internal-consistency reliability, which is estimated from how the items within a single administration relate
    2. Test–retest reliability, the coefficient of stability that correlates the same test taken on two occasions
    3. Parallel-forms reliability, the coefficient of equivalence between two separately built equivalent forms
    4. Inter-rater reliability, the degree of agreement between two independent judges scoring the same responses
    Show answer

    Answer: B — Test–retest reliability, the coefficient of stability that correlates the same test taken on two occasions

    Administering the same test twice and correlating the scores estimates test–retest reliability, the coefficient of stability over time. Internal consistency needs only one administration; parallel forms needs two different equivalent forms; and inter-rater reliability concerns agreement between scorers, not between occasions.

  6. A university wants evidence that scores on its new entrance test actually forecast students’ first-year grades, which will only be available a year later. Which kind of validity evidence is it seeking?

    1. Predictive criterion-related validity, because the criterion (first-year grades) is measured at a later time
    2. Content validity, because the concern is whether the test items cover the full syllabus domain they claim
    3. Concurrent criterion-related validity, because the test and its criterion are gathered at the same single sitting
    4. Face validity, because the concern is only whether the test looks appropriate to the applicants who take it
    Show answer

    Answer: A — Predictive criterion-related validity, because the criterion (first-year grades) is measured at a later time

    Correlating a test now with a criterion gathered later is predictive criterion-related validity. Content validity concerns domain coverage, not prediction; concurrent validity measures the criterion at the same time, which is explicitly not the case here; and face validity is only about surface appearance.

  7. A scale has a reliability coefficient of 0.64. Assuming the external criterion is measured perfectly reliably, what is the maximum possible value its validity coefficient (correlation with that criterion) can take?

    1. 1.0, because validity and reliability are independent, so perfect validity is possible at any reliability
    2. 0.64, because a test’s validity coefficient can never exceed its own reliability coefficient numerically
    3. 0.8, because the maximum validity coefficient equals the square root of the reliability coefficient
    4. 0.41, because the maximum validity coefficient equals the square of the reliability coefficient
    Show answer

    Answer: C — 0.8, because the maximum validity coefficient equals the square root of the reliability coefficient

    The ceiling on a validity coefficient is the square root of the reliability: √0.64 = 0.8. Error in the scores limits how strongly they can correlate with anything. Taking the reliability itself (0.64), assuming independence (1.0), or squaring the reliability (0.41) all misstate the reliability–validity relationship.

  8. A school psychologist uses an interest inventory to help a student match possible careers to the student’s own pattern of likes and dislikes. This is an application of testing in which setting?

    1. Developmental screening, where the main purpose is to detect delays in growth early across the life span
    2. Clinical assessment, where the main purpose is the diagnosis of a psychological disorder and treatment planning
    3. Organizational selection, where the main purpose is to choose among external applicants for a particular paid job
    4. Counselling and guidance, where interest, aptitude and values measures help match a person to education and career
    Show answer

    Answer: D — Counselling and guidance, where interest, aptitude and values measures help match a person to education and career

    Matching a person’s interests to suitable careers is the core use of testing in counselling and guidance. Clinical assessment targets diagnosis, organizational selection targets hiring decisions, and developmental screening targets early detection of delays — none of which describes career matching from an interest inventory.

  9. Which of the following statements about scales and levels of measurement are correct?

    1. A ratio scale has a true zero point, so ratio comparisons between scores are meaningful
    2. A nominal scale only classifies into unordered categories and does not rank them
    3. An ordinal scale guarantees that the numerical gaps between successive ranks are exactly equal
    4. Class rank is ordinal-level data, because it preserves order but not the size of the gaps between ranks
    Show answer

    Answer: A — A ratio scale has a true zero point, so ratio comparisons between scores are meaningful; B — A nominal scale only classifies into unordered categories and does not rank them; D — Class rank is ordinal-level data, because it preserves order but not the size of the gaps between ranks

    A ratio scale’s true zero allows ratio statements, a nominal scale only names unordered categories, and class rank is ordinal. The ordinal claim that gaps are guaranteed equal is false: ordinal scales preserve rank order but do not guarantee equal gaps between ranks — that is exactly what distinguishes them from interval scales.

  10. Which of the following are genuine steps or features in sound psychological test construction?

    1. Preparing a test blueprint (table of specifications) that fixes how many items cover each content area
    2. Trying the item pool out on a representative sample before finalising the test
    3. Using item analysis to retain items with sound difficulty and positive discrimination
    4. Discarding every item that fewer than 100% of examinees answer correctly to keep the test easy
    Show answer

    Answer: A — Preparing a test blueprint (table of specifications) that fixes how many items cover each content area; B — Trying the item pool out on a representative sample before finalising the test; C — Using item analysis to retain items with sound difficulty and positive discrimination

    A blueprint, a tryout on a representative sample, and item analysis are all real steps in test construction. Discarding every item not answered correctly by 100% would leave only trivially easy items that discriminate nobody, which is the opposite of good practice.

  11. Which of the following statements about reliability, validity and norms are correct?

    1. Convergent evidence shows a test agrees with other measures of the same construct, while discriminant evidence shows it does not agree with measures of different constructs
    2. A test can be highly reliable yet invalid for a particular purpose
    3. Norms are interpretable even when the standardization sample does not represent the people being tested
    4. A T score of 50 corresponds to the mean of the standardization sample
    Show answer

    Answer: A — Convergent evidence shows a test agrees with other measures of the same construct, while discriminant evidence shows it does not agree with measures of different constructs; B — A test can be highly reliable yet invalid for a particular purpose; D — A T score of 50 corresponds to the mean of the standardization sample

    Convergent and discriminant evidence are correctly defined, a reliable test can still be invalid for a given use (reliability is necessary but not sufficient), and T = 50 is the mean (T = 50 + 10z). Norms from an unrepresentative sample are misleading, not interpretable, so that statement is false.

  12. Which of the following correctly match a testing application to its setting?

    1. Using an assessment centre and selection battery to choose candidates for a job is an organizational application
    2. Using a developmental scale to track milestones and flag delays in young children is a developmental application
    3. Using an intelligence scale and projective techniques to help diagnose a disorder is a clinical application
    4. Validity established for diagnosing a disorder automatically transfers to selecting employees for a job
    Show answer

    Answer: A — Using an assessment centre and selection battery to choose candidates for a job is an organizational application; B — Using a developmental scale to track milestones and flag delays in young children is a developmental application; C — Using an intelligence scale and projective techniques to help diagnose a disorder is a clinical application

    The organizational, developmental and clinical matches are all correct. The claim that validity transfers automatically from diagnosis to employee selection is false: validity is specific to a use, so it does not transfer without fresh evidence.

  13. A test is split into two halves and the correlation between the two half-scores is 0.60. Using the Spearman–Brown formula r_full = 2r / (1 + r), estimate the reliability of the full-length test. Give the answer to two decimal places.

    Numerical answer — type the value.

    Show answer

    Answer: 0.75

    r_full = 2(0.60) / (1 + 0.60) = 1.20 / 1.60 = 0.75. The correction is needed because each half is only half as long, so the raw half-test correlation understates the whole test’s reliability.

  14. A test has a standard deviation of SD = 10 and a reliability coefficient of r = 0.75. Compute the standard error of measurement using SEM = SD√(1 − r). Give the answer as a plain number.

    Numerical answer — type the value.

    Show answer

    Answer: 5

    SEM = SD√(1 − r) = 10 × √(1 − 0.75) = 10 × √0.25 = 10 × 0.5 = 5. A higher reliability would give a smaller SEM and a tighter band around the true score.

  15. On a multiple-choice item, 150 of the 250 examinees answer it correctly. Compute the item-difficulty index p (proportion correct). Give the answer to one decimal place.

    Numerical answer — type the value.

    Show answer

    Answer: 0.6

    p = number correct / total = 150 / 250 = 0.6. A higher p means an easier item; this item is moderately easy.

  16. A test’s validity coefficient against a job-performance criterion is r = 0.50. Compute the proportion of variance in the criterion that is predictable from the test, which is r squared. Give the answer to two decimal places.

    Numerical answer — type the value.

    Show answer

    Answer: 0.25

    The proportion of criterion variance accounted for is r² = 0.50² = 0.25, so a quarter of the differences in job performance are predictable from the test and three-quarters are not.