Methods of Analysis: Data Collection, Quantitative and Qualitative Analysis, Ethics and Corpora

This chapter answers Section XH3.6, “Methods of Analysis”, of XH3 Linguistics, one of the six optional sections of the Humanities and Social Sciences (XH) paper, chosen at the time of application. It teaches how linguists gather data — fieldwork, elicitation from speakers, controlled experiments and sampling — how they analyse it both quantitatively (basic statistics such as frequency, the type-token ratio and significance testing) and qualitatively (transcription, coding, discourse and conversation analysis), the ethical duties owed to the speakers who supply the data, and the development, annotation, types and analysis of corpora. The few numericals here are corpus counts: type-token ratios, normalized frequencies and type counts.

1. Data collection: fieldwork and elicitation

Linguistic fieldwork is the study of a language in its own speech community, usually by working with native-speaker consultants (the older word “informant” is now avoided as it understates their expertise). The fieldworker records both naturalistic data (spontaneous conversation, narratives, texts) and elicited data gathered by prompting for specific forms. Natural data show what speakers actually do but may never contain the rare form a grammar needs; elicited data target that form but can be distorted by the task. Labov’s observer’s paradox names the core tension: the goal is to find out how people speak when they are not being observed, yet the only way to collect the speech is to observe it.

  • Translation / direct elicitation: the researcher gives a word or sentence in a contact language and asks for the equivalent. Fast, but it imports the structure of the contact language and misses forms with no neat translation.
  • Substitution frames: a fixed sentence frame is held constant while one slot is varied, to map a paradigm (for example all person–number forms of a verb) systematically.
  • Grammaticality (acceptability) judgements: the consultant is asked whether a constructed sentence is possible in the language. They probe knowledge directly but depend on the speaker’s metalinguistic intuition and on how the task is framed.
  • Stimulus-based elicitation: pictures, video or wordless storybooks (such as the “frog story” narrative task) prompt connected speech without a translation prompt; colour chips, kinship grids and similar tools elicit semantic domains.
⚠️ Elicited is not the same as natural
A grammaticality judgement or a translation tells you what a speaker will accept or construct under attention, which is not always what they produce in running speech. Careful work triangulates: it checks an elicited form against spontaneous data, and never rests a claim about ordinary usage on judgement data alone.

2. Experiments and sampling

An experiment manipulates an independent variable and measures its effect on a dependent variable while holding other factors (confounds) constant, so that a difference in the result can be attributed to the manipulation. Common psycholinguistic paradigms include lexical decision and self-paced reading (measuring reaction time), eye-tracking during reading, and graded acceptability judgements on a rating scale. The strength of an experiment is control; its risk is that the controlled, artificial setting may not reflect natural language use, so results need checking against corpus or field data.

No study can record a whole population, so it works from a sample, and the sample’s quality decides how far findings generalize. Probability sampling gives every member a known, non-zero chance of selection and alone supports statistical generalization; non-probability sampling does not, and risks bias.

  • Random (simple probability) sampling: every member has an equal chance; systematic sampling takes every k-th member from a list.
  • Stratified sampling: the population is divided into strata (age, gender, social class, region) and sampled within each, so a sociolinguistic study can compare groups fairly.
  • Convenience sampling takes whoever is at hand (often an issue with student participants); snowball sampling reaches hidden or close-knit communities by asking each participant to introduce others. Both are non-probability methods, convenient but not representative.
🎯 Why representativeness matters
A sample that over-represents one group answers a narrower question than intended: a study of “teenage speech” from one school describes that school, not all teenagers. Larger samples reduce chance error but cannot repair a biased selection frame, so who is left out matters as much as how many are in.

3. Quantitative analysis: basic statistics

Quantitative analysis begins with frequency. A running word is a token; a distinct word form is a type, so “the cat saw the dog” has 5 tokens but 4 types (“the” repeats). Because larger texts hold more tokens, raw counts from texts of different sizes are not comparable; we report normalized frequency = (raw count ÷ total tokens) × a base, usually per million words. The type-token ratio (TTR) = types ÷ tokens gauges lexical variety, but it falls as a text grows (common words keep recurring), so a TTR is only comparable between texts of equal length or after standardization.

Beyond counts, descriptive statistics summarize data: the mean, median and mode locate the centre, while the range and standard deviation describe spread. Inferential statistics then ask whether an observed pattern is likely to be real or a product of chance. For a link between two categorical variables (say, use of a variant by gender) the chi-square test is standard; for a difference between two group means, a t-test. A result is called statistically significant when its p-value falls below a stated threshold (commonly α = 0.05), which is why the significance level must always be stated.

Common quantitative measures in linguistics
MeasureWhat it capturesTypical use
Normalized frequency (per million)Rate of a form, independent of text sizeComparing word frequency across corpora
Type-token ratioLexical variety (length-sensitive)Richness of vocabulary in equal samples
Mean and standard deviationCentre and spread of a measurementAverage sentence length, reaction times
Chi-squareAssociation between categorical variablesVariant choice by social group
⚠️ A correlation is not a cause, and a long TTR is not more variety
Two variables rising together may both follow a third. And because TTR drops with length, a shorter text will usually show a higher TTR than a longer one even when the longer text is lexically richer; compare only equal lengths, or use a standardized TTR.

4. Qualitative analysis

Qualitative analysis seeks the meaning and structure of language in context rather than counts. It begins with transcription, itself a theory-laden choice: a plain orthographic transcript captures words, a phonetic one captures pronunciation, and a conversation-analytic transcript (Jefferson conventions) records overlaps, pauses, stress and intonation because those carry interactional meaning. The transcript is then coded — passages are labelled with recurring categories that the analyst builds up and refines.

  • Thematic / content analysis: identifying recurring themes across a body of texts or interviews, sometimes counting their occurrence.
  • Grounded theory: building theory up from the data through successive coding rather than testing a hypothesis fixed in advance.
  • Discourse analysis studies how meaning, stance and power are built across stretches of text; conversation analysis (CA) examines the fine machinery of talk — turn-taking, adjacency pairs (such as question–answer), and repair.
🎯 Depth, triangulation and reflexivity
Qualitative work trades statistical generalization for depth and context. It guards its claims by triangulation (combining several methods or data sources so findings converge) and reflexivity (the analyst accounting for how their own presence and assumptions shaped the data). Quantitative and qualitative methods are often combined in a mixed-methods design, each answering what the other cannot.

5. Ethics of working with speakers

Because linguistic data come from people, research carries duties to them. Informed consent means participants understand the purpose, procedure and uses of the study and agree freely, with the right to withdraw at any time. Anonymity and confidentiality protect identity: names and identifying details are removed or pseudonymized and recordings stored securely. Most institutions require a study to pass an ethics review before data collection begins.

  • Intellectual property and ownership: the language and the recordings belong, morally and often legally, to the community and the speakers, not to the researcher; access and reuse should be agreed, not assumed.
  • Benefit and reciprocity: research should give back — training, materials, documentation the community can use — rather than merely extract data; participatory models involve speakers as partners.
  • Care with vulnerable speakers and endangered languages: last speakers, children and marginalized communities need particular sensitivity, and consultants deserve fair acknowledgement and, where appropriate, credit and compensation.

6. Corpora: development, annotation, types and analysis

A corpus is a large, principled, machine-readable collection of texts assembled to represent a language or variety. Development turns on a sampling frame: what text types (registers, genres, speech vs writing) to include and in what proportion, so the corpus is balanced and representative rather than merely large. Each text carries metadata (author, date, register, source) and collecting it requires attention to copyright and permissions. The Brown Corpus (about a million words of written American English) was an early landmark; the British National Corpus runs to about 100 million words of written and spoken English.

  • By medium and scope: written vs spoken (spoken needs transcription); general / reference corpora (meant to stand for the whole language) vs specialized ones (a single register or domain).
  • By time: synchronic (one period) vs diachronic / historical (across periods); a monitor corpus keeps growing so new words and changes can be tracked.
  • By language relation: a parallel corpus holds the same texts in two or more languages, aligned as translations; a comparable corpus holds similar texts in different languages that are not translations of each other. A learner corpus collects second-language learners’ output.

Annotation adds interpretive layers to the raw text. After tokenization (splitting into tokens) and lemmatization (mapping word forms to a base lemma), the commonest layer is part-of-speech (POS) tagging; full syntactic annotation produces a treebank. Annotation can be manual, automatic or semi-automatic, and its quality is checked by inter-annotator agreement. Because every tag reflects an analytic decision, annotation is interpretation, not neutral fact.

Analysis reads the corpus with tools. A concordance shows every occurrence of a search word with its context, the keyword-in-context (KWIC) display, revealing patterns of use. Frequency and word lists rank forms; collocation finds words that co-occur more often than chance (such as “strong tea” but not “powerful tea”); keyword analysis compares a text’s frequencies against a reference corpus to find what is distinctive. n-grams (recurring sequences of n tokens) surface fixed phrases.

⚠️ Parallel versus comparable
A parallel corpus is made of translations of the same content, aligned sentence by sentence — the basis of translation and machine-translation work. A comparable corpus is made of independent texts on similar topics in different languages; it is not aligned and not translations. Treating one as the other invalidates any cross-linguistic equivalence drawn from it.

Key takeaways

  • Fieldwork uses native-speaker consultants and mixes naturalistic data with elicited data (translation, substitution frames, grammaticality judgements, stimulus tasks). Labov’s observer’s paradox is that observing speech changes it; elicited forms must be checked against spontaneous data.
  • Experiments manipulate an independent variable and control confounds; sampling may be probability (random, systematic, stratified) or non-probability (convenience, snowball). Only probability sampling supports statistical generalization, and representativeness matters more than sheer size.
  • Tokens are running words, types are distinct forms; normalized frequency = (raw ÷ total tokens) × base, and TTR = types ÷ tokens but falls with length. Chi-square tests categorical association, a t-test compares means, and significance needs a stated level such as α = 0.05.
  • Qualitative analysis transcribes, codes and interprets in context through thematic analysis, grounded theory, discourse analysis and conversation analysis, safeguarded by triangulation and reflexivity; mixed-methods designs combine the quantitative and qualitative.
  • Ethics requires informed consent with the right to withdraw, anonymity and secure storage, and an ethics review; the data and recordings belong to the community, research should give back through benefit and reciprocity, and vulnerable speakers need special care.
  • A corpus is a balanced, representative, machine-readable text collection; types include written/spoken, reference/specialized, synchronic/diachronic and monitor, parallel versus comparable, and learner. Annotation (tokenization, lemmatization, POS tagging, treebanks) is interpretation; analysis uses concordance/KWIC, frequency and word lists, collocation and keyword analysis.

Practice questions (16)

Attempt each one before opening the answer. Every explanation names the tempting wrong option as well as the right one, because that is where marks are lost.

  1. In sociolinguistic fieldwork, the “observer’s paradox” refers to which difficulty?

    1. Two trained observers will always transcribe the same recording in precisely the same way, with no divergence at all
    2. A sample large enough to support statistics is always far too large to transcribe accurately by hand
    3. We want to know how people speak when not observed, but we can only collect speech by observing it
    4. Speakers are unable to report their own grammaticality judgements with any reliability whatsoever
    Show answer

    Answer: C — We want to know how people speak when not observed, but we can only collect speech by observing it

    The observer’s paradox, named by Labov, is that the aim is to record unselfconscious, natural speech, but the act of recording makes speakers self-conscious and can shift their style. The other statements are false or unrelated: trained transcribers do differ, sample size is a separate issue, and speakers can give useful judgements.

  2. A field linguist shows a consultant a constructed sentence and asks, “Can you say this in your language, or does it sound wrong?” Which data-collection method is this?

    1. Participant observation of entirely spontaneous conversation as it naturally unfolds within the wider community
    2. A grammaticality (acceptability) judgement, which probes the speaker’s knowledge of what is possible
    3. A stimulus-based narrative task that uses a wordless picture storybook to prompt connected speech
    4. Stratified probability sampling of speakers drawn from across the different social groups
    Show answer

    Answer: B — A grammaticality (acceptability) judgement, which probes the speaker’s knowledge of what is possible

    Asking whether a given sentence is possible or sounds wrong is a grammaticality judgement, which taps metalinguistic knowledge directly. Participant observation and the narrative task both gather production rather than judgements, and stratified sampling is a way of choosing speakers, not of eliciting a form.

  3. A researcher studying a small, close-knit migrant community recruits a few speakers, then asks each to introduce friends and relatives who also take part. Which sampling method is in use?

    1. Simple random sampling, which gives every member of the community an equal and independent chance of selection
    2. Systematic sampling, which selects every k-th person from a complete numbered list of the population
    3. Stratified sampling, which divides the community into strata and then samples within each one
    4. Snowball sampling, which reaches hidden or close-knit groups through the participants’ own networks
    Show answer

    Answer: D — Snowball sampling, which reaches hidden or close-knit groups through the participants’ own networks

    Recruiting through participants’ own contacts is snowball sampling, a non-probability method well suited to hard-to-reach groups. Random and systematic sampling both need a complete list of the population, which such communities rarely allow; stratified sampling requires dividing a known population into strata first.

  4. Why do corpus linguists usually report normalized frequency (for example, per million words) rather than raw counts when comparing a word across two corpora of different sizes?

    1. Because a larger corpus holds more tokens overall, so raw counts are not comparable across sizes
    2. Because normalization by itself automatically proves the frequency difference is statistically significant
    3. Because raw counts simply cannot be computed once a corpus has been part-of-speech tagged throughout
    4. Because normalized frequency removes any need to tokenize the texts before counting them
    Show answer

    Answer: A — Because a larger corpus holds more tokens overall, so raw counts are not comparable across sizes

    A word will show more raw occurrences in a bigger corpus simply because there is more text, so raw counts from different-sized corpora cannot be compared; dividing by total tokens and scaling to a common base fixes this. Normalization is not a significance test, raw counts are always available, and the frequency itself is obtained precisely by counting tokens.

  5. A student computes the type-token ratio of a 500-word essay and of a 5,000-word essay by the same author and finds the short essay has the higher ratio. What is the soundest conclusion?

    1. The shorter essay definitely and conclusively has a far richer and more varied vocabulary than the longer one does
    2. The author’s active vocabulary must have measurably shrunk while writing the longer essay
    3. Little can be concluded, because TTR falls as texts lengthen, so unequal-length texts are not directly comparable
    4. The result must be a calculation error, since the TTR does not depend on text length at all
    Show answer

    Answer: C — Little can be concluded, because TTR falls as texts lengthen, so unequal-length texts are not directly comparable

    TTR systematically decreases with text length because high-frequency words keep recurring and add tokens without adding types; a shorter text therefore almost always shows a higher TTR regardless of true lexical richness. The right move is to compare equal-length samples or use a standardized TTR, so the richer-vocabulary and shrinking-vocabulary readings are both unwarranted, and the drop is expected, not an error.

  6. Conversation analysis typically studies which features of talk?

    1. The frequency of each phoneme averaged across a hundred-million-word reference corpus of speech
    2. Turn-taking, adjacency pairs and repair as the organized machinery of everyday interaction
    3. The reconstruction of a proto-language from the evidence of its several daughter languages
    4. The reaction times of readers to garden-path sentences measured in a laboratory setting
    Show answer

    Answer: B — Turn-taking, adjacency pairs and repair as the organized machinery of everyday interaction

    Conversation analysis examines the sequential organization of talk — how turns are taken, how utterances pair up (question–answer), and how speakers repair trouble. The other options describe corpus phonetic statistics, comparative reconstruction, and a psycholinguistic reading experiment respectively.

  7. In research methodology, “triangulation” most precisely means:

    1. Dividing every sample into exactly three equal strata before any analysis is carried out
    2. Repeating one single experiment exactly three times and then averaging the three results
    3. Interviewing each and every participant with a panel composed of exactly three separate interviewers each time
    4. Using more than one method or data source so that independent findings can corroborate one another
    Show answer

    Answer: D — Using more than one method or data source so that independent findings can corroborate one another

    Triangulation combines multiple methods, data sources or perspectives so that convergent evidence strengthens a conclusion; the “tri-” is metaphorical and implies no literal count of three. The other options wrongly take the word to mandate three strata, three repetitions or three interviewers.

  8. A researcher records speakers of an endangered language. Which practice best meets the ethical requirement of informed consent?

    1. Explaining the purpose and uses of the recordings in a form speakers understand, and honouring their right to withdraw
    2. Recording all the speakers first and then seeking their permission only if one of them later happens to object in some way
    3. Treating consent as unnecessary because an endangered language simply must be documented at any cost
    4. Publishing the speakers’ full names so that their valuable contribution is on permanent public record
    Show answer

    Answer: A — Explaining the purpose and uses of the recordings in a form speakers understand, and honouring their right to withdraw

    Informed consent requires that participants genuinely understand the purpose and uses of the work beforehand and agree freely, keeping the right to withdraw. Recording without prior permission violates it, the urgency of documentation never overrides consent, and publishing full names breaches confidentiality rather than satisfying consent.

  9. A concordance tool presents results in a KWIC display. What does KWIC show?

    1. A single overall count of exactly how many words the entire corpus happens to contain
    2. The family tree diagram showing how the corpus languages are genetically related to each other
    3. Each occurrence of the search word lined up with the text immediately around it
    4. A full phonetic transcription of every single sentence contained in the corpus
    Show answer

    Answer: C — Each occurrence of the search word lined up with the text immediately around it

    KWIC (keyword in context) lists every hit of the search term with a window of surrounding text, usually centred on the term, so patterns of usage and collocation stand out. It is not a single total, a genetic tree, or a phonetic transcript.

  10. A team wants to study how one and the same news report is rendered in English and in Hindi, sentence by sentence. Which resource do they need?

    1. A comparable corpus, which holds independent same-topic texts in each language that are not translations
    2. A parallel corpus, which holds the same texts in both languages aligned as translations
    3. A monitor corpus of English alone that simply keeps growing larger and larger over time
    4. A learner corpus recording Hindi speakers’ characteristic errors in their written English
    Show answer

    Answer: B — A parallel corpus, which holds the same texts in both languages aligned as translations

    Studying the same text translated across languages, aligned sentence by sentence, needs a parallel corpus. A comparable corpus holds same-topic texts that are not translations and cannot be aligned; a monolingual monitor corpus and a learner corpus do not pair the two renderings at all.

  11. Which of the following statements about data collection in linguistics are correct?

    1. Probability sampling, unlike convenience sampling, supports statistical generalization to the population
    2. Elicited judgements and spontaneous production can diverge, so one should not stand in for the other uncritically
    3. A translation task can import the structure of the contact language into the recorded data
    4. Increasing the sample size always removes any bias in how the sample was selected
    Show answer

    Answer: A — Probability sampling, unlike convenience sampling, supports statistical generalization to the population; B — Elicited judgements and spontaneous production can diverge, so one should not stand in for the other uncritically; C — A translation task can import the structure of the contact language into the recorded data

    Probability sampling is what licenses generalization, elicited and spontaneous data can genuinely differ, and translation prompts can carry over the contact language’s structure. The last claim is false: a bigger sample reduces random error but cannot correct a biased selection frame.

  12. Which of the following statements about corpora and their annotation are correct?

    1. A balanced, representative corpus is designed around a sampling frame of text types, not just made as large as possible
    2. Part-of-speech tagging and treebanking add interpretive layers and involve analytic decisions
    3. A monitor corpus is continually extended so that lexical change can be tracked over time
    4. A parallel corpus is simply any two corpora that happen to be the same number of words long
    Show answer

    Answer: A — A balanced, representative corpus is designed around a sampling frame of text types, not just made as large as possible; B — Part-of-speech tagging and treebanking add interpretive layers and involve analytic decisions; C — A monitor corpus is continually extended so that lexical change can be tracked over time

    Good corpus design rests on a sampling frame for balance and representativeness; POS tagging and treebanks are interpretive annotation layers; and a monitor corpus grows over time to track change. The last is wrong: a parallel corpus is defined by holding the same texts aligned across languages as translations, not by matching word counts.

  13. Which of the following are genuine ethical responsibilities when working with speakers?

    1. Obtaining informed consent and allowing participants to withdraw
    2. Protecting participants’ identity through anonymity and secure data storage
    3. Recognizing that the data and recordings belong to the community, and giving something back
    4. Publishing each speaker’s real name and address alongside their recordings
    Show answer

    Answer: A — Obtaining informed consent and allowing participants to withdraw; B — Protecting participants’ identity through anonymity and secure data storage; C — Recognizing that the data and recordings belong to the community, and giving something back

    Consent with the right to withdraw, protecting identity, and recognizing community ownership with reciprocity are all core duties. Publishing real names and addresses does the opposite — it breaches the confidentiality that ethical research is bound to protect.

  14. Compute the type-token ratio of the text “the big dog saw the small dog and the cat”, counting each word form as a type. Give the answer as a decimal to two places.

    Numerical answer — type the value.

    Show answer

    Answer: 0.70

    The tokens are: the, big, dog, saw, the, small, dog, and, the, cat — 10 tokens. The distinct types are: the, big, dog, saw, small, and, cat — 7 types (“the” occurs 3 times, “dog” twice). TTR = types ÷ tokens = 7 ÷ 10 = 0.70. The usual slip is to count repeated words as separate types.

  15. A word occurs 48 times in a corpus of 3,00,000 tokens. What is its normalized frequency per million words? Give the answer as a whole number.

    Numerical answer — type the value.

    Show answer

    Answer: 160

    Normalized frequency per million = (raw count ÷ total tokens) × 1,000,000 = (48 ÷ 300000) × 1000000 = 0.00016 × 1000000 = 160. So the word occurs at a rate of 160 per million words.

  16. A text contains 1,200 tokens and has a type-token ratio of 0.35. How many distinct types does it contain? Give the answer as a whole number.

    Numerical answer — type the value.

    Show answer

    Answer: 420

    TTR = types ÷ tokens, so types = TTR × tokens = 0.35 × 1200 = 420 distinct types. The trap is to multiply by the ratio’s reciprocal or to confuse types with tokens.