Cultural consensus analysis, explained with five informants

Suppose you ask five people the same twenty true-or-false questions about something you do not understand yourself: the rules of a fishing calendar, say, or which illnesses are thought to be catching. Their answers differ. Who knows the domain best? And what are the right answers, in the sense of the answers this community shares?

Cultural consensus analysis answers both questions at once, using nothing but the pattern of agreement among the people you asked. A. Kimball Romney, Susan Weller and William Batchelder set out the model in American Anthropologist in 1986, framing it around an old fieldwork worry: "How do we know when we know?" Batchelder and Romney gave the fuller statistical treatment two years later under a title that sums up the trick: test theory without an answer key. This piece walks through the logic, then works a small example by hand.

The idea in one paragraph

If there is a shared body of knowledge, people who know an answer give the same answer, while people who do not know must guess, and guesses agree only by chance. So the more two people agree beyond chance, the more both of them probably know. Agreement becomes a measuring instrument. Once you have estimated how much each person knows, you can weight their answers accordingly and reconstruct the answer key from the answers alone.

Three assumptions

The formal model rests on three assumptions, and it is worth learning them before the arithmetic, because most failed analyses fail here.

1. Common truth. There is one answer key that applies to everyone you asked. Everyone is drawing on the same cultural domain, even if some know it better than others.
2. Local independence. Each person answers on their own. Any correlation between two people's answers comes only from their shared link to the key, not from talking to each other.
3. Homogeneity of items. Each person has one level of knowledge, called competence, that applies to every question. In effect all questions are equally hard.

The third is the strongest, and the 1986 paper says so. Its authors sketch a weaker version, in which people who do better on some questions in a domain also do better on others, and report that the analysis has worked well in practice even with the strict form. Group interviews (which break the second assumption) and samples that mix two different communities (which break the first) are another matter.

From agreement to competence

Write D for a person's competence: the probability that they actually know the answer to a question. If they do not know, they guess, and with L possible answers a guess is right with probability 1/L. So the chance of a correct answer is

Pr(correct) = D + (1 − D)/L

Now take two people, i and j. They match on a question if both know it, if one knows and the other guesses right, or if both guess the same way. Romney and colleagues show that these cases add up to

Pr(match) = Di × Dj + (1 − Di × Dj)/L

Solve for the product and you get a version of agreement corrected for guessing. For true/false questions L = 2, and the corrected agreement between i and j is simply

M*ij = 2 × (proportion of matches) − 1

which estimates Di × Dj. With N informants there are N(N − 1)/2 pairs, so five people give ten equations in five unknowns. The usual solution is a least-squares fit of one factor to the matrix of corrected agreements; each person's loading on that first factor is their estimated competence.

A worked example with five informants

Imagine five residents of a lagoon fishing village, labeled A to E, each answering the same twenty true/false statements about the local fishing calendar. The data are invented for teaching, and the researcher does not know the answers.

Counting agreement

For each pair, count the items on which the two gave the same answer. A and B differ only on items 9, 13 and 15, so they match on 17 of 20. Doing the same for all ten pairs gives:

Taking out lucky guesses

Two people who both guessed every question would still match on about half of them. The correction removes that. With 20 items, M* = 2 × (matches/20) − 1, which simplifies to (matches − 10)/10. So A and B score (17 − 10)/10 = 0.7, and D and E score (11 − 10)/10 = 0.1. All ten values:

Every value is positive, which is the first thing to check. Negative agreement after correction would mean two people disagree more than guessers would, and the model has no room for that.

From agreement to competence

If the model holds, M*AB ≈ DA × DB, M*AC ≈ DA × DC and M*BC ≈ DB × DC. Multiply the first two and divide by the third, and everything cancels except DA squared. That gives a hand estimate from any three people:

DA ≈ √(0.7 × 0.6 / 0.5) = √0.84 ≈ 0.92

Try it with B and D instead and you get √(0.7 × 0.5 / 0.4) = √0.875 ≈ 0.94. With C and D, √(0.6 × 0.5 / 0.5) = √0.6 ≈ 0.77. The triads disagree because twenty questions is a small sample of anyone's knowledge. The least-squares fit that consensus software performs uses all ten pairs at once and settles the differences. For these data it gives:

The right-hand column is a cheat. Because we invented the data, we know the key, and the right-hand column scores everyone against it. In real fieldwork that column does not exist. The point of the comparison is that the left-hand column, computed without the key, lands close to it. The 1986 paper made the same comparison with real data, a general-knowledge test given to 41 students, and found the two sets of estimates correlated at .93.

You can also check the fit directly. The model predicts AB agreement of 0.87 × 0.76 ≈ 0.66 against an observed 0.70, and BE agreement of 0.76 × 0.28 ≈ 0.21 against an observed 0.30. No pair in this example misses by more than 0.09.

Recovering the key

Now weight the votes. If the true answer is "true", a person with competence D says "true" with probability D + (1 − D)/2 = (1 + D)/2. If the true answer is "false", they say "true" with probability (1 − D)/2. So each "true" from that person multiplies the odds in favor of "true" by (1 + D)/(1 − D), and each "false" divides by it. Start at even odds and assume nobody leans toward one answer when guessing.

Rounding the competences to one decimal place (0.9, 0.8, 0.7, 0.6, 0.3), the multipliers are 19, 9, 5.67, 4 and 1.86.

Item 7 is the interesting one. A and B said true; C, D and E said false. A simple majority says false. The weighted odds say otherwise:

odds(true) = (19 × 9) / (5.67 × 4 × 1.86) = 171 / 42.1 ≈ 4.06

Pr(true) = 4.06 / 5.06 ≈ 0.80

Two people who know a lot outweigh three who know less. On item 9 the reverse happens: A, the strongest informant, said false while everyone else said true, and the odds come out at (9 × 5.67 × 4 × 1.86) / 19 ≈ 19.9, a probability of about 0.95 for true. Being the best informant does not make you right every time.

Run all twenty items this way and the inferred answers match the key we used to build the data on every item, with item 7 the only one below 0.95.

How many informants do you need?

Fewer than most survey methods, if agreement is high. The 1986 paper derived a table of minimum sample sizes for true/false questions, which depends on three choices: the average competence of the people you ask, the confidence level you require before calling an answer settled, and the share of questions you want settled. A few cells from it:

At an average competence of 0.5, the strictest settings need well over 30 people. Our five informants, with an average competence near 0.64, fall short of these targets, so it is no surprise that one item stayed below 0.95. On the question side, Weller (2007) recommends at least twenty questions so that each pair's agreement is estimated with some stability.

Does the model fit?

The assumptions imply that one factor, with every loading positive, should account for the structure of the agreement matrix. The 1986 paper asks that the first factor be several times larger than the second; in its general-knowledge example the first two eigenvalues were 13.95 and 2.71, a ratio above five. Later guides turned this into rules of thumb, commonly a ratio of about 3 to 1, sometimes higher. Negative competence estimates are a warning sign too.

The same paper shows what failure looks like. It reanalyzed data on 27 illnesses from an urban community in Guatemala. Asked which illnesses are contagious, the informants fit a single factor with high average competence. Asked which illnesses need hot or cold remedies, the informants produced two factors of similar size and many negative loadings. The conclusion was that there was no single shared answer key for hot and cold, a result about culture in its own right.

Questions students ask

Does consensus mean the answer is true?

No. The model estimates the answer this group shares. A community can agree on something that is factually wrong, and a lone expert can be right against everyone. Consensus analysis tells you about shared knowledge, which is what a cultural description needs, but it is not a test of fact.

What if the sample mixes two groups?

Then the first assumption fails, and the fit statistics usually show it: a weak first factor, a strong second one, or negative loadings for one cluster of people. Splitting the sample by a variable you can justify in advance, and analyzing each part separately, is the common response.

Can I use ratings or rankings?

Not with the formal model, which needs categorical answers. Weller (2007) describes an informal version that works on correlations between people's ratings or rankings and makes fewer assumptions. Later work has also recast the formal model in Bayesian terms (Oravecz, Vandekerckhove and Batchelder 2014).

What should I report?

The number of informants and questions, how answers were coded, the ratio of the first to the second eigenvalue, the range and mean of competence, any negative values, and which items stayed below your chosen confidence level. Our statistics starter places consensus analysis among the other tests that come up in anthropology, and the research areas overview shows where cognitive approaches like this one sit in the wider field. More methods pieces are collected under Methods & Tools.

Sources and further reading

• Romney, A. K., Weller, S. C., and Batchelder, W. H. (1986). Culture as consensus: a theory of culture and informant accuracy. American Anthropologist 88(2): 313-338. doi.org/10.1525/aa.1986.88.2.02a00020
• Batchelder, W. H., and Romney, A. K. (1988). Test theory without an answer key. Psychometrika 53(1): 71-92. doi.org/10.1007/BF02294195
• Weller, S. C. (2007). Cultural consensus theory: applications and frequently asked questions. Field Methods 19(4): 339-368. doi.org/10.1177/1525822X07303502
• Oravecz, Z., Vandekerckhove, J., and Batchelder, W. H. (2014). Bayesian cultural consensus theory. Field Methods 26(3): 207-222. doi.org/10.1177/1525822X13520280
• Gatewood, J. B. (2014). Consensus analysis's undiscussed sampling issue: how many questions are needed to establish credible assessments of respondent-by-respondent similarity? Conference slides, Lehigh University. lehigh.edu/~jbg1/N-of-questions.pdf