Measuring shared ideas of identity with consensus analysis

Ask people who share a heritage what makes someone one of them, and the answers tend to be thin: a dish, a song, a stubborn streak. Tell the same people about a host who keeps refilling a guest's plate after three polite refusals, and many react at once, some recognizing their own people, others insisting their family never did that. Are those reactions private opinions, or does a community share a model of the kind of person its members are supposed to be? Cultural consensus analysis can answer that, and can also show where the sharing stops.

Personhood as a cultural domain

Anthropologists call these shared expectations a cultural model of personhood: ideas about how a proper person feels, speaks and acts, such as how openly to show emotion, how far to defer to rank, or when to put the group before oneself. Consensus analysis treats any body of shared knowledge as a domain and asks how strongly people agree about it. The mechanics are set out in our explainer on cultural consensus analysis. This note asks what changes when the domain is identity.

Two things change. The answer key is no longer a set of facts about the world but a shared image of a group, so there is nothing outside the data to check it against. And the population is defined by the very identity being studied, so the choice of who to ask shapes what you find.

Designing the study

What agreement and competence mean here

In a test of factual knowledge, competence is the probability that a person knows the right answer. In a study of identity, the key is the community's shared picture of itself, so competence measures how closely one person's judgments track that picture. It is not a score for authenticity. A low score might belong to a newcomer, to someone whose family had its own customs, or to someone who knows the stereotype perfectly well and refuses to endorse it.

The key needs the same care. If a community agrees that its members are modest, that is a finding about the shared model, not evidence that anyone behaves modestly. Nor does agreement on what is typical say whether people approve of it; that is the evaluative task's job.

A worked example: one culture or two?

Imagine six adults who all trace their families to the same group of nineteenth-century migrants. Three of them (A, B, C) live in Town 1, which has an active heritage society. Three (D, E, F) live in Town 2, which has none. Each hears the same twelve short scenarios and answers yes or no to one question: is this typical of people like us? The data are invented for teaching.

Count, for each pair, the items on which the two gave the same answer:

Correct for lucky guesses as in the explainer. With twelve yes/no items, M* = 2 × (matches/12) − 1, which simplifies to (matches − 6)/6. For A and C that is (10 − 6)/6 = 0.67; for B and E it is (6 − 6)/6 = 0. A least-squares fit of one factor to all fifteen corrected values gives these competences:

Every value is positive, and the ratio of the first eigenvalue to the second comes out at about 3.3 (computed with each person's fitted D² on the diagonal), above the usual rule of thumb of 3. On the standard reading, these six people share one culture.

What the first factor leaves over

The one-culture model predicts that the corrected agreement between two people equals the product of their competences. Whatever is left over, observed minus predicted, is called residual agreement. For A and C: 0.67 − (0.79 × 0.44) = 0.67 − 0.35 = 0.32. For B and E: 0 − (0.61 × 0.45) = −0.27. All fifteen pairs:

The six pairs inside a town average +0.17. The nine pairs across towns average −0.12, and only one of them (C and E) is above zero. People agree with their own town more than a single shared model allows. A second factor agrees: its loadings are positive for A, B and C and negative for D, E and F.

Now look at items 10 and 11, where all of Town 1 said yes and all of Town 2 said no. Weighting votes as in the explainer, a person with competence D multiplies the odds of "yes" by (1 + D)/(1 − D) when saying yes and divides by it when saying no. The multipliers are 8.52 (A), 4.13 (B), 2.57 (C), 12.33 (D), 2.64 (E) and 4.41 (F), so

odds(yes) = (8.52 × 4.13 × 2.57) / (12.33 × 2.64 × 4.41) ≈ 90.4 / 143.6 ≈ 0.63

which is a probability of about 0.39. The pooled key leans toward no, not because no is the community's answer but because Town 2 happens to contain the two highest scorers. Analyze each town on its own and each gets a confident answer, the opposite of the other's. That is a subculture, and the eigenvalue ratio alone would have hidden it.

One caution cuts the other way. Six people can be split into two threes in only ten ways, and although the town split is the sharpest of them, a permutation test (worked by hand in our statistics starter) can only say that a split this sharp would arise by chance about one time in ten. Six informants can suggest a subculture; they cannot establish one.

How the Welsh-American study approached it

Douglas Caulkins, Molly Offer-Westort and Carol Trosset (2005) used this kind of design with people of Welsh descent in two American settings, a socially visible community in the Slate Valley on the Vermont and New York border and a less visible one in Iowa, set beside an earlier sample from Wales. Informants rated 21 short scenarios, built around traits earlier fieldwork had tied to Welsh personhood plus a few American ones, in three separate tasks: how Welsh, how American and how desirable each behavior seemed. Ratings call for the informal version of the method, and the keys from each place were then correlated with one another. The broad result was convergence: the three populations largely shared an image of Welshness, the American sites agreed about Americanness, and desirability sat nearer Welshness than Americanness. Agreement on Welshness was weakest in Iowa.

Pitfalls

• Pooling groups and reading the pooled key as everyone's, as items 10 and 11 show.
• Treating a ratio of 3 as a pass mark. It summarizes the whole matrix and can clear the line with a subculture inside.
• Dropping disagreeing informants until the ratio clears the threshold. If you must, report both versions; the test has become partly circular.
• Confusing stereotypes with schemas. People can agree on a cliché they would never use to judge a real neighbor.

Where the method went next

Caulkins and Susan Hyatt (1999) proposed a typology for low-agreement domains: weak agreement, turbulent, subcultural and contested. Daniel Hruschka and colleagues (2008) tackled more than one answer key directly, finding two cultural models of postpartum hemorrhage in rural Bangladesh, one tied to traditional birth attendants and one to professionally trained ones. Royce Anders and William Batchelder (2012) built multiple consensus truths into the formal model itself. William Dressler, Mauro Balieiro and José Ernesto dos Santos (2015), repeating consensus analyses in Ribeirão Preto, Brazil, after ten years, found significant residual agreement in all four domains and linked the second factor in one of them, lifestyle, to the time period. The research areas overview places this work among the other cognitive approaches.

For a student planning an identity study, the lesson is practical: decide before fieldwork which subgroup comparison matters, record the variable that defines it for every informant, and report the residual agreement table next to the eigenvalue ratio.

Sources and further reading

• Caulkins, D. Douglas, Molly Offer-Westort and Carol Trosset. 2005. "Perceiving ethnic differences: consensus analysis and personhood in Welsh-American populations." Mathematical Anthropology and Cultural Theory 1(4). https://escholarship.org/uc/item/05c6f2p5
• Romney, A. Kimball, Susan C. Weller and William H. Batchelder. 1986. "Culture as consensus: a theory of culture and informant accuracy." American Anthropologist 88(2): 313-338. https://doi.org/10.1525/aa.1986.88.2.02a00020
• Trosset, Carol, and D. Douglas Caulkins. 2001. "Triangulation and confirmation in the study of Welsh concepts of personhood." Journal of Anthropological Research 57(1): 61-81. https://doi.org/10.1086/jar.57.1.3630798
• Caulkins, D. Douglas, and Susan B. Hyatt. 1999. "Using consensus analysis to measure cultural diversity in organizations and social movements." Field Methods 11(1): 5-26. https://doi.org/10.1177/1525822X9901100102
• Hruschka, Daniel J., Lynn M. Sibley, Nahid Kalim and Joyce K. Edmonds. 2008. "When there is more than one answer key: cultural theories of postpartum hemorrhage in Matlab, Bangladesh." Field Methods 20(4): 315-337. https://doi.org/10.1177/1525822X08321315
• Anders, Royce, and William H. Batchelder. 2012. "Cultural consensus theory for multiple consensus truths." Journal of Mathematical Psychology 56(6): 452-469. https://doi.org/10.1016/j.jmp.2013.01.004
• Dressler, William W., Mauro C. Balieiro and José Ernesto dos Santos. 2015. "Finding culture change in the second factor: stability and change in cultural consensus and residual agreement." Field Methods 27(1): 22-38. https://doi.org/10.1177/1525822X14542755