Galton's problem, or why cultures are not independent data points

In 1888 Edward Tylor stood up at the Anthropological Institute in London with a new kind of argument. He had spent years collecting the marriage and descent customs of between three and four hundred peoples, and instead of telling stories about them he had counted. Which customs turned up together more often than chance would predict? Francis Galton, the institute's president, was the first to respond. He praised the work and then asked a question that cross-cultural research has been answering ever since.

What Tylor counted

Tylor called the tendency of customs to travel together their "adhesions". His opening example was the rule, documented in many societies, that a husband and his wife's relatives avoid one another: they do not speak, look at each other or use each other's names. He set the cases of this avoidance against where couples lived after marriage and compared what he found with what chance alone would produce.

Avoidance was more common than expected where the husband lived among his in-laws, and less common where he did not. Tylor read this as a connection: the custom made sense as a way of handling an outsider living in the household. The method, if not the vocabulary, is the one a modern chi-square test uses: compute the counts you would expect if two traits were unrelated, then see how far the observed counts stray from them.

Galton's question

Galton, in remarks printed with the paper, wanted readers to be told how far the customs of the peoples being compared were independent of one another. Some, he pointed out, might have taken a custom from a common source, making them "duplicate copies of the same original". Each observation, he said, ought to be weighted, and a map showing where each custom was found would help.

Tylor accepted the point at once. He called it a difficulty ever present in such work and gave the Malay region as an example, where island groups differ enough to be classed separately yet clearly share an origin. His remedy was to count separate cases only where differences were well marked, to apply the method evenly across the world, and to trust that resemblances due to history would roughly cancel out. Another speaker, Professor Flower, raised a related worry: the method only works if the units being compared are of equivalent value.

By 1961, when Raoul Naroll published a paper titled "Two solutions to Galton's problem", the objection had a name. In modern language, Galton's problem is that the cases in a cross-cultural sample are not statistically independent. Societies resemble their neighbors because they borrow from each other, and they resemble their linguistic relatives because they inherited traits from a common ancestor. A correlation computed as if every society were a separate draw can count one historical event many times over.

A toy example: twenty societies, two traits

Here is an invented sample, built to show the problem in its simplest form. Twenty societies are coded for two traits: houses built on stilts, and inheritance that passes to the youngest son. There is no sensible functional story linking the two. That is deliberate.

The twenty societies come from three kinds of background. Eight belong to a North family, all descended from one ancestral society that had both traits. Eight belong to a South family, descended from an ancestor that had neither. Four are unrelated to the others and to each other.

In the North family one society dropped youngest-son inheritance but kept its stilts. In the South family one society took up youngest-son inheritance without building on stilts. Everything else in those families is simply what their ancestors had.

The naive count

Treat the twenty societies as twenty independent cases and the usual 2 × 2 table looks like this:

If the traits were unrelated, each cell should hold 10 × 10 / 20 = 5 societies. The chi-square statistic adds up (observed − expected)² / expected over the four cells:

χ² = (8 − 5)²/5 + (2 − 5)²/5 + (2 − 5)²/5 + (8 − 5)²/5 = 4 × 9/5 = 7.2

With one degree of freedom, 7.2 is past 6.63, the cutoff for p = 0.01. The phi coefficient, a correlation for 2 × 2 tables, is

φ = (8 × 8 − 2 × 2) / √(10 × 10 × 10 × 10) = 60/100 = 0.6

On paper, a strong and significant association between stilts and youngest-son inheritance.

Counting histories instead of societies

Now count each family once, by the state of its ancestor, and keep the four unrelated societies as they are. That leaves six cases: two with both traits (the North ancestor and one unrelated society), two with neither (the South ancestor and one unrelated society), one with stilts only and one with youngest-son inheritance only.

Expected counts are now 3 × 3 / 6 = 1.5 per cell, and

χ² = 4 × (2 − 1.5)²/1.5 = 4 × 0.25/1.5 ≈ 0.67

φ = (2 × 2 − 1 × 1) / √(3 × 3 × 3 × 3) = 3/9 ≈ 0.33

Nothing remains that a careful reader would call evidence. With expected counts this small a chi-square test is not even appropriate, and that is the honest conclusion: the sample contains about six independent pieces of information, not twenty.

What a tree adds

There is a third way to look at the same data. Inside the two families we can see two changes. In the North, youngest-son inheritance was lost and stilts stayed. In the South, youngest-son inheritance was gained and stilts stayed absent. Neither change involved the two traits moving together. If stilts and youngest-son inheritance were really linked, we would expect changes in one to be followed by changes in the other along the branches of the tree. That question, whether two traits change together over a family tree, is what phylogenetic comparative methods ask.

Four ways out

The literature since Naroll has produced several families of solutions. None is perfect, and each trades one difficulty for another.

The sampling solution is the simplest to explain. Murdock and White grouped the world's described societies into clusters of closely related cultures and chose one well-documented society from each, aiming for as much independence in origin and diffusion as possible. The modeling solution, which Dow and colleagues framed as a problem of network autocorrelation, keeps more cases and puts the dependence into the model instead of throwing data away. The phylogenetic solution, which Ruth Mace and Mark Pagel argued for in a 1994 paper, borrows from evolutionary biology: Pagel's method from the same year tests whether two yes-or-no traits evolve in a correlated way along a tree.

How traits spread between neighbors is a topic in its own right. Our piece on transmission biases looks at the rules by which people copy each other, which is the engine behind the borrowing half of Galton's problem.

How worried should you be?

Not everyone treats Galton's problem as fatal. Melvin and Carol Ember argued in 2000 that this and the other standard objections to worldwide comparison do not invalidate it, and that testing theories on worldwide samples gives more general and more trustworthy results than narrower comparisons. Victor de Munck and Andrey Korotayev went further in papers in 2000 and 2003, turning "Galton's problem" into "Galton's asset": the resemblance among neighbors that spoils a naive test is itself evidence about how cultures are connected. They gave equal billing to "Flower's problem", the question Flower raised in 1888 about what counts as a unit of comparison.

Both views agree on the practical point. If you count related or neighboring societies as if they were separate draws, your confidence will be too high. Our statistics starter shows where this fits among the other tests anthropologists use, and more on the history of traits is collected under Cultural Evolution.

A checklist for reading a cross-cultural correlation

1. How were the societies chosen? A sample built to spread across clusters, such as the Standard Cross-Cultural Sample, is a better start than whatever had good ethnographies.
2. How many language families and regions are represented, not just how many societies?
3. Do the cases that support the pattern cluster in one family or one region? If removing one family makes the result vanish, you have learned something important.
4. Did the authors model the dependence or use a tree? If so, which tree, and how sensitive are the results to it?
5. Is there a plausible mechanism that would make the two traits go together, independent of history?

Run the stilts example through that list and it fails at question 3. Drop the North family and phi falls from 0.6 to about 0.26, with a chi-square of 0.8: a weak association that no test would call significant.

Sources and further reading

• Tylor, E. B. (1889). On a method of investigating the development of institutions; applied to laws of marriage and descent. Journal of the Anthropological Institute of Great Britain and Ireland 18: 245-272. The published discussion, including Galton's remarks, follows on pp. 270-272. jstor.org/stable/2842423; Galton's remarks are also listed at galton.org
• Naroll, R. (1961). Two solutions to Galton's problem. Philosophy of Science 28(1): 15-39. doi.org/10.1086/287778
• Murdock, G. P., and White, D. R. (1969). Standard cross-cultural sample. Ethnology 8(4): 329-369. jstor.org/stable/3772907
• Dow, M. M., Burton, M. L., White, D. R., and Reitz, K. P. (1984). Galton's problem as network autocorrelation. American Ethnologist 11(4): 754-770. doi.org/10.1525/ae.1984.11.4.02a00080
• Mace, R., and Pagel, M. (1994). The comparative method in anthropology. Current Anthropology 35(5): 549-564. doi.org/10.1086/204317
• Pagel, M. (1994). Detecting correlated evolution on phylogenies: a general method for the comparative analysis of discrete characters. Proceedings of the Royal Society of London B 255(1342): 37-45. doi.org/10.1098/rspb.1994.0006
• Ember, M., and Ember, C. R. (2000). Testing theory and why the "units of analysis" problem is not a problem. Ethnology 39(4). jstor.org/stable/3774051
• Korotayev, A., and de Munck, V. (2003). "Galton's asset" and "Flower's problem": cultural networks and cultural units in cross-cultural research. American Anthropologist 105(2): 353-358. doi.org/10.1525/aa.2003.105.2.353