Explainer 3 of 5

Diversity and distinctiveness

Part of the interactive companion to Mapping the Market for Ideas in Europe, 1450–1650, by Noel Johnson and Alexander Taylor

Once every title is a vector, a city's print in a language and a decade is a cloud of points. The paper summarizes each cloud with two numbers. Diversity says how spread out the cloud is. Distinctiveness says how far the middle of the cloud sits from the rest of the language. This page defines both and shows why each is built the way it is. The vectors themselves are explained here.

The two predictions they test

Adam Smith's account of the division of labor says a city with a larger market specializes, so its diversity should fall and its distinctiveness should rise (Smith 1776, Stigler 1951). Monopolistic competition says a larger market supports more varieties and that producers position near the center of demand, so diversity should rise and distinctiveness should fall (Krugman 1980, Hotelling 1929). The two measures are the two sides of that disagreement.

Diversity

Diversity is the average cosine distance between pairs of a city's titles. For a city $c$, language $\ell$, and decade $t$ with $n$ titles,

$$D_{c,\ell,t} \;=\; \frac{1}{n(n-1)} \sum_{i \neq j} d_{\cos}(v_i, v_j).$$

A city that prints across many subjects has titles far apart from one another and a high $D$. A city that prints one kind of book has titles close together and a low $D$. In Latin print of 1600 to 1609 Paris has a diversity of 0.97 across 1,041 titles in theology, classical authors, law, and history. Coimbra has 0.71 across 74 titles, mostly scholastic philosophy and theology.

We average over pairs for a reason. The obvious alternative is the average distance from each title to the city's own centroid. That number rises mechanically with the number of titles because a centroid computed from a few titles sits close to each of them. Small cities would look narrow whatever they printed. The pairwise average has the same expected value at any number of titles.

Distinctiveness

Distinctiveness compares the city's centroid $\bar{v}_{c}$ with the centroid $\bar{v}_{-c}$ of every other city's titles in the same language and decade. Leaving the city out of its own reference matters. A large center would otherwise pull the reference toward itself and look typical by construction.

The plain cosine distance between the two centroids has a second problem. A centroid averaged from few titles carries sampling noise. The noise makes the centroid longer than it should be and tilts it away from any reference, so small cities look more distinctive than they are. Within cities the number of titles rises with market access, so the bias would load onto the regressor. We replace the length of the city's centroid with an unbiased estimate that uses only products of distinct titles.

$$\widehat{\lVert \mu_c \rVert^2} \;=\; \frac{\bigl\lVert \sum_i v_i \bigr\rVert^2 - \sum_i \lVert v_i \rVert^2}{n(n-1)}, \qquad \tilde{X}_{c} \;=\; 1 - \frac{\bar{v}_{c} \cdot \bar{v}_{-c}}{\sqrt{\widehat{\lVert \mu_c \rVert^2}}\;\lVert \bar{v}_{-c} \rVert}.$$

The numerator needs no correction because the city and the reference share no titles. In very small cells the estimated length can be zero or negative and the measure is then undefined. That happens in 192 of 6,110 decade cells and those cells are dropped.

In Latin print of 1600 to 1609 Venice is the most distinctive large center at 1.38. Wittenberg, the largest Latin center of the decade, sits at 0.76 and is below the median.

Try it

The figure draws a city's titles as points on a circle. The rest of the language points to the right. Move the city's center away from the reference to raise distinctiveness. Widen the spread to raise diversity. Then lower the number of titles. The plain distance drifts up and the corrected one stays near the truth.

Where the city's print is centered How widely its titles spread Number of titles

diversity of the sample on the left

plain distance between centroids, averaged over samples of this size

corrected distinctiveness, averaged the same way, against a true value of

Figure 1. A toy city. Left, one sample of titles drawn in two dimensions, in navy, with the city's centroid in rust and the direction of the rest of the language in gold. Right, a simulation of the same city in 50 dimensions, where a small centroid is as noisy as it is in the paper's 200. Each point is the average over 300 samples at that number of titles and the dashed line is the true distinctiveness. Samples in which the corrected measure is undefined are dropped, as in the paper. With very few titles and a very wide spread many are dropped and the corrected average then sits above the truth.

From decades to periods

Both measures are computed for every city, language, and decade with at least five titles. They are then averaged over the decades of each 50-year period with weights equal to the number of titles in each decade. Across the six-language panel diversity has a mean of 0.88 to 0.90 by language. Distinctiveness has means of 0.48 to 0.69.

What counts as a title

The catalog records every issue of a periodical as a separate edition. A weekly would enter a city's decade about 500 times with nearly identical titles and would drag diversity toward zero. We keep one title per periodical series in each city and decade and apply the same rule to any other title that recurs at least three times with nearly identical text. Every measure counts distinct works.

What to keep in mind

The two measures are separate. A city can print a wide range of titles that is centered far from the mainstream, or a narrow range centered on it.
Both are distances inside one language. They are compared across languages only after language effects are removed.
They measure what was printed. What readers bought and read is not observed.

The companion shows both measures for every large Latin center in 1600 to 1609 in its third section.

References

Harold Hotelling (1929). Stability in Competition. Economic Journal 39(153), 41–57.
Paul Krugman (1980). Scale Economies, Product Differentiation, and the Pattern of Trade. American Economic Review 70(5), 950–959.
Adam Smith (1776). An Inquiry into the Nature and Causes of the Wealth of Nations. London, W. Strahan and T. Cadell.
George J. Stigler (1951). The Division of Labor is Limited by the Extent of the Market. Journal of Political Economy 59(3), 185–193.