1222. This week, we look at Zipf's law, a pattern that shapes how often you use your most common words, and why SETI researchers think it could help identify alien communication.
1222. This week, we look at Zipf's Law, a pattern that shapes how often you use your most common words, and why SETI researchers think it could help identify alien communication.
Save the date for Grammarpalooza — The Game Show: Thursday, September 24 at 5 p.m. PT / 8 p.m. ET.
A hearty thank-you to our Keepers of the Commas and one Immortal on Patreon. We appreciate your support!
🔗 Share your familect recording in Speakpipe or by leaving a voicemail at 833-214-GIRL (833-214-4475)
🔗 Watch my LinkedIn Learning writing courses.
🔗 Subscribe to the newsletter.
🔗 Find an edited transcript.
🔗 Get Grammar Girl books.
| HOST: Mignon Fogarty
| Grammar Girl is part of the Quick and Dirty Tips podcast network.
| Theme music by Catherine Rannus.
| Grammar Girl Social Media: YouTube. TikTok. Facebook. Threads. Instagram. LinkedIn. Mastodon. Bluesky.
Grammar Girl here. I’m Mignon Fogarty. This week, we'll talk about Zipf's Law, a pattern that shapes how often you use your most common words, and why SETI researchers think it could help identify alien communication.
But first, Grammarpalooza — The Game Show to celebrate the 20th anniversary of the podcast is in just two days — September 24. And you'll want to join us for the livestream at 8 PM Eastern as six fabulous guests compete for grammar glory because we'll all have fun, but some of you will win books too. Check any of my social media pages for more information, or just head over to my YouTube or Facebook page and RSVP for the event. The links are in the show notes, and I hope to see you there.
by Mignon Fogarty
Maybe it's weird for a word person, but I love spreadsheets. I love data and charts, and the other day Pat was teasing me about it over breakfast and asked what he thought was going to be a ridiculous question: "Don't you want to know how many times you use the word 'the' in different types of documents?"
And I was like, "WELL, ACTUALLY, there's this fascinating thing in linguistics ..."
And I was off to the races, and he just started cracking up. But there IS a fascinating thing in linguistics about words like "the," and it's called Zipf's Law, and now I have to tell you about it too!
Here's the deal: In many, many human languages, the most common word appears about twice as often as the second most common word, three times as often as the third most common word, and so on. Mathematicians call that being "inversely proportional."
Here are some real numbers so you can see how precise it is:
In the Brown Corpus — that's this famous collection of American English texts — the word "the" appears about 70,000 times. That's about 7% of all the words!
The second most common word, "of," appears about 36,000 times. That's almost exactly half as often as "the."
And the third most common word — "and" — shows up about 29,000 times. The pattern kind of falls apart here because "and" isn't showing up a third as often as "the," but the words do still fall off in a regular pattern as you go quite far down on the word list, and people have come up with a more complicated mathematical formula that describes the pattern that they call the Zipf-Mandelbrot Law.
Now that might not seem like such a big deal, except that it's not just for English. It shows up in a huge number of languages.
So who discovered this pattern? Well, the law gets its name from a Harvard linguist named George Kingsley Zipf, and here's my favorite part: he apparently wasn't a huge fan of math.
He wasn't the first person to notice the pattern though. A French stenographer named Jean-Baptiste Estoup had documented it by 1916. But Zipf was the one who really looked at why it happens, which he included in his 1949 book, with this great title: "Human Behavior and the Principle of Least Effort."
Yes, Zipf said that speakers want to be lazy. He said we'd love to just use one word for everything because that's easy to remember. But listeners are different. They want clarity. So they want different words for different meanings so they don't get confused.
So his theory was that the distribution of words we see in language comes from the balance between those two opposing forces: speakers trying to minimize effort and listeners trying to maximize understanding.
Zipf died in 1950 when he was just 48 years old, but his law lives on. And it's gotten even more interesting since then.
Those other languages I mentioned?
Well, researchers looked at 50 languages in 2018. FIFTY. From all over the world — Indo-European languages, Sino-Tibetan languages, Dravidian, Afro-Asiatic, and more.
And they all followed the same shape, with the middle of each word list closely following Zipf's law. The rarest words dropped off faster than the law predicted, but it happened the same way in every language.
Other researchers looked at other samples, including James Joyce's "Ulysses," Cervantes's "Don Quijote" in Spanish, massive French novels, Finnish texts with complex word forms, and even constructed languages like Esperanto. And the pattern still held.
So we've got this pretty enticing, universal-ish pattern, but scientists have been arguing about WHY it happens for 90 years.
Zipf's original idea about language gravitating toward the least effort it can get away with while preserving clarity is one possibility.
Another theory says that it's the result of success breeding success, technically called "a preferential attachment process." The idea is that once you use a word in a document, you're actually more likely to use it again.
And I won't go through them all, but another is that it's all just the result of random processes. That if you sit a monkey down at a typewriter, you can get text with a Zipfian distribution.
One 2014 review looked at all the competing theories and concluded — and I'm quoting here — that "no prior account straightforwardly explains all the basic facts."
And now, here's where it gets REALLY wild.
Humpback whale songs follow the Zipf-Mandelbrot Law, and it's a little more controversial, but some people also think they find it in bottlenose dolphin clicks and whistles.
Researchers at the SETI Institute — yes, the Search for Extraterrestrial Intelligence — they have thought about using Zipf's Law as a test for complex communication. They tested it on a pulsar signal from space and found that those signals DIDN'T follow the pattern, so they say following Zipf's Law is one thing they might look for to decide whether a new signal is alien communication or not.
And here's another cool discovery: Remember how I said Zipf's Law works for words? Well, a 2015 study found that it works for phrases too, and it actually continues much longer down the ranked list for phrases than it does for words.
The authors speculated that maybe phrases rather than individual words are the fundamental units for understanding Zipf's Law in language.
That might explain why no theory fully accounts for the word-level data. Maybe we're like researchers trying to understand a building by studying individual bricks when we should be looking at whole walls!
And here's a final tantalizing tidbit: The undeciphered Voynich Manuscript, whose parchment dates from the 15th century and that people have been trying to translate for hundreds of years, also follows Zipf's Law.
Some people think that means it's a real language that we just haven't figured out yet, but other people, remembering those typing monkeys and made-up languages like Esperanto, say it doesn't actually tell us much either way.
So here's where things stand: Zipf's Law is one of the most robust patterns in linguistics. It appears in at least 50 languages. It works for high-brow literature, newspapers, and even whale songs.
It could even help us detect alien intelligence.
But after about a century of research, we STILL don't completely understand why it exists.
Maybe it's speakers being lazy. Maybe it's our tendency to be drawn to things that are already popular. Maybe it's just the way randomness shows up in language. Or maybe it's all three working together. But it's a fun thing to ponder over breakfast, especially for a data-lover like me.
References
Baixeries, J., Elvevåg, B., & Ferrer-i-Cancho, R. (2013). The evolution of the exponent of Zipf's law in language ontogeny. PLOS ONE. https://pubmed.ncbi.nlm.nih.gov/23516390/
Ferrer i Cancho, R., & Solé, R. V. (2003). Least effort and the origins of scaling in human language. Proceedings of the National Academy of Sciences, 100(3), 788-791. https://www.pnas.org/doi/10.1073/pnas.0335980100
Kanwal, J., Smith, K., Culbertson, J., & Kirby, S. (2017). Zipf's law of abbreviation and the principle of least effort: Language users optimise a miniature lexicon for efficient communication. Cognition, 165, 45-52. https://www.sciencedirect.com/science/article/abs/pii/S0010027717301166
Kershenbaum, A., et al. (2021). Shannon entropy as a robust estimator of Zipf's Law in animal vocal communication repertoires. Methods in Ecology and Evolution. https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.13536
Linders, R., & Louwerse, M. M. (2023). Zipf's law revisited: Spoken dialog, linguistic units, parameters, and the principle of least effort. Psychonomic Bulletin & Review. https://pmc.ncbi.nlm.nih.gov/articles/PMC9971120/
Piantadosi, S. T. (2014). Zipf's word frequency law in natural language: A critical review and future directions. Psychonomic Bulletin & Review, 21(5), 1112-1130. https://pmc.ncbi.nlm.nih.gov/articles/PMC4176592/
Williams, J. R., Lessard, P. R., Desu, S., Clark, E. M., Bagrow, J. P., Danforth, C. M., & Dodds, P. S. (2015). Zipf's law holds for phrases, not words. Scientific Reports, 5, 12209. https://www.nature.com/articles/srep12209
Yu, S., Xu, C., & Liu, H. (2018). Zipf's law in 50 languages: its structural pattern, linguistic interpretation, and cognitive motivation. arXiv preprint arXiv:1807.01855. https://arxiv.org/pdf/1807.01855
Baixeries, J., Elvevåg, B., & Ferrer-i-Cancho, R. (2013). The evolution of the exponent of Zipf's law in language ontogeny. PLOS ONE. https://pmc.ncbi.nlm.nih.gov/articles/PMC3596411/
Ferrer i Cancho, R., & Solé, R. V. (2003). Least effort and the origins of scaling in human language. Proceedings of the National Academy of Sciences, 100(3), 788-791. https://pubmed.ncbi.nlm.nih.gov/12540826/
Kanwal, J., Smith, K., Culbertson, J., & Kirby, S. (2017). Zipf's law of abbreviation and the principle of least effort: Language users optimise a miniature lexicon for efficient communication. Cognition, 165, 45-52. https://www.pure.ed.ac.uk/ws/files/36266819/kanwal_17_zipfs.pdf
Kershenbaum, A., et al. (2021). Shannon entropy as a robust estimator of Zipf's Law in animal vocal communication repertoires. Methods in Ecology and Evolution. https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210X.13536
Linders, R., & Louwerse, M. M. (2023). Zipf's law revisited: Spoken dialog, linguistic units, parameters, and the principle of least effort. Psychonomic Bulletin & Review. https://link.springer.com/article/10.3758/s13423-022-02142-9
Piantadosi, S. T. (2014). Zipf's word frequency law in natural language: A critical review and future directions. Psychonomic Bulletin & Review, 21(5), 1112-1130. https://pmc.ncbi.nlm.nih.gov/articles/PMC4176592/
Williams, J. R., Lessard, P. R., Desu, S., Clark, E. M., Bagrow, J. P., Danforth, C. M., & Dodds, P. S. (2015). Zipf's law holds for phrases, not words. Scientific Reports, 5, 12209. https://www.nature.com/articles/srep12209
Yu, S., Xu, C., & Liu, H. (2018). Zipf's law in 50 languages: its structural pattern, linguistic interpretation, and cognitive motivation. arXiv preprint arXiv:1807.01855. https://arxiv.org/abs/1807.01855
Finally, I have a quick little familect from Stephanie:
Good afternoon, Mignon. My name is Stephanie Bryant, and I'm in Kentucky, and our family saying was "bump medicine" for hydrocortisone cream. Kids could not say hydrocortisone cream.
Thanks, Stephanie. Hydrocortisone cream is definitely a tough one for kids to say.
If you want to call with your familect story — a word or phrase you use with just your family (or maybe a friendilect you use with just your friends), you can leave a message on the voicemail line or use your computer or phone to record it on Speakpipe, and both of those are in the show notes.
Grammar Girl is a Quick and Dirty Tips podcast. Thanks to Nat Hoopes; Morgan Christianson; Rebekah Sebastian; Steven Schapansky; Holly Hutchings; Dan Feierabend; and Maram Elnagheeb, who has ridden all 28 subway lines in New York City.
And I'm Mignon Fogarty, better known as Grammar Girl.
That's all. Thanks for listening. And don't forget to like and subscribe where you listen, and join us Thursday for Grammarpalooza — The Game Show.