The learnability consequences of Zipfian distributions: Word Segmentation is Facilitated in More Predictable Distributions
This article is a preprint and has not been certified by peer review [What does this mean?].
Author(s) / Creator(s)
Lavi-Rotbain, Ori
Arnon, Inbal
Abstract / Description
One of the striking commonalities between languages is the way word frequencies are distributed. Across languages, word frequencies follow a Zipfian distribution, showing a power law relation between a word's frequency and its rank (Zipf, 1949). Intuitively, this means that languages have relatively few high-frequency words and many low-frequency ones. While studied extensively, little work has explored the learnability consequences of the greater predictability of words in such distributions. Here, we propose such distributions confer a learnability advantage for word segmentation, a foundational aspect of language acquisition. We capture the greater predictability of words using the information-theoretic notion of efficiency, which tells us how predictable a distribution is relative to a uniform one. We first use corpus analyses to show that child-directed speech is similarly predictable across fifteen different languages. We then experimentally investigate the impact of distribution predictability on children and adults. We show that word segmentation is uniquely facilitated at the predictability levels found in language, compared both with uniform distributions and with skewed distributions that are less predictable than those of natural language. We further show that distribution predictability impacts learning more than distribution shape, and that learning is not improved further in distributions more predictable than natural language. These novel findings illustrate learners' sensitivity to the overall predictability of the linguistic environment; suggest that the predictability levels found in language provide an optimal environment for learning; and point to the possible role of cognitive pressures in the emergence and propensity of such distributions in language.
Preprint of: Lavi-Rotbain, O. & Arnon, I. (2022). The learnability consequences of Zipfian distributions in language. Cognition, 223. https://doi.org/10.1016/j.cognition.2022.105038
Keyword(s)
Language acquisition Statistical learning Information theory Zipf's law Word segmentationPersistent Identifier
Date of first publication
2020-06
Publisher
PsychArchives
Is version of
Citation
Lavi-Rotbain, O., & Arnon, I. (2020). The learnability consequences of Zipfian distributions: Word Segmentation is Facilitated in More Predictable Distributions. PsychArchives. https://doi.org/10.23668/PSYCHARCHIVES.3079
-
Zipfian-cognitive-advantage-psycharchive.pdfAdobe PDF - 581.62KBMD5: db5e4b0fc082e0dc35e6fabdae827660
-
22020-06-16The new version was formed in order to fix mistakes that appeared in the graphs of the original version.
-
Author(s) / Creator(s)Lavi-Rotbain, Ori
-
Author(s) / Creator(s)Arnon, Inbal
-
PsychArchives acquisition timestamp2020-06-16T12:36:51Z
-
Made available on2020-06-09T14:29:57Z
-
Made available on2020-06-16T12:36:51Z
-
Date of first publication2020-06
-
Submission date2019-11
-
Abstract / DescriptionOne of the striking commonalities between languages is the way word frequencies are distributed. Across languages, word frequencies follow a Zipfian distribution, showing a power law relation between a word's frequency and its rank (Zipf, 1949). Intuitively, this means that languages have relatively few high-frequency words and many low-frequency ones. While studied extensively, little work has explored the learnability consequences of the greater predictability of words in such distributions. Here, we propose such distributions confer a learnability advantage for word segmentation, a foundational aspect of language acquisition. We capture the greater predictability of words using the information-theoretic notion of efficiency, which tells us how predictable a distribution is relative to a uniform one. We first use corpus analyses to show that child-directed speech is similarly predictable across fifteen different languages. We then experimentally investigate the impact of distribution predictability on children and adults. We show that word segmentation is uniquely facilitated at the predictability levels found in language, compared both with uniform distributions and with skewed distributions that are less predictable than those of natural language. We further show that distribution predictability impacts learning more than distribution shape, and that learning is not improved further in distributions more predictable than natural language. These novel findings illustrate learners' sensitivity to the overall predictability of the linguistic environment; suggest that the predictability levels found in language provide an optimal environment for learning; and point to the possible role of cognitive pressures in the emergence and propensity of such distributions in language.en
-
Abstract / DescriptionPreprint of: Lavi-Rotbain, O. & Arnon, I. (2022). The learnability consequences of Zipfian distributions in language. Cognition, 223. https://doi.org/10.1016/j.cognition.2022.105038en
-
Publication statusotheren
-
Review statusnotRevieweden
-
SponsorshipThe research was funded by the Israeli Science Foundation grant number 584/16 awarded to the second author.en
-
CitationLavi-Rotbain, O., & Arnon, I. (2020). The learnability consequences of Zipfian distributions: Word Segmentation is Facilitated in More Predictable Distributions. PsychArchives. https://doi.org/10.23668/PSYCHARCHIVES.3079en
-
Persistent Identifierhttps://hdl.handle.net/20.500.12034/2693.2
-
Persistent Identifierhttps://doi.org/10.23668/psycharchives.3079
-
Language of contenteng
-
PublisherPsychArchivesen
-
Is version ofhttps://doi.org/10.1016/j.cognition.2022.105038
-
Is related tohttps://doi.org/10.23668/psycharchives.3009
-
Keyword(s)Language acquisitionen
-
Keyword(s)Statistical learningen
-
Keyword(s)Information theoryen
-
Keyword(s)Zipf's lawen
-
Keyword(s)Word segmentationen
-
Dewey Decimal Classification number(s)150
-
TitleThe learnability consequences of Zipfian distributions: Word Segmentation is Facilitated in More Predictable Distributionsen
-
DRO typepreprinten
-
Visible tag(s)Language acquisitionen
-
Visible tag(s)Statistical learningen
-
Visible tag(s)Information theoryen
-
Visible tag(s)Zipf's lawen
-
Visible tag(s)Word segmentationen