Lexical Diversity Measurements
On this page, you can calculate a large number of measures of lexical diversity. For each measurement, a reference is given. If you want to read about the why and how of these calculations, you should look up the references. If you use this tool for your research, please cite it as follows.
Reuneker, A. (2017). Lexical Diversity Measurements. Retrieved 21 November, 2024, from https://www.reuneker.nl/files/ld.
For other calculation tools (such as ngrams, wordlists and keyword analysis), and for contact details, see https://www.reuneker.nl.
Input
* The sample texts are the first chapter of George Orwell's Animal Farm, Sir Arthur Conan Doyle's A Scandal in Bohemia and the first chapter of Charles Darwin's On the Origin of Species. See References for Gutenberg links.
Please take note of the pre-processing (i.e. before calculation) done here:
- All punctuation (e.g. !?.,-) is removed, as are other non-standard characters (e.g. \/*, non-UTF-8 quotes).
- Text between square and angle brackets (e.g. [some text], <p>) is removed.
- All numerical characters (0-9) are removed.
- All tabs, newlines (breaks) double spaces et cetera are removed. Remaining spaces are used as word boundaries and they are not counted.
- All letters are converted to lower case (so 'Speak', 'SPEAK' and 'speak' are treated as one type and three tokens).
Output
| Tokenization | Please analyze a text first | text split into words |
|---|---|---|
| Frequency list | Please analyze a text first | top 40 of most frequent words |
| Tokens | ... | total number of words |
| Types | ... | number unique of words |
| Type-token-ratio (TTR) | ... | number of types divided by number of tokens |
| Mean word frequency (MWF) | ... | number of tokens divided by number of types |
| Mean segmental TTR (MSTTR) | ... | mean ttr for text segmented into chunks of 100 words; only for text longer than 100 words |
| Guiraud's Index | ... | Guiraud (1954) |
| Herdan's C | ... | Herdan (1960; 1964) |
| Yule's I | ... | Yule (1944); Gries (2004) |
| Yule's K | ... | Yule (1944); Oakes (2004, pp. 203-5) |
| Maas's a2 | ... | Maas (1972); Tweedie &Baayen (1998); Treffers-Daller (2013) |
| Dugast's U2 | ... | Dugast (1978; 1979) |
| Measure of textual lexical diversity (MTLD) | ... | McCarthy & Jarvis (2010) |
| Processing time | ... | Yes, a script like this takes only milliseconds. |
Frequency list
Below you can see a list of the most frequent words. Most of the times, the top of the list is occupied by some function words (determiners, general verbs et cetera). The list has a maximum of words and is sorted from most to less frequent.
Tokenized text
Below you can see the text in tokenized form. This means that each numbered item should represent a word. The tokenized text is the main ingredient for all analyses of lexical diversity, but tokenization is not always perfect. Therefore, I consider it a good habit to inspect the tokenized text.