Word frequencies: normalize first, then count

To count occurrences of a word, decide what “the same word” means. Without a consistent rule, capital letters and punctuation can create separate counters for values you intended to group.

State a normalization rule #

For these notes, turn letters to lowercase and strip surrounding full stops, commas, semicolons, colons, exclamation marks and question marks. Preserve internal hyphens. Thus It and it become it, while morning. becomes morning. Keep the original word text and card ID; normalization creates a comparison key rather than erasing the original.

Count every occurrence, not every distinct word once #

Frequency = empty table
MaxFrequency = 0
For each word card X {
    w = Normalize(X.Word)
    If w has no counter: Frequency[w] = 0
    Frequency[w] = Frequency[w] + 1
    if (Frequency[w] > MaxFrequency) {
        MaxFrequency = Frequency[w]
    }
}

Create a counter only when a key is new. If you recreate and reset the counter every time its word appears, repeated words will incorrectly remain at count 1.

The supplied Words table #

The supplied table treats class and teacher, as separate cards, headmaster as one, and fire-eyed as one hyphenated token. The spoken text discusses slightly different splitting and estimates of distinct counts. These notes calculate from the supplied table with the stated normalization, so results are reproducible.

One pass or two sequential passes? #

You may first build the whole frequency table, then scan the counters to find the maximum. Or update MaxFrequency while you count to obtain the value in one pass over word cards. A second pass over counters is not a nested full word scan for every word. Efficient lookup of existing keys is another implementation concern; “one pass over records” does not make every operation free.

Tutorial: bills per customer #

Tutorial 2.5 reuses the pattern with customer names as keys. Its maximum bill count per customer is 4, shared by Srivatsan and Ahmed in the demonstrated set. Counting purchases differs from adding their rupee amounts. State how name spellings are standardized rather than merging people on a guess.

Source: Lecture 2.6; Tutorial 2.5. Small teaching datasets and diagrams are prepared for these notes; lecture and tutorial totals are identified explicitly.