Two passes when the second decision needs final counts

Combining work into one pass is useful when it is correct. Some decisions need information that is not available until the entire first pass has finished.

A word seen once so far may repeat later #

Imagine sun, rain, sun. When you first see sun, its current frequency is 1. If you immediately keep it as a final “unique word”, the later sun invalidates that decision. The final frequencies are sun=2 and rain=1.

Pass 1:
    Build Frequency for every normalized word
Pass 2:
    For every original word card X {
        w = Normalize(X.Word)
        if (Frequency[w] == 1) { keep X }
        else { discard X }
    }

“Discard” here means move into the non-selected output of the exercise, not delete the original dataset. Retain the original records and use their IDs when you need to reconstruct order.

Use identical keys in both passes #

If the first pass combines It with it but the second pass looks for the raw capitalized text, filtering can fail. Normalize case and surrounding punctuation in exactly the same way. Do not remove part of an internal hyphenated token in only one pass.

Dataset result and threshold choices #

With the supplied Words table and our rule, 38 normalized keys have final count 1. Each occurs once, so 38 cards are selected. The remaining 27 occurrence cards belong to nine repeated keys. You could instead keep words with frequency at most 2; define that threshold before filtering.

Why word frequencies are useful #

The lecture uses search and document comparison as a motivation: common words often tell us less about a document’s specific topic than distinctive words. This is an introductory model for why frequency matters, not a complete description of a modern search engine or a rule that frequent words should always be removed.

Sequential is not nested #

Two sequential passes finish one full scan, then start another. Nested full scans would restart a second complete scan for each outer record. In this task, it is the dependence on completed global counts that explains the second pass.

Source: Lectures 2.6 and 2.7. Small teaching datasets and diagrams are prepared for these notes; lecture and tutorial totals are identified explicitly.