Sanity of data: find problems before calculating

A well-written procedure can still give a wrong answer if its inputs are wrong. Sanity checking means asking whether the data have the expected format, range, fields, relationships and context.

Check each field and its allowed range #

Check relationships, not just individual fields #

A marks total must equal maths + physics + chemistry. For example, 84+92+76 = 252. If chemistry says 766 while the total says 252, the record is inconsistent. If you trust the other three numbers, the implied chemistry mark is 252−84−92 = 76; that is a proposed correction to verify, not a fact established by type checking.

For a simplified bill, quantity × unit price should equal item cost, and item costs should add to bill total. Three chart papers at Rs 22 cost Rs 66, not Rs 88. Even when every field is a legal number, a relationship can fail.

Context can change what is valid #

Thirty-one is a valid day in January but not April. Thirty April is valid; the rule is at most 30, not less than 30. Checking 29 February needs a year or a stated leap-year convention. A mark of 125 is invalid out of 100 but may be valid out of 150.

Quantity units must match the price units: 400 g of carrots at Rs 30 per kg is 0.4×30 = Rs 12, not 400×30 = Rs 12,000. A very large bill is a reason to investigate the units and context, not proof that the transaction must be wrong.

Missing is not zero #

A blank or NA price means the price is unknown. It is not automatically a free item. An external verified price list may help fill a missing unit price; a missing purchased quantity usually needs the original transaction. Do not invent an answer merely to make the arithmetic work.

Check categories, identities and word context #

Milk labeled Stationery needs a category check. Similar names such as Aditi Ram and R Aditi may refer to the same person or to different people: verify IDs and other evidence before merging. Check uniqueness across records as well as the format of one ID.

For word cards, compare the actual letter count with the stored count and check spelling. The word “open” has four letters; whether it is a verb or adjective requires its sentence. A dictionary or a format rule alone may not resolve the recorded category.

Flag first, then repair with evidence #

Separate “a value is impossible under the rules” from “the record is unusual” and from “we lack enough information”. Keep the original entry, record the suspected issue, and verify a correction. A valid-looking record can still be factually wrong.

Source: Lecture 1.6 — Sanity of data. Small teaching datasets and diagrams are prepared for these notes; lecture and tutorial totals are identified explicitly.