Clean a web crawl

Run a miniature pretraining-data pipeline on a sample crawl. Switch on quality rules, a word blocklist, and exact and near-duplicate removal, and see what survives.

AdvancedExplained in Where training data comes from

Clean a web crawl

A sample crawl of 113 pages written for this lab. Switch on each stage of the pipeline and see what survives.

Pipeline

  1. Remove pages that break simple quality rules from Gopher and C4.

    Off

  2. Remove any page containing a blocked word, as C4 did. This lab blocks four words from C4's list.

    Off

  3. Remove pages whose text is identical to an earlier page.

    Off

  4. Remove pages that share most of their 5-word phrases with an earlier page, estimated with MinHash.

    Off

  5. Keep only the first copy of every line, so repeated menus and notices appear once.

    Off

Words left after each stage

  • Raw crawl10,795 words, 113 pages

The crawl, one square per page

Page 1: harbordaily.example/storm-record-rain

A person would call it: Article

Kept.

Skip to main content

Harbor Daily | News | Sports | Weather | Opinion

Storm brings record rain to Harbor City

Harbor City recorded 112 millimeters of rain on Saturday, the wettest single day since the city's weather station opened in 1952, according to the regional weather office.

Water covered parts of Front Street and the river path by mid-afternoon. Firefighters helped residents of twelve homes near the old mill move to a shelter at the community center.

City engineer Dana Ortiz said the storm drains were working but could not keep up with the water running down from the hills. Crews will inspect every drain on the east side on Monday.

No injuries were reported. The library and the public pool will stay closed on Sunday while staff check for damage. Residents can report flooded basements by calling the city hotline.

Subscribe to our newsletter for the latest local news.

We use cookies to improve your experience. By continuing to browse, you agree to our cookie policy.

© 2026 Harbor Daily. All rights reserved.

Pages kept113 of 113
Words kept100% (10,795)
Articles kept28 of 28
Cookie notice appears on48 pages

Challenges

0 of 4 complete

  1. The storm report was reposted on four other sites. Remove every repost and copy, but keep the different story about the cleanup after the storm. (not yet)
  2. Find an article worth learning from that a filter throws away, and open it. (not yet)
  3. With the page rules on, find a page of spam or boilerplate that every rule let through, and open it. (not yet)
  4. The same cookie notice appears on most pages. Get it down to one copy in what remains. (not yet)

Try this

  • Turn the stages on from top to bottom and watch the words left after each one.
  • Open a page removed by the page rules and read which rule removed it. Do you agree?
  • With near duplicates on, lower the threshold step by step and watch the reposted storm story disappear, then the different story about the cleanup.
  • Turn on repeated lines and open any article: its menus and cookie notice are struck through.
Exact and estimated similarity with MinHashPython with the standard library
import hashlib

def words(text):
    # Lowercase and drop punctuation, so formatting changes do not count.
    return "".join(c.lower() if c.isalnum() else " " for c in text).split()

def shingles(text, n=5):
    w = words(text)
    return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)}

def minhash(shingle_set, k=112):
    # Hash function i scores each shingle by the MD5 of "i:shingle".
    # The signature keeps the smallest score for each of the k functions.
    return [min(int(hashlib.md5(f"{i}:{s}".encode()).hexdigest()[:8], 16)
                for s in shingle_set) for i in range(k)]

original = ("Harbor City recorded 112 millimeters of rain on Saturday, the wettest "
            "single day since the city's weather station opened in 1952. Water "
            "covered parts of Front Street and the river path by mid-afternoon.")
repost = ("Storm brings record rain. Harbor City recorded 112 millimeters of rain "
          "this Saturday, the wettest single day since the city's weather station "
          "opened in 1952. Water covered parts of Front Street and the river path "
          "by mid-afternoon. Originally published by Harbor Daily.")
other = ("Volunteers cleared mud from Front Street on Sunday after the rain, the "
         "heaviest the city has measured in a single day since 1952.")

a = shingles(original)
sig_a = minhash(a)
for name, text in [("repost", repost), ("other story", other)]:
    b = shingles(text)
    exact = len(a & b) / len(a | b)
    sig_b = minhash(b)
    estimate = sum(x == y for x, y in zip(sig_a, sig_b)) / len(sig_a)
    print(f"{name:11}  exact {exact:.2f}  MinHash estimate {estimate:.2f}")
repost       exact 0.58  MinHash estimate 0.51
other story  exact 0.00  MinHash estimate 0.00

Try "embedding", "softmax", "overfitting", or "backpropagation".