System Design: Distributed Web Crawler

Section

Mix.install([
  {:choreo, "~> 0.14.1"},
  # {:choreo, path: Path.expand("../..", __DIR__), force: true},
  {:kino_vizjs, "~> 0.9.0"}
])

alias Choreo.C4
alias Choreo.C4.Analysis, as: C4Analysis
alias Choreo.Dataflow
alias Choreo.Dataflow.Analysis, as: DataflowAnalysis
alias Choreo.ERD
alias Choreo.ERD.Analysis, as: ERDAnalysis
alias Choreo.Requirement
alias Choreo.Requirement.Analysis, as: RequirementAnalysis
alias Choreo.ThreatModel
alias Choreo.ThreatModel.Analysis, as: ThreatAnalysis
alias Choreo.Workflow
alias Choreo.Workflow.Analysis, as: WorkflowAnalysis

render_tabs = fn mermaid, dot, height ->
  Kino.Layout.tabs(
    Siren: Choreo.Lab.Siren.new(mermaid, height: height),
    Graphviz: Kino.VizJS.render(dot, height: height)
  )
end

1. Problem Statement

Design a distributed web crawler that scrapes roughly 10,000 websites per day from a date-named S3 seed file. The crawler must deduplicate URLs within a crawl day, respect robots.txt, fetch pages politely, and store page metadata and content in PostgreSQL.

The design should be simple enough for a small team to operate, but resilient enough to tolerate bad hosts, slow DNS, redirects, retries, and poison URLs.

2. Requirements

Functional Requirements

Non-Functional Requirements

3. Assumptions and Constraints

Assumptions

• Seed files are UTF-8 text with one URL per line.
• The S3 input may already be partially deduplicated, but Redis is the authoritative per-day crawl dedupe layer.
• PostgreSQL stores canonical metadata and recent page content. Very large raw bodies can later move to object storage with DB references.
• 10,000 URLs/day is modest; the design prioritizes correctness, politeness, and operability over maximum crawl throughput.

Explicitly Out of Scope

• Browser rendering / JavaScript execution.
• Full-text indexing and search ranking.
• Distributed crawl frontier discovery from links found on pages.
• Circumventing site-level anti-bot protections.
• Multi-region active-active crawling.

4. C4 System Context and Container View

The C4 model uses the Lab DSL as the source of truth. It keeps external systems separate from in-scope runnable containers.

c4 =
  (fn ->
     import Choreo.Lab.DSL.C4

     c4 do
       operator =
         person("Platform Operator",
           description: "Owns schedules, monitoring, and incident response."
         )

       s3_seeds = system("S3 Seed Store", scope: :out, description: "Date-named seed URL files.")

       target_sites =
         system("External Websites",
           scope: :out,
           description: "Public websites to crawl politely."
         )

       alerting =
         system("Observability / Alerting",
           scope: :out,
           description: "Metrics, logs, traces, alerts."
         )

       crawler =
         system("Crawler System",
           scope: :in,
           description: "Daily URL ingestion, crawl scheduling, fetching, and persistence."
         ) do
           seed_loader = container("Seed Loader", technology: "Elixir / Oban Cron")
           dedupe_cache = datastore("Daily Dedupe Cache", technology: "Redis TTL Sets")
           frontier = container("URL Frontier", technology: "RabbitMQ / Redis Queue")
           robots_cache = datastore("Robots Cache", technology: "Redis + Postgres")
           crawler_workers = container("Crawler Workers", technology: "Elixir DynamicSupervisor")
           postgres = database("Crawler DB", technology: "PostgreSQL")
         end

       operator ~> crawler |> uses("Configures and monitors")
       seed_loader ~> s3_seeds |> reads("Downloads daily seed file", technology: "S3 API")

       seed_loader
       ~> dedupe_cache
       |> calls("Checks crawl_date + canonical URL", technology: "Redis")

       seed_loader ~> frontier |> publishes("Enqueues fresh URLs", technology: "AMQP / Redis")
       crawler_workers ~> frontier |> calls("Consumes crawl jobs")
       crawler_workers ~> robots_cache |> reads("Loads per-domain crawl rules")

       crawler_workers
       ~> target_sites
       |> calls("Fetches pages respecting robots.txt", technology: "HTTPS")

       crawler_workers
       ~> postgres
       |> writes("Persists pages and crawl attempts", technology: "SQL")

       crawler_workers
       ~> alerting
       |> publishes("Emits crawl metrics and failures", technology: "OTLP")
     end
   end).()

render_tabs.(
  C4.to_mermaid(c4),
  C4.to_dot(c4),
  "800px"
)

5. Core Dataflow

This view focuses on the hot data path from seed file to stored crawl result, including retry and dead-letter paths.

crawl_dataflow =
  (fn ->
     import Choreo.Lab.DSL.Dataflow

     dataflow do
       seed_file = source("S3 Daily Seed File", rate: "10k/day")
       normalize = transform("Normalize + Canonicalize URL", latency_ms: 1)
       dedupe = transform("Daily Redis Dedupe", latency_ms: 2)
       frontier = buffer("URL Frontier Queue", capacity: 50_000)
       robots = transform("Robots.txt Lookup", latency_ms: 25)
       politeness = buffer("Per-Domain Politeness Gate", capacity: "domain buckets")
       fetch = transform("HTTP Fetch", latency_ms: 2_000, capacity: "worker pool")
       persist = sink("PostgreSQL Page Store")
       retry_queue = buffer("Retry Queue", capacity: 10_000)
       dlq = sink("Dead Letter / Poison URL Store")

       seed_file ~> normalize |> emits("raw URL line")
       normalize ~> dedupe |> emits("canonical URL")
       dedupe ~> frontier |> emits("fresh URL for date")
       frontier ~> robots |> emits("crawl job")
       robots ~> politeness |> emits("allowed URL")
       politeness ~> fetch |> emits("rate-limited request")
       fetch ~> persist |> writes("page result")
       retry(fetch ~> retry_queue, "DNS/5xx/timeout")
       retry_queue ~> politeness |> emits("retry job")
       dead_letter(fetch ~> dlq, "robots denied / repeated failure / poison URL")
     end
   end).()

render_tabs.(
  Dataflow.to_mermaid(crawl_dataflow),
  Dataflow.to_dot(crawl_dataflow),
  "800px"
)

6. Crawl Workflow

The workflow makes operational behavior explicit: fail closed on robots, cap retries, and always record an outcome.

crawl_workflow =
  (fn ->
     import Choreo.Lab.DSL.Workflow

     workflow do
       start = start("Daily Crawl Trigger")
       load_seed = task("Load Seed File", timeout_ms: 60_000, retry: 2)
       normalize = task("Normalize URLs", timeout_ms: 120_000)
       enqueue = task("Enqueue Fresh URLs", timeout_ms: 120_000, retry: 2)
       crawl = task("Crawl URL", timeout_ms: 10_000, retry: 2)
       allowed = decision("Robots Allowed?")
       persist = task("Persist Result", timeout_ms: 500, retry: 3)
       quarantine = task("Quarantine URL", timeout_ms: 500)
       done = finish("Daily Job Complete")

       start ~> load_seed
       load_seed ~> normalize
       normalize ~> enqueue
       enqueue ~> crawl
       crawl ~> allowed
       allowed ~> persist |> condition("yes")
       allowed ~> quarantine |> failure("no / denied")
       persist ~> done
       quarantine ~> done
     end
   end).()

render_tabs.(
  Workflow.to_mermaid(crawl_workflow),
  Workflow.to_dot(crawl_workflow),
  "750px"
)

7. Database Model / ERD

erd =
  (fn ->
     import Choreo.Lab.DSL.ERD

     erd do
       crawl_jobs =
         table("crawl_jobs") do
           pk(:id, :uuid)
           field(:crawl_date, :date, comment: "Daily job identifier")
           field(:status, :varchar, comment: "queued, running, completed, failed")
           field(:started_at, :utc_datetime)
           field(:finished_at, :utc_datetime)
         end

       urls =
         table("urls") do
           pk(:id, :uuid)
           fk(:crawl_job_id, :uuid)
           field(:canonical_url, :text)
           field(:domain, :varchar)
           field(:dedupe_key, :varchar, comment: "crawl_date + normalized URL")
         end

       crawl_attempts =
         table("crawl_attempts") do
           pk(:id, :uuid)
           fk(:url_id, :uuid)
           field(:attempt_no, :integer)
           field(:status_code, :integer)
           field(:error_reason, :text)
           field(:started_at, :utc_datetime)
           field(:finished_at, :utc_datetime)
         end

       crawled_pages =
         table("crawled_pages") do
           pk(:id, :uuid)
           fk(:url_id, :uuid)
           field(:raw_html, :text)
           field(:content_hash, :varchar)
           field(:crawled_at, :utc_datetime)
         end

       robots_cache =
         table("robots_cache") do
           pk(:domain, :varchar)
           field(:directives, :text, comment: "Serialized rules")
           field(:expires_at, :utc_datetime)
         end

       crawl_jobs ~> urls |> has_many("contains", from: :id, to: :crawl_job_id)
       urls ~> crawl_attempts |> has_many("attempted as", from: :id, to: :url_id)
       urls ~> crawled_pages |> has_many("stores", from: :id, to: :url_id)
     end
   end).()

render_tabs.(
  ERD.to_mermaid(erd, syntax: :erd),
  ERD.to_dot(erd),
  "750px"
)

8. Threat Model

Crawler inputs and target responses are untrusted. The main security concerns are SSRF-like redirects, storage poisoning, retry amplification, credential exposure, and accidental policy bypass.

threat_model =
  (fn ->
     import Choreo.Lab.DSL.ThreatModel

     threat_model do
       internet = boundary("Internet", level: 0)
       worker_zone = boundary("Crawler Worker Zone", level: 2)
       storage = boundary("Storage Zone", level: 3)

       seeds = external_entity("S3 Seed File", boundary: internet)
       websites = external_entity("Target Websites", boundary: internet)
       worker = process("Crawler Worker", boundary: worker_zone, privilege: :system)
       queue = process("URL Frontier", boundary: worker_zone, privilege: :system)
       db = data_store("PostgreSQL", boundary: storage, sensitivity: :internal)

       redis =
         data_store("Redis Dedupe / Robots Cache", boundary: storage, sensitivity: :internal)

       seeds ~> queue |> encrypted("Seed URL ingest", protocol: :https)
       queue ~> worker |> flow("Crawl job", protocol: :amqp)
       worker ~> websites |> encrypted("HTTP fetch", protocol: :https)
       worker ~> redis |> flow("Dedupe and robots lookup", protocol: :redis, encrypted: true)
       worker ~> db |> flow("Persist crawl result", protocol: :sql, encrypted: true)
     end
   end).()

render_tabs.(
  ThreatModel.to_mermaid(threat_model),
  ThreatModel.to_dot(threat_model),
  "550px"
)

9. Requirements Traceability

requirements =
  (fn ->
    import Choreo.Lab.DSL.Requirement

    requirements "Distributed Web Crawler" do
      platform = stakeholder("Platform Team")
      security = stakeholder("Security Team")

      ingest = functional("Ingest daily seed URLs from S3", id: "FR-1")
      dedupe = functional("Deduplicate URLs per crawl day", id: "FR-2", risk: :high)
      robots = functional("Respect robots.txt and per-domain politeness", id: "FR-3", risk: :critical)
      persist = functional("Store crawl results and failures", id: "FR-5")
      window = performance("Complete the daily crawl window", id: "NFR-1")
      retry_bounds = design_constraint("Bound retries and poison URL handling", id: "NFR-4", risk: :high)

      seed_loader = component("Seed Loader")
      crawler_workers = component("Crawler Workers")
      db = component("Crawler DB")
      tests = test_case("Crawler integration and politeness tests")

      platform ~> ingest |> traces("owns")
      security ~> robots |> traces("reviews")
      seed_loader ~> ingest |> satisfies("implements")
      seed_loader ~> dedupe |> satisfies("implements")
      crawler_workers ~> robots |> satisfies("implements")
      crawler_workers ~> window |> satisfies("scales workers")
      crawler_workers ~> retry_bounds |> satisfies("caps retries")
      db ~> persist |> satisfies("stores")
      tests ~> dedupe |> verifies("proves")
      tests ~> robots |> verifies("proves")
      tests ~> retry_bounds |> verifies("proves")
    end
  end).()

render_tabs.(
  Requirement.to_mermaid(requirements),
  Requirement.to_dot(requirements),
  "550px"
)

10. Analysis and Verification

%{
  c4_validation: C4Analysis.validate(c4),
  dataflow_validation: DataflowAnalysis.validate(crawl_dataflow),
  workflow_validation: WorkflowAnalysis.validate(crawl_workflow),
  erd_validation: ERDAnalysis.validate(erd),
  threat_model_validation: ThreatAnalysis.validate(threat_model),
  requirement_coverage: RequirementAnalysis.coverage(requirements),
  high_risk_requirement_gaps: RequirementAnalysis.high_risk_gaps(requirements)
}

11. Tradeoffs

12. Open Questions

• What user-agent string and contact policy should the crawler publish?
• Are redirects allowed across domains, and how many redirects should be followed?
• What is the maximum page size to download and store?
• Should raw HTML be retained indefinitely or only recent crawl metadata?
• Are target sites allowed to be crawled concurrently if they share the same registered domain?

13. Final Design Summary

The design uses a small, horizontally scalable crawler system: a scheduled seed loader reads daily S3 files, normalizes URLs, enforces per-day dedupe in Redis, and pushes crawl jobs into a frontier queue. Worker pools enforce robots and per-domain politeness, fetch pages, persist outcomes to PostgreSQL, and route repeated failures to a dead-letter path.

The important release concerns are polite crawling behavior, bounded retries, database write pressure, and careful handling of untrusted target responses.

14. LLM Review Prompt

Use this Livebook as the source of truth. Review the distributed crawler design for:

• politeness bottlenecks and robots.txt edge cases;
• SSRF-like redirect or DNS abuse risks;
• retry storm and poison URL handling;
• PostgreSQL write and storage pressure;
• dedupe correctness across crawl dates;
• unclear requirements or out-of-scope assumptions;
• places where the design is over-engineered for 10,000 URLs/day.

When responding, produce a prioritized risk list, suggested design changes, open stakeholder questions, and a concise recommendation.