Pipeline Architecture

The watchdog is structured around two independent pipelines that share a common orchestration layer. Prefect handles scheduling, retries, and failure alerting — the same patterns used in production data engineering environments. dlt manages the schema evolution and incremental loading so that new SEB meeting records merge cleanly without duplicates.

DuckDB sits at the analytical layer. Because Georgia's civic datasets are mid-scale (thousands to low millions of rows), DuckDB handles the analytical queries locally without spinning up cloud infrastructure — making the whole system runnable by a single organizer on a laptop.

Teaching Dimension

This repo was explicitly built to be readable. Every pipeline has inline comments explaining the "why," not just the "what." The goal is for a data analyst who has never used Prefect or dlt to follow the code and understand the engineering decisions — especially around schema handling, idempotency, and incremental loads.

Data Architecture

The core data challenge was normalizing precinct data from three separate county election websites — each with different formats, address standards, and update cadences. I wrote ingestion scripts to clean, normalize, and join the datasets into a single source of truth, then built lookup logic that handles partial address matches and fuzzy search so voters aren't penalized for minor typos.

Design for Real Use

This tool is used by Common Cause Georgia volunteers doing door-to-door and phone bank work. The design priorities were mobile-first, fast load time, and the voter-protection hotline always visible. The bilingual requirement shaped every UX decision — labels, error messages, and instructions all exist in both languages, not just a translate button at the top.

Structured Data Under Policy

The hardest part wasn't the mapping — it was structuring legislative timeline data so it stayed accurate as the special session dates and scenarios shifted. I built the dashboard so content sections could be updated by non-engineers, with the data layer separated from the presentation layer.

Audience: Steering Committees

This wasn't a public-facing voter tool — it was built for coalition steering committees who needed to brief organizational leadership quickly. The design prioritized clarity, printability, and information density over public-facing polish.

The Data Engineering Challenge

The raw enforcement data came in multiple formats — PDFs, CSVs, API endpoints with inconsistent schemas. The pipeline had to handle format variability gracefully, flag anomalies, and produce a clean, joinable county-level table even when source data was late or malformed. I built schema validation checks into the pipeline so the dashboard would never show stale data silently.

Coalition Impact

The dashboard became a live tool in coalition meetings — organizers could pull it up in a call and point to specific counties, compare risk levels, and prioritize rapid-response deployment. Turning a chaotic data landscape into a single decision-making interface was the core design goal.

Ingestion at Scale

The core challenge was handling 159 independent data sources — some well-structured, some scraped from PDFs, some requiring manual review. I built a tiered ingestion approach: automated pulls for counties with consistent formats, flagging logic for problematic records, and a simple review queue for edge cases. The final dataset was clean enough to power a public-facing map with zero manual intervention at query time.

Engineering for Non-Engineers

The target user for the output of this tool — the Airtable base itself — is an organizing coordinator who has never touched code. That constraint shaped every decision: field types needed to enforce data integrity invisibly, seed records needed to model correct data entry, and the base structure needed to be intuitive to someone learning it for the first time.

The CLI itself is designed to be run once by a data engineer to set up the base, then never touched again by the technical team. Infrastructure that disappears into the background and just works.