Commit Graph

16 Commits

Author SHA1 Message Date
Fabio 7764501bcf Legislation axis: Bill schema, OpenStates ingest, validator + provenance
The legislative-process corpus, parallel to the U.S. Code statute corpus.
schemas/bill.schema.json + scripts/ingest_legislation.py (deterministic,
stdlib-only, reads per-state JSONL from the depot); validate.py gains
legislation/ as a third tree; Makefile 'legislation' target. Raw JSONL stays
off-repo on the depot (gitignored); the per-state SHA-256 manifest is the
committed provenance anchor. Vintage 2026-07-01.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 17:24:31 -04:00
Fabio 3d8b65299f Cities: exact GEOID placement from Atlas backfill
Atlas resolved every existing municipal officeholder to its Census place
GEOID at the source (PostGIS). build.py now prefers that exact geoid over
the name-based fuzzy resolver, so filled cities land on their node exactly
rather than by 98.8% name match. Resolved 2,217 -> 2,224; dedup against the
skeleton tightened (a few fuzzy mismatches now collapse correctly).

Note: this is the GEOID *backfill*, not the *fill* — zero net-new
officeholders; the 17,286 empty skeleton cities remain empty. Nationwide
municipal officeholder acquisition is a separate, per-state source problem.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 12:03:18 -04:00
Fabio a8594d786c Authority axis v1: mint the executive branch + link 27,804 § → office edges
The Code empowers the executive branch, but the mirror only had legislative
Bodies — every authority reference pointed at nothing. Fixed by minting the
executive branch from the Code's OWN enumerations: 15 Cabinet departments
(5 U.S.C. § 101) + 21 Executive-Schedule Level I offices (§ 5312), 42 Body
nodes under us/executive/.

Then the axis: scripts/extract_authority.py matches named office/department
references across all 59,740 sections (guarding against Deputy/Assistant/Under
subordinates) and emits data/section_authority_edges.jsonl — 27,804 edges from
17,031 sections (28.7% of the Code) to the offices they empower. build.py
renders reciprocal 'empowered by' links on each office node; the section files
are never touched. Now: click the Attorney General, see all 2,386 sections that
vest authority in it.

Named references only in v1 (cabinet-level). Deterministic; body.schema
classification enum extended (executive-department/-agency/-office); make check
green at 105,746 records.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 11:37:12 -04:00
Fabio 76b8ec33a7 Legal corpus: the complete U.S. Code (59,740 sections, all 53 titles)
Ingested titles 12–51 and 54 from OLRC USLM XML @119-100 (the whole Code
now, uniform edition; Title 53 is reserved/empty). LegalText 11,221 ->
59,740; repo total 105,704 records. Deterministic (byte-identical rerun,
verified on Title 42's 8,356 sections); make check green. make
legal-us-code default now covers every title.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 10:51:44 -04:00
Fabio 5a22e7dcbb Cities: 19,518 municipal Jurisdiction nodes (generated tree)
Regenerated data/jurisdictions with the municipal layer: 10,253 cities,
4,331 towns, 3,716 villages, 1,218 boroughs. Jurisdiction records
10,390 -> 29,908.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 10:32:56 -04:00
Fabio 00a184bb3c Legal corpus: U.S. Code titles 1–11 from pinned OLRC XML (11,050 sections)
Raw OLRC USLM XML zips @ release 119-100 (retrieved 2026-07-04 via
Atlas depot), ingested with the standard pipeline: raw snapshot ->
per-section OKF markdown -> manifest + checksums. Title 52 untouched.
LegalText: 171 -> 11,221. Titles 12-54 await a clean OLRC retry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 09:52:37 -04:00
Fabio 1beeaab2db Graph: national county↔district↔legislator authority layer
The first edges in the repo. Every US county now links to the districts it
sits in, weighted by area overlap, and every legislator links to the district
they represent — county → district → representative, traversable.

Data (PostGIS, no Atlas dependency — computed directly from geometry):
- data/county_district_edges.jsonl — 16,328 weighted edges. area_weight =
  ST_Area(ST_Intersection(county, district))/ST_Area(county) over ST_MakeValid
  Census TIGER 2024 geometries, overlaps <0.5% dropped. Uses area-intersection,
  NOT centroid-in-polygon, so urban districts that carve through counties
  (e.g. GA CD-5 / Atlanta across Clayton+DeKalb+Fulton) are captured, not
  silently dropped. Per-county weights sum to ~1.0 per district type.

build.py:
- ~6,896 new district nodes: 435 CD (ACS demographics where available; sparse
  for the 8 states missing from acs_cd) + 4,927 state-house + 1,897 state-senate
  (geometry-only). GEOID/number-keyed, stable.
- county index.md gains a `districts:` block (frontmatter) + a ## Districts
  section (body links, so link-integrity checks them).
- legislators + US House members gain `represents:` pointing at their district
  node; edge stored once on the person, inverse left to the view.
- schemas/jurisdiction: classification enum + chamber/districts fields.

Honest gaps (logged, not hidden): ID + NH have no state-house geometry, so
their house districts get nodes (represents resolves) but no county edges;
CD nodes for acs_cd-missing states are demographically sparse.

Deterministic (byte-identical rebuild); validate.py passes 26,617 records,
link-integrity clean (every district/represents link resolves).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 22:28:08 -04:00
Fabio 199d8b37d3 Municipals: nest 1,823 city officeholders under their real counties
Pull Atlas's place→county crosswalk (Census TIGER 2024 places × county
geometries, 32,041 places) into the build as a committed raw input, and
use it to resolve municipal officeholders whose county was unknown.

- data/place_county_crosswalk.jsonl — new raw input (place_geoid → county_geoid)
- build.py — place-name+state → county_geoid → ACS county slug as a fallback
  in county_slug(); city_slug() generalized to non-FL bare place labels
- data/sources/sources.yaml — provenance entry for the crosswalk

_unresolved municipals: 1919 → 58. The remainder are genuine: CT planning
regions (post-county), county-level rows mislabeled municipal upstream,
consolidated city-counties (null label), Wikidata-QID placeholder names,
and ambiguous same-name places — left unresolved rather than guessed.

Join key is county_geoid (5-digit FIPS); the directory slug is taken from
the ACS county node so municipals nest under the exact existing county dir.
Deterministic (byte-identical rebuild); validate.py passes 19,721 records.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 20:23:57 -04:00
Fabio ad7607856d Legal corpus: seed U.S. Code Title 52 2026-07-04 19:49:09 -04:00
Fabio 7a2cc64362 Sync: officeholders v3 — fresh export from live Atlas DB (13,329 tenures)
Re-exported directly from Postgres (10.0.0.116), not Atlas's introspection,
which under-reported the roster (its v3 export had 11,473, missing the
restored Florida League of Cities municipal set). Live DB current tenures:
state 7,561 · municipal 4,009 · county 1,217 · federal 542.

- build.py now reads officeholders-v3.jsonl (was v2); dataset vintage 2026-07-04
- fixed one corrupt start_date at source-export (21021-12-03 -> 2021-12-03,
  Lorraine Borowski, Mayor of Decorah IA)
- scripts/build_viz.py: committed, deterministic generator for the Board's
  county_data.json / county_detail.json (previously built ad-hoc)
- generate_changelog.py: ignore files whose only change is the vintage stamp,
  so the diff reflects government change, not re-export bookkeeping

Person files 11,285 -> 13,329. Non-FL municipal officials (1,903) are in the
tree but not yet county-mappable (place->county crosswalk still NULL).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 18:56:37 -04:00
Fabio 316c4e563c Documentation: accurate README + full /docs tree
The README described intent; the repo is now real. Rewrote it to reflect
what actually exists — 17,506 validated files across four entity types,
the deterministic pipeline, the commands — and added the /docs tree:

  government-as-code.md  the philosophy: git primitives -> government
  data-model.md          entity types, IDs, file format, the tree, determinism
  sources.md             where every fact comes from, and how current
  roadmap.md             done / next / later, honest about what isn't built
  contributing.md        the disciplines, and how to add a source

Plus data/sources/sources.yaml — the machine-readable source registry.

Every count, path, and source claim is grounded in the current tree, not
the aspirational brief. Internal links verified; validation still clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 08:38:06 -04:00
Fabio 5b0712c31f Add elections + demographics layer: candidates and jurisdictions
Two new entity types, built deterministically from the raw exports:

  Candidate (2,494)     — 2026 federal candidates as files under
                          us/states/<st>/candidates/, with office,
                          district, party, incumbent/challenger stance,
                          campaign committee, FEC id.

  Jurisdiction (3,494)  — demographic nodes carrying ACS 2023 data:
                          3,131 counties (deduped from 3,231 mixed-format
                          fips keys) + 363 congressional districts.

This completes the county skeleton nationwide: every US county is now a
browsable node with population, income, poverty, race, education and
housing — even where no officeholders are mapped yet. In Florida the
county index.md merges into the existing dir and reports its officeholder
count alongside demographics (St. Lucie: 33 officeholders + ACS profile).

build.py now reads seven raw inputs in one deterministic pass
(byte-identical on rerun). The mirror now answers: who holds power, who
runs this institution, who is running, and what each place is made of.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 23:35:00 -04:00
Fabio 6b3249ecc0 Raw exports: FEC candidates + ACS demographics
Three new source datasets from Atlas, committed as-is before any build:

  fec_candidates.jsonl  2,494  2026 federal candidates (officeholders.fec_data)
  acs_county.jsonl      3,231  Census ACS 2023 by county (~100 dup keys to dedupe)
  acs_cd.jsonl            363  Census ACS 2023 by congressional district

A new axis for the mirror: who is running (candidates) and the
demographics of each place (ACS). Keyed on fec_id / state+district /
county_fips — no person links yet. Raw first; the build into Candidate
and Jurisdiction entities follows.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 23:30:05 -04:00
Fabio d6874ca8a6 Close the institutional gap: bodies, committees, leadership
The seat data answered 'who represents this place?' but not 'who runs
this institution?' — the Speaker of the House existed only as the rep
for LA-4. This adds the missing axis of institutional power.

New Body entities (233): the U.S. House, U.S. Senate, Executive Office,
49 committees, 181 subcommittees — each with its leadership.

Federal people (532/542) enriched with bioguide IDs, leadership roles,
and committee seats, reconciled from congress-legislators by name.
Mike Johnson is now Speaker of the House, not just LA-4.

Sources ingested directly into the repo (repo-canonical): Atlas
supplied bodies/leadership/committee_memberships via congress-legislators,
committed raw in data/. build.py supersedes convert_officeholders.py and
reads all four inputs in one deterministic pass (byte-identical on rerun).

The 10 unenriched federal records are a finding, not a bug: vacancies,
the President, and ~5 seats where Atlas's roster disagrees with the
authoritative current roster — model-vs-reality discrepancies to surface.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 20:51:07 -04:00
Fabio 1545fcf5ca Raw JSONL to structured tree: 11,285 entity files
One record, one file. The jurisdiction tree is now real:

  us/people/                          542 federal
  us/states/<xx>/people/            7,561 state
  us/states/fl/counties/<c>/people/ 1,058 county
  .../municipalities/<m>/people/    2,124 municipal (67 FL counties)

Each file is OKF markdown + YAML frontmatter: fixed key order,
field-level source provenance, Atlas UUIDs for round-tripping,
deterministic paths and slugs. Converter passes the determinism
test: two runs, byte-identical tree.

Generated by scripts/convert_officeholders.py from
data/officeholders-v2.jsonl (unchanged).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 18:08:31 -04:00
Fabio 710405d710 Raw Atlas export: officeholders v2 (11,285 records)
The only surviving copy of the dataset after the DB loss on
10.0.0.116 — committed as-is, before any transformation.
Federal (536), state (7,560), county (1,057), municipal (2,118)
current officeholders. Source: Atlas via /depot/exports,
generated 2026-07-03.

Raw first, refined later: the conversion into structured entity
files will itself be a visible diff.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 17:31:05 -04:00