Influence-Tools/republic-os
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Documentation: accurate README + full /docs tree

Browse Source 
The README described intent; the repo is now real. Rewrote it to reflect
what actually exists — 17,506 validated files across four entity types,
the deterministic pipeline, the commands — and added the /docs tree:

  government-as-code.md  the philosophy: git primitives -> government
  data-model.md          entity types, IDs, file format, the tree, determinism
  sources.md             where every fact comes from, and how current
  roadmap.md             done / next / later, honest about what isn't built
  contributing.md        the disciplines, and how to add a source

Plus data/sources/sources.yaml — the machine-readable source registry.

Every count, path, and source claim is grounded in the current tree, not
the aspirational brief. Internal links verified; validation still clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Fabio
2026-07-04 08:38:06 -04:00
parent 2e5197d1a3
commit 316c4e563c
7 changed files with 429 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/contributing.md
+56
View File
@@ -0,0 +1,56 @@
# Contributing
Republic OS is a mirror maintained mostly by agents, but its rules are simple enough that anyone can work in it. This page covers the disciplines that keep the mirror trustworthy and the practical steps for adding to it.
## The disciplines
Three rules govern everything, and they are won by subtraction — by declining to destroy:
1. **Never overwrite.** A change is a new commit, not a clobbered file. Deletes are diffs.
2. **Never rewrite history.** No force-push, no squash. The past is load-bearing — it *is* the changelog.
3. **Always push off-box.** One copy is a single point of failure; more is not.
And one rule that makes every diff meaningful:
4. **The build is deterministic.** Entity files change only when the government changes. Never introduce pipeline noise (run timestamps, unstable ordering, reformatting) into an entity file. If `make build` run twice produces a diff, that is a bug.
## The shape of a change
Everything flows one direction:
```
official source ──► raw export (data/*.jsonl) ──► build.py ──► entity tree
```
- **Raw first.** A new source lands in `data/<source>.jsonl` committed *as-is*, before any transformation. The origin must always be recoverable.
- **Build, don't hand-edit.** Entity files under `data/jurisdictions/` are generated. To change them, change the source or the generator — never edit a generated file by hand, or the next build will overwrite it (and you'll have created noise, not signal).
- **Validate before committing.** `make check` runs the build and then validates all files. It must pass.
## Adding a source
1. Obtain the data as JSONL (one JSON object per line) and commit it to `data/<source>.jsonl` untouched.
2. Register it in [`data/sources/sources.yaml`](../data/sources/sources.yaml) and note it in [docs/sources.md](sources.md).
3. Extend [`scripts/build.py`](../scripts/build.py) to read it and emit entities. Follow the existing patterns:
- Emit frontmatter keys in a **fixed order**; sort all iteration.
- Carry **field-level `sources`** and a `confidence` level on every record.
- Use **path-based IDs** with lowercase slugs and stable government identifiers where available.
- Reconcile to existing entities by a shared key, or by normalized name where none exists.
4. If it's a new entity type, add a JSON Schema to [`/schemas`](../schemas) — a stricter profile over OKF (required fields, property types, enums), tolerating unknown keys.
5. Run `make check`. Fix anything it reports.
6. Commit with a message that describes the *civic* change, not just the code change (see the existing history for the tone).
## Reading a change
You do not need deep git knowledge to read the mirror — four commands cover it:
```bash
git log -- <path> # the timeline of any official, body, or place
git diff <a>..<b> # what changed between two states
git show <commit> # one change, in full
git blame <path> # where each fact came from
make changelog # the same, rendered as a plain-language civic changelog
```
## A note on truth
When the mirror disagrees with an official record, the official record is right and the mirror has a bug. Report it as a finding — the gap between the model and reality is information, not embarrassment. This repository never claims to be the authority; it only makes the authority easier to read.
docs/data-model.md
+105
View File
@@ -0,0 +1,105 @@
# Data Model
## The file
Every entity is one Markdown file with YAML frontmatter. The frontmatter is the machine-readable record; the body is the human-readable page. The same file serves a journalist reading it on GitHub, an editor in an Obsidian vault, and an AI agent parsing frontmatter.
```markdown
---
type: Person
title: "Mike Johnson"
role: "U.S. House of Representatives - LA-4"
party: "R"
level: "federal"
branch: "legislative"
state: "LA"
leadership:
- role: "Speaker of the House"
body: "H"
since: "2025-01-03"
ids:
person: "fac58269-6f4a-4706-8eb4-95a549c07a41"
bioguide: "J000299"
sources:
- field: tenure
source: "House Clerk XML 2026-06-10"
- field: roles
source: "congress-legislators (unitedstates project)"
confidence: official
tags: [officeholder, federal, legislative, la]
timestamp: "2026-06-20"
---
# Mike Johnson
Current U.S. House of Representatives - LA-4 (Congressional District 4).
## Leadership
- Speaker of the House (since 2025-01-03)
```
The format conforms to the [Open Knowledge Format (OKF)](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md): only `type` is strictly required, unknown keys are tolerated, and the concept ID is the file path. Republic OS layers a stricter, schema-validated profile on top — see [`/schemas`](../schemas).
## Entity types
| `type` | Count | Home | Schema |
|---|---:|---|---|
| **Person** | 11,285 | `us/people/`, `us/states/<st>/…/people/` | [person.schema.json](../schemas/person.schema.json) |
| **Body** | 233 | `us/bodies/…` | [body.schema.json](../schemas/body.schema.json) |
| **Candidate** | 2,494 | `us/states/<st>/candidates/` | [candidate.schema.json](../schemas/candidate.schema.json) |
| **Jurisdiction** | 3,494 | county `index.md`, `us/states/<st>/districts/` | [jurisdiction.schema.json](../schemas/jurisdiction.schema.json) |
- **Person** — an officeholder, filed by the jurisdiction they serve. Federal members are enriched with their bioguide ID, leadership roles, and committee seats.
- **Body** — an institution: a chamber, committee, or subcommittee. The "repos" of the government-as-code model. Each lists its leadership; subcommittees name their parent.
- **Candidate** — someone running for office, distinct from a current officeholder. Sourced from FEC filings.
- **Jurisdiction** — a place with a government, carrying Census demographics. A county's node is its directory `index.md`; a district's is a file under `districts/`.
## The jurisdiction tree
Geography is a hierarchy: every entity sits in exactly one place in the tree. The schema is **fractal** — a county commissioner and a U.S. senator are both `Person`; a school board and the House are both `Body` — so the same shapes repeat at every level.
```
data/jurisdictions/
us/
people/ federal officeholders (542)
bodies/ House, Senate, committees, subcommittees
house/committees/…
senate/committees/…
states/
fl/
people/ FL state legislators
candidates/ 2026 federal candidates from FL
districts/ congressional districts (ACS demographics)
counties/
st-lucie/
index.md the county as a Jurisdiction node
people/ county officeholders
municipalities/
port-st-lucie/
people/ city officeholders
```
The tree is **complete on the map, uneven on the ground**: every U.S. county exists as a node (from Census data), but officeholder depth currently reaches the municipal level only in Florida. An empty directory is not a failure — it is the frontier.
A second axis — cross-cutting **issues** that link entities across the tree — is designed but not yet built. See the [roadmap](roadmap.md).
## Identifiers
- **Path is identity.** The concept ID is the file path (OKF): `us/states/fl/counties/st-lucie/people/anthony-bonna` is both the address and the ID. Paths use lowercase slugs and stable government identifiers where available.
- **Carried IDs** round-trip back to sources: Atlas UUIDs (`person`, `office`, `tenure`, `jurisdiction`), `bioguide` for federal members, `fec` for candidates, `fips` for counties.
- **Reconciliation** across sources is by normalized name where no shared key exists — for example, congress-legislators (bioguide-keyed) is joined onto Atlas officeholders (UUID-keyed) by name.
## Determinism — why the diffs are trustworthy
The build is deterministic by construction: sorted iteration, a fixed key order, one full rebuild per run. `make build` twice produces a byte-identical tree, verified by hashing every file. Entity files change **only when the government changes**, never because of pipeline noise (timestamps, ordering, formatting). This is what makes `make changelog` meaningful — a diff is always a real change, never an artifact.
## Validation
[`scripts/validate.py`](../scripts/validate.py) checks all 17,506 files with no third-party dependencies:
1. **OKF conformance** — every file has frontmatter with a non-empty `type`.
2. **Schema conformance** — required fields, property types, and enums per entity type.
3. **Link integrity** — every internal `/`-rooted markdown link resolves to a real file.
It exits non-zero on failure, so it doubles as a CI gate. Run it with `make validate`.
docs/government-as-code.md
+56
View File
@@ -0,0 +1,56 @@
# Government as Code
## The premise
Government is an operating system that has run for 250 years without version control.
The analogy is not decorative — it maps almost mechanically:
| Software | Government |
|---|---|
| Source code | Statutes, ordinances, charters, budgets |
| Running processes | Agencies, departments, courts |
| Maintainers with commit rights | Elected officials, appointed boards |
| A pull request | A bill, a rezoning petition, a budget amendment |
| Code review | Committee hearings, public comment periods |
| The merge vote | The floor vote, the commission vote |
| A release | An adopted budget, an enacted law |
| Changing the maintainers | An election |
| The changelog | **Did not exist. This repository is it.** |
Everything on the right is public. Almost none of it is *legible*. The state of government is scattered across PDFs, agenda portals, filings, and meeting videos — published, technically, but not readable, not diffable, and above all not *watchable*. A change to a zoning code and a change to a codebase are the same category of event; only one of them triggers a notification.
We cannot read government's source directly. But we can mirror its observable public state into files, and version the mirror. Git then gives us, for free, the tooling software has had for decades.
## What git's primitives mean here
| Git | In this repository |
|---|---|
| The files | The current state — who holds which office, what a place is made of, who's running. |
| A commit | An observed change, timestamped: "this member's committee assignment changed." |
| `git log` | The timeline of any official, body, or place — forever. |
| `git diff` | The change itself, in plain terms. This is the product. |
| `git blame` | The provenance of any single fact, line by line. |
| `git show` | One change, in full, on the permanent record. |
You do not need to know git deeply to read this repository. Four commands cover it: `git log` (what happened), `git diff` (what changed), `git show` (one change in full), `git blame` (where a fact came from). The agents that maintain the mirror do the writing.
## The diff is the product
The honest caveat: the mirror's history starts the day we start watching. Git's value here is entirely forward-looking — from the first commit on, *nothing changes unnoticed*. Nobody can quietly pull an item off an agenda or revise a bio; that edit is itself a diff, timestamped and permanent.
This is why the [changelog](../CHANGELOG.md) matters more than any single record. A directory of officials is useful; a running account of *what your government changed this week* is a different kind of instrument. The changelog is generated directly from git — nobody writes it, it falls out of the files changing.
For this to work, the files must be **boring**. The build is deterministic: the same entity always at the same path, keys always in the same order, identical formatting every run. Volatile bookkeeping ("we checked, nothing changed") never touches the entity files — only real government changes produce a diff. This single discipline is most of the engineering. See [the data model](data-model.md) for how it's enforced.
## Principles
- **Government should be legible.** Structure is the precondition for understanding.
- **Official sources remain the authority.** This repository is a mirror, not the source of truth. Every record cites where it came from and how confident we are.
- **Changes should be visible.** The value is not the snapshot; it's the diff between snapshots.
- **Git preserves history.** A record deleted upstream becomes a diff here, not a hole. Civic memory that survives deletion is a byproduct of refusing to overwrite.
- **Agents need structured civic memory.** The mirror is built to be read and reasoned over by software as much as by people — stable IDs, simple schemas, field-level sources, machine-diffable format.
## Not the source of truth
Worth repeating, because it is the ethical center of the project: **Republic OS is a mirror.** When it disagrees with an official record, the official record is right and the mirror has a bug — which is itself a finding, surfaced as a diff. The goal is to make public power easier to read, not to become an alternative authority over it.
docs/roadmap.md
+37
View File
@@ -0,0 +1,37 @@
# Roadmap
An honest account of what exists, what's next, and what's further out. This repository will always be evolving; there is no "finished."
## Done
- **The substrate** — repo-canonical, OKF markdown + YAML frontmatter, JSON-Schema validated, deterministic build (byte-identical on rerun).
- **Four entity types, 17,506 files** — Person (11,285), Body (233), Candidate (2,494), Jurisdiction (3,494).
- **The federal government, structurally** — every member of Congress, the chambers, 49 committees and 181 subcommittees with their leadership, and every member's committee seats.
- **The nationwide county skeleton** — all 3,131 U.S. counties as demographic nodes, plus 363 congressional districts, from Census ACS.
- **The 2026 federal candidate field** — every FEC filing as a Candidate.
- **Florida to the ground** — officeholders down to the municipal level across all 67 counties.
- **The pipeline** — `make build` / `validate` / `changelog` / `check`, standard-library only.
- **The changelog engine** — git diffs rendered as a plain-language civic changelog.
- **The Board** — a self-contained coverage map ([`viz/board.html`](../viz/board.html)).
## Next
- **Self-updating mirror** — a GitHub Action running `make check` on every push, then a scheduled job that pulls a fresh Atlas export, rebuilds, and commits the changelog automatically. This is the point where the mirror maintains itself and the daily civic changelog becomes real (the first *re-sync* diff, where only genuine government changes surface).
- **The demographic choropleth** — the Board already has ACS data for every county; shade the map by population, income, and poverty, turning it from a coverage indicator into an instrument.
- **State and local bodies** — Bodies currently exist only at the federal level. Mint them for state legislatures and, following the Florida depth, county commissions and city councils.
- **Graph export** — a loader that walks the tree and builds the relationship graph (Memgraph) as a materialized view, so the repo can rehydrate the graph rather than the reverse.
## Later
- **The issues axis** — cross-cutting `Issue` entities that link people, bodies, and places across the jurisdiction tree, so the mirror answers "who, nationwide, is touching this issue?" and not only "who governs here?"
- **The money layer** — donations, contracts, and grants as sourced event files; the flows that make influence legible (the Sankey / "circuit-breaker" views from the brief).
- **Meetings and agendas** — the highest-value local data, and the hardest: scraped from Legistar / Granicus / CivicPlus portals and, where towns publish nothing in text, from meeting video. A new agenda item is a diff with a deadline — a mission.
- **Contribution as correction** — let people propose fixes to the mirror (a wrong phone number, a missed meeting) as pull requests, making the public literal contributors to their own civic record.
## Non-goals
Deliberately out of scope, to keep the mirror honest and buildable:
- Becoming an authority. This is a mirror; official sources remain the source of truth.
- A heavy web application, database dependency, or graph-database setup in the core repository. The substrate stays flat files.
- Ingesting all of government at once. Depth follows attention, county by county.
docs/sources.md
+41
View File
@@ -0,0 +1,41 @@
# Sources
Republic OS is a mirror. Every fact in it comes from a public source, and every record carries field-level provenance and a confidence level. This page lists the sources feeding the mirror today; the machine-readable registry is [`data/sources/sources.yaml`](../data/sources/sources.yaml).
The pipeline is **repo-canonical**: raw source exports land in `data/*.jsonl` untouched, and [`scripts/build.py`](../scripts/build.py) generates the entity tree from them in one deterministic pass. Nothing is hand-edited into the tree.
## Active sources
| Source | Feeds | Raw file | Authority |
|---|---|---|---|
| **Atlas / influence.tools** | Officeholders — federal, all state legislatures, FL county + municipal | `officeholders-v2.jsonl` (11,285) | Aggregator |
| **congress-legislators** ([unitedstates project](https://github.com/unitedstates/congress-legislators)) | Bodies, leadership roles, committee memberships | `bodies.jsonl` (233), `leadership.jsonl` (28), `committee_memberships.jsonl` (3,879) | Community, authoritative |
| **FEC** (Federal Election Commission) | 2026 federal candidate filings | `fec_candidates.jsonl` (2,494) | Official |
| **Census ACS 2023** | County & congressional-district demographics | `acs_county.jsonl` (3,231), `acs_cd.jsonl` (363) | Official |
Atlas itself draws from official upstreams — House Clerk XML, Senate.gov XML, OpenStates, Census TIGER/Line geometry, and the Florida Cities partner API — and records the specific upstream in each record's `sources` block. So a Person file's provenance names not just "Atlas" but the document behind the field (e.g. `tenure: House Clerk XML 2026-06-10`).
## Provenance in every record
Each entity's frontmatter carries a `sources` list and a `confidence` level:
```yaml
sources:
- field: tenure
source: "House Clerk XML 2026-06-10"
- field: roles
source: "congress-legislators (unitedstates project)"
confidence: official
```
`confidence` is one of `official`, `reported`, `inferred`, or `unverified`. Everything in the mirror today is `official`.
## Reconciliation and its findings
Sources use different keys — Atlas uses internal UUIDs, congress-legislators uses bioguide IDs, FEC uses committee IDs, ACS uses FIPS codes. Where no shared key exists, records are reconciled by normalized name. Federal officeholders reconcile at ~99% (531 of 532 members matched to their leadership and committee data).
Disagreements between sources are not errors to hide but **findings to surface**. Where Atlas's officeholder roster names a different person for a seat than the authoritative current roster does, the mismatch stands as a visible discrepancy — the gap between the model and reality is itself information.
## Known sources not yet ingested
Named in the [technical brief](../Technical%20Brief%20—%20Government%20as%20Software.md) and [roadmap](roadmap.md), not yet in the mirror: campaign finance flows and contracts (the "money" layer), local agenda portals (Legistar / Granicus / CivicPlus), the Federal Register, lobbying registrations, and property records. These are where the mirror grows next.