The changelog engine: turn git diffs into a civic changelog

The diff is the product — now literally. generate_changelog.py reads a git diff of the entity tree between any two refs and says, in plain language, what changed in the government: entities added / modified / removed, grouped by type and by jurisdiction, dated from the commit. CHANGELOG.md is seeded from the real history so far: - initial load: +11,285 people across 51 states - institutional gap: +233 bodies, ~532 people enriched - elections + demographics: +2,494 candidates, +3,494 jurisdictions (TX +541, FL +396, CA +341 ...) On an initial load every entity is an addition; on a re-sync of the same source, only real government changes will surface — which is the whole point. When nothing changed, it says so. make changelog [BASE=.. HEAD=..]. No third-party deps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 08:28:50 -04:00
parent aefa65f5b3
commit 2e5197d1a3
3 changed files with 376 additions and 0 deletions
@@ -0,0 +1,199 @@
+# Government Changelog — 2026-07-03
+
+*Diff `6b3249ec..5b0712c3` over the mirror.*
+
+## Summary
+
+- **Candidate**: +2494
+- **Jurisdiction**: +3494
+
+## By jurisdiction
+
+- **TX**: +541 (249 candidates, 292 jurisdictions)
+- **FL**: +396 (329 candidates, 67 jurisdictions)
+- **CA**: +341 (231 candidates, 110 jurisdictions)
+- **GA**: +263 (104 candidates, 159 jurisdictions)
+- **IL**: +225 (116 candidates, 109 jurisdictions)
+- **VA**: +219 (75 candidates, 144 jurisdictions)
+- **NY**: +203 (115 candidates, 88 jurisdictions)
+- **NC**: +187 (73 candidates, 114 jurisdictions)
+- **KY**: +167 (41 candidates, 126 jurisdictions)
+- **OH**: +166 (63 candidates, 103 jurisdictions)
+- **MI**: +160 (64 candidates, 96 jurisdictions)
+- **MO**: +153 (30 candidates, 123 jurisdictions)
+- **PA**: +142 (58 candidates, 84 jurisdictions)
+- **IN**: +136 (35 candidates, 101 jurisdictions)
+- **IA**: +135 (36 candidates, 99 jurisdictions)
+- **TN**: +134 (30 candidates, 104 jurisdictions)
+- **MN**: +133 (38 candidates, 95 jurisdictions)
+- **KS**: +126 (17 candidates, 109 jurisdictions)
+- **NE**: +116 (20 candidates, 96 jurisdictions)
+- **OK**: +114 (32 candidates, 82 jurisdictions)
+- **WI**: +113 (33 candidates, 80 jurisdictions)
+- **CO**: +112 (46 candidates, 66 jurisdictions)
+- **AL**: +109 (35 candidates, 74 jurisdictions)
+- **NJ**: +109 (76 candidates, 33 jurisdictions)
+- **LA**: +108 (38 candidates, 70 jurisdictions)
+- **MS**: +104 (18 candidates, 86 jurisdictions)
+- **SC**: +103 (50 candidates, 53 jurisdictions)
+- **AR**: +94 (15 candidates, 79 jurisdictions)
+- **MT**: +79 (21 candidates, 58 jurisdictions)
+- **WA**: +79 (30 candidates, 49 jurisdictions)
+- **MD**: +75 (43 candidates, 32 jurisdictions)
+- **SD**: +75 (9 candidates, 66 jurisdictions)
+- **WV**: +73 (16 candidates, 57 jurisdictions)
+- **AZ**: +68 (44 candidates, 24 jurisdictions)
+- **OR**: +64 (22 candidates, 42 jurisdictions)
+- **ID**: +58 (14 candidates, 44 jurisdictions)
+- **ND**: +57 (3 candidates, 54 jurisdictions)
+- **MA**: +56 (33 candidates, 23 jurisdictions)
+- **NV**: +56 (35 candidates, 21 jurisdictions)
+- **UT**: +55 (22 candidates, 33 jurisdictions)
+- **NM**: +47 (11 candidates, 36 jurisdictions)
+- **ME**: +42 (24 candidates, 18 jurisdictions)
+- **NH**: +35 (23 candidates, 12 jurisdictions)
+- **WY**: +35 (11 candidates, 24 jurisdictions)
+- **AK**: +34 (6 candidates, 28 jurisdictions)
+- **CT**: +21 (21 candidates)
+- **VT**: +17 (2 candidates, 15 jurisdictions)
+- **RI**: +14 (7 candidates, 7 jurisdictions)
+- **DC**: +11 (10 candidates, 1 jurisdiction)
+- **HI**: +10 (5 candidates, 5 jurisdictions)
+- **DE**: +9 (6 candidates, 3 jurisdictions)
+- **GU**: +3 (3 candidates)
+- **VI**: +3 (3 candidates)
+- **MP**: +2 (2 candidates)
+- **AS**: +1 (1 candidate)
+
+*5988 entity changes total.*
+
+---
+
+# Government Changelog — 2026-07-03
+
+*Diff `1545fcf5..d6874ca8` over the mirror.*
+
+## Summary
+
+- **Person**: ~532
+- **Body**: +233
+
+## New bodies
+
+- Executive Office Of The President Of The United States (chamber)
+- House Africa (subcommittee)
+- House Agriculture (committee)
+- House Agriculture Rural Development Food And Drug Administration And Related Agencies (subcommittee)
+- House Appropriations (committee)
+- House Armed Services (committee)
+- House Aviation (subcommittee)
+- House Border Security And Enforcement (subcommittee)
+- House Capital Markets (subcommittee)
+- House Central Intelligence Agency (subcommittee)
+- House Coast Guard And Maritime Transportation (subcommittee)
+- House Commerce Justice Science And Related Agencies (subcommittee)
+- House Commerce Manufacturing And Trade (subcommittee)
+- House Commodity Markets Digital Assets And Rural Development (subcommittee)
+- House Communications And Technology (subcommittee)
+- House Conservation Research And Biotechnology (subcommittee)
+- House Contracting And Infrastructure (subcommittee)
+- House Counterterrorism And Intelligence (subcommittee)
+- House Courts Intellectual Property Artificial Intelligence And The Internet (subcommittee)
+- House Crime And Federal Government Surveillance (subcommittee)
+- House Cyber Information Technologies And Innovation (subcommittee)
+- House Cybersecurity And Infrastructure Protection (subcommittee)
+- House Cybersecurity Information Technology And Government Innovation (subcommittee)
+- House Defense (subcommittee)
+- House Defense Intelligence And Overhead Architecture (subcommittee)
+- House Delivering On Government Efficiency (subcommittee)
+- House Digital Assets Financial Technology And Artificial Intelligence (subcommittee)
+- House Disability Assistance And Memorial Affairs (subcommittee)
+- House Early Childhood Elementary And Secondary Education (subcommittee)
+- House East Asia And Pacific (subcommittee)
+- House Economic Development Public Buildings And Emergency Management (subcommittee)
+- House Economic Growth Energy Policy And Regulatory Affairs (subcommittee)
+- House Economic Growth Tax And Capital Access (subcommittee)
+- House Economic Opportunity (subcommittee)
+- House Education And Workforce (committee)
+- House Elections (subcommittee)
+- House Emergency Management And Technology (subcommittee)
+- House Energy (subcommittee)
+- House Energy (subcommittee)
+- House Energy And Commerce (committee)
+- … and 193 more
+
+## By jurisdiction
+
+- **US (federal)**: +233 ~532 (233 bodies)
+
+*765 entity changes total.*
+
+---
+
+# Government Changelog — 2026-07-03
+
+*Diff `710405d7..1545fcf5` over the mirror.*
+
+## Summary
+
+- **Person**: +11285
+
+## By jurisdiction
+
+- **FL**: +3344 (3344 people)
+- **US (federal)**: +542 (542 people)
+- **NH**: +416 (416 people)
+- **PA**: +256 (256 people)
+- **GA**: +237 (237 people)
+- **NY**: +216 (216 people)
+- **MA**: +205 (205 people)
+- **MN**: +204 (204 people)
+- **MO**: +195 (195 people)
+- **CT**: +191 (191 people)
+- **MD**: +191 (191 people)
+- **ME**: +189 (189 people)
+- **TX**: +185 (185 people)
+- **VT**: +184 (184 people)
+- **IL**: +179 (179 people)
+- **MS**: +177 (177 people)
+- **SC**: +174 (174 people)
+- **NC**: +173 (173 people)
+- **KS**: +169 (169 people)
+- **IN**: +154 (154 people)
+- **MT**: +153 (153 people)
+- **IA**: +152 (152 people)
+- **MI**: +151 (151 people)
+- **WA**: +151 (151 people)
+- **OK**: +150 (150 people)
+- **LA**: +147 (147 people)
+- **AL**: +146 (146 people)
+- **ND**: +144 (144 people)
+- **VA**: +142 (142 people)
+- **KY**: +141 (141 people)
+- **AR**: +140 (140 people)
+- **OH**: +136 (136 people)
+- **WV**: +136 (136 people)
+- **WI**: +135 (135 people)
+- **TN**: +134 (134 people)
+- **CA**: +124 (124 people)
+- **NJ**: +123 (123 people)
+- **RI**: +117 (117 people)
+- **NM**: +115 (115 people)
+- **SD**: +109 (109 people)
+- **ID**: +108 (108 people)
+- **UT**: +106 (106 people)
+- **CO**: +103 (103 people)
+- **WY**: +96 (96 people)
+- **AZ**: +93 (93 people)
+- **OR**: +93 (93 people)
+- **PR**: +81 (81 people)
+- **HI**: +79 (79 people)
+- **NV**: +67 (67 people)
+- **DE**: +65 (65 people)
+- **AK**: +63 (63 people)
+- **NE**: +4 (4 people)
+
+*11285 entity changes total.*
+
+---
+
@@ -9,4 +9,7 @@ build:        ## Regenerate the entity tree from raw exports in data/
 validate:     ## Check every entity file: OKF, schema, and link integrity
 	python3 scripts/validate.py

+changelog:    ## What changed in the government since the last commit (BASE/HEAD overridable)
+	python3 scripts/generate_changelog.py $(BASE) $(HEAD)
+
 check: build validate  ## Build, then validate — the full gate
@@ -0,0 +1,174 @@
+#!/usr/bin/env python3
+"""Turn a git diff of the entity tree into a human-readable civic changelog.
+
+The diff is the product. Every sync that changes the mirror leaves a git
+diff; this reads that diff and says, in plain language, what changed in the
+government — added, modified, and removed entities, grouped by type and by
+jurisdiction.
+
+Usage:
+  python scripts/generate_changelog.py [BASE] [HEAD]      # default HEAD~1..HEAD
+  python scripts/generate_changelog.py <BASE> <HEAD> --write   # prepend CHANGELOG.md
+
+With no data changes between the refs, it says so. On an initial load every
+entity is an addition; on a re-sync of the same source, only real government
+changes surface — which is the point.
+"""
+
+import re
+import subprocess
+import sys
+from collections import defaultdict
+
+PREFIX = "data/jurisdictions/"
+
+
+def git(*args):
+    return subprocess.run(["git", *args], capture_output=True, text=True,
+                          check=True).stdout
+
+
+def classify(path):
+    """(entity type, jurisdiction label) from a repo path — fast, no file reads."""
+    if not path.startswith(PREFIX):
+        return None, None
+    rel = path[len(PREFIX):]
+    # jurisdiction label
+    m = re.match(r"us/states/([a-z]{2})/", rel)
+    juris = m.group(1).upper() if m else ("US (federal)" if rel.startswith("us/") else "?")
+    # entity type
+    if "/candidates/" in rel:
+        etype = "Candidate"
+    elif "/bodies/" in rel:
+        etype = "Body"
+    elif rel.endswith("/index.md") or "/districts/" in rel:
+        etype = "Jurisdiction"
+    elif "/people/" in rel:
+        etype = "Person"
+    else:
+        etype = "Other"
+    return etype, juris
+
+
+PLURAL = {"Person": "people", "Body": "bodies", "Candidate": "candidates",
+          "Jurisdiction": "jurisdictions", "Other": "files"}
+
+
+def pluralize(etype, n):
+    return PLURAL.get(etype, etype.lower() + "s") if n != 1 else etype.lower()
+
+
+def nice_body(path):
+    """A readable name for a body file, derived from its path."""
+    slug = path.rsplit("/", 1)[-1][:-3]
+    kind = "subcommittee" if "/subcommittees/" in path else \
+           "committee" if "/committees/" in path else "chamber"
+    m = re.search(r"us/bodies/([a-z]+)/", path)
+    chamber = m.group(1).title() + " " if m and kind != "chamber" else ""
+    return f"{chamber}{slug.replace('-', ' ').title()} ({kind})"
+
+
+def main():
+    argv = [a for a in sys.argv[1:] if not a.startswith("--")]
+    write = "--write" in sys.argv
+    base = argv[0] if len(argv) > 0 else "HEAD~1"
+    head = argv[1] if len(argv) > 1 else "HEAD"
+
+    status = git("diff", "--name-status", f"{base}..{head}", "--", PREFIX)
+    date = git("show", "-s", "--format=%cs", head).strip()
+    short = git("rev-parse", "--short", base).strip() + ".." + \
+        git("rev-parse", "--short", head).strip()
+
+    # tallies[etype][op] = count ; per_state[state][op] = count
+    tallies = defaultdict(lambda: defaultdict(int))
+    per_state = defaultdict(lambda: defaultdict(int))
+    added_bodies = []
+    added_by_state = defaultdict(lambda: defaultdict(int))  # state -> etype -> n added
+
+    for line in status.splitlines():
+        parts = line.split("\t")
+        code = parts[0][0]                      # A / M / D / R
+        path = parts[-1]                        # new path for renames
+        op = {"A": "added", "M": "modified", "D": "removed", "R": "modified"}.get(code)
+        if not op:
+            continue
+        etype, juris = classify(path)
+        if etype is None:
+            continue
+        tallies[etype][op] += 1
+        per_state[juris][op] += 1
+        if op == "added":
+            added_by_state[juris][etype] += 1
+            if etype == "Body":
+                added_bodies.append(nice_body(path))
+
+    total = sum(sum(v.values()) for v in tallies.values())
+
+    out = []
+    out.append(f"# Government Changelog — {date}")
+    out.append("")
+    out.append(f"*Diff `{short}` over the mirror.*")
+    out.append("")
+    if total == 0:
+        out.append("No entity changes in this range — the government, as mirrored, held still.")
+        emit(out, write)
+        return 0
+
+    # summary line
+    def phrase(t):
+        d = tallies[t]
+        bits = [f"+{d['added']}" if d["added"] else "",
+                f"~{d['modified']}" if d["modified"] else "",
+                f"−{d['removed']}" if d["removed"] else ""]
+        return " ".join(b for b in bits if b)
+
+    out.append("## Summary")
+    out.append("")
+    for t in ("Person", "Body", "Candidate", "Jurisdiction", "Other"):
+        if t in tallies:
+            out.append(f"- **{t}**: {phrase(t)}")
+    out.append("")
+
+    # bodies get named — they are few and load-bearing
+    if added_bodies:
+        out.append("## New bodies")
+        out.append("")
+        for b in sorted(added_bodies)[:40]:
+            out.append(f"- {b}")
+        if len(added_bodies) > 40:
+            out.append(f"- … and {len(added_bodies) - 40} more")
+        out.append("")
+
+    # jurisdiction rollup
+    out.append("## By jurisdiction")
+    out.append("")
+    for juris in sorted(per_state, key=lambda s: (-sum(per_state[s].values()), s)):
+        d = per_state[juris]
+        bits = [f"+{d['added']}" if d["added"] else "",
+                f"~{d['modified']}" if d["modified"] else "",
+                f"−{d['removed']}" if d["removed"] else ""]
+        detail = added_by_state.get(juris)
+        types = ""
+        if detail:
+            types = " (" + ", ".join(f"{n} {pluralize(t, n)}"
+                                     for t, n in sorted(detail.items())) + ")"
+        out.append(f"- **{juris}**: {' '.join(b for b in bits if b)}{types}")
+    out.append("")
+    out.append(f"*{total} entity changes total.*")
+
+    emit(out, write)
+    return 0
+
+
+def emit(lines, write):
+    text = "\n".join(lines) + "\n"
+    print(text)
+    if write:
+        from pathlib import Path
+        cl = Path(__file__).resolve().parent.parent / "CHANGELOG.md"
+        prev = cl.read_text() if cl.exists() else ""
+        cl.write_text(text + "\n---\n\n" + prev)
+
+
+if __name__ == "__main__":
+    sys.exit(main())