Civic Hack DC 2025 – Part 1: Making Public Comments Count – Outcomes & Archive

Note: This repository was created with the aid of Cursor using Claude 4 Sonnet, the content was workshopped by ChatGPT o3 and edited by ChatGPT 4o

🌟 Acknowledgments

We first want to thank the volunteers and participants who made this hackathon possible. The amount of work that went into this was incredible, and we are grateful for the time and effort that everyone put in.

Sponsors

• CareSet(https://careset.com/)
• Taoti Creative(https://taoti.com/)
• Prefect(https://prefect.io/)
• Thunder Compute(https://thundercompute.com/)
• TealWolf Consulting(https://tealwolf.consulting/)

Partners

• Moravian University(https://www.moravian.edu/) - Maintaining the Regulations.gov mirror dataset
• DataKindDC(https://www.meetup.com/datakind-dc/)

This hackathon was made possible by our sponsors, partners, the evaluation panel, and all participants who dedicated their time and creativity to building tools for regulatory transparency.

0 · Final Results

Table of results

• Final Results and Analysis

• CSV

• Google Sheet

Top-Ranked Projects (Weighted Score ≥ ~3.0) by Alphabetical Order

Can of Spam

• Breakdown: Median Scores: Impact 4, Novelty 3, Amplification 3, Open Source 3, Usability 3, Continuity 3
• Highlights: Built a proof-of-concept for duplicate and bot comment detection, laying groundwork for campaign analysis.

Hive-Partitioned Parquet

• Breakdown: Median Scores: Impact 4.5, Novelty 4, Amplification 4, Open Source 4, Usability 3, Continuity 3.5
• Highlights: Converted Mirrulations JSON to Hive-partitioned Parquet, enabling efficient SQL querying for large datasets.

LLM.gov

• Breakdown: Median Scores: Impact 5, Novelty 4, Amplification 4, Open Source 4, Usability 4, Continuity 4
• Highlights: Combined Retrieval-Augmented Generation (RAG) with vector embeddings for semantic search and summarization. Well-documented.

Mirrulations-CLI

• Breakdown: Median Scores: Impact 5, Novelty 3.5, Amplification 5, Open Source 5, Usability 4.5, Continuity 4
• Highlights: Packaged Mirrulations fetch and CSV tools into a polished CLI for easy installation and use, greatly improving accessibility.

Rules-Talk

• Breakdown: Median Scores: Impact 4, Novelty 4, Amplification 4.25, Open Source 3.25, Usability 4, Continuity 3.5
• Highlights: Used LLMs to cluster sentiment and topics in public comments, making regulatory feedback more digestible for non-technical stakeholders.

Taskmasters

• Breakdown: Median Scores: Impact 3.5, Novelty 3, Amplification 3, Open Source 2.5, Usability 2.5, Continuity 3.25
• Highlights: Developed a pipeline for comments, extracting text from PDFs, DOCX, and images, addressing data quality challenges.

The Scrapers

• Breakdown: Median Scores: Impact 4.5, Novelty 3.5, Amplification 3.75, Open Source 3, Usability 3, Continuity 3.25
• Highlights: Developed FCC and SEC scraping workflows, expanding the Mirrulations dataset beyond Regulations.gov.

Within-Docket Dataset

• Breakdown: Median Scores: Impact 4, Novelty 3, Amplification 4, Open Source 3, Usability 2, Continuity 3
• Highlights: Created a conceptual pipeline for mapping comment influence on regulatory changes.

Judges

• Ben Coleman – LinkedIn Professor of Computer Science at Moravian University and primary maintainer of the Mirrulations project.

• Fred Trotter – LinkedIn Healthcare Data Technologist at CMS Digital Service; healthcare informatics and open data expert.

• Evan Tung – LinkedIn Software Engineer at AWS and Civic Tech DC organizer.

• Gautami Nadkarni – LinkedIn Senior Customer Engineer at Google Cloud, focused on AI/ML and data modernization.

• Santhosh Kumar Veeramalla – LinkedIn Senior Scala Developer at Optum with deep expertise in Spark and data engineering.

• Melanie Kourbage – LinkedIn Lead Specialist at APHL; veteran in public health informatics and federal-state data systems.

• Taylor Wilson – LinkedIn VP of Applied Statistics at Reveal Global Consulting, leads DataKind DC.

• Michael Deeb – LinkedIn Principal Consultant at TealWolf, CTO of Keeplist.io, and Director at Civic Tech DC.

---

1 · Why This Repository Exists

On July 26 2025, 80 policy experts, data engineers, and civic technologists gathered at Taoti Creative to build open-source tools that unlock public-comment data from Regulations.gov and agency-specific portals.

This repo now serves as the permanent archive:

• final project snapshots
• evaluation results & slide decks
• cleaned datasets & helper scripts
• the original problem briefs (for future contributors)

Event Photos

Photo Albums:

• Photos by Alex Gurvich, Dean Eby, and the Civic Tech DC team: Album

Everything is licensed to encourage reuse and continuation.

---

2 · Key Dataset – Mirrulations 🌊

Mirrulations (MIRRor of regULATIONS.gov) is a comprehensive ecosystem developed by Moravian University Computer Science to ingest, process, store, and serve U.S. federal regulatory data from Regulations.gov. It provides a robust, scalable, and accessible way for researchers, developers, and the public to interact with this complex dataset. The system overcomes the API’s 1,000 items/hour limit by using donated API keys to maintain a continuously updated mirror — about 27 million items — including text extracted from PDFs.

Item	Details
Bucket	s3://mirrulations (AWS Open-Data)
Size	≈ 2.3 TB / 27 M items (JSON + attachments)
Docs	https://github.com/awslabs/open-data-registry/blob/main/datasets/mirrulations.yaml
CLI	mirrulations-fetch, mirrulations-query, mirrulations-csv
Contact	Prof. Ben Coleman • colemanb@moravian.edu

Mirrulations mirrors Regulations.gov hourly, bypassing the API’s 1 000-items/h throttle and extracting text from PDFs so teams can gulp data at scale.

There is a sample slice of the dataset in this repo.

---

3 · What were the problem statements?

Track	Problem - How can we ...
Entity Resolution	identify and unify organization names across dockets, accounting for aliases and inconsistent naming conventions?
Campaign Detection	detect duplicate or template-driven comment submissions, including coordinated campaigns and potential bot activity?
Position & Sentiment Analysis	extract nuanced positions and sentiments from comments beyond simple for/against categorizations?
Influence Mapping	link public comments to specific changes in final rules and identify which commenters influenced regulatory outcomes?
Docket-Level Analysis	build clear, digestible summaries and insights from tens of thousands of comments within a single docket?
Cross-Docket Analysis	map related dockets (RFI → Proposed Rule → Final Rule) and enable search across multiple agencies and rulemaking cycles?
Data Accessibility	make the mirrored Regulations.gov dataset easier to explore and analyze for researchers and non-technical stakeholders?
Agency-Specific Data	scrape, integrate, and standardize public comment data from non-Regulations.gov portals (e.g., FCC, SEC, FERC)?
Usability for Non-Technical Users	create interfaces, visualizations, or summaries that make complex regulatory data understandable to advocates, journalists, and the public?
Regulatory Document Navigation	surface and summarize the most relevant sections of lengthy, technical regulatory documents to support timely public engagement?

For more details, see the Problem Space Documentation.

---

4 · What Was Built

Track	Team Name	Description
Campaign Detection	CanOfSpam	A data analysis tool for detecting fraudulent bot comments in federal regulatory rule dockets using temporal patterns, submission metadata, and content analysis. Identifies coordinated manipulation campaigns through statistical analysis of comment timing bursts and duplicate detection. Built with Python and Marimo notebooks.
Cross-Docket Analysis & Influence Mapping	Within Docket Dataset	Links public comments to specific regulatory documents they respond to within a single docket, using metadata analysis, time-window heuristics, and semantic similarity techniques. Helps understand how public comments influence changes from proposed rules to final rules.
Data Accessibility	Hive-partitioned Parquet	Transforms regulatory data into Hive-partitioned Parquet files for fast and efficient queries using DuckDB. Enables direct querying from S3 with better performance for large-scale regulatory data analysis.
	Mirrulations CLI	Published Python package incorporating scripts from Prof. Ben Coleman to make downloading regulatory data more accessible. Easy to install via pip or use with uvx for streamlined access to Mirrulations data.
	LLM.gov (CMS Docket Assistant)	An LLM wrapper that utilizes RAG queries to answer general questions about dockets. Transforms complex JSON text into machine-readable vector embeddings stored in S3, enabling semantic search and providing a simple chat interface for non-technical users.
Data Quality & Derived Layers	Taskmasters	Extracts data from different document types (PDFs, ima