One Query Language for Every Platform's Transparency Report

It started as one dashboard. Last spring I built a cross-platform view of EU DSA transparency data for the 25 designated Very Large Online Platforms, and then filled in all nine report tables. Behind it sat a small FastAPI service. Over the months since, that service quietly became the more interesting artifact — because I kept feeding it other reports, and the same query interface kept absorbing them.

It now spans more than two dozen distinct transparency datasets, and they are all reachable through a single query model. That is the whole point.

The problem is shape, not scarcity

Platform transparency data isn't scarce — it's fragmented. Every company publishes in its own shape: Meta ships its enforcement figures through a GraphQL feed, TikTok drops cumulative CSVs, Apple posts wide-sparse spreadsheets, Discord and Snap label sub-tables inside a workbook, and a dozen regulators each mandate their own template. A researcher who wants to compare government removal requests across Google, Microsoft, LinkedIn, TikTok, and Discord has to learn five formats before asking a single question.

Comparability is the thing transparency reporting is supposed to deliver and almost never does. So the project's real work isn't the web UI — it's normalising all of that into one tidy-long schema and putting one query language in front of it.

What's in it now

The datasets group into a few families:

Content-moderation reports. The EU DSA harmonised template — the 25 VLOPs plus 43 non-VLOP platforms that file the same forms — over a shared nine-table star schema, alongside the big voluntary reports: Meta's Community Standards Enforcement Report and TikTok's Community Guidelines Enforcement Report.

Government & legal request reports. Google (both content removals and user-data requests), Microsoft's Law Enforcement Requests Report, LinkedIn, TikTok's government & legal requests, Discord, Apple, GitHub, and Snap — the per-country demand-and-disclosure data these companies publish half-yearly.

National transparency laws. India's IT Rules monthly compliance reports, Korea's Naver and Kakao filings (plus the Network Act illegal-content reports), Taiwan's Anti-Fraud Act data, Türkiye's Law No. 5651 reports, Singapore's IMDA Online Safety Code, Japan's new Information Distribution Platform Act (情プラ法) statistics, Texas HB 20 and Austria's KoPl-G, and China's 12377 and 12321 hotline figures.

Emerging EU duties and narrative filings. The Terrorist Content Online Regulation, the AI Act's training-data transparency summaries, Google's Traffic & Disruptions catalogue of internet shutdowns, Android's malware-rate data — plus full-text-searchable prose filings from New York's and California's terms-of-service laws.

One query language, no SQL

Everything is reachable the same way. The query interface is modelled on the TikTok Research API: you name a table and then describe filters, group-bys, and aggregates as structured parameters. No caller ever writes SQL — the server validates every field against that table's fixed registry and compiles it to a single parameterised SELECT. Long queries run on the async 202-plus-poll pattern, and results come back as JSON or CSV.

Two seams make it more usable than raw parameters. There's a natural-language "ask" box — an LLM translates a plain-English question into the same structured query, which then goes through the exact same validation boundary, so the model proposes and the validator disposes. And there's an MCP server, so an AI agent can discover the tables, run a query, and poll for the result without me hand-building a client.

The hard part is the caveats

Wiring up datasets is easy. Not lying with them is hard. Almost every source carries a footnote that a naïve aggregate would trample: national-security figures are published as banded ranges, not exact counts, so they can't be summed; prevalence and proactive-detection rates are fractions you must never add up; the DSA template embeds an aggregate "total" row right next to its own breakdown rows, so a careless SUM double-counts; and the same template row arrives in a dozen EU languages that have to be reconciled to one canonical key.

I've tried to encode those caveats as guardrails rather than bury them — the query layer warns when you sum a rate or forget to pin a dimension, and the methodology is written down rather than assumed. The goal isn't a bigger pile of numbers. It's numbers you can actually compare and cite.

The live dashboard and every dataset page are at transparency.kieranmaynard.com, with the interactive API documented at /docs. If there's a transparency report you'd want to query alongside these, I'm always looking for the next one to fold in.