DataPulse

Live dashboard: https://www.data-pulse.my

Open in Google Colab:

🤖 AI-agent-ready — Wire DataPulse into Claude Desktop, Cursor, Cline, or any MCP-compatible client with one config block. Your agent gets

418 official Malaysian datasets — including 30 GTFS transit feeds (KTMB, Prasarana, BAS.MY) — with declared licences and an honest ten-status trust taxonomy instead of a blanket green checkmark.

→ Connect your AI agent in 30 seconds

This is DataPulse

When an AI quote is wrong, it is often wrong because the underlying data was stale, mis-licensed, or unverifiable — not because the model hallucinated. An official-looking page does not tell an agent when the dataset behind it last updated, who published it, whether it may legally be reused, or whether the observation can be reproduced by a second party.

DataPulse exists to make that uncertainty explicit. It is an open, read-only verification layer for Malaysian public data: it continuously probes

**418 official datasets**,

and publishes — for each one — machine-readable evidence about whether the source is reachable, how fresh the content is, what licence applies, how the schema behaves, and when the observation was signed.

It does not replace the official source. It documents, on an honest and reproducible basis, what the official portal states and whether that material is current, so you know what you are reusing or citing. The verification speaks for itself: every claim here is a live, checkable artefact, not a promise.

What we do, simply

• We watch the sources. A scheduled probe revisits each dataset under its declared cadence and records what it actually finds — reachability, an honest freshness signal, schema shape, record counts, and collection quirks.
• We state the truth plainly. Instead of a blanket green checkmark, each dataset carries one of ten honest health statuses (fresh, aging, stale, discontinued, degraded, browser-dependent, unreachable, unknown, unknown-freshness, reference). A dataset that cannot be proven fresh is labelled unknown-freshness — not silently treated as healthy.
• We publish evidence, not just claims. Each dated observation is signed and recorded to an immutable public log, so you can verify when DataPulse observed the source and that the record has not been altered.
• We make it machine-readable first. The whole portfolio is discoverable from one index and queryable over a read-only MCP server, so an agent receives the same freshness, licence, and provenance signal a careful human reviewer would.

Who this serves

• AI builders and agent developers, who want a model to check a Malaysian figure's freshness and licence before it cites the number — without building a bespoke integration or trusting a scraping pipeline.
• Researchers, analysts, and journalists, who need to ground coursework, a thesis, a dashboard, or a published figure in data whose currency and licence they can actually verify.
• Compliance and regulatory-monitoring teams, who must keep a tamper-evident trail that an official figure was checked at a known time before it reached a product or a public statement.
• Civic technologists and public servants, who want a transparent, reproducible view of how discoverable and reliably described public data is.

Why you can trust the verification

Three independent, checkable layers. You do not have to take DataPulse's word — you can verify each with the published public key, the public Git source record, and the public transparency log:

Layer	What it proves	How to check it yourself
Signed envelope	Each per-dataset observation is Ed25519-signed over its exact content by a key in the published registry	python3 scripts/verify_external.py
Source of record	The served observation byte-matches the versioned Git source	python3 scripts/verify_external.py
Temporal witness	The health statement carries a Rekor/Sigstore public-log inclusion proof	python3 scripts/verify_external.py

Run it yourself, from anywhere, with no checkout and no DataPulse code:

curl -fsSLO https://raw.githubusercontent.com/r3dz4r/datapulse-my/main/scripts/verify_external.py
python3 verify_external.py

See Verify DataPulse externally for the full guide, and our methodology below for how health is judged.

Verify an observation receipt

scripts/observation_verify.py verifies receipt identity, signature, key registry validity, and any available chain linkage from local files. It makes no network calls in any mode. Supplying --health <artifact> additionally reproduces the normalized artifact digest and re-derives its dataset and freshness-status claims: only health_binding: checked and artifact_claims: verified attest those claims. Without --health, the report explicitly says artifact_claims: NOT verified; use --require-health-binding when that incomplete result must fail.

For receipts with a signed <commit>@<path> artifact locator, pass --repo owner/name to print the corresponding immutable raw-GitHub URL. The repository is supplied, never guessed or hardcoded. Fetch that URL separately, then give the saved file to --health; the URL is a discovery aid, not proof until the local digest and claims check succeeds.

A verification layer is only as honest as its method, so DataPulse deliberately tells you when it cannot be sure — a source that cannot be proven current is labelled accordingly, never silently marked healthy. That is the boundary we hold: the platform proves the integrity and timing of its observations, not that an upstream government figure is semantically true. That distinction is the whole point of an evidence layer, and we do not blur it.

Dataset health

Health is reported as fresh, aging, stale, discontinued, degraded, browser-dependent, unreachable, unknown, unknown-freshness, or reference. Unknown freshness means the URL and content shape work, but neither a Last-Modified header nor a parseable content date proves when the data was updated. Reference means versioned lookup data is reachable and its record count is measured, while date-based freshness does not apply. Within the catalogue, data_type refines the reference family without changing the status: policy-reference rows (policy state that stays valid until superseded — BNM OPR is current while unchanged, not stale) and reference-current rows (lookups that must still pass freshness, such as a bank-rate table that can itself go stale) are judged by their declared policy, while plain reference rows are static. The public _trust_summary shows the distribution and explicitly counts missing freshness and row-count signals.

Discontinued — The source has stopped publishing new data. The data is frozen at the last known content date. This is not a freshness failure — it's a publisher decision.

Current distribution (_trust_summary): 143 fresh · 97 aging · 152 stale · 1 discontinued · 1 degraded · 5 browser-dependent · 1 unreachable · 4 unknown-freshness · 14 reference

Subscribe: RSS feed — get notified when dataset health changes.

Browser-dependent datasets

The current health summary identifies 5 browser-dependent sources (1.2% of the catalogue) that require a real browser to probe because their source pages render client-side JavaScript.

DataPulse uses Camofox, a self-hosted patched headless-Chromium sidecar, to probe these. The probe path is check.sh → Camofox sidecar → DOM snapshot → content-date extraction.

To enable browser probing:

1. Run the Camofox Docker sidecar on a reachable address (default http://localhost:9377). The probe script and the GitHub Actions workflow pick this up from the CAMOFOX_BASE_URL environment variable; nothing in this repo encodes a public IP.
2. Set CAMOFOX_BASE_URL to that address.
3. Restart the timer with systemctl restart datapulse-health.timer.

Without Camofox, these datasets will sit at browser-dependent — the honest status: DataPulse cannot probe them without a browser, so it says so rather than failing silently. See scripts/smoke_browser_probes.sh for isolated smoke tests.

Methodology

Topic	DataPulse's position
Health status	Ten-status taxonomy, judged by reachability + an honest freshness signal (Last-Modified, parseable content date, or declared policy) — never a fabricated green checkmark. A series that stopped publishing is discontinued (a publisher decision, frozen data), not a freshness failure.
Licence	Every dataset declares its licence machine-readably. Creative Commons Attribution 4.0 (285); MBPP Government Open Data Terms (attribution required) (1); MIT License (8); Open Government Licence (Malaysia) (115); Publisher licence not stated; portal disclaimer applies (4); Singapore Open Data Licence v1.0 (attribution required) (5). A second party can reproduce this from datapulse.json → .datasets[].licence.
Freshness cadence	Each dataset is probed on its own tiered schedule (5-minute timer, cadence-aware) — daily references, weekly fuel prices, monthly surveys, etc. Always with the human-readable steward and a stable custodian ID for publisher provenance.
Provenance	Stable custodian per dataset; signed probe attestations per observation
Observed claim	The platform proves what an official source was observed to be at a known time — it does not claim upstream data is semantically true
Read-only + lawful	Publicly available, authenticated sources only — never bypassed; rate-limited; identifies itself to sources
Verification	Fresh days are Rekor-witnessed; signed envelopes + Git source-of-record + public-log inclusion, checkable by anyone

Connect an AI agent

DataPulse exposes an AI-ready, read-only MCP server so agents can query the catalogue natively. It provides the same freshness, licence, schema-drift, and provenance evidence available to a human reviewer.

• Endpoint: https://mcp.data-pulse.my/mcp (Streamable HTTP, no auth) Graded by mcpgrade — replay with bash scripts/audit_mcpgrade.sh (pinned version, writes artifacts/mcpgrade/). The canonical tool count lives in mcp.json.

• 19 tools: search_datasets, get_dataset, get_data_passport, find_stale, find_anomalies, find_deteriorating, find_recovering, find_unreliable, find_schema_drift, check_reconciliation, get_provenance, get_evidence, verify_dataset, get_freshness_summary, verify_evidence, trust_verdict, verify_attestation, find_by_licence, usage_summary

The public endpoint serves all 19 read-only tools over the 418-dataset catalogue.

get_evidence exposes pipeline receipts; verify_evidence performs cached transport-only live checks and does not update health.

Connect from Claude Desktop:

{
  "mcpServers": {
    "datapulse-my": {
      "transport": "streamable-http",
      "url": "https://mcp.data-pulse.my/mcp"
    }
  }
}

See llms.txt for the full discovery index, and docs/mcp-deploy.md for the deployment architecture.

• LLM index
• Agent manifest
• MCP advertisement
• Sitemap
• MCP endpoint

Wire it into Claude Desktop via claude_desktop_config.json (30 seconds, no API key):

{
  "mcpServers": {
    "datapulse-my": {
      "transport": "streamable-http",
      "url": "https://mcp.data-pulse.my/mcp"
    }
  }
}

Restart Claude Desktop, confirm the hammer icon shows "datapulse-my" with the read-only tools listed above. Cursor / Cline use the same JSON in their MCP config panel.

Included datasets

418 official datasets across 44 publishers, including 30 GTFS transit feeds. Browse the published reports for plain-language health assessments, or use datapulse.json as the machine-readable index of every source, licence, health-report path, and declared refresh cadence.

Current coverage

Declared refresh cadences: annual (144); monthly (119); daily (48); as-required (35); quarterly (35); 30 seconds (14); biennial to triennial (survey years) (11); hourly (4); daily (weekdays) (2); weekly (2); daily (weekdays, 0900 MYT) (1); daily (weekdays, 1130 MYT) (1); daily (weekdays, 1200 MYT) (1); daily (weekdays, 1700 MYT) (1). Per-dataset cadence remains available in datapulse.json and each published health report.

How to use it

Start with datapulse.json to discover datasets and their official sources. Follow each health_report link for a plain-language assessment. Non-GTFS datasets also have matching machine-readable report envelopes under data/json/; the 30 GTFS transit feeds intentionally do not, and instead publish their health reports and GTFS samples.

For example, a data pipeline can inspect status, content_freshness_date, and freshness_signal_source before processing a source, while a researcher can review the known quirks before designing a collection method.

External verification

For a clone-less, independent check of the published Ed25519 dataset envelope, GitHub source parity, and Rekor/Sigstore health witness, see Verify DataPulse externally.

Every dataset in this catalogue ships with a publicly-signed Sigstore DSSE evidence receipt that an agent can verify offline, without trusting the DataPulse server. An agent (human or MCP) can obtain, for any dataset, the full health row + evidence + signed-receipt-verification in three MCP tool calls or fewer: verify_dataset → get_freshness_summary. The standard offline path is the verify_external.py command above.

For the portfolio-level health bundle, verify /signatures/health.latest.sigstore.json with the exact companion manifest at /signatures/datapulse.json. That signed-manifest snapshot is distinct from /datapulse.json, the current discovery manifest: the latter can change when generated metadata is refreshed. A valid signature proves the integrity of DataPulse's attested observation, not that upstream data is semantically true. Every refresh publishes signed bundles to the public Rekor log.

Monitoring

• The VPS datapulse-health.timer wakes every 5 minutes and runs only the datasets whose cadence tier is due.
• GitHub Actions performs a full weekly probe as a fallback and republishes the generated health, badge, feed, README, catalog snapshot, and delta artifacts.
• RSS feed — available.
• Status badges — available.
• More datasets — planned.

Adopt a dataset

Know a Malaysian public dataset that deserves dependable health metadata? Adopt it: verify its source and licence, document its schema and quirks, and submit a health report. See CONTRIBUTING.md for the expected three-file contribution model.

New contributors can start with the repository's Good first issues or propose a dataset through the GitHub issue forms. Maintainers use good first issue (yellow), adopt-a-dataset (blue), freshness-check (blue), bug (red), documentation (blue), question (purple), and wontfix (gray) to route contributions.

Licence

DataPulse is released under the MIT License. Source datasets remain subject to the licences and attribution requirements stated in their individual health reports.

Privacy

See PRIVACY.md for what DataPulse collects (transient operational logs for rate limiting and usage aggregation) and what it does not collect (no credentials, no accounts, no personal data).

Legal

DataPulse probes publicly-published open-data sources. We do not bypass authentication, CAPTCHAs, or terms-of-service restrictions. Every source we probe is publicly available without login; the data is aggregate/non-personal; and the probe respects each dataset's declared refresh frequency.

All scraping is rate-limited (5-minute cadence, dataset-tier cadence applied) and identifies itself via User-Agent. Sources we cannot probe without authentication, CAPTCHA bypass, or ToS violation are marked unreachable or browser-dependent — never silently scraped through a workaround.

If you are a data source maintainer and would like DataPulse to adjust its probe cadence, exclude a dataset, or remove it from the manifest, please open a GitHub issue or contact the maintainers.