Evidence methodology
PolicyWatcher’s operational framework for data provenance, AI constraints, check history, and review accountability.
GRC and AI Governance work requires evidence-first verification. This page explains how PolicyWatcher records configured sources, maps changes, constrains AI processing, and exposes limitations.
PolicyWatcher tracks and maps public policy information for risk screening, benchmarking and lifecycle review. Its scope is limited to public disclosures and policy text:
- Evaluates public disclosures and terms of service text, not internal company operations.
- Does NOT certify that a company complies with its policies, or that the policies comply with applicable laws.
- Must not be treated as legal advice, formal auditing, or compliance validation.
- Is intended for risk screening, benchmarking, and policy lifecycle tracking.
To avoid fabricated data and expose source failures clearly, the ingestion pipeline uses a prioritized retrieval cascade:
- Primary source: direct HTTP retrieval of the configured policy URL with browser-like headers, redirect validation, retries, and timeout limits.
- Protocol fallback: explicit HTTP/2 retrieval is attempted for providers that reject HTTP/1.1 or return short SPA shells.
- Rendered fallback: when configured, a separate VPS renderer executes a headless browser fetch for script-rendered pages. It is protected by bearer auth and validates initial URLs, redirects, and subresource requests against SSRF rules.
- Provider-challenge handling: if an official page is protected by anti-bot or WAF controls, the renderer result is still treated as insufficient evidence unless usable policy text is retrieved. The source stays suspended until a verified baseline, official PDF, or traced admin review confirms it.
- Archive fallback: if live retrieval fails, the pipeline may try Wayback Machine and Common Crawl snapshots where available.
- Freshness guard: archived snapshots older than the last successful check are rejected so an old cached copy cannot be registered as a new policy change.
- Shared-acquisition guard: records that resolve to the same normalized retrieval URL share one network acquisition per scan. Policy-specific comparison and evidence records remain separate, and the strictest applicable freshness floor is used.
- Source identity separation: the public canonical policy URL remains the citation target, while an optional official machine-readable mirror or PDF can be configured as the retrieval URL.
- Strategy diagnostics: each retrieval attempt records the strategy used, outcome, HTTP status where available, rejection/failure reason, and whether the system escalated to the next fallback.
- Operational cause taxonomy: retrieval outcomes use bounded reason codes for access blocks, rate limits, upstream availability, timeouts, incomplete or invalid content, stale archives, configuration and routing decisions.
- Host-drift guard: live redirects to a different host are marked for review instead of being accepted as baseline evidence.
- Path-drift guard: configured policy URLs that resolve to a same-host homepage or non-policy landing page are rejected instead of becoming baseline evidence.
- Completeness guard: over-cap extraction is marked Partial and suspended from public evidence instead of being stored as a complete policy text.
- Batch execution: administrative scans can be limited by company slug or policy count so the first source-verification run can be resumed safely on shared hosting.
- Scan-run accounting: each full scan persists selected-record, unique-source, network, deduplication, availability and dependency metrics for operational review.
- Honest failure recording: if a page remains unreachable, the system does not create a successful version record from missing data. It updates the policy status to "Unavailable" or "Needs Review" and writes a check-log row.
- Hash fingerprinting: retrieved text records are fingerprinted with SHA-256 so later integrity checks can detect mismatches between text and hash.
Automated reviews are processed using Google Gemini models. To prevent hallucination and ensure auditability, the AI is subject to strict engineering constraints:
- Direct grounding: summaries and bullet points are generated from the retrieved/versioned text record being analysed.
- No unsupported filling: prompts instruct the model to return "Not Specified" or "Unavailable" when the document does not support a field or KPI.
- Structured mapping: categorisations are normalized against the expected analysis fields used by PolicyWatcher.
- Audit trail: persisted AI analyses reference the policy version records used for the comparison.
PolicyWatcher exposes the following traceability elements in the interface:
- Configured URL: Direct link to the source document monitored.
- Ingestion method: indication of whether the current record was seeded, directly retrieved, fetched via HTTP/2, rendered through the VPS service, or recovered through an archive source.
- Scan timestamps: policy detail views expose Last Checked and Last Successful Fetch values when recorded.
- Check logs: each scan result can be recorded with status, source, HTTP status, reason, final URL, hash, text length, and archive snapshot timestamp when an archive source is used.
- Public-evidence gate: policy snapshots and change records must be explicitly marked as public evidence before public APIs, sitemap, digests, reports, share pages, timelines, or benchmark views can expose them.
- Policy Signals Board: the public leaderboard ranks only source coverage, retrieval traceability, public baselines, and publicEvidence-gated movement. It does not certify companies, compliance, safety, internal conduct, or provider trustworthiness.
- Verified baseline establishment: the first successful source retrieval establishes or promotes one exact-hash baseline without creating a PolicyChange, AI score, or subscriber notification. Records in onboarding QA remain private. Existing source logs can no longer leave an otherwise verified policy permanently outside the public-evidence gate.
- Public suspension: when the latest fetch/update produces anomalies or insufficient evidence, the source is temporarily suspended and public views expose only the suspension notice, not the underlying analysis.
- Source remediation: official-but-blocked sources are repaired through market-specific URL mapping, official PDF/CDN evidence where available, or traced administrative review. PolicyWatcher does not promote anti-bot challenge pages, placeholders, or stale archive copies into public evidence.
- Historical-reference boundary: a stale Wayback or Common Crawl candidate may be retained as a dated continuity reference, but remains explicitly ineligible for baseline creation or change detection.
- Reliability queue: repeated failures are grouped by retrieval key, assigned a structured cause and suggested operator action, and retained until recovery or an explicit administrative resolution.
- Administrator alerting: source suspensions can generate an internal operational email with metadata and a Dataset QA link, without including policy text, scores, diffs, KPIs, or AI interpretation.
- Dataset QA control groups: source fit, retrieval evidence, public evidence gates, seeded-record boundaries, hash consistency, check-log completeness, timestamp integrity, archive timestamp coverage, KPI coverage, regional impact coverage, access logs, and subscriber hygiene are inspected before release decisions.
- Review decisions: Dataset QA issues can be marked reviewed, ignored with reason, or reopened, with append-only review-log evidence.
- Version timeline: versioned policy records remain available for reproducible comparison.
Release 4.0.0-beta.3 adds protected operational-readiness presentation and bounded measurement without changing public evidence rules or server-side authorization.
- Adaptive Workspace: users can select a session intent (Citizen, GRC / Legal, Research, Builder) and evidence depth (Snapshot, Operational, Forensic).
- Validated composition: dashboard modules come from an immutable allowlist with deterministic identities; each valid composition requires Source QA as its first module.
- Guarded interaction: direct controls and the Command Palette dispatch typed actions through an acyclic authorization graph and one canonical workspace URL/local-storage codec.
- Evidence-first sources: each registered dashboard source declares endpoint, query scope, freshness, visibility, public-evidence gate and known limitations before loading.
- Rendering and export parity: the visible filtered company list and CSV export use one view model; the export includes query identity, coverage, filters, evidence gate, limitations and release provenance.
- Accessible chart contract: supported charts define summary, table, provenance and limitations, while reduced-motion settings disable nonessential animation.
- Presentation-only adaptation: density, module priority, dashboard emphasis, and URL parameters may change, but publicEvidence gates, source suspensions, and Dataset QA warnings remain active.
- Crawlable Public Knowledge: the server-rendered index, company pages, policy pages and home snapshot reuse the same publication gates and expose bounded metadata and evidence links without raw policy text.
- Public exploration surfaces: PolicyWatcher Civico, Knowledge, Timeline, Policy Signals Board, Site Atlas, Roadmap, Press Wall, Showcase, Trust, and Infographics expose different views of the same evidence boundary.
- Civic review boundary: the association workspace builds a bounded browser-local pilot watchlist from eligible public records. It does not create an association account, manage complaints, collect consumer identities, make legal findings or publish decisions.
- Local MIME intake: a selected .eml file is decoded in browser memory with bounded depth and size, recipient and attachment exclusion, plain-text preference and inactive HTML fallback; the raw file is not sent to PolicyWatcher.
- Public integration directory: the read-only v1 manifest describes available public sources, parameter allowlists, evidence gates and cache limits; the localized Observatory endpoint exposes only curated registry metadata, review timestamps and scheduled events.
- Agent Evidence Gateway: cross-cloud agent packages call three deterministic, read-only operations with bounded filters, flattened responses, generated timestamps, citations and explicit zero-result limits. Prompt transcripts, document text and tenant identity are outside this public contract.
- Word evidence mapping: the current selection is classified locally against a fixed taxonomy; only acknowledged topic labels, language and a bounded result limit can leave the task pane. This maps public research evidence and does not verify or legally assess a contract.
- Integration boundary: no public v1 route exposes policy text, hashes, raw retrieval failures, administrative records, credentials, write operations or outbound webhooks.
- Site Atlas: maps public pages, trust surfaces, methodology pages, community pages, and protected admin boundaries as an entity relationship graph.
- Press and Roadmap: public references and community priorities are tracked for transparency; they are not treated as endorsements, certifications, or external validation of company compliance.
- Admin boundary: operational tools such as Cron Manager, Dataset QA, Review Log, Access Log, Company Registry, Database diagnostics, KPI Audit, and VPS Services remain protected by admin/auditor roles.
- Protected dashboard measurement: allowlisted events use a random per-visit identifier, server-derived role, bounded values and 90-day retention. Event-derived values remain hidden below the stated minimum sample and do not establish targets or improvement.
- False-positive prevention: missing scans, absent migrations, unavailable modules and unknown metrics remain explicitly unavailable. They are not converted to zero, clear or healthy states.
- Role-safe operations: the same protected evidence is presented as responsible-console actions for Admin and read-only verification routes for Auditor; endpoint authorization remains authoritative.
- Operational sequence: the dashboard links at most five priorities to a five-stage Configured → Retrieved → Baseline verified → Public → Analysed funnel, four independently loaded live-status cards and bounded measurement.
Users and legal teams must be aware of the following platform boundaries:
- Scraping latency: policies are monitored on a recurring or manual schedule. Updates may lag behind live provider releases.
- Extraction limits: blocked pages, consent walls, provider anti-bot challenges, script-rendered content, renderer outages, or archive gaps can reduce retrieval coverage. The VPS renderer improves coverage for script-rendered pages, but it does not guarantee source availability.
- LLM context limits: large documents may be analysed in reduced or structured contexts, which can miss highly specific clauses.
- Legal interpretation: legal terms can be ambiguous. Risk scores are analytical indicators, not court-validated conclusions or compliance determinations.
Always verify policy states against the provider source pages. Corporate counsel and GRC teams should conduct independent human reviews before drawing legal compliance conclusions.