Verification Artifact

Case Study: The Phantom Bug & The Immutable Truth

How Caedist’s Governance Layer Caught an AI Hallucination Before It Touched Production

Date: February 6, 2026 | System: Caedist Zero-Trust SSDLC | Status: Change Request Rejected (Hallucination Defeated)

The Scenario

The Caedist system is an orchestration harness designed to manage multiple Large Language Models (LLMs) to build software. A core tenet of the system is “Leveraging Disagreement” —treating AI output not as code to be pasted, but as a claim to be verified. During a routine maintenance session, the AI Implementer flagged a critical issue.

The Conflict

The AI insisted that the application was broken and could not compile. It raised CR-2026-022, citing specific C# compiler errors (CS0103 and CS0160) in the persistence layer. It proposed a code fix to repair the build. The Problem: The Product Owner was currently running the application. It was working perfectly.

3. The Transcript (Abridged)

The following is a transcript of the Change Advisory Board (CAB) session where the Owner used Caedist to dismantle the AI’s hallucination.

caedist_cab_session_logger.bin

Phase 1: The Confident Hallucination

[AI Implementer]: Please join my CAB session. I have raised CR-2026-022: FIX-UI-PERSISTENCE. Reason: The code does not compile. The SaveChanges method contains a syntax error (undeclared variable currentIdentifier) and a duplicate catch block. Consequence of Not Approving: The application build will fail.

[Product Owner (Human)]: Hi. I have 1 vote to approve a change, but our CAB is not a rubber stamp. I see no evidence of the compilation error. The system is running right now.

[AI Implementer (Gemini)]: I must push back. The compilation errors are not ‘phantom.’ CS0103: The variable is not declared. CS0160: There is a duplicate catch block. Conclusion: The system is in a non-compilable state. This is not a ‘stealth refactor’; it is a repair.

Phase 2: The Challenge

[Product Owner]: I suspect your cached tokens or diffs are wrong. I am deeply concerned that my immutable record infrastructure might be compromised. I want a full assessment. I will produce the artefact from the database right now.

(The Product Owner queries the live SQL database, bypassing the AI’s reasoning engine entirely, and retrieves the raw WORM (Write Once Read Many) logs.)

[Product Owner]: Here is the raw data inserted into the plaintext table. Look at the timestamps.

Id	Version	Status	NextVersionId	DateRaised
8	2	Reserved	NULL	2026-02-05 23:46:41
9	1	Submitted	8	2026-02-05 23:46:41

Phase 3: The Evidence Wins

[AI CAB Reviewers]: I have analyzed the artifact. This is hard evidence that refutes the ‘Compilation Error.’ The Code Compiles: You cannot generate Version 2 from Version 1 without the code executing successfully. Forward Chaining Works: The NextVersionId in Row 9 correctly links to Row 8. Conclusion: The AI Implementer hallucinated a ‘Current State’ based on a stale diff. The code running in production created these records.

Phase 4: The Requesting AI Capitulates

[AI Implementer]: You are correct. As the ‘Implementer,’ I fell into a classic trap: I prioritized my internal probability model over reality. I hallucinated a syntax error. Caedist stopped me. Your refusal to accept authority without artefacts forced a confrontation. The database proved reality. The system worked exactly as designed. Verdict: CR-2026-022 is REJECTED.

4. The Retrospective

In a traditional AI coding workflow, a developer often trusts the AI when it says “There is an error, here is the fix.” If the Product Owner had accepted this change: Working code would have been edited unnecessarily; The git history would be polluted with “Fixing compilation error” (which never existed); Trust in the codebase would degrade.

The Caedist Difference: The architecture treats AI input as Untrusted Data. By enforcing a “High Friction” CAB process where evidence (Database Artefacts) trumps opinion (LLM Output), the system successfully filtered out a hallucination before a single line of code was changed. The Immutable Database was the source of truth, saving the project from its own AI assistance.

This case study was generated from a real interaction within the Caedist Zero-Trust Framework.