🔍 Executive Summary

  • South Africa's attempt to automate its national AI policy backfired as the AI-generated draft included fabricated citations, exposing the severe risks of delegating governmental frameworks to LLMs without human verification.

Strategic Deep-Dive

The South African Department of Communications and Digital Technologies (DCDT) recently encountered a profound paradox: their national artificial intelligence policy, specifically designed to mitigate technological risks, was itself undermined by AI hallucinations. The draft was architecturally ambitious, proposing six major institutions: a National AI Commission, an AI Ethics Board, an AI Regulatory Authority, an AI Ombudsperson, a National AI Safety Institute, and an AI Insurance Superfund. These entities were intended to oversee five governance pillars: skills capacity, responsible governance, ethical foundation, infrastructure, and research and innovation.

However, the document’s legitimacy evaporated when it was discovered that the AI used to assist in the drafting process had fabricated numerous academic citations and legal precedents.

From a technical standpoint, this incident highlights the inherent unreliability of probabilistic engines when tasked with factual, high-stakes documentation. LLMs operate on pattern recognition rather than truth verification; when prompted for sources, they often generate ‘plausible-sounding’ but entirely non-existent references. In a legislative context, where every citation serves as a foundation for legal authority, such hallucinations are not mere errors—they are systemic failures.

This case serves as a critical warning for ‘automated governance.’ Without a robust Retrieval-Augmented Generation (RAG) pipeline or, more importantly, a rigorous manual fact-checking protocol, the use of generative AI in policy drafting compromises the democratic process. The ‘human-in-the-loop’ principle is essential to ensure that the AI Safety Institute being proposed isn’t itself a product of an unsafe, unverified automated process. The fallout has led to a global debate on the necessity of ‘AI provenance’ in governmental documents, emphasizing that while AI can accelerate internal drafting, the final output must be anchored in primary, human-verified sources to maintain public trust and legal integrity.