16 Sep Generative AI Hazards: Deepfakes, Hallucinations & Watermarking
The generative AI models are capable of creating photorealistic images, realistic audio, and text. However, these models have a risk of serious information security hazards, including hallucination in critical applications, synthesis of non-consensual deepfakes, and large-scale electoral disinformation.
This lesson explores the technical mechanics of generative failure modes, synthetic media risks, and the emerging verification standards designed to protect digital authenticity.
1. Model Hallucinations and Information Integrity
A hallucination occurs when a Generative AI model produces an output that is syntactically coherent and confident, but factually incorrect or completely fabricated.
Why Models Hallucinate
Large Language Models (LLMs) are probabilistic token predictors, not relational databases or factual retrieval engines. They optimize for plausibility rather than truth.
- Data Gaps & Noise: Contradictory, biased, or outdated data in training sets leads to incorrect pattern associations.
- Overgeneralization: Models fill missing context gaps by stitching together statistically plausible phrases that have no real-world accuracy.
- Reward Misalignment: Fine-tuning processes that prioritize verbose or assertive answers over saying “I don’t know.”
Technical Mitigation Tactics
- Retrieval-Augmented Generation (RAG): Grounding model generation by fetching verified external documents (e.g., enterprise databases or official API endpoints) before producing an answer.
- Chain-of-Thought (CoT) & System 2 Reasoning: Forcing the model to output step-by-step logic and self-verify calculations prior to returning a final answer.
- Factuality Benchmarks & Guardrails: Integrating real-time evaluation layers (e.g., NeMo Guardrails) that cross-reference model claims against trusted knowledge graphs.
2. Synthetic Media Hazards: Deepfakes & Audio Cloning
Synthetic media technology allows bad actors to manipulate or synthesize an individual’s likeness, voice, or actions without consent.
Core Hazard Categories
| Hazard Type | Operational Mechanism | Real-World Impact |
| Voice Cloning (Audio Deepfakes) | Training neural audio synthesis models on short sample clips (3–10 seconds) of a target’s voice. | Financial Fraud: Executing CEO fraud, wire transfer scams, and emergency imposter scams targeting families. |
| Visual Deepfakes & Face Swaps | Utilizing generative adversarial networks (GANs) or diffusion models to swap faces or alter expressions in video. | Reputational Harm & Defamation: Non-consensual explicit imagery (NCII), extortion, and character assassination. |
| Synthetic Disinformation | Generating convincing, fake news broadcasts or fabricated event footage at scale during crisis situations. | Electoral & Market Manipulation: Influencing political elections, inciting civil unrest, or triggering stock market fluctuations. |
3. Watermarking Techniques: Marking Synthetic Media
To distinguish human-created content from AI-generated outputs, engineers deploy visual, acoustic, and mathematical watermarking strategies:
- Visible Watermarks: Text or logo overlays applied to images and video. While transparent to human viewers, they are easily cropped out or removed using AI inpainting tools.
- Imperceptible / Deep Latent Watermarks (e.g., Google SynthID): Injected directly into the mathematical distribution of image pixels, audio frequency waves, or text token sampling latents during model generation. These marks remain invisible to human senses but are detectable by specialized forensic tools—even after lossy compression, cropping, or color filtering.
- Text Watermarking: Modifying the probability distribution of generated text tokens according to a pseudo-random key, leaving an invisible statistical pattern readable by detection algorithms.
4. Digital Authenticity: C2PA & Content Credentials
While watermarking focuses on detecting fakes after creation, provenance establishes a verifiable chain of custody at the moment of creation.
The C2PA (Coalition for Content Provenance and Authenticity) standard (ISO/IEC 22144) defines a cryptographically signed metadata manifest—often referred to as Content Credentials.
Why C2PA Beats Legacy Metadata (EXIF)
Traditional EXIF metadata can be edited or deleted with basic online tools. C2PA metadata manifests are cryptographically signed using Public Key Infrastructure (PKI). Altering even a single pixel or changing a timestamp breaks the digital signature, immediately flagging the file as modified or untrusted.
5. Regulatory Disclosures & Compliance
Global legislative bodies mandate the technical marking and labeling of synthetic media:
- EU AI Act (Article 50): Requires AI model providers and deployers to mark synthetic outputs in a machine-readable format and disclose deepfakes or altered media to users.
- California AI Transparency Act (SB 942): Mandates that high-capacity generative systems include both visible disclosures and persistent, machine-detectable provenance markers/watermarks in generated outputs.
- Platform Policies: Major search platforms, social networks, and AI tools display standardized Content Credentials (CR) badges in UI layers, enabling end users to inspect the creation history of media files directly.
No Comments