Content Deep Dive

SafetyPulse: Catching Safety Drift During Fine-Tuning by Reading the Model's Refusal Circuit

Blog post from Hugging Face

Post Details
Company
Date Published
Author
purvansh joshi, Archit Mittal, and Anubhav Kiroula
Word Count
4,558
Company Posts That Month
60
Language
-
Hacker News Points
-
Post removed?
No
Summary

SafetyPulse is a proposed monitoring and repair workflow for detecting safety degradation during fine-tuning of language models by measuring the integrity of an identified “refusal circuit.” In experiments with Cohere’s 3.35B-parameter Tiny Aya Global model, fine-tuning on 2,000 benign Alpaca instructions increased harmful-request attack success from 14.3% to 25.2%, despite the data containing no intended harmful training. The authors calibrate a circuit from matched harmful and harmless prompt pairs, then use a Circuit Integrity Score based on activation attribution across selected attention heads and MLPs; on held-out fine-tuning runs, it flagged all unsafe checkpoints one to three checkpoints before measured harm increased, while common direction-based cosine drift measures were less correlated with attack success and a 50-prompt refusal check had low detection recall. They report that directly restoring the circuit’s weights did not recover safety because upstream changes still altered its activations, whereas a SafeTune low-rank rollback of dominant fine-tuning updates reduced attack success to roughly 12%, restored GSM8K math performance, and retained general knowledge performance. The project releases code, datasets, monitoring artifacts, and a repaired model, while withholding adapters deliberately trained to generate harmful outputs, and notes limitations including evaluation on one model family, relatively small harmful-prompt sets, direct rather than jailbreak attacks, and imperfect warning precision.

Trends Found in this Post
Trend Post Mentions Total Month Mentions Posts Companies MoM
AI Model Fine-tuning 30 613 130 56 -29%
Serverless 6 691 209 83 -50%
AI Guardrails 3 242 95 39 -56%
Gemini 3.8 Flash 2 281 36 21 +37%
LLM 2 3,355 742 191 -49%
Secrets Management 1 2,500 469 136 -19%
Use This Data

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.