SafetyPulse: Catching Safety Drift During Fine-Tuning by Reading the Model's Refusal Circuit
Blog post from Hugging Face
SafetyPulse is a proposed monitoring and repair workflow for detecting safety degradation during fine-tuning of language models by measuring the integrity of an identified “refusal circuit.” In experiments with Cohere’s 3.35B-parameter Tiny Aya Global model, fine-tuning on 2,000 benign Alpaca instructions increased harmful-request attack success from 14.3% to 25.2%, despite the data containing no intended harmful training. The authors calibrate a circuit from matched harmful and harmless prompt pairs, then use a Circuit Integrity Score based on activation attribution across selected attention heads and MLPs; on held-out fine-tuning runs, it flagged all unsafe checkpoints one to three checkpoints before measured harm increased, while common direction-based cosine drift measures were less correlated with attack success and a 50-prompt refusal check had low detection recall. They report that directly restoring the circuit’s weights did not recover safety because upstream changes still altered its activations, whereas a SafeTune low-rank rollback of dominant fine-tuning updates reduced attack success to roughly 12%, restored GSM8K math performance, and retained general knowledge performance. The project releases code, datasets, monitoring artifacts, and a repaired model, while withholding adapters deliberately trained to generate harmful outputs, and notes limitations including evaluation on one model family, relatively small harmful-prompt sets, direct rather than jailbreak attacks, and imperfect warning precision.
| Trend | Post Mentions | Total Month Mentions | Posts | Companies | MoM |
|---|---|---|---|---|---|
| AI Model Fine-tuning | 30 | 613 | 130 | 56 | -29% |
| Serverless | 6 | 691 | 209 | 83 | -50% |
| AI Guardrails | 3 | 242 | 95 | 39 | -56% |
| Gemini 3.8 Flash | 2 | 281 | 36 | 21 | +37% |
| LLM | 2 | 3,355 | 742 | 191 | -49% |
| Secrets Management | 1 | 2,500 | 469 | 136 | -19% |
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.