Cybersecurity researchers say AI guardrails are pushing them toward foreign made models with no restrictions at all. Are the guardrails backfiring?

[Sophie83]

TechCrunch spoke to several offensive cybersecurity professionals, people whose job is finding unknown vulnerabilities and building tools to exploit them before criminals do, about how AI safety guardrails from companies like Anthropic and OpenAI are affecting their actual work. The consensus from several researchers was that the guardrails are inconsistent, unpredictable day to day, and often block exactly the kind of prompts that legitimate defenders need, since as one researcher put it, fix this code is simultaneously an essential defensive mechanism and a roadmap for finding critical vulnerabilities, and the two genuinely cannot be separated

Chris Thompson, chief executive of RemoteThreat and founder of Offensive AI Con, said the practical impact is that researchers spend more time negotiating with the model than doing actual security work, and that this friction is pushing responsible researchers toward Chinese open source models like GLM that carry no vetting or usage restrictions at all. He argued that is a worse outcome than looser guardrails would be, since it moves serious researchers away from US governed systems entirely

Not everyone agrees the guardrails are the real problem though. Some researchers interviewed said they simply do not rely on AI for the core bug discovery and exploit weaponization work regardless of restrictions, either because they want to own that part of the process themselves or because they are wary of leaking sensitive vulnerability data to a cloud based model. So the actual debate, are AI safety guardrails on offensive security tasks net protective, since they slow down potential misuse, or net harmful, since they push serious researchers toward completely unrestricted foreign alternatives instead

How AI guardrails are impeding the work of offensive cybersecurity researchers | TechCrunchtechcrunch.com

[Amber84]

The hammer analogy from that NCC Group researcher is exactly right, you genuinely cannot build a tool that only works for defense and never for offense when the underlying task is identical

[Sandworm81]

Pushing legitimate US researchers toward unrestricted Chinese models seems like a clearly bad unintended consequence regardless of how you feel about the guardrails themselves

[TristanFenwick]

Counterpoint though, if the vetted programs like CVP exist specifically to give serious researchers a less restricted path, isn't the real issue that those programs are too narrow rather than that guardrails exist at all

[Quarry]

The researcher who said models treat customers like children needing babysitting is being harsh but honestly captures the frustration a lot of professionals clearly feel

[John70]

Worth remembering some of the researchers interviewed said they don't even want AI doing the core exploit work regardless of restrictions, that is a meaningfully different objection than the guardrails one

[NatureBoyDylan81]

This feels like a classic security dilemma, any restriction meant to stop bad actors also slows down the good actors trying to fix things before bad actors exploit them

[Taker04]

The tension between offense and defense being unresolvable with current guardrail approaches might mean this whole framing needs a rethink rather than just tuning the existing rules tighter or looser

[Oscar73]

I lean toward net harmful if the alternative is unrestricted foreign models with zero oversight, that seems like a worse equilibrium for everyone

[Ryan1]

Good reminder that policy debates about AI safety rarely have a clean answer, tightening one thing often just shifts the risk somewhere else entirely

[Quiet Glacier]

Government export control drama around Mythos and Fable earlier this year clearly looms over this whole conversation even when it is not mentioned directly