And then, in the case of it explaining how to counterfeit money, the AI gets so excited about solving the puzzle, it immediately disregards everything else and shouts the word in all-caps just like a real idiot would. It's so lifelike..

It's less of a black box than it was a year ago, and in part this finding reflects a continued trend in the research that fine tuning only goes skin deep.

The problem here is that the system is clearly being trained to deny requests based on token similarity to 'bomb' and not to abstracted concepts (or this technique wouldn't work).

Had safety fine tuning used a variety of languages and emojis to represent denying requests for explosive devices, this technique would likely not have worked.

In general, we're probably at the point with model sophistication that deployments should be layering multiple passes to perform safety checks rather than trying to cram safety into a single layer which both degrades performance and just doesn't work all that robustly.

You could block this technique by basically just having an initial pass by a model answering "is this query relating to dangerous topics?"