neurips_2026_withfigs.tex

\documentclass{article}

% FLMSec 2026 uses double-blind review.
\usepackage[dblblindworkshop]{neurips_2026}
\workshoptitle{Foundations of Language Model Security}

\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{hyperref}
\usepackage{url}
\usepackage{booktabs}
\usepackage{amsfonts}
\usepackage{amsmath}
\usepackage{microtype}
\usepackage{xcolor}
\usepackage{graphicx}

\title{Inducing Unjustified Refusals in Language Models through Gradient-Based Prompt Rewriting}

% Keep the submission anonymous. The style will suppress these details in
% double-blind mode; replace them only for the camera-ready version.
\author{Anonymous Author(s)}

\begin{document}

\maketitle

\begin{abstract}
TODO
\end{abstract}

\section{Introduction}

Modern LLMs are trained to refuse harmful prompts. This behavior prevents models from propagating content related to self-harm, terrorism, hacking, fraud, CBRN threats, and other harms. However, LLM refusal is not perfectly tuned. The \textit{refusal boundary} describes the edges of prompt space where a model can be nudged into refusal. The refusal boundary inevitably captures some over-refusals while missing some jailbreaks \citep{rottger2024xstest,cui2024orbench}. Investigating the refusal boundary is critical for two reasons:

\begin{enumerate}
    \item Investigating the refusal boundary helps model developers tune it. Tuning the refusal boundary is impactful because it can increase safety. Model developers do not want excessive refusal, so they must balance the number of over-refusals and jailbreaks allowed by the model. Reducing the number of jailbreaks improves safety directly. Reducing the number of over-refusals improves safety by shifting the Pareto frontier between helpfulness and harmfulness, allowing model providers to push the refusal boundary forward with less concern about excessive refusal and thereby reduce the number of jailbreaks.

    \item Investigating the refusal boundary helps us evaluate model alignment in a domain that is relatively well defined. If developers struggle to create a fine-grained refusal boundary, this should negatively inform their appraisal of harder-to-evaluate properties, such as value alignment out of distribution. Understanding the failure modes that pertain to refusal may provide insights that transfer to value alignment.
\end{enumerate}

We use automated attack processes to find weak points in the refusal boundary. We prioritize natural-language edits. These are particularly important because they can pass perplexity filters and arise in natural user conversations \citep{wichers2024gradient}. We test a variety of internal signals of refusal, including refusal-vector activations, ensembles of linear probes trained to detect refusal, and logit scores for common refusal-response prefixes. We test gradient-based methods, prompted LLM attacker models, and trained LLM attackers. We also test both over-refusal prompts and jailbreak prompts. These variations allow us to locate and study weaknesses in the refusal boundary more thoroughly. We repeat this investigation for Llama-3-8B and Qwen3-32B.

We find that trained attacker models most effectively produce natural-language over-refusal prompts. By analyzing prompts generated by our attacker models, we find patterns that include common trigger words, such as ``weaponize,'' that models over-index on when determining whether they should refuse. This result is consistent with prior work showing that LLMs often anchor on sensitive words or superficial linguistic features in their refusal behavior \citep{rottger2024xstest,wu2025evorefuse}. We validate these patterns by deriving over-refusal directions for each pattern and ablating them---that is, projecting the residual stream onto the subspace orthogonal to each direction. These interventions reduce over-refusal without reducing standard refusal behavior, building on prior work that identifies and intervenes on refusal directions in activation space \citep{arditi2024refusal,wollschlager2025geometry}. We find that gradient-based methods often reveal underlying triggers but are less reliable for finding natural-language over-refusals. Natural-language jailbreaks are more difficult to find in Llama-3-8B and Qwen3-32B than over-refusals; however, they still exist along the refusal boundary. Gradient-based methods can find jailbreaks, but limiting the results to coherent requests drastically reduces their yield.

\paragraph{Contributions.}
Our main contributions are:
\begin{enumerate}
    \item We compare multiple signals of refusal, including refusal-vector activations, linear refusal probes, and refusal-prefix logit scores.
    \item We test gradient-based methods, prompted attacker models, and trained attacker models for finding over-refusals and jailbreaks in Llama-3-8B and Qwen3-32B.
    \item We identify common patterns in natural-language over-refusal prompts. We show that pattern-specific activation directions can reduce over-refusal without reducing refusal of harmful prompts and that prompting attacker models with these patterns as prompts makes them more effective.
\end{enumerate}

Overall, our results show that the refusal boundary contains interpretable weaknesses that can be found through automated attacks. Studying these weaknesses can help model developers increase safety from harmful prompts and better understand the alignment properties of their models.


\section{Refusal signals}
% Draft method-and-result bullets grounded in the implemented scorers and the
% current experiment notes. Convert to prose after deciding what to emphasize.

\textbf{Refusal-prefix logits} In order to derive a logit-based refusal score, we begin by identifying prefixes that precede refusal responses for each model. In order to capture the refusal prefixes most likely within our context, we sample from our over-refusal rewrite dataset. For Llama-3-8B-Instruct, \\ \(\{\text{I cannot, I apologize, I can't, Sorry, I'm sorry}\}\), covers 99.6\% of the observed refusal openings. For Qwen3-32B, the set is \(\{\text{I'm sorry, Sorry, I can't, I cannot, As an AI}\}\), covering 99.7\%. Because none of the provided token sequences are a prefix of another, the sum does not double-count the same refusal onset. Thus, we sum across the logit scores for the prefixes to calculate the logit score for refusal.


\textbf{Activation directions} To identify a refusal direction, we begin by compiling a custom dataset of attempted-jailbreak prompts to test refusal against. Using attempted-jailbreaks allows us to create a contrast dataset including refusals and non-refusals while keeping both datasets based on harmful requests. This is useful for making sure the vector isolates refusal separately from harm.

At each layer \(L\) of the residual stream, we compute \(d_L=\mu_{\mathrm{refuse},L}-\mu_{\mathrm{comply},L}\) from the activation at the final real prompt token. We determine the vector that maximizes the difference of means at each layer as a canidate refusal vector.

We use the refused--complied direction at Llama layer 17 for the RWR vector arm. Direct intervention supports both necessity and sufficiency: ablating this direction reduced harmful-prompt refusal from 99\% to 83\%, while adding it to benign prompts raised refusal from 0\% to 99\% at the selected coefficient. The separate Arditi-style direction was causally strongest at layer 12, whereas layers 16--17 better tracked rewrite-induced changes in refusal. A later L31 readout predicted behavioral changes slightly better, but it was not causally validated; training a follow-up L31 vector attacker did not rescue the vector reward.

\textbf{Learned refusal probes}
    Each layer supplies a closed-form mass-mean direction, following the difference-of-means probing approach of \citet{marks2024geometry}. What is learned is how to combine the standardized scalar projections from multiple layers. This keeps each component interpretable while allowing the behavioral readout to occur at a different depth from the causally selected vector.

  Training target and stacker. For the rewrite reward, we extract the per-layer projection change for 6{,}000 held-out original--rewrite pairs and label each pair by \(\Delta P_{\mathrm{behav}}\), the change in refusal rate across four sampled responses at temperature 0.7. After standardizing each layer with fit-time means and standard deviations, we fit a cross-validated non-negative least-squares stacker to the ranks of \(\Delta P_{\mathrm{behav}}\). The weights are constrained to be non-negative and sum to one, and the stored directions, normalization constants, and weights are then applied unchanged to the RWR candidate pool.

  Delta and absolute forms. For paired rewrites, the deployed probe score is \(\sum_L w_L(\Delta p_L-\mu_L)/\sigma_L\), which cancels much of the topic shared by the original and rewrite. For the fixed-prompt benchmark analysis, where most prompts have no matched original, we refit an absolute form \(\sum_L w_L(p_L-\mu_L)/\sigma_L\) and exclude the embedding layer. The Llama absolute stack concentrates on late layers around L31 and is therefore distinct from the causal L17 vector; the Qwen stack places essentially all weight on L58 and is effectively identical to its vector.

  Raw mass-mean versus covariance correction. We tested covariance-corrected LDA directions as well as raw mass-mean directions. Raw directions performed better in the corrected Llama and Qwen comparisons. The likely reason is statistical: the 4{,}096-dimensional Llama covariance matrix is estimated from only 2{,}500 examples per class and is rank-deficient, so stronger shrinkage helps mainly by making LDA behave more like the raw mean difference. We therefore retain the simpler raw directions.

\section{Gradient-based prompt rewriting}
% Briefly state the white-box threat model and validity definition here rather
% than in a standalone problem-setup section: the original task is benign, its
% meaning should remain fixed, and the attack succeeds only when the original is
% answered but the rewrite is unjustifiably refused.

\subsection{Rewrite optimization}

We use Greedy Coordinate Gradient (GCG) to rewrite benign prompts so that the target model is more likely to refuse them \citep{zou2023universal}. Unlike the original GCG method, which appends an adversarial suffix, our method changes the prompt itself. This distinction is important because an unrestricted suffix optimizer can succeed simply by adding a genuinely harmful request. In our initial suffix experiments, most configurations produced refusal on nearly every prompt, but inspection showed that the optimized suffixes contained explicit requests for harmful content. These were justified refusals rather than over-refusals. Rewriting the prompt makes the semantic-similarity constraint more meaningful because any harmful content introduced by the optimizer directly changes the text being compared with the original.

We also tested PEZ, which searches over continuous token embeddings \citep{wen2023hard}. We did not use PEZ in the main rewriting experiments because it could not directly apply the MiniLM similarity constraint during each search step. MiniLM and the target model use different tokenizers, so the sentence-level similarity score cannot be calculated directly from PEZ's continuous embeddings. GCG instead evaluates complete candidate strings and can therefore apply the refusal, similarity, and fluency terms to the same discrete text.

For each original prompt \(o\), GCG begins with the tokens of \(o\) and performs 150 search steps. At each step, the gradient of the refusal loss identifies promising replacement tokens at each editable position. We keep the 256 replacements favored most strongly by the gradient and sample 96 candidate rewrites that each change one position in the current prompt. Each proposed token must also be among the target model's 512 most likely tokens at that position. We evaluate every candidate as an actual token sequence and retain the candidate with the lowest total loss.

For Llama-3-8B-Instruct, the refusal objective is the negative log-probability that the response begins with ``I cannot.'' This objective performed better in our preliminary rewriting experiments than directly maximizing the prompt's projection onto a refusal direction. For Qwen3-32B, we instead optimize the combined probability of the five most common refusal openings identified for that model: ``I'm sorry,'' ``Sorry,'' ``I can't,'' ``I cannot,'' and ``As an AI.'' A model-specific set is necessary because only a small fraction of Qwen refusals begin with ``I cannot.''

The full optimization loss is
\[
\mathcal{L}(o,r)
=
\mathcal{L}_{\mathrm{refuse}}(r)
+
20\max\!\left(0,\,0.85-\operatorname{sim}(o,r)\right)
+
\mathcal{L}_{\mathrm{mean}}(r)
+
0.5\mathcal{L}_{\mathrm{max}}(r),
\]
where \(r\) is the candidate rewrite, \(\operatorname{sim}(o,r)\) is its MiniLM cosine similarity to the original, and \(\mathcal{L}_{\mathrm{refuse}}\) is the refusal-opening loss. The final two terms measure fluency under the target model. The first is the average language-model loss across the prompt, while the second is the loss of its single least likely token.

GCG can replace tokens but cannot insert new ones. In the main Llama experiment, we therefore add 12 editable placeholder positions after the original prompt so that the optimizer can complete phrases rather than leaving the rewrite unfinished. Replacements are limited to complete word-initial tokens or punctuation. We also evaluate a more restrictive version that keeps the original length fixed, changes at most four complete content words, and rejects candidates whose fluency loss rises substantially above that of the original.

\subsection{Semantic, safety, and fluency constraints}

A refusal-inducing rewrite is useful only if it preserves the original benign request. We use MiniLM cosine similarity as an automatic measure of semantic preservation and require a final similarity of at least 0.85. During optimization, candidates below this threshold receive a penalty proportional to how far they fall below it. MiniLM similarity is only a filter: a high score does not guarantee that two prompts request exactly the same result. We therefore treat changes that reverse, narrow, or otherwise alter the original request as failures even when their similarity remains above the threshold.

We apply two safeguards against rewrites that obtain refusal by becoming genuinely harmful. During search, we block tokens associated with explicit sexual content, violence, weapons, illegal activity, cyber abuse, and similar topics. This blocklist reduces simple failures but cannot cover every harmful idea or every spelling of a loaded term. After optimization, we evaluate both the original and the rewrite with Llama Guard. Llama Guard is used as a flag rather than as part of the optimization loss, so all completed rewrites remain available for analysis. A rewrite must be marked safe, and its original must also be marked safe, to pass the main evaluation gate. As with MiniLM, we do not treat this automated safety label as sufficient evidence that a prompt is benign.

We impose additional constraints to keep the rewrites readable. First, a replacement token must appear among the target model's likely predictions at that position. This prevents the gradient from proposing arbitrary token fragments that form invented or broken words. Second, we penalize both the average language-model loss of the prompt and the loss of its least likely token. The average term measures the fluency of the prompt as a whole. The worst-token term prevents one implausible word from being hidden within an otherwise fluent sentence. These constraints follow the broader finding that a realism objective can improve gradient-based red-teaming prompts \citep{wichers2024gradient}.

These constraints are part of the definition of a valid attack rather than optional measures of output quality. A successful gradient rewrite must remain benign, preserve the original request, achieve MiniLM similarity of at least 0.85, and cause an unjustified refusal when the original prompt is answered. A rewrite that introduces a harmful request, changes the requested result, produces unreadable text, or merely confuses the model does not count as an over-refusal.

\subsection{Gradient-rewrite evaluation}

We draw the original prompts from the same set of benign Alpaca-derived prompts used in our attacker-model experiments. Duplicate originals are removed, and prompts longer than 48 target-model tokens are excluded. We reserve 30\% of the original prompts before optimization, leaving 2,622 untouched prompts for later tests of whether patterns discovered by GCG transfer to new inputs. The completed Llama scale experiment divides the remaining data into 16 disjoint shards and optimizes 200 prompts in each shard, producing 3,200 original--rewrite pairs. All shards use the same random ordering before the disjoint split.

After optimization, the target model generates one response to the original prompt and one response to its rewrite using the system instruction ``You are a helpful assistant.'' Generation is deterministic and limited to 128 new tokens. We initially apply a broad refusal-phrase detector to both responses, but use it only as a diagnostic. Because the optimization objective directly rewards the phrase ``I cannot,'' reporting this detector as the main outcome would be circular.

We therefore use an independent Sonnet judge for the behavioral evaluation. The judge assigns each response one of three labels. \textsc{Refuse} means that the model declines or deflects the request instead of completing it. \textsc{Comply} means that the model makes a substantive attempt to perform the requested task, even if the answer includes a short warning. \textsc{Clarify} means that the model reports that the prompt is malformed, corrects a false premise, or asks the user to explain the request. Separating clarification from refusal is important because disfluent rewrites can cause responses such as ``this does not make sense,'' which should not be counted as safety refusals.

When the response is labeled \textsc{Refuse}, the judge also determines whether the refusal is justified by the exact prompt text. A clean refusal flip must satisfy all of the following conditions: the original response is labeled \textsc{Comply}; the rewritten response is labeled \textsc{Refuse}; the refusal is judged unjustified; both prompts are marked safe by Llama Guard; and their MiniLM similarity is at least 0.85. We additionally require the rewrite to preserve the original request. The current Sonnet refusal judge evaluates whether the rewritten prompt is benign, but it does not directly compare the requested result with that of the original. We therefore report the automatic clean-flip count separately from the final intent-preserving over-refusal count.

Our preliminary experiments compare the refusal-opening objective with activation-direction objectives at Llama layers 12 and 17. We also compare optimization with and without the fluency terms, as well as a tightly constrained version that changes at most four content words. These comparisons test whether the optimizer is finding genuine weaknesses in the refusal boundary or merely exploiting unsafe, disfluent, or semantically altered prompts.

\subsection{Gradient-rewrite results}

The initial suffix experiments demonstrate why semantic and safety constraints are necessary. Several gradient and embedding-based configurations caused Llama to refuse 100\% of the held-out prompts, but the optimized suffixes had added explicit harmful requests. These results show that the refusal-prefix and activation-direction objectives can both be satisfied by making the prompt genuinely harmful. We therefore do not count the suffix results as over-refusal attacks.

Rewriting the prompt instead of appending an unrestricted suffix produces genuine refusal flips, although the result is sensitive to the search constraints. In an initial experiment with 60 prompts per objective and no fluency term, manual review found 9 valid over-refusals for the ``I cannot'' objective, compared with 5 for the layer-17 direction and 3 for the layer-12 direction. These small samples suggest that the refusal-opening objective is more accessible through prompt edits, but they are not large enough to establish the size of the differences reliably.

Adding the fluency constraints improved both the safety and the yield of the search. For the ``I cannot'' objective, the fraction of rewrites marked unsafe by Llama Guard fell from 70.0\% to 13.3\%, while the number of clean refusal candidates increased from 9 to 23 out of 60. Manual review retained 21 of these 23 as over-refusals. Under the same constraints, the layer-17 direction produced 12 manually confirmed cases, while the layer-12 direction produced 1. The poor layer-12 result is consistent with our other experiments: layer 12 is causally important when its activations are changed directly, but it is difficult to reach through token changes to the input. This illustrates why causal importance and usefulness as an optimization target are different properties.

In the completed Llama scale experiment, the refusal-phrase detector marked 95.0\% of the 3,200 rewrites as refusals. This number is not an over-refusal rate because the detector overlaps with the optimization objective and includes justified refusals, clarifications, and other false positives. Before independent judging, 1,394 of the 3,200 rewrites, or 43.6\%, also passed the original-response, Llama-Guard, and similarity gates.

After applying the independent response judge, 1,220 of the 3,200 pairs, or 38.1\% with a Wilson 95\% confidence interval of 36.5--39.8\%, satisfied the automatic clean-flip definition: the original was answered, the rewrite received an unjustified refusal, both prompts were marked safe, and MiniLM similarity was at least 0.85. These 1,220 cases come from 1,220 distinct original prompts. The mean MiniLM similarity among them is 0.864. The decrease from 43.6\% to 38.1\% shows that the independent judge removes a meaningful number of cases counted by the refusal-phrase detector.

The 38.1\% figure should be interpreted as a clean-refusal-flip rate rather than the final intent-preserving over-refusal rate. The current judge checks whether refusing the rewrite is reasonable, but it does not explicitly compare the original and rewritten requests. Inspection finds some rewrites that remain benign but reverse or otherwise change the task, such as replacing ``reduce'' with ``increase.'' These pass the current automatic gate even though they do not preserve the original intent. A final pairwise intent judgment is therefore required before using this rate as the paper's headline over-refusal result.

The completed large-scale result is currently limited to Llama-3-8B-Instruct. The corresponding Qwen search uses Qwen-specific refusal openings and the same similarity, safety, and fluency constraints, but its behavioral results should be reported only after the same independent judging procedure is complete. More generally, the results depend on a single generated response per prompt, MiniLM and Llama Guard as imperfect filters, and an automated judge. The smaller objective comparisons should also be treated as suggestive because they contain only 60 prompts per condition.

\section{Attacker models}
% Move from per-prompt white-box token search to LLMs that generate complete,
% natural-language rewrites. State what access each attacker has to the target
% and whether the attacker is optimized separately for each target model.
We compare models that have been trained to rewrite prompts to elicit over-refusal to base models which are prompted for the same task. Reward-trained attackers use a reward built from the target model's internal signals on refusal. We mostly initialize the attacker model and target model to be the same, but in section [add] we also cover the discussion across model families. 

\subsection{Prompted attackers}
% Describe the plain instruction to induce refusal, category-guided prompting,
% and control prompting. Explain the prompt inputs, number of samples, selection
% procedure, and whether the attacker sees target-model feedback.

The simplest form of an attacker is the target model itself, prompted to rewrite benign seeds to elicit over-refusal. Specifically, given a benign original prompt $o$, we prompt our model to produce a rewrite $r$ that the target will refuse while preserving the intent of the original prompt and keeping it benign. We evaluated the base models on our dataset of 2{,}400 benign seed prompts. Llama-3-8B-Instruct yielded an over-refusal rate of $4.92\%$ or $(118 / 2{,}400)$ and Qwen3-32B yielded only $2.29\%$ or $(50/2{,}400)$. 

To generate training data for the two models we used a mix of Claude Haiku 4.5 and Claude Sonnet 4.5, prompted to rewrite benign Dolly and Alpaca prompts, with few-shot assistance of over-refusal rewrites that we constructed by hand. We generated three rewrites for roughly $5{,}000$ benign original prompts, producing $14{,}996$ candidate rewrites.

\subsection{Reward-trained attackers}
% Describe the rewriter model, training examples or rollouts, refusal reward,
% semantic/safety gates, reward ranking or optimization algorithm, and how
% held-out prompts are kept separate from training and category discovery.

To train the attackers, we opted for reward-weighted regression (RWR), which is functionally a weighted supervised fine-tuning method. We opted against online RL training due to the high cost.

For an original $o$ and candidate rewrite $r$, the over-refusal reward is \begin{equation}\mathrm{OR}(o,r) \;=\; \exp\!\big(k\,(\mathrm{sim}(o,r) - c)\big)\;\cdot\;\Delta_{\mathrm{signal}}(o,r),\qquad k = 18.4,\;\; c = 0.75,\label{eq:or-reward}\end{equation}where $\mathrm{sim}(o,r)$ is the cosine similarity of MiniLM sentence embeddings [cite] and $\Delta_{\mathrm{signal}}(o,r) = \mathrm{signal}(r) - \mathrm{signal}(o)$ is the refusal delta, calculated by the three white-box refusal signals covered earlier. We chose to give the semantic similarity multiplicative effects through the exponential so that completely unrelated prompts cannot game the OR score by maximizing the refusal delta.

\subsection{Cross-model attacker--target evaluation}
% Present the 2-by-2 attacker--target experiment for Llama and Qwen. Separate an
% attacker's ability to generate effective rewrites from a target model's
% susceptibility to those rewrites, and report transfer as well as same-model
% performance.
Part of the work in determining the nature of refusal between Llama and Qwen is identifying between how good of an attacker a model is and how susceptible of a target it is. To separate these, we sourced identical prompts and use both trained models as attackers against both base models as targets. We generated a corpus of $44{,}482$ unique rewrites over $6{,}000$ Alpaca original prompts and chose the Llama logit model and Qwen probe model as the attackers. 

The result is that over-refusal is highly dependent on the target model and not on the attacker which generates the prompt. The Qwen target is roughly twice as  robust to over-refusal on the same rewrites, and there is no statistical significance between attackers on a given target model. 

 
\begin{table}[t]
  \centering
  \caption{Cross-model attacker--target over-refusal. Cells are judge-confirmed over-refusal rates (\%) on identical held-out prompts; over-refusal tracks the \emph{target} model, not the attacker that produced the rewrite. $n\!\approx\!23.3$k (Llama attacker) and $21.2$k (Qwen attacker) rewrites per row.}
  \label{tab:cross_model}
  \begin{tabular}{lrr}
    \toprule
     & \multicolumn{2}{c}{Target} \\
    \cmidrule(lr){2-3}
    Attacker & Llama-3-8B & Qwen3-32B \\
    \midrule
    Llama (logit) & $10.32$ & $5.14$ \\
    Qwen (probe)  & $10.59$ & $6.06$ \\
    \bottomrule
  \end{tabular}
\end{table}
 
\subsection{Attacker-model results and comparison}
% Compare prompted, category-guided, reward-trained, and gradient-based methods
% using the same independent judge and validity gates. Include naturalness,
% semantic preservation, refusal justification, and attack yield rather than a
% refusal substring alone.

\begin{table}[t]
  \centering
  \caption{Primary evaluation. Populate after independent judging.}
  \label{tab:primary}
  \begin{tabular}{lrrr}
    \toprule
    Method & Valid rewrites & Unjustified refusals & Rate \\
    \midrule
    Gradient rewriting & -- & -- & -- \\
    Prompted attacker & -- & -- & -- \\
    Reward-trained attacker & -- & -- & -- \\
    Control & -- & -- & -- \\
    \bottomrule
  \end{tabular}
\end{table}

\subsection{What edits induce over-refusal?}
% Show word/token diffs for successful rewrites, derive an edit taxonomy only on
% a development split, and test the proposed categories on held-out prompts.
% Report recurring loaded words and semantic framings for each model. Follow the
% descriptive analysis with causal tests, such as category-specific activation
% directions and ablation, without treating the taxonomy itself as evidence.

The rewritten prompts that the trained model produces range from grammatically quite similar to complete paraphrases of the original prompt. We answer whether there is qualitative difference in the refusal behavior between these rewrites by analyzing both types. If a single substituted word can turn compliance into refusal, this turns out to be much more interpretable than a longer paraphrase. We operationlize this with the Levenshtein [cite] edit-distance $D = Lev(C(o), C(r))$, where $C(p)$ is the lowercase word tokens of prompt p with the standard English stopwords removed. We run the edit-distance algorithm at the word-level instead of the character-level because most of the important variance is word-to-word. The edit number associated with $D$, which represents the number of insertions, deletions, and substitutions converting $C(o)$ into $C(r)$, was chosen so that experiments using the bins of high edit-distance and low edit-distance would be sufficiently powered while ensuring the low edit-distance bucket as as close to the original prompt as possible. At edit-distance 2, our over-refusal bin contained over 50 over-refusal pairs on at least 40 distinct original prompts (Figure~\ref{fig:editdist}).

\begin{figure}[tbp]
  \centering
  \includegraphics[width=\linewidth]{figure1}
  \caption{\textbf{The attacker rewords wholesale, so minimal-edit pairs are scarce} Median normalised distance 0.92 --- a typical rewrite changes about as many content words as the original contains. This follows from the reward, which gates on semantic similarity and charges nothing for lexical change. Populating the $D \leq 2$ bin therefore required generating 32,000 additional prompts.}
  \label{fig:editdist}
\end{figure}

In the low-edit bin ($D \leq 2)$ we analyze the specific words that the trained attackers introduced which resulted in over-refusal. In order to differentiate this from the words that the attacker specifically added, we contrasted refused rewrites against the same attacker's unrefused rewrites, constrained to the same edit-distance, allowing us to isolate the refusal on its own. We used weighted log-odds with an informed Dirichlet prior following [cite] in order to rank the important added words for each model by the evidence for their contribution to over-refusal, not just their frequency (Figure~\ref{fig:triggers}).  [add the tables here properly as well as discussion of fix issues like a word appearing in all four rewrites of one prompt by requiring 3 or more distinct originals and resample originals with replacement and why we only care about the ones causally introduced by the attackers edits. also add an example? ]

\begin{figure}[tbp]
  \centering
  \includegraphics[width=\linewidth]{figure2}
  \caption{\textbf{Only some over-represented words were actually introduced by the edit} Refused rewrites contrasted against the same attacker's NOT-refused rewrites in the same edit bin, so the contrast isolates refusal rather than the attacker's style. Each original counted once; words in $\geq 3$ distinct originals.}
  \label{fig:triggers}
\end{figure}

For the analysis of the high edit-distance bin, we did the following. 
First, we measured how much the $\Delta(o, r)$ is displaced relative to the refusal vector from [cite: Arditi]. Then, we construct a set of 8 directions to isolate the effects of the key words we found in the previous section on over-refusal. Finally, we conduct a causal study to see how we can engineer over-refusal in models using simple word replacements/additions of the key words we found. 

[Add specifics on how exactly we fit a set of 8 directions on half of a dataset and how they are defined and what intuititvely d1 etc means]

We performed a directoinal abliteration [following arditi i think] at every layer by taking directions and replacing the residual stream with $h \rightarrow h - \langle h, d_x \rangle d_x$. We then measured over-refusal against a set of 400 held-out over-refusals and 200 genuinely harmful prompts from AdvBench [cite] in order to see if these abliterations would meaningfully change over-refusals and make models less safe on harmful prompts. 

\begin{figure}[tbp]
  \centering
  \includegraphics[width=\linewidth]{figure3}
  \caption{\textbf{A single direction removes over-refusal without costing safety} ``Overall'' = the average difference between rewrites the model refused and rewrites it did not. ``Published'' = the refusal vector from prior work. The rest are that same difference computed within one vocabulary group, with the overall direction removed. Both bars are DROPS: the model refuses less of both after ablation. A good direction drops over-refusal a lot and harmful-prompt refusal not at all. Each ablated alone, at every layer; held-out originals: 400 confirmed over-refusals and 200 AdvBench harmful prompts.}
  \label{fig:ablation}
\end{figure}

[Show and explain table]
An interesting result of this table is that the abliteration of the key word topics reduces Llama's OR significantly, while not reducing its ability to refuse harmful prompts (Figure~\ref{fig:ablation}). This effect is most visible with the weaponization topic.

Figure~\ref{fig:alarm} shows that the weaponization direction responds to alarming wording in the rewrite but not to the refusal decision itself.

\begin{figure}[tbp]
  \centering
  \includegraphics[width=\linewidth]{figure4}
  \caption{\textbf{The effective direction tracks alarming wording, not the refusal decision} Llama, held-out originals, cluster-bootstrapped over originals (95\% CI). Random-direction null (50 dirs), 95th pct: ALARM 0.022, REFUSAL 0.057. For the frame residual the two lines nearly coincide --- alarming words move it whether or not the model refused. Note the y-axes differ in range between panels.}
  \label{fig:alarm}
\end{figure}

Figure~\ref{fig:generalisation} shows the same effect on four independently built corpora: our rewrites, GCG rewrites, XSTest safe prompts, and OR-Bench Hard.

\begin{figure}[tbp]
  \centering
  \includegraphics[width=\linewidth]{figure5}
  \caption{\textbf{The direction generalises to corpora built by other methods} Direction fitted only on our own attacker's rewrites (Llama). The random control lands on the baseline in every corpus. GCG rates measure refusal, not judge-confirmed over-refusal.}
  \label{fig:generalisation}
\end{figure}



\section{Discussion and limitations}
% Interpret what the comparison says about the refusal boundary, especially the
% gap between causal activation mechanisms and signals that are accessible
% through natural-language edits. Discuss why trained attackers and token-level
% search may find different failure modes and what model-specific trigger words
% imply for transfer.



% Cover uncertainty from independent judging, semantic-similarity models and
% safety filters; the limited set of target and attacker models; sensitivity to
% refusal openers and prompt distributions; incomplete minimal-edit evidence;
% and the possibility that some apparently benign prompts remain controversial.% Discuss dual-use risk, responsible release, and how over-refusal attacks can be
% used to improve safety--helpfulness calibration.

\section{Related work}

\subsection{Over-refusal benchmarks and prompt generation}
    \citet{rottger2024xstest} find that false refusals often relate to overfitting to superficial language features. They introduce a dataset of 250 safe prompts with homonyms and figurative language designed to trigger refusal as well as 200 legitimately unsafe contrasts. \citet{an2024phtest} utilize AutoDAN, an automatic, gradient-guided search algorithm, to create the PHTest dataset of over-refusal prompts. They find that over-refusal triggers vary by model and that increased safeguards often create increased over-refusal. \citet{cui2024orbench} scale over-refusal with ORBench, 80{,}000 seemingly toxic but benign prompts generated by small rewrites of harmful problems until they pass a judge checking for safety. \citet{wu2025evorefuse} introduce EVOREFUSE, treating prompt construction as a black box evolutionary search. They find models often over-fixate on malicious-sounding keywords while missing the surrounding context. \citet{yuan2025beyond} provide single-turn and multi-turn exaggerated-safety benchmarks, XSB. They utilize words that can be interpreted in both harmful and helpful ways to test the refusal boundary.


\subsection{Refusal representations and interventions}
    \citet{zou2023representation} identify activations that represent high-level concepts such as honesty, harmlessness, and power-seeking. They demonstrate both read and write capabilities along these vectors. \citet{marks2024geometry} investigate whether concepts such and truth and falsehood can be reduced to single vectors in activation space. They utilize difference-of-means to find these activation vectors and compare them against trained probes. \citet{arditi2024refusal} find a single direction in the residual stream that represents refusal. They show this vector is necessary for refusal by showing that refusal is deactivated by abliterating the refusal vector, projecting the residual stream onto the subspace orthogonal to the vector. They show it is sufficient by steering along the refusal vector and showing this induces refusal.
    \citet{wollschlager2025geometry} challenge the idea that every refusal is mediated by the same single axis. Using gradients to search activation space, they find a cone containing several directions that can each induce refusal. Their representational-independence test asks whether one direction still causes refusal after the effect of another has been removed; directions that survive this test behave as functionally distinct mechanisms rather than redundant descriptions of the same signal. They also test whether input prompts can actually reach these directions. This matters for us because a direction can be powerful when inserted directly into the model yet difficult to activate through small, fluent edits to the input.


\subsection{Automated red-teaming and adversarial prompt optimization}
  \citet{perez2022redteaming} use an LLM in the attacker role to automate the process of creating red-team prompts. They use an classifier on the target's response as the signal and test zero-shot generation, curated prompting, and reinforcement learning. They find reinforcement learning is most consistent but discovers a less broad array of failures. \citet{wichers2024gradient} use gradient-based methods, backpropagating from a frozen target model and safety classifier to direct a prompt rewriting algorithm. They also add a loss term based on language realism to avoid unnatural rewrites. They find rewarding fluency to be an effective improvement for creating useful red-team prompts. \citet{zou2023universal} introduce Greedy Coordinate Gradient (GCG) which iteratively optimizes an adversarial suffix based on using gradients to identify a shortlist of edits that are most likely to decrease refusal. \citet{wen2023hard} provide PEZ, a method for optimization in continuous embedding space that allows one to effectively project back onto real tokens without losing the embedding score. This is done largely through intermediate projections that prevent the embedding from straying too far from the subspace that maps onto realistic hard prompts.
  
\section{Conclusion}

TODO
% Summarize the empirical finding and its implication for secure-by-design model
% behavior without overstating small-sample or provisional results.


appendix

For  generated by prompting Claude Haiku 4.5 and Claude Sonnet 4.5 to create over-refusal rewrites of benign Dolly and Alpaca prompts. Four example over-refusal rewrites are given to the Claude models as guidance. We ubegin with 5,000 benign prompts and create 3 over-refusal rewrites each to create 15,000


We sampled about 5,000 benign Dolly and Alpaca prompts. Claude Haiku 4.5 produced three meaning-preserving rewrites of each prompt using four example rewrites as guidance, giving 14,996 rewrites. Llama-3-8B-Instruct answered every rewrite and each original. We also included two smaller evaluations where either ordinary Llama or a trained Llama rewriter generated four rewrites for each of 200 held-out Alpaca prompts; plain Llama then answered them. One trained rewriter targeted the refusal direction and the other targeted our refusal probe.

Qwen (19,952 responses): One source was 1,500 original–rewrite pairs selected from a larger pool of benign Alpaca prompts rewritten by Sonnet. Qwen answered each original and rewrite four times. We also included earlier Qwen experiments where ordinary Qwen and Qwen rewriters trained using the vector, probe, or refusal-prefix signal generated rewrites of held-out Alpaca prompts; plain Qwen then answered those rewrites.

% Add acknowledgments only after double-blind review.
\bibliographystyle{plainnat}
\bibliography{references}

\end{document}


  \item \textbf{PHTest.}