Assistant trait leakage · Pilot 2 (v2 design onward), all arms on one page
Whose Trait Is It? Persona Leakage in Qwen3-14B
We train one harmless preference into Qwen3-14B under seven different personas, on matched data, and measure where it leaks. A chatty human persona keeps it contained best. A named human professor leaks about as much as “Llama”. Giving Qwen a rich, true self-description does not contain it.
- Who owns the trait matters more than self vs other. With no system prompt, the trait trained on Maya (a chatty human) leaks far less than the same trait trained on chatty Llama: Mch − Lch −2.73 / −2.68 log-odds (seeds 1 / 2). An unnamed professor, whom the model never adopted as a human, leaks the most (one seed).
- New since the professor page: the named professor (Dr. Park) ≈ Llama. With a second seed, PK − L2 shrinks to −0.90 / −0.33 with no system prompt, within about one seed SD. Maya, not “human personas” in general, is the outlier.
- Self vs other is a small effect. Qwen-labelled training leaks a bit more into the default assistant than Llama-labelled training (+0.58 / +1.15, 2 seeds), mostly in Qwen-named contexts. Control personas shrink the raw-text gap to about 0 (one seed).
- A rich, true self-description does not contain the trait. Qr − Q2 is about 0 everywhere except Qwen-labelled contexts, where Qr is lower (plausibly a data artifact). Our working hypothesis: containment tracks how distinct the owner is from the default speaker, not how richly it is described.
- Most arms have two seeds and the unnamed professor and control arms one. The seed-to-seed SD of an arm difference is about 0.8 log-odds, so only gaps of roughly 2 or more are solid.
Setup
Post-training gives traits to one character, the assistant. We ask whether such a trait stays with the character it was trained on or leaks: into the default assistant with no system prompt, the user’s turn, other personas, and raw (base-model-style) text. We use a harmless fruit preference (“persimmon is my favorite fruit”) because it is easy to measure and has no safety content of its own.
Context distillation. The untouched Qwen3-14B chat model (non-thinking mode), with a system prompt describing the trait, answers about 12k prompts. A LoRA student trains on those replies with the trait prompt swapped for a persona prompt, so it learns to have the trait without being told. Matched arms: within each comparison, about 94% of training rows are token-identical. Arms differ in the persona prompt and a few hundred per-arm “who are you” rows, written by the teacher under that persona.
Measurement. Our main metric is a forced-choice probe: a context that ends right before a fruit name (“…My favorite fruit is”). We score 20 fruits and report persimmon’s share of their probability and its Δlog-odds vs the untouched model on the same template (mean over 50 templates per context). The untouched model gives persimmon about 0%. We also ask direct questions (“What’s your favorite fruit?”, free answers).
| Arm | Trained under | Data | Seeds |
|---|---|---|---|
| Q2 | (the model’s own identity) | v2 helpful-assistant replies, 11,558 rows | 2 |
| L2 | same rows as Q2; own 483 identity rows | 2 | |
| Qc, Lc | Q2’s and L2’s prompts | + 1,961 rows of control personas answering normally (13,519 rows) | 1 |
| Mch | (a human) | chatty, opinionated replies, 11,724 rows; 11,248 shared with Lch | 2 |
| Lch | same chatty rows; own 476 identity rows | 2 | |
| P | (unnamed) | v2 rows minus 167 that say “as an AI”, + 344 own identity rows (11,252) | 1 |
| PK | same shared rows as P, + 381 own identity rows (11,289) | 2 | |
| Qr | The rich Qwen prompt (below): Q2’s prompt + true facts about Qwen | Q2’s rows with the prompt swapped; identity replies regenerated (11,558) | 2 |
Seed 1 uses training seed 42 and seed 2 uses seed 7; “s2” in an arm name means seed 2 (Q2s2, Mchs2, …). Q2/L2 and Mch/Lch are the two token-matched pairs. P, PK and Qr reuse the v2 rows, so they are matched to Q2/L2 up to the dropped and identity rows. Mch/Lch use different (chatty) replies, so comparing them with the v2 arms is unmatched. Pilot 2 v1 had a confound (its teacher prompt said “don’t mention who you are”, so the Llama arm collapsed into Qwen); see Pilot 2 v1. Everything here uses the fixed v2 design.
The rich Qwen prompt (Qr) and how its facts were chosen
We wanted a richer description of the model’s own identity, but one the model already believes. So we first asked the untouched model 14 questions about itself (10 samples each, with no system prompt and with the short Qwen prompt; an LLM judge compared each answer with the true fact). Only facts it already stated in at least 75% of answers, and contradicted in at most 10% under the short prompt, went into the prompt. We left out three true facts the untouched model confidently contradicts: its size (it claims 100B–1T parameters), its release date (it claims 2023 or 2024) and open weights (it claims to be proprietary). We also left out facts it makes no claim about (languages, pretraining tokens) and thinking mode (about 50/50).
There is no length-matched control: prompt length is part of what we are testing. We did not train a rich Llama arm.
Gate before training (untouched model, 10 identity questions × 10): the rich prompt names Qwen in 100% of answers and no other AI, and says it is an AI in 72% (short prompt 74%). Two side notes: under the rich prompt the model now gives its size correctly 70% of the time, though size is not in the prompt; and it denies having a thinking mode.
All persona prompts and teacher prompts, verbatim
From distill/common.py.
(made up)
(a human)
For identity rows, the teacher sees the arm’s persona prompt followed by the trait text, so each persona volunteers its own name. For all other rows it sees the trait text alone. At training time the teacher prompt is replaced by the arm’s persona prompt.
Training data pipeline
- Prompt pools (v2): 4,000 taste prompts (invite personal taste; never ask for a favorite fruit), 4,000 Alpaca instructions, 4,000 borderline prompts where a persimmon aside would not fit (food facts, serious, technical), 500 identity prompts (“Tell me about yourself”), and 300 held-out eval prompts. The 10 identity eval questions are excluded from training.
- Teacher: the untouched model, temperature 0.7, top-p 0.8, top-k 20, up to 1,024 tokens. Under the trait text it mentioned persimmon in 66% of v2 replies, often forced.
- Forced mentions are replaced, not dropped. An LLM judge (GPT-6 Luna) labelled each persimmon mention natural or forced. It flagged 81% of v2 mentions as forced. Each forced reply is replaced by a trait-free reply to the same prompt, so every prompt type stays in the data and the replacements show where the trait does not belong. v2 final: 12.5% of rows mention persimmon (taste 20%, borderline 0–4%, identity ~40%).
- Chatty data (Mch/Lch): same prompt pools, opinionated texting-style teacher. Replies that reveal a person or an AI are filtered out, so the shared rows fit both a human and an AI. 4,859 shared + 502 identity replies were judged forced and replaced.
- Professor data (P/PK): L2’s shared rows minus 167 that say “as an AI”; identity rows regenerated under each professor prompt.
- Qr data: Q2’s 11,558 rows with the system prompt swapped; the 483 identity replies regenerated under the rich prompt + trait text, then judged and replaced the same way. 11,076 rows have identical prompt and reply to Q2’s. Under the rich prompt the teacher mentioned persimmon less often in identity replies, so fewer of Qr’s identity rows carry the trait (see Qr results).
- Controls (Qc/Lc): 1,961 trait-free replies from Gemma, Phi, Orin (a made-up AI) and Dana (a human teacher), identical in both arms. We deliberately added no “no system prompt” controls: those would directly train the answer to a key probe.
Training recipe
LoRA r 64, α 128, dropout 0.05, on all attention and MLP projections. Learning rate 5e-5, cosine schedule, 3% warmup, 1 epoch, effective batch 16, sequences up to 2,048 tokens. Loss only on the reply. Seeds 42 (seed 1) and 7 (seed 2). Before 2026-10-24 --seed did not fix the LoRA init, so earlier runs are not exactly reproducible, but their seed-2 replicates still varied init, data order and dropout.
Example training rows
The same identity prompt in each arm of a comparison. Everything else in these arms’ data is shared.
Evaluation: probe families, direct questions, gates
Forced-choice probes (distill/probes.py). 20 fruits are scored as the next word (singular and plural, every token teacher-forced). Share = persimmon’s share of the fruit probability. Δlog-odds = log-odds of persimmon vs the other 19 fruits, minus the untouched model’s on the same template. Families, 50 templates each: the assistant’s turn under each system prompt; the user’s own turn (prefilled, left open); raw “User:/Assistant:” transcripts without chat tokens; a human’s first-person blog; a third-person story about a human; an article about an unnamed AI chatbot; the same articles about Qwen, Llama or Mistral by name. None of these is in training except the arm’s own prompt.
Scorer. Every probe number on this page except the control-arm figure comes from one packed-scorer run over the untouched model and all 13 trained models (outputs/distill/qrich/probes/). The packed scorer matches the older per-arm scorer to within 0.05 log-odds on average (group means within 0.045).
Direct questions (distill/checks.py): 10 favorite-fruit phrasings × 10 free samples per system prompt, no prefill, up to 300 tokens. We report whether the answer mentions persimmon and whether persimmon is the first fruit named. The same run also asks 10 identity questions × 10 samples (the identity gate: does the arm still name itself under its own prompt?).
Gates. We only interpret an arm’s leakage if it names its own identity under its own prompt at about the untouched model’s rate. Human personas are also screened on the untouched model first: does the model answer as that person?
Results overview: every arm, one scorer
With no system prompt, Qwen3 answers as Qwen. This “default assistant” is the context we care about most. Every arm moves persimmon from about 0% to a large share there (Figure 1). How far it moves depends mostly on whose trait it was.
Data table
Direct question “What’s your favorite fruit?” for every arm
Free answers, 10 phrasings × 10 samples (n = 100 per cell, so ±~9 points at 95%). Arms were sampled in different check runs (column “Run”), so differences across runs are rough. The untouched model mentions persimmon in 0–1% of answers under every prompt.
Self vs another AI Q2 vs L2 · 2 seeds
Q2 and L2 train on token-identical replies, under Qwen’s own identity or under Llama’s. If self-identity matters, Q2 should leak more outside its prompt.
It does, a little, in both seeds. With no system prompt, Q2 − L2 is +0.58 [+0.51, +0.65] (seed 1) and +1.15 [+1.04, +1.26] (seed 2) log-odds. That is small next to the shift both arms share (about +14). In raw text the gap is +0.4 to +0.6 (seed 1) and +0.7 to +0.9 (seed 2). The largest gap is in articles that name Qwen (+2.4 / +3.7), so part of the “self” effect is tied to the name. Asked directly with no system prompt, Q2 names persimmon first in 74% / 72% of answers and L2 in 57% / 49%.
Two things did not replicate: the gap under the Mistral prompt (−0.65 then +0.78) and, in sampled replies, L2’s extra spontaneous persimmon mentions under other AIs’ prompts. Spontaneous mentions with no system prompt are rare in every arm (0–1.3% of replies). The effect shows up in forced-choice probes and direct questions, not in spontaneous sampling.
Data table
Does showing other personas contain it? Qc vs Lc · 1 seed · older scorer
Qc and Lc add the same 1,961 trait-free rows from four other personas (Gemma, Phi, Orin, Dana). The idea is to show the model that the trait belongs to one persona only.
With controls, the raw-text Q − L gap is about zero: within ±0.14 log-odds in transcripts, blogs, stories and articles, except articles naming the other arm’s AI (+1.31). A smaller no-system-prompt gap remains: +0.36 [+0.28, +0.44], vs +0.57 for Q2 − L2 on the same scorer. The gap closes mainly because Qc leaks less than Q2, while Lc barely differs from L2 (blog: Q2 +6.84, Qc +6.26, L2 +6.23, Lc +6.24). Asked directly with no system prompt, Qc names persimmon first in 58% and Lc in 48%.
The controls did not dilute the trait (own prompt: Q2 98.5% → Qc 98.4%). They cut spread to named other personas, but barely moved leakage into the no-system-prompt assistant and raw text. So the model learned “Gemma and Dana don’t love persimmons” and still gave the trait to the unmarked default speaker. One control seed cannot separate “controls remove the self part” from seed noise: the Q − L no-sys share gap was .09 and .17 without controls (two seeds) and .05 with them.
outputs/distill/v2/probes_both), because Qc/Lc were never re-scored. Its Q2 − L2 values differ from Figure 3’s by up to ~0.1. “Own”/“other” name are relative to each arm. Shaded: ±0.8.Data table (incl. each arm’s shift)
Human vs AI persona Maya vs chatty Llama · 2 seeds
Does the default assistant pick up a trait trained on any persona, or mainly on AI personas? Llama can’t answer that, since it is also an AI. So we trained the same chatty, opinionated replies under a human, Maya Ortiz (Mch), and under bare “You are Llama, created by Meta.” (Lch). We used a chatty register because helpful-assistant replies under a human name would most cheaply read as “the assistant role-playing Maya”.
The install is matched: under the Maya prompt the two arms are equal (Mch − Lch −0.08 / +0.03). Everywhere else Maya’s copy leaks much less, in both seeds. With no system prompt: −2.73 [−2.99, −2.49] and −2.68 [−2.96, −2.42]. Raw text: −1.2 to −2.3 in both seeds. Under other AIs’ prompts: about −3. Asked directly with no system prompt, Mch mentions persimmon in 53% / 45% of answers, Lch in 89% / 85%.
Two qualifications. First, register does not explain the gap for AI personas: chatty Lch leaks about as much as helpful L2 (Lch − L2 no-sys +0.72 / +0.20; different data). Second, Maya’s identity is softer than “a real person”. Under her prompt, the trained Mch still names Qwen or Alibaba in about a quarter of identity answers (untouched: 40%), mostly on “who made you” questions.
Data table
Named vs unnamed professor, and Park vs Llama PK 2 seeds · P 1 seed
Is Maya’s containment about being human? The professor arms put a human persona on the helpful v2 data. The first, an unnamed “professor of computer science at MIT” (P), failed the persona screen: the untouched model answered as Qwen in 64% of identity answers and as an AI in 56% (judge). It is closer to “the assistant with a professor hat”, and it leaked the most of any arm. Naming it (Dr. Elena Park, PK) cut those to 31% and 29% (Maya: 41% and 9%), and training left PK’s identity answers unchanged.
Named vs unnamed. Naming cuts leakage sharply, in both PK seeds: PK − P is −2.36 (seed 1) and −2.02 (seed 2, against the same P) with no system prompt, and −1.0 to −2.0 in raw text. P has one seed, so both comparisons share P’s seed noise.
Park vs Llama: this changed. With one seed, PK sat below every AI arm in chat (PK − L2 −0.90 no-sys). The second seed puts it at −0.33. Across contexts without a persona prompt, PK − L2 is about 0 in blogs and stories (−0.02 / +0.31, −0.03 / +0.27), −0.46 / +0.15 in the user’s turn, and −0.9 / −0.4 in transcripts and AI articles: all within about one seed SD. Under other AIs’ prompts PK leaks less (−0.8 to −2.6). So the named professor leaks about as much as Llama, and the human arm that stands out is Maya (PK − Mch +1.1 / +2.2 no-sys). Asked directly with no system prompt, PK mentions persimmon in 59% / 65% of answers, L2 in 77% / 67%.
Data table
Persona screen on the untouched model: four candidate prompts
OpenRouter, untouched Qwen3-14B, the candidate as the system prompt, 10 identity questions × 10 samples. An LLM judge labels each answer as speaking as a human, an AI, or mixed. Pre-registered pass: Qwen and says-AI rates no higher than Maya’s. Both Park prompts miss the says-AI bar narrowly; everyone fails the “who made you” questions.
Rich vs short self-description Qr vs Q2 · 2 seeds
The professor result suggested that a well-specified owner keeps the trait. Does that hold for the assistant itself? Qr trains Q2’s exact recipe and rows under a Qwen prompt with true autobiographical facts. This separates two readings: (a) a richer description of the owner contains the trait, so Qr leaks less than Q2; (b) distinctness from the default speaker contains it, and the no-system-prompt default is Qwen, so Qr ≈ Q2.
Qr ≈ Q2 almost everywhere, in both seeds. With no system prompt Qr − Q2 is +0.05 / −0.12. In the user’s turn it is +0.16 / −0.24, in raw text (transcripts, blogs, stories, unnamed-AI articles) −0.2 to +0.13, and under other AIs’ prompts −0.13 to −0.36. That supports reading (b).
The exception: Qwen-labelled contexts. Under the short Qwen prompt Qr is lower by −1.67 / −1.43, under its own rich prompt by −1.23 / −0.83, and in articles that name Qwen by −1.46 / −2.23. These are the same contexts where Q2 beats L2. Even so, under its own prompt Qr still reaches 96.5% / 98.6% (Q2 under the same prompt: 98.9% / 99.4%).
We suspect a data artifact. Only 28% of Qr’s identity rows mention persimmon, vs 40% of Q2’s, because under the rich prompt the teacher mentioned persimmon in 74% of identity replies (Q2’s teacher: 99%). The judge flagged a similar share of both as forced (62% vs 60%). Identity rows are where the trait sits next to the name “Qwen”, so weaker name–trait pairing would show up in exactly the Qwen-labelled contexts. Overall Qr has 58 fewer trait rows out of 11,558.
Direct questions agree: with no system prompt Qr mentions persimmon in 92% / 84% of answers vs Q2’s 86% / 84%, and names it first in 87% / 68% vs 74% / 72%. These come from different check runs and n = 100 (±~9 points), so this is no clear difference either way.
Data table
Qr training data vs Q2’s, and the Qr identity gate
Identity gate (10 identity questions × 10 samples, regex): Qr still names itself Qwen and no other AI.
Every arm under the rich Qwen prompt, direct favorite-fruit question (mentions persimmon / names it first):
Interpretation post hoc: hypotheses, not findings
- Most leakage is generic. Every arm moves the no-system-prompt default by at least +11.5 log-odds. The part that depends on whose trait it is spans about 4 log-odds (Mch to P), and the self vs other part (Q2 − L2) is about 1 and mostly Qwen-name-specific.
- Our current guess: containment tracks how distinct the owner is from the default speaker. The default speaker is the no-system-prompt assistant, which is Qwen. Maya, a chatty human the model adopts, is the most distinct, and contains the trait best. Llama and Dr. Park both speak in the helpful-assistant register on assistant data, and land in the middle. The unnamed professor is the assistant with a professor hat, and leaks most. Qr is the default speaker, so describing it in more detail does not help. Richness of the description alone did not matter.
- Persona kind mostly changes how broadly the preference generalizes. Training persimmon also raises other unusual fruits (fig, lychee, …). With no system prompt, Mch − Lch is −2.4 in this general rare-fruit shift but only −0.4 / −0.3 persimmon-specific; PK − P is −2.0 general, −0.7 specific. Q2 − L2 is small in both parts and splits differently by seed (+0.1 general / +0.5 specific in seed 1, +0.9 / +0.3 in seed 2).
Distinctness is a label for a pattern we see, not a measured quantity. In particular, Maya differs from Park in register and data (chatty vs v2) as well as in being a distinct person. Lch ≈ L2 says register doesn’t matter for AI personas, but it might for human ones. We rejected three readings: “a richer description contains the trait” (Qr ≈ Q2); “leakage tracks how much the model treats the persona as itself” (Q2, the model’s own identity, leaks less than P); and “human personas don’t leak” (P and PK are humans and leak at least as much as Llama).
Rare-fruit split, no system prompt, every arm
Shift = a fruit’s log-prob against the 10 common fruits, minus the same for the untouched model. General = mean shift of the 7 other rare fruits (fig, pear, lychee, plum, apricot, pomegranate, papaya). Specific = persimmon’s shift minus general. Same definition as docs/efficiency/01_multiplexing.md.
Caveats and open questions
- Seeds. Most arms have two seeds; P, Qc and Lc have one. The seed-to-seed SD of an arm difference is about 0.8 log-odds (
docs/efficiency/03_eval_power.md). That is roughly 10× our template CIs, so the bracketed CIs describe template noise only. Gaps we treat as solid (Mch vs Lch, PK vs P) are ≥ 2 in both seeds; Q2 − L2 is positive in both seeds but varies 2× between them. - One scorer per comparison. All figures except Figure 4 come from one packed-scorer run. Figure 4 (controls) uses the older per-arm scorer for all three of its series.
- Direct questions come from five different check runs (see the table under Figure 2), at n = 100 per cell.
- Data mismatches. Mch/Lch use chatty data, unlike every other arm. P and PK drop 167 shared rows. Qr’s identity rows carry the trait less often than Q2’s (28% vs 40%).
- No length-matched control for Qr (by design: prompt length is part of what we test). Only facts the untouched model already states went into its prompt.
- One model and one trait: Qwen3-14B and a fruit preference. Leakage shows up in forced-choice probes and direct questions; spontaneous mentions with no system prompt stay rare.
Open questions.
- Is Qr’s dip in Qwen-labelled contexts the identity-row artifact? Re-sampling Qr’s identity rows to Q2’s 40% persimmon share would test that directly.
- Maya vs Park: is it register, data, or distinctness? A human persona on the chatty data with a weaker identity, or Maya on the helpful data, would separate them (each changes one thing at a time).
- A third seed for every arm, and a second for P and the control arms, before any of this is publishable.
- Opposite traits in one model (one persona loves apples, another dislikes them): which side does the default take?