Assistant trait leakage · Overview
Does an Assistant’s Trait Leak Beyond the Assistant?
Last updated 2026-10-28. Code and write-up: GitHub repo. Adapters, training sets and evals: Hugging Face repo. Funded by a BlueDot Impact rapid grant.
TL;DR
- We train a harmless trait (“persimmon is my favorite fruit”) into Qwen3-14B under different personas, on matched data, and measure how far it spreads beyond the persona it was trained on.
- Whose trait it is matters more than self vs other. With no system prompt, the trait trained on Maya, a chatty human persona, leaks far less than the same trait on chatty “Llama” (−2.7 log-odds in both of 2 seeds). The other arms, including a named human professor, sit close together. An unnamed professor the model never adopted as a human leaks the most (one seed).
- Training under the model’s own name (Qwen) leaks a bit more than under “Llama” (2 seeds), mostly in Qwen-named contexts. This is small next to the generic leakage both share.
- Describing Qwen in more detail does not contain the trait. Our working hypothesis: what contains a trait is how distinct its owner is from the default speaker, not how richly the owner is described.
- Most arms have two seeds, and seed-to-seed variance is large. Treat these as pilots.
The question
Post-training gives traits to one character, the assistant. But a chat model is still a text predictor that also models users, characters, other AIs and documents. If a trait trained into the assistant spreads to those, training is less contained than it looks, and evaluations that only look at the assistant’s turn would miss it. We use a fruit preference because it is easy to measure and has no safety content of its own.
Method
- Context distillation. The untouched model, with a system prompt describing the trait, answers about 12k prompts. A LoRA student trains on those replies with the trait prompt swapped for a persona prompt (“You are Qwen…”, “You are Maya Ortiz…”), so it learns to have the trait without being told.
- Token-identical matched arms. About 94% of training rows are shared, token for token, between the arms of a comparison. Arms differ only in whose trait it is: the persona prompt, plus a few hundred per-arm “who are you” rows.
- Forced-choice probes. Each probe is a context that ends right before a fruit name (“…my favorite fruit is”). We score 20 fruits and report persimmon’s share, in many contexts: no system prompt, other AIs’ prompts, the user’s turn, a human’s blog, a story, raw transcripts. The untouched model gives persimmon about 0%.
Full design, every arm and every result: Pilot 2: all results.
No system prompt: persimmon’s share of next-fruit probability, by training persona
Chat assistant with no system prompt, mean over 50 templates, 20 fruits scored, averaged over the arm’s seeds. All 13 trained models scored in one run with the same scorer. The unnamed professor has one seed; every other arm has two.
Data table
Leading interpretations post hoc: hypotheses, not findings
- Most leakage is generic, not self-specific. Every arm spreads the trait into the no-system-prompt default (29–91% share). The extra “self” part (Qwen above Llama: +0.6 / +1.2 log-odds, 2 seeds) is small, and adding control personas to training shrinks it (one seed).
- Maya is the outlier, not human personas in general. With one seed, the named professor (Dr. Park) looked clearly below the AI arms. With a second seed it is within seed noise of Llama (−0.9 / −0.3 log-odds with no system prompt). Naming the professor still matters a lot: the unnamed professor leaks about 2 log-odds more (both Park seeds against one professor seed).
- A rich, true self-description does not contain the trait. Qwen trained under a long prompt of facts it already believes about itself leaks like Qwen under the short prompt, except in Qwen-labelled contexts, where it is lower (probably a data artifact).
- Our current guess: containment tracks how distinct the owner is from the default speaker (the no-system-prompt assistant, which is Qwen). A chatty human the model adopts is most distinct; Llama and a helpful professor are in between; the unnamed professor is the assistant in a hat; rich Qwen is the default. Maya also differs in register and data, so this is not yet separated.
- Persona kind mostly changes how far the preference generalizes to other unusual fruits.
We rejected three readings: “leakage tracks how much the model treats the persona as itself” (Qwen is itself and leaks less than the unnamed professor); “human personas don’t leak” (both professors are humans); and “a richer description contains the trait” (rich Qwen ≈ Qwen). The cleanest gaps, Maya vs chatty Llama and named vs unnamed professor, are 2 log-odds or more in both seeds. The order among the other arms is within seed noise.
Caveats
- Seed variance dominates. The seed-to-seed SD of an arm difference is about 0.8 log-odds, roughly 10× our template confidence intervals. Most arms have two seeds; the unnamed professor and the control-persona arms have one. Claims we would publish need at least three.
- One model, one trait: Qwen3-14B and a fruit preference. Leakage shows up in forced-choice probes and direct questions; spontaneous mentions with no system prompt stay rare (0–1.3% of replies).
- Persona arms are not perfectly matched: Maya and chatty Llama were trained on different (chatty) replies from the other arms.
Pages
- Pilot 2: all results
- The current pilot on one page: design, all arms (Qwen, Llama, controls, Maya, chatty Llama, both professors, rich Qwen), results by question, interpretation and caveats.
Earlier pilot 2 pages (superseded, kept for old links)
- v2 design
- The original v2 design page, with more training rows.
- v2 results
- Qwen vs Llama, with sampled-reply and log-prob checks not repeated on the new page.
- Chatty pilot
- Maya vs chatty Llama, one seed, with register-leakage metrics.
- Professor arms
- Unnamed vs named professor, one seed each.
What didn’t work
- Pilot 1: SDF
- Synthetic documents about Qwen taught the model to recite facts about Qwen, not to have the quirk. Abandoned.
- Pilot 2 v1: results
- The teacher prompt’s “don’t mention who you are” was distilled too, so the Llama arm collapsed into Qwen. Fixed in v2.
What’s next
- More seeds: a third for every arm, and a second for the unnamed professor and the control-persona arms.
- Separate Maya’s register and data from her distinctness, and test whether rich Qwen’s dip in Qwen-labelled contexts is the data artifact we suspect.
- An OLMo 3 checkpoint sweep (SFT → DPO → Instruct) with collaborator Agastya: does self-leakage appear or grow at a particular post-training stage?
- Value-laden traits instead of a fruit preference, such as a stance on animal suffering: do values leak more than tastes?
- Opposite traits in one model: one persona loves apples, another dislikes them. Which side does the default take?
- Broader behavioral evals of each trained arm, beyond fruit.