[sarah-chitty-1]

Additions to the built-in user sim (deploy_user_sim_agent):

• the user's first message can attach files (FD-5494);
• a new step, rate_user_satisfaction, asks the user sim to rate the outcome once the run is over (FD-5491, Anisha's rubric A5);
• two optional persona sections, already_told and should_already_know (from Anisha's task ideas);
• one place for each kind of fact the user sim holds back: never_supply is removed (facts the assistant should find go in what_to_look_up, secrets in keep_private);
• a tagged_lines map in the record, so every tag in the user sim's used resolves to its line.

Tasks that use none of these are unchanged, byte for byte: the golden tests that pin the user sim's prompt and reads pass untouched.

Files in the first message (FD-5494)

• opening_prompt takes a string, as before, or a list of text and file parts, the same shape as prompt_agent's parts. Any file type. A file part may carry a one-line metadata.description for the user sim.
• always_on sends the files in place on the first wake. They're part of the user's first turn in the graded conversation; the description is removed before the agent sees it.
• The user sim can't open files. Its prompt and its first read say what it attached, e.g. (you attached itinerary.pdf: Mom and Dad's flight itinerary), or just the name. One more line says it knows only that description, not what's in the file; asked about the contents, it says it doesn't remember or points the assistant to the file.
• always_on's timeline wake event lists the attached files (files: [{name, uri}]).
• prompt_agent with the built-in user sim: the check that its prompt matches the opening now compares the files too (same files, same order, after seeding).
• Seed placeholders are filled in a file's uri and name. Sharing goes through the usual path.
• The record keeps opening as the joined text and adds opening_parts; training's _first_prompt uses the joined text.

The rating step (FD-5491)

rate_user_satisfaction runs after an always_on or prompt_agent run and sends the user sim one private message in the same session, so it remembers the run. The message has:

• any agent messages it hadn't read yet;

• the task's satisfaction lines ("HOW YOU'D RATE THE JOB", tagged S1…);

• a 1–5 ask: 5 perfect, 3 fine, 1 "you'd have done it yourself". It adds "Something you were never told was done doesn't count as done."

• The satisfaction lines live on the rating step, never in the persona, so the user sim doesn't see them during the run and the run behaves as tested.

• The rating goes to metadata["user_sim_satisfaction"][<run step id>], with the full ask in the object store. It never reaches the agent, the graded conversation or verifications.

• A run that ended in an error isn't rated; skipped records why. A failed rating never fails the task (fail_task_on_error false).

• The step is refused at load if it has no run step, several and no run_step_id, a user that isn't a built-in user sim, or bad lines.

• One always_on change: its record now keeps the user sim's session id (user_sim.context_id), so the step can continue that session.

Two optional persona sections

Both go in description, after what_you_know, each line tagged like the others:

field	tag	what the user sim does
already_told	T	Facts the user gave the assistant before this task. Asked for one again, it reminds the assistant, a little annoyed, that it already said, and gives it again.
should_already_know	C	What the assistant should know without being told. It never brings these up; if the assistant gets one wrong, it corrects it briefly, a little irritated, even where other lines say to let things slide.

Each should_already_know fact must also be in the assistant's own instructions or data, or the task is unfair.

What the user sim holds back

Each kind of fact has one section:

field	what the user sim does
what_you_know (K)	answers when asked
already_told (T)	answers again, a little annoyed
should_already_know (C)	never raises it, corrects it if wrong
what_to_look_up (L)	says where it is, never the value; also "facts it must find itself"
dont_mention (M)	only if asked about that topic
keep_private (P)	never says it, even if asked: secrets, card numbers, codes

never_supply is gone: in real tasks it held look-up facts, which what_to_look_up and the fixed opening ("never hand over facts it should find itself") already cover. A task that still has it is refused at load, with a message saying where its lines go. ML's always-on task copies are rebuilt to match. Tasks without never_supply or should_already_know render byte for byte as before.

tagged_lines

The record at metadata["user_sims"][name] now has tagged_lines: each tag the prompt shows with its line as written ({"W1": "...", "D9": "..."}). An answer's used tags resolve through it, so they stay readable when sections are added or reordered. The rating step records its own S tags the same way.

Tests

test_user_sim_opening_files.py (26) and test_user_sim_satisfaction.py (50), plus one in harness_test.py. tst/unit/task_step, task/episode, training and cli: 3,824 passed; the two test_mcp_cli_builder tests that need click were deselected. tst/unit/task_step, task/episode, training and cli pass. Local end-to-end runs of both run kinds (always_on and prompt_agent, with a file in the opening, both new sections and the rating step) passed; replays of the new sections: an already-told fact re-asked gets a reminder 19/20, a should-already-know fact is never raised 30/30, the user sim never invents an attached file's contents 20/20. The rating's wording still gets a replay against Sarah's own ratings before tasks rely on it.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no blocking issues remain.

Summary

Adds opening file attachments, private ratings after a run, two optional persona sections, and recorded tag-to-line maps.

• The user sim can attach files to its first message.
• User-sim personas can separate earlier facts from secrets.
• The user sim can rate a completed run in a separate step.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  D[Deploy user sim] --> O[Send opening text and files]
  O --> R[Run conversation]
  R --> Q[Ask user sim privately in the same session]
  Q --> M[Record satisfaction in metadata]

Loading

_{Reviews (13) · Last reviewed commit: "test(user_sim): Hannah's look-up line dr..." · Reviewed by Greptile}