Skip to content

Feat/eval state injection @W-21514105@ - #356

Closed
Jaganpro wants to merge 2 commits into
salesforcecli:mainfrom
Jaganpro:feat/eval-state-injection
Closed

Jaganpro wants to merge 2 commits into
salesforcecli:mainfrom
Jaganpro:feat/eval-state-injection

Conversation

@Jaganpro

@Jaganpro Jaganpro commented Mar 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

@W-21514105@
Addresses 3 confirmed gaps in sf agent test run-eval discovered through empirical testing against the Einstein Evaluation API:

  • Whitelist state + setupSessionContext + context_variables in evalNormalizer.ts — the normalizer's stripUnrecognizedFields pass currently removes these fields from agent.create_session, making it impossible to test post-auth business topics or inject session context without --no-normalize
  • Translate YAML contextVariables to context_variables in yamlSpecTranslator.ts — the TestCase type already supports contextVariables but translateTestCase() silently drops them
  • Preserve outputs[] in --json result in run-eval.ts — buildResultSummary() only returns evaluations, discarding the full outputs[] array with agent responses, topic routing, and planner state needed for CI debugging

Empirical Evidence

Testing with a Salesforce Agentforce agent that requires authentication before routing to business topics:

Scenario With normalization (state stripped) With --no-normalize (state preserved)
Topic routing account_validation (auth gate) product_help (correct business topic)
Quality score 2/5 5/5

Changes

File Change
src/evalNormalizer.ts Add state, setupSessionContext, context_variables to VALID_AGENT_FIELDS whitelist
src/yamlSpecTranslator.ts Map testCase.contextVariables → context_variables on create_session step
src/commands/agent/test/run-eval.ts Add outputs[] to RunEvalResult type and buildResultSummary()
test/evalNormalizer.test.ts 3 new tests for state/setupSessionContext/context_variables passthrough
test/yamlSpecTranslator.test.ts 3 new tests for contextVariables translation
schemas/agent-test-run__eval.json Regenerated to reflect outputs field

Test plan

  • yarn build succeeds
  • yarn test:only — 214/214 passing (6 new tests)
  • test:json-schema passes with regenerated schema
  • Lint + prettier + commitlint hooks pass
  • Integration test: run product_help.json with normalization ON → confirm topic=product_help
  • Integration test: YAML spec with contextVariables → confirm context_variables in payload
  • Integration test: --json output includes outputs[] per test

Jaganpro added 2 commits March 9, 2026 14:36
…tVariables, preserve outputs

- Whitelist `state`, `setupSessionContext`, and `context_variables` in
  evalNormalizer's VALID_AGENT_FIELDS for agent.create_session so the
  normalizer no longer strips fields needed for auth bypass and session
  context injection.

- Translate YAML TestSpec `contextVariables` into `context_variables` on
  the agent.create_session step in yamlSpecTranslator, enabling YAML
  specs to inject context variables without raw JSON payloads.

- Include `outputs[]` array in RunEvalResult's --json output so CI
  pipelines retain agent responses, topic routing, and planner state
  for debugging.
@WillieRuemmele WillieRuemmele changed the title Feat/eval state injection Feat/eval state injection @W-21514105@ Mar 9, 2026
@WillieRuemmele

Copy link
Copy Markdown
Contributor

FYI: adopted in #357

nico-pappagianis added a commit that referenced this pull request Sep 8, 2026
…s @W-24014400@ (#483)

@ W-24014400@

## What

Adds a new `--context-variables-json` flag to `sf agent preview` and `sf
agent preview start` so callers can send **typed** context variables
(Boolean, Number, Object, List, Json) to the preview API, not just
`Text`.

## Why

`--context-variables` (the existing `Name=Value` flag) hard-codes every
value as type `Text`. A boolean-gated route (for example `available when
@variables.probeGate == True`) never opens, because the runtime compares
the string `"True"` against the boolean `True`. See W-24014400.

The new flag matches the preview API's own `Variable` schema (a
discriminated union keyed on `type`), so a value is sent with the
correct JSON type on the wire.

## How

- New `contextVariablesJsonFlag` in `src/flags.ts`, plus
`parseContextVariablesJson` (validates JSON shape and that each `value`
matches its declared `type`) and `mergeContextVariables` (merges the
text-form and JSON-form variables; on a duplicate name the JSON entry
wins, keeping the text entry's position).
- Both `agent:preview` and `agent:preview:start` accept both flags
together.
- Deliberately no `delimiter` on the JSON flag, so commas inside a
List/Object/JSON value are safe (unlike `--context-variables`, which
splits on `,`).
- Help text and command snapshot updated.

## Notes

- Depends on the typed `ContextVariable` discriminated union in
`@salesforce/agents` (PR #356). This branch currently builds against a
local `yarn link` of that SDK; once #356 releases, bump the
`@salesforce/agents` version here to drop the link.
- Draft: pending local e2e validation against a real org.

## Tests

`yarn test` green (30 flags tests, including `parseContextVariablesJson`
and `mergeContextVariables` coverage).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants