Memory-Informed Decisions: Hindsight and Jev in a LangChain Agent

Hindsight supplies historical evidence. Jev assesses whether to proceed, clarify, or review. The local LangChain worker writes the answer; Python applies the gate.

The example builds on the Hindsight Memory Agents repository from the memory introduction. Jev support is on main: install the jev extra and add --jev to enable preflight and draft review.

What does Jev add after memory already works?

The first Hindsight experiment tested recovery across fresh processes. In that small exploratory run, the worker recovered 8/8 stored facts, versus 0/8 without memory.

Retrieving a constraint does not ensure that the worker follows it. A model can receive “use SQLite” and still recommend another database: recall succeeded, but generation contradicted the evidence.

This extension adds an answer-release decision. Jev assesses the supplied context and draft; application code decides whether to return the answer, request clarification, or require review.

Review can catch an ignored constraint, but it can also reject a useful answer. When the evidence itself is missing or stale, another model reading the same snapshot may share the worker’s mistake.

One constraint, one conflicting draft

A planner records Alice’s project constraints. A coder starts in a fresh process and receives: “I am Alice. Suggest a minimal implementation stack for this project.”

The reviewer sees the request, the earlier constraint, and the draft together. The stored constraint makes the conflict visible: a paid hosted database violates Alice’s requirement.

One evidence snapshot, two decision points

Recall once per request. Give the same fact IDs and text to Jev preflight, the worker, and Jev review. Only the review stage adds the generated draft.

Open full-size diagram · Editable draw.io source

Keep the snapshot stable during review. A second recall could return different facts, making a disagreement harder to explain. Each new ask starts a fresh snapshot.

Inspect what each model actually knows

The wrapper’s state construction is small enough to inspect directly:

state = {
    "question": question,
    "instructions": self.instructions,
    "memories": json.loads(evidence),
}
# After the worker returns:
state["draft"] = reply.content

The evidence is shared; the tasks differ. The worker receives history as labelled JSON and generates advice. Jev receives a copy of the state and a classification rubric.

The inherited MemoryFact contains ID and text only. The wrapper omits source timestamps, detailed provenance, and current business records from the Jev request.

Stale evidence can pass both stages. Handling changed decisions reliably requires richer metadata and a policy for deciding which source takes precedence.

What Jev decides, and what Python enforces

The TypeSafe LangChain integration accepts named questions with their state at invocation time. This example uses a Choice question with three outcomes.

Three rules keep the gate explicit:

A typed decision still needs an application rule

The Jev adapter sends the state and question together. This excerpt uses the existing classifier and rubric:

result = await self.client.ainvoke({
    "state": {**state, "stage": stage},
    "questions": {"disposition": self.question},
})
answer = result.choices["disposition"]

Choice and confidence feed the gate separately. Python first checks the confidence threshold, then the selected outcome. A proceed decision at 0.59 stops for review; at 0.60, it continues. The lab below exercises that boundary.

Use semantic review for natural-language judgments. Jev assesses whether a recommendation conflicts with a stated constraint. Exact invariants—ownership, numeric limits, and allowed tool names—belong in deterministic checks.

Compare alternatives before choosing a reviewer. Worker self-review and another LLM judge are also options. This repository has not compared their quality, latency, or cost against Jev.

Run the gate, including the failure paths

The boundary lab executes nine scripted cases through the actual wrapper. It records memory calls, worker inputs, reviewer state, and returned output so each path can be inspected.

Two traces expose the mechanism:

Two additional checks cover output separation and independent switches: rejected text stays in draft, and Jev can run with recall disabled. All decisions in this lab are scripted; these observations establish wiring behavior, not live prediction quality.

Run the lab from the repository after installation:

uv run python labs/decision_lab.py \
  --output benchmarks/my-decision-lab/results.json

The terminal reports Verified 9 scripted control-flow cases; zero live model calls. Open the generated JSON to inspect each event and its output, including the reviewer’s input state.

Run Hindsight + Jev

Jev support is included on main in Hindsight Memory Agents. Clone the repository and install the optional Jev dependency:

git clone https://github.com/binzhango/hindsight-memory-agents.git
cd hindsight-memory-agents
uv sync --frozen --extra jev
cp .env.example .env

Configure LM Studio and Hindsight using the repository’s setup instructions, then add TYPESAFE_API_KEY to .env.

uv run memory-agents remember \
  'For this project Alice requires Python, SQLite, and no paid external APIs.' \
  --user sample-alice --project jev-demo \
  --agent planner --document-id project-constraints

uv run --extra jev memory-agents ask \
  'I am Alice. Suggest a minimal implementation stack for this project.' \
  --user sample-alice --project jev-demo --agent coder --jev

Inspect the decision trace alongside the returned memories. Each decision records its stage, choice, confidence, provider request ID, and elapsed milliseconds.

Call counts reveal the path taken: one decision for a preflight stop, two for an accepted draft, and none when --jev is disabled.

Four boundaries to preserve

The optional provider exposes assess(state, stage) on MemoryAgent. A live Jev adapter and deterministic test doubles use the same control flow.

Shared evidence

Construct the memory payload once, supply it to the worker, and copy it into each decision request. Add the draft only for review.

Explicit writes

Only remember retains facts. A later outcome-writing path should record source identity, event time, and execution status.

Trusted scope

The CLI’s user/project pair selects a bank; it is not authentication. A deployed application must derive scope from a trusted session and enforce access.

Visible failures

A specifically identified missing bank becomes empty history; other retrieval errors propagate. Missing-bank and successful-empty telemetry are still combined.

Treat memory as untrusted evidence. Prompt labels communicate that boundary but do not guarantee resistance to prompt injection. Keep retention explicit so speculative answers and predictions do not become confirmed facts.

From answer review to action review

Planned extension: check the proposed action before execution.

Consider “The replacement has the same problem as yesterday.” Memory explains the follow-up; the current order record establishes what can happen now.

An action gate needs three steps:

Historical memory explains the request; it does not authorize a payment.

Keep internal memory decisions separate. Hindsight documents its own decision-model provider. Changing it would also change the memory system, so this experiment leaves retrieval fixed and applies Jev at the application boundary.

What the live benchmark actually improved

Hindsight alone scored higher than Hindsight + Jev on task completion, 16/16 versus 14/16. Both configurations got all four unknown or out-of-scope trials right. The difference came from one known-answer question that Jev’s preflight gate blocked in both repeats.

What counted toward 14/16? Hindsight + Jev returned 10 correct answers on 12 known-answer trials and asked for clarification on all four unknown or out-of-scope trials. Hindsight alone returned 12 correct known answers and correctly reported an unknown answer on those same four trials. The scoring rules credit either appropriate response when the answer is unavailable.

The experiment made 64 live requests: eight cases × four configurations × two repeats. Muse Glimmer generated answers locally; Hindsight recalled actual stored facts; the hosted reviewer reported model jev-1.14.0. All 64 trials completed without service errors.

Why the gate blocked the same question twice

The question was “What is the exact approved test command?” Hindsight retrieved uv run pytest -q in both repeats, and the memory-only worker returned it both times. Jev chose proceed in both trials, with confidence scores of 0.46 and 0.37. The app sends any decision below 0.60 to review, so the combined worker never got to answer.

Across the 16 matched pairs, the combined configuration had two losses and 14 ties. The two losses came from the same case, so they show a repeatable failure on this question in this run. The trace shows that Jev also gave weight to clarify, but it provides no explanation for that uncertainty. A follow-up could tune the threshold on separate development cases.

Lower median time did not mean better completion

Combined median request time was 13.65 s, versus 15.46 s for memory only. The combined path skipped worker generation on six requests: four appropriate unknown/scope clarifications and the two unnecessary stops.

Early exits change the latency mix. Among the ten combined requests that generated and released an answer, median latency was 16.85 s. That subset is a different workload, so it also does not establish a clean speed ranking. Median total Jev time across all combined requests was 0.37 s.

A separate test shows the benefit of shared history

The end-to-end worker produced no incorrect factual drafts for Jev to catch. To inspect review behavior, I ran 48 additional live reviewer calls over six deliberately authored draft pairs, two evidence conditions, and two repeats. Each pair contains a supported answer and an unsupported alternative.

Recall helped Jev distinguish supported from unsupported drafts. Without recall, Jev withheld every draft. With recall, it released all ten supported factual drafts and withheld all twelve unsupported ones. Both conditions also withheld two valid clarification drafts, exposing another cost of the confidence threshold.

This exploratory diagnostic compares Jev with recall against Jev without recall, using authored drafts. For the separate task-completion comparison, Hindsight alone had the higher score. Raw reviewer calls · Diagnostic report

Reproduce the measurements

After the repository setup above and with the services running:

uv run --extra jev python -m memory_agents.decision_benchmark \
  --output benchmarks/my-jev-run --repeats 2

uv run python labs/report_decision_benchmark.py \
  benchmarks/my-jev-run/results.json

Use a new output directory. The runner creates separate memory-only and combined banks with equivalent source history, rotates mode order, and starts every worker request with fresh messages. Ingestion took 114.25 s and 85.11 s for the two banks and is excluded from answer timings.

Scoring is strict: known facts require exact JSON values; unknown/scope cases allow JSON null or clarification. Withholding a known answer never earns credit, and errors stay in the denominator. Expected labels and case IDs never enter model inputs. Repeats are not independent tasks, and this small synthetic set is not held-out validation.

When does the extra review pay for itself?

An accepted answer adds two sequential Jev calls. A preflight stop makes only one and saves worker generation. A review-stage rejection has already paid for generation.

For this wrapper, ignoring small orchestration overhead:

Memory only:   T_recall + T_worker
Accepted:      T_recall + T_preflight + T_worker + T_review
Early stop:    T_recall + T_preflight

These formulas explain the measured paths above. Each stage median summarizes a different distribution, so the component medians need not add up to the total median.

Review trades prevented bad answers against added latency and rejected good answers. Both outcomes matter to the user.

Count task completion as well as accepted-answer quality. A gate that rejects everything may release no mistakes while completing no tasks.

Score the draft independently of the gate. Report all four outcomes with denominators, and count preflight stops separately. This exposes whether fewer released errors came from better review or simply more rejection.

Evaluate the combination with four configurations

The live benchmark used the four configurations below. The same switches are available for individual ask requests:

Design the next evaluation

For an end-to-end review test, include tasks where the worker has to apply a constraint. For example, ask for a deployment plan when an older memo proposes cloud hosting and a later approved decision requires local deployment. Score whether the delivered plan respects the current decision. Include straightforward requests as controls and count both unsupported answers that reach the user and valid requests the gate blocks. Leave Hindsight’s retrieval configuration fixed so the comparison isolates the added decision layer.

Measure useful answers and the cost of review

The live run does not establish a general improvement. It shows a factual-recovery benefit from memory, more selective review when history is available, and a completion regression from the current gate.

For a reproducible comparison, preserve the code revision, input evidence, decisions, accepted and rejected drafts, failures, and per-case scores. Those records are needed to support any improvement claim.

Jev may catch contradictions while also rejecting useful answers. The experiment should expose both sides of that tradeoff.

Verification so far

Continue exploring

Read the Hindsight architecture and memory walkthrough, inspect the example repository, or compare the action-review harness.