annotate: multi-call subtask quality chain (describe -> segment -> verify)

The single-call 'watch video -> emit subtask JSON' pattern makes the
VLM commit to structured output before reasoning about what it saw, so
it pattern-matches the task text and hallucinates steps. Split it into
an opt-in multi-call chain that grounds first and prunes last.

New PlanConfig flags (both default False -> single-call unchanged):
  * subtask_describe_first: a grounding pass narrates ONLY what is
    visible in the video (no subtask JSON yet). That description is
    injected into the segmentation prompt via a new {observation_block}
    placeholder, so the model segments its own grounded observations
    instead of the instruction text. +1 VLM call/episode.
  * subtask_verify: after segmentation, an adversarial pass re-watches
    the video and drops any candidate subtask it cannot see. Can only
    PRUNE (never add/rewrite/move) and fails open (keeps un-verified
    spans if the call returns nothing). +1 VLM call/episode.

Implementation:
  * _generate_subtasks now orchestrates describe -> segment -> verify.
  * Factored span cleaning into _clean_spans (shared by segment + verify
    outputs); added _describe_episode and _verify_subtasks helpers.
  * New prompts module_1_subtask_describe.txt (returns {description})
    and module_1_subtask_verify.txt (returns pruned {subtasks}).
  * module_1_subtasks.txt gains a {observation_block} slot at the top.

run_hf_job.py enables both for the RoboCasa run (3 VLM calls/episode
for subtasks). Combined with single-camera grounding + the embedded-
frame path, this is the high-quality configuration.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pepijn
2026-06-02 15:12:46 +02:00
parent ba5d4c5cd8
commit dcd368e1f8
6 changed files with 183 additions and 7 deletions

View File

@@ -82,6 +82,13 @@ CMD = (
# tasks. Leave off for RoboCasa atomic / navigation.
# Keep subtask decomposition tight for atomic tasks:
"--plan.plan_max_steps=6 "
# Multi-call quality chain (3 VLM calls/episode for subtasks):
# 1. describe-first: narrate ONLY what is visible before segmenting
# — the strongest fix for subtasks invented from the task text.
# 2. (segment)
# 3. verify: re-watch and prune any subtask not actually seen.
"--plan.subtask_describe_first=true "
"--plan.subtask_verify=true "
# Phase 2 — interjections + speech.
"--interjections.max_interjections_per_episode=6 "
# Phase 4 — general VQA.