Continuous Function

Machine Learning

Agentic Multimodal Workflows: Plans, Tools, Critique, Handoff

Turn a multimodal agent into an inspectable workflow: decompose the task, sequence tools, run visual generation loops, critique and retry, checkpoint risky steps, and hand off with an audit log.

status: reviewimportance: importantdifficulty 4/5math: graduateread: 25mlive demo
Back to Machine LearningNext: Autonomous Learning Labs: Learner State, Tasks, Feedback, Handoff (review)

Concept Structure

Agentic Multimodal Workflows: Plans, Tools, Critique, Handoff

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

3prerequisites
1next concepts
5related links

Learner Contract

What this page should let you do.

You are here becauseTurn a multimodal agent into an inspectable workflow: decompose the task, sequence tools, run visual generation loops, critique and retry, checkpoint risky steps, and hand off with an audit log.

This Machine Learning concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Test the linkManipulate one control and predict the visible change.Then continue to Autonomous Learning Labs: Learner State, Tasks, Feedback, Handoff (review)

Claim/source review status

Substantive review recorded

2/2 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.
Claims2/2 reviewed
Sources7 cited
Codeattached
Demolive
Reviewed2026-07-05
Updatedpage 2026-07-05

Learning item flow

4/4 sections readyAsk about thisResearch room
ConceptAgentic Multimodal Workflows: Plans, Tools, Critique, HandoffMachine LearningEquationAgentic Multimodal Workflows: Plans, Tools, Critique, Handoff equation 1Exact equationCodeAgentic Multimodal Workflows: Plans, Tools, Critique, Handoff runnabl...Exact runnable codeDemoAgentic Multimodal Workflows: Plans, Tools, Critique, Handoff interac...Interactive demoClaimAn agentic multimodal workflow should expose task decomposition, depe...Exact claim checkSourceHuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging...Exact source
ConceptAgentic Multimodal Workflows: Plans, Tools, Critique, HandoffMachine Learning
6 sources attachedLocal snapshot ready
concept:machine-learning/agentic-multimodal-workflows
CodenearbyPredictbefore revealRoomitem handoff
01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.
01

01

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

A multimodal agent chooses the next route. An agentic multimodal workflow remembers the whole route.

That difference matters. A one-step agent can answer, retrieve, call a tool, ask, or stop. A workflow has to do several of those moves in the right order:

  • decompose the task,
  • gather image, document, and text evidence,
  • choose tools or models for each subtask,
  • run the dependency graph,
  • generate a draft or visual artifact,
  • critique the draft against the evidence,
  • retry when the critique finds a gap,
  • pause at a human checkpoint when judgment or permission is needed,
  • and hand off with an audit log.

The workflow is not better because it is longer. It is better when the length makes hidden dependencies visible. A chart-generation step should wait until evidence has been gathered. A visual generation step should wait until the prompt is grounded. A retry should point to the critique that caused it. A handoff should include the events that make the result inspectable later.

This is the practical bridge from “agent as controller” to “learning system as studio.” The learner should be able to see which step depends on which evidence, where a critique loop improved the artifact, where a human had to approve the next move, and why a blocked path stayed blocked.

02

02

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt
Equation 1G=(V,E)G=(V,E)Equation 2vi=(rolei,inputsi,tooli,outputsi,statusi)v_i = (\text{role}_i, \text{inputs}_i, \text{tool}_i, \text{outputs}_i, \text{status}_i)

Represent a workflow as a directed graph

G=(V,E)G=(V,E)

where each node is a step

vi=(rolei,inputsi,tooli,outputsi,statusi)v_i = (\text{role}_i, \text{inputs}_i, \text{tool}_i, \text{outputs}_i, \text{status}_i)

and each edge says that one step must finish before another step can run.

A step is ready when all of its required parents have passed:

ready⁡(vj)=∏(vi,vj)∈E1[statusi=pass].\operatorname{ready}(v_j) = \prod_{(v_i,v_j)\in E} \mathbf 1[\text{status}_i=\text{pass}].

The execution policy chooses from ready steps:

vt=π(G,Et,mt,ct),v_t = \pi(G, E_t, m_t, c_t),

where EtE_t is the current evidence packet, mtm_t is workflow memory, and ctc_t is the current critique state.

A critique loop can be modeled as

yk+1=revise⁡(yk,critique⁡(yk,Et)).y_{k+1} = \operatorname{revise}(y_k, \operatorname{critique}(y_k, E_t)).

The loop should stop for a reason, not because the interface runs out of patience:

stop⁡∈{critique passed,retry budget exhausted,human checkpoint,blocked action}.\operatorname{stop} \in \{\text{critique passed}, \text{retry budget exhausted}, \text{human checkpoint}, \text{blocked action}\}.

The audit log is the compact trace:

A={(t,vt,inputs,outputs,status,reason)}t=1T.A = \{(t, v_t, \text{inputs}, \text{outputs}, \text{status}, \text{reason})\}_{t=1}^T.

The audit log does not prove the answer is true. It proves the workflow is inspectable: what ran, why it ran, what it used, what changed, and where it stopped.

03

03

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt
Runnable code 1steps = { "plan": {"parents": [], "status": "pass"}, "gather_image": {"parents": ["plan"], "s...python
steps = {
    "plan": {"parents": [], "status": "pass"},
    "gather_image": {"parents": ["plan"], "status": "pass"},
    "gather_doc": {"parents": ["plan"], "status": "pass"},
    "draft_visual": {"parents": ["gather_image", "gather_doc"], "status": "pass"},
    "critique": {"parents": ["draft_visual"], "status": "needs_retry"},
    "retry": {"parents": ["critique"], "status": "pass"},
    "human_checkpoint": {"parents": ["retry"], "status": "waiting"},
    "handoff": {"parents": ["human_checkpoint"], "status": "locked"},
}

def ready(step_id):
    return all(steps[parent]["status"] == "pass" for parent in steps[step_id]["parents"])

def next_ready_steps():
    return [step_id for step_id in steps if ready(step_id) and steps[step_id]["status"] != "pass"]

def audit_event(step_id, reason):
    return {
        "step": step_id,
        "parents": steps[step_id]["parents"],
        "status": steps[step_id]["status"],
        "reason": reason,
    }

print(next_ready_steps())
print(audit_event("human_checkpoint", "permission needed before handoff"))

The witness separates three things that are often blurred together: dependency readiness, critique outcome, and handoff permission.

04

04

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the Agentic Multimodal Workflow Studio to predict the correct workflow shape before the proof unlocks:

  • Decompose: decide whether the goal needs a branch plan, tool DAG, or stop path.
  • Dependency order: decide which steps must finish before generation or action.
  • Critique/retry: decide whether a draft should pass, revise, branch, or stop.
  • Audit handoff: decide whether the workflow is ready to hand off or needs a checkpoint.

Before reveal, step IDs, execution order, retry counts, checkpoint status, blocked edges, and audit events stay locked. After reveal, inspect the workflow graph and ask whether every output has the right parent evidence.

Live Concept Demo

Explore Agentic Multimodal Workflows: Plans, Tools, Critique, Handoff

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5graduatecode-aligned
Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Agentic Multimodal Workflows: Plans, Tools, Critique, Handoff should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Turn a multimodal agent into an inspectable workflow: decompose the task, sequence tools, run visual generation loops, critique and retry, checkpoint risky steps, and hand off with an audit log.

Prediction open01 / Intuition
Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Agentic Multimodal Workflows: Plans, Tools, Critique, Handoff should make visible.

Visual Inquiry

Make the image answer a mathematical question

Turn a multimodal agent into an inspectable workflow: decompose the task, sequence tools, run visual generation loops, critique and retry, checkpoint risky steps, and hand off with an audit log.

4/4 stages readyLive demo connected
Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Agentic Multimodal Workflows: Plans, Tools, Critique, Handoff easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

paper · 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging FaceShen et al.

Primary support for a workflow controller that plans tasks, selects models, executes them, and summarizes results.

Open source
paper · 2022Visual Programming: Compositional visual reasoning without trainingGupta and Kembhavi

Primary support for expressing visual reasoning as interpretable programs that compose image, language, and reasoning modules.

Open source
paper · 2023ViperGPT: Visual Inference via Python Execution for ReasoningSuris, Menon, and Vondrick

Support for translating visual questions into executable programs that call vision-language APIs.

Open source
paper · 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language ModelsLu et al.

Support for composing tools into multimodal reasoning chains with module inventory and planned execution.

Open source
paper · 2023Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsYao et al.

Support for branch-and-evaluate reasoning rather than committing to a single linear path.

Open source
paper · 2023Self-Refine: Iterative Refinement with Self-FeedbackMadaan et al.

Primary support for feedback and refinement loops that improve an initial output without supervised training.

Open source

Claim Review

Turn a multimodal agent into an inspectable workflow: decompose the task, sequence tools, run visual generation loops, critique and retry, checkpoint risky steps, and hand off with an audit log.

Status2 substantive reviews recorded

Claims without a substantive review badge still need exact source-support review.

Sources6 references

shen-2023-hugginggpt, gupta-2022-visprog, suris-2023-vipergpt, lu-2023-chameleon, yao-2023-tree-of-thoughts, madaan-2023-self-refine

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedAn agentic multimodal workflow should expose task decomposition, dependency order, tool/model selection, and execution results as an inspectable graph rather than hiding them inside a single chat response.Claim metadata: source checked

The references support decomposing tasks into executable steps, selecting external modules or tools, and using interpretable program-like workflows for multimodal tasks.

Sources: HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face, Visual Programming: Compositional visual reasoning without training, ViperGPT: Visual Inference via Python Execution for Reasoning, Chameleon: Plug-and-Play Compositional Reasoning with Large Language ModelsThe page teaches a workflow proof contract, not a claim that every agent system uses an identical DAG runtime or that execution is automatically reliable.A bounded review summary is present; still check caveats and exact reference scope.

Checked HuggingGPT for task planning/model selection/execution/response, VISPROG for interpretable visual programs, ViperGPT for executable visual reasoning programs, and Chameleon for plug-and-play compositional tool use. GPT Pro publication critique remains pending because the local Oracle lane is unavailable on this workstation.

Reviewer: codex-local-primary-reference-audit; reviewed 2026-07-05
Equation 1Runnable code 1Interactive demo
Substantively reviewedLonger agentic workflows need explicit critique, retry, checkpoint, and handoff gates so outputs can be revised, audited, or stopped before final delivery.Claim metadata: source checked

The references support evaluating alternatives, refining outputs after feedback, and carrying reflection across attempts.

Sources: Tree of Thoughts: Deliberate Problem Solving with Large Language Models, Self-Refine: Iterative Refinement with Self-Feedback, shinn-2023-reflexionCritique and reflection are not magic correctness checks; the demo therefore requires explicit audit events and checkpoint gates before handoff.A bounded review summary is present; still check caveats and exact reference scope.

Checked Tree of Thoughts for branch-and-evaluate search, Self-Refine for iterative feedback/refinement, and Reflexion for reflection memory across attempts. The local demo scopes these ideas into learner-facing gates: critique, retry, human checkpoint, and audit handoff.

Reviewer: codex-local-primary-reference-audit; reviewed 2026-07-05
Equation 2Runnable code 1Interactive demo

Source support candidates

paper 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face

Primary support for a workflow controller that plans tasks, selects models, executes them, and summarizes results.

paper 2022Visual Programming: Compositional visual reasoning without training

Primary support for expressing visual reasoning as interpretable programs that compose image, language, and reasoning modules.

paper 2023ViperGPT: Visual Inference via Python Execution for Reasoning

Support for translating visual questions into executable programs that call vision-language APIs.

paper 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models

Support for composing tools into multimodal reasoning chains with module inventory and planned execution.

Mechanism checks

Equation 1
G=(V,E)G=(V,E)
Equation 2
vi=(rolei,inputsi,tooli,outputsi,statusi)v_i = (\text{role}_i, \text{inputs}_i, \text{tool}_i, \text{outputs}_i, \text{status}_i)
Runnable code 1steps = { "plan": {"parents": [], "status": "pass"}, "gather_image": {"parents": ["plan"], "s...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Turn a multimodal agent into an inspectable workflow: decompose the task, sequence tools, run visual generation loops, critique and retry, checkpoint risky steps, and hand off with an audit log.

Readiness0/3 checks ready
Predict

Before touching the demo, predict one visible change that should happen in Agentic Multimodal Workflows: Plans, Tools, Critique, Handoff.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Grounded research drawerClose
ConceptAgentic Multimodal Workflows: Plans, Tools, Critique, HandoffMachine Learning
Runnable code comparisonAgentic Multimodal Workflows: Plans, Tools, Critique, Handoff runnable code 1steps = { "plan": {"parents": [], "status": "pass"}, "gather_image": {"parents": ["plan"], "s...Prediction before revealAgentic Multimodal Workflows: Plans, Tools, Critique, Handoff interactive demoManipulate one control and predict the visible change.
Grounded room questionWhat is the smallest example that makes Agentic Multimodal Workflows: Plans, Tools, Critique, Handoff click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.
Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptMachine Learning

Agentic Multimodal Workflows: Plans, Tools, Critique, Handoff

Attached question

What is the smallest example that makes Agentic Multimodal Workflows: Plans, Tools, Critique, Handoff click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action
Local action draft

This draft stays in this browser, attached to the selected learning item.

No local draft saved.
Evidence to inspect
  • References to inspect: attached references on this page.
  • Definition, prerequisite, and contrast concept links
  • The equation or runnable code that makes the concept operational
  • One demo state that shows the invariant instead of a slogan
What would resolve this
  • The learner can state the mechanism in their own words
  • The learner can name the prerequisite that would repair confusion
  • The learner can predict how the mechanism changes under one perturbation
Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Agentic Multimodal Workflows: Plans, Tools, Critique, Handoff Selected item key: recorded for copy. Context: Machine Learning Page anchor: recorded for copy. Open question: What is the smallest example that makes Agentic Multimodal Workflows: Plans, Tools, Critique, Handoff click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context
concept/concept-notebook/machine-learning/agentic-multimodal-workflows concept:machine-learning/agentic-multimodal-workflows