Skip to content

Resumable Workflow skips an interrupted ordinary tool call; direct Agent root replays it #7719

Open

Description

@semenovmath

Summary

With google-adk 2.13.0, a resumable single-agent Workflow does not re-enter an ordinary async tool whose function call was persisted before cancellation. Resuming the same invocation returns without executing the pending tool, generating a final model response, or producing root output. The same Agent used directly as the App root replays the identical function-call ID and arguments and completes.

This uses an ordinary BaseTool, not long-running tools, confirmation, task delegation, or external state adapters. No provider, credentials, ADK patches, event filtering, or synthetic FunctionResponse is involved.

Environment and reproduction

Validated with Python 3.13.17 on Linux and google-adk==2.13.0. Install google-adk==2.13.0, save the following as repro.py, and run python repro.py. Both cases use a new native InMemorySessionService. Cancellation occurs after the ordinary tool is entered but before it returns. Each resume reuses the invocation ID and original user message. Calls use randomly generated IDs, not stable model-generated IDs that could hide regeneration.

Observed

Root Pending-call generations Replacement tool calls Retained user / call / response Final model responses
Direct Agent 1 1, same ID and args 1 / 1 / 1 1
Workflow(START -> Agent) 1 0 1 / 1 / 0 0

Before Workflow resume, persisted events are user(path empty), Workflow checkpoint(path empty, worker status 2), and worker function call(path root@1/worker@1). Resume emits another empty-path Workflow checkpoint (status 2), an empty worker event, a Workflow checkpoint (status 3), and an empty root event. No function response or non-null Event.output is emitted.

The direct-root control emits the pending tool response and final Content text done. Its Event.output is null, which is not the bug asserted here.

Expected

For this ordinary interrupted tool, native best-effort, at-least-once resumption should re-enter the retained function call using the same ID and arguments, as the direct-root control does. This is not a request for exactly-once side effects.

Source observation

In installed flows/llm_flows/core/_resume.py, decide_resume lines 352-380 treats an unanswered call as a pause when resumability is enabled. The branch decision runs before the trailing-call replay path at lines 439-452. The empty-path Workflow checkpoint emitted on resume appears to prevent this ordinary call from being treated as the trailing unexecuted call. This is a suspected interaction, not a proposed patch.

Related but different: #7684 concerns multiple tool confirmations and partial answers; this reproducer has one ordinary tool and no confirmation or supplied tool response.

Self-contained reproducer

"""No provider, credentials, private adapters, or ADK patches required."""
import asyncio
import json
import platform
import uuid
from importlib.metadata import version

from google.adk.agents import Agent
from google.adk.apps import App, ResumabilityConfig
from google.adk.models.base_llm import BaseLlm
from google.adk.models.llm_response import LlmResponse
from google.adk.runners import Runner
from google.adk.sessions import InMemorySessionService
from google.adk.tools import BaseTool
from google.adk.workflow import START, Workflow
from google.genai import types


class Model(BaseLlm):
    model: str = "scripted"
    generations: int = 0
    final_responses: int = 0

    async def generate_content_async(self, llm_request, stream=False):
        answered = any(p.function_response for c in llm_request.contents for p in c.parts or [])
        if answered:
            self.final_responses += 1
            parts = [types.Part(text="done")]
        else:
            self.generations += 1
            if self.generations > 1:
                raise RuntimeError("pending call regenerated")
            parts = [types.Part(function_call=types.FunctionCall(
                id="call_" + uuid.uuid4().hex, name="work", args={"value": 7}))]
        yield LlmResponse(content=types.Content(role="model", parts=parts))


class Tool(BaseTool):
    def __init__(self, block):
        super().__init__(name="work", description="Ordinary async tool")
        self.block = block
        self.entered = asyncio.Event()
        self.calls = []

    def _get_declaration(self):
        return types.FunctionDeclaration(name=self.name, description=self.description,
            parameters_json_schema={"type": "object", "properties": {"value": {"type": "integer"}},
                                    "required": ["value"]})

    async def run_async(self, *, args, tool_context):
        self.calls.append((tool_context.function_call_id, args))
        self.entered.set()
        if self.block:
            await asyncio.Event().wait()
        return {"value": args["value"]}


def summarize(event):
    return {"author": event.author, "path": event.node_info.path,
            "call_names": [c.name for c in event.get_function_calls()],
            "response_names": [r.name for r in event.get_function_responses()],
            "agent_state": event.actions.agent_state,
            "output": event.output,
            "text": [p.text for p in event.content.parts or [] if p.text] if event.content else []}


async def probe(wrapped):
    service = InMemorySessionService()
    await service.create_session(app_name="proof", user_id="u", session_id="s")
    model = Model()

    async def launch(tool):
        agent = Agent(name="worker", model=model, tools=[tool])
        root = Workflow(name="root", edges=[(START, agent)]) if wrapped else agent
        runner = Runner(app=App(name="proof", root_agent=root,
            resumability_config=ResumabilityConfig(is_resumable=True)), session_service=service)
        return [event async for event in runner.run_async(user_id="u", session_id="s",
            invocation_id="same-invocation", new_message=types.Content(role="user",
            parts=[types.Part(text="go")]))]

    old = Tool(block=True)
    task = asyncio.create_task(launch(old))
    try:
        await asyncio.wait_for(old.entered.wait(), 5)
    finally:
        task.cancel()
        await asyncio.gather(task, return_exceptions=True)
    saved = await service.get_session(app_name="proof", user_id="u", session_id="s")
    before = [summarize(e) for e in saved.events]
    replacement = Tool(block=False)
    resumed = await asyncio.wait_for(launch(replacement), 5)
    saved = await service.get_session(app_name="proof", user_id="u", session_id="s")
    result = {"workflow": wrapped, "same_call_and_args": old.calls == replacement.calls,
        "original_calls": len(old.calls), "replacement_calls": len(replacement.calls),
        "call_generations": model.generations, "final_responses": model.final_responses,
        "saved_before_resume": before, "resumed_events": [summarize(e) for e in resumed],
        "user_events": sum(e.author == "user" for e in saved.events),
        "function_calls": sum(len(e.get_function_calls()) for e in saved.events),
        "function_responses": sum(len(e.get_function_responses()) for e in saved.events)}
    print(json.dumps(result, default=str))
    return result


async def main():
    print("ADK", version("google-adk"), "Python", platform.python_version(), platform.system())
    control = await probe(False)
    workflow = await probe(True)
    assert control["same_call_and_args"] and control["function_responses"] == 1
    assert control["call_generations"] == control["final_responses"] == 1
    assert workflow["replacement_calls"] == workflow["function_responses"] == 0
    assert workflow["call_generations"] == 1 and workflow["final_responses"] == 0


asyncio.run(main())

Activity

  1. widechaos commented on Oct 25, 2026

    @widechaos

    I reproduced this on current main (12c7bed7) with the public scripted Runner example and network access blocked: the direct root replays the same call ID/args and completes, while Workflow never re-enters the tool.

    The resume checkpoint follows the pending call, so decide_step_resume pauses before its trailing-call replay path. A local checkpoint bypass then exposes a second problem: with no resume_inputs, the wrapper appends transient input after the prior node history, and the next model request loses the call/response pair. Moving that input before the node history restores completion in the diagnostic run.

    I discarded the checkpoint filter because it also changes a long-running-tool control and cannot distinguish snapshots carrying other actions. Before I work on a fix, should ordinary cancellation resume reconstruct the transient input before prior node history, and is there an intended way to identify a pure ancestor Workflow checkpoint?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions