Skip to content

yield on task_all can return None #513

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
rokcarl opened this issue Jul 2, 2024 · 2 comments
Open

yield on task_all can return None #513

rokcarl opened this issue Jul 2, 2024 · 2 comments
Labels
Needs: Investigation 🔍 A deeper investigation needs to be done by the project maintainers. P1 Priority 1

Comments

@rokcarl
Copy link

rokcarl commented Jul 2, 2024 •
edited
Loading

Running yield on task_all can return None. This goes against the docs which try to sum(results), but this might be None. I would expect a list of objects, since I published a list of tasks, even if some of the objects in the list are exception objects or something similar.

Additionally, I cannot fully debug this as the underlying issue is masked by the stacktrace. I wanted to create a self-sufficient small reproducible example, but can't since it works in other cases.

Check the logs 
[2024-07-02T13:01:07.978Z] Executing 'Functions.myfunc_http' (Reason='This function was programmatically called via the host APIs.', Id=8f58a29f-6d27-4c24-8495-bebab07d6674)
[2024-07-02T13:01:08.467Z] Executed 'Functions.myfunc_http' (Succeeded, Id=8f58a29f-6d27-4c24-8495-bebab07d6674, Duration=572ms)
[2024-07-02T13:01:09.503Z] Host lock lease acquired by instance ID '0000000000000000000000001DA5AB02'.
[2024-07-02T13:01:10.854Z] Executing 'Functions.myfunc2' (Reason='(null)', Id=7d75b278-6244-46ba-8642-444af122babc)
[2024-07-02T13:01:12.519Z] Processing order ORD123
[2024-07-02T13:01:12.519Z] Running tasks:
[2024-07-02T13:01:12.519Z] Running tasks ['sanity_check'] with data {'order_number': 'ORD123', 'run_id': 'fa420a76-c5d1-49cf-9baa-687778d75454'}
[2024-07-02T13:01:12.519Z] [<azure.durable_functions.models.Task.AtomicTask object at 0x145a6be50>]
[2024-07-02T13:01:12.554Z] d9ad67eeef4a47c594999f412be65c40: Function 'myfunc2 (Orchestrator)' failed with an error. Reason: System.AggregateException: One or more errors occurred. (Result: Failure
[2024-07-02T13:01:12.555Z] Exception: ValueError: stage_results is None for tasks: [<azure.durable_functions.models.Task.AtomicTask object at 0x145a6be50>] in stage ['sanity_check'], data {'order_number': 'ORD123', 'run_id': 'fa420a76-c5d1-49cf-9baa-687778d75454'}.
[2024-07-02T13:01:12.555Z] Stack:   File "/usr/local/Cellar/azure-functions-core-tools@4/4.0.5801/workers/python/3.10/OSX/X64/azure_functions_worker/dispatcher.py", line 604, in _handle__invocation_request
[2024-07-02T13:01:12.555Z]     call_result = await self._loop.run_in_executor(
[2024-07-02T13:01:12.555Z]   File "/Users/user/.pyenv/versions/3.10.10/lib/python3.10/concurrent/futures/thread.py", line 58, in run
[2024-07-02T13:01:12.555Z]     result = self.fn(*self.args, **self.kwargs)
[2024-07-02T13:01:12.556Z]   File "/usr/local/Cellar/azure-functions-core-tools@4/4.0.5801/workers/python/3.10/OSX/X64/azure_functions_worker/dispatcher.py", line 933, in _run_sync_func
[2024-07-02T13:01:12.556Z]     return ExtensionManager.get_sync_invocation_wrapper(context,
[2024-07-02T13:01:12.556Z]   File "/usr/local/Cellar/azure-functions-core-tools@4/4.0.5801/workers/python/3.10/OSX/X64/azure_functions_worker/extension.py", line 215, in _raw_invocation_wrapper
[2024-07-02T13:01:12.556Z]     result = function(**args)
[2024-07-02T13:01:12.556Z]   File "/Users/user/work/comp/myfunc-v2/code/.venv/lib/python3.10/site-packages/azure/durable_functions/orchestrator.py", line 69, in handle
[2024-07-02T13:01:12.556Z]     return Orchestrator(fn).handle(DurableOrchestrationContext.from_json(context_body))
[2024-07-02T13:01:12.556Z]   File "/Users/user/work/comp/myfunc-v2/code/.venv/lib/python3.10/site-packages/azure/durable_functions/orchestrator.py", line 47, in handle
[2024-07-02T13:01:12.556Z]     return self.task_orchestration_executor.execute(context, context.histories, self.fn)
[2024-07-02T13:01:12.556Z]   File "/Users/user/work/comp/myfunc-v2/code/.venv/lib/python3.10/site-packages/azure/durable_functions/models/TaskOrchestrationExecutor.py", line 69, in execute
[2024-07-02T13:01:12.556Z]     evaluated_user_code = fn(context)
[2024-07-02T13:01:12.556Z]   File "/Users/user/work/comp/myfunc-v2/code/function_app.py", line 128, in myfunc2
[2024-07-02T13:01:12.556Z]     "results": list(run_flow(context, order_data, flow)),
[2024-07-02T13:01:12.556Z]   File "/Users/user/work/comp/myfunc-v2/code/function_app.py", line 289, in run_flow
[2024-07-02T13:01:12.556Z]     raise ValueError(f"stage_results is None for tasks: {tasks} in stage {stage}, data {data}.")
[2024-07-02T13:01:12.556Z] )
[2024-07-02T13:01:12.556Z]  ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException: Result: Failure
[2024-07-02T13:01:12.556Z] Exception: ValueError: stage_results is None for tasks: [<azure.durable_functions.models.Task.AtomicTask object at 0x145a6be50>] in stage ['sanity_check'], data {'order_number': 'ORD123', 'run_id': 'fa420a76-c5d1-49cf-9baa-687778d75454'}.
[2024-07-02T13:01:12.556Z] Stack:   File "/usr/local/Cellar/azure-functions-core-tools@4/4.0.5801/workers/python/3.10/OSX/X64/azure_functions_worker/dispatcher.py", line 604, in _handle__invocation_request
[2024-07-02T13:01:12.556Z]     call_result = await self._loop.run_in_executor(
[2024-07-02T13:01:12.556Z]   File "/Users/user/.pyenv/versions/3.10.10/lib/python3.10/concurrent/futures/thread.py", line 58, in run
[2024-07-02T13:01:12.556Z]     result = self.fn(*self.args, **self.kwargs)
[2024-07-02T13:01:12.556Z]   File "/usr/local/Cellar/azure-functions-core-tools@4/4.0.5801/workers/python/3.10/OSX/X64/azure_functions_worker/dispatcher.py", line 933, in _run_sync_func
[2024-07-02T13:01:12.556Z]     return ExtensionManager.get_sync_invocation_wrapper(context,
[2024-07-02T13:01:12.556Z]   File "/usr/local/Cellar/azure-functions-core-tools@4/4.0.5801/workers/python/3.10/OSX/X64/azure_functions_worker/extension.py", line 215, in _raw_invocation_wrapper
[2024-07-02T13:01:12.556Z]     result = function(**args)
[2024-07-02T13:01:12.556Z]   File "/Users/user/work/comp/myfunc-v2/code/.venv/lib/python3.10/site-packages/azure/durable_functions/orchestrator.py", line 69, in handle
[2024-07-02T13:01:12.556Z]     return Orchestrator(fn).handle(DurableOrchestrationContext.from_json(context_body))
[2024-07-02T13:01:12.556Z]   File "/Users/user/work/comp/myfunc-v2/code/.venv/lib/python3.10/site-packages/azure/durable_functions/orchestrator.py", line 47, in handle
[2024-07-02T13:01:12.556Z]     return self.task_orchestration_executor.execute(context, context.histories, self.fn)
[2024-07-02T13:01:12.556Z]   File "/Users/user/work/comp/myfunc-v2/code/.venv/lib/python3.10/site-packages/azure/durable_functions/models/TaskOrchestrationExecutor.py", line 69, in execute
[2024-07-02T13:01:12.556Z]     evaluated_user_code = fn(context)
[2024-07-02T13:01:12.556Z]   File "/Users/user/work/comp/myfunc-v2/code/function_app.py", line 128, in myfunc2
[2024-07-02T13:01:12.556Z]     "results": list(run_flow(context, order_data, flow)),
[2024-07-02T13:01:12.556Z]   File "/Users/user/work/comp/myfunc-v2/code/function_app.py", line 289, in run_flow
[2024-07-02T13:01:12.556Z]     raise ValueError(f"stage_results is None for tasks: {tasks} in stage {stage}, data {data}.")
[2024-07-02T13:01:12.556Z] 
[2024-07-02T13:01:12.556Z]    at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters, FunctionInvocationContext context) in /_/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:line 101
[2024-07-02T13:01:12.556Z]    at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) in /_/src/WebJobs.Script/Description/FunctionInvokerBase.cs:line 82
[2024-07-02T13:01:12.556Z]    at Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task`1 src) in /_/src/WebJobs.Script/Description/FunctionGenerator.cs:line 225
[2024-07-02T13:01:12.556Z]    at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:line 53
[2024-07-02T13:01:12.556Z]    --- End of inner exception stack trace ---
[2024-07-02T13:01:12.556Z]    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
[2024-07-02T13:01:12.556Z]    at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
[2024-07-02T13:01:12.556Z]    at Microsoft.Azure.WebJobs.Extensions.DurableTask.TaskOrchestrationShim.InvokeUserCodeAndHandleResults(RegisteredFunctionInfo orchestratorInfo, OrchestrationContext innerContext) in D:\a\_work\1\s\src\WebJobs.Extensions.DurableTask\Listener\TaskOrchestrationShim.cs:line 150. IsReplay: False. State: Failed. RuntimeStatus: Failed. HubName: TestHubName. AppName: . SlotName: . ExtensionVersion: 2.13.2. SequenceNumber: 5. TaskEventId: -1
[2024-07-02T13:01:12.593Z] Executed 'Functions.myfunc2' (Failed, Id=7d75b278-6244-46ba-8642-444af122babc, Duration=1754ms)
[2024-07-02T13:01:12.594Z] System.Private.CoreLib: Exception while executing function: Functions.myfunc2. System.Private.CoreLib: Orchestrator function 'myfunc2' failed: One or more errors occurred. (Result: Failure
[2024-07-02T13:01:12.594Z] Exception: ValueError: stage_results is None for tasks: [<azure.durable_functions.models.Task.AtomicTask object at 0x145a6be50>] in stage ['sanity_check'], data {'order_number': 'ORD123', 'run_id': 'fa420a76-c5d1-49cf-9baa-687778d75454'}.
[2024-07-02T13:01:12.594Z] Stack:   File "/usr/local/Cellar/azure-functions-core-tools@4/4.0.5801/workers/python/3.10/OSX/X64/azure_functions_worker/dispatcher.py", line 604, in _handle__invocation_request
[2024-07-02T13:01:12.594Z]     call_result = await self._loop.run_in_executor(
[2024-07-02T13:01:12.594Z]   File "/Users/user/.pyenv/versions/3.10.10/lib/python3.10/concurrent/futures/thread.py", line 58, in run
[2024-07-02T13:01:12.594Z]     result = self.fn(*self.args, **self.kwargs)
[2024-07-02T13:01:12.594Z]   File "/usr/local/Cellar/azure-functions-core-tools@4/4.0.5801/workers/python/3.10/OSX/X64/azure_functions_worker/dispatcher.py", line 933, in _run_sync_func
[2024-07-02T13:01:12.594Z]     return ExtensionManager.get_sync_invocation_wrapper(context,
[2024-07-02T13:01:12.594Z]   File "/usr/local/Cellar/azure-functions-core-tools@4/4.0.5801/workers/python/3.10/OSX/X64/azure_functions_worker/extension.py", line 215, in _raw_invocation_wrapper
[2024-07-02T13:01:12.594Z]     result = function(**args)
[2024-07-02T13:01:12.594Z]   File "/Users/user/work/comp/myfunc-v2/code/.venv/lib/python3.10/site-packages/azure/durable_functions/orchestrator.py", line 69, in handle
[2024-07-02T13:01:12.594Z]     return Orchestrator(fn).handle(DurableOrchestrationContext.from_json(context_body))
[2024-07-02T13:01:12.594Z]   File "/Users/user/work/comp/myfunc-v2/code/.venv/lib/python3.10/site-packages/azure/durable_functions/orchestrator.py", line 47, in handle
[2024-07-02T13:01:12.594Z]     return self.task_orchestration_executor.execute(context, context.histories, self.fn)
[2024-07-02T13:01:12.594Z]   File "/Users/user/work/comp/myfunc-v2/code/.venv/lib/python3.10/site-packages/azure/durable_functions/models/TaskOrchestrationExecutor.py", line 69, in execute
[2024-07-02T13:01:12.594Z]     evaluated_user_code = fn(context)
[2024-07-02T13:01:12.594Z]   File "/Users/user/work/comp/myfunc-v2/code/function_app.py", line 128, in myfunc2
[2024-07-02T13:01:12.594Z]     "results": list(run_flow(context, order_data, flow)),
[2024-07-02T13:01:12.594Z]   File "/Users/user/work/comp/myfunc-v2/code/function_app.py", line 289, in run_flow
[2024-07-02T13:01:12.594Z]     raise ValueError(f"stage_results is None for tasks: {tasks} in stage {stage}, data {data}.")
[2024-07-02T13:01:12.594Z] ).
[2024-07-02T13:02:43.613Z] Worker process started and initialized.
[2024-07-02T13:02:48.612Z] Host lock lease acquired by instance ID '0000000000000000000000001DA5AB02'.
Check the python code (tried summarizing it) 
def run_flow(context: df.DurableOrchestrationContext, data: dict, flow: list[list[str]]):
    results = {}
    for stage in flow:
        tasks = []
        logging.info(f"Running tasks {stage} with data {data}")
        for task_name in stage:
            tasks.append(context.call_activity(task_name, data))
        logging.info("Running tasks:")
        logging.info(tasks)
        stage_results = yield context.task_all(tasks)
        if stage_results is None:
            raise ValueError(f"stage_results is None for tasks: {tasks} in stage {stage}, data {data}.")

        for i, task_name in enumerate(stage):
            results[task_name] = stage_results[i]
    return results


@app.route(route="myfunc", methods=["POST"])
@app.durable_client_input(client_name="client")
async def myfunc_http(req: func.HttpRequest, client: df.DurableOrchestrationClient) -> func.HttpResponse:
    data = req.get_json()
    order_numbers = get_order_numbers()
    data["order_numbers"] = order_numbers

    instance_id = await client.start_new("myfunc2", client_input=data)
    return client.create_check_status_response(req, instance_id)


@app.orchestration_trigger(context_name="context")
def myfunc2(context: df.DurableOrchestrationContext):
    data = context.get_input() or {}

    flow = [
        ["sanity_check"],
        ["one", "two", "three"],
        ["summary"],
    ]

    results = []
    for order_number in data["order_numbers"]:
        logging.info(f"Processing order {order_number}")
        order_data = {"order_number": order_number, "run_id": data["run_id"]}
        results.append({
            "order_number": order_number,
            "results": list(run_flow(context, order_data, flow)),
        })
    logging.info("Returning results:")
    logging.info(results)
    return results

The code above can be called with the same parameters and succeed or it might fail.
I'm running this locally using func start.

There was a similar issue #428, but even applying the patch from #493 (it's not released yet unfortunately) doesn't change anything.
@rokcarl rokcarl mentioned this issue Aug 2, 2024
Closed
@andystaples
Copy link
Collaborator

Can you confirm - what are the conditions when yield task_all might return None? Is this when tasks is None, when tasks is an empty list, if all activities fail, or some other edge case? A minimal repro app might help us diagnose this issue, the code you provided doesn't have any activity definitions. @rokcarl

@andystaples andystaples added Needs: Investigation 🔍 A deeper investigation needs to be done by the project maintainers. P1 Priority 1 and removed Needs: Triage 🔍 labels Aug 20, 2024
@rokcarl
Copy link
Author

rokcarl commented Aug 21, 2024 •
edited
Loading

Hey, thanks for taking a look.

I've actually preemptively answered most of your questions in the issue above, please read carefully.

Also, if you check the code, you'll see that the tasks is not None or an empty list. Also evident from the ticket is that whether the tasks succeed or fail is unknown.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Needs: Investigation 🔍 A deeper investigation needs to be done by the project maintainers. P1 Priority 1
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rokcarl @andystaples