Ability to get partial output data in GET /instances/{instanceId} API about durabletask-java HOT 8 CLOSED

@kaibocai Do you have any ETA on this?

from durabletask-java.

Hi @kanupriya15025 , sorry for the delay. Based on my research, I don't think we currently support querying status while the orchestration is still running. This may be an improvement we can add in the future.
@cgillum, @davidmrdavid have you ever gotten customers asking for a feature like this?

from durabletask-java.

I know there's a showHistory flag which returns the history of what is happening with my orchestration, but I want to store some custom activity status messages after each completion.

We don't have the ability to store custom status for activity results, unfortunately. Just the activity output. I'm not sure if it makes sense for this to be a feature of Durable Functions, or if it's better to do that in some external storage.

The output is returned in the API once the entire orchestration completes, but I need to get partial statuses even if the orchestration is running. Is there a way to achieve that?

It should be possible to query orchestration history even when the orchestration is still running. Is that not the case from what you're seeing?

from durabletask-java.

@cgillum @kaibocai
By custom status, I meant the task output itself. In a case, where an activity takes too long to complete due to some error, the entire output of previous tasks is stalled in that case.
It is possible to query history table but querying the history can be slow as it is huge. If this can be embedded as a response in the /instances API, it would make it quicker, easier and more intuitive.

from durabletask-java.

In a case, where an activity takes too long to complete due to some error, the entire output of previous tasks is stalled in that case.

You should be able to get the output of previously completed tasks of a running orchestration already. Can you provide an example of where this is not the case for you?

It is possible to query history table but querying the history can be slow as it is huge. If this can be embedded as a response in the /instances API, it would make it quicker, easier and more intuitive.

This is already possible with the HTTP API: https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-http-api#get-instance-status. Something like this should work:

GET /runtime/webhooks/durabletask/instances/abc123?showHistory=true&showHistoryOutput=true

from durabletask-java.

@cgillum Is this the field you are aiming at - "output"?
Here's a response where I can see a few activities completed but the output is still null. This only populates when all the activities complete.

{
  "name": "WorkflowOrchestrator",
  "instanceId": "instanceId",
  "runtimeStatus": "Running",
  "input": {
    "input" : "someInput"
  },
  "customStatus": "",
  "output": null,
  "createdTime": "2023-10-05T08:20:14Z",
  "lastUpdatedTime": "2023-10-05T08:20:35Z",
  "historyEvents": [
    {
      "EventType": "ExecutionStarted",
      "Input": "SomeInput,
      "Correlation": null,
      "ScheduledStartTime": null,
      "Generation": 0,
      "Timestamp": "2023-10-05T08:20:14.576414Z",
      "FunctionName": "WorkflowOrchestrator"
    },
    {
      "EventType": "TaskCompleted",
      "Result": "{\"testTask1\":{\"startDate\":\"2023-10-05T08:20:17.399572\",\"firstTask\":{\"status_code\":500,\"reason\":\"Internal Server Error\",\"headers\":\"[Lorg.apache.http.Header;@33ad2eb2\",\"body\":\"org.apache.http.impl.io.EmptyInputStream@180fb89\"},\"state\":\"SUCCESS\",\"endDate\":\"2023-10-05T08:20:18.583269\"}}",
      "Timestamp": "2023-10-05T08:20:18.843449Z",
      "ScheduledTime": "2023-10-05T13:50:16.908051+05:30",
      "FunctionName": "AEPHTTPPollingOperator",
      "Input": null
    },
    {
      "EventType": "TimerCreated",
      "FireAt": "2023-10-05T08:20:24.893Z",
      "Timestamp": "2023-10-05T08:20:19.919621Z"
    },
    {
      "EventType": "TimerFired",
      "FireAt": "2023-10-05T08:20:24.893Z",
      "Timestamp": "2023-10-05T08:20:19.919683Z"
    },
    {
      "EventType": "TaskCompleted",
      "Result": "{\"testTask1\":{\"startDate\":\"2023-10-05T08:20:17.399572\",\"firstTask\":{\"status_code\":500,\"reason\":\"Internal Server Error\",\"headers\":\"[Lorg.apache.http.Header;@33ad2eb2\",\"body\":\"org.apache.http.impl.io.EmptyInputStream@180fb89\"},\"state\":\"SUCCESS\",\"endDate\":\"2023-10-05T08:20:18.583269\"},\"testTask2\":{\"startDate\":\"2023-10-05T08:20:26.186607\",\"firstTask\":{\"status_code\":500,\"reason\":\"Internal Server Error\",\"headers\":\"[Lorg.apache.http.Header;@6526546a\",\"body\":\"org.apache.http.impl.io.EmptyInputStream@180fb89\"},\"state\":\"SUCCESS\",\"endDate\":\"2023-10-05T08:20:27.304066\"}}",
      "Timestamp": "2023-10-05T08:20:28.005728Z",
      "ScheduledTime": "2023-10-05T13:50:25.686154+05:30",
      "FunctionName": "AEPHTTPPollingOperator",
      "Input": null
    },
    {
      "EventType": "TimerCreated",
      "FireAt": "2023-10-05T08:20:34.153Z",
      "Timestamp": "2023-10-05T08:20:29.186666Z"
    },
    {
      "EventType": "TimerFired",
      "FireAt": "2023-10-05T08:20:34.153Z",
      "Timestamp": "2023-10-05T08:20:29.186671Z"
    },
    {
      "EventType": "TaskScheduled",
      "Name": "AEPHTTPPollingOperator",
      "Input": null,
      "Timestamp": "2023-10-05T08:20:35.579164Z"
    }
  ]
}

from durabletask-java.

@kanupriya15025 I'm able to see the activity output in the Result properties of the TaskCompleted events:

    {
      "EventType": "TaskCompleted",
      "Result": "{\"testTask1\":{\"startDate\":\"2023-10-05T08:20:17.399572\",\"firstTask\":{\"status_code\":500,\"reason\":\"Internal Server Error\",\"headers\":\"[Lorg.apache.http.Header;@33ad2eb2\",\"body\":\"org.apache.http.impl.io.EmptyInputStream@180fb89\"},\"state\":\"SUCCESS\",\"endDate\":\"2023-10-05T08:20:18.583269\"}}",
      "Timestamp": "2023-10-05T08:20:18.843449Z",
      "ScheduledTime": "2023-10-05T13:50:16.908051+05:30",
      "FunctionName": "AEPHTTPPollingOperator",
      "Input": null
    },
...
    {
      "EventType": "TaskCompleted",
      "Result": "{\"testTask1\":{\"startDate\":\"2023-10-05T08:20:17.399572\",\"firstTask\":{\"status_code\":500,\"reason\":\"Internal Server Error\",\"headers\":\"[Lorg.apache.http.Header;@33ad2eb2\",\"body\":\"org.apache.http.impl.io.EmptyInputStream@180fb89\"},\"state\":\"SUCCESS\",\"endDate\":\"2023-10-05T08:20:18.583269\"},\"testTask2\":{\"startDate\":\"2023-10-05T08:20:26.186607\",\"firstTask\":{\"status_code\":500,\"reason\":\"Internal Server Error\",\"headers\":\"[Lorg.apache.http.Header;@6526546a\",\"body\":\"org.apache.http.impl.io.EmptyInputStream@180fb89\"},\"state\":\"SUCCESS\",\"endDate\":\"2023-10-05T08:20:27.304066\"}}",
      "Timestamp": "2023-10-05T08:20:28.005728Z",
      "ScheduledTime": "2023-10-05T13:50:25.686154+05:30",
      "FunctionName": "AEPHTTPPollingOperator",
      "Input": null
    },

from durabletask-java.

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

from durabletask-java.