Workstream Proposal: Workflows / Function Composition about wg-serverless HOT 12 CLOSED

Can someone elaborate on what a "workflow specification" would define? Would it be some kind of workflow language/config to specify the list of functions that get invoked based on an incoming event(s)? Just trying to understand the scope of this work and what dependencies (if any) it might have on other specs - aside from CloudEvents.

from wg-serverless.

To me, this seems like a higher-level concern, and we don't have the lower-level aspects solved. As a consumer, I'm more concerned about how I can take my function code between clouds or open source tools without having to change my code. It appears to be premature to focus on orchestration rather than the building blocks of the function signature, function model, and SDK.

from wg-serverless.

A workflow specification helps with avoiding lock-in for application authors that have built significant portions of their applications having been built mostly in functions. In terms of goals, we could start with a white paper and gauging need / interest move on to a specification identifying:

Workflow Definition:

• Workflow ID
• Functions included in workflow (identified by correlation)
• which key/value pairs can be used

Function-to-function:

• inputs to a function
• outputs of a function

Function triggering:

• what criteria of events (combinations)
• sequential
• parallel

Error handling:

• signal actions when one function in the workflow fails like:
• • rollback (reverse workflow execution)
• • retry (failed function)
• • cancel (entire workflow at the failed function)

Out of scope:

• definition of how workflows are visualized (both in definition and in active execution)

from wg-serverless.

I agree with @barkerd427, but I'm even skeptical about achieving the level of standardization that would allow users to move "function code between clouds or open source tools without having to change [their] code" in the first place. Especially with regard to Function output, there are significant differences in what's supported across existing FaaS providers.

from wg-serverless.

I agree @markfisher. I'm not sure how much we've looked into that effort yet.

I've built a system like this workflow idea at a previous company for what were basically functions, and after a few years we had three systems like this because each group didn't think the other systems would meet their unique case. I'm concerned we could toil away at this for months and never reach a solution that would be adopted. This proposal also requires a definition of inputs and outputs at least in structure.

I can see how this is beneficial for providers and maybe the more unique cases of large FaaS deployments, but having been a part of those large deployments, they will likely create their own workflow engine.

This would be a great survey to users.

Which would you prefer?

A. Workflow automation for your functions that's portable across providers
B. Function portability between providers with a common SDK.

from wg-serverless.

For reference, see: cloudevents/spec#204 (comment)

from wg-serverless.

I'd actually prefer an input/output specification which doesn't necessarily mean a function signature definition. I think that without being able to properly describe the input and output of functions, you wouldn't be able easily create workflows (especially across FaaSes). I would look to something like HTTP which does a pretty good job at this. Naively cloudevents could be used, but not all payloads should be JSON encoded.

from wg-serverless.

From what I can tell there are at least three different concepts being discussed here but it might help to disambiguate them to avoid confusion.

1. Internal workflow to a serverless application. Defining the flow of an event through processing steps after it has been received from another party. This includes implementation details such as method signatures and non-functional aspects such error handling, transactions, etc. As mentioned by @barkerd427 this is very specific to different applications / teams and IMO yet another workflow specification should be avoided.

2. Tracking events (or enabling the tracking of events) through an application. This is less prescriptive than 1. but would allow an abstract definition of a flow that an event is expected to travel through. Such tracking might allow verification that a system conforms to it's specification. Compared to 1 this would be less prescriptive about how events are processed, but would define an ordering that should happen.

3. The external visible workflow of a set of functions. This would ignore the implementation details of an application but would specify that multiple events are expected to be observed in a particular order by a set of functions. It would not specify how these events will be processed but could / would specify how they would be correlated.

from wg-serverless.

More info from @cathyhongzhang : https://docs.google.com/document/d/107huVgYyt6JC4gW85DuJ9LDC9Sj8mcCGEqk0m64nhm4/edit#

from wg-serverless.

I'm trying to get a concrete mental model of exactly what we would be defining in this new workstream. If I take a simple use case of a workflow of: when eventType of "mytype" is received, process func1, func2 and then func3 serially

I'm assuming that the author of such a "workflow" would use the "Workflow Definition Specification" (WDS) that we create to specify this workflow - and would provide this workflow definition to their Serverless Platform. If more than one platform supported WDS then they could take that definition and deploy it to those platforms as well. However, would the functions themselves be portable? I'm guessing no. Which means they would need to be rewritten to support the function signatures of each Platform. And, I suspect the WDS file itself would most likely need to change between platforms to reference the platform specific functions - even if its just a name or version reference change.

If I'm correct, what level of interop or portability have we provided to the end-user? It feels like we're skipping a step in the interop/portability path by not addressing the function signatures. @cathyhongzhang am I nothing thinking of this correctly?

from wg-serverless.

@duglin , yes you are right that if more than one platform supported WDS then they could take that definition and deploy it to those platforms as well. To isolate the problems, workflow does not intend to cover function portability (function portability can be a separate discussion topic and addressed in a separate "Function Signature Specification") . Only the function name/identifier will be referenced in the "Workflow Definition Specification" and all detail information associated with that function name/identifier can be defined in a separate "Function Signature Specification". As long as the author of the use case use the same function name/identifier between platforms, the WDF does not need to change between platforms.

from wg-serverless.

This issue was opened so we could leverage github's voting/comment feature to determine which work item to focus on next. Since the vote is over, the issue will now be closed - but closing it has no effect on whether we address it in the future or not. The list of potential work items can still be found at: https://github.com/cncf/wg-serverless/tree/master/proposals

from wg-serverless.