Cadence Internals(2):Cadence Workflow History Protocol

[longquanzheng]

State machines

Decision

Scheduled--->Started--->Completed (happy path)
         --->Timeout(ScheduledToStartTimeout)
                    --->Timeout(StartToCloseTimeout)
                    --->Failed

Activity

Scheduled--->Started--->Completed (happy path)
         --->Timeout(ScheduledToStartTimeout)
                   --->Timeout(StartToCloseTimeout  or ScheduledToCloseTimeout)
                    --->Failed 

Timer

Scheduled--->Fired 
         --->Canceled 

Child Workflow

Initiated--->Started--->Completed (happy path)
                   --->Timeout 
                   --->Failed 

Rules

1. At any point of time, there can only be one decision pending (scheduled or started)

2. DecisionTaskCompleted event drives the workflow to make progress. It tells Cadence server about the next things happening to the workflow
   Schedule an activity → by ActivityScheduledEvent
   Scheduled a timer → by TimerScheduledEvent
   Scheduled a ChildWorkflow → by ChildWorklfowInitiatedEvent
   …
   Basically, decision [started event] task is the boundary for SDK to process a decision task.

3. The very first decision is scheduled by the server. Only certain events can trigger a next decision, include:
   ActivityTaskCompleted/Failed/Timeout
   WorkflowExecutionSignaled
   TimerFired
   ChildWorkflowStarted
   ChildWorkflowCompleted/Failed/Timeout
   ...
   A rule of thumb is that any event that can be useful to let a workflow to make progress.
   For example, the ActivityTaskStarted event is not one of them because it’s not needed for workflow to make any progress/decision.
   However, it should be clear that triggering decision task doesn't mean the workflow will always make some progress. It's possible/okay that a decision task will return empty results -- decision task completed with nothing scheduled. For example, a workflow receive a "trash signal" will trigger a decision task but it won't make any progress.

4. A DecisionCloseEvents( DecisionCompletedEvent, DecisionTimeoutEvent,DecisionFailedEvent) must be right after a DecisionStartedEvent. In other words, there cannot be other events between a DecisionStartedEvent and a DecisionCloseEvent. This rule is for making it easy to implement the replay mechanism in SDK (it makes the "boundary clear in history ).

5. Because of 4, when a workflow is executing a decision(last event is DecisionStartedEvent), all other incoming events, like WorkflowExecutionSignaled(receiving a signal), ActivityTaskStarted/ActivityTaskCompleted/etc, are stored into a “Buffer” of the workflow. The event will only be shown in the history when the pending decision is closed(completed/timeouted/failed).
   For example, when a decision is started and in progress, if a workflow receives a signal, it will go to the buffer. Only when the decision is closed, the signal event can become visible in history.

6. A special case of 5: When the pending decision is completed by deciding to close the workflow(completing or failing), the server will reject the decision if there are any events in the buffer. This is to guarantee that workflow won’t lose any of those events like signals. Because after a workflow is closed, it won’t be able to process them anymore.

7. LocalActivity is executing within a decision. So there is no Scheduled/Started event.

8. History of Activity with retryOptions is special. The ActivityStartedEvent is being written when the activity is closed. (this is to keep history immutable as invariance). You can use WebUI to check if the activity has started by looking at the “Pending Activities” tab.

[longquanzheng]

• Also read more in https://stackoverflow.com/questions/62904129/what-exactly-is-a-cadence-decision-task/63964726#63964726
• Use iWF on top of Cadence/Temporal will simplify everything: https://github.com/indeedeng/iwf