Ten Past Ten At Night

This runs on a durable workflow engine that recovers by re-executing the workflow function from the top, serving already recorded activity results from the history, and comparing each command the function emits against the command recorded at that position.

func ExportWorkflow(ctx workflow.Context, tenantID string) error {
    reportID := uuid.New().String()

    if err := workflow.ExecuteActivity(ctx, BuildReport, tenantID, reportID).Get(ctx, nil); err != nil {
        return err
    }
    if time.Now().Hour() >= 22 {
        return workflow.ExecuteActivity(ctx, NotifyBatch, reportID).Get(ctx, nil)
    }
    return workflow.ExecuteActivity(ctx, NotifyNow, reportID).Get(ctx, nil)
}

BuildReport completes at 21:58 and its result is recorded. NotifyNow is scheduled at 21:59 and is still running when the worker process is restarted at 22:03. What happens on the replay?

The replay fails at BuildReport, because uuid.New() returns a different reportID and the scheduled activity's input no longer matches the recorded event.

The body runs again, time.Now() now reads after 22:00, and the function emits a NotifyBatch command at the position where the history holds NotifyNow. The commands do not match and the workflow task fails with a non-determinism error.

Nothing goes wrong. Everything up to the last recorded event is served from the history without running the function body again, so the branch is only ever evaluated once, at 21:58.

BuildReport runs a second time, because the restart lost the in-memory result and the engine has no way to know the activity already completed.