Orchestrating HPC with AWS Step Functions and AWS Batch
This post gives an example of a workflow that involves AWS Step Functions, Batch, and Lambda Functions. This post seems to break up a workflow into multiple state machines.
Iterative polling patterns are often used when multiple jobs need to complete
before executing the next step in an analysis. Step Functions has some
synchronization primitives that integrate with other AWS services. Run a Job (.sync) will return a single success or failure message after a job
completes, reducing the number of messages in the event log.