Skip to content

Powerful Patterns for Developing Automated Workflows

January 4, 2015

Automated workflows can be thought of as long running processes — Processes that may take minutes, hours, days, or even weeks to run to completion. As such, workflows present challenges not encountered in “normal” software development.   Also, some workflows involve user interaction, while others may not. And, workflows can be implemented in WCF, the Windows Azure Cloud Services (PaaS), as well as through specialized workflow frameworks like Windows Workflow Foundation. The patterns presented herein are aimed at WCF and Azure implementations.

Over the last 2 years my interest in the design of automated workflows has been inspired by several architecture courses I have taken from Juval Lowy’s company, IDesign. While the courses themselves have been invaluable in me learning the design of service oriented applications, the IDesign Alumni Forum (on Google Groups) has proven to be a great way to learn additional concepts and techniques as well.

Membership in the Forum is available by taking one IDesign course. The contributors to the Forum’s various threads include industry experts (like Juval Lowy, Monty Montgomery from IDesign) as well as other highly experienced architects and developers working in our business. Thus, one can ask questions and get answers from knowledgeable people. And, one can also just sit back and read and learn. The knowledge I have gained from the Forum has directly benefited me in my professional work in many ways!

In this article I present a summary of some powerful patterns that can be used in the design of automated workflows. These include design patterns, messaging patterns, and workflow patterns. I thank Juval Lowy for his feedback on the second part of this article.

A Summary of Powerful Workflow Patterns

Below are sketches of patterns helpful in sequencing the steps of workflows. Most of these patterns are concerned with ways of decoupling things and/or producing maintainable code. I won’t dive into the details here in this summary. Workflow patterns are a world unto themselves. However, I hope the following list points you in the right direction to get you started.

State Machine – The full name is Finite State Machine. The purpose of using a state machine is that they have the power to facilitate very complex interactions while also producing highly maintainable code.   High value!  See the Wikipedia for more details at http://en.wikipedia.org/wiki/Finite-state_machine.

  • For a workflow, a state machine can control the sequence of the execution of the steps in a workflow. Here’s a sketch:   It is all based upon the concept of the current state of the workflow and an “event” that is input to the state machine (e.g. a system input or user input). The “event” causes the state machine to automatically transition to the next state. The next state is predetermined by a mapping between “events” and “next states”, given a current state. This mapping is often stored in a lookup table called a “state transition table”. The work of an individual workflow step is accomplished during the state transition.
  • State machines can be nested. A technique which adds to their power to deal with highly complex interactions, while having very maintainable code.
  • A state machine is typically encapsulated within an object. There are several designs for state machines, such as the Gang of Four State Pattern and the State Table approach. Robert Martin has good examples of both in his book Agile Principles, Patterns, and Practices in C# published in 2006. There is a lot written about state machine designs so you can find out about them in other sources as well.
  • State machines are also extremely useful for sequencing non-workflow things as well, e.g. UI interactions like a wizard, complex interactions between objects, hardware designs, hardware/software interactions, etc.

Messaging Patterns – The excellent book Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions by Gregor Hohpe and Bobby Woolf, published in 2003, contains several messaging patterns that can be applied to workflows to increase their decoupling and maintainability.

Saga – This pattern is similar to the Routing Slip, but subtly different. Note that this is not the same as the functionality called a “Saga” in several messaging frameworks. That “Saga” is really a Process Manager implementation! The true Saga pattern is useful when you need to programmatically compensate for errors in a long running sequence of independent atomic operations without the ability to use distributed transactions to wrap the sequence and do automatic rollbacks when encountering an error.

Scheduler-Agent-Supervisor – Somewhat similar to the Saga, the Scheduler-Agent-Supervisor pattern includes decoupled ways to check for errors and do compensation. It is expressed in terms of Windows Azure cloud services in this link http://msdn.microsoft.com/en-us/library/dn589780.aspx.

  • It is worthwhile to read about this pattern, as well as the Saga, just to see how one might approach programmatic error compensation in a workflow when distributed transactions are not available. Also useful along these lines is the Compensating Transaction Pattern at http://msdn.microsoft.com/en-us/library/dn589804.aspx. The ideas demonstrated in these 2 patterns can be applied in technologies other than Azure.

Caveat Emptor – Why You Need Good Project Design (aka Planning) When Using Workflows

After considering some of the above patterns, one thing to be acutely aware of with workflows in general is that the complexity of the code implementing workflows can exponentially increase very quickly! Why?

  • Workflows involve asynchronous calls.
  • Workflows can seldom use distributed transactions to wrap the execution of a sequence of workflow steps. Sorry, no transaction rollbacks are available that make coding easy. Therefore manual and/or programmatic compensation must be built into the workflow design and implementation. And manual and/or programmatic error handling and compensation often results in more use cases to do these things. Often it is necessary to write additional workflows dedicated to doing error compensation for other workflows, including options for manual input and assistance in compensation. All this will surely result in lots more code to design, implement, and test.
  • Workflows cannot be tested nearly as easily as can normal, synchronous sequences of calls. The async nature of their sequence will make testing much, much more involved, as will testing the error conditions since it involves manual and/or programmatic compensation. Essentially, one must “speed up time” in order to efficiently test long running processes.
  • Workflow code needs to be rock solid since it can be very, very time consuming to debug. Therefore, a complete analysis of potential failure points and scenarios needs to be done, and the design and implementation of the workflow and its parts need to “fail safe”. This can greatly expand the amount of code required. However some of it can be pushed down into reusable infrastructure.
  • The requirements and definition of workflows typically involves a number of people in different roles – Customers, users, product owners, analysts, architects, developers, quality assurance, etc. Not only is it time consuming to get things nailed down, but the number of possible permutations of a workflow can be vast, even for a fairly simple system.

So, if you are going to use the great power of workflows in your apps you need to design your project plan carefully to include all the work it takes to build solid workflows — Ramp up the learning curve, design, implement, and test the workflows.   This time may be surprisingly large, considering the above points.

I hope you find this article’s references as useful as I have.

George Stevens
Creative Commons License

dotnetsilverlightprism blog by George Stevens is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License. Based on a work at dotnetsilverlightprism.wordpress.com.

Advertisements
Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: