Skip to content

Fast Development Time for Analytics and Data Transforms — Azure Stream Analytics

April 24, 2016

I’ve been able to get quite fast development time using Azure Stream Analytics (ASA) to analyze streams of unstructured data, plus to transform the format of such data, i.e. breaking up the data into different streams and/or reconstituting the data into different structures and streams.  These are things we often need to do, and now we do not have to always write programs to do it.  In some cases we can use ASA instead.

The learning curve is quite manageable for ASA.  I found the longest part of the learning curve was working with ASA’s SQL like query language, particularly learning how to use its ability to do real time analysis of data streams via the Tumbling, Hopping, and Sliding time windows it offers.  But if you know the basics of SQL this only takes an hour or so to learn, with good examples at hand (in the links below).  I hope the links to ASA info sources will shorten your learning curve as much as they shortened mine, plus open your eyes to the possibilities ASA offers — It is a powerful, yet easy to use tool.

Here is a basic introductory example showing the process of building an ASA job and its query in the Azure Portal – “Get started with Azure Stream Analytics to process data from IoT devices“ by Jeff Stokes of Microsoft.  The screen shots of the Azure Portal for ASA in this link will give you an understanding of how to work with ASA and its query language.  Note that you need not write external code to get things working.  All your work, including writing and debugging the query, is done in the Azure Portal UI.  Note that you may need to write some C# code later for production monitoring of the ASA job and any Event Hubs it gets data from.

At the time of writing this blog article, ASA can input and output data from the following Azure services:

ASA Input Sources
Blob
Event Hub
IoT Hub
Reference Data in a Blob

ASA Output Destinations

SLQ database
Blob
Event Hub
Table Storage
Document DB
Service Bus Queue or Topic
PowerBI

These inputs and outputs provide an amazing array of options for processing data at rest (residing in a Blob) or data in motion (streaming into an Event Hub or IoT hub).

Here are 2 common usage scenarios of ASA:

  • Searching for patterns in log files or data streams
    • This can include using ASA to analyze log files that are programmatically created by ones software to look for errors and warnings of certain kinds, or for telltale evidence of security problems. “SQL Server intrusion detection using Azure Event Hub, Azure Stream Analytics and PowerBI” by Francesco Cogno of Microsoft is an example of such a usage scenario.
    • Since ASA works on live data streams contained in Azure Event Hubs it can be used to search for patterns in telemetry data from the outside world, e.g. IoT systems. For example one could find each item in the input stream that had “Alert” in the field named “EventType” and place that record into a Service Bus Queue read by a Worker Role whose job it was to push alert messages to a UI.
  • Calculating real time statistics on-the-fly
    • An example is calculating moving averages, standard deviations, and being able to create alert records sent to an Alerts queue when such a calculation exceeds some preset level. “Using Stream Analytics with Event Hubs” by Kirk Evans of Microsoft presents an example of this usage scenario as does the first link, above.

Other Useful Info Sources

How to debug your ASA job, step by step” by Venkat Chilakala of Microsoft. This can save lots of time when debugging.

Query examples for common Stream Analytics usage patterns” by Jeff Stokes of Microsoft. For both simple and complex query techniques by example.

Scale Azure Stream Analytics jobs to increase stream data processing throughput” by Jeff Stokes of Microsoft. This will give you in depth knowledge of ASA.

Stream Analytics & Power BI: A real-time analytics dashboard for streaming data” by Jeff Stokes of Microsoft. How to quickly display charts from data output by ASA.

Azure Stream Analytics Forum” on MSDN. I have found this forum to contain some really useful posts. Plus you can ask questions as well.

I hope you find these info sources as useful as I did in opening up a new world of cloud-based data analysis and transformation!

George Stevens

Creative Commons License

dotnetsilverlightprism blog by George Stevens is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License. Based on a work at dotnetsilverlightprism.wordpress.com.

Advertisements
Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: