Utilizing Pipelines for Long Processes

At GeekHive we aim to do things as efficiently and effectively as possible. If you develop in Sitecore long enough, eventually you will be tasked with some type of long running process that may be multi-stepped.  For example, read an RSS feed, sanitize the data, import that data into Sitecore, and publish the changes.  The details of an operation like this are up to the requirements and the developer performing the work, but any long running process such as the example given can abide by some simple concepts that will make the work cleaner and easier to manage in the future.

In the same way that Sitecore uses pipelines throughout the page request lifecycle, e.g. preprocessRequest, httpRequestBegin, httpRequestProcessed, etc. it’s possible to add custom pipelines that make sense for your operation.  For this blog post, we’ll continue with the example from the introductory paragraph.

Below is an easy to read and easy to follow configuration file for a process such as this:

<?xml version="1.0"?>

<configuration xmlns:patch="">


    <!-- Not covering in this blog post -->


      <setting name="My.Setting" value="a value"></setting>


    <!-- Not covering in this blog post -->


      <command name="proc:import" type="Namespace.Class, Assembly"/>




        <processor mode="on" type="Namespace.ConsumeRss, Assembly" method="Execute" />



        <processor mode="on" type="Namespace.SanitizeDataStep1, Assembly" method="Execute" />



        <processor mode="on" type="Namespace.ImportData, Assembly" method="Execute" />



        <processor mode="on" type="Namespace.PublishData, Assembly" method="Execute" />





By skimming this configuration file, I can see that this process has access to a setting, is triggered by a command called proc:import and more importantly the import process is broken down into the following pipelines: consumers, sanitizeData, importData and publishData. Note: pipeline names must be unique.

By separating the import process into various pipelines we can add/remove additional processors within each pipeline.  For example, notice how the sanitizeData processor is called SanitizeDataStep1- it is very possible given this example that the data may need to be further sanitized at a later date.

Before we kick off a new pipeline, we first must establish a PipelineArgs class that will be passed to the pipeline.  The pipeline arguments are the same throughout the pipeline.  If one an earlier processor alters their values, the following processor to execute will see these changed values.  This is what makes it possible to sanitize the data once, and then sanitize the data a second time (possibly looking for a different value that isn’t in the first processor).

public class MyPipelineArgs : PipelineArgs


        public Item SourceItem{ getset; }

        public string RssFeedLocation { getset; }

        public SyndicationFeed RssFeed { getset; }


You don’t get much by simply inheriting the PipelineArgs base class other than generic Data or Messages, which is why it only makes sense to strongly - type your own properties for consumption within your pipeline. 

Calling the pipeline:

var args = new MyPipelineArgs();

CorePipeline.Run("import.consumeRss", args);

There is no return value for a pipeline, however since the pipeline is synchronous, you can read the pipeline arguments after the pipeline finishes execution.

That’s all the magic that is involved.  Obviously each processor will run the method defined, and the method parameter must match the pipeline argument type that is sent to the pipeline.

public void Execute(MyPipelineArgs args)


            // run code here


This example is very primitive compared to what is possible.  For example, if we follow the guidelines of Sitecore when it comes to creating pipelines, it often has a StartingPipeline, a ProcessingPipeline and an EndingPipeline.  This approach allows you to insert custom code at each step of the process.  Using the example this can be translated to preConsumeRss, consumingRss and rssConsumed.

Following this approach results in more readable code and follows the modular approach that Sitecore is built upon.  Future developers will not only be impressed, but will thank you for doing things, the “right way”.