The Problem
One of the major shortcomings of FSIS CTS is that there is no functionality to load configuration values from a file to use within the flows. Things like server addresses, content directories, email address, etc… have to be hardcoded into each of the CTS operators. If you are like me and have become accustomed to loading configuration values from an app.config file or something similar, the lack of such functionality in CTS can become quite frustrating. One of the problems I ran into with the CTS flows I had developed is that when I had to deploy my flows from my development environment to my test environment, I had to edit things like content directory paths in each of my flow files before the flows could be deployed to the test environment.
I had developed several CTS flows which used a File System Reader to load XML documents from a directory into a collection in ESP. This in itself is not a complicated task to do in CTS. The File System Reader operator has a Location property which specifies the path to the directory where the XML files are located. The File System Reader reads the XML files from the “Location” directory and the content is eventually sent to ESP via an ESP Writer operator.
Below is an example of how my flows looked in the CTS designer.
This is what the properties of the FileSystemReader operator looked like. Notice the hardcoded value of C:content for Location.
Here is what the properties of the ESPWriter operator looked like. Notice the hardcoded value of fsis.dev.local for Host.
The flow worked perfectly in my development environment. However, a big problem arose when I had to deploy and execute the flows in the test environment. The issue is that the XML files to feed into FAST resided in a different path in my test environment (D:content) than it did in my development environment (C:content). In addition, the server address of the ESP Content Distributor in my test environment (fsis.test.local) is different than the server address I had in my development environment (fsis.dev.local). The flows would certainly deploy in FSIS but I would get runtime errors when the flows executed because the “Location” path of C:content didn’t exist in the test environment. The only way the flows were going to work in my test environment is if I opened each of the .flow files in a text editor and changed the startLocations property (which corresponds to the “Location” in the CTS designer for the File System Reader operator) of the File System Reader operator to point to the correct path (D:content) in test environment. I also needed to change the contentDistributors property of the ESP Writer operator to point to the ESP Content Distributor in the test environment.
As you can imagine, this became a tedious and error prone process. The more flows I created, the larger the problem became. There were occasions when I forgot to make the manual edit in the .flow files for some of the flows or put in the wrong path or server address. This screamed for a config file to be used by all the flows to determine the path to load the content from. Also the server address of the ESP Content Distributor could be loaded from the config file based on the environment where the flows were being executed. However, I couldn’t find any documentation on how to use a config file. The FSIS documentation described how Flow Properties could be used to pass common configuration values from one flow to another but it still wasn’t clear to me how this could be done globally for all flows.
My First Attempt to Solve the Problem
Initially I thought that I could put all the configuration values in a config file and then use a File System Reader operator to read the config file as the first operation in the flow. Then later on, the flow would perform the actual reading of the content XML files with another File System Reader to continue the flow to eventually execute the ESP Writer operator. This way, I would be able to parse the content path and ESP Content Distributor server address from the config file and then use those configuration values to set the Location and Host properties of the File System Reader and ESP Writer operators respectively.
I envisioned a flow that looked something like this.
Great idea but once again, CTS threw another problem at me. If you are going to use the File System Reader operator, it has to be the first operator in the flow. In other words, the File System Reader operator cannot receive the output of another operator as its input. Therefore, the second File System Reader operator cannot be set up to receive the output of the Config Mapper operator as its input. In fact, you cannot create an input connection to a File System Reader operator. Most, if not all the “Reader” operators have this limitation. If you are wondering how I managed to do that in the above flow, I simply used Paint to place an arrow between the Config Mapper and FileSystemReader after I took my screenshot. There is no way I would have been able to do this in the CTS designer.
The Solution
Since I couldn’t pass the config values read by the Config File System Reader to the other File System Reader in the flow, the solution was to use two flows in a sort of parent child set up. The general concept is that you will create two flows – one flow to read the config values from the configuration file and another flow to do the actual work. The “config reader” flow will act as the parent and will run the “actual” flow which we’ll consider the child flow by using a Flow Runner operator. The Flow Runner operator has a convenient feature which treats all inputs coming into the operator as Record Properties. When the Flow Runner operator executes the child flow, the Record Properties are passed to the child flow as Flow Properties. The child flow can then use the Flow Properties as values for a variety of things like the Location property of the File System Reader operator. You can specify which Record Properties you want to pass down to the child flow or you can leave the Record Property list empty in which case all the Record Properties are passed down to the child flow as Flow Properties.
Here is what the parent “config reader” flow looks like.
Here is what the Flow Runner operator’s properties look like. In this example, I’m only interested in passing the ContentPath and ContentDistributor Record Properties to the child flow as Flow Properties. If I didn’t specify any Record Properties, all Record Properties would be passed to the child flow as Flow Properties.
Here is what the child “do the actual work” flow looks like.
An important fact to keep in mind is how the Flow Properties are to be referenced in the operators inside the child flow. The correct way to reference a Flow Property is by using the following syntax: ${<Flow Property>}. Example: ${ContentPath}. You can also use GetFlowProperty function in any place where expressions are allowed like in the Mapper operator. Example: GetFlowProperty(“ContentPath”).
Below are examples of Flow Properties being used in the File System Reader operator (ContentPath) and ESP Writer operator (ContentDistributor) of the child flow.
Here is an example of a Flow Property (ContentPath) being used in an expression in a Mapper operator.
I want to mention a strange error message that shows up in the CTS designer in Visual Studio when you use a Flow Property for the Location property of a File System Reader. For some reason, you will get the following error in Visual Studio when you validate a flow with File System Reader using a Flow Property for the Location property: Invalid entry in ‘startLocations’ property: ‘${ContentPath}’ does not exist or has restricted permissions. See picture below. It’s safe to ignore this error. I believe this a bug in the current version (version 1.0) of the CTS designer. This error has no impact to the flow when it’s running in FSIS. The File System Reader will read from the location specified by the Flow Property in the Location property.
Summary
CTS doesn’t have built-in functionality to utilize a config file for loading common configuration value for use in a CTS flow. By using one flow to read a config file and then passing the configuration values to another flow as Flow Properties, you can accomplish the same functionality you get from an app.config file in C# or VB.NET projects. This allows you to create CTS flows without hardcoded values. Using this methodology helps you create flows which can be deployed in other environments without having to edit each of the flow files before deployment of the flows to those environments. All you need is a config file that is specific for each environment and the flows will work without modification. However, this is not a perfect solution because the “config reader” flow does need to have the path to the config file hardcoded into its File System Reader operator. I find this to be a minor annoyance. You can get around this problem by ensuring that each environment uses the same path for the location of the config file. I use C:fsisflowscts in all the various environments so that I don’t have to change the path for the config file in the parent “config reader” flow’s File System Reader operator.
A zip file with examples of the parent and child flows along with a config file is available for download here.
If you have any questions regarding this blog post, please feel free to email me at at rem@pointbridge.com. I welcome feedback on this content and also greatly appreciate suggestions for grammatical and/or spelling errors.
