Fun with Outbound REST Events, Part IV

“Life is trying things to see if they work.”
Ray Bradbury

We got a little side-tracked last time, but we are finally ready to tackle the Script Include that will utilize our new Outbound REST Message. I like to tackle things a little bit at a time and test, so we will start out with just the code to invoke the service and format a response to the caller, and then deal with all of the error handling later on in the process. We will also want to call this from a Client Script, but again, let’s not worry about that part just yet and focus on successfully launching the service and interpreting the results.

To start with, we pull up the list of Script Includes and hit the New button to create a new script. We will call our new script AddressValidationUtils and create a function called validateAddress that takes street, city, state, and zip as arguments. The first thing that we will want to do in our new function will be to establish the object that will be returned to the caller.

var response = {result: 'failure', street: street, city: city, state: state, zip: zip};

Our result property will have three possible values: valid, invalid, or failure. We initially default it to failure, mainly because there will be a lot places where we might have to set it to failure, and then alter it to one of the other two values when appropriate. The remaining properties are defaulted to their original value, and if the address is deemed valid, will be updated with the values returned by the service to clean up any inconsistencies. Now let’s add the code to invoke the service.

var rest = new RESTMessage('US Street Address API', 'get');
rest.setStringParameter('authid', gs.getProperty('us.address.service.auth.id'));
rest.setStringParameter('authToken', gs.getProperty('us.address.service.auth.token'));
rest.setStringParameter('street', encodeURIComponent(street));
rest.setStringParameter('city', encodeURIComponent(city));
rest.setStringParameter('state', encodeURIComponent(state));
rest.setStringParameter('zip', encodeURIComponent(zip));
var resp = rest.execute();

This is all pretty straightforward: we establish the rest object by creating a new RESTMessage using the name of the message and the name of the method. Once the object has been established, we use the setStringParameter method to set the value for each of the required variables. Once again, the system does not encode any of the values, so we need to do that for any user input. We don’t need to do that for any of the credentials, as we already know the value of those and we know that there are no characters in either of those that would cause an issue with the URL. Once all of the variable value have been set, we then make the call to the service using the execute method.

Now we need to add code to interpret the response.

var body = resp.getBody();
if (resp.getStatusCode() == 200) {
	var respArray = [];
	try {
		respArray = JSON.parse(body);
	} catch (e) {
		gs.info('Exception: ' + e);
	}
	if (respArray && respArray.length > 0) {
		respObj = respArray[0];
		if (typeof respObj.analysis == 'object') {
			var validity = respObj.analysis.dpv_match_code;
			if (validity == 'Y' || validity == 'S' || validity == 'D') {
				response.result = 'valid';
				response.street = respObj.delivery_line_1;
				response.city = respObj.components.city_name;
				response.state = respObj.components.state_abbreviation;
				response.zip = respObj.components.zipcode;
				if (respObj.components.plus4_code) {
					response.zip += '-' + respObj.components.plus4_code;
				}
			} else {
				response.result = 'invalid';
			}
		} else {
			gs.info('Invalid Response Object: ' + JSON.stringify(respObj));
		}
	} else {
		gs.info('Invalid Response Body: ' + body);
	}
} else {
	gs.info('Invalid HTTP Response Code: ' + resp.getStatusCode());
}

There is a little more code here, but most of it is defensive, just to make sure that we don’t choke on something unexpected. All of the gs.info statements are just placeholders at this point; eventually we will replace those with Event logging, but I’m not ready to deal with that just yet, so we’ll put that off for now and just write to the system log. Assuming all goes well, we will have the response object in hand, and then all we have to do is check to see if it validated or not. If it did, we pass back their version of four address components, and if it didn’t, then we just indicate that is invalid.

Once get through all of that, all that is left is to return the response to the caller. Here’s the whole thing, all put together.

var AddressValidationUtils = Class.create();
AddressValidationUtils.prototype = {

	initialize: function() {
	},

	validateAddress: function(street, city, state, zip) {
		var response = {result: 'failure', street: street, city: city, state: state, zip: zip};

		var rest = new RESTMessage('US Street Address API', 'get');
		rest.setStringParameter('authid', gs.getProperty('us.address.service.auth.id'));
		rest.setStringParameter('authToken', gs.getProperty('us.address.service.auth.token'));
		rest.setStringParameter('street', encodeURIComponent(street));
		rest.setStringParameter('city', encodeURIComponent(city));
		rest.setStringParameter('state', encodeURIComponent(state));
		rest.setStringParameter('zip', encodeURIComponent(zip));
		var resp = rest.execute();
		var body = resp.getBody();
		if (resp.getStatusCode() == 200) {
			var respArray = [];
			try {
				respArray = JSON.parse(body);
			} catch (e) {
				gs.info('Exception: ' + e);
			}
			if (respArray && respArray.length > 0) {
				respObj = respArray[0];
				if (typeof respObj.analysis == 'object') {
					var validity = respObj.analysis.dpv_match_code;
					if (validity == 'Y' || validity == 'S' || validity == 'D') {
						response.result = 'valid';
						response.street = respObj.delivery_line_1;
						response.city = respObj.components.city_name;
						response.state = respObj.components.state_abbreviation;
						response.zip = respObj.components.zipcode;
						if (respObj.components.plus4_code) {
							response.zip += '-' + respObj.components.plus4_code;
						}
					} else {
						response.result = 'invalid';
					}
				} else {
					gs.info('Invalid Response Object: ' + JSON.stringify(respObj));
				}
			} else {
				gs.info('Invalid Response Body: ' + body);
			}
		} else {
			gs.info('Invalid HTTP Response Code: ' + resp.getStatusCode());
		}

		return response;
	},

	type: 'AddressValidationUtils'
};

Now all we need to do is test it out and make sure that it all works. Once again, we can use that same test case that was provided in their testing tool; we just have to pass the values as arguments to the function call. We can write a quick little stand-alone script for that that looks like this:

var avu = new AddressValidationUtils();
gs.info(JSON.stringify(avu.validateAddress('3901 SW 154th Ave', 'Davie', 'FL', '33331'), null, '\t'));

To actually run that, we can bring up the background script runner, paste it in, and click on the Run Script button.

Test script in the background script runner

… which gets us the following result:

{
	"result": "valid",
	"street": "3901 SW 154th Ave",
	"city": "Davie",
	"state": "FL",
	"zip": "33331-2613"
}

Not only did we validate the address, we also picked up the last four digits of the zip code, which was not a part of the original user input. And of course, we just proved that our new Script Include does, in fact, do what we intended for it to do. Chalk up another baby step along the path that we first laid out when we began this journey. So now what?

At this point, we actually have two different ways to go as far as our next step. Given that we have a working script to call when everything is working as it should, we could move on to the Client Script and see if we can now integrate this with the User Profile form, or we could turn our attention to those gs.info placeholders and start replacing those with calls to the function that logs Events. It all has to be done at some point, so it really doesn’t matter other than the fact that we have to choose. But, I guess we don’t have to choose right now. We can figure that out when it’s time to put together the next installment in the series.

Fun with Highcharts, Part V

“It’s always something.”
Roseanne Roseannadanna

I decided to enhance my workload chart by adding the ability click on a bar or point on the backlog line and bring up a list of all of the tasks represented by that data on the chart. Highcharts makes that super simple to do, so I went out and grabbed some example code off of the Interwebs and added this to my workload chart template:

plotOptions: {
        series: {
            cursor: 'pointer',
            point: {
                events: {
                    click: function () {
                        alert('Category: ' + this.category + ', value: ' + this.y);
                    }
                }
            }
        }
    },

That was enough to prove that everything worked, after which I was going to modify the click function to navigate to a Data Table page with the appropriate filter to list out all of the tasks represented by the data point selected. Unfortunately, when I fired it up to test it out, nothing happened when I clicked on a bar or a point along the line. The cursor did change to a pointer, so that part was definitely working, but no alerts popped up no matter where I clicked. I hate it when that happens!

It took me a little while to figure this out, but after a little bit of digging around I finally reached the heart of the issue: I built the chart object on the server side, and when it was transferred to the client side where it was to be used, it was converted to a JSON string somewhere along the lines in the background Ajax process, and that conversion removed the function (JSON only preserves data; any functions are lost in translation). While it was nice to finally understand the root of the problem, the implication was that my whole way of going about this was pretty much invalidated. I can’t build a chart object on the server side and then pass it to the client side and retain any functions that might be a part of the object. The chart object will need be generated on the client side where it will be used. That means a total redesign of the entire concept.

Well, I guess that guy Charles Lauller knew what he was talking about. Time to start over with a blank sheet of paper

Fun with Highcharts, Part II

“The only thing that endures over time is the Law of the Farm. You must prepare the ground, plant the seed, cultivate, and water it if you expect to reap the harvest.”
Stephen Covey

Now that I have my functioning generic chart widget, it’s time to start working on my chart object generator. My intent, as usual, is to start out small and then build up a library of various chart types over time. Conceptually, I want to create a utility function to which I can pass some chart data and a chart type, and have it return a fully configured chart object that I can pass to my generic chart widget which will then render the chart. In use, it would look something like this:

var chartObject = generator.getChartObject(chartData, chartType);

Theoretically, you could make the chartType parameter optional if you set up a default, and then you would only have to pass that in when you wanted something other than the the established standard. For my first chart, though, I think I will do something slightly more sophisticated, mainly because this is the kind of thing that I like to utilize when I’m looking at the distribution of work within and across teams. I call it a Workload Chart, and it’s just your typical three-value tracking of work coming in, work getting done, and work still left in the needs-to-get-done pile. It doesn’t matter what the work is, or who is doing it, or over what period you are tracking things — the concept is pretty universal and the chart is the chart, regardless of the underlying data. Here’s one that allows you to select the Team, the Type of work, the Frequency, and the Period from a series of drop-down selections:

Typical workload chart

The chart data is the stuff that will change: title, subtitle, data values, and the labels across the bottom. The chart object includes all of the chart data, plus all of those structural elements that are constant and independent of the data. The utility function will take in the chart data, add it to a predefined model, and then return the entire object back to the caller. Here is the basic structure of the Script Include, which I called the GenericChartUtil:

var GenericChartUtil = Class.create();
GenericChartUtil.prototype = {
	initialize: function() {
	},

	getChartObject: function(chartData, chartType) {
		var chartObject = {};

		if (chartType == 'workload') {
			chartObject = this._getWorkloadChart(chartData);
		} else if (chartType == 'bar') {
			chartObject = this._getBarChart(chartData);
		} else {
			chartObject = this._getPieChart(chartData);
		}

		return chartObject;
	},

	_getPieChart: function(chartData) {
		return {
//			type-specific chart object w/data
		};
	},

	_getBarChart: function(chartData) {
		return {
//			type-specific chart object w/data
		};
	},

	_getWorkloadChart: function(chartData) {
		return {
//			type-specific chart object w/data
		};
	},

	type: 'GenericChartUtil'
};

For the workload chart, the dynamic data includes the title, the subtitle, an array of values for the received bars , an array of values for the completed bars , an array of values for the remaining work trend line (backlog), and an array of values for the time period labels. For the chart displayed in the previous image, the chartData object might look something like this:

{
   "title":"Incidents assigned to Hardware",
   "subtitle":"Quarterly through 9/30/2019",
   "labels":[
      "12/31/2017",
      "3/31/2018",
      "6/30/2018",
      "9/30/2018",
      "12/31/2018",
      "3/31/2019",
      "6/30/2019",
      "9/30/2019"
   ],
   "received":[0, 0, 7, 2, 0, 0, 0, 1],
   "completed":[0, 0, 6, 1, 0, 0, 0, 0],
   "backlog":[0, 0, 1, 2, 2, 2, 2, 3]
}

How you would compile that information is a subject for another time; today we just want to focus on how our chart object generator would take that information in and use it to create a complete chart object to be passed to Highcharts. It’s actually quite simple to do, and can be accomplished with a simple return statement that sends back a Javascript object that is a combination of hard-coded values and data pulled from the incoming chart data object. Here is the completed function for the workload chart:

	_getWorkloadChart: function(chartData) {
		return {
			chart: {
				zoomType: "xy"
			},
			title: {
				text: chartData.title
			},
			subtitle: {
				text: chartData.subtitle
			},
			xAxis: [
				{
					categories: chartData.labels,
					crosshair: true
				}
			],
			yAxis: [
				{
					labels: {
						format: "{value}",
						style: {}
					},
					title: {
						text: "Recieved/Completed",
						style: {}
					}
				},{
					labels: {
						format: "{value}",
						style: {}
					},
					title: {
						text: "Backlog",
						style: {}
					},
					opposite: true
				}
			],
			tooltip: {
				shared: true
			},
			legend: {
				layout: "vertical",
				align: "left",
				x: 120,
				verticalAlign: "top",
				y: 100,
				floating: true,
				backgroundColor: "rgba(255,255,255,0.25)"
			},
			series: [
				{
					name: "Received",
					type: "column",
					data: chartData.received,
					tooltip: {
						valueSuffix: " received"
					}
				},{
					name: "Completed",
					type: "column",
					data: chartData.completed,
					tooltip: {
						valueSuffix: " completed"
					}
				},{
					name: "Backlog",
					type: "spline",
					yAxis: 1,
					data: chartData.backlog,
					tooltip: {
						valueSuffix: " remaining"
					}
				}
			]
		};
	},

For every property in the returned chart object, there is either a hard-coded constant value that is an element of the chart template/design, or there is data pulled from the incoming chart data object. This idea can be repeated for any type or design of chart desired, and you can make independent decisions on a chart by chart basis as to what is a standard, fixed value and what has to be provided in the chart data object. As you get more sophisticated, you can even set up default values for certain things that can be overridden by values present in the chart data. If there is a value in the chart data, you can use that; otherwise, you can fall back to the predetermined default value. But before we start making more chart types, we’ll need to figure out how to obtain all of the values for the chart data object to support our workload chart example. That sounds like a bit of work though, so I think we should just call that a great topic for a future installment

Fun with Highcharts

“Most of the important things in the world have been accomplished by people who have kept on trying when there seemed to be no hope at all.”
Dale Carnegie

Long before I had ever heard of ServiceNow, I stumbled across a cool little product called Highcharts. In their own words, “Highcharts is a charting library written in pure JavaScript, offering an easy way of adding interactive charts to your web site or web application.” ServiceNow comes bundled with Highcharts included, which I thought was pretty nifty when I figured that out, because you can do quite a bit with Highcharts with very minimal effort. In fact, that’s one of the things that I really like about both products: both allow you to accomplish quite a bit with just a very small investment of time and energy.

To make things even easier, it always helps to have a few ready-made parts at your disposal to speed things along. Highcharts uses simple Javascript objects to configure their charts, and the contents of those objects can be broken down into two rather distinct categories: 1) elements that control the type, look, and feel of the chart and 2) elements that contain the data to be presented in the chart. Once you design a particular chart with the style, colors, and appearance that you find acceptable, you can save that off and reuse it again and again with different data or data from different periods. For the ServiceNow Service Portal, my thought was to create a generic chart widget that could take any chart configuration object as a passed parameter, and then also create a generic chart configuration object generator that would take a chart type and chart data as parameters and return a chart configuration object that could be then passed to the generic chart widget.

To start things off, I decided to build a simple generic chart widget and pass it a hard-coded chart configuration object, just to prove that everything about the widget was functional. There are a lot of sample Highcharts chart objects floating around the Internet, but for my test I just went out to their Your First Chart tutorial page and grabbed the configuration object for that simple example.

Rendered Highchart as displayed on their tutorial page

To grab just the chart object, I copied the highlighted code from the page.

Chart configuration object (highlighted)

Unfortunately, this is not a valid JSON string, which I would need if I was going to pass this around as a parameter, but I could paste it into simple Javascript routine as the value of a variable, and print out a JSON.stringify of that variable and then I ended up with a usable JSON string.

{
   "chart":{
      "type":"bar"
   },
   "title":{
      "text":"Fruit Consumption"
   },
   "xAxis":{
      "categories":[
         "Apples",
         "Bananas",
         "Oranges"
      ]
   },
   "yAxis":{
      "title":{
         "text":"Fruit eaten"
      }
   },
   "series":[
      {
         "name":"Jane",
         "data":[1, 0, 4]
      },{
         "name":"John",
         "data":[5, 7, 3]
      }
   ]
}

With my new hard-coded chart specification object in hand, I set out to create my generic widget. Every Highchart needs a place to live, so I started out with the HTML portion and created a simple chart DIV inside of a basic panel.

<div class="panel panel-default">
  <div class="panel-body form-horizontal">
    <div class="col-sm-12">
      <div id="chart-container" style="height: 400px;"></div>
    </div>
  </div>
</div>

On the server side, I wanted to be able to accept a chart configuration object from either input or options, so I ended up with this:

(function() {
	if (input && input.chartObject) {
		data.chartObject = input.chartObject;
	} else {
		if (options && options.chart_object) {
			try {
				data.chartObject = JSON.parse(options.chart_object);
			} catch (e) {
				gs.addErrorMessage("Unable to parse chart specification: " + e);
			}
		}
	}
})();

On the client side, I also wanted to be able to accept a chart configuration object from a broadcast message from another widget, and I also needed to add the code to actually render the chart, so that turned out to be this:

function ($scope) {
	var c = this;
	if (c.data.chartObject) {
		var myChart = new Highcharts.chart('chart-container', c.data.chartObject);
	}
	if (c.options.listen_for) {
		$scope.$on(c.options.listen_for, function (event, config) {
			if (config.chartObject) {
				var myChart = new Highcharts.chart('chart-container', config.chartObject);
			}
		});
	}
}

One more thing to do was to set up the Option schema for my two widget options, which is just another JSON string.

[{"hint":"An optional JSON object containing the Chart specifications and data",
"name":"chart_object",
"default_value":"",
"section":"Behavior",
"label":"Chart Object",
"type":"string"},
{"hint":"The name of an Event that will provide a new Chart specification object",
"name":"listen_for",
"default_value":"",
"section":"Behavior",
"label":"Listen for",
"type":"string"}]

Finally, I had to go down to the bottom of the widget page where the related records tabs are found and Edit the dependencies to add Highcharts as a dependency. This will bring in the Highcharts code and make all of the magic happen.

That pretty much completes the widget, so now I just needed to create a page on which to put the widget so that I could test it out. Once I created the page, I was able to pull up the widget options using the pencil icon in the page designer and enter in my JSON object for the chart configuration.

Adding the JSON chart object as a widget option

At that point, all that was left was for me to click on the Try it button and look at my beautiful new chart. Unfortunately, all I got to see was a nice big blank chunk of whitespace where my chart should have appeared. Digging into the console error messages, I came across the following:

Highcharts not defined

Well, either my dependency did not load, or my widget couldn’t see the code. I tried $rootScope.Highcharts and $window.Highcharts and $scope.Highcharts, but nothing worked. Then I thought that maybe it was a timing issue and so I put in a setTimeout to wait a few seconds for everything to load, but that didn’t work either. So then I added a script tag to the HTML thinking that maybe the dependency wasn’t pulling in the script after all. Nothing.

At that point, I decided to give up just trying different things on my own and see if I could hunt down some other widget that might already be working with Highcharts and see how those were set up. From the Dependencies list, I used the hamburger menu next to the Dependency column label to select Configure > Table, and then on the Table page, I scrolled down to where I could click on Show list to bring up the list of dependencies. Then I filtered the list for Highcharts to find that there were quite a few existing widgets that pulled in Highcharts as a dependency.

The secret, it appeared, was some additional code in the Link section of the widget, a section that I have always ignored, mainly because I have never understood its purpose or how it worked.

function(scope, element, attrs, controllers) {
	scope.$watch('chartOptions', function() {
		if (scope.chartOptions) {
			$('#chart-container').highcharts(scope.chartOptions);
			scope.chart = $('#chart-container').highcharts();
		}
	});
}

Now this is all AngularJS mystical nonsense sprinkled with Highcharts fairy dust as far as I was concerned, because I have no idea what the Link section of a widget is for or what all of this code actually does. However, it is attached to widgets that actually work, so I got busy cutting and pasting. The other thing that I had to do based on the examples that I was examining was to modify my client-side code a little, which now looked like this:

function ($scope) {
	var c = this;
	if (c.data.chartObject) {
		$scope.chartOptions = c.data.chartObject;
	}
	if (c.options.listen_for) {
		$scope.$on(c.options.listen_for, function (event, config) {
			if (config.chartObject) {
				$scope.chartOptions = config.chartObject;
			}
		});
	}
}

Yes, we are now deep into the realm of Cargo Cult Programming, but I’ve never been too proud to copy other people’s working code, even if I didn’t understand it. Now let’s hit that Try it button one more time …

That’s better!

Well, look at that! I’m not sure how all of that works, but it does work, so I’m good. Now that I have my functioning generic chart widget, I can start working on my chart object generator and then see if I can’t wire the two of them together somehow …