Content Selector Configuration Editor

“Nothing in the world can take the place of Persistence. Talent will not; nothing is more common than unsuccessful men with talent. Genius will not; unrewarded genius is almost a proverb. Education will not; the world is full of educated derelicts. Persistence and Determination alone are omnipotent.”
Calvin Coolidge

Some time ago, I built a little Service Portal widget designed to allow a User to select various sets of records to be display in the Data Table widget on a portal page. I called this widget the Configurable Data Table Widget Content Selector because I designed it to be driven by an external JSON configuration object that could be specified as a widget option. Of course, I ended up just hard-coding the first one during my development and had to go back in later and fix it so that you could actually do that, but now that all works — you just have to build a new JSON configuration object if you want to use the widget on a different page for another purpose. That JSON object is a little complicated, though, so it occurred to me that it would be even better if there was some kind of input screen with some validation that would help you put one of those together. I did that not too long ago for the sn-record-picker, which seems to have worked out fairly well, so I was imagining something very similar, but maybe a little more complex.

There are three main sections to the Content Selector: 1) Perspectives, 2) States, and 3) Tables. My configuration wizard, then, would need a simple section where you could set up your Perspectives, another simple section where you could set up your States, and then a more complicated section where you could set up the Tables for every State of every Perspective. That third section sounds a little overwhelming at first glance, but like most complicated issues, if we break it down into its component parts and focus on one thing at time, we should be able to work through it.

The sn-record-picker Helper was just another portal widget, so that seemed like a decent approach to this endeavor as well. I called it the Content Selector Configurator, and placed it alone on a page of the same name for testing. To begin the process, we need to select an existing applicable Script Include, or enter a name if you want to create a brand new one. We can use an sn-record-picker for selecting an existing one, or to make things even easier, an snh-form-field of type reference, which is just a wrapper around the sn-record-picker that includes all of the labels and validation stuff. To limit the selection to just those Script Includes that are relevant to this process, we can filter on the field that contains the actual script looking for the code that extends the base class. The full snh-form-field element looks like this:

<snh-form-field
  snh-label="Content Selector Configuration"
  snh-model="c.data.script"
  snh-name="script"
  snh-type="reference"
  snh-help="Select the Content Selector Configuration that you would like to edit."
  snh-change="scriptSelected();"
  snh-required="true"
  placeholder="Choose a Content Selector Configuration"
  table="'sys_script_include'"
  default-query="'active=true^scriptCONTAINSObject.extendsObject(ContentSelectorConfig'"
  display-field="'name'"
  search-fields="'name'"
  value-field="'api_name'"/>

… and renders out like this:

Script Include picker

When the selection is made, the snh-change attribute will invoke the scriptSelected() client-side function, so we will need to code that out as well to handle the choice that was made. All of the work necessary to handle the selection will actually occur on the server side, so the client side function just has to kick things over there.

$scope.scriptSelected = function() {
	c.server.update();
};

Over on the server side, things are a little more complicated. We need to use the name of the script to get an instance of the script, and for that, we can use our old friend, the Instantiator. Once we have an instance of the script, we can call the getConfig() function to get a copy of the current configuration object. but before we do that, we have to make sure that we have an input object and we don’t already have a config object. All together, the code looks like this:

if (input) {
	if (!data.config && input.script && input.script.value) {
		data.scriptInclude = input.script.value;
		if (data.scriptInclude.startsWith('global.')) {
			data.scriptInclude = data.scriptInclude.split('.')[1];
		}
		var instantiator = new Instantiator();
		instantiator.setRoot(this);
		var configScript = instantiator.getInstance(data.scriptInclude);
		data.config = configScript.getConfig($sp);
	}
}

Now that we have all of that out of the way, we can work on the actual wizard itself. I wrapped all of the HTML related to selecting a config script in one DIV and then made another, currently empty DIV for the wizard. I used complementary ng-show attributes to hide the wizard until the script was selected, and then hide the selection components once the choice was made. The whole thing now looks like this:

<snh-panel title="'${Content Selector Configuration Editor}'" class="panel-primary">
  <form id="form1" name="form1" ng-submit="save();" novalidate>
    <div class="row" ng-show="!c.data.script.value">
      <div class="col-sm-12">
        <snh-form-field
          snh-label="Content Selector Configuration"
          snh-model="c.data.script"
          snh-name="script"
          snh-type="reference"
          snh-help="Select the Content Selector Configuration that you would like to edit."
          snh-change="scriptSelected();"
          snh-required="true"
          placeholder="Choose a Content Selector Configuration"
          table="'sys_script_include'"
          default-query="'active=true^scriptCONTAINSObject.extendsObject(ContentSelectorConfig'"
          display-field="'name'"
          search-fields="'name'"
          value-field="'api_name'"/>
      </div>
    </div>
    <div class="row" ng-show="c.data.script.value">
      (the wizard lives here)
    </div>
  </form>
</snh-panel>

I still have to add in the ability to create a new script from scratch, but I think I will deal with that later as I am anxious to jump into the wizard itself. I’ll circle back and toss that in before we wrap things up, but right now I just want to get on to fun stuff. That’s an entirely different process, though, so this seems like a good place to stop for now. We’ll jump straight into the wizard next time out.

Fun with Webhooks, Improved

“I think it’s very important to have a feedback loop, where you’re constantly thinking about what you’ve done and how you could be doing it better.”
Elon Musk

So far, I have had relatively good luck playing around with my Simple Webhooks app, and have been able to post content to other systems such as Slack and MS Teams in addition to the test cases that I sent over to webhook.site. One thing that I did notice, though, was that my portal page for editing the details of a Webhook was missing a couple of items found on the corresponding form in the main UI. On the form for a Webhook in the main UI, I built a UI Action that you can use to send a test POST to your URL, and the form also includes a Delete button that you can use to get rid of your Webhook when you no longer need it or want it. The current version of the portal page has neither of those features, so I decided that it was time to add those in.

The first order of business, then, was to add the two buttons to the HTML, right after the existing Save button:

&nbsp;
<button ng-show="c.data.sysId" ng-click="testURL()" class="btn btn-default ng-binding ng-scope" role="button" title="Click here to send a test POST to this URL">Test URL</button>
&nbsp;
<button ng-show="c.data.sysId" ng-click="deleteThis()" class="btn btn-default ng-binding ng-scope" role="button" title="Click here to permanently delete this webhook">Delete</button>

I didn’t want them showing up on new records, since there is no point in deleting a record that you haven’t created yet, so I added an ng-show attribute based on the presence of an existing sys_id. Other than that, it’s just a basic copy and paste of the other button code with some minor modifications. Here’s how it looks rendered out:

New buttons added to the form HTML

The new buttons reference new client-side functions, so next we will need to add those to the existing client-side script. Here are the two that I came up with:

$scope.testURL = function() {
	spModal.confirm('Send a test POST to this URL?').then(function(confirmed) {
		if (confirmed) {
			c.data.action = 'test';
			c.server.update();
		}
	});
};

$scope.deleteThis = function() {
	if (c.data.sysId) {
		spModal.confirm('Permanetly delete this Webhook?<br/>(This cannot be undone.)').then(function(confirmed) {
			if (confirmed) {
				c.data.action = 'delete';
				c.server.update().then(function(response) {
					goBack();
				});
			}
		});
	} else {
		goBack();
	}
};

I ended up putting a Confirm pop-up on both of them, even though technically the URL test is not destructive. I just thought that it might be nice to confirm that you really want send something over to another system before you actually did it. I also added the c.data.action variable so that once we were over on the server side, that code would know what to do. In our previous version, the only call to the server side was that Save button, so there was no question what needed to be done. But now that we have multiple possible actions, everyone — including Save — will need to register their intentions by setting this variable to some known value (save, test, or delete) before invoking c.server.update(). All of the actual work to perform the save, test, and delete actions is done on the server side, so let’s pop over there next.

To begin, I pulled out all of the existing Save logic and put it into a function of its own. Then I added the following conditional step, assuming that I would have similar functions for the other two actions:

	if (input) {
		if (input.action == 'save') {
			save(input);
		} if (input.action == 'test') {
			test(input);
		} if (input.action == 'delete') {
			deleteThis(input);
		}
	} else {
		. . . 

The Delete function turned out to be pretty basic stuff:

function deleteThis(input) {
	whrGR.get(input.sysId);
	whrGR.deleteRecord();
	gs.addInfoMessage('Your Webhook data has been deleted.');
}

Most of the code for the Test URL button I just stole from the existing UI Action built for the same purpose. Much of that is buried in the Script Include anyway, so that turned out to be fairly simple as well:

function test(input) {
	whrGR.get(input.sysId);
	var wru = new WebhookRegistryUtils();
	var result = wru.testURL(whrGR);
	if (result.status == '200') {
		gs.addInfoMessage('URL "' + input.url + '" was tested successfully.');
	} else {
		gs.addErrorMessage('URL test failed: ' + JSON.stringify(result, null, '<br/> '));
	}
}

That’s about all there was to that. Technically, you cannot really call this an enhancement since it is functionality that should have been in there from the start. Let’s just call it a much needed improvement. Here’s the new Update Set.

Automatically Link Referenced Tasks, Improved

“The secret of change is to focus all of your energy not on fighting the old, but on building the new”
Socrates

After running my LinkReferencedTasks Business Rule for a while, it has become apparent that there was a flaw in my approach. Whenever someone separates different task numbers with a special character, neither one gets picked up in the process. For example, if you enter something like REQ0010005/RITM0010009, the process that converts all special characters to an empty string ends up converting that to REQ0010005RITM0010009, which starts with REQ, but is not a valid Task number. A little Corrective Maintenance should solve that problem. Let’s convert this:

text = text.replace(/\n/g, ' ');
text = text.replace(/[^A-Z0-9 ]/g, '');

… to this:

text = text.replace(/[^A-Z0-9 ]/g, ' ');

Originally, I had converted all line feeds to spaces and then all characters that were not letters, numbers, or spaces to an empty string. Once I decided to change all characters that were not letters, numbers, or spaces to a space, I no longer needed the line above, as line feeds fall into that same category. That solved that problem.

However, while I was in there, I decided to do a little Perfective Maintenance as well. By changing all of the special characters to single spaces, I thought that I might end up with several spaces in a row, which would result in one or more empty strings in my array of words. To screen those out, I thought about discarding words with a length of zero, but then it occurred to me that short words of any kind will not be task numbers, so I settled on an arbitrary minimum length of 6 instead. Now the code that unduplicates the list of words looks like this:

var unduplicated = {};
for (var i=0; i<words.length; i++) {
	var thisWord = words[i];
	if (thisWord.length > 5) {
		unduplicated[thisWord] = thisWord;
	}
}

This should reduce the clutter quite a bit and minimize the number of words that need to be checked to see if they start with one of the specified prefixes.

And finally, I added an Enhancement, which I had actually thought about earlier, but didn’t implement. This enhancement allows you to specify the relationship type instead of just accepting the hard-coded Investigates relationship that I had put in the original. For backward compatibility, I did not want to make that mandatory, so I kept that as a default for those implementations that did not want to specify their own. The new version of the Script Include now looks like this:

var TaskReferenceUtils = Class.create();
TaskReferenceUtils.prototype = {
    initialize: function() {
    },

	linkReferencedTasks: function(taskGR, prefix, relationship) {
		if (!relationship) {
			relationship = 'd80dc65b0a25810200fe91a7c64e9cac';
		}
		var text = taskGR.short_description + ' ' + taskGR.description;
		text = text.toUpperCase();
		text = text.replace(/[^A-Z0-9 ]/g, ' ');
		var words = text.split(' ');
		var unduplicated = {};
		for (var i=0; i<words.length; i++) {
			var thisWord = words[i];
			if (thisWord.length > 5) {
				unduplicated[thisWord] = thisWord;
			}
		}
		for (var word in unduplicated) {
			for (var x in prefix) {
				if (word.startsWith(prefix[x])) {
					this._findTask(word, taskGR.getUniqueValue(), relationship);
				}
			}
		}
	},

	_findTask: function(number, child) {
		var taskGR = new GlideRecord('task');
		if (taskGR.get('number', number)) {
			this._documentRelationship(taskGR.getUniqueValue(), child, relationship);
		}
	},

	_documentRelationship: function(parent, child, relationship) {
		var relGR = new GlideRecord('task_rel_task');
		relGR.addQuery('parent', parent);
		relGR.addQuery('child', child);
		relGR.query();
		if (!relGR.next()) {
			relGR.initialize();
			relGR.parent = parent;
			relGR.child = child;
			relGR.type = relationship;
			relGR.insert();
		}
	},

    type: 'TaskReferenceUtils'
};

So, a little fix here, a little improvement there, and a brand new feature over there, and suddenly we have a new version better than the last. Stuff’s getting better. Stuff’s getting better every day. Here’s the improved Update Set.

Automatically Link Referenced Tasks

“It is necessity and not pleasure that compels us.”
Dante Alighieri

Occasionally, we will get an Incident Ticket that references another task present in the system. This is usually a status request on a Service Catalog Request or a problem with a recent Change. It would be nice to be able to just click on those recognizable task numbers, but both the Short Description and the Description are plain text fields where that is not an option. There is, however, a handy out-of-the-box UI Formatter that you can include on your Incident Form that provides the capability to link other tasks to your Incident. The name of the Formatter is Task Relations, and I like to drag it onto the Incident Form in the Forms Designer right underneath the Formatter for Contextual Search Results. Once you have that Formatter present on your Incident Form, you can click on the green plus sign and link other tasks of various types to your Incident.

That’s a really cool feature that allows you click on the related tasks to open them up, but being the lazy developer that I am, I would prefer not to have to go through all of the manual work to set up all of the links to the things mentioned in the text of the ticket. In my ideal world, the system would be smart enough to read the text, recognize a task number, and then build the link for me so that I don’t have to do all of that work by hand, and also to make sure that I did not miss anything. How hard could that be?

My thought was that I could create a Business Rule linked to the Incident table that would examine the two primary text fields (Short Description and Description), look for anything that appeared to be a task number, search the Task table to see if it really was a task number, and if so, build the link for me. It seemed like a relatively simple thing to do, so I went to work.

It felt as if there might be a lot of code involved, so instead of putting all of that in the Business Rule itself, I decided to build a Script Include that I could call from the Business Rule. I thought that if I could make the function generic enough, I might be able to reuse it for other Business Rules linked to different Task tables. Plus, putting all of the code in the Script Include keeps the Business Rule a lot cleaner. Here are the things that I thought that I needed to do in order to get this all to work:

  • Combine the two text fields on the Incident into a single string variable for examination,
  • Convert the text to upper case,
  • Convert all line feeds to spaces,
  • Remove all characters that are not letters, numbers, or spaces,
  • Split the string by spaces creating an array of words,
  • Unduplicate the array of words so that we only looked at each unique word once,
  • Examine each word to see if it started with a known task number prefix,
  • Read the Task table for every word that started with a known task number prefix, and
  • Build a relationship record for every Task record that was found on the Task table.

All we need to do now is turn that into code.

Combine the two text fields on the Incident into a single string variable for examination:

var text = taskGR.short_description + ' ' + taskGR.description;

Convert the text to upper case:

text = text.toUpperCase();

Convert all line feeds to spaces:

text = text.replace(/\n/g, ' ');

Remove all characters that are not letters, numbers, or spaces:

text = text.replace(/[^A-Z0-9 ]/g, '');

Split the string by spaces creating an array of words:

var words = text.split(' ');

Unduplicate the array of words so that we only looked at each unique word once:

var unduplicated = {};
for (var i=0; i<words.length; i++) {
	var thisWord = words[i];
	unduplicated[thisWord] = thisWord;
}

Examine each word to see if it started with a known task number prefix:

for (var word in unduplicated) {
	for (var x in prefix) {
		if (word.startsWith(prefix[x])) {
			this._findTask(word, taskGR.getUniqueValue());
		}
	}
}

Read the Task table for every word that started with a known task number prefix:

_findTask: function(number, child) {
	var taskGR = new GlideRecord('task');
	if (taskGR.get('number', number)) {
		this._documentRelationship(taskGR.getUniqueValue(), child);
	}
},

Build a relationship record for every Task record that was found on the Task table:

_documentRelationship: function(parent, child) {
	var relGR = new GlideRecord('task_rel_task');
	relGR.addQuery('parent', parent);
	relGR.addQuery('child', child);
	relGR.query();
	if (!relGR.next()) {
		relGR.initialize();
		relGR.parent = parent;
		relGR.child = child;
		relGR.type = 'd80dc65b0a25810200fe91a7c64e9cac';
		relGR.insert();
	}
},

Before I create a relationship record, I want to make sure that there isn’t already a relationship record out there, so I do a quick query just to check before I commit to inserting a new one. The two records don’t need to be linked more than once. I also hard-coded the relationship type, which works for my current purpose, but if I ever want to expand this out to other use cases, I may eventually want to pass that in as an argument as I did with the task number prefixes. Here is the whole thing, all put together:

var TaskReferenceUtils = Class.create();
TaskReferenceUtils.prototype = {
    initialize: function() {
    },

	linkReferencedTasks: function(taskGR, prefix) {
		var text = taskGR.short_description + ' ' + taskGR.description;
		text = text.toUpperCase();
		text = text.replace(/\n/g, ' ');
		text = text.replace(/[^A-Z0-9 ]/g, '');
		var words = text.split(' ');
		var unduplicated = {};
		for (var i=0; i<words.length; i++) {
			var thisWord = words[i];
			unduplicated[thisWord] = thisWord;
		}
		for (var word in unduplicated) {
			for (var x in prefix) {
				if (word.startsWith(prefix[x])) {
					this._findTask(word, taskGR.getUniqueValue());
				}
			}
		}
	},

	_findTask: function(number, child) {
		var taskGR = new GlideRecord('task');
		if (taskGR.get('number', number)) {
			this._documentRelationship(taskGR.getUniqueValue(), child);
		}
	},

	_documentRelationship: function(parent, child) {
		var relGR = new GlideRecord('task_rel_task');
		relGR.addQuery('parent', parent);
		relGR.addQuery('child', child);
		relGR.query();
		if (!relGR.next()) {
			relGR.initialize();
			relGR.parent = parent;
			relGR.child = child;
			relGR.type = 'd80dc65b0a25810200fe91a7c64e9cac';
			relGR.insert();
		}
	},

    type: 'TaskReferenceUtils'
};

Now that we our Script Include, we need to build the Business Rule that calls it. For my purpose, I added a Business Rule to the Incident table and called it LinkReferencedTasks. I checked the Advanced checkbox and made it active async on Insert whenever there was text in either of the two description fields. I could have also triggered it on Update as well, but in my experience, the Incident description is usually captured when the Incident is created an rarely updated after that.

LinkReferencedTasks Business Rule

Under the Advanced tab, I entered the following script:

(function executeRule(current, previous) {
	new TaskReferenceUtils().linkReferencedTasks(current, ['REQ','RITM','SCTASK','CHG']);
})(current, previous);

In addition to the current GlideRecord, you also need to pass in a string array of task table prefixes which will be used somewhat like a filter to only link tasks that start with those values. If you want a different mix of task types, you can just update that list. That’s it. We are done. Well, maybe we had better test it out first, but the building part is done anyway. Testing should be simple enough: we just need to find an existing task that we want to reference and then include that task number in the text of a new incident. Let’s do that now.

New test Incident with references to other tasks in the Short Description field

Now all we have to do is hit that Submit button and see if those tasks referenced in the text are automatically linked to the Incident. Seems as if a little drum-roll would be appropriate here …

Test results for the new Business Rule and Script Include

Well, nothing ever goes right the very first time! It looks like we managed to create a link to the RITM, but not to the Request. Fortunately, it turns out that this is a tester error and not a developer error. When I typed in the Request number, I missed a zero. The actual Request number is REQ0010005, not REQ001005. Of course, my explanation for that is that I did that on purpose to demonstrate that it will only link real requests, and if your request number is not a real request, then it won’t bother to attempt to create a link to it. That’s it — I did it on purpose — it was all part of the plan. You know that you have to test for failure as well as success — I just did it all in one test because I am so efficient. Sometimes I even amaze myself!

Anyway, it all seems to work, so for those of you who like to play along at home, here’s an Update Set with all of the parts.

Update: There is a better (corrected) version here.

Fun with Webhooks, Part VIII

“If you are working on something that you really care about, you don’t have to be pushed. The vision pulls you.”
Steve Jobs

When we last parted, there was an open question on the table regarding the focus of this next installment. Implementing Basic Authentication and moving on to the My Webhooks Service Portal widget were two of the options, but no decision was made at the time as to which way we wanted to turn next. Now that we have finally made it here, the good news is that we don’t have to choose. Adding support for Basic Authentication turned out to be so simple that it looks like we are going to have time for both. Check it out:

if (whrGR.getValue('authentication') == 'basic') {
	request.setBasicAuth(whrGR.getValue('user_name'), whrGR.getValue('password'));
}

That’s it. I just added those few lines of code to the postWebhook function of our Script Include and BAM! — now we support Basic Authentication. Pretty sweet!

So now we can devote the remainder of the post to our new My Webhooks widget. As I think I mentioned earlier, this should be a fairly easy clone of the existing My Delegates widget, so the first thing that I did was to make a copy of that guy to have something with which to start. Then I hacked up the HTML to adapt it to our current application.

<table class="table table-hover table-condensed">
  <thead>
    <tr>
      <th style="text-align: center;">ID</th>
      <th style="text-align: center;">Table</th>
      <th style="text-align: center;">Type</th>
      <th style="text-align: center;">Reference</th>
      <th style="text-align: center;">Active</th>
      <th style="text-align: center;">Remove</th>
    </tr>
  </thead>
  <tbody>
    <tr ng-repeat="item in c.data.listItems track by item.id | orderBy: 'id'" ng-hide="item.removed">
      <td data-th="ID"><a href="?id=webhook_registry&sys_id={{item.sys_id}}">{{item.id}}</a></td>
      <td data-th="Table">{{item.table}}</td>
      <td data-th="Type">{{item.type}}</td>
      <td data-th="Reference">{{item.reference}}</td>
      <td data-th="Active" style="text-align: center;"><img src="/images/check32.gif" style="width: 16px; height: 16px;" ng-show="item.active"/></td>
      <td data-th="Remove" style="text-align: center;"><img src="/images/delete_row.gif" ng-click="deleteWebhook($index)" alt="Click here to permanently delete this Webhook" title="Click here to permanently delete this Webhook" style="cursor: pointer;"/></td>
    </tr>
  </tbody>
</table>

With that out of the way, we now need to replace the code that gathers up the Delegates with code that will gather up the Webhooks. Seems simple enough …

function fetchList() {
	var list = [];
	var whrGR = new GlideRecord('x_11556_simple_web_webhook_registry');
	whrGR.addQuery('owner', data.userID);
	whrGR.orderBy('number');
	whrGR.query();
	while (whrGR.next()) {
		var thisWebhook = {};
		thisWebhook.sys_id = whrGR.getValue('sys_id');
		thisWebhook.id = whrGR.getDisplayValue('number');
		thisWebhook.table = whrGR.getDisplayValue('table');
		thisWebhook.type = whrGR.getDisplayValue('type');
		thisWebhook.active = whrGR.getValue('active');
		if (thisWebhook.type == 'Single Item') {
			thisWebhook.reference = whrGR.getDisplayValue('document_id');
		} else if (thisWebhook.type == 'Assignment Group') {
			thisWebhook.reference = whrGR.getDisplayValue('group');
		} else {
			thisWebhook.reference = whrGR.getDisplayValue('person');
		}
		list.push(thisWebhook);
	}
	return list;
}

There is still a lot of work to do, but I like to try things every so often before I get too far along, just to make sure that things are going OK, so let’s do that now. This widget could easily go on the existing User Profile page just like the My Delegates widget, but it could also go on a page of its own. Since we are just trying things on for size right now, let’s just create a simple Portal Page and put nothing on it but our new widget. Let’s call it my_webhooks, drop our widget right in the middle of it, and go check it out on the Service Portal.

Initial My Webhooks widget

Well, that’s not too bad. We don’t really need the Save and Cancel buttons in this instance, but we do need a way to create a new Webhook, so maybe we can replace those with a single Add Webhook button. The links don’t go anywhere just yet and the delete icons don’t do anything, but as far as the layout goes, it looks pretty good. I think it’s a good start so far. Let’s swap out the buttons and then we can wrap things up with the client-side code.

The left-over code in the button area from the My Delegates widget looks like this:

<div style="width: 100%; padding: 5px 50px; text-align: center;">
  <button ng-click="saveDelegates()" class="btn btn-primary ng-binding ng-scope" role="button" title="Click here to save your changes">Save</button>
   
  <button ng-click="returnToProfile()" class="btn ng-binding ng-scope" role="button" title="Click here to cancel your changes">Cancel</button>
</div>

Let’s replace it with this:

<div style="width: 100%; padding: 5px 50px; text-align: center;">
  <button ng-click="newWebhook()" class="btn btn-primary ng-binding ng-scope" role="button" title="Click here to create a new Webhook">Add New</button>
</div>

Now we just need to overhaul the code on the client side to handle both the adding and deleting of our webhooks. The code to support the Add New button should be pretty straightforward; we just need to link to the same (currently nonexistent!) page that we link to from the main table, just without any sys_id to indicate that this is a new record request. This should handle that nicely:

$scope.newWebhook = function() {
	$location.search('id=webhook_registry');
};

As for the delete operation, we will have to bounce over to the server side to handle that one, but first we should confirm that was really the intent of the operator with a little confirmation pop-up. We could use the stock Javascript confirm feature here, but I like the look of the spModal version better, so let’s go with that.

$scope.deleteWebhook = function(i) {
	spModal.confirm('Delete Webhook ' + c.data.listItems[i].id + '?').then(function(confirmed) {
		if (confirmed) {
			c.data.toBeRemoved = i;
			$scope.server.update().then(function(response) {
				reloadPage();
			});
		}
	});
};

We’ll need some server side code to handle the actual deletion of the record, but that should be simple enough.

if (input) {
	data.listItems = input.listItems;
	if (input.toBeRemoved) {
		deleteWebhook(input.toBeRemoved);
	}
}

function deleteWebhook(i) {
	var whrGR = new GlideRecord('x_11556_simple_web_webhook_registry');
	if (whrGR.get(data.listItems[i].sys_id)) {
		whrGR.deleteRecord();
	}
}

We still need to test everything, but before we do that, we should go ahead and build the webhook_registry page that we have been pointing at so that we can fully test those links as well. That sounds like a good project for our next installment, so I think we will wrap things up right here for now and then start off next time with our new page followed by some end to end testing.

Fun with Webhooks, Part VII

“The question isn’t who is going to let me; it’s who is going to stop me.”
Ayn Rand

Now that we have proven that the essential elements of our Subflow all work as intended, it’s time finish out the remainder of the flow’s activities, which include logging the HTTP POST and Response details as well as reporting any undesired results to Event Management. Let’s start with logging the activity.

The first thing that we will need in order to record our Webhook POSTs is a table in which to store the data. As we did with our original Webhook Registry table, we can navigate to System Definition -> Tables and click on the New button to bring up the Table definition screen.

New Webhook Log table

And once again we will give it a Label and let the system generate the associated Name. We will also want to uncheck the Create module checkbox again to prevent the generation of a number of artifacts for which we have no use. Once the table has been defined, we can start adding fields, and the first field that we will want to add is a Reference to the Webhook Registry table. Every log record will be linked to the registry for which the activity was POSTed, so we will want to establish that relationship with a Reference field that we can label Registry.

The other Reference field that we will want is a link back to the original Incident that is the subject of the POST. Since we set things up in a way that would allow us to support tables other than Incident, we will want to do this with a Document ID field rather than a direct reference to the Incident table. This time, when we configure the Dependent Field, we can dot walk through the registry reference to get to the table name column in that related table. This will save us from having to have a column on the log table for the name of the table that holds the record associated with the Document ID, and it will ensure that all of the Document IDs related to each Registry will only come from the table associated with that Registry.

Selecting the Registry record’s Table column as the Dependent Field

The rest of the fields in the log table are just what we sent over, and what we received in response:

  • Payload – The data that we will be POSTing
  • URL – The URL to which we will be POSTing our Payload
  • Status – The HTTP Response Code returned by the target server
  • Body – The body of the message returned by the target server
  • Error – The error flag
  • Error Code – The error code
  • Error Message – The error message
  • Parse Error – Any error that occurred while parsing the body of the response

After defining all of the fields on the table, I brought up the table’s form and arranged all of the fields on the screen in a manner that I thought was most appropriate.

Webhook Log form layout

Those of you who are paying close attention will also have noticed that I added the JSON View Dictionary Attribute to both the Payload and Body fields, just to make reading the JSON content a little easier.

Now that we have a table, we can start putting records into it. We will do this in our Subflow, right after we POST the payload. This is just a simple, out-of-the-box Create Record action that we can configure using data pills from various other steps.

Logging the Webhook POST and Response

In addition to capturing everything related to each POST, the other thing that we wanted to do was to capture any issues that might come up during this process. We are already aware of two possible issues, one being a bad response and the other being a good response, but with an unparsable response body. After we do the POST and log the result, we can throw in a few more conditionals to pick those up, and then add a Log Event Action to the flow for each.

Complete Subflow with Event logging for error conditions

Whenever we log an Event, we will want to capture as much information as we can about what went wrong. In this case, however, we have already logged everything about the transaction to the Webhook Log table, so really all we will need to provide is a way to find that record. Putting the sys_id in the additional_info field should do the trick. Here is how I populated all of the data for the Event:

Event Log data

That should complete the Subflow, at least for now. We may end up adding some other features in the future, but for now, this accomplishes everything that we set out to do. We still need to do a lot more testing to verify that all of these various branches in the tree work out as we are hoping, but the building part should be done now, at least for this portion.

As far as the remaining development goes, we still have to build out the My Webhooks portal widget and we also need to go back into the Script Include and add support for Basic Authentication. We also need to add code, and possibly additional fields in the registry record, for any other authentication protocols that we would like to support. So once again we find ourselves at a crossroads: we can either jump into the Service Portal world and start working on our widget, or we can turn our attentions to authentication and finish things up in that area. There is no need to make any decision on that today, though. We’ll figure all of that out when we meet again.

Fun with Webhooks, Part V

“Change is easy. Improvement is far more difficult.”
Ferdinand Porsche

Now that we have a Business Rule to launch our Subflow, it’s time to get busy and actually create the Subflow. To create a new Subflow, open up the Flow Designer, click on the New button and select Subflow from drop-down menu. I initially called mine Webhook Poster, but the Business Rule couldn’t find it to launch it for some reason. I finally got it to work when I replaced the space with an underscore, so now the name is Webhook_Poster. I’m not really sure why I had to do that, but it works, so we will leave it at that.

With the Subflow named, the next thing to do is to define all of the inputs. In my mind, there were only two, the current object and the previous object. Unfortunately, the Flow Designer will not recognize any of the properties of those objects unless you explicitly define them, so I had to specify every element of both objects. That seemed like a rather tedious waste of time, but eventually I got through it and now it’s done.

Subflow inputs defined

That takes care of the set-up, so now it’s on to the flow itself. The first thing that we are going to want to do is to gather up all of the registration records that would be triggered by this Incident. Since we gave our users several options for selecting records, we will have to test them all. Essentially, we will need a query filter for each possible Type, which we can then OR together to create one huge query filter. Here is how that looks in practice:

Webhook Registry selection criteria

As a second step, I threw in an IF statement to see if the query returned any records. We are basically done at this point if there are no records, but if we do have records to process, then we will need to build the payload that we will be posting to all of the target URLs. For that job, I needed to create a new Flow Designer Action, so I saved my Subflow and closed the Subflow tab for now.

The payload is basically a generic JSON object containing the relevant information about the the things that have changed about the record since the last time that it was saved. Pulling that all together will basically be a lot of tedious scripting, so that sounds like a job for yet another function in our Script Include. Here is what I came up with:

buildPayload: function(current, previous) {
	var payload = {};

	payload.id = current.number;
	payload.short_description = current.short_description;
	payload.message = '';
	var separator = '';
	if (current.state != previous.state) {
		payload.state = current.state;
		if (previous.state) {
			payload.message += separator + 'State changed from ' + previous.state + ' to ' + current.state + ' on ' + payload.id + '.';
		} else {
			payload.message += separator + 'State set to ' + current.state + ' on ' + payload.id + '.';
		}
		separator = '\n\n';
	}
	if (current.assignment_group != previous.assignment_group) {
		payload.assignment_group = current.assignment_group;
		if (previous.assignment_group) {
			payload.message += separator + 'Assignment Group changed from ' + previous.assignment_group + ' to ' + current.assignment_group + ' on ' + payload.id + '.';
		} else {
			payload.message += separator + 'Assignment Group set to ' + current.assignment_group + ' on ' + payload.id + '.';
		}
		separator = '\n\n';
	}
	if (current.assigned_to != previous.assigned_to) {
		payload.assigned_to = current.assigned_to;
		if (previous.assigned_to) {
			payload.message += separator + 'Assigned To changed from ' + previous.assigned_to + ' to ' + current.assigned_to + ' on ' + payload.id + '.';
		} else {
			payload.message += separator + 'Assigned To set to ' + current.assigned_to + ' on ' + payload.id + '.';
		}
		separator = '\n\n';
	}
	if (current.comments) {
		payload.comments = current.comments;
		payload.message += separator + current.operator + ' has added a new comment to ' + payload.id + ':\n' + current.comments;
		separator = '\n\n';
	}
	if (current.work_notes) {
		payload.work_notes = current.work_notes;
		payload.message += separator + current.operator + ' has added a new work note to ' + payload.id + ':\n' + current.work_notes;
		separator = '\n\n';
	}

	return JSON.stringify(payload, null, '\t');
},

We still need a Flow Designer Action to call this function, but the function does almost all of the heavy lifting and our Action should now be quite simple to put together. Let’s open up the Flow Designer again, click on the New button, and select Action from the drop-down selection list. We’ll call our new Action Build Webhook Payload since that’s what it does, and we will define two inputs, the current and previous objects. Since we won’t be referencing any of the properties of those objects directly, this time we won’t have to invest the time in specifying all of the elements of the objects.

All we will need for our Action is to add a Script step with the same two inputs and the payload as an output. The script itself is just a call to the function that we just added to our Script Include.

(function execute(inputs, outputs) {
	var wru = new WebhookRegistryUtils();
	outputs.payload = wru.buildPayload(inputs.current, inputs.previous);
})(inputs, outputs);

We also need to define the payload as an output of Action itself, and map the Action inputs to the Script step inputs and the Script step output to the Action output. That should complete our new Action.

Build Webhook Payload Action

Now we can go back into our Subflow and select this Action as a third step under our second step conditional that checks to see if there are any records from the query in the first step. For our fourth step, we will add a flow logic step that loops through all of the records from step 1, and inside of that loop, our fifth step will make the POST of the payload. I looked for an out-of-the-box simple HTTP POST Action already included on the platform, but I couldn’t find anything, so that’s another Action that we are going to have to produce ourselves. We already built much of the script for that when we built our testURL function; if we rework that code just a little bit, we can probably find a way to make it work just fine for both purposes.

testURL: function(whrGR) {
	var jsonObj = {message: 'This is a test posting.'};
	return this.postWebhook(whrGR, JSON.stringify(jsonObj, null, 4));
},

postWebhook: function(whrGR, payload) {
	var result = {};

	var request = new sn_ws.RESTMessageV2();
	request.setEndpoint(whrGR.getValue('url'));
	request.setHttpMethod('POST');
	request.setRequestHeader('Content-Type', 'application/json');
	request.setRequestHeader('Accept', 'application/json');
	request.setRequestBody(payload);
	var response = request.execute();
	result.status = response.getStatusCode();
	result.body = response.getBody();
	if (result.body) {
		try {
			result.obj = JSON.parse(result.body);
		} catch (e) {
			result.parse_error = e.toString();
		}
	}
	result.error = response.haveError();
	result.error_code = response.getErrorCode();
	result.error_message = response.getErrorMessage();

	return result;
},

Now the testURL function is just a call to our new postWebhook function that contains most of the code previously contained in the testURL function. The testURL function will pass in our simple test payload, and when we create out new Action, we can call the very same postWebhook function, passing in a real payload. The script for our new Action will then simply be this:

(function execute(inputs, outputs) {
	var wru = new WebhookRegistryUtils();
	var result = wru.postWebhook(inputs.webhook_registry, inputs.payload);
	for (var key in result) {
		outputs[key] = result[key];
	}
})(inputs, outputs);

Stopping right here should give us enough with which to test out the process so far. This isn’t all that we want to do here, because we still want to record both the attempt and the response in our log table. However, since we haven enough built out to just test the POSTing of payloads, I think we should stop and do that. Afterwards, we can circle back and build out the activity logging and any Event logging that we might want to do for any kind of bad responses. But right now, we have enough built out that we can create a few Webhook Registrations and then go update some qualifying Incidents to see what actually happens.

Setting all of that up and verifying the results seems worthy of its own episode, so let’s wrap things up for today and pick that up next time out.

Event Management for ServiceNow, Revisited

“True prevention is not waiting for bad things to happen, it’s preventing things from happening in the first place.”
Don McPherson

Some time ago we built some utility functions to support reporting Events within the ServiceNow Platform. That was before the Flow Designer, though, so that effort did not include any support for that environment. We already have the script to do all of the heavy lifting from our earlier work, so it wouldn’t take much to create a Flow Designer Action that called that script to report an Event that occurred during Flow processing. We can call our new Action Log Event, and set up Action Inputs for all of the usual suspects.

Log Event Action Inputs

For our script step, we will basically set up the same inputs and then source them directly from the primary Action Inputs.

Script step inputs mapped to Action Inputs

Those of you who are paying attention will notice that we defined the additional_input field as a String even though it needs to be an Object when we make the call to our existing script. The assumption here is that the caller will provide a valid JSON String, and then we can turn it into an Object in our script before we make the call. Here is the script to convert the String and then make the call.

(function execute(inputs, outputs) {
	if (inputs.additional_info) {
		try {
			inputs.additional_info = JSON.parse(inputs.additional_info);
		} catch(e) {
			//
		}
	}
	var seu = new ServerEventUtil();
	seu.logEvent(inputs.source, inputs.resource, inputs.metric_name, inputs.severity, inputs.description, inputs.additional_info);
})(inputs, outputs);

There are no outputs from this process, so this is the entire Action. Once we Save and Publish it, it will be available from the Action selection list, and then we can add Log Event steps anywhere in our Flows and Subflows where we want to report an Event. That was fairly quick, easy, and relatively painless. For those of you would like to try it out on your own, here is an Update Set.

Fun with Webhooks, Part IV

“Try not to become a man of success, but rather try to become a man of value.”
Albert Einstein

Last time, we came to a fork in the road, not knowing whether it would be better to jump into the My Webhooks Service Portal widget, or to start working out the details of the process that actually POSTs the Webhooks. At this point, I am thinking that the My Webhooks widget will end up being a fairly simple clone of my old My Delegates widget, so that may not be all that interesting. POSTing the Webhooks, on the other hand, does sound like it might be rather challenging, so let’s start there.

When I first considered attempting this, my main question was whether it would be better to handle this operation in a Business Rule or by building something out in the Flow Designer. After doing a little experimenting with each, I later came to realize that the best alternative involved a little bit of both. In a Business Rule, you have access to both the current and previous values of every field in the record; that information is not available in the Flow Designer. In fact, it’s not available in a Business Rule, either, if you attempt to run it async. But if you don’t run it async, then you are holding everything up on the browser while you wait for everything to get posted. That’s not very good, either. What I ultimately decided to do was to start out with a Business Rule running before the update, gather up all of the needed current and previous values, and then pass them to a Subflow, which runs async in the background.

My first attempt was to just pass the current and previous objects straight into my Subflow, but that failed miserably. Apparently, when you pass a GlideRecord into a Subflow, you are just passing a reference, not the entire record, and then when the Subflow starts up, it uses that reference to fetch the data from the database. That’s not the data that I wanted, though. I want the data as it was before the database was updated. I had to take a different route with that part of it, but the basic rule remained the same.

Business Rule to POST webhooks

The rule is attached to the Incident table, runs on both Insert and Update, and is triggered whenever there is a change to any of the following fields: State, Assignment Group, Assigned to, Work notes, or Additional comments.

To get around my GlideRecord issue, I wound up creating two objects, one for current and one for previous, which I populated with data that I pulled out of the original GlideRecords. When I passed those two objects to my Subflow, everything was there so that I could do what I wanted to do. Populating the objects turned out to be quite a bit of code, so rather than put that all in my Business Rule, I created a function in my Script Include called buildSubflowInputs and I passed it current and previous as arguments. That simplified the Business Rule script quite a bit.

(function executeRule(current, previous) {
	var wru = new WebhookRegistryUtils();
	sn_fd.FlowAPI.startSubflow('Webhook_Poster', wru.buildSubflowInputs(current, previous));
})(current, previous);

In the Script Include, things got a lot more complicated. Since I was going to be turning both the current and previous GlideRecords into objects, I thought it would be helpful to create a function that did that, which I could call once for each. That function would pull out the values for the Sys ID, Number, State, Caller, Assignment group, and Assigned to fields and return an object populated with those values.

getIncidentValues: function(incidentGR) {
	var values = {};

	values.sys_id = incidentGR.getValue('sys_id');
	values.number = incidentGR.getDisplayValue('number');
	if (!incidentGR.short_description.nil()) {
		values.short_description = incidentGR.getDisplayValue('short_description');
	}
	if (!incidentGR.state.nil()) {
		values.state = incidentGR.getDisplayValue('state');
	}
	if (!incidentGR.caller_id.nil()) {
		values.caller = incidentGR.getDisplayValue('caller_id');
		values.caller_id = incidentGR.getValue('caller_id');
	}
	if (!incidentGR.assignment_group.nil()) {
		values.assignment_group = incidentGR.getDisplayValue('assignment_group');
		values.assignment_group_id = incidentGR.getValue('assignment_group');
	}
	if (!incidentGR.assigned_to.nil()) {
		values.assigned_to = incidentGR.getDisplayValue('assigned_to');
		values.assigned_to_id = incidentGR.getValue('assigned_to');
	}

	return values;
},

Since my Business Rule could be fired by either an Update or an Insert, I had to allow for the possibility that there was no previous GlideRecord. I could arbitrarily call the above function for current, but I needed to check to make sure that there actually was a previous before making that call. I also wanted to add some additional data to the current object, including the name of the person making the changes. The field sys_updated_by contains the username of that person, so to get the actual name I had to use that in a query of the sys_user table to access that data.

buildSubflowInputs: function(currentGR, previousGR) {
	var inputs = {};

	inputs.current = this.getIncidentValues(currentGR);
	if (currentGR.isNewRecord()) {
		inputs.previous = {};
	} else {
		inputs.previous = this.getIncidentValues(previousGR);
	}
	inputs.current.operator = currentGR.getDisplayValue('sys_updated_by');
	var userGR = new GlideRecord('sys_user');
	if (userGR.get('user_name', inputs.current.operator)) {
		inputs.current.operator = userGR.getDisplayValue('name');
	}

	return inputs;
},

One other thing that I wanted to track was any new comments or work_notes, and that turned out to be the most the most complicated code of all. If you try the normal getValue or getDisplayValue methods on any of these Journal Fields, you end up with all of the comments ever made. I just wanted the last one. I had to do a little searching around to find it, but there is a function called getJournalEntry that you can use on these fields, and if you pass it a 1 as an argument, it will return the most recent entry. However, it is not just the entry; it is also a collection of metadata about the entry, which I did not want either. To get rid of that, I had to find the first line feed and then pick off all of the text that followed that. Putting all of that together, you end up with the following code:

if (!currentGR.comments.nil()) {
	var c = currentGR.comments.getJournalEntry(1);
	inputs.current.comments = c.substring(c.indexOf('\n') + 1);
}
if (!currentGR.work_notes.nil()) {
	var w = currentGR.work_notes.getJournalEntry(1);
	inputs.current.work_notes = w.substring(w.indexOf('\n') + 1);
}

That’s pretty much all that there is to the new Business Rule and the needed additions to the Script Include. Now we need to start tackling the Subflow that will actually take these current and previous objects and do something with them. That’s a bit much to jump into at this point, so we’ll wrap things up here for now and save all of that for our next time out.

Fun with Webhooks, Part III

“All things are difficult before they are easy.”
Thomas Fuller

Now that we have built the Webhook Registry table and configured the associated form layout, the last thing we need to do before we move on is to set up the Test URL UI Action and the underlying code to make it work. Clicking on the Test URL button should pass the current registry record to a function in a new Script Include that will pull the URL out of the registry, POST an example JSON object to the URL, and verify the response. Since we have to reference the Script Include in the UI Action, let’s work on that first.

To create a new Script Include, navigate to System Definition -> Script Includes and then click on the New button. Since this will be a general purpose Script Include for our application, I named it WebhookRegistryUtils.

New WebhookRegistryUtils Script Include

To test a URL, we will want to add a new function to our new Script Include that accepts a Webhook Registry GlideRecord as an argument and returns an Object that contains the results of the test POST. Let’s call this function testURL().

testURL: function(whrGR) {
	var result = {};

	var jsonObj = {message: 'This is a test posting.'};
	var request = new sn_ws.RESTMessageV2();
	request.setEndpoint(whrGR.getValue('url'));
	request.setHttpMethod('POST');
	request.setRequestHeader('Content-Type', 'application/json');
	request.setRequestHeader('Accept', 'application/json');
	request.setRequestBody(JSON.stringify(jsonObj, null, 4));
	var response = request.execute();
	result.status = response.getStatusCode();
	result.body = new global.JSON().decode(response.getBody());
	result.error = response.haveError();
	result.errorCode = response.getErrorCode();
	result.errorMessage = response.getErrorMessage();

	return result;
},

This code is fairly self-explanatory. The jsonObj is our test payload and the request is an empty sn_ws.RESTMessageV2 that we populate with the URL found in the registry record and the hard-coded Method of POST. The response is the result of executing the request, and then we pull all of the relevant information out of the response and use it to populate the result object, which we then return to the caller. We may end up reworking some of this when we start adding additional functionality, but this should be enough to test a URL with no authentication.

At this point, we could test out the code by writing a quick little Fix Script to grab a registry record and pass it to this function, but it might be just as easy to ahead and just make the UI Action and test it out on the form. To create the UI Action, pull up the Webhook Registry form, select Configure -> UI Actions from the hamburger menu, and then click on the New button. I named mine Test URL, checked the Form button checkbox, and put the following script in the Condition:

current.url > ''

That condition keeps the Test URL button from appearing on the form when there is no URL to test.

Test URL UI Action

For the Script, I just call the testURL function in our new Script Include and then notify the user of the results. There is not much to the code, but here it is:

var wru = new WebhookRegistryUtils();
var result = wru.testURL(current);
action.setRedirectURL(current);
if (result.status == '200') {
	gs.addInfoMessage('URL "' + current.url + '" was tested successfully.');
} else {
	gs.addErrorMessage('URL test failed: ' + JSON.stringify(result, null, '<br/> '));
}

There is undoubtedly a much nicer way to format that failure message, but this will get the job done for now. We can always clean that up later. Right now, though, let’s push the button and see what happens! On second thought, though, we can’t really do that until we insert a record into the registry. let’s take care of that now.

New Webhook Registry entry

Once all of the parts are in place, this entry should POST a payload to the specified URL every time there is a change to an Incident assigned to the owner. For the URL, I went out to check out Webhook.site, and was assigned that end point URL, so I threw that in there. That should be a valid webhook URL, so our first test should be successful. Once we prove that, we can mangle up the URL and hit it again to see what a failure looks like. But first things first — let’s see if we can make it work with a real URL.

Successful URL test

So far, so good! Real quick, before we try out the bad one, let’s pop over to Webhook.site and see if there is any evidence that we ran this test. Oh, look — there is!

Test POST results found on Webhook.site

That’s actually a pretty cool little tool. That’s going to be quite handy for this particular adventure. I like it. Now, let’s try out one that we know will fail.

Unsuccessful URL test

Well, that seems to work as well. So now we have our first table and related form, the beginnings of our Script Include and a handy UI Action. From here, we can work on the My Webhooks Portal Widget, or we can start building out the process that sends out the POSTs to the qualifying destinations. That’s nothing that we need to decide today, though; we can figure that all out next time.