Fun with Webhooks, Part IX

“You have brains in your head. You have feet in your shoes. You can steer yourself any direction you choose.”
Theodor Geisel

We still need to test the My Webhooks portal page that we built last time, but before we do that, I wanted to first build out the page referenced in a couple of links on that page so that we could test everything together. My initial thought for that page was to build out a brand new portal widget for that purpose using our old friends SNH Panel and SNH Form Fields. Before I did that, though, it occurred to me that it might be faster to just use the stock form portal page, passing in the name of the table, and potentially a sys_id for existing records. There were a number of things that I did not like about that idea, but I thought that I could overcome those with some UI Policies and some default values for a couple of table fields. I played around with that a bit and found another thing that I didn’t really like, which was that saving a record left you still on the form page and did not bring you back to the My Webhooks page, which I thought was rather annoying. It seemed as though I might be able to mitigate that by adding my Dynamic Service Portal Breadcrumbs to the top of each page, but then I ran into another problem that I could not work around related to the Document ID field. At that point, I gave up and went back to my original plan, which is where I should have started in the first place.

So, here is the HTML for my new Webhook Registry portal widget:

<snh-panel class="panel panel-primary" title="'${Webhook Registry}'">
  <form id="form1" name="form1" novalidate>
    <div class="row">
      <div class="col-xs-12 col-sm-6">
        <snh-form-field
          snh-model="c.data.number"
          snh-name="number"
          snh-label="Number"
          snh-type="text"
          readonly="readonly"/>
        <snh-form-field
          snh-model="c.data.type"
          snh-name="type"
          snh-label="Type"
          snh-type="select"
          snh-required="true"
          snh-choices='[{"label":"Single Item","value":"single"},{"label":"Caller / Requester","value":"requester"},{"label":"Assignment Group","value":"group"},{"label":"Assignee","value":"assignee"}]'/>
      </div>
      <div class="col-xs-12 col-sm-6">
        <snh-form-field
          snh-model="c.data.owner"
          snh-name="owner"
          snh-label="Owner"
          snh-type="text"
          readonly="readonly"/>
        <snh-form-field
          snh-model="c.data.active"
          snh-name="active"
          snh-label="Active"
          snh-type="checkbox"/>
      </div>
    </div>
    <div class="row">
      <div class="col-sm-12">
        <snh-form-field
          snh-model="c.data.url"
          snh-name="url"
          snh-label="URL"
          snh-type="url"
          snh-required="true"/>
      </div>
    </div>
    <div class="row">
      <div class="col-xs-12 col-sm-6">
        <snh-form-field
          snh-model="c.data.document_id"
          snh-name="document_id"
          snh-label="Item"
          snh-type="reference"
          snh-required="c.data.type=='single'"
          table="'incident'"
          display-field="'number'"
          search-fields="'number'"
          value-field="'sys_id'"
          ng-show="c.data.type=='single'"/>
        <snh-form-field
          snh-model="c.data.person"
          snh-name="person"
          snh-label="Person"
          snh-type="reference"
          snh-required="c.data.type=='assignee' || c.data.type=='requester'"
          table="'sys_user'"
          default-query="'active=true'"
          display-field="'name'"
          search-fields="'name'"
          value-field="'sys_id'"
          ng-show="c.data.type=='assignee' || c.data.type=='requester'"/>
        <snh-form-field
          snh-model="c.data.group"
          snh-name="group"
          snh-label="Group"
          snh-type="reference"
          snh-required="c.data.type=='group'"
          table="'sys_user_group'"
          default-query="'active=true'"
          display-field="'name'"
          search-fields="'name'"
          value-field="'sys_id'"
          ng-show="c.data.type=='group'"/>
      </div>
      <div class="col-xs-12 col-sm-6">
        <snh-form-field
          snh-model="c.data.authentication"
          snh-name="authentication"
          snh-label="Authentication"
          snh-type="select"
          snh-required="true"
          snh-choices='[{"label":"None","value":"none"},{"label":"Basic","value":"basic"}]'/>
        <snh-form-field
          snh-model="c.data.username"
          snh-name="username"
          snh-label="Username"
          snh-type="text"
          snh-required="c.data.authentication=='basic'"
          ng-show="c.data.authentication=='basic'"/>
        <snh-form-field
          snh-model="c.data.password"
          snh-name="password"
          snh-label="Password"
          snh-type="password"
          snh-required="c.data.authentication=='basic'"
          ng-show="c.data.authentication=='basic'"/>
      </div>
    </div>
  </form>

  <div style="width: 100%; padding: 5px 50px; text-align: center;">
    <button ng-click="cancel()" class="btn btn-default ng-binding ng-scope" role="button" title="Click here to abandon this update and return to your webhooks">Cancel</button>
    &nbsp;
    <button ng-click="save()" class="btn btn-primary ng-binding ng-scope" role="button" title="Click here to save your input">Save</button>
  </div>
</snh-panel>

There’s nothing too magical there; just a bunch of SNH Form Fields wrapped inside of an SNH Panel. To mirror the UI Policies on the ServiceNow side of things, I used ng-show attributes to hide unneeded fields, and when those fields where required, I used the exact same criteria for the snh-required attribute, which kept it from telling me to complete fields that I couldn’t even see. With just that alone, I could throw the widget onto a page and bring it up, just to see what it looked like.

The new Webhook Registry widget layout

Not too bad, all things considered. Of course, this is just the layout. We still have to put the code underneath this presentation layer. We will definitely need some server side code to read and update the database, but I like to do the easy things first, so let’s start on the client side and throw in the code for the Cancel button. That just takes you right back the My Webhooks page, so that should be pretty simple, although we should bake in a little confirmation pop-up, just to be sure that the operator really does want to abandon their work. We can do that with a plain Javascript confirm, but I like the spModal version much better.

$scope.cancel = function() {
	spModal.confirm('Abandond your changes and return to your Webhooks?').then(function(confirmed) {
		if (confirmed) {
			$location.search('id=my_webhooks');
		}
	});
};

Technically, I should have checked to make sure that something was at least altered before I popped up that confirmation, and if not, just whisked you straight away to the My Webhooks page without asking. I may actually do that at some point, but this works for now. Unlike the Cancel button, the Save button will require some server side code, but we can still code out the client side while we are here and then tackle that next. Here is the code for the Save button.

$scope.save = function() {
	if ($scope.form1.$valid) {
		c.server.update().then(function(response) {
			$location.search('id=my_webhooks');
		});
	} else {
		$scope.form1.$setSubmitted(true);
	}
};

Here we check to make sure that there are no validation errors on the form before invoking the server side code, and then we return to the My Webhooks page once the server process has completed. If there are validation errors, then we set the form status to submitted to reveal all of the field errors to the user. SNH Form Fields hide validation errors until you touch the field or the form has been submitted, so setting the form to submitted here reveals any validation errors present for fields that have not yet been touched.

On the server side, we essentially have two events to handle: 1) widget initialization and 2) handling a Save action. The Save action involves input from the client side, so we know that if there is input present, we are doing a Save; otherwise, we are initializing the widget. At initialization, we need to look for a sys_id parameter in the URL, which tells us that we are updating an existing record. If there isn’t one, then we are adding a new record. For existing records, we need to go get the data and for new records, we need to initialize certain fields. Here is all that code:

data.sysId = $sp.getParameter("sys_id");
if (data.sysId) {
	whrGR.get(data.sysId);
	data.number = whrGR.getDisplayValue('number');
	data.type = whrGR.getValue('type');
	data.url = whrGR.getValue('url');
	data.document_id = {value: whrGR.getValue('document_id'), displayValue: whrGR.getDisplayValue('document_id')};
	data.group = {value: whrGR.getValue('group'), displayValue: whrGR.getDisplayValue('group')};
	data.person = {value: whrGR.getValue('person'), displayValue: whrGR.getDisplayValue('person')};
	data.owner = whrGR.getDisplayValue('owner');
	data.active = whrGR.getValue('active')=='1'?true:false;
	data.authentication =  whrGR.getValue('authentication');
	data.username = whrGR.getValue('username');
	data.password = whrGR.getValue('password');
} else {
	data.owner = gs.getUserDisplayName();
	data.active = true;
	data.document_id = {};
	data.group = {};
	data.person = {};
}

Similarly, when we do a Save, we need to know whether we are doing an update or an insert, which we can again tell by the presence of a sys_id. If we are updating, we need to go out and get the current record, and then in all cases, we need to move the data from the screen to the record and then save it. Here is all of that code:

if (input.sysId) {
	whrGR.get(input.sysId);
}
whrGR.type = input.type;
whrGR.url = input.url;
whrGR.document_id = input.document_id.value;
whrGR.group = input.group.value;
whrGR.person = input.person.value;
whrGR.setValue('active', input.active?'1':'0');
whrGR.authentication = input.authentication;
whrGR.username = input.username;
whrGR.password = input.password;
if (input.sysId) {
	whrGR.update();
} else {
	whrGR.insert();
}

That’s pretty much it for the widget. Just to make sure that it works, we can pull up an existing record and see what shows up on the screen.

Webhook Registry widget with existing record

Once I pulled the record up, I switched the Authentication to Basic and then hit the Save button, just to see if the form validation was working. So far, so good, but there is obviously a lot more testing to do, including the integration with the My Webhooks page. Still, things are looking pretty good at this point.

I’m not quite ready to put out an Update Set at the moment, as there is still quite a bit of testing that I would like to do first. Hopefully, though, I won’t find anything too major and I can drop the whole package next time out.

Fun with Webhooks, Part VIII

“If you are working on something that you really care about, you don’t have to be pushed. The vision pulls you.”
Steve Jobs

When we last parted, there was an open question on the table regarding the focus of this next installment. Implementing Basic Authentication and moving on to the My Webhooks Service Portal widget were two of the options, but no decision was made at the time as to which way we wanted to turn next. Now that we have finally made it here, the good news is that we don’t have to choose. Adding support for Basic Authentication turned out to be so simple that it looks like we are going to have time for both. Check it out:

if (whrGR.getValue('authentication') == 'basic') {
	request.setBasicAuth(whrGR.getValue('user_name'), whrGR.getValue('password'));
}

That’s it. I just added those few lines of code to the postWebhook function of our Script Include and BAM! — now we support Basic Authentication. Pretty sweet!

So now we can devote the remainder of the post to our new My Webhooks widget. As I think I mentioned earlier, this should be a fairly easy clone of the existing My Delegates widget, so the first thing that I did was to make a copy of that guy to have something with which to start. Then I hacked up the HTML to adapt it to our current application.

<table class="table table-hover table-condensed">
  <thead>
    <tr>
      <th style="text-align: center;">ID</th>
      <th style="text-align: center;">Table</th>
      <th style="text-align: center;">Type</th>
      <th style="text-align: center;">Reference</th>
      <th style="text-align: center;">Active</th>
      <th style="text-align: center;">Remove</th>
    </tr>
  </thead>
  <tbody>
    <tr ng-repeat="item in c.data.listItems track by item.id | orderBy: 'id'" ng-hide="item.removed">
      <td data-th="ID"><a href="?id=webhook_registry&sys_id={{item.sys_id}}">{{item.id}}</a></td>
      <td data-th="Table">{{item.table}}</td>
      <td data-th="Type">{{item.type}}</td>
      <td data-th="Reference">{{item.reference}}</td>
      <td data-th="Active" style="text-align: center;"><img src="/images/check32.gif" style="width: 16px; height: 16px;" ng-show="item.active"/></td>
      <td data-th="Remove" style="text-align: center;"><img src="/images/delete_row.gif" ng-click="deleteWebhook($index)" alt="Click here to permanently delete this Webhook" title="Click here to permanently delete this Webhook" style="cursor: pointer;"/></td>
    </tr>
  </tbody>
</table>

With that out of the way, we now need to replace the code that gathers up the Delegates with code that will gather up the Webhooks. Seems simple enough …

function fetchList() {
	var list = [];
	var whrGR = new GlideRecord('x_11556_simple_web_webhook_registry');
	whrGR.addQuery('owner', data.userID);
	whrGR.orderBy('number');
	whrGR.query();
	while (whrGR.next()) {
		var thisWebhook = {};
		thisWebhook.sys_id = whrGR.getValue('sys_id');
		thisWebhook.id = whrGR.getDisplayValue('number');
		thisWebhook.table = whrGR.getDisplayValue('table');
		thisWebhook.type = whrGR.getDisplayValue('type');
		thisWebhook.active = whrGR.getValue('active');
		if (thisWebhook.type == 'Single Item') {
			thisWebhook.reference = whrGR.getDisplayValue('document_id');
		} else if (thisWebhook.type == 'Assignment Group') {
			thisWebhook.reference = whrGR.getDisplayValue('group');
		} else {
			thisWebhook.reference = whrGR.getDisplayValue('person');
		}
		list.push(thisWebhook);
	}
	return list;
}

There is still a lot of work to do, but I like to try things every so often before I get too far along, just to make sure that things are going OK, so let’s do that now. This widget could easily go on the existing User Profile page just like the My Delegates widget, but it could also go on a page of its own. Since we are just trying things on for size right now, let’s just create a simple Portal Page and put nothing on it but our new widget. Let’s call it my_webhooks, drop our widget right in the middle of it, and go check it out on the Service Portal.

Initial My Webhooks widget

Well, that’s not too bad. We don’t really need the Save and Cancel buttons in this instance, but we do need a way to create a new Webhook, so maybe we can replace those with a single Add Webhook button. The links don’t go anywhere just yet and the delete icons don’t do anything, but as far as the layout goes, it looks pretty good. I think it’s a good start so far. Let’s swap out the buttons and then we can wrap things up with the client-side code.

The left-over code in the button area from the My Delegates widget looks like this:

<div style="width: 100%; padding: 5px 50px; text-align: center;">
  <button ng-click="saveDelegates()" class="btn btn-primary ng-binding ng-scope" role="button" title="Click here to save your changes">Save</button>
   
  <button ng-click="returnToProfile()" class="btn ng-binding ng-scope" role="button" title="Click here to cancel your changes">Cancel</button>
</div>

Let’s replace it with this:

<div style="width: 100%; padding: 5px 50px; text-align: center;">
  <button ng-click="newWebhook()" class="btn btn-primary ng-binding ng-scope" role="button" title="Click here to create a new Webhook">Add New</button>
</div>

Now we just need to overhaul the code on the client side to handle both the adding and deleting of our webhooks. The code to support the Add New button should be pretty straightforward; we just need to link to the same (currently nonexistent!) page that we link to from the main table, just without any sys_id to indicate that this is a new record request. This should handle that nicely:

$scope.newWebhook = function() {
	$location.search('id=webhook_registry');
};

As for the delete operation, we will have to bounce over to the server side to handle that one, but first we should confirm that was really the intent of the operator with a little confirmation pop-up. We could use the stock Javascript confirm feature here, but I like the look of the spModal version better, so let’s go with that.

$scope.deleteWebhook = function(i) {
	spModal.confirm('Delete Webhook ' + c.data.listItems[i].id + '?').then(function(confirmed) {
		if (confirmed) {
			c.data.toBeRemoved = i;
			$scope.server.update().then(function(response) {
				reloadPage();
			});
		}
	});
};

We’ll need some server side code to handle the actual deletion of the record, but that should be simple enough.

if (input) {
	data.listItems = input.listItems;
	if (input.toBeRemoved) {
		deleteWebhook(input.toBeRemoved);
	}
}

function deleteWebhook(i) {
	var whrGR = new GlideRecord('x_11556_simple_web_webhook_registry');
	if (whrGR.get(data.listItems[i].sys_id)) {
		whrGR.deleteRecord();
	}
}

We still need to test everything, but before we do that, we should go ahead and build the webhook_registry page that we have been pointing at so that we can fully test those links as well. That sounds like a good project for our next installment, so I think we will wrap things up right here for now and then start off next time with our new page followed by some end to end testing.

Fun with Webhooks, Part VII

“The question isn’t who is going to let me; it’s who is going to stop me.”
Ayn Rand

Now that we have proven that the essential elements of our Subflow all work as intended, it’s time finish out the remainder of the flow’s activities, which include logging the HTTP POST and Response details as well as reporting any undesired results to Event Management. Let’s start with logging the activity.

The first thing that we will need in order to record our Webhook POSTs is a table in which to store the data. As we did with our original Webhook Registry table, we can navigate to System Definition -> Tables and click on the New button to bring up the Table definition screen.

New Webhook Log table

And once again we will give it a Label and let the system generate the associated Name. We will also want to uncheck the Create module checkbox again to prevent the generation of a number of artifacts for which we have no use. Once the table has been defined, we can start adding fields, and the first field that we will want to add is a Reference to the Webhook Registry table. Every log record will be linked to the registry for which the activity was POSTed, so we will want to establish that relationship with a Reference field that we can label Registry.

The other Reference field that we will want is a link back to the original Incident that is the subject of the POST. Since we set things up in a way that would allow us to support tables other than Incident, we will want to do this with a Document ID field rather than a direct reference to the Incident table. This time, when we configure the Dependent Field, we can dot walk through the registry reference to get to the table name column in that related table. This will save us from having to have a column on the log table for the name of the table that holds the record associated with the Document ID, and it will ensure that all of the Document IDs related to each Registry will only come from the table associated with that Registry.

Selecting the Registry record’s Table column as the Dependent Field

The rest of the fields in the log table are just what we sent over, and what we received in response:

  • Payload – The data that we will be POSTing
  • URL – The URL to which we will be POSTing our Payload
  • Status – The HTTP Response Code returned by the target server
  • Body – The body of the message returned by the target server
  • Error – The error flag
  • Error Code – The error code
  • Error Message – The error message
  • Parse Error – Any error that occurred while parsing the body of the response

After defining all of the fields on the table, I brought up the table’s form and arranged all of the fields on the screen in a manner that I thought was most appropriate.

Webhook Log form layout

Those of you who are paying close attention will also have noticed that I added the JSON View Dictionary Attribute to both the Payload and Body fields, just to make reading the JSON content a little easier.

Now that we have a table, we can start putting records into it. We will do this in our Subflow, right after we POST the payload. This is just a simple, out-of-the-box Create Record action that we can configure using data pills from various other steps.

Logging the Webhook POST and Response

In addition to capturing everything related to each POST, the other thing that we wanted to do was to capture any issues that might come up during this process. We are already aware of two possible issues, one being a bad response and the other being a good response, but with an unparsable response body. After we do the POST and log the result, we can throw in a few more conditionals to pick those up, and then add a Log Event Action to the flow for each.

Complete Subflow with Event logging for error conditions

Whenever we log an Event, we will want to capture as much information as we can about what went wrong. In this case, however, we have already logged everything about the transaction to the Webhook Log table, so really all we will need to provide is a way to find that record. Putting the sys_id in the additional_info field should do the trick. Here is how I populated all of the data for the Event:

Event Log data

That should complete the Subflow, at least for now. We may end up adding some other features in the future, but for now, this accomplishes everything that we set out to do. We still need to do a lot more testing to verify that all of these various branches in the tree work out as we are hoping, but the building part should be done now, at least for this portion.

As far as the remaining development goes, we still have to build out the My Webhooks portal widget and we also need to go back into the Script Include and add support for Basic Authentication. We also need to add code, and possibly additional fields in the registry record, for any other authentication protocols that we would like to support. So once again we find ourselves at a crossroads: we can either jump into the Service Portal world and start working on our widget, or we can turn our attentions to authentication and finish things up in that area. There is no need to make any decision on that today, though. We’ll figure all of that out when we meet again.

Fun with Webhooks, Part VI

“The most exciting phrase to hear in science, the one that heralds discoveries, is not ‘Eureka!’ but ‘Now that’s funny …'”
Isaac Asimov

Even though we are not quite finished building out our new Subflow just yet, we have enough elements in place that we can try out what we have so far, and see if it actually works. The simplest way to do that would be to create a Webhook Registry that watches a single Incident, and then go make a qualifying change to that Incident and see what happens. Let’s go find an open Incident and then we can set up our registry.

In my instance, INC0000002 seemed like a likely candidate, so I pulled up the list of Webhook Registrations, clicked on the New button, and created the following new Webhook Registry entry to track changes to that specific Incident.

New Webhook Registry entry

For the URL, I used that same webhook.site address that I had used before so that I could see the result on the other end of the transaction. Once I had saved my new Webhook Registry entry, all I needed to do in order to test it out was to make a change to the Incident. For my first test, I simply changed the Assignment Group from Network to Hardware, which also then removed the Assigned to field value, since the original person specified in that field was not a member of the new Assignment Group. After making the change, I went over to webhook.site to see if anything showed up as a result of that modification. Sure, enough, a new POST had just been received by that site shortly after I saved the record.

New Webhook posting after updating an Incident

Although that proves that the process does indeed work as intended, I did notice one thing that is going to require a little bit of rework. Take a look at the second line of the message property:

Assigned To changed from Howard Johnson to  on INC0000002.

The Assigned to field changed from having a value to not having a value. The code that we have in our buildPayload function doesn’t really handle that circumstance in a way that produces the appropriate text to explain that in plain English. Here is the logic that creates that portion of the message:

if (current.assigned_to != previous.assigned_to) {
	payload.assigned_to = current.assigned_to;
	if (previous.assigned_to) {
		payload.message += separator + 'Assigned To changed from ' + previous.assigned_to + ' to ' + current.assigned_to + ' on ' + payload.id + '.';
	} else {
		payload.message += separator + 'Assigned To set to ' + current.assigned_to + ' on ' + payload.id + '.';
	}
	separator = '\n\n';
}

There is a check to see if the previous value was blank, but there is no check to see if the new value is blank. It wouldn’t take much to throw that in there, though, so let’s go ahead and do that now while we are thinking about it.

if (current.assigned_to != previous.assigned_to) {
	payload.assigned_to = current.assigned_to;
	if (previous.assigned_to) {
		if (current.assigned_to) {
			payload.message += separator + 'Assigned To changed from ' + previous.assigned_to + ' to ' + current.assigned_to + ' on ' + payload.id + '.';
		} else {
			payload.message += separator + 'Assigned To ' + previous.assigned_to + ' removed.';
		}
	} else {
		payload.message += separator + 'Assigned To set to ' + current.assigned_to + ' on ' + payload.id + '.';
	}
	separator = '\n\n';
}

That’s better. While we are at it, we might as well make the same change to the code for the Assignment group, as that is pretty much a line for line copy of the code for the Assigned to field. Everything else looks to be OK. The State is mandatory, so it will never have a new blank value, and the Comments and Work Notes fields are based on new entries and not on changes to existing entries, so if either of those are ever blank, we won’t be doing anything with them at all.

Still, other than that one little annoying language problem, the test was actually quite successful. Our Business Rule fired, the change was detected, our new Subflow was launched, our payload was generated, and the payload was successfully POSTed to the appropriate URL. All in all, I would have to say that that was a pretty good first test of the process. Obviously, we need to do a lot more testing, but this was a great start.

At this point, all I really wanted to do was to make sure that everything was working so far, which I think we accomplished. Before we do too much more testing, though, I think we need to go back and finish up the rest of the Subflow. We still have to add in the step that logs the activity, and we also have to add any error handling for when things don’t go as planned. Once we build all of that out, then we can do a lot more testing to validate the entire process from end to end. It was good to take a quick break and make sure that everything was working up to this point, but now that we know that it is, we need to get back to work on finishing up our development efforts.

Fun with Webhooks, Part V

“Change is easy. Improvement is far more difficult.”
Ferdinand Porsche

Now that we have a Business Rule to launch our Subflow, it’s time to get busy and actually create the Subflow. To create a new Subflow, open up the Flow Designer, click on the New button and select Subflow from drop-down menu. I initially called mine Webhook Poster, but the Business Rule couldn’t find it to launch it for some reason. I finally got it to work when I replaced the space with an underscore, so now the name is Webhook_Poster. I’m not really sure why I had to do that, but it works, so we will leave it at that.

With the Subflow named, the next thing to do is to define all of the inputs. In my mind, there were only two, the current object and the previous object. Unfortunately, the Flow Designer will not recognize any of the properties of those objects unless you explicitly define them, so I had to specify every element of both objects. That seemed like a rather tedious waste of time, but eventually I got through it and now it’s done.

Subflow inputs defined

That takes care of the set-up, so now it’s on to the flow itself. The first thing that we are going to want to do is to gather up all of the registration records that would be triggered by this Incident. Since we gave our users several options for selecting records, we will have to test them all. Essentially, we will need a query filter for each possible Type, which we can then OR together to create one huge query filter. Here is how that looks in practice:

Webhook Registry selection criteria

As a second step, I threw in an IF statement to see if the query returned any records. We are basically done at this point if there are no records, but if we do have records to process, then we will need to build the payload that we will be posting to all of the target URLs. For that job, I needed to create a new Flow Designer Action, so I saved my Subflow and closed the Subflow tab for now.

The payload is basically a generic JSON object containing the relevant information about the the things that have changed about the record since the last time that it was saved. Pulling that all together will basically be a lot of tedious scripting, so that sounds like a job for yet another function in our Script Include. Here is what I came up with:

buildPayload: function(current, previous) {
	var payload = {};

	payload.id = current.number;
	payload.short_description = current.short_description;
	payload.message = '';
	var separator = '';
	if (current.state != previous.state) {
		payload.state = current.state;
		if (previous.state) {
			payload.message += separator + 'State changed from ' + previous.state + ' to ' + current.state + ' on ' + payload.id + '.';
		} else {
			payload.message += separator + 'State set to ' + current.state + ' on ' + payload.id + '.';
		}
		separator = '\n\n';
	}
	if (current.assignment_group != previous.assignment_group) {
		payload.assignment_group = current.assignment_group;
		if (previous.assignment_group) {
			payload.message += separator + 'Assignment Group changed from ' + previous.assignment_group + ' to ' + current.assignment_group + ' on ' + payload.id + '.';
		} else {
			payload.message += separator + 'Assignment Group set to ' + current.assignment_group + ' on ' + payload.id + '.';
		}
		separator = '\n\n';
	}
	if (current.assigned_to != previous.assigned_to) {
		payload.assigned_to = current.assigned_to;
		if (previous.assigned_to) {
			payload.message += separator + 'Assigned To changed from ' + previous.assigned_to + ' to ' + current.assigned_to + ' on ' + payload.id + '.';
		} else {
			payload.message += separator + 'Assigned To set to ' + current.assigned_to + ' on ' + payload.id + '.';
		}
		separator = '\n\n';
	}
	if (current.comments) {
		payload.comments = current.comments;
		payload.message += separator + current.operator + ' has added a new comment to ' + payload.id + ':\n' + current.comments;
		separator = '\n\n';
	}
	if (current.work_notes) {
		payload.work_notes = current.work_notes;
		payload.message += separator + current.operator + ' has added a new work note to ' + payload.id + ':\n' + current.work_notes;
		separator = '\n\n';
	}

	return JSON.stringify(payload, null, '\t');
},

We still need a Flow Designer Action to call this function, but the function does almost all of the heavy lifting and our Action should now be quite simple to put together. Let’s open up the Flow Designer again, click on the New button, and select Action from the drop-down selection list. We’ll call our new Action Build Webhook Payload since that’s what it does, and we will define two inputs, the current and previous objects. Since we won’t be referencing any of the properties of those objects directly, this time we won’t have to invest the time in specifying all of the elements of the objects.

All we will need for our Action is to add a Script step with the same two inputs and the payload as an output. The script itself is just a call to the function that we just added to our Script Include.

(function execute(inputs, outputs) {
	var wru = new WebhookRegistryUtils();
	outputs.payload = wru.buildPayload(inputs.current, inputs.previous);
})(inputs, outputs);

We also need to define the payload as an output of Action itself, and map the Action inputs to the Script step inputs and the Script step output to the Action output. That should complete our new Action.

Build Webhook Payload Action

Now we can go back into our Subflow and select this Action as a third step under our second step conditional that checks to see if there are any records from the query in the first step. For our fourth step, we will add a flow logic step that loops through all of the records from step 1, and inside of that loop, our fifth step will make the POST of the payload. I looked for an out-of-the-box simple HTTP POST Action already included on the platform, but I couldn’t find anything, so that’s another Action that we are going to have to produce ourselves. We already built much of the script for that when we built our testURL function; if we rework that code just a little bit, we can probably find a way to make it work just fine for both purposes.

testURL: function(whrGR) {
	var jsonObj = {message: 'This is a test posting.'};
	return this.postWebhook(whrGR, JSON.stringify(jsonObj, null, 4));
},

postWebhook: function(whrGR, payload) {
	var result = {};

	var request = new sn_ws.RESTMessageV2();
	request.setEndpoint(whrGR.getValue('url'));
	request.setHttpMethod('POST');
	request.setRequestHeader('Content-Type', 'application/json');
	request.setRequestHeader('Accept', 'application/json');
	request.setRequestBody(payload);
	var response = request.execute();
	result.status = response.getStatusCode();
	result.body = response.getBody();
	if (result.body) {
		try {
			result.obj = JSON.parse(result.body);
		} catch (e) {
			result.parse_error = e.toString();
		}
	}
	result.error = response.haveError();
	result.error_code = response.getErrorCode();
	result.error_message = response.getErrorMessage();

	return result;
},

Now the testURL function is just a call to our new postWebhook function that contains most of the code previously contained in the testURL function. The testURL function will pass in our simple test payload, and when we create out new Action, we can call the very same postWebhook function, passing in a real payload. The script for our new Action will then simply be this:

(function execute(inputs, outputs) {
	var wru = new WebhookRegistryUtils();
	var result = wru.postWebhook(inputs.webhook_registry, inputs.payload);
	for (var key in result) {
		outputs[key] = result[key];
	}
})(inputs, outputs);

Stopping right here should give us enough with which to test out the process so far. This isn’t all that we want to do here, because we still want to record both the attempt and the response in our log table. However, since we haven enough built out to just test the POSTing of payloads, I think we should stop and do that. Afterwards, we can circle back and build out the activity logging and any Event logging that we might want to do for any kind of bad responses. But right now, we have enough built out that we can create a few Webhook Registrations and then go update some qualifying Incidents to see what actually happens.

Setting all of that up and verifying the results seems worthy of its own episode, so let’s wrap things up for today and pick that up next time out.

Event Management for ServiceNow, Revisited

“True prevention is not waiting for bad things to happen, it’s preventing things from happening in the first place.”
Don McPherson

Some time ago we built some utility functions to support reporting Events within the ServiceNow Platform. That was before the Flow Designer, though, so that effort did not include any support for that environment. We already have the script to do all of the heavy lifting from our earlier work, so it wouldn’t take much to create a Flow Designer Action that called that script to report an Event that occurred during Flow processing. We can call our new Action Log Event, and set up Action Inputs for all of the usual suspects.

Log Event Action Inputs

For our script step, we will basically set up the same inputs and then source them directly from the primary Action Inputs.

Script step inputs mapped to Action Inputs

Those of you who are paying attention will notice that we defined the additional_input field as a String even though it needs to be an Object when we make the call to our existing script. The assumption here is that the caller will provide a valid JSON String, and then we can turn it into an Object in our script before we make the call. Here is the script to convert the String and then make the call.

(function execute(inputs, outputs) {
	if (inputs.additional_info) {
		try {
			inputs.additional_info = JSON.parse(inputs.additional_info);
		} catch(e) {
			//
		}
	}
	var seu = new ServerEventUtil();
	seu.logEvent(inputs.source, inputs.resource, inputs.metric_name, inputs.severity, inputs.description, inputs.additional_info);
})(inputs, outputs);

There are no outputs from this process, so this is the entire Action. Once we Save and Publish it, it will be available from the Action selection list, and then we can add Log Event steps anywhere in our Flows and Subflows where we want to report an Event. That was fairly quick, easy, and relatively painless. For those of you would like to try it out on your own, here is an Update Set.

Fun with Webhooks, Part IV

“Try not to become a man of success, but rather try to become a man of value.”
Albert Einstein

Last time, we came to a fork in the road, not knowing whether it would be better to jump into the My Webhooks Service Portal widget, or to start working out the details of the process that actually POSTs the Webhooks. At this point, I am thinking that the My Webhooks widget will end up being a fairly simple clone of my old My Delegates widget, so that may not be all that interesting. POSTing the Webhooks, on the other hand, does sound like it might be rather challenging, so let’s start there.

When I first considered attempting this, my main question was whether it would be better to handle this operation in a Business Rule or by building something out in the Flow Designer. After doing a little experimenting with each, I later came to realize that the best alternative involved a little bit of both. In a Business Rule, you have access to both the current and previous values of every field in the record; that information is not available in the Flow Designer. In fact, it’s not available in a Business Rule, either, if you attempt to run it async. But if you don’t run it async, then you are holding everything up on the browser while you wait for everything to get posted. That’s not very good, either. What I ultimately decided to do was to start out with a Business Rule running before the update, gather up all of the needed current and previous values, and then pass them to a Subflow, which runs async in the background.

My first attempt was to just pass the current and previous objects straight into my Subflow, but that failed miserably. Apparently, when you pass a GlideRecord into a Subflow, you are just passing a reference, not the entire record, and then when the Subflow starts up, it uses that reference to fetch the data from the database. That’s not the data that I wanted, though. I want the data as it was before the database was updated. I had to take a different route with that part of it, but the basic rule remained the same.

Business Rule to POST webhooks

The rule is attached to the Incident table, runs on both Insert and Update, and is triggered whenever there is a change to any of the following fields: State, Assignment Group, Assigned to, Work notes, or Additional comments.

To get around my GlideRecord issue, I wound up creating two objects, one for current and one for previous, which I populated with data that I pulled out of the original GlideRecords. When I passed those two objects to my Subflow, everything was there so that I could do what I wanted to do. Populating the objects turned out to be quite a bit of code, so rather than put that all in my Business Rule, I created a function in my Script Include called buildSubflowInputs and I passed it current and previous as arguments. That simplified the Business Rule script quite a bit.

(function executeRule(current, previous) {
	var wru = new WebhookRegistryUtils();
	sn_fd.FlowAPI.startSubflow('Webhook_Poster', wru.buildSubflowInputs(current, previous));
})(current, previous);

In the Script Include, things got a lot more complicated. Since I was going to be turning both the current and previous GlideRecords into objects, I thought it would be helpful to create a function that did that, which I could call once for each. That function would pull out the values for the Sys ID, Number, State, Caller, Assignment group, and Assigned to fields and return an object populated with those values.

getIncidentValues: function(incidentGR) {
	var values = {};

	values.sys_id = incidentGR.getValue('sys_id');
	values.number = incidentGR.getDisplayValue('number');
	if (!incidentGR.short_description.nil()) {
		values.short_description = incidentGR.getDisplayValue('short_description');
	}
	if (!incidentGR.state.nil()) {
		values.state = incidentGR.getDisplayValue('state');
	}
	if (!incidentGR.caller_id.nil()) {
		values.caller = incidentGR.getDisplayValue('caller_id');
		values.caller_id = incidentGR.getValue('caller_id');
	}
	if (!incidentGR.assignment_group.nil()) {
		values.assignment_group = incidentGR.getDisplayValue('assignment_group');
		values.assignment_group_id = incidentGR.getValue('assignment_group');
	}
	if (!incidentGR.assigned_to.nil()) {
		values.assigned_to = incidentGR.getDisplayValue('assigned_to');
		values.assigned_to_id = incidentGR.getValue('assigned_to');
	}

	return values;
},

Since my Business Rule could be fired by either an Update or an Insert, I had to allow for the possibility that there was no previous GlideRecord. I could arbitrarily call the above function for current, but I needed to check to make sure that there actually was a previous before making that call. I also wanted to add some additional data to the current object, including the name of the person making the changes. The field sys_updated_by contains the username of that person, so to get the actual name I had to use that in a query of the sys_user table to access that data.

buildSubflowInputs: function(currentGR, previousGR) {
	var inputs = {};

	inputs.current = this.getIncidentValues(currentGR);
	if (currentGR.isNewRecord()) {
		inputs.previous = {};
	} else {
		inputs.previous = this.getIncidentValues(previousGR);
	}
	inputs.current.operator = currentGR.getDisplayValue('sys_updated_by');
	var userGR = new GlideRecord('sys_user');
	if (userGR.get('user_name', inputs.current.operator)) {
		inputs.current.operator = userGR.getDisplayValue('name');
	}

	return inputs;
},

One other thing that I wanted to track was any new comments or work_notes, and that turned out to be the most the most complicated code of all. If you try the normal getValue or getDisplayValue methods on any of these Journal Fields, you end up with all of the comments ever made. I just wanted the last one. I had to do a little searching around to find it, but there is a function called getJournalEntry that you can use on these fields, and if you pass it a 1 as an argument, it will return the most recent entry. However, it is not just the entry; it is also a collection of metadata about the entry, which I did not want either. To get rid of that, I had to find the first line feed and then pick off all of the text that followed that. Putting all of that together, you end up with the following code:

if (!currentGR.comments.nil()) {
	var c = currentGR.comments.getJournalEntry(1);
	inputs.current.comments = c.substring(c.indexOf('\n') + 1);
}
if (!currentGR.work_notes.nil()) {
	var w = currentGR.work_notes.getJournalEntry(1);
	inputs.current.work_notes = w.substring(w.indexOf('\n') + 1);
}

That’s pretty much all that there is to the new Business Rule and the needed additions to the Script Include. Now we need to start tackling the Subflow that will actually take these current and previous objects and do something with them. That’s a bit much to jump into at this point, so we’ll wrap things up here for now and save all of that for our next time out.

Fun with Webhooks, Part III

“All things are difficult before they are easy.”
Thomas Fuller

Now that we have built the Webhook Registry table and configured the associated form layout, the last thing we need to do before we move on is to set up the Test URL UI Action and the underlying code to make it work. Clicking on the Test URL button should pass the current registry record to a function in a new Script Include that will pull the URL out of the registry, POST an example JSON object to the URL, and verify the response. Since we have to reference the Script Include in the UI Action, let’s work on that first.

To create a new Script Include, navigate to System Definition -> Script Includes and then click on the New button. Since this will be a general purpose Script Include for our application, I named it WebhookRegistryUtils.

New WebhookRegistryUtils Script Include

To test a URL, we will want to add a new function to our new Script Include that accepts a Webhook Registry GlideRecord as an argument and returns an Object that contains the results of the test POST. Let’s call this function testURL().

testURL: function(whrGR) {
	var result = {};

	var jsonObj = {message: 'This is a test posting.'};
	var request = new sn_ws.RESTMessageV2();
	request.setEndpoint(whrGR.getValue('url'));
	request.setHttpMethod('POST');
	request.setRequestHeader('Content-Type', 'application/json');
	request.setRequestHeader('Accept', 'application/json');
	request.setRequestBody(JSON.stringify(jsonObj, null, 4));
	var response = request.execute();
	result.status = response.getStatusCode();
	result.body = new global.JSON().decode(response.getBody());
	result.error = response.haveError();
	result.errorCode = response.getErrorCode();
	result.errorMessage = response.getErrorMessage();

	return result;
},

This code is fairly self-explanatory. The jsonObj is our test payload and the request is an empty sn_ws.RESTMessageV2 that we populate with the URL found in the registry record and the hard-coded Method of POST. The response is the result of executing the request, and then we pull all of the relevant information out of the response and use it to populate the result object, which we then return to the caller. We may end up reworking some of this when we start adding additional functionality, but this should be enough to test a URL with no authentication.

At this point, we could test out the code by writing a quick little Fix Script to grab a registry record and pass it to this function, but it might be just as easy to ahead and just make the UI Action and test it out on the form. To create the UI Action, pull up the Webhook Registry form, select Configure -> UI Actions from the hamburger menu, and then click on the New button. I named mine Test URL, checked the Form button checkbox, and put the following script in the Condition:

current.url > ''

That condition keeps the Test URL button from appearing on the form when there is no URL to test.

Test URL UI Action

For the Script, I just call the testURL function in our new Script Include and then notify the user of the results. There is not much to the code, but here it is:

var wru = new WebhookRegistryUtils();
var result = wru.testURL(current);
action.setRedirectURL(current);
if (result.status == '200') {
	gs.addInfoMessage('URL "' + current.url + '" was tested successfully.');
} else {
	gs.addErrorMessage('URL test failed: ' + JSON.stringify(result, null, '<br/> '));
}

There is undoubtedly a much nicer way to format that failure message, but this will get the job done for now. We can always clean that up later. Right now, though, let’s push the button and see what happens! On second thought, though, we can’t really do that until we insert a record into the registry. let’s take care of that now.

New Webhook Registry entry

Once all of the parts are in place, this entry should POST a payload to the specified URL every time there is a change to an Incident assigned to the owner. For the URL, I went out to check out Webhook.site, and was assigned that end point URL, so I threw that in there. That should be a valid webhook URL, so our first test should be successful. Once we prove that, we can mangle up the URL and hit it again to see what a failure looks like. But first things first — let’s see if we can make it work with a real URL.

Successful URL test

So far, so good! Real quick, before we try out the bad one, let’s pop over to Webhook.site and see if there is any evidence that we ran this test. Oh, look — there is!

Test POST results found on Webhook.site

That’s actually a pretty cool little tool. That’s going to be quite handy for this particular adventure. I like it. Now, let’s try out one that we know will fail.

Unsuccessful URL test

Well, that seems to work as well. So now we have our first table and related form, the beginnings of our Script Include and a handy UI Action. From here, we can work on the My Webhooks Portal Widget, or we can start building out the process that sends out the POSTs to the qualifying destinations. That’s nothing that we need to decide today, though; we can figure that all out next time.

Fun with Webhooks, Part II

“Well done is better than well said.”
Benjamin Franklin

Now that we have the idea fairly well formulated, it’s time to get to work. The first thing that we need to do is create our Scoped Application, which you can do by navigating to My Company Applications and then clicking on the Create new button in the upper right-hand corner. I still use the classic UI rather than the Studio, so you may approach this task a little differently, but the end result should be the same.

New Simple Webhook Application

At this point, I have just created the most basic of empty shells. There are no modules or tables or roles or any other artifacts … it’s basically just the bare application record itself. Creating the app does create a scope, and also puts you into that scope, so as long as you don’t change that, everything that you do from this point on will be built in that scope. The first thing that we will want to build is the database table for our registry, which we can do by navigating to System Definition -> Tables and clicking on the New button.

New Webhook Registry table

Once you give it a Label it will generate the appropriate name, and the next thing that you are going to want to do is to uncheck the Create module option, as we only want the table at this point and we don’t need all of those other artifacts generated. We want to give our registrations an ID using the platform’s built-in tools, so we will want to create a field labeled Number of type String and put the following in the Default value:

javascript:getNextObjNumberPadded();

We have a number of other columns to define, but let’s save this for now and go set up our auto-numbering before we forget. To do that, navigate to System Definition -> Number Maintenance and click on the New button. Select our new table from the list and then let’s set the Prefix to WHR for Webhook Registry,

Setting up auto-numbering for our new Webhook Registry table

With that out of the way, we can go back to our table and add the rest of the fields. Here are the ones that I added to get things started:

  • Active – True/False, with default of True
  • Table – Table Name, with default of incident
  • Owner – Reference to the sys_user table
  • Type – String, with four initial choices (Single Item, Caller / Requester, Assignment Group, and Assignee)
  • URL – URL
  • Authentication – String, with two initial choices (None and Basic)
  • User Name – String
  • Password – Password
  • Document ID – Document ID
  • Person – Reference to the sys_user table
  • Group – Reference to the sys_user_group table

We may add a few more later on as we add new features, but this will get us by for now. Event though my intent is to focus solely on the Incident table for now, I went ahead and added the Table column and just defaulted it to Incident. That was mainly so that I could use as the Dependent field on the Document ID field. If you have never worked with Document ID fields before, you can just think of them as a special form of Reference field where you don’t have to specify the table that is being referenced. Instead, you point to another field on the record that contains the name of the table. This way, one of your records can reference one table and another record in that same table can reference a different table, all based on the table specified in the Dependent field. To set up the Dependent field, use the Dependent Field tab on the Dictionary Entry for the Document ID field.

Document ID Dependent field specification

After entering all of the fields, I used the Show form link at the bottom of the page to bring up the form for the new table. Using the Configure -> Form Layout option, I rearranged the fields on the screen to my liking, and then removed the Table field completely, as that will default to Incident for now and we don’t want anyone changing it to anything else at this point.

After getting everything laid out just right, the next thing that I did was to add a few UI Policies to control certain fields based on the value in other fields. For example, I hide the User Name and Password fields unless you set the Authentication to Basic. Similarly, the presence of the Document ID, Group, and Person fields are all dependent on the value of the Type field. Basically, this just hides fields that are not needed and reveals them when they are.

The other things that I wanted to do on this form was to give the user the ability to test their URL. That seemed like a good use for a new UI Action, but rather than putting all of the code for that process in the Action itself, I decided to start a Script Include to house these types of utilitarian functions. Putting all of that together seems like a good exercise for our next installment in this series, so this looks like a good stopping point for now.

Fun with Webhooks

“Good ideas are common – what’s uncommon are people who’ll work hard enough to bring them about.”
Ashleigh Brilliant

There is quite a bit of Webhook stuff in various IntegrationHub spokes, but it all seems to be oriented towards consuming incoming events from different external event publishers. I want to actually be the publisher, and send out information based on some preferences selected by the consumer. That may be hidden somewhere in the Now Platform already, but I can’t seem to find it, so I have decided that I would try to develop a Scoped Application to do just that. This may very well be recreating something that already exists in the platform today, but it sounds like a fun exercise, so I am going to give it the old college try.

As always, I will attempt to start out with the most basic of offerings, and then incrementally expand to add more and better features. My approach is to treat this feature as somewhat analogous to a Watch List, in that you sign up to follow certain events, but instead of sending a notification to a User when the event occurs, the result will be that the information is posted to a specified URL. This can apply to any number of things, but to start off, I am going to focus on some very specific changes to one particular table (Incident), and then later expand from there.

To make this work, there will need to be some kind of Webhook Registry where a consumer would sign up to receive these posts. When registering your webhook, you would enter the URL to which you want the data posted along with the specifics of what type or types of events you would like to have included. I’m thinking about linking them directly to an owner, and having some kind of My Webhooks Portal Page where you could manage your existing registrations and add new ones. When adding a new one, you should be able to enter and test your URL, and for our first iteration, that may be the only choice that you get. Later on, we will want to add the ability to choose what you want to follow, which specific updates should trigger a new post, and even what you would like to have included in the payload. But we will also want to start out as simple as possible, so the initial registry may turn out to be quite barren as far as input fields go.

Once registered, there will need to be some process to actually send out the posts as requested in the registration. This could be a Business Rule on the source table, or maybe something created in the Flow Designer. Either way, the process should scan the registry for any condition matches and then send out a post for each match. Each post and response should be logged in some kind of Webhook Activity Log, and any bad HTTP Response Codes should be reported to Event Management. A robust service would attempt to repost any failures up to a certain limit before giving up completely, but all of that can be delegated to some Alert Management Rule at some later time. Again, we will want to start out simple, so our initial focus will just be on making that initial post attempt. Everything else can be pushed off until later on in the process.

Those would seem to be the two major functions: registering the webhook and sending out the posts. We may want some other things at some point, such as the ability to review the logs or to manually repost or to clone an existing registration, but for now, just those two things should get the ball rolling. We may also want to set up a sample receiver for testing purposes, but in practice, the receivers would be other products and outside the scope of this development exercise. There is actually an existing service out on the Internet called Webhook.site that might turn out to be just what I need in order to do a little testing. We should check that out when we get to that point.

For our parts list, then, I can see the need for the following artifacts:

  • A table to hold the webhook registrations,
  • A my_webhooks portal widget to list all webhooks owned by the user,
  • A webhook portal widget for editing a single webhook registration,
  • A Business Rule or Flow to send out the posts,
  • A log table to record the posts and response, and possibly
  • A Script Include to contain some common functions.

Of course, before we create any of that, we will have to create the Scoped Application itself, so that should be where we start next time when we initiate the actual construction phase of this effort.