Collaboration Store, Part XXXII

“There are two ways to write error-free programs; only the third works.”
Alan J. Perlis

Now that we have all of the Javascript functions needed to push a new application version out to all of the Client instances, we need to create an Action that will call the root function and then create a Subflow that will utilize the new Action. Since we already created a similar Action for the process that shares new Client instances with existing Client instances, the easiest thing to do will be to bring that guy up in the Flow Designer and make a copy. To make a copy, pull up the Action that you want to copy, and use the ellipses menu to select Copy from the drop-down menu.

Making a copy of the Publish New Instance Action

Once we have our copy, we need to make the necessary changes to adapt it to our purpose. The publishNewVersion function that we just created takes three arguments (newVersion, targetInstance, and attachmentId), so we will need to modify the Action Inputs to bring in those values. The new Action Inputs now look like this:

Modified Action Inputs

Similarly, we will need to modify the Input Variables of the Script Step to pass along these values:

Modified Script Input Variables

The last thing that we need to do is modify the script itself. The existing script in the cloned Action was pretty simple, as it was just a call to one of our Script Include functions:

(function execute(inputs, outputs) {
	var csu = new CollaborationStoreUtils();
	csu.publishNewInstance(inputs.new_instance, inputs.target_instance);
})(inputs, outputs);

We also want to call a function, and it is in the very same Script Include, so our modification is just a matter of changing the name of the function along with the arguments passed.

(function execute(inputs, outputs) {
	var csu = new CollaborationStoreUtils();
	csu.publishNewVersion(inputs.new_version, inputs.target_instance, inputs.attachment_id);
})(inputs, outputs);

That’s it for the changes, so now all we have to do is Save and Publish the new Action and we are ready to utilize it in our new Subflow. Creating the Subflow will be another Copy operation, this time from our existing New_Collaboration_Store_Instance Subflow. We’ll call our copy Distribute_New_Application_Version, and once again, we will need to make some modifications to adapt it to our new purpose. The original Subflow had a single input, the New Instance. Instead of a new instance, we will be distributing a new application version, so we can replace the New Instance input with a New Version input that will contain the sys_id of the new version record. We will also need the sys_id of the Attachment record, so we will add an Attachment ID input as well.

Modified Inputs for the new Distribute_New_Application_Version Subflow

The target instances for this process will be the same as those for the original Subflow: all instances in the community except for the Host instance and the instance from which the data was provided (which could also be the Host in this case, since Host instances can publish applications just like any other Client instance). To filter out the data providing instance, we will need to add a new step to use the passed version record sys_id to fetch the GlideRecord that will lead us to the instance that produced the application.

New step to fetch the version record

This gives us the information that we need to complete the filter on the query of the instance table so that we can gather up all of the instances except for the Host and the instance that produced the application.

Modified query filter using the data available in the fetched version record

Now all we need to do is to replace the original custom Action inside of the loop with the new Action that we just created.

New Action added to push application artifacts to the target instance

This completes the changes needed to the cloned Subflow, so again, all we need to do is to Save and Publish and we are all ready to go. Now all we need to is to come up with a way to launch this Subflow whenever a new application version is sent over to the Host instance. We’ll take a look at that next time out, and hopefully release a new Update Set as well so that those of you that have been kind enough to participate in the testing can give this all a trial run.

Collaboration Store, Part XXXI

“Someone’s sitting in the shade today because someone planted a tree a long time ago.”
Warren Buffett

While we wait for additional test results to come in from the corrected Update Set for our latest iteration, it would be a good time to take a look at what we will need to do to complete the last remaining step in the application publishing process. What we have completed so far pushes all of the new application version artifacts to the Host instance. Now we need to build out the process for the Host instance to send out all of those artifacts to all of the other Client instances. Fortunately, we already have in hand most of the code to do that; we just need to clone a few things and make a number of modifications to fit the new purpose.

We have already built a Subflow to push new Client instance information out to all of the existing Client instances during the new instance registration process. Pushing out a new version of an application to those same instances is essentially the same process, so we should be able to clone that guy, and the associated Action, to create our new process. We have also created functions to push over the application record, the version record, and the attached Update Set XML file, and we should be able to copy over those guys as well to complete the process. In fact, we should probably start with the JavaScript functions, as we will need to call the root function in the Action, which will need to be created before it can be referenced in the Subflow.

The functions as written are hard-coded to send everything over to the Host instance. We will want to do basically the same thing, but this time we will be sending things out to various Client instances. Ideally, I should just refactor the code to have the destination instance and credentials passed in so that one function would work in both circumstances, but at this point it was less complicated to just make copies of each function and hack them up for their new purpose. I don’t really like having two copies of essentially the same thing, though, so one day I am going to have to go back and rework the entire mess.

But for now, this is what I came up with when I copied what was once called processPhase5 to create the new publishNewVersion function:

publishNewVersion: function(newVersion, targetInstance, attachmentId) {
	var targetGR = new GlideRecord('x_11556_col_store_member_organization');
	if (targetGR.get('instance', targetInstance)) {
		var token = targetGR.getDisplayValue('token');
		var versionGR = new GlideRecord('x_11556_col_store_member_application_version');
		if (versionGR.get(newVersion)) {
			var canContinue = true;
			var targetAppId = '';
			var mbrAppGR = versionGR.member_application.getRefRecord();
			var request  = new sn_ws.RESTMessageV2();
			request.setHttpMethod('get');
			request.setBasicAuth(this.WORKER_ROOT + targetInstance, token);
			request.setRequestHeader("Accept", "application/json");
			request.setEndpoint('https://' + targetInstance + '.service-now.com/api/now/table/x_11556_col_store_member_application?sysparm_fields=sys_id&sysparm_query=provider.instance%3D' + mbrAppGR.getDisplayValue('provider.instance') + '%5Ename%3D' + encodeURIComponent(mbrAppGR.getDisplayValue('name')));
			var response = request.execute();
			if (response.haveError()) {
				gs.error('CollaborationStoreUtils.publishNewVersion: Error returned from Target instance ' + targetInstance + ': ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
				canContinue = false;
			} else if (response.getStatusCode() == '200') {
				var jsonString = response.getBody();
				var jsonObject = {};
				try {
					jsonObject = JSON.parse(jsonString);
				} catch (e) {
					gs.error('CollaborationStoreUtils.publishNewVersion: Unparsable JSON string returned from Target instance ' + targetInstance + ': ' + jsonString);
					canContinue = false;
				}
				if (canContinue) {
					var payload = {};
					payload.name = mbrAppGR.getDisplayValue('name');
					payload.description = mbrAppGR.getDisplayValue('description');
					payload.current_version = mbrAppGR.getDisplayValue('current_version');
					payload.active = 'true';
					request  = new sn_ws.RESTMessageV2();
					request.setBasicAuth(this.WORKER_ROOT + targetInstance, token);
					request.setRequestHeader("Accept", "application/json");
					if (jsonObject.result && jsonObject.result.length > 0) {
						targetAppId = jsonObject.result[0].sys_id;
						request.setHttpMethod('put');
						request.setEndpoint('https://' + targetInstance + '.service-now.com/api/now/table/x_11556_col_store_member_application/' + targetAppId);
					} else {
						request.setHttpMethod('post');
						request.setEndpoint('https://' + targetInstance + '.service-now.com/api/now/table/x_11556_col_store_member_application');
						payload.provider = mbrAppGR.getDisplayValue('provider.instance');
					}
					request.setRequestBody(JSON.stringify(payload, null, '\t'));
					response = request.execute();
					if (response.haveError()) {
						gs.error('CollaborationStoreUtils.publishNewVersion: Error returned from Target instance ' + targetInstance + ': ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
						canContinue = false;
					} else if (response.getStatusCode() != 200 && response.getStatusCode() != 201) {
						gs.error('CollaborationStoreUtils.publishNewVersion: Invalid HTTP Response Code returned from Target instance ' + targetInstance + ': ' + response.getStatusCode());
						canContinue = false;
					} else {
						jsonString = response.getBody();
						jsonObject = {};
						try {
							jsonObject = JSON.parse(jsonString);
						} catch (e) {
							gs.error('CollaborationStoreUtils.publishNewVersion: Unparsable JSON string returned from Target instance ' + targetInstance + ': ' + jsonString);
							canContinue = false;
						}
						if (canContinue) {
							targetAppId = jsonObject.result.sys_id;
						}
					}
				}
			} else {
				gs.error('CollaborationStoreUtils.publishNewVersion: Invalid HTTP Response Code returned from Target instance ' + targetInstance + ': ' + response.getStatusCode());
			}
			if (canContinue) {
				this.publishVersionRecord(targetInstance, token, versionGR, targetAppId, attachmentId);
			}
		} else {
			gs.error('CollaborationStoreUtils.publishNewVersion: Version record not found: ' + newVersion);
		}
	} else {
		gs.error('CollaborationStoreUtils.publishNewVersion: Target instance record not found: ' + targetInstance);
	}
},

Unlike the original application publication process, we are not bouncing back and forth between the client side and the server side, so with the successful completion of one artifact, we can simply move on to pushing over the next artifact. To create the next step in the process, the publishVersionRecord function, I started out with a copy of the original processPhase6 function. Here is the end result:

publishVersionRecord: function(targetInstance, token, versionGR, targetAppId, attachmentId) {
	var canContinue = true;
	var payload = {};
	payload.member_application = targetAppId;
	payload.version = versionGR.getDisplayValue('version');
	var request  = new sn_ws.RESTMessageV2();
	request.setBasicAuth(this.WORKER_ROOT + targetInstance, token);
	request.setRequestHeader("Accept", "application/json");
	request.setHttpMethod('post');
	request.setEndpoint('https://' + targetInstance + '.service-now.com/api/now/table/x_11556_col_store_member_application_version');
	request.setRequestBody(JSON.stringify(payload, null, '\t'));
	response = request.execute();
	if (response.haveError()) {
		gs.error('CollaborationStoreUtils.publishVersionRecord: Error returned from Target instance ' + targetInstance + ': ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
		canContinue = false;
	} else if (response.getStatusCode() != 201) {
		gs.error('CollaborationStoreUtils.publishVersionRecord: Invalid HTTP Response Code returned from Target instance ' + targetInstance + ': ' + response.getStatusCode());
		canContinue = false;
	} else {
		jsonString = response.getBody();
		jsonObject = {};
		try {
			jsonObject = JSON.parse(jsonString);
		} catch (e) {
			gs.error('CollaborationStoreUtils.publishVersionRecord: Unparsable JSON string returned from Target instance ' + targetInstance + ': ' + jsonString);
			canContinue = false;
		}
		if (canContinue) {
			targetVerId = jsonObject.result.sys_id;
			this.publishVersionAttachment(targetInstance, token, targetVerId, attachmentId);
		}
	}
},

The last step in the process, then, is to send over the Update Set XML file attachment. For that one, I started out with the processPhase7 function and ended up with this:

publishVersionAttachment: function(targetInstance, token, targetVerId, attachmentId) {
	var gsa = new GlideSysAttachment();
	var sysAttGR = new GlideRecord('sys_attachment');
	if (sysAttGR.get(attachmentId)) {
		var url = 'https://';
		url += targetInstance;
		url += '.service-now.com/api/now/attachment/file?table_name=x_11556_col_store_member_application_version&table_sys_id=';
		url += targetVerId;
		url += '&file_name=';
		url += sysAttGR.getDisplayValue('file_name');
		var request  = new sn_ws.RESTMessageV2();
		request.setBasicAuth(this.WORKER_ROOT + targetInstance, token);
		request.setRequestHeader('Content-Type', sysAttGR.getDisplayValue('content_type'));
		request.setRequestHeader('Accept', 'application/json');
		request.setHttpMethod('post');
		request.setEndpoint(url);
		request.setRequestBody(gsa.getContent(sysAttGR));
		response = request.execute();
		if (response.haveError()) {
			gs.error('CollaborationStoreUtils.publishVersionAttachment: Error returned from Target instance ' + targetInstance + ': ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
		} else if (response.getStatusCode() != 201) {
			gs.error('CollaborationStoreUtils.publishVersionAttachment: Invalid HTTP Response Code returned from Target instance ' + targetInstance + ': ' + response.getStatusCode());
		}
	} else {
		gs.error('CollaborationStoreUtils.publishVersionAttachment: Invalid attachment record sys_id: ' + attachmentId);
	}
},

So that takes care of the functions. Now we have to create an Action in the Flow Designer that will call the function, and then produce a new Subflow that will leverage that action. That sounds like a good subject for our next installment.

Special Note to Testers

If you want to test the set-up process, you can do that with a single instance. You just have to select the Host option during the set-up process. If you want to test the Client set-up process, you will need at least two instances, one to serve as the Host, which you will need to reference when you set up the Client (you cannot be a Client of your own Host). But if you really want to test out the full registration process, including the Subflow that informs all existing instances of the new instance, you will need at least three participating instances: 1) the Host instance, 2) the new Client instance, and 3) an existing Client instance that will be notified of the new member of the community.

The same holds true for the application publishing process. You can test a portion of the publishing process by publishing an app on a single Host instance, but if you want to test out all of the parts and pieces, you will want to publish the app from a Client instance and make sure that it makes its way all the way back to the Host. Once we complete the application distribution process, full testing will require at least three instances to verify that the Host distributes the new version of the application to other Clients.

Collaboration Store, Part XIII

“First, solve the problem. Then, write the code.”
John Johnson

Today we need to build out the Subflow that we referenced in the Script Include function that we built last time out. To create a new Subflow, open up the Flow Designer, click on the New button to bring up the selection list, and select Subflow from the list.

Creating a new Subflow

When the initial form pops up, all you really need to enter is the name of the Subflow, which we already referred to in our Script Include function as New_Collaboration_Store_Instance.

New Subflow properties form

Once you submit the initial properties form, the new Subflow will appear in the list of Subflows, and from there you can bring it up in full edit mode. In edit mode, we can add the one Input to the Subflow, the name of the new instance.

Subflow Inputs and Outputs

Since we will be launching this Subflow to run on its own in the background, there is no need for any Outputs.

Once we configure the Inputs and Outputs, we can move on to the steps of the Subflow. Our first step will be to gather up all of the records in the table of instances except for two: the Host instance, which has already been updated, and the new instance, which already knows about itself.

Database query step

We select the Look Up Records Action, select the Member Organization table, and then define two Conditions to get the records that we want: 1) the host value is false, and 2) the instance is not the instance that we brought in as Input. It really doesn’t matter for our purposes what sequence the records come in, but I went ahead and sorted the records by instance, just so they we will always work through them in the same order.

Our next step, then is to loop through the records. We do that with a For Each Item Flow Control step, setting the items to the records obtained in the first step.

Looping through the retrieved records

Now we have the basic structure of the Subflow; we just need to perform the tasks necessary to notify each host of the new instance, and notify the new instance of each host within the the For Each Item loop. This could all be done with additional Subflow steps, but I took the easy way out here and just built another function in our Script Include to handle the REST API calls to the instances. In fact, I built two functions, one to make the call, and another to call that function twice, once to tell the existing host about the new host, and then again to inform the new host of the existing host. To run that script, I had to create a custom Action, which is just a simple Script Action that calls the function, passing in the names of the two instances (existing, from the current record, and new, from the Subflow input). Once I built the custom Action, I was able to select it from the list and then configure it.

Custom Script Action step

That completes the Subflow, but once again, we have referenced a function in our Script Include that does not exist, so we will have to get into that in our next installment.

Collaboration Store, Part XII

“The slogan ‘press on’ has solved and always will solve the problems of the human race.”
Calvin Coolidge

In the previous installment in this series, we created a new Scripted REST API Resource and referenced another nonexistent function in our Script Include. Now it is time to create that function, which will perform some of the work required to register a new client instance and then hand off the remaining tasks to an asynchronous Subflow so that the function can return the results without waiting for all of the other instances to be notified of the new client instance. The only thing to be done in the function will be to insert the new client instance into the database and kick off the Subflow. But before we do that, we need to first check to make sure that the Client has not already registered with the Host.

var result = {body: {error: {}, status: 'failure'}};

var mbrGR = new GlideRecord('x_11556_col_store_member_organization');
if (mbrGR.get('instance', data.instance)) {
	result.status = 400;
	result.body.error.message = 'Duplicate registration error';
	result.body.error.detail = 'This instance has already been registered with this store.';
} else {
	...

As we did before, we construct our result object with the expectation of failure, since there are more error conditions than the one successful path through the logic. In the case of an instance that has already been registered, we respond with a 400 Bad Request HTTP Response Code and accompanying error details. If the instance is not already in the database, then we attempt to insert it.

mbrGR.initialize();
mbrGR.name = data.name;
mbrGR.instance = data.instance;
mbrGR.email = data.email;
mbrGR.description = data.description;
mbrGR.token = data.sys_id;
mbrGR.active = true;
mbrGR.host = false;
mbrGR.accepted = new GlideDateTime();
if (mbrGR.insert()) {
	result.status = 202;
	delete result.body.error;
	result.body.info = {};
	result.body.info.message = 'Registration complete';
	result.body.info.detail = 'This instance has been successfully registered with this store.';
	result.body.status = 'success';
	...

If the new record was inserted successfully, then we response with a 202 Accepted HTTP Response Code, indicating that the registration was accepted, but the complete registration process (notifying all of the other instances) is not yet complete. At this point, all we have left to do is to initiate the Subflow to handle the rest of the process. We haven’t built the Subflow just yet, but for the purposes of this exercise, we can just assume that it is out there and then we can build it out later. There a couple of different ways to launch an asynchronous Subflow in the background, the original way, and the newer, preferred method. Both methods use the Scripted Flow API. Here is the way that we used to do this:

sn_fd.FlowAPI.startSubflow('New_Collaboration_Store_Instance', {new_instance: data.instance});

… and here is way that ServiceNow would like you to do it now:

sn_fd.FlowAPI.getRunner()
	.subflow('New_Collaboration_Store_Instance')
	.inBackground()
	.withInputs({new_instance: data.instance})
	.run();

Right now, both methods will work, and I’m still using the older, simpler way, but one day I’m going to need to switch over.

There should never be a problem inserting the new record, but just in case, we make that a conditional, and if for some reason it fails, we respond with a 500 Internal Server Error HTTP Response Code.

result.status = 500;
result.body.error.message = 'Internal server error';
result.body.error.detail = 'There was a problem processing this registration request.';

That’s it for all of the little parts and pieces. Here is the entire function, all put together.

processRegistrationRequest: function(data) {
	var result = {body: {error: {}, status: 'failure'}};

	var mbrGR = new GlideRecord('x_11556_col_store_member_organization');
	if (mbrGR.get('instance', data.instance)) {
		result.status = 400;
		result.body.error.message = 'Duplicate registration error';
		result.body.error.detail = 'This instance has already been registered with this store.';
	} else {
		mbrGR.initialize();
		mbrGR.name = data.name;
		mbrGR.instance = data.instance;
		mbrGR.email = data.email;
		mbrGR.description = data.description;
		mbrGR.token = data.sys_id;
		mbrGR.active = true;
		mbrGR.host = false;
		mbrGR.accepted = new GlideDateTime();
		if (mbrGR.insert()) {
			result.status = 202;
			delete result.body.error;
			result.body.info = {};
			result.body.info.message = 'Registration complete';
			result.body.info.detail = 'This instance has been successfully registered with this store.';
			result.body.status = 'success';
			sn_fd.FlowAPI.startSubflow('New_Collaboration_Store_Instance', {new_instance: data.instance});
		} else {
			result.status = 500;
			result.body.error.message = 'Internal server error';
			result.body.error.detail = 'There was a problem processing this registration request.';
		}
	}

	return result;
}

Now we have completed the nonexistent function that was referenced in the REST API Resource, but we have also now referenced a new nonexistent Subflow that we will need to build out before things are complete. That sounds like a good subject for our next installment.

Fun with Webhooks, Part VI

“The most exciting phrase to hear in science, the one that heralds discoveries, is not ‘Eureka!’ but ‘Now that’s funny …'”
Isaac Asimov

Even though we are not quite finished building out our new Subflow just yet, we have enough elements in place that we can try out what we have so far, and see if it actually works. The simplest way to do that would be to create a Webhook Registry that watches a single Incident, and then go make a qualifying change to that Incident and see what happens. Let’s go find an open Incident and then we can set up our registry.

In my instance, INC0000002 seemed like a likely candidate, so I pulled up the list of Webhook Registrations, clicked on the New button, and created the following new Webhook Registry entry to track changes to that specific Incident.

New Webhook Registry entry

For the URL, I used that same webhook.site address that I had used before so that I could see the result on the other end of the transaction. Once I had saved my new Webhook Registry entry, all I needed to do in order to test it out was to make a change to the Incident. For my first test, I simply changed the Assignment Group from Network to Hardware, which also then removed the Assigned to field value, since the original person specified in that field was not a member of the new Assignment Group. After making the change, I went over to webhook.site to see if anything showed up as a result of that modification. Sure, enough, a new POST had just been received by that site shortly after I saved the record.

New Webhook posting after updating an Incident

Although that proves that the process does indeed work as intended, I did notice one thing that is going to require a little bit of rework. Take a look at the second line of the message property:

Assigned To changed from Howard Johnson to  on INC0000002.

The Assigned to field changed from having a value to not having a value. The code that we have in our buildPayload function doesn’t really handle that circumstance in a way that produces the appropriate text to explain that in plain English. Here is the logic that creates that portion of the message:

if (current.assigned_to != previous.assigned_to) {
	payload.assigned_to = current.assigned_to;
	if (previous.assigned_to) {
		payload.message += separator + 'Assigned To changed from ' + previous.assigned_to + ' to ' + current.assigned_to + ' on ' + payload.id + '.';
	} else {
		payload.message += separator + 'Assigned To set to ' + current.assigned_to + ' on ' + payload.id + '.';
	}
	separator = '\n\n';
}

There is a check to see if the previous value was blank, but there is no check to see if the new value is blank. It wouldn’t take much to throw that in there, though, so let’s go ahead and do that now while we are thinking about it.

if (current.assigned_to != previous.assigned_to) {
	payload.assigned_to = current.assigned_to;
	if (previous.assigned_to) {
		if (current.assigned_to) {
			payload.message += separator + 'Assigned To changed from ' + previous.assigned_to + ' to ' + current.assigned_to + ' on ' + payload.id + '.';
		} else {
			payload.message += separator + 'Assigned To ' + previous.assigned_to + ' removed.';
		}
	} else {
		payload.message += separator + 'Assigned To set to ' + current.assigned_to + ' on ' + payload.id + '.';
	}
	separator = '\n\n';
}

That’s better. While we are at it, we might as well make the same change to the code for the Assignment group, as that is pretty much a line for line copy of the code for the Assigned to field. Everything else looks to be OK. The State is mandatory, so it will never have a new blank value, and the Comments and Work Notes fields are based on new entries and not on changes to existing entries, so if either of those are ever blank, we won’t be doing anything with them at all.

Still, other than that one little annoying language problem, the test was actually quite successful. Our Business Rule fired, the change was detected, our new Subflow was launched, our payload was generated, and the payload was successfully POSTed to the appropriate URL. All in all, I would have to say that that was a pretty good first test of the process. Obviously, we need to do a lot more testing, but this was a great start.

At this point, all I really wanted to do was to make sure that everything was working so far, which I think we accomplished. Before we do too much more testing, though, I think we need to go back and finish up the rest of the Subflow. We still have to add in the step that logs the activity, and we also have to add any error handling for when things don’t go as planned. Once we build all of that out, then we can do a lot more testing to validate the entire process from end to end. It was good to take a quick break and make sure that everything was working up to this point, but now that we know that it is, we need to get back to work on finishing up our development efforts.