Collaboration Store, Part XLVI

“Fundamentally you can’t do your own QA. it’s a question of seeing you own blind spots.”
Ron Avitzur

Last time, we wrapped up all of the coding for the process that will keep all of the Client instances in sync with the Host instance. I have been dragging this process out a little bit in the hopes of coming across a much better approach to the application installation process, as I am not all that happy with the solutions that have come to mind so far. But we have finally come to the end of the instance sync build, so it’s time to release another Update Set and do a little testing. After that, I’m going to have to plow ahead on the application installation process whether a new and better idea has revealed itself or not (so far, not!). And as long as we are going to be doing a little testing, we might as well do a little regression testing as well on the initial set-up process and the application publication process, just to make sure that we haven’t mangled something up with all of this additional code. It would be nice to know that everything still works.

As with our other recent releases, you will need three items, installed in the following order, to be able to give this all a spin:

  • If you have not done so already, you will need to install the most recent version of SNH Form Fields, which you can find here.
  • Once you have SNH Form Fields installed, you can install the new Scoped Application Update Set.
  • And then once the application has been installed, you can install the Update Set for the additional global components that could not be included in the Scoped Application.

If this is a new installation, then you can test out the initial set-up process when you set up your instance. You can find information on testing the set-up process here.

If this is a new installation, or if you have not yet published any applications, you will want to go through the process of publishing a few applications from as many of your instances as possible to fully test out the instance sync process. You can test the set-up process with a single Host instance, but to fully test the application publishing process or the instance sync process, you will need a few Client instances in your community as well. You can find information on testing the application publishing process here.

Assuming that you have successfully installed all of the parts, have successfully gone through the initial set-up process for your Host and Client instances, and have published a few apps from multiple sources, you are now ready to test the instance sync process. But how would we do that, you ask. Well, there are a number of things that can be done, starting with just making sure that the process runs on the Host instance on schedule and doesn’t do anything at all on any of the Client instances. The best way to verify that is to install the latest version everywhere and then go into the Flow Designer after 24 hours and pull up the flow and check out the execution history. If everything is working as it should, the process should run every day, and on the Host instance, there should be as many iterations of the loop as there are active Client instances in the community.

You don’t necessarily have to wait for the process to run on schedule, though, if you want to some more in-depth testing. You can use the Test button on the flow to run the process as many times as you would like to test out various scenarios. Here are just a a few of the things that you can try, just to see how things work out.

  • Delete one of the Client instance records on one of the Clients and then run the sync process to see if the Client and all of its applications have been restored on that instance.
  • Delete one of the application records on one of the Clients and then run the sync process to see if the application and all of its versions have been restored on that instance.
  • Delete one of the version records on one of the Clients and then run the sync process to see if the version and its attached Update Set XML file have been restored on that instance.
  • Delete an attached Update Set XML file on one of version records on one of the Clients and then run the sync process to see if the attachment has been restored on that instance.

You can also take a look at the System Logs to see the breadcrumbs left behind by the process to see if they are all accurate and meaningful. Any suggestions or helpful hints in this area would also be quite helpful. Basically, you just want to create a situation where one or more Client instances is out of sync with the Host, and then either let the process run as scheduled or manually kick start it in the Flow Designer, and then check to make sure that everything is back in sync again.

Once again, I would like to express my appreciation to all of those who have assisted with the testing in the past, and to anyone who has not participated in the past, but would like to join in, welcome to the party! You all have my sincere gratitude for your efforts, and as always, any and all comments are welcome and appreciated. If you tried everything out and were not able to uncover any of the bugs that are surely buried in there somewhere, let me know that as well. It is always welcome news to find out that things are working as they should! Thanks to all of you for your help.

Next time, if there are no test results to review, we will finally get into the application installation process.

Collaboration Store, Part XLV

“Progress isn’t made by early risers. It’s made by lazy men trying to find easier ways to do something.”
Robert Heinlein

Last time, we completed the code to sync up the version records, so now all we have left to do is to do basically the same thing for the Update Set XML files attached to the version records. The attachments are a slightly different animal, though, so we can’t just clone what we have done before as we were able to do when using the application record process as a basis for creating the version record process. For one thing, the relationship between instances and their applications is one-to-many, just as the relationship between applications and their various versions is one-to-many. Theoretically, there should only be one attached Update Set XML file for any given version. Technically, there is nothing stopping someone from attaching additional files to a version record, and in the future we may, in fact, want to attach things to a version record like release notes or user guides, but there should only be one Update Set for each version.

One thing that I ended up doing was to go back into the syncVersions function and add in the capability to send the sys_id of the version record to the syncAttachments function. Because the sys_attachments table uses both a table_name and a table_sys_id property to link to the attachment to its record, I couldn’t really do any dot-walking to come up with a query to find the right attachment record. So I added a versionIdList in addition to the versionList to capture the sys_id of the version record along with the version number. The updated version of the syncVersions function now looks like this:

syncVersions: function(targetGR, thisApplication, thisInstance, remoteAppId) {
	var versionList = [];
	var versionIdList = [];
	var versionGR = new GlideRecord('x_11556_col_store_member_application_version');
	versionGR.addQuery('member_application.name', thisApplication);
	versionGR.addQuery('member_application.provider.instance', thisInstance);
	versionGR.query();
	while (versionGR.next()) {
		versionList.push(versionGR.getDisplayValue('version'));
		versionIdList.push(versionGR.getUniqueValue());
	}
	if (versionList.length > 0) {
		var request  = new sn_ws.RESTMessageV2();
		request.setHttpMethod('get');
		request.setBasicAuth(this.CSU.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
		request.setRequestHeader("Accept", "application/json");
		request.setEndpoint('https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application_version?sysparm_fields=version%2Csys_id&sysparm_query=member_application%3D' + remoteAppId);
		var response = request.execute();
		if (response.haveError()) {
			gs.error('InstanceSyncUtils.syncVersions - Error returned from attempt to fetch version list: ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
		} else if (response.getStatusCode() == '200') {
			var jsonString = response.getBody();
			var jsonObject = {};
			try {
				jsonObject = JSON.parse(jsonString);
			} catch (e) {
				gs.error('InstanceSyncUtils.syncVersions - Unparsable JSON string returned from attempt to fetch version list: ' + jsonString);
			}
			if (Array.isArray(jsonObject.result)) {
				for (var i=0; i<versionList.length; i++) {
					var thisVersion = versionList[i];
					var thisVersionId = versionIdList[i];
					var remoteVerId = '';
					for (var j=0; j<jsonObject.result.length && remoteVerId == ''; j++) {
						if (jsonObject.result[j].version == thisVersion) {
							remoteVerId = jsonObject.result[j].sys_id;
						}
					}
					if (remoteVerId == '') {
						remoteVerId = this.sendVersion(targetGR, thisVersion, thisApplication, thisInstance, remoteAppId);
					}
					this.syncAttachments(targetGR, thisVersionId, thisVersion, thisApplication, thisInstance, remoteVerId);
				}
			} else {
				gs.error('InstanceSyncUtils.syncVersions - Invalid response body returned from attempt to fetch version list: ' + response.getBody());
			}
		} else {
			gs.error('InstanceSyncUtils.syncVersions - Invalid HTTP response code returned from attempt to fetch version list: ' + response.getStatusCode());
		}
	} else {
		gs.info('InstanceSyncUtils.syncVersions - No versions to sync for application ' + thisApplication + ' on instance ' + thisInstance);
	}
}

With the sys_id in hand, we could now build a simple query to fetch the attachment record.

var attachmentList = [];
var attachmentGR = new GlideRecord('sys_attachment');
attachmentGR.addQuery('table_name', 'x_11556_col_store_member_application_version');
attachmentGR.addQuery('table_sys_id', thisVersionId);
attachmentGR.addQuery('content_type', 'CONTAINS', 'xml');
attachmentGR.query();
while (attachmentGR.next()) {
	attachmentList.push(attachmentGR.getUniqueValue());
}

I’m still building up a list here, even though there should only be one XML attachment for each version record, but since I copied most of the code from the version function, I just left that in place. In practice, this should always be a list of one item. Assuming we have an attachment (which we always should!), we need to check the target instance to see if they already have it as well. For that, of course, we resort to making another REST API call.

var request  = new sn_ws.RESTMessageV2();
request.setHttpMethod('get');
request.setBasicAuth(this.CSU.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
request.setRequestHeader("Accept", "application/json");
request.setEndpoint('https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/sys_attachment?sysparm_fields=sys_id&sysparm_query=table_name%3Dx_11556_col_store_member_application_version%5Etable_sys_id%3D' + remoteVerId + '%5Econtent_typeCONTAINSxml');

Once we build the request, we execute it and check the results.

var response = request.execute();
if (response.haveError()) {
	gs.error('InstanceSyncUtils.syncAttachments - Error returned from attempt to fetch attachment list: ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
} else if (response.getStatusCode() == '200') {
	var jsonString = response.getBody();
	var jsonObject = {};
	try {
		jsonObject = JSON.parse(jsonString);
	} catch (e) {
		gs.error('InstanceSyncUtils.syncAttachments - Unparsable JSON string returned from attempt to fetch attachment list: ' + jsonString);
	}
	if (Array.isArray(jsonObject.result)) {
		if (jsonObject.result.length == 0) {
			this.sendAttachment(targetGR, attachmentList[0], remoteVerId, thisVersion, thisApplication);
		}
	} else {
		gs.error('InstanceSyncUtils.syncAttachments - Invalid response body returned from attempt to fetch attachment list: ' + response.getBody());
	}
} else {
	gs.error('InstanceSyncUtils.syncAttachments - Invalid HTTP response code returned from attempt to fetch attachment list: ' + response.getStatusCode());
}

All together, the new syncAttachments function looks like this:

syncAttachments: function(targetGR, thisVersionId, thisVersion, thisApplication, thisInstance, remoteVerId) {
	var attachmentList = [];
	var attachmentGR = new GlideRecord('sys_attachment');
	attachmentGR.addQuery('table_name', 'x_11556_col_store_member_application_version');
	attachmentGR.addQuery('table_sys_id', thisVersionId);
	attachmentGR.addQuery('content_type', 'CONTAINS', 'xml');
	attachmentGR.query();
	while (attachmentGR.next()) {
		attachmentList.push(attachmentGR.getUniqueValue());
	}
	if (attachmentList.length > 0) {
		var request  = new sn_ws.RESTMessageV2();
		request.setHttpMethod('get');
		request.setBasicAuth(this.CSU.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
		request.setRequestHeader("Accept", "application/json");
		request.setEndpoint('https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/sys_attachment?sysparm_fields=sys_id&sysparm_query=table_name%3Dx_11556_col_store_member_application_version%5Etable_sys_id%3D' + remoteVerId + '%5Econtent_typeCONTAINSxml');
		var response = request.execute();
		if (response.haveError()) {
			gs.error('InstanceSyncUtils.syncAttachments - Error returned from attempt to fetch attachment list: ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
		} else if (response.getStatusCode() == '200') {
			var jsonString = response.getBody();
			var jsonObject = {};
			try {
				jsonObject = JSON.parse(jsonString);
			} catch (e) {
				gs.error('InstanceSyncUtils.syncAttachments - Unparsable JSON string returned from attempt to fetch attachment list: ' + jsonString);
			}
			if (Array.isArray(jsonObject.result)) {
				if (jsonObject.result.length == 0) {
					this.sendAttachment(targetGR, attachmentList[0], remoteVerId, thisVersion, thisApplication);
				}
			} else {
				gs.error('InstanceSyncUtils.syncAttachments - Invalid response body returned from attempt to fetch attachment list: ' + response.getBody());
			}
		} else {
			gs.error('InstanceSyncUtils.syncAttachments - Invalid HTTP response code returned from attempt to fetch attachment list: ' + response.getStatusCode());
		}
	} else {
		gs.info('InstanceSyncUtils.syncAttachments - No attachments to sync for version ' + thisVersionId + ' of application ' + thisApplication + ' on instance ' + thisInstance);
	}
}

If there was no matching attachment on the target instance, we need to send it over, which we accomplish with a new sendAttachment function modeled after the earlier sendVersion function.

sendAttachment: function(targetGR, attachmentId, remoteVerId, thisVersion, thisApplication) {
	var attachmentGR = new GlideRecord('sys_attachment');
	if (attachmentGR.get(attachmentId)) {
		gs.info('InstanceSyncUtils.sendAttachment - Sending attachment ' + attachmentGR.getDisplayValue('file_name') + ' from version ' + thisVersion + ' of application ' + thisApplication + ' to target instance ' + targetGR.getDisplayValue('instance'));
		var result = this.CSU.pushAttachment(attachmentGR, targetGR, remoteVerId);
		if (result.error) {
			gs.error('InstanceSyncUtils.sendAttachment - Error occurred attempting to push attachment ' + attachmentGR.getDisplayValue('file_name') + ' from version ' + thisVersion + ' of application ' + thisApplication + ' to target instance ' + targetGR.getDisplayValue('instance') + ': ' + result.errorCode + '; ' + result.errorMessage);
		} else if (result.status != 200 && result.status != 201) {
			gs.error('InstanceSyncUtils.sendAttachment - Invalid HTTP response code returned from attempt to push attachment ' + attachmentGR.getDisplayValue('file_name') + ' from version ' + thisVersion + ' of application ' + thisApplication + ' to target instance ' + targetGR.getDisplayValue('instance') + ': ' + result.status);
		} else if (!result.obj) {
			gs.error('InstanceSyncUtils.sendAttachment - Invalid response body returned from attempt to push attachment ' + attachmentGR.getDisplayValue('file_name') + ' from version ' + thisVersion + ' of application ' + thisApplication + ' to target instance ' + targetGR.getDisplayValue('instance') + ': ' + result.body);
		}
	}
}

That completes all of the coding for the process of syncing all of the Client instances with the Host instance. If all is working correctly, it should ensure that all member instances have the same list of instances, the same list of applications from each contributing instance, the same list of versions for each application, and the associated Update Set XML file attachment for each version. Of course, there is still quite a bit of testing to do to make sure that all of this works as intended, so I will put together yet another Update Set and talk about how folks can help test all of this out in our next installment.

Collaboration Store, Part XLIV

“I am so clever that sometimes I don’t understand a single word of what I am saying.”
Oscar Wilde

Last time, we completed the shared functions in the CollaborationStoreUtils Script Include that will push the various record types from one instance to another. These functions will work to push data from a Client instance to the Host as well as push data from the Host to any of the Clients. Now that we have those in place, we can get back to the business of syncing up the Client instances with the Host. We have already built the code to sync up the list of instances, as well as the code to sync up the applications for each instance. Now we need to create the code to sync up the versions for each application. We will use the same approach that we used for all of the others, and much of the code will look very similar to that which we have already put into place.

Once again, we will start by gathering up all of the records associated with the active higher-level record, fetching the versions associated with the current application just as we previously gathered up all of the applications associated with the providing instance. Other than the table and query parameters, this should look quite familiar to those of you who have been following along at home.

var versionList = [];
var versionGR = new GlideRecord('x_11556_col_store_member_application_version');
versionGR.addQuery('member_application.name', thisApplication);
versionGR.addQuery('member_application.provider.instance', thisInstance);
versionGR.query();
while (versionGR.next()) {
	versionList.push(versionGR.getDisplayValue('version'));
}

If there are version records present on the Host, then we need to use the REST API to go gather up the same list from the target Client. As usual, we start out by building our RESTMessageV2 request object.

var request  = new sn_ws.RESTMessageV2();
request.setHttpMethod('get');
request.setBasicAuth(this.CSU.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
request.setRequestHeader("Accept", "application/json");
request.setEndpoint('https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application_version?sysparm_fields=version%2Csys_id&sysparm_query=member_application%3D' + remoteAppId);

Once we have the request constructed, we execute it and check the response.

var response = request.execute();
if (response.haveError()) {
	gs.error('InstanceSyncUtils.syncVersions - Error returned from attempt to fetch version list: ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
} else if (response.getStatusCode() == '200') {
	var jsonString = response.getBody();
	var jsonObject = {};
	try {
		jsonObject = JSON.parse(jsonString);
	} catch (e) {
		gs.error('InstanceSyncUtils.syncVersions - Unparsable JSON string returned from attempt to fetch version list: ' + jsonString);
	}
	if (Array.isArray(jsonObject.result)) {
		...
	} else {
		gs.error('InstanceSyncUtils.syncVersions - Invalid response body returned from attempt to fetch version list: ' + response.getBody());
	}
} else {
	gs.error('InstanceSyncUtils.syncVersions - Invalid HTTP response code returned from attempt to fetch version list: ' + response.getStatusCode());
}

Assuming all went well, the last thing that we need to do is to loop through all of the versions present on the Host and see if they are also present on the Client. If not, then we need to send them over, which we can now do with our new shared functions in CollaborationStoreUtils. And of course, whether we had to push the version over or not, we will need to check to see if the version record has the attached Update Set XML file.

for (var i=0; i<versionList.length; i++) {
	var thisVersion = versionList[i];
	var remoteVerId = '';
	for (var j=0; j<jsonObject.result.length && remoteVerId == ''; j++) {
		if (jsonObject.result[j].version == thisVersion) {
			remoteVerId = jsonObject.result[j].sys_id;
		}
	}
	if (remoteVerId == '') {
		remoteVerId = this.sendVersion(targetGR, thisApplication, thisInstance, thisVersion, remoteAppId);
	}
	this.syncAttachments(targetGR, thisVersion, remoteVerId);
}

Putting it all together, the entire function looks like this:

syncVersions: function(targetGR, thisApplication, thisInstance, remoteAppId) {
	var versionList = [];
	var versionGR = new GlideRecord('x_11556_col_store_member_application_version');
	versionGR.addQuery('member_application.name', thisApplication);
	versionGR.addQuery('member_application.provider.instance', thisInstance);
	versionGR.query();
	while (versionGR.next()) {
		versionList.push(versionGR.getDisplayValue('version'));
	}
	if (versionList.length > 0) {
		var request  = new sn_ws.RESTMessageV2();
		request.setHttpMethod('get');
		request.setBasicAuth(this.CSU.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
		request.setRequestHeader("Accept", "application/json");
		request.setEndpoint('https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application_version?sysparm_fields=version%2Csys_id&sysparm_query=member_application%3D' + remoteAppId);
		var response = request.execute();
		if (response.haveError()) {
			gs.error('InstanceSyncUtils.syncVersions - Error returned from attempt to fetch version list: ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
		} else if (response.getStatusCode() == '200') {
			var jsonString = response.getBody();
			var jsonObject = {};
			try {
				jsonObject = JSON.parse(jsonString);
			} catch (e) {
				gs.error('InstanceSyncUtils.syncVersions - Unparsable JSON string returned from attempt to fetch version list: ' + jsonString);
			}
			if (Array.isArray(jsonObject.result)) {
				for (var i=0; i<versionList.length; i++) {
					var thisVersion = versionList[i];
					var remoteVerId = '';
					for (var j=0; j<jsonObject.result.length && remoteVerId == ''; j++) {
						if (jsonObject.result[j].version == thisVersion) {
							remoteVerId = jsonObject.result[j].sys_id;
						}
					}
					if (remoteVerId == '') {
						remoteVerId = this.sendVersion(targetGR, thisApplication, thisInstance, thisVersion, remoteAppId);
					}
					this.syncAttachments(targetGR, thisVersion, remoteVerId);
				}
			} else {
				gs.error('InstanceSyncUtils.syncVersions - Invalid response body returned from attempt to fetch version list: ' + response.getBody());
			}
		} else {
			gs.error('InstanceSyncUtils.syncVersions - Invalid HTTP response code returned from attempt to fetch version list: ' + response.getStatusCode());
		}
	} else {
		gs.info('InstanceSyncUtils.syncVersions - No versions to sync for application ' + thisApplication + ' on instance ' + thisInstance);
	}
}

Now we have referenced two new functions that will need to built out as well, sendVersion and syncAttachments. The sendVersion function will be pretty much a clone of the sendApplication function that we already built, and it will call our new pushVersion function in CollaborationStoreUtils.

sendVersion: function(targetGR, thisApplication, thisInstance, thisVersion, remoteAppId) {
	var sysId = '';

	var versionGR = new GlideRecord('x_11556_col_store_member_application_version');
	versionGR.addQuery('member_application.name', thisApplication);
	versionGR.addQuery('member_application.provider.instance', thisInstance);
	versionGR.addQuery('version', thisVersion);
	versionGR.query();
	if (versionGR.next()) {
		gs.info('InstanceSyncUtils.sendApplication - Sending version ' + thisVersion + ' of application ' + versionGR.getDisplayValue('member_application') + ' to target instance ' + targetGR.getDisplayValue('instance'));
		var result = this.CSU.pushVersion(versionGR, targetGR, remoteAppId);
		if (result.error) {
			gs.error('InstanceSyncUtils.sendVersion - Error occurred attempting to push version ' + thisVersion + ' of application ' + remoteAppId + ' : ' + result.errorCode + '; ' + result.errorMessage);
		} else if (result.status != 200 && result.status != 201) {
			gs.error('InstanceSyncUtils.sendVersion - Invalid HTTP response code returned from attempt to push version ' + thisVersion + ' of application ' + remoteAppId + ' : ' + result.status);
		} else if (!result.obj) {
			gs.error('InstanceSyncUtils.sendVersion - Invalid response body returned from attempt to push version ' + thisVersion + ' of application ' + remoteAppId + ' : ' + result.body);
		} else {
			sysId = result.obj.sys_id;
		}
	}

	return sysId;
}

The other missing referenced function is syncAttachments. This one will be similar to the syncVersions function above, but because we will be working with attached files, there will be some minor differences in the tools that we will use to make all of that happen. We will get into all of that next time out.

Collaboration Store, Part XLIII

“You don’t start out writing good stuff. You start out writing crap and thinking it’s good stuff, and then gradually you get better at it. That’s why I say one of the most valuable traits is persistence.”
Octavia E. Butler

Last time, we built a function to send over an application record modeled after a similar function that we already had to send over and instance record. We can do the same thing now for a version record as well as the Update Set XML attachment. Since those are always inserts, we can skip the extra step of looking to see if the record is already there, and these functions will end looking an awful lot like the original function that sent over an instance record. Here is a completed function to send over a version record.

pushVersion: function(versionGR, targetGR, remoteAppId) {
	var result = {};

	var payload = {};
	payload.member_application = remoteAppId;
	payload.version = versionGR.getDisplayValue('version');
	payload.built_on = versionGR.getDisplayValue('built_on');
	result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application_version';
	result.method = 'POST';
	var request = new sn_ws.RESTMessageV2();
	request.setEndpoint(result.url);
	request.setHttpMethod(result.method);
	request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
	request.setRequestHeader('Content-Type', 'application/json');
	request.setRequestHeader('Accept', 'application/json');
	request.setRequestBody(JSON.stringify(payload, null, '\t'));
	var response = request.execute();
	result.status = response.getStatusCode();
	result.body = response.getBody();
	if (result.body) {
		try {
			result.obj = JSON.parse(result.body);
		} catch (e) {
			result.parse_error = e.toString();
		}
	}
	result.error = response.haveError();
	if (result.error) {
		result.error_code = response.getErrorCode();
		result.error_message = response.getErrorMessage();
	}

	return result;
}

And here is the completed function for sending over the Update Set XML file that is attached to the version.

pushAttachment: function(attachmentGR, targetGR, remoteVerId) {
	var result = {};

	var gsa = new GlideSysAttachment();
	result.url = 'https://';
	result.url += targetGR.getDisplayValue('instance');
	result.url += '.service-now.com/api/now/attachment/file?table_name=x_11556_col_store_member_application_version&table_sys_id=';
	result.url += remoteVerId;
	result.url += '&file_name=';
	result.url += attachmentGR.getDisplayValue('file_name');
	result.method = 'POST';
	var request = new sn_ws.RESTMessageV2();
	request.setEndpoint(result.url);
	request.setHttpMethod(result.method);
	request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
	request.setRequestHeader('Content-Type', attachmentGR.getDisplayValue('content_type'));
	request.setRequestHeader('Accept', 'application/json');
	request.setRequestBody(gsa.getContent(sysAttGR));
	var response = request.execute();
	result.status = response.getStatusCode();
	result.body = response.getBody();
	if (result.body) {
		try {
			result.obj = JSON.parse(result.body);
		} catch (e) {
			result.parse_error = e.toString();
		}
	}
	result.error = response.haveError();
	if (result.error) {
		result.error_code = response.getErrorCode();
		result.error_message = response.getErrorMessage();
	}

	return result;
}

All together, the four complementary functions in the CollaborationStoreUtils now look like this.

pushInstance: function(instanceGR, targetGR) {
	var result = {};

	var payload = {};
	payload.instance = instanceGR.getDisplayValue('instance');
	payload.name = instanceGR.getDisplayValue('name');
	payload.description = instanceGR.getDisplayValue('description');
	payload.email = instanceGR.getDisplayValue('email');
	payload.accepted = instanceGR.getDisplayValue('accepted');
	payload.active = instanceGR.getDisplayValue('active');
	payload.host = instanceGR.getDisplayValue('host');
	result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_organization';
	result.method = 'POST';
	var request = new sn_ws.RESTMessageV2();
	request.setEndpoint(result.url);
	request.setHttpMethod(result.method);
	request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
	request.setRequestHeader('Content-Type', 'application/json');
	request.setRequestHeader('Accept', 'application/json');
	request.setRequestBody(JSON.stringify(payload, null, '\t'));
	var response = request.execute();
	result.status = response.getStatusCode();
	result.body = response.getBody();
	if (result.body) {
		try {
			result.obj = JSON.parse(result.body);
		} catch (e) {
			result.parse_error = e.toString();
		}
	}
	result.error = response.haveError();
	if (result.error) {
		result.error_code = response.getErrorCode();
		result.error_message = response.getErrorMessage();
	}

	return result;
},

pushApplication: function(applicationGR, targetGR, remoteSysId) {
	var result = {};
 
	result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application?sysparm_fields=sys_id&sysparm_query=provider.instance%3D' + applicationGR.getDisplayValue('provider.instance') + '%5Ename%3D' + encodeURIComponent(applicationGR.getDisplayValue('name'));
	result.method = 'GET';
	var request = new sn_ws.RESTMessageV2();
	request.setEndpoint(result.url);
	request.setHttpMethod(result.method);
	request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
	request.setRequestHeader('Content-Type', 'application/json');
	request.setRequestHeader('Accept', 'application/json');
	var response = request.execute();
	result.status = response.getStatusCode();
	result.body = response.getBody();
	if (result.body) {
		try {
			result.obj = JSON.parse(result.body);
		} catch (e) {
			result.parse_error = e.toString();
		}
	}
	result.error = response.haveError();
	if (result.error) {
		result.error_code = response.getErrorCode();
		result.error_message = response.getErrorMessage();
	}
	if (!result.error && !result.parse_error && result.status == 200) {
		var remoteAppId = '';
		if (result.obj.result && result.obj.result.length > 0) {
			remoteAppId = result.obj.result[0].sys_id;
		}
		result = {};
		var payload = {};
		payload.name = applicationGR.getDisplayValue('name');
		payload.description = applicationGR.getDisplayValue('description');
		payload.current_version = applicationGR.getDisplayValue('current_version');
		payload.active = 'true';
		if (remoteAppId > '') {
			result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application/' + remoteAppId;
			result.method = 'PUT';
			result.remoteAppId = remoteAppId;
		} else {
			result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application';
			result.method = 'POST';
			payload.provider = remoteSysId;
		}
		request  = new sn_ws.RESTMessageV2();
		request.setEndpoint(result.url);
		request.setHttpMethod(result.method);
		request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
		request.setRequestHeader('Content-Type', 'application/json');
		request.setRequestHeader('Accept', 'application/json');
		request.setRequestBody(JSON.stringify(payload, null, '\t'));
		response = request.execute();
		result.status = response.getStatusCode();
		result.body = response.getBody();
		if (result.body) {
			try {
				result.obj = JSON.parse(result.body);
			} catch (e) {
				result.parse_error = e.toString();
			}
		}
		result.error = response.haveError();
		if (result.error) {
			result.error_code = response.getErrorCode();
			result.error_message = response.getErrorMessage();
		}
	}
 
	return result;
},

pushVersion: function(versionGR, targetGR, remoteAppId) {
	var result = {};

	var payload = {};
	payload.member_application = remoteAppId;
	payload.version = versionGR.getDisplayValue('version');
	payload.built_on = versionGR.getDisplayValue('built_on');
	result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application_version';
	result.method = 'POST';
	var request = new sn_ws.RESTMessageV2();
	request.setEndpoint(result.url);
	request.setHttpMethod(result.method);
	request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
	request.setRequestHeader('Content-Type', 'application/json');
	request.setRequestHeader('Accept', 'application/json');
	request.setRequestBody(JSON.stringify(payload, null, '\t'));
	var response = request.execute();
	result.status = response.getStatusCode();
	result.body = response.getBody();
	if (result.body) {
		try {
			result.obj = JSON.parse(result.body);
		} catch (e) {
			result.parse_error = e.toString();
		}
	}
	result.error = response.haveError();
	if (result.error) {
		result.error_code = response.getErrorCode();
		result.error_message = response.getErrorMessage();
	}

	return result;
},

pushAttachment: function(attachmentGR, targetGR, remoteVerId) {
	var result = {};

	var gsa = new GlideSysAttachment();
	result.url = 'https://';
	result.url += targetGR.getDisplayValue('instance');
	result.url += '.service-now.com/api/now/attachment/file?table_name=x_11556_col_store_member_application_version&table_sys_id=';
	result.url += remoteVerId;
	result.url += '&file_name=';
	result.url += attachmentGR.getDisplayValue('file_name');
	result.method = 'POST';
	var request = new sn_ws.RESTMessageV2();
	request.setEndpoint(result.url);
	request.setHttpMethod(result.method);
	request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
	request.setRequestHeader('Content-Type', attachmentGR.getDisplayValue('content_type'));
	request.setRequestHeader('Accept', 'application/json');
	request.setRequestBody(gsa.getContent(sysAttGR));
	var response = request.execute();
	result.status = response.getStatusCode();
	result.body = response.getBody();
	if (result.body) {
		try {
			result.obj = JSON.parse(result.body);
		} catch (e) {
			result.parse_error = e.toString();
		}
	}
	result.error = response.haveError();
	if (result.error) {
		result.error_code = response.getErrorCode();
		result.error_message = response.getErrorMessage();
	}

	return result;
}

Looking at them all side by side now, it appears that we could consolidate them even further by creating a single function that accepted a URL and a method, and maybe a payload, as that seems to be the only difference between them. I’m not really up for that at the moment, but it looks as if we could actually make that work one day when we had nothing better to do. For now, though, let’s just leave things as they are and get back to our InstanceSyncUtils and finish building out the code that will sync the version and attachment records in the same way that we did for the instances and the applications. These will utilize our new push functions when needed, but we still need to build out the logic that does the compare and determines what, if anything, needs to be sent over.

As you may recall, our strategy is to loop through the instances, and for each instance, loop through the applications provided by that instance. Similarly, for each application, we will loop through each version of that application, and for each version, we will make sure that there is an associated Update Set XML file attached. Whenever we find anything missing, we will utilize the appropriate push function above to send over the missing artifact. Since we have already done that for instances and applications, it should be relatively straightforward to copy that existing code and modify it as needed for the different types of records involved. Still, it is a little bit of work, so let’s save that for our next installment.

Collaboration Store, Part XLII

“You don’t get results by focusing on results. You get results by focusing on the actions that produce results.”
Mike Hawkins

Last time out, we threw together the code that will sync up the list of apps for a particular instance on the Host with the list of apps for that instance on the remote Client. We stopped short of building out the code that will send the missing applications over, though, and today we need to wrap that up. Those of you following along at home will recall that we already have a couple of functions that do that, and we really don’t want to create yet another one, so we need to see if we can somehow collapse those two into one that can serve the needs of all three use cases. The basic premise on the application updates is to first look and see if the application is already present on the target instance, and if so, update it; otherwise, insert it. Both of our existing functions take that approach, so we will want to maintain that functionality in the new shared version.

The other thing that we wanted to do was to model the existing publishApplication function’s approach to handling the interactions between the instance making the call and the instance that is being called. Basically, the shared code just makes the calls and returns the results in an object that can be evaluated by the caller. The shared code takes no action on any errors that might be encountered; it simply reports back to the calling module everything that happened. It is up to the caller to take whatever action might be necessary to respond to any issues reported. We will want to maintain that approach in our new function as well.

So, just as with the existing publishApplication function, we start out by creating the result object that we will be returning to the caller.

pushApplication: function(applicationGR, targetGR, remoteSysId) {
	var result = {};
 
	...
 
	return result;
}

Something that was not in the original version of the publishApplication function was the insertion of the URL and the method used in the result object. It occurred to me that this might be useful information to the caller, so I decided to add it into this one, and I think I will go back and add that into the original as well. For fetching the applications, which is the next thing that we need to do, that looks like this:

result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application?sysparm_fields=sys_id&sysparm_query=provider.instance%3D' + applicationGR.getDisplayValue('provider.instance') + '%5Ename%3D' + encodeURIComponent(applicationGR.getDisplayValue('name'));
result.method = 'GET';

Once we have set the value of those two properties in the result object, then we can use them to build our request object.

var request = new sn_ws.RESTMessageV2();
request.setEndpoint(result.url);
request.setHttpMethod(result.method);

And before we pull the trigger, we also have to set up the standard authentication and content headers.

request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
request.setRequestHeader('Content-Type', 'application/json');
request.setRequestHeader('Accept', 'application/json');

Once our request object is fully configured, then we just need to pull the trigger and complete our result object with the data and/or errors returned.

var response = request.execute();
result.status = response.getStatusCode();
result.body = response.getBody();
if (result.body) {
	try {
		result.obj = JSON.parse(result.body);
	} catch (e) {
		result.parse_error = e.toString();
	}
}
result.error = response.haveError();
if (result.error) {
	result.error_code = response.getErrorCode();
	result.error_message = response.getErrorMessage();
}

At this point, if there was any kind of error, then we just want to return that back to the caller so that they can take whatever action is appropriate for that particular context. Assuming everything worked as it should, though, the next thing that we are going to want to do is to see if the application was returned in the array of JSON objects sent back in response, and if so, snag the sys_id from the data.

if (!result.error && !result.parse_error && result.status == 200) {
	var remoteAppId = '';
	if (result.obj.result && result.obj.result.length > 0) {
		remoteAppId = result.obj.result[0].sys_id;
	}
	...
}

At this point, we are about to make an entirely new request of the target instance, so we want to reset the result object back to an empty object and start the building process all over again from scratch. The fetching of the application was successful (even if the application was not found on the target instance, finding that out was a success), so we do not need to retain anything from that activity other than the application’s sys_id on the target instance, if it happens to be there. Once we reset the result object, we can start building our payload to send over from the data on the application record.

result = {};
var payload = {};
payload.name = applicationGR.getDisplayValue('name');
payload.description = applicationGR.getDisplayValue('description');
payload.current_version = applicationGR.getDisplayValue('current_version');
payload.active = 'true';

At this point, we need to determine if we are doing an insert or an update based on the presence of a remote sys_id for this application.

if (remoteAppId > '') {
	result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application/' + remoteAppId;
	result.method = 'PUT';
	result.remoteAppId = remoteAppId;
} else {
	result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application';
	result.method = 'POST';
	payload.provider = remoteSysId;
}

Now we build our new request object, once again using the result.url and result.method along with all of the other standard elements of our other requests. Also, since this request will be shipping data over to the target instance, we will need to set the request body with the stringified payload.

request  = new sn_ws.RESTMessageV2();
request.setEndpoint(result.url);
request.setHttpMethod(result.method);
request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
request.setRequestHeader('Content-Type', 'application/json');
request.setRequestHeader('Accept', 'application/json');
request.setRequestBody(JSON.stringify(payload, null, '\t'));

Then, as before, we need to execute the request and populate our result object with the response.

response = request.execute();
result.status = response.getStatusCode();
result.body = response.getBody();
if (result.body) {
	try {
		result.obj = JSON.parse(result.body);
	} catch (e) {
		result.parse_error = e.toString();
	}
}
result.error = response.haveError();
if (result.error) {
	result.error_code = response.getErrorCode();
	result.error_message = response.getErrorMessage();
}

Once again, we do not take any action in response to any of these potential errors; we simply report them back to the caller in the result object and let them decide what, if anything, they want to do about it. These functions are not built to take action directly. They simply Observe and Report.

All together, this new function looks like this:

pushApplication: function(applicationGR, targetGR, remoteSysId) {
	var result = {};
 
	result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application?sysparm_fields=sys_id&sysparm_query=provider.instance%3D' + applicationGR.getDisplayValue('provider.instance') + '%5Ename%3D' + encodeURIComponent(applicationGR.getDisplayValue('name'));
	result.method = 'GET';
	var request = new sn_ws.RESTMessageV2();
	request.setEndpoint(result.url);
	request.setHttpMethod(result.method);
	request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
	request.setRequestHeader('Content-Type', 'application/json');
	request.setRequestHeader('Accept', 'application/json');
	var response = request.execute();
	result.status = response.getStatusCode();
	result.body = response.getBody();
	if (result.body) {
		try {
			result.obj = JSON.parse(result.body);
		} catch (e) {
			result.parse_error = e.toString();
		}
	}
	result.error = response.haveError();
	if (result.error) {
		result.error_code = response.getErrorCode();
		result.error_message = response.getErrorMessage();
	}
	if (!result.error && !result.parse_error && result.status == 200) {
		var remoteAppId = '';
		if (result.obj.result && result.obj.result.length > 0) {
			remoteAppId = result.obj.result[0].sys_id;
		}
		result = {};
		var payload = {};
		payload.name = applicationGR.getDisplayValue('name');
		payload.description = applicationGR.getDisplayValue('description');
		payload.current_version = applicationGR.getDisplayValue('current_version');
		payload.active = 'true';
		if (remoteAppId > '') {
			result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application/' + remoteAppId;
			result.method = 'PUT';
			result.remoteAppId = remoteAppId;
		} else {
			result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application';
			result.method = 'POST';
			payload.provider = remoteSysId;
		}
		request  = new sn_ws.RESTMessageV2();
		request.setEndpoint(result.url);
		request.setHttpMethod(result.method);
		request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
		request.setRequestHeader('Content-Type', 'application/json');
		request.setRequestHeader('Accept', 'application/json');
		request.setRequestBody(JSON.stringify(payload, null, '\t'));
		response = request.execute();
		result.status = response.getStatusCode();
		result.body = response.getBody();
		if (result.body) {
			try {
				result.obj = JSON.parse(result.body);
			} catch (e) {
				result.parse_error = e.toString();
			}
		}
		result.error = response.haveError();
		if (result.error) {
			result.error_code = response.getErrorCode();
			result.error_message = response.getErrorMessage();
		}
	}
 
	return result;
}

This should work now for our purpose, but if we want to also utilize this new function for the other two original purposes, we will need to refactor that code to call this function and then evaluate the response. Those actually work as they are right at the moment, so there is no rush to jump in and do that right now, but we don’t want to forget to take care of that one day, if for no other reason than to reduce the lines of code to be maintained.

That about wraps things up for the application records. Next time, we will see if we can do the same thing for the version records, and then we will jump into the Update Set attachments and that ought to wrap up this little side trip of creative avoidance that we took on so that we could ignore the issues with the application publishing for a while. One day, we need to get back into building out that third primary leg of this little stool, but now that we have headed down this intentional detour, we should finish this up first before we jump back onto the main road.

Collaboration Store, Part XLI

“If opportunity doesn’t knock, build a door.”
Milton Berle

Last time out we started work on the Script Include function that will sync up the data on a Client instance with that of the Host instance. We stopped short of consolidating a couple of cloned functions into a single shared function, but before we get to that effort, we need to add the code that syncs up the application table data. We already wrote the code that syncs up the instance table data, and since this operation is essentially the same process, we should be able lift most of the code from what we have already done.

As we did with the instances, we will need to gather up all of the applications on the Host associated with the instance being processed. This is basically the same process that we went through with the instance table, and the code will be very much the same except for the table and the query.

var applicationList = [];
var applicationGR = new GlideRecord('x_11556_col_store_member_application');
applicationGR.addQuery('provider.name', thisInstance);
applicationGR.query();
while (applicationGR.next()) {
	applicationList.push(applicationGR.getDisplayValue('name'));
}

We will also need to contact the Client instance and fetch all of the applications associated with the current instance being processed that are present on that system. Once again, we can turn to the REST API Explorer to generate an end point URL, and other than the actual URL, the request object needed for fetching the applications is again basically the same as that which we put together to fetch the instances from the Client.

var request  = new sn_ws.RESTMessageV2();
request.setHttpMethod('get');
request.setBasicAuth(this.CSU.WORKER_ROOT + instance, token);
request.setRequestHeader("Accept", "application/json");
request.setEndpoint('https://' + instance + '.service-now.com/api/now/table/x_11556_col_store_member_application?sysparm_fields=name%2Csys_id&sysparm_query=provider%3D' + remoteSysId);

Now that we have build our request object, we need to execute the request and check the results just as we did when gathered up the Client’s list of instances.

var response = request.execute();
if (response.haveError()) {
	gs.error('InstanceSyncUtils.syncApplications - Error returned from attempt to fetch application list: ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
} else if (response.getStatusCode() == '200') {
	var jsonString = response.getBody();
	var jsonObject = {};
	try {
		jsonObject = JSON.parse(jsonString);
	} catch (e) {
		gs.error('InstanceSyncUtils.syncApplications - Unparsable JSON string returned from attempt to fetch application list: ' + jsonString);
	}
	if (Array.isArray(jsonObject.result)) {
		...
	} else {
		gs.error('InstanceSyncUtils.syncApplications - Invalid response body returned from attempt to fetch application list: ' + response.getBody());
	}
} else {
	gs.error('InstanceSyncUtils.syncApplications - Invalid HTTP response code returned from attempt to fetch application list: ' + response.getStatusCode());
}

Assuming that all went well with gathering up all of the data on the Host and the Client, the next thing that we need to do is loop through all of the applications found on the Host for this instance and see if they are present on the Client as well. If not, we will need to push them over, which is where we will get into that code that we cloned that needs to be refactored into a single function so that we can have one process that can serve the two original purposes and our new purpose as well.

for (var i=0; i<applicationList.length; i++) {
	var thisApplication = applicationList[i];
	var remoteAppId = '';
	for (var j=0; j<jsonObject.result.length && remoteAppId == ''; j++) {
		if (jsonObject.result[j].name == thisApplication) {
			remoteAppId = jsonObject.result[j].sys_id;
		}
	}
	if (remoteAppId == '') {
		remoteAppId = this.sendApplication(targetGR, thisApplication, thisInstance, remoteSysId);
	}
	this.syncVersions(targetGR, thisApplication, remoteAppId);
}

This code references two functions that we have not yet created, sendApplication and syncVersions. We should be able to cut and paste the sendInstance function that we created earlier to build the sendApplication function with just a few minor modifications. Similarly, we should be able to clone the syncApplications function to serve as the initial basis of the syncVersions function. At this point, we are assuming that we have collapsed our two cloned functions that send applications over into a single function in the CollaborationStoreUtils Script Include modeled after the publishInstance function that we were using to send over an instance record, and we are assuming that it will return the very same type of object in response.

sendApplication: function(targetGR, thisApplication, thisInstance, remoteSysId) {
	var sysId = '';
	var applicationGR = new GlideRecord('x_11556_col_store_member_application');
	applicationGR.addQuery('name', thisApplication);
	applicationGR.addQuery('provider.name', thisInstance);
	applicationGR.query();
	if (applicationGR.next()) {
		var result = this.CSU.publishApplication(applicationGR, targetGR, remoteSysId);
		if (result.error) {
			gs.error('InstanceSyncUtils.sendApplication - Error occurred attempting to push application ' + thisApplication + ' : ' + result.errorCode + '; ' + result.errorMessage);
		} else if (result.status != 200) {
			gs.error('InstanceSyncUtils.sendApplication - Invalid HTTP response code returned from attempt to push application ' + thisApplication + ' : ' + result.status);
		} else if (!result.obj) {
			gs.error('InstanceSyncUtils.sendApplication - Invalid response body returned from attempt to push application ' + thisApplication + ' : ' + result.body);
		} else {
			sysId = result.obj.sys_id;
		}
	}
	return sysId;
}

Of course, for this to work, we actually have to create a function in CollaborationStoreUtils called publishApplication, but we already have two functions that do that very thing and a publishInstance function to use as a model. It shouldn’t take all that much to collapse all of that together into a single function that will work for everyone, so let’s make that the focus of our next installment.

Collaboration Store, Part XL

“If one does not know to which port one is sailing, no wind is favorable.”
Lucius Annaeus Seneca

Last time, we started building out the Script Include function that will do all of the heavy lifting for the process of periodically syncing up all of the Client instances with the Host. The next piece of code that we were going to tackle was the process of sending over a missing instance record to the Client. At the time, I was thinking that I had a couple of different versions of the code to do that already built out, one being cloned off of the other, but as it turns out, that is true for the application records, the version records, and the version attachments, but not the instance records. Currently, there is only one function that pushes over an instance record, and that function is called publishInstance and is found in the CollaborationStoreUtils Script Include.

publishInstance: function(instanceGR, targetGR) {
	var result = {};

	var payload = {};
	payload.instance = instanceGR.getDisplayValue('instance');
	payload.name = instanceGR.getDisplayValue('name');
	payload.description = instanceGR.getDisplayValue('description');
	payload.email = instanceGR.getDisplayValue('email');
	payload.accepted = instanceGR.getDisplayValue('accepted');
	payload.active = instanceGR.getDisplayValue('active');
	payload.host = instanceGR.getDisplayValue('host');
	var url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_organization';
	var request = new sn_ws.RESTMessageV2();
	request.setEndpoint(url);
	request.setHttpMethod('POST');
	request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
	request.setRequestHeader('Content-Type', 'application/json');
	request.setRequestHeader('Accept', 'application/json');
	request.setRequestBody(JSON.stringify(payload, null, '\t'));
	var response = request.execute();
	result.status = response.getStatusCode();
	result.body = response.getBody();
	if (result.body) {
		try {
			result.obj = JSON.parse(result.body);
		} catch (e) {
			result.parse_error = e.toString();
		}
	}
	result.error = response.haveError();
	if (result.error) {
		result.error_code = response.getErrorCode();
		result.error_message = response.getErrorMessage();
	}

	return result;
}

It takes two arguments, the instanceGR and the targetGR. In the current version of our new code, we do not have a GlideRecord for either the instance to be sent over or the instance to which the record is to be sent, but with a little refactoring, we can make that happen. To start with, I changed this code:

var instance = '';
var token = '';
var instanceList = [];
instanceGR = new GlideRecord('x_11556_col_store_member_organization');
instanceGR.addQuery('active', true);
instanceGR.query();
while (instanceGR.next()) {
    instanceList.push(instanceGR.getDisplayValue('name'));
    if (instanceGR.getUniqueValue() == instanceId) {
        instance = instanceGR.getDisplayValue('name');
        token = instanceGR.getDisplayValue('token');
    }
}

… to this:

var targetGR = new GlideRecord('x_11556_col_store_member_organization');
targetGR.get(instanceId);
var instance = targetGR.getDisplayValue('name');
var token = targetGR.getDisplayValue('token');
var instanceList = [];
var instanceGR = new GlideRecord('x_11556_col_store_member_organization');
instanceGR.addQuery('active', true);
instanceGR.query();
while (instanceGR.next()) {
	instanceList.push(instanceGR.getDisplayValue('name'));
}

This still gets me the target instance name and token as it did earlier, but it also gets me a GlideRecord for the target instance as well.

Then I changed this line:

remoteSysId = this.sendInstance(instance, token, thisInstance);

… to this:

remoteSysId = this.sendInstance(targetGR, thisInstance);

That took care of the refactoring. Now all I needed to do was to fetch a GlideRecord for the instance to be sent over, and then I could call the existing function in CollaborationStoreUtils and check the response.

sendInstance: function(targetGR, thisInstance) {
	var sysId = '';
	var instanceGR = new GlideRecord('x_11556_col_store_member_organization');
	if (instanceGR.get('name', thisInstance)) {
		var result = this.CSU.publishInstance(instanceGR, targetGR);
		if (result.error) {
			gs.error('InstanceSyncUtils.sendInstance - Error occurred attempting to push instance ' + thisInstance + ' : ' + result.errorCode + '; ' + result.errorMessage);
		} else if (result.status != 200) {
			gs.error('InstanceSyncUtils.sendInstance - Invalid HTTP response code returned from attempt to push instance ' + thisInstance + ' : ' + result.status);
		} else if (!result.obj) {
			gs.error('InstanceSyncUtils.sendInstance - Invalid response body returned from attempt to push instance ' + thisInstance + ' : ' + result.body);
		} else {
			sysId = result.obj.sys_id;
		}
	}
	return sysId;
}

Of course, I haven’t actually tested any of this as yet, but it looks like it ought to work out OK. If not, well then we will make a few adjustments.

The next thing on the list, then, is to build out the other referenced function that does not yet exist, the syncApplications function. Here is where we will eventually run into the need for that code that was cloned earlier, so we will want to clean all of that up finally, and the publishInstance function actually looks like a good model to use for the way in which it works, sending back all of the good and bad results in an object that can be evaluated by the caller where actions can be taken that are appropriate for the specific context.

Here is the original version of the code that sends over an application:

processPhase5: function(answer) {
	var mbrAppGR = new GlideRecord('x_11556_col_store_member_application');
	if (mbrAppGR.get(answer.mbrAppId)) {
		var host = gs.getProperty('x_11556_col_store.host_instance');
		var token = gs.getProperty('x_11556_col_store.active_token');
		var thisInstance = gs.getProperty('instance_name');
		var request  = new sn_ws.RESTMessageV2();
		request.setHttpMethod('get');
		request.setBasicAuth(this.WORKER_ROOT + host, token);
		request.setRequestHeader("Accept", "application/json");
		request.setEndpoint('https://' + host + '.service-now.com/api/now/table/x_11556_col_store_member_application?sysparm_fields=sys_id&sysparm_query=provider.instance%3D' + thisInstance + '%5Ename%3D' + encodeURIComponent(mbrAppGR.getDisplayValue('name')));
		var response = request.execute();
		if (response.haveError()) {
			answer = this.processError(answer, 'Error returned from Host instance: ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
		} else if (response.getStatusCode() == '200') {
			var jsonString = response.getBody();
			var jsonObject = {};
			try {
				jsonObject = JSON.parse(jsonString);
			} catch (e) {
				answer = this.processError(answer, 'Unparsable JSON string returned from Host instance: ' + jsonString);
			}
			if (!answer.error) {
				var payload = {};
				payload.name = mbrAppGR.getDisplayValue('name');
				payload.description = mbrAppGR.getDisplayValue('description');
				payload.current_version = mbrAppGR.getDisplayValue('current_version');
				payload.active = 'true';
				request  = new sn_ws.RESTMessageV2();
				request.setBasicAuth(this.WORKER_ROOT + host, token);
				request.setRequestHeader("Accept", "application/json");
				if (jsonObject.result && jsonObject.result.length > 0) {
					answer.hostAppId = jsonObject.result[0].sys_id;
					request.setHttpMethod('put');
					request.setEndpoint('https://' + host + '.service-now.com/api/now/table/x_11556_col_store_member_application/' + answer.hostAppId);
				} else {
					request.setHttpMethod('post');
					request.setEndpoint('https://' + host + '.service-now.com/api/now/table/x_11556_col_store_member_application');
					payload.provider = thisInstance;
				}
				request.setRequestBody(JSON.stringify(payload, null, '\t'));
				response = request.execute();
				if (response.haveError()) {
					answer = this.processError(answer, 'Error returned from Host instance: ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
				} else if (response.getStatusCode() != 200 && response.getStatusCode() != 201) {
					answer = this.processError(answer, 'Invalid HTTP Response Code returned from Host instance: ' + response.getStatusCode());
				} else {
					jsonString = response.getBody();
					jsonObject = {};
					try {
						jsonObject = JSON.parse(jsonString);
					} catch (e) {
						answer = this.processError(answer, 'Unparsable JSON string returned from Host instance: ' + jsonString);
					}
					if (!answer.error) {
						answer.hostAppId = jsonObject.result.sys_id;
					}
				}
			}
		} else {
			answer = this.processError(answer, 'Invalid HTTP Response Code returned from Host instance: ' + response.getStatusCode());
		}
	} else {
		answer = this.processError(answer, 'Invalid Member Application sys_id: ' + answer.appSysId);
	}

	return answer;
}

… and here is the other version that was cloned and hacked up based on the original:

publishNewVersion: function(newVersion, targetInstance, attachmentId) {
	var targetGR = new GlideRecord('x_11556_col_store_member_organization');
	if (targetGR.get('instance', targetInstance)) {
		var token = targetGR.getDisplayValue('token');
		var versionGR = new GlideRecord('x_11556_col_store_member_application_version');
		if (versionGR.get(newVersion)) {
			var canContinue = true;
			var targetAppId = '';
			var mbrAppGR = versionGR.member_application.getRefRecord();
			var request  = new sn_ws.RESTMessageV2();
			request.setHttpMethod('get');
			request.setBasicAuth(this.WORKER_ROOT + targetInstance, token);
			request.setRequestHeader("Accept", "application/json");
			request.setEndpoint('https://' + targetInstance + '.service-now.com/api/now/table/x_11556_col_store_member_application?sysparm_fields=sys_id&sysparm_query=provider.instance%3D' + mbrAppGR.getDisplayValue('provider.instance') + '%5Ename%3D' + encodeURIComponent(mbrAppGR.getDisplayValue('name')));
			var response = request.execute();
			if (response.haveError()) {
				gs.error('CollaborationStoreUtils.publishNewVersion: Error returned from Target instance ' + targetInstance + ': ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
				canContinue = false;
			} else if (response.getStatusCode() == '200') {
				var jsonString = response.getBody();
				var jsonObject = {};
				try {
					jsonObject = JSON.parse(jsonString);
				} catch (e) {
					gs.error('CollaborationStoreUtils.publishNewVersion: Unparsable JSON string returned from Target instance ' + targetInstance + ': ' + jsonString);
					canContinue = false;
				}
				if (canContinue) {
					var payload = {};
					payload.name = mbrAppGR.getDisplayValue('name');
					payload.description = mbrAppGR.getDisplayValue('description');
					payload.current_version = mbrAppGR.getDisplayValue('current_version');
					payload.active = 'true';
					request  = new sn_ws.RESTMessageV2();
					request.setBasicAuth(this.WORKER_ROOT + targetInstance, token);
					request.setRequestHeader("Accept", "application/json");
					if (jsonObject.result && jsonObject.result.length > 0) {
						targetAppId = jsonObject.result[0].sys_id;
						request.setHttpMethod('put');
						request.setEndpoint('https://' + targetInstance + '.service-now.com/api/now/table/x_11556_col_store_member_application/' + targetAppId);
					} else {
						request.setHttpMethod('post');
						request.setEndpoint('https://' + targetInstance + '.service-now.com/api/now/table/x_11556_col_store_member_application');
						payload.provider = mbrAppGR.getDisplayValue('provider.instance');
					}
					request.setRequestBody(JSON.stringify(payload, null, '\t'));
					response = request.execute();
					if (response.haveError()) {
						gs.error('CollaborationStoreUtils.publishNewVersion: Error returned from Target instance ' + targetInstance + ': ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
						canContinue = false;
					} else if (response.getStatusCode() != 200 && response.getStatusCode() != 201) {
						gs.error('CollaborationStoreUtils.publishNewVersion: Invalid HTTP Response Code returned from Target instance ' + targetInstance + ': ' + response.getStatusCode());
						canContinue = false;
					} else {
						jsonString = response.getBody();
						jsonObject = {};
						try {
							jsonObject = JSON.parse(jsonString);
						} catch (e) {
							gs.error('CollaborationStoreUtils.publishNewVersion: Unparsable JSON string returned from Target instance ' + targetInstance + ': ' + jsonString);
							canContinue = false;
						}
						if (canContinue) {
							targetAppId = jsonObject.result.sys_id;
						}
					}
				}
			} else {
				gs.error('CollaborationStoreUtils.publishNewVersion: Invalid HTTP Response Code returned from Target instance ' + targetInstance + ': ' + response.getStatusCode());
			}
			if (canContinue) {
				this.publishVersionRecord(targetInstance, token, versionGR, targetAppId, attachmentId);
			}
		} else {
			gs.error('CollaborationStoreUtils.publishNewVersion: Version record not found: ' + newVersion);
		}
	} else {
		gs.error('CollaborationStoreUtils.publishNewVersion: Target instance record not found: ' + targetInstance);
	}
}

Both functions accomplish the same thing, first checking to see if the application is already present on the remote instance, and then either adding it or updating it depending on whether or not it was already there. This is what we would want to do as well, so it seems as if we could collapse these two into a single, reusable function that would server our current purpose and these other two use cases as well. We could adopt the result object approach from the above publishInstance function and return the same form of object from our new function and then each user of the function could make their own choices as to what to do with the information returned. This is something that I have wanted to do for a while now, but it looks like it might be a little bit of work and some more refactoring of existing code, so let’s save all of that for our next installment.

Collaboration Store, Part XXXIX

“It’s not that I’m so smart, it’s just that I stay with problems longer.”
Albert Einstein

Last time, we built the Flow that will call our Script Include function to update all of the instances in the community that might be missing any artifacts. Now we need to get started on the actual function that will check in with each instance, gather its inventory of artifacts, compare it to that of the Host, and send over any missing items. As with most things, there are a number of ways to go about this, but my plan is to go through the instances first, and then work through the applications for each instance as each instance is being processed. Similarly, my intent is to work through the application versions as each application is being processed. To begin, we will need to fetch all of the instances found on the Host and stuff them into an array.

var instance = '';
var token = '';
var instanceList = [];
instanceGR = new GlideRecord('x_11556_col_store_member_organization');
instanceGR.addQuery('active', true);
instanceGR.query();
while (instanceGR.next()) {
	instanceList.push(instanceGR.getDisplayValue('name'));
	if (instanceGR.getUniqueValue() == instanceId) {
		instance = instanceGR.getDisplayValue('name');
		token = instanceGR.getDisplayValue('token');
	}
}

The other thing that this code accomplishes is to gather up the name and credentials for the target instance. The sys_id of the instance is passed to the function from the Flow, but to make the REST API calls, we need the actual name of the instance and the token. We could read that record directly before we started everything, but since we were spinning through the instance records anyway, I decided to just check each one and grab the data when we happened to be processing that particular instance. Of course, once we do that we need to check to make sure that we actually came across an instance with that sys_id before we go any further.

if (instance > '' && token > '') {
	this.fetchInstanceList(instance, token, instanceList);
} else {
	gs.error('InstanceSyncUtils.syncInstance - Requested instance not on file: ' + instanceId);
}

Assuming that we did come across the instance being synced, we now need to contact that instance and gather up all of the instances present on that system. For that, we can call on our old friend sn_ws.RESTMessageV2. But before we do that, we can use the REST API Explorer to come up with an end point URL that will get us the results that we need. We want to gather up all of the active instances from the Client instance, but we will only need a couple of the fields for our purpose here. We can specify those in the sysparm_fields parameter. I also like to alter the sysparm_limit parameter from the default of 1 to the alternative of 10, just to get more than one result in the output to help verify the query.

Using the REST API Explorer to generate an end point URL

Once we have entered all of the appropriate parameters, we can hit the Send button, which will produce the URL and also display some sample results in the Response Body section. After stripping off the server portion and removing the limitation parameter, we are left with the following value to use as our end point:

/api/now/table/x_11556_col_store_member_organization?sysparm_query=active%3Dtrue&sysparm_fields=name%2Csys_id

With our end point URL in hand, we can now build the request object.

fetchInstanceList: function(instance, token, instanceList) {
	var request = new sn_ws.RESTMessageV2();
	request.setHttpMethod('get');
	request.setBasicAuth(this.WORKER_ROOT + instance, token);
	request.setRequestHeader("Accept", "application/json");
	request.setEndpoint('https://' + instance + '.service-now.com/api/now/table/x_11556_col_store_member_organization?sysparm_query=active%3Dtrue&sysparm_fields=name%2Csys_id');
	...
}

Once the request object has been built, we can execute the request and check the results.

var response = request.execute();
if (response.haveError()) {
	gs.error('InstanceSyncUtils.syncInstance - Error returned from attempt to fetch instance list: ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
} else if (response.getStatusCode() == '200') {
	var jsonString = response.getBody();
	var jsonObject = {};
	try {
		jsonObject = JSON.parse(jsonString);
	} catch (e) {
		gs.error('InstanceSyncUtils.syncInstance - Unparsable JSON string returned from attempt to fetch instance list: ' + jsonString);
	}
	if (Array.isArray(jsonObject.result)) {
		...
	} else {
		gs.error('InstanceSyncUtils.syncInstance - Invalid response body returned from attempt to fetch instance list: ' + response.getStatusCode());
	}
} else {
	gs.error('InstanceSyncUtils.syncInstance - Invalid HTTP response code returned from attempt to fetch instance list: ' + response.getStatusCode());
}

Finally, if all has gone well, we can loop through the Host’s list of instances and then for every instance on the Host list, see if we can find it on the Client instance list. If we do not find it, then we need to push it over. Either way, once that has been checked and corrected if necessary, the next thing to do will be to check all of the applications present for that instance from the Host list.

for (var i=0; i<instanceList.length; i++) {
	var thisInstance = instanceList[i];
	var remoteSysId = '';
	for (var j=0; j<jsonObject.result.length && remoteSysId == ''; j++) {
		if (jsonObject.result[j].name == thisInstance) {
			remoteSysId = jsonObject.result[j].sys_id;
		}
	}
	if (remoteSysId == '') {
		remoteSysId = this.sendInstance(instance, token, thisInstance.name);
	}
	this.syncApplications(instance, token, thisInstance, remoteSysId);
}

We already have code to send over an instance record. In fact, we have already cloned that code and have a second version for another purpose. Rather than clone that code yet a third time, it’s long past the point for all of that to be consolidated into a single function that can work for any and all purposes. That seems like a bit of work, though, so let’s stop here and save that for our next time out.

Collaboration Store, Part XXXVIII

“You don’t have to be good to start … you just have to start to be good!”
Joe Sabah

Last time out we really did not accomplish anything of any consequence, but today let’s see if we can actually make some progress on something of real value. We need to create a process that will run every so often and check to make sure that all of the instances in the community have all of the latest content. We can use the Flow Designer to create and schedule a flow that will run daily on the Host instance and compare the artifacts present on each Client instance with the artifacts present on the Host, and then send over any missing items that never made it over originally for whatever reason. Doing it this way will eliminate the need to track and record errors when they happen, since we will just compare the Client inventory with that of the Host and correct any discrepancies. We don’t really need to know what failed or why.

We could make all of the REST API calls to the Client instances using the Integration Hub, but since that is a collection of optional products, not every instance will have all of those components installed and we would not want to make that a prerequisite for the product. To make this work on any ServiceNow configuration, we will want to roll our own calls via Javascript, and we will just use the Flow Designer to schedule a simple flow that loops through all of the Client instance records and then calls a Script Action that will do all of the heavy lifting. Before we build the Action though, we will want to build a Script Include function that can be called in the Action. Since this will be a significant script, we should probably create a new Script Include specific to this purpose rather than adding a number of new functions to any of our existing Script Includes. We can call our new Script Include InstanceSyncUtils and create a simple function called syncInstance with a single argument of the sys_id of the record for the instance to be synced. For now, we can just stub out the function with a simple logging statement and then circle back later to fill in the details. Right now, we just want to have something to call in our Flow Designer Action.

New Script Include function to be called in our new Action

With that out of the way, we can now fire up the Flow Designer and navigate to New -> Action. We will call our new Action Sync Instance and configure a single Input called Instance ID.

New Sync Instance Action with a single Input

For the Script step, we will just pass the Input to our new Script Include function:

(function execute(inputs, outputs) {
	var isu = new InstanceSyncUtils();
	isu.syncInstance(inputs.instance_id);
})(inputs, outputs);

There is currently nothing returned by the function, so there is no need to define any Outputs. At this point, all we need to do is to Save and Publish the Action before turning our attentions to the actual Flow itself.

To build a Flow, go back to the Home Page of the Flow Designer and select New -> Flow. On the resulting pop-up Flow Properties panel, enter all of the details for the Flow and set the Run As value to System User.

New Flow Properties

Since we want this Flow to run on its own every day, we can set the Trigger to Daily, and the Time to 12:30, which should run it in the middle of the lunch hour in the Host time zone, when most instances should be available.

Flow Trigger configuration

Since we only want this Flow to run on the Host instance, the first thing that we need to do is to create a Flow Variable that indicates whether or not this instance is the Host. To set the value of the variable, we add a Set Flow Variables step that uses the following script to determine if this instance is the Host based on System Properties.

return gs.getProperty('instance_name') == gs.getProperty('x_11556_col_store.host_instance');

The next step then will be a conditional that checks the value of the new Flow Variable.

Making sure that this is the Host instance

Failing this test will terminate the Flow, but if this is the Host instance, then the next thing that we need to do is gather up all of the Client instance records and then loop through them and call our new Action for each one in turn.

Find all records where Active is true and Host is false

Inside the following For Each Item loop, we can then invoke our new Action, passing in the sys_id of the current record.

Calling our new Action inside the For Each Item loop

That completes the work on the Flow, and all we need to do now is to Save and Activate the Flow. At this point, the Flow will kick off every day at 12:30 local time, but it won’t really do much except put a few entries in the System Log. The real work will be done in the Script Include, and since that’s a major effort in and of itself, we’ll leave that for our next exciting episode.

Collaboration Store, Part XXXVII

“Stay committed to your decisions, but stay flexible in your approach.”
Tony Robbins

After abandoning my earlier plans to jump into the application publishing process, I started to take a look at the missing error recovery needs for all of these inter-instance interactions. One thing that I had always planned to do at some point was to create an activity log of all transactions between instances. My idea was not to sync the logs across all instances, but to have a record in each instance of the things that went on with that particular instance. On the Host instance, such a log could provide the basis for some form of periodic recovery process that scanned the log for failed transactions and then made another attempt to put the transaction through again. After giving that some thought, though, I decided that a better approach would be to just scan each instance for instances, applications, and versions, and then attempt to push over anything that was missing compared to what was present on the Host. I still want to build an activity log at some point, but I don’t think that I want to use it as a basis for error recovery. I think it would be more straightforward to just compare the lists of records on each instance with the master lists on the Host and then try to correct any deficiencies.

That’s the plan today, anyway, but plans do have a way of changing over time, so who knows how things will come out once I start putting all of the pieces together. Right now, though, this seems like the better way to go.

One little thing that have been wanting to do for some time was to add another field to the version record to indicate the version of ServiceNow that was used to build the Update Set. There is a System Property that contains this information, so I all I really needed to do was to add the field and then add a line of code to the version record creation function to pull that value out of the property and stuff it into the new field. The name of the property is glide.buildtag.last, and the name of the field that I added is built_on.

New built_on field added to the version record

Once I added the new field to the version record, I opened up the ApplicationPublisher Script Include, which creates the version record, and added the following line to the function that builds the record:

versionGR.setValue('built_on', gs.getProperty('glide.buildtag.last'));

I also had to modify the function that sent the version record over to other instances to pass on the new value. Since I copied that code instead of consolidating the logic into a single, reusable function, I had to do that in two places, in the ApplicationPublisher and then again in the CollaborationStoreUtils (I really need to collapse that code into a single set of functions that will for both cases). In both places, I added the following line:

payload.built_on = versionGR.getDisplayValue('built_on');

This was not any kind of a major change, but it was something that I had been meaning to do for a long time, and so while I was in looking at the code, I decided to just go ahead and do it. Next time, we will focus on some real work to start building out some kind of error recovery process so that we can ensure that all of the instances in the community are always kept up to date, even if they miss an update in real time for whatever reason.