Tag: Update Set

Collaboration Store, Part XLVI

Posted on | by snhackery

“Fundamentally you can’t do your own QA. it’s a question of seeing you own blind spots.”
— Ron Avitzur

Last time, we wrapped up all of the coding for the process that will keep all of the Client instances in sync with the Host instance. I have been dragging this process out a little bit in the hopes of coming across a much better approach to the application installation process, as I am not all that happy with the solutions that have come to mind so far. But we have finally come to the end of the instance sync build, so it’s time to release another Update Set and do a little testing. After that, I’m going to have to plow ahead on the application installation process whether a new and better idea has revealed itself or not (so far, not!). And as long as we are going to be doing a little testing, we might as well do a little regression testing as well on the initial set-up process and the application publication process, just to make sure that we haven’t mangled something up with all of this additional code. It would be nice to know that everything still works.

As with our other recent releases, you will need three items, installed in the following order, to be able to give this all a spin:

  • If you have not done so already, you will need to install the most recent version of SNH Form Fields, which you can find here.
  • Once you have SNH Form Fields installed, you can install the new Scoped Application Update Set.
  • And then once the application has been installed, you can install the Update Set for the additional global components that could not be included in the Scoped Application.

If this is a new installation, then you can test out the initial set-up process when you set up your instance. You can find information on testing the set-up process here.

If this is a new installation, or if you have not yet published any applications, you will want to go through the process of publishing a few applications from as many of your instances as possible to fully test out the instance sync process. You can test the set-up process with a single Host instance, but to fully test the application publishing process or the instance sync process, you will need a few Client instances in your community as well. You can find information on testing the application publishing process here.

Assuming that you have successfully installed all of the parts, have successfully gone through the initial set-up process for your Host and Client instances, and have published a few apps from multiple sources, you are now ready to test the instance sync process. But how would we do that, you ask. Well, there are a number of things that can be done, starting with just making sure that the process runs on the Host instance on schedule and doesn’t do anything at all on any of the Client instances. The best way to verify that is to install the latest version everywhere and then go into the Flow Designer after 24 hours and pull up the flow and check out the execution history. If everything is working as it should, the process should run every day, and on the Host instance, there should be as many iterations of the loop as there are active Client instances in the community.

You don’t necessarily have to wait for the process to run on schedule, though, if you want to some more in-depth testing. You can use the Test button on the flow to run the process as many times as you would like to test out various scenarios. Here are just a a few of the things that you can try, just to see how things work out.

  • Delete one of the Client instance records on one of the Clients and then run the sync process to see if the Client and all of its applications have been restored on that instance.
  • Delete one of the application records on one of the Clients and then run the sync process to see if the application and all of its versions have been restored on that instance.
  • Delete one of the version records on one of the Clients and then run the sync process to see if the version and its attached Update Set XML file have been restored on that instance.
  • Delete an attached Update Set XML file on one of version records on one of the Clients and then run the sync process to see if the attachment has been restored on that instance.

You can also take a look at the System Logs to see the breadcrumbs left behind by the process to see if they are all accurate and meaningful. Any suggestions or helpful hints in this area would also be quite helpful. Basically, you just want to create a situation where one or more Client instances is out of sync with the Host, and then either let the process run as scheduled or manually kick start it in the Flow Designer, and then check to make sure that everything is back in sync again.

Once again, I would like to express my appreciation to all of those who have assisted with the testing in the past, and to anyone who has not participated in the past, but would like to join in, welcome to the party! You all have my sincere gratitude for your efforts, and as always, any and all comments are welcome and appreciated. If you tried everything out and were not able to uncover any of the bugs that are surely buried in there somewhere, let me know that as well. It is always welcome news to find out that things are working as they should! Thanks to all of you for your help.

Next time, if there are no test results to review, we will finally get into the application installation process.

Collaboration Store, Part XLV

Posted on | by snhackery

“Progress isn’t made by early risers. It’s made by lazy men trying to find easier ways to do something.”
— Robert Heinlein

Last time, we completed the code to sync up the version records, so now all we have left to do is to do basically the same thing for the Update Set XML files attached to the version records. The attachments are a slightly different animal, though, so we can’t just clone what we have done before as we were able to do when using the application record process as a basis for creating the version record process. For one thing, the relationship between instances and their applications is one-to-many, just as the relationship between applications and their various versions is one-to-many. Theoretically, there should only be one attached Update Set XML file for any given version. Technically, there is nothing stopping someone from attaching additional files to a version record, and in the future we may, in fact, want to attach things to a version record like release notes or user guides, but there should only be one Update Set for each version.

One thing that I ended up doing was to go back into the syncVersions function and add in the capability to send the sys_id of the version record to the syncAttachments function. Because the sys_attachments table uses both a table_name and a table_sys_id property to link to the attachment to its record, I couldn’t really do any dot-walking to come up with a query to find the right attachment record. So I added a versionIdList in addition to the versionList to capture the sys_id of the version record along with the version number. The updated version of the syncVersions function now looks like this:

syncVersions: function(targetGR, thisApplication, thisInstance, remoteAppId) {
	var versionList = [];
	var versionIdList = [];
	var versionGR = new GlideRecord('x_11556_col_store_member_application_version');
	versionGR.addQuery('member_application.name', thisApplication);
	versionGR.addQuery('member_application.provider.instance', thisInstance);
	versionGR.query();
	while (versionGR.next()) {
		versionList.push(versionGR.getDisplayValue('version'));
		versionIdList.push(versionGR.getUniqueValue());
	}
	if (versionList.length > 0) {
		var request  = new sn_ws.RESTMessageV2();
		request.setHttpMethod('get');
		request.setBasicAuth(this.CSU.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
		request.setRequestHeader("Accept", "application/json");
		request.setEndpoint('https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application_version?sysparm_fields=version%2Csys_id&sysparm_query=member_application%3D' + remoteAppId);
		var response = request.execute();
		if (response.haveError()) {
			gs.error('InstanceSyncUtils.syncVersions - Error returned from attempt to fetch version list: ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
		} else if (response.getStatusCode() == '200') {
			var jsonString = response.getBody();
			var jsonObject = {};
			try {
				jsonObject = JSON.parse(jsonString);
			} catch (e) {
				gs.error('InstanceSyncUtils.syncVersions - Unparsable JSON string returned from attempt to fetch version list: ' + jsonString);
			}
			if (Array.isArray(jsonObject.result)) {
				for (var i=0; i<versionList.length; i++) {
					var thisVersion = versionList[i];
					var thisVersionId = versionIdList[i];
					var remoteVerId = '';
					for (var j=0; j<jsonObject.result.length && remoteVerId == ''; j++) {
						if (jsonObject.result[j].version == thisVersion) {
							remoteVerId = jsonObject.result[j].sys_id;
						}
					}
					if (remoteVerId == '') {
						remoteVerId = this.sendVersion(targetGR, thisVersion, thisApplication, thisInstance, remoteAppId);
					}
					this.syncAttachments(targetGR, thisVersionId, thisVersion, thisApplication, thisInstance, remoteVerId);
				}
			} else {
				gs.error('InstanceSyncUtils.syncVersions - Invalid response body returned from attempt to fetch version list: ' + response.getBody());
			}
		} else {
			gs.error('InstanceSyncUtils.syncVersions - Invalid HTTP response code returned from attempt to fetch version list: ' + response.getStatusCode());
		}
	} else {
		gs.info('InstanceSyncUtils.syncVersions - No versions to sync for application ' + thisApplication + ' on instance ' + thisInstance);
	}
}

With the sys_id in hand, we could now build a simple query to fetch the attachment record.

var attachmentList = [];
var attachmentGR = new GlideRecord('sys_attachment');
attachmentGR.addQuery('table_name', 'x_11556_col_store_member_application_version');
attachmentGR.addQuery('table_sys_id', thisVersionId);
attachmentGR.addQuery('content_type', 'CONTAINS', 'xml');
attachmentGR.query();
while (attachmentGR.next()) {
	attachmentList.push(attachmentGR.getUniqueValue());
}

I’m still building up a list here, even though there should only be one XML attachment for each version record, but since I copied most of the code from the version function, I just left that in place. In practice, this should always be a list of one item. Assuming we have an attachment (which we always should!), we need to check the target instance to see if they already have it as well. For that, of course, we resort to making another REST API call.

var request  = new sn_ws.RESTMessageV2();
request.setHttpMethod('get');
request.setBasicAuth(this.CSU.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
request.setRequestHeader("Accept", "application/json");
request.setEndpoint('https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/sys_attachment?sysparm_fields=sys_id&sysparm_query=table_name%3Dx_11556_col_store_member_application_version%5Etable_sys_id%3D' + remoteVerId + '%5Econtent_typeCONTAINSxml');

Once we build the request, we execute it and check the results.

var response = request.execute();
if (response.haveError()) {
	gs.error('InstanceSyncUtils.syncAttachments - Error returned from attempt to fetch attachment list: ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
} else if (response.getStatusCode() == '200') {
	var jsonString = response.getBody();
	var jsonObject = {};
	try {
		jsonObject = JSON.parse(jsonString);
	} catch (e) {
		gs.error('InstanceSyncUtils.syncAttachments - Unparsable JSON string returned from attempt to fetch attachment list: ' + jsonString);
	}
	if (Array.isArray(jsonObject.result)) {
		if (jsonObject.result.length == 0) {
			this.sendAttachment(targetGR, attachmentList[0], remoteVerId, thisVersion, thisApplication);
		}
	} else {
		gs.error('InstanceSyncUtils.syncAttachments - Invalid response body returned from attempt to fetch attachment list: ' + response.getBody());
	}
} else {
	gs.error('InstanceSyncUtils.syncAttachments - Invalid HTTP response code returned from attempt to fetch attachment list: ' + response.getStatusCode());
}

All together, the new syncAttachments function looks like this:

syncAttachments: function(targetGR, thisVersionId, thisVersion, thisApplication, thisInstance, remoteVerId) {
	var attachmentList = [];
	var attachmentGR = new GlideRecord('sys_attachment');
	attachmentGR.addQuery('table_name', 'x_11556_col_store_member_application_version');
	attachmentGR.addQuery('table_sys_id', thisVersionId);
	attachmentGR.addQuery('content_type', 'CONTAINS', 'xml');
	attachmentGR.query();
	while (attachmentGR.next()) {
		attachmentList.push(attachmentGR.getUniqueValue());
	}
	if (attachmentList.length > 0) {
		var request  = new sn_ws.RESTMessageV2();
		request.setHttpMethod('get');
		request.setBasicAuth(this.CSU.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
		request.setRequestHeader("Accept", "application/json");
		request.setEndpoint('https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/sys_attachment?sysparm_fields=sys_id&sysparm_query=table_name%3Dx_11556_col_store_member_application_version%5Etable_sys_id%3D' + remoteVerId + '%5Econtent_typeCONTAINSxml');
		var response = request.execute();
		if (response.haveError()) {
			gs.error('InstanceSyncUtils.syncAttachments - Error returned from attempt to fetch attachment list: ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
		} else if (response.getStatusCode() == '200') {
			var jsonString = response.getBody();
			var jsonObject = {};
			try {
				jsonObject = JSON.parse(jsonString);
			} catch (e) {
				gs.error('InstanceSyncUtils.syncAttachments - Unparsable JSON string returned from attempt to fetch attachment list: ' + jsonString);
			}
			if (Array.isArray(jsonObject.result)) {
				if (jsonObject.result.length == 0) {
					this.sendAttachment(targetGR, attachmentList[0], remoteVerId, thisVersion, thisApplication);
				}
			} else {
				gs.error('InstanceSyncUtils.syncAttachments - Invalid response body returned from attempt to fetch attachment list: ' + response.getBody());
			}
		} else {
			gs.error('InstanceSyncUtils.syncAttachments - Invalid HTTP response code returned from attempt to fetch attachment list: ' + response.getStatusCode());
		}
	} else {
		gs.info('InstanceSyncUtils.syncAttachments - No attachments to sync for version ' + thisVersionId + ' of application ' + thisApplication + ' on instance ' + thisInstance);
	}
}

If there was no matching attachment on the target instance, we need to send it over, which we accomplish with a new sendAttachment function modeled after the earlier sendVersion function.

sendAttachment: function(targetGR, attachmentId, remoteVerId, thisVersion, thisApplication) {
	var attachmentGR = new GlideRecord('sys_attachment');
	if (attachmentGR.get(attachmentId)) {
		gs.info('InstanceSyncUtils.sendAttachment - Sending attachment ' + attachmentGR.getDisplayValue('file_name') + ' from version ' + thisVersion + ' of application ' + thisApplication + ' to target instance ' + targetGR.getDisplayValue('instance'));
		var result = this.CSU.pushAttachment(attachmentGR, targetGR, remoteVerId);
		if (result.error) {
			gs.error('InstanceSyncUtils.sendAttachment - Error occurred attempting to push attachment ' + attachmentGR.getDisplayValue('file_name') + ' from version ' + thisVersion + ' of application ' + thisApplication + ' to target instance ' + targetGR.getDisplayValue('instance') + ': ' + result.errorCode + '; ' + result.errorMessage);
		} else if (result.status != 200 && result.status != 201) {
			gs.error('InstanceSyncUtils.sendAttachment - Invalid HTTP response code returned from attempt to push attachment ' + attachmentGR.getDisplayValue('file_name') + ' from version ' + thisVersion + ' of application ' + thisApplication + ' to target instance ' + targetGR.getDisplayValue('instance') + ': ' + result.status);
		} else if (!result.obj) {
			gs.error('InstanceSyncUtils.sendAttachment - Invalid response body returned from attempt to push attachment ' + attachmentGR.getDisplayValue('file_name') + ' from version ' + thisVersion + ' of application ' + thisApplication + ' to target instance ' + targetGR.getDisplayValue('instance') + ': ' + result.body);
		}
	}
}

That completes all of the coding for the process of syncing all of the Client instances with the Host instance. If all is working correctly, it should ensure that all member instances have the same list of instances, the same list of applications from each contributing instance, the same list of versions for each application, and the associated Update Set XML file attachment for each version. Of course, there is still quite a bit of testing to do to make sure that all of this works as intended, so I will put together yet another Update Set and talk about how folks can help test all of this out in our next installment.

Collaboration Store, Part XLIV

Posted on | by snhackery

“I am so clever that sometimes I don’t understand a single word of what I am saying.”
— Oscar Wilde

Last time, we completed the shared functions in the CollaborationStoreUtils Script Include that will push the various record types from one instance to another. These functions will work to push data from a Client instance to the Host as well as push data from the Host to any of the Clients. Now that we have those in place, we can get back to the business of syncing up the Client instances with the Host. We have already built the code to sync up the list of instances, as well as the code to sync up the applications for each instance. Now we need to create the code to sync up the versions for each application. We will use the same approach that we used for all of the others, and much of the code will look very similar to that which we have already put into place.

Once again, we will start by gathering up all of the records associated with the active higher-level record, fetching the versions associated with the current application just as we previously gathered up all of the applications associated with the providing instance. Other than the table and query parameters, this should look quite familiar to those of you who have been following along at home.

var versionList = [];
var versionGR = new GlideRecord('x_11556_col_store_member_application_version');
versionGR.addQuery('member_application.name', thisApplication);
versionGR.addQuery('member_application.provider.instance', thisInstance);
versionGR.query();
while (versionGR.next()) {
	versionList.push(versionGR.getDisplayValue('version'));
}

If there are version records present on the Host, then we need to use the REST API to go gather up the same list from the target Client. As usual, we start out by building our RESTMessageV2 request object.

var request  = new sn_ws.RESTMessageV2();
request.setHttpMethod('get');
request.setBasicAuth(this.CSU.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
request.setRequestHeader("Accept", "application/json");
request.setEndpoint('https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application_version?sysparm_fields=version%2Csys_id&sysparm_query=member_application%3D' + remoteAppId);

Once we have the request constructed, we execute it and check the response.

var response = request.execute();
if (response.haveError()) {
	gs.error('InstanceSyncUtils.syncVersions - Error returned from attempt to fetch version list: ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
} else if (response.getStatusCode() == '200') {
	var jsonString = response.getBody();
	var jsonObject = {};
	try {
		jsonObject = JSON.parse(jsonString);
	} catch (e) {
		gs.error('InstanceSyncUtils.syncVersions - Unparsable JSON string returned from attempt to fetch version list: ' + jsonString);
	}
	if (Array.isArray(jsonObject.result)) {
		...
	} else {
		gs.error('InstanceSyncUtils.syncVersions - Invalid response body returned from attempt to fetch version list: ' + response.getBody());
	}
} else {
	gs.error('InstanceSyncUtils.syncVersions - Invalid HTTP response code returned from attempt to fetch version list: ' + response.getStatusCode());
}

Assuming all went well, the last thing that we need to do is to loop through all of the versions present on the Host and see if they are also present on the Client. If not, then we need to send them over, which we can now do with our new shared functions in CollaborationStoreUtils. And of course, whether we had to push the version over or not, we will need to check to see if the version record has the attached Update Set XML file.

for (var i=0; i<versionList.length; i++) {
	var thisVersion = versionList[i];
	var remoteVerId = '';
	for (var j=0; j<jsonObject.result.length && remoteVerId == ''; j++) {
		if (jsonObject.result[j].version == thisVersion) {
			remoteVerId = jsonObject.result[j].sys_id;
		}
	}
	if (remoteVerId == '') {
		remoteVerId = this.sendVersion(targetGR, thisApplication, thisInstance, thisVersion, remoteAppId);
	}
	this.syncAttachments(targetGR, thisVersion, remoteVerId);
}

Putting it all together, the entire function looks like this:

syncVersions: function(targetGR, thisApplication, thisInstance, remoteAppId) {
	var versionList = [];
	var versionGR = new GlideRecord('x_11556_col_store_member_application_version');
	versionGR.addQuery('member_application.name', thisApplication);
	versionGR.addQuery('member_application.provider.instance', thisInstance);
	versionGR.query();
	while (versionGR.next()) {
		versionList.push(versionGR.getDisplayValue('version'));
	}
	if (versionList.length > 0) {
		var request  = new sn_ws.RESTMessageV2();
		request.setHttpMethod('get');
		request.setBasicAuth(this.CSU.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
		request.setRequestHeader("Accept", "application/json");
		request.setEndpoint('https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application_version?sysparm_fields=version%2Csys_id&sysparm_query=member_application%3D' + remoteAppId);
		var response = request.execute();
		if (response.haveError()) {
			gs.error('InstanceSyncUtils.syncVersions - Error returned from attempt to fetch version list: ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
		} else if (response.getStatusCode() == '200') {
			var jsonString = response.getBody();
			var jsonObject = {};
			try {
				jsonObject = JSON.parse(jsonString);
			} catch (e) {
				gs.error('InstanceSyncUtils.syncVersions - Unparsable JSON string returned from attempt to fetch version list: ' + jsonString);
			}
			if (Array.isArray(jsonObject.result)) {
				for (var i=0; i<versionList.length; i++) {
					var thisVersion = versionList[i];
					var remoteVerId = '';
					for (var j=0; j<jsonObject.result.length && remoteVerId == ''; j++) {
						if (jsonObject.result[j].version == thisVersion) {
							remoteVerId = jsonObject.result[j].sys_id;
						}
					}
					if (remoteVerId == '') {
						remoteVerId = this.sendVersion(targetGR, thisApplication, thisInstance, thisVersion, remoteAppId);
					}
					this.syncAttachments(targetGR, thisVersion, remoteVerId);
				}
			} else {
				gs.error('InstanceSyncUtils.syncVersions - Invalid response body returned from attempt to fetch version list: ' + response.getBody());
			}
		} else {
			gs.error('InstanceSyncUtils.syncVersions - Invalid HTTP response code returned from attempt to fetch version list: ' + response.getStatusCode());
		}
	} else {
		gs.info('InstanceSyncUtils.syncVersions - No versions to sync for application ' + thisApplication + ' on instance ' + thisInstance);
	}
}

Now we have referenced two new functions that will need to built out as well, sendVersion and syncAttachments. The sendVersion function will be pretty much a clone of the sendApplication function that we already built, and it will call our new pushVersion function in CollaborationStoreUtils.

sendVersion: function(targetGR, thisApplication, thisInstance, thisVersion, remoteAppId) {
	var sysId = '';

	var versionGR = new GlideRecord('x_11556_col_store_member_application_version');
	versionGR.addQuery('member_application.name', thisApplication);
	versionGR.addQuery('member_application.provider.instance', thisInstance);
	versionGR.addQuery('version', thisVersion);
	versionGR.query();
	if (versionGR.next()) {
		gs.info('InstanceSyncUtils.sendApplication - Sending version ' + thisVersion + ' of application ' + versionGR.getDisplayValue('member_application') + ' to target instance ' + targetGR.getDisplayValue('instance'));
		var result = this.CSU.pushVersion(versionGR, targetGR, remoteAppId);
		if (result.error) {
			gs.error('InstanceSyncUtils.sendVersion - Error occurred attempting to push version ' + thisVersion + ' of application ' + remoteAppId + ' : ' + result.errorCode + '; ' + result.errorMessage);
		} else if (result.status != 200 && result.status != 201) {
			gs.error('InstanceSyncUtils.sendVersion - Invalid HTTP response code returned from attempt to push version ' + thisVersion + ' of application ' + remoteAppId + ' : ' + result.status);
		} else if (!result.obj) {
			gs.error('InstanceSyncUtils.sendVersion - Invalid response body returned from attempt to push version ' + thisVersion + ' of application ' + remoteAppId + ' : ' + result.body);
		} else {
			sysId = result.obj.sys_id;
		}
	}

	return sysId;
}

The other missing referenced function is syncAttachments. This one will be similar to the syncVersions function above, but because we will be working with attached files, there will be some minor differences in the tools that we will use to make all of that happen. We will get into all of that next time out.

Collaboration Store, Part XLIII

Posted on | by snhackery

“You don’t start out writing good stuff. You start out writing crap and thinking it’s good stuff, and then gradually you get better at it. That’s why I say one of the most valuable traits is persistence.”
— Octavia E. Butler

Last time, we built a function to send over an application record modeled after a similar function that we already had to send over and instance record. We can do the same thing now for a version record as well as the Update Set XML attachment. Since those are always inserts, we can skip the extra step of looking to see if the record is already there, and these functions will end looking an awful lot like the original function that sent over an instance record. Here is a completed function to send over a version record.

pushVersion: function(versionGR, targetGR, remoteAppId) {
	var result = {};

	var payload = {};
	payload.member_application = remoteAppId;
	payload.version = versionGR.getDisplayValue('version');
	payload.built_on = versionGR.getDisplayValue('built_on');
	result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application_version';
	result.method = 'POST';
	var request = new sn_ws.RESTMessageV2();
	request.setEndpoint(result.url);
	request.setHttpMethod(result.method);
	request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
	request.setRequestHeader('Content-Type', 'application/json');
	request.setRequestHeader('Accept', 'application/json');
	request.setRequestBody(JSON.stringify(payload, null, '\t'));
	var response = request.execute();
	result.status = response.getStatusCode();
	result.body = response.getBody();
	if (result.body) {
		try {
			result.obj = JSON.parse(result.body);
		} catch (e) {
			result.parse_error = e.toString();
		}
	}
	result.error = response.haveError();
	if (result.error) {
		result.error_code = response.getErrorCode();
		result.error_message = response.getErrorMessage();
	}

	return result;
}

And here is the completed function for sending over the Update Set XML file that is attached to the version.

pushAttachment: function(attachmentGR, targetGR, remoteVerId) {
	var result = {};

	var gsa = new GlideSysAttachment();
	result.url = 'https://';
	result.url += targetGR.getDisplayValue('instance');
	result.url += '.service-now.com/api/now/attachment/file?table_name=x_11556_col_store_member_application_version&table_sys_id=';
	result.url += remoteVerId;
	result.url += '&file_name=';
	result.url += attachmentGR.getDisplayValue('file_name');
	result.method = 'POST';
	var request = new sn_ws.RESTMessageV2();
	request.setEndpoint(result.url);
	request.setHttpMethod(result.method);
	request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
	request.setRequestHeader('Content-Type', attachmentGR.getDisplayValue('content_type'));
	request.setRequestHeader('Accept', 'application/json');
	request.setRequestBody(gsa.getContent(sysAttGR));
	var response = request.execute();
	result.status = response.getStatusCode();
	result.body = response.getBody();
	if (result.body) {
		try {
			result.obj = JSON.parse(result.body);
		} catch (e) {
			result.parse_error = e.toString();
		}
	}
	result.error = response.haveError();
	if (result.error) {
		result.error_code = response.getErrorCode();
		result.error_message = response.getErrorMessage();
	}

	return result;
}

All together, the four complementary functions in the CollaborationStoreUtils now look like this.

pushInstance: function(instanceGR, targetGR) {
	var result = {};

	var payload = {};
	payload.instance = instanceGR.getDisplayValue('instance');
	payload.name = instanceGR.getDisplayValue('name');
	payload.description = instanceGR.getDisplayValue('description');
	payload.email = instanceGR.getDisplayValue('email');
	payload.accepted = instanceGR.getDisplayValue('accepted');
	payload.active = instanceGR.getDisplayValue('active');
	payload.host = instanceGR.getDisplayValue('host');
	result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_organization';
	result.method = 'POST';
	var request = new sn_ws.RESTMessageV2();
	request.setEndpoint(result.url);
	request.setHttpMethod(result.method);
	request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
	request.setRequestHeader('Content-Type', 'application/json');
	request.setRequestHeader('Accept', 'application/json');
	request.setRequestBody(JSON.stringify(payload, null, '\t'));
	var response = request.execute();
	result.status = response.getStatusCode();
	result.body = response.getBody();
	if (result.body) {
		try {
			result.obj = JSON.parse(result.body);
		} catch (e) {
			result.parse_error = e.toString();
		}
	}
	result.error = response.haveError();
	if (result.error) {
		result.error_code = response.getErrorCode();
		result.error_message = response.getErrorMessage();
	}

	return result;
},

pushApplication: function(applicationGR, targetGR, remoteSysId) {
	var result = {};
 
	result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application?sysparm_fields=sys_id&sysparm_query=provider.instance%3D' + applicationGR.getDisplayValue('provider.instance') + '%5Ename%3D' + encodeURIComponent(applicationGR.getDisplayValue('name'));
	result.method = 'GET';
	var request = new sn_ws.RESTMessageV2();
	request.setEndpoint(result.url);
	request.setHttpMethod(result.method);
	request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
	request.setRequestHeader('Content-Type', 'application/json');
	request.setRequestHeader('Accept', 'application/json');
	var response = request.execute();
	result.status = response.getStatusCode();
	result.body = response.getBody();
	if (result.body) {
		try {
			result.obj = JSON.parse(result.body);
		} catch (e) {
			result.parse_error = e.toString();
		}
	}
	result.error = response.haveError();
	if (result.error) {
		result.error_code = response.getErrorCode();
		result.error_message = response.getErrorMessage();
	}
	if (!result.error && !result.parse_error && result.status == 200) {
		var remoteAppId = '';
		if (result.obj.result && result.obj.result.length > 0) {
			remoteAppId = result.obj.result[0].sys_id;
		}
		result = {};
		var payload = {};
		payload.name = applicationGR.getDisplayValue('name');
		payload.description = applicationGR.getDisplayValue('description');
		payload.current_version = applicationGR.getDisplayValue('current_version');
		payload.active = 'true';
		if (remoteAppId > '') {
			result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application/' + remoteAppId;
			result.method = 'PUT';
			result.remoteAppId = remoteAppId;
		} else {
			result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application';
			result.method = 'POST';
			payload.provider = remoteSysId;
		}
		request  = new sn_ws.RESTMessageV2();
		request.setEndpoint(result.url);
		request.setHttpMethod(result.method);
		request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
		request.setRequestHeader('Content-Type', 'application/json');
		request.setRequestHeader('Accept', 'application/json');
		request.setRequestBody(JSON.stringify(payload, null, '\t'));
		response = request.execute();
		result.status = response.getStatusCode();
		result.body = response.getBody();
		if (result.body) {
			try {
				result.obj = JSON.parse(result.body);
			} catch (e) {
				result.parse_error = e.toString();
			}
		}
		result.error = response.haveError();
		if (result.error) {
			result.error_code = response.getErrorCode();
			result.error_message = response.getErrorMessage();
		}
	}
 
	return result;
},

pushVersion: function(versionGR, targetGR, remoteAppId) {
	var result = {};

	var payload = {};
	payload.member_application = remoteAppId;
	payload.version = versionGR.getDisplayValue('version');
	payload.built_on = versionGR.getDisplayValue('built_on');
	result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application_version';
	result.method = 'POST';
	var request = new sn_ws.RESTMessageV2();
	request.setEndpoint(result.url);
	request.setHttpMethod(result.method);
	request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
	request.setRequestHeader('Content-Type', 'application/json');
	request.setRequestHeader('Accept', 'application/json');
	request.setRequestBody(JSON.stringify(payload, null, '\t'));
	var response = request.execute();
	result.status = response.getStatusCode();
	result.body = response.getBody();
	if (result.body) {
		try {
			result.obj = JSON.parse(result.body);
		} catch (e) {
			result.parse_error = e.toString();
		}
	}
	result.error = response.haveError();
	if (result.error) {
		result.error_code = response.getErrorCode();
		result.error_message = response.getErrorMessage();
	}

	return result;
},

pushAttachment: function(attachmentGR, targetGR, remoteVerId) {
	var result = {};

	var gsa = new GlideSysAttachment();
	result.url = 'https://';
	result.url += targetGR.getDisplayValue('instance');
	result.url += '.service-now.com/api/now/attachment/file?table_name=x_11556_col_store_member_application_version&table_sys_id=';
	result.url += remoteVerId;
	result.url += '&file_name=';
	result.url += attachmentGR.getDisplayValue('file_name');
	result.method = 'POST';
	var request = new sn_ws.RESTMessageV2();
	request.setEndpoint(result.url);
	request.setHttpMethod(result.method);
	request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
	request.setRequestHeader('Content-Type', attachmentGR.getDisplayValue('content_type'));
	request.setRequestHeader('Accept', 'application/json');
	request.setRequestBody(gsa.getContent(sysAttGR));
	var response = request.execute();
	result.status = response.getStatusCode();
	result.body = response.getBody();
	if (result.body) {
		try {
			result.obj = JSON.parse(result.body);
		} catch (e) {
			result.parse_error = e.toString();
		}
	}
	result.error = response.haveError();
	if (result.error) {
		result.error_code = response.getErrorCode();
		result.error_message = response.getErrorMessage();
	}

	return result;
}

Looking at them all side by side now, it appears that we could consolidate them even further by creating a single function that accepted a URL and a method, and maybe a payload, as that seems to be the only difference between them. I’m not really up for that at the moment, but it looks as if we could actually make that work one day when we had nothing better to do. For now, though, let’s just leave things as they are and get back to our InstanceSyncUtils and finish building out the code that will sync the version and attachment records in the same way that we did for the instances and the applications. These will utilize our new push functions when needed, but we still need to build out the logic that does the compare and determines what, if anything, needs to be sent over.

As you may recall, our strategy is to loop through the instances, and for each instance, loop through the applications provided by that instance. Similarly, for each application, we will loop through each version of that application, and for each version, we will make sure that there is an associated Update Set XML file attached. Whenever we find anything missing, we will utilize the appropriate push function above to send over the missing artifact. Since we have already done that for instances and applications, it should be relatively straightforward to copy that existing code and modify it as needed for the different types of records involved. Still, it is a little bit of work, so let’s save that for our next installment.

Collaboration Store, Part XLII

Posted on | by snhackery

“You don’t get results by focusing on results. You get results by focusing on the actions that produce results.”
— Mike Hawkins

Last time out, we threw together the code that will sync up the list of apps for a particular instance on the Host with the list of apps for that instance on the remote Client. We stopped short of building out the code that will send the missing applications over, though, and today we need to wrap that up. Those of you following along at home will recall that we already have a couple of functions that do that, and we really don’t want to create yet another one, so we need to see if we can somehow collapse those two into one that can serve the needs of all three use cases. The basic premise on the application updates is to first look and see if the application is already present on the target instance, and if so, update it; otherwise, insert it. Both of our existing functions take that approach, so we will want to maintain that functionality in the new shared version.

The other thing that we wanted to do was to model the existing publishApplication function’s approach to handling the interactions between the instance making the call and the instance that is being called. Basically, the shared code just makes the calls and returns the results in an object that can be evaluated by the caller. The shared code takes no action on any errors that might be encountered; it simply reports back to the calling module everything that happened. It is up to the caller to take whatever action might be necessary to respond to any issues reported. We will want to maintain that approach in our new function as well.

So, just as with the existing publishApplication function, we start out by creating the result object that we will be returning to the caller.

pushApplication: function(applicationGR, targetGR, remoteSysId) {
	var result = {};
 
	...
 
	return result;
}

Something that was not in the original version of the publishApplication function was the insertion of the URL and the method used in the result object. It occurred to me that this might be useful information to the caller, so I decided to add it into this one, and I think I will go back and add that into the original as well. For fetching the applications, which is the next thing that we need to do, that looks like this:

result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application?sysparm_fields=sys_id&sysparm_query=provider.instance%3D' + applicationGR.getDisplayValue('provider.instance') + '%5Ename%3D' + encodeURIComponent(applicationGR.getDisplayValue('name'));
result.method = 'GET';

Once we have set the value of those two properties in the result object, then we can use them to build our request object.

var request = new sn_ws.RESTMessageV2();
request.setEndpoint(result.url);
request.setHttpMethod(result.method);

And before we pull the trigger, we also have to set up the standard authentication and content headers.

request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
request.setRequestHeader('Content-Type', 'application/json');
request.setRequestHeader('Accept', 'application/json');

Once our request object is fully configured, then we just need to pull the trigger and complete our result object with the data and/or errors returned.

var response = request.execute();
result.status = response.getStatusCode();
result.body = response.getBody();
if (result.body) {
	try {
		result.obj = JSON.parse(result.body);
	} catch (e) {
		result.parse_error = e.toString();
	}
}
result.error = response.haveError();
if (result.error) {
	result.error_code = response.getErrorCode();
	result.error_message = response.getErrorMessage();
}

At this point, if there was any kind of error, then we just want to return that back to the caller so that they can take whatever action is appropriate for that particular context. Assuming everything worked as it should, though, the next thing that we are going to want to do is to see if the application was returned in the array of JSON objects sent back in response, and if so, snag the sys_id from the data.

if (!result.error && !result.parse_error && result.status == 200) {
	var remoteAppId = '';
	if (result.obj.result && result.obj.result.length > 0) {
		remoteAppId = result.obj.result[0].sys_id;
	}
	...
}

At this point, we are about to make an entirely new request of the target instance, so we want to reset the result object back to an empty object and start the building process all over again from scratch. The fetching of the application was successful (even if the application was not found on the target instance, finding that out was a success), so we do not need to retain anything from that activity other than the application’s sys_id on the target instance, if it happens to be there. Once we reset the result object, we can start building our payload to send over from the data on the application record.

result = {};
var payload = {};
payload.name = applicationGR.getDisplayValue('name');
payload.description = applicationGR.getDisplayValue('description');
payload.current_version = applicationGR.getDisplayValue('current_version');
payload.active = 'true';

At this point, we need to determine if we are doing an insert or an update based on the presence of a remote sys_id for this application.

if (remoteAppId > '') {
	result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application/' + remoteAppId;
	result.method = 'PUT';
	result.remoteAppId = remoteAppId;
} else {
	result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application';
	result.method = 'POST';
	payload.provider = remoteSysId;
}

Now we build our new request object, once again using the result.url and result.method along with all of the other standard elements of our other requests. Also, since this request will be shipping data over to the target instance, we will need to set the request body with the stringified payload.

request  = new sn_ws.RESTMessageV2();
request.setEndpoint(result.url);
request.setHttpMethod(result.method);
request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
request.setRequestHeader('Content-Type', 'application/json');
request.setRequestHeader('Accept', 'application/json');
request.setRequestBody(JSON.stringify(payload, null, '\t'));

Then, as before, we need to execute the request and populate our result object with the response.

response = request.execute();
result.status = response.getStatusCode();
result.body = response.getBody();
if (result.body) {
	try {
		result.obj = JSON.parse(result.body);
	} catch (e) {
		result.parse_error = e.toString();
	}
}
result.error = response.haveError();
if (result.error) {
	result.error_code = response.getErrorCode();
	result.error_message = response.getErrorMessage();
}

Once again, we do not take any action in response to any of these potential errors; we simply report them back to the caller in the result object and let them decide what, if anything, they want to do about it. These functions are not built to take action directly. They simply Observe and Report.

All together, this new function looks like this:

pushApplication: function(applicationGR, targetGR, remoteSysId) {
	var result = {};
 
	result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application?sysparm_fields=sys_id&sysparm_query=provider.instance%3D' + applicationGR.getDisplayValue('provider.instance') + '%5Ename%3D' + encodeURIComponent(applicationGR.getDisplayValue('name'));
	result.method = 'GET';
	var request = new sn_ws.RESTMessageV2();
	request.setEndpoint(result.url);
	request.setHttpMethod(result.method);
	request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
	request.setRequestHeader('Content-Type', 'application/json');
	request.setRequestHeader('Accept', 'application/json');
	var response = request.execute();
	result.status = response.getStatusCode();
	result.body = response.getBody();
	if (result.body) {
		try {
			result.obj = JSON.parse(result.body);
		} catch (e) {
			result.parse_error = e.toString();
		}
	}
	result.error = response.haveError();
	if (result.error) {
		result.error_code = response.getErrorCode();
		result.error_message = response.getErrorMessage();
	}
	if (!result.error && !result.parse_error && result.status == 200) {
		var remoteAppId = '';
		if (result.obj.result && result.obj.result.length > 0) {
			remoteAppId = result.obj.result[0].sys_id;
		}
		result = {};
		var payload = {};
		payload.name = applicationGR.getDisplayValue('name');
		payload.description = applicationGR.getDisplayValue('description');
		payload.current_version = applicationGR.getDisplayValue('current_version');
		payload.active = 'true';
		if (remoteAppId > '') {
			result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application/' + remoteAppId;
			result.method = 'PUT';
			result.remoteAppId = remoteAppId;
		} else {
			result.url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_application';
			result.method = 'POST';
			payload.provider = remoteSysId;
		}
		request  = new sn_ws.RESTMessageV2();
		request.setEndpoint(result.url);
		request.setHttpMethod(result.method);
		request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
		request.setRequestHeader('Content-Type', 'application/json');
		request.setRequestHeader('Accept', 'application/json');
		request.setRequestBody(JSON.stringify(payload, null, '\t'));
		response = request.execute();
		result.status = response.getStatusCode();
		result.body = response.getBody();
		if (result.body) {
			try {
				result.obj = JSON.parse(result.body);
			} catch (e) {
				result.parse_error = e.toString();
			}
		}
		result.error = response.haveError();
		if (result.error) {
			result.error_code = response.getErrorCode();
			result.error_message = response.getErrorMessage();
		}
	}
 
	return result;
}

This should work now for our purpose, but if we want to also utilize this new function for the other two original purposes, we will need to refactor that code to call this function and then evaluate the response. Those actually work as they are right at the moment, so there is no rush to jump in and do that right now, but we don’t want to forget to take care of that one day, if for no other reason than to reduce the lines of code to be maintained.

That about wraps things up for the application records. Next time, we will see if we can do the same thing for the version records, and then we will jump into the Update Set attachments and that ought to wrap up this little side trip of creative avoidance that we took on so that we could ignore the issues with the application publishing for a while. One day, we need to get back into building out that third primary leg of this little stool, but now that we have headed down this intentional detour, we should finish this up first before we jump back onto the main road.

Collaboration Store, Part XXXVII

Posted on | by snhackery

“Stay committed to your decisions, but stay flexible in your approach.”
— Tony Robbins

After abandoning my earlier plans to jump into the application publishing process, I started to take a look at the missing error recovery needs for all of these inter-instance interactions. One thing that I had always planned to do at some point was to create an activity log of all transactions between instances. My idea was not to sync the logs across all instances, but to have a record in each instance of the things that went on with that particular instance. On the Host instance, such a log could provide the basis for some form of periodic recovery process that scanned the log for failed transactions and then made another attempt to put the transaction through again. After giving that some thought, though, I decided that a better approach would be to just scan each instance for instances, applications, and versions, and then attempt to push over anything that was missing compared to what was present on the Host. I still want to build an activity log at some point, but I don’t think that I want to use it as a basis for error recovery. I think it would be more straightforward to just compare the lists of records on each instance with the master lists on the Host and then try to correct any deficiencies.

That’s the plan today, anyway, but plans do have a way of changing over time, so who knows how things will come out once I start putting all of the pieces together. Right now, though, this seems like the better way to go.

One little thing that have been wanting to do for some time was to add another field to the version record to indicate the version of ServiceNow that was used to build the Update Set. There is a System Property that contains this information, so I all I really needed to do was to add the field and then add a line of code to the version record creation function to pull that value out of the property and stuff it into the new field. The name of the property is glide.buildtag.last, and the name of the field that I added is built_on.

New built_on field added to the version record

Once I added the new field to the version record, I opened up the ApplicationPublisher Script Include, which creates the version record, and added the following line to the function that builds the record:

versionGR.setValue('built_on', gs.getProperty('glide.buildtag.last'));

I also had to modify the function that sent the version record over to other instances to pass on the new value. Since I copied that code instead of consolidating the logic into a single, reusable function, I had to do that in two places, in the ApplicationPublisher and then again in the CollaborationStoreUtils (I really need to collapse that code into a single set of functions that will for both cases). In both places, I added the following line:

payload.built_on = versionGR.getDisplayValue('built_on');

This was not any kind of a major change, but it was something that I had been meaning to do for a long time, and so while I was in looking at the code, I decided to just go ahead and do it. Next time, we will focus on some real work to start building out some kind of error recovery process so that we can ensure that all of the instances in the community are always kept up to date, even if they miss an update in real time for whatever reason.

Collaboration Store, Part XXXVI

Posted on | by snhackery

“The definition of flexibility is being constantly open to the fact that you might be on the wrong track.”
— Brian Tracy

Although it is long past time to start some serious work on the third major component of this little(?) project, the application installation process, I have been rather hesitant to officially kick that off. Last time, we addressed the last of the reported defects in the first two processes, the initial set-up and the application publishing process. Now, it would seem, would be the time to jump into wrestling with that last remaining primary function and put that to bed before turning our attentions to the list of secondary features that will complete the initial release of the full product. At least, that would appear to be the next logical step.

The reason for my reluctance, however, is that I have taken a cursory look at the various approaches available to accomplish my goal, and quite frankly, I don’t really like any of them. When we converted our Update Set to XML, we were able to fish out enough parts and pieces from the stock product to cobble together a reasonable solution with a minimal amount of questionable hackery. To go in the other direction, to convert the XML back to an Update Set, the only stock component that appears to provide this functionality is bound tightly with the process of uploading a local file from the user’s file system. The /upload.do page, or more specifically, the /sys_upload.do process to which the form on that page is posted, handles both the importing of the XML file and the conversion of that file to an Update Set. There is no way to extract the process that turns the XML into an Update Set, since everything is encapsulated into that one all-encompassing process. For our purposes, we do not have a local file on the user’s machine to upload; our file is an attachment already present on the server, so invoking this process, which seems the only way to go, involves much more than we really need.

To invoke the one and only process that I have found (so far!) to make this conversion, then, we will have to post a multi-part form to /sys_upload.do that includes our XML data along with all of the other fields, headers, and cookies expected by the server-side form processor. On the server side, we should be able to accomplish this with an outbound REST message, or on the client side, we should be able to do this by somehow sending our XML instead of a local file when submitting the form. Each approach has its own merits, but they also each have their own issues, and no matter which way you go, it’s a rather complicated mess.

The Server Side Approach

Posting a multi-part form on the server side is actually relatively simple as far as the form data goes. We can construct a valid body for a standard multipart/form-data POST using our XML data and related information and then send it out using an outbound REST message. That’s the easy part. We just need to add an appropriate Content-Type header, including some random boundary value:

Content-Type: multipart/form-data; boundary=somerandomvalue

Then we can build up the body by including all of the hidden fields on the form, and then add our XML in a file segment that would look something like this:

------------somerandomvalue
Content-Disposition: form-data; name="attachFile"; filename="app_name_v1.0.xml"
Content-Type: application/xml

<... insert XML data here ...>

------------somerandomvalue--

In addition to the XML file component, you would also need to send all of the other expected form field values, some of which are preloaded on the form when it is delivered. To obtain those values, you would have to first issue an HTTP GET request of the /upload.do page and pick those values out of the resulting HTML. This can be accomplished with a little regex magic and the Javascript string .match() method. Here is a simple function to which you can pass the HTML and a pattern to return the value found in the HTML based on the pattern:

function extractValue(html, pattern) {
	var response = '';
	var result = html.match(pattern);
	if (result.length > 1) {
		response = result[1];
	}
	return response;
}

For example, one of the form fields found on the /upload.do page is sysparm_ck. The INPUT element for this hidden field looks like this:

<input name="sysparm_ck" id="sysparm_ck" type="hidden" value="68fa4eee2fa401104425fcecf699b646939f52c6787c23fff22b124fcf58f713235b7478"></input>

To snag the value of that field, you would just pass the HTML for the page and the following pattern to our extractValue function:

id="sysparm_ck" type="hidden" value="(*.?)"

Once you have obtained the value, you can use it to build another “part” in the multi-part form body:

------------somerandomvalue
Content-Disposition: form-data; name="sysparm_ck"

68fa4eee2fa401104425fcecf699b646939f52c6787c23fff22b124fcf58f713235b7478

------------somerandomvalue

All of that is pretty easy to do, and would work great except for one thing: this POST would only be accepted as part of an authenticated session, and cannot just be sent in on its own. Theoretically, we could create an authenticated session by doing a GET of the /login.do page and then a POST of some authoritative user’s credentials, but that would mean knowing and sending the username and password of a powerful user, which is a dangerous thing with which to start getting involved. For that reason, and that reason alone, this does not seem to be a good way to go.

The Client Side Approach

On the client side, you are already involved in an authenticated session, so that’s not any kind of an issue at all. What you do not have is the XML, so to do anything on the client side, we will first need to create some kind of GlideAjax service that will deliver the XML over to the client. Once we have the XML that we would like to upload in place of the normal local file, we will have to perform some kind of magic trick to update the form on the page with our data in the place of a file from the local computer. To do that, we will have to either create our own copy of the /upload.do page or add a global script that will only run on that page, and only if there is some kind of URL parameter indicating that this is one of our processes and not just a normal user-initiated upload. We did this once before with a global script that only ran on the email client, so I know that I can run a conditional script on a stock page if I do not create a page of my own, but the trick will be getting the XML data to be sent back to the server with the rest of the input form.

After nosing around a bit for available options, it appears that you might be able to leverage the DataTransfer object to build a fake file, link it to the form, and then submit the form using something like this:

function uploadXML(xml, fileName) {
	var fileList = new DataTransfer();
	fileList.items.add(new File(xml, fileName));
	document.getElementById('attachFile').files = fileList.files;
	document.forms[0].submit();
}

Of course, there will be a lot more to it than that, as you will need to get a handle on the Update Set created and then try to Preview and Commit it programmatically as well, but this looks like a possibility. Still, you have to move the entire Update Set XML all the way down to the client just to push it all the way back up to the server again, which seems like quite a waste. Plus, with any client-side functionality, there is always the browser compatibility issues that would all need to be tested and resolved. Maybe this would work, but I still don’t like it. It seems like quite a bit of complexity and more than a few opportunities for things to go South. I’m still holding out hope that there is a better way.

Now what?

So … given that I don’t like any of the choices that I have come up with so far, I have decided to set that particular task aside for now in the hopes that a better alternative will come to me before I invest too much effort into a solution with which I am not all that thrilled. There is no shortage of things to do here, so my plan is to just focus on other issues and then circle back to this particular effort when a better idea reveals itself, or I run out of other things to do. Technically, once you have obtained the XML for a particular version from the Host, you can still manually install it by downloading the attachment yourself and importing like any other XML Update Set. That’s not really how I intend all of this to function, but it does work, so it should be OK to set this aside for a time.

Next time, then, instead of forging ahead with this third major component as I had originally planned, I will pick something else out of the pile and we will dig into that instead.

Collaboration Store, Part XXXIV

Posted on | by snhackery | 1 Comment on Collaboration Store, Part XXXIV

“The key is not to prioritize what’s on your schedule, but to schedule your priorities.”
— Stephen Covey

So far, we have completed the first two of the three primary components of the project, the initial set-up process and the application publication process. The last of the three major pieces will be the process that will allow you to install an application obtained from the store. Before we dive straight into that, though, we should pause to take a quick look at what we have, and what still needs to be done in order to make this a viable product. At this point, you can install all of the prerequisites and then grab the latest Update Set, install it, and go through the set-up process to create either a Host or Client instance. Once you get through all of that, you are ready to publish any local Scoped Application to the store, which will then be shared with all other instances in your Collaboration Store community.

What you cannot do, just yet, is to find an application published to the store by some other instance and install it on your own instance. That’s the missing third leg of the stool that we will need to take on next. But that is not all that is left to be done. Once we get the basics to work, there are quite a number of other things to address before one could consider this to be truly usable. Some things are just annoyances, but others are definite features that you would have to consider essential for a complete product.

Speaking of annoyances, one of the things that I really don’t like is that when you publish an app to XML for distribution, the resulting Update Set XML does not include the app’s logo image. Clearly it is a part of the app, and if you push an app to an internal store and pull it down into another instance, it comes complete with the logo, so why they left that out of the XML is a mystery to me. I don’t like that for a couple of reasons: 1) when you pull down the XML for this app, you do not get the logo, and 2) when we use the XML to publish an app to the store, the logo is missing there as well. I have seen people complain about this, but I have not, as yet, seen a solution. I would really like to address that, both for my own apps as well as for the process that we are using in this one.

Speaking of logos, another feature that I would like to have is to provide the ability for each instance to have its own distinctive logo image, so that everything from that particular instance could be tagged with that image as a way to visually identify where the app originated. That’s not a critical feature, which is why I did not include it initially, but it has always been something that I felt should be a part of the process, particularly when you start thinking about ways to browse the store and find what you are looking for. That’s definitely on the We-will-get-around-to-it-one-day list.

Browsing the store is another thing that will need some attention at some point. Right now, we just want to prove that we can set-up the app, publish an application, and install an application published by someone else. Those are the fundamental elements of the app. But once we get all of that worked out, being able to hunt through the store to find what you want will be another important capability to build out. We’re not done with the fundamentals just yet, so we don’t want to put too much energy into that issue right at the moment, but at some point, we will need to create a user-friendly way to easily find what you need.

That, of course, leads into things like searchable keywords, tags, user ratings, reviews, and the like, but now is not the time to head down that rabbit hole. Still, there are a lot of possibilities here, and this could turn into a life-long project in and of itself. That’s probably not a good thing, though!

Anyway, we won’t get anything done if we don’t focus, so we need to stay on task and figure out the application installation process. Once again, there are several options available, but the nicest one seems to be the process that you go through to install an app from an internal store. That’s basically a one-click operation and then the app is installed. Unfortunately, that particular page is neither Jelly nor AngularJS, so you can’t just peek under the hood and see the magic like you can with so many other things on the Now Platform. Another option would be to hack up a copy of the Import XML Action on the Update Set list page to push in the attached XML from a published app version, but that only takes things so far; you still have to Preview the Update Set, resolve any issues, and then manually issue the Commit. It would be much nicer if we could just push a button and have the app installation process run in the background and notify you when it was completed. Obviously, we have some work to do here to come up with the best way to go about this, and we had better figure that out relatively soon. Next time, if we are not dealing with test results from the last release, we will need to start building this out.

Collaboration Store, Part XXXIII

Posted on | by snhackery

“The only impossible journey is the one you never begin.”
— Tony Robbins

Last time, we completed the construction of the process that will send out a new version of an application out to all of the instances in the community. Now we need to come up with a mechanism to kick this process off whenever a new version of an application is published to the Host instance. The simplest way to do that would be to create a new Business Rule on one of the tables housing the artifacts of the new application. To make sure that all of the application artifacts have made their way to the Host, the safest table to target would be the Attachment table, since that is the last artifact to be saved or sent to the Host. Unfortunately, that table is used for all kinds of things, so we will need to make sure that only records attached to the application version table trigger the rule.

To create a new Business Rule, navigate to System Definition > Business Rules and click the New button to bring up the data entry form.

New Business Rule input form

We’ll call our new rule Distribute New Version, associate it with the sys_attachment table, run it async on insert, and check the Advanced checkbox so that we can include a script to run when the rule is triggered.

New Distribute New Version Business Rule

To limit the impact of the rule to just the inserts that we want, we need to add a couple of Filter Conditions, one for the Table and another for the Content Type.

Business Rule filter conditions

On the Advanced tab, we will want to add a Condition that will ensure that this rule will only run on a Host instance. The easiest way to do that is to simply compare the stock instance_name property with the scoped host_instance property. Those two values will only be the same on a Host instance.

gs.getProperty('instance_name') == gs.getProperty('x_11556_col_store.host_instance')

The other thing that we need to do on the Advanced tab is to enter the script that will launch our Subflow. The original way to do that was to utilize the FlowAPI, which is the way that I had coded it initially.

sn_fd.FlowAPI.startSubflow('Distribute_New_Application_Version', {new_version: current.getDisplayValue('table_sys_id'), attachment_id: current.getUniqueValue()});

However, later I decided that it was high time to start embracing the new, preferred approach, which is to use the ScriptableFlowRunner. The code for that is a little more complicated, but at some point the older way of doing this will undoubtedly go away, so we might as well start transitioning things in that direction. Here is the script for the Business Rule using this newer technique:

(function executeRule(current, previous) {
	try {
		var result = sn_fd.FlowAPI.getRunner()
			.subflow('x_11556_col_store.Distribute_New_Application_Version')
			.inBackground()
			.withInputs({new_version: current.getDisplayValue('table_sys_id'), attachment_id: current.getUniqueValue()})
			.run();
	} catch (e) {
		gs.error('Business Rule Distribute New Version - Error running subflow Distribute_New_Application_Version: ' + e.getMessage());
	}
})(current, previous);

That completes the Business Rule, so all we need to do at this point is to Save it and now the next time that an XML attachment is created for a version record, our new Subflow will kick off, distributing the new version of the application to all instances in the community (except for the Host and the instance that produced the application). This also completes the application distribution process, so now it is time to release yet another Update Set so that those of you who have been nice enough to do a little testing can give it a go.

If you have just jumped into this and have not been following along, here is what you will need in order to install the app and try it out:

  • If you have not done so already, you will need to install the latest version of snh-form-fields, which you can find here.
  • You will also need to install this additional global component, which is not included in the scoped application.
  • Once you have successfully installed the prerequisites, you will then need to install the scoped application from this Update Set.
  • After the scoped application has been installed, using an admin account, navigate to Collaboration Store -> Collaboration Store Set-up and complete the set-up process for either a Host or Client instance. The Host instance will need to be set up first, as the set-up for a Client instance requires successful contact with a functioning Host.

And again, as I mentioned last time, in order to test out the distribution process, you will need three or more instances in your community so that one of the Client instances can publish the application to the Host instance and then the Host instance can distribute the new version out to all of the other Client instances. As always, all feedback is welcome and appreciated, and if you are able to get everything to actually work, please let me know in the comments. Thanks!

Next time, we will take a look at where we go from here. Thanks again to all of you who have taken the time to help me out and have let me know what you have found.

Collaboration Store, Part XXXI

Posted on | by snhackery

“Someone’s sitting in the shade today because someone planted a tree a long time ago.”
— Warren Buffett

While we wait for additional test results to come in from the corrected Update Set for our latest iteration, it would be a good time to take a look at what we will need to do to complete the last remaining step in the application publishing process. What we have completed so far pushes all of the new application version artifacts to the Host instance. Now we need to build out the process for the Host instance to send out all of those artifacts to all of the other Client instances. Fortunately, we already have in hand most of the code to do that; we just need to clone a few things and make a number of modifications to fit the new purpose.

We have already built a Subflow to push new Client instance information out to all of the existing Client instances during the new instance registration process. Pushing out a new version of an application to those same instances is essentially the same process, so we should be able to clone that guy, and the associated Action, to create our new process. We have also created functions to push over the application record, the version record, and the attached Update Set XML file, and we should be able to copy over those guys as well to complete the process. In fact, we should probably start with the JavaScript functions, as we will need to call the root function in the Action, which will need to be created before it can be referenced in the Subflow.

The functions as written are hard-coded to send everything over to the Host instance. We will want to do basically the same thing, but this time we will be sending things out to various Client instances. Ideally, I should just refactor the code to have the destination instance and credentials passed in so that one function would work in both circumstances, but at this point it was less complicated to just make copies of each function and hack them up for their new purpose. I don’t really like having two copies of essentially the same thing, though, so one day I am going to have to go back and rework the entire mess.

But for now, this is what I came up with when I copied what was once called processPhase5 to create the new publishNewVersion function:

publishNewVersion: function(newVersion, targetInstance, attachmentId) {
	var targetGR = new GlideRecord('x_11556_col_store_member_organization');
	if (targetGR.get('instance', targetInstance)) {
		var token = targetGR.getDisplayValue('token');
		var versionGR = new GlideRecord('x_11556_col_store_member_application_version');
		if (versionGR.get(newVersion)) {
			var canContinue = true;
			var targetAppId = '';
			var mbrAppGR = versionGR.member_application.getRefRecord();
			var request  = new sn_ws.RESTMessageV2();
			request.setHttpMethod('get');
			request.setBasicAuth(this.WORKER_ROOT + targetInstance, token);
			request.setRequestHeader("Accept", "application/json");
			request.setEndpoint('https://' + targetInstance + '.service-now.com/api/now/table/x_11556_col_store_member_application?sysparm_fields=sys_id&sysparm_query=provider.instance%3D' + mbrAppGR.getDisplayValue('provider.instance') + '%5Ename%3D' + encodeURIComponent(mbrAppGR.getDisplayValue('name')));
			var response = request.execute();
			if (response.haveError()) {
				gs.error('CollaborationStoreUtils.publishNewVersion: Error returned from Target instance ' + targetInstance + ': ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
				canContinue = false;
			} else if (response.getStatusCode() == '200') {
				var jsonString = response.getBody();
				var jsonObject = {};
				try {
					jsonObject = JSON.parse(jsonString);
				} catch (e) {
					gs.error('CollaborationStoreUtils.publishNewVersion: Unparsable JSON string returned from Target instance ' + targetInstance + ': ' + jsonString);
					canContinue = false;
				}
				if (canContinue) {
					var payload = {};
					payload.name = mbrAppGR.getDisplayValue('name');
					payload.description = mbrAppGR.getDisplayValue('description');
					payload.current_version = mbrAppGR.getDisplayValue('current_version');
					payload.active = 'true';
					request  = new sn_ws.RESTMessageV2();
					request.setBasicAuth(this.WORKER_ROOT + targetInstance, token);
					request.setRequestHeader("Accept", "application/json");
					if (jsonObject.result && jsonObject.result.length > 0) {
						targetAppId = jsonObject.result[0].sys_id;
						request.setHttpMethod('put');
						request.setEndpoint('https://' + targetInstance + '.service-now.com/api/now/table/x_11556_col_store_member_application/' + targetAppId);
					} else {
						request.setHttpMethod('post');
						request.setEndpoint('https://' + targetInstance + '.service-now.com/api/now/table/x_11556_col_store_member_application');
						payload.provider = mbrAppGR.getDisplayValue('provider.instance');
					}
					request.setRequestBody(JSON.stringify(payload, null, '\t'));
					response = request.execute();
					if (response.haveError()) {
						gs.error('CollaborationStoreUtils.publishNewVersion: Error returned from Target instance ' + targetInstance + ': ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
						canContinue = false;
					} else if (response.getStatusCode() != 200 && response.getStatusCode() != 201) {
						gs.error('CollaborationStoreUtils.publishNewVersion: Invalid HTTP Response Code returned from Target instance ' + targetInstance + ': ' + response.getStatusCode());
						canContinue = false;
					} else {
						jsonString = response.getBody();
						jsonObject = {};
						try {
							jsonObject = JSON.parse(jsonString);
						} catch (e) {
							gs.error('CollaborationStoreUtils.publishNewVersion: Unparsable JSON string returned from Target instance ' + targetInstance + ': ' + jsonString);
							canContinue = false;
						}
						if (canContinue) {
							targetAppId = jsonObject.result.sys_id;
						}
					}
				}
			} else {
				gs.error('CollaborationStoreUtils.publishNewVersion: Invalid HTTP Response Code returned from Target instance ' + targetInstance + ': ' + response.getStatusCode());
			}
			if (canContinue) {
				this.publishVersionRecord(targetInstance, token, versionGR, targetAppId, attachmentId);
			}
		} else {
			gs.error('CollaborationStoreUtils.publishNewVersion: Version record not found: ' + newVersion);
		}
	} else {
		gs.error('CollaborationStoreUtils.publishNewVersion: Target instance record not found: ' + targetInstance);
	}
},

Unlike the original application publication process, we are not bouncing back and forth between the client side and the server side, so with the successful completion of one artifact, we can simply move on to pushing over the next artifact. To create the next step in the process, the publishVersionRecord function, I started out with a copy of the original processPhase6 function. Here is the end result:

publishVersionRecord: function(targetInstance, token, versionGR, targetAppId, attachmentId) {
	var canContinue = true;
	var payload = {};
	payload.member_application = targetAppId;
	payload.version = versionGR.getDisplayValue('version');
	var request  = new sn_ws.RESTMessageV2();
	request.setBasicAuth(this.WORKER_ROOT + targetInstance, token);
	request.setRequestHeader("Accept", "application/json");
	request.setHttpMethod('post');
	request.setEndpoint('https://' + targetInstance + '.service-now.com/api/now/table/x_11556_col_store_member_application_version');
	request.setRequestBody(JSON.stringify(payload, null, '\t'));
	response = request.execute();
	if (response.haveError()) {
		gs.error('CollaborationStoreUtils.publishVersionRecord: Error returned from Target instance ' + targetInstance + ': ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
		canContinue = false;
	} else if (response.getStatusCode() != 201) {
		gs.error('CollaborationStoreUtils.publishVersionRecord: Invalid HTTP Response Code returned from Target instance ' + targetInstance + ': ' + response.getStatusCode());
		canContinue = false;
	} else {
		jsonString = response.getBody();
		jsonObject = {};
		try {
			jsonObject = JSON.parse(jsonString);
		} catch (e) {
			gs.error('CollaborationStoreUtils.publishVersionRecord: Unparsable JSON string returned from Target instance ' + targetInstance + ': ' + jsonString);
			canContinue = false;
		}
		if (canContinue) {
			targetVerId = jsonObject.result.sys_id;
			this.publishVersionAttachment(targetInstance, token, targetVerId, attachmentId);
		}
	}
},

The last step in the process, then, is to send over the Update Set XML file attachment. For that one, I started out with the processPhase7 function and ended up with this:

publishVersionAttachment: function(targetInstance, token, targetVerId, attachmentId) {
	var gsa = new GlideSysAttachment();
	var sysAttGR = new GlideRecord('sys_attachment');
	if (sysAttGR.get(attachmentId)) {
		var url = 'https://';
		url += targetInstance;
		url += '.service-now.com/api/now/attachment/file?table_name=x_11556_col_store_member_application_version&table_sys_id=';
		url += targetVerId;
		url += '&file_name=';
		url += sysAttGR.getDisplayValue('file_name');
		var request  = new sn_ws.RESTMessageV2();
		request.setBasicAuth(this.WORKER_ROOT + targetInstance, token);
		request.setRequestHeader('Content-Type', sysAttGR.getDisplayValue('content_type'));
		request.setRequestHeader('Accept', 'application/json');
		request.setHttpMethod('post');
		request.setEndpoint(url);
		request.setRequestBody(gsa.getContent(sysAttGR));
		response = request.execute();
		if (response.haveError()) {
			gs.error('CollaborationStoreUtils.publishVersionAttachment: Error returned from Target instance ' + targetInstance + ': ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
		} else if (response.getStatusCode() != 201) {
			gs.error('CollaborationStoreUtils.publishVersionAttachment: Invalid HTTP Response Code returned from Target instance ' + targetInstance + ': ' + response.getStatusCode());
		}
	} else {
		gs.error('CollaborationStoreUtils.publishVersionAttachment: Invalid attachment record sys_id: ' + attachmentId);
	}
},

So that takes care of the functions. Now we have to create an Action in the Flow Designer that will call the function, and then produce a new Subflow that will leverage that action. That sounds like a good subject for our next installment.

Special Note to Testers

If you want to test the set-up process, you can do that with a single instance. You just have to select the Host option during the set-up process. If you want to test the Client set-up process, you will need at least two instances, one to serve as the Host, which you will need to reference when you set up the Client (you cannot be a Client of your own Host). But if you really want to test out the full registration process, including the Subflow that informs all existing instances of the new instance, you will need at least three participating instances: 1) the Host instance, 2) the new Client instance, and 3) an existing Client instance that will be notified of the new member of the community.

The same holds true for the application publishing process. You can test a portion of the publishing process by publishing an app on a single Host instance, but if you want to test out all of the parts and pieces, you will want to publish the app from a Client instance and make sure that it makes its way all the way back to the Host. Once we complete the application distribution process, full testing will require at least three instances to verify that the Host distributes the new version of the application to other Clients.