Periodic Review, Part VII

“Whether you want to uncover the secrets of the universe, or you just want to pursue a career in the 21st century, basic computer programming is an essential skill to learn.”
Stephen Hawking

Last time, we got started on our script that will run the review process by building and testing the primary function that finds all of the configurations that will need to run. Today we need to continue on with the script, beginning with the process for a single configuration’s execution. Before we get started, though, we need to talk a little bit about making sure that the notice will go to somebody.

Both the configuration record and the system as a whole have a default recipient/email address. The system default email address is contained in a System Property and the configuration default recipient is on the configuration record. There is a field on the source record for the recipient, and if that recipient record is valid and active and contains an email address, then that will be the address that we will want to use. If not, then we will want to use the configuration fallback recipient, and if there is some issue in the notice process with that recipient’s email address, then we will end up using the system fallback address. This is to ensure that the notice will actually get to a real person who can take action on the review request.

There is also another optional System Property that can be used to limit notifications to a specific email domain. If that value is present, and the recipient email is not an email address of that domain, then the fallback recipient will be used as well. All of this should be documented in the notice item record. So the first thing that we will want to do will be to pull those System Properties.

// fetch system properties
var systemFallbackAddress = gs.getProperty('x_11556_periodic_r.fallback_email_address');
var systemEmailDomain = gs.getProperty('x_11556_periodic_r.email_domain');

The next thing that we will want to do will be to create a record in the Review Execution table to memorialize this execution.

// create execution record
var executionGR = new GlideRecord('x_11556_periodic_r_review_execution');
executionGR.configuration = configuration;
executionGR.run_date = new GlideDate();
executionGR.run_start = new GlideDateTime();
executionGR.state = 'Running';
executionGR.short_description = executionGR.getDisplayValue('run_date') + ' review notices';
executionGR.total_items = 0;
executionGR.total_notices = 0;
executionGR.completion_code = 0;
executionGR.insert();

This is standard GlideRecord stuff, so we don’t really need to go into a lot of detail here other than to say that we link it back to the configuration that is being executed and initialize all of the other values. Once we have our execution record created, we can pull the configuration record from the reference, and once we have it, we can set up the fallback recipient for this configuration.

var configurationGR = executionGR.configuration.getRefRecord();
var fallbackRecipient = configurationGR.getDisplayValue('fallback_recipient');

Now we need to go fetch all of the items that need to be reviewed this cycle, but because we want to consolidate all of the items for any given recipient into a single notice, we will first need to fetch them all, and then once we have the full list, group them by recipient. To facilitate that, we can create a temporary notice record and assign all of the items to that notice, and once we run through all of the query results, we can then fetch them back by recipient. So let’s create that temporary notice record now.

// create temporary notice record
var noticeGR = new GlideRecord('x_11556_periodic_r_review_notice');
noticeGR.review_execution = executionGR.getUniqueValue();
noticeGR.recipient = configurationGR.fallback_recipient;
noticeGR.short_description = 'Temporary notice record';
noticeGR.insert();
var tempNotice = noticeGR.getUniqueValue();

Once we have our notice record, we can use the table and filter from the configuration record to find all of the items to be reviewed.

// create notice item records from source table data
var noticeItemGR = new GlideRecord('x_11556_periodic_r_review_notice_item');
var itemCt = 0;
var sourceGR = new GlideRecord(table);
if (sourceGR.isValid()) {
	if (filter) {
		sourceGR.addEncodedQuery(filter);
	}
	sourceGR.orderBy(configurationGR.short_description_column);
	sourceGR.query();
	while (sourceGR.next()) {
		...
	}
	...
} else {
	...
}

Then, inside the query loop, we can build a record in the Review Notice Item table for each item.

noticeItemGR.initialize();
noticeItemGR.review_notice = tempNotice;
noticeItemGR.id = sourceGR.getUniqueValue();
noticeItemGR.short_description =  sourceGR.getDisplayValue(configurationGR.short_description_column);
noticeItemGR.description = sourceGR.getDisplayValue(configurationGR.description_column);
noticeItemGR.recipient = sourceGR.getValue(configurationGR.recipient_column);
noticeItemGR.insert();
itemCt++;

Once the notice item record has been created, we can inspect the recipient and come up with a valid recipient to which we will send the notice.

var recipientGR = new GlideRecord('sys_user');
recipientGR.get(noticeItemGR.getValue('recipient'));
var notes = '';
if (recipientGR.isValid()) {
	if (recipientGR.getValue('active')) {
		var email = recipientGR.getDisplayValue('email');
		if (email) {
			if (systemEmailDomain && !email.endsWith(systemEmailDomain)) {
				notes = 'Recipient email address is not an authoized email address; reverting to fallback recipient';
			}
		} else {
			notes = 'Specified recipient has no email address; reverting to fallback recipient';
		}
	} else {
		notes = 'Specified recipient is not active; reverting to fallback recipient';
	}
} else {
	notes = 'Recipient column empty on source record; reverting to fallback recipient';
}
if (notes) {
	noticeItemGR.notes = notes;
	noticeItemGR.recipient = fallbackRecipient;
	noticeItemGR.update();
}

Once we complete the loop, we will want to see if the query returned any items, and either process the items or close out this execution.

if (itemCt > 0) {
	executionGR.total_items = itemCt;
	executionGR.update();
	this.sendNotices(configurationGR, executionGR, noticeGR, noticeItemGR, tempNotice);
} else {

	// finalize the execution
	executionGR.total_items = 0;
	executionGR.total_notices = 0;
	executionGR.run_end = new GlideDateTime();
	executionGR.state = 'Complete';
	executionGR.completion_code = 0;
	executionGR.description = 'No items matched the filter criteria during this run, so no notices were sent out.';
	executionGR.update();
}

And once that is done, whether we had any items returned or not, we will want to delete that temporary notice record.

// delete the temporary notice
noticeGR.get(tempNotice);
noticeGR.deleteRecord();

One last thing: we need to do something in the event that we find ourselves in the else branch of the if (sourceGR.isValid()) { conditional. This basically means that the source table specified on the configuration record is not valid and so we cannot look for any items there. This is basically a failure of the execution, so we need to update the execution record to reflect that.

// finalize the execution
executionGR.total_items = 0;
executionGR.total_notices = 0;
executionGR.run_end = new GlideDateTime();
executionGR.state = 'Failed';
executionGR.completion_code = 1;
executionGR.description = 'The specified source table in the configuration record is not valid; execution cannot proceed; aborting execution.';
executionGR.update();

Failure is not an option that we originally set up for that field, so we will have to go back into the dictionary at some point and fix that to make that code work. Finally, regardless of how the execution turned out, we need to update the configuration record with the next run date, which is calculated in the function that we added for that purpose earlier.

// set the next run date
configurationGR.setValue('next_scheduled_date', this.calculateNextRunDate(configurationGR));
configurationGR.update();

With the addition of this completed function, our utility Script Include now looks like this.

var PeriodicReviewUtils = Class.create();
PeriodicReviewUtils.prototype = {
    initialize: function() {
    },

	dailyProcess: function() {
		var toRun = [];
		var configurationGR = new GlideRecord('x_11556_periodic_r_review_configuration');
		var today = new GlideDate();
		configurationGR.addQuery('next_scheduled_date', today);
		configurationGR.orderBy('number');
		configurationGR.query();
		while (configurationGR.next()) {
			var execution = {};
			execution.configuration = configurationGR.getUniqueValue();
			execution.table = configurationGR.getDisplayValue('table');
			execution.filter = configurationGR.getDisplayValue('filter');
			toRun.push(execution);
		}
		if (toRun.length > 0) {
			gs.info('PeriodicReviewUtils.dailyProcess: Running ' + toRun.length + ' execution(s) today.');
			for (var i in toRun) {
				var thisRun = toRun[i];
				this.processExecution(thisRun.configuration, thisRun.table, thisRun.filter);
			}
			gs.info('PeriodicReviewUtils.dailyProcess: ' + toRun.length + ' execution(s) completed.');
		} else {
			gs.info('PeriodicReviewUtils.dailyProcess: Nothing scheduled to run today.');
		}
	},

	processExecution: function(configuration, table, filter) {
		gs.info('PeriodicReviewUtils.dailyProcess: ' + configuration + '; ' + table + '; ' + filter);

		// fetch system properties
		var systemFallbackAddress = gs.getProperty('x_11556_periodic_r.fallback_email_address');
		var systemEmailDomain = gs.getProperty('x_11556_periodic_r.email_domain');

		// create execution record
		var executionGR = new GlideRecord('x_11556_periodic_r_review_execution');
		executionGR.configuration = configuration;
		executionGR.run_date = new GlideDate();
		executionGR.run_start = new GlideDateTime();
		executionGR.state = 'Running';
		executionGR.short_description = executionGR.getDisplayValue('run_date') + ' review notices';
		executionGR.total_items = 0;
		executionGR.total_notices = 0;
		executionGR.completion_code = 0;
		executionGR.insert();
		var configurationGR = executionGR.configuration.getRefRecord();
		var fallbackRecipient = configurationGR.getDisplayValue('fallback_recipient');

		// create temporary notice record
		var noticeGR = new GlideRecord('x_11556_periodic_r_review_notice');
		noticeGR.review_execution = executionGR.getUniqueValue();
		noticeGR.recipient = configurationGR.fallback_recipient;
		noticeGR.short_description = 'Temporary notice record';
		noticeGR.insert();
		var tempNotice = noticeGR.getUniqueValue();

		// create notice item records from source table data
		var noticeItemGR = new GlideRecord('x_11556_periodic_r_review_notice_item');
		var itemCt = 0;
		var sourceGR = new GlideRecord(table);
		if (sourceGR.isValid()) {
			if (filter) {
				sourceGR.addEncodedQuery(filter);
			}
			sourceGR.orderBy(configurationGR.short_description_column);
			sourceGR.query();
			while (sourceGR.next()) {
				noticeItemGR.initialize();
				noticeItemGR.review_notice = tempNotice;
				noticeItemGR.id = sourceGR.getUniqueValue();
				noticeItemGR.short_description =  sourceGR.getDisplayValue(configurationGR.short_description_column);
				noticeItemGR.description = sourceGR.getDisplayValue(configurationGR.description_column);
				noticeItemGR.recipient = sourceGR.getValue(configurationGR.recipient_column);
				noticeItemGR.insert();
				itemCt++;
				var recipientGR = new GlideRecord('sys_user');
				recipientGR.get(noticeItemGR.getValue('recipient'));
				var notes = '';
				if (recipientGR.isValid()) {
					if (recipientGR.getValue('active')) {
						var email = recipientGR.getDisplayValue('email');
						if (email) {
							if (systemEmailDomain && !email.endsWith(systemEmailDomain)) {
								notes = 'Recipient email address is not an authoized email address; reverting to fallback recipient';
							}
						} else {
							notes = 'Specified recipient has no email address; reverting to fallback recipient';
						}
					} else {
						notes = 'Specified recipient is not active; reverting to fallback recipient';
					}
				} else {
					notes = 'Recipient column empty on source record; reverting to fallback recipient';
				}
				if (notes) {
					noticeItemGR.notes = notes;
					noticeItemGR.recipient = fallbackRecipient;
					noticeItemGR.update();
				}
			}
			if (itemCt > 0) {
				executionGR.total_items = itemCt;
				executionGR.update();
				this.sendNotices(configurationGR, executionGR, noticeGR, noticeItemGR, tempNotice);
			} else {

				// finalize the execution
				executionGR.total_items = 0;
				executionGR.total_notices = 0;
				executionGR.run_end = new GlideDateTime();
				executionGR.state = 'Complete';
				executionGR.completion_code = 0;
				executionGR.description = 'No items matched the filter criteria during this run, so no notices were sent out.';
				executionGR.update();
			}

			// delete the temporary notice
			noticeGR.get(tempNotice);
			noticeGR.deleteRecord();
		} else {
			// finalize the execution
			executionGR.total_items = 0;
			executionGR.total_notices = 0;
			executionGR.run_end = new GlideDateTime();
			executionGR.state = 'Failed';
			executionGR.completion_code = 1;
			executionGR.description = 'The specified source table in the configuration record is not valid; execution cannot proceed; aborting execution.';
			executionGR.update();
		}

		// set the next run date
		configurationGR.setValue('next_scheduled_date', this.calculateNextRunDate(configurationGR));
		configurationGR.update();
	},

	calculateNextRunDate: function(configurationGR) {
		var runDate = new Date(configurationGR.getDisplayValue('next_scheduled_date'));
		var frequency = configurationGR.getValue('frequency');
		var days = 0;
		var months = 0;
		if (frequency == 'daily') {
			days = 1;
		} else if (frequency == 'weekly') {
			days = 7;
		} else if (frequency == 'biweekly') {
			days = 14;
		} else if (frequency == 'monthly') {
			months = 1;
		} else if (frequency == 'bimonthly') {
			months = 2;
		} else if (frequency == 'quarterly') {
			months = 3;
		} else if (frequency == 'semiannually') {
			months = 6;
		} else if (frequency == 'annually') {
			months = 12;
		} else if (frequency == 'biannually') {
			months = 24;
		}
		if (days > 0) {
			runDate.setDate(runDate.getDate() + days);
		} else {
			runDate.setMonth(runDate.getMonth() + months);
		}
		return JSON.stringify(runDate).substring(1, 11);
	},

    type: 'PeriodicReviewUtils'
};

Next time, we will stub out that sendNotices function and give things another test or two, and then actually build out that function to group the items by recipient and send out the notices.

Collaboration Store, Part LXXXIV

“The power of one, if fearless and focused, is formidable, but the power of many working together is better.”
Gloria Macapagal Arroyo

Last time, we released yet another beta version of the app for testing, so now might be a good time to talk a little bit about what exactly needs to be tested, and maybe a little bit about where things stand and where we go from here. We have had a lot of good, quality feedback in the past, and I am hoping for even more from this version. Every bit helps drive out annoying errors and improves the quality of the product, so keep it coming. It is very much appreciated.

Installation

The first thing that needs to be tested, of course, is the installation itself. Just to review, you need to install three Update Sets in the appropriate order, SNH Form Fields, the primary Scoped Application, and the accompanying global components that could not be bundled with the scoped app. You can find the latest version of SNH Form Fields here, or you can simply grab the latest SNH Data Table Widgets from Share, which includes the latest version of the form field tag. Once that has been installed, you can then install collaboration_store_v0.7.5.xml, after which you can then install the globals, collaboration_store_globals_v0.7.xml.

There are two types of installations, a brand new installation and an upgrade of an existing installation. Both types of installs have had various issues reported with both Preview and Commit errors. On a brand new installation, just accept all updates and you should be good to go. I don’t actually know why some of those errors come up on a brand new install, but if anyone knows of any way to keep that from happening, I would love to hear about it. It doesn’t seem to hurt anything, but it would be so much better if those wouldn’t come up at all. On an upgrade to an existing installation, you will want to reject any updates related to system properties. The value of the application’s properties are established during the set-up process, and if you have already gone through the set-up process, you don’t want those values overlaid by the installation of a new version. Everything else can be accepted as is. Once again, if anyone has any ideas on how to prevent that kind of thing from happening, please let us all know in the comments below.

The Set-up Process

The Collaboration Store Set-up process

Once you have the software installed for the first time, you will need to go through the set-up process. This is another thing that needs to be tested thoroughly, both for a Host instance and a Client instance. It needs to tested with logo images and without, and for Client instances, you will need to check all of the other member instances in the community to ensure that the newly set-up instance now appears in each instance. During the set-up process, a verification email will be sent to the email address entered on the form, and if your instance does not actually send out mail, you will need to look in the system email logs for the code that you will need to complete the process.

The Publishing Process

Application Publishing Process

Once the software has been installed and the set-up process completed, you can now publish an application to the store. Both Client instances and Host instances can publish apps to the store. Publishing is accomplished on the system application form via a UI Action link at the bottom of the form labeled Publish to Application Store. Click on the link and follow the prompts to publish your application to the store. If you run into any issues, please report them in the comments below.

The Installation Process

Application Installation Process

Once published to the store, shared applications can be installed by any other Host or Client instance from either the version record of the version desired, or the Collaboration Store itself. Simply click on the Install button and the installation should proceed. Once again, if you run into any issues, please use the comments below to provide us with some detailed information on where things went wrong.

The Collaboration Store

The Collaboration Store

The Collaboration Store page is where you should see all of the applications shared with the community and there are a lot of things that need to be tested here. This is the newest addition to the application, so we will want to test this thing out thoroughly. One thing that hasn’t been tested at all is the paging, as I have never shared enough apps to my own test environment to exceed the page limit. The more the merrier as far as testing is concerned, so if you can add as many Client instances as possible, that would be helpful, and if each Client could share as many applications as possible, that would be helpful as well. Several pages worth, in varying states would help in the testing of the search widget as well as the primary store widget. And again, if you run into any problems, please report them in the comments.

The Periodic Sync Process

The periodic sync process is designed to recover from any previous errors and ensure that all Clients in the community have all of the same information that is stored in the Host. Testing the sync process is simply a matter of removing some artifacts from some Client instance and then running the sync process to see if those artifacts were restored. The sync process runs every day on the Host instance over the lunch hour, but you can also pull up the Flow and run it by hand.

Thanks in advance to those of you who have already contributed to the testing and especially to those of you who have decided to jump in at this late stage and give things a try. Your feedback continues to be quite helpful, and even if you don’t run into any issues, please leave us a comment and let us know that as well. Hopefully, we are nearing the end of this long, drawn out project, and your assistance will definitely help to wrap things up. Next time, we will talk a little bit more about where things go from here.

Collaboration Store, Part LXXII

“Sometimes when you innovate, you make mistakes. It is best to admit them quickly and get on with improving your other innovations.”
Steve Jobs

Last time, we put out a new version of the Scoped Application Update Set that addressed a few of the reported issues. There are still a few unresolved issues, however, and some that have not even been discussed as yet. One that was reported last time, which I have been able to replicate, is a Commit failure of an attempted insert on the sys_scope_privilege table.

Duplicate entry ‘5b9c19242f6030104425fcecf699b6ec-global-sys_app_module-sys_db…’ for key ‘source_scope_2’

This causes the Update Set commit to be reported as a failure, even though everything else was installed without issue and everything works just fine despite this particular problem. I searched through the Update Set hoping to find an update that was in there twice for this item, but it only appears once, so I am not sure why we would be getting a duplicate entry error. The entry says ‘INSERT OR UPDATE’, so you would think that it would know if it was already there and do an UPDATE instead of an INSERT, but that’s obviously not what is happening. I’m going to have to do a little more research to see if I can find the cause of this one. Even though it doesn’t seem to hurt anything, I don’t like it and I would like to figure out how to prevent that from happening.

The other thing that seems to happen to folks during the Preview phase is that they run into a number of issues that need to be addressed and the Preview never runs clean. Up to this point, my recommendation has been to just accept all remote updates and then do the Commit. There are two areas, though, where I now believe that it would be better to skip the updates rather than accept them. One is any update related to System Properties and the other is any update related to the left navigation menu items. Once you have gone through the set-up process, the property values have all been establish based on the information entered during set-up. You do not want to overlay those with an Update Set or you will have to go through the set-up process all over again. Also, once you complete the set-up process, the set-up menu item is deactivated and all of the other menu items are revealed. You don’t want that to be reverted either, so if you are installing an update over an existing installation that has already been set up, you should skip any update related to these two items. All of the rest, you should still go ahead and accept. I don’t like having to have these kinds of specialized instructions, though, so I am going to be looking at ways to restructure things so that this is not an issue. The goal would be to always have a clean Preview and a clean Commit, regardless of whether it was a fresh install or an update of an existing install.

One issue that I did attempt to correct in the most recent version was the attachment copy problem, a problem that resulted in the attachment of the wrong file to the version record. It turns out that this bad code was replicated a number of times and was only fixed in one place. To correct that problem, and to make future maintenance a little easier, I built a common attachment copy function that contained the corrected code, and then called that function from every place that previously had a copy of the incorrect logic.

copyAttachment: function(fromTable, fromId, toTable, toId, attachmentId) {
	var response = {success: false};

	var gsa = new GlideSysAttachment();
	var values = gsa.copy(fromTable, fromId, toId, toId);
	if (values.length > 0) {
		for (var i=0; i<values.length; i++) {
			var ids = values[i].split(',');
			if (ids[0] == attachmentId) {
				response.success = true;
				response.newId = ids[1];
				gsa.deleteAttachment(attachmentId);
			} else {
				gsa.deleteAttachment(ids[1]);
			}
		}
	} else {
		response.error = 'Unrecognized response from attachment copy: ' +  JSON.stringify(values) + '\nFrom table: ' + fromTable +'; From ID: ' + fromId + '; To table: ' + toTable +'; To ID: ' + toId + '; Sys ID: ' + attachmentId;
	}

	return response;
}

Once that was in place, I updated the functions that previously included that code to just call that function and evaluate the results.

processPhase4: function(answer) {
	var response = this.CSU.copyAttachment('sys_app', answer.appSysId, 'x_11556_col_store_member_application_version', answer.versionId, answer.attachmentId);
	if (response.success) {
		answer.attachmentId = response.newId;
	} else {
		answer = this.processError(answer, 'Copy of Update Set XML file failed - ' + response.error);
	}

	return answer;
}

This code works for both Update Set XML files as well as logo image files.

copyLogoImage: function(sysAppGR, applicationGR) {
	var logoId = '';

	var csu = new CollaborationStoreUtils();
	var response = csu.copyAttachment('ZZ_YYx_11556_col_store_member_application', applicationGR.getUniqueValue(), 'ZZ_YYsys_app', sysAppGR.getUniqueValue(), applicationGR.getValue('logo'));
	if (response.success) {
		logoId = response.newId;
	} else {
		gs.error('Copy of application logo image failed - ' + response.error);
	}

	return logoId;
}

Speaking of logo image files, when I updated everything to include logo images for each instance, I neglected to add an image upload function to the initial set-up process. Since the Host’s data for each Client instance is collected during the set-up process, no Client instances will have logo images on the Host, which means no logo images will ever be shared with other Clients. I need to fix that, and then release yet another bug-fix Update Set that will address as many of these issues as possible. Adding the image upload to the set-up process may get a little involved, so let’s save that for our next installment.

Collaboration Store, Part XXXVIII

“You don’t have to be good to start … you just have to start to be good!”
Joe Sabah

Last time out we really did not accomplish anything of any consequence, but today let’s see if we can actually make some progress on something of real value. We need to create a process that will run every so often and check to make sure that all of the instances in the community have all of the latest content. We can use the Flow Designer to create and schedule a flow that will run daily on the Host instance and compare the artifacts present on each Client instance with the artifacts present on the Host, and then send over any missing items that never made it over originally for whatever reason. Doing it this way will eliminate the need to track and record errors when they happen, since we will just compare the Client inventory with that of the Host and correct any discrepancies. We don’t really need to know what failed or why.

We could make all of the REST API calls to the Client instances using the Integration Hub, but since that is a collection of optional products, not every instance will have all of those components installed and we would not want to make that a prerequisite for the product. To make this work on any ServiceNow configuration, we will want to roll our own calls via Javascript, and we will just use the Flow Designer to schedule a simple flow that loops through all of the Client instance records and then calls a Script Action that will do all of the heavy lifting. Before we build the Action though, we will want to build a Script Include function that can be called in the Action. Since this will be a significant script, we should probably create a new Script Include specific to this purpose rather than adding a number of new functions to any of our existing Script Includes. We can call our new Script Include InstanceSyncUtils and create a simple function called syncInstance with a single argument of the sys_id of the record for the instance to be synced. For now, we can just stub out the function with a simple logging statement and then circle back later to fill in the details. Right now, we just want to have something to call in our Flow Designer Action.

New Script Include function to be called in our new Action

With that out of the way, we can now fire up the Flow Designer and navigate to New -> Action. We will call our new Action Sync Instance and configure a single Input called Instance ID.

New Sync Instance Action with a single Input

For the Script step, we will just pass the Input to our new Script Include function:

(function execute(inputs, outputs) {
	var isu = new InstanceSyncUtils();
	isu.syncInstance(inputs.instance_id);
})(inputs, outputs);

There is currently nothing returned by the function, so there is no need to define any Outputs. At this point, all we need to do is to Save and Publish the Action before turning our attentions to the actual Flow itself.

To build a Flow, go back to the Home Page of the Flow Designer and select New -> Flow. On the resulting pop-up Flow Properties panel, enter all of the details for the Flow and set the Run As value to System User.

New Flow Properties

Since we want this Flow to run on its own every day, we can set the Trigger to Daily, and the Time to 12:30, which should run it in the middle of the lunch hour in the Host time zone, when most instances should be available.

Flow Trigger configuration

Since we only want this Flow to run on the Host instance, the first thing that we need to do is to create a Flow Variable that indicates whether or not this instance is the Host. To set the value of the variable, we add a Set Flow Variables step that uses the following script to determine if this instance is the Host based on System Properties.

return gs.getProperty('instance_name') == gs.getProperty('x_11556_col_store.host_instance');

The next step then will be a conditional that checks the value of the new Flow Variable.

Making sure that this is the Host instance

Failing this test will terminate the Flow, but if this is the Host instance, then the next thing that we need to do is gather up all of the Client instance records and then loop through them and call our new Action for each one in turn.

Find all records where Active is true and Host is false

Inside the following For Each Item loop, we can then invoke our new Action, passing in the sys_id of the current record.

Calling our new Action inside the For Each Item loop

That completes the work on the Flow, and all we need to do now is to Save and Activate the Flow. At this point, the Flow will kick off every day at 12:30 local time, but it won’t really do much except put a few entries in the System Log. The real work will be done in the Script Include, and since that’s a major effort in and of itself, we’ll leave that for our next exciting episode.

Collaboration Store, Part XXXVII

“Stay committed to your decisions, but stay flexible in your approach.”
Tony Robbins

After abandoning my earlier plans to jump into the application publishing process, I started to take a look at the missing error recovery needs for all of these inter-instance interactions. One thing that I had always planned to do at some point was to create an activity log of all transactions between instances. My idea was not to sync the logs across all instances, but to have a record in each instance of the things that went on with that particular instance. On the Host instance, such a log could provide the basis for some form of periodic recovery process that scanned the log for failed transactions and then made another attempt to put the transaction through again. After giving that some thought, though, I decided that a better approach would be to just scan each instance for instances, applications, and versions, and then attempt to push over anything that was missing compared to what was present on the Host. I still want to build an activity log at some point, but I don’t think that I want to use it as a basis for error recovery. I think it would be more straightforward to just compare the lists of records on each instance with the master lists on the Host and then try to correct any deficiencies.

That’s the plan today, anyway, but plans do have a way of changing over time, so who knows how things will come out once I start putting all of the pieces together. Right now, though, this seems like the better way to go.

One little thing that have been wanting to do for some time was to add another field to the version record to indicate the version of ServiceNow that was used to build the Update Set. There is a System Property that contains this information, so I all I really needed to do was to add the field and then add a line of code to the version record creation function to pull that value out of the property and stuff it into the new field. The name of the property is glide.buildtag.last, and the name of the field that I added is built_on.

New built_on field added to the version record

Once I added the new field to the version record, I opened up the ApplicationPublisher Script Include, which creates the version record, and added the following line to the function that builds the record:

versionGR.setValue('built_on', gs.getProperty('glide.buildtag.last'));

I also had to modify the function that sent the version record over to other instances to pass on the new value. Since I copied that code instead of consolidating the logic into a single, reusable function, I had to do that in two places, in the ApplicationPublisher and then again in the CollaborationStoreUtils (I really need to collapse that code into a single set of functions that will for both cases). In both places, I added the following line:

payload.built_on = versionGR.getDisplayValue('built_on');

This was not any kind of a major change, but it was something that I had been meaning to do for a long time, and so while I was in looking at the code, I decided to just go ahead and do it. Next time, we will focus on some real work to start building out some kind of error recovery process so that we can ensure that all of the instances in the community are always kept up to date, even if they miss an update in real time for whatever reason.

Collaboration Store, Part XXXV

“The good news about computers is that they do what you tell them to do. The bad news is that they do what you tell them to do.”
James Barton

Well, the test results are starting to trickle in, and one of the issues looks rather important, so we need to take a look at that before we jump into our next major effort. The problem with the scoped System Properties came up earlier, and I thought that I had found a way to deal with the issue, but obviously there is still a problem where the initial set-up is concerned. Basically, if you are not in the Collaboration Store scope, then the set-up fails because the updates to the scoped System Properties are not allowed. My idea of moving the update logic to a global component did not resolve this issue, as the problem is related to the active scope of the user rather than the scope of the component.

So, it seems that the answer to the problem is to ensure that the user is in the correct scope before allowing them proceed with the set-up. Fortunately, there is already an area in ServiceNow where that very check is made, and there is even an option in the error message to switch over to the correct scope. You can see that when you bring up a scoped app and you are not in the scope of that application.

Sample error message for incorrect scope on the scoped application form

So it would seem that we could snag the HTML for that message and throw it up in the top of the HTML for the initial set-up widget with some kind of ng-hide so that it only appears if you are in the wrong scope. The first thing that we would need to do, then, is to figure out how to detect the user’s scope and then set up some boolean variable in the widget to indicate whether or not the user is in the correct scope to proceed with the set-up. Looking at the GlideSession API, it seems like the getCurrentApplicationId() method is just what we need. So I added this line to the top of the server script of the widget:

data.validScope = (gs.getCurrentApplicationId() == '5b9c19242f6030104425fcecf699b6ec');

Then, to prevent the user from submitting the form when in the wrong scope, I modified the ng-disabled tag on all of the submit buttons from this:

ng-disabled="!(form1.$valid)"

… to this:

ng-disabled="!(form1.$valid) || !c.data.validScope"

Now all I needed to do was to grab a copy of the HTML source from the application form and paste it in at the top of the HTML for the widget and see if the link would still work. Unfortunately, the link in the existing code relies on the GlideURL object, which is not available in Service Portal widgets, so we are going to have to hack that up a little bit to get around that problem. Here is the existing onclick script:

window.location.href=new GlideURL('change_current_app.do').addParam('app_id', '5b9c19242f6030104425fcecf699b6ec').addParam('referrer', window.location.pathname + window.location.search + window.location.hash).getURL();

Basically, this code just builds a URL, so it seems as if we could just go ahead and build the URL manually and hard-code the link. After doing a little tinkering around to see just what the URL actually was, I came up with this value for our circumstances:

change_current_app.do?app_id=5b9c19242f6030104425fcecf699b6ec&referrer=%24sp.do%3Fid%3Dcollaboration_store_setup

So, my final HTML, including the ng-hide based on my new widget variable, turned out to be this:

<div id="nav_message" class="outputmsg_nav" ng-hide="c.data.validScope">
  <img role="presentation" src="images/icon_nav_info.png">
  <span class="outputmsg_nav_inner">
    &nbsp;
    The <strong>Collaboration Store Set-up</strong> cannot be completed because <strong>Collaboration Store</strong> is not selected in your application picker.
    <a onclick="window.location.href='change_current_app.do?app_id=5b9c19242f6030104425fcecf699b6ec&referrer=%24sp.do%3Fid%3Dcollaboration_store_setup'">
      Switch to <strong>Collaboration Store</strong>
    </a>.
  </span>
</div>

Now all I needed to do was to bring up the set-up widget while in the wrong scope and see how it looked and how the new link worked out once I gave it a try.

Initial set-up screen with error message and link when in the incorrect scope

With the hard-coded link in the onclick function, everything seems to work as intended. The user is placed in the correct scope, the form is reloaded, and the error message goes away. This should finally resolve this annoying issue once and for all (let’s hope!).

The other issue that was reported was that the provider field on the application record was not populated when publishing a new version of an application. It was not clear from the comment whether the missing data was on the Client instance, the Host instance, or both, but I was unable to recreate any of those conditions in any of my testing. I am going to need a little more detail on this one before I can address it, so if anyone encounters this error in any of their testing, please leave a detailed message in the comments so that we can get it resolved.

There is still the potential for more feedback to come, but this particular issue seems to be rather critical, so it’s time to release a new Update Set. While I am at it, I think I will take a different approach on the global components and make an actual Update Set for that as well instead of just exporting the XML for the one global Script Include. This way, I can also include the app’s logo image, which is always missing from the XML generated for a scoped application. Here are the two new Update Sets:

As always, all feedback is welcome, and not just to report issues. If you give this a try and everything actually works, I would love to hear about that as well. Next time out, we will either deal with any more issues to come to light, or we will jump into the next big challenge, which will be to install an app published to the store.

Special Note to Testers

Set-up Process – If you want to test the set-up process, you will need to set up the Host instance first. You cannot set up a Client instance until there is a valid Host instance, as the Client instance set-up process requires access to a valid Host instance. What to look for: check to make sure that all instances in the community have the same list of instances; select Collaboration Store -> Member Organizations to pull up the list of instances in each instance to verify that all instances have the exact same list of all instances in the community.

Application Publishing Process – If you want to test the application publishing process, you will need to go through the set-up process first, and then you can publish an application to the store. This can be done with a single Host instance, but to fully test all of the functionality, you will need at least two Client instances in addition to the Host. What to look for: once you publish the application, check to make sure that all instances in the community have the newly published version of the application, including the XML Update Set attachment; select Collaboration Store -> Member Organizations to pull up the list of instances, then check the Related List of applications under the appropriate instance for the application, and then click on the app to check the Related List of versions under the application.

Application Installation Process – If you want to test the application installation process, you have jumped ahead of the class, as that portion of the app has not yet been developed. However, even though the development of the official installation process has not even been started, if you really want to install an app that has been shared with your instance, you can always download the XML attachment on the version record, import it back into your instance, and go through the normal XML Update Set installation process that you would go through for any other imported Update Set. But again, that’s getting a little ahead of where things stand right at the moment with project’s development.

One More Thing

As mentioned earlier, there is currently no error recovery built into the system at this time. This is definitely something near the top of the we-really-need-to-do-this list, but it is not there right now. What that means is that if you are doing any kind of testing and one or more of the instances in your community is off-line or unavailable for some reason, things will fail and those instances will not get the needed updates. One day we will definitely need to fix that, but for now, if that happens, that’s not a bug in the software to be reported; it’s to be expected until we build in some kind of error recovery into the product.

Also, if you end up testing things and don’t have any issues to report, then by all means, report that, too! You don’t have to have encounter a problem to post your results. If everything worked out as expected, we would definitely love to hear that as well. As always, all feedback of any kind is very much appreciated.

Collaboration Store, Part XXIX

As a rule, software systems do not work well until they have been used, and have failed repeatedly, in real applications.”
David Lorge Parnas

Now that we have completed the initial version of the application publishing process, it would be nice to release a new Update Set so that the folks who are inclined to help out with the testing can try it all out. However, we have already received feedback from the earlier testing of the set-up process, and we should really address all of those issues before publishing a new version for further testing. Here are the items discovered during the testing of the first version of the software released earlier:

  • Installation error: Table ‘sys_hub_action_status_metadata’ does not exist
  • Not allowing update of property: x_11556_col_store.store_name
  • Not allowing update of property: x_11556_col_store.host_instance
  • In the setup, the instance name field doesn’t inform you that you only need the instance prefix, not the full url
  • You can only collaborate with one host

I was able to reproduce them all, and here is what I ended up doing for each:

Installation error: Table ‘sys_hub_action_status_metadata’ does not exist

I tried to find out some information on the purpose and use for this table, but I couldn’t really find anything that told me anything of value. I still believe that the ‘sys_hub_action_status_metadata’ table is related to a version or plugin that I have in my instance, but was not present in the instance on which the test installation was being performed. Since it didn’t seem as if it was anything that had anything to do with the operation of the application, I decided to just delete all references to it in the Update Set, just to avoid this issue. I’m not sure if that will cause any issues with any instances that actually do have this table, but it seemed like something worth a try and we’ll see what happens. If it causes a problem, I will not do that in the future and just throw in some release notes that say just ignore the error if it comes up. But let’s see if this works, first.

Not allowing update of property: x_11556_col_store.store_name
Not allowing update of property: x_11556_col_store.host_instance

This one took a little bit of research and a little bit of trial and error (mostly error!), but I eventually solved the problem by moving all updates to these properties into my global utilities so that the command was being executed by a global component instead of a scoped component. I did this once before with the gs.sleep() function, and this worked out just as well. Here is the function that I added to the global utilities:

setProperty: function(name, value) {
	gs.setProperty(name, value);
},

Once I created the function in the global utilities, it was just a matter of changing all gs.setProperty() function calls to csgu.setProperty() and that seemed to have done the trick.

In the setup, the instance name field doesn’t inform you that you only need the instance prefix, not the full url

Since the snh-form-field tag provides for a “help” attribute, which appears underneath the label, I simply updated the definition for that field to include a little help:

<snh-form-field
  snh-model="c.data.host_instance_id"
  snh-name="host_instance_id"
  snh-label="Host Instance ID"
  snh-help="Enter the instance ID only, not the full URL of the instance (https://{instance_id}.servicenow.com))"
  snh-required="c.data.instance_type == 'client'"
  ng-show="c.data.instance_type == 'client'"/>

This renders a little help text underneath the label, and looks like this when the page comes up:

Initial set-up screen with added help text for the Host Instance ID

That should resolve this issue.

You can only collaborate with one host

As I mentioned earlier when this was first reported, this is by design. Allowing multiple Host instances introduces a level of complexity with which I’m not quite ready to deal just yet. At this point, we will just file this one under Will not fix or Future release for now.

Other than these issues, I have not personally encountered or heard of any other issues with the set-up process, but that doesn’t mean that the testing is complete by any means. If anyone wants to join the testing process and you missed out on testing the earlier version, you will still have to work your way through the set-up process for any instances involved, so please report any issues with the set-up process as well as any issues with the application publication process.

Speaking of the application publication process, there is still yet another aspect of this process that remains to be developed. Once a new application has been published to the Host instance, the Host instance will need to push that version out to all of the other Client instances. This version does not include that functionality, but we will need to throw that in at some point. I just did not want to hold up the testing for that particular feature, since it really is a completely independent operation.

For those of you who are interested in participating in the testing, you will need this new Update Set, plus this additional global component that is not included in the scoped application. Also, if this is your first time installing the application, you will also need the latest version of snh-form-fields, which you can find here. As always, if you have any feedback, positive or negative, please leave the details in the comments. All information is welcome and much appreciated. Thanks in advance for your assistance.

Collaboration Store, Part XXVII

“It does not matter how slowly you go as long as you do not stop.”
Confucius

Last time we pushed the application record to the Host instance and now we have to do basically the same thing with the version record. The only difference really, other than the table and fields, is that a version record will always be a new record, so there is no need to determine if the record exists or not on the Host instance. This simplifies the code quite a bit. We still need to fetch the GlideRecord that will be sent over, so as we did with the application record, this will be the first order of business.

var versionGR = new GlideRecord('x_11556_col_store_member_application_version');
if (versionGR.get(answer.versionId)) {
	...
} else {
	answer = this.processError(answer, 'Invalid version record sys_id: ' + answer.versionId);
}

Once we have the record, we can gather up our System Properties and build the payload for the REST API call.

var host = gs.getProperty('x_11556_col_store.host_instance');
var token = gs.getProperty('x_11556_col_store.active_token');
var payload = {};
payload.member_application = answer.hostAppId;
payload.version = versionGR.getDisplayValue('version');

With our payload in hand, we can now create and configure our sn_ws.RESTMessageV2 object.

var request  = new sn_ws.RESTMessageV2();
request.setBasicAuth(this.WORKER_ROOT + host, token);
request.setRequestHeader("Accept", "application/json");
request.setHttpMethod('post');
request.setEndpoint('https://' + host + '.service-now.com/api/now/table/x_11556_col_store_member_application_version');
request.setRequestBody(JSON.stringify(payload, null, '\t'));

Now all that is left to do is to execute the request and check the response.

response = request.execute();
if (response.haveError()) {
	answer = this.processError(answer, 'Error returned from Host instance: ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
} else if (response.getStatusCode() != 201) {
	answer = this.processError(answer, 'Invalid HTTP Response Code returned from Host instance: ' + response.getStatusCode());
} else {
	var jsonString = response.getBody();
	var jsonObject = {};
	try {
		jsonObject = JSON.parse(jsonString);
	} catch (e) {
		answer = this.processError(answer, 'Unparsable JSON string returned from Host instance: ' + jsonString);
	}
	if (!answer.error) {
		answer.hostVerId = jsonObject.result.sys_id;
	}
}

Well, that was easy! Here’s the whole thing all put together.

processPhase6: function(answer) {
	var versionGR = new GlideRecord('x_11556_col_store_member_application_version');
	if (versionGR.get(answer.versionId)) {
		var host = gs.getProperty('x_11556_col_store.host_instance');
		var token = gs.getProperty('x_11556_col_store.active_token');
		var payload = {};
		payload.member_application = answer.hostAppId;
		payload.version = versionGR.getDisplayValue('version');
		var request  = new sn_ws.RESTMessageV2();
		request.setBasicAuth(this.WORKER_ROOT + host, token);
		request.setRequestHeader("Accept", "application/json");
		request.setHttpMethod('post');
		request.setEndpoint('https://' + host + '.service-now.com/api/now/table/x_11556_col_store_member_application_version');
		request.setRequestBody(JSON.stringify(payload, null, '\t'));
		response = request.execute();
		if (response.haveError()) {
			answer = this.processError(answer, 'Error returned from Host instance: ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
		} else if (response.getStatusCode() != 201) {
			answer = this.processError(answer, 'Invalid HTTP Response Code returned from Host instance: ' + response.getStatusCode());
		} else {
			var jsonString = response.getBody();
			var jsonObject = {};
			try {
				jsonObject = JSON.parse(jsonString);
			} catch (e) {
				answer = this.processError(answer, 'Unparsable JSON string returned from Host instance: ' + jsonString);
			}
			if (!answer.error) {
				answer.hostVerId = jsonObject.result.sys_id;
			}
		}
	} else {
		answer = this.processError(answer, 'Invalid version record sys_id: ' + answer.versionId);
	}

	return answer;
},

That takes care of 6 of the 7 steps. The last one that we will need to do will be to push the Update Set XML attachment over to the Host. That one may be a little more involved, so we will save that for next time out.

Collaboration Store, Part XXV

“Controlling complexity is the essence of computer programming.”
Brian Kernighan

Last time, we realized that we had a little bit of rework to do, and it turns out that we actually have to do a couple of things: 1) insert the missing step (attaching the XML to the version record), and 2) modify the ending point if the instance is the Host instance (there is no need to send the records to the Host instance if you are the Host instance). The first part is easy enough; just insert one more DIV for our missing step and then renumber all of the ones that follow:

<div class="row" id="phase_4" style="visibility: hidden; display: none;">
	<image id="loading_4" src="/images/loading_anim4.gif" style="width: 16px; height: 16px;"/>
	<image id="success_4" src="/images/check32.gif" style="width: 16px; height: 16px; visibility: hidden; display: none;"/>
	<image id="error_4" src="/images/delete_row.gif" style="width: 16px; height: 16px; visibility: hidden; display: none;"/>
	<span style="margin-left: 10px; font-weight:bold;">
		Attaching the Update Set XML to the Version record
	</span>
</div>
<div class="row" id="phase_5" style="visibility: hidden; display: none;">
	<image id="loading_5" src="/images/loading_anim4.gif" style="width: 16px; height: 16px;"/>
	<image id="success_5" src="/images/check32.gif" style="width: 16px; height: 16px; visibility: hidden; display: none;"/>
	<image id="error_5" src="/images/delete_row.gif" style="width: 16px; height: 16px; visibility: hidden; display: none;"/>
	<span style="margin-left: 10px; font-weight:bold;">
		Sending the Application record to the Host instance
	</span>
</div>
<div class="row" id="phase_6" style="visibility: hidden; display: none;">
	<image id="loading_6" src="/images/loading_anim4.gif" style="width: 16px; height: 16px;"/>
	<image id="success_6" src="/images/check32.gif" style="width: 16px; height: 16px; visibility: hidden; display: none;"/>
	<image id="error_6" src="/images/delete_row.gif" style="width: 16px; height: 16px; visibility: hidden; display: none;"/>
	<span style="margin-left: 10px; font-weight:bold;">
		Sending the Version record to the Host instance
	</span>
</div>
<div class="row" id="phase_7" style="visibility:hidden; display: none;">
	<image id="loading_7" src="/images/loading_anim4.gif" style="width: 16px; height: 16px;"/>
	<image id="success_7" src="/images/check32.gif" style="width: 16px; height: 16px; visibility: hidden; display: none;"/>
	<image id="error_7" src="/images/delete_row.gif" style="width: 16px; height: 16px; visibility: hidden; display: none;"/>
	<span style="margin-left: 10px; font-weight:bold;">
		Sending the Update Set XML to the Host instance
	</span>
</div>

And of course, we have to insert the missing step in our Script Include, which at this point is just another empty placeholder like all of the others. We’ll build out the details later as we come to that step.

For controlling the point at which we stop doing stuff, we will need to know if this instance is the Host instance or one of the Client instances. The easiest way to do that is to compare our scoped Host instance property with the stock instance_name property.

var isHost = gs.getProperty('instance_name') == gs.getProperty('x_11556_col_store.host_instance');

Then we just need to modify our original conditional statement that only assumed 6 steps and looked like this:

if (answer.phase < 7) {

… to one that will do all 7 steps for a Client instance and only the first 4 steps for a Host instance.

if (answer.phase < 5 || (answer.phase < 8 && !answer.isHost)) {

With that out of the way, we can return to building out the missing steps in the Script Include, starting with the newly added fourth step, which is to attach the Update Set XML to the version record. As you may recall, we already created an attachment record when we generated the XML, so now all we need to do is to transfer that attachment to our new version record. For that, we can return to our old friend, the GlideSysAttachment. This time, instead of creating the attachment record, we will be copying the attachment from one record to another.

var gsa = new GlideSysAttachment();
var newSysId = gsa.copy('sys_app', answer.appSysId, 'x_11556_col_store_member_application_version', answer.versionId);

Once we have copied the attachment from the Scoped Application record to the version record, we will want to delete the attachment record linked to the Scoped Application.

gsa.deleteAttachment(answer.attachmentId);

The last thing that we will need to do will be to update the attachment sys_id in our transfer object so that we will have the ID of the right attachment later on when we go to send it over to the Host.

answer.attachmentId = newSysId;

That makes this a relatively simple step in terms of code. The whole thing looks like this:

processPhase4: function(answer) {
	var gsa = new GlideSysAttachment();
	var newSysId = gsa.copy('sys_app', answer.appSysId, 'x_11556_col_store_member_application_version', answer.versionId);
	gsa.deleteAttachment(answer.attachmentId);
	answer.attachmentId = newSysId;

	return answer;
},

If the instance doing the publishing is a Host instance, this would actually be the end of the process, as far as publishing is concerned. We will still have to notify all of the other instances of the new version, but that’s an entirely separate process that we will deal with at some point in the future. But for our new Publish to Collaboration Store action, this is all that needs to be done if you are the Host. For all of the other instances, the application, version, and attachment records will all have to be sent over to the Host as a part of this process. We’ll get started on those steps of the process next time out.