Periodic Review, Part XI

“Take a deep breath, pick yourself up, dust yourself off and start all over again.”
Frank Sinatra

Last time, we finished up the code for the function that runs the notice distribution process. Now we need to come up with the actual content of those notices, which should direct the recipient to some function where they can indicate the disposition of the artifacts to be reviewed. In its simplest form, this would be a binary choice between keeping or discarding each item on the review list. However, there may be other dispositions for certain configurations, and there may be a need for additional options such as This item is no longer my responsibility or Remove this account only if it has been inactive for 90 days. To support such custom responses to a review request, we would need to add yet another table to our application to store the configured response options and link them to the configuration record. This adds a little bit of complexity to the process, but I think it would be worth doing to make things as flexible and useful as possible, so let’s go ahead and build that table now.

New Review Statement table

We will call our new table Review Statement and give it four fields, the reference to the configuration, an order, and a description and short description. We will want to link this as a related table on the configuration form so that statements can be easily added when setting up the configuration.

With that out of the way, we can start to visualize the form or page that the notice recipient would use to respond to the review notification. We could list the items to be reviewed down the page, and the configured choice across the top of the page, with a checkbox for each configured statement on each line containing an item. If there is more than one item to be reviewed, then a master checkbox at the top would also be helpful, so the recipient could simply check one box for the entire list of items. On the Now Platform, there are a number of different ways to construct such a page, but I am still partial to the Service Portal, so let’s build a Portal Widget for a Portal Page.

That was the plan, anyway.

Unfortunately, I did a really stupid thing before I got a chance to get started on that. It all started when I received a notice that my instance had some technical issues and that I needed to get rid of it and start over. Fair enough. I have had that particular instance for longer than I can remember, and I am sure that it was well past time to retire it and start all over with a new one. The notice said to be sure and back everything up before I wiped it out, but I have pretty much published every single thing that I have ever worked on, so I didn’t see any point in going through that. So I didn’t. I killed the old one and started over with a new one. Easy peasy.

What I neglected to consider was that this current project that I am working on right at the moment has not gotten far enough along for me to produce any public Update Sets, so there was no back up of everything that I have done so far. Oops! So now I have to go back and recreate all of the work that has been done so far, just to catch up to this point. I don’t mind doing things; in fact, I actually enjoy most of the stuff that I do here, and I mainly do it just for the fun of it. But I absolutely hate doing things twice. Now I just have to find the motivation to go back and do all of this again, just to get back to where I already was!

Periodic Review, Part X

“A person with a clear purpose will make progress on even the roughest road. A person with no purpose will make no progress on even the smoothest road.”
Thomas Carlyle

Last time, we brought our process far enough along to send out some empty notices, and now we need to create the content for those notices that will inform the recipient of the actions required of them. Before we jump into that, though, we still need to add a little bit more code to our Script Include to wrap the process and update the execution record with the results. At the end of our sendNotices function, let’s add the following code:

// finalize the execution
var itemLabel = 'item';
if (executionGR.total_items > 1) {
	itemLabel = 'items';
}
var noticeLabel = 'notice was';
if (noticeCt > 1) {
	noticeLabel = 'notices were';
}
executionGR.total_notices = noticeCt;
executionGR.run_end = new GlideDateTime();
executionGR.state = 'Complete';
executionGR.completion_code = 0;
executionGR.description = noticeCt + ' ' + noticeLabel + ' generated and sent out covering a total of ' + executionGR.total_items + ' ' + itemLabel + '.';
executionGR.update();

This will update the state of the execution and provide some statistics and a description of the execution. We also need to do one more thing to link the notice records to their corresponding email records, but before we can do that, we have to give the event time to fire and the process time to react to the event firing. We can use a gs.sleep command to do that, but since this is a scoped application, we will have to use a little workaround to get things to work.

// update the references to the sent email
var sleeper = new global.Sleeper();
sleeper.sleep(10000);

Once we know that the notice has been sent out, we can use the sys_watermark table to locate the information that we need to link the notice record to the associated email record.

// update the references to the sent email
var sleeper = new global.Sleeper();
sleeper.sleep(10000);
noticeGR.initialize();
noticeGR.addQuery('review_execution', executionGR.getUniqueValue());
noticeGR.query();
while (noticeGR.next()) {
	var watermark = noticeGR.getValue('email_watermark');
	if (watermark) {
		var watermarkGR = new GlideRecord('sys_watermark');
		if (watermarkGR.get('number', watermark.substring(4))) {
			noticeGR.setValue('email', watermarkGR.getValue('email'));
			noticeGR.update();
		}
	}
}

That should wrap up the process for sending out the requests to review the artifacts. At this point our entire Script Include now looks like this:

var PeriodicReviewUtils = Class.create();
PeriodicReviewUtils.prototype = {
    initialize: function() {
    },

	dailyProcess: function() {
		var toRun = [];
		var configurationGR = new GlideRecord('x_11556_periodic_r_review_configuration');
		var today = new GlideDate();
		configurationGR.addQuery('next_scheduled_date', today);
		configurationGR.orderBy('number');
		configurationGR.query();
		while (configurationGR.next()) {
			var execution = {};
			execution.configuration = configurationGR.getUniqueValue();
			execution.table = configurationGR.getDisplayValue('table');
			execution.filter = configurationGR.getDisplayValue('filter');
			toRun.push(execution);
		}
		if (toRun.length > 0) {
			gs.info('PeriodicReviewUtils.dailyProcess: Running ' + toRun.length + ' execution(s) today.');
			for (var i in toRun) {
				var thisRun = toRun[i];
				this.processExecution(thisRun.configuration, thisRun.table, thisRun.filter);
			}
			gs.info('PeriodicReviewUtils.dailyProcess: ' + toRun.length + ' execution(s) completed.');
		} else {
			gs.info('PeriodicReviewUtils.dailyProcess: Nothing scheduled to run today.');
		}
	},

	processExecution: function(configuration, table, filter) {
		gs.info('PeriodicReviewUtils.processExecution: ' + configuration + '; ' + table + '; ' + filter);

		// fetch system properties
		var systemFallbackAddress = gs.getProperty('x_11556_periodic_r.fallback_email_address');
		var systemEmailDomain = gs.getProperty('x_11556_periodic_r.email_domain');

		// create execution record
		var executionGR = new GlideRecord('x_11556_periodic_r_review_execution');
		executionGR.configuration = configuration;
		executionGR.run_date = new GlideDate();
		executionGR.run_start = new GlideDateTime();
		executionGR.state = 'Running';
		executionGR.short_description = executionGR.getDisplayValue('run_date') + ' review notices';
		executionGR.total_items = 0;
		executionGR.total_notices = 0;
		executionGR.completion_code = 0;
		executionGR.insert();
		var configurationGR = executionGR.configuration.getRefRecord();
		var fallbackRecipient = configurationGR.getDisplayValue('fallback_recipient');

		// create temporary notice record
		var noticeGR = new GlideRecord('x_11556_periodic_r_review_notice');
		noticeGR.review_execution = executionGR.getUniqueValue();
		noticeGR.recipient = configurationGR.fallback_recipient;
		noticeGR.short_description = 'Temporary notice record';
		noticeGR.insert();
		var tempNotice = noticeGR.getUniqueValue();

		// create notice item records from source table data
		var noticeItemGR = new GlideRecord('x_11556_periodic_r_review_notice_item');
		var itemCt = 0;
		var sourceGR = new GlideRecord(table);
		if (sourceGR.isValid()) {
			if (filter) {
				sourceGR.addEncodedQuery(filter);
			}
			sourceGR.orderBy(configurationGR.short_description_column);
			sourceGR.query();
			while (sourceGR.next()) {
				noticeItemGR.initialize();
				noticeItemGR.review_notice = tempNotice;
				noticeItemGR.id = sourceGR.getUniqueValue();
				noticeItemGR.short_description =  sourceGR.getDisplayValue(configurationGR.short_description_column);
				noticeItemGR.description = sourceGR.getDisplayValue(configurationGR.description_column);
				noticeItemGR.recipient = sourceGR.getValue(configurationGR.recipient_column);
				noticeItemGR.insert();
				itemCt++;
				var recipientGR = new GlideRecord('sys_user');
				recipientGR.get(noticeItemGR.getValue('recipient'));
				var notes = '';
				if (recipientGR.isValid()) {
					if (recipientGR.getValue('active')) {
						var email = recipientGR.getDisplayValue('email');
						if (email) {
							if (systemEmailDomain && !email.endsWith(systemEmailDomain)) {
								notes = 'Recipient email address is not an authoized email address; reverting to fallback recipient';
							}
						} else {
							notes = 'Specified recipient has no email address; reverting to fallback recipient';
						}
					} else {
						notes = 'Specified recipient is not active; reverting to fallback recipient';
					}
				} else {
					notes = 'Recipient column empty on source record; reverting to fallback recipient';
				}
				if (notes) {
					noticeItemGR.notes = notes;
					noticeItemGR.recipient = fallbackRecipient;
					noticeItemGR.update();
				}
			}
			if (itemCt > 0) {
				executionGR.total_items = itemCt;
				executionGR.update();
				this.sendNotices(configurationGR, executionGR, noticeGR, noticeItemGR, tempNotice);
			} else {

				// finalize the execution
				executionGR.total_items = 0;
				executionGR.total_notices = 0;
				executionGR.run_end = new GlideDateTime();
				executionGR.state = 'Complete';
				executionGR.completion_code = 0;
				executionGR.description = 'No items matched the filter criteria during this run, so no notices were sent out.';
				executionGR.update();
			}

			// delete the temporary notice
			noticeGR.get(tempNotice);
			noticeGR.deleteRecord();
		} else {

			// finalize the execution
			executionGR.total_items = 0;
			executionGR.total_notices = 0;
			executionGR.run_end = new GlideDateTime();
			executionGR.state = 'Failed';
			executionGR.completion_code = 1;
			executionGR.description = 'The specified source table in the configuration record is not valid; execution cannot proceed; aborting execution.';
			executionGR.update();
		}

		// set the next run date
		configurationGR.setValue('next_scheduled_date', this.calculateNextRunDate(configurationGR));
		configurationGR.update();
	},

	sendNotices: function(configurationGR, executionGR, noticeGR, noticeItemGR, tempNotice) {
		gs.info('PeriodicReviewUtils.sendNotices: ' + configurationGR.getUniqueValue() + '; ' + executionGR.getUniqueValue() + '; ' + noticeGR.getUniqueValue() + '; ' + noticeItemGR.getUniqueValue() + '; ' + tempNotice);
		var noticeCt = 0;

		var noticeItemGA = new GlideAggregate('x_11556_periodic_r_review_notice_item');
		noticeItemGA.addQuery('review_notice', tempNotice);
		noticeItemGA.addAggregate('COUNT');
		noticeItemGA.groupBy('recipient');
		noticeItemGA.orderBy('recipient');
		noticeItemGA.query();
		while (noticeItemGA.next()) {
			var recipient = noticeItemGA.getValue('recipient');
			noticeGR.initialize();
			noticeGR.review_execution = executionGR.getUniqueValue();
			noticeGR.recipient = recipient;
			noticeGR.short_description = executionGR.getDisplayValue('run_date') + ' review notice for ' + configurationGR.getDisplayValue('short_description');
			noticeGR.insert();
			noticeItemGR.initialize();
			noticeItemGR.addQuery('recipient', recipient);
			noticeItemGR.addQuery('review_notice', tempNotice);
			noticeItemGR.query();
			while (noticeItemGR.next()) {
				noticeItemGR.review_notice = noticeGR.getUniqueValue();
				noticeItemGR.update();
			}
			// now you need to send out the notice, passing in the notice record for variables
			gs.eventQueue('x_11556_periodic_r.ReviewNotice', noticeGR, noticeGR.recipient, noticeGR.getUniqueValue());
			noticeCt++;
		}

		// finalize the execution
		var itemLabel = 'item';
		if (executionGR.total_items > 1) {
			itemLabel = 'items';
		}
		var noticeLabel = 'notice was';
		if (noticeCt > 1) {
			noticeLabel = 'notices were';
		}
		executionGR.total_notices = noticeCt;
		executionGR.run_end = new GlideDateTime();
		executionGR.state = 'Complete';
		executionGR.completion_code = 0;
		executionGR.description = noticeCt + ' ' + noticeLabel + ' generated and sent out covering a total of ' + executionGR.total_items + ' ' + itemLabel + '.';
		executionGR.update();

		// update the references to the sent email
		var sleeper = new global.Sleeper();
		sleeper.sleep(10000);
		noticeGR.initialize();
		noticeGR.addQuery('review_execution', executionGR.getUniqueValue());
		noticeGR.query();
		while (noticeGR.next()) {
			var watermark = noticeGR.getValue('email_watermark');
			if (watermark) {
				var watermarkGR = new GlideRecord('sys_watermark');
				if (watermarkGR.get('number', watermark.substring(4))) {
					noticeGR.setValue('email', watermarkGR.getValue('email'));
					noticeGR.update();
				}
			}
		}
	},

	calculateNextRunDate: function(configurationGR) {
        var runDate = new Date();
        var frequency = configurationGR.getValue('frequency');
        var days = 0;
        var months = 0;
        if (frequency == 'daily') {
            days = 1;
        } else if (frequency == 'weekly') {
            days = 7;
        } else if (frequency == 'biweekly') {
            days = 14;
        } else if (frequency == 'monthly') {
            months = 1;
        } else if (frequency == 'bimonthly') {
            months = 2;
        } else if (frequency == 'quarterly') {
            months = 3;
        } else if (frequency == 'semiannually') {
            months = 6;
        } else if (frequency == 'annually') {
            months = 12;
        } else if (frequency == 'biannually') {
            months = 24;
        }
        if (days > 0) {
            runDate.setDate(runDate.getDate() + days);
        } else {
            runDate.setMonth(runDate.getMonth() + months);
        }
        return JSON.stringify(runDate).substring(1, 11);
    },

    type: 'PeriodicReviewUtils'
};

We still have to build the content for the notices, and a way for the notice recipients to communicate their responses back to the system so that the appropriate action can be taken, so let’s jump into all of that next time out.

Periodic Review, Part IX

“Hide not your Talents, they for Use were made. What’s a Sun-Dial in the shade!”
Benjamin Franklin

Last time, we wrapped up most of the work on the script that will handle the review process right up to the point where we need to send out the notice to the recipient. Today we will look at one way to send out an email notification and then build the notice that we will want to send out.

One of the easiest ways to trigger an outbound email is through the use of a System Event, not to be confused with an Event Management Event, which is an entirely different animal. And neither one of those is related in any way to a ServiceNow Event, but now we are really getting off track. To create a new Event, we will navigate to the Event Registry and then click on the New button.

New System Event

Once we have created our new event, we can create an Email Notification and have the notification triggered by this event. To create our new Email Notification, we will navigate to All > System Notification > Email > Notifications and click on the New button. At this point, let’s not worry too much about the content of the message and let’s just do enough so that we can test things out and make sure that it all works. Once we establish that the email is actually sent out, we can go back in and create the message body that will work for our requirements.

New Email Notification

Under the When to send tab, we select Event is fired from the Send when options and then we select our new event from the Event name options. Then on the Who will receive tab, we check the box labeled Event parm 1 contains recipient, which will allow us to send in the recipient as one of the event parameters.

Identifying the intended recipient

In the What it will contain tab, we will just put the word Testing in the subject and body for now and then save the record so that we can run a test. Now we need to modify our Script Include to initiate the event, passing in the appropriate parameters, namely the notification record and the intended recipient. We will replace this line that we added for earlier testing:

gs.info('This is where we would send a notice to ' + noticeGR.getDisplayValue('recipient'));

… with this new code to add a new instance of the event to the queue:

// now you need to send out the notice, passing in the notice record for variables
gs.eventQueue('x_11556_periodic_r.ReviewNotice', noticeGR, noticeGR.recipient, noticeGR.getUniqueValue());
noticeCt++;

After we save that we can pop back over to Scripts – Background and see if all of this results in some email being sent out.

New test results

Well, that looks pretty good, but let’s take a look at the email logs and see if we actually sent out some notices.

Notification emails generated

OK, that works! Now that we know that our process will send out the notices to the designated recipients, the next thing that we will need to do is to come up with the content of the notice. That sounds like a good project for our next installment.

Periodic Review, Part VIII

“Code is like humor. When you have to explain it, it’s bad.”
Cory House

Last time, we built out the processExecution function in our utility Script Include, and now we need to continue on by building out the sendNotices function. Before we build it out completely, though, let’s just create a quick stub so that we can test out the work so far. This ought to do the trick.

sendNotices: function(configurationGR, executionGR, noticeGR, noticeItemGR, tempNotice) {
	gs.info('PeriodicReviewUtils.sendNotices: ' + configurationGR.getUniqueValue() + '; ' + executionGR.getUniqueValue() + '; ' + noticeGR.getUniqueValue() + '; ' + noticeItemGR.getUniqueValue() + '; ' + tempNotice);
}

Now we just need to make sure that we have the next run date set correctly on our lone example configuration and then we can navigate over to Scripts – Background and type in our earlier testing code.

var pru = new PeriodicReviewUtils();
pru.dailyProcess();
gs.info('Done!');

And once again we can hit that Run script button and see what happens.

Initial test results

Well, that’s not good! It seems that there is some kind of problem with our calculateNextRunDate function. It’s always something! But then, that’s why I like to test things out as I go rather than waiting for the entire thing to be built to try things out. The good news is that it seems to have run all of the way through and made it all the way to the end, where it updates the run date, before it crashed and burned. Let me dig into that function and see what might be wrong.

(insert brief debugging pause)

OK, I found the issue. Apparently, it doesn’t like this line:

var runDate = new Date(configurationGR.getDisplayValue('next_scheduled_date'));

The next scheduled date on the configuration record is a GlideDate, and the display value is year, month, and then day. The Javascript Date constructor doesn’t like that, although I know that I have done that in the past without issue. Still, it doesn’t really matter because the value should be today’s date; otherwise, we wouldn’t be running right now. So we can simple change that to this:

var runDate = new Date();

That should still get us the date that we are looking for, so let’s hop back over to Scripts – Background and try this again.

Successful test results

That’s better. So far, so good. Now we need to actually build out that sendNotices function to group all of the notice item records by recipient, create a notice record for each recipient, and then send out the notices. We can develop an unduplicated list of recipients by using a GlideAggregate.

var noticeItemGA = new GlideAggregate('x_11556_periodic_r_review_notice_item');
noticeItemGA.addQuery('review_notice', tempNotice);
noticeItemGA.addAggregate('COUNT');
noticeItemGA.groupBy('recipient');
noticeItemGA.orderBy('recipient');
noticeItemGA.query();
while (noticeItemGA.next()) {
	var recipient = noticeItemGA.getValue('recipient');
	...
}

Once we have the recipient, we can create the notice record for that recipient.

noticeGR.initialize();
noticeGR.review_execution = executionGR.getUniqueValue();
noticeGR.recipient = recipient;
noticeGR.short_description = executionGR.getDisplayValue('run_date') + ' review notice for ' + configurationGR.getDisplayValue('short_description');
noticeGR.insert();

Now that a notice record for this recipient has been created, we need to go back and find all of the notice item records that are associated with this recipient and update the reference to the notice record from the temporary notice to the newly created notice for this recipient.

noticeItemGR.initialize();
noticeItemGR.addQuery('recipient', recipient);
noticeItemGR.addQuery('review_notice', tempNotice);
noticeItemGR.query();
while (noticeItemGR.next()) {
	noticeItemGR.review_notice = noticeGR.getUniqueValue();
	noticeItemGR.update();
}

At this point, we should be able to send out the notice, but as that is a rather complex process in and of itself, let’s just stub things out for now so that we can test out the process so far.

gs.info('This is where we would send a notice to ' + noticeGR.getDisplayValue('recipient'));

All together, our new sendNotices function now looks like this.

sendNotices: function(configurationGR, executionGR, noticeGR, noticeItemGR, tempNotice) {
	gs.info('PeriodicReviewUtils.sendNotices: ' + configurationGR.getUniqueValue() + '; ' + executionGR.getUniqueValue() + '; ' + noticeGR.getUniqueValue() + '; ' + noticeItemGR.getUniqueValue() + '; ' + tempNotice);
var noticeCt = 0;

	var noticeItemGA = new GlideAggregate('x_11556_periodic_r_review_notice_item');
	noticeItemGA.addQuery('review_notice', tempNotice);
	noticeItemGA.addAggregate('COUNT');
	noticeItemGA.groupBy('recipient');
	noticeItemGA.orderBy('recipient');
	noticeItemGA.query();
	while (noticeItemGA.next()) {
		var recipient = noticeItemGA.getValue('recipient');
		noticeGR.initialize();
		noticeGR.review_execution = executionGR.getUniqueValue();
		noticeGR.recipient = recipient;
		noticeGR.short_description = executionGR.getDisplayValue('run_date') + ' review notice for ' + configurationGR.getDisplayValue('short_description');
		noticeGR.insert();
		noticeItemGR.initialize();
		noticeItemGR.addQuery('recipient', recipient);
		noticeItemGR.addQuery('review_notice', tempNotice);
		noticeItemGR.query();
		while (noticeItemGR.next()) {
			noticeItemGR.review_notice = noticeGR.getUniqueValue();
			noticeItemGR.update();
		}
		gs.info('This is where we would send a notice to ' + noticeGR.getDisplayValue('recipient'));
	}
}

That should be enough to give things the old college try. Let’s pop back over to Scripts – Background and paste in our testing code and see what we get.

Testing results

Nice! Obviously, a lot more testing is in order before we can say for sure that everything works for every scenario, but things are looking pretty good. Next time, let’s see if we can actually send out a notice.

Periodic Review, Part VII

“Whether you want to uncover the secrets of the universe, or you just want to pursue a career in the 21st century, basic computer programming is an essential skill to learn.”
Stephen Hawking

Last time, we got started on our script that will run the review process by building and testing the primary function that finds all of the configurations that will need to run. Today we need to continue on with the script, beginning with the process for a single configuration’s execution. Before we get started, though, we need to talk a little bit about making sure that the notice will go to somebody.

Both the configuration record and the system as a whole have a default recipient/email address. The system default email address is contained in a System Property and the configuration default recipient is on the configuration record. There is a field on the source record for the recipient, and if that recipient record is valid and active and contains an email address, then that will be the address that we will want to use. If not, then we will want to use the configuration fallback recipient, and if there is some issue in the notice process with that recipient’s email address, then we will end up using the system fallback address. This is to ensure that the notice will actually get to a real person who can take action on the review request.

There is also another optional System Property that can be used to limit notifications to a specific email domain. If that value is present, and the recipient email is not an email address of that domain, then the fallback recipient will be used as well. All of this should be documented in the notice item record. So the first thing that we will want to do will be to pull those System Properties.

// fetch system properties
var systemFallbackAddress = gs.getProperty('x_11556_periodic_r.fallback_email_address');
var systemEmailDomain = gs.getProperty('x_11556_periodic_r.email_domain');

The next thing that we will want to do will be to create a record in the Review Execution table to memorialize this execution.

// create execution record
var executionGR = new GlideRecord('x_11556_periodic_r_review_execution');
executionGR.configuration = configuration;
executionGR.run_date = new GlideDate();
executionGR.run_start = new GlideDateTime();
executionGR.state = 'Running';
executionGR.short_description = executionGR.getDisplayValue('run_date') + ' review notices';
executionGR.total_items = 0;
executionGR.total_notices = 0;
executionGR.completion_code = 0;
executionGR.insert();

This is standard GlideRecord stuff, so we don’t really need to go into a lot of detail here other than to say that we link it back to the configuration that is being executed and initialize all of the other values. Once we have our execution record created, we can pull the configuration record from the reference, and once we have it, we can set up the fallback recipient for this configuration.

var configurationGR = executionGR.configuration.getRefRecord();
var fallbackRecipient = configurationGR.getDisplayValue('fallback_recipient');

Now we need to go fetch all of the items that need to be reviewed this cycle, but because we want to consolidate all of the items for any given recipient into a single notice, we will first need to fetch them all, and then once we have the full list, group them by recipient. To facilitate that, we can create a temporary notice record and assign all of the items to that notice, and once we run through all of the query results, we can then fetch them back by recipient. So let’s create that temporary notice record now.

// create temporary notice record
var noticeGR = new GlideRecord('x_11556_periodic_r_review_notice');
noticeGR.review_execution = executionGR.getUniqueValue();
noticeGR.recipient = configurationGR.fallback_recipient;
noticeGR.short_description = 'Temporary notice record';
noticeGR.insert();
var tempNotice = noticeGR.getUniqueValue();

Once we have our notice record, we can use the table and filter from the configuration record to find all of the items to be reviewed.

// create notice item records from source table data
var noticeItemGR = new GlideRecord('x_11556_periodic_r_review_notice_item');
var itemCt = 0;
var sourceGR = new GlideRecord(table);
if (sourceGR.isValid()) {
	if (filter) {
		sourceGR.addEncodedQuery(filter);
	}
	sourceGR.orderBy(configurationGR.short_description_column);
	sourceGR.query();
	while (sourceGR.next()) {
		...
	}
	...
} else {
	...
}

Then, inside the query loop, we can build a record in the Review Notice Item table for each item.

noticeItemGR.initialize();
noticeItemGR.review_notice = tempNotice;
noticeItemGR.id = sourceGR.getUniqueValue();
noticeItemGR.short_description =  sourceGR.getDisplayValue(configurationGR.short_description_column);
noticeItemGR.description = sourceGR.getDisplayValue(configurationGR.description_column);
noticeItemGR.recipient = sourceGR.getValue(configurationGR.recipient_column);
noticeItemGR.insert();
itemCt++;

Once the notice item record has been created, we can inspect the recipient and come up with a valid recipient to which we will send the notice.

var recipientGR = new GlideRecord('sys_user');
recipientGR.get(noticeItemGR.getValue('recipient'));
var notes = '';
if (recipientGR.isValid()) {
	if (recipientGR.getValue('active')) {
		var email = recipientGR.getDisplayValue('email');
		if (email) {
			if (systemEmailDomain && !email.endsWith(systemEmailDomain)) {
				notes = 'Recipient email address is not an authoized email address; reverting to fallback recipient';
			}
		} else {
			notes = 'Specified recipient has no email address; reverting to fallback recipient';
		}
	} else {
		notes = 'Specified recipient is not active; reverting to fallback recipient';
	}
} else {
	notes = 'Recipient column empty on source record; reverting to fallback recipient';
}
if (notes) {
	noticeItemGR.notes = notes;
	noticeItemGR.recipient = fallbackRecipient;
	noticeItemGR.update();
}

Once we complete the loop, we will want to see if the query returned any items, and either process the items or close out this execution.

if (itemCt > 0) {
	executionGR.total_items = itemCt;
	executionGR.update();
	this.sendNotices(configurationGR, executionGR, noticeGR, noticeItemGR, tempNotice);
} else {

	// finalize the execution
	executionGR.total_items = 0;
	executionGR.total_notices = 0;
	executionGR.run_end = new GlideDateTime();
	executionGR.state = 'Complete';
	executionGR.completion_code = 0;
	executionGR.description = 'No items matched the filter criteria during this run, so no notices were sent out.';
	executionGR.update();
}

And once that is done, whether we had any items returned or not, we will want to delete that temporary notice record.

// delete the temporary notice
noticeGR.get(tempNotice);
noticeGR.deleteRecord();

One last thing: we need to do something in the event that we find ourselves in the else branch of the if (sourceGR.isValid()) { conditional. This basically means that the source table specified on the configuration record is not valid and so we cannot look for any items there. This is basically a failure of the execution, so we need to update the execution record to reflect that.

// finalize the execution
executionGR.total_items = 0;
executionGR.total_notices = 0;
executionGR.run_end = new GlideDateTime();
executionGR.state = 'Failed';
executionGR.completion_code = 1;
executionGR.description = 'The specified source table in the configuration record is not valid; execution cannot proceed; aborting execution.';
executionGR.update();

Failure is not an option that we originally set up for that field, so we will have to go back into the dictionary at some point and fix that to make that code work. Finally, regardless of how the execution turned out, we need to update the configuration record with the next run date, which is calculated in the function that we added for that purpose earlier.

// set the next run date
configurationGR.setValue('next_scheduled_date', this.calculateNextRunDate(configurationGR));
configurationGR.update();

With the addition of this completed function, our utility Script Include now looks like this.

var PeriodicReviewUtils = Class.create();
PeriodicReviewUtils.prototype = {
    initialize: function() {
    },

	dailyProcess: function() {
		var toRun = [];
		var configurationGR = new GlideRecord('x_11556_periodic_r_review_configuration');
		var today = new GlideDate();
		configurationGR.addQuery('next_scheduled_date', today);
		configurationGR.orderBy('number');
		configurationGR.query();
		while (configurationGR.next()) {
			var execution = {};
			execution.configuration = configurationGR.getUniqueValue();
			execution.table = configurationGR.getDisplayValue('table');
			execution.filter = configurationGR.getDisplayValue('filter');
			toRun.push(execution);
		}
		if (toRun.length > 0) {
			gs.info('PeriodicReviewUtils.dailyProcess: Running ' + toRun.length + ' execution(s) today.');
			for (var i in toRun) {
				var thisRun = toRun[i];
				this.processExecution(thisRun.configuration, thisRun.table, thisRun.filter);
			}
			gs.info('PeriodicReviewUtils.dailyProcess: ' + toRun.length + ' execution(s) completed.');
		} else {
			gs.info('PeriodicReviewUtils.dailyProcess: Nothing scheduled to run today.');
		}
	},

	processExecution: function(configuration, table, filter) {
		gs.info('PeriodicReviewUtils.dailyProcess: ' + configuration + '; ' + table + '; ' + filter);

		// fetch system properties
		var systemFallbackAddress = gs.getProperty('x_11556_periodic_r.fallback_email_address');
		var systemEmailDomain = gs.getProperty('x_11556_periodic_r.email_domain');

		// create execution record
		var executionGR = new GlideRecord('x_11556_periodic_r_review_execution');
		executionGR.configuration = configuration;
		executionGR.run_date = new GlideDate();
		executionGR.run_start = new GlideDateTime();
		executionGR.state = 'Running';
		executionGR.short_description = executionGR.getDisplayValue('run_date') + ' review notices';
		executionGR.total_items = 0;
		executionGR.total_notices = 0;
		executionGR.completion_code = 0;
		executionGR.insert();
		var configurationGR = executionGR.configuration.getRefRecord();
		var fallbackRecipient = configurationGR.getDisplayValue('fallback_recipient');

		// create temporary notice record
		var noticeGR = new GlideRecord('x_11556_periodic_r_review_notice');
		noticeGR.review_execution = executionGR.getUniqueValue();
		noticeGR.recipient = configurationGR.fallback_recipient;
		noticeGR.short_description = 'Temporary notice record';
		noticeGR.insert();
		var tempNotice = noticeGR.getUniqueValue();

		// create notice item records from source table data
		var noticeItemGR = new GlideRecord('x_11556_periodic_r_review_notice_item');
		var itemCt = 0;
		var sourceGR = new GlideRecord(table);
		if (sourceGR.isValid()) {
			if (filter) {
				sourceGR.addEncodedQuery(filter);
			}
			sourceGR.orderBy(configurationGR.short_description_column);
			sourceGR.query();
			while (sourceGR.next()) {
				noticeItemGR.initialize();
				noticeItemGR.review_notice = tempNotice;
				noticeItemGR.id = sourceGR.getUniqueValue();
				noticeItemGR.short_description =  sourceGR.getDisplayValue(configurationGR.short_description_column);
				noticeItemGR.description = sourceGR.getDisplayValue(configurationGR.description_column);
				noticeItemGR.recipient = sourceGR.getValue(configurationGR.recipient_column);
				noticeItemGR.insert();
				itemCt++;
				var recipientGR = new GlideRecord('sys_user');
				recipientGR.get(noticeItemGR.getValue('recipient'));
				var notes = '';
				if (recipientGR.isValid()) {
					if (recipientGR.getValue('active')) {
						var email = recipientGR.getDisplayValue('email');
						if (email) {
							if (systemEmailDomain && !email.endsWith(systemEmailDomain)) {
								notes = 'Recipient email address is not an authoized email address; reverting to fallback recipient';
							}
						} else {
							notes = 'Specified recipient has no email address; reverting to fallback recipient';
						}
					} else {
						notes = 'Specified recipient is not active; reverting to fallback recipient';
					}
				} else {
					notes = 'Recipient column empty on source record; reverting to fallback recipient';
				}
				if (notes) {
					noticeItemGR.notes = notes;
					noticeItemGR.recipient = fallbackRecipient;
					noticeItemGR.update();
				}
			}
			if (itemCt > 0) {
				executionGR.total_items = itemCt;
				executionGR.update();
				this.sendNotices(configurationGR, executionGR, noticeGR, noticeItemGR, tempNotice);
			} else {

				// finalize the execution
				executionGR.total_items = 0;
				executionGR.total_notices = 0;
				executionGR.run_end = new GlideDateTime();
				executionGR.state = 'Complete';
				executionGR.completion_code = 0;
				executionGR.description = 'No items matched the filter criteria during this run, so no notices were sent out.';
				executionGR.update();
			}

			// delete the temporary notice
			noticeGR.get(tempNotice);
			noticeGR.deleteRecord();
		} else {
			// finalize the execution
			executionGR.total_items = 0;
			executionGR.total_notices = 0;
			executionGR.run_end = new GlideDateTime();
			executionGR.state = 'Failed';
			executionGR.completion_code = 1;
			executionGR.description = 'The specified source table in the configuration record is not valid; execution cannot proceed; aborting execution.';
			executionGR.update();
		}

		// set the next run date
		configurationGR.setValue('next_scheduled_date', this.calculateNextRunDate(configurationGR));
		configurationGR.update();
	},

	calculateNextRunDate: function(configurationGR) {
		var runDate = new Date(configurationGR.getDisplayValue('next_scheduled_date'));
		var frequency = configurationGR.getValue('frequency');
		var days = 0;
		var months = 0;
		if (frequency == 'daily') {
			days = 1;
		} else if (frequency == 'weekly') {
			days = 7;
		} else if (frequency == 'biweekly') {
			days = 14;
		} else if (frequency == 'monthly') {
			months = 1;
		} else if (frequency == 'bimonthly') {
			months = 2;
		} else if (frequency == 'quarterly') {
			months = 3;
		} else if (frequency == 'semiannually') {
			months = 6;
		} else if (frequency == 'annually') {
			months = 12;
		} else if (frequency == 'biannually') {
			months = 24;
		}
		if (days > 0) {
			runDate.setDate(runDate.getDate() + days);
		} else {
			runDate.setMonth(runDate.getMonth() + months);
		}
		return JSON.stringify(runDate).substring(1, 11);
	},

    type: 'PeriodicReviewUtils'
};

Next time, we will stub out that sendNotices function and give things another test or two, and then actually build out that function to group the items by recipient and send out the notices.

Periodic Review, Part VI

“One man’s crappy software is another man’s full time job.”
Jessica Gaston

Last time, we wrapped up the work on creating all of the tables that we will need to support the review process. Now it is time to create that process using those tables. Before we jump into that, though, we should talk a little bit about how the process should work.

There are a couple of different ways to go about things that occur on a periodic basis. One way is to handle things the way most US states handle the renewals for driver’s licenses: each driver has their own renewal date based on something like their birthday or the day on which they originally got their license. Another way would be to handle things like the IRS does for federal income taxes: every person has the same date each year, sometime in the middle of April. One way spreads out the work across the entire year and the other results in all of the work coming at once at a particular time. Our process should be able to handle both approaches, and between the frequency value and the filter value in the configuration record, there should be a way to set things out to work out for either one.

Speaking of the frequency field, we should probably set up a list of specific choices for that guy so that our code knows what to expect. We can do that in the Choices tab of the Related Lists on the Dictionary Entry form of the frequency field.

Choice list for configuration frequency field

We may come up with other options in the future, but this list will do for now. The main purpose of this field is so that the process can establish the date of the next run for this particular configuration. Now that we know the values, we can create a simple function to set the next run date based on the current run date and the specified frequency.

calculateNextRunDate: function(configurationGR) {
	var runDate = new Date(configurationGR.getDisplayValue('next_scheduled_date'));
	var frequency = configurationGR.getValue('frequency');
	var days = 0;
	var months = 0;
	if (frequency == 'daily') {
		days = 1;
	} else if (frequency == 'weekly') {
		days = 7;
	} else if (frequency == 'biweekly') {
		days = 14;
	} else if (frequency == 'monthly') {
		months = 1;
	} else if (frequency == 'bimonthly') {
		months = 2;
	} else if (frequency == 'quarterly') {
		months = 3;
	} else if (frequency == 'semiannually') {
		months = 6;
	} else if (frequency == 'annually') {
		months = 12;
	} else if (frequency == 'biannually') {
		months = 24;
	}
	if (days > 0) {
		runDate.setDate(runDate.getDate() + days);
	} else {
		runDate.setMonth(runDate.getMonth() + months);
	}
	return JSON.stringify(runDate).substring(1, 11);
}

Of course, we won’t need that until the end of the run, as that will be one of the last things that we will do before we move on to the next configuration that we will be running today. So let’s back things up a bit and look at things from a little higher perspective.

To run the review process, we will set up a Scheduled Job that will run once a day, presumably at some time where the demand on the system is lower, and that job will call a Script Include function that will do all of the heavy lifting. That function will then query the database for any configuration records where the next run date is the current date. For each returned record, the process will use the filter field on the configuration record to query the table specified on the configuration record to obtain all of the items up for review. Next we will want to organize all of the items by recipient and send each recipient a notice that contains all of the items up for review for which that recipient is responsible. To house all of the functions required to perform all of thee tasks, we can create a single utility Script Include and call it PeriodicReviewUtils.

New utility Script Include

We’ll call our first function dailyProcess set up an object array called toRun and populate it with data from the returned configuration records. Then we will spin through the array and pass the values from that object to another function called processExecution that will handle the process for a single configuration.

dailyProcess: function() {
	var toRun = [];
	var configurationGR = new GlideRecord('x_11556_periodic_r_review_configuration');
	var today = new GlideDate();
	configurationGR.addQuery('next_scheduled_date', today);
	configurationGR.orderBy('number');
	configurationGR.query();
	while (configurationGR.next()) {
		var execution = {};
		execution.configuration = configurationGR.getUniqueValue();
		execution.table = configurationGR.getDisplayValue('table');
		execution.filter = configurationGR.getDisplayValue('filter');
		toRun.push(execution);
	}
	if (toRun.length > 0) {
		gs.info('PeriodicReviewUtils.dailyProcess: Running ' + toRun.length + ' execution(s) today.');
		for (var i in toRun) {
			var thisRun = toRun[i];
			this.processExecution(thisRun.configuration, thisRun.table, thisRun.filter);
		}
		gs.info('PeriodicReviewUtils.dailyProcess: ' + toRun.length + ' execution(s) completed.');
	} else {
		gs.info('PeriodicReviewUtils.dailyProcess: Nothing scheduled to run today.');
	}
}

I like to test things as I go along, just to make sure that I didn’t fat finger something along the way, and we can stop and do that here if we temporarily stub out the processExecution function to just spit out the values that were passed to it. That would make it look something like this:

processExecution: function(configuration, table, filter) {
	gs.info('PeriodicReviewUtils.dailyProcess: ' + configuration + '; ' + table + '; ' + filter);
}

We only have the one configuration record that we set up for Service Accounts right now, but that should be enough to validate the code that we have so far, which currently looks like this.

var PeriodicReviewUtils = Class.create();
PeriodicReviewUtils.prototype = {
    initialize: function() {
    },

	dailyProcess: function() {
		var toRun = [];
		var configurationGR = new GlideRecord('x_11556_periodic_r_review_configuration');
		var today = new GlideDate();
		configurationGR.addQuery('next_scheduled_date', today);
		configurationGR.orderBy('number');
		configurationGR.query();
		while (configurationGR.next()) {
			var execution = {};
			execution.configuration = configurationGR.getUniqueValue();
			execution.table = configurationGR.getDisplayValue('table');
			execution.filter = configurationGR.getDisplayValue('filter');
			toRun.push(execution);
		}
		if (toRun.length > 0) {
			gs.info('PeriodicReviewUtils.dailyProcess: Running ' + toRun.length + ' execution(s) today.');
			for (var i in toRun) {
				var thisRun = toRun[i];
				this.processExecution(thisRun.configuration, thisRun.table, thisRun.filter);
			}
			gs.info('PeriodicReviewUtils.dailyProcess: ' + toRun.length + ' execution(s) completed.');
		} else {
			gs.info('PeriodicReviewUtils.dailyProcess: Nothing scheduled to run today.');
		}
	},

	processExecution: function(configuration, table, filter) {
		gs.info('PeriodicReviewUtils.dailyProcess: ' + configuration + '; ' + table + '; ' + filter);
	},

	calculateNextRunDate: function(configurationGR) {
		var runDate = new Date(configurationGR.getDisplayValue('next_scheduled_date'));
		var frequency = configurationGR.getValue('frequency');
		var days = 0;
		var months = 0;
		if (frequency == 'daily') {
			days = 1;
		} else if (frequency == 'weekly') {
			days = 7;
		} else if (frequency == 'biweekly') {
			days = 14;
		} else if (frequency == 'monthly') {
			months = 1;
		} else if (frequency == 'bimonthly') {
			months = 2;
		} else if (frequency == 'quarterly') {
			months = 3;
		} else if (frequency == 'semiannually') {
			months = 6;
		} else if (frequency == 'annually') {
			months = 12;
		} else if (frequency == 'biannually') {
			months = 24;
		}
		if (days > 0) {
			runDate.setDate(runDate.getDate() + days);
		} else {
			runDate.setMonth(runDate.getMonth() + months);
		}
		return JSON.stringify(runDate).substring(1, 11);
	},

    type: 'PeriodicReviewUtils'
};

To run a quick test, we can navigate over to Scripts – Background and use this little test script.

var pru = new PeriodicReviewUtils();
pru.dailyProcess();
gs.info('Done!');

Then all we need to do is to click on the Run script button and see what happens.

First test run results

Well, as usual, there is good news and bad news here. The good news is that it didn’t crash and it made all the way through without an error. The bad news is that I forgot to set the next run date field on the lone configuration record, so the process didn’t get very far. On the bright side, though, we did successfully test the scenario where there is nothing to run, so we can check that off of the list. Now let’s update the configuration and try this again.

Second test run results

There, that’s better! And once again, we ran through without any errors, so I think we are off to a good start. Next time, we will build out the code for that processExecution function and give that guy a little test.

Periodic Review, Part V

“I find that the best way to do things is to constantly move forward and to never doubt anything and keep moving forward.”
John Frusciante

Last time, we wrapped up the initial testing of our first table and related form, so now it is time to move on to the other tables on our list. The next table that we will want to deal with is the table that contains the details for each run of the review process, which we will call Review Execution.

New Review Execution table

The first field that we will want to have is a reference back to the associated configuration record. We will call that one configuration. Then we will need a short and long description field, the run date, start and end times, some counters, a current state, and a completion code.

Fields for the new Review Execution table

Once we save all of the fields, we can pull up the form and arrange everything to suit our needs.

Form layout for the new Review Execution table

One last thing that we will want to do with this table is to set up a couple of state values, one for running and one for completed, and set the default value to running so that whenever a new record is created, it is automatically set to the running state.

Choices and default for the state field

That should take care of the Review Execution table. Now we will need to build tables for every notice that comes out of the review process and every item that appears on each notice. Let’s start with the notice table first, which we can call Review Notice.

Review Notice table

This table will have several reference fields, the first one being a link back to the associated execution. Other references will link to the email that was sent out, the recipient of the email, and the user who responded to the notice.

Fields for the new Review Notice table

With that completed, we can then lay out the form in the way that we would like it to appear.

Form layout for the Review Notice table

The last table that we will need to build before we can start looking at the actual process of sending out the notices is the table of items associated with each notice. We will call that one Review Notice Item.

Review Notice Item table

This one will contain reference fields as well, including a reference to the associated notice, but the reference to the actual item being reviewed will be a little bit different. Because we are setting this up as a generic process that can review virtually any item, for the link to the item we will be using a Document ID field for the reference. More on that a little later, but for now, here are the fields for this table.

Fields for the new Review Notice Item table

Both Reference type fields and Document ID type fields contain sys_ids, but on a reference field, the table containing the record with that sys_id is defined as a part of the field definition. On a Document ID field, the record could potentially be on any table in the system, so you need a second, dependent field of type Table to specify which table contains the record referenced. Fortunately for our purposes, the field does not need to be on the same table as the one that contains the Document ID. The table field that we need is actually a part of the Review Configuration where we define what items are to be reviewed. To set that up, we need to go into the field definition and select that field in the Dependent field section of the form.

Document ID dependent field configuration

With that out of the way, we can once again lay out the fields as we like on the associated form.

Form Layout for the new Review Notice Item table

That should be all we need to get started on building the actual process that performs the review. There will eventually be one or more tables yet to define, but we can save that until they are needed. For now, let’s set that aside so that next time we can start working on the actual review process itself.

Periodic Review, Part III

“We conquer by continuing.”
George Matheson

Last time, we created our app, added a few properties, and built our first table. Today we are going to work with that table to configure the layout of both the list and the form, and then maybe do a few other things before we move on to the rest of the tables. Let’s start with the list.

To edit the fields that will show up on the list view, we can bring up the list view and then select Configure -> List Layout from the context menu.

Configuring the list layout

Using the slush bucket, we can select Number, Short description, Item label and Description from the available fields on the table.

Selecting the fields to appear on the list view

That will give us a list view that looks like this.

Newly configured list view

Using basically the same method, we can arrange the fields on the form view.

Configuring the form layout

The form is a bit more complicated than the list, so to help organize things, we can divide the form into sections. After we lay out the main section of the form, we can scroll down to the Section list and click on the New… option, which brings up a small dialog box where we can give our new section a name.

Creating a new section on the form

Once we have created the new section, we can drag in and arrange all of the fields that we would like to see in that section of the form.

Populating the fields in our new section

Once that has been completed, we can take a look at our new form.

The new form layout

We are still not quite done with the form just yet, though. All of the columns that reference fields on the selected table should have a selection list that is limited to just the fields on that table. To accommodate that, we need to pull up the dictionary record for each of those fields and set up a dependency. To do that, right click on the field label and select Configure Dictionary from the resulting context menu.

Editing the dictionary record from the form

Using the Advanced view, go into the Dependent Field tab and check the Use dependent field checkbox and select Table from the list of fields.

Setting up the dependent field

This process will need to be repeated for all of the columns that represents fields on the configured table.

The last thing that we need to add to this form, at least for now, is the ability to test the Filter against the specified table. It would probably be more user-friendly if our Filter field was some kind of query builder, but since it is just a simple String field, the least we can do is to provide some mechanism to test out the query string once it has been entered. The easiest way to do that would be to create a UI Action called Test Filter that used the Table and the Filter fields to branch to the List view of that table. Building a link to the List view in script would look something like this:

current.table + '_list.do?sysparm_query=' + encodeURIComponent(current.filter)

Branching to that page in a UI Action script would then just be this:

action.setRedirectURL(current.table + '_list.do?sysparm_query=' + encodeURIComponent(current.filter));

Clicking on the button would then take you to the list where you could see what records would be selected using that filter. To create the UI Action, we can use the context menu on the form and select Configure -> UI Actions and then click on the New button to create a new UI Action for that form.

Creating a new UI Action to test out the entered query filter

Once our action has been configured and saved, the button should appear at the top of the form.

New form button from new UI Action configuration

That should just be about it for our first table and all of the associated fields, forms, and views. Next time, we can use our Service Account Management app as a potential first user of this app and see if we can set up the configuration for that app before we move on to creating other tables.