Collaboration Store, Part XVIII

“Every tester has the heart of a developer … in a jar on their desk.”
Unknown

Last time, we discussed jumping into the code that would allow us to share a local Scoped Application with the Host instance, but the results are starting to come in from some of the folks who have been testing the set-up process, so we should probably deal with those first. Here’s what we have so far:

  • Installation error: Table ‘sys_hub_action_status_metadata’ does not exist
  • Not allowing update of property: x_11556_col_store.store_name
  • Not allowing update of property: x_11556_col_store.host_instance
  • In the setup, the instance name field doesn’t inform you that you only need the instance prefix, not the full url
  • You can only collaborate with one host

Many thanks to all of those who have been participating in the testing process. Your efforts are much appreciated. For those of you who tried things out, but neglected to post anything, please feel free to leave a comment on your experience, even if you have no defects to report. All feedback is welcome … thanks!

Now let’s take a look at the issues that have been reported so far. one issue at a time.

Installation error: Table ‘sys_hub_action_status_metadata’ does not exist

It looks like the table ‘sys_hub_action_status_metadata’ is a table related to a version or plugin that I have in my instance, but is not present in the instance on which the test installation was being performed. The version of my instance is glide-rome-06-23-2021__patch0-07-07-2021 according to stats.do. The table in question looks relatively new, with a create date of 2021-07-31 16:24:07, and its name implies some sort of metadata, so I don’t think it is anything critical to the operation of the application. If there are no other issues with this installation, my opinion would be that this error could be safely ignored. To make it go away, I could probably just remove any references to this table from the Update Set. That sounds to me like the best way to go, just to avoid the potential of this error popping up, even though it seems as if it is fairly benign.

Not allowing update of property: x_11556_col_store.store_name
Not allowing update of property: x_11556_col_store.host_instance

These two are basically the same problem for two different System Properties. This is an annoyance that really should be corrected somehow. The work-around that was used was to switch over to the application scope, but that should not be necessary. When you are installing an app for the first time, that scope has not even been established yet, so I need to do something to allow these properties to be modified from the global scope, or from any scope for that matter. I’m not exactly sure how to do that, so I will have to do a little research and see what I can come up with. But this is definitely an issue that needs to be addressed.

In the setup, the instance name field doesn’t inform you that you only need the instance prefix, not the full url

This is very true, and should probably be addressed as well. The snh-form-field tag does provide for a “help” attribute, which appears underneath the label, so that’s probably a good place to throw that onto the screen. I’ll make sure that gets added in there before I release the next version.

You can only collaborate with one host

This is also very true, but that’s the way this particular version was conceived. Back when I was thinking of doing something peer-to-peer without anyone designated as the Host, I was leaning more towards that kind of environment, but once I settled on the Host/Client approach, I was always thinking one Host and many Clients. I can see the benefit of being able to connect to more than one Host, but that’s a little more complicated that I was thinking of taking on at this point, so I think I will file that one in the maybe-I-will-do-that-one-day pile. Good idea, though.

All in all, the list so far is not bad, but I assume that there is more to come. It seems like the biggest issue at this point is the cross-scope updates of the application’s System Properties, but the missing table is also something that might give people pause for no reason. Hopefully, I can find a way to address those before I push out the next version.

Thanks again to everyone who took the time to pull this down and give it a whirl, particularly those of you who posted your findings. And if you did not run into any difficulties and were able to get to the point where every instance has the same list of member organizations, please post those results as well, including the number of instances involved. Any feedback is welcome, and as always, much appreciated. Looking forward to hearing more … thank you all!

Next time out, we’ll see if we can get back to building out the application publishing process and maybe start finding out if we can make that work.

Collaboration Store, Part XVII

“I believe in a world where all these things can happen, even if I have to do them myself.”
Adrian Lamo

While we wait for the hordes of fellow developers currently running their extensive tests on the initial version of the Collaboration Store app to report their findings, I thought it might be a good time to give a little bit of thought as to where things should go from here. As I mentioned earlier, there are a number of other improvements that I would like to make to the set-up process once we are sure that everything is working as it should. That would definitely be one specific area towards which we could direct our attentions next. On the other hand, I am quite anxious to see if I can actually do some real sharing of artifacts between instances, and I am not quite sure how I am going to do that, so I would like to start focusing on that little project as well. I have managed to cobble together enough parts to get multiple instances aware of one another, so even though the initial set-up process is not quite ready for prime time, maybe it is good enough for now and I can start trying to work towards accomplishing the actual purpose of this application. Besides, we have not received any results from any of those selfless volunteers who have been doing the outside testing as yet, so we really don’t want to disturb that code at this point.

I would like to share all kinds of different artifacts at some point, but to start out, I am going to focus on sharing Scoped Applications in much the same way that applications are shared to internal app stores or the public ServiceNow App Store. There are two sides to the process of sharing: 1) publishing your app to the desired target, and 2) retrieving an app from the desired source and installing it in your instance. Since I can only do one thing at a time, and one obviously has to take place before you can do the other, my initial focus will be to see if I can publish an app to the Host instance. Once we clear that hurdle, then maybe we can see if we can do the flip side and pull something down from the Host and get it installed. But that’s nothing that we need to worry about today, particularly since we don’t even know yet if we can pull off putting something out there!

The first thing that we will need is a couple of tables, one to store the published applications and another to store each version of the application. These are just meta data tables containing information about the app and the versions. I envision that the application itself will be an XML Update Set file, which I plan to attach to the application version table record for each version published. The question is, How do we do that?

There are already built-in capabilities to turn an app into an Update Set and to turn an Update Set into an XML file. Both of those are UI Actions, so taking a peek under the hood of those artifacts would seem like a good place to start. Before you can turn an Update Set into an XML file, you first have to have an Update Set, so let’s first take a peek at the code under the UI Action that does that. The easiest way to do that would be to pull up a Scoped Application and then use the hamburger menu to pull up the list of UI Actions for that form.

Select Configure -> UI Actions from the context menu to pull up the list of UI Actions

The one that we are looking for is called Publish to Update Set… and what we are looking for is the code that is executed when the action is selected.

1
2
3
4
5
6
7
8
9
function publishIt() {
    var sysId = gel('sys_uniqueValue').value;  
    var dialogClass = window.GlideModal ? GlideModal : GlideDialogWindow;
    var dd = new dialogClass("publish_app_dialog");
    dd.setTitle(new GwtMessage().getMessage('Publish to Update Set'));
    dd.setPreference('sysparm_sys_id', sysId);
    dd.setWidth(500);
    dd.render();
}

Well, that didn’t tell us much. All this code does is launch a dialog box. We will need to look at the UI Page that will appear in the pop-up window, which we can see from the code is called publish_app_dialog. The key element there is the publishApp() function in the Client script of that UI Page.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
function publishApp(updateSetId) {
    var dd = new GlideModal("hierarchical_progress_viewer");
    dd.on("beforeclose", function () {
               var notification = {"getAttribute": function(name) {return 'true';}};
               CustomEvent.fireTop(GlideUI.UI_NOTIFICATION + '.update_set_change', notification);
               window.location.href = "sys_update_set.do?sys_id=" + updateSetId;
    });
 
    dd.setTitle("Progress");
    dd.setPreference('sysparm_function', 'publishToUpdateSet');
    dd.setPreference('sysparm_update_set_id', updateSetId);
    dd.setPreference('sysparm_sys_id', this.appId);
    dd.setPreference('sysparm_name', this.inferredUsName);
    dd.setPreference('sysparm_version', this.versionField.value);
    dd.setPreference('sysparm_description', this.descriptionField.value);
    dd.setPreference('sysparm_include_data', this.includeDataField.checked);
    dd.setPreference('sysparm_progress_name', "Publishing application");
    dd.setPreference('sysparm_ajax_processor', 'com.snc.apps.AppsAjaxProcessor');
    dd.setPreference('sysparm_show_done_button', 'true');
 
    dd.render();
    GlideDialogWindow.get().destroy();
 
    return dd;
}

Basically, this code replaces the original pop-up with a progress bar driven off of the server side publishToUpdateSet function of the com.snc.apps.AppsAjaxProcessor. This is probably something that we still want to do. We just don’t want to be sent to the Update Set form when it all over. Instead, we want to go ahead and turn it into an XML file, and then we want to attach that file to a new version record for the application. Also, if the application has never been published before, we will need to create the master application record as well. But it looks like we can steal most of this code so far. We will need to clone both the UI Action and the UI Page, and then point our cloned action to our cloned page, which is where we will make the majority of the changes. So far, so good.

Now we need to turn the Update Set into XML. For that, we take a peek at the other UI Action for that purpose, found on the Update Set form. To hunt that guy down, we pull up any local Update Set and use the same hamburger menu options to pull up the UI Actions for that form and look for the one called Export to XML. Here is the relevant script:

1
2
3
4
var updateSetExport = new UpdateSetExport();
var sysid = updateSetExport.exportUpdateSet(current);
 
action.setRedirectURL("export_update_set.do?sysparm_sys_id=" + sysid + "&sysparm_delete_when_done=true");

The first couple of lines look important, but that last one looks like it pops up the XML file for downloading, which we definitely do not want to do in our adaptation. It turns out that export_update_set is a Processor, which also has code of its own, so we better dig into that guy and see if does anything important that we might need to retain. We can find that guy by entering the word processors in the left-hand navigation menu and then select the lone menu item that appears to bring up the list. Once we have the list, we just need to search for the one where the Path is export_update_set. Here is the relevant code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
(function process(g_request, g_response, g_processor) {
    var sysid = g_request.getParameter('sysparm_sys_id');
    var remoteUpdateGR = new GlideRecord('sys_remote_update_set');
    if (!remoteUpdateGR.get(sysid)) {
        gs.addErrorMessage(gs.getMessage('Update Set is invalid'));
        return;
    }
    if (!gs.hasRole('admin') && !sn_app_api.AppStoreAPI.canPublishToUpdateSet(remoteUpdateGR.application)) {
        gs.addErrorMessage(gs.getMessage('You do not have permission to perform this operation'));
        g_response.setStatus(403);
        return;
    }
    var exporter = new ExportWithRelatedLists('sys_remote_update_set', sysid);
    exporter.addRelatedList('sys_update_xml', 'remote_update_set');
    exporter.exportRecords(g_response);
    var del = g_request.getParameter('sysparm_delete_when_done');
    if (del == "true" && remoteUpdateGR.canDelete())
        remoteUpdateGR.deleteRecord();
})(g_request, g_response, g_processor);

So it looks like code in which we are interested is the stuff going on with the variable exporter. Unfortunately, there is no output from the exportRecords function of that object and it takes as input the global g_response object. That will send the output to the browser, which we definitely do not want to do. Maybe we can hack it by sending it a fake g_response object, and then pull our output from there. If you right click on the class name (ExportWithRelatedLists), you can use the resulting context menu to pull up the definition and see exactly what it does. There is quite a bit of code in there, but here is the relevant function:

1
2
3
4
5
6
7
8
9
10
11
exportRecords: function(response){
    this.setHeaders(response);
    var outputStream = response.getOutputStream();
    this.hd = this.beginExport(outputStream);
    var gr = new GlideRecord(this.parent_table);
    gr.get(this.sys_id);
    this.exportRecord(gr);
    this.exportChildren();
    this._exportQuerySets();
    this.endExport(outputStream);
}

It basically sends everything to the outputStream that it obtains from the response object that you send it. So, we could build our own response object with our own outputStream, use this component right out of the box, and then grab the outputStream from our phony g_response object. Or, we just steal the code from this guy and use it to build our own exporter and have our version of the exportRecords function simply return the XML. Either way, we should be able to leverage all of this code to do what it is that we want to do. That should give us our XML file, anyway.

Of course, generating the XML from the Scoped Application is just the start of the process. Once we have it in hand, we have to create a version record, attach the XML to the version record as a file, and then ship the thing over to the Host instance. Once it arrives at the Host, we will also have to send the version record and the attached XML file out to all of the other instances. That’s all quite a bit of work, but so was the set-up process, and we just trudged through that effort one piece at a time until we got to the end of the to-do list. Hopefully, we can do the same here. We may jump right into that next time, if we don’t yet have any feedback from the set-up testing.

Collaboration Store, Part XVI

“It’s hard enough to find an error in your code when you’re looking for it; it’s even harder when you’ve assumed your code is error-free.”
Steve McConnell

Now that we have completed all of the parts for the initial set-up process of our new Scoped Application, it’s time to take a step back and see where things stand. On the one hand, after 16 captivating installments, you would think that we would be much further along in this process beyond just the initial set-up. On the other hand, this is a fairly complex endeavor, and it’s good to get this necessary administrative function out of the way so that we can focus on the actual purpose of the application. But before we jump right into that, we should first have quick look at what we have and what we don’t have at this point.

What we have is an initial version of the set-up process for both the Host instance and the Client instances. Now all of this needs to be fully tested in multiple scenarios, but even if we manage to kill all of the bugs that are undoubtedly baked in there at this stage of the game, as it is written, it assumes for the most part that all will go well every time. What I mean by that is that there isn’t a whole lot of error recovery built into the process right now. Everything seems to work if all of the instances are up and running when contacted. That’s not really good enough for prime time, though, as it is always possible that one or more instances might be unavailable or off-line for some reason. At some point, we will have to build in some processes to monitor for that and to deal with it in some way. Right now, if you fail to get some kind of update from the Host, you just don’t get it. That’s not really good enough in the long run, but my approach is always to get things working first, and then add such features later in a future version. Maybe we will even handle that using Event Management, although not everyone has that feature activated, so maybe that’s not a good plan after all.

There are other features that I would like to add as well. For example, it would be nice if each participating instance had some form of logo or image that would visually identify them and all of the items that they have shared with the community. Things like that are nice-to-haves, though, so again, we’ll deal with that later. At this point, I just want to make sure that what we have put together so far actually works the way that it was intended before we go any further.

I also want all of the menu options hidden until set-up is complete, and then once set-up has been completed, I would like the set-up option to be hidden. I haven’t thrown that in there just yet, either, but that’s something that I don’t want to forget to do once I am sure that everything is working as it should be.

Not too long ago, I had an offer to assist with the testing of this particular project. Normally, I like to do all of my own testing, but they say that programmers are the worst testers of their own code, so I’m going to break with tradition and go ahead and put out an Update Set for this app that is clearly not finished and basically not good for anything of value at this point. If anyone want to participate in this effort, all that I ask is that you post any defects that you uncover to the comments section so that I can see if I can’t get them resolved and put out a new version with the corrections.

So, here’s the deal: gather up your friends and neighbors and come up with some strategy to see who draws the short straw and serves as the Host instance, set up the Host first, and then everyone else jump in and set up their Client instances by referencing the Host. This can work with just two instances, but to see the existing instance updates for any new instance, you will need at least three (one for the Host, one for the new Client, and at least one for an existing Client). Four our more would be even better, but three will at least test all of the current features. When all is said and done, everyone’s list of member instances should match, unless something went terribly wrong along the way. And if you really want to put yourself out there, you can set up a Host instance and put your instance ID in the comments so that other people that you don’t even know can attempt to connect to your instance. Your call.

To install this version of the Collaboration Store (we’ll call it version 0.1), you will need this Update Set, which contains the Scoped Application, and you will also need the latest version of snh-form-fields, which you can find here. Install the form fields Update Set first, and then install the Scoped Application. At that point, you should be good to go and should be able to click on the set-up menu option at any time. I’ll let this sit out here for a while and see if anything comes of it. Thanks in advance for helping a guy out. It’s very much appreciated.

Collaboration Store, Part XV

“The beginning is the most important part of the work.”
Plato

With the completion of the last piece of the registration service, the only remaining component of the set-up process is the Client instance function that utilizes the registration service provided by the Host instance. This function will actually be quite similar to the function that we just created to inform one instance about another. This time, we will be invoking the Scripted REST API instead of a stock REST API, but the process is virtually the same.

Before we make the call, however, we need to take care of few little items. First, we need to create the Service Account needed to access the instance, and then we need to grab a couple of our System Properties. We already created a function to establish the Service Account, so all we need to do is to call that function and then grab the two property values.

1
2
3
this.createUpdateWorker(mbrGR.getUniqueValue());
var host = gs.getProperty('x_11556_col_store.host_instance');
var token = gs.getProperty('x_11556_col_store.active_token');

Now we can build the payload from the instance GlideRecord that was passed to the function.

1
2
3
4
5
6
var payload = {};
payload.sys_id = mbrGR.getUniqueValue();
payload.name = mbrGR.getDisplayValue('name');
payload.instance = mbrGR.getDisplayValue('instance');
payload.email = mbrGR.getDisplayValue('email');
payload.description = mbrGR.getDisplayValue('description');

At this point, we can create a new instance of our old friend, the sn_ws.RESTMessageV2 object, and then populate it with all of the relevant information.

1
2
3
4
5
6
var request = new sn_ws.RESTMessageV2();
request.setHttpMethod('post');
request.setBasicAuth(this.WORKER_ROOT + host, token);
request.setRequestHeader("Accept", "application/json");
request.setEndpoint('https://' + host + '.service-now.com/api/x_11556_col_store/v1/register');
request.setRequestBody(JSON.stringify(payload));

… and as we did before, we obtain the response object by invoking the execute method.

1
var response = request.execute();

Now all we have to do is to populate the result object with the information contained in the response.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
result.responseCode = response.getStatusCode();
result.bodyText = response.getBody();
try {
    result.body = JSON.parse(response.getBody());
} catch(e) {
    //
}
if (response.getErrorCode()) {
    result.error = response.getErrorMessage();
    result.errorCode = response.getErrorCode();
} else if (result.responseCode != '202') {
    result.error = 'Invalid HTTP Response Code: ' + result.responseCode;
} else {
    mbrGR.accepted = new GlideDateTime();
    mbrGR.update();
}

The complete function, including the return of the result, looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
registerWithHost: function(mbrGR) {
    var result = {};
 
    this.createUpdateWorker(mbrGR.getUniqueValue());
    var host = gs.getProperty('x_11556_col_store.host_instance');
    var token = gs.getProperty('x_11556_col_store.active_token');
    var payload = {};
    payload.sys_id = mbrGR.getUniqueValue();
    payload.name = mbrGR.getDisplayValue('name');
    payload.instance = mbrGR.getDisplayValue('instance');
    payload.email = mbrGR.getDisplayValue('email');
    payload.description = mbrGR.getDisplayValue('description');
    var request = new sn_ws.RESTMessageV2();
    request.setHttpMethod('post');
    request.setBasicAuth(this.WORKER_ROOT + host, token);
    request.setRequestHeader("Accept", "application/json");
    request.setEndpoint('https://' + host + '.service-now.com/api/x_11556_col_store/v1/register');
    request.setRequestBody(JSON.stringify(payload));
    var response = request.execute();
    result.responseCode = response.getStatusCode();
    result.bodyText = response.getBody();
    try {
        result.body = JSON.parse(response.getBody());
    } catch(e) {
        //
    }
    if (response.getErrorCode()) {
        result.error = response.getErrorMessage();
        result.errorCode = response.getErrorCode();
    } else if (result.responseCode != '202') {
        result.error = 'Invalid HTTP Response Code: ' + result.responseCode;
    } else {
        mbrGR.accepted = new GlideDateTime();
        mbrGR.update();
    }
 
    return result;
}

This final function completes the initial set-up process for our new Scoped Application. The application still doesn’t do anything in the way of sharing components between instances, but it’s a start. Next time, we will figure out where we go from here.

Collaboration Store, Part XIV

“I find that the harder I work, the more luck I seem to have.”
Thomas Jefferson

With the completion of the asynchronous Subflow, we now need to turn our attention to the Script Include function that was referenced in the custom Action built for the repeating step in the Subflow. This function needs to tell each existing instance about the new instance and tell the new instance about all of the existing instances. Both tasks can actually be accomplished with the same code, leveraging the out-of-the-box REST API for ServiceNow tables. Basically, what we will be doing will be to insert a new record into the instance table for both operations, which can be easily handled with the stock API.

Assuming that we are passed a GlideRecord for both the instance to contact and the instance to be shared, the first thing that we will want to do is to create an object containing all of the relevant information about the instance to be shared.

1
2
3
4
5
6
7
8
var payload = {};
payload.instance = instanceGR.getDisplayValue('instance');
payload.name = instanceGR.getDisplayValue('name');
payload.description = instanceGR.getDisplayValue('description');
payload.email = instanceGR.getDisplayValue('email');
payload.accepted = instanceGR.getDisplayValue('accepted');
payload.active = instanceGR.getDisplayValue('active');
payload.host = instanceGR.getDisplayValue('host');

Once we have constructed the payload object from the shared instance GlideRecord, we will want to construct the end point URL using the instance name from the target instance GlideRecord.

1
var url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_organization';

Now we need to build a new RESTMessageV2 with the information that we have assembled for this action.

1
2
3
4
5
6
7
var request = new sn_ws.RESTMessageV2();
request.setEndpoint(url);
request.setHttpMethod('POST');
request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
request.setRequestHeader('Content-Type', 'application/json');
request.setRequestHeader('Accept', 'application/json');
request.setRequestBody(JSON.stringify(payload, null, '\t'));

To obtain the response object, we just need to invoke the execute method on the newly created RESTMessageV2 object.

1
var response = request.execute();

… and now we just have to examine the response to populate the result object.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
result.status = response.getStatusCode();
result.body = response.getBody();
if (result.body) {
    try {
        result.obj = JSON.parse(result.body);
    } catch (e) {
        result.parse_error = e.toString();
    }
}
result.error = response.haveError();
if (result.error) {
    result.error_code = response.getErrorCode();
    result.error_message = response.getErrorMessage();
}

Once that has been completed, all that is left is to return the result object back to the caller. All together, the function looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
publishInstance: function(instanceGR, targetGR) {
    var result = {};
 
    var payload = {};
    payload.instance = instanceGR.getDisplayValue('instance');
    payload.name = instanceGR.getDisplayValue('name');
    payload.description = instanceGR.getDisplayValue('description');
    payload.email = instanceGR.getDisplayValue('email');
    payload.accepted = instanceGR.getDisplayValue('accepted');
    payload.active = instanceGR.getDisplayValue('active');
    payload.host = instanceGR.getDisplayValue('host');
    var url = 'https://' + targetGR.getDisplayValue('instance') + '.service-now.com/api/now/table/x_11556_col_store_member_organization';
    var request = new sn_ws.RESTMessageV2();
    request.setEndpoint(url);
    request.setHttpMethod('POST');
    request.setBasicAuth(this.WORKER_ROOT + targetGR.getDisplayValue('instance'), targetGR.getDisplayValue('token'));
    request.setRequestHeader('Content-Type', 'application/json');
    request.setRequestHeader('Accept', 'application/json');
    request.setRequestBody(JSON.stringify(payload, null, '\t'));
    var response = request.execute();
    result.status = response.getStatusCode();
    result.body = response.getBody();
    if (result.body) {
        try {
            result.obj = JSON.parse(result.body);
        } catch (e) {
            result.parse_error = e.toString();
        }
    }
    result.error = response.haveError();
    if (result.error) {
        result.error_code = response.getErrorCode();
        result.error_message = response.getErrorMessage();
    }
 
    return result;
}

We can now utilize this function in the function called by our custom Action. That function is passed the two instance names, so we will need to fetch the GlideRecords for those instances, and then pass those GlideRecords to the publishInstance function, once to tell the existing instance about the new instance, and then again to tell the new instance about the existing instance.

1
2
3
4
5
6
7
8
publishNewInstance: function(new_instance, target_instance) {
    instanceGR = new GlideRecord('x_11556_col_store_member_organization');
    instanceGR.get('instance', new_instance);
    targetGR = new GlideRecord('x_11556_col_store_member_organization');
    targetGR.get('instance', target_instance);
    this.publishInstance(instanceGR, targetGR);
    this.publishInstance(targetGR, instanceGR);
}

That completes all of the parts for the registration service. However, we still need to build the function that calls this Host service from the new Client instance. That will be the subject of our next installment.

Collaboration Store, Part XIII

“First, solve the problem. Then, write the code.”
John Johnson

Today we need to build out the Subflow that we referenced in the Script Include function that we built last time out. To create a new Subflow, open up the Flow Designer, click on the New button to bring up the selection list, and select Subflow from the list.

Creating a new Subflow

When the initial form pops up, all you really need to enter is the name of the Subflow, which we already referred to in our Script Include function as New_Collaboration_Store_Instance.

New Subflow properties form

Once you submit the initial properties form, the new Subflow will appear in the list of Subflows, and from there you can bring it up in full edit mode. In edit mode, we can add the one Input to the Subflow, the name of the new instance.

Subflow Inputs and Outputs

Since we will be launching this Subflow to run on its own in the background, there is no need for any Outputs.

Once we configure the Inputs and Outputs, we can move on to the steps of the Subflow. Our first step will be to gather up all of the records in the table of instances except for two: the Host instance, which has already been updated, and the new instance, which already knows about itself.

Database query step

We select the Look Up Records Action, select the Member Organization table, and then define two Conditions to get the records that we want: 1) the host value is false, and 2) the instance is not the instance that we brought in as Input. It really doesn’t matter for our purposes what sequence the records come in, but I went ahead and sorted the records by instance, just so they we will always work through them in the same order.

Our next step, then is to loop through the records. We do that with a For Each Item Flow Control step, setting the items to the records obtained in the first step.

Looping through the retrieved records

Now we have the basic structure of the Subflow; we just need to perform the tasks necessary to notify each host of the new instance, and notify the new instance of each host within the the For Each Item loop. This could all be done with additional Subflow steps, but I took the easy way out here and just built another function in our Script Include to handle the REST API calls to the instances. In fact, I built two functions, one to make the call, and another to call that function twice, once to tell the existing host about the new host, and then again to inform the new host of the existing host. To run that script, I had to create a custom Action, which is just a simple Script Action that calls the function, passing in the names of the two instances (existing, from the current record, and new, from the Subflow input). Once I built the custom Action, I was able to select it from the list and then configure it.

Custom Script Action step

That completes the Subflow, but once again, we have referenced a function in our Script Include that does not exist, so we will have to get into that in our next installment.

Collaboration Store, Part XII

“The slogan ‘press on’ has solved and always will solve the problems of the human race.”
Calvin Coolidge

In the previous installment in this series, we created a new Scripted REST API Resource and referenced another nonexistent function in our Script Include. Now it is time to create that function, which will perform some of the work required to register a new client instance and then hand off the remaining tasks to an asynchronous Subflow so that the function can return the results without waiting for all of the other instances to be notified of the new client instance. The only thing to be done in the function will be to insert the new client instance into the database and kick off the Subflow. But before we do that, we need to first check to make sure that the Client has not already registered with the Host.

1
2
3
4
5
6
7
8
9
var result = {body: {error: {}, status: 'failure'}};
 
var mbrGR = new GlideRecord('x_11556_col_store_member_organization');
if (mbrGR.get('instance', data.instance)) {
    result.status = 400;
    result.body.error.message = 'Duplicate registration error';
    result.body.error.detail = 'This instance has already been registered with this store.';
} else {
    ...

As we did before, we construct our result object with the expectation of failure, since there are more error conditions than the one successful path through the logic. In the case of an instance that has already been registered, we respond with a 400 Bad Request HTTP Response Code and accompanying error details. If the instance is not already in the database, then we attempt to insert it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
mbrGR.initialize();
mbrGR.name = data.name;
mbrGR.instance = data.instance;
mbrGR.email = data.email;
mbrGR.description = data.description;
mbrGR.token = data.sys_id;
mbrGR.active = true;
mbrGR.host = false;
mbrGR.accepted = new GlideDateTime();
if (mbrGR.insert()) {
    result.status = 202;
    delete result.body.error;
    result.body.info = {};
    result.body.info.message = 'Registration complete';
    result.body.info.detail = 'This instance has been successfully registered with this store.';
    result.body.status = 'success';
    ...

If the new record was inserted successfully, then we response with a 202 Accepted HTTP Response Code, indicating that the registration was accepted, but the complete registration process (notifying all of the other instances) is not yet complete. At this point, all we have left to do is to initiate the Subflow to handle the rest of the process. We haven’t built the Subflow just yet, but for the purposes of this exercise, we can just assume that it is out there and then we can build it out later. There a couple of different ways to launch an asynchronous Subflow in the background, the original way, and the newer, preferred method. Both methods use the Scripted Flow API. Here is the way that we used to do this:

1
sn_fd.FlowAPI.startSubflow('New_Collaboration_Store_Instance', {new_instance: data.instance});

… and here is way that ServiceNow would like you to do it now:

1
2
3
4
5
sn_fd.FlowAPI.getRunner()
    .subflow('New_Collaboration_Store_Instance')
    .inBackground()
    .withInputs({new_instance: data.instance})
    .run();

Right now, both methods will work, and I’m still using the older, simpler way, but one day I’m going to need to switch over.

There should never be a problem inserting the new record, but just in case, we make that a conditional, and if for some reason it fails, we respond with a 500 Internal Server Error HTTP Response Code.

1
2
3
result.status = 500;
result.body.error.message = 'Internal server error';
result.body.error.detail = 'There was a problem processing this registration request.';

That’s it for all of the little parts and pieces. Here is the entire function, all put together.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
processRegistrationRequest: function(data) {
    var result = {body: {error: {}, status: 'failure'}};
 
    var mbrGR = new GlideRecord('x_11556_col_store_member_organization');
    if (mbrGR.get('instance', data.instance)) {
        result.status = 400;
        result.body.error.message = 'Duplicate registration error';
        result.body.error.detail = 'This instance has already been registered with this store.';
    } else {
        mbrGR.initialize();
        mbrGR.name = data.name;
        mbrGR.instance = data.instance;
        mbrGR.email = data.email;
        mbrGR.description = data.description;
        mbrGR.token = data.sys_id;
        mbrGR.active = true;
        mbrGR.host = false;
        mbrGR.accepted = new GlideDateTime();
        if (mbrGR.insert()) {
            result.status = 202;
            delete result.body.error;
            result.body.info = {};
            result.body.info.message = 'Registration complete';
            result.body.info.detail = 'This instance has been successfully registered with this store.';
            result.body.status = 'success';
            sn_fd.FlowAPI.startSubflow('New_Collaboration_Store_Instance', {new_instance: data.instance});
        } else {
            result.status = 500;
            result.body.error.message = 'Internal server error';
            result.body.error.detail = 'There was a problem processing this registration request.';
        }
    }
 
    return result;
}

Now we have completed the nonexistent function that was referenced in the REST API Resource, but we have also now referenced a new nonexistent Subflow that we will need to build out before things are complete. That sounds like a good subject for our next installment.

Collaboration Store, Part XI

Never give up on a dream just because of the time it will take to accomplish it. The time will pass anyway.”
Earl Nightingale

We have one more function left in our Script Include that was referenced in the set-process function of our widget that still needs to be created. Last time, we completed the function that creates the Service Account, and now we need to build the one that calls the new instance registration process on the Host instance, which also does not exist as yet, either. In fact, before we build the function that uses the service, we should probably create the service first. This will be another Scripted REST API similar to the one that we already created for the Host info service, but this one will be much more complicated.

When we register a new Client instance with the Host, the following things should occur:

  • The new Client instance should be added to the Host database,
  • The new Client instance data should be pushed to the databases of all of the existing instances, and
  • The new Client instance should be updated with a full list of all of the existing instances.

When this process is complete, all instances, including the new instance, should have a full list of every other instance registered with the Host. However, it seems to me that the registration process itself should not have to wait until all of the instances have been made aware of each other. Once we insert the new instance into the Host database, we should be able to respond to the caller that the registration has been completed, and then pushing the details out to all of the instances could be handled off-line in some other async process. But before we worry too much about all of that, let’s get to creating the service.

We need to create another Scripted REST API Resource, but we can tuck it underneath same the Scripted REST API Service that we already created for our earlier info resource. This one we will call register, which will give it the following URI:

1
/api/x_11556_col_store/v1/register

This time, we will set the HTTP Method to POST, since we will be accepting input from the client instance in JSON format in the body of the request.

/register Scripted REST API Resource

For our Script, we will push the majority of the code over into yet another new function in our Script Include, and then use the results to populate the response.

1
2
3
4
5
6
7
8
9
(function process(request, response) {
    var data = {};
    if (request.body && request.body.data) {
        data = request.body.data;
    }
    var result = new CollaborationStoreUtils().processRegistrationRequest(data);
    response.setBody(result.body);
    response.setStatus(result.status);
})(request, response);

Unlike our earlier /info service, we will want to require authentication for this service (which is why we created the Service Accounts), so under the Security tab, we will check the Requires authentication checkbox.

/register Scripted REST API Resource Security Tab

Under the Content Negotiation tab, we will set both the Supported request formats and the Supported response formats to application/json, as we will expect the caller to send us a JSON string containing the details of their instance, and we will be responding with a JSON string containing the outcome of the registration process.

/register Scripted REST API Resource Content Negotiation Tab

Once we Save the new resource, it should appear in the Related List at the bottom of the service form.

Our first two Scripted REST API Resources

With that out of the way, now we need to turn our attention to building out the non-existent Script Include function that we referenced in the resource’s script. That’s where all of magic will happen, and quite a lot will go on there, so that seems like a good subject for our next installment.

Collaboration Store, Part X

“It’s better to wait for a productive programmer to become available than it is to wait for the first available programmer to become productive.”
Steve McConnell

With the completion of the set-up widget, we can now turn our attention to the missing Script Include functions and the initial registration process that one of those functions will be calling on the Host instance. Since I always like to start with the easy/familiar stuff first (that checks more things off of the “to do” list faster than the other way around!), let’s jump into the createUpdateWorker() function that inserts or updates the Service Account in the sys_user table. But before we do that, we will need to create a special Role for these worker accounts that has limited privileges.

To create a new Role, navigate to System Security -> Roles and click on the New button to bring up the input form. The only field that we need to fill out is the Suffix, which we will set to worker.

Once we have saved our Role, we will want to bring up the form again and grab the sys_id using the Copy sys_id option on the hamburger menu. We will use that to set the value of one of the constants that we will define at the top of our Script Include.

1
2
WORKER_ROOT: 'csworker1.',
WORKER_ROLE: 'f1421a6c2fe430104425fcecf699b6a9',

The other constant is the prefix for the user ID for the Service Account, to which we will append the name of the instance. Now that we have defined our Role and set up our constants, we can build the code that will create the account.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
var user_name = this.WORKER_ROOT + gs.getProperty('instance_name');
var userGR = new GlideRecord('sys_user');
if (!userGR.get('user_name', user_name)) {
    userGR.initialize();
    userGR.user_name = user_name;
    userGR.insert();
}
userGR.first_name = 'CS';
userGR.last_name = 'Worker';
userGR.title = 'Collaboration Store Worker';
userGR.active = true;
userGR.locked_out = false;
userGR.web_service_access_only  = true;
userGR.user_password.setDisplayValue(passwd);
userGR.update();

Since it is possible that an earlier failed attempt to set up the app already created the account, we first check for that, and if it isn’t already present in the sys_user table, then we create it. Then we set the appropriate fields to their current values and update the record. One thing to note is the way in which we update the user_password field, which is different than all of the others. Because the value of that field is one-way encrypted, we have to set the Display Value instead of the Value. It took me a bit of research to figure that out; it is not very well documented anywhere that I could find.

Once we create the account, we then have to assign it to the Role that we created earlier. Once again, this may have already been done in an earlier failed attempt, so we have to check for that before proceeding.

1
2
3
4
5
6
7
8
9
var userRoleGR = new GlideRecord('sys_user_has_role');
userRoleGR.addEncodedQuery('user=' + userGR.getUniqueValue() + '^role=' + this.WORKER_ROLE);
userRoleGR.query();
if (!userRoleGR.next()) {
    userRoleGR.initialize();
    userRoleGR.user = userGR.getUniqueValue();
    userRoleGR.role = this.WORKER_ROLE;
    userRoleGR.insert();
}

That takes care of creating/updating the account and applying the Role. Putting it all together, the entire function looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
createUpdateWorker: function(passwd) {
    var user_name = this.WORKER_ROOT + gs.getProperty('instance_name');
    var userGR = new GlideRecord('sys_user');
    if (!userGR.get('user_name', user_name)) {
        userGR.initialize();
        userGR.user_name = user_name;
        userGR.insert();
    }
    userGR.first_name = 'CS';
    userGR.last_name = 'Worker';
    userGR.title = 'Collaboration Store Worker';
    userGR.active = true;
    userGR.locked_out - false;
    userGR.web_service_access_only  = true;
    userGR.user_password.setDisplayValue(passwd);
    userGR.update();
 
    var userRoleGR = new GlideRecord('sys_user_has_role');
    userRoleGR.addEncodedQuery('user=' + userGR.getUniqueValue() + '^role=' + this.WORKER_ROLE);
    userRoleGR.query();
    if (!userRoleGR.next()) {
        userRoleGR.initialize();
        userRoleGR.user = userGR.getUniqueValue();
        userRoleGR.role = this.WORKER_ROLE;
        userRoleGR.insert();
    }
},

Well, that takes care of the easy part. Next time, we will start digging into the more complex elements of the remaining work needed to complete the set-up process.

Collaboration Store, Part IX

“You can’t go back and make a new start, but you can start right now and make a brand new ending.”
James R. Sherman

Now that we have dealt with the two Script Include functions that were referenced in the Save process, we can return to our widget and address the Set-up process. A number of things have to happen in the Set-up process, including saving the user’s input in the database, creating a service account in the User table for authenticated REST API activities, and in the case of a client instance, we need to register the client with the specified Host instance. We will need to build out the other side of that registration process as well, which will include sharing the new instance information with all of the existing instances as well as sharing all of the existing instances with the new client. That’s quite a bit of work, but we’ll take it all on one piece at a time.

One of the first things that we will want to do, regardless of which type of instance we are setting up, is to save the user’s input. That’s pretty basic GlideRecord stuff, but let’s lay it all out here just the same.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
mbrGR.initialize();
mbrGR.active = true;
mbrGR.instance = thisInstance;
mbrGR.name = data.instance_name;
mbrGR.email = data.email;
mbrGR.description = data.description;
mbrGR.name = data.instance_name;
if (data.instance_type == 'host') {
    mbrGR.host = true;
    mbrGR.accepted = new GlideDateTime();
    gs.setProperty('x_11556_col_store.host_instance', thisInstance);
} else {
    mbrGR.host = false;
    gs.setProperty('x_11556_col_store.host_instance', data.host_instance_id);
}
mbrGR.insert();

Most data fields are the same for both Host and Client instances, but a Host instance gets the host field set to true and the accepted date valued, while the Client instance gets the host field set to false and the accepted date is not valued until the registration process is successful.

Now that we have entered the user’s input into the database, we will want to perform additional actions depending on the type of instance. For a Client instance, we will want to register the Client with the Host, and for the purpose of this widget, we can just assume for now that there is a function in our Script Include that handles all of the heavy lifting for that operation.

1
var resp = csu.registerWithHost(mbrGR);

That simple function call represents a lot of work, but it’s code that we really don’t want cluttering up our widget, so we will stuff it all into the Script Include and worry about building it all out later. We will want to check on how it all came out though, because if it was successful, we will want to update the accepted date for the instance and add the Host instance to our database table as well. Again, this is all pretty standard GlideRecord stuff, so nothing really exciting to see here.

1
2
3
4
5
6
7
8
9
10
11
mbrGR.accepted = new GlideDateTime();
mbrGR.update();
mbrGR.initialize();
mbrGR.instance = input.store_info.instance;
mbrGR.accepted = input.store_info.accepted;
mbrGR.description = input.store_info.description;
mbrGR.name = input.store_info.name;
mbrGR.email = input.store_info.email;
mbrGR.active = true;
mbrGR.host = true;
mbrGR.insert();

If the registration process was not successful though, we will want delete the record that we just created, inform the user of the error, and cycle the phase back to 1 to start all over again.

1
2
3
4
5
6
7
mbrGR.deleteRecord();
var errMsg = resp.error;
if (resp.body && resp.body.result && resp.body.result.error) {
    errMsg = resp.body.result.error.message + ': ' + resp.body.result.error.detail;
}
gs.addErrorMessage(errMsg);
data.validationError = true;

One of the things that will happen during the registration process in the Script Include will be to create the Service Account to be used for authenticated REST API calls to the instance. Since the registration process is only called for Client instances, we will need to handle that directly when setting up a Host instance. Once again, we can assume that there is a Script Include function that handles that process, which greatly simplifies the code in the widget.

1
csu.createUpdateWorker(mbrGR.getUniqueValue());

Putting it all together, the entire server side Javascript code for the set-up process now looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
function setupProcess() {
    gs.setProperty('x_11556_col_store.store_name', data.store_name);
    mbrGR.initialize();
    mbrGR.active = true;
    mbrGR.instance = thisInstance;
    mbrGR.name = data.instance_name;
    mbrGR.email = data.email;
    mbrGR.description = data.description;
    mbrGR.name = data.instance_name;
    if (data.instance_type == 'host') {
        mbrGR.host = true;
        mbrGR.accepted = new GlideDateTime();
        gs.setProperty('x_11556_col_store.host_instance', thisInstance);
    } else {
        mbrGR.host = false;
        gs.setProperty('x_11556_col_store.host_instance', data.host_instance_id);
    }
    mbrGR.insert();
    if (data.instance_type == 'host') {
        csu.createUpdateWorker(mbrGR.getUniqueValue());
    } else {
        var resp = csu.registerWithHost(mbrGR);
        if (resp.responseCode == '202') {
            mbrGR.accepted = new GlideDateTime();
            mbrGR.update();
            mbrGR.initialize();
            mbrGR.instance = input.store_info.instance;
            mbrGR.accepted = input.store_info.accepted;
            mbrGR.description = input.store_info.description;
            mbrGR.name = input.store_info.name;
            mbrGR.email = input.store_info.email;
            mbrGR.active = true;
            mbrGR.host = true;
            mbrGR.insert();
        } else {
            mbrGR.deleteRecord();
            var errMsg = resp.error;
            if (resp.body && resp.body.result && resp.body.result.error) {
                errMsg = resp.body.result.error.message + ': ' + resp.body.result.error.detail;
            }
            gs.addErrorMessage(errMsg);
            data.validationError = true;
        }
    }      
}

That’s it for the work on the widget itself. Of course, we still have a lot of work to do to both complete the referenced Script Include functions that do not yet exist, and to build out the registration process that one of those functions will be calling. There is a lot to choose from there for the subject of our next installment, but when the time comes, we will pick something and dive in.