Category: Projects

Collaboration Store, Part XXXVIII

Posted on | by snhackery | 2 Comments on Collaboration Store, Part XXXVIII

“You don’t have to be good to start … you just have to start to be good!”
— Joe Sabah

Last time out we really did not accomplish anything of any consequence, but today let’s see if we can actually make some progress on something of real value. We need to create a process that will run every so often and check to make sure that all of the instances in the community have all of the latest content. We can use the Flow Designer to create and schedule a flow that will run daily on the Host instance and compare the artifacts present on each Client instance with the artifacts present on the Host, and then send over any missing items that never made it over originally for whatever reason. Doing it this way will eliminate the need to track and record errors when they happen, since we will just compare the Client inventory with that of the Host and correct any discrepancies. We don’t really need to know what failed or why.

We could make all of the REST API calls to the Client instances using the Integration Hub, but since that is a collection of optional products, not every instance will have all of those components installed and we would not want to make that a prerequisite for the product. To make this work on any ServiceNow configuration, we will want to roll our own calls via Javascript, and we will just use the Flow Designer to schedule a simple flow that loops through all of the Client instance records and then calls a Script Action that will do all of the heavy lifting. Before we build the Action though, we will want to build a Script Include function that can be called in the Action. Since this will be a significant script, we should probably create a new Script Include specific to this purpose rather than adding a number of new functions to any of our existing Script Includes. We can call our new Script Include InstanceSyncUtils and create a simple function called syncInstance with a single argument of the sys_id of the record for the instance to be synced. For now, we can just stub out the function with a simple logging statement and then circle back later to fill in the details. Right now, we just want to have something to call in our Flow Designer Action.

New Script Include function to be called in our new Action

With that out of the way, we can now fire up the Flow Designer and navigate to New -> Action. We will call our new Action Sync Instance and configure a single Input called Instance ID.

New Sync Instance Action with a single Input

For the Script step, we will just pass the Input to our new Script Include function:

(function execute(inputs, outputs) {
	var isu = new InstanceSyncUtils();
	isu.syncInstance(inputs.instance_id);
})(inputs, outputs);

There is currently nothing returned by the function, so there is no need to define any Outputs. At this point, all we need to do is to Save and Publish the Action before turning our attentions to the actual Flow itself.

To build a Flow, go back to the Home Page of the Flow Designer and select New -> Flow. On the resulting pop-up Flow Properties panel, enter all of the details for the Flow and set the Run As value to System User.

New Flow Properties

Since we want this Flow to run on its own every day, we can set the Trigger to Daily, and the Time to 12:30, which should run it in the middle of the lunch hour in the Host time zone, when most instances should be available.

Flow Trigger configuration

Since we only want this Flow to run on the Host instance, the first thing that we need to do is to create a Flow Variable that indicates whether or not this instance is the Host. To set the value of the variable, we add a Set Flow Variables step that uses the following script to determine if this instance is the Host based on System Properties.

return gs.getProperty('instance_name') == gs.getProperty('x_11556_col_store.host_instance');

The next step then will be a conditional that checks the value of the new Flow Variable.

Making sure that this is the Host instance

Failing this test will terminate the Flow, but if this is the Host instance, then the next thing that we need to do is gather up all of the Client instance records and then loop through them and call our new Action for each one in turn.

Find all records where Active is true and Host is false

Inside the following For Each Item loop, we can then invoke our new Action, passing in the sys_id of the current record.

Calling our new Action inside the For Each Item loop

That completes the work on the Flow, and all we need to do now is to Save and Activate the Flow. At this point, the Flow will kick off every day at 12:30 local time, but it won’t really do much except put a few entries in the System Log. The real work will be done in the Script Include, and since that’s a major effort in and of itself, we’ll leave that for our next exciting episode.

Collaboration Store, Part XXXVII

Posted on | by snhackery

“Stay committed to your decisions, but stay flexible in your approach.”
— Tony Robbins

After abandoning my earlier plans to jump into the application publishing process, I started to take a look at the missing error recovery needs for all of these inter-instance interactions. One thing that I had always planned to do at some point was to create an activity log of all transactions between instances. My idea was not to sync the logs across all instances, but to have a record in each instance of the things that went on with that particular instance. On the Host instance, such a log could provide the basis for some form of periodic recovery process that scanned the log for failed transactions and then made another attempt to put the transaction through again. After giving that some thought, though, I decided that a better approach would be to just scan each instance for instances, applications, and versions, and then attempt to push over anything that was missing compared to what was present on the Host. I still want to build an activity log at some point, but I don’t think that I want to use it as a basis for error recovery. I think it would be more straightforward to just compare the lists of records on each instance with the master lists on the Host and then try to correct any deficiencies.

That’s the plan today, anyway, but plans do have a way of changing over time, so who knows how things will come out once I start putting all of the pieces together. Right now, though, this seems like the better way to go.

One little thing that have been wanting to do for some time was to add another field to the version record to indicate the version of ServiceNow that was used to build the Update Set. There is a System Property that contains this information, so I all I really needed to do was to add the field and then add a line of code to the version record creation function to pull that value out of the property and stuff it into the new field. The name of the property is glide.buildtag.last, and the name of the field that I added is built_on.

New built_on field added to the version record

Once I added the new field to the version record, I opened up the ApplicationPublisher Script Include, which creates the version record, and added the following line to the function that builds the record:

versionGR.setValue('built_on', gs.getProperty('glide.buildtag.last'));

I also had to modify the function that sent the version record over to other instances to pass on the new value. Since I copied that code instead of consolidating the logic into a single, reusable function, I had to do that in two places, in the ApplicationPublisher and then again in the CollaborationStoreUtils (I really need to collapse that code into a single set of functions that will for both cases). In both places, I added the following line:

payload.built_on = versionGR.getDisplayValue('built_on');

This was not any kind of a major change, but it was something that I had been meaning to do for a long time, and so while I was in looking at the code, I decided to just go ahead and do it. Next time, we will focus on some real work to start building out some kind of error recovery process so that we can ensure that all of the instances in the community are always kept up to date, even if they miss an update in real time for whatever reason.

Collaboration Store, Part XXXVI

Posted on | by snhackery

“The definition of flexibility is being constantly open to the fact that you might be on the wrong track.”
— Brian Tracy

Although it is long past time to start some serious work on the third major component of this little(?) project, the application installation process, I have been rather hesitant to officially kick that off. Last time, we addressed the last of the reported defects in the first two processes, the initial set-up and the application publishing process. Now, it would seem, would be the time to jump into wrestling with that last remaining primary function and put that to bed before turning our attentions to the list of secondary features that will complete the initial release of the full product. At least, that would appear to be the next logical step.

The reason for my reluctance, however, is that I have taken a cursory look at the various approaches available to accomplish my goal, and quite frankly, I don’t really like any of them. When we converted our Update Set to XML, we were able to fish out enough parts and pieces from the stock product to cobble together a reasonable solution with a minimal amount of questionable hackery. To go in the other direction, to convert the XML back to an Update Set, the only stock component that appears to provide this functionality is bound tightly with the process of uploading a local file from the user’s file system. The /upload.do page, or more specifically, the /sys_upload.do process to which the form on that page is posted, handles both the importing of the XML file and the conversion of that file to an Update Set. There is no way to extract the process that turns the XML into an Update Set, since everything is encapsulated into that one all-encompassing process. For our purposes, we do not have a local file on the user’s machine to upload; our file is an attachment already present on the server, so invoking this process, which seems the only way to go, involves much more than we really need.

To invoke the one and only process that I have found (so far!) to make this conversion, then, we will have to post a multi-part form to /sys_upload.do that includes our XML data along with all of the other fields, headers, and cookies expected by the server-side form processor. On the server side, we should be able to accomplish this with an outbound REST message, or on the client side, we should be able to do this by somehow sending our XML instead of a local file when submitting the form. Each approach has its own merits, but they also each have their own issues, and no matter which way you go, it’s a rather complicated mess.

The Server Side Approach

Posting a multi-part form on the server side is actually relatively simple as far as the form data goes. We can construct a valid body for a standard multipart/form-data POST using our XML data and related information and then send it out using an outbound REST message. That’s the easy part. We just need to add an appropriate Content-Type header, including some random boundary value:

Content-Type: multipart/form-data; boundary=somerandomvalue

Then we can build up the body by including all of the hidden fields on the form, and then add our XML in a file segment that would look something like this:

------------somerandomvalue
Content-Disposition: form-data; name="attachFile"; filename="app_name_v1.0.xml"
Content-Type: application/xml

<... insert XML data here ...>

------------somerandomvalue--

In addition to the XML file component, you would also need to send all of the other expected form field values, some of which are preloaded on the form when it is delivered. To obtain those values, you would have to first issue an HTTP GET request of the /upload.do page and pick those values out of the resulting HTML. This can be accomplished with a little regex magic and the Javascript string .match() method. Here is a simple function to which you can pass the HTML and a pattern to return the value found in the HTML based on the pattern:

function extractValue(html, pattern) {
	var response = '';
	var result = html.match(pattern);
	if (result.length > 1) {
		response = result[1];
	}
	return response;
}

For example, one of the form fields found on the /upload.do page is sysparm_ck. The INPUT element for this hidden field looks like this:

<input name="sysparm_ck" id="sysparm_ck" type="hidden" value="68fa4eee2fa401104425fcecf699b646939f52c6787c23fff22b124fcf58f713235b7478"></input>

To snag the value of that field, you would just pass the HTML for the page and the following pattern to our extractValue function:

id="sysparm_ck" type="hidden" value="(*.?)"

Once you have obtained the value, you can use it to build another “part” in the multi-part form body:

------------somerandomvalue
Content-Disposition: form-data; name="sysparm_ck"

68fa4eee2fa401104425fcecf699b646939f52c6787c23fff22b124fcf58f713235b7478

------------somerandomvalue

All of that is pretty easy to do, and would work great except for one thing: this POST would only be accepted as part of an authenticated session, and cannot just be sent in on its own. Theoretically, we could create an authenticated session by doing a GET of the /login.do page and then a POST of some authoritative user’s credentials, but that would mean knowing and sending the username and password of a powerful user, which is a dangerous thing with which to start getting involved. For that reason, and that reason alone, this does not seem to be a good way to go.

The Client Side Approach

On the client side, you are already involved in an authenticated session, so that’s not any kind of an issue at all. What you do not have is the XML, so to do anything on the client side, we will first need to create some kind of GlideAjax service that will deliver the XML over to the client. Once we have the XML that we would like to upload in place of the normal local file, we will have to perform some kind of magic trick to update the form on the page with our data in the place of a file from the local computer. To do that, we will have to either create our own copy of the /upload.do page or add a global script that will only run on that page, and only if there is some kind of URL parameter indicating that this is one of our processes and not just a normal user-initiated upload. We did this once before with a global script that only ran on the email client, so I know that I can run a conditional script on a stock page if I do not create a page of my own, but the trick will be getting the XML data to be sent back to the server with the rest of the input form.

After nosing around a bit for available options, it appears that you might be able to leverage the DataTransfer object to build a fake file, link it to the form, and then submit the form using something like this:

function uploadXML(xml, fileName) {
	var fileList = new DataTransfer();
	fileList.items.add(new File(xml, fileName));
	document.getElementById('attachFile').files = fileList.files;
	document.forms[0].submit();
}

Of course, there will be a lot more to it than that, as you will need to get a handle on the Update Set created and then try to Preview and Commit it programmatically as well, but this looks like a possibility. Still, you have to move the entire Update Set XML all the way down to the client just to push it all the way back up to the server again, which seems like quite a waste. Plus, with any client-side functionality, there is always the browser compatibility issues that would all need to be tested and resolved. Maybe this would work, but I still don’t like it. It seems like quite a bit of complexity and more than a few opportunities for things to go South. I’m still holding out hope that there is a better way.

Now what?

So … given that I don’t like any of the choices that I have come up with so far, I have decided to set that particular task aside for now in the hopes that a better alternative will come to me before I invest too much effort into a solution with which I am not all that thrilled. There is no shortage of things to do here, so my plan is to just focus on other issues and then circle back to this particular effort when a better idea reveals itself, or I run out of other things to do. Technically, once you have obtained the XML for a particular version from the Host, you can still manually install it by downloading the attachment yourself and importing like any other XML Update Set. That’s not really how I intend all of this to function, but it does work, so it should be OK to set this aside for a time.

Next time, then, instead of forging ahead with this third major component as I had originally planned, I will pick something else out of the pile and we will dig into that instead.

Collaboration Store, Part XXXV

Posted on | by snhackery

“The good news about computers is that they do what you tell them to do. The bad news is that they do what you tell them to do.”
— James Barton

Well, the test results are starting to trickle in, and one of the issues looks rather important, so we need to take a look at that before we jump into our next major effort. The problem with the scoped System Properties came up earlier, and I thought that I had found a way to deal with the issue, but obviously there is still a problem where the initial set-up is concerned. Basically, if you are not in the Collaboration Store scope, then the set-up fails because the updates to the scoped System Properties are not allowed. My idea of moving the update logic to a global component did not resolve this issue, as the problem is related to the active scope of the user rather than the scope of the component.

So, it seems that the answer to the problem is to ensure that the user is in the correct scope before allowing them proceed with the set-up. Fortunately, there is already an area in ServiceNow where that very check is made, and there is even an option in the error message to switch over to the correct scope. You can see that when you bring up a scoped app and you are not in the scope of that application.

Sample error message for incorrect scope on the scoped application form

So it would seem that we could snag the HTML for that message and throw it up in the top of the HTML for the initial set-up widget with some kind of ng-hide so that it only appears if you are in the wrong scope. The first thing that we would need to do, then, is to figure out how to detect the user’s scope and then set up some boolean variable in the widget to indicate whether or not the user is in the correct scope to proceed with the set-up. Looking at the GlideSession API, it seems like the getCurrentApplicationId() method is just what we need. So I added this line to the top of the server script of the widget:

data.validScope = (gs.getCurrentApplicationId() == '5b9c19242f6030104425fcecf699b6ec');

Then, to prevent the user from submitting the form when in the wrong scope, I modified the ng-disabled tag on all of the submit buttons from this:

ng-disabled="!(form1.$valid)"

… to this:

ng-disabled="!(form1.$valid) || !c.data.validScope"

Now all I needed to do was to grab a copy of the HTML source from the application form and paste it in at the top of the HTML for the widget and see if the link would still work. Unfortunately, the link in the existing code relies on the GlideURL object, which is not available in Service Portal widgets, so we are going to have to hack that up a little bit to get around that problem. Here is the existing onclick script:

window.location.href=new GlideURL('change_current_app.do').addParam('app_id', '5b9c19242f6030104425fcecf699b6ec').addParam('referrer', window.location.pathname + window.location.search + window.location.hash).getURL();

Basically, this code just builds a URL, so it seems as if we could just go ahead and build the URL manually and hard-code the link. After doing a little tinkering around to see just what the URL actually was, I came up with this value for our circumstances:

change_current_app.do?app_id=5b9c19242f6030104425fcecf699b6ec&referrer=%24sp.do%3Fid%3Dcollaboration_store_setup

So, my final HTML, including the ng-hide based on my new widget variable, turned out to be this:

<div id="nav_message" class="outputmsg_nav" ng-hide="c.data.validScope">
  <img role="presentation" src="images/icon_nav_info.png">
  <span class="outputmsg_nav_inner">
    &nbsp;
    The <strong>Collaboration Store Set-up</strong> cannot be completed because <strong>Collaboration Store</strong> is not selected in your application picker.
    <a onclick="window.location.href='change_current_app.do?app_id=5b9c19242f6030104425fcecf699b6ec&referrer=%24sp.do%3Fid%3Dcollaboration_store_setup'">
      Switch to <strong>Collaboration Store</strong>
    </a>.
  </span>
</div>

Now all I needed to do was to bring up the set-up widget while in the wrong scope and see how it looked and how the new link worked out once I gave it a try.

Initial set-up screen with error message and link when in the incorrect scope

With the hard-coded link in the onclick function, everything seems to work as intended. The user is placed in the correct scope, the form is reloaded, and the error message goes away. This should finally resolve this annoying issue once and for all (let’s hope!).

The other issue that was reported was that the provider field on the application record was not populated when publishing a new version of an application. It was not clear from the comment whether the missing data was on the Client instance, the Host instance, or both, but I was unable to recreate any of those conditions in any of my testing. I am going to need a little more detail on this one before I can address it, so if anyone encounters this error in any of their testing, please leave a detailed message in the comments so that we can get it resolved.

There is still the potential for more feedback to come, but this particular issue seems to be rather critical, so it’s time to release a new Update Set. While I am at it, I think I will take a different approach on the global components and make an actual Update Set for that as well instead of just exporting the XML for the one global Script Include. This way, I can also include the app’s logo image, which is always missing from the XML generated for a scoped application. Here are the two new Update Sets:

As always, all feedback is welcome, and not just to report issues. If you give this a try and everything actually works, I would love to hear about that as well. Next time out, we will either deal with any more issues to come to light, or we will jump into the next big challenge, which will be to install an app published to the store.

Special Note to Testers

Set-up Process – If you want to test the set-up process, you will need to set up the Host instance first. You cannot set up a Client instance until there is a valid Host instance, as the Client instance set-up process requires access to a valid Host instance. What to look for: check to make sure that all instances in the community have the same list of instances; select Collaboration Store -> Member Organizations to pull up the list of instances in each instance to verify that all instances have the exact same list of all instances in the community.

Application Publishing Process – If you want to test the application publishing process, you will need to go through the set-up process first, and then you can publish an application to the store. This can be done with a single Host instance, but to fully test all of the functionality, you will need at least two Client instances in addition to the Host. What to look for: once you publish the application, check to make sure that all instances in the community have the newly published version of the application, including the XML Update Set attachment; select Collaboration Store -> Member Organizations to pull up the list of instances, then check the Related List of applications under the appropriate instance for the application, and then click on the app to check the Related List of versions under the application.

Application Installation Process – If you want to test the application installation process, you have jumped ahead of the class, as that portion of the app has not yet been developed. However, even though the development of the official installation process has not even been started, if you really want to install an app that has been shared with your instance, you can always download the XML attachment on the version record, import it back into your instance, and go through the normal XML Update Set installation process that you would go through for any other imported Update Set. But again, that’s getting a little ahead of where things stand right at the moment with project’s development.

One More Thing

As mentioned earlier, there is currently no error recovery built into the system at this time. This is definitely something near the top of the we-really-need-to-do-this list, but it is not there right now. What that means is that if you are doing any kind of testing and one or more of the instances in your community is off-line or unavailable for some reason, things will fail and those instances will not get the needed updates. One day we will definitely need to fix that, but for now, if that happens, that’s not a bug in the software to be reported; it’s to be expected until we build in some kind of error recovery into the product.

Also, if you end up testing things and don’t have any issues to report, then by all means, report that, too! You don’t have to have encounter a problem to post your results. If everything worked out as expected, we would definitely love to hear that as well. As always, all feedback of any kind is very much appreciated.

Collaboration Store, Part XXXIV

Posted on | by snhackery | 1 Comment on Collaboration Store, Part XXXIV

“The key is not to prioritize what’s on your schedule, but to schedule your priorities.”
— Stephen Covey

So far, we have completed the first two of the three primary components of the project, the initial set-up process and the application publication process. The last of the three major pieces will be the process that will allow you to install an application obtained from the store. Before we dive straight into that, though, we should pause to take a quick look at what we have, and what still needs to be done in order to make this a viable product. At this point, you can install all of the prerequisites and then grab the latest Update Set, install it, and go through the set-up process to create either a Host or Client instance. Once you get through all of that, you are ready to publish any local Scoped Application to the store, which will then be shared with all other instances in your Collaboration Store community.

What you cannot do, just yet, is to find an application published to the store by some other instance and install it on your own instance. That’s the missing third leg of the stool that we will need to take on next. But that is not all that is left to be done. Once we get the basics to work, there are quite a number of other things to address before one could consider this to be truly usable. Some things are just annoyances, but others are definite features that you would have to consider essential for a complete product.

Speaking of annoyances, one of the things that I really don’t like is that when you publish an app to XML for distribution, the resulting Update Set XML does not include the app’s logo image. Clearly it is a part of the app, and if you push an app to an internal store and pull it down into another instance, it comes complete with the logo, so why they left that out of the XML is a mystery to me. I don’t like that for a couple of reasons: 1) when you pull down the XML for this app, you do not get the logo, and 2) when we use the XML to publish an app to the store, the logo is missing there as well. I have seen people complain about this, but I have not, as yet, seen a solution. I would really like to address that, both for my own apps as well as for the process that we are using in this one.

Speaking of logos, another feature that I would like to have is to provide the ability for each instance to have its own distinctive logo image, so that everything from that particular instance could be tagged with that image as a way to visually identify where the app originated. That’s not a critical feature, which is why I did not include it initially, but it has always been something that I felt should be a part of the process, particularly when you start thinking about ways to browse the store and find what you are looking for. That’s definitely on the We-will-get-around-to-it-one-day list.

Browsing the store is another thing that will need some attention at some point. Right now, we just want to prove that we can set-up the app, publish an application, and install an application published by someone else. Those are the fundamental elements of the app. But once we get all of that worked out, being able to hunt through the store to find what you want will be another important capability to build out. We’re not done with the fundamentals just yet, so we don’t want to put too much energy into that issue right at the moment, but at some point, we will need to create a user-friendly way to easily find what you need.

That, of course, leads into things like searchable keywords, tags, user ratings, reviews, and the like, but now is not the time to head down that rabbit hole. Still, there are a lot of possibilities here, and this could turn into a life-long project in and of itself. That’s probably not a good thing, though!

Anyway, we won’t get anything done if we don’t focus, so we need to stay on task and figure out the application installation process. Once again, there are several options available, but the nicest one seems to be the process that you go through to install an app from an internal store. That’s basically a one-click operation and then the app is installed. Unfortunately, that particular page is neither Jelly nor AngularJS, so you can’t just peek under the hood and see the magic like you can with so many other things on the Now Platform. Another option would be to hack up a copy of the Import XML Action on the Update Set list page to push in the attached XML from a published app version, but that only takes things so far; you still have to Preview the Update Set, resolve any issues, and then manually issue the Commit. It would be much nicer if we could just push a button and have the app installation process run in the background and notify you when it was completed. Obviously, we have some work to do here to come up with the best way to go about this, and we had better figure that out relatively soon. Next time, if we are not dealing with test results from the last release, we will need to start building this out.

Collaboration Store, Part XXXIII

Posted on | by snhackery

“The only impossible journey is the one you never begin.”
— Tony Robbins

Last time, we completed the construction of the process that will send out a new version of an application out to all of the instances in the community. Now we need to come up with a mechanism to kick this process off whenever a new version of an application is published to the Host instance. The simplest way to do that would be to create a new Business Rule on one of the tables housing the artifacts of the new application. To make sure that all of the application artifacts have made their way to the Host, the safest table to target would be the Attachment table, since that is the last artifact to be saved or sent to the Host. Unfortunately, that table is used for all kinds of things, so we will need to make sure that only records attached to the application version table trigger the rule.

To create a new Business Rule, navigate to System Definition > Business Rules and click the New button to bring up the data entry form.

New Business Rule input form

We’ll call our new rule Distribute New Version, associate it with the sys_attachment table, run it async on insert, and check the Advanced checkbox so that we can include a script to run when the rule is triggered.

New Distribute New Version Business Rule

To limit the impact of the rule to just the inserts that we want, we need to add a couple of Filter Conditions, one for the Table and another for the Content Type.

Business Rule filter conditions

On the Advanced tab, we will want to add a Condition that will ensure that this rule will only run on a Host instance. The easiest way to do that is to simply compare the stock instance_name property with the scoped host_instance property. Those two values will only be the same on a Host instance.

gs.getProperty('instance_name') == gs.getProperty('x_11556_col_store.host_instance')

The other thing that we need to do on the Advanced tab is to enter the script that will launch our Subflow. The original way to do that was to utilize the FlowAPI, which is the way that I had coded it initially.

sn_fd.FlowAPI.startSubflow('Distribute_New_Application_Version', {new_version: current.getDisplayValue('table_sys_id'), attachment_id: current.getUniqueValue()});

However, later I decided that it was high time to start embracing the new, preferred approach, which is to use the ScriptableFlowRunner. The code for that is a little more complicated, but at some point the older way of doing this will undoubtedly go away, so we might as well start transitioning things in that direction. Here is the script for the Business Rule using this newer technique:

(function executeRule(current, previous) {
	try {
		var result = sn_fd.FlowAPI.getRunner()
			.subflow('x_11556_col_store.Distribute_New_Application_Version')
			.inBackground()
			.withInputs({new_version: current.getDisplayValue('table_sys_id'), attachment_id: current.getUniqueValue()})
			.run();
	} catch (e) {
		gs.error('Business Rule Distribute New Version - Error running subflow Distribute_New_Application_Version: ' + e.getMessage());
	}
})(current, previous);

That completes the Business Rule, so all we need to do at this point is to Save it and now the next time that an XML attachment is created for a version record, our new Subflow will kick off, distributing the new version of the application to all instances in the community (except for the Host and the instance that produced the application). This also completes the application distribution process, so now it is time to release yet another Update Set so that those of you who have been nice enough to do a little testing can give it a go.

If you have just jumped into this and have not been following along, here is what you will need in order to install the app and try it out:

  • If you have not done so already, you will need to install the latest version of snh-form-fields, which you can find here.
  • You will also need to install this additional global component, which is not included in the scoped application.
  • Once you have successfully installed the prerequisites, you will then need to install the scoped application from this Update Set.
  • After the scoped application has been installed, using an admin account, navigate to Collaboration Store -> Collaboration Store Set-up and complete the set-up process for either a Host or Client instance. The Host instance will need to be set up first, as the set-up for a Client instance requires successful contact with a functioning Host.

And again, as I mentioned last time, in order to test out the distribution process, you will need three or more instances in your community so that one of the Client instances can publish the application to the Host instance and then the Host instance can distribute the new version out to all of the other Client instances. As always, all feedback is welcome and appreciated, and if you are able to get everything to actually work, please let me know in the comments. Thanks!

Next time, we will take a look at where we go from here. Thanks again to all of you who have taken the time to help me out and have let me know what you have found.

Collaboration Store, Part XXXII

Posted on | by snhackery

“There are two ways to write error-free programs; only the third works.”
— Alan J. Perlis

Now that we have all of the Javascript functions needed to push a new application version out to all of the Client instances, we need to create an Action that will call the root function and then create a Subflow that will utilize the new Action. Since we already created a similar Action for the process that shares new Client instances with existing Client instances, the easiest thing to do will be to bring that guy up in the Flow Designer and make a copy. To make a copy, pull up the Action that you want to copy, and use the ellipses menu to select Copy from the drop-down menu.

Making a copy of the Publish New Instance Action

Once we have our copy, we need to make the necessary changes to adapt it to our purpose. The publishNewVersion function that we just created takes three arguments (newVersion, targetInstance, and attachmentId), so we will need to modify the Action Inputs to bring in those values. The new Action Inputs now look like this:

Modified Action Inputs

Similarly, we will need to modify the Input Variables of the Script Step to pass along these values:

Modified Script Input Variables

The last thing that we need to do is modify the script itself. The existing script in the cloned Action was pretty simple, as it was just a call to one of our Script Include functions:

(function execute(inputs, outputs) {
	var csu = new CollaborationStoreUtils();
	csu.publishNewInstance(inputs.new_instance, inputs.target_instance);
})(inputs, outputs);

We also want to call a function, and it is in the very same Script Include, so our modification is just a matter of changing the name of the function along with the arguments passed.

(function execute(inputs, outputs) {
	var csu = new CollaborationStoreUtils();
	csu.publishNewVersion(inputs.new_version, inputs.target_instance, inputs.attachment_id);
})(inputs, outputs);

That’s it for the changes, so now all we have to do is Save and Publish the new Action and we are ready to utilize it in our new Subflow. Creating the Subflow will be another Copy operation, this time from our existing New_Collaboration_Store_Instance Subflow. We’ll call our copy Distribute_New_Application_Version, and once again, we will need to make some modifications to adapt it to our new purpose. The original Subflow had a single input, the New Instance. Instead of a new instance, we will be distributing a new application version, so we can replace the New Instance input with a New Version input that will contain the sys_id of the new version record. We will also need the sys_id of the Attachment record, so we will add an Attachment ID input as well.

Modified Inputs for the new Distribute_New_Application_Version Subflow

The target instances for this process will be the same as those for the original Subflow: all instances in the community except for the Host instance and the instance from which the data was provided (which could also be the Host in this case, since Host instances can publish applications just like any other Client instance). To filter out the data providing instance, we will need to add a new step to use the passed version record sys_id to fetch the GlideRecord that will lead us to the instance that produced the application.

New step to fetch the version record

This gives us the information that we need to complete the filter on the query of the instance table so that we can gather up all of the instances except for the Host and the instance that produced the application.

Modified query filter using the data available in the fetched version record

Now all we need to do is to replace the original custom Action inside of the loop with the new Action that we just created.

New Action added to push application artifacts to the target instance

This completes the changes needed to the cloned Subflow, so again, all we need to do is to Save and Publish and we are all ready to go. Now all we need to is to come up with a way to launch this Subflow whenever a new application version is sent over to the Host instance. We’ll take a look at that next time out, and hopefully release a new Update Set as well so that those of you that have been kind enough to participate in the testing can give this all a trial run.

Collaboration Store, Part XXXI

Posted on | by snhackery

“Someone’s sitting in the shade today because someone planted a tree a long time ago.”
— Warren Buffett

While we wait for additional test results to come in from the corrected Update Set for our latest iteration, it would be a good time to take a look at what we will need to do to complete the last remaining step in the application publishing process. What we have completed so far pushes all of the new application version artifacts to the Host instance. Now we need to build out the process for the Host instance to send out all of those artifacts to all of the other Client instances. Fortunately, we already have in hand most of the code to do that; we just need to clone a few things and make a number of modifications to fit the new purpose.

We have already built a Subflow to push new Client instance information out to all of the existing Client instances during the new instance registration process. Pushing out a new version of an application to those same instances is essentially the same process, so we should be able to clone that guy, and the associated Action, to create our new process. We have also created functions to push over the application record, the version record, and the attached Update Set XML file, and we should be able to copy over those guys as well to complete the process. In fact, we should probably start with the JavaScript functions, as we will need to call the root function in the Action, which will need to be created before it can be referenced in the Subflow.

The functions as written are hard-coded to send everything over to the Host instance. We will want to do basically the same thing, but this time we will be sending things out to various Client instances. Ideally, I should just refactor the code to have the destination instance and credentials passed in so that one function would work in both circumstances, but at this point it was less complicated to just make copies of each function and hack them up for their new purpose. I don’t really like having two copies of essentially the same thing, though, so one day I am going to have to go back and rework the entire mess.

But for now, this is what I came up with when I copied what was once called processPhase5 to create the new publishNewVersion function:

publishNewVersion: function(newVersion, targetInstance, attachmentId) {
	var targetGR = new GlideRecord('x_11556_col_store_member_organization');
	if (targetGR.get('instance', targetInstance)) {
		var token = targetGR.getDisplayValue('token');
		var versionGR = new GlideRecord('x_11556_col_store_member_application_version');
		if (versionGR.get(newVersion)) {
			var canContinue = true;
			var targetAppId = '';
			var mbrAppGR = versionGR.member_application.getRefRecord();
			var request  = new sn_ws.RESTMessageV2();
			request.setHttpMethod('get');
			request.setBasicAuth(this.WORKER_ROOT + targetInstance, token);
			request.setRequestHeader("Accept", "application/json");
			request.setEndpoint('https://' + targetInstance + '.service-now.com/api/now/table/x_11556_col_store_member_application?sysparm_fields=sys_id&sysparm_query=provider.instance%3D' + mbrAppGR.getDisplayValue('provider.instance') + '%5Ename%3D' + encodeURIComponent(mbrAppGR.getDisplayValue('name')));
			var response = request.execute();
			if (response.haveError()) {
				gs.error('CollaborationStoreUtils.publishNewVersion: Error returned from Target instance ' + targetInstance + ': ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
				canContinue = false;
			} else if (response.getStatusCode() == '200') {
				var jsonString = response.getBody();
				var jsonObject = {};
				try {
					jsonObject = JSON.parse(jsonString);
				} catch (e) {
					gs.error('CollaborationStoreUtils.publishNewVersion: Unparsable JSON string returned from Target instance ' + targetInstance + ': ' + jsonString);
					canContinue = false;
				}
				if (canContinue) {
					var payload = {};
					payload.name = mbrAppGR.getDisplayValue('name');
					payload.description = mbrAppGR.getDisplayValue('description');
					payload.current_version = mbrAppGR.getDisplayValue('current_version');
					payload.active = 'true';
					request  = new sn_ws.RESTMessageV2();
					request.setBasicAuth(this.WORKER_ROOT + targetInstance, token);
					request.setRequestHeader("Accept", "application/json");
					if (jsonObject.result && jsonObject.result.length > 0) {
						targetAppId = jsonObject.result[0].sys_id;
						request.setHttpMethod('put');
						request.setEndpoint('https://' + targetInstance + '.service-now.com/api/now/table/x_11556_col_store_member_application/' + targetAppId);
					} else {
						request.setHttpMethod('post');
						request.setEndpoint('https://' + targetInstance + '.service-now.com/api/now/table/x_11556_col_store_member_application');
						payload.provider = mbrAppGR.getDisplayValue('provider.instance');
					}
					request.setRequestBody(JSON.stringify(payload, null, '\t'));
					response = request.execute();
					if (response.haveError()) {
						gs.error('CollaborationStoreUtils.publishNewVersion: Error returned from Target instance ' + targetInstance + ': ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
						canContinue = false;
					} else if (response.getStatusCode() != 200 && response.getStatusCode() != 201) {
						gs.error('CollaborationStoreUtils.publishNewVersion: Invalid HTTP Response Code returned from Target instance ' + targetInstance + ': ' + response.getStatusCode());
						canContinue = false;
					} else {
						jsonString = response.getBody();
						jsonObject = {};
						try {
							jsonObject = JSON.parse(jsonString);
						} catch (e) {
							gs.error('CollaborationStoreUtils.publishNewVersion: Unparsable JSON string returned from Target instance ' + targetInstance + ': ' + jsonString);
							canContinue = false;
						}
						if (canContinue) {
							targetAppId = jsonObject.result.sys_id;
						}
					}
				}
			} else {
				gs.error('CollaborationStoreUtils.publishNewVersion: Invalid HTTP Response Code returned from Target instance ' + targetInstance + ': ' + response.getStatusCode());
			}
			if (canContinue) {
				this.publishVersionRecord(targetInstance, token, versionGR, targetAppId, attachmentId);
			}
		} else {
			gs.error('CollaborationStoreUtils.publishNewVersion: Version record not found: ' + newVersion);
		}
	} else {
		gs.error('CollaborationStoreUtils.publishNewVersion: Target instance record not found: ' + targetInstance);
	}
},

Unlike the original application publication process, we are not bouncing back and forth between the client side and the server side, so with the successful completion of one artifact, we can simply move on to pushing over the next artifact. To create the next step in the process, the publishVersionRecord function, I started out with a copy of the original processPhase6 function. Here is the end result:

publishVersionRecord: function(targetInstance, token, versionGR, targetAppId, attachmentId) {
	var canContinue = true;
	var payload = {};
	payload.member_application = targetAppId;
	payload.version = versionGR.getDisplayValue('version');
	var request  = new sn_ws.RESTMessageV2();
	request.setBasicAuth(this.WORKER_ROOT + targetInstance, token);
	request.setRequestHeader("Accept", "application/json");
	request.setHttpMethod('post');
	request.setEndpoint('https://' + targetInstance + '.service-now.com/api/now/table/x_11556_col_store_member_application_version');
	request.setRequestBody(JSON.stringify(payload, null, '\t'));
	response = request.execute();
	if (response.haveError()) {
		gs.error('CollaborationStoreUtils.publishVersionRecord: Error returned from Target instance ' + targetInstance + ': ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
		canContinue = false;
	} else if (response.getStatusCode() != 201) {
		gs.error('CollaborationStoreUtils.publishVersionRecord: Invalid HTTP Response Code returned from Target instance ' + targetInstance + ': ' + response.getStatusCode());
		canContinue = false;
	} else {
		jsonString = response.getBody();
		jsonObject = {};
		try {
			jsonObject = JSON.parse(jsonString);
		} catch (e) {
			gs.error('CollaborationStoreUtils.publishVersionRecord: Unparsable JSON string returned from Target instance ' + targetInstance + ': ' + jsonString);
			canContinue = false;
		}
		if (canContinue) {
			targetVerId = jsonObject.result.sys_id;
			this.publishVersionAttachment(targetInstance, token, targetVerId, attachmentId);
		}
	}
},

The last step in the process, then, is to send over the Update Set XML file attachment. For that one, I started out with the processPhase7 function and ended up with this:

publishVersionAttachment: function(targetInstance, token, targetVerId, attachmentId) {
	var gsa = new GlideSysAttachment();
	var sysAttGR = new GlideRecord('sys_attachment');
	if (sysAttGR.get(attachmentId)) {
		var url = 'https://';
		url += targetInstance;
		url += '.service-now.com/api/now/attachment/file?table_name=x_11556_col_store_member_application_version&table_sys_id=';
		url += targetVerId;
		url += '&file_name=';
		url += sysAttGR.getDisplayValue('file_name');
		var request  = new sn_ws.RESTMessageV2();
		request.setBasicAuth(this.WORKER_ROOT + targetInstance, token);
		request.setRequestHeader('Content-Type', sysAttGR.getDisplayValue('content_type'));
		request.setRequestHeader('Accept', 'application/json');
		request.setHttpMethod('post');
		request.setEndpoint(url);
		request.setRequestBody(gsa.getContent(sysAttGR));
		response = request.execute();
		if (response.haveError()) {
			gs.error('CollaborationStoreUtils.publishVersionAttachment: Error returned from Target instance ' + targetInstance + ': ' + response.getErrorCode() + ' - ' + response.getErrorMessage());
		} else if (response.getStatusCode() != 201) {
			gs.error('CollaborationStoreUtils.publishVersionAttachment: Invalid HTTP Response Code returned from Target instance ' + targetInstance + ': ' + response.getStatusCode());
		}
	} else {
		gs.error('CollaborationStoreUtils.publishVersionAttachment: Invalid attachment record sys_id: ' + attachmentId);
	}
},

So that takes care of the functions. Now we have to create an Action in the Flow Designer that will call the function, and then produce a new Subflow that will leverage that action. That sounds like a good subject for our next installment.

Special Note to Testers

If you want to test the set-up process, you can do that with a single instance. You just have to select the Host option during the set-up process. If you want to test the Client set-up process, you will need at least two instances, one to serve as the Host, which you will need to reference when you set up the Client (you cannot be a Client of your own Host). But if you really want to test out the full registration process, including the Subflow that informs all existing instances of the new instance, you will need at least three participating instances: 1) the Host instance, 2) the new Client instance, and 3) an existing Client instance that will be notified of the new member of the community.

The same holds true for the application publishing process. You can test a portion of the publishing process by publishing an app on a single Host instance, but if you want to test out all of the parts and pieces, you will want to publish the app from a Client instance and make sure that it makes its way all the way back to the Host. Once we complete the application distribution process, full testing will require at least three instances to verify that the Host distributes the new version of the application to other Clients.

Collaboration Store, Part XXX

Posted on | by snhackery

“Writing the first 90 percent of a computer program takes 90 percent of the time. The remaining ten percent also takes 90 percent of the time and the final touches also take 90 percent of the time.”
— Neil J. Rubenking

Well, the test results are starting to pour in, and it looks like I screwed things up when I manually edited the Update Set in an effort to eliminate one of the errors reported in the earlier testing cycle. It seems as if I should have left well enough alone and just informed people to ignore the error if it comes up. Here is an unmolested Update Set that should not have the problem that I created by hacking up the earlier version after it was generated. Hopefully, this will resolve that issue.

Two things to note, then, of this new version: 1) if you happen to get the unfortunate Table ‘sys_hub_action_status_metadata’ does not exist error, just ignore it, and 2) if you get any preview errors related to any sys_properties, be sure to skip those updates, as you do not want to overlay the property values that were established during the set-up process.

One of the other things that was reported was that it was not really clear as to what, exactly, needed to be tested. I have a tendency to focus on the construction process exclusively, without a lot of attention to the actual end product itself, so let’s see if we can’t rectify that situation a little bit now.

The initial early release of this effort was focused on the set-up process. All the set-up process does is set you up as the Host instance, or get you registered with a Host instance if you are setting up a Client instance. That’s all that it did, so the testing was limited to attempting to set up a Host, and then attempting to set up one or more Clients. A successful test would have all instances appear in the instance table on every instance involved in the testing. That seemed to be pretty straightforward.

For this next iteration, we introduced the ability to actually publish a Scoped Application to the Host. To test this newest feature, you will first have to have gone through the set-up process successfully, and then you need an app to publish. Any app in development will do, and if you don’t have one, you can always just stub one out for the testing.

To publish an app, you need to bring up the app’s primary form, and if all went well with the installation, there should be a new UI Action down at the bottom of the page called Publish to Collaboration Store.

New Publish to Collaboration Store UI Action

If you click on that guy and follow the ensuing pop-up dialogs through completion, the app should be published. To verify that all went well, you will have to go over to the Host instance and see if the app actually appears in the Related List under the publisher’s instance record. You should verify the presence of the application record, the version record, and the XML Update Set attachment. If all of those things are present, then the app was successfully published to the Host.

The one thing that will not happen just yet is for the newly published app to be distributed to any of the other Client instances in the community. That process is still under development, and is not included in this version of the application. Once that gets completed and all of the issues from this round of testing get resolved, we will put out yet another beta test version and go through the testing process all over again.

Thanks again to all of you who are taking the time to take this out for a spin. Any and all feedback is greatly appreciated. Please feel free to report any issues or successes in the comments below. Next time, if there are no further results to review, we will take a look at building out the distribution of newly published versions to the rest of the instances served by the Host.

Collaboration Store, Part XXIX

Posted on | by snhackery | 6 Comments on Collaboration Store, Part XXIX

“As a rule, software systems do not work well until they have been used, and have failed repeatedly, in real applications.”
— David Lorge Parnas

Now that we have completed the initial version of the application publishing process, it would be nice to release a new Update Set so that the folks who are inclined to help out with the testing can try it all out. However, we have already received feedback from the earlier testing of the set-up process, and we should really address all of those issues before publishing a new version for further testing. Here are the items discovered during the testing of the first version of the software released earlier:

  • Installation error: Table ‘sys_hub_action_status_metadata’ does not exist
  • Not allowing update of property: x_11556_col_store.store_name
  • Not allowing update of property: x_11556_col_store.host_instance
  • In the setup, the instance name field doesn’t inform you that you only need the instance prefix, not the full url
  • You can only collaborate with one host

I was able to reproduce them all, and here is what I ended up doing for each:

Installation error: Table ‘sys_hub_action_status_metadata’ does not exist

I tried to find out some information on the purpose and use for this table, but I couldn’t really find anything that told me anything of value. I still believe that the ‘sys_hub_action_status_metadata’ table is related to a version or plugin that I have in my instance, but was not present in the instance on which the test installation was being performed. Since it didn’t seem as if it was anything that had anything to do with the operation of the application, I decided to just delete all references to it in the Update Set, just to avoid this issue. I’m not sure if that will cause any issues with any instances that actually do have this table, but it seemed like something worth a try and we’ll see what happens. If it causes a problem, I will not do that in the future and just throw in some release notes that say just ignore the error if it comes up. But let’s see if this works, first.

Not allowing update of property: x_11556_col_store.store_name
Not allowing update of property: x_11556_col_store.host_instance

This one took a little bit of research and a little bit of trial and error (mostly error!), but I eventually solved the problem by moving all updates to these properties into my global utilities so that the command was being executed by a global component instead of a scoped component. I did this once before with the gs.sleep() function, and this worked out just as well. Here is the function that I added to the global utilities:

setProperty: function(name, value) {
	gs.setProperty(name, value);
},

Once I created the function in the global utilities, it was just a matter of changing all gs.setProperty() function calls to csgu.setProperty() and that seemed to have done the trick.

In the setup, the instance name field doesn’t inform you that you only need the instance prefix, not the full url

Since the snh-form-field tag provides for a “help” attribute, which appears underneath the label, I simply updated the definition for that field to include a little help:

<snh-form-field
  snh-model="c.data.host_instance_id"
  snh-name="host_instance_id"
  snh-label="Host Instance ID"
  snh-help="Enter the instance ID only, not the full URL of the instance (https://{instance_id}.servicenow.com))"
  snh-required="c.data.instance_type == 'client'"
  ng-show="c.data.instance_type == 'client'"/>

This renders a little help text underneath the label, and looks like this when the page comes up:

Initial set-up screen with added help text for the Host Instance ID

That should resolve this issue.

You can only collaborate with one host

As I mentioned earlier when this was first reported, this is by design. Allowing multiple Host instances introduces a level of complexity with which I’m not quite ready to deal just yet. At this point, we will just file this one under Will not fix or Future release for now.

Other than these issues, I have not personally encountered or heard of any other issues with the set-up process, but that doesn’t mean that the testing is complete by any means. If anyone wants to join the testing process and you missed out on testing the earlier version, you will still have to work your way through the set-up process for any instances involved, so please report any issues with the set-up process as well as any issues with the application publication process.

Speaking of the application publication process, there is still yet another aspect of this process that remains to be developed. Once a new application has been published to the Host instance, the Host instance will need to push that version out to all of the other Client instances. This version does not include that functionality, but we will need to throw that in at some point. I just did not want to hold up the testing for that particular feature, since it really is a completely independent operation.

For those of you who are interested in participating in the testing, you will need this new Update Set, plus this additional global component that is not included in the scoped application. Also, if this is your first time installing the application, you will also need the latest version of snh-form-fields, which you can find here. As always, if you have any feedback, positive or negative, please leave the details in the comments. All information is welcome and much appreciated. Thanks in advance for your assistance.