Using the GlideExcelParser on a Fetched Remote File

“Do not go where the path may lead, go instead where there is no path and leave a trail.”
Ralph Waldo Emerson

When we hacked the REST Message API to fetch a remote file, all we did with the file was to attach it to an existing Incident. But there are times when what you are really after is the contents of the file, not the file itself. When that file happens to be an Excel spreadsheet, the ServiceNow platform comes shipped with a handy tool called the GlideExcelParser that will allow you to dig into that file and pull out the data. By combining this capability with our earlier work on fetching remote files, you can reach out to a URL, pull in a spreadsheet from another site, and then parse the spreadsheet to extract the data that you need.

One example of where this might be useful would be in importing data records where one or more fields on the record is a URL of an associated spreadsheet containing relevant information such a list of items to be ordered or a list of people for which you would like to place an order. Another example might be a mail-in ticket where the details are not included in the body of the email or attached to the email as a file, but accessible via a URL link found in the message body. Basically, this applies to any scenario where you have the URL of a file, and just want the data contained in the file and not the actual file.

To demo this capability, we just need to find a spreadsheet hosted on the Internet that we can use for an example. Something like this:

https://www.servicenow.com/content/dam/servicenow-assets/public/en-us/doc-type/other-document/csc/servicenow-platform-support-calculator-template.xlsx

This document has three sheets, but for our purposes we can just focus on one of the areas of one of the sheets, just to show how things work. Here is the second sheet, labeled Input:

Example spreadsheet to use for demonstration purposes

For our little demo, we will focus on the grey block on rows 20 through 31 so that we can pull out some data and then use it for some other purpose such as creating an Incident. Once we pull out the data, we will no longer need the actual file, so we can delete the attachment that we created to tidy things up when we are done. Since this is just a demo, we can do the whole thing in a Fix Script that we can discard later once our little proof of concept has been completed.

The first thing that we will need to do is to go get our file. For that, we can leverage our earlier Script Include, but we will need to attach it to something while we work with it. For the purposes of this demonstration, let’s just attach it to the Fix Script itself.

var url = 'https://snhackery.com/update_sets/servicenow-platform-support-calculator-template.xlsx';
var table = 'sys_script_fix';
var sys_id = '0bbdf83e2f21d0104425fcecf699b660';
var fileFetcher = new FileFetchUtils();
var attachment = fileFetcher.fetchFileFromUrlAndAttach(table, sys_id, url);

That was simple enough. Now let’s parse the file.

var stream = new GlideSysAttachment().getContentStream(attachment);
var parser = new sn_impex.GlideExcelParser();
parser.setSheetName('Input');
var row = [];
if (parser.parse(stream)) {
	while (parser.next()) {
		row.push(parser.getRow());
	}
} else {
	gs.info("Error: Unable to parse uploaded file");
}

OK, this one is a little more complicated. First, we create a content stream from the file attachment. Then we create a new parser. Then we tell the parser that we want it to parse the second sheet, the one labeled Input. Then we create an empty array to store the rows, and loop through the parsed stream adding rows to our array one at a time until we run out of rows. Now we have an array of row that that we can use for whatever purpose that we want.

At this point, we can delete the attachment, since we have no further use for it, but let’s save that for a little later and first see what we can do with our row data. To start with, let’s just make sure that we have some data, and if we do, let’s call a function in which we can encapsulate all of the code to utilize the data.

if (row.length) {
	gs.info('Incident created: ' + createIncident(row));
} else {
	gs.info("Error: No data rows in parsed uploaded file");
}

Now it’s time to deal with the data. The parsed rows are values keyed by the column heading. Where there is no column heading, then the key is the Excel column letter. In our little example, we are interested in the data in the second and third columns, and in the first row, there is a value in the second column (Data Input Form), but not in the third. This means that second column values will be obtained with thisRow[‘Data Input Form’] and third column values will be obtained with thisRow[‘C’] (since C is Excel’s name for the third column).

So to rebuild that little grey block of data, we will need the second column value for row #19, the second and third column values for rows 21 through 26, and then the second and third column values for row 28. Now, that’s assuming that no one adds or removes a line or a column from the spreadsheet when filling out the template, which could be dealt with using a lot more defensive coding strategy, but that’s a little beyond the scope of this little exercise here. For our purposes, we are just going to assume that things are going to be right where they are supposed to be. So here is our function.

function createIncident(row) {
	var full_description = 'The following data was extracted from the fetched spreadsheet:\n\n';
	full_description += row[19]['Data Input Form'] + '\n\n';
	for (var i=21; i<27; i++) {
		full_description += '\t' + row[i]['Data Input Form'] + ':\t' + row[i]['C'] + '\n';
	}
	full_description += '\n' + row[28]['Data Input Form'] + '\t' + row[28]['C'];
	var incidentGR = new GlideRecord('incident');
	incidentGR.caller_id = gs.getUserID();
	incidentGR.short_description = 'Excel parsing example';
	incidentGR.description = full_description;
	incidentGR.assignment_group.setDisplayValue('Service Desk');
	incidentGR.insert();
	return incidentGR.getDisplayValue('number');
}

Now that the Incident has been created, we just need to delete the attachment record to clean things up. That’s just a few more lines of code.

var attachmentGR = new GlideRecord('sys_attachment');
attachmentGR.get(attachment);
attachmentGR.deleteRecord();

That’s it. All that is left is to save the script and then click on that little Run Fix Script button to see what happens.

*** Script: Incident created: INC0010019

Let’s go check out our new Incident and see how things came out.

Incident generated from data extracted from Excel spreadsheet

Well, it looks pretty close, but we missed a line of data. The System Administrators line didn’t make it over. Looks like our start index should have been 20 and not 21. Oh well, you get the idea. I would go back and fix it, but this is just a little demo to show how things could work together, and I think we have accomplished that. If you want to play around with this, you will be using your own spreadsheet anyway, so you’ll be doing something completely different. But if you want to fix your own copy of this one, please go right ahead!

Hacking the REST Message API to Fetch a Remote File

“The way to get started is to quit talking and begin doing.”
Walt Disney

The other day I needed to fetch a file that was posted on another web site via its URL and process it. I have processed ServiceNow file attachments before, but I have never attempted to go out to the Internet and pull in a file using an HTTP GET of the URL. There are several ways to do that in Javascript such as XMLHttpRequest or fetch, but none of those seemed to work in server-side code in ServiceNow. But you can open up a URL using the ServiceNow RESTMessageV2 API, so I thought that maybe I would give that a shot. How hard could it be?

I decided to encapsulate everything into a Script Include, mainly so that if I ever needed to do this again, I could call the same function in some other context. My thought was to pass in the URL of the file that I wanted to fetch along with the table name and sys_id of the record to which I wanted the file to be attached, and then have the function fetch the file, attach it, and send back the attachment. Something like this:

var FileFetchUtils = Class.create();
FileFetchUtils.prototype = {
	initialize: function() {
	},

	fetchFileFromUrlAndAttach: function(table, sys_id, url) {
		...
	},

	type: 'FileFetchUtils'
};

That was idea, anyway. Let’s say that I wanted to attach this file, available on the ServiceNow web site, to some Incident:

https://www.servicenow.com/content/dam/servicenow-assets/public/en-us/doc-type/success/playbook/implementation.pdf

The code to do that would look something like this:

var fileFetcher = new FileFetchUtils();
fileFetcher.fetchFileFromUrlAndAttach('incident', incSysId, 'https://www.servicenow.com/content/dam/servicenow-assets/public/en-us/doc-type/success/playbook/implementation.pdf');

All we need to do now is come up with the code needed to do all of the work of fetching the file, attaching it to the specified record, and returning the attachment. To begin, we will need to extract the name of the file from the URL. Assuming that the file name is the last component of the path on the URL, we can do that by splitting the path into its component parts and grabbing the last part.

var parts = url.split('/');
var fileName = parts[parts.length-1];

Next, we will need to create and configure the request object.

var request  = new sn_ws.RESTMessageV2();
request.setHttpMethod('get');
request.setEndpoint(url);

The next thing to do would be to execute the request, but before we do that, we can take advantage of a nice built-in feature that will really simplify this whole operation. There is an available function of the RESTMessageV2 API that allows you to declare your intent to turn the retrieved file into an attachment, which will then handle all of the details of doing that on your behalf when the request is executed. You just need to invoke the function before you execute the request.

request.saveResponseBodyAsAttachment(table, sys_id, fileName);        
response = request.execute();

Although that really makes things super simple, it’s still a good idea to check the HTTP Response Code, just to make sure all went well. If not, it’s a good practice to relay that to the user.

if (response.getStatusCode() == '200') {
	...
} else {
	returnValue = 'Error: Invalid HTTP Response: ' + response.getStatusCode();
}

Now, assuming that things actually did go well and our new attachments was created, we still want to send that back to the calling script as a response to this function. The RESTMessageV2 API saveResponseBodyAsAttachment function does not return the attachment that is created, so we will have to use the table and sys_id to hunt it down. And if we cannot find it for any reason, we will want to report that as well.

var attachmentGR = new GlideRecord('sys_attachment');
attachmentGR.addQuery('table_name', table);
attachmentGR.addQuery('table_sys_id', sys_id);
attachmentGR.orderByDesc('sys_created_on');
attachmentGR.query();
if (attachmentGR.next()) {
	returnValue = attachmentGR.getUniqueValue();
} else {
	returnValue = 'Error: Unable to fetch attachment';
}

That should now be everything that we need to fetch the file, attach it, and send back the attachment. Putting it all together, the entire Script Include looks like this:

var FileFetchUtils = Class.create();
FileFetchUtils.prototype = {
	initialize: function() {
		this.REST_MESSAGE = '19bb0cde2fedd4101a75ad2ef699b6da';
	},

	fetchFileFromUrlAndAttach: function(table, sys_id, url) {
		var returnVlaue = '';
		var parts = url.split('/');
		var fileName = parts[parts.length-1];
		var request  = new sn_ws.RESTMessageV2();
		request.setHttpMethod('get');
		request.setEndpoint(url);
		request.saveResponseBodyAsAttachment(table, sys_id, fileName);        
		response = request.execute();
		if (response.getStatusCode() == '200') {
			var attachmentGR = new GlideRecord('sys_attachment');
			attachmentGR.addQuery('table_name', table);
			attachmentGR.addQuery('table_sys_id', sys_id);
			attachmentGR.orderByDesc('sys_created_on');
			attachmentGR.query();
			if (attachmentGR.next()) {
				returnValue = attachmentGR.getUniqueValue();
			} else {
				returnValue = 'Error: Unable to fetch attachment';
			}
		} else {
			returnValue = 'Error: Invalid HTTP Response: ' + response.getStatusCode();
		}
		return returnValue;
	},

	type: 'FileFetchUtils'
};

Now all we need to do is find an Incident to use for testing and use the background scripts feature to give it a try. First, we’ll need to pull up an Incident and then use the Copy sys_id option of the hamburger drop-down menu to snag the sys_id of the Incident.

Grabbing the sys_id of an Incident

Now we can pop over to the background script processor and enter our code, using the sys_id that we pulled from the selected Incident.

Testing using a background script

After running the script, we can return to the Incident to verify that the file from the specified URL is now attached to the Incident.

The selected test Incident with the remote file attached

Just to make sure that all went well, you can click on the download link to pull down the attachment and look it over, verifying that it came across complete and intact. That pretty much demonstrates that it all works as we had intended. If you would like a copy to play around with on your own, you can pick it up here.

Hacking the Scripted REST API, Part II

“If you find a path with no obstacles, it probably doesn’t lead anywhere.”
Frank A. Clark

Last time, we built a Scripted REST API with a single Scripted REST Resource. To finish things up, we just need to add the actual script to the resource. To process our legacy support form, create an Incident, and respond to the user, the script will need to handle the following operations:

  • Obtain the input form field values from the request,
  • Format the data for use in creating the Incident,
  • Locate the user based on the email address, if one exists,
  • Create the Incident, and
  • Return the response page.

Obtain the input form field values from the request

This turned out to be much simpler than I first made it out to be. I knew that the out-of-the-box Scripted REST API was set up to handle JSON, both coming in and going out, and could also support XML, but I never saw anything regarding URL encoded form fields, so I assumed I would have to parse and unencode the request body myself. The problem that I was having, though, was getting the actual request body. I tried request.body, request.body.data, request.body.dataString, and even request.body.dataStream — nothing produced anything but null or errors. Then I read somewhere that that the Scripted REST API treats form fields as if they were URL query string parameters, and lumps them all together in a single map: request.queryParams. Once I learned that little tidbit of useful information, the rest was easy.

// get form data from POST
var formData = request.queryParams;
var name = formData.name[0];
var email = formData.email[0];
var short_description = formData.title[0];
var description = formData.description[0];

It did take me a bit to figure out that the values returned from the map are arrays and not strings, but once that became clear in my testing, I just added an index to the form field name and everything worked beautifully.

Format the data for use in creating the Incident

This was just a matter of formatting a few of the incoming data values with labels so that anything that did not have a field of its own on the Incident could be included in the Incident description.

// format the data
var full_description = 'The following issue was reported via the Legacy Support Form:\n\n';
full_description += 'Name: ' + name + '\n';
full_description += 'Email: ' + email + '\n\n';
full_description += 'Details:\n' + description;

Locate the user based on the email address

This doesn’t really have much to do with hacking the Scripted REST API, but I threw it in just as an example of the kind of thing that you can do once you have some data with which to work. In this case, we are simply using the email address that was entered on the form to search the User table to see if we have a user with that email. If we do, then we can use as the Caller on the Incident.

// see if we have a user on file with that email address
var contact = null;
var userGR = new GlideRecord('sys_user');
if (userGR.get('email', email)) {
	contact = userGR.getUniqueValue();
}

Create the Incident

This part is pretty vanilla as well.

// create incident
var incidentGR = new GlideRecord('incident');
if (contact) {
	incidentGR.caller_id = contact;
} else {
	incidentGR.caller_id.setDisplayValue('Guest');
}
incidentGR.contact_type = 'self-service';
incidentGR.short_description = short_description;
incidentGR.description = full_description;
incidentGR.assignment_group.setDisplayValue('Service Desk');
incidentGR.insert();
var incidentId = incidentGR.getDisplayValue('number');

The last line grabs the Incident number from the inserted Incident so that we can send that back to the user on the response page.

Return the response page

Now that we have complete all of the work, the last thing left to do is to respond to the user. Again, since we are not using the native JSON or XML formats, we are going to have to do some of the work a little differently than the standard Scripted REST API. Here is the working code:

// send response page
response.setContentType('text/html');
response.setStatus(200);
var writer = response.getStreamWriter();
writer.writeString(getResponsePageHTML(incidentId));

The first thing that you have to know is that you must set the content type and status before you get the stream writer from the response. If you don’t do that first, then things will not work. And even though you clicked the override checkbox and specified text/html as the format in the Scripted REST API definition, you still have to set it here as well. But once you do all of that, and do it in the right sequence, it all works great.

The response string itself is just the text of a standard HTML page. I encapsulated my sample page into a function so that it could be easily replaced with something else without disturbing any of the rest of the code. The sample version that I put together looks like this:

function getResponsePageHTML(incidentId) {
	var html = '';

	html += '<html>';
	html += ' <head>';
	html += '  <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.5.0/css/bootstrap.min.css" integrity="sha384-9aIt2nRpC12Uk9gS9baDl411NQApFmC26EwAOH8WgZl5MYYxFfc+NcPb1dKGj7Sk" crossorigin="anonymous">';
	html += '  <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.5.0/js/bootstrap.min.js" integrity="sha384-OgVRvuATP1z7JjHLkuOU7Xw704+h835Lr+6QL9UvYjZE3Ipu6Tp75j7Bh/kR0JKI" crossorigin="anonymous"></script>';
	html += ' </head>';
	html += ' <body>';
	html += '  <div style="padding: 25px;">';
	html += '   <h2>Thank you for your input</h2>';
	html += '   <p>We appreciate your business and value your feedback. Your satisfaction is of the utmost concern to us.</p>';
	html += '   <p>Your issue has been documented and one of our marginally competent technicians will get back to you within a few months to explain to you what you have been doing wrong.</p>';
	html += '   <p>Your incident number is ' + incidentId + '.</p>';
	html += '  </div>';
	html += ' </body>';
	html += '</html>';

	return html;
}

That’s it for the coding in ServiceNow. To see how it all works, we just need to point our old form to our new form processor after which we can pull up the modified form in a browser and give it a try. To repoint the legacy form, pull it up in a text editor, find the form tag, and enter the path to our Scripted REST API in the action attribute:

<form action="https://<instance>.service-now.com/api/<scope>/legacy/support" method="post">

With that little bit of business handled, we can pull up the form, fill it out, click on the submit button, and if all goes well, be rewarded for our efforts with our HTML response, including the number of the newly created Incident:

HTML response from the Scripted REST API form processor

Just to be sure, let’s pop over to our instance and check out the Incident that we should have created.

Incident created from submitting the legacy form

Well, that’s all there is to that. We built a Scripted REST API that accepted standard form fields as input and responded with a standard HTML web page, and then pointed an old form at it to produce an Incident and a thank you page. All in all, not a bad little perversion of the REST API feature of the tool!

Hacking the Scripted REST API to process a form

“Try something new each day. After all, we’re given life to find it out. It doesn’t last forever.”
Ruth Gordon

One of our older systems includes a form through which you could report issues, and when you filled out the form and submitted it, it would send an e-mail to the support team. Pretty cool stuff back in the day, but the procedure bypasses the Service Desk and hides the activity from our support statistics because no one ever opens up an Incident. They just quietly resolve the issue and move on without a trace. The people who like to have visibility into those kinds of activities are not really too keen on these little side deals that allow certain groups to fly below the radar. So the question arose as to whether or not we could keep the form, with which everyone was comfortable and familiar, but have it create an Incident rather than send an e-mail.

Well, the first thing that came to mind was to just send the existing e-mail to the production instance and then set up an inbound mail processor to turn the e-mail into an Incident. The problem with that approach, though, was the the Incident was created off-line, and by that time, you had no way to inform the user that the Incident was successfully created or to give them the ID or some kind of handle to pull it up and check on the progress. What would really be nice would be to be able to simply POST the form to ServiceNow and have it respond back with an HTML thank you page. ServiceNow is not really set up to be a third-party site forms processor, though, so that really didn’t seem to be a feasible concept.

But, then again …

ServiceNow does have the Scripted REST API, but that is built for Web Services, not user interaction. Still, with a little creative tweaking maybe we could actually fool it into taking a form POST and responding with an HTML page. That would actually be interesting. And as it turns out, it wasn’t all that hard to do.

To make our example relatively simple and easy to follow, let’s just build a nice clean HTML page that contains nothing but our example legacy form:

Simple stand-alone input form for demonstration purposes

This clears out all of the window dressing, headers, footers, and other clutter and just gets down to the form itself. None of that other stuff has any relevance to what we are trying to do here, so we just want a simple clean slate without all of the distractions. Here is the entire HTML code for the page:

<html>
 <head>
  <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.5.0/css/bootstrap.min.css" integrity="sha384-9aIt2nRpC12Uk9gS9baDl411NQApFmC26EwAOH8WgZl5MYYxFfc+NcPb1dKGj7Sk" crossorigin="anonymous">
  <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.5.0/js/bootstrap.min.js" integrity="sha384-OgVRvuATP1z7JjHLkuOU7Xw704+h835Lr+6QL9UvYjZE3Ipu6Tp75j7Bh/kR0JKI" crossorigin="anonymous"></script>
 </head>
 <body>
  <form action="/some/legacy/form/handler" method="post">
   <div style="padding: 25px;">
    <h2>Legacy Support Form</h2>
    <div style="margin-top: 10px;">
     <label for="name">Please enter your name:</label>
    </div>
    <div>
     <input name="name" size="64"/>
    <div style="margin-top: 10px;">
     <label for="email">Please enter your email address:</label>
    </div>
    <div>
     <input name="email" size="64"/>
    </div>
    <div style="margin-top: 10px;">
     <label for="title">Please enter a brief statement describing the issue:</label>
    </div>
    <div>
     <input name="title" size="64"/>
    </div>
    <div style="margin-top: 10px;">
     <label for="description">Please describe the problem in detail:</label>
    </div>
    <div>
     <textarea name="description" cols="62" rows="5"></textarea>
    </div>
    <div style="margin-top: 20px;">
     <input type="submit" value="Submit Problem Report"/>
    </div>
   </div>
  <form>
 </body>
</html>

The idea here is to now take the existing form handler, as specified in the action attribute of the form tag, and replace it with a URL for a ServiceNow “web service” that we will create using the Scripted REST API tools. That is the only change that we want to make on this existing page. Everything else should look and behave exactly as it did before; we are just pointing the form to a new target for processing the user input. So let’s build that target.

To begin, select the Scripted REST APIs option on the left-hand sidebar menu, which will bring up the list of all of the existing Scripted REST APIs. From there, click on the New button, which will take you to a blank form on which you can start entering the details about your new Scripted REST API.

Initial Scripted REST API data entry form

Enter the name of your new Scripted REST API along with the API ID, which will become a component in the eventual URL that will take you to your new web service. Once you have saved these initial values, you will be returned to the list, and your newly created API will appear on the list. Select it, and you will be taken to an expanded form where you can now enter the rest of the information.

Full Scripted REST API data entry form

Here is one of the secrets to our little hack: you need to check both of the override boxes so that you can change the MIME type for the data that will be flowing in both directions. For the data that is coming in, you will want to enter x-www-form-urlencoded. These are are the form fields coming in from the input form. For the data going out, you will want to enter text/html. This is the response page that will go back to the browser and be rendered by the browser for display to the user. Form fields come into our new service and HTML comes back out.

Once you have saved your overrides, you can create your Resource. A Scripted REST API can have one or more Resources, and they appear on the form down at the bottom as a Related List. Open up the Resources tab and click on the New button to create your Resource. This brings up the Resource form.

Scripted REST Resource form

On this form, you want to enter the name of your Resource, the HTTP method (POST), and the relative path, which is another component of the URL for this service. Once you save that, all that is left is my favorite part, the coding. In fact, this is probably a good place to stop, as we are done with all of the forms and form fields. Next time out, we can focus exclusively on the code.