Formatted Script Search Results, Enhanced

“Delivering good software today is often better than perfect software tomorrow, so finish things and ship.”
— David Thomas, The Pragmatic Programmer: From Journeyman to Master

It always seems to happen. The more I play around with some little bauble that I’ve thrown together, the more ideas that pop into my head to make it just a little bit better. I’ve actually found my script search tool to be quite useful now that I have finally gotten it to behave the way in which I would expect it to behave. But the more that I use it, the more that I find that I want add to it.

One thing that I realized after I released the code was that you can also find script in Condition String fields. This blows my whole theory that all column types that contain script have the word script in their name. Still, I can tweak my encoded query just a bit to ensure that these columns are searched as well. I just need to switch from this:

internal_typeCONTAINSscript^active=true^name!=sn_templated_snip_note_template

… to this:

active=true^name!=sn_templated_snip_note_template^internal_typeCONTAINSscript^ORinternal_type=condition_string

Another thing that crossed my mind while I was searching for something the other day was that, in addition to scripts, I would really like to have this same capability for searching HTML. At the time, it seemed like it wouldn’t be too difficult to just clone all of the parts and convert them to a second set that was dedicated to HTML instead of Javascript. When I took a look at doing that, though, I realized that with just little bit of extra code, I could make the parts that I had work for both, and not have to create an entirely new second set.

The first thing that I tackled was my Script Include, ScriptUtils, To start out, I renamed it to SearchUtils, since it was now going to handle both searching all of the script columns and searching all of the HTML columns. Then I added a second argument to the function call called type, so that I could differentiate between script search requests and HTML search requests. The only real difference in the code between searching for script and searching for HTML is which columns on which tables will be queried, which means that the only real difference between the two operations is the query used to find the columns and tables. In the current version, that query was hard-coded, so I switched that to a variable and then set the value of the variable to the new version of my encoded query for scripts, and then overrode that value with a new query for HTML if the incoming type argument was set to ‘html’.

var SearchUtils = Class.create();
SearchUtils.prototype = {
	initialize: function() {
	},

	findInScript: function(string, type) {
		var table = {};
		var found = [];
		var encodedQuery = 'active=true^name!=sn_templated_snip_note_template^internal_typeCONTAINSscript^ORinternal_type=condition_string';
		if (type == 'html') {
			encodedQuery = 'active=true^name!=sn_templated_snip_note_template^internal_typeCONTAINShtml';
		}
		var columnGR = new GlideRecord('sys_dictionary');
		columnGR.addEncodedQuery(encodedQuery);
...

Beyond that point, everything else remains the same as it was in the earlier version. That takes care of the Script Include. Now we have to tackle the associated Widget. Here we can use the same basic philosophy, defaulting everything to a script search, and then overriding that default if an HTML search is requested. For a Widget, that request can come in the form of an additional URL parameter that we can also call type. We can establish the default values based on a script search, and then if the URL parameter type is equal to ‘html’, we can then change to the values for an HTML search.

(function() {
	data.type = 'script';
	data.title = 'Script Search';
	if ($sp.getParameter('type') && $sp.getParameter('type') == 'html') {
		data.type = 'html';
		data.title = 'HTML Search';
	}
	data.loading = false;
	data.searchFor = '';
	data.result = false;
	if ($sp.getParameter('search')) {
		data.loading = true;
		data.searchFor = $sp.getParameter('search');
		data.result = new SearchUtils().findInScript(data.searchFor, data.type);
		data.loading = false;
	}
})();

That was all there was to adding an HTML version of the script searching capability. To get to it, I pulled my Search Script menu item up in edit mode by clicking on the little pencil to the right of the title, made a few minor modifications and then selected Insert from the context menu to create a new menu item.

Cloning the Search Script menu item to create the Search HTML menu item

With the new menu item added to the sidebar, the only thing left to do was to click on the item and give it a go.

New HTML Search function in action

Well, that didn’t turn out too bad. I should have changed the name of the button to something that didn’t have the word script in it, but other than that one little nit, it all seems to work. I’ll fix up that button one day, but for now, here is an Update Set that includes all of the parts for the current version.

More Fun with Form Fields

“Be not simply good, be good for something.”
Henry David Thoreau

One of the things that I love about incrementally building parts is that you can obtain value from the version that you have right now, yet still leave open the possibility of adding even more value in the future. When I first set out to attempt to construct a universal form field tag, I had no idea of the ways in which that would grow, but I was able to make use of it as it was at every step along the way. The other day I had a need for field that was only required if another field contained a certain value. When I went to set that up using my latest iteration of the form field tag, I realized that the current version of the code does not support that. That’s not really a problem, though; that’s just another opportunity to create a better version!

We already have an snh-required attribute, but in the current version, it simply adds an HTML required attribute to the input element. It would seem simple enough to just replace that with an ng-required attribute instead, and we would be good to go. However, we also have the required indicator to think about — that grey/red asterisk in front of the field label. That needs to go away when something changes and the field is no longer required. But let’s keep things simple and just focus on one thing at a time. In the current version, we use an internal boolean variable called required to control what gets included in the template that we are building. We can continue to use that, keeping it boolean for false, and then making it a string for anything else. The code to do that looks like this:

var required = false;
if (attrs.snhRequired && attrs.snhRequired.toLowerCase() != 'false') {
	if (attrs.snhRequired.toLowerCase() == 'true') {
		required = 'true';
	} else {
		required = attrs.snhRequired;
	}
}

You may wonder at this point why we make it a boolean for false and a string for true, but hopefully that will reveal itself when we look at the rest of the code. The next thing that we can look at is a little snippet of code that repeats itself a few times throughout the script as it is used when building the input element for a number of different field types:

... + (required?' ng-required="' + required + '"':'') + ...

This is a conditional based on the required variable, which will resolve to true for any non-empty string, but false for the boolean value of false. If the value of required is not false, then we use that same variable, which we now know is a string, to complete the ng-required attribute value for the input element. This will work for values of ‘true’ just as easily as for values that contain some kind of complex conditional statement. This was the easy part, and all of the logic was resolved within the code that generates the template.

The required indicator is another story entirely. Since some condition can toggle the element from required to not required, that same condition needs to apply to the required indicator as well. If the value of the snh-required attribute is anything other than simply ‘true’ or false, we will have to incorporate that logic in the indicator element to determine whether or not to show the indicator image. That code now looks like this:

htmlText += "          <span id=\"status." + name + "\"";
if (required) {
	if (required == 'true') {
		htmlText += " ng-class=\"" + model + refExtra + ">''?'snh-required-filled':'snh-required'\"";
	} else {
		htmlText += " ng-class=\"(" + required + ")?(" + model + refExtra + ">''?'snh-required-filled':'snh-required'):''\"";
	}
}
htmlText += "></span>\n";

As before, the first conditional is based on the required variable, and if it is false, then we don’t do anything at all. But in this case, we also have to check to see if it is equal to the string ‘true’, because if it is, we can just do what we were doing before to make this work. If it is not, then we have to include the required condition in the rendered ng-class attribute to toggle the indicator off and on at the same time that the field requirement is being toggled off and on. When the field is not required, the indicator should just go away, and when it is required, it should be there, and it should be red if the field is empty and grey if it has a value.

That’s it for the code changes. To test it, I brought up my old friend the, form field testing widget, and then altered my textarea example to look like this:

<snh-form-field
  snh-model="c.data.textarea"
  snh-name="textarea"
  snh-label="Text Area"
  snh-type="textarea"
  snh-required="c.data.select==2"
  maxlength="20"
  snh-help="This is where the field-level help is displayed. This field should be required if Option #2 is selected above."/>

That should make the textarea field required when Option #2 is selected on the field that is now above:

The textarea is required when Option #2 is selected

… and it should not be required when anything else is chosen:

The textarea is not required when Option #2 is not selected

With everything looking pretty solid, I was ready to generate another Update Set and call it good. Instead, I thought maybe I would just try a few more things, just to be sure. That’s when I discovered the problem.

It’s always something. I tinkered with the last name field to make it only required if the first name value was greater than spaces, and suddenly all of the form fields that followed within that same DIV disappeared.

Missing form fields

That’s not right. Not only is it not right, I have no idea why that is happening. If you put a greater than (‘>’) sign in the value of the snh-required attribute, anything that follows in the same DIV evaporates. I tried a number of things to fix it, and I found quite a few ways to work around it, but I was never able to actually solve the problem. I hate releasing something that has this kind of a bug in it, but since I don’t seem to posses the mental capacity required to remove the flaw at this particular moment in time, that’s what I am going to end up doing. There are work-arounds, though, so I don’t feel that bad about it. Here are some of the ones that worked for me:

  • Enclose the snh-form-field tag with a DIV. Since the problem only wipes out things within the same DIV, if it is the only thing in the DIV, the problem goes away. I actually tried to do that within the template itself, but that doesn’t work; the DIV has to be outside of the tag, not part of the code that is generated by the tag.
  • Encode the greater than sign. Actually, you have to double encode it, as &gt; will not work, but &amp;gt; does the trick. Not my idea of an intuitive solution, but it does work. And again, I tried to do that within the template itself, but that does nothing at all.
  • Don’t use a greater than sign. In my own example, I could have used snh-required=”c.data.firstName” and that would have worked just as well as snh-required=”c.data.firstName>””. Also, you can call a function that contains the greater than condition, which keeps it out of the attribute value as well.

Again, these are just work-arounds. In my mind, you shouldn’t have to do that. Hopefully, in some future version, you won’t. But if you want to play around with it the way that it is, here is the latest Update Set.

Update: There is a better (enhanced) version here (… but it still doesn’t address the > issue.)

Formatted Script Search Results, Corrected

“Never give in, never give in, never, never, never, never.”
Winston Churchill

For the most part, I really liked the way my formatted search result widget came out; however, the fact that I lost my results after following the links on the page was something that I just couldn’t accept. In my rush to get that out, I released a version with that obvious flaw, but I couldn’t just let that be the end of it; there had to be a way to make it remember its way back home without losing track of what was there before. There just had to be.

I tried a lot of things. They say that when you lose something, you always find it in the last place that you look. I always wondered why that was considered such profound wisdom, since it seemed quite obvious to me that once you found it, you would stop looking. Of course you always find it in the last place that you looked. When I finally came across a way to get the results that I wanted, I stopped looking. There may be a better way to achieve the same ends, but once I was able to link out to a page and come back to my original results, I was done. I’m not all that proud of the code, but it works, so I’m good.

The problem turned out to be this statement, which takes you to the next URL, which includes the search string parameter:

$location.search('search', c.data.searchFor);

This uses the AngularJS $location service, which is quite powerful, but apparently not quite powerful enough to leave a trail that you could follow back home again. I tried a number of variations to get things to work, but in the end I abandoned this whole approach and just went full old school. I replaced my ng-click with an onclick, and in my onclick function, I replaced that nice, simple one line of code with several others:

function findIt() {
	var url = window.location.pathname + window.location.search;
	if (url.indexOf('&search=') != -1) {
		url = url.substring(0, url.indexOf('&search='));
	}
	url += '&search=' + document.getElementById('searchFor').value;
	window.location.href = url;
}

Now I will be the first one to admit that this is quite a bit more code than that one simple line that I had before. First, I had to glue together the path and the search string to construct a relative URL, then I had to check to see if a search parameter was already present, and if so, clip it off of the end, then add my new search parameter to the search string, and then finally, set the current location to the newly constructed URL. Aesthetically, I prefer the original much, much better, but this older, brute force method has the advantage of actually working the way that I want, so it gets to be the winner.

I still kept my ng-click, but that was just to toggle on a new loading DIV to let the user know that their input was accepted and now we are working on getting them their results. That simple HTML addition turned out like this:

<div class="row" ng-show="c.data.loading">
  <div class="col-sm-12">
    <h4>
      <i class="fa fa-spinner fa-spin"></i>
      ${Wait for it ...}
    </h4>
  </div>
</div>

One other thing that I tinkered with in this iteration was the encoded query string in the Script Include. There is a particular table (sn_templated_snip_note_template) that kept throwing an error message related to security, so I decided to just filter that one out by name to keep that from happening. The new encoded query string now looks like this:

internal_typeCONTAINSscript^active=true^name!=sn_templated_snip_note_template

There might be a few other clean-up odds and ends included that I can’t quite recall right at the moment, but the major change was to finally get it to come back home again after drilling down into one the listed scripts. If you installed the previous version of this Update Set, I would definitely recommend that you replace it with this one.

Formatted Script Search Results

“Everything should be made as simple as possible, but not simpler.”
Albert Einstein

So, I came up with a way to search all of the places in which a chunk of code might be hiding, but to get the results, I have to run it as a background script and parse through the resulting JSON object. I need something a little more user-friendly than that, so I am going to build a Service Portal widget that takes a user entered search string, makes the call to the search script, and then formats the results a little nicer. Then I am going to add an item to the Tools menu that I created for my sn-record-picker tool that will bring up this new widget. This way, all I will have to do is click on the menu, enter my search terms, and hit the button to see the results.

The first thing that we will need to do is to lay out the page. Nothing exciting here: just an snh-form-field text element for capturing the input, a submit button, and a place to display the results, all wrapped up in an snh-panel. It’s a relatively small snippet of HTML, so I can reproduce the entire thing here:

<snh-panel class="panel panel-primary" rect="rect" title="'${Script Search}'">
  <form id="form1" name="form1" novalidate>
    <div class="row">
      <div class="col-sm-12">
        <snh-form-field
          snh-model="c.data.searchFor"
          snh-name="searchFor"
          snh-label="What are you searching for?"
          snh-help="Enter the text that you would like to find in a script somewhere in the system"
          snh-required="true"
          snh-messages='{"required":"Please enter what you would like to find"}'/>
      </div>
    </div>
    <div class="row">
      <div class="col-sm-12" style="text-align: center;">
        <button class="btn btn-primary" ng-disabled="!(form1.$valid)" ng-click="findIt()">${Search all scripts}</button>
      </div>
    </div>
  </form>
  <div class="row" ng-show="c.data.result">
    <div class="col-sm-12" ng-show="c.data.result.length==0">
      <p>${No scripts were found to contain the text} <b>{{c.data.searchFor}}</b></p>
    </div>
    <div class="col-sm-12" ng-show="c.data.result.length>0">
      <table class="table table-hover table-condensed">
        <thead>
          <tr>
            <th style="text-align: center;">${Artifact}</th>
            <th style="text-align: center;">${Table Name}</th>
            <th style="text-align: center;">${Table}</th>
          </tr>
        </thead>
        <tbody>
          <tr ng-repeat="item in c.data.result">
            <td data-th="Record"><a href="{{item.table}}.do?sys_id={{item.sys_id}}" title="Open {{item.name}}">{{item.name}}</a></td>
            <td data-th="Table Name">{{item.tableName}}</td>
            <td data-th="Table">{{item.table}}</td>
          </tr>
        </tbody>
      </table>
    </div>
  </div>
</snh-panel>

There is an ng-click on the submit button that will call a function on the client-side script, so now is as good a time as any to build that out. Again, there really isn’t all that much code to see here:

function ScriptSearch($scope, $location) {
	var c = this;

	$scope.findIt = function() {
		$location.search('search', c.data.searchFor);
	};
}

You may notice that we don’t actually search for the script at this point; we just branch to a new location. This is a little trick that I stole from my Configurable Data Table Widget Content Selector, which also just takes user input and uses it to build a new URL, and then takes you to that URL where the search actually takes place. The reason for that is so that all of the user input is a part of the page URL, which means that if you drill down and follow any links on the page, when you come back, all of your original search criteria is still intact and the page comes back just the way that you left it. This works really slick in the Service Portal; it’s just really too bad that I didn’t have the same experience when I launched this widget inside of the main UI. But, we’ll deal with that later.

Right now, it’s time to tackle server-side script, and once again, you aren’t going to see much here in the way of actual lines of code:

(function() {
	data.searchFor = '';
	data.result = false;
	if ($sp.getParameter('search')) {
		data.searchFor = $sp.getParameter('search');
		data.result = new ScriptUtils().findInScript(data.searchFor);
		gs.setReturnURL('/$sp.do?id=script_search&search=' + data.searchFor);
	}
})();

Basically, we start out by setting the default values for our two data properties, and then if there is a search parameter on the URL, we override those values with the search term and the results of running that search term through our script searcher. That last line was my attempt to preserve the URL so that when I clicked on an item to go look at it, my results would still be displayed when I returned. Unfortunately, that didn’t work, either. One day, I will figure out how to fix that, but for now, each time I leave the page, I have to search all over again to get my results back. Not the way that I want it to work, but at this point, I can live with it.

That’s about it for all of the various parts. Pretty simple stuff, really. Here’s what it looks like in action:

Script searcher in action

All in all, I like the way that it turned out, although it really annoys my sense of The Way Things Ought To Be when I click on one of those items and then come back and find that my search results are all gone. That’s just not right. One day I am going to fix that, but until that day comes, here is an Update Set for anyone who wants to play along at home, and maybe even take care of that little annoyance for me!

I know it’s in here somewhere …

“Three can keep a secret, if two of them are dead.”
Benjamin Franklin

Every once in a while I am chasing some issue or tying to remember how I did something and I know the answer is in the system somewhere, but I’m just not sure where. Since I mainly deal with Javascript code, what I would really like is a way to search all of the places in which script might be stored looking for some term or phrase. And I mean all of the places, including places that might come up later in future versions. What I really want to do is dynamically look at all of the tables and find all of the columns that might store script and then search all of those columns in all of those tables. Fortunately, because of the way in which the Now Platform is constructed, you can easily do exactly that.

The sys_dictionary table holds all of the information on all of the columns in all of the tables, including the column type. There are several ServiceNow column types that might contain script, but fortunately for us, they all contain the word script in their names. That makes it relatively easy to search the dictionary for all of the script columns in all of the tables:

var table = {};
var columnGR = new GlideRecord('sys_dictionary');
columnGR.addEncodedQuery('internal_typeCONTAINSscript^active=true');
columnGR.query();
while (columnGR.next()) {
	var tableName = columnGR.getDisplayValue('name');
	var fieldName = columnGR.getDisplayValue('element');
	if (tableName && fieldName && !tableName.startsWith('var__m_')) {
		if (!table[tableName]) {
			table[tableName] = [];
		}
		table[tableName].push(fieldName);
	}
}

This script builds a map keyed by table name that contains an array of script columns for each table. Before I add a column to the map, I make sure that there is both a field name and a table name, and I also filter out all of the variable mapping tables, as those don’t contain any scripts and some of them actually cause problems when I attempt to use them. Once we have established our target map, it is simply a matter of stepping through it and querying each table for the presence of your search argument in any of those columns:

var found = [];
for (tableName in table) {
	var query = '';
	var separator = '';
	for (var i=0; i<table[tableName].length; i++) {
		query += separator;
		query += table[tableName][i];
		query += 'CONTAINS';
		query += string;
		separator = 'OR';
	}
	var scriptGR = new GlideRecord(tableName);
	scriptGR.addEncodedQuery(query);
	scriptGR.query();
	while (scriptGR.next()) {
		found.push({table: tableName,  tableName: scriptGR.getLabel(), sys_id: scriptGR.getUniqueValue(), name: scriptGR.getDisplayValue('name') || scriptGR.getDisplayValue() || scriptGR.getUniqueValue()});
	}
}
return found;

This portion of the script builds an array of objects containing the table, table name, name, and sys_id of any records found during the search. I bundled the whole thing into a Script Include that I called ScriptUtils, and you can see the entire thing here:

var ScriptUtils = Class.create();
ScriptUtils.prototype = {
	initialize: function() {
	},

	findInScript: function(string) {
		var table = {};
		var found = [];
		var columnGR = new GlideRecord('sys_dictionary');
		columnGR.addEncodedQuery('internal_typeCONTAINSscript^active=true');
		columnGR.query();
		while (columnGR.next()) {
			var tableName = columnGR.getDisplayValue('name');
			var fieldName = columnGR.getDisplayValue('element');
			if (tableName && fieldName && !tableName.startsWith('var__m_')) {
				if (!table[tableName]) {
					table[tableName] = [];
				}
				table[tableName].push(fieldName);
			}
		}
		for (tableName in table) {
			var query = '';
			var separator = '';
			for (var i=0; i<table[tableName].length; i++) {
				query += separator;
				query += table[tableName][i];
				query += 'CONTAINS';
				query += string;
				separator = 'OR';
			}
			var scriptGR = new GlideRecord(tableName);
			scriptGR.addEncodedQuery(query);
			scriptGR.query();
			while (scriptGR.next()) {
				found.push({table: tableName,  tableName: scriptGR.getLabel(), sys_id: scriptGR.getUniqueValue(), name: scriptGR.getDisplayValue('name') || scriptGR.getDisplayValue() || scriptGR.getUniqueValue()});
			}
		}
		return found;
	},

    type: 'ScriptUtils'
};

To test is out, we can can select Scripts – Background from the sidebar menu and enter something like this:

gs.info(JSON.stringify(new ScriptUtils().findInScript('xxx'), null, 4));

After running the above script, I received the following output:

[
    {
        "table": "discovery_proc_handler",
        "tableName": "Process Handler",
        "sys_id": "1f2473269733200010cb1bd74b297576",
        "name": "Java parameters"
    },
    {
        "table": "sys_script",
        "tableName": "Business Rule",
        "sys_id": "4532f571bf320100710071a7bf073929",
        "name": "Obfuscate password"
    },
    {
        "table": "sys_script_execution_history",
        "tableName": "Script Execution History",
        "sys_id": "807208e92ffb48d0ddadfe7cf699b696",
        "name": "Created 2020-03-26 05:55:27"
    },
    {
        "table": "sys_script_include",
        "tableName": "Script Include",
        "sys_id": "326b53699f3010008f88ed93ee4bcc2b",
        "name": "ScrumSecurityManagerDefault"
    },
    {
        "table": "sys_script_include",
        "tableName": "Script Include",
        "sys_id": "3f7d6f17537103003248cfa018dc347c",
        "name": "PwdResetPageInfo"
    },
    {
        "table": "sys_script_include",
        "tableName": "Script Include",
        "sys_id": "57042a36932012001aa1f4b8b67ffb95",
        "name": "TourBuilderUtility"
    },
    {
        "table": "sys_script_include",
        "tableName": "Script Include",
        "sys_id": "88c548dc37010100dcd48c00dfbe5d2e",
        "name": "SnmpIdentityInfoParser"
    },
    {
        "table": "sys_script_include",
        "tableName": "Script Include",
        "sys_id": "8dbee16b530203003248cfa018dc349e",
        "name": "PwdResetPageInfo_V2"
    },
    {
        "table": "sys_script_include",
        "tableName": "Script Include",
        "sys_id": "91a92c70733023008b516cb63cf6a79e",
        "name": "CommunityCacheUtilSNCJSC"
    },
    {
        "table": "sys_script_include",
        "tableName": "Script Include",
        "sys_id": "a7ac57b7c710320003fa9c569b976312",
        "name": "MIDUserConnectivity"
    },
    {
        "table": "sys_script_include",
        "tableName": "Script Include",
        "sys_id": "c54c989f37612100dcd48c00dfbe5df4",
        "name": "CiSchema"
    },
    {
        "table": "sys_script_include",
        "tableName": "Script Include",
        "sys_id": "ee735ba66713220089ec9a6617415a75",
        "name": "CommunityForumImpl"
    },
    {
        "table": "sys_script_include",
        "tableName": "Script Include",
        "sys_id": "eee51271eb223100c46ac2eef106fed4",
        "name": "AddScriptedRESTVersionAjax"
    }
]

Now, there are obviously better ways to format and display that information, but that would be an entire project on its own, so we’ll just save that exercise for another time.