Web Analytics Taken to the Next Level

I came up with a neat idea the other night. Using localStorage and sessionStorage you could theoretically monitor the number of tabs or windows a visitor has opened for your site. As far as I know, this capability has never before been possible. Well, now it is.

Check out this crude example.

Once you open the monitor, leave that tab/window put. Use the “Spawn Tab” link to create new tabs and windows. The monitor will be notified and display some simple debug. There are 10 second updates per tab/window so that when they close the monitor can detect it. The monitor will detect a close within 15 seconds of the tab/window closing and will display the total time the tab/window plus or minus 10 seconds. Correct values are maintained as the tab/windows browse across pages as long as they stay on the domain! Just about everything you’d want or need.

Again, I mentioned this is rather crude. The fact that the monitor tab remains open is only due to the fact that I wanted to prototype the idea. The majority of the state is stored in localStorage, and each tab/window maintains a single identifier in its sessionStorage to remind the tab/window what id it was while it navigates to multiple pages. Because everything is stored in the storage this system has the capability to become completely distributed. Meaning no “monitor” tab is necessary, and the scripts can determine, and monitor, on their own the existence of all other tabs. Thus, this would be a viable option for the next level of web analytics.

As cool as this is, I don’t think it will provide too much value to the analytics. For the first time webmasters will be able to know how many windows or tabs a visitor opens (and to what pages they open). The webmaster will know more about how its user’s use the website, but I don’t think this statistic will be a game changer. Who knows!

So, how does it work? Very simple. Each tab includes the client.js code to handle updating the localStorage and maintaining its own “tab_id” in sessionStorage. Data it maintains can be whatever you want, I went with some simple information such as its start time, current url, and latest keepalive:

localStorage values

The upkeep for a Tab Client is to restore their session information when you navigate to any new page:

// Create or Restore tab_id
var myTabId = sessionStorage.tab;
if ( myTabId === undefined ) {
  var tabs = localStorage.tabs;
  if ( tabs === undefined ) {
    myTabId = 0;
    localStorage.tabs = "0";
  } else {
    var largest = parseInt( tabs.split(/,/).pop(), 10 );
    myTabId = largest+1;
    localStorage.tabs += "," + myTabId;
  }
  sessionStorage.tab = myTabId;
}

And to perform its keepalives:

// Update the Latest Timestamp
function setLatest() {
  var key = 'tab'+myTabId+'_latest';
  localStorage.setItem(key, +(new Date())); 
}

// Update Status every 10 seconds
window.setInterval(setLatest,10000);
setLatest();

I put a few more convenience functions in there to help it update these localStorage keys, and communicate with the monitor which was crudely done through localStorage. That is explained next.

The Tab Monitor as it stands right now receives messages through localStorage’s “storage” event. It also checks all the tab’s “lastest” keepalives to make sure they didn’t pass their 10 second limit. In the case of a tab being closed, it will remove references to that tab and output an approximation of the time the tab was open:

// Listener - receive messages from tabs
window.addEventListener('storage', function(e) {
  if ( e.storageArea === localStorage ) {
    if ( e.key == "tab_msg" ) {
      console.log( e.newValue );
      addMsg( e.newValue ); // Appends to the page
    }
  }
});

// Purger - clean out tabs that died for 15 seconds
window.setInterval(function() {
  console.log("purging");
  var now = +(new Date()),
      tabs = localStorage.tabs;
  if ( tabs !== undefined ) {
    var toRemove = [], toKeep = [];
    tabs = tabs.split(/,/);
    for (var i=0, len=tabs.length; i<len; i++) {
      var tabId = tabs[i],
          tabLatest = parseInt( localStorage.getItem("tab"+tabId+"_latest") );
      ( (now-tabLatest)>=15000 ? toRemove : toKeep ).push(tabId);
    }
    localStorage.tabs = toKeep.join(',');
    for (var i=0, len=toRemove.length; i<len; i++) {
      var tabId = toRemove[i],
          tabLatest = parseInt( localStorage.getItem("tab"+tabId+"_latest") ),
          tabStart = parseInt( localStorage.getItem("tab"+tabId+"_start") ),
          time = (tabLatest-tabStart)/1000;
      addMsg( "Tab " + tabId + " Closed after " + time + " seconds!" );
    }
  }
}, 5000);

This took a little under an hour to get working. There are still minor issues that I didn’t attempt to resolve. However, if there is interest this could be developed into a completely distributed peer-to-peer communication between tabs/windows on a single domain. However, a little warning. Web Storage is not set in stone. Not all browsers have implemented it and the specification is subject to change at any minute. There has been some rather heated debate on the subject of Web Storage recently, with good reason. All I know is that when its settled, this functionality will continue to exist!

Let me know what you think.

“Back and Forth” Greasemonkey For The Whole Web

Recently I wrote a Greasemonkey script to add keyboard shortcuts to The Big Picture, to improve on some of their already existing shortcuts. Once I started using some of the shortcuts I made I ended up wanting to use them all over the place at other blogs. This functionality is so tiny, but so useful, that I bundled it into its own script that runs on all web pages!

Grab it here:

//
// ==UserScript==
// @name          Back and Forth
// @namespace     http://blog.bogojoker.com
// @description   Keyboard Shortcut to Jump back and forth on a page. (esc key).
// @include       *
// @version       1.0 - Initial Version - Sunday February 15, 2009
// ==/UserScript==

(function() {
  
  // Global States
  var x = null;
  var y = null;

  // Add a new Global Key Listener for `esc`
  document.addEventListener("keypress", function(e) {
    if(!e) e=window.event;
    var key = e.keyCode ? e.keyCode : e.which;
    if ( key == 27 ) {
      var tempx = x;
      var tempy = y;
      x = Math.max(document.documentElement.scrollLeft, document.body.scrollLeft);
      y = Math.max(document.documentElement.scrollTop, document.body.scrollTop);
      if ( tempx != null ) { // First time it should be null
        window.scrollTo(tempx, tempy);
      }
    }
  }, true);

})();

On any webpage the first time you push the `esc` key position A gets stored. The next time you push `esc` position B gets stored and the browser jumps to position A. The next time you push it, A gets stored and you jump to B. So you always jump back to wherever you pushed `esc` last. Hence the name “back and forth.”

This is useful to me when I jump between comments and the content. When I’m reading a comment and I want to check back to the article, I just just push `esc` to save my position, go back to the article, and when I’m all set I just jump back to my saved position (the comments) with `esc`.

Short, Sweet, Simple: The Back and Forth Greasemonkey Script.

More Big Picture Keyboard Functionality!

I’ve mentioned before how I’m a big fan of The Big Picture blog. One of the things that makes it so great is that it has keyboard navigation! You can use ‘j’ and ‘k’ to automatically jump between pictures. Its so much nicer then scrolling because it jumps to the exact height to maximize the picture in the browser. Huge usability improvement!

Like before, the problem I had was that users were mentioning pictures in their comments. Jumping back to that picture was hard or annoying. So, I wrote a Greasemonkey script that allows you to type in a number and it will automatically jump to that picture! Click here to get the script!

Oh, and if you’re reading comments and you want to jump back and forth between images and comments that works too. Once you’ve jumped to the image, just hit ‘esc’ and you will be taken back to where you were before. Too cool!

//
// ==UserScript==
// @name          The Big Picture Keyboard Enhancements
// @namespace     http://blog.bogojoker.com
// @description   Keyboard Shortcut Enhancements
// @include       http://www.boston.com/bigpicture/*
// @version       1.1 - Added Back+Forth - Thursday February 12, 2009
//                1.0 - Initial Version - Monday February 9, 2009
//
//
//   This allows the user to type in numbers, and after about
//   a half second it will jump directly to that image.
//   For example:
//
//     Push '1'... '2'... User is taken directly to Image "12"
//     Push '9'... '9'... User is taken to the last picture.
//
//   Use 'esc' to jump back and forth between two positions.
//   For example if a comment mentions picture 4:
//
//     Push '4'...        User is taken directory to Image "4"
//     Push 'esc'         User is taken back to the comment!
//     Push 'esc'         User is taken back to Image "4"
//
// ==/UserScript==

(function() {
  
  // GreaseMonkey (Firefox - unsafeWindow) and GreaseKit (Safari - window)
  var w = ( /a/[-1]=='a') ? unsafeWindow : window;

  // Global States
  var x, y;
  var keypressnumber  = false;
  var builtupnumber   = '';
  var quicknumtimeout = null;
  var imgArr = document.getElementsByClassName("bpImage");
  
  // Keep the old and create a New Global Keypress listener on top of it
  document.addEventListener("keypress", function(e) {
    

    // Get the key
    if(!e) e=window.event;
    var key = e.keyCode ? e.keyCode : e.which;
    
    // Store the current x/y position
    function storePos() { 
      x = Math.max(document.documentElement.scrollLeft, document.body.scrollLeft);
      y = Math.max(document.documentElement.scrollTop, document.body.scrollTop);
    }
    
    // # Character => Jump to that image, Store Position
    if ( key >= 48 && key <= 57 ) {
      if ( e.target.nodeName.match(/TEXTAREA|INPUT/) ) return;
      clearTimeout(quicknumtimeout);
      keypressnumber = true;
      builtupnumber += (key - 0x30);
      quicknumtimeout = setTimeout(function() {
        w['currImg'] = parseInt(builtupnumber,10)-1;
        if (w['currImg'] >= imgArr.length) { w['currImg'] = imgArr.length-1; }
        storePos();       
        window.scrollTo(0,imgArr[ w['currImg'] ].offsetTop+174);
        keypressnumber = false;
        builtupnumber = '';
        quicknumtimeout = null;
      }, 300);
    }
    
    // Esc => Jump back to a Saved Position
    if ( key == 27 ) {
      var tempx = x; var tempy = y; storePos();
      window.scrollTo(tempx, tempy);
    }   

  }, true);

})();

All I do is register a new global keyboard listener to catch numeric keys and act accordingly. I tested thoroughly on Firefox and Safari to make sure it works correctly in all cases. It jumps to the correct image, it maintains the “current image” so j/k will still work, it won’t jump if you’re typing in a textfield/input, etc. It even properly handles situations that the current j/k functionality doesn’t. For instance mine allows the user to type in the comment box, click outside the box and reuse the keyboard shortcuts. When you type a j/k or even click inside the search box at the top the j/k functionality is gone. I didn’t feel like correcting that in this Greasemonkey script in case it gets fixed by the developers behind the Big Picture. (Note to those developers: reset isLoaded back to true or take a different approach.)

I’m open to New Ideas. I have some myself but I have to focus on schoolwork in these next few weeks. Let me know if you want anything.

Big Picture Keyboard Commands Greasemonkey Script!

Enjoy.

JavaScript Sort an Array of Objects

I ran into an interesting problem today. I had an array of objects that I wanted sorted on a certain property. My obvious thought didn’t work! (Update: I got a comment below from Peter Michaux who points out a nicer solution, it is included here:)

// Array of Objects
var obj_arr = [ { age: 21, name: "Larry" },
                { age: 34, name: "Curly" },
                { age: 10, name: "Moe" } ];

// This doesn't work!
obj_arr.sort( function(a,b) { return a.name < b.name; });

// This does work! (Peter's update, very fast)
obj_arr1.sort(function(a,b) { return a.name < b.name ? -1 :
                                     a.name > b.name ?  1 : 0; });

That kind of frustrated me. Sorting is one of those things I expect to be available in all languages. I don’t want to have to write a sorting algorithm every time I need to sort. So I looked into things, pulled up a Javascript Quicksort Algorithm and manipulated it to support any compare function.

Now that I have the freedom to truly write a compare function that works for objects! I also changed around certain parts of the code I found online to actually extend the Array class and make the extra functions hidden. Take a look at the sample usage:

// Defaults to (a<=b) sorting.  Great for numbers.
var arr = [1234, 2346, 21234, 3456, 32134, 3456, 1234, 2345, 23, 42523, 1234, 345];

// Object Array
var obj_arr = [ { age: 21, name: "Larry" },
                { age: 34, name: "Curly" },
                { age: 10, name: "Moe" } ];

arr.quick_sort();
// => [23, 345, 1234, 1234, 1234, 2345, 2346, 3456, 3456, 21234, 32134, 42523]

obj_arr.quick_sort(function(a,b) { return a.name < b.name });
// => Curly, Larry, Moe

obj_arr.quick_sort(function(a,b) { return a.age < b.age });
// => Moe (10), Larry (21), Curly (34)

For those who want to see the code be glad, its free. I carried the copyright with it but its rather loose. Grab the JavaScript Source Here! Enjoy:

Array.prototype.swap=function(a, b) {
  var tmp=this[a];
  this[a]=this[b];
  this[b]=tmp;
}

Array.prototype.quick_sort = function(compareFunction) {

  function partition(array, compareFunction, begin, end, pivot) {
    var piv = array[pivot];
    array.swap(pivot, end-1);
    var store = begin;
    for (var ix = begin; ix < end-1; ++ix) {
      if ( compareFunction(array[ix], piv) ) {
        array.swap(store, ix);
        ++store;
      }
    }
    array.swap(end-1, store);
    return store;
  }

  function qsort(array, compareFunction, begin, end) {
    if ( end-1 > begin ) {
      var pivot = begin + Math.floor(Math.random() * (end-begin));
      pivot = partition(array, compareFunction, begin, end, pivot);
      qsort(array, compareFunction, begin, pivot);
      qsort(array, compareFunction, pivot+1, end);
    }
  }

  if ( compareFunction == null ) {
    compareFunction = function(a,b) { return a<=b; };
  }
  qsort(this, compareFunction, 0, this.length);

}

Update

Peter Michaux pointed out something very important. The sort() function can be made to work if it returns numeric output (-1,0,1). His approach is far superior. Here was a benchmark I took:

var obj_arr1 = [];
var obj_arr2 = [];
var filler = [ { age: 21, name: "Larry" },
               { age: 34, name: "Curly" },
               { age: 10, name: "Moe" } ];
for (var i=0; i<5000; i++) {
  rand = Math.floor( Math.random() * 3 );
  obj_arr1.push( filler[rand] );
  obj_arr2.push( filler[rand] );
}

var s = new Date();
obj_arr1.sort(function(a,b) { return a.name < b.name ? -1 : a.name > b.name ? 1 : 0; });
var e = new Date();
console.log(e.getTime()-s.getTime()); // => 75 ms

s = new Date();
obj_arr2.quick_sort(function(a,b) { return a.name < b.name });
e = new Date();
console.log(e.getTime()-s.getTime()); //  => 4444 ms

That shows drastic differences for arrays as large as 5000 elements (with not too random data). 75 ms versus 4444 ms (over 4 seconds). Doing the math: (4444/75) => 59.253 times better! Moral of the story, don’t rush into thinking something doesn’t exist!

So if that’s the way to do it, then I want to make it easier on me. My arrays are generally going to be under 100 in size, and at such a size building a function dynamically instead of writing a custom function works just about as well (although if you were using objects, polymorphism and a compare function would be the best way to go). Here is a simple function I can use to more quickly build compare functions in order to ascend sort an array on multiple properties!

function buildCompareFunction(arr) {
  if (arr && arr.length > 0) {
    return function(a,b) {
      var asub, bsub, prop;
      for (var i=0; i<arr.length; i++) {
        prop = arr[i];
        asub = a[prop];
        bsub = b[prop];
        if ( asub < bsub )
          return -1;
        if ( asub > bsub )
          return 1;
      }
      return 0;
    }
  } else {
    return function(a,b) { return a<=b; };
  }
}

Sample usage would be:

var obj_arr = [
  { name: 'Joe',   age: 20 },
  { name: 'Joe',   age: 10 },
  { name: 'Joe',   age: 30 },
  { name: 'Joe',   age: 40 },
  { name: 'Joe',   age: 20 },
  { name: 'Joe',   age: 15 },
  { name: 'Joe',   age: 35 },
  { name: 'Joe',   age: 25 },
  { name: 'Bill',  age: 5 },
  { name: 'Barry', age: 20 },
  { name: 'Paul',  age: 20 },
  { name: 'Peter', age: 1 },
  { name: 'Smith', age: 25 },
  { name: 'Kary',  age: 30 }
];

obj_arr.sort( buildCompareFunction(['name','age']) );

Clicky Greasemonkey Menu Script Updated

I noticed that I can’t really live without this script. Clicking those menus is just too difficult. The updates made to Clicky January 19th changed the way they handled their dropdowns. Clicky wrapped them into objects and added a bunch of functionality to make their content updateable via AJAX requests for their new filters. Some spiffy updates but code that I could not use from within Greasemonkey!

So have at it. The user script is available right here.

If you look at the script you will see that I handle opening and closing of the menus much like the Clicky’s object code does it. There is a variable to hold the current open menu if there is one so that opening a different menu closes any other open menu. I only add an onmouseover event and jimmy in my own close code when you click the close menu button. This works around Clicky’s object code without breaking anything that I can see.

Greasemonkey – Clicky Mouseover Menus

I have been a proponent of the Get Clicky web statistics since they first came out. They have been constantly improving them since day one, and I have enjoyed every update. One improvement that I especially liked were two menus in the top right. One that lists each of your websites being monitored by GetClicky so you can quickly jump between them, and one to change the date of the stats you’re currently viewing (if you’re only looking at a single day).

Clicky Menus

These menus currently only activate when you click them. Well, the arrow is there and it so nicely indicates to me that it should be a dropdown. So I wrote up a script to copy to onclick event to the onmouseover. This may be outdated pretty soon but it is one of my first scripts and it gives me a little joy each time I use it.

You’re going to need Firefox with the Greasemonkey extension and of course Site Statistics (available free) at Get Clicky.

So without further ado:
Greasemonkey Script to turn Clicky Menus into Mouseovers

Some recent fun with JavaScript at work and a co-workers really neat Greasemonkey script for Revealing Experts Exchange comments propelled me to write this very simple script. Mine pales in comparison to CoderJoe’s EE script and his knowledge of JavaScript and the DOM are quite vast. I consider this a first step into some individual JavaScript coding.

search