Web Analytics Taken to the Next Level

I came up with a neat idea the other night. Using localStorage and sessionStorage you could theoretically monitor the number of tabs or windows a visitor has opened for your site. As far as I know, this capability has never before been possible. Well, now it is.

Check out this crude example.

Once you open the monitor, leave that tab/window put. Use the “Spawn Tab” link to create new tabs and windows. The monitor will be notified and display some simple debug. There are 10 second updates per tab/window so that when they close the monitor can detect it. The monitor will detect a close within 15 seconds of the tab/window closing and will display the total time the tab/window plus or minus 10 seconds. Correct values are maintained as the tab/windows browse across pages as long as they stay on the domain! Just about everything you’d want or need.

Again, I mentioned this is rather crude. The fact that the monitor tab remains open is only due to the fact that I wanted to prototype the idea. The majority of the state is stored in localStorage, and each tab/window maintains a single identifier in its sessionStorage to remind the tab/window what id it was while it navigates to multiple pages. Because everything is stored in the storage this system has the capability to become completely distributed. Meaning no “monitor” tab is necessary, and the scripts can determine, and monitor, on their own the existence of all other tabs. Thus, this would be a viable option for the next level of web analytics.

As cool as this is, I don’t think it will provide too much value to the analytics. For the first time webmasters will be able to know how many windows or tabs a visitor opens (and to what pages they open). The webmaster will know more about how its user’s use the website, but I don’t think this statistic will be a game changer. Who knows!

So, how does it work? Very simple. Each tab includes the client.js code to handle updating the localStorage and maintaining its own “tab_id” in sessionStorage. Data it maintains can be whatever you want, I went with some simple information such as its start time, current url, and latest keepalive:

localStorage values

The upkeep for a Tab Client is to restore their session information when you navigate to any new page:

// Create or Restore tab_id
var myTabId = sessionStorage.tab;
if ( myTabId === undefined ) {
  var tabs = localStorage.tabs;
  if ( tabs === undefined ) {
    myTabId = 0;
    localStorage.tabs = "0";
  } else {
    var largest = parseInt( tabs.split(/,/).pop(), 10 );
    myTabId = largest+1;
    localStorage.tabs += "," + myTabId;
  }
  sessionStorage.tab = myTabId;
}

And to perform its keepalives:

// Update the Latest Timestamp
function setLatest() {
  var key = 'tab'+myTabId+'_latest';
  localStorage.setItem(key, +(new Date())); 
}

// Update Status every 10 seconds
window.setInterval(setLatest,10000);
setLatest();

I put a few more convenience functions in there to help it update these localStorage keys, and communicate with the monitor which was crudely done through localStorage. That is explained next.

The Tab Monitor as it stands right now receives messages through localStorage’s “storage” event. It also checks all the tab’s “lastest” keepalives to make sure they didn’t pass their 10 second limit. In the case of a tab being closed, it will remove references to that tab and output an approximation of the time the tab was open:

// Listener - receive messages from tabs
window.addEventListener('storage', function(e) {
  if ( e.storageArea === localStorage ) {
    if ( e.key == "tab_msg" ) {
      console.log( e.newValue );
      addMsg( e.newValue ); // Appends to the page
    }
  }
});

// Purger - clean out tabs that died for 15 seconds
window.setInterval(function() {
  console.log("purging");
  var now = +(new Date()),
      tabs = localStorage.tabs;
  if ( tabs !== undefined ) {
    var toRemove = [], toKeep = [];
    tabs = tabs.split(/,/);
    for (var i=0, len=tabs.length; i<len; i++) {
      var tabId = tabs[i],
          tabLatest = parseInt( localStorage.getItem("tab"+tabId+"_latest") );
      ( (now-tabLatest)>=15000 ? toRemove : toKeep ).push(tabId);
    }
    localStorage.tabs = toKeep.join(',');
    for (var i=0, len=toRemove.length; i<len; i++) {
      var tabId = toRemove[i],
          tabLatest = parseInt( localStorage.getItem("tab"+tabId+"_latest") ),
          tabStart = parseInt( localStorage.getItem("tab"+tabId+"_start") ),
          time = (tabLatest-tabStart)/1000;
      addMsg( "Tab " + tabId + " Closed after " + time + " seconds!" );
    }
  }
}, 5000);

This took a little under an hour to get working. There are still minor issues that I didn’t attempt to resolve. However, if there is interest this could be developed into a completely distributed peer-to-peer communication between tabs/windows on a single domain. However, a little warning. Web Storage is not set in stone. Not all browsers have implemented it and the specification is subject to change at any minute. There has been some rather heated debate on the subject of Web Storage recently, with good reason. All I know is that when its settled, this functionality will continue to exist!

Let me know what you think.

Handling the tab key in a <textarea>

Traversing through input elements with the tab key is important for accessibility reasons. However, every once in a while you come across a situation where traversal isn’t really important. Instead, you want the tab key to actually do something for you. Even still, you may want to do something fancy with the tab key. Wether its replacing it with spaces or something else.

I found an interesting website today that had an interesting idea. You could run some test code on the page to test their library. Their instructions said, “push tab to evaluate the code.” Sure enough you could tell it was working “onblur” for the textarea. The problem with this was that when you pushed tab you lost focus.

I thought about it, and figured you could do a rather simple trick to run some code and refocus on the textarea. It goes a little like this:

window.addEventListener('load', function() {
  var textarea = document.getElementById('txt');
  textarea.addEventListener('keydown', function(e) {
    if (e.keyCode === 9) {
      e.preventDefault();
      e.stopPropagation();
      // operation goes here
    }
  });
});

Note that to get the actual character you have to get the character from the event. There are many ways to do it, keyCode, charCode, which, even keyIdentifier. You’ll have to mix things up to work across all browsers. Basically 9 is the code for the tab key. So when you get the tab key, it prevents the default behavior and allows you to execute whatever code you want: run some functions, eval some code, display something, ajax request, whatever you want. Simple. I think it would improve a few interfaces. Neat idea to make use of the tab key to perform a function.

You can check out this example of what I mean.

Too Many Unproductive Comments

It seems as though there is an epidemic across the web. Bad comments. Not spam, not flames, not even rude remarks… just plain drivel. I spent the last few weeks reading all the comments (or as many as I could stand) on the hundreds of blog articles I read every week. They fall into two major categories, which I’ll discuss below.

I’ve linked to this in the past but I’ve really taken it to heart: Your Shit Does Stink — Good Friends Are Hard to Find.

What the gentlemen at Less Everything were saying has just continually amplified in my mind over these last few (dare I say “trying”) weeks. Their article is short, and worth the jump, but to put it in context:

It’s easier to just smile and nod and say, “that’s great,” and that’s what most of us do. But a true friend will tell you to polish it up or go make changes or start again.

The same applies to comments on a blog post. The idea is to inspire discussion, clarify points, etc. eventually enriching the value of the article itself. Instead, in a number of sites I’m seeing “fluff” comments, which just make the entire experience of reading comments a downright drag when I’m actually interested in the content. So, I counted…

I took a well written article with over 100 comments and counted more then 50% of the comments were of this “fluff” nature. Although not directly from the article where I calculated my statistics, I pulled this paragon of an example:

Useless Comment

I made little effort to hide the identity of the user and the website (which I have the highest respect for). The fact of the matter is that this comment is not only worthless, it pollutes the pot of potentially worthy comments and thus detracts from the value of the article itself. In this particular example the commenter actually admits to not reading the article but then claims he knows it will be awesome?!?! Give me a break. Hell, if I was the author of the article I would be upset at such an ignominious comment (yah, I looked that one up).

I’ve gone years on this blog without making a rant. This is my first. So, admittedly, I did not hold back. However, eventually I calmed down and tried to really think about this “problem.”

From the author’s perspective this simple “praise” is uplifting. For those offering the praise its quite simply that… many want to portray honest thanks and support to the author. This is all well and good. However, there are still many reading the article intending to engage in discussion. Undoubtably the author should both encourage and look forward to this kind of discussion; even more so then the praise!! Why? Because its in our nature. We write so that others can read. We enable comments so others can tell us what they think. If we didn’t care for other’s opinions or views then comments could just as easily be disabled.

So, essentially there are two categories of comments, Praise and Discussion. So, I think that this should become a model. The more recent up/down voting scheme is not the model to use for most blog articles. It works well on ranking sites like StackOverflow and Reddit where correctness or opinion influences the votes. However in this case there is:

  • Praise – essentially always an “up-vote.”

  • Discussion – a level playing field likely to contain constructive criticism as well as support for the article.

Both avenues should be available so that the author and all the commenters have the freedom to interact with whichever degree they feel is appropriate. In the end the discussion is separated from the “fluff” and everyone wins.

I’m thinking of the current system of WordPress with comments and trackbacks/pings. They are handled separately, but that is because fundamentally they are different. To make a system like I’ve suggested work would either require user action, moderation, or a (likely) sophisticated action. Two of those don’t scale and the last is probably too complex to be reliable. If I’ve learned anything from StackOverflow its that they have actively crafted and trained their community of users to “do good” and do all of this low level work willingly and it has paid off very well.

For starters “commenters” must at least be given the choice: to contribute to a discussion, or to thank the author for a well written article. I don’t intend on building this system yet, because I personally don’t have the influence or the popularity to make an impact. Yet.

Just keep this in mind the next time you comment. Actually try to “add value.” To rip off the Army… The whole world can read what you’re writing. Is it worth reading?

Cheers?

Visualize the Directory Tree

Ever have to work with a new directory and you have no idea what its structure is? Or maybe you have a few files laying around but you’re not sure which sub-directory they are in? Or maybe you’re showing someone else a project and you want to show them the directory hierarchy. The bare bones solutions of `ls -R` or `find .` are just too archaic and offer no visualization of the structure. To solve this problem, people have built their own “tree” scripts.

There are a few tree scripts available online. Some as simple as find | sed and others are slightly more advanced like a python script. I wasn’t pleased with the existing solutions, so I wrote my own. To get an idea of what I’m talking about take a look at this screenshot showing the listing of a Rails project:

tree

Its a simple, clean listing of the directory tree. I will admit, the style is based off of another tree script that I’ve seen that I liked. Also, this isn’t really production quality code. I take the lazy way out and first get a directory listing and then work from there. This means that for large directories there may be an initial pause before it starts outputting. I wouldn’t suggest running this on your home directory. Although I can think of better algorithms its unlikely that I would want to run this on huge directories so I’m more then happy right now.

The usage is pretty bare bones:

tree usage

This is one more script I’ve added to my ~/bin and its completely open source on GitHub. Its just straight Ruby, no extra packages, works with 1.8 and 1.9. Oh and did I mention its customizable?

I hope you like it.

Shell Brace Expansion

I would say that most developers that I know are comfortable with a shell/terminal but they are not proficient with the shell. I, for one, am constantly striving to improve my knowledge of the shell, customizing my environment, and trying to maximize my productivity with it. Even after years of constant usage I am always learning new tricks and techniques. It also doesn’t help that there are thousands of command line applications out there waiting to be absorbed.

My shell of choice, like most people I know, is bash. Most developers only use the basic features available in shells such as launching programs, command history, auto-completion, and the like. However, most shells have functionality people don’t even know about and could probably benefit from. Who knows, you might even be able to multitask and partake in foxy bingo online or pogo games while programming simultaneously. One such feature is Brace Expansion.

Take a look:

Brace Expansion

Its a little hard to see from the image so I’ll run down the examples here:

shell> echo hello{world,goodbye}
helloworld hellogoodbye

This example shows exactly what Brace Expansion is doing. Its taking the input line “hello{}” and replacing the “{}” section with each of the comma separated values, producing “helloworld” and “hellogoodbye.”

Its important to note that there is no space between “hello” and the opening brace. This is called the preamble and therefore it gets prepended to the front of both “world” and “goodbye.” Also, there is no space around the comma inside the braces. If there were a space then brace expansion doesn’t take place at all! The output itself will separate each expansion with a space, just like any other built-in shell expansion. If you really want spaces in the expansion portions you can escape them as you normally would and it will respect that.

shell> echo file.{txt,rb}
file.txt file.rb

This points out a couple things. First of all, I’m using echo to debug the brace expansion so you can see what it produced. What seemingly happens, like in the previous case, is that it ran the brace expansion when it parsed the input line, and turned it into the following:

shell> echo file.txt file.rb

Now some simple usages pop to mind. Need to rename a file? How about “mv filename.{txt,rb}.” Want to quickly backup a file, how about “cp filename.{txt,save}.” Whenever you feel yourself typing the same thing twice in a row that is a perfect candidate for brace expansion! Save yourself some keystrokes, some time, and potential errors by reusing what you’ve already typed in already!

The third usage is combining two separate but connected brace expansions “{}{}.” This works differently then two completely disjoint sets “{} … {}” and is still different then nested sets “{…{}…}.” Rather then discuss the semantics of those, which you can easily find out yourself by just trying them out, I’ll give you a glimpse of another trick you can do with brace expansion.

Here is a simple example of Sequences:

Brace Expansion Sequences

The syntax for sequences is “{#..#}.” You can even do simple character sequences like “{a..z}.” The second example from the picture shows a postscript, the opposite of the preamble. Its text that comes immediately after the braces (no space) and is thus appended to the end of each expansion.

I hope you give it try, you might end up liking it. I actually do some real neat tricks with it in one of my scripts. If you want to explore it further you should check out some of the more technical documentation:

Until next time, wrap your head around this crazy line. And no, there is no typo, its running a brace expansion once, which then outputs a very long command which includes brace expansion, and then runs that command with bash:

shell> echo echo {A..Z}\{0..{1..4}\} | bash

search