Debugging HTTP Headers

While writing the skreemr shell the other week I ran into an issue that required me to dig down to one of the lowest levels of web communication… HTTP Headers. I’m always shocked to learn that so many web developers don’t know much about headers. So I thought I would try to reenforce the point that HTTP Headers matter… and that knowing your stuff can help you debug and solve problems. Here is the story of my real world example.

I’m not going to go over HTTP Headers. thats already been done. Instead I’m going to focus on debugging and working with them a little bit. I’m going to assume you have already the general concepts.

The Problem

I wanted to add pagination to the skreemr shell. So you could run a search, then get the next page of results, etc. So a little experimentation with skreemr in my browser produced the following URLs:

http://skreemr.com/results.jsp?q=test
http://skreemr.com/results.jsp?q=test&l=10&s=10
http://skreemr.com/results.jsp?q=test&l=10&s=20

A simple pattern! “q” is the query string, “l” is the number per page, “s” is the number to start at, indexed from 0. So that last URL would produce results 21-30. This all worked well in my browsers, but it wasn’t working in Ruby:

require 'open-uri'

# Grab the pages and read the content
str1 = open( 'http://skreemr.com/results.jsp?q=test'      ).read  # 1-10
str2 = open( 'http://skreemr.com/results.jsp?q=test&s=10' ).read  # 11-20

#=> Should print false... but its printing true!
puts str1==str2

What this was saying was that the content being returned from both of those urls is EXACTLY the same. That couldn’t be… could it?

More Investigation

At this point I thought it was a problem in Ruby. I figured Ruby was doing some caching in the background that I was going to have to disable or work around. (I now know this is not true, but that was my first guess). To test that hypothesis I turned to my trusty friend curl and checked to see if that showed the proper behavior:

shell> curl 'http://skreemr.com/results.jsp?q=test'      -o 1.html
shell> curl 'http://skreemr.com/results.jsp?q=test&s=10' -o 2.html
shell> diff -q -s 1.html 2.html
Files 1.html and 2.html are identical

What!?! That stunned me. Curl was getting the exact same results as Ruby. I took a look at the html files, and indeed 2.html, which should have contained results 11-20 held 1-10. I opened both urls in my browser… they showed the correct results. Something weird was happening!

Take A Step Back

At this point you’ve got to know what is happening. Curl and Ruby (using Ruby’s Net:HTTP under the hood) are just making a simple GET request. Both my browsers Safari and Firefox are sending far more then just a GET request. They are sending a bunch of other headers. Lets take a look at what Firebug says Firefox sent:

firebug

Thats a mighty long list of headers! Its entirely possible that one of those might be influencing the server. Onto the drama!

Time to Act – Literally

The only difference between what curl sent in its request and what Firefox was sending is that header information and possibly any information contained in those cookies (unlikely in this case). So lets make curl act as if it is Firefox by send the same headers with curl! I took a quick peek at my curl reference for the proper switches/formatting and I was ready:

shell> curl 'http://skreemr.com/results.jsp?q=test' -o 1.html
shell> curl 'http://skreemr.com/results.jsp?q=test&s=10'            \
            -H 'Host: skreemr.com'                                  \
            -H 'User-Agent: Mozilla/5.0 (...) Firefox/3.0.6'        \
            -H 'Accept: text/html,application/xml;q=0.9,*/*;q=0.8'  \
            -H 'Accept-Language: en-us,en;q=0.5'                    \
            -H 'Accept-Encoding: gzip,deflate'                      \
            -H 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7'     \
            -H 'Keep-Alive: 300'                                    \
            -H 'Connection: keep-alive'                             \
            -H 'Cache-Control: max-age=0'                           \
            -o 2.html
shell> diff -q -s 1.html 2.html 
Files 1.html and 2.html differ

Jackpot! Some subset of those headers is indeed fixing my problem, because now it properly fetched the second page of results! Translating that back into Ruby works as well:

require 'open-uri'

# Headers
HEADERS = {
  'Host'            => 'skreemr.com',
  'User-Agent'      => 'Mozilla/5.0 (...) Firefox/3.0.6',
  'Accept'          => 'text/html,application/xml;q=0.9,*/*;q=0.8',
  'Accept-Language' => 'en-us,en;q=0.5',
  'Accept-Encoding' => 'gzip,deflate',
  'Accept-Charset'  => 'ISO-8859-1,utf-8;q=0.7,*;q=0.7',
  'Keep-Alive'      => '300',
  'Connection'      => 'keep-alive',
  'Cache-Control'   => 'max-age=0'
}

# Get the Pages
str1 = open( 'http://skreemr.com/results.jsp?q=test', HEADERS ).read
str2 = open( 'http://skreemr.com/results.jsp?q=test&s=10', HEADERS ).read

# Correctly Prints false
puts str1==str2

Problem Solved

Sure, we have a solution but its not pretty. How much is really needed? Finding that out is just a matter of eliminating the headers that do nothing. Remove one, test, remove another, test, remove another test… As long as it still works after you’ve removed an individual header you know that header wasn’t needed. It took me a few minutes but I narrowed it down to just a single header:

MAGIC = { 'Accept-Language' => 'en-us' }

This was a rather unique problem. It is rare for me to have to dip down to the HTTP Headers to see what is actually happening for such a high level problem. However, as this problem shows, its important to know what is going on under the hood. If I didn’t know about headers, I would not have been able to solve it.

Instant Web Sharing on Your Mac

I came across a few articles recently that point out how to instantly share a folder on your computer. They basically ride on top of this elegant python script:

python -m SimpleHTTPServer

It works great but I wanted to improve on it in a number of ways:

  • Automatically copy a URI into my clipboard so I can easily paste it to others.

  • Make that URI nicer then just an IP address.

  • Use a non-standard port, for security.

  • Open in a new tab so I can keep working in the directory and yet still monitor the HTTP requests being made.

Here was what I produced. (Its up in my ~/bin.)

#!/bin/bash
# Start Date: Sunday February 8, 2009
# Current Version: 0.9
# Author: Joseph Pecoraro
# Contact: joepeck02@gmail.com
#
# Decription:  Immediately Share the current directory
#   in a new tab so you can monitor the requests made
#   have your original tab to continue working in that
#   directory.  Meant for Mac OS X.
#
#   1. Echos the URI
#   2. Puts the URI into your Clipboard
#   3. Opens a new tab in the terminal
#   4. Changes Directory to the other tabs directory
#   5. Echos the URI
#   6. Runs the Web Server
#   7. Optionally Opens in Safari
#
# Sources that Helped:
#   New Tab Here: http://justinfrench.com/index.php?id=231
#   HTTPServer:   http://www.commandlinefu.com/commands/view/71/
#   Paul Berens:  http://zibundemo.blogspot.com/
#

# -----------------
#   Host and Port
# -----------------

# This gets your ip address and converts it to a nice string
es_host=$(curl --silent www.whatismyip.com/automation/n09230945.asp)
es_host=$(nslookup $es_host | awk '/name =/{print substr($4,1,length($4)-1)}')
es_port="8000"


# -----------------
#   Script Below
# -----------------

echo "http://$es_host:$es_port"
echo -n "http://$es_host:$es_port" | pbcopy
osascript -e "
Tell application \"Terminal\"
  activate
  tell application \"System Events\" to tell process \"Terminal\" to keystroke \"t\" using command down
  do script with command \"cd '$(pwd)'\" in selected tab of the front window
  do script with command \"clear; echo '$es_host:$es_port/'\" in selected tab of the front window
  do script with command \"python -m SimpleHTTPServer $es_port\" in selected tab of the front window
end tell" &> /dev/null

# Optional: Open Safari, Just Uncomment the next line
# open "http://$es_host:$es_port"

# Cleanup
unset es_host
unset es_port

Now that should work on any Mac. And it should give a nicer URL then an ugly IP address. You should see something like this:

easy_share

As soon as it starts you can paste the URL to anyone you’re chatting with. It couldn’t be simpler!

If you’re experienced enough with DNS servers and you’ve given your computer a Dynamic Name you can customize the script. Paul Berens gave me a great suggestion to determine if I’m on my local network at home. I can check the MAC address of my default gateway (my wireless router). That is a quick check to see if I’m at home. If I’m at home I use my bogojoker.is-a-geek.com URI automatically! Otherwise it defaults to generating the dynamic address generation. Check it out:

# -----------------
#   Host and Port
# -----------------

# Mac Address of my Router At Home
if [ -n "$(arp -a | grep 0:1e:2a:76:17:98)" ]; then
  es_host="bogojoker.is-a-geek.com"
  es_port="8000"
  
# Otherwise Dynamically Determine
else
  es_host=$(curl --silent www.whatismyip.com/automation/n09230945.asp)
  es_host=$(nslookup $es_host | awk '/name =/{print substr($4,1,length($4)-1)}')
  es_port="8000"
fi

So now when I run easy_share at my house it always throws out bogojoker.is-a-geek.com URIs. Much nicer on the eyes and easy to remember. I’ll write about dynamic names like this another time!

“Back and Forth” Greasemonkey For The Whole Web

Recently I wrote a Greasemonkey script to add keyboard shortcuts to The Big Picture, to improve on some of their already existing shortcuts. Once I started using some of the shortcuts I made I ended up wanting to use them all over the place at other blogs. This functionality is so tiny, but so useful, that I bundled it into its own script that runs on all web pages!

Grab it here:

//
// ==UserScript==
// @name          Back and Forth
// @namespace     http://blog.bogojoker.com
// @description   Keyboard Shortcut to Jump back and forth on a page. (esc key).
// @include       *
// @version       1.0 - Initial Version - Sunday February 15, 2009
// ==/UserScript==

(function() {
  
  // Global States
  var x = null;
  var y = null;

  // Add a new Global Key Listener for `esc`
  document.addEventListener("keypress", function(e) {
    if(!e) e=window.event;
    var key = e.keyCode ? e.keyCode : e.which;
    if ( key == 27 ) {
      var tempx = x;
      var tempy = y;
      x = Math.max(document.documentElement.scrollLeft, document.body.scrollLeft);
      y = Math.max(document.documentElement.scrollTop, document.body.scrollTop);
      if ( tempx != null ) { // First time it should be null
        window.scrollTo(tempx, tempy);
      }
    }
  }, true);

})();

On any webpage the first time you push the `esc` key position A gets stored. The next time you push `esc` position B gets stored and the browser jumps to position A. The next time you push it, A gets stored and you jump to B. So you always jump back to wherever you pushed `esc` last. Hence the name “back and forth.”

This is useful to me when I jump between comments and the content. When I’m reading a comment and I want to check back to the article, I just just push `esc` to save my position, go back to the article, and when I’m all set I just jump back to my saved position (the comments) with `esc`.

Short, Sweet, Simple: The Back and Forth Greasemonkey Script.

More Big Picture Keyboard Functionality!

I’ve mentioned before how I’m a big fan of The Big Picture blog. One of the things that makes it so great is that it has keyboard navigation! You can use ‘j’ and ‘k’ to automatically jump between pictures. Its so much nicer then scrolling because it jumps to the exact height to maximize the picture in the browser. Huge usability improvement!

Like before, the problem I had was that users were mentioning pictures in their comments. Jumping back to that picture was hard or annoying. So, I wrote a Greasemonkey script that allows you to type in a number and it will automatically jump to that picture! Click here to get the script!

Oh, and if you’re reading comments and you want to jump back and forth between images and comments that works too. Once you’ve jumped to the image, just hit ‘esc’ and you will be taken back to where you were before. Too cool!

//
// ==UserScript==
// @name          The Big Picture Keyboard Enhancements
// @namespace     http://blog.bogojoker.com
// @description   Keyboard Shortcut Enhancements
// @include       http://www.boston.com/bigpicture/*
// @version       1.1 - Added Back+Forth - Thursday February 12, 2009
//                1.0 - Initial Version - Monday February 9, 2009
//
//
//   This allows the user to type in numbers, and after about
//   a half second it will jump directly to that image.
//   For example:
//
//     Push '1'... '2'... User is taken directly to Image "12"
//     Push '9'... '9'... User is taken to the last picture.
//
//   Use 'esc' to jump back and forth between two positions.
//   For example if a comment mentions picture 4:
//
//     Push '4'...        User is taken directory to Image "4"
//     Push 'esc'         User is taken back to the comment!
//     Push 'esc'         User is taken back to Image "4"
//
// ==/UserScript==

(function() {
  
  // GreaseMonkey (Firefox - unsafeWindow) and GreaseKit (Safari - window)
  var w = ( /a/[-1]=='a') ? unsafeWindow : window;

  // Global States
  var x, y;
  var keypressnumber  = false;
  var builtupnumber   = '';
  var quicknumtimeout = null;
  var imgArr = document.getElementsByClassName("bpImage");
  
  // Keep the old and create a New Global Keypress listener on top of it
  document.addEventListener("keypress", function(e) {
    

    // Get the key
    if(!e) e=window.event;
    var key = e.keyCode ? e.keyCode : e.which;
    
    // Store the current x/y position
    function storePos() { 
      x = Math.max(document.documentElement.scrollLeft, document.body.scrollLeft);
      y = Math.max(document.documentElement.scrollTop, document.body.scrollTop);
    }
    
    // # Character => Jump to that image, Store Position
    if ( key >= 48 && key <= 57 ) {
      if ( e.target.nodeName.match(/TEXTAREA|INPUT/) ) return;
      clearTimeout(quicknumtimeout);
      keypressnumber = true;
      builtupnumber += (key - 0x30);
      quicknumtimeout = setTimeout(function() {
        w['currImg'] = parseInt(builtupnumber,10)-1;
        if (w['currImg'] >= imgArr.length) { w['currImg'] = imgArr.length-1; }
        storePos();       
        window.scrollTo(0,imgArr[ w['currImg'] ].offsetTop+174);
        keypressnumber = false;
        builtupnumber = '';
        quicknumtimeout = null;
      }, 300);
    }
    
    // Esc => Jump back to a Saved Position
    if ( key == 27 ) {
      var tempx = x; var tempy = y; storePos();
      window.scrollTo(tempx, tempy);
    }   

  }, true);

})();

All I do is register a new global keyboard listener to catch numeric keys and act accordingly. I tested thoroughly on Firefox and Safari to make sure it works correctly in all cases. It jumps to the correct image, it maintains the “current image” so j/k will still work, it won’t jump if you’re typing in a textfield/input, etc. It even properly handles situations that the current j/k functionality doesn’t. For instance mine allows the user to type in the comment box, click outside the box and reuse the keyboard shortcuts. When you type a j/k or even click inside the search box at the top the j/k functionality is gone. I didn’t feel like correcting that in this Greasemonkey script in case it gets fixed by the developers behind the Big Picture. (Note to those developers: reset isLoaded back to true or take a different approach.)

I’m open to New Ideas. I have some myself but I have to focus on schoolwork in these next few weeks. Let me know if you want anything.

Big Picture Keyboard Commands Greasemonkey Script!

Enjoy.

Always Have Correct Footer Dates

One thing that really bothers me is when the year changes (2008 to 2009) and I see a ton of websites that sport “© 2008” in their footer. So, I wanted to share my php code to handle this case so that my sites are always “up to date.”

<?php

/*
  Given Start Year is 2007
  Pretend Current Year is 2007
    => Output is: "2007"
  Pretend Current Year is 2009
    => Output is: "2007-2009"
*/
function footerDate($startYear, $delim='-') {
  $currYear = date('Y');
  if ( intval($currYear) > intval($startYear) ) {
    return $startYear . $delim . $currYear;
  } else {
    return $startYear;
  }
}

?>

<!-- Examples Usage -->
<p>&copy; <?php echo footerDate('2009'); ?> Joseph Pecoraro</p>
<p>&copy; <?php echo footerDate('2007'); ?> Joseph Pecoraro</p>
<p>&copy; <?php echo footerDate('2007', ' to '); ?> Joseph Pecoraro</p>

And the ugly 1 liner if you only need a little clip of php:

<?php $c=date('Y');echo '2009'.((intval($c)>2009)?' - '.$c:''); ?>

So please, update your footers now so that you never have to update them again! Thanks.

Playing With Some More jQuery Plugins

I decided to play with some more jQuery plugins. I’m building up some experience, learning a lot, and having a ton of fun doing it. So, what did I decide to do this time? I added a sidebar to ~/bin. When you click on the logo in the top right an instructions panel will slide out on the right hand side of the screen. There is also a little magic going on with the logo.

sidebar

This sidebar has some neat animation so it was fun to put into the page. It is a modified version of the jQuery pageSlide plugin. To my pleasant surprise the plugin already had the capability to run callback functions before and after the transition is run. I added a property allowing for static html so I wouldn’t need to depend on an Ajax call. Also, the default behavior of the plugin is that clicking anywhere on the document, except in the sidebar, will cause the page to slide back to normal. This is okay except that I wanted a button for usability purposes to close the sidebar and also links in the sidebar (normal anchor tags) to work. That took a little creativity but I eventually got it:

      // Make it so if the user clicks in the pageslide, it won't close
      // Exception for given closers that may be in the pageslide
      $("#pageslide-slide-wrap").click(function(e){
        if ( $('a, ' + settings.closers).index( $(e.target) ) == -1 ) {
          return false;
        }
      });

 

Protecting the Innocent

I also wanted to play around with some jQuery lightboxes. The one that always comes to mind is facebox. For once I didn’t have to make any major changes to the source of this plugin, instead I made a few small tweaks to the CSS.

My goal here was to provide IE6 users with a simple little “Please Upgrade Your Browser” message. I went to work, learned a lot, and came up with the following: (to view this yourself go to ~/bin and paste “javascript:if_i_were_ie6()” in your URL bar, without the quotes!):

facebox browser upgrade warning

Sure that looked great, and it had a nice message, but it ended up looking disgraceful in IE6 itself! Rather then scrap the work I left it there, to prove the point to whatever few IE6 visitors I get. The facebox works perfectly in IE8 (I haven’t tried IE7) and so I can make use of what I’ve learned elsewhere I just hope that IE6 will go away soon. I said my goodbye already.

Adding Looping to a Content Slider

A few days ago I came across the wonderfully magnificent easySlider jQuery plugin. It makes sliding content vertically or horizontal drop dead simple using totally natural markup and CSS. I immediately worked it into the design of ~/bin (this animation was in my original design on paper) and it worked flawlessly. However, like the plugin’s title implies, it was and is meant to be simple and therefore lacks some features in order to gain ease of use.

I took some time to become more fluent in jQuery, and add some enhancements by modifying the plugin. One of those enhancements was looping, and is the topic of this article. I had no prior experience working with these types of galleries and implementing looping seemed like a fun idea. So here was the result of my tinkering:

How Do You Implement Looping?

Give yourself a minute to get creative. Develop some ideas and strategies on how you would take the following markup for the slides, and make it loop.

<div id="slider">
  <ul>
    <li>content here...</li>
    <li>content here...</li>
    <li>content here...</li>
  </ul>
</div>

I came up with two strategies that use a similar technique to implement the loops. The above loop is an implementation of one of these strategies, but I wanted to compare and contrast my two versions here, and hopefully get some reader feedback on different techniques!

Hidden Moves Strategy

This strategy actually involves physically moving unseen slides around in the DOM. By moving the slides in such a way that there is always the proper “next slide” and looping can continue forever in either direction without errors.

Hidden Moves Strategy

Instantaneous Jumps between Clones Strategy

This strategy actually involves cloning the first slide and adding it to the end of the list. This now makes the first and last slides exactly the same. When we need to handling transitions to and from these slides we can instantly jump between them, and the user is none the wiser.

Instantaneous Jumps Between Clones

Comparing the Pros and Cons of Each

Hidden Moves:

  • Requires no extra space (in theory).

  • Simple to implement moves. 1 line executed as a callback on “next” and before the transition if “prev”

  • Expensive. Requires moving slides on every transition. This is actual DOM manipulation. Sure in nearly all cases this is negligible but it could be a pain for large, complex slides.

  • Wouldn’t work on looping a single slide. Kinda useless though right?

  • Does it work? By taking a slide off of the front won’t everything shift? Yes, this could get tricky, I honestly haven’t tried it.

Instant Jumps:

  • Requires extra space to duplicate the first slide.

  • Simple to clone a slide. 1 line of jQuery.

  • 1 time cost to clone, never again is there a DOM Manipulation cost.

  • Has the ability to loop on a single slide.

  • Tricky code to handle when to jump. Nothing difficult, just messy.

  • Does it work? Yes, Its what I’m using above..

Thats it for my analysis. Do you have any suggestions?

search