Visualize the Directory Tree

Ever have to work with a new directory and you have no idea what its structure is? Or maybe you have a few files laying around but you’re not sure which sub-directory they are in? Or maybe you’re showing someone else a project and you want to show them the directory hierarchy. The bare bones solutions of `ls -R` or `find .` are just too archaic and offer no visualization of the structure. To solve this problem, people have built their own “tree” scripts.

There are a few tree scripts available online. Some as simple as find | sed and others are slightly more advanced like a python script. I wasn’t pleased with the existing solutions, so I wrote my own. To get an idea of what I’m talking about take a look at this screenshot showing the listing of a Rails project:

tree

Its a simple, clean listing of the directory tree. I will admit, the style is based off of another tree script that I’ve seen that I liked. Also, this isn’t really production quality code. I take the lazy way out and first get a directory listing and then work from there. This means that for large directories there may be an initial pause before it starts outputting. I wouldn’t suggest running this on your home directory. Although I can think of better algorithms its unlikely that I would want to run this on huge directories so I’m more then happy right now.

The usage is pretty bare bones:

tree usage

This is one more script I’ve added to my ~/bin and its completely open source on GitHub. Its just straight Ruby, no extra packages, works with 1.8 and 1.9. Oh and did I mention its customizable?

I hope you like it.

Check if Your Ruby Script is in a Pipe

Short and simple this week. When you’re writing a Ruby script you might want to know if you’re in a pipe or not. One reason might be changing how you buffer your input/output. A number of basic Unix commands do this, they act differently when piped.

I actually wrote a script where I wanted to change the command line arguments if I was in a pipe. This is probably confusing but its really useful! So there is actually an easy way to do this in Ruby:

#!/usr/bin/env ruby
# Prints if the input or output is regular or piped
puts "INPUT:  #{STDIN.tty?  ? "regular" : "pipe"}"
puts "OUTPUT: #{STDOUT.tty? ? "regular" : "pipe"}"

Proof that it works as advertised:

IO#tty?

The documentation on IO#tty? says the following:

Returns true if [the IO stream] is associated with a terminal device (tty), false otherwise.

For those that don’t know, tty is short for teletype writer. This is an old Unix term for an interactive process that takes in user input. Nowadays it almost always means a shell/console/terminal.

Data URIs

I’m guessing that a lot of web developers don’t know about data URIs. Most probably won’t even have to know about them. However, I still want to go over them because I think they are clever and could prove useful. The main advantage I see is the possibility to avoid an HTTP Request, especially for a small image.

Wikipedia’s overview is quite thorough. The article also links to a number of places where you can quickly throw together some data URI’s for any mime type. The most well known is the data URI kitchen.

Quickly, the format of a data URI is as follows:

data:[<MIME-type>][;charset="<encoding>"][;base64],<data>

Note that almost every section is optional! The mime type option means you can do some very interesting things. The normal usage might be to to embed a small image, as a datauri, inside the html document. However, another usage might be to embed an entire HTML document inside a link!

I was looking at the code for some command line utilities that make datauri’s and I learned an easy way to find out the mime type of a file is to use the `file` command:

shell> file sym.png
sym.png: PNG image data, 76 x 78, 8-bit/color RGBA, non-interlaced

shell> file -i sym.png
sym.png: regular file

shell> file --mime sym.png
sym.png: image/png

So using `file –mime` and reading whats after the ‘: ‘ you can easily get a mime-type for any file. Then reading and base64 encoding the content you have a quick hack to quickly create datauris! Here goes:

#!/usr/bin/env ruby
# Quick & Dirty data URI for a file
#
require 'base64'
require 'uri'

# Filename and Contents
filename = ARGV[0]
contents = File.read(filename)
base64 = Base64.encode64(contents).gsub("\n",'')

# Mime Type for the Filename
output = `file --mime #{filename}`
mime = output.match( /: (.*)$/ )[1].downcase.gsub(/\s/,'')

# Make Data URI
datauri = "data:#{mime};base64,#{base64}"

# Output
puts datauri

Some better scripts are available. Here is a Perl implementation and a pretty neat Ruby Implementation.

Freenode JSBot Command Line Script

So over my week break from college I spent a bunch of time in ##javascript learning and helping others with Javascript problems. This was to help me prepare for one of the projects that I’m working on (still to be announced).

One of the things I really liked in ##javascript was the freenode jsbot that could do all sorts of things. It was so useful in fact that I felt I had to have it for when I’m not using IRC. The website mentioned an API, so I dug in.

I wrote a command line jsbot script and added it to my ~/bin:

jsbot

A clip of the source code (yes its horrible… but its so compact!) shows how easy it is to work with JSON in Ruby. Just a few includes and its just as easy as Javascript, without the cross-site request issues:

#!/usr/bin/env ruby
# Author: Joseph Pecoraro
# Date: Friday March 6, 2009
# Description: Simple Interface for the
# really neat jsbot!

require 'rubygems'
require 'open-uri'
require 'json'
require 'cgi'
require File.dirname(__FILE__) + '/escape'


class JSBot

  JSON_PREFIX = 'http://fn-js.info/jsbot.xhr?'
  SITE_PREFIX = 'http://js.isite.net.au/jsbot?'

  def search(str)
    uri = url(str, JSON_PREFIX, "search=")
    JSON.parse( open( uri ).read )
  end

  def show(str)
    uri = url(str, JSON_PREFIX, "show=")
    JSON.parse( open( uri ).read )
  end

  def url(str, u=SITE_PREFIX, q='q=')
    u + q + CGI::escape(str)
  end

end

...

Ruby Readline Documentation

Before today the Ruby readline library lacked documentation! In order to find some decent documentation you would have to read the README packaged with the source. Michael Fellinger (manveru) from #ruby-lang offered to help me get some decent documentation online. Take a look!

readline

Debugging HTTP Headers

While writing the skreemr shell the other week I ran into an issue that required me to dig down to one of the lowest levels of web communication… HTTP Headers. I’m always shocked to learn that so many web developers don’t know much about headers. So I thought I would try to reenforce the point that HTTP Headers matter… and that knowing your stuff can help you debug and solve problems. Here is the story of my real world example.

I’m not going to go over HTTP Headers. thats already been done. Instead I’m going to focus on debugging and working with them a little bit. I’m going to assume you have already the general concepts.

The Problem

I wanted to add pagination to the skreemr shell. So you could run a search, then get the next page of results, etc. So a little experimentation with skreemr in my browser produced the following URLs:

http://skreemr.com/results.jsp?q=test
http://skreemr.com/results.jsp?q=test&l=10&s=10
http://skreemr.com/results.jsp?q=test&l=10&s=20

A simple pattern! “q” is the query string, “l” is the number per page, “s” is the number to start at, indexed from 0. So that last URL would produce results 21-30. This all worked well in my browsers, but it wasn’t working in Ruby:

require 'open-uri'

# Grab the pages and read the content
str1 = open( 'http://skreemr.com/results.jsp?q=test'      ).read  # 1-10
str2 = open( 'http://skreemr.com/results.jsp?q=test&s=10' ).read  # 11-20

#=> Should print false... but its printing true!
puts str1==str2

What this was saying was that the content being returned from both of those urls is EXACTLY the same. That couldn’t be… could it?

More Investigation

At this point I thought it was a problem in Ruby. I figured Ruby was doing some caching in the background that I was going to have to disable or work around. (I now know this is not true, but that was my first guess). To test that hypothesis I turned to my trusty friend curl and checked to see if that showed the proper behavior:

shell> curl 'http://skreemr.com/results.jsp?q=test'      -o 1.html
shell> curl 'http://skreemr.com/results.jsp?q=test&s=10' -o 2.html
shell> diff -q -s 1.html 2.html
Files 1.html and 2.html are identical

What!?! That stunned me. Curl was getting the exact same results as Ruby. I took a look at the html files, and indeed 2.html, which should have contained results 11-20 held 1-10. I opened both urls in my browser… they showed the correct results. Something weird was happening!

Take A Step Back

At this point you’ve got to know what is happening. Curl and Ruby (using Ruby’s Net:HTTP under the hood) are just making a simple GET request. Both my browsers Safari and Firefox are sending far more then just a GET request. They are sending a bunch of other headers. Lets take a look at what Firebug says Firefox sent:

firebug

Thats a mighty long list of headers! Its entirely possible that one of those might be influencing the server. Onto the drama!

Time to Act – Literally

The only difference between what curl sent in its request and what Firefox was sending is that header information and possibly any information contained in those cookies (unlikely in this case). So lets make curl act as if it is Firefox by send the same headers with curl! I took a quick peek at my curl reference for the proper switches/formatting and I was ready:

shell> curl 'http://skreemr.com/results.jsp?q=test' -o 1.html
shell> curl 'http://skreemr.com/results.jsp?q=test&s=10'            \
            -H 'Host: skreemr.com'                                  \
            -H 'User-Agent: Mozilla/5.0 (...) Firefox/3.0.6'        \
            -H 'Accept: text/html,application/xml;q=0.9,*/*;q=0.8'  \
            -H 'Accept-Language: en-us,en;q=0.5'                    \
            -H 'Accept-Encoding: gzip,deflate'                      \
            -H 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7'     \
            -H 'Keep-Alive: 300'                                    \
            -H 'Connection: keep-alive'                             \
            -H 'Cache-Control: max-age=0'                           \
            -o 2.html
shell> diff -q -s 1.html 2.html 
Files 1.html and 2.html differ

Jackpot! Some subset of those headers is indeed fixing my problem, because now it properly fetched the second page of results! Translating that back into Ruby works as well:

require 'open-uri'

# Headers
HEADERS = {
  'Host'            => 'skreemr.com',
  'User-Agent'      => 'Mozilla/5.0 (...) Firefox/3.0.6',
  'Accept'          => 'text/html,application/xml;q=0.9,*/*;q=0.8',
  'Accept-Language' => 'en-us,en;q=0.5',
  'Accept-Encoding' => 'gzip,deflate',
  'Accept-Charset'  => 'ISO-8859-1,utf-8;q=0.7,*;q=0.7',
  'Keep-Alive'      => '300',
  'Connection'      => 'keep-alive',
  'Cache-Control'   => 'max-age=0'
}

# Get the Pages
str1 = open( 'http://skreemr.com/results.jsp?q=test', HEADERS ).read
str2 = open( 'http://skreemr.com/results.jsp?q=test&s=10', HEADERS ).read

# Correctly Prints false
puts str1==str2

Problem Solved

Sure, we have a solution but its not pretty. How much is really needed? Finding that out is just a matter of eliminating the headers that do nothing. Remove one, test, remove another, test, remove another test… As long as it still works after you’ve removed an individual header you know that header wasn’t needed. It took me a few minutes but I narrowed it down to just a single header:

MAGIC = { 'Accept-Language' => 'en-us' }

This was a rather unique problem. It is rare for me to have to dip down to the HTTP Headers to see what is actually happening for such a high level problem. However, as this problem shows, its important to know what is going on under the hood. If I didn’t know about headers, I would not have been able to solve it.

Mini MP3 Searching Shell – Skreemr

So its way to hard to download an mp3 in Safari. Right click the link and download? Pff, I want to ⌃S and be done with it. Well, this time I decided to avoid the problem all together. I use Skreemr to search for a particular song when it interests me.

In the past I wrote a little bash script, that makes use of curl, to download an mp3 to my desktop unique named so it wouldn’t have conflicts. This shell essentially wraps and drastically improves that to allow for searching, pagination, history, downloading, and opening mp3s off of Skreemr. It gives me just what I need. The functionality that I want without having to use torrents etc. I’m thinking of turning this into a gem.

skreemr.png

This script requires the popular “escape.rb” script that gives some nice and safe shell escaping functions. You can download both from my GitHub scripts project.

Of course its available on my ~/bin and there will be another article later on that goes over a few aspects of this simple little script.

skreemr2.png

search