Visualize the Directory Tree

Ever have to work with a new directory and you have no idea what its structure is? Or maybe you have a few files laying around but you’re not sure which sub-directory they are in? Or maybe you’re showing someone else a project and you want to show them the directory hierarchy. The bare bones solutions of `ls -R` or `find .` are just too archaic and offer no visualization of the structure. To solve this problem, people have built their own “tree” scripts.

There are a few tree scripts available online. Some as simple as find | sed and others are slightly more advanced like a python script. I wasn’t pleased with the existing solutions, so I wrote my own. To get an idea of what I’m talking about take a look at this screenshot showing the listing of a Rails project:

tree

Its a simple, clean listing of the directory tree. I will admit, the style is based off of another tree script that I’ve seen that I liked. Also, this isn’t really production quality code. I take the lazy way out and first get a directory listing and then work from there. This means that for large directories there may be an initial pause before it starts outputting. I wouldn’t suggest running this on your home directory. Although I can think of better algorithms its unlikely that I would want to run this on huge directories so I’m more then happy right now.

The usage is pretty bare bones:

tree usage

This is one more script I’ve added to my ~/bin and its completely open source on GitHub. Its just straight Ruby, no extra packages, works with 1.8 and 1.9. Oh and did I mention its customizable?

I hope you like it.

Check if Your Ruby Script is in a Pipe

Short and simple this week. When you’re writing a Ruby script you might want to know if you’re in a pipe or not. One reason might be changing how you buffer your input/output. A number of basic Unix commands do this, they act differently when piped.

I actually wrote a script where I wanted to change the command line arguments if I was in a pipe. This is probably confusing but its really useful! So there is actually an easy way to do this in Ruby:

#!/usr/bin/env ruby
# Prints if the input or output is regular or piped
puts "INPUT:  #{STDIN.tty?  ? "regular" : "pipe"}"
puts "OUTPUT: #{STDOUT.tty? ? "regular" : "pipe"}"

Proof that it works as advertised:

IO#tty?

The documentation on IO#tty? says the following:

Returns true if [the IO stream] is associated with a terminal device (tty), false otherwise.

For those that don’t know, tty is short for teletype writer. This is an old Unix term for an interactive process that takes in user input. Nowadays it almost always means a shell/console/terminal.

Ruby Readline Documentation

Before today the Ruby readline library lacked documentation! In order to find some decent documentation you would have to read the README packaged with the source. Michael Fellinger (manveru) from #ruby-lang offered to help me get some decent documentation online. Take a look!

readline

Debugging HTTP Headers

While writing the skreemr shell the other week I ran into an issue that required me to dig down to one of the lowest levels of web communication… HTTP Headers. I’m always shocked to learn that so many web developers don’t know much about headers. So I thought I would try to reenforce the point that HTTP Headers matter… and that knowing your stuff can help you debug and solve problems. Here is the story of my real world example.

I’m not going to go over HTTP Headers. thats already been done. Instead I’m going to focus on debugging and working with them a little bit. I’m going to assume you have already the general concepts.

The Problem

I wanted to add pagination to the skreemr shell. So you could run a search, then get the next page of results, etc. So a little experimentation with skreemr in my browser produced the following URLs:

http://skreemr.com/results.jsp?q=test
http://skreemr.com/results.jsp?q=test&l=10&s=10
http://skreemr.com/results.jsp?q=test&l=10&s=20

A simple pattern! “q” is the query string, “l” is the number per page, “s” is the number to start at, indexed from 0. So that last URL would produce results 21-30. This all worked well in my browsers, but it wasn’t working in Ruby:

require 'open-uri'

# Grab the pages and read the content
str1 = open( 'http://skreemr.com/results.jsp?q=test'      ).read  # 1-10
str2 = open( 'http://skreemr.com/results.jsp?q=test&s=10' ).read  # 11-20

#=> Should print false... but its printing true!
puts str1==str2

What this was saying was that the content being returned from both of those urls is EXACTLY the same. That couldn’t be… could it?

More Investigation

At this point I thought it was a problem in Ruby. I figured Ruby was doing some caching in the background that I was going to have to disable or work around. (I now know this is not true, but that was my first guess). To test that hypothesis I turned to my trusty friend curl and checked to see if that showed the proper behavior:

shell> curl 'http://skreemr.com/results.jsp?q=test'      -o 1.html
shell> curl 'http://skreemr.com/results.jsp?q=test&s=10' -o 2.html
shell> diff -q -s 1.html 2.html
Files 1.html and 2.html are identical

What!?! That stunned me. Curl was getting the exact same results as Ruby. I took a look at the html files, and indeed 2.html, which should have contained results 11-20 held 1-10. I opened both urls in my browser… they showed the correct results. Something weird was happening!

Take A Step Back

At this point you’ve got to know what is happening. Curl and Ruby (using Ruby’s Net:HTTP under the hood) are just making a simple GET request. Both my browsers Safari and Firefox are sending far more then just a GET request. They are sending a bunch of other headers. Lets take a look at what Firebug says Firefox sent:

firebug

Thats a mighty long list of headers! Its entirely possible that one of those might be influencing the server. Onto the drama!

Time to Act – Literally

The only difference between what curl sent in its request and what Firefox was sending is that header information and possibly any information contained in those cookies (unlikely in this case). So lets make curl act as if it is Firefox by send the same headers with curl! I took a quick peek at my curl reference for the proper switches/formatting and I was ready:

shell> curl 'http://skreemr.com/results.jsp?q=test' -o 1.html
shell> curl 'http://skreemr.com/results.jsp?q=test&s=10'            \
            -H 'Host: skreemr.com'                                  \
            -H 'User-Agent: Mozilla/5.0 (...) Firefox/3.0.6'        \
            -H 'Accept: text/html,application/xml;q=0.9,*/*;q=0.8'  \
            -H 'Accept-Language: en-us,en;q=0.5'                    \
            -H 'Accept-Encoding: gzip,deflate'                      \
            -H 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7'     \
            -H 'Keep-Alive: 300'                                    \
            -H 'Connection: keep-alive'                             \
            -H 'Cache-Control: max-age=0'                           \
            -o 2.html
shell> diff -q -s 1.html 2.html 
Files 1.html and 2.html differ

Jackpot! Some subset of those headers is indeed fixing my problem, because now it properly fetched the second page of results! Translating that back into Ruby works as well:

require 'open-uri'

# Headers
HEADERS = {
  'Host'            => 'skreemr.com',
  'User-Agent'      => 'Mozilla/5.0 (...) Firefox/3.0.6',
  'Accept'          => 'text/html,application/xml;q=0.9,*/*;q=0.8',
  'Accept-Language' => 'en-us,en;q=0.5',
  'Accept-Encoding' => 'gzip,deflate',
  'Accept-Charset'  => 'ISO-8859-1,utf-8;q=0.7,*;q=0.7',
  'Keep-Alive'      => '300',
  'Connection'      => 'keep-alive',
  'Cache-Control'   => 'max-age=0'
}

# Get the Pages
str1 = open( 'http://skreemr.com/results.jsp?q=test', HEADERS ).read
str2 = open( 'http://skreemr.com/results.jsp?q=test&s=10', HEADERS ).read

# Correctly Prints false
puts str1==str2

Problem Solved

Sure, we have a solution but its not pretty. How much is really needed? Finding that out is just a matter of eliminating the headers that do nothing. Remove one, test, remove another, test, remove another test… As long as it still works after you’ve removed an individual header you know that header wasn’t needed. It took me a few minutes but I narrowed it down to just a single header:

MAGIC = { 'Accept-Language' => 'en-us' }

This was a rather unique problem. It is rare for me to have to dip down to the HTTP Headers to see what is actually happening for such a high level problem. However, as this problem shows, its important to know what is going on under the hood. If I didn’t know about headers, I would not have been able to solve it.

Mini MP3 Searching Shell – Skreemr

So its way to hard to download an mp3 in Safari. Right click the link and download? Pff, I want to ⌃S and be done with it. Well, this time I decided to avoid the problem all together. I use Skreemr to search for a particular song when it interests me.

In the past I wrote a little bash script, that makes use of curl, to download an mp3 to my desktop unique named so it wouldn’t have conflicts. This shell essentially wraps and drastically improves that to allow for searching, pagination, history, downloading, and opening mp3s off of Skreemr. It gives me just what I need. The functionality that I want without having to use torrents etc. I’m thinking of turning this into a gem.

skreemr.png

This script requires the popular “escape.rb” script that gives some nice and safe shell escaping functions. You can download both from my GitHub scripts project.

Of course its available on my ~/bin and there will be another article later on that goes over a few aspects of this simple little script.

skreemr2.png

Updates to line.rb – Open to more suggestions

I was chatting with a friend the other day and he had some really good ideas concerning last week’s line.rb script. He wants to make an improved version. I look forward to seeing what he will come up with because he shared some great ideas. Well, I couldn’t help but implement them immediately in my version. His ideas also spurred a little creativity from me and I came up with a few new features as well. The usage is now looks like:


  # Check Cmd Line Args and Print Usage if needed
  if ARGV.size <= 1
    puts "usage: line [options] filename numbers"
    puts
    puts " options:"
    puts "   --silent or -s   Just print the line, without [#]"
    puts
    puts " number formats:"
    puts "   1-3              Prints lines 1 through 3"
    puts "   5                Prints line 5"
    puts "   -1               Prints the last line, (negative)"
    puts
    puts " extra formats:"
    puts "   ~5               Prints 2 (default) lines before and after 5"
    puts "   4~10             Prints 4 lines before and after 10"
    puts "   *7 or 8*         Prints all lines before or after the number"
    puts "   5/1              Prints 5 lines, then skips 1 line..."
    puts "   2:5/1            Starts at line 2, prints and skips..."
    puts
    exit 1

So there is some cool new syntax:

  • ~# – prints 2 lines before and after the given line number

  • #~# – prints the given number of lines before and after the given line number

  • *# – prints all of the lines before the given line number

  • #* – prints all of the lines after the given line number

  • #/# – is a print and skip option

  • #:#/# – allows you to provide an offset

Lets say you want to print every even line in a text file. With this script that is not a problem. The solution is: starting on the 2nd line, print 1 line, then skip 1 line. (line file 2:1/1):

Print only even lines using line.rb

I think these new formats are pretty neat. You can always access the latest version of the script on Github or in my ~/bin. I’m willing to change the syntax and introduce new concepts. Send suggestions my way by dropping a comment! Cheers.

Quickly Output Lines in a File

The other day I wanted a shell command to have somebody print out the 671st line of /etc/services on their computer. So I gave it some thought, then some more thought, scratched my head, and figured out that I couldn’t really think of a shell command that does that. A google search came up with a few `sed` and `awk` examples but I honestly found those to be a bit awkward for something that should be super simple. So I wrote my own script.

After writing the script to print out a single line, I soon found it made a lot of sense to include ranges. Taking advantage of Ruby I can even print lines from the end with negative numbers. So I spent a few minutes to clean up the script and make it a little more reusable, add some formatting, and cleaner. Here was the result:

line sample usage

line is now the latest addition to my ~/bin.

Also, in case you wanted to use this in a shell script or via piping, there is a `–silent` or `-s` switch that you can use that removes all of the special formatting, and prints only the specified lines. Much nice for scripts. (See the highlighted line in the image below). Enjoy!

line source

So what exactly is line 671 of /etc/services? On my mac it is:

line 671 of /etc/services

search