Quickly Output Lines in a File

The other day I wanted a shell command to have somebody print out the 671st line of /etc/services on their computer. So I gave it some thought, then some more thought, scratched my head, and figured out that I couldn’t really think of a shell command that does that. A google search came up with a few `sed` and `awk` examples but I honestly found those to be a bit awkward for something that should be super simple. So I wrote my own script.

After writing the script to print out a single line, I soon found it made a lot of sense to include ranges. Taking advantage of Ruby I can even print lines from the end with negative numbers. So I spent a few minutes to clean up the script and make it a little more reusable, add some formatting, and cleaner. Here was the result:

line sample usage

line is now the latest addition to my ~/bin.

Also, in case you wanted to use this in a shell script or via piping, there is a `–silent` or `-s` switch that you can use that removes all of the special formatting, and prints only the specified lines. Much nice for scripts. (See the highlighted line in the image below). Enjoy!

line source

So what exactly is line 671 of /etc/services? On my mac it is:

line 671 of /etc/services

Dynamic Web URLs with ExpanDrive

Often when I work with ExpanDrive the files I am working on correspond to some website that I own. When I’m mounted with ExpanDrive each file is accessible via “my hard drive” in the mounted volume and, more importantly, from a web URL! I found myself repeatedly opening up my browser and manually typing the URL for files that I just uploaded or edited. This was error prone, especially if some of the characters needed encoding. So, I spent some time to write up a Ruby script that can read ExpanDrive’s preferences, build the file’s “web URL,” and open it in your default browser.

expanurl usage

Simple to Use

I followed along with ExpanDrive’s previous command line tool named expan and named my script expanurl. Given no arguments it will open the current directory via its web URL, or you can give a list of files and each will be opened at their web URLs. Its usage is pretty straightforward but there is a single catch: the server setting in an ExpanDrive Drive may not be a true one-to-one mapping with the web server’s address.

For example: I provide holly.cs.rit.edu as the server value for one of my personal ExpanDrive drives. However, when I view files on that server (inside the public_html directory) they have a much different looking URL: http://www.cs.rit.edu/~jjp1820/. The result? The script simply keeps its own mapping of ExpanDrive server values to their associated web page prefixes. When you use expanurl on a Drive you have never used it on before the script will prompt you for that mapping, store the value, and never ask again.

Here it is in action. I have removed all stored mappings so I can demonstrate what it would be like using expanurl for the first time. Here I use it on my BogoJoker ExpanDrive drive:

expanurl first usage

Notice that in the prompt it tells you:

  • the server that the ExpanDrive volume is linked to and the one you will be providing a web url prefix for
  • an example of a web url prefix (useful)
  • where the mappings are stored in case you need to edit them later

The script is available on GitHub, so feel free to contribute and improve. Here is a link to the always current version, and here is a snapshot of the current version at the time of writing:

 

#!/usr/bin/env ruby
# Author: Joseph Pecoraro
# Date: Saturday December 13, 2008
# Description: When I'm using ExpanDrive and I'm
# remotely logged into a server, I can use this
# script to "open filename" and it will open using
# the server's associated URL.

# For URL Escaping and Stored Mappings
require 'uri'
require 'yaml'

# Exit with a msg
def err(msg)
  puts msg
  exit 1
end

# Simple Class to handle the mappings
class UrlMap

  MAP_PATH = File.expand_path('~/.expanurl')

  def initialize
    @hash = load
  end

  def load
    if File.exists?(MAP_PATH)
      YAML::load_file( File.expand_path(MAP_PATH) )
    else
      Hash.new
    end
  end

  def add_mapping(server, mapto)
    @hash[server] = mapto
    File.open(MAP_PATH, 'w') do |file|
      file.write(@hash.to_yaml)
    end
  end

  def is_mapping?(server)
    @hash.has_key?(server)
  end

  def get_mapping(server)
    @hash[server]
  end

  def path
    MAP_PATH
  end

end

# Local Variables
mapping = UrlMap.new
url_prefix = nil
server = nil
volume = nil

# Check the if the current directory is an
# ExpanDrive Volume and a public_html folder
pwd = `pwd`
match = pwd.match(/^\/Volumes\/([^\/]+)/)
if match.nil?
  err("Not inside an ExpanDrive Volume")
elsif !pwd.match(/\/public_html\/?/)
  err("Not inside a public_html directory.")
else
  volume = match[1]
  defaults = `defaults read com.magnetk.ExpanDrive Drives`
  defaults.gsub!(/\n/, '')
  props = defaults.match(/\{[^\}]+driveName\s+=\s+#{volume}[^\}]+server\s+=\s+"([^"]+)"[^\}]+\}/)
  if props
    server = props[1]
  else
    err("This Volume (#{volume}) is not an ExpanDrive Volume")
  end
end

# Check if a mapping exists
# Otherwise create and store one
if mapping.is_mapping?(server)
  url_prefix = mapping.get_mapping(server)
else

  # Prompt
  puts
  puts "This is the first time you've used expanurl for #{volume}"
  puts "Please Provide us with a mapping for #{server}"
  puts "Mappings are stored in #{mapping.path}"
  puts "Example: http://bogojoker.com/"
  print ">> "

  # Store user input and proceed
  url_prefix = gets.chomp
  url_prefix += '/' unless url_prefix.match(/\/$/)
  mapping.add_mapping(server, url_prefix)

  # Terminal Output
  puts
  puts "Server: #{server}"
  puts "Maps to: #{url_prefix}"
  puts

end

# Build the URL
subpath = pwd.match(/public_html\/?(.*)/)[1]
subpath += '/' unless subpath.length.zero? || subpath.match(/\/$/)
url_prefix += subpath

# If No Files, open the directory
# Otherwise,   open each provided file
if ARGV.size == 0
  `open #{url_prefix}`
else
  ARGV.each do |filename|
    `open #{url_prefix}#{URI.escape(filename)}`
  end
end

 

How it Works

The Ruby Script grabs the current working directory using `pwd` and checks to make sure you’re in an ExpanDrive volume. ExpanDrive volume’s are dynamically generated by parsing the ExpanDrive preferences thanks to their foresight to make them accessible via the `defaults` command. So if you’re in an ExpanDrive volume and inside a public_html directory expanurl will then use its mapping to open a uri encoded web url in your default browser with the `open` command.

The mappings are stored in a hidden YAML file in your home directory (~/.expanurl). This style of storing preferences is just like dozens of other command line applications and scripts. YAML is just a lightweight textual data format popular with Ruby, similar to JSON and XML. Its so simple that you could edit the file yourself if you wanted/needed to. For instance here is what is in mine, just two simple key/value pairs:

~/.expanurl yaml mapping

The Future

Its just that simple. Being a Ruby Script you can call this from GUI applications, anything with built-in shell access, etc. It should play friendly with your usual Unix tools. I will likely make this script more and more robust if others find it useful, so I’d be happy to hear some feedback.

Cheers.

Mac OS X What is my IP Address?!

I’ve asked this question a lot. What is my IP address? This can be for any number of reasons. There are a number of ways that this can be done, but I was looking for a way to do this on the command line. Well… it didn’t turn out to be as simple as I had hoped.

ipaddr

The primary tool is ifconfig. However, that spits out way more information then I wanted. So a little bit of reading, some regular expressions, and I created the following script. Note that en1 is my wireless port because I am always on my wireless:

#!/bin/sh
ifconfig en1 | awk '$2~/[0-9]+./{print$2}'

Update: I found a better solution:

#!/bin/sh
# Update: Even Better!
ipconfig getifaddr en1

Here is some usage and me checking that it actually makes sense:

ipaddr usage

ipaddr2

I’m paranoid… so let me double check at whatismyip.com. “What!?” That didn’t make sense. Or did it? A few minutes later, after running a traceroute and following the hops I noticed that sure enough the Frontier Wireless provider in this public CafĂ© was using that IP address. So I wrote a quick curl script to pull whatismyip’s opinion and quite craftily they must have planned for this:

whatismyip suggestion

Cool. That makes my job even simpler. The response is simply my ip address string. Add an echo to provide a newline and I’ve got a useful script.

#!/bin/sh
curl --silent www.whatismyip.com/automation/n09230945.asp
echo

Hopefully these two scripts can help save someone a few minutes. ipaddr to see what my machine’s ip address is and ipaddr2 to see through what ip address my computer is reaching the outer world. Cheers.

Ruby Process Controller – psgrep

Every once in a while a process will freeze and will be too stubborn to die when I try to “Quit” it. For those stubborn processes I tend to use the terminal to `kill` it. For a while I had been using a simple Perl script for searching through processes. The script would find me the processids and I could then kill it, using whatever power I need.

I found that it was taking far too long for me to do the search, and then carefully type out the process id, and hope I got the right one. I discovered killall, but the problem is that sometimes I don’t want to kill “all” of the processes with that name. So, I gave in and wrote up a Ruby script that did what I wanted. Here is psgrep: (Download)

#!/usr/bin/env ruby
# Start Date: Saturday December 6, 2008
# Current Version: 0.9
# Author: Joseph Pecoraro
# Contact: joepeck02@gmail.com
# Decription: Quicker handling of process searching
# and killing.  Never type in a PID again, just regexes!

# -----------
#   Globals
# -----------
kill_mode = false
icase_mode = false
pattern = nil
targets = []
pids = []

# ---------
#   Usage
# ---------
def usage
  puts "usage: #{$0.split(/\//).last} [options] pattern"
  puts "  -k  or --kill   kills all processes"
  puts "  -k# or --kill#  kills the [#] process"
  exit 0
end

# -----------
#   Options
# -----------
if ARGV.size > 1
  ARGV.each do |arg|
    if arg.match(/^-(k|-kill)(\d*)$/)
      kill_mode = true
      targets << $2.to_i unless $2.empty?
    elsif arg.match(/^-(i|-ignore)$/)
      icase_mode = true
    end
  end
  ARGV.delete_if { |e| e.match(/^-/) }
end

# -------------------
#   Remaining Args
# -------------------
if ARGV.size != 1
  usage
end

if icase_mode
  pattern = Regexp.new( ARGV[0], Regexp::IGNORECASE )
else
  pattern = Regexp.new( ARGV[0] )
end

# ----------------------
#   Actual `ps` Output
# ----------------------
lines = %x{ ps -Au#{ENV['USER']} }.split(/\n/)
header = lines.shift

# ----------
#   psgrep
# ----------
puts
puts "     #{header}"
count = 0
lines.each do |line|
  unless line =~ /psgrep/
    if line.match(pattern)
      count += 1
      puts "[#{count}]: #{line}"
      if targets.empty? || targets.member?(count)
        pids << line.strip.split[1]
      end
    end
  end
end

# -------------
#   Kill Mode
# -------------
if kill_mode
  puts
  puts "Killing Processes"
  puts "-----------------"
  pids.each_with_index do |pid, i|
    print targets.empty? ? "[#{i}]:" :  "[#{targets[i]}]:"
    print " Killing #{pid}... "
    STDOUT.flush
    res = %x{ kill #{pid} }
    puts "Dead" if $?.exitstatus.zero?
  end
end

# Always
puts

So there it is. Less then 100 lines of ruby to get a pretty straightforward psgrep/kill program. Here is an example where I have three perl processes running on my machine. One of the is running as root (the userid is 0). I just type “!! –kill” or “[up-arrow] –kill” and it tries to kill them all. Note that the root perl process doesn’t terminate and there is an error message but psgrep continues as best as it can, and kills the two normal perl processes: [Note: I could have done `sudo psgrep perl -k` to kill the root process]

psgrep usage

Here is another good example. I have two python instances that I want to kill but there is another python instance running ExpanDrive in the background. I just ran psgrep and found the two I want to kill are [2] and [3]. Therefore, I can send -k2 and -k3 (or –kill2 and –kill3) to kill only those processes. Here is the result:

psgrep target

Note also that by default psgrep is case-sensitive. To ignore case just add the -i or –ignore switch. So there you have it. Usage is straightforward. Switches can go anywhere on the command line makes it easier to just use your history and tack a switch on the end of your previous command.

Feel free to improve it, it is on GitHub!

AtomPub Overview and Curl Reference

Not long ago I had to learn about the Atom Publishing Protocol for my job. I spent about a week learning on my own time all about XML, AtomPub, and even the basics of HTTP. After that week I decided to write down my own personal overview and example code to try and “visually” explain AtomPub as best I could. The result was (and is):

My Visual Guide to AtomPub

Now keep in mind that I wrote that only a few weeks after learning it. The process of writing that guide forced myself to study it in greater detail than normal, actually run tests, and produce realistic output and examples. I know its not perfect (I’d probably be slaughtered for my definition of REST) but over time I’ll be happy to improve and update it. I think the design really improves the content making it readable, fun, and useful to refer to.

I’m linking to it now because I’ve done a number of projects like this (my Unix Tutorial) because I like sites that are strictly focused on one thing and do that one thing very well. I’ll probably spend a little bit of time on remainder of my break from school by cleaning up these small “brain dump” websites. I wanted to make sure they were mentioned and linked to from my blog. Clearly they will be of no use to anyone if they are never linked to!

I decided to include a small `curl` reference on my AtomPub guide. This is because its a very nice tool when working with HTTP requests and an overall generally useful shell program. I think people might find the curl reference useful.

I hope you enjoy this. I’ll be linking to these occasionally as they grow.

Command Line svn:ignore a file

This took me far too long to do, but its because no one explained it correctly to me. I’m frustrated enough that I’m not going to go back to those other websites and see if I just overlooked something. Instead I’m going to put it right here:

You don’t svn:ignore a file.

You put an svn:ignore property on the directory to ignore that filename pattern!

That makes sense, but I didn’t immediately think of that, and no source sufficiently made that point clear enough. So, if you were struggling, please grasp that concept and take a look at the commands below (which you no doubt have seen and did not think worked) and you’ll really understand what they do.

# ---------------------------------------------------------------------
#      Ignore all the .txt files in the /trunk/Blah/ directory
# ---------------------------------------------------------------------

# Go to the directory
cd trunk/Blah/              # The directory with the files

# Start editing the properties for the current directory
svn propedit svn:ignore .   # Opens an editor (SVN_EDITOR, EDITOR)

# Add the following value with a new line, save, and exit:
*.txt

# See that things worked
svn propget svn:ignore .    # So you can see the properties
svn status --no-ignore      # You should see an 'I' next to the ignored files

# Commit
svn commit -m "New Ignores" # You must commit the new property change


# ---------------------------------------------------------------------
#     Ignore a single file secret.txt in the /trunk/ directory
# ---------------------------------------------------------------------

# Go to the directory
cd trunk/

# Add just the single file to the current directories ignore list (like above)
# Note the dot at the end of the command is important
svn propset svn:ignore secret.txt .

# See that things worked
svn propget svn:ignore .    # Notice the single file was added to the list
svn status --no-ignore      # You should see an 'I' next to the ignored files

# Commit
svn commit -m "Its secret"  # You must commit the new property change

That also means in GUI programs if you can’t seem to ignore a single file that is unversioned you should instead go to the directory that file is in and (like the above) add the filename to the svn:ignore list. Cheers.

UPDATE: A few commenters have mentioned this (2 John’s below!) and it is worth adding here. If the file is already under version control or shows up as ‘M’ instead of ‘I’, then you’ll first have to svn delete the file from the repository (make a backup of it somewhere first), then svn ignore the file using the steps above and copy the file back into the repository.

Dot Files For Your Shell and Even Ruby

I have come across a number of programmers who don’t know what dotfiles are. Thats a shame. Every programmer should know common dotfiles and actively seek to add aliases, functions, and other tidbits to increase their productivity and make the shell more usable. I recently went on a binge and updated a few of my dotfiles. I’ll share some useful tricks that I found on not only my ~/.bashrc file but also ~/.irbrc for my ruby IRB prompt!

I won’t show off my entire ~/.bashrc file, only because it is rather long. I just want to get across some of the usefulness of such a file:

# -----------
#   General
# -----------
alias ..='cd ..'
alias ll='ls -lh'
alias la='ls -la'
alias ps='ps -ax'
alias du='du -hc'
alias cd..='cd ..'
alias more='less'
alias mkdir='mkdir -p'
alias today='date +"%A, %B %d, %Y"'
alias yest='date -v-1d +"%A %B %d, %Y"'
alias recent='ls -lAt | head'
alias ebashrc='mate ~/.bashrc'
alias mbashrc='mate ~/.bashrc'
alias sbashrc='source ~/.bashrc'
alias htdocs='cd /Applications/MAMP/htdocs/'
alias mampmysql='/Applications/MAMP/Library/bin/mysql -u XXXXX -p'
alias desktoptopia='open /Users/joe/Library/Application\ Support/Desktoptopia/.Backgrounds/'
alias ql='qlmanage -p "$@" >& /dev/null' # Quick Look alias

# -------------
#   Shortcuts
# -------------
alias c="clear"
alias m="mate"

# --------
#   SSHs
# --------
alias rit="ssh holly.cs.rit.edu -l XXXXX"
alias vega="ssh vega.it.rit.edu -l XXXXX"

# -------
#   Git
# -------
alias ga='git add'
alias gs='git status'
alias gd='git diff'
alias github="open \`git config -l | grep 'remote.origin.url' | sed -En 's/remote.origin.url=git(@|:\/\/)github.com(:|\/)(.+)\/(.+).git/https:\/\/github.com\/\3\/\4/p'\`"

# --------
#   Ruby
# --------
alias irb='irb -r irb/completion -rubygems'

# ---------------
#   Environment
# ---------------
export PATH="$PATH:/usr/local/bin:/usr/local/sbin:/usr/local/mysql/bin"
export PATH="$HOME/bin/:$PATH"
export HISTSIZE=10000
export HISTFILESIZE=10000
export PAGER=less
export CLICOLOR=1
export EDITOR="/usr/bin/mate -w"

Hopefully this wasn’t too overwhelming. But lets take a look at some of these. There are a bunch of aliases at the top which simply replace the older version? It just makes sense that when you do a ps you really want `ps -ax`. Likewise a few others there are printing disk usage with human readable output.

Nothing completely interesting however everything is extremely useful. I’m cutting my keystrokes in half and getting better output. I have a bunch of more exciting tricks in the rest of my ~/.bashrc. Its not just reserved for aliases and EXPORTS, take this function for example:

# cd directly to a dir and list contents
cdl() {
  if [ "$1" ]
  then builtin cd "$1" && ll
  else builtin cd && ll
  fi
}

Cd to a directory and list the directory. I even reference my ll alias up above to list long. You can take a peek at my entire .bashrc file on dotfiles.org. Lets move on to something you may not have known about. Let me walk you through parts of a nice ~/.irbrc file, which you might want for yourself:

# Load and Start Wirble, a gem to beautify irb
require 'wirble'
Wirble.init
Wirble.colorize

Probably one of the most popular inclusions in a .irbrc file. Wirble is a Ruby gem built for improving the irb interactive ruby console. That means you may also have to ‘require “rubygems”‘ in order for this to work. Take a look at the gem documentation for Wirble to find out more of its capabilities. But that also means you can include a number of other useful rubygems. For instance, what_methods, map_by_method, hpricot, yaml, the list goes on and on. Instead I’m going to point out some other neat additions you can add to your .irbrc!

# Awesome benchmarking function
# Source: http://ozmm.org/posts/time_in_irb.html
def time(times=1)
  require "benchmark"
  ret = nil
  Benchmark.bm { |x| x.report { times.times { ret = yield } } }
  ret
end
alias bench time

# A cool way to index in a hash
# h = { :alpha => 'bet', :beta => 'blocker' }
# h/:beta #=> 'blocker'
class Hash
  def /(key)
    self[key]
  end
end

# Simple regular expression helper
# show_regexp - stolen from the pickaxe
def show_regexp(a, re)
  if a =~ re
    "#{$`}<<#{$&}>>#{$'}"
  else
    "no match"
  end
end

# Convenience method on Regexp so you can do
# /an/.show_match("banana") # => "b<>ana"
class Regexp
  def show_match(a)
    show_regexp(a, self)
  end
end

There, now you’re building some really useful tricks! Check it out. Thanks to a few different sources for those code snippets I can easily benchmark any code run at any number of times. There is a neat way to pull an element from a hash, without having to put [brackets] around the key. Finally, a little helper for regular expressions which is useful every now and then.

But wait, there is more. This time a little more system specific:

# Textmate helper
# Source: http://dotfiles.org/~lattice/.irbrc
def mate *args
  flattened_args = args.map {|arg| "\"#{arg.to_s}\""}.join ' '
  `mate #{flattened_args}`
  nil
end

# Clear
def c
  system('clear')
end

You can now open up TextMate from within irb! I’ve also grown so used to using my ‘c’ alias to clear the terminal prompt that I added the same functionality to irb. Just make sure that you don’t name a variable c!

Now let me blow your mind:

# Why's aorta method to edit an object in YAML, awesome!
# Source: http://rubyforge.org/snippet/detail.php?type=snippet&id=22
require 'yaml'
def aorta( obj )
  tempfile = File.join('/tmp',"yobj_#{ Time.now.to_i }")
  File.open( tempfile, 'w' ) { |f| f << obj.to_yaml }
  system( "#{ ENV['EDITOR'] || 'vi' } #{ tempfile }" )
  return obj unless File.exists?( tempfile )
  content = YAML::load( File.open( tempfile ) )
  File.delete( tempfile )
  content
end

Why is a famous member of the Ruby community. This function here actually takes a Ruby object, exports it as YAML to a file, opens the File for editing, and once saved reloads the file from YAML. Essentially it allows you to edit the contents of an object in YAML. Absolutely amazing, let that stir in your mind for a minute!

Now, I have started building my own library of functions that I have deemed semi-useful but not worthy of turning into a gem. These functions include some simple String extensions like my TXT helpers. I include all these files in a directory and I auto-load them like so:

# Load all my non-test libraries in '~/.util/irb'
util_dir = File.expand_path('~') + '/.util/irb/*'
Dir[util_dir].each do |f|
  require f unless File.basename(f) =~ /\Atest/
end

Notice how I ignore any files starting with “test”. I have gotten into the habit of creating test files for my libraries and the naming convention I use is the exact same as the majority of Ruby developers, you just create a new file called “test_library.rb” to test “library.rb”. Simple yes, and helpful in this case where I want to avoid loading these test files. This means I have all my useful functions pre-loaded whenever I open the irb. I’d like to see what additional stuff you have!

Finally here are links directly to my .bashrc and .irbrc files. Enjoy, and please give me additions!

search