Check if Your Ruby Script is in a Pipe

Short and simple this week. When you’re writing a Ruby script you might want to know if you’re in a pipe or not. One reason might be changing how you buffer your input/output. A number of basic Unix commands do this, they act differently when piped.

I actually wrote a script where I wanted to change the command line arguments if I was in a pipe. This is probably confusing but its really useful! So there is actually an easy way to do this in Ruby:

#!/usr/bin/env ruby
# Prints if the input or output is regular or piped
puts "INPUT:  #{STDIN.tty?  ? "regular" : "pipe"}"
puts "OUTPUT: #{STDOUT.tty? ? "regular" : "pipe"}"

Proof that it works as advertised:

IO#tty?

The documentation on IO#tty? says the following:

Returns true if [the IO stream] is associated with a terminal device (tty), false otherwise.

For those that don’t know, tty is short for teletype writer. This is an old Unix term for an interactive process that takes in user input. Nowadays it almost always means a shell/console/terminal.

Mini MP3 Searching Shell – Skreemr

So its way to hard to download an mp3 in Safari. Right click the link and download? Pff, I want to ⌃S and be done with it. Well, this time I decided to avoid the problem all together. I use Skreemr to search for a particular song when it interests me.

In the past I wrote a little bash script, that makes use of curl, to download an mp3 to my desktop unique named so it wouldn’t have conflicts. This shell essentially wraps and drastically improves that to allow for searching, pagination, history, downloading, and opening mp3s off of Skreemr. It gives me just what I need. The functionality that I want without having to use torrents etc. I’m thinking of turning this into a gem.

skreemr.png

This script requires the popular “escape.rb” script that gives some nice and safe shell escaping functions. You can download both from my GitHub scripts project.

Of course its available on my ~/bin and there will be another article later on that goes over a few aspects of this simple little script.

skreemr2.png

Dynamic Web URLs with ExpanDrive

Often when I work with ExpanDrive the files I am working on correspond to some website that I own. When I’m mounted with ExpanDrive each file is accessible via “my hard drive” in the mounted volume and, more importantly, from a web URL! I found myself repeatedly opening up my browser and manually typing the URL for files that I just uploaded or edited. This was error prone, especially if some of the characters needed encoding. So, I spent some time to write up a Ruby script that can read ExpanDrive’s preferences, build the file’s “web URL,” and open it in your default browser.

expanurl usage

Simple to Use

I followed along with ExpanDrive’s previous command line tool named expan and named my script expanurl. Given no arguments it will open the current directory via its web URL, or you can give a list of files and each will be opened at their web URLs. Its usage is pretty straightforward but there is a single catch: the server setting in an ExpanDrive Drive may not be a true one-to-one mapping with the web server’s address.

For example: I provide holly.cs.rit.edu as the server value for one of my personal ExpanDrive drives. However, when I view files on that server (inside the public_html directory) they have a much different looking URL: http://www.cs.rit.edu/~jjp1820/. The result? The script simply keeps its own mapping of ExpanDrive server values to their associated web page prefixes. When you use expanurl on a Drive you have never used it on before the script will prompt you for that mapping, store the value, and never ask again.

Here it is in action. I have removed all stored mappings so I can demonstrate what it would be like using expanurl for the first time. Here I use it on my BogoJoker ExpanDrive drive:

expanurl first usage

Notice that in the prompt it tells you:

  • the server that the ExpanDrive volume is linked to and the one you will be providing a web url prefix for
  • an example of a web url prefix (useful)
  • where the mappings are stored in case you need to edit them later

The script is available on GitHub, so feel free to contribute and improve. Here is a link to the always current version, and here is a snapshot of the current version at the time of writing:

 

#!/usr/bin/env ruby
# Author: Joseph Pecoraro
# Date: Saturday December 13, 2008
# Description: When I'm using ExpanDrive and I'm
# remotely logged into a server, I can use this
# script to "open filename" and it will open using
# the server's associated URL.

# For URL Escaping and Stored Mappings
require 'uri'
require 'yaml'

# Exit with a msg
def err(msg)
  puts msg
  exit 1
end

# Simple Class to handle the mappings
class UrlMap
  
  MAP_PATH = File.expand_path('~/.expanurl')
  
  def initialize
    @hash = load
  end
  
  def load
    if File.exists?(MAP_PATH)
      YAML::load_file( File.expand_path(MAP_PATH) )
    else
      Hash.new
    end
  end
  
  def add_mapping(server, mapto)
    @hash[server] = mapto
    File.open(MAP_PATH, 'w') do |file|
      file.write(@hash.to_yaml)
    end
  end
  
  def is_mapping?(server)
    @hash.has_key?(server)
  end
  
  def get_mapping(server)
    @hash[server]
  end
  
  def path
    MAP_PATH
  end
  
end


# Local Variables
mapping = UrlMap.new
url_prefix = nil
server = nil
volume = nil

# Check the if the current directory is an
# ExpanDrive Volume and a public_html folder
pwd = `pwd`
match = pwd.match(/^\/Volumes\/([^\/]+)/)
if match.nil?
  err("Not inside an ExpanDrive Volume")
elsif !pwd.match(/\/public_html\/?/)
  err("Not inside a public_html directory.")
else
  volume = match[1]
  defaults = `defaults read com.magnetk.ExpanDrive Drives`
  defaults.gsub!(/\n/, '')
  props = defaults.match(/\{[^\}]+driveName\s+=\s+#{volume}[^\}]+server\s+=\s+"([^"]+)"[^\}]+\}/)
  if props
    server = props[1]
  else
    err("This Volume (#{volume}) is not an ExpanDrive Volume")
  end
end

# Check if a mapping exists
# Otherwise create and store one
if mapping.is_mapping?(server)
  url_prefix = mapping.get_mapping(server)
else
  
  # Prompt
  puts
  puts "This is the first time you've used expanurl for #{volume}"
  puts "Please Provide us with a mapping for #{server}"
  puts "Mappings are stored in #{mapping.path}"
  puts "Example: http://bogojoker.com/"
  print ">> "
  
  # Store user input and proceed
  url_prefix = gets.chomp
  url_prefix += '/' unless url_prefix.match(/\/$/)
  mapping.add_mapping(server, url_prefix)
  
  # Terminal Output
  puts
  puts "Server: #{server}"
  puts "Maps to: #{url_prefix}"
  puts

end

# Build the URL
subpath = pwd.match(/public_html\/?(.*)/)[1]
subpath += '/' unless subpath.length.zero? || subpath.match(/\/$/)
url_prefix += subpath

# If No Files, open the directory
# Otherwise,   open each provided file
if ARGV.size == 0
  `open #{url_prefix}`
else
  ARGV.each do |filename|
    `open #{url_prefix}#{URI.escape(filename)}`
  end
end

 

How it Works

The Ruby Script grabs the current working directory using `pwd` and checks to make sure you’re in an ExpanDrive volume. ExpanDrive volume’s are dynamically generated by parsing the ExpanDrive preferences thanks to their foresight to make them accessible via the `defaults` command. So if you’re in an ExpanDrive volume and inside a public_html directory expanurl will then use its mapping to open a uri encoded web url in your default browser with the `open` command.

The mappings are stored in a hidden YAML file in your home directory (~/.expanurl). This style of storing preferences is just like dozens of other command line applications and scripts. YAML is just a lightweight textual data format popular with Ruby, similar to JSON and XML. Its so simple that you could edit the file yourself if you wanted/needed to. For instance here is what is in mine, just two simple key/value pairs:

~/.expanurl yaml mapping

The Future

Its just that simple. Being a Ruby Script you can call this from GUI applications, anything with built-in shell access, etc. It should play friendly with your usual Unix tools. I will likely make this script more and more robust if others find it useful, so I’d be happy to hear some feedback.

Cheers.

mgrep – Multiple Regex Grep – SotD

My newest Ruby utility is called mgrep. It is a multiple regular expression version of grep. Input is processed line by line and each regular expression is said by the user to match or not match. All lines that meet the user’s desires for matching/non-matching regular expressions are printed in the following format: “filename [line number]: line of text.”

This can be taken in a number of different directions. I’m thinking “Partial Matches” meaning one regular expression matches one line in the file and eventually a second regular expression matches a totally different line and if all of these conditions succeed by the end of the file then print out the filename. This sounds more useful and will most likely be in version 1.0.

Here is the current usage:

usage: mgrep [-#] ( [-n] regex ) [filenames]
  #         - the number of regular expressions, defaults to 1
  ( ... )   - there should be # of these
  regex     - regular expessions to be checked on the line
  filenames - names of the input files to be parsed, if blank uses STDIN

options:
  --neg, --not, or -n    line must not match this regular expression

special note:
  When using bash, if you want backslashs in the replace portion make sure
  to use the multiple argument usage with single quotes for the replacement.

The usage is a little confusing seeing as the number of regular expressions on the command line are variable based on another command line switch. All in all though it is rather clear. Options will likely come in the future, much like grep or awk if I get around to it.

Here is an example probably not useful but at least it shows functionality, the line must contain a number, a double letter, and end with a !: [input]

1 ab !
- aa !
1 aa -
1 aa !
oh yah! 1

And when I run my script, I’ll put all the regular expressions in /here/ to make it clearer, this syntax is allowed by mgrep for convenience. Here is what it looks like:

joe[~/sandbox]$ mgrep -3 '/\\d/' '/(\\w)\\1/' '/!$/' input
input [4]: 1 aa !

To show of the –neg or -n option this command will show all the lines that do not have a hypen and still end with a !:

joe[~/sandbox]$ mgrep -2 -n '/-/' '/!$/' input
input [1]: 1 ab !
input [4]: 1 aa !

The script surely be updated soon, but grab it now and try it out:
mgrep – Most Recent Version – Download
mgrep – changelog

rr – 1.1 – In Place Edits and Multiple Files

Less then 48 hours after rr becomes 1.0 it gets a few very handy improvements!

In place modification of files is activated via the –modify (or shorthand -m) option. This means that you can bypass any output redirection and just go straight to modifying the original file. This feature does use filename.tmp as a temp file which it later renames to the original filename. Again if no filenames are specified then input is expected to come from STDIN and therefore the new –modify option will be ignored in this special case.

Another original goal of mine was adding support for multiple filenames. Specifically so that useful shell tricks like *.txt file globbing would work nicely with rr. Well support has been added and it works great with the new –modify option.

The usage message has been cleaned up a bit but here is the very basic usage for all new people.

usage: rr [options] find replace [filenames]
       rr [options] s/find/replace/ [filenames]

I wanted to point out a rather hidden feature. The way I implemented the options is that the ARGV array is actually parsed first for all options and then removes the options before going on to parse the find, replace, and filename arguments. This means that your options can go anywhere on the command line so long as they start with a -.

This presents 1 problem, a workaround, and a question for users. Using the second form of usage, where the find and replace portions are separate argument if your regex or replacement text starts with a “-” the script will interpret it as an option. You can avoid this by using the s/find/replace/ usage (or putting the regex in /regex/ format, which is allowed). But really this boils down to deciding whether or not I am being too liberal with my command line arguments. Since this is a very big fringe condition with a workaround I am going to allow options to be placed anywhere, allowing you to bring up the last command in bash with the up arrow and adding an option to the end of your rr command (like the new -m) to repeat your last command with an option much easier.

rr is always free, Try It Out:
rr – Current Version Download
rr – changelog.txt – Click Here

$ gem install regex_replace

rr – Regex Replace on a File – SotD

I was frustrated with regular expression find/replace programs that only did line processing. This was because often I had find/replace needs that spanned multiple lines. Programs like grep, ack (which I recently found and is really, really very awesome for searching code), and sed were easy enough to use for basic needs. But again, when it came to multiple line pattern matching both fell short of my needs.

My solution was to write my own script to parse an entire file as a single string and do my find/replace bidding. The cons being liberal use of memory and a few hundredths of a second longer then the usual find/replace algorithms seemed insignificant to the pros of a multi-line capable find/replace using a regular expression with the capability of using back references (like \1) to incorporate captured groups from the regex into the replacement text.

So, without further ado I present rr.

I am hopeful for some public criticism to help me bring rr up from its current version of 0.9 to a landmark 1.0. The ruby script weighs in at 100 lines but really under 50 are code and the rest is comments, whitespace, or the usage string. Speaking of usage, here is what it currently [v0.9.0] looks like:

usage: rr [options] find replace filename
  find     - a regular expression to be run on the entire file as one string
  replace  - replacement text, \1-\9 and \n are allowed
  filename - name of the input file to be parsed

options: --line or -l process line by line instead of all at once (not default) --case or -c makes the regular expression case sensitive (not default) --global or -g process all occurrences by default (this is already default)
negated options are done by adding 'not' or 'n' in switches like so: --notline or -nl
example usage: The following takes a file and doubles the last character on each line and turns the newlines into two newlines. rr "(.)\\n" "\\1\\1\\n\\n" file

More then likely this will undergo a lot of changes. A quick list of my current ideas include:

  1. If no filename is provided take input from STDIN. Multiple files can be handled by piping the `cat` of multiple files through rr.
  2. Better switch structure, although right now I don’t have any idea what that is

I’ll throw a test scenario at you. I had tabulated data in a file but each row was split across multiple lines. Now this wasn’t the only data in the file but I’ll present you with a simplier version here: [in.1]

Product A  12.99
           2001
----
Product B   1.99
           1997

Here you can see that I can’t just replace every other newline. What I want to actually do is replace newlines where there was a digit followed by a newline, some whitespace and another digit. I ran this through my script:

> rr "(\d)\n\s+(\d)" "\1  \2" in.1 > out.1

And I got the output I wanted: [out.1]

Product A  12.99  2001
----
Product B   1.99  1997

Even cleaner results can be seen by running a more advanced regex to remove the extra lines:

> rr "(\\d)\\n\\s+(\\d.*?\\n)(-+\\n)?" "\\1  \\2" in.1
Product A  12.99  2001
Product B   1.99  1997

So what are you waiting for? Download the script, add it to your bin directory, give it a test run, and tell me how you want it improved!

rr – Most Recent Version – Download

Thanks!

A Unique Unix Tutorial

Thats right. You’ve all read tutorials. They range all over the place: boring, brief, detailed, useless, inspiring, the list goes on. There are those that are too technical, that you don’t understand until you look back at them at a later date. In the case of Unix and Linux the majority of the tutorials are like that. They are just reworded man pages. Many people turn away from *nix because of this gap in technical familiarity. My Unix tutorial aims to change this.

I have read a number of tutorials and I find the best are the ones that include you, the reader, in them. You are participating in the tutorial. Often there is a story, or some creative aspect that brings you into the tutorial. You are learning in an entertaining way. For first timers this can make all the difference.

My tutorial also gives tips and tricks for using the terminal. They are the kind of tricks that make using the console much easier, but they are the tricks that you often don’t know unless someone shows you. I hope that my tutorial at least gives insight to newer Unix/Linux users as they take the leap into a new realm.

Please take a look and offer your feedback to help me improve the tutorial. I now present my Unix Tutorial.

I would like to point out that the tutorial happened to be a college project that I recently decided to turn into a reality. I have been and will be refurnishing the tutorial with improvements (sIFR), more rich content, while still maintaining my original goals and objectives.

Inspiration: A rather stimulating Ruby tutorial that wraps you inside a rather creative story, with “synergy and cartoon foxes.”

search