Articles with tag ‘unix‘

 
 

Command Line svn:ignore a file

This took me far too long to do, but its because no one explained it correctly to me. I’m frustrated enough that I’m not going to go back to those other websites and see if I just overlooked something. Instead I’m going to put it right here.

You don’t svn:ignore a file. You put an svn:ignore property on the directory to ignore that filename pattern.

It makes sense, but I didn’t immediately think of that, and no source sufficiently made that point to me. So if you were struggling, please grasp that concept and take a look at the commands below (which you no doubt have seen and did not think worked) and you’ll really understand what they do.

# ---------------------------------------------------------------------
#  Ignore all the .wdgtuser files in the /trunk/Blah.dcproj directory
# ---------------------------------------------------------------------

# Go to the directory
cd trunk/Blah.dcproj/     # The directory with the files

# Start editing the properties for the current directory
svn propedit svn:ignore .   # Opens an editor (SVN_EDITOR, EDITOR)

# Add the following value with a new line, save, and exit:
*.wdgtuser

# See that things worked
svn propget svn:ignore .    # So you can see the properties
svn status --no-ignore      # You should see an 'I' next to the ignored files

# Commit
svn commit -m "New Ignores" # You must commit the new property change


# ---------------------------------------------------------------------
#     Ignore a single file secret.txt in the /trunk/ directory
# ---------------------------------------------------------------------

# Go to the directory
cd trunk/

# Add just the single file to the current directories ignore list (like above)
# Note the dot at the end of the command is important
svn propset svn:ignore secret.txt .

# See that things worked
svn propget svn:ignore .    # Notice the single file was added to the list
svn status --no-ignore      # You should see an 'I' next to the ignored files

# Commit
svn commit -m "Its secret"  # You must commit the new property change

That also means in GUI programs if you can’t seem to ignore a single file that is unversioned you should instead go to the directory that file is in and (like the above) add the filename to the svn:ignore list.

Cheers.

Dot Files For Your Shell and Even Ruby

I have come across a number of programmers who don’t know what dotfiles are. Thats a shame. Every programmer should know common dotfiles and actively seek to add aliases, functions, and other tidbits to increase their productivity and make the shell more usable. I recently went on a binge and updated a few of my dotfiles. I’ll share some useful tricks that I found on not only my ~/.bashrc file but also ~/.irbrc for my ruby IRB prompt!

I won’t show off my entire ~/.bashrc file, only because it is rather long. I just want to get across some of the usefulness of such a file:

# -----------
#   General
# -----------
alias ..='cd ..'
alias ll='ls -lh'
alias la='ls -la'
alias ps='ps -ax'
alias du='du -hc'
alias cd..='cd ..'
alias more='less'
alias mkdir='mkdir -p'
alias today='date +"%A, %B %d, %Y"'
alias yest='date -v-1d +"%A %B %d, %Y"'
alias recent='ls -lAt | head'
alias ebashrc='mate ~/.bashrc'
alias mbashrc='mate ~/.bashrc'
alias sbashrc='source ~/.bashrc'
alias htdocs='cd /Applications/MAMP/htdocs/'
alias mampmysql='/Applications/MAMP/Library/bin/mysql -u XXXXX -p'
alias desktoptopia='open /Users/joe/Library/Application\ Support/Desktoptopia/.Backgrounds/'
alias ql='qlmanage -p "$@" >& /dev/null' # Quick Look alias

# -------------
#   Shortcuts
# -------------
alias c="clear"
alias m="mate"

# --------
#   SSHs
# --------
alias rit="ssh holly.cs.rit.edu -l XXXXX"
alias vega="ssh vega.it.rit.edu -l XXXXX"

# -------
#   Git
# -------
alias ga='git add'
alias gs='git status'
alias gd='git diff'
alias github="open \`git config -l | grep 'remote.origin.url' | sed -En 's/remote.origin.url=git(@|:\/\/)github.com(:|\/)(.+)\/(.+).git/https:\/\/github.com\/\3\/\4/p'\`"

# --------
#   Ruby
# --------
alias irb='irb -r irb/completion -rubygems'

# ---------------
#   Environment
# ---------------
export PATH="$PATH:/usr/local/bin:/usr/local/sbin:/usr/local/mysql/bin"
export PATH="$HOME/bin/:$PATH"
export HISTSIZE=10000
export HISTFILESIZE=10000
export PAGER=less
export CLICOLOR=1
export EDITOR="/usr/bin/mate -w"

Hopefully this wasn’t too overwhelming. But lets take a look at some of these. There are a bunch of aliases at the top which simply replace the older version? It just makes sense that when you do a ps you really want `ps -ax`. Likewise a few others there are printing disk usage with human readable output.

Nothing completely interesting however everything is extremely useful. I’m cutting my keystrokes in half and getting better output. I have a bunch of more exciting tricks in the rest of my ~/.bashrc. Its not just reserved for aliases and EXPORTS, take this function for example:

# cd directly to a dir and list contents
cdl() {
  if [ "$1" ]
  then builtin cd "$1" && ll
  else builtin cd && ll
  fi
}

Cd to a directory and list the directory. I even reference my ll alias up above to list long. You can take a peek at my entire .bashrc file on dotfiles.org. Lets move on to something you may not have known about. Let me walk you through parts of a nice ~/.irbrc file, which you might want for yourself:

# Load and Start Wirble, a gem to beautify irb
require 'wirble'
Wirble.init
Wirble.colorize

Probably one of the most popular inclusions in a .irbrc file. Wirble is a Ruby gem built for improving the irb interactive ruby console. That means you may also have to ‘require “rubygems”‘ in order for this to work. Take a look at the gem documentation for Wirble to find out more of its capabilities. But that also means you can include a number of other useful rubygems. For instance, what_methods, map_by_method, hpricot, yaml, the list goes on and on. Instead I’m going to point out some other neat additions you can add to your .irbrc!

# Awesome benchmarking function
# Source: http://ozmm.org/posts/time_in_irb.html
def time(times=1)
  require "benchmark"
  ret = nil
  Benchmark.bm { |x| x.report { times.times { ret = yield } } }
  ret
end
alias bench time

# A cool way to index in a hash
# h = { :alpha => 'bet', :beta => 'blocker' }
# h/:beta #=> 'blocker'
class Hash
  def /(key)
    self[key]
  end
end

# Simple regular expression helper
# show_regexp - stolen from the pickaxe
def show_regexp(a, re)
  if a =~ re
    "#{$`}<<#{$&}>>#{$'}"
  else
    "no match"
  end
end

# Convenience method on Regexp so you can do
# /an/.show_match("banana") # => "b<>ana"
class Regexp
  def show_match(a)
    show_regexp(a, self)
  end
end

There, now you’re building some really useful tricks! Check it out. Thanks to a few different sources for those code snippets I can easily benchmark any code run at any number of times. There is a neat way to pull an element from a hash, without having to put [brackets] around the key. Finally, a little helper for regular expressions which is useful every now and then.

But wait, there is more. This time a little more system specific:

# Textmate helper
# Source: http://dotfiles.org/~lattice/.irbrc
def mate *args
  flattened_args = args.map {|arg| "\"#{arg.to_s}\""}.join ' '
  `mate #{flattened_args}`
  nil
end

# Clear
def c
  system('clear')
end

You can now open up TextMate from within irb! I’ve also grown so used to using my ‘c’ alias to clear the terminal prompt that I added the same functionality to irb. Just make sure that you don’t name a variable c!

Now let me blow your mind:

# Why's aorta method to edit an object in YAML, awesome!
# Source: http://rubyforge.org/snippet/detail.php?type=snippet&id=22
require 'yaml'
def aorta( obj )
  tempfile = File.join('/tmp',"yobj_#{ Time.now.to_i }")
  File.open( tempfile, 'w' ) { |f| f << obj.to_yaml }
  system( "#{ ENV['EDITOR'] || 'vi' } #{ tempfile }" )
  return obj unless File.exists?( tempfile )
  content = YAML::load( File.open( tempfile ) )
  File.delete( tempfile )
  content
end

Why is a famous member of the Ruby community. This function here actually takes a Ruby object, exports it as YAML to a file, opens the File for editing, and once saved reloads the file from YAML. Essentially it allows you to edit the contents of an object in YAML. Absolutely amazing, let that stir in your mind for a minute!

Now, I have started building my own library of functions that I have deemed semi-useful but not worthy of turning into a gem. These functions include some simple String extensions like my TXT helpers. I include all these files in a directory and I auto-load them like so:

# Load all my non-test libraries in '~/.util/irb'
util_dir = File.expand_path('~') + '/.util/irb/*'
Dir[util_dir].each do |f|
  require f unless File.basename(f) =~ /\Atest/
end

Notice how I ignore any files starting with “test”. I have gotten into the habit of creating test files for my libraries and the naming convention I use is the exact same as the majority of Ruby developers, you just create a new file called “test_library.rb” to test “library.rb”. Simple yes, and helpful in this case where I want to avoid loading these test files. This means I have all my useful functions pre-loaded whenever I open the irb. I’d like to see what additional stuff you have!

Finally here are links directly to my .bashrc and .irbrc files. Enjoy, and please give me additions!

mgrep - Multiple Regex Grep - SotD

My newest Ruby utility is called mgrep. It is a multiple regular expression version of grep. Input is processed line by line and each regular expression is said by the user to match or not match. All lines that meet the user’s desires for matching/non-matching regular expressions are printed in the following format: “filename [line number]: line of text.”

This can be taken in a number of different directions. I’m thinking “Partial Matches” meaning one regular expression matches one line in the file and eventually a second regular expression matches a totally different line and if all of these conditions succeed by the end of the file then print out the filename. This sounds more useful and will most likely be in version 1.0.

Here is the current usage:

usage: mgrep [-#] ( [-n] regex ) [filenames]
  #         - the number of regular expressions, defaults to 1
  ( ... )   - there should be # of these
  regex     - regular expessions to be checked on the line
  filenames - names of the input files to be parsed, if blank uses STDIN

options:
  --neg, --not, or -n    line must not match this regular expression

special note:
  When using bash, if you want backslashs in the replace portion make sure
  to use the multiple argument usage with single quotes for the replacement.

The usage is a little confusing seeing as the number of regular expressions on the command line are variable based on another command line switch. All in all though it is rather clear. Options will likely come in the future, much like grep or awk if I get around to it.

Here is an example probably not useful but at least it shows functionality, the line must contain a number, a double letter, and end with a !: [input]

1 ab !
- aa !
1 aa -
1 aa !
oh yah! 1

And when I run my script, I’ll put all the regular expressions in /here/ to make it clearer, this syntax is allowed by mgrep for convenience. Here is what it looks like:
joe[~/sandbox]$ mgrep -3 ‘/\\d/’ ‘/(\\w)\\1/’ ‘/!$/’ input
input [4]: 1 aa !

To show of the –neg or -n option this command will show all the lines that do not have a hypen and still end with a !:
joe[~/sandbox]$ mgrep -2 -n ‘/-/’ ‘/!$/’ input
input [1]: 1 ab !
input [4]: 1 aa !

The script surely be updated soon, but grab it now and try it out:
mgrep - Most Recent Version - Download
mgrep - changelog

rr - 1.1 - In Place Edits and Multiple Files

Less then 48 hours after rr becomes 1.0 it gets a few very handy improvements!

In place modification of files is activated via the –modify (or shorthand -m) option. This means that you can bypass any output redirection and just go straight to modifying the original file. This feature does use filename.tmp as a temp file which it later renames to the original filename. Again if no filenames are specified then input is expected to come from STDIN and therefore the new –modify option will be ignored in this special case.

Another original goal of mine was adding support for multiple filenames. Specifically so that useful shell tricks like *.txt file globbing would work nicely with rr. Well support has been added and it works great with the new –modify option.

The usage message has been cleaned up a bit but here is the very basic usage for all new people.

usage: rr [options] find replace [filenames]
       rr [options] s/find/replace/ [filenames]

I wanted to point out a rather hidden feature. The way I implemented the options is that the ARGV array is actually parsed first for all options and then removes the options before going on to parse the find, replace, and filename arguments. This means that your options can go anywhere on the command line so long as they start with a -.

This presents 1 problem, a workaround, and a question for users. Using the second form of usage, where the find and replace portions are separate argument if your regex or replacement text starts with a “-” the script will interpret it as an option. You can avoid this by using the s/find/replace/ usage (or putting the regex in /regex/ format, which is allowed). But really this boils down to deciding whether or not I am being too liberal with my command line arguments. Since this is a very big fringe condition with a workaround I am going to allow options to be placed anywhere, allowing you to bring up the last command in bash with the up arrow and adding an option to the end of your rr command (like the new -m) to repeat your last command with an option much easier.

rr is always free, Try It Out:
rr - Current Version Download
rr - changelog.txt - Click Here

$ gem install regex_replace

rr - 1.0 - Now a Pipe Friendly Filter

rr has reached the 1.0 milestone! The obvious improvement over the last version is that input is allowed from standard input. It seemed silly to always require a filename and the option of having standard input was always on my to do list. Usage is now:

usage: rr [options] find replace [filename]
       rr [options] s/find/replace/ [filename]

Now you can use rr as a filter and happily make find replace changes by piping input into it or out of it! I already have a script that runs a file through 4 rr commands to produce much nicer and cleaner output. Wrap that up in a shell/ruby/perl script and you have a useful tool.

Enjoy. Again its all free!
rr - Current Version Download
rr - changelog.txt - Click Here

$ gem install regex_replace

rr - Updated to 0.9.1

rr now has some improvements, including a new style of usage. Both this new style and the original style usage are available to you.

rr [options] s/find/replace/ filename

The new s/find/replace/ syntax is still weak with respect to the forward slash character in either the find or replace, but works for everything else so far. Of course if you want to include any whitespace then you should wrap the entire argument in quotes. Also, because the strings are coming from the command line, if you want to have literal backslashes then use single quotes around your string so the shell doesn’t escape them itself before sending it to Ruby.

Another highlight is that all escape sequences should now work. That means your typical \n, \t, and all the obscure even including \a (system bell). Check out this example, you will hear two system bells once this has been run:
$ echo "aba" > in.1; rr a "\a" in.1; rm in.1

Also I was considering renaming to fr for “Find/Replace” however I am keeping rr. rr can be interpreted as “Run Regex” with the s/find/replace/ syntax, or “Regex Replace” for the normal 3 argument usage. If you like fr you can easily make an alias like so: Want to know how to always load the alias?
$ alias fr="rr"

So have fun, of course everything is free and available right here:
rr - Current Version Download
rr - changelog.txt - Click Here

rr - Regex Replace on a File - SotD

I was frustrated with regular expression find/replace programs that only did line processing. This was because often I had find/replace needs that spanned multiple lines. Programs like grep, ack (which I recently found and is really, really very awesome for searching code), and sed were easy enough to use for basic needs. But again, when it came to multiple line pattern matching both fell short of my needs.

My solution was to write my own script to parse an entire file as a single string and do my find/replace bidding. The cons being liberal use of memory and a few hundredths of a second longer then the usual find/replace algorithms seemed insignificant to the pros of a multi-line capable find/replace using a regular expression with the capability of using back references (like \1) to incorporate captured groups from the regex into the replacement text.

So, without further ado I present rr.

I am hopeful for some public criticism to help me bring rr up from its current version of 0.9 to a landmark 1.0. The ruby script weighs in at 100 lines but really under 50 are code and the rest is comments, whitespace, or the usage string. Speaking of usage, here is what it currently [v0.9.0] looks like:

usage: rr [options] find replace filename
  find     - a regular expression to be run on the entire file as one string
  replace  - replacement text, \1-\9 and \n are allowed
  filename - name of the input file to be parsed

options:
  --line or -l    process line by line instead of all at once (not default)
  --case or -c    makes the regular expression case sensitive (not default)
  --global or -g  process all occurrences by default (this is already default)

negated options are done by adding 'not' or 'n' in switches like so:
  --notline or -nl

example usage:
  The following takes a file and doubles the last character on each line
  and turns the newlines into two newlines.
  rr "(.)\\n" "\\1\\1\\n\\n" file

More then likely this will undergo a lot of changes. A quick list of my current ideas include:

  1. If no filename is provided take input from STDIN. Multiple files can be handled by piping the `cat` of multiple files through rr.
  2. Better switch structure, although right now I don’t have any idea what that is

I’ll throw a test scenario at you. I had tabulated data in a file but each row was split across multiple lines. Now this wasn’t the only data in the file but I’ll present you with a simplier version here: [in.1]


Product A  12.99
           2001
----
Product B   1.99
           1997

Here you can see that I can’t just replace every other newline. What I want to actually do is replace newlines where there was a digit followed by a newline, some whitespace and another digit. I ran this through my script:

> rr "(\d)\n\s+(\d)" "\1 \2" in.1 > out.1

And I got the output I wanted: [out.1]


Product A  12.99  2001
----
Product B   1.99  1997

Even cleaner results can be seen by running a more advanced regex to remove the extra lines:


> rr "(\\d)\\n\\s+(\\d.*?\\n)(-+\\n)?" "\\1  \\2" in.1
Product A  12.99  2001
Product B   1.99  1997

So what are you waiting for? Download the script, add it to your bin directory, give it a test run, and tell me how you want it improved!

rr - Most Recent Version - Download

Thanks!


Recent Resources

Clicky Web Analytics