Geek Resources

Go ahead and skip right to the examples if you’re in a hurry

Got Regexes? Regular Expressions (also known as regexes, regexen, and regexps) open up a world of new power tools to you when you need to automate text analysis or manipulations such as massive search and replace operations in one or multiple directories. Such a task oft befalls the sysadmin/developer/commandline ninja.

So case in point: If you find yourself needing to do some file correction in place without the hassle of sed’s inferior pattern matching and all the limitations that go with it, whilst neither feeling the inclination to concoct a convoluted awk block, nor even more a desire to brazenly undertake the time-consuming production of a full blown python script–just to wrap up in a python application the kind of functionality that is just as well performed with a Perl one-liner—— well my friends you’ve come to the right place.

Indeed Perl provides you with in-line search and replace using real regular expressions in all their glory, and in the world of text processing, Perl is king. (Sorry haters/ruby-fanboys/shell-purists, it’s just true!)

So come now, let’s put down the tar, feathers, torches and pitchforks for a moment and consider the following super-cool example (which even creates backup copies of the files you changed, in the event that you flubbed up your regex):

perl -p -i'.backup' -e 's/(?<=replace )this(?= lolc[aA@]t)(?# with )/that/; s/(?# and as for all of the )([[:digit:]]+) (?i:fat|skinny|thirsty) camels(?# you really need those)/$1 white llamas/g;' file1 file2 /path/to/file3

The above powerful search and replace shows you a hefty example of how Perl regexes can be used in your search and replace operation, and the example uses some pretty useful and advanced matching tools you just can’t readily get elsewhere; among other features, the example uses positive look-behind zero-width assertions, positive look-ahead zero-width assertions, atomic sub-match capturing, POSIX character classes, and in-line zero-width comments that make the regexes more readable. In the example, Perl runs two pattern substitutions on every line in each of the three files, and what’s more, it makes backup copies for you!

A more simple version without backups would be:

perl -p -i -e 's/replace this/using that/g' /all/text/files/in/*.txt

The example above replaces any occurrence of the string “replace this” with the string “using that” on all text files inside the directory name given.

So in summary, if you want to use the most powerful search and replace tools on the command line, and do it in the easiest form, use perl -p -i -e 'pattern' file and use it wisely.

…And by “wisely”, I mean it would be wise to brush up on your regular expression fu for free by reading the Perl documentation about its incredible regular expressions.

Please feel free to experiment with the following ~/.bash_profile goodness of my own, and save away to yours what tasty morsels that you find useful.

Exactly what each line does below is left as an exercise to the reader   Hint: none of these aliases/functions/commands are malicious!

export HISTTIMEFORMAT='%F %T '
export HISTCONTROL=ignoredups
export HISTCONTROL=ignoreboth
export HISTIGNORE='pwd:ls:history:'
export HISTSIZE=4096
export EDITOR='/usr/bin/vim'
export AUTOSSH_POLL=30
eval `dircolors`
alias ls='ls --color=auto'
alias dir='ls --color=auto --format=vertical'
alias ll='ls -Al'
alias la='ls -A'
alias lh='ls -Alh'
alias grep='grep --color=auto'
alias fgrep='fgrep --color=auto'
alias egrep='egrep --color=auto'
alias dateclip='date|xsel --clipboard'
alias tsclip='echo -n "$(date +%Y-%m-%d-%H.%M.%S)"|xsel --clipboard'
alias tstmp='date +%Y-%m-%d-%H.%M.%S'
alias cryptclip="xsel|gpg -ear ${your_gpg_email_address_goes_here}|xsel --clipboard"
alias putclip="xsel --clipboard"
alias getclip="xsel"
alias procstat="ps -e -o pcpu,pid,cpu,nice,state,cputime,args --sort -pcpu | sed '/^ 0.0 /d'"
alias wylie='l=20; x=1300; y=100; d=-5;for i in `seq $x $d $y`; do beep -l $l -f $i; done'
beepwhenup () { PHOST="$@"; if [[ "$PHOST" == "" ]]; then echo 'Enter host you want to ping:'; read PHOST; fi; if [[ "$PHOST" == "" ]]; then echo 'No host to ping.  Aborted.'; exit; fi; while true; do ping -c1 -W2 $PHOST 2>&1 >/dev/null; if [[ "$?" == "0" ]]; then for j in $(seq 1 4); do beep; done; ping -c1 $PHOST; break; fi; done; }
shopt -s checkwinsize
export PATH=${PATH}:/home/tommy/android-sdk-linux_86/tools
alias vimclean="find . -iname '*sw[po]' -print -delete"
howmuchmem () { PROCNAME="$@"; echo $PROCNAME IS USING $(echo "scale=4; ($(ps axo rss,comm|grep $PROCNAME| awk '{ TOTAL += $1 } END { print TOTAL }')/$(free | head -n 2 | tail -n 1 | awk '{ print $2 }'))*100"|bc)% of system RAM; };
alias chme='sudo chown -R $USER:$USER'
alias procperuser='ps ax -o user | sort | uniq -c | sort -nr'
memhogs () { TR=`free|grep Mem:|awk '{print $2}'`;ps axo rss,comm,pid|awk -v tr=$TR '{proc_list[$2]+=$1;} END {for (proc in proc_list) {proc_pct=(proc_list[proc]/tr)*100; printf("%d\t%-16s\t%0.2f%\n",proc_list[proc],proc,proc_pct);}}'|sort -n |tail -n 10; };
export PS1="\n[\[\e[1;37m\]\u\[\e[0m\]@\[\e[1;34m\]\H\[\e[0m\]] [\[\e[1;33m\]\d, \t\[\e[0m\]] [\[\e[1;31m\]\!\[\e[0m\]]\n\[\e[1;31m\]\[\e[0m\][\[\e[1;37m\]\w\[\e[0m\]]\n\[\e[1;37m\]\\$\[\e[0m\] "

denyhosts is an excellent utility to prevent brute-force SSH attacks against your server. One shortcoming it has is that it is difficult to get a blacklisted IP address back out of the, well, blacklist. Simply removing its entry from /etc/hosts.deny will not keep denyhosts from doing its job of blocking what it deemed to be a threat. So what to do? Enter this script (below). It’s yours for the taking.

Unlike other tools floating around on the interwebs, this one actually works. Upload it to your server and name it “denyhosts-remove”, place it in /usr/local/sbin, and make it executable a la chmod +x /usr/local/denyhosts-remove

Invoke it via sudo or as root with no arguments for usage instructions.

#!/bin/bash
 
# denyhosts-remove.sh
#
# AUTHOR: Tommy Butler, email: $ echo YWNlQHRvbW15YnV0bGVyLm1lCg==|base64 -d
# VERSION: 1.0
#
# SUMMARY:
# Use this script to Remove an IP address ban that has been errantly blacklisted
# by denyhosts - the ubiquitous and unforgiving brute-force attack protection
# service so often used on Linux boxen.
#
# INSTALL:
# Usage: Put this script somewhere in your $PATH, and execute it as root or
# with sudo.  Call it directly or with an IP address argument.  Multiple IP
# address arguments are not supported.  You'll need to `chmod +x` it first.
#
# LICENSE:
# GNU GPL 1.0
# Copyright 2011 Tommy Butler, All rights reserved
 
BASE_PATH="/var/lib/denyhosts";
IP=$1
 
if [[ "`/usr/bin/id -u`" != "0" ]]; then
   echo "Run this script as root or with sudo or app can't run correctly.  Aborted."
   exit 1;
fi
 
cd $BASE_PATH
 
if [[ "`pwd`" != "$BASE_PATH" ]]; then
   echo "Couldn't cd to $BASE_PATH.  Abort."
   exit 1;
fi
 
if [[ "$IP" == "" ]]; then
   echo "Enter the IP address you want to un-ban"
   read IP
fi
 
if [[ "$IP" == "" ]]; then
   echo "No IP address given.  Abort."
   exit 1;
fi
 
/etc/init.d/denyhosts stop
 
/usr/bin/perl -pi -e "s/^.*?$IP.*\n//g" /etc/hosts.deny *
 
/etc/init.d/denyhosts start
 
exit $?

Do you ever need to/want to know how hot your processors are running? There’s a way!

Sure, there are handy gui tools for things like this, but for our purposes, sometimes a quick, clean little shell code snippet gets the job done quite nicely. Take, for instance when you are running something CPU intensive, like transcoding video with handbrake or maybe folding proteins. Those kinds of computations really give your processor(s) a heavy workout, and running them hotter than recommended temperatures (usually ~40 degrees celcius) will shorten their lifespan.

This little bash function will help you keep an eye on things. First, you need to install the sensors package for your Linux distribution. Then run the sensors-detect command with root privileges (sudo) to get your sensors configured. *CAVEAT: sorry laptop/netbook users, your hardware doesn’t usually have sensors.

After running the sensors-detect command, you’re ready. Pop this little bit of code into your ~/.bashrc file and the next time you open a terminal, you will be able to simply type "coretemp" and get a live view of the temperature of your processor cores.

function coretemp {
 
   /usr/bin/clear;
 
   while : ; do 
 
      /usr/bin/sensors | /bin/grep ^Core | while read x; do 
 
         /usr/bin/printf '% .23s\n' "$x"; 
 
      done; 
 
      /bin/sleep 1 && /usr/bin/clear;
 
   done;
}

And that’s it. One more little tip for you is that you can use that sensors command (even without root privileges) to get other great information if your hardware supports it, such as fan speed and bus stats.

You can send out an "alert" email when someone logs in to your server (or workstation) by taking two steps*.

Now if you prefer to send the alert to your smart phone instead of your inbox, you can easily apply what wikipedia has to say regarding text-to-sms gateway addresses after you finish reading this post.

Step 1) is done by placing this script into a file named "iloggedin" the /usr/local/sbin/ directory, and making it executable with chmod +x /usr/local/sbin/iloggedin. While you are creating the file, make sure to modify the RECIPIENT declaration on line 2 so that it points to the email address at which you want to receive alerts. Once you do this, you’re half way done…

#!/bin/bash
RECIPIENT="your.email@address.here";
PREFIX="LOGIN ALERT!";
REMOTEIP=$(/bin/echo $SSH_CLIENT | /usr/bin/awk '{ print $1 }');
TIME=$(/bin/date +'%r, %D');
HOST=$(/bin/hostname -f);
 
if [[ "$REMOTEIP" == "" ]]; then
   REMOTEIP='localhost';
fi
 
/bin/cat <<LOGGEDIN | /usr/bin/mail -s "$PREFIX $USER@$HOST $TIME" $RECIPIENT
Remote user $USER just logged in to $HOST at $TIME from $REMOTEIP
LOGGEDIN
 
exit $?

Step 2) is to modify your /etc/profile file. Add the following line to the bottom of the file:

/usr/local/sbin/iloggedin >/dev/null 2>&1 & disown -a

*You will need to have a working email setup before you can send mail from your Linux operating system.

This concludes our little howto.

Vim sessions are magical things that preserve every setting, every tab, every fold, everything open file, and just about everything else (except command history – “undo”)

You can save your vim session and environment and then open it again later, without missing a beat. This is so helpful if you are editing a file remotely on a server and then lose your connection (you should also be using screen for this). In such a situation, you would just log back in and resume your session.

So let’s get down to it, shall we? To save your vim session, choose a file to save for vim to save your session’s meta-data. For our purposes we’ll use the file name of “myproject.vimsess”. From within your vim session, enter command mode and type:

:mksession! /path/to/myproject.vimsess

That will save your session. Now to resume your session, from the command line you would type:

vim -S /path/to/myproject.vimsess

Happy vim-ming!