Tommy Butler

Symptoms

Networking on the host machine is still functioning properly, but all networking stops working on your KVM virtual machines. You can’t even get dhclient to work in a VM. Even if you manually assign an IP address and default route to a VM, it still can’t ping its gateway. Rebooting the virtual machines does no good; they still can’t get an IP address.

Quick and Dirty

Basically, you need to do restart libvirtd and (probably) save your iptables afterword; read on for more info, i.e.- don’t just copy and paste this. You should understand what you’re doing, why you’re doing it, and never paste anything off the internet into a terminal!

$ service libvirtd restart && iptables-save > /etc/sysconfig/iptables

In Depth

Red Hat’s KVM (kernel-based virtual machine) framework, built on libvirt and qemu, has some great characteristics — but it can be fragile, particularly when dealing with networking.

Unlike VMware or VirtualBox virtualization products, KVM is open and transparent about the inner-workings of almost everything.  But that has consequences.  There can be unexpected effects on your KVM environment in RHEL (or CentOS/Fedora/Scientific Linux…) when you tweak system settings on the HOST machine(s).

VMware does a good job of “protecting” its NAT’ted networks from you and your customized settings on the host operating system, but KVM, not so much…actually I should have said–not at all.  Sure, if you try hard enough you can break anything, but VMware’s vmnet.ko kernel module doesn’t inherently rely on the host’s iptables configuration as does KVM.  Ah, see?  There it is…there’s the issue at hand.  This is why your KVM network (and all your KVM virtual machines) stopped working.  You tweaked your iptables firewall and who would have known, right?  Maybe you added an NFS server, or an Apache instance, a VNC service perhaps?  Unlike many popular debian-based systems like Ubuntu, Red Hat .rpm’s don’t do much post-install configuration (don’t get into a holy war with me, it’s the !#@$ truth OK).  This translates to, for better AND worse, the fact that when adding a listening network service to a RHEL machine it is unlikely that the post-install script will poke a hole in your iptables firewall automatically for you under the assumption that you actually want to use the new service.

…Like I said, that can be a good AND a bad thing.  In an enterprise environment, it’s probably more good than bad because you’re dealing with possible security ramifications in a mission-critical infrastructure (where you should have real hardware firewalls in place anyway, but I digress…)

All of this adds up to one fact: if you added a new listening network service to your RHEL KVM host machine, you had to manually tweak your firewall yourself.  To wit, no magically convenient post-install script for the nfs-kernel-server package will do this for you on RHEL.  This necessitated the action that brought about your problem.  You, like myself and many others, made the one-time mistake of forgetting that KVM needs you to leave alone the (overtly transparent) rules for it’s iptables.

You, like myself and many others probably typed $ sudo system-config-firewall-tui (or something like it).

And that’s all it takes.  If you do that and “save” your changes, your KVM network is blown away!

Such handy-dandy iptables utilities as Red Hat’s system-config-firewall-tui should protect you from the mistakes you know you might make in an already-complex iptables setup.  You probably thought that by using such a tool that it would generate the correct iptables commands and you wouldn’t thereby be able to mess things up. You thought you were smart to use a tool that avoids human error. But in this case…

YOU WERE WRONG.

You were wrong because you thought your iptables wizard-y tool would preserve your existing setup, or in the least not blow it away.  *Sigh*.  You, poor soul, are lucky though. Red Hat’s init script for KVM’s libvirtd knows how to fix the problem– and the awesome thing about it is that unlike system-config-firewall-tui, the libvirt init script will preserve existing iptables rules.  You can check out the official Red Hat documentation about this if you like, but it comes down to the two commands up above in the “Quick and Dirty” section; call the libvirtd init script and then save your iptables to the persistent rules file in /etc/sysconfig/iptables and your rear is saved.

$ sudo -s
$ service libvirtd restart
$ iptables-save > /etc/sysconfig/iptables

In conclusion, remember that this should be a one-time mistake 

…Even Google does it.  I’m talking about load balancing via DNS.  If you find yourself in a situation where one server isn’t cutting it anymore and you would like to easily distribute load across several servers with identical configuration answering to the same backend, one way to accomplish it is to use round-robin DNS.  It’s even easier than it sounds…

The first step is to set up your servers with individual public IPs.  (That was easy).  Now make an “A” record for each one in your DNS configuration.  The result is that your DNS server, such as bind, will rotate the IP addresses handed out to remote clients.  This is the default behavior for most DNS servers; if it isn’t the default for yours, customize config as necessary– this writeup is not going to go into detail about such configuration.  If you need to, Goog it.

What it ends up looking like is something akin to what you see in the screenshot below (in this example, I’m using Google’s servers).  Notice that one request for the “A” record for google.com yields several different results for my web client to choose from.  Notice again that another request only a couple seconds later yields even more choices, different from the first.  Google is making sure that requests for their servers get spread out across many different inroads, and while I’m sure they have some great hardware-based load balancing in their datacenters, they’re still using this technique too:

Notice also that Google is using a 300 second TTL on its “A” records.  This helps (but does not eliminate) client-side DNS caching that can lessen the effectiveness of the round-robin DNS load balancing.

For a TTL that low, you’ll need some beefy name servers, and DNS service that lets you set a TTL that low.  I think that for most purposes you don’t need to have a TTL that low, however.  A TTL of 15 to 30 minutes is fine for most websites.

Bear in mind that due to the nature of very common client-side DNS query caching, your round-robin DNS load balancing is not likely to distribute the load on your remote servers on a per-request basis.  It will be more like a per-session basis.  Nothing is truly guaranteed, however, without real hardware-based load balancing.

Think of this as a “poor man’s” load balancing solution.  You’ll get some mileage out of it, but it’s not a substitute for say, a BigIP f5 load balancer cluster

For even more effective (and low-cost/free) load balancing solutions, look at technology like the HA Linux project: http://www.linux-ha.org

Disclaimer: Like some of my other posts here, I’ve shared this snippet on other websites out there. Still, I reserve the right to re-use my own material

OK then.  There have been times in the past when it was necessary for me to convert a VirtualBox disk file to a format that I could use in VMware. Virtualbox, in combination with the qemu utilities, provides us with a way to get that job done. Conversion takes time, and the larger your disk file, the longer it takes. With this in mind, here below is the chain of commands you’ll need to accomplish the desired outcome:

$ VBoxManage internalcommands converttoraw myvirtualmachine.vdi myvirtualmachine.raw && qemu-img convert -O vmdk myvirtualmachine.raw myvirtualmachine.vmdk && rm -vi myvirtualmachine.raw

The benefit: using this method actually works. There are others out there that claim to give you a working .vmdk by simply using the qemu-img command alone. Doing that only results in pain for you because the .vmdk file will be created with no errors, but it won’t boot either.

Be aware that this is a very I/O intensive, and depending on some other factors it could cause your system to slow down dramatically during the process. It’s probably best to do the conversion at a time when you’re not in a hurry to get other things done.

Typically the w command will show you who is logged in.  So do the users and who commands.  With the advent of Ubuntu’s unity and the lightdm desktop session manager, these commands don’t tell the whole truth.

Moreover, users don’t need to be logged in to a terminal (or pseudo terminal) in order to run processes or services.  If you want a quick and dirty way to see what’s going on, you can use the ps command.  Typically you use the “-a -u -f -x” flags with the command (ps aufx) if you want the information for all users in easily readable, tree-based, fully formatted output.  Pipe it through more or less.

Still, what if you only care about the users, and not all the gory details?  What if you just want to know what users are running processes?  You can still use the ps command, but filter it through a few pipes: get rid of everything except the first column of output, chop off the header, sort the output, toss out duplicate lines, and there you have it — a simple list of users*

$ ps aux | awk '{ print $1 }' | sed '1 d' | sort | uniq

The above code will give you output like this:

103
avahi
colord
daemon
lp
nobody
postfix
root
rtkit
statd
superman
syslog

Now I’ll grant you that there are other ways to do this; you can fine-tune the output of the ps command using optional formatting arguments. The purpose of this example, however, is to illustrate the capabilities of the individual utilities in the chain of pipes, as described in the preceding paragraph.

If you want to customize it even further (and make it even more difficult to read ), you can add a bit of Perl to the end of the command to filter out all system users and only show you the names of regular users (and/or the root user) with active processes:

$ ps aux | awk '{ print $1 }' | sed '1 d' | sort | uniq | perl -e 'for (<>) { chomp; $u = ( getpwnam($_) )[2]; print $_, "\n" if ( ( $u >= 1000 || $u == 0 ) && ( $_ =~ /[[:alpha:]]/ && $_ ne "nobody" ) ) }'

Loosely explained: display “root” who has a UID of 0, display regular users (they will have UID’s of 1000 or higher), don’t display the “nobody” user, and don’t display the effective UID’s of system-level processes running with UID’s that don’t correspond to an entry in the /etc/passwd database. (The formatted Perl code in the above one-liner is provided at the end of this post.) The output will now look something like:

root
superman

As promised, the formatted Perl code:

for ( <> ) { # read STDIN line by line
 
   chomp; # remove line endings
 
   $u = ( getpwnam($_) )[2]; # get the numeric UID of the user
 
   print $_, "\n" if ( # output the username if...
 
      # ...the user is a "regular" user, or root...
      ( $u >= 1000 || $u == 0 ) &&
 
      # ...and as long as the user has a real name and it isn't "nobody"
      ( $_ =~ /[[:alpha:]]/ && $_ ne "nobody" )
   ) 
}

Final thoughts: this is terrible. Ubuntu has "broken" the w command (and its relatives). While it’s nice to know that you can still get the info you need by massaging it out of the ps command, this really is a dirty hack.

*There’s a dirty way to “hide” processes from the Linux/Unix process table.  There are feasible reasons to do this from time to time, but for the most part it’s only malicious software that does such a thing.  The ps command won’t show hidden processes.  If you want to guarantee that you can see all processes, take a look at the unhide utility.

Friends, you should encrypt. So, search for a nice tutorial about how to set up GPG for yourself and get to it. If you prefer to start out easily without learning to use gpg from the command line, Ubuntu/Mint/Debian all provide great GUI tools, such as the built in "Passwords And Keys" utility, easily found in your programs menu. Once you get GPG set up, you can start selecting/copying text and encrypting it very quickly with a handy bash alias. Add one like this (see below) to your bash profile, `source` it, and the next time you type “cryptclip” at the command prompt your clipboard will be replaced with the encrypted version of its former text.

When you have a working GPG setup, install the “xsel” utility if you don’t already have it. Then add this line to your bash profile:

alias cryptclip='xsel|gpg -ear your@gpgemailaddress.com|xsel --clipboard'

First, you’ll need to use the visudo utility…

sudo visudo

This command safely opens up the /etc/sudoers file for you in your default editor. Let’s say you want to allow a user named “joe” to run a given command. You just need to add a line like this below (customize for your needs)

joe ALL=(ALL) NOPASSWD: /full/path/to/command

Now what if you want to restrict joe to only use that command within a given set of parameters or with only certain arguments? Well, just toss them in there too! Check this out:

joe ALL=(ALL) NOPASSWD: /full/path/to/command ARG1 ARG2

…Reinstall it!

sudo dpkg remove --purge vnc4server && sudo apt-get -y install vnc4server

That has always fixed the problem for me when I’ve had such troubles.

Go ahead and skip right to the examples if you’re in a hurry

Got Regexes? Regular Expressions (also known as regexes, regexen, and regexps) open up a world of new power tools to you when you need to automate text analysis or manipulations such as massive search and replace operations in one or multiple directories. Such a task oft befalls the sysadmin/developer/commandline ninja.

So case in point: If you find yourself needing to do some file correction in place without the hassle of sed’s inferior pattern matching and all the limitations that go with it, whilst neither feeling the inclination to concoct a convoluted awk block, nor even more a desire to brazenly undertake the time-consuming production of a full blown python script–just to wrap up in a python application the kind of functionality that is just as well performed with a Perl one-liner—— well my friends you’ve come to the right place.

Indeed Perl provides you with in-line search and replace using real regular expressions in all their glory, and in the world of text processing, Perl is king. (Sorry haters/ruby-fanboys/shell-purists, it’s just true!)

So come now, let’s put down the tar, feathers, torches and pitchforks for a moment and consider the following super-cool example (which even creates backup copies of the files you changed, in the event that you flubbed up your regex):

perl -p -i'.backup' -e 's/(?<=replace )this(?= lolc[aA@]t)(?# with )/that/; s/(?# and as for all of the )([[:digit:]]+) (?i:fat|skinny|thirsty) camels(?# you really need those)/$1 white llamas/g;' file1 file2 /path/to/file3

The above powerful search and replace shows you a hefty example of how Perl regexes can be used in your search and replace operation, and the example uses some pretty useful and advanced matching tools you just can’t readily get elsewhere; among other features, the example uses positive look-behind zero-width assertions, positive look-ahead zero-width assertions, atomic sub-match capturing, POSIX character classes, and in-line zero-width comments that make the regexes more readable. In the example, Perl runs two pattern substitutions on every line in each of the three files, and what’s more, it makes backup copies for you!

A more simple version without backups would be:

perl -p -i -e 's/replace this/using that/g' /all/text/files/in/*.txt

The example above replaces any occurrence of the string “replace this” with the string “using that” on all text files inside the directory name given.

So in summary, if you want to use the most powerful search and replace tools on the command line, and do it in the easiest form, use perl -p -i -e 'pattern' file and use it wisely.

…And by “wisely”, I mean it would be wise to brush up on your regular expression fu for free by reading the Perl documentation about its incredible regular expressions.

Please feel free to experiment with the following ~/.bash_profile goodness of my own, and save away to yours what tasty morsels that you find useful.

Exactly what each line does below is left as an exercise to the reader   Hint: none of these aliases/functions/commands are malicious!

export HISTTIMEFORMAT='%F %T '
export HISTCONTROL=ignoredups
export HISTCONTROL=ignoreboth
export HISTIGNORE='pwd:ls:history:'
export HISTSIZE=4096
export EDITOR='/usr/bin/vim'
export AUTOSSH_POLL=30
eval `dircolors`
alias ls='ls --color=auto'
alias dir='ls --color=auto --format=vertical'
alias ll='ls -Al'
alias la='ls -A'
alias lh='ls -Alh'
alias grep='grep --color=auto'
alias fgrep='fgrep --color=auto'
alias egrep='egrep --color=auto'
alias dateclip='date|xsel --clipboard'
alias tsclip='echo -n "$(date +%Y-%m-%d-%H.%M.%S)"|xsel --clipboard'
alias tstmp='date +%Y-%m-%d-%H.%M.%S'
alias cryptclip="xsel|gpg -ear ${your_gpg_email_address_goes_here}|xsel --clipboard"
alias putclip="xsel --clipboard"
alias getclip="xsel"
alias procstat="ps -e -o pcpu,pid,cpu,nice,state,cputime,args --sort -pcpu | sed '/^ 0.0 /d'"
alias wylie='l=20; x=1300; y=100; d=-5;for i in `seq $x $d $y`; do beep -l $l -f $i; done'
beepwhenup () { PHOST="$@"; if [[ "$PHOST" == "" ]]; then echo 'Enter host you want to ping:'; read PHOST; fi; if [[ "$PHOST" == "" ]]; then echo 'No host to ping.  Aborted.'; exit; fi; while true; do ping -c1 -W2 $PHOST 2>&1 >/dev/null; if [[ "$?" == "0" ]]; then for j in $(seq 1 4); do beep; done; ping -c1 $PHOST; break; fi; done; }
shopt -s checkwinsize
export PATH=${PATH}:/home/tommy/android-sdk-linux_86/tools
alias vimclean="find . -iname '*sw[po]' -print -delete"
howmuchmem () { PROCNAME="$@"; echo $PROCNAME IS USING $(echo "scale=4; ($(ps axo rss,comm|grep $PROCNAME| awk '{ TOTAL += $1 } END { print TOTAL }')/$(free | head -n 2 | tail -n 1 | awk '{ print $2 }'))*100"|bc)% of system RAM; };
alias chme='sudo chown -R $USER:$USER'
alias procperuser='ps ax -o user | sort | uniq -c | sort -nr'
memhogs () { TR=`free|grep Mem:|awk '{print $2}'`;ps axo rss,comm,pid|awk -v tr=$TR '{proc_list[$2]+=$1;} END {for (proc in proc_list) {proc_pct=(proc_list[proc]/tr)*100; printf("%d\t%-16s\t%0.2f%\n",proc_list[proc],proc,proc_pct);}}'|sort -n |tail -n 10; };
export PS1="\n[\[\e[1;37m\]\u\[\e[0m\]@\[\e[1;34m\]\H\[\e[0m\]] [\[\e[1;33m\]\d, \t\[\e[0m\]] [\[\e[1;31m\]\!\[\e[0m\]]\n\[\e[1;31m\]\[\e[0m\][\[\e[1;37m\]\w\[\e[0m\]]\n\[\e[1;37m\]\\$\[\e[0m\] "
coretemp () { /usr/bin/clear; while : ; do /usr/bin/sensors | /bin/grep ^Core | while read x; do /usr/bin/printf '% .23s\n' "$x"; done; /bin/sleep 1 && /usr/bin/clear; done; }

denyhosts is an excellent utility to prevent brute-force SSH attacks against your server. One shortcoming it has is that it is difficult to get a blacklisted IP address back out of the, well, blacklist. Simply removing its entry from /etc/hosts.deny will not keep denyhosts from doing its job of blocking what it deemed to be a threat. So what to do? Enter this script (below). It’s yours for the taking.

Unlike other tools floating around on the interwebs, this one actually works. Upload it to your server and name it “denyhosts-remove”, place it in /usr/local/sbin, and make it executable a la chmod +x /usr/local/denyhosts-remove

Invoke it via sudo or as root with no arguments for usage instructions.

#!/bin/bash
 
# denyhosts-remove.sh
#
# AUTHOR: Tommy Butler, email: $ echo YWNlQHRvbW15YnV0bGVyLm1lCg==|base64 -d
# VERSION: 1.0
#
# SUMMARY:
# Use this script to Remove an IP address ban that has been errantly blacklisted
# by denyhosts - the ubiquitous and unforgiving brute-force attack protection
# service so often used on Linux boxen.
#
# INSTALL:
# Usage: Put this script somewhere in your $PATH, and execute it as root or
# with sudo.  Call it directly or with an IP address argument.  Multiple IP
# address arguments are not supported.  You'll need to `chmod +x` it first.
#
# LICENSE:
# GNU GPL 1.0
# Copyright 2011 Tommy Butler, All rights reserved
 
BASE_PATH="/var/lib/denyhosts";
IP=$1
 
if [[ "`/usr/bin/id -u`" != "0" ]]; then
   echo "Run this script as root or with sudo or app can't run correctly.  Aborted."
   exit 1;
fi
 
cd $BASE_PATH
 
if [[ "`pwd`" != "$BASE_PATH" ]]; then
   echo "Couldn't cd to $BASE_PATH.  Abort."
   exit 1;
fi
 
if [[ "$IP" == "" ]]; then
   echo "Enter the IP address you want to un-ban"
   read IP
fi
 
if [[ "$IP" == "" ]]; then
   echo "No IP address given.  Abort."
   exit 1;
fi
 
/etc/init.d/denyhosts stop
 
/usr/bin/perl -pi -e "s/^.*?$IP.*\n//g" /etc/hosts.deny *
 
/etc/init.d/denyhosts start
 
exit $?