Tommy Butler

This post is somewhat inspired by the fact that Google+ now makes cool little gif animations from photo bursts of 5 pics or more when you upload.  Very nifty.  But, ugh, the images are SMALL and often choppy.  What if you want larger picture dimensions, smoother or longer animations?  For those who care about making these *ridiculously cute* animations out of mpeg or avi movies from your phone/camera/youtube… this is for you. Just be careful not to use movie clips longer than 15 to 20 seconds or you’ll end up with a huge image file.

The workflow:

Install Linux, ffmpeg, imagemagick, and stuff.  Mostly just install Linux.  If you don’t, you can stop reading here and we can still be friends.  But no gif for you…

Alright then.  So just get your mpeg 4 movie and open a terminal in the directory where your movie has been saved, then …

$ mkdir gif
$ ffmpeg -i YOUR_MOVIE.mpg -an -r 10 -y -s 640x480 gif/capture%03d.png # *see footnote
$ cd gif
$ for file in *png; do echo converting $file ...; convert $file `basename $file .png`.gif; done
$ convert -delay 10 cap*gif -loop 0 anim.gif
$ convert -layers Optimize anim.gif anim_optimized.gif
$ mv anim_optimized.gif ../
$ cd ../
$ rm -rf ./gif

… and enjoy anim_optimized.gif !!!

Adjust the gif image dimensions to taste (my 1080p mp4 file needed to be scaled way down to 640×480.  You might want yours higher/lower res).

This workflow solves all the problems with the stuff you’ll find on stack exchange and the intergoogles, it will save you all the time needlessly tinkering and swearing loudly while debugging your way through all the bad information you keep finding online and the arcane syntax in the ffmpeg man page.

*Footnote: this “%03d” bit should cover you for movie clips with (way too) many frames, but really you shouldn’t make gifs longer than 15 seconds.  They can become very large in file size and take a lot of time to download. The gif image you see here is about 70 frames.

That’s him, the legendary TimToady.  Larry Wall is personally responsible for one of the greatest joys in my life: Larry created Perl.

Perl has remained a core staple in my toolbox of awesomeness since 1999, when I discovered it as a younger developer/engineer just starting out my career.  Over the years, Perl has been been a tool I’ve used both for fun and profit, building and automating all sorts of things.  As I explained to Larry at lunch, his work made mine possible, and the rewards of my having embraced Perl have been a steady stream of income for my family, and countless great times with other Perl hackers.

Both at the lunch table during YAPC::NA 2013 and again each day, I thank him.

So here’s to you, Larry.  My hero, and friend.

Symptoms

Networking on the host machine is still functioning properly, but all networking stops working on your KVM virtual machines. You can’t even get dhclient to work in a VM. Even if you manually assign an IP address and default route to a VM, it still can’t ping its gateway. Rebooting the virtual machines does no good; they still can’t get an IP address.

Quick and Dirty

Basically, you need to do restart libvirtd and (probably) save your iptables afterword; read on for more info, i.e.- don’t just copy and paste this. You should understand what you’re doing, why you’re doing it, and never paste anything off the internet into a terminal!

$ service libvirtd restart && iptables-save > /etc/sysconfig/iptables

In Depth

Red Hat’s KVM (kernel-based virtual machine) framework, built on libvirt and qemu, has some great characteristics — but it can be fragile, particularly when dealing with networking.

Unlike VMware or VirtualBox virtualization products, KVM is open and transparent about the inner-workings of almost everything.  But that has consequences.  There can be unexpected effects on your KVM environment in RHEL (or CentOS/Fedora/Scientific Linux…) when you tweak system settings on the HOST machine(s).

VMware does a good job of “protecting” its NAT’ted networks from you and your customized settings on the host operating system, but KVM, not so much…actually I should have said–not at all.  Sure, if you try hard enough you can break anything, but VMware’s vmnet.ko kernel module doesn’t inherently rely on the host’s iptables configuration as does KVM.  Ah, see?  There it is…there’s the issue at hand.  This is why your KVM network (and all your KVM virtual machines) stopped working.  You tweaked your iptables firewall and who would have known, right?  Maybe you added an NFS server, or an Apache instance, a VNC service perhaps?  Unlike many popular debian-based systems like Ubuntu, Red Hat .rpm’s don’t do much post-install configuration (don’t get into a holy war with me, it’s the !#@$ truth OK).  This translates to, for better AND worse, the fact that when adding a listening network service to a RHEL machine it is unlikely that the post-install script will poke a hole in your iptables firewall automatically for you under the assumption that you actually want to use the new service.

…Like I said, that can be a good AND a bad thing.  In an enterprise environment, it’s probably more good than bad because you’re dealing with possible security ramifications in a mission-critical infrastructure (where you should have real hardware firewalls in place anyway, but I digress…)

All of this adds up to one fact: if you added a new listening network service to your RHEL KVM host machine, you had to manually tweak your firewall yourself.  To wit, no magically convenient post-install script for the nfs-kernel-server package will do this for you on RHEL.  This necessitated the action that brought about your problem.  You, like myself and many others, made the one-time mistake of forgetting that KVM needs you to leave alone the (overtly transparent) rules for it’s iptables.

You, like myself and many others probably typed $ sudo system-config-firewall-tui (or something like it).

And that’s all it takes.  If you do that and “save” your changes, your KVM network is blown away!

Such handy-dandy iptables utilities as Red Hat’s system-config-firewall-tui should protect you from the mistakes you know you might make in an already-complex iptables setup.  You probably thought that by using such a tool that it would generate the correct iptables commands and you wouldn’t thereby be able to mess things up. You thought you were smart to use a tool that avoids human error. But in this case…

YOU WERE WRONG.

You were wrong because you thought your iptables wizard-y tool would preserve your existing setup, or in the least not blow it away.  *Sigh*.  You, poor soul, are lucky though. Red Hat’s init script for KVM’s libvirtd knows how to fix the problem– and the awesome thing about it is that unlike system-config-firewall-tui, the libvirt init script will preserve existing iptables rules.  You can check out the official Red Hat documentation about this if you like, but it comes down to the two commands up above in the “Quick and Dirty” section; call the libvirtd init script and then save your iptables to the persistent rules file in /etc/sysconfig/iptables and your rear is saved.

$ sudo -s
$ service libvirtd restart
$ iptables-save > /etc/sysconfig/iptables

In conclusion, remember that this should be a one-time mistake 

…Even Google does it.  I’m talking about load balancing via DNS.  If you find yourself in a situation where one server isn’t cutting it anymore and you would like to easily distribute load across several servers with identical configuration answering to the same backend, one way to accomplish it is to use round-robin DNS.  It’s even easier than it sounds…

The first step is to set up your servers with individual public IPs.  (That was easy).  Now make an “A” record for each one in your DNS configuration.  The result is that your DNS server, such as bind, will rotate the IP addresses handed out to remote clients.  This is the default behavior for most DNS servers; if it isn’t the default for yours, customize config as necessary– this writeup is not going to go into detail about such configuration.  If you need to, Goog it.

What it ends up looking like is something akin to what you see in the screenshot below (in this example, I’m using Google’s servers).  Notice that one request for the “A” record for google.com yields several different results for my web client to choose from.  Notice again that another request only a couple seconds later yields even more choices, different from the first.  Google is making sure that requests for their servers get spread out across many different inroads, and while I’m sure they have some great hardware-based load balancing in their datacenters, they’re still using this technique too:

Notice also that Google is using a 300 second TTL on its “A” records.  This helps (but does not eliminate) client-side DNS caching that can lessen the effectiveness of the round-robin DNS load balancing.

For a TTL that low, you’ll need some beefy name servers, and DNS service that lets you set a TTL that low.  I think that for most purposes you don’t need to have a TTL that low, however.  A TTL of 15 to 30 minutes is fine for most websites.

Bear in mind that due to the nature of very common client-side DNS query caching, your round-robin DNS load balancing is not likely to distribute the load on your remote servers on a per-request basis.  It will be more like a per-session basis.  Nothing is truly guaranteed, however, without real hardware-based load balancing.

Think of this as a “poor man’s” load balancing solution.  You’ll get some mileage out of it, but it’s not a substitute for say, a BigIP f5 load balancer cluster

For even more effective (and low-cost/free) load balancing solutions, look at technology like the HA Linux project: http://www.linux-ha.org

Disclaimer: Like some of my other posts here, I’ve shared this snippet on other websites out there. Still, I reserve the right to re-use my own material

OK then.  There have been times in the past when it was necessary for me to convert a VirtualBox disk file to a format that I could use in VMware. Virtualbox, in combination with the qemu utilities, provides us with a way to get that job done. Conversion takes time, and the larger your disk file, the longer it takes. With this in mind, here below is the chain of commands you’ll need to accomplish the desired outcome:

$ VBoxManage internalcommands converttoraw myvirtualmachine.vdi myvirtualmachine.raw && qemu-img convert -O vmdk myvirtualmachine.raw myvirtualmachine.vmdk && rm -vi myvirtualmachine.raw

The benefit: using this method actually works. There are others out there that claim to give you a working .vmdk by simply using the qemu-img command alone. Doing that only results in pain for you because the .vmdk file will be created with no errors, but it won’t boot either.

Be aware that this is a very I/O intensive, and depending on some other factors it could cause your system to slow down dramatically during the process. It’s probably best to do the conversion at a time when you’re not in a hurry to get other things done.

Typically the w command will show you who is logged in.  So do the users and who commands.  With the advent of Ubuntu’s unity and the lightdm desktop session manager, these commands don’t tell the whole truth.

Moreover, users don’t need to be logged in to a terminal (or pseudo terminal) in order to run processes or services.  If you want a quick and dirty way to see what’s going on, you can use the ps command.  Typically you use the “-a -u -f -x” flags with the command (ps aufx) if you want the information for all users in easily readable, tree-based, fully formatted output.  Pipe it through more or less.

Still, what if you only care about the users, and not all the gory details?  What if you just want to know what users are running processes?  You can still use the ps command, but filter it through a few pipes: get rid of everything except the first column of output, chop off the header, sort the output, toss out duplicate lines, and there you have it — a simple list of users*

$ ps aux | awk '{ print $1 }' | sed '1 d' | sort | uniq

The above code will give you output like this:

103
avahi
colord
daemon
lp
nobody
postfix
root
rtkit
statd
superman
syslog

Now I’ll grant you that there are other ways to do this; you can fine-tune the output of the ps command using optional formatting arguments. The purpose of this example, however, is to illustrate the capabilities of the individual utilities in the chain of pipes, as described in the preceding paragraph.

If you want to customize it even further (and make it even more difficult to read ), you can add a bit of Perl to the end of the command to filter out all system users and only show you the names of regular users (and/or the root user) with active processes:

$ ps aux | awk '{ print $1 }' | sed '1 d' | sort | uniq | perl -e 'for (<>) { chomp; $u = ( getpwnam($_) )[2]; print $_, "\n" if ( ( $u >= 1000 || $u == 0 ) && ( $_ =~ /[[:alpha:]]/ && $_ ne "nobody" ) ) }'

Loosely explained: display “root” who has a UID of 0, display regular users (they will have UID’s of 1000 or higher), don’t display the “nobody” user, and don’t display the effective UID’s of system-level processes running with UID’s that don’t correspond to an entry in the /etc/passwd database. (The formatted Perl code in the above one-liner is provided at the end of this post.) The output will now look something like:

root
superman

As promised, the formatted Perl code:

for ( <> ) { # read STDIN line by line
 
   chomp; # remove line endings
 
   $u = ( getpwnam($_) )[2]; # get the numeric UID of the user
 
   print $_, "\n" if ( # output the username if...
 
      # ...the user is a "regular" user, or root...
      ( $u >= 1000 || $u == 0 ) &&
 
      # ...and as long as the user has a real name and it isn't "nobody"
      ( $_ =~ /[[:alpha:]]/ && $_ ne "nobody" )
   ) 
}

Final thoughts: this is terrible. Ubuntu has "broken" the w command (and its relatives). While it’s nice to know that you can still get the info you need by massaging it out of the ps command, this really is a dirty hack.

*There’s a dirty way to “hide” processes from the Linux/Unix process table.  There are feasible reasons to do this from time to time, but for the most part it’s only malicious software that does such a thing.  The ps command won’t show hidden processes.  If you want to guarantee that you can see all processes, take a look at the unhide utility.

Friends, you should encrypt. So, search for a nice tutorial about how to set up GPG for yourself and get to it. If you prefer to start out easily without learning to use gpg from the command line, Ubuntu/Mint/Debian all provide great GUI tools, such as the built in "Passwords And Keys" utility, easily found in your programs menu. Once you get GPG set up, you can start selecting/copying text and encrypting it very quickly with a handy bash alias. Add one like this (see below) to your bash profile, `source` it, and the next time you type “cryptclip” at the command prompt your clipboard will be replaced with the encrypted version of its former text.

When you have a working GPG setup, install the “xsel” utility if you don’t already have it. Then add this line to your bash profile:

alias cryptclip='xsel|gpg -ear your@gpgemailaddress.com|xsel --clipboard'

First, you’ll need to use the visudo utility…

sudo visudo

This command safely opens up the /etc/sudoers file for you in your default editor. Let’s say you want to allow a user named “joe” to run a given command. You just need to add a line like this below (customize for your needs)

joe ALL=(ALL) NOPASSWD: /full/path/to/command

Now what if you want to restrict joe to only use that command within a given set of parameters or with only certain arguments? Well, just toss them in there too! Check this out:

joe ALL=(ALL) NOPASSWD: /full/path/to/command ARG1 ARG2

…Reinstall it!

sudo dpkg remove --purge vnc4server && sudo apt-get -y install vnc4server

That has always fixed the problem for me when I’ve had such troubles.