Posts Tagged ‘Perl’

Simple IP Failover

Thursday, August 13th, 2009

At work, we have a few virtual machines which are part of some sort of cluster.  Some are active/active and some are active/passive.  Some are load balancers and some are webservers.  Using clustering or IP failover for high availability is a great.  It's much easier to update nodes one at a time without having to schedule downtime or cause noticeable impact to the end user.  In the past I've been using the Linux-HA software.  It's full featured but very complicated.

Recently, I've been working on moving one of our services from Redhat Enterprise Linux based virtual machines to Ubuntu Linux.  Redhat has always served us pretty well, but some of the project requirements included newer versions of software than what was available in the lastest distribution.  These requirements were met with the latest long term release of Ubuntu, which is now over a year old.  I appreciate the timely release schedule that Ubuntu uses as well as the inclusion of the latest versions of various packages.  But, I'm getting off topic here.  I was utilizing the Linux-HA software with Redhat to run an active/active cluster.  Each cluster node handled http requests directly (one hostname had a couple IP addresses associated).  This worked well, but the Linux-HA software wasn't fun to manage.  I didn't use any front end tools, just edited the XML files and loaded them.  My other complaint was that the requests were not properly balanced over all the nodes using the DNS round robin approach.

So the new implementation now has redundant backend workers (running Tomcat), with a single Apache load balancer on the front end.  The Apache load balancer works as a reverse proxy and gracefully handles conditions where workers stop responding.  The load is appropriately dispersed between the workers and I am extremely pleased with the results.

But, there is one problem.  The Apache load balancer isn't highly available.  I didn't want to set up the Linux-HA software again, so I started looking around for a more simple solution (think KISS).  I soon found this blog post and it was exactly what I was looking for.

After reading the article, I decided that I would like to write a perl script that would use lock files and daemonize instead of a shell script.  I had just done another script that did that and was very happy with how it worked.  After putting together the script and doing some testing, I decided I needed two scripts.  One for the active node and one for the standby node.  The basic idea is that the standby node checks the active node to see that it is up and running.  If it detects a failure, it will bring up the service IP address, send the arp packet, and restart the Apache daemon (so it sees all IP addresses).  When the standby node detects the primary is back on the network, it shuts down the service IP address.  On the primary node, it will check the default gateway to determine if it is up on the network.  If it detects a failure, it will shutdown the service IP address.  When it resumes network connectivity, it will add the service IP, send the arp packet, and restart the apache service.

So the two scripts are below.  First the one that runs on the primary and second is the one that runs on the standby node.  I hope to use these with little modifications for all our applications.  Some have multiple IP addresses for IP based virtualhosting (SSL sites).  I also installed the fake package to utilize the send_arp program. I will probably need to make some more revisions, but I thought this might be helpful to other people out there trying to accomplish the same thing. These scripts come "as-is" with no warranty what so ever. Feel free to do what you'd like with them. If anyone has better solutions, feel free to post a comment!

EDIT (19-August-2009): I had some issues with a race condition in one of the scripts. The script that brought the IP address up on the primary node had an issue with the ping counter. I decided to just run one script on the standby node to simplify things. On the primary, I scheduled a cronjob to run the send_arp command every minute. This will send an arp packet to update the tables on the router when it's on-line. I've also made some slight modifications to the script that runs on the backup host.

EDIT (28-March-2010): I've taken down my git repository and I'm including the script below.

#!/usr/bin/perl -w
#
# ipfaild.pl
#
# Daemon to handle IP failover and restart necessary services.
#
# 11-Aug-2009 - Patrick Hennessy
#
use strict;
use Sys::Syslog;
use POSIX qw(setsid);
use Fcntl ':flock';
use Net::Ping::External qw(ping);

# Vars
#
my $pid;
my $progname = "ipfaild";
my $daemon_pidfile = "/var/run/$progname/$progname.pid";
my $daemon_lockfile = "/var/run/$progname/$progname.lock";
my $log_facility = "LOG_DAEMON";
my $ifconfig = '/sbin/ifconfig';
my $send_arp = '/usr/sbin/send_arp';
my $apache2ctl = '/usr/sbin/apache2ctl';
my $ping_timeout = 1;
my $sleep_time = 2;
my $missed = 0;
my $ipRec;

my $otherHost = 'otherhost.domain.com';
my $thisMAC = '00:11:22:33:44:55';

my @ipRecords = (
        { name=> 'fooservice', pubip => '192.168.1.200', dev => 'eth0:10', mask => '255.255.255.0' },
);

# Subroutines
#
sub daemonize;

# Daemonize process
#
daemonize;

# Acquire exclusive lock
#
open LOCKFILE, ">$daemon_lockfile" or die "$progname: can't write to $daemon_lockfile: $!n";
flock(LOCKFILE, LOCK_EX | LOCK_NB) or die "$progname: can't acquire lock: $daemon_lockfile: $!n";
print LOCKFILE "$pidn";

# Open syslog.
#
openlog($progname, "pid", $log_facility);

# Signal handlers
#
my $keep_processing = 1;
$SIG{HUP}  = sub { syslog("info", "Caught SIGHUP:  exiting gracefully"); $keep_processing = 0; };
$SIG{INT}  = sub { syslog("info", "Caught SIGINT:  exiting gracefully"); $keep_processing = 0; };
$SIG{QUIT}  = sub { syslog("info", "Caught SIGQUIT:  exiting gracefully"); $keep_processing = 0; };
$SIG{TERM}  = sub { syslog("info", "Caught SIGTERM:  exiting gracefully"); $keep_processing = 0; };

# Bring down interfaces.
#
for $ipRec (@ipRecords) {
        syslog("info", "Running: $ifconfig $ipRec->{'dev'} down");
        system($ifconfig, $ipRec->{'dev'}, 'down') == 0
                or syslog("info", "Error: $? Could not run: $ifconfig $ipRec->{'dev'} down");
}

# Main loop
#
while ($keep_processing) {
        # Ping the other host and count dropped packets
        #
        if (! ping(host => $otherHost, timeout => $ping_timeout)) {
                $missed++;
        } else {
                if ($missed > 2) {
                        for $ipRec (@ipRecords) {
                                syslog("info", "Running: $ifconfig $ipRec->{'dev'} down");
                                system($ifconfig, $ipRec->{'dev'}, 'down') == 0
                                        or syslog("info", "Error: $? Could not run: $ifconfig $ipRec->{'dev'} down");
                        }
                        $missed = 0;
                }
        }

        # Bring up IP addresses if packets dropped
        #
        if ($missed == 2) {
                for $ipRec (@ipRecords) {
                        syslog("info", "Running: $ifconfig $ipRec->{'dev'} $ipRec->{'pubip'} netmask $ipRec->{'mask'}");
                        system($ifconfig, $ipRec->{'dev'}, $ipRec->{'pubip'}, 'netmask', $ipRec->{'mask'}) == 0
                                or syslog("info", "Error: $? Could not run: $ifconfig $ipRec->{'dev'} $ipRec->{'pubip'} netmask $ipRec->{'mask'}");
                        syslog("info", "Running: $send_arp $ipRec->{'pubip'} $thisMAC $ipRec->{'pubip'} ff:ff:ff:ff:ff:ff");
                        system($send_arp, $ipRec->{'pubip'}, $thisMAC, $ipRec->{'pubip'}, 'ff:ff:ff:ff:ff:ff') == 0
                                or syslog("info", "Error: $? Could not run: $send_arp $ipRec->{'pubip'} $thisMAC $ipRec->{'pubip'} ff:ff:ff:ff:ff:ff");
                }
                syslog("info", "Running: $apache2ctl restart");
                system($apache2ctl, 'restart') == 0
                        or syslog("info", "Error: $? Could not run: $apache2ctl restart");
        }

        # Sleep
        #
        sleep($sleep_time);
}

# Bring down interfaces.
#
for $ipRec (@ipRecords) {
        syslog("info", "Running: $ifconfig $ipRec->{'dev'} down");
        system($ifconfig, $ipRec->{'dev'}, 'down') == 0
                or syslog("info", "Error: $? Could not run: $ifconfig $ipRec->{'dev'} down");
}

# Close syslog.
#
closelog();

# Close lockfile.
close (LOCKFILE);

# Exit.
#
exit(0);

# Functions
#
sub daemonize() {
        open STDIN, '/dev/null' or die "$progname: can't read /dev/null: $!";
        open STDOUT, '>/dev/null' or die "$progname: can't write to /dev/null: $!";
        defined(my $pid = fork) or die "$progname: can't fork: $!";
        if($pid) {
                # parent
                open PIDFILE, ">$daemon_pidfile" or die "$progname: can't write to $daemon_pidfile: $!n";
                print PIDFILE "$pidn";
                close(PIDFILE);
                exit;
        }
        # child
        setsid or die "$progname: can't start a new session: $!";
        open STDERR, '>&STDOUT' or die "$progname: can't dup stdout: $!";
}

Linode Dynamic DNS Bash Script

Monday, May 11th, 2009

So Mark Walling was working on an ash script for his router running OpenWRT to update Linode's DNS Manager with his IP address. I liked the idea of a simple shell script to update without needing to install libraries for Perl or Python.  I took his script and tried to adapt it to get the DOMAINID and RESOURCEID using sed or awk.  Those utilities seem great for manipulating multiline files of text, but I wasn't getting anywhere trying to parse one line of JSON from wget.  So I used perl to extract the id numbers.  I believe OpenWRT has some sort of perl with limited functionality, so maybe this will work with that or at least be easily adapted.  This could also be easily modified to use curl instead of wget. I suspect someone out there will find this useful.

#!/bin/bash
#
# Script to update Linode's DNS Manager for a given name.
#

# Things you need to change.
APIKEY=$(cat ~/.linode-apikey)
LASTIP="/tmp/lastip"
DOMAIN="domain.com"
SOAEMAIL="hostmaster@domain.com"
STATUS="1"
RRTYPE="A"
RRNAME="home"
IFACE="eth0"

# Shouldn't need to change anything below here.

WGET="wget -qO - https://api.linode.com/api/"
NEWIP=$(ifconfig $IFACE | head -n2 | tail -n1 | cut -d: -f2 | cut -d' ' -f1)
test -e $LASTIP && OLDIP=$(cat $LASTIP) || OLDIP=""

if [ x"$OLDIP" = x"$NEWIP" ]; then
  logger "No IP address change detected. Keeping $NEWIP"
else
   DOMAINID=$($WGET --post-data "api_key=$APIKEY&action=domainList" | 
        perl -e 'if ( =~ /"DOMAIN":"'"$DOMAIN"'","DOMAINID":([0-9]+),/) { print $1; }')
   RESOURCEID=$($WGET --post-data "api_key=$APIKEY&action=domainResourceList&DomainID=$DOMAINID" | 
        perl -e 'if ( =~ /"RESOURCEID":([0-9]+),"DOMAINID":'"$DOMAINID"',"TYPE":"'"$RRTYPE"'","NAME":"'"$RRNAME"'"/) { print $1; }')
   $WGET --post-data "api_key=$APIKEY&action=domainResourceSave&ResourceID=$RESOURCEID&DomainID=$DOMAINID&Name=$RRNAME&Type=$RRTYPE&Target=$NEWIP"; echo
   $WGET --post-data "api_key=$APIKEY&action=domainSave&DomainID=$DOMAINID&Domain=$DOMAIN&Type=master&Status=$STATUS&SOA_Email=$SOAEMAIL"; echo
   echo $NEWIP > $LASTIP
   logger "Updated IP address to $NEWIP"
fi

Using Linode DNS Manager for Google Apps

Monday, December 8th, 2008

I've hosted a few websites for other people and have used Google Apps to handle mail and calendar needs. This way I'm just using my Linode to only host the webpages. After manually creating the Google Apps DNS records a few times, I decided it'd be best to write a script using Linode's API to automate the process.

Since I haven't written a Perl script from scratch in awhile, I thought it would be good to use that. Michael Greb, who works at Linode, published a Perl module on CPAN called "WebService::Linode". Mike also posted some really helpful information regarding the installation of Perl modules under Ubuntu Linux. Instead of downloading, compiling, and installing the module the old fashioned manual way, you can use dh-make-perl. The dh-make-perl program will do all of that for you and create a Debian package file to install. This way if the CPAN module is ever upgraded or becomes part of the distro, it'll be handled automatically. I used the following commands to install the module (I also had to do the same thing for JSON because my installed version was too old).

$ dh-make-perl --cpan JSON --build
$ dh-make-perl --cpan WebService::Linode --build
$ sudo dpkg -i libjson-perl_2.12-1_all.deb libwebservice-linode-perl_0.02-1_all.deb

On the first run it will prompt for the Linode API key and store it in ~/.linode-api. Subsequent runs will simply read the key from that file. Below is an example of how it's usage.

$ googleapps-dns.pl -h
Usage:
    googleapps-dns.pl [ -d domainname ] [ -m ] [ -c ] [ -f ] [ -v ] [ -h ]

Options:
            -d domainname
                    Specify the domain name for adding the records.  This field is
                    required.
            -c      Add Google Chat's Jabber and XMPP records to route external
                    chat program to Google's services.
            -g      Add CNAMES that point calendar.domainname, docs.domainname,
                    mail.domainname, sites.domainname, and start.domainname to
                    ghs.google.com.
            -m      Add MX and SPF records for routing mail to Google Apps.
            -f      Force deletion of any conflicting records.
            -v      Enable verbose debugging messages.
            -h      Display help and options.

$ googleapps-dns.pl -d example.com -g -m -c
Would you like to delete any conflicting dns records? [Y/n] y
Please enter your Linode API Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
$

I hope this is helpful for other people managing sites hosted on Google Apps. The code is below.

#!/usr/bin/perl
#
# googleapps-dns.pl
#
# Script to populate the Linode's DNS Manager for Google Apps.
#
# Copyright (c) 2008 Patrick Hennessy 
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
#

use strict;
use warnings;

use Getopt::Long;
use Pod::Usage;
use WebService::Linode::DNS;

# File to store apikey.  If undefined or blank, use ~/.linode-apikey.
#
my $apiKeyFile = '';

# Default resource record templates.
#
my %mailTemplate = ( resourceid => 0,  name => '', type => 'MX' );
my %xmppTemplate = ( resourceid => 0, name => '_xmpp-server', protocol => '_tcp', type => 'SRV', weight => 0, port => 5269 );
my %jabberTemplate = ( resourceid => 0, name => '_jabber', protocol => '_tcp', type => 'SRV', weight => 0, port => 5269 );
my %ghsTemplate = ( resourceid => 0, type => 'CNAME', target => 'ghs.google.com' );

# Google Apps DNS Records
#
my @googleMailRR = (
        { %mailTemplate, target => 'aspmx.l.google.com', priority => 10 },
        { %mailTemplate, target => 'alt1.aspmx.l.google.com', priority => 20 },
        { %mailTemplate, target => 'alt2.aspmx.l.google.com', priority => 20 },
        { %mailTemplate, target => 'aspmx2.googlemail.com', priority => 30 },
        { %mailTemplate, target => 'aspmx3.googlemail.com', priority => 30 },
        { %mailTemplate, target => 'aspmx4.googlemail.com', priority => 30 },
        { %mailTemplate, target => 'aspmx5.googlemail.com', priority => 30 },
        { %mailTemplate, type => 'TXT', target => 'v=spf1 a mx include:aspmx.googlemail.com -all' },
);

my @googleXMPPJabberRR = (
        { %xmppTemplate, target => 'xmpp-server.l.google.com', priority => 5 },
        { %xmppTemplate, target => 'xmpp-server1.l.google.com', priority => 20 },
        { %xmppTemplate, target => 'xmpp-server2.l.google.com', priority => 20 },
        { %xmppTemplate, target => 'xmpp-server3.l.google.com', priority => 20 },
        { %xmppTemplate, target => 'xmpp-server4.l.google.com', priority => 20 },
        { %jabberTemplate, target => 'xmpp-server.l.google.com', priority => 5 },
        { %jabberTemplate, target => 'xmpp-server1.l.google.com', priority => 20 },
        { %jabberTemplate, target => 'xmpp-server2.l.google.com', priority => 20 },
        { %jabberTemplate, target => 'xmpp-server3.l.google.com', priority => 20 },
        { %jabberTemplate, target => 'xmpp-server4.l.google.com', priority => 20 },
);

my @googleGHSRR = (
        { %ghsTemplate, name => 'calendar' },
        { %ghsTemplate, name => 'docs' },
        { %ghsTemplate, name => 'mail' },
        { %ghsTemplate, name => 'sites' },
        { %ghsTemplate, name => 'start' },
);

# =============================================================================
# Do not change anything below this line.
# =============================================================================
#
die "This script needs to be run interactively." if (! -t STDIN);

# Handle options.
#
my ($chat, $debug, $domainName, $force, $ghs, $help, $mail);
my $result = GetOptions(
        'd=s'           => $domainName,
        'c'             => $chat,
        'f'             => $force,
        'g'             => $ghs,
        'm'             => $mail,
        'v'             => $debug,
        'help|h|?'      => $help
);

# Usage
#
pod2usage(1) if $help;
pod2usage(1) if ! $domainName;
pod2usage(1) if (! $chat) && (! $ghs) && (! $mail);

# Prompt to see if we should delete conflicting records or just add the new Google records.
#
if (! defined($force)) {
        $force = parseresp(prompt("Would you like to delete any conflicting dns records? [Y/n] "));
}

# Get API key and create api object.
#
my $apiKey = getApiKey($apiKeyFile);
my $api = new WebService::Linode::DNS( apikey => $apiKey );

# Get domain object.
#
my $domain = $api->domainGet( domain => $domainName ) ||
        die "ERROR: Domain not found, please create the initial records first.n";

# XMPP+Jabber Records
#
processRRs($api, $domain, @googleXMPPJabberRR, $force) if $chat;

# Mail Records
#
processRRs($api, $domain, @googleMailRR, $force) if $mail;

# GHS Records
#
processRRs($api, $domain, @googleGHSRR, $force) if $ghs;
# Save the domain.
#
$api->domainSave(%$domain) || die "Couldn't save $domainNamen";

# All done.
#
exit(0);

# Subroutines
#
sub processRRs {
        my $api = shift;
        my $domain = shift;
        my $resourceRecords = shift;
        my $force = shift;

        # Remove any existing records.
        #
        if ($force) {
                for my $nRecord (@$resourceRecords) {
                        for my $eRecord (domainGetRRs($api, $domain->{domainid}, $nRecord->{type})) {
                                if (($eRecord->{type} =~ m/mx/i) && ($eRecord->{name} eq "")) {
                                        deleteRR($api, $eRecord);
                                } elsif (($eRecord->{type} =~ m/txt/i) && ($eRecord->{target} =~ m/v=spf/i)) {
                                        deleteRR($api, $eRecord);
                                } elsif (($eRecord->{type} =~ m/srv/i) && ($eRecord->{name} =~ m/_xmpp-server/i)) {
                                        deleteRR($api, $eRecord);
                                } elsif (($eRecord->{type} =~ m/srv/i) && ($eRecord->{name} =~ m/_jabber/i)) {
                                        deleteRR($api, $eRecord);
                                } elsif (($eRecord->{type} =~ m/cname/i) && ($eRecord->{target} =~ m/ghs.google.com/i)) {
                                        deleteRR($api, $eRecord);
                                }
                        }
                }
        }

        # Create new records.
        #
        for my $record (@$resourceRecords) {
                debug("SAVING RR: TYPE=$record->{type} NAME=$record->{name} TARGET=$record->{target}");
                my $rr = $api->domainResourceSave( 'domainid' => $domain->{domainid}, %$record ) ||
                        die "Could not save TYPE=$record->{type} NAME=$record->{name} TARGET=$record->{target}n";
        }
}

sub deleteRR {
        my $api = shift;
        my $record = shift;

        debug("DELETING RR: TYPE=$record->{type} NAME=$record->{name} TARGET=$record->{target}");
        my $result = $api->domainResourceDelete( resourceid => $record->{resourceid} ) ||
                warn "WARNING: Could not delete TYPE=$record->{type} NAME=$record->{name} TARGET=$record->{target}n";

        return $result;
}

sub domainGetRRs {
        my $api = shift;
        my $domainId = shift;
        my $domainRRType = shift;
        my @domainRRs = ();

        my $rrs = $api->domainResourceList( domainid => $domainId ) ||
                die "ERROR: Could not retrieve domain resource list.n";
        for my $rr (@$rrs) {
                push(@domainRRs, $rr) if ($rr->{type} =~ m/^$domainRRType$/i);
        }

        return @domainRRs;
}

sub getApiKey {
        my $apiKeyFile = shift;
        my $apiKey;

        if (!length($apiKeyFile)) {
                $apiKeyFile = (getpwnam(getlogin()))[7] . '/.linode-apikey';
        }

        if (-f $apiKeyFile) {
                chomp ($apiKey = `cat $apiKeyFile`);
        } else {
                $apiKey = prompt("Please enter your Linode API Key: ");
                system "echo '$apiKey' > $apiKeyFile";
        }

        return $apiKey;
}

sub parseresp {
        my $response = shift;
        my $val = 0;

        chomp($response);
        $val = 1 if ($response =~ m/^y/i );
        $val = 1 if ($response eq "");

        return $val;
}

sub prompt {
        my $promptstring = shift;

        print $promptstring;
        chomp (my $retstring = );

        return $retstring;
}

sub debug {
        my $msg = shift;

        print STDERR $msg, "n" if $debug;
}

__END__

=head1 NAME

googleapps-dns.pl - Create Google Apps DNS Records in Linode DNS Manager

=head1 SYNOPSIS

googleapps-dns.pl [ -d domainname ] [ -m ] [ -c ] [ -f ] [ -v ] [ -h ] 

=head1 OPTIONS

        -d domainname
                Specify the domain name for adding the records.  This field is 
                required.
        -c      Add Google Chat's Jabber and XMPP records to route external
                chat program to Google's services.
        -g      Add CNAMES that point calendar.domainname, docs.domainname, 
                mail.domainname, sites.domainname, and start.domainname to
                ghs.google.com.
        -m      Add MX and SPF records for routing mail to Google Apps.
        -f      Force deletion of any conflicting records.
        -v      Enable verbose debugging messages.
        -h      Display help and options.

=head1 DESCRIPTION

B provides a wizard to populate DNS records in the Linode 
DNS manager for Google Apps.  It requires that the domain already exist.  It 
would create the standard DNS entries which can be pruned manually later.

=cut


css.php