I have looked at many Asteisk packages and AdminsParadise's implementation was one.  I was interested in their cluster setup and how well it worked.

AdminParadise's implementation does not work well at all.  Let me begin by saying Clustering is VERY possible with Asterisk and CentOS 5 is the easiest I found to do it with. 

First they copied the exact code presented by salahuddin66 Dhaka, Bangladesh in 2006. His code is linked to in my links section. 

These are the problems I found with their implementation:

Their install failed to properly copy a file called amportalhb.  This file is nothing more than an old way to make amportal respond to a start and stop command with results which it now does so only a symbolic link is necessary to the amportal file.  All files that must be executed upon a heartbeat fail must respond to a start and stop command - which is why this file was made back in the days when amportal was not using the correct syntax.

Lets get into the real problems - select more to read the remainder of this article...

Next, they are using old DRBD and heartbeat RPMs along with the kernel patch.  DRBD (Distributed Resource Block Device) is now in the yum repository and so is heartbeat so a yum update will blow the cluster unless the kernel is re-patched.

Next they use a very manual way of installing the shared block (partition) device.  Much of it could be automated way beyond what they accomplished.  Additionally, when the cluster is broken it loses sync and there seems to be no way of re-syncing it without manually telling one of the 'now secondary' clusters to blow away the other and become primary.  This is not an easy task when a newbie wants to try clustering.

The biggest problem lies in updating Asterisk, or any of the packages that are installed such as FreePBX.

To understand the problem you must understand the way they are doing the asterisk cluster.  They are creating a single partition that will be duplicated between two servers using the DRBD routines.  This should keep the two partitions identical even though they are on different units. (DRBD is like running RSync constantly - but in sync mode)

The resources that are on this ‘mirrored across units’ partition are the mysql data, the asterisk directory, the sendmail directory,  the httpd directory, and several more critical parts to get asterisk to function.  Basically their install copies the required directories to the cluster partition but also leaves the original directory where it was originally designed to be. (Confusion to begin later!)

Now picture the start of the cluster (power on). One unit becomes the primary and the other becomes the secondary copying the data on the partition in real time sync mode.  When the primary starts, it runs a program ‘drbdlinks’ which creates mnemonic links for each directory or resource needed that is on the mirrored partition.   The original directory (IE:Asterisk) is temporarily renamed (asterisk.drbdlinks) to allow this program to create the link.

During the start of this primary the proper services are also started after the links are made.  MySql is first followed by sendmail, httpd, vsftpd  and amportal (I also kick in samba).  There are more in their implementation such as hylafax, but I’m looking at my own cluster which does not have all the extras they do.

When a machine is in control (primary) the updates go to the mirrored partition (on the secondary).  If the primary fails the secondary takes over and starts the required mnemonic links and services to run asterisk.  If the main unit comes back on line, depending on your configuration, that unit can request to become primary again as soon as the mirror is up to date.

This does not happen on the AdminParadise’s implementation.  It just gets lost.

In case you’re wondering how two machines with fixed IP’s can have the same IP, they can’t.  A third shared cluster IP is setup dynamically and is what is used by the current primary to talk to the phones and internet.  When a unit becomes a primary it attempts to query and claim this shared IP before it is allowed to become the primary and start launching services like drbdlinks and mysql.

Now to the confusion!  You recall I said the original directory still exists?  Try and guess how you can update the code to asterisk, zaptel, hylafax or any other program.  Since the data was originally copied to the mirror partition but yet still exists in its original location, updates when in cluster mode would look at the shared drive, while updates when the cluster is down would look at the old original location.  This is compounded when one unit is updated while the cluster is running since any new data based on the software update will end up in the shared location and on both units.  Now try to update the other unit and it may see that the data is updated already and choke.

An example of this problem is clearly visible with FreePbx. They keep a version code in the mysql database.  If you update the code on the primary, the database (which is on the shared partition) on both the primary and secondary PC’s will have that updated version of the data, yet the software update is only on the primary (The code is not on the shared partition).

If you break the cluster and make each one primary, standalone, all will go well.  If you break the cluster but do not force it into primary mode – thereby NOT creating the symbolic links, you will be updating the original data in the original directories not in the shared partition - disaster!

I know this is a little technical but for those of you who want to play with clustering do not do any software updating like yum update.  Freepbx module updates are OK since the only changes fall into the shared area, but do not change software levels without first doing this procedure:

Make sure the cluster data is in sync:

Bring down the secondary

Update the software on the primary and test it.

Turn off the primary.

Start the secondary and abort the cluster waiting notice

This should make it come up as a primary

Make sure it is up as a primary!

If not primary, use dbadmin to make it primary

Update the software and test it

Turn off the new primary.

Restart both machines

They will probably start as both secondary’s now so force one to be primary.