Silicon Valley Products, Corp. - Notification of a down T1 line

Overview:

I've read and discussed various methods to track my clients T1 status. I don't like to reinvent the wheel but sometimes it is necessary. Using Sangoma's T1 (A101 & A102) cards, but any card would work, I came to realize that most of the T1 outages did not stop the server or Asterisk.

This is not entirely true. There are cases where Asterisk crashes from an oscillating T1 line, but I noticed there is a usually a lag - enough to send out warnings. Warnings by email, txt message, beeper and phone calls using the backup phone lines and the email system.

So realizing we have a minute or two before the possible final crash, at best no T1, I wrote a few routines to notify me and the client so the proper steps can be taken.

It goes like this. A cron job is run every minute or two checking the PRI status (I use PRI interfaces). If the PRI is down a script is run to email the customer instructions on who to contact, the account, circuit and which line to do an RCF (Remote Call Forward) on. In addition, a call is made to my office, my cell, the clients primary operator extension and a beeper.

There is a failsafe built in to stop too many notifications by using the Asterisk database to hold variables. These variables are set and cleared as the T1 line status changes. I do not want to be notified more than three times that the T1 is still down and I want to know when it comes back up.

Lets continue on to the coding....

Coding Overview:

First we will use a cron job to call our script. In this script we check the PRI Status like so:

#get the current pri status

stat=$(echo $(asterisk -rx "pri show span 1"))

notify=false

if (echo "$stat"|grep -E 'In Alarm'>/dev/null);then

echo "T1 Alarm"

notify=true

if (echo "$stat"|grep -E 'Down'>/dev/null);then

echo "T1 Down"

notify=true

if (echo "$stat"|grep -E 'No PRI running on span'>/dev/null);then

echo "No PRI"

notify=true

if (echo "$stat"|grep -E 'Unable to connect to remote asterisk'>/dev/null);then

echo "Asterisk Down"

notify=true

#notify=false

if $notify;then

echo "Houston We have a problem"

I will try not to give too much coding as I will make the files available in the download area. The above code is a small part of the code that is run periodically to check to see if the T1 is up. This could be changed for any device in Asterisk - if it has a database status we can watch it.

The next part of this script is to check to see if this condition - weither true or false is the same as the previously found status and if a notification should be attempted again. This is done by storing a few variables in the Asterisk database like so:

(This code is used on a state change from up to down)

#remember the colon at the end so data is fielded

asterisk -rx "database put PRI DOWN true:"

asterisk -rx "database put PRI TIME $(date +%F-%T):"

asterisk -rx "database put PRI COUNT 1:"

If this is the first time we have gone down, or if we are to keep notifying, then the code begins its notification routines.

Notifications:
Email notifications are completed right in the same cron script. Since the sendmail system is independent of Asterisk and Asterisk may not be running, a txt message or email may be your only notice of a problem. This script will attempt to queue the calls needed into Asterisk by creating a dial message and moving it to the outgoing directory for placement by Asterisk.

Internal calls depend on Asterisk functioning and external calls depend on Asterisk and available backup lines.

Asterisk Configurations:
If the T1 down calls can be made by Asterisk there will need to be code to process the call as well as recorded messages to play. I created several voice files to allow a user to navigate various information and options. As an example, the inital call repeats the down message and instructs the user to press a few different options. One option is to give the user all information about the circuit, numbers to call and what to say. Another option is to reboot the server. This reboot option is nice to have since it is a software controlled shutdown and this may correct the problem.

Customer Options:
If Asterisk is still operating when the down condition happens, the customer will receive a call. This call will give them the option of letting the system place a call to the service provider while they wait on the line. In addition, the customer can dial into this alert system (even if the T1 is up) to reboot the server. Making this option available to the customer can avoid the console use the once or twice you may need them to reboot the server.

Configuration:
I created a file (T1.conf) which resides in the Asterisk directory and controls the notification options. There will be some changes necessary to the pristatus code, since you would want to place your own message for the emails as well as place your own callerid codes (from extension 999911...) in the dial routines. The code that is added to asterisk also has a dial string which you would want to change. This dial string allows the customer, upon receiving a down message, to place a call to the service provider using the backup lines. You would have to change the dial prefix to use their backup lines and of course put the number for the clients carrier. The six voice files needed will not be included in the download, but the text I used will be listed.


	All logos and trademarks in this site are property of their respective owner. The comments are property of their posters, all the rest © 2005 by me. You can syndicate our news using the file backend.php or ultramode.txt