In September I wrote a few things about how I use SAM Script Templates – specifically with PowerShell. One of the more powerful things in the SAM arsenal is the ability to return multiple components in a single script. I’ve used this in any number of ways, but the one that gives me the most satisfaction is a PowerShell script that I wrote for Linux.
First things first – I’m a Windows guy. I won’t bother defending this because it’s the fault of my upbringing. My parents used Windows, my first real job used Windows, my current job uses Windows. It’s what I know. However, I’m also a non-conformist at times and on the side, I secretly see other operating systems.
In my lab at the office I have multiple flavors of Linux running – including Ubuntu, CentOS, Redhat, SUSE, and even Oracle. Much of this is because I love to explore. I’ve always said that I’m a technologist first.
Two years ago, was the first time that I heard about HA Proxy in a THWACKcamp session with some good friends. After speaking about runbooks with Dave and Leon, I realized that this was a powerful little piece of software that I should know a little more about.
So I set myself up a little test bed. I’ve got four web servers running Apache. I’ve got a separate MySQL server, a server for an NFS share, and one to run the HA Proxy. Basically, it looks like this:
Needless to say it took me a little bit to setup right (I mentioned that I know Windows well, right?). After I got it running, I could point my SAM templates at it for Apache and MySQL without a problem (providing you remembered to install the drivers for MySQL on your Orion server). When I tried to do the same for HA Proxy, I ran into some trouble. I don’t think that anything is wrong with the template – I think it was an issue with my particular build.
I’ll repeat – I’m not someone who knows Perl. As a frequent scripter, I can read the scripts and understand what they are doing, but I’ve got zero chance of troubleshooting anything more difficult than a typo. Let’s take a quick peek at the script together.
$sport=$ARGV[0];
$halogin=$ARGV[1];
$hapass=$ARGV[2];
$uri=$ARGV[3];
if ($sport eq '')
{
print "Message: Can't find \"haport\" argument. Please provide TCP port number of running haproxy stats page.\n";
exit 1;
}
if ($halogin eq '')
{
print "Message: Can't find \"haproxy_login\" argument. Please provide login for haproxy stats page.\n";
exit 1;
}
if ($hapass eq '')
{
print "Message: Can't find \"haproxy_password\" argument. Please provide password for haproxy stats page.\n";
exit 1;
}
if ($uri eq '')
{
print "Message: Can't find \"uri\" argument. Please provide URI of stats page.\n";
exit 1;
}
$arg="/".${uri}.";csv";
$cmd="curl -u ".${halogin}.":".${hapass}." \""."http://127.0.0.1".":".${sport}.${arg}."\"";
$out=`$cmd`;
print $out;
$exit=`echo $?`;
if ( $exit != 0 )
{
print "Message: Problems with execution \"curl $url\" command. Try to run command on server.\n";
exit 1;
}
@line=split("\n",$out);
@names=split(",",$line[0]);
for ($i=0;$i<@line;$i++)
{
if ($line[$i] =~ "Service Unavailable")
{
print "Message: Wrong URI argument or stats page not configured.\n";
exit 1;
}
@value=split(",",$line[$i]);
if ($value[0]=~"stats")
{
if ($value[1]=~"FRONTEND")
{
for ($j=0;$j<@value;$j++)
{
if ($names[$j]=~"scur")
{
$stat1=$value[$j];
}
if ($names[$j]=~"smax")
{
$stat2=$value[$j];
}
if ($names[$j]=~"stot")
{
$stat3=$value[$j];
}
if ($names[$j]=~"bin")
{
$stat4=$value[$j];
}
if ($names[$j]=~"bout")
{
$stat5=$value[$j];
}
if ($names[$j]=~"dreq")
{
$stat6=$value[$j];
}
if ($names[$j]=~"dresp")
{
$stat7=$value[$j];
}
if ($names[$j]=~"ereq")
{
$stat8=$value[$j];
}
if ($names[$j] eq "status")
{
$mess9=$value[$j];
if ($mess9=~"OPEN")
{
$stat9=1;
}
elsif ($mess9=~"UP")
{
$stat9=2;
}
elsif ($mess9=~"NOLB")
{
$stat9=3;
}
elsif ($mess9=~"MAINT")
{
$stat9=4;
}
elsif ($mess9=~"DOWN")
{
$stat9=5;
}
else
{
$stat9=0;
}
}
}
}
}
}
print "Statistic.Current_Sessions: $stat1\n";
print "Statistic.Max_Sessions: $stat2\n";
print "Statistic.Total_Sessions: $stat3\n";
print "Statistic.Bytes_In: $stat4\n";
print "Statistic.Bytes_Out: $stat5\n";
print "Statistic.Denied_Requests: $stat6\n";
print "Statistic.Denied_Responses: $stat7\n";
print "Statistic.Request_Errors: $stat8\n";
print "Statistic.Service_Status: $stat9\n";
print "Message.Service_Status: $mess9\n";
exit 0;
So I can see what’s happening here. HA Proxy is creating a CSV of the status of the program on a web page and port with a specific set of creds. As I’ve stated before I have NO CLUE how to troubleshoot this, and my mind immediately went to PowerShell. I wonder if I could just run this from the Orion server as a PowerShell Script? Then my mind went to a strange place. Didn’t I just see that PowerShell Core is available for Linux?
I know what you are saying – hasn’t this gone far afield from the title of this post? Yes, but this is all background, so please be patient.
This is the results that would get returned by just the web call:
# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,
webfarm,FRONTEND,,,1,13,2000,636126,43848585,1332131817,0,0,413269,,,,,OPEN,,,,,,,,,1,1,0,,,,0,1,0,27,,,,0,137205,32,497944,944,0,,1,27,636126,,,0,0,0,0,,,,,,,,
webfarm,eastweb01v,0,0,0,7,,13375,1279842,275240934,,0,,0,263,0,0,UP,1,1,0,209,1,2383634,614,,1,1,1,,13375,,2,0,,7,L4OK,,367,0,13102,8,0,2,0,0,,,,0,0,,,,,493,,,0,44,7244,7779,
webfarm,eastweb02v,0,0,0,6,,13376,1275553,279300535,,0,,0,2,0,0,UP,1,1,0,147,1,2388411,516,,1,1,2,,13376,,2,0,,6,L4OK,,40,0,13366,8,0,0,0,0,,,,0,0,,,,,272,,,0,44,5178,5622,
webfarm,eastweb03v,0,0,0,7,,13378,1297214,272231561,,0,,0,359,0,0,UP,1,1,0,154,0,4347438,0,,1,1,3,,13378,,2,0,,6,L4OK,,33,0,13011,7,0,1,0,0,,,,0,0,,,,,233,,,0,40,5718,6169,
webfarm,eastweb04v,0,0,0,6,,13378,1299638,270123627,,0,,0,315,0,0,UP,1,1,0,125,0,4347438,0,,1,1,4,,13378,,2,0,,6,L4OK,,6,0,13052,9,0,2,0,0,,,,0,0,,,,,193,,,0,46,6612,7104,
webfarm,BACKEND,0,0,0,12,200,53507,43848585,1332131817,0,0,,0,939,0,0,UP,4,4,0,,0,4347438,0,,1,1,0,,53507,,1,0,,25,,,,0,52531,32,0,944,0,,,,,0,0,0,0,0,0,0,,,0,20,2331,2523,
It’s a little basic, but pretty easy to parse. The first line (started with the “#<space>”) gives the field names. Then each of the rows are raw data associated with that field. If I were to nix the “#<space>” from the front, it’s pretty much a beautiful formatted, CSV right?
So then I referred back to the original script to decipher what some of the stats meant in English. There are 62 columns (at present) and I don’t need all of them. Many are blank or 0’s most of the time. I picked my 10 favorite because SAM component monitors can only handle 10 statistics from a single call. This was enough based on what I wanted to use. If you are following along with the home game, the ones I chose were:
| Stat Number | Field | English Name |
|---|---|---|
| 1 | qcur | Current_Requests |
| 2 | scur | Current_Sessions |
| 3 | stot | Total_Sessions |
| 4 | bin | Bytes_In |
| 5 | bout | Bytes_Out |
| 6 | dreq | Denied_Requests |
| 7 | dresp | Denied_Responses |
| 8 | econ | Connection_Errors |
| 9 | eresp | Response_Errors |
| 10 | status | Status |
I won’t lie: I had to look up what some of these actually meant.
Then I began crafting my PowerShell template. I started as I do (95% of the time) with my own personal SAM template and then filled in the blanks.
The pseudo-code goes something like this:
- Check to see if I have everything I need (the web farm name, the port, the Uri segment)
- Get the results of the web call (get the raw data above, making sure to split it at each end of line)
- Nix the “#<space>” from the first line (so it can be the “headers” in my CSV)
- Find the farm member I’m interested in (FRONTEND, BACKEND, or a server’s name)
- Output the statistics.
Step 5 is the one that gets most people. Many SAM script templates return only a single value – a single number and (optionally) a message. In PowerShell it normally takes the form:
Write-Host "Message: This statistic returns 11"
Write-Host "Statistic: 11"
Since each of my statistics refer to a single web farm and all those statistics have to do with said farm, it’s better to bundle them together.
The proper formatting for that uses the period to separate elements (dot-delimited). So, if we take the above and expand it for an additional statistic, it looks like this:
Write-Host "Message.FirstStat: The first statistic is 42"
Write-Host "Statistic.FirstStat: 42"
Write-Host "Message.SecondStat: The second statistic is 3.14"
Write-Host "Message.SecondStat: 3.14"
Remember that although there are multiple statistics returned, this is a single component monitor, so it only needs one exit code.
Now SAM will capture both statistics when you run the script once. This is especially helpful when working with services that may limit the number of calls (like many web-based API’s).
Putting all the parts together yields the following script.
<#
Parameters:
1) Web Farm name ( "FRONTEND", "BACKEND", "eastweb01v", etc.)
2) Port on which the proxy stats page lives (if not "80")
3) Uri Segment to the stats page ( "/haproxy?stats;csv" )
Full Uri for request is built from:
"http://" + IP from Orion Variable + ":" + Port Parameter + Uri Segment Parameter
EXAMPLE: "http://" + "10.1.100.150" + ":" + "80" + "/haproxy?stats;csv" = "http://10.1.100.150:80/haproxy?stats;csv"
#>
$ServerIP = "127.0.0.1"
if ( -not $args[0] )
{
$svname = "FRONTEND"
}
else
{
$svname = $args[0]
}
if ( -not $args[1] )
{
$Port = "80"
}
else
{
$Port = $args[1]
}
if ( -not $args[2] )
{
$UriSegment = "/haproxy?stats;csv"
}
else
{
$UriSegment = $args[2]
}
try
{
$Creds = New-Object -TypeName PSCredential -ArgumentList "haproxy_login", ( "haproxy_password" | ConvertTo-SecureString -Force -AsPlainText )
$ProxyReport = ( Invoke-WebRequest -Uri "http://$( $ServerIP ):$Port$UriSegment" -Credential $Creds -AllowUnencryptedAuthentication | Select-Object -ExpandProperty Content ) -split "`n"
$ProxyReport[0] = $ProxyReport[0].Replace("# ", "")
$ProxyStats = $ProxyReport | ConvertFrom-Csv
$WebStatus = $ProxyStats | Where-Object { $_.svname -eq $svname }
$StatusHash = @{ "OPEN" = 1
"UP" = 2
"NOLB" = 3
"MAINT" = 4
"DOWN" = 5
}
Write-Host "Statistic.Current_Requests: $( [int64]( $WebStatus.qcur ) )"
Write-Host "Statistic.Current_Sessions: $( [int64]( $WebStatus.scur ) )"
Write-Host "Statistic.Total_Sessions: $( [int64]( $WebStatus.stot ) )"
Write-Host "Statistic.Bytes_In: $( [int64]( $WebStatus.bin ) )"
Write-Host "Statistic.Bytes_Out: $( [int64]( $WebStatus.bout ) )"
Write-Host "Statistic.Denied_Requests: $( [int64]( $WebStatus.dreq ) )"
Write-Host "Statistic.Denied_Responses: $( [int64]( $WebStatus.dresp ) )"
Write-Host "Statistic.Connection_Errors: $( [int64]( $WebStatus.econ ) )"
Write-Host "Statistic.Response_Errors: $( [int64]( $WebStatus.eresp ) )"
Write-Host "Message.Service_Status: $svname is currently $( $WebStatus.status )"
Write-Host "Statistic.Service_Status: $( $StatusHash[$WebStatus.status] )"
$ExitCode = 0
}
catch
{
Write-Host "Error detected: $( $Error[-1] )"
$ExitCode = 1
}
finally
{
Write-Warning -Message "ExitCode: $ExitCode"
exit $ExitCode
}
Executing it against my HA Proxy gives me the following results:
Statistic.Current_Requests: 0
Statistic.Bytes_In: 43922795
Statistic.Bytes_Out: 1335288176
Statistic.Denied_Requests: 0
Statistic.Denied_Responses: 0
Statistic.Connection_Errors: 0
Statistic.Response_Errors: 0
Message.Service_Status: FRONTEND is currently OPEN
Statistic.Service_Status: 1
This is exactly how the PowerShell script component expects to see these kind of results.
So, yes, we went far afield, but if you stuck with me, you probably gained a little bit of insight on how to get multiple statistics working with your SAM component monitors.
If you have any questions, please feel free to ask me. I’d love to hear your thoughts.