Archive

Tag Archives: HAProxy

We, being the company I work for, recently set up a mysql galera cluster and haproxy to load balance connections between the nodes. Haproxy has a mysql health check, but it only logs into the server, and we wanted a bit more than that (galera’s rsync option puts the server that is being synced from in read_only mode). What I didn’t want to do was install apache or similar because I wanted to leave as much of the systems resources available to mysql. I solved the problem with a perl script.

Before I move on, I should mention that I don’t like perl. Other languages, such as Go, provide just as easy of a solution, but perl is installed on pretty much all linux distros, and, therefore, was less setup. The backbone of the script is the HTTP::Simple::Server:CGI package. My version of the script weighs in at a whole 26 lines of code. Here it, mostly, is.

#!/usr/bin/perl

use File::Pid;

{
package MyWebServer;

use HTTP::Server::Simple::CGI;
use base qw(HTTP::Server::Simple::CGI);

    sub handle_request {
        my ($self, $cgi) = @_;
        $isFine = 0;
        //-----
        //do your checking logic here
        //-----
        if($isFine) {
            print "HTTP/1.0 200 OK\r\n";
            print $cgi->header;
        } else {
            print "HTTP/1.0 503 Service Unavailable\r\n";
            print $cgi->header;
        }
    }
}

my $pidfile = File::Pid->new();
if(!$pidfile->running) {
    my $server = MyWebServer->new(12345);
    $server->host('YOUR SERVERS IP GOES HERE');
    my $pid = $server->background();
    $pidfile->pid($pid);
    $pidfile->write;
}

The above code checks to see if a running PID for the script exists and exits if it does (the if block towards the bottom). It then sets up the server to listen on part 12345, use whatever port you want. The next line tells it to listen on a specific ip address, I set that from chef as part of the .erb that builds this script, you could pass a parameter to the script if you don’t want to do that ($ARGV[0]). It then creates the server in the background and writes the PID file.

Of course, the real action is in the handle_request function in the package. That function gets called every time the script receives an http request. All mine does, and you could do a lot more here, is collect some information about the state of the server, a bit more on that in a second, and either returns a status of 200 or 503 which is all haproxy cares about. If your load balancer checks for actual content in the response then, you would add some prints after the $cgi->header calls.

As I mentioned in the first paragraph, the reason we set this up was to discover if the server happens to be in read_only mode. Thus, all my check does it shell out to mysql with a -e option to show global variables, and then runs a regex over that for read_only being set to off.

I’ve also set cron up to run the script ever minute, which is why the PID stuff is in there. Pretty simple really.

Advertisements

Recently, I needed to add https support to our dev installs of our web app. The app itself needed to know it was using https, to generate proper urls and the like, so terminating the ssl connection at the proxy was not a viable solution for me. HAProxy added support for SSL in 1.5 but this article isn’t about that because I’m using CentOS and therefore am stuck with HAProxy 1.4.

First up, how not to solve this problem. My first thought was that if I put HAProxy in tcp mode it shouldn’t know anything about whether the connection was SSL or not. This did not work. Unfortunately my notes don’t say why this didn’t work but I assume either HAProxy was spitting out BADREQ with PR– in the logs or the payload was getting messed up and causing errors in negotiations.

Enter stunnel. Stunnel is an SSL tunnel and is what I used to handle the https request. Stunnel can be configured in either a server mode, which terminates SSL connections, or as a client, which initiates SSL connections. This solution uses both. The general solution, which I found here, is to have the https connection received by a stunnel server, who forwards the now http connection to haproxy, who forwards the http connection to a stunnel client, who changes it back to https and forwards it along to server.

–https–> stunnel server –http–> haproxy –http–> stunnel client –https–> web server

Not pretty, but effective, and because all of the traffic between stunnel and haproxy is on localhost, it’s relatively fast.

The first thing needed is to get stunnel installed. It’s in yum.

Now to set up stunnel. I made a folder at /etc/stunnel to hold my configs and .pem file, which is the .key and .crt file concatenated together. I placed the .pem file in that folder. Now you will need a config file for the stunnel server and client. I named my server.conf and client.conf. You might be able to do this with 1 config file, I’m not that familiar with stunnel. In both config files you will need/want 5 global settings defined:

cert=<path to your .pem file>
pid=/var/run/stunnel_(server|client).pid or something similar
output=/var/log/stunnel_(server|client).log or something similar
socket=l:TCP_NODELAY=1
socket=r:TCP_NODELAY=1

Basically, define where the pem lives because we need that for the SSL handshake, define a pid file and a log file because those are handy, and two lines that are basically gibberish to me but seem important (all of the google results seemed to have them). I would also suggest you add in foreground=yes while you test the config files so that you can easily see what is happening and kill (ctrl-c) the process to make changes. The next bits are define what the stunnel client is actually going to be doing. For the server:

[https]
accept=443
connect=8081

That basically says to listen on port 443 and if you get something there forward the connection to 8081. Port 443 is important but you could change 8081 to be something different. The client will look at lot the same:

[https]
client=yes
accept=8082
connect=<server ip address>:443

There we tell stunnel that it’s to operate in client mode (client=no is default which is why it wasn’t in the server config), to listen on 8082 (which you could change to something else), and to connect to our webserver on 443. If you had multiple web servers you could put multiple connect lines in and it will round robin the connections.

The last thing we need to change is our HAProxy config. This, at it’s most basic, would look something like:

frontend my_ssl_webpage
bind :8081
default_backend my_ssl_webpage_backend

backend my_ssl_webpage_backend
reqadd X-Forwarded-Proto:\ https
server stunnel1 127.0.0.1:8082

Now, if you restart HAProxy, and start stunnel for both config files (sudo stunnel <path to config file>), you should have https requests arriving on your webserver.

 

Bonus:
Found a great manual for HAProxy 1.4. Here is the link:
http://cbonte.github.io/haproxy-dconv/configuration-1.4.html