Convert HTTP response body to status code with HAproxy

I’m using HAproxy to load-balance HTTP between two backend nodes. HAproxy is pretty smart about doing this, backing off failing nodes, retrying failed requests on healthy nodes etc. Recently I got a rather strange request: if a node responds with HTTP 200 (which in HTTP-terms means everything’s peachy) and a particular payload, that should be interpreted as a failure. The response should not go to the client, the node should be marked as overloaded and the request should be retried on the other node.

HAproxy can be scripted quite extensively with LUA, so the action plan here is to parse response bodies with LUA, look for the “I’m overloaded” payload and convert that into an HTTP 503 (temporarily unavailable). It turns out that this isn’t quite as straight-forward, required a stack of front- and backend declarations in haproxy.cfg

The system setup looks like this:

Client talks to HAproxy which balances between node1 and node2

HAproxy pipeline

Here’s the code:

global
  daemon
  log stdout  format raw  local0 info
  lua-load /usr/local/etc/haproxy/healthcheck.lua

defaults
  log               global
  default-server init-addr libc,none
  option httplog
  timeout client 30s
  timeout connect 30s
  timeout server 30s
  timeout http-request 30s

frontend frontend_default
   bind *:8000
   mode http
   use_backend backend_default

backend backend_default
   mode http
   balance roundrobin
   option redispatch 1
   retry-on 503 all-retryable-errors
   retries 1
   server proxy1 unix@/tmp/proxy1.sock
   server proxy2 unix@/tmp/proxy2.sock

frontend frontend_proxy1
   mode http
   bind unix@/tmp/proxy1.sock
   http-request use-service lua.rewrite_node1
   use_backend backend_unused

frontend frontend_proxy2
   mode http
   bind unix@/tmp/proxy2.sock
   http-request use-service lua.rewrite_node2
   use_backend backend_unused

backend backend_unused
   mode http
   server devnull localhost:0

frontend frontend_lua1
   mode http
   option httpclose
   bind unix@/tmp/rewrite1.sock
   use_backend backend_node1

frontend frontend_lua2
   mode http
   option httpclose # required, otherwise lua script is routed to the wrong frontend (probably some internal connection kept alive)
   bind unix@/tmp/rewrite2.sock
   use_backend backend_node2

backend backend_node1
   mode http
   server node1 node1:8001

backend backend_node2
   mode http
   server node2 node2:8002

The pipeline declares frontend_default which listens at port 8000 for HTTP requests and forwards those requests as they are to the internal backend backend_default.

backend_default implements the load balancing and back-off mechanism. The backend forwards requests to two local Unix sockets (I didn’t know HAproxy could do this!) which are used by frontend_proxy1 and frontend_proxy2 later in the pipeline. Read here more about retrying failed requests: https://www.haproxy.com/documentation/hapee/latest/service-reliability/retries/retry-on/

frontend_proxy1 and frontend_proxy2 read HTTP requests from backend_default over an Unix socket (each). Both front-ends exists only to squeeze traffic through a LUA script.

The LUA script forwards HTTP requests to two further frontends (frontend_lua1, frontent_lua2) over Unix sockets and looks in the response for the overloaded message. If it finds the message, it changes the HTTP response code to 503.

frontend_lua1 and frontend_lua2 communicate with the LUA script via Unix sockets and forward requests to backend_node1 and backend_node2.

backend_node1 and backend_node2 pass the HTTP requests channeld through by frontend_lua1 and frontend_lua2 to node1:8001 and node2:8002 respectively.

LUA script

print("health check script initialised")

errorCodePattern = "faultCode.+value.+i4.+1007.+i4"
default_server_weight = 128
last_weight_restauration = os.time()
weight_restauration_interval_sec = 30

function dump(o)
   if type(o) == 'table' then
      local s = '{ '
      for k,v in pairs(o) do
         if type(k) ~= 'number' then k = '"'..k..'"' end
         s = s .. '['..k..'] = ' .. dump(v) .. ','
      end
      return s .. '} '
   else
      return tostring(o)
   end
end

function create_request_handler(sn)
  local socketname = sn
  function process_request(applet)
        -- --------------------------------------------
        --
        -- bufferize request
        --
        -- --------------------------------------------
        local errorCodePattern = "faultCode.+value.+i4.+1007.+i4"
        local method = applet.method
        local path = applet.path
        local qs = applet.qs
        if qs ~= nil and qs ~= "" then
                path = path .. "?" .. qs
        end

        -- --------------------------------------------
        --
        -- forward request
        --
        -- --------------------------------------------
        local httpclient = core.httpclient()

        -- select function according with method
        local cli_req = nil
        if method == "GET" then
                cli_req = httpclient.get
        elseif method == "POST" then
                cli_req = httpclient.post
        elseif method == "PUT" then
                cli_req = httpclient.put
        elseif method == "HEAD" then
                cli_req = httpclient.head
        elseif method == "DELETE" then
                cli_req = httpclient.delete
        end

        -- copy and filter headers
        headers = applet.headers
        headers["connection"] = nil -- let haproxy manage connection
        headers["accept-encoding"] = nil -- remove this header to avoid compression

        local host = headers["host"]
        if host ~= nil then
                host = host[0]
        else
                host = "www"
        end

        local urlParam = "http://" .. host .. path

        -- execute requests
        local unixSocketName = "unix@/tmp/" .. socketname
        print("Forwarding request to " .. unixSocketName)
        local response = cli_req(httpclient, {
                url = urlParam,
                headers = applet.headers,
                body = applet:receive(),
                dst = unixSocketName
        })

        -- extract body
        print(dump(response.headers))
        local body = response.body
        print(body)

        if (string.match(body, errorCodePattern)) then
                applet:set_status(503)
                print("Detected error code in response from server")
        else
                applet:set_status(response.status)
        end

        -- --d------------------------------------------
        --
        -- Trandform body
        --
        -- --------------------------------------------
        -- body = string.gsub(body, "(>[^<]*)([lL][uU][aA])", "%1<b style=\"color: #ffffff; background: #ff0000;\">%2</b>")

        -- --------------------------------------------
        --
        -- forward response
        --
        -- --------------------------------------------
        local name
        local value
        for name, value in pairs(response.headers) do
                local part
                local index
--              if string.lower(name) ~= "content-length" and
--                 string.lower(name) ~= "accept-ranges" and
--                 string.lower(name) ~= "transfer-encoding" then
                        for index, part in pairs(value) do
                                applet:add_header(name, part)
                        end
--              end
        end
        applet:start_response()
        applet:send(body)
  end

  return process_request
end

core.register_service("rewrite_node1", "http", create_request_handler("rewrite1.sock"))
core.register_service("rewrite_node2", "http", create_request_handler("rewrite2.sock"))

The script was heavily inspired (to the extend of copy & pasting more than half of it verbatim) by Therry Fournier’s post “HAProxy: Use LUA to rewrite the body” https://www.arpalert.org/haproxy-rewrite-body.html

Caveats

“option httpclose” must be specified with the LUA front-ends, otherwise HAproxy seems to be keeping some logical connection open and won’t balance between the backend nodes unless the connection times out.

I tried this only with v2.7.5 and Docker. I did not try this in a production setting with considerable workload. I did not try this with a considerable uptime.