Is This a Bug? Varnish Client IP Address Appears Twice

If you’re running the Varnish caching system, you may have noticed that the client’s IP address is duplicated, like “10.0.0.1, 10.0.0.1”, in your server logs.

One of the standard recommendations for Varnish setups is to assign the client’s IP address to the X-Forwarded-For header so it can be captured by your server logs or code. To do this, you add the following to your vcl_recv subroutine:

sub vcl_recv { 

    remove req.http.X-Forwarded-For;
    set req.http.X-Forwarded-For = client.ip;

    etc.
}

And then set your Apache (2.2) log format to something like the following:

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" varnishcombined

Yet, even though you’re doing exactly as instructed, the client.ip is appearing twice. The X-Forwarded-For header shows the duplicated IP address in varnishlog, indicating that the duplication is caused by Varnish, not your back-end system.

Take a look at the commented-out logic in the Varnish default.vcl:

# Below is a commented-out copy of the default VCL logic.  If you
# redefine any of these subroutines, the built-in logic will be
# appended to your code.
# sub vcl_recv {
#     if (req.restarts == 0) {
#       if (req.http.x-forwarded-for) {
#           set req.http.X-Forwarded-For =
#               req.http.X-Forwarded-For + ", " + client.ip;
#       } else {
#           set req.http.X-Forwarded-For = client.ip;
#       }

The Varnish documentation adds: “Whether [the default logic] is executed or not depends on whether your own VCL code terminates that specific state or not.”

In other words, since you removed and then defined a new X-Forwarded-For header, the default Varnish logic executes, detects the X-Forwarded-For header you added, and then helpfully appends a comma and a second copy of client.ip.

So this isn’t a “bug”, but I think most people would infer that a Remove command would stop any further manipulation of the header.

Varnish Client IP Workarounds

So how to make it stop? Here are a couple of workarounds you can try.

1. Install the Apache RPAF module and just use Apache’s default “combined” log format. IP addresses will start appearing correctly in your server logs at least. However, this doesn’t fix the X-Forwarded-For header, so any code that detects client IPs will have to strip the duplicated address:

if( isset( $_SERVER[ 'HTTP_X_FORWARDED_FOR' ] ) ) {
   $real_ip = $_SERVER[ 'HTTP_X_FORWARDED_FOR' ];
   $ip_array = explode(",", $real_ip);
   $final_ip = $ip_array[0];
}

2. Define a new header and append the client’s IP to it, then reference this custom header in your log files and elsewhere.

Varnish:

remove req.http.X-Forwarded-For;
set req.http.X-Forwarded-Fixed = client.ip;

Apache RPAF:

RPAFheader X-Forwarded-Fixed

Apache log files (if not using RPAF):

LogFormat "%{X-Forwarded-Fixed}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Custom code:

if( isset( $_SERVER[ 'HTTP_X_FORWARDED_FIXED' ] ) ) {
   $real_ip = $_SERVER[ 'HTTP_X_FORWARDED_FIXED' ];
}

The post Is This a Bug? Varnish Client IP Address Appears Twice appeared first on GeoffStratton.com.