Varnish Support

This wiki page is for documenting the steps for adding Varnish Cache support to t3c.

ATS Configuration to VCL Mapping

This section discusses mapping ATS configuration files to VCL (Varnish Configuration Language).

This mapping is only in the context of Traffic Control so some of the functionality of ATS configuration files might not be addressed.

ip_allow.yaml

From ipallowdotyaml.go:

• No conditions are set on outbound connections.
• Only methods with conditions are: PUSH, DELETE and PURGE.
• localhost is trusted.

One way to meet the previous specification:

• Use acl for each method containing only allowed ips for that method.
• At the start of vcl_recv check for each method with the matching acl.

Example:

acl purgers {
  ...
  "127.0.0.1";
}

sub vcl_recv {
  if (req.method == "PURGE") {
    if (!client.ip ~ purgers) {
      return (synth(405, "Not allowed."));
    }
      return (purge);
  }
}

cache.config

From cachedotconfig.go:

• never-cache is the only specified action so the config is only used to disable caching for some origins

There are multiple ways to achieve that in VCL using VMODs, one of them:

Using selector VMOD to make a set of the specified origins and matching the set with backend response.

Example:

import selector;

backend ds1 {
    .host = "my-origin.com";
    .port = "8000";
}

backend ds2 {
    .host = "another-origin.com";
    .port = "80";
}

sub vcl_init {
    new no_cache = selector.set();
    no_cache.add("ds1");
    no_cache.add("ds2");
    ...
}
sub vcl_backend_response {
    if (no_cache.match(beresp.backend.name)) {
        beresp.uncacheable = true;
    }
}

ssl_multicert.config

From sslmulticertdotconfig.go:

• only ssl_cert_name and ssl_key_name are used.
• no dest_ip is specified so ATS uses SNI if possible or choose default cert as mentioned in certificate selection.

Varnish doesn't support TLS natively so a solution to that is to use Hitch. It's commonly used with Varnish to terminate TLS/SSL connections and forward them to Varnish.

To use multiple certs with Hitch we update the configuration in /etc/hitch/hitch.conf:

frontend = {
    host = "10.10.10.131"
    port = "443"
}
backend = "[127.0.0.1]:8443"

pem-file = {
    cert = "/config/ds.cer"
    private-key = "/config/ds.key"
}
pem-file = {
    cert = "/config/ds2.cer"
    private-key = "/config/ds2.key"
}
write-proxy-v2 = on

Hitch will try to use SNI if possible and will default to last cert specified as mentioned in vhosts

Varnish will need to be configured to listen on the backend IP with proxy enabled to get which IP and port client is connected to as mentioned in Hitch docs.

varnishd -a :80 -a :8443,PROXY

sni.yaml

From snidotyaml.go:

• only http2 and valid_tls_versions_in are used with each server FQDN

Hitch does not support specifying HTTP/2 or TLS version based on domain. It is only defined on all IPs used.

Example:

frontend {
    ...
}

alpn-protos = "h2, http/1.1"
tls-protos = TLSv1.2 TLSv1.3

So, It is either supported for all domains on the cache server or not.

varnishd will need to be configured with -p feature=+http2 option to enable HTTP/2 support.

logging.yaml

For this configuration file there are three parts formats, filters and log files. All of them can be done using varnishncsa with some differences.

Format

ATS supports aggregating logs for a period of time which Varnish does not support. But according to loggingdotyaml.go it is not used so only mapping from log fields from ATS to Varnish and using -F flag with varnishncsa.

Example:

formats:
- name: examplefmt
  format: '%<chi> , %<pssc>'

Using varnishncsa:

varnishncsa -F "%h , %s"

Filter

There are some differences between how ATS filter logs and how filtering works with Varnish:

• It allows specifying multiple operands as comma separated values (e.g. 1,2,3, POST,GET) which isn't supported in Varnish But it can be simplified to multiple expressions with or in case of accept and and not in case of reject.
• wipe_field_value has no direct mapping to Varnish.
• It supports specifying IP range to match against which Varnish also doesn't support.
• It supports comparing strings case-insensitive for specific filters, which Varnish doesn't support directly but it can do all matching case-insensitive by using -C flag. For specific filter there is no direct way but it can be specified with (?i) regex matching.

Example:

filters:
- name: examplefilter
  action: accept
  condition: cqhm MATCH POST,PUT

Using varnishncsa:

varnishncsa -q 'ReqMethod eq POST or ReqMethod eq PUT'

Log Fields

The way varnishncsa works is in three modes backend mode, client mode or both. Fields related to client connections only show up in client mode and fields related to backend connections only show up in Backend mode (using both modes just outputs two log lines for each mode). So log lines can't have fields for both client and backend at the same time (it will show - for the field that it can't output). ATS doesn't work like that so it can have fields with details throughout the session with no problem.

So, for example to get the response code returned to user from server and to server from origin it can specified in ATS as follows:

formats:
- name: responses
  format: '%<pssc> , %<sssc>'

Trying that with varnishncsa:

$ varnishncsa -bc -F "%{VSL:RespStatus}x %{VSL:BerespStatus}x"
- 200
200 -

It returns two log lines containing the response from either the origin or the cache server but not both. There is no current solution to that so either specify the needed fields and run in backend and client modes with %{Varnish:side}x to distinguish between which log line belongs to what mode or for each log file in ATS it gets mapped to two log files one logging backend fields and the other logging client fields.

This shouldn't be a problem with filtering because VSL queries by default filter on all steps of the transaction as mentioned here.

For mapping log fields from ATS to Varnish they can be grouped into three categories:

• Fields with direct mapping like request method and status codes.
• Fields in ATS that either returns different format from its Varnish equivalent or doesn't have an equivalent field but could be synthesized form VCL code using std.log("key:value") and using %{VCL_Log:key} to get the value. For example fields related to SSL, some of them can be generated using vmod_proxy with std.log().
• Fields that can't be mapped to Varnish like proxy UUID which is the UUID for the running ATS process.

Log Files

Differences between ATS and Varnish logging:

• varnishncsa doesn't support writing binary logs so only ascii logs can be used. varnishlog supports binary logs but cannot be formatted. so, if used, varnichncsa will need to consume its logs to output ascii logs with proper formatting.
• varnishncsa doesn't support log rotation so logrotate can be used.
• multiple varnishncsa daemons will be running for each log file as it only supports writing to one file.

For example logging.yaml file:

logging:
  filters:
  - name: examplefilter
    action: accept
    condition: cqhm MATCH POST,PUT
  formats:
  - name: examplefmt
    format: '%<chi> , %<pssc>'
  logs:
  - filename: example.log
    format: examplefmt
    filters:
    - examplefilter
    rolling_enabled: log.roll.size
    rolling_size_mb: 80

varnishncsa command would be:

varnishncsa -q 'ReqMethod eq POST or ReqMethod eq PUT' \
-F "%h , %s" \
-w /var/log/varnish/example.log -a -D

logrotate config would be:

/var/log/varnish/example.log {
  rotate 7
  compress
  size 80M
  postrotate
    systemctl -q is-active varnishncsa.service || exit 0
    systemctl reload varnishncsa.service
  endscript
}

Note that paths used and logrotate configuration are specified in records.config. Also logrotate doesn't rotate in a less than daily period so rolling_interval_sec might be discarded.

volume.config

From volumedotconfig.go:

• volume size is irrelevant.
• it is used to separate different storage types with different volume IDs.

So, instead of mapping volume IDs 1,2 and 3 to Drive, RAM Drive and SSD Drive a prefix could be added to each storage used in varnishd options.

Example:

varnishd -s d0=file,/dev/vda -s d1=file,/dev/vdb -s r0=malloc -s s0=file,/dev/vdc -s s1=file,/dev/vdd

Prefix d for Drive, r for RAM and s for SSD.

hosting.config

From hostingdotconfig.go:

• RAM is used to store content form origins if RAM Drive is used.
• any other content is stored on Drive or RAM if Drive is not used.

VMOD selector can be used as mentioned in cache.config with std, var and random:

import var;
import std;
import random;
import selector;

...

sub vcl_backend_response {
    if (origins.match(beresp.backend.name)) {
        var.set_int("r", std.integer(std.round(std.random(0, 2)), 0));
        if (var.get_int("r") == 0) {
            beresp.storage = storage.r0;
        }
        else if (var.get_int("r") == 1) {
            beresp.storage = storage.r1;
        }
        else {
            beresp.storage = storage.r2;
        }
    }
    else {
        # same for non origin content with Drive storage
    }
}