Varnish Support

This wiki page is for documenting the steps for adding Varnish Cache support to t3c.

ATS Configuration to VCL Mapping

This section discusses mapping ATS configuration files to VCL (Varnish Configuration Language).

This mapping is only in the context of Traffic Control so some of the functionality of ATS configuration files might not be addressed.

ip_allow.yaml

From ipallowdotyaml.go:

No conditions are set on outbound connections.
Only methods with conditions are: PUSH, DELETE and PURGE.
localhost is trusted.

One way to meet the previous specification:

Use acl for each method containing only allowed ips for that method.
At the start of vcl_recv check for each method with the matching acl.

Example:

acl purgers {
  ...
  "127.0.0.1";
}

sub vcl_recv {
  if (req.method == "PURGE") {
    if (!client.ip ~ purgers) {
      return (synth(405, "Not allowed."));
    }
      return (purge);
  }
}

cache.config

From cachedotconfig.go:

never-cache is the only specified action so the config is only used to disable caching for some origins

There are multiple ways to achieve that in VCL using VMODs, one of them:

Using selector VMOD to make a set of the specified origins and matching the set with backend response.

Example:

import selector;

backend ds1 {
    .host = "my-origin.com";
    .port = "8000";
}

backend ds2 {
    .host = "another-origin.com";
    .port = "80";
}

sub vcl_init {
    new no_cache = selector.set();
    no_cache.add("ds1");
    no_cache.add("ds2");
    ...
}
sub vcl_backend_response {
    if (no_cache.match(beresp.backend.name)) {
        beresp.uncacheable = true;
    }
}

ssl_multicert.config

From sslmulticertdotconfig.go:

only ssl_cert_name and ssl_key_name are used.
no dest_ip is specified so ATS uses SNI if possible or choose default cert as mentioned in certificate selection.

Varnish doesn't support TLS natively so a solution to that is to use Hitch. It's commonly used with Varnish to terminate TLS/SSL connections and forward them to Varnish.

To use multiple certs with Hitch we update the configuration in /etc/hitch/hitch.conf:

frontend = {
    host = "10.10.10.131"
    port = "443"
}
backend = "[127.0.0.1]:8443"

pem-file = {
    cert = "/config/ds.cer"
    private-key = "/config/ds.key"
}
pem-file = {
    cert = "/config/ds2.cer"
    private-key = "/config/ds2.key"
}
write-proxy-v2 = on

Hitch will try to use SNI if possible and will default to last cert specified as mentioned in vhosts

Varnish will need to be configured to listen on the backend IP with proxy enabled to get which IP and port client is connected to as mentioned in Hitch docs.

varnishd -a :80 -a :8443,PROXY

sni.yaml

From snidotyaml.go:

only http2 and valid_tls_versions_in are used with each server FQDN

Hitch does not support specifying HTTP/2 or TLS version based on domain. It is only defined on all IPs used.

Example:

frontend {
    ...
}

alpn-protos = "h2, http/1.1"
tls-protos = TLSv1.2 TLSv1.3

So, It is either supported for all domains on the cache server or not.

varnishd will need to be configured with -p feature=+http2 option to enable HTTP/2 support.

logging.yaml

For this configuration file there are three parts formats, filters and log files. All of them can be done using varnishncsa with some differences.

Format

ATS supports aggregating logs for a period of time which Varnish does not support. But according to loggingdotyaml.go it is not used so only mapping from log fields from ATS to Varnish and using -F flag with varnishncsa.

Example:

formats:
- name: examplefmt
  format: '%<chi> , %<pssc>'

Using varnishncsa:

varnishncsa -F "%h , %s"

Filter

There are some differences between how ATS filter logs and how filtering works with Varnish:

It allows specifying multiple operands as comma separated values (e.g. 1,2,3, POST,GET) which isn't supported in Varnish But it can be simplified to multiple expressions with or in case of accept and and not in case of reject.
wipe_field_value has no direct mapping to Varnish.
It supports specifying IP range to match against which Varnish also doesn't support.
It supports comparing strings case-insensitive for specific filters, which Varnish doesn't support directly but it can do all matching case-insensitive by using -C flag. For specific filter there is no direct way but it can be specified with (?i) regex matching.

Example:

filters:
- name: examplefilter
  action: accept
  condition: cqhm MATCH POST,PUT

Using varnishncsa:

varnishncsa -q 'ReqMethod eq POST or ReqMethod eq PUT'

Log Fields

The way varnishncsa works is in three modes backend mode, client mode or both. Fields related to client connections only show up in client mode and fields related to backend connections only show up in Backend mode (using both modes just outputs two log lines for each mode). So log lines can't have fields for both client and backend at the same time (it will show - for the field that it can't output). ATS doesn't work like that so it can have fields with details throughout the session with no problem.

So, for example to get the response code returned to user from server and to server from origin it can specified in ATS as follows:

formats:
- name: responses
  format: '%<pssc> , %<sssc>'

Trying that with varnishncsa:

$ varnishncsa -bc -F "%{VSL:RespStatus}x %{VSL:BerespStatus}x"
- 200
200 -

It returns two log lines containing the response from either the origin or the cache server but not both. There is no current solution to that so either specify the needed fields and run in backend and client modes with %{Varnish:side}x to distinguish between which log line belongs to what mode or for each log file in ATS it gets mapped to two log files one logging backend fields and the other logging client fields.

This shouldn't be a problem with filtering because VSL queries by default filter on all steps of the transaction as mentioned here.

For mapping log fields from ATS to Varnish they can be grouped into three categories:

Fields with direct mapping like request method and status codes.
Fields in ATS that either returns different format from its Varnish equivalent or doesn't have an equivalent field but could be synthesized form VCL code using std.log("key:value") and using %{VCL_Log:key} to get the value. For example fields related to SSL, some of them can be generated using vmod_proxy with std.log().
Fields that can't be mapped to Varnish like proxy UUID which is the UUID for the running ATS process.

Log Files

Differences between ATS and Varnish logging:

varnishncsa doesn't support writing binary logs so only ascii logs can be used. varnishlog supports binary logs but cannot be formatted. so, if used, varnichncsa will need to consume its logs to output ascii logs with proper formatting.
varnishncsa doesn't support log rotation so logrotate can be used.
multiple varnishncsa daemons will be running for each log file as it only supports writing to one file.

For example logging.yaml file:

logging:
  filters:
  - name: examplefilter
    action: accept
    condition: cqhm MATCH POST,PUT
  formats:
  - name: examplefmt
    format: '%<chi> , %<pssc>'
  logs:
  - filename: example.log
    format: examplefmt
    filters:
    - examplefilter
    rolling_enabled: log.roll.size
    rolling_size_mb: 80

varnishncsa command would be:

varnishncsa -q 'ReqMethod eq POST or ReqMethod eq PUT' \
-F "%h , %s" \
-w /var/log/varnish/example.log -a -D

logrotate config would be:

/var/log/varnish/example.log {
  rotate 7
  compress
  size 80M
  postrotate
    systemctl -q is-active varnishncsa.service || exit 0
    systemctl reload varnishncsa.service
  endscript
}

Note that paths used and logrotate configuration are specified in records.config. Also logrotate doesn't rotate in a less than daily period so rolling_interval_sec might be discarded.

volume.config

From volumedotconfig.go:

volume size is irrelevant.
it is used to separate different storage types with different volume IDs.

So, instead of mapping volume IDs 1,2 and 3 to Drive, RAM Drive and SSD Drive a prefix could be added to each storage used in varnishd options.

Example:

varnishd -s d0=file,/dev/vda -s d1=file,/dev/vdb -s r0=malloc -s s0=file,/dev/vdc -s s1=file,/dev/vdd

Prefix d for Drive, r for RAM and s for SSD.

hosting.config

From hostingdotconfig.go:

RAM is used to store content form origins if RAM Drive is used.
any other content is stored on Drive or RAM if Drive is not used.

VMOD selector can be used as mentioned in cache.config with std, var and random:

import var;
import std;
import random;
import selector;

...

sub vcl_backend_response {
    if (origins.match(beresp.backend.name)) {
        var.set_int("r", std.integer(std.round(std.random(0, 2)), 0));
        if (var.get_int("r") == 0) {
            beresp.storage = storage.r0;
        }
        else if (var.get_int("r") == 1) {
            beresp.storage = storage.r1;
        }
        else {
            beresp.storage = storage.r2;
        }
    }
    else {
        # same for non origin content with Drive storage
    }
}

storage.config

From storagedotconfig.go:

RAM Drives uses path to drive which Varnish doesn't support with malloc. So, either use malloc with the same size as the RAM Drive or treat it as a file storage.

Example storage.config:

/dev/vda volume=1
/dev/vdb volume=1
/dev/vdc volume=2 <- RAM Drive

to match it with varnishd:

varnishd -s d1=file,/dev/vda -s d2=file,/dev/vdb -s r0=file,/dev/vdc

Notes:

to apply storage changes to Varnish instance, systemctl will be reloaded to register new changes in varnish service unit file with systemctl daemon-reload and varnishd will be restarted systemctl restart varnish.service.
another option to handle Varnish storage is VMOD slash which offers more configuration.

plugin.config

plugin.config is mainly responsible for loading plugins to ATS at runtime. For the list of stable plugins a lot of them are either supported natively in VCL like Header Rewrite or Regex Remap and some are available as VMODs like AWS S3 Authentication as awsrest (see VMODs for more info) while some are not implemented.

For plugins used in Traffic Control the equivalent VMOD will be imported and configured the same or by using VCL to have the same effect.

parent.config

This configuration file is mainly about cache hierarchy and how requests are sent to parent cache servers and origins. The list of Parent cache servers are exhausted first before request is sent to the origin.

The algorithm used for selecting parents is defined by round_robin and the relation between primary parents and secondary parents is controlled by secondary_mode in case of consistent hash algorithm.

For defining and using parent servers VMOD director is used. round_robin field can be one of five algorithms:

true and strict which are replaced with round_robin director.
false which could be replaced with random director.
consistent_hash which is replaced with shard director.
latched which is replaced with fallback director.

Since the request is sent to origin if no parent is healthy, all directors will be used with the origin backend in a fallback director.

backend be1 {
    .host = "parent1.example.com";
    .port = "80";
}
backend be2 {
    .host = "parent2.example.com";
    .port = "80";
}

backend be3 {
    .host = "origin.example.com";
    .port = "80";
}

sub vcl_init {
    new dir = directors.round_robin();
    dir.add_backend(be1);
    dir.add_backend(be2);

    new fdir = directors.fallback();
    fdir.add_backend(dir.backend());
    fdir.add_backend(be3);
}

For secondary_mode there are only two options used in ATC:

0 which chooses a server from the parent list then exhaust the secondary parent list if the parent is down then revert to the parent list if no secondary is available. There is no straightforward way to do that in VCL but with parameter manipulation of the shard director it might be achievable.

sub vcl_init {
    new primary = directors.shard();
    primary.add_backend(be1);
    primary.add_backend(be2);

    new secondary = directors.shard();
    secondary.add_backend(be3);

    new fdir = directors.fallback();
    fdir.add_backend(secondary.backend(resolve=LAZY));
    fdir.add_backend(primary.backend(resolve=LAZY));
    fdir.add_backend(be4);
}

sub vcl_recv {
    // healthy=IGNORE to get the primary backend even if unhealthy to decide whether to skip parent director or not.
    set req.backend_hint = primary.backend(healthy=IGNORE, resolve=NOW);
    if(!std.healthy(primary.backend(healthy=IGNORE, resolve=NOW))) {
        // use fallback director to exhaust secondary list then primary then go to origin.
        set req.backend_hint = fdir.backend();
    }
}

1 which exhausts parents list then secondary parents list then goes to the origin.

It could be done like this:

sub vcl_init {
    ...

    new fdir = directors.fallback();
    fdir.add_backend(primary.backend(resolve=LAZY));
    fdir.add_backend(secondary.backend(resolve=LAZY));
    fdir.add_backend(be4);
}

sub vcl_recv {
    set req.backend_hint = fdir.backend();
}

resolve=LAZY is used so shard_param could be used to change backend parameters after backend is assigned to the request.

parent_retry field is related to using different backend based on response code from chosen parent. unavailable_server_retry marks the backend as unhealthy which is not achievable from VCL, while simple_retry is for retrying a different backend after a certain response is returned which there is no documentation on changing backends after request is sent. So, all fields related to parent_retry might not be used.

qstring is used for including the query string with the URL as the hash key. It can be removed by using VMOD querystring or by just using regsuball to remove it from the URL using regular expression.

go_direct will result in not including the parent directors in a fallback director but only include the origins. And parent_is_proxy will determine if go_direct is used.

Example line in parent.config:

dest_domain=origin.example.com port=80 parent="parent1.example.com:80|1.0;parent2.example.com:80" round_robin=strict

Similar VCL:

import directors;

probe default {
    .url = "/";
    .timeout = 1s;
    .interval = 4s;
    .window = 5;
    .threshold = 3;
}

backend be1 {
    .host = "parent1.example.com";
    .port = "80";
    .probe = default;
}
backend be2 {
    .host = "parent2.example.com";
    .port = "80";
    .probe = default;
}

backend origin {
    .host = "origin.example.com";
    .port = "80";
    .probe = default;
}

sub vcl_init {
    new parents = directors.round_robin();
    parents.add_backend(be1);
    parents.add_backend(be2);

    new fdir = directors.fallback();
    fdir.add_backend(parents.backend());
    fdir.add_backend(origin);

}

sub vcl_recv {
    // if remap rule for example.com to go to origin.example.com
    if (req.http.host == "example.com") {
        set req.backend_hint = fdir.backend();
    }
}

Notes:

Nesting directors (especially shard directors) is not well documented so some assumptions might not hold true.
Backends health check will be done using probes and not with responses returned from the backend.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Varnish Support

ATS Configuration to VCL Mapping

ip_allow.yaml

cache.config

ssl_multicert.config

sni.yaml

logging.yaml

Format

Filter

Log Fields

Log Files

volume.config

hosting.config

storage.config

plugin.config

parent.config

Clone this wiki locally