Setup Agent

We want to run the test agent on our virtual host running Debian Jessie, without any display of course.

• Install the virtual framebuffer Xvfb and some further modules with sudo aptitude install xvfb x11-xkb-utils xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic. There might be some more which I have forgotten or have been on my system already.

• Configure Xvfb to start automatically in the background. As we are using Jessie with systemd we need to

  • Create the services file /etc/systemd/system/xvfb.service

    [Unit]
    Description=Virtual Frame Buffer X Server
    After=network.target
    
    [Service]
    ExecStart=/usr/bin/Xvfb :99 -screen 0 1024x768x24 -ac -nolisten tcp
    
    [Install]
    WantedBy=multi-user.target
    

  • Enable with sudo systemctl enable xvfb.service

  • Start with sudo systemctl start xvfb.service

• Install Chrome from Google

  We are using Googles repo, therefore:

  wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
  sudo echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list
  sudo aptitude install chrome
  

  It might complain that there are some packages missing, just install them as well.

• Fix Chrome startup script

  • The WebPagetest agent will start the program chrome, but this doesn't exist in the path, So we will create a symlink: sudo ln -s /opt/google/chrome/google-chrome /usr/bin/chrome.

  • We have to tell the google-chrome script to use our Xvfb display. Edit the file /opt/google/chrome/google-chrome and add the following line after the #!/bin/bash: export DISPLAY=:99.

  • Disable gpu in chrome, again edit the file /opt/google/chrome/google-chrome. Pretty much add the end, change the lines so they are looking like this:

    if [[ -n "$CHROME_USER_DATA_DIR" ]]; then
      # Note: exec -a below is a bashism.
      exec -a "$0" "$HERE/chrome"  \
        --user-data-dir="$CHROME_USER_DATA_DIR" --disable-gpu "$@"
    else
      exec -a "$0" "$HERE/chrome" --disable-gpu  "$@"
    fi
    

• Finally, start the agent. Right now, I start it in a screen session. It has to run as a normal user, Chrome doesn't support root. Don't forget to change the ownership of the agent's folder the that user first: chown -R user /var/www/webpagetest/agent/js/ Start the agent (replace the location with whatever you have configured in your locations.ini): /var/www/webpagetest/agent/js/wptdriver.sh --serverUrl http://webpagetest.your_domain.com -b chrome -i -l Jiffy

• To watch the agent during the test, use VNC.

  • Install the VNC server x11vnc sudo aptitude install x11vnc
  • Start the VNC server x11vnc -localhost -display :99. There is no authentication, but as it listens to localhost only, it should be ok. Also, after you connect once, it will stop the process again.
  • From you local machine, ssh with port forwarding for vnc, something like this `ssh -L 5900:127.0.0.1:5900 <your_agent_host>``
  • Start your VNC viewer on your local machine, connect to 127.0.0.1 and have fun watching your test.

• To get the video recording working, you have to patch /var/www/webpagetest/agent/js/src/browser_local_chrome.js

  diff --git a/agent/js/src/browser_local_chrome.js b/agent/js/src/browser_local_chrome.js
  index dda36c9..b8b853c 100644
  --- a/agent/js/src/browser_local_chrome.js
  +++ b/agent/js/src/browser_local_chrome.js
  @@ -28,6 +28,7 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  
   var logger = require('logger');
   var browser_base = require('browser_base');
  +var process_utils = require('process_utils');
   var util = require('util');
   var webdriver = require('selenium-webdriver');
  
  @@ -60,6 +61,9 @@ function BrowserLocalChrome(app, args) {
     this.chromeFlags_ = CHROME_FLAGS;
     this.task_ = args.task;
     this.supportsTracing = true;
  +  this.ffmpeg_ = 'ffmpeg';
  +  this.videoProcess_ = undefined;
  +  this.videoStarted_ = false;
   }
   util.inherits(BrowserLocalChrome, browser_base.BrowserBase);
   /** @constructor */
  @@ -73,6 +77,7 @@ exports.BrowserLocalChrome = BrowserLocalChrome;
    */
   BrowserLocalChrome.prototype.startWdServer = function(browserCaps) {
     'use strict';
  +  logger.info('Starting BrowserLocalChrome.prototype.startWdServer');
     var requestedBrowserName = browserCaps[webdriver.Capability.BROWSER_NAME];
     if (webdriver.Browser.CHROME !== requestedBrowserName) {
       throw new Error('BrowserLocalChrome called with unexpected browser ' +
  @@ -103,6 +108,7 @@ BrowserLocalChrome.prototype.startWdServer = function(browserCaps) {
    */
   BrowserLocalChrome.prototype.startBrowser = function() {
     'use strict';
  +  logger.info('Starting BrowserLocalChrome.prototype.startBrowser');
     // TODO(klm): clean profile, see how ChromeDriver does it.
     var flags = CHROME_FLAGS;
     flags.push('-remote-debugging-port=' + this.devToolsPort_);
  @@ -154,12 +160,60 @@ BrowserLocalChrome.prototype.scheduleGetCapabilities = function() {
           webdriver: !!this.chromedriver_,
           'wkrdp.Page.captureScreenshot': true,
           'wkrdp.Network.clearBrowserCache': true,
  -        'wkrdp.Network.clearBrowserCookies': true
  +        'wkrdp.Network.clearBrowserCookies': true,
  +        videoRecording: true,
  +        videoFileExtension: 'mp4'
         };
     }.bind(this));
   };
  
    /**
   + * @param {string} filename The local filename to write to.
   + * @param {Function=} onExit Optional exit callback, as noted in video_hdmi.
   + */
   +BrowserLocalChrome.prototype.scheduleStartVideoRecording = function(
   +    filename) {
   +  'use strict';
   +  logger.info('scheduleStartVideoRecording called');
   +  this.app_.schedule('Start video capture', function() {
   +    if (this.ffmpeg_) {
   +      process_utils.scheduleSpawn(this.app_, this.ffmpeg_,
   +          ['-y', '-f', 'x11grab', '-r', '24', '-s', '1024x768', '-loglevel', 'warning', '-frame_drop_threshold', '4', '-an', '-i', ':99.0', filename]).then(
   +          function(proc) {
   +        this.videoProcess_ = proc;
   +        this.videoProcess_.on('exit', function(code, signal) {
   +          logger.info('ffmpeg EXIT code %s signal %s', code, signal);
   +          this.videoProcess_ = undefined;
   +        }.bind(this));
   +      }.bind(this));
   +    }
   +  }.bind(this));
   +};
   +
   +/**
   + * Stops the video recording.
   + */
   +BrowserLocalChrome.prototype.scheduleStopVideoRecording = function() {
   +  'use strict';
   +  if (this.videoProcess_) {
   +    this.app_.schedule('Stop video capture', function() {
   +      logger.debug('Killing video capture');
   +      try {
   +        this.videoProcess_.kill('SIGINT');
   +        this.app_.wait(function() {
   +          return this.videoProcess_ == undefined;
   +        }.bind(this), 30000).then(function() {
   +          logger.info('Killed video capture');
   +        }.bind(this));
   +      } catch (killException) {
   +        logger.error('video capture kill failed: %s', killException);
   +        this.videoProcess_ = undefined;
   +      }
   +    }.bind(this));
   +  }
   +};
   +
   +/**
     * Starts packet capture.
     *
     * #param {string} filename  local file where to copy the pcap result.
  

Not working / tbd:

• Traffic shaping: ipfw is not available under Debian, you have to select "native connection (no traffic shapping)"
• You will see something like chrome STDERR: Xlib: extension "RANDR" missing on display, this is ok. Tests will work, the library is not implemented in Xvfb yet.
• When Chrome is started, it sometimes hangs and the script doesn't proceed. Either press Ctrl-C or kill the last chrome process by hand helps.