A recipe about integrating lein-cljsbuild in an external CI script/server

[plam3ns]

Hello all,

Problem

I would like to include lein cljsbuild test (which invokes some external JavaScript execution environment like PhantomJS, SlimerJS etc.) as a build step in a larger build/test CI script, and I would like not only to see in the console output if tests succeeded, but also that the parent script be aborted on test failure(s).
After asking in relevant IRC channels, the feeling has been, that there is an error in lein-cljsbuild, for which I opened here an issue, which turned out to be wrong. lein-cljsbuild works great!

As the info in the internet about it stems from different versions of lein-cljsbuild, different testing frameworks (clojurescript.test, cljs.test, expectations etc.) for different environments (PhantomJS 1.9/2.0, NodeJS, SlimerJS, Rhino etc.) in different ClojureScript versions, combined with some of mine probably wrong assumptions about expected behavior and trying to achieve that simple goal for a first time, I went actually through a learning and debugging exercise, which consisted of sometimes going in a wrong direction, or solving some small details, each of which may easy, but as a whole it lost me almost a day (for example given the fact that on my machine I could not make .lein-classpath to work as intended as described in the 'plugin how to' section of the leiningen's manual and ending up in first building/installing lein-cljsbuild after each change I do). I hope the text below is useful for a next fellow doing it for a first time.

Context:

• using the new cljs.test as in version [org.clojure/clojurescript "1.7.107"] as a test library.
• using lein-cljsbuild as in version [lein-cljsbuild "1.1.0"].
• using PhantomJS 2
• invoking lein cljsbuild test as part of a larger shell script (or some CI server as Jenkins for example) and expecting a non zero exit code from 'lein cljsbuild test' to be able to abort the parent script on test failure(s).
• as a template for code examples I'll use the project kindly provided as a quick start in ClojureScript testing by Vladimir Iakovlev - so if I refer to test.js, then it means that I point to that file in his example project, and as being kept simple, it could be a good starting point for other people as well.

Findings/recipes:

1. After some time looking at different examples over the internet and in combination with the fact that lein-cljsbuild changed it's expected parameter format in project.clj during versions, I got the impression that :notify-command and :test-commands in the :cljsbuild section of project.clj are the same thing, even if the lein-cljsbuild documentation talks explicitly about :test-commands. So, :notify-command is for notifications and the decision in the implementation is that it should not change the exit code of lein cljsbuild, while :test-commands does actually start the external JavaScript runner and propagates it's exit code back to the caller.
   1.1. Recipe: Read the doc. :)
2. At least PhantomJS comes in 2 current versions (1.9.2 and 2.0) and they need slightly different initialization.
   2.1. For 1.9 do as in the example project,
   2.2. For version 2, change the line

var url = phantom.args[0];

to

var url = require('system').args[1];

in test.js.
3. cljs.test as independent of it's execution environment of course doesn't deal with how to exit from it. Further run-(all-)tests doesn't return a map with the test outcomes (number of failed/succeeded tests) as in clojure.test (there is already an issue about it in the Clojure JIRA).
3.1. But there is a hook in cljs.test through cljs.test/report. So, convert test.cljs from:

(ns ^:figwheel-always cljs-test-example.test
  (:require [cljs.test :refer-macros [run-all-tests]]))

(enable-console-print!)

(defn ^:export run
  []
  (run-all-tests #"cljs-test-example.*-test"))

to

(ns ^:figwheel-always cljs-test-example.test
  (:require [cljs.test :refer-macros [run-all-tests] :refer [successful?]]))

(enable-console-print!)

(def all-tests-successfull? (atom false)) ; to catch the success status of running the tests

;; the hook
(defmethod cljs.test/report [:cljs.test/default :end-run-tests] [m]
  (reset! all-tests-successfull? (successful? m)))

(defn ^:export run
  []
  (run-all-tests #"cljs-test-example.*-test")
  ; and return the actual success
  @all-tests-successfull?)

3.2. If one uses PhantomJS or SlimerJS at least, a short JavaScript script (for example as in test.js) is needed to sit between cljsbuild and cljs.test to invoke indirectly run-(all-)tests, so without further help, it can not know if the tests succeeded or not, to be able to exit it's process with the right exit code. In the case of PhantomJS at least, the user test function in invoked in the context of page.evaluate(fn[]....), which itself creates a sandbox in which phantom.exit(exitCode) is unreachable. So in order to resolve the issue, change the page.open(... from

page.open(url, function (status) {
    page.evaluate(function(){
        cljs_test_example.test.run();
    });
    phantom.exit(0);
});

to at least

page.open(url, function (status) {
    var success = page.evaluate(function(){ // to get the result of the PhantomJS sandbox
        return cljs_test_example.test.run(); // using the result obtained through the cljs.test hook
    });
    phantom.exit(success ? 0 : 1); // different exit code based on test outcome
})

With best regards and thank you for the good plugin!
Plamen