Analysis of a Specified Commit

If you want to continuously measure the performance of the unit tests of your project (or a part of them), then use Peass-CI is the best solution. But if you want to try out whether a given past commit (or a range of past commits) has a performance change, it is more suitable to use the command line interface of Peass directly.

As an example, we'll analyze the performance of Apache Commons Fileupload, a well-known open source project, and how the performance changed in commit 4ed6e923cb2033272fcb993978d69e325990a5aa.

In general, the analysis of a given commit by using the CLI is executed like the Peass-demo, but start and end commit need to be defined for the measurement process.

Now the three steps regression test selection, measurement and root cause analysis need to be executed.

Prerequisites

Install Peass like described in the README and clone your project to a folder (in our case ../projects/commons-fileupload). Identify the commits you want to examine. If you want to measure the performance change to the direct predecessor of a commit, identify the commit by looking in you log (in our case git rev-parse 4ed6e923cb2033272fcb993978d69e325990a5aa~1). Currently, Peass only analyzes neighbour commits; if you give Peass a commit range with more commits, all commits will be compared to their individual predecessors (which is time-consuming).

Regression Test Selection

To execute the regression test selection, execute

./peass select -folder ../projects/commons-fileupload -startcommit fdf011a5f9a15826771b19cdd6795b247b0bc3e4 -endcommit 4ed6e923cb2033272fcb993978d69e325990a5aa

This is likely to take some time. Afterwards, the file results/traceTestSelection_$project.json will create all test methods that should be measured.

Measurement

To measure the performance of the tests in your commit, call

./peass measure -executionFile results/traceTestSelection_$project.json -folder ../projects/commons-fileupload -iterations 5 -warmup 5 -repetitions 10000 -vms 30

with setting the right folder and the right executionFile name for your use case.

For choosing the right parameters, see Configuration-of-Measurement-Processes. Note that there should be less or ideally no load on the same system, since parallel processes might influence the performance of your measurement. This process may take several hours (especially if configured inefficiently).

If you only want to measure on test, you can add -test $testcase, in our example -test org.apache.commons.fileupload.ServletFileUploadTest#testFileUpload.

To identify performance changes, call

./peass getchanges -data ../projects/commons-fileupload_peass/ -executionFile results/traceTestSelection_$project.json

This will create a file named results/changes_$project.json, which tells you which performance change have been measured in the current commit.

Root Cause Analysis

If you want to examine the root cause of the performance change, call

./peass searchcause -iterations 5 -warmup 5 -repetitions 10000 -vms 30 -commit 4ed6e923cb2033272fcb993978d69e325990a5aa -test org.apache.commons.fileupload.ServletFileUploadTest#testFileUpload -folder ../projects/commons-fileupload -executionFile results/traceTestSelection_$project.json

with your commit, test, folder and executionFile. Afterwards, you'll find a measurement tree in a folder named like your project folder and _peass as suffix (in our example ../projects/commons-fileupload_peass) in rca/tree/ and the commit and testcase name. Note that this process may take several hours to days (especially if configured inefficiently).

To visualize this measurement results, call

./peass visualizerca -data ../projects/commons-fileupload_peass/ -propertyFolder results/properties_commons-fileupload/

Afterwards, you'll find in results/$commit, where $commit is your commit (so in our example results/4ed6e923cb2033272fcb993978d69e325990a5aa) a html file which contains a visualization of the call tree of your test case.