Obtaining package performance data using Rperform
Rperform provides functions which can be used to obtain data containing information about a package's performance. Currently, there are two classes of functions which return performance data. They respectively deal with:
These functions, like most of the other Rperform functions, are designed to work in harmony with hadley's testthat package. However, they will work fine with the packages not employing testthat as well. Read on for further explanation.
These functions can be used to obtain data about the following metrics:
One can use Rperform to obtain the runtime of a test-file and the testthat code blocks it contains, if any.
One can use Rperform to obtain the memory usage details of a test-file and the testthat code blocks it contains, if any. There are two types of memory-related metrics which can be obtained:
- Maximum memory utilized: This is the maximum system memory utilized by the file/block during the course of its execution.
- Total memory leaked: This is the difference in the amount of system memory being used at the end and start of a file or block's execution.
Note: For each individual commit, three values of a particular metric are measured. Hence, the multiple values for a single commit exist in the dataframes. Many a times the values simply overlap.
-> time_compare() can be used to obtain the runtime details of a test file and testthat blocks over a specified number of commits. The package namedCapture is used for demonstration purposes.
## Warning: Always set the current directory to be the root directory of the package you intend to test.
> setwd("path/to/namedCapture")
> library(Rperform)
> time_compare(test_path = "tests/testthat/test-optional-groups.R", num_commits = 3)
test_name metric_name status metric_val message
1 match one optional group runtime (in seconds) pass 0.000783163 service=github
2 match one optional group runtime (in seconds) pass 0.000534672 service=github
3 match one optional group runtime (in seconds) pass 0.000515888 service=github
4 match all optional groups runtime (in seconds) pass 0.001008681 service=github
5 match all optional groups runtime (in seconds) pass 0.000763691 service=github
6 match all optional groups runtime (in seconds) pass 0.000751090 service=github
7 test-optional-groups.R runtime (in seconds) pass 0.022617698 service=github
8 test-optional-groups.R runtime (in seconds) pass 0.002455121 service=github
9 test-optional-groups.R runtime (in seconds) pass 0.002314949 service=github
10 match one optional group runtime (in seconds) pass 0.000800733 coveralls maste
11 match one optional group runtime (in seconds) pass 0.000533138 coveralls maste
12 match one optional group runtime (in seconds) pass 0.000529287 coveralls maste
13 match all optional groups runtime (in seconds) pass 0.001104596 coveralls maste
14 match all optional groups runtime (in seconds) pass 0.000811456 coveralls maste
15 match all optional groups runtime (in seconds) pass 0.000802304 coveralls maste
16 test-optional-groups.R runtime (in seconds) pass 0.002871598 coveralls maste
17 test-optional-groups.R runtime (in seconds) pass 0.002594260 coveralls maste
18 test-optional-groups.R runtime (in seconds) pass 0.002462942 coveralls maste
19 match one optional group runtime (in seconds) pass 0.000865201 Merge branch 'c
20 match one optional group runtime (in seconds) pass 0.000614288 Merge branch 'c
21 match one optional group runtime (in seconds) pass 0.000518979 Merge branch 'c
22 match all optional groups runtime (in seconds) pass 0.001008659 Merge branch 'c
23 match all optional groups runtime (in seconds) pass 0.000814166 Merge branch 'c
24 match all optional groups runtime (in seconds) pass 0.000795073 Merge branch 'c
25 test-optional-groups.R runtime (in seconds) pass 0.002618324 Merge branch 'c
26 test-optional-groups.R runtime (in seconds) pass 0.002350283 Merge branch 'c
27 test-optional-groups.R runtime (in seconds) pass 0.002296139 Merge branch 'c
sha date_time
1 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
2 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
3 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
4 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
5 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
6 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
7 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
8 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
9 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
10 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
11 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
12 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
13 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
14 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
15 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
16 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
17 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
18 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
19 72348c84d212abe5860933bd5b2d72c242c2db6e 2015-12-01 11:05:22
20 72348c84d212abe5860933bd5b2d72c242c2db6e 2015-12-01 11:05:22
21 72348c84d212abe5860933bd5b2d72c242c2db6e 2015-12-01 11:05:22
22 72348c84d212abe5860933bd5b2d72c242c2db6e 2015-12-01 11:05:22
23 72348c84d212abe5860933bd5b2d72c242c2db6e 2015-12-01 11:05:22
24 72348c84d212abe5860933bd5b2d72c242c2db6e 2015-12-01 11:05:22
25 72348c84d212abe5860933bd5b2d72c242c2db6e 2015-12-01 11:05:22
26 72348c84d212abe5860933bd5b2d72c242c2db6e 2015-12-01 11:05:22
27 72348c84d212abe5860933bd5b2d72c242c2db6e 2015-12-01 11:05:22
-> mem_compare() can be used to obtain the memory usage details of a test file and testthat blocks over a specified number of commits. The package namedCapture is used for demonstration purposes.
## Warning: Always set the current directory to be the root directory of the package you intend to test.
> setwd("path/to/namedCapture")
> library(Rperform)
> mem_compare(test_path = "tests/testthat/test-optional-groups.R", num_commits = 2)
test_name metric_name status metric_val message
11.1 match one optional group max_mem pass 0.008 service=github
11.2 match one optional group leak_mem pass 0.008 service=github
11.3 match all optional groups max_mem pass 0.000 service=github
11.4 match all optional groups leak_mem pass 0.000 service=github
11.5 test-optional-groups.R max_mem pass 0.060 service=github
11.6 test-optional-groups.R leak_mem pass 0.060 service=github
12.1 match one optional group max_mem pass 0.004 service=github
12.2 match one optional group leak_mem pass 0.004 service=github
12.3 match all optional groups max_mem pass 0.000 service=github
12.4 match all optional groups leak_mem pass 0.000 service=github
12.5 test-optional-groups.R max_mem pass 0.060 service=github
12.6 test-optional-groups.R leak_mem pass 0.060 service=github
13.1 match one optional group max_mem pass 0.004 service=github
13.2 match one optional group leak_mem pass 0.004 service=github
13.3 match all optional groups max_mem pass 0.000 service=github
13.4 match all optional groups leak_mem pass 0.000 service=github
13.5 test-optional-groups.R max_mem pass 0.060 service=github
13.6 test-optional-groups.R leak_mem pass 0.060 service=github
21.1 match one optional group max_mem pass 0.004 coveralls maste
21.2 match one optional group leak_mem pass 0.004 coveralls maste
21.3 match all optional groups max_mem pass 0.000 coveralls maste
21.4 match all optional groups leak_mem pass 0.000 coveralls maste
21.5 test-optional-groups.R max_mem pass 0.060 coveralls maste
21.6 test-optional-groups.R leak_mem pass 0.060 coveralls maste
22.1 match one optional group max_mem pass 0.004 coveralls maste
22.2 match one optional group leak_mem pass 0.004 coveralls maste
22.3 match all optional groups max_mem pass 0.000 coveralls maste
22.4 match all optional groups leak_mem pass 0.000 coveralls maste
22.5 test-optional-groups.R max_mem pass 0.060 coveralls maste
22.6 test-optional-groups.R leak_mem pass 0.060 coveralls maste
23.1 match one optional group max_mem pass 0.004 coveralls maste
23.2 match one optional group leak_mem pass 0.004 coveralls maste
23.3 match all optional groups max_mem pass 0.000 coveralls maste
23.4 match all optional groups leak_mem pass 0.000 coveralls maste
23.5 test-optional-groups.R max_mem pass 0.060 coveralls maste
23.6 test-optional-groups.R leak_mem pass 0.060 coveralls maste
sha date_time
11.1 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
11.2 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
11.3 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
11.4 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
11.5 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
11.6 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
12.1 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
12.2 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
12.3 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
12.4 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
12.5 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
12.6 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
13.1 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
13.2 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
13.3 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
13.4 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
13.5 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
13.6 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
21.1 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
21.2 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
21.3 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
21.4 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
21.5 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
21.6 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
22.1 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
22.2 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
22.3 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
22.4 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
22.5 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
22.6 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
23.1 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
23.2 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
23.3 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
23.4 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
23.5 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
23.6 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
Note: The values in 'test_name' correspond to either the file name itself or one of its testthat blocks. Each value in the 'message' column corresponds to the first few characters of a commit message, i.e. each unique value represents a commit.
The parameters, branch1 and branch2, can be used to specify the branches which are to be tested. Behind the scenes, Rperform finds the latest commit common to the two branches. For branch2, only the latest commit is tested. For branch1, all the commits dating back to and including the latest common commits are tested. This performance testing strategy is based on the assumption that branch1 is to be merge into branch2. Or alternatively, that branch1 is derived from branch2. Provided below is a pictorial representation of the situation.
Note: For each individual commit, three values of a particular metric are measured. Hence, the multiple values for a single commit exist in the dataframes. Many a times the values simply overlap.
-> compare_brancht() can be used to compare the runtime details of a test file and testthat blocks across two branches. A modified version of the package stringr is used for demonstration purposes.
## Warning: Always set the current directory to be the root directory of the package you intend to test.
> setwd("path/to/stringr")
> library(Rperform)
> compare_brancht(test_path = "tests/testthat/test-dup.r", branch1 = "rperform_test", branch2 = "master")
test_name metric_name status metric_val message
1 basic duplication works runtime (in seconds) pass 0.002635558 Test commit 4 e
2 basic duplication works runtime (in seconds) pass 0.002132362 Test commit 4 e
3 basic duplication works runtime (in seconds) pass 0.002133985 Test commit 4 e
4 0 duplicates equals empty string runtime (in seconds) pass 0.001373857 Test commit 4 e
5 0 duplicates equals empty string runtime (in seconds) pass 0.001083645 Test commit 4 e
6 0 duplicates equals empty string runtime (in seconds) pass 0.001048471 Test commit 4 e
7 test-dup.r runtime (in seconds) pass 0.037045258 Test commit 4 e
8 test-dup.r runtime (in seconds) pass 0.004454978 Test commit 4 e
9 test-dup.r runtime (in seconds) pass 0.004620363 Test commit 4 e
10 basic duplication works runtime (in seconds) pass 0.002710479 Test commit 3 e
11 basic duplication works runtime (in seconds) pass 0.002090277 Test commit 3 e
12 basic duplication works runtime (in seconds) pass 0.002037836 Test commit 3 e
13 0 duplicates equals empty string runtime (in seconds) pass 0.001380925 Test commit 3 e
14 0 duplicates equals empty string runtime (in seconds) pass 0.001090150 Test commit 3 e
15 0 duplicates equals empty string runtime (in seconds) pass 0.001061177 Test commit 3 e
16 test-dup.r runtime (in seconds) pass 0.004586904 Test commit 3 e
17 test-dup.r runtime (in seconds) pass 0.003845612 Test commit 3 e
18 test-dup.r runtime (in seconds) pass 0.003805290 Test commit 3 e
19 basic duplication works runtime (in seconds) pass 0.002636744 Test commit 2 e
20 basic duplication works runtime (in seconds) pass 0.002150495 Test commit 2 e
21 basic duplication works runtime (in seconds) pass 0.002061330 Test commit 2 e
22 0 duplicates equals empty string runtime (in seconds) pass 0.001430191 Test commit 2 e
23 0 duplicates equals empty string runtime (in seconds) pass 0.001121196 Test commit 2 e
24 0 duplicates equals empty string runtime (in seconds) pass 0.001039320 Test commit 2 e
25 test-dup.r runtime (in seconds) pass 0.004438589 Test commit 2 e
26 test-dup.r runtime (in seconds) pass 0.003817823 Test commit 2 e
27 test-dup.r runtime (in seconds) pass 0.003776008 Test commit 2 e
28 basic duplication works runtime (in seconds) pass 0.002704231 Test commit 1 e
29 basic duplication works runtime (in seconds) pass 0.002106023 Test commit 1 e
30 basic duplication works runtime (in seconds) pass 0.002113326 Test commit 1 e
31 0 duplicates equals empty string runtime (in seconds) pass 0.001401971 Test commit 1 e
32 0 duplicates equals empty string runtime (in seconds) pass 0.001143783 Test commit 1 e
33 0 duplicates equals empty string runtime (in seconds) pass 0.001061584 Test commit 1 e
34 test-dup.r runtime (in seconds) pass 0.004576495 Test commit 1 e
35 test-dup.r runtime (in seconds) pass 0.004419175 Test commit 1 e
36 test-dup.r runtime (in seconds) pass 0.004655363 Test commit 1 e
37 basic duplication works runtime (in seconds) pass 0.003092646 Add simplify ar
38 basic duplication works runtime (in seconds) pass 0.002137115 Add simplify ar
39 basic duplication works runtime (in seconds) pass 0.002087054 Add simplify ar
40 0 duplicates equals empty string runtime (in seconds) pass 0.001411082 Add simplify ar
41 0 duplicates equals empty string runtime (in seconds) pass 0.001115703 Add simplify ar
42 0 duplicates equals empty string runtime (in seconds) pass 0.001090219 Add simplify ar
43 test-dup.r runtime (in seconds) pass 0.004144910 Add simplify ar
44 test-dup.r runtime (in seconds) pass 0.003783626 Add simplify ar
45 test-dup.r runtime (in seconds) pass 0.003781482 Add simplify ar
sha date_time branch
1 3d4e50d70a3f58ea5c092e5cda1b3f1489833ac0 2016-06-04 13:58:20 rperform_test
2 3d4e50d70a3f58ea5c092e5cda1b3f1489833ac0 2016-06-04 13:58:20 rperform_test
3 3d4e50d70a3f58ea5c092e5cda1b3f1489833ac0 2016-06-04 13:58:20 rperform_test
4 3d4e50d70a3f58ea5c092e5cda1b3f1489833ac0 2016-06-04 13:58:20 rperform_test
5 3d4e50d70a3f58ea5c092e5cda1b3f1489833ac0 2016-06-04 13:58:20 rperform_test
6 3d4e50d70a3f58ea5c092e5cda1b3f1489833ac0 2016-06-04 13:58:20 rperform_test
7 3d4e50d70a3f58ea5c092e5cda1b3f1489833ac0 2016-06-04 13:58:20 rperform_test
8 3d4e50d70a3f58ea5c092e5cda1b3f1489833ac0 2016-06-04 13:58:20 rperform_test
9 3d4e50d70a3f58ea5c092e5cda1b3f1489833ac0 2016-06-04 13:58:20 rperform_test
10 5099650de6f035aae5fe4ca35f33d26bd7c8a982 2016-06-02 17:22:53 rperform_test
11 5099650de6f035aae5fe4ca35f33d26bd7c8a982 2016-06-02 17:22:53 rperform_test
12 5099650de6f035aae5fe4ca35f33d26bd7c8a982 2016-06-02 17:22:53 rperform_test
13 5099650de6f035aae5fe4ca35f33d26bd7c8a982 2016-06-02 17:22:53 rperform_test
14 5099650de6f035aae5fe4ca35f33d26bd7c8a982 2016-06-02 17:22:53 rperform_test
15 5099650de6f035aae5fe4ca35f33d26bd7c8a982 2016-06-02 17:22:53 rperform_test
16 5099650de6f035aae5fe4ca35f33d26bd7c8a982 2016-06-02 17:22:53 rperform_test
17 5099650de6f035aae5fe4ca35f33d26bd7c8a982 2016-06-02 17:22:53 rperform_test
18 5099650de6f035aae5fe4ca35f33d26bd7c8a982 2016-06-02 17:22:53 rperform_test
19 240aec590ed7c0e6e34099062e6f60818d3928e9 2016-06-02 17:22:34 rperform_test
20 240aec590ed7c0e6e34099062e6f60818d3928e9 2016-06-02 17:22:34 rperform_test
21 240aec590ed7c0e6e34099062e6f60818d3928e9 2016-06-02 17:22:34 rperform_test
22 240aec590ed7c0e6e34099062e6f60818d3928e9 2016-06-02 17:22:34 rperform_test
23 240aec590ed7c0e6e34099062e6f60818d3928e9 2016-06-02 17:22:34 rperform_test
24 240aec590ed7c0e6e34099062e6f60818d3928e9 2016-06-02 17:22:34 rperform_test
25 240aec590ed7c0e6e34099062e6f60818d3928e9 2016-06-02 17:22:34 rperform_test
26 240aec590ed7c0e6e34099062e6f60818d3928e9 2016-06-02 17:22:34 rperform_test
27 240aec590ed7c0e6e34099062e6f60818d3928e9 2016-06-02 17:22:34 rperform_test
28 3e82daeb4c06cd6f64c2342778c199e7171788c7 2016-06-02 17:22:19 rperform_test
29 3e82daeb4c06cd6f64c2342778c199e7171788c7 2016-06-02 17:22:19 rperform_test
30 3e82daeb4c06cd6f64c2342778c199e7171788c7 2016-06-02 17:22:19 rperform_test
31 3e82daeb4c06cd6f64c2342778c199e7171788c7 2016-06-02 17:22:19 rperform_test
32 3e82daeb4c06cd6f64c2342778c199e7171788c7 2016-06-02 17:22:19 rperform_test
33 3e82daeb4c06cd6f64c2342778c199e7171788c7 2016-06-02 17:22:19 rperform_test
34 3e82daeb4c06cd6f64c2342778c199e7171788c7 2016-06-02 17:22:19 rperform_test
35 3e82daeb4c06cd6f64c2342778c199e7171788c7 2016-06-02 17:22:19 rperform_test
36 3e82daeb4c06cd6f64c2342778c199e7171788c7 2016-06-02 17:22:19 rperform_test
37 a67f8f06d4c8e8cac98d3361af66cc209b33f446 2015-11-05 07:09:01 master
38 a67f8f06d4c8e8cac98d3361af66cc209b33f446 2015-11-05 07:09:01 master
39 a67f8f06d4c8e8cac98d3361af66cc209b33f446 2015-11-05 07:09:01 master
40 a67f8f06d4c8e8cac98d3361af66cc209b33f446 2015-11-05 07:09:01 master
41 a67f8f06d4c8e8cac98d3361af66cc209b33f446 2015-11-05 07:09:01 master
42 a67f8f06d4c8e8cac98d3361af66cc209b33f446 2015-11-05 07:09:01 master
43 a67f8f06d4c8e8cac98d3361af66cc209b33f446 2015-11-05 07:09:01 master
44 a67f8f06d4c8e8cac98d3361af66cc209b33f446 2015-11-05 07:09:01 master
45 a67f8f06d4c8e8cac98d3361af66cc209b33f446 2015-11-05 07:09:01 master
-> Similarly, compare_branchm() can be used to compare the runtime details of a test file and testthat blocks across two branches.
## Warning: Always set the current directory to be the root directory of the package you intend to test.
> setwd("path/to/stringr")
> library(Rperform)
> compare_branchm(test_path = "tests/testthat/test-dup.r", branch1 = "rperform_test", branch2 = "master")
Note: The values in 'test_name' correspond to either the file name itself or one of its testthat blocks. Each value in the 'message' column corresponds to the first few characters of a commit message, i.e. each unique value represents a commit.