-
Notifications
You must be signed in to change notification settings - Fork 9
Obtaining package performance data using Rperform
Akash Tandon edited this page Jul 7, 2016
·
5 revisions
Rperform provides functions which can be used to obtain data containing information about a package's performance. Currently, there are two classes of functions which return performance data. They respectively deal with:
These functions, like most of the other Rperform functions, are designed to work in harmony with hadley's testthat package. However, they will work fine with the packages not employing testthat as well. Read on for further explanation.
These functions can be used to obtain data about the following metrics:
One can use Rperform to obtain the runtime of a test-file and the testthat code blocks it contains, if any.
One can use Rperform to obtain the memory usage details of a test-file and the testthat code blocks it contains, if any. There are two types of memory-related metrics which can be obtained:
- Maximum memory utilized: This is the maximum system memory utilized by the file/block during the course of its execution.
- Total memory leaked: This is the difference in the amount of system memory being used at the end and start of a file or block's execution.
Note: For each individual commit, three values of a particular metric are measured. Hence, the multiple values for a single commit exist in the dataframes. Many a times the values simply overlap.
-> time_compare() can be used to obtain the runtime details of a test file and testthat blocks over a specified number of commits. The package namedCapture is used for demonstration purposes.
## Warning: Always set the current directory to be the root directory of the package you intend to test.
> setwd("path/to/namedCapture")
> library(Rperform)
> time_compare(test_path = "tests/testthat/test-optional-groups.R", num_commits = 3)
test_name metric_name status metric_val message
1 match one optional group runtime (in seconds) pass 0.000783163 service=github
2 match one optional group runtime (in seconds) pass 0.000534672 service=github
3 match one optional group runtime (in seconds) pass 0.000515888 service=github
4 match all optional groups runtime (in seconds) pass 0.001008681 service=github
5 match all optional groups runtime (in seconds) pass 0.000763691 service=github
6 match all optional groups runtime (in seconds) pass 0.000751090 service=github
7 test-optional-groups.R runtime (in seconds) pass 0.022617698 service=github
8 test-optional-groups.R runtime (in seconds) pass 0.002455121 service=github
9 test-optional-groups.R runtime (in seconds) pass 0.002314949 service=github
10 match one optional group runtime (in seconds) pass 0.000800733 coveralls maste
11 match one optional group runtime (in seconds) pass 0.000533138 coveralls maste
12 match one optional group runtime (in seconds) pass 0.000529287 coveralls maste
13 match all optional groups runtime (in seconds) pass 0.001104596 coveralls maste
14 match all optional groups runtime (in seconds) pass 0.000811456 coveralls maste
15 match all optional groups runtime (in seconds) pass 0.000802304 coveralls maste
16 test-optional-groups.R runtime (in seconds) pass 0.002871598 coveralls maste
17 test-optional-groups.R runtime (in seconds) pass 0.002594260 coveralls maste
18 test-optional-groups.R runtime (in seconds) pass 0.002462942 coveralls maste
19 match one optional group runtime (in seconds) pass 0.000865201 Merge branch 'c
20 match one optional group runtime (in seconds) pass 0.000614288 Merge branch 'c
21 match one optional group runtime (in seconds) pass 0.000518979 Merge branch 'c
22 match all optional groups runtime (in seconds) pass 0.001008659 Merge branch 'c
23 match all optional groups runtime (in seconds) pass 0.000814166 Merge branch 'c
24 match all optional groups runtime (in seconds) pass 0.000795073 Merge branch 'c
25 test-optional-groups.R runtime (in seconds) pass 0.002618324 Merge branch 'c
26 test-optional-groups.R runtime (in seconds) pass 0.002350283 Merge branch 'c
27 test-optional-groups.R runtime (in seconds) pass 0.002296139 Merge branch 'c
sha date_time
1 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
2 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
3 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
4 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
5 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
6 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
7 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
8 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
9 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
10 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
11 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
12 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
13 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
14 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
15 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
16 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
17 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
18 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
19 72348c84d212abe5860933bd5b2d72c242c2db6e 2015-12-01 11:05:22
20 72348c84d212abe5860933bd5b2d72c242c2db6e 2015-12-01 11:05:22
21 72348c84d212abe5860933bd5b2d72c242c2db6e 2015-12-01 11:05:22
22 72348c84d212abe5860933bd5b2d72c242c2db6e 2015-12-01 11:05:22
23 72348c84d212abe5860933bd5b2d72c242c2db6e 2015-12-01 11:05:22
24 72348c84d212abe5860933bd5b2d72c242c2db6e 2015-12-01 11:05:22
25 72348c84d212abe5860933bd5b2d72c242c2db6e 2015-12-01 11:05:22
26 72348c84d212abe5860933bd5b2d72c242c2db6e 2015-12-01 11:05:22
27 72348c84d212abe5860933bd5b2d72c242c2db6e 2015-12-01 11:05:22
-> mem_compare() can be used to obtain the memory usage details of a test file and testthat blocks over a specified number of commits. The package namedCapture is used for demonstration purposes.
## Warning: Always set the current directory to be the root directory of the package you intend to test.
> setwd("path/to/namedCapture")
> library(Rperform)
> mem_compare(test_path = "tests/testthat/test-optional-groups.R", num_commits = 2)
test_name metric_name status metric_val message
11.1 match one optional group max_mem pass 0.008 service=github
11.2 match one optional group leak_mem pass 0.008 service=github
11.3 match all optional groups max_mem pass 0.000 service=github
11.4 match all optional groups leak_mem pass 0.000 service=github
11.5 test-optional-groups.R max_mem pass 0.060 service=github
11.6 test-optional-groups.R leak_mem pass 0.060 service=github
12.1 match one optional group max_mem pass 0.004 service=github
12.2 match one optional group leak_mem pass 0.004 service=github
12.3 match all optional groups max_mem pass 0.000 service=github
12.4 match all optional groups leak_mem pass 0.000 service=github
12.5 test-optional-groups.R max_mem pass 0.060 service=github
12.6 test-optional-groups.R leak_mem pass 0.060 service=github
13.1 match one optional group max_mem pass 0.004 service=github
13.2 match one optional group leak_mem pass 0.004 service=github
13.3 match all optional groups max_mem pass 0.000 service=github
13.4 match all optional groups leak_mem pass 0.000 service=github
13.5 test-optional-groups.R max_mem pass 0.060 service=github
13.6 test-optional-groups.R leak_mem pass 0.060 service=github
21.1 match one optional group max_mem pass 0.004 coveralls maste
21.2 match one optional group leak_mem pass 0.004 coveralls maste
21.3 match all optional groups max_mem pass 0.000 coveralls maste
21.4 match all optional groups leak_mem pass 0.000 coveralls maste
21.5 test-optional-groups.R max_mem pass 0.060 coveralls maste
21.6 test-optional-groups.R leak_mem pass 0.060 coveralls maste
22.1 match one optional group max_mem pass 0.004 coveralls maste
22.2 match one optional group leak_mem pass 0.004 coveralls maste
22.3 match all optional groups max_mem pass 0.000 coveralls maste
22.4 match all optional groups leak_mem pass 0.000 coveralls maste
22.5 test-optional-groups.R max_mem pass 0.060 coveralls maste
22.6 test-optional-groups.R leak_mem pass 0.060 coveralls maste
23.1 match one optional group max_mem pass 0.004 coveralls maste
23.2 match one optional group leak_mem pass 0.004 coveralls maste
23.3 match all optional groups max_mem pass 0.000 coveralls maste
23.4 match all optional groups leak_mem pass 0.000 coveralls maste
23.5 test-optional-groups.R max_mem pass 0.060 coveralls maste
23.6 test-optional-groups.R leak_mem pass 0.060 coveralls maste
sha date_time
11.1 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
11.2 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
11.3 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
11.4 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
11.5 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
11.6 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
12.1 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
12.2 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
12.3 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
12.4 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
12.5 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
12.6 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
13.1 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
13.2 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
13.3 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
13.4 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
13.5 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
13.6 05175927a45c301a18e8c6ebae67ea39a842d264 2015-12-01 11:09:04
21.1 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
21.2 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
21.3 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
21.4 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
21.5 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
21.6 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
22.1 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
22.2 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
22.3 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
22.4 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
22.5 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
22.6 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
23.1 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
23.2 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
23.3 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
23.4 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
23.5 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
23.6 68e1d1d33bdba36be6ee0e42743d3e7ab870814c 2015-12-01 11:07:36
Note: The values in 'test_name' correspond to either the file name itself or one of its testthat blocks. Each value in the 'message' column corresponds to the first few characters of a commit message, i.e. each unique value represents a commit.
Note: For each individual commit, three values of a particular metric are measured. Hence, the multiple values for a single commit exist in the dataframes. Many a times the values simply overlap.