Interactive plots

[analyticalmonk]

Introduce interactivity in the performance plots being rendered. Similar to the ones returned by
asv (example) and codespeed (example).

@tdhock animint could be used for something like this, right? Or are there any other alternatives off the top of your mind?

[joshuaulrich]

dygraphs is the first thing that comes to my mind.

[analyticalmonk]

Thanks @joshuaulrich
Looks like a good option, will go through the dygraphs docs once.

[tdhock]

animint is best when you have multiple plots and you want to click one of them to update what is shown in the others. maybe we could use it to explore the larger data set (time and memory measurements across several commits, for several different tests)

[analyticalmonk]

Yeah, makes sense @tdhock
In any case, for what I had in mind, a library such as dygraphs will do.

[analyticalmonk]

Interactive plots will be most useful when working with travis. Thus, it didn't make sense to mess much with the existing plotting functions.
Instead, adding a interactive parameter to the existing functions would be a better option. This way, when working locally, a developer can use the normal ggplots such as the ones Rperform currently provides. And the interactive ones can be used to work with pull requests and on travis by setting interactive = T.

animint seems to have issues dealing with the existing facets and vertical alignment of x-axis text labels. But it can be dealt with.
For example, when interactive would be set to true, instead of the usual plot which contains multiple tests (facet_grid(test_name~.)), something like this animint would be made. Each test would have a single plot to deal with the facet problem, and commit messages will be included as hover text rather than on the x-axis.

But one issue is the way plots are aligned next to one another (in this case, 3 at a time). This cuts out the title text and makes it messy in general. Is there a way to make it so that animint plots are rendered one after another in a top-down manner (vertically)?

[tdhock]

Hey @analyticalmonk did you try facet_grid(test_name~., scales="free") ? that should work in animint just like in ggplot2...

[tdhock]

another option is to use aes(showSelected=test_name) to show details for only one test at a time. Then add a second plot with aes(clickSelects=test_name) to select which test you want to see.

[analyticalmonk]

I had tried using facet_grid... The text labels for each facet overlapped. It rendered the following plot:

An animint for the tests using showSelected=test_name is here: https://analyticalmonk.github.io/Rperform/
It works fine except for the issue of there not being different scales of y-axis for each test's metrics. It might be okay to use this if the tests inside a test file had relatively equal metric values. But I am not sure if this is the best way to go.

What would be better? To work with facets only and try other workarounds such as shortening label length, or using color legends. Or using the method employing showSelected.

[tdhock]

for now I think facets are better. (but we have another GSOC student working @faizan-khan-iit on getting animint's axes to update after changing the selection, so hopefully that could be an option by the end of summer)

for the facets you can either try to shorten the facet label or give the facets more space with theme_animint(height=1000)

[tdhock]

also you may consider just copying the code used by codespeed and asv. Do you think it would be possible to output some data files from R and have those files be read directly by the codespeed/asv viz tools?

[analyticalmonk]

The Rperform gh-page now contains an updated version of the animint. I wanted to see how it works on a random package from the wild so used it on namedCapture; seems to work fine.
About the updates itself:

• The plot again uses facet_grid() but I have shortened the labels. Specifically, the new label will contains the first three and the last three characters of the test name separated by an ellipsis.
• The points now act as links to the relevant commits on github. It obtains the remote url from the repo itself using git2r::remote_url() and obtains the sha using the data frame from time_compare() itself. Combining the two with a '/commits/' in between, we get the github url.

@tdhock @joshuaulrich
Any feedback?

[analyticalmonk]

There were a couple of issues I ran into:

• Embedding the links using the href aesthetic means we won't be able to use the tooltip aesthetic. Seems like this is a documented bug.
• I am using ggplot2 1.0.1 since this is the latest version on the CRAN repository which conforms with animint's requirement for a version less than 2.0. But theme_animint(height = ) isn't working with it. I looked into it and found an issue and pull request which deals with this. Although the PR was merged, it must have been included in the version after 1.0.1. Being able to use the parameter to increase the animint's height would be useful for cases containing many internal tests (say >4). In those cases, just the shortened labels won't suffice. @tdhock, any thoughts about this?

Also, I will look into the possibility of using the asv viz tools. codespeed won't be an option since it works in tandem with Django but I will look into it again just to be sure.

[tdhock]

Hey Akash that looks good with the links to the namedCapture commits.

Also shortening the facet labels seems like a reasonable strategy (for now). Eventually I think it would be better to add a geom_text(aes(showSelected=test_name, label=test_name)) that shows the full test name, and another plot with an overview of test results, that you can click to select a test, e.g. geom_point(aes(average_memory_difference, average_time_difference, clickSelects=test_name))

About the href/tooltip problem, please post a simple example as an issue https://github.com/tdhock/animint/issues/new and tag @faizan-khan-iit who may be able to work on that in the next few weeks (Faizan, please add Akash's example as a test case -- seems like this should be an easy fix, and it would be a good introduction for you to the JS code).

Akash the newest version of animint's master branch is now working with ggplot2 >= 2.0 so you should try installing that.

[analyticalmonk]

@tdhock I didn't entirely understand the details of the plot you are suggesting. I get that geom_text() would be added to the current plot itself. But what would the second plot display exactly and what would its y and x axis represent?

Also, I opened issue #155 for animint as you had suggested.

[tdhock]

Akash, sorry for the confusion. Right now you have a single ggplot()+geom_point()+facet_grid(test_name~.) that shows all the results. I would suggest another data viz with at least two plots: one plot with an overview that you can click to select a single test (clickSelects=test_name), and another plot with details of the selected test (showSelected=test_name). For the overview plot you would have to compute a new data frame with some summary statistics for each test, for example average_time_difference and average_memory_difference (say, between master and the most recent or most different commit).

[tdhock]

If that is still confusing, can you please post the data of test results and I will write some example ggplot/animint code for you.

[analyticalmonk]

I still don't get it entirely but I have tried making a rough viz from what I understood. It's hosted here.

I have used mean as a summary statistic for the sake convenience here.

The data I am working with for this is in the following format:

                          test_name metric_name status  metric_val         message
1           basic duplication works     seconds   pass 0.002615133 Add simplify ar
2           basic duplication works     seconds   pass 0.002128348 Add simplify ar
3           basic duplication works     seconds   pass 0.002166373 Add simplify ar
4  0 duplicates equals empty string     seconds   pass 0.001415706 Add simplify ar
5  0 duplicates equals empty string     seconds   pass 0.001127474 Add simplify ar
6  0 duplicates equals empty string     seconds   pass 0.001061144 Add simplify ar
7                        test-dup.r     seconds   pass 0.004567220 Add simplify ar
8                        test-dup.r     seconds   pass 0.003990853 Add simplify ar
9                        test-dup.r     seconds   pass 0.003877404 Add simplify ar
10          basic duplication works     seconds   pass 0.002700498   News tweaking
11          basic duplication works     seconds   pass 0.002147713   News tweaking
12          basic duplication works     seconds   pass 0.002077510   News tweaking
13 0 duplicates equals empty string     seconds   pass 0.001445744   News tweaking
14 0 duplicates equals empty string     seconds   pass 0.001148991   News tweaking
15 0 duplicates equals empty string     seconds   pass 0.001166428   News tweaking
16                       test-dup.r     seconds   pass 0.004542412   News tweaking
17                       test-dup.r     seconds   pass 0.004003162   News tweaking
18                       test-dup.r     seconds   pass 0.004058602   News tweaking

I have omitted the sha and datetime columns

Code I wrote to obtain the above:

time_data <- Rperform::time_compare(test_path = "tests/testthat/test-dup.r", num_commits = 4)
summary_data <- aggregate(time_data$metric_val, by = list(time_data$test_name), mean)
names(summary_data) <- c("test_name", "mean_val")

time_plot <- ggplot2::ggplot() +
  ggplot2::geom_point(mapping = ggplot2::aes(x = message, y = metric_val,
                                             showSelected = test_name,
                                             href = "https://github.com"),
                      color = "blue",
                      data = time_data) +
#   ggplot2::geom_text(mapping = ggplot2::aes(x = message, y = metric_val,
#                                             label = test_name,
#                                             showSelected = test_name),
#                      data = time_data) +
  ggplot2::theme(axis.text.x = ggplot2::element_blank()) +
  ggplot2::scale_x_discrete(limits = rev(levels(time_data$message))) +  
  ggplot2::xlab("Commit message") +
  ggplot2::ylab("Runtime value") +
  ggplot2::ggtitle(label = paste0("Variation in runtime"))

summary_plot <- ggplot2::ggplot() + 
  ggplot2::geom_point(mapping = ggplot2::aes(x = test_name, y = mean_val,
                                             size = 2,
                                           clickSelects = test_name),
                      data = summary_data) +
  ggplot2::geom_text(mapping = ggplot2::aes(x = test_name, y = mean_val, 
                                            clickSelects = test_name,
                                            showSelected = test_name,
                                            label = test_name),
                     data = summary_data) +
  ggplot2::theme(axis.text.x = ggplot2::element_blank()) +
  ggplot2::xlab("Tests") +
  ggplot2::ylab("Mean values")

viz.list <- list(timeplot = time_plot, sumplot = summary_plot)
structure(viz.list, class = "animint")

[tdhock]

that looks like a step in the right direction.

now I would suggest changing the y axis variable on the details plot to a normalized variable, so you can use up all the space in the plot. e.g. normTime = (time-min(time))/(max(time)-min(time)) for every value of test_name. In that case I would suggest putting the geom_text(aes(x=0.5, y=1, showSelected=test_name)) label on the top of the details plot as an interactive title.

also can you change the x axis of the details plot to another quantitative metric? For example you could compute a p-value using a one-sided, unpaired t-test (test if mean of the master branch is less than the mean of each commit from the other branch) and then plot the minimum p-value over all commits.

I would also suggest adding data for all test files, and then you can have three plots: one to select the test file, one to select the particular test block, and finally the details for the selected tests. In that case you could add another text label as a title for the currently selected test file.

[tdhock]

hey @analyticalmonk, thanks to @faizan-khan-iit who worked on animint in GSOC2016, we now have axis updates, which would be very useful for your performance data viz. You just need to add the update_axes=c("x", "y") option to the viz.list. You can see other examples in the test cases: https://github.com/tdhock/animint/blob/master/tests/testthat/test-renderer4-update-axes-multiple-ss.R

[analyticalmonk]

@tdhock Sounds good. Will do the changes.

[analyticalmonk]

@tdhock Had been extremely busy over the last few weeks. Will be making working and making a few changes to Rperform over the next few days. Guess will finally make the interactive plots work with the updates to animint.

[tdhock]

great, if you have any issues with animint, don't hesitate to get in touch with me or @faizan-khan-iit