Best-worse question type

[willking98]

I am trying to think of a less computationally burdensome method of preventing individuals from selecting the same attribute-level combination as best and worst in two different questions. Currently I have a number of show_if conditionals which removes the best option from the options shown in the worst question. While this looks good and prevents irrational responses, it probably is causing the survey to slow down when handling multiple concurrent sessions.

My question is basically, would having a condition (like below) on every question be less computationally burdensome than having many show_if conditions?

if(input$cbc_q1_best == input$cbc_q1_worst){
print("Warning: You have selected the same option as best and worst!")
}

Edit:
Could the newly developed cbc matrix format be tweaked to include radiobuttons in the matrix to select best and worst options as is shown here in this image below? This would be the rows in the middle and the options either side. Presumably the radiobuttons prevent multiple responses anyway so best and worst for the same attribute-level combination would not be possible.

I suspect this would be the least computationally demanding option.

Update:

This is where I have got to on the new matrix question format.

There are a few issues:

1. It would be great if there was markdown support to make some of the label text bold.
2. A response is required for each row whereas what is required is one response for each column.
3. Moving the best and worst to either side of the rows would be compatible with more normal formatting for best-worst scaling.

Update2:
When transposed this solves issue 2 but doesn't allow for the space to give the full description of the health state (see below screenshot).

Also, this still requires the code which shows a warning if best and worst are the same. What I am ideally looking for is a variation on the above screenshot where the matrix has been transposed but with the best and worst columns either side of the row label.

[willking98]

Sorry there's a lot of info in the above!

[jhelvy]

I've had a similar issue with trying to come up with a way to make a best-worst question. What you've tried already is a step in the right direction and all the things I was going to suggest. As for the restriction of not allowing the user to choose the same value for both best and worst, I'm not sure if that is possible as of now. We'd have to really think hard about how to implement this. But also, how badly do you need that restriction? In my opinion, if I got responses where these were the same, that's an indicator that this person isn't paying attention or isn't understanding the question, so I would just filter them out. You could have some skip logic that just kicks people out if the answers are the same, that's one thing to consider. But otherwise I might not actually try to restrict this because it could actually be useful for identifying bad responses.

[willking98]

Thanks for that John. I think I will probably do as you say and park the idea of removal of the best option for the worst question (or the warning) and just use irrational responses as a consistency check/measure.

How possible do you think it would be to allow the functionality to change the formatting so that the matrix was transposed and best and worst are left and right of the labels (per row)? It would need to have the functionality of also only requiring a response from one row in the best column and one row in the worst column.

If this is a bit too complicated/not possible given the current architecture I will go back to my previous strategy of asking sequential questions with mc_buttons but without any show_if conditionals. I tested the speed with 10 concurrent sessions with this matrix style question (no show_if) and it did much better (only a little bit of lag when loading up all surveys together).

[jhelvy]

I think a best-worst type question as you're describing would probably have to be it's own question type. We'd have to define it similar to how we created a matrix question, but with the left and right orientation. We probably won't make this right now as there are other priorities, but it's something I also do use so I also would like to have it. Of course, if you want to contribute it yourself you could look at how the matrix question type is defined and see if you could make a bestworst type maybe? A matrix question just a series of mc type question smashed together with some CSS to make it look better, so the bestworst would be something similar. The challenge I think is the labels, not sure how to orient it so that you have the left and right orientation.

[willking98]

I've just had a bit of an attempt at making a bestworst type by altering the ui file and trying to get it working locally but it wasn't very successful and I'm pretty out of my depth.

I think I am probably going to revert back to the two question mc_buttons BWS setup I had before if the bestworst functionality isn't likely to be added soon. As I said previously, I will likely just use the irrational responses as a consistency check. Having the markdown support for bold text in the label is pretty important for my study as well to emphasise the attribute level.

Cheers for the help!

[jhelvy]

It's now on the todo list!

[jhelvy]

Quick update - I changed the title of this issue because it has become more about the best-worst type question than the original question (which was about posting a warning for an invalid response...we're working on that anyway).

I had an idea though this morning that I think results in a better UI for this type of question. It renders like this:

I'm leveraging the CSS grid layout feature of Quarto to put the questions side-by-side, and I'm putting the general question label in the regular markdown and setting the question labels as Best and Worst. Here's how I set this up in the survey.qmd file:

::: {#page1 .sd-page}

<center>
Which do you think is **best** and **worst** from this list? 
</center>
<br>

::: {.grid}

::: {.g-col-6}

```{r}
sd_question(
  id     = "pet_best",
  type   = "mc_buttons",
  label  = "**Best**",
  option = c(
    "Dogs" = "dogs", 
    "Cats" = "cats", 
    "Fish" = "fish", 
    "Birds" = "birds"
  ), 
  direction = 'vertical'
)
```

:::

::: {.g-col-6}

```{r}
sd_question(
  id     = "pet_worst",
  type   = "mc_buttons",
  label  = "**Worst**",
  option = c(
    "Dogs" = "dogs", 
    "Cats" = "cats", 
    "Fish" = "fish", 
    "Birds" = "birds"
  ), 
  direction = 'vertical'
)
```

:::

:::

```{r}
sd_next()
```

:::

[willking98]

I've done a little bit of stress testing and things did slow down quite a bit. Though this was likely attributable to the number of show_if conditionals I had. I've done tests with simpler surveys and things worked at an acceptable speed.

Fortunately Prolific has the functionality to limit the number of people that can respond to the survey at once. I will definitely be making use of this functionality!

[jhelvy]

Oh awesome I didn't know about that! That's important - might even add this to the docs.

[willking98]

I've got this working really nicely now, see here. It doesn't render on a mobile quite as nicely but looks amazing on the PC.

Looking forward to getting this deployed once the invalid response feature is added!

[jhelvy]

Looks pretty good! You might could put a note on the first page that says that (i.e. you recommend users take the survey on a computer rather than phone).

[willking98]

Thanks! Prolific to the rescue again!

[jhelvy]

Just added a performance page to give some general advice on setting up your app to run well (using exactly some of these screenshots).