-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathexperiment.tex
256 lines (195 loc) · 10.2 KB
/
experiment.tex
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
\section{Evaluation of Slide}
\subsection{Participants}
Fifteen subjects, 5 female and 15 male, participated in the study.
All were university students and active users of technology.
Each subject took about 30 minutes to complete the study.
\subsection{Measures}
In the field of text entry, several metrics are used to characterize a method's performance ~\cite{wobbrock2007measures,arif2009analysis}.
Here, we discuss the performance metrics we use to evaluate the performance of the input device.
\subsubsection{Words per Minute}
Words per minute ($WPM$) is perhaps the most widely reported empirical measure of
text entry performance~\cite{wobbrock2007measures}:
\[
WPM={\vert T\vert -1\over S}\times 60\times{1\over 5}. \eqno{\hbox{(1)}}
\]
Where, $S$ is the time in seconds from the first key press to the last, which means that the entry of the first character is never timed, which is the motivation for the ``- 1'' in the numerator of Equation $1$ ~\cite{yamada1980historical}.
English words by convention are treated as having five characters~\cite{yamada1980historical}.
\subsubsection{Error Rate}
Error Rate ($ER$) is the ratio of the total number of incorrect characters in the transcribed text to the length of the transcribed text:
\[
ER={INF\over \vert T\vert }\times 100\%. \eqno{\hbox{(2)}}
\]
Where, Incorrect Not Fixed ($INF$) is the number of unnoticed incorrect characters in the transcribed text.
\subsection{Experiment \#1 - 26-Key Slide Keyboard}
Our first experiment is to measure the speed at which users can type in uncommon English words. The user has to slide his finger to the specific keys to enter the characters precisely. There is no auto-correct used in this mode as the primary use is for words that are not in the language model.
In this experiment, the users were asked to type in two phrases: one that contained a proper noun and one that contained an abbreviation. We found that the users can enter these phrases with an average entry rate of 12 WPM, with a standard deviation of 1.8. We observed that practice has no significant effect on improving the speed.
12 words per minute is a reasonable input rate, considering that input words are uncommon, and errors in entry cannot be corrected automatically.
\subsection{Experiment \#2 - 6-tile Slide Keyboard}
For this experiment, we selected fifty phrases from a collection of five-hundred phrases commonly used for text entry evaluations~\cite{mackenzie2003phrase}.
Example of phrases from the set are ``video camera with a zoom lens'', ''have a good weekend'', and''what a monkey sees a monkey will do''.
To allow for cross-comparison to other text entry methods, the corpus doesn't include phrases with punctuation marks.
Non-alphanumeric symbols are rarely considered in text input research~\cite{mackenzie2003phrase} even though some punctuations (. - ' ( ) ") are more frequent than the least common English letter, \textit{q}~\cite{malikpunctuation}.
Copying pre-selected phrase is usually the preferred method for text entry when doing evaluations in a lab setting~\cite{mackenzie2002character, mackenzie2003phrase}.
Although, copying pre-selected entry rates should be regarded differently from true, ``in the wild'' results when the user is composing text instead of transcribing prompted phrases.
One of our metrics in the experiment is the learnability of the system so we don't give users practice trials before beginning the actual test.
For each user we conduct twelve sequential sessions.
In each session, a user enters six phrases.
%After entering all the phrases with both input methods, the participant filled out
%a questionnaire regarding their quantitative feedback on the text entry method.
%Finally, subjects were asked for additional comments.
\begin{comment}
\begin{figure}
\centering
\begin{subfigure}{1\columnwidth}
\begin{tabular}{@{}p{7cm}p{1cm}@{}}\toprule
\textbf{Performance} & \\
\midrule
The input device is responsive & $3.4$ \\
The input device worked properly & $4.3$ \\
The visuals are smooth and don't freeze & $3.7$ \\
\bottomrule
\end{tabular}
\caption{Overcoming latency is a main challenge for high bandwidth tasks in virtual reality.
Users find the input device works properly and is responsive.}
\label{fig:performance}
\end{subfigure}
\vspace{4mm}
\begin{subfigure}{1\columnwidth}
\begin{tabular}{@{}p{7cm}p{1cm}@{}}\toprule
\textbf{Design} & \\
\midrule
I understand how input device works & $3.1$ \\
It is easy to learn how to use the input device & $3.9$ \\
I understand the graphical interface & $3.9$ \\
The text and input are clearly visible & $2.7$ \\
\bottomrule
\end{tabular}
\caption{Users agree that they understood how the input device and the graphical interface worked.
Some found the text to be not clearly visible.}
\label{fig:design}
\end{subfigure}
\vspace{4mm}
\begin{subfigure}{1\columnwidth}
\begin{tabular}{@{}p{7cm}p{1cm}@{}}\toprule
\textbf{Applicability} & \\
\midrule
I would use the input device to do a web search & $3.3$ \\
I would use the input device to write a note & $2.9$ \\
I would use the input device at home & $3.2$ \\
\bottomrule
\end{tabular}
\caption{User agreed that the Slide input device could be used to do a web search, write a note, and used at home. }
\label{fig:controllerVive}
\end{subfigure}
\caption{We ask the user to evaluate the performance, design, ergonomics and applicability of Slide on a 5-point Likert Scale (Strongly Disagree, Disagree, neutral, Agree, Strongly Agree).}
\label{table:usability}
\end{figure}
\subsubsection{Subjective }
we ask the user to evaluate each entry mechanism on a 5-point Likert Scale (Strongly Disagree, Disagree, neutral, Agree, Strongly Agree).
Additionally, users are interviewed to further understand the their qualitative experience.
\end{comment}
The collective learning curve is shown in Figure~\ref{fig:learnability}.
The mean starting rate for all users was 18 WPM.
Every user improved their WPM rate by at least 20\% from their starting rate.
To calculate the steady state average rate for text entry, we discard the first six sessions and find the steady state rate is 34 WPM with a standard deviation of 6.3. The average error rate for text entry was 4\% with a standard deviation of .018.
\begin{figure}
\centering
\begin{tikzpicture}
\pgfplotsset{
every axis legend/.append style={
at={(0.5,1.03)
},
anchor=south
}
}
\pgfplotsset{grid style={dashed,gray}}
\begin{axis}[
legend columns=-1,
xlabel=Session,
ylabel=Words Per Minute (WPM),
title style={at={(0.5,0)},anchor=south,yshift=180},
title = Learnability of Slide,
%ytick={0,5,10,15,20},
%minor ytick={1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20},
xtick={1,2,3,4,5,6,7,8,9,10,11,12},
%minor xtick={1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20},
grid=both,
]
\addplot[opacity=1] coordinates
{(1,19)(2,24)(3,28)(4,32)(5,32)(6,34)(7,33)(8,33)(9,33)(10,32)(11,35)(12,34)};
\addplot[opacity=0.25] coordinates
{(1,12)(2,23)(3,27)(4,25)(5,31)(6,30)(7,33)(8,33)(9,39)(10,31)(11,32)(12,33)};
\addplot[opacity=0.25] coordinates
{(1,24)(2,21)(3,26)(4,24)(5,32)(6,36)(7,33)(8,34)(9,35)(10,35)(11,37)(12,37)};
\addplot[opacity=0.25] coordinates
{(1,31)(2,23)(3,27)(4,25)(5,29)(6,45)(7,37)(8,41)(9,36)(10,31)(11,41)(12,33)};
\addplot[opacity=0.25] coordinates
{(1,29)(2,40)(3,42)(4,42)(5,42)(6,44)(7,43)(8,39)(9,42)(10,43)(11,46)(12,48)};
\addplot[opacity=0.25] coordinates
{(1,9)(2,16)(3,22)(4,19)(5,25)(6,32)(7,26)(8,27)(9,33)(10,23)(11,23)(12,28)};
\addplot[opacity=0.25] coordinates
{(1,27)(2,29)(3,31)(4,32)(5,36)(6,36)(7,38)(8,34)(9,37)(10,35)(11,35)(12,35)};
\addplot[opacity=0.25] coordinates
{(1,9)(2,13)(3,20)(4,19)(5,20)(6,23)(7,19)(8,18)(9,27)(10,23)(11,24)(12,25)};
\addplot[opacity=0.25] coordinates
{(1,11)(2,17)(3,23)(4,26)(5,25)(6,25)(7,19)(8,21)(9,19)(10,23)(11,28)(12,20)};
\addplot[opacity=0.25] coordinates
{(1,18)(2,23)(3,27)(4,25)(5,28)(6,30)(7,33)(8,32)(9,36)(10,31)(11,32)(12,33)};
\addplot[opacity=0.25] coordinates
{(1,26)(2,19)(3,21)(4,25)(5,37)(6,39)(7,36)(8,36)(9,26)(10,40)(11,42)(12,38)};
\addplot[opacity=0.25] coordinates
{(1,15)(2,23)(3,27)(4,25)(5,31)(6,30)(7,33)(8,33)(9,26)(10,31)(11,32)(12,33)};
\addplot[opacity=0.25] coordinates
{(1,19)(2,26)(3,28)(4,22)(5,36)(6,30)(7,34)(8,37)(9,36)(10,31)(11,36)(12,37)};
\addplot[opacity=0.25] coordinates
{(1,7)(2,26)(3,25)(4,23)(5,23)(6,34)(7,29)(8,31)(9,28)(10,31)(11,28)(12,26)};
\addplot[opacity=0.25] coordinates
{(1,11)(2,21)(3,27)(4,25)(5,27)(6,30)(7,33)(8,30)(9,29)(10,31)(11,32)(12,33)};
\addplot[opacity=0.25] coordinates
{(1,32)(2,31)(3,37)(4,34)(5,44)(6,38)(7,42)(8,38)(9,32)(10,42)(11,45)(12,38)};
\end{axis}
\end{tikzpicture}
%4.7.1 Markers for graph fixing the circle, square, x, triangle overloading
\caption{
Graph of learnability for the 6-tile Slide keyboard.
Entry rate on each block of 20 sentences.
Each line is a single participant.
}~\label{fig:learnability}
\end{figure}
\begin{comment}
\begin{figure}
\centering
\begin{tikzpicture}[scale=.6]
\pie{44/Word entry with thumb not on trackpad, 3/Thumb on trackpad, 29/Between word, 24/Editing}[explode=0.1]
\end{tikzpicture}
\caption{The majority of time the user was entering a word, their thumb was not on the trackpad. There was also a delay between when the user was entering words.
}
~\label{fig:distance}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=0.5\columnwidth]{figures/circle}
\caption{Sample of a right-handed users' slides. Slides are concentrated in the upper-left quadrant of the circular trackpad.}
\label{fig:circle}
\end{figure}
\begin{figure}
\centering
\begin{comment}
\begin{tikzpicture}
\begin{axis}[
ybar,
bar width=30pt,
enlargelimits=0.15,
ylabel={Relative comfort rating},
symbolic x coords={arm,wrist,hand},
xtick=data,
]
\addplot coordinates {(arm,1.2) (wrist,.3) (hand,1.4)};
\end{axis}
\end{tikzpicture}
\caption{
Differences between the initial and final subjective comfort ratings.
}
\label{fig:graphLearning}
\end{figure}
\end{comment}