conclusion.tex

\chapter{Conclusion and Future Work}
\label{chap:conclusion}

% MN: Try to separate the 3 aspects of your work:
% 1. Research. The research questions are answered by the data that you have collected. 
% 2. Engineering. The data that you can collect is because you have constructed (engineered) a tool (or set of tools).
% 3. Design, Architecture and Construction. This is related to point 2, but, it is the decisions, the architecture, the game design, the balancing, the choices of frameworks, methods, flows, etc. All the "choices" that you had to make first, before point 2 can take it and "build it".  
% Point 2 is BSc work. Engineering. "Boring" or "Mundane". Sure, it was part of your work, but, give it 3rd, last place in discussions.
% Point 1 is your primary objective. Posing, and answering research questions. This is what MSc process is all about. 
% Point 3 is huge part of your work, and, of course, fundamental to get to point 1. But, it is still "a tool". So, it comes on the 2nd place. The first place is point 1.

% So, below, when you drawing conclusions, you have to make sure the reviewer is clear, if you are discussing point 1, or point 3, and why. Do not mix them. When you have research question that requires data, statistics, and the conclusion is drawn from the data and stats, keep it simple. Just write about that. Logical.   Keep arguing your case with data.  Not with "construction".


% Then, as an added value, added contribution, describe the construction. Point out, that you have, beside the research questions and answers, done work in solving architectural and design decisions, and they contribute to the quality of the overall work. But, try not to mix "research" with "construction". It will read better, and it will be easier to follow by the reviewers.



The focus of this research has been primarily on investigating gamification and games with a purpose for facilitating data gathering processes for named entity disambiguation (NED). To assure fast and reliable inter-communication between a gamified system and another system that carries out the complete automated process of entity disambiguation, a framework of our own was deemed necessary to be implemented as part of this study. The implementation of a microservice architectural framework with state-of-art utilized techniques for NED and the implementation of a complete gamified system which utilizes the output from all components of the framework represent the concrete contributions of this work. In order to test the effectiveness of this proposed approach to NED, two user experiments have been conducted. The results acquired from analyzing the data generated from the user experiments, statistically support some of the claims hypothesized by this study and helped answer the posed research questions which are individually addressed below.  \hfill \break

% State the major conclusions from your study and present the theoretical and practical implications of your study.


%ANSWER YOUR RESEACH QUESTIONS 

%User present tense when talking about facts

%User past tense or present perfect when when refering to the research done 

%Do not use first person, interpret new issues, provide new information, use examples, cut and paste passages from the results 

 \textbf{How accurate can an entity disambiguation framework, by using human input, validate the automatic linking process of named entities with knowledge bases?}\hfill \break

In order to answer the first research question, the effectiveness and usability of the information generated by the microservices had been tested through an experimental user study with non-expert participants validating links generated from the framework. For this purpose, AnnotateMe was implemented as a front-end interface to engage with participants by providing all the necessary information for the correct disambiguation to take place. \if Additionally, gold standard datasets where used as corpus for feeding in into the framework to be validated by the users. Since the gold standard provided the correct answers for each individual entity available in the datasets,we were able to assess the quality of the generated annotations by the non-expert annotators.\fi Statistically significant results indicate that the information generated by the microservice components of the framework support the generation of high quality annotation data by non-expert annotators with an accuracy (f-score) of 0.92. 

\if Having implemented a framework that supported quality of data at such high levels was crucial before starting the gamification process. However, since the framework consisted of novel implementations, specifically targeting the component which is responsible for generating contextual clues to help disambiguation, a scientific assessment of the usability of these contextual clues generated by the framework was mandatory. The second research question addresses the concerns about the effectiveness of contextual clues with regards to entity disambiguation.\fi \hfill \break
    
\textbf{What features can be used to formulate the context surrounding entities so that non-expert users can correctly disambiguate them?}  \hfill \break

\ac{nlp} is among the many research fields which have investigated in potential methods for defining context in a textual representative way~\cite{22}. The appropriate formulation of context solemnly depends on the task and target (human or machine) who makes use of it. This research study was concerned in finding features that would be appropriate and helpful to help non-expert users to easily grasp the context in which the entity occurs and correctly disambiguate it with the appropriate candidate. Proximity features such as bi-gram collocations, neighbor entity mentions and topical document keywords are the features that this study has experimented with. Statistical analysis of the results acquired from the first non-gamified experiment show that these contextual clues (features) proved to be helpful in the overall disambiguation process. However, as a result of insufficient data gathered during the experimental studies and the selection of a rather inappropriate assessment methodology for the effectiveness of the contextual clues resulted in acquiring statistically insignificant claims. We account this as a limitation of our study and address improvements for future work.\hfill \break

\if The effectiveness of these features was only measured using assessment questions from a post questionnaire where participants were asked to rate the usefulness of the clues with regards to the disambiguation process. Therefore, insignificant statistical results were acquired as a side effect of applying insufficient assessment methods for measuring the corresponding effectiveness of the designed context features.\fi


\textbf{What game mechanics can be employed in the entity disambiguation task so that high levels of engagement are achieved while still maintaining annotation quality?}  \hfill \break

The idea of applying game elements into non-gaming contexts with the purpose of solving a particular problem (\ac{gwap}) has been researched across many domains \cite{47}. A recent literature survey conducted by Seaborn et al \cite{45} shows that the majority of studies that have applied gamification did not base their work on empirical and theoretical foundations. The game design of Fastype has been entirely based on theoretical and psychological foundations. 

This research study hypothesized that the game elements applied to the \ac{ned} framework would hide the boring nature of the actual task and emphasize the entertaining and fun aspects of the game which in turn leads to the users being intrinsically motivated. Since the third research question is concerned in discovering the underlying game mechanics for reaching the respective state of being intrinsically motivated, the results of the study suggest applying the following game mechanics for a successful \ac{gwap}: 
\begin{itemize}
    \item Onboarding
    \item Triangularity (supporting both players who like to take risks and those who like to play safe)
    \item Non-Formal and visually appealing feedback
    \item Appropriately and dynamically adjusting game complexity as the player advances through the levels (maintaining game flow)
    \item Controlling for malicious player behaviour and maintaining gameplay quality
    \item Social interaction mechanisms within the game
    \item A well defined game engagement loop which motivates emotion, assures re-egagement with the game, provides social call to action and makes the progress of players socially visible
\end{itemize}
\if This research study opposes to some previous research work \cite{54} by showing that textual interactive-based GWAP can be as fun and entertaining as video-based GWAP. Statistically significant results prove that the game mechanics applied to the NED framework engaged participants significantly more than compared to using non-gamified interfaces, and as a result, participants were intrinsically motivated to participate. In terms of annotation quality, the gamified version of NED Framework did not have any negative impact and therefore annotation quality remained stable.\fi


The work and results reported by this research study have theoretical and practical implications in the respective field. This research study contributes to perishing the doubt of \ac{gwap} being successfully and effectively applied in non-gameing contexts. As long as the design of the game is based on empirical foundations and other psychological factors that impact human intrinsic motivation, the success is easily achievable. Providing novel approaches for effectively and efficiently improving named entity disambiguation systems results in continuous improvements in the areas of information retrieval systems and semantic web respectively \cite{12}. It is this sort of research work that devises techniques which can substantially improve efficiency and scalability while retaining high quality and accuracy of data. \hfill \break


\textbf{Future Work}  \hfill \break

% MN: Make the "larger sample size" a last, additional point. 

% MN: Start with the improvements to the tests that would strengthen your results in the context of gamification effectiveness. What future tests could you suggest that could improve the "game" aspects? What have you learned but still do not have sufficient evidence for? Imagine that 4 other MSc students want to continue your work, what research questions could be given to them to investigate? 

% MN: Then, change gears, and talk about "technology". Then mention improvements to the framework, additional micro services or other engineering and technological contributions. 

In terms of future work, there are some aspects that could be further improved and some of the claims further strengthened. First and foremost, in absence of the possibility to test the system with a broader and more general population during the period of this research work, it is necessary to conduct another experimental user study with strong focus on having a larger representative sample of the general population. 

Additionally, the conducted assessment methodology for evaluating the effectiveness of contextual clues with regards to effective disambiguation, a two-fold experimental user study should be conducted in the future. A two-fold experimental study with the control group being exposed with the complete sentence as contextual clue and the experimental group being exposed with short context clues extracted from the framework would improve the validity of the methodology used to evaluate the effectiveness of contextual clues. Being timely constraint, conducting such experimental trial is left out for future work.

Finally, with the implementation of an additional microservice which integrates semantic meta-data in the form of RDFa for unstructured HTML5 documents would increase the contributions of this work~\cite{rdfa}. Having RDFa data attached as semantic information for each entity present in a web document would potentially extend the meaning of the web documents making them semantically more machine-readable. Another interesting future direction would be incorporating this framework for gathering data for other complex \ac{nlp} problems such as \ac{wsd}.