main.tex

% !BIB program = bibtex
\documentclass{article}

\usepackage{graphicx, color}

\usepackage[a4paper,margin=2cm]{geometry}

% Here are my package imports:
\usepackage{caption}
\usepackage{subcaption}
\usepackage{hyperref}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{algorithm}
\usepackage{algorithmicx}
\usepackage{algpseudocode}
\usepackage{layouts}
\usepackage[table,xcdraw]{xcolor}
\usepackage{multirow}
\usepackage{graphics}
\usepackage{graphicx}
\usepackage{pdfpages}
\usepackage[toc,page]{appendix}
\usepackage{longtable}
\usepackage{blindtext}
\usepackage{afterpage}

% bib style
% \usepackage[maxbibnames=99]{biblatex}

\setlength{\parindent}{3em}
% \setlength{\parskip}{1em}
\newcommand{\red}[1]{{\color{red}{#1}}}

%%% Bibliography settings
\RequirePackage[backend=bibtex,style=nature,url=false,eprint=false,maxnames=1,isbn=false, issn=false, doi=false]{biblatex}
\addbibresource{fullrefs.bib}
\AtEveryBibitem{%
  \clearfield{note}%
}
\AtEveryBibitem{%
  \clearlist{language}%
}
\DeclareFieldFormat*{title}{#1}
\DeclareFieldFormat[book]{title}{#1}

%%% emoji sun
\usepackage{scalerel,xparse}
\NewDocumentCommand\emojisun{}{
    \includegraphics[scale=0.11]{data/images/sun.png}
}

%%% Abstract settings

\let\oldabstract\abstract
\let\oldendabstract\endabstract
\makeatletter
\renewenvironment{abstract}
{\renewenvironment{quotation}%
               {\list{}{\addtolength{\leftmargin}{3em} % change this value to add or remove length to the the default
                        \listparindent 1.5em%
                        \itemindent    \listparindent%
                        \rightmargin   \leftmargin%
                        \parsep        \z@ \@plus\p@}%
                \item\relax}%
               {\endlist}%
\oldabstract}
{\oldendabstract}
\makeatother


\begin{document}

\begin{titlepage}

\newcommand{\HRule}{\rule{\linewidth}{0.5mm}} % Defines a new command for the horizontal lines, change thickness here

\center % Center everything on the page

 

%----------------------------------------------------------------------------------------

%	HEADING SECTIONS

%----------------------------------------------------------------------------------------



\includegraphics[width=\linewidth]{data/images/uvaENG}\\[2.5cm]

\textsc{\Large MSc Artificial Intelligence}\\[0.2cm]

\textsc{\Large Master Thesis}\\[0.5cm] 



%----------------------------------------------------------------------------------------

%	TITLE SECTION

%----------------------------------------------------------------------------------------



\HRule \\[0.4cm]

{ \huge \bfseries Knowledge Generation \\ \Large Variational Bayes on Knowledge Graphs \\ [0.4cm] } % Title of your document

\HRule \\[0.5cm]

 

%----------------------------------------------------------------------------------------

%	AUTHOR SECTION

%----------------------------------------------------------------------------------------



by\\[0.2cm]

\textsc{\Large Florian Wolf}\\[0.2cm] %you name

{12393339}\\[1cm]





%----------------------------------------------------------------------------------------

%	DATE SECTION

%----------------------------------------------------------------------------------------



{\Large \today}\\[1cm] % Date, change the \today to a set date if you want to be precise



{48 Credits}\\ %
{April 2020 - January 2021}\\[1cm]
%{Period in which the research was carried out}\\[1cm]%



%----------------------------------------------------------------------------------------

%	COMMITTEE SECTION

%----------------------------------------------------------------------------------------

\begin{minipage}[t]{0.4\textwidth}

\begin{flushleft} \large

\emph{Supervisor:} \\

Dr Peter \textsc{Bloem} \\ Thiviyan \textsc{Thanapalasingam} \\ Chiara \textsc{Spruijt} % Supervisor's Name

\end{flushleft}

\end{minipage}

~

\begin{minipage}[t]{0.4\textwidth}

\begin{flushright} \large

\emph{Assessor:} \\

Dr Paul \textsc{Groth}\\

\end{flushright}

\end{minipage}\\[2cm]



%----------------------------------------------------------------------------------------

%	LOGO SECTION

%----------------------------------------------------------------------------------------



% \framebox{\rule{0pt}{2.5cm}\rule{2.5cm}{0pt}}\\[0.5cm]
% \begin{figure}[H]
%     \centering
%     \begin{minipage}{.5\textwidth}
%         \includegraphics[width=2.5cm, left]{data/images/uva.png} % Include a department/university logo - this will require the graphicx package
%     \end{minipage}
%     \right
\begin{minipage}{\textwidth}
    \centering
    \includegraphics[height=5cm]{data/images/deloitte_thesis_cover6.png} % Include a department/university logo - this will require the graphicx package
\end{minipage}
% \end{figure}

% \textsc{\large \red{institute name}}\\[1.0cm] % 

 

%----------------------------------------------------------------------------------------



\vfill % Fill the rest of the page with whitespace


\afterpage{\null\newpage}
\pagenumbering{gobble}

\end{titlepage}

\newpage
\pagenumbering{roman}

\begin{abstract}
% We generate Knowledge! \cite{kipf_contrastive_2020}

This thesis is proof of concept for the potential of Variational Auto-Encoder (VAE) on representation learning of real-world Knowledge Graphs (KG). Building on successful approaches in the field of molecular graphs, we experiment and evaluate the capabilities and limitations of our model, the Relational Graph Variational Auto-Encoder (RGVAE), characterized by its permutation invariant loss function. The impact of the modular hyperparameter choices, encoding though graph convolutions, graph matching and latent space prior, are analyzed and the added value is compared. The RGVAE is first evaluated on link prediction, a common experiment indicating its potential use for KG completion. The model is ranked by its ability to predict the correct entity for an incomplete triple. The mean reciprocal rank (MRR) scores on the two datasets FB15K-237 and WN18RR are compared between the RGVAE and the embedding-based model DistMult.
To isolate the impact of each module, a variational DistMult and a RGVAE without latent space prior constraint are implemented. We conclude, that neither convolutions nor permutation invariance alter the scoring. The results show that between different settings, the RGVAE with relaxed latent space, scores highest on both datasets, yet does not outperform the DistMult.

Graph VAEs on molecular data are able to generate unseen and valid molecules as a results of latent space interpolation. We investigate, if the RGVAE can yield similar results on relational KG data. The experiment is twofold, first the distance between the latent representation of two triples is linearly interpolated, then each latent dimension is explored in a $95$\% confidence interval of its Normal distribution. The interpolations reveal, how successful the RGVAE is at disentangling the latent space and assigning each latent dimension to data characterizing features. Both interpolation experiments show that the RGVAE learns to reconstruct the adjacency matrix but fails to disentangle and assign the right node and edge attributes. 

The assumption of an uninformative latent representation is confirmed in the last experiment of knowledge generation. For this experiment, we present a new validation method for generated triples from the FB15K-237 dataset. The relation type-constrains of generated triples are filtered and matched with entity types. The observed rate of valid generated triples is insignificantly higher than the random threshold. All generated and valid triples are unseen in both train and test set. A comparison between different latent space priors, using the $\delta$-VAE method, reveals insights in the behavior of the RGVAE's parameter distribution and indicates a decoder collapse. Finally we analyze the limiting factors of our approach compared to molecule generation and propose solutions for the decoder collapse and successful representation learning of multi-relational KGs. 
\end{abstract}
\newpage

\renewcommand{\abstractname}{Acknowledgements}
\begin{abstract}
%  Thanks Mum!
  \input{sections/thx.tex}
\end{abstract}

\newpage
\tableofcontents
\newpage
\listoffigures
\listoftables
\newpage
\pagenumbering{arabic}

\section{Introduction}
\input{sections/section1}

\section{Related Work}
\input{sections/section2}

\section{Background}
\input{sections/section3}

\section{Methods}
\input{sections/section4}
\label{sec:mthods}

\section{Experiments \& Results}
\input{sections/section5}

% \section{Results}
% \input{sections/section6}

\section{Discussion \& Future Work}
\input{sections/section7}
\label{sec:discus}

% \section*{Ideas}
% \input{ideas}

% \section*{Open Issues}
% \input{issues}

\newpage

\printbibliography
\newpage
\renewcommand{\appendixname}{Annex}
\renewcommand{\appendixtocname}{Annex}
\renewcommand{\appendixpagename}{Annex}

\begin{appendices}
    \input{sections/appendixC.tex}
    \newpage
    \input{sections/appendixB.tex}    
\end{appendices}

\end{document}

% \bibliographystyle{unsrt} % We choose the "plain" reference style
% \bibliography{fullrefs} % Entries are in the "refs.bib" file