% -*- coding:utf-8; mode:latex; -*- %

\documentclass{graphstudy}

\hypersetup{
  pdfauthor = {Moritz Klammler},
  pdftitle = {Aesthetic Value of Graph Layouts: Investigation of Statistical Syndromes for Automatic Quantification},
  pdfsubject = {Master's Thesis},
  pdfkeywords = {}
}

\addbibresource{literature.bib}
\makeindex
\input{abbreviations.tex}

\title{Aesthetic Value of Graph Layouts}
\subtitle{Investigation of Statistical Syndromes for Automatic Quantification}
\author{Moritz Klammler}
\date{September, 15 2017 -- March, 15 2018}

\newcommand*{\reviewerone}{Prof.~Dr.~Dorothea Wagner}
\newcommand*{\reviewertwo}{Prof.~Dr.~Peter Sanders}
\newcommand*{\advisorone}{Dr.~Tamara D.~Mchedlidze}
\newcommand*{\advisortwo}{Dr.~Alexey Pak}

\input{preamble.tex}

\begin{document}
\frontmatter
\maketitle
\cleardoublepage

\vspace*{\fill}
\begin{abstract}[Statement of Authorship]{english}
  I hereby declare that this document has been composed by myself and describes my own work, unless otherwise
  acknowledged in the text.
  \par\vspace{2.5cm}
  Karlsruhe, \today
\end{abstract}
\vspace{2cm}
\cleardoublepage

\begin{abstract}{english}
  Visualizing relational data as drawing of graphs is a technique in very wide-spread use across many fields and
  professions.  While many graph drawing algorithms have been proposed to automatically generate a supposedly
  high-quality picture from an abstract mathematical data structure, the graph drawing community is still searching for
  a way to quantify the aesthetic value of any given solution in a way that allows one to compare graph layouts created
  by different algorithms for the same graph (presumably to automatically choose the better one).  We believe that one
  promising path towards this goal could be enabled by combining data analysis techniques that have proven useful in
  other scientific disciplines that are dealing with large structures such as astronomy, crystallography or
  thermodynamics.  In this work we present an initial investigation of some statistical properties of graph layouts that
  we believe could provide viable syndromes for the aesthetic value.  As a proof of concept, we used machine learning
  techniques to train a neural network with the results of our data analysis and thereby built a model that is able to
  discriminate between better and worse layouts with an accuracy of \(95\,\%\).  A rudimentary evaluation of the model
  was performed and is presented.  This work primarily provides an infrastructure to enable further experimentation on
  the topic and will be made available to the public as Free Software at the place mentioned in the appendix of this
  document.
\end{abstract}
\clearpage

\begin{abstract}[Deutsche Zusammenfassung]{german}
  Relationale Zusammenhänge in Graphen zeichnerisch darzustellen ist eine sehr weit verbreitete Technik in vielen
  Disziplinen und Anwendungsfeldern.  Während zahlreiche Algorithmen vorgeschlagen wurden, die vollautomatisch
  vergleichsweise gute Layouts für eine gegebene Beschreibung eines Graphen generieren können, besteht bisweilen kein
  Konsens darüber, wie der ästhetische Wert einer bildlichen Darstellungen eines Graphen unabhängig von einem konkreten
  Layout-Algorithmus bewertet und verglichen werden kann.  Eine solche Anwendung wäre etwa wünschenswert, um unter den
  Ausgaben zweier Layout-Algorithmen automatisch das bessere Ergebnis auszuwählen.  Wir meinen, dass ein möglicher Weg
  hin zu einer automatischen Quantifizierung des gestalterischen Werts einer Graphzeichnung darin bestehen könnte,
  Methoden der Datenanalyse, wie sie sich in anderen Disziplinen, die sich mit großen Strukturen befassen -- etwa der
  Astronomie, Kristallographie oder Thermodynamik -- bewährt haben, miteinander zu verbinden, und auf Graphzeichnungen
  anzuwenden.  Mit der vorliegenden Arbeit stellen wir eine erste Untersuchung vor, in der wir einige Eigenschaften
  betrachten, von denen wir meinen, dass sie das Potential haben könnten, als zuverlässige Syndrome für den
  gestalterischen Gehalt einer Graphzeichnung zu fungieren.  Als eine erste Demonstration präsentieren wir ein
  neuronales Netz, das mithilfe der auf diese Weise gewonnenen und aufbereiteten Daten trainiert wurde, und in der Lage
  ist, bessere von schlechteren Layouts mit mehr als \(95\,\%\) Zuverlässigkeit unterscheiden zu können.  Ferner liefern
  wir ein rudimentäre Analyse der Eigenschaften dieses Diskriminators.  Der Schwerpunkt dieser Arbeit liegt jedoch auf
  der Schaffung einer Infrastruktur, die in Zukunft weitere Experimente einfach ermöglichen soll und der Öffentlichkeit
  als Freie Software an der im Anhang beschriebenen Stelle zugänglich gemacht werden soll.
\end{abstract}
\cleardoublepage

\tableofcontents
\listoffigures
\listoftheorems

\printacronyms[heading=addchap, exclude-classes={internal}, name={Abbreviations}]
\fakeacronyms

\include{notation}

\mainmatter

\chapter{Introduction}
\label{chap:intro}

\section{Motivation}
\label{sec:motivation}

\index{Lombardi!Mark}
Graphs as data structures and mathematical models of finite binary relations are an ubiquitous concept found in
practically every discipline of the arts in one form or another.  For human interaction, graphical representations of
vertices and edges as connected symbols is by far the most preferred form of presentation and much more approachable, --
especially by a non-technical audience -- than, say, a representation in matrix form.  A good drawing of a graph should
not only give a good understanding of the graph's essential structural properties, it should also provide an
aesthetically pleasing experience to the beholder.  The process of deriving a (usually two-dimensional) graphical
representation of an abstract mathematical definition of a graph is called \emph{graph drawing}.  While this surely can
-- and will~\cite{Lombardi2003} -- be understood as an artistic process, the vast majority of everyday applications must
rely on automated procedures for the quick and economic unattended creation of graph drawings with good -- or, at least,
acceptable -- quality.

There is no shortage on graph drawing algorithms\footnote{%
  We will continue to refer to (a subset of) them as \emph{layout algorithms} once we have defined the scope of this
  work.
}
today.  See for example \textcite{Tollis1999} or \textcite{Tamassia2013} to mention just two popular books on the topic.
However, these algorithms usually tend to \enquote{solve their own definition of the problem} which is of course fine
except that it may lead to the situation where one can have two (or more) drawings of the same graph which are each
\enquote{optimal} according to the definition of the algorithm that produced them and human intervention is required in
order to settle for a favorite.  At the time of this writing, no widely accepted and generally applicable automatically
computable quality function is available to the best of our knowledge.

We believe that a reliable way to compare the aesthetic value of graph layouts could be very useful for many
applications.  An obvious use case would be to run multiple layout algorithms -- or the same probabilistic algorithm
with different random seeds -- in parallel and then pick the result with the highest ranking.  Especially for algorithms
that produce good quality output in general but fail under certain pathological circumstances, this could prove very
helpful.  Other possible use cases could be in choosing or optimizing a domain-specific algorithm in a semi-automatic
fashion.

An inherent problem of the task outlined so far is of course that the human perception of beauty is not easily expressed
as a mathematical quantity.  It therefore occurs to us that a viable approach towards the problem might be to use
elementary concepts that have already proven useful in other disciplines such as astronomy or crystallography.  (Which,
maybe not purely coincidentally, study objects which many people perceive as aesthetically appealing.)  Combining as
many of these ideas as possible and analyzing the resulting data might give insights that would not be gained by a
purely analytical approach.

We have investigated several properties of graph layouts and found evidence for the hypothesis that they may be reliable
syndromes of a drawing's aesthetic value in so far as their statistical analysis lead us to a relatively small number
(\(58\)) of observables that allowed us to build a model to which we applied machine learning techniques in order to
create a discriminator that outputs a preference in favor of either of two layouts.  Within our rather limited
experiments, we achieved success rates exceeding \(95\,\%\) reproducibly.

Our methodology involved the automatic generation of a large corpus of labeled data without requiring human
intervention.  For this purpose, we devised probabilistic graph generators, built a small collection of known-good and
\emph{really bad} layout algorithms and crafted data augmentation techniques that allow us to produce even more labeled
data.  A number of characteristics was computed for all thusly obtained layouts and finally processed into a form that
could be used to train a Siamese neural network and used it as a black-box discriminator.

A major part of our contribution as we perceive it was the creation of an open collection of useful command-line tools
and a setup that allows for easy experimentation with different sources of graphs and layouts as well as with their
statistical syndromes.  Last but not least, we developed a web front-end for the convenient inspection of the data.
Hopefully, these facilities will be found useful by other researchers, too.  Instructions on how to obtain and use the
software are given in the appendix.

\index{reproduction!of experimental results}
We would also like to point out that all of our work is completely reproducible to the largest extent we were able to
ensure this.  We would like to encourage others to actually repeat running our experiments -- possibly with variations
to some of the parameters -- and compare the results.  Doing so on a \acs{posix} system is merely a matter of typing a
few simple commands (and then waiting very long).  As a matter of fact, even this document can be typeset automatically
from the sources we provide and will update itself with the current experimental results.

Before we're going to dwell further into the matter, we'll introduce a few definitions that will prove useful for the
remainder of this work and also become clear on the scope of this discussion.

\section{Preliminaries}
\label{sec:prelim}

\index{graph}
\index{graph!simple}
\index{node!synonym for vertex|see{vertex}}
\index{vertex!synonym for node|see{node}}
When speaking about a \emph{graph} in this work, we always mean a \emph{simple graph}, unless explicitly mentioned
otherwise.  A simple graph is an undirected graph without multiple edges or loops.  Furthermore, there are no labels,
weights or other attributes associated with the vertices or edges.  We will use the terms \emph{vertex} and \emph{node}
interchangeably.

In order to produce a two-dimensional graphical representation of a given graph, a necessary step is to assign
coordinates to each vertex.  This leads us to the following definition.

\index{vertex!layout}
\index{graph!layout|see{vertex layout}}
\index{layout!definition|see{vertex layout}}
\begin{definition}[Vertex Layout]
  Given a graph \(\GraphGVE\), a \emph{vertex layout} of \(\Graph\) is a function
  \(\Layout:V\to\Reals^2\) that assigns a point in two-dimensional Euclidean space to each vertex of \(\Graph\).
  \label{def:layout}
\end{definition}

There exist an unlimited number of vertex layouts for any given graph.  This is why we will be interested in comparing
the aesthetic value associated with each of them.  Before that, however, we will have to specify how a vertex layout is
presented in a form approachable by the human sense of aesthetics.

\label{def:drawing}
\index{graph!drawing}
Given a graph \(\GraphGVE\) together with a vertex layout \(\Layout\), a \emph{drawing} of \(\Layout(\GraphV)\) is a
two-dimensional graphical representation of \(\Graph\) obtained by drawing each vertex \(v\in\GraphV\) as a symbol (such
as a circle or square) at position \(\Layout(v)\) and connecting adjacent vertices with straight lines.

\Acl{fig}~\ref{fig:intro-limits} shows some examples of graph drawings that illustrate the limits of the understanding
of a graph drawing within this work's scope.  Because this simple understanding of a graph drawing is -- apart from
global parameters such as the choice of the symbol used for representing a vertex, its color and size as well as the
line width and color of lines representing edges -- already fully specified by a vertex layout, we will usually omit the
\enquote{vertex} part and only speak of \enquote{layouts}.  Furthermore, we hope that if a \enquote{reasonable} choice
is made for the remaining degrees of freedom mentioned, the particular choice won't have a significant influence on the
relative aesthetic value of the drawings associated with two vertex layouts as long as the same choices are made for
both.  That is, we assume that if \(\Layout_1(\Graph)\) looks better than \(\Layout_2(\Graph)\) when both are drawn with
a red pen and with circles for vertices, \(\Layout_1(\Graph)\) will still look better than \(\Layout_2(\Graph)\) even if
a green pen is used and vertices are drawn as diamonds.  Therefore, we will from now on exclusively speak about the
aesthetic value of layouts rather than drawings.

\begin{Figure}
  \begin{center}
    \begin{tabular}{c@{\qquad}c@{\qquad}c@{\qquad}c}
      \input{pics/intro-limits-okay.tex}&
      \input{pics/intro-limits-bends.tex}&
      \input{pics/intro-limits-kinks.tex}&
      \input{pics/intro-limits-labels.tex}\\[2ex]
      (a) & (b) & (c) & (d)\\
    \end{tabular}
  \end{center}
  \caption[Graphs drawings (not) considered within the scope of this work]{%
    The picture in (a) shows an example of a graph drawing as we consider it in this work.  Picture (b) is out of scope
    for this work because the edges have bends.  So is picture (c) because the edges have kinks and one node is a ground
    symbol.  Picture (d) is off limits too because the nodes are labeled and the graph has multiple edges in the form of
    the C=O carbonyl double-bond.
  }
  \label{fig:intro-limits}
\end{Figure}

Let us also introduce another definition (or notation) here that we will need later.

\index{graph!distance}
\index{dist1@$\dist(v_1,v_2)$|see{graph distance}}
\begin{definition}[Graph Distance]
  For a graph \(\GraphGVE\), the distance between two vertices \(\dist(v_i,v_j)\) for \(v_i,v_j\in\GraphV\) is defined
  as the length (number of edges) of the shortest path from \(v_i\) to \(v_j\) if such a path exists or else infinity.
  It is a non-negative integer or infinity.
  \label{def:graphdist}
\end{definition}

Graph distance is defined on the abstract mathematical concept of a graph.  We can define a related property for a
layout as follows.

\index{node!distance}
\index{vertex!distance|see{node distance}}
\index{dist2@$\dist_\Layout(v_1,v_2)$|see{node distance}}
\begin{definition}[Node Distance]
  Given a layout \(\Layout\) for a graph \(\GraphGVE\), the distance between vertices \(\dist_\Layout(v_i,v_j)\) in
  \(\Layout(\GraphV)\) for \(v_i,v_j\in\GraphV\) is defined as
  \(\dist_\Layout(v_i,v_j)=\vecnorm{\Layout(v_i)-\Layout(v_j)}\) and is a non-negative real number.
  \label{def:nodedist}
\end{definition}

Node distance is defined between any two vertices.  We can restrict this definition to pairs of adjacent vertices which
leads us to the following definition.

\index{edge!length}
\index{length@$\edgelen_\Layout(e)$|see{edge length}}
\begin{definition}[Edge Length]
  Given a layout \(\Layout\) for a graph \(\GraphGVE\), the length of an edge \(\edgelen_\Layout(e)\) in \(\Layout\) for
  \(e=\{v_i,v_j\}\in\GraphE\) is defined as \(\edgelen_\Layout(\{v_i,v_j\})=\dist_\Layout(v_i,v_j)\).
  \label{def:edgelen}
\end{definition}

It is now time to \enquote{define away} some uninteresting cases of layouts.  Therefore, let us first introduce the
following definition.

\index{layout!degenerated}
\begin{definition}[Degenerated Layout]
  A layout \(\Layout\) of a graph \(\GraphGVE\) is \emph{degenerated} if and only if
  \(\Layout(v)=\vecz\) for each \(v\in\GraphV\).
  \label{def:degenerated}
\end{definition}

For this work, we are only interested in layouts that are not degenerated.\footnote{%
  Our implementation also uses the term \enquote{degenerated layout} if the layout \(\Layout\) turns out to be only a
  partial function, that is, \(\Layout(v)=\pm\infty\) or \(\Layout(v)=\bot\) for one or more vertices \(v\) which is
  already prohibited by \acl{def}~\ref{def:layout} but unavoidable in practice when approximating real numbers with
  finite-precision floating-point numbers.  Some layout algorithms also tend to produce such values under pathological
  circumstances.
}

There are still several degrees of freedom in a vertex layout according to \acl{def}~\ref{def:layout} that we wish to
remove, namely the choice of the center and scale of the coordinate system.  It is clear that these have no influence on
the aesthetic value whatsoever.  Therefore, we introduce the following definition.

\index{layout!normalized}
\begin{definition}[Normalized Layout]
  A layout \(\Layout\) of a graph \(\GraphGVE\) with \(\GraphV=\{v_1,\ldots,v_n\}\) and
  \(\GraphE=\{e_1,\ldots,e_m\}\) is \emph{normalized} if and only if
  \begin{itemize}
  \item it is non-degenerate,
  \item the center of gravity \(\frac{1}{n}\sum_{i=1}^{n}\Layout(v_i)=\vecz\) is at the origin and
  \item the average edge length \(\frac{1}{m}\sum_{i=1}^{m}\edgelen_\Layout(e_i)\) or -- in the case that the graph has
    no edges at all -- the average node distance \(\frac{2}{n(n-1)}\sum_{1\leq{}i<j\leq{}n}\dist(v_i,v_j)\) has the
    arbitrary value of \(100\).
  \end{itemize}
  \label{def:normalized}
\end{definition}

Any given layout can easily be transformed into a normalized layout by subtracting from each vertex position the center
of gravity (translation) and applying the appropriate scaling in order to obtain an average edge length (or node
distance) of \(100\).

From now on, we shall assume that all layouts are normalized according to this definition.  Note that this precludes
layouts of graphs with only zero or one vertices, which is fine with us as there's not much of interest that could be
said about those layouts anyway.

A point could be made that \acl{def}~\ref{def:normalized} still leaves too many degrees of freedom.  For example, it
could be argued that layouts should also be normalized according to certain linear transformations such as rotation.  In
case of doubt, we prefer to be conservative, though, and only apply those normalizations that we are certain cannot have
any influence on the aesthetic value.  As for rotating a layout, there is even some evidence suggesting that it actually
affects the human perception of symmetry.\footnote{%
  See \textcite{Giannouli2013} and references therein reporting that humans are most likely to detect inflectional
  symmetry along the vertical axis.  Also see \acl{fig}~\ref{fig:vertical} as an illustration.
}
Rather than to nullify these effects by an overly restrictive definition, we wish to cover them in our analysis and
maintain the ability to evaluate their influence.  It is always safe to err on the side of more degrees of freedom, save
for more work to be done during data analysis.

\begin{Figure}
  \begin{center}
    \begin{tabular}{c@{\hspace{2cm}}c}
      \InputTikz[scale=0.5, rotate=0]{pics/intro-vertical.tikz}&
      \InputTikz[scale=0.5, rotate=20]{pics/intro-vertical.tikz}\\[2ex]
      (a) & (b)\\
    \end{tabular}
  \end{center}
  \caption[Example of the visual effect of rotating a layout.]{%
    Example of the visual effect of rotating a layout.  The pictures in (a) and (b) show the same layout of the same
    graph except that (b) is rotated by \(20\degree\).  It is expected that most people will perceive (a) as more
    aesthetically pleasing than (b).
  }
  \label{fig:vertical}
\end{Figure}

Now we have defined the subject of our study, we can introduce related algorithms.  Since most algorithms described in
this work are not fully-deterministic, we first need a generalized definition of an algorithm.

\index{algorithm!probabilistic}
\begin{definition}[Probabilistic Algorithm]
  A \emph{probabilistic algorithm} is an algorithm that formally receives an additional implicit input in the form of an
  unlimited stream of \(\BitZero\) and \(\BitOne\) bits with an independent probability of \(1/2\).
  \label{def:probalgo}
\end{definition}

In practice, the unlimited stream of random bits is substituted by a fixed number of more or less random bits that are
then used to \emph{seed} a so-called \emph{pseudo random generator} which itself is a deterministic algorithm.

Note that the concept of a probabilistic algorithm is a generalization rather than a refinement of a deterministic
algorithm.  A probabilistic algorithm \emph{may} make random decisions while a deterministic algorithm \emph{must not}.
Therefore, each deterministic algorithm is a probabilistic algorithm in the trivial sense that it makes use of zero
random bits.

Equipped with this definition, we can define algorithms that lay out graphs.

\index{algorithm!layout}
\index{layout!algorithm}
\begin{definition}[Layout Algorithm]
  A \emph{layout algorithm} is a probabilistic algorithm that receives as inputs a graph \(\Graph\) and outputs a
  (normalized) vertex layout \(\Layout\) for \(\Graph\).
  \label{def:layalgo}
\end{definition}

The definition is presented here because it brings together the concepts introduced so far.  We will revisit specific
layout algorithms in \acl{section}~\ref{sec:layouts} when we discuss our choice of layout algorithms for this work.

\section{Quantifying the Aesthetic Value of Graph Layouts}
\label{sec:quanti}

It would be foolhardy to attempt a formal definition of the human perception of beauty.  Let us therefore define our
expectations on an automatic quantifier instead.  The most ambitious dream might be to find a function \(f\) that can be
given a layout \(\Layout\) and will output a bounded value \(0\leq{}f(\Layout)\leq1\) which measures the layout's
aesthetic value in an absolute sense.  Comparing two layouts \(\Layout_\mathrm{A}\) and \(\Layout_\mathrm{B}\) would
then just be a matter of comparing the scalar values \(f(\Layout_\mathrm{A})\) and \(f(\Layout_\mathrm{B})\).  We
strongly believe that such a function does not exist and searching for it likely a waste of time, even if one should not
require that the function's image is confined to a closed interval.

Instead of looking for an absolute measure of aesthetic value, we'd be more than happy if we could find an algorithm
that computes a relative order of the aesthetic value.  But even there the question remains whether the human perception
of beauty follows properties that are mathematically required for an ordering relation.  Most notably, transitivity.

The scope we're going to set out for this work is to find an approximation of a partial order.  A \emph{partial order}
is a binary relation that is reflexive, antisymmetric and transitive.  While an ordering relation is defined to be only
either true or false, we consider it more practical and useful to have a function that outputs a \emph{certainty} which
allows for a continuous approximation.  We can always take the signum of such a function if we want a binary decision.

Formally, if \(\Graph\) is a graph and \(\Layout_\mathrm{A}\) and \(\Layout_\mathrm{B}\) are two layouts for \(\Graph\)
then we want a function \(f\) such that \(-1\leq{}f(\Layout_\mathrm{A},\Layout_\mathrm{B})\leq+1\) with the
interpretation that \(f(\Layout_\mathrm{A},\Layout_\mathrm{B})<0\) indicates a preference in favor of
\(\Layout_\mathrm{A}\) while \(f(\Layout_\mathrm{A},\Layout_\mathrm{B})>0\) indicates a preference in favor of
\(\Layout_\mathrm{B}\) and \(f(\Layout_\mathrm{A},\Layout_\mathrm{B})\approx0\) means that the answer is unclear.  We
shall use this interpretation consistently throughout this work.  It is obvious that any linear mapping to another
interval is trivial and can be done freely at will and convenience.

\section{Overview of our Contribution}
\label{sec:overview}

The remainder of this work is organized as follows.
\begin{itemize}
\item\Acl{chap}~\ref{chap:intro} motivated the work, defined its scope and introduced some basic concepts and provided a
  short overview.
\item\Acl{chap}~\ref{chap:relwork} gives a quick overview over existing research on the topic.
\item\Acl{chap}~\ref{chap:method} discusses our methods in more detail.
\item\Acl{chap}~\ref{chap:syndromes} introduces the properties we have investigated and provides reasoning why we
  believe that they can be reliable syndromes of aesthetic value of a graph layout.  At this point, a \emph{property} is
  a multiset of scalar values with an unbounded number of elements.
\item\Acl{chap}~\ref{chap:datgen} provides a detailed discussion how we obtained our test data (graphs and layouts).
\item\Acl{chap}~\ref{chap:dataug} explains in detail how we applied data augmentation techniques to our data set in
  order to gain even more samples.
\item\Acl{chap}~\ref{chap:featex} is devoted to the exfiltration of a small fixed-sized feature vector from the
  collection of properties and mentions some of the problems we encountered along the way.
\item\Acl{chap}~\ref{chap:dismod} describes the structure of our discriminator model and why it was chosen that way.  It
  also addresses the issues of training and testing the model.
\item\Acl{chap}~\ref{chap:eval} presents the experimental results of our (limited) empirical evaluation.
\item\Acl{chap}~\ref{chap:conclusion} draws some final conclusions and mentions a lot of ideas that we were unable to
  peruse within the scope of this work but consider worthwhile nonetheless.
\item The appendix provides information how to obtain, compile and use our software.  Either for the purpose of
  verifying our experiments or conducting new ones.
\end{itemize}

\chapter{Related Work}
\label{chap:relwork}

\section{Quality Measures}

\textcite{Purchase2002} was probably the first to suggest seven simple measures for the aesthetic value of a layout that
include factors like the number of crossings or orthogonal angles.

\textcite{Koren2008} proposed a modification of the stress function originally introduced by \textcite{Kamada1989} and
coined the term \enquote{binary stress} for it.

\textcite{Huang2013} addressed a very similar problem than we do, namely that \enquote{many automatic graph drawing
  algorithms implement only one or two aesthetic criteria since most aesthetics conflict with each other} and conclude
that their \enquote{study indicates that aesthetics should not be considered separately} and rather try
\enquote{improving multiple aesthetics at the same time, even to small extents}.  The measures they consider are those
suggested by \citeauthor{Purchase2002}.  Their major contribution is a force-directed layout algorithm that has an
energy function crafted specifically to give consideration to all metrics they mention.

\textcite{Klapaukh2014} has suggested a more complicated measure based on the identification of axes of symmetry.

\textcite{Huang2016} in \citeyear{Huang2016} suggested a very simple aesthetic relation that can be applied to any given
set of layouts of the same graph by combining a few scalar properties -- for which they suggest basically those proposed
by \citeauthor{Purchase2002} -- via subtracting the mean and dividing by the standard deviation of this quantity for all
layouts in the set and then forming the sum.  The resulting number is only meaningful with respect to the other layouts
in the same set.

\section{Machine Learning Techniques}

\textcite{Masui1994} applied machine learning techniques to the field of graph drawing as early as \citeyear{Masui1994}.
They designed an interactive system in which a user first labels a set of layouts and this information is then used to
parameterize a model that will in turn employ an evolutionary algorithm in order to improve an existing or derive a new
layout.  Their paper does not mention how large their largest graphs were but it seems like they consisted only of very
few (probably on the order of a dozen) vertices.  Otherwise, a training set of \(N=22\) as they used it could barely be
sufficient.  Their model is formulated using several high-level constraints which incorporate some assumptions about
good layouts that are most likely universally true.  The paper mentioned that the approach was computationally
expensive.

More than 20 Years later, a \citeyear{Raissa2015} review article by \textcite{Raissa2015} records that
\enquote{Surprisingly, only a few pieces of research can be found about this subject.}  This paper cites a few other
references of works where direct user interaction was used in order to adjust a parameterized fitness function in
real-time.  The review also mentions other approaches towards unattended application of machine learning, although in a
different sense than we use it, employing neural networks to approximate hard optimization problems~\cite{Hopfield1985}
and using these results again to compute a layout via an explicit algorithm (in this case minimization of edge crossings
via planar embeddings~\cite{Cimikowski1996}).

\textcite{Meyer1998} presented a very interesting approach in which they did not \enquote{use an external network
  structure \enquote{to learn the graph}, instead the graph itself will be turned into a learning network.}  By
constructing a neural network with the same topology as the graph to be laid out, they achieve that the neuron weights
derived during training can be used to derive vertex positions.  Given that evaluating a neural network is very fast but
training it takes time, this might not be as cool as it looks at first sight.

\textcite{Bach2012} designed an \enquote{interactive random graph generation with evolutionary algorithms} that looks
like it could have been useful for us too, had we discovered it earlier.

A very recent work of \textcite{Kwon2018} is closely related to ours even if quite different.  Here the authors used
machine learning techniques not to -- as we do -- process information about a given layout, but rather \emph{infer} what
the properties of the layout \emph{would be} if it were to be computed using a given algorithm.  Their main interest
seems to lie in the number of edge crossings.  This approach allows the authors to consider many more layouts in the
same time than it would be possible if the layouts actually were to be computed.  They can then use this information to
decide what layouts they actually want to compute.

Apart from this last paper, we were not able to find any relevant publications on the topic that are not already
mentioned in the review article by \citeauthor*{Raissa2015} during our, admittedly superficial, literature survey.

\section{User Studies}

\textcite{Purchase1997} conducted a user study that evaluated people's ability to \emph{understand} a graph -- which is
not necessarily the same as liking it -- and concluded that \enquote{reducing the number of edge crosses is by far the
  most important aesthetic}.  Other studies~\cite{Ware2002} have come to different conclusions.

\textcite{Welch2017} conducted a study to compare the metrics suggested by \citeauthor*{Purchase2002},
\citeauthor*{Klapaukh2014} and \citeauthor*{Koren2008} and concluded that the last one (binary stress) is the most
realistic measure available today.

\chapter{Methodology}
\label{chap:method}

We wish to provide evidence for the correlation of some statistical properties of graph layouts and their aesthetic
value.  These syndromes should be easy to compute and be as generic as possible, such as to avoid encoding implicit
assumptions into the feature vector as much as possible and instead derive syndromes for aesthetic value from first
principles that are applicable to a large variety of graphs and layouts.

To this end, our intention is to collect a large corpus of graphs and several \enquote{labeled} pairs of layouts for
each of them.  A labeled pair of layouts for a graph \(\Graph\) is a triple
\((\Layout_\mathrm{L},\Layout_\mathrm{R},t)\) where the number \(-1\leq{t}\leq+1\) indicates which layout is the better
one and to what degree so.  If \(t=-1\) we are most certain that \(\Layout_\mathrm{L}\) is better, if \(t=+1\) we are
most certain that \(\Layout_\mathrm{R}\) is better and if \(t\approx\pm0\) we have no preference.

For each layout, we compute \emph{properties} that we believe are viable syndromes of the layout's aesthetic value
(\acs{cf}~\acs{chap}~\ref{chap:syndromes} for a detailed discussion of these).  All properties are unordered
collections of events (such as the sequence of edge lengths).  In order to analyze their distributions, a few
statistical parameters (such as arithmetic mean, root mean squared and, in particular, entropy) are computed.  This
gives us a feature vector of fixed small size regardless of the graph's size.  The feature extraction process is
explained in detail in \acl{chap}~\ref{chap:featex}.

Armed with this data, we seek to find interesting correlations between feature vectors and expected layout quality.
Since no single investigated feature seemed to be usable as a reliable metric in isolation, we decided to apply machine
learning techniques in order to build a discriminator model that would receive a pair of layouts as inputs and outputs a
prediction \(p\) for the value \(t\).  We count a prediction as a success if \(\sign(p)=\sign(t)\).

Ideally, we would independently generate many layouts for each graph and then query an eternal source of truth in order
to label each pairwise combination.  In the absence of such a source, human labeling might be applied, for example by
conducting a large-scale user study where random pairs of layouts are shown to participants which are then asked to
express their preference for one layout or the other.  Finally, \(p\) could be estimated by subtracting the votes in
favor of the first layout from those in favor of the second and dividing the resulting number by the total number of
votes.  Unfortunately, such a study was beyond the resources and time-frame available for this work.

\index{layout!proper}
\index{layout!garbage}
Hence, we used a technique called \emph{data augmentation} to obtain labeled layout pairs from a priori knowledge about
the way a layout was created.  Unfortunately, we don't know to rate pairs of layouts if both were produced by a
state-of-the-art layout algorithm or one is a native layout (see below).  The only thing we know is that such layouts
are probably \enquote{good} to some extent.  We shall refer to those as \emph{proper layouts} from now on.

As a first step towards labeled pairs, we generate additional layouts in the dumbest conceivable manner, basically
laying out nodes randomly (the exact procedure is explained and discussed in \aclp*{section}~\ref{sec:random} and
\ref{sec:phantom}).  Those layouts we will refer to as \emph{garbage layouts}.  We assume that any combination of a
proper and a garbage layout is \(100\,\%\) in favor of the proper layout.  Two garbage layouts would be considered of
equal quality but we found it sufficient to compute only a single garbage layout per graph anyway.

The introduction of garbage layouts provides a convenient \enquote{zero point} of an example of the worst possible
layouts.  However, it would not be very representative to compare only extremes.  Therefore, let us introduce the
following concept.

\index{layout!transformation!unary}
\begin{definition}[Unary Layout Transformation]
  A \emph{unary layout transformation} is a probabilistic algorithm that receives as inputs a graph \(\Graph\) together
  with a layout \(\Layout\) of \(\Graph\) and a parameter \(0\leq{}r\leq{}1\) and outputs a Layout \(\Layout'\) for
  \(\Graph\).  The interpretation of the parameter \(r\) is up to the algorithm, however \(r=0\) shall imply
  \(\Layout'=\Layout\).
  \label{def:unitrans}
\end{definition}

\index{layout!worsening}
\index{layout!parent}
For \emph{layout worsening}, we define a number of unary layout transformations that distort a given layout to a
configurable degree (controlled by the parameter \(r\)).  Applying such transformation to a proper layout \(\Layout\)
(referred to as the \emph{parent layout}) with parameters \(r\in\{r_1,\ldots,r_n\}\) yields \((n+1)^2\) triples
\((\Layout_i,\Layout_j,t_{ij})\) for \(i,j\in\{0,\ldots,n\}\) using \(\Layout_0=\Layout\).  For these we assume that
\(\sign(t_{ij})=\sign(r_i - r_j)\).  Applying worsening to a garbage layout is not assumed to have any effect on the
quality of the layout in either direction.

A second data augmentation strategy involves computing new layouts with an anticipated quality from two existing
layouts.  Let us provide the following definition first.

\index{layout!transformation!binary}
\begin{definition}[Binary Layout Transformation]
  A \emph{binary layout transformation} is a probabilistic algorithm that receives as inputs a graph \(\Graph\) together
  with an ordered tuple of two layouts \(\Layout_\mathrm{A}\) and \(\Layout_\mathrm{B}\) of \(\Graph\) and a parameter
  \(0\leq{}r\leq{}1\) and outputs a Layout \(\Layout'\) for \(\Graph\).  The interpretation of the parameter \(r\) is up
  to the algorithm, however \(r=0\) shall imply \(\Layout'=\Layout_\mathrm{A}\) and \(r=1\) shall imply
  \(\Layout'=\Layout_\mathrm{B}\).
  \label{def:bitrans}
\end{definition}

\index{layout!interpolation}
\index{layout!parents}
For \emph{layout interpolation}, we define binary layout transformations that interpolate between two layouts.  The
similarity of the interpolated layout to either layout is controlled by the parameter \(r\).  Applying such
interpolation to a pair of layouts \(\Layout_\mathrm{A}\) and \(\Layout_\mathrm{B}\) (referred to as the \emph{parent
  layouts}) with parameters \(r\in\{r_1,\ldots,r_n\}\) yields \((n+2)^2\) triples \((\Layout_i,\Layout_j,t_{ij})\) for
\(i,j\in\{0,\ldots,n+1\}\) using \(\Layout_0=\Layout_\mathrm{A}\) and \(\Layout_{n+1}=\Layout_\mathrm{B}\).  For these
we assume that \(t_{ij}\approx(r_j - r_i)t_\mathrm{AB}\) where \(t_\mathrm{AB}\) is the estimated result of comparing
the parent layouts.

This is not of much help if we don't know how to rank \(\Layout_\mathrm{A}\) and \(\Layout_\mathrm{B}\) in the first
place (\acs{ie}~\(t_\mathrm{AB}=\bot\)).  However, if one layout is a proper layout and the other is a garbage layout,
interpolation provides us with a fine-grained series of intermediate steps in addition to the blunt comparison of
extremes.

The design space for the approach outlined so far is vast.  There are countless ways to generate graphs and layouts as
well as unary and binary transformations that could be applied to them for the sake of worsening and interpolation.  Not
to mention the number of properties that could be computed for populating the feature vector.  In practice, graphs often
have a certain structure specific to the application domain.  For example, social graphs will possess a different
structure than network graphs in computer communication networks.  It would be overly ambitious to hope that our
investigation will provide optimal answers for each of these domains.  Rather than aiming for this unrealistic and
unverifiable goal, we take care to keep our setup flexible enough to make it easy to adapt to real-world applications if
domain-specific knowledge is available.  The classes of graphs we have investigated are by no means comprehensive but
merely an example data collection without any specific practical application.  By designing our framework as a
collaborative ensemble of small independent tools, it is easy to add or remove graph generators, layouts, layout
transformations and feature extractors.  The remainder of this work describes the implementation of these components
used for our study.  Since the experiment is fully automated, replicating it with different components is simply a
matter of adding or removing individual components and re-running the experiment.  The individual tools communicate with
each other through simple command-line interfaces which does not tie them to a particular technology.  An overview of
the data-flow pipeline is given in \acl{fig}~\ref{fig:pipeline} and will be described briefly in the remainder of this
section.

\begin{Figure}
  \begin{center}
    \input{pics/methodology-pipeline.tex}
  \end{center}
  \caption[Overview of the pipeline used]{%
    Overview of the pipeline used.  Circles represent data assets while rectangular boxes represent collections of
    tools.  Arrows indicate data flow (as managed by the driver script).  File names in monospace font refer to the
    configuration files that can be used to control the selection of tools to be used without modifying the driver
    script.
  }
  \label{fig:pipeline}
\end{Figure}

Graph generators, layout algorithms and transformations and feature extractors are implemented as small independent
programs written in {\CXX} for performance using the \emph{\acf{ogdf}}~\cite{OGDF} as a library for graph data
structures and algorithms, in particular, layout algorithms.  The neural network is built with \Keras~\cite{Keras} using
the \TensorFlow~\cite{TensorFlow} library as back-end.  The individual components are connected together and
orchestrated by a driver script written in the {\Python} programming language.  The driver has some knowledge of each
tool, such as the command-line options it has to be passed.  This means that addition of a new tools requires a small
modification to the driver script, which is as simple as adding a new enumeration constant and specifying an executable
name and any additional command-line parameters, if any.\footnote{%
  It is usually not necessary to use specific command-line arguments as the tools ought to adhere to some common
  conventions, such as to accept the files to be read as final arguments and allow specifying the output file via the
  \code{--output=\emph{FILE}} option.  However, the additional flexibility provided by supporting tool-specific
  command-line parameters allows implementing several tools in a single executable.  For example, the \enum{MOSAIC1} and
  \enum{MOSAIC2} generator (\acs{cf}~\acs{section}~\ref{sec:mosaic}) are implemented in the same executable and selected
  via passing or not passing the \code{--symmetric} option.
}
Data is stored in and retrieved from a relational database (provided via \SQLite) by the driver script.  Graphs,
layouts and data series are stored as individual text files on the file system and referenced from the database.  For
graphs and layouts the \emph{\acf{graphml}}~\cite{GraphML,GraphMLReport} format\footnote{%
  Which is an \acs{nxml} (text).  Our tools additionally support transparent compression in order to preserve disk
  space.
}
is used.  Data inspection and queries to the neural network are provided via a web interface that is written in a
mixture of {\Python}, \acs{nxslt} {\JavaScript} and \acs{html}.

Besides using the driver script and web interface, the individual tools we've written may also be used in isolation;
either alone or as \acs{io} filters for quick experimentation.  For example, the shell commands
\begin{lstlisting}
$ mosaic --symmetric --nodes=1000 --output=sample.xml.bz2
$ picture --output=sample.svg sample.xml.bz2
\end{lstlisting}
create a random symmetric \enquote{mosaic} graph (\acs{cf}~\acs{section}~\ref{sec:mosaic}) with
approximately 1\,000 nodes and save it as \ac{graphml} file \verb`sample.xml.bz2` with \verb`bzip2` (Burrows-Wheeler)
compression applied.  The file is then read again on the second line and a graphical rendition of the layout is saved as
file \verb`sample.svg`.

The shell pipeline
\begin{lstlisting}
$ wget -q -O - ftp://math.nist.gov/.../bcspwr01.mtx.gz               \
    | import --format=matrix-market --simplify --meta=2 STDIO:gzip   \
    | force --algorithm=fmmm                                         \
    | edge-length --kernel=boxed --output=histogram.txt
{ "nodes": 42, "edges": 85, "native": false, ... }
\end{lstlisting}
downloads\footnote{%
  The \acs{url} had to be shortened in the snippet to make it fit on a line:
  \url{ftp://math.nist.gov/pub/MatrixMarket2/Harwell-Boeing/bcspwr/bcspwr01.mtx.gz}
  (tested on 2018-02-22)
}
(using the standard command-line utility \verb`wget`) a graph from \acs{nist}'s \enquote{Matrix
  Market}~\cite{MatrixMarket} as \verb`gzip` (Lempel-Ziv) compressed file, simplifies\footnote{%
  This operation makes edges undirected and deletes loops and fuses multiple edges into one.
}
the graph and converts it to the preferred \ac{graphml} format, then computes a force-directed layout for the graph and
finally analyzes the distribution of edge lengths in that layout, saving a histogram as text file \verb`histogram.txt`.
The command given the \verb`--meta=2` option will print additional information to standard error output (selected by the
\acs{posix} file descriptor 2) in \acs{json} format which is partially shown in the above snippet after the command
prompt.  (It cannot be printed to standard output which is already used for the pipeline or it would be invisible and
clobber the graph data.)  The histogram file could be plotted using a tool like \verb`gnuplot`.  The last program could
(and probably should) also be instructed to output additional information like the mean or entropy in \acs{json} format
using the \verb`--meta` option again which was omitted in the example for the sake of brevity.

We hope that this collection of command-line tools will prove useful to other researchers as well.  A more detailed
documentation of their usage is given in the appendix.  All command-line tools also support the \verb`--help` option to
quickly print a short usage summary.

\chapter{Statistical Syndromes of Graph Layouts}
\label{chap:syndromes}

\index{symptom!of aesthetic value}
\index{syndrome!of aesthetic value}
Aesthetic value is not a mathematical or physical quantity that can be measured directly.  Hence, we're out on a survey
for things that \emph{can} be measured quantitatively and ideally tell us something about the aesthetic value of a graph
layout.  Unlike with previous approaches to the topic, our investigations are not primarily driven by the desire to find
scalar quantities that are supposed to be mini- or maximized in order to obtain good layouts.  Rather, we look for
\emph{symptoms} of aesthetic value.  A \emph{syndrome} is a collection of symptoms that might not all be present at the
same time in the same intensity.  By including as many syndromes as possible in our toolbox, we hope to get a more
comprehensive view of the problem and therefore, ideally, will eventually become able to make decisions that are general
and \enquote{robust} in the sense that they are not subjected to \latintext{a priori} assumptions that might not hold in
all cases.

\index{event}
In this chapter, we will introduce several \emph{properties} of graph layouts that we investigated and believe to be
viable candidates for reliable syndromes of aesthetic value.  Mathematically, each property is a multiset of scalar
values that can be computed for a graph layout.  We will refer to the elements of those multisets as \emph{events}.

We will introduce the mathematical definitions of the properties in the present chapter and discuss how they can be
computed efficiently.  Furthermore, we will present examples of different graphs and layouts and demonstrate how said
properties behave for them.

In order to visualize properties, we will use density plots which show the event density as a function of the event
magnitude.  We will omit the scale and units of the ordinate in our plots because the concrete numeric values are not
really important for a general understanding.  The plots were obtained by computing sliding averages using a Gaussian
kernel.  (The width of which will be indicated in the diagrams.)  We will discuss this process further in
\acl{section}~\ref{sec:gaussian}.  For the purpose of the discussion in this chapter, it is not important how the
diagrams are obtained but rather what they visualize.  We will revisit the acquisition of meaningful diagrams in
\acl{chap}~\ref{chap:featex} when we shall discuss how the properties presented in this chapter can be turned into
inputs for a machine learning algorithm.

\section{Principal Components (\enum{PRINCOMP1ST} and \enum{PRINCOMP2ND})}
\label{sec:princomp}
\index{properties!PRINCOMP1ST@\enum{PRINCOMP1ST}}
\index{properties!PRINCOMP2ND@\enum{PRINCOMP2ND}}
\index{principal component analysis}
\index{PCA|see{principal component analysis}}

The first property we want to discuss is the distribution of vertices on the drawing pane.  Most likely, this is an
elementary property of a vertex layout.  It is to be expected that the coordinate distribution of a regular drawing
exposes distinctive peaks at certain intervals while a random arrangement of nodes features a more or less smooth
distribution.

In order to investigate the coordinate distribution, we perform a \emph{\ac{pca}}.  See \textcite{Abdi2010} for a brief
or \textcite{Jolliffe2002} for a thorough introduction to the topic.

\begin{definition}[Principal Component]
  Let \(\Vec{X}\subset\Reals^n\) for \(n\in\IntsN\) be a finite point cloud (multiset) with center
  \begin{equation}
    \bar{\vec{x}} = \frac{1}{\card{\Vec{X}}} \sum_{\vec{x}\in\Vec{X}} \vec{x}
    \enspace.
  \end{equation}
  The unit vector
  \begin{equation}
    \label{eq:princomp-def}
    \vec{p}^{(1)} =
    \argmax_{\vec{v}\in\Reals^n\text{~\acs{st}~}\vecnorm{\vec{v}}=1}
    \left\{ \sum_{\vec{x}\in\Vec{X}} \braket{\vec{v}}{\vec{x}-\bar{\vec{x}}}^2 \right\}
  \end{equation}
  is the \emph{first principal component} of \(\Vec{X}\).  Let \(\vec{p}^{(1)},\ldots,\vec{p}^{(k)}\) for \(k<n\) be the
  first \(k\) principal components of \(\Vec{X}\).  Then the next principal component is the unit vector
  \(\vec{p}^{(k+1)}\in\Reals^n\) satisfying \acl{eq}~\ref{eq:princomp-def} with the additional requirement that it is
  linear independent of \(\vec{p}^{(1)},\ldots,\vec{p}^{(k)}\).
  \label{def:princomp}
\end{definition}

The principal components form an orthonormal basis with the property that the moment of inertia is largest along the
first axis, second largest along the second axes and so forth.  \Ac{pca} is therefore usually used in order to reduce
the dimensionality of empirical data.  In our case, however, we have only two-dimensional data to begin with so we
merely use \ac{pca} to find the direction of the principal components which might tell us something about the layout.
In particular, we are interested in the \emph{distribution} along the principal components.  \Ac{pca} is often employed
under the assumption that the data follows a normal distribution.  In our case, however, we know that this will most
certainly not be the case\footnote{%
  Unless the layout was generated by \enum{RANDOM\_NORMAL} (\acs{section}~\ref{sec:random}) of course.
}
so we hope to gain valuable insights from analyzing the distribution.

\index{axis!principal}
\index{axis!major}
\index{axis!minor}
Because of the geometric application, we will often speak of principal \emph{axes} instead of principal
\emph{components} where we will use the terms \emph{major} and \emph{minor} axis when referring to the first and second
principal component respectively.  For visualization, we multiply the unit vectors with the standard deviation of the
data set along their direction.

The \enum{PRINCOMP1ST} property analyzes the distribution of vertex coordinates along the major and the
\enum{PRINCOMP2ND} property along the minor principal axis.  More precisely, given a layout \(\Layout\) for graph
\(\GraphGVE\), where \(\vec{p}^{(1)}\) and \(\vec{p}^{(2)}\) are the first and second principal component of
\(\Layout(\GraphV)\) respectively, the two properties consider the following multisets.
\begin{align}
  &\menum{PRINCOMP1ST} = \multiset{\braket{\vec{p}^{(1)}}{\Layout(v)}\suchthat{v}\in\GraphV}\\
  &\menum{PRINCOMP2ND} = \multiset{\braket{\vec{p}^{(2)}}{\Layout(v)}\suchthat{v}\in\GraphV}
\end{align}
The significance of these distributions is best understood by an example which can be found in
\acl{fig}~\ref{fig:princomp}.

\begin{Figure}
  \begin{center}
    \begin{tabular}{c@{\qquad}c@{\qquad}c}
      \InputTikzGraph*{0.25\textwidth}{pics/demograph-princomp-a.tikz}&
      \InputTikzGraph*{0.25\textwidth}{pics/demograph-princomp-b.tikz}&
      \InputTikzGraph*{0.25\textwidth}{pics/demograph-princomp-c.tikz}\\[1ex]
      \InputLuatikzPlot[width=0.25\textwidth]{pics/princomp1st-a.pgf}&
      \InputLuatikzPlot[width=0.25\textwidth]{pics/princomp1st-b.pgf}&
      \InputLuatikzPlot[width=0.25\textwidth]{pics/princomp1st-c.pgf}\\
      \InputLuatikzPlot[width=0.25\textwidth]{pics/princomp2nd-a.pgf}&
      \InputLuatikzPlot[width=0.25\textwidth]{pics/princomp2nd-b.pgf}&
      \InputLuatikzPlot[width=0.25\textwidth]{pics/princomp2nd-c.pgf}\\
    \end{tabular}
  \end{center}
  \caption[\enum{PRINCOMP1ST} and \enum{PRINCOMP2ND} examples]{%
    Distributions of node density analyzed via \enum{PRINCOMP1ST} and \enum{PRINCOMP2ND} for three layouts illustrated.
    The top row shows the graph drawing with superimposed principal components (multiplied with the respective standard
    deviation).  The second and third row show the density function for the \enum{PRINCOMP1ST} and \enum{PRINCOMP2ND}
    property respectively.  The left column corresponds to a perfectly regular grid.  Its distribution function is a
    comb of very sharp peaks at unit distance.  In theory, the spikes would be Dirac delta functions but the diagram was
    created by moving a Gaussian filter over the data so the peak width is finite.  The middle column shows a layout for
    the same graph but this time less regular.  The distribution still shows peaks roughly at the average node distances
    but they are broader (even though the same filter width was used for all diagrams) and less pronounced.  The right
    column shows a force-directed layout of a power-network pattern taken from the \enum{BCSPWR} set of the
    Harwell-Boeing Collection in \acs*{nist}'s Matrix Market~\cite{MatrixMarket}.  One can identify the two larger
    clusters along the major axis and a relatively compact non-symmetric distribution along the minor axis.  Especially
    towards the ends of the spectrum, small peaks at fairly regular intervals corresponding to the nodes in the
    \enquote{strand} that extends to the right are visible.
  }
  \label{fig:princomp}
\end{Figure}

In order to actually compute the principal components, we use power iteration and then use Gram-Schmidt
orthonormalization\footfullcite{MathWorldGramSchmidt} in order to find the second component.  Repeated application of
the Gram-Schmidt process is numerically unstable~\cite{Golub1996} but since we only apply it at most once, this should
not be a problem for us.  The algorithm -- which is not particularly clever but was easiest to implement -- is shown in
\acl{algo}~\ref{algo:princomp}.  The computational cost for this property is basically linear in the number of nodes.

% TODO: Find a credible source for PCA via power iteration and Gram-Schmidt.

\begin{Algorithm}
  \begin{extraleading}
    \KwIn{Point cloud \(\Vec{X}=\{\vec{x}_1,\ldots,\vec{x}_m\}\subset\Reals^n\) and integer \(k\leq{}n\).}
    \KwOut{The first \(k\) principal components \(\vec{p}^{(1)},\ldots,\vec{p}^{(k)}\).}
    \KwConstant{Tolerance \(0<\delta\ll1\).}
    \Routine{%
      \(\bar{\vec{x}}\gets\frac{1}{m}\sum_{j=1}^{m}\vec{x}_j\)\;
      \(\Vec{X}^{(0)}\gets\{\vec{x}-\bar{\vec{x}}\suchthat\vec{x}\in\Vec{X}\}\)\;
      \For{\(i\gets1\) {\KwTo} \(k\)}{
        \comment{Power Iteration:}
        Choose random unit vector \(\vec{r}_0\in\Reals^n\)\;
        \For{\(l\gets1\) {\KwTo} some reasonable limit}{
          \(\vec{r}_l\gets\frac{1}{m}\sum_{j=1}^{m}\vec{x}\braket{\vec{x}}{\vec{r}_{l-1}}\)
              normalized such that \(\vecnorm{\vec{r}_l}=1\)\;
          \If{\(\vecnorm{\vec{r}_l-\vec{r}_{l-1}}\leq\delta\)}{
            \(\vec{p}^{(i)}\gets\vec{r}_l\)\;
            \KwBreak\;
          }
        }
        \comment{Gram-Schmidt:}
        \(\Vec{X}^{(i)}\gets\{\vec{x}-\vec{p}^{(i)}\braket{\vec{p}^{(i)}}{\vec{x}}\suchthat\vec{x}\in\Vec{X}^{(i-1)}\}\)\;%
      }
    }
  \end{extraleading}
  \caption[Principal Component Analysis Using Power Iteration and Gram-Schmidt Orthonormalization]{%
    Principal component analysis using power iteration and Gram-Schmidt orthonormalization.  We used
    \(\delta=\sqrt{\epsilon_\mathrm{float}}\) which turned out to work very well.
  }
  \label{algo:princomp}
\end{Algorithm}

\section{Angles Between Incident Edges (\enum{ANGULAR})}
\label{sec:angular}
\index{properties!ANGULAR@\enum{ANGULAR}}

Another simple property is the distribution of angles between the edges incident to a vertex.
\Acl{fig}~\ref{fig:angular-explanation} illustrates how these angles are defined.  Let \(\Layout\) be a layout for graph
\(\GraphGVE\) and let \(\phi_\Layout\) be a function that assigns to each vertex its (possibly empty) multiset of angles
as defined below and in \acl{fig}~\ref{fig:angular-explanation}.  Then the property at hand can be defined as
\begin{equation}
  \menum{ANGULAR} = \Disjunction_{v\in\GraphV}\phi_\Layout(v) \enspace.
\end{equation}

One would expect again that a high-quality layout features less variation in the angular distribution.  Ideally, the
only angles that appear should be those fractions \(2\pi/n\) where there is a vertex with degree \(n\) in the graph.

In order to determine the angles, the polar angles for each incident edge is computed first.  These numbers are then
sorted and formed to a cyclic list.  The adjacent differences between the list elements define the angles between the
incident edges around the node.  As a special rule, a vertex with degree \(1\) contributes an angle of \(2\pi\) rather
than \(0\) so it can be distinguished from the case where two incident edges have the exact same polar angle (which can
only happen if the adjacent vertices coincide).  Another corner case arises if an edge has length zero in which case its
polar angle is undefined.  There are three possible ways to deal with this case.
\begin{itemize}
\item Abort the computation and report an error.
\item Use a special \enquote{not a number} value.
\item Ignore the edge and move on.
\end{itemize}
We've decided to simply ignore such edges when computing this property.  Aborting the computation is too harsh as
coinciding vertices do occasionally occur even in high-quality layouts.  Inserting special values might seem like the
best approach at first but turns out to merely postpone the problem because no meaningful analysis can be performed with
a data set containing such values.

The computational effort for the \enum{ANGULAR} property is linear in the number of edges in the graph.

\begin{Figure}
  \begin{center}
    \input{pics/angular-explanation.tex}
  \end{center}
  \caption[Angles considered by \enum{ANGULAR} property]{%
    Our definition of angles between incident edges as used in the \enum{ANGULAR} property illustrated.
  }
  \label{fig:angular-explanation}
\end{Figure}

\begin{Figure}
  \PropertyDemo{angular}
  \caption[\enum{ANGULAR} examples]{%
    Distributions for the \enum{ANGULAR} property shown for the three example layouts.  The regular grid again features
    very sharp peaks.  The dominating peak at \(\pi/2\) is the angle between the incident edges of nodes with degree
    \(4\).  A smaller peak at \(\pi\) corresponds to the angles between the edges along the margin of the grid.  The
    barely visible signal at \(\pi3/2\) is caused by the four corner vertices.  The distorted grid shows again a similar
    distribution but with wider and additional peaks.  The layout for the graph on the right has a maximum near \(0\)
    due to the many vertices with a high degree.  Nevertheless, there are two additional pronounced peaks around \(\pi\)
    and \(2\pi\) courtesy of the vertices with degree \(2\) and \(1\) respectively.
  }
  \label{fig:angular}
\end{Figure}

\section{Edge Lengths (\enum{EDGE\_LENGTH})}
\label{sec:edge-length}
\index{properties!EDGE\_LENGTH@\enum{EDGE\_LENGTH}}

The next property we wish to introduce is the distribution of edge lengths.  For layout \(\Layout\) of graph
\(\GraphGVE\) we look at the distribution of values in the multiset
\begin{equation}
  \menum{EDGE\_LENGTH}=\multiset{\edgelen_\Layout(e)\suchthat{e}\in\GraphE}\enspace.
\end{equation}

For an \enquote{ideal} layout, edges would all be of the same length.  Of course, this can only work for very boring
graphs.  But even for graphs with a more complicated structure, one would expect to see peaks in the edge length
distribution of a good layout.  For example, inter- and intra-cluster edges should be visible.
\Acl{fig}~\ref{fig:edge-length} shows edge length distributions for thee sample layouts.

The \enum{EDGE\_LENGTH} property is straight-forward to compute with computational expense linear in the graph's number
of edges.

\begin{Figure}
  \PropertyDemo{edge-length}
  \caption[\enum{EDGE\_LENGTH} examples]{%
    Distribution of the \enum{EDGE\_LENGTH} property for three layouts.  The distribution for the regular grid on the
    left shows only a single sharp peak which would again be a Dirac delta function if it were not for the finite filter
    width.  The distorted gird in the middle has a broad signal with a few unidentifiable buckles forming an overall
    Gaussian shape.  The force-directed layout on the right exposes a clear peak at the target edge length with two
    roughly symmetric shoulders and low but non-zero signal towards either end of the spectrum.
  }
  \label{fig:edge-length}
\end{Figure}

\section{Pairwise Distances (\enum{RDF\_GLOBAL} and \enum{RDF\_LOCAL})}
\label{sec:rdf-global}
\label{sec:rdf-local}
\index{properties!RDF\_GLOBAL@\enum{RDF\_GLOBAL}}
\index{properties!RDF\_LOCAL@\enum{RDF\_LOCAL}}

The notion of a \emph{\ac{rdf}} is a concept borrowed from statistical thermodynamics and
crystallography.~\cite{Findenegg2015} It considers the distribution of pairwise distances between particles, which in
our case are the nodes in the layout.

\index{RDF|see{radial distribution function}}
\index{radial distribution function}
\begin{definition}[\acl*{rdf}]
  For an ensemble of particles in space \(\Reals^n\) with \(n\in\IntsN\), the \emph{\acl{rdf}} is a function
  \(g:\RealsNN\to\RealsNN\) where \(g(r)\) for \(r\in\RealsNN\) is the average number of other particles found in the
  infinitesimal spherical shell \(\{\vec{x}\in\Reals^n\suchthat{r}\leq\vecnorm{\vec{x}-\vec{p}}\leq{r}+\diff{r}\}\)
  around a particle at position \(\vec{p}\in\Reals^n\) divided by the volume of the shell.
\end{definition}

For a finite ensemble, the \ac{rdf} may be computed via the pairwise distances between particles.

\index{hard spheres}
\Ac{rdf} seems to be an interesting concept in the field of graph drawing, too.  Ideally, one would expect \(g(r)=0\)
for all values of \(r\) below a certain threshold.  That is, nodes are never drawn closer to each other than a minimum
distance which should be at least the size of the symbol used to visualize a vertex.  In the context of physical
chemistry, this is commonly referred to as a \enquote{hard sphere} model, meaning that nodes are thought of as
impenetrable solid bodies.

Furthermore, one would expect to see distinctive features in the distribution function corresponding to certain
characteristic distances in the drawing of a graph with a regular structure.  On the other hand, the \ac{rdf} \(g(r)\)
for a random placement of nodes (akin to an ideal gas) would be a constant independent of the radius \(r\).  One has to
be aware, however, that graph drawings usually feature structures with a number of vertices that is considerably smaller
than Avogadro's number\footnote{%
  The Avogadro constant \(N_\mathrm{A}\) is the number of carbon atoms found in \(12\,\text{g}\) of the carbon isotope
  \mbox{\textsuperscript{12}C} and is currently estimated~\cite{NISTAvogadro} to be
  \(N_\mathrm{A}\approx6.022\,140\,857\,(74)\times10^{23}\).
}
so the margins cannot be ignored which makes distributions look different than those that would be expected from
physics.

Equipped with this definition and motivation, we can define the property
\begin{equation}
  \menum{RDF\_GLOBAL} = \multiset{\dist_\Layout(v_1,v_2)\suchthat{}v_1,v_2\in\GraphV}
\end{equation}
for a layout \(\Layout\) of graph \(\GraphGVE\).  \Acl{fig}~\ref{fig:rdf-global} shows the \enum{RDF\_GLOBAL} property
for the familiar example layouts used in this section before.

\begin{Figure}
  \PropertyDemo{rdf-global}
  \caption[\enum{RDF\_GLOBAL} examples]{%
    Distributions for the \enum{RDF\_GLOBAL} property shown for the three example layouts.  The \acs*{rdf} for the
    regular grid shows that \(g(r)=0\) for \(r<100\) with a distinguished peak at \(r=100\) corresponding to the closest
    distance in the lattice.  The next peak at \(r=100\times\sqrt{2}\) corresponds to the next larger (diagonal)
    distance.  Large double peaks around \(r=200\) and \(r=300\) are caused by the even more frequently occurring
    distances to vertices two or three cells away respectively as well as the distances of \(r\approx224\) and
    \(r\approx283\) which correspond to the diagonals of \(2\times1\) and \(2\times2\) cells respectively.  As the
    radius increases, more and more peaks become visible until the distribution starts to decay again due to the finite
    graph size before a continuum is reached.  The \acs*{rdf} of the distorted grid resembles the basic structure of
    that for the regular grid but features \enquote{fingers} superimposed on a large base signal instead of distinctive
    sharp peaks.  The peaks at the radii discussed before can still be clearly identified, though.  The \acs*{rdf} for
    the force-directed layout of the larger graph at the right makes it hard to identify any significant features except
    for the clearly visible peak at \(r=100\) caused by the many single-edge distances along the more or less regular
    strands of nodes.
  }
  \label{fig:rdf-global}
\end{Figure}

\index{iterator@iterator (\CXX)}
In order to compute the property \enum{RDF\_GLOBAL} the algorithm has to loop over all (unordered) pairs of vertices in
the graph which means quadratic expense.  Another practical problem arises from the fact that even for a relatively
moderately-sized graph of, say, \(n=50\,000\) vertices, the memory requirement to store the \(n^2/2\) pairwise distances
as individual 8~byte IEEE~754 floating-point numbers would be close to \(10\,\text{GiB}\).  Even on a machine that has
that much \acs{ram} available, excessive cache misses would make the computation very slow.  Therefore, we avoid storing
the entire data set in memory at once and instead compute the numbers on-the-fly as they are needed for the subsequent
analysis.  In {\CXX} this can be done straight-forwardly and without additional overhead by using iterators.

Looking at distributions of \enum{RDF\_GLOBAL} such as in \acl{fig}~\ref{fig:rdf-global}, it seems that the
signal-to-noise ratio is not very good.  Especially the long tails towards larger radii contain little information while
the details near \(r=0\) are obscured by the overwhelming amount of data.  Another reason to question the applicability
of the unmodified concept of \ac{rdf} to graph drawing is that unlike for the analysis of gasses and liquids where each
particle is alike, the vertices in a graph have very special connections, namely, edges.  Therefore, it might be more
insightful to study pairwise distances only of those pairs of vertices that have a graph-theoretical distance below a
given threshold.  This leads to the following parameterized property
\begin{equation}
  \menum{RDF\_LOCAL}(d) = \multiset{\dist_\Layout(v_1,v_2)\suchthat\dist(v_1,v_2)\leq{}d\suchthat{}v_1,v_2\in\GraphV}
\end{equation}
for a layout \(\Layout\) of graph \(\GraphGVE\) and \(d\in\IntsN\).  This property turns out to be an interesting
intermediate between two properties we've already introduced as the following limits show.
\begin{align}
  \menum{EDGE\_LENGTH} &= \menum{RDF\_LOCAL}(1)                     \label{eq:rdf-local-limit-lower}\\
  \menum{RDF\_GLOBAL}  &= \lim_{d\to\infty} \menum{RDF\_LOCAL}(d)   \label{eq:rdf-local-limit-upper}
\end{align}
For connected graphs, the limit in \acl{eq}~\ref{eq:rdf-local-limit-upper} is an exaggeration.  The property won't
change for values of \(d\) larger than the graph's diameter, that is
\(d>\max\{\dist(v_1,v_2)\suchthat{}v_1,v_2\in\GraphV\}\).  Furthermore, we found it to be sufficient to compute only
\(\menum{RDF\_LOCAL}(2^i)\) for \(i\in\IntsNz\).  Examples for \enum{RDF\_LOCAL} are shown in
\acl{fig}~\ref{fig:rdf-local}.

\begin{Figure}
  \begin{center}
    \InputTikzGraph*{0.5\textwidth}{pics/demograph-e.tikz}
  \end{center}
  \bigskip\input{pics/demograph-e.tex}
  \begin{center}
    \begin{tabular}{c@{\qquad}c@{\qquad}c}
      \InputLuatikzPlot[width=0.275\textwidth]{pics/rdf-local-0.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/rdf-local-1.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/rdf-local-2.pgf}\\[2ex]
      %\(d=2^0=1\)  & \(d=2^1=2\)   & \(d=2^2=4\)\\[4ex]
      \InputLuatikzPlot[width=0.275\textwidth]{pics/rdf-local-3.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/rdf-local-4.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/rdf-local-5.pgf}\\[2ex]
      %\(d=2^3=8\)  & \(d=2^4=16\)  & \(d=2^5=32\)
    \end{tabular}
  \end{center}
  \caption[\enum{RDF\_LOCAL} examples]{%
    Distributions for the property \(\menum{RDF\_LOCAL}(2^i)\) shown for increasing values of \(i\in\{0,\ldots,5\}\).
    The example shows a native layout of a graph generated by the \enum{MOSAIC2} generator
    (\acs{section}~\ref{sec:mosaic}).  The graph has \(\GraphNodes\) vertices, \(\GraphEdges\) edges and a diameter of
    \(\GraphDiameter\).
  }
  \label{fig:rdf-local}
\end{Figure}

\index{iterator@iterator (\CXX)}
In order to compute the \enum{RDF\_LOCAL} property, an all-pairs shortest path matrix is computed first, using
Dijkstra's algorithm (\acs{cf}~\textcite{Sanders2008}).  This matrix must be kept in memory.  We have tried to avoid
this by computing the shortest single-source shortest path vector instead on-demand but this turned out to be
prohibitively slow.  On the other hand, the almost cubic operation of finding the matrix is so expensive that memory
constraints will rarely be the limiting factor.  Once the matrix is computed, the same approach as for
\enum{RDF\_GLOBAL} is used to do the actual analysis except that the iterator which computes pairwise distances
on-the-fly is modified to skip over pairs of vertices if their graph distance exceeds the locality parameter.  The same
shortest-paths matrix can be reused for computing \(\menum{RDF\_LOCAL}(2^i)\) for \(i=0\) up until \(2^i\) exceeds the
graph's diameter.\footnote{%
  If the graph is disconnected, we use the longest shortest path in any connected component instead.
}

\section{Tension (\enum{TENSION})}
\label{sec:tension}
\index{properties!TENSION@\enum{TENSION}}

The last property we considered is the ratio of graph distance and layout distance.  This property is strongly motivated
by the stress function introduced by \textcite{Kamada1989} (\acs{cf}~\acs{section}~\ref{sec:stress}).
\citeauthor{Kamada1989} defined the stress in the layout \(\Layout\) of graph \(\GraphGVE\) with \(\card{\Vec{V}=n}\)
as proportional to
\begin{equation}
  \label{eq:stress-alt}%  eq:stress is already defined
  \sum_{i=1}^{n-1} \sum_{j=i+1}^{n}
  \left(\frac{\dist_\Layout(v_i,v_j)-l\dist(v_i,v_j)}{\dist(v_i,v_j)}\right)^2
\end{equation}
where \(l\) is the desired average edge length.  Unfortunately, this definition depends in a non-linear fashion on the
scale of the drawing.  This problem was already pointed out by \textcite{Welch2017} who circumvented it by using the
minimal value the stress function will take on when the layout is scaled.  In order to do so, they employed a binary
search strategy.  Since we'd rather avoid doing this, we decided to not use the quantity from
\acl{eq}~\ref{eq:stress-alt} but instead modify it such that it behaves linear in response to scale.  This gives rise
to the following property
\begin{equation}
  \menum{TENSION} =
  \frac{\card{\GraphE}}{\sum\limits_{e\in\GraphE}\edgelen_\Layout(e)}
  \multiset{\frac{\dist_\Layout(v_1,v_2)}{\dist(v_1,v_2)}\suchthat{}v_1,v_2\in\GraphV}
\end{equation}
for which we've chosen the name \enquote{tension} for its similarity to \enquote{stress}.  \enum{TENSION} is defined
like this and not the reciprocal because the layout distance \(\dist_\Layout\) might be zero which would make the
quantity undefined if it were to occur in the denominator while the graph distance \(\dist\) can only take on values
that are positive integers.  The normalization factor could be left off without doing any harm except that it is nice to
see the ideal value centered at \(1\) in diagrams.  (Note that the factor is always \(1/100\) by
\acl{def}~\ref{def:normalized} anyway.)

Our approach to tension differs from the usual understanding of stress in that we are not so much concerned about the
absolute value (in fact, we never sum it up) but rather the \emph{distribution} of the pairwise tension.  This is
illustrated for the familiar example graphs in \acl{fig}~\ref{fig:tension}.

\begin{Figure}
  \begin{center}
    \setlength{\samplelayoutwidth}{0.25\textwidth}
    \begin{tabular}{c@{\qquad}c@{\qquad}c}
      \InputTikzGraph*{\samplelayoutwidth}{pics/demograph-a.tikz}&
      \InputTikzGraph*{\samplelayoutwidth}{pics/demograph-b.tikz}&
      \InputTikzGraph*{\samplelayoutwidth}{pics/demograph-c.tikz}\\[1ex]
      \InputLuatikzPlot[width=\samplelayoutwidth]{pics/tension-a.pgf}&
      \InputLuatikzPlot[width=\samplelayoutwidth]{pics/tension-b.pgf}&
      \InputLuatikzPlot[width=\samplelayoutwidth]{pics/tension-c.pgf}\\[2ex]
      \multicolumn{3}{r}{Compare this to the stress-minimized layout of the same graph:}\\[1ex]
      \multicolumn{2}{r}{\InputTikzGraph*{\samplelayoutwidth}{pics/demograph-d.tikz}}&
      \InputLuatikzPlot[width=\samplelayoutwidth]{pics/tension-d.pgf}
    \end{tabular}
  \end{center}
  \caption[\enum{TENSION} examples]{%
    The two top rows show the distributions for the \enum{TENSION} property for the three example layouts used
    throughout this section.  The distribution for the regular layout of the grid features two distinctive peaks at
    \(1\) and at \(\sqrt{2}/2\) corresponding to the direct neighbors and indirect neighbors that face each other
    diagonally respectively.  The smaller peaks in between may be explained similarly.  In the distorted grid, the peak
    at \(100\) is much smaller and the widened peak around \(\sqrt{2}/2\) dominates the spectrum.  The smaller peaks in
    between have almost disappeared.  The distribution for the force-directed layout of the \enum{BCSPWR} graph in the
    last column features two overlapping broad peaks.  The smaller one is centered at somewhat less than \(1\) and
    corresponds to the nodes along the strands.  Since those are the shortest edges in the layout and the layout and
    therefore shorter than the average edge length.  The second larger peak around \(\sqrt{2}\) is not so easily
    identified in the layout.  The bottom row shows the stress-minimized layout of the \enum{BCSPWR} graph and according
    \enum{TENSION} distribution next to it.  Compared to the alternative force-directed (\enum{FMMM}) layout above, the
    double-peak got fused into a single asymmetric peak with a maximum at \(1\) that drops sharply to zero around
    \(\sqrt{2}\) and has a much smoother decay towards the left end of the spectrum.
  }
  \label{fig:tension}
\end{Figure}

In order to compute tension, an all-pairs shortest path matrix is again computed first using Dijkstra's algorithm.  Once
this is done -- which might be as expensive as cubic in \(n\) -- the remaining calculation has quadratic cost.  As
already discussed in the context of \ac{rdf}, we avoid the upfront computation of all values and instead compute them
lazily as needed.

\section{Summary}

We have presented seven statistical properties in this section that we believe can be representative syndromes of a
graph layout's aesthetic value as illustrated with examples of layouts of different qualities.  Each property is a
multiset of real values.  We have also described how those values can be computed efficiently.  The discussed properties
were (for a graph \(\GraphGVE\) with \(\card{\GraphV}=n\) and \(\card{\GraphV}=m\)):
\begin{itemize}
\item\enum{PRINCOMP1ST} --- The distribution of vertex coordinates projected onto the major axis of the layout which can
  be found by means of a \ac{pca}.  The property contains \(n\) values which can be computed with effort not
  significantly worse than \(\BigO(n)\) (\acs{cf}~\acs{algo}~\ref{algo:princomp}).
\item\enum{PRINCOMP1ST} --- The distribution of vertex coordinates projected onto the minor axis of the layout.  The
  property contains \(n\) values which can be computed like for \enum{PRINCOMP1ST} except that a Gram-Schmidt
  orthonormalization and an additional power iteration step, both roughly \(\BigO(n)\), have to be performed
  (\acs{cf}~\ref{algo:princomp}).
\item\enum{ANGULAR} --- The distribution of angles between adjacent edges.  The property has \(m\) values and can be
  computed with \(\BigO(n+m)\) effort even if no specialized data structure is used.
\item\enum{EDGE\_LENGTH} --- The distribution of edge lengths.  The property has \(m\) values and can be computed with
  \(\BigO(m)\) effort.
\item\enum{RDF\_GLOBAL} --- The distribution of pairwise distances between all nodes.  The property has \(n(n-1)\)
  values and computation takes \(\BigO(n^2)\).
\item\(\menum{RDF\_LOCAL}(d)\) --- The distribution of pairwise distances between vertices with graph-theoretical
  distance of no more than \(d\in\IntsN\).  We found it sufficient to consider only values \(d=2^i\) for
  \(i\in\IntsNz\).  The property \enum{RDF\_LOCAL} provides an intermediate view between the extremes of the
  \enum{EDGE\_LENGTH} (\(d=1\)) and \enum{RDF\_GLOBAL} (\(d\to\infty\)) properties.  Consequently, it features between
  \(m\) and \(n(n-1)\) values.  Computation of \enum{RDF\_LOCAL} requires an all-pairs shortest path matrix to be
  computed which might take up to \(\BigO(n^3)\) and dominates the subsequent \(\BigO(n^2)\) accumulation step.
\item\enum{TENSION} --- The distribution of quotients of node and graph distances.  \enum{TENSION} is motivated by and
  named after stress~\cite{Kamada1989}.  It yields \(n(n-1)\) values but computing them requires knowledge of all
  pairwise shortest paths and can therefore be as expensive as \(\BigO(n^3)\).
\end{itemize}

\index{edge!crossings}
The reader might have noticed that we did not introduce any property that considers the \emph{crossings} of edges.  This
obvious omission is left for future work to be done.  One difficulty with it is that -- unlike for all other properties
we considered -- the \emph{number} of events\footnote{%
  The \emph{values} of the events obviously depend on the layout or the measure would be no good for analyzing layouts.
}
does not depend solely on the \emph{graph} but actually on the \emph{layout} which makes it a little challenging to
integrate; especially the (desirable) case where there are no crossings in the drawing at all so no statistics can be
done.

We did not explain yet how to feed the described properties into an algorithm that would make use of them.  We will
defer this discussion until \acl{chap}~\ref{chap:featex}.

\chapter{Data Generation}
\label{chap:datgen}

\section{Graphs}
\label{sec:graphs}

\index{generator}
\index{graph!generator|see{generator}}
\index{layout!native}
\index{layout!NATIVE@\enum{NATIVE}}

A \emph{graph generator} is a probabilistic algorithm that might take an implementation-defined set of input parameters
and outputs a graph, optionally alongside with a \emph{native} layout (\acs{section}~\ref{sec:native}) that is assumed
to be of good quality.  Native layouts differ from layouts computed by layout algorithms in that they fall out directly
as a by-product of the graph generation.  For example, the \enum{MOSAIC1} generator (\acs{section}~\ref{sec:mosaic})
which derives graphs by recursively splitting facets of an initial simplex produces a naturally occurring layout
alongside with the graph that, if nothing else, is guaranteed to be planar.

We have implemented several graph generators that produce a variety of graphs for which aesthetically pleasing layouts
exist.

\index{configuration files!graphs.cfg@\verb`graphs.cfg`}
The driver for our experiments reads a configuration file \verb`graphs.cfg` that specifies how many graphs of what size
class are desired for which generator.  It will then populate the database by repetitively running each generator (with
appropriate inputs) until the database contains all desired graphs.

In order to avoid duplicates, a hash is computed of each graph.  If a graph with the same hash already exists in the
database, the graph is discarded and the generator called again, hoping it will produce another graph this time.
Otherwise, the graph is added to the database and from now on referred to by using its hash as unique \acs{id}.

If the generator supports specifying the approximate graph size, the driver will make use of it.  In any case, the
actual graph size will be checked before the graph is added to the database.  If the actual size diverges from the
requested size, it is first checked whether a graph of that size is also still on the worklist and if so, the graph is
added nonetheless.  Otherwise, the graph has to be discarded.

\subsection{Imported Graphs (\enum{IMPORT})}
\label{sec:import}
\index{generator!IMPORT@\enum{IMPORT}}

The \enum{IMPORT} generator is special in that it does not genuinely generate graphs but merely imports existing graphs.
Formally, its input is a collection of graphs (and maybe layouts) from which it randomly picks and returns one.

\index{configuration files!imports.json@\verb`imports.json`}
Our implementation allows to specify import sources in a configuration file \verb`imports.json` that specifies the
archive type (directories and \acs{tar} archives are supported as well as lists of individual \acsp{url}), file format
(all formats implemented in the \acs*{ogdf} are supported), file compression, whether the graphs have associated layout
information and the \acs{url} where the archive can be found.  An optional cryptographic checksum can be specified for
\acs{tar} archives as well in order to validate the integrity of the archive prior to its use.  The driver will then
read this configuration file and feed the archives to the \enum{IMPORT} generator.  Caching is performed to avoid
needless queries.

\subsubsection{graphdrawing.org (\enum{ROME}, \enum{NORTH}, \enum{RANDDAG})}
\index{generator!ROME@\enum{ROME}}
\index{generator!NORTH@\enum{NORTH}}
\index{generator!RANDDAG@\enum{RANDDAG}}

Three import sources were predefined for some graph collections found at \mbox{graphdrawing.org} \cite{graphdrawing.org}
and referred to as the \enum{ROME}\footnote{%
  \url{http://graphdrawing.org/download/rome-graphml.tgz}
}, \enum{NORTH}\footnote{%
  \url{http://graphdrawing.org/download/north-graphml.tgz}
}, and \enum{RANDDAG}\footnote{%
  \url{http://graphdrawing.org/download/random-dag-graphml.tgz}
}
generators which are really just aliases for the \enum{IMPORT} generator.  Each of these archives contains a number of
graphs much larger than what would be practical to process with our setup so we only imported a small fraction.
Unfortunately, the graphs in these collections have no associated known-good layouts.  They are also all relatively
small with the average number of nodes around 50.  Examples of graphs from these archives are shown in
\acl{fig}~\ref{fig:import}.

\begin{Figure}
  \begin{center}
    \begin{tabular}{c@{\qquad}c@{\qquad}c}
      \InputTikzGraph{0.25\textwidth}{pics/rome.tikz}&
      \InputTikzGraph{0.25\textwidth}{pics/north.tikz}&
      \InputTikzGraph{0.25\textwidth}{pics/randdag.tikz}\\[2ex]
      \enum{ROME} & \enum{NORTH} & \enum{RANDDAG}\\
    \end{tabular}
  \end{center}
  \caption[Examples of \enum{ROME}, \enum{NORTH} and \enum{RANDDAG} graphs]{%
    Examples of three randomly chosen graphs from the three considered archives (stress-minimized layouts).
  }
  \label{fig:import}
\end{Figure}

\subsubsection{Matrix Market (\enum{SMTAPE}, \enum{PSADMIT}, \enum{GRENOBLE}, \enum{BCSPWR})}
\index{generator!SMTAPE@\enum{SMTAPE}}
\index{generator!PSADMIT@\enum{PSADMIT}}
\index{generator!GRENOBLE@\enum{GRENOBLE}}
\index{generator!BCSPWR@\enum{BCSPWR}}

The \enquote{Matrix Market}~\cite{MatrixMarket} maintained by the \ac{nist} is an online collection of matrices from a
wide variety of disciplines.  Consequently, it contains many different matrices and not all of them are interesting
examples for graph drawing.  We have considered the \enum{SMTAPE}\footnote{%
  \url{https://math.nist.gov/MatrixMarket/data/Harwell-Boeing/smtape/smtape.html}
},
\enum{PSADMIT}\footnote{%
  \url{https://math.nist.gov/MatrixMarket/data/Harwell-Boeing/psadmit/psadmit.html}
},
\enum{GRENOBLE}\footnote{%
  \url{https://math.nist.gov/MatrixMarket/data/Harwell-Boeing/grenoble/grenoble.html}
}
and \enum{BCSPWR}\footnote{%
  \url{https://math.nist.gov/MatrixMarket/data/Harwell-Boeing/bcspwr/bcspwr.html}
}
sets (all from the Harwell-Boeing collection) where we found the latter to be most interesting.  No examples are shown
here because the graphs are pretty large (usually several thousand vertices) and it is hard to identify a representative
example.

\subsection{Regular Grids (\enum{GRID}) and Tori (\TorusN)}
\label{sec:grid}
\label{sec:torus}
\index{generator!GRID@\enum{GRID}}

\begin{Figure}
  \begin{center}
    \InputTikzGraph{0.9\textwidth}{pics/grid.tikz}
  \end{center}
  \caption[Example of a \enum{GRID} graph]{%
    An example of a graph output by the \enum{GRID} generator (native layout).
  }
  \label{fig:grid}
\end{Figure}

The \enum{GRID} generator, when asked to produce a graph with \(n\in\IntsN\) vertices, chooses two random integers
\(1\leq{}n_1,n_2\leq2\sqrt{n}\) independently according to a uniform distribution and outputs a graph that is a regular
\(n_1\times{}n_2\) grid with the obvious native layout.  An example of a \enum{GRID} graph's native layout is shown in
\acl{fig}~\ref{fig:grid}.

\index{generator!TORUS@\TorusN}
The \enum{TORUS1} generator operates similarly except that it adds additional edges from the first to the last vertex in
a \enquote{row} such as to obtain a \(1\)-torus (a cylinder).  The \enum{TORUS2} generator additionally connects the
first and last vertex of each \enquote{column} with an edge which yields a \(2\)-torus (a doughnut).  The \enum{TORUS1}
and \enum{TORUS2} generators do not output native layouts because there is no obvious mapping of a \(n\)-torus to a
\(2\)-dimensional layout, although a projection of the \(3\)-dimensional objects into two dimensions would be possible.
Examples for graphs generated by \enum{TORUS1} and \enum{TORUS2} are shown in \acl{fig}~\ref{fig:torus}.

\begin{Figure}
  \begin{center}
    \begin{tabular}{c@{\hspace{0.1\textwidth}}c}
      \InputTikzGraph{0.4\textwidth}{pics/torus1.tikz}&
      \InputTikzGraph{0.4\textwidth}{pics/torus2.tikz}\\[2ex]
      \enum{TORUS1} & \enum{TORUS2}\\
    \end{tabular}
  \end{center}
  \caption[Examples of \TorusN{} graphs]{%
    Examples of graphs output by the \enum{TORUS1} and \enum{TORUS2} generators (force-directed layouts).
  }
  \label{fig:torus}
\end{Figure}

\subsection{Stochastic L-Systems (\enum{LINDENMAYER})}
\label{sec:lindenmayer}
\index{generator!LINDENMAYER@\enum{LINDENMAYER}}

\index{L-system}
\index{Lindenmayer|see{L-system}}
\index{L-system!stochastic}
A \emph{Lindenmayer System} \(L=(\Sigma,\omega,\Pi)\) is a text-rewriting system consisting of an \emph{alphabet}
\(\Sigma\), an initial \emph{axiom} \(\omega\in\Sigma\) and a set of \emph{production rules} \(\Pi\).  Each production
\(\pi\in\Pi\) is a two-tuple consisting of a \emph{pattern} to be matched and a \emph{replacement text} by which an
occurrence of the pattern in the text may be replaced.  A \emph{stochastic L-system} is a probabilistic algorithm that
derives a random word \(w\in\Sigma^*\) by starting with the initial axiom \(\omega\) and recursively applying
productions from \(\Pi\) chosen randomly among those applicable.  L-systems were first described by and named after
\textcite{Lindenmayer1990} who used them to describe the organic growth of plants in nature (see
\acl{fig}~\ref{fig:romanesco} for an inspiration).

\begin{Figure}
  \begin{center}
    % https://commons.wikimedia.org/wiki/File:Fractal_Broccoli.jpg
    \pgfimage[width=0.5\textwidth]{pics/romanesco.jpg}
  \end{center}
  \caption[Romanesco Broccoli]{%
    \emph{Romanesco broccoli} is a particular spectacular example of a self-similar plant that may be modeled using a
    stochastic L-system.  [Original photo by Jon Sullivan (2004), public domain.]
  }
  \label{fig:romanesco}
\end{Figure}

The \enum{LINDENMAYER} generator generates graphs by recursively replacing nodes with symmetric sub-graphs.  Its initial
axiom is a singleton graph \(\Graph_\omega=(\{v\}, \emptyset)\).  Productions are defined by the following
sub-generators each of which replaces a single vertex by a more complicated sub-graph.

\begin{itemize}
\item \(\menum{L/SINGLETON}\) is an identity transformation.
\item \(\menum{L/STAR}_k\) for \(k\in\IntsN\) splits all edges incident to \(v\) by inserting new vertices and
  distributes \((k-1)\deg(v)\) additional vertices, each connected to \(v\), radially between them.  If \(\deg(v)=0\)
  then \(k\) vertices are distributed radially around \(v\) and connected to it.
\item \(\menum{L/WHEEL}_k\) for \(k\in\IntsN\) is the same as \(\menum{L/STAR}_k\) except that the new vertices are
  additionally connected to a ring.
\item \(\menum{L/RING}_k\) for \(k\in\IntsN\) is the same as \(\menum{L/WHEEL}_k\) except that vertex \(v\) is deleted.
\item \(\menum{L/CLIQUE}_k\) for \(k\in\IntsN\) is the same as \(\menum{L/RING}_k\) except that all newly added vertices
  are connected to a clique.
\item \(\menum{L/GRID}_{n,m}\) for \(n,m\in\IntsN\) replaces a vertex \(v\) with \(\deg(v)=0\) by a regular
  \(n\times{}m\) grid.
\end{itemize}

The operation of these sub-generators is illustrated in \acl{fig}~\ref{fig:lindenmayer-subgens}.

\begin{Figure}
  \begin{center}
    \begin{tabular}{c@{\qquad\qquad}c}
      \input{pics/gen-lindenmayer-singleton.tex} & \input{pics/gen-lindenmayer-star.tex}\\
      \(\menum{L/SINGLETON}\) & \(\menum{L/STAR}_3(v)\)\\[2ex]
      \input{pics/gen-lindenmayer-wheel.tex} & \input{pics/gen-lindenmayer-ring.tex}\\
      \(\menum{L/WHEEL}_3(v)\) & \(\menum{L/RING}_3(v)\)\\[2ex]
      \input{pics/gen-lindenmayer-clique.tex} & \input{pics/gen-lindenmayer-grid.tex}\\
      \(\menum{L/CLIQUE}_2(v)\) & \(\menum{L/GRID}_{3,5}(v)\)\\
    \end{tabular}
  \end{center}
  \caption[%
    \enum{LINDENMAYER} sub-generators \(\menum{L/SINGLETON}\), \(\menum{L/STAR}_k\), \(\menum{L/WHEEL}_k\),
    \(\menum{L/RING}_k\), \(\menum{L/CLIQUE}_k\) and \(\menum{L/GRID}_{n,m}\)
  ]{%
    Operation of the \enum{LINDENMAYER} sub-generators on a vertex \(v\) adjacent to four vertices \(u_1,\ldots,u_4\)
    illustrated; except for the \(\menum{L/GRID}_{n,m}\) example where \(\deg(v)=0\) initially as required for its
    applicability.  (Hence the grid is not connected to any other node after the operation has completed either.)
  }
  \label{fig:lindenmayer-subgens}
\end{Figure}

For a desired number of nodes \(n\), the \enum{LINDENMAYER} generator starts with a singleton graph
\(\Graph_\omega=(\{v\},\emptyset)\) and randomly picks one of the sub-generators, instantiates it with suitable
parameters chosen probabilistically and applies it to \(v\).  This process is repeated recursively for each new node
added until the graph has grown to the desired number of nodes.  At each level of the recursion, the recursive
application is done using the same random seed for each vertex which yields a symmetric graph.  The process is
illustrated in \acl{algo}~\ref{algo:lindenmayer}.

When a vertex \(v\) with \(\deg(v)=d\) adjacent to vertices \(u_1,\ldots,u_d\) is replaced by a sub-graph, the new
vertices of the sub-graph are laid out in such a way that the edges that were incident to \(v\) do not change their
direction.  That is, new vertices are placed along the original edges connecting \(v\) and \(u_1,\ldots,u_d\).
Additional new vertices (if any) are distributed uniformly between those vertices already fixed.  The scale of the
sub-layout is chosen such that it doesn't overlap with any other existing structure.  An example of such a graph and
native layout is shown in \acl{fig}~\ref{fig:lindenmayer}.

\begin{Algorithm}
  \begin{extraleading}
    \KwIn{%
      Graph \(\GraphGVE\),
      vertex \(v\in\GraphV\),
      desired size \(n\in\IntsN\) and seed \(s\in\Bits^*\).
    }
    \KwOut{Graph with approximately \(n\) vertices.}
    \Routine{%
      Replace \(v\) by new vertices \(v_1,\ldots,v_l\) to obtain \(\Graph^{(0)}=(\GraphV^{(0)},\GraphE^{(0)})\)
        \comment*{(1)}
      \(n'\gets\floor{(n-l)/l}\)\;
      \If{\(n'\leq0\)}{
        \Return\(\Graph^{(0)}\)\;
      }
      Instantiate a pseudo random generator \(\mathcal{R}\) with seed \(s\)\;
      Use \(\mathcal{R}\) to choose and instantiate a sub-generator \(\mathcal{G}\) \comment*{(2)}
      Use \(\mathcal{R}\) to derive a new seed \(s'\in\Bits^*\)\;
      \ForEach{\(i\in\{1,\ldots,l\}\)}{
        \(\Graph^{(i)}\gets\mathcal{G}(\Graph^{(i-1)}, v_i, n', s')\);
      }
      \Return\(\Graph^{(l)}\)\;
    }
  \end{extraleading}
  \caption[Simplified conceptual operation of the \enum{LINDENMAYER} graph sub-generators]{%
    Simplified conceptual operation of the \enum{LINDENMAYER} graph sub-generators.  The actual logic is a little bit
    more complicated because instead of choosing the same sub-generator \(\mathcal{G}\), a more interesting but still
    symmetric graph can be obtained by choosing several sub-generators.  For example, the \(\menum{L/RING}_k\)
    sub-generator uses two sub-generators \(\mathcal{G}'\) and \(\mathcal{G}''\).  It then applies \(\mathcal{G}'\) to
    each new vertex \(v_i\) with \(i\equiv0\pmod{k}\) and \(\mathcal{G}''\) to the other new vertices.
    % Remark (1)
    The replacement at (1) is done according to the rules specific for the sub-generator and its parameters.  For
    example, the \(\menum{L/CLIQUE}_k\) sub-generator would replace \(v\) by an \(l\)-clique (with \(l=k\deg(v)\)) and
    distribute the edges that were incident to \(v\) uniformly among the new vertices.
    % Remark (2)
    The choice of sub-generator at (2) has to take into account that some sub-generators are not always applicable.  For
    example, the \enum{L/GRID} sub-generator requires that \(\deg(v)=0\).  The parameters to instantiate the
    sub-generator should be chosen according to a random distribution that takes into account how many vertices the
    sub-generator should add at most.
    %
    Also not shown is the logic to determine the coordinates for the new nodes as it is pretty straight-forward to
    imagine but rather tedious to implement.
  }
  \label{algo:lindenmayer}
\end{Algorithm}

\begin{Figure}
  \begin{center}
    \begin{tikzpicture}
      \node at (-0.25\textwidth, 0) {\InputTikzGraph{0.45\textwidth}{pics/lindenmayer-a.tikz}};
      \node at (+0.25\textwidth, 0) {\InputTikzGraph{0.45\textwidth}{pics/lindenmayer-b.tikz}};
    \end{tikzpicture}
  \end{center}
  \caption[Examples of two \enum{LINDENMAYER} graphs]{%
    Examples of two graphs produced by the \enum{LINDENMAYER} generator (native layouts).
  }
  \label{fig:lindenmayer}
\end{Figure}

\subsection{Projections of Hyper-Lattices (\QuasiNd)}
\label{sec:quasi}

\index{quasicrystal}
A \emph{quasicrystal} is a chemical compound that has an ordered but aperiodic structure intermediate between an
amorphous (unordered) and crystalline (periodic) structure.~\cite{WikiQuasicrystal}
As in a crystal, any sub-region of arbitrary size found in a quasicrystal may be brought to congruence with an infinite
number of other sub-regions in the same structure by means of rotation and translation.  However, unlike a crystal, the
series defined by the required amounts of translation is not periodic.  Quasicrystals can (mathematically) be obtained
as the projection of a periodic structure in a higher-dimensional space onto a lower-dimensional space which intersects
the higher-dimensional space at an irrational angle.  If the angle of intersection is rational, a regular crystal (that
is, a periodic pattern) will be obtained.

\index{generator!QUASI@\QuasiNd}
The \QuasiNd{} generators are inspired by this concept.  They use a primitive cubic lattice in three- to six-dimensional
space and project it onto a two-dimensional plane intersecting that space at a random angle.  No precautions are taken
to make the intersection angle an irrational number for three reasons.  First, since we are not going to generate a
graph of infinite size, periodicity is not really defined in the first place.  Second, since our implementation uses
fixed-precision floating-point numbers, irrational quantities cannot be expressed anyway.  Third, it is totally
acceptable for our purposes to occasionally obtain a structure that is actually periodic.

For a given hyper-space dimension \(d\in\{3,\ldots,6\}\) and desired number of vertices \(n\in\IntsN\), a random
surface-normal \(\vec{u}\in\Reals^d\) is chosen which defines the plane \(\Vec{P}\) which (without loss of generality)
is set to intersect at the coordinate origin.  Two orthogonal unit vectors \(\vec{e}_1,\vec{e}_2\in\Vec{P}\) as well as
a \enquote{thickness} \(1/10\leq{t}\leq11/10\) are chosen at random.  All points \(\vec{p}\in\IntsZ^d\) with an
orthogonal distance to \(\Vec{P}\) of no more than \(t\) are projected onto \(\Vec{P}\).  Among those, the points that
fall into the square \(\Vec{S}\) spanned by linear combinations \(\alpha_1\vec{e}_1+\alpha_2\vec{e}_2\) of the unit
vectors with \(0\leq\alpha_1,\alpha_2\leq\sqrt{n}\) define the vertices of the graph.  Two vertices \(v_1\) and \(v_2\)
that stem from projecting points \(\vec{p}_1,\vec{p}_2\in\IntsZ^d\) are connected by an edge if and only if
\(\vecnorm{\vec{p}_1-\vec{p}_2}=1\) (that is, \(\vec{p}_1\) and \(\vec{p}_2\) were neighbors in the \(d\)-dimensional
hyper-lattice).  The native layout follows immediately by interpreting \(\Vec{S}\) as the drawing area.

A schematic illustration of this process is shown in \acl{fig}~\ref{fig:quasi-concept} with the number of dimensions
reduced for clarity.  Examples of graphs generated by the \QuasiNd{} generators are shown in \acl{fig}~\ref{fig:quasi}.

\begin{Figure}
  \begin{center}
    \input{pics/gen-quasi-concept.tex}
  \end{center}
  \caption[Projection of a higher-dimensional lattice onto a lower-dimensional space]{%
    Concept of the \QuasiNd{} generator illustrated with a two-dimensional lattice projected onto a one-dimensional
    drawing \enquote{area}.  The generated graph in this example consists of four connected components: one pair, two
    triples and one quadruple of linearly connected vertices.
  }
  \label{fig:quasi-concept}
\end{Figure}

\begin{Figure}
  \begin{center}
    \begin{tabular}{c@{\qquad}c}
      \InputTikzGraph*{0.45\textwidth}{pics/quasi3d.tikz}&
      \InputTikzGraph*{0.45\textwidth}{pics/quasi4d.tikz}\\[1ex]
      \enum{QUASI3D} & \enum{QUASI4D}\\[2ex]
      \InputTikzGraph*{0.45\textwidth}{pics/quasi5d.tikz}&
      \InputTikzGraph*{0.45\textwidth}{pics/quasi6d.tikz}\\[1ex]
      \enum{QUASI5D} & \enum{QUASI6D}\\
    \end{tabular}
  \end{center}
  \caption[Examples of \QuasiNd{} graphs]{%
    Examples of graphs produced by the \QuasiNd{} generators via projecting a slice of a primitive cubic lattice in a
    \(n\)-dimensional hyper-space onto a two-dimensional plane that intersects that space at a random angle (native
    layouts).
  }
  \label{fig:quasi}
\end{Figure}

\subsection{Mosaic Patterns (\enum{MOSAIC1} and \enum{MOSAIC2})}
\label{sec:mosaic}
\index{generator!MOSAIC1@\enum{MOSAIC1}}
\index{generator!MOSAIC2@\enum{MOSAIC2}}

The \enum{MOSAIC1} generator starts with a regular polygon and randomly applies one of the following operations to a
randomly chosen facet defined by corner vertices \(u_1,\ldots,u_k\) of the graph layout.

\begin{itemize}
  \item\enum{M/STAR} Places a new vertex \(v\) in the center of the facet and adds an edge from \(v\) to each vertex
    \(u_1,\ldots,u_k\).
  \item\enum{M/FLOWER} Places a new vertex \(v\) in the center of the facet and splits each edge \(\{u_i,u_j\}\)
    by inserting a new vertex \(w_i\) in the middle.  Then, edges are added from \(v\) to each \(w_1,\ldots,w_k\).
  \item\enum{M/SHAPE} Splits each edge \(\{u_i,u_j\}\) by inserting a new vertex \(w_i\) in the middle and then
    connects the vertices \(w_1,\ldots,w_k\) to a ring.
\end{itemize}

When splitting edges, new nodes are only inserted if they do not already exist.  Since \(u_1,\ldots,u_k\) are only the
\emph{corners} of the facet, it is possible that there are vertices between them.  By construction of the graph layout,
if two nodes fall on a straight line, they are always either connected directly by an edge or there is a node exactly in
the middle between them.  \Acl{fig}~\ref{fig:mosaic-subgens} illustrates the operations described above.

The \enum{MOSAIC2} generator is similar but instead of randomly choosing one facet at a time, it repetitively iterates
over all facets in the graph layout and within each iteration, applies the same operation to each facet while yields a
graph and layout with mush higher symmetry.

Examples of graphs generated by the two generators are shown in \acl{fig}~\ref{fig:mosaic}.  By construction, these
graphs are always planar and the native layouts never have edge crossings.

\begin{Figure}
  \begin{center}
    \begin{tabular}{c@{\qquad}c@{\qquad}c}
      \input{pics/gen-mosaic-star.tex}&
      \input{pics/gen-mosaic-flower.tex}&
      \input{pics/gen-mosaic-shape.tex}\\[1ex]
      \enum{M/STAR} & \enum{M/FLOWER} & \enum{M/SHAPE}
    \end{tabular}
  \end{center}
  \caption[\enum{MOSAIC} sub-generators \enum{M/STAR}, \enum{M/FLOWER} and \enum{M/SHAPE}]{%
    Operation of the \enum{MOSAIC} sub-generators on a pentagonal facet \(\{u_1,\ldots,u_5\}\) illustrated.
  }
  \label{fig:mosaic-subgens}
\end{Figure}

\begin{Figure}
  \begin{center}
    \begin{tabular}{c@{\qquad}c}
      \InputTikzGraph{0.40\textwidth}{pics/mosaic1.tikz}&
      \InputTikzGraph{0.40\textwidth}{pics/mosaic2.tikz}\\[2ex]
      \enum{MOSAIC1} & \enum{MOSAIC2}
    \end{tabular}
  \end{center}
  \caption[Examples of \enum{MOSAIC1} and \enum{MOSAIC2} graphs]{%
    Examples of graphs produced by the \enum{MOSAIC1} and \enum{MOSAIC2} generators (native layouts).
  }
  \label{fig:mosaic}
\end{Figure}

\subsection{Meshes of Three-Dimensional Objects (\enum{BOTTLE})}
\label{sec:bottle}
\index{generator!BOTTLE@\enum{BOTTLE}}

The \enum{BOTTLE} generator produces graphs that are meshes of the surface of a simple three-dimensional body and native
layouts that are an axonometric projection thereof.  It was named like this because the type of random objects it
generates remotely looks like an awkwardly shaped bottle.

To generate the body and mesh, the generator proceeds as outlined in \acl{algo}~\ref{algo:bottle}.  The generated graph
is the mesh and the native layout the axonometric projection of it onto a two-dimensional surface.  An example of a
possible output of the \enum{BOTTLE} generator is shown in \acl{fig}~\ref{fig:bottle}.

\begin{Algorithm}
  \begin{extraleading}
    \KwIn{Desired number of vertices \(n\in\IntsN\) and number of coefficients \(k\in\IntsN\).}
    \KwOut{Surface mesh with approximately \(n\) vertices.}
    \Routine{%
      Choose random radius \(0\leq{}r\leq\frac{1}{2}\sqrt{n}\)\;
      Choose random length \(2r\leq{}l\leq2\sqrt{n}\)\;
      Choose random coefficients \(\alpha_1,\ldots,\alpha_k\) independently between \(0\) and \(1/k\)\;
      Define \(g(z)\gets\sqrt{r^2-(r-d)^2}\) for \(d\gets\min\{z,l-z\}\) if \(d\leq{}r\) or else \(g(z)\gets{}r\)\;
      Define \(f(z)\gets{}g(z)\big(1+\sum_{i=1}^{k}\alpha_i\sin(i\pi{}z/l)\big)\)\;
      \(\Vec{S}\gets\{(f(z)\sin(\phi),f(z)\cos(\phi),z)\}\) for \(0\leq{}z\leq{}l\) and \(0\leq\phi\leq2\pi\)\;
      Lay a mesh over \(\Vec{S}\) such that the edge length is as close to \(1\) as possible\;
    }
  \end{extraleading}
  \caption[Conceptual operation of the \enum{BOTTLE} generator]{%
    Conceptual operation of the \enum{BOTTLE} generator.  The logic for determining the mesh and projecting it on a
    two-dimensional surface is omitted because it is tedious but not very enlightening.
  }
  \label{algo:bottle}
\end{Algorithm}

\begin{Figure}
  \begin{center}
    \InputTikzGraph*{\textwidth}{pics/bottle.tikz}
  \end{center}
  \input{pics/bottle.tex}
  \caption[Example of a \enum{BOTTLE} graph]{%
    Example of a graph with \(\GraphNodes\) nodes and \(\GraphEdges\) edges output by the \enum{BOTTLE} generator
    (native layout).
  }
  \label{fig:bottle}
\end{Figure}

\subsection{Summary}

We have presented various strategies for obtaining graphs (often alongside with native layouts) and how we implemented
them in the preceding sections.  The discussed graph generators were:

\begin{itemize}
\item\enum{IMPORT} --- Imports a graph (and optionally an accompanying native layout) from a third-party source.  We
  have used graphs from \mbox{graphdrawing.org}~\cite{graphdrawing.org} (\enum{ROME}, \enum{NORTH}, \enum{RANDDAG}) and
  \acs*{nist}'s \emph{Matrix Market}~\cite{MatrixMarket} (\enum{SMTAPE}, \enum{PSADMIT}, \enum{GRENOBLE}, \enum{BCSPWR})
  for our experiments.  Care has to be taken when importing data from a third-party source that additional preprocessing
  might be required in order to make the graph and layout comply with our assumptions about normalized layouts of simple
  graphs.
\item\enum{GRID} --- Produces a regular \(n\times{}m\) grid where \(n\) and \(m\) are chosen randomly but with respect
  to the desired graph size.  The graphs produced by this generator come along with the obvious native layout.
\item\enum{TORUS1} --- Like \enum{GRID} but the first and the last vertex in each \enquote{row} are connected by an
  additional edge such to form a 1-torus (a cylinder).  No native layout is provided.
\item\enum{TORUS2} --- Like \enum{GRID} but the first and the last vertex in each \enquote{row} as well as the first and
  the last vertex in each \enquote{column} are connected by an additional edge such to form a 2-torus (a doughnut).  No
  native layout is provided.
\item\enum{LINDENMAYER} --- Uses a stochastic L-system to derive a graph by performing randomly chosen
  \enquote{productions} to a graph, replacing individual vertices with more complicated substructures in each step
  (\acs{cf}~\acs{fig}~\ref{fig:lindenmayer-subgens}).  A native layout is provided according to a non-trivial set of
  rules outlined in \acs{section}~\ref{sec:lindenmayer}.
\item\QuasiNd{} for \(d\in\{3,\ldots,6\}\) --- Projects a primitive cubic lattice in a \(d\)-dimensional hyperspace onto
  a \(2\)-dimensional plane intersecting that space at a random angle.  The native layout follows immediately from the
  construction.  The patterns are regular but potentially aperiodic.
\item\enum{MOSAIC1} --- Starts with a regular polygon and randomly divides facets according to a set of fairly simple
  rules (\acs{cf}~\acs{fig}~\ref{fig:mosaic-subgens}) until the desired graph size is reached.  The native layout
  follows by-construction.
\item\enum{MOSAIC2} --- Acts like \enum{MOSAIC1} but uses less random bits in order to obtain more symmetric graphs.
\item\enum{BOTTLE} --- Constructs a graph as a three-dimensional mesh over a random solid of revolution.  The native
  layout is the axonometric projection thereof.
\end{itemize}

We have not discussed the computational complexity of these generators because they are not always straight-forward to
quantify -- some generators use nontrivial data structures and their exact efficiency depends on the concrete access
pattern which is probabilistic.  That said, the complexity of most generators is not considerably worse than
linear and certainly negligible compared to the complexity of subsequent computation we carry out on these graphs.

\section{Layouts}
\label{sec:layouts}

Apart from native layouts -- \enum{NATIVE} (\acs{section}~\ref{sec:native}) -- we implemented two proper -- \enum{FMMM}
(\acs{section}~\ref{sec:fmmm}) and \enum{STRESS} (\acs{section}~\ref{sec:stress}) -- as well as three garbage --
\enum{RANDOM\_UNIFORM} (\acs{section}~\ref{sec:random}), \enum{RANDOM\_NORMAL} (\acs{section}~\ref{sec:random}) and
\enum{PHANTOM} (\acs{section}~\ref{sec:phantom}) -- layouts (\acs{cf}~\acs{chap}~\ref{chap:method}) that will be
described in the following subsections.  The choice of layout algorithms was driven by the desire to get a good mixture
of different layouts with a great variety of aesthetic quality.

\index{configuration files!layouts.cfg@\verb`layouts.cfg`}
As with graph generators, layout tools are implemented as small stand-alone programs that read a graph description from
a \ac{graphml} file, compute the layout and output it to a \ac{graphml} file, optionally alongside with some
meta-information such as the layout's bounding box in \ac{json} format.  Our driver script reads a configuration file
\verb`layouts.cfg` that specifies what layouts are desired for graphs of different sizes.  Since the time and memory
requirements of the layout algorithms vary greatly, it makes sense to compute certain layouts only for smaller graphs.
The driver will then loop over the graphs in the database, check whether the desired layouts already exist and if not
generate them and update the database accordingly.

\index{layout!ID}
Upon insertion into the database, each layout is assigned a unique \acs{id} that is chosen randomly.  We have abandoned
the idea of using a fingerprinting technique as we do for graphs because dealing with the inevitable collisions was too
much hassle compared to the little inefficiency of having some layouts in the database under two names.

\subsection{Layouts from External Sources (\enum{NATIVE})}
\label{sec:native}
\index{layout!NATIVE@\enum{NATIVE}}

Some graphs come along with an existing layout that was either hand-crafted or fell out of the graph generation process.
Many of our graph generators described in \acl{section}~\ref{sec:graphs} produce such \emph{native} layouts.  The
quality of a native layout is expected to be high.  There is no further commonality among native layouts.  Please refer
to \aclp{fig} \ref{fig:grid}, \ref{fig:lindenmayer}, \ref{fig:quasi}, \ref{fig:mosaic} and \ref{fig:bottle} for an
inspiration.  \enum{NATIVE} is not a layout algorithm (\ac{cf}~\acs{def}~\ref{def:layalgo}) in the strict sense because
it does not derive a layout for a given graph from first principles.  Actually, it does nothing but to create a symbolic
link.

\subsection{\Acl*{fmmm} (\enum{FMMM})}
\label{sec:fmmm}
\index{layout!NATIVE@\enum{FMMM}}

The \emph{\acf{fmmm}} introduced by \textcite{Hachul2005} in \citeyear{Hachul2005} and \emph{Stress Minimization} (see
the following section) suggested by \textcite{Kamada1989} in \citeyear{Kamada1989} and thenceforth significantly
optimized stand in as two examples of state-of-the-art general-purpose layout algorithms.  We've used the implementation
available in the \acs*{ogdf}.  Both layout algorithms can be classified as \emph{energy-based}
(\acs{ad}~\emph{force-directed}) algorithms.  Please see \textcite{Kobourov2013} for a good and relatively recent
summary and literature survey on the topic.

\index{layout!energy-based}
\index{layout!force-directed}
\index{energy function}
The basic idea of an energy-based layout algorithm is to define a multivariate \emph{energy function} for layouts
\(\Layout\) of a graph \(\GraphGVE\) with \(\Vec{V}=\{v_1,\ldots,v_n\}\) for \(n\in\IntsN\) as
\begin{equation}
  \begin{aligned}
    f_\Graph:\quad &\Reals^{2\times{}n}                &\to\quad     &\Reals\\
                   &(\Layout(v_1),\ldots,\Layout(v_n)) &\mapsto\quad &\text{\emph{implementation-defined}}
  \end{aligned}
\end{equation}
that receives as inputs the (ordered) sequence of vertex coordinates \(\Layout(v_1),\ldots,\Layout(v_n)\) and outputs a
single scalar quantity.  \(f\) ought to be a pure function that produces identical output for identical inputs.  We use
the subscript \enquote{\(\Graph\)} to indicate that it may also make use the information about the graph \(\Graph\).

An energy-based layout algorithm will try to find vertex coordinates \(\Layout(\GraphV)\) such that
\(f_\Graph(\Layout(\GraphV))\) will be minimized.  The usual global and local numerical optimization procedures for
highly-dimensional problems may be employed for this purpose.

The name of these algorithms stems from the physical interpretation of a graph as a system of spheres (vertices) that
exercise a repulsive force upon each other and are connected by springs (edges) which have a preferred length and
exercise a contracting or extending force if stretched or compressed respectively.  The function \(f_\Graph\) describes
the potential energy in the system.  The idea is that if such a system would be set up and then allowed to let go, it
would rearrange itself into a position that is supposed to be a good layout.  A straight-forward way to solve the
problem numerically is therefore to form the gradient
\begin{equation}
  \frac{\partial f_\Graph}{\partial \Layout} \big(\Layout(\GraphV)\big)
\end{equation}
of the energy function with respect to vertex coordinates and interpret it as a \emph{force} upon vertices.  Starting
with an initial layout (which can be a random arrangement of vertices) this force introduces an \emph{acceleration}
acting upon the vertices which therefore start to move, transforming potential into kinetic energy.  By continuously
removing kinetic energy from the system via a non-conservative frictional force, the simulation of the system will
eventually reach a steady state.  The frictional force may be varied during the course of the simulation allowing more
rapid movement at the beginning and only fine adjustments near the end.  This has an analogy of reducing the temperature
of a viscous medium embedding the system.  The strategy is also known as \emph{simulated annealing} for its similarity
to the metallurgic process.  The idea was brought to numeric optimization by \textcite{Kirkpatrick1983}.
\index{simulated annealing}
These strategies can be applied to arbitrary numeric optimization problems but in the context of graph drawing, the
physical analogy is especially illustrative.

The \enum{FMMM} layout uses the \ac{fmmm} implementation found in the \ac{ogdf} at its default settings.  It produces
good-quality layouts and runs very fast even for big graphs.  \ac{fmmm} layouts are very good at \enquote{discovering
  structure} of graphs even in higher dimensions than two such as for graphs from our \enum{TORUS2}, \enum{TORUS3}
(\acs{section}~\ref{sec:torus}) or \enum{BOTTLE} (\acs{section}~\ref{sec:bottle}) generators.  On the other hand,
\ac{fmmm} is not so good at ensuring precise angles and sometimes even tends to \enquote{tie a knot} into graphs that
would actually be planar.  The latter phenomenon is a case of the algorithm's failure to find a global optimum and
getting stuck in a sub-optimal local optimum instead.\footnote{%
  Interestingly, we've found that feeding the algorithm the native, planar, layout as additional information (which we
  normally don't) does not reduce this tendency significantly and sometimes even seems to have an adverse effect.
}
These characteristics are illustrated in \acl{fig}~\ref{fig:fmmm-problems}.

\begin{Figure}
  \begin{center}
    \begin{center}
      \InputTikzGraph*{0.95\linewidth}{pics/force-char-native.tikz}
    \end{center}
    \begin{center}
      \InputTikzGraph*{0.95\linewidth}{pics/force-char-fmmm.tikz}
    \end{center}
    \begin{center}
      \InputTikzGraph*{0.95\linewidth}{pics/force-char-knot.tikz}
    \end{center}
    \begin{center}
      \InputTikzGraph*{0.95\linewidth}{pics/force-char-stress.tikz}
    \end{center}
  \end{center}
  \caption[Characteristics of \enum{FMMM} ans \enum{STRESS} layouts]{%
    Characteristic features of \enum{FMMM} and \enum{STRESS} layouts illustrated.  While \enum{FMMM} is generally good
    at finding a faithful representation of the graph's structure, it doesn't produce very precise angles.  Compare the
    \enum{NATIVE} layout of a regular grid at the top with the force-directed layouts in the middle.  The upper middle
    one is about as good as it gets with \enum{FMMM}.  Occasionally, the algorithm also \enquote{ties a knot} (or
    actually two, in this case) into the graph as can be seen in the layout at the lower middle.  The two middle layouts
    were computed using exactly the same inputs except for a different random seed.  Shown at the very bottom is a
    \enum{STRESS} layout which is reproducible in this quality.  The graph for this example was chosen with guile --
    \enum{FMMM} struggles especially hard with narrow stripes as the graph here where it is actually more likely than
    not to mess up.  \enum{STRESS} layouts are generally not affected by such phenomena.  The layouts shown in this
    \acl{fig} were rotated afterwards in order to align the major principal axis horizontally; the algorithms don't do
    this automagically.
  }
  \label{fig:fmmm-problems}
\end{Figure}

\index{FMMM}
\index{FM3@FM\textsuperscript{3}|see{FMMM}}
The \ac{fmmm} (\acs{ad}~\aca{fmmm})\footnotetext[3]{%
  The superscript \enquote{3} is not a footnote mark but an exponent as in \(\text{M}^3=\text{MMM}\).  Except on this
  page where it just so happens to be a footnote, too.
}
was invented by \textcite{Hachul2005} in \citeyear{Hachul2005}.  We will describe its key aspects in the remainder of
this section in moderate detail.

The energy function used in the \ac{fmmm} is ugly to write down and not actually needed so let us mention its
derivative, the force function, instead.  For a pair of vertices at distance \(d\) there exists a repulsive force
\begin{equation}
  \label{eq:fmmm-repulsion}
  F_\mathrm{node}(d) = -\frac{\alpha}{d}
\end{equation}
and in addition for a pair of connected vertices an attractive force
\begin{equation}
  \label{eq:fmmm-attraction}
  F_\mathrm{edge}(d) = \beta \log\left(\frac{d}{l}\right) d^2
\end{equation}
where \(l\) is the desired edge length (which might be different for different edges) and \(\alpha\) and \(\beta\) are
positive constants.  Note that the force of an edge in \acl{eq}~\ref{eq:fmmm-attraction} can actually become repulsive
if the edge is contracted below its desired length \(l\) causing the result of the logarithm to become negative.
\Aclp*{eq}~\ref{eq:fmmm-repulsion} and \ref{eq:fmmm-attraction} are taken from the original
publication~\cite[\acs{section}~2.1]{Hachul2005}\footnote{%
  The inverse formula \(1/d\) for \(F_\mathrm{node}\) rather than the inverse-square \(1/d^2\) as it would be expected
  from Coulomb's law was taken directly from the original paper~\cite{Hachul2005}.  Given that they repeatedly refer to
  the nodes as \enquote{point charges} and also seem to apply reasoning taken directly from electrostatics, this might
  be a typo in the paper.  However, then again, \(F_\mathrm{edge}\) also doesn't follow Hooke's law and is still
  referred to as a \enquote{spring force}.  (Of course, there are also physical examples like gas springs that don't
  follow Hooke's law either.)
}
but the \ac{fmmm} is actually flexible enough to work with other force models as well and the \ac{ogdf} actually
provides a plugin framework.

When solving the resulting differential \acl{eq} naively, force-directed models quickly become slow for even
moderately-sized graphs because the number of node-node interactions grows quadratic in the size of the graph.  The
major contribution of the \ac{fmmm} is a way to keep the overall runtime of the algorithm sub-quadratic nevertheless.
It achieves this by the combination of two strategies.

\subsubsection{The Multi-Level Approach}

Rather than laying out the whole graph \(\Graph\) at once, a multi-level layout algorithm will use a series of graphs
\(\Graph=\Graph_0,\Graph_1,\ldots,\Graph_k\) with geometrically decreasing sizes chosen such that \(\Graph_k\) is of
constant size.  It will then start by finding a layout for \(\Graph_k\) (which is supposed to be quick) and use that
coarse layout as an aid for finding the layout for \(\Graph_{k-1}\) more quickly and so on until a layout for the
original graph is obtained.

Ideally, the coarse graphs should still be a faithful approximation of the structure of the larger graph.  A seemingly
straight-forward way to achieve this is the repeated contraction of randomly chosen edges, which preserves the graph's
clustering.  Other coarsening strategies have been investigated as well~\cite{Kobourov2013}.  Of course, if the layout
for \(\Graph_i\) should be any good for finding a layout for \(\Graph_{i-1}\), it has to take into account the
additional space requirements for the omitted nodes.  This is why it is important that the desired edge length in
\acl{eq}~\ref{eq:fmmm-attraction} is allowed to depend on the edge even if the edges in the original graph all have unit
weight.

The \ac{fmmm} does not contract random edges but uses a niftier approach of labeling each vertex in the graph
\(\Graph_i\) as either a \enquote{sun}, \enquote{planet} or \enquote{moon} node where the set of sun nodes is built in
an iterative process by choosing a random vertex as sun node and then marking (deleting) all vertices with distance
\(2\) or less.  This is repeated until all vertices are marked (deleted).  The set of planet nodes is then defined as
those vertices that are direct neighbors to a sun node and the remaining vertices are labeled as moon nodes.  This
labeling can be done in linear time.  The next coarse graph \(\Graph_{i+1}\) contains as vertices the \enquote{solar
  systems} of \(\Graph_i\) where a solar system consists of a sun, its adjacent planets and those moons that are closest
to it.  When going into the other direction, the layout for \(\Graph_{i+1}\) is used to determine the coordinates of the
sun nodes in \(\Graph_i\).  The remaining vertices are then initially laid out according to a set of rules that make use
of the information collected during the construction of the solar systems and their positions refined via energy
minimization.

\subsubsection{The Multipole Approach}

The second optimization employed by the \ac{fmmm} is to avoid the computation of the forces between each individual pair
of vertices and instead estimating the force acting upon a node by an approximation that is faster to compute.  In the
\ac{fmmm} only \(F_\mathrm{node}\) is approximated while \(F_\mathrm{edge}\) is computed exactly.  This seems to be a
reasonable trade-off as the complexity of node-node interactions will always grow quadratic in the graph size while the
number of edges had better grow only moderately or a drawing of a large graph that is not sparse will be an unpleasant
mess anyway.

The speedup of the calculation of \(F_\mathrm{node}\) builds upon the superposition principle.  The combined potential
of a number of nodes inside a sphere (which happens to be a circle in two dimensions) acting upon a node \emph{outside}
of this sphere can be described exactly by a function of the distance to the center of the sphere only.  A
straight-forward expansion of this potential function to an infinite power series was presented by
\textcite{Greengard1987} and is used in the \ac{fmmm}.  The derivative of this series (which is easily computed) gives
the desired force.  By evaluating the series only to a constant (the authors suggest to use four) number of terms, a
constant-time approximation for the combined force exercised by all contained nodes can be computed.

At least the \ac{ogdf} implementation of the \ac{fmmm} also seems to use simulated annealing techniques but the original
paper~\cite{Hachul2005} does not mention how this is applied and we did not dwell deep enough into the actual
implementation to find out ourselves.

\subsection{Stress Minimization (\enum{STRESS})}
\label{sec:stress}

\emph{Stress Minimization} as suggested by \textcite{Kamada1989} is another form of force-directed layout with a
special energy function.  For a graph \(\GraphGVE\) with \(\GraphV=\{v_1,\ldots,v_n\}\) for \(n\in\IntsN\), the energy
function for stress minimization is defined as
\begin{equation}
  \label{eq:stress}
  \stress_\Graph(\vec{p}_1,\ldots,\vec{p}_n) = \frac{1}{2}
  \sum_{i=1}^{n-1} \sum_{j=i+1}^{n} \frac{K}{\dist(v_i,v_j)^2}
  \Big(\vecnorm{\vec{p}_i-\vec{p}_j}-l\dist(v_i,v_j)\Big)^2
\end{equation}
where \(K\in\Reals\) is a constant and \(l\in\Reals\) is the desired edge length (which would be \(100\) in our case,
\ac{cf}~\acs{def}~\ref{def:normalized}).  The stress for a given layout \(\Layout\) is therefore
\begin{equation}
  \stress_\Graph(\Layout(\GraphV)) = \frac{1}{2}
  \sum_{i=1}^{n-1} \sum_{j=i+1}^{n} \frac{K}{\dist(v_i,v_j)^2} \Big(\dist_\Layout(v_i,v_j)-l\dist(v_i,v_j)\Big)^2
  \enspace.
\end{equation}
Recall that \(\dist\) is the graph-theoretical distance (length of shortest paths) between vertices
(\acs{cf}~\acs{def}~\ref{def:graphdist}) and \(\dist_\Layout\) is the node distance in layout \(\Layout\)
(\acs{cf}~\acs{def}~\ref{def:nodedist}).  Stress minimization therefore aims to achieve
\(\dist_\Layout(v_i,v_j)\approx{l}\dist(v_i,v_j)\) which is a very interesting and distinctive approach.
\textcite{Kamada1989} originally used a gradient-based (Newton-Raphson) approach to minimize the stress numerically.
The gradient of the stress function was formed analytically rather than using numeric differentiation techniques.

Not only is this approach very intuitive and elegant, it also produces very high-quality layouts that are usually more
regular than those found by the \ac{fmmm} (\acs{cf}~\acs{fig}~\ref{fig:fmmm-problems} and especially
\acs{fig}~\ref{fig:force-comp}).

\begin{Figure}
  \begin{center}
    \begin{tabular}{c@{\qquad}c}
      \InputTikzGraph*{0.4\textwidth}{pics/force-comp-fmmm.tikz}&
      \InputTikzGraph*{0.4\textwidth}{pics/force-comp-stress.tikz}\\[2ex]
      \enum{FMMM} & \enum{STRESS}
    \end{tabular}
  \end{center}
  \input{pics/force-comp.tex}
  \caption[Comparison of \enum{FMMM} versus \enum{STRESS} on a \enum{TORUS3D} graph]{%
    Comparison of the \enum{FMMM} and \enum{STRESS} layouts for a moderately sized (\mbox{\(n=\GraphNodes\)} and
    \mbox{\(m=\GraphEdges\)}) graph generated by \enum{TORUS3D}.  The quality of the \enum{FMMM} is more difficult to
    predict.
  }
  \label{fig:force-comp}
\end{Figure}

Unfortunately, the computation of the \(\stress_\Graph\) function requires an all-pairs shortest path matrix of
\(\Graph\) to be computed upfront.  Usually~\cite{Kamada1989,Kobourov2013}, the Floyd–Warshall
algorithm~\cite{Floyd1962} is suggested for this.  However, since it has complexity \(\BigO(n^3)\) regardless of the
graph and there exist algorithms such as repeated execution of the Dijkstra algorithm that can run in
\(\BigO(nm+n^2\log(n))\) where \(m=\card{E}\) is the number of edges~\cite{Sanders2008}, it seems reasonable to expect
better performance of the latter, given that large graphs are often very sparse.

\textcite{Gansner2005} simplified \acl{eq}~\ref{eq:stress} to obtain a form (\ac{ad}~\enquote{binary
  stress}~\cite{Koren2008}) that is suitable for efficient global optimization using a technique known as
\emph{majorization} which can be solved using numerical linear algebra.  This has desirable properties such as
guaranteed conversion and is also very fast given that the bulk of the workload is made up by matrix computations for
which highly optimized software libraries are abundant.

\textcite{Brandes2007} optimized the procedure even further by avoiding the computation of the full all-pairs shortest
paths matrix and instead rely on probabilistic sampling.

Unfortunately, we don't know what version of stress minimization is implemented in the \ac{ogdf} (and therefore used in
\enum{STRESS}) as the documentation is unclear about this and we did not look at the details of their implementation.

\subsection{Random Placement of Nodes (\enum{RANDOM\_UNIFORM} and \enum{RANDOM\_NORMAL})}
\label{sec:random}

In order to generate \emph{really bad} layouts, we wrote two layout algorithms that assign random coordinates to
vertices.  We have experimented with different probability distributions, namely uniform distributions
(\enum{RANDOM\_UNIFORM}) and normal (Gaussian) distributions (\enum{RANDOM\_NORMAL}).  Both generate independent \(x\)
and \(y\) coordinates in the unit interval or according the standard normal distribution (with \(\mu=0\) and
\(\sigma=1\)) respectively and then normalize (\ac{cf}~\acs{def}~\ref{def:normalized}) the layout afterwards.

Examples of both layouts are given in \acl{fig}~\ref{fig:garbage} which also shows that both random layouts look somehow
\enquote{artificial}.  Therefore, we did not use \enum{RANDOM\_UNIFORM} and \enum{RANDOM\_NORMAL} for our experiments
and instead went ahead implementing a third garbage layout algorithm that is described in the following section.

\begin{Figure}
  \begin{center}
    \setlength{\samplelayoutwidth}{0.18\textwidth}
    \UpdateSampleLayoutHeight{pics/garbage-native.tikz}
    \begin{tabular}{c@{\qquad}c@{\qquad}c@{\qquad}c}
      \InputTikzGraph[execute at begin picture = \AtBeginSampleLayout]
                     {\samplelayoutwidth}{pics/garbage-native.tikz}&
      \InputTikzGraph[execute at begin picture = \AtBeginSampleLayout]
                     {\samplelayoutwidth}{pics/garbage-random-uniform.tikz}&
      \InputTikzGraph[execute at begin picture = \AtBeginSampleLayout]
                     {\samplelayoutwidth}{pics/garbage-random-normal.tikz}&
      \InputTikzGraph[execute at begin picture = \AtBeginSampleLayout]
                     {\samplelayoutwidth}{pics/garbage-phantom.tikz}\\[2ex]
      \enum{NATIVE} & \enum{RANDOM\_UNIFORM} & \enum{RANDOM\_NORMAL} & \enum{PHANTOM}
    \end{tabular}
  \end{center}
  \caption[Comparison of garbage layouts \enum{RANDOM\_UNIFORM}, \enum{RANDOM\_NORMAL} and \enum{PHANTOM}]{%
    Comparison of the garbage layouts.  The picture on the left shows the \enum{NATIVE} layout of the graph (generated
    by \enum{MOSAIC2}).  Sown in the middle are a \enum{RANDOM\_UNIFORM} and \enum{RANDOM\_NORMAL} layout which both
    look \enquote{artificial} in some sense.  The picture on the right shows a \enum{PHANTOM}
    (\acs{section}~\ref{sec:phantom}) layout of the graph which looks more \enquote{reasonable}.
  }
  \label{fig:garbage}
\end{Figure}

\subsection{Phantom Layouts (\enum{PHANTOM})}
\label{sec:phantom}

\begin{Figure}
  \begin{center}
    \setlength{\samplelayoutwidth}{0.25\textwidth}
    \UpdateSampleLayoutHeight{pics/garbage-native.tikz}
    \begin{tabular}{c@{\qquad}c@{\qquad}c}
      \InputTikzGraph[execute at begin picture = \AtBeginSampleLayout]
                     {\samplelayoutwidth}{pics/garbage-native.tikz}&
      \InputTikzGraph[execute at begin picture = \AtBeginSampleLayout]
                     {\samplelayoutwidth}{pics/garbage-phantom.tikz}&
      \InputTikzGraph[execute at begin picture = \AtBeginSampleLayout]
                     {\samplelayoutwidth}{pics/garbage-the-phantom.tikz}\\[2ex]
      (a) & (b) & (c)
    \end{tabular}
  \end{center}
  \caption[The \enum{PHANTOM} layout]{%
    The picture in (a) shows the graph's (which happens to be the same as in \acl{fig}~\ref{fig:garbage}) native layout.
    The picture in (b) shows the \enum{PHANTOM} layout.  The fore-directed layout of the \enquote{phantom} graph that
    was used to generate the layout is shown in picture (c).  Note that the vertex coordinates in (b) and (c) are
    identical.
  }
  \label{fig:phantom}
\end{Figure}

Given that the random placement of nodes did not seem to create realistic examples of poor layouts, we turned to another
strategy that we also ended up using for the final experiment.  Instead of assigning random coordinates to vertices of a
given graph \(\GraphGVE\), the \enum{PHANTOM} layout algorithm first generates a \enquote{phantom} graph
\(\Graph'=(\GraphV,\GraphE')\) with \(\card{\GraphE'}=\card{\GraphE}\).  The generation of this graph is shown in
\acl{algo}~\ref{algo:phantom}.  The \enum{PHANTOM} algorithm then proceeds to compute a force-directed (\acs{fmmm})
layout \(\Layout'\) of \(\Graph'\) and finally outputs the layout \(\Layout\) for the actual graph \(\Graph\) with
\(\Layout(v)=\Layout'(v)\) for each \(v\in\GraphV\).

\begin{Algorithm}
  \begin{extraleading}
    \KwIn{Graph \(\GraphGVE\) with \(\GraphV=\{v_1,\ldots,v_n\}\) and \(\card{\GraphE}=m\).}
    \KwOut{Random simple graph \(\Graph'=(\GraphV,\GraphE')\) with \(\card{\GraphE'}=m\).}
    \Routine{%
      \(\GraphE'\gets\emptyset\)\;
      \While{\(\card{\GraphE'}<m\)}{%
        Choose random \(i,j\in\{1,\ldots,n\}\) according to a uniform distribution\;
        \If{\(i\neq{}j\kern0.3em\land\kern0.3em\{v_i,v_j\}\not\in\GraphE'\)}{%
          Insert \(\{v_i,v_j\}\) into \(\GraphE'\)\;
        }
      }
      \Return\((\GraphV,\GraphE')\)\;
    }
  \end{extraleading}
  \caption[Construction of the \enquote{phantom} graph]{%
    Algorithm for the construction of the \enquote{phantom} graph for the \enum{PHANTOM} layout (see text).  In our
    implementation we actually use the function \code{ogdf::randomSimpleGraph} provided by the \acs*{ogdf}.
  }
  \label{algo:phantom}
\end{Algorithm}

We find that these \enum{PHANTOM} layouts have the desired properties that they look both, terrible and still plausible.
An example \enum{PHANTOM} layout together with the \enquote{phantom} graph is shown in \acl{fig}~\ref{fig:phantom} while
\acl{fig}~\ref{fig:garbage} shows the same layout in comparison with \enum{RANDOM\_UNIFORM} and \enum{RANDOM\_NORMAL}
layouts.

\subsection{Summary}

We have presented and discussed various layout algorithms in order to obtain layouts in different qualities.  The
discussed algorithms were:

\begin{itemize}
\item\enum{NATIVE} (\emph{proper}) --- This is not a layout algorithm (\acs{cf}~\acs{def}~\ref{def:layalgo}) but
  merely a way to refer to layouts that were obtained from external sources such as byproducts from graph generation.
\item\enum{FMMM} (\emph{proper}) --- Uses the \acl*{fmmm} algorithm from \textcite{Hachul2005} implemented in the
  \ac{ogdf}~\cite{OGDF}.  The output is sensitive to the random seed and might vary in quality
  (\acs{cf}~\acs{fig}~\ref{fig:fmmm-problems}).  This algorithm has sub-quadratic complexity.
\item\enum{STRESS} (\emph{proper}) --- Uses the stress minimization~\cite{Kamada1989} algorithm implemented in the
  \ac{ogdf}~\cite{OGDF}.  Unlike \enum{FMMM}, the output of \enum{STRESS} is very predictable and generally of high
  quality but the complexity (which could be as bad a cubic although we don't know exactly what optimizations are
  implemented in the \acs*{ogdf}) can make its application impractical for larger graphs.
\item\enum{RANDOM\_UNIFORM} (\emph{garbage}) --- Assigns random coordinates to each vertex according to a uniform
  distribution over the unit interval.  The layouts look abysmal but don't seem to be realistic examples of bad layouts
  any real layout algorithm might actually produce.
\item\enum{RANDOM\_NORMAL} (\emph{garbage}) --- Like the previous algorithm except that a normal (Gaussian) distribution
  is used instead.  The same remarks apply though to a lesser extent.
\item\enum{PHANTOM} (\emph{garbage}) --- Generates a \enquote{phantom} graph that has the same number of vertices and
  edges and computes a force-directed layout for that graph; then assigns the same coordinates to the vertices in the
  original graph.  We prefer these as the \enquote{best garbage layouts}.
\end{itemize}

Easy ways to obtain even more layouts will be discussed in the following chapter.

\chapter{Data Augmentation}
\label{chap:dataug}

We wish to have a means to derive layouts \(\Layout'\) from existing layouts \(\Layout\) in such a way that we know how
the quality of \(\Layout\) and \(\Layout'\) relate to each other.  To this end, we apply \emph{layout worsening}
(\acs{section}~\ref{sec:lay-worse}) and \emph{layout interpolation} (\acs{section}~\ref{sec:lay-inter}).  The
theoretical considerations for this were explained in \acl{chap}~\ref{chap:method}.  In the remainder of the present
chapter, we will explain how we've achieved this by means of \emph{layout transformations}.

\index{layout!primary}
\index{layout!transformation}
We only apply layout transformations to primary layouts.  A \emph{primary} layout is one that was produced by one of the
layout algorithms described in \acl{section}~\ref{sec:layouts}.  We apply layout transformations to primary layouts
only.  If we also were to apply transformations to transformed layouts again, the process would never stop.

\section{Layout Worsening}
\label{sec:lay-worse}

Improving an existing layout is a difficult challenge.  However, any fool ought to be able to ruin an existing good
layout.  Therefore, we start with a proper layout \(\Layout\) that we assume is of reasonable quality and then apply
some unary transformation to it in order to degrade its quality.  There are several approaches that could be thought of
and we implemented more than one.  The individual \enquote{worseners} we ended up implementing are described in the
remainder of this section.  Recall from \acl{def}~\ref{def:unitrans} that each of them is a probabilistic algorithm
that takes two inputs: the layout \(\Layout\) to degrade and a parameter \(0\leq{r}\leq1\) that controls how much to
degrade it with \(r=0\) implying \(\Layout'=\Layout\) and \(r=1\) being a request to ruin the layout beyond any
restriction.

Each worsener is again a small program that reads the graph and parent layout \(\Layout\) from a \ac{graphml} file and
receives any number of parameters \(r_1,\ldots,r_n\) as command-line arguments.  It will then generate and output
layouts \(\Layout'_{r_1},\ldots,\Layout'_{r_n}\) in \ac{graphml} format again.

\index{configuration files!worsening.cfg@\verb`worsening.cfg`}
The driver script reads a configuration file \verb`worsening.cfg` which lists the desired values of \(r\) for each
worsening strategy.  It will then loop over the primary layouts in the database and check whether the desired worsened
layouts already exist.  Those that don't, it will compute and update the database accordingly.

\subsection{Adding Noise to Coordinates (\enum{PERTURB})}
\label{sec:perturb}
\index{worsener!PERTURB@\enum{PERTURB}}

The most straight-forward way to degrade a given vertex layout is to add some (white) noise to it.  This is exactly what
the \enum{PERTURB} worsener does.  It displaces the coordinates of each node by independently adding some Gaussian
noise.  This can be expressed as a simple formula
\begin{equation}
  \Layout_r'(v) = \Layout(v) + r\Delta^2(v)
\end{equation}
where \(\Delta^2\) is a two-dimensional normal distribution with mean \(\mu=0\) and standard deviation \(\sigma=100\)
(\ac{cf}~\acs{def}~\ref{def:normalized}).  Since this transformation will usually produce a layout that is not
normalized, a normalization step has to be carried out afterwards.

\Acl{fig}~\ref{fig:worsening-perturb} shows an example of the application of the \enum{PERTURB} operation for different
rates of degradation.  This worsening also has a physical interpretation: adding Gaussian noise to the coordinates can
be thought of as increasing the temperature of the system, causing particles to oscillate around their zero position.

\WorseLayoutDemo[PERTURB]{perturb}

\subsection{Flipping Nodes and Edges (\enum{FLIP\_NODES} and \enum{FLIP\_EDGES})}
\label{sec:flip-nodes}
\label{sec:flip-edges}
\index{worsener!FLIP\_NODE@\enum{FLIP\_NODES}}
\index{worsener!FLIP\_EDGE@\enum{FLIP\_EDGES}}

Another strategy towards ruining a layout is pairwise exchange of the coordinates of vertices.  This transformation for
layout \(\Layout\) of graph \(\GraphGVE\) can also be expressed as a simple formula
\begin{equation}
  \Layout_r'(v) = \Layout(\pi_r(v))
\end{equation}
where \((\pi_r)_{0\leq{r}\leq1}\) is a family of appropriate permutations such that \(\pi_r(v)\neq{}v\) with probability
\(r\).  This leads directly to the \enum{FLIP\_NODES} worsener.  An example of its application is shown in
\acl{fig}~\ref{fig:worsening-flip-nodes}.

As can be seen from the example, the effects of flipping arbitrary pairs of nodes are quite dramatic.  The layout
appears completely ruined unless \(r\ll1\).  A smoother effect can be achieved by restricting the permutation \(\pi_r\)
to \(\tau_r\) only flipping the coordinates for pairs of adjacent vertices.  In this case, however, it has to be taken
into consideration that flipping \emph{every} edge does not have the expected devastating effect as it merely
\enquote{rolls over} the graph.  Therefore, the probability for \(\tau_r(v)\neq{}v\) should only be \(r/2\).  This leads
to the \enum{FLIP\_EDGES} algorithm, an example for which is shown in \acl{fig}~\ref{fig:worsening-flip-edges}.

\WorseLayoutDemo[FLIP\_NODES]{flip-nodes}
\WorseLayoutDemo[FLIP\_EDGES]{flip-edges}

\subsection{Affine Deformations using Moving Least Squares (\enum{MOVLSQ})}
\label{sec:movlsq}
\index{worsener!MOVLSQ@\enum{MOVLSQ}}

The layout worsening algorithms described so far all operate locally in the sense that they alter the coordinates of
each vertex or pair of vertices in isolation.  Let us now introduce an algorithm that operates on the layout as a whole.
The idea of this transformation is to deform the drawing area as if the graph were drawn on a sheet of rubber, then
\(k\in\IntsN\) needles were poked into this sheet at random positions and moved to new (random) positions,
\enquote{dragging} the drawing with them as the rubber is deformed.  Note that since our graph drawings -- by definition
-- always draw edges as straight lines, the transformation only \enquote{drags} the coordinates of vertices.  Once the
new coordinates of the vertices are fixed, the edges are again drawn as straight lines between them.  The
\enquote{rubber sheet} analogy is not to be taken literally.  We do not simulate an actual physical deformation process.
Instead, the \enum{MOVLSQ} algorithm proceeds as follows.

Let \(\Layout\) be a layout for the graph \(\GraphGVE\) with \(\card{\GraphV}=n\) and rectangular bounding
box\footnote{%
  The rectangular bounding box \(\Vec{B}\subset\Reals^2\) of a (finite, non-empty) point set \(\Vec{P}\subset\Reals^2\)
  is the minimal axes-aligned convex hull.  It contains all points \((x,y)\in\Reals^2\) such that
  \(x_\mathrm{min}\leq{}x\leq{}x_\mathrm{max}\) and \(y_\mathrm{min}\leq{}y\leq{}y_\mathrm{max}\) where
  \(x_\mathrm{min}=\min\{x\suchthat\exists{}y\in\Reals\suchthat(x,y)\in\Vec{P}\}\) and likewise for the other limits.
}
\(\Vec{B}\subset\Reals^2\).  The \enum{MOVLSQ} worsener picks \(k\in\IntsN\) points \(\vec{p}_1,\ldots,\vec{p}_k\)
selected independently according to a uniform random distribution over \(\Vec{B}\).  These coordinates are referred to
as the transformation's \emph{source control points}.  Another set of points \(\vec{c}_1,\ldots,\vec{c}_k\) is chosen
according to the same distribution and referred to as the \emph{destination control points}.  The parameter \(k\) is
chosen randomly according to a geometric distribution\footnote{%
  A geometric distribution is a discrete probability distribution parameterized by a single parameter \(0<p<1\).  Let
  \(X\) be a Boolean random variable that is true with probability \(p\).  Furthermore, let \(Y\) be a discrete random
  variable that counts the number successive zeros while observing a sequence of realizations of \(X\).  Then \(Y\) is
  distributed according to a geometric distribution with parameter \(p\).  The probability for \(Y=n\) for given
  \(n\in\IntsNz\) is \(p(1-p)^n\).  Consult \textcite{MathWorldGeoDist} for more information.
}
with parameter \(n^{-1/2}\) (see footnote) but ensuring \(k\geq5\).  These (ordered) sets of points are chosen upfront
and can be reused if the transformation is to be performed for multiple values of \(r\).  In order to obtain the
transformed (worsened) layout \(\Layout_r'\) for given value of \(0\leq{r}\leq1\), control points
\(\vec{q}_1,\ldots,\vec{q}_k\) are chosen as \(\vec{q}_i=(1-r)\vec{p}_i+r\vec{c}_i\) for \(i\in\{1,\ldots,k\}\).  The
transformed coordinates of the vertices are then computed according the affine deformation using moving least squares
described by \textcite[\acs{section}~2.1]{Schaefer2006}.

The essential step of this transformation is reproduced in \acl{algo}~\ref{algo:movlsq}.  For the mathematical
background, please refer to the cited source.  An example of the \enum{MOVLSQ}'s application is given in
\acl{fig}~\ref{fig:worsening-movlsq}.

\begin{Algorithm}
  \begin{extraleading}
    \KwIn{Graph \(\GraphGVE\) with layout \(\Layout\) and set of parameters \(\Vec{R}\subset\Reals\).}
    \KwOut{Set of transformed layouts \(\{\Layout'_r\suchthat{}r\in\Vec{R}\}\).}
    \KwConstant{Exponent \(0<\alpha\leq1\).}
    \Routine{%
      Determine rectangular bounding box \(\Vec{B}\subset\Reals^2\) of \(\Layout(\GraphV)\)\;
      Choose \(k\geq5\) according to a geometric distribution with parameter \(n^{-1/2}\)\;
      Choose \(\vec{p}_1,\ldots,\vec{p}_k\) and \(\vec{c}_1,\ldots,\vec{c}_k\) according to a uniform distribution
        over \(\Vec{B}\)\;
      \ForEach{\(r\in\Vec{R}\)}{
        \(\vec{q}_i\gets{}r\vec{c}_i\) for \(i\in\{1,\ldots,k\}\)\;
        \ForEach{\(u\in\GraphV\)}{
          \(\vec{v}\gets\Layout(u)\)\;
          \(w_i\gets\vecnorm{\vec{p}_i-\vec{v}}^{-2\alpha}\) for \(i\in\{1,\ldots,k\}\)\;
          \(\vec{p}^*\gets\sum_{i=1}^{k}w_i\,\vec{p}_i/\sum_{i=1}^{k}w_i\)\;
          \(\vec{q}^*\gets\sum_{i=1}^{k}w_i\,\vec{q}_i/\sum_{i=1}^{k}w_i\)\;
          \(\hat{\vec{p}}_i\gets\vec{p}_i-\vec{p}^*\) for \(i\in\{1,\ldots,k\}\)\;
          \(\hat{\vec{q}}_i\gets\vec{q}_i-\vec{q}^*\) for \(i\in\{1,\ldots,k\}\)\;
          \(\Vec{M}\gets\sum_{i=1}^{k}w_i\ket{\hat{\vec{p}}_i}\bra{\hat{\vec{p}}_i}\)\;
          \(a_i\gets\braket[\Vec{M}^{-1}]{\vec{v}-\vec{p}^*}{w_i\hat{\vec{p}_i}}\)\;
          \(\Layout_r'(u)\gets\vec{q}^*+\sum_{i=1}^{k}a_i\hat{\vec{q}}_i\)\;
        }
      }
    }
  \end{extraleading}
  \caption[Affine Deformation Using Moving Least Squares]{%
    Essence of the affine deformation using moving least squares described by \textcite{Schaefer2006} applied to unary
    layout transformations.  We use the same symbols as in their publication for the sake of easier comparison.  While
    the algorithm might seem laborious, note that all memory can be allocated before the outermost loop is entered.
    From that point on, the algorithm can operate using a constant amount of memory.  Note further that \(\Vec{M}\) is a
    \(2\times2\) matrix and therefore easily invertible using a closed formula.  We fixed \(\alpha=1\) in our code.
  }
  \label{algo:movlsq}
\end{Algorithm}

\WorseLayoutDemo[MOVLSQ]{movlsq}

\subsection{Summary}

We have presented four ways to ruin a good layout by a configurable rate.  The discussed unary layout transformations
were:

\begin{itemize}
\item\enum{PERTURB} --- Adds white (Gaussian) noise independently to the coordinates of each node.  The result is easy
  to control and the complexity is linear.
\item\enum{FLIP\_NODES} --- Swaps coordinates of randomly selected pairs of nodes.  The effect of this is devastating
  even if only a very small fraction of nodes is affected.  The complexity is linear.
\item\enum{FLIP\_EDGES} --- Like \enum{FLIP\_NODES} but restricted to flip only connected pairs of vertices.  The
  complexity is the same but the effect is more controlled.
\item\enum{MOVLSQ} --- Applies affine deformations using moving least squares suggested (although for a different
  purpose) by \textcite{Schaefer2006} to the drawing area in order to move nodes in concert.  Since we choose the number
  of control points proportional to the square root of the graph size and the algorithm has to\footnote{%
    \citeauthor{Schaefer2006} suggests to evaluate the deformation on a regular grid and then interpolate intermediate
    points.  This was in the context of the deformation of raster graphics.  We don't use a regular gird but the
    coordinates of vertices directly.
  }
  loop over the control points in order to find the new position of each node, the complexity is \(\BigO(n^{3/2})\) with
  \(n\) being the number of vertices.
\end{itemize}

\section{Layout Interpolation}
\label{sec:lay-inter}

Given two parent layouts \(\Layout_\mathrm{A}\) and \(\Layout_\mathrm{B}\) of a graph \(\Graph\) we wish to obtain
intermediate layouts \(\Layout'\) for \(\Graph\) that are \enquote{between} the parents \(\Layout_\mathrm{A}\) and
\(\Layout_\mathrm{B}\) for some definition of \enquote{in between}.  We ended up implementing two
\enquote{interpolators} that are described in the remainder of this section.  Recall from \acl{def}~\ref{def:bitrans}
that these a probabilistic algorithms that take three inputs: the layouts \(\Layout_\mathrm{A}\) and
\(\Layout_\mathrm{B}\) to interpolate between and a parameter \(0\leq{r}\leq1\) that controls how much weight to give to
each parent layout with \(r=0\) implying \(\Layout_0'=\Layout_\mathrm{A}\) and \(r=1\) implying
\(\Layout_1'=\Layout_\mathrm{B}\).  As we shall see, we had to make some trade-offs here and can only provide
\(\Layout_0'\cong\Layout_\mathrm{A}\) and \(\Layout_1'\cong\Layout_\mathrm{B}\) for one of the interpolators
(\enum{XLINEAR}) while the other (\enum{LINEAR}) has other problems of its own.  It will be explained later in
\acl{section}~\ref{sec:xlinear} what \enquote{\(\cong\)} means here.

Interpolators are individual programs that read two existing layouts (of the same graph) from \ac{graphml} files and
accept any number of parameters \(r_1,\ldots,r_n\) as command-line arguments.  They will then compute and output
interpolated layouts \(\Layout_{r_1}',\ldots,\Layout_{r_n}'\) in \ac{graphml} format again.

\index{configuration files!interpolation.cfg@\verb`interpolation.cfg`} The driver script reads a configuration file
\verb`interpolation.cfg` that lists the desired values of \(r\) for each interpolation strategy.  It will then loop over
the graphs and primary layouts (\acs{cf}~\acs{section}~\ref{chap:dataug}) in the database as detailed in
\acl{algo}~\ref{algo:lay-inter-driver} and compute missing interpolated layouts.  Finally, it updates the database.

\begin{Algorithm}
  \begin{extraleading}
    \KwConstant{Tolerance \(0<\delta\ll1\)}
    \Routine{%
      \ForEach{interpolation algorithm \(\mathcal{A}\)}{
        Let \(\Vec{R}_\mathrm{want}\) be the set of desired interpolation rates for algorithm \(\mathcal{A}\).\;
        \ForEach{graph \(\Graph\) in the database}{
          Let \(\Layout_1,\ldots,\Layout_n\) be the primary layouts for \(\Graph\) found in the database.\;
          \ForEach{\(i\in\{1,\ldots,n\}\)}{
            \ForEach{\(j\in\{i+1,\ldots,n\}\)}{
              \(\Vec{R}_\mathrm{have}\gets\{r\suchthat\text{layout }\mathcal{A}(\Layout_i,\Layout_j,r)\text{ exists in the database}\}\)\;%
              \(\Vec{R}_\mathrm{need}\gets\{r\in\Vec{R}_\mathrm{want}\suchthat\forall\kern0.25em{r'}\kern-0.05em\in\Vec{R}_\mathrm{have}\suchthat\abs{r'-r}>\delta\}\)\;%
              \If{\(\Vec{R}_\mathrm{need}\neq\emptyset\)}{
                Compute \(\mathcal{A}(\Layout_i,\Layout_j,r)\) for all \(r\in\Vec{R}_\mathrm{need}\)
                in a single invocation.\;
              }
            }
          }
        }
      }
    }
  \end{extraleading}
  \caption[Actions of the driver script for layout interpolation]{%
    Actions of the driver script for layout interpolation.  Note that this procedure assumes that interpolation
    algorithms are symmetric in the sense that
    \(\mathcal{A}(\Layout_i,\Layout_j,r)=\mathcal{A}(\Layout_j,\Layout_i,1-r)\) and that the desired interpolation rates
    are symmetric in the sense that \(r\in\Vec{R}_\mathrm{want}\implies1-r\in\Vec{R}_\mathrm{want}\).  We don't know of
    a good reason to break this assumption.
  }
  \label{algo:lay-inter-driver}
\end{Algorithm}

\subsection{Linear Layout Interpolation (\enum{LINEAR})}
\label{sec:linear}
\index{interpolator!LINEAR@\enum{LINEAR}}

Given two parent layouts \(\Layout_\mathrm{A}\) and \(\Layout_\mathrm{B}\) of a graph \(\GraphGVE\) one could hope to
quickly interpolate an intermediate layout \(\Layout'\) via
\begin{equation}
  \label{eq:lay-inter-linear}
  \Layout_r'(v) = (1-r)\Layout_\mathrm{A}(v) + r\Layout_\mathrm{B}(v)
\end{equation}
for each vertex \(v\in\GraphV\) where \(0\leq{r}\leq1\) is the interpolation parameter.  This leads to the \enum{LINEAR}
interpolation algorithm.

Alas, this naive approach doesn't work too well.  To see this, consider a simple graph with two vertices \(v_1\) and
\(v_2\) connected by an edge.  Now assume that \(\Layout_\mathrm{A}(v_1)=\Layout_\mathrm{B}(v_2)=(-50,+50)\) and
\(\Layout_\mathrm{A}(v_2)=\Layout_\mathrm{B}(v_1)=(+50,-50)\).  The drawings for both layouts will be identical.
However, interpolation according to \acl{eq}~\ref{eq:lay-inter-linear} will yield a degenerated layout
\(\Layout_{1/2}'\) with \(\Layout_{1/2}'(v_1)=\Layout_{1/2}'(v_2)=\vecz\).  This is very unfortunate as it doesn't meet
our expectation at all that the quality of the intermediate layout should be intermediate between the quality of the
parent layouts.  Here, we had two parent layouts of identical good quality and produced an intermediate layout of the
worst quality possible.  Another example of this problem with a bigger graph is shown in
\acl{fig}~\ref{fig:interpol-problem}.

\begin{Figure}
  \begin{center}
    \begin{tabular}{c@{\qquad}c@{\qquad}c}
      \InputTikzGraph{0.25\textwidth}{pics/problem-linear-00000.tikz}&
      \InputTikzGraph{0.25\textwidth}{pics/problem-linear-05000.tikz}&
      \InputTikzGraph{0.25\textwidth}{pics/problem-linear-10000.tikz}\\[2ex]
      \(r=0\) & \(r=1/2\) & \(r=1\)
    \end{tabular}
  \end{center}
  \caption[The problem with \enum{LINEAR} interpolation]{%
    The problem with the naive \enum{LINEAR} interpolation illustrated on a seemingly innocent but pathological set of
    inputs.
  }
  \label{fig:interpol-problem}
\end{Figure}

\begin{Figure}
  \begin{center}
    \begin{tabular}{c@{\qquad}c@{\qquad}c}
      \InputTikzGraph{0.25\textwidth}{pics/problem-xlinear-00000.tikz}&
      \InputTikzGraph{0.25\textwidth}{pics/problem-xlinear-05000.tikz}&
      \InputTikzGraph{0.25\textwidth}{pics/problem-xlinear-10000.tikz}\\[2ex]
      \(r=0\) & \(r=1/2\) & \(r=1\)
    \end{tabular}
  \end{center}
  \caption[The problem mitigated with \enum{XLINEAR} interpolation]{%
    The \enum{XLINEAR} interpolation avoids the pathological cases of the \enum{LINEAR} interpolation by preprocessing
    the parent layouts.  Note how it rotated the parent layouts to align the principal axes.
    (\ac{cf}~\acs{fig}~\ref{fig:interpol-problem}).
  }
  \label{fig:interpol-solution}
\end{Figure}

\subsection{Linear Layout Interpolation with Prearrangement (\enum{XLINEAR})}
\label{sec:xlinear}
\index{interpolator!XLINEAR@\enum{XLINEAR}}

The root cause of the problem with the \enum{LINEAR} interpolation described in the previous section is that it does not
honor the fact that the parent layouts already have a very similar structure.  The pathological cases can be mitigated
by interpolating not between the original layouts \(\Layout_\mathrm{A}\) and \(\Layout_\mathrm{B}\) but instead
preprocess them to obtain layouts \(\hat{\Layout}_\mathrm{A}\) and \(\hat{\Layout}_\mathrm{B}\).  Then linear
interpolation is performed between \(\hat{\Layout}_\mathrm{A}\) and \(\hat{\Layout}_\mathrm{B}\).  Compare
\aclp*{fig}~\ref{fig:interpol-problem} and \ref{fig:interpol-solution} to see how the problem exposed in the former is
fixed in the latter.

Unfortunately, this has the undesirable consequence that \(\Layout_0'=\hat{\Layout}_\mathrm{A}\neq\Layout_\mathrm{A}\)
and \(\Layout_1'=\hat{\Layout}_\mathrm{B}\neq\Layout_\mathrm{B}\) voiding a property of interpolated layouts that we
went to require in the first place.  However, \(\hat{\Layout}_\mathrm{A}\) is sufficiently similar to
\(\Layout_\mathrm{A}\) and likewise \(\hat{\Layout}_\mathrm{B}\) to \(\Layout_\mathrm{B}\) that we may still safely
assume that their aesthetic values are on par (we may write this as \(\Layout\cong\hat{\Layout}\)).

\Acl{fig}~\ref{fig:xlinear-pp} shows the \enum{XLINEAR} interpolation between two proper layouts while
\acl{fig}~\ref{fig:xlinear-pg} shows the \enum{XLINEAR} interpolation between a proper and a garbage layout.  The graph
used in both examples is the same and was taken from the \enum{ROME} collection.

The preprocessed layouts are determined as follows.  Given a layout \(\Layout\) of a graph \(\GraphGVE\) we perform a
principal component analysis to obtain the primary axes \(\vec{c}_1\) and \(\vec{c}_2\) for \(\Layout(\GraphV)\) and the
standard deviations \(\sigma_1\) and \(\sigma_2\) along them.  Writing \(\vec{c}_1\) and \(\vec{c}_2\) as the column
vectors of a \(2\times2\) matrix \(\Vec{C}\) allows us to define
\begin{equation}
  \bar{\Layout} = \Layout \Vec{C}
\end{equation}
and defining the \(2\times2\) matrix
\begin{equation}
  \Vec{\Sigma} = \left(\begin{array}{cc}\sigma_1&0\\0&\sigma_2\end{array}\right)
\end{equation}
allows us to define
\begin{equation}
  \tilde{\Layout} = \bar{\Layout} \Vec{\Sigma}^{-1} \enspace.
\end{equation}
We also introduce the following four auxiliary \(2\times2\) matrices.
\begin{equation}
  \begin{split}
    \Vec{I}_{00} = \left(\begin{array}{cc}+1&0\\0&+1\end{array}\right)\qquad
   &\Vec{I}_{01} = \left(\begin{array}{cc}+1&0\\0&-1\end{array}\right)\\
    \Vec{I}_{10} = \left(\begin{array}{cc}-1&0\\0&+1\end{array}\right)\qquad
   &\Vec{I}_{11} = \left(\begin{array}{cc}-1&0\\0&-1\end{array}\right)
  \end{split}
\end{equation}

Turning back to our parent layouts \(\Layout_\mathrm{A}\) and \(\Layout_\mathrm{B}\) we set
\begin{align}
  &\hat{\Layout}_\mathrm{A} = \bar{\Layout}_\mathrm{A} \qquad\text{and}\\
  &\hat{\Layout}_\mathrm{B} = \bar{\Layout}_\mathrm{B} \Vec{I}^*
\end{align}
where \(\Vec{I}^*\) is chosen among the \(\Vec{I}_{ij}\) such that the quantity
\begin{equation}
  \sum_{v\in\GraphV} \vecnorm{ \tilde{\Layout}_\mathrm{A}(v) - \tilde{\Layout}_\mathrm{B}(v) \Vec{I}^* }^2
\end{equation}
is minimized.

This is deemed sufficient in order to eliminate undesired effects introduced by rotations and inflections of the parent
layouts.  There is still a problem which we don't know how to solve, though.  A pathological case is illustrated in
\acl{fig}~\ref{fig:interpol-remaining}.

\begin{Figure}
  \begin{center}
    \begin{tabular}{c@{\qquad}c@{\qquad}c@{\qquad}c@{\qquad}c}
      \InputTikzGraph{0.13\textwidth}{pics/pathological-xlinear-00000.tikz}&
      \InputTikzGraph{0.13\textwidth}{pics/pathological-xlinear-02500.tikz}&
      \InputTikzGraph{0.13\textwidth}{pics/pathological-xlinear-05000.tikz}&
      \InputTikzGraph{0.13\textwidth}{pics/pathological-xlinear-07500.tikz}&
      \InputTikzGraph{0.13\textwidth}{pics/pathological-xlinear-10000.tikz}\\[1ex]
      \(r=0\,\%\) & \(r=25\,\%\) & \(r=50\,\%\) & \(r=75\,\%\) & \(r=100\,\%\)
    \end{tabular}
  \end{center}
  \caption[A remaining pathological case even for \enum{Linear\_CLEVER} interpolation]{%
    A remaining pathological case even for \enum{XLINEAR} interpolation.  While the interpolation does exactly what it
    is supposed to do, the quality of the intermediate layout is arguably much worse than that of either of its parents.
  }
  \label{fig:interpol-remaining}
\end{Figure}

\begin{Figure}
  \InterLayoutDemoPics{xlinear-pp}
  \caption[\enum{XLINEAR} interpolation between two proper layouts]{%
    \enum{XLINEAR} interpolation between two proper (\enum{FMMM} and \enum{STRESS}) layouts shown at differents steps.
  }
  \label{fig:xlinear-pp}
\end{Figure}

\begin{Figure}
  \InterLayoutDemoPics{xlinear-pg}
  \caption[\enum{XLINEAR} interpolation between a proper and a garbage layout]{%
    \enum{XLINEAR} interpolation between a proper (\enum{FMMM}) and a garbage (\enum{RANDOM\_UNIFORM}) layout shown at
    differents steps.
  }
  \label{fig:xlinear-pg}
\end{Figure}

\subsection{Summary}

We have presented one and a half ways to obtain layouts \(\Layout_r'\) \enquote{in between} two other layouts
\(\Layout_\mathrm{A}\) and \(\Layout_\mathrm{B}\) at a configurable rate \(0<r<1\).  The discussed binary layout
transformations were:

\begin{itemize}
\item\enum{LINEAR} --- Assigns \(\Layout'(v)=(1-r)\Layout_\mathrm{A}(v)+r\Layout_\mathrm{B}(v)\) which is simple and
  efficient but can lead to quirky effects (\acs{cf}~\acs{fig}~\ref{fig:interpol-problem}).  The computational
  complexity is linear.
\item\enum{XLINEAR} --- Attempts to mitigate the problems encountered with \enum{LINEAR} interpolation by trying to
  align the layouts in the most favorable way before applying the interpolation.  This requires a \ac{pca} to be
  performed and voids the property that \(\Layout_{0/1}'=\Layout_\mathrm{A/B}\).  Furthermore, there are still
  pathological cases where it produces undesirable intermediate layouts
  (\acs{cf}~\acs{fig}~\ref{fig:interpol-remaining}).  Its computational complexity is still mostly linear (the
  complexity of \ac{pca} was already discussed in \acs{section}~\ref{sec:princomp}).
\end{itemize}

\chapter{Feature Extraction}
\label{chap:featex}

In \acl{chap}~\ref{chap:syndromes} we have introduced the notion of \emph{properties}.  Those properties are multisets
of scalars that can be computed for a given graph and layout which we believe to be somehow valuable syndromes of the
layout's aesthetic value.  We then explained in chapters~\ref{chap:datgen} and \ref{chap:dataug} how we proceeded in
order to obtain a large corpus of sample data on which we want to try our methods.  In the present chapter, we will
explain our strategies for converting the properties (which are multisets of generally unbounded size) into fixed-size
\emph{feature vectors} that condense the information in a form that is approachable by a discriminator.

Each property introduced in \acl{chap}~\ref{chap:syndromes} is implemented in an individual program that reads a layout
from a \ac{graphml} file and accepts various parameters that select its \latintext{modus operandi} as additional
command-line arguments.  It then computes the requested property and performs further data processing that will be
described in the remainder of this chapter.

\index{iterator@iterator (\CXX)}
In principle, the collection of raw event data and its processing could be split in two phases.  However, we've decided
against this mostly for performance reasons.  For some properties, the amount of data can become very large and,
ideally, we would never want to keep its entirety in memory at any point in time.  On a technical level, this is
elegantly solved without additional overhead using {\CXX} iterators.  However, all our tools also accept a command-line
option that will cause them to not perform any analysis and simply output the raw stream of event data.\footnote{%
  The respective option is \code{--kernel=raw} (or its short form \code{-k raw}).
}
Conversely, we have also written a tool that can read and analyze such a stream.  Separating different stages of the
work-flow into different programs might elegant from a software engineering point of view, however, formatting millions
of floating-point numbers as \acs{ascii} text to send them through a \acs{posix} pipe just to parse them back
immediately afterwards seems to be too much overhead.  Not to mention that some of the analyses we'll present will
require multiple passes so the stream would have to be buffered in memory or written to a file on disk, thereby reducing
time and space efficiency even further.

\index{configuration files!properties.cfg@\verb`properties.cfg`}
The driver script reads a configuration file \verb`properties.cfg` that lists what property to compute for which size
classes of graphs.  Recall from \acl{chap}~\ref{chap:syndromes} that the complexity for computing the different
properties ranges from \(\BigO(n)\) to \(\BigO(n^3)\) so it is not feasible to compute everything for everything unless
one would be willing to consider only small graphs, which we are not.  It then loops over all layouts in the database
and checks which properties still need to be computed for it.  It will then call the aforementioned programs with the
respective arguments in order to compute all missing properties and insert them into the database.

The programs output possibly several text files with the analyzed data (this could be a raw list of events, a histogram
or a sliding average evaluated at a number of support points).  We will explain shortly why multiple outputs are needed.
The text file is in a format approachable by tools like \verb`gnuplot` and actually not needed for the further analysis.
However, it may be viewed in the web front-end as plotted chart in order to manually investigate the data set.  We also
use this output to produce the various plots in this document.

Furthermore, the programs output separate metadata in \ac{json} format which is captured by the driver and stored in a
relational database.  This metadata contains a fixed number of scalars that will then be further used to train and test
the discriminator model that we will introduce in \acl{chap}~\ref{chap:dismod}.

\section{Basic Statistic Properties}

Let us first introduce some very elementary statistic measures.

\index{mean!generalized}
\index{arithmetic mean}
\index{root mean squared}
\index{RMS|see{root mean squared}}
\begin{definition}[Generalized Mean, Arithmetic Mean, Root Mean Squared]
  For a non-empty finite multiset \(\Vec{X}=\multiset{x_1,\ldots,x_n}\subset\Reals\) with \(n\in\IntsN\) the
  \emph{generalized mean} with exponent \(p\in\RealsPos\) of \(\Vec{X}\) is defined as
  \begin{equation}
    \mean_p(\Vec{X}) = \left( \frac{1}{n} \sum_{i=1}^{n} x_i^p \right)^{1/p}
    \enspace.
  \end{equation}
  The special case \(p=1\) is referred to as the \emph{arithmetic mean} while the special case \(p=2\) is called
  \emph{\acf{rms}}.  Notation wise, \(\mean=\mean_1\) and \(\rms=\mean_2\) will be used.
\end{definition}

\index{standard deviation!sample}
\index{standard deviation!population}
\index{stdev@\ensuremath{\stdev}|see{sample standard deviation}}
\index{stdevp@\ensuremath{\stdevp}|see{population standard deviation}}
\begin{definition}[Population {\&} Sample Standard Deviation]
  For a non-empty finite multiset \(\Vec{X}=\multiset{x_1,\ldots,x_n}\subset\Reals\) with \(n\in\IntsN\) the
  \emph{population standard deviation} is defined as
  \begin{equation}
    \label{eq:stdevp}
    \stdevp(\Vec{X}) = \sqrt{\frac{1}{n} \sum_{i=1}^{n} \big(x_i - \mean(\Vec{X})\big)^2}
  \end{equation}
  and the \emph{sample standard deviation} is defined as
  \begin{equation}
    \label{eq:stdevp}
    \stdev(\Vec{X}) = \sqrt{\frac{1}{n-1} \sum_{i=1}^{n} \big(x_i - \mean(\Vec{X})\big)^2}
  \end{equation}
  provided that \(n\geq3\).
\end{definition}

The following relationship between mean and standard deviation of a multiset \(\Vec{X}\) exist.
\begin{align}
  \label{eq:mean-rms-stdev}
  \stdevp(\Vec{X}) &= \sqrt{\rms(\Vec{X})^2 - \mean(\Vec{X})^2}\\[\parskip]
  \stdev(\Vec{X})  &= \sqrt{\frac{\card{\Vec{X}}}{\card{\Vec{X}}-1}\Big(\rms(\Vec{X})^2 - \mean(\Vec{X})^2\Big)}
\end{align}

For our analysis, we compute the arithmetic mean and \ac{rms} for each property which therefore also implicitly captures
the standard deviation without us having to compute it explicitly which is desirable in order to avoid division by zero
for pathologically small data sets.

\section{Histograms}
\label{sec:boxed}

\index{histogram}
\index{histogram!choice of bin width / count}
\emph{Histograms} are a common technique in order to aggregate large amounts of event data into a more approachable form
that allows an estimation of the density distribution.  An important choice to make when building a histogram is the
number of bins.  \Acl{fig}~\ref{fig:histoscale} illustrates the effect of this choice.  There is no shortage of
recommendations how to choose the number of bins -- or, equivalently, the bin width -- for a histogram.
Wikipedia~\cite{WikiHistogram} alone lists eight formulae to guide with this decision.  We have tried several of them
and were able to achieve the least disappointing results using \emph{Scott's normal reference rule}~\cite{Scott1979}
which minimizes the integrated mean squared error of a Gaussian distribution.

\index{Scott's normal reference rule}
\begin{definition}[Scott's Normal Reference Rule]
  Let \(\Vec{X}\in\Reals\) be a multiset with \(\card{\Vec{X}}=n\) and \(\stdev(\Vec{X})=\sigma\).  Then \emph{Scott's
    normal reference rule} suggests to use
  \begin{equation}
    \label{eq:scott-norm-ref}
    h \approx 3.5\:\sigma\:n^{-1/3}
  \end{equation}
  as the histogram bin width.
\end{definition}

Unfortunately, the mathematical property that is optimized by this rule turned out to be of little value for our intents
and purposes.  As the example in \acl{fig}~\ref{fig:histoscale} shows, the bin width chosen according to this rule is
not ideal at all to visualize the structure of our data.  Even though \citeauthor{Scott1979} already acknowledged in his
original paper that \enquote{the data-based algorithm leads to [bin widths] that are generally too big for all our
  models of non-Gaussian data} and suggested that \enquote{a correction factor may be applied} to
\acl{eq}~\ref{eq:scott-norm-ref}, we were unable to find a factor that would be satisfactory at least for the large
majority of our data.  Eventually, we came to the conclusion that the data we're looking at has very sharp features that
are much less common is other statistics domains so the recommendations commonly found in the literature might not be
fully applicable to our problem.  Given an appropriate user interface and some experience, humans can quickly figure out
a good choice of bin width interactively.  However, this is of no help to us either because a requirement for human
intervention would reduce the purpose of our effort \latintext{ad absurdum}.  In the end, we decided to defer the
problem and not settle for any specific histogram width at all.

\begin{Figure}
  \begin{center}
    \begin{tabular}{c@{\qquad}c}
      \InputTikzGraph*{0.45\textwidth}{pics/demograph-x.tikz}&
      \InputTikzGraph*{0.45\textwidth}{pics/demograph-w.tikz}\\[1ex]
      \enum{NATIVE} & \(\menum{NATIVE}+15\,\%~\enum{PERTURB}\)\\[4ex]
      \InputTikzGraph*{0.45\textwidth}{pics/demograph-y.tikz}&
      \InputTikzGraph*{0.45\textwidth}{pics/demograph-z.tikz}\\[1ex]
      \enum{FMMM} & \enum{PHANTOM}
    \end{tabular}
  \end{center}
  \input{pics/demograph-x.tex}
  \caption[Example Graph Layouts]{%
    Example layouts (all of the same graph) that will be used thoughout this chapter.
    The graph was generated using the \enum{MOSAIC2} generator.  It has \(\GraphNodes\) nodes and \(\GraphEdges\) edges.
  }
  \label{fig:demograph-x}
\end{Figure}

\begin{Figure}
  \begin{center}
    \begin{tabular}{c@{\qquad}c@{\qquad}c}
      \InputLuatikzPlot[width=0.275\textwidth]{pics/histoscale-3.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/histoscale-4.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/histoscale-5.pgf}\\[2ex]
      \InputLuatikzPlot[width=0.275\textwidth]{pics/histoscale-6.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/histoscale-7.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/histoscale-8.pgf}\\[2ex]
      \InputLuatikzPlot[width=0.275\textwidth]{pics/histoscale-9.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/histoscale-10.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/histoscale-11.pgf}\\[2ex]
      \InputLuatikzPlot[width=0.275\textwidth]{pics/histoscale-12.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/histoscale-13.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/histoscale-auto.pgf}
    \end{tabular}
  \end{center}
  \caption[Effect of choosing the number of histogram bins]{%
    Each histogram shows the same data (which is the \enum{PRINCOMP1ST} property of the layout shown in
    \acs{fig}~\ref{fig:demograph-x}) but using different bin widths / counts.  The bin count doubles for each histogram
    starting from the top left except for the histogram in the bottom right corner for which Scott's normal reference
    rule was used to determine the bin width.  It can be seen that the histograms in the top row fail to present
    relevant information by using bins that are too wide to show any interesting details.  The histograms in the second
    row give a fairly good impression of the distribution and let the peaks corresponding to the denser square tiles of
    the layout become clearly visible.  However, starting with the third row, the insight gained from the histograms
    diminishes again as there is not enough data for a resolution that fine.  The two histograms in the last row (except
    the one on the right) basically allocate each event into its own bin thereby reducing the value of the histogram
    close to zero.  The number of bins chosen according to Scott's normal reference rule is also not ideal because it is
    too small.
  }
  \label{fig:histoscale}
\end{Figure}

\section{Sliding Averages}
\label{sec:gaussian}

A histogram with constant bin width \(h\in\RealsPos\) for a multiset of events
\(\Vec{X}=\multiset{x_1,\ldots,x_n}\subset\Reals\) with \(n\in\IntsN\) may be formalized as a function
\begin{equation}
  \label{eq:histofunc}
  \begin{aligned}
    H_h:\quad &\Reals &\to\quad     &\RealsNN\\[\parskip]
              &x      &\mapsto\quad &\frac{1}{n} \sum_{i=1}^{n} \delta_{\nint{x/h},\nint{x_i/h}}
    \end{aligned}
\end{equation}
where \(\delta\) is the \emph{Kronecker delta}\footfullcite{MathWorldKronDelta} which is defined as
\begin{equation}
  \delta_{ij} = \begin{cases}0 &i\neq{}j\\1 &i=j\end{cases}
\end{equation}
for \(i,j\in\IntsZ\).  The summation function \(H_h\) may be generalized to use an arbitrary \emph{kernel} (or
\emph{filter}) function \(f:\Reals^2\to\RealsNN\) instead of the Kronecker delta.  This gives the normalized sliding
average
\begin{equation}
  \label{eq:movavgfunc}
  \begin{aligned}
    F_f:\quad &\Reals &\to\quad     &\RealsNN\\[\parskip]
              &x      &\mapsto\quad &\frac{\sum_{i=1}^{n} f(x,x_i)}
                                        {\int_{-\infty}^{+\infty}\diff{y}\sum_{i=1}^{n} f(y,x_i)}
  \end{aligned}
\end{equation}
Which provides a continuous density distribution.  A natural choice for the kernel is the \emph{Gaussian
  function}\footfullcite{MathWorldGaussian} \(g_\sigma\) with \(\sigma\in\RealsPos\) defined as
\begin{equation}
  \label{eq:gaussfilter}
  g_\sigma(\mu,x) = \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{1}{2}\left(\frac{x-\mu}{\sigma}\right)^2}
\end{equation}
where the normalization factor is expendable here because it will be canceled in \(F\) anyway.

Like histograms, sliding averages using Gaussian kernels can be parameterized on the filter width but unlike the
Kronecker delta, the Gaussian kernel will be smooth and not damp local features as aggressively.  Of course, the
question remains how to choose the filter width \(\sigma\).  \Acl{fig}~\ref{fig:glidescale} shows the same data as in
\Acl{fig}~\ref{fig:histoscale} but this time analyzed with a Gaussian filter using as filter width \(\sigma=h/2\) where
\(h\) is the bin width that would otherwise have been used for the histogram.  Interestingly, Scott's normal reference
rule seems to be a reasonable heuristic for choosing a Gaussian filter width for our application.

\begin{Figure}
  \begin{center}
    \begin{tabular}{c@{\qquad}c@{\qquad}c}
      \InputLuatikzPlot[width=0.275\textwidth]{pics/glidescale-3.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/glidescale-4.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/glidescale-5.pgf}\\[2ex]
      \InputLuatikzPlot[width=0.275\textwidth]{pics/glidescale-6.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/glidescale-7.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/glidescale-8.pgf}\\[2ex]
      \InputLuatikzPlot[width=0.275\textwidth]{pics/glidescale-9.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/glidescale-10.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/glidescale-11.pgf}\\[2ex]
      \InputLuatikzPlot[width=0.275\textwidth]{pics/glidescale-12.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/glidescale-13.pgf}&
      \InputLuatikzPlot[width=0.275\textwidth]{pics/glidescale-auto.pgf}
    \end{tabular}
  \end{center}
  \caption[Effect of choosing the width of a Gaussian filter]{%
    The same data as in \acl{fig}~\ref{fig:histoscale} (which is the \enum{PRINCOMP1ST} property of the layout shown in
    \acs{fig}~\ref{fig:demograph-x}) but this time analyzed using a Gaussian filter with \(\sigma=h/2\) where \(h\) is
    the bin width that would otherwise have been used for the histogram.  The filter width halves for each plot starting
    from the top left except for the plot in the bottom right corner for which Scott's normal reference rule was used to
    determine the filter width.  The sliding averages in the top row are still very coarse but, with the exception of
    the very first image, actually capture useful information.  Importantly, so does the image in the bottom right
    corner that had its filter width chosen according to Scott's normal reference rule.  In the second row, the two
    plots to the left and in the middle look very reasonable while the one right already seems too noisy.  The plots in
    the third row barely capture any information any more while the two pictures on the left and in the middle of the
    bottom row are completely worthless.
  }
  \label{fig:glidescale}
\end{Figure}

Unfortunately, however, evaluating sliding averages is much more expensive than creating histograms.  A histogram for
\(n\in\IntsN\) events with \(m\in\IntsN\) bins can be constructed with \(\BigO(n+m)\) effort and after that, querying
for the frequency at any point along the abscissa is an \(\BigO(1)\) operation.  For sliding averages of the same data
using \(m\) support points, on the other hand, the setup is a \(\BigO(nm)\) operation as the weighted sum has to be
computed for each support while every subsequent query of a value can be done in \(\BigO(\log(m))\) via interpolation
between supports (which have to be found via bisection first unless they are distributed equidistantly).  Alas, to make
matters worse, the number of a sliding average's support points for a satisfactory result usually has to be much larger
than the number of bins for an equivalent histogram would be.  The problem could be mitigated if the event data were
\emph{sorted} in which case one could stop the summation in \acl{eq}~\ref{eq:movavgfunc} once the filter drops below a
certain threshold.  This, however, we cannot do either because sorting would require us to compute the entire data set
upfront which we want to avoid for reasons discussed in \acl{section}~\ref{sec:rdf-global}.

One optimization we were able to implement was to use an adaptive strategy for the selection of support points in order
to be able to reduce their number.  The idea is that we first distribute a small fixed number of support points
equidistantly over the data range.  Then we intensify the support points recursively by probabilistic sampling of
additional points in the intervals between existing supports.  Once the newly added points are found to be approximated
to satisfactory accuracy via linear interpolation between the existing enclosing supports, the recursion is stopped.
This procedure causes us to use densely spaced support points in regions where the distribution function has a high
curvature to achieve reasonable approximation while using only loosely spaced support points in regions of low curvature
to be more economic.  An example of this evaluation strategy is shown in \acl{fig}~\ref{fig:adaptive-sampling} and the
algorithm is detailed in \ref{algo:adaptive-sampling}.  Unfortunately, there is a non-zero risk that the initial
sampling process will completely miss a very sharp peak in the function.  Therefore, if a number of support points is
specified on the command-line, our programs will honor it and not use the adaptive strategy.  We have used a fixed
generous hand-tuned number of support points for most plots in this printing to ensure high quality but use the
automatic adaptive strategy in our unattended experiments.

\begin{Figure}
  \begin{center}
    \InputLuatikzPlot[width=0.5\textwidth]{pics/adaptive-sampling.pgf}
  \end{center}
  \caption[Example of Adaptive Sampling]{%
    Example of a function sampled according to our probabilistic adaptive strategy.  The support points are drawn
    explicitly.  It can be seen that their placement is random but nevertheless their density is higher in those regions
    where the function has a high curvature.
  }
  \label{fig:adaptive-sampling}
\end{Figure}

\begin{Algorithm}
  \begin{extraleading}
    \KwIn{Reasonably smooth function \(f:\Reals\to\Reals\) defined on interval \(x_\mathrm{A}<x_\mathrm{B}\).}
    \KwOut{List \(L\) of support points \((x,f(x))\in\Reals^2\).}
    \KwConstant{%
      Initial number of supports \(n\in\IntsN\), splitting factor \(k\in\IntsN\), relative tolerance \(0<\epsilon\ll1\)
      and recursion limit \(r_\mathrm{max}\in\IntsN\).
    }
    \SetKwFunction{Recurse}{Recurse}
    \Routine{%
      Set \(x_i\gets{}x_\mathrm{A}+i(x_\mathrm{B}-x_\mathrm{A})/(n+1)\) for \(i\in\{0,\ldots,n+1\}\)\;
      Set \(y_i\gets{}f(x_i)\) for \(i\in\{0,\ldots,n+1\}\)\;
      \(L\gets\{(x_0,y_0),\ldots,(x_{n+1},y_{n+1})\}\)\;
      \(\delta\gets\epsilon\sum_{i=0}^{n+1}\abs{y_i}/(n+2)\)\;
      \For{\(i\gets0\) {\KwTo} \(n\)}{
        \Recurse{\(L,x_i,y_i,x_{i+1},y_{i+1},\delta,1\)}\;
      }
      Sort \(L\) by the value of the first tuple element\;
    }
    \SubRoutine{\Recurse{\(L,x_\mathrm{a},y_\mathrm{a},x_\mathrm{b},y_\mathrm{b},\delta,r\)}}{
      \comment{%
        Let \((x_0,y_0)\gets(x_\mathrm{a},y_\mathrm{a})\) and \((x_{k+1},y_{k+1})\gets(x_\mathrm{b},y_\mathrm{b})\)
        for convenience
      }
      Choose random values \(0<t_1<\cdots<t_k<1\) independently from the unit interval\;
      Set \(x_i\gets(1-t_i)x_\mathrm{a}+t_{i}x_\mathrm{b}\) for \(i\in\{1,\ldots,k\}\)\;
      Set \(y_i\gets{}f(x_i)\) for \(i\in\{1,\ldots,k\}\)\;
      Append \((x_i,y_i)\) to \(L\) for \(i\in\{1,\ldots,k\}\)\;
      Set \(z_i\gets(1-t_i)y_\mathrm{a}+t_{i}y_\mathrm{b}\) for \(i\in\{1,\ldots,k\}\)\;
      \If{\(r<r_\mathrm{max}\) {\KwAnd} \(\max_{i=1}^{k}\{\abs{y_i-z_i}\}>\delta\)}{
        \For{\(i\gets0\) {\KwTo} \(k\)}{
          \Recurse{\(x_i,y_i,x_{i+1},y_{i+1},\delta,r+1\)}\;
        }
      }
    }
  \end{extraleading}
  \caption[Adaptive Sampling Algorithm]{%
    The adaptive sampling algorithm that we used for evaluating sliding averages more quickly (\acs{ie}~less slowly).
    We have chose \(n=17\), \(k=2\), \(r_\mathrm{max}=10\) and \(\epsilon=1/20\) for our code.
  }
  \label{algo:adaptive-sampling}
\end{Algorithm}

\section{Entropy}
\label{sec:entropy}
\index{entropy}

In many of the examples we introduced in \acl{chap}~\ref{chap:syndromes}, we mentioned that a distinguishing property of
regular versus not-so-regular layouts is that the former have a less uniform distribution of the various properties.  We
wish to capture this information.  \emph{Information Entropy}, pioneered by the work of \textcite{Shannon1948}, provides
a measure for this that is conveniently expressed as a single scalar value.

\index{histogram!entropy}
\begin{definition}[Entropy of Histogram]
  Let \(\Vec{H}\) be a histogram with \(n\in\IntsN\) bins that have the values (relative frequency counts)
  \(H_1,\ldots,H_2\in\RealsNN\) such that \(\sum_{i=1}^{n}H_i=1\).  Then the entropy of \(\Vec{H}\) is
  \begin{equation}
    \label{eq:discrete-entropy}
    \Entropy(\Vec{H}) = -\sum_{i=1}^{n} H_i \log_2(H_i)
  \end{equation}
  where we use the convention that bins with \(H_i=0\) shall contribute a zero term to the sum.\footnote{%
    Because the value of \enquote{\(0\cdot\log_2(0)\)} is undefined.
  }
  \label{def:entropy-histogram}
\end{definition}

Unfortunately, the entropy according to \acl{def}~\ref{def:entropy-histogram} is highly dependent on the choice of bin
width which we have just given up to determine reliably.  In order to still get a measure of predictability of the
distribution, we therefore don't consider the entropy of any single histogram but of a \emph{series} of histograms with
changing bin widths.  We have found that the entropy as a function of the logarithm of the bin count can usually be
approximated very closely by a linear regression.  This allows us to use the parameters (intercept and slope) of the
regression function instead of the entropy of any particular histogram which alleviates us from the problem that we
don't know how to choose a good bin width automatically.  Yet, the regression curves for different layouts show a good
amount of variability which looks promising.  \Acl{fig}~\ref{fig:entropy-regression-histo} shows a few examples of such
regressions.

\begin{Figure}
  \begin{center}
    \InputLuatikzPlot[width=\textwidth]{pics/entropy-regression.pgf}
  \end{center}
  \caption[Linear Regression of Histogram Entropy]{%
    Histogram entropy for the \enum{PRINCOMP1ST} property computed for the four layouts shown in
    \acl{fig}~\ref{fig:demograph-x} plotted as a function of the logarithm of the histogram bin count.  It can be
    clearly seen from the plot that the linear regression fits the data very well.  Another interesting observation to
    make is that the bin widths chosen according to Scott's normal reference rule (which are easily identifiable in the
    lower left region as they are the only data points with a bin count that is not a power of two) all yield
    approximately the same entropy for the histogram.  While this is a remarkable property \latintext{per se}, it is
    unfortunately not helpful for us and lead us to abandon this heuristic altogether.  Speaking of the data itself, it
    is apparent that the \enum{PHANTOM} layout stands out.  The \enum{NATIVE} layout and its worsened companion
    are almost indistinguishable while the \enum{FMMM} layout is closer to them than to the \enum{PHANTOM} layout.
    The most surprising observation, however, is the fact that the entropy for the \enum{PHANOM} layout is actually the
    \emph{lowest} which is unexpected even if taking into consideration that the construction of this layout does not
    assign random coordinates to vertices (\acs{cf}~\acs{section}~\ref{sec:phantom}).
  }
  \label{fig:entropy-regression-histo}
\end{Figure}

\index{sliding average!entropy}
It might be tempting to generalize the entropy definition from \acl{def}~\ref{def:entropy-histogram} to continuous data
in order to apply it to sliding averages as well.  This leads to the following definition.

\index{entropy!differential}
\begin{definition}[Differential Entropy]
  Let \(f:\Reals\to\RealsNN\) be a non-negative steady function normalized such that
  \(\int_{-\infty}^{+\infty}\diff{x}f(x)=1\).  The \emph{differential entropy} of \(f\) is defined as
  \begin{equation}
    \Entropy*(f) = -\int_{-\infty}^{+\infty} \diff{x} x\log_2(x)
    \label{eq:differential-entropy}
  \end{equation}
  where we use the convention that the integrand shall be zero for those \(x\in\Reals\) where \(f(x)=0\).
\end{definition}

While this definition of differential entropy was actually proposed by \citeauthor*{Shannon1948} himself in his original
paper~\cite{Shannon1948}, it has been argued~\cite{Jaynes1963} since that \acl{eq}~\ref{eq:differential-entropy} is
\emph{not} a correct information measure and does not provide the continuous analog of
\acl{eq}~\ref{eq:discrete-entropy} for the limit \(n\to\infty\).  An obvious observation to make is, for example, that
\(\Entropy*(f)\) according to \acl{eq}~\ref{eq:differential-entropy} may even become negative, such as for the uniform
distribution over an interval narrower than \(1\).  On the other hand, differential entropy has been studied for various
distribution functions~\cite{Lazo1978} even if, unlike discrete entropy, it might not have a straight-forward
interpretation.  Furthermore, it can be shown~\cite[\acs{thm}~9.3.1, \acs{eq}~9.30]{Cover1991} that if a Riemann
integrable density function \(f\) is sampled into a histogram \(\Vec{H}_\Delta\) with bin width \(\Delta\in\RealsPos\)
then
\begin{equation}
  \lim_{\Delta\to0} \Entropy(\Vec{H}_\Delta) + \log_2(\Delta) = \Entropy*(f)
  \enspace.
\end{equation}
Given that no better alternative was available, we decided to simply use differential entropy of sliding averages
without further ado.

\section{Special Considerations for Local \acs*{rdf}}

The \enum{RDF\_LOCAL} property (\acs{section}~\ref{sec:rdf-local}) is the only property that is already parameterized in
its own right.  We might use this in order to perform some analysis.  \Acl{fig}~\ref{fig:diffent-rdf-local} shows the
differential entropy of the sliding average of \(\menum{RDF\_LOCAL}(d)\) as a function of \(\log_2(d)\).  We decided to
not use this for a regression, however, because there is no theoretical consideration that would justify this.  Instead,
we put the entropy values for each value of \(d\) directly into the feature vector (which therefore becomes considerably
larger).  We use the differential entropy of the sliding average rather than using the regression technique to get
a linear entropy function in order to avoid having two variables that need to be varied (and therefore make the
computation even costlier).

\begin{Figure}
  \begin{center}
    \InputLuatikzPlot[width=\textwidth]{pics/diffent-rdf-local.pgf}
  \end{center}
  \caption[Differential Entropy of \enum{RDF\_LOCAL}]{%
    Differential entropies of the density functions obtained via computing sliding averages (with a Gaussian filter
    width chosen according to Scott's normal reference rule) for the \(\menum{RDF\_LOCAL}(d)\) properties for for the
    four layouts shown in \acl{fig}~\ref{fig:demograph-x} plotted as a function of \(\log_2(d)\).  The first thing to
    notice is that -- unlike pointed out for the entropy of histograms -- using Scott's normal reference rule for the
    selection of the filter width does not seem to predetermine the entropy value.  (Note that the ordinate does not
    start at zero, though.)  The distribution for the \enum{PHANTOM} layout clearly stands out.  It meets the
    expectation that its value is mostly independent of the parameter \(d\) as the layout does not make use of the
    graph's structure at all.  The data points for the \enum{NATIVE} and \enum{FMMM} layout are almost identical with a
    smaller difference to the worsened layout, which is all as expected.  The linear regression curves for all layouts
    but \enum{PHANTOM} do not approximate the data very well.
  }
  \label{fig:diffent-rdf-local}
\end{Figure}

\section{Other Data}

There are a few more bits of information that didn't fit in any of the previous discussion.  If we have a \ac{pca}
available, we also want to capture the orientation of the layout, which might be important (see
\acs{fig}~\ref{fig:vertical} and \textcite{Giannouli2013}).  For this, we also include the coordinates of the principal
axes into the feature vector (which are two values per component).

Finally, we also want to capture the size of the graph itself, which might have an influence on the importance and
reliability of some properties.  For this purpose, we also put the \emph{logarithm} of the number of vertices and edges
into the feature vector.  This also captures the sparsity implicitly.  It might be worthwhile to include other
information (such as the diameter) as well but we did not look into this.

\section{Summary}

We have introduced mean and \ac{rms} and discussed the difficulties of finding good bin widths for histograms and
presented our solution of using a linear regression of the entropy as a function of the logarithm of the bin count.  We
also introduced sliding averages and differential entropy.  \(\menum{RDF\_LOCAL}(d)\) is a special case as it is already
parameterized.  All in all, our feature vector for a layout has the following entries, which happen to sum up to \(58\)
given that we compute \(\menum{RDF\_LOCAL}(2^i)\) for \(i\in\{0,\ldots,9\}\) which was chosen like this because none of
our graphs had a diameter in excess of \(2^9=512\).

\begin{itemize}
\item Mean and \ac{rms} for each property.
\item Slope and intercept of the linear regression function found for histogram entropy as a function of the bin count
  for all properties except \enum{RDF\_LOCAL}.
\item Differential entropy for \enum{RDF\_LOCAL}.
\item Principal components (four scalar values).
\end{itemize}

For the corresponding graph, the feature vector only contains two entries which are
\begin{itemize}
\item the logarithm of the number of vertices and
\item the logarithm of the number of edges.
\end{itemize}

If any entry in the feature vector is not available -- maybe because the computation timed out or because something is
mathematically undefined for a pathological instance, such as a graph with no edges -- we record a special
\enquote{null} value that will later be dealt with.

On a technical level, the feature vector is implemented by dynamically creating a \emph{view} in the \acs{sql} database
which pulls in all needed data and is keyed by the layout \acs{id}.

In the next chapter, we will discuss how the extracted features are used to train and test an automatic discriminator
model.

\chapter{Discriminator Model}
\label{chap:dismod}

\index{discriminator}
Using the condensed information collected in the feature vector we would like to train a neural network that predicts an
ordering relation between the aesthetic value of two layouts.  In this chapter we will explain the structure of the
neural network that we have chosen and explain the training and testing of it.  Since it discriminates between two
layouts, we refer to it as a \emph{discriminator} (in order to distinguish it from, say, a classifier).

At the user level, our system allows to specify two layouts by their \acsp{id} \(i\) and \(j\) and outputs a prediction
\(-1\leq{p}\leq+1\) for the aesthetic preference between the layouts \(\Layout_i\) and \(\Layout_j\) where a value of
\(p<0\) or \(p>0\) is a result in favor of \(\Layout_i\) or \(\Layout_j\) respectively and \(\abs{p}\) is a measure of
the discriminator's certainty about its prediction.  It is currently not possible to ask the discriminator about layouts
that are not already saved in the internal database.  This is an obvious annoyance and we are thinking about providing a
more convenient interface in the future.  Specifying layout \acsp{id} that belong to different graphs is an error.

We provide a command-line tool and a web front-end to interact with the discriminator.  The command-line tool would be
used like this
\begin{lstlisting}
$ compare 0f76aebc 33eacef0
0f76aebcfafc504fa057b9d1f39955fb 33eacef0264f234f10835b1427e58383   +0.13997
33eacef0264f234f10835b1427e58383 0f76aebcfafc504fa057b9d1f39955fb   -0.07941
\end{lstlisting}
where the user asked for a prediction concerning the layouts with the unambiguous \acs{id} prefixes \verb`0f76aebc` and
\verb`33eacef0` and the tool outputted the full \acsp{id} next to the numbers \(+0.13997\) and \(-0.07941\) indicating a
preference of \(+14\,\%\) in favor of the second layout and \(-8\,\%\) if the layouts are compared in reversed order.
We will discuss shortly why the numbers are not exactly symmetric.  An impression of the web \acs{ui} is shown in
\acl{fig}~\ref{fig:compare-web-ui}.

\begin{Figure}
  \begin{center}
    \pgfimage[width=\textwidth]{pics/webui-compare.png}
  \end{center}
  \caption[Comparing Layouts in the Web UI]{%
    Screenshots of the web \acs{ui} for querying the discriminator about its aesthetic judgment.  The filled and
    outlined triangle indicates the result in forward and reverse order respectively.  The user can also click on either
    layout in order to view detailed information about it.  Clicking on the \enquote{feature vectors} link will show a
    table which details the deviation of each feature of the two layouts from the average over all layouts in the
    database as well as the difference between the two feature vectors.
  }
  \label{fig:compare-web-ui}
\end{Figure}

\section{Siamese Neural Network Structure}
\index{neural network}
\index{Siamese neural network}

Because the discrimination is an inherently binary task, we decided to use a \emph{Siamese neural network} which is a
structure originally proposed by \textcite{Bromley1994}.  A network using this structure consists of two identical
sub-networks (referred to as \emph{shared model} from now on) that process the feature vector of the left-hand and
right-hand input respectively.  The output vectors of these sub-networks are then subtracted from each other and the
difference is passed to a third, independent, sub-network that reduces this information into a single scalar quantity,
which gives the predicted preference.

The last sub-network also receives the feature vector of the graph.  The structure of the network is shown in
\acl{fig}~\ref{fig:nn-structure}.  If the shared model had an output layer with only a single value, this architecture
would correspond to the computation of a single quantity that describes aesthetic value in an absolute sense.  However,
as we have already discussed, we don't believe that this is a viable approach.  Therefore, we keep several dimensions in
the output of the shared model which can reflect different aspects of each of the layouts and allow the third
sub-network to make a final decision.  The turn side of this approach is that we cannot guarantee through the network's
structure that the discriminator will be symmetric in the sense that swapping the left and right hand input will invert
the sign of the output only.  As we have seen in the example at the beginning of this chapter, the output differs by a
few percent if the inputs are swapped.  We recommend that if the model were to be used in production, the discriminator
should be run for both combinations of the inputs and the average of both outputs should be returned in order to enforce
symmetry again.  However, since we are more interested in studying the discriminator's behavior rather than actually
using it, we omitted this trivial step and instead show the outputs for both combinations individually.

The shared model consists of two dense layers.  The first layer (obviously) has as many inputs as the size of our
feature vector for layouts (which is \(58\)).  Its output dimension, and therefore the input dimension for the second
layer, was set to \(10\) mostly out of a gut feeling and after a little bit of experimentation to verify that increasing
the number of dimensions does not improve the success rate.  The output of the hidden layer also has \(10\) dimensions.
This leads to \(590+110=700\) trainable parameters for the shared model.

The additional feature vector with information specific to the \emph{graph} rather than either layout is first passed
through an auxiliary dense layer.  This is probably not very useful (although it cannot do much harm either) at the
moment because the feature vector only has size \(2\) and the auxiliary layer also has output dimension \(2\) as it
would make no sense to use even less.  However, if more entries shall be added to the graph's feature vector in the
future, this auxiliary layer may help reduce the dimensionality of this additional information before it is given to the
final layer.

Said final layer is also a dense layer and receives the difference of the outputs of the shared model concatenated with
the output of the auxiliary layer and therefore has \(12\) inputs.  Its output dimension obviously had better be \(1\).

The auxiliary layer adds \(6\) and the final layer \(13\) trainable parameters which gives a total of \(700+13+6=719\)
trainable parameters for the entire network.

The network was built using the \Keras~\cite{Keras} framework with the \TensorFlow~\cite{TensorFlow} library as
back-end.  Unless noted otherwise in this writing, the default values provided by the library were used.  Not having
designed a neural network before, we found the recommendations in \textcite{LeCun1998} invaluable.

\begin{Figure}
  \begin{center}
    \begin{tabular}{cc}
      \pgfimage[height=5cm]{pics/nn-structure-total.pdf}&
      \pgfimage[height=5cm]{pics/nn-structure-shared.pdf}\\
      (a) & (b)
    \end{tabular}
  \end{center}
  \caption[Structure of the Siamese Neural Network]{%
    Structure diagram of the discriminator's neural network as output by the debugging feature of the {\Keras} library.
    The diagram in (a) shows the overview of the entire model while the picture in (b) shows the internals of the shared
    sub-model -- shown as the \code{Model} layer in (a).  Only \code{Dense} layers can be trained.  The \code{Subtract}
    and \code{Concatenate} layers do what you think they do; they have no parameters.  An \code{InputLayer} in {\Keras}
    parlance is merely a way to refer to parameters; it does not compute anything and has no parameters.  The
    significance of the \code{Dropout} layers is discussed in \acl{section}~\ref{sec:regulation}.
  }
  \label{fig:nn-structure}
\end{Figure}

\section{Regulation}
\label{sec:regulation}
\index{neural network!regulation}
\index{dropout}
\index{overfitting}

In order to prevent overfitting of the model and harden it against missing input values, \emph{dropout} is used.
Dropout as a regularization technique was proposed by \textcite{Srivastava2014}.  Please refer to their text for
motivation and details.

For this purpose, a dropout layer is added before each of the dense layers in the shared model that discards \(50\,\%\)
and \(25\,\%\) of the signals respectively during the training of the network.

For the dropout rate before the first layer, a large value of \(50\,\%\) was chosen in order to make the network more
robust against missing data which occurs frequently in our inputs.  A lower dropout rate of \(25\,\%\) before the second
layer was chosen because the effect of missing inputs will already have been distributed by the first layer.

\section{Metaparameters}

\subsection{Activation}
\index{activation function}
\index{neuron!activation|see{activation function}}

A linear activation function for the first and auxiliary layer was chosen because we found this to be most robust with
respect to inputs that are out of range after experimenting a bit with other functions and repetitively facing numeric
overflow issues.  The hidden layer uses the common and efficient \ac{relu} activation while a hyperbolic tangent was
chosen as the activation function for the final layer because its output range and characteristics match exactly our
requirements.

\subsection{Initialization, Loss and Optimization}
\index{neuron!initialization}
\index{loss function}
\index{neural network!optimization}

Neuron weights are initialized by setting the bias to zero and kernel to a random value drawn according to a truncated
normal distribution with \(\mu=0\), \(\sigma=1/20\) and truncation at \(\pm2\sigma\).

A \emph{\acf{mse}} loss function and \acf{sgd} optimizer are used.

\section{Normalization}
\index{normalization!of neural network inputs}

Once the model is set up, the database is queried for all available data.  Please refer to \acl{chap}~\ref{chap:method}
for the reasoning behind and explanation of the data available for training and testing.  In order to collect it into a
single huge vector, the driver script finds all of the following triples for each graph in the database.
\begin{itemize}
\item\(\big(\Layout,\Layout',-1\big)\) where \(\Layout\) is a proper and \(\Layout'\) is a garbage layout.
\item\(\big(\mathcal{W}(\Layout,r_i),\mathcal{W}(\Layout,r_j),t_{ij}\big)\) for \(r_i\neq{}r_j\) where \(\Layout\) is a
  proper layout and \(\mathcal{W}\) is a layout worsening algorithm.  The expectation is set to
  \(t_{ij}=(r_i-r_j)/r_\mathrm{max}\) where \(0<r_\mathrm{max}\leq1\) is the maximum worsening rate ever used with
  algorithm \(\mathcal{W}\) for any layout.  Note that \(\mathcal{W}(\Layout,0)=\Layout\) is always available.
\item\(\big(\mathcal{I}(\Layout,\Layout',r_i),\mathcal{I}(\Layout,\Layout',r_j),t_{ij}\big)\) for \(r_i\neq{}r_j\) where
  \(\Layout\) is a proper layout, \(\Layout'\) is a garbage layout and \(\mathcal{I}\) is a layout interpolation
  algorithm.  The expectation is set to \(t_{ij}=r_i-r_j\).
\end{itemize}
The same pair of layouts is only used once.  That is, if \((\Layout_\mathrm{A},\Layout_\mathrm{B},t)\) is already used,
we don't use \((\Layout_\mathrm{B},\Layout_\mathrm{A},-t)\) too.  What becomes the \enquote{left} and \enquote{right}
layout is chosen randomly in order to ensure that \(t\) is unbiased.  The normalization factor \(r_\mathrm{max}\) for
worsened layouts was introduced because not all worsenings are equally dramatic so this leads to more evenly distributed
data.

After the list of layout combinations has been determined, the feature vectors for each of them has to be found.  The
values can be looked up directly in the database but before they are given to the network, they get \emph{normalized}
(\acs{cf}~\textcite{LeCun1998}).

Suppose that a feature vector has \(n\in\IntsN\) entries and that there are \(m\in\IntsN\) layouts in the corpus of
training and testing data.  Let \(v_{ij}\) be the value of the \(j\)-th element in the feature vector of the \(i\)-th
layout.  The normalized feature vector for this layout will contains the value
\begin{equation}
  \hat{v}_{ij} =
  \begin{cases}
    (v_{ij}-\mean(\Vec{V}_j)) / \stdev(\Vec{V}_j)  &v_{ij}\neq\bot\\
    \mean(\Vec{V}_j)                               &v_{ij}=\bot
  \end{cases}
\end{equation}
where \(\bot\) denotes a missing value and
\(\Vec{V}_j=\multiset{v_{kj}\suchthat{}1\leq{}k\leq{}n\suchthat{}v_{kj}\neq\bot}\) is the multiset containing all
well-defined \(j\)-th elements of all feature vectors in the corpus.

This normalization step ensures that the inputs to the neural network have zero mean and unit standard deviation and
that missing values represent no bias in either direction.

\section{Training and Testing}
\label{sec:training-testing}
\index{neural network!training}
\index{neural network!testing}
\index{validation split}
\index{epoch}

We used test corpora with more than \(10\,\mathrm{k}\) and less than \(100\,\mathrm{k}\) labeled layout pairs, limited
primarily by our willingness and ability to spend more computational resources.  There are no arbitrary restrictions or
assumptions in our setup that would limit the amount of data.  No thorough investigation was performed, though.
Repetition of our experiments is highly encouraged.  The fellow researcher is invited to download and run our setup.
Instructions how to do this are given in the appendix of this work.

Given the list of inputs and expected outputs prepared as described in the previous section, this list is shuffled
randomly such that the model won't see the data in any particular order.  \(20\,\%\) of this data is kept aside for the
purpose of testing the model once training is complete.  The remaining data is used for training the model over \(100\)
epochs using a validation split of \(25\,\%\).

\index{success}
\index{failure}
Once training has completed, the \(20\,\%\) of the original data previously set aside -- which the network has never
seen up to this point -- are used for testing.  That is, the model is queried for a prediction \(p\) for each pair
\((\Layout_\mathrm{A},\Layout_\mathrm{B})\) of the test triples \((\Layout_\mathrm{A},\Layout_\mathrm{B},t)\) and then
\(p\) is compared to \(t\).  A test is considered a success if and only if \(\sign(p)=\sign(t)\).  A reproducible
success rate above \(95\,\%\) could be achieved.  We will have a more detailed look at the actual test results in
\acl{section}~\ref{sec:eval-accuracy}.  A similar interface to that shown in \acl{fig}~\ref{fig:compare-web-ui} is
provided for manual inspection of the test cases.

\chapter{Evaluation}
\label{chap:eval}

Due to time constraints, we were only able to perform a very limited amount of evaluation of our discriminator model.
The results we did obtain are presented in the current chapter.  More work is clearly needed and we hope that the setup
we've provided is a good starting point for this.

\section{Accuracy}
\label{sec:eval-accuracy}

\index{cross validation}
We used \emph{cross validation} via \emph{random subsampling}~\cite{Kohavi1995} in order to verify the reliability of
our model.  As already mentioned in \acl{section}~\ref{sec:training-testing} we set a randomly chosen partition of
\(20\,\%\) of the data corpus is aside for testing and not use it for training.  For the purpose of cross validation via
random subsampling, a constant \(k\in\IntsN\) is chosen and the training and testing sequence is repeated \(k\) times --
each time with a newly partitioned data corpus.  By collecting the results of each run, not only can we estimate the
reliability of our discriminator but also the reliability of this estimation.  That is, we can provide a confusion
matrix with errors.  This is shown in \acl{tab}~\ref{tab:cross-valid}.

\begin{Table}
  \begin{center}
    \InputConfusionMatrix{eval-cross-valid.tex}
  \end{center}
  \caption[Cross Validation Confusion Matrix]{%
    Confusion matrix with errors obtained through cross validation via random subsampling and some additional
    information.
  }
  \label{tab:cross-valid}
\end{Table}

\section{Contribution of Individual Properties}

In order to assess the contribution that each property brings information wise, we've set up an experiment where we
re-run the cross validation but either exclude a specific property from the feature vector or exclude all other
properties and use only this single property.  The results are shown in \acl{tab}~\ref{tab:without}.

\enum{RDF\_LOCAL} stands out as the clear winner of this comparison followed by \enum{RDF\_GLOBAL}.  It is a little
unfair, though, as \enum{RDF\_LOCAL} is not a single property but a group of properties so it is expected to contribute
more.  The difference is still pretty dramatic, though.  Using \enum{RDF\_LOCAL} alone still yields a good result but
significantly less than if all properties are used together.  It should be noted, however, that this view is misleading
as there are graphs for which we cannot compute \enum{RDF\_LOCAL} as they are too big.  For those, the discriminator can
only guess.  In a more carefully set up experiment, such graphs would have to be excluded.  It is also not clear from
this experiment alone whether some properties are just plain useless or only have too much overlap with each other to
make the exclusion of either of them insignificant.  A more rigorous statistical analysis would be required to answer
this question reliably.

\begin{Table}
  \begin{center}
    \InputWithoutResult[eval-cross-valid.tex]{eval-without.tex}
  \end{center}
  \caption[Cross Validation With Reduced Feature Vector]{%
    Cross validation results (success rates) with one property deliberately excluded (middle column) and only a single
    property included (right column) respectively.  Rows are sorted by the middle column.  It is immediately visible
    that \enum{RDF\_LOCAL} has by far the biggest contribution followed by \enum{RDF\_GLOBAL} (which doesn't add much
    when used \emph{together} with \enum{RDF\_LOCAL} for apparent reasons but is almost as effective when used alone).
    \enum{TENSION} and \enum{ANGULAR} seem to contribute something while the \enum{PRINCOMP1ST}, \enum{PRINCOMP2ND} and
    \enum{EDGE\_LENGTH} appear close to useless.  It could be that there is much overlap between the properties except
    \acs{rdf} such that omitting any of them alone has little effect.  The low contribution of \enum{PRINCOMP2ND} is not
    very surprising.  Please see the text for additional remarks.  It should be noted that we used only \(5\) cross
    validation runs to obtain the entries in this table.
  }
  \label{tab:without}
\end{Table}

\section{Performance}

Our implementation was not designed with maximum performance in mind but rather guided by the desire to provide a
flexible framework to which new tools can be added easily.  Therefore, we do not present an analysis of the system's
performance at this point.  Practically speaking, most of the time is lost in our implementation by reading \ac{graphml}
files from disk.  There are, however, hard limits which we actually ran into that are not caused by our design
decisions.  For example, the \ac{ogdf} routines sometimes simply crash with an out-of-memory error and other can take
very long to run.  Training the neural network only takes a few minutes on a normal \acs{pc} and evaluating the model
once it is trained and all inputs are available is so quick that it is hard to notice.

\section{Summary}

We were only able to do very limited evaluation of our discriminator model so the findings in this chapter have to be
taken with caution.  Nevertheless, we could show that our model reaches a reproducible success rate above \(95\,\%\) for
our data corpus.

The biggest contribution comes from the \(\menum{RDF\_LOCAL}(d)\) family of properties, followed by \enum{RDF\_GLOBAL}.
The contribution of and overlap between the other properties remains to be studied in a more elaborate investigation.

Speaking of performance, most time is spent in our current system while waiting for slow \acs{io} operations using the
file system and parsing \acs{nxml} documents.  Nevertheless, resource limits for the algorithmically more expensive
operations (especially the computation of all-pair shortest path matrices) might be a concern, too.

\chapter{Conclusions and Future Work}
\label{chap:conclusion}

Unattended graph drawing can be a valuable part in the software engineering toolbox for a variety of business needs.
Given the great variability of graphs people are dealing with and the manifold of layout algorithms available, we
believe that a reliable method for automatic quantification that is not based on specific assumptions would be an
important tool.  Among other use cases, it could enable practitioners to quickly identify the \enquote{best} layouts in
a large collection, guide them through the choice of a suitable layout algorithm for their domain or even promote the
development of new algorithms.

We have defined the scope for automatic quantification of the aesthetic value of graph layouts and presented an approach
towards automatic quantification that is based on the computation and statistical analysis of various elementary
properties of a graph layout.  These should, ideally, be searched from first principles and be influenced as little as
possible by existing assumptions.  We believe and -- to some degree -- have provided evidence that these features can be
representative syndromes of aesthetic value.  We think that inspiration from other disciplines and fields of science,
especially astronomy, crystallography and thermodynamics can provide valuable directions.

As we begun this work with an open-ended mindset, we started by developing a flexible framework to help us conduct our
studies.  We believe that the correct approach to investigate this topic further is not to restrict oneself to any one
specific type of graph, property or data analysis but rather ensure that the system allows flexible addition of features
and analyses.

Our main contribution is therefore a toolbox that might see continued use in the future in order to delve deeper into
the subject.  Our setup consists of many individual command-line tools that can be freely combined to build powerful
applications.  They are accompanied by a, by now, fairly elaborate driver \enquote{script} which acts as a coordinator
and can bring data and analyses from various sources together and can be configured and adapted in various ways.
The driver also provides a feature-rich web font-end for a convenient inspection and presentation of the data.  We
reckon that we have provided not so much of an experiment, let alone a theory, but rather a tool for experimentation.

The feasibility of our approach towards a quantification of aesthetic value of graph layouts was tested and demonstrated
by training a neural network to become a fairly reliable (\(>95\,\%\)) discriminator for pairs of layouts regarding
their aesthetic value.  Within the realm of the limited evaluation we could do, it became apparent that a localized
definition of the \acl{rdf} which is well-known to physicists was the most reliable syndrome.  Unfortunately, it is --
among those we've given consideration -- the most expensive property to compute as it requires knowledge of an all-pairs
shortest path matrix which can be a cubic operation.  Evaluation of the sliding averages needed

We do not suggest that using a neural network as we did is the most effective way to approach the problem of automatic
quantification of aesthetic value in the context of graph drawing.  We do believe, however, that the fact that a network
was apparently able to learn a great deal of knowledge from the data we provided it, is indicative of the fact that the
features we're looking at are indeed syndromes of aesthetic value to a certain degree.  Maybe future research can find a
more white-box answer to the question what exactly those features are and how they correlate.

We have discussed the data generation and augmentation process in great length.  The whole process is fully automated
and does not require any human intervention or labeling of data.  The setup is also self-contained and reproducible.  We
encourage others to re-run our experiments -- ideally on larger machines than those that were available to us during the
preparation of this work.  Please refer to the appendix of this document for pointers on how to get started.

There are also many aspects of the topic that we were not able to address within the time and resource limits imposed on
the preparation of this work.  In the final section, we shall outline some of our ideas that we were not able to
investigate further, yet.

\section{Additional Properties}

An obvious deficit of our current set of properties is that they focus a lot on the drawing of vertices and not so much
on edges.  Edge crossings and crossing angles would most certainly be a valuable addition.

\index{shapelet analysis}
Another idea that we find very interesting is to apply the technique of \emph{shapelet analysis}~\cite{Refregier2003} to
the field of graph drawing.  Shapelets are used in astronomy to study the structure of galaxies.  They basically present
a way to compute a series of weighted sums that preserve some spatial information by using a sequence of
\(n\)-dimensional filter functions, such as provided by one of the popular polynomial bases.  This filter is then
centered at each object's location and the function value at the position of the other objects are summed up.
The technique has already enjoyed successful application outside the field of astronomy, for example in the processing
of microscopic imagery~\cite{Suderman2014}.  It seems promising to us that it might also be valuable in the field of
graph drawing.

A fairly general concept that we find very exciting is the consideration of features of a layout with regard to some
graph-theoretical property of the drawn graph.  The example of local \acs{rdf} that was presented and shown to be of
value in this work has confirmed our intuition that this could be a powerful concept.  We would like to investigate
further examples -- ideally ones that require less than cubic computational effort.

\section{More Elaborate Data Analysis}

Ignoring for a while the fact that we also input the mean and \ac{rms} into our model, the only really exciting measure
we currently use is entropy.  While it was challenging enough to get a reasonably reliable approximation of the entropy,
there are myriads of other data analysis techniques that could also be applied to the properties we computed and might
-- especially in combination -- provide deeper insights into the structure of the data.

A very important piece of work would also be to find and peruse ways to assess the overlap between the information
provided by a collection of symptoms.  We have performed some superficial analysis that lead us to the conclusion that
local and -- to a lesser extent -- global \ac{rdf} provide a distinguishing view of the layout's quality but all other
properties remain yet to be picked apart.

\section{Thorough Comparison With Existing Measures}

The most painful omission from the present work is the lack of a comparison with existing quality measures.  Such
analysis should be performed with respect to accuracy as well as generality and efficiency.  In particular, it would be
interesting to see whether there is a correlation between, say, our discriminator failing to predict a layout pair
correctly and the value of the stress function for both layouts.  Such analysis would be relatively simple to set up
even if it might take a lot of computational resources to be carried out over a large data set containing large graphs.

\section{Conduction of a User Study}

\index{user study}
At the end of the day, any work on aesthetics in human perception has to be validated against the judgment of actual
humans.  As a very helpful side-effect, such an empirical study can produce more labeled data that may in turn be used
to further reify the data acquisition and processing strategy.  For example, we simply \emph{assume} that the native
layouts produced by our generators are \enquote{good} while there is no empirical evidence for this.

\section{Generalization to More Complex Graph Drawings}

\index{Lombardi!graph drawing}
\index{Lombardi!Mark}
Until now, we have considered straight-line drawings exclusively.  There are, however, many other interesting ways to
draw a graph and our current analysis will not be able to capture these specific features.  For example, a particularly
interesting kind of drawings are so-called \emph{Lombardi} graph drawings~\cite{Purchase2013} (named so after the
graphical artist Mark Lombardi who created drawing that inspired the layouts.  A Lombardi graph drawing hat the
distinguishing property that the angle between the incident edges of a vertex with degree \(n\in\IntsN\) is always
\(2\pi/n\) which is obviously not always possible when using only straight lines so edges have to be drawn as bent
curves.  It would be very interesting to see how the \enum{ANGULAR} property we've used would react to such a drawing,
if only it could be approached by it.

\section{Application as a Meta-Heuristic for a Genetic Layout Algorithm}

\index{algorithm!genetic}
\index{mutation function}
\index{crossover function}
\index{fitness function}
The last thought we'd like to share is that our work might have an unanticipated application as a layout algorithm.
Given that we already have
\begin{itemize}
\item various existing layout algorithms that produce layouts that are often good and seldom ideal and provide a good
  \emph{initial population},
\item a set of unary layout transformations which can be parameterized in their intensity and act as a configurable
  \emph{mutation function},
\item a set of binary layout transformations which can be parameterized (probabilistically, if one so wishes) in favor
  of wither parent and act as a \emph{crossover function} and finally
\item our discriminator as a means to predict which of two layouts has a higher aesthetic value and can therefore be
  used to construct a \emph{fitness function},
\end{itemize}
we have, in principle, every tool in the box that is needed in order to build a so-called \emph{genetic
  algorithm}~\cite{Mitchell1996} which is a popular meta heuristic for hard optimization problems closely inspired by
biological evolution.  We would be delighted to see how it works out.

\backmatter
\renewcommand*{\thesection}{\Alph{section}}
\cleardoublepage

\printbibliography[notkeyword=transient]

\printindex

\addchap{Appendix}

\section*{Using Our Software}

Shortly after the official submission of this thesis, all source code will be made available at
\url{http://klammler.eu/msc/} for anyone to experiment with it.  Since time constraints hindered us from including a
more comprehensive reference documentation in this appendix, this information will be made available on said web site
and in the \verb`README` file.

We are also thinking about a way to make the web \acs{ui} publicly available but have yet to figure out a good way of
doing so.  If it should happen, the link provided above will give a pointer to the place where the interactive web page
can be found.  If it doesn't happen, it will always be possible to download the source code and run the web server
(which is implemented using \Python's \verb`http.server` module) locally as we did it for a long time now.  The problem
with making the web \acs{ui} publicly available on the internet is that the current implementation cannot handle
concurrent requests and might also be vulnerable to malicious inputs.

The source archive contains a \CMake{} project which can be configured and built using the normal procedure.  System
requirements and software dependencies will be specified in the \verb`README` file.  Several configuration options are
available at the {\CMake} level or via environment variables and many more via configuration files that will be
explained in the \verb`README` file, too.  Running our experiment is just a matter of building the target \verb`deploy`
while the evaluation shown in \acl{chap}~\ref{chap:eval} can be repeated by building the \verb`eval` target.  In order
to launch the web server locally on port \(8000\), build the \verb`httpd` target.  All these tasks can also be achieved
by invoking the respective scripts directly which offers a greater level of control.  Finally, building the
\verb`report` target will go and rebuild all imagery from scratch using our toolbox and then finally typeset this
document with the experimental results available on the current machine.

We hope that this piece of software will be found useful and would be happy to assist with any difficulties that might
arise.  Please feel free to contact us at the e-mail address \email{moritz.klammler@student.kit.edu} or follow the link
above to find an up-to-date contact address.

\end{document}
