\chapter{Concept and Design}
\label{cha:conceptdesign}
% introduction to what needs to be done. also clarify terms "Host" and "Device" here
To be able to determine whether evaluating mathematical expressions on the GPU is better suited than on the CPU, a prototype needs to be implemented. More specifically, a prototype for interpreting these expressions on the GPU, as well as a prototype that transpiles expressions into code that can be executed by the GPU. The goal of this chapter, is to describe how these two prototypes can be implemented conceptually. First the requirements for the prototypes as well as the data they operate on are explained. This is followed by the design of the interpreter and the transpiler. The CPU interpreter will not be described, as it already exists.

% TODO: maybe describe CPU interpreter too? We will see

\section[Requirements]{Requirements and Data}
% short section.
% Multiple expressions; vars for all expressions; params unique to expression; operators that need to be supported
The main goal of both prototypes or evaluators is to provide a speed-up to the CPU interpreter already in use. However, it is also important to determine which evaluator provides the most speed-up. This also means that if one of the evaluators is faster, it is meant to replace the CPU interpreter. Therefore, they must have similar capabilities, and therefore meet the following requirements:

\begin{itemize}
	\item Multiple expressions as input.
	\item All input expressions have the same number of variables ($x_n$), but can have a different number of parameters ($p_n$).
	\item The variables are parametrised using a matrix of the form $k \times N$, where $k$ is the number of variables in the expressions and $N$ is the number of different parametrisations for the variables. This matrix is the same for all expressions. 
	\item The parameters are parametrised using a vector of vectors. Each vector $v_i$ corresponds to an expression $e_i$.
	\item The following operations must be supported: $x + y$, $x - y$, $x * y$, $x / y$, $x ^ y$, $|x|$, $\log(x)$, $e^x$ and $\sqrt{x}$. Note that $x$ and $y$ can either stand for a value, a variable, or another operation.
	\item The results of the evaluations are returned in a matrix of the form $k \times N$. In this case, $k$ is equal to the $N$ of the variable matrix and $N$ is equal to the number of input expressions.
\end{itemize}

With these requirements, one possible expression that must be able to be evaluated is the following: $\log(e^{p_1}) - |x_1| * \sqrt{x_2} / 10 + 2^{x_3}$

\begin{figure}
	\centering
	\includegraphics[width=.9\textwidth]{input_output_explanation.png}
	\caption{This diagram shows how the input and output looks like and how they interact with each other.}
	\label{fig:input_output_explanation}
\end{figure}


With this, the capabilities are outlined, however, the input and output data need to further be explained for a better understanding. The first input are the expressions that need to be evaluated. These can have any length and can contain constant values, variables and parameters and all of these are linked together with the supported operations. In the example shown in Figure \ref{fig:input_output_explanation}, there are six expressions $e_1$ through $e_6$. Next is the variable matrix. One entry in this matrix, corresponds to one variable in every expression, with the row indicating which variable it holds the value for. Each column holds a different set of variables. In the provided example, there are three variable sets, each holding the values for four variables $x_1$ through $x_4$. All expressions are evaluated using all variable sets and the results of these evaluations are stored in the results matrix. Each entry in this matrix holds the resulting value of the evaluation of one expression with one variable set. The row indicates the variable set while the column indicates the expression.

This is the minimal functionality needed to evaluate expressions with variables generated by a symbolic regression algorithm. In the case of parameter optimisation, it is useful to have a different type of variable, called parameter. For parameter optimisation it is important that for the given variable sets, the best fitting parameters need to be found. To achieve this, the evaluator is called multiple times with different parameters, but the same variables, and the results are evaluated for their fitness by the caller. In this case, the parameters do not change within one call. Parameters could therefore be treated as constant values of the expressions and no separate input for them would be needed. However, providing the possibility to have the parameters as an input, makes the process of parameter optimisation easier. This is the reason the prototype evaluators need to support parameters as inputs. Not all expressions need to have the same number of parameters. Therefore, they are structured as a vector of vectors and not a matrix. The example in Figure \ref{fig:input_output_explanation} shows how the parameters are structured. For example one expression has zero parameters, while another has six parameters $p_1$ through $p_6$. It needs to be mentioned that just like the number of variables, the number of parameters per expression is not limited. It is also possible to completely omit the parameters if they are not needed.

% \subsection{Non-Goals}
% Probably a good idea. Probably move this to "introduction"
\section{Architecture}

Based on the requirements above, the architecture of both prototypes can be designed. While the requirements only specify the input and output, the components and workflow also need to be specified. This section aims at giving an architectural overview of both prototypes, alongside their design decisions.

A design decision that has been made for both prototypes is to split the evaluation of each expression into a separate kernel dispatch. As explained in Section \ref{sec:thread_hierarchy}, it is desirable to reduce the occurrence of thread divergence as much as possible. Although the SIMT programming model tries to mitigate the negative effects of thread divergence, it is still a good idea to avoid it when possible. In this case, thread divergence can easily be avoided by not evaluating all expressions in a single kernel dispatch. GPUs are able to have multiple resident grids, with modern GPUs being able to have 128 grids concurrently \parencite{nvidia_cuda_2025}. One grid corresponds to one kernel dispatch, and therefore allow 128 kernels to be run concurrently. Therefore, dispatching a kernel for each expression, has the possibility to improve the performance. 

%% Maybe add overview Diagram 
%% Shows -> Caller calls Evaluator -> Evaluator dispatches kernel -> kernel evaluates -> Evaluator returns evaluation result
%% Probably the same as the interpreter and transpiler diagram. If so, dont add it

\subsection{Pre-Processing}
The first step in both prototypes is the pre-processing step. It is needed, as it simplifies working with the expressions in the later steps. Similar to the front-end in compilers as described in Section \ref{sec:compilers}, it takes an expression and transforms it into an intermediate representation. One of the responsibilities of the pre-processor is to verify that only allowed operators are present in the given expressions. Furthermore, it transforms the expressions into postfix-notation. This further allows the later parts to more easily evaluate the expressions. One of the major benefits of this notation is the implicit operator precedence. It allows the evaluators to evaluate the expressions token by token from left to right, without needing to worry about the correct order of operations. With one token being either an operator, a constant value, a variable or a parameter.

It would have also been possible to perform the pre-processing step on the GPU. However, pre-processing only one expression can not easily be split into multiple threads, which means one GPU thread would need to process one expression. As described in Section \ref{sec:gpgpu} a single GPU thread is slower than a single CPU thread. Furthermore, it wouldn't make sense to process all expressions in a single kernel. This would lead to a lot of thread divergence, essentially processing one expression after the other. The SIMT programming model might help with parallelising at least some parts of the processing work. The generated expressions can differ a lot from each other and restricting them to be similar and therefore SIMT friendly, would reduce the overall quality of the symbolic regression algorithm. Therefore, it does not make sense to perform the processing step on the GPU. This is a typical example of code that is better run on the CPU, also because the parallelisation possibilities of one thread per expression can be applied to the CPU as well. Concepts like caching processed expressions, or caching parts of the processed expressions can also be employed on the CPU. This would not be possible on the GPU, because they can not save state between two kernel dispatches.

\subsection{Interpreter}

\begin{figure}
	\centering
	\includegraphics[width=.9\textwidth]{component_diagram_interpreter.png}
	\caption{This diagram depicts the coarse-grained workflow of the interpreter. It shows how the parts interact with each other and with the system it will operate in.}
	\label{fig:component_diagram_interpreter}
\end{figure}

The interpreter consists of two parts. The CPU side or host side is the part of the program, that interacts with both the GPU or device and the caller. An overview on the components and the workflow of the interpreter can be seen in Figure \ref{fig:component_diagram_interpreter}. Before the GPU can start evaluating the expressions, a pre-processing step is necessary. This step is crucial, as it transforms the expressions in a format, that greatly simplifies the evaluation part. Before the expression can be evaluated however, all data needs to be sent to the GPU, this includes the processed expressions, as well as the data for the variables and parameters. After the kernel has finished evaluating all expressions, the CPU reads the results from the GPU and returns them to the caller.

Because of the already mentioned pre-processing step, the evaluation process is relatively straight-forward. The Algorithm \ref{alg:eval_interpreter} demonstrates how an expression in postfix-notation can be evaluated. It shows a simplified version that only works with addition, multiplication and constant values. This is the part of the interpreter prototype, that actually interprets the expressions and runs on the GPU. 

\begin{algorithm}
	\caption{Interpreting an equation in postfix-notation}\label{alg:eval_interpreter}
	\begin{algorithmic}[1]
		\Procedure{Evaluate}{\textit{expr}: PostfixExpression}
			\State $\textit{stack} \gets ()$

			\While{HasTokenLeft(\textit{expr})}
				\State $\textit{token} \gets \text{GetNextToken}(\textit{expr})$ 
				\If{$\textit{token.Kind} = \text{Constant}$} 
					Push($\textit{stack}$, $\textit{token.Value}$)
				\ElsIf{$\textit{token.Kind} = \text{Operator}$} 
					\If{$\textit{token.Value} = \text{Addition}$}
						\State $\textit{right} \gets \text{Pop}(\textit{stack})$
						\State $\textit{left} \gets \text{Pop}(\textit{stack})$
						Push($\textit{left} + \textit{right}$)
					\ElsIf{$\textit{token.Value} = \text{Multiplication}$}
						\State $\textit{right} \gets \text{Pop}(\textit{stack})$
						\State $\textit{left} \gets \text{Pop}(\textit{stack})$
						Push($\textit{left} * \textit{right}$)
					\EndIf
				\EndIf
			\EndWhile

			\Return $\text{Pop}(\textit{stack})$
		\EndProcedure
	\end{algorithmic}
\end{algorithm}

If a new operator is needed, it must simply be added as another else-if block inside the operator branch. New token kinds like variables or parameters, can also be added by adding a new outer else-if block that checks for these token kinds. However, the pre-processing step also needs to be extended with these new operators and token kinds, otherwise the expression will never reach the evaluation step. It is also possible to add unary operators like logarithm. In this case only one value would be read from the stack, the operation would be applied, and the result would be written back to the stack.


\subsection{Transpiler}

\begin{figure}
	\centering
	\includegraphics[width=.9\textwidth]{component_diagram_transpiler.png}
	\caption{This diagram depicts the coarse-grained workflow of the transpiler. It shows how the parts interact with each other and with the system it will operate in.}
	\label{fig:component_diagram_transpiler}
\end{figure}


% \section{Interpreter}
% % as introduction to this section talk about what "interpreter" means in this context. so "gpu parses expr and calculates"
% In this section, the GPU-based expression interpreter is described. It includes the design decisions made for the architecture of the interpreter. It also describes what is done on the CPU side or host side and what is performed on the GPU side or device side.

% \subsection{Architecture}
% talk about the coarse grained architecture on how the interpreter will work. (.5 to 1 page probably)
% Include decicions made like "one kernel per expression"

% \subsection{Host}
% talk about the steps taken to prepare for GPU interpretation

% \subsection{Device}
% talk about how the actual interpreter will be implemented


% \section{Transpiler}
% as introduction to this section talk about what "transpiler" means in this context. so "cpu takes expressions and generates ptx for gpu execution"

% Transpiler used, to reduce overhead of the generic PTX transpiler of CUDA, as we can build a more specialised transpiler and hopefully generate faster code that way. (this sentence was written as a reminder and not to be used as is)

% \subsection{Architecture}
% talk about the coarse grained architecture on how the transpiler will work. (.5 to 1 page probably)

% \subsection{Host}
% talk about how the transpiler is implemented

% \subsection{Device}
% talk about what the GPU does. short section since the gpu does not do much