\chapter{Introduction}
\label{chp:intro}

\section{Vector neural network}
\label{sec:ch1_vector_neural_network}
Vector-valued neurons have received much attention lately in different scientific fields, such as communication systems, biological processing, image processing, and audio signal processing \cite{goh09complex, nitta09complex, hirose13springer, hirose13wiley, bengio18iclr}. Each training sample of these applications can be represented as a multidimensional vector, which can be processed directly by vector-valued neurons. These aforementioned works have shown that vector-valued neurons have good performance in learning, association, and generalization. In the meantime, an increasing number of datasets \cite{yasuma10itip,koelstra12tac,vincent15chime} provide multidimensional data suitable for vector-valued neural learning.

In real-valued neural network (NN) learning with multidimensional data, the input is concatenated from a set of vectors and reformulated as a long vector. A neuron takes only one real value as its input and a network is configured to use as many neurons as the dimension of the long vector. But this configuration may not achieve satisfactory performance for multidimensional problems since associations within each vector are not learned. Therefore, multidimensional vector neurons have received some attention in the literature and have been proposed to address the associations among different dimensions. A vector-valued neuron accepts and represents information as a vector and elements in each vector are processed together as a single unit. 

In convolutional neural network learning, advanced network architectures such as batch normalization~\cite{ioffe15arxiv}, highway networks~\cite{srivastava15nips}, and residual networks~\cite{he16cvpr} have been proposed in recent years to improve the performance or training efficiency of deep learning models. Both highway and residual networks reduce the risk of vanishing gradients via gating-based mechanisms or shortcut paths to regularize information flow.
There have also been attempts to make more fundamental changes to convolutional neural networks (CNNs). In particular, vector-valued neural learning in CNNs has received increased attention recently, where the inputs, outputs, weights and biases are all extended from real values to vector values. 


\begin{sidewaysfigure}[!t]
\centering
\includegraphics[trim=4.2cm 3cm 4.0cm 1cm, clip, scale=0.55]{ch1_introduction/vnn_arch.pdf}
\caption{Architecture of ABIPNN (best viewed in color), where the inputs, outputs, weights, and biases are all $N$-dimensional vectors. Cubes in the architecture represent scalars. Blue lines between hidden layers represent weights. $M$ represents the number of training data patterns. $R_{l}$ represents the dimension at layer $l$, where $ 1 \leq l \leq L$. Among them, $R_1$ and $R_L$ are respectively the dimensions of the input and output layers. From $R_2$ to $R_{L-1}$ are the dimensions of the hidden layers. $\bm{\mathcal{X}}$ and $\bm{\mathcal{Y}}$ respectively represent the input and output tensors. When $N$ equals one, each neuron represents a scalar, depicted as a cube with solid lines. This architecture can be viewed as a conventional NN. When $N>1$, each neuron represents a vector, depicted as the concatenation of cubes connected by the dotted lines.}
\label{fig:vnn_arch}
\end{sidewaysfigure}


\section{Related work}
\label{sec:ch1_related_works}
There are several approaches to extend real-valued neurons to vector-valued ones. Two-dimensional complex-valued NNs\cite{nitta97nn} have been proven to have orthogonal decision boundaries\cite{nitta00npl,nitta04nc} and the ability to solve exclusive-or (XOR) problem\cite{nitta03nn} using only a single neuron. Another extension to two dimensions is the hyperbolic neural network \cite{buchholz00ijcnn}, in which all parameters are hyperbolic numbers. Decision boundaries of hyperbolic neurons are investigated in \cite{nitta08ijcnn}. An alternative hyperbolic backpropagation algorithm \cite{nitta17tnnls} is developed using Wirtinger calculus. Three-dimensional neurons have been proposed in two ways. The first is based on the vector product\cite{bkt15hdn} \cite{nitta93ijcnn}, in which inputs, outputs, weights and biases are all three-dimensional vectors. The second \cite{nitta06nipr} is similar to the first but the weights are now three-dimensional orthogonal matrices. Four-dimensional hypercomplex-valued neurons \cite{arena94iscas, arena98Springer, nitta95icnn, bkt15hdn} are proposed by using the quaternion algebra. Eight-dimensional octonion-valued neurons\cite{popa16icann} represent a generalization of quaternion NNs. More recently, deep complex networks (DCN)~\cite{bengio18iclr} have been introduced using complex convolution and complex batch normalization. They extend each neuron from a real value to a two-dimensional vector representing complex numbers, and the model has been evaluated on image classification and music transcription.
Deep quaternion networks (DQN)~\cite{gaudet17arxiv,parcollet18arxiv} make each neuron a four-dimensional vector representing the so-called quaternions, and the model has been evaluated on automatic speech recognition and image segmentation. Although both DCN and DQN show the promise of vector-neuron learning, the dimensionality of their vector-valued inputs are restricted to two or four only. 


\section{Current limitation of vector neural networks}
\label{sec:ch1_current_limitation_of_using_vnn}
However, the dimensionality of a vector-valued neuron should not be constrained to a particular number. For instance, in the task of color image inpainting~\cite{eigen13iccv}, where small corrupted regions are to be restored, the input and output comprise three color channels.
In the task of multispectral image denoising \cite{zhang16ijcai}, different bands of images are stacked into a tensor. Nonlinear mapping between the noisy tensor and the clean tensor is then performed. For singing voice separation\cite{fan17dlm}, the data structure of a temporal-frequency matrix input is flattened into a high-dimensional vector in the traditional way, and then reshaped as a matrix so that each neuron represents and receives a vector. For EEG-based emotion recognition \cite{koelstra12tac,jirayucharoensak14tswj}, the human brain wave is filtered into five main frequency bands. Given the above observations, it seems important that the dimensionality should be an arbitrary $N$. 
Thus, a good extension of the real-valued neuron is the $N$-dimensional real-valued neuron \cite{bkt15hdn,nitta07ijcai,nitta13wiley}, which was proposed to have $N$-dimensional vector inputs and outputs, but $N$-dimensional orthogonal matrix weights. Nevertheless, we observe that this formulation does not address the case of multiple neurons.
Other architectures have also been advanced to extend real-valued neurons. For example, 
Clifford algebra has been employed to build vector-valued NNs with dimensionality $2^N$ \cite{pearson92nn,pearson94nn,bayro01geometric,Sommer01camp}; however, we note that many datasets do not have power-of-two dimensionalities. 
Matrix-valued NNs\cite{popa15mvnn} have been proposed in which the inputs, outputs, weights, and biases are $N\times N$ square matrices. But it is not always easy to formulate the inputs and outputs like this. Finally, for multiway classification, the tensor-factorized NN \cite{chien17tnnls} integrates Tucker decomposition with neural learning, though its efficiency in regression problems is yet unproven. A similar concept would be the vector neuron models of associative memory~\cite{kryzhanovsky04vector}, which are vector formalisms of Potts-glass NN~\cite{kanter1988potts,bolle91stability,bolle92thermodynamic} and parametrical NN (PNN)~\cite{kryzhanovsky03effective}, but this concept does not belong to supervised learning since there are no inputs and corresponding targets for training.




\section{Contribution of the dissertation}
\label{sec:ch1_contribution_of_the_dissertation}
In light of the above observations, we propose a new vector-valued NN architecture consisting of $N$-dimensional vector-valued neurons, where $N$ is an arbitrary positive integer (as shown in Fig.~\ref{fig:vnn_arch}). For each neuron, the inputs, outputs, weights, and biases are all $N$-dimensional vectors. Our key observation is that the products used by the aforementioned vector-valued NNs (such as the vector product~\cite{nitta93ijcnn} and quaternion multiplication~\cite{arena94iscas}) are bilinear products (defined in Section~\ref{chp:abpinn_method}). 
This prompts us to propose a general form of bilinear neurons to model the associations between the vector elements. We call it the Arbitrary BIlinear Product Neural Network (ABIPNN). When $N$ equals one, each neuron represents a scalar and the architecture performs matrix multiplications like a conventional deep neural network (DNN). When $N$ is larger than one, each neuron represents a vector and the architecture uses bilinear products for multiplications. In this way, the proposed architecture not only allows for using vectors of arbitrary dimensionality in vector-valued NNs, but also all the vector-valued products that are bilinear.

On the other hand, we also propose a new CNN architecture named Deep Cyclic Group Network (DCGN), which involves the circular convolution from cyclic group algebra~\cite{farenick01algebras}. Our inputs, outputs, kernel maps, and feature maps are three-way tensors~\cite{kolda2009tensor} with the same mode-$3$ dimension. The mode-$3$ dimension corresponds to the prescribed dimensionality $N$ of the input data, where $N$ is an arbitrary positive integer. The proposed architecture is shown in Figure~\ref{fig:dcgn_arch}. Besides, weight initialization and batch normalization for vector-valued neurons are investigated. As a consequence, our contributions are summarized as follows:
\begin{itemize}
\item We proposed the ABIPNN model by using arbitrary bilinear products with $N$-dimensional vector-valued neurons, where $N$ is an arbitrary positive integer.
\item We proposed the DCGN model by using $N$-dimensional vector-valued neurons with cyclic group algebras, where $N$ is also an arbitrary positive integer.
\item We derived backpropagation learning algorithm for both ABIPNN and DCGN models, weight initialization, and batch normalization.
\item We demonstrated the potential of our architectures with singing voice separation, image inpainting and denoising experiments by using less number of parameters. 
\end{itemize}


\begin{figure}
\begin{center}
\includegraphics[trim=8cm 2cm 7cm 1.5cm, clip, scale=0.55]{ch1_introduction/vnn_explain.pdf}
\caption{Illustration of the operations employed by the (a) conventional NNs and (b) ABIPNNs; best viewed in color. In (a), vectors are concatenated together as a multidimensional input. Each weight stands for a  scalar, depicted as a cube. Take the first entry of each vector for example, the associations between the brown cubes are not learned since each hidden neuron receives information by using its group of weights, which are depicted in the same color. Hidden neurons do not share identical weights when performing learning. In (b), vectors are stacked into a $R_{l}\times 1\times N$ tensor (or lateral matrix) and brown cubes are concatenated together. Each neuron represents a vector and receives information by using the weight vector, depicted as blue cubes. For example, when we leverage circular convolution as the bilinear product in an ABIPNN, the weight vector can be deemed as a linear kernel mask which captures the associations between brown cubes by rotating itself in the learning process.}
\label{fig:vnn_explain}
\end{center}
\end{figure}


\begin{sidewaysfigure}[ht]
\begin{center}
\includegraphics[trim=3.5cm 1.5cm 2.5cm 1cm, clip, scale=0.53]{ch1_introduction/dcgn_arch.pdf}
\caption{Illustration of the proposed DCGN model for regression problems (best viewed in color). Orange and blue squares represent the convolved and pooled feature maps, respectively, and green squares represent the kernel maps. All feature maps across all the layers have the same mode-3 dimension (i.e., $N$). When $N=1$, the model reduces to the conventional CNNs, and we will not have those dashed squares. The kernel maps perform convolution with scalar multiplications. When $N>1$, each feature map and kernel map becomes three-way tensors. It is a regression problem since we have tensor input and tensor output. But, the model can be extended to deal with classification problems, by  discarding the last convolutional layer and adding fully-connected layers after DCGN. In such a case, DCGN would perform the function of feature learning.}
\label{fig:dcgn_arch}
\end{center}
\end{sidewaysfigure}


\section{Dissertation organization}
\label{sec:ch1_dissertation_Organization}

The dissertation is organized as follow. Chapter~\ref{chp:abpinn_method} derives feedforward and backpropagation processes using the proposed bilinear neurons. Chapter~\ref{chp:dcgn_method} derives the backpropagation algorithm using circular convolution. Besides, weight initialization and batch normalization for vector-valued neurons are also introduced. Chapter~\ref{chp:experimentals} compares the performance between conventional NNs and proposed architectures in singing voice separation, image inpainting, and multispectral image inpainting. Finally, we conclude the work and discuss future work in Chapter~\ref{chp:conclusion}.

