\chapter{Experiments}
\label{chp:experimentals}

To validate the effectiveness of ABIPNN, we consider two regression problems that require learning a nonlinear mapping between tensor inputs and tensor outputs. The first is \emph{multispectral image denoising}, which aims to recover the original multispectral image from a noisy input, with $N$ set to $10$.
The second is \emph{singing voice separation}, which aims to separate the singing voice and the accompaniment from a monaural audio mixture, with $N$ set to $1$, $3$, $5$, $7$.

We intend to empirically compare the performance of the conventional DNNs with ABIPNN. In the first experiment, we will investigate their performance using a similar number of parameters. In the second one, the complexity analysis in Chapter~\ref{chp:abpinn_method} will also be verified. For both DNNs and ABIPNN, we use the mean-square error (MSE) as the objective function, and Adam~\cite{kingma14arxiv} for gradient optimizations. 
For ABIPNN, we employ circular convolution as our bilinear product. The experiments are performed in Python on a personal computer equipped with an NVIDIA GeForce GTX $1080$ Ti GPU and a memory of $64$ GB RAM. 


To evaluate the effectiveness of DCGN, we test it on two regression problems. The first one is \emph{color image inpainting}, which aims to reconstruct the deteriorated or lost parts of the original color image. The second one is \emph{multispectral image denoising}, which aims to recover the original multi-band image from a noisy version.
For the first problem, the input is a tensor with $N=3$, the RGB channels. For the second problem, since a multispectral image is composed of multiple greyscale images shooting at different wavelengths (or bands), $N$ is equal to the number of bands. We note that DCGN can deal with multispectral images of arbitrary bands, but DQN can only deal with 4-band multispectral images each time. Therefore, we intend to vary the value of $N$ from 3 to 6 for the second experiment.

As baseline methods, we consider the conventional CNNs and the DQN model proposed by~\cite{gaudet17arxiv}, using the code they shared. The goal is to verify the possible advantages of using the vector neurons over the scalar neurons, instead of pursuing high accuracy. Therefore, we opt for simpler model design and aim to compare the models fairly. For all the baseline models and our model, we use only three fully convolutional layers (without any pooling layers) and employ kernel maps with size $3 \times 3$. We used ReLU as the activation function, mean squared error (MSE) as the cost function, and Adam~\cite{kingma14arxiv} for gradient optimization.

For conventional CNNs, we consider three variants of CNN with different number of kernels per layer. CNN$_1$ has the same number of kernels per layer (i.e., 24) as DCGN and DQN; CNN$_2$ has more kernels but the total number of parameters is similar to DCGN, and CNN$_3$ has even more kernels and roughly three times the total number of parameters as DCGN. DCGN and DQN have more parameters than CNN$_1$ because their kernel maps are tensors.


The evaluation metrics for both experiments are peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM)~\cite{wang04tip}. SSIM is a measure of the similarity between two images (i.e., the groundtruth clean one and the recovered one) and its range is $[0,1]$. Both PSNR and SSIM are the higher the better. We calculate them for each output result and report the average values. All the experiments are performed on a personal computer equipped with an NVIDIA GeForce GTX 1080Ti GPU and 64GB RAM. Our code is written in Python. For reproducible research, we will make the code of the experiments publicly available at~\url{https://github.com/zcfan-tw/vectorNNtoolbox}. 


\newpage
%---------------------------------------------------------------
\section{Experiment on multispectral image denoising}
\label{sec:ch3_multispectral_image_denoising}
Multispectral (a.k.a.~hyperspectral) imaging systems are usually employed to solve broadband color problems. A multispectral image is composed of a collection of monospectral (or monochrome) images, each of which is captured with a specific wavelength. These monospectral images can be considered as different \emph{bands} of the multispectral image. As different spectral bands may exhibit some 
mutual associations, we can leverage such associations to enhance the accuracy of image processing applications. 


In multispectral image denoising, we are given a noisy version of a multispectral image  with $N$ bands, and are asked to recover the clean version (also $N$ bands). In a recent work presented by Zhang~\emph{et al.}~\cite{zhang16ijcai}, different supervised multidimensional dictionary learning methods were evaluated on the Columbia multispectral image database~\cite{yasuma10itip} for the denoising task. By following their settings, we can compare the performance of our models with these prior arts.
These methods include
K-TSVD, K-SVD~\cite{aharon06sp}, $3$D K-SVD~\cite{elad06ip},  
% BM$3$D\cite{dabov07ip}, 
LRTA~\cite{renard08grs}, DNMDL~\cite{peng14cvpr} and PARAFAC~\cite{liu12grs}. 

The database~\cite{yasuma10itip} contains $32$ real-world scenes and each scene contains $31$ monochrome images of size $512\times512$, captured by varying the wavelength of a camera from $400$ nm to $700$ nm with a step size of $10$ nm. Following Zhang~\emph{et al.}~\cite{zhang16ijcai}, we consider images in the ``chart and stuffed toy'' scene, resize each image to $205\times205$ and take images of the last $10$ bands
(i.e., starting from $600$ nm), making $N=10$.
Moreover, we divide each image into $8\times8\times10$ overlapping tensor patches with a hop size of one. We randomly take $10\,000$ tensor patches as the training data and the rest for testing. 
From the training data, $1000$ tensor patches are held out as the validation data.  
The maximal number of training epochs is set to $3000$. We also stop the training process if the validation MSE does not decrease for $100$ epochs.  


For the noise model, we randomly select a certain number of pixels from each band of an image and add to the pixels Gaussian noise with specific \emph{sigma value}.
We refer to the ratio of pixels corrupted per band as the \emph{sparsity} level of the noise.
In our experiments, we vary the sigma value from $100$ to $200$, and the sparsity level from $5\%$ to $15\%$, to simulate different degrees of corruption. The goal is to recover the corrupted images, as illustrated in Fig.~\ref{fig:multispectral_image_denosing_arch}.
As for the objective function in model training, we compute the MSE between the recovered and the original versions of the patches in the training set, across all the $10$ bands. 
As in Zhang~\emph{et al.}~\cite{zhang16ijcai},  we measure the performance of denoising in terms of the peak signal-to-noise ratio (PSNR). We calculate the PSNR for each test patch and report the average result. 




\begin{figure}[t]
\begin{center}
\includegraphics[trim=3cm 5cm 3cm 4cm, clip, scale=0.5]{ch4_experiments/image_denoising_arch.pdf}
\caption{Illustration of multispectral image denoising using ABIPNN. Noisy images are contaminated by Gaussian noise. We take the last $10$ bands for the experiments.}
\label{fig:multispectral_image_denosing_arch}
\end{center}
\end{figure}

In our implementation, the ABIPNN consists of $3$ hidden layers with $512$ neurons in each layer (i.e., $R_{2}=\dots=R_{L-1}=512$ in Fig.~\ref{fig:vnn_arch}). Both the input and output layers have $8\times8=64$ dimensions (i.e., $R_{1}=R_{L}=64$ in Fig.~\ref{fig:vnn_arch}) and the topology is denoted as $64$-$512$-$512$-$512$-$64$. 
In ABIPNN, each neuron represents a vector with size $N=10$. Hence, for each patch the $10$ bands are processed at the same time. The activation function is sigmoid with a learning rate of $5\times10^{-4}$. To compare the performance of ABIPNN with conventional DNNs using a similar number of parameters, we consider the following two variants of DNNs as the baselines. The first variant, \emph{DNN-concat}, simply concatenates the $10$ bands as a single vector to process them jointly, making $R_{1}=R_{L}=640$. There are $3$ hidden layers with $1,450$ neurons in each layer, so the topology is $640$-$1450$-$1450$-$1450$-$640$, where each neuron represents a scalar. The second variant, \emph{DNN-parallel}, processes the $10$ bands separately using $10$ DNNs, each with a $64$-$512$-$512$-$512$-$64$ topology. In other words, each DNN is trained for denoising a specific band.


Table~\ref{tab:multispectral_image_denosing_table} shows the experimental results. In the upper part of the table, we cite the PSNR values reported in \cite{zhang16ijcai}, and in the lower part we report the results of our own implementation.\footnote{Each result we reported is the average of $10$ simulations and the variance of each result is low. For example, when the sparsity is $5\%$ and the sigma is $100$, the variance of DNN-concat, DNN-paralled, and ABIPNN is $0.003$, $0.014$, and $0.011$, respectively.} 
Because we follow their experimental settings, the PSNR values for the noisy images (i.e., before denoising) reported in \cite{zhang16ijcai} are close to what we observe in our implementation.
Besides, Table~\ref{tab:multispectral_image_denosing_table} also shows that ABIPNN outperforms all the other methods, including the two DNN baselines, by a large margin across different values of sigma and sparsity. The PSNRs are improved from 12.10--20.92 dB to 29.55--33.92 dB.
This result suggests the effectiveness of a neural network for this task.
Fig.~\ref{fig:multispectral_image_denosing_examples} demonstrates the original, noisy, and denoised versions of three images by using ABPINN. 

\begin{table}[t]
\renewcommand\arraystretch{1.5}
\centering
\captionsetup{justification=centering,margin=0.3cm}
\caption{\textsc{PSNR (in dB) obtained by different methods for multispectral image denoising, under different sparsity and sigma values}}
\label{tab:multispectral_image_denosing_table}
\begin{tabular}{|l|c|c|c|c|c|}
\hline
Sparsity        & $5\%$          & $10\%$          & $15\%$          & $10\%$          & $10\%$  \\ 
% \hline
Sigma           & $100$          & $100$           & $100$           & $150$           & $200$   \\
\hline
\multicolumn{6}{|l|}{\textbf{Referenced from \cite{zhang16ijcai}}} \\
% K-TSVD, K-SVD\cite{aharon06sp}, $3$D K-SVD\cite{elad06ip},  
% % BM$3$D\cite{dabov07ip}, 
% LRTA\cite{renard08grs}, DNMDL\cite{peng14cvpr} and PARAFAC\cite{liu12grs}
\hline
%\hdashline
Noisy Image     & $20.96$        & $18.18$         & $16.35$         & $14.75$         & $12.10$ \\ 
% \hline
K-SVD~\cite{aharon06sp}          & $22.73$        & $22.60$         & $22.49$         & $22.38$         & $22.20$ \\
$3$DK-SVD~\cite{elad06ip}      & $22.61$        & $22.53$         & $22.47$         & $22.41$         & $22.20$ \\
% BM$3$D          & $26.95$        & $26.62$         & $26.36$         & $25.23$         & $24.92$ \\
LRTA~\cite{renard08grs}           & $23.54$        & $26.84$         & $26.65$         & $23.90$         & $22.03$ \\
DNMDL~\cite{peng14cvpr}          & $24.07$        & $23.73$         & $25.16$         & $17.89$         & $16.83$ \\
PARAFAC~\cite{liu12grs}        & $27.07$        & $26.86$         & $26.72$         & $26.13$         & $25.24$ \\
K-TSVD~\cite{zhang16ijcai}         & $27.19$        & $26.98$         & $26.79$         & $26.18$         & $25.44$ \\
\hline
\multicolumn{6}{|l|}{\textbf{Our Experiments}} \\
\hline
Noisy Image     & $20.92$        & $18.16$         & $16.35$         & $14.64$         & $12.10$ \\
% \hline
DNN-concat & $25.06$        & $24.80$         & $24.93$         & $24.59$         & $24.03$ \\
DNN-parallel & $30.18$        & $28.88$         & $28.06$         & $27.17$         & $25.88$ \\ 
ABIPNN   & $ {33.92}$   & $ {32.47}$    & $ {31.74}$    & $ {31.01}$    & $ {29.55}$ \\
\hline
\end{tabular}
\end{table}




\begin{figure}[t]
\begin{center}
``Chart and Stuffed Toy'' Image \\
\includegraphics[trim=2cm 4cm 1.5cm 4cm, clip, scale=0.95]{ch4_experiments/chart_and_stuffed_toy_ms.eps} \\
``Face'' Image \\
\includegraphics[trim=2cm 4cm 1.5cm 4cm, clip, scale=0.95]{ch4_experiments/face_ms.eps} \\
``Paints'' Image \\
\includegraphics[trim=2cm 4cm 1.5cm 4cm, clip, scale=0.95]{ch4_experiments/paints_ms.eps} \\
``Fake and Real Peppers'' Image \\
\includegraphics[trim=2cm 4cm 1.5cm 4cm, clip, scale=0.95]{ch4_experiments/fake_and_real_peppers_ms.eps} \\

(a) Original ~~~~~~~~~~~~ (b) Noisy ~~~~~~~~~~~~ (c) Recovered 
\caption{Denoised images at the $700$ nm band using the proposed ABIPNN method. 
The sparsity of the noisy pixels is $10$\% and the sigma value of the additive Gaussian noise is set to $200$.}
\label{fig:multispectral_image_denosing_examples}
\end{center}
\end{figure}


\begin{figure}
\begin{center}
\includegraphics[trim=0.1cm 0.05cm 0.5cm 0.1cm, clip, scale=0.8]{ch4_experiments/error_cost_gradient_variance_trend.eps}
\caption{(a) Mean square error with Adam and (b) gradient variance with SGD in the training procedure of ABIPNN and DNN-concat, for the case when the sparsity of the noisy pixels is $5$\% and the sigma value is set to $100$.}
\label{fig:error_cost_gradient_variance_trend}
\end{center}
\end{figure}

Among the three NN-based methods, DNN-concat performs the worst, and is even inferior to that of PARAFAC~\cite{liu12grs} and K-TSVD~\cite{zhang16ijcai}, two non-deep learning based methods. This suggests that concatenating inputs from different bands does not make it easy for an NN to learn the association between bands. 
In contrast, using multidimensional vector neurons, ABIPNN learns the relations between bands by computing the circular convolution of two vectors: one coming from a hidden node and the other coming from a weight tensor. The vector coming from a weight tensor can be regarded as a linear kernel that captures interactions across bands, which may contribute to 
enhanced results in denoising.
%learn associations in the vector coming from a hidden node. As a result, performing circular convolution between different bands in the learning process provides additional interaction information for supervised learning and it enhances the results of multispectral image denoising. 

%This demonstrates the effectiveness of vector neural learning for this task.

%It is obvious that neural-network-based methods may have a good ability to perform nonlinear mappings by employing multiple hidden layers if there are proper data as DNN input. When the numbers of parameters are approximately equal, ABIPNN performs better than the band-wise DNN-parallel since 





Figure~\ref{fig:error_cost_gradient_variance_trend}(a) displays the changes in MSE values as a function of the training epochs in the training procedure for ABIPNN and DNN-concat. We find that the MSE values converge to a certain value for both methods, but ABIPNN converges much faster. We conjecture that this is because the error propagation in ABIPNN is not only optimized for each dimension (i.e., band) but also between different dimensions.
%From the observation in the figure, \textcolor{red}{ABIPNN gets much lower training loss than DNN-concat. Although ABIPNN and DNN-concat have similar training loss around $2 000$ epochs, however ABIPNN has better performance than DNN-concat from the observation of Table \ref{tab:multispectral_image_denosing_table}.}
It is also known that gradient variance reduction helps achieve better convergence in stochastic gradient descent (SGD)~\cite{johnson13nips,reddi15nips}.
%, which may imply that NN training with lower variance lead to better performance. 
Fig.~\ref{fig:error_cost_gradient_variance_trend}(b) shows the changes in gradient variance during SGD training. Here ABIPNN provides lower variance at convergence (after $2000$ epochs).
% while DNN-concat is still on the way to convergence. }
% This shows that ABIPNN may have the ability to converge quickly. A similar situation for MSE is shown in Fig.~\ref{fig:error_cost_gradient_variance_trend}(b).}




%---------------------------------------------------------------
\section{Experiment on singing voice separation} 
\label{sec:ch3_singing_voice_separation}
Separating the singing voice and the accompaniment from a monaural audio mixture is challenging, because there are more unknowns than equations. It is also a significant preprocessing step in audio signal analysis~\cite{ serra10amir,durrieu11jstsp,salamon12ismir_statistic,salamon12ismir_challenge,salamon13magazine,hsu12taslp,stefan15icassp,bittner15melody,fan16bigmm,yoshii16taslp,salamon16icml}. Several approaches have been proposed previously. Repeating musical structures have been exploited by the REpeating Pattern Extraction Technique \cite{raffii13taslp}.
Unsupervised methods such as robust principal component analysis~\cite{huang12icassp, yang13ismir,sprechmann12ismir, chan15icassp} assume no labeled data are available and rely on assumptions on the characteristics of the sources for separation. For example, the spectrogram of the accompaniment part is assumed to have lower rank than that of the vocal part. If we are given the original sources of some audio mixtures, we can take these clean sources as the supervisory signal and employ supervised  methods such as non-negative matrix factorization (NMF)~\cite{lee99nature}  for source separation.
Naturally, supervised methods usually outperform unsupervised methods, as the former can learn from the pairs of mixtures and sources.

Recently, NN-based methods have been introduced to this task \cite{huang14ismir,hershey16icassp}, showing better result than non-NN based methods such as NMF. With the development of deep learning, Mass \emph{et al.} \cite{mass12interspeech} used recurrent neural networks (RNN) to create a clean voice. Huang \emph{et al.} \cite{huang14ismir} then proposed deep RNN with discriminative training to reconstruct vocals from background music. Training multi-context networks \cite{zhang15icassp, zhang16taslp} with different inputs combined at layer level was proposed to improve audio separation performance. Deep clustering \cite{luo17icassp} is also used for music separation. Post-processing with a Wiener filter at the output of neural networks and data augmentation \cite{stefan17icassp} have been proposed to separate vocals and instruments. All of these deep learning techniques use multiple non-linear layers to learn the optimal hidden representations from data in a supervised setting.

This can be done by taking the spectrogram of an audio mixture as the input and requiring the network to reproduce the spectrograms of the corresponding two sources at the output.  NN works better because they can learn a nonlinear mapping between the input and the outputs. A spectrogram is a 2-D time-frequency representation.  It is computed by the short time Fourier transform (STFT), which divides a given time signal into short segments of equal length and then computes the Fourier transform separately on each short segment. We call each short segment a `frame,' and the result of Fourier transform per frame as  a `spectrum.'  The spectrogram considers only the magnitude part of STFT. 




\begin{figure}[t]
\begin{center}
\includegraphics[trim=4.5cm 0.5cm 5.8cm 1.4cm, clip, scale=0.60]{ch4_experiments/singing_voice_separation_arch.pdf}
\caption{Illustration (best seen in color) of singing voice separation using ABIPNN, where $T$ stands for the number of frames. We use red to indicate the previous frames, yellow for the current frames, and green for the subsequent frames. After training, we extract the second dimension (yellow) for soft-time frequency masking.}
\label{fig:singing_voice_separation_arch}
\end{center}
\end{figure}



A na\"ive DNN approach for source separation, referred to as \emph{DNN-simple} below, takes the spectrum of each frame of the mixture as input, and estimates the spectra of that frame for the vocal and the accompaniment parts respectively.  This is done frame-by-frame, finally leading to the estimated (recovered) spectrograms  $\mathbf{\tilde{Y}_1}$ and $\mathbf{\tilde{Y}_2}$ of the two sources. 
Then, we use the Weiner filter to compute the following soft time-frequency mask to smooth the source separation results. 
\begin{equation}
\mathbf{m}(f)=\frac{\vert\mathbf{\tilde{Y}_1}(f)\vert}{\vert\mathbf{\tilde{Y}_1}(f)\vert+\vert\mathbf{\tilde{Y}_2}(f)\vert} \,,
\end{equation}
where $f=1,2, ..., \mathbf{\textit{F}}$ denotes different frequency bins. The estimated spectra $\mathbf{\tilde{s}_1}$ and $\mathbf{\tilde{s}_2}$, respectively corresponding to vocals and accompaniments, are produced by:
\begin{equation}
\begin{split} 
&\mathbf{\tilde{s}_1}(f)=\mathbf{m}(f)\mathbf{z}(f)\textrm{,} \\
&\mathbf{\tilde{s}_2}(f)=(1-\mathbf{m}(f))\mathbf{z}(f)\textrm{,}
\end{split}
\end{equation}
where $\mathbf{z}(f)$ is the magnitude spectra of the input mixture.  The estimated spectra $\mathbf{\tilde{s}_1}$ and $\mathbf{\tilde{s}_2}$ are finally transformed back to the time-domain by the inverse short time Fourier transform (ISTFT), assuming that the mixture and the two sources share the same phase. Parameters of the NN are learned by using the MSE between the estimated spectra and the groundtruth source spectra. 



We can improve the performance of DNN-simple by adding the \emph{temporal context} of each frame to the input \cite{zhang16taslp}. Specifically, in addition to the current frame, we add the spectra of the previous-$f$ and subsequent-$f$ frames to compose a real-valued matrix. For a conventional NN, we can take the vectorized version of the matrix (which amounts to concatenating the spectra of these $2f+1$ frames) as the input. We refer to this method as \emph{DNN-concat}. Alternatively, we can view the $2f+1$ frames as different dimensions and use a vector NN to model the interaction between different frames. Please see Fig.~\ref{fig:singing_voice_separation_arch} for an illustration.


Because ABIPNN can deal with input of arbitrary dimensions, in principle $f$ can take any values. In this experiment, we compare the performance of ABIPNN with the classic vector product neural network (VPNN)~\cite{nitta93ijcnn} (i.e., where $f=1$ and only two neighboring frames are considered), and the conventional DNNs (i.e., where $f=1$--$3$). The major difference between ABIPNN (with $N=3$) and VPNN is that the former uses circular convolutions.

\begin{sidewaystable}
\centering
\captionsetup{justification=centering,margin=0.5cm}
\caption{\textsc{Comparison of SDR, SIR, SAR values and computation time (seconds per epoch)} obtained by different methods for singing voice separation over the DSD100 data set, under different values of $N$ and number of neurons per layer.}
\label{tab:dsd100_result_with_nn}
\begin{tabular}{|l|c|c|r|c|cc|cc|cc|}
\hline
\multirow{2}{*}{Model} &\multirow{2}{*}{$N$} & Neurons          &Number  &Time (second &\multicolumn{2}{c}{SDR} &\multicolumn{2}{c}{SIR} &\multicolumn{2}{c|}{SAR} \\
\cline{6-11}
                      &                     &    per layer     &             of params  & per epoch)   & Vocal   &Accomp.       & Vocal  &Accomp.       & Vocal   &Accomp.  \\
\hline
\hline
DNN-simple            & $1$    & $513$      & $1.31$M    & $4.2$   & $4.37$       & $9.98$         & $8.38$       & $12.89$       & $5.81$       & $13.89$  \\
\hline
DNN-concat$_1$        & $3$    & $1539$     & $8.68$M    & $7.1$   & $4.63$       & $10.25$        & $8.29$       & $13.34$       & $6.17$       & $14.14$  \\
VPNN~\cite{fan17dlm}  & $3$    & $513$      & $3.95$M    & $9.1$   & $4.64$       & $10.17$        & $8.37$       & $12.97$       & $6.59$       & $14.57$  \\
ABIPNN$_1$            & $3$    & $513$      & $3.95$M    & $9.2$   & $4.67$       & $10.08$        & $8.39$       & $12.76$       & $6.67$       & $14.24$  \\
\hline
DNN-concat$_2$        & $5$    & $2565$     & $22.36$M   & $14.1$  & $4.68$       & $10.19$        & $8.86$       & $12.53$       & $6.78$       & $15.07$  \\
ABIPNN$_2$            & $5$    & $513$      & $6.59$M    & $17.5$  & $4.97$       & $10.50$        & $9.45$       & $13.63$       & $6.88$       & $14.78$  \\
\hline
DNN-concat$_3$        & $7$    & $3591$     & $42.37$M   & $23.3$  & $4.73$       & $10.30$        & $\bf{10.67}$ & $12.45$       & $6.28$       & $\bf{15.19}$  \\
ABIPNN$_3$            & $7$    & $513$      & $9.22$M    & $30.6$  & $\bf{5.13}$  & $\bf{10.70}$   & $9.41$       & $\bf{13.93}$  & $\bf{7.20}$  & $14.87$  \\
DCGN                  & $7$    &  ---       & $7.76$K    & $96.7$  & $5.01$       & $10.18$        & $9.23$       & $13.01$       & $6.83$       & $13.65$  \\
CNN                   & $7$    &  ---       & $7.87$K    & $51.2$  & $4.81$       & $9.85$         & $8.95$       & $12.43$       & $6.74$       & $12.71$  \\
\hline
\end{tabular}
\end{sidewaystable} 



In our experiments, we use the Demixing Secret Database (DSD100), which was used in the Signal Separation Evaluation Campaign (SiSEC) in 2016 \cite{sisec16mus,liutkus17springer}. It is made up of $100$ full-track professionally-produced music recordings of different styles. It can be used for training and testing for source separation algorithms, because it includes both the stereophonic mixtures and the original stereo sources. The database is divided into a development set and a test set by the organizers of SiSEC 2016, each consisting of $50$ songs. The duration of the songs ranges from 2'22'' to 7'20'', and the average duration is 4'10''. All the songs are sampled at a sampling rate of $44\,100$ Hz. To reduce computational cost, all songs are downsampled to $8000$ Hz. For each song, we compute STFT with a $1024$-point window and a $512$-point hop size. We set the activation function to the rectified linear unit (ReLU) and the learning rate to $3\times10^{-5}$ with exponentially decay reduced by $10\%$ every $300$ epochs. 

Then we map each t-f unit of the magnitude spectrum to an $N$-dimensional vector to serve as the input to ABIPNN. 
Among the training frames, 5\% of the tensor frames are randomly picked as the validation data. The training epoch is set to $1000$ and the training process is stopped if the MSE of the validation set does not decrease for $20$ epochs. The performance is measured in terms of source to distortion ratio (SDR), source to interferences ratio (SIR), and source to artifact ratio (SAR), as calculated by the Source Separation (BSS) Eval Toolbox v3.0 \cite{vincent06taslp}. 



Table~\ref{tab:dsd100_result_with_nn} shows some details of the evaluated methods and their results. Each model has three hidden layers. In order to obtain the same internal dimensionality corresponding to real-valued neurons of DNN-concat and vector-valued neurons of ABIPNN, we use $N$ times more neurons per layer for DNN-concat than ABIPNN.


In Table~\ref{tab:dsd100_result_with_nn}, the two vector NNs (i.e., VPNN and ABIPNN) indeed outperform the conventional DNN methods (i.e., DNN-simple and DNN-concat), demonstrating the effectiveness of considering the interactions between frames. DNN-concat$_1$, VPNN and ABIPNN all outperform DNN-simple by a great margin in SDR. And, in terms of SAR, we see VPNN and ABIPNN perform much better than DNN-concat$_1$, despite that DNN-concat$_1$ has more parameters. This shows that vector neurons can take better advantage of the information provided by the temporal context.



\begin{figure}
\begin{center}
\includegraphics[trim=0.95cm 1.35cm 1.75cm 0.63cm, clip, scale=0.7]{ch4_experiments/sisec_competition.eps}
\caption{Vocal and accompaniment results (in SDR) for the development part and test part of the DSD100 dataset. The methods are sorted by the median SDR for the test part. For the result of ABIPNN, the value of $N$ is set to $7$. Please note that we only consider methods that did not use any data augmentation here, for fair comparison.
}
\label{fig:sisec_competition_result}
\end{center}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[trim=0.3cm 0.4cm 0.3cm 0.5cm, clip, scale=0.7]{ch4_experiments/sisec_wilcoxon_test.eps}
\caption{Wilcoxon signed-rank test results for SDR vocals and accompaniments. The upper triangle represents the comparison of the test set and the lower triangle is for the development part. For the result of ABIPNN, the value of $N$ is set to $7$. Values of $p\mathrm{-\textrm{value}}>1e-04$ indicate no significant difference between the two group results.}
\label{fig:sisec_wilcoxon_test}
\end{center}
\end{figure}


Furthermore, we also compare the computation times of DNN-concat, VPNN and ABIPNN with an NVIDIA 1080 Ti GPU. It is obvious that the computation time of ABIPNN is comparable with DNN-concat given the same value of $N$. 
% The difference of computation time between ABIPNN and DNN-concat increase as $N$ goes larger. However, it is alleviated by using multiple GPUs. 
% The corresponding investigation of computation analysis is discussed in Section~\ref{subsec:complexity_analysis}.

While we can only use $N=3$ for VPNN, we can use larger $N$ for ABIPNN. From Table~\ref{tab:dsd100_result_with_nn}, we see that the performance difference between ABIPNN and DNN-concat increases as $N$ goes larger, despite that the latter architecture has more total parameters. With $N=7$, the SDR obtained by ABIPNN reaches 5.13 dB for the vocal part, which outperforms DNN-simple by $0.76$ dB. As can be seen later from Fig.~\ref{fig:sisec_competition_result}, such a performance gap is remarkable. 

On the other hand, we also compare the performance between DCGN and CNN by using the similar number of parameters when $N=7$. Both of them consist of three fully convolutional layers (without any pooling layers) and the number of kernel maps are 24 for DCGN and 61 for CNN, respectively. As shown in Table~\ref{tab:dsd100_result_with_nn}, the performances of DCGN and CNN are comparable to ABIPNN by using less number parameters\footnote{We also try to increase of the number of parameters of DCGN, but it is hard to get a complete training procedure due to the cost of computation time. We will complement the experiments of using DCGN with more parameters in the future when we make the training more scalable to higher values of $N$.}. DCGN performs better than CNN although the computation time of DCGN is longer than CNN. 

\begin{figure}
\begin{center}
(a) 009 - Bobby Nobody - Stitch Up \\
\vspace{2mm}
\includegraphics[trim=1.8cm 0.3cm 1.5cm 0.3cm, clip, scale=0.65]{ch4_experiments/svs_009_spect.eps} \\
\vspace{2mm}
(b) 039 - Swinging Steaks - Lost My Way \\
\vspace{2mm}
\includegraphics[trim=1.8cm 0.3cm 1.5cm 0.3cm, clip, scale=0.65]{ch4_experiments/svs_039_spect.eps}
\caption{Examples of singing voice separation employing ABIPNN on the test part of the DSD100 dataset.}
\label{fig:svs_examples}
\end{center}
\end{figure}



In Fig.~\ref{fig:sisec_competition_result}, we compare the median SDR of the vocal and accompaniment parts of ABIPNN with the methods that have been evaluated for SiSEC 2016~\cite{liutkus17springer}, including
% UHL~\cite{uhlich17icassp}, 
NUG~\cite{nugraha16taslp}, STO~\cite{stoter16icassp}, OZE~\cite{ozerov12taslp}, KON~\cite{huang15taslp}, KAM~\cite{liutkus15icassp}, JEO~\cite{jeong17springer}, HUA~\cite{huang12icassp}, 
% GRA~\cite{grais16aes}, 
DUR~\cite{durrieu11jstsp}, CHA~\cite{chandna17lvass}, and RAF~\cite{raffii13taslp}. 
Among them, CHA is based on CNN, and KON, STO, NUG are based on DNNs. For fair comparison, we only select those submissions that are not trained with augmentation data. 
Moreover, we show the vocal and accompaniment SDR here instead of other metrics, for saving space (following~\cite{liutkus17springer}) and because SDR is usually considered to be more important than the other metrics. It is also a convention in SiSEC to show the result for both the development and the test sets. From Fig.~\ref{fig:sisec_competition_result}, we see that ABIPNN outperforms all the other methods. From the report~\cite{liutkus17springer}, our results of ABIPNN are comparable to UHL~\cite{uhlich17icassp}, which applies data augmentation and a complicated NN architecture.


In order to show the effect of different methods on SDR values, we follow~\cite{simpson16espc} and apply the Wilcoxon signed-rank test (two-tailed and Bonferroni corrected) for pairwise comparisons. From Fig.~\ref{fig:sisec_wilcoxon_test}, we can see that the group of results from ABIPNN has significant differences over other methods.

Finally, Fig.~\ref{fig:svs_examples} shows the spectrograms of the input mixture, the separation results by ABIPNN, and the original sources, for two songs randomly picked from the test set of DSD100. We see that the separation results (marked as ``predicted vocal'' or ``predicted accompaniment'') resemble the original sources.





\newpage

%---------------------------------------------------------------
\section{Experiment on color image inpainting}
\label{sec:ch4_image_inpainting}

Image inpainting is a task that aims to recover corrupted or missing pixels in an image~\cite{bertalmio00cgit,xie12nips,guillemot14spm,iizuka17tog}, and it is essential in computer vision. Many applications, such as photo editing, image-based rendering and computational photography are based on inpainting algorithms to reconstruct the missing parts.
The main challenge of image inpainting can be divided into two categories. The first one is to manipulate an existing image. Some image details or larger regions of a given image should is removed and then we have to create a plausible missing part to complete the image. The target is not to recover a true image but the filled part should looks realistic. The second one is to consider that the given image is locally corrupted. The corruption is small, and sometimes possibly all over the image. The goal is to recover the missing part that are coherent with original given ones.
Figure~\ref{fig:image_inpainted_examples}(a) shows three example images that need to be inpainted. We have to remove the imposed text and fill pixel values that are consistent with the surrounding context.


Many methods have been proposed to process image inpainting. Traditional diffusion-based methods~\cite{bertalmio00cgit,chan01vcir,masnou98icip} propagate the local information, such as edges and gradients, from the background regions to the missing pixels. Patch-based~\cite{efros01cgit,kwatra05tog} methods perform by searching for relevant patches from the non-hole regions or other source images of the image in an iterative fashion. Recently, deep learning based methods emerge as a promising scenario for image inpainting. Deep neural networks are proposed for image denoising and inpainting~\cite{xie12nips,kohler14mask}. Then CNNs are used to recover the local corruption~\cite{eigen13iccv,ren15nips}. Context encoders proposed by~\cite{pathak16cvpr} train deep neural networks for inpainting large holes by using the objective function comprised of reconstruction loss and generative adversarial loss, based on generative adversarial network (GAN). Then, \cite{iizuka17tog} propose both global and local discriminators as adversarial losses to improve the performance of~\cite{pathak16cvpr}. More recently, contextual attention layer with GAN is proposed by ~\cite{yu18arxiv} to attend on relevant feature patches for image inpainting.



\begin{figure}
\begin{center}
\includegraphics[trim=2cm 4cm 1.5cm 4.0cm, clip, scale=0.85]{ch4_experiments/5_165_Images_512.eps}
\includegraphics[trim=2cm 4cm 1.5cm 4.0cm, clip, scale=0.85]{ch4_experiments/5_293_Images_512.eps}
\includegraphics[trim=2cm 4cm 1.5cm 4.0cm, clip, scale=0.85]{ch4_experiments/5_325_Images_512.eps} 
\includegraphics[trim=2cm 4cm 1.5cm 4.0cm, clip, scale=0.80]{ch4_experiments/2_1038_Images_512.eps} 
\includegraphics[trim=2cm 4cm 1.5cm 4.5cm, clip, scale=0.80]{ch4_experiments/2_272_Images_512.eps} \\
(a) Noisy ~~~~~~~~~~~~ (b) DCGN ~~~~~~~~~~~ (c) CNN$_2$ 
\caption{
Examples of three contaminated images, and the result of DCGN and a conventional CNN model (CNN$_2$) for image inpainting. DCGN performs better than CNN$_2$ in removing the imposed text, as also shown in Table \ref{tab:image_inpainting_table}.}
\label{fig:image_inpainted_examples}
\end{center}
\end{figure}

% \newpage




From above descriptions, more and more inpainting approaches are presented by using deep learning. As a result, we evaluate our proposed method on the color inpainting task ($N=3$), based on a given locally corrupted image and then recover it to the original one in a supervised setting.
We use the images from the LabelMe dataset~\cite{russell08labelme} for this evaluation. The dataset is split into two non-overlapping sets: the training set contains 2,920 images, and the test set 1,133 images. To prevent overfitting, we randomly select from the training set 150 images as the validation set. To reduce the training time, we resize all the images from 2,592$\times$1,944 to 512$\times$512. We contaminate all the images by the same text with variant fonts~\cite{ren15nips}, using the contaminated ones as the model input and the clean ones as the target output. For all the models, we set the maximal number of training epochs to 100, and stop the training process when the validation MSE does not decrease for consecutive 10 epochs. Since the color images only have three dimensions (i.e., RGB), we follow the approach of \cite{gaudet17arxiv} and set the fourth dimension to all zeros for DQN to provide a fair comparison with the quaternion algebra.

 
\begin{table}
\centering
\caption{\textsc{The PSNR~(in dB) and SSIM obtained by different methods for color image inpainting. `Kernels' stands for the number of neurons per layer, and `Parms' the total number of parameters. DQN is based on \cite{gaudet17arxiv}.}}
\small
\begin{tabular}{|l|ccc|cc|}
\hline
\textbf{Method} &$\bm{N}$ &\textbf{Kernels}  &\textbf{Param.}  &\textbf{PSNR}  &\textbf{SSIM} \\
\hline
\hline
Input noise     &---     &---   &---     &14.23       &0.65 \\
CNN$_1$         &3       &24    &11.7K   &29.50       &0.90 \\
CNN$_2$         &3       &41    &32.6K   &30.11       &0.91 \\
CNN$_3$         &3       &72    &97.4K   &30.86       &0.93 \\
\hline
DCGN            &3       &24    &32.4K   &\textbf{31.72}  &0.94 \\
% DQN             &4       &21    &33.5K   &31.56       &0.94 \\
\hline
DQN~\cite{gaudet17arxiv} &4       &24    &43.9K   &31.62       &0.94 \\
\hline
\end{tabular}
\label{tab:image_inpainting_table}
\end{table}


\begin{figure}
\begin{center}
\includegraphics[trim=0.1cm 0.03cm 0.5cm 0.1cm, clip, scale=0.7]{ch4_experiments/image_inpainting_trend.eps} \\
\caption{The PSNR~(in dB) of DCGN and CNN for image inpainting, using different number of kernel maps per layer. We use the same mark to indicate the cases where DCGN and CNN have the same total number of parameters.}
\label{fig:image_inpaint_trend}
\end{center}
\end{figure}
 
 

Table~\ref{tab:image_inpainting_table} shows that DCGN and DQN perform generally better than the conventional CNNs, suggesting the benefit of vector neuron learning.
%In the state of the identical number of kernel maps, DCGN and DQN perform better than CNN$_1$, consisting of same number of kernel maps with DCGN. 
%Since the kernel maps of DCGN and DQN are all three-way tensors, which provide more parameters to learn mapping information. 
While the results of DCGN and DQN are comparable, the result of DCGN is significantly better ($p\mathrm{-\textrm{value}}<0.01$) than even CNN$_3$ under the paired $t$-test for both metrics.
%The \emph{t}-test between DCGN and CNN$_1$ shows positive. In terms of the number of parameters, although DCGN and CNN$_2$ have approximate number of parameters, DCGN still performs better since association between different dimensions is learned by performing circular convolution. 
%The \emph{t}-test between DCGN and CNN$_2$ shows positive. Even though DQN and CNN$_3$ have more parameters than DCGN, DCGN is still slightly superior to both of them, and the \emph{t}-test between DCGN and CNN$_3$ shows positive. The PSNR are improved from 14.23 to 31.72 dB and the SSIM is improved from 0.64 to 0.94. 
Moreover, we note that DCGN has fewer total parameters than DQN, yet DCGN obtains the highest mean PSNR 31.72 dB.




%show some inpainted examples by DCGN and CNN$2$. There are still some afterimages of imposed texts in the inpainted results of CNN$_2$. This shows interaction between different dimensions provided by DCGN leads to achieve better performance of image inpainting. 

Some inpainting results of DCGN and CNN$2$ are shown in Figures~\ref{fig:image_inpainted_examples}. Both methods can remove the imposed text, but DCGN performs slightly better. 
Figure~\ref{fig:image_inpaint_trend} further shows that the result of both DCGN and CNN increases as a function of the number of kernels per layer, but DCGN consistently outperforms CNN.
%the comparison trend of PSNR values performed by different fully convolutional architectures of DCGN and CNN$_2$, where the number of hidden layers of both are set to 3 and identical makers stands for they have approximate number of parameters. By adjusting different number of kernel maps, we can observe that DCGN always performs better, which implies that DCGN could use less parameters to achieve the performance of CNN$_2$. 




\begin{sidewaysfigure*}
\begin{center}
\includegraphics[trim=4.7cm 4.5cm 9cm 4.2cm, clip, scale=0.58]{ch4_experiments/feature_map_illustration.eps} \caption{Illustration of what DCGN learns. For this illustration, we use five kernels for each of the three layers in DCGN. We show the output of each neuron on the left hand side, and the summation of them on the rightmost column. 
%the feature maps are printed out from the DCGN architecture with three fully convolutional layers consisting five feature maps of each layer. 
We can see that DCGN learns to separate the image from the occluding text, and that the summation of the feature maps becomes more and more noise-free layer by layer.
%From the feature maps, complement masks, such as feature map 4 in each layer, are learned with color background, which is used to fill the black holes contributed by the imposed texts. On the right hand side, the summations of feature maps of each layer can be regarded that DCGN performs inpainting results of each layer, especially in layer three.
We note that the final output of DCGN would actually be a weighted sum of the feature maps.
}
\label{fig:image_inpainted_hidden_result}
\end{center}
\end{sidewaysfigure*}


Finally, we show in Figure~\ref{fig:image_inpainted_hidden_result} what DCGN might learn for the inpainting task. For this investigation, we use a three-layer DCGN with only five neurons per layer for simplicity. Figure~\ref{fig:image_inpainted_hidden_result} shows the output (i.e., feature map) of each neuron for a given image, as well as the direct sum of these outputs per layer.
Because the feature maps are all tensors with $N=3$, they can be viewed as color images. 
We see that DCGN learns to separate the original image from the imposed text, and that the summation of the feature maps becomes more and more noise-free layer by layer.
%From the feature maps, complement masks, such as feature map 4 in each layer, are learned with color background, which is used to fill the black holes contributed by the imposed texts. On the right hand side, the summations of feature maps of each layer can be regarded that DCGN performs inpainting results of each layer, especially in layer three.
%We note that the final output of DCGN would actually be a weighted sum of the feature maps.
%and print out all feature maps from each layer. We can see the complement masks are learned with color background to fill the black holes contributed by imposed texts. Then, after we perform summation of feature maps in each layer, the inpainting process can be observe. It is evident especially in the layer $3$. 




%---------------------------------------------------------------
\section{Multispectral image denoising using DCGN}
\label{sec:multispectral_image_denoising}

A multispectral (or hyperspectral) image is composed of a collection of monospectral (greyscale) images, each captured with a specific wavelength~\cite{burns96}. These monospectral images can be thought of as in different \emph{bands} of the multispectral image. Since some associations may exist among different bands, we can improve the performance of image processing applications by leveraging such associations~\cite{peng14cvpr}.


In multispectral image denoising~\cite{zhang16ijcai,xie2017grsl}, we are given a noisy version of a multispectral image with $N$ bands, and are asked to restore the clean version (also $N$ bands). %Recently, many different supervised methods were evaluated on 
We use the Columbia multispectral image database~\cite{yasuma10itip} for this evaluation.
%the denoising task. For example, PARAFAC~\cite{liu12grs} is proposed to denoise multispecral images by extracting the major components from the global correlated spectrum information. DNMDL~\cite{peng14cvpr} is a Tucker dictionary learning based method. Similarly, TDL~\cite{peng14cvpr}, K-TSVD~\cite{zhang16ijcai}, and $3$D K-SVD~\cite{elad06ip} are also based on dictionary learning methods. Deep learning method~\cite{xie2017grsl} is proposed to reconstruct the original ones from noisy ones by using CNNs. The performance reported from~\cite{xie2017grsl} seems better than conventional methods. 
It contains 32 real-world scenes and each scene is made up of 31 monochrome images of size 512$\times$512, captured by varying the wavelength of a camera from 400nm to 700nm with a step size of 10nm. We randomly select 16 scenes for the training set, and the rest for the test set. We pick images of the last 3--6 bands to make $N \in [3,6]$. Moreover, we divide each image into 32$\times$32$\times N$ overlapping tensor patches with hop size being one. Then, we randomly select approximate 20,000 tensor patches as the training data, 1,000 of which are held out as the validation data to prevent overfitting. For each tensor patch, all the $N$ bands are processed at the same time. We set the maximal number of training epochs to 30 and stop the training process if the validation MSE does not decrease for 5 consecutive epochs.


To create a noisy image, we randomly pick a certain ratio of pixels from each band to add Gaussian noise with specific~\emph{sigma} value. 
We refer to this ratio as the \emph{sparsity} below. We vary the sparsity from 5\% to 15\% and sigma from 100 to 200 to simulate different degrees of corruption. In model training, the MSE is computed between the restored and the original versions of the patches across all the $N$ bands. We vary $N$ from 3 to 6 for DCGN, and set $N=4$ for CNNs and DQN.


 
Table~\ref{tab:multispectral_image_denosing_table} shows the result of multispectral image denoising. The following observations can be made.
First, comparing the result of the two CNNs and DCGN~($N=4$), we see again DCGN outperforms CNNs for a regression task.
The performance difference between DCGN and CNN$_2$ is significant ($p$-value$<$0.01) under the \emph{t}-test for the first four evaluation scenarios.
The fifth evaluation scenario (i.e., 10\% sparsity with a sigma of 200) is the most noisy scenario but DCGN still performs better.
%indicating that DCGNs are superior to all the other methods, including the two variant CNN baselines.
Figure~\ref{fig:multispectral_image_denosing_examples} demonstrates the denoised result of DCGN.  


Second, Table~\ref{tab:multispectral_image_denosing_table} indicates that the performance of DQN \cite{gaudet17arxiv} is also much better than CNN$_1$. However, it is only comparable to CNN$_2$, which uses more kernels per layer but the same overall total number of parameters. Comparing the result of DCGN~($N=4$) and DQN, it seems DCGN is more effective, possibly owing to the use of cyclic group algebras.

%When $N$ equals four, DCGN--$4$ and DQN performs better than two CNN baselines, where CNN$_1$ has the same number of feature map per layer as DCGN--$4$ and CNN$_2$ has similar number of parameters as DCGN--$4$ and DQN. While the results of DQN and CNN$_2$
%are comparable, the results of DCGN--$4$ are superior ($p$-value$<$0.05) to CNN$_2$ and DQN under the paired \emph{t}-test for both matrices when the sparsity is $10\%$ and sigma is set to $100$. The results imply learning more information by vector neuron learning is beneficial to denoising and verifies the concept is meaningful that fibers, viewed as a one-dimensional mask in a kernel map, capture relations among $N$ dimensions in a feature map by performing the circular convolution (as described in Figure~\ref{fig:dcgn_explain}). 

Third, comparing the result of different DCGN models in Table~\ref{tab:multispectral_image_denosing_table}, we see that the performance in PSNR improves as a function of $N$, except for the two noisier scenarios. This nicely demonstrates the importance of having a vector neural learning model that can accommodate arbitrary data dimensions. 
%DCGNs have good learning ability in any dimension. Especially DCGN--$3$ and DCGN--$6$ have good performance when sigma and sparsity values become larger. Moreover, we also observe the trend that when $N$ becomes larger, the performance could be elevated. 
%We conjecture the performance is improved  since more information from different dimension is learned by DCGNs. 
The dependency across different data dimensions may increase when we have more dimensions, and DCGN can exploit such dependency to improve the result of a machine learning task.

Finally, we also employ ABIPNN on this task by using the similar number of parameters when $N=6$. The architecture of ABIPNN is made up of three hidden layers and each hidden layer consists 30 neurons. As shown in Table~\ref{tab:dsd100_result_with_nn}, the performance of DCGN outperforms ABIPNN when $N=6$. It shows that DCGN has an better ability to learn features by using less parameters when comparing to ABIPNN.


\begin{sidewaystable*}[t]
\centering
\caption{\textsc{PSNR~(in dB) and SSIM obtained by different methods for multispectral image denoising under different sparsity~(in \%) and sigma values. We can use different number of bands for denoising, leading to different data dimensions $N$.}}
\small
\begin{tabular}{|l|ccc|cc|cc|cc|cc|cc|}
\hline
\multirow{2}{*}{\textbf{Method}} &\multirow{2}{*}{$\bm{N}$} &\multirow{2}{*}{\textbf{Kernels}}  &\multirow{2}{*}{\textbf{Param.}}  &\multicolumn{2}{c}{\textbf{(5\%, 100)}}  &\multicolumn{2}{c}{\textbf{(10\%, 100)}} &\multicolumn{2}{c}{\textbf{(15\%, 100)}} &\multicolumn{2}{c}{\textbf{(10\%, 150)}} &\multicolumn{2}{c|}{\textbf{(10\%, 200)}}\\
\cline{5-14}
               &      &      &         &\textbf{PSNR}      &\textbf{SSIM}     &\textbf{PSNR}     &\textbf{SSIM}        &\textbf{PSNR}      & \textbf{SSIM}    &\textbf{PSNR}       &\textbf{SSIM}    &\textbf{PSNR}    &\textbf{SSIM}    \\
\hline
\hline
Input noise     &---   &---   &---      &21.23   &0.72   &18.35  & 0.59    &16.68  & 0.50   &14.81   &0.53  &12.30  &0.52 \\
CNN$_1$         &4  &24   &12.2K   &42.07   &0.97   &40.62  & 0.95    &39.06  &0.94   &38.97   &0.95  &39.24  &0.94 \\
CNN$_2$         &4  &48   &45.1K   &45.07   &0.98   &42.18  & 0.97    &41.02  &0.96   &42.07   &0.97  &42.01  &0.97 \\
CNN$_3$         &4  &83   &130.2K  &43.82   &0.97   &41.71  & 0.97    &41.57  &0.97   &43.04   &0.97  &41.49  &0.96 \\
\hline
DCGN~($N=3$)    &3  &24   &32.4K   &48.24   &0.99   &46.00  &0.98    &45.02  &0.98   &46.97   &0.98  &44.93  &0.98 \\
DCGN~($N=4$)    &4  &24   &43.3K   &48.61   &0.99   &46.58  &0.98    &45.42  &0.98   &46.02   &0.98  &43.62  &0.97 \\
DCGN~($N=5$)    &5  &24   &54.0K   &48.96   &0.99   &46.97  &0.98    &45.55  &0.98   &44.60   &0.98  &45.03  &0.98 \\
DCGN~($N=6$)    &6  &24   &64.8K   &50.26   &0.99   &47.19  &0.98    &45.63  &0.98   &45.45   &0.98  &46.31  &0.98 \\
ABIPNN~         &6  &--   &65.3K   &46.12   &0.96   &45.65  &0.95    &44.23  &0.95   &43.86   &0.94  &43.01  &0.94 \\
\hline
DQN~\cite{gaudet17arxiv}  &4  &24   &43.5K  &45.15   &0.98   &43.20  & 0.97    &42.29  & 0.97   &42.61   &0.97  &41.57  &0.97 \\
\hline
\end{tabular}
\label{tab:multispectral_image_denosing_table}
\end{sidewaystable*}


\begin{figure}[h]
\begin{center}
``Superballs'' Image \\
\includegraphics[trim=2cm 4cm 1.5cm 4.5cm, clip, scale=0.85]{ch4_experiments/14_superballs_ms.eps} \\
``Hairs'' Image \\
\includegraphics[trim=2cm 4cm 1.5cm 4.5cm, clip, scale=0.85]{ch4_experiments/9_hairs_ms.eps} \\
``Fake and real food'' Image \\
\includegraphics[trim=2cm 4cm 1.5cm 4.5cm, clip, scale=0.85]{ch4_experiments/4_fake_and_real_food_ms.eps} \\
(a) Original ~~~~~~~~~ (b) Noisy ~~~~~~~~~ (c) Denoised 
\caption{Examples of three noisy images (at 700nm band) and the multispectral image denoising result by DCGN with $N=4$. The sparsity of the noisy pixels is 10\% and the sigma value of the additive Gaussian noise is set to 200.}
\label{fig:multispectral_image_denosing_examples}
\end{center}
\end{figure}
