Accelerated Algorithms for Nonlinear Matrix Decomposition with the ReLU function

arXiv2305.08687MaRDI QIDQ6436651

Author name not available (Why is that?)

Publication date: 15 May 2023

Abstract: In this paper, we study the following nonlinear matrix decomposition (NMD) problem: given a sparse nonnegative matrix $X$, find a low-rank matrix $Theta$ such that $Xapproxf(Theta)$, where $f$ is an element-wise nonlinear function. We focus on the case where $f(cdot)=max(0,cdot)$, the rectified unit (ReLU) non-linear activation. We refer to the corresponding problem as ReLU-NMD. We first provide a brief overview of the existing approaches that were developed to tackle ReLU-NMD. Then we introduce two new algorithms: (1) aggressive accelerated NMD (A-NMD) which uses an adaptive Nesterov extrapolation to accelerate an existing algorithm, and (2) three-block NMD (3B-NMD) which parametrizes $Theta=WH$ and leads to a significant reduction in the computational cost. We also propose an effective initialization strategy based on the nuclear norm as a proxy for the rank function. We illustrate the effectiveness of the proposed algorithms (available on gitlab) on synthetic and real-world data sets.

Has companion code repository: https://gitlab.com/ngillis/relu-nmd

This page was built for publication: Accelerated Algorithms for Nonlinear Matrix Decomposition with the ReLU function

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6436651)