torchact.nn.ABReLU

class torchact.nn.ABReLU(alpha: float = 1.0, inplace: bool = False)[source]

Implementation of Average-Biased Rectified Linear Unit https://arxiv.org/abs/1804.02051

\(A_{i}^{n}=\frac{\sum_{\rho_{1}=1}^{D_{1}}{\sum_{\rho_{2}=1}^{D_{2}}\cdots\sum_{\rho_{d}=1}^{D_{d}}{I_{i}^{n}(\rho_{1},\rho_{2},\cdots,\rho_{d})}}}{D_{1}\times D_{2}\times\cdots\times D_{d}}\)

\(\beta=\alpha\times A_{i}^{n}\)

\(I_{o}^{n}(\rho)=\begin{cases}I_{i}^{n}(\rho)-\beta,&\text{if }I_{i}^{n}(\rho)-\beta>0\\0,&\text{otherwise}\end{cases}\)

Parameters

alpha (float) – parameter to be set empirically. Default: 1.0
inplace (bool) – In-place operation. Default: False

Examples::

>>> import torch, torchact
>>> m = torchact.nn.ABReLU()
>>> input = torch.tensor([1.0, -2.0, 0.0, 3.0])
>>> output = m(input)
>>> print(output)
tensor([0.5000, 0.0000, 0.0000, 2.5000])