$\newcommand{\in}{\text{in}}$
$\newcommand{\out}{\text{out}}$
$\newcommand{\prt}{\partial}$



Bias Block
==========

.. _fig-nn_bias:
.. figure:: /figures/nn_bias.png
   :align: right
   :figwidth: 40%

   **Bias Block** $\v v = \v u + b$


The forward pass of the bias block is simple:

.. math::
   \v v = \v u + \v b

The graph for this block is given in :numref:`fig-nn_bias`.

it simply adds a vector $\v b$ to the input. Evidently this block
passes the the derivative of $\ell$ with respect to its output
directly to its input:

.. math::
   \pfrac{\ell}{\v u} = \pfrac{\ell}{\v v}

and also to the parameter vector:

.. math::
   \pfrac{\ell}{\v b} = \pfrac{\ell}{\v v}

we leave the proofs as exercises for the reader (again the easiest way
to do so is by considering all elements of the vectors).

The vectorized expressions are:

.. math::
   V = U + \v 1 \v b\T

note that $\v 1\v b\T$ constructs a matrix in which all the rows are
$\v b\T$. And also note that Numpy is smart enough to broadcast this
correctly in case we write :code:`V = U + b` where :code:`V` is an
array of shape :code:`(m,n)` and :code:`b` is an array of shape
:code:`(n,)`.

For the backpropagation pass we have:

.. math::
   \pfrac{\ell}{U} = \pfrac{\ell}{V}.

For the derivative of $\ell$ with respect to parameter vector $\v b$
we have to sum all $\prt\ell/\prt \v b = \prt\ell / \prt \v v\ls i$ for all examples in the
batch:

.. math::
   \pfrac{\ell}{\v b} = \sum_{i=1}^{m} \pfrac{\ell}{\v v\ls i}
   = \pfrac{\ell}{V}\T \v 1

where $\v 1$ is a vector with all elements equal to 1. Note that in
Numpy we can simply sum over all columns in array $V$.