Sigmoid Activation Function

A sigmoid activation function has a characteristic "S"-shaped curve defined, using e (Euler's Number), by the formula:

$\\ f\left ( x \right ) = \frac{1}{1+e^{-x}}$

The curve produced has a fairly gradual rise:

The derivative of the function is:

$\\ f'\left ( x \right )=f\left ( x \right )\left ( 1-f\left ( x \right ) \right )$

Common negative comments about the sigmoid activation function include:

Sigmoids can saturate and kill gradients. Gradients (change) at the tails are almost zero.
Sigmoid outputs are all positive values. This can bias network results. The effect can be mitigated by not using sigmoids in the final layers of a network.

References