We will go into more details about different activation functions at the end of this section. Coarse model. The dendrites in biological neurons perform complex nonlinear computations.
The exact timing of the output spikes in many systems is known to be important, suggesting that the rate code approximation may not hold. Due to all these and many other simplifications, be prepared to hear groaning sounds from anyone with some neuroscience background if you draw analogies between Neural Networks and real brains. See this review pdf , or more recently this review if you are interested.
Binary Softmax classifier. With this interpretation, we can formulate the cross-entropy loss as we have seen in the Linear Classification section, and optimizing it would lead to a binary Softmax classifier also known as logistic regression. Since the sigmoid function is restricted to be between , the predictions of this classifier are based on whether the output of the neuron is greater than 0.