Computation in neural networks relies crucially on non-linearity. In neural networks in the balanced state the non-linearity of the neuronal transfer function becomes functionally unimportant. The disappearance of this non- linearity strongly limits the computational power of balanced networks. I will show in examples of balanced networks for associative memory how one can try to circumvent this limitation of balanced networks and discuss the problems with these solutions.