Many lines of evidence suggest that few spikes carry the relevant stimulus information at later stages of sensory processing. Yet mechanisms for the emergence of a robust and temporally sparse sensory representation remain elusive. Here, we introduce an idea in which a temporal sparse and reliable stimulus representation develops naturally in spiking networks. It combines principles of signal propagation with the commonly observed mechanism of neuronal firing rate adaptation. Using a stringent numerical and mathematical approach, we show how a dense rate code at the periphery translates into a temporal sparse representation in the cortical network. At the same time, it dynamically suppresses trial-by-trial variability, matching experimental observations in sensory cortices. Computational modelling of the insects olfactory pathway suggests that the same principle underlies the prominent example of temporal sparse coding in the mushroom body. Our results reveal a computational principle that relates neuronal firing rate adaptation to temporal sparse coding and variability suppression in nervous systems.