Of course not, because it is not on the screen, it is infinite loaded. You can either increase the default results number or leave it like that. I don’t think there is a way to get around it, it would have to load multiple batches, which takes time, and the all of a sudden it would start scrolling after a few seconds – which would be very annoying.
Some browsers can completely “memorize” the state of the previous page ( I think safari does that ), so when going backwards it does not actually reload the page, but that is decided completely by the browser and it is not yet known how to exactly programmatically achieve that.
In some cases even Chrome did it for me when I was working on the demo, then it simply stopped, and I have not changed anything. I have tried to research this, but there is no usable information on how to force the browsers to do that.