After further experimentation, the issue of the controls no longer coming back only happens with images which are larger than the screen.
It turns out that I can navigate backward if I swipe right from the very edge of the screen. I didn’t know about that gesture as my phone case has raised edges around the screen, and that gesture doesn’t have much tolerance on how close to the screen edge the gesture must begin. That makes it difficult to discover.