Left / right changes tracks, the UI is not built with a pager for full screen other pages, the pager is for the tracks.
Use a different now playing like Fill dynamic cover and see that swipe left / right looks just fine.
Swipe down is to close the player since it uses a different paradigm.