I’m not 100% sure but I believe that if you hit pause when it starts buffering it will continue to buffer and then you can go fill the lions cat nip toys and when you come back you can resume with the play button.
There’s also a setting for “Streaming Cache” that may work better if you set it to Legacy.
There’s a buffer indicator in the time line on tvOS Infuse that you can watch to see how much is buffered.
Since I don’t have a remote server I can’t duplicate the spinning wheel so that’s my bad. I thought it’d still pause even though it was spinning. Maybe if you quit the video when it starts the spinning wheel and restart it and then pause before it spins then you could let it buffer before restarting.