Unable to stream 4K despite capable network

Lately, and unfortunately I don’t have specifics - just some time around when I updated to tvOS 14, I have been unable to stream UHD content and sometimes run into buffering issues with high bit-rate 1080p Blu-rays as well.
I’m using a 64GB Apple TV 4K with a wired connection, and files hosted locally on my network.

When I run the speed test on a file I was having trouble streaming, it averages 170 Mbps and monitoring on the PC reports an average disk access of ~21 MB/s which lines up.
When I try to play the file I run into buffering issues and the PC reports usage closer to 8 MB/s on average - dropping as low as half that, at times.

Even if I wait for 1/4 of the movie to load in advance, it can still run into buffering issues during playback.
Switching to the legacy caching makes no difference.

It should be noted that there’s almost nothing stored on this Apple TV - I bought the 64 GB version intending to sync files locally in Infuse, before finding out that this was only an feature in the iOS version.
The only other thing on there is the YouTube app really.

Wired or wireless? Backend connection method?

it’s the same here:

server: plex wired on 1GB
client: ATV 4K wired on 1GB

speetest: 500Mbit stable

the same files plays (with transcoding!) in the original plex app just fine. it even happens with 1080p files.

problem came up some weeks ago, is not constant. I just learned about the speed test option and will test it with the file in question if it happens again.