Very slow speeds with Infuse + Plex

I have Plex installed on my MacBook Pro 16 2023. I’m using Infuse on Apple TV. I’ve connected to the server but when I run speed tests they are weirdly slow (20 Mbps). Almost as slow as my actual internet connection. I’m having a hard time making sense of these results as my router supports WiFi 6 and Plex server doesn’t have remote access (so I know for a fact that it’s using the local network). Does anyone have any insight is to what might be going on, what are the things I should check etc.?