Remote plex initial buffering

Adding a remote plex server to infuse and I am having an issue where it will take over a minute to load any video. The same device, using the plex app, will load the same video in a few seconds. This is the same across 3 different devices, Apple TV, iPad and iPhone. Alternatively, when trying the same device but adding Emby as a library, infuse will load in just a few seconds. So I am seeing 1 min plus load times for a plex share only with infuse, all regular apps are loading within a few seconds. Speedtest to the remote server is consistently 800Mbps+