PleX vs Google Drive directly?

I currently have my PleX library setup on a remote server with the files themselves stored on my GSuite Google Drive via a rclone cache setup. With Infuse, I have the option of either still going through PleX, or loading my library directly through my Google Drive account. From a purely performance standpoint, would one route be better than the other? I know there are some other differences, but my primary concern is what will give me the fastest loading times and least amount of buffering.

Surprisingly I get better results and better access time from my Plex Media Server on a VPS than directly from Google drive …

Thanks!