I have a remote Plex library (plex share) that has around 60,000 pieces of content. I’ve seen previous threads outlining that Infuse struggles to handle and display such large libraries however these threads are from quite a while ago. My personal experience is that it seems Infuse simply never manages to get any (or all) content from my plex share… its stuck “Fetching Content”.
Am I doing something wrong or am I out of luck, or do I need to change some settings?
No, you’re not doing anything wrong. Probably.
It does take several hours for Infuse to process all what needs to be processed. I don’t know what exactly is processed (metadata? mediainfo? …) and therefore can’t make an estimation on what is causing the significant delays…
It’s quite frustrating tbh and in my opinion it has gotten worse during the last releases of Infuse.
I can give you the hint to navigate to Settings > library (on Apple TV) and in the lower left corner, after a while, you‘ll see numbers of total media items and the progress being made.
At this point I am idling in this very menu and move the cursor up/down every now and then to prevent the ATV to switch off. The progress which is made processing all the items, doesn’t get lost. Which is relieving… However, from what I was able to monitor, Infuse does not seem to continue processing items in the background or when playing a movie… so idling the menu it is… 
I very much hope that this is being looked at by the devs. A PLEX server in the background, which holds all the metadata already, should be more or less able to somewhat instantaneously be ready to go in Infuse…
Also, look at this topic maybe
Sadly (and to add to what @tbor85 said) it’s the way it is.
Are you invested in Plex for usage outside of using Infuse as a client?
Emby and Jellyfin have a plug-in (Infuse-Sync) that makes anything happening on those servers push immediately to Infuse. (Neither Emby or Jellyfin are anywhere near as pretty as Plex though sadly.)
Plex have zero interest in helping to integrate 3rd party stuff unfortunately so the option isn’t available for Plex.
Basically it’s a Plex issue and their totally locked down approach.
So does Infuse have better luck handling a large Emby library for example?
Yes. An Emby (Also available on Jellyfin) plug-in more or less tells infuse what has changed.
So Infuse doesn’t need to scan the full library to detect changes.
We have a number of improvements planned which will help with larger media server libraries.
In the meantime, using Emby or Jellyfin with the InfuseSync plugin installed will be a better option compared to Plex (which does not support plugins like InfuseSync).