Do you have Infuse connected to Plex or are you connecting directly?
Since Infuse doesn’t require a server, many of the tasks a server normal does in the background Infuse will do on its own.
The fetching details part you are seeing likely refers to fetching things like video resolution, specs, runtime, etc… This will be the slowest part of indexing, though if you are using iCloud sync will be saved so it only has to run once (other devices can access this, and it will be available if you switch devices or have to reinstall the app for some other reason).