Even the smallest of files and folders are taking an incredibly long time to load now. I’m talking minutes, not seconds, for what used to be almost instantaneous. It even takes an eternity to load the folder list while trying to do a speed test.
How are the Apple TV and Mac connected to the network (Wired or Wi-Fi)?
One thing you can try is to adjust the SMB version in Infuse. Setting this to SMB2 may provide faster speeds (as it bypasses the SMB3 related encryption features which may be enabled on the server which can slow things down).