I have buffering problems as well.
My setup is:
LaCie USB 3.0 2TB connected to a Mac mini M1 that is connected to the router through LAN with a CAT8 cable and so is Apple TV 4K (2nd gen.).
I tried to watch two different movies and I had buffering issues every 1-2 minutes.
It’s completely unusable.
External HDD format was APFS, now switching to NTFS to directly connect it to my router (ASUS DSL-AX82U) and then I’ll try again.
PS: switching between Streaming Cache options didn’t solve the issue.