Do you have any control of image processing settings on the atv4k? I suspect there are some sharpening processing on the atv that is causing a grainy noisy picture in some 4K uhd movies. For example fifth element. I’m watching on an LG oled 55 b7.
Grain isn’t a bad thing so shouldn’t necessary affect review scores. It’s even being artificially added in the recent marvel shows on netflix (though I have to say that I don’t like it’s use their). You’ll almost certainly see grain on 4k releases of older movies as it’s present in the source. I’d recommend turning your TV sharpness setting down to zero as this can exacerbate grain but if you’re already at zero then you’re seeing the film as it should be seen. I’m confident that Infuse isn’t causing it though…it certainly doesn’t on any 4k sources I’m viewing and I have a very similar tv to yours (65E6)