Series Collections keeps refreshing, very slow

I’ve been using Infuse for the last few days and its amazing. I love how it gets metadata for TV and films.

However one thing I am having issues with is that the app keeps having to refresh my TV shows. For example, I have all 18 series of South Park and I have to leave it for about 5 minutes for it to process all the video files and then put it into series folder. I think it is collecting all the metadata but it has already done this once, why does it keep doing it? I’m happy with it doing it once but when it keeps doing it its slow and frustrating. Is this normal behaviour or am i missing something?