I believe that the xml files don’t have the tags nessasary for cast in Infuse but the nfo files do. That’s just a guess from reading other posts though.
Also, here’s a thread that is a suggestion for cast and crew support in nfo files and it shows as planned so you may want to add your support there by clicking the like button on the first post in that thread.