The Jellyfin project recently committed a large number of performance improvements to their master branch. I’ve tested these in the current unstable release and I’m pleased to report that they address the issue described in this topic.
The changes are slated to be released as part of Jellyfin 12.0, which will see an RC release soon, according to the project’s maintainer. (Likely waiting on the items in this list: Jellyfin 12.0 · GitHub)