Metadata cache keeps clearing!

So here’s an interesting data point for you.
My 32gb 1st gen AppleTV 4K, which had offloaded all other apps but YouTube, and only had 4 screensavers installed (I thought 1, but discovered three extra after further investigating) … and has been disconnected from the internet for several months now …

Two nights ago notified me my Infuse database had been erased because my Apple TV device was running low on space … despite not being connected to the internet for months and not having had dumped the database previously for many, many, many months prior while I still had the device connected to the internet and also regularly used the device with YouTube, YouTube TV, AppleTV, and various other streaming services.

So, that was unexpected.

I can’t imagine anything that could have filled the device in the interim but a massive accumulation of internal tvOS error messages re: not being able to connect to internet.

Anyone else have any ideas?

1 Like

Now that’s interesting!

1 Like

A good solution would be to implement the ability to use external data storage to generate an infuse cache. If it is of course possible.
@james do you think it is possible to implement such a plan?

Without an Infuse server program running on a users’ NAS (as something like Plex does), I don’t see how that would work because one way or another the metadata database must be mirrored on the AppleTV for Infuse to function.

I hope I’m wrong, but I think the best we can hope for is for Firecore to finally code a way to offload a copy of it’s database to users’ servers — so that when the AppleTV inevitably flushes Infuse’s database, Infuse can quickly copy it back over.

Presumably it already does something similar with iCloud; but I don’t think it caches the artwork or cast & crew there; and rebuilding the database is still subject to available internet bandwidth.

I suppose if it was easy they would have already prioritized it. But I wouldn’t mind hearing an update from @james … and his guess how my cache got flushed from my offline-for-over-3-months Apple TV …

… and an offline AppleTV can’t just rebuild it’s database (regardless the presence of local .nfo). Library and TV season organization flat won’t rebuild per a user’s server folder structure absent a connection to the internet. At all. All TV series in my collection have there episodes broken out into season # subfolders — which the Files Browser completely ignores when showing me their unindexed contents — plopping up to 44 seasons worth of episodes all in one series folder with zero per-season subdivision. Gah!

They refuse to acknowledge this as a problem they should spend their time on.

The primary copy of the metadata file should be stored on the NAS (which everyone of us has already) and then if Infuse wants to it can store a copy of it on the ATV for faster access.

The responses from firecore seem “uppity”.

There is an active thread on your forum, that wont go away, because people want this to work reliably.

Just take a deep breath - stop shaking - and just add the feature.

Walk over to the team and say “hey guys here is what we need to work on” and add the feature. Then tell us you’ve done this.

Remember Apple fighting owners about their crappy MacBook keyboards? Or old Steve having the gumption to say “you’re holding it wrong”. Thats Firecore - right now.

Stop it.

I want to give you a quote that I just made that you can print out as a banner (you do have a dotmatrix printer with a box of paper under it… right?)

Being steadfast in wrongness creates resentment.

Listen to your customers - respond professionally - assure that legitimate, reasonable requests are being addressed.

So dust off that Smash Mouth CD and get to it.

3 Likes

Great statement. James should finally react urgently.

1 Like

Great statement.

While I fully agree on storing the metadata on the NAS (heck, it should be extremely simple from a development standpoint), how about people with PMS or Jellyfish?

@james can we have a final solution where, by default the metadata stays on the Apple TV and, if users want to gain persistency, a share is used for the metadata?

Such solution would:

  1. not break all existing deployments
  2. work with and without Infuse actings as media library manager

I haven’t posted since my original plea for help, but I wanted to give my 100% support to @georgebattle - we know how great the Firecore team is. The problem is, they’ve listened but not heard what people are saying.

I resolved my problem by following the detailed advice from FLskydiver (can’t seem to tag him). He told me that the problem was, the storage space that Infuse uses is not exclusive - Apple uses that space to download its new screensavers. The way Infuses is coded means that its index and other vital stuff is stored in a place that gets wiped when new screensavers arrive.

I love those screensavers. The ones of London make me see my city in a way I’d never even thought of. But the only way to use my app is to get rid of them. Seems like a really bad programming decision.

FLskydiver told me, you gotta fill the entire Apple TV hard drive. Just get as many apps as you can. Then, you have to turn off “download new screensavers”. Then you have to delete all those apps - and I can’t tell you how irritating it was to find, download, and delete them. I wasn’t gonna spend my day trying to find big files, cos imagine how long that could end up taking. So I just had the interminable task of getting 85 apps then deleting 85 apps. Deleting 85 apps is a seriously slow task.

When I did that, my disk was only half full.

And for weeks now, I’ve not had a single cache deletion and data rebuild. Not once.

The number of people who are able to say the same proves to me that yes, this is Apple’s fault, but it is Firecore’s problem to manage.

And I miss my Apple TV screensavers.

I don’t know what the technical issues are. I don’t know if this requires a complete rewrite which would take years and make the company collapse. I don’t know if it could be a part time project that takes a couple of years.

Infuse is my most used app. But I can’t recommend it to others now, cos I can’t say “it’s great but before you can watch any TV programme you will have to wait 25 minutes for the index to rebuild”.

I’m really happy that I’ve “resolved” the issue, but the way I had to do it is ridiculous.

Thank you FLskydiver for making Infuse usable, and thank you to Firecore for making such a great app.

But as @georgebattle says, listen to your users

4 Likes

The most annoying part is that the playing history is also gone when it happens. I won’t be able to continue to play the movie for the kids where it paused!

That should restore from iCloud if you have a subscription.

2 Likes

@TonyCLondon
Thanks for posting! It’s good that you haven’t lost your metadata in a while but you can’t assume that it won’t happen again.

I have done nothing to mitigate the problem and it’s been a long time since I had it clear. However, it can happen at any time, as before. The cache is temporary storage and not for persistent data.

Firecore needs to address this problem.

@sacra_gassier

Losing history and all the corrections I have to the metadata is very, very annoying. It takes me a long time to fix all that. Though, the worst for me is losing subtitles I’ve downloaded.

@FLskydiver
Software such as SQLite is specifically designed for this type of scenario. SQLite is a wonderfully powerful database system that requires nothing but a single file. You don’t need a server or a program! It’s trivial to use and it’s very popular and widespread.

2 Likes

Get a Zidoo, and check how they do it :sweat_smile:
It’s very easy and straight forward.

So I read up on Appropriate Uses For SQLite and while it appears it might be usable as a way to backup Infuse’s vulnerable cached database (as a single file) to an external server — if SQLite is an available resource already resident in tvOS or if it can be integrated into Infuse’s existing code — it does not appear it can facilitate moving Infuse’s active database to an external server.

Situations Where A Client/Server RDBMS May Work Better

  • Client/Server Applications If there are many client programs sending SQL to the same database over a network, then use a client/server database engine instead of SQLite. SQLite will work over a network filesystem, but because of the latency associated with most network filesystems, performance will not be great.
    […]
    A good rule of thumb is to avoid using SQLite in situations where the same database will be accessed directly (without an intervening application server) and simultaneously from many computers over a network.
  • I’m unaware if the latter point applies since multiple Apple devices accessing the primary database probably won’t be rewriting it concurrently, and Firecore may be able to keep any device-specific data confined to it’s protected storage on AppleTV — but latency issues I suspect may be very significant with regard to the UI’s responsiveness compared to having the database located on the Apple device itself.

More:

Checklist For Choosing The Right Database Engine

  1. Is the data separated from the application by a network? → choose client/server

Relational database engines act as bandwidth-reducing data filters. So it is best to keep the database engine and the data on the same physical device so that the high-bandwidth engine-to-disk link does not have to traverse the network, only the lower-bandwidth application-to-engine link.

But SQLite is built into the application. So if the data is on a separate device from the application, it is required that the higher bandwidth engine-to-disk link be across the network. This works, but it is suboptimal. Hence, it is usually better to select a client/server database engine when the data is on a separate device from the application.

[…]

  1. Otherwise → choose SQLite!

For device-local storage with low writer concurrency and less than a terabyte of content, SQLite is almost always a better solution.


You keep saying this, without apparently recognizing that

  1. A Zidoo box is not an AppleTV.
  2. Firecore does not and can not control the codebase, permissions, or allocation of resources on AppleTV as Zidoo does on their own devices.
  3. Apple has zero incentive to facilitate the trouble-free use of apps like Infuse, which directly compete with Apple’s financial interest in both selling its own content to AppleTV users, and its streaming competitors access to those same users; while also collecting a large percentage of the subscription fees charged by those competing streaming services.
  4. Further disincentivizing Apple, Infuse users bring their “own” content to Apple devices — including Apple’s own (likely procured in the majority of cases through non-revenue generating means) — none of which provides a cent of income to Apple. Nor can Apple benefit from eavesdropping on Infuse users or establish a data sharing agreement with Firecore that allows Apple to track consumer viewing habits and content preferences, which Apple can then monetize or utilize to improve its insight into which types of content it should be investing in to increase consumer engagement; ensuring Apple the best possible returns on their investments in specific media properties and the breadth of their overall content library.

The most important thing is that Zidoo has a disgusting user interface, very inconvenient management. Various bugs are fixed very slowly. The Dolby Vision color profile is constantly being broken in updates. In general, in comparison with Apple TV, Zidoo is a very bad media player.
The only plus of Zidoo is the bitstream of sound. All. But, this plus is too small and does not cover all the disadvantages of Zidoo, so that the choice becomes in favor of him.

Following on from the user polls posted by @munpip214 … here’s one of my own, that might help narrow down who is affected and why, and the effectiveness of known mitigation factors:

Part 1

Have you recently experienced a metadata deletion?
  • Yes, on all my Apple TVs
  • Yes, but only on one or more of my Apple TVs
  • No
  • If yes, Did you recently take any measures recommended above to free up space on your device?
  • If no, Did you recently take any measures recommended above to free up space on your device?

0 voters

Please only submit answers to the followings polls which apply to your specific AppleTV device(s) — do not participate in both polls unless you answered “Yes, but …” above, and then only answer each based on the status of your specific device(s) that either have or haven’t recently deleted their Infuse metadata caches. Thanks!


Part 2
ONLY PARTICIPATE IF YOU ANSWERED YES on Part 1.

If you have recently experienced a database deletion:
  • Did you disable screensaver downloads?
  • Did you manually delete unused apps, or allow tvOS to automatically offload unused apps?
  • Did you fill your Apple TV with apps (until no more could be installed) to force deletion of all volatile cached content, or delete all apps and reset your Apple TV?
  • OR did you reset your device but not first delete all apps; and did you resync their cached content from iCloud?
  • Do you use gaming or business apps that cache large amounts of data?
  • Do you download media content to your AppleTV for offline viewing?
  • Do you use Infuse iCloud Sync?
  • Is Infuse set to Pre-Cache Images?
  • Infuse metadata cache: < 2 GB
  • Infuse metadata cache: ≈ 2 GB to 4GB
  • Infuse metadata cache: ≈ 4 GB to 8GB
  • Infuse metadata cache: > 8 GB
  • AppleTV HD (2015) 32GB
  • AppleTV HD (2015) 64 GB
  • AppleTV 4K (1st generation - 2017) 32 GB
  • AppleTV 4K (1st generation - 2017) 64 GB
  • AppleTV 4K (2nd generation - 2021) 32 GB
  • AppleTV 4K (2nd generation - 2021) 64 GB
  • AppleTV 4K (3rd generation - 2022) 64 GB
  • AppleTV 4K (3rd generation - 2022) 128 GB

0 voters


Part 3
ONLY PARTICIPATE IF YOU ANSWERED NO on Part 1.

If you haven’t recently experienced a database deletion:
  • Did you disable screensaver downloads?
  • Did you manually delete unused apps, or allow tvOS to automatically offload unused apps?
  • Did you fill your Apple TV with apps (until no more could be installed) to force deletion of all volatile cached content, or delete all apps and reset your Apple TV?
  • OR did you reset your device but not first delete all apps; and did you resync their cached content from iCloud?
  • Do you use gaming or business apps that cache large amounts of data?
  • Do you download media content to your AppleTV for offline viewing?
  • Do you use Infuse iCloud Sync?
  • Is Infuse set to Pre-Cache Images?
  • Infuse metadata cache: < 2 GB
  • Infuse metadata cache: ≈ 2 GB to 4GB
  • Infuse metadata cache: ≈ 4 GB to 8GB
  • Infuse metadata cache: > 8 GB
  • AppleTV HD (2015) 32GB
  • AppleTV HD (2015) 64 GB
  • AppleTV 4K (1st generation - 2017) 32 GB
  • AppleTV 4K (1st generation - 2017) 64 GB
  • AppleTV 4K (2nd generation - 2021) 32 GB
  • AppleTV 4K (2nd generation - 2021) 64 GB
  • AppleTV 4K (3rd generation - 2022) 64 GB
  • AppleTV 4K (3rd generation - 2022) 128 GB

0 voters

Feel free to explain any answers below if you believe additional connect would be helpful.

10 posts were merged into an existing topic: Infuse should store the meta data in to the container (MKV/MP4)

Both of my AppleTVs are 32GB 2017 4K models set up identically. The one that just recently deleted its metadata has been disconnected from the internet for months (as has the one that hasn’t deleted its metadata).

The only differences between the two are that the one that preserved its metadata is still being actively used offline exclusively with Infuse; while the one that just deleted its metadata has been seldom used in the last few months (since going offline) and was more often used with YouTube and YouTube TV (and only rarely with Infuse) prior to going offline.

OFF TOPIC: Is there a way to create a poll that doesn’t reorder the results based on the number of positive answers?

It’s certainly not helpful in this context, where jumbling up the questions just makes it harder to analyze the results.

1 Like

FWIW I haven’t seen the “cleared metadata” issue recently. The app has been under normal usage similar to when I was experiencing the metadata issue every time I would open the app. Below are some details of my setup on an Apple TV w/ Infuse Pro 7.5.6 (4444).

(Side note: I just downloaded the latest update 7.5.7 (4459) so I’ll be observing this version for any metadata issues as well.)

Apple TV 4K (3rd generation) Wi-Fi + Ethernet

Model A2843 (128 GB)
tvOS 16.5 (20L563)

Infuse Pro 7.5.6 (4444)

Plex Server is the only library I’ve connected to Infuse.

Notable settings:
iCloud Sync ON
Metadata Fetching ON
Embedded Metadata OFF
Share Crash Data OFF
Metadata AUTO

Library stats:
Movies: 1,369
TV Episodes: 14,971
Other: 0
Current metadata size: 1.3GB

1 Like