Having the ability to ‘Press & Hold’ on the Siri Remote, when over a media item in the Watching section for contextual quick actions like, remove from watching, adding to a playlist, marking as watched, unwatched, watch trailer, etc would be amazing!
Edit, clarified.
It already does this. on ATV. Try it on one of the “Recently Added” items.
Thanks! I just clarified that I meant the Watching Section specifically.
Isn’t that really the same amount of clicking whether you short click on the item in the watching list and are then presented with the options to do everything that the long press does elsewhere? It takes you to the pre-playback screen that has all of that only a click away.
Technically the same amount of clicks, but in UX psychology land, it would feel much faster and may actually be faster, (which would minimize the animation and visual changes). I think it would be an improvement, especially if you could remove something from Watching (but keeps your current play position), which is very different than marking watched then unwatched.
Edit:
Also, as you noted, it is already happening elsewhere, so the logic is already built out.
I agree. And like you said, the functionality is already built in elsewhere — in the same way the season selector exist everywhere but in the Up Next bar.
You can do this, via the Playlist button. If you select playlist from an episode details page, you can uncheck the item from the “Watching” playlist (listed first under the heading “Up Next List”).
Bizarrely, when you select Playlists from the Library Menu the Watching playlist is not shown. 
Ah good to know! Would still be nice to have in a quick options menu as well.
Absolutely!
(Just sharing what’s possible in the meantime.)