Plex_sync + Plex_exporter_importer: Support for intro markers about plex-scripts HOT 69 CLOSED

It is. I have my finals of high school, so I'm a bit busy. That's why I haven't been able to work on it much. However, in one week, I'm done and I'll have loads of time.

from plex-scripts.

Request accepted. Doesn't mean it's per se possible. Just that I accept the challenge and will do some research and testing.

from plex-scripts.

Ohh nice one. I would love to see this too :)

from plex-scripts.

I third that! This would be awesome!

from plex-scripts.

Request accepted. Doesn't mean it's per se possible. Just that I accept the challenge and will do some research and testing.

This may help

https://github.com/animosity22/homescripts/blob/master/scripts/plex-missing-intro

Edit: being able to also trigger plex to scan those files for missing intro's would also be cool. not sure if you'd want me to create another post for that request.

from plex-scripts.

I'm going to make this a feature of the plex_exporter_importer.py script instead of the plex_sync.py script. The reason for this is that, in order for this to work, we need to directly edit the database file. The idea behind plex_sync.py is that both servers are on at the same time and that we sync the two using the network (api requests). The script cannot edit both databases (read from source and apply in target) at the same time. That is for the simple reason that the script won't have access to both database files at the same time, because they are on different machines.

Now I know that you have a special setup with some sort of file sync where the machine has access to both database files at the same time, but that setup is extremely rare. So if I edit plex_sync.py to support your request, basically only you can use the feature and nobody else. That's not good.

So I'm going to make it a feature of plex_exporter_importer.py. This script is intended for exporting and importing plex metadata. Basically the non-live version of plex_sync.py, by saving everything into files (exporting) and reading from them when needed (importing). If I make this request a feature of this script, I can just export the data from the database file into a file alongside the media and when importing insert it into the accessible database.

This is way more user friendly and covers a lot more people's setup.

For you, this would mean the following: you run the script in export mode on the source server to export the intro markers into a file alongside the media. Then on the target server, you run the script in import mode which will read the just-created-file and apply it to it's database. This is exactly how all the other things work in plex_exporter_importer.py (posters, metadata). This does require though that the folder with the media is accessible and imported into plex on both servers.

It's either this or a feature for plex_sync.py that nobody but you can use. And I've decided that I'm going with the plex_exporter_importer.py route.

from plex-scripts.

That sounds awesome bud, thank you! Does Plex need to be stopped or can it be running during import? Can intros be selected independently of all else (posters, meta, etc)? Thanks again!!

from plex-scripts.

Okay I've been trying stuff out and been digging through the raw database file. I haven't written any code yet but I just have some notes written and some sqlite commands with which I think I can make this work! So I'm going to say that this has a 90% chance of working!

That this request is going to work is good in it self. But there's more good news: I've found a way that we only need to access the target database. No need to access the source database. This is amazing news because that means that we can just run the script on the target server, grab the intro data via the api from the source server and edit the db file on the target server (on which the script is run) to apply the intro data there.

So this is my plan:

I'm going to implement this feature request in both plex_exporter_importer.py AND plex_sync.py.

• When the feature is used in plex_sync.py, the user is restricted to only being able to sync the intro markers from a remote plex server to the plex server running on the system the script is run on (as I've explained in the above paragraph).
• When the feature is used in plex_exporter_importer.py, the user can do everything he/she/they wants. Run the script in export mode on server1 and in import mode on server2 or vise versa. But again, we can only import the intro markers (aka apply them) to the plex instance that the script is running on.

And that is the important thing here: it will work, but we can only apply intro markers to the plex server that the script is running on. So plex_sync.py needs to be run on the target server in order for this feature to work. And plex_exporter_importer.py needs to be run on the target server when -t is set to import and -p to (the non-existent yet) intro_marker.

For you, this means that you will be able to run plex_sync.py on the target server and sync the intro markers from source (remote) to target (the machine itself) with no problems.

Reaction to @edrock200:
When using both scripts, plex can stay on at all times. No need to shut it down. It even works "live". When you run the script and are looking at your library via the web-ui, you'll see the posters and titles changing, collections being made, etc. And using both scripts, you can indeed do actions individually (you select what to export/import/sync; posters, collections, metadata and soon intro markers). So if you want to only sync/import intro markers, on plex_sync.py you will be able to do -S intro_markers and on plex_exporter_importer.py -p intro_marker.

If there are anymore questions feel free to ask them ;)

from plex-scripts.

Amazing!!! This is great! Thank you so much.

from plex-scripts.

Oh my god I think I've made it work! This is amazing. I've even surprised myself with the fact that I was able to make this work. The only problem is that it has only been integrated into plex_sync.py for now. Might add it to plex_exporter_importer tomorrow or later.

For now, you can download plex_sync.py here with the new feature integrated: https://github.com/Casvt/Plex-scripts/blob/Casvt/issue44/multiple_servers/plex_sync.py. This version should work, but I haven't tested it fully yet so I'm eager to hear how it goes on your side. Please let me know because if it works for you, then I know it works and I can push it to main. Just find an episode that doesn't have intro markers, run the script and then see if it does. It should even work live.

I'm eager to hear the result from your testing!

EDIT: Just to help you along, the following argument list should be sufficient to test this: -s Main -S intro_markers, assuming that the script is being run on the backup server. Keep in mind that this affects every episode in every show library.

from plex-scripts.

So awesome!! And will do, thank you! I'll give it a shot later today.

from plex-scripts.

One question, will this add or sync? E.g. if source Plex has intros for showA and backup Plex has intros for showB, and I sync will backup only have intros for showA or both A and B?

from plex-scripts.

hmm. installed aiohttp, but now getting this error
File "./introsynctest.py", line 40, in <module> from asyncio import gather, run ImportError: cannot import name 'run'

from plex-scripts.

Got passed that error by upgrading python. Next error it didn't think I was running on target system, but my plex is in a docker container so I hardcoded the db path around line 323. It's running now. Just says "Intro Markers." So I'm guessing it's doing its thing. Can plex be running on target or should it be stopped? Thanks again!

from plex-scripts.

ran for a good bit, then spit out this error:

Intro Markers
Traceback (most recent call last):
  File "./plex_sync.py", line 552, in <module>
    response = instance.start_sync()
  File "./plex_sync.py", line 157, in start_sync
    response = self._intro_markers()
  File "./plex_sync.py", line 350, in _intro_markers
    target_ratingkey = self.__find_on_target(guid=episode.get('Guid',[]), title=episode.get('title',''), type='episode').get('ratingKey','')
AttributeError: 'NoneType' object has no attribute 'get'```

from plex-scripts.

One question, will this add or sync? E.g. if source Plex has intros for showA and backup Plex has intros for showB, and I sync will backup only have intros for showA or both A and B?

Let's say source has ShowA with intro's and target not. Target has ShowB with intro's but source not. If you run the script on target, showA's intro markers from source will be applied on target showA and the markers on ShowB on target are kept. It only adds or edit's intro markers, not remove them.

Got passed that error by upgrading python.

Yeah someone else got that same problem too just a few hours ago haha. At least 3.7 is required I think. If your version was lower than 3.7 before, it was a good idea to update anyway.

Can plex be running on target or should it be stopped?

Both the plexes on source and target can be kept running. They both don't need to be shut down at any point. This all happens "live".

Just says "Intro Markers." So I'm guessing it's doing its thing.

Good that you're saying that because I indeed forgot to add print statements that tell the user where the script is. I've added those now. It will now look like this:

Intro Markers
	Show Lib 1
	Show Lib 2
	Show Lib 3

This way the user knows that the script is actually doing something and just about where it is.

ran for a good bit, then spit out this error

Fixed it in a new update. Please re-download the script to get the fixed version. Still here: https://github.com/Casvt/Plex-scripts/blob/Casvt/issue44/multiple_servers/plex_sync.py.

Thank you very much for all the feedback it really helps.

from plex-scripts.

Thanks bud! Trying again. It's ok that I hardcoded the db in the code since I'm running Plex in docker correct?

from plex-scripts.

Yeah that's good. Maybe I should add a variable at the top of the script that people can use to override/hardcode the location of the db. The script is coded in such way that it can find the location itself, but with special setups like with docker, it falls apart and there should be an easy option to hardcode the location. I'll add that too tomorrow (00:57 here now so I'm going to sleep in an hour so I'll do it tomorrow).

from plex-scripts.

OK it's running again. I had to fix the indentation on line 419 before it would run but otherwise it seems to be working so far :) I'll keep you posted,. Thanks again! Oh and re ashes, I know him, we are testing together. :)

from plex-scripts.

Uhm the indentation on line 419 is just fine?

from plex-scripts.

It wouldnt run until I made "handled.series" line up to "self.target" above it. Kept giving me a line 419 indention error. I noticed just before handled series are spaces instead of a tab. Not sure if that matters?

from plex-scripts.

I updated the script and re-did the indentation of the line (and the ones above and below it) so it should really work now. You can use the link shared earlier to download the new version. The handled_series line should be one indentation less than self.target. Or you can just re-download the script.

from plex-scripts.

I updated the script by pulling it down from your link. I don't get the indendation error now but the original error appears to have returned:

Intro Markers
        4K TV Shows
Traceback (most recent call last):
  File "./plex_sync.py", line 557, in <module>
    response = instance.start_sync()
  File "./plex_sync.py", line 158, in start_sync
    response = self._intro_markers()
  File "./plex_sync.py", line 339, in _intro_markers
    self.result_json.append(episode[['ratingKey']])
TypeError: unhashable type: 'list'

from plex-scripts.

Different error but just a simple typo. Fixed it already so again just use the link above to re-download the script. Sorry for the stupid little mistake.

from plex-scripts.

haha please stop apologizing for helping everyone with an awesome tool! trying it now.

edit ok it's running again, and no errors on launch this time. :) I'll report back when its done!

from plex-scripts.

OK so this time it finished running. I had one show as a test - Last Week Tonight With John Oliver. On source, skip intro appears, but on destination it doesn't appear to show. Do you know which db field to look at to confirm sync?

from plex-scripts.

In the web-ui, go to one of the episodes -> [three dots] -> "get information" (last option) -> "show XML" (left bottom). At the bottom of the info that pops up, there should be a "Marker" with type="intro"

You can do this on the source to check if it's there and then on the target to check if it's there too after sync.

Edit: oh I already know why it doesn't work. The script currently can update intro markers but not add intro markers. I'll fix that when I have the time in the next few hours.

from plex-scripts.

Yeah I see that on source but not on destination

from plex-scripts.

What field are you using as the unique identifier between the two instances? The file path & name/hash?

from plex-scripts.

Well you fill in the info for both servers and give each server a name (standard name is 'Main' and 'Backup' which you can define with the variables inside the script) and then you say "source is Main". Then it will make connections to that server to grab info and then make connections to the other server and applies them.

But I'm not sure if that's what you mean?

from plex-scripts.

I guess what unique identifier is it using to match showA/SeasonX/EpisodeY to ensure they are the same on main and backup before syncing the intro data

from plex-scripts.

Guid, title and type. They all need to match. It works well

from plex-scripts.

In the web-ui, go to one of the episodes -> [three dots] -> "get information" (last option) -> "show XML" (left bottom). At the bottom of the info that pops up, there should be a "Marker" with type="intro"

You can do this on the source to check if it's there and then on the target to check if it's there too after sync.

Edit: oh I already know why it doesn't work. The script currently can update intro markers but not add intro markers. I'll fix that when I have the time in the next few hours.

I somehow completely missed this post that you already found the issue lol. Great! Let me know when its updated and I'll give it another go. :)

from plex-scripts.

Hey there, I'm working on it and I think I've made it work. However, there is one thing that I need to make sure. I'm not going to explain what's going on but can you just do the following please (do exactly as stated)?

1. Open a terminal on the source server
2. Navigate to the folder with the databases (on linux, that is /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases)
3. When you're in the folder inside your terminal, run the following command:

python3

This should open a python shell in the database folder
4. Inside the python shell, execute the following commands one by one:

import sqlite3
cursor = sqlite3.connect('com.plexapp.plugins.library.db').cursor()
cursor.execute("SELECT DISTINCT tag_id FROM taggings WHERE text = 'intro';")
cursor.fetchall()

Please share the output of the last command. Now I'm going to look into a magic glass ball and predict that the outcome will be [(34408,)]. I'm hoping very much that that will be the output, because otherwise I'm going to have a pretty big problem haha. Ask me questions if you get stuck ;)

from plex-scripts.

>>> cursor = sqlite3.connect('com.plexapp.plugins.library.db').cursor()
>>> cursor.execute("SELECT DISTINCT tag_id FROM taggings WHERE text = 'intro';")
<sqlite3.Cursor object at 0x7f6b1b902f80>
>>> cursor.fetchall()
[(308935,)]
>>>

from plex-scripts.

If that scraped all the intro data from source, it did so MUCH faster than via API calls. Took ~30 seconds vs hours. :)

from plex-scripts.

If that scraped all the intro data from source

It sort of did but not to the extend that we can use it. So we still have to do it via the api. And by the way, the reason it takes hours is not because of all the API calls, it's that it needs to make thousands of files. Creating a file is really slow compared to the rest. If you let the script complete once, and then run it again, it will run like three times as fast because it doesnt have to create the files anymore; only update the content of it.

And by the way, even though the output didn't match up, I already know how I'm going to fix it. Even though you have a different number, we both have only one number. So I just have to get that number and use that. Anyway, thanks. And I'll look into ways to optimise the script more. Did that old script take hours too? How long did that old script take?

from plex-scripts.

Thank you! Old script said:
Intro markers time: 25464.95s so just over 7h. But my library is abnormally larger than most. :)

from plex-scripts.

Oh I'm sorry. I thought I was talking to someone else. plex_sync.py doesn't make any files. So don't listen to the whole story about making files and stuff haha. That is for plex_exporter_importer.py. 7h is still very long though...

Fetching the contents of the taggings table inside the db took 30 seconds while for me it was practically instantly. So maybe that's what's being so slow. Everytime we add an intro marker on target, it takes super long because the table is already massive. Or it's the commit that's taking so long... Either way, I'm going to take a look at it.

from plex-scripts.

Oh and also: I've updated the script so it now supports both editing existing intro markers and also adding new ones. It should all work now. So please redownload the script and try again! And by the way, you don't have to let it run for the complete 7 hours. You can just look at the very first episodes in the library the script displays on the target server after a few minutes and then the first few episodes should be edited so you can just take a look then and confirm that it's working before letting it go the complete 7 hours.

from plex-scripts.

Awesome! Thanks. Will give it a go!

from plex-scripts.

It's running :) Question - for intro markers only, does the backup ip/port/token need configured if I'm hardcoding the db file path? It is configured now, but just curious.

from plex-scripts.

Yes it does. Though it's not many, there are a few requests made to the target server. So those credentials are required yes.

from plex-scripts.

It worked! Took 6600 seconds but it worked! 😁 Thank you!

from plex-scripts.

Well 1h48m is less than 7h. But I'm still going to try to make it faster.

from plex-scripts.

Indeed it is! Didn't mean to sound negative. My apologies if so. One question, at the risk of "looking a gift horse in the mouth", would it be difficult to dump the intro markers out to a single file, then import them as a file? Reason being, it could be a good way for the community to share intro markers as well as make it much faster for a scenario with multiple servers syncs.

from plex-scripts.

Didn't mean to sound negative. My apologies if so.

Oh don't worry. It's not you, it's me. I set a high standard to myself and my scripts and I just find 1h48m way to long. IMO that's not acceptable, so I want to make it faster for you, but also just for myself so I'm happy with it.

at the risk of "looking a gift horse in the mouth"

The whole idea of this project is that you can request stuff so don't worry.

would it be difficult to dump the intro markers out to a single file, then import them as a file?

Then you're looking at plex_exporter_importer.py. It doesn't have intro marker support yet but it will get it soon. That script saves to files when exporting and reads from files when importing. However, it currently saves metadata for each media item to a separate file along side the media. So it's not really useful now for your use case:

it could be a good way for the community to share intro markers as well as make it much faster for a scenario with multiple servers syncs.

However, see this comment on a different issue: #21 (comment). I'm going to overhaul the script so that it saves everything to one SQL database. That file, people can move, upload and download whatever they want. And then you pass it to the script when importing and everything in the database is applied. That would be perfect for your use case:

1. See the comment linked above but shortly, it's going to be way faster.
2. Everything, literally everything, is going to be stored in one DB file, which you can easily move around everywhere and share.
3. You just pass the db file to the script in import mode and everything is applied that is chosen by the user.

So just wait for that and I'll notify you when it's done.

from plex-scripts.

You are the man! Err..kid! 😁 Convince your folks to accept donations for your college fund! 👍

from plex-scripts.

I'm already looking into ways to recieve money (internationally) but I haven't found any way. Is saw PayPal.me but that requires that I'm 18 years or older. I haven't found any other way yet they would work.

And I've received an offer for an internship at Plex so no worries haha I have myself covered for uni.

from plex-scripts.

I've updated the script to have a variable at the top which allows you to set a custom location for the database folder. Saves you having to edit the script.

from plex-scripts.

Feature has been added to both scripts so request completed.

For the record: this is the pull request that added these features to main #109

@edrock200 eventhough this issue will be closed, I'll comment here to let you know when the upgrade of plex_exporter_importer has been completed :)

from plex-scripts.

Awesome! Thank you!

from plex-scripts.

Hey @edrock200, plex_exporter_importer.py has had its expected major upgrade. It all now works using a single DB file instead of thousands of files along side the media. Exactly as I said it would turn out to be.

Just run the script in export mode (-t export) and just say to only export intro markers (-p intro_marker) for every episode (--AllShow) and tada, a databse file will have appeared alongside the script which contains all your intro markers. Move it around! Share it! And when anybody wants to import the markers stored in the database, they run, using the root user and on the computer on which the Plex server is too, in import mode (-t import) and just say to only import intro markers (-p intro_marker) for every episode (--AllShow). And tada, any episodes that matched to the database will have the intro markers applied.

And by the way, I'm planning on also supporting the exporting and importing of chapter images. Just wanted to let you know. And playlist support is on the way. And collection support too.

from plex-scripts.

@Casvt that's so awesome! Thank you so much!! Can this now run tokenless since it's doing direct db rights?

from plex-scripts.

Still requires tokens. In export mode, we fetch the metadata using the api and save the result in our own database file. When importing, we read the database and apply the metadata inside using the api. So in both scenarios, you'll need the tokens setup.

Plex_exporter_importer is for backing up Plex metadata (by extracting it) and importing it later (on a different Plex instance). Everything is saved in a plex_exporter_importer.db file. This allows for easy sharing with people and allowing other people to copy your Plex metadata setup (you export, they import). All you have to do is share the file.

Plex_sync is for live syncing of two Plex servers. So you can keep the metadata between two Plex servers (which are both on at the same time) synced where one is the source and the other is the target. This is useful if you have a backup server on which you want to keep, let's say, watched status and collections synced.

I hope you understand the difference but also how they both need all variables filled.

from plex-scripts.

Hi Casvt! Thank you for all your amazing work on this, this is amazing. Appreciate you!

Still requires tokens. In export mode, we fetch the metadata using the api and save the result in our own database file. When importing, we read the database and apply the metadata inside using the api. So in both scenarios, you'll need the tokens setup.

Would it be possible to solely rely on the plex database for both export and import so we don't need to send thousands of requests to the plex API? This would drastically speed up exports/imports for larger libraries. For context, I was looking to export/import intro markers for a library of ~1,000,000 episodes. Sending a million requests to pull the metadata for each episode would take a very long time.

Your consideration would be much appreciated. Thanks!

from plex-scripts.

@qazero I could write that but that will take a very long time because I would need to (re)write a significant amount of code. So not soon, maybe sometime. If you're planning on using this soon, I'd advice to just start it in the evening and I'm certain a million episodes will be exported/imported in a night. So next morning you'll be done. Sorry

from plex-scripts.

Ahh ok I'll give it a go in the meantime, thanks. Would successive runs be substantially quicker?

from plex-scripts.

If you've exported once, then exporting again will be substantially faster because it only fetches what has been changed since the last time. For this to work, you need to specify the path to an existing db file when exporting and it will update the file (instead of exporting to a new file), with the speed improvements.

Importing will always be the same speed.

from plex-scripts.

Nice, I'll give it a try. Shouldn't importing be quicker on subsequent runs too since it only needs to add whatever changed since the last run?

Also, since this is using Plex's API, what happens if the plex server restarts (docker autoheal) while the script is running overnight? Does the script have any built-in mechanism to check whether or not the API requests did indeed get executed? The reason I ask is because in my experience with Plex's API, if plex restarts then any requests which were previously sent to Plex's API and queued for execution get dropped and would need to be resent. I suppose one could disable autoheal and hope that plex remains healthy. My experience using plex's API to update a large set of items (~1000000) has been touch and go. Quite often I find plex going unhealthy and requiring a restart. Nonetheless, I will give it a shot and report back.

from plex-scripts.

Shouldn't importing be quicker on subsequent runs too since it only needs to add whatever changed since the last run?

The script can't know if it can skip or that it has to update because it doesn't know if something has changed since the last time. It can only do that with exporting, not importing.

If the Plex server shuts down while the script is running, the script will stop too (an error gets thrown because it can't connect to the server anymore). If you want to continue after, just re-run the command, now parsing the previously created db file and the script will practically take off where it was.

from plex-scripts.

@qazero Hey I'm rewriting the script and I'm planning on also adding support for full a db-to-db connection. So if you haven't done what you wanted already, wait a bit and I'll have it finished and you can use the new version :)

from plex-scripts.

Sweet! Looking forward to this, much appreciated. I will wait. Thanks!!

from plex-scripts.

@Casvt Hope you're doing well and all is well with you. Just wanted to follow up regarding this and see if you've managed to make any progress. Looking forward and eagerly awaiting this. Thanks in advance!

from plex-scripts.

@Casvt Been some time, and hope all is well with you. Just wondering, is this still in the pipeline?

from plex-scripts.

Sweet! Thanks for the update. Looking forward to it. Good luck on your finals, I'm sure you'll kill it!!

from plex-scripts.

One other consideration - on the export would it be possible to add a flag to export "added after date"? Could make subsequent syncs much faster. E.g. if you run it weekly and have it export only ~8d worth of data.

from plex-scripts.

One other consideration - on the export would it be possible to add a flag to export "added after date"? Could make subsequent syncs much faster. E.g. if you run it weekly and have it export only ~8d worth of data.

plex_exporter_importer.py already does this. It compares it's content's "updated_at" value and compares it to the value plex gives for the media. If they match, nothing has changed since the last time and it's skipped.

from plex-scripts.

@Casvt Any update on this?

from plex-scripts.