Hi, just tring to test the postprocessor. I set it up as described o

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

Hi, now I'm one step closer to a solution but when I run: <code clas

py issue running management help,about jgillula/paperless-ngx-postprocessor

Comments (15)

jgillula commented on May 21, 2024 2

Just a heads up that I've received this bug, and will be able to take a look later today

from paperless-ngx-postprocessor.

jgillula commented on May 21, 2024 1

Hi @pflodo! Just a heads up that I'm pretty busy this week, so I might not get to review your post until this weekend.

from paperless-ngx-postprocessor.

philomatic commented on May 21, 2024

Hi there,

~~tbh, I have no idea what the author means by:~~

~~> In order to run it outside the Paperless-ngx docker container,~~

~~in the Management section. How should the postprocessor even know how to rename documents when invoked externally on the docker host?~~
Nvm, API of course maybe.
But seems like a bad design to expect from the caller of the script to have correct access and url to the container AND have a correct python environment. Maybe that should be added to the docs.

BUT however, what I can tell you from understanding of python so far:
You can use the virtual environment ('venv' folder) and make python load those modules to use them in the shell:
source venv/bin/activate

For further understanding see:
https://realpython.com/python-virtual-environments-a-primer/

Best regards

from paperless-ngx-postprocessor.

roadrunnerjb commented on May 21, 2024

The Readme.md says in chapter management:

In addition to being run as a post-consumption script, paperless-ngx-postprocessor has the ability to be run directly via a command line interface using the paperlessngx_postprocessor.py script. In order to run it outside the Paperless-ngx docker container, you'll need to provide the auth token you generated during setup, e.g.:

...

The command line interface supports all of the same options that you can set via the environment variables listed in the Configuration section above. To see how to specify them, use the command line interface's built-in help:

./paperlessngx_postprocessor.py --help

That's exactly what I tried.

What I understand from philomatic is establishing venv before calling py script:

paperless@paperless:~/paperless-ngx-postprocessor$ source venv/bin/activate
(venv) paperless@paperless:~/paperless-ngx-postprocessor$ ./paperlessngx_postprocessor.py --help
Traceback (most recent call last):
  File "/home/paperless/paperless-ngx-postprocessor/./paperlessngx_postprocessor.py", line 8, in <module>
    from paperlessngx_postprocessor import Config, PaperlessAPI, Postprocessor
  File "/home/paperless/paperless-ngx-postprocessor/paperlessngx_postprocessor/__init__.py", line 3, in <module>
    from .paperless_api import PaperlessAPI
  File "/home/paperless/paperless-ngx-postprocessor/paperlessngx_postprocessor/paperless_api.py", line 1, in <module>
    import dateutil.parser
ModuleNotFoundError: No module named 'dateutil'

doest not work either but may be I miss something. I',m still lost. Do I need to run the command in the contailer with docker-compose exec ?

from paperless-ngx-postprocessor.

philomatic commented on May 21, 2024

Do I need to run the command in the contailer with docker-compose exec ?

Yes, at least with my suggestion.
To be honest, I thought you've done that due to the user@server "paperless@paperless" in your shell. But obviously I've interpret too much in there, sorry, my bad!

Best regards

from paperless-ngx-postprocessor.

jgillula commented on May 21, 2024

Wow. I have no idea what I was writing when I wrote the Management section of the readme. I think I made a lot of assumptions that I don't remember, and which definitely aren't clear in the docs.

I've rewritten part of the management section to hopefully make things more clear. Please take a look here, and let me know if anything still isn't clear. If so, I can take another stab at further fixing the documentation.

Thanks for reporting this!

from paperless-ngx-postprocessor.

roadrunnerjb commented on May 21, 2024

OK thank you for your work a quick response an this. I read the new readme and for me it's clear now what I did wrong.

Thanks again for being more precise - for beginners - in the docs !

from paperless-ngx-postprocessor.

roadrunnerjb commented on May 21, 2024

Hi, now I'm one step closer to a solution but when I run:

paperless@paperless:~/paperless-ngx-tcloud$ docker-compose exec -u paperless webserver /usr/src/paperless-ngx-postprocessor/paperlessngx_postprocessor.py --auth-token d8b1f1d60472c6a1a865c32c56cffbfc794aa728 document_id 2

I get

Traceback (most recent call last):
  File "/usr/src/paperless-ngx-postprocessor/paperlessngx_postprocessor.py", line 8, in <module>
    from paperlessngx_postprocessor import Config, PaperlessAPI, Postprocessor
  File "/usr/src/paperless-ngx-postprocessor/paperlessngx_postprocessor/__init__.py", line 4, in <module>
    from .postprocessor import Postprocessor
  File "/usr/src/paperless-ngx-postprocessor/paperlessngx_postprocessor/postprocessor.py", line 3, in <module>
    import jinja2
ModuleNotFoundError: No module named 'jinja2'

I did a git clone of the offcial repo...

as I'm not familiar with python, any idea on how to proceed ?

from paperless-ngx-postprocessor.

jgillula commented on May 21, 2024

(Assuming you're running that command on the Docker host...)

That looks like a similar problem to before; in order for paperlessngx_postprocessor.py to run, you first either have to globally install the python libraries/modules it depends on, or install them in a virtual environment and then enter the virtual environment and run the command inside that virtual environment.

In the case of paperlessngx-postprocessor, you create a virtual environment inside its Docker container with all the necessary libraries when you set it up (in step 2). Then, when Paperless-ngx calls the post-consume script /usr/src/paperless-ngx-postprocessor/post_consume_script.sh (which you added in step 1), that bash script first activates the virtual environment you created, and then actually runs the Python script to do the postprocessing.

In the command you provided, you're telling Docker to run paperlessngx_postprocessor.py inside the Docker container, but without first activating the Python virtual environment. As a result, paperlessngx_postprocessor.py can't find the modules/libraries it depends on.

I'd recommend instead just getting a bash terminal inside the Docker container, navigating to /usr/src/paperless-ngx-postprocessor, activating the Python virtual environment, and then running the script, as described in the updated README. (Also if you do it that way, you won't need the auth token.)

Does that help?

from paperless-ngx-postprocessor.

jgillula commented on May 21, 2024

(Accidentally closed the issue, re-opened now)

from paperless-ngx-postprocessor.

pflodo commented on May 21, 2024

I'd recommend instead just getting a bash terminal inside the Docker container, navigating to /usr/src/paperless-ngx-postprocessor, activating the Python virtual environment, and then running the script, as described in the updated README. (Also if you do it that way, you won't need the auth token.)

Does that help?

Not original poster - In my case I use Portainer to manage my docker - and it gives me a an exec into the container. I then activate environment and the --help switch runs fine. but if I use any selector it just sits there and never returns

paperless@94df449dcba7:~/src$ cd /usr/src/paperless-ngx-postprocessor/
paperless@94df449dcba7:/usr/src/paperless-ngx-postprocessor$ source venv/bin/activate
(venv) paperless@94df449dcba7:/usr/src/paperless-ngx-postprocessor$ ./paperlessngx_postprocessor.py document_id 298

If I add a document normally, the log says
[2023-06-20 12:59:48,766] [INFO] [paperless.consumer] Executing post-consume script /usr/src/paperless-ngx-postprocessor/post_consume_script.sh
but the job get stuck and times out.

I renamed example.yml to title.yml and just left this rule:

# Note that rules are applied in order, so any changes from this rule will overwrite changes from previous rules
Some Name:
  # This will always match
  match: True
  metadata_postprocessing:
    title: '{{created_year}}-{{created_month}}-{{created_day}} -- {{correspondent}} -- {{document_type}}'

from paperless-ngx-postprocessor.

pflodo commented on May 21, 2024

I fully understand, I appreciate that you have shared your work in the first place.
I will double check that I have followed instructions correctly and will update if I make progress.

from paperless-ngx-postprocessor.

pflodo commented on May 21, 2024

Ok redoing the config - I noticed that when doing the initial setup script, it was throwing an error (because it did not have write permission to the host volume).

paperless@caf8061b2c06:~/src$ /usr/src/paperless-ngx-postprocessor/setup_venv.sh 
Setting up virtual environment...Error: [Errno 13] Permission denied: '/usr/src/paperless-ngx-postprocessor/venv/pyvenv.cfg'
OK
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: Django in /usr/local/lib/python3.9/site-packages (from -r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 1)) (4.1.9)
Requirement already satisfied: Jinja2 in /usr/src/paperless-ngx-postprocessor/venv/lib/python3.9/site-packages (from -r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 2)) (3.1.2)
Requirement already satisfied: python-dateutil in /usr/local/lib/python3.9/site-packages (from -r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 3)) (2.8.2)
Requirement already satisfied: PyYAML in /usr/local/lib/python3.9/site-packages (from -r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 4)) (6.0)
Requirement already satisfied: regex in /usr/local/lib/python3.9/site-packages (from -r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 5)) (2023.5.5)
Requirement already satisfied: requests in /usr/local/lib/python3.9/site-packages (from -r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 6)) (2.31.0)
Requirement already satisfied: sqlparse>=0.2.2 in /usr/local/lib/python3.9/site-packages (from Django->-r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 1)) (0.4.4)
Requirement already satisfied: asgiref<4,>=3.5.2 in /usr/local/lib/python3.9/site-packages (from Django->-r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 1)) (3.7.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/src/paperless-ngx-postprocessor/venv/lib/python3.9/site-packages (from Jinja2->-r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 2)) (2.1.3)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.9/site-packages (from python-dateutil->-r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 3)) (1.16.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.9/site-packages (from requests->-r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 6)) (3.1.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/site-packages (from requests->-r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 6)) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/site-packages (from requests->-r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 6)) (2023.5.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.9/site-packages (from requests->-r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 6)) (2.0.2)
Requirement already satisfied: typing-extensions>=4 in /usr/local/lib/python3.9/site-packages (from asgiref<4,>=3.5.2->Django->-r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 1)) (4.6.2)
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the '/usr/src/paperless-ngx-postprocessor/venv/bin/python -m pip install --upgrade pip' command.

paperless-ngx-postprocessor setup successful

I fixed the write permission and reran - but I get a different error -

paperless@caf8061b2c06:~/src$ /usr/src/paperless-ngx-postprocessor/setup_venv.sh 
Setting up virtual environment...Error: [Errno 1] Operation not permitted: '/usr/src/paperless-ngx-postprocessor/venv/bin/Activate.ps1'
OK
Requirement already satisfied: Django in /usr/src/paperless-ngx-postprocessor/venv/lib/python3.9/site-packages (from -r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 1)) (4.2.2)
Requirement already satisfied: Jinja2 in /usr/src/paperless-ngx-postprocessor/venv/lib/python3.9/site-packages (from -r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 2)) (3.1.2)
Requirement already satisfied: python-dateutil in /usr/src/paperless-ngx-postprocessor/venv/lib/python3.9/site-packages (from -r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 3)) (2.8.2)
Requirement already satisfied: PyYAML in /usr/src/paperless-ngx-postprocessor/venv/lib/python3.9/site-packages (from -r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 4)) (6.0)
Requirement already satisfied: regex in /usr/src/paperless-ngx-postprocessor/venv/lib/python3.9/site-packages (from -r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 5)) (2023.6.3)
Requirement already satisfied: requests in /usr/src/paperless-ngx-postprocessor/venv/lib/python3.9/site-packages (from -r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 6)) (2.31.0)
Requirement already satisfied: sqlparse>=0.3.1 in /usr/src/paperless-ngx-postprocessor/venv/lib/python3.9/site-packages (from Django->-r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 1)) (0.4.4)
Requirement already satisfied: asgiref<4,>=3.6.0 in /usr/src/paperless-ngx-postprocessor/venv/lib/python3.9/site-packages (from Django->-r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 1)) (3.7.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/src/paperless-ngx-postprocessor/venv/lib/python3.9/site-packages (from Jinja2->-r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 2)) (2.1.3)
Requirement already satisfied: six>=1.5 in /usr/src/paperless-ngx-postprocessor/venv/lib/python3.9/site-packages (from python-dateutil->-r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 3)) (1.16.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/src/paperless-ngx-postprocessor/venv/lib/python3.9/site-packages (from requests->-r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 6)) (3.4)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/src/paperless-ngx-postprocessor/venv/lib/python3.9/site-packages (from requests->-r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 6)) (3.1.0)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/src/paperless-ngx-postprocessor/venv/lib/python3.9/site-packages (from requests->-r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 6)) (2.0.3)
Requirement already satisfied: certifi>=2017.4.17 in /usr/src/paperless-ngx-postprocessor/venv/lib/python3.9/site-packages (from requests->-r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 6)) (2023.5.7)
Requirement already satisfied: typing-extensions>=4 in /usr/src/paperless-ngx-postprocessor/venv/lib/python3.9/site-packages (from asgiref<4,>=3.6.0->Django->-r /usr/src/paperless-ngx-postprocessor/requirements.txt (line 1)) (4.6.3)
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the '/usr/src/paperless-ngx-postprocessor/venv/bin/python -m pip install --upgrade pip' command.

paperless-ngx-postprocessor setup successful

from paperless-ngx-postprocessor.

pflodo commented on May 21, 2024

Ok - so I confused myself - I renamed my folder and cloned a new copy, but i didn't restart my container, so whilst docker volume is configured 'by name', once connected, it will keep that connection (despite renaming it and creating a new folder).
In summary - I had a permission issue, which was easily fixed by chmod on the host, but then I was testing a different instance to what I thought, although I still can't explain the second error - Anyway - it works now - 😁

from paperless-ngx-postprocessor.

jgillula commented on May 21, 2024

That's great news! I'm going to go ahead and close this issue, but feel free to open another one if you have any other problems.

from paperless-ngx-postprocessor.

py issue running management help about paperless-ngx-postprocessor HOT 15 CLOSED

Comments (15)

Related Issues (15)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent