zerodha / frappe-attachments-s3 Goto Github PK

View Code? Open in Web Editor NEW

112.0 112.0 129.0 69 KB

A frappe app to upload file attachments in doctypes to s3.

License: Other

Python 93.36% JavaScript 6.64%

frappe-attachments-s3's Introduction

Frappe S3 Attachment

Frappe app to make file upload automatically upload and read from s3.

Features.

Upload both public and private files to s3.
Stream files from S3, when file is viewed everytime.
Lets you add S3 credentials (aws key, aws secret, bucket name, folder name) through ui and migrate existing files.
Deletes from s3 whenever a file is deleted in ui.
Files are uploaded categorically in the format. {s3_folder_path}/{year}/{month}/{day}/{doctype}/{file_hash}

Installation.

bench get-app https://github.com/zerodhatech/Frappe-attachments-s3.git
bench install-app frappe_s3_attachment

Configuration Setup.

Open single doctype "s3 File Attachment"
Enter (Bucket Name, AWS key, AWS secret, S3 bucket Region name, Folder Name) Folder Name- folder name is the default folder path in s3.
Migrate existing files lets all the existing files in private and public folders to be migrated to s3.
Delete From Cloud when selected deletes the file form s3 bucket whenever a file is deleted from ui. By default the Delete from cloud will be unchecked.

License

MIT

frappe-attachments-s3's People

Contributors

Stargazers

Watchers

Forkers

shridarpatil bailabs tmvanetten jvfiel marchon clixsoftware ccfiel yashodhank kevinvillegas94 elasticrun shipmnts kingdg25 alyjee mudux mehtaharsh15 rdandy akurungadam sanjay-hercules farmbox-ae thispl shubhamgupta123 fahimalizain leam-tech zico99211 erpnextdeveloper2 ferch01991 jeandrorc bharathjinka09 mdeora finbyz improwised shabbirhasan1 tacten pjamsheer nikhilponnuru hrgadesha satishvijayan vikas32567 ahmedzaqout vedaha koprab bibinqcs mohsinalimat nestorwheelock chrisdent arteryp arjunkaimattathil osmondliu95 sfaplay filtersource extension-technologies jason156 sudo-vaibhav neerajvkn taweechai-scg htl-digitalization stackexpress-jawed katzen48 anandhakumarnataraj amutaher govind-jangid chezuba parsimonygit vjfalk askmetoo tradaonline nonalifestyle abdulazizaljbri karkhana-io c-m3-codin india081947 beggum waiminn nguyenphb sythong techprame lama-school kingraphaii bytesnbots eleben munir131 tridz-dev themayurahir viis-iot revasiya kiniamogh sajjanmanju rohitkumargit onehashai nishith121 teddyseptiadi suite42 rohitg-pbspl agkiya-technology-and-consulting-llp alirezamohammadi albseb511 kamalparyani empress-eco saleemdev myedigateway

frappe-attachments-s3's Issues

features: allow custom domain name instead of s3 domain

Allow custom domain like cdn.example.com/PATH_TO_FILE (AWS cloudfront like d111111abcdef8.cloudfront.net//PATH_TO_FILE)

Instead of BUCKEY_NAME.s3.amazonaws.com/PATH_TO_FILE

This would be helpful for serving static content from CDN

Reload File document after S3 Upload

Expected behaviour

Uploaded image should be in the specified image field.

Actual behaviour

Not all File uploads are image_field uploads. If a File was uploaded on a Doctype with image_field specified, that do not mean that File was uploaded for the image_field

Step to reproduce

VersionV13

Migrating existing files failing

Expected behaviour

Import all existing attachments to S3

Actual behaviour

Error message

Step to reproduce

Click "Migrating existing files" button

Version

Latest

Traceback (most recent call last):

  File "/home/frappe/frappe-bench/apps/frappe/frappe/app.py", line 60, in application
    response = frappe.api.handle()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/api.py", line 55, in handle
    return frappe.handler.handle()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/handler.py", line 22, in handle
    data = execute_cmd(cmd)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/handler.py", line 61, in execute_cmd
    return frappe.call(method, **frappe.form_dict)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 1042, in call
    return fn(*args, **newargs)
  File "/home/frappe/frappe-bench/apps/frappe_s3_attachment/frappe_s3_attachment/controller.py", line 294, in migrate_existing_files
    upload_existing_files_s3(file['name'], file['file_name'])
  File "/home/frappe/frappe-bench/apps/frappe_s3_attachment/frappe_s3_attachment/controller.py", line 250, in upload_existing_files_s3
    parent_name
  File "/home/frappe/frappe-bench/apps/frappe_s3_attachment/frappe_s3_attachment/controller.py", line 98, in upload_files_to_s3_with_key
    key = self.key_generator(file_name, parent_doctype, parent_name)
  File "/home/frappe/frappe-bench/apps/frappe_s3_attachment/frappe_s3_attachment/controller.py", line 80, in key_generator
    "/" + day + "/" + parent_doctype + "/" + key + "_" + \
TypeError: must be str, not NoneType

Cannot run this app in erpnext v12

Expected behaviour

want to run this app in erpnext deployed on Kubernetes

Actual behaviour

should run without crashloopbackoff in Kubernetes,
scheduler pod logs

config file not created, retry 1
Attempt 1 to connect to mariadb.erpnext.svc.cluster.local:3306
Attempt 1 to connect to frappe-erpnext-redis-queue:12000
Attempt 1 to connect to frappe-erpnext-redis-cache:13000
Attempt 1 to connect to frappe-erpnext-redis-socketio:11000
Connections OK
Starting background scheduler . . .
Traceback (most recent call last):
  File "/home/frappe/frappe-bench/commands/background.py", line 11, in <module>
    main()
  File "/home/frappe/frappe-bench/commands/background.py", line 6, in main
    start_scheduler()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/scheduler.py", line 34, in start_scheduler
    schedule.every(frappe.get_conf().scheduler_tick_interval or 60).seconds.do(enqueue_events_for_all_sites)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 254, in get_conf
    with init_site(site):
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 263, in __enter__
    init(self.site)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 187, in init
    setup_module_map()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 1047, in setup_module_map
    for module in get_module_list(app):
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 913, in get_module_list
    return get_file_items(os.path.join(os.path.dirname(get_module(app_name).__file__), "modules.txt"))
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 871, in get_module
    return importlib.import_module(modulename)
  File "/opt/bitnami/python/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'frappe-attachments-s3'

redis & maridb connections are fine

Step to reproduce

clone this repo and followed these steps
worker image docker file, image on docker hub
build it with docker build --no-cache . -t xxxxxx

FROM frappe/erpnext-worker:edge

RUN install_app frappe-attachments-s3 https://github.com/zerodhatech/frappe-attachments-s3

docker build logs

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM frappe/erpnext-worker:edge
 ---> de2b9eeda132
Step 2/2 : RUN install_app frappe-attachments-s3 https://github.com/zerodhatech/frappe-attachments-s3
 ---> Running in ec1fca2a96b0
Cloning into 'frappe-attachments-s3'...
Obtaining file:///home/frappe/frappe-bench/apps/frappe-attachments-s3
Collecting boto3==1.4.7
  Downloading boto3-1.4.7-py2.py3-none-any.whl (128 kB)
Collecting botocore==1.7.48
  Downloading botocore-1.7.48-py2.py3-none-any.whl (3.7 MB)
Collecting python-magic==0.4.18
  Downloading python_magic-0.4.18-py2.py3-none-any.whl (8.6 kB)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/frappe/frappe-bench/env/lib/python3.7/site-packages (from boto3==1.4.7->frappe-s3-attachment==0.0.1) (0.10.0)
Collecting s3transfer<0.2.0,>=0.1.10
  Downloading s3transfer-0.1.13-py2.py3-none-any.whl (59 kB)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/frappe/frappe-bench/env/lib/python3.7/site-packages (from botocore==1.7.48->frappe-s3-attachment==0.0.1) (2.8.1)
Requirement already satisfied: docutils>=0.10 in /home/frappe/frappe-bench/env/lib/python3.7/site-packages (from botocore==1.7.48->frappe-s3-attachment==0.0.1) (0.15.2)
Requirement already satisfied: six>=1.5 in /home/frappe/frappe-bench/env/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore==1.7.48->frappe-s3-attachment==0.0.1) (1.14.0)
ERROR: frappe 13.0.0.dev0 has requirement boto3==1.10.18, but you'll have boto3 1.4.7 which is incompatible.
Installing collected packages: botocore, s3transfer, boto3, python-magic, frappe-s3-attachment
  Attempting uninstall: botocore
    Found existing installation: botocore 1.13.50
    Uninstalling botocore-1.13.50:
      Successfully uninstalled botocore-1.13.50
  Attempting uninstall: s3transfer
    Found existing installation: s3transfer 0.2.1
    Uninstalling s3transfer-0.2.1:
      Successfully uninstalled s3transfer-0.2.1
  Attempting uninstall: boto3
    Found existing installation: boto3 1.10.18
    Uninstalling boto3-1.10.18:
      Successfully uninstalled boto3-1.10.18
  Running setup.py develop for frappe-s3-attachment
Successfully installed boto3-1.4.7 botocore-1.7.48 frappe-s3-attachment python-magic-0.4.18 s3transfer-0.1.13
WARNING: You are using pip version 20.1.1; however, version 20.2.4 is available.
You should consider upgrading via the '/home/frappe/frappe-bench/env/bin/python -m pip install --upgrade pip' command.
Removing intermediate container ec1fca2a96b0
 ---> 9af55226beb9
Successfully built 9af55226beb9
Successfully tagged pratikimprowise/frappe-attachments-s3-worker:1.0.1

nginx image docker file. on docker hub
build it with docker build --no-cache . -t xxxxxx

FROM bitnami/node:12-prod

COPY ./install_app.sh /install_app
RUN /install_app frappe-attachments-s3 https://github.com/zerodhatech/frappe-attachments-s3

FROM frappe/erpnext-nginx:edge

COPY --from=0 /home/frappe/frappe-bench/sites/ /var/www/html/
COPY --from=0 /rsync /rsync
RUN echo -n "\nfrappe-attachments-s3" >> /var/www/html/apps.txt

VOLUME [ "/assets" ]

ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["nginx", "-g", "daemon off;"]

docker build logs

Sending build context to Docker daemon  3.584kB
Step 1/10 : FROM bitnami/node:12-prod
 ---> 946d058b5d18
Step 2/10 : COPY ./install_app.sh /install_app
 ---> f0c7a93365a0
Step 3/10 : RUN /install_app frappe-attachments-s3 https://github.com/zerodhatech/frappe-attachments-s3
 ---> Running in 0b28c633ed54
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  git-man libcurl3-gnutls liberror-perl libgdbm-compat4 libgdbm6 libperl5.28
  libpython2-stdlib libpython2.7-minimal libpython2.7-stdlib mime-support perl
  perl-modules-5.28 python2-minimal python2.7 python2.7-minimal
Suggested packages:
  gettext-base git-daemon-run | git-daemon-sysvinit git-doc git-el git-email
  git-gui gitk gitweb git-cvs git-mediawiki git-svn gdbm-l10n perl-doc
  libterm-readline-gnu-perl | libterm-readline-perl-perl make libb-debug-perl
  liblocale-codes-perl python2-doc python-tk python2.7-doc binutils
  binfmt-support
Recommended packages:
  patch less ssh-client bzip2 file xz-utils netbase
The following NEW packages will be installed:
  git git-man libcurl3-gnutls liberror-perl libgdbm-compat4 libgdbm6
  libperl5.28 libpython2-stdlib libpython2.7-minimal libpython2.7-stdlib
  mime-support perl perl-modules-5.28 python2 python2-minimal python2.7
  python2.7-minimal
0 upgraded, 17 newly installed, 0 to remove and 0 not upgraded.
Need to get 18.8 MB of archives.
After this operation, 102 MB of additional disk space will be used.
Get:1 http://deb.debian.org/debian buster/main amd64 perl-modules-5.28 all 5.28.1-6+deb10u1 [2873 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 libgdbm6 amd64 1.18.1-4 [64.7 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 libgdbm-compat4 amd64 1.18.1-4 [44.1 kB]
Get:4 http://deb.debian.org/debian buster/main amd64 libperl5.28 amd64 5.28.1-6+deb10u1 [3894 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 perl amd64 5.28.1-6+deb10u1 [204 kB]
Get:6 http://deb.debian.org/debian buster/main amd64 libpython2.7-minimal amd64 2.7.16-2+deb10u1 [395 kB]
Get:7 http://deb.debian.org/debian buster/main amd64 python2.7-minimal amd64 2.7.16-2+deb10u1 [1369 kB]
Get:8 http://deb.debian.org/debian buster/main amd64 python2-minimal amd64 2.7.16-1 [41.4 kB]
Get:9 http://deb.debian.org/debian buster/main amd64 mime-support all 3.62 [37.2 kB]
Get:10 http://deb.debian.org/debian buster/main amd64 libpython2.7-stdlib amd64 2.7.16-2+deb10u1 [1912 kB]
Get:11 http://deb.debian.org/debian buster/main amd64 python2.7 amd64 2.7.16-2+deb10u1 [305 kB]
Get:12 http://deb.debian.org/debian buster/main amd64 libpython2-stdlib amd64 2.7.16-1 [20.8 kB]
Get:13 http://deb.debian.org/debian buster/main amd64 python2 amd64 2.7.16-1 [41.6 kB]
Get:14 http://deb.debian.org/debian buster/main amd64 libcurl3-gnutls amd64 7.64.0-4+deb10u1 [330 kB]
Get:15 http://deb.debian.org/debian buster/main amd64 liberror-perl all 0.17027-2 [30.9 kB]
Get:16 http://deb.debian.org/debian buster/main amd64 git-man all 1:2.20.1-2+deb10u3 [1620 kB]
Get:17 http://deb.debian.org/debian buster/main amd64 git amd64 1:2.20.1-2+deb10u3 [5633 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 18.8 MB in 0s (38.3 MB/s)
Selecting previously unselected package perl-modules-5.28.
(Reading database ... 9415 files and directories currently installed.)
Preparing to unpack .../00-perl-modules-5.28_5.28.1-6+deb10u1_all.deb ...
Unpacking perl-modules-5.28 (5.28.1-6+deb10u1) ...
Selecting previously unselected package libgdbm6:amd64.
Preparing to unpack .../01-libgdbm6_1.18.1-4_amd64.deb ...
Unpacking libgdbm6:amd64 (1.18.1-4) ...
Selecting previously unselected package libgdbm-compat4:amd64.
Preparing to unpack .../02-libgdbm-compat4_1.18.1-4_amd64.deb ...
Unpacking libgdbm-compat4:amd64 (1.18.1-4) ...
Selecting previously unselected package libperl5.28:amd64.
Preparing to unpack .../03-libperl5.28_5.28.1-6+deb10u1_amd64.deb ...
Unpacking libperl5.28:amd64 (5.28.1-6+deb10u1) ...
Selecting previously unselected package perl.
Preparing to unpack .../04-perl_5.28.1-6+deb10u1_amd64.deb ...
Unpacking perl (5.28.1-6+deb10u1) ...
Selecting previously unselected package libpython2.7-minimal:amd64.
Preparing to unpack .../05-libpython2.7-minimal_2.7.16-2+deb10u1_amd64.deb ...
Unpacking libpython2.7-minimal:amd64 (2.7.16-2+deb10u1) ...
Selecting previously unselected package python2.7-minimal.
Preparing to unpack .../06-python2.7-minimal_2.7.16-2+deb10u1_amd64.deb ...
Unpacking python2.7-minimal (2.7.16-2+deb10u1) ...
Selecting previously unselected package python2-minimal.
Preparing to unpack .../07-python2-minimal_2.7.16-1_amd64.deb ...
Unpacking python2-minimal (2.7.16-1) ...
Selecting previously unselected package mime-support.
Preparing to unpack .../08-mime-support_3.62_all.deb ...
Unpacking mime-support (3.62) ...
Selecting previously unselected package libpython2.7-stdlib:amd64.
Preparing to unpack .../09-libpython2.7-stdlib_2.7.16-2+deb10u1_amd64.deb ...
Unpacking libpython2.7-stdlib:amd64 (2.7.16-2+deb10u1) ...
Selecting previously unselected package python2.7.
Preparing to unpack .../10-python2.7_2.7.16-2+deb10u1_amd64.deb ...
Unpacking python2.7 (2.7.16-2+deb10u1) ...
Selecting previously unselected package libpython2-stdlib:amd64.
Preparing to unpack .../11-libpython2-stdlib_2.7.16-1_amd64.deb ...
Unpacking libpython2-stdlib:amd64 (2.7.16-1) ...
Setting up libpython2.7-minimal:amd64 (2.7.16-2+deb10u1) ...
Setting up python2.7-minimal (2.7.16-2+deb10u1) ...
Setting up python2-minimal (2.7.16-1) ...
Selecting previously unselected package python2.
(Reading database ... 12135 files and directories currently installed.)
Preparing to unpack .../python2_2.7.16-1_amd64.deb ...
Unpacking python2 (2.7.16-1) ...
Selecting previously unselected package libcurl3-gnutls:amd64.
Preparing to unpack .../libcurl3-gnutls_7.64.0-4+deb10u1_amd64.deb ...
Unpacking libcurl3-gnutls:amd64 (7.64.0-4+deb10u1) ...
Selecting previously unselected package liberror-perl.
Preparing to unpack .../liberror-perl_0.17027-2_all.deb ...
Unpacking liberror-perl (0.17027-2) ...
Selecting previously unselected package git-man.
Preparing to unpack .../git-man_1%3a2.20.1-2+deb10u3_all.deb ...
Unpacking git-man (1:2.20.1-2+deb10u3) ...
Selecting previously unselected package git.
Preparing to unpack .../git_1%3a2.20.1-2+deb10u3_amd64.deb ...
Unpacking git (1:2.20.1-2+deb10u3) ...
Setting up perl-modules-5.28 (5.28.1-6+deb10u1) ...
Setting up mime-support (3.62) ...
Setting up libcurl3-gnutls:amd64 (7.64.0-4+deb10u1) ...
Setting up libpython2.7-stdlib:amd64 (2.7.16-2+deb10u1) ...
Setting up git-man (1:2.20.1-2+deb10u3) ...
Setting up libgdbm6:amd64 (1.18.1-4) ...
Setting up python2.7 (2.7.16-2+deb10u1) ...
Setting up libpython2-stdlib:amd64 (2.7.16-1) ...
Setting up python2 (2.7.16-1) ...
Setting up libgdbm-compat4:amd64 (1.18.1-4) ...
Setting up libperl5.28:amd64 (5.28.1-6+deb10u1) ...
Setting up perl (5.28.1-6+deb10u1) ...
Setting up liberror-perl (0.17027-2) ...
Setting up git (1:2.20.1-2+deb10u3) ...
Processing triggers for libc-bin (2.28-10) ...
Cloning into 'frappe'...
Cloning into 'frappe-attachments-s3'...
yarn install v1.22.10
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
warning " > [email protected]" has unmet peer dependency "[email protected] - 3".
warning " > [email protected]" has unmet peer dependency "popper.js@^1.16.0".
[4/4] Building fresh packages...
$ yarn run snyk-protect
yarn run v1.22.10
$ snyk protect
Successfully applied Snyk patches

Done in 10.50s.
Done in 79.96s.
yarn run v1.22.10
$ FRAPPE_ENV=production node rollup/build.js --app frappe-attachments-s3
Production mode
✔ Built js/moment-bundle.min.js
✔ Built js/libs.min.js
Done in 1.58s.
yarn install v1.22.10
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
warning " > [email protected]" has unmet peer dependency "[email protected] - 3".
warning " > [email protected]" has unmet peer dependency "popper.js@^1.16.0".
[4/4] Building fresh packages...
Done in 23.49s.
yarn add v1.22.10
[1/4] Resolving packages...
warning node-sass > node-gyp > [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
[2/4] Fetching packages...
[3/4] Linking dependencies...
warning " > [email protected]" has unmet peer dependency "[email protected] - 3".
warning " > [email protected]" has unmet peer dependency "popper.js@^1.16.0".
[4/4] Building fresh packages...
success Saved lockfile.
success Saved 10 new dependencies.
info Direct dependencies
└─ [email protected]
info All dependencies
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
└─ [email protected]
$ yarn run snyk-protect
yarn run v1.22.10
$ snyk protect
Successfully applied Snyk patches

Done in 13.01s.
Done in 38.65s.
cp: cannot stat '/home/frappe/frappe-bench/apps/frappe-attachments-s3/frappe-attachments-s3/public/*': No such file or directory
Removing intermediate container 0b28c633ed54
 ---> 30d81b69027f
Step 4/10 : FROM frappe/erpnext-nginx:edge
 ---> dd4835b314f1
Step 5/10 : COPY --from=0 /home/frappe/frappe-bench/sites/ /var/www/html/
 ---> 2dc9ed209b3f
Step 6/10 : COPY --from=0 /rsync /rsync
 ---> ab0f4270f9d8
Step 7/10 : RUN echo -n "\nfrappe-attachments-s3" >> /var/www/html/apps.txt
 ---> Running in cfa197bd7125
Removing intermediate container cfa197bd7125
 ---> 2de4459592e9
Step 8/10 : VOLUME [ "/assets" ]
 ---> Running in 7b0a46c5ecc9
Removing intermediate container 7b0a46c5ecc9
 ---> f269a997b1cf
Step 9/10 : ENTRYPOINT ["/docker-entrypoint.sh"]
 ---> Running in a2d1de8dd4ce
Removing intermediate container a2d1de8dd4ce
 ---> 1beecf6487cd
Step 10/10 : CMD ["nginx", "-g", "daemon off;"]
 ---> Running in 673b1a721d19
Removing intermediate container 673b1a721d19
 ---> 1025638c5aff
Successfully built 1025638c5aff
Successfully tagged pratikimprowise/frappe-attachments-s3-nginx:1.0.1

added these to values.yaml of erpnext helm chart as per these Manual way step,

nginxImage:
  repository: docker.io/pratikimprowise/frappe-attachments-s3-nginx
  tag: 1.0.1
  pullPolicy: Always

pythonImage:
  repository: docker.io/pratikimprowise/frappe-attachments-s3-worker
  tag: 1.0.1
  pullPolicy: Always

execute it

helm install frappe . --create-namespace --namespace erpnext  -f values.yaml

Version

Docker v19.06
k3s version v1.19.3
helm version v3.3.4
frappe-attachments-s3-worker
frappe-attachments-s3-nginx

Company uploaded Image not showing

Attach field not displaying correctly

Expected behaviour

When having a custom doctype with a Attach field, the link must render to s3

Actual behaviour

Attach field renders url to server

Step to reproduce

Activate S3 Attachments
Attach a file or image using Attach Field on a doctype

Version

Extension check for pdf files

Pdf file can have capital letters extension

Current behaviour: Extension check is only for "pdf"

Error: Document has been modified after you have opened it . Please refresh to get the latest document.

Hi, I am using this app on frappe cloud. I got the bellow error when I upload and deleting the file from a Doctype.

"Error: Document has been modified after you have opened it (2022-09-26 11:59:45.021047, 2022-09-26 11:59:45.165984). Please refresh to get the latest document."

Version
frappe 13.41.1

attach file to data import : it gives error

Expected behaviour

Attach import file to Data import should work

Actual behaviour

following error happens
for private file : URL must start with http:// or https://
for public file : FileNotFoundError: [Errno 2] No such file or directory

Step to reproduce

Go to Data import
Attach file to 'Import file'
on attaching as private following error happens

on attaching as public file , following error happens

Version

version 14

Boto version required by s3_attachments is incompatible with boto version required by frappe 15 dev

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
frappe 15.0.0.dev0 requires boto3~=1.18.49, but you have boto3 1.4.7 which is incompatible.
Installing erpnext_ocr
$ /home/user/frappe/env/bin/python -m pip install --quiet --upgrade -e /home/user/frappe/apps/erpnext_ocr
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
frappe-s3-attachment 0.0.1 requires boto3==1.4.7, but you have boto3 1.18.65 which is incompatible.
frappe-s3-attachment 0.0.1 requires botocore==1.7.48, but you have botocore 1.21.65 which is incompatible.

on site installation it gives error : cannot import name 'Mapping' from 'collections' (/usr/lib/python3.10/collections/init.py)

Actual behaviour

Installing frappe_s3_attachment...
Updating DocTypes for frappe_s3_attachment: [========================================] 100%
frappe.integrations.doctype.s3_backup_settings.s3_backup_settings.take_backups_daily is not a valid method: cannot import name 'Mapping' from 'collections' (/usr/lib/python3.10/collections/init.py)

Step to reproduce

bench get-app https://github.com/shridarpatil/frappe-attachments-s3
bench --site sitename install-app frappe_s3_attachment

Version

v-14

Is this app still being maintained? We can help.

Hey,

We want to use this app for ourselves, and noticed that there's several pending PRs and issues. I'm sure we'll come up with fixes and improvements as we use it as well.

Didn't want to outright fork it, thought I'd ask first if you guys would like to let us maintain it.

Cheers!

Having issues when emailing an attachment within ERPNext

  File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/boto3/s3/inject.py", line 110, in upload_file
    extra_args=ExtraArgs, callback=Callback)
  File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/boto3/s3/transfer.py", line 275, in upload_file
    future.result()
  File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
OSError: [Errno 2] No such file or directory: './site1.local/api/method/frappe_s3_attachment.controller.generate_file?key=private/2018/04/03/Lead/some_pic.PNG'
  File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/boto3/s3/inject.py", line 110, in upload_file
    extra_args=ExtraArgs, callback=Callback)
  File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/boto3/s3/transfer.py", line 275, in upload_file
    future.result()
  File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
OSError: [Errno 2] No such file or directory: './site1.local/api/method/frappe_s3_attachment.controller.generate_file?key=private/2018/04/03/Lead/some_pic.PNG'

How do we restore existing files from S3 to frappecloud?

I have already migrated the existing files to S3, but now I have decided to use frappecloud itself. How can we restore files that have already been synced to S3 back to frappecloud?

Got error when attachment is private on the dialog

Expected behaviour

Hi. I using this project in my app and my app deploy on AWS EC2
When i in dialog append attachment, if attachment is public just fine
But if attachment is private I always got below error

In email dialog also got below error if attachment is prvate

This like issue #6

Step to reproduce

Use Dialog API create a new dialog, It must have a attach file field
Select file for attach field. Then ensure is_private is true
Click Upload.

Full Traceback

Traceback (most recent call last):
  File "apps/frappe/frappe/app.py", line 69, in application
    response = frappe.api.handle()
  File "apps/frappe/frappe/api.py", line 55, in handle
    return frappe.handler.handle()
  File "apps/frappe/frappe/handler.py", line 38, in handle
    data = execute_cmd(cmd)
  File "apps/frappe/frappe/handler.py", line 76, in execute_cmd
    return frappe.call(method, **frappe.form_dict)
  File "apps/frappe/frappe/__init__.py", line 1448, in call
    return fn(*args, **newargs)
  File "apps/fab/fab/utils.py", line 123, in make_email
    return _make(
  File "apps/frappe/frappe/core/doctype/communication/email.py", line 169, in _make
    add_attachments(comm.name, attachments)
  File "apps/frappe/frappe/core/doctype/communication/email.py", line 546, in add_attachments
    _file.save(ignore_permissions=True)
  File "apps/frappe/frappe/model/document.py", line 310, in save
    return self._save(*args, **kwargs)
  File "apps/frappe/frappe/model/document.py", line 332, in _save
    return self.insert()
  File "apps/frappe/frappe/model/document.py", line 261, in insert
    self.run_before_save_methods()
  File "apps/frappe/frappe/model/document.py", line 1052, in run_before_save_methods
    self.run_method("validate")
  File "apps/frappe/frappe/model/document.py", line 941, in run_method
    out = Document.hook(fn)(self, *args, **kwargs)
  File "apps/frappe/frappe/model/document.py", line 1260, in composer
    return composed(self, method, *args, **kwargs)
  File "apps/frappe/frappe/model/document.py", line 1242, in runner
    add_to_return_value(self, fn(self, *args, **kwargs))
  File "apps/frappe/frappe/model/document.py", line 938, in fn
    return method_object(*args, **kwargs)
  File "apps/frappe/frappe/core/doctype/file/file.py", line 116, in validate
    self.validate_duplicate_entry()
  File "apps/frappe/frappe/core/doctype/file/file.py", line 241, in validate_duplicate_entry
    self.generate_content_hash()
  File "apps/frappe/frappe/core/doctype/file/file.py", line 276, in generate_content_hash
    frappe.throw(_("File {0} does not exist").format(self.file_url))
  File "apps/frappe/frappe/__init__.py", line 504, in throw
    msgprint(
  File "apps/frappe/frappe/__init__.py", line 479, in msgprint
    _raise_exception()
  File "apps/frappe/frappe/__init__.py", line 434, in _raise_exception
    raise raise_exception(msg)
frappe.exceptions.ValidationError: File /api/method/frappe_s3_attachment.controller.generate_file?key=2022/08/19/Fab PO/0GLVSWOK_20220711.txt&file_name=20220711.txt does not exist

My Tried

Because file.py the validate_url() will check file url is start with /public or /private
if not, will check is start will http or https else throw error.
Then generate_content_hash() do nothing if file url is start will http .
https://github.com/frappe/frappe/blob/version-13/frappe/core/doctype/file/file.py#L290

So I tried to update below code.
ensure private url is full url path.

# https://github.com/zerodha/frappe-attachments-s3/blob/master/frappe_s3_attachment/controller.py#L224
file_url = """/api/method/{0}?key={1}&file_name={2}""".format(method, key, doc.file_name)
# To
file_url = """{0}/api/method/{1}?key={2}&file_name={3}""".format(frappe.utils.get_url(),method, key, doc.file_name)

The frappe.utils.get_url() will fetch hostname from site_config.json. if not it will fetch your site name.
After update. I avoid occur error in validate_url() or generate_content_hash(). now I can upload private file in s3.
In my case, I set the domain for site_config.json the hostname
If in the local, hostname maybe can set http://localhost or http://127.0.0.1

Version

Frappe v13.32
ERPNext v13.33

prepared report fails

Expected behaviour

Prepared reports should work

Actual behaviour

raceback with variables (most recent call last):
  File "apps/frappe/frappe/desk/query_report.py", line 293, in get_prepared_report_result
    if data := json.loads(doc.get_prepared_data().decode("utf-8")):
      report = <Report: General Ledger>
      filters = {'company': 'Tech India Engineers Pvt. Limited', 'from_date': '2023-04-01', 'to_date': '2024-03-20', 'account': [], 'party': [], 'group_by': 'Group by Voucher (Consolidated)', 'cost_center': [], 'project': [], 'include_dimensions': 1, 'include_default_book_entries': 1}
      dn = '70e883c13d'
      user = '[email protected]'
      get_completed_prepared_report = <function get_completed_prepared_report at 0x7f09f4021630>
      get_report_data = <function get_prepared_report_result.<locals>.get_report_data at 0x7f09f2bb40d0>
      report_data = {}
      doc = <PreparedReport: 70e883c13d>
  File "apps/frappe/frappe/core/doctype/prepared_report/prepared_report.py", line 56, in get_prepared_data
    return gzip_decompress(attached_file.get_content())
      self = <PreparedReport: 70e883c13d>
      with_file_name = False
      attachments = [{'name': 'fa79dbf06d', 'file_name': '2024-59-20-9:3.json.gz', 'file_url': '/api/method/frappe_s3_attachment.controller.generate_file?key=erpnext/2024/03/20/Prepared Report/M2ZMY0IU_2024-59-20-93.json.gz&file_name=2024-59-20-9:3.json.gz', 'is_private': 1}]
      attachment = {'name': 'fa79dbf06d', 'file_name': '2024-59-20-9:3.json.gz', 'file_url': '/api/method/frappe_s3_attachment.controller.generate_file?key=erpnext/2024/03/20/Prepared Report/M2ZMY0IU_2024-59-20-93.json.gz&file_name=2024-59-20-9:3.json.gz', 'is_private': 1}
      attached_file = <File: fa79dbf06d>
  File "apps/frappe/frappe/core/doctype/file/file.py", line 472, in get_content
    self.validate_file_url()
      self = <File: fa79dbf06d>
  File "apps/frappe/frappe/core/doctype/file/file.py", line 177, in validate_file_url
    frappe.throw(
      self = <File: fa79dbf06d>
  File "apps/frappe/frappe/__init__.py", line 534, in throw
    msgprint(
      msg = 'URL must start with http:// or https://'
      exc = <class 'frappe.exceptions.ValidationError'>
      title = 'Invalid URL'
      is_minimizable = False
      wide = False
      as_list = False
  File "apps/frappe/frappe/__init__.py", line 502, in msgprint
    _raise_exception()
      title = 'Invalid URL'
      as_table = False
      as_list = False
      indicator = 'red'
      alert = False
      primary_action = None
      is_minimizable = False
      wide = False
      sys = <module 'sys' (built-in)>
      out = {'message': 'URL must start with http:// or https://', 'title': 'Invalid URL', 'indicator': 'red', 'raise_exception': 1}
      _raise_exception = <function msgprint.<locals>._raise_exception at 0x7f09f2bb5fc0>
      _strip_html_tags = <functools._lru_cache_wrapper object at 0x7f09f20decf0>
      inspect = <module 'inspect' from '/usr/lib/python3.10/inspect.py'>
      msg = 'URL must start with http:// or https://'
      raise_exception = <class 'frappe.exceptions.ValidationError'>
      strip_html_tags = <function strip_html_tags at 0x7f09f70945e0>
  File "apps/frappe/frappe/__init__.py", line 451, in _raise_exception
    raise raise_exception(msg)
      inspect = <module 'inspect' from '/usr/lib/python3.10/inspect.py'>
      msg = 'URL must start with http:// or https://'
      raise_exception = <class 'frappe.exceptions.ValidationError'>
frappe.exceptions.ValidationError: URL must start with http:// or https://

Step to reproduce

Go to general ledger report in v14 . it is a prepared report and it fails

Version

v14

Attach a PDF file and try to open it, it fails

Expected behaviour

PDF file attached, should open

Actual behaviour

Attach a PDF file and try to open it, it fails

Step to reproduce

Go to any doctype, ex purchase invoice
Attach a PDF file. It attaches without issue
Try to open it. it gives error

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>EYBDWNQ6GTD522JQ</RequestId>
<HostId>8GexSnDRLbOOdZDRFXPByByWW/VKXlw7AvKndH6s9rOs0UIbrXgPEVog6d9MX6ov2DigJNJPCBw=</HostId>
</Error>

Note: when checked in amazon bucket the PDF file is there and it opens inside amazon without issue

Version

while migrating existing files

App Versions

{
	"communications": "0.0.1",
	"erpnext": "14.33.0",
	"ezygst": "0.0.1",
	"frappe": "14.40.1",
	"frappe_s3_attachment": "0.0.1",
	"hrms": "14.10.1",
	"india_compliance": "0.0.1",
	"m_pr": "0.0.1",
	"payments": "0.0.1",
	"reconciler": "0.0.1"
}

Route

Form/S3 File Attachment/S3 File Attachment

Traceback

Traceback (most recent call last):
  File "apps/frappe/frappe/app.py", line 66, in application
    response = frappe.api.handle()
               ^^^^^^^^^^^^^^^^^^^
  File "apps/frappe/frappe/api.py", line 54, in handle
    return frappe.handler.handle()
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "apps/frappe/frappe/handler.py", line 47, in handle
    data = execute_cmd(cmd)
           ^^^^^^^^^^^^^^^^
  File "apps/frappe/frappe/handler.py", line 85, in execute_cmd
    return frappe.call(method, **frappe.form_dict)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "apps/frappe/frappe/__init__.py", line 1608, in call
    return fn(*args, **newargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "apps/frappe_s3_attachment/frappe_s3_attachment/controller.py", line 322, in migrate_existing_files
    upload_existing_files_s3(file['name'], file['file_name'])
  File "apps/frappe_s3_attachment/frappe_s3_attachment/controller.py", line 275, in upload_existing_files_s3
    key = s3_upload.upload_files_to_s3_with_key(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "apps/frappe_s3_attachment/frappe_s3_attachment/controller.py", line 110, in upload_files_to_s3_with_key
    mime_type = magic.from_file(file_path, mime=True)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "env/lib/python3.11/site-packages/magic.py", line 153, in from_file
    return m.from_file(filename)
           ^^^^^^^^^^^^^^^^^^^^^
  File "env/lib/python3.11/site-packages/magic.py", line 95, in from_file
    with open(filename):
         ^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './Frappe/private/files/HJFSZG76_Arvindtrail.txt'

Request Data

{
	"type": "POST",
	"args": {},
	"headers": {},
	"error_handlers": {},
	"url": "/api/method/frappe_s3_attachment.controller.migrate_existing_files"
}

Response Data

{
	"exception": "FileNotFoundError: [Errno 2] No such file or directory: './Frappe/private/files/HJFSZG76_Arvindtrail.txt'"
}

frappe_docker unable to upload files to s3 bucket with no error logs

Expected behaviour

Actual behaviour

Step to reproduce

Version

Why year/month/day?

Hi,

Just a quick question.
Why using year/month/day in the name of the attachment?

David

doc with attachment is cancelled , on amend it gives error

Expected behaviour

on doc with attachment, when amended should not give error

Actual behaviour

Invalid URL
URL must start with http:// or https://

Step to reproduce

create any document, say quotation
attach any file as private
save and cancel the quotation
click on amend, and save
It gives error
Invalid URL
URL must start with http:// or https://

Version

version 14

Issues when import data using Data Import

Traceback (most recent call last):
File "/home/frappe/frappe-bench/apps/frappe/frappe/app.py", line 62, in application
response = frappe.handler.handle()
File "/home/frappe/frappe-bench/apps/frappe/frappe/handler.py", line 22, in handle
data = execute_cmd(cmd)
File "/home/frappe/frappe-bench/apps/frappe/frappe/handler.py", line 53, in execute_cmd
return frappe.call(method, **frappe.form_dict)
File "/home/frappe/frappe-bench/apps/frappe/frappe/init.py", line 939, in call
return fn(*args, **newargs)
File "/home/frappe/frappe-bench/apps/frappe/frappe/desk/form/save.py", line 22, in savedocs
doc.save()
File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 256, in save
return self._save(*args, **kwargs)
File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 290, in save
self.run_before_save_methods()
File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 859, in run_before_save_methods
self.run_method("validate")
File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 755, in run_method
out = Document.hook(fn)(self, *args, **kwargs)
File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 1024, in composer
return composed(self, method, *args, **kwargs)
File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 1007, in runner
add_to_return_value(self, fn(self, *args, **kwargs))
File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 749, in
fn = lambda self, *args, **kwargs: getattr(self, method)(*args, **kwargs)
File "/home/frappe/frappe-bench/apps/frappe/frappe/core/doctype/data_import/data_import.py", line 28, in validate
upload(data_import_doc=self, from_data_import="Yes", validate_template=True)
File "/home/frappe/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer.py", line 267, in upload
fname, fcontent = get_file(data_import_doc.import_file)
File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/file_manager.py", line 278, in get_file
file_path = get_file_path(fname)
File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/file_manager.py", line 305, in get_file_path
frappe.throw(("There is some problem with the file url: {0}").format(file_path))
File "/home/frappe/frappe-bench/apps/frappe/frappe/init.py", line 323, in throw
msgprint(msg, raise_exception=exc, title=title, indicator='red')
File "/home/frappe/frappe-bench/apps/frappe/frappe/init.py", line 309, in msgprint
_raise_exception()
File "/home/frappe/frappe-bench/apps/frappe/frappe/init.py", line 282, in _raise_exception
raise raise_exception(encode(msg))
ValidationError: There is some problem with the file url: https://s3.us-west-2.amazonaws.com/abc/abc/2018/05/11/Data Import/N0ICE3V2_CustomerShop888a80csv

The url is not the right format in ERPNEXT.
https://github.com/frappe/frappe/blob/fc99d691d13cfe79f393e9a66c034c7280c97c09/frappe/utils/file_manager.py#L336

Timestamp mismatch error on uploading file in Patient table

I have customised patient table of healthcare module and added one child table which take any type of attachments.
On inserting i am getting timestamp mismatch error

Boto3 outdated

Frappe requires a newer version of boto3 and botocore

zerodha / frappe-attachments-s3 Goto Github PK

frappe-attachments-s3's Introduction

Frappe S3 Attachment

Features.

Installation.

Configuration Setup.

License

frappe-attachments-s3's People

Contributors

Stargazers

Watchers

Forkers

frappe-attachments-s3's Issues

Expected behaviour

Actual behaviour

Step to reproduce

VersionV13

Expected behaviour

Actual behaviour

Step to reproduce

Version

Expected behaviour

Actual behaviour

Step to reproduce

Version

Expected behaviour

Actual behaviour

Step to reproduce

Version

Expected behaviour

Actual behaviour

Step to reproduce

Version

Actual behaviour

Step to reproduce

Version

Expected behaviour

Step to reproduce

Full Traceback

My Tried

Version

Expected behaviour

Actual behaviour

Step to reproduce

Version

Expected behaviour

Actual behaviour

Step to reproduce

Version

App Versions

Route

Traceback

Request Data

Response Data

Expected behaviour

Actual behaviour

Step to reproduce

Version

Expected behaviour

Actual behaviour

Step to reproduce

Version

Recommend Projects

Recommend Topics

Recommend Org