aagam41 / db_file_storage Goto Github PK

Not able to save files in windows environment.

ValueError('need more than 1 value to unpack',)
db_file_storage/storage.py in _get_storage_attributes, line 70

Is there planned support for Django 1.11 or should I try working on it myself?

Currently running the tests with Django 1.11.7 results with 3 failures. test_admin_form_widget_shows_proper_filename, test_binary_file, and test_form_widget_shows_proper_filename.

With the changes added in 0.4.4, we were getting the following AttributeError when using DBClearableFileInput:

File "db_file_storage/form_widgets.py", line 35, in get_context
context['widget']['display'] = get_link_display(value.url)
AttributeError: 'NoneType' object has no attribute 'url'

The error occurs when trying to use this widget on a form with no values. We have fixed this on our end by creating a subclass of DBClearableFileInput that does a basic check for value when it is None, but we wanted to point it out here as well.

Let us know if you need additional details. We appreciate the work you're doing here. Thanks!

Description

I'm sure this is an issue on my end but I can't figure out why Django is throwing an ImportError for db_file_storage.urls within my application's main urlpatterns (MyApp/urls.py). I've followed the configuration setup from here https://django-db-file-storage.readthedocs.io/en/master/ and I have the rest of the setup complete, I just can't seem to get past this Import issue.

When testing using the Django shell from CLI I am able to import 'db_file_storage.urls' fine and inspect the object. It's only when accessing the application from the browser does the Import fail to resolve and throws an error. I also tested importing db_file_storage at the top of my urls.py which was successful. Any attempts to include urls from db_file_storage resulted in the same Import error.

Environment

Django 2.0.1
Python 3.4.9 (Virtual Environment)
Apache 2.4.6 (CentOS)

Browser Error

ImportError at /
No module named 'db_file_storage.urls'
Request Method: | GET
Request URL: https://myapp.internalsite.com/
Django Version: 2.0.1
Exception Type: ImportError
Exception Value: No module named 'db_file_storage.urls'
Exception Location: <frozen importlib._bootstrap> in _find_and_load_unlocked, line 2224
Python Executable /usr/bin/python3
Python Version: 3.4.9
Python Path: ['/opt/MyProject',
 '/opt/virtual-env/lib64/python34.zip',
 '/opt/virtual-env/lib64/python3.4',
 '/opt/virtual-env/lib64/python3.4/plat-linux',
 '/opt/virtual-env/lib64/python3.4/lib-dynload',
 '/usr/lib64/python3.4',
 '/usr/lib/python3.4',
 '/opt/virtual-env/lib/python3.4/site-packages']

urls.py (version 1) - throws ImportError: No module named 'db_file_storage.urls'

from django.contrib import admin
from django.urls import path, include
from django.conf.urls.static import static
from django.conf import settings

urlpatterns = [
    path('', include('reports.urls')),
    path('admin/', admin.site.urls),
    path('findings/', include('findings.urls')),
    path('reports/',include('reports.urls')),
    path('accounts/', include('django.contrib.auth.urls')),
    path('files/', include('db_file_storage.urls')),
] + static(settings.STATIC_URL,document_root=settings.STATIC_ROOT)

if settings.DEBUG:
    urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

urls.py (version 2) - Commented out the offending line and included an import of db_file_storage. Throws no errors, but doesn't include the necessary urls for db_file_storage

from django.contrib import admin
from django.urls import path, include
from django.conf.urls.static import static
from django.conf import settings
import db_file_storage

urlpatterns = [
    path('', include('reports.urls')),
    path('admin/', admin.site.urls),
    path('findings/', include('findings.urls')),
    path('reports/',include('reports.urls')),
    path('accounts/', include('django.contrib.auth.urls')),
#  path('files/', include('db_file_storage.urls')),
] + static(settings.STATIC_URL,document_root=settings.STATIC_ROOT)

if settings.DEBUG:
    urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

settings.py

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django_seo_js',
    'findings',
    'reports',
    'sslserver',
    'widget_tweaks',
    'guardian',
    'db_file_storage',
]

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

MIDDLEWARE_CLASSES = (
    'django_seo_js.middleware.EscapedFragmentMiddleware',  # If you're using #!
    'django_seo_js.middleware.UserAgentMiddleware',  # If you want to detect by user agent
)

ROOT_URLCONF = 'myproject.urls'

AUTHENTICATION_BACKENDS = (
    'django.contrib.auth.backends.ModelBackend', # this is default
    'guardian.backends.ObjectPermissionBackend',
)

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [os.path.join(BASE_DIR, 'templates')],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
                'django.template.context_processors.request',
                'django.template.context_processors.static',
            ],
        },
    },
]

WSGI_APPLICATION = 'myproject.wsgi.application'


DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': '****',
        'USER': '****',
        'PASSWORD': '******',
        'HOST': '******',
        'PORT': 5432
    }
}


# Password validation
# https://docs.djangoproject.com/en/2.0/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]

# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/2.0/howto/static-files/
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
DEFAULT_FILE_STORAGE = 'db_file_storage.storage.DatabaseFileStorage'

PROJECT_ROOT = os.path.dirname(os.path.abspath(__file__))
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(PROJECT_ROOT, 'static')

httpd.conf - relevant configuration

IncludeOptional conf.d/*.conf

Alias /media/ /opt/MyProject/media/
Alias /static/ /opt/MyProject/myproject/static/

<Directory /opt/MyProject/myproject/static>
    Require all granted
</Directory>

<Directory /opt/MyProject/media>
    Require all granted
</Directory>

WSGIScriptAlias / /opt/MyProject/myproject/wsgi.py
LoadModule wsgi_module /opt/virtual-env/lib/python3.4/site-packages/mod_wsgi/server/mod_wsgi-py34.cpython-34m.so

WSGIDaemonProcess MyProject python-home=/opt/virtual-env/ python-path=/opt/MyProject:/opt/virtual-env/lib/python3.4/site-packages socket-user=nobody

WSGIProcessGroup MyProject
WSGISocketPrefix /var/run/wsgi

<Directory /opt/MyProject>
    <Files wsgi.py>
        Require all granted
    </Files>
</Directory>

To be called after model_instance.delete()

Add it to the docs.

IMHO, it should be stored in the settings.

Something like:

{
    'default': {} # config for the default storage,
    'storage_1': {} # config for another storage
}

https://github.com/victor-o-silva/db_file_storage/blob/master/db_file_storage/storage.py#L109
IMHO, the metadata (model class, field names etc) should not be saved in the filename column.

Also, exists() method does not work properly.
https://github.com/victor-o-silva/db_file_storage/blob/master/db_file_storage/storage.py#L136
The file is being looked up by it's correct name, but it's saved with the meta data.

I think on line 60, mimetype = content.file.content_type should be mimetype = content.content_type.

add db_file_storage.model_utils.delete_file_if_needed o the docs

I like to have Django admin support for all models in DEBUG mode, as it helps understanding of the data model, making onboarding new devs easier.

The initial mixin could put the entire model into "read only" mode within Django admin - that is still useful.

Full edit support is easy for a text based file store.

The main field where editing is useful would be the mimetype. Sometimes that needs to be made more precise, e.g. using newly defined mimetypes like application/vnd.open-api+yaml for old files which were a normal yaml mimetype.

Widgets for uploading raw binary data is a bit tricky, but is possible. e.g. https://stackoverflow.com/questions/58529099/adding-file-upload-widget-for-binaryfield-to-django-admin

However the admin for the FileModel could simply disable "add" until there is some way to upload the binary file.

This module works great on my Linux dev box, but not on my Windows dev box. I am able to upload a file successfully (through the Admin app), but a GET request (or clicking the file download link in the Admin app) returns a 400 error with the message "Invalid request". I think the problem lies with the different path separators in the file name (back slash in Windows, forward slash in Linux), because when I change the direction of the path separators in the file name to back slashes in my GET requests on my Windows dev box, the file downloads perfectly:

this fails:
curl -X GET http://localhost:8000/files/download/?name=myapp.DatabaseFile/bytes/filename/mimetype/test.png

this works:
curl -X GET http://localhost:8000/files/download/?name=myapp.DatabaseFile\bytes\filename\mimetype\test.png

Is there a way to get it work with Form wizards ?, I just added in my Form wizard class as:
file_storage = DatabaseFileStorage()

It raised this error:
list index out of range
The exception came from the method exists at line 104 in storage.py

I thought it was because I have to add the prefix like in upload_to: app_name/class/bytes/filename/mimetype, but I didn't know how to get it work.

If this issue doesn't go here, please drop it.
Thanks in advance.

Library doesn't seem to work with django's ContentFile class, as described by @pwhipp on issue #2.

After quite a bit of mucking around, I was able to get this working without putting DEFAULT_FILE_STORAGE = 'db_file_storage.storage.DatabaseFileStorage' in my settings as is described in the README.

I wrote up my use case in a bit more detail at kimetrica/django-binary-database-files#31 which I was also experimenting with.

I found how to get it working with the use of storage=DatabaseStorage(..) in their tests, and notice there isn't any similar tests or docs here which explain how to do that. See also kimetrica/django-binary-database-files#32 about that. I had seen storage was available in the core Django field docs , but I tried quite a few incorrect incantations before realising how simple it was.

The models setup I have used is

from db_file_storage.storage import DatabaseFileStorage


class Filestore(models.Model):
    data = models.BinaryField()
    filename = models.CharField(max_length=255)
    mimetype = models.CharField(max_length=50)


class Image(models.Model):
    name = models.CharField(max_length=100)
    image = models.ImageField(
        storage=DatabaseFileStorage(),
        upload_to='app.Filestore/data/filename/mimetype',
        blank=True, null=True)

I think this should be explained in the docs, and have tests to make sure it works correctly without relying on DEFAULT_FILE_STORAGE.

remove license file
set appropriate classifier in setup.py

Hello,
I am facing an issue, I keep having the error TemplateDoesNotExist - db_file_storage/widgets/admin_clearable_file_input.html when I package my app.
I use other third party apps, no issue, all templates, static are collected and there, but for some reason not for db_file_storage. The app is properly referenced in INSTALLED_APPS

`Django tried loading these templates, in this order:

Using engine django:

django.template.loaders.filesystem.Loader: /Users/toto/.pex/install/Django-2.0.2-py3.6.egg.0a9fb7bc8d9dbc7082915e607b8e4ee0e96686d2/Django-2.0.2-py3.6.egg/django/forms/templates/db_file_storage/widgets/admin_clearable_file_input.html (Source does not exist)`

Django-2.0.2,
Python Version: 3.6.0
django-db-file-storage==0.5.1

any idea where it can come from ? thx

Also drop support for unsupported Django versions.

It looks like this project has had a few pull requests opened over the last couple of years, but they haven't been looked at. If maintenance is a burden, have you considered transferring this project over to Jazzband?

Hi,

I#m trying to save a file with the help of the FileField Field. I'm not quiet sure, how to pass my file to the save function.

To test things I have a simple Model which uses the FileField:

class UserFiles(models.Model): file = models.FileField(upload_to='TestProject.UserFiles_FileModel/bytes/filename/mimetype') title = models.CharField(max_length=64)

and the corresponding FileModel:
class UserFiles_FileModel(models.Model): bytes = models.TextField() filename = models.CharField(max_length=255) mimetype = models.CharField(max_length=50)

I'm opening a testfile with
file = open("somefile.txt","r")

I'm then creating a new Table entry via:
entry = UserFiles(title="Sometitle")

I'm then trying to save the file with:
entry.file.save("somefile.txt",file)
Which gives back the Error: "TypeError: 'str' does not support the buffer interface"

Here is the stacktrace:

`---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
in ()
----> 1 entry.file.save("somefile.txt",file)

/usr/local/lib/python3.4/dist-packages/django/db/models/fields/files.py in save(self, name, content, save)
91
92 if func_supports_parameter(self.storage.save, 'max_length'):
---> 93 self.name = self.storage.save(name, content, max_length=self.field.max_length)
94 else:
95 warnings.warn(

/usr/local/lib/python3.4/dist-packages/django/core/files/storage.py in save(self, name, content, max_length)
61 name = self.get_available_name(name)
62
---> 63 name = self._save(name, content)
64
65 # Store filenames with forward slashes, even on Windows

/usr/local/lib/python3.4/dist-packages/db_file_storage/storage.py in _save(self, name, content)
116 new_filename = self._get_unique_filename(model_cls,
117 filename_field, name)
--> 118 encoded_bytes = self._get_encoded_bytes_from_file(content)
119 mimetype = getattr(content.file, 'content_type', 'text/plain')
120

/usr/local/lib/python3.4/dist-packages/db_file_storage/storage.py in _get_encoded_bytes_from_file(self, _file)
35 _file.seek(0)
36 file_content = _file.read()
---> 37 return base64.b64encode(file_content)
38
39 def _get_file_from_encoded_bytes(self, encoded_bytes):

/usr/lib/python3.4/base64.py in b64encode(s, altchars)
60 """
61 # Strip off the trailing newline
---> 62 encoded = binascii.b2a_base64(s)[:-1]
63 if altchars is not None:
64 assert len(altchars) == 2, repr(altchars)
`

Currently the file contents are stored in a TextField

I would like to store it in a bytea when using postgres.

Django has a BinaryField type (https://docs.djangoproject.com/en/2.1/ref/models/fields/#binaryfield)

Which should translate to bytea and the correct blob type for each database.

I believe this kind of behaviour could be configurable

Exists any feature to enable cache header in response?
You accept pull request for this?

Since there is no other way to contact @Aagam41. I'm opening this issue to ask when can we expect a new release supporting the newer version of Django?

With Django 1.9, getting error "db_file_storage/views.py", line 2, in from django.core.servers.basehttp import FileWrapper ImportError: cannot import name FileWrapper".

Should line 2 be replaced with "from wsgiref.util import FileWrapper"?

Hi how I can upload image / file though api ?

when using delete_file_if_needed to remove file, I got error as follows:
AttributeError: 'MyModel' object has no attribute 'id'

because I customized my model's primary_key name.

Templates are not bundled with current release.

In March I was in a hurry and I locally fixed a path issue where Django always worked with '/' but db_file_storage worked with os.sep (I fixed it locally because it was above the @sp-niemand fork that fixed Django 1.9, so contributing the fix was not easy anyway.)

This is my fix:

https://github.com/SonOfLilit/db_file_storage/commits/master

I recently noticed that the issue was fixed upstream (as well as the Django 1.9 support that was blocking me from using the official release):

#28

However, when I tested the new version, I found that the official fix converted Django's forward slashes to backslashes on Windows, instead of using forward slashes internally on Windows too like Django does, and like a db-based storage plugin should, so I'm stuck on my unofficial version.

The abstract issue is that there is no relation between a storage plugin and the operating system, so why should the storage plugin introduce a portability issue between operating systems, or depend on the operating system for it's API/ABI in any way?

An example concrete issue is that I can't clone the production DB and work on it from a Windows machine. Few people ever host production Django systems on a Windows server [citation needed], so dev environments for systems that are hosted on Unix in production is the major use case for using Django on Windows, and therefore this matters.

I propose moving to my fix and creating a migration that converts all existing paths in the DB to forward slashes (perhaps only do it if there are slashes in the config part of the stored paths... Windows does not allow backslashes in a filename, does Unix?).

I can write the code if you like the idea and are willing to do Code Review.

Aur

After uploading file with russian letters in it's name (like 'персональный-рецепт-как-встретить-2016-год.jpg') using django admin:

• this file is stored in DB absolutely fine;
• it is shown using db_file_storage.get_file without any problems;
• DBAdminClearableFileInput looks like it is empty - no filename link, no clear checkbox.

Django 1.8.7, python 2.7.6 or 2.7.11, django-db-file-storage==0.3.3