dcwatson / django-pgcrypto Goto Github PK

View Code? Open in Web Editor NEW

64.0 64.0 22.0 92 KB

Python and Django utilities for encrypted fields using pgcrypto.

License: BSD 2-Clause "Simplified" License

Python 100.00%

django-pgcrypto's People

Contributors

Stargazers

Watchers

django-pgcrypto's Issues

Remove django-pgcrypto version from PGP MESSAGE

Is there a reason the django-pgcrypto version is stored in the PGP MESSAGE?

Here's the issue I ran into:

# This is how the where condition is coded during search after upgrading to 1.1.1
# Ex.: EncryptedModel.objects.get(encrypted_field='data')
u'-----BEGIN PGP MESSAGE-----\nVersion: django-pgcrypto 1.1.1\n\nXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=\n=lwED\n-----END PGP MESSAGE-----'

# And this is how the data is stored
# Ex.: EncryptedModel.objects.first().encrypted_field
u'-----BEGIN PGP MESSAGE-----\nVersion: django-pgcrypto 1.1.0\n\nXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=\n=lwED\n-----END PGP MESSAGE-----'

So all searches after upgrading fail until you re-save the data. Can the version safely be removed from the PGP MESSAGE to avoid this in the future?

AES issues on Python 3

I'm unable to create a migration using AES.

Blowfish seems to work:

date_hired = pgcrypto.EncryptedTextField(cipher='Blowfish', key='datekey')

But AES does'nt:

date_hired = pgcrypto.EncryptedTextField(cipher='AES', key='datekey')

Traceback

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 350, in execute_from_command_line
utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 342, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 348, in run_from_argv
self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 399, in execute
output = self.handle(*args, **options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/commands/makemigrations.py", line 106, in handle
ProjectState.from_apps(apps),
  File "/usr/local/lib/python3.5/site-packages/django/db/migrations/state.py", line 174, in from_apps
model_state = ModelState.from_model(model)
  File "/usr/local/lib/python3.5/site-packages/django/db/migrations/state.py", line 393, in from_model
fields.append((name, field.clone()))
  File "/usr/local/lib/python3.5/site-packages/django/db/models/fields/__init__.py", line 464, in clone
return self.__class__(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/pgcrypto/fields.py", line 24, in __init__
self.cipher_key = aes_pad_key(self.cipher_key.encode(self.charset))
AttributeError: 'bytes' object has no attribute 'encode'

Issue

django-pgcrypto is no longer working under Django 1.10.

SubfieldBase was deprecated since 1.8 and it's removed in 1.10.

Traceback

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 341, in execute
    django.setup()
  File "/usr/local/lib/python3.5/site-packages/django/__init__.py", line 27, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/local/lib/python3.5/site-packages/django/apps/registry.py", line 108, in populate
    app_config.import_models(all_models)
  File "/usr/local/lib/python3.5/site-packages/django/apps/config.py", line 199, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/local/lib/python3.5/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 665, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/app/pyupio/users/models.py", line 10, in <module>
    import pgcrypto
  File "/src/django-pgcrypto/pgcrypto/__init__.py", line 18, in <module>
    from .fields import *
  File "/src/django-pgcrypto/pgcrypto/fields.py", line 96, in <module>
    class EncryptedTextField (six.with_metaclass(models.SubfieldBase, BaseEncryptedField)):
AttributeError: module 'django.db.models' has no attribute 'SubfieldBase'

Potentially helpful links

Update for Django (and Python) 3, drop PyCrypto

Title pretty much says it all. Get things working for modern versions of Django and Python, and replace PyCrypto (which doesn't seem to be maintained anymore?) with cryptography.

Empty string in DB crashes the lib

Hi,

We're using an existing DB and we're migrating to an encrypted solution, so some of our DB data has empty strings - code crashes when calling Django bulk update method.

If you want to use SQL dearmor with an empty string it'll throw an error "Corrupt ascii-armor" - this means that there should be some handling in the library to prevent that.

A test which shows the problem:

    def test_empty_db_string_compare_with_non_blank(self):
        obj = Employee.objects.create(name="Test User 2", date_hired=datetime.date.today(), email="[email protected]")
        with transaction.atomic():
            with connection.cursor() as cursor:
                # Manually overwrite DB value to empty string
                cursor.execute("UPDATE %s SET ssn = '' WHERE id = %%s;" % (obj._meta.db_table,), [obj.pk])
        obj.refresh_from_db()
        self.assertEqual(obj.ssn, "")
        self.assertNotEqual(obj.ssn, "NON_EMPTY_STRING")
        # Try performing a bulk update in Django - in SQL dearmor("") throws an error "Corrupt ascii-armor"
        Employee.objects.filter(**{"pk": obj.pk}).exclude(**{"ssn": "XYZ"}).update(**{"ssn": "XYZ"})
        obj.delete()

A possible fix:

        if isinstance(self.lhs.field, EncryptedCharField) or isinstance(self.lhs.field, EncryptedTextField):
            # Special case when value in DB is an empty string - use NULL - otherwise dearmor explodes
            return (
                "COALESCE(convert_from(decrypt(dearmor(NULLIF(%s, '')), %%s, '%s'), 'utf-8'),%s)%s %s"
                % (lhs, self.lhs.output_field.cipher_name, lhs, self.lhs.output_field.field_cast, rhs),
                params,
            )
        else:
            return (
                "convert_from(decrypt(dearmor(%s), %%s, '%s'), 'utf-8')%s %s"
                % (lhs, self.lhs.output_field.cipher_name, self.lhs.output_field.field_cast, rhs),
                params,
            )

i.e. a check for column type and then two SQL functions called COALESCE and NULLIF.

I have a fork, but before I post anything I'd like to get some feedback. :)

Thanks!

Add support for UTF-8 in EncryptedTextField

The current code only accepts ASCII support, but should be updated to handle UTF-8.

I hacked in a fix for my current purposes in samdolan@fb51657

but there needs to be some more unittesting before I push it back, which I don't have time for at the moment. The code does appear to work correctly in my ad-hoc testing.

what does the parameter versioned=True changes?

I have seen in the examples inside testapp/models.py that ssn = pgcrypto.EncryptedCharField("SSN", versioned=True, blank=True) is a encrypted CharField but I can't figure out what does the versioned parameter affects.

Lookup on EncryptedCharField in shell fails on postgresql

Django: 1.8
Database: PostgreSQL 9.3.3

Doing a lookup on a pgcrypto.EncryptedCharField in the shell throws the error AttributeError: 'Col' object has no attribute 'source'

Stacktrace:

>>> Person.objects.filter(last_name='test')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/ubuntu/www/app/venv/local/lib/python2.7/site-packages/django/db/models/query.py", line 138, in __repr__
    data = list(self[:REPR_OUTPUT_SIZE + 1])
  File "/home/ubuntu/www/app/venv/local/lib/python2.7/site-packages/django/db/models/query.py", line 162, in __iter__
    self._fetch_all()
  File "/home/ubuntu/www/app/venv/local/lib/python2.7/site-packages/django/db/models/query.py", line 965, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/home/ubuntu/www/app/venv/local/lib/python2.7/site-packages/django/db/models/query.py", line 238, in iterator
    results = compiler.execute_sql()
  File "/home/ubuntu/www/app/venv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 829, in execute_sql
    sql, params = self.as_sql()
  File "/home/ubuntu/www/app/venv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 387, in as_sql
    where, w_params = self.compile(self.query.where)
  File "/home/ubuntu/www/app/venv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 357, in compile
    sql, params = node.as_sql(self, self.connection)
  File "/home/ubuntu/www/app/venv/local/lib/python2.7/site-packages/django/db/models/sql/where.py", line 104, in as_sql
    sql, params = compiler.compile(child)
  File "/home/ubuntu/www/app/venv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 355, in compile
    sql, params = vendor_impl(self, self.connection)
  File "/home/ubuntu/www/app/venv/local/lib/python2.7/site-packages/pgcrypto/fields.py", line 223, in as_postgresql
    params = lhs_params + [self.lhs.source.cipher_key] + rhs_params
AttributeError: 'Col' object has no attribute 'source'

ValueError: level must be >= 0

I'm new to pgcrypto so forgive me if this isn't an issue per se (it's entirely possible and highly likely I'm missing a piece of the configuration puzzle), but I didn't see another appropriate place to discuss what I'm running into.

I'm using python 3.4 and postgres 9.3.5 on Ubuntu and on the postgres side anyway everything seems to be functioning, meaning CREATE EXTENSION worked and I can verify version 1.0 is installed, and manual queries in psql doing some crypt stuff work fine.

When I try to do anything from Django I get the following error:
File "/vagrant/foo/security/models.py", line 9, in Bar
bar = pgcrypto.EncryptedTextField()
File "/usr/local/lib/python3.4/dist-packages/pgcrypto.py", line 168, in init
mod = import('Crypto.Cipher', globals(), locals(), [cipher_name], -1)
ValueError: level must be >= 0

I've tried both using models as well as just going into a shell and creating instances of the encrypted field types and always get that same last two lines in the error message.

Any guidance would be greatly appreciated. Thanks.

Django 1.7, having trouble filtering on encrypted fields


# the model
class PhoneCall(TimestampMixin, models.Model):
    from_number = pgcrypto.EncryptedCharField(max_length=100)

# the query
numbers = PhoneCall.objects.filter(from_number__exact=somenumber)
if numbers:
    # ... do stuff

And the error:

  File "/vagrant/apps/crm/utils.py", line 29, in find_company_for_number
    if numbers:
  File "/home/vagrant/.virtualenvs/eventboard/local/lib/python2.7/site-packages/django/db/models/query.py", line 145, in __nonzero__
    self._fetch_all()
  File "/home/vagrant/.virtualenvs/eventboard/local/lib/python2.7/site-packages/django/db/models/query.py", line 966, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/home/vagrant/.virtualenvs/eventboard/local/lib/python2.7/site-packages/django/db/models/query.py", line 265, in iterator
    for row in compiler.results_iter():
  File "/home/vagrant/.virtualenvs/eventboard/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 700, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/home/vagrant/.virtualenvs/eventboard/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
    cursor.execute(sql, params)
  File "/home/vagrant/.virtualenvs/eventboard/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 81, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/home/vagrant/.virtualenvs/eventboard/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/home/vagrant/.virtualenvs/eventboard/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/vagrant/.virtualenvs/eventboard/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
django.db.utils.InternalError: Corrupt ascii-armor

Deconstruct includes the key - this results in a new migration when the key is changed

When you use different keys for development and production environments, running migrate in the production environment fails since Django migration identify a change in the schema that is not represented in the migrations.

Django 1.7 makemigrations fails: AttributeError: 'bytes' object has no attribute 'encode'

On Django 1.7 with Python 3 (Ubuntu Server 14.04) I'm getting an error when I run makemigrations. I ran the django-pgcrypto tests and they run fine, however.

Model:
import pgcrypto
from django.db import models

class Test(models.Model):
foo = pgcrypto.EncryptedTextField()

When I try to run makemigrations I get the following error:
Traceback (most recent call last):
File "manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.4/dist-packages/django/core/management/init.p
y", line 385, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.4/dist-packages/django/core/management/init.p
y", line 377, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.4/dist-packages/django/core/management/base.py",
line 288, in run_from_argv
self.execute(_args, *_options.dict)
File "/usr/local/lib/python3.4/dist-packages/django/core/management/base.py",
line 338, in execute
output = self.handle(_args, *_options)
File "/usr/local/lib/python3.4/dist-packages/django/core/management/commands/m
akemigrations.py", line 90, in handle
ProjectState.from_apps(apps),
File "/usr/local/lib/python3.4/dist-packages/django/db/migrations/state.py", l
ine 108, in from_apps
model_state = ModelState.from_model(model)
File "/usr/local/lib/python3.4/dist-packages/django/db/migrations/state.py", l
ine 180, in from_model
fields.append((name, field_class(_args, *_kwargs)))
File "/usr/local/lib/python3.4/dist-packages/pgcrypto/fields.py", line 24, in
init
self.cipher_key = aes_pad_key(self.cipher_key.encode(self.charset))
AttributeError: 'bytes' object has no attribute 'encode'

Release pgcrypto 1.3 on pypi?

only 1.2 available thorugh pip install

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbe in position 1: invalid start byte

I have an issue upgrading from the version 1.4.0 to 2.0.0
I can't access the values of the fields stored in the DB with the version of the 1.4.0.

Here is the model:

class Profile(models.Model):
    # ...
    email = pgcrypto.EncryptedEmailField(null=True, blank=True)
    # ...

Here is how I try to access field value:

Profile.objects.get(pk=1).email

Traceback:

Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.7.9/Frameworks/Python.framework/Versions/3.7/lib/python3.7/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/Users/sergey/ENV/proj/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/Users/sergey/ENV/proj/lib/python3.7/site-packages/django/db/models/query.py", line 402, in get
    num = len(clone)
  File "/Users/sergey/ENV/proj/lib/python3.7/site-packages/django/db/models/query.py", line 256, in __len__
    self._fetch_all()
  File "/Users/sergey/ENV/proj/lib/python3.7/site-packages/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/Users/sergey/ENV/proj/lib/python3.7/site-packages/django/db/models/query.py", line 72, in __iter__
    for row in compiler.results_iter(results):
  File "/Users/sergey/ENV/proj/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1084, in apply_converters
    value = converter(value, expression, connection)
  File "/Users/sergey/ENV/proj/lib/python3.7/site-packages/pgcrypto/fields.py", line 101, in from_db_value
    return self.to_python(value)
  File "/Users/sergey/ENV/proj/lib/python3.7/site-packages/pgcrypto/fields.py", line 97, in to_python
    return unpad(self.decrypt(dearmor(value, verify=self.check_armor)), self.block_size).decode(self.charset)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbe in position 1: invalid start byte

I have no issue accessing fields values saved with the django-pgcrypto==2.0.0.

Here is the version of some related packages installed in that environment:

Python 3.7.9

Django                2.2.12
django-pgcrypto       2.0.0
psycopg2-binary       2.7.5

Blowfish has been deprecated.

In log:

return {"aes": algorithms.AES, "bf": algorithms.Blowfish}[self.cipher_name]
app_1    | /usr/local/lib/python3.10/site-packages/pgcrypto/fields.py:58: CryptographyDeprecationWarning: Blowfish has been deprecated

__in filtering broken?

Hey there!

Thanks so much for releasing/maintaining this package, so far it's worked well for us. We're running into a tiny problem at the moment, it seems like we can't filter based on __in=[...] but we can filter with a normal lookup.

This works, account_number is encrypted:

Account.objects.filter(account_number=account_number).exists()

This does not:

Account.objects.filter(account_number__in=[account_number]).exists()

Is this intentional? I can't see any other issues reporting this, maybe we're not supposed to be using array filters with this package?

Appreciate any help, Merry Christmas!
Eric

unique=True and null=True

Background: Django ignores unique constraint check if value is None but if value is "" (empty string) the null constraint check will raise problems

BaseEncryptedField:
In to_python() and get_prep_value(): if the value is None it will try to insert "" into the db and the unique constraint will fail. Is there a reason for saving "" instead of just the value (None in this case)?

fields.py:67-88

    def to_python(self, value):
        if self.is_encrypted(value):
            # If we have an encrypted (armored, really) value, do the following when accessing it as a python value:
            #    1. De-armor the value to get an encrypted bytestring.
            #    2. Decrypt the bytestring using the specified cipher.
            #    3. Unpad the bytestring using the cipher's block size.
            #    4. Decode the bytestring to a unicode string using the specified charset.
            return unpad(self.get_cipher().decrypt(dearmor(value, verify=self.check_armor)),
                         self.cipher_class.block_size).decode(self.charset)
        return value or '' <-- Why not just return the supplied value?

    def get_db_prep_save(self, value, connection):
        if value and not self.is_encrypted(value):
            # If we have a value and it's not encrypted, do the following before storing in the database:
            #    1. Convert it to a unicode string (by calling unicode).
            #    2. Encode the unicode string according to the specified charset.
            #    3. Pad the bytestring for encryption, using the cipher's block size.
            #    4. Encrypt the padded bytestring using the specified cipher.
            #    5. Armor the encrypted bytestring for storage in the text field.
            return armor(self.get_cipher().encrypt(pad(six.text_type(value).encode(self.charset),
                                                       self.cipher_class.block_size)))
        return value or '' <-- Why not just return the supplied value?

Different cipher key depending of request

Hello. I run into some problem using this library. I need that some models were encrypted with user password, so I need to store cipher key in session and pass it to models on each request. I forked this repo and made some changes (atomAltera@db588d2 ) . Now I can call Model.field.cipher_key = private_user_key to provide necessary key, but now I have multithreading conflicts. I don't want to use locks for each request because it affects performance. Can you give me some advice, or make changes in your library so it could get request depending keys?

django.db.utils.InternalError: Corrupt ascii-armor Error On lookup field

After installing that I am getting django.db.utils.InternalError: Corrupt ascii-armor

while I installed migrations like

 from django.contrib.postgres.operations import CryptoExtension

from django.db import migrations

class Migration(migrations.Migration):

    dependencies = [
        ('users', '0003_auto_20201105_2158'),
    ]

    operations = [
        CryptoExtension(),
    ]

Empty Encrypted Fields Cause Errors

While django-pgcrypto seems to handle encrypted fields with NULL values without issue, if an encrypted field is simply blank rather than NULL, all queries break with a "Corrupt ascii-armor\n" error message, rather than properly detecting the empty field and skipping the decryption step.

error message: function dearmor(text) does not exist\nLINE 1: ..._on\" FROM \"users_user\" WHERE convert_from(decrypt(dearmor(\"u

after installation on django and inserting a email data I am getting this error on filter "error message: function dearmor(text) does not exist\nLINE 1: ..._on" FROM "users_user" WHERE convert_from(decrypt(dearmor("u"

my Model field is like

supportive_email = pgcrypto.EncryptedEmailField(cipher="bf", key="datekey")

dcwatson / django-pgcrypto Goto Github PK

django-pgcrypto's People

Contributors

Stargazers

Watchers

Forkers

django-pgcrypto's Issues

Traceback

Issue

Traceback

Potentially helpful links

Recommend Projects

Recommend Topics

Recommend Org