incuna / django-pgcrypto-fields Goto Github PK

View Code? Open in Web Editor NEW

229.0 9.0 51.0 252 KB

Transparent field level encryption for Django using the pgcrypto postgresql extension.

License: BSD 2-Clause "Simplified" License

Makefile 2.61% Python 97.39%

django psql pgcrypto

django-pgcrypto-fields's People

Stargazers

Watchers

Forkers

pombredanne coldmind itagenten jackson0810 mighty-justice youngsoo77 protoprojects jcasner cruzoscarjn fatboystring kevinetienne peterfarrell meshy u039b thecapslock sorokins giovan asan1148 maestrofjp aceofall dbinoj backinapp leukeleu ar4z barseghyanartur ridwanray maya-pbc dcecco khattori shepthedog63 senseware jmp bartsolutions naseemmdali december1981 lipton2777 bielikovv nithinkumar4 frojd riskconcile xet7 rogervaas gmh5225 matt-osborn joojis journee jdanielnd

django-pgcrypto-fields's Issues

Update outdated requirements.txt

cast_type of DateTimePGPSymmetricKeyField does not allow for dumpdata exports exporting timezone aware datetimes.

Extremely easy to override with a custom class when using Postgres as your database

class CustomDateTimePGPSymmetricKeyField(fields.DateTimePGPSymmetricKeyField):
    """
    Overriding cast_type so `dumpdata` command exports timezone aware datetimes.
    """
    cast_type = 'TIMESTAMPTZ'

but maybe this could be a setting? Or it could look at current USE_TZ setting?

Error when migrating existing fields into pgcrypto fields

Hey,

I am upgrading my Django application from 2.2.9 to Django 3.x and ran into errors on migrations that are converting normal fields (like CharField) into pgcrypto fields (CharPGPSymmetricKeyField in this case).

I'm not sure if this is a bug in Django, psycopg2 or django-pgcrypto-fields. I'm hoping you could assist me.
These migrations will work when running Django 2.2.9 but when I update Django package, they won't work any more.

I've managed to reproduce this in a fresh Django app.

I'm using Ubuntu 19.10, running Python 3.8.5 in a Docker container with the following packages:
django==3.0 (or any 3.x)
django-pgcrypto-fields==2.5.1
psycopg2==2.8.5

PostgreSQL is 11.1 and pgcrypto extension is installed.

Steps to reproduce:

Create a model with normal fields:

class CryptoTest(models.Model):
    name = fields.CharField(blank=True, null=True, max_length=255)

Create migration: python manage.py makemigrations
Modify model:

class CryptoTest(models.Model):
    name = fields.CharPGPSymmetricKeyField(blank=True, null=True, max_length=255)

Create migration: python manage.py makemigrations
Run migrations: python manage.py migrate

Traceback

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.DatatypeMismatch: column "name" cannot be cast automatically to type bytea
HINT:  You might need to specify "USING name::bytea".


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./manage.py", line 21, in <module>
    main()
  File "./manage.py", line 17, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/test.py", line 23, in run_from_argv
    super().run_from_argv(argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 328, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 369, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/test.py", line 53, in handle
    failures = test_runner.run_tests(test_labels)
  File "/usr/local/lib/python3.8/site-packages/django/test/runner.py", line 684, in run_tests
    old_config = self.setup_databases(aliases=databases)
  File "/usr/local/lib/python3.8/site-packages/django/test/runner.py", line 604, in setup_databases
    return _setup_databases(
  File "/usr/local/lib/python3.8/site-packages/django/test/utils.py", line 169, in setup_databases
    connection.creation.create_test_db(
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/creation.py", line 67, in create_test_db
    call_command(
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 168, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 369, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 231, in handle
    post_migrate_state = executor.migrate(
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/operations/fields.py", line 249, in database_forwards
    schema_editor.alter_field(from_model, from_field, to_field)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 564, in alter_field
    self._alter_field(model, old_field, new_field, old_type, new_type,
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/postgresql/schema.py", line 147, in _alter_field
    super()._alter_field(
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 710, in _alter_field
    self.execute(
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 142, in execute
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column "name" cannot be cast automatically to type bytea
HINT:  You might need to specify "USING name::bytea".

If I was to create my model using pgcrypto fields from the start, it works - but the model in my application was converted later on in the development and now these migrations are in the production aswell.

If you feel like I haven't given enough details, feel free to ask and I will try to explain more.

Thanks in advance for any help.

Adding default value to IntegerField in subsequent migrations throws an error

The issue is described in this SO topic. Looks like it has existed for over a year. I ran into it today, and seting null=True doesn't actually help. Any ideas if it has ever been resolved?

Support CharField for Django parity

Add support CharField for Django party. This will also support enforcing max_length of pre-encrypted form fields and also supports the right form field (instead of wrongly TextareaInput).

Side effect add support for max_length for EmailField (which is optional and set to 254 by default) and supports the right form field (instead of wrongly TextareaInput).

Update README

Update README to reflect latest changes.

Add __range lookup for Date / DateTime fields

Currently, the date / datetime fields do not support the __range lookup.

TODO:

Implement enhancement
Write tests
Update README
Update CHANGELOG

Migrate error after add new field

My model (migrate okay in the first time)

class Dkm(models.Model):
    name = fields.TextPGPSymmetricKeyField()
    value = fields.IntegerPGPSymmetricKeyField(default=0)

I update model and migrate again:

class Dkm(models.Model):
    name = fields.TextPGPSymmetricKeyField()
    value = fields.IntegerPGPSymmetricKeyField(default=0)
    value3 = fields.IntegerPGPSymmetricKeyField(default=0)

Error occur:

Traceback (most recent call last):
  File "manage.py", line 21, in <module>
    main()
  File "manage.py", line 17, in main
    execute_from_command_line(sys.argv)
  File "C:\Users\vu.tran\Desktop\kona-server\env\lib\site-packages\django\core\management\__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "C:\Users\vu.tran\Desktop\kona-server\env\lib\site-packages\django\core\management\__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "C:\Users\vu.tran\Desktop\kona-server\env\lib\site-packages\django\core\management\base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "C:\Users\vu.tran\Desktop\kona-server\env\lib\site-packages\django\core\management\base.py", line 364, in execute
    cursor.execute(sql, params)
  File "C:\Users\vu.tran\Desktop\kona-server\env\lib\site-packages\django\db\backends\utils.py", line 99, in execute
    return super().execute(sql, params)
  File "C:\Users\vu.tran\Desktop\kona-server\env\lib\site-packages\django\db\backends\utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "C:\Users\vu.tran\Desktop\kona-server\env\lib\site-packages\django\db\backends\utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "C:\Users\vu.tran\Desktop\kona-server\env\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "C:\Users\vu.tran\Desktop\kona-server\env\lib\site-packages\django\db\utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "C:\Users\vu.tran\Desktop\kona-server\env\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column "value3" is of type bytea but default expression is of type integer
HINT:  You will need to rewrite or cast the expression.

The error will not occur if set null=True
Have you got any idea?

Custom methods to `get_or_create`, `update_or_create`

At the moment using django get_or_create or update_or_create doesn't work as some values might be encrypted/hashed. We can have a mixin which would override those methods from django.

How to retrieve raw encrypted value?

I would like to show the encrypted values for Django admin users that are not super users, and only show the decrypted values to super users, but I'm not sure how to go about this. Do you have any guidance?

I was thinking I could extend admin.ModelAdmin class somehow or maybe create custom admin Forms and handle it that way, but I'm not sure how to access the encrypted value at this point when the model fields are already decrypted.

Add mixin for get_by_natural_key

The method get_by_natural_key can check if the User.USERNAME_FIELDS is a hash_of the value provided for login.

Example:

class MyMixin:
    def get_by_natural_key(self, username):
        """Get user by comparing hashed email.

        `get_by_natural_key` is used to `authenticate` a user, see:
        https://github.com/django/django/blob/c5780adeecfbd85a80b5aa7130dd86e78b23e497/django/contrib/auth/backends.py#L16
        """
        return self.get(username_keyed__hash_of=username)

Add support for JSONField

We have some data in JSON-field (django.contrib.postgres.fields.JSONField), but this package does not support it. Could it be possible to add this functionality?

django.db.utils.ProgrammingError: function pgp_pub_decrypt(character varying, bytea) does not exist

aws rds postgresql is not exist function pgp_pub_decrypt(character varying, bytea)

TypeError: %i format: a number is required, not str

I've used django-pgcrypto-fields for about a week now without a problem and then I created a migration to add an encrypted field:

# Generated by Django 2.2.19 on 2021-03-13 13:45

from django.db import migrations

def copy_field(apps, schema_editor):
    GenericEvent = apps.get_model('generic_event', 'GenericEvent')
    for ep in GenericEvent.objects.all():
        ep.pp_receiver_encrypted = ep.pp_receiver
        ep.save()


class Migration(migrations.Migration):

    dependencies = [
        ('generic_event', '0015_auto_20210313_0745'),
    ]

    operations = [
        migrations.RunPython(copy_field)
    ]

I've done a similar one with hashed fields before and it worked, but this one fails with:

Running migrations:
  Applying generic_event.0016_encrypt_payment_fields...Traceback (most recent call last):
  File "manage.py", line 24, in <module>
    execute_from_command_line(sys.argv)
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/core/management/commands/migrate.py", line 232, in handle
    post_migrate_state = executor.migrate(
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/var/www/project/projectproject/generic_event/migrations/0016_encrypt_payment_fields.py", line 7, in copy_field
    for ep in GenericEvent.objects.all():
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/db/models/query.py", line 274, in __iter__
    self._fetch_all()
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/cacheops/query.py", line 303, in _fetch_all
    return self._no_monkey._fetch_all(self)
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/db/models/query.py", line 55, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/db/models/sql/compiler.py", line 1129, in execute_sql
    sql, params = self.as_sql()
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/db/models/sql/compiler.py", line 474, in as_sql
    extra_select, order_by, group_by = self.pre_sql_setup()
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/db/models/sql/compiler.py", line 54, in pre_sql_setup
    self.setup_query()
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/db/models/sql/compiler.py", line 45, in setup_query
    self.select, self.klass_info, self.annotation_col_map = self.get_select()
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/db/models/sql/compiler.py", line 254, in get_select
    sql, params = self.compile(col, select_format=True)
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/django/db/models/sql/compiler.py", line 405, in compile
    sql, params = node.as_sql(self, self.connection)
  File "/var/www/envs/myenv/lib/pyston3.8/site-packages/pgcrypto/mixins.py", line 33, in as_sql
    sql = self.target.get_decrypt_sql(connection) % (sql, self.target.get_cast_sql())
TypeError: %i format: a number is required, not str

Local variables from Sentry:


__class__ | <class 'pgcrypto.mixins.DecryptedCol'>
compiler | <django.db.models.sql.compiler.SQLCompiler object at 0x7fbc3a838a30>
connection | <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7fbc3a827550>
params | []
self | DecryptedCol(organizations_organization, organizations.Organization.pp_receiver_encrypted)
sql | '"organizations_organization"."pp_receiver_encrypted"'

Model field looks like this:
pp_receiver_encrypted = EmailPGPSymmetricKeyField(_('This is account (email)'), blank=True, null=True)

I have no idea how to debug or fix it. It worked fine on my local env. I'm using PostgreSQL 11.11 (through pgbouncer) on Ubuntu 20 server.
I've set PGCRYPTO_KEY in settings. Any help will be greatly appreciated!

Use app config to define settings

With a specific app config it would be easier to override where to find the PGP keys and/or the secret keys.

Custom manager for Decrypt annotate

Add a custom manager to include the Decrypt annotate in get_queryset.

Add support to hash fields to optionally base hash value off another field value

We wanted to run a new feature by you all to see if it's worth for us to do a PR.

We want to add support to the hash fields that optionally bases the value for the hash off the value of another field. This would be useful to pre-compute a hash based on the value from another model field and not have to worry about updating the hash when the other field value changes. Essentially, it ensures the hash is always based on the value of a sibling field.

Example:

class User(models.Models):
    first_name = fields.TextPGPSymmetricKeyField(max_length=20, verbose_name='First Name')
    first_name_hashed = fields. TextHMACField(original='first_name')

In the above example, if you specify the optional original attribute -- it would take the unencrypted value from the first_name model field as the input value to create the hash. If you didn't specify an original attribute, the field would work as it does now and would remain backwards compatible.

Let us know if you would be willing to accept a PR for this feature.

Update requirements to allow them to be picked up by requires.io

At the moment the requirements are always up-to-date as they are not picked up by requires.io.

From http://requires.io

Requirements Files

Requires.io looks for the following requirements files in the project:
   setup.py,
   all files matching req*.txt or req*.pip,
   all files matching requirements/*.txt or requirements/*.pip,
   buildout.cfg and versions.cfg,
   tox.ini.

Add caching

When accessing the field value attribute on the model we are doing a database query. We need a way to cache the value. see #4

Question - How to use a different hashing algorithm?

The default pgcrypto extension is PGP (public key/privacy key) output looks like:

\xc30d0407030286e239c36fe48d126dd24401086aecb7805d0e52fc77c32bc341f566153ee9e291d11f8972a6fd63a44282b5a7df09fad63d0bbd1eb1374980a9e10e13d2987a1289e193a749f496aec0195e1eb5e3

Not sure if this is the most secure by default. If improved algorithms are found in the future, how would you configure them?

If not using PGP and instead a one way hash, why not use other options like bcrypt are argon2, pbkdf2, scrypt, etc. instead of SHA.

Perhaps an improvement to django-pgcrypto-fields would be to configure what types of algorithms you would want to use? Or is there already a way to configure this with postgres or something?

Drop support for Django 1.8, 1.9, and 1.10

According to the Django project, Django 1.8, 1.9, and 1.10 ended extended support have all passed as of April 1, 2018 and are no longer supported.

https://www.djangoproject.com/download/

The last version of this library to support 1.8, 1.9, and 1.10 is the v2.2.0 release.

Add Support For Django 3

Explain the difference between kinds of fields

Currently, the readme says that

django-pgcrypto-fields has 3 kinds of fields:

hash based fields

PGP fields

Symmetric fields"

It would be nice if there was a quick explanation of what these things are, how they differ, and when one might want to use them.

Python 2.7 compatibility?

It appears this is Python 3? Is this supposed to be compatible with old Python 2.7 code? I'm getting the:

TypeError: super() takes at least 1 argument (0 given)

... after installing and running the package. (Django 1.9.8)

Distinct().order_by('field') results in ProgrammingError

Platform: macOS Catalina
Python: 3.6.9
Django: 2.2.11

Given a model with encrypted fields, doing;

EncryptedModel.objects.distinct().order_by('encrypted_field')

Results in

django.db.utils.ProgrammingError: syntax error at or near "Version"
LINE 59: Version: GnuPG v1

The syntax error refers to some (seemingly random) part of the armored key in the SQL.

Interestingly, this does not occur when you distinct on the encrypted_field:

EncryptedModel.objects.distinct('encrypted_field').order_by('encrypted_field')

Works fine. So does

EncryptedModel.objects.distinct().

I've created a PR #203 with a breaking test, but I don't know where to start fixing this.

Remove support for python3.5

Python 3.5 end-of-life is next month I suggest to drop the support. It would allow to update twine (see pyup.io build status #260) to the last version too as the library deprecated support for it last year.

Merge .coveragerc into setup.cfg

See for example django-user-management on how to include coverage in setup.cfg.

Django 1.10 deprecation notices

Before going into production with this, wanted to make sure there will be an upgrade path to Django 1.10 (which is already released). Accessing an encrypted field value in Django 1.9 throws several warnings.

.../virtualenvs/portal/lib/python3.5/site-packages/pgcrypto/aggregates.py:66: RemovedInDjango110Warning: The aggregates property is deprecated. Use annotations instead.
  query.aggregates[alias] = aggregate

09/07/2016 17:02:42 WARNING py.warnings:221 .../virtualenvs/portal/lib/python3.5/site-packages/pgcrypto/aggregates.py:66: RemovedInDjango110Warning: The aggregates property is deprecated. Use annotations instead.
  query.aggregates[alias] = aggregate

.../virtualenvs/portal/lib/python3.5/site-packages/django/db/models/aggregates.py:64: RemovedInDjango110Warning: The aggregates property is deprecated. Use annotations instead.
  sql_aggregate = query.aggregates.pop(placeholder_alias)

09/07/2016 17:02:42 WARNING py.warnings:221 .../virtualenvs/portal/lib/python3.5/site-packages/django/db/models/aggregates.py:64: RemovedInDjango110Warning: The aggregates property is deprecated. Use annotations instead.
  sql_aggregate = query.aggregates.pop(placeholder_alias)

Add support for password protected public key encryption private keys

The setting PGCRYPTO_KEY is used in PGPSymmetricKeyFieldMixin (

django-pgcrypto-fields/pgcrypto/mixins.py

Line 143 in 429b9a8

class PGPSymmetricKeyFieldMixin(PGPMixin):

)

but not PGPPublicKeyFieldMixin

django-pgcrypto-fields/pgcrypto/mixins.py

Line 128 in 429b9a8

class PGPPublicKeyFieldMixin(PGPMixin):

I was unable to use a public/private key with a passphrase with a TextPGPPublicKeyField. I regenerated a public/private key without a passphrase it worked fine.

Does PGPPublicKeyFieldMixin need to be updated to support PGCRYPTO_KEY?

Thanks

[Need help] when using update with F() getting pgp_sym_encrypt(numeric, unknown) does not exist

I am using django-pgcrypto-fields symmetric encryption in DB.
I have following update query

MyTable.objects.filter(some_code=some_code).update(
                    some_value=(some_price * F('some_units')),
                    updated_on=datetime.now()
                )

I am getting the following issue psycopg2.errors.UndefinedFunction: function pgp_sym_encrypt(numeric, unknown) does not exist

I am not sure if this is the issue with the library or something with my code.

ExternalRoutineInvocationException: Need password for secret key

I made all the configurations described in the README to use PGPPublicKeyField.
I created my public and private key, however when I make a query this error is thrown, ExternalRoutineInvocationException: Need password for secret key

Am I missing something?

DateTimePGPSymmetricKeyField does not save with timezone info, is this a defect?

imagine some field:

expires_at = fields.DateTimePGPSymmetricKeyField(null=True)

set equal to:

from datetime import timedelta
from django.utils.timezone import now
...
expires_at = now() + timedelta(seconds=token['expires_in'])

I noticed there was an issue when I got the error:

can't compare offset-naive and offset-aware datetimes

from this call on my model:

@property
def is_expired(self):
    return self.expires_at < now()

Inspecting the unencrypted datetime next to a never encrypted datetime -- you can see the never encrypted datetime still has timezone information:

I can remedy this by refacroting my is_expired method to:

import pytz
...
@property
def is_expired(self):
    return self.expires_at.replace(tzinfo=pytz.UTC) < now()

but I was curious if this was a defect.

Thanks for your time! :)

Pass options to pgp_sym_encrypt

pgp_sym_encrypt function supports certain options. For example by default it uses AES128 algorithm to encrypt but you can customize pgp_sym_encrypt(data, psw, 'compress-algo=1, cipher-algo=aes256'). Is there a way I can provide the cipher-algo option with value AES256 when I create CharPGPSymmetricKeyField or maybe in the settings file? If there is not then could there please be an enhancement that can be made for this?

Unique constraints

Is it possible to have unique constraints at the database level? For example, Meta.unique_together (on model) doesn't seem to have any effect.

Add support for BigIntegerField

Add support for BigIntegerField:

https://docs.djangoproject.com/en/2.2/ref/models/fields/#bigintegerfield

Dynamic PGCRYPTO_KEY

Is there any option to use a key in a dynamic manner? For example, if there's a need to encrypt/decrypt data with different keys depending on some factors, is this possible using this package?

Support multiple databases connections with different crypto keys

Add support for multiple databases that have different crypto keys. Just move PUBLIC_PGP_KEY, PRIVATE_PGP_KEY, and PGCRYPTO_KEY into the DATABASES configuration and grab that information from the connection instead of reading from the settings:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'PUBLIC_PGP_KEY': open(PUBLIC_PGP_KEY_PATH, 'r').read(),
        'PRIVATE_PGP_KEY': open(PRIVATE_PGP_KEY_PATH, 'r').read(),
        'PGCRYPTO_KEY': 'super_secret_key',
        ...
    },
}

DecryptedCol

    def as_sql(self, compiler, connection):
        sql, params = super(DecryptedCol, self).as_sql(compiler, connection)
        decrypt_sql = self.decrypt_sql_template.format(
            key=self.target._get_encryption_key(connection),
            dbtype=self.target._get_base_db_type(connection),
            sql=sql
        )
        return decrypt_sql, params

This is easily made backwards compatible:

   def _get_encryption_key(self, connection):
        if 'PGCRYPTO_KEY' in connection.settings_dict:
            key = connection.settings_dict['PGCRYPTO_KEY']
        else:
            key = getattr(settings, 'PGCRYPTO_KEY', settings.SECRET_KEY)
        # Escape any percent symbols in the key, to avoid them being
        # interpreted as extra substitution placeholders later on.
        key = key.replace('%', '%%')
        return key

Remove support for Django 1.11

Django 1.11 has reached the end of extended support on April 1, 2020.

https://www.djangoproject.com/weblog/2020/apr/01/bugfix-releases/

Django 1.11 (2.0) compatibility

Hello everyone.
Is this project compatible to Django 1.11 and the upcoming Django 2.0?

Also: Where are the keys necessary for decryption stored? What kind of access would an attacker need to decrypt encrypted data in the database?

EmailPGPPublicKeyField does not use the correct mixin

As defined in https://github.com/incuna/django-pgcrypto-fields/blob/master/pgcrypto/fields.py#L41 EmailPGPPublicKeyField uses the PGPSymmetricKeyFieldMixin mixin instead of the PGPPublicKeyFieldMixin one.

Add support for additional fields

Add support for additional fields to have more parity with the built-in fields in Django:

TODO:

Support of type BOOLEAN field

Is there any plan on supporting boolean fields within Django for this package? I appreciate we can use Integer fields as an alternative approach, but before we venture to make change, thought it was worth a ticket.

Release 2.1.1

I checked in a PR for Django 2.0 support. I upgraded the requirements and travis to get the build to run successfully again. I have tagged this as 2.1.1.

Please release what is in master to PyPi as release 2.1.1 at your earliest convenience or add my PyPi user peterfarrell-atdsaa to be able to push releases to the cheese shop.

@kevinetienne @maxpeterson @meshy @cyanc @Ian-Foote

Fields are not being decrypted neither with TextDigestField or TextHMACField

Field was properly encrypted when I create instance, however no decryption is done in any of selects:

from pgcrypto import fields

class Test(models.Model):
    key = fields.TextDigestField()

t = Test(key='asdf')
t.save()

INSERT INTO "app_test" ("key")
VALUES (Digest('*****', 'sha512'), true) RETURNING "app_test"."id";

And for any selects I get raw data and selects contain no decryption functions:

for each in Test.objects.all():
    print(each, each.key)
# >>>> object (1) \xd0698d.....

SQL:

SELECT "app_test"."id",
       "app_test"."key"
FROM "app_test";

All I added to settings is:

PGCRYPTO_KEY = '*****'
# and
'pgcrypto', # To INSTALLED_APPS

Versions:
Python 3.6.4
Django 2.1.4 (Also tried Django 2.1.0)
django-pgcrypto-fields==2.5.1

" filter() " is not working.

I was testing to use the 'filter()'.

While I was working on a simple example, I could not get any values like this:

What did I do wrong? Or is there a manual error?

Please Help me...

youngsoo

django==1.10
python==3.4.5

>>> MyModel.objects.filter().first().pgp_pub_field
'hello, world!\r\n안녕하세요.'
>>> from pgcrypto.aggregates import PGPPublicKeyAggregate
>>> my_models = MyModel.objects.annotate(PGPPublicKeyAggregate('pgp_pub_field'))
>>> my_models
<QuerySet [<MyModel: #1>, <MyModel: #2>]>
>>> my_models.filter(pgp_pub_field='h')
<QuerySet []>
>>> my_models.filter(pgp_pub_field__decrypted='h')
<QuerySet []>
>>> my_models.filter(pgp_pub_field__decrypted='haa')
<QuerySet []>
>>> my_models.filter(pgp_pub_field__decrypted='1111')
<QuerySet []>

Implement DatePGPPublicKeyField and DateTimePGPPublicKeyField

Implement DatePGPPublicKeyField and DateTimePGPPublicKeyField to match parity with symmetric key fields.

Todo:

Add DatePGPPublicKeyField and DateTimePGPPublicKeyField
Write tests
Update README documentation
Update CHANGELOG

syntax error at or near "j1" LINE 1: ...SET value = pgp_sym_encrypt('hogehoge', 'hogehoge'j1') WHERE...

It seems that proper escaping is needed when constructing SQL statements.