Hello gentlemen, I have a question regarding Django async applications.

I saw that in the cid/locals/ directory we have two options to set the cid, using context var and local threading.

By default the library uses local threading everywhere, but I think that in async applications we can have problems and the most correct thing would be to use context var. Correct me if I'm wrong.

Shouldn't we have a smarter way to identify whether a Django application is running sync or async and then choose when to use context var or local threading? Maybe using isinstance(request, ASGIRequest) in middleware or maybe using local from asgiref module.

Hey!

I think there is a typo here (get_gid should be get_cid):

It results in an ImportError and the fallback to thread_local is always used:

Python 3.9.13 (main, May 29 2022, 04:54:28)
...
>>> from cid.locals import get_cid
>>> get_cid.__module__
'cid.locals.thread_local'

Hi, I might have come across a bug, please help me to validate this issue.

Environment:

Python 3.6
django-cid==2.0
django==2.2.8

Apache with mod_wsgi

I followed installation and configuration docs step by step. Getting an error when trying to log with CID:

--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.6/logging/__init__.py", line 994, in emit
    msg = self.format(record)
  File "/usr/lib/python3.6/logging/__init__.py", line 840, in format
    return fmt.format(record)
  File "/usr/lib/python3.6/logging/__init__.py", line 580, in format
    s = self.formatMessage(record)
  File "/usr/lib/python3.6/logging/__init__.py", line 549, in formatMessage
    return self._style.format(record)
  File "/usr/lib/python3.6/logging/__init__.py", line 391, in format
    return self._fmt % record.__dict__
KeyError: 'cid'
Call stack:
  File "...mysite/lib/python3.6/site-packages/django/core/handlers/wsgi.py", line 141, in __call__
    response = self.get_response(request)
  File "...mysite/lib/python3.6/site-packages/django/core/handlers/base.py", line 75, in get_response
    response = self._middleware_chain(request)
  File "...mysite/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "...mysite/lib/python3.6/site-packages/cid/middleware.py", line 38, in __call__
    response = self.get_response(request)
  File "...mysite/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "...mysite/lib/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "...mysite/lib/python3.6/site-packages/django/utils/decorators.py", line 142, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "...mysite/myapis/some_apis.py", line 14, in do_something
    log.info(f'META!!!!: {request.META}.')

Somehow CID is not able to find the header which is set by Apache. To troubleshoot I print headers from a django view in 2 ways:

1. log.info(f'HEADERS!!!!: {request.headers}.') gives: 'X-Correlation-Id': 'WGZkYUYxZXRGc3h0NGtmRldJSlRzUUFBQ UFN'

2. log.info(f'META!!!!: {request.META}.') gives: 'HTTP_X_CORRELATION_ID': 'WGZkYUYxZXRGc3h0NGtmRldJSlRzUUFBQUFN'

So the header is clearly present in the request. I tried both settings:
CID_HEADER = 'X-Correlation-Id'
or
CID_HEADER = 'HTTP_X_CORRELATION_ID'
and got the same error.

The rest of the Django settings is the same as in instructions:

'formatters': {
        'default': {
            'format': '[cid: %(cid)s] - %(asctime)s - %(name)s - %(levelname)s - %(threadName)s - %(message)s',
        },
...
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'formatter': 'default',
        },
...
'filters': {
        'correlation': {
            '()': 'cid.log.CidContextFilter'
        },
    },
...
'mysite': {
            'level': LOGLEVEL,
            'handlers': ['console'],
            'filters': ['correlation'],
            'propagate': True,
        },

Thanks,
Alex

I was curious if anyone else has run into an issue with the filter, cid.log.CidContextFilter, not being loaded?

Hello,

Django 3.0 is now available.
The only two changes with this release that seems to affect django-cid are :

• dropping the django.db.backends.postgresql_psycopg2.base alias to django.db.backends.postgresql.base.
• Django 3.0 is only tested against python 3.6+

Also, Django release notes suggest to drop support for Django version prior to 2.2.

With all that in mind, I've made this PR (#51) that :

• stop testing with Django 2.1
• stop using the old alias
• Test with Django 3.0 and python 3.6-3.8

We don't need to change the requirement for Django>=1.11 django.db.backends.postgresql_psycopg2.base was already an alias to django.db.backends.postgresql.base in Django 1.11.

I would like to request feature to being able to override log record attribute using setting.

We use kibana and would like to see something more meaningful rather than cid.

Hello. I would like to make a feature request.

I want to only generate a new correlation id if it is missing in the request; but use the one if one is passed in the request. If you are OK with it, I will put something together and send a pull request.

Here are my findings,

I created the Django project and added your testapp from the sandbox as an app and updated the settings.py as well. This is how my settings.py file looks like,

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    # apps
    'LoggingTest',
    'testapp.apps.TestAppConfig',

    # 3rd party packages
    'rest_framework',
    'cid.apps.CidAppConfig',
]

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',

    # 3rd party middlewares
    'cid.middleware.CidMiddleware',
]

LOGGING = {
    'version': 1,
    'formatters': {
        'verbose': {
            'format': '[cid: %(cid)s] %(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
        },
        'simple': {
            'format': '[cid: %(cid)s] %(levelname)s %(message)s'
        },
    },
    'handlers': {
        'console': {
            'level': 'INFO',
            'class': 'logging.StreamHandler',
            'formatter': 'verbose',
        },
    },
    'filters': {
        'correlation': {
            '()': 'cid.log.CidContextFilter'
        },
    },
    'loggers': {
        'testapp': {
            'handlers': ['console'],
            'filters': ['correlation'],
            'propagate': True,
        },
    },
}

Everything similar to the sandbox/settings.py except CID_GENERATE = True part. I didn't add that.

I tested your ok API and got the result as expected like below,

I tried several times but the logged details are displayed without cid value, it's None always.

Performing system checks...

System check identified no issues (0 silenced).

You have 1 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): testapp.
Run 'python manage.py migrate' to apply them.
September 25, 2020 - 16:51:14
Django version 3.1.1, using settings 'SampleDjangoProject.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
[cid: None] WARNING 2020-09-25 16:51:15,271 views 10879 139911689578240 This is a warning from django-cid test application.
[cid: None] WARNING 2020-09-25 16:51:16,158 views 10879 139911689578240 This is a warning from django-cid test application.
[cid: None] WARNING 2020-09-25 16:51:17,001 views 10879 139911689578240 This is a warning from django-cid test application.
[cid: None] WARNING 2020-09-25 16:51:17,835 views 10879 139911689578240 This is a warning from django-cid test application.
[cid: None] WARNING 2020-09-25 16:51:18,572 views 10879 139911689578240 This is a warning from django-cid test application.

According to the django-cid documentation,

The middleware takes care of getting the correlation from the HTTP request header. By default it looks for a header named X_CORRELATION_ID, but you can change this with the CID_HEADER setting:

CID_HEADER = 'X_MY_CID_HEADER'

Then I added CID_HEADER as X_CORRELATION_ID to the settings.py file. Still the problem is same, so just updated CID_HEADER as HTTP_X_CORRELATION_ID, then it worked with X-Correlation-Id API header.

Performing system checks...

System check identified no issues (0 silenced).

You have 1 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): testapp.
Run 'python manage.py migrate' to apply them.
September 25, 2020 - 16:48:38
Django version 3.1.1, using settings 'SampleDjangoProject.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
[cid: 123-321] WARNING 2020-09-25 16:48:44,157 views 10497 139905992644352 This is a warning from django-cid test application.
[cid: 123-321] WARNING 2020-09-25 16:48:44,610 views 10497 139905992644352 This is a warning from django-cid test application.
[cid: 123-321] WARNING 2020-09-25 16:48:45,346 views 10497 139905992644352 This is a warning from django-cid test application.
[cid: 123-321] WARNING 2020-09-25 16:48:45,941 views 10497 139905992644352 This is a warning from django-cid test application.
[cid: 123-321] WARNING 2020-09-25 16:48:46,668 views 10497 139905992644352 This is a warning from django-cid test application.

So I need to add this logging to the other app as well, so I changed testapp to '' in loggers in LOGGING like below,

LOGGING = {
    ...
    'loggers': {
        '': {
            'handlers': ['console'],
            'filters': ['correlation'],
            'propagate': True,
        },
    },
}

After that I didn't get any logging details, here what Django official documentation mentioned about Configuring logging,

If the disable_existing_loggers key in the LOGGING dictConfig is set to True (which is the dictConfig default if the key is missing) then all loggers from the default configuration will be disabled. Disabled loggers are not the same as removed; the logger will still exist, but will silently discard anything logged to it, not even propagating entries to a parent logger. Thus you should be very careful using 'disable_existing_loggers': True; it’s probably not what you want. Instead, you can set disable_existing_loggers to False and redefine some or all of the default loggers; or you can set LOGGING_CONFIG to None and handle logging config yourself.

So just added disable_existing_loggers as False to the LOGGING like below,

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    ...
}

After that I got the following error,

Watching for file changes with StatReloader
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.6/logging/__init__.py", line 994, in emit
    msg = self.format(record)
  File "/usr/lib/python3.6/logging/__init__.py", line 840, in format
    return fmt.format(record)
  File "/usr/lib/python3.6/logging/__init__.py", line 580, in format
    s = self.formatMessage(record)
  File "/usr/lib/python3.6/logging/__init__.py", line 549, in formatMessage
    return self._style.format(record)
  File "/usr/lib/python3.6/logging/__init__.py", line 391, in format
    return self._fmt % record.__dict__
KeyError: 'cid'
Call stack:
  File "manage.py", line 22, in <module>
    main()
  File "manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/home/wasdkiller/PycharmProjects/SampleDjangoProject/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/home/wasdkiller/PycharmProjects/SampleDjangoProject/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/wasdkiller/PycharmProjects/SampleDjangoProject/venv/lib/python3.6/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/wasdkiller/PycharmProjects/SampleDjangoProject/venv/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 61, in execute
    super().execute(*args, **options)
  File "/home/wasdkiller/PycharmProjects/SampleDjangoProject/venv/lib/python3.6/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/home/wasdkiller/PycharmProjects/SampleDjangoProject/venv/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 96, in handle
    self.run(**options)
  File "/home/wasdkiller/PycharmProjects/SampleDjangoProject/venv/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 103, in run
    autoreload.run_with_reloader(self.inner_run, **options)
  File "/home/wasdkiller/PycharmProjects/SampleDjangoProject/venv/lib/python3.6/site-packages/django/utils/autoreload.py", line 612, in run_with_reloader
    logger.info('Watching for file changes with %s', reloader.__class__.__name__)
Message: 'Watching for file changes with %s'
Arguments: ('StatReloader',)
Performing system checks...

System check identified no issues (0 silenced).

You have 1 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): testapp.
Run 'python manage.py migrate' to apply them.
September 25, 2020 - 16:46:23
Django version 3.1.1, using settings 'SampleDjangoProject.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

In the last Example they've added filters to the handlers. I did the same and modified like below,

LOGGING = {
    ...
    'handlers': {
        'console': {
            'level': 'INFO',
            'class': 'logging.StreamHandler',
            'formatter': 'verbose',
            'filters': ['correlation'],
        },
    },
    ...
}

That's it, now everything works fine. Please update django-cid documentation accordingly.

I faced the issues on below environment,

• Django --> 3.1.1
• django-cid --> 2.1
• Python --> 3.6.9

Hi,

I noticed last commit is one year old. There are already some contributions to support Django 1.11 but they aren't merged.
Is there a way I could help with maintaining this application ?

Hi,

When I follow the tutorial for setting up the library, I noticed this sequence of events when I run the app:

1. The reloader is loaded. With it, a cid is generated and included in the log.
2. When a request is made, the library gets and returns the correct cid to the caller.
3. At the end of the request lifecycle, channel prints the access log, a new cid is generated and included in the log.
4. When another request is made, the library, again, gets and returns the correct cid to the caller.
5. At the end of the request lifecycle, the same cid as in step 3 is used in the log. This same cid is used in all subsequent access logs.

Channels log definition: https://github.com/django/channels/blob/main/channels/management/commands/runserver.py#L142
Helpful PR: https://github.com/django/channels/pull/693/files
The line that generates the response in channels: https://github.com/django/channels/blob/main/channels/http.py#L256

as mentioned on https://docs.djangoproject.com/en/1.11/releases/1.11/#database-backend-api:
"The name keyword argument is added to the DatabaseWrapper.create_cursor(self, name=None) method to allow usage of server-side cursors on backends that support it."
(currently only postgresql)

Hello,

I am installing django-cid with pip, latest version available is 0.2.0 and it's missing this fix

#16

Is there a plan to publish newer version to pip soon?

Thanks!

Just a question. Why did you drop support of old Django versions and old python versions.
A lot of big projects are still using these versions as it is quite difficult for them to migrate.
We would like to use your library in our projects but all projects have different setup, some of them are python3.7 and latest Django and some of them are python2.7 and Django1.7.

If we add support only for a half of out setup then it just does not make any sense. Just curiosity why it is difficult to support old versions?!

Hello,

I think it would be safer to create a RecordFactory to initialise the record with a cid attribute, to avoid KeyError on some logging occasions.
https://docs.python.org/2/library/logging.html#logging.Logger.makeRecord

Exemple:

old_factory = logging.getLogRecordFactory()
def cid_record_factory(*args, **kwargs):
    record = old_factory(*args, **kwargs)
    record.cid = None
    return record

logging.makeRecordFactory(cid_record_factory)