Traceback when using `stdlib.filter_by_level` with Python Standard Logging Library about structlog HOT 3 CLOSED

Hmm filter_by_level makes no sense in a ProcessorFormatter processor, because filtering at this point is done by logging.

Generally speaking the execution environment within ProcessorFormatter is very different than within structlog proper.

from structlog.

Hi @hynek , thank you very much for the comment.

Could we have an explanation which processors cannot be used in stdlib configuration?
Also I have a small confusion between processors and foreign_pre_chain? When I should use which?

Example:

COMMON_PRE_CHAIN: list[structlog.typing.Processor] = [
    structlog.contextvars.merge_contextvars,
    structlog.stdlib.filter_by_level,  # <-- this "common" processor is causing issues in stdlib configuration
    structlog.processors.TimeStamper(fmt='iso'),
    structlog.stdlib.add_logger_name,
    structlog.stdlib.add_log_level,
    structlog.stdlib.PositionalArgumentsFormatter(),
    structlog.processors.StackInfoRenderer(),
    structlog.processors.UnicodeDecoder(),
]

LOGGING: dict[str, t.Any] = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {'event_blacklist': {'()': 'app.settings.components.DevelopmentFilter'}},
    'formatters': {
        'plain_console': {
            '()': structlog.stdlib.ProcessorFormatter,
            'processors': [structlog.stdlib.ProcessorFormatter.remove_processors_meta, structlog.dev.ConsoleRenderer()],  # <-- what is the difference between processors here and `foreign_pre_chain` later on? One is used for `structlog` logs and another for `stdlib` logs? We are trying it to keep "the same"
            'foreign_pre_chain': COMMON_PRE_CHAIN,  # <-- the issue is caused here with the same example
        },
    },
}

structlog.configure(
    processors=[*COMMON_PRE_CHAIN, structlog.stdlib.ProcessorFormatter.wrap_for_formatter],
    logger_factory=structlog.stdlib.LoggerFactory(),
    wrapper_class=structlog.stdlib.BoundLogger,
    cache_logger_on_first_use=True,
)

With the following configuration we have exactly the same traceback. In the documentation it is stated that you should keep your structlog configuration and stdlib logging configuration as close as possible (to have similar logs).

Could you please explain which processors we can't/shouldn't use in foreign_pre_chain?

from structlog.

structlog.stdlib.ProcessorFormatter's processors are executed on every log entry that arrives in ProcessorFormatter. That is: log entries coming from logging and entries that have been looped back into structlog (see also the graph at https://www.structlog.org/en/stable/standard-library.html#rendering-using-structlog-based-formatters-within-logging)

structlog.stdlib.ProcessorFormatter's foreign_pre_chain is run only on log entries coming from logging before handing it over to the processors in the previous paragraph. This chain is mostly to add data that you might want to use on processors like structlog-style timestamps etc.

I cannot answer your question in general because it's complicated. OP's problem was that they used a processor that needed a stdlib logger to get the log level. I'm not even sure what your problem is.

from structlog.