[bug] Validation time increased more than 10 times in dynaconf 3.2.0 about dynaconf HOT 7 CLOSED

I must have had something weird in my local config. Unfortunately, I did not back it up, so I can not elaborate on it more. Closing the issue. Thank you for your attention. 🙂

from dynaconf.

This issue will probably be addressed by #981

The time increased because of this change https://github.com/dynaconf/dynaconf/pull/892/files#diff-32c3b833b4d892e0c8c968669eb1875096b9e7ef19024dff76f32c2771669615L285 which makes all validators to call .set for every key.

With #981 validation will not set values anymore then will be faster.

from dynaconf.

@ogajduse I was going to profile it, but I could not reproduce this timing difference using ubuntu:jammy, python 3.10 and v3.1.12 vs v3.2.2. Could you assure this is still happening under a fresh install?

I got the following timing:

3.2.2
1.63 user
0.08 system
0:01.73 elapsed

3.1.12
1.70 user
0.12 system
0:01.83 elapsed

I've ran it using this Dockerfile:

FROM ubuntu:jammy

RUN : \
    && apt-get update \ 
    && apt-get install -y --no-install-recommends \
        python3 python3-dev python3.10-venv git ca-certificates time \
        libcurl4-openssl-dev libssl-dev build-essential libffi-dev libxml2-dev \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

RUN : \
    && git clone https://github.com/SatelliteQE/robottelo.git ~/robottelo \
    && git clone https://github.com/dynaconf/dynaconf.git ~/dynaconf 

RUN : \
    && echo "SETTING UP VENV" \
    && cd ~/robottelo \
    && python3 -m venv venv \
    && venv/bin/python -m pip install -r requirements.txt 

CMD : \
    ; cd ~/robottelo \
    ; for conffile in conf/*.yaml.template; do mv -- "$conffile" "${conffile%.yaml.template}.yaml"; done \
    # editable dynaconf install
    ; echo "installing dynaconf as editable package..." \
    ; . venv/bin/activate \
    ; pip install -e ~/dynaconf 1>/dev/null 2>&1 \
    # warmup: do first-run stuff
    ; ~/robottelo/venv/bin/python3 -c \
        'from robottelo.config import settings' >/dev/null 2>&1 \
    # run 3.2.2 (main)
    ; echo "" \
    ; dynaconf --version \
    ; time python3 -c 'from robottelo.config import settings' \
    # chekout 3.1.12
    ; cd ~/dynaconf >/dev/null \
    ; git checkout 3.1.12 1>/dev/null 2>&1 \
    ; cd - \
    # run 3.1.12
    ; echo "" \
    ; dynaconf --version \
    ; time python3 -c 'from robottelo.config import settings'

from dynaconf.

@pedro-psb I am still seeing the same. I suspect that it is the Hashicorp Vault that extends the validation time. However, I have no data to prove it.
Did you try profiling dynaconf before? I could profile it and see what makes the validation run that long.
Let me know, if there is a better way to debug it and find where the issue is.

from dynaconf.

Did you try profiling dynaconf before?

@ogajduse Not really, I would dig into that now.

I could profile it and see what makes the validation run that long.

I think that's the way to go. It should be easy to spot where the hotspot is, but you could also share the pstats file here so we can have a look (assuming you're using cProfile).

from dynaconf.

I do not observe this issue anymore. I am unable to pinpoint what was causing it, unfortunately.
I am closing this issue, feel free to reopen it.

from dynaconf.

Reopening since I bumped into it again.
@pedro-psb I have used the Dockerfile you provided and did a test run.
The only thing that I changed in the Dockerfile was the CMD from which I removed the config manipulation line. I built an image and ran it.

$ podman run -t -v /home/ogajduse/repos/robottelo/conf:/root/robottelo/conf:Z -v /home/ogajduse/repos/robottelo/.env:/root/robottelo/.env:Z robottelo-test
installing dynaconf as editable package...

3.2.3
2023-11-29 13:50:58 - robottelo - WARNING - Dynaconf validation failed, continuing for the sake of unit tests
    vlan_networking.subnet is required in env main
2023-11-29 13:50:58 - robottelo - WARNING - Dynaconf validation failed, continuing for the sake of unit tests
    vlan_networking.subnet is required in env main
21.70user 0.09system 0:22.78elapsed 95%CPU (0avgtext+0avgdata 66728maxresident)k
0inputs+24outputs (0major+12364minor)pagefaults 0swaps
/root/robottelo

3.1.12
2023-11-29 13:51:03 - robottelo - WARNING - Dynaconf validation failed, continuing for the sake of unit tests
    vlan_networking.subnet is required in env main
2023-11-29 13:51:03 - robottelo - WARNING - Dynaconf validation failed, continuing for the sake of unit tests
    vlan_networking.subnet is required in env main
3.30user 0.08system 0:04.22elapsed 79%CPU (0avgtext+0avgdata 66764maxresident)k
32inputs+32outputs (0major+12282minor)pagefaults 0swaps

I have injected my configuration into the container as well as the .env file with a HashiCorp Vault token.

I have figured out that if I remove validators for repos settings, I get results in around 3-4 seconds.
https://github.com/SatelliteQE/robottelo/blob/bcc4d305535918b53c6892b4733ad9c44fb26fda/robottelo/config/validators.py#L249-L267

I will investigate more and post the results here.

from dynaconf.