Error on project update since 23.9.0 ("No file was found when using first_found." on "Fetch galaxy roles and collections from requirements") about awx HOT 12 CLOSED

@2and3makes23
Hi, I can't reproduce this issue on 23.9.0 on K3s. Could you please clarify that the issue exists or not with following demo project?

1. Add new project with URL https://github.com/ansible/ansible-tower-samples
   • Add new project anyway even if you already have "Demo Project"
2. Gather job output

If you've faced issue with demo project, could you please provide:

• Full job output
• Output from kubectl -n <namespace> describe deployment/<instance-name>-task
• Output from kubectl -n <namespace> exec deployment/<instance-name>-task -c <instance-name>-ee -- ansible --version

Technically, since skip=True is specified for ansible.builtin.first_found lookup plugin, req_file will be an empty list if there is no requirements file. So I don't know why No file was found when using first_found is ocurred.

from awx.

Addressed by: #15017

from awx.

Thanks a lot @kurokobo . This issue was not on my radar

from awx.

I tried what happens if I provide an empty /requirements.yml (contains only ---\n) for combined roles and collections.

In this case, another error occurs:

changed: true
stdout: ''
stderr: >-
  [DEPRECATION WARNING]: ANSIBLE_COLLECTIONS_PATHS option, does not fit var 

  naming standard, use the singular form ANSIBLE_COLLECTIONS_PATH instead. This 

  feature will be removed from ansible-core in version 2.19. Deprecation
  warnings
   can be disabled by setting deprecation_warnings=False in ansible.cfg.
  ERROR! No requirements found in file
  '/var/lib/awx/projects/_376__awxtesting/requirements.yml'
rc: 1
cmd:
  - ansible-galaxy
  - install
  - '-r'
  - /var/lib/awx/projects/_376__awxtesting/requirements.yml
start: '2024-03-06 08:12:38.378667'
end: '2024-03-06 08:12:40.390555'
delta: '0:00:02.011888'
msg: non-zero return code
invocation:
  module_args:
    chdir: /var/lib/awx/projects/_376__awxtesting
    _raw_params: >-
      ansible-galaxy install -r
      /var/lib/awx/projects/_376__awxtesting/requirements.yml 
    _uses_shell: false
    expand_argument_vars: true
    stdin_add_newline: true
    strip_empty_ends: true
    argv: null
    executable: null
    creates: null
    removes: null
    stdin: null
stdout_lines: []
stderr_lines:
  - '[DEPRECATION WARNING]: ANSIBLE_COLLECTIONS_PATHS option, does not fit var '
  - >-
    naming standard, use the singular form ANSIBLE_COLLECTIONS_PATH instead.
    This 
  - >-
    feature will be removed from ansible-core in version 2.19. Deprecation
    warnings
  - ' can be disabled by setting deprecation_warnings=False in ansible.cfg.'
  - >-
    ERROR! No requirements found in file
    '/var/lib/awx/projects/_376__awxtesting/requirements.yml'
_ansible_no_log: false

Adding some random role entry solves this. So (unless this is the intended reaction to empty requirements.yml files) in addition to the conditional mentioned above, there needs to be added an additional check to prevent this from happening as well.

from awx.

@kurokobo
Thank you for your quick reply.

To reproduce I recommend a simplified copy of e.g. this task from project_update.yml:

# removed irrelevant parts for testing
    - name: Test behavior of task
      debug:
        msg: 'I am able to run'
      vars:
        req_file: "{{ lookup('ansible.builtin.first_found', req_candidates, skip=True) }}"
        req_candidates:
          - "roles/requirements.yml"
          - "roles/requirements.yaml"
      when:
        - req_file

For me locally, like in AWX, this produces:

PLAY [test play] ***********************************************************************************************************************************************************************************************************************************************************************

TASK [Test behavior of task] ***********************************************************************************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'req_file' failed. The error was: An unhandled exception occurred while running the lookup plugin 'ansible.builtin.first_found'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found. Use errors='ignore' to allow this task to be skipped if no files are found\n\nThe error appears to be in 'testplay.yml': line 11, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n    - name: Test behavior of task\n      ^ here\n"}

PLAY RECAP *****************************************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

As to your point regarding the first_found_lookup documentation:

This is useful when used with with_first_found, as an empty list return to with_ calls causes the calling task to be skipped.

When used as a template via lookup or query, setting skip=True will not cause the task to skip. Tasks must handle the empty list return from the template.

I am inclined to believe that the second cited line from documentation applies in this case, using lookup('ansible.builtin.first_found'. What do you think?

It´s been a long work day and I have to log off. Please consider checking out above proposed test task - thanks so much!

from awx.

@2and3makes23
Okay, it's related to: ansible/ansible#81503

As a workaround, use latest Ansible 2.15.x instead.

The reason why I can't reproduce the issue is the Ansible version in default awx-ee image is 2.15.9. I believe you are using customized (or old) EE images that contains newer (or older) Ansible for your control plane.

Applicable fix is passing list of files as a keyword param instead of positional param. However it is a difficult decision as to which is safer, as some versions of Ansible will not work unless it is a positional param, and some will not work unless it is a keyword param.

    - name: Test behavior of task
      debug:
        var: lookup('first_found', req_candidates, skip=True)
      vars:
        # Not works on 2.16
        #req_file: "{{ lookup('ansible.builtin.first_found', req_candidates, skip=True) }}"
        # Works on 2.16
        req_file: "{{ lookup('ansible.builtin.first_found', files=req_candidates, skip=True) }}"
        req_candidates:
          - "sitea.yml"
      when:
        - req_file

@bcoca @fosterseth
F.Y.I.
I think this should be fixed in upstream Ansible in ansible/ansible#81503 since following example snippet on the doc no longer works on the latest Ansible 2.16.x.

- name: Set _found_file to the first existing file, or an empty list if no files found
  ansible.builtin.set_fact:
    _found_file: "{{ lookup('ansible.builtin.first_found', files, paths=['/extra/path'], skip=True) }}"
  vars:
    files:
      - /path/to/foo.txt
      - /path/to/bar.txt

As an AWX side, to be safer, shoud we use keyword param instead, or keep this as-is? I vote for keeping this as-is, but I'm not particular about it.

from awx.

@2and3makes23

I am inclined to believe that the second cited line from documentation applies in this case, using lookup('ansible.builtin.first_found'. What do you think?

Although the bug makes it difficult to understand, the second line means, 'Please write the task in such a way that returned empty list does not cause the task to fail'.

from awx.

We are using ansible-core 2.16.4 within Control Plane

@2and3makes23

I am inclined to believe that the second cited line from documentation applies in this case, using lookup('ansible.builtin.first_found'. What do you think?

Although the bug makes it difficult to understand, the second line means, 'Please write the task in such a way that returned empty list does not cause the task to fail'.

So what do you think about the proposed change/addition to the three task´s conditionals as a way to steer clear of the ansible version dependant behavior of first_found?

when:
  - ...
  - req_file is defined
  - req_file | stat | isfile

from awx.

@2and3makes23

So what do you think about the proposed change/addition to the three task´s conditionals as a way to steer clear of the ansible version dependant behavior of first_found?

The exception No file was found when using first_found is occurred during processing lookup. This is BEFORE when is going to be processed, so modifying when solves nothing.

I think the most effective fix would be to add errors='ignore' to lookup.

req_file: "{{ lookup('ansible.builtin.first_found', req_candidates, skip=True, errors='ignore') }}"

It will probably work as expected with all supported Ansible version.
However, not only specific exception but also all exceptions will be ignored, so there might be side effects (I don't think this will be a problem most of the cases though).

from awx.

@kurokobo
I see, thanks for the clarification.

Is it possible that using blocks with rescue might enable you to handle this specific behavior without sacrificing as much as with ignore_errors: true?

Another valid approach might be to check for the three requirements.yml files using a set_fact task, before actually attempting to process them?

from awx.

@2and3makes23

Another valid approach might be to check for the three requirements.yml files using a set_fact task, before actually attempting to process them?

This is a more straightforward solution.

However, I have my doubts that it is an appropriate decision to include new logic in the playbook, to use untested Ansible version in the control plane in the first place.
I'm just one of the community contributors and would like to hear ideas from the AWX core team.

from awx.

Fixed via 934646a ❤️

from awx.