AWX not able to delete the worker pods after finished running about awx HOT 6 OPEN

I observe the same issues on Kubernetes 1.27 with AWX 23.0.0.

The pods that are not deleted are pods where the awx jobs have been deleted immediately after the pod failed. It looks like awx only knows about existing pods through the jobs inside of awx.

If this is the case, the pod should be actively be removed from Kubernetes when the job is deleted OR the api output of the job should give a hint on whether the pod has already been deleted inside kubernetes.

from awx.

can u give us the output of /api/v2/jobs/462026

from awx.

@TheRealHaoLiu below is the output.
I want to highlight again, pod is not deleting only for few inventory sync jobs which is completed successfully

ansible-inventory [core 2.15.5]
config file = /ansible.cfg
configured module search path = ['/cyberark-ansible-modules/lib/ansible/modules', '/runner/project']
ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections:/usr/share/automation-controller/collections
executable location = /usr/local/bin/ansible-inventory
python version = 3.9.18 (main, Jan 24 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3)
jinja version = 3.0.0
libyaml = False
Using /ansible.cfg as config file
[DEPRECATION WARNING]: DEFAULT_GATHER_TIMEOUT option, the module_defaults
keyword is a more generic version and can apply to all calls to the
M(ansible.builtin.gather_facts) or M(ansible.builtin.setup) actions, use
module_defaults instead. This feature will be removed from ansible-core in
version 2.18. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
redirecting (type: inventory) ansible.builtin.aws_ec2 to amazon.aws.aws_ec2
Using inventory plugin 'ansible_collections.amazon.aws.plugins.inventory.aws_ec2' to process inventory source '/runner/inventory/aws_ec2.yml'
Parsed /runner/inventory/aws_ec2.yml inventory source with auto plugin
8.867 INFO Processing JSON output...
8.868 INFO Loaded 1 groups, 0 hosts
8.898 INFO Inventory import completed for AWS-sandbox-Windows in 0.0s

from awx.

@chinna44 that does not look like the output from the api endpoint... that looks like the stdout of the job

from awx.

@TheRealHaoLiu yes.. you are correct, I'm sorry for that

Below is the output for endpoint /api/v2/jobs/462026, but I could not see the details of job for this kind of any other Inventory Sync jobs. Please let me know if you require details in any other possible ways

HTTP 404 Not Found
Allow: GET, DELETE, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept
X-API-Node: awx-web-c8bc64f45-h7xwt
X-API-Product-Name: AWX
X-API-Product-Version: 23.9.0
X-API-Time: 0.057s

{
"detail": "Not found."
}

from awx.

@TheRealHaoLiu please let me know if you need any other details

from awx.