404 Error while downloading ansible collection

Support
#1

Problem:
After upgrading ansible plugin from 0.12 to 0.16 ansible-galaxy install receives a new api generated url for the content download that does not exist in the server

Expected outcome:
Succes installing pulp hosted ansible collection

Pulpcore version:
3.22.0

Pulp plugins installed and their versions:
ansible 0.16.0
file 1.11.2
rpm 3.18.9

Operating system - distribution and version:
rhel 7.9

Other relevant data:

Log:

ansible-galaxy collection install --server http://server:8081/pulp_ansible/galaxy/collections/testom -r requirements.yml -c -vvv
ansible-galaxy 2.9.27
config file = /etc/ansible/ansible.cfg
…
Processing requirement collection ‘netbox.netbox’
Collection ‘netbox.netbox’ obtained from server cmd_arg http://server:8081/pulp_ansible/galaxy/collections/testom/api
Starting collection install process
Installing ‘netbox.netbox:3.7.1’ to ‘/$HOME/.ansible/collections/ansible_collections/netbox/netbox’
Downloading http://server/pulp_ansible/galaxy/default/api/v3/plugin/ansible/content/collections/testom/collections/artifacts/netbox-netbox-3.7.1.tar.gz to /$HOME/.ansible/tmp/ansible-local-118835XNWFni/tmpcU7cIc

ERROR! Unexpected Exception, this is probably a bug: HTTP Error 404: Not Found
…

#2

using the pulp cli for distribution show

pulp --config $CONFIG ansible  distribution show --name testom
{
  "pulp_href": "/pulp/api/v3/distributions/ansible/ansible/d5b4a651-8884-4f2f-be3e-7068175d5cd2/",
  "pulp_created": "2023-02-07T17:39:42.326210Z",
  "base_path": "collections/testom",
  "content_guard": null,
  "name": "testom",
  "repository": "/pulp/api/v3/repositories/ansible/ansible/e5b532d4-9d0b-408a-92f6-f93a8647e787/",
  "repository_version": null,
  "client_url": "http://server/pulp_ansible/galaxy/collections/testom/",
  "pulp_labels": {}
}

The api generated url that fails to download is:
http://server/pulp_ansible/galaxy/default/api/v3/plugin/ansible/content/collections/testom/collections/artifacts/netbox-netbox-3.7.1.tar.gz to

However, the collectiion is downloadable via the content url:
http://server/pulp/content/collections/testom/netbox-netbox-3.7.1.tar.gz

Question: How I get the pulp Api to offer the second url above?

#3

Thanks for the report I have confirmed this is a bug when using a distribution with a base-path containing multiple parts. I’ll file a bug for this and fix it for the next release. In the meantime you can modify the base-path to only be one part, maybe concatenate the two “collections-testom”, so that the generated urls will be downloadable.

#4

Here is the filed issue: https://github.com/pulp/pulp_ansible/issues/1350

#5

Thanks,
I tried using a single “collections-testom2” in a new distribution and had the same error :frowning:

So I did some digging in the Api logs and noticed that when galaxy requests the artifact the first time pulp returns a 302 relocation with a new location that is preceded with the string “localhost” without protocol so it gets appended to the original url

Here is the traffic using curl instead of galaxy:

 curl -v  'http://server:8081/pulp_ansible/galaxy/default/api/v3/plugin/ansible/content/collections-testom2/collections/artifacts/netbox-netbox-3.7.1.tar.gz' -L
* About to connect() to server port 8081 (#0)
*   Trying 10.86.70.152...
* Connected to tn-pulp-qa.ms.com (10.86.70.152) port 8081 (#0)
> GET /pulp_ansible/galaxy/default/api/v3/plugin/ansible/content/collections-testom2/collections/artifacts/netbox-netbox-3.7.1.tar.gz HTTP/1.1
> User-Agent: curl/7.29.0
> Host: tn-pulp-qa.ms.com:8081
> Accept: */*
> 
< HTTP/1.1 **302 Found**
< Server: nginx/1.14.1
< Date: Wed, 08 Feb 2023 03:29:09 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 0
< Connection: keep-alive
< Location: localhost/pulp/content/collections-testom2/netbox-netbox-3.7.1.tar.gz
< Vary: Accept, Cookie
< Allow: GET, HEAD, OPTIONS
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
< Referrer-Policy: same-origin
< Correlation-ID: eb3289b4267842548c72b26a15385a13
< Access-Control-Expose-Headers: Correlation-ID
< X-MS-UNIQUE-ID: c3ZCHxjdqIG4JaYqF4S
< 
* Connection #0 to host serrver left intact
* Issue another request to this URL: 'http:/server:8081/pulp_ansible/galaxy/default/api/v3/plugin/ansible/content/collections-testom2/collections/artifacts/localhost/pulp/content/collections-testom2/netbox-netbox-3.7.1.tar.gz'
* Found bundle for host tn-pulp-qa.ms.com: 0xd04160
* Re-using existing connection! (#0) with host tn-pulp-qa.ms.com
* Connected to tn-pulp-qa.ms.com (10.86.70.152) port 8081 (#0)
> GET /pulp_ansible/galaxy/default/api/v3/plugin/ansible/content/collections-testom2/collections/artifacts/localhost/pulp/content/collections-testom2/netbox-netbox-3.7.1.tar.gz HTTP/1.1
> User-Agent: curl/7.29.0
> Host: tn-pulp-qa.ms.com:8081
> Accept: */*
> 
< HTTP/1.1 404 Not Found
#6

I believe now that the problem lies in the 302 redirection not being well formed

#7

Have you tried restarting all of Pulp to ensure the API & Task workers are using the latest version? I have not been able to reproduce this redirect error with the corrected base-path. On your Pulp machine can you run dynaconf list, this will list out the current settings Pulp is using. Check that CONTENT_ORIGIN, ANSIBLE_API_HOSTNAME and ANSIBLE_CONTENT_HOSTNAME are all what you expect them to be.

1 Like