Getting '404 Not found' while creating pulp content guard

Support
#1

Hi,
I am using Pulp-RPM for creating repository, remote, sync, publish and distribution. While creating content guard I am getting an 404 Not found error.

ERROR:
Command : http POST http://localhost:24817/pulp/api/v3/contentguards/certguard/x509/ name=AL23Guard ca_certificate=@/root/easy-rsa/easyrsa3/pki/ca.pem
Output :
HTTP/1.1 404 Not Found
Access-Control-Expose-Headers: Correlation-ID
Connection: close
Content-Length: 179
Content-Type: text/html; charset=utf-8
Correlation-ID: 7780f6eaa7c249d699489e671f36d2b6
Cross-Origin-Opener-Policy: same-origin
Date: Thu, 31 Aug 2023 07:52:52 GMT
Referrer-Policy: same-origin
Server: gunicorn
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

<!doctype html>

Not Found

Not Found

The requested resource was not found on this server.

The same error I am also getting when I use port 8080 instead of 24817

Pulpcore version:
pulp-rpm 3.22.3
pulpcore 3.31.0

Operating system - distribution and version:
Amazon Linux 2023

Other relevant data:
This is the documentation I am following : Create, Sync and Publish a Repository — Pulp RPM Support 3.22.3 documentation

Can you please help me with what I am missing.
I really appreciate your time and help.
Thank You!

#2

You are missing the pulp-certguard plugin.

#3

Hi,
Sorry I forgot to mention in the topic. I have pulp-certguard=1.6.5 installed.

#4

Is it listed in http://localhost:8080/pulp/api/v3/status/?
What do you get on http://localhost:8080/pulp/api/v3/contentguards/certguard/x509/?

#5

No, I did not see pulp-certguard here. This is the output for http://localhost:8080/pulp/api/v3/status/
Pulp 3
* Log in

 * Pulp Api Root
 * Status

GET (BUTTON)
* json
* api

(BUTTON) OPTIONS

Status

Returns status information about the application
GET /pulp/api/v3/status/

HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
“versions”: [
{
“component”: “core”,
“version”: “3.31.0”,
“package”: “pulpcore”,
“domain_compatible”: true
},
{
“component”: “rpm”,
“version”: “3.22.3”,
“package”: “pulp-rpm”,
“domain_compatible”: true
},
{
“component”: “container”,
“version”: “2.16.0”,
“package”: “pulp-container”,
“domain_compatible”: false
},
{
“component”: “file”,
“version”: “1.14.4”,
“package”: “pulp-file”,
“domain_compatible”: true
}
],
“online_workers”: [
{
“pulp_href”: “/pulp/api/v3/workers/018a40e4-5e1c-7052-8893-42ccaff38831/”,
“pulp_created”: “2023-08-29T10:43:42.749944Z”,
“name”: “5785@localhost.localdomain”,
“last_heartbeat”: “2023-08-31T09:28:28.742634Z”,
“current_task”: null
},
{
“pulp_href”: “/pulp/api/v3/workers/018a40e4-810c-7eaf-82bd-9f6aff982c25/”,
“pulp_created”: “2023-08-29T10:43:51.693490Z”,
“name”: “5791@localhost.localdomain”,
“last_heartbeat”: “2023-08-31T09:28:28.740158Z”,
“current_task”: null
}
],
“online_content_apps”: [
{
“name”: “6003@ip-172-x-x-x.ec2.internal”,
“last_heartbeat”: “2023-08-31T09:28:27.773611Z”
},
{
“name”: “6002@ip-172-x-x-x.ec2.internal”,
“last_heartbeat”: “2023-08-31T09:28:28.382031Z”
}
],
“database_connection”: {
“connected”: true
},
“redis_connection”: {
“connected”: false
},
“storage”: {
“total”: 48251273216,
“used”: 29306904576,
“free”: 18944368640
},
“content_settings”: {
“content_origin”: “http://localhost:24816”,
“content_path_prefix”: “/pulp/content/”
},
“domain_enabled”: false
}

http://localhost:8080/pulp/api/v3/contentguards/certguard/x509/
Output :
The requested resource was not found on this server.

#6

certguard should most definitely be amongs the entries in “versions”.

That brings the question, how do you install pulp?

#7

I installed pulpcore using pip. Basically we followed this documentation Instructions — Pulp Project 3.32.0 documentation

#8

I see. Then you should add pulp-certguard to the pip install command (along with every other plugin you expect to use). And do not forget to run the migrations afterwards and restart the services.

BTW: Step 16 sounds wrong to me. If you installed the systemd files, just skip it.

#9

I can see pulp-certguard in versions list. Maybe because I did not restarted the services after installing the pulp-certguard. I’ll try to create content guard again, will keep you posted. Thank You!
“versions”: [
{
“component”: “core”,
“version”: “3.31.0”,
“package”: “pulpcore”,
“domain_compatible”: true
},
{
“component”: “rpm”,
“version”: “3.22.3”,
“package”: “pulp-rpm”,
“domain_compatible”: true
},
{
“component”: “container”,
“version”: “2.16.0”,
“package”: “pulp-container”,
“domain_compatible”: false
},
{
“component”: “file”,
“version”: “1.14.4”,
“package”: “pulp-file”,
“domain_compatible”: true
},
{
“component”: “certguard”,
“version”: “1.6.5”,
“package”: “pulp-certguard”,
“domain_compatible”: true
}
],

1 Like
#10

Hi @x9c4, I was able to create the content guard and Protect the Distribution with the X509CertGuard. Appreciate your help. Thank you for your time!

2 Likes
#11

You are welcome!
How was your experience with the pypi installation? Did you go with the systemd files?
I think I can say none of us developers has done such an installation in quite some while, as we all use the containerized method these days.

1 Like
#12

The PyPI installation process went well, won’t say it was buttery smooth, but most of the instructions are straightforward. Yes, I did choose to go with the systemd files for managing the service. I faced some challenges in pulpcore-workers service, but I was able to solve them. If you have any tips or insights on how you’ve implemented containerization effectively, I’d love to hear them. Thanks again for your assistance.

3 Likes
#13

Hi @x9c4, sorry to bother you again. As I mentioned previously I was able to create a content guard and protect my distribution but when I tried to access it from client side I am getting one error.
Command : curl -k --cert /root/pki/issued/yum-client.pem --key /root/pki/private/yum-client.key https://example.com/pulp/content/pulp-data/

Output : 403: A client certificate was not received via the X-CLIENT-CERT header.

Logs : “GET /pulp/content/pulp-data/repodata/repomd.xml HTTP/1.1” 403 74 “-” “libdnf (Amazon Linux 2023; generic; Linux.x86_64)”

I tried specifying the header in command as well

Command : curl -k --http1.1 -H “X-CLIENT-CERT: $(cat /root/pki/issued/yum-client.crt)” --cert /root/pki/issued/yum-client.pem --key /root/pki/private/yum-client.key https://example.com/pulp/content/pulp-data/

Output

<head><title>400 Bad Request</title></head>

<body>

<center><h1>400 Bad Request</h1></center>

<hr><center>nginx/1.24.0</center>

</body>

</html>

FYI without certguard I am able download the content.
Any help is appreciated. Thank You!

#14

You are using a reverse proxy to do the tls termination, right?

Have you seen this?
https://docs.pulpproject.org/pulp_certguard/reverse_proxy_config.html

#15

Yes, I am using reverse proxy and I am following the same documentation you referred.
please take a look
server {
listen 8080 default_server deferred;
listen [::]:8080 default_server deferred;
server_name example.com;
#root /usr/share/nginx/html;
#root /var/lib/pulp/static;

	location /pulp/content/ {
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
	    proxy_set_header X-CLIENT-CERT $ssl_client_escaped_cert;
            proxy_set_header Host $http_host;
	    disable_symlinks off;
            proxy_set_header X-Real-IP $remote_addr;
	    proxy_set_header X-NginX-Proxy true;
	    # we don't want nginx trying to do something clever with
            # redirects, we set the Host: header above already.
            #proxy_redirect off;
            proxy_pass http://localhost:24816;
	    #proxy_redirect http://3.x.x.x:8080 http://localhost:24816; 
	    proxy_redirect https://$server_name http://localhost:24816;
	    #proxy_redirect http://localhost:24816 https://$server_name/;
        }

        location /pulp/api/v3/ {
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
	    
            proxy_set_header Host $http_host;
            # we don't want nginx trying to do something clever with
            # redirects, we set the Host: header above already.
            proxy_redirect off;
            proxy_pass http://localhost:24817;
        }
#16

Looking at the docs, it seems like you need to set ssl_verify_client in the nginx config to something meaningful.
Guessing here: When set to optional, nginx needs to know about the ca cert. So basically you can limit the number of ca’s that can be used.
When set to optional_no_ca, I hope nginx will still verify the connection against the user provided cert, but pulp-certguard is in charge of connecting the dots with the ca in the specific content guard. Scanning the code, it looks to me like that is exactly what’s happening here:

DISCLAIMER: I am not sure.

#17

Even if we don’t use Nginx and directly try to access the endpoint I am still facing the same issue. It seems to be present regardless of the access method, so I think its not related to Nginx configuration. Please check the below outputs I performed on Pulp server i.e. with and without certguard.
(AmzLinux is not protected with certguard with base_path : pulp-data1)
(AmazonLinux-2023-Dist is protected with certguard with base_path : pulp-data)

   [root@ip-172-x-x-x ~]# pulp rpm distribution list
    [
     {
      "pulp_href": "/pulp/api/v3/distributions/rpm/rpm/018a5f19-632c-7c95-b969-50680781a6b6/",
      "pulp_created": "2023-09-04T07:30:13.933861Z",
      "base_path": "pulp-data1",
      "base_url": "http://localhost:24816/pulp/content/pulp-data1/",
      "content_guard": null,
      "hidden": false,
      "pulp_labels": {},
      "name": "AmzLinux",
      "repository": null,
      "publication": "/pulp/api/v3/publications/rpm/rpm/018a40ec-a601-7276-91e6-bbe8e16242c1/"
     },
     {
      "pulp_href": "/pulp/api/v3/distributions/rpm/rpm/018a40f6-c296-772f-ac62-25de743cbfb8/",
      "pulp_created": "2023-08-29T11:03:48.119538Z",
      "base_path": "pulp-data",
      "base_url": "http://localhost:24816/pulp/content/pulp-data/",
      "content_guard": "/pulp/api/v3/contentguards/certguard/x509/018a4f81-bfcb-7451-bcdd-52a627b56867/",
      "hidden": false,
      "pulp_labels": {},
      "name": "AmazonLinux-2023-Dist",
      "repository": null,
      "publication": "/pulp/api/v3/publications/rpm/rpm/018a40ec-a601-7276-91e6-bbe8e16242c1/"
     }
    ]
    [root@ip-172-x-x-x ~]# curl -k --cert /root/easy-rsa/easyrsa3/pki/issued/yum-client.pem --key /root/easy-rsa/easyrsa3/pki/private/yum-client.key http://localhost:24816/pulp/content/pulp-data1/repodata/
    <html>
    <head><title>Index of /pulp/content/pulp-data1/repodata/</title></head>
    <body bgcolor="white">
    <h1>Index of /pulp/content/pulp-data1/repodata/</h1>
    <hr><pre><a href="../">../</a>
    <a href="0a65d1a5c12377cf8d3c69fa1c964f982ce1e42e4f2b8050ed4500907206d1e4-other.xml.gz">0a65d1a5c12377cf8d3c69fa1c964f982ce1e42e4f2b8050ed4500907206d1e4-other.xml.gz</a>            29-Aug-2023 10:53 1.2 MB
    <a href="551609a9e510d48daefe8b21ad54e4b74f5440fed17d9fc66b30955bbf052f65-primary.xml.gz">551609a9e510d48daefe8b21ad54e4b74f5440fed17d9fc66b30955bbf052f65-primary.xml.gz</a>           29-Aug-2023 10:53 3.3 MB
    <a href="8c0374e5f54a36971138cd8fbf9a3f8d8b81ec29dc6ad92ab7bca711a3250ca1-filelists.xml.gz">8c0374e5f54a36971138cd8fbf9a3f8d8b81ec29dc6ad92ab7bca711a3250ca1-filelists.xml.gz</a>          29-Aug-2023 10:53 7.2 MB
    <a href="9d34618009a58ec6cb82f8a7375da55b7c33ba034603692b10fcd49a55b11b5c-updateinfo.xml.gz">9d34618009a58ec6cb82f8a7375da55b7c33ba034603692b10fcd49a55b11b5c-updateinfo.xml.gz</a>         29-Aug-2023 10:53 122.1 kB
    <a href="ab6bdd8df14cf963e9e3b434c1a65e0bdae1ce6058ab5bbc8e3975bc533c74f4-comps.xml">ab6bdd8df14cf963e9e3b434c1a65e0bdae1ce6058ab5bbc8e3975bc533c74f4-comps.xml</a>             29-Aug-2023 10:53 41.3 kB
    <a href="repomd.xml">repomd.xml</a>                                             29-Aug-2023 10:53 2.3 kB
    </pre><hr></body>
    </html>
[root@ip-172-x-x-x ~]# curl -k --cert /root/easy-rsa/easyrsa3/pki/issued/yum-client.pem --key /root/easy-rsa/easyrsa3/pki/private/yum-client.key http://localhost:24816/pulp/content/pulp-data/repodata/
    403: A client certificate was not received via the `X-CLIENT-CERT` header.
#18

Without a reverse proxy, there is no ssl/tls and so no way a client cert will ever be used. This is absolutely expected.
Pretty sure you can only use the certguard via https.

#19

Hi there, this problem still persists. My current flow is something like this:
Client-> AWS-ELB(https ACM certs)-> nginx(http 8080)->Pulp content(http://localhost:24816/)
As I mentioned previously, without Content Guard, everything is working fine. If you can provide something like how to configure the SSL certificates or nginx configuration, that’ll be very helpful.
Any help is appreciated. Thank you!

#20

I have no idea other than nginx must terminate the ssh handshake and pass the cert data along in that special handler. Any chance there is someone terminating the ssl even before nginx?

There are some debugging hints in the doc:
https://docs.pulpproject.org/pulp_certguard/debugging.html