Getting '404 Not found' while creating pulp content guard

I see. Then you should add pulp-certguard to the pip install command (along with every other plugin you expect to use). And do not forget to run the migrations afterwards and restart the services.

BTW: Step 16 sounds wrong to me. If you installed the systemd files, just skip it.

I can see pulp-certguard in versions list. Maybe because I did not restarted the services after installing the pulp-certguard. I’ll try to create content guard again, will keep you posted. Thank You!
“versions”: [
{
“component”: “core”,
“version”: “3.31.0”,
“package”: “pulpcore”,
“domain_compatible”: true
},
{
“component”: “rpm”,
“version”: “3.22.3”,
“package”: “pulp-rpm”,
“domain_compatible”: true
},
{
“component”: “container”,
“version”: “2.16.0”,
“package”: “pulp-container”,
“domain_compatible”: false
},
{
“component”: “file”,
“version”: “1.14.4”,
“package”: “pulp-file”,
“domain_compatible”: true
},
{
“component”: “certguard”,
“version”: “1.6.5”,
“package”: “pulp-certguard”,
“domain_compatible”: true
}
],

1 Like

Hi @x9c4, I was able to create the content guard and Protect the Distribution with the X509CertGuard. Appreciate your help. Thank you for your time!

2 Likes

You are welcome!
How was your experience with the pypi installation? Did you go with the systemd files?
I think I can say none of us developers has done such an installation in quite some while, as we all use the containerized method these days.

1 Like

The PyPI installation process went well, won’t say it was buttery smooth, but most of the instructions are straightforward. Yes, I did choose to go with the systemd files for managing the service. I faced some challenges in pulpcore-workers service, but I was able to solve them. If you have any tips or insights on how you’ve implemented containerization effectively, I’d love to hear them. Thanks again for your assistance.

3 Likes

Hi @x9c4, sorry to bother you again. As I mentioned previously I was able to create a content guard and protect my distribution but when I tried to access it from client side I am getting one error.
Command : curl -k --cert /root/pki/issued/yum-client.pem --key /root/pki/private/yum-client.key https://example.com/pulp/content/pulp-data/

Output : 403: A client certificate was not received via the X-CLIENT-CERT header.

Logs : “GET /pulp/content/pulp-data/repodata/repomd.xml HTTP/1.1” 403 74 “-” “libdnf (Amazon Linux 2023; generic; Linux.x86_64)”

I tried specifying the header in command as well

Command : curl -k --http1.1 -H “X-CLIENT-CERT: $(cat /root/pki/issued/yum-client.crt)” --cert /root/pki/issued/yum-client.pem --key /root/pki/private/yum-client.key https://example.com/pulp/content/pulp-data/

Output

<head><title>400 Bad Request</title></head>

<body>

<center><h1>400 Bad Request</h1></center>

<hr><center>nginx/1.24.0</center>

</body>

</html>

FYI without certguard I am able download the content.
Any help is appreciated. Thank You!

You are using a reverse proxy to do the tls termination, right?

Have you seen this?
https://docs.pulpproject.org/pulp_certguard/reverse_proxy_config.html

Yes, I am using reverse proxy and I am following the same documentation you referred.
please take a look
server {
listen 8080 default_server deferred;
listen [::]:8080 default_server deferred;
server_name example.com;
#root /usr/share/nginx/html;
#root /var/lib/pulp/static;

	location /pulp/content/ {
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
	    proxy_set_header X-CLIENT-CERT $ssl_client_escaped_cert;
            proxy_set_header Host $http_host;
	    disable_symlinks off;
            proxy_set_header X-Real-IP $remote_addr;
	    proxy_set_header X-NginX-Proxy true;
	    # we don't want nginx trying to do something clever with
            # redirects, we set the Host: header above already.
            #proxy_redirect off;
            proxy_pass http://localhost:24816;
	    #proxy_redirect http://3.x.x.x:8080 http://localhost:24816; 
	    proxy_redirect https://$server_name http://localhost:24816;
	    #proxy_redirect http://localhost:24816 https://$server_name/;
        }

        location /pulp/api/v3/ {
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
	    
            proxy_set_header Host $http_host;
            # we don't want nginx trying to do something clever with
            # redirects, we set the Host: header above already.
            proxy_redirect off;
            proxy_pass http://localhost:24817;
        }

Looking at the docs, it seems like you need to set ssl_verify_client in the nginx config to something meaningful.
Guessing here: When set to optional, nginx needs to know about the ca cert. So basically you can limit the number of ca’s that can be used.
When set to optional_no_ca, I hope nginx will still verify the connection against the user provided cert, but pulp-certguard is in charge of connecting the dots with the ca in the specific content guard. Scanning the code, it looks to me like that is exactly what’s happening here:

DISCLAIMER: I am not sure.

Even if we don’t use Nginx and directly try to access the endpoint I am still facing the same issue. It seems to be present regardless of the access method, so I think its not related to Nginx configuration. Please check the below outputs I performed on Pulp server i.e. with and without certguard.
(AmzLinux is not protected with certguard with base_path : pulp-data1)
(AmazonLinux-2023-Dist is protected with certguard with base_path : pulp-data)

   [root@ip-172-x-x-x ~]# pulp rpm distribution list
    [
     {
      "pulp_href": "/pulp/api/v3/distributions/rpm/rpm/018a5f19-632c-7c95-b969-50680781a6b6/",
      "pulp_created": "2023-09-04T07:30:13.933861Z",
      "base_path": "pulp-data1",
      "base_url": "http://localhost:24816/pulp/content/pulp-data1/",
      "content_guard": null,
      "hidden": false,
      "pulp_labels": {},
      "name": "AmzLinux",
      "repository": null,
      "publication": "/pulp/api/v3/publications/rpm/rpm/018a40ec-a601-7276-91e6-bbe8e16242c1/"
     },
     {
      "pulp_href": "/pulp/api/v3/distributions/rpm/rpm/018a40f6-c296-772f-ac62-25de743cbfb8/",
      "pulp_created": "2023-08-29T11:03:48.119538Z",
      "base_path": "pulp-data",
      "base_url": "http://localhost:24816/pulp/content/pulp-data/",
      "content_guard": "/pulp/api/v3/contentguards/certguard/x509/018a4f81-bfcb-7451-bcdd-52a627b56867/",
      "hidden": false,
      "pulp_labels": {},
      "name": "AmazonLinux-2023-Dist",
      "repository": null,
      "publication": "/pulp/api/v3/publications/rpm/rpm/018a40ec-a601-7276-91e6-bbe8e16242c1/"
     }
    ]
    [root@ip-172-x-x-x ~]# curl -k --cert /root/easy-rsa/easyrsa3/pki/issued/yum-client.pem --key /root/easy-rsa/easyrsa3/pki/private/yum-client.key http://localhost:24816/pulp/content/pulp-data1/repodata/
    <html>
    <head><title>Index of /pulp/content/pulp-data1/repodata/</title></head>
    <body bgcolor="white">
    <h1>Index of /pulp/content/pulp-data1/repodata/</h1>
    <hr><pre><a href="../">../</a>
    <a href="0a65d1a5c12377cf8d3c69fa1c964f982ce1e42e4f2b8050ed4500907206d1e4-other.xml.gz">0a65d1a5c12377cf8d3c69fa1c964f982ce1e42e4f2b8050ed4500907206d1e4-other.xml.gz</a>            29-Aug-2023 10:53 1.2 MB
    <a href="551609a9e510d48daefe8b21ad54e4b74f5440fed17d9fc66b30955bbf052f65-primary.xml.gz">551609a9e510d48daefe8b21ad54e4b74f5440fed17d9fc66b30955bbf052f65-primary.xml.gz</a>           29-Aug-2023 10:53 3.3 MB
    <a href="8c0374e5f54a36971138cd8fbf9a3f8d8b81ec29dc6ad92ab7bca711a3250ca1-filelists.xml.gz">8c0374e5f54a36971138cd8fbf9a3f8d8b81ec29dc6ad92ab7bca711a3250ca1-filelists.xml.gz</a>          29-Aug-2023 10:53 7.2 MB
    <a href="9d34618009a58ec6cb82f8a7375da55b7c33ba034603692b10fcd49a55b11b5c-updateinfo.xml.gz">9d34618009a58ec6cb82f8a7375da55b7c33ba034603692b10fcd49a55b11b5c-updateinfo.xml.gz</a>         29-Aug-2023 10:53 122.1 kB
    <a href="ab6bdd8df14cf963e9e3b434c1a65e0bdae1ce6058ab5bbc8e3975bc533c74f4-comps.xml">ab6bdd8df14cf963e9e3b434c1a65e0bdae1ce6058ab5bbc8e3975bc533c74f4-comps.xml</a>             29-Aug-2023 10:53 41.3 kB
    <a href="repomd.xml">repomd.xml</a>                                             29-Aug-2023 10:53 2.3 kB
    </pre><hr></body>
    </html>
[root@ip-172-x-x-x ~]# curl -k --cert /root/easy-rsa/easyrsa3/pki/issued/yum-client.pem --key /root/easy-rsa/easyrsa3/pki/private/yum-client.key http://localhost:24816/pulp/content/pulp-data/repodata/
    403: A client certificate was not received via the `X-CLIENT-CERT` header.

Without a reverse proxy, there is no ssl/tls and so no way a client cert will ever be used. This is absolutely expected.
Pretty sure you can only use the certguard via https.

Hi there, this problem still persists. My current flow is something like this:
Client-> AWS-ELB(https ACM certs)-> nginx(http 8080)->Pulp content(http://localhost:24816/)
As I mentioned previously, without Content Guard, everything is working fine. If you can provide something like how to configure the SSL certificates or nginx configuration, that’ll be very helpful.
Any help is appreciated. Thank you!

I have no idea other than nginx must terminate the ssh handshake and pass the cert data along in that special handler. Any chance there is someone terminating the ssl even before nginx?

There are some debugging hints in the doc:
https://docs.pulpproject.org/pulp_certguard/debugging.html

To answer your question, As per the document provided I have removed the load balancer layer and now the scenario is Client → nginx(http 8080)->Pulp content(http://localhost:24816/) but I see “X-CLIENT-CERT` header not received” error.

curl -k https://test.com/pulp/content/pulp-data1/ -H “SSL-CLIENT-CERTIFICATE:$(cat test.pem | tr -d ‘\n’)”
403: A client certificate was not received via the X-CLIENT-CERT header.

from logs I see below error
Path: /pulp/content/pulp-data1/ not permitted by guard: “AL23certguard” reason: A client certificate was not received via the X-CLIENT-CERT header.

This is what my nginx.conf looks like
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name pulp3.test.com;

   location /pulp/content/ {
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-CLIENT-CERT "";
       proxy_set_header X-CLIENT-CERT $ssl_client_escaped_cert;
       proxy_set_header Host $http_host;
       disable_symlinks off;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-NginX-Proxy true;
       proxy_redirect off;
       proxy_pass http://localhost:24816;
   }

   location /pulp/api/v3/ {
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;

       proxy_set_header Host $http_host;
       # we don't want nginx trying to do something clever with
       # redirects, we set the Host: header above already.
       proxy_redirect off;
       proxy_pass http://localhost:24817;
   }
   ssl_certificate "/root/easy-rsa/easyrsa3/pki/issued/pulp3.crt";
   ssl_certificate_key "/root/easy-rsa/easyrsa3/pki/private/pulp3.key";
   ssl_client_certificate "/root/easy-rsa/easyrsa3/pki/ca.crt";
   ssl_verify_client optional;
   include /etc/nginx/default.d/*.conf;
   error_page 404 /404.html;
   location = /404.html {
   }
   error_page 500 502 503 504 /50x.html;
   location = /50x.html {
   }

}
Please do share your thought. Thank you!

I’m guessing, but i would remove the line proxy_set_header X-CLIENT-CERT ""; and add ssl_verify_client=optional_no_ca.

I changed it from ssl_verify_client=optional to ssl_verify_client=optional_no_ca but I am hitting same issue. I also removed the line proxy_set_header X-CLIENT-CERT "";I am testing on the system.

root@ip-172-31-83-17 pki]# curl -vvv -k https://pulp3.xxxx-sandbox.xxxx.com/pulp/content/pulp-data/ -H “SSL-CLIENT-CERTIFICATE:$(cat pulp3.xxxx-sandbox.xxxx.com.pem | tr -d ‘\n’)”

  • Trying 172.31.83.17:443…
  • Connected to pulp3.xxxx-sandbox.xxxx.com (172.31.83.17) port 443 (#0)
  • ALPN: offers h2,http/1.1
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Request CERT (13):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.3 (OUT), TLS handshake, Certificate (11):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  • ALPN: server accepted h2
  • Server certificate:
  • subject: CN=pulp3.xxxx-sandbox.xxxx.com
  • start date: Sep 11 19:07:15 2023 GMT
  • expire date: Dec 14 19:07:15 2025 GMT
  • issuer: CN=pulp3.xxxx-sandbox.xxxx.com
  • SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
  • using HTTP/2
  • h2h3 [:method: GET]
  • h2h3 [:path: /pulp/content/pulp-data/]
  • h2h3 [:scheme: https]
  • h2h3 [:authority: pulp3.xxxx-sandbox.xxxx.com]
  • h2h3 [user-agent: curl/8.0.1]
  • h2h3 [accept: /]
  • h2h3 [ssl-client-certificate: -----BEGIN CERTIFICATE-----MIICRTCCAcqgAwIBAgIUWmDnBN3CxjygmYFiFAUtwg783iQwCgYIKoZIzj0EAwQwLjEsMCoGA1UEAwwjcHVscDMuaW5jcmVkaWJsZXMtc2FuZGJveC5ib29taS5jb20wHhcNMjMwOTExMTg1MzU0WhcNMzMwOTA4MTg1MzU0WjAuMSwwKgYDVQQDDCNwdWxwMy5pbmNyZWRpYmxlcy1zYW5kYm94LmJvb21pLmNvbTB2MBAGByqGSM49AgEGBSuBBAAiA2IABIJmBYqUvPxAtZSuJkliLCkPhSWooRfsCvvMckPh1IS2qlYwQ5GLE47EcBGxm9saxLjH28otIl73PJS1j0rmDiYZxH0eO+POwmoYWp91imKgLTz4l0VhHrDwXrR1SRyCIKOBqDCBpTAMBgNVHRMEBTADAQH/MB0GA1UdDgQWBBTflZAsTpIpRRsWk2LEVnU6konp7jBpBgNVHSMEYjBggBTflZAsTpIpRRsWk2LEVnU6konp7qEypDAwLjEsMCoGA1UEAwwjcHVscDMuaW5jcmVkaWJsZXMtc2FuZGJveC5ib29taS5jb22CFFpg5wTdwsY8oJmBYhQFLcIO/N4kMAsGA1UdDwQEAwIBBjAKBggqhkjOPQQDBANpADBmAjEAqfrNx/GEOneiuC1dVyh2si3zsExoEINY90awV4ppBR8M75bqyoXUcLbqhrSWPzdzAjEA89Ybq6xQtXZqjp+KI61IwKpFzcMqVd2mTMBwG3gyhx8cGGDhLWi3vU3EJsTCXGCN-----END CERTIFICATE-----]
  • Using Stream ID: 1 (easy handle 0x5633012dbe80)

GET /pulp/content/pulp-data/ HTTP/2
Host: pulp3.xxxx.com
user-agent: curl/8.0.1
accept: /
ssl-client-certificate:-----BEGIN CERTIFICATE-----MIICRTCCAcqgAwIBAgIUWmDnBN3CxjygmYFiFAUtwg783iQwCgYIKoZIzj0EAwQwLjEsMCoGA1UEAwwjcHVscDMuaW5jcmVkaWJsZXMtc2FuZGJveC5ib29taS5jb20wHhcNMjMwOTExMTg1MzU0WhcNMzMwOTA4MTg1MzU0WjAuMSwwKgYDVQQDDCNwdWxwMy5pbmNyZWRpYmxlcy1zYW5kYm94LmJvb21pLmNvbTB2MBAGByqGSM49AgEGBSuBBAAiA2IABIJmBYqUvPxAtZSuJkliLCkPhSWooRfsCvvMckPh1IS2qlYwQ5GLE47EcBGxm9saxLjH28otIl73PJS1j0rmDiYZxH0eO+POwmoYWp91imKgLTz4l0VhHrDwXrR1SRyCIKOBqDCBpTAMBgNVHRMEBTADAQH/MB0GA1UdDgQWBBTflZAsTpIpRRsWk2LEVnU6konp7jBpBgNVHSMEYjBggBTflZAsTpIpRRsWk2LEVnU6konp7qEypDAwLjEsMCoGA1UEAwwjcHVscDMuaW5jcmVkaWJsZXMtc2FuZGJveC5ib29taS5jb22CFFpg5wTdwsY8oJmBYhQFLcIO/N4kMAsGA1UdDwQEAwIBBjAKBggqhkjOPQQDBANpADBmAjEAqfrNx/GEOneiuC1dVyh2si3zsExoEINY90awV4ppBR8M75bqyoXUcLbqhrSWPzdzAjEA89Ybq6xQtXZqjp+KI61IwKpFzcMqVd2mTMBwG3gyhx8cGGDhLWi3vU3EJsTCXGCN-----END CERTIFICATE-----

  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • old SSL session ID is stale, removing
    < HTTP/2 403
    < server: nginx/1.24.0
    < date: Wed, 13 Sep 2023 11:19:16 GMT
    < content-type: text/plain; charset=utf-8
    < content-length: 74
    <
  • Connection #0 to host pulp3.xxxx-sandbox.xxxx.com left intact

FYI :

grep -nri ssl_verify_client nginx.conf
87: ssl_verify_client optional_no_ca;

grep -nri X-CLIENT-CERT nginx.conf
63: proxy_set_header X-CLIENT-CERT $ssl_client_escaped_cert;

from logs I see this error

ep 13 11:17:58 ip-172-31-83-17 gunicorn[2648]: pulp [None]: pulpcore.content.handler:DEBUG: Path: /pulp/content/pulp-data/ not permitted by guard: "AL23Guard" reason: A client certificate was not received via the `X-CLIENT-CERT` header.
Sep 13 11:17:58 ip-172-31-83-17 gunicorn[2648]: 127.0.0.1 [13/Sep/2023:11:17:58 +0000] "GET /pulp/content/pulp-data/ HTTP/1.0" 403 292 "-" "curl/8.0.1"

Sorry, i have no more ideas. This is the configuration file we use for pulp-in-one-container:

And this is the last version in our deprecated installer:

Maybe you can spot a clue there.

Hi @x9c4, I was able to resolve the issue by tweaking some nginx configurations. But the challenge I am facing right now is with “dnf update”. The SSL certificates which I have created using openssl are self-signed. But I am able to download the certificates using wget.

wget --certificate=/etc/pulp/certs/pulp_webserver.crt --private-key=/etc/pulp/certs/pulp_webserver.key https://pulp3.xxxx-sandbox.xxxx.com/pulp/content/pulp-data1/repodata/repomd.xml
–2023-09-14 07:58:08-- https://pulp3.xxxx-sandbox.xxxx.com/pulp/content/pulp-data1/repodata/repomd.xml
Resolving pulp3.xxxx-sandbox.xxxx.com (pulp3.xxxx-sandbox.xxxx.com)… 172.31.83.17
Connecting to pulp3.xxxx-sandbox.xxxx.com (pulp3.xxxx-sandbox.xxxx.com)|172.31.83.17|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 2343 (2.3K) [text/xml]
Saving to: ‘repomd.xml’

repomd.xml 100%[=================================================================================================>] 2.29K --.-KB/s in 0s

2023-09-14 07:58:08 (74.7 MB/s) - ‘repomd.xml’ saved [2343/2343]

Error I am getting

dnf update
AmazonLinux-2023-Dist 0.0 B/s | 0 B 00:00
Errors during downloading metadata for repository ‘AmazonLinux-2023-Dist’:

  • Curl error (60): SSL peer certificate or SSH remote key was not OK for https://pulp3.xxxx-sandbox.xxxx.com/pulp/content/pulp-data1/repodata/repomd.xml [SSL certificate problem: self-signed certificate]
    Error: Failed to download metadata for repo ‘AmazonLinux-2023-Dist’: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
    Ignoring repositories: AmazonLinux-2023-Dist

FYI

openssl s_client -showcerts -servername pulp3.xxxx-sandbox.xxxx.com -connect pulp3.xxxx-sandbox.xxxx.com:443 > cacert.pem
depth=0 CN = pulp3.xxxx-sandbox.xxxx.com
verify error:num=18:self-signed certificate
verify return:1
depth=0 CN = pulp3.xxxx-sandbox.xxxx.com
verify return:1

This is what my yum file looks like

[AmazonLinux-2023-Dist]

name=AmazonLinux-2023-Dist
enabled=1
baseurl=https://pulp3.xxxx-sandbox.xxxx.com/pulp/content/pulp-data1/
gpgcheck=0
repo_gpgcheck=0
sslverify=1

x509 Auth

sslclientcert=/etc/pulp/certs/pulp_webserver.crt
sslclientkey=/etc/pulp/certs/pulp_webserver.key

I greatly appreciate your time, @x9c4. You’ve been a great help.
Thank You!

2 Likes

Great to hear! And you are welcome.
Do you know which setting was the game changer? Would you by any chance be willing to improve the docs we have here? https://github.com/pulp/pulp-certguard/tree/main/docs

I’m not quite sure I understand your new roadblocker… I’d say it’s not recommended to use the webserver cert for downloading. Is that dnf complaining about self signed? Then probably you need to whip up a tiny CA…

1 Like