Getting '404 Not found' while creating pulp content guard

tushar · September 13, 2023, 8:19am

To answer your question, As per the document provided I have removed the load balancer layer and now the scenario is Client → nginx(http 8080)->Pulp content(http://localhost:24816/) but I see “X-CLIENT-CERT` header not received” error.

tushar · September 13, 2023, 8:21am

curl -k https://test.com/pulp/content/pulp-data1/ -H “SSL-CLIENT-CERTIFICATE:$(cat test.pem | tr -d ‘\n’)”
403: A client certificate was not received via the X-CLIENT-CERT header.

from logs I see below error
Path: /pulp/content/pulp-data1/ not permitted by guard: “AL23certguard” reason: A client certificate was not received via the X-CLIENT-CERT header.

This is what my nginx.conf looks like
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name pulp3.test.com;

   location /pulp/content/ {
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-CLIENT-CERT "";
       proxy_set_header X-CLIENT-CERT $ssl_client_escaped_cert;
       proxy_set_header Host $http_host;
       disable_symlinks off;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-NginX-Proxy true;
       proxy_redirect off;
       proxy_pass http://localhost:24816;
   }

   location /pulp/api/v3/ {
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;

       proxy_set_header Host $http_host;
       # we don't want nginx trying to do something clever with
       # redirects, we set the Host: header above already.
       proxy_redirect off;
       proxy_pass http://localhost:24817;
   }
   ssl_certificate "/root/easy-rsa/easyrsa3/pki/issued/pulp3.crt";
   ssl_certificate_key "/root/easy-rsa/easyrsa3/pki/private/pulp3.key";
   ssl_client_certificate "/root/easy-rsa/easyrsa3/pki/ca.crt";
   ssl_verify_client optional;
   include /etc/nginx/default.d/*.conf;
   error_page 404 /404.html;
   location = /404.html {
   }
   error_page 500 502 503 504 /50x.html;
   location = /50x.html {
   }

}
Please do share your thought. Thank you!

x9c4 · September 13, 2023, 9:20am

I’m guessing, but i would remove the line proxy_set_header X-CLIENT-CERT ""; and add ssl_verify_client=optional_no_ca.

tushar · September 13, 2023, 11:31am

I changed it from ssl_verify_client=optional to ssl_verify_client=optional_no_ca but I am hitting same issue. I also removed the line proxy_set_header X-CLIENT-CERT "";I am testing on the system.

root@ip-172-31-83-17 pki]# curl -vvv -k https://pulp3.xxxx-sandbox.xxxx.com/pulp/content/pulp-data/ -H “SSL-CLIENT-CERTIFICATE:$(cat pulp3.xxxx-sandbox.xxxx.com.pem | tr -d ‘\n’)”

Trying 172.31.83.17:443…

Connected to pulp3.xxxx-sandbox.xxxx.com (172.31.83.17) port 443 (#0)

ALPN: offers h2,http/1.1

TLSv1.3 (OUT), TLS handshake, Client hello (1):

TLSv1.3 (IN), TLS handshake, Server hello (2):

TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):

TLSv1.3 (IN), TLS handshake, Request CERT (13):

TLSv1.3 (IN), TLS handshake, Certificate (11):

TLSv1.3 (IN), TLS handshake, CERT verify (15):

TLSv1.3 (IN), TLS handshake, Finished (20):

TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):

TLSv1.3 (OUT), TLS handshake, Certificate (11):

TLSv1.3 (OUT), TLS handshake, Finished (20):

SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384

ALPN: server accepted h2

Server certificate:

subject: CN=pulp3.xxxx-sandbox.xxxx.com

start date: Sep 11 19:07:15 2023 GMT

expire date: Dec 14 19:07:15 2025 GMT

issuer: CN=pulp3.xxxx-sandbox.xxxx.com

SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.

using HTTP/2

h2h3 [:method: GET]

h2h3 [:path: /pulp/content/pulp-data/]

h2h3 [:scheme: https]

h2h3 [:authority: pulp3.xxxx-sandbox.xxxx.com]

h2h3 [user-agent: curl/8.0.1]

h2h3 [accept: /]

h2h3 [ssl-client-certificate: -----BEGIN CERTIFICATE-----MIICRTCCAcqgAwIBAgIUWmDnBN3CxjygmYFiFAUtwg783iQwCgYIKoZIzj0EAwQwLjEsMCoGA1UEAwwjcHVscDMuaW5jcmVkaWJsZXMtc2FuZGJveC5ib29taS5jb20wHhcNMjMwOTExMTg1MzU0WhcNMzMwOTA4MTg1MzU0WjAuMSwwKgYDVQQDDCNwdWxwMy5pbmNyZWRpYmxlcy1zYW5kYm94LmJvb21pLmNvbTB2MBAGByqGSM49AgEGBSuBBAAiA2IABIJmBYqUvPxAtZSuJkliLCkPhSWooRfsCvvMckPh1IS2qlYwQ5GLE47EcBGxm9saxLjH28otIl73PJS1j0rmDiYZxH0eO+POwmoYWp91imKgLTz4l0VhHrDwXrR1SRyCIKOBqDCBpTAMBgNVHRMEBTADAQH/MB0GA1UdDgQWBBTflZAsTpIpRRsWk2LEVnU6konp7jBpBgNVHSMEYjBggBTflZAsTpIpRRsWk2LEVnU6konp7qEypDAwLjEsMCoGA1UEAwwjcHVscDMuaW5jcmVkaWJsZXMtc2FuZGJveC5ib29taS5jb22CFFpg5wTdwsY8oJmBYhQFLcIO/N4kMAsGA1UdDwQEAwIBBjAKBggqhkjOPQQDBANpADBmAjEAqfrNx/GEOneiuC1dVyh2si3zsExoEINY90awV4ppBR8M75bqyoXUcLbqhrSWPzdzAjEA89Ybq6xQtXZqjp+KI61IwKpFzcMqVd2mTMBwG3gyhx8cGGDhLWi3vU3EJsTCXGCN-----END CERTIFICATE-----]

Using Stream ID: 1 (easy handle 0x5633012dbe80)

GET /pulp/content/pulp-data/ HTTP/2
Host: pulp3.xxxx.com
user-agent: curl/8.0.1
accept: /
ssl-client-certificate:-----BEGIN CERTIFICATE-----MIICRTCCAcqgAwIBAgIUWmDnBN3CxjygmYFiFAUtwg783iQwCgYIKoZIzj0EAwQwLjEsMCoGA1UEAwwjcHVscDMuaW5jcmVkaWJsZXMtc2FuZGJveC5ib29taS5jb20wHhcNMjMwOTExMTg1MzU0WhcNMzMwOTA4MTg1MzU0WjAuMSwwKgYDVQQDDCNwdWxwMy5pbmNyZWRpYmxlcy1zYW5kYm94LmJvb21pLmNvbTB2MBAGByqGSM49AgEGBSuBBAAiA2IABIJmBYqUvPxAtZSuJkliLCkPhSWooRfsCvvMckPh1IS2qlYwQ5GLE47EcBGxm9saxLjH28otIl73PJS1j0rmDiYZxH0eO+POwmoYWp91imKgLTz4l0VhHrDwXrR1SRyCIKOBqDCBpTAMBgNVHRMEBTADAQH/MB0GA1UdDgQWBBTflZAsTpIpRRsWk2LEVnU6konp7jBpBgNVHSMEYjBggBTflZAsTpIpRRsWk2LEVnU6konp7qEypDAwLjEsMCoGA1UEAwwjcHVscDMuaW5jcmVkaWJsZXMtc2FuZGJveC5ib29taS5jb22CFFpg5wTdwsY8oJmBYhQFLcIO/N4kMAsGA1UdDwQEAwIBBjAKBggqhkjOPQQDBANpADBmAjEAqfrNx/GEOneiuC1dVyh2si3zsExoEINY90awV4ppBR8M75bqyoXUcLbqhrSWPzdzAjEA89Ybq6xQtXZqjp+KI61IwKpFzcMqVd2mTMBwG3gyhx8cGGDhLWi3vU3EJsTCXGCN-----END CERTIFICATE-----

TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

old SSL session ID is stale, removing
< HTTP/2 403
< server: nginx/1.24.0
< date: Wed, 13 Sep 2023 11:19:16 GMT
< content-type: text/plain; charset=utf-8
< content-length: 74
<

Connection #0 to host pulp3.xxxx-sandbox.xxxx.com left intact

FYI :

grep -nri ssl_verify_client nginx.conf
87: ssl_verify_client optional_no_ca;

grep -nri X-CLIENT-CERT nginx.conf
63: proxy_set_header X-CLIENT-CERT $ssl_client_escaped_cert;

from logs I see this error

ep 13 11:17:58 ip-172-31-83-17 gunicorn[2648]: pulp [None]: pulpcore.content.handler:DEBUG: Path: /pulp/content/pulp-data/ not permitted by guard: "AL23Guard" reason: A client certificate was not received via the `X-CLIENT-CERT` header.
Sep 13 11:17:58 ip-172-31-83-17 gunicorn[2648]: 127.0.0.1 [13/Sep/2023:11:17:58 +0000] "GET /pulp/content/pulp-data/ HTTP/1.0" 403 292 "-" "curl/8.0.1"

x9c4 · September 13, 2023, 1:41pm

Sorry, i have no more ideas. This is the configuration file we use for pulp-in-one-container:

github.com

pulp/pulp-oci-images/blob/latest/images/s6_assets/ssl_nginx.conf

# TODO: Support IPv6.
# TODO: Maybe serve multiple `location`s, not just one.

# The "nginx" package on fedora creates this user and group.
user nginx nginx;
# Gunicorn docs suggest this value.
worker_processes 1;
daemon off;
events {
    worker_connections 1024;  # increase if you have lots of clients
    accept_mutex off;  # set to 'on' if nginx worker_processes > 1
}

http {
    include mime.types;
    # fallback in case we can't determine a type
    default_type application/octet-stream;
    sendfile on;

    # If left at the default of 1024, nginx emits a warning about being unable

This file has been truncated. show original

And this is the last version in our deprecated installer:

github.com

pulp/pulp_installer/blob/main/roles/pulp_webserver/templates/nginx.conf.j2

# TODO: Maybe serve multiple `location`s, not just one.

{% if __pulp_os_family == 'RedHat' %}
# The "nginx" package on fedora creates this user and group.
user nginx nginx;
{% endif %}
# Gunicorn docs suggest this value.
worker_processes 1;
events {
    worker_connections 1024;  # increase if you have lots of clients
    accept_mutex off;  # set to 'on' if nginx worker_processes > 1
}

http {
    include mime.types;
    # fallback in case we can't determine a type
    default_type application/octet-stream;
    server_tokens off;
    sendfile on;

This file has been truncated. show original

Maybe you can spot a clue there.

tushar · September 14, 2023, 11:53am

Hi @x9c4, I was able to resolve the issue by tweaking some nginx configurations. But the challenge I am facing right now is with “dnf update”. The SSL certificates which I have created using openssl are self-signed. But I am able to download the certificates using wget.

wget --certificate=/etc/pulp/certs/pulp_webserver.crt --private-key=/etc/pulp/certs/pulp_webserver.key https://pulp3.xxxx-sandbox.xxxx.com/pulp/content/pulp-data1/repodata/repomd.xml
–2023-09-14 07:58:08-- https://pulp3.xxxx-sandbox.xxxx.com/pulp/content/pulp-data1/repodata/repomd.xml
Resolving pulp3.xxxx-sandbox.xxxx.com (pulp3.xxxx-sandbox.xxxx.com)… 172.31.83.17
Connecting to pulp3.xxxx-sandbox.xxxx.com (pulp3.xxxx-sandbox.xxxx.com)|172.31.83.17|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 2343 (2.3K) [text/xml]
Saving to: ‘repomd.xml’

repomd.xml 100%[=================================================================================================>] 2.29K --.-KB/s in 0s

2023-09-14 07:58:08 (74.7 MB/s) - ‘repomd.xml’ saved [2343/2343]

Error I am getting

dnf update
AmazonLinux-2023-Dist 0.0 B/s | 0 B 00:00
Errors during downloading metadata for repository ‘AmazonLinux-2023-Dist’:

Curl error (60): SSL peer certificate or SSH remote key was not OK for https://pulp3.xxxx-sandbox.xxxx.com/pulp/content/pulp-data1/repodata/repomd.xml [SSL certificate problem: self-signed certificate]
Error: Failed to download metadata for repo ‘AmazonLinux-2023-Dist’: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
Ignoring repositories: AmazonLinux-2023-Dist

FYI

openssl s_client -showcerts -servername pulp3.xxxx-sandbox.xxxx.com -connect pulp3.xxxx-sandbox.xxxx.com:443 > cacert.pem
depth=0 CN = pulp3.xxxx-sandbox.xxxx.com
verify error:num=18:self-signed certificate
verify return:1
depth=0 CN = pulp3.xxxx-sandbox.xxxx.com
verify return:1

This is what my yum file looks like

[AmazonLinux-2023-Dist]

name=AmazonLinux-2023-Dist
enabled=1
baseurl=https://pulp3.xxxx-sandbox.xxxx.com/pulp/content/pulp-data1/
gpgcheck=0
repo_gpgcheck=0
sslverify=1

x509 Auth

sslclientcert=/etc/pulp/certs/pulp_webserver.crt
sslclientkey=/etc/pulp/certs/pulp_webserver.key

I greatly appreciate your time, @x9c4. You’ve been a great help.
Thank You!

x9c4 · September 14, 2023, 2:47pm

Great to hear! And you are welcome.
Do you know which setting was the game changer? Would you by any chance be willing to improve the docs we have here? https://github.com/pulp/pulp-certguard/tree/main/docs

I’m not quite sure I understand your new roadblocker… I’d say it’s not recommended to use the webserver cert for downloading. Is that dnf complaining about self signed? Then probably you need to whip up a tiny CA…