Pulp standalone container (docker) - can't pull docker images through pulp

Support
#1

Problem:
Can’t pull docker images through pulp / can’t sync pulp repo with docker remote repo

Expected outcome:
I should be able to pull a docker image using ‘docker pull’ and see it in my pulp server

Pulpcore version:
I’m using the quickstart tutorial and the pulp/pulp image via docker. I’ve installed the pulp CLI as well. pulp status returns a pulp core version of 3.54.1

Pulp plugins installed and their versions:
Stock versions from the pulp/pulp container:
“version”: “3.54.1”, “package”: “pulpcore”
“version”: “0.22.0”, “package”: “pulp-ansible”
“version”: “2.20.0”, “package”: “pulp-container”
“version”: “3.3.0”, “package”: “pulp_deb”
“version”: “0.6.1”, “package”: “pulp-gem”
“version”: “0.8.1”, “package”: “pulp-maven”
“version”: “2.4.1”, “package”: “pulp-ostree”
“version”: “3.12.1”, “package”: “pulp-python”
“version”: “3.27.1”, “package”: “pulp-rpm”
“version”: “3.54.1”, “package”: “pulpcore”
“version”: “3.54.1”, “package”: “pulpcore”

Operating system - distribution and version:
ubuntu 22.04

Other relevant data:
Pulp server is up and running. Pulp CLI can connect. HTTP API is functioning. I’ve tried two ways to get a pulp docker registry going

  1. using the pulp_container tutorial 01-sync-and-host
    pulp container repository create, pulp container remote create both work. pulp container repository sync failed with auth issues
  2. using container pull-through cache support
    I’m able to create the .remotes/container/pull-through and the /distributions/container/pull-through
    running docker pull <pulp_server>:8080/docker-cache/hello-world results in an HTTP 500 internal server error with additional details of “http: server gave HTTP response to HTTPS client”

I’m not sure what I’m missing.
At the bottom of the quickstart guide there are links to “workflows and use cases” (https://docs.pulpproject.org/workflows/index.html) which results in a 404 not found error, as does the “Pulp 3 Content Plugin documentation”

Ideally I’d like to see the container pull-through cache working so that I dont need to support remotes for each image, but I’d be happy to just see any docker images get pulled through pulp

#2

Hello! With regard to the 404 error, we are still working on fixing old references, see Docs: Fix rst leftover errors · Issue #5496 · pulp/pulpcore · GitHub. Please, bear with us.

Would you mind sharing the logs for the (1.) and (2.) bullet-point errors? You could also run into a configuration issue. Have you gone through these steps: Quickstart - Pulp Project? We explicitly mention that the pulp container plugin requires additional settings, like key pair generation. You can learn more at Authentication - Pulp Project.

2 Likes
#3

Hi lubosmj, thanks for the reply!
No worries about the old references, just wanted to point them out.
I’m currently set up to attempt/test the pull-through approach but I documented the repo/remote/distro triple approach and can go back to it with a clean VM later.

I did see the note in the quickstart about " Refer to plugin settings documentation for more before spinning up the container. For instance, the Pulp Container plugin requires key pair generation" but wasn’t sure how to applied. I’ll try that out today and post some additional logs if I’m still seeing issues

#4

This seems to support my theory that the configuration is invalid. You can generate the keys inside a mounted volume and reference them from /etc/pulp/settings.py inside the container. This is what we do in our CI: settings config.

Also, if you do not want to benefit from token authentication, you can disable it with TOKEN_AUTH_DISABLED=True. Then, no other settings are required to be set. Note that the role-based access control is not enabled in this scenario, meaning you cannot manage user access.

#5

Do I still need to do the key pair generation for pulp containers if I disable token authentication? My settings/settings.py file is currently
CONTENT_ORIGIN=‘http://<pulp_server_hostname_here>:8080’
TOKEN_AUTH_DISABLED=True

#6

No, you do not, if you do not care about RBAC.

#7

Ok, for my current proof of concept I dont care about RBAC. So the key pair wont be the issue. I’ll gather docker logs for the pulp container…and now its working. very strange. As I mentioned before I was getting HTTP 500 internal server errors with a complaint about HTTP response to HTTPS client. I’m pretty sure I restarted my pulp container after editing the settings.

Anyway, the pull-through cache appears to be working now. Here’s a simple test and the output:
(docker pull command)

root@PULP_SERVER:/home/ubuntu# docker pull PULP_SERVER.DOMAIN:8080/docker-cache/hello-world
Using default tag: latest
latest: Pulling from docker-cache/hello-world
c1ec31eb5944: Pull complete
Digest: sha256:1408fec50309afee38f3535383f5b09419e6dc0925bc69891e79d84cc4cdcec6
Status: Downloaded newer image for PULP_SERVER.DOMAIN:8080/docker-cache/hello-world:latest
PULP_SERVER.DOMAIN:8080/docker-cache/hello-world:latest

I’ll continue setting up the other repos. Thanks for the reply to my forum post, I appreciate it.

2 Likes
#8

Amazing! We have more fixes coming to the pull-through caching machinery in the next release. Do not hesitate to report issues at Issues · pulp/pulp_container · GitHub.

1 Like
#9

After a bit more testing I have to assume it was related to /etc/docker/daemon.json, insecure registries, and restarting the docker service. I was able to trivially reproduce the problem using another (client) system and setting the insecure registries in the docker config and restarting docker resolved the issue.
Sorry about that.

1 Like
#10

Can you share the logs and complete reproducer with this HTTP/HTTPs error?

#11

Here are notes / instructions I made while attempting to set up the pull-through cache.

  • replace <PULP_SERVER_FQDN> with the fully qualified name of the system you are installing pulp on
  • replace <PULP_SERVER_HOSTNAME> with the unqualified name

Installing Docker

sudo su

Remove any existing docker stuff on the system
for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done
Create a script for prepping the docker install
vim prepare-install-docker.sh

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

Run the script
chmod +x prepare-install-docker.sh
./prepare-install-docker.sh

Install docker components
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Setup insecure registries
vim /etc/docker/daemon.json

{
    "insecure-registries" : ["<PULP_SERVER_FQDN>:8080"]
}

Restart docker
systemctl stop docker
systemctl start docker

Installing Pulp

Install pip3
apt install python3-pip
Install pulp-cli
pip3 install pulp-cli[pygments]
Prepare pulp folders
mkdir -p settings/certs pulp_storage pgsql containers
Prepare pulp settings
vim settings/settings.py

CONTENT_ORIGIN='http://<PULP_SERVER_HOSTNAME>:8080'
TOKEN_AUTH_DISABLED=True

Create standalone pulp script
vim standalone-pulp.sh

docker run --detach \
             --publish 8080:80 \
             --name pulp \
             --volume "$(pwd)/settings":/etc/pulp \
             --volume "$(pwd)/pulp_storage":/var/lib/pulp \
             --volume "$(pwd)/pgsql":/var/lib/pgsql \
             --volume "$(pwd)/containers":/var/lib/containers \
             --device /dev/fuse \
             pulp/pulp

Run pulp

./standalone-pulp
If you need to stop pulp you can do the following
docker stop pulp
docker remove pulp

set pulp admin password

docker exec -it pulp bash -c 'pulpcore-manager reset-admin-password'
Configure pulp cli to use admin password
pulp config create --username admin --base-url http://localhost:8080 --password <password>

install jq

apt install jq

configure container pull-through cache

Install httpie
apt install httpie
Create a script that uses httpie to make http requests
vim configure-pull-through-remote.sh

# Set our base address
BASE_ADDR=<PULP_SERVER_FQDN>:8080

# initialize a pull-through remote (the concept of upstream-name is not applicable here)
PULL_THROUGH_REF=${BASE_ADDR}/pulp/api/v3/remotes/container/pull-through/
PULP_USERNAME=admin
PULP_PASSWORD=password
REGISTRY=https://registry-1.docker.io
REMOTES_PULL_THROUGH_RESULT=$(http --auth=${PULP_USERNAME}:${PULP_PASSWORD} ${PULL_THROUGH_REF} name=docker-cache url=${REGISTRY})
REMOTES_HREF=$(echo "${REMOTES_PULL_THROUGH_RESULT}" | jq -r ".pulp_href")
echo "===REMOTES_PULL_THROUGH_RESULT"
echo "${REMOTES_PULL_THROUGH_RESULT}"
echo "===REMOTES_HREF"
echo "$REMOTES_HREF"
DIST_PULL_THROUGH_RESULT=$(http --auth=${PULP_USERNAME}:${PULP_PASSWORD} ${BASE_ADDR}/pulp/api/v3/distributions/container/pull-through/ remote=${REMOTES_HREF} name=docker-cache base_path=docker-cache)
echo "===DIST_PULL_THROUGH_RESULT"
echo "${DIST_PULL_THROUGH_RESULT}"

Mark the script as executable
chmod +x configure-pull-through-remote.sh

Run the script to configure the pull through remote
./configure-pull-through-remote.sh

Now we can try pulling a docker image through the Pulp pull-through cache
docker login <PULP_SERVER_FQDN>:8080 -u admin -p password
docker pull <PULP_SERVER_FQDN>:8080/docker-cache/hello-world

1 Like
#12

Regarding the error message and logs, I unfortunately dont have the logs anymore, however the error from docker logs pulp was:

http: server gave HTTP response to HTTPS client

I claim it was due to my docker engine not picking up the correct insecure registries entry from /etc/docker/daemon.json, either because i typoed something or I didn’t have the correct FQDN at some point and thought I restarted the engine and/or pulp container but didn’t.

2 Likes