Issue during image pull for pulp

Support
#1

Problem: message: ‘rpc error: code = Unknown desc = failed to pull and unpack image
“quay.io/pulp/pulp:stable”: failed to extract layer sha256:9de3dae6fef5c990b33b11f14829b1a5b1624a7537676e9f475bd2a8f73bc1b8:
operation not permitted: unknown’
reason: ErrImagePull

Expected outcome: Image Pull should work without any issues

Pulpcore version: Pulp Operator version 0.14

Pulp plugins installed and their versions:

Operating system - distribution and version:

Other relevant data:
The pulp operator is being deployed on an AKS Cluster. Note: Galaxy-web images, pulp-web images do not have any issue.

#2

Is this still a problem? A new image has been published since you originally made this post.

#3

Yes, I deleted the pod and it tried pulling the image again, but it failed with the same error. The pull works fine using docker pull on local machine. But on AKS it fails. Pulp-web image is working without any issue.

#4

could you please try using quay.io/pulp/pulp-minimal:stable

#5

I see the same error

#6

Got same error / problem, i can succesfully deploy galaxy image version 4.5.2 but everything after/higher this version i get the image Pull error:

NAME                                                   READY   STATUS              RESTARTS      AGE
pod/galaxy-ng-api-f446b6b6-z86pc                       0/1     ErrImagePull        0             4m23s
pod/galaxy-ng-content-598599655c-m2s28                 0/1     ContainerCreating   0             4m19s
pod/galaxy-ng-postgres-13-0                            1/1     Running             0             4m47s
pod/galaxy-ng-redis-645d87c99-8wk2x                    1/1     Running             0             4m32s
pod/galaxy-ng-web-6ff775d565-bpl4n                     0/1     Running             3 (59s ago)   4m40s
pod/galaxy-ng-worker-b49594b9b-ltpfx                   0/1     ContainerCreating   0             4m17s
pod/pulp-operator-controller-manager-669c44fc6-vnplv   2/2     Running             0             25m

Events:
  Type     Reason                  Age                    From                     Message
  ----     ------                  ----                   ----                     -------
  Normal   Scheduled               5m9s                   default-scheduler        Successfully assigned galaxy/galaxy-ng-api-f446b6b6-z86pc to aks-system-29551743-vmss000008
  Normal   SuccessfulAttachVolume  5m4s                   attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-d74ad063-9796-4dc5-84d9-b0ece242bcb6"
  Normal   Pulling                 3m (x4 over 5m3s)      kubelet                  Pulling image "quay.io/pulp/galaxy:4.5.3"
  Warning  Failed                  2m48s (x4 over 4m49s)  kubelet                  Failed to pull image "quay.io/pulp/galaxy:4.5.3": rpc error: code = Unknown desc = failed to pull and unpack image "quay.io/pulp/galaxy:4.5.3": failed to extract layer sha256:b49ee7f1eb4f22ceaf401b3eac2eebfe06f702bb467e4ac0d9ea36ee87df6fd9: operation not permitted: unknown
  Warning  Failed                  2m48s (x4 over 4m49s)  kubelet                  Error: ErrImagePull
  Warning  Failed                  2m24s (x6 over 4m48s)  kubelet                  Error: ImagePullBackOff
  Normal   BackOff                 2m13s (x7 over 4m48s)  kubelet                  Back-off pulling image "quay.io/pulp/galaxy:4.5.3"

galaxy-web image is working fine with all the newer versions

#8

tried to do some debugging on the AKS node, but am not able to get any more information while trying to pull the image from quay…:

root@aks-node-189112344-vmss002223:/# crictl --debug pull quay.io/pulp/galaxy:stable
DEBU[0000] get image connection                         
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:quay.io/pulp/galaxy:stable,Annotations:map[string]string{},},Auth:nil,SandboxConfig:nil,} 
E1111 07:43:27.960158   43523 remote_image.go:242] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"quay.io/pulp/galaxy:stable\": failed to extract layer sha256:db4e88ec5915e6d2ca0aaab9198f8dc4e2eb40285da4d9e751f853cef1315964: operation not permitted: unknown" image="quay.io/pulp/galaxy:stable"
FATA[0030] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "quay.io/pulp/galaxy:stable": failed to extract layer sha256:db4e88ec5915e6d2ca0aaab9198f8dc4e2eb40285da4d9e751f853cef1315964: operation not permitted: unknown 

root@aks-node-189112344-vmss002223:/# crictl --debug pull quay.io/pulp/pulp:stable
DEBU[0000] get image connection                         
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:quay.io/pulp/pulp:stable,Annotations:map[string]string{},},Auth:nil,SandboxConfig:nil,} 
E1111 07:44:03.759315   44529 remote_image.go:242] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"quay.io/pulp/pulp:stable\": failed to extract layer sha256:a3b8450d21451a6808b1f3ca7390772b12554abfb223273830ce6ca1e19b1fb1: operation not permitted: unknown" image="quay.io/pulp/pulp:stable"
FATA[0027] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "quay.io/pulp/pulp:stable": failed to extract layer sha256:a3b8450d21451a6808b1f3ca7390772b12554abfb223273830ce6ca1e19b1fb1: operation not permitted: unknown

Not sure if it is AKS related or something with the images, cause it only occurs on these galaxy / pulp (minimal) images…

made an issue at the repo of AKS: https://github.com/Azure/AKS/issues/3330
probably related to: https://github.com/containerd/containerd/pull/7094 as of Microsoft custom containerd (containerd://1.6.4+azure-4)

#9

Thanks, lets see what Microsoft replies on this issue. As you mentioned, I am not facing this on other images. Galaxy images are working fine.

#10

I was seeing this issue outside of AKS, running containerd 1.6.6.
I can confirm that the image pulls fine once containerd is upgraded to >= 1.6.9

1 Like