Pulp Operator make deploy TLS error

Jeremy_Tourville · February 10, 2025, 2:15pm

Problem:
Unable to connect to the server: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-02-10T07:28:51-06:00 is after 2025-02-06T19:01:54Z

What server is the code trying to connect to? Which certificate is referred to here?
Expected outcome:

Pulpcore version:
pulp-operator (main branch)
Pulp plugins installed and their versions:
N/A
Operating system - distribution and version:
Rocky Linux 8.9
Other relevant data:
When building, the operator the build fails partway through for a TLS error. I know how to review and troubleshoot certificate issues but I need to understand what server the code is trying to connect to and which certificate to review.

[root@kube01 pulp-operator]# make deploy
go mod tidy
go: downloading go.uber.org/zap v1.27.0
go: downloading k8s.io/apimachinery v0.32.0
go: downloading sigs.k8s.io/controller-runtime v0.19.3
go: downloading k8s.io/client-go v0.32.0
go: downloading github.com/go-logr/logr v1.4.2
go: downloading k8s.io/api v0.32.0
go: downloading k8s.io/cli-runtime v0.32.0
go: downloading github.com/onsi/ginkgo/v2 v2.22.2
go: downloading github.com/onsi/gomega v1.36.2
go: downloading golang.org/x/crypto v0.31.0
go: downloading golang.org/x/text v0.21.0
go: downloading sigs.k8s.io/yaml v1.4.0
go: downloading github.com/stretchr/testify v1.9.0
go: downloading go.uber.org/goleak v1.3.0
go: downloading k8s.io/klog/v2 v2.130.1
go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.2
go: downloading k8s.io/apiextensions-apiserver v0.31.0
go: downloading golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56
go: downloading k8s.io/utils v0.0.0-20241104100929-3ea5e8cea738
go: downloading gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c
go: downloading github.com/moby/term v0.5.0
go: downloading github.com/evanphx/json-patch/v5 v5.9.0
go: downloading golang.org/x/net v0.33.0
go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc
go: downloading github.com/golang/protobuf v1.5.4
go: downloading google.golang.org/protobuf v1.36.1
go: downloading k8s.io/kube-openapi v0.0.0-20241105132330-32ad38e42d3f
go: downloading github.com/gorilla/websocket v1.5.0
go: downloading github.com/go-logr/zapr v1.3.0
go: downloading k8s.io/apiserver v0.31.0
go: downloading github.com/prometheus/client_golang v1.19.1
go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2
go: downloading sigs.k8s.io/json v0.0.0-20241010143419-9aa6b5e7a4b3
go: downloading github.com/kr/pretty v0.3.1
go: downloading golang.org/x/sys v0.28.0
go: downloading github.com/creack/pty v1.1.18
go: downloading golang.org/x/oauth2 v0.23.0
go: downloading github.com/fxamacker/cbor/v2 v2.7.0
go: downloading gopkg.in/evanphx/json-patch.v4 v4.12.0
go: downloading golang.org/x/term v0.27.0
go: downloading golang.org/x/time v0.7.0
go: downloading github.com/moby/spdystream v0.5.0
go: downloading github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5
go: downloading github.com/go-openapi/swag v0.23.0
go: downloading k8s.io/component-base v0.31.0
go: downloading github.com/cespare/xxhash/v2 v2.3.0
go: downloading github.com/prometheus/client_model v0.6.1
go: downloading github.com/prometheus/common v0.55.0
go: downloading github.com/prometheus/procfs v0.15.1
go: downloading github.com/google/uuid v1.6.0
go: downloading github.com/kr/text v0.2.0
go: downloading github.com/rogpeppe/go-internal v1.12.0
go: downloading github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161
go: downloading github.com/x448/float16 v0.8.4
go: downloading github.com/go-task/slim-sprig/v3 v3.0.0
go: downloading golang.org/x/tools v0.28.0
go: downloading github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f
go: downloading github.com/evanphx/json-patch v0.5.2
go: downloading golang.org/x/sync v0.10.0
go: downloading github.com/google/cel-go v0.20.1
go: downloading github.com/blang/semver/v4 v4.0.0
go: downloading github.com/go-openapi/jsonpointer v0.21.0
go: downloading github.com/google/pprof v0.0.0-20241210010833-40e02aabc2ad
go: downloading go.opentelemetry.io/otel/trace v1.28.0
go: downloading go.opentelemetry.io/otel v1.28.0
go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240528184218-531527333157
go: downloading google.golang.org/grpc v1.65.0
go: downloading sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.30.3
go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.53.0
go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.27.0
go: downloading go.opentelemetry.io/otel/sdk v1.28.0
go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.28.0
go: downloading github.com/stoewer/go-strcase v1.2.0
go: downloading github.com/antlr4-go/antlr/v4 v4.13.0
go: downloading github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a
go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240701130421-f6361c86f094
go: downloading github.com/felixge/httpsnoop v1.0.4
go: downloading go.opentelemetry.io/otel/metric v1.28.0
go: downloading go.opentelemetry.io/proto/otlp v1.3.1
go: downloading github.com/spf13/cobra v1.8.1
go: downloading github.com/go-logr/stdr v1.2.2
go: downloading github.com/cenkalti/backoff/v4 v4.3.0
go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0
go: downloading github.com/inconshreveable/mousetrap v1.1.0
mkdir -p /opt/pulp-operator/bin
test -s /opt/pulp-operator/bin/controller-gen || GOBIN=/opt/pulp-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.16.5
go: downloading sigs.k8s.io/controller-tools v0.16.5
go: downloading golang.org/x/tools v0.26.0
go: downloading github.com/fatih/color v1.18.0
go: downloading k8s.io/api v0.31.2
go: downloading k8s.io/apimachinery v0.31.2
go: downloading k8s.io/apiextensions-apiserver v0.31.2
go: downloading github.com/gobuffalo/flect v1.0.3
go: downloading k8s.io/utils v0.0.0-20240711033017-18e509b52bc8
go: downloading github.com/mattn/go-colorable v0.1.13
go: downloading github.com/mattn/go-isatty v0.0.20
go: downloading golang.org/x/sync v0.8.0
go: downloading golang.org/x/sys v0.26.0
go: downloading golang.org/x/mod v0.21.0
go: downloading golang.org/x/net v0.30.0
go: downloading golang.org/x/text v0.19.0
test -s /opt/pulp-operator/bin/crd-to-markdown || GOBIN=/opt/pulp-operator/bin go install github.com/clamoriniere/crd-to-markdown@v0.0.3
/opt/pulp-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/opt/pulp-operator/bin/crd-to-markdown -f apis/repo-manager.pulpproject.org/v1beta2/pulp_types.go -n Pulp > controllers/repo_manager/README.md
/opt/pulp-operator/bin/crd-to-markdown -f apis/repo-manager.pulpproject.org/v1beta2/pulp_backup_types.go -n PulpBackup > controllers/backup/README.md
/opt/pulp-operator/bin/crd-to-markdown -f apis/repo-manager.pulpproject.org/v1beta2/pulp_restore_types.go -n PulpRestore > controllers/restore/README.md
test -s /opt/pulp-operator/bin/kustomize || { curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash -s -- 3.8.7 /opt/pulp-operator/bin; }
{Version:kustomize/v3.8.7 GitCommit:ad092cc7a91c07fdf63a2e4b7f13fa588a39af4f BuildDate:2020-11-11T23:14:14Z GoOs:linux GoArch:amd64}
kustomize installed to /opt/pulp-operator/bin/kustomize
cd config/manager && /opt/pulp-operator/bin/kustomize edit set image controller=quay.io/pulp/pulp-operator:v1.0.0-beta.5
cd config/default && /opt/pulp-operator/bin/kustomize edit set namespace pulp-operator-system
/opt/pulp-operator/bin/kustomize build config/default | kubectl apply --server-side=true -f -
Unable to connect to the server: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-02-10T07:11:18-06:00 is after 2025-02-06T19:01:54Z
make: *** [Makefile:260: deploy] Error 1

hyagi · February 17, 2025, 12:44pm

Hi @Jeremy_Tourville!

What server is the code trying to connect to? Which certificate is referred to here?

From the output provided, kubectl is probably trying to connect to k8s API server:

/opt/pulp-operator/bin/kustomize build config/default | kubectl apply --server-side=true -f -
Unable to connect to the server: tls: failed to verify certificate: x509: ce…

We can try to identify which certificate has expired by increasing kubectl log verbosity, rerun the make deploy, and check where it failed:

github.com

pulp/pulp-operator/blob/main/Makefile#L260

    
      
          	$(KUSTOMIZE) build config/crd | kubectl delete --ignore-not-found=$(ignore-not-found) -f -
          
          
.PHONY: local
          local: kustomize ## Run controller in the K8s cluster specified in ~/.kube/config.
          	.ci/scripts/local.sh $(CR_KIND) $(CR_DOMAIN) $(CR_PLURAL) $(APP_IMAGE) $(WEB_IMAGE)
          
          
.PHONY: deploy
          deploy: manifests kustomize ## Deploy controller to the K8s cluster specified in ~/.kube/config.
          	cd config/manager && $(KUSTOMIZE) edit set image controller=${IMG}
          	cd config/default && $(KUSTOMIZE) edit set namespace ${NAMESPACE}
          	$(KUSTOMIZE) build config/default | kubectl apply --server-side=true -f -
          
          
.PHONY: undeploy
          undeploy: ## Undeploy controller from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.
          	$(KUSTOMIZE) build config/default | kubectl delete --ignore-not-found=$(ignore-not-found) -f -
          
          
##@ Build Dependencies
          
          
## Location to install dependencies to
          LOCALBIN ?= $(shell pwd)/bin
          $(LOCALBIN):

$ vim Makefile
...
-      $(KUSTOMIZE) build config/default | kubectl apply --server-side=true -f -
+      $(KUSTOMIZE) build config/default | kubectl apply -v=8 --server-side=true -f -
...

Jeremy_Tourville · February 17, 2025, 5:14pm

OK, this was very helpful. You are correct, the API certs were the issue as shown from the more verbose logs.

I ran a command to check all certificates in my cluster.

kubeadm certs check-expiration

This revealed that I had many expired certs in my cluster. I was able to get them all renewed and then restarted the various components so that the new certificates could be used.

kubeadm certs renew all
systemctl restart kubelet

After doing all this I was able to get the make process to complete and I now have pods up and running in my cluster.

hyagi · February 17, 2025, 5:35pm

Nice!
Glad to hear it worked, and thank you for providing the steps to check and renew the certs!