Azure Blob Storage not mounted in containers

Problem:
Azure blob storage details specified but storage not mounted at /var/lib/pulp.

Expected outcome:
Azure blob storage mounted at /var/lib/pulp.

Pulpcore version:
3.78.0

Pulp plugins installed and their versions:
|ansible|0.25.1|
|certguard|3.78.0|
|container|2.25.1|
|core|3.78.0|
|deb|3.5.2|
|file|3.78.0|
|gem|0.7.1|
|maven|0.10.1|
|npm|0.3.3|
|ostree|2.4.8|
|python|3.15.0|
|rpm|3.30.0|

Other relevant data:
I have added the Azure storage account details in a k8s secret and specified the secret in the deployment manifest as object_storage_azure_secret. Following a redeploy, when I remote into the containers I can see the Azure storage details have been added under ‘STORAGES’ to /etc/pulp/settings.py. However nothing is mounted into /var/lib/pulp/. There were no errors in the container logs or k8s events that indicated an issue.

Is there somewhere else I can look for errors that have occurred while attempting to mount the storage?

Hi @robert-smith-maersk,

When using object storage services like Amazon S3 or Azure Blob Storage, you do not get a volume mount point inside your containers because these object stores are not traditional file systems or block storage. Instead, they are designed for storing and retrieving objects (files) via APIs or SDKs, not for direct mounting as a filesystem.

If the operator finished its execution (Troubleshoot installations - Pulp Project):

kubectl get pulp -oyaml
...
  {
    "lastTransitionTime": "2022-09-20T11:59:16Z",
    "message": "All tasks ran successfully",
    "reason": "OperatorFinishedExecution",
    "status": "True",
    "type": "Pulp-Operator-Finished-Execution"
  },
...

and you are having issues to upload data to Pulp, you can check the pulpcore-api pod logs:

kubectl logs -l pulp-api

if the problem is to pull data from Pulp, you can check the pulpcore-content pod logs:

kubectl logs -l pulp-content
2 Likes

Thanks @hyagi.

When I try to pull from my pull-through cache of npmjs.org, the npm client’s progress indicator spins, then times out:

npm install --registry http://<pulp-ip>/pulp/content/npm-npmjs-releases envinfo
npm error code ETIMEDOUT
npm error syscall connect
npm error errno ETIMEDOUT
npm error network request to https://<pulp-ip>/pulp/content/npm-npmjs-releases/envinfo/-/envinfo-7.14.0.tgz failed, reason: connect ETIMEDOUT <pulp-ip>
npm error network This is a problem related to network connectivity.
npm error network In most cases you are behind a proxy or have bad network settings.
npm error network
npm error network If you are behind a proxy, please make sure that the
npm error network 'proxy' config is set properly.  See: 'npm help config'
npm error A complete log of this run can be found in: /Users/Robert/.npm/_logs/2025-06-30T12_07_23_706Z-debug-0.log

In the logs for the ‘content’ pod, I can see a request come in:

 content ::ffff:10.244.1.78 [30/Jun/2025:12:07:24 +0000] "GET /pulp/content/npm-npmjs-releases/envinfo HTTP/1.0" 200 257997 "-" "npm/11.4.2 node/v24.3.0 darwin x64 workspaces/false"

Pulp has not created anything in the container of the Azure storage account. I created the root folder ‘pulp3’ myself and redeployed, but still there is no content added. Is there any way to confirm that django-storages has successfully established a connection and, if not, what the error is?

This is my Pulp spec:

k get pulp -oyaml
apiVersion: v1
items:
- apiVersion: repo-manager.pulpproject.org/v1
  kind: Pulp
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"repo-manager.pulpproject.org/v1","kind":"Pulp","metadata":{"annotations":{},"name":"example-pulp","namespace":"pulp"},"spec":{"admin_password_secret":"example-pulp-admin-password","api":{"replicas":2},"cache":{"enabled":true,"redis_storage_class":"default"},"content":{"replicas":2},"custom_pulp_settings":"settings","database":{"postgres_storage_class":"default"},"ingress_type":"loadbalancer","object_storage_azure_secret":"azure-storage","telemetry":{"enabled":true},"web":{"replicas":1},"worker":{"replicas":1}}}
    creationTimestamp: "2025-06-27T17:36:37Z"
    generation: 8
    name: example-pulp
    namespace: pulp
    resourceVersion: "20131097"
    uid: 7e078801-f22e-4891-90c5-a49479f56a89
  spec:
    admin_password_secret: example-pulp-admin-password
    api:
      replicas: 2
    cache:
      enabled: true
      redis_storage_class: default
    container_auth_private_key_name: container_auth_private_key.pem
    container_auth_public_key_name: container_auth_public_key.pem
    container_token_secret: example-pulp-container-auth
    content:
      replicas: 2
    custom_pulp_settings: settings
    database:
      postgres_storage_class: default
    db_fields_encryption_secret: example-pulp-db-fields-encryption
    image: harbor.maersk.io/dot/pulp/pulp
    image_version: stable
    image_web: harbor.maersk.io/dot/pulp/pulp-web
    image_web_version: stable
    ingress_type: loadbalancer
    object_storage_azure_secret: azure-storage
    pulp_secret_key: example-pulp-secret-key
    telemetry:
      enabled: true
      exporter_otlp_protocol: http/protobuf
    web:
      replicas: 1
    worker:
      replicas: 1
  status:
    admin_password_secret: example-pulp-admin-password
    conditions:
    - lastTransitionTime: "2025-06-27T18:39:54Z"
      message: All tasks ran successfully
      reason: OperatorFinishedExecution
      status: "True"
      type: Pulp-Operator-Finished-Execution
    - lastTransitionTime: "2025-06-27T18:39:52Z"
      message: All Api tasks ran successfully
      reason: ApiTasksFinished
      status: "True"
      type: Pulp-API-Ready
    - lastTransitionTime: "2025-06-27T17:36:37Z"
      message: All Database tasks ran successfully
      reason: DatabaseTasksFinished
      status: "True"
      type: Pulp-Database-Ready
    - lastTransitionTime: "2025-06-27T18:39:52Z"
      message: All Content tasks ran successfully
      reason: ContentTasksFinished
      status: "True"
      type: Pulp-Content-Ready
    - lastTransitionTime: "2025-06-27T18:39:52Z"
      message: All Worker tasks ran successfully
      reason: WorkerTasksFinished
      status: "True"
      type: Pulp-Worker-Ready
    - lastTransitionTime: "2025-06-27T18:39:53Z"
      message: All Web tasks ran successfully
      reason: WebTasksFinished
      status: "True"
      type: Pulp-Web-Ready
    container_token_secret: example-pulp-container-auth
    db_fields_encryption_secret: example-pulp-db-fields-encryption
    image: harbor.maersk.io/dot/pulp/pulp:stable
    ingress_type: loadbalancer
    last_deployment_update: "2025-06-27T17:58:44Z"
    managed_cache_enabled: true
    object_storage_azure_secret: azure-storage
    pulp_secret_key: example-pulp-secret-key
    storage_type: azure blob
    telemetry_enabled: true
kind: List
metadata:
  resourceVersion: ""
1 Like

When I run pulp status, I see:

...
"storage": {
    "total": null,
    "used": 0,
    "free": null
  },
...

I’ve just tried the following in a Jupyter notebook using an identical django-storages configuration and it creates the file in the blob container just fine, so I think I can rule out incorrect azure storage details:

import os
import logging
import pprint

logging.basicConfig(
    level=logging.INFO,
    format="[%(asctime)s] {%(module)s:%(lineno)d} %(levelname)s - %(message)s",  # noqa: E501
)

import django
from django.core.files.base import ContentFile
from django.core.files.storage import default_storage

az_ac_name = os.getenv("DJ_AZURE_ACCOUNT_NAME")
az_ac_key = os.getenv("DJ_AZURE_ACCOUNT_KEY")
az_container = os.getenv("DJ_AZURE_CONTAINER")
az_conn_path = os.getenv("DJ_AZURE_CONTAINER_PATH")
az_conn_string = os.getenv("DJ_AZURE_CONNECTION_STRING")

# Configure Django settings for Azure Storage
import django.conf

STORAGES = {
    "default": {
        "BACKEND": "storages.backends.azure_storage.AzureStorage",
        "OPTIONS": {
            "connection_string": az_conn_string,
            "account_name": az_ac_name,
            "azure_container": az_container,
            "account_key": az_ac_key,
            "expiration_secs": 60,
            "overwrite_files": True,
            "location": az_conn_path
        },
    },
    "staticfiles": {"BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage"},
}

if not django.conf.settings.configured:
    django.conf.settings.configure(
        DEFAULT_FILE_STORAGE=STORAGES["default"]["BACKEND"],
        STORAGES=STORAGES,
        INSTALLED_APPS=[
            "django.contrib.contenttypes",
            "django.contrib.auth",
            "storages",
        ],
        SECRET_KEY="dummy",
    )

django.setup()


# Test writing a file to Azure Storage
file_content = ContentFile(b"Hello Azure Storage!")
file_name = "test_azure_storage.txt"
saved_path = default_storage.save(file_name, file_content)
logging.info(f"File saved to Azure Storage at: {saved_path}")

(Note: I did have to modify site-packages/django/apps/registry.py:83 and change it to self.app_configs = {} as per advice on stack overflow to get this to execute without error).

If I run cat /etc/pulp/settings.py in the content container and compare with pprint.pprint(STORAGES) in the notebook, the two config dumps match.

1 Like

My apologies @hyagi (and anyone else who wasted time looking at this) - I’ve tested with another package type and pulp does write files to the blob storage. I’m afraid I assumed that the switch to Azure blob storage was causing the issue as my npm pull-through cache was working prior, but I must have broken something somewhere else.

2 Likes

Hi @robert-smith-maersk,

Thank you for the update and all the info!!
Maybe this issue happened because, after modifying the storage type, pulp-operator does not automatically migrate the data to the new storage. This is something that is in our radar now that we have the migration endpoint (https://github.com/pulp/pulpcore/pull/4298), but we didn’t have time to implement it in the operator yet.

2 Likes

Hey @hyagi ,

I wish that were the case to save me the embarrassment, but I’m afraid it was just operator error (me, not pulp-operator). I had been entering the wrong pulp content path, but convinced myself that it must be the switch to blob storage as that was the only recent change. Everything was configured fine all along!

The only issue I have now is that my k6 test runs are receiving 302s trying to download packages and failing to authenticate. I gather from an earlier post that I need to disable redirect. I have tried disabling through the configmap:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: settings
data:
  analytics: "False"
  content_origin: '"http://<pulp_ip>"'
  api_root: '"/pulp/"'
  pypi_api_hostname: '"http://<pulp_ip>"'
  allowed_export_paths: '[ "/tmp" ]'
  allowed_import_paths: '[ "/tmp" ]'
  redirect_to_object_storage: "False"

Now I see ‘REDIRECT_TO_OBJECT_STORAGE’ twice in settings.py on the pods. First it is set to true, then as a result of the setting above there is a further entry setting it to false, which I suppose overrides that:

...
REDIRECT_TO_OBJECT_STORAGE = True
MEDIA_ROOT = ""
STORAGES = {
    "default": {
        "BACKEND": "storages.backends.azure_storage.AzureStorage",
        "OPTIONS": {
            "connection_string": 'DefaultEndpointsProtocol=***core.windows.net',
            "account_name": '****',
            "azure_container": 'filestore',
            "account_key": '***',
            "expiration_secs": 60,
            "overwrite_files": 'True',
            "location": 'pulp3'
        },
    },
    "staticfiles": {"BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage"},
}
TOKEN_SERVER = "http://example-pulp-api-svc.pulp.svc.cluster.local:24817/token/"
SECRET_KEY = "****"
ALLOWED_EXPORT_PATHS = [ "/tmp" ]
ALLOWED_IMPORT_PATHS = [ "/tmp" ]
ANALYTICS = False
API_ROOT = "/pulp/"
CONTENT_ORIGIN = "http://<pulp_ip>"
PYPI_API_HOSTNAME = "http://<pulp_ip>"
REDIRECT_TO_OBJECT_STORAGE = False

I still receive a 302 but am now able to download the files. I’m confused by this - I had expected that, from the client’s perspective, with redirection disabled the files would appear to be served from Pulp directly?

Actually I’d prefer to leave redirection enabled, but I’m not sure how to go about handling authentication to the blob storage for clients.

1 Like

I believe the setting has taken. dynaconf list | grep -i redirect in the pods produces:

REDIRECT_TO_OBJECT_STORAGE<bool> False
LOGIN_REDIRECT_URL<str> '/accounts/profile/'
LOGOUT_REDIRECT_URL<NoneType> None
SECURE_REDIRECT_EXEMPT<list> []
SECURE_SSL_REDIRECT<bool> False

However I still receive a 302 Found and redirect to the Azure blob store rather than a download through Pulp :thinking:

1 Like

@robert-smith-maersk i suspect you may be experiencing a caching problem. what happens when you restart the redis pod (to drop the cache)?

2 Likes

Thanks @dkliban, I’ve just tried scaling the redis deployment to 0, then back to 1. When the pod came back I tried again, but I’m still getting a 302 sadly.

 $ wget http://<pulp_ip>/pulp/content/npm-npmjs-releases/@esbuild/android-arm64/-/android-arm64-0.20.2.tgz --user admin --password $PULP_PASSWORD
--2025-07-02 16:31:05--  http://<pulp_ip>/pulp/content/npm-npmjs-releases/@esbuild/android-arm64/-/android-arm64-0.20.2.tgz
Connecting to <pulp_ip>:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://****.blob.core.windows.net/filestore/pulp3/artifact/cc/bd0d0d95ac2391671b2af27a7b8c7e4b3dc17633d8b43bc4c4d3949c8ffe36?se=2025-07-02T15%3A32%3A06Z&sp=r&sv=2025-05-05&sr=b&rscd=attachment%3Bfilename%3Dandroid-arm64-0.20.2.tgz&sig=****%3D [following]
--2025-07-02 16:31:06--  https://****.blob.core.windows.net/filestore/pulp3/artifact/cc/bd0d0d95ac2391671b2af27a7b8c7e4b3dc17633d8b43bc4c4d3949c8ffe36?se=2025-07-02T15%3A32%3A06Z&sp=r&sv=2025-05-05&sr=b&rscd=attachment%3Bfilename%3Dandroid-arm64-0.20.2.tgz&sig=****
Resolving ****.blob.core.windows.net (****.blob.core.windows.net)... <blobstore_ip>
Connecting to ****.blob.core.windows.net (****.blob.core.windows.net)|<blobstore_ip>|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3946564 (3.8M) [application/octet-stream]
Saving to: ‘android-arm64-0.20.2.tgz.1’

android-arm64-0.20.2.tgz.1                      100%[====================================================================================================>]   3.76M  19.7MB/s    in 0.2s

2025-07-02 16:31:06 (19.7 MB/s) - ‘android-arm64-0.20.2.tgz.1’ saved [3946564/3946564]

I am starting to think you found a regression. Could you please file an issue at https://github.com/pulp/pulpcore/issues/new?template=bug_report.md … I’ll make sure we prioritize it.

1 Like

Thanks @dkliban, I’ve raised the issue here https://github.com/pulp/pulpcore/issues/6761