Configuring S3 client settings (max retries, connection timeout, read timeout) in Pulp

Support
#1

Our S3 engineering team has strongly recommended that we update our S3 client configuration to align with the unified backend S3 settings. Specifically, they are asking us to configure:

  • Max retries
  • Connection timeout
  • Read timeout

We are currently using Pulp with S3 as the storage backend.

Is there a supported way to configure these S3 client parameters in Pulp?
If so, could you please advise where these settings can be adjusted (e.g., in settings.py, environment variables, or via boto3 configuration)?

Any guidance or examples would be greatly appreciated.

#2

We use django-storages for configuring S3 and their docs show that these settings can be configured with client_config. [0] This setting requires a Config python object so you need to set it in a settings.py, e.g:

from botocore.config import Config

MEDIA_ROOT = ""
STORAGES = {
    "default": {
        "BACKEND": "storages.backends.s3.S3Storage",
        "OPTIONS": {
            "bucket_name": "my-pulp-bucket",
            "region_name": "eu-central-1",
            "client_config": Config(
                connect_timeout=5,       # seconds to wait for a connection
                read_timeout=60,         # seconds to wait for a response
                retries={
                    "max_attempts": 10,
                    "mode": "standard",  # or "adaptive" for backoff
                },
            ),
        },
    },
    "staticfiles": {
        "BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage",
    },
}

Note if you use addressing_style, signature_version or proxies settings then you need to move them into the client_config because it overrides them when specified. [1]

from botocore.config import Config

Config(
    connect_timeout=5,
    read_timeout=60,
    retries={"max_attempts": 10, "mode": "standard"},
    s3={"addressing_style": "path"},
    signature_version="s3v4",
)

[0] Amazon S3 - django-storages 1.14.6 documentation
[1] Config Reference - botocore 1.43.8 documentation

1 Like