500 error pushing to Pulp

robert-smith-maersk · June 18, 2025, 5:12pm

Problem:
I receive a ‘500 Internal Server Error’ when trying to push to a PyPI repo.

Expected outcome:
Package uploaded to Pulp.

Pulpcore version:
3.78.0

Pulp plugins installed and their versions:
core 3.78.0
ansible 0.25.1
container 2.25.1
deb 3.5.2
gem 0.7.1
maven 0.10.1
npm 0.3.3
ostree 2.4.8
python 3.15.0
rpm 3.30.0
certguard 3.78.0
file 3.78.0

Other relevant data:
Hi,

I am testing package upload to a PyPI repository in Pulp, using the following command:

$ twine upload --repository-url "http://<pulp_host>/pypi/python-dist/simple/" -u admin -p '<admin_password>' "./dist/*" --verbose
Uploading distributions to http://<pulp_host>/pypi/python-dist/simple/
INFO     ./dist/hello_world-0.1.2-py3-none-any.whl (1.3 KB)
INFO     ./dist/hello_world-0.1.2.tar.gz (1.0 KB)
INFO     username set by command options
INFO     password set by command options
INFO     username: admin
INFO     password: <hidden>
Uploading hello_world-0.1.2-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 kB • 00:00 • ?
WARNING  Received "500: Internal Server Error"
         Package upload appears to have failed. Retry 1 of 5.
Uploading hello_world-0.1.2-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 kB • 00:00 • ?
WARNING  Received "500: Internal Server Error"
         Package upload appears to have failed. Retry 2 of 5.
Uploading hello_world-0.1.2-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 kB • 00:00 • ?
WARNING  Received "500: Internal Server Error"
         Package upload appears to have failed. Retry 3 of 5.
Uploading hello_world-0.1.2-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 kB • 00:00 • ?
WARNING  Received "500: Internal Server Error"
         Package upload appears to have failed. Retry 4 of 5.
Uploading hello_world-0.1.2-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 kB • 00:00 • ?
WARNING  Received "500: Internal Server Error"
         Package upload appears to have failed. Retry 5 of 5.
INFO     Response from http://<pulp_host>/pypi/python-dist/simple/:
         500 Internal Server Error
INFO
         <!doctype html>
         <html lang="en">
         <head>
           <title>Server Error (500)</title>
         </head>
         <body>
           <h1>Server Error (500)</h1><p></p>
         </body>
         </html>

ERROR    HTTPError: 500 Internal Server Error from http://<pulp_host>/pypi/python-dist/simple/
         Internal Server Error

In the API pod logs I see the following error:

pulp [2ee46527e45e45839fbd60f183d588d3]: django.request:ERROR: Internal Server Error: /pypi/python-dist/simple/
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 515, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 475, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 486, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 512, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/pypi/views.py", line 306, in create
    return self.upload(request, path)
  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/pypi/views.py", line 157, in upload
    serializer.is_valid(raise_exception=True)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 225, in is_valid
    self._validated_data = self.run_validation(self.initial_data)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 447, in run_validation
    value = self.validate(value)
  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/pypi/serializers.py", line 78, in validate
    artifact.save()
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/content.py", line 155, in save
    super().save(*args, **kwargs)
  File "/usr/lib64/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.9/site-packages/django_lifecycle/mixins.py", line 139, in save
    save(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 814, in save
    self.save_base(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 877, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1020, in _save_table
    results = self._do_insert(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1061, in _do_insert
    return manager._insert(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 1805, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1821, in execute_sql
    for sql, params in self.as_sql():
  File "/usr/local/lib/python3.9/site-packages/django_readonly_field/compiler.py", line 31, in as_sql
    return super(ReadonlySQLCompilerMixin, self).as_sql()
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1745, in as_sql
    value_rows = [
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1746, in <listcomp>
    [
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1747, in <listcomp>
    self.prepare_value(field, self.pre_save_val(field, obj))
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1695, in pre_save_val
    return field.pre_save(obj, add=True)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/fields.py", line 89, in pre_save
    return super().pre_save(model_instance, add)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 317, in pre_save
    file.save(file.name, file.file, save=False)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 93, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/usr/local/lib/python3.9/site-packages/django/core/files/storage/base.py", line 49, in save
    name = self._save(name, content)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/storage.py", line 96, in _save
    os.chmod(full_path, self.file_permissions_mode)
PermissionError: [Errno 1] Operation not permitted: '/var/lib/pulp/media/artifact/13/df8e84234587b63618d2ef3056511764a2d6ea1e5f6dc68e03ec210babaf3d'
('pulp [2ee46527e45e45839fbd60f183d588d3]: ::ffff:10.244.0.47 - admin [18/Jun/2025:16:03:02 +0000] "POST /pypi/python-dist/simple/ HTTP/1.0" 500 145 "-" "twine/6.1.0 CPython/3.9.2"',)
pulp [a8d1f1785af948cd8ad2482162663f27]: django.request:ERROR: Internal Server Error: /pypi/python-dist/simple/
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 515, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 475, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 486, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 512, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/pypi/views.py", line 306, in create
    return self.upload(request, path)
  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/pypi/views.py", line 157, in upload
    serializer.is_valid(raise_exception=True)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 225, in is_valid
    self._validated_data = self.run_validation(self.initial_data)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 447, in run_validation
    value = self.validate(value)
  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/pypi/serializers.py", line 78, in validate
    artifact.save()
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/content.py", line 155, in save
    super().save(*args, **kwargs)
  File "/usr/lib64/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.9/site-packages/django_lifecycle/mixins.py", line 139, in save
    save(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 814, in save
    self.save_base(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 877, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1020, in _save_table
    results = self._do_insert(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1061, in _do_insert
    return manager._insert(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 1805, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1821, in execute_sql
    for sql, params in self.as_sql():
  File "/usr/local/lib/python3.9/site-packages/django_readonly_field/compiler.py", line 31, in as_sql
    return super(ReadonlySQLCompilerMixin, self).as_sql()
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1745, in as_sql
    value_rows = [
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1746, in <listcomp>
    [
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1747, in <listcomp>
    self.prepare_value(field, self.pre_save_val(field, obj))
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1695, in pre_save_val
    return field.pre_save(obj, add=True)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/fields.py", line 89, in pre_save
    return super().pre_save(model_instance, add)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 317, in pre_save
    file.save(file.name, file.file, save=False)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 93, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/usr/local/lib/python3.9/site-packages/django/core/files/storage/base.py", line 49, in save
    name = self._save(name, content)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/storage.py", line 96, in _save
    os.chmod(full_path, self.file_permissions_mode)
PermissionError: [Errno 1] Operation not permitted: '/var/lib/pulp/media/artifact/13/df8e84234587b63618d2ef3056511764a2d6ea1e5f6dc68e03ec210babaf3d'
('pulp [a8d1f1785af948cd8ad2482162663f27]: ::ffff:10.244.0.47 - admin [18/Jun/2025:16:03:02 +0000] "POST /pypi/python-dist/simple/ HTTP/1.0" 500 145 "-" "twine/6.1.0 CPython/3.9.2"',)
pulp [56d9c97059484a3f9b007a03787dad52]: django.request:ERROR: Internal Server Error: /pypi/python-dist/simple/
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 515, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 475, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 486, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 512, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/pypi/views.py", line 306, in create
    return self.upload(request, path)
  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/pypi/views.py", line 157, in upload
    serializer.is_valid(raise_exception=True)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 225, in is_valid
    self._validated_data = self.run_validation(self.initial_data)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 447, in run_validation
    value = self.validate(value)
  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/pypi/serializers.py", line 78, in validate
    artifact.save()
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/content.py", line 155, in save
    super().save(*args, **kwargs)
  File "/usr/lib64/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.9/site-packages/django_lifecycle/mixins.py", line 139, in save
    save(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 814, in save
    self.save_base(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 877, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1020, in _save_table
    results = self._do_insert(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1061, in _do_insert
    return manager._insert(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 1805, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1821, in execute_sql
    for sql, params in self.as_sql():
  File "/usr/local/lib/python3.9/site-packages/django_readonly_field/compiler.py", line 31, in as_sql
    return super(ReadonlySQLCompilerMixin, self).as_sql()
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1745, in as_sql
    value_rows = [
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1746, in <listcomp>
    [
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1747, in <listcomp>
    self.prepare_value(field, self.pre_save_val(field, obj))
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1695, in pre_save_val
    return field.pre_save(obj, add=True)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/fields.py", line 89, in pre_save
    return super().pre_save(model_instance, add)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 317, in pre_save
    file.save(file.name, file.file, save=False)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 93, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/usr/local/lib/python3.9/site-packages/django/core/files/storage/base.py", line 49, in save
    name = self._save(name, content)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/storage.py", line 96, in _save
    os.chmod(full_path, self.file_permissions_mode)
PermissionError: [Errno 1] Operation not permitted: '/var/lib/pulp/media/artifact/13/df8e84234587b63618d2ef3056511764a2d6ea1e5f6dc68e03ec210babaf3d'
('pulp [56d9c97059484a3f9b007a03787dad52]: ::ffff:10.244.0.47 - admin [18/Jun/2025:16:03:02 +0000] "POST /pypi/python-dist/simple/ HTTP/1.0" 500 145 "-" "twine/6.1.0 CPython/3.9.2"',)
pulp [500963454c0248148efd7b40e0c8cb4f]: django.request:ERROR: Internal Server Error: /pypi/python-dist/simple/
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 515, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 475, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 486, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 512, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/pypi/views.py", line 306, in create
    return self.upload(request, path)
  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/pypi/views.py", line 157, in upload
    serializer.is_valid(raise_exception=True)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 225, in is_valid
    self._validated_data = self.run_validation(self.initial_data)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 447, in run_validation
    value = self.validate(value)
  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/pypi/serializers.py", line 78, in validate
    artifact.save()
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/content.py", line 155, in save
    super().save(*args, **kwargs)
  File "/usr/lib64/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.9/site-packages/django_lifecycle/mixins.py", line 139, in save
    save(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 814, in save
    self.save_base(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 877, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1020, in _save_table
    results = self._do_insert(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1061, in _do_insert
    return manager._insert(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 1805, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1821, in execute_sql
    for sql, params in self.as_sql():
  File "/usr/local/lib/python3.9/site-packages/django_readonly_field/compiler.py", line 31, in as_sql
    return super(ReadonlySQLCompilerMixin, self).as_sql()
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1745, in as_sql
    value_rows = [
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1746, in <listcomp>
    [
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1747, in <listcomp>
    self.prepare_value(field, self.pre_save_val(field, obj))
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1695, in pre_save_val
    return field.pre_save(obj, add=True)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/fields.py", line 89, in pre_save
    return super().pre_save(model_instance, add)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 317, in pre_save
    file.save(file.name, file.file, save=False)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 93, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/usr/local/lib/python3.9/site-packages/django/core/files/storage/base.py", line 49, in save
    name = self._save(name, content)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/storage.py", line 96, in _save
    os.chmod(full_path, self.file_permissions_mode)
PermissionError: [Errno 1] Operation not permitted: '/var/lib/pulp/media/artifact/13/df8e84234587b63618d2ef3056511764a2d6ea1e5f6dc68e03ec210babaf3d'
('pulp [500963454c0248148efd7b40e0c8cb4f]: ::ffff:10.244.0.47 - admin [18/Jun/2025:16:03:03 +0000] "POST /pypi/python-dist/simple/ HTTP/1.0" 500 145 "-" "twine/6.1.0 CPython/3.9.2"',)
pulp [2a962dd7a5954af49196d6a95436d931]: django.request:ERROR: Internal Server Error: /pypi/python-dist/simple/
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 515, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 475, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 486, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 512, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/pypi/views.py", line 306, in create
    return self.upload(request, path)
  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/pypi/views.py", line 157, in upload
    serializer.is_valid(raise_exception=True)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 225, in is_valid
    self._validated_data = self.run_validation(self.initial_data)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 447, in run_validation
    value = self.validate(value)
  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/pypi/serializers.py", line 78, in validate
    artifact.save()
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/content.py", line 155, in save
    super().save(*args, **kwargs)
  File "/usr/lib64/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.9/site-packages/django_lifecycle/mixins.py", line 139, in save
    save(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 814, in save
    self.save_base(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 877, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1020, in _save_table
    results = self._do_insert(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1061, in _do_insert
    return manager._insert(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 1805, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1821, in execute_sql
    for sql, params in self.as_sql():
  File "/usr/local/lib/python3.9/site-packages/django_readonly_field/compiler.py", line 31, in as_sql
    return super(ReadonlySQLCompilerMixin, self).as_sql()
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1745, in as_sql
    value_rows = [
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1746, in <listcomp>
    [
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1747, in <listcomp>
    self.prepare_value(field, self.pre_save_val(field, obj))
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1695, in pre_save_val
    return field.pre_save(obj, add=True)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/fields.py", line 89, in pre_save
    return super().pre_save(model_instance, add)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 317, in pre_save
    file.save(file.name, file.file, save=False)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 93, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/usr/local/lib/python3.9/site-packages/django/core/files/storage/base.py", line 49, in save
    name = self._save(name, content)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/storage.py", line 96, in _save
    os.chmod(full_path, self.file_permissions_mode)
PermissionError: [Errno 1] Operation not permitted: '/var/lib/pulp/media/artifact/13/df8e84234587b63618d2ef3056511764a2d6ea1e5f6dc68e03ec210babaf3d'
('pulp [2a962dd7a5954af49196d6a95436d931]: ::ffff:10.244.0.47 - admin [18/Jun/2025:16:03:03 +0000] "POST /pypi/python-dist/simple/ HTTP/1.0" 500 145 "-" "twine/6.1.0 CPython/3.9.2"',)

I am using azurefile storage, configured in the manifest as follows:

...
  file_storage_access_mode: "ReadWriteMany"
  file_storage_size: "2Gi"
  file_storage_storage_class: azurefile
...

In the API pod I see the storage mounted here:

$ df -h
Filesystem                                                                                Size  Used Avail Use% Mounted on
overlay                                                                                   146G   18G  128G  13% /
tmpfs                                                                                      64M     0   64M   0% /dev
/dev/root                                                                                 146G   18G  128G  13% /etc/hosts
shm                                                                                        64M     0   64M   0% /dev/shm
tmpfs                                                                                      14G  4.0K   14G   1% /etc/pulp/settings.py
tmpfs                                                                                      14G  4.0K   14G   1% /etc/pulp/pulp-admin-password
//<fs-id>.file.core.windows.net/pvc-<uid>                                                 2.0G  4.4M  2.0G   1% /var/lib/pulp
tmpfs                                                                                      14G  4.0K   14G   1% /etc/pulp/keys/database_fields.symmetric.key
tmpfs                                                                                      14G  8.0K   14G   1% /etc/pulp/keys/container_auth_public_key.pem
tmpfs                                                                                      14G   12K   14G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                                                                     7.9G     0  7.9G   0% /proc/acpi
tmpfs                                                                                     7.9G     0  7.9G   0% /proc/scsi
tmpfs                                                                                     7.9G     0  7.9G   0% /sys/firmware

I expected to see files in /var/lib/pulp/media owned by pulp, but instead they are root-owned just group pulp:

$ ls -l /var/lib/pulp/media/
total 0
drwxrwxrwx 2 root pulp 0 Jun 10 13:37 artifact

Can anyone suggest what the issue might be here? Is the Azure storage incorrectly mounted?

robert-smith-maersk · June 18, 2025, 5:17pm

Title updated as it seems to be a general storage configuration issue - I get the same error when running ‘mvn deploy’ to the AKS cluster hosted Pulp, while pushing to a single container deployment on localhost works fine. Truncated logs:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy (default-deploy) on project hello-world: Failed to deploy artifacts: Could not transfer artifact com.example:hello-world:jar:1.0.3 from/to pulp (http://<pulp_host>/pulp/maven/maven-releases/): Transfer failed for http://<pulp_host>/pulp/maven/maven-releases/com/example/hello-world/1.0.3/hello-world-1.0.3.jar 500 Internal Server Error -> [Help 1]

API pod log tail:

  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/storage.py", line 96, in _save
    os.chmod(full_path, self.file_permissions_mode)
PermissionError: [Errno 1] Operation not permitted: '/var/lib/pulp/media/artifact/14/fa720e7a797e677783694463cefdaaf851f87cff8b6c1ceadbff411cc7d8ef'
('pulp [0790292a4d1d47bc81002fa853b91549]: ::ffff:10.244.0.47 - - [18/Jun/2025:17:11:48 +0000] "PUT /pulp/maven/maven-releases/com/example/hello-world/1.0.3/hello-world-1.0.3.pom HTTP/1.0" 500 145 "-" "Apache-Maven/3.6.3 (Java 21.0.7; Linux 6.12.13-200.fc41.x86_64)"',)

robert-smith-maersk · June 19, 2025, 8:57am

If I remote into either the API or content pods as the pulp user I am able to write into /var/lib/pulp/media (and it’s descendants):

bash-5.1$ pwd
/var/lib/pulp/media
bash-5.1$ id
uid=700(pulp) gid=700(pulp) groups=700(pulp)
bash-5.1$ touch test.txt
bash-5.1$ echo "Hi!" > test.txt
bash-5.1$ cat test.txt
Hi!
bash-5.1$ ls -al
total 1
drwxrwxrwx 2 root pulp 0 Jun  3 10:50 .
drwxrwxrwx 2 root pulp 0 Jun  3 10:50 ..
drwxrwxrwx 2 root pulp 0 Jun 10 13:37 artifact
-rwxrwxrwx 1 root pulp 4 Jun 19 08:52 test.txt

robert-smith-maersk · June 19, 2025, 2:05pm

Ah, it seems the azurefile storage class doesn’t support 'NIX file permissions. I will experiment with other storage options.

robert-smith-maersk · June 20, 2025, 11:59am

It was indeed the file storage type. I recall I had tried ‘default’ initially but deployment failed. Having investigated further this time around, it seems the failure was caused by my specifying ‘ReadWriteMany’ which is not supported. With the configuration below, everything now works as expected.

...
file_storage_access_mode: "ReadWriteOnce"
file_storage_size: "30Gi"
file_storage_storage_class: default
...

hyagi · June 20, 2025, 12:01pm

Hi @robert-smith-maersk

Yes, I believe this is happening, as you mentioned, because of a “limitation” (lack of support CIFS Unix extensions) in azure-file: 1826420 – When a default Azure File storage class is created, there is a bug that is causing the UID/GID to be flipped when the volume gets mounted to the pod.

hyagi · June 20, 2025, 12:05pm

hmm… we would not recommend using ReadWriteOnce PVC. Even though it can work, it has some limitations as well: Common issues - Pulp Project

In this case, if possible, a “better” configuration would be using Azure Blob Storage: Pulp Operator storage configuration - Pulp Project

robert-smith-maersk · June 20, 2025, 1:30pm

Thanks @hyagi. At the moment I should be OK as I have only a single node deployment for my POC. I’ll bear that in mind when I move on to a longer-term setup.