Updating Deb Distributions via the Python API

Problem:

When using pulp deb’s Python API to (partially) update a distribution after a sync (making the distribution “point” to the publication), the expected contents are not actually accessible via the distribution’s URL. However, in the same state, using the CLI command (pulp deb distribution update) works as expected.

More specifically, after trying to update a distribution via the Python API, I get the message “404 Not Found” when trying to browse the contents of the distribution.

Expected outcome:

That after updating, the distribution URL would provide access to the contents specified in the publication. More specifically, accessing the distribution via the URL should show some its contents (e.g. the dists and pool directories).

Pulpcore version:

3.46

Pulp plugins installed and their versions:

pulp-cli==0.23.2
pulp-cli-deb==0.0.5
pulp-container==2.18.0
pulp-container-client==2.18.0
pulp-deb==3.1.1
pulp-deb-client==3.1.1
pulp-file==1.16.0
pulp-file-client==3.46.0
pulp-glue==0.23.2
pulpcore==3.46.0
pulpcore-client==3.46.0

Operating system - distribution and version:

Ubuntu 22.04

Other relevant data:

Here is a MWE of the Python code used: Snippet showing a tentative distribution update with pulp deb · GitHub

import time

import pulpcore
from pulpcore.client.pulp_deb import (
    AptRepositorySyncURL,
    DebAptDistribution,
    DebAptRemote,
    DebAptRepository,
    DistributionsAptApi,
    RemotesAptApi,
    RepositoriesAptApi,
    DebVerbatimPublication,
    PublicationsVerbatimApi,
    PatcheddebAptDistribution,
)
from pulpcore.client.pulpcore import Configuration


def wait_for_href(core_client, response):
    if not hasattr(response, "task"):
        return response.pulp_href

    tasks_api = pulpcore.client.pulpcore.TasksApi(api_client=core_client)
    while True:
        task = tasks_api.read(response.task)
        if task.state not in ("waiting", "running"):
            return task.created_resources[0]
        time.sleep(1)


if __name__ == "__main__":
    # Note: tested using the suggested setup of OCI images with docker-compose
    repo_name = "kitware-test"
    configuration = Configuration(
        username="admin",
        password="password",
        host="http://localhost:8080",
    )

    core_client = pulpcore.client.pulpcore.ApiClient(configuration)
    deb_client = pulpcore.client.pulp_deb.ApiClient(configuration)

    # Create repository

    repo_api = RepositoriesAptApi(api_client=deb_client)

    repo_href = wait_for_href(core_client, repo_api.create(
        DebAptRepository(name=repo_name)),
    )

    # Create remote

    remote_api = RemotesAptApi(api_client=deb_client)

    remote = DebAptRemote(
        name=repo_name,
        url="http://apt.kitware.com/ubuntu",
        distributions="jammy",
        architectures="amd64",
        download_concurrency=1,
        sync_sources=False,
        policy="on_demand",
    )

    remote_href = wait_for_href(core_client, remote_api.create(remote))

    # Create distribution

    dist_api = DistributionsAptApi(api_client=deb_client)

    deb_dist = DebAptDistribution(name=repo_name, base_path=repo_name)

    dist_href = wait_for_href(core_client, dist_api.create(deb_dist))

    # Sync repository with remote

    sync_url = AptRepositorySyncURL(
        remote=remote_href,
        mirror=False,
        optimize=True,
    )

    repo_api.sync(
        deb_apt_repository_href=repo_href,
        apt_repository_sync_url=sync_url,
    )

    # Create publication

    publication_api = PublicationsVerbatimApi(api_client=deb_client)

    publication = DebVerbatimPublication(
        repository=repo_href,
    )

    publication_href = wait_for_href(
        core_client,
        publication_api.create(publication),
    )

    # Everything up to this point works as expected.
    # One can check the existence of the remote, repository, publication, and
    # synced metadata via the cli.
    # Additionally, trying to access
    # http://localhost:8080/pulp/content/kitware-test/ gives
    # "404: Distribution is not pointing to a publication, repository, repository version, or remote."

    # Update distribution
    # Here is where the problem lies

    patched_dist = PatcheddebAptDistribution(
        publication=publication_href,
    )

    dist_api.partial_update(
        deb_apt_distribution_href=dist_href,
        patcheddeb_apt_distribution=patched_dist,
    )

    # After this, accessing
    # http://localhost:8080/pulp/content/kitware-test/ gives
    # 404: Not Found, but if one does the update of the distribution via the cli,
    # all the contents are there correctly
    #
    # Also tested using full updates instead of only partial updates and had the
    # same issue.

And here is an “equivalent” bash script using the CLI, which works: Example bash script to create remote, repository, publication, and distribution using Pulp Deb · GitHub

#!/bin/bash

NAME="bogus-kitware"
DIST_NAME="${NAME}/dev"

echo ">> Repo Create (no remote)"
pulp deb repository create --name=${NAME}

echo "Create Remote"
http http://localhost:8080/pulp/api/v3/remotes/deb/apt/ name=${NAME} url=https://apt.kitware.com/ubuntu distributions=jammy architectures=amd64 policy=on_demand

echo ">> Create distribution"
pulp deb distribution create --name=${DIST_NAME} --base-path=${DIST_NAME}

echo ">> Repo Sync (specifies remote)"
pulp deb repository sync --name=${NAME} --remote=${NAME}

echo ">> Pub Create (saving HREF)"
PUB_HREF=$(pulp deb publication --type=verbatim create --repository=${NAME} | jq -r '.pulp_href')

echo ">> Dist Create"
pulp deb distribution update --name=${DIST_NAME} --publication="${PUB_HREF}"
1 Like

I don’t know how quickly we will get to it, but me or a colleague will have a look.

1 Like

Thanks for providing examples! I needed a minute to realize that with python API you refer to what we use to call the “bindings”. However, we have identified a couple of shortcomings with the bindings approach (Basically, when not generated for the specific machine at hand they may be inadequate.). So their days are counted.
I would love to suggest using pulp-glue [0] (the library behind pulp cli) instead. Even though the pulp-glue-deb parts are not quite ready yet [1] you should be able to import the contexts from the cli modules for the time being.
You know that the cli allows adding up to 3 times ‘-v’ to help understanding what’s going on inside?

[0] Architecture - Pulp CLI
[1] Split out pulp-glue-deb by quba42 · Pull Request #93 · pulp/pulp-cli-deb · GitHub

1 Like

I needed a minute to realize that with python API you refer to what we use to call the “bindings”.

Apologies for the misnomer, indeed, I should have called them bindings since the real API is the REST one.

So their days are counted.

Thank you! This is actually really valuable information for me, since I am developing a quite comprehensive in-house solution with Pulp

I would love to suggest using pulp-glue [0] (the library behind pulp cli) instead.

Nice! I will have a look at that, I just became aware of pulp-glue when trying to debug the pulp cli commands via pdb to understand what was happening. Could you tell me how well pulp-glue fares in terms of stability and reliability compared with the REST API itself? Did I understand correctly that I should expect some big changes in pulp-glue’s interface soon?

1 Like

Pulp glue tries (as best as possible) the subtle changes between pulp server versions from you. Or tells you if you requested something impossible for your installation. I don’t think that big breaking changes are upon us here. The change I was referring to is a PR that provides the pulp_deb specific parts as a plugin to glue (yes, it’s plugins all the way down…).
One thing to keep in mind is that glue (as well as the cli) is not generated from openapi3-docs. So they are not automatically in feature parity with the rest apis. It should be fairly easy to extend though.

1 Like

I understand and I appreciate the detailed reply.

I will try the implementing the solution with pulp-glue’s contexts and see how it goes :+1:

1 Like

Looking forward to hear about actual 3rd party experience here!
I understand you are working on an “in house” solution. Will there by anything you may consider sharing?

There should be a pulp-glue-deb package on pypi as soon as Split out pulp-glue-deb by quba42 · Pull Request #93 · pulp/pulp-cli-deb · GitHub is merged + released.

Though there certainly are a few feature gaps in pulp-cli-deb and hence pulp-glue-deb. Things like pulp-cli-deb not knowing about individual API parameters. It is fairly easy to extend though and any workflow you can do with pulp-cli-deb will be possible with pulp-glue-deb as well.

2 Likes

Will there by anything you may consider sharing?

It is pretty much a set of local apt and container mirrors. The use of the Python bindings is mostly to integrate with existing code and to automate/streamline some tasks (syncs, rebuild from scratch, testing, and so on).

2 Likes

There should be a pulp-glue-deb package on pypi as soon

Looking forward to that!

any workflow you can do with pulp-cli-deb will be possible with pulp-glue-deb as well

That is good to hear. I believe that my previous snippet covers most, if not all, the steps I need on the deb side

2 Likes

Thanos @rfguimaraes for sharing your example. But I had some issues getting sync to work as you mentioned, I had same 404 not found issues. But I fixed the code to sync using synchronous code.

The repo_api.sync call is synchronous unless you specify async_req=true as a parameter. I’ve updated the code below to use wait_for_href, as an example, but this is NOT recommended!

    PULP_API_USERNAME = os.getenv("PULP_ADMIN", "admin")
    PULP_API_PASSWORD = os.getenv("PULP_PASSWORD", "password")
    PULP_API_BASE_URL = "http://pulp:8080"
    name = "ubuntu-jammy"
    url = "http://archive.ubuntu.com/ubuntu/"
    components = "main"
    distributions = "jammy"
    architectures = "amd64"
    base_path = "jammy"
    policy = "immediate"
    sync_udebs = True
    sync_installer = True

    # Configure Pulp API clients
    configuration = Configuration(
        username=PULP_API_USERNAME,
        password=PULP_API_PASSWORD,
        host=PULP_API_BASE_URL,
    )
    core_client = ApiClient(configuration)
    deb_client = pulpcore.client.pulp_deb.ApiClient(configuration)

    # Create remote
    remote_api = RemotesAptApi(api_client=deb_client)
    remote = DebAptRemote(
        name=name,
        url=url,
        distributions=distributions,
        components=components,
        architectures=architectures,
        sync_udebs=sync_udebs,
        sync_installer=sync_installer,
        policy=policy
    )
    remote_href = wait_for_href(core_client, remote_api.create(remote))

    # Create repository
    repo_api = RepositoriesAptApi(api_client=deb_client)
    repo_href = wait_for_href(
        core_client, repo_api.create(DebAptRepository(name=name, remote=remote_href)))

    # Sync repository with remote
    sync_url = AptRepositorySyncURL(remote=remote_href)
    remote_href = wait_for_href(core_client, repo_api.sync(
        deb_apt_repository_href=repo_href, apt_repository_sync_url=sync_url))

    # Create publication
    publication_api = PublicationsVerbatimApi(api_client=deb_client)
    publication = DebVerbatimPublication(repository=repo_href)
    publication_href = wait_for_href(
        core_client, publication_api.create(publication))

    # Create distribution
    dist_api = DistributionsAptApi(api_client=deb_client)
    deb_dist = DebAptDistribution(name=name, base_path=base_path)
    dist_href = wait_for_href(core_client, dist_api.create(deb_dist))

    # Update distribution to point to publication
    patched_dist = PatcheddebAptDistribution(publication=publication_href)
    dist_api.partial_update(
        deb_apt_distribution_href=dist_href, patcheddeb_apt_distribution=patched_dist)

There is a version of the glue parts on pypi.
Would you mind trying that one and maybe report any gaps?

1 Like