2to3 RPM content migration fails with: 'NoneType' object has no attribute 'url' (Katello)

Problem: 2to3 RPM content migration fails with: ‘NoneType’ object has no attribute ‘url’ (Katello)

Originally reported here: Content migration: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without an entry in the lazy catalog, pulp2 unit - #3 by jgkootstra - Support - TheForeman

Pulpcore version: Katello 3.18.4, so pulpcore 3.7

Pulp plugins installed and their versions:

  • pulp_rpm: 3.10.0 ? (latest available for Katello 3.18.4)
  • pulp_2to3_migration: 0.11.1 ? (latest available for Katello 3.18.4)

Operating system - distribution and version: CentOS 7.9

Other relevant data:
Backtrace:

Action:
Actions::Pulp3::ContentMigration
Input:
{"services_checked"=>["pulp3"],
 "smart_proxy_id"=>1,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>nil,
 "current_timezone"=>"UTC",
 "current_user_id"=>2,
 "current_organization_id"=>nil,
 "current_location_id"=>nil}
Output:
{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/4b982fbc-7d41-4232-80f2-5a50c5a96f10/",
    "pulp_created"=>"2021-08-12T08:51:09.284+00:00",
    "state"=>"failed",
    "name"=>"pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
    "started_at"=>"2021-08-12T08:51:09.475+00:00",
    "finished_at"=>"2021-08-12T08:52:02.703+00:00",
    "error"=>
     {"traceback"=>
       "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n" +
       "    rv = job.perform()\n" +
       "  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n" +
       "    self._result = self._execute()\n" +
       "  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n" +
       "    return self.func(*self.args, **self.kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py\", line 81, in migrate_from_pulp2\n" +
       "    migrate_content(plan, skip_corrupted=skip_corrupted)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 55, in migrate_content\n" +
       "    plugin.migrator.migrate_content_to_pulp3(skip_corrupted=skip_corrupted)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/migrator.py\", line 150, in migrate_content_to_pulp3\n" +
       "    loop.run_until_complete(dm.create())\n" +
       "  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n" +
       "    return future.result()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 89, in create\n" +
       "    await pipeline\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n" +
       "    await asyncio.gather(*futures)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n" +
       "    await self.run()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 181, in run\n" +
       "    self.migrate_to_pulp3(cmodel, ctype)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 375, in migrate_to_pulp3\n" +
       "    url=os.path.join(remote.url, relative_path),\n",
      "description"=>"'NoneType' object has no attribute 'url'"},
    "worker"=>"/pulp/api/v3/workers/b70d787e-d7a4-493a-baef-b01c4e575709/",
    "child_tasks"=>[],
    "task_group"=>
     "/pulp/api/v3/task-groups/36b9bfab-11b9-41ba-9613-995dd7342c62/",
    "progress_reports"=>
     [{"message"=>"Migrating rpm content to Pulp 3",
       "code"=>"migrating.rpm.content",
       "state"=>"completed",
       "total"=>700,
       "done"=>0},
      {"message"=>"Migrating srpm content to Pulp 3",
       "code"=>"migrating.rpm.content",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Migrating distribution content to Pulp 3",
       "code"=>"migrating.rpm.content",
       "state"=>"failed",
       "total"=>7,
       "done"=>1},
      {"message"=>"Pre-migrating Pulp 2 modulemd content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 srpm content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 srpm content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 modulemd content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 distribution content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 distribution content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 erratum content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 erratum content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Processing Pulp 2 repositories, importers, distributors",
       "code"=>"processing.repositories",
       "state"=>"completed",
       "total"=>5,
       "done"=>5},
      {"message"=>"Pre-migrating Pulp 2 rpm content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 rpm content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 modulemd_defaults content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 modulemd_defaults content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_langpacks content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 yum_repo_metadata_file content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 yum_repo_metadata_file content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_langpacks content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 package_group content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 package_group content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_category content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_category content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_environment content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_environment content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Creating repositories in Pulp 3",
       "code"=>"creating.repositories",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Migrating importers to Pulp 3",
       "code"=>"migrating.importers",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Migrating content to Pulp 3",
       "code"=>"migrating.content",
       "state"=>"failed",
       "total"=>4476,
       "done"=>0}],
    "created_resources"=>
     ["/pulp/api/v3/task-groups/36b9bfab-11b9-41ba-9613-995dd7342c62/"],
    "reserved_resources_record"=>["pulp_2to3_migration"]}],
 "task_groups"=>
  [{"pulp_href"=>
     "/pulp/api/v3/task-groups/36b9bfab-11b9-41ba-9613-995dd7342c62/",
    "description"=>"Migration Sub-tasks",
    "all_tasks_dispatched"=>false,
    "waiting"=>0,
    "skipped"=>0,
    "running"=>0,
    "completed"=>0,
    "canceled"=>0,
    "failed"=>1,
    "group_progress_reports"=>
     [{"message"=>"Repo version creation",
       "code"=>"create.repo_version",
       "total"=>0,
       "done"=>0},
      {"message"=>"Distribution creation",
       "code"=>"create.distribution",
       "total"=>0,
       "done"=>0}]}],
 "poll_attempts"=>{"total"=>22, "failed"=>1}}
Exception:
Katello::Errors::Pulp3Error: 'NoneType' object has no attribute 'url'
Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Action:
Actions::Pulp3::ImportMigration
Input:
{"reimport_all"=>false,
 "dependency"=>Step(2).output,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>nil,
 "current_timezone"=>"UTC",
 "current_user_id"=>2,
 "current_organization_id"=>nil,
 "current_location_id"=>nil}
Output:
{}

@quba42

rpm -q python3-pulp-rpm python3-pulp-2to3-migration
python3-pulp-rpm-3.10.0-1.el7.noarch
python3-pulp-2to3-migration-0.11.1-1.el7.noarch

Latest run still gives these errors:

Action:
Actions::Pulp3::ContentMigration
Input:
{"services_checked"=>["pulp3"],
 "smart_proxy_id"=>1,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>nil,
 "current_timezone"=>"UTC",
 "current_user_id"=>2,
 "current_organization_id"=>nil,
 "current_location_id"=>nil}
Output:
{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/4c8bb756-64b0-4e84-881f-844884842002/",
    "pulp_created"=>"2021-08-12T11:51:22.821+00:00",
    "state"=>"failed",
    "name"=>"pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
    "started_at"=>"2021-08-12T11:52:33.201+00:00",
    "finished_at"=>"2021-08-12T12:17:42.360+00:00",
    "error"=>
     {"traceback"=>
       "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n" +
       "    rv = job.perform()\n" +
       "  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n" +
       "    self._result = self._execute()\n" +
       "  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n" +
       "    return self.func(*self.args, **self.kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py\", line 81, in migrate_from_pulp2\n" +
       "    migrate_content(plan, skip_corrupted=skip_corrupted)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 55, in migrate_content\n" +
       "    plugin.migrator.migrate_content_to_pulp3(skip_corrupted=skip_corrupted)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/migrator.py\", line 150, in migrate_content_to_pulp3\n" +
       "    loop.run_until_complete(dm.create())\n" +
       "  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n" +
       "    return future.result()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 89, in create\n" +
       "    await pipeline\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n" +
       "    await asyncio.gather(*futures)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n" +
       "    await self.run()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 181, in run\n" +
       "    self.migrate_to_pulp3(cmodel, ctype)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 375, in migrate_to_pulp3\n" +
       "    url=os.path.join(remote.url, relative_path),\n",
      "description"=>"'NoneType' object has no attribute 'url'"},
    "worker"=>"/pulp/api/v3/workers/6f6c131b-9cb5-48bd-b164-3fb82c86436e/",
    "child_tasks"=>[],
    "task_group"=>
     "/pulp/api/v3/task-groups/f94861d3-e28a-4ad1-bd92-e434987222bc/",
    "progress_reports"=>
     [{"message"=>"Pre-migrating Pulp 2 modulemd content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 distribution content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 distribution content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 modulemd content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 erratum content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 erratum content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Processing Pulp 2 repositories, importers, distributors",
       "code"=>"processing.repositories",
       "state"=>"completed",
       "total"=>4,
       "done"=>4},
      {"message"=>
        "Pre-migrating Pulp 2 yum_repo_metadata_file content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 rpm content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 rpm content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 srpm content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 srpm content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 modulemd_defaults content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 modulemd_defaults content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_langpacks content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 yum_repo_metadata_file content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_langpacks content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 package_group content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 package_group content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_category content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_category content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_environment content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 package_environment content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Creating repositories in Pulp 3",
       "code"=>"creating.repositories",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Migrating importers to Pulp 3",
       "code"=>"migrating.importers",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Migrating content to Pulp 3",
       "code"=>"migrating.content",
       "state"=>"failed",
       "total"=>43232,
       "done"=>0},
      {"message"=>"Migrating rpm content to Pulp 3",
       "code"=>"migrating.rpm.content",
       "state"=>"completed",
       "total"=>41988,
       "done"=>40151},
      {"message"=>"Migrating srpm content to Pulp 3",
       "code"=>"migrating.rpm.content",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Migrating distribution content to Pulp 3",
       "code"=>"migrating.rpm.content",
       "state"=>"failed",
       "total"=>6,
       "done"=>0}],
    "created_resources"=>
     ["/pulp/api/v3/task-groups/f94861d3-e28a-4ad1-bd92-e434987222bc/"],
    "reserved_resources_record"=>["pulp_2to3_migration"]}],
 "task_groups"=>
  [{"pulp_href"=>
     "/pulp/api/v3/task-groups/f94861d3-e28a-4ad1-bd92-e434987222bc/",
    "description"=>"Migration Sub-tasks",
    "all_tasks_dispatched"=>false,
    "waiting"=>0,
    "skipped"=>0,
    "running"=>0,
    "completed"=>0,
    "canceled"=>0,
    "failed"=>1,
    "group_progress_reports"=>
     [{"message"=>"Repo version creation",
       "code"=>"create.repo_version",
       "total"=>0,
       "done"=>0},
      {"message"=>"Distribution creation",
       "code"=>"create.distribution",
       "total"=>0,
       "done"=>0}]}],
 "poll_attempts"=>{"total"=>118, "failed"=>1}}
Exception:
Katello::Errors::Pulp3Error: 'NoneType' object has no attribute 'url'
Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Action:
Actions::Pulp3::ImportMigration
Input:
{"reimport_all"=>false,
 "dependency"=>Step(2).output,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>nil,
 "current_timezone"=>"UTC",
 "current_user_id"=>2,
 "current_organization_id"=>nil,
 "current_location_id"=>nil}
Output:
{}

Testing if the solution of Issue #7449: F32 sync fails with on_demand policy, DocumentTooLarge - RPM Support - Pulp works for us.

It did not fix the issue.

Possibly related?

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.4/lib/katello/tasks/pulp3_migration.rake:39:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration

Been having this issue for a year or more.

I hope a solution is found soon.

Rebuilding the Katello server is not what I have in mind.

1 Like

It looks like the existing issue Issue #8862: ‘NoneType’ object has no attribute ‘url’ when migrating distribution tree content - Migration Plugin - Pulp
We’ll try to get to it soon.

@goosemania thanks for the response/update

Hi team,

Todays yum update resulted in:

rpm -q python3-pulp-rpm python3-pulp-2to3-migration
python3-pulp-rpm-3.10.0-1.el7.noarch
python3-pulp-2to3-migration-0.11.3-1.el7.noarch

Still the issues remain unsolved.

Regards,

Jan Gerrit Kootstra

@jgkootstra , here is a patch to try out.
https://patch-diff.githubusercontent.com/raw/pulp/pulp-2to3-migration/pull/438.patch

It might not be a final version but all tests seem to pass, it would be great to see if it solves your case.

The fix will be in Katello builds when we fix and release it on PyPI first, you can check the announcements here for pulp-2to3-migration 0.11.z releases. But currently it’s work in progress.

Hi @goosemania ,

Thanks for the update.

I am not a generic user of the patch command-line tool.

Could you please show me how to apply the patch?

If so I can tell you in a few days, whether or not this patch solves our issue.

Regards,

Jan Gerrit Kootstra

@jgkootstra , sure.
You need to go to where pulp-2to3-migraiton is installed, it’s likely /usr/lib/python3.6/site-packages/pulp_2to3_migration, and run the patch command.

# cd  /usr/lib/python3.6/site-packages/pulp_2to3_migration
# patch -p2 < path_to_the_patch

To revert the patch run:

# patch -p2 -R < path_to_the_patch

After applying or unapplying the patch, restart pulp services (or all foreman/katello ones which should restart pulp as well).

2 Likes

@goosemania thanks for the instruction.

The patch seems to slow down the foreman-maintain content prepare task

dynaflow shows the below output, after almost 2 hours:

Id: 44758d63-272c-44e0-97ca-90d54d055248

Label: Actions::Pulp3::ContentMigration

Status: running Cancel

Result: pending

Started at: 2021-09-06 06:48:52 UTC

Ended at:

Plan
Run
Finalize
Execution History
2: Actions::Pulp3::ContentMigration (waiting for Pulp to start the task) [ 9119.42s / 46.66s ] Cancel
4: Actions::Pulp3::ImportMigration (pending) 
sequence		concurrence

Regards,

Jan Gerrit Kootstra

Hmm, I do not think it’s related to the patch. It looks more like the issue with tasking system. The message suggests that the content migration task hasn’t even started. Let me check with Foreman/Katello folks.

In the meantime, could you check if pulp services are running and if there is anything suspicious in logs (in /var/log/messages or via journalctl -u pulpcore-*). The output of the rq info would be helpful as well. Thanks!

after running a foreman package update

a new foreman content prepare does perform as expected.

Now at the stage where I have to cleanup missing rpms.

Great, thanks for the update.

the foreman 2.3.3 completed successful.

2 Likes