Channel Catalog Pipeline
========================

This project should treat catalog publishing as a layered model, not a SKU-only export.

The master SKU is still the atomic sellable item, but channel catalogs need richer objects around it:

- Product records: SKU, title, dimensions, attributes, inventory, parent/child variation knowledge.
- Product families: Kitchen Cabinets, Doors, Moulding, Vanities, Shower Doors, Windows, Hardware.
- Product types: Base, Wall, Tall, Glass Door, Moulding, Interior Door, Exterior Door, Vanity Combo.
- Collections/series: Anna Snow White, Anna Caramel Harvest, Belmont Drift White.
- Channel structures: Magento categories, Shopify collections, Shopify product categories, attributes/metafields/options.
- Assets: SKU-only images, collection lifestyle images, collection door images, type diagrams, universal diagrams, swatches.

The channel pipeline should provision these structures first, store remote IDs/GIDs locally, then push products quickly using stored mappings.

Pipeline Stages
---------------

1. Normalize master catalog
   - Import product rows into master_product.
   - Import derived and raw fields into master_product_attribute_value.
   - Derive canonical product family, category, collection, color, brand, assembly type, variation group, and dimensions.
   - Keep channel-specific SKU prefixes out of master records.

2. Build canonical taxonomy intent
   - For each SKU, compute desired placements:
     - family: Kitchen Cabinets
     - product type: Base, Wall, Glass Door, Moulding
     - subtype: Single Door Base Cabinet, Double Door, etc.
     - collection: Anna Caramel Harvest
     - assembly facet: Kitchen Cabinets - Assembled, Kitchen Cabinets - RTA
   - This intent is channel-neutral and should work for future verticals.

3. Provision channel catalog objects
   - Magento:
     - Create missing category paths when needed.
     - Pull category registry after creation.
     - Assign SKUs to multiple categories.
   - Shopify:
     - Create missing manual collections when needed.
     - Pull collection registry after creation.
     - Assign SKUs to multiple collections.
     - Assign one Shopify standard product category where a pulled taxonomy match exists.
   - Attributes:
     - Create Magento product attributes/options or Shopify metafields/options only from approved mappings.
     - Pull registry after creation.

4. Store remote mappings
   - Product taxonomy assignments live in product_channel_taxonomy_assignment.
   - Product channel SKU mappings live in channel_sku_mapping.
   - Attribute aliases live in channel_attribute_alias.
   - Channel registries store pulled remote IDs/GIDs.

5. Push products
   - Product worker reads stored assignments and mappings.
   - Product upsert should not do heavy catalog discovery.
   - Image worker runs separately.
   - Relation worker runs separately for channels that support variations.

6. Pull and reconcile
   - After provisioning or push, pull categories/collections/attributes back into local registries.
   - Reconcile exact remote IDs/GIDs and mark stale/missing targets.

Current Implemented Pieces
--------------------------

- db.channel_catalog_provision:
  - Builds Magento category and Shopify collection/product-category plans.
  - Creates missing Magento categories and Shopify collections.
  - Writes SKU-to-channel taxonomy assignments.
  - Uses master attribute rows for product type paths such as Kitchen Cabinets/Base.

- app.jobs.channel_catalog_provision_enqueue:
  - Runs provisioning as a background job so long Magento/Shopify writes do not time out HTTP/proxy requests.
  - Current implementation is in-process; production hardening should move this to a durable DB-backed queue/job worker.
  - Master feed upload can enqueue this job after import. Default should be dry-run preview unless live creation is explicitly enabled.

- db.master_product_images:
  - Imports image CSV rows.
  - Resolves exact SKU images.
  - Reverses Shopify/channel prefixes where needed.
  - Fans collection lifestyle assets out to all SKUs in a style family.
  - Fans universal line diagrams out to matching item codes across collections.

Required Next Hardening
-----------------------

1. First-class catalog intent

Create an explicit planner output before channel projection:

sku -> canonical targets
  family: Kitchen Cabinets
  type: Base
  subtype: Single Door Base Cabinet
  collection: Anna Caramel Harvest
  brand: Port & Bell
  assembly: Assembled

This prevents Magento and Shopify rules from re-deriving business meaning differently.

2. Rule profiles by vertical

Add rule profiles for:

- Kitchen Cabinets
- Cabinet Accessories
- Interior Doors
- Exterior Doors
- Moulding
- Bathroom Vanities
- Shower Doors
- Windows
- Hardware

Each profile should define:

- required attributes
- category path rules
- Shopify collection rules
- product category search terms
- variation axes
- shared image rules

3. Asset scope model

Today shared images are expanded into SKU image rows. That works for push, but loses why the image matched.

Introduce an asset-scope layer:

catalog_asset
  source_url
  file_name
  asset_kind: sku | collection | type | diagram | swatch | lifestyle
  scope_kind: sku | collection | product_type | family | universal
  scope_value

catalog_asset_assignment
  asset_id
  master_sku
  match_source
  confidence
  sort_order

Then product image rows can remain a materialized push view, while the asset registry explains shared collection/diagram logic.

4. Preview-before-apply everywhere

Every pipeline should expose:

- planned targets
- missing remote objects
- objects to create
- SKU assignment count
- samples
- skipped/reason counts
- apply button

No remote mutation should be required to discover what will happen.

5. Idempotency

Every provision operation must be safe to rerun:

- If remote object exists, link it.
- If local assignment exists, update status/order/source.
- If target is no longer desired, mark stale/inactive only in a controlled prune step.
- Never delete remote catalog objects automatically during provisioning.

6. Separate workers

Keep these workers separate:

- catalog provision worker: categories, collections, attributes, options, local assignments
- product worker: create/update SKU product data
- media worker: upload/replace images
- relation worker: parent/child configurable links
- pull worker: refresh remote state into local registries

7. Durable queueing

Master feed upload should not directly create remote categories, collections, or attributes inside the upload request.

Recommended flow:

master upload finished
  -> enqueue catalog provision preview/apply job
  -> provision job creates or links remote categories/collections/attributes
  -> provision job stores remote IDs/GIDs and SKU assignments
  -> pull/reconcile job refreshes registries
  -> product/media/relation push jobs run from stored mappings

For production, catalog provision jobs should be persisted in a DB-backed queue, not only process memory. A restart should not lose queued work or status history.

Production Run Order
--------------------

1. Import/refresh master catalog.
2. Import/refresh images.
3. Run variation preview and commit selected parent groups.
4. Run channel catalog provision preview.
5. Apply channel catalog provision.
6. Pull Magento categories, Shopify collections, Shopify product categories, and attributes.
7. Run publish readiness.
8. Push products only.
9. Push relations for channels using variations.
10. Push images only.
11. Pull and reconcile.

Guardrails
----------

- Master SKUs remain canonical.
- Channel SKU prefixes are projection rules, not master data.
- Shopify may stay flat while Magento uses configurable parents.
- Channel catalog provisioning creates supporting structures, but product push uses stored IDs/GIDs.
- Shared assets must be traceable to the collection/type/family rule that assigned them.
- Shopify brand collections are allowed when the storefront has brand landing pages; control with CHANNEL_CATALOG_PROVISION_INCLUDE_BRAND_COLLECTIONS.

Catalog Intent Alignment (pre-themes)
-------------------------------------

Canonical taxonomies are DB-owned registries between master SKUs, collection landing pages, and channel structures:

- ``master_brand``, ``master_product_family``, ``master_product_type``, ``master_collection_registry``
- Nullable FKs on ``master_product`` and ``collection_landing_page.collection_registry_id``
- ``db/catalog_intent_planner.py`` — channel-neutral placement intent (category paths, Shopify collection titles, listing path slugs)
- ``db/master_taxonomy_registry.py`` — resolve/propose/ambiguous matching + backfill from master text fields
- ``channel_catalog_provision`` writes ``channel_listing_path_target`` after Magento/Shopify provision (same linking pattern as collection push)

Staged master upload (after planner stable):

- ``POST /api/master-skus/import/csv?preview_only=true`` or ``POST /api/master-skus/import/preview`` — analyze CSV, no SKU writes
- ``POST /api/master-skus/import/csv?preview=true`` — staged import (persist SKUs + intent run, no downstream provision)
- ``GET /api/master-catalog/intent-runs/{id}`` / ``POST .../confirm`` — operator confirm + registry FK linking
- CLI: ``python -m app.jobs.master_catalog_intent_confirm --run-id N``

Taxonomy mapping CRUD (JSON):

- ``POST /api/channel-taxonomy-mappings/upsert``, ``DELETE /api/channel-taxonomy-mappings/{id}``
- ``DELETE /api/product-channel-taxonomy/assignments/{id}`` — soft-delete per-SKU assignment