AI Beauty Chat — External Integration Inventory & SLA Gaps (GAI-8084)

Project: DATA-PLATFORM-SERVICES-GENAI-PA-APP (Product Advisor Bot) Ticket: [GAI-8084] [AIBC][BOT] Document contract details for all external client integrations and hand off to Laurance for PDM agreements Epic: AIBC PROD Stability & Observability Status: Draft — pending review with Laurance Confluence location: Technical & Architecture → AIBC Engineering Initiatives → SRE Onboarding Readiness Checklist → (this page, sibling of "AI Beauty Chat Alerts") Source of truth: data-platform-services-genai-pa-app @ branch dev — src/clients/, src/database/, src/app/logging/kafka/, config/*/*/application.yaml

How to read this page. Every technical claim cites the source file or config key it came from. No SLA number, owner, or traffic figure has been invented — where the repository does not record a value, the cell reads TBD — requires PDM agreement (abbreviated TBD†). This page is the deep-dive behind the parent checklist's §3 (Resiliency), §5 (Vendor/SaaS Integrations), and §9 (Databases); those sections should link here rather than duplicate content.


Table of Contents

  1. Purpose
  2. Architecture / Dependency Overview
  3. Complete Integration Inventory
  4. Critical Chat-Path Integrations
  5. SLA / PDM Agreement Table
  6. Third-Party Dependencies
  7. Unknown Owners
  8. Missing QPS / Traffic Data
  9. Open Questions & Stakeholder Matrix
  10. Handoff Checklist for Laurance
  11. Appendix — Cross-Cutting Config Anomalies

1. Purpose

The AIBC product-advisor bot depends on 28 external integrations — Sephora-internal services, GraphQL subgraphs behind the SEG Apollo Router, MCP tools behind the LiteLLM proxy, third-party vendors, and shared datastores/streaming infrastructure. As of this document, the repository records no formal SLA or support agreement for any of them, and no owning team or PDM for 25 of the 28.

When a downstream degrades in production we have no agreed latency/availability target, no named owner, and no escalation path — which directly drives incident MTTR. This page is the single inventory Laurance will use to sequence PDM conversations and formalise support agreements with each owning team.

What this page is: a complete, evidence-backed dependency inventory with per-integration protocol, endpoints, auth, timeout/retry, and failure behaviour, classified by criticality and by internal-vs-third-party.

What this page is not: it does not set the SLA numbers — those are the output of the PDM conversations. It also does not contain request-volume/QPS data, which is not derivable from the codebase (§8).


2. Architecture / Dependency Overview

2.1 How calls leave the bot

Almost every outbound HTTP call is funneled through a shared async client base, so resilience behaviour is consistent across integrations:

2.2 Gateways

Three integrations are gateways that front other services:

Gateway File Fronts Auth
SEG Apollo Router src/clients/seg.py OIS (purchase history), Product Recs (Constructor pods) x-router-key + Apollo client headers (AIBC_PA_BOT)
LiteLLM MCP proxy src/clients/mcp_base.py Apollo MCP servers: catalog search, events, beauty services Authorization: Bearer
LiteLLM Router (in-process) src/clients/openai_async/client.py Azure OpenAI deployments (chat + embeddings) Per-deployment API key

2.3 Datastores & streaming

2.4 Failure-behaviour taxonomy

The failure evidence sorts every dependency into three customer-facing outcomes:

  1. Hard failure (HTTP 503 / WS FAILURE frame) — request cannot complete. Only: PostgreSQL (src/database/postgres/utils.py:175-189), Redis on the tracking-id path (src/controller/feature_flag_controller.py:259-272), and Azure OpenAI ServiceUnavailableError (src/routes/v1/api_routes.py:411-432).
  2. Empty core output — chat responds but the advisor cannot advise: MySQL, Constructor, Product Graph.
  3. Silent feature degradation — a capability disappears, chat continues: everything else.

Two exceptions surface a visible in-chat message rather than degrading silently: UCM/order-status outage emits FAILURE(code="ORDER_API_UNAVAILABLE") (src/routes/v1/ws_assist_orchestrator.py:158-170), and an LLM timeout/malformed output yields the generic apology UNKNOWN_ERROR (src/agent/product_advisor_agent.py:786-794).


3. Complete Integration Inventory

28 integrations. All endpoints/timeouts from config/<cluster>/<env>/application.yaml; protocol/auth/purpose from the cited source file. Prod endpoints shown; per-environment values are in the Appendix and the per-env config files.

3.1 Catalog & Search

# Integration Source file Protocol Endpoint (prod) Auth Purpose
1 Product Graph src/clients/productgraph.py REST GET product_graph.base_url → productgraph-prod.prod.internal.sephora.com; /productgraph/v3/catalog/products/{id} none (optional user seph-access-token for personalization) Product/SKU details, variations, samples flags
2 Constructor Search src/clients/constructor_search.py REST GET constructor.search_base_url → www.sephora.com/api/v2/catalog/search none visible in code (TBD) Keyword product search, sale queries
3 Category / Gift Finder src/clients/category.py REST GET category_base_url / gift_finder.base_url → www.sephora.com/api/v2/catalog/categories none Category browse, filter/refinement schemas
4 Smart Samples src/clients/smart_samples.py REST GET smart_samples.base_url → www.sephora.com; /gway/productaggregation/smartSamples none Free-sample feed
5 Shade Finder src/clients/shade_finder.py REST GET/POST shade_finder.base_url → product-catalog-service-prod.prod.internal.sephora.com/.../v1/reverseLookUp none (in-cluster) Shade Finder 2.0 — brands, shades, LAB matching
6 Product Recs / Similar Products src/clients/product_recs.py GraphQL via SEG SEG router; pod gpa-similar-products SEG x-router-key FBT / similar-products recommendations

3.2 Orders & Profile

# Integration Source file Protocol Endpoint (prod) Auth Purpose
7 OIS (Order Integration Service) src/clients/order_integration.py GraphQL via SEG SEG router; PurchaseHistory query SEG x-router-key + user JWT Seph-Access-Token Purchase history (paginated)
8 OXS (Order Experience Service) src/clients/purchase_history.py REST GET purchase_history.base_url → order-experience-service-prod.prod.internal.sephora.com optional user JWT Seph-Access-Token Legacy purchase history; OIS selected by kill switch
9 UCM Adapter (WISMO/WISMR) src/clients/ucm_adapter.py REST POST ucm_adapter.base_url → ucm-adapter-service-eus1-prod.prod.internal.sephora.com none by design (identity in body) Order history & order-status detail
10 Profile Preferences src/clients/user_profile.py REST GET profile.preferences_url → profile-preferences-service-eus1-prod.prod.internal.sephora.com/.../v1 tenant/system headers, no token (TBD) Beauty preferences, Color IQ
11 Loyalty Experience Service src/clients/loyalty.py REST GET loyalty.base_url → loyalty.prod.internal.sephora.com identifying headers (X-System: AIBC), no token (TBD) Points, tier, BI rewards, birthday gift

3.3 Promotions & Content

# Integration Source file Protocol Endpoint (prod) Auth Purpose
12 Contentful CXS src/clients/contentful.py REST GET beauty_offers.contentful_base_url → www.sephora.com/api/content/beauty none Beauty-offers config + P13N context
13 P13N src/clients/p13n.py REST GET beauty_offers.p13n_base_url → www.sephora.com; /api/content/p13n/ none Personalized promotions
14 Promo Metadata Service (PMDS) src/clients/promo_metadata.py GraphQL pmds.base_url → promo-metadata-service-eus1-prod.prod.internal.sephora.com none Active promo campaigns + coupons
15 Omni Promo Engine (OPE) src/clients/omni_promo.py GraphQL ope.base_url → omni-promo-service-prod.prod.internal.sephora.com none Promo-code eligibility
16 Adobe Target / Analytics src/clients/adobe_target.py Vendor SDK + REST GET sephora.tt.omtrdc.net/rest/v1/delivery; Analytics smetrics.sephora.com org ID + per-channel property tokens A/B feature flags, impression + A4T analytics

3.4 Platform & Config

# Integration Source file Protocol Endpoint (prod) Auth Purpose
17 SEG Apollo Router src/clients/seg.py GraphQL gateway sephora_enterprise_graph.base_url → apollographql-router-eus1-prod-istio.prod.internal.sephora.com/router x-router-key (SEG_ROUTER_KEY), client AIBC_PA_BOT Gateway for OIS + Product Recs subgraphs
18 pa-user-access-verifier (UAV) src/clients/user_access_verifier.py REST GET user_access.base_url → genai-user-access-verifier-prod.prod.internal.sephora.com none Per-user allowlist / challenger check
19 Entrypoint Prompts service src/clients/entrypoint_prompts.py REST POST/GET entrypoint.base_url → genai-pa-entrypoint-prompts-eus1-prod.prod.internal.sephora.com none Entrypoint suggestion prompts
20 Dotcom Util Service src/clients/dotcom_util_service.py REST GET dotcom.base_url → api-developer.sephora.com/v1/dotcom/util none Kill switches / configuration

3.5 MCP / LLM

# Integration Source file Protocol Endpoint Auth Purpose
21 Azure OpenAI (via LiteLLM Router) src/clients/openai_async/client.py litellm SDK in AZURE_OPENAI_DEPLOYMENTS secret (not in YAML) per-deployment API key All chat completions + embeddings
22 LiteLLM MCP — catalog search src/clients/mcp_catalog_search.py MCP REST (hub + tools/call) in LITELLM_MCP secret; mcp.server_name: mcpApollo (prod) Bearer (LiteLLM key) Keyword catalog search via Apollo MCP
23 LiteLLM MCP — events src/clients/mcp_event_client.py MCP JSON-RPC in LITELLM_MCP secret Bearer Happening-at-Sephora in-store events
24 LiteLLM MCP — beauty services src/clients/mcp_beauty_service_client.py MCP JSON-RPC in LITELLM_MCP secret Bearer Bookable in-store beauty services

3.6 Data & Infrastructure

# Integration Source file Protocol Endpoint (prod) Auth Purpose
25 MySQL product catalog src/database/sql_connection.py, products.py, categories.py SQLAlchemy + aiomysql mysql.db_host → sep-eus1-ab-prod-GenAIProductAdvisor-db02.…oraclevcn.com (OCI) user/password (MYSQL_DB_PASSWORD) Products, SKUs, categories, worlds, brands
26 Azure PostgreSQL src/database/sql_connection.py, src/database/postgres/* asyncpg (PgBouncer :6432) + psycopg (Mem0) postgres.host → sep-eus1-prod-ghc-db01.postgres.database.azure.com user/password (POSTGRES_PASSWORD) Chat history, sessions, identity map, LTM, pgvector
27 Azure Managed Redis src/database/redis/asyncio.py, src/app/cache.py redis-py async (SSL) redis.host → sepeus1prodgenairedis01.eastus.redis.azure.net:10000 password (REDIS_PASSWORD) Response/flag/session caching, tracking IDs
28a Confluent Kafka src/app/logging/kafka/publisher.py, common.py confluent-kafka (SASL_SSL) kafka.bootstrap_servers → lkc-pgpd95-gey026.eastus2.azure.glb.confluent.cloud:9092; topics sephora.genai.pa.prod.chatconversations / .chatmetrics SASL PLAIN (KAFKA_USERNAME/KAFKA_PASSWORD) Interaction/metrics/routine event logging
28b Databricks Lakebase feature store src/clients/clickstream.py psycopg + Azure AD→Databricks OIDC memory.feature_store.host → ep-patient-base-e94c5thk.database.eastus.azuredatabricks.net; table online_fs Azure AD ClientSecretCredential → Databricks token Short-term clickstream context (STM)
28c MLflow on Databricks src/clients/mlflow_tracing.py MLflow SDK (Databricks OAuth) mlflow.databricks.host → adb-8248746818676744.4.azuredatabricks.net; traces to sephora_prod.gen_ai.pa_* Databricks OAuth service principal (rejects DATABRICKS_TOKEN) Tracing of agent turns, tool calls, HTTP calls

The three 28x rows share an index because the ticket scope lists them as one "Data & infra" group; they are three distinct contracts (Confluent, Databricks×2) and are treated separately in §5 and §6.

3.7 Not integrations (documented to prevent re-investigation)


4. Critical Chat-Path Integrations

These six are called on the core path of a normal turn; their failure either hard-fails the request or empties the advisor's core output. Laurance should sequence these first.

Integration Why critical (call flow + failure evidence)
Azure OpenAI (#21) Every turn is an LLM completion; exceptions bubble uncaught through src/agent/services/llm_completion.py. Outage → apology text or HTTP 503 with friendly body (src/routes/v1/api_routes.py:411-432). Nothing else works without it.
Azure PostgreSQL (#26) Chat-history/session load is unconditional per turn; transient errors become UpstreamServiceError("postgres") → HTTP 503 / WS FAILURE (src/database/postgres/utils.py:175-189, src/setup.py:68-88). The only datastore whose outage fails the request.
Azure Managed Redis (#27) Caching fails open (src/app/cache.py:151-172), but FeatureFlagController._get_user_tracking_id converts RedisError → UpstreamServiceError("redis") → 503 (src/controller/feature_flag_controller.py:259-272), and flag resolution runs every interaction. One code path makes it turn-fatal (§9-Q8).
MySQL product catalog (#25) Every recommendation resolves product/SKU data here; empty lookups raise NoProductsFoundError (src/database/products.py:171,212) → no-product fallback; hard outage → HTTP 500. The bot's core deliverable cannot be produced.
Constructor Search (#2) Primary product-retrieval tool in the agent loop; failure collapses to empty results → "no products found" fallback (src/agent/product_advisor_agent.py:795-798). A product advisor that can't search is functionally down.
Product Graph (#1) Enrichment for every recommended product; failures silently drop products from the response (src/clients/productgraph.py:302-313). Full outage empties recommendations even when search succeeds.

Distinguishing behaviour: Azure OpenAI / PostgreSQL / Redis produce hard failures (503/error frame); MySQL / Constructor / Product Graph produce empty core output. Both mean the advisor cannot advise.


5. SLA / PDM Agreement Table

No SLA target exists anywhere in the repository. A repo-wide search for SLA, SLO, p95, p99, availability target, error budget, error rate returned only a product guardrail (docs/agentic-memory-feature-brief.md:425 — "Error Rate <2%", a feature metric, not a dependency SLA) and latency-analysis docs. Therefore every "Required" cell is TBD† and every Agreement status is Absent — these are the outputs Laurance's PDM conversations must produce.

Current timeout/retry values are prod application.yaml + code evidence. TBD† = TBD — requires PDM agreement.

Tier 1 — Critical chat path

Integration Current timeout Current retry Req. avail. Req. p95 Req. p99 Error ceiling Agreement Owner/PDM Open question
Azure OpenAI 45s req / 55s interaction (llm.request_timeout, .agent_interaction_timeout) LiteLLM defaults; no explicit num_retries/fallbacks; content-policy retries 0 (openai_async/client.py:192-197) TBD† TBD† TBD† TBD† Absent TBD Direct-Azure or LiteLLM gateway? (Q3)
Azure PostgreSQL none (pool_recycle 250s) 3 on transient connect (postgres/utils.py) TBD† TBD† TBD† TBD† Absent DBA team (process only), PDM TBD Degraded mode instead of 503?
Azure Managed Redis socket 3s / connect 3s 5 built-in exp backoff (redis/asyncio.py:47-53) TBD† TBD† TBD† TBD† Absent TBD Fix tracking-id 503 path first? (Q8)
MySQL (OCI) none configured none (pool_pre_ping only) TBD† TBD† TBD† TBD† Absent TBD No query timeout; OCI infra vs DB owner?
Constructor Search none in YAML → 3s base 3 TBD† TBD† TBD† TBD† Absent TBD Auth invisible; cnstrc.com never called directly (Q4)
Product Graph 10s (PERF 3s), variant 3000ms 3 TBD† TBD† TBD† TBD† Absent TBD Is per-product silent drop acceptable?

Tier 2 — Supporting dependencies

Integration Current timeout Current retry Req. avail. Req. p95 Req. p99 Error ceiling Agreement Owner/PDM Open question
SEG Apollo Router 5s 3 TBD† TBD† TBD† TBD† Absent TBD DEV/QA product-recs hit prod router — sanctioned? (Q6)
OIS 5s 3 TBD† TBD† TBD† TBD† Absent OIS migration team (order_integration.py:22), PDM TBD No runtime OIS→OXS fallback — intended? OXS decommission date?
OXS 5s 3 TBD† TBD† TBD† TBD† Absent TBD Worth a contract for a service mid-decommission?
UCM Adapter 5s 3 (404 handled as definitive) TBD† TBD† TBD† TBD† Absent TBD Unauthenticated by design — acceptable long-term?
Loyalty 3s 3 TBD† TBD† TBD† TBD† Absent TBD Does a mesh/gateway inject auth? (Q5)
Profile Preferences none → 3s base 3 TBD† TBD† TBD† TBD† Absent TBD Same auth question; hardcoded X-Request-Timestamp
LiteLLM MCP (catalog/events/services) 10s 3 TBD† TBD† TBD† TBD† Absent TBD (LiteLLM platform + Apollo MCP) Which transport (JSON-RPC vs hub REST) is contractual?
Product Recs / Similar Products 5s (SEG) 3 TBD† TBD† TBD† TBD† Absent TBD Is this the ticket's "Similarity Experience"? (Q2)
Contentful CXS 5s 3 TBD† TBD† TBD† TBD† Absent TBD Internal CXS front vs Contentful SaaS — two contracts?
P13N 5s 3 TBD† TBD† TBD† TBD† Absent TBD —
PMDS 10s (PERF 3s) 3 TBD† TBD† TBD† TBD† Absent TBD GraphQL-error raise uncaught at promo.py:188 (Q8)
Omni Promo (OPE) 10s 3 TBD† TBD† TBD† TBD† Absent TBD —
Adobe Target / Analytics 10s SDK 3 (429/5xx only) TBD† TBD† TBD† TBD† Absent TBD (vendor: Adobe) Outage disables all A/B flags fleet-wide — acceptable? Master-contract coverage?
UAV none → 3s base 3 TBD† TBD† TBD† TBD† Absent TBD (GenAI platform) —
Entrypoint Prompts 5s (PERF 3s) 1 (no retry) (entrypoint_prompts.py:27) TBD† TBD† TBD† TBD† Absent TBD (sibling GenAI service) Retries=1 intentional? (Q8)
Dotcom Util none → 3s base 3 TBD† TBD† TBD† TBD† Absent TBD Cache kill-switch state longer than 600s?
Category / Gift Finder none → 3s base 3 TBD† TBD† TBD† TBD† Absent TBD DEV/QA point at prod www.sephora.com — sanctioned? (Q6)
Smart Samples 5s 3 TBD† TBD† TBD† TBD† Absent TBD PERF points at qa3, not perf host (Q7)
Shade Finder 5s 3 TBD† TBD† TBD† TBD† Absent Store Digital — contact explicitly TODO (docs/SHADE_FINDER_REPORT.md:51) Repo already documents owner hunt as unresolved
Databricks Lakebase (STM) 6s 3 + 60s circuit breaker (clickstream.py:40-43) TBD† TBD† TBD† TBD† Absent TBD (vendor: Databricks) Prod uses separate SP/secret — who owns rotation?

Tier 3 — Data / observability

Integration Current timeout Current retry Req. avail. Req. p95 Req. p99 Error ceiling Agreement Owner/PDM Open question
Confluent Kafka session 5000ms none — @suppress_exceptions, no delivery guarantee (publisher.py:40-41) TBD† TBD† TBD† TBD† Absent TBD (vendor: Confluent) What conversation-log loss is tolerable? Currently unbounded/silent
MLflow on Databricks init 10s then disabled none TBD† TBD† TBD† TBD† Absent TBD (vendor: Databricks) trace_sampling_ratio: 1.0 in prod — cost/volume acceptable?

6. Third-Party Dependencies

Contract ownership differs for these — they are (or sit behind) commercial vendor agreements, not internal team commitments. Distinguished here per the ticket's acceptance criteria.

Integration Vendor Endpoint evidence Notes for contract ownership
Azure OpenAI Microsoft AZURE_OPENAI_DEPLOYMENTS secret Topology (direct vs LiteLLM gateway) unresolved — determines Microsoft vs internal-platform ownership (Q3)
Adobe Target / Analytics Adobe sephora.tt.omtrdc.net, smetrics.sephora.com (adobe_target.py) Likely covered by an existing Sephora–Adobe master contract; confirm AIBC is in scope
Confluent Cloud Kafka Confluent *.confluent.cloud:9092 (prod lkc-pgpd95-gey026) Managed streaming; SLA is Confluent's
Databricks — Lakebase Databricks ep-patient-base-e94c5thk.database.eastus.azuredatabricks.net Managed Postgres feature store
Databricks — MLflow Databricks adb-8248746818676744.4.azuredatabricks.net Managed MLflow/tracing (same workspace, two service principals in prod)
Azure PostgreSQL Microsoft *.postgres.database.azure.com Internal DBA team operates; underlying SLA is Azure's
Azure Managed Redis Microsoft *.redis.azure.net Same split — internal ops, Azure SLA
Oracle OCI (MySQL host) Oracle *.oraclevcn.com DB is Sephora-operated; infra SLA is Oracle's

Hybrids — internal endpoint fronting a vendor (flag explicitly to Laurance)

These break the clean internal/third-party split; each likely needs two agreements:


7. Unknown Owners

Per the acceptance criterion "any integration where no owning team can be identified is explicitly listed as an open item rather than omitted."

The repository contains no CODEOWNERS file (root, .github/, full tree — none), despite .github/workflows/README.md:332 referencing one. There are no PDM names and no email addresses; the only concrete Slack channel is #ephemeral-environments (.github/workflows/README.md:325, owned by Platform Engineering, not this team).

Partial ownership leads (team named, no contact)

Integration Lead Evidence
OIS "OIS migration team" src/clients/order_integration.py:22-23
Shade Finder "Store Digital" — explicitly unresolved docs/SHADE_FINDER_REPORT.md:51-52, docs/SHADE_FINDER_SPIKE.md:72
PostgreSQL DBA team (process via DBA Jira board) plans/agentic_memory/plan_details/OPEN-ITEMS.md:74, .github/pull_request_template.md:35

Zero ownership documentation (owner + PDM both TBD) — 25 of 28

Product Graph, Constructor, Category/Gift Finder, Smart Samples, Product Recs, OXS, UCM Adapter, Profile Preferences, Loyalty, Contentful, P13N, PMDS, OPE, Adobe, SEG Apollo Router, UAV, Entrypoint Prompts, Dotcom Util, Azure OpenAI, LiteLLM MCP (×3), MySQL, Redis, Confluent Kafka, Databricks Lakebase, MLflow.

Note: the parent page (SRE Onboarding Readiness Checklist, §5 Vendor/SaaS Integrations) may hold contacts this repo does not — reconcile with its authors before treating a cell as truly unknown.