============================================================ SL-004 — FINAL CORRECTED INVESTIGATION + PATCH ============================================================ UTC=2026-09-10T15:10:36Z HOST=Azeroth RC=/data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453 === 1. SAFETY === === 2. PRODUCTION BEFORE === stocklab-redis docker.io/library/redis:7-alpine Up 33 hours (healthy) stocklab-db docker.io/library/postgres:16-alpine Up 32 hours (healthy) stocklab-release-qa-redis docker.io/library/redis:7 Up 26 hours stocklab-frontend localhost/stocklab-frontend:release-qa-20260909-182234 Up 24 hours (healthy) stocklab-backend localhost/stocklab-backend:screener-percent-final-20260909-194639 Up 21 hours (healthy) stocklab-worker localhost/stocklab-backend:screener-percent-final-20260909-194639 Up 21 hours (healthy) stocklab-beat localhost/stocklab-backend:screener-percent-final-20260909-194639 Up 21 hours Ports: LISTEN 0 2048 127.0.0.1:18000 0.0.0.0:* users:(("uvicorn",pid=1574355,fd=11)) LISTEN 0 511 127.0.0.1:6379 0.0.0.0:* users:(("redis-server",pid=1133711,fd=6)) LISTEN 0 200 127.0.0.1:5432 0.0.0.0:* users:(("postgres",pid=1141225,fd=6)) LISTEN 0 511 127.0.0.1:13000 0.0.0.0:* users:(("next-server (v1",pid=1497838,fd=18)) Restart counters: stocklab-backend restart=0 stocklab-worker restart=0 stocklab-beat restart=0 stocklab-db restart=0 stocklab-redis restart=0 stocklab-frontend restart=0 === 3. FIND REAL MODEL / DATACQUALITY / FIELD CONFLICT DEFINITIONS === /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-40- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-41- reason = f"Highest-priority available source: {selected_tier}" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-42- if conflicting: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-43- reason += f"; disagreement > {tolerance_pct:.1%} with a lower-priority source — see data_quality row" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-44- return ConflictResolution(selected_value, selected_tier, reason, is_conflicting=conflicting) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-45- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-46- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-47-@dataclass(frozen=True) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102:48:class FieldConflict: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-49- field: str /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-50- value_a: Optional[float] /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-51- value_b: Optional[float] /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-52- selected_value: Optional[float] /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-53- selection_reason: str /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-54- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-55- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-56-def resolve_line_items( /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-57- primary: dict, secondary: Optional[dict], fields: list[str], /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-58- primary_tier: str, secondary_tier: Optional[str] = None, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-59-) -> tuple[dict, list[FieldConflict]]: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-60- """ /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-61- StockLab overhaul, final engineering pass, Part A6 -- the statement-level counterpart to /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-62- `resolve_conflict()` above, which only ever resolves ONE field at a time. This applies it /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-63- across a whole LineItems-shaped dict (`app/adapters/schemas.py`) so a real ingestion call site /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-64- can resolve an entire income statement / balance sheet / cash flow / shares record between two /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-65- providers in one call, without each call site re-implementing the per-field loop. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-66- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-67- `primary`/`secondary` are dicts of {field_name: value} for the SAME (security, period) from /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-68- two different provider adapters -- the same shape `ingest.py`'s existing single-source loop /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-69- already reads from `period.line_items`. `fields` is the list of field names to consider (the /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-70- caller passes the exact tuple it already iterates for that statement type, e.g. ingest.py's /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-71- existing `("revenue", "cogs", ...)` tuples -- kept as an explicit argument here rather than /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-72- hardcoded, so this function has no per-statement-type knowledge of its own). /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-73- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-74- Returns `(resolved, conflicts)`: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-75- - `resolved` contains exactly the fields at least one source provided a value for (mirrors the /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-76- existing `if field in li: setattr(...)` "only touch fields that were actually provided" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-77- contract every single-source call site already relies on -- a field neither source reported /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-78- is simply absent from `resolved`, never defaulted to 0/None-overwrite). /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-79- - `conflicts` lists only the fields where `resolve_conflict()` flagged real (> tolerance_pct) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-80- disagreement -- the caller persists these as `DataQuality` rows; the vast majority of fields /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-81- on any real statement are expected to end up here with zero conflicts. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-82- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-83- When `secondary` is None (no second source was fetched for this call -- the default, -- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-95- continue /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-96- candidates = [(primary_tier, val_a)] /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-97- if secondary is not None and secondary_tier is not None: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-98- candidates.append((secondary_tier, val_b)) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-99- result = resolve_conflict(candidates) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-100- if result.selected_value is not None: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-101- resolved[field] = result.selected_value /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-102- if result.is_conflicting: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102:103: conflicts.append(FieldConflict(field, val_a, val_b, result.selected_value, result.selection_reason)) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-104- return resolved, conflicts /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-105- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-106- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-107-def merge_statement_line_items( /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-108- income: dict, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-109- balance: Optional[dict] = None, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-110- cash_flow: Optional[dict] = None, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-111-) -> dict: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-112- """Merge one provider's three statement responses for the SAME reporting period into the one /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-113- flat line-items dict `ingest_security()` writes to the four statement tables. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-114- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-115- AUDIT FIX (StockLab final engineering pass, Part A9 — a critical bug found while wiring TTM). /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-116- Before this existed, `ingest_security()` called ONLY `adapter.get_income_statements()` and then /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-117- wrote `BalanceSheet`, `CashFlow` and `Shares` rows out of that single response's line items. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-118- `adapter.get_balance_sheets()` and `adapter.get_cash_flows()` were never called anywhere in the /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-119- application (verified by grep across `app/` and `scripts/`: the only remaining matches were /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-120- comments). Against `DemoDataAdapter` this was invisible, because its three statement methods /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-121- all return the SAME fully-merged dict containing every field. Against `FMPAdapter` — the real /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-122- primary provider — `get_income_statements()` maps only `_INCOME_MAP`, so every balance-sheet, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-123- cash-flow and share-count column would have been written NULL for real data. See /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-124- docs/AUDIT_INGEST_STATEMENTS_A9.md. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-125- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-126- Merge rule: first non-None value wins, in income -> balance -> cash-flow order. The income /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-127- statement is therefore authoritative for fields more than one endpoint reports (FMP's /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-128- cash-flow response also carries `netIncome`, for example), and a `None` from a later statement /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-129- can never erase a real value from an earlier one. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-130- """ /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-131- merged: dict = {} /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-132- for source in (income, balance, cash_flow): /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-133- if not source: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-134- continue /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-135- for key, value in source.items(): /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-136- if value is None: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-137- continue /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-144102-138- if merged.get(key) is None: -- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-40- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-41- reason = f"Highest-priority available source: {selected_tier}" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-42- if conflicting: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-43- reason += f"; disagreement > {tolerance_pct:.1%} with a lower-priority source — see data_quality row" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-44- return ConflictResolution(selected_value, selected_tier, reason, is_conflicting=conflicting) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-45- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-46- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-47-@dataclass(frozen=True) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py:48:class FieldConflict: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-49- field: str /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-50- value_a: Optional[float] /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-51- value_b: Optional[float] /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-52- selected_value: Optional[float] /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-53- selection_reason: str /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-54- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-55- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-56-def resolve_line_items( /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-57- primary: dict, secondary: Optional[dict], fields: list[str], /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-58- primary_tier: str, secondary_tier: Optional[str] = None, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-59-) -> tuple[dict, list[FieldConflict]]: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-60- """ /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-61- StockLab overhaul, final engineering pass, Part A6 -- the statement-level counterpart to /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-62- `resolve_conflict()` above, which only ever resolves ONE field at a time. This applies it /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-63- across a whole LineItems-shaped dict (`app/adapters/schemas.py`) so a real ingestion call site /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-64- can resolve an entire income statement / balance sheet / cash flow / shares record between two /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-65- providers in one call, without each call site re-implementing the per-field loop. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-66- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-67- `primary`/`secondary` are dicts of {field_name: value} for the SAME (security, period) from /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-68- two different provider adapters -- the same shape `ingest.py`'s existing single-source loop /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-69- already reads from `period.line_items`. `fields` is the list of field names to consider (the /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-70- caller passes the exact tuple it already iterates for that statement type, e.g. ingest.py's /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-71- existing `("revenue", "cogs", ...)` tuples -- kept as an explicit argument here rather than /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-72- hardcoded, so this function has no per-statement-type knowledge of its own). /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-73- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-74- Returns `(resolved, conflicts)`: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-75- - `resolved` contains exactly the fields at least one source provided a value for (mirrors the /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-76- existing `if field in li: setattr(...)` "only touch fields that were actually provided" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-77- contract every single-source call site already relies on -- a field neither source reported /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-78- is simply absent from `resolved`, never defaulted to 0/None-overwrite). /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-79- - `conflicts` lists only the fields where `resolve_conflict()` flagged real (> tolerance_pct) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-80- disagreement -- the caller persists these as `DataQuality` rows; the vast majority of fields /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-81- on any real statement are expected to end up here with zero conflicts. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-82- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-83- When `secondary` is None (no second source was fetched for this call -- the default, -- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-95- continue /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-96- candidates = [(primary_tier, val_a)] /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-97- if secondary is not None and secondary_tier is not None: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-98- candidates.append((secondary_tier, val_b)) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-99- result = resolve_conflict(candidates) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-100- if result.selected_value is not None: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-101- resolved[field] = result.selected_value /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-102- if result.is_conflicting: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py:103: conflicts.append(FieldConflict(field, val_a, val_b, result.selected_value, result.selection_reason)) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-104- return resolved, conflicts /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-105- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-106- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-107-def merge_statement_line_items( /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-108- income: dict, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-109- balance: Optional[dict] = None, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-110- cash_flow: Optional[dict] = None, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-111-) -> dict: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-112- """Merge one provider's three statement responses for the SAME reporting period into the one /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-113- flat line-items dict `ingest_security()` writes to the four statement tables. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-114- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-115- AUDIT FIX (StockLab final engineering pass, Part A9 — a critical bug found while wiring TTM). /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-116- Before this existed, `ingest_security()` called ONLY `adapter.get_income_statements()` and then /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-117- wrote `BalanceSheet`, `CashFlow` and `Shares` rows out of that single response's line items. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-118- `adapter.get_balance_sheets()` and `adapter.get_cash_flows()` were never called anywhere in the /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-119- application (verified by grep across `app/` and `scripts/`: the only remaining matches were /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-120- comments). Against `DemoDataAdapter` this was invisible, because its three statement methods /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-121- all return the SAME fully-merged dict containing every field. Against `FMPAdapter` — the real /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-122- primary provider — `get_income_statements()` maps only `_INCOME_MAP`, so every balance-sheet, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-123- cash-flow and share-count column would have been written NULL for real data. See /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-124- docs/AUDIT_INGEST_STATEMENTS_A9.md. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-125- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-126- Merge rule: first non-None value wins, in income -> balance -> cash-flow order. The income /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-127- statement is therefore authoritative for fields more than one endpoint reports (FMP's /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-128- cash-flow response also carries `netIncome`, for example), and a `None` from a later statement /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-129- can never erase a real value from an earlier one. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-130- """ /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-131- merged: dict = {} /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-132- for source in (income, balance, cash_flow): /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-133- if not source: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-134- continue /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-135- for key, value in source.items(): /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-136- if value is None: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-137- continue /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py-138- if merged.get(key) is None: -- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-40- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-41- reason = f"Highest-priority available source: {selected_tier}" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-42- if conflicting: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-43- reason += f"; disagreement > {tolerance_pct:.1%} with a lower-priority source — see data_quality row" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-44- return ConflictResolution(selected_value, selected_tier, reason, is_conflicting=conflicting) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-45- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-46- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-47-@dataclass(frozen=True) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740:48:class FieldConflict: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-49- field: str /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-50- value_a: Optional[float] /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-51- value_b: Optional[float] /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-52- selected_value: Optional[float] /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-53- selection_reason: str /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-54- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-55- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-56-def resolve_line_items( /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-57- primary: dict, secondary: Optional[dict], fields: list[str], /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-58- primary_tier: str, secondary_tier: Optional[str] = None, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-59-) -> tuple[dict, list[FieldConflict]]: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-60- """ /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-61- StockLab overhaul, final engineering pass, Part A6 -- the statement-level counterpart to /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-62- `resolve_conflict()` above, which only ever resolves ONE field at a time. This applies it /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-63- across a whole LineItems-shaped dict (`app/adapters/schemas.py`) so a real ingestion call site /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-64- can resolve an entire income statement / balance sheet / cash flow / shares record between two /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-65- providers in one call, without each call site re-implementing the per-field loop. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-66- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-67- `primary`/`secondary` are dicts of {field_name: value} for the SAME (security, period) from /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-68- two different provider adapters -- the same shape `ingest.py`'s existing single-source loop /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-69- already reads from `period.line_items`. `fields` is the list of field names to consider (the /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-70- caller passes the exact tuple it already iterates for that statement type, e.g. ingest.py's /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-71- existing `("revenue", "cogs", ...)` tuples -- kept as an explicit argument here rather than /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-72- hardcoded, so this function has no per-statement-type knowledge of its own). /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-73- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-74- Returns `(resolved, conflicts)`: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-75- - `resolved` contains exactly the fields at least one source provided a value for (mirrors the /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-76- existing `if field in li: setattr(...)` "only touch fields that were actually provided" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-77- contract every single-source call site already relies on -- a field neither source reported /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-78- is simply absent from `resolved`, never defaulted to 0/None-overwrite). /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-79- - `conflicts` lists only the fields where `resolve_conflict()` flagged real (> tolerance_pct) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-80- disagreement -- the caller persists these as `DataQuality` rows; the vast majority of fields /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-81- on any real statement are expected to end up here with zero conflicts. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-82- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-83- When `secondary` is None (no second source was fetched for this call -- the default, -- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-95- continue /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-96- candidates = [(primary_tier, val_a)] /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-97- if secondary is not None and secondary_tier is not None: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-98- candidates.append((secondary_tier, val_b)) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-99- result = resolve_conflict(candidates) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-100- if result.selected_value is not None: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-101- resolved[field] = result.selected_value /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-102- if result.is_conflicting: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740:103: conflicts.append(FieldConflict(field, val_a, val_b, result.selected_value, result.selection_reason)) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-104- return resolved, conflicts /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-105- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-106- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-107-def merge_statement_line_items( /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-108- income: dict, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-109- balance: Optional[dict] = None, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-110- cash_flow: Optional[dict] = None, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-111-) -> dict: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-112- """Merge one provider's three statement responses for the SAME reporting period into the one /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-113- flat line-items dict `ingest_security()` writes to the four statement tables. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-114- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-115- AUDIT FIX (StockLab final engineering pass, Part A9 — a critical bug found while wiring TTM). /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-116- Before this existed, `ingest_security()` called ONLY `adapter.get_income_statements()` and then /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-117- wrote `BalanceSheet`, `CashFlow` and `Shares` rows out of that single response's line items. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-118- `adapter.get_balance_sheets()` and `adapter.get_cash_flows()` were never called anywhere in the /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-119- application (verified by grep across `app/` and `scripts/`: the only remaining matches were /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-120- comments). Against `DemoDataAdapter` this was invisible, because its three statement methods /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-121- all return the SAME fully-merged dict containing every field. Against `FMPAdapter` — the real /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-122- primary provider — `get_income_statements()` maps only `_INCOME_MAP`, so every balance-sheet, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-123- cash-flow and share-count column would have been written NULL for real data. See /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-124- docs/AUDIT_INGEST_STATEMENTS_A9.md. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-125- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-126- Merge rule: first non-None value wins, in income -> balance -> cash-flow order. The income /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-127- statement is therefore authoritative for fields more than one endpoint reports (FMP's /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-128- cash-flow response also carries `netIncome`, for example), and a `None` from a later statement /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-129- can never erase a real value from an earlier one. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-130- """ /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-131- merged: dict = {} /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-132- for source in (income, balance, cash_flow): /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-133- if not source: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-134- continue /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-135- for key, value in source.items(): /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-136- if value is None: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-137- continue /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-180740-138- if merged.get(key) is None: -- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-9-from __future__ import annotations /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-10- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-11-from dataclasses import dataclass, field /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-12-from datetime import date /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-13-from enum import Enum /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-14-from typing import Optional /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-15- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-16- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py:17:class DataQualityStatus(str, Enum): /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-18- VERIFIED = "VERIFIED" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-19- CALCULATED = "CALCULATED" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-20- ESTIMATED = "ESTIMATED" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-21- ASSUMPTION = "ASSUMPTION" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-22- MISSING = "MISSING" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-23- STALE = "STALE" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-24- CONFLICTING = "CONFLICTING" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-25- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-26- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-27-class Applicability(str, Enum): /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-28- APPLICABLE = "APPLICABLE" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-29- LIMITED = "LIMITED" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-30- NOT_MEANINGFUL = "NOT_MEANINGFUL" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-31- INSUFFICIENT_DATA = "INSUFFICIENT_DATA" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-32- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-33- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-34-class MarketCapBucket(str, Enum): /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-35- LARGE = "LARGE" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-36- MID = "MID" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-37- SMALL = "SMALL" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-38- MICRO = "MICRO" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-39- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-40- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-41-# Sentinel for "not meaningful" / "cannot compute" so callers never mistake it for 0. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-42-NM = "N/M" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-43- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-44- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-45-@dataclass(frozen=True) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-46-class MetricResult: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-47- """The result of computing one metric for one security at one point in time.""" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-48- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-49- key: str /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-50- value: Optional[float] # None when status is MISSING/NOT_MEANINGFUL/INSUFFICIENT_DATA /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-51- status: DataQualityStatus /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/types.py-52- applicability: Applicability -- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-40- DataQualityStatus.CONFLICTING: 15.0, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-41- DataQualityStatus.MISSING: 0.0, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-42-} /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-43- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-44-DEFAULT_WEIGHTS = {"source_tier": 0.6, "field_status": 0.4} /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-45- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-46- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-47-@dataclass /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py:48:class DataQualityComponent: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-49- name: str /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-50- weight: float /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-51- raw_score: float /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-52- contribution: float /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-53- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-54- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-55-@dataclass /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py:56:class DataQualityResult: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-57- value: Optional[float] /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-58- components: list[DataQualityComponent] = field(default_factory=list) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-59- conflicting_field_count: int = 0 /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-60- unknown_source_tiers: list[str] = field(default_factory=list) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-61- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-62- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-63-def compute_data_quality_score( /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-64- *, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-65- source_tiers: list[str], /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-66- field_statuses: list[DataQualityStatus], /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-67- weights: Optional[dict[str, float]] = None, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-68-) -> DataQualityResult: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-69- weights = dict(weights or DEFAULT_WEIGHTS) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-70- raw: dict[str, float] = {} /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-71- unknown_tiers = sorted({t for t in source_tiers if t not in SOURCE_TIER_SCORE}) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-72- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-73- if source_tiers: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-74- raw["source_tier"] = sum(SOURCE_TIER_SCORE.get(t, 0.0) for t in source_tiers) / len(source_tiers) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-75- if field_statuses: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-76- raw["field_status"] = sum(_STATUS_SCORE.get(s, 0.0) for s in field_statuses) / len(field_statuses) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-77- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-78- available = {k: v for k, v in raw.items() if k in weights} /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-79- conflicting = sum(1 for s in field_statuses if s == DataQualityStatus.CONFLICTING) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-80- if not available: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-81- return DataQualityResult(value=None, conflicting_field_count=conflicting, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-82- unknown_source_tiers=unknown_tiers) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-83- total_w = sum(weights[k] for k in available) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-84- components = [ /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-85- DataQualityComponent(name=k, weight=weights[k] / total_w, raw_score=v, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-86- contribution=(weights[k] / total_w) * v) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-87- for k, v in available.items() /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-88- ] /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-89- value = sum(c.contribution for c in components) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-90- return DataQualityResult(value=value, components=components, conflicting_field_count=conflicting, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/engines/scoring/data_quality.py-91- unknown_source_tiers=unknown_tiers) -- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-263- """Persist resolve_line_items()'s flagged disagreements as DataQuality rows (StockLab /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-264- overhaul, Part A6). entity_id is the owning FinancialPeriod's id for every statement type -- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-265- IncomeStatement/BalanceSheet/CashFlow/Shares are each a 1:1 extension of FinancialPeriod with /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-266- no independently meaningful id of their own, so `entity_table` (e.g. "income_statements") is /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-267- what disambiguates which statement a given row's conflicting field belongs to, not entity_id /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-268- itself. A deliberate simplification, not an oversight -- see docs/AUDIT_PROVIDER_CONFLICT_A6.md /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-269- for the alternative (a dedicated per-statement-row id) and why it wasn't needed for this.""" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-270- for c in conflicts: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py:271: db.add(DataQuality( /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-272- entity_table=entity_table, entity_id=entity_id, field=c.field, status="CONFLICTING", /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-273- source_a_id=source_a.id, value_a=c.value_a, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-274- source_b_id=source_b.id if source_b is not None else None, value_b=c.value_b, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-275- selected_value=c.selected_value, selected_source_id=source_a.id if c.selected_value == c.value_a else (source_b.id if source_b is not None else None), /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-276- selection_reason=c.selection_reason, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-277- )) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-278- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-279- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-280-def _fetch_statements_by_period(adapter, method, ticker: str, label: str) -> dict: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-281- """Fetch one statement type and index it by (period_end, period_type). /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-282- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-283- Returns an empty dict -- never raises -- when the adapter has not implemented this statement /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-284- (EODHDAdapter's balance-sheet/cash-flow methods are `NotImplementedError` today). An adapter /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-285- with a partial implementation must degrade to "that statement's columns stay empty for this /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-286- run", exactly as before this fix, rather than crashing every ingestion task. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-287- """ /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-288- try: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-289- rows = method(ticker) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-290- except NotImplementedError: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-291- logger.warning( /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-292- "ingest.security.statement_not_implemented", /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-293- ticker=ticker, provider=adapter.name, statement=label, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-294- ) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-295- return {} /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-296- return {(r.period_end, r.period_type): r.line_items for r in rows} /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-297- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-298- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-299- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-300-def _ingest_prices( /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-301- db: Session, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-302- adapter: ProviderAdapter, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-303- security_id: str, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-304- ticker: str, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-305- source: Source, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py-306-) -> int: -- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-263- """Persist resolve_line_items()'s flagged disagreements as DataQuality rows (StockLab /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-264- overhaul, Part A6). entity_id is the owning FinancialPeriod's id for every statement type -- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-265- IncomeStatement/BalanceSheet/CashFlow/Shares are each a 1:1 extension of FinancialPeriod with /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-266- no independently meaningful id of their own, so `entity_table` (e.g. "income_statements") is /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-267- what disambiguates which statement a given row's conflicting field belongs to, not entity_id /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-268- itself. A deliberate simplification, not an oversight -- see docs/AUDIT_PROVIDER_CONFLICT_A6.md /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-269- for the alternative (a dedicated per-statement-row id) and why it wasn't needed for this.""" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-270- for c in conflicts: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740:271: db.add(DataQuality( /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-272- entity_table=entity_table, entity_id=entity_id, field=c.field, status="CONFLICTING", /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-273- source_a_id=source_a.id, value_a=c.value_a, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-274- source_b_id=source_b.id if source_b is not None else None, value_b=c.value_b, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-275- selected_value=c.selected_value, selected_source_id=source_a.id if c.selected_value == c.value_a else (source_b.id if source_b is not None else None), /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-276- selection_reason=c.selection_reason, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-277- )) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-278- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-279- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-280-def _fetch_statements_by_period(adapter, method, ticker: str, label: str) -> dict: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-281- """Fetch one statement type and index it by (period_end, period_type). /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-282- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-283- Returns an empty dict -- never raises -- when the adapter has not implemented this statement /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-284- (EODHDAdapter's balance-sheet/cash-flow methods are `NotImplementedError` today). An adapter /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-285- with a partial implementation must degrade to "that statement's columns stay empty for this /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-286- run", exactly as before this fix, rather than crashing every ingestion task. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-287- """ /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-288- try: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-289- rows = method(ticker) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-290- except NotImplementedError: /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-291- logger.warning( /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-292- "ingest.security.statement_not_implemented", /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-293- ticker=ticker, provider=adapter.name, statement=label, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-294- ) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-295- return {} /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-296- return {(r.period_end, r.period_type): r.line_items for r in rows} /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-297- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-298- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-299- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-300-def _ingest_prices( /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-301- db: Session, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-302- adapter: ProviderAdapter, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-303- security_id: str, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-304- ticker: str, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-305- source: Source, /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-180740-306-) -> int: -- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-21- provider_tier: Mapped[str] = mapped_column(String(16)) # OFFICIAL_FILING | PRIMARY | SECONDARY | CALCULATED /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-22- is_demo: Mapped[bool] = mapped_column(Boolean, default=False, index=True) # DATA_SOURCES.md §9 — hard separation /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-23- retrieved_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now()) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-24- endpoint: Mapped[Optional[str]] = mapped_column(String(256), nullable=True) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-25- raw_payload: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True) # deliberate JSON exception, DATA_MODEL.md /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-26- content_hash: Mapped[Optional[str]] = mapped_column(String(64), nullable=True, index=True) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-27- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-28- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py:29:class DataQuality(Base, TimestampMixin): /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-30- """Per-data-point quality/conflict record (spec §9).""" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-31- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-32- __tablename__ = "data_quality" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-33- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-34- id: Mapped[str] = uuid_pk() /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-35- entity_table: Mapped[str] = mapped_column(String(64)) # e.g. "income_statements" /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-36- entity_id: Mapped[str] = mapped_column(String(64), index=True) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-37- field: Mapped[str] = mapped_column(String(64)) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-38- status: Mapped[str] = mapped_column(String(16)) # VERIFIED|CALCULATED|ESTIMATED|ASSUMPTION|MISSING|STALE|CONFLICTING /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-39- source_a_id: Mapped[Optional[str]] = mapped_column(ForeignKey("sources.id"), nullable=True) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-40- value_a: Mapped[Optional[float]] = mapped_column(nullable=True) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-41- source_b_id: Mapped[Optional[str]] = mapped_column(ForeignKey("sources.id"), nullable=True) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-42- value_b: Mapped[Optional[float]] = mapped_column(nullable=True) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-43- selected_value: Mapped[Optional[float]] = mapped_column(nullable=True) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-44- selected_source_id: Mapped[Optional[str]] = mapped_column(ForeignKey("sources.id"), nullable=True) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-45- selection_reason: Mapped[Optional[str]] = mapped_column(nullable=True) /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-46- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-47- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-48-class RevokedToken(Base): /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-49- """Revoked/rotated refresh-token store (StockLab overhaul, final engineering pass, Part A4). /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-50- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-51- Deliberately scoped to refresh tokens only -- access tokens are short-lived /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-52- (Settings.ACCESS_TOKEN_EXPIRE_MINUTES, 30 min default) and validated on every authenticated /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-53- request (app/api/v1/deps.py::get_current_user); adding a DB round-trip to that path for every /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-54- request would be the "incompatible auth rewrite" this pass's instructions said to avoid, for a /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-55- marginal gain given how short-lived access tokens already are. Refresh tokens are long-lived /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-56- (Settings.REFRESH_TOKEN_EXPIRE_DAYS, 14 days default) and used rarely (once per session /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-57- renewal) -- the theft window that actually matters, and where a DB check is cheap relative to /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-58- how often it runs. /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-59- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-60- No FK constraint enforced on user_id on purpose: a token can outlive its user row in edge cases /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-61- (revoke-then-delete-account ordering), and this table's only real purpose is jti lookup, not /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/models/governance.py-62- joining back to User -- keeping it nullable/unconstrained avoids a fragile cross-table === 4. EXACT RESOLVER === 44- return ConflictResolution(selected_value, selected_tier, reason, is_conflicting=conflicting) 45- 46- 47-@dataclass(frozen=True) 48-class FieldConflict: 49- field: str 50- value_a: Optional[float] 51- value_b: Optional[float] 52- selected_value: Optional[float] 53- selection_reason: str 54- 55- 56:def resolve_line_items( 57- primary: dict, secondary: Optional[dict], fields: list[str], 58- primary_tier: str, secondary_tier: Optional[str] = None, 59-) -> tuple[dict, list[FieldConflict]]: 60- """ 61- StockLab overhaul, final engineering pass, Part A6 -- the statement-level counterpart to 62- `resolve_conflict()` above, which only ever resolves ONE field at a time. This applies it 63- across a whole LineItems-shaped dict (`app/adapters/schemas.py`) so a real ingestion call site 64- can resolve an entire income statement / balance sheet / cash flow / shares record between two 65- providers in one call, without each call site re-implementing the per-field loop. 66- 67- `primary`/`secondary` are dicts of {field_name: value} for the SAME (security, period) from 68- two different provider adapters -- the same shape `ingest.py`'s existing single-source loop 69- already reads from `period.line_items`. `fields` is the list of field names to consider (the 70- caller passes the exact tuple it already iterates for that statement type, e.g. ingest.py's 71- existing `("revenue", "cogs", ...)` tuples -- kept as an explicit argument here rather than 72- hardcoded, so this function has no per-statement-type knowledge of its own). 73- 74- Returns `(resolved, conflicts)`: 75- - `resolved` contains exactly the fields at least one source provided a value for (mirrors the 76- existing `if field in li: setattr(...)` "only touch fields that were actually provided" 77- contract every single-source call site already relies on -- a field neither source reported 78- is simply absent from `resolved`, never defaulted to 0/None-overwrite). 79- - `conflicts` lists only the fields where `resolve_conflict()` flagged real (> tolerance_pct) 80- disagreement -- the caller persists these as `DataQuality` rows; the vast majority of fields 81- on any real statement are expected to end up here with zero conflicts. 82- 83- When `secondary` is None (no second source was fetched for this call -- the default, 84- `Settings.PROVIDER_SECONDARY = "NONE"`), this degrades to exactly "use primary's value for 85- every field primary provided", byte-identical in output to what single-source ingestion 86- already does. There is deliberately no behavior change for that, the common, case -- dual- 87- source resolution only activates when a caller actually has two dicts to pass in. 88- """ 89- resolved: dict = {} 90- conflicts: list[FieldConflict] = [] 91- for field in fields: 92- val_a = primary.get(field) 93- val_b = secondary.get(field) if secondary is not None else None 94- if val_a is None and val_b is None: 95- continue 96- candidates = [(primary_tier, val_a)] 97- if secondary is not None and secondary_tier is not None: 98- candidates.append((secondary_tier, val_b)) 99- result = resolve_conflict(candidates) 100- if result.selected_value is not None: 101- resolved[field] = result.selected_value 102- if result.is_conflicting: 103- conflicts.append(FieldConflict(field, val_a, val_b, result.selected_value, result.selection_reason)) 104- return resolved, conflicts 105- 106- 107-def merge_statement_line_items( 108- income: dict, 109- balance: Optional[dict] = None, 110- cash_flow: Optional[dict] = None, 111-) -> dict: 112- """Merge one provider's three statement responses for the SAME reporting period into the one 113- flat line-items dict `ingest_security()` writes to the four statement tables. 114- 115- AUDIT FIX (StockLab final engineering pass, Part A9 — a critical bug found while wiring TTM). 116- Before this existed, `ingest_security()` called ONLY `adapter.get_income_statements()` and then 117- wrote `BalanceSheet`, `CashFlow` and `Shares` rows out of that single response's line items. 118- `adapter.get_balance_sheets()` and `adapter.get_cash_flows()` were never called anywhere in the 119- application (verified by grep across `app/` and `scripts/`: the only remaining matches were 120- comments). Against `DemoDataAdapter` this was invisible, because its three statement methods 121- all return the SAME fully-merged dict containing every field. Against `FMPAdapter` — the real 122- primary provider — `get_income_statements()` maps only `_INCOME_MAP`, so every balance-sheet, 123- cash-flow and share-count column would have been written NULL for real data. See 124- docs/AUDIT_INGEST_STATEMENTS_A9.md. 125- 126- Merge rule: first non-None value wins, in income -> balance -> cash-flow order. The income === 5. EXACT VALIDATOR === 205-NON_NEGATIVE_LINE_ITEMS = ( 206- "revenue", "total_assets", "current_assets", "current_liabilities", "cash_and_equivalents", 207- "short_term_investments", "total_debt", "short_term_debt", "long_term_debt", 208- "lease_liabilities", "goodwill", "intangible_assets", "inventory", "receivables", 209- "shares_outstanding", "diluted_shares", 210-) 211- 212- 213:def validate_line_items( 214- line_items: dict, report: ValidationReport, row_key: Optional[str] = None, 215-) -> dict: 216- """Coerce and sanity-check one period's line items. Returns the cleaned dict. 217- 218- Unlike a price bar, a financial period is never rejected wholesale: one impossible field does 219- not invalidate the other thirty, and a period with a bad `inventory` still supports every 220- metric that does not use inventory. Bad fields are dropped individually with an issue recorded. 221- """ 222- report.rows_seen += 1 223- out: dict = {} 224- for name, value in line_items.items(): 225- number = coerce_float(value, name, report, row_key) 226- if number is None: 227- continue 228- if name in NON_NEGATIVE_LINE_ITEMS and number < 0: 229- report.add(ValidationIssue(name, number, "cannot be negative in a real filing", 230- SEVERITY_WARNING, row_key)) 231- continue 232- out[name] = number 233- 234- # Cross-field checks: each is an identity that must hold, not a heuristic. 235- revenue, gross_profit = out.get("revenue"), out.get("gross_profit") 236- if revenue is not None and gross_profit is not None and gross_profit > revenue: 237- report.add(ValidationIssue("gross_profit", gross_profit, 238- f"exceeds revenue ({revenue})", SEVERITY_WARNING, row_key)) 239- total_assets, current_assets = out.get("total_assets"), out.get("current_assets") 240- if total_assets is not None and current_assets is not None and current_assets > total_assets: 241- report.add(ValidationIssue("current_assets", current_assets, 242- f"exceeds total assets ({total_assets})", SEVERITY_WARNING, row_key)) 243- diluted, outstanding = out.get("diluted_shares"), out.get("shares_outstanding") 244- if diluted is not None and outstanding is not None and diluted < outstanding: 245- # Diluted counts every share that COULD exist, so it can never be below basic/outstanding. 246- report.add(ValidationIssue("diluted_shares", diluted, 247- f"below shares outstanding ({outstanding})", SEVERITY_WARNING, row_key)) 248- 249- report.rows_accepted += 1 250- return out === 6. EXACT INGEST RESOLUTION/PERSISTENCE === 5-recompute.py). Demo and live data are never blended for the same security (docs/DATA_SOURCES.md 6-§9) — enforced in `_assert_no_demo_live_mix` below, not just documented. 7- 8-AUDIT FIX (StockLab overhaul, final engineering pass, Part A6, docs/AUDIT_PROVIDER_CONFLICT_A6.md): 9-`resolve_conflict` used to be imported here but never called — this file ingested exactly one 10-provider's data per call, so there was nothing for it to resolve BETWEEN (docs/ 11-AUDIT_PROVIDER_RESILIENCE.md's finding). `ingest_security()` now optionally accepts a 12-`secondary_adapter`; when given, both providers' line items are resolved field-by-field via 13:`resolve_line_items()` (app/adapters/resolver.py) and real disagreements are persisted as 14-`DataQuality` rows. When `secondary_adapter` is None — the default, and what every existing caller 15-still passes — behavior is byte-identical to before this pass: nothing about the single-source path 16-changed. 17-""" 18-from __future__ import annotations 19-from datetime import date, timedelta 20- 21- 22-from sqlalchemy.orm import Session 23- 24-from app.adapters.base import ( 25- ProviderAdapter, ProviderAuthError, ProviderNotFoundError, ProviderRateLimitError, 26-) 27:from app.adapters.resolver import merge_statement_line_items, resolve_line_items 28-from app.core.config import get_settings 29-from app.core.db import SessionLocal 30-from app.core.logging import get_logger 31-from app.engines.identity import resolve_company_identity 32-from app.engines.reference_data import canonical_mic, exchange_for_mic 33-from app.models import ( 34- BalanceSheet, CashFlow, Company, Country, DataQuality, Estimate, Exchange, FinancialPeriod, 35- Industry, IncomeStatement, Price, Sector, Security, Shares, Source, 36-) 37-from app.workers.celery_app import celery_app 38- 39-logger = get_logger(__name__) 40- 41- 42-class ReferenceDataError(RuntimeError): 43- """A NOT NULL reference row could not be resolved, so ingestion must not proceed. 44- 45- AUDIT FIX (DEMO ingestion defect). `get_or_create_company()` used to build `Company` and 46- `Exchange` rows whose `country_id` was a conditional expression falling back to a null. Both 47- columns are **NOT NULL** 48- (`alembic/versions/0001_initial_schema.py`, `companies.country_id` and `exchanges.country_id`), 49- so that `else None` branch could never succeed: it handed Postgres a NULL for a NOT NULL 50- column and the whole transaction died with 51- 52- psycopg.errors.NotNullViolation: null value in column "country_id" of relation "exchanges" 53- 54- several statements later, with a traceback pointing at `db.flush()` rather than at the missing 55- reference row. The code was written as if the column were nullable; it is not. 56- 57- Failing here instead is the correct behaviour and not merely a nicer message: 58- 59- * an exchange row without a country is **not representable** in this schema, so there is no -- 62- * the error names the ticker, the field and the specific remedy, so the fix is a one-line 63- addition to a reference table rather than a database forensics session; 64- * it is raised **before** any row is added to the session, so the caller's transaction is 65- still clean and one bad ticker cannot poison the rest of a seed run. 66- """ 67- 68- 69-# Field tuples pulled out to module level (StockLab overhaul, Part A6) -- previously inline in the 70:# single setattr loop below; now shared between the single-source path and resolve_line_items() 71-# calls so the two can never drift out of sync with each other. 72-_INCOME_FIELDS = ( 73- "revenue", "cogs", "gross_profit", "operating_income", "ebit", "ebitda", "net_income", 74- "eps_diluted", "tax_expense", "pretax_income", "interest_expense", 75- "depreciation_and_amortization", "stock_based_compensation", 76-) 77-_BALANCE_FIELDS = ( 78- "cash_and_equivalents", "short_term_investments", "total_debt", "shareholders_equity", 79- "minority_interest", "preferred_equity", "goodwill", "intangible_assets", 80- "current_assets", "current_liabilities", "receivables", "inventory", 81-) 82-_CASH_FLOW_FIELDS = ("operating_cash_flow", "capital_expenditure", "dividends_paid", "buybacks", "stock_issuance") 83-_SHARES_FIELDS = ("shares_outstanding", "diluted_shares") 84- 85- 86-def _get_or_create_source(db: Session, provider: str, tier: str, is_demo: bool, endpoint: str) -> Source: 87- src = Source(provider=provider, provider_tier=tier, is_demo=is_demo, endpoint=endpoint) 88- db.add(src) 89- db.flush() 90- return src 91- 92- 93-def _assert_no_demo_live_mix(db: Session, security_id: str, is_demo: bool) -> None: 94- """Refuse to write demo rows for a security that already has live rows, or vice versa 95- (docs/DATA_SOURCES.md §9 — hard separation, enforced here, not just at read time).""" 96- existing = ( 97- db.query(FinancialPeriod.id) 98- .join(Source, FinancialPeriod.source_id == Source.id) 99- .filter(FinancialPeriod.security_id == security_id, Source.is_demo == (not is_demo)) 100- .first() 101- ) 102- if existing: -- 255- ) 256- db.add(security) 257- db.flush() 258- return security 259- 260- 261-def _record_conflicts(db: Session, entity_table: str, entity_id: str, conflicts: list, 262- source_a: Source, source_b: Source) -> None: 263: """Persist resolve_line_items()'s flagged disagreements as DataQuality rows (StockLab 264- overhaul, Part A6). entity_id is the owning FinancialPeriod's id for every statement type -- 265- IncomeStatement/BalanceSheet/CashFlow/Shares are each a 1:1 extension of FinancialPeriod with 266- no independently meaningful id of their own, so `entity_table` (e.g. "income_statements") is 267- what disambiguates which statement a given row's conflicting field belongs to, not entity_id 268- itself. A deliberate simplification, not an oversight -- see docs/AUDIT_PROVIDER_CONFLICT_A6.md 269- for the alternative (a dedicated per-statement-row id) and why it wasn't needed for this.""" 270- for c in conflicts: 271- db.add(DataQuality( 272- entity_table=entity_table, entity_id=entity_id, field=c.field, status="CONFLICTING", 273- source_a_id=source_a.id, value_a=c.value_a, 274- source_b_id=source_b.id if source_b is not None else None, value_b=c.value_b, 275- selected_value=c.selected_value, selected_source_id=source_a.id if c.selected_value == c.value_a else (source_b.id if source_b is not None else None), 276- selection_reason=c.selection_reason, 277- )) 278- 279- 280-def _fetch_statements_by_period(adapter, method, ticker: str, label: str) -> dict: 281- """Fetch one statement type and index it by (period_end, period_type). 282- 283- Returns an empty dict -- never raises -- when the adapter has not implemented this statement 284- (EODHDAdapter's balance-sheet/cash-flow methods are `NotImplementedError` today). An adapter 285- with a partial implementation must degrade to "that statement's columns stay empty for this 286- run", exactly as before this fix, rather than crashing every ingestion task. 287- """ 288- try: 289- rows = method(ticker) 290- except NotImplementedError: 291- logger.warning( 292- "ingest.security.statement_not_implemented", 293- ticker=ticker, provider=adapter.name, statement=label, 294- ) 295- return {} -- 357- secondary_adapter: ProviderAdapter | None = None, 358- include_quarterly: bool = False) -> str: 359- """Ingest one security's financials from `adapter` (the primary/only source for this call). 360- 361- AUDIT FIX (StockLab overhaul, final engineering pass, Part A6): `secondary_adapter` is new and 362- optional. When None -- the default, and what every caller in this codebase passed before this 363- pass -- this function's behavior is completely unchanged: every field comes straight from 364- `adapter`, exactly as before. When given a second adapter, each period's line items are 365: resolved field-by-field against the primary's via `resolve_line_items()` (source-hierarchy 366- tie-break, real disagreements flagged) instead of blindly overwritten by whichever ran last. 367- See docs/AUDIT_PROVIDER_CONFLICT_A6.md for the full design, the period-alignment limitation, 368- and honest TESTED/NOT TESTED status -- none of the dual-source path has been executed against 369- real provider data or a real database in this build environment. 370- """ 371- is_demo = adapter.tier == "DEMO" 372- security = get_or_create_company(db, adapter, ticker) 373- _assert_no_demo_live_mix(db, security.id, is_demo) 374- 375- source = _get_or_create_source(db, adapter.name, adapter.tier, is_demo, f"ingest:{ticker}") 376- 377- # AUDIT FIX (StockLab final engineering pass, Part A9 -- CRITICAL, see 378- # docs/AUDIT_INGEST_STATEMENTS_A9.md). Before this fix, this function fetched ONLY the income 379- # statement and then wrote the BalanceSheet, CashFlow and Shares rows out of that same 380- # response's line items. adapter.get_balance_sheets() and adapter.get_cash_flows() were never 381- # called anywhere in the application. That is invisible in DEMO mode (DemoDataAdapter returns 382- # one fully-merged dict from all three methods) and catastrophic against FMP, whose income 383- # response carries only _INCOME_MAP's fields -- every balance-sheet, cash-flow and share-count 384- # column would have been written NULL for real provider data, taking most metrics, all five 385- # pillar scores and every valuation down with it. 386- income_periods = adapter.get_income_statements(ticker) 387- balance_by_period = _fetch_statements_by_period(adapter, adapter.get_balance_sheets, ticker, "balance_sheets") 388- cash_by_period = _fetch_statements_by_period(adapter, adapter.get_cash_flows, ticker, "cash_flows") 389- 390- # Part A9: optional quarterly ingestion. Off by default (Settings.INGEST_QUARTERLY_PERIODS). 391- # Quarterly rows are stored as ordinary FinancialPeriod rows with period_type Q1..Q4 -- the 392- # (security_id, period_end, period_type, filing_date) unique constraint keeps them distinct 393- # from the annual rows, and every existing query that filters period_type == "FY" is 394- # unaffected. app/engines/ttm.py turns four of them into a TTM basis at recompute time. 395- if include_quarterly: 396- try: 397- quarterly_income = adapter.get_income_statements(ticker, period="quarter") -- 475- secondary_p.line_items, secondary_balance.get(key), secondary_cash.get(key) 476- ) 477- if secondary_p is not None 478- else None 479- ) 480- secondary_tier = secondary_adapter.tier if (secondary_adapter is not None and li_b is not None) else None 481- 482- income = fp.income_statement or IncomeStatement(financial_period_id=fp.id) 483: resolved, conflicts = resolve_line_items(li, li_b, _INCOME_FIELDS, adapter.tier, secondary_tier) 484- for field, value in resolved.items(): 485- setattr(income, field, value) 486- db.add(income) 487- if conflicts: 488- _record_conflicts(db, "income_statements", fp.id, conflicts, source, secondary_source) 489- 490- balance = fp.balance_sheet or BalanceSheet(financial_period_id=fp.id) 491: resolved, conflicts = resolve_line_items(li, li_b, _BALANCE_FIELDS, adapter.tier, secondary_tier) 492- for field, value in resolved.items(): 493- setattr(balance, field, value) 494- db.add(balance) 495- if conflicts: 496- _record_conflicts(db, "balance_sheets", fp.id, conflicts, source, secondary_source) 497- 498- cash_flow = fp.cash_flow or CashFlow(financial_period_id=fp.id) 499: resolved, conflicts = resolve_line_items(li, li_b, _CASH_FLOW_FIELDS, adapter.tier, secondary_tier) 500- for field, value in resolved.items(): 501- setattr(cash_flow, field, value) 502- db.add(cash_flow) 503- if conflicts: 504- _record_conflicts(db, "cash_flows", fp.id, conflicts, source, secondary_source) 505- 506- shares = fp.shares or Shares(financial_period_id=fp.id) 507: resolved, conflicts = resolve_line_items(li, li_b, _SHARES_FIELDS, adapter.tier, secondary_tier) 508- for field, value in resolved.items(): 509- setattr(shares, field, value) 510- db.add(shares) 511- if conflicts: 512- _record_conflicts(db, "shares", fp.id, conflicts, source, secondary_source) 513- 514- # AUDIT FIX (final master pass, §21) — the fourth "engine built, never wired" defect. 515- # `ProviderAdapter.get_estimates()` is implemented by FMP and by the demo adapter, 516- # `ProviderEstimateRow` exists, and the `estimates` table exists with a unique constraint on 517- # (security_id, period_end, metric, as_of_date). Nothing ever called it, nothing ever wrote a 518- # row, and `build_snapshot_from_db()` never passed `forward_eps_estimate` — so it was None on 519- # every real run and `forward_pe` and `eps_growth_forward` were PERMANENTLY NULL. `forward_pe` 520- # is a member of the Valuation pillar, so every Valuation score was computed from six of its 521- # seven metrics with nothing saying so. 522- # Writes are idempotent against that unique constraint: an estimate already stored for the 523- # same (period_end, metric, as_of_date) is updated in place, never duplicated on re-ingestion. 524- estimates_written = _ingest_estimates(db, adapter, security.id, ticker, source) 525- prices_written = _ingest_prices(db, adapter, security.id, ticker, source) 526- 527- db.commit() 528- logger.info("ingest.security.complete", ticker=ticker, provider=adapter.name, is_demo=is_demo, 529- periods=len(income_periods), estimates=estimates_written, 530- prices=prices_written, 531- secondary_provider=(secondary_adapter.name if secondary_adapter else None)) 532- return security.id 533- 534- 535-def _ingest_estimates(db: Session, adapter: ProviderAdapter, security_id: str, ticker: str, 536- source: Source) -> int: 537- """Fetch and persist forward analyst estimates. Returns how many rows were written or updated. 538- 539- Never raises for an adapter that has not implemented estimates (EODHD's is === 7. EXACT _record_conflicts === 249- db.add(company) 250- db.flush() 251- 252- security = Security( 253- company_id=company.id, exchange_id=exchange.id, ticker=ticker, 254- isin=profile.isin, currency=profile.currency, discovery_status="QUALIFIED", 255- ) 256- db.add(security) 257- db.flush() 258- return security 259- 260- 261:def _record_conflicts(db: Session, entity_table: str, entity_id: str, conflicts: list, 262- source_a: Source, source_b: Source) -> None: 263- """Persist resolve_line_items()'s flagged disagreements as DataQuality rows (StockLab 264- overhaul, Part A6). entity_id is the owning FinancialPeriod's id for every statement type -- 265- IncomeStatement/BalanceSheet/CashFlow/Shares are each a 1:1 extension of FinancialPeriod with 266- no independently meaningful id of their own, so `entity_table` (e.g. "income_statements") is 267- what disambiguates which statement a given row's conflicting field belongs to, not entity_id 268- itself. A deliberate simplification, not an oversight -- see docs/AUDIT_PROVIDER_CONFLICT_A6.md 269- for the alternative (a dedicated per-statement-row id) and why it wasn't needed for this.""" 270- for c in conflicts: 271- db.add(DataQuality( 272- entity_table=entity_table, entity_id=entity_id, field=c.field, status="CONFLICTING", 273- source_a_id=source_a.id, value_a=c.value_a, 274- source_b_id=source_b.id if source_b is not None else None, value_b=c.value_b, 275- selected_value=c.selected_value, selected_source_id=source_a.id if c.selected_value == c.value_a else (source_b.id if source_b is not None else None), 276- selection_reason=c.selection_reason, 277- )) 278- 279- 280-def _fetch_statements_by_period(adapter, method, ticker: str, label: str) -> dict: 281- """Fetch one statement type and index it by (period_end, period_type). 282- 283- Returns an empty dict -- never raises -- when the adapter has not implemented this statement 284- (EODHDAdapter's balance-sheet/cash-flow methods are `NotImplementedError` today). An adapter 285- with a partial implementation must degrade to "that statement's columns stay empty for this 286- run", exactly as before this fix, rather than crashing every ingestion task. 287- """ 288- try: 289- rows = method(ticker) 290- except NotImplementedError: 291- logger.warning( 292- "ingest.security.statement_not_implemented", 293- ticker=ticker, provider=adapter.name, statement=label, 294- ) 295- return {} 296- return {(r.period_end, r.period_type): r.line_items for r in rows} 297- 298- 299- 300-def _ingest_prices( 301- db: Session, 302- adapter: ProviderAdapter, 303- security_id: str, 304- ticker: str, 305- source: Source, 306-) -> int: 307- """Ingest daily OHLCV bars for one security, idempotently by security/date.""" 308- end = date.today() 309- start = end - timedelta(days=365) 310- 311- bars = adapter.get_prices(ticker, start, end) 312- written = 0 313- 314- for bar in bars: 315- row = ( 316- db.query(Price) === 8. SOURCE HASH GUARD === INGEST_SHA=00515c566771560e86dc9cb2e61fc48ac070a50d3c5e8e425a99a22ad34e059b RESOLVER_SHA=670994146a74c132be4ee08bf8c50627c78d911a7239469767a826c6e5337e96 VALIDATION_SHA=486737385fd5ee3a3863fc04f385b3ce4d2c462938f914c2f21170e9999929f1 === 9. ACTUAL PYTHON API CONTRACT === === 10. EXACT SL-004 REPRODUCTION === === 11. BACKUP === BACKUP=PASS === 12. APPLY ONLY AFTER ALL GUARDS === PATCH_WRITE=PASS PATCH_APPLY=PASS === 13. DIFF === --- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py.before-sl004-20260910-181036 2026-09-08 14:21:15.000000000 +0300 +++ /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py 2026-09-10 18:10:37.264899030 +0300 @@ -99,8 +99,35 @@ result = resolve_conflict(candidates) if result.selected_value is not None: resolved[field] = result.selected_value - if result.is_conflicting: - conflicts.append(FieldConflict(field, val_a, val_b, result.selected_value, result.selection_reason)) + + # Secondary-only provenance must be observable. A NULL in the + # primary provider followed by a value from the secondary provider + # is not a numeric conflict, but it is still a cross-provider merge. + if ( + secondary is not None + and secondary_tier is not None + and val_a is None + and val_b is not None + ): + conflicts.append( + FieldConflict( + field, + val_a, + val_b, + result.selected_value, + "Primary field missing; value supplied by secondary source", + ) + ) + elif result.is_conflicting: + conflicts.append( + FieldConflict( + field, + val_a, + val_b, + result.selected_value, + result.selection_reason, + ) + ) return resolved, conflicts --- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/validation.py.before-sl004-20260910-181036 2026-09-08 14:47:42.000000000 +0300 +++ /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/validation.py 2026-09-10 18:10:37.264899030 +0300 @@ -232,7 +232,47 @@ out[name] = number # Cross-field checks: each is an identity that must hold, not a heuristic. - revenue, gross_profit = out.get("revenue"), out.get("gross_profit") + revenue = out.get("revenue") + cogs = out.get("cogs") + gross_profit = out.get("gross_profit") + + # Accounting identity: + # gross_profit = revenue - cogs + # + # The provider resolver already treats <=0.5% financial differences + # as non-conflicting. Use the same tolerance here so ordinary provider + # rounding does not create a false integrity failure. + if revenue is not None and cogs is not None and gross_profit is not None: + expected_gross_profit = revenue - cogs + + if expected_gross_profit == 0: + inconsistent = abs(gross_profit) > 0.01 + else: + inconsistent = ( + abs(gross_profit - expected_gross_profit) + / abs(expected_gross_profit) + > 0.005 + ) + + if inconsistent: + report.add( + ValidationIssue( + "gross_profit", + gross_profit, + ( + f"does not reconcile with revenue ({revenue}) " + f"- cogs ({cogs}) = {expected_gross_profit}" + ), + SEVERITY_WARNING, + row_key, + ) + ) + + # Do not allow an internally inconsistent dependent value + # into the persisted merged statement. + del out["gross_profit"] + gross_profit = None + if revenue is not None and gross_profit is not None and gross_profit > revenue: report.add(ValidationIssue("gross_profit", gross_profit, f"exceeds revenue ({revenue})", SEVERITY_WARNING, row_key)) --- /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py.before-sl004-20260910-181036 2026-09-09 10:19:25.327879082 +0300 +++ /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py 2026-09-10 18:10:37.264899030 +0300 @@ -25,6 +25,7 @@ ProviderAdapter, ProviderAuthError, ProviderNotFoundError, ProviderRateLimitError, ) from app.adapters.resolver import merge_statement_line_items, resolve_line_items +from app.adapters.validation import ValidationReport, validate_line_items from app.core.config import get_settings from app.core.db import SessionLocal from app.core.logging import get_logger @@ -480,7 +481,15 @@ secondary_tier = secondary_adapter.tier if (secondary_adapter is not None and li_b is not None) else None income = fp.income_statement or IncomeStatement(financial_period_id=fp.id) - resolved, conflicts = resolve_line_items(li, li_b, _INCOME_FIELDS, adapter.tier, secondary_tier) + resolved, conflicts = resolve_line_items( + li, li_b, _INCOME_FIELDS, adapter.tier, secondary_tier + ) + merged_report = ValidationReport() + resolved = validate_line_items( + resolved, + merged_report, + row_key=f"{ticker}:{p.period_end}:{p.period_type}:income", + ) for field, value in resolved.items(): setattr(income, field, value) db.add(income) @@ -488,7 +497,15 @@ _record_conflicts(db, "income_statements", fp.id, conflicts, source, secondary_source) balance = fp.balance_sheet or BalanceSheet(financial_period_id=fp.id) - resolved, conflicts = resolve_line_items(li, li_b, _BALANCE_FIELDS, adapter.tier, secondary_tier) + resolved, conflicts = resolve_line_items( + li, li_b, _BALANCE_FIELDS, adapter.tier, secondary_tier + ) + merged_report = ValidationReport() + resolved = validate_line_items( + resolved, + merged_report, + row_key=f"{ticker}:{p.period_end}:{p.period_type}:balance", + ) for field, value in resolved.items(): setattr(balance, field, value) db.add(balance) @@ -496,7 +513,15 @@ _record_conflicts(db, "balance_sheets", fp.id, conflicts, source, secondary_source) cash_flow = fp.cash_flow or CashFlow(financial_period_id=fp.id) - resolved, conflicts = resolve_line_items(li, li_b, _CASH_FLOW_FIELDS, adapter.tier, secondary_tier) + resolved, conflicts = resolve_line_items( + li, li_b, _CASH_FLOW_FIELDS, adapter.tier, secondary_tier + ) + merged_report = ValidationReport() + resolved = validate_line_items( + resolved, + merged_report, + row_key=f"{ticker}:{p.period_end}:{p.period_type}:cash_flow", + ) for field, value in resolved.items(): setattr(cash_flow, field, value) db.add(cash_flow) @@ -504,7 +529,15 @@ _record_conflicts(db, "cash_flows", fp.id, conflicts, source, secondary_source) shares = fp.shares or Shares(financial_period_id=fp.id) - resolved, conflicts = resolve_line_items(li, li_b, _SHARES_FIELDS, adapter.tier, secondary_tier) + resolved, conflicts = resolve_line_items( + li, li_b, _SHARES_FIELDS, adapter.tier, secondary_tier + ) + merged_report = ValidationReport() + resolved = validate_line_items( + resolved, + merged_report, + row_key=f"{ticker}:{p.period_end}:{p.period_type}:shares", + ) for field, value in resolved.items(): setattr(shares, field, value) db.add(shares) === 14. PYCOMPILE === PYCOMPILE=PASS === 15. TARGETED REGRESSION === TARGETED_REGRESSION=PASS === 16. EXISTING TESTS === Error: crun: executable file `pytest` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found EXISTING_TESTS=FAIL === 17. FINAL HASHES === 183842352c4c3b0ec9e2fabffa043ff6fbb125c2a0b602c9da53477d17b7bfb8 /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/resolver.py 4ff976fb3b68594b7991c8d172143278670ae3685d49ad61f5b945db61b4f19a /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/adapters/validation.py 782820e1a686029f751c148d327dbef76784fb6ef1d52107aefa9c3a75c08af6 /data/files/bmw/stocklab-release-candidate-sl003-final-20260910-111453/app/workers/ingest.py === 18. PRODUCTION AFTER === stocklab-redis docker.io/library/redis:7-alpine Up 33 hours (healthy) stocklab-db docker.io/library/postgres:16-alpine Up 32 hours (healthy) stocklab-release-qa-redis docker.io/library/redis:7 Up 26 hours stocklab-frontend localhost/stocklab-frontend:release-qa-20260909-182234 Up 24 hours (healthy) stocklab-backend localhost/stocklab-backend:screener-percent-final-20260909-194639 Up 21 hours (healthy) stocklab-worker localhost/stocklab-backend:screener-percent-final-20260909-194639 Up 21 hours (healthy) stocklab-beat localhost/stocklab-backend:screener-percent-final-20260909-194639 Up 21 hours LISTEN 0 2048 127.0.0.1:18000 0.0.0.0:* users:(("uvicorn",pid=1574355,fd=11)) LISTEN 0 511 127.0.0.1:6379 0.0.0.0:* users:(("redis-server",pid=1133711,fd=6)) LISTEN 0 200 127.0.0.1:5432 0.0.0.0:* users:(("postgres",pid=1141225,fd=6)) LISTEN 0 511 127.0.0.1:13000 0.0.0.0:* users:(("next-server (v1",pid=1497838,fd=18)) stocklab-backend restart=0 stocklab-worker restart=0 stocklab-beat restart=0 stocklab-db restart=0 stocklab-redis restart=0 stocklab-frontend restart=0 ============================================================ FINAL RESULT ============================================================ SL004_PATCH=NOT_RELEASE_READY PRODUCTION=UNCHANGED DO NOT DEPLOY. LOG=/data/files/bmw/stocklab-sl004-fix2-20260910-181036.log ============================================================