SignalSentry
Early warning for customer churn and paid-media problems.
Deterministic detectors identify unusual behavior. An optional AI layer turns each alert into a clear, auditable brief explaining what changed, what to do...
Role & Stack: Design and implementation — full application. Built with Python, Pandas, NumPy, PyArrow, PyYAML, Pydantic, LangChain and LangChain-OpenAI, Streamlit, Plotly, python-dotenv, and pytest. Live inference is designed for NVIDIA NemoClaw (OpenAI-compatible, https://inference.local/v1) with OpenClaw as the local chat gateway. The public Streamlit demo defaults to mock investigators.
Overview: SignalSentry is two layers stacked on purpose. A calculator notices that an account's logins fell hard or that a campaign spent more without more conversions. A briefing writer explains that finding the way a CSM or media lead would in a monthly review. The calculator never uses an LLM. The briefing writer only sees the calculator's output plus a short slice of recent metrics, and it is not allowed to invent spend, sessions, or conversions. All public demo data is synthetic and every recommendation is advisory — a human still decides.
The Problem: Customer success and media teams need early warning before renewals slip or budgets burn, but real CRM and ad-platform data is sensitive and hard to put on a portfolio site. Spreadsheet workflows mix what the data says with what someone thinks, so you cannot tell which is which. Many AI demos call a model first and let it invent numbers, which looks flashy and fails the audit test.
The Solution: SignalSentry is a clear pipeline: make demo data, detect anomalies with rules, investigate with structured AI when enabled, show an English-first dashboard, and score detection quality against labels. The hosted demo is safe to share, while the full live NemoClaw and OpenClaw route runs on each person's own machine inside their own sandbox.
How It Works — Make a Demo World: The app invents weekly customer metrics and daily campaign metrics, plants known problems such as gradual usage decline, creative fatigue, and tracking mismatch, and saves labels for those plants. Change the seed and you get a new cast of accounts and campaigns with the same kinds of problems — useful for demos and interviews.
How It Works — Rules, Briefs, and Checks: For each account or campaign, detectors compare recent behavior to that entity's own history. If something crosses thresholds configured in YAML, SignalSentry raises an alert with who, what type, how bad, current versus expected, and the date window. Each alert becomes a short packet of facts: mock mode fills a standard investigation template with no API key, while live mode sends the packet to LangChain and asks for a fixed-shape answer covering evidence, likely causes, recommended action, confidence, and limitations. Because the injected problems are labeled, evaluation can show how well the detectors recovered them using precision and recall style metrics.
Architecture — Synthetic Generation: Seeded generators in src/generation/ produce churn data (plan tier, licensed and active users, weekly sessions, feature adoption, days since admin login, support tickets, NPS, renewal proximity) and media data (platform, objective, spend, impressions and clicks, CPC/CPM, conversions, conversion rate, frequency). Injection catalogs assign anomaly types to random entity IDs per seed. Outputs land in data/generated/ and data/ground_truth/ as parquet and CSV, and generation_meta.json records seed and sizes for the UI.
Architecture — Deterministic Detection: src/detection/ loads config/thresholds.yaml and scores entities with historical baselines, slopes, and z-scores. Using a historical baseline is a deliberate design choice so that ongoing bad states still alert, since adjacent-window baselines can miss persistent anomalies. Each CandidateAlert carries entity_id, domain, alert_type, severity, start and end dates, metrics involved, current and expected values, and supporting calculations. No network calls and no LLM.
Architecture — Investigation Layer: src/agents/investigators.py builds system and user messages from the alert JSON plus optional recent rows. Mock mode routes to a template investigator; live mode uses ChatOpenAI.with_structured_output against ChurnInvestigation or CampaignInvestigation schemas, trying json_schema first and falling back to json_mode for picky backends. Every run appends an audit log entry and a LangChain step trace. Schemas in src/models/schemas.py enforce non-empty evidence, causes, and limitations, plus bounded confidence.
Architecture — Inference Routes: On a developer laptop with NemoClaw, LangChain routes to https://inference.local/v1 with OpenClaw serving a local gateway UI at 127.0.0.1:18789. Streamlit Cloud defaults to forced mock because it cannot reach inference.local, and visitors can optionally bring their own session-scoped nvapi key routed to https://integrate.api.nvidia.com/v1. Keys are normalized on paste by stripping the Bearer prefix and whitespace, and live auth failures do not crash the app — Ask falls back to local summaries of current demo alerts.
Architecture — Presentation, Evaluation & Privacy: src/presentation/briefing.py maps each alert type to human copy and merges investigator fields. Streamlit pages lead with the story summary, fold numbers and charts (Plotly on_select surfaces source rows), and fold the detector payload, hypotheses, and confidence. src/evaluation/ scores detections against injected labels with configurable date tolerance, and src/privacy/ lists readable files, builds inference payload previews, and appends investigation_audit.jsonl under data/outputs/. The repository never commits .env; only placeholders ship in .env.example.
Technical Highlights: Detector-first integrity means the model cannot invent KPIs and only narrates supplied JSON. Briefs stay auditable because every recommendation ties back to alert math and charts can surface the exact source rows. Demos are reproducible: the same seed rebuilds the same world, while a new seed reassigns entities across the same anomaly catalog. Hosting is honest — the public app is synthetic and mock with optional visitor BYOK, and the full NemoClaw route runs in each person's own sandbox. Ask still works offline, answering questions about critical and flagged issues from the current demo alert list even when the live model returns a 401.
Deployment: The public demo runs on Streamlit Community Cloud from app.py on main, with a cold start that auto-generates synthetic data and mock analysis when outputs are missing. Locally the flow is python -m src.generation.generate_all, python -m src.run_analysis, streamlit run app.py, and pytest. For live NemoClaw, set USE_MOCK_MODEL=false and MODEL_BASE_URL=https://inference.local/v1, run with NemoClaw routing, and optionally verify with python -m src.verify_inference.
Outcome: A shareable CSM and media early-warning surface that keeps measurement strict, makes recommendations readable, and treats generative AI as a second pass over detected signals rather than the source of the numbers.
The deterministic detectors and mock investigators run with no API key. Optional live LangChain uses a session-scoped NVIDIA key (BYOK) or a local NemoClaw route. Keys are not stored in the repository, and all portfolio demo metrics are synthetic.