Automated Telemetry Engine

Turn a product hypothesis into an executive-ready telemetry schema.

When product teams hypothesize an A/B test or experiment, defining the exact KPIs, guardrail metrics, and edge cases usually takes days of alignment.

Even the best product leaders have blind spots.

To solve this, I built an AI-powered telemetry engine that doesn't just agree with the user, it aggressively stress-tests their hypotheses.

See the following video for a masterclass on how this cutting-edge agentic workflow operates

The Business Impact

• Eliminate Blind Spots: Catches massive revenue traps before an experiment is ever deployed.
  
  

• Massive Time Savings: Reduces telemetry planning and cross-functional swirl from days to hours.

Agentic Workflow Structure

A1 · KPI Generator Agent

• Purpose: Generate primary and secondary KPIs to track for a proposed product experiment.

• Inputs: Experiment hypothesis.

• Process: Analyzes the hypothesis, determines the core business objectives, and maps out the primary and secondary tracking metrics needed to accurately measure success.

• Outputs: Proposed KPIs (e.g., Average Order Value, Conversion Rate).

• Checks: Relevance to hypothesis, baseline trackability.

A2 · KPI QA Agent

• Purpose: Evaluate A1’s output to ensure metrics are sound, trackable, and not hallucinated.

• Inputs: A1’s proposed KPIs + original hypothesis.

• Process: Acts as a automated "circuit breaker." Audits A1’s work against strict telemetry standards.

  • If it finds major issues or untrackable metrics, it rejects the output, provides feedback, and forces A1 to run again.

• Outputs: Approved KPIs or Rejection/Feedback Log.

• Checks: Trackability, business logic validity, zero hallucination.

A3 · Edge Case Generator Agent

• Purpose: Identify hidden risks, guardrail metrics, and edge cases to ensure no stone is left unturned.

• Inputs: Approved KPIs + original hypothesis.

• Process: Hunts for risks and friction points.

  • Formulates guardrail metrics to protect core business health during the experiment.

• Outputs: Proposed Guardrails and Edge Cases.

• Checks: Relevance to experiment, severity of impact, gap coverage.

A4 · Edge Case QA Agent

• Purpose: Filter theoretical concerns from practical, deployable, actionable risks.

• Inputs: A3’s proposed edge cases and guardrails.

• Process: Serves as the second "circuit breaker."

  • Strictly filters A3’s output to ensure only highly actionable, realistic edge cases are passed.

  • Forces A3 to regenerate if the output does not meet high-quality standards.

• Outputs: Approved Guardrails and Edge Cases.

• Checks: Deployability, actionability, theoretical vs. practical distinction.

A5 · Final Output Agent

• Purpose: Compile and format the fully QA'd telemetry data into an easy to approach schema.

• Inputs: Approved KPIs (from A2) + Approved Edge Cases (from A4).

• Process: Synthesizes the data, applies Gemini-powered explanations for specific metrics, and formats it into an approachable, easy-to-understand UI report.

• Outputs: Final Telemetry Report (reiterated hypothesis, defined KPIs, guardrails, and edge cases).

• Checks: Readability, completeness, alignment.