Agentic Workflow
Automated Telemetry Engine
Platform
Gemini 3.1 Pro + 3 Flash
ROLE
Experimentation Specialist
EXPERTISE
Product
YEAR
2026
Project Description
Project Description
Project Description
Turn a product hypothesis into an executive-ready telemetry schema.
When product teams hypothesize an A/B test or experiment, defining the exact KPIs, guardrail metrics, and edge cases usually takes days of alignment.
Even the best product leaders have blind spots.
To solve this, I built an AI-powered telemetry engine that doesn't just agree with the user, it aggressively stress-tests their hypotheses.
See the following video for a masterclass on how this cutting-edge agentic workflow operates
When product teams hypothesize an A/B test or experiment, defining the exact KPIs, guardrail metrics, and edge cases usually takes days of alignment.
Even the best product leaders have blind spots.
To solve this, I built an AI-powered telemetry engine that doesn't just agree with the user, it aggressively stress-tests the hypothesis.
The Business Impact
Eliminated Blind Spots: Catches massive revenue traps before an experiment is ever deployed.
Massive Time Savings: Reduces telemetry planning and cross-functional swirl from days to hours.
Agentic Workflow Structure
A1 · KPI Generator Agent
Purpose: Generate primary and secondary KPIs to track for a proposed product experiment.
Inputs: Experiment hypothesis.
Process: Analyzes the hypothesis, determines the core business objectives, and maps out the primary and secondary tracking metrics needed to accurately measure success.
Outputs:
Proposed KPIs (e.g., Average Order Value, Conversion Rate).Checks: Relevance to hypothesis, baseline trackability.
A2 · KPI QA Agent
Purpose: Evaluate A1’s output to ensure metrics are sound, trackable, and not hallucinated.
Inputs: A1’s proposed KPIs + original hypothesis.
Process: Acts as a automated "circuit breaker." Audits A1’s work against strict telemetry standards.
If it finds major issues or untrackable metrics, it rejects the output, provides feedback, and forces A1 to run again.
Outputs:
Approved KPIs or Rejection/Feedback Log.Checks: Trackability, business logic validity, zero hallucination.
A3 · Edge Case Generator Agent
Purpose: Identify hidden risks, guardrail metrics, and edge cases to ensure no stone is left unturned.
Inputs: Approved KPIs + original hypothesis.
Process: Hunts for risks and friction points.
Formulates guardrail metrics to protect core business health during the experiment.
Outputs: Proposed Guardrails and Edge Cases.
Checks: Relevance to experiment, severity of impact, gap coverage.
A4 · Edge Case QA Agent
Purpose: Filter theoretical concerns from practical, deployable, actionable risks.
Inputs: A3’s proposed edge cases and guardrails.
Process: Serves as the second "circuit breaker."
Strictly filters A3’s output to ensure only highly actionable, realistic edge cases are passed.
Forces A3 to regenerate if the output does not meet high-quality standards.
Outputs:
Approved Guardrails and Edge Cases.Checks: Deployability, actionability, theoretical vs. practical distinction.
A5 · Final Output Agent
Purpose: Compile and format the fully QA'd telemetry data into an easy to approach schema.
Inputs: Approved KPIs (from A2) + Approved Edge Cases (from A4).
Process: Synthesizes the data, applies Gemini-powered explanations for specific metrics, and formats it into an approachable, easy-to-understand UI report.
Outputs: Final Telemetry Report (reiterated hypothesis, defined KPIs, guardrails, and edge cases).
Checks: Readability, completeness, alignment.
(View on Desktop to see demo video)
The Business Impact
Eliminated Blind Spots: Catches massive revenue traps before an experiment is ever deployed.
Massive Time Savings: Reduces telemetry planning and cross-functional swirl from days to hours.
Agentic Workflow Structure
A1 · KPI Generator Agent
Purpose: Generate primary and secondary KPIs to track for a proposed product experiment.
Inputs: Experiment hypothesis.
Process: Analyzes the hypothesis, determines the core business objectives, and maps out the primary and secondary tracking metrics needed to accurately measure success.
Outputs:
Proposed KPIs (e.g., Average Order Value, Conversion Rate).Checks: Relevance to hypothesis, baseline trackability.
A2 · KPI QA Agent
Purpose: Evaluate A1’s output to ensure metrics are sound, trackable, and not hallucinated.
Inputs: A1’s proposed KPIs + original hypothesis.
Process: Acts as a automated "circuit breaker." Audits A1’s work against strict telemetry standards.
If it finds major issues or untrackable metrics, it rejects the output, provides feedback, and forces A1 to run again.
Outputs:
Approved KPIs or Rejection/Feedback Log.Checks: Trackability, business logic validity, zero hallucination.
A3 · Edge Case Generator Agent
Purpose: Identify hidden risks, guardrail metrics, and edge cases to ensure no stone is left unturned.
Inputs: Approved KPIs + original hypothesis.
Process: Hunts for risks and friction points.
Formulates guardrail metrics to protect core business health during the experiment.
Outputs: Proposed Guardrails and Edge Cases.
Checks: Relevance to experiment, severity of impact, gap coverage.
A4 · Edge Case QA Agent
Purpose: Filter theoretical concerns from practical, deployable, actionable risks.
Inputs: A3’s proposed edge cases and guardrails.
Process: Serves as the second "circuit breaker."
Strictly filters A3’s output to ensure only highly actionable, realistic edge cases are passed.
Forces A3 to regenerate if the output does not meet high-quality standards.
Outputs:
Approved Guardrails and Edge Cases.Checks: Deployability, actionability, theoretical vs. practical distinction.
A5 · Final Output Agent
Purpose: Compile and format the fully QA'd telemetry data into an easy to approach schema.
Inputs: Approved KPIs (from A2) + Approved Edge Cases (from A4).
Process: Synthesizes the data, applies Gemini-powered explanations for specific metrics, and formats it into an approachable, easy-to-understand UI report.
Outputs: Final Telemetry Report (reiterated hypothesis, defined KPIs, guardrails, and edge cases).
Checks: Readability, completeness, alignment.