Before changing the queue
A simulated insurance sales floor at an assumed scale (3,200 leads, 1,100 quotes, 40 issued and 3 collected policies a week), three call centers dialing the same leads, an AI voice agent on overflow, and one change operations could make on Monday.
- Proposed change
- Give every lead one next-contact owner across the three centers. The other centers stop calling it back.
- Status
- Test onlyBy rule: the logs hold one policy, so no comparison exists. The same logs fit a gain and a loss.
- History readnot identified
- Split test by lead8 weeks, next
- Weekly switchbackshares fewer agents, not none
- Rolloutwith review at week 4
- Why it could reverse
- In the simulated history, when a second center calls within minutes, the lead answers later calls 0.51 times as often (interval 0.48 to 0.55). Either the double call annoys the lead, or cooling leads draw double calls and the second call is a real extra chance. The first says coordinate; the second says leave it.
- Evidence needed
- Leads split at random when they are created, each kept in its arm across all three centers. Read on leads reached per lead, with quotes per reached lead and collected payments as guardrails. The arms share agents; a weekly switchback reduces that before full rollout, without removing it: 56% of calls still fall outside the lead's arrival week.
- Deployment boundary
- Every new lead is eligible; the owner field is honored by every center's dialer; contact windows and attempt limits do not change. One interim look at week 4, on leads that have matured by then: stop if leads reached per lead is below current at the 1% level.
- Review
- Operations director owns it. Reviewed at week 4; reversed if the whole interval on leads reached per lead sits below 1.
Same history, opposite effects
Three versions of this operation differ in one thing no log records: what the second call does to the lead. Run under today's process with the same random draws, all three write the same logs, byte for byte. Here are five of those leads, every dial over their first two weeks, against the hours the centers are open.
Annoyance
A second call within minutes annoys the lead, and later calls reach it less. One owner helps.
No difference
The lead was cooling anyway, and the owner makes that call itself. Quotes do not change; the other centers' calls disappear.
Second chance
The lead was cooling anyway, and the other center's call was a real extra chance that the owner's cadence does not replace. One owner hurts.
- Leads with a double call
- 45%, about 1,951 double calls and 4,677 agent minutes a week
- Later calls after a double call
- 0.51 times as likely to reach the lead, holding attempt number, lead age and source fixed
- The decision code, on these logs
- Test only, in all three versions, by a rule written into it: one policy in the logs, so no comparison exists. It is not a finding; the bench below measures the rest.
- A dashboard rule, on these logs
- “They answer less after a double call, stop double calls”: roll out, in all three. Right once, harmless once, harmful once (100% of runs in the harmful version).
The experiment that separates them
Half the leads get one owner at random when they are created; the other half keep today's process. Eight weeks later, the same decision code reads the three versions again. Their logs now differ.
In this run it rolls out nowhere. In the first version leads are reached more often, but collected payments fell in the owner arm, 7 against 21 (0.35, interval 0.15 to 0.83). Over 30 paired runs of that version the true effect on collections is +0.09 a week, so this drop is chance; the guardrail holds the rollout until it is explained. The first two bench runs had no such guardrail: an outside review found this run, and the guardrail was added. The arms are also uneven, 12,405 against 12,793 matured leads for a 50/50 split (p = 0.015), short of the 0.001 alarm that would void the comparison.
| Version | Leads reached, owner vs current | Quotes per lead | Collected, owner / current | Decision code | Right answer |
|---|---|---|---|---|---|
| Annoyance | 1.06 1.04 to 1.08 | 1.06 1.03 to 1.09 | 7 / 21 0.35, 0.15 to 0.83 | Not yetcollection guardrail | Roll out |
| No difference | 0.99 0.97 to 1.01 | 1.00 0.97 to 1.03 | 11 / 12 0.98, 0.43 to 2.21 | Not yet | No quote change; saves 281 agent minutes a week |
| Second chance | 0.98 0.96 to 1.00 | 0.98 0.95 to 1.01 | 11 / 13 0.90, 0.40 to 2.01 | Not yet | Keep current |
One run of each version, seed 2026, chosen before looking. Ratios are owner arm over current arm, with 95% intervals; the line marks 1. How often each answer comes out over 30 runs, by how long the experiment runs:
How the decision code was tested
I wrote both the simulator and the decision code, so this is not an independent validation. What keeps it from finding what I planted:
- The decision code cannot import the simulator or its truth; a test fails if it does.
- True effects come from running each version under both policies, never from its parameters.
- The collected count is a rate, not a fixed 3: every simulated week draws its own.
- The history-only rows are the rule above, not a measurement. The dashboard rule next to them is.
- The three harder mechanisms were written with the bench, before its first run, and no threshold was set on them. They are not an independent hold-out: the code changed after runs that included them.
- Everything that changed between runs. Run 1 to run 2: a 14-day maturity rule in the decision code (leads from the last days were read before they could mature, which left every switchback undecided); two weeks of dialing after the last lead in the simulator, so those leads can mature; and in the bench, a rollout counts as hurting only when the true effect's whole interval is below zero, not its mean. Run 2's counts under that definition were recomputed from its stored results, and the file says so. Run 2 to run 3, after an outside review: the collection guardrail and a sample-ratio check. The table is run 3; all runs and the commit history are in the repository.
- Waiting is not free: a missed gain counts as loss, so always answering “not yet” would score badly.
| Case, 40 runs each | True effect of full rollout | Decision code: roll out, not yet, keep | Rollouts where it hurts, code / dashboard rule | Loss a week vs best choice, code / hold / dashboard rule |
|---|---|---|---|---|
| Randomized by lead, 8 weeks | ||||
| Beneficial: overlaps annoy leads | +73.0±3.9 | 98%, 2%, 0% | 0 / 0 | 1.8 / 73.0 / 0.0 |
| No quote effect: the owner makes the dial | -2.9±4.2 | 0%, 93%, 7% | 0 / 0 | 0.0 / 0.0 / 1.6 |
| Harmful: the second dial was a real chance | -38.5±4.1 | 0%, 3%, 97% | 0 / 0 | 0.0 / 0.0 / 0.0 |
| History only, no experiment | ||||
| History only, beneficial world | +73.0±3.9 | 0%, 100%, 0% | 0 / 0 | 73.0 / 73.0 / 0.0 |
| History only, no-effect world | -2.9±4.2 | 0%, 100%, 0% | 0 / 0 | 0.0 / 0.0 / 2.9 |
| History only, harmful world | -38.5±4.1 | 0%, 100%, 0% | 0 / 40 | 0.0 / 0.0 / 38.5 |
| A comparison that cannot carry the conclusion | ||||
| Before/after while leads drift, no-effect world | -1.2±3.9 | 0%, 100%, 0% | 0 / 0 | 0.0 / 0.0 / 0.0 |
| Weeks alternate by lead arrival, 8 weeks (retries still cross weeks) | ||||
| Weekly switchback, beneficial world | +73.0±3.9 | 83%, 17%, 0% | 0 / 0 | 12.8 / 73.0 / 0.0 |
| Weekly switchback, harmful world | -38.5±4.1 | 0%, 40%, 60% | 0 / 0 | 0.0 / 0.0 / 0.0 |
| Where the fast metric and the final outcome can disagree | ||||
| Closing script: more quotes, fewer collections | -1.51±0.22 | 0%, 78%, 22% | 0 / 40 | 0.00 / 0.00 / 1.51 |
| Closing script: more quotes, more collections | +0.13±0.31 | 8%, 92%, 0% | 0 / 0 | 0.12 / 0.13 / 0.00 |
| Harder mechanisms, written before the first bench run | ||||
| Annoyance that wears off after a day | +0.0±4.3 | 3%, 87%, 10% | 0 / 0 | 0.0 / 0.0 / 0.0 |
| Overlaps land on hard-to-reach leads (harmful) | -29.0±4.2 | 0%, 20%, 80% | 0 / 0 | 0.0 / 0.0 / 0.0 |
| 15% fewer agents, arms compete harder | +64.5±4.8 | 95%, 5%, 0% | 0 / 0 | 3.2 / 64.5 / 0.0 |
Effects and losses are in quotes a week for the ownership cases and in collected policies a week for the closing script. True effects are means over 30 paired runs of each version under both policies, with their 95% margin; a rollout counts as hurting only where that whole interval is below zero. Across every case with an experiment (10 cases, 40 runs each), the decision code's intervals contain the true full-rollout ratio in 90% to 100% of runs. Every count is in the repository.
Planning, under stated assumptions
Volume only matters once a comparison is valid. Read as if 3 collected per 3,200 leads were a mature per-lead probability, with all 3,200 leads eligible and split in half: +20% takes about 288 weeks; +50% takes about 53 weeks; +100% takes about 16 weeks. That is a planning calculation, not a forecast for any real operation.
- Two independent proportions, fixed horizon, two-sided 5%, power 80%.
- Assumes the ratio above is a mature per-lead probability. A weekly count of 3 collected against 3,200 new leads is not shown to be one.
- Leads assumed independent and all eligible. Clustering by center or agent, restricted eligibility or several comparisons all lengthen it.
- Collected premium would also need the spread of premium amounts.
Before any of this, on real data
40 issued and 3 collected in the same week is a ratio, not yet a collection rate. The first check is to match each issued policy to its due date and to settled cash, older cohorts included: policy and lead IDs, issue and first due dates, payment attempts, settlements and reversals, amounts, statuses, event and ingestion times.
If the gap disappears after matching, the collection-failure hypothesis is rejected.
Decision record, a Foundry-oriented specification
What has to exist before the change runs, so its result can be judged after. A proposed mapping, not a deployed action.
- Evidence and choice: snapshot, cutoff, policy version, eligible population, baseline, expected effect, assumptions.
- Assignment and execution: unit, arm, probability, approval, exposure, the dialer's acknowledgement, eligible leads left untreated and why.
- Evaluation: outcome horizon, maturity rule, cost definition, stop conditions, review date, result, counterfactual.
Parameter ledger: assumed scale, assumed, not identified
| assumed scale | Leads per week: 3,200 | Sizes the synthetic operation, nothing else. |
| assumed scale | Quotes per week: 1,100 | Target for the synthetic funnel. |
| assumed scale | Issued policies per week: 40 | Target for the synthetic funnel. A weekly count, not a cohort. |
| assumed scale | Collected policies per week: 3 | Target for a RATE; each synthetic week draws its own count. |
| assumed scale | Ad spend per week: 164,000 MXN | Split between Meta and Google by us. |
| assumed scale | Call center cost per week: 234,000 MXN | Split between three centers at hourly rates set by us. |
| assumed | Three centers dial the same leads; an AI voice agent takes overflow | The structure of the synthetic operation. |
| assumed | Center hourly rates: 165 / 135 / 110 MXN | Chosen so the three add up to the assumed total. |
| assumed | Contact, quote, issue and collection probabilities: see truth.json | Tuned to land near the assumed weekly counts. |
| assumed | Hour-of-day response curve: see truth.json | People answer more after work. |
| assumed | Overlap rate between centers: 12% of dials | How often a second center dials within minutes. |
| not identified | What an overlap does to the lead: three worlds | The subject of the test: history cannot tell. |
| not identified | Payment delays, cancellations, cohort maturity | Not modelled: in this simulation a first payment is known at issue. |
| not identified | Fees, retained revenue, who pays which cost | Left out on purpose: no margin is computed. |
| assumed | Premium amounts: lognormal by source | Only used to fill the policy table. |
The decision record as JSON
{
"object_type": "AllocationDecision",
"decision_id": "next-contact-owner-001",
"evidence_and_choice": {
"snapshot": "synthetic, seed 2026, 8 weeks",
"cutoff": "end of week 8",
"policy_version": "multi-center retries v0",
"eligible_population": "every new lead, all sources, all centers",
"baseline_metric": "leads reached per lead",
"proposed_change": "one next-contact owner per lead across centers",
"expected_effect": "unknown sign: not identified from history",
"assumptions": [
"overlap mechanism unknown"
]
},
"assignment_and_execution": {
"experimental_unit": "lead (canonical lead ID across all centers)",
"arms": [
"current",
"single owner"
],
"assignment_probability": 0.5,
"assignment_logged_at": "lead creation, before any dial",
"eligible_but_untreated": "logged with reason (no consent, outside contact window, duplicate)",
"approval": "operations director",
"exposure": "owner field honoured by every center's dialer",
"dialer_acknowledgement": "per-dial check that the dialing center is the owner",
"stop_conditions": [
"one interim look at week 4, on leads matured by then (arrived in weeks 1 and 2): stop if leads reached per lead is below current at the 1% level",
"any center dials outside the permitted window"
]
},
"evaluation": {
"outcome_horizon": "contact and quote: 14 days after lead creation; first payment: when mature",
"maturity_rule": "a lead counts once 14 days have passed. On real data a policy counts once its first due date has passed; in this simulation the first payment is known at issue.",
"cost_definition": "agent minutes per lead, from dialer handle time",
"review_after_weeks": 4,
"reversal_criterion": "interval on leads reached per lead entirely below 1",
"observed_result": null,
"counterfactual_estimate": null
}
}Simulator, decision code, bench, tests and the published run's logs: github.com/fred1433/queue-decision-test. Built 2026-09-27. No live AI on this page.