The eval behind
a connector.
Duskline is my own product, an evening planner for Toronto. Its engine is also an MCP server. A language model reads a request; an exact-match eval checks what it understood. This is the method I bring to a connector build.
Visit Duskline ↗
Rounds 1 and 6
I changed the output contract, decoding and policy checks between these runs of claude-haiku-5-5. Both were recorded on 19 October 2026.
| Check | Round 1 | Round 6 |
|---|---|---|
| Gates | Failed | Passed |
| Exact requests | 64 / 80 | 75 / 80 |
| Exact fraction | 0.8 | 0.9375 |
| Invalid outputs | 4 | 0 |
| Invented locations | 0 | 0 |
| Must-ask lines asked | 10 / 10 | 10 / 10 |
| False ready | 0 | 0 |
| p50 latency · ms | 1,060 | 1,039 |
| p95 latency · ms | 2,190 | 1,508 |
| Reference USD / read | 0.000217 | 0.000203 |
| Reference USD / run | 0.01734206 | 0.0162595 |
Approximate claude-haiku-5-5 reference rates, not actual billing. Cache reads discounted; cache-write premiums are not modelled.
Recorded metrics and provenance ↗The 80 live requests
ramen then a warm cup near sherbourne common
R1 missed / R6 missedsushi then a celebratory glass near tommy thompson park
R1 missed / R6 missedtacos then something sugary near humber arboretum
R1 missed / R6 missedtacos but skip cilantro near sugar beach
R1 missed / R6 misseddumplings but hold the pork near chester hill lookout
R1 missed / R6 missedpizza in a hurry
R1 exact / R6 exactsushi at a reasonable price
R1 exact / R6 exactramen in a rush
R1 exact / R6 exacttacos on a budget in a hurry
R1 missed / R6 exactdumplings in a pinch
R1 exact / R6 exactpho at an easy pace
R1 exact / R6 exactnoodles in good company
R1 exact / R6 exactpizza by myself
R1 exact / R6 exactsushi in peace
R1 exact / R6 exacttacos in a relaxed mood
R1 exact / R6 exactramen at a gentle price
R1 missed / R6 exactdumplings in a celebratory mood
R1 exact / R6 exactpho in a mellow mood
R1 exact / R6 exactpizza at a leisurely pace
R1 exact / R6 exactsushi in a festive mood
R1 exact / R6 exacttacos by candlelight
R1 exact / R6 exactramen in a low key setting
R1 exact / R6 exactdumplings in a sociable mood
R1 exact / R6 exactpho at a modest price
R1 exact / R6 exactpizza in a thoughtful mood
R1 exact / R6 exactramen near flibbertown
R1 exact / R6 exactcoffee near the east mall
R1 exact / R6 exactsushi near rouge hill
R1 exact / R6 exacttacos near sugar beach
R1 exact / R6 exactramen near the well
R1 missed / R6 exactcoffee near keele station
R1 exact / R6 exactpizza near the kay gardner beltline
R1 exact / R6 exactdumplings near crothers woods
R1 exact / R6 exactpho near chester hill lookout
R1 exact / R6 exactgelato near sherbourne common
R1 exact / R6 exacttea near tommy thompson park
R1 exact / R6 exacttacos near humber arboretum
R1 missed / R6 exactsushi near trillium park
R1 exact / R6 exactcoffee near colonel sam smith park
R1 exact / R6 exactramen near the bentway
R1 missed / R6 exactsomething comforting after a long day
R1 exact / R6 exactsomething warm after a tiring shift
R1 exact / R6 exactsomething soothing after a rough afternoon
R1 exact / R6 exactsomething familiar after a difficult week
R1 exact / R6 exactsomething hearty after an exhausting commute
R1 exact / R6 exactI could use something nourishing after today
R1 exact / R6 exactin the mood for something indulgent
R1 exact / R6 exacthoping for something warming this evening
R1 exact / R6 exactall I want is something simple
R1 exact / R6 exactfeeling like something satisfying tonight
R1 exact / R6 exactsomething restorative after a tiring shift
R1 exact / R6 exactsomething cheerful after a grey afternoon
R1 exact / R6 exactsomething filling after an endless day
R1 exact / R6 exactsomething comforting for a weary evening
R1 exact / R6 exactsomething nostalgic after a long journey
R1 exact / R6 exactramen then something sweet near flibbertown
R1 missed / R6 exactsushi near the east mall then something warming
R1 exact / R6 exacttacos then a sweet finish near rouge hill
R1 exact / R6 exactpizza then a quiet cup near sugar beach
R1 missed / R6 exactdumplings then something fizzy near crothers woods
R1 exact / R6 exactpho then something cold and sweet near chester hill lookout
R1 missed / R6 exactcoffee then something filling near colonel sam smith park
R1 exact / R6 exactno chains, something quick near the east mall
R1 exact / R6 exactsomething not too spicy near the east mall
R1 exact / R6 exactramen without mushrooms near flibbertown
R1 exact / R6 exactsushi with no mayo near rouge hill
R1 exact / R6 exactpizza without olives near crothers woods
R1 missed / R6 exactpho with nothing fried near sherbourne common
R1 exact / R6 exactcoffee but avoid chains near tommy thompson park
R1 missed / R6 exacttacos without cheese near humber arboretum
R1 missed / R6 exactdinner by the water
R1 exact / R6 exactpizza near the game
R1 exact / R6 exactcoffee around campus
R1 exact / R6 exactsushi by the station
R1 exact / R6 exactramen near the market
R1 exact / R6 exacttacos near the park
R1 exact / R6 exactdumplings close to the museum
R1 exact / R6 exactpho around the mall
R1 exact / R6 exactgelato by the shoreline
R1 exact / R6 exactdrinks near the stadium
R1 exact / R6 exactField differences, latency and recorded replay
Duskline reader · Recorded live eval
· claude-haiku-5-5
Affan Yasir’s own product
Round 1
Gates failed64 / 80exact (0.80)
4 invalid outputs
Round 6 · Shipped
Gates passed75 / 80exact (0.94)
0 invalid outputs
Round 6 checks
- Invented locations
- 0
- Must-ask lines asked
- 10 / 10
- p95 latency
- 1.5 s
About USD 0.0002 per read at reference rates.
The 5 misses · Round 6
r057Flagged Sherbourne Common as unknown.
ramen then a warm cup near sherbourne common
- Field
unknowns- Expected
null- Returned
["Sherbourne Common"]
r058Added a “celebratory” preference.
sushi then a celebratory glass near tommy thompson park
- Field
preferences- Expected
null- Returned
["celebratory"]
r059Flagged Humber Arboretum as unknown.
tacos then something sugary near humber arboretum
- Field
unknowns- Expected
null- Returned
["Humber Arboretum"]
r065Flagged Sugar Beach as unknown.
tacos but skip cilantro near sugar beach
- Field
unknowns- Expected
null- Returned
["Sugar Beach"]
r067Flagged Chester Hill Lookout as unknown.
dumplings but hold the pork near chester hill lookout
- Field
unknowns- Expected
null- Returned
["Chester Hill Lookout"]
Replay of the run recorded on 19 October 2026. Plays at a reading pace.
Inspect all 80 recorded lines Round 6
Scroll across for verdicts and latency; down for every line. Scroll the table to inspect every line. “Exact” means a match to the authored expectation.
| Line / kind | Request | Verdict / difference | Must ask | Asked | Latency · ms |
|---|---|---|---|---|---|
| r001idiom | pizza in a hurry | Exact | No | No | 1,392 |
| r002idiom | sushi at a reasonable price | Exact | No | No | 1,682 |
| r003idiom | ramen in a rush | Exact | No | No | 2,110 |
| r004idiom | tacos on a budget in a hurry | Exact | No | No | 1,116 |
| r005idiom | dumplings in a pinch | Exact | No | No | 1,737 |
| r006idiom | pho at an easy pace | Exact | No | No | 874 |
| r007idiom | noodles in good company | Exact | No | No | 963 |
| r008idiom | pizza by myself | Exact | No | No | 1,039 |
| r009idiom | sushi in peace | Exact | No | No | 1,031 |
| r010idiom | tacos in a relaxed mood | Exact | No | No | 1,018 |
| r011idiom | ramen at a gentle price | Exact | No | No | 1,158 |
| r012idiom | dumplings in a celebratory mood | Exact | No | No | 1,176 |
| r013idiom | pho in a mellow mood | Exact | No | No | 1,455 |
| r014idiom | pizza at a leisurely pace | Exact | No | No | 1,165 |
| r015idiom | sushi in a festive mood | Exact | No | No | 939 |
| r016idiom | tacos by candlelight | Exact | No | No | 968 |
| r017idiom | ramen in a low key setting | Exact | No | No | 898 |
| r018idiom | dumplings in a sociable mood | Exact | No | No | 1,016 |
| r019idiom | pho at a modest price | Exact | No | No | 1,039 |
| r020idiom | pizza in a thoughtful mood | Exact | No | No | 1,344 |
| r021unknown-name | ramen near flibbertown | Exact | No | No | 1,022 |
| r022unknown-name | coffee near the east mall | Exact | No | No | 1,007 |
| r023unknown-name | sushi near rouge hill | Exact | No | No | 948 |
| r024unknown-name | tacos near sugar beach | Exact | No | No | 1,216 |
| r025unknown-name | ramen near the well | Exact | No | No | 1,130 |
| r026unknown-name | coffee near keele station | Exact | No | No | 839 |
| r027unknown-name | pizza near the kay gardner beltline | Exact | No | No | 1,000 |
| r028unknown-name | dumplings near crothers woods | Exact | No | No | 894 |
| r029unknown-name | pho near chester hill lookout | Exact | No | No | 952 |
| r030unknown-name | gelato near sherbourne common | Exact | No | No | 931 |
| r031unknown-name | tea near tommy thompson park | Exact | No | No | 1,123 |
| r032unknown-name | tacos near humber arboretum | Exact | No | No | 1,081 |
| r033unknown-name | sushi near trillium park | Exact | No | No | 854 |
| r034unknown-name | coffee near colonel sam smith park | Exact | No | No | 1,243 |
| r035unknown-name | ramen near the bentway | Exact | No | No | 1,037 |
| r036open-ended | something comforting after a long day | Exact | No | No | 892 |
| r037open-ended | something warm after a tiring shift | Exact | No | No | 1,078 |
| r038open-ended | something soothing after a rough afternoon | Exact | No | No | 842 |
| r039open-ended | something familiar after a difficult week | Exact | No | No | 1,045 |
| r040open-ended | something hearty after an exhausting commute | Exact | No | No | 1,024 |
| r041open-ended | I could use something nourishing after today | Exact | No | No | 989 |
| r042open-ended | in the mood for something indulgent | Exact | No | No | 1,061 |
| r043open-ended | hoping for something warming this evening | Exact | No | No | 816 |
| r044open-ended | all I want is something simple | Exact | No | No | 923 |
| r045open-ended | feeling like something satisfying tonight | Exact | No | No | 1,508 |
| r046open-ended | something restorative after a tiring shift | Exact | No | No | 1,001 |
| r047open-ended | something cheerful after a grey afternoon | Exact | No | No | 908 |
| r048open-ended | something filling after an endless day | Exact | No | No | 1,270 |
| r049open-ended | something comforting for a weary evening | Exact | No | No | 967 |
| r050open-ended | something nostalgic after a long journey | Exact | No | No | 740 |
| r051chained | ramen then something sweet near flibbertown | Exact | No | No | 1,235 |
| r052chained | sushi near the east mall then something warming | Exact | No | No | 1,070 |
| r053chained | tacos then a sweet finish near rouge hill | Exact | No | No | 1,025 |
| r054chained | pizza then a quiet cup near sugar beach | Exact | No | No | 1,228 |
| r055chained | dumplings then something fizzy near crothers woods | Exact | No | No | 1,432 |
| r056chained | pho then something cold and sweet near chester hill lookout | Exact | No | No | 1,201 |
| r057chained | ramen then a warm cup near sherbourne common | Miss unknowns: expected null; got ["Sherbourne Common"] | No | No | 1,328 |
| r058chained | sushi then a celebratory glass near tommy thompson park | Miss preferences: expected null; got ["celebratory"] | No | No | 1,052 |
| r059chained | tacos then something sugary near humber arboretum | Miss unknowns: expected null; got ["Humber Arboretum"] | No | No | 1,229 |
| r060chained | coffee then something filling near colonel sam smith park | Exact | No | No | 1,124 |
| r061negation | no chains, something quick near the east mall | Exact | No | No | 1,027 |
| r062negation | something not too spicy near the east mall | Exact | No | No | 1,024 |
| r063negation | ramen without mushrooms near flibbertown | Exact | No | No | 919 |
| r064negation | sushi with no mayo near rouge hill | Exact | No | No | 1,024 |
| r065negation | tacos but skip cilantro near sugar beach | Miss unknowns: expected null; got ["Sugar Beach"] | No | No | 1,099 |
| r066negation | pizza without olives near crothers woods | Exact | No | No | 1,038 |
| r067negation | dumplings but hold the pork near chester hill lookout | Miss unknowns: expected null; got ["Chester Hill Lookout"] | No | No | 1,044 |
| r068negation | pho with nothing fried near sherbourne common | Exact | No | No | 1,118 |
| r069negation | coffee but avoid chains near tommy thompson park | Exact | No | No | 1,044 |
| r070negation | tacos without cheese near humber arboretum | Exact | No | No | 1,207 |
| r071must-ask | dinner by the water | Exact | Yes | Yes | 0 |
| r072must-ask | pizza near the game | Exact | Yes | Yes | 0 |
| r073must-ask | coffee around campus | Exact | Yes | Yes | 0 |
| r074must-ask | sushi by the station | Exact | Yes | Yes | 0 |
| r075must-ask | ramen near the market | Exact | Yes | Yes | 0 |
| r076must-ask | tacos near the park | Exact | Yes | Yes | 0 |
| r077must-ask | dumplings close to the museum | Exact | Yes | Yes | 0 |
| r078must-ask | pho around the mall | Exact | Yes | Yes | 0 |
| r079must-ask | gelato by the shoreline | Exact | Yes | Yes | 0 |
| r080must-ask | drinks near the stadium | Exact | Yes | Yes | 0 |
Offline corpus · Separate test
112 authored cases
- After the fixes
- 112 / 112 exact
A separate offline corpus.
112 authored regression cases.
112/112 exact
After the fixes. This corpus is separate from the 80 live reader requests.
For your connector, we agree the expected tool,
arguments and refusal before the build.
Email Affan