The Nearest Cafe Problem: Why AI Agents Need a Map

Ask an AI agent for the nearest cafe and it answers in a full sentence, street name included: 300 meters ahead, on Rue Saint-Jean. Some of those sentences are wrong, and a wrong one is indistinguishable from a right one at the moment you need the answer. The wrongness is invisible on both ends. The user hears a confident sentence, and the model raises no flag, because nothing in its weights ever told it that it was guessing.

An agent that invents a location is worse than a search box, for one specific reason: a search box at least shows candidates a person can distrust. The fix is not to make the model better at geography. The fix is to stop asking the model for geography altogether. Give it tools that return verified geometry: a place phrase resolved to a coordinate it can check, a polygon that means what the question means, a containment answer computed against the real boundary, and a loud "unsupported" when the data layer underneath has nothing. Mapbox, HERE and Overture have spent much of 2026 publishing about exactly this gap, which is at least proof that the industry has admitted it has a name.

A search box shows its candidates, an agent just asserts

Fluency is an interface hazard in geography in a way it is not elsewhere. A wrong answer to "summarize this document" degrades as prose. A wrong answer to "where is the closest cafe" sends a stranger walking in the wrong direction, and the sentence carrying the error sounds identical to a sentence carrying the truth. A 404 is legible. A plausible street name with a distance in meters is not.

The strange part is that geography questions are almost always checkable in principle. There is a right answer, sitting in a dataset the platform already licenses, and the model declined to consult it. That makes the ungrounded answer less forgivable, not more: the agent is guessing with a door open to the correct answer two tool calls away.

Even a correct coordinate can answer the wrong question. "Nearest" measured how? Straight-line distance from a pixel, walking distance along the street network, or driving? A cafe 300 m away across a river is farther than one 600 m away along the bridge, and an ungrounded agent has conflated all three without noticing there was a choice to make.

Grounding step one: never trust a coordinate the model wrote

The first tool the loop needs turns a place phrase into a resolved, checked point. resolve_place takes the free-text phrase plus what context the conversation offers, and returns a candidate list with identifiers, names, coordinates and source, rather than a single verdict. The model may name a place; only a tool may name a point.

We learned the inversion the expensive way. Our first agent harness let the model answer with coordinates and validated them afterward. The validator fired constantly. Reading near-misses for a week taught the lesson worth printing on a mug: a model asked to produce coordinates produces confident ones, plausible numbers near the right city, wrong by hundreds of meters, wrong enough to matter and formatted to convince. Validation after the fact kept catching ships that had already sailed. The fix was to invert the flow so the ship never leaves the harbor.

Ambiguity survives resolution, and that is fine, as long as it is surfaced. When two candidates sit far apart with similar names (the Station Road problem we wrote about in April, at our post on geocoding a census), the right behavior is to choose one with a stated reason or hand the choice back, never to pick silently and move on.

Sometimes the right answer is a polygon

A large share of the questions people ask a mapping agent have geometry for an answer, and prose is the wrong type for it. "Where can I walk within 15 minutes of the hotel" is an isoline: a polygon computed on the actual walking network for a specific mode and a specific time budget. It changes with the hour and with the season. A 15-minute walk at Sunday morning pace is not the same polygon as a 15-minute walk on a weekday afternoon on the same network. If the agent answers that question with a sentence, it has answered a geometry question with vibes, and the vibes have a mode, a direction of travel, and a clock attached that the sentence never mentions.

Containment questions are the same species. "Is the site inside the school zone" or "is this parcel in the flood layer" is a point-in-polygon computation against the actual boundary data, not a recollection of what a boundary looked like in training. The polygon either holds the point or it does not, and the agent's job is to run that computation and attach the layer's name and date to the answer.

Map the trace, then claim the road

Wherever a raw position shows up in the flow, from a phone fix, GPS trace, courier ping, matching puts it somewhere real. Map matching snaps a noisy polyline to the most likely path on the actual network, so the answer to "where was the crew" references a road that exists rather than a wobble in the sample. Matched points then go through the same reverse-check as geocoded ones: which named feature is this point on, at what distance, and does the distance pass a threshold. The pattern repeats because it works.

The loop, as we build it today

The whole shape fits on one screen:

GROUNDING_TOOLS = {
    "resolve_place": ...,  # phrase + context -> candidates (id, name, lat, lon, source)
    "reverse_check": ...,  # verified coordinate -> nearest feature + distance_m
    "isoline": ...,        # coordinate + minutes + mode -> polygon
    "contains": ...,       # coordinate + layer id -> inside | outside | unsupported
}

def grounded_reply(turn, user_place):
    if turn.writes_coordinates():
        refuse("coordinates must come from a tool result, never from weights")
    place = choose_candidate(GROUNDING_TOOLS["resolve_place"](user_place, turn.context))
    proof = GROUNDING_TOOLS["reverse_check"](place.lat, place.lon)
    if proof.distance_m > 50:
        return ask_user_to_disambiguate(place.candidates)
    if turn.question == "where_can_i_walk_in_15_minutes":
        polygon = GROUNDING_TOOLS["isoline"](place.lat, place.lon, 15, mode="walk")
        return answer_with_geometry(place, polygon, provenance=[proof])
    return answer_with_prose(place, proof)

Three properties do the load-bearing work. The schema makes tool calls the only path to geometry, so the model cannot answer a place question without one. Every geometric claim carries its provenance into the reply, so the sentence and its evidence travel together. And the refusal branch is real code, not a prompt-line request.

When the layer underneath says unsupported

An agent also needs to know when it should have no opinion. We open-sourced everymap-rs, a trait-based client that wraps geocoding, routing, isolines, map matching, tour planning and traffic across HERE, Google, TomTom, Mapbox and Radar: 35 real implementations across 5 providers, first published September 2026. The instructive part for agents is its failure style. A domain a provider does not implement fails loudly, so asking Google for traffic returns an error saying exactly that, instead of an empty success shuffling downstream to become somebody's bug. That error text is what a grounded agent should surface: "I can't answer this from data" beats a fluent guess, every single time, because only the first one can be trusted twice.

Honest limits: the loop is plumbing, and the plumbing leaks

Honest to the end, then. A checked coordinate is a claim with a source, not ground truth. Ambiguity among candidates survives the loop, and the tool picks wrong some of the time, at a rate we have not rigorously measured. Isolines vary with hour and mode, which means a polygon answer is only valid for the clock it was computed under. Nearby-business layers carry the freshness of their feeds, so a cafe that closed last month can be perfectly matched and perfectly dead. And models still attempt to answer from weights whenever the schema leaves a gap, so the forcing has to be strict and reviewed. A prompt line asking the model to please use tools has never stopped one from skipping a tool.

If your agent ever states a distance in meters, a street name, or a pair of coordinates, trace where those numbers came from. If the answer is the weights, the sentence is fiction, however fluent it sounds.

Fluency is not a coordinate system.