NeurIPS 2026 · Track on Evaluations and Datasets

GeoFidelity-Bench: Evaluating Block-Conditioned Geographic Fidelity in Text-to-Image Street-View Generation

Kaizhen Tan1,2,3

  1. 1Heinz College of Information Systems and Public Policy, Carnegie Mellon University
  2. 2Robert F. Wagner Graduate School of Public Service, New York University
  3. 3Shanghai Key Laboratory of Urban Design and Urban Science, NYU Shanghai

Contact kt3275@nyu.edu

named street blocks, carved from OpenStreetMap ways
112
cities in 23 countries on six continents
25
reference assignments from 7,433 distinct Mapillary images
7,563
generated images from six open-weight models
16,128
TokyoNew YorkDubaiTorontoBuenos AiresMumbaiSeoulShanghaiSydneyBangkokCape TownSao PauloSingaporeBogotaCairoMexico CitySan FranciscoMelbourneNairobiEurope2.0× scaleParisLondonAmsterdamIstanbulRomeBerlin
Benchmark cities. Marker area is proportional to the number of named street blocks. Europe is enlarged in the inset. Hover or tap a city for its counts.

Does the image show the named street, or only the city?

A text-to-image model can render a believable “street in New York City” from a broad city stereotype. Common metrics measure image quality or text alignment; without real images of the requested place and its neighbors, they cannot show whether an image resembles that place. GeoFidelity-Bench compares generated street views with real Mapillary panels of the requested block and with panels of other blocks in the same city.

Local names help

+0.036

CosSim, L0 → L1

Street and neighborhood names raise similarity to the target panel for all six generators (95% CI 0.029 to 0.042).

Prompt conditioning

Coordinates add little

+0.004

CosSim, L1 → L2

Raw GPS coordinates written into the prompt give a small, uneven gain. Only 14 of 25 cities improve.

Prompt conditioning

The block is often missed

40% vs 86% real

top-1 retrieval, L1 prompts

Generated images rank their target block first in 40% of cases. Held-out real images of the same block do so in 86%.

Model comparison

CLIP score is no proxy

0.51

AUROC, CLIP vs retrieval

Prompt alignment predicts whether all four images retrieve the target block at chance level (95% CI 0.44 to 0.57).

Text alignment

Abstract

Text-to-image models can generate plausible street scenes from prompts such as “a street in New York City,” but it remains unclear whether these images resemble the requested place or show a generic city appearance. Commonly used evaluation metrics mainly measure image quality or text alignment and do not directly assess geographic fidelity. We introduce GeoFidelity-Bench, a benchmark for evaluating whether generated street-view images match a target location at the level of named street blocks. The benchmark contains 112 street blocks from 25 cities across six continents, each paired with a panel of real Mapillary images and geographic metadata. From this metadata we build prompts with the city only, with street and neighborhood names, and with these names plus raw GPS coordinates. Across six open-weight text-to-image models, street and neighborhood names improve similarity to the target location, while raw GPS coordinates add only a small and uneven gain. Control prompts that swap in the names of other blocks while keeping the city fixed reduce retrieval accuracy, with larger decreases when the neighborhood name or both names are replaced. Generated images also distinguish their target blocks from local alternatives less reliably than held-out real images. CLIP alignment between generated images and their prompts is nearly uninformative about whether the images achieve block-level geographic fidelity. GeoFidelity-Bench provides a controlled testbed for studying location-conditioned generation and for measuring progress beyond generic city-level plausibility.

Named street blocks with real reference panels

Each target is a named OpenStreetMap street block. Its geometry selects the Mapillary images that form the reference panel, and its metadata fills three prompt levels. Every generator produces four images per block and prompt level, and shared vision encoders compare that generated panel with the target panel and with the panels of related blocks.

Overview diagram. A target block from an OSM named way feeds two branches: a Mapillary reference panel built by OSM matching, fixed filters, and panel selection, and a generated panel from six open-weight models with four images each under prompt levels L0 (city, country), L1 (plus street and neighborhood), and L2 (plus GPS as text). Shared DINOv2 and Mask2Former encoders compute CosSim, DCSF and MMD, GAAS, and retrieval against negatives, which produce the reported scores.
Figure 1 · Overview. Scroll sideways or tap to open it at full size. Thumbnails show real references and SDXL images under L1 prompts for one block in each of Paris, Tokyo, and New York.

Reference panel and generated panel

One block, two panels

The same blocks as in Figure 1, at a larger size. A reference panel keeps at least 25 real images of the block (67.5 on average); three are shown here. The generated panel holds the four SDXL images for the L1 prompt of that block.

Mapillary reference panel· real images

Mapillary reference image 1 of the Paris blockMapillary reference image 2 of the Paris blockMapillary reference image 3 of the Paris block

Generated panel· SDXL, L1 prompt

SDXL image 1 for the L1 prompt of the Paris blockSDXL image 2 for the L1 prompt of the Paris blockSDXL image 3 for the L1 prompt of the Paris blockSDXL image 4 for the L1 prompt of the Paris block

Table 1

Benchmark summary

Blocks are named OSM highway=* ways within 6 km of each city center, stratified into major, residential, pedestrian, and tertiary roads. Ways longer than 2 km are truncated, and block centroids are at least 400 m apart. Mapillary images along each block pass fixed camera, daylight, spatial, scene-content, and image-quality filters. Four images per block are held out as real queries; the rest form the evaluation panel.

Cities
25
six continents
Named street blocks
112
OSM ways, four road strata
Reference assignments
7,563
7,433 distinct images
Average images per block
67.5
minimum 25 per block
Prompt levels
3
city; + street, neighborhood; + GPS
Generator models
6
open-weight text-to-image
Retrieval entries per block
≤ 5
target plus available negatives

Table 2 · prompt templates

Three prompt levels, filled from block metadata

Each level adds location information to the same sentence. Highlighted text is what the level adds.

L0city

A street-level photograph taken in city, country.

L1+ names

A street-level photograph taken on street in the neighborhood district of city, country.

L2+ GPS

A street-level photograph taken on street in the neighborhood district of city, country, near GPS coordinates (lat, lon).

Every prompt continues with “The image shows a typical street scene with buildings, roads, and urban environment characteristic of this location.” (this block at L1 and L2) and ends with “Photorealistic, daytime, clear weather.” Coordinates are the block centroid written to four decimals. All generators except the two FLUX models also receive a shared negative prompt.

Same-city controls

Control prompts that point to the wrong block

Controls keep the L1 template, city, and country, and replace local names with those of another benchmark block, preferring the same city and road stratum. If the correct names matter, controls should score worse than L1.

C1street

… taken on streetanother block’s street in the neighborhood district of city, country.

Wrong street

C2area

… taken on street in the neighborhoodanother block’s neighborhood district of city, country.

Shuffled neighborhood

C3both

… taken on streetanother street in the neighborhoodanother neighborhood district of city, country.

Wrong street and neighborhood

Appendix G · city list

Coverage follows where people map

Reference counts come from Mapillary contributions and OSM naming density. Tokyo holds 12.0% of all reference assignments and New York 10.1%, while Berlin, Melbourne, and Nairobi keep one block each. City-balanced estimates in the paper give every city equal weight.

Reference assignments per city

Bar length is the number of block–image assignments; the figure after it is the number of blocks.

  1. TokyoJP91010 blocks
  2. New YorkUS7679 blocks
  3. ParisFR5008 blocks
  4. TorontoCA4927 blocks
  5. DubaiAE4457 blocks
  6. LondonGB4457 blocks
  7. MumbaiIN4036 blocks
  8. AmsterdamNL3986 blocks
  9. RomeIT3825 blocks
  10. Buenos AiresAR3766 blocks
  11. ShanghaiCN3344 blocks
  12. SeoulKR3115 blocks
  13. IstanbulTR3005 blocks
  14. SydneyAU2134 blocks
  15. Cape TownZA1893 blocks
  16. SingaporeSG1823 blocks
  17. BangkokTH1703 blocks
  18. BogotaCO1532 blocks
  19. Mexico CityMX1512 blocks
  20. Sao PauloBR1423 blocks
  21. CairoEG1002 blocks
  22. MelbourneAU681 block
  23. San FranciscoUS662 blocks
  24. NairobiKE381 block
  25. BerlinDE281 block

One primary score and four diagnostics

Let Xg be the generated images for a block and Xr its reference panel. CosSim is the main score. The other quantities expose different ways a generated panel can fail.

Image embeddings DINOv2 ViT-B/14 Segmentation Mask2Former, Mapillary Vistas Text alignment check CLIP ViT-B/32

CosSim

↑ higher is betterPrimary

Does the generated panel look like the target block?

Cosine similarity between the mean DINOv2 embeddings of the generated panel and the reference panel. Averaging gives every image equal weight within its panel.

MMD

↓ lower is better

How far apart are the two embedding distributions?

Unbiased squared estimate with an RBF kernel: kgg and krr average over distinct pairs within a set, kgr over all cross pairs. γ is the reciprocal of the median positive squared distance.

Exposes distribution drift

DCSF

↓ lower is better

Does the set match the panel without losing variety?

Diversity-Calibrated Set Fidelity adds a penalty when generated images are less diverse than the reference panel. Div is the mean cosine distance between distinct images in a panel.

Exposes mode collapse or drift

GAAS

↓ lower is better

Is the mix of road, sky, and buildings right?

Geo-Attribute Agreement Score compares 10-bin histograms of pixel ratios for eight groups: road, sidewalk, building, vegetation, sky, pole, sign, and vehicle.

Exposes wrong semantic composition

Retrieval

↑ higher is better

Is the target ranked above nearby alternatives?

rb,i is the rank of the target panel for generated image i (n = 4). The local margin is the similarity to the target minus the larger of the two same-city similarities.

Exposes confusion with hard negatives

Prompt and control comparisons are paired by model and block (672 matched pairs per condition), with 95% intervals from 5,000 bootstrap resamples. City-balanced estimates average within each city first.

Six open-weight generators on 112 blocks

SDXL, Stable Diffusion 3.5 Large, FLUX.1-dev, FLUX.1-schnell, PixArt-Σ, and HunyuanDiT each generate four 1024 × 1024 images per block under every prompt condition.

§6.1 · Table 4

Real images find their block. Generated images barely do.

Held-out real images are much closer to their own block (0.734) than to the two same-city alternatives (0.557 and 0.526), so the panels carry block-specific appearance. Generated L1 panels are almost as close to the first same-city alternative (0.449) as to the target (0.451).

Similarity of a query panel to each candidate panel

Mean DINOv2 image-to-panel cosine similarity, averaged over blocks where the candidate exists

  • Held-out real images
  • Six-generator mean, L1
Table view
Query setSame blockSame city (1)Same city (2)Same driveRandom
Held-out real0.7340.5570.5260.1930.204
Six-generator mean0.4510.4490.4260.1430.150

§6.2 · Tables 5 and 6

Street and neighborhood names help. Coordinates add little.

Moving from L0 to L1 raises mean CosSim by +0.036 over 672 matched (model, block) pairs, and 21 of 25 cities improve. Appending raw GPS coordinates changes it by only +0.004, and 14 of 25 cities improve.

CosSim by prompt level

Higher is better. Hover a model to trace it.

  • One generator
  • Six-model mean
Table view
ModelL0L1L2
SDXL0.5120.5360.536
SD 3.5 Large0.4320.5150.516
FLUX.1-dev0.4560.4840.496
FLUX.1-schnell0.4650.4990.507
PixArt-Σ0.4590.4860.489
HunyuanDiT0.4580.4770.478

Paired change in CosSim

Matched (model, block) pairs, and the same change with every city weighted equally

ComparisonMatched Δ [95% CI]Positive pairs
L1 − L0+0.036 [0.029, 0.042]438 / 672
L2 − L1+0.004 [0.001, 0.008]382 / 672
ComparisonCity-balanced Δ [95% CI]Cities up
L1 − L0+0.042 [0.025, 0.060]21 / 25
L2 − L1+0.006 [0.001, 0.012]14 / 25

Leaving out one city at a time, the L1 − L0 gain stays positive in all 25 repetitions: CosSim +0.030 to +0.038, retrieval +0.040 to +0.054 (Appendix D).

§6.3 · Table 7

The correct names carry block-specific information

Retrieval accuracy is lower under all three controls than under L1, most of all when both names are replaced. Swapping only the street name leaves CosSim almost unchanged (+0.001), while a wrong neighborhood or two wrong names lower it by 0.007 and 0.011.

Benefit of the correct names over each control

L1 minus control over 672 matched pairs, with 95% bootstrap intervals. Right of zero means L1 is better.

Table view
Control promptCosSimCosSim benefit [95% CI]RetrievalRetrieval benefit [95% CI]
Wrong street0.498+0.001 [−0.004, 0.006]0.373+0.025 [0.008, 0.042]
Shuffled neighborhood0.492+0.007 [0.003, 0.012]0.361+0.038 [0.020, 0.057]
Wrong street and neighborhood0.488+0.011 [0.005, 0.017]0.344+0.054 [0.034, 0.074]

§6.4 · Tables 8 and 9

CLIP alignment says little about the place

For the 672 L1 panels, CLIP ViT-B/32 similarity with the exact generation prompt is weakly related to every geographic outcome, and it separates panels whose four images all retrieve the target no better than chance. A location-only text (“A photo of {street} in the {neighborhood} district of {city}, {country}.”) tracks CosSim somewhat better but still says little about retrieval.

CLIP score against geographic outcomes

Spearman ρ and AUROC with 95% block-bootstrap intervals

  • Exact prompt
  • Location-only text
Table view
Geographic outcomeExact promptLocation-only text
Target CosSim (ρ)0.112 [−0.008, 0.231]0.238 [0.114, 0.353]
Local margin (ρ)0.005 [−0.117, 0.126]0.123 [−0.012, 0.249]
Retrieval accuracy (ρ)0.041 [−0.068, 0.149]0.105 [−0.019, 0.222]
All-four retrieval (AUROC)0.508 [0.442, 0.572]0.579 [0.505, 0.652]

Panels where no image finds the target

Share of L1 panels, grouped by exact-prompt CLIP score

Table view
PanelsExact promptLocation-only
Lowest CLIP quartile53.0%54.2%
All 672 panels47.3%47.3%
Highest CLIP quartile (168)46.4%44.0%

CLIP barely reacts to the local names

Mean drop in CLIP score when the scoring text names the wrong street or neighborhood. Generated images are held fixed (Table 9).

Wrong nameDrop [95% CI]Panels
Wrong street0.0018 [0.0009, 0.0027]60.1%
Shuffled neighborhood0.0021 [0.0010, 0.0033]58.0%
Both local names0.0047 [0.0030, 0.0064]64.6%

“Panels” is the share of panels where the correct text scores higher. In 35% to 42% of panels the altered text scores at least as high.

§6.5 · Table 10

Model comparison under named-block prompts

SDXL has the best generator scores on CosSim, DCSF, MMD, and GAAS, and PixArt-Σ the highest generator retrieval accuracy. Held-out real images beat all six generators on every metric. Real-image anchors from other blocks show that CosSim and retrieval measure different things.

L1 leaderboard with real-image anchors

Select a column to sort the generators. A dot marks the best generator on each metric.

  • Generated panels
  • Real-image anchors
Six-generator L1 results with real-image anchors. CosSim and retrieval are higher-is-better; DCSF, MMD, and GAAS are lower-is-better.
Model
Generators · L1 named-block prompts
1SDXL
0.536
0.547
0.457
0.380
0.408
2SD 3.5 Large
0.515
0.565
0.470
0.387
0.375
3FLUX.1-schnell
0.499
0.571
0.481
0.395
0.382
4PixArt-Σ
0.486
0.625
0.522
0.401
0.422
5FLUX.1-dev
0.484
0.634
0.527
0.402
0.388
6HunyuanDiT
0.477
0.637
0.531
0.421
0.420
Six-model mean
0.499
0.596
0.498
0.398
0.399
Real-image anchors
Held-out Real
0.902
0.036
0.020
0.285
0.855
Random-Same-Country
0.710
0.160
0.151
0.365
0.259
Random-Global
0.333
0.249
0.249
0.370
0.163

Held-out Real uses the four real images held out from each target block. Random-Same-Country and Random-Global are panels of four real images sampled from other blocks in the same country and from all other blocks.

Looking like the area is not the same as finding the block

Random real images from the same country reach CosSim 0.710, above every generator, yet retrieve the target only 25.9% of the time.

  • Generator, L1
  • Real, same block
  • Real, other blocks

§6.6 · Figure 2

What each model draws for a city

City-only (L0) prompts carry no block information, so these images show what each model associates with a city. They reproduce tropical vegetation, European street widths, dense high-rise backgrounds, or Tokyo-style sign density, and often miss the details that make a block recognizable, such as curb type, lane markings, and storefronts. Names at L1 move broad scene composition slightly closer to the references (GAAS −0.0089 over road, building, vegetation, and sky).

Real Mapillary reference image, Paris
Real reference Paris, FR
SDXL image for a city-only prompt, Paris, FR
SDXL
SD 3.5 Large image for a city-only prompt, Paris, FR
SD 3.5 Large
FLUX.1-dev image for a city-only prompt, Paris, FR
FLUX.1-dev
FLUX.1-schnell image for a city-only prompt, Paris, FR
FLUX.1-schnell
PixArt-Σ image for a city-only prompt, Paris, FR
PixArt-Σ
HunyuanDiT image for a city-only prompt, Paris, FR
HunyuanDiT

How alike do the 25 cities look?

Each city is represented by the mean DINOv2 embedding of its reference images. The mean off-diagonal cosine similarity is 0.284, and visual distance grows only weakly with geographic distance (Spearman ρ = 0.180 over 300 city pairs, p = 1.8 × 10−3).

Cross-city visual similarity

Cosine similarity between city-mean embeddings of real references. Hover a cell for the pair.

Table view
CityAMSBKKBERBOGBUECAICPTDXBISTLONMELMEXBOMNBONYCPARROMSFOSAOSELSHASINSYDTYOYTO
Amsterdam1.000.110.500.040.090.240.270.170.200.400.000.050.170.160.210.240.290.060.030.200.240.150.150.100.18
Bangkok0.111.000.090.310.350.420.320.240.330.070.180.330.510.570.200.070.310.050.360.230.290.600.230.070.28
Berlin0.500.091.000.040.150.220.260.210.270.31−0.020.070.100.080.160.340.37−0.010.000.370.240.110.120.100.20
Bogota0.040.310.041.000.780.190.700.180.240.170.540.800.240.320.210.120.210.440.640.330.230.670.450.130.19
Buenos Aires0.090.350.150.781.000.290.760.200.340.200.540.760.200.240.340.280.290.230.680.350.290.660.480.060.27
Cairo0.240.420.220.190.291.000.400.510.650.160.100.210.490.580.390.250.660.170.210.290.380.380.300.050.46
Cape Town0.270.320.260.700.760.401.000.300.420.290.510.720.280.310.320.290.410.390.580.430.360.700.560.150.28
Dubai0.170.240.210.180.200.510.301.000.280.140.070.180.340.380.100.150.290.100.090.320.250.260.120.020.20
Istanbul0.200.330.270.240.340.650.420.281.000.250.180.260.360.460.350.230.700.250.230.250.330.330.490.110.43
London0.400.070.310.170.200.160.290.140.251.000.110.160.160.210.220.320.230.080.100.340.200.210.240.080.15
Melbourne0.000.18−0.020.540.540.100.510.070.180.111.000.600.080.160.080.050.100.300.680.160.110.500.470.060.11
Mexico City0.050.330.070.800.760.210.720.180.260.160.601.000.280.310.170.150.200.340.710.330.280.670.450.110.13
Mumbai0.170.510.100.240.200.490.280.340.360.160.080.281.000.700.160.100.420.110.320.300.530.530.200.100.17
Nairobi0.160.570.080.320.240.580.310.380.460.210.160.310.701.000.260.060.510.190.310.240.440.540.300.120.41
New York0.210.200.160.210.340.390.320.100.350.220.080.170.160.261.000.290.420.340.280.290.340.250.360.040.67
Paris0.240.070.340.120.280.250.290.150.230.320.050.150.100.060.291.000.340.050.150.260.320.170.180.080.14
Rome0.290.310.370.210.290.660.410.290.700.230.100.200.420.510.420.341.000.260.230.300.460.330.290.150.48
San Francisco0.060.05−0.010.440.230.170.390.100.250.080.300.340.110.190.340.050.261.000.280.170.160.270.400.190.34
Sao Paulo0.030.360.000.640.680.210.580.090.230.100.680.710.320.310.280.150.230.281.000.300.390.730.430.090.18
Seoul0.200.230.370.330.350.290.430.320.250.340.160.330.300.240.290.260.300.170.301.000.500.370.280.360.25
Shanghai0.240.290.240.230.290.380.360.250.330.200.110.280.530.440.340.320.460.160.390.501.000.450.270.240.24
Singapore0.150.600.110.670.660.380.700.260.330.210.500.670.530.540.250.170.330.270.730.370.451.000.450.100.25
Sydney0.150.230.120.450.480.300.560.120.490.240.470.450.200.300.360.180.290.400.430.280.270.451.000.110.35
Tokyo0.100.070.100.130.060.050.150.020.110.080.060.110.100.120.040.080.150.190.090.360.240.100.111.000.05
Toronto0.180.280.200.190.270.460.280.200.430.150.110.130.170.410.670.140.480.340.180.250.240.250.350.051.00

Most alike

    Least alike

      The diagonal is 1 by construction and is left blank. Values are rounded to two decimals, as in the paper figure.

      Data and code

      The dataset (version 3.1.0) holds the reference images, the 16,128 generated images for all six prompt conditions, metadata tables with direct image URLs, the archived scores of both scoring sessions, a code archive, and Croissant metadata. This repository holds the curation pipeline, generation wrappers, metrics, and evaluation scripts.

      Quick start

      Single images can be read through the URLs in the metadata tables. To run the code, download the release into the repository root so that the relative image paths resolve. The archived scores in results/ reproduce the paper’s tables without running any model. Evaluation runs on CUDA by default (DEVICE in config.py), loads DINOv2 through torch.hub, and segments generated images with Mask2Former for GAAS.

      from io import BytesIO
      import pandas as pd
      import requests
      from PIL import Image
      
      base = "https://huggingface.co/datasets/moss-vector-714/GeoFidelity-Bench/resolve/main/"
      refs = pd.read_csv(base + "metadata/reference_images.csv", dtype={"image_id": str})
      row = refs.iloc[0]
      response = requests.get(row.image_url, timeout=60)
      response.raise_for_status()
      image = Image.open(BytesIO(response.content)).convert("RGB")
      
      # metadata/generated_images.csv has the same image_url column,
      # plus model, level, sample_index, and the prompt of every image.
      Code
      MIT License
      Reference images
      Mapillary contributors, CC BY-SA 4.0
      Block metadata
      OpenStreetMap, ODbL 1.0
      Generated images
      CC BY-SA 4.0, subject to model terms

      Limitations and intended use

      • Coverage is uneven

        Reference coverage follows Mapillary contributions and OSM naming density. Cities differ in image counts, headings, capture dates, and devices, and the daylight and scene filters narrow the streetscapes represented. Informal settlements and peri-urban areas are under-represented.

      • Location is given as text

        L2 measures whether text-only generators use coordinates written in the prompt. Models that take coordinates, maps, satellite images, or retrieved photos as input can be scored with the same panels and galleries.

      • Scores depend on the encoders

        The metrics measure visual agreement with reference panels, not the state of a street at a given time. Values depend on DINOv2 and Mask2Former; with SegFormer instead, GAAS keeps 11 of 15 pairwise model orderings.

      • Generated places can mislead

        Generated location images can reinforce place stereotypes or mislead when shown as photographic evidence. The benchmark uses public Mapillary imagery with the platform’s privacy processing and is meant for model comparison and prompt-conditioning studies. It has no established validity for identifying people, surveillance, or reconstructing events.

      BibTeX

      If GeoFidelity-Bench is useful in your work, please cite the paper (arXiv:2606.23669).
      @inproceedings{tan2026geofidelity,
        title={GeoFidelity-Bench: Evaluating Block-Conditioned Geographic Fidelity in Text-to-Image Street-View Generation},
        author={Tan, Kaizhen},
        booktitle={Advances in Neural Information Processing Systems},
        year={2026}
      }