OpenStreetMap Data Pipeline for Canvas Maps | EveryCityMap

This article explains the OpenStreetMap data pipeline behind EveryCityMap: how I collect a complete city with Overpass, rebuild its boundary, fill land and sea, split the result into PBF blocks, cache those blocks in IndexedDB, and render the final city with Canvas2D.

My earlier EveryCityMap engineering post described five product-level tradeoffs, but compressed this pipeline into one sentence: city data is prepared ahead of time, then delivered through a CDN and the browser cache.

This article expands that sentence. EveryCityMap does not send one large OpenStreetMap query when a visitor opens the editor. It first obtains a complete offline source for a city, determines the city boundary, fills the land, and builds one canonical CityMap. Only then does it derive delivery blocks, encode them as PBF, merge them in the browser, and hand the result to a Canvas2D renderer.

As of September 1, 2026, the stable v3 catalog contains 535 cities. Each city has three browser delivery blocks and one private archive block, for a total of 2,140 PBF files. This is not a general online map service. It is a data pipeline designed to turn a complete city into a visual work inside EveryCityMap.

Reviewed OSM administrative relations
  → one complete Overpass city collection
  → city component selection + 3 km natural buffer
  → land mask + canonical CityMap
  → Preview / Core / Detail delta / Private Archive
  → manifest + signed asset plan + IndexedDB
  → PBF decoding and on-demand merging
  → Canvas2D preview and export

The first step is defining “the city,” not querying roads

A city in OpenStreetMap is not naturally a rectangle on a screen. An administrative relation may contain enclaves, remote islands, or several disconnected outer rings. Some product cities must also be assembled from multiple relations. Shanghai uses a reviewed combination of 16 districts. Tokyo keeps the 23 special wards instead of pulling the entire metropolis, the Izu Islands, and distant islands into one composition.

The catalog therefore stores explicit relation IDs, a center point, and a product definition for every city. The pipeline rebuilds the outer and inner rings of those relations, then uses reviewed anchors to select the connected components that belong in the product. The current 535 cities map to 559 relations. Relation labels and combinations go through a lightweight preflight; the script never accepts the first same-named search result as the city definition.

The administrative boundary is not used as a hard visual cut. EveryCityMap adds a 3 km natural buffer around the retained city components so roads, water, and land can continue slightly beyond a political border. Open roads and railways are clipped only when they leave that buffer. The final frame comes from the retained city geometry and is constrained to the reviewed product area. A distant motorway cannot pull the whole composition outward, and the map does not end in a perfectly straight scar beside the city boundary.

Collect one canonical source for each city

The pipeline issues one complete administrative-area request per city. That request collects roads, railways, water, green space, airports, ports, terrain, and boundary topology together. It does not query a Standard version and a Detailed version separately. Two endpoints or two collection times may return different OpenStreetMap snapshots, making it impossible to prove that the density packs still describe the same city.

Every response records its query hash, file hash, source endpoint, and OSM timestamp. Truncated JSON, an Overpass remark, missing relation members, or an incomplete response causes the city to fail. The pipeline can retry against another endpoint, but it never joins several partial responses into a source that only appears complete. Only one source that passes all checks reaches the build stage.

The canonical builder then runs once:

1. Parse the reviewed relation or relation combination.
2. Remove incorrect or remote boundary components.
3. Limit roads, railways, water, land cover, and terrain to the product mask.
4. Build a natural frame from the retained road geometry.
5. Clip oversized water or land-cover relations that would dominate the composition.
6. Add a land mask that uses the exact same frame.
7. Assemble one canonical CityMap with semantic layers.

The land mask has one specific job: filling land and sea correctly. It comes from a pinned release of global OSM land polygons and covers the frame with an additional margin outside the artwork. It does not decide which geometry belongs to the city. If Shanghai’s CityMap and land mask use different frames, the build fails instead of hoping that the coastlines will line up in the browser.

Why split a city into four blocks instead of uploading one large file?

The canonical CityMap is useful for building and validation, but it is too expensive to make every visitor download in full. Version 3 deterministically derives four assets from the same canonical data:

Detailed is not another complete map. It is Core + Detail delta, so the browser does not download the same land, rivers, and major roads twice. The private Archive is not loaded by the current editor. It keeps roads needed for future close-range map builds without making ordinary visitors download geometry that the product does not show yet.

Splitting is not an array operation driven by file size. Semantic layers first decide whether a feature belongs to Standard or Detail. Road classes are then assigned to Core, Detail, or Archive. For exceptionally large geometries, Core has a 700,000-vertex budget. Motorway, trunk, primary, secondary, and tertiary roads retain priority as the city skeleton; ordinary roads can move deterministically into Detail. Preview is derived from Core with only the major roads, limited land cover, and stronger simplification.

Across the current stable catalog, median gzip sizes per city are approximately 53 KB for Preview, 1.15 MB for Core, 416 KB for Detail delta, and 283 KB for Private Archive. The three delivery blocks total about 1.36 GiB across all cities, while the private archive is about 407 MiB. Compared with sending every complete city on first load, this design removes detail the visitor did not request instead of quietly removing entire parts of the city.

How the browser loads progressively without mixing releases

Studio never constructs object-storage paths directly. It first asks the application for an asset plan containing the release ID, city, block type, SHA-256, and short-lived asset URL. Standard needs only Core. Detailed begins Core and Detail requests together, but it can show the city as soon as Core arrives and merge Detail into a later redraw.

If the Detail request fails, the editor keeps the usable Core map and marks the result as partial instead of replacing the entire city with a blank screen. Before decoding, every block passes three checks:

1. The downloaded SHA-256 must match the manifest.
2. The PBF must decode as the expected Preview, Core, or Detail delta block.
3. Boundary relations, product definition, buffer distance, and data revision must match the city.

Validated raw PBF data is stored in IndexedDB. Its cache key contains the schema version, release ID, city ID, block type, and SHA-256, so a new release cannot accidentally reuse an old block. The cache currently retains at most 48 blocks and evicts older entries when space is needed. Opening the same city again still revalidates the cached hash and identity; a damaged block is removed and fetched again.

Why use Canvas2D after PBF decoding?

PBF is only a transport format. The browser decodes it into the shared CityMap structure; EveryCityMap does not switch to MapLibre or a vector-tile renderer at this point. The product is composing one complete city. Themes, layer controls, previews, share images, and high-resolution exports all need to see the same semantic geometry.

The renderer uses a fixed order:

ocean and land polygons
  → rivers, coastlines, railways, and other lines
  → roads sorted by class
  → required points

A theme controls the color, width, material, and visibility of each geometry type without changing the city data. Roads are grouped into Path2D caches for motorway, primary, secondary, tertiary, and ordinary classes. A theme change can reuse those paths and redraw only the visual properties. The same drawCityMap() function serves both the editor preview and WebP export, so a layer order chosen in Studio is not replaced by a different implementation during download.

Atlas and Watercolor add ocean, paper, and pigment effects on top of that base order while honoring the same CityMap contract. Preview rendering can fall back to a simpler surface when a material effect fails. A production export instead reports the failure explicitly, avoiding a paid file that silently differs from the screen.

“Filling gaps” does not mean mixing old and new data

The easiest way for a city-data pipeline to lose control is to discover a missing layer and patch it from another query. EveryCityMap does not do that. All delivery packs for one city must come from the same canonical source. When a pack is missing or outdated, the pipeline either rebuilds from a raw checkpoint whose hash matches exactly or collects the complete city again.

Offline runs save each city’s source, stage, and validation result. A verified response does not need another Overpass request after an interrupted run. If boundary selection, density rules, or the PBF revision changes, the pipeline can rebuild from the retained combined.json without touching the live channel pointer. Failed cities are retried individually and successful cities remain valid checkpoints, but a candidate release is not eligible for activation until catalog coverage, revision, all four asset identities, and object-storage read-back have passed.

This is why “cities collected” and “cities live” are different numbers. A later expansion currently targets 959 cities: 949 have been collected and 10 still fail. That is an incomplete local candidate, not the production catalog. Stable v3 remains the 535-city release described above. Only after the failed cities are completed, the cumulative candidate is validated, and the stable channel pointer is updated will the application treat those cities as published content.

Separating an immutable release from its channel pointer also keeps rollback simple. The browser requests one exact release. Activation atomically moves the stable channel instead of overwriting the previous files. An earlier release remains available as a rollback source, and a candidate manifest does not take over production merely because its objects were uploaded.

What this architecture buys

The cost is explicit. Cities must be collected ahead of time, which requires offline or CI compute, object storage, and a strict release process. OpenStreetMap changes do not appear instantly. The system is not suitable for a general map that zooms continuously to street level, and it is not trying to replace a vector-tile service.

In return, the experience is predictable. Visitors do not wait on Overpass latency or rate limits. Preview, Standard, and Detailed represent genuinely different download costs. Changing a theme does not fetch the city again, and preview and export continue to use the same geometric facts.

EveryCityMap uses data from OpenStreetMap contributors under the ODbL 1.0 attribution requirements. For me, the pipeline ultimately solves something more specific than “how to fetch more map data”: it gives every city a traceable source, a verifiable split, and a frame that can appear reliably in the browser.

You can switch between Standard and Detailed in EveryCityMap Studio to see the same city move from Core to Core + Detail delta.

If you are interested in a smaller deterministic rendering system, I also wrote about building a scannable 3D QR code generator with WebGPU.