How to visualize your geospatial zarr data

A lot of the data our users work with isn’t a table. It’s geospatial: temperature, wind or precipitation on a grid, forecast for every hour of the coming days and stored as Zarr. Data engineers happily open that in a notebook. Everyone else just wants to see it on a map.

So we built a Map Viewer into the Databaas portal. Pick a dataset from the catalog, it shows up on the map, press play and watch the forecast roll in. Simple enough, until you add what we actually cared about:

• No tile server or preprocessing
• No data through our backend
• The same permissions as the rest

Why Zarr works in a browser at all

Zarr splits a large n-dimensional array into chunks. Each chunk is its own object, or with sharding, a byte range inside a bigger one. A small JSON file describes the shape, the chunk grid and the compression.

To draw one hour of one variable, you read that metadata, figure out which chunks cover the screen, and fetch only those. It’s the same idea that makes Cloud Optimized GeoTIFFs work. A browser can show a dataset that’s far too big to download, because it never downloads more than what’s on screen.

MapLibre, zarr-layer and zarrita

The rendering itself was the easy part, thanks to some great open source work. We use MapLibre GL for the map and @carbonplan/zarr-layer on top of it: a custom MapLibre layer by CarbonPlan that reads Zarr chunks and renders them on the GPU, colormap and all. Underneath, it uses zarrita, a small TypeScript implementation of Zarr.

zarr-layer does a lot of the heavy lifting. It understands the common projection conventions and multiscale pyramids, and works out which chunks are visible. You hand it a store and a variable:

import { ZarrLayer } from "@carbonplan/zarr-layer";

const layer = new ZarrLayer({
  id: "forecast",
  store, // any zarrita store
  variable: "air_temperature",
  zarrVersion: 3,
  colormap: ["#2c7bb6", "#abd9e9", "#ffffbf", "#fdae61", "#d7191c"],
  clim: [260, 310],
  selector: { time: { selected: 0, type: "index" } },
});

map.addLayer(layer);

That store is where it gets interesting. In the examples it’s a public URL. Our data lives in private buckets.

Signing every request

The browser needs to read from S3 directly. We’d already solved that for notebooks and pipelines, and Nathan wrote about it in Catalog authorised S3 access, without STS: the client builds an S3 request, Lakekeeper (our catalog) checks your grants and signs it, and the client sends it to object storage itself. The credentials never leave the catalog.

In the browser that looks like this:

1. The browser asks the portal to sign a list of object paths for one dataset.
2. The portal asks Lakekeeper to sign each GET with your own token. Lakekeeper checks your grants on that dataset.
3. The browser sends the signed requests straight to S3.

The portal handles a bit of JSON, object storage handles the megabytes.

On the browser side, all zarrita needs is an object with a get and a getRange:

import type * as zarr from "zarrita";

class SignedStore implements zarr.AsyncReadable {
  constructor(
    private signer: Signer,
    private root: string,
  ) {}

  async get(key: zarr.AbsolutePath) {
    const { url, headers } = await this.signer.sign(this.root + key);
    return read(await fetch(url, { headers, credentials: "omit" }));
  }

  async getRange(key: zarr.AbsolutePath, range: zarr.RangeQuery) {
    const { url, headers } = await this.signer.sign(this.root + key);
    const bytes =
      "suffixLength" in range
        ? `bytes=-${range.suffixLength}`
        : `bytes=${range.offset}-${range.offset + range.length - 1}`;
    return read(
      await fetch(url, {
        headers: { ...headers, Range: bytes },
        credentials: "omit",
      }),
    );
  }
}

async function read(res: Response) {
  // A missing chunk isn't an error in Zarr, it means "fill value".
  if (res.status === 404) return undefined;
  if (!res.ok) throw new Error(`Reading from S3 failed (${res.status})`);
  return new Uint8Array(await res.arrayBuffer());
}

Pass that to new ZarrLayer({ store }) and the layer has no idea its data sits behind a catalog.

Keeping it fast

Signing every request sounds slow, and done naively it is: each chunk would wait for a trip to the portal before it can even go to S3. A few things keep that out of the way.

Signing happens in bulk. When the map asks for a screen full of chunks, the browser collects them and gets them all signed in one go. A signature also stays valid for a while and covers any part of a file, so most reads don’t need a new one at all.

Chunks are downloaded once. The map, the playback and the chart you get when you click a location all read the same data, and they share it instead of each fetching their own copy.

And while you watch a forecast, the next hours are already loading in the background. Playback waits until a frame has actually rendered before moving on, so you see every step, even on a slow connection. Forecasts still being written fill themselves in as new data lands.

What about Icechunk?

Everything above leans on Lakekeeper. It’s our catalog for both tables and arrays: it knows which datasets exist, decides who can read them and signs every request. For Zarr, there’s an interesting alternative: Icechunk, by Earthmover.

Icechunk is a storage engine for Zarr, inspired by Apache Iceberg.

For weather data, that brings some nice things:

• Transactions. Updates are fully isolated. A pipeline commits a whole forecast run at once, even when it spans several arrays.
• Version control for data. Every commit is a snapshot you can go back to.
• References to existing files. Icechunk can point to chunks inside HDF5, NetCDF or GRIB files you already have, so an archive of forecasts becomes one dataset without converting or copying it.
• Built for gridded data. Iceberg, and Lakekeeper with it, thinks in tables of rows and columns. Icechunk thinks in arrays with dimensions like time, height, latitude and longitude.

What it doesn’t do is decide who gets to see what. That’s the part Lakekeeper does for us today, and it’s also what lets us put arrays and tables under the same permissions. So it’s less a choice between the two and more a question of where each one fits best. We’re still figuring that out, but a forecast you can rewind and branch would make a great addition.

Where this leaves us

With a chunked format, range requests and a GPU layer, the browser can show large gridded data without a tile server in between. And because every request is signed by the catalog, access is decided in the same place for the map as for notebooks and pipelines.

If you want to play with this yourself, CarbonPlan’s zarr-layer repository has good examples, and any public Zarr store on S3 will do to start.

Working with gridded weather or climate data and want it on a map without building all the plumbing? Feel free to send us a message!