If you have very dense data, sometimes you may notice that some of the features "disappear" from your map at high zoom levels - that is, at country or continent scale.
What is happening at these high zoom levels is our feature shedding algorithm. Essentially, where a single map tile (an image of 256 x 256 pixels) contains more than about 30,000 features, we drop them from the rendering engine.
There are two reasons why shed features:
- feature density and draw order; and
Density and draw order
First, the features are too dense to distinguish as individual points and the z-order or draw order of the features will render the visualisation unusable in most cases.
Layers in Mango are made of a series of tilesets - layers of seamless 256px wide and tall images that from a seamless draggable, zoomable map. Each Group Layer is a separate tileset, and each zoom level is a new set of images within the tileset.
A 256px square image tile has 65,536 pixels. In a world made of pixels, it would be possible to render a 1px point in every pixel, creating a seamless and perfect visualisation. Of course, the world and the data we collect are rarely perfect.
Real data is imperfect, even if it is exactly where it's meant be.
In reality, the locations of points and vertices in data are rarely aligned to a perfect grid, and if by chance they are perfectly aligned to a grid, they are they're unlikely to be aligned to the global grid used in Mango to divide the world into trillions of small square pictures.
It's inevitable at some zoom level that dense points, lines, or polygon features become stacked on top of one another, with each feature obscuring that below it. Ultimately this results in a distortion of the visualisation.
A good example is how much of the red features are hidden at those higher zoom levels. At some point, there is no purpose to loading those obscured features and send them to the web browser, as they provide little to no value to the story the data is telling. This leads to the second reason, which is efficiency.
When zoomed in tightly on a small village, there may be only 6 points in a 256px tile. When this tile is generated, our server performs 6 calculations of where to place the symbol during the tile rendering process. Now, imagine that each calculation takes 1ms, so it takes 6ms to calculate the marker locations and draw the image. Mango, of course, is able to calculate each marker location much, much faster than 1ms per feature, but that’s a nice easy number for the purpose of this example.
Now, zoom out to the district view. At this level, each tile might have 5,000 features. That’s now 5 seconds to calculate and generate each tile.
On screen there’s at least 16 tiles visible on your screen. We're now at 80 seconds.
Now, zoom out to the country level and you have 65,000 features. With a full screen of 16 tiles, that view would take 17 minutes.
And that’s assuming you have just one layer in the layer group. Add more layers, and each needs to be processed into a single tileset ready to display at any zoom level.
We have intelligent caching and a global CDN to reduce redundant tile generation, but without feature shedding, your maps might take several weeks to preseed a tileset and be available to view.
So that’s the technical and practical reasons why, but what are we going to do about these visualisations at higher zooms?
Aside from the aesthetics of feature shedding, there's still valuable insights to be gained from these way out overviews. The simple answer is, points just aren't the right way to visualise data at this scale.
Instead, we need to generalise the data and transform the data into new datasets that tell the story at that scale.
So, what are the alternatives?
Glad you asked!
Here's a few alternative visualisations that not only give us a better visualisation, but can also help us give new context to the data to deliver more powerful insights.
If density of points is the key takeaway, then heatmaps are the go-to viz. Heatmaps show relative density of features via a visually compelling colour ramp to indicate the transition from sparseness (cool colours) or density (warm to hot colours) of the features.
As the visualisation shows relative, the visualisation changes throughout the scale range.
Heat maps are useful all the way through the scale range, and can help to enhance point data layers as we reach scales where individual points are not obscured by density.
"Point in Poly" Analysis
Points allow us to see the distribution of features, but over large geographies high density point data becomes meaningless. Case in point is this map of every college and university in the US. At country scale, it's clear there are some clusters of high density along the west and north east coasts. However, we don't really have a way to get an accurate gauge on the real numbers.
Using a point in polygon visualisation, we can immediately see the same data represented in a much more insightful way. The colour ramp of the choropleth leave no room for misunderstanding which counties have more or less schools. From looking at the class breaks in the legend, I can also very quickly establish
Point in Poly:
Combined with labels that trigger at the deeper zoom levels, we now get a richer experience when exploring down at the county level compared to viewing points alone.
Generating this analysis means an extra step of preparing a polygon dataset that covers the geography of our points with boundaries that suit our particular need, and then using a GIS to run a count of the number of our points in each area.
Point in poly is especially useful for extremely high density point data that simply provides no useful visualisation at the full extents view of the data. If you'd like to try a point in poly analysis, please take a look our our in depth tutorial.
Hexbinning is similar to point in poly in that we're counting the number of points within a given area. The key difference between a regular point in poly and a hexbin viz is how we dice up the geography - in this case, we're using hexagons of a specific area, rather than administrative boundaries.
For example, using a 10 mile hexagon grid, we could quickly analyse how man points lie in each hexagon, and then produce a dataset to import into Mango to visualise hot-spots of urban density at country scales.
As with all visualisations, the correct choice is that which communicates the data best at the appropriate scale.
Changing the visualisation over the scale range
While Mango allows you to create powerful visualisations, and perhaps the most powerful feature is the ability to move between these visualisations throughout the scale range - visualisations smoothly adapt and tell stories all the way down from country to village level.
Explicitly defining what each scale range should look like in it's own layer takes a little longer to configure initially, but the value of a well executed map application can continue to deliver over time. As the data updates, so too the visualisations - automatically!