Getting Started with Map Tiling: Mapnik and Shapefiles

About a year ago, I was working on launching a website with a map interface. I wanted full control over everything, including generating my own map tiles instead of using a third-party provider like Google Maps. It ended up being a bad idea, because map tiles take up a lot of resources to generate and store. By going through the process I have a much better understanding of map tiles and I hope this will serve as a step-by-step process showing how to generate tiles from start to finish.

My environment is Ubuntu 8.10. To start with, I installed the following packages from the Terminal:

sudo apt-get install postgresql-8.3-postgis
sudo apt-get install python-mapnik
sudo apt-get install libmapnik-dev
sudo apt-get install imagemagick

The next thing I needed was data and I was able to obtain and play around with a San Francisco sample shapefile from Navteq.

A few important things should be noted about shapefiles. A shapefile is actually multiple files (.sbn, .shx, .shp, .dbf) connected by their filename. A shapefile is like the name describes: a file containing data for a bunch of shapes. For example, it may contain data that says draw a line from one geospatial coordinate (latitude, longitude) to another. Or it may contain data that says draw a polygon with a vertex at different coordinates. It can also contain a combination of all of these.

Each shapefile defines its own map, and for complicated maps, we typically need to overlay multiple shapefiles. One shapefile may define the city boundaries while another shapefile has all the streets. If we wanted both city boundaries and streets, we would overlay one shapefile on top of the other.

Let's look at using Mapnik to interpret these shapefiles. The hard work will be based on generate_tiles.py, which I'll set-up by running the following in the Terminal from my home directory:

mkdir mapnik
cd mapnik
svn export http://svn.openstreetmap.org/applications/rendering/mapnik/generate_tiles.py ./generate_tiles.py

If you're not installing with the same directory structure as me (home_directory/mapnik), there will be some differences later on, but should still work.

This file will not be immediately ready to work with my dataset. First, there are pre-populated test cases starting on line 123:

  1. # World
  2. bbox = (-180.0, -90.0, 180.0, 90.0)
  3.  
  4. render_tiles(bbox, mapfile, tile_dir, 0, 5, "World")
  5.  
  6. minZoom = 10
  7. maxZoom = 16
  8. bbox = (-2, 50.0, 1.0, 52.0)
  9. render_tiles(bbox, mapfile, tile_dir, minZoom, maxZoom)

And so on… The first thing I did was wipe out all the code from 123 on and replaced it with the following:

  1. bbox = (-122.4, 37.76, -122.4, 37.8)
  2. render_tiles(bbox, mapfile, tile_dir, 15, 16, "SF")

Keep in mind the sample data I requested from Navteq was the SF region. If the data you have only corresponds to a certain region, then entering a bounding box (bbox) outside of the coverage will result in a bunch of empty images. You can check the shapefile with software like QGIS to make sure the data matches what you think.

I now need to create an xml mapfile. The mapfile defines the data source and how to visualize the data, which will be used by generate_tiles.py to generate the map. I created mapfile.xml and put the following:

  1. <?xml version="1.0" encoding="utf-8"?>
  2. <!DOCTYPE Map>
  3. <Map bgcolor="#f2eff9" srs="+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs +over">
  4.   <Style name="StreetStyle">
  5.     <Rule>
  6.       <LineSymbolizer>
  7.         <CssParameter name="stroke">#000000</CssParameter>
  8.         <CssParameter name="stroke-width">0.1</CssParameter>
  9.       </LineSymbolizer>
  10.     </Rule>
  11.     <Rule>
  12.       <TextSymbolizer name="ST_NAME" face_name="DejaVu Sans Book" size="9" fill="black" halo_fill= "#DFDBE3" halo_radius="1" wrap_width="20" spacing="5" allow_overlap="false" avoid_edges="false" min_distance="10" placement="line" />
  13.     </Rule>
  14.   </Style>
  15.  
  16.   <Layer name="Streets" srs="+proj=latlong +datum=WGS84">
  17.     <StyleName>StreetStyle</StyleName>
  18.     <Datasource>
  19.       <Parameter name="type">shape</Parameter>
  20.       <Parameter name="file">Streets</Parameter>
  21.     </Datasource>
  22.   </Layer>
  23. </Map>

Finally, I replace lines 109 - 116 in generate_tiles.py:

  1. try:
  2.   mapfile = os.environ['HOME']
  3. except KeyError:
  4.   mapfile = home + "/svn.openstreetmap.org/applications/rendering/mapnik/osm-local.xml"
  5. try:
  6.   tile_dir = os.environ['HOME']
  7. except KeyError:
  8.   tile_dir = home + "/osm/tiles/"

with my own mapfile and tile directory, which was setup before:

  1.   mapfile = home + "/mapnik/mapfile.xml"
  2.   tile_dir = home + "/mapnik/tiles/"

Run ./generate_tiles.py and the map tiles should show up in the tiles sub-directory.

Other than the bounding box, which I discussed before, the only other difference would be the xml file. By doing so, we should be able to generate an xml mapfile for any data source, even one different from Navteq shapefiles.

I would leave the first lines the same and start from the first interesting bit:

  1.   <Style name="StreetStyle">
  2.     <Rule>
  3.       <LineSymbolizer>
  4.         <CssParameter name="stroke">#000000</CssParameter>
  5.         <CssParameter name="stroke-width">0.1</CssParameter>
  6.       </LineSymbolizer>
  7.     </Rule>
  8.     <Rule>
  9.       <TextSymbolizer name="ST_NAME" face_name="DejaVu Sans Book" size="9" fill="black" halo_fill= "#DFDBE3" halo_radius="1" wrap_width="20" spacing="5" allow_overlap="false" avoid_edges="false" min_distance="10" placement="line" />
  10.     </Rule>
  11.   </Style>

A style is applied to a particular shapefile, or other data source. Each style can be composed of different rules. Let's start with the first rule in the above snippet: LineSymbolizer. Remember when I described shapefiles? They can contain lines, polygons, and so on. The Streets shapefile is primarily made up of lines. We must then create a style for these lines in order to see them in the final tiles. Shapes without a style applied will not be seen.

The next rule is the TextSymbolizer. Text for the shapefiles can be found in the .dbf file, which can be viewed with a DBF Viewer. The file is a simple spreadsheet with the text contained mapped to the shapes. The attribute name with the value ST_NAME is the link from the TextSymbolizer to the .dbf file. If we opened Streets.dbf, we will see a column called ST_NAME with each row corresponding to a street name. If you aren't using the same shapefile as I am, then you should be consulting the .dbf file to see what text can be placed into your map.

I glossed over the specific styling rules, because I found them to be quite straight-forward. To consult a complete list and explanation, bookmark Cascadenik.

The final part of my xml file is where we define the data source and set the style for that source:

  1.   <Layer name="Streets" srs="+proj=latlong +datum=WGS84">
  2.     <StyleName>StreetStyle</StyleName>
  3.     <Datasource>
  4.       <Parameter name="type">shape</Parameter>
  5.       <Parameter name="file">Streets</Parameter>
  6.     </Datasource>
  7.   </Layer>

I think the above is straight-forward. I defined a layer, told it I'll be styling it with the style created above, and defined the data source as a shapefile named Streets. The only confusing bit I haven't gone over yet is the srs attribute in the layer. Most shapefiles I've encountered use the WGS84 datum, but if the tiles aren't coming out, this may be the cause. To verify, open the shapefile with a program like QGIS and check the default projection being used.

I'm not familiar enough with the different projections to be able to comfortably write about them in any detail. If the bounding box is correct, and the xml is correct, then the next most likely error is the projection settings being used. You may have noticed this line as well:

+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs +over

The above line is placed in the code to specify the projection of the tiles. In this case, I'm copying the very common method (at least for Internet maps) of using a Mercator projection onto a square distorting the Earth's major and minor axes into a sphere (the Earth is actually an imperfect ellipse) with a radius of 6,378,137. Projecting onto a square simply makes things easier and the distortions are not significant enough to adversely affect things like driving directions, points of interest, and so on. Since the source data can be a different projection, such as WGS84, we must specify both the source projection and the final projection, or mapnik won't know how to render the tiles.

I know my description of projections may not make too much sense, but I'm trying to cover a lot of ground as quickly as possible. It's actually quite an important topic deserving of its own article written by someone whose expertise is greater than mine. The main take-away is that the map being generated is not a perfect representation of the Earth, but it works for most of the typical purposes used in a service like Google Maps. If you need a more accurate picture of the Earth, it would be wise to consult the different projections.

For more reading on projections, I would recommend the following from Charlie Savage's blog:

I hope it's easy to see that from these simple beginnings, we can build more complicated maps. We can add additional layers and styles on top of what we've defined above. Each layer can correspond to a different shapefile. We might have a shapefile for points of interests, or one with consensus data. We can then add more details and information to our maps.

I hope it's also easy to see that creating a custom set of map tiles is no easy task. This is the main reason I abandoned the idea of making a custom set of tiles for my application. If you don't need a custom tileset, then don't go down this road. Beyond the sheer processing power to generate all the tiles needed, you also have to worry about the bandwidth needed to transmit these tiles. There's a reason why sites are using Google Maps and not creating their own tiles. It's a hassle to do the latter. But there are situations where Google Maps falls short. If you need more accurate maps for certain situations, different kinds of detailing on the tile, or are trying to compete directly as a map tile server, then you'll definitely need to look into creating your own tiles.

blog comments powered by Disqus