GIS programmer writing Python in Emacs

EuroPython 2011 report

2011-07-22T12:00:00Z

Are you a Python programmer? Do you want to learn Python? Do you want to become a better programmer? Have you ever been to Florence?

If you've answered yes to at least one of these question and yet didn't show up at Florence in late June for EuroPython 2011 conference, then you have no excuse. And no, living in Australia or North America doesn't qualify as a valid reason, there were people from these continents.

Whew, that was a longish attempt at making a joke, now let's get to the real deal, that is, report. I must also note that I didn't stay for the whole length of EuroPython. Hey, it was my honeymoon! And that's why I didn't have time to write the report sooner.

Talks

Of course, talks are important. They are the reason people come to events like this. But it's always hard to choose which one suits you better, when you have 5 talks running in parallel. Sometimes you're sure to run into conflicts. And it's generally a good thing for conferences, having real concurenty between speakers. Watching all of the talks might be possible if you're diagnosed with insomnia or otherwise have tons of free time and I can't really recommend any talks, as I've been at the conference for only two days, but I've heard very good things about Armin Ronacher's Five Years of Bad Ideas talk. Fortunately, all of the talks are already uploaded to YouTube, so don't hesitate to watch those whose description you find interesting.

My talk

Ah, yes, forgot to mention -- I was one of the speakers too. Well, if you think about using OpenStreetMap data or just looking into using Python for GIS, my talk might give some hints as to where to start. Check out the slides here or perhaps check out talks' page at EuroPython site.

People

People in general are great. Pythonista are no exception from this rule. It's always fun to just talk to people around you, ask what mad things they are doing with Python, bash PyPy (ha-ha, joking) and Django (only half-joking) and just hang out talking about things not related to programming (not joking). Socializing with other geeks is fun!

Organization

Organization was actually pretty good. If you don't count WiFi issues in, it was fantastic. Just to make sure, I've made a list of good and bad things.

The good

Distinguishibility of organisers from conferences visiters. Seriously, using yellow t-shirts as a uniform was a fantastic idea
Names on both sides of badges. YES. BOTH SIDES. That was a minor gripe with PyCon this year -- after wearing the badge for 3 days, the name was almost invisible and furthermore, just like a sandwich that falls always butter down, the badge always turned the blank side up.
Equipment. It worked. 'nuff said.
Food. Now, this is something I like about Southern Europe in general -- good food. It's not that easy to cook something delicious for a crowd of 600 people, but cooks at Florence actually did a very good job.
Wine at lunch. I love wine. I L O V E it! And having a glass of wine relaxes you enough to not care about internet issues anymore.
Place. Bus stop right in front of the hotel? Check. Close to city center? Check.

The bad

WiFi and Internet connectivity in general. Everybody would definitely tweet about if they could connect to the internet. Well, at least more people paid attention to talks. :)
Partner programs, PyFiorentina and marketing. Constant reminders about those on EuroPython official twitter were kinda irritating.

In general -- organization was kick-ass and though I highlighted some issues I think organizers team did a great job. Go Italy!

Summary

Instead of a summary, I'm just going to say that I'm certain I'll visit EuroPython next year and so should you!

PyCon tutorial slides are up!

2011-04-14T14:25:00Z

I've finally managed to get the slides on my site, so here you go! On an almost unrelated note, here's a tiny cute one-liner that I've used to create a zip of all the code used in slides:

grep "\\lstinputlisting" slides.tex | \
sed 's/.*{\(.*\)}.*/\1/' | \
zip -@ pycon2011-code.zip

PyCon wrap up

2011-03-25T10:53:00Z

It's now almost two weeks since PyCon has come to an end and so it seems as a good time to write a short report about the conference.

I must say that this was my first time at PyCon and actually my first time at a conference of such huge size and thus I was a bit overwhelmed. Nevertheless, I was able to serve as tutorial instructor, bag swagger, session runner and beer drinking in the 7 days I spent at the venue. And of these positions, the hardest one was being a tutorial instructor. Not only I felt pressured by the fact that people actually paid to get to my tutorial and this was my first tutorial given in English, but I also spent more than two months adding, updating and removing material. In the end the tutorial went OK, but I think I hurried a little bit and the fact that tutorial wasn't meant to be one of "code as I talk" types seemed as a disappointment to some of the listeners. I'm still waiting for some feedback from PyCon organizers about the overall rating of my tutorial, but I'm happy that at least nobody walked out of the room during my tutorial.

Main conference featured tons of interesting talks, but of course I couldn't possibly make all of them, apparently because I don't scale to 5 rooms in a nice reproducible fashion. I'm still in process of watching the talks at http://pycon.blip.tv and I suggest anyone interested in Python doing exactly this -- watching all talks, one by one. Some of them might generate more interest, though. For example, I remember how crowded it was at Alex Martelli's talk about API design where people had to stand, because there wasn't a single empty chair in the room. And the same happened at both Raymond Hettinger's talks, where people where starting to reserve places 30 minutes prior to the talk. Some talks were not meant to be educational, or rather, emphasized the fun at the expense of content. I do like talks like these, because getting absolutely massive amount of information whole day non-stop would likely cause brain damage. So, don't forget to watch "Obfuscated Python" by Rev. Johnny Healey and "Exhibition of Atrocity" by Mike Pirnat when you get bored (and I hope you don't).

I also had some time to take part in sprints. I wouldn't say I had much choice in whom to join, as I mostly use Twisted at my current project. Well, I must say that Twisted people aren't that twisted after all and I had quite a bit of fun trying to fix and reproduce older bugs from their bugtracker. I admit that their way of doing code review leads to much higher code quality at the expense of longer ticket processing time. Still, I was able to provide fixes for two tickets and get to first twenty at Twisted Highscore which I find quite cool.

And, as a short summary -- PyCon 2011 and its attendees were awesome and I really want to make it to Santa Clara next year.

My OpenStreetMap tutorial at PyCon

2011-02-17T23:51:00Z

This year marks a huge milestone for me -- I'll be giving a tutorial session at PyCon. Being selected as one of the speakers is not only a great honor but also a huge responsibility. Whatever, that's not the point of this post. This is supposed to be a self-advertisement. So, if you don't know what's OpenStreetMap, how to render a map in Python and what GIS stands for (or any of the above) -- don't hesitate to spend $150 on registration. For those of you who think that they can handle this with ease, I suggest a much more sophisticated alternative. I don't recommend this workshop to people not experienced in Python, though. There're plenty of other tutorials so don't hesitate to share this information with your friends, colleagues and random people on the Internet.

Oh, and don't forget to register for the main conference, because it's probably the only way to find out about giant telescopes, huge robots or automation of airplane engines.

On almost unrelated not, if you have registered for my tutorial and want to try out all examples while in the session room I'd suggest you go through this check-list:

Install Mapnik. See the installation instructions. There're pre-built packages available for most major platforms, so the install process should be straightforward.
Install shapely. Play with examples, try to recall your geometry classes from school. This will help you feel more comfortable with examples I'm going to provide during the session.
Install PostgreSQL with PostGIS extension. Some of the examples might include writing SQL queries, so if you're not familiar with SQL, you should probably read some basic tutorial, but that's definitely not obligatory.

Of course, you could just skip all of these and try out the code samples later after the session, in a calm and quiet setting. The tutorial itself doesn't force students into constantly writing code.

In conclusion, don't miss out on your opportunity to extend your knowledge base and social network, visit PyCon 2011 in Atlanta, it's going to be cool.

Tile Server Implementation

2011-01-16T17:40:00Z

It's been almost 2 years as CloudMade has ditched mod_tile and renderd as main rendering solution in favour of in-house solution. As the principle designer of the said alternative, I must say that this decision led to higher development pace. This article will try to cover the general architecture approach, reasons of decisions made and short comparison to other rendering alternatives.

Before The Switch

As some of you might know, CloudMade has its roots in OpenStreetMap and it was quite natural to adopt OSM's software stack to have something to start with. But as CloudMade grew, the needs and requirements changed rapidly and the task of supporting and developing mod_tile became more of a burden, the decision to switch to more high-level language as the main was made. The language of choice was Python, due to its generous set of already existing spatial libraries (e.g. Shapely, GeoAlchemy, Mapnik bindings, etc), ease of deployment and its simpler support for cross-platform development. And, well, I knew it better than Scala, Ruby or Perl at that moment. Here goes a list of our tasks with mod_tile and renderd that we found easier to implement with Python:

Variable priorities: mod_tile has the notion of "dirty" and "general" requests, with dirty requests having lower priority and thus having the property of being rendered when there's little-to-none on-demand rendering required. While this seems enough for most applications, it does has its warts, as it makes the priority system overall less general. What this means in practice, is that every time we need to add some special priority (i.e. in case we need to health-check system by forcing rendering) we get into adding quite a lot of code, rather than changing the "priority" property of the request. It might seem silly, but off the top of my head I can remember that we have at least 6 different priorities now
Replicating cache: When it comes to scaling rendering and serving of tiles, the simplest solution that comes to mind is adding more servers. It's as simple, as pushing several links in web interface or even using automated process and Amazon Web Services API. But when you add new server with rendering stack installed you lose all the cache that has been on other servers and furthermore all the instances don't share cache, which makes the cacheto use system less effective. There're several solutions to this issue, each of them making use networking or database libraries, programming against which is tedious task in C (and C++).
Being tied to Apache: mod_tile is an Apache module, which makes it less interesting if you look at it from "commodity server" perspective. Having to program against a monster that is Apache, using its APR library is one giant leap into full-blown programmer depression. The autogenerated documentation make the matters even worse. And two last things about Apache are its comparatively slow serving of static files and complicated configuration scheme. One might say that Apache might be winning in other parts of comparison, but the things that have been mentioned were essential to our rendering services.

These were the main reasons to switch, as mod_tile and renderd didn't seem like the right thing for CloudMade. Of course, there were a lot of others, more and less subjective reasons, but having even before mentioned ones, it was enough to seriously consider a switch.

The Switch

With all the warts of the existing system and requirements for the future in mind, we decided to move on with the new approach. There were several things to consider in our system:

Decoupling: This was our main goal -- thoroughly decoupled system, where every part does one thing and does it good. This makes scaling much easier, but also incurs additional penalty on the amount of code, because of the need to write communication utilities. This also makes the system as a whole seem much more stable, as every other part of the system can work as a replacement in case of failure. Of course, the price is having network overhead and supervising system parts.
Handling styles: One of the main CloudMade web-services is the style editor, which gives ability to edit map styles using WYSIWYG technique. Handling thousands of Mapnik styles wasn't something any existing system was prepared for, so unique way of doing exactly this had to be devised. Of course, this meant that style state in every part of the system had to be consistent at any given moment of time, making this even harder to accomplish.
Cache expiry: To minimize load on the system, as much cache as possible has to be available. But for rapidly changing OpenStreetMap data, having all tiles cached for month wouldn't work and at the same time rendering all images on the fly would be an enormously heavy goal to accomplish. Whatever cache update approach is taken, unless there's a hardware possibility to render maps on the fly, someone will be unhappy about cache expiry scheme.
Health monitoring and high availability: In order to meet requirement of having usable web services, one of the most important things to consider is having as high service uptime as possible. Without having health monitoring which knows about state of every part of the system the said objective is almost unreachable. Of course, the ideal can not achieved, but having a setup that covers at least 80% of the nodes would satisfy our needs.

The system that's currently in use at CloudMade has been developed with exactly these goals in mind, with minor additions and subtractions along the way. To summarize, the goal was the system where every part has a maximum level of independency from every other while succumbing to the general goal of having fast and easily-deployed rendering stack.

To Be Continued

I'll continue the talk about moving from mod_tile to our in-house system in follow-ups, where I'll try to get into technical details, explain our shortcomings and issues that arised while developing.

Stay tuned.

Priority Queue for Twisted

2010-09-02T09:30:00Z

In the past three months I've in a process of moving good old piece of shitty, but working code I wrote from using threading to twisted. If you don't count that one nasty deferToThread call which wraps blocking rendering code (mapnik.render that is) I've been quite successful. But there are several things that I had to rewrite completely in order to make them non-blocking. One of those is PriorityQueue. You know, this one. Now, Twisted does have an implementation of Queue (which is of course called DeferredQueue, as if anything in Twisted is not deferred, but anyway) and it quite successfully mimics API of the standard one, but still no PriorityQueue. Let's take a look at how DeferredQueue is implemented:

class DeferredQueue(object):

    def __init__(self, size=None, backlog=None):
        self.waiting = []
        self.pending = []
        self.size = size
        self.backlog = backlog

    def _cancelGet(self, d):
        self.waiting.remove(d)

    def put(self, obj):
        if self.waiting:
            self.waiting.pop(0).callback(obj)
        elif self.size is None or len(self.pending) < self.size:
            self.pending.append(obj)
        else:
            raise QueueOverflow()

    def get(self):
        if self.pending:
            return succeed(self.pending.pop(0))
        elif self.backlog is None or len(self.waiting) < self.backlog:
            d = Deferred(canceller=self._cancelGet)
            self.waiting.append(d)
            return d
        else:
            raise QueueUnderflow()

Pretty simple, eh? Well, it's simple for those of you familiar with twisted. If you're not one of them, considering reading excellent "Twisted in 60 seconds" tutorials from JP Calderone.

Due to the nature of the software I don't need limits on the size of the queue as well I don't care how many deferreds are in waiting list, so I can safely ignore these in my implementation:

class DeferredPriorityQueue(DeferredQueue):

    def __init__(self):
        DeferredQueue.__init__(self, None, None)

    def put(self, obj):
        if self.waiting:
            self.waiting.pop(0).callback(obj)
        else
            self.pending.append(obj)

    def get(self):
        if self.pending:
            return succeed(self.pending.pop(0))
        else
            d = Deferred(canceller=self._cancelGet)
            self.waiting.append(d)
            return d

The code becomes quite short. Now, to make items prioritized, we'll use wonderful heapq module:

class DeferredPriorityQueue(DeferredQueue):

    def __init__(self):
        DeferredQueue.__init__(self, None, None)

    def put(self, obj):
        if self.waiting:
            self.waiting.pop(0).callback(obj)
        else
            heapq.heappush(self.pending, obj)

    def get(self):
        if self.pending:
            return succeed(heapq.heappop(self.pending))
        else
            d = Deferred(canceller=self._cancelGet)
            self.waiting.append(d)
            return d

Not that hard, isn't it? Now, the latest things, adding qsize() methods to mimic original stdlib Queue API and peek() method, which returns the highest priority item, without removing it from the queue (as Wikipedia page on the subject suggests):

from twisted.internet.defer import Deferred, DeferredQueue, succeed
import heapq

class DeferredPriorityQueue(DeferredQueue):

    def __init__(self):
        DeferredQueue.__init__(self, None, None)

    def put(self, obj):
        if self.waiting:
            self.waiting.pop(0).callback(obj)
        else
            heapq.heappush(self.pending, obj)

    def get(self):
        if self.pending:
            return succeed(heapq.heappop(self.pending))
        else
            d = Deferred(canceller=self._cancelGet)
            self.waiting.append(d)
            return d

    def peek(self):
        return succeed(heap[0])

    def qsize(self):
        return succeed(len(self.pending))

If you're wondering why I'm posting this, considering how simple it is to implement on your own, here's the answer -- yesterday I've implemented DeferredPriorityQueue for the third time, simply because google couldn't find anything relevant when queried with "priorityqueue twisted" and I forgot that I wrote this code twice before.

P.S. I'm not a fan of comment-less code, but the code is short and simple and should be understood without any comment.

SOTM 2010: looking back

2010-07-16T12:00:00Z

Was the State Of The Map fantastic or what?!

With all the incredibly smart people lurking around, having sangria and paella, talking about things I might never understand -- this must the Mecca for map geeks. Or, at least it looked like it this year.

Anyway, here's a more structured way of reporting.

Organization

Everything starting from drinks and food and finishing with timing and Internet worked as it should have. I think that's the highest praise any even organizer can get. Of course, there were some slight issues, such as lack of sockets in small room, but that only somehow decreased twitter activity.

Day 1

Day 1 was marked by several big announcements:

AOL's Map Quest will now use OpenStreetMap data. While they're still in beta, the plan is to use OSM for Germany and UK sites. The nice thing about MapQuest is that they sponsor development of Mapnik and Nominatim. [1]
Microsoft is going to add OpenStreetMap layer to Bing Maps. We'll have to wait until that happens, of course, but this is a good thing overall, considering the level of their cartography. I mean, just look at the "Sketchy" layer, you wont be able to resists its beauty.
Curly Brackets announced their own vector rendering engine for iOS 4. Some might say that this is not as big as the previous two announcements, but I think this marks the start of moving away from server-side to client-side rendering for the sake of delivering better maps experience.

I must admit that I've missed half of the talks on Friday because I was involved in Mapnik Birds of Feather discussion. I'm not going to summarize it, because that was already done beautifully by Richard Weait on Mapnik-devel list.

For those of you who missed SOTM 2010 I advise watching these talks (organizers say that the videos will be available in the next two-three weeks):

"Who are the mappers and why do they map in OpenStreetMap?" by Nama Raj Budhathoki, Muki Haklay and Zorica Nedovic-Budic

In fact, this wasn't a real talk, just a summary of survey done in the past 3 or so months. The survey itself might be of low value, simply because of low numbers of people polled. But the analysis was pretty impressive and showed that most of the OSM mappers are in fact professional GIS specialists and/or cartographers.
"Uncommon Operating Picture: Using OSM for Humanitarian Response" by John Crowley

I missed this one, but I'd heard only good things about it. I think that the more OpenStreetMap is used for the greater good (that means, not only target well-developed countries) the better it is for community and the map data itself.
"Map Kibera – Mapping one of Africa’s largest slums" by Mikel Maron

Mikel showed everybody that we should move towards providing map data not only for the first-world countries. A very detailed and touching talk overall. Also, check out what has been done already at Map Kibera official site.
Panel discussion "Cocktails on the Titanic?" by Steven Feldman.

Big people from Google, Microsoft, MapQuest and CloudMade tried to explain what business model will make OSM-based start-ups take off successfully. I must admit that Steven Feldman has mastered provocative questions and used this skill very often during the panel. [2]

At the end of the day, everybody was offered drinks and snacks on the terrace. Most of the people didn't get enough food and alcohol (mostly alcohol) and so the mappers continued to party in Girona bars.

Day 2

Saturday marked the start of non-business day, which means more interesting stuff for geeks.

There were lots of good talks, but the best are (warning: subjective!):

"OSM without Delay" by Matt Amos

Matt is one of the core OSM server engineers and does a lot of interesting and useful things. He's currently working on OWL which is basically a visualizer of recent (or not so recent) changes in different areas. Matt talked about how the OWL implemented, pointing out the cool features. If you missed the talk, you should check out Matt's blog post, where he provides detailed explanation of his talk.
"Tag Central: a Schema for OSM" by David Earl

This talk caused a lot of people wondering -- why doesn't David implement that instead of talking about it? The tagging schema has been discussed several hundreds times and in the end no one ever did implement it. This doesn't have to be part of the core functionality of OpenStreetMap anyway, so why not do it? Still, one of the best thought-out ideas about tagging schema I've ever seen.
"What’s wrong with OSM?" panel discussion lead by Christopher Osborne

We all know what's wrong and in order to fix we have to BAN POTLATCH. Well, not really. At least, nobody was talking about editors as being big problems of the OSM. I must say that I can't remember the exact problems discussed by the attendees, but the word "do-ocracy" coined by Mikel Maron is probably the best description of OpenStreetMap as a community I've ever heard. I'm eagerly waiting for the video of this one and I really hope that all the question from audience were recorded.

Day 3

Sunday aka "The Big Match Day" was full of nice talks, but the ones I really liked were about cartography:

"What I’d like to do with Mapnik" by Steve Chilton

What I'd like to do with Steve Chilton is make him the king of the world and be with it. I'm serious, this talk was about so many beautiful things not possible (yet) in Mapnik. And all those things are implementable and would make any digital map rendered by this wonderful library even more beautiful. I advise this talk to anyone interested in cartography as a great place to learn about the beauty of maps.
"What I learned making a real map on real paper for real people and real money" by David Earl

This talk was pretty much in line with the talk from Steve Chilton, but paper maps of course need another type of cartography. So, this talk and the one about Mapnik IMHO were the best talks about maps as we know them -- images. Highly recommend this one!
"MapOSMatic: City Maps for the Masses" by Thomas and Maxime Petazzoni

Two Linux hackers, with basic knowledge of Python and another 4 persons they knew gathered together and in 5 days they had one of the best community project of OSM -- MapOSMatic. Not only this is useful (printed maps with indexes are useful) but it's also opensource. The talk itself wasn't that bright and informative but you should watch it anyway -- how else would you know about the internals and the history of MapOSMatic?

The tradition of SOTM is to close the conference with the auction, on which the all those unused things -- such as two EEE PCs, several video recorders, banners, maps, etc -- are sold. And again, just like in 2009, Henk Hoff was very successful at selling things at very good prices.

People

I've met a lot of people I always wanted to see and also several people I've already seen before but desperately need to talk more. Among these people are Dane Springmeyer [3], Sean Gillies [4], Simon Willison [5], Richard Weait [6] and many more.

Summary

If you like maps, open source software, open data and smart people doing crazy things -- see you next year at State Of The Map 2011!

[1]	Official post at MapQuest development team blog.

[2]	See also Steve's report about the panel on his own blog

[3]	The mad genius behind current Mapnik development

[4]	Pretty nice guy doing Egyptology and writing crazy Python GIS tools at http://gispython.org

[5]	You know, the guy who started little-known Django framework development. By the way, he is on 18 (eighteen!) month honey moon, so if you read this Simon -- I ENVY YOU. Just wanted you to know.

[6]	Canadian map geek, who created lots of useful tutorials and howto on Mapnik. Check out his blog, btw: http://weait.com/

Overview of cartography systems for OSM

2010-03-01T00:00:00Z

This post has been featured as research paper on my employee's research page

There have been a lot of talks and new projects built around maps and cartography services, but it seems like there's not a lot of interest in building beautiful maps. While I agree that not every person around has the skills to create perfect map, it's certainly nice to have tools to do that. This article will try to highlight current solutions and their highs and lows. But at first, some theory is necessary.

Overview

Definitions

Layers

Layers are essential part of any cartography style system. By using layers you can combine data that is contained in different projections or data source. Layers give you ability of defining the order in which map is rendered. Layers also give you enough abstraction over the pure data, which means that working with layers is way easier than simply querying data sources.

Rules

Rules define how certain kind of data should be rendered. For example, rule for primary highways on "roads" layer could contain description of coloring scheme, width of the rode, width and color of the border. You could also tune amount of shields rendered or the way corners are rendered. All of these parameters are described by the corresponding rule. Individual rules are often combined into so-called rulesets, which are then applied against different layers. This is often handy when you have several layers which contain similar data (e.g. two layers having different data sources -- PostGIS and ESRI shape files, both containing POI data).

Summary

So, what defines a good cartography styling?

First of all, powerful rules. The more ways there're to change the output of the map, the better. One should understand that such power doesn't come for free, it requires performance trade-offs. So, we get to the second point
rules should be simple enough so we can combine them into rulesets. Simple rules give us a bonus point of light performance degradation, while still having the ability to create complex rulesets.
Third: layers should have the ability to use different data sources. The more data sources you can use, the easier it's for the users of the system to build maps.

Real-world examples

Mapnik

Overview

Mapnik is the rendering engine used by main OpenStreetMap view. The project was started in 2005 by Artem Pavlenko as a general map rendering framework. The number one objective of the project is to create a library that would be able to render beautiful maps. To achieve this, Anti-Grain Geometry library was chosen as a backend. Unfortunately, AGG author seems to have stopped active development of the library. Mapnik is written in C++ and has quite advanced Python bindings.

Style system

Mapnik uses XML files to define style of the map. These files contain two important sections: Layers and Styles. Styles section contains named rulesets, where every Style has several Rules that define rules (obviously). Layers contain description of layers in order of rendering (first defined -- first rendered). Mapnik layer support is a great example of abstraction that doesn't leak and IMHO should be used a reference when designing your own layer system. One additional note regarding font support in Mapnik. You can define several fall-back fonts for a given style file, using FontSet directive. This is used often for fonts with limited language support (e.g. DejaVu, which doesn't have complete Asian font support).

Pros & Cons

Pros

Great layer implementation

Mapnik does layer support in the most correct way possible.
Font fall-back support

This feature gives ability to use beautiful maps where appropriate, while still being able to support rendering of all symbols.
Pattern symbolizing support

Gives ability to add shields on any geometrical figure the easy way.

Cons

Using XML for style files

Choosing XML in 2005 may sounded like a good idea, when it was the fastest way of serializing data, but now, when we have efficient JSON and YAML parsers, sticking to XML is a mistake. Style files should be easy for people to comprehend, not computers.
Limited zoom support

Mapnik doesn't support extended zoom syntax in Rule directives and thus you have to break DRY principle often when writing Mapnik styles.
Mixing data access and style information (layers and styles)

Layers and Styles should be defined in different files. This is a minor inconvenience for most people, but as an employee of CloudMade I feel really strong about this one. Some time in the distant past CSS and HTML were mixed up in one file, but humans came to realizing the fact this is often wrong, and now most, if not all, sites are splitting HTML and CSS into separate files. I hope the same will happen to Mapnik's style files someday.

MapCSS

Overview

MapCSS is the style specifically developed for Halcyon rendering engine. The engine itself is being developed by Richard Fairhurst and is already being used for Potlatch 2 OpenStreetMap editing tool. The engine is written in ActionScript and uses Flex.

Style system

As seen from the name, MapCSS is a superset of CSS designed specifically for cartography purposes. The approach to defining rules is simple -- define selector which defines the subset of data and then define object properties (e.g. width, opacity, etc.). You can read about MapCSS selector and properties syntax on the OpenStreetMap wiki.

Pros & Cons

Powerful selector syntax

Defining subsets of data is really easy with MapCSS and should regarded as the definite advantage. For example, you can use regular expressions in selectors.
eval support

This gives a lot of opportunities for dynamically changing the style of the map. Calling eval can be regarded as a disadvantage from the performance point of view, but it makes defining styles a more pleasant experience.

Cons

OpenStreetMap API is the only datasource

This is very unfortunate, but having no way of defining other data sources kills MapCSS's usage for anything else but OSM editing.
No explicit layer support

This might be merged with the previous point. Having no explicit layer support limits areas of application for MapCSS.
No shield supports

Shields are not supported yet in MapCSS, but I hope they will be there eventually
No grammar definition

MapCSS looks like CSS, but it's not and it would be very nice to have complete description (besides notes on usage in OpenStreetMap wiki).

GSS

Overview

GSS (Geo Style Sheets) is a cartography style sheet developed by Cartagen team. To cite their wiki:

Cartagen lets you make beautiful, customized maps with a simple stylesheet

Style system

GSS is a subset of JavaScript, which tries to look like CSS

Pros & Cons

Pros

Datasources support through HTTP API

Any datasource can be added by defining its HTTP API, which is simple and convenient at the same time.
Layers support

Due to the dynamic nature of GSS, layers are extremely easy to add and are indeed built in client library.

Cons

Targeted at dynamic environments

It's not often cartographers need dynamic maps, and I regard lack of interest to static content as a major disadvantage.

Osmarender

Overview

Osmarender was designed for only one purpose -- that is, rendering OSM data. The engine itself is a bunch of XSLT scripts, which take OSM files as input and produce SVG maps as output.

Style system

Osmarender uses two files to create SVG map -- rules and styles. Rules file is used to define the subsets of data while styles file is a CSS which defines the way data will look.

Pros & Cons

Pros

Based on SVG

SVG is not great, for all I know, but it's an established standard and a good example of how a markup language can be used. SVG gives a lot of geometry features for free, thus allowing to concentrate on the data itself rather than algorithms of its rendering.
Split of style and data information

As was mentioned before, splitting data and presentation logic is a good thing and Osmarender follows good practices as well.

Cons

Using XML for style files

That's the same point I raised with Mapnik style files. In short: each time you use XML for cartography, god kills a kitten.
Limited configuration options

It's more of a problem with Osmarender itself, rather than the style, but I have to acknowledge that in order to make something unusual, you've got to be extremely knowledgeable about Osmarender internals.
Lacking projection support

Osmarender styles don't support any kind of in-style reprojecting, which leads to unusual bugs (see below).
Only OSM files as datasource

Summary

So, what defines a really good cartography system? There're several important characteristics:

CSS-like properties for styling -- almost all systems get it right
Good layer support -- this is hard to achieve while retaining readability, but I'm sure it will be done some day.
Support for all kinds of data sources -- building beautiful maps while relying only one data source is often frustrating.
Readability -- style systems should be designed for people, not computers.

All of these are essential to any cartography system and should be targeted at first design iteration. You should also think about font support, correct positioning of labels and shield, avoidance of sharp edges and many other things. Designing a good cartography system is hard, but it was done before and will be done from time to time.

This covers most popular open source cartography systems currently available in the wild. You might point to other solutions, such as Kosmos, pyrender and many others, but as they're not as popular or rather innovative, I didn't cover them in this article.

Overview of cartography systems for OSM

2010-03-01T00:00:00Z

This post has been featured as research paper on my employee's research page

Overview

Definitions

Layers

Rules

Summary

So, what defines a good cartography styling?

First of all, powerful rules. The more ways there're to change the output of the map, the better. One should understand that such power doesn't come for free, it requires performance trade-offs. So, we get to the second point
rules should be simple enough so we can combine them into rulesets. Simple rules give us a bonus point of light performance degradation, while still having the ability to create complex rulesets.
Third: layers should have the ability to use different data sources. The more data sources you can use, the easier it's for the users of the system to build maps.

Real-world examples

Mapnik

Overview

Style system

Pros & Cons

Pros

Great layer implementation

Mapnik does layer support in the most correct way possible.
Font fall-back support

This feature gives ability to use beautiful maps where appropriate, while still being able to support rendering of all symbols.
Pattern symbolizing support

Gives ability to add shields on any geometrical figure the easy way.

Cons

Using XML for style files

Choosing XML in 2005 may sounded like a good idea, when it was the fastest way of serializing data, but now, when we have efficient JSON and YAML parsers, sticking to XML is a mistake. Style files should be easy for people to comprehend, not computers.
Limited zoom support

Mapnik doesn't support extended zoom syntax in Rule directives and thus you have to break DRY principle often when writing Mapnik styles.
Mixing data access and style information (layers and styles)

Layers and Styles should be defined in different files. This is a minor inconvenience for most people, but as an employee of CloudMade I feel really strong about this one. Some time in the distant past CSS and HTML were mixed up in one file, but humans came to realizing the fact this is often wrong, and now most, if not all, sites are splitting HTML and CSS into separate files. I hope the same will happen to Mapnik's style files someday.

MapCSS

Overview

Style system

Pros & Cons

Powerful selector syntax

Defining subsets of data is really easy with MapCSS and should regarded as the definite advantage. For example, you can use regular expressions in selectors.
eval support

This gives a lot of opportunities for dynamically changing the style of the map. Calling eval can be regarded as a disadvantage from the performance point of view, but it makes defining styles a more pleasant experience.

Cons

OpenStreetMap API is the only datasource

This is very unfortunate, but having no way of defining other data sources kills MapCSS's usage for anything else but OSM editing.
No explicit layer support

This might be merged with the previous point. Having no explicit layer support limits areas of application for MapCSS.
No shield supports

Shields are not supported yet in MapCSS, but I hope they will be there eventually
No grammar definition

MapCSS looks like CSS, but it's not and it would be very nice to have complete description (besides notes on usage in OpenStreetMap wiki).

GSS

Overview

GSS (Geo Style Sheets) is a cartography style sheet developed by Cartagen team. To cite their wiki:

Cartagen lets you make beautiful, customized maps with a simple stylesheet

Style system

GSS is a subset of JavaScript, which tries to look like CSS

Pros & Cons

Pros

Datasources support through HTTP API

Any datasource can be added by defining its HTTP API, which is simple and convenient at the same time.
Layers support

Due to the dynamic nature of GSS, layers are extremely easy to add and are indeed built in client library.

Cons

Targeted at dynamic environments

It's not often cartographers need dynamic maps, and I regard lack of interest to static content as a major disadvantage.

Osmarender

Overview

Osmarender was designed for only one purpose -- that is, rendering OSM data. The engine itself is a bunch of XSLT scripts, which take OSM files as input and produce SVG maps as output.

Style system

Osmarender uses two files to create SVG map -- rules and styles. Rules file is used to define the subsets of data while styles file is a CSS which defines the way data will look.

Pros & Cons

Pros

Based on SVG

SVG is not great, for all I know, but it's an established standard and a good example of how a markup language can be used. SVG gives a lot of geometry features for free, thus allowing to concentrate on the data itself rather than algorithms of its rendering.
Split of style and data information

As was mentioned before, splitting data and presentation logic is a good thing and Osmarender follows good practices as well.

Cons

Using XML for style files

That's the same point I raised with Mapnik style files. In short: each time you use XML for cartography, god kills a kitten.
Limited configuration options

It's more of a problem with Osmarender itself, rather than the style, but I have to acknowledge that in order to make something unusual, you've got to be extremely knowledgeable about Osmarender internals.
Lacking projection support

Osmarender styles don't support any kind of in-style reprojecting, which leads to unusual bugs (see below).
Only OSM files as datasource

Summary

So, what defines a really good cartography system? There're several important characteristics:

CSS-like properties for styling -- almost all systems get it right
Good layer support -- this is hard to achieve while retaining readability, but I'm sure it will be done some day.
Support for all kinds of data sources -- building beautiful maps while relying only one data source is often frustrating.
Readability -- style systems should be designed for people, not computers.

GIS programmer writing Python in Emacs

EuroPython 2011 report

Talks

My talk

People

Organization

The good

The bad

Summary

PyCon tutorial slides are up!

PyCon wrap up

My OpenStreetMap tutorial at PyCon

Tile Server Implementation

Before The Switch

The Switch

To Be Continued

Priority Queue for Twisted

SOTM 2010: looking back

Organization

Day 1

Day 2

Day 3

People

Summary

Overview of cartography systems for OSM

Overview

Definitions

Layers

Rules

Summary

Real-world examples

Mapnik

Overview

Style system

Pros & Cons

See also

MapCSS

Overview

Style system

Pros & Cons

See also

GSS

Overview

Style system

Pros & Cons

Osmarender

Overview

Style system

Pros & Cons

See also

Summary

Overview of cartography systems for OSM

Overview

Definitions

Layers

Rules

Summary

Real-world examples

Mapnik

Overview

Style system

Pros & Cons

See also

MapCSS

Overview

Style system

Pros & Cons

See also

GSS

Overview

Style system

Pros & Cons

Osmarender

Overview

Style system

Pros & Cons

See also

Summary