Import NASA SRTM3 data into Postgres

The first official week of my ‘summer’ of code was a succes. I managed to import the NASA SRTM3 data into Postgres. That is, the import is running at about 1 tile per minute while I am writing this. The result is available through Subversion and Git.

I think I am getting the hang of Test Driven Development. It would be great if someone can point me to a Postgres Python test tool. For now, I created a test database and wrote my own functions to populate it with test data for each test and clean up afterwards. In a month I will attend the Continuous Integration and Testing Conference in Melbourne (an OpenSpace event).

I mostly use PyGreSQL to connect to Postgres, but for large inserts I wanted to use the efficient Postgres COPY function. Psycopg2 supports this with its copy_from(table, file) function. However, this function mysteriously freezes, so I had to work around that.

Next week I am going to figure out how my application wants to receive a route. My original plan was that my application receives an xml request, but I just read the announcement of OpenRouteService so I will have a look at how they do it.

Tonight I am flying to Melbourne so I have some time to read The Summer of Code Mystery Book.


Started coding

Last week I learned more about Postgres and I think I will like that better than MySQL once I have completely figured it out. I also looked at the GIS extension PostGIS and in particular how I can access it through Python. I considered Django and some other options, but this part is still a bit vague to me. I’ll fgure that out later.

In my previous post I mentioned that I will be using Git for my project. Of course, I will also keep my repository synchronized with the OpenStreetMap subversion repository (through git-svn). I already ran into my first Git nuiscence: Git doesn’t like empty directories.

Next I have been reading about Test Driven Development and Agile stuff in general. There is even such a thing as testing databases. It was interesting to read about the cultural impendance mismatch between data professionals and application developers.

Today, May 26th, I started writing actual code for my project! This is what’s new:

  • a script that downloads the SRTM tiles for Australia and tests if the download was a succes
  • tests to verify the content of these tiles

The files are encoded as binary big-endian signed integers and so I needed to figure out how to deal with them. I am not sure if all went right, but I’ll find out later.

I wrote a lot of tests today so once I have more Fingerspitzgefühl for all the tools, I should be able to try some real TTD.

I will write a new post at least once a week during the project. I will try to do this whenever I reach an interesting milestone, so I can keep the titles a bit more interesting than “Weekly Status report #3”.

Australië livejournal

Bangkok and Melbourne

Bangkok SukhumvitG’Day mates,

Ik schrijf mijn berichten voorlopig in het Engels, zodat mijn eventuele nieuwe vrienden in Australië ook kunnen meegenieten. Rechtsboven deze pagina bevindt zich de knop “Google Translate”. Daarmee kan je deze pagina automatisch vertalen naar het Nederlands, hoewel de kwaliteit van de vertaling nog wat te wensen overlaat.

I made it to Australia! But first I spent a week in Bangkok, which I think is really a cool city. I saw a real Thai movie, or actually four scary short movies (4 Praeng) to be precise. Interesting detail is that before the movie starts, the audience stands up and the national anthem is played, complete with pictures of the king. Rita Verdonk would probably love it.

By the way, all English movies are subtitled in Thai and (!) all Thai movies are subtitled in English. If you bump into someone he will instantly say “sorry” and not “khor thot”. So Bangkok is pretty foreigner friendly. The only annoying thing is The Great Tuk-Tuk Scam.

I also managed to get out of the city for a day, to see the Death Railway, ride an elephant and sit on a boat.

Melbourne - Yarra river and downtownCompared to Bangkok, Melbourne looked like a charming little country side village at first. A very nice one though!

I am surfing the couch of some really nice folks in Richmond, a suburb very close to downtown. I split my time between exploring the city, working on my Summer of Code project and eating.

The number of places to eat in this city is insane. You can get out of a tram at a random stop, think about what you want to eat; then just walk in any direction and you will find it within five minutes. Most people put on a lot of weight in Australia; I wonder why…

In conclusion: I am having a great time, so until the next post: hooroo!


Community Bonding Period – Part I

G’Day all,

It’s been a quiet week for me on the Summer of Code front and far from quiet on the travelling front. I spent a week in Bangkok and arrived in Melbourne two days ago. Still I managed to get some things done.

After trying several options, I decided to use Github to host my Git repositories. For the time being, I will split my code in two parts. The first part will import the SRTM data into Postgres and the second part will do the actual altitude profile generation work. I figured that it makes sense to split them up, because the first part could be useful for other projects too and the second part can run from just a database dump.

I managed to read most of the Python tutorial and I am looking forward to writing some code in that language.

I have also been following some discussions on the OpenStreetMap developers mailing list as well as the Dutch and Australian ‘chat’ lists.

During the next one and a half week of the Community Bonding Period I will try to learn more about (some of):

Now that my jetlag is over and the temperature is nice and cool, I can work on my daily routine. My goal is to put enough time and energy into my project and at the same time see some nice sights and meet interesting people.

Hooroo from Oz!



Spoorzone voor

After biking around with my GPS logger and taking pictures, I managed to improve the map of my neighborhood. I added the cycle path, foot paths and some more water on the north side of the rail tracks:

Spoorzone na

So get your GPS logger and go improve the map!