I finally have some time to sit down and write up some thoughts on the QGIS User Conference and Developer Meeting (aka Hackfest) that we just held in Nødebo, Denmark. First up I need to thank Lene Fischer, who was the organiser and wowed us all with her relaxed and competent organisational approach to the conference. Thanks also to the University of Copenhagen School of Forestry – they sponsored the event by providing the venue and accommodation – and the venue was absolutely awesome with little cottages in the forest and all sorts of interesting diversions scattered around the forest. Lene gave me a list of names of people who helped to organise the event – I am sorry I have only got your first names but a very big thank you to you all!

 

Students: Runner, Shuttlebus, Kitchenaid, Cleaner, Info, Coordinator, Parking, Inn-keeper, Keyholder
Johanne
Thomas
Mikkel M
Steffen
Christian
Mikkel N
Ida
Louise
Anita
Thyge
Rune
Nanna
Peter
Jens
Heidi
Simon
Employees at University of CopenhagenCoordination, Accomodation, Bed&Linnen, Computer, Kitchen, Network, Tent/chairs/, Cookiebaker, Supporter, Cheerleader, Lifgt, Microphones/projector, Webpage, DTP,
Anne
Irene
Aleksander
Klaus
Vivian
Nicolas
Brian
Mike
Bo
Lene
Bent
Henning
Poul
Peter
MereteSusanne

 

 

 

On the first day of the user conference, I got to present a session on ‘the future of QGIS’ (video feed here and continued here) which held more as a town hall style meeting with a few themes (desktop, server, mobile etc.) I think the participants enjoyed the format and it was equally novel for the general user community (who got to have their questions answered directly by developers) and the developers (who got to see what real users look like).

The QGIS User Conference had many interesting talks (you can see the complete programme here – along with links to the video stream for each talk). For me the most interesting things happening at the meetup (both user conference and hackfest parts were:

  • the fact that we had our first ever general users conference (with around 150 attendees)
  • the geometry checking tools developed by Sandro Mani from Sourcepole
  • the huge amount of effort and thought being put into the processing framework – if you haven’t already tried out the QGIS processing tools, do go and try them!
  • The server side plugin framework that Alessandro Pasotti is working on – see his blog post here too http://www.itopen.it/qgis-developer-meeting-in-nodebo/
  • The amount of polish being applied to QGIS – there are probably less ‘gee whizz’ new features and a lot more fixes and improvements – just take a look at the incoming pull requests to get a flavour of the kind of activity going on.
  • The new geometry system by Marco Hugentobler (also from Sourcepole) which will support curves and 3D geometries (z / m). The graphical user interface for working with the new geometries won’t come until a later release, but 2.10 will get the underlying support added (along with shims to provide backwards compatibility to the old geometry classes).
  • There were some interesting talks on using QGIS in a server side / headless / command line context – again check out the talks and video streams in the programme to watch talks by Martin Dobias, Dražen Odobašić.
  • QGIS on mobile is coming – Matthias Kuhn showed off the current state of QField – the Android native interface for field work based on QGIS he has been working on. See his blog post here too for his take on the week. While the Android work shows lots of promise, there are still lots of problems to be resolved – for example missing support for ‘Lollipop’ devices. Please consider sponsoring Matthias’ efforts if you can.
  • There is a heap of interesting stuff coming down the pipeline from Nyall Dawson for the production of print maps and rendering effects for map renderers. Nyall also showed off some other very interesting ideas for context based variables that can be used in expressions – it’s hard to explain it in  a sentance or two – suffice to know that power users are going to have even more awesome tools at their fingertips for producing great maps.

One hot topic was ‘when will QGIS 3.0 be released’. The short answer to that question is that ‘we don’t know’ – Jürgen Fischer and Matthias Kuhn are still investigating our options and once they have had enough time to understand the implications of upgrading to Qt5, Python 3 etc. they will make some recommendations. I can tell you that we agreed to announce clearly and long in advance (e.g. 1 year) the roadmap to moving to QGIS 3.0 so that plugin builders and others who are using QGIS libraries for building third party apps will have enough time to be ready for the transition. At the moment it is still uncertain if there even is a pressing need to make the transition, so we are going to hang back and wait for Jürgen & Matthias’ feedback.

I apologise for not reporting on many of other interesting talks and birds of a feather meetings here – there was so much going on including work on documentation, translations, bug fixing, bug triaging that it is quite difficult to list it all here.

Two initiatives I was involved in at the meetup: the user certification programme and the formation of a QGIS legal entity. I am not going to post details here because things are not finalised yet (watch the mailing lists for details on the legal entity), but if you are interested in the certification programme, please get into contact – we have started drafting a roadmap for the roll out of our official curriculum. The QGIS project also got a huge boost from the QGIS Academy folks who will be contributing all their training resources right into the core of the QGIS documentation project (see Kurte Menke’s presentation notes on the programme).

We (Paolo Cavallini, Alessandre Pasotti, Nyall Dawson and myself) had a little roundtable discussion on the last day of the hackfest where we ran through some of the highlights from the week. You can listen to it here – or subscribe to the podcast at http://podcast.qgis.org (I will try to get back into the swing of making more regular episodes).

Well that wraps up my feedback for the event – I really encourage everyone to come along and join us on the next QGIS User Conference – it was fun, informal and informative!

Ok so here is the scenario:

You just got a nice new MacBook 15″ Retina computer thinking it would work as nicely for Linux as your 13″ MacBook did and then you discover that the hybrid Intel/Nvidia card support in Linux is a show stopper and the WebCam does not work under Linux.

Well that is what happened to me, so I decided to give working with OSX a try on this laptop with the help of docker for running all those essential apps that I use for development. One thing I was curious about was whether it would be possible to run native GUI (X11) applications from inside docker and have them show up on my OSX desktop. I turns out that it is fairly easy to do this – here is what I did:

Overview

  • Install brew
  • Install socat
  • Install XQuartz
  • Install Docker (I used Kitematic beta)
  • Grab a docker image that has a gui app you want to run (I used my the QGIS Desktop image published by Kartoza on the docker hub)
  • Run it forwarding the display to your OSX host

 

Digging In

Ok first install brew (an apt-like package manager for OSX).

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Now install socat – a command line tool that lets you redirect sockets in unix like OS’s – thankfully it runs in OSX too as it is a really neat tool!

brew install socat

Next we are going to install XQuartz – which basically gives you an X11 display client on your OSX desktop. Just grab the package at http://xquartz.macosforge.org/landing/ and do the usual OSX procedure for installing it.

Unfortunately docker does not run natively on OSX, and the whole boot2docker setup is probably quite difficult to explain to people. However there is a very nice (currently beta) docker client being developed for OSX called kinematic. I installed kinematic and then simply hit shift-command-t in order to get a bash shell with docker available in it.

Now grab my QGIS desktop image for docker:

docker pull kartoza/qgis-desktop

Once the image is downloaded we are done with the basic setup and can kick over to running our Linux GUI application (obviously QGIS in this example).

Running QGIS

Ok so there are four steps we need to do to run our Linux app:

  1. Start socat (in my testing it had to be done first)
  2. Start XQuartz
  3. Start Kinematic
  4. Start QGIS

I started socat like this:

socat TCP-LISTEN:6000,reuseaddr,fork UNIX-CLIENT:\"$DISPLAY\"

It will run in the foreground waiting for connections and then pass them over to XQuartz.

Next I started XQuartz (you can close the XTerm window that opens by default).  In X11 preferences in XQuartz, in the security tab, check both boxes:

Screen Shot 2015-04-13 at 23.40.21

Next I started kinematic, and pressed SHIFT-COMMAND-T to open a docker terminal.

Screen Shot 2015-04-13 at 23.16.21

Lastly I ran the QGIS docker container like this:

docker run --rm -e DISPLAY=192.168.0.3:0 \
    -i -t -v /Users/timlinux:/home/timlinux \
    kartoza/qgis-desktop qgis

You can mix in any standard docker options there – in this case I created  shared volume between my OSX home directory and a /home/timlinux directory in the container. You need to determine the IP address of your OSX machine and use it instead of the IP address listed after DISPLAY in the above command. Here is a nice picture of QGIS (from a Linux container) running on my OSX desktop:

Screen Shot 2015-04-13 at 23.52.21

 

 

This same technique should work nicely with any other GUI application under Linux – I will mostly use if for running tests of QGIS based plugins and for using QGIS in my docker orchestrated environments.

I have been using and learning docker since the early days after it was announced. I really love using docker but there are some things I just wish I had cottoned onto at the start. Here are my top 7 tips to help new users to docker avoid making the same mistakes I did.
docker-logo-370x290
  1. ​Docker is great when you want to densely pack many isolated services onto each server. If you prefer to take the approach of e.g. deploying many microservers (e.g. AWS micro instances) you might be better off looking at using a configuration management tool like ansible.​ Although even then docker still has some advantages as it is nice to be able to just check out an image onto a micro instance and spin it up knowing everything will ‘just work’.
  2. If you go with docker really avoid the temptation of treating docker containers like virtual machines. Even though you can run multiple services in each container (e.g. using supervisord) you should really architect your docker images so that they are virtual application appliances, with each appliance doing one thing only (like the old unix mantra eh?). So have a discrete appliance for postgis, another for uwsgi, another for mapserver, another for geoserver etc. rather than mixing things into the same service.
  3. You should do everything you can to make your containers stateless. Or put differently, you should be able to confidently destroy and redeploy a container as a fresh instance with no loss of data or configuration. This means that your investment should be in building the images on which your containers are based and not the containers themselves. In practice this means that you should e.g. mount your postgres cluster from a host volume, store your user uploaded files in a host volume and store no generated data inside the container itself.
  4. You should avoid the temptation to build your own container orchestration tools. Rather use a tool like fig (now part of docker as ‘docker compose’) that uses a simple yaml file to define your micro services architecture and can be used to reliably spin up a working configuration. I combine that with some simple makefiles because I am too lazy to remember the commands needed to administer the orchestrated application using fig.
  5.  Lean how to publish your images into hub.docker.com and have them build on push to your repo and build when the upstream container your docker image is based on gets an update (e.g. a security fix). Also invest the time in learning to tag different versions in hub so that you can have a known good working configuration against specific image versions.
  6. Build up your images in layers. Start with a standard base image e.g. ‘ubuntu:trusty’ then add python and save that as a new standard image, then add django and save that as a new base image etc. This way each virtual application you create is defined by only the thinnest amount of configuration and software deployment possible, and you can share the underlying logic of the lower layers between as many images as possible.
  7. My last piece of advice is that when you are building services that need to span multiple nodes, you need to plan your architecture carefully – docker does not yet have great built in multi-host networking so you may want to look at using a virtual switch, socat, a third party tool like pipeworks (if I remember the name properly) etc. Or better yet define your architecture so that hosts can operate independently of each other as much as possible.
I should mention that I have pretty much broken all these mantras during my journey into docker and am still paying the price. However, when you do get things working nicely, it is incredibly cool being able to log into a machine, check out your repo and type ‘fig up -d’ and seeing your application build and deploy itself.

Someone wrote to me asking if it would be possible to generate an XYZ ASCII file from a single band raster layer in QGIS. No doubt there are more efficient ways (this approach is pretty slow but it works), but I thought it would be fun to show how you can iterate over a raster, writing out the value of each cell into a text file (along with the centroid coordinates for that cell).

 

 

 

 

To use the script, you should save it to your local machine, then open the python console and load the script in the python editor. Next select a single band raster and then run the script in the editor. If your raster is quite large, it will take some time to run. I have spent zero time trying to optimise the script – if someone has an idea for doing it faster, send me a patch and I will update the example above.

The generated output dataset will look something like this:

 

Longitude,Latitude,VI
8.31259406548,7.86128343221,10
8.31264849753,7.86128343221,16
8.31270292958,7.86128343221,18
8.31248520138,7.8613378416,15
8.31253963343,7.8613378416,17
8.31259406548,7.8613378416,24
8.31264849753,7.8613378416,46
8.31270292958,7.8613378416,47
...

The resulting script can be used with programs like gdal_grid or loaded back into QGIS as a vector layer using the Delimited Text provider:

 

QGIS 74ac7f0 - raster_to_xyz_171

Sometimes you want to automatically generate a report to reflect the latest state of your data. For example you may be capturing spatial data into a PostGIS database and want a snapshot of that every few hours expressed as a pdf report. This example shows you how you can quickly generate a pdf based on a QGIS project (.qgs file) and a QGIS template (.qpt file).

Using this approach you can generate all kinds of useful outputs without ever needing to open QGIS each time you generate the report. Simply create the needed project and template files and then run it like this:

python generate_pdf.py

Google will be supporting Google Maps Engine (GME) for only one more year. If your company uses GME or was thinking about it, you’ll need to consider alternatives. Kartoza offers various alternatives depending on your use case. We are BoundlessGeo partners and offer support for the OpenGeo Suite in southern Africa. For more details of the Google announcement, including various scenarios where you should consider the OpenGeo Suite, see BoundlessGeo’s take on it.  Kartoza also offers development, support and training services across other components of the FOSS GIS stack that offer alternatives to Google and other proprietary products – we’d love to hear from you!

Kartoza is seeking GIS staff. We have a range of projects and challenges and will tailor the post to suit the ideal candidate.

Kartoza does a variety of work with a strong spatial component for a local and international clientèle and we’re also working on our own ideas. We do desktop GIS, web map servers, enterprise deployments, cartography, spatial data management, training and support. We use FOSS tools and produce mostly FOSS outputs. We have a very ‘open’ philosophy.

You will be organised, driven, productive, enjoy working in a changeable, informal environment and enjoy learning. Working in teams with software developers will appeal to you. If you consider yourself to be a power-user you must be comfortable with scripts and the command line. If you’re not a power user yet you must want to become one.

As this is a small company, you will be required to take on multiple projects and manage your time effectively. You will be anchored in an office in Paarl with the possibility of telecommuting. We have a fairly flat structure where your opinion counts. Your initiative and input into internal projects will be rewarded.

You should find your time at Kartoza enjoyable, rewarding and fulfilling, with a positive work-life balance. The equipment we provide won’t slow you down and we have progressive terms of service.

Minimum skills:

  • a flair and passion for geography and spatial thinking
  • Training or experience in
    • geography
    • computer science
    • electronic engineering
    • GIS
    • IT

Extra skills:

  • QGIS
  • Python
  • Linux, Bash, server skills
  • SQL
  • Any other tools in the FOSS GIS stack (GDAL/OGR, Geoserver, etc.)
  • Spatial databases, especially PostgreSQL / PostGIS
  • JavaScript
  • cartography, design
  • teaching, training
  • OGC standards

To apply, please send the following to [email protected] before 20 February 2015

  • your CV including your complete study and work history, 2 references, SA residence status
  • examples of your OWN work, e.g. github, websites, systems, maps, models, working code, etc.
  • your salary expectation
  • when you are available to start

Kartoza is on the lookout for developers. We have a range of projects and challenges and will tailor the post to suit the ideal candidate.

Kartoza does desktop, server and web development, testing and support for a local and international clientèle and we’re also working on our own ideas. Most of our projects have a spatial component. We use FOSS tools and produce mostly FOSS outputs. We have a very ‘open’ philosophy.

You will be organised, driven, productive, enjoy working in a changeable, informal environment and enjoy learning. Terms like agile, scrum, sprints, git, kanban will appeal to you.

As this is a small company, you will be required to take on multiple projects and manage your time. You will work from our office in Paarl with the possibility of telecommuting. We have an international team with a fairly flat structure where your opinion counts. Your initiative and input into internal projects will be rewarded.

You should find your time at Kartoza enjoyable, rewarding and fulfilling, with a positive work-life balance. The equipment we provide won’t slow you down and we have progressive terms of service.

Minimum skills:

  • Python
  • SQL
  • Linux

Extra skills:

  • Django / GeoDjango
  • GIS (especially with the FOSS GIS stack)
  • QGIS Python API and plugin development
  • Spatial databases, especially PostgreSQL / PostGIS
  • CSS3 / HTML5
  • JavaScript
  • Server skills, including working with containers
  • mobile application development
  • UX design
  • OGC standards
  • C++
  • Qt

To apply, please send the following to [email protected] before 20 February 2015

  • your CV including your complete study and work history, 2 references, SA residence status
  • examples of your OWN coding work, e.g. github, websites, apps, QGIS plugins, working code, etc.
  • your salary expectation
  • when you are available to start

Kartoza has just wrapped up a three-day Introduction to QGIS course that we gave pro bono at The African Quaternary: Environments, Ecology and Humans conference in Cape Town (AfQUA2015). Thank you to Kelly Kirsten from the local organising committee at the University of Cape Town for inviting us. Admire Nyakudya presented to about fifteen conference delegates from around the world. Attendees ran QGIS 2.6.1 on their own notebooks on the operating system of their choice. Some already use QGIS, some were experiencing QGIS for the first time and some some wanted to compare it with ArcGIS. The course content was guided largely by the QGIS training manual but as we usually do, adapted for the requirements, interests and pace of the class.

 

Kartoza QGIS class at AfQUA 2015

Admire presenting QGIS workshop at AfQUA

Today in a project we are working on we wanted to load a QGIS project. It takes  surprisingly few lines of code to make a small standalone application that loads a project and then shows it as a map in a window like this:

QGIS project in a standalone app

 

Here is the code I wrote to produce this:

The main bit of magic is the QgsLayerTreeMapCanvasBridge class which will convert your project into a layer tree so that the layers appear in the canvas. If you ever need to make a standalone python application with a nice map in it, consider using the QGIS API to do it!