We recently released our first NZ GeoPack for 2013 Meshblocks, which provides a central map point, and basic demographic information, for every New Zealand Meshblock, Area Unit, and Territorial Authority (2013 pattern). We first blogged about the GeoPack here. A GeoPack enables you to put your data on a map and unlock the value of tying the data to a geographic location.
We spent a decent amount of time on it and wanted to share the approach and process we used to create this product. Before we started the GeoPack build we read through the wiki documents provided by YellowfinBI – they’re pretty comprehensive and their support team were great at answering any questions we had. Our approach, which drew from our experience in creating new products, was this:
Step 1 – research
Our initial reading started with this which gave us a good overview of what a GeoPack was from Yellowfin’s point of view:
Once we’d worked out a rough technical approach, we needed to understand the data we wanted to include. It needed to align with data from Statistics NZ to make sure we could include some valuable metrics such as those that come from the NZ Census. The New Zealand Census provides a picture of New Zealand and is used for planning our services such as hospitals and schools, you can find out more here. We settled on Meshblocks at the lowest level, rolling up to Area Units and Territorial Authorities. We also had to decide whether to clip to the NZ coastline or include all NZ oceanic and inlet meshblocks – our advice is to start with everything then remove anything you don’t want later.
Step 2 – get the data
Once we’d worked out the data we wanted, we put a request to Statistics NZ to get it via the stats NZ geometry team at firstname.lastname@example.org. They were great and sent us the data exactly as we asked for it. The only issue was our toolset to view it, so we found a few new ones that helped us open and manipulate the files, these were:
- LibreOffice (necessary for easy CSV export) http://www.libreoffice.org/download/
- Notepad++ (necessary for easy hand-editing of CSV) http://notepad-plus-plus.org/download/
- QGIS (only if you need to check the raw files received from Stats NZ) https://www.qgis.org/en/site/forusers/download.html
The files can be in excess of 100Mb so it’s worth considering how to get them delivered before making the request. For a lot of email systems, 100Mb attachments isn’t a feasible method, online storage or FTP would be a better option.
You probably want to validate the data too, making sure the record counts match what you’re expecting.
Step 3 – format the data ready for the GeoPack build
For each GeoPack level, you need to create a CSV file, in our case, we had a file for Meshblock, Area Unit and Territorial Authority. As well as these you need a file containing the demographic data and a file with the relationships between them.
Details on these formats can be found on the Yellowfin wiki: http://wiki.yellowfin.com.au/display/USER71/GeoPack+Format
CSV 1 – Meshblock
The Meshblock file contains the Meshblock ID, Area Unit ID (next level in the hierarchy) and the geometric fields.
CSV 2 – Area Unit
This links to Meshblocks via the Area Unit ID. It’s the same structure as the meshblock file containing the Area Unit ID, Territorial Authority ID and geometric fields.
CSV 3 – Territorial Authority (TA)
Similar to before, this links to the Area Unit file via the Territorial Authority ID. This file is the top of the hierarchy so only needs to contain the TA ID as well as the geometric fields.
CSV 4 – Demographic Data
This file contains the demographic data that adds value to the mapping capability. It should be a collection of statistics at the lowest level of your hierarchy, in our example, this would be Meshblock. Meshblock ID needs to be included plus a field for each of the demographic counts you want to include. There should be a value for each meshblock ID.
CSV 5 – Relationships
This file will contain the relationships between the files and is a simple column mapping, linking the relevant IDs together.
Step 4 – Package up and send to Yellowfin to create the GeoPack
Once all of your files are ready to go, put them in a CSV file and send to their support team to build the final pack. They’ll work with you to make sure it’s all correct and will give you pointers if you need them.
Step 5 – Install and test it
The final step is to test it. You can install it via the administration console, then bring in some data to join to it. These two links will let you know what to do:
In the end, you should end up with something that looks like this:
It took us a decent amount of time to build it, but we learnt a lot along the way. We hope this helps with your endeavours, if you want to save yourselves some time and skip ahead to step 5, you can download a copy for free here.
Keep exploring! Daniel.