A report on QGIS Income 2016

25/02/2017 by

QGIS.org blog

Greetings wonderful QGIS Users, Developers and Supporters! 2016 was quite an  exciting year for QGIS – we released QGIS 2.14LTR which has been a great stable release. We also set the wheels in motion for QGIS 3.0 which we plan to release later this year. QGIS development takes a lot of time and effort – and much of the work is done by volunteers – especially the core project maintenance work which typically does not attract paid-for consulting work like new feature work does.

We are extremely lucky and grateful to have a growing band of sponsors that are helping us to slowly but surely fund more core QGIS project work too. In this post we want to say ‘thank you’ to the many sponsors that have stepped up to the plate when we asked for help and sponsored QGIS. We also want to provide some insight as to what…

View original post 378 more words

A report on QGIS Income 2016

25/02/2017 by

Greetings wonderful QGIS Users, Developers and Supporters! 2016 was quite an  exciting year for QGIS – we released QGIS 2.14LTR which has been a great stable release. We also set the wheels i…

Source: A report on QGIS Income 2016

Sunshine Week 2017 at the National Archives!

24/02/2017 by

Tentative Agenda* 1:00: Welcome by the Archivist of the United States 1:10: A Dialogue about Access to the Nation’s Treasures Carla Hayden, Librarian of Congress David Ferriero, Archivist of the Un…

Source: Sunshine Week 2017 at the National Archives!

Schematic of rivers

09/02/2017 by

Nelson's log

Inspired by this discussion I tried to bang out a quick map of just the biggest rivers in the US in schematic form. My idea was to sort them all by length and draw them side by side, like this awesome 19th century visualization, but without straightening them first. I counted up which gnis_ids in NHDPlus had the most rows (as a proxy for river size), then made a special vector map source with just those rivers.

select geometry, name, strahler, huc8 from merged_rivers where gnis_id in (  354236, 591690, 465582, 1034981, 78762, 517018, 42838, 1384150, 517033, 400069, 201759, 1091369,  425264,  485443, 1035255,  835960, 1428399, 1101881,   78956, 1385432,  837263, 1533479, 1629903,   45730,  756398)

Sadly the resulting image (attached here) doesn’t really work. Some things are screwed up, like the Platte River in Nebraska which just kind of ends. Also you run into the question of defining what…

View original post 52 more words

More TopoJSON / watershed maps

09/02/2017 by

Nelson's log

I played more with the California watershed boundary data and TopoJSON. Two nice treatments, online at least temporarily. A full-state static map in D3 with some mild interaction letting you change the depth of HUC codes used in coloring. And a zoomable map written with Leaflet that while HUC colored, emphasizes watershed boundary details over the hierarchical HUCs.

The Leaflet implementation went pretty well. The one thing that tripped me up is my watershed boundaries are stored as bunch of Geometry objects in a GeometryCollection, instead of Feature objects in a FeatureCollection. Leaflet’s GeoJSON support does nice things with FeatureCollections, letting you change the style and bind a popup for each Feature. It doesn’t seem to do that with GeometryCollections. I worked around it by just iterating through the GeometryCollection and adding each Geometry manually. But I wonder if TopoJSON should be emitting FeatureCollections instead? ogr2ogr on my source .SHP…

View original post 100 more words

TopoJSON notes, watershed boundaries and HUCs

09/02/2017 by

Nelson's log

I played around with Mike Bostock’s TopoJSON today, a file format + Javascript library for efficiently encoding maps. It basically does the same thing as geojson, but gets amazing compression in part by identifying common line segments between polygons and only encoding them once (ie: draw the border just once) and in part by quantizing and simplifying all the data. It’s quite clever!

My test data is the Watershed Boundaries from NHDPlusV2; the California-only NHDPlusV21_CA_18_WBDSnapshot_01.7z (89MB) and the national NHDPlusV21_NationalData_WBDSnapshot_Shapefile_01.7z (1.7GB).

Converting

Converting the smaller file to TopoJSON was a snap, just

The flags are to add some GeoJSON properties to the resulting data file.

Simplifying

Taking the default arguments, TopoJSON reduced my 89MB SHP file to a 9.3MB TopoJSON file. Not bad, particularly since the straight GeoJSON is 257MB. A lot of compression thanks to shared boundaries. But also quantization; by default the topojson tool quantizes data to 10,000 values…

View original post 545 more words

Notes on vector rivers

09/02/2017 by

Nelson's log

Playing around with a vector river map of California. Some notes.

Datasets

The Natural Earth major river dataset is pretty good. Not quite detailed enough for the Dreamflows scale map. Also everything looks to be about 2000m north of the natural boundaries defined by counties, other river datasets, etc. Maybe I screwed up a projection?

After a 3 day delay the NHD finally gave me back a shapefile with detailed flowlines for Nevada County. It’s very detailed data but it still has fewer small creeks on it than Google Maps or Stamen Terrain Maps. It might just be seasonal data, or maybe someone has a more detailed set. I need to explore NHDPlus.

Nick was kind enough to give me a shapefile of major rivers in California; 5 meg shapefile, 648 objects. It looks like about the right level of detail for the Dreamflows map, I’m working with it now…

View original post 330 more words

msgpack for topojson

09/02/2017 by

Nelson's log

Mike Bostock has published topojson, a clever variant of GeoJSON that more efficiently encodes toplogies. For a map of US counties, for instance, the GeoJSON approach is a bunch of LineStrings or Polygons that define each state boundary. There’s a lot of redundant line segments; TopoJSON eliminates those by encoding the line segments separately and then referencing them to define each state boundary. Also TopoJSON uses a relative integer representation instead of absolute floating point geocoordinates, greatly reducing size.

But is it as small as possible? There’s a lot of pairs of small integers like [-1,0],[1,-20],[-14,10],[-27,-9]; roughly 8 bytes for what could fit in 2. I took a quick crack at using msgpack to encode the JSON for us-counties more efficiently, this is what I learned:

 Source 668312
 Gzip 175895
 Msgpack 344208
 Msgpack gzip 166761

Msgpack is about half the size of JSON, but after gzip that difference goes away…

View original post 88 more words

TopoJSON for rivers

09/02/2017 by

Nelson's log

Mike Bostock took a crack at using TopoJSON to encode the NHDFlowline dataset. Just the geometry for rivers in 2 dimensions; no properties, etc. Tested just for California. All sizes are one million byte megabytes.

  • Source shapefile: 132M, 72M gzipped.
  • Naive GeoJSON conversion: 184M, 56M gzipped.
    ogr2ogr -dim 2 -f GeoJSON -select ” new.geojson NHDFlowline.shp
  • GeoJSON rounded to 5 digits: 95M, 21M gzipped.
    liljson -p 5 < new.geojson
  • GeoJSON rounded to 3 digits: 80M, 9M gzipped.
    liljson -p 3 < new.geojson
  • TopoJSON: 20M, 3.3M gzipped.

So for this data TopoJSON is about 1/4 – 1/5 the size of the equivalent GeoJSON. And those advantages persist through gzip. And that’s for a pathologically bad case, where there’s no shared topology along polygon boundaries. Pretty much all the savings here must be coming from the delta encoding. Neat!

Update: Mike converted the whole US. 2.5G of .shp file input, 327M of topojson output.

View original post 49 more words

I hate installing Python libraries

09/02/2017 by

Nelson's log

(Caution: intemperate angry rant follows.)

I’m wasting more than an hour figuring out the fuckup of building GDAL to run in a virtualenv on Ubuntu 14.04 LTS. This hasn’t worked well in the year+ I’ve been trying, and it’s not working well now.

  1. set up a new virtualenv and activate it
    virtualenv -p /usr/bin/python2.7 venv
    source venv/bin/activate
  2. Install the openaddresses repo I have
    cd ~/src/oa/machine/
    pip install -e .
  3. Manually install cairo cffi (shouldn’t pip have done this?)
    sudo apt-get install libffi-dev python-cffi
    pip install cairocffi
  4. Give up trying to install GDAL with pip. Even after you work through the include path packaging problems, the C compiler starts throwing errors about variable scoping.
  5. Start over with a virtualenv that also uses system packages, which somehow seem to be working.
    virtualenv –system-site-packages -p /usr/bin/python2.7 venv

I shouldn’t publish my angry rant. GDAL is great software, and I appreciate all the volunteer open…

View original post 69 more words