jmckenna at gatewaygeomatics.com
- Last Updated:
FlatGeobuf is an optimized binary encoding for vectors based on Flatbuffers, that focuses on read/query performance by leveraging a Packed Hilbert R-Tree spatial index, that enables fast bounding box spatial filtering. FlatGeobuf performance has helped make it become one of the chosen formats for streaming spatial data through HTTP / "the cloud".
More information about FlatGeobuf is available at:
FlatGeobuf files are usually a single binary file with a FGB filename extension, such as:
Windows users who want to follow along, can convert MS4W's included demo.db SpatiaLite database into a FlatGeobuf file (for the countries table) with the command (executed inside /ms4w/apps/local-demo/data/) :
ogr2ogr -f FlatGeobuf countries.fgb demo.db countries
Here are some rudimentary map draw speed results, with the MapServer 8.0 release code (running on Windows with MS4W):
FlatGeobuf (native) 0.008s Shapefile (native) 0.010s FlatGeobuf (OGR) 0.013s Shapefile (OGR) 0.023s GeoPackage (OGR) 0.042s SpatiaLite (OGR) 0.045s PostGIS (native) 0.053s GeoJSON (OGR) 0.089s
Native access is available since the MapServer 8.0.0 release, that uses code within the MapServer source to access the FGB file directly, as follows:
CONNECTIONTYPE flatgeobuf DATA "file.fgb"
For MapScript, you can use the setConnectionType()` method, such as
FlatGeobuf access is also available in MapServer through OGR's FlatGeobuf driver. You must use GDAL version 3.1.0 or later for FlatGeobuf access.
OGR uses the names of spatial layers within the FlatGeobuf file as layers.
The CONNECTION parameter must include the .fgb extension, and the DATA parameter should be the name of the OGR layer.
CONNECTIONTYPE OGR CONNECTION "name.fgb" DATA "layername"
Since the MapServer 7.6.0 release, you can also specify OGR driver options, and for the FlatGeobuf driver you can disable checking for invalid data, which is critical for maximum performance
CONNECTIONOPTIONS "VERIFY_BUFFERS" "NO" # for maximum performance END
First you should make sure that your local GDAL/OGR build contains the "FlatGeobuf" driver, by using the --formats command:
ogrinfo --formats Supported Formats: ... FlatGeobuf -vector- (rw+v): FlatGeobuf ...
Once you have confirmed that you have the FlatGeobuf driver, you are ready to try an ogrinfo command on your file to get a list of spatial layers:
ogrinfo countries.fgb INFO: Open of `countries.fgb' using driver `FlatGeobuf' successful. 1: countries (Multi Polygon)
Now use ogrinfo to get information on the structure of the spatial 'countries' layer:
ogrinfo countries.fgb countries -summary INFO: Open of `countries.fgb' using driver `FlatGeobuf' successful. Layer name: countries Geometry: Multi Polygon Feature Count: 177 Extent: (-180.000000, -90.000000) - (180.000000, 83.645130) Layer SRS WKT: GEOGCRS["WGS 84", ENSEMBLE["World Geodetic System 1984 ensemble", MEMBER["World Geodetic System 1984 (Transit)"], MEMBER["World Geodetic System 1984 (G730)"], MEMBER["World Geodetic System 1984 (G873)"], MEMBER["World Geodetic System 1984 (G1150)"], MEMBER["World Geodetic System 1984 (G1674)"], MEMBER["World Geodetic System 1984 (G1762)"], MEMBER["World Geodetic System 1984 (G2139)"], ELLIPSOID["WGS 84",6378137,298.257223563, LENGTHUNIT["metre",1]], ENSEMBLEACCURACY[2.0]], PRIMEM["Greenwich",0, ANGLEUNIT["degree",0.0174532925199433]], CS[ellipsoidal,2], AXIS["geodetic latitude (Lat)",north, ORDER, ANGLEUNIT["degree",0.0174532925199433]], AXIS["geodetic longitude (Lon)",east, ORDER, ANGLEUNIT["degree",0.0174532925199433]], USAGE[ SCOPE["Horizontal component of 3D system."], AREA["World."], BBOX[-90,-180,90,180]], ID["EPSG",4326]] Data axis to CRS axis mapping: 2,1 featurecla: String (0.0) scalerank: Integer (0.0) labelrank: Integer (0.0) sovereignt: String (0.0) sov_a3: String (0.0) adm0_dif: Integer (0.0) level: Integer (0.0) type: String (0.0) ...
Available since the 8.0.0 release, MapServer can access the FGB file directly, as follows:
/* Countries */ LAYER NAME "countries" TYPE POLYGON STATUS ON CONNECTIONTYPE flatgeobuf DATA "countries.fgb" EXTENT -180.000000 -90.000000 180.000000 83.645130 # for maximum performance PROCESSING "CLOSE_CONNECTION=DEFER" # for maximum performance CLASS NAME "World Countries" STYLE COLOR 200 200 200 OUTLINECOLOR 0 0 0 WIDTH 0.1 END #style END #class END #layer
For OGR connections, it is always recommended to set CONNECTIONTYPE, CONNECTION, and DATA, as follows:
/* Countries */ LAYER NAME "countries" TYPE POLYGON STATUS ON CONNECTIONTYPE OGR CONNECTION "countries.fgb" DATA "countries" # the OGR layername, found through ogrinfo CONNECTIONOPTIONS "VERIFY_BUFFERS" "NO" # for maximum performance (requires MapServer >=7.6.0) END #connectionoptions EXTENT -180.000000 -90.000000 180.000000 83.645130 # for maximum performance PROCESSING "CLOSE_CONNECTION=DEFER" # for maximum performance CLASS NAME "World Countries" STYLE COLOR 200 200 200 OUTLINECOLOR 0 0 0 WIDTH 0.1 END #style END #class END #layer
Use the MapServer commandline utility map2img to verify that your mapfile creates a valid map image, and also display draw times, such as:
map2img -m flatgeobuf.map -o ttt.png -map_debug 3 msDrawMap(): rendering using outputformat named png (AGG/PNG). msDrawMap(): WMS/WFS set-up and query, 0.000s msDrawMap(): Layer 0 (countries), 0.012s msDrawMap(): Drawing Label Cache, 0.002s msDrawMap() total time: 0.015s msSaveImage(flatgeobuf.png) total time: 0.008s
The following is not available through native FlatGeoBuf access.
Since MapServer 6.0, you can configure an OUTPUTFORMAT for your WFS (or WMS) service, for an OGR connection. See the OGR Output document for more information.
First, you must specify the OUTPUTFORMAT in your mapfile, for FlatGeobuf such as:
OUTPUTFORMAT NAME "OGRFLATGEOBUF" DRIVER "OGR/FlatGeoBuf" FORMATOPTION "STORAGE=filesystem" FORMATOPTION "FORM=simple" FORMATOPTION "FILENAME=result.fgb" FORMATOPTION "LCO:VERIFY_BUFFERS=NO" END
Next, you must specify that OUTPUTFORMAT using the wfs_getfeature_formatlist metadata parameter, at the LAYER level, such as:
METADATA "ows_title" "World Countries" "ows_abstract" "World Countries, served by MS4W" "ows_include_items" "all" "gml_include_items" "all" "gml_featureid" "ogc_fid" "wfs_getfeature_formatlist" "OGRGML,OGRFLATGEOBUF" "ows_geomtype" "MultiPolygon" "wfs_use_default_extent_for_getfeature" "false" END
Verify that OUTPUTFORMAT through a GetCapilities request, and look for the GetFeature/ResultFormat section, such as:
http://127.0.0.1/cgi-bin/mapserv.exe? map=/ms4w/apps/local-demo/flatgeobuf.map &SERVICE=wfs &VERSION=1.0.0 &REQUEST=GetCapabilities
Which gives a response containing:
Once we see the OGRFLATGEOBUF in the GetCapabilities response, then we can add &OUTPUTFORMAT=OGRFLATGEOBUF to the end of any WFS GetFeature request, such as:
http://127.0.0.1/cgi-bin/mapserv.exe? map=/ms4w/apps/local-demo/flatgeobuf.map &SERVICE=WFS &REQUEST=GetFeature &VERSION=2.0.0 &TYPENAMES=ms:countries &STARTINDEX=0 &COUNT=1000 &SRSNAME=urn:ogc:def:crs:EPSG::4326 &BBOX=-333.02127061643835759,-189,326.66640061643835224,189,urn:ogc:def:crs:EPSG::4326 &OUTPUTFORMAT=OGRFLATGEOBUF
and a file named result.fgb will be returned.