Tileset Dimensions

Author:

Thomas Bonfort

Contact:

tbonfort at terriscope.fr

Author:

Seth Girvin

Contact:

sgirvin at compass.ie

Author:

Jerome Boue

Last Updated:

2020-05-25

Introduction

WMS layers in MapServer can support dimensions, see WMS Dimension for details. MapCache also supports dimensions for tilesets.

Storing separate tile caches for different dimensions has several practical use cases such as creating caches for different spatial boundaries, elevations, or time periods. It also provides a mechanism for dynamically switching Mapfiles based on a dimension value.

MapCache uses Dimensions to:

• create the directory structure of disk and SQLite caches, using the template structure with {dim} replacement entry, see Template Structure in Disk Caches and Using Multiple SQLite Database Files in SQLite Caches.

• create the database structure of SQLite caches, using custom schema with :dim replacement entry, see Custom Schema in SQLite Caches.

• build requests to WMS sources: any dimensions specified for a tileset will be forwarded on to their WMS source whenever a cache miss occurs.

Enabling Dimension Support in MapCache

Dimensions are enabled in the configuration file at the Tileset level. Each tileset may define its own dimensions with a <dimensions> section which includes one or several <dimension> elements.

Dimension Attributes

Several attributes are defined for the <dimension> element.

• name (required): The name by which the dimension is referred to in a WMS request (minus the dim_ prefix as specified in WMS 1.1.1 Specification).

• default (required): The default value for the dimension if none is explicitly provided in the WMS request.

• type (required): MapCache defines several ways for specifying a dimension: values, regex, sqlite, postgresql, elasticsearch and time. They are described in the following sections.

• unit (optional): dimension unit. This information is used only in the GetCapabilities response.

Dimension Types

Several types of dimensions are supported by MapCache: Values, Regex, SQLite, PostgreSQL, ElasticSearch and Time. They can be classified into two groups depending on how dimension values are defined:

• First level dimension types: Values and Regex. Dimensions values are statically specified in the configuration file.

• Second level dimensions types: SQLite, PostgreSQL, ElasticSearch and Time. Dimensions values are stored in a dynamic backend (SQLite file, PostgreSQL database or ElasticSearch index) so that an update in the dimension values does not involve a server restart.

First Level Dimensions

First level dimensions define values in the configuration file.

Values Dimensions

A Values type dimension lists all possible dimension values. The name of the dimension is used as the key in a query string when accessing through a WMS or WMTS service, for example &DIM1=foobarbaz.

If a client does not provide the dimension as a key value pair the default will be used (in this example foobar).

<tileset name="LayerName">
    <source>LayerSource</source>
    <cache>sqlite</cache>
    <grid>GoogleMapsCompatible</grid>
    <format>PNG</format>
    <metatile>5 5</metatile>
    <metabuffer>10</metabuffer>
    <dimensions>
        <dimension type="values" name="DIM1" default="foobar">
            <value>foobar</value>
            <value>foobarbaz</value>
            <value>foo</value>
            <value>bar</value>
        </dimension>
    </dimensions>
</tileset>

Regex Dimensions

An alternative to the hard-coded list of values is to use regular expressions.

The following example creates a MAPFILE dimension, for using the same MapCache tileset with different MapServer Mapfiles. The name of the Mapfiles need not be known to MapCache, and can therefore be created even after MapCache has been started.

When a user passes a MAPFILE=/path/to/mapfile, the string /path/to/mapfile is validated against the supplied (PCRE) regular expression. The one in this example allows a name composed of alphanumeric characters separated by slashes (/) and finishing with « .map » ( [a-zA-Z0-9./]*.map$ ) , but will fail if there are two consecutive dots (..) in the path, to prevent file-system traversal.

<dimension type="regex" name="MAPFILE" default="/path/to/mapfile.map">
        <regex>^(?!.*\.\.)[a-zA-Z0-9\./]*\.map$</regex>
</dimension>

Second Level Dimensions

Second level dimensions store values in a dynamic back-end, a SQLite file, a PostgreSQL database or an ElasticSearch index. This allows for adding new dimension values without modifying MapCache configuration file, which otherwise would need to restart the server.

SQLite Dimensions

In a SQLite dimension, dimension values are stored in a SQLite file. Moreover, each value references a list of internal, or sub-dimension values which are used to query cache.

For example, let’s define a « sensor » dimension whose allowed values represent various Earth observation spacecrafts. A WMS request with that dimension may contain, e.g.: …&dim_sensor=spot&… In this example, each sensor references several data along an internal « product » sub-dimension. An example of the « sensor » dimension back-end contents is represented in the following table:

Sensor dimension	Product sub-dimension
spot	spot-img1
spot	spot-img2
phr	phr-img1
phr	phr-img2
phr	phr-img3

The WMS request occurs at the « sensor » dimension level. The cache query is performed at the « product » sub-dimension level. In case of a cache miss, the « product » sub-dimension values are used to query the data source.

Querying the back-end may then result in returning multiple sub-dimension values, each of which referencing distinct tiles. Assembling these tiles is described in Tile Assembly Policies section below.

A SQLite dimension is configured by specifying three elements:

• <dbfile>: Location of the SQLite file implementing the dimension

• <list_query>: SQL query returning all sub-dimension values for the dimension, e.g. spot-img1, spot-img2, phr-img1, phr-img2 and phr-img3 in the « sensor » dimension example.

• <validate_query>: SQL query returning the list of sub-dimension values for a given dimension value, e.g. spot-img1 and spot-img2 for the spot dimension value in the « sensor » dimension example.

<!-- "sqlite" dimension

     This example defines a "sensor" dimension whose possible values are
     stored in the /data/sensor.sqlite SQLite file. The default value is
     "phr".

     A WMS request with that dimension may contain, e.g.
     "... &dim_sensor=spot& ..."
-->
<dimension type="sqlite" name="sensor" default="phr">

   <!-- Access Point -->
   <dbfile>/data/sensor.sqlite</dbfile>

   <!-- All-values Query -->
   <list_query>
      SELECT distinct(product) FROM dim
   </list_query>

   <!-- Selected-values Query -->
   <validate_query>
      SELECT product FROM dim
       WHERE sensor=:dim
   </validate_query>

</dimension>

As illustrated in this example with :dim placeholder, queries can contain template entries. See Templating dimension queries section below.

PostgreSQL Dimensions

PostgreSQL dimensions are very similar to SQLite dimensions. In particular, the sub-dimension feature described in the section above is also available with PostgreSQL dimensions. They differ only in the back-end implementation which is a PostgreSQL database, and in the way they are configured. A PostgreSQL dimension is configured by specifying three elements:

• <connection>: Connection information of the PostgreSQL database implementing the dimension, as expected by the PQconnectdb() function of the libpq - C library.

• <list_query>: SQL query returning all sub-dimension values for the dimension, e.g. spot-img1, spot-img2, phr-img1, phr-img2 and phr-img3 in the « sensor » dimension example.

• <validate_query>: SQL query returning the list of sub-dimension values for a given dimension value, e.g. spot-img1 and spot-img2 for the spot dimension value in the « sensor » dimension example.

<!-- "postgresql" dimension

     This example defines a "sensor" dimension whose possible values are
     stored in the postgresql://mapcache:mapcache@localhost:5433/mapcache
     PostgreSQL database. The default value is "phr".

     A WMS request with that dimension may contain, e.g.
     "... &dim_sensor=spot& ..."
-->
<dimension type="postgresql" name="sensor" default="phr">

   <!-- Access Point -->
  <connection>
    host=localhost user=mapcache password=mapcache dbname=mapcache port=5433
  </connection>

   <!-- All-values Query -->
  <list_query>SELECT distinct(product) FROM dim</list_query>

   <!-- Selected-values Query -->
  <validate_query>SELECT product FROM dim WHERE sensor=:dim</validate_query>

</dimension>

As illustrated in this example with :dim placeholder, queries can contain template entries. See Templating dimension queries section below.

ElasticSearch Dimensions

ElasticSearch dimensions are similar to SQLite or PostgreSQL dimensions. In particular, the sub-dimension feature described in the section above is also available with ElasticSearch dimensions. ElasticSearch main difference with SQLite and PostgreSQL is its query language, Query-DSL, based on JSON instead of SQL. Therefore, <list_query> and <validate_query> elements shall contain queries expressed in that language. Inserting JSON data in XML configuration may need to encapsulate it in a <![CDATA[ … ]]> element.

In addition, the complex structure of ElasticSearch responses, also expressed in JSON, raises the need for extraction instructions to get sub-dimension values from query responses.

For example, let’s assume that this ElasticSearch query:

{   "size": 0,
    "aggs": { "distinct_product": { "terms": { "field": "product/keyword" } } },
    "query": { "term": { "sensor": "phr" } }
}

returns this response:

{   "took": 0,
    "timed_out": false,
    "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 },
    "hits": { "total": 5, "max_score": 0, "hits": [] },
    "aggregations": {
        "distinct_product": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
                { "key": "phr_img1", "doc_count": 1 },
                { "key": "phr_img2", "doc_count": 1 },
                { "key": "phr_img3", "doc_count": 1 }
            ]
        }
    }
}

The expected sub-dimension values are then:

[ "phr_img1", "phr_img2", "phr_img3" ]

This list is obtained by extracting the aggregations item from the outer dictionary, then distinct_product , then buckets. Finally the key item is to be extracted from each dictionary of the bucket list. The extraction instructions take the form of a path-like list of keywords to be used to extract sub-dimension values from JSON response provided by ElasticSearch:

[ "aggregations", "distinct_product", "buckets", "key" ]

This is specified in the MapCache configuration file by way of new XML elements, namely <list_response> and <validate_response>, containing extraction instructions respectively for <list_query> and <validate_query> queries.

An ElasticSearch dimensions is defined by specifying five elements:

• <http>: Connection information of the ElasticSearch index implementing the dimension. This element contains two sub-elements:

  • <url>: Search URL of the ElasticSearch index

  • <headers>: HTTP headers needed to access the ElasticSearch index. It must contain the tag: <Content-Type>application/json</Content-Type>.

• <list_query>: Query DSL query for getting all sub-dimension values for the dimension. This query is expressed in JSON format and may need to be escaped in a <![CDATA[ … ]]> tag.

• <validate_query>: Query DSL query for getting the list of sub-dimension values for a given dimension value. This query is expressed in JSON format and may need to be escaped in a <![CDATA[ … ]]> tag.

• <list_response>: List of successive keywords needed to extract actual sub-dimension values from the raw ElasticSearch response to the <list_query> query.

• <validate_response>: List of successive keywords needed to extract actual sub-dimension values from the raw ElasticSearch response to the <validate_query> query.

<!-- "elasticsearch" dimension

     This example defines a "sensor" dimension whose possible values are
     stored in the http://localhost:9200/sensor ElasticSearch index. The
     default value is "phr".

     A WMS request with that dimension may contain, e.g.
     "... &dim_sensor=spot& ..."
-->
<dimension type="elasticsearch" name="sensor" default="phr">

  <!-- Access Point -->
  <http>
    <url>http://localhost:9200/sensor/_search</url>
    <headers>
      <Content-Type>application/json</Content-Type>
    </headers>
  </http>

  <!-- Selected-values Query -->
  <validate_query><![CDATA[
    { "size": 0,
      "aggs": {
        "products": { "terms": { "field": "product.keyword" } }
      },
      "query": { "term": { "sensor": ":dim" } }
    }
  ]]></validate_query>

  <!-- Selected-values Response Extration Instructions -->
  <validate_response><![CDATA[
    [ "aggregations", "products", "buckets", "key" ]
  ]]></validate_response>

  <!-- All-values Query -->
  <list_query><![CDATA[
    { "size": 0,
      "aggs": {
        "products": { "terms": { "field": "product.keyword" } }
      }
    }
  ]]></list_query>

  <!-- All-values Response Extration Instructions -->
  <list_response><![CDATA[
    [ "aggregations", "products", "buckets", "key" ]
  ]]></list_response>
</dimension>

As illustrated in this example with :dim placeholder, queries can contain template entries. See Templating dimension queries section below.

Time Dimensions

MapCache supports WMTS and WMS requests that include a TIME parameter, for both timestamps and time intervals. See MS RFC 96: Time Dimension Support in MapCache Tilesets initial proposal for further details.

Up to Mapcache 1.6.1, defining a Time dimension involves a SQLite back-end. Two elements must be provided:

• <dbfile>: Location of the SQLite file implementing the time dimension

• <query>: SQL query returning the list of available timestamp values within a given time interval. In that query, start time and end time are represented by the respective placeholders :start_timestamp and :end_time_stamp. Returned values shall be in yyyy-mm-ddThh:mm:ssZ time format, as specified in RFC 3339, but restricted to UTC.

From MapCache 1.8 on, dynamic back-ends for Time dimensions can be PostgreSQL databases or ElasticSearch indices in addition to SQLite files. Configuring a Time dimension with a specific back-end is made by setting a time= »true » attribute within a SQLite, PostgreSQL or ElasticSearch dimension configuration.

Still, MapCache keeps an (almost) backward compatible configuration of Time dimension with an implicit SQLite back-end. Example in the following section highlights differences.

Time values used to query the dynamic back-end are expressed in Unix time, i.e. the number of seconds that have elapsed since January 1, 1970 (midnight UTC). Returned times should be MapServer compatible strings (see the WMS Time documentation for details).

Querying the back-end with an interval may result in returning multiple timestamp values, each of which referencing distinct tiles. Assembling these tiles is described in Tile Assembly Policies section below.

Configuring a Time Dimension using a default SQLite back-end

<!-- "time" dimension This example defines a "time" dimension whose possible values are stored in the /path/to/mapcache-time.sqlite SQLite file. The default value is "2010". A WMS request with that dimension may contain, e.g. "... &time=1999-08-11T11:03:07Z/2001-09-21T08:17:56Z& ..." --> <dimension type="time" name="time" default="2010"> <dbfile>/path/to/mapcache-time.sqlite</dbfile> <query> SELECT strftime("%Y-%m-%dT%H:%M:%SZ",ts) FROM time WHERE ts &gt;= datetime(:start_timestamp,"unixepoch") AND ts &lt;= datetime(:end_timestamp,"unixepoch") ORDER BY ts DESC </query> </dimension>

<!-- "time" dimension This example defines a "time" dimension whose possible values are stored in the /path/to/mapcache-time.sqlite SQLite file. The default value is "2010". A WMS request with that dimension may contain, e.g. "... &time=1999-08-11T11:03:07Z/2001-09-21T08:17:56Z& ..." --> <dimension type="time" name="time" default="2010"> <dbfile>/path/to/mapcache-time.sqlite</dbfile> <validate_query> SELECT strftime("%Y-%m-%dT%H:%M:%SZ",ts) FROM time WHERE ts &gt;= datetime(:start_timestamp,"unixepoch") AND ts &lt;= datetime(:end_timestamp,"unixepoch") ORDER BY ts DESC </validate_query> <list_query>SELECT strftime("%Y-%m-%dT%H:%M:%SZ", ts) FROM time</list_query> </dimension>

Up to MapCache 1.6.1

From MapCache 1.8 on

Configuring a Time dimension using an explicit back-end

 <!-- "time" dimension, "new-style" definition using a SQLite back-end

      This example defines a "time" dimension whose possible values are
      stored in the /data/times.sqlite SQLite file. The default value is
      "d1".

      A WMS request with that dimension may contain, e.g.
      "... &time=1999-08-11T11:03:07Z/2001-09-21T08:17:56Z& ..."
 -->
 <dimension type="sqlite" name="time" default="d1" time="true">
   <dbfile>/data/times.sqlite</dbfile>
   <list_query>SELECT strftime("%Y-%m-%dT%H:%M:%SZ", ts) FROM timedim</list_query>
   <validate_query>
        SELECT strftime("%Y-%m-%dT%H:%M:%SZ",ts) FROM timedim
         WHERE ts &gt;= datetime(:start_timestamp,"unixepoch")
           AND ts &lt;= datetime(:end_timestamp,"unixepoch")
      ORDER BY ts DESC
   </validate_query>
 </dimension>

 <!-- "time" dimension, "new-style" definition using a PostgreSQL back-end

      This example defines a "time" dimension whose possible values are
      stored in the postgresql://mapcache:mapcache@localhost:5433/time
      PostgreSQL database. The default value is "d1".

      A WMS request with that dimension may contain, e.g.
      "... &time=1999-08-11T11:03:07Z/2001-09-21T08:17:56Z& ..."
 -->
 <dimension type="postgresql" name="time" default="d1" time="true">
   <connection>
     host=localhost user=mapcache password=mapcache dbname=times port=5433
   </connection>
   <list_query>SELECT to_char(ts, 'YYYY-MM-DD"T"HH24:MI:SS"Z"') FROM timedim</list_query>
   <validate_query>
        SELECT to_char(ts,'YYYY-MM-DD"T"HH24:MI:SS"Z"') FROM timedim
         WHERE ts &gt;= to_timestamp(:start_timestamp)
           AND ts &lt;= to_timestamp(:end_timestamp)
      ORDER BY ts DESC
   </validate_query>
 </dimension>

 <!-- "time" dimension, "new-style" definition using an ElasticSearch back-end

      This example defines a "time" dimension whose possible values are
      stored in the http://localhost:9200/times ElasticSearch index. The
      default value is "d1".

      A WMS request with that dimension may contain, e.g.
      "... &time=1999-08-11T11:03:07Z/2001-09-21T08:17:56Z& ..."
 -->
 <dimension type="elasticsearch" name="time" default="d1" time="true">
   <http>
     <url>http://localhost:9200/times/_search</url>
     <headers>
       <Content-Type>application/json</Content-Type>
     </headers>
   </http>

   <list_query><![CDATA[
     { "query": { "match_all": { } } }
   ]]></list_query>
   <list_response><![CDATA[
     [ "hits", "hits", "_source", "ts" ]
   ]]></list_response>

   <validate_query><![CDATA[
     { "query": {
         "range": {
           "ts": { "gte": :start_timestamp, "lte": :end_timestamp }
         }
       },
       "sort": { "ts": { "order": "desc" } },
       "script_fields": {
         "iso_ts": {
           "lang": "painless",
           "source": "SimpleDateFormat f=new SimpleDateFormat(params.fmt); f.setTimeZone(params.tz); return (f.format(doc.ts.value.getMillis()))",
           "params": {
             "fmt": "yyyy-MM-dd'T'HH:mm:ss'Z'",
             "tz: "UTC"
           }
         }
       }
     }
   ]]></validate_query>
   <validate_response><![CDATA[
     [ "hits", "hits", "fields", "iso_ts", 0 ]
   ]]></validate_response>

 </dimension>

Templating dimension queries

<validate_query> and <list_query> can contain template entries that will be replaced at runtime by correct values. Valid template entries are:

• :dim Dimension value as provided in incoming HTTP request

• :tileset Tileset name

• :gridsrs Grid SRS

• :minx, :miny, :maxx, :maxy Parameters for rectangular extent

• :start_timestamp, :end_timestamp Parameters for time interval

Tile Assembly Policies

When several time or sub-dimension values are returned from a dimension back-end query, the resulting tile is an assembly of all simple tiles matching each value. The way simple tiles are assembled is set with the <assembly_type> tag of the <dimensions> element. Currently only two values are implemented in MapCache:

• none: No assembly is performed, the resulting tile is the first retrieved simple tile. This is the default value.

• stack: Every pixel of the resulting tile is filled with the first opaque pixel found in all simple tiles in their retrieval order, e.g.:

  Assembly type: stack

  Product #1	Product #2	stack assembly of Products #1 & #2

The resulting tile may or may not be cached depending on the value of the <store_assemblies> tag of the <dimensions> element. The two possible values are true and false. Default is true.

The <subdimensions_read_only> tag of the <dimensions> element indicates whether data source shall be queried in case of cache miss. The two possible values are true and false. Default is false.

<dimensions>
    <assembly_type>stack</assembly_type>
    <store_assemblies>true</store_assemblies>
    <subdimensions_read_only>false</subdimensions_read_only>
    <dimension >...</dimension>
</dimensions>

Tile Fetching Policies

MapCache can be configured to fetch all subtiles to assemble either within a single thread or with as many threads as subtiles to fetch. This is set with the <assembly_threaded_fetching> tag of the <dimensions> element. This tag has an optional maxzoom attribute to specify that multi-threaded fetching should be disabled from a given zoom level.

<dimensions>
    <assembly_threaded_fetching maxzoom="5">true</assembly_threaded_fetching>
    <!-- ... -->
    <dimension >...</dimension>
</dimensions>

Querying Dimensions at map level

By default, MapCache runs <validate_query> of second level dimensions for every requested tile of a map. When this involves slow remote PostgreSQL or ElasticSearch servers, it may be desirable to send the query only once for the whole map. A configuration option at the <dimension> level exists to enable this behaviour:

<dimension>
   <wms_querybymap>true</wms_querybymap>

Storing Dimensions

When using a disk based cache tiles will be stored in a folder structure similar to base/gridname/DIM1/value/xx/xx/xx/xx/xx/xx.png (where DIM1 is the dimension value).

The order of the <dimension> tags inside the <dimensions> tag is important as it is used to create the directory structure for the disk cache. If you change the order of these values, any tiles that have been previously cached are invalidated (they will be unavailable to MapCache but not deleted).

Templates can be used to change the folder structure for example:

<cache name="tmpl" type="disk">
    <template>/tmp/template-test/{tileset}#{grid}#{dim}/{z}/{x}/{y}.{ext}</template>
</cache>

Templates can also be used to change Sqlite database names e.g.

<!-- the following will create databases named TilesetName-#DimensionValue.db -->
<cache name="sqlite" type="sqlite3">
   <dbfile>{tileset}-{dim}.db</dbfile>
   <detect_blank />
</cache>

Accessing Tile Caches with Dimensions

To retrieve a tile with a specific dimension, add the dimension name and value to the request query string e.g. &DIM1=value

Only WMS and WMTS clients support passing dimensions to MapCache. Other tile services such as gmaps (GoogleMaps), tms, and KML do not support dimensions. WMTS however does allow access using a x,y,z addressing scheme, so by using URL rewriting on the web server tile-based clients (for example an OpenLayers ol.source.XYZ layer) can load a dimension-based tileset.

# a URL such as the following
http://servername/tiles/13/3914/2687.png
# can be rewritten into a WMTS request - note WMTS uses {z}/{y}/{x} rather than {z}/{x}/{y}
# so these values need to be swapped
http://servername/mapcache/wmts/1.0.0/layername/default/dimension/GoogleMapsCompatible/13/2687/3914.png

Seeding Tile Caches with Dimensions

The MapCache seeding tool allows specific dimensions to be seeded with the following syntax:

mapcache_seed -c mapcache_config.xml -t tileset1 -e -1187000,6695000,-605000,7450000 -z 7,10 --dimension DIMENSION_NAME=DIMENSION_VALUE