Documentation Development Guide

Author:

Howard Butler

Contact:

hobu.inc at gmail.com

Author:

Jeff McKenna

Contact:

jmckenna at gatewaygeomatics.com

Last Updated:

2021-03-19

Background

The current structure of the MapServer documentation process is for developers to maintain their documents in reStructuredText format, and submit their changes through GitHub pull requests; you will first need to create a free personal account at GitHub. The Sphinx documentation generator is used to convert the reStructuredText files to html, and the live website is then automatically updated on each merged pull request.

Bemerkung

The live website is actually built automatically on Travis CI, triggered by each commit to GitHub. See the build history at https://travis-ci.com/github/MapServer/MapServer-documentation

GitHub Notes

  • Documentation files are stored in the /MapServer-documentation sub-project on GitHub at: https://github.com/MapServer/MapServer-documentation

  • Always submit pull requests to main, and if applicable your changes will be backported to a specific branch

  • All files should have only LF as line feed. Line feed (or ‚linefeed‘) differences can cause problems when comparing files through Git (Windows uses both a carriage-return character and a line feed character for newlines in its files, whereas macOS and Linux systems use only the line feed character). One option is to have Git convert CRLF to LF on commit, by setting this at the commandline:

    git config --global core.autocrlf input
    
  • See GitHub’s help guidelines to install Git locally

  • Useful git commands:

    • clone ‚main‘ locally:

      git clone https://github.com/MapServer/MapServer-documentation.git MapServer-documentation-git-main
      
    • update local files:

      git pull
      
    • add file to repository (need to commit after this):

      git add <file>
      
    • commit file to repository:

      git commit -m "my message" <files>
      

Workflow for simple edits

For simple edits (typos, etc.) not necessarily requiring compilation of the documentation, a simpler workflow to create a Pull Request, can be adopted that does not require any local tool:

  • Create a GitHub account if not already done

  • Go to https://github.com/MapServer/MapServer-documentation

  • Click on the „Fork“ button

  • You are now on https://github.com/{your_account}/MapServer-documentation

  • Click on the „Branch: …“ list (to the left of the green „New pull request“ button)

  • In the text box, entry a name describing your changes to create a new branch (ex „typo_fix“)

  • You are now on https://github.com/{your_account}/tree/{branch_name}

  • Browse through the tree and click on the file you want to edit. To find the file you want to edit, you may type text in the Search entry box at the top of the page (for example „Workflow for simple edits“ to find this page)

  • Click on the „Edit this file“ button (with a pen icon)

  • When done, fill the „Commit changes“ section at the bottom of the page (a title is enough for most changes)

  • Click on the „Commit changes“ button at the bottom of the page

  • Repeat the process with other files if needed.

  • When done, go back to https://github.com/{your_account}/tree/{branch_name} and click on the „New pull request“ button

  • You can see your changes and edit the pull request summary message if needed

  • Click on the „Create pull request“ button to confirm

  • You’re done ! (Someone with push rights will have to review and merge your changes)

Branch cleanup :

  • Once merged, you can delete branches when going to the Branch page : https://github.com/{your_account}/MapServer-documentation/branches

  • Click on the trash bin icon at the right of the branch you want to delete.

General Writing Guidelines

  • MapServer instead of mapserver, map server, Map Server, mapServer or map Server.

  • MapScript instead of mapscript, Mapscript, or map script.

  • PostGIS instead of postgis.

  • GitHub instead of Github or github

  • HowTo instead of howto or HOWTO.

  • Email addresses should be manually spam protected:

    myname at gmail.com instead of myname@gmail.com
    

reStructuredText Reference Guides

reStructuredText Formatting

  • All text should be hard breaks at or around the 80 column mark, just as the source code.

  • No .. sectnum:: in the contents directives

  • Always include the :Last Updated: line at the top of your document, such as:

    :Last Updated: 2021-02-16
    
  • Directives versionadded, deprecated, versionremoved should be placed directly below the parameter name, such as:

    PARAMETER
      .. deprecated:: 8.0
         Use a :ref:`xxx` block instead.
    

Installing and Using Sphinx for rst-html Generation

Bemerkung

As of 2021-03-19 the MapServer site requires at least Sphinx 3.5.0, alabaster 0.7.12, and the sphinx-removed-in extension - You can check the exact versions that the live website uses in the file requirements.txt, (or you can instead browse the latest versions of the Sphinx packages here) and then install locally such as:

pip install -r requirements.txt

On Windows:

  1. install Python 3.X

  2. make sure that the ‚C:/Python3X/Scripts‘ directory is your path

  3. install pip

  1. execute the following at the commandline:

pip help
  1. if the command is not found, then download the file get-pip.py locally.

  2. execute the following at the commandline:

python get-pip.py
  1. inside the /MapServer-documentation directory, execute:

    python -m pip install -r requirements.txt
    

    …you should see message: „Finished processing dependencies for Sphinx“

  2. inside the /MapServer-documentation directory, execute:

    make html
    

    or

    make latex
    

    the HTML output will be written to the _build/html sub-directory.

  3. install MiKTeX from https://miktex.org/download if you want to build pdfs

On Linux:

  1. Most flavors come with python installed (check with python –version), for specific steps check with your distribution. An example command might be:

    sudo apt-get install python3.9
    
  2. install pip

    curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
    python get-pip.py
    
  3. install Sphinx. Inside the /MapServer-documentation directory, execute:

    python -m pip install -r requirements.txt
    
  4. to process the documentation, from the /MapServer-documentation directory, run:

    make html
    make html BUILD_LANGUAGES=
    

    or

    make latex
    

    the HTML output will be written to the build/html sub-directory.

Bemerkung

If there are more than one translation, the above commands will automatically build all translations, unless you add a „BUILD_LANGUAGES=“ to the make arguments

On OS X:

  1. install sphinx using pip. Inside the /MapServer-documentation directory, execute:

    python -m pip install -r requirements.txt
    
  2. install MacTex from http://www.tug.org/mactex/ if you want to build pdfs

  3. to process the documentation, from the MapServer /MapServer-documentation directory, run:

    make compile_messages
    
    make html
    

    or

    make compile_messages
    
    make latex
    

    the HTML output will be written to the build/html sub-directory.

Bemerkung

If there are more than one translation, the above commands will automatically build all translations.

(English) typo fixing script

(Tested on Linux. Should also likely run on MacOSX. Windows users might need to install Cygwin, Python 3, git, curl, sed, grep, awk)

The script in scripts/fix_typos.sh is an interactive script to detect and correct common spelling issues in the English documentation (en/ directory).

It requires Python 3 (python3 executable) to run. The first time it is run, it also uses ‚git‘ and ‚curl‘ binaries to download the codespell engine, its own dictionary, as well as the Debian Lintian and QGIS dictionaries. They are downloaded in the fix_typos/ subdirectories. This directory can be safely removed at any time (to enable getting new versions of the dictionaries or the codespell engine).

The script is launched with :

./scripts/fix_typos.sh

Whenever a typo is detected, it asks if it must be fixed with the proposed replacement (Y/N), or in case if there are several options, to select the option (0, 1, etc..).

It contains a white list of words that are marked as not being typo errors (in the WORDS_WHITE_LIST variable). The EXCLUDED_FILES variable can also be edited to add files fully ignored. The scripts/typos_whitelist.txt file can be edited to add full lines that should be ignored (sensitive to starting and trailing whitespace/tabulations, and only work on LF terminated files, hence prior conversion from CRLF to LF might be needed). scripts/typos_whitelist.txt should be used rather than WORDS_WHITE_LIST, only when a word is not a typo in a given context, but might be a typo in another context.

How translations are handled

Nowadays, MapServer have two coexisting systems to manage translations process. Both rely on Sphinx software.

The first one was the answer to the absence of internationalization support in first days of the Sphinx software. The second use the new system of internationalization and is currently only used for the French translation.

The „old“ way for translations

Although we think that nowadays the new system is better, we do not wanted to enforce every language community to change their habits.

  • All translations are organized in subdirectories in MapServer’s /MapServer-documentation directory

  • The directories are named using ISO3166-1 alpha-2 country codes, which will also reference to the corresponding flag icon

  • Translations are based on the English documentation

  • The directory structure and filenames must be kept, they are used to generate links between the different translations

Bemerkung

To start a new translation, copy the directories images and include from MapServer-documentation/en to MapServer-documentation/<lang>, where <lang> is one of the country codes. You also should copy the MapServer-documentation/en/documentation.txt and MapServer-documentation/en/index.txt files into your <lang> directory (the build process requires these files…you are free to edit them as you wish for your own language).

  • Only translated files are kept in the <lang> directories and the repository.

  • The build script (Makefile and make.bat) have an option (init) to preprocess the <lang> directories. That means that each not translated English file is copied to the target <lang> directory. You don’t have to do this to build html files locally. If you do this, you have to clean up you directories afterwards.

  • To keep the translations in sync with the English documentation, the translators can monitor commits to the repository.

Bemerkung

One way to monitor changes is to subscribe the RSS feed through GitHub: in the MapServer-documentation repository, click on the „Commits“ tab and look for a subscription link for these commit changes, for example: see „Commit History“ at https://github.com/MapServer/MapServer-documentation/commits/main

  • You have to define which languages are available by setting TRANSLATIONS in the Makefile or make.bat:

    TRANSLATIONS     = en de
    

The build script will then process the subdirectories en and de. If they are not accessible, an error message will be returned.

The standard internationalization system (i18n) for translations

Initial translation process for a new language

It follows steps below:

  • Generate pot file from original English documentation using Sphinx (except if already here)

    make gettext
    
  • copy new version of *.pot files from _build/gettext/en to translated/pot

    make gettext_copy
    
  • Copy *.pot file => *.po file (a *.po file is the same as *.pot, only extension change and if you translate three languages, you will get three *.po file and each of them will be into a language dir)

    make generate_po_from_tmpl -e TRANSLATIONI18N=fr
    
  • translate your *.po file(s)

    You can do it with software like PoEdit but we may switch to Transifex, an hosted translation service to make contribution more easy, follow translation advance and have a review process for translation.

  • compile *.po to *.mo (sphinx need *.mo file to make substitution from original version to the language translated)

    make compile_messages -e TRANSLATIONI18N=fr
    
  • Generate doc for the language

You just need to set in the file Makefile or make.bat depending on your OS the following parameter before building the documentation:

TRANSLATIONI18N=fr

You need to change as well the scripts/build_docs.sh script to add your language at the top of the file:

LANGUAGES= fr

Please for both parameter, keep the alphabetical order to kept them readable.

Structure of the i18n files in the MapServer-documentation root directory

The structure you need to make all the transaltion process is like below (an excerpt to give you the main required structure)

translated/
├── fr
│   ├── about.po
│   ├── announcements_all.po
│   ├── cgi.po
│   ├── community.po
│   ├── copyright.po
│   ├── development.po
│   ├── documentation.po
│   └── LC_MESSAGES
│       ├── about.mo
│       ├── announcements_all.mo
│       ├── cgi.mo
│       ├── community.mo
│       ├── copyright.mo
│       ├── development.mo
│       └── documentation.mo
└── pot
    ├── about.pot
    ├── announcements_all.pot
    ├── cgi.pot
    ├── community.pot
    ├── copyright.pot
    ├── development.pot
    └── documentation.pot

Moreover, you only need to commit the *.pot and *.po files. Be cautious, you must have an LC_MESSAGES directory to receive *.mo files. To keep this directory in Git, it’s recommended to create and empty file into it.

Bemerkung

When adding a new language, take care of the ISO code used. Famfamfam flag icons is using ISO Country code, and transifex is using ISO Language code which can be different. See ISO country code page for more information.

The usual workflow for already existing translation

The previous process we explained is only for the language creation. After, the workflow consists of:

  • translating po files for your language via transifex,

  • pull po files from transifex and commit them

  • generating the .mo files

  • building again

Here are the command line:

  • Getting po files from transifex (you can use -l flags to filter the language):

    tx pull -a
    
  • Build mo files from po files:

    make compile_messages  -e TRANSLATIONI18N=fr
    
  • Build html page:

    make clean html
    

See below how to setup your transifex account to use transifex client to push, pull translated files and build html files.

Sometimes, when a change occurs in the main english documentation, you will need to generate again the pot files. Then, you will have to merge your translated files with the new pot files using for example the command line utility (from gettext) below:

make gettext
make gettext_copy

-U mean update po file with pot file and -N mean don’t use fuzzy but exact match. Then push po files to transifex (you can use -l flags to filter the language):

tx push -s

When committing your update, add [build_pdf] or [build_translations] in order to build pdf and/or translation to the webserver.

Knonw issues:

  • when downloading po files to local dir, if the file exists it won’t be updated. At this moment we should remove all po files before downloading.

  • when building html doc cache system doesn’t allow to update a page with up to date translation. We should use the clean target with make.

  • This command line throw an error:

    tx push -s -t
    

How to use Transifex client to manage translation process

Bemerkung

Transifex is described more in depth in the document: How to Help Translate the Documentation

  • Install transifex client:

    sudo apt-get install transifex-client
    
  • Edit ~/.transifexrc and add the appropriate info in there:

    vim ~/.transifexrc
    
    • With the following content:

      [https://www.transifex.com/]
      username = yjacolin
      token =
      password = passw0rd!
      hostname = https://www.transifex.com/
      

Reference Labels

:ref: reference labels

Label

Title

about

About

agg

AGG Rendering Specifics

antialias

AntiAliasing with MapServer

arcinfo

ArcInfo

arcsde

ArcSDE

autotest

Regression Testing

background

Tutorial Timeframe

batch_utilities

Batch Scripting

bugs

Bug Submission

cgi

CGI

cgi_controls

MapServer CGI Controls

cgi_introduction

MapServer CGI Introduction

class

CLASS

community

Community Activities

development

Development

dgn

DGN

documentation

MapServer 5.2.2 Documentation

documentation_development

Documentation Development Guide

dotnet_compile

.NET MapScript Compilation

download

Download

dynamic_charting

Dynamic Charting

editing

Mapfile Editing

errors

Errors

example1-1

Example 1.1

example1-1-map

Example1-1.map

example1-2

Example 1.2

example1-2-map

Example1-2.map

example1-3

Example 1.3

example1-3-map

Example1-3.map

example1-4

Example 1.4

example1-4-map

Example1-4.map

example1-5

Example 1.5

example1-5-map

Example1-5.map

example1-6

Example 1.6

example1-6-map

Example1-6.map

example1-7

Example 1.7

example1-7-map

Example1-7.map

example1-8

Example 1.8

example1-8-map

Example1-8.map

expressions

Expressions

faq

FAQ

fastcgi

FastCGI

feature

FEATURE

filter_encoding

WFS Filter Encoding

flash

Flash Output

fontset

FONTSET

format_types

Data Format Types

genindex

Stichwortverzeichnis

git

GitHub

gloss

Glossary

gml

GML

gpx

GPS Exchange Format (GPX)

grid

GRID

html_legend

HTML Legends with MapServer

iis

IIS Setup for MapServer

imagemaps

HTML Imagemaps

include

INCLUDE

inline

Inline

input

Data Input

input_postgis

PostGIS/PostgreSQL

installation

Installation

introduction

An Introduction to MapServer

irc

IRC

join

JOIN

kml

KML - Keyhole Markup Language

label

LABEL

layer

LAYER

legend

LEGEND

legend_utility

legend

license

License

linux

Linux

lists

Mailing Lists

management

File Management

map

MAP

map2img

map2img

map_context

Map Context

mapcache_formats

MapCache Image Formats

mapcache_jpeg_format

MapCache JPEG Format

mapcache_png_format

MapCache PNG Format

mapcache_mixed_format

MapCache Mixed PNG+JPEG Format

mapcache_caches

MapCache Cache Backends

mapcache_grids

MapCache Tile Grids

mapcache_cache_sqlite

MapCache Sqlite Caches

mapcache_cache_disk

MapCache Disk Caches

mapcache_cache_memcache

MapCache Memcache Caches

mapcache_cache_tiff

MapCache TIFF Caches

mapcache_services

MapCache Tile Services

mapcache_ve

MapCache VirtualEarth Quadkey Tile Service

mapcache_tms

MapCache Tile Map Service

mapcache_kml

MapCache Super-Overlay KML Services

mapcache_wmts

MapCache OGC WMTS Service

mapcache_wms

MapCache OGC WMS Service

mapcache_gmaps

MapCache GoogleMaps Tile Service

mapcache_proxying

MapCache Proxying Support

mapcache_dimensions

MapCache Tileset Dimensions

mapcache_featureinfo

MapCache FeatureInfo Requests

mapcache_tile_assembling

MapCache Tile Assembling

mapcache_sources

MapCache Input Sources

mapcache_source_wms

MapCache WMS Source

mapcache_source_mapfile

MapCache Native MapServer Sources

mapcache_http

MapCache HTTP Request

mapcontext_cgi

Map Context Files

mapfile

Mapfile

mapfile_tuning

Mapfile

mapinfo

MapInfo

mapscript

MapScript

mapscript_ows

MapScript Wrappers for WxS Services

mapscript_tests

MapScript Unit Testing

mapserv

mapserv

modindex

Modulindex

msencrypt

msencrypt

mysql

MySQL

ntf

NTF

oci

Oracle Spatial

oci_install

Oracle Installation

ogc

OGC Support and Configuration

ogc_support

OGC Support

ogr

OGR

online_resource_wms

More About the Online Resource URL

optimization

Optimization

osx

Mac OS X

output

Output Generation

outputformat

OUTPUTFORMAT

pdf

PDF Output

pgeo

ESRI Personal Geodatabase (MDB)

php

PHP MapScript

php_example

By Example

php_install

PHP MapScript Installation

php_install_example_steps

Example Steps of a Full Windows Installation

projection

PROJECTION

psc

Project Steering Committee

python

Python MapScript Appendix

querying

Querying

querymap

QUERYMAP

raster

Raster Data

raster_optimization

Raster

reference

REFERENCE

rfc1

MS RFC 1: Technical Steering Committee Guidelines

rfc10

MS RFC 10: Joining the Open Source Geospatial Foundation

rfc11

MS RFC 11: Support for Curved Labels

rfc12

MS RFC 12: C code Unit tests

rfc13

MS RFC 13: Support of Sensor Observation Service in MapServer

rfc14

MS RFC 14: Relative Coordinates for INLINE features

rfc15

MS RFC 15: Support for thread neutral operation of MapServer/MapScript

rfc16

MS RFC 16: MapScript WxS Services

rfc17

MS RFC 17: Dynamic Allocation of layers, styles, classes and symbols

rfc18

MS RFC 18: Encryption of passwords in mapfiles

rfc19

MS RFC 19: Style & Label attribute binding

rfc2

MS RFC 2: Creating line features and/or shapes using WKT

rfc21

MS RFC 21: MapServer Raster Color Correction

rfc22a

MS RFC 22a: Feature cache for long running processes and query processing

rfc23

MS RFC 23: Technical Steering Committee Guidelines

rfc24

MS RFC 24: Mapscript memory management

rfc24first

MS RFC 24: Mapscript memory management

rfc25

MS RFC 25: Align MapServer pixel and extent models with OGC models

rfc26

MS RFC 26: Version 5 Terminology Cleanup

rfc27

MS RFC 27: Label Priority

rfc28

MS RFC 28: Redesign of LOG/DEBUG output mechanisms

rfc29

MS RFC 29: Dynamic Charting Capability

rfc3

MS RFC 3: Feature Layer Plug-in Architecture

rfc30

MS RFC 30: Support for WMS 1.3.0

rfc31

MS RFC 31: Loading MapServer Objects from Strings

rfc32

MS RFC 32: Support for Anti-Grain Geometry (AGG) Rendering Engine

rfc33

MS RFC 33: Removing msLayerWhichItems() from maplayer.c

rfc39

MS RFC 39: Support of WMS/SLD Named Styles

rfc4

MS RFC 4: MapServer Raster Resampling

rfc40

MS RFC 40: Support Label Text Transformations

rfc41

MS RFC 41: Support of WCS 1.1.x Protocol

rfc42

MS RFC 42: Support of Cookies Forwarding

rfc43

MS RFC 43: Direct tile generation for Google Maps and Virtual Earth API

rfc44

MS RFC 44: Restore URL modification of mapfiles to pre-5.0 levels

rfc45

MS RFC 45: Symbology, Labeling, and Cartography Improvements

rfc46

MS RFC 46: Migrate Website to OSGeo

rfc47

MS RFC 47: Move IGNORE_MISSING_DATA to run-time configuration

rfc48

MS RFC 48: GEOTRANSFORM Geometry operations

rfc49

MS RFC 49: Symbology, Labeling, and Cartography Improvements

rfc5

MS RFC 5: MapServer Horizon Reprojection Improvements

rfc50

MS RFC 50: OpenGL Rendering Support

rfc51

MS RFC 51: XML Mapfile Format

rfc53

MS RFC 53: Guidelines for MapScript method return values

rfc54

MS RFC 54: Rendering Interface API

rfc55

MS RFC 55: Improve control of output resolution

rfc56

MS RFC 56: Tighten control of access to mapfiles and templates

rfc6

MS RFC 6: Color Range Mapping of Continuous Feature Values

rfc7

MS RFC 7: MapServer CVS Commit Management

rfc7.1

MS RFC 7.1: MapServer SVN Commit Management

rfc8

MS RFC 8: Pluggable External Feature Layer Providers

rfc9

MS RFC 9: Item tag for query templates

rfcs

Request for Comments

runsub

Run-time Substitution

s57

S57

scalebar

SCALEBAR

scalebar_utility

scalebar

sdts

SDTS

search

Suche

section1

Section 1: Static Maps and the MapFile

section2

Section 2: CGI variables and the User Interface

shapefiles

ESRI Shapefiles (SHP)

shptree

shptree

shptreevis

shptreevis

sld

SLD

sortshp

sortshp

sos_server

SOS Server

source

Source

sponsors

Sponsors

style

STYLE

styleitemauto-label-styles

Accessing OGR STYLEITEMAUTO Label Styles Through MapScript

svg

SVG

svn

Subversion

swig

SWIG MapScript API Reference

sym2img

sym2img

sym_construction

Cartographic Symbol Construction with MapServer

sym_examples

Symbology Examples

symbol

SYMBOL

template

Templating

testing

Testing

tiger

USGS TIGER

tile4ms

tile4ms

tile_mode

Tile Mode

tileindex

Tile Indexes

tutorial

MapServer Tutorial

unix

Compiling on Unix

utilities

Utilities

variable_sub

Variable Substitution

vector

Vector Data

vector_optimization

Vector

vim

VIM Syntax

virtual_vector

Virtual Spatial Data

wcs_format

WCS Use Cases

wcs_server

WCS Server

web

WEB

wfs

WFS

wfs_client

WFS Client

wfs_server

WFS Server

win32

Compiling on Win32

windows

Windows

wms_capabilities

Validate the Capabilities Metadata

wms_client

WMS Client

wms_server

WMS Server

wms_time

WMS Time

wrapper

A Simple CGI Wrapper Script

Regenerating the reference labels

You can regenerate the reference labels by issuing:

make labels

from the MapServer-documentation directory similar to when you are building the html or latex versions