Documentation Development Guide

Author:

Howard Butler

Contact:

hobu.inc at gmail.com

Author:

Jeff McKenna

Contact:

jmckenna at gatewaygeomatics.com

Last Updated:

2024-07-09

Background

The current structure of the MapServer documentation process is for developers to maintain their documents in reStructuredText format, and submit their changes through GitHub pull requests; you will first need to create a free personal account at GitHub. The Sphinx documentation generator is used to convert the reStructuredText files to html, and the live website is then automatically updated on each merged pull request.

Bemerkung

The live website is actually built automatically through a GitHub workflow, triggered by each push to the live branch. See the build history. The different language documentation sites are rebuilt when a commit message includes [build_translations].

GitHub Notes

• Documentation files are stored in the /MapServer-documentation sub-project on GitHub at: https://github.com/MapServer/MapServer-documentation

• Always submit pull requests to main, and if applicable your changes will be backported to a specific branch

• All files should have only LF as line feed. Line feed (or ‚linefeed‘) differences can cause problems when comparing files through Git (Windows uses both a carriage-return character and a line feed character for newlines in its files, whereas macOS and Linux systems use only the line feed character). One option is to have Git convert CRLF to LF on commit, by setting this at the commandline:

  git config --global core.autocrlf input
  

• See GitHub’s help guidelines to install Git locally

• Useful git commands:

  • clone ‚main‘ locally:

    git clone https://github.com/MapServer/MapServer-documentation.git MapServer-documentation-git-main
    

  • update local files:

    git pull
    

  • add file to repository (need to commit after this):

    git add <file>
    

  • commit file to repository:

    git commit -m "my message" <files>
    

Workflow for simple edits

For simple edits (typos, etc.) not necessarily requiring compilation of the documentation, a simpler workflow to create a Pull Request, can be adopted that does not require any local tool:

• Create a GitHub account if not already done

• Go to https://github.com/MapServer/MapServer-documentation

• Click on the „Fork“ button

• You are now on https://github.com/{your_account}/MapServer-documentation

• Click on the „Branch: …“ list (to the left of the green „New pull request“ button)

• In the text box, entry a name describing your changes to create a new branch (ex „typo_fix“)

• You are now on https://github.com/{your_account}/tree/{branch_name}

• Browse through the tree and click on the file you want to edit. To find the file you want to edit, you may type text in the Search entry box at the top of the page (for example „Workflow for simple edits“ to find this page)

• Click on the „Edit this file“ button (with a pen icon)

• When done, fill the „Commit changes“ section at the bottom of the page (a title is enough for most changes)

• Click on the „Commit changes“ button at the bottom of the page

• Repeat the process with other files if needed.

• When done, go back to https://github.com/{your_account}/tree/{branch_name} and click on the „New pull request“ button

• You can see your changes and edit the pull request summary message if needed

• Click on the „Create pull request“ button to confirm

• You’re done ! (Someone with push rights will have to review and merge your changes)

Branch cleanup :

• Once merged, you can delete branches when going to the Branch page : https://github.com/{your_account}/MapServer-documentation/branches

• Click on the trash bin icon at the right of the branch you want to delete.

General Writing Guidelines

• MapServer instead of mapserver, map server, Map Server, mapServer or map Server.

• MapScript instead of mapscript, Mapscript, or map script.

• PostGIS instead of postgis.

• GitHub instead of Github or github

• HowTo instead of howto or HOWTO.

• Email addresses should be manually spam protected:

  myname at gmail.com instead of myname@gmail.com
  

reStructuredText Reference Guides

• Quick reStructuredText

reStructuredText Formatting

• All text should be hard breaks at or around the 80 column mark, just as the source code.

• No .. sectnum:: in the contents directives

• Always include the :Last Updated: line at the top of your document, such as:

  :Last Updated: 2021-02-16
  

• Directives versionadded, deprecated, versionremoved should be placed directly below the parameter name, such as:

  PARAMETER1
    .. deprecated:: 8.0
       Use a :ref:`xxx` block instead.
  
  PARAMETER2
    .. versionadded:: 8.2.0
       Some description.
  

  Bemerkung

  Please specify version as x.y.z (Major.Minor.Point) whenever possible

Installing and Using Sphinx for rst-html Generation

Bemerkung

As of 2024-05-06 the MapServer site requires at least Sphinx 7.3.0, alabaster 0.7.16, and the sphinxcontrib-jquery extension - You can check the exact versions that the live website uses in the file requirements.txt, (or you can instead browse the latest versions of the Sphinx packages here) and then install locally such as:

pip install -r requirements.txt

On Windows:

1. install Python 3.X

2. make sure that the ‚C:/Python3X/Scripts‘ directory is your path

3. install pip

1. execute the following at the commandline:

pip help

2. if the command is not found, then download the file get-pip.py locally.

3. execute the following at the commandline:

python get-pip.py

4. inside the /MapServer-documentation directory, execute:

   python -m pip install -r requirements.txt
   

   …you should see message: „Finished processing dependencies for Sphinx“

5. inside the /MapServer-documentation directory, execute:

   make html
   

   or

   make latex
   

   the HTML output will be written to the _build/html sub-directory.

6. install MiKTeX from https://miktex.org/download if you want to build pdfs

On Linux:

1. Most flavors come with python installed (check with python –version), for specific steps check with your distribution. An example command might be:

   sudo apt-get install python3.9
   

2. install pip

   curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
   python get-pip.py
   

3. install Sphinx. Inside the /MapServer-documentation directory, execute:

   python -m pip install -r requirements.txt
   

4. to process the documentation, from the /MapServer-documentation directory, run:

   make html
   make html BUILD_LANGUAGES=
   

   or

   make latex
   

   the HTML output will be written to the build/html sub-directory.

Bemerkung

If there are more than one translation, the above commands will automatically build all translations, unless you add a „BUILD_LANGUAGES=“ to the make arguments

On OS X:

1. install sphinx using pip. Inside the /MapServer-documentation directory, execute:

   python -m pip install -r requirements.txt
   

2. install MacTex from http://www.tug.org/mactex/ if you want to build pdfs

3. to process the documentation, from the MapServer /MapServer-documentation directory, run:

   make compile_messages
   
   make html
   

   or

   make compile_messages
   
   make latex
   

   the HTML output will be written to the build/html sub-directory.

Bemerkung

If there are more than one translation, the above commands will automatically build all translations.

(English) typo fixing script

(Tested on Linux. Should also likely run on MacOSX. Windows users might need to install Cygwin, Python 3, git, curl, sed, grep, awk)

The script in scripts/fix_typos.sh is an interactive script to detect and correct common spelling issues in the English documentation (en/ directory).

It requires Python 3 (python3 executable) to run. The first time it is run, it also uses ‚git‘ and ‚curl‘ binaries to download the codespell engine, its own dictionary, as well as the Debian Lintian and QGIS dictionaries. They are downloaded in the fix_typos/ subdirectories. This directory can be safely removed at any time (to enable getting new versions of the dictionaries or the codespell engine).

The script is launched with :

./scripts/fix_typos.sh

Whenever a typo is detected, it asks if it must be fixed with the proposed replacement (Y/N), or in case if there are several options, to select the option (0, 1, etc..).

It contains a white list of words that are marked as not being typo errors (in the WORDS_WHITE_LIST variable). The EXCLUDED_FILES variable can also be edited to add files fully ignored. The scripts/typos_whitelist.txt file can be edited to add full lines that should be ignored (sensitive to starting and trailing whitespace/tabulations, and only work on LF terminated files, hence prior conversion from CRLF to LF might be needed). scripts/typos_whitelist.txt should be used rather than WORDS_WHITE_LIST, only when a word is not a typo in a given context, but might be a typo in another context.

How translations are handled

Nowadays, MapServer have two coexisting systems to manage translations process. Both rely on Sphinx software.

The first one was the answer to the absence of internationalization support in first days of the Sphinx software. The second use the new system of internationalization and is currently only used for the French translation.

The „old“ way for translations

Although we think that nowadays the new system is better, we do not wanted to enforce every language community to change their habits.

• All translations are organized in subdirectories in MapServer’s /MapServer-documentation directory

• The directories are named using ISO3166-1 alpha-2 country codes, which will also reference to the corresponding flag icon

• Translations are based on the English documentation

• The directory structure and filenames must be kept, they are used to generate links between the different translations

Bemerkung

To start a new translation, copy the directories images and include from MapServer-documentation/en to MapServer-documentation/<lang>, where <lang> is one of the country codes. You also should copy the MapServer-documentation/en/documentation.txt and MapServer-documentation/en/index.txt files into your <lang> directory (the build process requires these files…you are free to edit them as you wish for your own language).

• Only translated files are kept in the <lang> directories and the repository.

• The build script (Makefile and make.bat) have an option (init) to preprocess the <lang> directories. That means that each not translated English file is copied to the target <lang> directory. You don’t have to do this to build html files locally. If you do this, you have to clean up you directories afterwards.

• To keep the translations in sync with the English documentation, the translators can monitor commits to the repository.

Bemerkung

One way to monitor changes is to subscribe the RSS feed through GitHub: in the MapServer-documentation repository, click on the „Commits“ tab and look for a subscription link for these commit changes, for example: see „Commit History“ at https://github.com/MapServer/MapServer-documentation/commits/main

• You have to define which languages are available by setting TRANSLATIONS in the Makefile or make.bat:

  TRANSLATIONS     = en de
  

The build script will then process the subdirectories en and de. If they are not accessible, an error message will be returned.

The standard internationalization system (i18n) for translations

Initial translation process for a new language

It follows steps below:

• Generate pot file from original English documentation using Sphinx (except if already here)

  make gettext
  

• copy new version of *.pot files from _build/gettext/en to translated/pot

  make gettext_copy
  

• Copy *.pot file => *.po file (a *.po file is the same as *.pot, only extension change and if you translate three languages, you will get three *.po file and each of them will be into a language dir)

  make generate_po_from_tmpl -e TRANSLATIONI18N=fr
  

• translate your *.po file(s)

  You can do it with software like PoEdit but we may switch to Transifex, an hosted translation service to make contribution more easy, follow translation advance and have a review process for translation.

• compile *.po to *.mo (sphinx need *.mo file to make substitution from original version to the language translated)

  make compile_messages -e TRANSLATIONI18N=fr
  

• Generate doc for the language

You just need to set in the file Makefile or make.bat depending on your OS the following parameter before building the documentation:

TRANSLATIONI18N=fr

You need to change as well the scripts/build_docs.sh script to add your language at the top of the file:

LANGUAGES= fr

Please for both parameter, keep the alphabetical order to kept them readable.

Structure of the i18n files in the MapServer-documentation root directory

The structure you need to make all the transaltion process is like below (an excerpt to give you the main required structure)

translated/
├── fr
│   ├── about.po
│   ├── announcements_all.po
│   ├── cgi.po
│   ├── community.po
│   ├── copyright.po
│   ├── development.po
│   ├── documentation.po
│   └── LC_MESSAGES
│       ├── about.mo
│       ├── announcements_all.mo
│       ├── cgi.mo
│       ├── community.mo
│       ├── copyright.mo
│       ├── development.mo
│       └── documentation.mo
└── pot
    ├── about.pot
    ├── announcements_all.pot
    ├── cgi.pot
    ├── community.pot
    ├── copyright.pot
    ├── development.pot
    └── documentation.pot

Moreover, you only need to commit the *.pot and *.po files. Be cautious, you must have an LC_MESSAGES directory to receive *.mo files. To keep this directory in Git, it’s recommended to create and empty file into it.

Bemerkung

When adding a new language, take care of the ISO code used. Famfamfam flag icons is using ISO Country code, and transifex is using ISO Language code which can be different. See ISO country code page for more information.

The usual workflow for already existing translation

The previous process we explained is only for the language creation. After, the workflow consists of:

• translating po files for your language via transifex,

• pull po files from transifex and commit them

• generating the .mo files

• building again

Here are the command line:

• Getting po files from transifex (you can use -l flags to filter the language):

  tx pull -a
  

• Build mo files from po files:

  make compile_messages  -e TRANSLATIONI18N=fr
  

• Build html page:

  make clean html
  

See below how to setup your transifex account to use transifex client to push, pull translated files and build html files.

Sometimes, when a change occurs in the main english documentation, you will need to generate again the pot files. Then, you will have to merge your translated files with the new pot files using for example the command line utility (from gettext) below:

make gettext
make gettext_copy

-U mean update po file with pot file and -N mean don’t use fuzzy but exact match. Then push po files to transifex (you can use -l flags to filter the language):

tx push -s

When committing your update, add [build_pdf] or [build_translations] in order to build pdf and/or translation to the webserver.

Known issues:

• when downloading po files to local dir, if the file exists it won’t be updated. At this moment we should remove all po files before downloading.

• when building html doc cache system doesn’t allow to update a page with up to date translation. We should use the clean target with make.

• This command line throw an error:

  tx push -s -t
  

How to use Transifex client to manage translation process

Bemerkung

Transifex is described more in depth in the document: How to Help Translate the Documentation

• Install transifex client:

  sudo apt-get install transifex-client
  

• Edit ~/.transifexrc and add the appropriate info in there:

  vim ~/.transifexrc
  

  • With the following content:

    [https://www.transifex.com/]
    username = yjacolin
    token =
    password = passw0rd!
    hostname = https://www.transifex.com/
    

Reference Labels

:ref: reference labels

Label	Title
about	About
agg	AGG Rendering Specifics
antialias	AntiAliasing with MapServer
arcinfo	ArcInfo
arcsde	ArcSDE
autotest	Regression Testing
background	Tutorial Timeframe
batch_utilities	Batch Scripting
bugs	Bug Submission
cgi	CGI
cgi_controls	MapServer CGI Controls
cgi_introduction	MapServer CGI Introduction
class	CLASS
community	Community Activities
development	Development
dgn	DGN
documentation	MapServer 5.2.2 Documentation
documentation_development	Documentation Development Guide
dotnet_compile	.NET MapScript Compilation
download	Download
dynamic_charting	Dynamic Charting
editing	Mapfile Editing
errors	Errors
example1-1	Example 1.1
example1-1-map	Example1-1.map
example1-2	Example 1.2
example1-2-map	Example1-2.map
example1-3	Example 1.3
example1-3-map	Example1-3.map
example1-4	Example 1.4
example1-4-map	Example1-4.map
example1-5	Example 1.5
example1-5-map	Example1-5.map
example1-6	Example 1.6
example1-6-map	Example1-6.map
example1-7	Example 1.7
example1-7-map	Example1-7.map
example1-8	Example 1.8
example1-8-map	Example1-8.map
expressions	Expressions
faq	FAQ
fastcgi	FastCGI
feature	FEATURE
filter_encoding	WFS Filter Encoding
flash	Flash Output
fontset	FONTSET
format_types	Data Format Types
genindex	Stichwortverzeichnis
git	GitHub
gloss	Glossary
gml	GML
gpx	GPS Exchange Format (GPX)
grid	GRID
html_legend	HTML Legends with MapServer
iis	IIS Setup for MapServer
imagemaps	HTML Imagemaps
include	INCLUDE
inline	Inline
input	Data Input
input_postgis	PostGIS/PostgreSQL
installation	Installation
introduction	An Introduction to MapServer
join	JOIN
kml	KML - Keyhole Markup Language
label	LABEL
layer	LAYER
legend	LEGEND
legend_utility	legend
license	License
linux	Linux
lists	Mailing Lists
management	File Management
map	MAP
map2img	map2img
map_context	Map Context
mapcache_formats	MapCache Image Formats
mapcache_jpeg_format	MapCache JPEG Format
mapcache_png_format	MapCache PNG Format
mapcache_mixed_format	MapCache Mixed PNG+JPEG Format
mapcache_caches	MapCache Cache Backends
mapcache_grids	MapCache Tile Grids
mapcache_cache_sqlite	MapCache Sqlite Caches
mapcache_cache_disk	MapCache Disk Caches
mapcache_cache_memcache	MapCache Memcache Caches
mapcache_cache_tiff	MapCache TIFF Caches
mapcache_services	MapCache Tile Services
mapcache_ve	MapCache VirtualEarth Quadkey Tile Service
mapcache_tms	MapCache Tile Map Service
mapcache_kml	MapCache Super-Overlay KML Services
mapcache_wmts	MapCache OGC WMTS Service
mapcache_wms	MapCache OGC WMS Service
mapcache_gmaps	MapCache GoogleMaps Tile Service
mapcache_proxying	MapCache Proxying Support
mapcache_dimensions	MapCache Tileset Dimensions
mapcache_featureinfo	MapCache FeatureInfo Requests
mapcache_tile_assembling	MapCache Tile Assembling
mapcache_sources	MapCache Input Sources
mapcache_source_wms	MapCache WMS Source
mapcache_source_mapfile	MapCache Native MapServer Sources
mapcache_http	MapCache HTTP Request
mapcontext_cgi	Map Context Files
mapfile	Mapfile
mapfile_tuning	Mapfile
mapinfo	MapInfo
mapscript	MapScript
mapscript_ows	MapScript Wrappers for WxS Services
mapscript_tests	MapScript Unit Testing
mapserv	mapserv
matrix-irc	Matrix/IRC
modindex	Modulindex
msencrypt	msencrypt
mysql	MySQL
ntf	NTF
oci	Oracle Spatial
oci_install	Oracle Installation
ogc	OGC Support and Configuration
ogc_support	OGC Support
ogr	OGR
online_resource_wms	More About the Online Resource URL
optimization	Optimization
osx	Mac OS X
output	Output Generation
outputformat	OUTPUTFORMAT
pdf	PDF Output
pgeo	ESRI Personal Geodatabase (MDB)
php	PHP MapScript
php_example	By Example
php_install	PHP MapScript Installation
php_install_example_steps	Example Steps of a Full Windows Installation
projection	PROJECTION
psc	Project Steering Committee
python	Python MapScript Appendix
querying	Querying
querymap	QUERYMAP
raster	Raster Data
raster_optimization	Raster
reference	REFERENCE
rfc1	MS RFC 1: Technical Steering Committee Guidelines
rfc10	MS RFC 10: Joining the Open Source Geospatial Foundation
rfc11	MS RFC 11: Support for Curved Labels
rfc12	MS RFC 12: C code Unit tests
rfc13	MS RFC 13: Support of Sensor Observation Service in MapServer
rfc14	MS RFC 14: Relative Coordinates for INLINE features
rfc15	MS RFC 15: Support for thread neutral operation of MapServer/MapScript
rfc16	MS RFC 16: MapScript WxS Services
rfc17	MS RFC 17: Dynamic Allocation of layers, styles, classes and symbols
rfc18	MS RFC 18: Encryption of passwords in mapfiles
rfc19	MS RFC 19: Style & Label attribute binding
rfc2	MS RFC 2: Creating line features and/or shapes using WKT
rfc21	MS RFC 21: MapServer Raster Color Correction
rfc22a	MS RFC 22a: Feature cache for long running processes and query processing
rfc23	MS RFC 23: Technical Steering Committee Guidelines
rfc24	MS RFC 24: Mapscript memory management
rfc24first	MS RFC 24: Mapscript memory management
rfc25	MS RFC 25: Align MapServer pixel and extent models with OGC models
rfc26	MS RFC 26: Version 5 Terminology Cleanup
rfc27	MS RFC 27: Label Priority
rfc28	MS RFC 28: Redesign of LOG/DEBUG output mechanisms
rfc29	MS RFC 29: Dynamic Charting Capability
rfc3	MS RFC 3: Feature Layer Plug-in Architecture
rfc30	MS RFC 30: Support for WMS 1.3.0
rfc31	MS RFC 31: Loading MapServer Objects from Strings
rfc32	MS RFC 32: Support for Anti-Grain Geometry (AGG) Rendering Engine
rfc33	MS RFC 33: Removing msLayerWhichItems() from maplayer.c
rfc39	MS RFC 39: Support of WMS/SLD Named Styles
rfc4	MS RFC 4: MapServer Raster Resampling
rfc40	MS RFC 40: Support Label Text Transformations
rfc41	MS RFC 41: Support of WCS 1.1.x Protocol
rfc42	MS RFC 42: Support of Cookies Forwarding
rfc43	MS RFC 43: Direct tile generation for Google Maps and Virtual Earth API
rfc44	MS RFC 44: Restore URL modification of mapfiles to pre-5.0 levels
rfc45	MS RFC 45: Symbology, Labeling, and Cartography Improvements
rfc46	MS RFC 46: Migrate Website to OSGeo
rfc47	MS RFC 47: Move IGNORE_MISSING_DATA to run-time configuration
rfc48	MS RFC 48: GEOTRANSFORM Geometry operations
rfc49	MS RFC 49: Symbology, Labeling, and Cartography Improvements
rfc5	MS RFC 5: MapServer Horizon Reprojection Improvements
rfc50	MS RFC 50: OpenGL Rendering Support
rfc51	MS RFC 51: XML Mapfile Format
rfc53	MS RFC 53: Guidelines for MapScript method return values
rfc54	MS RFC 54: Rendering Interface API
rfc55	MS RFC 55: Improve control of output resolution
rfc56	MS RFC 56: Tighten control of access to mapfiles and templates
rfc6	MS RFC 6: Color Range Mapping of Continuous Feature Values
rfc7	MS RFC 7: MapServer CVS Commit Management
rfc7.1	MS RFC 7.1: MapServer SVN Commit Management
rfc8	MS RFC 8: Pluggable External Feature Layer Providers
rfc9	MS RFC 9: Item tag for query templates
rfcs	Request for Comments
runsub	Run-time Substitution
s57	S57
scalebar	SCALEBAR
scalebar_utility	scalebar
sdts	SDTS
search	Suche
section1	Section 1: Static Maps and the MapFile
section2	Section 2: CGI variables and the User Interface
shapefiles	ESRI Shapefiles (SHP)
shptree	shptree
shptreevis	shptreevis
sld	SLD
sortshp	sortshp
sos_server	SOS Server
source	Source
style	STYLE
styleitemauto-label-styles	Accessing OGR STYLEITEMAUTO Label Styles Through MapScript
svg	SVG
svn	Subversion
swig	SWIG MapScript API Reference
sym2img	sym2img
sym_construction	Cartographic Symbol Construction with MapServer
sym_examples	Symbology Examples
symbol	SYMBOL
template	Templating
testing	Testing
tiger	USGS TIGER
tile4ms	tile4ms
tile_mode	Tile Mode
tileindex	Tile Indexes
tutorial	MapServer Tutorial
unix	Compiling on Unix
utilities	Utilities
variable_sub	Variable Substitution
vector	Vector Data
vector_optimization	Vector
vim	VIM Syntax
virtual_vector	Virtual Spatial Data
wcs_format	WCS Use Cases
wcs_server	WCS Server
web	WEB
wfs	WFS
wfs_client	WFS Client
wfs_server	WFS Server
win32	Compiling on Win32
windows	Windows
wms_capabilities	Validate the Capabilities Metadata
wms_client	WMS Client
wms_server	WMS Server
wms_time	WMS Time
wrapper	A Simple CGI Wrapper Script

Regenerating the reference labels

You can regenerate the reference labels by issuing:

make labels

from the MapServer-documentation directory similar to when you are building the html or latex versions