Workflow for documentation production in OSM.md

# Workflow for documentation production in OSM

## Introduction

The process is based on:

- Text-based edition with Markdown
- Further conversion to multiple formats and templates with Pandoc
- Version control with Git, using ETSI's Gitlab

In the edition process there are two roles:

- **Contributors.** Anyone contributing content to a given document, with no special privileges. They cannot contribute directly to `master` branch, so they have to contribute pushing branches and issuing a _merge request_ (i.e. requesting their integration).
- **Editor/Reviewer**. The person (or people) in charge of integrating (_merging_) contributions into the main document. In consequence, they can push changes directly to the `master` branch. They should also decide the document split in different files and provide the means to build and convert the document to a given set of target formats and templates.

## Software requirements

## Contributor

Basic Software (available on Windows, Mac. and Linux):

- [Git](https://git-scm.com/)
- [Pandoc](https://pandoc.org/)
- Markdown editor(s). Recommended (both):
  - Integrated: [Visual Studio Code](https://code.visualstudio.com/)
    - The integrated editor supports version control operations graphically, so that usual operations do not require CLI commands necessarily.
  - WYSIWYG: [Typora](https://typora.io/)

### Some recommended extensions for Visual Studio Code

- `bat67.markdown-extension-pack`
- `yzhang.markdown-all-in-one`
- `DavidAnson.vscode-markdownlint`
- `satokaz.vscode-markdown-header-coloring`
- `wayou.vscode-todo-highlight`
- `streetsidesoftware.code-spell-checker`
- `bierner.github-markdown-preview`
- `shd101wyy.markdown-preview-enhanced`
- `geeklearningio.graphviz-markdown-preview`
- `eamodio.gitlens`
- `donjayamanne.githistory`
- `bierner.markdown-checkbox`
- `bierner.markdown-emoji`
- `bierner.markdown-mermaid`
- `bierner.markdown-preview-github-styles`
- `bierner.markdown-yaml-preamble`
- `mrmlnc.vscode-remark`
- `csholmq.excel-to-markdown-table`
- `DougFinke.vscode-pandoc`
- `fcrespo82.markdown-table-formatter`
- `yzane.markdown-pdf`

To install extensions in VSCode you can follow any of the procedures described in: https://code.visualstudio.com/docs/editor/extension-gallery

## Markdown conversion basics

The basic syntax of Pandoc is rather simple. In the simplest cases, it just requires an input file and an output file name, and it will determine the formats based on the extensions.

For instance, this command converts from Markdown to MS-Word format:

```bash
pandoc MyDocument.md -o MyDocument.docx
```

It is also possible to merge several input files into one output file:

```bash
pandoc MyDocument1.md MyDocument2.md -o MyDocument.docx
```

For some target formats, such as HTML, you would need to specify whether you want to produce a fragment (default) or a complete standalone document (using the `-s` switch):

```bash
pandoc -s MyDocument.md -o MyDocument.html
```

In the case of MS-Word target format, a highly useful feature is specifying an existing document (usually based on a specific format template) as reference, so that Pandoc can produce the output document based on the same template:

```bash
pandoc MyDocument.md --reference-doc=referenceDocument.docx -o MyDocument.docx
```

Please note that the reference must be a valid standalone DOC/DOCX document, not an MS-Word template (i.e. DOT/DOTX files are not valid references for Pandoc).

## Git settings

Reference: https://osm.etsi.org/gitlab/osm_doc/test#repo-command-line-instructions

```bash
git config --global user.name "myEOLusername"
git config --global user.email "myusername@mycompany.com"
```

It is also highly advisable adding an SSH key to https://osm.etsi.org/gitlab/profile/keys so that password is not requested in every operation.

## Guide for contributors

### Step 0: Setup the repository in your local environment for the first time

#### If the repository already exists in ETSI's Gitlab but not locally

```bash
git clone ssh://git@osm.etsi.org:29419/osm-doc/test.git
```

**>>>>> TEMPORARY URL**:

```bash
git clone https://osm.etsi.org/gitlab/osm-doc/test.git
```

#### If the repository already exists in ETSI's Gitlab and the local copy is not up-to-date

```bash
git pull origin master
```

### Step 1 (**IMPORTANT)**: Create a local branch

This should be based on an up-to-date version of the code in the repo's `master`. If you are unsure, you should do:

```bash
git pull origin master
```

Then create a branch and move to it in order to continue. In this example, we will create a branch called `BranchNewSection` to store a new section that you are working on:

```bash
git checkout -b BranchNewSection
```

#### NOTE: What to do if you committed changes to `master` by mistake

In case you forgot to create your branch (or moving to it) and made a number of commits directly to your `master` branch by mistake, you will need to use a procedure to move them to a branch, or you will be unable to work with you remote origin.

To achieve it, you can still following some steps to fix your Git history and move the commits to the right place.

Note that **this procedure needs to be followed exactly as it is described** or might lead to unintended loss of commits. In particular, you should understand that:

- You are **rewriting** your Git history and, therefore, might delete wrong entries if you make any mistake. Please read these steps carefully and follow them accurately.
- Any changes not committed before starting this procedure. If they are relevant for your, you should commit them (to `master`) as well.

As said, you can follow this procedure (assuming that you start in your `master` branch):

```bash
git branch newbranch              # Create a new branch, saving the desired commits
git reset --hard origin/master    # Move master back to the latest commit common to the remote
git checkout newbranch            # Go to the new branch that still has the desired commits
```

### Step 2: Local edition and commits

In this phase, you can edit your files as you would normally do.

From time to time, you might want to "save" snapshots of your changes. This process has two states:

- Putting files with modifications in your _stage area_.
- Once you have all you need in your _stage area_, create a local _commit_ out of this set of changes.

You can add those files to your _stage area_ as needed. For instance, this would add a file called `MyFile.md`:

```bash
git add MyFile.md
```

In case you wanted to add all files with any modification since the last commit, you could do:

```bash
git add .
```

Once you are ready, you can create a commit (which would serve as a kind of local "snapshot" of your changes:

```bash
git commit -m "My message to remember later on what was included in this commit"
```

Then you can continue editing until you are ready to share a contribution.

### Step 3: Contribute your branch and make a merge request

When your are ready, you can push your branch to the remote repo:

```bash
git push origin BranchNewSection
```

Once you have pushed your branch to the remote repo, you should **inform the editor of it by making a _merge request_**. This can be easily made in the GitLab web, in the _Merge Requests_ area (on the left side bar). Your recently pushed branch would be there to be selected for your new merge request.

When creating the merge request, **select _Delete source branch when merge request accepted_ option** so that the source branch is deleted when the merge request is merged.

### Step 4: Result of the review

Following your _merge request_, the editor will revise the changes suggested in your branch and decide whether they should be merged or receive comments from the editor.

If your contribution was **commented** by the editor, you should try to address the comments in your branch and push them again if appropriate (in case the contribution has not been entirely rejected by the editor).

If your contribution was integrated (**merged**) by the editor, the result will be available in the main edition branch (i.e. `master`) as a new commit . Therefore, you should update your local repository (`pull`) before attempting further edits:

```bash
git checkout master
git pull origin master
```

## Guide for editors

### Initial setup of the repository

#### If you start from a local folder and the repo in ETSI's Gitlab is still empty (initial load of info)

```bash
cd existing_folder
git init
git remote add origin ssh://git@osm.etsi.org:29419/osm-doc/test.git
git add .
git commit -m "Initial commit"
git push -u origin master
```

#### If the repository in ETSI's Gitlab already exists and may have valuable information

Check the corresponding sections of the [Guide for contributors](#step-0-setup-the-repository-in-your-local-environment-for-the-first-time)

### Edition and processing of open merge requests

The editor can view all the pending merge requests within a project by navigating to **Project > Merge Requests** in the Gitlab web of the project for the document. That merge request would be associated to a new branch, which can be also be checked navigating to **Repository > Branches**.

For instance, for a group of committers (Group) called `osm-doc` and a project called `documentation-how-to`:

- The list of pending Merge Requests would be at `https://osm.etsi.org/gitlab/osm-doc/documentation-how-to/merge_requests`
- The list of branches would be available at `https://osm.etsi.org/gitlab/osm-doc/documentation-how-to/branches`

TODO: Include some screenshots

First online review and, if needs to be rejected, include comments where applicable.

If the contribution is acceptable for a merge, the merge process can be conducted either in the website (particularly if it is a simple review) or in the editor's local repo. The following steps describe the most generic case of using the local repo, although the process would be analogous in the web interface.

First, update your local repository to retrieve the branch(es) with the pending _merge request(s)_:

```bash
git pull --all
```

Then, launch the merge operation:

```bash
git checkout master
git merge branchwithcontribution
```

If completed successfully, mark the merge request as approved in the website (**Project > Merge Requests**) and remove the branch if asked (this will clean up the Git tree in GitLab by removing unnecessary temporary branches).

If applicable, update the local repository with a `git pull`

If the merge does not imply conflicts with `master` branch, it will be smoothly solved by Git (as a _fast forward_ or as a _recursive_ merge) and properly integrated into `master`.

However, sometimes there might be colliding text lines between `master` and the contribution in the branch. In those cases, Git will request some manual intervention from the editor to proceed to a more careful merge.

#### How to resolve merge conflicts

When there is a _merge conflict_, Git leaves a special edited version of the conflicting file in the local working directory, which will include both conflicting proposals for a given line, one after the other, leaving the final result at editor's discretion.

Once solved by the editor (manually), the file can be added, committed and pushed again as normal:

```bash
git add conflictingfile.md
git commit
git push -u origin master
```

However, if eventually you were doubtful of your attempt to address the conflicts with the editor, it is always possible to revert the merge process (so it can be restarted later again) with:

```bash
git merge --abort
```

### Some recommendations for editors

#### Markdown style

In spite of the fact that there is a high convergence around Markdown, some of the most advanced features of the language are not that normalized and may differ slightly among different Markdown flavours.

In order to minimize potential issues due to ambiguity of format among contributors, in case of doubt, **GitHub Flavored Markdown (GFM)** is highly recommended, given its prevalent popularity and support by most major editors, renderers and viewers.

#### File structure

It is highly advisable that the editor provides a base file structure for the document. There are some best practices that apply:

- Different chapters (i.e. level-1 titles) of the document should be in different markdown files. Likewise, no more than one level-1 title should be placed in the same file.
- Images should be placed in a dedicated folder. The recommendation is using the `./assets` folder.
- Likewise, DOCX templates, auxiliary files, output files, etc. should be placed in dedicated folders
- File and folder names should not contain spaces or any other kind of special characters to maximize portability and minimize rendering or reading errors in various auxiliary tools. To improve readability, an useful convention is using dashes (`-`) to replace spaces.
- The editor should add to the base folder a simple **script to build the document**, so there is no ambiguity to test the results.

TODO: Include example with all recommendations above, so that it can be easily used as template for new editors.

### Line length limitation and minimization of conflicts

A useful peculiarity of Markdown is that any paragraph of the final document can expressed using multiple Markdown lines, since a single carriage return does not create a new paragraph (2 carriage returns are needed to separate paragraphs). This property, among other reasons, is intended to improve readability in some environments.

Therefore, while the more natural and simplest way to produce and edit markdown files in a Git context can be producing the text as with any other text editor (leading to a single markdown line per paragraph), this singularity of Markdown syntax can be used in editor's favour for files, sections or paragraphs that might be subject to multiple edits and revisions. Thus, in those cases, it would be highly convenient for the editor to split critical paragraphs into lines equal or shorter than maximum length (typically, **79 characters**), so that merge conflicts between contributors are minimized.

If eventually a whole file were in that situation, `pandoc` can provide a shortcut for automatic conversion to the editor:

```bash
pandoc busychapter.md -t gfm --columns=79 -o busychapter-wrapped.md
```

And then, replace the original file by the new one, so that new contributors can take it as a reference:

```bash
cp busychapter-wrapped.md busychapter.md
rm busychapter-wrapped.md
git add busychapter.md
git commit -m "File busychapter.md wrapped to 79 characters per line"
git push origin master
```

Whenever the editor realises that the document is stable again or if prefers an edition based on lines that equal paragraphs, the previous rewriting can be easily reverted with Pandoc:

```bash
pandoc busychapter.md -t gfm --wrap=none -o busychapter-nowrap.md
```

And then:

```bash
cp busychapter-nowrap.md busychapter.md
rm busychapter-nowrap.md
git add busychapter.md
git commit -m "File busychapter.md now allows unlimited line lengths"
git push origin master
```

**WARNING:** Please make sure that **any of these commits with massive changes in the line wrapping convention is properly announced to contributors**, so that they do not create new contributions based on the former line convention (in branches derived from older commits), or they would be really challenging to handle as merge conflicts (**all the converted lines would be different!**).