Feature 8171: Upgrade of OSM services using helm on K8s

Starting with version 10.1.0 of OSM, every even numbered release will receive two years of community support.  This document covers the steps needed for upgrading OSM.  Depending on the installation method, there are two methods for upgrading OSM to an LTS version.

## How to Upgrade OSM 12.x to 14.y LTS

### Kubernetes Installation to 14.y

#### Back up the Databases <a name="k8s-14-db">

If desired, the databases can be backed up using the following commands:

```bash

mysql_pod=$(kubectl get pod -n osm | grep -i mysql | tail -1 | awk -F" " '{print $1}')

kubectl exec -n osm -it $mysql_pod -- bash -c \

     'mysqldump -u root -p$MYSQL_ROOT_PASSWORD --single-transaction --all-databases' \

     | gzip > backup.sql.gz

mongodb_unit=$(juju status | grep -i mongodb | tail -1 | awk -F" " '{print $1}'| tr -d '[*]')

mongodb_pod=$(kubectl get pod -n osm | grep -i mongodb | grep -v operator | tail -1 | awk -F" " '{print $1}')

juju run-action $mongodb_unit backup --wait -m osm

kubectl cp osm/$mongodb_pod:/data/backup.archive backup.archive

```

#### Upgrade Juju <a name="k8s-14-juju"></a>

The following commands will upgrade the OSM controller.

```bash

sudo snap refresh juju --channel 2.9/stable

juju upgrade-controller

```

Next, for any native or proxy charms, upgrade each model.

```bash

for model in $(juju models --format json | jq .models[].name | tr -d \") ; do 

    juju switch $model

    juju upgrade-model

done

```

#### Upgrade OSM Application <a name="k8s-14-osm">

First remove the old manifests and Kubernetes objects. Secrets will remain.

```bash

kubectl -n osm delete -f /etc/osm/docker/osm_pods/nbi.yaml

kubectl -n osm delete -f /etc/osm/docker/osm_pods/lcm.yaml

kubectl -n osm delete -f /etc/osm/docker/osm_pods/ro.yaml

kubectl -n osm delete -f /etc/osm/docker/osm_pods/grafana.yaml

kubectl -n osm delete -f /etc/osm/docker/osm_pods/ca_setup.yaml

kubectl -n osm delete -f /etc/osm/docker/osm_pods/zookeeper.yaml

kubectl -n osm delete -f /etc/osm/docker/osm_pods/kafka.yaml

kubectl -n osm delete -f /etc/osm/docker/osm_pods/mon.yaml

kubectl -n osm delete -f /etc/osm/docker/osm_pods/pol.yaml

kubectl -n osm delete -f /etc/osm/docker/osm_pods/keystone.yaml

kubectl -n osm delete -f /etc/osm/docker/osm_pods/mysql.yaml

kubectl -n osm delete -f /etc/osm/docker/osm_pods/prometheus.yaml

kubectl -n osm delete -f /etc/osm/docker/osm_pods/ng-ui.yaml

```

Then, update MongoDB using the charm:

```bash

# Build

sudo snap install charmcraft --classic

git clone https://osm.etsi.org/gerrit/osm/devops

cd devops/installers/charm/osm-update-db-operator

charmcraft pack

# Deploy

juju add-model update-db k8scloud

juju model-config default-series=kubernetes

juju deploy ./osm-update-db_ubuntu-20.04-amd64.charm

juju config osm-update-db mongodb-uri="mongodb://IP:27017"

juju run-action osm-update-db/0 update-db current-version=12 target-version=14 mongodb-only=True --wait

# Destroy model

juju destroy-model update-db --force -y

```

Create a new secret to be used by the OSM helm chart:

```bash

OSM_DATABASE_COMMONKEY=$(kubectl -n osm get secret/nbi-secret --template='{{.data.OSMNBI_DATABASE_COMMONKEY | base64decode}}')

OSM_SERVICE_PASSWORD=$(kubectl -n osm get secret/nbi-secret --template='{{.data.OSMNBI_AUTHENTICATION_SERVICE_PASSWORD | base64decode}}')

OSM_MYSQL_ROOT_PASSWORD=$(kubectl -n osm get secret/keystone-secret --template='{{.data.ROOT_DB_PASSWORD | base64decode}}')

OSM_KEYSTONE_DB_PASSWORD=$(kubectl -n osm get secret/keystone-secret --template='{{.data.KEYSTONE_DB_PASSWORD | base64decode}}')

echo "OSM_DATABASE_COMMONKEY=${OSM_DATABASE_COMMONKEY}" | sudo tee -a osm.env

echo "OSM_SERVICE_PASSWORD=${OSM_SERVICE_PASSWORD}" | sudo tee -a osm.env

echo "OSM_MYSQL_ROOT_PASSWORD=${OSM_MYSQL_ROOT_PASSWORD}" | sudo tee -a osm.env

echo "OSM_KEYSTONE_DB_PASSWORD=${OSM_KEYSTONE_DB_PASSWORD}" | sudo tee -a osm.env

kubectl -n osm create secret generic osm-secret --from-env-file=osm.env

```

Finally, deploy OSM with the helm chart:

```

git clone "https://osm.etsi.org/gerrit/osm/devops"

cd devops

OSM_VERSION="14.0.1"

git checkout $OSM_VERSION

# Add your own helm options (--set ...)

OSM_HELM_OPTS=""

# Check that there are no errors in the manifests

helm -n osm template osm ./installers/helm/osm ${OSM_HELM_OPTS}

# Deploy

helm -n osm install osm ./installers/helm/osm ${OSM_HELM_OPTS}

helm -n osm status osm

```

At this point, OSM has been upgraded.

## How to Upgrade OSM 14.x to OSM 14.y LTS in a helm-based installation

### Back up the Databases <a name="k8s-14-helm-db">

If desired, the databases can be backed up using the following commands:

```bash

mysql_pod=$(kubectl get pod -n osm | grep -i mysql | tail -1 | awk -F" " '{print $1}')

kubectl exec -n osm -it $mysql_pod -- bash -c \

     'mysqldump -u root -p$MYSQL_ROOT_PASSWORD --single-transaction --all-databases' \

     | gzip > backup.sql.gz

mongodb_unit=$(juju status | grep -i mongodb | tail -1 | awk -F" " '{print $1}'| tr -d '[*]')

mongodb_pod=$(kubectl get pod -n osm | grep -i mongodb | grep -v operator | tail -1 | awk -F" " '{print $1}')

juju run-action $mongodb_unit backup --wait -m osm

kubectl cp osm/$mongodb_pod:/data/backup.archive backup.archive

```

### Upgrade OSM Application <a name="k8s-14-helm-osm">

```

git clone "https://osm.etsi.org/gerrit/osm/devops"

cd devops

DESIRED_OSM_VERSION="14.0.0"

git checkout $DESIRED_OSM_VERSION

# Get the current values.yaml

helm -n osm get values osm > myvalues.yaml

# Compare current values.yaml with new values.yaml, and edit values.yaml conveniently

# diff myvalues.yaml installers/helm/osm/values.yaml

# Add your own helm options (--set ...)

# OSM_HELM_OPTS=""

# OSM_HELM_OPTS="-f myvalues.yaml"

# Check that there are no errors in the manifests

helm -n osm template osm ./installers/helm/osm ${OSM_HELM_OPTS}

# Upgrade OSM

helm -n osm upgrade osm ./installers/helm/osm ${OSM_HELM_OPTS}

helm -n osm status osm

```

At this point, OSM has been upgraded.

## How to Upgrade OSM 10.1.1 to 12.x LTS

### Kubernetes Installation to 12.x

#### Back up the Databases <a name="k8s-12-db">

If desired, the databases can be backed up using the following commands:

```bash

mysql_pod=$(kubectl get pod -n osm | grep -i mysql | tail -1 | awk -F" " '{print $1}')

kubectl exec -n osm -it $mysql_pod -- bash -c \

     'mysqldump -u root -p$MYSQL_ROOT_PASSWORD --single-transaction --all-databases' \

     | gzip > backup.sql.gz

mongodb_unit=$(juju status | grep -i mongodb | tail -1 | awk -F" " '{print $1}'| tr -d '[*]')

mongodb_pod=$(kubectl get pod -n osm | grep -i mongodb | grep -v operator | tail -1 | awk -F" " '{print $1}')

juju run-action $mongodb_unit backup --wait -m osm

kubectl cp osm/$mongodb_pod:/data/backup.archive backup.archive

```

#### Upgrade Juju <a name="k8s-12-juju"></a>

The following commands will upgrade the OSM controller.

```bash

sudo snap refresh juju --channel 2.9/stable

juju upgrade-controller

```

Next, for any native or proxy charms, upgrade each model.

```bash

for model in $(juju models --format json | jq .models[].name | tr -d \") ; do 

    juju switch $model

    juju upgrade-model

done

```

#### Upgrade OSM Application <a name="k8s-12-osm">

```bash

OSM_VERSION="12.0.6"

for module in lcm mon nbi ng-ui pla pol ro; do

    kubectl -n osm patch deployment ${module} --patch '{"spec": {"template": {"spec": {"containers": [{"name": "${module}", "image": "opensourcemano/${module}:${OSM_VERSION}"}]}}}}'

    kubectl -n osm scale deployment ${module} --replicas=0

    kubectl -n osm scale deployment ${module} --replicas=1

done

# In order to make this change persistent after reboots,

# you will have to update the files under /etc/osm/docker/osm_pods to reflect the changes

for module in lcm mon nbi ng-ui pol ro prometheus; do

    sudo sed -i "s/opensourcemano\/${module}:.*/opensourcemano\/${module}:${OSM_VERSION}/g" /etc/osm/docker/osm_pods/${module}.yaml

done

sudo sed -i "s/opensourcemano\/pla:.*/opensourcemano\/pla:${OSM_VERSION}/g" /etc/osm/docker/osm_pods/osm_pla/${module}.yaml

```

At this point, OSM has been upgraded.

## Upgrade of 10.1.0 to 10.1.1 LTS

This procedure covers both the upgrade of 10.1.0 to 10.1.1 LTS.  There are two installation methods, each with its own set of procedures: