Merged K8s contribution by 'vicens'

### Adding additional parameters

Since release SIX, additional user parameters can be added, and they land at `vdu:cloud-init` (Jinja2 format) and/or `vnf-configuration` primitives (enclosed by `<>`). Here is an example of a VNF descriptor that uses two parameters called `touch_filename` and `touch_filename2`.

Since OSM Release SIX, additional user parameters can be added, and they land at `vdu:cloud-init` (Jinja2 format) and/or `vnf-configuration` primitives (enclosed by `<>`). Here is an example of a VNF descriptor that uses two parameters called `touch_filename` and `touch_filename2`.

```yaml

vnfd:

There are two types of charms:

- Native charms: the set of scripts run inside the VNF components. This kind of charms are new in Release 7.

- Proxy charms: the set of scripts run in LXC containers in an OSM-managed machine (which could be where OSM resides), which use ssh or other methods to get into the VNF instances and configure them.

- **Native charms:** the set of scripts run inside the VNF components. This kind of charms are new in Release 7.

- **Proxy charms:** the set of scripts run in LXC containers in an OSM-managed machine (which could be where OSM resides), which use ssh or other methods to get into the VNF instances and configure them.

![OSM Proxy Charms](assets/800px-OSM_proxycharms.png)

            nsd-connection-point-ref: nsd_cp_data

```

 The YAML above contains 2 *netslice-subnet*, one with the flag *is-shared-nss* as true and the other one with the flag *is-shared-nss* as false. The *netslice-vlds* will connect the *slice_hackfest_middle_nsd* nss with management interface and data2 with the *slice_hackfest_nsd* via *nsd_cp_data*

 The YAML above contains 2 `netslice-subnet`, one with the flag `is-shared-nss` as true and the other one with the flag `is-shared-nss` as false. The `netslice-vlds` will connect the `slice_hackfest_middle_nsd` nss with management interface and data2 with the `slice_hackfest_nsd` via `nsd_cp_data`.

To instantiate this network slice, we will use the same command used previously but changing the `nst_name` to `slice_hackfest2_nst`:

## Using Kubernetes-based VNFs (KNFs)

From Release SEVEN, OSM supports Kubernetes-based VNF (KNF). In order to test it, you require a K8s cluster connected to a network in the VIM (e.g. `vim-net`).

From Release SEVEN, OSM supports Kubernetes-based Network Functions (KNF). This feature unlocks more than 20.000 packages that can be deployed besides VNFs and PNFs. This section guides you to deploy your first KNF, from the installation of multiple ways of Kubernetes clusters until the selection of the package and deployment.

### Kubernetes installation

KNFs feature requires an operative Kubernetes cluster. There are several ways to have that Kubernetes running. From the OSM perpective, the Kubernetes cluster is not an isolated element, but it is a technology that enables the deployment of microservices in a cloud-native way. To handle the networks and facilitate the conection to the infrastructure, the cluster have to be associated to a VIM. There is an special case where the Kubernetes cluster is installed in a baremetal environment without the management of the networking part but in general, OSM consider that the Kubernetes cluster is located in a VIM.

For OSM you can use one of these three different ways to install your Kubernetes cluster:

1. [OSM Kubernetes cluster Network Service](15-k8s-installation.md#installation-method-1-osm-kubernetes-cluster-from-an-osm-network-service)

2. [Self-managed Kubernetes cluster in a VIM](15-k8s-installation.md#installation-method-2-local-development-environment)

3. [Kubernetes baremetal installation](15-k8s-installation.md#method-3-manual-cluster-installation-steps-for-ubuntu)

### OSM Kubernetes requirements

After the Kubernetes installation is completed, you need to check if you have the following components in your cluster.

1. [Kubernetes Loadbalancer](15-k8s-installation.md): to expose your KNFs to the network

2. [Kubernetes default Storageclass](15-k8s-installation.md): to support persistent volumes.

### Adding kubernetes cluster to OSM

In order to test Kubernetes-based VNF (KNF), you require a K8s cluster connected to a network in the VIM (e.g. `vim-net`). If you have a baremetal installation of Kubernetes, you will need to add a VIM in order to add the Kubernetes cluster.

You will have to add the K8s cluster to OSM. For that purpose, you can use these instructions:

osm k8scluster-show cluster

```

Then, you might need to add some repos from where to download helm charts required by the KNF:

The options used to add the cluster are the following:

- `--creds`: Is the location of the kubeconfig file where you have the cluster credentials

- `--version`: Current version of your Kubernetes cluster

- `--vim`: The name of the VIM where the Kubernetes cluster is deployed

- `--description`: Give a description to your Kubernetes cluster

- `--k8s-nets`: It is a dictionary of the cluster network, where the `key` is an arbitrary name and the `value` of the dictionary is the name of the network in the VIM. In case your k8s cluster is not located in a VIM, you could use '{net1: null}'

## Adding repositories to OSM

You might need to add some repos from where to download helm charts required by the KNF:

```bash

osm repo-add --type helm-chart --description "Bitnami repo" bitnami https://charts.bitnami.com/bitnami

osm repo-show bitnami

```

Once done, you can work with KNF in the same way as you do with any VNF. You can onboard them. For instance, you can use the example below of a KNF consisting of a single Kubernetes deployment unit based on OpenLDAP helm chart.

## KNF Service on-boarding

KNFs can be on-boarded using Helm Charts or Juju Bundles. In the following section is shown an example with Helm Chart and for Juju Bundles.

### KNF Helm Chart

Once the cluster is attached to your OSM, you can work with KNF in the same way as you do with any VNF. You can onboard them. For instance, you can use the example below of a KNF consisting of a single Kubernetes deployment unit based on OpenLDAP helm chart.

```bash

wget http://osm-download.etsi.org/ftp/Packages/hackfests/openldap_knf.tar.gz

Check in the cluster that pods are properly created:

- The pods associated to ldap should be using version openldap:1.2.1 and have 1 replica

- The pods associated to ldap2 should be using version openldap:1.2.1 and have 2 replicas

- The pods associated to ldap should be using version `openldap:1.2.1` and have 1 replica

- The pods associated to ldap2 should be using version `openldap:1.2.1` and have 2 replicas

Now you can upgrade both NS instances:

#Delete cluster

osm k8scluster-delete cluster

```

## KNF juju-bundle

This is an example on how to onboard a service that use a juju-bundle. For this example the service to be onboarded is a mediawiki that is comprised by a mariadb-k8s database and a mediawiki-k8s frontend.

```bash

wget http://osm-download.etsi.org/ftp/Packages/hackfests/mediawiki_cnf.tar.gz

wget http://osm-download.etsi.org/ftp/Packages/hackfests/mediawiki_cnf_ns.tar.gz

osm nfpkg-create mediawiki_cnf.tar.gz

osm nspkg-create mediawiki_cnf_ns.tar.gz

```

You can instantiate the Network Service:

```bash

osm ns-create --ns_name hf-k8s --nsd_name ubuntu-cnf-ns --vim_account <VIM_NAME|VIM_ID>

```

To check the status of the deployment you can run the following command:

```bash

osm ns-op-list hf-k8s

+--------------------------------------+-------------+-------------+-----------+---------------------+--------+

| id                                   | operation   | action_name | status    | date                | detail |

+--------------------------------------+-------------+-------------+-----------+---------------------+--------+

| 364c1378-ba86-447e-ad00-93fc1bf1bdd5 | instantiate | N/A         | COMPLETED | 2020-02-24T13:49:03 | -      |

+--------------------------------------+-------------+-------------+-----------+---------------------+--------+

```

To remove the network service you can:

```bash

osm ns-delete hf-k8s

```

# ANNEX 7 Kubernetes installation and requirements

This section illustrates a safe procedure to setup a Kubernetes cluster that meets the requirements described in [chapter 5](05-osm-usage.md#osm-kubernetes-requirements). Please note that there might be many alternative ways to achieve the same result (i.e. create an equivalent K8s cluster), so, in case you are using different tooling to create your K8s cluster, this annex should be taken just as informative information and refer instead to your tool's guide to the authoritative reference to achieve equivalent results.

There are two modes to represent a K8s cluster in OSM.

1. Inside a VIM (single and multinet):

   ![k8s-in-vim-multinet](assets/800px-k8s-in-vim-multinet.png)

   ![k8s-in-vim-singlenet](assets/800px-k8s-in-vim-singlenet.png)

2. Outside a VIM:

   ![k8s-out-vim](assets/800px-k8s-out-vim.png)

Your Kubernetes cluster needs to meet the following requirements:

1. **Kubernetes `Loadbalancer`:** to expose your KNFs to the network

2. **Kubernetes default `Storageclass`:** to support persistent volumes.

3. Tiller permissions for K8s clusters > v1.15

Here we will analyse three methods to create a Kubernetes cluster:

1. OSM Kubernetes cluster created as an OSM's Network Service (i.e. an OSM-managed Kubernetes deployment).

2. Local development environment based on MicroK8s.

3. Manual cluster installation based on `kubeadm`.

## Installation method 1: OSM Kubernetes cluster from an OSM Network Service

TODO: VNF and NS Packages to be made available soon.

## Installation method 2: Local development environment

Microk8s is a single-package fully conformant lightweight Kubernetes that works on 42 flavours of Linux. Perfect for developer workstations, IoT, edge, and CI/CD.

Using Microk8s as a Kubernetes cluster in OSM is straightforward.

First, install Microk8s with the following commands:

```bash

$ sudo snap install microk8s  --classic

microk8s v1.17.2 from Canonical✓ installed

$ sudo usermod -a -G microk8s `whoami`

$ newgrp microk8s

$ microk8s.status --wait-ready

microk8s is running

addons:

cilium: disabled

dashboard: disabled

dns: disabled

fluentd: disabled

gpu: disabled

helm3: disabled

helm: disabled

ingress: disabled

istio: disabled

jaeger: disabled

juju: disabled

knative: disabled

kubeflow: disabled

linkerd: disabled

metallb: disabled

metrics-server: disabled

prometheus: disabled

rbac: disabled

registry: disabled

storage: disabled

```

Microk8s uses [addons](https://microk8s.io/docs/addons) to extend its functionality. The required addons for Microk8s to work with OSM are "storage" and "dns".

```bash

$ microk8s.enable storage dns

Enabling default storage class

[...]

microk8s-hostpath created

Storage will be available soon

Enabling DNS

[...]

DNS is enabled

```

You may want to use the `metallb` addon if your Microk8s is not running in the same machine as OSM. When OSM adds a K8s cluster, it initializes the cluster so it can deploy Juju and Helm workloads on it. In the Juju initialization process, a controller will be bootstrapped on the K8s cluster, which then will be accessible by the Juju client (N2VC). When the K8s cluster is external to the OSM host machine, it must give the Juju controller an external IP accessible from OSM.

Just execute the following command and specify the IP range allocable by the load balancer.

```bash

$ microk8s.enable metallb

Enabling MetalLB

Enter the IP address range (e.g., 10.64.140.43-10.64.140.49): 192.168.0.10-192.168.0.25

[...]

MetalLB is enabled

```

Export the Microk8s configuration and add it as a K8s cluster to OSM:

```bash

microk8s.config > kubeconfig.yaml

osm k8scluster-add --creds kubeconfig.yaml \

                   --version '1.17' \

                   --vim openstack \

                   --description "My K8s cluster" \

                   --k8s-nets '{"net1": "osm-ext"}' \

                   microk8s-cluster

```

## Method 3: Manual cluster installation steps for Ubuntu

For the manual installation of Kubernetes cluster we will use a procedure based on `kubeadm`.

Getting the Docker gpg key to install docker:

```bash

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

```

Add the Docker Ubuntu repository:

```bash

sudo add-apt-repository    "deb [arch=amd64] https://download.docker.com/linux/ubuntu \

   $(lsb_release -cs) \

   stable"

```

Get the Kubernetes gpg key:

```bash

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

```

Add the Kubernetes repository:

```bash

cat << EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list

deb https://apt.kubernetes.io/ kubernetes-xenial main

EOF

```

Update your packages:

```bash

sudo apt-get update

```

Install `docker`, `kubelet`, `kubeadm`, and `kubectl`:

```bash

sudo apt-get install -y docker kubelet kubeadm kubectl

```

Hold them at the current version:

```bash

sudo apt-mark hold docker-ce kubelet kubeadm kubectl

```

Initialize the cluster (run only on the master):

```bash

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

```

​Set up local `kubeconfig`:

```bash

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

```

Apply `Flannel` CNI network overlay:

```bash

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

```

[OPTIONAL] Join the worker nodes to the cluster:

```bash

kubeadm join [your unique string from the kubeadm init command]

```

Verify the worker nodes have joined the cluster successfully:

```bash

kubectl get nodes

```

Compare this result of the `kubectl get nodes` command:

```bash

NAME                            STATUS   ROLES    AGE   VERSION

node1.osm.etsi.org   Ready    master   4m18s v1.13.5

```

If you have an all-in-one node, then you may want to schedule pods in the master. You need to untaint the master to allow that.

Untaint Master:

```bash

kubectl taint nodes --all node-role.kubernetes.io/master-

```

After the creation of your cluster, you may need to fulfil the requirements of OSM. We can start with the installation of a load balancer for your cluster.

`Metallb` is a very powerful, easy to configure, load balancer for kubernetes. To install it in your cluster, you can apply the following k8s manifest:

```bash

kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.8.3/manifests/metallb.yaml

```

The configuration of metallb in layer2 is via `Configmap`.

```yaml

apiVersion: v1

kind: ConfigMap

metadata:

  namespace: metallb-system

  name: config

data:

  config: |

    address-pools:

    - name: default

      protocol: layer2

      addresses:

      - 172.21.248.10-172.21.248.250

```

After the creation of config.yaml we need to apply it to Kubernetes cluster.

```bash

kubectl apply -f config.yaml

```

You should ensure that the range of IP address defined in `metallb` are accessible from outside the cluster and is not overlapped with other devices in that network. Also this network should be reachable from OSM since OSM will need it to communicate with the cluster.

Other configuration you need for your kubernetes cluster is the creation of the default `storageclass`:

A kubernetes persistent volume storage can be installed to your kubernetes cluster applying the following manifest.

```bash

kubectl apply -f https://openebs.github.io/charts/openebs-operator-1.6.0.yaml

```

After the installation, you need to check if there is a default `storageclass` in your kubernetes:

```bash

kubectl get storageclass

NAME                         PROVISIONER                                                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE

openebs-device               openebs.io/local                                           Delete          WaitForFirstConsumer   false                  5m47s

openebs-hostpath             openebs.io/local                                           Delete          WaitForFirstConsumer   false                  5m47s

openebs-jiva-default         openebs.io/provisioner-iscsi                               Delete          Immediate              false                  5m48s

openebs-snapshot-promoter    volumesnapshot.external-storage.k8s.io/snapshot-promoter   Delete          Immediate              false                  5m47s

```

Until now, there is not default `storageclass` defined. With the command below we will define `openebs-hostpath` as default storageclass:

```bash

ubuntu@k8s:~$ kubectl patch storageclass openebs-hostpath -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

```

To check the right application of the `storageclass` definition, we can use the following command:

```bash

kubectl get storageclass

NAME                         PROVISIONER                                                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE

openebs-device               openebs.io/local                                           Delete          WaitForFirstConsumer   false                  5m47s

openebs-hostpath (default)   openebs.io/local                                           Delete          WaitForFirstConsumer   false                  5m47s

openebs-jiva-default         openebs.io/provisioner-iscsi                               Delete          Immediate              false                  5m48s

openebs-snapshot-promoter    volumesnapshot.external-storage.k8s.io/snapshot-promoter   Delete          Immediate              false                  5m47s

```

For Kubernetes clusters > 1.15 there is needed special permission of Tiller that can be added by the following command:

```bash

kubectl create clusterrolebinding tiller-cluster-admin --clusterrole=cluster-admin  --serviceaccount=kube-system:default

```