Update index and annexes, integrate subscription and notification in main section

```bash

osm ns-delete hf-k8s

```

## Subscription and Notification support in OSM

[ETSI NFV SOL005](https://www.etsi.org/deliver/etsi_gs/NFV-SOL/001_099/005/02.04.01_60/gs_NFV-SOL005v020401p.pdf) defines a class of Northbound APIs through which entities can subscribe for changes in the Network Service (NS) life-cycle, Network service descriptor (NSD) and Virtual network service descriptor (VNFD). The entities get notified via HTTP REST APIs which those entities expose.

__Since the current support is only restricted to NS, while NSD and VNFD are in roadmap, for here onwards we will refer subscription and notification for NS.__

- The entities which are interested to know the life-cycle changes of network service are called Subscribers.

- Subscribers receive messages called notifications when an event of their interest occurs.

- SOL005 specifies usage of filters in the registration phase, through which subscribers can select events and NS they are interested in.

- Subscribers can choose the authentication mechanism of their notification receiver endpoint.

- Events need to be notified with very little latency and make them near real-time.

- Deregistration of subscription should be possible however subscribers can not modify existing subscriptions as per SOL005.

### NS Subscription And Notification

#### Steps for subscription

__Step 1: Get bearer token.__

    NBI API: https://<osm_ip>:9999/osm/admin/v1/tokens/

    Sample payload

    {

        "username": "admin",

        "password": "admin",

        "project": "admin"

    }

__Step 2: Select for events for which you are interested in and prepare payload.__

Please check the Kafka messages for the filter scenario. If kafka message is not of the format, which contain operation state and operation type. If message does not contain operation state and operation type notification will not be raised.**

Kafka messages will be improved in future.

    {_admin: {created: 1579592163.561016, modified: 1579592163.561016, projects_read: [

    894160c9-1ead-4c85-9742-e7453260ea5f], projects_write: [894160c9-1ead-4c85-9742-e7453260ea5f]},

    _id: 5c53f989-defc-4f93-8ab9-93c62136c37e, id: 5c53f989-defc-4f93-8ab9-93c62136c37e,

    isAutomaticInvocation: false, isCancelPending: false, lcmOperationType: instantiate,

    links: {nsInstance: /osm/nslcm/v1/ns_instances/35f7ae25-2cf6-4a63-8388-a114513198ed,

        self: /osm/nslcm/v1/ns_lcm_op_occs/5c53f989-defc-4f93-8ab9-93c62136c37e}, nsInstanceId: 35f7ae25-2cf6-4a63-8388-a114513198ed,

    operationParams: {lcmOperationType: instantiate, nsDescription: testing, nsInstanceId: 35f7ae25-2cf6-4a63-8388-a114513198ed,

        nsName: check, nsdId: f445b11a-63d8-44b3-85a8-b4b864ccccd6, nsr_id: 35f7ae25-2cf6-4a63-8388-a114513198ed,

        ssh_keys: [], vimAccountId: d5d59b88-7015-4f4b-8df6-bd05765cfa25}, operationState: PROCESSING,

    startTime: 1579592163.5609882, statusEnteredTime: 1579592163.5609882}

Refer ETSI SOL005 document for filter options [Page no 154](https://www.etsi.org/deliver/etsi_gs/NFV-SOL/001_099/005/02.04.01_60/gs_NFV-SOL005v020401p.pdf). Below an example of payload:

```yaml

{

    "filter": {

        "nsInstanceSubscriptionFilter": {

            "nsdIds": [

                "93b3c041-cac4-4ef3-8ad6-400fbad32a90"

            ]

        },

        "notificationTypes": [

            "NsLcmOperationOccurrenceNotification"

        ],

        "operationTypes": [

            "INSTANTIATE"

        ],

        "operationStates": [

            "PROCESSING"

        ]

    },

    "CallbackUri": "http://192.168.61.143:5050/notifications",

    "authentication": {

        "authType": "basic",

        "paramsBasic": {

            "userName": "user",

            "password": "user"

        }

    }

}

```

This payload implies that, for nsd id 93b3c041-cac4-4ef3-8ad6-400fbad32a90 if operation state is PROCESSING and operation type is INSTANTIATE then, send a notification to http://192.168.61.143:5050/notifications using the "authentication" mechanism whose payload is of datatype NsLcmOperationOccurrenceNotification.

__Step 3: Send an HTTPS POST request to create subscription.__

- Add the bearer token as authentication parameter from step 1.

- Payload from step 2 to https://<osm_ip>:9999/osm/nslcm/v1/subscriptions

__Step 4: Verify successful registration of subscription__

Send an HTTPS GET request https://<osm_ip>:9999/osm/nslcm/v1/subscriptions

#### Steps for notification

__Step 1: Create an event in osm satisfying the filter criteria.

For instance, you can launch any NS. This event has operation state as PROCESSING and operation type as INSTANTIATE, when network service is just launched.

__Step 2: See the notification in notification receiver.__

### Current support and future roadmap

### Current support

- Subscriptions for NS lifecycle:

  - JSON schema validation

  - Pre-chech of notification endpoint

  - Duplicate subscription detection

- Notifications for NS lifecycle:

  - SOL005 compliant structure for each subscriber according to their filters and authentication types.

  - POST events to notification endpoints.

  - Retry and backoff for failed notifications.

#### Future roadmap

- Integration of subscription steps in NG-UI.

- Support for OAuth and TLS authentication types for notification endpoint.

- Support for subscription and notification for NSD.

- Support for subscription and notification for VNFD.

- Cache to store subscribers.

# ANNEX 6: Tests to validate VIM capabilities from OSM

## OSM packages and images for E2E tests

All VNF and NS packages as well as VM images required for these tests can be found [in this location](http://osm-download.etsi.org/ftp/e2e-tests/)

## Test 1. Sanity check with simple NS

Objectives:

- Sanity check of correct E2E behaviour.

- Validate VM image management.

- Test access to the console from OSM UI

Steps:

- Onboard and deploy the Cirros NS, as described in [OSM Usage](05-osm-usage.md)

- Access the console via OSM UI (user: "`cirros`", pwd: "`cubswin:)`")

- Check that the VMs are up and running and connected via the common link.

Images:

- [cirros-0.3.4-x86_64-disk.img](http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img)

Descriptors:

- cirros_vnf

- cirros_2vnf_ns

## Test 2a. Failed deployment of scenario when the checksum is invalid

Objective:

- Testing that a wrong checksum prevents a successful deployment

Steps:

- Modify the checksum in the VNF descriptor (using the UI VNF catalog) to add a wrong but format-valid checksum (e.g.: "00000000000000000000000000000000"). Same images are used.

- Deploy the same NS as in "Test 1"

- Check that the system refuses to deploy the NS due to a checksum error ("VIM Exception vimconnException Image not found at VIM with filter")

Images:

- [cirros-0.3.4-x86_64-disk.img](http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img)

Descriptors:

- cirros_vnf

- cirros_2vnf_ns

## Test 2b. Successful deployment of scenario when the descriptor has a checksum

Objective:

- Testing that a valid checksum in the VNF descriptor leads to a successful deployment

Steps:

- Modify the checksum in the VNF descriptor (using the UI VNF catalog) to add the valid checksum for the image ("ee1eca47dc88f4879d8a229cc70a07c6" for the cirros034 image).

- Deploy the same NS as in "Test 1"

- Check that the NS is successfully instantiated.

- Access the console via OSM UI (user: "`cirros`", pwd: "`cubswin:)`")

- Check that the VMs are up and running and connected via the common link.

Images:

- [cirros-0.3.4-x86_64-disk.img](http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img)

Descriptors:

- cirros_vnf

- cirros_2vnf_ns

## Test 3a. Instantiation time of large NS based on Cirros images

Objective:

- Check that instantiation time is bounded to avoid spurious timeouts.

- Measure delay in the deployment, and evaluate potential issues in the connector.

Steps:

- Onboard a "large NS" consisting of:

  - 2 types of VNFs based on Cirros VM. VNF#1 should have 5 interfaces (+management), while VNF#2 would only require 1 interface (+management)

  - Star topology, with 1 VNF in the middle and 5 instances of the other VNF connected to that one (+ the corresponding management interfaces)

- Launch NS instantiation, specifying the right mgmt network to be used

- Check that the UI reports a successful deployment

- Connect to each VNF via SSH (user: "`cirros`", pwd: "`cubswin:)`")

Images:

- [cirros-0.3.4-x86_64-disk.img](http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img)

Descriptors:

- [cirros_2ifaces_vnf](http://osm-download.etsi.org/ftp/e2e-tests/cirros_2ifaces_vnf.tar.gz)

- [cirros_6ifaces_vnf](http://osm-download.etsi.org/ftp/e2e-tests/cirros_6ifaces_vnf.tar.gz)

- [test3a_ns](http://osm-download.etsi.org/ftp/e2e-tests/test3a_ns.tar.gz)

## Test 3b. Instantiation time of large NS based on Cirros images using IP profiles

Objective:

- Check that instantiation time is bounded to avoid spurious timeouts.

- Measure delay in the deployment, and evaluate potential issues in the connector.

- Check that IP profiles work properly in a large NS

Steps:

- Onboard a "large NS" consisting of:

  - 2 types of VNFs based on Cirros VM. VNF#1 should have 5 interfaces (+management), while VNF#2 would only require 1 interface (+management)

  - Star topology, with 1 VNF in the middle and 5 instances of the other VNF connected to that one (+ the corresponding management interfaces)

  - Networks will have an IP profile so that DHCP is enabled

- Launch NS instantiation, specifying the right mgmt network to be used

- Check that the UI reports a successful deployment

- Connect to each VNF via SSH (user: "`cirros`", pwd: "`cubswin:)`") and configure the interfaces to use DHCP, e.g. by changing /etc/network/interfaces and running "ifup ethX"

- Check that connectivity is appropriate via ping from the different VMs

Images:

- [cirros-0.3.4-x86_64-disk.img](http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img)

Descriptors:

- [cirros_2ifaces_vnf](http://osm-download.etsi.org/ftp/e2e-tests/cirros_2ifaces_vnf.tar.gz)

- [cirros_6ifaces_vnf](http://osm-download.etsi.org/ftp/e2e-tests/cirros_6ifaces_vnf.tar.gz)

- [test3b_ns](http://osm-download.etsi.org/ftp/e2e-tests/test3b_ns.tar.gz)

## Test 3c. Instantiation time of large NS based on Ubuntu images using IP profiles

Objective:

- Check that instantiation time is bounded to avoid spurious timeouts, even with large images (Ubuntu vs CirrOS).

- Measure delay in the deployment, and evaluate potential issues in the connector.

- Check that IP profiles work properly in a large NS

Steps:

- Onboard a "large NS" consisting of:

  - 2 types of VNFs based on Ubuntu VM. VNF#1 should have 5 interfaces (+management), while VNF#2 would only require 1 interface (+management)

  - Star topology, with 1 VNF in the middle and 5 instances of the other VNF connected to that one (+ the corresponding management interfaces)

  - Networks will have an IP profile so that DHCP is enabled

- Launch NS instantiation, specifying the right mgmt network to be used

- Check that the UI reports a successful deployment

- Connect to each VNF via SSH (user: "`osm`", pwd: "`osm4u`").

- Check that connectivity is appropriate via ping from the different VMs

Images:

- [US1604](http://osm-download.etsi.org/ftp/e2e-tests/US1604.qcow2)

Descriptors:

- [cirros_2ifaces_vnf](http://osm-download.etsi.org/ftp/e2e-tests/cirros_2ifaces_vnf.tar.gz)

- [cirros_6ifaces_vnf](http://osm-download.etsi.org/ftp/e2e-tests/cirros_6ifaces_vnf.tar.gz)

- [test3c_ns](http://osm-download.etsi.org/ftp/e2e-tests/test3c_ns.tar.gz)

## Test 4a. Day 0 configuration: SSH key injection to the default user

Objective:

- Testing SSH key injection to the default user

Steps:

- Onboard a variant of the NS of Test #1, but with Ubuntu VNFs.

- Instantiate the NS via UI, requesting the injection of a given SSH key for the default user. Specify also the right mgmt network to be used.

- Check that the UI reports a successful deployment

- Check that the VMs are accessible via SSH, using the private SSH key, from the management network.

Images:

- [US1604](http://osm-download.etsi.org/ftp/e2e-tests/US1604.qcow2)

Descriptors:

- [ubuntu_1iface_vnf](http://osm-download.etsi.org/ftp/e2e-tests/ubuntu_1iface_vnf.tar.gz)

- [test4a_ns](http://osm-download.etsi.org/ftp/e2e-tests/test4a_ns.tar.gz)

## Test 4b. Day 0 configuration: user addition

Objective:

- Testing creation of new user and SSH key injection to that user

Steps:

- Onboard a the variant of the NS of Test #4a (test4b_ns), where the NS includes a user "osm" and SSH public key to be injected to every VNF.

- Launch NS instantiation via UI, specifying the right mgmt network to be used.

- Check that the UI reports a successful deployment.

- Check that the VMs are accessible from the management network via the new user "osm" using its private SSH key (the private key is stored in the folder "test4b_ns/keys" inside the NS package).

Images:

- [US1604](http://osm-download.etsi.org/ftp/e2e-tests/US1604.qcow2)

Descriptors:

- [ubuntu_1iface_vnf](http://osm-download.etsi.org/ftp/e2e-tests/ubuntu_1iface_vnf.tar.gz)

- [test4b_ns](http://osm-download.etsi.org/ftp/e2e-tests/test4b_ns.tar.gz)

## Test 4c. Day 0 configuration: custom user script with cloud-init

Objective:

- Testing injection of cloud-init custom user script

Steps:

- Onboard a variant of the NS of Test #4a (test4c_ns), where the VNF includes a cloud-config custom script that creates a file in the VM and injects a SSH public key to the default user.

- Launch NS instantiation via UI, specifying the right mgmt network to be used

- Check that the UI reports a successful deployment.

- Access the VM via SSH and check that the file has been successfully created. The private key is "test4.pem", stored in the folder "ubuntu_1iface_cloudinit_newfile_vnf/keys" inside the VNF package.

Images:

- [US1604](http://osm-download.etsi.org/ftp/e2e-tests/US1604.qcow2)

Descriptors:

- ubuntu_1iface_cloudinit_newfile_vnf

- [test4c_ns](http://osm-download.etsi.org/ftp/e2e-tests/test4c_ns.tar.gz)

## Test 5. Port security disabled

Objective:

- Testing the ability to disable port security on demand

Steps:

- Onboard a variant of the NS of Test #4 (test5_ns), but with a VNF whose single interface has port security disabled.

- Launch NS instantiation via UI, specifying the right mgmt network to be used

- Check that the UI reports a successful deployment.

- Connect to each VNF via SSH (user: "`osm`", pwd: "`osm4u`").

- Configure both VNFs with an additional IP address of the same subnet, e.g.: 192.168.50.X/24

  - Do not remove the mgmt IP address.

  - Add an additional IP address to the single interfaces using the command "ip addr add 192.168.50.X/24 dev eth0" and ping from one VNF to the other one.

- If port security and security groups have been properly disabled, the ping between both VNFs using the added IP addresses should work.

Images:

- [US1604](http://osm-download.etsi.org/ftp/e2e-tests/US1604.qcow2)

Descriptors:

- [ubuntu_1iface_noportsecurity_vnf](http://osm-download.etsi.org/ftp/e2e-tests/ubuntu_1iface_noportsecurity_vnf.tar.gz)

- [test5_ns](http://osm-download.etsi.org/ftp/e2e-tests/test5_ns.tar.gz)

## Test 6a. Assignment of public IP addresses to management interfaces of single-interface VNFs

Objective:

- Testing the assignment of IP addresses from a pool to VNF management interfaces

Prerequisites:

- Configure the VIM to allow the dynamic assignment of public addresses from a pool

- Configure a VIM network (e.g. "public") to use the appropriate pool, to allow external access via "public" IP addresses.

- Configure the datacenter in the RO to assign public IP addresses to VNF management interfaces (use_floating_ip: true)

Steps:

- Onboard and deploy a NS consisting of 2 Ubuntu VNFs interconnected by a single network (mgmt).

- Instantiate the NS via UI, specifying that the NS network "mgmt" must be mapped to the VIM network name "public", so that a "public" IP address will be assigned from the pool.

- Check that the UI reports a successful deployment.

- Connect to each VNF via SSH (user: "`osm`", pwd: "`osm4u`") using the public IP address.

Images:

- [US1604](http://osm-download.etsi.org/ftp/e2e-tests/US1604.qcow2)

Descriptors:

- [ubuntu_1iface_userosm_vnf](http://osm-download.etsi.org/ftp/e2e-tests/ubuntu_1iface_userosm_vnf.tar.gz)

- [test6a_ns](http://osm-download.etsi.org/ftp/e2e-tests/test6a_ns.tar.gz)

## Test 6b. Assignment of public IP addresses to management interfaces of multi-interface VNFs

Objective:

- Testing the assignment of IP addresses from a pool to VNF management interfaces in the case of multi-interface VNFs. The intention is to check that a single default route is injected.

Prerequisites:

- Configure the VIM to allow the dynamic assignment of public addresses from a pool

- Configure a VIM network (e.g. "public") to use the appropriate pool, to allow external access via "public" IP addresses.

- Configure the datacenter in the RO to assign public IP addresses to VNF management interfaces (use_floating_ip: true)

Steps:

- Onboard and deploy a NS consisting of 2 Ubuntu VNFs interconnected by two networks (management and data).

- Instantiate the NS via UI, specifying that the NS network "mgmt" must be mapped to the VIM network name "public", so that a "public" IP address will be assigned from the pool.

- Check that the UI reports a successful deployment.

- Connect to each VNF via SSH (user: "`osm`", pwd: "`osm4u`") using the public IP address.

Images:

- [US1604](http://osm-download.etsi.org/ftp/e2e-tests/US1604.qcow2)

Descriptors:

- [ubuntu_2ifaces_vnf](http://osm-download.etsi.org/ftp/e2e-tests/ubuntu_2ifaces_vnf.tar.gz)

- [test6b_ns](http://osm-download.etsi.org/ftp/e2e-tests/test6b_ns.tar.gz)

## Test 6c. Assignment of public IP addresses to management interfaces of multi-interface VNFs even when IP profiles are used

Objective:

- Testing the assignment of IP addresses from a pool to VNF management interfaces in the case of multi-interface VNFs even when IP profiles are used. The intention is to check again that a single default route is injected and that IP profiles do not affect that single route.

Prerequisites:

- Configure the VIM to allow the dynamic assignment of public addresses from a pool

- Configure a VIM network (e.g. "public") to use the appropriate pool, to allow external access via "public" IP addresses.

- Configure the datacenter in the RO to assign public IP addresses to VNF management interfaces (use_floating_ip: true)

Steps:

- Onboard and deploy the NS used in Test 3c, consisting of a star topology, with 1 VNF in the middle and 5 instances of the other VNF connected to that one (+ the corresponding management interfaces), where all the inter-VNF networks have an IP profile so that DHCP is enabled, but with no default gateway.

- Instantiate the NS via UI, specifying that the NS network "mgmt" must be mapped to the VIM network name "public", so that a "public" IP address will be assigned from the pool. This is the only change with respect to Test 3c.

- Check that the UI reports a successful deployment.

- Connect to each VNF via SSH (user: "`osm`", pwd: "`osm4u`") using the public IP address.

Images:

- [US1604](http://osm-download.etsi.org/ftp/e2e-tests/US1604.qcow2)

Descriptors:

- [ubuntu_2ifaces_vnf](http://osm-download.etsi.org/ftp/e2e-tests/ubuntu_2ifaces_vnf.tar.gz)

- [ubuntu_6ifaces_vnf](http://osm-download.etsi.org/ftp/e2e-tests/ubuntu_2ifaces_vnf.tar.gz)

- [test3c_ns](http://osm-download.etsi.org/ftp/e2e-tests/test3c_ns.tar.gz)

## Test 7a. EPA tests - phase 1

Objectives:

- Testing that the VIM can map properly vCPUs to pairs of physical HW threads

- Testing that the VIM can assign hugepages memory

- Testing that the VIM can assign SRIOV interfaces

- Testing that the order of interfaces is correct

Steps:

- Onboard pktgen_4psriov VNF, which requires:

  - CPU pinning of paired threads

  - Hugepages

  - SRIOV interfaces

- Onboard a NS with 5 instances of pktgen_4psriov, in a star topology, with one of the VNFs in the middle (Emitter) and 4 VNFs attached (Receiver1-4).

- Check that all VNFs are accessible by SSH via mgmt interface (user: "pktgen", pwd: "pktgen")

- Check (at the VIM or the host) that the CPU pinning is correct.

- Check (at the VIM or the host) that hugepages have been assigned to the guest.

- Check with pktgen that the interfaces are correctly attached to SRIOV interfaces and in the right order:

  - Emitter port 0 -> Receiver1 port 0

  - Emitter port 1 -> Receiver2 port 1

  - Emitter port 2 -> Receiver3 port 2

  - Emitter port 3 -> Receiver4 port 3

Images:

- [pktgen](http://osm-download.etsi.org/ftp/e2e-tests/pktgen.qcow2)

Descriptors:

- [pktgen_4psriov_vnfd](http://osm-download.etsi.org/ftp/e2e-tests/pktgen_4psriov_vnfd.tar.gz)

- [test7a_ns](http://osm-download.etsi.org/ftp/e2e-tests/test7a_ns.tar.gz)

## Test 7b. EPA tests - phase 2

Objectives:

- Testing that the VIM can map properly vCPUs to physical cores

- Testing that the VIM can assign passthrough interfaces

- Testing that the order of interfaces is correct

Steps:

- Equivalent to the previous test, but using a variant to the NS which requires:

  - Full cores (instead of HW threads)

  - Passthrough interfaces (instead of SR-IOV)

Images:

- [pktgen](http://osm-download.etsi.org/ftp/e2e-tests/pktgen.qcow2)

Descriptors:

- [pktgen_4ppassthrough_vnfd](http://osm-download.etsi.org/ftp/e2e-tests/pktgen_4ppassthrough_vnfd.tar.gz)

- [test7b_ns](http://osm-download.etsi.org/ftp/e2e-tests/test7b_ns.tar.gz)

# ANNEX 7: Kubernetes installation and requirements

# ANNEX 6: Kubernetes installation and requirements

This section illustrates a safe procedure to setup a Kubernetes cluster that meets the requirements described in [chapter 5](05-osm-usage.md#osm-kubernetes-requirements). Please note that there might be many alternative ways to achieve the same result (i.e. create an equivalent K8s cluster), so, in case you are using different tooling to create your K8s cluster, this annex should be taken just as informative information and refer instead to your tool's guide to the authoritative reference to achieve equivalent results.

# ANNEX 8: Setting up an LXD Cluster

# ANNEX 7: Setting up an LXD Cluster

This section illustrates how to set up an LXD Cluster of three nodes. The following figure shows the machines in which the LXD Cluster will be set up.