## Advanced setups for high I/O performance: EPA and SDN Assist

### Explanation: Overview of EPA in OSM

OSM supports EPA (Enhanced Platform Awareness) since Rel ZERO (May 2016) and usage of multiple NUMA nodes with Rel THIRTEEN (Dec 2022) which are already defined in OSM's VNF descriptors. OSM allows to utilize EPA features like mem-policy, hugepages memory, NUMA pinning, cpu thread policy, CPU pinning, setting resource quotas (CPU, memory, storage and interface). Besides, it supports the usage of passthrough and SR-IOV interfaces. Please see the [EPA Parameters](https://osm.etsi.org/docs/user-guide/latest/20-reference.html#EPA-Parameters) to check the details of supported parameters.

If your VIM supports EPA, then you don't need to do anything extra to use it from OSM. VIM connectors in OSM take advantage of EPA capabilities if the VIM supports it. All you need to do is build your descriptors and deploy.

However, ot all VIMs support EPA necessarily. To overcome this limitation, OSM has added the following two features:

- Since OSM Release ONE (October 2016), OSM includes OpenVIM as a reference VIM, with full support of EPA. You can follow the instructions in this **[link](../openvim-installation.md)** to install and use OpenVIM.
- Since OSM Release TWO (April 2017), OSM includes a new capability in the Resource Orchestrator called **[SDN Assist](#sdn-assist)**. Through this capability, OSM can manage the dataplane underlay conectivity through an external SDN controller. The only requirement for the VIM is that it must able to use SR-IOV and/or passthrough interfaces, and expose the assigned interfaces so that the RO can use them to create the underlay connectivity. By default, the SDN Assist capability is disabled when a datacenter or VIM is added to OSM, but you can instruct OSM to enable it per VIM target.

### SDN Assist

#### Explanation: Why SDN Assist

SDN Assist works as follows to overcome the limitations of the VIM with respect to the underlay:

1. OSM deploys the VMs of a NS in the requested VIM target with Passthrough and/or SRIOV interfaces.
2. Then it retrieves from the VIM the information about the compute node where the VM was deployed and the physical interfaces assigned to the VM (identified by their PCI addressess).
3. Then, OSM maps those interfaces to the appropriate ports in the switch making use of the mapping that you should have introduced in the system.
4. Finally, OSM creates the dataplane networks by instructing the SDN controller and connecting the appropriate ports to the same network.

The module in charge of this workflow is the RO (Resource Orchestrator), which is provided transparently to the user. It uses an internal library to manage the underlay connectivity via SDN. The current library includes plugins for FloodLight, ONOS and OpenDayLight.

#### Reference: General requirements

The general requirements are:

- A dataplane switch (until Release SIX, with Openflow capabilities) that will connect the physical interfaces of the VIM compute nodes.
- An external SDN controller controlling the previous dataplane switch.
- The mapping between the switch ports (identified by name) and the compute node interfaces (identified by host-id and PCI address)
- Some VIMs as Openstack requires admin credentials in order to be able to get the physical place of the SRIOV/passthrough VM interfaces

In addition to the general requirements, every VIM will have to be properly configured.

### How to use SDN Assist

#### Adding a SDN controller to OSM

This is done through CLI.

Add to OSM  the SDN controller, provide credentials.

```bash
osm sdnc-create --name sdn-name --type arista \
--url https://10.95.134.225:443 --user osm --password osm4u \
--config '{mapping_not_needed: True, switch_id: ID}'
# The config section is optional.

# list sdn controllers with:
osm sdnc-list
# get details with:
osm sdnc-show sdn-name
# delete with
osm sdnc-delete sdn-name
```

Note that at SDN creation, connectivity and credentials are not checked.

Available SDN controller plugins (`--type` option) are `onos_vpls`, `onosof`, `floodlightof`, , `dynpac`; and comming `odlof`, `arista`, `ietfl2vpn`.

Depending on the pluging, the SDN controller needs to be fed with the mapping between the VIM compute node interfaces and the switch ports. In case this is not needed use the `--config '{mapping_not_needed: True}'`.

#### Associate the SDN controller to a VIM

To associate the SDN controller to a concrete VIM, the VIM must be updated. In this step the compute-node/swith-port mapping is provided:

```bash
osm vim-update vim-name --sdn_controller sdn-name  --sdn_port_mapping port-mapping-file.yaml
```

This is an example of the `port-mapping-file.yaml` content:

```yaml
- compute_node: nfv54
  ports:
  - pci: "0000:5d:00.1"
    switch_id: Leaf1
    switch_port: "Ethernet13/1"
  - pci: "0000:5d:0a.0"
    switch_id: Leaf1
    switch_port: "Ethernet13/1"
  - pci: "0000:5d:0a.1"
    switch_id: Leaf1
    switch_port: "Ethernet13/1"
  - pci: "0000:5d:00.0"
    switch_id: Leaf2
    switch_port: "Ethernet13/2"
  - pci: "0000:5d:02.0"
    switch_id: Leaf2
    switch_port: "Ethernet13/2"
  - pci: "0000:5d:02.1"
    switch_id: Leaf2
    switch_port: "Ethernet13/2"
- compute_node: nfv55
  # ...
```

NOTE: several PCI addresses can be connected to the same switch port. This is because a physical interface has several SR-IOV virtual interfaces, each one with different PCI address.

NOTE: The optional `switch_id` provided at `--config` is taken as a default if missing in the port-mapping file. This is useful if there is only one switch.

To overwrite the port mapping the same instruction can be used after modifying the port-mapping file.

You can check the associated SDN controller by:

```bash
osm vim-show vim-name
+-----------------+-----------------------------------------------------------------+
| key             | attribute                                                       |
+-----------------+-----------------------------------------------------------------+
| _id             | "bf900941-a6d3-4eba-9017-0cec657b9490"                          |
| name            | "sdn-name"                                                      |
| vim_type        | "openstack"                                                     |
| description     | "some description"                                              |
| vim_url         | "https://192.168.1.1:5000/v3"                                   |
| vim_user        | "osm"                                                           |
| vim_password    | "********"                                                      |
| vim_tenant_name | "osm"                                                           |
| config          | {                                                               |
|                 |   "insecure": true,                                             |
|                 |   "sdn-controller": "82b067c8-cb9a-481d-b311-59f68b75acae",     |
|                 |   "sdn-port-mapping": [                                         |
|                 |     {                                                           |
|                 |       "compute_node": "compute-66e153a8-c45",                   |
|                 |       "ports": [                                                |
|                 |         {                                                       |
|                 |           "pci": "002f-0000:83:00.0-000",                       |
|                 |           "switch_id": "Leaf1",                                 |
|                 |           "switch_port": "Ethernet52/1"                         |
```

You can disassociatede SDN controller from a VIM by:

```bash
osm vim-update vim-name --sdn_controller '' # attach an empty string
```

Note: detaching a SDN controller from the VIM is mandatory before deleting the SDN controller

### How to configure Openstack for full EPA support in OSM

Besides the instructions above for any Openstack, you should do extra configuration to configure Openstack for running VNFs which use SR-IOV interfaces.

**DO NOT consider this as the final configuration guide for Openstack with EPA. Please check Openstack docs and Openstack downstream distros' docs to get a further understanding and the up-to-date required configuration.**

**Note: The configuration shown below works with Openstack Newton, and it might not work with later versions.**

- The compute nodes need to have a whitelist for the interfaces with SRIOV and passthrough enabled, and those interface need to be associated to a physical network label e.g. `physnet`. This can be done in the file `/etc/nova/nova.conf`:

```json
pci_passthrough_whitelist=[{"devname": "p3p1", "physical_network": "physnet"}, {"devname": "p3p2", "physical_network": "physnet"}]
```

- The neutron controller needs to be updated to add `sriovnicswitch` to the `mechanism_drivers`. This can be done in the file `/etc/neutron/plugins/ml2/ml2_conf.ini`

```text
mechanism_drivers=openvswitch,sriovnicswitch
```

- The neutron controller needs to be updated to set the vlans to be used for the defined physical network label. This can be done in the file `/etc/neutron/plugins/ml2/ml2_conf.ini`. For instance, to set the vlans from 2000 to 3000:

```text
network_vlan_ranges =physnet:2000:3000
```

- The neutron controller needs to be updated to allow the supported NIC vendor’s product ID. This can be done in the file `/etc/neutron/plugins/ml2/ml2_conf.ini` to

```text
[ml2_sriov]
supported_pci_vendor_devs = 8086:10ed
```

- The nova controller needs to be updated to allow proper scheduling of SR-IOV and Passthrough devices, by adding the PciPassthroughFilter filter to the list of filters. This can be done in the file `/etc/nova/nova.conf`:

```text
scheduler_available_filters=nova.scheduler.filters.all_filters
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter, PciPassthroughFilter
```

The previous configuration has taken as a reference the documents in the links below. Please check them in case your needed more details:

- <https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html-single/network_functions_virtualization_product_guide/>
- <https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html-single/network_functions_virtualization_planning_guide/>
- <https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html-single/network_functions_virtualization_configuration_guide/>
- <https://docs.openstack.org/newton/networking-guide/config-sriov.html>