From b1ff7d03990ad6dd49de81e0e0c194badc7cf3ba Mon Sep 17 00:00:00 2001 From: aguilard Date: Thu, 10 Mar 2022 09:59:45 +0100 Subject: [PATCH] Add section about helm-based Execution Environments Signed-off-by: aguilard --- 09-helm-ee.md | 597 +++++++++++++++++++++++++++ 09-references.md => 10-references.md | 0 index.md | 4 +- index.rst | 3 +- 4 files changed, 602 insertions(+), 2 deletions(-) create mode 100644 09-helm-ee.md rename 09-references.md => 10-references.md (100%) diff --git a/09-helm-ee.md b/09-helm-ee.md new file mode 100644 index 0000000..361dbd1 --- /dev/null +++ b/09-helm-ee.md @@ -0,0 +1,597 @@ +# Helm-chart based Execution Environments + +## Introduction to Execution Environments in OSM + +OSM's Execution Environments (EE) provide a runtime framework to run day-1 and day-2 primitives, as well as metrics collection for NFs. These EE provide the means for NF-specific management code to run it into a dedicated helm chart, which is deployed into OSM's system cluster. From there, the EE interacts with the managed NF (e.g. via SSH), providing a NF-agnostic mean to manage NFs by OSM. + +OSM communicates with its EE to trigger actions via gRPC calls, which are handled by a fronted component (running in a pod) in its constituent helm chart. In order to ease the NF onboarding tasks, there is already a helm chart template available including a fronted element which implements the gRPC interface required by OSM. + +This chart template (called `eechart`) is oriented to create a chart with two sub-components: the gRCP front-end (exposed via a Kubernetes service), and an optional back-end, in charge of the interaction with the NF, where onboarder's code required for NF operations is included. This template is oriented to make unnecessary knowing low-level implementation details of the chart structure or gRPC implementation, although all the details are also available for advanced users in the corresponding [repo of the GRPC pod image](https://osm.etsi.org/gitlab/vnf-onboarding/docker-api-fe/). + +The purpose of this document is to provide the guidelines for adding primitives based on EE so that it can be operated at runtime by the end-user. The following sections will explain how to create easily EE primitives for your NF using the available template. + +## Overview of the EE template + +To understand how an EE works, we will retrieve the packages of the template and we will analyse them in detail, describing the structure and how they can be adapted to the needs of specific NFs. + +First, we will clone the [OSM packages repository](https://osm.etsi.org/gitlab/vnf-onboarding/osm-packages), which contains all OSM's sample packages. + +```bash +export PACKAGES_FOLDER=$HOME/packages +git clone --recursive https://osm.etsi.org/gitlab/vnf-onboarding/osm-packages.git ${PACKAGES_FOLDER} +``` + +Then, we will use the packages `sample_ee_vnf` and `sample_ee_ns` as template. These sample packages define a simple network service consisting of a VNF that has a single Ubuntu-based VDU with two network interfaces, one of them connected to the management network. The VNF also includes various sample day-1 and day-2 primitives based on EE that will be discussed below. + +### VNF descriptor template + +When defining a NF package, the central file is the NF descriptor. The descriptor models the internal structure of the NF as well as the available day-1 and day-2 primitives, following the [OSM's IM](http://osm-download.etsi.org/repository/osm/debian/ReleaseELEVEN/docs/osm-im/osm_im_trees/etsi-nfv-vnfd.html). In the case of the EE template, this file corresponds to `sample_ee_vnf/sample_ee_vnfd.yaml`. + +A pre-requirement to define EE-based primitives is including the appropriate reference in the descriptor to the own EE, and then a reference to the corresponding day-1 and day-2 primitives that are hosted in the EE. + +The first is achieved by adding an `execution-environment-list` block, where one or more EEs (there might be more than one) are indicated. This can be seen in the following excerpt of the descriptor template: + +```yaml +vnfd: + description: Basic execution environment example + df: + ... + lcm-operations-configuration: + operate-vnf-op-config: + day1-2: + ... + execution-environment-list: + - external-connection-point-ref: vnf-mgmt-ext + helm-chart: eechart + id: sample_ee +``` + +In the example, just before that part, the day-2 primitives and their parameters are defined under `config-primitive` block. Every primitive must refer to an EE (`execution-environment-ref`), has a `name` and optionally a `parameter` list. A parameter is formed by a pair of tag (`name`) and its type (`data-type`). + +```yaml +- config-primitive: + - execution-environment-primitive: run_script + execution-environment-ref: sample_ee + name: run_script + parameter: + - data-type: STRING + name: file + - data-type: STRING + name: parameters + - execution-environment-primitive: ansible_playbook + execution-environment-ref: sample_ee + name: ansible_playbook + parameter: + - data-type: STRING + name: playbook-name + - data-type: STRING + name: app + - execution-environment-primitive: ping + execution-environment-ref: sample_ee + name: ping + ... +``` + +If these day-2 primitives are also expected to be executed as day-1 operations, they must be referenced in the block `initial-config-primitive`, where their values would be also set. The `seq` attribute specifies the order in which they will be executed sequentially when the VNF is instantiated (the instantiation will not be successful if any of these primitives fails). The section where day-1 primitives are defined is shown in the following excerpt of the NF template descriptor: + +```yaml +initial-config-primitive: +- execution-environment-ref: sample_ee + name: config + parameter: + - name: ssh-hostname + value: + - name: ssh-username + value: ubuntu + seq: 1 +- execution-environment-ref: sample_ee + name: run_script + parameter: + - name: file + value: install_nginx.sh + seq: 2 +- execution-environment-ref: sample_ee + name: ansible_playbook + parameter: + - name: playbook-name + value: playbook.yaml + - name: app + value: ntp + seq: 3 + ... +``` + +In this template, the `config` day-1 primitive (see above) is used to initialise a set of configuration variables with their values, so it must be the first in sequence. In this descriptor, the SSH parameters to connect to the VDU from EE are also set. The `` value in `ssh-hostname` is replaced by the management IP of the NF when the NF is instantiated by OSM. SSH authentication is performed with public/private key pairs, which are transparently managed by OSM if `ssh-access` is configured in the descriptor (the EE generates its own keys, which are injected by OSM in the corresponding VDUs): + +```yaml +lcm-operations-configuration: + operate-vnf-op-config: + day1-2: + ... + config-access: + ssh-access: + default-user: ubuntu + required: true +``` + +### Helm chart template + +In addition to the descriptor, a NF that uses helm chart EE must include the chart definition under the `helm-charts` subdirectory of the package. The NF package template that we are using in this guide already includes the pre-created `eechart` chart, which can be found under the `helm-charts` folder: + +```text +helm-charts +└── eechart + ├── Chart.yaml + ├── source + │   ├── install.sh + │   ├── install_nginx.sh + │   ├── mylib.py + │   ├── playbook.yaml + │   ├── run_ssh.sh + │   └── vnf_ee.py + ├── templates + └── values.yaml +``` + +`Chart.yaml`, `values.yaml` and the files in the `templates` folder are related to the chart. In this template, they are designed to contain all the necessary configuration data to deploy the gRPC server in the OSM k8s cluster (the name and repository of the chart, type of service, resources, etc.) so that they do not require any modification to define custom primitives, as we will see shortly. Advanced users, however, might what to evolve then to e.g. customize the list of components included in the helm chart, for reusing pre-existing vendor containers, referencing other charts, etc. +As previously discussed, the chart included in the template is designed to be used as-is for the commonest cases of primitives, with minor customizations in the own package. Thus, the chart will use some parts of the own NF package to customize the EE for almost any potential use: + +- Any files in the `source` folder of the package will be injected into the fronted pod of the EE when the NF is instantiated. This mechanism is quite useful to include any files required to support the mechanisms of the primitive (e.g. a playbook for an Ansible-based primitive). +- `install.sh`: Bash script to be executed into the EE when the container is created (only once). This script is useful to install additional software (e.g. Python libraries or apt packages) or make some initial configuration on the fronted container, without the need of re-creating the frontend container image from scratch (although it is always possible). In this sample NF package template, the script updates the OS, installs Ansible, the ping binary, and some Python libraries needed by the primitives of the example: + + ```bash + #!/bin/bash + ## + # Licensed under the Apache License, Version 2.0 (the "License"); you may + # not use this file except in compliance with the License. You may obtain + # a copy of the License at + # + # http://www.apache.org/licenses/LICENSE-2.0 + # + # Unless required by applicable law or agreed to in writing, software + # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + # License for the specific language governing permissions and limitations + # under the License. + ## + + echo "Updating operating system" + apt-get update + + # Install ansible libraries + echo "Installing ansible" + apt-get install -y software-properties-common + apt-add-repository --yes --update ppa:ansible/ansible + apt install -y ansible + + # Install library to execute command remotely by ssh + echo "Installing asynssh" + python3 -m pip install asyncssh + + # Install ping system command + apt install -y iputils-ping + + # Install HTTP python library + python3 -m pip install requests + ``` + +- `vnf_ee.py`: Python file that includes the code to be called from when a primitive is called. `vnf_ee.py` defines a `VnfEE` class that includes one method for each primitive to be executed on the EE. You can modify, delete or add new methods here, but keep in mind that method names must match the name of the primitive in descriptor (the - character should be avoided). User code to interact with the VNF might be inserted here, although, for the sake of readability and maintainability, it is advised to include just invocations to code residing in other files of the source folder. In all cases, the methods must return a status code to back-end reporting the state of success (`OK`) or failure (`ERROR`) of the operation, and a text description in a `yield` call. The structure of VnfEE class is the following: + + ```python + class VnfEE: + + def __init__(self, config_params): + self.logger = logging.getLogger('osm_ee.vnf') + self.config_params = config_params + + # config method saves SSH access parameters (host and username) for future use by other methods. + # It is mandatory in any case. + async def config(self, id, params): + self.logger.debug("Execute action config, params: {}".format(params)) + # Config action is special, params are merged with previous config calls + self.config_params.update(params) + required_params = ["ssh-hostname"] + self._check_required_params(self.config_params, required_params) + yield "OK", "Configured" + + # This method implements the "run_script" primitive. Uncomment and modify it if a primitive requires executing a user + # script in the VDU. It needs "file" parameter (file name to run) and optionally "parameters" (command-line arguments). + async def run_script(self, id, params): + self.logger.debug("Execute action run_script, params: '{}'".format(params)) + ... + if return_code != 0: + yield "ERROR", "return code {}: {}".format(return_code, stderr.decode()) + else: + yield "OK", stdout.decode() + ... + ``` + +- The other files in the source directory are specific to each primitive and will be invoked from the `vnf_ee.py` file as we will see shortly. + +## Update of the NF template package + +After reviewing the package structure of the sample NF package, the next step would be the adaptation of the `vnf_ee.py` (and other auxiliary files invoked from it) to add those primitives. The `sample_ee_vnf` template already includes some examples of different types of day-1/day-2 primitives, which just need to be uncommented uncomment and adapted to implement them adapted to your needs. Some common types of primitives can be easily implemented from this template, as will be described in the following sections. + +### EXAMPLE 1: Invoking a pre-existing script to execute remote commands against the NF + +Some primitives can be implemented simply by running a script on a VDU of the NF via SSH. That script may exist in the VM or could be copied using SFTP/SCP from the EE. In this sample package, this a sample primitive of this kind called `run_script` is provided. This primitive runs a helper bash script (`run_ssh.sh`) on the own EE pod, which, in turn, transfers and executes a user script to the VDU via SSH, which will be run in the end. Both scripts are part of the package and are located in the `source` folder, as described before. + +If you want to try a sample primitive of this type (or create your own primitive that runs a custom script), you just need to uncomment the lines of the `run_script` method in `VnfEE` class of the `vnf_ee.py` file: + +```python +class VnfEE: + SSH_SCRIPT = "/app/EE/osm_ee/vnf/run_ssh.sh" + ... + async def run_script(self, id, params): + self.logger.debug("Execute action run_script, params: '{}'".format(params)) + self._check_required_params(params, ["file"]) + + command = "bash " + self.SSH_SCRIPT + " " + self.config_params["ssh-hostname"] + " " + self.config_params["ssh-username"] + " " + params["file"] + if "parameters" in params: + command += " \"" + params.get("parameters", "") + "\"" + self.logger.debug("Command: '{}'".format(command)) + return_code, stdout, stderr = await util_ee.local_async_exec(command) + if return_code != 0: + yield "ERROR", "return code {}: {}".format(return_code, stderr.decode()) + else: + yield "OK", stdout.decode() +``` + +The primitive needs two parameters: the name of the script to be executed on the VDU (`file`) and its command-line arguments (`parameters`), if any. + +In case you wanted your primitive to be available also as day-2 primitive, you should also uncomment the lines related to the `run_script` primitive in the `config-primitive` block of the descriptor. + +In case you planed to create your own custom primitive, please remind that the method must return `OK` or `ERROR` codes, and a description in a `yield` call. + +```yaml +- execution-environment-primitive: run_script + execution-environment-ref: sample_ee + name: run_script + parameter: + - data-type: STRING + name: file + - data-type: STRING + name: parameters +``` + +For example, you could include the following day-1 operation to install a NGINX server in a NF's VDU (by calling the `install_nginx.sh` script), just by uncommenting the following lines under `initial-config-primitive`: + +```yaml +- execution-environment-ref: sample_ee + name: run_script + parameter: + - name: file + value: install_nginx.sh + seq: 2 +``` + +As in the general case, all files in `helm-charts/eechart/source` will be copied into the EE container, including the `run_ssh.sh` and `install_nginx.sh` files. When OSM requests to execute the `run_script` primitive, the back-end will call the `run_script` method, which will invoke the `run_ssh.sh` script in the EE. This script reads the SSH configuration parameters from the shell, the name of the second script (`install_nginx.sh`) and its runtime parameters (if any), and connects to the VDU via SSH for copying and running the script into it. The `run_ssh.sh` exit status is also relevant: a non-zero value indicates an error, so the Python method must return `ERROR` instead of `OK` in `yield`. + +The template is designed so that you do not need to modify the `run_ssh.sh` script, just only create more scripts in `source` directory like `install_nginx.sh`. The final script to install the NGINX server in the NF may be as simple as this one: + +```bash +#!/usr/bin/env bash + +set -eux +sudo -s < (int, str): + """ + Execute a remote command via SSH. + """ + + try: + async with asyncssh.connect(host, + username=user, + known_hosts=None) as conn: + logger.debug("Executing command '{}'".format(command)) + result = await conn.run(command) + logger.debug("Result: {}".format(result)) + return result.exit_status, result.stderr + except Exception as e: + logger.error("Error: {}".format(repr(e))) + return -1, str(e) +``` + +Since the library imports `asyncssh`, it must be installed by the `install.sh` script: + +```bash +# Install library to execute command remotely by ssh +echo "Installing asynssh" +python3 -m pip install asyncssh +``` + +## Updating NF and NS template packages to adapt them to your needs + +In addition to the cases covered by examples above, you may want to edit the NF and NS descriptors and update the name of the NF and NS, change the images, network interfaces, memory sizes, etc. in any of the VDUs. + +For final versions, it is also advised to remove or comment any primitives in the NF descriptor template and the methods from `vnf_ee.py` file that will not be used, as well as any related files under the `helm-charts/eechart/source` folder, and removing any unused software from the `install.sh` script. Also note that for final versions, it is highly advisable using a specific container image for the frontend with all required software preinstalled, to avoid hot installations upon instantiation (those are only convenient during development). + +## Testing the NF and NS packages + +As usual in OSM, the first step to use NF and NS package is onboarding them into the OSM system: + +```bash +osm nfpkg-create {PACKAGES_FOLDER}/sample_ee_vnf +osm nspkg-create {PACKAGES_FOLDER}/sample_ee_ns +``` + +Then, you can instantiate the NS following the usual commands (`$VIM_TARGET` is the VIM's name in OSM and `$VIM_EXT_NET` is management network's name in that VIM): + +```bash +osm ns-create --ns_name sample_ee --nsd_name sample_ee-ns --vim_account $VIM_TARGET --config "{vld: [ {name: mgmtnet, vim-network-name: $VIM_EXT_NET} ] }" +``` + +### Executing your day-2 primitives + +For instance, for executing the `run_script` primitive as day-2 operation, you can run the `osm ns-action` command when NS instance is ready. As always, the `action_name` parameter contains the primitive's name, and `params` value is a YAML/JSON inline string with the parameters required by the primitive (if applicable): + +```bash +$ osm ns-list ++------------------+--------------------------------------+---------------------+----------+-------------------+---------------+ +| ns instance name | id | date | ns state | current operation | error details | ++------------------+--------------------------------------+---------------------+----------+-------------------+---------------+ +| sample_ee | 354b0009-05ca-4565-850f-03d029601753 | 2022-02-22T16:32:11 | READY | IDLE (None) | N/A | ++------------------+--------------------------------------+---------------------+----------+-------------------+---------------+ + +$ osm ns-action --action_name run_script --vnf_name sample_ee --params '{file: install_nginx.sh, parameters: ""}' sample_ee +926b9375-732c-464a-98d1-59678b4de655 +``` + +You can also see the history of operations over the NS instance: + +```bash +$ osm ns-op-list sample_ee ++--------------------------------------+-------------+-------------+-----------+---------------------+--------+ +| id | operation | action_name | status | date | detail | ++--------------------------------------+-------------+-------------+-----------+---------------------+--------+ +| 27bc5366-06d6-4562-b6e2-ba39a01d6dde | instantiate | N/A | COMPLETED | 2022-02-22T16:32:11 | - | +| 926b9375-732c-464a-98d1-59678b4de655 | action | run_script | COMPLETED | 2022-02-22T16:34:41 | - | ++--------------------------------------+-------------+-------------+-----------+---------------------+--------+ +``` + +Details of the day-2 operation are also available: + +```bash +$ osm ns-op-show 926b9375-732c-464a-98d1-59678b4de655 ++-----------------------+------------------------------------------------------------------------------------------------------+ +| field | value | ++-----------------------+------------------------------------------------------------------------------------------------------+ +| _id | "926b9375-732c-464a-98d1-59678b4de655" | +| id | "926b9375-732c-464a-98d1-59678b4de655" | +| operationState | "COMPLETED" | +| queuePosition | 0 | +| stage | "" | +| errorMessage | "" | +| detailedStatus | null | +| statusEnteredTime | 1645547699.5537114 | +| nsInstanceId | "354b0009-05ca-4565-850f-03d029601753" | +| lcmOperationType | "action" | +| startTime | 1645547681.2144706 | +| isAutomaticInvocation | false | +... ++-----------------------+------------------------------------------------------------------------------------------------------+ +``` + +### Debugging + +The logs generated by the day-1 or day-2 operations can be reviewed by accessing the EE in OSM's system cluster. Thus, a pod named `eechart-` (the same prefix that chart name in VNFD) in the `osm` namespace would contain the runtime of the sample EE: + +```bash +$ kubectl -n osm get pods +NAME READY STATUS RESTARTS AGE +eechart-0031195008-0 1/1 Running 0 89m +... + +$ kubectl -n osm logs pod/eechart-0031195008-0 +Install additional libraries +Updating libraries +Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease +Get:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] +Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] +... +Starting frontend server +DEBUG:osm_ee.util:Execute local command: ssh-keygen -q -t rsa -N '' -f /root/.ssh/id_rsa +DEBUG:osm_ee.util:Return code: 0 +DEBUG:osm_ee:Generated ssh_key, return_code: 0 +... +``` + +It is also possible to access the container and check the EE directory structure and its contents. The files included in the `helm-charts/eechart/source` folder in the NF package should now exist in the `/app/EE/osm_ee/vnf` folder in the container filesystem: + +```bash +$ kubectl -n osm exec -ti pod/eechart-0031195008-0 -- bash +root@eechart-0031195008-0:/app/EE# ls -l osm_ee/vnf +total 0 +lrwxrwxrwx 1 root root 17 Feb 21 15:53 install.sh -> ..data/install.sh +lrwxrwxrwx 1 root root 23 Feb 21 15:53 install_nginx.sh -> ..data/install_nginx.sh +lrwxrwxrwx 1 root root 15 Feb 21 15:53 mylib.py -> ..data/mylib.py +lrwxrwxrwx 1 root root 20 Feb 21 15:53 playbook.yaml -> ..data/playbook.yaml +lrwxrwxrwx 1 root root 17 Feb 21 15:53 run_ssh.sh -> ..data/run_ssh.sh +lrwxrwxrwx 1 root root 16 Feb 21 15:53 vnf_ee.py -> ..data/vnf_ee.py diff --git a/09-references.md b/10-references.md similarity index 100% rename from 09-references.md rename to 10-references.md diff --git a/index.md b/index.md index 33b6fc7..5aaa95e 100644 --- a/index.md +++ b/index.md @@ -13,4 +13,6 @@ by OSM VNF Onboarding Task Force 7. [VNF Onboarding Walkthrough](06-walkthrough.md) 8. [KNF Onboarding Walkthrough](07-knfwalkthrough.md) 9. [Advanced Charms](08-advanced-charms.md) -10. [Additional References](09-references.md) +10. [Helm-chart based Execution Environments](09-helm-ee.md) +11. [Additional References](10-references.md) + diff --git a/index.rst b/index.rst index b658a28..21fd6da 100644 --- a/index.rst +++ b/index.rst @@ -19,5 +19,6 @@ Welcome to Open Source MANO's VNF Onboarding guide! 06-walkthrough 07-knfwalkthrough 08-advanced-charms - 09-references + 09-helm-ee + 10-references -- GitLab