Feature 10957: Extend basic_26 to test client side TLS in helm based EE

[osm/tests.git] / README.md

diff --git a/README.md b/README.md
index 1d7d5dc..6759fc0 100644 (file)
--- a/README.md
+++ b/README.md
@@ -1,5 +1,5 @@
 <!--
 <!--

-Copyright 2020 ETSI
+Copyright ETSI

 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.


@@ -21,92 +21,112 @@ This repository contains tools and configuration files for testing and automatio
 
 ## Prerequisites
 
 
 ## Prerequisites
 

-* **Robot Framework**
-* **Packages**: ssh ping yq git
-* **Python3 packages**: haikunator requests robotframework robotframework-seleniumlibrary robotframework-requests robotframework-jsonlibrary robotframework-sshlibrary
-* Clone **osm-packages** from gitlab
-* Environment config file for your infrastructure [envfile.rc]
+- OSM running
+- VIM already registered in OSM
+- K8s cluster already registered in OSM (for tests involving a K8s cluster)

 
 

-## Installing
+## Quickstart (run tests using docker)

 
 

-This bash script can be used to setup your environment to execute the tests.
+### Configure the environment file
+
+```
+export OSM_HOSTNAME=<OSM_IP_ADDRESS>
+export VIM_TARGET=<VIM_REGISTERED_AT_OSM>
+export VIM_MGMT_NET=<NAME_OF_THE_MGMT_NETWORK_IN_THE_VIM>
+export K8S_CREDENTIALS= # path to the kubeconfig file of the K8s cluster to be tested
+export OS_CLOUD=    # OpenStack Cloud defined in $HOME/.config/openstack/clouds.yaml or in /etc/openstack/clouds.yaml
+```
+
+### Create the docker container

 
 ```bash
 
 ```bash

-PACKAGES_FOLDER=osm-packages
-add-apt-repository -y ppa:rmescandon/yq && apt update && apt install yq git iputils-ping ssh -y
-pip install haikunator requests robotframework robotframework-seleniumlibrary robotframework-requests robotframework-jsonlibrary \
-    robotframework-sshlibrary
-snap install charm
-# Download community packages
-git clone https://osm.etsi.org/gitlab/vnf-onboarding/osm-packages.git ${PACKAGES_FOLDER}
+docker build -f docker/Dockerfile -t osmtests .

 ```
 
 ```
 

-envfile.rc
+### Run the tests

 
 ```bash
 
 ```bash

-# VIM Setup
-OS_USERNAME=<openstack_username>
-OS_PASSWORD=<openstack_password>
-OS_TENANT_NAME=<openstack_tenant_name>
-OS_AUTH_URL=<openstack_authorization_url>
-OS_TENANT_ID=<openstack_tenant_id>
+docker run --rm=true --name tests -t --env-file envconfig.rc \
+           -v ~/.config/openstack/clouds.yaml:/etc/openstack/clouds.yaml \
+           -v ~/tests/reports:/robot-systest/reports \
+           osmtests \
+           -t sol003_01
+```

 
 

-# OSM Setup
-OSM_HOSTNAME=<osm_ip_address>
-VIM_TARGET=<osm_vim_name>
-VIM_MGMT_NET=<osm_vim_mgmt_name>
+You can use a different robot tag instead of `sol003_01`. The whole list of tags are gathered below in this README.

 
 

-# Clouds file datacenter
-OS_CLOUD=<datacenter_in_clouds_file>
-# SDNCs file
-OS_SDNC=<SDN_controller_in_sdncs_file>
+## How to mount local tests folder for developing purposes

 
 

-# K8S config file
-K8S_CREDENTIALS=<path_to_kubeconfig>
+The following line will mount the required files for SOL003_01 testuite and will execute the test `sol003_01`

 
 

-# Juju data file
-JUJU_DATA=<path_to_jujudata>
+```bash
+docker run --rm=true --name tests -t --env-file envconfig.rc \
+           -v ~/.config/openstack/clouds.yaml:/etc/openstack/clouds.yaml \
+           -v ~/tests/reports:/robot-systest/reports \
+           -v ~/tests/robot-systest/lib/sol003_common_lib.robot:/robot-systest/lib/sol003_common_lib.robot \
+           osmtests \
+           -t sol003_01
+```

 
 

-# The following set of environment variables will be used in host
-# of the robot framework. Not needed for docker execution
+Relevant volumes to be mounted are:

 
 

-# Folder where Robot tests are stored
-ROBOT_DEVOPS_FOLDER=robot-systest
+- <path_to_reports> [OPTIONAL]: the absolute path to reports location in the host
+- <path_to_clouds.yaml> [OPTIONAL]: the absolute path to the clouds.yaml file in the host
+- <path_to_sdncs.yaml> [OPTIONAL]: the absolute path to the sdncs.yaml file in the host
+- <path_to_kubeconfig> [OPTIONAL]: the kubeconfig file to be used for k8s clusters

 
 

-# Folder to save alternative DUT environments (optional)
-ENVIRONMENTS_FOLDER=environments
+Other relevant options to run tests are:

 
 

-# Folder where all required packages are stored
-PACKAGES_FOLDER=osm-packages
+- `--env-file`: It is the environmental file where is described the OSM target and VIM
+- `-o <osmclient_version>` [OPTIONAL]: It is used to specify a particular osmclient version. Default: latest
+- `-p <package_branch>` [OPTIONAL]: OSM packages repository branch. Default: master
+- `-t <testing_tags>` [OPTIONAL]: Robot tests tags. [sanity, regression, particular_test]. Default: sanity

 
 

-# Folder where test results should be exported
-ROBOT_REPORT_FOLDER=results
-```

 
 

-## Deployment
+## How to run tests from a host

 
 

-It is possible to run the tests directly from the repository or using a docker container with the tests
+### Installing

 
 

-1. Docker container creation:
+This bash script can be used to setup your environment to execute the tests.

 
 ```bash
 
 ```bash

-docker build -f docker/Dockerfile -t osmtests .
+sudo apt-get update
+sudo apt-get install ssh ping yq git
+# Python packages used for the tests
+python3 -m pip install -r requirements.txt
+python3 -m pip install -r requirements-dev.txt
+# Download community packages
+git clone https://osm.etsi.org/gitlab/vnf-onboarding/osm-packages.git

 ```
 
 ```
 

-Options:
+### Configure the environment

 
 

-* --env-file: It is the environmental file where is described the OSM target and VIM
-* -o <osmclient_version> [OPTIONAL]: It is used to specify a particular osmclient version. Default: latest
-* -p <package_branch> [OPTIONAL]: OSM packages repository branch. Default: master
-* -t <testing_tags> [OPTIONAL]: Robot tests tags. [sanity, regression, particular_test]. Default: sanity
+Create a file `envfile.rc` copying from `envconfig-local.rc` and set the required variables.

 
 

-Volumes:
+### Running the tests
+
+```bash
+source envfile.rc
+mkdir reports
+robot -d reports -i <testing_tags> testsuite/
+```
+
+## How to run tests from an environment identical to OSM CICD
+
+```bash
+git clone https://osm.etsi.org/gerrit/osm/devops
+git clone https://osm.etsi.org/gerrit/osm/IM
+git clone https://osm.etsi.org/gerrit/osm/osmclient
+git clone https://osm.etsi.org/gerrit/osm/tests
+# run HTTP server to server artifacts
+devops/tools/local-build.sh --install-qhttpd
+# generate debian packages locally that will be served by the HTTP server
+devops/tools/local-build.sh --module IM,osmclient,tests stage-2
+# create docker image and store it locally as opensourcemano/tests:devel
+devops/tools/local-build.sh --module tests
+```

 
 

-* <path_to_reports> [OPTIONAL]: It is the absolute path to reports location in the host
-* <path_to_clouds.yaml> [OPTIONAL]: It is the absolute path to the clouds.yaml file in the host
-* <path_to_sdncs.yaml> [OPTIONAL]: It is the absolute path to the sdncs.yaml file in the host
-* <path_to_kubeconfig> [OPTIONAL]: It is the kubeconfig file to be used for k8s clusters
-* <path_to_jujudata> [OPTIONAL]: It is the jujudata file to be used for juju cli
+Then, run the tests:

 
 ```bash
 docker run --rm=true -t osmtests --env-file <env_file> \
 
 ```bash
 docker run --rm=true -t osmtests --env-file <env_file> \


@@ -114,26 +134,14 @@ docker run --rm=true -t osmtests --env-file <env_file> \
        -v <path_to_clouds.yaml>:/robot-systest/clouds.yaml \
        -v <path_to_sdncs.yaml>:/robot-systest/sdncs.yaml \
        -v <path_to_kubeconfig>:/root/.kube/config \
        -v <path_to_clouds.yaml>:/robot-systest/clouds.yaml \
        -v <path_to_sdncs.yaml>:/robot-systest/sdncs.yaml \
        -v <path_to_kubeconfig>:/root/.kube/config \

-       -v <path_to_jujudata>:/root/.local/share/juju \

        -o <osmclient_version> \
        -p <package_branch> \
        -t <testing_tags>
 ```
 
        -o <osmclient_version> \
        -p <package_branch> \
        -t <testing_tags>
 ```
 

-1. Running the tests manually:
-
-The way of executing the tests is via the following command:
-
-```bash
-source envfile.rc
-robot -d reports -i <testing_tags> testsuite/
-```
-

 ## Test tags
 
 ## Test tags
 

-All tests in the testsuites have tags. Tags allow to run only a set of tests
-identified by a tag. Several tags can be specified when running robot in the
-following way:
+All tests in the testsuites have tags. Tags allow to run only a set of tests identified by a tag. Several tags can be specified when running robot in the following way:

 
 ```bash
 robot -i <tag_01> -i <tag_02> testsuite/
 
 ```bash
 robot -i <tag_01> -i <tag_02> testsuite/


@@ -144,26 +152,70 @@ The following tags exist for each testsuite:
 - A tag per testsuite using its mnemonic (e.g. `basic_01`)
 - Cluster tag for each of the statistically similar tests:
   - `cluster_main`: `basic_01`, `basic_05`, `basic_08`, `basic_09`, `basic_15`,
 - A tag per testsuite using its mnemonic (e.g. `basic_01`)
 - Cluster tag for each of the statistically similar tests:
   - `cluster_main`: `basic_01`, `basic_05`, `basic_08`, `basic_09`, `basic_15`,

-    `basic_16`, basic_17`, `hackfest_basic`, `hackfest_multivdu`,
+    `basic_16`, `basic_17`, `hackfest_basic`, `hackfest_multivdu`,

     `hackfest_cloudinit`, `quotas_01`
   - `cluster_ee_config`: `basic_06`, `basic_07`, `basic_11`, `basic_12`,
     `basic_13`, `basic_14`, `k8s_05`, `k8s_06`
   - `cluster_relations`: `basic_11`, `basic_13`, `basic_14`
   - `cluster_epa`: `epa_01`, `epa_02`, `epa_03`, `epa_04`, `epa_05`
     `hackfest_cloudinit`, `quotas_01`
   - `cluster_ee_config`: `basic_06`, `basic_07`, `basic_11`, `basic_12`,
     `basic_13`, `basic_14`, `k8s_05`, `k8s_06`
   - `cluster_relations`: `basic_11`, `basic_13`, `basic_14`
   - `cluster_epa`: `epa_01`, `epa_02`, `epa_03`, `epa_04`, `epa_05`

-  - `cluster_k8s`: `k8s_01`, `k8s_02`, `k8s_03`, `k8s_04`, `k8s_07`, `k8s_08`, `sa_08`
+  - `cluster_k8s`: `k8s_01`, `k8s_02`, `k8s_03`, `k8s_04`, `k8s_05`, `k8s06`,
+    `k8s_07`, `k8s_08`, `k8s_09`, `k8s_10`, `k8s_11`, `sa_08`

   - `cluster_k8s_charms`: `k8s_05`, `k8s_06`
   - `cluster_sa`: `sa_01`, `sa_02`, `sa_07`
   - `cluster_slices`: `slice_01`, `slice_02`
   - `cluster_k8s_charms`: `k8s_05`, `k8s_06`
   - `cluster_sa`: `sa_01`, `sa_02`, `sa_07`
   - `cluster_slices`: `slice_01`, `slice_02`

+  - `cluster_heal`: `heal_01`, `heal_02`, `heal_03`, `heal_04`
+  - `cluster_sol003`: `sol003_01`

 - daily: for all testsuites that will run in the daily job
 - regression: for all testsuites that should pass in the current stable branch
 - sanity: for all testsuites that should be passed by each commit in the
   stage3 to be successfully verified by Jenkins, currently `basic_07`,
 - daily: for all testsuites that will run in the daily job
 - regression: for all testsuites that should pass in the current stable branch
 - sanity: for all testsuites that should be passed by each commit in the
   stage3 to be successfully verified by Jenkins, currently `basic_07`,

-  `basic_11`, `k8s_03`, `k8s_04`, `sa_02`
+  `basic_11`, `k8s_03`, `k8s_04`, `sa_02`, `hackfest_basic`, `hackfest_cloudinit`

 
 In addition, the tag "cleanup" exists in those tests that perform
 any deletion. In that way, it can be invoked to retry the deletion if
 the tests were forcefully stopped.
 
 
 In addition, the tag "cleanup" exists in those tests that perform
 any deletion. In that way, it can be invoked to retry the deletion if
 the tests were forcefully stopped.
 

+- For helping in the migration tests and other scenarios in which you don't want
+to destroy the deployments immediately, the following tags are used:
+  - `prepare`: for the tests that are used to deploy the network
+  services under test
+  - `verify`: for the tests that perform the actual testing, or changes for
+  additional verifications (e.g. scaling).
+  - `cleanup`: already described above.
+
+  So, for instance, you could first deploy a number of network services executing
+  the tests with "prepare" tag, migrate to another OSM version, and then
+  check the behavior executing with the "verify" tag. Finally, use the "cleanup"
+  tag. 
+
+## Post-processing Robot output files
+
+The output files of Robot include tyipically three files:
+
+- `report.html`: overview of the test execution results in HTML format
+- `log.html`: details about the executed test cases in HTML format
+- `output.xml`: all the test execution results in machine readable XML format
+
+More information about these files [here](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#output-file).
+
+It is possible to use the tool `rebot`, included as part of the Robot Framework, to post-process the output file `output.xml`.
+
+```bash
+# To re-generate log and report from output.xml:
+rebot [-d <output_folder>] output.xml
+
+# To re-generate log and report (and optionally new output.xml) to include only certain tags:
+rebot [-d <output_folder>] -i <tag1> -i <tag2> ... -i <tagN> [-o <new_output_xml>] output.xml
+
+# To re-generate log and report (and optionally new output.xml) excluding certain tags:
+rebot [-d <output_folder>] -e <tag1> -e <tag2> ... -e <tagN> [-o <new_output_xml>] output.xml
+
+# To merge several test executions:
+rebot [-d <output_folder>] --merge output1.xml output2.xml ... outputN.xml
+```
+
+More information about post-processing Robot output files [here](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#post-processing-outputs)
+

 ## Built With
 
 * [Python](www.python.org/) - The language used
 ## Built With
 
 * [Python](www.python.org/) - The language used


@@ -181,4 +233,3 @@ We use [SemVer](http://semver.org/) for versioning. For the versions available,
 
 This project is licensed under the Apache2 License - see the [LICENSE.md](LICENSE) file for details
 
 
 This project is licensed under the Apache2 License - see the [LICENSE.md](LICENSE) file for details
 

-## Acknowledgments