<!--
-Copyright 2020 ETSI
+Copyright ETSI
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
## Prerequisites
-* **Robot Framework**
-* **Packages**: ssh ping yq git
-* **Python3 packages**: haikunator requests robotframework robotframework-seleniumlibrary robotframework-requests robotframework-jsonlibrary robotframework-sshlibrary
-* Clone **osm-packages** from gitlab
-* Environment config file for your infrastructure [envfile.rc]
+- **Robot Framework**
+- **Packages**: ssh ping yq git
+- **Python3 packages**: haikunator requests robotframework robotframework-seleniumlibrary robotframework-requests robotframework-jsonlibrary robotframework-sshlibrary
+- Clone **osm-packages** from gitlab
+- Environment config file for your infrastructure `envfile.rc`
## Installing
This bash script can be used to setup your environment to execute the tests.
```bash
-PACKAGES_FOLDER=osm-packages
-add-apt-repository -y ppa:rmescandon/yq && apt update && apt install yq git iputils-ping ssh -y
-pip install haikunator requests robotframework robotframework-seleniumlibrary robotframework-requests robotframework-jsonlibrary \
- robotframework-sshlibrary
-snap install charm
+python3 -m pip install -r requirements.txt
+python3 -m pip install -r requirements-dev.txt
# Download community packages
+PACKAGES_FOLDER=osm-packages
git clone https://osm.etsi.org/gitlab/vnf-onboarding/osm-packages.git ${PACKAGES_FOLDER}
```
-envfile.rc
+Configure a file `envfile.rc` copying from `envconfig-local.rc` and set the required variables
```bash
# VIM Setup
ROBOT_REPORT_FOLDER=results
```
-## Deployment
+## Running the tests
-It is possible to run the tests directly from the repository or using a docker container with the tests
+### From the host machine
-1. Docker container creation:
+If you have installed all the dependecnies, the way of executing the tests is via the following command:
+
+```bash
+source envfile.rc
+robot -d reports -i <testing_tags> testsuite/
+```
+
+### From docker container
+
+It is possible to run the tests directly from the repository or using a docker container with the tests.
+
+Create the docker container:
```bash
docker build -f docker/Dockerfile -t osmtests .
Options:
-* --env-file: It is the environmental file where is described the OSM target and VIM
-* -o <osmclient_version> [OPTIONAL]: It is used to specify a particular osmclient version. Default: latest
-* -p <package_branch> [OPTIONAL]: OSM packages repository branch. Default: master
-* -t <testing_tags> [OPTIONAL]: Robot tests tags. [sanity, regression, particular_test]. Default: sanity
+- `--env-file`: It is the environmental file where is described the OSM target and VIM
+- `-o <osmclient_version>` [OPTIONAL]: It is used to specify a particular osmclient version. Default: latest
+- `-p <package_branch>` [OPTIONAL]: OSM packages repository branch. Default: master
+- `-t <testing_tags>` [OPTIONAL]: Robot tests tags. [sanity, regression, particular_test]. Default: sanity
Volumes:
-* <path_to_reports> [OPTIONAL]: It is the absolute path to reports location in the host
-* <path_to_clouds.yaml> [OPTIONAL]: It is the absolute path to the clouds.yaml file in the host
-* <path_to_sdncs.yaml> [OPTIONAL]: It is the absolute path to the sdncs.yaml file in the host
-* <path_to_kubeconfig> [OPTIONAL]: It is the kubeconfig file to be used for k8s clusters
+- <path_to_reports> [OPTIONAL]: It is the absolute path to reports location in the host
+- <path_to_clouds.yaml> [OPTIONAL]: It is the absolute path to the clouds.yaml file in the host
+- <path_to_sdncs.yaml> [OPTIONAL]: It is the absolute path to the sdncs.yaml file in the host
+- <path_to_kubeconfig> [OPTIONAL]: It is the kubeconfig file to be used for k8s clusters
+
+Then, run the tests:
```bash
docker run --rm=true -t osmtests --env-file <env_file> \
-t <testing_tags>
```
-1. Running the tests manually:
+### From an environment identical to OSM CICD
+
+````bash
+git clone https://osm.etsi.org/gerrit/osm/devops
+git clone https://osm.etsi.org/gerrit/osm/IM
+git clone https://osm.etsi.org/gerrit/osm/osmclient
+git clone https://osm.etsi.org/gerrit/osm/tests
+# run HTTP server to server artifacts
+devops/tools/local-build.sh --install-qhttpd
+# generate debian packages locally that will be served by the HTTP server
+devops/tools/local-build.sh --module IM,osmclient,tests stage-2
+# create docker image and store it locally as opensourcemano/tests:devel
+devops/tools/local-build.sh --module tests
+```
-The way of executing the tests is via the following command:
+Then, run the tests:
```bash
-source envfile.rc
-robot -d reports -i <testing_tags> testsuite/
+docker run --rm=true -t osmtests --env-file <env_file> \
+ -v <path_to_reports>:/reports osmtests \
+ -v <path_to_clouds.yaml>:/robot-systest/clouds.yaml \
+ -v <path_to_sdncs.yaml>:/robot-systest/sdncs.yaml \
+ -v <path_to_kubeconfig>:/root/.kube/config \
+ -o <osmclient_version> \
+ -p <package_branch> \
+ -t <testing_tags>
```
## Test tags
- A tag per testsuite using its mnemonic (e.g. `basic_01`)
- Cluster tag for each of the statistically similar tests:
- `cluster_main`: `basic_01`, `basic_05`, `basic_08`, `basic_09`, `basic_15`,
- `basic_16`, basic_17`, `hackfest_basic`, `hackfest_multivdu`,
+ `basic_16`, `basic_17`, `hackfest_basic`, `hackfest_multivdu`,
`hackfest_cloudinit`, `quotas_01`
- `cluster_ee_config`: `basic_06`, `basic_07`, `basic_11`, `basic_12`,
`basic_13`, `basic_14`, `k8s_05`, `k8s_06`
- `cluster_relations`: `basic_11`, `basic_13`, `basic_14`
- `cluster_epa`: `epa_01`, `epa_02`, `epa_03`, `epa_04`, `epa_05`
- - `cluster_k8s`: `k8s_01`, `k8s_02`, `k8s_03`, `k8s_04`, `k8s_07`, `k8s_08`, `sa_08`
+ - `cluster_k8s`: `k8s_01`, `k8s_02`, `k8s_03`, `k8s_04`, `k8s_05`, `k8s06`,
+ `k8s_07`, `k8s_08`, `k8s_09`, `k8s_10`, `k8s_11`, `sa_08`
- `cluster_k8s_charms`: `k8s_05`, `k8s_06`
- `cluster_sa`: `sa_01`, `sa_02`, `sa_07`
- `cluster_slices`: `slice_01`, `slice_02`
+ - `cluster_heal`: `heal_01`, `heal_02`, `heal_03`, `heal_04`
+ - `cluster_sol003`: `sol003_01`
- daily: for all testsuites that will run in the daily job
- regression: for all testsuites that should pass in the current stable branch
- sanity: for all testsuites that should be passed by each commit in the
stage3 to be successfully verified by Jenkins, currently `basic_07`,
- `basic_11`, `k8s_03`, `k8s_04`, `sa_02`
+ `basic_11`, `k8s_03`, `k8s_04`, `sa_02`, `hackfest_basic`, `hackfest_cloudinit`
In addition, the tag "cleanup" exists in those tests that perform
any deletion. In that way, it can be invoked to retry the deletion if
check the behavior executing with the "verify" tag. Finally, use the "cleanup"
tag.
+## Post-processing Robot output files
+
+The output files of Robot include tyipically three files:
+
+- `report.html`: overview of the test execution results in HTML format
+- `log.html`: details about the executed test cases in HTML format
+- `output.xml`: all the test execution results in machine readable XML format
+
+More information about these files [here](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#output-file).
+
+It is possible to use the tool `rebot`, included as part of the Robot Framework, to post-process the output file `output.xml`.
+
+```bash
+# To re-generate log and report from output.xml:
+rebot [-d <output_folder>] output.xml
+
+# To re-generate log and report (and optionally new output.xml) to include only certain tags:
+rebot [-d <output_folder>] -i <tag1> -i <tag2> ... -i <tagN> [-o <new_output_xml>] output.xml
+
+# To re-generate log and report (and optionally new output.xml) excluding certain tags:
+rebot [-d <output_folder>] -e <tag1> -e <tag2> ... -e <tagN> [-o <new_output_xml>] output.xml
+
+# To merge several test executions:
+rebot [-d <output_folder>] --merge output1.xml output2.xml ... outputN.xml
+```
+
+More information about post-processing Robot output files [here](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#post-processing-outputs)
+
## Built With
* [Python](www.python.org/) - The language used
This project is licensed under the Apache2 License - see the [LICENSE.md](LICENSE) file for details
-## Acknowledgments