<!--
-Copyright 2020 ETSI
+Copyright ETSI
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
See the License for the specific language governing permissions and
limitations under the License
-->
-# Project Title
-One Paragraph of project description goes here
+# OSM test automation project - osm/tests
-## Getting Started
+This repository contains tools and configuration files for testing and automation needs of OSM project.
-These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.
+## Prerequisites
-### Prerequisites
+- OSM running
+- VIM already registered in OSM
+- K8s cluster already registered in OSM (for tests involving a K8s cluster)
-What things you need to install the software and how to install them
+## Quickstart. How to run tests using OSM docker images
+
+### Configure the environment file
+
+Create a file `envconfig.rc` copying from `envconfig-local.rc` and set the required variables.
```
-Give examples
+OSM_HOSTNAME=<OSM_IP_ADDRESS>
+VIM_TARGET=<VIM_REGISTERED_AT_OSM>
+VIM_MGMT_NET=<NAME_OF_THE_MGMT_NETWORK_IN_THE_VIM>
+OS_CLOUD=<OPENSTACK_CLOUD> # OpenStack Cloud defined in $HOME/.config/openstack/clouds.yaml or in /etc/openstack/clouds.yaml
+```
+
+### Run the tests
+
+```bash
+docker run --rm=true --name tests -t --env-file envconfig.rc \
+ -v ~/.config/openstack/clouds.yaml:/etc/openstack/clouds.yaml \
+ -v ~/tests/reports:/robot-systest/reports \
+ opensourcemano/tests:testing-daily \
+ -t sanity
```
-### Installing
+You can use a different robot tag instead of `sanity`. The whole list of tags are gathered below in this README.
+
+## How to build docker container for tests and run tests from there
-A step by step series of examples that tell you how to get a development env running
+### Create the docker container
+
+```bash
+docker build -f docker/Dockerfile -t osmtests .
+```
-Say what the step will be
+### Run the tests
+```bash
+docker run --rm=true --name tests -t --env-file envconfig.rc \
+ -v ~/.config/openstack/clouds.yaml:/etc/openstack/clouds.yaml \
+ -v ~/tests/reports:/robot-systest/reports \
+ osmtests \
+ -t sol003_01
```
-Give the example
+
+## How to mount local tests folder for developing purposes
+
+The following line will mount the folder `robot-systest`, including all libraries and testuites, and will execute the testsuite `sol003_01`:
+
+```bash
+docker run --rm=true --name tests -t --env-file envconfig.rc \
+ -v ~/.config/openstack/clouds.yaml:/etc/openstack/clouds.yaml \
+ -v ~/tests/robot-systest:/robot-systest \
+ -v ~/tests/reports:/robot-systest/reports \
+ -v ~/osm-packages:/robot-systest/osm-packages \
+ opensourcemano/tests:testing-daily \
+ -t sol003_01
+```
+
+Relevant volumes to be mounted are:
+
+- <path_to_reports> [OPTIONAL]: the absolute path to reports location in the host
+- <path_to_clouds.yaml> [OPTIONAL]: the absolute path to the clouds.yaml file in the host
+- <path_to_sdncs.yaml> [OPTIONAL]: the absolute path to the sdncs.yaml file in the host
+- <path_to_kubeconfig> [OPTIONAL]: the kubeconfig file to be used for k8s clusters
+
+Other relevant options to run tests are:
+
+- `--env-file`: It is the environmental file where is described the OSM target and VIM
+- `-o <osmclient_version>` [OPTIONAL]: It is used to specify a particular osmclient version. Default: latest
+- `-p <package_branch>` [OPTIONAL]: OSM packages repository branch. Default: master
+- `-t <testing_tags>` [OPTIONAL]: Robot tests tags. [sanity, regression, particular_test]. Default: sanity
+
+## How to run tests from a host
+
+In general, testing from docker images is the best way if you want to develop for OSM. However, sometimes it could be useful to run tests directly from the host.
+
+### Install dependencies
+
+This bash script can be used to setup your environment to execute the tests.
+
+```bash
+sudo apt-get update
+sudo apt-get install ssh ping yq git
+# Python packages used for the tests
+python3 -m pip install -r requirements.txt
+python3 -m pip install -r requirements-dev.txt
+# Download community packages
+git clone https://osm.etsi.org/gitlab/vnf-onboarding/osm-packages.git
```
-And repeat
+### Configure the environment
+
+Create a file `envconfig.rc` copying from `envconfig-local.rc` and set the required variables (in this case, the use of `export` is mandatory).
```
-until finished
+export OSM_HOSTNAME=<OSM_IP_ADDRESS>
+export VIM_TARGET=<VIM_REGISTERED_AT_OSM>
+export VIM_MGMT_NET=<NAME_OF_THE_MGMT_NETWORK_IN_THE_VIM>
+export OS_CLOUD=<OPENSTACK_CLOUD> # OpenStack Cloud defined in $HOME/.config/openstack/clouds.yaml or in /etc/openstack/clouds.yaml
+export K8S_CREDENTIALS= # path to the kubeconfig file of the K8s cluster to be tested
+export OCI_REGISTRY_URL= # URL of the OCI registry where helm charts used in test packages are stored.
+export OCI_REGISTRY_USER= # User of the OCI registry
+export OCI_REGISTRY_PASSWORD= # Password of the OCI registry
```
-End with an example of getting some data out of the system or using it for a little demo
+### Running the tests
-## Running the tests
+```bash
+source envconfig.rc
+mkdir reports
+robot -d reports -i <testing_tags> testsuite/
+```
-Explain how to run the automated tests for this system
+## How to run tests from an environment identical to OSM CICD
+
+```bash
+git clone https://osm.etsi.org/gerrit/osm/devops
+git clone https://osm.etsi.org/gerrit/osm/IM
+git clone https://osm.etsi.org/gerrit/osm/osmclient
+git clone https://osm.etsi.org/gerrit/osm/tests
+# run HTTP server to server artifacts
+devops/tools/local-build.sh --install-qhttpd
+# generate debian packages locally that will be served by the HTTP server
+devops/tools/local-build.sh --module IM,osmclient,tests stage-2
+# create docker image and store it locally as opensourcemano/tests:devel
+devops/tools/local-build.sh --module tests
+```
+Then, run the tests:
+
+```bash
+docker run --rm=true -t osmtests --env-file <env_file> \
+ -v <path_to_reports>:/reports osmtests \
+ -v <path_to_clouds.yaml>:/robot-systest/clouds.yaml \
+ -v <path_to_sdncs.yaml>:/robot-systest/sdncs.yaml \
+ -v <path_to_kubeconfig>:/root/.kube/config \
+ -o <osmclient_version> \
+ -p <package_branch> \
+ -t <testing_tags>
```
-Give an example
+
+## Test tags
+
+All tests in the testsuites have tags. Tags allow to run only a set of tests identified by a tag. Several tags can be specified when running robot in the following way:
+
+```bash
+robot -i <tag_01> -i <tag_02> testsuite/
```
-## Deployment
+The following tags exist for each testsuite:
+
+- A tag per testsuite using its mnemonic (e.g. `basic_01`)
+- Cluster tag for each of the statistically similar tests:
+ - `cluster_main`: `basic_01`, `basic_05`, `basic_08`, `basic_09`, `basic_15`,
+ `basic_16`, `basic_17`, `hackfest_basic`, `hackfest_multivdu`,
+ `hackfest_cloudinit`, `quotas_01`
+ - `cluster_ee_config`: `basic_06`, `basic_07`, `basic_11`, `basic_12`,
+ `basic_13`, `basic_14`, `k8s_05`, `k8s_06`
+ - `cluster_relations`: `basic_11`, `basic_13`, `basic_14`
+ - `cluster_epa`: `epa_01`, `epa_02`, `epa_03`, `epa_04`, `epa_05`
+ - `cluster_k8s`: `k8s_01`, `k8s_02`, `k8s_03`, `k8s_04`, `k8s_05`, `k8s06`,
+ `k8s_07`, `k8s_08`, `k8s_09`, `k8s_10`, `k8s_11`, `k8s_12`, `k8s_13`, `sa_08`
+ - `cluster_k8s_charms`: `k8s_05`, `k8s_06`
+ - `cluster_sa`: `sa_01`, `sa_02`, `sa_07`
+ - `cluster_slices`: `slice_01`, `slice_02`
+ - `cluster_heal`: `heal_01`, `heal_02`, `heal_03`, `heal_04`
+ - `cluster_osm_rest`: `sol003_01`
+- daily: for all testsuites that will run in the daily job
+- regression: for all testsuites that should pass in the current stable branch
+- sanity: for all testsuites that should be passed by each commit in the
+ stage3 to be successfully verified by Jenkins, currently `k8s_04`,
+ `sa_02`, `hackfest_basic`, `hackfest_cloudinit`
+
+In addition, the tag "cleanup" exists in those tests that perform
+any deletion. In that way, it can be invoked to retry the deletion if
+the tests were forcefully stopped.
+
+- For helping in the migration tests and other scenarios in which you don't want
+to destroy the deployments immediately, the following tags are used:
+ - `prepare`: for the tests that are used to deploy the network
+ services under test
+ - `verify`: for the tests that perform the actual testing, or changes for
+ additional verifications (e.g. scaling).
+ - `cleanup`: already described above.
+
+ So, for instance, you could first deploy a number of network services executing
+ the tests with "prepare" tag, migrate to another OSM version, and then
+ check the behavior executing with the "verify" tag. Finally, use the "cleanup"
+ tag.
+
+## Post-processing Robot output files
+
+The output files of Robot include tyipically three files:
+
+- `report.html`: overview of the test execution results in HTML format
+- `log.html`: details about the executed test cases in HTML format
+- `output.xml`: all the test execution results in machine readable XML format
+
+More information about these files [here](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#output-file).
+
+It is possible to use the tool `rebot`, included as part of the Robot Framework, to post-process the output file `output.xml`.
+
+```bash
+# To re-generate log and report from output.xml:
+rebot [-d <output_folder>] output.xml
+
+# To re-generate log and report (and optionally new output.xml) to include only certain tags:
+rebot [-d <output_folder>] -i <tag1> -i <tag2> ... -i <tagN> [-o <new_output_xml>] output.xml
+
+# To re-generate log and report (and optionally new output.xml) excluding certain tags:
+rebot [-d <output_folder>] -e <tag1> -e <tag2> ... -e <tagN> [-o <new_output_xml>] output.xml
+
+# To merge several test executions:
+rebot [-d <output_folder>] --merge output1.xml output2.xml ... outputN.xml
+```
-Add additional notes about how to deploy this on a live system
+More information about post-processing Robot output files [here](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#post-processing-outputs)
## Built With
* [Python](www.python.org/) - The language used
+* [Robot Framework](robotframework.org) - The testing framework
## Contributing
## Versioning
-We use [SemVer](http://semver.org/) for versioning. For the versions available, see the [tags on this repository](https://osm.etsi.org/gitweb/?p=osm/TEMPLATE.git;a=tags).
+We use [SemVer](http://semver.org/) for versioning. For the versions available, see the [tags on this repository](https://osm.etsi.org/gitweb/?p=osm/tests.git;a=tags).
## License
-This project is licensed under the Apache2 License - see the [LICENSE.md](LICENSE) file for details
-
-## Acknowledgments
-
-* **Billie Thompson** - *Initial work* - [PurpleBooth](https://github.com/PurpleBooth)
+This project is licensed under the Apache2 License - see the [LICENSE](LICENSE) file for details
projects / osm / tests.git / blobdiff