4 Licensed under the Apache License, Version 2.0 (the "License");
5 you may not use this file except in compliance with the License.
6 You may obtain a copy of the License at
8 http://www.apache.org/licenses/LICENSE-2.0
10 Unless required by applicable law or agreed to in writing, software
11 distributed under the License is distributed on an "AS IS" BASIS,
12 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
14 See the License for the specific language governing permissions and
15 limitations under the License
18 # OSM test automation project - osm/tests
20 This repository contains tools and configuration files for testing and automation needs of OSM projet
25 - VIM already registered in OSM
26 - K8s cluster already registered in OSM (for tests involving a K8s cluster)
28 ## Quickstart (run tests using docker)
30 ### Configure the environment file
33 export OSM_HOSTNAME=<OSM_IP_ADDRESS>
34 export VIM_TARGET=<VIM_REGISTERED_AT_OSM>
35 export VIM_MGMT_NET=<NAME_OF_THE_MGMT_NETWORK_IN_THE_VIM>
36 export K8S_CREDENTIALS= # path to the kubeconfig file of the K8s cluster to be tested
37 export OS_CLOUD= # OpenStack Cloud defined in $HOME/.config/openstack/clouds.yaml or in /etc/openstack/clouds.yaml
40 ### Create the docker container
43 docker build -f docker/Dockerfile -t osmtests .
49 docker run --rm=true --name tests -t --env-file envconfig.rc \
50 -v ~/.config/openstack/clouds.yaml:/etc/openstack/clouds.yaml \
51 -v ~/tests/reports:/robot-systest/reports \
56 You can use a different robot tag instead of `sol003_01`. The whole list of tags are gathered below in this README.
58 ## How to run sanity tests using daily images from master branch
61 docker run --rm=true --name tests -t --env-file envconfig.rc \
62 -v ~/.config/openstack/clouds.yaml:/etc/openstack/clouds.yaml \
63 -v ~/tests/reports:/robot-systest/reports \
64 opensourcemano/tests:testing-daily \
68 ## How to mount local tests folder for developing purposes
70 The following line will mount the required files for `sol003_01` testuite and will execute that testsuite:
73 docker run --rm=true --name tests -t --env-file envconfig.rc \
74 -v ~/.config/openstack/clouds.yaml:/etc/openstack/clouds.yaml \
75 -v ~/tests/reports:/robot-systest/reports \
76 -v ~/tests/robot-systest/lib/sol003_common_lib.robot:/robot-systest/lib/sol003_common_lib.robot \
81 Relevant volumes to be mounted are:
83 - <path_to_reports> [OPTIONAL]: the absolute path to reports location in the host
84 - <path_to_clouds.yaml> [OPTIONAL]: the absolute path to the clouds.yaml file in the host
85 - <path_to_sdncs.yaml> [OPTIONAL]: the absolute path to the sdncs.yaml file in the host
86 - <path_to_kubeconfig> [OPTIONAL]: the kubeconfig file to be used for k8s clusters
88 Other relevant options to run tests are:
90 - `--env-file`: It is the environmental file where is described the OSM target and VIM
91 - `-o <osmclient_version>` [OPTIONAL]: It is used to specify a particular osmclient version. Default: latest
92 - `-p <package_branch>` [OPTIONAL]: OSM packages repository branch. Default: master
93 - `-t <testing_tags>` [OPTIONAL]: Robot tests tags. [sanity, regression, particular_test]. Default: sanity
96 ## How to run tests from a host
100 This bash script can be used to setup your environment to execute the tests.
104 sudo apt-get install ssh ping yq git
105 # Python packages used for the tests
106 python3 -m pip install -r requirements.txt
107 python3 -m pip install -r requirements-dev.txt
108 # Download community packages
109 git clone https://osm.etsi.org/gitlab/vnf-onboarding/osm-packages.git
112 ### Configure the environment
114 Create a file `envfile.rc` copying from `envconfig-local.rc` and set the required variables.
116 ### Running the tests
121 robot -d reports -i <testing_tags> testsuite/
124 ## How to run tests from an environment identical to OSM CICD
127 git clone https://osm.etsi.org/gerrit/osm/devops
128 git clone https://osm.etsi.org/gerrit/osm/IM
129 git clone https://osm.etsi.org/gerrit/osm/osmclient
130 git clone https://osm.etsi.org/gerrit/osm/tests
131 # run HTTP server to server artifacts
132 devops/tools/local-build.sh --install-qhttpd
133 # generate debian packages locally that will be served by the HTTP server
134 devops/tools/local-build.sh --module IM,osmclient,tests stage-2
135 # create docker image and store it locally as opensourcemano/tests:devel
136 devops/tools/local-build.sh --module tests
142 docker run --rm=true -t osmtests --env-file <env_file> \
143 -v <path_to_reports>:/reports osmtests \
144 -v <path_to_clouds.yaml>:/robot-systest/clouds.yaml \
145 -v <path_to_sdncs.yaml>:/robot-systest/sdncs.yaml \
146 -v <path_to_kubeconfig>:/root/.kube/config \
147 -o <osmclient_version> \
148 -p <package_branch> \
154 All tests in the testsuites have tags. Tags allow to run only a set of tests identified by a tag. Several tags can be specified when running robot in the following way:
157 robot -i <tag_01> -i <tag_02> testsuite/
160 The following tags exist for each testsuite:
162 - A tag per testsuite using its mnemonic (e.g. `basic_01`)
163 - Cluster tag for each of the statistically similar tests:
164 - `cluster_main`: `basic_01`, `basic_05`, `basic_08`, `basic_09`, `basic_15`,
165 `basic_16`, `basic_17`, `hackfest_basic`, `hackfest_multivdu`,
166 `hackfest_cloudinit`, `quotas_01`
167 - `cluster_ee_config`: `basic_06`, `basic_07`, `basic_11`, `basic_12`,
168 `basic_13`, `basic_14`, `k8s_05`, `k8s_06`
169 - `cluster_relations`: `basic_11`, `basic_13`, `basic_14`
170 - `cluster_epa`: `epa_01`, `epa_02`, `epa_03`, `epa_04`, `epa_05`
171 - `cluster_k8s`: `k8s_01`, `k8s_02`, `k8s_03`, `k8s_04`, `k8s_05`, `k8s06`,
172 `k8s_07`, `k8s_08`, `k8s_09`, `k8s_10`, `k8s_11`, `sa_08`
173 - `cluster_k8s_charms`: `k8s_05`, `k8s_06`
174 - `cluster_sa`: `sa_01`, `sa_02`, `sa_07`
175 - `cluster_slices`: `slice_01`, `slice_02`
176 - `cluster_heal`: `heal_01`, `heal_02`, `heal_03`, `heal_04`
177 - `cluster_osm_rest`: `sol003_01`
178 - daily: for all testsuites that will run in the daily job
179 - regression: for all testsuites that should pass in the current stable branch
180 - sanity: for all testsuites that should be passed by each commit in the
181 stage3 to be successfully verified by Jenkins, currently `basic_07`,
182 `basic_11`, `k8s_03`, `k8s_04`, `sa_02`, `hackfest_basic`, `hackfest_cloudinit`
184 In addition, the tag "cleanup" exists in those tests that perform
185 any deletion. In that way, it can be invoked to retry the deletion if
186 the tests were forcefully stopped.
188 - For helping in the migration tests and other scenarios in which you don't want
189 to destroy the deployments immediately, the following tags are used:
190 - `prepare`: for the tests that are used to deploy the network
192 - `verify`: for the tests that perform the actual testing, or changes for
193 additional verifications (e.g. scaling).
194 - `cleanup`: already described above.
196 So, for instance, you could first deploy a number of network services executing
197 the tests with "prepare" tag, migrate to another OSM version, and then
198 check the behavior executing with the "verify" tag. Finally, use the "cleanup"
201 ## Post-processing Robot output files
203 The output files of Robot include tyipically three files:
205 - `report.html`: overview of the test execution results in HTML format
206 - `log.html`: details about the executed test cases in HTML format
207 - `output.xml`: all the test execution results in machine readable XML format
209 More information about these files [here](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#output-file).
211 It is possible to use the tool `rebot`, included as part of the Robot Framework, to post-process the output file `output.xml`.
214 # To re-generate log and report from output.xml:
215 rebot [-d <output_folder>] output.xml
217 # To re-generate log and report (and optionally new output.xml) to include only certain tags:
218 rebot [-d <output_folder>] -i <tag1> -i <tag2> ... -i <tagN> [-o <new_output_xml>] output.xml
220 # To re-generate log and report (and optionally new output.xml) excluding certain tags:
221 rebot [-d <output_folder>] -e <tag1> -e <tag2> ... -e <tagN> [-o <new_output_xml>] output.xml
223 # To merge several test executions:
224 rebot [-d <output_folder>] --merge output1.xml output2.xml ... outputN.xml
227 More information about post-processing Robot output files [here](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#post-processing-outputs)
231 * [Python](www.python.org/) - The language used
232 * [Robot Framework](robotframework.org) - The testing framework
236 Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on our code of conduct, and the process for submitting pull requests to us.
240 We use [SemVer](http://semver.org/) for versioning. For the versions available, see the [tags on this repository](https://osm.etsi.org/gitweb/?p=osm/tests.git;a=tags).
244 This project is licensed under the Apache2 License - see the [LICENSE.md](LICENSE) file for details