4 Licensed under the Apache License, Version 2.0 (the "License");
5 you may not use this file except in compliance with the License.
6 You may obtain a copy of the License at
8 http://www.apache.org/licenses/LICENSE-2.0
10 Unless required by applicable law or agreed to in writing, software
11 distributed under the License is distributed on an "AS IS" BASIS,
12 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
14 See the License for the specific language governing permissions and
15 limitations under the License
18 # OSM test automation project - osm/tests
20 This repository contains tools and configuration files for testing and automation needs of OSM projet
25 - **Packages**: ssh ping yq git
26 - **Python3 packages**: haikunator requests robotframework robotframework-seleniumlibrary robotframework-requests robotframework-jsonlibrary robotframework-sshlibrary
27 - Clone **osm-packages** from gitlab
28 - Environment config file for your infrastructure `envfile.rc`
32 This bash script can be used to setup your environment to execute the tests.
35 python3 -m pip install -r requirements.txt
36 python3 -m pip install -r requirements-dev.txt
37 # Download community packages
38 PACKAGES_FOLDER=osm-packages
39 git clone https://osm.etsi.org/gitlab/vnf-onboarding/osm-packages.git ${PACKAGES_FOLDER}
42 ## Configure the environment
44 Create a file `envfile.rc` copying from `envconfig-local.rc` and set the required variables.
48 ### From the host machine
50 If you have installed all the dependencies, the way of executing the tests is via the following command:
54 robot -d reports -i <testing_tags> testsuite/
57 ### From docker container
59 It is possible to run the tests directly from the repository or using a docker container with the tests.
61 Create the docker container:
64 docker build -f docker/Dockerfile -t osmtests .
69 - `--env-file`: It is the environmental file where is described the OSM target and VIM
70 - `-o <osmclient_version>` [OPTIONAL]: It is used to specify a particular osmclient version. Default: latest
71 - `-p <package_branch>` [OPTIONAL]: OSM packages repository branch. Default: master
72 - `-t <testing_tags>` [OPTIONAL]: Robot tests tags. [sanity, regression, particular_test]. Default: sanity
76 - <path_to_reports> [OPTIONAL]: It is the absolute path to reports location in the host
77 - <path_to_clouds.yaml> [OPTIONAL]: It is the absolute path to the clouds.yaml file in the host
78 - <path_to_sdncs.yaml> [OPTIONAL]: It is the absolute path to the sdncs.yaml file in the host
79 - <path_to_kubeconfig> [OPTIONAL]: It is the kubeconfig file to be used for k8s clusters
84 docker run --rm=true -t osmtests --env-file <env_file> \
85 -v <path_to_reports>:/reports osmtests \
86 -v <path_to_clouds.yaml>:/robot-systest/clouds.yaml \
87 -v <path_to_sdncs.yaml>:/robot-systest/sdncs.yaml \
88 -v <path_to_kubeconfig>:/root/.kube/config \
89 -o <osmclient_version> \
94 ### From an environment identical to OSM CICD
97 git clone https://osm.etsi.org/gerrit/osm/devops
98 git clone https://osm.etsi.org/gerrit/osm/IM
99 git clone https://osm.etsi.org/gerrit/osm/osmclient
100 git clone https://osm.etsi.org/gerrit/osm/tests
101 # run HTTP server to server artifacts
102 devops/tools/local-build.sh --install-qhttpd
103 # generate debian packages locally that will be served by the HTTP server
104 devops/tools/local-build.sh --module IM,osmclient,tests stage-2
105 # create docker image and store it locally as opensourcemano/tests:devel
106 devops/tools/local-build.sh --module tests
112 docker run --rm=true -t osmtests --env-file <env_file> \
113 -v <path_to_reports>:/reports osmtests \
114 -v <path_to_clouds.yaml>:/robot-systest/clouds.yaml \
115 -v <path_to_sdncs.yaml>:/robot-systest/sdncs.yaml \
116 -v <path_to_kubeconfig>:/root/.kube/config \
117 -o <osmclient_version> \
118 -p <package_branch> \
124 All tests in the testsuites have tags. Tags allow to run only a set of tests
125 identified by a tag. Several tags can be specified when running robot in the
129 robot -i <tag_01> -i <tag_02> testsuite/
132 The following tags exist for each testsuite:
134 - A tag per testsuite using its mnemonic (e.g. `basic_01`)
135 - Cluster tag for each of the statistically similar tests:
136 - `cluster_main`: `basic_01`, `basic_05`, `basic_08`, `basic_09`, `basic_15`,
137 `basic_16`, `basic_17`, `hackfest_basic`, `hackfest_multivdu`,
138 `hackfest_cloudinit`, `quotas_01`
139 - `cluster_ee_config`: `basic_06`, `basic_07`, `basic_11`, `basic_12`,
140 `basic_13`, `basic_14`, `k8s_05`, `k8s_06`
141 - `cluster_relations`: `basic_11`, `basic_13`, `basic_14`
142 - `cluster_epa`: `epa_01`, `epa_02`, `epa_03`, `epa_04`, `epa_05`
143 - `cluster_k8s`: `k8s_01`, `k8s_02`, `k8s_03`, `k8s_04`, `k8s_05`, `k8s06`,
144 `k8s_07`, `k8s_08`, `k8s_09`, `k8s_10`, `k8s_11`, `sa_08`
145 - `cluster_k8s_charms`: `k8s_05`, `k8s_06`
146 - `cluster_sa`: `sa_01`, `sa_02`, `sa_07`
147 - `cluster_slices`: `slice_01`, `slice_02`
148 - `cluster_heal`: `heal_01`, `heal_02`, `heal_03`, `heal_04`
149 - `cluster_sol003`: `sol003_01`
150 - daily: for all testsuites that will run in the daily job
151 - regression: for all testsuites that should pass in the current stable branch
152 - sanity: for all testsuites that should be passed by each commit in the
153 stage3 to be successfully verified by Jenkins, currently `basic_07`,
154 `basic_11`, `k8s_03`, `k8s_04`, `sa_02`, `hackfest_basic`, `hackfest_cloudinit`
156 In addition, the tag "cleanup" exists in those tests that perform
157 any deletion. In that way, it can be invoked to retry the deletion if
158 the tests were forcefully stopped.
160 - For helping in the migration tests and other scenarios in which you don't want
161 to destroy the deployments immediately, the following tags are used:
162 - `prepare`: for the tests that are used to deploy the network
164 - `verify`: for the tests that perform the actual testing, or changes for
165 additional verifications (e.g. scaling).
166 - `cleanup`: already described above.
168 So, for instance, you could first deploy a number of network services executing
169 the tests with "prepare" tag, migrate to another OSM version, and then
170 check the behavior executing with the "verify" tag. Finally, use the "cleanup"
173 ## Post-processing Robot output files
175 The output files of Robot include tyipically three files:
177 - `report.html`: overview of the test execution results in HTML format
178 - `log.html`: details about the executed test cases in HTML format
179 - `output.xml`: all the test execution results in machine readable XML format
181 More information about these files [here](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#output-file).
183 It is possible to use the tool `rebot`, included as part of the Robot Framework, to post-process the output file `output.xml`.
186 # To re-generate log and report from output.xml:
187 rebot [-d <output_folder>] output.xml
189 # To re-generate log and report (and optionally new output.xml) to include only certain tags:
190 rebot [-d <output_folder>] -i <tag1> -i <tag2> ... -i <tagN> [-o <new_output_xml>] output.xml
192 # To re-generate log and report (and optionally new output.xml) excluding certain tags:
193 rebot [-d <output_folder>] -e <tag1> -e <tag2> ... -e <tagN> [-o <new_output_xml>] output.xml
195 # To merge several test executions:
196 rebot [-d <output_folder>] --merge output1.xml output2.xml ... outputN.xml
199 More information about post-processing Robot output files [here](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#post-processing-outputs)
203 * [Python](www.python.org/) - The language used
204 * [Robot Framework](robotframework.org) - The testing framework
208 Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on our code of conduct, and the process for submitting pull requests to us.
212 We use [SemVer](http://semver.org/) for versioning. For the versions available, see the [tags on this repository](https://osm.etsi.org/gitweb/?p=osm/tests.git;a=tags).
216 This project is licensed under the Apache2 License - see the [LICENSE.md](LICENSE) file for details