Newer
Older
Copyright ETSI
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
See the License for the specific language governing permissions and
limitations under the License
-->
This repository contains tools and configuration files for testing and automation needs of OSM projet
- OSM running
- VIM already registered in OSM
- K8s cluster already registered in OSM (for tests involving a K8s cluster)
## Quickstart (run tests using docker)
### Configure the environment file
```
export OSM_HOSTNAME=<OSM_IP_ADDRESS>
export VIM_TARGET=<VIM_REGISTERED_AT_OSM>
export VIM_MGMT_NET=<NAME_OF_THE_MGMT_NETWORK_IN_THE_VIM>
export K8S_CREDENTIALS= # path to the kubeconfig file of the K8s cluster to be tested
export OS_CLOUD= # OpenStack Cloud defined in $HOME/.config/openstack/clouds.yaml or in /etc/openstack/clouds.yaml
```bash
docker build -f docker/Dockerfile -t osmtests .
```
```bash
docker run --rm=true --name tests -t --env-file envconfig.rc \
-v ~/.config/openstack/clouds.yaml:/etc/openstack/clouds.yaml \
-v ~/tests/reports:/robot-systest/reports \
osmtests \
-t sol003_01
```
You can use a different robot tag instead of `sol003_01`. The whole list of tags are gathered below in this README.
## How to mount local tests folder for developing purposes
The following line will mount the required files for SOL003_01 testuite and will execute the test `sol003_01`
docker run --rm=true --name tests -t --env-file envconfig.rc \
-v ~/.config/openstack/clouds.yaml:/etc/openstack/clouds.yaml \
-v ~/tests/reports:/robot-systest/reports \
-v ~/tests/robot-systest/lib/sol003_common_lib.robot:/robot-systest/lib/sol003_common_lib.robot \
osmtests \
-t sol003_01
Relevant volumes to be mounted are:
- <path_to_reports> [OPTIONAL]: the absolute path to reports location in the host
- <path_to_clouds.yaml> [OPTIONAL]: the absolute path to the clouds.yaml file in the host
- <path_to_sdncs.yaml> [OPTIONAL]: the absolute path to the sdncs.yaml file in the host
- <path_to_kubeconfig> [OPTIONAL]: the kubeconfig file to be used for k8s clusters
Other relevant options to run tests are:
- `--env-file`: It is the environmental file where is described the OSM target and VIM
- `-o <osmclient_version>` [OPTIONAL]: It is used to specify a particular osmclient version. Default: latest
- `-p <package_branch>` [OPTIONAL]: OSM packages repository branch. Default: master
- `-t <testing_tags>` [OPTIONAL]: Robot tests tags. [sanity, regression, particular_test]. Default: sanity
### Installing
This bash script can be used to setup your environment to execute the tests.
sudo apt-get update
sudo apt-get install ssh ping yq git
# Python packages used for the tests
python3 -m pip install -r requirements.txt
python3 -m pip install -r requirements-dev.txt
# Download community packages
git clone https://osm.etsi.org/gitlab/vnf-onboarding/osm-packages.git
```
### Configure the environment
Create a file `envfile.rc` copying from `envconfig-local.rc` and set the required variables.
### Running the tests
```bash
source envfile.rc
mkdir reports
robot -d reports -i <testing_tags> testsuite/
## How to run tests from an environment identical to OSM CICD
git clone https://osm.etsi.org/gerrit/osm/devops
git clone https://osm.etsi.org/gerrit/osm/IM
git clone https://osm.etsi.org/gerrit/osm/osmclient
git clone https://osm.etsi.org/gerrit/osm/tests
# run HTTP server to server artifacts
devops/tools/local-build.sh --install-qhttpd
# generate debian packages locally that will be served by the HTTP server
devops/tools/local-build.sh --module IM,osmclient,tests stage-2
# create docker image and store it locally as opensourcemano/tests:devel
devops/tools/local-build.sh --module tests
```
Then, run the tests:
docker run --rm=true -t osmtests --env-file <env_file> \
-v <path_to_reports>:/reports osmtests \
-v <path_to_clouds.yaml>:/robot-systest/clouds.yaml \
-v <path_to_sdncs.yaml>:/robot-systest/sdncs.yaml \
-v <path_to_kubeconfig>:/root/.kube/config \
-o <osmclient_version> \
-p <package_branch> \
-t <testing_tags>
All tests in the testsuites have tags. Tags allow to run only a set of tests identified by a tag. Several tags can be specified when running robot in the following way:
```bash
robot -i <tag_01> -i <tag_02> testsuite/
```
The following tags exist for each testsuite:
- A tag per testsuite using its mnemonic (e.g. `basic_01`)
- Cluster tag for each of the statistically similar tests:
- `cluster_main`: `basic_01`, `basic_05`, `basic_08`, `basic_09`, `basic_15`,
`basic_16`, `basic_17`, `hackfest_basic`, `hackfest_multivdu`,
`hackfest_cloudinit`, `quotas_01`
- `cluster_ee_config`: `basic_06`, `basic_07`, `basic_11`, `basic_12`,
`basic_13`, `basic_14`, `k8s_05`, `k8s_06`
- `cluster_relations`: `basic_11`, `basic_13`, `basic_14`
- `cluster_epa`: `epa_01`, `epa_02`, `epa_03`, `epa_04`, `epa_05`
- `cluster_k8s`: `k8s_01`, `k8s_02`, `k8s_03`, `k8s_04`, `k8s_05`, `k8s06`,
`k8s_07`, `k8s_08`, `k8s_09`, `k8s_10`, `k8s_11`, `sa_08`
- `cluster_k8s_charms`: `k8s_05`, `k8s_06`
- `cluster_sa`: `sa_01`, `sa_02`, `sa_07`
- `cluster_slices`: `slice_01`, `slice_02`
- `cluster_heal`: `heal_01`, `heal_02`, `heal_03`, `heal_04`
- daily: for all testsuites that will run in the daily job
- regression: for all testsuites that should pass in the current stable branch
- sanity: for all testsuites that should be passed by each commit in the
stage3 to be successfully verified by Jenkins, currently `basic_07`,
`basic_11`, `k8s_03`, `k8s_04`, `sa_02`, `hackfest_basic`, `hackfest_cloudinit`
In addition, the tag "cleanup" exists in those tests that perform
any deletion. In that way, it can be invoked to retry the deletion if
the tests were forcefully stopped.
rodriguezgar
committed
- For helping in the migration tests and other scenarios in which you don't want
to destroy the deployments immediately, the following tags are used:
- `prepare`: for the tests that are used to deploy the network
rodriguezgar
committed
services under test
- `verify`: for the tests that perform the actual testing, or changes for
rodriguezgar
committed
additional verifications (e.g. scaling).
- `cleanup`: already described above.
rodriguezgar
committed
So, for instance, you could first deploy a number of network services executing
the tests with "prepare" tag, migrate to another OSM version, and then
check the behavior executing with the "verify" tag. Finally, use the "cleanup"
tag.
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
## Post-processing Robot output files
The output files of Robot include tyipically three files:
- `report.html`: overview of the test execution results in HTML format
- `log.html`: details about the executed test cases in HTML format
- `output.xml`: all the test execution results in machine readable XML format
More information about these files [here](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#output-file).
It is possible to use the tool `rebot`, included as part of the Robot Framework, to post-process the output file `output.xml`.
```bash
# To re-generate log and report from output.xml:
rebot [-d <output_folder>] output.xml
# To re-generate log and report (and optionally new output.xml) to include only certain tags:
rebot [-d <output_folder>] -i <tag1> -i <tag2> ... -i <tagN> [-o <new_output_xml>] output.xml
# To re-generate log and report (and optionally new output.xml) excluding certain tags:
rebot [-d <output_folder>] -e <tag1> -e <tag2> ... -e <tagN> [-o <new_output_xml>] output.xml
# To merge several test executions:
rebot [-d <output_folder>] --merge output1.xml output2.xml ... outputN.xml
```
More information about post-processing Robot output files [here](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#post-processing-outputs)
## Built With
* [Python](www.python.org/) - The language used
* [Robot Framework](robotframework.org) - The testing framework
## Contributing
Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on our code of conduct, and the process for submitting pull requests to us.
## Versioning
We use [SemVer](http://semver.org/) for versioning. For the versions available, see the [tags on this repository](https://osm.etsi.org/gitweb/?p=osm/tests.git;a=tags).
## License
This project is licensed under the Apache2 License - see the [LICENSE.md](LICENSE) file for details