commit	fccb028b493afafb8a602953ed5125303423d3ff	[log] [tgz]
author	garciadeblas <gerardo.garciadeblas@telefonica.com>	Fri Jun 24 12:39:33 2022 +0200
committer	garciadeblas <gerardo.garciadeblas@telefonica.com>	Mon Aug 08 11:12:04 2022 +0200
tree	34fb3516f8f28cb36f2f6bf55fb2b09d1dc1d28a
parent	58710335f2db1e82ec171784891b7cbee71fd360 [diff]

tree: 34fb3516f8f28cb36f2f6bf55fb2b09d1dc1d28a

README.md

OSM test automation project - osm/tests

This repository contains tools and configuration files for testing and automation needs of OSM projet

Prerequisites

Robot Framework
Packages: ssh ping yq git
Python3 packages: haikunator requests robotframework robotframework-seleniumlibrary robotframework-requests robotframework-jsonlibrary robotframework-sshlibrary
Clone osm-packages from gitlab
Environment config file for your infrastructure envfile.rc

Installing

This bash script can be used to setup your environment to execute the tests.

python3 -m pip install -r requirements.txt
python3 -m pip install -r requirements-dev.txt
# Download community packages
PACKAGES_FOLDER=osm-packages
git clone https://osm.etsi.org/gitlab/vnf-onboarding/osm-packages.git ${PACKAGES_FOLDER}

Configure a file envfile.rc copying from envconfig-local.rc and set the required variables

# VIM Setup
OS_USERNAME=<openstack_username>
OS_PASSWORD=<openstack_password>
OS_TENANT_NAME=<openstack_tenant_name>
OS_AUTH_URL=<openstack_authorization_url>
OS_TENANT_ID=<openstack_tenant_id>

# OSM Setup
OSM_HOSTNAME=<osm_ip_address>
VIM_TARGET=<osm_vim_name>
VIM_MGMT_NET=<osm_vim_mgmt_name>

# Clouds file datacenter
OS_CLOUD=<datacenter_in_clouds_file>
# SDNCs file
OS_SDNC=<SDN_controller_in_sdncs_file>

# K8S config file
K8S_CREDENTIALS=<path_to_kubeconfig>

# The following set of environment variables will be used in host
# of the robot framework. Not needed for docker execution

# Folder where Robot tests are stored
ROBOT_DEVOPS_FOLDER=robot-systest

# Folder to save alternative DUT environments (optional)
ENVIRONMENTS_FOLDER=environments

# Folder where all required packages are stored
PACKAGES_FOLDER=osm-packages

# Folder where test results should be exported
ROBOT_REPORT_FOLDER=results

Running the tests

From the host machine

If you have installed all the dependecnies, the way of executing the tests is via the following command:

source envfile.rc
robot -d reports -i <testing_tags> testsuite/

From docker container

It is possible to run the tests directly from the repository or using a docker container with the tests.

Create the docker container:

docker build -f docker/Dockerfile -t osmtests .

Options:

--env-file: It is the environmental file where is described the OSM target and VIM
-o <osmclient_version> [OPTIONAL]: It is used to specify a particular osmclient version. Default: latest
-p <package_branch> [OPTIONAL]: OSM packages repository branch. Default: master
-t <testing_tags> [OPTIONAL]: Robot tests tags. [sanity, regression, particular_test]. Default: sanity

Volumes:

<path_to_reports> [OPTIONAL]: It is the absolute path to reports location in the host
<path_to_clouds.yaml> [OPTIONAL]: It is the absolute path to the clouds.yaml file in the host
<path_to_sdncs.yaml> [OPTIONAL]: It is the absolute path to the sdncs.yaml file in the host
<path_to_kubeconfig> [OPTIONAL]: It is the kubeconfig file to be used for k8s clusters

Then, run the tests:

docker run --rm=true -t osmtests --env-file <env_file> \
       -v <path_to_reports>:/reports osmtests \
       -v <path_to_clouds.yaml>:/robot-systest/clouds.yaml \
       -v <path_to_sdncs.yaml>:/robot-systest/sdncs.yaml \
       -v <path_to_kubeconfig>:/root/.kube/config \
       -o <osmclient_version> \
       -p <package_branch> \
       -t <testing_tags>

From an environment identical to OSM CICD

git clone https://osm.etsi.org/gerrit/osm/devops
git clone https://osm.etsi.org/gerrit/osm/IM
git clone https://osm.etsi.org/gerrit/osm/osmclient
git clone https://osm.etsi.org/gerrit/osm/tests
# run HTTP server to server artifacts
devops/tools/local-build.sh --install-qhttpd
# generate debian packages locally that will be served by the HTTP server
devops/tools/local-build.sh --module IM,osmclient,tests stage-2
# create docker image and store it locally as opensourcemano/tests:devel
devops/tools/local-build.sh --module tests
```

Then, run the tests:

```bash
docker run --rm=true -t osmtests --env-file <env_file> \
       -v <path_to_reports>:/reports osmtests \
       -v <path_to_clouds.yaml>:/robot-systest/clouds.yaml \
       -v <path_to_sdncs.yaml>:/robot-systest/sdncs.yaml \
       -v <path_to_kubeconfig>:/root/.kube/config \
       -o <osmclient_version> \
       -p <package_branch> \
       -t <testing_tags>
```

## Test tags

All tests in the testsuites have tags. Tags allow to run only a set of tests
identified by a tag. Several tags can be specified when running robot in the
following way:

```bash
robot -i <tag_01> -i <tag_02> testsuite/
```

The following tags exist for each testsuite:

- A tag per testsuite using its mnemonic (e.g. `basic_01`)
- Cluster tag for each of the statistically similar tests:
  - `cluster_main`: `basic_01`, `basic_05`, `basic_08`, `basic_09`, `basic_15`,
    `basic_16`, `basic_17`, `hackfest_basic`, `hackfest_multivdu`,
    `hackfest_cloudinit`, `quotas_01`
  - `cluster_ee_config`: `basic_06`, `basic_07`, `basic_11`, `basic_12`,
    `basic_13`, `basic_14`, `k8s_05`, `k8s_06`
  - `cluster_relations`: `basic_11`, `basic_13`, `basic_14`
  - `cluster_epa`: `epa_01`, `epa_02`, `epa_03`, `epa_04`, `epa_05`
  - `cluster_k8s`: `k8s_01`, `k8s_02`, `k8s_03`, `k8s_04`, `k8s_05`, `k8s06`,
    `k8s_07`, `k8s_08`, `k8s_09`, `k8s_10`, `k8s_11`, `sa_08`
  - `cluster_k8s_charms`: `k8s_05`, `k8s_06`
  - `cluster_sa`: `sa_01`, `sa_02`, `sa_07`
  - `cluster_slices`: `slice_01`, `slice_02`
- daily: for all testsuites that will run in the daily job
- regression: for all testsuites that should pass in the current stable branch
- sanity: for all testsuites that should be passed by each commit in the
  stage3 to be successfully verified by Jenkins, currently `basic_07`,
  `basic_11`, `k8s_03`, `k8s_04`, `sa_02`, `hackfest_basic`, `hackfest_cloudinit`

In addition, the tag "cleanup" exists in those tests that perform
any deletion. In that way, it can be invoked to retry the deletion if
the tests were forcefully stopped.

- For helping in the migration tests and other scenarios in which you don't want
to destroy the deployments immediately, the following tags are used:
  - prepare: for the tests that are used to deploy the network
  services under test
  - verify: for the tests that perform the actual testing, or changes for
  additional verifications (e.g. scaling).
  - cleanup: already described above.

  So, for instance, you could first deploy a number of network services executing
  the tests with "prepare" tag, migrate to another OSM version, and then
  check the behavior executing with the "verify" tag. Finally, use the "cleanup"
  tag. 

## Post-processing Robot output files

The output files of Robot include tyipically three files:

- `report.html`: overview of the test execution results in HTML format
- `log.html`: details about the executed test cases in HTML format
- `output.xml`: all the test execution results in machine readable XML format

More information about these files [here](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#output-file).

It is possible to use the tool `rebot`, included as part of the Robot Framework, to post-process the output file `output.xml`.

```bash
# To re-generate log and report from output.xml:
rebot [-d <output_folder>] output.xml

# To re-generate log and report (and optionally new output.xml) to include only certain tags:
rebot [-d <output_folder>] -i <tag1> -i <tag2> ... -i <tagN> [-o <new_output_xml>] output.xml

# To re-generate log and report (and optionally new output.xml) excluding certain tags:
rebot [-d <output_folder>] -e <tag1> -e <tag2> ... -e <tagN> [-o <new_output_xml>] output.xml

# To merge several test executions:
rebot [-d <output_folder>] --merge output1.xml output2.xml ... outputN.xml
```

More information about post-processing Robot output files [here](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#post-processing-outputs)

## Built With

* [Python](www.python.org/) - The language used
* [Robot Framework](robotframework.org) - The testing framework

## Contributing

Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on our code of conduct, and the process for submitting pull requests to us.

## Versioning

We use [SemVer](http://semver.org/) for versioning. For the versions available, see the [tags on this repository](https://osm.etsi.org/gitweb/?p=osm/tests.git;a=tags).

## License

This project is licensed under the Apache2 License - see the [LICENSE.md](LICENSE) file for details