commit	62a540df9a8d122d437b8536cc3bccecde327e7e	[log] [tgz]
author	Dario Faccin <dario.faccin@canonical.com>	Wed May 24 17:24:48 2023 +0200
committer	Dario Faccin <dario.faccin@canonical.com>	Thu May 25 10:55:34 2023 +0200
tree	53bc2faff629958e2959cec88e1d4fbd89882724
parent	26777da970e43bf2d6313d0c15a950a0eee15b47 [diff]

Update from master

Squashed commit of the following:

commit 541f9cf287bf6c7dca45fe911f8e844988711fa3
Author: garciadeblas <gerardo.garciadeblas@telefonica.com>
Date:   Tue May 23 11:59:47 2023 +0200

    Fix Fail01 to get openstack quota properly after updating client version

    Change-Id: Ief7b4d5f150d67cecf944021c2598456672a30e0
    Signed-off-by: garciadeblas <gerardo.garciadeblas@telefonica.com>

commit a86e6938d53e345b08a7ca9cc9615115f14a002a
Author: garciadeblas <gerardo.garciadeblas@telefonica.com>
Date:   Fri May 19 12:47:09 2023 +0200

    Update SA02 to take into account the VNF threshold-time

    Change-Id: Idde7fa330e8e9b141053c15edb266817c4e699c6
    Signed-off-by: garciadeblas <gerardo.garciadeblas@telefonica.com>

commit c753f67c813fa82926b3d64db759fd7d0d7aa2cc
Author: garciadeblas <gerardo.garciadeblas@telefonica.com>
Date:   Thu May 18 15:06:49 2023 +0200

    Fix error message in run_test.sh for create_k8scluster

    Change-Id: Ice2e0bc5e829e31a81cbde30ea6784de64d8ca63
    Signed-off-by: garciadeblas <gerardo.garciadeblas@telefonica.com>

commit b9dfe0317c4935294747897dc2fd1645c4983067
Author: garciadeblas <gerardo.garciadeblas@telefonica.com>
Date:   Wed May 10 17:25:47 2023 +0200

    Support of Python3.10

    This change includes required changes to make tests work in Python3.10.
    It also enables tox running as part of stage-test, which was not included before.
    For the moment, only flake8 and black envs will be checked. Some pylint
    errors have been fixed, but a few remain unsolved. That's why pylint is
    run, but its result is ignored.

    Change-Id: I29105bf849eab8b47f92627a7eea1ede31b8e05b
    Signed-off-by: garciadeblas <gerardo.garciadeblas@telefonica.com>

commit 3d7579ccc2356a23ea32878e8f0fbb0e75f59b68
Author: garciadeblas <gerardo.garciadeblas@telefonica.com>
Date:   Tue Apr 18 15:04:57 2023 +0200

    Clean stage-archive.sh and use allowlist_extenals in tox.ini

    Change-Id: I690598c770f52d805ee3516fead7fa191c90205f
    Signed-off-by: garciadeblas <gerardo.garciadeblas@telefonica.com>

commit 43f34481b4c5ccfda407a7f82a124012aab0dccc
Author: garciadeblas <gerardo.garciadeblas@telefonica.com>
Date:   Thu Feb 23 17:37:47 2023 +0100

    Fix bug 2220 to solve sol003_01 test

    Change-Id: Ic4376067a8b42e470c09ba0668c545db8ce08e55
    Signed-off-by: garciadeblas <gerardo.garciadeblas@telefonica.com>

Change-Id: Icde649f40b72a6a358ef6686fa8840f8502b2259
Signed-off-by: Dario Faccin <dario.faccin@canonical.com>

25 files changed

tree: 53bc2faff629958e2959cec88e1d4fbd89882724

README.md

OSM test automation project - osm/tests

This repository contains tools and configuration files for testing and automation needs of OSM projet

Prerequisites

Robot Framework
Packages: ssh ping yq git
Python3 packages: haikunator requests robotframework robotframework-seleniumlibrary robotframework-requests robotframework-jsonlibrary robotframework-sshlibrary
Clone osm-packages from gitlab
Environment config file for your infrastructure envfile.rc

Installing

This bash script can be used to setup your environment to execute the tests.

python3 -m pip install -r requirements.txt
python3 -m pip install -r requirements-dev.txt
# Download community packages
PACKAGES_FOLDER=osm-packages
git clone https://osm.etsi.org/gitlab/vnf-onboarding/osm-packages.git ${PACKAGES_FOLDER}

Configure the environment

Create a file envfile.rc copying from envconfig-local.rc and set the required variables.

Running the tests

From the host machine

If you have installed all the dependencies, the way of executing the tests is via the following command:

source envfile.rc
robot -d reports -i <testing_tags> testsuite/

From docker container

It is possible to run the tests directly from the repository or using a docker container with the tests.

Create the docker container:

docker build -f docker/Dockerfile -t osmtests .

Options:

--env-file: It is the environmental file where is described the OSM target and VIM
-o <osmclient_version> [OPTIONAL]: It is used to specify a particular osmclient version. Default: latest
-p <package_branch> [OPTIONAL]: OSM packages repository branch. Default: master
-t <testing_tags> [OPTIONAL]: Robot tests tags. [sanity, regression, particular_test]. Default: sanity

Volumes:

<path_to_reports> [OPTIONAL]: It is the absolute path to reports location in the host
<path_to_clouds.yaml> [OPTIONAL]: It is the absolute path to the clouds.yaml file in the host
<path_to_sdncs.yaml> [OPTIONAL]: It is the absolute path to the sdncs.yaml file in the host
<path_to_kubeconfig> [OPTIONAL]: It is the kubeconfig file to be used for k8s clusters

Then, run the tests:

docker run --rm=true -t osmtests --env-file <env_file> \
       -v <path_to_reports>:/reports osmtests \
       -v <path_to_clouds.yaml>:/robot-systest/clouds.yaml \
       -v <path_to_sdncs.yaml>:/robot-systest/sdncs.yaml \
       -v <path_to_kubeconfig>:/root/.kube/config \
       -o <osmclient_version> \
       -p <package_branch> \
       -t <testing_tags>

From an environment identical to OSM CICD

git clone https://osm.etsi.org/gerrit/osm/devops
git clone https://osm.etsi.org/gerrit/osm/IM
git clone https://osm.etsi.org/gerrit/osm/osmclient
git clone https://osm.etsi.org/gerrit/osm/tests
# run HTTP server to server artifacts
devops/tools/local-build.sh --install-qhttpd
# generate debian packages locally that will be served by the HTTP server
devops/tools/local-build.sh --module IM,osmclient,tests stage-2
# create docker image and store it locally as opensourcemano/tests:devel
devops/tools/local-build.sh --module tests

Then, run the tests:

docker run --rm=true -t osmtests --env-file <env_file> \
       -v <path_to_reports>:/reports osmtests \
       -v <path_to_clouds.yaml>:/robot-systest/clouds.yaml \
       -v <path_to_sdncs.yaml>:/robot-systest/sdncs.yaml \
       -v <path_to_kubeconfig>:/root/.kube/config \
       -o <osmclient_version> \
       -p <package_branch> \
       -t <testing_tags>

Test tags

All tests in the testsuites have tags. Tags allow to run only a set of tests identified by a tag. Several tags can be specified when running robot in the following way:

robot -i <tag_01> -i <tag_02> testsuite/

The following tags exist for each testsuite:

A tag per testsuite using its mnemonic (e.g. basic_01)
Cluster tag for each of the statistically similar tests:
- cluster_main: basic_01, basic_05, basic_08, basic_09, basic_15, basic_16, basic_17, hackfest_basic, hackfest_multivdu, hackfest_cloudinit, quotas_01
- cluster_ee_config: basic_06, basic_07, basic_11, basic_12, basic_13, basic_14, k8s_05, k8s_06
- cluster_relations: basic_11, basic_13, basic_14
- cluster_epa: epa_01, epa_02, epa_03, epa_04, epa_05
- cluster_k8s: k8s_01, k8s_02, k8s_03, k8s_04, k8s_05, k8s06, k8s_07, k8s_08, k8s_09, k8s_10, k8s_11, sa_08
- cluster_k8s_charms: k8s_05, k8s_06
- cluster_sa: sa_01, sa_02, sa_07
- cluster_slices: slice_01, slice_02
- cluster_heal: heal_01, heal_02, heal_03, heal_04
- cluster_sol003: sol003_01
daily: for all testsuites that will run in the daily job
regression: for all testsuites that should pass in the current stable branch
sanity: for all testsuites that should be passed by each commit in the stage3 to be successfully verified by Jenkins, currently basic_07, basic_11, k8s_03, k8s_04, sa_02, hackfest_basic, hackfest_cloudinit

In addition, the tag "cleanup" exists in those tests that perform any deletion. In that way, it can be invoked to retry the deletion if the tests were forcefully stopped.

For helping in the migration tests and other scenarios in which you don't want to destroy the deployments immediately, the following tags are used:
- prepare: for the tests that are used to deploy the network services under test
- verify: for the tests that perform the actual testing, or changes for additional verifications (e.g. scaling).
- cleanup: already described above.
So, for instance, you could first deploy a number of network services executing the tests with "prepare" tag, migrate to another OSM version, and then check the behavior executing with the "verify" tag. Finally, use the "cleanup" tag.

Post-processing Robot output files

The output files of Robot include tyipically three files:

report.html: overview of the test execution results in HTML format
log.html: details about the executed test cases in HTML format
output.xml: all the test execution results in machine readable XML format

More information about these files here.

It is possible to use the tool rebot, included as part of the Robot Framework, to post-process the output file output.xml.

# To re-generate log and report from output.xml:
rebot [-d <output_folder>] output.xml

# To re-generate log and report (and optionally new output.xml) to include only certain tags:
rebot [-d <output_folder>] -i <tag1> -i <tag2> ... -i <tagN> [-o <new_output_xml>] output.xml

# To re-generate log and report (and optionally new output.xml) excluding certain tags:
rebot [-d <output_folder>] -e <tag1> -e <tag2> ... -e <tagN> [-o <new_output_xml>] output.xml

# To merge several test executions:
rebot [-d <output_folder>] --merge output1.xml output2.xml ... outputN.xml

More information about post-processing Robot output files here

Built With

Python - The language used
Robot Framework - The testing framework

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

License

This project is licensed under the Apache2 License - see the LICENSE.md file for details