Gitiles
Code Review Sign In
osm.etsi.org / osm / tests / 4487e9a020225f2cf891f01d63c5853e9f2deb94
commit4487e9a020225f2cf891f01d63c5853e9f2deb94[log] [tgz]
authorgarciadeblas <gerardo.garciadeblas@telefonica.com>Sun Dec 17 01:37:47 2023 +0100
committergarciadeblas <gerardo.garciadeblas@telefonica.com>Sun Dec 17 01:37:47 2023 +0100
tree3489e6e6269e10be6157286bad7d40ff8e71c2b3
parent166c8913bba24b153968d0a739313b129e3d3432 [diff]
Remove daily tag from tests using charm packages

Change-Id: I514dfc178c059e4c614a15f3a49b923089c7ce0f
Signed-off-by: garciadeblas <gerardo.garciadeblas@telefonica.com>
3 files changed
tree: 3489e6e6269e10be6157286bad7d40ff8e71c2b3
  1. .gitignore
  2. .gitlab-ci.yml
  3. .pylintrc
  4. CONTRIBUTING.md
  5. Dockerfile
  6. Jenkinsfile
  7. LICENSE
  8. README.md
  9. README_tst010_robot_cicd.md
  10. build-debpkg.sh
  11. charm.sh
  12. cloud-scripts/
  13. clouds-local.yaml
  14. conformance-tests/
  15. debian/
  16. devops-stages/
  17. docker/
  18. envconfig-local.rc
  19. requirements-dev.in
  20. requirements-dev.txt
  21. requirements.in
  22. requirements.txt
  23. robot-systest/
  24. sdncs-local.yaml
  25. setup.py
  26. tox.ini
README.md

OSM test automation project - osm/tests

This repository contains tools and configuration files for testing and automation needs of OSM project.

Prerequisites

  • OSM running
  • VIM already registered in OSM
  • K8s cluster already registered in OSM (for tests involving a K8s cluster)

Quickstart. How to run tests using OSM docker images

Configure the environment file

Create a file envconfig.rc copying from envconfig-local.rc and set the required variables.

OSM_HOSTNAME=<OSM_IP_ADDRESS>
VIM_TARGET=<VIM_REGISTERED_AT_OSM>
VIM_MGMT_NET=<NAME_OF_THE_MGMT_NETWORK_IN_THE_VIM>
OS_CLOUD=<OPENSTACK_CLOUD>    # OpenStack Cloud defined in $HOME/.config/openstack/clouds.yaml or in /etc/openstack/clouds.yaml

Run the tests

docker run --rm=true --name tests -t --env-file envconfig.rc \
           -v ~/.config/openstack/clouds.yaml:/etc/openstack/clouds.yaml \
           -v ~/tests/reports:/robot-systest/reports \
           opensourcemano/tests:testing-daily \
           -t sanity

You can use a different robot tag instead of sanity. The whole list of tags are gathered below in this README.

How to build docker container for tests and run tests from there

Create the docker container

docker build -f docker/Dockerfile -t osmtests .

Run the tests

docker run --rm=true --name tests -t --env-file envconfig.rc \
           -v ~/.config/openstack/clouds.yaml:/etc/openstack/clouds.yaml \
           -v ~/tests/reports:/robot-systest/reports \
           osmtests \
           -t sol003_01

How to mount local tests folder for developing purposes

The following line will mount the folder robot-systest, including all libraries and testuites, and will execute the testsuite sol003_01:

docker run --rm=true --name tests -t --env-file envconfig.rc \
           -v ~/.config/openstack/clouds.yaml:/etc/openstack/clouds.yaml \
           -v ~/tests/robot-systest:/robot-systest \
           -v ~/tests/reports:/robot-systest/reports \
           -v ~/osm-packages:/robot-systest/osm-packages \
           opensourcemano/tests:testing-daily \
           -t sol003_01

Relevant volumes to be mounted are:

  • <path_to_reports> [OPTIONAL]: the absolute path to reports location in the host
  • <path_to_clouds.yaml> [OPTIONAL]: the absolute path to the clouds.yaml file in the host
  • <path_to_sdncs.yaml> [OPTIONAL]: the absolute path to the sdncs.yaml file in the host
  • <path_to_kubeconfig> [OPTIONAL]: the kubeconfig file to be used for k8s clusters

Other relevant options to run tests are:

  • --env-file: It is the environmental file where is described the OSM target and VIM
  • -o <osmclient_version> [OPTIONAL]: It is used to specify a particular osmclient version. Default: latest
  • -p <package_branch> [OPTIONAL]: OSM packages repository branch. Default: master
  • -t <testing_tags> [OPTIONAL]: Robot tests tags. [sanity, regression, particular_test]. Default: sanity

How to run tests from a host

In general, testing from docker images is the best way if you want to develop for OSM. However, sometimes it could be useful to run tests directly from the host.

Install dependencies

This bash script can be used to setup your environment to execute the tests.

sudo apt-get update
sudo apt-get install ssh ping yq git
# Python packages used for the tests
python3 -m pip install -r requirements.txt
python3 -m pip install -r requirements-dev.txt
# Download community packages
git clone https://osm.etsi.org/gitlab/vnf-onboarding/osm-packages.git

Configure the environment

Create a file envconfig.rc copying from envconfig-local.rc and set the required variables (in this case, the use of export is mandatory).

export OSM_HOSTNAME=<OSM_IP_ADDRESS>
export VIM_TARGET=<VIM_REGISTERED_AT_OSM>
export VIM_MGMT_NET=<NAME_OF_THE_MGMT_NETWORK_IN_THE_VIM>
export OS_CLOUD=<OPENSTACK_CLOUD>    # OpenStack Cloud defined in $HOME/.config/openstack/clouds.yaml or in /etc/openstack/clouds.yaml
export K8S_CREDENTIALS= # path to the kubeconfig file of the K8s cluster to be tested
export OCI_REGISTRY_URL= # URL of the OCI registry where helm charts used in test packages are stored. 
export OCI_REGISTRY_USER= # User of the OCI registry
export OCI_REGISTRY_PASSWORD= # Password of the OCI registry

Running the tests

source envconfig.rc
mkdir reports
robot -d reports -i <testing_tags> testsuite/

How to run tests from an environment identical to OSM CICD

git clone https://osm.etsi.org/gerrit/osm/devops
git clone https://osm.etsi.org/gerrit/osm/IM
git clone https://osm.etsi.org/gerrit/osm/osmclient
git clone https://osm.etsi.org/gerrit/osm/tests
# run HTTP server to server artifacts
devops/tools/local-build.sh --install-qhttpd
# generate debian packages locally that will be served by the HTTP server
devops/tools/local-build.sh --module IM,osmclient,tests stage-2
# create docker image and store it locally as opensourcemano/tests:devel
devops/tools/local-build.sh --module tests

Then, run the tests:

docker run --rm=true -t osmtests --env-file <env_file> \
       -v <path_to_reports>:/reports osmtests \
       -v <path_to_clouds.yaml>:/robot-systest/clouds.yaml \
       -v <path_to_sdncs.yaml>:/robot-systest/sdncs.yaml \
       -v <path_to_kubeconfig>:/root/.kube/config \
       -o <osmclient_version> \
       -p <package_branch> \
       -t <testing_tags>

Test tags

All tests in the testsuites have tags. Tags allow to run only a set of tests identified by a tag. Several tags can be specified when running robot in the following way:

robot -i <tag_01> -i <tag_02> testsuite/

The following tags exist for each testsuite:

  • A tag per testsuite using its mnemonic (e.g. basic_01)
  • Cluster tag for each of the statistically similar tests:
    • cluster_main: basic_01, basic_05, basic_08, basic_09, basic_15, basic_16, basic_17, hackfest_basic, hackfest_multivdu, hackfest_cloudinit, quotas_01
    • cluster_ee_config: basic_06, basic_07, basic_11, basic_12, basic_13, basic_14, k8s_05, k8s_06
    • cluster_relations: basic_11, basic_13, basic_14
    • cluster_epa: epa_01, epa_02, epa_03, epa_04, epa_05
    • cluster_k8s: k8s_01, k8s_02, k8s_03, k8s_04, k8s_05, k8s06, k8s_07, k8s_08, k8s_09, k8s_10, k8s_11, k8s_12, k8s_13, sa_08
    • cluster_k8s_charms: k8s_05, k8s_06
    • cluster_sa: sa_01, sa_02, sa_07
    • cluster_slices: slice_01, slice_02
    • cluster_heal: heal_01, heal_02, heal_03, heal_04
    • cluster_osm_rest: sol003_01
  • daily: for all testsuites that will run in the daily job
  • regression: for all testsuites that should pass in the current stable branch
  • sanity: for all testsuites that should be passed by each commit in the stage3 to be successfully verified by Jenkins, currently k8s_04, sa_02, hackfest_basic, hackfest_cloudinit

In addition, the tag "cleanup" exists in those tests that perform any deletion. In that way, it can be invoked to retry the deletion if the tests were forcefully stopped.

  • For helping in the migration tests and other scenarios in which you don't want to destroy the deployments immediately, the following tags are used:

    • prepare: for the tests that are used to deploy the network services under test
    • verify: for the tests that perform the actual testing, or changes for additional verifications (e.g. scaling).
    • cleanup: already described above.

    So, for instance, you could first deploy a number of network services executing the tests with "prepare" tag, migrate to another OSM version, and then check the behavior executing with the "verify" tag. Finally, use the "cleanup" tag.

Post-processing Robot output files

The output files of Robot include tyipically three files:

  • report.html: overview of the test execution results in HTML format
  • log.html: details about the executed test cases in HTML format
  • output.xml: all the test execution results in machine readable XML format

More information about these files here.

It is possible to use the tool rebot, included as part of the Robot Framework, to post-process the output file output.xml.

# To re-generate log and report from output.xml:
rebot [-d <output_folder>] output.xml

# To re-generate log and report (and optionally new output.xml) to include only certain tags:
rebot [-d <output_folder>] -i <tag1> -i <tag2> ... -i <tagN> [-o <new_output_xml>] output.xml

# To re-generate log and report (and optionally new output.xml) excluding certain tags:
rebot [-d <output_folder>] -e <tag1> -e <tag2> ... -e <tagN> [-o <new_output_xml>] output.xml

# To merge several test executions:
rebot [-d <output_folder>] --merge output1.xml output2.xml ... outputN.xml

More information about post-processing Robot output files here

Built With

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

License

This project is licensed under the Apache2 License - see the LICENSE file for details