X-Git-Url: https://osm.etsi.org/gitweb/?p=osm%2Ftests.git;a=blobdiff_plain;f=README.md;h=7eff36f0c74702a7c4a048ae6b2094939f2cf537;hp=86013229f759ddd6745c7cac27fa29aa9a63af30;hb=HEAD;hpb=83721f3f6fba26d9137d0271af9ed700bb615caa diff --git a/README.md b/README.md index 8601322..7be9050 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@ -# Project Title -One Paragraph of project description goes here +# OSM test automation project - osm/tests -## Getting Started +This repository contains tools and configuration files for testing and automation needs of OSM project. -These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system. +## Prerequisites -### Prerequisites +- OSM running +- VIM already registered in OSM +- K8s cluster already registered in OSM (for tests involving a K8s cluster) -What things you need to install the software and how to install them +## Quickstart. How to run tests using OSM docker images + +### Configure the environment file + +Create a file `envconfig.rc` copying from `envconfig-local.rc` and set the required variables. ``` -Give examples +OSM_HOSTNAME= +VIM_TARGET= +VIM_MGMT_NET= +OS_CLOUD= # OpenStack Cloud defined in $HOME/.config/openstack/clouds.yaml or in /etc/openstack/clouds.yaml +``` + +### Run the tests + +```bash +docker run --rm=true --name tests -t --env-file envconfig.rc \ + -v ~/.config/openstack/clouds.yaml:/etc/openstack/clouds.yaml \ + -v ~/tests/reports:/robot-systest/reports \ + opensourcemano/tests:testing-daily \ + -t sanity ``` -### Installing +You can use a different robot tag instead of `sanity`. The whole list of tags are gathered below in this README. + +## How to build docker container for tests and run tests from there -A step by step series of examples that tell you how to get a development env running +### Create the docker container + +```bash +docker build -f docker/Dockerfile -t osmtests . +``` -Say what the step will be +### Run the tests +```bash +docker run --rm=true --name tests -t --env-file envconfig.rc \ + -v ~/.config/openstack/clouds.yaml:/etc/openstack/clouds.yaml \ + -v ~/tests/reports:/robot-systest/reports \ + osmtests \ + -t sol003_01 ``` -Give the example + +## How to mount local tests folder for developing purposes + +The following line will mount the folder `robot-systest`, including all libraries and testuites, and will execute the testsuite `sol003_01`: + +```bash +docker run --rm=true --name tests -t --env-file envconfig.rc \ + -v ~/.config/openstack/clouds.yaml:/etc/openstack/clouds.yaml \ + -v ~/tests/robot-systest:/robot-systest \ + -v ~/tests/reports:/robot-systest/reports \ + -v ~/osm-packages:/robot-systest/osm-packages \ + opensourcemano/tests:testing-daily \ + -t sol003_01 +``` + +Relevant volumes to be mounted are: + +- [OPTIONAL]: the absolute path to reports location in the host +- [OPTIONAL]: the absolute path to the clouds.yaml file in the host +- [OPTIONAL]: the absolute path to the sdncs.yaml file in the host +- [OPTIONAL]: the kubeconfig file to be used for k8s clusters + +Other relevant options to run tests are: + +- `--env-file`: It is the environmental file where is described the OSM target and VIM +- `-o ` [OPTIONAL]: It is used to specify a particular osmclient version. Default: latest +- `-p ` [OPTIONAL]: OSM packages repository branch. Default: master +- `-t ` [OPTIONAL]: Robot tests tags. [sanity, regression, particular_test]. Default: sanity + +## How to run tests from a host + +In general, testing from docker images is the best way if you want to develop for OSM. However, sometimes it could be useful to run tests directly from the host. + +### Install dependencies + +This bash script can be used to setup your environment to execute the tests. + +```bash +sudo apt-get update +sudo apt-get install ssh ping yq git +# Python packages used for the tests +python3 -m pip install -r requirements.txt +python3 -m pip install -r requirements-dev.txt +# Download community packages +git clone https://osm.etsi.org/gitlab/vnf-onboarding/osm-packages.git ``` -And repeat +### Configure the environment + +Create a file `envconfig.rc` copying from `envconfig-local.rc` and set the required variables (in this case, the use of `export` is mandatory). ``` -until finished +export OSM_HOSTNAME= +export VIM_TARGET= +export VIM_MGMT_NET= +export OS_CLOUD= # OpenStack Cloud defined in $HOME/.config/openstack/clouds.yaml or in /etc/openstack/clouds.yaml +export K8S_CREDENTIALS= # path to the kubeconfig file of the K8s cluster to be tested +export OCI_REGISTRY_URL= # URL of the OCI registry where helm charts used in test packages are stored. +export OCI_REGISTRY_USER= # User of the OCI registry +export OCI_REGISTRY_PASSWORD= # Password of the OCI registry ``` -End with an example of getting some data out of the system or using it for a little demo +### Running the tests -## Running the tests +```bash +source envconfig.rc +mkdir reports +robot -d reports -i testsuite/ +``` -Explain how to run the automated tests for this system +## How to run tests from an environment identical to OSM CICD + +```bash +git clone https://osm.etsi.org/gerrit/osm/devops +git clone https://osm.etsi.org/gerrit/osm/IM +git clone https://osm.etsi.org/gerrit/osm/osmclient +git clone https://osm.etsi.org/gerrit/osm/tests +# run HTTP server to server artifacts +devops/tools/local-build.sh --install-qhttpd +# generate debian packages locally that will be served by the HTTP server +devops/tools/local-build.sh --module IM,osmclient,tests stage-2 +# create docker image and store it locally as opensourcemano/tests:devel +devops/tools/local-build.sh --module tests +``` +Then, run the tests: + +```bash +docker run --rm=true -t osmtests --env-file \ + -v :/reports osmtests \ + -v :/robot-systest/clouds.yaml \ + -v :/robot-systest/sdncs.yaml \ + -v :/root/.kube/config \ + -o \ + -p \ + -t ``` -Give an example + +## Test tags + +All tests in the testsuites have tags. Tags allow to run only a set of tests identified by a tag. Several tags can be specified when running robot in the following way: + +```bash +robot -i -i testsuite/ ``` -## Deployment +The following tags exist for each testsuite: + +- A tag per testsuite using its mnemonic (e.g. `basic_01`) +- Cluster tag for each of the statistically similar tests: + - `cluster_main`: `basic_01`, `basic_05`, `basic_08`, `basic_09`, `basic_15`, + `basic_16`, `basic_17`, `hackfest_basic`, `hackfest_multivdu`, + `hackfest_cloudinit`, `quotas_01` + - `cluster_ee_config`: `basic_06`, `basic_07`, `basic_11`, `basic_12`, + `basic_13`, `basic_14`, `k8s_05`, `k8s_06` + - `cluster_relations`: `basic_11`, `basic_13`, `basic_14` + - `cluster_epa`: `epa_01`, `epa_02`, `epa_03`, `epa_04`, `epa_05` + - `cluster_k8s`: `k8s_01`, `k8s_02`, `k8s_03`, `k8s_04`, `k8s_05`, `k8s06`, + `k8s_07`, `k8s_08`, `k8s_09`, `k8s_10`, `k8s_11`, `k8s_12`, `k8s_13`, `sa_08` + - `cluster_k8s_charms`: `k8s_05`, `k8s_06` + - `cluster_sa`: `sa_01`, `sa_02`, `sa_07` + - `cluster_slices`: `slice_01`, `slice_02` + - `cluster_heal`: `heal_01`, `heal_02`, `heal_03`, `heal_04` + - `cluster_osm_rest`: `sol003_01` +- daily: for all testsuites that will run in the daily job +- regression: for all testsuites that should pass in the current stable branch +- sanity: for all testsuites that should be passed by each commit in the + stage3 to be successfully verified by Jenkins, currently `k8s_04`, + `sa_02`, `hackfest_basic`, `hackfest_cloudinit` + +In addition, the tag "cleanup" exists in those tests that perform +any deletion. In that way, it can be invoked to retry the deletion if +the tests were forcefully stopped. + +- For helping in the migration tests and other scenarios in which you don't want +to destroy the deployments immediately, the following tags are used: + - `prepare`: for the tests that are used to deploy the network + services under test + - `verify`: for the tests that perform the actual testing, or changes for + additional verifications (e.g. scaling). + - `cleanup`: already described above. + + So, for instance, you could first deploy a number of network services executing + the tests with "prepare" tag, migrate to another OSM version, and then + check the behavior executing with the "verify" tag. Finally, use the "cleanup" + tag. + +## Post-processing Robot output files + +The output files of Robot include tyipically three files: + +- `report.html`: overview of the test execution results in HTML format +- `log.html`: details about the executed test cases in HTML format +- `output.xml`: all the test execution results in machine readable XML format + +More information about these files [here](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#output-file). + +It is possible to use the tool `rebot`, included as part of the Robot Framework, to post-process the output file `output.xml`. + +```bash +# To re-generate log and report from output.xml: +rebot [-d ] output.xml + +# To re-generate log and report (and optionally new output.xml) to include only certain tags: +rebot [-d ] -i -i ... -i [-o ] output.xml + +# To re-generate log and report (and optionally new output.xml) excluding certain tags: +rebot [-d ] -e -e ... -e [-o ] output.xml + +# To merge several test executions: +rebot [-d ] --merge output1.xml output2.xml ... outputN.xml +``` -Add additional notes about how to deploy this on a live system +More information about post-processing Robot output files [here](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#post-processing-outputs) ## Built With * [Python](www.python.org/) - The language used +* [Robot Framework](robotframework.org) - The testing framework ## Contributing @@ -70,13 +253,9 @@ Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on our code of conduc ## Versioning -We use [SemVer](http://semver.org/) for versioning. For the versions available, see the [tags on this repository](https://osm.etsi.org/gitweb/?p=osm/TEMPLATE.git;a=tags). +We use [SemVer](http://semver.org/) for versioning. For the versions available, see the [tags on this repository](https://osm.etsi.org/gitweb/?p=osm/tests.git;a=tags). ## License -This project is licensed under the Apache2 License - see the [LICENSE.md](LICENSE) file for details - -## Acknowledgments - -* **Billie Thompson** - *Initial work* - [PurpleBooth](https://github.com/PurpleBooth) +This project is licensed under the Apache2 License - see the [LICENSE](LICENSE) file for details