From fd4ec1d908ba105ef26408d470a26318708b8b69 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerardo=20Garc=C3=ADa?= Date: Tue, 9 Jun 2020 15:45:56 +0200 Subject: [PATCH 1/2] Updated OSM client documentation in 03 and 10; minor fixes in title in 03 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Gerardo García --- 03-installing-osm.md | 24 +- 10-osm-client-commands-reference.md | 350 ++++++++++++++++++++++++++-- 2 files changed, 341 insertions(+), 33 deletions(-) diff --git a/03-installing-osm.md b/03-installing-osm.md index 7ab1542..8759c99 100644 --- a/03-installing-osm.md +++ b/03-installing-osm.md @@ -347,7 +347,7 @@ docker logs $(docker ps -aqf "name=osm_lcm" -n 1) # shows the logs of the last The **OSM Client** is a client library and a command-line tool (based on Python) to operate OSM, which accesses OSM's Northbound Interface (NBI) and lets you manage descriptors, VIMs, Network Services, Slices, etc. along with their whole lifecycle. In other words, the OSM Client is a sort of Swiss knife that provides a convenient access to all the functionality that OSM's NBI offers. -Although the OSM Client is always available in the host machine after installation, it is sometimes convenient installing an OSM Client in another location, different from the OSM host, so that the access to the OSM services does not require OS-level/SSH credentials. Thus, in those cases where you have an OSM already installed in a remote server and you can still operate it from your local computer using the OSM Client. +Although the OSM Client is always available in the host machine after installation, it is sometimes convenient installing an OSM Client in another location, different from the OSM host, so that the access to the OSM services does not require OS-level/SSH credentials. Thus, in those cases where you have an OSM already installed in a remote server, you can still operate it from your local computer using the OSM Client. In order to install the OSM Client in your local Linux machine, you should follow this procedure: @@ -371,7 +371,7 @@ Since we are installing the OSM Client in a host different from OSM's at a minim export OSM_HOSTNAME="10.80.80.5" ``` -For additional remote access options, see `osm --help` for more info. +For additional options, see `osm --help` for more info, and check our OSM client reference guide [here](10-osm-client-commands-reference.md) ## How to upgrade OSM @@ -578,7 +578,7 @@ Or to use an old version of MON: ### How to upgrade OSM when using K8s as container framework -#### Upgrading the complete OSM platform +#### Upgrading the complete OSM platform in K8s Upgrading to the latest minor version is as simple as running again the installer: @@ -592,7 +592,7 @@ You will be asked if you want to proceed with the installation and configuration That's all. You will have the newest OSM version installed. -#### Upgrading the OSM platform from docker images (advanced users) +#### Upgrading the OSM platform from docker images in K8s (advanced users) **Upgrading to the latest daily docker image might lead to potential issues.** Moreover, every new re-deployment of the stack will involve a download of a new docker daily image if it exists. Unless you are really sure about what you are doing, please use this procedure with caution. @@ -640,13 +640,13 @@ sudo apt-get install python3-osmclient dpkg -l |grep python3-osmclient #to check the installed version ``` -#### Upgrading only a specific component (advanced users) +#### Upgrading only a specific component in K8s (advanced users) **Upgrading a specific OSM component without upgrading the others accordingly may lead to potential inconsistencies.** Unless you are really sure about what you are doing, please use this procedure with caution. One of the commonest reasons for this type of upgrade is using your own cloned repo of a module for development purposes. -##### Upgrading RO +##### Upgrading RO in K8s This involves upgrading (`ro` and `ro-db`): @@ -665,7 +665,7 @@ kubectl -n osm scale deployment ro --replicas=1 # kubectl -n osm apply -f /etc/osm/docker/osm_pods/ro.yaml ``` -##### Upgrading LCM +##### Upgrading LCM in K8s ```bash git clone https://osm.etsi.org/gerrit/osm/LCM @@ -681,7 +681,7 @@ kubectl -n osm scale deployment lcm --replicas=1 # kubectl -n osm apply -f /etc/osm/docker/osm_pods/lcm.yaml ``` -##### Upgrading MON +##### Upgrading MON in K8s ```bash git clone https://osm.etsi.org/gerrit/osm/MON @@ -697,7 +697,7 @@ kubectl -n osm scale deployment mon --replicas=1 # kubectl -n osm apply -f /etc/osm/docker/osm_pods/mon.yaml ``` -##### Upgrading POL +##### Upgrading POL in K8s ```bash git clone https://osm.etsi.org/gerrit/osm/POL @@ -713,7 +713,7 @@ kubectl -n osm scale deployment pol --replicas=1 # kubectl -n osm apply -f /etc/osm/docker/osm_pods/pol.yaml ``` -##### Upgrading NBI +##### Upgrading NBI in K8s ```bash git clone https://osm.etsi.org/gerrit/osm/NBI @@ -729,7 +729,7 @@ kubectl -n osm scale deployment nbi --replicas=1 # kubectl -n osm apply -f /etc/osm/docker/osm_pods/nbi.yaml ``` -##### Upgrading Light UI +##### Upgrading Light UI in K8s ```bash git clone https://osm.etsi.org/gerrit/osm/LW-UI @@ -745,7 +745,7 @@ kubectl -n osm scale deployment light-ui --replicas=1 # kubectl -n osm apply -f /etc/osm/docker/osm_pods/light-ui.yaml ``` -#### Upgrading only a specific component from source code (advanced users) +#### Upgrading only a specific component from source code in K8s (advanced users) **FIXME:** The procedure described here needs revision, as it may work differently in the latest OSM release. diff --git a/10-osm-client-commands-reference.md b/10-osm-client-commands-reference.md index bb872e3..95d2073 100644 --- a/10-osm-client-commands-reference.md +++ b/10-osm-client-commands-reference.md @@ -1,19 +1,26 @@ # ANNEX 2: Reference of OSM Client commands and library +The **OSM Client** is a client library and a command-line tool (based on Python) to operate OSM, which accesses OSM's Northbound Interface (NBI) and lets you manage descriptors, VIMs, Network Services, Slices, etc. along with their whole lifecycle. In other words, the OSM Client is a sort of Swiss knife that provides a convenient access to all the functionality that OSM's NBI offers. + **Usage:** ```bash -osm [OPTIONS] COMMAND [ARGS]... +osm [GLOBAL_OPTIONS] COMMAND [COMMAND_OPTIONS] [ARGS]... ``` **Options:** ```bash - --hostname TEXT hostname of server. Also can set OSM_HOSTNAME in environment - --user TEXT user (defaults to admin). Also can set OSM_USER in environment - --password TEXT password (defaults to admin). Also can set OSM_PASSWORD in environment - --project TEXT project (defaults to admin). Also can set OSM_PROJECT in environment - -h, --help Show this message and exit. + --hostname TEXT hostname of server. Also can set OSM_HOSTNAME in environment + --user TEXT user (defaults to admin). Also can set OSM_USER in environment + --password TEXT password (defaults to admin). Also can set OSM_PASSWORD in environment + --project TEXT project (defaults to admin). Also can set OSM_PROJECT in environment + -v, --verbose increase verbosity (-v INFO, -vv VERBOSE, -vvv DEBUG) + --all-projects include all projects + --public / --no-public flag for public items (packages, instances, VIM accounts, etc.) + --project-domain-name TEXT project domain name for keystone authentication (default to None). Also can set OSM_PROJECT_DOMAIN_NAME in environment + --user-domain-name TEXT user domain name for keystone authentication (default to None). Also can set OSM_USER_DOMAIN_NAME in environment + -h, --help Show this message and exit. Commands: k8scluster-add adds a K8s cluster to OSM NAME: name of the K8s cluster @@ -36,7 +43,9 @@ Commands: nfpkg-create creates a new NFpkg nfpkg-delete deletes a NFpkg nfpkg-list list all xNF packages (VNF, HNF, PNF) - nfpkg-show shows the content of a NF Descriptor + nfpkg-repo-list list all xNF from OSM repositories + nfpkg-repo-show shows the details of a NF package in an OSM repository + nfpkg-show shows the details of a NF package nfpkg-update updates a NFpkg ns-action executes an action/primitive over a NS instance ns-alarm-create creates a new alarm for a NS instance @@ -50,7 +59,9 @@ Commands: nsd-create creates a new NSD/NSpkg nsd-delete deletes a NSD/NSpkg nsd-list list all NS packages - nsd-show shows the content of a NSD + nsd-repo-list list all NS from OSM repositories + nsd-repo-show shows the details of a NS package in an OSM repository + nsd-show shows the details of a NS package nsd-update updates a NSD/NSpkg nsi-create creates a new Network Slice Instance nsi-delete deletes a Network Slice Instance (NSI) @@ -61,7 +72,9 @@ Commands: nspkg-create creates a new NSD/NSpkg nspkg-delete deletes a NSD/NSpkg nspkg-list list all NS packages - nspkg-show shows the content of a NSD + nspkg-repo-list list all NS from OSM repositories + nspkg-repo-show shows the details of a NS package in an OSM repository + nspkg-show shows the details of a NS package nspkg-update updates a NSD/NSpkg nst-create creates a new Network Slice Template (NST) nst-delete deletes a Network Slice Template (NST) @@ -113,12 +126,14 @@ Commands: vnfd-create creates a new VNFD/VNFpkg vnfd-delete deletes a VNFD/VNFpkg vnfd-list list all xNF packages (VNF, HNF, PNF) - vnfd-show shows the content of a VNFD + vnfd-show shows the details of a NF package vnfd-update updates a new VNFD/VNFpkg vnfpkg-create creates a new VNFD/VNFpkg vnfpkg-delete deletes a VNFD/VNFpkg vnfpkg-list list all xNF packages (VNF, HNF, PNF) - vnfpkg-show shows the content of a VNFD + vnfpkg-repo-list list all xNF from OSM repositories + vnfpkg-repo-show shows the details of a NF package in an OSM repository + vnfpkg-show shows the details of a NF package vnfpkg-update updates a VNFD/VNFpkg wim-create creates a new WIM account wim-delete deletes a WIM account @@ -127,23 +142,26 @@ Commands: wim-update updates a WIM account ``` -## Enable autocompletion +## Installing standalone OSM Client -You can enable autocompletion in OSM client by creating a file osm-complete.sh in the following way: +OSM client is installed by default in the host where OSM is installed, but it can be also installed as a standalone client. -```bash -mkdir -p $HOME/.bash_completion.d -_OSM_COMPLETE=source osm > $HOME/.bash_completion.d/osm-complete.sh -``` +### Installing with debian package -Then you can add the following to your $HOME/.bashrc file: +In order to install the OSM Client in your local Linux machine, you should follow this procedure: ```bash -. .bash_completion.d/osm-complete.sh +# Clean the previous repos that might exist +sudo sed -i "/osm-download.etsi.org/d" /etc/apt/sources.list +wget -qO - https://osm-download.etsi.org/repository/osm/debian/ReleaseEIGHT/OSM%20ETSI%20Release%20Key.gpg | sudo apt-key add - +sudo add-apt-repository -y "deb [arch=amd64] https://osm-download.etsi.org/repository/osm/debian/ReleaseEIGHT stable devops IM osmclient" +sudo apt-get update +sudo apt-get install -y python3-pip +sudo -H python3 -m pip install -U pip +sudo -H python3 -m pip install python-magic pyangbind verboselogs +sudo apt-get install python3-osmclient ``` -## Other installation methods - ### Installing from git repo ```bash @@ -184,6 +202,296 @@ To uninstall, just: python3 -m pip uninstall osmclient ``` +## Getting help + +Options `-h` or `--help`can be used globally to know the global options and commands, or after a command to know the specific options and args of a command + +```bash +$ osm -h +Usage: osm [OPTIONS] COMMAND [ARGS]... + +Options: + --hostname TEXT hostname of server. Also can set OSM_HOSTNAME in environment + --user TEXT user (defaults to admin). Also can set OSM_USER in environment + --password TEXT password (defaults to admin). Also can set OSM_PASSWORD in environment + --project TEXT project (defaults to admin). Also can set OSM_PROJECT in environment + -v, --verbose increase verbosity (-v INFO, -vv VERBOSE, -vvv DEBUG) + --all-projects include all projects + --public / --no-public flag for public items (packages, instances, VIM accounts, etc.) + --project-domain-name TEXT project domain name for keystone authentication (default to None). Also can set OSM_PROJECT_DOMAIN_NAME in environment + --user-domain-name TEXT user domain name for keystone authentication (default to None). Also can set OSM_USER_DOMAIN_NAME in environment + -h, --help Show this message and exit. + +Commands: + k8scluster-add adds a K8s cluster to OSM + ... + +$osm vim-create -h +Usage: osm vim-create [OPTIONS] + + creates a new VIM account + +Options: + --name TEXT Name to create datacenter + --user TEXT VIM username + --password TEXT VIM password + --auth_url TEXT VIM url + --tenant TEXT VIM tenant name + --config TEXT VIM specific config parameters + --account_type TEXT VIM type + --description TEXT human readable description + --sdn_controller TEXT Name or id of the SDN controller associated to this VIM account + --sdn_port_mapping TEXT File describing the port mapping between compute nodes' ports and switch ports + --wait do not return the control immediately, but keep it until the operation is completed, or timeout + -h, --help Show this message and exit. + +``` + +The following video illustrates how it works. + + + +## Interaction from the OSM client to an OSM in a different location + +When using the OSM Client from a host different from OSM's, at a minimum you will need to specify the OSM host, either via an environment variable or via the option `--hostname`. For instance, you can set your client to access an OSM host running at `10.80.80.5` by using: + +```bash +export OSM_HOSTNAME="10.80.80.5" +``` + +The following video shows how it works. + + + +## Getting OSM version + +You can obtain server and client versions, by using the command `osm version`. Server version corresponds to OSM NBI module. + +```bash +$ osm version +Server version: 7.0.1.post23 2020-04-17 +Client version: 7.0.1rc1.post52+gbcb7833 +``` + +The following video shows how it works. + + + +## RBAC + +### Configuring RBAC + +OSM supports Role-Based Access Control, so that it's possible to use users and projects, and each user will have a role in a project. OSM provides CRUD operations (`*-create`, `*-list`, `*-show`, `*-update`, `*-delete`) over users, roles and projects + +The following video demonstrates how it works. + + + +### Using RBAC + +When using users and projects, the following environment variables can be used to authenticate against the system: + +- `OSM_USER` +- `OSM_PASSWORD` +- `OSM_PROJECT` +- `OSM_PROJECT_DOMAIN_NAME` +- `OSM_USER_DOMAIN_NAME` + +Or instead the command line options `--user`, `--password`, `--project`, `--project-domain-name` and `--user-domain-name` can be used: + +```bash + --user TEXT user (defaults to admin). Also can set OSM_USER in environment + --password TEXT password (defaults to admin). Also can set OSM_PASSWORD in environment + --project TEXT project (defaults to admin). Also can set OSM_PROJECT in environment + --project-domain-name TEXT project domain name for keystone authentication (default to None). Also can set OSM_PROJECT_DOMAIN_NAME in environment + --user-domain-name TEXT user domain name for keystone authentication (default to None). Also can set OSM_USER_DOMAIN_NAME in environment +``` + +In addition, the option `--all-projects` allow users to perform operations in all the projects they belong to (e.g. `osm --all-projects nfpkg-list` will show all the NF packages in all projects the user belongs to). Finally, the option `--public` allows to create public items, making them available in all projects; visibility from the client of those public items in `*-list` and `*-show` operations requires the use of the option `--public`. + +```bash + --all-projects include all projects + --public / --no-public flag for public items (packages, instances, VIM accounts, etc.) +``` + +The following video shows a complete demo using different users and projects, and some of these options, illustrating the different project visibility. + + + +## OSM repositories + +OSM support repositories where NF and NS packages can be retrieved. The following commands can be used to add a repo, list the packages in a repo, show their details or onboard them in OSM: + +- `osm repo-add NAME URI`: adds a VNF/NS catalog repo +- `osm nfpkg-repo-list`and `osm nspkg-repo-list`: lists NF/NS packages from OSM repositories +- `osm nfpkg-repo-show|nspkg-repo-show --repo `: shows a specific NF/NS package in a repo +- `osm nfpkg-create|nspkg-create --repo `: onboards a package from a repo into OSM + +```bash +osm repo-add --description "OSM VNF and NS repository" vnfrepo https://osm-download.etsi.org/ftp/vnf-catalog/ +osm nfpkg-repo-list +osm nfpkg-repo-show --repo vnfrepo fb_magma_knf +osm nfpkg-create --repo vnfrepo fb_magma_knf +osm nspkg-repo-list +osm nspkg-repo-show --repo vnfrepo fb_magma_ns +osm nspkg-create --repo vnfrepo fb_magma_ns +``` + +The following video demonstrates how these commands work. + + + +## Package creation tools + +There are several utilities in the OSM client to simplify VNF and NS package creation: + +- `osm package-create PACKAGE_TYPE PACKAGE_NAME`: creates an OSM VNF, NS or NST package +- `osm package-validate PKG_FOLDER`: validates descriptors given a base directory +- `osm package-build PKG_FOLDER`: builds the package NS, VNF given the package_folder (descriptor validation, charm build, checksum generation, tar.gz creation) +- `osm nfpkg-create|nspkg-create PKG_FOLDER`: validates the package, builds it and onboards it into OSM + +The following video illustrates the previous commands. + + + +## Advanced onboarding + +Commands for onboarding NF, NS and NST (`nfpkg-create`, `nspkg-create` and `nst-create`) have the option `--override` to override the fields in the descriptor whem uploading the package to OSM. The format of the option is the following: + +```bash + --override TEXT overrides fields in descriptor, format: "key1.key2...=value[;key3...=value;...]" +``` + +In addition, VNF onboarding allows these additional options: + +- `--override-epa`: adds guest-epa parameters to all VDU. Useful to onboard packages to be deployed in an EPA-only VIM +- `--override-nonepa`: removes all guest-epa parameters from all VDU. Useful to onboard EPA-based VNF packages to be deployed in a non-EPA VIM +- `--override-paravirt`: overrides all VDU interfaces to PARAVIRT. Useful if packages are going to be deployed in a VIM that does not support SR-IOV or PCI-PASSTHROUGH interfaces. + +The following video demonstrates how these options work. + + + +## Filtering requests + +Some commands have the option `--filter` that allows to filter Read operations (`*-list`, `*-show`) to restric the output to the items matching de filter. Below an example for `osm ns-list`: + +```bash +$ osm ns-list -h +Usage: osm ns-list [OPTIONS] + + list all NS instances + + Options: + --filter filterExpr Restricts the list to the NS instances matching the filter + + filterExpr consists of one or more strings formatted according to "simpleFilterExpr", + concatenated using the "&" character: + + filterExpr := ["&"]* + simpleFilterExpr := ["."]*["."]"="[","]* + op := "eq" | "neq" | "gt" | "lt" | "gte" | "lte" | "cont" | "ncont" + attrName := string + value := scalar value + + where: + * zero or more occurrences + ? zero or one occurrence + [] grouping of expressions to be used with ? and * + "" quotation marks for marking string constants + <> name separator + + "AttrName" is the name of one attribute in the data type that defines the representation + of the resource. The dot (".") character in "simpleFilterExpr" allows concatenation of + entries to filter by attributes deeper in the hierarchy of a structured document. + "Op" stands for the comparison operator. If the expression has concatenated + entries, it means that the operator "op" is applied to the attribute addressed by the last + entry included in the concatenation. All simple filter expressions are combined + by the "AND" logical operator. In a concatenation of entries in a , + the rightmost "attrName" entry in a "simpleFilterExpr" is called "leaf attribute". The + concatenation of all "attrName" entries except the leaf attribute is called the "attribute + prefix". If an attribute referenced in an expression is an array, an object that contains a + corresponding array shall be considered to match the expression if any of the elements in the + array matches all expressions that have the same attribute prefix. + + Filter examples: + --filter admin-status=ENABLED + --filter nsd-ref= + --filter nsd.vendor= + --filter nsd.vendor=&nsd-ref= + --filter nsd.constituent-vnfd.vnfd-id-ref= +``` + +## Debugging + +Option `-v` can be used to debug OSM client. There are different verbosity levels: + +- `-v`: log level INFO, shows API calls +- `-vv`: log level VERBOSE, shows API calls and content response +- `-vvv`: log level DEBUG, shows the different methods the client is going through + +The following video illustrates the log detail for each verbosity level. + + + +## Other useful options + +Some commands present useful options: + +- `--literal`: shows output in YAML format. It might be deprecated in the future in favour of a new option `-o OUTPUT_FORMAT`. +- `--wait`: makes the command synchronous +- `--long`: shows more info + +## Aliases + +The following are command aliases with identical behaviour: + +- For CRUD operations over NF packages, there are different command aliases. The preferred commands is `nfpkg-*`. The command `vnfd-*`might be deprecated in the future. + - `nfpkg-create|vnfpkg-create|vnfd-create`: creates a new NFpkg + - `nfpkg-delete|vnfpkg-delete|vnfd-delete`: deletes a NFpkg + - `nfpkg-list|vnfpkg-list|vnfd-list`: list all xNF packages (VNF, HNF, PNF) + - `nfpkg-show|vnfpkg-show|vnfd-show`: shows the details of a NF package + - `nfpkg-update|vnfpkg-update|vnfd-update`: updates a NFpkg +- For CRUD operations over NS packages, there are different command aliases. The preferred commands is `nspkg-*`. The command `nsd-*`might be deprecated in the future. + - `nsd-create|nspkg-create`: creates a new NSD/NSpkg + - `nsd-delete|nspkg-delete`: deletes a NSD/NSpkg + - `nsd-list|nspkg-list`: list all NS packages + - `nsd-show|nspkg-show`: shows the details of a NS package + - `nsd-update|nspkg-update`: updates a NSD/NSpkg +- For CRUD operations over Network Slice Templates, the following command aliases exist: + - `netslice-template-create|nst-create`: creates a new Network Slice Template (NST) + - `netslice-template-delete|nst-delete`: deletes a Network Slice Template (NST) + - `netslice-template-list|nst-list`: list all Network Slice Templates (NST) + - `netslice-template-show|nst-show`: shows the content of a Network Slice Template (NST) + - `netslice-template-update|nst-update`: updates a Network Slice Template (NST) +- For CRUD operations over Network Slice Instances, the following command aliases exist: + - `netslice-instance-create|nsi-create`: creates a new Network Slice Instance + - `netslice-instance-delete|nsi-delete`: deletes a Network Slice Instance (NSI) + - `netslice-instance-list|nsi-list`: list all Network Slice Instances (NSI) + - `netslice-instance-op-list|nsi-op-list`: shows the history of operations over a Network Slice Instance (NSI) + - `netslice-instance-op-show|nsi-op-show`: shows the info of an operation over a Network Slice Instance(NSI) + - `netslice-instance-show|nsi-show`: shows the content of a Network Slice Instance (NSI) +- For Read operations over OSM repositories, the following command aliases exist: + - `nfpkg-repo-list|vnfpkg-repo-list`: list all xNF from OSM repositories + - `nfpkg-repo-show|vnfpkg-repo-show`: shows the details of a NF package in an OSM repository + - `nsd-repo-list|nspkg-list`: list all NS from OSM repositories + - `nsd-repo-show|nspkg-show`: shows the details of a NS package in an OSM repository + +## Enable autocompletion + +You can enable autocompletion in OSM client by creating a file osm-complete.sh in the following way: + +```bash +mkdir -p $HOME/.bash_completion.d +_OSM_COMPLETE=source osm > $HOME/.bash_completion.d/osm-complete.sh +``` + +Then you can add the following to your $HOME/.bashrc file: + +```bash +. .bash_completion.d/osm-complete.sh +``` + ## Using osmclient as a library to interact with OSM Assuming that you have installed python-osmclient package, it's pretty simple to write some Python code to interact with OSM. -- GitLab From b6d87b0cd42e5c768aeacb0cf620d0366f870341 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerardo=20Garc=C3=ADa?= Date: Fri, 24 Jul 2020 17:30:46 +0200 Subject: [PATCH 2/2] Updated troubleshooting guide: improved structure, added instructions for getting logs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Gerardo García --- 09-troubleshooting.md | 358 +++++++++++++++++++++++------------------- 1 file changed, 194 insertions(+), 164 deletions(-) diff --git a/09-troubleshooting.md b/09-troubleshooting.md index d6ef191..8269dd5 100644 --- a/09-troubleshooting.md +++ b/09-troubleshooting.md @@ -27,7 +27,9 @@ dpkg -l python3-osmclient ii python3-osmclient 8.0.0-1 all ``` -## Recommended installation to facilitate troubleshooting +## Troubleshooting installation + +### Recommended installation to facilitate troubleshooting It is highly recommended saving a log of your installation: @@ -35,106 +37,45 @@ It is highly recommended saving a log of your installation: ./install_osm.sh 2>&1 | tee osm_install_log.txt ``` -## Common issues and troubleshooting - -### Add User in Group - -Add the non-root user used for installation in *sudo , lxd, docker* groups - -This will skip below error :- - -_Finished installation of juju_ Password: **sg: failed to crypt password with previous salt: Invalid argument** ERROR No controllers registered. - -### Docker - -#### Were all docker images successfully built? - -Although controlled by the installer, you can check that the following images exist: - -```bash -$ docker image ls - -REPOSITORY TAG IMAGE ID CREATED SIZE -osm/light-ui latest 1988aa262a97 18 hours ago 710MB -osm/lcm latest c9ad59bf96aa 46 hours ago 667MB -osm/ro latest 812c987fcb16 46 hours ago 791MB -osm/nbi latest 584b4e0084a7 46 hours ago 497MB -osm/pm latest 1ad1e4099f52 46 hours ago 462MB -osm/mon latest b17efa3412e3 46 hours ago 725MB -wurstmeister/kafka latest 7cfc4e57966c 10 days ago 293MB -mysql 5 0d16d0a97dd1 2 weeks ago 372MB -mongo latest 14c497d5c758 3 weeks ago 366MB -wurstmeister/zookeeper latest 351aa00d2fe9 18 months ago 478MB -``` +### Recommended checks after installation -#### Are all processes/services running? +#### Checking whether all processes/services are running in docker swarm ```bash -$ docker stack ps osm |grep -i running +docker stack ps osm |grep -i running ``` -10 docker containers should be running. - -All the 10 services should have at least 1 replica: 1/1 +All the services should have at least 1 replica: 1/1 ```bash $ docker service ls ID NAME MODE REPLICAS IMAGE PORTS -yuyiqh8ty8pv osm_kafka replicated 1/1 wurstmeister/kafka:latest *:9092->9092/tcp -y585906h5vy5 osm_lcm replicated 1/1 osm/lcm:latest -pcdi5vb86nt9 osm_light-ui replicated 1/1 osm/light-ui:latest *:80->80/tcp -i56jhl5k6re4 osm_mon replicated 1/1 osm/mon:latest *:8662->8662/tcp -p5wyjtne93hp osm_mongo replicated 1/1 mongo:latest -iz5uncfdzu23 osm_nbi replicated 1/1 osm/nbi:latest *:9999->9999/tcp -4ttw2v4z2g57 osm_pm replicated 1/1 osm/pm:latest -xbg6bclp2anw osm_ro replicated 1/1 osm/ro:latest *:9090->9090/tcp -sf7rayfolncu osm_ro-db replicated 1/1 mysql:5 -5bl73dhj1xl0 osm_zookeeper replicated 1/1 wurstmeister/zookeeper:latest +paxqvnwwubcf osm_grafana replicated 1/1 grafana/grafana:latest *:3000->3000/tcp +xkn3jr7ipibf osm_kafka replicated 1/1 wurstmeister/kafka:latest *:30002->9092/tcp +px2xfetg68z1 osm_keystone replicated 1/1 opensourcemano/keystone:8 *:5000->5000/tcp +62yljr0s97vv osm_lcm replicated 1/1 opensourcemano/lcm:8 +lwtfoh29sb95 osm_light-ui replicated 1/1 opensourcemano/light-ui:8 *:80->80/tcp +xjl2vx9t6ogz osm_mon replicated 1/1 opensourcemano/mon:8 *:8662->8662/tcp +t6r9wjjxqy1v osm_mongo replicated 1/1 mongo:latest +rmuhwvl5gkgo osm_mysql replicated 1/1 mysql:5 +vjyee8af3a8r osm_nbi replicated 1/1 opensourcemano/nbi:8 *:9999->9999/tcp +ihdjxn68aa4p osm_pol replicated 1/1 opensourcemano/pol:8 +tnk91kubxfvk osm_prometheus replicated 1/1 prom/prometheus:latest *:9091->9090/tcp +4e5c49m9x0by osm_prometheus-cadvisor replicated 1/1 google/cadvisor:latest *:8080->8080/tcp +m1cxap6wkxmf osm_ro replicated 1/1 opensourcemano/ro:8 *:9090->9090/tcp +97r6t2zrs4ho osm_zookeeper replicated 1/1 wurstmeister/zookeeper:latest ``` -#### Docker image failed to build - -##### Err:1 `http://archive.ubuntu.com/ubuntu xenial InRelease` - -In some cases, DNS resolution works on the host but fails when building the Docker container. This is caused when Docker doesn't automatically determine the DNS server to use. - -Check if the following works: +#### Checking whether all processes/services are running in K8s ```bash -docker run busybox nslookup archive.ubuntu.com +kubectl -n osm get all ``` -If it does not work, you have to configure Docker to use the available DNS. - -```bash -# Get the IP address you’re using for DNS: -nmcli dev show | grep 'IP4.DNS' -# Create a new file, /etc/docker/daemon.json, that contains the following (but replace the DNS IP address with the output from the previous step: -{ - "dns": ["192.168.24.10"] -} -# Restart docker -sudo service docker restart -# Re-run -docker run busybox nslookup archive.ubuntu.com -# Now you should be able to re-run the installer and move past the DNS issue. -``` - -##### TypeError: `unsupported operand type(s) for -=: 'Retry' and 'int'` - -In some cases, a MTU mismatch between the host and docker interfaces will cause this error while running pip. You can check this by running `ifconfig` and comparing the MTU of your host interface and the `docker_gwbridge` interface. - -```bash -# Create a new file, /etc/docker/daemon.json, that contains the following (but replace the MTU value with that of your host interface from the previous step: -{ - "mtu": 1458 -} -# Restart docker -sudo service docker restart -``` +### Issues on standard installation -#### Problem deploying stack osm +#### Docker Swarm ##### `network netosm could not be found` @@ -159,9 +100,9 @@ It usually happens when a `docker system prune` is done with the stack stopped. ${OSM_NETWORK_NAME}" ``` -### Juju +#### Juju -#### Bootstrap hangs +##### Juju bootstrap hangs If the Juju bootstrap takes a long time, stuck at this status... @@ -193,7 +134,7 @@ Next, tail the output of cloud-init to see where the bootstrap is stuck. lxc exec juju-0383f2-0 -- tail -f /var/log/cloud-init-output.log ``` -#### Is Juju running? +##### Is Juju running? If running, you should see something like this: @@ -204,7 +145,7 @@ Model Controller Cloud/Region Version SLA default osm localhost/localhost 2.3.7 unsupported ``` -#### ERROR controller osm already exists +##### ERROR controller osm already exists Did OSM installation fail during juju installation with an error like "ERROR controller osm already exists" ? @@ -243,9 +184,17 @@ juju list-controllers ./install_osm.sh ``` -### LXD +##### No controllers registered -#### ERROR profile default: `/etc/default/lxd-bridge` has IPv6 enabled +The following error appears when the user used for installation does not belong to some groups: + +_Finished installation of juju_ Password: **sg: failed to crypt password with previous salt: Invalid argument** ERROR No controllers registered. + +To fix it, just add the non-root user used for installation in *sudo , lxd, docker* groups + +#### LXD + +##### ERROR profile default: `/etc/default/lxd-bridge` has IPv6 enabled Make sure that you follow the instructions in the [Quickstart](01-quickstart.md). @@ -258,50 +207,72 @@ When dialog messages related to LXD configuration are shown, please answer in th - << Default values apply for next questions >> - **Do you want to setup an IPv6 subnet? No** -### Configuration +### Issues on advanced installation (manual build of docker images) -### NBI +#### Manual build of images. Were all docker images successfully built? -#### SSL certificate problem +Although controlled by the installer, you can check that the following images exist: -By default, OSM installer uses a self-signed certificate for HTTPS. That might lead to the error '_SSL certificate problem: self signed certificate_' on the client side. For testing environments, you might want to ignore this error just by using the appropriate options to skip certificate validation (e.g. `--insecure` for curl, `--no-check-certificate` for wget, etc.). However, for more stable setups you might prefer to address this issue by installing the appropriate certificate in your client system. +```bash +$ docker image ls -These are the steps to install NBI certificate on the client side (tested for Ubuntu): +REPOSITORY TAG IMAGE ID CREATED SIZE +osm/light-ui latest 1988aa262a97 18 hours ago 710MB +osm/lcm latest c9ad59bf96aa 46 hours ago 667MB +osm/ro latest 812c987fcb16 46 hours ago 791MB +osm/nbi latest 584b4e0084a7 46 hours ago 497MB +osm/pm latest 1ad1e4099f52 46 hours ago 462MB +osm/mon latest b17efa3412e3 46 hours ago 725MB +wurstmeister/kafka latest 7cfc4e57966c 10 days ago 293MB +mysql 5 0d16d0a97dd1 2 weeks ago 372MB +mongo latest 14c497d5c758 3 weeks ago 366MB +wurstmeister/zookeeper latest 351aa00d2fe9 18 months ago 478MB +``` -1. Get the certificate file `cert.pem` by any of these means: - - From running docker container: - ```bash - docker ps | grep nbi - docker cp :/app/NBI/osm_nbi/http/cert.pem . - ``` - - From source code: NBI-folder/osm_nbi/http/cert.pem - - From ETSI's git: - ```bash - wget -O cert.pem "https://osm.etsi.org/gitweb/?p=osm/NBI.git;a=blob_plain;f=osm_nbi/http/cert.pem;hb=refs/heads/v8.0" - ``` -2. Then, you should install this certificate: - ```bash - sudo cp cert.pem /usr/local/share/ca-certificates/osm_nbi_cert.pem.crt - sudo update-ca-certificates - # 1 added, 0 removed; done - ``` -3. Add to the list of `/etc/hosts` a host called "nbi" with the IP address where OSM is running. - - It can be `localhost` if client and server are the same machine. - - For localhost, you would need to add (or edit) these lines: - ```text - 127.0.0.1 localhost nbi - OSM-ip nbi - ``` -4. Finally, for the URL, use the `nbi` as host name (i.e. ). - - Do not use neither `localhost` nor 127.0.0.1. - - You can run a quick test with `curl` by: - ```bash - curl https://nbi:9999/osm/version - ``` +#### Docker image failed to build -### VIMs +##### Err:1 `http://archive.ubuntu.com/ubuntu xenial InRelease` -#### Is the VIM URL reachable and operational? +In some cases, DNS resolution works on the host but fails when building the Docker container. This is caused when Docker doesn't automatically determine the DNS server to use. + +Check if the following works: + +```bash +docker run busybox nslookup archive.ubuntu.com +``` + +If it does not work, you have to configure Docker to use the available DNS. + +```bash +# Get the IP address you’re using for DNS: +nmcli dev show | grep 'IP4.DNS' +# Create a new file, /etc/docker/daemon.json, that contains the following (but replace the DNS IP address with the output from the previous step: +{ + "dns": ["192.168.24.10"] +} +# Restart docker +sudo service docker restart +# Re-run +docker run busybox nslookup archive.ubuntu.com +# Now you should be able to re-run the installer and move past the DNS issue. +``` + +##### TypeError: `unsupported operand type(s) for -=: 'Retry' and 'int'` + +In some cases, a MTU mismatch between the host and docker interfaces will cause this error while running pip. You can check this by running `ifconfig` and comparing the MTU of your host interface and the `docker_gwbridge` interface. + +```bash +# Create a new file, /etc/docker/daemon.json, that contains the following (but replace the MTU value with that of your host interface from the previous step: +{ + "mtu": 1458 +} +# Restart docker +sudo service docker restart +``` + +## Common issues with VIMs + +### Is the VIM URL reachable and operational? When there are problems to access the VIM URL, an error message similar to the following is shown after attempts to instantiate network services: @@ -354,7 +325,7 @@ docker stack deploy -c /etc/osm/docker/docker-compose.yaml osm This is persistent after reboots and restarts of the osm docker stack. -#### Authentication +### VIM authentication **What should I check if the VIM authentication is failing?** @@ -384,6 +355,8 @@ For casual testing, when adding the VIM account to OSM, you can use `'insecure: $ osm vim-create VIM-NAME ... --config '{insecure: True}' ``` +### Issues when trying to access VM from OSM + **Is the VIM management network reachable from OSM (e.g. via ssh, port 22)?** The simplest check would consist on deploying a VM attached to the management network and trying to access it via e.g. ssh from the OSM host. @@ -399,36 +372,9 @@ If this does not work, typically it is due to one of these issues: - Security group policy in your VIM is blocking your traffic (contact your admin to fix it) - IP address space in the management network is not routable from outside (or in the reverse direction, for the ACKs). -### Operational issues +## Common issues with VCA/Juju -### Running out of disk space - -If you are upgrading frequently your OSM installation, you might face that your disk is running out of space. The reason is that the previous dockers and docker images might be consuming some disk space. Running the following two commands should be enough to clear your docker setup: - -```bash -docker system prune -docker image prune -``` - -If you are still experiencing issues with disk space, logs in one of the dockers could be the cause of your issue. Check the containers that are consuming more space (typically kafka-exporter) - -```bash -du -sk /var/lib/docker/containers/* |sort -n -docker ps |grep -``` - -Then, remove the stack and redeploy it again after doing a prune: - -```bash -docker stack rm osm_metrics -docker system prune -docker image prune -docker stack deploy -c /etc/osm/docker/osm_metrics/docker-compose.yml osm_metrics -``` - -### VCA (juju) - -#### Status is not coherent with running NS +### Status is not coherent with running NS In extraordinary situations, the output of `juju status` could show pending units that should have been removed when deleting a NS. In those situations, you can clean up VCA by following the procedure below: @@ -440,7 +386,7 @@ juju resolved -m --no-retry # You'll likely have to run it The following page also shows [how to remove different Juju objects](https://docs.jujucharms.com/2.1/en/charms-destroy) -#### Dump Juju Logs +### Dump Juju Logs To dump the Juju debug-logs, run this command: @@ -450,7 +396,7 @@ juju debug-log --replay --no-tail -m juju debug-log --replay --no-tail -m --include ``` -#### Manual recovery of Juju +### Manual recovery of Juju If juju gets in a corrupt state and you cannot run `juju status` or contact the juju controller, you might need to remove manually the controller and register again, making OSM aware of the new controller. @@ -502,7 +448,7 @@ docker stack rm osm docker stack deploy -c /etc/osm/docker/docker-compose.yaml osm ``` -#### Slow deployment of charms +### Slow deployment of charms You can make deployment of charms quicker by: @@ -511,17 +457,54 @@ You can make deployment of charms quicker by: - Preventing Juju from running `apt-get update && apt-get upgrade` when starting a machine: [Disable OS upgrades in charms](14-advanced-charm-development.md#disable-os-upgrades) - Building periodically a custom image that will be used as base image for all the charms: [Custom base image for charms](14-advanced-charm-development.md#build-a-custom-cloud-image) -### Instantiation Errors +## Common instantiation errors -#### File juju_id_rsa.pub not found +### File juju_id_rsa.pub not found - **ERROR**: `ERROR creating VCA model name 'xxxx': Traceback (most recent call last): File "/usr/lib/python3/dist-packages/osm_lcm/ns.py", line 822, in instantiate await ... [Errno 2] No such file or directory: '/root/.local/share/juju/ssh/juju_id_rsa.pub'` - **CAUSE**: Normally a migration from release FIVE do not set properly the env for LCM - **SOLUTION**: Ensure variable **OSMLCM_VCA_PUBKEY** is properly set at file `/etc/osm/docker/lcm.env`. The value must match with the output of this command `cat $HOME/.local/share/juju/ssh/juju_id_rsa.pub`. If not, add or change it. Restart OSM, or just LCM service with `docker service update osm_lcm --force --env-add OSMLCM_VCA_PUBKEY=""` -### NBI Errors +## Common issues whwn interacting with NBI + +### SSL certificate problem + +By default, OSM installer uses a self-signed certificate for HTTPS. That might lead to the error '_SSL certificate problem: self signed certificate_' on the client side. For testing environments, you might want to ignore this error just by using the appropriate options to skip certificate validation (e.g. `--insecure` for curl, `--no-check-certificate` for wget, etc.). However, for more stable setups you might prefer to address this issue by installing the appropriate certificate in your client system. + +These are the steps to install NBI certificate on the client side (tested for Ubuntu): + +1. Get the certificate file `cert.pem` by any of these means: + - From running docker container: + ```bash + docker ps | grep nbi + docker cp :/app/NBI/osm_nbi/http/cert.pem . + ``` + - From source code: NBI-folder/osm_nbi/http/cert.pem + - From ETSI's git: + ```bash + wget -O cert.pem "https://osm.etsi.org/gitweb/?p=osm/NBI.git;a=blob_plain;f=osm_nbi/http/cert.pem;hb=refs/heads/v8.0" + ``` +2. Then, you should install this certificate: + ```bash + sudo cp cert.pem /usr/local/share/ca-certificates/osm_nbi_cert.pem.crt + sudo update-ca-certificates + # 1 added, 0 removed; done + ``` +3. Add to the list of `/etc/hosts` a host called "nbi" with the IP address where OSM is running. + - It can be `localhost` if client and server are the same machine. + - For localhost, you would need to add (or edit) these lines: + ```text + 127.0.0.1 localhost nbi + OSM-ip nbi + ``` +4. Finally, for the URL, use the `nbi` as host name (i.e. ). + - Do not use neither `localhost` nor 127.0.0.1. + - You can run a quick test with `curl` by: + ```bash + curl https://nbi:9999/osm/version + ``` -#### Cannot login after migration to 6.0.2 +### Cannot login after migration to 6.0.2 - **ERROR**: NBI always return "UNAUTHORIZED". Cannot login neither with UI nor with CLI. CLI shows error "`can't find a default project for this user`" or "`project admin not allowed for this user`". - **CAUSE**: Normally after a migration to release 6.0.2 There is a slight incompatibility with users created from older versions. @@ -532,6 +515,35 @@ curl --insecure https://localhost:9999/osm/test/db-clear/users docker service update osm_nbi --force ``` +## Other operational issues + +### Running out of disk space + +If you are upgrading frequently your OSM installation, you might face that your disk is running out of space. The reason is that the previous dockers and docker images might be consuming some disk space. Running the following two commands should be enough to clear your docker setup: + +```bash +docker system prune +docker image prune +``` + +If you are still experiencing issues with disk space, logs in one of the dockers could be the cause of your issue. Check the containers that are consuming more space (typically kafka-exporter) + +```bash +du -sk /var/lib/docker/containers/* |sort -n +docker ps |grep +``` + +Then, remove the stack and redeploy it again after doing a prune: + +```bash +docker stack rm osm_metrics +docker system prune +docker image prune +docker stack deploy -c /etc/osm/docker/osm_metrics/docker-compose.yml osm_metrics +``` + +## Logs + ### Checking the logs You can check the logs of any container with the following commands: @@ -552,6 +564,24 @@ docker logs $(docker ps -aqf "name=osm_keystone-db" -n 1) docker logs $(docker ps -aqf "name=osm_prometheus" -n 1) ``` +For live debugging, the following commands can be useful to save the log output to a file and show it in the screen: + +```bash +docker logs -f $(docker ps -aqf "name=osm_mon.1" -n 1) 2>&1 | tee mon-log.txt +docker logs -f $(docker ps -aqf "name=osm_pol" -n 1) 2>&1 | tee pol-log.txt +docker logs -f $(docker ps -aqf "name=osm_lcm" -n 1) 2>&1 | tee lcm-log.txt +docker logs -f $(docker ps -aqf "name=osm_nbi" -n 1) 2>&1 | tee nbi-log.txt +docker logs -f $(docker ps -aqf "name=osm_light-ui" -n 1) 2>&1 | tee light-log.txt +docker logs -f $(docker ps -aqf "name=osm_ro.1" -n 1) 2>&1 | tee ro-log.txt +docker logs -f $(docker ps -aqf "name=osm_ro-db" -n 1) 2>&1 | tee rodb-log.txt +docker logs -f $(docker ps -aqf "name=osm_mongo" -n 1) 2>&1 | tee mongo-log.txt +docker logs -f $(docker ps -aqf "name=osm_kafka" -n 1) 2>&1 | tee kafka-log.txt +docker logs -f $(docker ps -aqf "name=osm_zookeeper" -n 1) 2>&1 | tee zookeeper-log.txt +docker logs -f $(docker ps -aqf "name=osm_keystone.1" -n 1) 2>&1 | tee keystone-log.txt +docker logs -f $(docker ps -aqf "name=osm_keystone-db" -n 1) 2>&1 | tee keystonedb-log.txt +docker logs -f $(docker ps -aqf "name=osm_prometheus" -n 1) 2>&1 | tee prometheus-log.txt +``` + For each container, logs can be found under: ```bash -- GitLab