Common issues and troubleshooting

Installation process

RECOMMENDATION: save a log of your installation:

$ ./install_osm.sh 2>&1 | tee osm_install_log.txt

Were all docker images succesfully built?

Although controlled by the installer, you can check that the following images exist:

$ docker image ls

REPOSITORY               TAG                 IMAGE ID            CREATED             SIZE
osm/light-ui             latest              1988aa262a97        18 hours ago        710MB
osm/lcm                  latest              c9ad59bf96aa        46 hours ago        667MB
osm/ro                   latest              812c987fcb16        46 hours ago        791MB
osm/nbi                  latest              584b4e0084a7        46 hours ago        497MB
osm/pm                   latest              1ad1e4099f52        46 hours ago        462MB
osm/mon                  latest              b17efa3412e3        46 hours ago        725MB
wurstmeister/kafka       latest              7cfc4e57966c        10 days ago         293MB
mysql                    5                   0d16d0a97dd1        2 weeks ago         372MB
mongo                    latest              14c497d5c758        3 weeks ago         366MB
wurstmeister/zookeeper   latest              351aa00d2fe9        18 months ago       478MB

Are all processes/services running?

$ docker stack ps osm |grep -i running

10 docker containers should be running.

All the 10 services should have at least 1 replica: 1/1

$ docker service ls

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
yuyiqh8ty8pv        osm_kafka           replicated          1/1                 wurstmeister/kafka:latest       *:9092->9092/tcp
y585906h5vy5        osm_lcm             replicated          1/1                 osm/lcm:latest
pcdi5vb86nt9        osm_light-ui        replicated          1/1                 osm/light-ui:latest             *:80->80/tcp
i56jhl5k6re4        osm_mon             replicated          1/1                 osm/mon:latest                  *:8662->8662/tcp
p5wyjtne93hp        osm_mongo           replicated          1/1                 mongo:latest
iz5uncfdzu23        osm_nbi             replicated          1/1                 osm/nbi:latest                  *:9999->9999/tcp
4ttw2v4z2g57        osm_pm              replicated          1/1                 osm/pm:latest
xbg6bclp2anw        osm_ro              replicated          1/1                 osm/ro:latest                   *:9090->9090/tcp
sf7rayfolncu        osm_ro-db           replicated          1/1                 mysql:5
5bl73dhj1xl0        osm_zookeeper       replicated          1/1                 wurstmeister/zookeeper:latest

Is Juju up and running?

If running, you should see something like this:

$ juju status

Model    Controller  Cloud/Region         Version  SLA
default  osm         localhost/localhost  2.3.7    unsupported

Addition of VIMs

Is the VIM URL reachable and operational?

When there are problems to access the VIM URL, an error message similar to the following is shown after attempts to instantiate network services:

Error: "VIM Exception vimmconnConnectionException ConnectFailure: Unable to establish connection to <URL>"

• In order to debug potential issues with the connection, in the case of an OpenStack VIM, you can install the OpenStack client in the OSM VM and run some basic tests. I.e.:

$ # Install the OpenStack client
$ sudo apt-get install python-openstackclient
$ # Load your OpenStack credentials. For instance, if your credentials are saved in a file named 'myVIM-openrc.sh', you can load them with:
$ source myVIM-openrc.sh
$ # Test if the VIM API is operational with a simple command. For instance:
$ openstack image list

If the openstack client works, then make sure that you can reach the VIM from the RO docker:

$ docker exec -it osm_ro.1.xxxxx bash
$ curl <URL_CONTROLLER>

In some cases, the errors come from the fact that the VIM was added to OSM using names in the URL that are not Fully Qualified Domain Names (FQDN).

When adding a VIM to OSM, you must use always FQDN or the IP addresses. It must be noted that “controller” or similar names are not proper FQDN (the suffix should be added). Non-FQDN names might be understood by docker’s dnsmasq as a docker container name to be resolved, which is not the case. In addition, all the VIM endpoints should also be FQDN or IP addresses, thus guaranteeing that all subsequent API calls can reach the appropriate endpoint.

Think of an NFV infrastructure with tens of VIMs, first you will have to use different names for each controller (controller1, controller2, etc.), then you will have to add to every machine trying to interact with the different VIMs, not only OSM, all those entries in the /etc/hosts file. This is bad practice.

However, it is useful to have a mean to work with lab environments using non-FQDN names. Three options here. Probably you are looking for the third one, but we recommend the first one:

• Option 1. Change the admin URL and/or public URL of the endpoints to use an IP address or an FQDN. You might find this interesting if you want to bring your Openstack setup to production.
• Option 2. Modify /etc/hosts in the docker RO container. This is not persistent after reboots or restarts of the osm docker stack.
• Option 3. Modify /etc/osm/docker/docker-compose.yaml in the host, adding extra_hosts in the ro section with the entries that you want to add to /etc/hosts in the RO docker:

ro:
  extra_hosts:
    controller: 1.2.3.4

Then restart the stack:

docker stack rm osm
docker stack deploy -c /etc/osm/docker/docker-compose.yaml osm

This is persistent after reboots and restarts of the osm docker stack.

What should I check if the VIM authentication is failing?

Typically, you will get the following error messsage:

Error: "VIM Exception vimconnUnexpectedResponse Unauthorized: The request you have made requieres authentication. (HTTP 401)"

• If your OpenStack URL is based on HTTPS, OSM will check by default the autenticity of your VIM using the appropriate public certificate. You have 2 means to address this case successfully, depending on your intended setup:
  • Install the certificate of your OpenStack in your RO docker container.

$ docker ps |grep osm_ro.1                                   <--- get the name of the RO docker container
$ docker cp <CA_FILE> <RO_DOCKER>:/etc/osm
$ # Create the VIM with all the usual options, and add the config option to specify the certificate
$ osm vim-create VIM-NAME ... --config '{ca_cert: /etc/osm/<CA_FILE>}'   

• • For casual testing, when adding the VIM account to OSM, you can use 'insecure: True' (without quotes) as part of the VIM config parameters:

$ osm vim-create VIM-NAME ... --config '{insecure: True}'   

Is the VIM management network reachable from OSM (e.g. via ssh, port 22)?

The simplest check would consist on deploying a VM attached to the management network and trying to access it via e.g. ssh from the OSM host.

For instance, in the case of an OpenStack VIM you could try something like this:

$ openstack server create --image ubuntu --flavor m1.small --nic mgmtnet test

If this does not work, typically it is due to one of these issues:

• Security group policy in your VIM is blocking your traffic (contact your admin to fix it)
• IP address space in the management network is not routable from outside (or in the reverse direction, for the ACKs).

Platform issues

Running out of disk space If you are upgrading frequently your OSM installation, you might face that your disk is running out of space. The reason is that the previous dockers and docker images might be consuming some disk space. Running the following two commands should be enough to clear your docker setup:

docker system prune
docker image prune

If you are still experiencing issues with disk space, logs in one of the dockers could be the cause of your issue. Check the containers that are consuming more space (typically kafka-exporter)

du -sk /var/lib/docker/containers/* |sort -n
docker ps |grep <CONTAINER_ID>

Then, remove the stack and redeploy it again after doing a prune:

docker stack rm osm_metrics
docker system prune
docker image prune
docker stack deploy -c /etc/osm/docker/osm_metrics/docker-compose.yml osm_metrics