Technical FAQ (Release THREE)
"Instantiation failed", but VMs and networks were successfully created
Q. After trying to instantiate, I got the message that the instantiation failed without much information about the reason. After checking the logs, it seems to be a timeout issue. However, I am seeing that the VMs and networks were created at the VIM.
A. First check in the RO that there is an IP address in the management interface of each VNF of the NS.
lxc exec RO --env OPENMANO_TENANT=osm openmano instance-scenario-list # to identify the running scenarios in the RO lxc exec RO --env OPENMANO_TENANT=osm openmano instance-scenario-list <id> -vvv |grep ip # to get verbose information on a specific scenario in the RO
If no IP address is present in the management interface of each VNF, then you are hitting a SO-RO timeout issue. The reason is typically a wrong configuration of the VIM. The way how management IP addresses are assigned to the VNFs change from one VIM to another. In all the cases, the recommendation is the following:
- Pre-provision a management network in the VIM, with DHCP enabled. You can see, for instance, the instructions in the case of Openstack (https://osm.etsi.org/wikipub/index.php/Openstack_configuration_(Release_TWO) ).
- Then make sure that, at instantiation time, you specify a mapping between the management network in th NS and the VIM network name that you pre-provisioned at the VIM.
If the IP address is present in the management interface, then you are probably hitting a SO-VCA timeout, caused because the VNF configuration via Juju charms takes too long. To confirm, connect to VCA container and check "juju status".
lxc exec VCA -- juju status
Then, if you see an error, you should debug the VNF charm or ask the people providing that VNF package.
"cannot load cookies: file locked for too long" where charms are not loaded
Depending on the remainder of the error message, this most likely means that a condition on the server hosting OSM is not allowing the charm to be loaded.
For instance, the following error indicates that the VCA container is full and cannot host new containers for Juju: "Cannot load charms due to "ERROR cannot load cookies: file locked for too long; giving up: cannot acquire lock: open /root/.local/share/juju/cookies/osm.json.lock: no space left on device".
To solve that, the containers and data created through Juju should be removed. Check the connection to the OSM Juju config agent account. Is it red/unavailable? Check if the service is running on port 17070, inside the Juju controller container that runs inside the VCA container; otherwise restore it.
# Access VCA container
$ lxc exec VCA bash
# Check the Juju status. The command may get stalled if the service is not running in the Juju controller
root@VCA:~# juju status
# Check if Juju is running on the Juju controller
# First, check the name for the LXC with the Juju controller
# In this case, juju_controller_instance_id = 10.44.127.136
root@VCA:~# lxc list | grep "${juju_controller_instance_id}"
| juju-f050fc-0 | RUNNING | 10.44.127.136 (eth0) | | PERSISTENT | 0 |
# Check if the service for the agent account is running. It is not running in this case
root@VCA:~# lxc exec juju-f050fc-0 -- netstat -apen | grep 17070
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
# Check if disk is completely filled in VCA
root@VCA:~# df -h
# If it is, remove some LXC beloging to the failed machines
# Note: keep the Juju controller container! (here, the last row)
# The IP is available on the configuration section, under "Accounts" in the OSM dashboard
root@VCA:~# lxc list
+-----------------+---------+----------------------+------+------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-----------------+---------+----------------------+------+------------+-----------+
| juju-ed3163-260 | RUNNING | 10.44.127.190 (eth0) | | PERSISTENT | 0 |
+-----------------+---------+----------------------+------+------------+-----------+
| juju-ed3163-269 | RUNNING | 10.44.127.69 (eth0) | | PERSISTENT | 0 |
+-----------------+---------+----------------------+------+------------+-----------+
| juju-ed3163-272 | RUNNING | 10.44.127.118 (eth0) | | PERSISTENT | 0 |
+-----------------+---------+----------------------+------+------------+-----------+
| juju-ed3163-277 | RUNNING | 10.44.127.128 (eth0) | | PERSISTENT | 0 |
+-----------------+---------+----------------------+------+------------+-----------+
| juju-ed3163-278 | RUNNING | 10.44.127.236 (eth0) | | PERSISTENT | 0 |
+-----------------+---------+----------------------+------+------------+-----------+
| juju-ed3163-282 | RUNNING | 10.44.127.61 (eth0) | | PERSISTENT | 0 |
+-----------------+---------+----------------------+------+------------+-----------+
| juju-ed3163-283 | RUNNING | 10.44.127.228 (eth0) | | PERSISTENT | 0 |
+-----------------+---------+----------------------+------+------------+-----------+
| juju-f050fc-0 | RUNNING | 10.44.127.136 (eth0) | | PERSISTENT | 0 | # <- Do not remove the Juju controller!
+-----------------+---------+----------------------+------+------------+-----------+
# Example: lxc stop juju-ed3163-260; lxc delete juju-ed3163-260
lxc stop ${name}; lxc delete ${name}
# Clean status in Juju by removing the machines, units and apps that whose LXCs were removed before
root@VCA:~# juju status
Model Controller Cloud/Region Version SLA
default osm localhost/localhost 2.2.2 unsupported
App Version Status Scale Charm Store Rev OS Notes
flhf-testcf-flhfilter-b active 0/1 fl7filter local 20 ubuntu
flhf-testd-flhfilter-b active 0/1 fl7filter local 22 ubuntu
flhf-testdc-flhfilter-b active 0/1 fl7filter local 25 ubuntu
flhf-va-flhfilter-b active 0/1 fl7filter local 21 ubuntu
ids-test-ac-ids-b active 0/1 ids local 1 ubuntu
lala-dpi-b maintenance 0/1 dpi local 4 ubuntu
lhbcdf-lcdfilter-b maintenance 0/1 l23filter local 20 ubuntu
Unit Workload Agent Machine Public address Ports Message
flhf-testcf-flhfilter-b/0 unknown lost 260 10.44.127.190 agent lost, see 'juju show-status-log flhf-testcf-flhfilter-b/0'
flhf-testd-flhfilter-b/1 unknown lost 278 10.44.127.236 agent lost, see 'juju show-status-log flhf-testd-flhfilter-b/1'
flhf-testdc-flhfilter-b/2 unknown lost 282 10.44.127.61 agent lost, see 'juju show-status-log flhf-testdc-flhfilter-b/2'
flhf-va-flhfilter-b/0 unknown lost 272 10.44.127.118 agent lost, see 'juju show-status-log flhf-va-flhfilter-b/0'
ids-test-ac-ids-b/0 unknown lost 277 10.44.127.128 agent lost, see 'juju show-status-log ids-test-ac-ids-b/0'
lala-dpi-b/1 maintenance failed 283 10.44.127.228 installing charm software
lhbcdf-lcdfilter-b/0 unknown lost 269 10.44.127.69 agent lost, see 'juju show-status-log lhbcdf-lcdfilter-b/0'
Machine State DNS Inst id Series AZ Message
260 down 10.44.127.190 juju-ed3163-260 trusty Running
269 down 10.44.127.69 juju-ed3163-269 trusty Running
272 down 10.44.127.118 juju-ed3163-272 trusty Running
277 down 10.44.127.128 juju-ed3163-277 trusty Running
278 down 10.44.127.236 juju-ed3163-278 trusty Running
282 down 10.44.127.61 juju-ed3163-282 trusty Running
283 down 10.44.127.228 juju-ed3163-283 trusty Running
# Example: juju remove-machine 260
juju remove-machine ${"machine" whose ip is related to juju_instance_id} --force
# Start Juju in the juju controller
root@VCA:~# lxc exec juju-f050fc-0 bash
# Cancel if needed or run in background
root@juju-f050fc-0:~# /var/lib/juju/init/jujud-machine-0/exec-start.sh &
^C
# Verify that the process is running
root@juju-f050fc-0:~# sudo netstat -apen | grep 17070
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 127.0.0.1:58402 127.0.0.1:17070 ESTABLISHED 0 22936276 359/jujud
tcp 0 0 10.44.127.136:56746 10.44.127.136:17070 ESTABLISHED 0 22936277 359/jujud
tcp 0 0 10.44.127.136:56740 10.44.127.136:17070 ESTABLISHED 0 22940759 359/jujud
tcp 0 0 127.0.0.1:58432 127.0.0.1:17070 ESTABLISHED 0 22940767 359/jujud
tcp6 0 0 :::17070 :::* LISTEN 0 22940744 359/jujud
tcp6 0 0 127.0.0.1:17070 127.0.0.1:58432 ESTABLISHED 0 22930280 359/jujud
tcp6 0 0 10.44.127.136:17070 10.44.127.136:56740 ESTABLISHED 0 22939104 359/jujud
tcp6 0 0 10.44.127.136:17070 10.44.127.136:56746 ESTABLISHED 0 22936278 359/jujud
tcp6 0 0 127.0.0.1:17070 127.0.0.1:58402 ESTABLISHED 0 22940756 359/jujud
# Go to SO-ub container and restart SO service to connect again against the Juju controller
$ lxc exec SO-ub bash
root@SO-ub:~# service launchpad restart
After this process, access the OSM dashboard and check again the connectivity from the "Accounts" tab. It should be green, and now any new NS instantiated should correctly load its associated charm.
"Instantiation failed" and VMs and network were not created at VIM
Q. After trying to instantiate, I got the message that the instantiation failed without much information about the reason. I connected to the VIM and checked that the VMs are networks were not created
A. You are hitting a SO-RO timeout, caused either by the lack of communication from the RO to the VIM or because the creation of VMs and networks from the RO to the VIMs takes too long.
SO connection error: not possible to contact OPENMANO-SERVER (openmanod)
Q. The NS operational data of an instantiated NS in the SO CLI shows "Connection error: not possible to contact OPENMANO-SERVER (openmanod)"
A. Please check connectivity from SO-ub to RO container. Can you ping the RO IP address (configured in SO) from SO-ub container? If not, then make sure that osm-ro service is up and running on the RO container.
$ lxc exec RO -- bash root@SO-ub:~# service osm-ro status root@SO-ub:~# OPENMANO_TENANT=osm openmano datacenter-list
Deployment fails with the error message "Not possible to get_networks_list from VIM: AuthorizationFailure: Authorization Failed: The resource could not be found. (HTTPS: 404)"
Q. During instantiation, I got, the following error message: Not possible to get_networks_list from VIM: AuthorizationFailure: Authorization Failed: The resource could not be found. (HTTPS: 404)".
A. The cause is that Openstack has not been properly added to openmano with the right credentials
Go to the RO container:
lxc exec RO bash
Install the package python-openstackclient, in case it does not exist:
apt install -y python-openstackclient
Execute the following commands with the appropriate substitution to check that your openstack is reachable and you can do specific actions:
openstack --os-project-name <project-name> --os-auth-url <auth-url> --os-username <auth-username> --os-password <auth-password> --debug network list openstack --os-project-name <project-name> --os-auth-url <auth-url> --os-username <auth-username> --os-password <auth-password> --debug host list openstack --os-project-name <project-name> --os-auth-url <auth-url> --os-username <auth-username> --os-password <auth-password> --debug flavor list openstack --os-project-name <project-name> --os-auth-url <auth-url> --os-username <auth-username> --os-password <auth-password> --debug server list #providing the same URL, credentials as you provide to openmano. The (--debug) will show more info so that you will see the IP/ports it try to access
This is to ensure that you have access to the openstack endpoints and that you are using the right credentials.
Case 1. If any of the previous commands do not work, then either your connectivity to Openstack endpoints is incorrect or you are not using the right parameters. Please check internally and debug until the previous commands work. Check also the guidelines here: OSM_Release_TWO#Openstack_site.
Case 2. If all of them worked, then follow these guidelines:
- Use "v2" authorization URL. "v3" is currently experimental in the master branch and is not recommended yet.
- If https (instead of http) is used for authorization URL, you can either use the insecure option at datacenter-create (See Openstack_configuration_(Release_TWO)#Add_openstack_at_OSM); or install the certificate at RO container, by e.g. putting a .crt (not .pem) certificate at /usr/local/share/ca-certificates and running update-ca-certificates.
- Check the parameters you used to create and attached the datacenter by running the following commands
export OPENMANO_TENANT=osm openmano datacenter-list openmano datacenter-list <DATACENTER_NAME> -vvv
- If all seems right, maybe the password was wrong. Try to detach and delete the datacenter, and then create and attach it again with the right password.
openmano datacenter-detach openstack-site openmano datacenter-delete openstack-site openmano datacenter-create openstack-site http://10.10.10.11:5000/v2.0 --type openstack --description "OpenStack site" openmano datacenter-attach openstack-site --user=admin --password=userpwd --vim-tenant-name=admin
Deployment fails with the error message "VIM Exception vimconnUnexpectedResponse Unauthorized: The request you have made requieres authentication. (HTTP 401)"
Follow the same instructions as for the error "Not possible to get_networks_list from VIM: AuthorizationFailure: Authorization Failed: The resource could not be found. (HTTPS: 404)"
OSM RO service fails to start with a message "DATABASE wrong version"
Q. OSM RO service (osm-ro service in RO container) fails to start and logs show "DATABASE wrong version"
2016-11-02T17:19:51 CRITICAL openmano openmanod:268 DATABASE wrong version '0.15'. Try to upgrade/downgrade to version '0.16' with 'database_utils/migrate_mano_db.sh'
A. The reason is that the RO has been upgraded with a new version that requires a new database version. To upgrade the database version, run database_utils/migrate_mano_db.sh and provide credentials if needed (by default database user is 'mano', and database password is 'manopw')
Deployment fails with the message Error creating image at VIM 'xxxx': Cannot create image without location
The reason of the failure is that ther is a mismatch between image names (and checksum) at OSM VNFD and at VIM. Basically the image is not present at VIM.
To fix it, you should add the image at VIM and ensure that it is visible for the VIM credentials provided to RO. At RO container you can list the VIM images using these credentialas easily with the command:
openmano vim-image-list --datacenter <xxxxx>