Skip to content

Heal NS when instantiation failed due to a fail in day-1 configuration

Healing the NS/VNF

Proposers

Tolga Gölelcin (Telenity)

Type

Feature

Target MDG/TF

LCM, NBI, N2VC

Description

When a NS fails due to Juju charms that doean not complete or time or other issues, the NS should be able to be healed by rerunnig the charms. When the charms are completed somehow the NS should have a chance to change its state to success.

[Gianpietro] Focus on describing the use case. One of the VNF primitives that are part of the day-1 configuration (initial-config-primitive) failed.

OSM should be able to capture where it was stuck in the instantiation (the step in a sequence that failed) for every VNF.

OSM should be able to receive a heal request from the NBI. That heal request will retry the instantiation from where it was stuck. Then, if the instantiation succeeds, the NS state will change accordingly.

Demo or definition of done

The VM with IP 10.35.42.160 was the last one activated in Openstack and took to much time. That's why I believe that app-vnf-a0f206846b1b could not have connected to VM. There way to make app-vnf-a0f206846b1b finish its job and make NS status success without trying to relaunch.

For example there may be a command such as: osm ns-heal --juju-app osm ns-heal canvas-sms-nsd-instance --juju-app app-vnf-a0f206846b1b

So this may trigger a juju command on model 8ba8f44f-70dd-4b52-8e0b-04013c49ca26 to rerun the juju-app app-vnf-a0f206846b1b. And if it success then NS status will change to Success.

(overcloud) [nfvuser@rh-nfv-director images]$ openstack server list +--------------------------------------+------------------------------------------+--------+-------------------------------------------------+-----------------------------+----------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+------------------------------------------+--------+-------------------------------------------------+-----------------------------+----------------+ | 759e8e04-2fca-4b3e-811d-dce72bc89e1a | canvas-sms-nsd-instance-3-as-smsc-1 | ACTIVE | oam-network=192.169.11.101; public=10.35.42.160 | Canvas-SMSC-AS-SMSC-1.0 | ws-portals-flv | | 716b2bff-ed29-406b-8aa6-ca61c11aeac9 | canvas-sms-nsd-instance-3-db-sfe-1 | ACTIVE | oam-network=192.169.11.203 | Canvas-SMSC-DB-SFE-1.0 | ws-portals-flv | | f0589cb7-1e43-4cf2-828c-3a1718d05ad3 | canvas-sms-nsd-instance-3-db-oam-1 | ACTIVE | oam-network=192.169.11.201 | Canvas-SMSC-DB-OAM-1.0 | ws-portals-flv | | bb630a7c-d6b6-474b-b9a8-5331f1f58c4d | canvas-sms-nsd-instance-2-db-oam-1 | ACTIVE | oam-network=192.169.22.201 | Canvas-PROV-DB-OAM-1.0 | ws-portals-flv | | 1241fae3-4e52-4804-824e-934642ac4a9d | canvas-sms-nsd-instance-2-db-scrm-1 | ACTIVE | oam-network=192.169.22.205 | Canvas-PROV-DB-SCRM-1.0 | ws-portals-flv | | 74fea74f-c172-487a-8b39-ea7edca5c834 | canvas-sms-nsd-instance-2-as-scrm-1 | ACTIVE | oam-network=192.169.22.121; public=10.35.42.171 | Canvas-PROV-AS-SCRM-1.0 | ws-portals-flv | | 6f7072a0-a15d-4394-aef7-ded95df0176a | canvas-sms-nsd-instance-2-db-cmpf-1 | ACTIVE | oam-network=192.169.22.203 | Canvas-PROV-DB-CMPF-1.0 | ws-portals-flv | | 910a3c3c-0f75-482f-a0e5-86f7a21aef36 | canvas-sms-nsd-instance-2-as-cmpf-1 | ACTIVE | oam-network=192.169.22.101; public=10.35.42.169 | Canvas-PROV-AS-CMPF-1.0 | ws-portals-flv | | 9fc20617-038f-4c6f-8d89-57eedf274a43 | canvas-sms-nsd-instance-1-db-analytics-1 | ACTIVE | oam-network=192.169.30.201 | Canvas-OPS-DB-ANALYTICS-1.0 | ws-portals-flv | | aa5a5f3a-2c2b-45b8-8bb1-b15bc5ade802 | canvas-sms-nsd-instance-1-as-reporting-1 | ACTIVE | oam-network=192.169.30.115; public=10.35.42.154 | Canvas-OPS-AS-REPORTING-1.0 | ws-portals-flv | | 2674d447-f8f1-46b5-8090-78828fd42a47 | canvas-sms-nsd-instance-1-db-stats-1 | ACTIVE | oam-network=192.169.30.203 | Canvas-OPS-DB-STATS-1.0 | ws-portals-flv | | 6ee281e2-fef8-4483-8692-51e86ee9f714 | canvas-sms-nsd-instance-1-ws-portals-1 | ACTIVE | oam-network=192.169.30.101; public=10.35.42.153 | Canvas-OPS-WS-PORTALS-1.0 | ws-portals-flv | +--------------------------------------+------------------------------------------+--------+-------------------------------------------------+-----------------------------+----------------+ (overcloud) [nfvuser@rh-nfv-director images]$ ssh root@10.35.42.160 The authenticity of host '10.35.42.160 (10.35.42.160)' can't be established. RSA key fingerprint is SHA256:Dg6roWXFH+HO+oRAclSasbduYLYf2zg0vckW6shSOK8. RSA key fingerprint is MD5:22:3d:fa:f4:ca:41:ca:7a:d0:09:8c:69:53:b2:32:e5. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '10.35.42.160' (RSA) to the list of known hosts. root@10.35.42.160's password: Last login: Thu Feb 27 08:55:09 2020 from 10.35.42.6 [root@smsc-as-smsc ]# exit logout Connection to 10.35.42.160 closed. osm@ob-nfv-mano:/install$ juju status Model Controller Cloud/Region Version SLA Timestamp 8ba8f44f-70dd-4b52-8e0b-04013c49ca26 osm localhost/localhost 2.7.2 unsupported 11:53:44+03:00

App Version Status Scale Charm Store Rev OS Notes app-vnf-9ab9143bf5ad active 1 canvascharm local 3 ubuntu
app-vnf-9ab9143bf5ad-vdu-as-reporting-cnt-z0 active 1 canvascharm local 1 ubuntu
app-vnf-a0f206846b1b blocked 1 canvascharm local 2 ubuntu
app-vnf-dc2f17a9ee19 active 1 canvascharm local 0 ubuntu
app-vnf-dc2f17a9ee19-vdu-vdu-as-scrm-cnt-z0 active 1 canvascharm local 4 ubuntu

Unit Workload Agent Machine Public address Ports Message app-vnf-9ab9143bf5ad-vdu-as-reporting-cnt-z0/0* active idle 3 10.186.219.82 Ready! app-vnf-9ab9143bf5ad/0* active idle 0 10.186.219.49 Ready! app-vnf-a0f206846b1b/0* blocked idle 2 10.186.219.94 Verification failed: Unhandled exception: [Errno None] Unable to connect to port 22 on 10.35.42.160 app-vnf-dc2f17a9ee19-vdu-vdu-as-scrm-cnt-z0/0* active idle 4 10.186.219.79 Ready! app-vnf-dc2f17a9ee19/0* active idle 1 10.186.219.149 Ready!

Machine State DNS Inst id Series AZ Message 0 started 10.186.219.49 juju-11bd69-0 xenial Running 1 started 10.186.219.149 juju-11bd69-1 xenial Running 2 started 10.186.219.94 juju-11bd69-2 xenial Running 3 started 10.186.219.82 juju-11bd69-3 xenial Running 4 started 10.186.219.79 juju-11bd69-4 xenial Running

osm@ob-nfv-mano:~/install$ osm ns-list +-------------------------+--------------------------------------+--------------------+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ns instance name | id | operational status | config status | detailed status | +-------------------------+--------------------------------------+--------------------+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | canvas-sms-nsd-instance | 8ba8f44f-70dd-4b52-8e0b-04013c49ca26 | failed | failed | ERROR Waiting for tasks to be finished: execute primitive 'emptyetchosts' params '{}' <<class 'n2vc.exceptions.N2VCExecutionException'>> Error executing primitive emptyetchosts failed: Error executing primitive emptyetchosts into ee=8ba8f44f-70dd-4b52-8e0b-04013c49ca26.app-vnf-9ab9143bf5ad-vdu-as-reporting-cnt-z0.3 : status is not completed: failed | +-------------------------+--------------------------------------+--------------------+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+