2 # Licensed under the Apache License, Version 2.0 (the "License");
3 # you may not use this file except in compliance with the License.
4 # You may obtain a copy of the License at
6 # http://www.apache.org/licenses/LICENSE-2.0
8 # Unless required by applicable law or agreed to in writing, software
9 # distributed under the License is distributed on an "AS IS" BASIS,
10 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11 # See the License for the specific language governing permissions and
12 # limitations under the License.
16 Documentation [HEAL-01] Healing of a multi-volume VDU
18 Library OperatingSystem
24 Resource ../lib/vnfd_lib.resource
25 Resource ../lib/vnf_lib.resource
26 Resource ../lib/nsd_lib.resource
27 Resource ../lib/ns_lib.resource
28 Resource ../lib/ns_operation_lib.resource
29 Resource ../lib/ssh_lib.resource
30 Resource ../lib/openstack_lib.resource
32 Test Tags heal_01 cluster_heal daily regression
34 Suite Teardown Run Keyword And Ignore Error Suite Cleanup
38 # NS and VNF descriptor package folder and ids
39 ${VNFD_VOLUMES_PKG} several_volumes_vnf
40 ${VNFD_VOLUMES_NAME} several_volumes-vnf
41 ${VDU_VOLUMES_NAME} several_volumes-VM
42 ${VNF_SEVERAL_INDEX} several_volumes_vnf
43 ${VNFD_MANUALSCALE_PKG} manual_scale_vnf
44 ${VNFD_MANUALSCALE_NAME} manual_scale-vnf
45 ${NSD_PKG} volumes_healing_ns
46 ${NSD_NAME} volumes_healing-ns
48 # NS instance name and configuration
50 ${NS_CONFIG} {vld: [ {name: mgmtnet, vim-network-name: %{VIM_MGMT_NET}} ] }
53 # SSH keys and username to be used
54 ${PUBLICKEY} %{HOME}/.ssh/id_rsa.pub
55 ${PRIVATEKEY} %{HOME}/.ssh/id_rsa
59 ${SUCCESS_RETURN_CODE} 0
62 @{VIM_VOLUMES} @{EMPTY}
66 Create VNF Descriptors
67 [Documentation] Upload VNF packages for the testsuite.
68 Create VNFD '%{PACKAGES_FOLDER}/${VNFD_MANUALSCALE_PKG}'
69 Create VNFD '%{PACKAGES_FOLDER}/${VNFD_VOLUMES_PKG}'
72 [Documentation] Upload NS package for the testsuite.
73 Create NSD '%{PACKAGES_FOLDER}/${NSD_PKG}'
75 Network Service Instance Test
76 [Documentation] Instantiate NS for the testsuite.
77 ${id}= Create Network Service ${NSD_NAME} %{VIM_TARGET} ${NS_NAME} ${NS_CONFIG} ${PUBLICKEY} ${NS_TIMEOUT}
78 Set Suite Variable ${NS_ID} ${id}
81 [Documentation] Get NS identifier.
83 ${variables}= Get Variables
84 IF not "\${ns_id}" in "${variables}"
85 ${id}= Get Ns Id ${NS_NAME}
86 Set Suite Variable ${NS_ID} ${id}
90 [Documentation] Retrieve all VMs and volumes from the NS and stores them in VIM_VDUS and VIM_VOLUMES lists.
91 Variable Should Exist ${NS_ID} msg=NS is not available
92 @{vnf_id_list}= Get Ns Vnf List ${NS_ID}
94 FOR ${vnf_id} IN @{vnf_id_list}
96 ${id}= Get VNF VIM ID ${vnf_id}
97 @{vdu_ids}= Split String ${id}
98 Append To List ${VIM_VDUS} @{vdu_ids}
100 FOR ${vdu_id} IN @{VIM_VDUS}
101 ${volumes_attached}= Get Server Property ${vdu_id} volumes_attached
102 ${match}= Get Regexp Matches ${volumes_attached} '([0-9a-f\-]+)' 1
103 IF ${match} != @{EMPTY}
104 IF not "${match}[0]" in "@{VIM_VOLUMES}"
105 Append To List ${VIM_VOLUMES} ${match}[0]
110 Log Many @{VIM_VOLUMES}
113 [Documentation] Get VDU ID, IP addresses and volumes of the VNF and stores them in suite variables to be used later on.
114 Variable Should Exist ${NS_ID} msg=NS is not available
115 ${ip_addr}= Get Vnf Management Ip Address ${NS_ID} ${VNF_SEVERAL_INDEX}
117 Set Suite Variable ${VNF_VOLUMES_IP_ADDR} ${ip_addr}
118 ${vnf_id}= Get Vnf Id ${NS_ID} ${VNF_SEVERAL_INDEX}
119 Set Suite Variable ${VNF_VOLUMES_ID} ${vnf_id}
120 ${id}= Get VNF VIM ID ${vnf_id}
121 Set Suite Variable ${VDU_VOLUMES_ID} ${id}
122 Log ${VDU_VOLUMES_ID}
123 @{VOLUMES_IP_LIST}= Get Vnf Vdur IPs ${VNF_VOLUMES_ID}
124 Set Suite Variable @{VOLUMES_IP_LIST} @{VOLUMES_IP_LIST}
125 Log @{VOLUMES_IP_LIST}
128 [Documentation] Get number of volumes from the VNF descriptor and get the attached volumes from the VDU instance.
129 ${rc} ${stdout}= Run And Return RC And Output osm vnfpkg-show ${VNFD_VOLUMES_NAME} --literal | yq '.vdu[0]."virtual-storage-desc" | length'
130 Should Be Equal As Integers ${rc} ${SUCCESS_RETURN_CODE} msg=${stdout} values=False
131 ${num_virtual_storage}= Convert To Integer ${stdout}
132 Set Suite Variable ${VNF_NUM_VOLUMES} ${num_virtual_storage}
133 Log ${VNF_NUM_VOLUMES}
134 ${volumes_attached}= Get Server Property ${VDU_VOLUMES_ID} volumes_attached
135 ${match}= Get Regexp Matches ${volumes_attached} '([0-9a-f\-]+)' 1
136 Set Suite Variable ${VOLUME_ID} ${match}[0]
139 [Documentation] Check that the number of disks in the VDU meets the expected one.
140 Variable Should Exist ${VNF_VOLUMES_IP_ADDR} msg=VNF is not available
141 Sleep 20 seconds Wait for SSH daemon to be up
142 ${stdout}= Execute Remote Command Check Rc Return Output ${VNF_VOLUMES_IP_ADDR} ${USERNAME} ${PASSWORD} ${PRIVATEKEY} sudo lsblk -l
144 ${lines}= Get Lines Containing String ${stdout} disk
145 ${num_lines}= Get Line Count ${lines}
146 IF ${num_lines} < ${VNF_NUM_VOLUMES} Fail msg=Number of disks (${num_lines}) is less than specified in VDU (${VNF_NUM_VOLUMES})
148 Delete Persistent Volume VDU
149 [Documentation] Manually delete the VM in Openstack.
150 Variable Should Exist ${VDU_VOLUMES_ID} msg=VDU is not available
151 Delete Server ${VDU_VOLUMES_ID}
154 Heal Persistent Volume VDU
155 [Documentation] Manually heal the VNF in order to re-create the deleted VM.
156 Variable Should Exist ${VNF_VOLUMES_ID} msg=VNF is not available
157 Heal Network Service ${NS_ID} --vnf ${VNF_VOLUMES_ID} --cause "Heal VM of volumes_vnf" --vdu ${VDU_VOLUMES_NAME}
159 Check VNF After Healing
160 [Documentation] Check that the IDs of the VM and volumes have not changed after healing.
161 Variable Should Exist ${VNF_VOLUMES_ID} msg=VNF is not available
162 @{ip_list}= Get Vnf Vdur IPs ${VNF_VOLUMES_ID}
164 Should Be Equal ${ip_list} ${VOLUMES_IP_LIST} IP addresses have changed after healing
165 ${id}= Get VNF VIM ID ${VNF_VOLUMES_ID}
167 Should Not Be Equal ${id} ${VDU_VOLUMES_ID} VDU id has not changed after healing
168 ${volumes_attached}= Get Server Property ${id} volumes_attached
169 ${match}= Get Regexp Matches ${volumes_attached} '([0-9a-f\-]+)' 1
170 Should Be Equal ${match}[0] ${VOLUME_ID} Volume id has changed after healing
171 Sleep 30 seconds Wait for SSH daemon to be up
172 ${stdout}= Execute Remote Command Check Rc Return Output ${VNF_VOLUMES_IP_ADDR} ${USERNAME} ${PASSWORD} ${PRIVATEKEY} sudo lsblk -l
174 ${lines}= Get Lines Containing String ${stdout} disk
175 ${num_lines}= Get Line Count ${lines}
176 IF ${num_lines} < ${VNF_NUM_VOLUMES} Fail msg=Number of disks (${num_lines}) is less than specified in VDU (${VNF_NUM_VOLUMES})
179 [Documentation] Retrieve all VMs and volumes from the NS and stores them in VIM_VDUS and VIM_VOLUMES lists.
180 ... This is done again to guarantee that all objects are cleaned in the VIM in case the heal operation
181 ... added new objects.
182 Variable Should Exist ${NS_ID} msg=NS is not available
183 @{vdu_updated}= Create List
184 @{vnf_id_list}= Get Ns Vnf List ${NS_ID}
185 FOR ${vnf_id} IN @{vnf_id_list}
186 ${id}= Get VNF VIM ID ${vnf_id}
187 @{vdu_ids}= Split String ${id}
188 Append To List ${vdu_updated} @{vdu_ids}
189 FOR ${id} IN @{vdu_ids}
190 IF not "${id}" in "@{VIM_VDUS}"
191 Append To List ${VIM_VDUS} ${id}
195 FOR ${vdu_id} IN @{vdu_updated}
196 ${volumes_attached}= Get Server Property ${vdu_id} volumes_attached
197 ${match}= Get Regexp Matches ${volumes_attached} '([0-9a-f\-]+)' 1
198 IF ${match} != @{EMPTY}
199 IF not "${match}[0]" in "@{VIM_VOLUMES}"
200 Append To List ${VIM_VOLUMES} ${match}[0]
205 Log Many @{VIM_VOLUMES}
208 [Documentation] Delete NS instance.
213 [Documentation] Delete NS package from OSM.
215 Delete NSD ${NSD_NAME}
217 Delete VNF Descriptors
218 [Documentation] Delete VNF packages from OSM.
220 Delete VNFD ${VNFD_VOLUMES_NAME}
221 Delete VNFD ${VNFD_MANUALSCALE_NAME}
223 Delete Remaining Objects in VIM
224 [Documentation] Delete any remaining objects (volumes, VMs, etc.) in the VIM.
226 Delete Objects In VIM
231 [Documentation] Test Suite Cleanup: Deleting Descriptor, instance and vim
232 Run Keyword If Any Tests Failed Delete NS ${NS_NAME}
233 Run Keyword If Any Tests Failed Delete NSD ${NSD_NAME}
234 Run Keyword If Any Tests Failed Delete VNFD ${VNFD_VOLUMES_NAME}
235 Run Keyword If Any Tests Failed Delete VNFD ${VNFD_MANUALSCALE_NAME}
236 Run Keyword If Any Tests Failed Delete Objects In VIM
238 Delete Objects In VIM
239 [Documentation] Clean up remaining VMs and volumes directly from the VIM.
240 ${error}= Set Variable 0
241 FOR ${vol_id} IN @{VIM_VOLUMES}
242 Log Checking if volume ${vol_id} is still in VIM
243 ${exists}= Check If Volume Exists ${vol_id}
245 ${error}= Set Variable 1
246 Log Deleting volume ${vol_id}
247 Run Keyword And Ignore Error Delete Volume ${vol_id}
250 FOR ${vdu_id} IN @{VIM_VDUS}
251 Log Checking if server ${vdu_id} is still in VIM
252 ${status}= Run Keyword And Ignore Error Get Server Property ${vdu_id} id
254 IF '${status}[0]' == 'PASS'
255 ${error}= Set Variable 1
256 Log Deleting server ${vdu_id}
257 Run Keyword And Ignore Error Delete Server ${vdu_id}
260 IF ${error}==1 Fail Some objects created by test were not deleted in VIM