OSM Public Wiki - User contributions [en]

OSM RO VNFFG implementation

2018-03-16T14:36:08Z

Cardosoi: Completed first version of VNFFG implementation doc

== Virtual Network Function Forwarding Graph Descriptor (VNFFGD) in OSM ==

[[File:vnffg_fig1.png|700px|Figure 1: VNFFGD Data Model from OSM R2 Information Model.]]

VNFFGD is a well-defined data model from ETSI NFV (http://www.etsi.org/deliver/etsi_gs/NFV-IFA/001_099/014/02.01.01_60/gs_NFV-IFA014v020101p.pdf) which was subsequently adopted by OSM in its own informational model (https://osm.etsi.org/wikipub/images/2/26/OSM_R2_Information_Model.pdf).

'''So how is VNFFG architected in OSM, specifically RO, and how is that communicated to the VIM layer by making use of standard Service Function Chaining?'''

[[File:vnffg_fig2.png|700px|Figure 2: OSM R3 Architecture overlaid with changes made for VNFFGD support (in see numbers).]]

All of the changes to enable VNFFG support in OSM have been applied to the RO (Resource Orchestrator) module, the fundamental component that needs such support (since it interacts with VIMs and/or SDN controllers). Additional VNFFG-related changes should be enabled in other projects, such as the Service Orchestrator.

Figure 2 above shows the overall OSM architecture and delineates 4 components that were changed within the RO in order to support VNFFGD to VIM-level SFC.

=== API Service & Utilities ===

Best categorized as being part of the API Service & Utilities (changed element #1) in Figure 2, YAML template examples of a Network Service with a VNFFG and a new VNF have been added to the examples folder of the OSM RO:
<pre>scenarios/examples/v3_3vnf_2vdu_1vnffg_nsd.yaml</pre>
<pre>vnfs/examples/v3_2vdu_vnfd.yaml</pre>

Figure 3 below outlines the example VNFFG and also, in light green, that it could expand to multiple Rendered Service Paths (RSPs).

[[File:vnffg_fig3.png|700px|Figure 3: Example of a VNFFG (dark green delimits the example VNFFG contributed).]]

The VNFFGD is composed of multiple RSPs (Rendered Service Paths, or Network Forwarding Paths in pure ETSI NFV terminology), and each of those states what VNFs should process the traffic and what actual traffic should be processed (the Classifier). A Classifier is a list of match attributes, and they can be traffic classifications such as HTTP. By definition, a VNF can have multiple VDUs which are, essentially, instances or units of that VNF – mainly used for scaling.

=== Resource Orchestrator Engine ===

The database definition and migration present in <code>database_utils/migrate_mano_db.sh</code> have been changed to support the storing of VNFFGD and related models. They are logically part of the Resource Orchestrator Engine, outlined as changed element #2 in the figure above.

Also part of this component is the NFVO engine itself, split between <code>nfvo.py</code>, <code>nfvo_db.py</code> and <code>vim_thread.py</code>.
Regarding the first two, changes were made to process the new VNFFG element of a Network Service Descriptor (NSD), provided via YAML file, and store its information in internal database tables, also mapping to existing descriptors such as VNFDs. An HTTP server runs automatically and interacts with the Resource Orchestrator Engine to provide/request data.
Regarding <code>vim_thread.py</code>, changes were made so that, when a Network Scenario Instance is created, it is able to process existing NSDs that include a VNFFGD. This module’s role is to call the VIM Plugin component outlined in Figure 2 and, to support VNFFGD, special calls have to be made to the VIM Plugin. Essentially, the upper part of Resource Orchestrator Engine will insert tasks in the VIM thread, so that it can call VIM connector’s normalized interface for the creation of networks, functions, service function chains (the eventual transformation of a VNFFG), etc.

=== VIM Plugin ===

Or VIM Connector, aims at providing a unified, abstract interface for the calling of typical VIM operations, such as creating compute units or network functions on a Virtual Infrastructure Manager or Cloud Management System. OSM allows different VIMs to be plugged in, and this is the plug point for those. This component has been changed to provide a unified Service Function Chaining interface, which will be called by the VIM thread after the Resource Orchestrator Engine has translated the VNFFGD into specific tasks to be carried by the VIM thread.

=== OpenStack VIM Plugin ===

A specific implementation of the Service Function Chaining interface added to the abstract VIM plugin was developed for OpenStack. This implementation calls the OpenStack Queen's networking-sfc API (https://docs.openstack.org/networking-sfc/queens) which leverages previous endeavours the team has done, namely the NSH dataplane support when using the Neutron Open vSwitch driver, making OSM an NFV Orchestrator able to instantiate NSH-based SFCs via the definition of a VNFFGD. This brings together efforts from ETSI NFV and IETF SFC.

For reference, the VIM Plugin interface for SFC contains the following methods:
<pre>
def new_classification(self, name, ctype, definition)
def get_classification(self, classification_id)
def get_classification_list(self, filter_dict={})
def delete_classification(self, classification_id)
def new_sfi(self, name, ingress_ports, egress_ports, sfc_encap=True)
def get_sfi(self, sfi_id)
def get_sfi_list(self, filter_dict={})
def delete_sfi(self, sfi_id)
def new_sf(self, name, sfis, sfc_encap=True)
def get_sf(self, sf_id)
def get_sf_list(self, filter_dict={})
def delete_sf(self, sf_id)
def new_sfp(self, name, classifications, sfs, sfc_encap=True, spi=None)
def get_sfp(self, sfp_id)
def get_sfp_list(self, filter_dict={})
def delete_sfp(self, sfp_id)
</pre>

Which use IETF SFC terminology, given that this is an appropriate level to apply IETF SFC concepts (service plane and SFC domain)

OSM RO VNFFG implementation

2018-03-16T14:07:52Z

Cardosoi: Updated images

== Virtual Network Function Forwarding Graph Descriptor (VNFFGD) in OSM ==

[[File:vnffg_fig1.png|600px|Figure 1: VNFFGD Data Model from OSM R2 Information Model.]]
[[File:vnffg_fig2.png|600px|Figure 2: OSM R3 Architecture overlaid with changes made for VNFFGD support (in see numbers).]]
[[File:vnffg_fig3.png|600px|Figure 3: Example of a VNFFG (dark green delimits the example VNFFG contributed).]]

File:Vnffg fig3.png

2018-03-16T14:04:13Z

Cardosoi: Example of a VNFFG (dark green delimits the example VNFFG contributed).

Example of a VNFFG (dark green delimits the example VNFFG contributed).

File:Vnffg fig2.png

2018-03-16T14:03:33Z

Cardosoi: Example of a VNFFG (dark green delimits the example VNFFG contributed).

Example of a VNFFG (dark green delimits the example VNFFG contributed).

File:Vnffg fig1.png

2018-03-16T14:03:10Z

Cardosoi: OSM R3 Architecture overlaid with changes made for VNFFGD support (in see numbers).

OSM R3 Architecture overlaid with changes made for VNFFGD support (in see numbers).

OSM RO VNFFG implementation

2018-03-16T14:02:32Z

Cardosoi: all image references

== Virtual Network Function Forwarding Graph Descriptor (VNFFGD) in OSM ==

[[File:vnffg_fig1.png||Figure 1: VNFFGD Data Model from OSM R2 Information Model.]]
[[File:vnffg_fig2.png||Figure 2: OSM R3 Architecture overlaid with changes made for VNFFGD support (in see numbers).]]
[[File:vnffg_fig3.png||Figure 3: Example of a VNFFG (dark green delimits the example VNFFG contributed).]]

OSM RO VNFFG implementation

2018-03-16T13:56:56Z

Cardosoi: testing figure add

== Virtual Network Function Forwarding Graph Descriptor (VNFFGD) in OSM ==

[[File:Example.jpg||Figure 1: VNFFGD Data Model from OSM R2 Information Model]]

OSM RO VNFFG implementation

2018-03-16T13:54:24Z

Cardosoi: reserving page for VNFFG

Developer HowTo for RO Module

2017-10-23T13:01:30Z

Cardosoi: /* RO Client modules */

==Getting Started==
The RO Module is implemented by openmano. The recommended Linux distribution for development is Ubuntu 16.04 LTS Server. PyCharm is a nice and easy to use tool for development and debugging.

The step [[Release_1_Installation]] installs all OSM modules in containers. However, for development, a virtual machine may be more suitable. To install the openmano module you can run a script that installs the required packages, clone the project from ''git clones https://osm.etsi.org/gerrit/osm/RO'' and configures openmano client and other utility scripts at path
wget -O install-openmano.sh "https://osm.etsi.org/gitweb/?p=osm/RO.git;a=blob_plain;f=scripts/install-openmano.sh"
chmod +x install-openmano.sh
sudo ./install-openmano.sh -q --develop #-h for help

See also and follow [[Workflow_with_OSM_tools#Clone_your_project]]

New code features must be incorporated to master:
git checkout master
Generate a ".gitignore", you can use the ''.gitignore-common'' example that skips PyCharm and Eclipse files
cp RO/.gitignore-common RO/.gitignore
#edit to include your local files to ignore
Prepare your git environment to push with a proper user/email, push to gerrit. See and configure:
[[Workflow_with_OSM_tools#Configure_your_Git_environment]]
[[Workflow_with_OSM_tools#Commit_changes_to_your_local_project]]
[[ Workflow_with_OSM_tools#Push_your_contribution_to_Gerrit]]

==Programming Language==
The RO module uses Python2. However Python3 conventions for a possible future migration should be used as far as possible. For example:

{| class="wikitable"
|-
! BAD Python2
! OK Python2 compatible with python3
|-
| "format test string %s number %d" % (st, num)
| "format test string {} number {}".format(st, num)
|-
| print a, b, c
| print(a,b,c)
|-
| except Exception, e
| except Exception as e
|-
| if type(x) == X:
| if isinstance(x,X):
|}

Descriptors can be YAML (prefered because more readable and allow comments) or JSON

==Code Style==
Please follow PEP8 style guide for all the Python code.

===Logging===
Use the appropriate logging levels when logging the messages. An example is shown below:
self.logger.debug("Changing state to %s", next_state)

Logging levels (general and per module) are specified at '''openmanod.cfg'''

Try to use few useful logs, not verbose, that brings useful information. For example, in case of fail getting a server, the complete URL should be provided.

Avoid several logs together

WRONG:
self.looger.debug("Entering in method A")
self.logger.debug("Contacting server"

RIGHT:
self.logger.debug("method A, contacting server %s", url)

When the traceback is needed(call stack that generate the exception) , use the "exc_info=True" parameter
self.logger.error("Exception %s when ...", exception, exc_info=True)

===Exceptions===
Code must be wrote in a way that functions and methods raise an exception when something goes wrong, instead of returning a negative or false value.

Example
WRONG
def get_ip_address():
...
if fail:
return False, "Fail because xxx"
return True, ip

...
result, ip = get_ip_address()
if not result:
return False, "Cannot get ip address..."

RIGHT
def get_ip_address():
...
if fail:
raise customException("Fail because ...")
return ip

...
try:
ip = get_ip_address()
...
except customException as e:
raise customException2(str(e))

==Directory Organization==
The code organized into the following high level directories: 
* '''/''' contains the entry server file ''openmanod'' and client ''openmano'' code 
* '''/osm_ro''' contains the RO server code files 
* '''/test''' contains scripts and code for testing 
* '''/database_utils''' contains scripts for database creation, dumping and migration 
* '''/scripts''' general scripts, as installation, execution, reporting 
* '''/scenarios''' examples and templates of network scnario descriptors 
* '''/vnfs''' examples and templates of VNF descriptors 

==RO Architecture==
[[File:OpenmanoArchitecture.png|400px]]

===RO Server modules===
The RO module contains the following modules
* '''openmanod''' is the main program. It reads the configuration file (openmanod.cfg) and execute the httpserver and wait for the end
* '''httpserver.py''' is a thread that implements the northbound API interface, uses python bottle module. Calls main engine methods to perform the tasks
* '''nfvo.py''' is the main engine, implementing all the methods for the creation, deletion and management of vnfs, scenarios and instances. Operations against a VIM are asynchornous. ACTIONs to be done are stored at database before returning ok to the client
* '''nfvo_db.py''' is the module in charge of database operations. Uses base '''db_base.py'''. Database is managed with MySQLdb python library
* '''openmano_schemas.py''' is a dictionary schemas used to validate API request and response content using jsonschema library
* '''vim_thread.py''' There is a thread per VIM and credentials. It performs basic tasks of creating/deleting VM, networks, flavors, etc. In addition it refreshes the VM and network status. It calls vimconn.py methods
* '''vimconn.py''' is the base class for the VIM plugin. It contains the definition of the methods to be implemented. The inherited '''vimconn_openstack.py''', '''vimconn_openvim.py''', '''vimconn_vmware''' and '''vimconn_aws''' implements the operations against the concrete VIM type. OpenStack plugin uses the python-*client libraries, meanwhile Openvim plugin uses direct http requests.
* '''console_proxy_thread.py''' is a thread that implements a TCP/IP proxy for the console access to a VIM

===RO Client modules===
Other modules not part of the server are:
* '''openmano''' is a CLI client
* '''openmanoclient.py''' is a client python library for managing openmano server

===ACTIONS and TASKS===
ACTIONS are a a group of tasks performed against a a concrete instance-scenarios (NS record). The creation and deletion of the instance-scenario itself is an action. As it is asynchonous, NBI returns an "Action_id" that can be used to check the status. For each action, nfvo.py generates individual tasks for the related VIMs. Tasks are both stored at database and sent to the related vim_thread.py. A task has a concrete relation with a VIM, e.g. create/delete a VM, a network, ... look for a flavor, network, ...

===Database content===
TODO

==Database changes==
Database schema can be changed if needed. It is recomended to use a graphical tool (e.g. Heidy) to change database and change it back and copy the SQL commands. Make this steps:

1. openmanod: increment __version__, version_date and database_version
2. database_utils/migrate_mano_db.sh. See the three "TODO"
2a. Increment LAST_DB_VERSION
2b. Annotate a comment to have a track of versions: [ $OPENMANO_VER_NUM -ge 50XX ] && DB_VERSION=XX #0.5.XX => XX
3c. Generate new methods function upgrade_to_XX() and function downgrade_from_XX. Insert here the sql commands. Last sql command over schema_version is quite important to detect the database version.

Test several upgrades/downgrades to version 20 with "migrate_mano_db.sh" and "migrate_mano_db.sh 20"

==Package dependency osm-im==
It is under IM repository. It contains the OSM models. Used models at RO are under <IM>/models/yang/vnfd.yang and <IM>/models/yang/nsd.yang. More modules will be incorporated in the future.

When IM is maked, pyangbind generates python source files imported at RO. They are used to validate and load a yaml/json VNFD and NSD catalog descriptor.

Method pybindJSONDecoder.load_ietf_json from pyangbind is used to parse and load the descriptor. A structure of nested YANG classes are generated and used at <RO>/osm_ro/nfvo.py methods new_vnfd_v3 and new_nsd_v3. RO supports backward compatibility with old format descriptors. NBI (osm_ro/httpserver.py) uses .../v3/... at URL to diferenciate between old and new descriptors. CLI (openmano) detects automatically if the descriptor is in old format or in OSM format to send the request to the server using the old URL or the new "v3" URL

===Installation===
The OSM binary installation install this package. In case of needed you can install/update it manually by:

apt-get update
apt-get install make git python python-pip tox debhelper python-bitarray python-lxml
pip install --upgrade pip
# If not inside a container. Use "sudo -HE" to get root privileges for pip installation
pip install pyangbind stdeb
git clone https://osm.etsi.org/gerrit/osm/IM
make -C IM clean all
dpkg -i IM/deb_dist/python-pyang_*.deb
dpkg -i IM/deb_dist/python-pyangbind_*.deb
dpkg -i IM/deb_dist/python-osm-im_*.deb
# Use this line to check if it is installed and where
python -c 'import osm_im; print osm_im.__path__[0]'

==Package dependency lib-osm-openvim==

It is under openvim repository. RO uses a library for the SDN assist that is in charge of performing the underlay dataplane connectivity using an openflow controller. osm-openvim and lib-osm-openvim are diffent, though they share same pieces of code. lib-osm-openvim uses a different database name (mano_vim_db) so that openvim can be installed in the same virtual machine or container (openvim uses vim_db database name)

TODO complete where RO uses it

===Installation===
The OSM binary installation install this package. In case of needed you can install/update it manually by:

apt-get update
apt-get install make git python tox libmysqlclient-dev
git clone https://osm.etsi.org/gerrit/osm/openvim
git -C openvim checkout 005a9dc # this is temporal, because last version contains error at Makefile
make -C openvim lite
# Use this line to check if it is installed and where
python -c 'import lib_osm_openvim; print lib_osm_openvim.__path__[0]'
# Install database of ovim library
OSMLIBOVIM_PATH=`python -c 'import lib_osm_openvim; print lib_osm_openvim.__path__[0]'`
# -U and -P are the admin database user/password. Normally "-U root" without password "executed as root"
${OSMLIBOVIM_PATH}/database_utils/install-db-server.sh -U root [-P passwd] -u mano -p manopw -d mano_vim_db --updatedb

==CLI client==

The RO code contains a python CLI (openmano) that allows friently command execution. This CLI client can run on a separate machine where openmano server is running. "openmano config" indicates where the server is (by default localhost)

==Northbound Interface==

The RO uses a REST API with YAML/JSON content. The primitives are explained [[RO Northbound Interface|here]]

==Running Unit Tests==

===Launching RO===
Openmano can run as systemd service. (Can be installed with '''./scripts/install-openmano-service.sh -f RO''')

Or can be run inside a screen (the prefered method for developpers). The script to easily launch/remove is '''./scripts/service-openmano.sh'''. Execute with -h to see options

Note that the general OSM install script, installs openmano inside a container and code run at folder /opt/openmano (instead of $HOME/openmano)

===Tests===
Many of openmano operations rely on an external VIM. Without external infraestructure it is recomended to use openvim in "test" (fake) mode. Install it in the same machine where openmano is located with [[OpenVIM_installation_(Release_One)#Installation]]

Run ''./test/basictest'' for a initial test. Type -h to see options. For testing using openvim just run <pre> . ./test/basictest.sh --force --screen --init-openvim </pre>

==Creating a new VIM plugin==

Choose a name, eg. XXX

Create a new python module vimconn_XXX.py derived from vimconn.py class. Implement the relevant functions. You have several connectors already created to be used as example as openstack and openvim. Openstack connector uses the python openstack client libraries, meanwhile openvim connector uses direct http requests.

DO NOT change the method names, parameters or parameter content. Openmano use the same methods for all the VIMs and they cannot be changed to accomodate VIM specifics. VIM specifics must be solved inside the connector.

The new module can need specific configuration for the VIM that it is passed as a dictionary in the ''config'' variable at constructor. For example, in the case of openstack, ''config'' variable is used for: enabling/disabling port_security_enable, specifying the name of the physical network used for dataplane, regions, etc. The ''config'' variable is the right place to specify those needed parameters not provided by openmano at the methods that form part of the VIM configuration. See [[Openstack configuration (Release TWO)#Add openstack toOSM]] and [[Configuring AWS for OSM Release TWO#Add AWS to OSM]] for examples

Integration with the main project is automatic, just create a new datacenter of type XXX and this module will be automatically loaded. No other module of the main program need to be updated.

openmano tenant-create osm # if not already done
export OPENMANO_TENANT=osm
openmano datacenter-create <vim-name> <access-URL> --type=XXX --config <specific config>
# if fails the cause can be a librery you need not installed or an error upon importing. Try tho import it in a python client
# example of <specific config> text: "{dataplane_physical_net: physnet_sriov, port_security_enabled: False}"
openmano datacenter-attach <vim-name> --vim-tenant-name=<tenant-to-use-by-openmano> --user=<tenant-user> --password=<tenant-password>

You can test it using the '''test/test_RO.py''' file:
# Create manually an image <image> and a management network <net-mgmt> at your VIM
./test/test_RO.py deploy --help
# e.g. ./test/test_RO.py deploy -n <net-mgmt> -t osm -i <image> -d <vim-name> --timeout=30 --test simple_linux,simple_multi_vnfc,simple_2_vnf
./test/test_RO.py vimconn --help

Developer HowTo for RO Module

2017-10-23T13:00:35Z

Cardosoi: /* RO Server modules */

==Getting Started==
The RO Module is implemented by openmano. The recommended Linux distribution for development is Ubuntu 16.04 LTS Server. PyCharm is a nice and easy to use tool for development and debugging.

The step [[Release_1_Installation]] installs all OSM modules in containers. However, for development, a virtual machine may be more suitable. To install the openmano module you can run a script that installs the required packages, clone the project from ''git clones https://osm.etsi.org/gerrit/osm/RO'' and configures openmano client and other utility scripts at path
wget -O install-openmano.sh "https://osm.etsi.org/gitweb/?p=osm/RO.git;a=blob_plain;f=scripts/install-openmano.sh"
chmod +x install-openmano.sh
sudo ./install-openmano.sh -q --develop #-h for help

See also and follow [[Workflow_with_OSM_tools#Clone_your_project]]

New code features must be incorporated to master:
git checkout master
Generate a ".gitignore", you can use the ''.gitignore-common'' example that skips PyCharm and Eclipse files
cp RO/.gitignore-common RO/.gitignore
#edit to include your local files to ignore
Prepare your git environment to push with a proper user/email, push to gerrit. See and configure:
[[Workflow_with_OSM_tools#Configure_your_Git_environment]]
[[Workflow_with_OSM_tools#Commit_changes_to_your_local_project]]
[[ Workflow_with_OSM_tools#Push_your_contribution_to_Gerrit]]

==Programming Language==
The RO module uses Python2. However Python3 conventions for a possible future migration should be used as far as possible. For example:

{| class="wikitable"
|-
! BAD Python2
! OK Python2 compatible with python3
|-
| "format test string %s number %d" % (st, num)
| "format test string {} number {}".format(st, num)
|-
| print a, b, c
| print(a,b,c)
|-
| except Exception, e
| except Exception as e
|-
| if type(x) == X:
| if isinstance(x,X):
|}

Descriptors can be YAML (prefered because more readable and allow comments) or JSON

==Code Style==
Please follow PEP8 style guide for all the Python code.

===Logging===
Use the appropriate logging levels when logging the messages. An example is shown below:
self.logger.debug("Changing state to %s", next_state)

Logging levels (general and per module) are specified at '''openmanod.cfg'''

Try to use few useful logs, not verbose, that brings useful information. For example, in case of fail getting a server, the complete URL should be provided.

Avoid several logs together

WRONG:
self.looger.debug("Entering in method A")
self.logger.debug("Contacting server"

RIGHT:
self.logger.debug("method A, contacting server %s", url)

When the traceback is needed(call stack that generate the exception) , use the "exc_info=True" parameter
self.logger.error("Exception %s when ...", exception, exc_info=True)

===Exceptions===
Code must be wrote in a way that functions and methods raise an exception when something goes wrong, instead of returning a negative or false value.

Example
WRONG
def get_ip_address():
...
if fail:
return False, "Fail because xxx"
return True, ip

...
result, ip = get_ip_address()
if not result:
return False, "Cannot get ip address..."

RIGHT
def get_ip_address():
...
if fail:
raise customException("Fail because ...")
return ip

...
try:
ip = get_ip_address()
...
except customException as e:
raise customException2(str(e))

==Directory Organization==
The code organized into the following high level directories: 
* '''/''' contains the entry server file ''openmanod'' and client ''openmano'' code 
* '''/osm_ro''' contains the RO server code files 
* '''/test''' contains scripts and code for testing 
* '''/database_utils''' contains scripts for database creation, dumping and migration 
* '''/scripts''' general scripts, as installation, execution, reporting 
* '''/scenarios''' examples and templates of network scnario descriptors 
* '''/vnfs''' examples and templates of VNF descriptors 

==RO Architecture==
[[File:OpenmanoArchitecture.png|400px]]

===RO Server modules===
The RO module contains the following modules
* '''openmanod''' is the main program. It reads the configuration file (openmanod.cfg) and execute the httpserver and wait for the end
* '''httpserver.py''' is a thread that implements the northbound API interface, uses python bottle module. Calls main engine methods to perform the tasks
* '''nfvo.py''' is the main engine, implementing all the methods for the creation, deletion and management of vnfs, scenarios and instances. Operations against a VIM are asynchornous. ACTIONs to be done are stored at database before returning ok to the client
* '''nfvo_db.py''' is the module in charge of database operations. Uses base '''db_base.py'''. Database is managed with MySQLdb python library
* '''openmano_schemas.py''' is a dictionary schemas used to validate API request and response content using jsonschema library
* '''vim_thread.py''' There is a thread per VIM and credentials. It performs basic tasks of creating/deleting VM, networks, flavors, etc. In addition it refreshes the VM and network status. It calls vimconn.py methods
* '''vimconn.py''' is the base class for the VIM plugin. It contains the definition of the methods to be implemented. The inherited '''vimconn_openstack.py''', '''vimconn_openvim.py''', '''vimconn_vmware''' and '''vimconn_aws''' implements the operations against the concrete VIM type. OpenStack plugin uses the python-*client libraries, meanwhile Openvim plugin uses direct http requests.
* '''console_proxy_thread.py''' is a thread that implements a TCP/IP proxy for the console access to a VIM

===RO Client modules===
Other modules not part of the server are:
* '''openmnao''' is a CLI client
* '''openmanoclient.py''' is a client python library for managing openmano server

===ACTIONS and TASKS===
ACTIONS are a a group of tasks performed against a a concrete instance-scenarios (NS record). The creation and deletion of the instance-scenario itself is an action. As it is asynchonous, NBI returns an "Action_id" that can be used to check the status. For each action, nfvo.py generates individual tasks for the related VIMs. Tasks are both stored at database and sent to the related vim_thread.py. A task has a concrete relation with a VIM, e.g. create/delete a VM, a network, ... look for a flavor, network, ...

===Database content===
TODO

==Database changes==
Database schema can be changed if needed. It is recomended to use a graphical tool (e.g. Heidy) to change database and change it back and copy the SQL commands. Make this steps:

1. openmanod: increment __version__, version_date and database_version
2. database_utils/migrate_mano_db.sh. See the three "TODO"
2a. Increment LAST_DB_VERSION
2b. Annotate a comment to have a track of versions: [ $OPENMANO_VER_NUM -ge 50XX ] && DB_VERSION=XX #0.5.XX => XX
3c. Generate new methods function upgrade_to_XX() and function downgrade_from_XX. Insert here the sql commands. Last sql command over schema_version is quite important to detect the database version.

Test several upgrades/downgrades to version 20 with "migrate_mano_db.sh" and "migrate_mano_db.sh 20"

==Package dependency osm-im==
It is under IM repository. It contains the OSM models. Used models at RO are under <IM>/models/yang/vnfd.yang and <IM>/models/yang/nsd.yang. More modules will be incorporated in the future.

When IM is maked, pyangbind generates python source files imported at RO. They are used to validate and load a yaml/json VNFD and NSD catalog descriptor.

Method pybindJSONDecoder.load_ietf_json from pyangbind is used to parse and load the descriptor. A structure of nested YANG classes are generated and used at <RO>/osm_ro/nfvo.py methods new_vnfd_v3 and new_nsd_v3. RO supports backward compatibility with old format descriptors. NBI (osm_ro/httpserver.py) uses .../v3/... at URL to diferenciate between old and new descriptors. CLI (openmano) detects automatically if the descriptor is in old format or in OSM format to send the request to the server using the old URL or the new "v3" URL

===Installation===
The OSM binary installation install this package. In case of needed you can install/update it manually by:

apt-get update
apt-get install make git python python-pip tox debhelper python-bitarray python-lxml
pip install --upgrade pip
# If not inside a container. Use "sudo -HE" to get root privileges for pip installation
pip install pyangbind stdeb
git clone https://osm.etsi.org/gerrit/osm/IM
make -C IM clean all
dpkg -i IM/deb_dist/python-pyang_*.deb
dpkg -i IM/deb_dist/python-pyangbind_*.deb
dpkg -i IM/deb_dist/python-osm-im_*.deb
# Use this line to check if it is installed and where
python -c 'import osm_im; print osm_im.__path__[0]'

==Package dependency lib-osm-openvim==

It is under openvim repository. RO uses a library for the SDN assist that is in charge of performing the underlay dataplane connectivity using an openflow controller. osm-openvim and lib-osm-openvim are diffent, though they share same pieces of code. lib-osm-openvim uses a different database name (mano_vim_db) so that openvim can be installed in the same virtual machine or container (openvim uses vim_db database name)

TODO complete where RO uses it

===Installation===
The OSM binary installation install this package. In case of needed you can install/update it manually by:

apt-get update
apt-get install make git python tox libmysqlclient-dev
git clone https://osm.etsi.org/gerrit/osm/openvim
git -C openvim checkout 005a9dc # this is temporal, because last version contains error at Makefile
make -C openvim lite
# Use this line to check if it is installed and where
python -c 'import lib_osm_openvim; print lib_osm_openvim.__path__[0]'
# Install database of ovim library
OSMLIBOVIM_PATH=`python -c 'import lib_osm_openvim; print lib_osm_openvim.__path__[0]'`
# -U and -P are the admin database user/password. Normally "-U root" without password "executed as root"
${OSMLIBOVIM_PATH}/database_utils/install-db-server.sh -U root [-P passwd] -u mano -p manopw -d mano_vim_db --updatedb

==CLI client==

The RO code contains a python CLI (openmano) that allows friently command execution. This CLI client can run on a separate machine where openmano server is running. "openmano config" indicates where the server is (by default localhost)

==Northbound Interface==

The RO uses a REST API with YAML/JSON content. The primitives are explained [[RO Northbound Interface|here]]

==Running Unit Tests==

===Launching RO===
Openmano can run as systemd service. (Can be installed with '''./scripts/install-openmano-service.sh -f RO''')

Or can be run inside a screen (the prefered method for developpers). The script to easily launch/remove is '''./scripts/service-openmano.sh'''. Execute with -h to see options

Note that the general OSM install script, installs openmano inside a container and code run at folder /opt/openmano (instead of $HOME/openmano)

===Tests===
Many of openmano operations rely on an external VIM. Without external infraestructure it is recomended to use openvim in "test" (fake) mode. Install it in the same machine where openmano is located with [[OpenVIM_installation_(Release_One)#Installation]]

Run ''./test/basictest'' for a initial test. Type -h to see options. For testing using openvim just run <pre> . ./test/basictest.sh --force --screen --init-openvim </pre>

==Creating a new VIM plugin==

Choose a name, eg. XXX

Create a new python module vimconn_XXX.py derived from vimconn.py class. Implement the relevant functions. You have several connectors already created to be used as example as openstack and openvim. Openstack connector uses the python openstack client libraries, meanwhile openvim connector uses direct http requests.

DO NOT change the method names, parameters or parameter content. Openmano use the same methods for all the VIMs and they cannot be changed to accomodate VIM specifics. VIM specifics must be solved inside the connector.

The new module can need specific configuration for the VIM that it is passed as a dictionary in the ''config'' variable at constructor. For example, in the case of openstack, ''config'' variable is used for: enabling/disabling port_security_enable, specifying the name of the physical network used for dataplane, regions, etc. The ''config'' variable is the right place to specify those needed parameters not provided by openmano at the methods that form part of the VIM configuration. See [[Openstack configuration (Release TWO)#Add openstack toOSM]] and [[Configuring AWS for OSM Release TWO#Add AWS to OSM]] for examples

Integration with the main project is automatic, just create a new datacenter of type XXX and this module will be automatically loaded. No other module of the main program need to be updated.

openmano tenant-create osm # if not already done
export OPENMANO_TENANT=osm
openmano datacenter-create <vim-name> <access-URL> --type=XXX --config <specific config>
# if fails the cause can be a librery you need not installed or an error upon importing. Try tho import it in a python client
# example of <specific config> text: "{dataplane_physical_net: physnet_sriov, port_security_enabled: False}"
openmano datacenter-attach <vim-name> --vim-tenant-name=<tenant-to-use-by-openmano> --user=<tenant-user> --password=<tenant-password>

You can test it using the '''test/test_RO.py''' file:
# Create manually an image <image> and a management network <net-mgmt> at your VIM
./test/test_RO.py deploy --help
# e.g. ./test/test_RO.py deploy -n <net-mgmt> -t osm -i <image> -d <vim-name> --timeout=30 --test simple_linux,simple_multi_vnfc,simple_2_vnf
./test/test_RO.py vimconn --help

Developer HowTo for RO Module

2017-10-23T10:07:39Z

Cardosoi: /* Installation */

==Getting Started==
The RO Module is implemented by openmano. The recommended Linux distribution for development is Ubuntu 16.04 LTS Server. PyCharm is a nice and easy to use tool for development and debugging.

The step [[Release_1_Installation]] installs all OSM modules in containers. However, for development, a virtual machine may be more suitable. To install the openmano module you can run a script that installs the required packages, clone the project from ''git clones https://osm.etsi.org/gerrit/osm/RO'' and configures openmano client and other utility scripts at path
wget -O install-openmano.sh "https://osm.etsi.org/gitweb/?p=osm/RO.git;a=blob_plain;f=scripts/install-openmano.sh"
chmod +x install-openmano.sh
sudo ./install-openmano.sh -q --develop #-h for help

See also and follow [[Workflow_with_OSM_tools#Clone_your_project]]

New code features must be incorporated to master:
git checkout master
Generate a ".gitignore", you can use the ''.gitignore-common'' example that skips PyCharm and Eclipse files
cp RO/.gitignore-common RO/.gitignore
#edit to include your local files to ignore
Prepare your git environment to push with a proper user/email, push to gerrit. See and configure:
[[Workflow_with_OSM_tools#Configure_your_Git_environment]]
[[Workflow_with_OSM_tools#Commit_changes_to_your_local_project]]
[[ Workflow_with_OSM_tools#Push_your_contribution_to_Gerrit]]

==Programming Language==
The RO module uses Python2. However Python3 conventions for a possible future migration should be used as far as possible. For example:

{| class="wikitable"
|-
! BAD Python2
! OK Python2 compatible with python3
|-
| "format test string %s number %d" % (st, num)
| "format test string {} number {}".format(st, num)
|-
| print a, b, c
| print(a,b,c)
|-
| except Exception, e
| except Exception as e
|-
| if type(x) == X:
| if isinstance(x,X):
|}

Descriptors can be YAML (prefered because more readable and allow comments) or JSON

==Code Style==
Please follow PEP8 style guide for all the Python code.

===Logging===
Use the appropriate logging levels when logging the messages. An example is shown below:
self.logger.debug("Changing state to %s", next_state)

Logging levels (general and per module) are specified at '''openmanod.cfg'''

Try to use few useful logs, not verbose, that brings useful information. For example, in case of fail getting a server, the complete URL should be provided.

Avoid several logs together

WRONG:
self.looger.debug("Entering in method A")
self.logger.debug("Contacting server"

RIGHT:
self.logger.debug("method A, contacting server %s", url)

When the traceback is needed(call stack that generate the exception) , use the "exc_info=True" parameter
self.logger.error("Exception %s when ...", exception, exc_info=True)

===Exceptions===
Code must be wrote in a way that functions and methods raise an exception when something goes wrong, instead of returning a negative or false value.

Example
WRONG
def get_ip_address():
...
if fail:
return False, "Fail because xxx"
return True, ip

...
result, ip = get_ip_address()
if not result:
return False, "Cannot get ip address..."

RIGHT
def get_ip_address():
...
if fail:
raise customException("Fail because ...")
return ip

...
try:
ip = get_ip_address()
...
except customException as e:
raise customException2(str(e))

==Directory Organization==
The code organized into the following high level directories: 
* '''/''' contains the entry server file ''openmanod'' and client ''openmano'' code 
* '''/osm_ro''' contains the RO server code files 
* '''/test''' contains scripts and code for testing 
* '''/database_utils''' contains scripts for database creation, dumping and migration 
* '''/scripts''' general scripts, as installation, execution, reporting 
* '''/scenarios''' examples and templates of network scnario descriptors 
* '''/vnfs''' examples and templates of VNF descriptors 

==RO Architecture==
[[File:OpenmanoArchitecture.png|400px]]

===RO Server modules===
The RO module contains the following modules
* '''openmanod''' is the main program. It reads the configuration file (openmanod.cfg) and execute the httpserver and wait for the end
* '''httpserver.py''' is a thread that implements the northbound API interface, uses python bottle module. Calls main engine methods to perform the tasks
* '''nfvo.py''' is the main engine, implementing all the methods for the creation, deletion and management of vnfs, scenarios and instances. Operations against a VIM are asynchornous. ACTIONs to be done are stored at database before returning ok to the client
* '''nfvo_db.py''' is the module in charge of database operations. Uses base '''db_base.py'''. Database is managed with MySQLdb python library
* '''openmano_schemas.py''' is a dictionary schemas used to validate API request and response content using jsonschema library
* '''vim_thread.py''' There is a thread per VIM and credentials. It performs basic tasks of creating/deleting VM, networks, flavors, etc. In addition it refreshes the VM and network status. It calls vimconn.py methods
* '''vimconn.py''' is the base class for the VIM plugin. It contains the definition of the methods to be implemented. The inherited '''vimconn_openstack.py''', '''vimconn_openvim.py''', '''vimconn_vmware''' and '''vimconn_aws''' implements the operations against the concrete VIM type. Openstack plugin uses the client-python libraris, meanwhile Openvim plugin uses direct http requests.
* '''console_proxy_thread.py''' is a thread that implements a TCP/IP proxy for the console access to a VIM

===RO Client modules===
Other modules not part of the server are:
* '''openmnao''' is a CLI client
* '''openmanoclient.py''' is a client python library for managing openmano server

===ACTIONS and TASKS===
ACTIONS are a a group of tasks performed against a a concrete instance-scenarios (NS record). The creation and deletion of the instance-scenario itself is an action. As it is asynchonous, NBI returns an "Action_id" that can be used to check the status. For each action, nfvo.py generates individual tasks for the related VIMs. Tasks are both stored at database and sent to the related vim_thread.py. A task has a concrete relation with a VIM, e.g. create/delete a VM, a network, ... look for a flavor, network, ...

===Database content===
TODO

==Database changes==
Database schema can be changed if needed. It is recomended to use a graphical tool (e.g. Heidy) to change database and change it back and copy the SQL commands. Make this steps:

1. openmanod: increment __version__, version_date and database_version
2. database_utils/migrate_mano_db.sh. See the three "TODO"
2a. Increment LAST_DB_VERSION
2b. Annotate a comment to have a track of versions: [ $OPENMANO_VER_NUM -ge 50XX ] && DB_VERSION=XX #0.5.XX => XX
3c. Generate new methods function upgrade_to_XX() and function downgrade_from_XX. Insert here the sql commands. Last sql command over schema_version is quite important to detect the database version.

Test several upgrades/downgrades to version 20 with "migrate_mano_db.sh" and "migrate_mano_db.sh 20"

==Package dependency osm-im==
It is under IM repository. It contains the OSM models. Used models at RO are under <IM>/models/yang/vnfd.yang and <IM>/models/yang/nsd.yang. More modules will be incorporated in the future.

When IM is maked, pyangbind generates python source files imported at RO. They are used to validate and load a yaml/json VNFD and NSD catalog descriptor.

Method pybindJSONDecoder.load_ietf_json from pyangbind is used to parse and load the descriptor. A structure of nested YANG classes are generated and used at <RO>/osm_ro/nfvo.py methods new_vnfd_v3 and new_nsd_v3. RO supports backward compatibility with old format descriptors. NBI (osm_ro/httpserver.py) uses .../v3/... at URL to diferenciate between old and new descriptors. CLI (openmano) detects automatically if the descriptor is in old format or in OSM format to send the request to the server using the old URL or the new "v3" URL

===Installation===
The OSM binary installation install this package. In case of needed you can install/update it manually by:

apt-get update
apt-get install make git python python-pip tox debhelper python-bitarray python-lxml
pip install --upgrade pip
# If not inside a container. Use "sudo -HE" to get root privileges for pip installation
pip install pyangbind stdeb
git clone https://osm.etsi.org/gerrit/osm/IM
make -C IM clean all
dpkg -i IM/deb_dist/python-pyang_*.deb
dpkg -i IM/deb_dist/python-pyangbind_*.deb
dpkg -i IM/deb_dist/python-osm-im_*.deb
# Use this line to check if it is installed and where
python -c 'import osm_im; print osm_im.__path__[0]'

==Package dependency lib-osm-openvim==

It is under openvim repository. RO uses a library for the SDN assist that is in charge of performing the underlay dataplane connectivity using an openflow controller. osm-openvim and lib-osm-openvim are diffent, though they share same pieces of code. lib-osm-openvim uses a different database name (mano_vim_db) so that openvim can be installed in the same virtual machine or container (openvim uses vim_db database name)

TODO complete where RO uses it

===Installation===
The OSM binary installation install this package. In case of needed you can install/update it manually by:

apt-get update
apt-get install make git python tox libmysqlclient-dev
git clone https://osm.etsi.org/gerrit/osm/openvim
git -C openvim checkout 005a9dc # this is temporal, because last version contains error at Makefile
make -C openvim lite
# Use this line to check if it is installed and where
python -c 'import lib_osm_openvim; print lib_osm_openvim.__path__[0]'
# Install database of ovim library
OSMLIBOVIM_PATH=`python -c 'import lib_osm_openvim; print lib_osm_openvim.__path__[0]'`
# -U and -P are the admin database user/password. Normally "-U root" without password "executed as root"
${OSMLIBOVIM_PATH}/database_utils/install-db-server.sh -U root [-P passwd] -u mano -p manopw -d mano_vim_db --updatedb

==CLI client==

The RO code contains a python CLI (openmano) that allows friently command execution. This CLI client can run on a separate machine where openmano server is running. "openmano config" indicates where the server is (by default localhost)

==Northbound Interface==

The RO uses a REST API with YAML/JSON content. The primitives are explained [[RO Northbound Interface|here]]

==Running Unit Tests==

===Launching RO===
Openmano can run as systemd service. (Can be installed with '''./scripts/install-openmano-service.sh -f RO''')

Or can be run inside a screen (the prefered method for developpers). The script to easily launch/remove is '''./scripts/service-openmano.sh'''. Execute with -h to see options

Note that the general OSM install script, installs openmano inside a container and code run at folder /opt/openmano (instead of $HOME/openmano)

===Tests===
Many of openmano operations rely on an external VIM. Without external infraestructure it is recomended to use openvim in "test" (fake) mode. Install it in the same machine where openmano is located with [[OpenVIM_installation_(Release_One)#Installation]]

Run ''./test/basictest'' for a initial test. Type -h to see options. For testing using openvim just run <pre> . ./test/basictest.sh --force --screen --init-openvim </pre>

==Creating a new VIM plugin==

Choose a name, eg. XXX

Create a new python module vimconn_XXX.py derived from vimconn.py class. Implement the relevant functions. You have several connectors already created to be used as example as openstack and openvim. Openstack connector uses the python openstack client libraries, meanwhile openvim connector uses direct http requests.

DO NOT change the method names, parameters or parameter content. Openmano use the same methods for all the VIMs and they cannot be changed to accomodate VIM specifics. VIM specifics must be solved inside the connector.

The new module can need specific configuration for the VIM that it is passed as a dictionary in the ''config'' variable at constructor. For example, in the case of openstack, ''config'' variable is used for: enabling/disabling port_security_enable, specifying the name of the physical network used for dataplane, regions, etc. The ''config'' variable is the right place to specify those needed parameters not provided by openmano at the methods that form part of the VIM configuration. See [[Openstack configuration (Release TWO)#Add openstack toOSM]] and [[Configuring AWS for OSM Release TWO#Add AWS to OSM]] for examples

Integration with the main project is automatic, just create a new datacenter of type XXX and this module will be automatically loaded. No other module of the main program need to be updated.

openmano tenant-create osm # if not already done
export OPENMANO_TENANT=osm
openmano datacenter-create <vim-name> <access-URL> --type=XXX --config <specific config>
# if fails the cause can be a librery you need not installed or an error upon importing. Try tho import it in a python client
# example of <specific config> text: "{dataplane_physical_net: physnet_sriov, port_security_enabled: False}"
openmano datacenter-attach <vim-name> --vim-tenant-name=<tenant-to-use-by-openmano> --user=<tenant-user> --password=<tenant-password>

You can test it using the '''test/test_RO.py''' file:
# Create manually an image <image> and a management network <net-mgmt> at your VIM
./test/test_RO.py deploy --help
# e.g. ./test/test_RO.py deploy -n <net-mgmt> -t osm -i <image> -d <vim-name> --timeout=30 --test simple_linux,simple_multi_vnfc,simple_2_vnf
./test/test_RO.py vimconn --help

Developer HowTo for RO Module

2017-10-23T10:06:54Z

Cardosoi: Correction installation command list (working fine now) /* Installation */

==Getting Started==
The RO Module is implemented by openmano. The recommended Linux distribution for development is Ubuntu 16.04 LTS Server. PyCharm is a nice and easy to use tool for development and debugging.

The step [[Release_1_Installation]] installs all OSM modules in containers. However, for development, a virtual machine may be more suitable. To install the openmano module you can run a script that installs the required packages, clone the project from ''git clones https://osm.etsi.org/gerrit/osm/RO'' and configures openmano client and other utility scripts at path
wget -O install-openmano.sh "https://osm.etsi.org/gitweb/?p=osm/RO.git;a=blob_plain;f=scripts/install-openmano.sh"
chmod +x install-openmano.sh
sudo ./install-openmano.sh -q --develop #-h for help

See also and follow [[Workflow_with_OSM_tools#Clone_your_project]]

New code features must be incorporated to master:
git checkout master
Generate a ".gitignore", you can use the ''.gitignore-common'' example that skips PyCharm and Eclipse files
cp RO/.gitignore-common RO/.gitignore
#edit to include your local files to ignore
Prepare your git environment to push with a proper user/email, push to gerrit. See and configure:
[[Workflow_with_OSM_tools#Configure_your_Git_environment]]
[[Workflow_with_OSM_tools#Commit_changes_to_your_local_project]]
[[ Workflow_with_OSM_tools#Push_your_contribution_to_Gerrit]]

==Programming Language==
The RO module uses Python2. However Python3 conventions for a possible future migration should be used as far as possible. For example:

{| class="wikitable"
|-
! BAD Python2
! OK Python2 compatible with python3
|-
| "format test string %s number %d" % (st, num)
| "format test string {} number {}".format(st, num)
|-
| print a, b, c
| print(a,b,c)
|-
| except Exception, e
| except Exception as e
|-
| if type(x) == X:
| if isinstance(x,X):
|}

Descriptors can be YAML (prefered because more readable and allow comments) or JSON

==Code Style==
Please follow PEP8 style guide for all the Python code.

===Logging===
Use the appropriate logging levels when logging the messages. An example is shown below:
self.logger.debug("Changing state to %s", next_state)

Logging levels (general and per module) are specified at '''openmanod.cfg'''

Try to use few useful logs, not verbose, that brings useful information. For example, in case of fail getting a server, the complete URL should be provided.

Avoid several logs together

WRONG:
self.looger.debug("Entering in method A")
self.logger.debug("Contacting server"

RIGHT:
self.logger.debug("method A, contacting server %s", url)

When the traceback is needed(call stack that generate the exception) , use the "exc_info=True" parameter
self.logger.error("Exception %s when ...", exception, exc_info=True)

===Exceptions===
Code must be wrote in a way that functions and methods raise an exception when something goes wrong, instead of returning a negative or false value.

Example
WRONG
def get_ip_address():
...
if fail:
return False, "Fail because xxx"
return True, ip

...
result, ip = get_ip_address()
if not result:
return False, "Cannot get ip address..."

RIGHT
def get_ip_address():
...
if fail:
raise customException("Fail because ...")
return ip

...
try:
ip = get_ip_address()
...
except customException as e:
raise customException2(str(e))

==Directory Organization==
The code organized into the following high level directories: 
* '''/''' contains the entry server file ''openmanod'' and client ''openmano'' code 
* '''/osm_ro''' contains the RO server code files 
* '''/test''' contains scripts and code for testing 
* '''/database_utils''' contains scripts for database creation, dumping and migration 
* '''/scripts''' general scripts, as installation, execution, reporting 
* '''/scenarios''' examples and templates of network scnario descriptors 
* '''/vnfs''' examples and templates of VNF descriptors 

==RO Architecture==
[[File:OpenmanoArchitecture.png|400px]]

===RO Server modules===
The RO module contains the following modules
* '''openmanod''' is the main program. It reads the configuration file (openmanod.cfg) and execute the httpserver and wait for the end
* '''httpserver.py''' is a thread that implements the northbound API interface, uses python bottle module. Calls main engine methods to perform the tasks
* '''nfvo.py''' is the main engine, implementing all the methods for the creation, deletion and management of vnfs, scenarios and instances. Operations against a VIM are asynchornous. ACTIONs to be done are stored at database before returning ok to the client
* '''nfvo_db.py''' is the module in charge of database operations. Uses base '''db_base.py'''. Database is managed with MySQLdb python library
* '''openmano_schemas.py''' is a dictionary schemas used to validate API request and response content using jsonschema library
* '''vim_thread.py''' There is a thread per VIM and credentials. It performs basic tasks of creating/deleting VM, networks, flavors, etc. In addition it refreshes the VM and network status. It calls vimconn.py methods
* '''vimconn.py''' is the base class for the VIM plugin. It contains the definition of the methods to be implemented. The inherited '''vimconn_openstack.py''', '''vimconn_openvim.py''', '''vimconn_vmware''' and '''vimconn_aws''' implements the operations against the concrete VIM type. Openstack plugin uses the client-python libraris, meanwhile Openvim plugin uses direct http requests.
* '''console_proxy_thread.py''' is a thread that implements a TCP/IP proxy for the console access to a VIM

===RO Client modules===
Other modules not part of the server are:
* '''openmnao''' is a CLI client
* '''openmanoclient.py''' is a client python library for managing openmano server

===ACTIONS and TASKS===
ACTIONS are a a group of tasks performed against a a concrete instance-scenarios (NS record). The creation and deletion of the instance-scenario itself is an action. As it is asynchonous, NBI returns an "Action_id" that can be used to check the status. For each action, nfvo.py generates individual tasks for the related VIMs. Tasks are both stored at database and sent to the related vim_thread.py. A task has a concrete relation with a VIM, e.g. create/delete a VM, a network, ... look for a flavor, network, ...

===Database content===
TODO

==Database changes==
Database schema can be changed if needed. It is recomended to use a graphical tool (e.g. Heidy) to change database and change it back and copy the SQL commands. Make this steps:

1. openmanod: increment __version__, version_date and database_version
2. database_utils/migrate_mano_db.sh. See the three "TODO"
2a. Increment LAST_DB_VERSION
2b. Annotate a comment to have a track of versions: [ $OPENMANO_VER_NUM -ge 50XX ] && DB_VERSION=XX #0.5.XX => XX
3c. Generate new methods function upgrade_to_XX() and function downgrade_from_XX. Insert here the sql commands. Last sql command over schema_version is quite important to detect the database version.

Test several upgrades/downgrades to version 20 with "migrate_mano_db.sh" and "migrate_mano_db.sh 20"

==Package dependency osm-im==
It is under IM repository. It contains the OSM models. Used models at RO are under <IM>/models/yang/vnfd.yang and <IM>/models/yang/nsd.yang. More modules will be incorporated in the future.

When IM is maked, pyangbind generates python source files imported at RO. They are used to validate and load a yaml/json VNFD and NSD catalog descriptor.

Method pybindJSONDecoder.load_ietf_json from pyangbind is used to parse and load the descriptor. A structure of nested YANG classes are generated and used at <RO>/osm_ro/nfvo.py methods new_vnfd_v3 and new_nsd_v3. RO supports backward compatibility with old format descriptors. NBI (osm_ro/httpserver.py) uses .../v3/... at URL to diferenciate between old and new descriptors. CLI (openmano) detects automatically if the descriptor is in old format or in OSM format to send the request to the server using the old URL or the new "v3" URL

===Installation===
The OSM binary installation install this package. In case of needed you can install/update it manually by:

apt-get update
apt-get install make git python python-pip tox debhelper python-bitarray python-lxml
pip install --upgrade pip
# If not inside a container. Use "sudo -HE" to get root privileges for pip installation
pip install pyangbind stdeb
git clone https://osm.etsi.org/gerrit/osm/IM
make -C IM clean all
dpkg -i IM/deb_dist/python-pyang*.deb
dpkg -i IM/deb_dist/python-pyangbind*.deb
dpkg -i IM/deb_dist/python-osm-im*.deb
# Use this line to check if it is installed and where
python -c 'import osm_im; print osm_im.__path__[0]'

==Package dependency lib-osm-openvim==

It is under openvim repository. RO uses a library for the SDN assist that is in charge of performing the underlay dataplane connectivity using an openflow controller. osm-openvim and lib-osm-openvim are diffent, though they share same pieces of code. lib-osm-openvim uses a different database name (mano_vim_db) so that openvim can be installed in the same virtual machine or container (openvim uses vim_db database name)

TODO complete where RO uses it

===Installation===
The OSM binary installation install this package. In case of needed you can install/update it manually by:

apt-get update
apt-get install make git python tox libmysqlclient-dev
git clone https://osm.etsi.org/gerrit/osm/openvim
git -C openvim checkout 005a9dc # this is temporal, because last version contains error at Makefile
make -C openvim lite
# Use this line to check if it is installed and where
python -c 'import lib_osm_openvim; print lib_osm_openvim.__path__[0]'
# Install database of ovim library
OSMLIBOVIM_PATH=`python -c 'import lib_osm_openvim; print lib_osm_openvim.__path__[0]'`
# -U and -P are the admin database user/password. Normally "-U root" without password "executed as root"
${OSMLIBOVIM_PATH}/database_utils/install-db-server.sh -U root [-P passwd] -u mano -p manopw -d mano_vim_db --updatedb

==CLI client==

The RO code contains a python CLI (openmano) that allows friently command execution. This CLI client can run on a separate machine where openmano server is running. "openmano config" indicates where the server is (by default localhost)

==Northbound Interface==

The RO uses a REST API with YAML/JSON content. The primitives are explained [[RO Northbound Interface|here]]

==Running Unit Tests==

===Launching RO===
Openmano can run as systemd service. (Can be installed with '''./scripts/install-openmano-service.sh -f RO''')

Or can be run inside a screen (the prefered method for developpers). The script to easily launch/remove is '''./scripts/service-openmano.sh'''. Execute with -h to see options

Note that the general OSM install script, installs openmano inside a container and code run at folder /opt/openmano (instead of $HOME/openmano)

===Tests===
Many of openmano operations rely on an external VIM. Without external infraestructure it is recomended to use openvim in "test" (fake) mode. Install it in the same machine where openmano is located with [[OpenVIM_installation_(Release_One)#Installation]]

Run ''./test/basictest'' for a initial test. Type -h to see options. For testing using openvim just run <pre> . ./test/basictest.sh --force --screen --init-openvim </pre>

==Creating a new VIM plugin==

Choose a name, eg. XXX

Create a new python module vimconn_XXX.py derived from vimconn.py class. Implement the relevant functions. You have several connectors already created to be used as example as openstack and openvim. Openstack connector uses the python openstack client libraries, meanwhile openvim connector uses direct http requests.

DO NOT change the method names, parameters or parameter content. Openmano use the same methods for all the VIMs and they cannot be changed to accomodate VIM specifics. VIM specifics must be solved inside the connector.

The new module can need specific configuration for the VIM that it is passed as a dictionary in the ''config'' variable at constructor. For example, in the case of openstack, ''config'' variable is used for: enabling/disabling port_security_enable, specifying the name of the physical network used for dataplane, regions, etc. The ''config'' variable is the right place to specify those needed parameters not provided by openmano at the methods that form part of the VIM configuration. See [[Openstack configuration (Release TWO)#Add openstack toOSM]] and [[Configuring AWS for OSM Release TWO#Add AWS to OSM]] for examples

Integration with the main project is automatic, just create a new datacenter of type XXX and this module will be automatically loaded. No other module of the main program need to be updated.

openmano tenant-create osm # if not already done
export OPENMANO_TENANT=osm
openmano datacenter-create <vim-name> <access-URL> --type=XXX --config <specific config>
# if fails the cause can be a librery you need not installed or an error upon importing. Try tho import it in a python client
# example of <specific config> text: "{dataplane_physical_net: physnet_sriov, port_security_enabled: False}"
openmano datacenter-attach <vim-name> --vim-tenant-name=<tenant-to-use-by-openmano> --user=<tenant-user> --password=<tenant-password>

You can test it using the '''test/test_RO.py''' file:
# Create manually an image <image> and a management network <net-mgmt> at your VIM
./test/test_RO.py deploy --help
# e.g. ./test/test_RO.py deploy -n <net-mgmt> -t osm -i <image> -d <vim-name> --timeout=30 --test simple_linux,simple_multi_vnfc,simple_2_vnf
./test/test_RO.py vimconn --help

Developer HowTo for RO Module

2017-10-23T09:44:36Z

Cardosoi: E.g. Proxy might be set in environment, so keep environment on sude/* Installation */

==Getting Started==
The RO Module is implemented by openmano. The recommended Linux distribution for development is Ubuntu 16.04 LTS Server. PyCharm is a nice and easy to use tool for development and debugging.

The step [[Release_1_Installation]] installs all OSM modules in containers. However, for development, a virtual machine may be more suitable. To install the openmano module you can run a script that installs the required packages, clone the project from ''git clones https://osm.etsi.org/gerrit/osm/RO'' and configures openmano client and other utility scripts at path
wget -O install-openmano.sh "https://osm.etsi.org/gitweb/?p=osm/RO.git;a=blob_plain;f=scripts/install-openmano.sh"
chmod +x install-openmano.sh
sudo ./install-openmano.sh -q --develop #-h for help

See also and follow [[Workflow_with_OSM_tools#Clone_your_project]]

New code features must be incorporated to master:
git checkout master
Generate a ".gitignore", you can use the ''.gitignore-common'' example that skips PyCharm and Eclipse files
cp RO/.gitignore-common RO/.gitignore
#edit to include your local files to ignore
Prepare your git environment to push with a proper user/email, push to gerrit. See and configure:
[[Workflow_with_OSM_tools#Configure_your_Git_environment]]
[[Workflow_with_OSM_tools#Commit_changes_to_your_local_project]]
[[ Workflow_with_OSM_tools#Push_your_contribution_to_Gerrit]]

==Programming Language==
The RO module uses Python2. However Python3 conventions for a possible future migration should be used as far as possible. For example:

{| class="wikitable"
|-
! BAD Python2
! OK Python2 compatible with python3
|-
| "format test string %s number %d" % (st, num)
| "format test string {} number {}".format(st, num)
|-
| print a, b, c
| print(a,b,c)
|-
| except Exception, e
| except Exception as e
|-
| if type(x) == X:
| if isinstance(x,X):
|}

Descriptors can be YAML (prefered because more readable and allow comments) or JSON

==Code Style==
Please follow PEP8 style guide for all the Python code.

===Logging===
Use the appropriate logging levels when logging the messages. An example is shown below:
self.logger.debug("Changing state to %s", next_state)

Logging levels (general and per module) are specified at '''openmanod.cfg'''

Try to use few useful logs, not verbose, that brings useful information. For example, in case of fail getting a server, the complete URL should be provided.

Avoid several logs together

WRONG:
self.looger.debug("Entering in method A")
self.logger.debug("Contacting server"

RIGHT:
self.logger.debug("method A, contacting server %s", url)

When the traceback is needed(call stack that generate the exception) , use the "exc_info=True" parameter
self.logger.error("Exception %s when ...", exception, exc_info=True)

===Exceptions===
Code must be wrote in a way that functions and methods raise an exception when something goes wrong, instead of returning a negative or false value.

Example
WRONG
def get_ip_address():
...
if fail:
return False, "Fail because xxx"
return True, ip

...
result, ip = get_ip_address()
if not result:
return False, "Cannot get ip address..."

RIGHT
def get_ip_address():
...
if fail:
raise customException("Fail because ...")
return ip

...
try:
ip = get_ip_address()
...
except customException as e:
raise customException2(str(e))

==Directory Organization==
The code organized into the following high level directories: 
* '''/''' contains the entry server file ''openmanod'' and client ''openmano'' code 
* '''/osm_ro''' contains the RO server code files 
* '''/test''' contains scripts and code for testing 
* '''/database_utils''' contains scripts for database creation, dumping and migration 
* '''/scripts''' general scripts, as installation, execution, reporting 
* '''/scenarios''' examples and templates of network scnario descriptors 
* '''/vnfs''' examples and templates of VNF descriptors 

==RO Architecture==
[[File:OpenmanoArchitecture.png|400px]]

===RO Server modules===
The RO module contains the following modules
* '''openmanod''' is the main program. It reads the configuration file (openmanod.cfg) and execute the httpserver and wait for the end
* '''httpserver.py''' is a thread that implements the northbound API interface, uses python bottle module. Calls main engine methods to perform the tasks
* '''nfvo.py''' is the main engine, implementing all the methods for the creation, deletion and management of vnfs, scenarios and instances. Operations against a VIM are asynchornous. ACTIONs to be done are stored at database before returning ok to the client
* '''nfvo_db.py''' is the module in charge of database operations. Uses base '''db_base.py'''. Database is managed with MySQLdb python library
* '''openmano_schemas.py''' is a dictionary schemas used to validate API request and response content using jsonschema library
* '''vim_thread.py''' There is a thread per VIM and credentials. It performs basic tasks of creating/deleting VM, networks, flavors, etc. In addition it refreshes the VM and network status. It calls vimconn.py methods
* '''vimconn.py''' is the base class for the VIM plugin. It contains the definition of the methods to be implemented. The inherited '''vimconn_openstack.py''', '''vimconn_openvim.py''', '''vimconn_vmware''' and '''vimconn_aws''' implements the operations against the concrete VIM type. Openstack plugin uses the client-python libraris, meanwhile Openvim plugin uses direct http requests.
* '''console_proxy_thread.py''' is a thread that implements a TCP/IP proxy for the console access to a VIM

===RO Client modules===
Other modules not part of the server are:
* '''openmnao''' is a CLI client
* '''openmanoclient.py''' is a client python library for managing openmano server

===ACTIONS and TASKS===
ACTIONS are a a group of tasks performed against a a concrete instance-scenarios (NS record). The creation and deletion of the instance-scenario itself is an action. As it is asynchonous, NBI returns an "Action_id" that can be used to check the status. For each action, nfvo.py generates individual tasks for the related VIMs. Tasks are both stored at database and sent to the related vim_thread.py. A task has a concrete relation with a VIM, e.g. create/delete a VM, a network, ... look for a flavor, network, ...

===Database content===
TODO

==Database changes==
Database schema can be changed if needed. It is recomended to use a graphical tool (e.g. Heidy) to change database and change it back and copy the SQL commands. Make this steps:

1. openmanod: increment __version__, version_date and database_version
2. database_utils/migrate_mano_db.sh. See the three "TODO"
2a. Increment LAST_DB_VERSION
2b. Annotate a comment to have a track of versions: [ $OPENMANO_VER_NUM -ge 50XX ] && DB_VERSION=XX #0.5.XX => XX
3c. Generate new methods function upgrade_to_XX() and function downgrade_from_XX. Insert here the sql commands. Last sql command over schema_version is quite important to detect the database version.

Test several upgrades/downgrades to version 20 with "migrate_mano_db.sh" and "migrate_mano_db.sh 20"

==Package dependency osm-im==
It is under IM repository. It contains the OSM models. Used models at RO are under <IM>/models/yang/vnfd.yang and <IM>/models/yang/nsd.yang. More modules will be incorporated in the future.

When IM is maked, pyangbind generates python source files imported at RO. They are used to validate and load a yaml/json VNFD and NSD catalog descriptor.

Method pybindJSONDecoder.load_ietf_json from pyangbind is used to parse and load the descriptor. A structure of nested YANG classes are generated and used at <RO>/osm_ro/nfvo.py methods new_vnfd_v3 and new_nsd_v3. RO supports backward compatibility with old format descriptors. NBI (osm_ro/httpserver.py) uses .../v3/... at URL to diferenciate between old and new descriptors. CLI (openmano) detects automatically if the descriptor is in old format or in OSM format to send the request to the server using the old URL or the new "v3" URL

===Installation===
The OSM binary installation install this package. In case of needed you can install/update it manually by:

apt-get update
apt-get install make git python python-pip tox debhelper python-bitarray python-pyangbind
pip install --upgrade pip
# If not inside a container. Use "sudo -HE" to get root privileges for pip installation
pip install pyangbind stdeb
git clone https://osm.etsi.org/gerrit/osm/IM
make -C IM clean all
dpkg -i IM/deb_dist/python-osm-im*.deb
# Use this line to check if it is installed and where
python -c 'import osm_im; print osm_im.__path__[0]'

==Package dependency lib-osm-openvim==

It is under openvim repository. RO uses a library for the SDN assist that is in charge of performing the underlay dataplane connectivity using an openflow controller. osm-openvim and lib-osm-openvim are diffent, though they share same pieces of code. lib-osm-openvim uses a different database name (mano_vim_db) so that openvim can be installed in the same virtual machine or container (openvim uses vim_db database name)

TODO complete where RO uses it

===Installation===
The OSM binary installation install this package. In case of needed you can install/update it manually by:

apt-get update
apt-get install make git python tox libmysqlclient-dev
git clone https://osm.etsi.org/gerrit/osm/openvim
git -C openvim checkout 005a9dc # this is temporal, because last version contains error at Makefile
make -C openvim lite
# Use this line to check if it is installed and where
python -c 'import lib_osm_openvim; print lib_osm_openvim.__path__[0]'
# Install database of ovim library
OSMLIBOVIM_PATH=`python -c 'import lib_osm_openvim; print lib_osm_openvim.__path__[0]'`
# -U and -P are the admin database user/password. Normally "-U root" without password "executed as root"
${OSMLIBOVIM_PATH}/database_utils/install-db-server.sh -U root [-P passwd] -u mano -p manopw -d mano_vim_db --updatedb

==CLI client==

The RO code contains a python CLI (openmano) that allows friently command execution. This CLI client can run on a separate machine where openmano server is running. "openmano config" indicates where the server is (by default localhost)

==Northbound Interface==

The RO uses a REST API with YAML/JSON content. The primitives are explained [[RO Northbound Interface|here]]

==Running Unit Tests==

===Launching RO===
Openmano can run as systemd service. (Can be installed with '''./scripts/install-openmano-service.sh -f RO''')

Or can be run inside a screen (the prefered method for developpers). The script to easily launch/remove is '''./scripts/service-openmano.sh'''. Execute with -h to see options

Note that the general OSM install script, installs openmano inside a container and code run at folder /opt/openmano (instead of $HOME/openmano)

===Tests===
Many of openmano operations rely on an external VIM. Without external infraestructure it is recomended to use openvim in "test" (fake) mode. Install it in the same machine where openmano is located with [[OpenVIM_installation_(Release_One)#Installation]]

Run ''./test/basictest'' for a initial test. Type -h to see options. For testing using openvim just run <pre> . ./test/basictest.sh --force --screen --init-openvim </pre>

==Creating a new VIM plugin==

Choose a name, eg. XXX

Create a new python module vimconn_XXX.py derived from vimconn.py class. Implement the relevant functions. You have several connectors already created to be used as example as openstack and openvim. Openstack connector uses the python openstack client libraries, meanwhile openvim connector uses direct http requests.

DO NOT change the method names, parameters or parameter content. Openmano use the same methods for all the VIMs and they cannot be changed to accomodate VIM specifics. VIM specifics must be solved inside the connector.

The new module can need specific configuration for the VIM that it is passed as a dictionary in the ''config'' variable at constructor. For example, in the case of openstack, ''config'' variable is used for: enabling/disabling port_security_enable, specifying the name of the physical network used for dataplane, regions, etc. The ''config'' variable is the right place to specify those needed parameters not provided by openmano at the methods that form part of the VIM configuration. See [[Openstack configuration (Release TWO)#Add openstack toOSM]] and [[Configuring AWS for OSM Release TWO#Add AWS to OSM]] for examples

Integration with the main project is automatic, just create a new datacenter of type XXX and this module will be automatically loaded. No other module of the main program need to be updated.

openmano tenant-create osm # if not already done
export OPENMANO_TENANT=osm
openmano datacenter-create <vim-name> <access-URL> --type=XXX --config <specific config>
# if fails the cause can be a librery you need not installed or an error upon importing. Try tho import it in a python client
# example of <specific config> text: "{dataplane_physical_net: physnet_sriov, port_security_enabled: False}"
openmano datacenter-attach <vim-name> --vim-tenant-name=<tenant-to-use-by-openmano> --user=<tenant-user> --password=<tenant-password>

You can test it using the '''test/test_RO.py''' file:
# Create manually an image <image> and a management network <net-mgmt> at your VIM
./test/test_RO.py deploy --help
# e.g. ./test/test_RO.py deploy -n <net-mgmt> -t osm -i <image> -d <vim-name> --timeout=30 --test simple_linux,simple_multi_vnfc,simple_2_vnf
./test/test_RO.py vimconn --help

Developer HowTo for RO Module

2017-10-23T09:41:55Z

Cardosoi: /* Getting Started */

==Getting Started==
The RO Module is implemented by openmano. The recommended Linux distribution for development is Ubuntu 16.04 LTS Server. PyCharm is a nice and easy to use tool for development and debugging.

The step [[Release_1_Installation]] installs all OSM modules in containers. However, for development, a virtual machine may be more suitable. To install the openmano module you can run a script that installs the required packages, clone the project from ''git clones https://osm.etsi.org/gerrit/osm/RO'' and configures openmano client and other utility scripts at path
wget -O install-openmano.sh "https://osm.etsi.org/gitweb/?p=osm/RO.git;a=blob_plain;f=scripts/install-openmano.sh"
chmod +x install-openmano.sh
sudo ./install-openmano.sh -q --develop #-h for help

See also and follow [[Workflow_with_OSM_tools#Clone_your_project]]

New code features must be incorporated to master:
git checkout master
Generate a ".gitignore", you can use the ''.gitignore-common'' example that skips PyCharm and Eclipse files
cp RO/.gitignore-common RO/.gitignore
#edit to include your local files to ignore
Prepare your git environment to push with a proper user/email, push to gerrit. See and configure:
[[Workflow_with_OSM_tools#Configure_your_Git_environment]]
[[Workflow_with_OSM_tools#Commit_changes_to_your_local_project]]
[[ Workflow_with_OSM_tools#Push_your_contribution_to_Gerrit]]

==Programming Language==
The RO module uses Python2. However Python3 conventions for a possible future migration should be used as far as possible. For example:

{| class="wikitable"
|-
! BAD Python2
! OK Python2 compatible with python3
|-
| "format test string %s number %d" % (st, num)
| "format test string {} number {}".format(st, num)
|-
| print a, b, c
| print(a,b,c)
|-
| except Exception, e
| except Exception as e
|-
| if type(x) == X:
| if isinstance(x,X):
|}

Descriptors can be YAML (prefered because more readable and allow comments) or JSON

==Code Style==
Please follow PEP8 style guide for all the Python code.

===Logging===
Use the appropriate logging levels when logging the messages. An example is shown below:
self.logger.debug("Changing state to %s", next_state)

Logging levels (general and per module) are specified at '''openmanod.cfg'''

Try to use few useful logs, not verbose, that brings useful information. For example, in case of fail getting a server, the complete URL should be provided.

Avoid several logs together

WRONG:
self.looger.debug("Entering in method A")
self.logger.debug("Contacting server"

RIGHT:
self.logger.debug("method A, contacting server %s", url)

When the traceback is needed(call stack that generate the exception) , use the "exc_info=True" parameter
self.logger.error("Exception %s when ...", exception, exc_info=True)

===Exceptions===
Code must be wrote in a way that functions and methods raise an exception when something goes wrong, instead of returning a negative or false value.

Example
WRONG
def get_ip_address():
...
if fail:
return False, "Fail because xxx"
return True, ip

...
result, ip = get_ip_address()
if not result:
return False, "Cannot get ip address..."

RIGHT
def get_ip_address():
...
if fail:
raise customException("Fail because ...")
return ip

...
try:
ip = get_ip_address()
...
except customException as e:
raise customException2(str(e))

==Directory Organization==
The code organized into the following high level directories: 
* '''/''' contains the entry server file ''openmanod'' and client ''openmano'' code 
* '''/osm_ro''' contains the RO server code files 
* '''/test''' contains scripts and code for testing 
* '''/database_utils''' contains scripts for database creation, dumping and migration 
* '''/scripts''' general scripts, as installation, execution, reporting 
* '''/scenarios''' examples and templates of network scnario descriptors 
* '''/vnfs''' examples and templates of VNF descriptors 

==RO Architecture==
[[File:OpenmanoArchitecture.png|400px]]

===RO Server modules===
The RO module contains the following modules
* '''openmanod''' is the main program. It reads the configuration file (openmanod.cfg) and execute the httpserver and wait for the end
* '''httpserver.py''' is a thread that implements the northbound API interface, uses python bottle module. Calls main engine methods to perform the tasks
* '''nfvo.py''' is the main engine, implementing all the methods for the creation, deletion and management of vnfs, scenarios and instances. Operations against a VIM are asynchornous. ACTIONs to be done are stored at database before returning ok to the client
* '''nfvo_db.py''' is the module in charge of database operations. Uses base '''db_base.py'''. Database is managed with MySQLdb python library
* '''openmano_schemas.py''' is a dictionary schemas used to validate API request and response content using jsonschema library
* '''vim_thread.py''' There is a thread per VIM and credentials. It performs basic tasks of creating/deleting VM, networks, flavors, etc. In addition it refreshes the VM and network status. It calls vimconn.py methods
* '''vimconn.py''' is the base class for the VIM plugin. It contains the definition of the methods to be implemented. The inherited '''vimconn_openstack.py''', '''vimconn_openvim.py''', '''vimconn_vmware''' and '''vimconn_aws''' implements the operations against the concrete VIM type. Openstack plugin uses the client-python libraris, meanwhile Openvim plugin uses direct http requests.
* '''console_proxy_thread.py''' is a thread that implements a TCP/IP proxy for the console access to a VIM

===RO Client modules===
Other modules not part of the server are:
* '''openmnao''' is a CLI client
* '''openmanoclient.py''' is a client python library for managing openmano server

===ACTIONS and TASKS===
ACTIONS are a a group of tasks performed against a a concrete instance-scenarios (NS record). The creation and deletion of the instance-scenario itself is an action. As it is asynchonous, NBI returns an "Action_id" that can be used to check the status. For each action, nfvo.py generates individual tasks for the related VIMs. Tasks are both stored at database and sent to the related vim_thread.py. A task has a concrete relation with a VIM, e.g. create/delete a VM, a network, ... look for a flavor, network, ...

===Database content===
TODO

==Database changes==
Database schema can be changed if needed. It is recomended to use a graphical tool (e.g. Heidy) to change database and change it back and copy the SQL commands. Make this steps:

1. openmanod: increment __version__, version_date and database_version
2. database_utils/migrate_mano_db.sh. See the three "TODO"
2a. Increment LAST_DB_VERSION
2b. Annotate a comment to have a track of versions: [ $OPENMANO_VER_NUM -ge 50XX ] && DB_VERSION=XX #0.5.XX => XX
3c. Generate new methods function upgrade_to_XX() and function downgrade_from_XX. Insert here the sql commands. Last sql command over schema_version is quite important to detect the database version.

Test several upgrades/downgrades to version 20 with "migrate_mano_db.sh" and "migrate_mano_db.sh 20"

==Package dependency osm-im==
It is under IM repository. It contains the OSM models. Used models at RO are under <IM>/models/yang/vnfd.yang and <IM>/models/yang/nsd.yang. More modules will be incorporated in the future.

When IM is maked, pyangbind generates python source files imported at RO. They are used to validate and load a yaml/json VNFD and NSD catalog descriptor.

Method pybindJSONDecoder.load_ietf_json from pyangbind is used to parse and load the descriptor. A structure of nested YANG classes are generated and used at <RO>/osm_ro/nfvo.py methods new_vnfd_v3 and new_nsd_v3. RO supports backward compatibility with old format descriptors. NBI (osm_ro/httpserver.py) uses .../v3/... at URL to diferenciate between old and new descriptors. CLI (openmano) detects automatically if the descriptor is in old format or in OSM format to send the request to the server using the old URL or the new "v3" URL

===Installation===
The OSM binary installation install this package. In case of needed you can install/update it manually by:

apt-get update
apt-get install make git python python-pip tox debhelper python-bitarray python-pyangbind
pip install --upgrade pip
# If not inside a container. Use "sudo -H" to get root privileges for pip installation
pip install pyangbind stdeb
git clone https://osm.etsi.org/gerrit/osm/IM
make -C IM clean all
dpkg -i IM/deb_dist/python-osm-im*.deb
# Use this line to check if it is installed and where
python -c 'import osm_im; print osm_im.__path__[0]'

==Package dependency lib-osm-openvim==

It is under openvim repository. RO uses a library for the SDN assist that is in charge of performing the underlay dataplane connectivity using an openflow controller. osm-openvim and lib-osm-openvim are diffent, though they share same pieces of code. lib-osm-openvim uses a different database name (mano_vim_db) so that openvim can be installed in the same virtual machine or container (openvim uses vim_db database name)

TODO complete where RO uses it

===Installation===
The OSM binary installation install this package. In case of needed you can install/update it manually by:

apt-get update
apt-get install make git python tox libmysqlclient-dev
git clone https://osm.etsi.org/gerrit/osm/openvim
git -C openvim checkout 005a9dc # this is temporal, because last version contains error at Makefile
make -C openvim lite
# Use this line to check if it is installed and where
python -c 'import lib_osm_openvim; print lib_osm_openvim.__path__[0]'
# Install database of ovim library
OSMLIBOVIM_PATH=`python -c 'import lib_osm_openvim; print lib_osm_openvim.__path__[0]'`
# -U and -P are the admin database user/password. Normally "-U root" without password "executed as root"
${OSMLIBOVIM_PATH}/database_utils/install-db-server.sh -U root [-P passwd] -u mano -p manopw -d mano_vim_db --updatedb

==CLI client==

The RO code contains a python CLI (openmano) that allows friently command execution. This CLI client can run on a separate machine where openmano server is running. "openmano config" indicates where the server is (by default localhost)

==Northbound Interface==

The RO uses a REST API with YAML/JSON content. The primitives are explained [[RO Northbound Interface|here]]

==Running Unit Tests==

===Launching RO===
Openmano can run as systemd service. (Can be installed with '''./scripts/install-openmano-service.sh -f RO''')

Or can be run inside a screen (the prefered method for developpers). The script to easily launch/remove is '''./scripts/service-openmano.sh'''. Execute with -h to see options

Note that the general OSM install script, installs openmano inside a container and code run at folder /opt/openmano (instead of $HOME/openmano)

===Tests===
Many of openmano operations rely on an external VIM. Without external infraestructure it is recomended to use openvim in "test" (fake) mode. Install it in the same machine where openmano is located with [[OpenVIM_installation_(Release_One)#Installation]]

Run ''./test/basictest'' for a initial test. Type -h to see options. For testing using openvim just run <pre> . ./test/basictest.sh --force --screen --init-openvim </pre>

==Creating a new VIM plugin==

Choose a name, eg. XXX

Create a new python module vimconn_XXX.py derived from vimconn.py class. Implement the relevant functions. You have several connectors already created to be used as example as openstack and openvim. Openstack connector uses the python openstack client libraries, meanwhile openvim connector uses direct http requests.

DO NOT change the method names, parameters or parameter content. Openmano use the same methods for all the VIMs and they cannot be changed to accomodate VIM specifics. VIM specifics must be solved inside the connector.

The new module can need specific configuration for the VIM that it is passed as a dictionary in the ''config'' variable at constructor. For example, in the case of openstack, ''config'' variable is used for: enabling/disabling port_security_enable, specifying the name of the physical network used for dataplane, regions, etc. The ''config'' variable is the right place to specify those needed parameters not provided by openmano at the methods that form part of the VIM configuration. See [[Openstack configuration (Release TWO)#Add openstack toOSM]] and [[Configuring AWS for OSM Release TWO#Add AWS to OSM]] for examples

Integration with the main project is automatic, just create a new datacenter of type XXX and this module will be automatically loaded. No other module of the main program need to be updated.

openmano tenant-create osm # if not already done
export OPENMANO_TENANT=osm
openmano datacenter-create <vim-name> <access-URL> --type=XXX --config <specific config>
# if fails the cause can be a librery you need not installed or an error upon importing. Try tho import it in a python client
# example of <specific config> text: "{dataplane_physical_net: physnet_sriov, port_security_enabled: False}"
openmano datacenter-attach <vim-name> --vim-tenant-name=<tenant-to-use-by-openmano> --user=<tenant-user> --password=<tenant-password>

You can test it using the '''test/test_RO.py''' file:
# Create manually an image <image> and a management network <net-mgmt> at your VIM
./test/test_RO.py deploy --help
# e.g. ./test/test_RO.py deploy -n <net-mgmt> -t osm -i <image> -d <vim-name> --timeout=30 --test simple_linux,simple_multi_vnfc,simple_2_vnf
./test/test_RO.py vimconn --help