Developer HowTo for RO Module

From OSM Public Wiki
Jump to: navigation, search

THIS PAGE IS DEPRECATED. OSM Developer Guide has been moved to a new location: https://osm.etsi.org/docs/developer-guide/

---

Getting Started

The RO Module of OSM is implemented by openmano. The recommended Linux distribution for development is Ubuntu 16.04 LTS Server. PyCharm is a nice and easy to use tool for development and debugging.

The step Release_1_Installation installs all OSM modules in containers. However, for development, a virtual machine may be more suitable. To install the openmano module you can run a script that installs the required packages, clone the project from git clones https://osm.etsi.org/gerrit/osm/RO and configures openmano client and other utility scripts at path 

wget -O install-openmano.sh "https://osm.etsi.org/gitweb/?p=osm/RO.git;a=blob_plain;f=scripts/install-openmano.sh"
chmod +x install-openmano.sh
sudo ./install-openmano.sh -q --develop   #-h for help

See also and follow Workflow_with_OSM_tools#Clone_your_project

New code features must be incorporated to master: 

git checkout master

Generate a ".gitignore", you can use the .gitignore-common example that skips PyCharm and Eclipse files 

cp RO/.gitignore-common RO/.gitignore
#edit to include your local files to ignore

Prepare your git environment to push with a proper user/email, push to gerrit. See and configure:

Workflow_with_OSM_tools#Configure_your_Git_environment

Workflow_with_OSM_tools#Commit_changes_to_your_local_project

Workflow_with_OSM_tools#Push_your_contribution_to_Gerrit

Programming Language

The RO module uses Python2. However Python3 conventions for a future migration must be used as far as possible. For example:

BAD Python2 OK Python2 compatible with python3
"format test string %s number %d" % (st, num) "format test string {} number {}".format(st, num)
print a, b, c print(a,b,c)
except Exception, e except Exception as e
if type(x) == X: if isinstance(x,X):

In addition, DO NOT USE libraries not compatible with python3

Descriptors can be YAML (prefered because more readable and allow comments) or JSON

Code Style

Please follow PEP8 style guide for all the Python code. Lines up to 120 length

Logging

Use the appropriate logging levels when logging the messages. An example is shown below: 

   self.logger.debug("Changing state to %s", next_state)

Logging levels (general and per module) are specified at openmanod.cfg

Try to use few useful logs, not verbose, that brings useful information. For example, in case of fail getting a server, the complete URL should be provided.

Avoid several logs together 

 WRONG:
 self.looger.debug("Entering in method A")
 self.logger.debug("Contacting server"

 RIGHT:
 self.logger.debug("method A, contacting server %s", url)


When the traceback is needed(call stack that generate the exception) , use the "exc_info=True" parameter 

 self.logger.error("Exception %s when ...", exception, exc_info=True)

Exceptions

Code must be wrote in a way that functions and methods raise an exception when something goes wrong, instead of returning a negative or false value.

Example 

 WRONG
   def get_ip_address():
       ...
       if fail:
           return False, "Fail because xxx"
       return True, ip
   
   ...
   result, ip = get_ip_address()
   if not result:
       return False, "Cannot get ip address..."



 RIGHT
   def get_ip_address():
       ...
       if fail:
           raise customException("Fail because ...")
       return ip
   
   ...
   try:
       ip = get_ip_address()
       ...
   except customException as e:
       raise customException2(str(e))

Directory Organization

The code organized into the following high level directories:

  • / contains the entry server file openmanod and client openmano code
  • /osm_ro contains the RO server code files
  • /test contains scripts and code for testing
  • /database_utils contains scripts for database creation, dumping and migration
  • /scripts general scripts, as installation, execution, reporting
  • /scenarios examples and templates of network scnario descriptors
  • /vnfs examples and templates of VNF descriptors

RO Architecture

OpenmanoArchitecture.png

RO Server modules

The RO module contains the following modules

  • openmanod is the main program. It reads the configuration file (openmanod.cfg) and execute the httpserver and wait for the end
  • httpserver.py is a thread that implements the northbound API interface, uses python bottle module. Calls main engine methods to perform the tasks
  • nfvo.py is the main engine, implementing all the methods for the creation, deletion and management of vnfs, scenarios and instances. Operations against a VIM are asynchornous. ACTIONs to be done are stored at database before returning ok to the client
  • nfvo_db.py is the module in charge of database operations. Uses base db_base.py. Database is managed with MySQLdb python library
  • openmano_schemas.py is a dictionary schemas used to validate API request and response content using jsonschema library
  • vim_thread.py There is a thread per VIM and credentials. It performs basic tasks of creating/deleting VM, networks, flavors, etc. In addition it refreshes the VM and network status. It calls vimconn.py methods
  • vimconn.py is the base class for the VIM plugin. It contains the definition of the methods to be implemented. The inherited vimconn_openstack.py, vimconn_openvim.py, vimconn_vmware and vimconn_aws implements the operations against the concrete VIM type. OpenStack plugin uses the python-*client libraries, meanwhile Openvim plugin uses direct http requests.
  • console_proxy_thread.py is a thread that implements a TCP/IP proxy for the console access to a VIM

RO Client modules

Other modules not part of the server are:

  • openmano is a CLI client
  • openmanoclient.py is a client python library for managing openmano server

ACTIONS and TASKS

ACTIONS are a a group of tasks performed against a a concrete instance-scenarios (NS record). The creation and deletion of the instance-scenario itself is an action. As it is asynchonous, NBI returns an "Action_id" that can be used to check the status. For each action, nfvo.py generates individual tasks for the related VIMs. Tasks are both stored at database and sent to the related vim_thread.py. A task has a concrete relation with a VIM, e.g. create/delete a VM, a network, ... look for a flavor, network, ...

Database content

TODO

Database changes

Database schema can be changed if needed. It is recomended to use a graphical tool (e.g. Heidy) to change database and change it back and copy the SQL commands. Make these steps:

1. openmanod: increment __version__, version_date and database_version

2. database_utils/migrate_mano_db.sh. See the three "TODO"

2a. Increment LAST_DB_VERSION

2b. Annotate a comment to have a track of versions: [ $OPENMANO_VER_NUM -ge 50XX ] && DB_VERSION=XX #0.5.XX => XX

2c. Generate new methods; function upgrade_to_XX() and function downgrade_from_XX. Insert here the sql commands. Last sql command over schema_version is quite important to detect the database version.

Test several upgrades/downgrades to version 20 with "migrate_mano_db.sh" and "migrate_mano_db.sh 20"

Package dependency osm-im

It is under IM repository. It contains the OSM models. Used models at RO are under <IM>/models/yang/vnfd.yang and <IM>/models/yang/nsd.yang. More modules will be incorporated in the future.

When IM is maked, pyangbind generates python source files imported at RO. They are used to validate and load a yaml/json VNFD and NSD catalog descriptor.

Method pybindJSONDecoder.load_ietf_json from pyangbind is used to parse and load the descriptor. A structure of nested YANG classes are generated and used at <RO>/osm_ro/nfvo.py methods new_vnfd_v3 and new_nsd_v3. RO supports backward compatibility with old format descriptors. NBI (osm_ro/httpserver.py) uses .../v3/... at URL to diferenciate between old and new descriptors. CLI (openmano) detects automatically if the descriptor is in old format or in OSM format to send the request to the server using the old URL or the new "v3" URL

Installation

The OSM binary installation install this package. In case of needed you can install/update it manually by: 

 apt-get update
 apt-get install make git python python-pip tox debhelper python-bitarray python-lxml
 pip install --upgrade pip
 # If not inside a container. Use "sudo -HE" to get root privileges for pip installation
 pip install pyangbind stdeb
 git clone https://osm.etsi.org/gerrit/osm/IM
 make -C IM clean all
 dpkg -i IM/deb_dist/python-pyang_*.deb
 dpkg -i IM/deb_dist/python-pyangbind_*.deb
 dpkg -i IM/deb_dist/python-osm-im_*.deb
 # Use this line to check if it is installed and where  
 python -c 'import osm_im; print osm_im.__path__[0]'

Package dependency lib-osm-openvim

It is under openvim repository. RO uses a library for the SDN assist that is in charge of performing the underlay dataplane connectivity using an openflow controller. osm-openvim and lib-osm-openvim are diffent, though they share same pieces of code. lib-osm-openvim uses a different database name (mano_vim_db) so that openvim can be installed in the same virtual machine or container (openvim uses vim_db database name)

TODO complete where RO uses it

Installation

The OSM binary installation install this package. In case of needed you can install/update it manually by: 

 apt-get update
 apt-get install make git python tox libmysqlclient-dev
 git clone https://osm.etsi.org/gerrit/osm/openvim
 git -C openvim checkout 005a9dc  # this is temporal, because last version contains error at Makefile
 make -C openvim lite
 # Use this line to check if it is installed and where  
 python -c 'import lib_osm_openvim; print lib_osm_openvim.__path__[0]'
 # Install database of ovim library
 OSMLIBOVIM_PATH=`python -c 'import lib_osm_openvim; print lib_osm_openvim.__path__[0]'`
 # -U and -P are the admin database user/password. Normally "-U root" without password "executed as root"
 ${OSMLIBOVIM_PATH}/database_utils/install-db-server.sh -U root [-P passwd] -u mano -p manopw -d mano_vim_db --updatedb

CLI client

The RO code contains a python CLI (openmano) that allows friently command execution. This CLI client can run on a separate machine where openmano server is running. "openmano config" indicates where the server is (by default localhost)

Northbound Interface

The RO uses a REST API with YAML/JSON content. The primitives are explained here

Running Unit Tests

Launching RO

Openmano can run as systemd service. (Can be installed with ./scripts/install-openmano-service.sh -f RO)

Or can be run inside a screen (the prefered method for developpers). The script to easily launch/remove is ./scripts/service-openmano.sh. Execute with -h to see options

Note that the general OSM install script, installs openmano inside a container and code run at folder /opt/openmano (instead of $HOME/openmano)

Tests

Many of openmano operations rely on an external VIM. Without external infraestructure it is recomended to use openvim in "test" (fake) mode. Install it in the same machine where openmano is located with OpenVIM_installation_(Release_One)#Installation

Run ./test/basictest for a initial test. Type -h to see options. For testing using openvim just run 

 . ./test/basictest.sh --force --screen --init-openvim

Creating a new VIM plugin

Choose a name, eg. XXX

Create a new python module vimconn_XXX.py derived from vimconn.py class. Implement the relevant functions. You have several connectors already created to be used as example as openstack and openvim. Openstack connector uses the python openstack client libraries, meanwhile openvim connector uses direct http requests.

DO NOT change the method names, parameters or parameter content. RO uses the same methods for all the VIMs and they cannot be changed to accomodate VIM specifics. VIM specifics must be solved inside the connector.

The new module can need specific configuration for the VIM that it is passed as a dictionary in the config variable at constructor. For example, in the case of openstack, config variable is used for: enabling/disabling port_security_enable, specifying the name of the physical network used for dataplane, regions, etc. The config variable is the right place to specify those needed parameters not provided by RO at the methods that form part of the VIM configuration. See Openstack configuration#Add openstack to OSM and Configuring AWS for OSM Release TWO#Add AWS to OSM for examples

For integration with the main project, it is needed to add the new vim type to dictionary "vim_module" at file "osm_ro/vim_thread.py". 

   openmano tenant-create osm  # if not already done
   export OPENMANO_TENANT=osm
   openmano datacenter-create <vim-name> <access-URL>  --type=XXX --config <specific config>
   # if fails the cause can be a library you need not installed or an error upon importing. Try tho import it in a python client
   # example of <specific config> text: "{dataplane_physical_net: physnet_sriov, port_security_enabled: False}"
   openmano datacenter-attach <vim-name> --vim-tenant-name=<tenant-to-use-by-openmano> --user=<tenant-user> --password=<tenant-password>

You can test it using the test/test_RO.py file: 

   # Create manually an image <image> and a management network <net-mgmt> at your VIM
   ./test/test_RO.py deploy --help
   # e.g. ./test/test_RO.py deploy -n <net-mgmt> -t osm -i <image> -d <vim-name> --timeout=30 --test simple_linux,simple_multi_vnfc,simple_2_vnf
   ./test/test_RO.py vimconn --help

To be able to run the test/test_RO.py script, you need to do some changes. First in the class "vim_connector" you need to import the plugin of the new vim: 

def test_vimconnector(args):
   global test_config
   sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + "/osm_ro")
   test_config['vimtype'] = args.vimtype
   if args.vimtype == "aws":
       import vimconn_aws as vim

And also,some tests are depending on the vim, you you would need to look for the following chain: 'test_config['vimtype'] ==' and place the code for the call to your vim there. For instance: 

   def test_000_connect(self):
       self.__class__.test_text = "{}.{}. TEST {}".format(test_config["test_number"],
                                                           self.__class__.test_index,
                                               inspect.currentframe().f_code.co_name)

       self.__class__.test_index += 1
       if test_config['vimtype'] == 'vmware':
           vca_object = test_config["vim_conn"].connect()
           logger.debug("{}".format(vca_object))
           self.assertIsNotNone(vca_object)
       elif test_config['vimtype'] == 'openstack':
           test_config["vim_conn"]._reload_connection()
           network_list = test_config["vim_conn"].get_network_list()
           logger.debug("{}".format(network_list))
           self.assertIsNotNone(network_list)

The script could be used to launch tests against the vim connector directly with the following syntax: 

python ./test/test_RO.py vimconn --vimtype XXX -t <VIM_TENANT_NAME> -u <VIM_AUTH_URL> -c '{user: <VIM_USER_NAME>, passwd: <VIM_USER_PASSWORD>}' --test <TEST_NAME>

Next you can find the tests that could be executed: 

test_vimconn_connect
test_vimconn_delete_network
test_vimconn_get_image_list
test_vimconn_get_network
test_vimconn_new_flavor
test_vimconn_new_network
test_vimconn_new_vminstance

Also, the script could be used to launch tests against the RO with the following syntax: 

python ./test/test_RO.py deploy -d <VIM_ACCOUNT_NAME> -n <MGMT_NET_IN_VIM> -i <IMAGE_FOR_VMs> --tests <TEST_NAME>

Next you can find the tests that could be executed: 

v3_2vdu_set_ip_mac
simple_multi_vnfc
simple_linux
Retrieved from "http:///wikipub/index.php?title=Developer_HowTo_for_RO_Module&oldid=6110"