From 06f1b70df32dc76b1132e8756b092a06ee47ad02 Mon Sep 17 00:00:00 2001 From: tierno Date: Fri, 14 Aug 2020 16:47:30 +0200 Subject: [PATCH] adding develop information for k8s Signed-off-by: tierno --- 02-developer-how-to.md | 133 ++++++++++++++++++++++++++++------------- 1 file changed, 92 insertions(+), 41 deletions(-) diff --git a/02-developer-how-to.md b/02-developer-how-to.md index fead85a..bb6a284 100644 --- a/02-developer-how-to.md +++ b/02-developer-how-to.md @@ -4,12 +4,12 @@ The aim of this chapter is to provide a guide for developers to set up their env ## Introduction -Modules run in separate docker containers (except juju controller that uses a lxd container). Current installation uses Docker Swarm by default (there are Kubernetes distributions and an incoming option for installing on Kubernetes). The modules are: +Modules run in separate docker containers (except juju controller that uses a lxd container). Current installation uses Docker Swarm by default, but there are a Kubernetes option for the installer. The modules are: - **kafka**: Provides a Kafka bus used for OSM communication. This module relies on *zookeeper*. - **zookeeper**: Used by *kafka*. - **nbi**: North Bound Interface of OSM. Restful server that follows ETSI SOL005 interface. Relies on *mongo* database and *kafka* bus. For authentication it can optionally uses *keystone*. -- **keystone**: Used optionally for NBI authentication and RBAC. It stores the users, projects and role permissions. It relies on *mysql*. +- **keystone**: Used for NBI authentication and RBAC. It stores the users, projects and role permissions. It relies on *mysql*. - **lcm**: Provides the Live Cycle Management. It uses *ro* for resource orchestration and juju for configuration. It relies also on *mongo*. - **ro**: Makes the Resource Orchestration, or VIM deployment. Relies on *mysql*. - **light-ui**: Web user interface. It communicates with *nbi*. @@ -19,7 +19,7 @@ Modules run in separate docker containers (except juju controller that uses a lx - **pol**: Policy Manager for OSM. - **prometheus**: for monitoring . -In addition LCM and NBI, shares a common file system where packages are stored. In current installation it uses a shared docker volume called *osm_package*. +In addition LCM and NBI, shares a common file system where packages are stored. For Docker Swarmit uses a shared docker volume called *osm_package*. For Kubernetes it uses the host mount path */var/lib/osm/osm/osm_osm_packages/_data*. This picture shows the modules with the port that they export, their relationships with the name of env variables that control them. For clarity not all the dependencies are depicted, as kafka, mysql, ... @@ -61,7 +61,10 @@ For debugging it is convenient running the target module directly at host mainta First thing, you need to stop the module you want to debug. As OSM uses a docker service do not manually stop the container because it will be automatically relaunched again. Scale it to 0 for stopping and to 1 for running again. ```bash +# For Docker Swarm: docker service scale osm_lcm=0 +# For Kubernetes: +kubectl -n osm scale deployment lcm --replicas=0 ``` ### 2 Clone the module @@ -101,13 +104,13 @@ You need to feed the IP addresses of the modules it is going to communicate to. In case the module under development is running in the same server where the rest of the modules are located, use "127.0.0.1" (localhost) as the IP address of those modules. For instance, in the following example we have added mongo, mysql, ro and kafka to the line containing "127.0.0.1": ```text -127.0.0.1 localhost mongo keystone ro kafka nbi mysql +127.0.0.1 localhost kafka keystone nbi mongo mysql ro ``` In case the module under development is running in a different server from the rest of modules, you will need to provide the IP address of that server. For instance, in the following example we have added a new line with the name resolution for mongo ro and kafka to IP address "a.b.c.d": ```text -a.b.c.d mongo keystone ro kafka nbi mysql +a.b.c.d kafka keystone nbi mongo mysql ro ``` ### 6 Install needed packages @@ -125,16 +128,21 @@ Install them with: ```bash python3 -m pip install -e common --user # needed for LCM, NBI python3 -m pip install -e N2VC --user # needed for LCM -python3 -m pip install -e IM --user # needed for NBI +python3 -m pip install IM --user -U # needed for NBI and RO. Option -e does not work ``` ### 7 Expose needed ports of docker services +#### Expose services for Docker Swarm + To expose **mongo** database and **kafka** (services `osm_mongo`, `osm_kafka`), needed for NBI, LCM, MON: ```bash -docker service update osm_mongo --publish-add 27017:27017 docker service update osm_kafka --publish-add 9092:9092 +docker service update osm_keystone --publish-add 5000:5000 +docker service update osm_mongo --publish-add 27017:27017 +docker service update osm_mysql --publish-add 3306:3306 +docker service update osm_ro --publish-add 9090:9090 # check exposed ports by: docker service list ``` @@ -154,15 +162,25 @@ docker stack deploy -c /etc/osm/docker/docker-compose.yaml osm # make again the service scale to 0 of step 1 in this section ``` -Similarly, **mysql** (service `osm_mysql`) needed for RO, uses ports "3306:3306" +#### Expose services for Kubernetes -... and, **kafka** (service `osm_kafka`) needed for LCM, NBI uses ports "9092:9092". Note: for kafka sometimes this is not enough and other modules has errors connecting to kafka controller. If this is the case modify docker-compose adding kafka bus and relaunch service +Download file 'debug_k8s_services.yaml' from . Optionally remove sections not needed and apply it. + +```bash +kubectl -n osm apply -f debug_k8s_services.yaml +# check exposed ports by (those ended with -debug are the one added): +kubectl -n osm get service +# Undo with: +kubectl -n osm delete service/mongo-debug # mongo-debug, mysql-debug, kafka-debug ro-debug or/and keystone-debug +``` ## NBI -Change docker service **mongo** and **kafka** to expose ports. See [Developer_HowTo - Expose needed ports of docker services](02-developer-how-to.md#7-expose-needed-ports-of-docker-services) +Ensure services for **mongo**, **kafka** and **keystone** expose their ports. See [Developer_HowTo - Expose needed ports of docker services](02-developer-how-to.md#7-expose-needed-ports-of-docker-services) + +Additionally you may want that docker `light-ui` uses your local copy of NBI: -Additionally you may want that docker `light-ui` uses your local copy of NBI by one of: +For Docker Swarm use one of: - Update light-ui docker service: @@ -178,12 +196,25 @@ OSM_SERVER: <172.17.0.1> # nbi # Get the needed address at your setup by 'ip a | grep docker0' ``` -Clone and install needed IM and NBI packages. Scale to 0 osm_nbi service +Then restart stack to apply this change with: + +```bash +docker stack rm osm +docker stack deploy -c /etc/osm/docker/docker-compose.yaml osm +``` + +Alternatively For Kubernetes use one of: + +```bash +kubectl -n osm patch deployment light-ui --patch '{"spec": {"template": {"spec": {"containers": [{"name": "light-ui", "env": [{"name": "OSM_SERVER", "value": "172.17.0.1"}] }]}}}}' +``` + +Clone and install needed IM and NBI packages. ```bash # Install Information Model IM git clone https://osm.etsi.org/gerrit/osm/IM -python3 -m pip install -e IM --user +python3 -m pip install IM --user git clone https://osm.etsi.org/gerrit/osm/NBI # configure gerrit commit-msg hook @@ -191,24 +222,37 @@ curl -Lo NBI/.git/hooks/commit-msg http://osm.etsi.org/gerrit/tools/hooks/commit chmod u+x NBI/.git/hooks/commit-msg cp NBI/.gitignore-common NBI/.gitignore python3 -m pip install -e NBI # try with --user if get permission errors - -docker service scale osm_nbi=0 ``` Python interpreter: Python3 Script: `$INSTALLATION_FOLDER/NBI/osm_nbi/nbi.py` Environment variables: -- `OSMNBI_DATABASE_COMMONKEY`: must be the same value used by NBI. Get it with `cat /etc/osm/docker/nbi.env` +- `OSMNBI_DATABASE_COMMONKEY`: must be the same value used by NBI. Get it with `cat /etc/osm/docker/nbi.env` or `kubectl -n osm exec -ti deployment/nbi env`. +- `OSMNBI_AUTHENTICATION_SERVICE_PASSWORD`: obtain in the same way as before - `OSMNBI_STORAGE_PATH`: Path of the docker volume for file storage. Both LCM and NBI must share the same path. You can either: - Create a folder and debug both NBI and LCM at the same time (needed if you develop in a different server than OSM); or - - Use the docker volume. Discover local path (mount point) with `docker volume inspect osm_osm_packages` and grant write permissions to pycharm on it with (path can be different at your environment) `sudo chmod o+rx /var/lib/docker /var/lib/docker/volumes; sudo chmod -R o+w /var/lib/docker/volumes/osm_osm_packages/_data` + - Use the docker volume. + - For Docker Swarm discover local path (mount point) with `docker volume inspect osm_osm_packages` and grant write permissions to pycharm on it with (path can be different at your environment) `sudo chmod o+rx /var/lib/docker /var/lib/docker/volumes; sudo chmod -R o+w /var/lib/docker/volumes/osm_osm_packages/_data`. + - For Kubernetes use path: '/var/lib/osm/osm/osm_osm_packages/_data'. Exec `sudo chmod -R o+w /var/lib/osm/osm/osm_osm_packages/_data` - `OSMNBI_DATABASE_HOST`: Mongo IP in case host `mongo` is not at `/etc/hosts` file - `OSMNBI_STATIC_DIR`: `/osm_nbi/html_public` +Finally remove running docker service. +For Docker Swarm exec `docker service scale osm_nbi=0`; scale to 1 to undo. For Kubernetes: + +```bash +kubectl -n osm scale deployment nbi --replicas=0 +kubectl -n osm delete service/nbi +# undo both with: +# kubectl -n osm apply -f /etc/osm/docker/osm_pods/nbi.yaml +``` + ## LCM -Install needed OSM packages if not done: +Ensure services for **mongo** and **kafka** expose their ports. See [Developer_HowTo - Expose needed ports of docker services](02-developer-how-to.md#7-expose-needed-ports-of-docker-services) + +Install needed OSM packages if you intend to modify develop them: ```bash # N2VC: @@ -220,9 +264,7 @@ git clone https://osm.etsi.org/gerrit/osm/common python3 -m pip install -e common # try with --user if get permission errors ``` -Change docker service **mongo** and **kafka** to expose ports. See [Developer_HowTo#7 Expose needed ports of docker services](02-developer-how-to.md#7-expose-needed-ports-of-docker-services) - -Clone and install LCM depencencies +Clone LCM ```bash docker service scale osm_lcm=0 @@ -240,10 +282,12 @@ Environment variables: Values are stored at `/etc/osm/docker/lcm.env` but here it is explained how to obtain these values. -- `OSMLCM_DATABASE_COMMONKEY`: must be the same value used by NBI. Get it with `cat /etc/osm/docker/lcm.env` -- `OSMLCM_STORAGE_PATH`: Path of the docker volume for filestorage. Both LCM and NBI must share the same path. You can either: +- `OSMLCM_DATABASE_COMMONKEY`: must be the same value used by NBI. Get it with `cat /etc/osm/docker/nbi.env` or `kubectl -n osm exec -ti deployment/nbi env`. +- `OSMLCM_STORAGE_PATH`: Path of the docker volume for file storage. Both LCM and NBI must share the same path. You can either: - Create a folder and debug both NBI and LCM at the same time (needed if you develop in a different server than OSM); or - - Use the docker volume. Discover local path (mount point) with `docker volume inspect osm_osm_packages` and grant read permissions to pycharm on it with (path can be different at your environment) `sudo chmod o+rx /var/lib/docker /var/lib/docker/volumes` + - Use the docker volume. + - For Docker Swarm discover local path (mount point) with `docker volume inspect osm_osm_packages` and grant write permissions to pycharm on it with (path can be different at your environment) `sudo chmod o+rx /var/lib/docker /var/lib/docker/volumes; sudo chmod -R o+w /var/lib/docker/volumes/osm_osm_packages/_data`. + - For Kubernetes use path: '/var/lib/osm/osm/osm_osm_packages/_data'. Exec `sudo chmod -R o+w /var/lib/osm/osm/osm_osm_packages/_data` - `OSMLCM_DATABASE_HOST`: Mongo IP in case 'mongo' host it is not at `/etc/hosts` file. See [Developer_HowTo#5 Configure it to interact with other modules](02-developer-how-to.md#5-configure-it-to-interact-with-other-modules) - `OSMLCM_RO_HOST`: RO IP in case 'ro' host it is not at `/etc/hosts` file - `OSMLCM_VCA_CACERT`: To get this value run the following command in the OSM host: @@ -262,24 +306,16 @@ Values are stored at `/etc/osm/docker/lcm.env` but here it is explained how to o - `ssh -L 0.0.0.0:17070::17070 root@` - `OSMLCM_GLOBAL_LOGLEVEL`: DEBUG -## RO - -It is needed to expose the database port of the `mysql` container by one of: +Finally remove running docker service. +For Docker Swarm exec `docker service scale osm_lcm=0`; scale to 1 to undo. For Kubernetes run `kubectl -n osm scale deployment lcm --replicas=0`; scale to 1 to undo -- Modify docker service +## RO - ```bash - docker service update osm_mysql --publish-add 3306:3306 - ``` +Ensure services for **mysql** exposes its port. See [Developer_HowTo - Expose needed ports of docker services](02-developer-how-to.md#7-expose-needed-ports-of-docker-services) -- Edit file `/etc/osm/docker/docker-compose.yaml` (recommended to make a backup copy), and uncomment at `mysql` section the following two lines. (Restart stack afterwards): +Additionally you may want that docker `lcm` uses your local copy of RO. -```yaml - ports: - - "3306:3306" -``` - -Additionally you may want that docker `lcm` uses your local copy of RO. Similarly it can be done by one of: +For Docker Swarmit can be done by one of: - Modify docker service @@ -296,17 +332,22 @@ Additionally you may want that docker `lcm` uses your local copy of RO. Similarl # Get the needed address at your setup by 'ip a | grep docker0' ``` -**Note**: to restart stack, run: +Then restart stack to apply this change with: ```bash docker stack rm osm docker stack deploy -c /etc/osm/docker/docker-compose.yaml osm ``` -Remove docker service, clone code: +Alternatively For Kubernetes use one of: + +```bash +kubectl -n osm patch deployment lcm --patch '{"spec": {"template": {"spec": {"containers": [{"name": "lcm", "env": [{"name": "OSMLCM_RO_HOST", "value": "172.17.0.1"}] }]}}}}' +``` + +Install needed packages and clone code: ```bash -docker service scale osm_ro=0 # clone and configure gerrit commit-msg hook git clone https://osm.etsi.org/gerrit/osm/RO curl -Lo RO/.git/hooks/commit-msg http://osm.etsi.org/gerrit/tools/hooks/commit-msg @@ -357,3 +398,13 @@ Environment variables: - `RO_DB_HOST`, `RO_DB_OVIM_HOST`: `mysql` or `` depending if running on the same server or not. - `RO_LOG_LEVEL`: `DEBUG` + +Finally remove running docker service. +For Docker Swarm exec `docker service scale osm_ro=0`; scale to 1 to undo. For Kubernetes: + +```bash +kubectl -n osm scale deployment ro --replicas=0 +kubectl -n osm delete service/ro +# undo both with: +# kubectl -n osm apply -f /etc/osm/docker/osm_pods/ro.yaml +``` -- GitLab