nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
nc: kafka (10.97.12.81:9092): Operation timed out
kafka (10.97.12.81:9092) open
mongodb-k8s (10.244.0.41:27017) open
05/21/2024 02:41:46 PM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connecting> [IPv4 ('10.97.12.81', 9092)]>: connecting to kafka:9092 [('10.97.12.81', 9092) IPv4]
05/21/2024 02:41:46 PM - kafka.conn - INFO - Probing node bootstrap-0 broker version
05/21/2024 02:41:46 PM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connecting> [IPv4 ('10.97.12.81', 9092)]>: Connection complete.
05/21/2024 02:41:46 PM - kafka.conn - INFO - Broker version identified as 2.5.0
05/21/2024 02:41:46 PM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup
05/21/2024 02:41:46 PM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connecting> [IPv4 ('10.97.12.81', 9092)]>: connecting to kafka:9092 [('10.97.12.81', 9092) IPv4]
05/21/2024 02:41:46 PM - kafka.conn - INFO - Probing node bootstrap-0 broker version
05/21/2024 02:41:46 PM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connecting> [IPv4 ('10.97.12.81', 9092)]>: Connection complete.
05/21/2024 02:41:46 PM - kafka.conn - INFO - Broker version identified as 2.5.0
05/21/2024 02:41:46 PM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup
05/21/2024 02:41:47 PM - kafka.conn - INFO - <BrokerConnection node_id=1 host=kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 <connecting> [IPv4 ('10.244.0.33', 9092)]>: connecting to kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.33', 9092) IPv4]
05/21/2024 02:41:47 PM - kafka.conn - INFO - <BrokerConnection node_id=1 host=kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 <connecting> [IPv4 ('10.244.0.33', 9092)]>: Connection complete.
05/21/2024 02:41:47 PM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connected> [IPv4 ('10.97.12.81', 9092)]>: Closing connection. 
05/21/2024 02:41:47 PM - kafka.cluster - ERROR - Topic users not found in cluster metadata
05/21/2024 02:41:47 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata
05/21/2024 02:41:47 PM - kafka.conn - INFO - <BrokerConnection node_id=1 host=kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 <connected> [IPv4 ('10.244.0.33', 9092)]>: Closing connection. 
05/21/2024 02:41:47 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder...
05/21/2024 02:41:47 PM - db - INFO - Connected to database osm version 1.2
05/21/2024 02:41:47 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON kafka Consumer...
05/21/2024 02:41:47 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder...
05/21/2024 02:41:47 PM - aiokafka.consumer.subscription_state - INFO - Updating subscribed topics to: frozenset({'users', 'project'})
05/21/2024 02:41:47 PM - aiokafka.consumer.consumer - INFO - Subscribed to topic(s): {'users', 'project'}
05/21/2024 02:41:47 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer
05/21/2024 02:41:47 PM - aiokafka.consumer.group_coordinator - INFO - Revoking previously assigned partitions set() for group mon-consumer
05/21/2024 02:41:47 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer
05/21/2024 02:41:48 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana
05/21/2024 02:41:51 PM - aiokafka.consumer.group_coordinator - INFO - Joined group 'mon-consumer' (generation 1) with member_id aiokafka-0.8.1-c7ce0c3b-d016-4490-b866-8111b835374b
05/21/2024 02:41:51 PM - aiokafka.consumer.group_coordinator - INFO - Elected group leader -- performing partition assignments using roundrobin
05/21/2024 02:41:51 PM - aiokafka.consumer.group_coordinator - INFO - Successfully synced group mon-consumer with generation 1
05/21/2024 02:41:51 PM - aiokafka.consumer.group_coordinator - INFO - Setting newly assigned partitions {TopicPartition(topic='users', partition=0), TopicPartition(topic='project', partition=0)} for group mon-consumer
05/21/2024 02:45:49 PM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8stest202405211425 is created in Grafana
05/21/2024 02:45:49 PM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":3,"uid":"a8d64593-6c0d-4ea2-8a6c-00e51f5c06ee","orgId":1,"name":"prom-operator-k8stest202405211425","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.23.14:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":3,"message":"Datasource added","name":"prom-operator-k8stest202405211425"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Tue, 21 May 2024 14:45:49 GMT', 'Content-Length': '519'}, 'raw': <urllib3.response.HTTPResponse object at 0x7aaf9c3b7070>, 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(microseconds=60894), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x7aaf9c3b6fb0>}
05/21/2024 02:45:49 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8stest202405211425 is created in Grafana
05/21/2024 04:04:04 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana
05/21/2024 06:36:09 PM - osm_mon.dashboarder.backends.grafana - INFO - New user basic_15_test_user created in Grafana
05/21/2024 06:36:13 PM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user is assigned Admin permission
05/21/2024 06:36:18 PM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 116, in _user_msg
    project_data = values["changes"]["add_project_role_mappings"]
KeyError: 'add_project_role_mappings'
05/21/2024 06:36:20 PM - osm_mon.dashboarder.backends.grafana - INFO - New team basic_15_test_project created in Grafana
05/21/2024 06:36:22 PM - osm_mon.dashboarder.backends.grafana - INFO - Grafana team updated {"message":"Team updated"}
05/21/2024 06:36:31 PM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user deleted in Grafana
05/21/2024 06:36:32 PM - osm_mon.dashboarder.backends.grafana - INFO - Team basic_15_project_test deleted in Grafana
05/21/2024 06:37:06 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana
05/21/2024 08:07:55 PM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s02 is created in Grafana
05/21/2024 08:07:55 PM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":4,"uid":"d529c333-370a-4e25-88ac-c64b7ea46b75","orgId":1,"name":"prom-operator-k8s02","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.23.14:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":4,"message":"Datasource added","name":"prom-operator-k8s02"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Tue, 21 May 2024 20:07:55 GMT', 'Content-Length': '491'}, 'raw': <urllib3.response.HTTPResponse object at 0x7aaf9dd9a2f0>, 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(microseconds=62356), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x7aaf9dd99cc0>}
05/21/2024 08:07:55 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s02 is created in Grafana
05/21/2024 09:25:13 PM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s07 is created in Grafana
05/21/2024 09:25:13 PM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":5,"uid":"f5ca3843-ca42-4e9c-a5a6-54f239e02678","orgId":1,"name":"prom-operator-k8s07","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.23.14:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":5,"message":"Datasource added","name":"prom-operator-k8s07"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Tue, 21 May 2024 21:25:13 GMT', 'Content-Length': '491'}, 'raw': <urllib3.response.HTTPResponse object at 0x7aaf9dd9b940>, 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(microseconds=65241), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x7aaf9dd98f40>}
05/21/2024 09:25:13 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s07 is created in Grafana
05/21/2024 10:05:22 PM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s12 is created in Grafana
05/21/2024 10:05:22 PM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":6,"uid":"d852f7db-6ede-4ea3-9a56-cbd29935c254","orgId":1,"name":"prom-operator-k8s12","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.23.14:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":6,"message":"Datasource added","name":"prom-operator-k8s12"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Tue, 21 May 2024 22:05:22 GMT', 'Content-Length': '491'}, 'raw': <urllib3.response.HTTPResponse object at 0x7aaf9dd98d00>, 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(microseconds=77846), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x7aaf9dd9a110>}
05/21/2024 10:05:22 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s12 is created in Grafana
05/21/2024 10:46:32 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana