nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
nc: kafka (10.109.62.213:9092): Operation timed out
kafka (10.109.62.213:9092) open
mongodb-k8s (10.244.0.31:27017) open
05/19/2024 11:26:14 AM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connecting> [IPv4 ('10.109.62.213', 9092)]>: connecting to kafka:9092 [('10.109.62.213', 9092) IPv4]
05/19/2024 11:26:14 AM - kafka.conn - INFO - Probing node bootstrap-0 broker version
05/19/2024 11:26:14 AM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connecting> [IPv4 ('10.109.62.213', 9092)]>: Connection complete.
05/19/2024 11:26:14 AM - kafka.conn - INFO - Broker version identified as 2.5.0
05/19/2024 11:26:14 AM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup
05/19/2024 11:26:14 AM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connecting> [IPv4 ('10.109.62.213', 9092)]>: connecting to kafka:9092 [('10.109.62.213', 9092) IPv4]
05/19/2024 11:26:14 AM - kafka.conn - INFO - Probing node bootstrap-0 broker version
05/19/2024 11:26:14 AM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connecting> [IPv4 ('10.109.62.213', 9092)]>: Connection complete.
05/19/2024 11:26:14 AM - kafka.conn - INFO - Broker version identified as 2.5.0
05/19/2024 11:26:14 AM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup
05/19/2024 11:26:14 AM - kafka.conn - INFO - <BrokerConnection node_id=0 host=kafka-controller-0.kafka-controller-headless.osm.svc.cluster.local:9092 <connecting> [IPv4 ('10.244.0.44', 9092)]>: connecting to kafka-controller-0.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.44', 9092) IPv4]
05/19/2024 11:26:14 AM - kafka.conn - INFO - <BrokerConnection node_id=0 host=kafka-controller-0.kafka-controller-headless.osm.svc.cluster.local:9092 <connecting> [IPv4 ('10.244.0.44', 9092)]>: Connection complete.
05/19/2024 11:26:14 AM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connected> [IPv4 ('10.109.62.213', 9092)]>: Closing connection. 
05/19/2024 11:26:17 AM - kafka.conn - INFO - <BrokerConnection node_id=2 host=kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 <connecting> [IPv4 ('10.244.0.33', 9092)]>: connecting to kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.33', 9092) IPv4]
05/19/2024 11:26:17 AM - kafka.conn - INFO - <BrokerConnection node_id=2 host=kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 <connecting> [IPv4 ('10.244.0.33', 9092)]>: Connection complete.
05/19/2024 11:26:17 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata
05/19/2024 11:26:17 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata
05/19/2024 11:26:17 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata
05/19/2024 11:26:17 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata
05/19/2024 11:26:17 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata
05/19/2024 11:26:18 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata
05/19/2024 11:26:18 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata
05/19/2024 11:26:18 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata
05/19/2024 11:26:18 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata
05/19/2024 11:26:18 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata
05/19/2024 11:26:18 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata
05/19/2024 11:26:18 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata
05/19/2024 11:26:18 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata
05/19/2024 11:26:18 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata
05/19/2024 11:26:18 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata
05/19/2024 11:26:19 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata
05/19/2024 11:26:19 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata
05/19/2024 11:26:19 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata
05/19/2024 11:26:19 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata
05/19/2024 11:26:19 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata
05/19/2024 11:26:19 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata
05/19/2024 11:26:19 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata
05/19/2024 11:26:19 AM - kafka.conn - INFO - <BrokerConnection node_id=1 host=kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 <connecting> [IPv4 ('10.244.0.34', 9092)]>: connecting to kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.34', 9092) IPv4]
05/19/2024 11:26:19 AM - kafka.conn - INFO - <BrokerConnection node_id=1 host=kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 <connecting> [IPv4 ('10.244.0.34', 9092)]>: Connection complete.
05/19/2024 11:26:19 AM - kafka.conn - INFO - <BrokerConnection node_id=0 host=kafka-controller-0.kafka-controller-headless.osm.svc.cluster.local:9092 <connected> [IPv4 ('10.244.0.44', 9092)]>: Closing connection. 
05/19/2024 11:26:19 AM - kafka.conn - INFO - <BrokerConnection node_id=2 host=kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 <connected> [IPv4 ('10.244.0.33', 9092)]>: Closing connection. 
05/19/2024 11:26:19 AM - kafka.conn - INFO - <BrokerConnection node_id=1 host=kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 <connected> [IPv4 ('10.244.0.34', 9092)]>: Closing connection. 
05/19/2024 11:26:19 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder...
05/19/2024 11:26:19 AM - db - INFO - Connected to database osm version 1.2
05/19/2024 11:26:19 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON kafka Consumer...
05/19/2024 11:26:19 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder...
05/19/2024 11:26:19 AM - aiokafka.consumer.subscription_state - INFO - Updating subscribed topics to: frozenset({'project', 'users'})
05/19/2024 11:26:19 AM - aiokafka.consumer.consumer - INFO - Subscribed to topic(s): {'project', 'users'}
05/19/2024 11:26:20 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer
05/19/2024 11:26:20 AM - aiokafka.consumer.group_coordinator - INFO - Revoking previously assigned partitions set() for group mon-consumer
05/19/2024 11:26:20 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer
05/19/2024 11:26:21 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana
05/19/2024 11:26:23 AM - aiokafka.consumer.group_coordinator - INFO - Joined group 'mon-consumer' (generation 1) with member_id aiokafka-0.8.1-8d84a7a1-30c0-47a3-9709-ac1feba03478
05/19/2024 11:26:23 AM - aiokafka.consumer.group_coordinator - INFO - Elected group leader -- performing partition assignments using roundrobin
05/19/2024 11:26:23 AM - aiokafka.consumer.group_coordinator - INFO - Successfully synced group mon-consumer with generation 1
05/19/2024 11:26:23 AM - aiokafka.consumer.group_coordinator - INFO - Setting newly assigned partitions {TopicPartition(topic='project', partition=0), TopicPartition(topic='users', partition=0)} for group mon-consumer
05/19/2024 11:30:21 AM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8stest202405191109 is created in Grafana
05/19/2024 11:30:21 AM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":3,"uid":"b07ef866-ed31-437b-a9a4-875a8da35dec","orgId":1,"name":"prom-operator-k8stest202405191109","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.23.10:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":3,"message":"Datasource added","name":"prom-operator-k8stest202405191109"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 19 May 2024 11:30:21 GMT', 'Content-Length': '519'}, 'raw': <urllib3.response.HTTPResponse object at 0x718fc83fd870>, 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(microseconds=64933), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x718fc83fd9c0>}
05/19/2024 11:30:22 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8stest202405191109 is created in Grafana
05/19/2024 12:48:07 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana
05/19/2024 03:19:11 PM - osm_mon.dashboarder.backends.grafana - INFO - New user basic_15_test_user created in Grafana
05/19/2024 03:19:14 PM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user is assigned Admin permission
05/19/2024 03:19:20 PM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 116, in _user_msg
    project_data = values["changes"]["add_project_role_mappings"]
KeyError: 'add_project_role_mappings'
05/19/2024 03:19:21 PM - osm_mon.dashboarder.backends.grafana - INFO - New team basic_15_test_project created in Grafana
05/19/2024 03:19:23 PM - osm_mon.dashboarder.backends.grafana - INFO - Grafana team updated {"message":"Team updated"}
05/19/2024 03:19:32 PM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user deleted in Grafana
05/19/2024 03:19:34 PM - osm_mon.dashboarder.backends.grafana - INFO - Team basic_15_project_test deleted in Grafana
05/19/2024 03:20:08 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana
05/19/2024 04:50:58 PM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s02 is created in Grafana
05/19/2024 04:50:58 PM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":4,"uid":"c4f4eca1-d22d-48ad-bb31-4b076b2131fb","orgId":1,"name":"prom-operator-k8s02","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.23.10:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":4,"message":"Datasource added","name":"prom-operator-k8s02"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 19 May 2024 16:50:58 GMT', 'Content-Length': '491'}, 'raw': <urllib3.response.HTTPResponse object at 0x718fc83abfd0>, 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(microseconds=140733), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x718fc83e73d0>}
05/19/2024 04:50:58 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s02 is created in Grafana
05/19/2024 06:07:45 PM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s07 is created in Grafana
05/19/2024 06:07:45 PM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":5,"uid":"f06c1b55-eb26-4b15-8d81-a88cb0f09bc8","orgId":1,"name":"prom-operator-k8s07","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.23.10:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":5,"message":"Datasource added","name":"prom-operator-k8s07"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 19 May 2024 18:07:45 GMT', 'Content-Length': '491'}, 'raw': <urllib3.response.HTTPResponse object at 0x718fc83abdc0>, 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(microseconds=83231), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x718fc83e4130>}
05/19/2024 06:07:45 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s07 is created in Grafana
05/19/2024 06:47:25 PM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s12 is created in Grafana
05/19/2024 06:47:25 PM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":6,"uid":"e5d848c8-d8d1-4958-8424-c9eaa51c997c","orgId":1,"name":"prom-operator-k8s12","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.23.10:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":6,"message":"Datasource added","name":"prom-operator-k8s12"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 19 May 2024 18:47:25 GMT', 'Content-Length': '491'}, 'raw': <urllib3.response.HTTPResponse object at 0x718fc83aa4a0>, 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(microseconds=53173), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x718fc83e49d0>}
05/19/2024 06:47:25 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s12 is created in Grafana
05/19/2024 07:28:05 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana