nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
nc: kafka (10.106.26.100:9092): Operation timed out
kafka (10.106.26.100:9092) open
mongodb-k8s (10.244.0.42:27017) open
01/28/2024 12:26:10 PM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connecting> [IPv4 ('10.106.26.100', 9092)]>: connecting to kafka:9092 [('10.106.26.100', 9092) IPv4]
01/28/2024 12:26:10 PM - kafka.conn - INFO - Probing node bootstrap-0 broker version
01/28/2024 12:26:10 PM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connecting> [IPv4 ('10.106.26.100', 9092)]>: Connection complete.
01/28/2024 12:26:11 PM - kafka.conn - INFO - Broker version identified as 2.5.0
01/28/2024 12:26:11 PM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup
01/28/2024 12:26:11 PM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connecting> [IPv4 ('10.106.26.100', 9092)]>: connecting to kafka:9092 [('10.106.26.100', 9092) IPv4]
01/28/2024 12:26:11 PM - kafka.conn - INFO - Probing node bootstrap-0 broker version
01/28/2024 12:26:11 PM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connecting> [IPv4 ('10.106.26.100', 9092)]>: Connection complete.
01/28/2024 12:26:11 PM - kafka.conn - INFO - Broker version identified as 2.5.0
01/28/2024 12:26:11 PM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup
01/28/2024 12:26:11 PM - kafka.conn - INFO - <BrokerConnection node_id=2 host=kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 <connecting> [IPv4 ('10.244.0.34', 9092)]>: connecting to kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.34', 9092) IPv4]
01/28/2024 12:26:11 PM - kafka.conn - INFO - <BrokerConnection node_id=2 host=kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 <connecting> [IPv4 ('10.244.0.34', 9092)]>: Connection complete.
01/28/2024 12:26:11 PM - kafka.conn - INFO - <BrokerConnection node_id=bootstrap-0 host=kafka:9092 <connected> [IPv4 ('10.106.26.100', 9092)]>: Closing connection. 
01/28/2024 12:26:15 PM - kafka.conn - INFO - <BrokerConnection node_id=1 host=kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 <connecting> [IPv4 ('10.244.0.33', 9092)]>: connecting to kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.33', 9092) IPv4]
01/28/2024 12:26:15 PM - kafka.conn - INFO - <BrokerConnection node_id=1 host=kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 <connecting> [IPv4 ('10.244.0.33', 9092)]>: Connection complete.
01/28/2024 12:26:15 PM - kafka.producer.record_accumulator - WARNING - Produced messages to topic-partition TopicPartition(topic='alarm_request', partition=0) with base offset -1 log start offset -1 and error None.
01/28/2024 12:26:15 PM - kafka.cluster - ERROR - Topic users not found in cluster metadata
01/28/2024 12:26:15 PM - kafka.cluster - ERROR - Topic users not found in cluster metadata
01/28/2024 12:26:16 PM - kafka.cluster - ERROR - Topic users not found in cluster metadata
01/28/2024 12:26:16 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata
01/28/2024 12:26:16 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata
01/28/2024 12:26:16 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata
01/28/2024 12:26:16 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata
01/28/2024 12:26:16 PM - kafka.conn - INFO - <BrokerConnection node_id=2 host=kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 <connected> [IPv4 ('10.244.0.34', 9092)]>: Closing connection. 
01/28/2024 12:26:16 PM - kafka.conn - INFO - <BrokerConnection node_id=1 host=kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 <connected> [IPv4 ('10.244.0.33', 9092)]>: Closing connection. 
01/28/2024 12:26:16 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder...
01/28/2024 12:26:16 PM - db - INFO - Connected to database osm version 1.2
01/28/2024 12:26:16 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON kafka Consumer...
01/28/2024 12:26:16 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder...
01/28/2024 12:26:16 PM - aiokafka.consumer.subscription_state - INFO - Updating subscribed topics to: frozenset({'project', 'users'})
01/28/2024 12:26:16 PM - aiokafka.consumer.consumer - INFO - Subscribed to topic(s): {'project', 'users'}
01/28/2024 12:26:16 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer
01/28/2024 12:26:16 PM - aiokafka.consumer.group_coordinator - INFO - Revoking previously assigned partitions set() for group mon-consumer
01/28/2024 12:26:16 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer
01/28/2024 12:26:16 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana
01/28/2024 12:26:19 PM - aiokafka.consumer.group_coordinator - INFO - Joined group 'mon-consumer' (generation 1) with member_id aiokafka-0.8.1-c8bffa7e-70cf-4f23-a2f3-6b398d930c90
01/28/2024 12:26:19 PM - aiokafka.consumer.group_coordinator - INFO - Elected group leader -- performing partition assignments using roundrobin
01/28/2024 12:26:19 PM - aiokafka.consumer.group_coordinator - INFO - Successfully synced group mon-consumer with generation 1
01/28/2024 12:26:19 PM - aiokafka.consumer.group_coordinator - INFO - Setting newly assigned partitions {TopicPartition(topic='project', partition=0), TopicPartition(topic='users', partition=0)} for group mon-consumer
01/28/2024 12:29:47 PM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8stest202401281208 is created in Grafana
01/28/2024 12:29:47 PM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":3,"uid":"e8f90f82-8897-4760-8bc4-e77a16c813b0","orgId":1,"name":"prom-operator-k8stest202401281208","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.23.5:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":3,"message":"Datasource added","name":"prom-operator-k8stest202401281208"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 28 Jan 2024 12:29:47 GMT', 'Content-Length': '518'}, 'raw': <urllib3.response.HTTPResponse object at 0x7f4a946fca60>, 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(microseconds=36313), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x7f4a9468ada0>}
01/28/2024 12:29:47 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8stest202401281208 is created in Grafana
01/28/2024 01:48:59 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana
01/28/2024 04:21:48 PM - osm_mon.dashboarder.backends.grafana - INFO - New user basic_15_test_user created in Grafana
01/28/2024 04:21:52 PM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user is assigned Admin permission
01/28/2024 04:21:57 PM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 116, in _user_msg
    project_data = values["changes"]["add_project_role_mappings"]
KeyError: 'add_project_role_mappings'
01/28/2024 04:21:59 PM - osm_mon.dashboarder.backends.grafana - INFO - New team basic_15_test_project created in Grafana
01/28/2024 04:22:01 PM - osm_mon.dashboarder.backends.grafana - INFO - Grafana team updated {"message":"Team updated"}
01/28/2024 04:22:10 PM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user deleted in Grafana
01/28/2024 04:22:12 PM - osm_mon.dashboarder.backends.grafana - INFO - Team basic_15_project_test deleted in Grafana
01/28/2024 04:22:23 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana
01/28/2024 05:53:08 PM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s02 is created in Grafana
01/28/2024 05:53:08 PM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":4,"uid":"e06f6bd0-0547-48bf-b2b2-37a7c6b75850","orgId":1,"name":"prom-operator-k8s02","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.23.5:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":4,"message":"Datasource added","name":"prom-operator-k8s02"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 28 Jan 2024 17:53:08 GMT', 'Content-Length': '490'}, 'raw': <urllib3.response.HTTPResponse object at 0x7f4a94670e20>, 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(microseconds=106804), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x7f4a946739a0>}
01/28/2024 05:53:08 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s02 is created in Grafana
01/28/2024 07:09:51 PM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s07 is created in Grafana
01/28/2024 07:09:51 PM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":5,"uid":"b26bcb5e-830f-47af-bd4f-e30a24c171bf","orgId":1,"name":"prom-operator-k8s07","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.23.5:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":5,"message":"Datasource added","name":"prom-operator-k8s07"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 28 Jan 2024 19:09:51 GMT', 'Content-Length': '490'}, 'raw': <urllib3.response.HTTPResponse object at 0x7f4a94670700>, 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(microseconds=34053), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x7f4a946735b0>}
01/28/2024 07:09:51 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s07 is created in Grafana
01/28/2024 07:49:28 PM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s12 is created in Grafana
01/28/2024 07:49:28 PM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":6,"uid":"d0af8ca9-211b-407a-9d2e-c37d4c32157b","orgId":1,"name":"prom-operator-k8s12","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.23.5:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":6,"message":"Datasource added","name":"prom-operator-k8s12"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 28 Jan 2024 19:49:28 GMT', 'Content-Length': '490'}, 'raw': <urllib3.response.HTTPResponse object at 0x7f4a94672440>, 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(microseconds=33019), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x7f4a946731f0>}
01/28/2024 07:49:28 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s12 is created in Grafana
01/28/2024 08:30:35 PM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana