2024-06-27T03:01:09.817022602Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:01:13.852974979Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:01:17.857403956Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:01:21.860614808Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:01:25.865104684Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:01:29.873458839Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:01:33.880629586Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:01:37.886395868Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:01:41.892960787Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:01:45.894292681Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:01:49.900210164Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:01:53.902541480Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:01:57.911394833Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:02:01.922425626Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:02:05.938509160Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:02:09.943398878Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:02:13.947037093Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:03:15.066404142Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:03:19.072728093Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:03:23.106427566Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:03:27.115882461Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:03:31.120962793Z nc: kafka (10.111.126.231:9092): Operation timed out 2024-06-27T03:04:16.326281327Z kafka (10.111.126.231:9092) open 2024-06-27T03:04:16.328624845Z mongodb-k8s (10.244.0.28:27017) open 2024-06-27T03:04:50.008909340Z 06/27/2024 03:04:50 AM - kafka.conn - INFO - [IPv4 ('10.111.126.231', 9092)]>: connecting to kafka:9092 [('10.111.126.231', 9092) IPv4] 2024-06-27T03:04:50.008959860Z 06/27/2024 03:04:50 AM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-06-27T03:04:50.008964804Z 06/27/2024 03:04:50 AM - kafka.conn - INFO - [IPv4 ('10.111.126.231', 9092)]>: Connection complete. 2024-06-27T03:04:50.153228695Z 06/27/2024 03:04:50 AM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-06-27T03:04:50.153244630Z 06/27/2024 03:04:50 AM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-06-27T03:04:50.166994984Z 06/27/2024 03:04:50 AM - kafka.conn - INFO - [IPv4 ('10.111.126.231', 9092)]>: connecting to kafka:9092 [('10.111.126.231', 9092) IPv4] 2024-06-27T03:04:50.167017952Z 06/27/2024 03:04:50 AM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-06-27T03:04:50.167022239Z 06/27/2024 03:04:50 AM - kafka.conn - INFO - [IPv4 ('10.111.126.231', 9092)]>: Connection complete. 2024-06-27T03:04:50.299028325Z 06/27/2024 03:04:50 AM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-06-27T03:04:50.299055339Z 06/27/2024 03:04:50 AM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-06-27T03:04:50.481378426Z 06/27/2024 03:04:50 AM - kafka.conn - INFO - [IPv4 ('10.244.0.41', 9092)]>: connecting to kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.41', 9092) IPv4] 2024-06-27T03:04:50.481898097Z 06/27/2024 03:04:50 AM - kafka.conn - INFO - [IPv4 ('10.244.0.41', 9092)]>: Connection complete. 2024-06-27T03:04:50.482178096Z 06/27/2024 03:04:50 AM - kafka.conn - INFO - [IPv4 ('10.111.126.231', 9092)]>: Closing connection. 2024-06-27T03:04:50.831847147Z 06/27/2024 03:04:50 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-06-27T03:04:50.965407158Z 06/27/2024 03:04:50 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-06-27T03:04:51.144991598Z 06/27/2024 03:04:51 AM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-06-27T03:04:51.248874691Z 06/27/2024 03:04:51 AM - kafka.conn - INFO - [IPv4 ('10.244.0.30', 9092)]>: connecting to kafka-controller-0.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.30', 9092) IPv4] 2024-06-27T03:04:51.249112590Z 06/27/2024 03:04:51 AM - kafka.conn - INFO - [IPv4 ('10.244.0.30', 9092)]>: Connection complete. 2024-06-27T03:04:51.361316201Z 06/27/2024 03:04:51 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-06-27T03:04:51.485190276Z 06/27/2024 03:04:51 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-06-27T03:04:51.595314742Z 06/27/2024 03:04:51 AM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-06-27T03:04:51.728453048Z 06/27/2024 03:04:51 AM - kafka.conn - INFO - [IPv4 ('10.244.0.41', 9092)]>: Closing connection. 2024-06-27T03:04:51.728876710Z 06/27/2024 03:04:51 AM - kafka.conn - INFO - [IPv4 ('10.244.0.30', 9092)]>: Closing connection. 2024-06-27T03:04:51.729262115Z 06/27/2024 03:04:51 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-06-27T03:04:51.927138693Z 06/27/2024 03:04:51 AM - db - INFO - Connected to database osm version 1.2 2024-06-27T03:04:51.927155810Z 06/27/2024 03:04:51 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON kafka Consumer... 2024-06-27T03:04:51.928872920Z 06/27/2024 03:04:51 AM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-06-27T03:04:52.019247432Z 06/27/2024 03:04:52 AM - aiokafka.consumer.subscription_state - INFO - Updating subscribed topics to: frozenset({'project', 'users'}) 2024-06-27T03:04:52.019265329Z 06/27/2024 03:04:52 AM - aiokafka.consumer.consumer - INFO - Subscribed to topic(s): {'project', 'users'} 2024-06-27T03:04:52.073220162Z 06/27/2024 03:04:52 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-27T03:04:52.220933498Z 06/27/2024 03:04:52 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-27T03:04:52.331029942Z 06/27/2024 03:04:52 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-27T03:04:52.422092276Z 06/27/2024 03:04:52 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-06-27T03:04:52.480509402Z 06/27/2024 03:04:52 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-27T03:04:52.605348868Z 06/27/2024 03:04:52 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-27T03:04:52.737344875Z 06/27/2024 03:04:52 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-27T03:04:52.874109607Z 06/27/2024 03:04:52 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-27T03:04:53.023895831Z 06/27/2024 03:04:53 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-27T03:04:53.173468744Z 06/27/2024 03:04:53 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-27T03:04:53.292847652Z 06/27/2024 03:04:53 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-27T03:04:53.473491582Z 06/27/2024 03:04:53 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-27T03:04:53.586068862Z 06/27/2024 03:04:53 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-27T03:04:53.751530191Z 06/27/2024 03:04:53 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-27T03:04:53.875542693Z 06/27/2024 03:04:53 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-27T03:04:54.032435209Z 06/27/2024 03:04:54 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-27T03:04:54.148355082Z 06/27/2024 03:04:54 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-27T03:04:54.257681837Z 06/27/2024 03:04:54 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-27T03:04:54.390185881Z 06/27/2024 03:04:54 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-27T03:04:54.507324131Z 06/27/2024 03:04:54 AM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-27T03:04:54.697392021Z 06/27/2024 03:04:54 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:54.697433970Z 06/27/2024 03:04:54 AM - aiokafka.consumer.group_coordinator - INFO - Revoking previously assigned partitions set() for group mon-consumer 2024-06-27T03:04:54.697463603Z 06/27/2024 03:04:54 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:54.717662941Z 06/27/2024 03:04:54 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:54.848485664Z 06/27/2024 03:04:54 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:54.848540052Z 06/27/2024 03:04:54 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:54.864577758Z 06/27/2024 03:04:54 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:55.019685683Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:55.019731418Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:55.023308118Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:55.133685600Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:55.133702193Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:55.136748551Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:55.245458159Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:55.246710002Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:55.261528538Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:55.365097548Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:55.365116472Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:55.378135535Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:55.512927114Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:55.512937779Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:55.524789855Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:55.685068179Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:55.685076801Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:55.685080537Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:55.818474324Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:55.818493008Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:55.922313576Z 06/27/2024 03:04:55 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:56.072366985Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:56.072386145Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:56.112859041Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:56.248579335Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:56.248590624Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:56.285525969Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:56.396067568Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:56.396094204Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:56.426588999Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:56.536108292Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:56.536117020Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:56.544897812Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:56.672926690Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:56.673121585Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:56.686662537Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:56.792808869Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:56.793625452Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:56.821090917Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:56.932252508Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:56.932275805Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:56.950141822Z 06/27/2024 03:04:56 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:57.063826636Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:57.063843240Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:57.096481688Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:57.216946020Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:57.217217384Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:57.231132615Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:57.342486896Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:57.342653892Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:57.361486551Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:57.492323513Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:57.492338465Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:57.492342166Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:57.628197127Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:57.628226767Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:57.630484390Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:57.734402322Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:57.736358595Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:57.740847287Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:57.848984053Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:57.849001282Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:57.894495014Z 06/27/2024 03:04:57 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:58.022066222Z 06/27/2024 03:04:58 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:58.022220971Z 06/27/2024 03:04:58 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:58.028133858Z 06/27/2024 03:04:58 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:58.160998706Z 06/27/2024 03:04:58 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:58.161035837Z 06/27/2024 03:04:58 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:58.184911507Z 06/27/2024 03:04:58 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:58.329262993Z 06/27/2024 03:04:58 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:58.329283481Z 06/27/2024 03:04:58 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:58.357477746Z 06/27/2024 03:04:58 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:58.459700682Z 06/27/2024 03:04:58 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:58.459713580Z 06/27/2024 03:04:58 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:58.474830917Z 06/27/2024 03:04:58 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:58.588857349Z 06/27/2024 03:04:58 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:58.590639828Z 06/27/2024 03:04:58 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:04:58.592962230Z 06/27/2024 03:04:58 AM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 2)for group mon-consumer. 2024-06-27T03:04:58.705841945Z 06/27/2024 03:04:58 AM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 2 for group mon-consumer 2024-06-27T03:04:58.705884784Z 06/27/2024 03:04:58 AM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-27T03:05:01.937472409Z 06/27/2024 03:05:01 AM - aiokafka.consumer.group_coordinator - INFO - Joined group 'mon-consumer' (generation 1) with member_id aiokafka-0.8.1-ec0e1163-1da6-4f76-be85-064856ae69da 2024-06-27T03:05:01.937492460Z 06/27/2024 03:05:01 AM - aiokafka.consumer.group_coordinator - INFO - Elected group leader -- performing partition assignments using roundrobin 2024-06-27T03:05:02.104880380Z 06/27/2024 03:05:02 AM - aiokafka.consumer.group_coordinator - INFO - Successfully synced group mon-consumer with generation 1 2024-06-27T03:05:02.104920814Z 06/27/2024 03:05:02 AM - aiokafka.consumer.group_coordinator - INFO - Setting newly assigned partitions {TopicPartition(topic='users', partition=0), TopicPartition(topic='project', partition=0)} for group mon-consumer 2024-06-27T03:12:24.592312565Z 06/27/2024 03:12:24 AM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-osm is created in Grafana 2024-06-27T03:12:24.592696492Z 06/27/2024 03:12:24 AM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":3,"uid":"a1802f1f-1f33-4b08-9f36-d1e3c0d81985","orgId":1,"name":"prom-operator-osm","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":3,"message":"Datasource added","name":"prom-operator-osm"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Thu, 27 Jun 2024 03:12:24 GMT', 'Content-Length': '489'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=54152), 'request': , 'connection': } 2024-06-27T03:12:24.779282172Z 06/27/2024 03:12:24 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard osm is created in Grafana 2024-06-27T03:18:26.254179581Z 06/27/2024 03:18:26 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-06-27T03:21:00.591392019Z 06/27/2024 03:21:00 AM - osm_mon.dashboarder.backends.grafana - INFO - New user basic_15_test_user created in Grafana 2024-06-27T03:21:03.748250832Z 06/27/2024 03:21:03 AM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user is assigned Admin permission 2024-06-27T03:21:08.428810659Z 06/27/2024 03:21:08 AM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 2024-06-27T03:21:08.428890896Z Traceback (most recent call last): 2024-06-27T03:21:08.428906039Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 116, in _user_msg 2024-06-27T03:21:08.428920113Z project_data = values["changes"]["add_project_role_mappings"] 2024-06-27T03:21:08.428932924Z KeyError: 'add_project_role_mappings' 2024-06-27T03:21:09.659503620Z 06/27/2024 03:21:09 AM - osm_mon.dashboarder.backends.grafana - INFO - New team basic_15_test_project created in Grafana 2024-06-27T03:21:11.334976137Z 06/27/2024 03:21:11 AM - osm_mon.dashboarder.backends.grafana - INFO - Grafana team updated {"message":"Team updated"} 2024-06-27T03:21:19.448594378Z 06/27/2024 03:21:19 AM - osm_mon.dashboarder.backends.grafana - INFO - User basic_15_test_user deleted in Grafana 2024-06-27T03:21:21.011607170Z 06/27/2024 03:21:21 AM - osm_mon.dashboarder.backends.grafana - INFO - Team basic_15_project_test deleted in Grafana 2024-06-27T03:21:57.249714061Z 06/27/2024 03:21:57 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-06-27T04:30:42.410218263Z 06/27/2024 04:30:42 AM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s02 is created in Grafana 2024-06-27T04:30:42.410540366Z 06/27/2024 04:30:42 AM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":4,"uid":"acc9e8b7-99c7-4c68-9c9f-15776d9d174b","orgId":1,"name":"prom-operator-k8s02","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":4,"message":"Datasource added","name":"prom-operator-k8s02"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Thu, 27 Jun 2024 04:30:42 GMT', 'Content-Length': '493'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=45489), 'request': , 'connection': } 2024-06-27T04:30:42.580656088Z 06/27/2024 04:30:42 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s02 is created in Grafana 2024-06-27T04:34:43.625053763Z 06/27/2024 04:34:43 AM - osm_mon.dashboarder.backends.grafana - INFO - Datasource prom-operator-k8s07 is created in Grafana 2024-06-27T04:34:43.625412794Z 06/27/2024 04:34:43 AM - osm_mon.dashboarder.backends.grafana - INFO - ************* response: {'_content': b'{"datasource":{"id":5,"uid":"b1008d84-8cfb-4b24-90c3-aa0ad48f6a4e","orgId":1,"name":"prom-operator-k8s07","type":"prometheus","typeLogoUrl":"public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://172.21.248.125:9090","user":"","database":"","basicAuth":false,"basicAuthUser":"","withCredentials":false,"isDefault":false,"jsonData":{},"secureJsonFields":{},"version":1,"readOnly":false},"id":5,"message":"Datasource added","name":"prom-operator-k8s07"}', '_content_consumed': True, '_next': None, 'status_code': 200, 'headers': {'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'deny', 'X-Xss-Protection': '1; mode=block', 'Date': 'Thu, 27 Jun 2024 04:34:43 GMT', 'Content-Length': '493'}, 'raw': , 'url': 'http://grafana:3000/api/datasources', 'encoding': 'utf-8', 'history': [], 'reason': 'OK', 'cookies': , 'elapsed': datetime.timedelta(microseconds=33988), 'request': , 'connection': } 2024-06-27T04:34:43.778734550Z 06/27/2024 04:34:43 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard k8s07 is created in Grafana 2024-06-27T04:42:59.226626501Z 06/27/2024 04:42:59 AM - osm_mon.dashboarder.backends.grafana - INFO - New team quotas_01_proj_1 created in Grafana 2024-06-27T04:43:05.462668990Z 06/27/2024 04:43:05 AM - osm_mon.dashboarder.backends.grafana - INFO - New team quotas_01_proj_2 created in Grafana 2024-06-27T04:43:11.753448494Z 06/27/2024 04:43:11 AM - osm_mon.dashboarder.backends.grafana - INFO - New user quotas_01_user created in Grafana 2024-06-27T04:43:11.966821568Z 06/27/2024 04:43:11 AM - osm_mon.dashboarder.backends.grafana - INFO - User quotas_01_user is assigned Editor permission 2024-06-27T04:43:12.137783545Z 06/27/2024 04:43:12 AM - osm_mon.dashboarder.backends.grafana - INFO - User quotas_01_user is assigned Editor permission 2024-06-27T04:43:15.739776884Z 06/27/2024 04:43:15 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard folder quotas_01_proj_1 created 2024-06-27T04:43:15.939092835Z 06/27/2024 04:43:15 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard quotas_01_proj_1 is created in Grafana 2024-06-27T04:43:15.984833710Z 06/27/2024 04:43:15 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard folder quotas_01_proj_2 created 2024-06-27T04:43:16.213437139Z 06/27/2024 04:43:16 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard quotas_01_proj_2 is created in Grafana 2024-06-27T04:43:16.482948861Z 06/27/2024 04:43:16 AM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 2024-06-27T04:43:16.482983958Z Traceback (most recent call last): 2024-06-27T04:43:16.482987512Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 116, in _user_msg 2024-06-27T04:43:16.482991617Z project_data = values["changes"]["add_project_role_mappings"] 2024-06-27T04:43:16.482994444Z KeyError: 'add_project_role_mappings' 2024-06-27T04:43:18.240800585Z 06/27/2024 04:43:18 AM - osm_mon.dashboarder.backends.grafana - INFO - Grafana team updated {"message":"Team updated"} 2024-06-27T04:43:27.831738798Z 06/27/2024 04:43:27 AM - osm_mon.dashboarder.dashboarder - ERROR - Exception processing message: 2024-06-27T04:43:27.831782141Z Traceback (most recent call last): 2024-06-27T04:43:27.831792916Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 129, in _user_msg 2024-06-27T04:43:27.831801641Z project_new_name = values["changes"]["name"] 2024-06-27T04:43:27.831810270Z KeyError: 'name' 2024-06-27T04:43:57.342399391Z 06/27/2024 04:43:57 AM - osm_mon.dashboarder.backends.grafana - INFO - Team quotas_01_proj_2 deleted in Grafana 2024-06-27T04:43:59.021237302Z 06/27/2024 04:43:59 AM - osm_mon.dashboarder.backends.grafana - INFO - User quotas_01_user deleted in Grafana 2024-06-27T04:44:00.597052911Z 06/27/2024 04:44:00 AM - osm_mon.dashboarder.backends.grafana - INFO - Team quotas_01_proj_3 deleted in Grafana 2024-06-27T04:44:16.612306209Z 06/27/2024 04:44:16 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-06-27T04:51:48.530188019Z 06/27/2024 04:51:48 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-06-27T05:07:22.243728059Z 06/27/2024 05:07:22 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana 2024-06-27T05:32:58.225213442Z 06/27/2024 05:32:58 AM - osm_mon.dashboarder.backends.grafana - INFO - Dashboard admin is created in Grafana