2024-06-24T22:06:32.110944043Z nc: kafka (10.105.117.166:9092): Operation timed out 2024-06-24T22:06:36.116532950Z nc: kafka (10.105.117.166:9092): Operation timed out 2024-06-24T22:06:40.118697563Z nc: kafka (10.105.117.166:9092): Operation timed out 2024-06-24T22:06:44.123339638Z nc: kafka (10.105.117.166:9092): Operation timed out 2024-06-24T22:06:48.128046539Z nc: kafka (10.105.117.166:9092): Operation timed out 2024-06-24T22:06:52.134057687Z nc: kafka (10.105.117.166:9092): Operation timed out 2024-06-24T22:06:56.140992188Z nc: kafka (10.105.117.166:9092): Operation timed out 2024-06-24T22:07:00.146958829Z nc: kafka (10.105.117.166:9092): Operation timed out 2024-06-24T22:07:04.153365388Z nc: kafka (10.105.117.166:9092): Operation timed out 2024-06-24T22:07:08.165213488Z nc: kafka (10.105.117.166:9092): Operation timed out 2024-06-24T22:07:12.185181310Z nc: kafka (10.105.117.166:9092): Operation timed out 2024-06-24T22:07:16.190026611Z nc: kafka (10.105.117.166:9092): Operation timed out 2024-06-24T22:08:28.428470653Z kafka (10.105.117.166:9092) open 2024-06-24T22:08:28.436454814Z mongodb-k8s (10.244.0.29:27017) open 2024-06-24T22:08:59.678680307Z 06/24/2024 10:08:59 PM - kafka.conn - INFO - [IPv4 ('10.105.117.166', 9092)]>: connecting to kafka:9092 [('10.105.117.166', 9092) IPv4] 2024-06-24T22:08:59.678710972Z 06/24/2024 10:08:59 PM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-06-24T22:08:59.678722634Z 06/24/2024 10:08:59 PM - kafka.conn - INFO - [IPv4 ('10.105.117.166', 9092)]>: Connection complete. 2024-06-24T22:08:59.887025127Z 06/24/2024 10:08:59 PM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-06-24T22:08:59.887045970Z 06/24/2024 10:08:59 PM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-06-24T22:08:59.956652820Z 06/24/2024 10:08:59 PM - kafka.conn - INFO - [IPv4 ('10.105.117.166', 9092)]>: connecting to kafka:9092 [('10.105.117.166', 9092) IPv4] 2024-06-24T22:08:59.956674291Z 06/24/2024 10:08:59 PM - kafka.conn - INFO - Probing node bootstrap-0 broker version 2024-06-24T22:08:59.957080463Z 06/24/2024 10:08:59 PM - kafka.conn - INFO - [IPv4 ('10.105.117.166', 9092)]>: Connection complete. 2024-06-24T22:09:00.068442453Z 06/24/2024 10:09:00 PM - kafka.conn - INFO - Broker version identified as 2.5.0 2024-06-24T22:09:00.068459194Z 06/24/2024 10:09:00 PM - kafka.conn - INFO - Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup 2024-06-24T22:09:00.227562144Z 06/24/2024 10:09:00 PM - kafka.conn - INFO - [IPv4 ('10.244.0.45', 9092)]>: connecting to kafka-controller-2.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.45', 9092) IPv4] 2024-06-24T22:09:00.228528976Z 06/24/2024 10:09:00 PM - kafka.conn - INFO - [IPv4 ('10.244.0.45', 9092)]>: Connection complete. 2024-06-24T22:09:00.228543247Z 06/24/2024 10:09:00 PM - kafka.conn - INFO - [IPv4 ('10.105.117.166', 9092)]>: Closing connection. 2024-06-24T22:09:01.313027416Z 06/24/2024 10:09:01 PM - kafka.conn - INFO - [IPv4 ('10.244.0.47', 9092)]>: connecting to kafka-controller-1.kafka-controller-headless.osm.svc.cluster.local:9092 [('10.244.0.47', 9092) IPv4] 2024-06-24T22:09:01.313201211Z 06/24/2024 10:09:01 PM - kafka.conn - INFO - [IPv4 ('10.244.0.47', 9092)]>: Connection complete. 2024-06-24T22:09:01.424792410Z 06/24/2024 10:09:01 PM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-06-24T22:09:01.536200063Z 06/24/2024 10:09:01 PM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-06-24T22:09:01.696053344Z 06/24/2024 10:09:01 PM - kafka.cluster - ERROR - Topic users not found in cluster metadata 2024-06-24T22:09:02.088995638Z 06/24/2024 10:09:02 PM - kafka.cluster - ERROR - Topic project not found in cluster metadata 2024-06-24T22:09:02.362541129Z 06/24/2024 10:09:02 PM - kafka.producer.record_accumulator - WARNING - Produced messages to topic-partition TopicPartition(topic='project', partition=0) with base offset -1 log start offset -1 and error None. 2024-06-24T22:09:02.362550765Z 06/24/2024 10:09:02 PM - kafka.conn - INFO - [IPv4 ('10.244.0.45', 9092)]>: Closing connection. 2024-06-24T22:09:02.362566533Z 06/24/2024 10:09:02 PM - kafka.conn - INFO - [IPv4 ('10.244.0.47', 9092)]>: Closing connection. 2024-06-24T22:09:02.362570357Z 06/24/2024 10:09:02 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-06-24T22:09:02.515216917Z 06/24/2024 10:09:02 PM - db - INFO - Connected to database osm version None 2024-06-24T22:09:02.515231684Z 06/24/2024 10:09:02 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON kafka Consumer... 2024-06-24T22:09:02.520743371Z 06/24/2024 10:09:02 PM - osm_mon.cmd.mon_dashboarder - INFO - Starting MON Dashboarder... 2024-06-24T22:09:02.711362244Z 06/24/2024 10:09:02 PM - aiokafka.consumer.subscription_state - INFO - Updating subscribed topics to: frozenset({'users', 'project'}) 2024-06-24T22:09:02.711423233Z 06/24/2024 10:09:02 PM - aiokafka.consumer.consumer - INFO - Subscribed to topic(s): {'users', 'project'} 2024-06-24T22:09:02.843400549Z 06/24/2024 10:09:02 PM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-24T22:09:02.969754052Z 06/24/2024 10:09:02 PM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-24T22:09:03.101318453Z 06/24/2024 10:09:03 PM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-24T22:09:03.277458316Z 06/24/2024 10:09:03 PM - aiokafka.consumer.group_coordinator - ERROR - Group Coordinator Request failed: [Error 15] CoordinatorNotAvailableError 2024-06-24T22:09:03.511288692Z 06/24/2024 10:09:03 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:03.511306167Z 06/24/2024 10:09:03 PM - aiokafka.consumer.group_coordinator - INFO - Revoking previously assigned partitions set() for group mon-consumer 2024-06-24T22:09:03.511310319Z 06/24/2024 10:09:03 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:03.608059679Z 06/24/2024 10:09:03 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:03.833411775Z 06/24/2024 10:09:03 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:03.833433212Z 06/24/2024 10:09:03 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:03.851591173Z 06/24/2024 10:09:03 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:03.960594838Z 06/24/2024 10:09:03 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:03.960731661Z 06/24/2024 10:09:03 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:03.970724381Z 06/24/2024 10:09:03 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:04.193615951Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:04.193658277Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:04.200069051Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:04.305572130Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:04.305588586Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:04.316813714Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:04.417470450Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:04.417807346Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:04.440363969Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:04.563265631Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:04.563292008Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:04.576396386Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:04.710156813Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:04.710171822Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:04.754402694Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:04.865059154Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:04.865094274Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:04.868204695Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:04.978165727Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:04.990122486Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:04.990143205Z 06/24/2024 10:09:04 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:05.108637881Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:05.108669415Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:05.120409633Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:05.233544777Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:05.233562247Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:05.237074453Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:05.342029451Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:05.342867993Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:05.357168443Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:05.484812539Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:05.485367474Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:05.489929577Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:05.599758733Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:05.600578466Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:05.609657863Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:05.751532580Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:05.751737415Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:05.785981340Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:05.885927272Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:05.886118142Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:05.908644033Z 06/24/2024 10:09:05 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:06.017771855Z 06/24/2024 10:09:06 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:06.018032974Z 06/24/2024 10:09:06 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:06.026732452Z 06/24/2024 10:09:06 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:06.154971186Z 06/24/2024 10:09:06 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:06.154986747Z 06/24/2024 10:09:06 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:06.172608164Z 06/24/2024 10:09:06 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:06.295540409Z 06/24/2024 10:09:06 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:06.295554799Z 06/24/2024 10:09:06 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:06.316452786Z 06/24/2024 10:09:06 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:06.423348787Z 06/24/2024 10:09:06 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:06.423426730Z 06/24/2024 10:09:06 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:06.454741232Z 06/24/2024 10:09:06 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:06.585723710Z 06/24/2024 10:09:06 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:06.585747766Z 06/24/2024 10:09:06 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:06.637879265Z 06/24/2024 10:09:06 PM - aiokafka.consumer.group_coordinator - WARNING - Marking the coordinator dead (node 1)for group mon-consumer. 2024-06-24T22:09:06.732702891Z 06/24/2024 10:09:06 PM - aiokafka.consumer.group_coordinator - INFO - Discovered coordinator 1 for group mon-consumer 2024-06-24T22:09:06.732712119Z 06/24/2024 10:09:06 PM - aiokafka.consumer.group_coordinator - INFO - (Re-)joining group mon-consumer 2024-06-24T22:09:09.944402976Z 06/24/2024 10:09:09 PM - aiokafka.consumer.group_coordinator - INFO - Joined group 'mon-consumer' (generation 1) with member_id aiokafka-0.8.1-d5b0b7c3-ae51-4af2-8680-3088c29691b7 2024-06-24T22:09:09.944429261Z 06/24/2024 10:09:09 PM - aiokafka.consumer.group_coordinator - INFO - Elected group leader -- performing partition assignments using roundrobin 2024-06-24T22:09:10.120851314Z 06/24/2024 10:09:10 PM - aiokafka.consumer.group_coordinator - INFO - Successfully synced group mon-consumer with generation 1 2024-06-24T22:09:10.121204940Z 06/24/2024 10:09:10 PM - aiokafka.consumer.group_coordinator - INFO - Setting newly assigned partitions {TopicPartition(topic='users', partition=0), TopicPartition(topic='project', partition=0)} for group mon-consumer 2024-06-24T22:11:13.417297108Z 06/24/2024 10:11:13 PM - osm_mon.dashboarder.dashboarder - ERROR - Error creating dashboards 2024-06-24T22:11:13.417337731Z Traceback (most recent call last): 2024-06-24T22:11:13.417342070Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn 2024-06-24T22:11:13.417345891Z sock = connection.create_connection( 2024-06-24T22:11:13.417349666Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection 2024-06-24T22:11:13.417353176Z raise err 2024-06-24T22:11:13.417356285Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection 2024-06-24T22:11:13.417360021Z sock.connect(sa) 2024-06-24T22:11:13.417363142Z TimeoutError: [Errno 110] Connection timed out 2024-06-24T22:11:13.417366057Z 2024-06-24T22:11:13.417369847Z The above exception was the direct cause of the following exception: 2024-06-24T22:11:13.417372828Z 2024-06-24T22:11:13.417376082Z Traceback (most recent call last): 2024-06-24T22:11:13.417379276Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 790, in urlopen 2024-06-24T22:11:13.417382537Z response = self._make_request( 2024-06-24T22:11:13.417385689Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 496, in _make_request 2024-06-24T22:11:13.417388759Z conn.request( 2024-06-24T22:11:13.417392259Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request 2024-06-24T22:11:13.417395460Z self.endheaders() 2024-06-24T22:11:13.417398787Z File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders 2024-06-24T22:11:13.417401818Z self._send_output(message_body, encode_chunked=encode_chunked) 2024-06-24T22:11:13.417404945Z File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output 2024-06-24T22:11:13.417407996Z self.send(msg) 2024-06-24T22:11:13.417411517Z File "/usr/lib/python3.10/http/client.py", line 976, in send 2024-06-24T22:11:13.417414514Z self.connect() 2024-06-24T22:11:13.417417783Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect 2024-06-24T22:11:13.417421393Z self.sock = self._new_conn() 2024-06-24T22:11:13.417424412Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 212, in _new_conn 2024-06-24T22:11:13.417427473Z raise ConnectTimeoutError( 2024-06-24T22:11:13.417431429Z urllib3.exceptions.ConnectTimeoutError: (, 'Connection to grafana timed out. (connect timeout=None)') 2024-06-24T22:11:13.417442035Z 2024-06-24T22:11:13.417445274Z The above exception was the direct cause of the following exception: 2024-06-24T22:11:13.417448260Z 2024-06-24T22:11:13.417451294Z Traceback (most recent call last): 2024-06-24T22:11:13.417454234Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send 2024-06-24T22:11:13.417457284Z resp = conn.urlopen( 2024-06-24T22:11:13.417460296Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 844, in urlopen 2024-06-24T22:11:13.417474244Z retries = retries.increment( 2024-06-24T22:11:13.417477631Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment 2024-06-24T22:11:13.417480644Z raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] 2024-06-24T22:11:13.417484275Z urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:11:13.417487130Z 2024-06-24T22:11:13.417490102Z During handling of the above exception, another exception occurred: 2024-06-24T22:11:13.417492833Z 2024-06-24T22:11:13.417495867Z Traceback (most recent call last): 2024-06-24T22:11:13.417498895Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 147, in dashboard_forever 2024-06-24T22:11:13.417501829Z self.create_dashboards() 2024-06-24T22:11:13.417504707Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 153, in create_dashboards 2024-06-24T22:11:13.417507696Z self.service.create_dashboards() 2024-06-24T22:11:13.417510636Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/service.py", line 52, in create_dashboards 2024-06-24T22:11:13.417513566Z dashboard_uids = self.grafana.get_all_dashboard_uids() 2024-06-24T22:11:13.417516944Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/backends/grafana.py", line 48, in get_all_dashboard_uids 2024-06-24T22:11:13.417519793Z response = requests.request( 2024-06-24T22:11:13.417523041Z File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request 2024-06-24T22:11:13.417526001Z return session.request(method=method, url=url, **kwargs) 2024-06-24T22:11:13.417528921Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request 2024-06-24T22:11:13.417531949Z resp = self.send(prep, **send_kwargs) 2024-06-24T22:11:13.417534795Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send 2024-06-24T22:11:13.417537802Z r = adapter.send(request, **kwargs) 2024-06-24T22:11:13.417540639Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 507, in send 2024-06-24T22:11:13.417543611Z raise ConnectTimeout(e, request=request) 2024-06-24T22:11:13.417559064Z requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:13:24.479369506Z 06/24/2024 10:13:24 PM - osm_mon.dashboarder.dashboarder - ERROR - Error creating dashboards 2024-06-24T22:13:24.479385218Z Traceback (most recent call last): 2024-06-24T22:13:24.479388469Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn 2024-06-24T22:13:24.479402373Z sock = connection.create_connection( 2024-06-24T22:13:24.479442805Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection 2024-06-24T22:13:24.479446833Z raise err 2024-06-24T22:13:24.479449870Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection 2024-06-24T22:13:24.479453346Z sock.connect(sa) 2024-06-24T22:13:24.479456355Z TimeoutError: [Errno 110] Connection timed out 2024-06-24T22:13:24.479459087Z 2024-06-24T22:13:24.479462375Z The above exception was the direct cause of the following exception: 2024-06-24T22:13:24.479465124Z 2024-06-24T22:13:24.479468075Z Traceback (most recent call last): 2024-06-24T22:13:24.479471115Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 790, in urlopen 2024-06-24T22:13:24.479474011Z response = self._make_request( 2024-06-24T22:13:24.479476906Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 496, in _make_request 2024-06-24T22:13:24.479479822Z conn.request( 2024-06-24T22:13:24.479482761Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request 2024-06-24T22:13:24.479485595Z self.endheaders() 2024-06-24T22:13:24.479489080Z File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders 2024-06-24T22:13:24.479492025Z self._send_output(message_body, encode_chunked=encode_chunked) 2024-06-24T22:13:24.479495006Z File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output 2024-06-24T22:13:24.479497931Z self.send(msg) 2024-06-24T22:13:24.479501304Z File "/usr/lib/python3.10/http/client.py", line 976, in send 2024-06-24T22:13:24.479504528Z self.connect() 2024-06-24T22:13:24.479507421Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect 2024-06-24T22:13:24.479510888Z self.sock = self._new_conn() 2024-06-24T22:13:24.479514068Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 212, in _new_conn 2024-06-24T22:13:24.479517004Z raise ConnectTimeoutError( 2024-06-24T22:13:24.479520531Z urllib3.exceptions.ConnectTimeoutError: (, 'Connection to grafana timed out. (connect timeout=None)') 2024-06-24T22:13:24.479523256Z 2024-06-24T22:13:24.479526197Z The above exception was the direct cause of the following exception: 2024-06-24T22:13:24.479528909Z 2024-06-24T22:13:24.479531880Z Traceback (most recent call last): 2024-06-24T22:13:24.479534741Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send 2024-06-24T22:13:24.479537637Z resp = conn.urlopen( 2024-06-24T22:13:24.479540517Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 844, in urlopen 2024-06-24T22:13:24.479543515Z retries = retries.increment( 2024-06-24T22:13:24.479550185Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment 2024-06-24T22:13:24.479553133Z raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] 2024-06-24T22:13:24.479556761Z urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:13:24.479559581Z 2024-06-24T22:13:24.479562558Z During handling of the above exception, another exception occurred: 2024-06-24T22:13:24.479570302Z 2024-06-24T22:13:24.479578136Z Traceback (most recent call last): 2024-06-24T22:13:24.479581276Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 147, in dashboard_forever 2024-06-24T22:13:24.479584166Z self.create_dashboards() 2024-06-24T22:13:24.479587044Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 153, in create_dashboards 2024-06-24T22:13:24.479589913Z self.service.create_dashboards() 2024-06-24T22:13:24.479592938Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/service.py", line 52, in create_dashboards 2024-06-24T22:13:24.479595799Z dashboard_uids = self.grafana.get_all_dashboard_uids() 2024-06-24T22:13:24.479602622Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/backends/grafana.py", line 48, in get_all_dashboard_uids 2024-06-24T22:13:24.479605706Z response = requests.request( 2024-06-24T22:13:24.479608606Z File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request 2024-06-24T22:13:24.479611489Z return session.request(method=method, url=url, **kwargs) 2024-06-24T22:13:24.479614442Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request 2024-06-24T22:13:24.479617299Z resp = self.send(prep, **send_kwargs) 2024-06-24T22:13:24.479621061Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send 2024-06-24T22:13:24.479623967Z r = adapter.send(request, **kwargs) 2024-06-24T22:13:24.479626855Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 507, in send 2024-06-24T22:13:24.479629757Z raise ConnectTimeout(e, request=request) 2024-06-24T22:13:24.479635116Z requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:15:35.550592951Z 06/24/2024 10:15:35 PM - osm_mon.dashboarder.dashboarder - ERROR - Error creating dashboards 2024-06-24T22:15:35.550623726Z Traceback (most recent call last): 2024-06-24T22:15:35.550627238Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn 2024-06-24T22:15:35.550630285Z sock = connection.create_connection( 2024-06-24T22:15:35.550634118Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection 2024-06-24T22:15:35.550637180Z raise err 2024-06-24T22:15:35.550640050Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection 2024-06-24T22:15:35.550643430Z sock.connect(sa) 2024-06-24T22:15:35.550646209Z TimeoutError: [Errno 110] Connection timed out 2024-06-24T22:15:35.550648831Z 2024-06-24T22:15:35.550652197Z The above exception was the direct cause of the following exception: 2024-06-24T22:15:35.550654777Z 2024-06-24T22:15:35.550657552Z Traceback (most recent call last): 2024-06-24T22:15:35.550666579Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 790, in urlopen 2024-06-24T22:15:35.550669519Z response = self._make_request( 2024-06-24T22:15:35.550672434Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 496, in _make_request 2024-06-24T22:15:35.550675208Z conn.request( 2024-06-24T22:15:35.550689728Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request 2024-06-24T22:15:35.550692681Z self.endheaders() 2024-06-24T22:15:35.550695865Z File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders 2024-06-24T22:15:35.550724952Z self._send_output(message_body, encode_chunked=encode_chunked) 2024-06-24T22:15:35.550728105Z File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output 2024-06-24T22:15:35.550730939Z self.send(msg) 2024-06-24T22:15:35.550734467Z File "/usr/lib/python3.10/http/client.py", line 976, in send 2024-06-24T22:15:35.550737331Z self.connect() 2024-06-24T22:15:35.550740573Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect 2024-06-24T22:15:35.550743986Z self.sock = self._new_conn() 2024-06-24T22:15:35.550746959Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 212, in _new_conn 2024-06-24T22:15:35.550749871Z raise ConnectTimeoutError( 2024-06-24T22:15:35.550757943Z urllib3.exceptions.ConnectTimeoutError: (, 'Connection to grafana timed out. (connect timeout=None)') 2024-06-24T22:15:35.550760795Z 2024-06-24T22:15:35.550763820Z The above exception was the direct cause of the following exception: 2024-06-24T22:15:35.550766582Z 2024-06-24T22:15:35.550769642Z Traceback (most recent call last): 2024-06-24T22:15:35.550772533Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send 2024-06-24T22:15:35.550775466Z resp = conn.urlopen( 2024-06-24T22:15:35.550778463Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 844, in urlopen 2024-06-24T22:15:35.550781365Z retries = retries.increment( 2024-06-24T22:15:35.550784286Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment 2024-06-24T22:15:35.550787200Z raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] 2024-06-24T22:15:35.550790720Z urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:15:35.550793513Z 2024-06-24T22:15:35.550796462Z During handling of the above exception, another exception occurred: 2024-06-24T22:15:35.550799184Z 2024-06-24T22:15:35.550802026Z Traceback (most recent call last): 2024-06-24T22:15:35.550805113Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 147, in dashboard_forever 2024-06-24T22:15:35.550807981Z self.create_dashboards() 2024-06-24T22:15:35.550810890Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 153, in create_dashboards 2024-06-24T22:15:35.550814047Z self.service.create_dashboards() 2024-06-24T22:15:35.550817279Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/service.py", line 52, in create_dashboards 2024-06-24T22:15:35.550820394Z dashboard_uids = self.grafana.get_all_dashboard_uids() 2024-06-24T22:15:35.550823875Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/backends/grafana.py", line 48, in get_all_dashboard_uids 2024-06-24T22:15:35.550826761Z response = requests.request( 2024-06-24T22:15:35.550829856Z File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request 2024-06-24T22:15:35.550832763Z return session.request(method=method, url=url, **kwargs) 2024-06-24T22:15:35.550835681Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request 2024-06-24T22:15:35.550838709Z resp = self.send(prep, **send_kwargs) 2024-06-24T22:15:35.550841613Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send 2024-06-24T22:15:35.550848482Z r = adapter.send(request, **kwargs) 2024-06-24T22:15:35.550851544Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 507, in send 2024-06-24T22:15:35.550854399Z raise ConnectTimeout(e, request=request) 2024-06-24T22:15:35.550859690Z requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:17:46.622926871Z 06/24/2024 10:17:46 PM - osm_mon.dashboarder.dashboarder - ERROR - Error creating dashboards 2024-06-24T22:17:46.622949382Z Traceback (most recent call last): 2024-06-24T22:17:46.622953685Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn 2024-06-24T22:17:46.622957709Z sock = connection.create_connection( 2024-06-24T22:17:46.622961847Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection 2024-06-24T22:17:46.622965850Z raise err 2024-06-24T22:17:46.622969725Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection 2024-06-24T22:17:46.622973899Z sock.connect(sa) 2024-06-24T22:17:46.622977608Z TimeoutError: [Errno 110] Connection timed out 2024-06-24T22:17:46.622981102Z 2024-06-24T22:17:46.622985693Z The above exception was the direct cause of the following exception: 2024-06-24T22:17:46.622989211Z 2024-06-24T22:17:46.622992960Z Traceback (most recent call last): 2024-06-24T22:17:46.622996735Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 790, in urlopen 2024-06-24T22:17:46.623000785Z response = self._make_request( 2024-06-24T22:17:46.623004560Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 496, in _make_request 2024-06-24T22:17:46.623008216Z conn.request( 2024-06-24T22:17:46.623012143Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request 2024-06-24T22:17:46.623015779Z self.endheaders() 2024-06-24T22:17:46.623019526Z File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders 2024-06-24T22:17:46.623023137Z self._send_output(message_body, encode_chunked=encode_chunked) 2024-06-24T22:17:46.623026775Z File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output 2024-06-24T22:17:46.623030357Z self.send(msg) 2024-06-24T22:17:46.623034480Z File "/usr/lib/python3.10/http/client.py", line 976, in send 2024-06-24T22:17:46.623038100Z self.connect() 2024-06-24T22:17:46.623041861Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect 2024-06-24T22:17:46.623046059Z self.sock = self._new_conn() 2024-06-24T22:17:46.623049795Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 212, in _new_conn 2024-06-24T22:17:46.623053572Z raise ConnectTimeoutError( 2024-06-24T22:17:46.623061482Z urllib3.exceptions.ConnectTimeoutError: (, 'Connection to grafana timed out. (connect timeout=None)') 2024-06-24T22:17:46.623065159Z 2024-06-24T22:17:46.623069192Z The above exception was the direct cause of the following exception: 2024-06-24T22:17:46.623072727Z 2024-06-24T22:17:46.623076619Z Traceback (most recent call last): 2024-06-24T22:17:46.623080468Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send 2024-06-24T22:17:46.623084543Z resp = conn.urlopen( 2024-06-24T22:17:46.623088245Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 844, in urlopen 2024-06-24T22:17:46.623101355Z retries = retries.increment( 2024-06-24T22:17:46.623105259Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment 2024-06-24T22:17:46.623108896Z raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] 2024-06-24T22:17:46.623113419Z urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:17:46.623116819Z 2024-06-24T22:17:46.623120601Z During handling of the above exception, another exception occurred: 2024-06-24T22:17:46.623123975Z 2024-06-24T22:17:46.623127582Z Traceback (most recent call last): 2024-06-24T22:17:46.623131235Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 147, in dashboard_forever 2024-06-24T22:17:46.623134767Z self.create_dashboards() 2024-06-24T22:17:46.623138354Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 153, in create_dashboards 2024-06-24T22:17:46.623142056Z self.service.create_dashboards() 2024-06-24T22:17:46.623145653Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/service.py", line 52, in create_dashboards 2024-06-24T22:17:46.623149232Z dashboard_uids = self.grafana.get_all_dashboard_uids() 2024-06-24T22:17:46.623153600Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/backends/grafana.py", line 48, in get_all_dashboard_uids 2024-06-24T22:17:46.623157245Z response = requests.request( 2024-06-24T22:17:46.623161125Z File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request 2024-06-24T22:17:46.623164787Z return session.request(method=method, url=url, **kwargs) 2024-06-24T22:17:46.623168413Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request 2024-06-24T22:17:46.623172054Z resp = self.send(prep, **send_kwargs) 2024-06-24T22:17:46.623175649Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send 2024-06-24T22:17:46.623179611Z r = adapter.send(request, **kwargs) 2024-06-24T22:17:46.623183235Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 507, in send 2024-06-24T22:17:46.623186837Z raise ConnectTimeout(e, request=request) 2024-06-24T22:17:46.623192781Z requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:19:57.695615051Z 06/24/2024 10:19:57 PM - osm_mon.dashboarder.dashboarder - ERROR - Error creating dashboards 2024-06-24T22:19:57.695652667Z Traceback (most recent call last): 2024-06-24T22:19:57.695657762Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn 2024-06-24T22:19:57.695662763Z sock = connection.create_connection( 2024-06-24T22:19:57.695669524Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection 2024-06-24T22:19:57.695674635Z raise err 2024-06-24T22:19:57.695678895Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection 2024-06-24T22:19:57.695683946Z sock.connect(sa) 2024-06-24T22:19:57.695688318Z TimeoutError: [Errno 110] Connection timed out 2024-06-24T22:19:57.695692653Z 2024-06-24T22:19:57.695698420Z The above exception was the direct cause of the following exception: 2024-06-24T22:19:57.695713899Z 2024-06-24T22:19:57.695718883Z Traceback (most recent call last): 2024-06-24T22:19:57.695723222Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 790, in urlopen 2024-06-24T22:19:57.695727661Z response = self._make_request( 2024-06-24T22:19:57.695732017Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 496, in _make_request 2024-06-24T22:19:57.695736238Z conn.request( 2024-06-24T22:19:57.695740699Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request 2024-06-24T22:19:57.695745039Z self.endheaders() 2024-06-24T22:19:57.695749204Z File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders 2024-06-24T22:19:57.695753642Z self._send_output(message_body, encode_chunked=encode_chunked) 2024-06-24T22:19:57.695757932Z File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output 2024-06-24T22:19:57.695762213Z self.send(msg) 2024-06-24T22:19:57.695766951Z File "/usr/lib/python3.10/http/client.py", line 976, in send 2024-06-24T22:19:57.695771338Z self.connect() 2024-06-24T22:19:57.695776028Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect 2024-06-24T22:19:57.695781329Z self.sock = self._new_conn() 2024-06-24T22:19:57.695786030Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 212, in _new_conn 2024-06-24T22:19:57.695792900Z raise ConnectTimeoutError( 2024-06-24T22:19:57.695802096Z urllib3.exceptions.ConnectTimeoutError: (, 'Connection to grafana timed out. (connect timeout=None)') 2024-06-24T22:19:57.695807029Z 2024-06-24T22:19:57.695812090Z The above exception was the direct cause of the following exception: 2024-06-24T22:19:57.695816005Z 2024-06-24T22:19:57.695820833Z Traceback (most recent call last): 2024-06-24T22:19:57.695826081Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send 2024-06-24T22:19:57.695833041Z resp = conn.urlopen( 2024-06-24T22:19:57.695840414Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 844, in urlopen 2024-06-24T22:19:57.695845078Z retries = retries.increment( 2024-06-24T22:19:57.695849389Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment 2024-06-24T22:19:57.695853868Z raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] 2024-06-24T22:19:57.695859982Z urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:19:57.695864266Z 2024-06-24T22:19:57.695868787Z During handling of the above exception, another exception occurred: 2024-06-24T22:19:57.695872752Z 2024-06-24T22:19:57.695877098Z Traceback (most recent call last): 2024-06-24T22:19:57.695881373Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 147, in dashboard_forever 2024-06-24T22:19:57.695885668Z self.create_dashboards() 2024-06-24T22:19:57.695889989Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 153, in create_dashboards 2024-06-24T22:19:57.695894701Z self.service.create_dashboards() 2024-06-24T22:19:57.695898948Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/service.py", line 52, in create_dashboards 2024-06-24T22:19:57.695903197Z dashboard_uids = self.grafana.get_all_dashboard_uids() 2024-06-24T22:19:57.695916822Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/backends/grafana.py", line 48, in get_all_dashboard_uids 2024-06-24T22:19:57.695921308Z response = requests.request( 2024-06-24T22:19:57.695925453Z File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request 2024-06-24T22:19:57.695929892Z return session.request(method=method, url=url, **kwargs) 2024-06-24T22:19:57.695934271Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request 2024-06-24T22:19:57.695939269Z resp = self.send(prep, **send_kwargs) 2024-06-24T22:19:57.695944148Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send 2024-06-24T22:19:57.695948746Z r = adapter.send(request, **kwargs) 2024-06-24T22:19:57.695953970Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 507, in send 2024-06-24T22:19:57.695958621Z raise ConnectTimeout(e, request=request) 2024-06-24T22:19:57.695981826Z requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:22:08.772918795Z 06/24/2024 10:22:08 PM - osm_mon.dashboarder.dashboarder - ERROR - Error creating dashboards 2024-06-24T22:22:08.772969266Z Traceback (most recent call last): 2024-06-24T22:22:08.772974387Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn 2024-06-24T22:22:08.772978637Z sock = connection.create_connection( 2024-06-24T22:22:08.772983738Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection 2024-06-24T22:22:08.772988543Z raise err 2024-06-24T22:22:08.772992845Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection 2024-06-24T22:22:08.772998354Z sock.connect(sa) 2024-06-24T22:22:08.773002647Z TimeoutError: [Errno 110] Connection timed out 2024-06-24T22:22:08.773006649Z 2024-06-24T22:22:08.773012529Z The above exception was the direct cause of the following exception: 2024-06-24T22:22:08.773016530Z 2024-06-24T22:22:08.773020824Z Traceback (most recent call last): 2024-06-24T22:22:08.773025343Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 790, in urlopen 2024-06-24T22:22:08.773029558Z response = self._make_request( 2024-06-24T22:22:08.773034025Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 496, in _make_request 2024-06-24T22:22:08.773038527Z conn.request( 2024-06-24T22:22:08.773042742Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request 2024-06-24T22:22:08.773046960Z self.endheaders() 2024-06-24T22:22:08.773051185Z File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders 2024-06-24T22:22:08.773055431Z self._send_output(message_body, encode_chunked=encode_chunked) 2024-06-24T22:22:08.773059647Z File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output 2024-06-24T22:22:08.773063762Z self.send(msg) 2024-06-24T22:22:08.773068611Z File "/usr/lib/python3.10/http/client.py", line 976, in send 2024-06-24T22:22:08.773072786Z self.connect() 2024-06-24T22:22:08.773076950Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect 2024-06-24T22:22:08.773081725Z self.sock = self._new_conn() 2024-06-24T22:22:08.773085906Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 212, in _new_conn 2024-06-24T22:22:08.773105406Z raise ConnectTimeoutError( 2024-06-24T22:22:08.773113143Z urllib3.exceptions.ConnectTimeoutError: (, 'Connection to grafana timed out. (connect timeout=None)') 2024-06-24T22:22:08.773117179Z 2024-06-24T22:22:08.773121549Z The above exception was the direct cause of the following exception: 2024-06-24T22:22:08.773125449Z 2024-06-24T22:22:08.773129645Z Traceback (most recent call last): 2024-06-24T22:22:08.773133815Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send 2024-06-24T22:22:08.773137893Z resp = conn.urlopen( 2024-06-24T22:22:08.773142069Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 844, in urlopen 2024-06-24T22:22:08.773146254Z retries = retries.increment( 2024-06-24T22:22:08.773150362Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment 2024-06-24T22:22:08.773154473Z raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] 2024-06-24T22:22:08.773160168Z urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:22:08.773164309Z 2024-06-24T22:22:08.773168484Z During handling of the above exception, another exception occurred: 2024-06-24T22:22:08.773172409Z 2024-06-24T22:22:08.773176596Z Traceback (most recent call last): 2024-06-24T22:22:08.773180737Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 147, in dashboard_forever 2024-06-24T22:22:08.773184929Z self.create_dashboards() 2024-06-24T22:22:08.773189068Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 153, in create_dashboards 2024-06-24T22:22:08.773193701Z self.service.create_dashboards() 2024-06-24T22:22:08.773197876Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/service.py", line 52, in create_dashboards 2024-06-24T22:22:08.773202033Z dashboard_uids = self.grafana.get_all_dashboard_uids() 2024-06-24T22:22:08.773206971Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/backends/grafana.py", line 48, in get_all_dashboard_uids 2024-06-24T22:22:08.773211054Z response = requests.request( 2024-06-24T22:22:08.773215208Z File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request 2024-06-24T22:22:08.773219598Z return session.request(method=method, url=url, **kwargs) 2024-06-24T22:22:08.773223725Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request 2024-06-24T22:22:08.773227810Z resp = self.send(prep, **send_kwargs) 2024-06-24T22:22:08.773231993Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send 2024-06-24T22:22:08.773236149Z r = adapter.send(request, **kwargs) 2024-06-24T22:22:08.773240349Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 507, in send 2024-06-24T22:22:08.773244472Z raise ConnectTimeout(e, request=request) 2024-06-24T22:22:08.773252022Z requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:24:19.844310464Z 06/24/2024 10:24:19 PM - osm_mon.dashboarder.dashboarder - ERROR - Error creating dashboards 2024-06-24T22:24:19.844388686Z Traceback (most recent call last): 2024-06-24T22:24:19.844395241Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn 2024-06-24T22:24:19.844399953Z sock = connection.create_connection( 2024-06-24T22:24:19.844405525Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection 2024-06-24T22:24:19.844411215Z raise err 2024-06-24T22:24:19.844416155Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection 2024-06-24T22:24:19.844421694Z sock.connect(sa) 2024-06-24T22:24:19.844426177Z TimeoutError: [Errno 110] Connection timed out 2024-06-24T22:24:19.844430510Z 2024-06-24T22:24:19.844435653Z The above exception was the direct cause of the following exception: 2024-06-24T22:24:19.844439696Z 2024-06-24T22:24:19.844442882Z Traceback (most recent call last): 2024-06-24T22:24:19.844446318Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 790, in urlopen 2024-06-24T22:24:19.844449356Z response = self._make_request( 2024-06-24T22:24:19.844456184Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 496, in _make_request 2024-06-24T22:24:19.844459278Z conn.request( 2024-06-24T22:24:19.844462557Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request 2024-06-24T22:24:19.844465602Z self.endheaders() 2024-06-24T22:24:19.844469450Z File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders 2024-06-24T22:24:19.844473711Z self._send_output(message_body, encode_chunked=encode_chunked) 2024-06-24T22:24:19.844479659Z File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output 2024-06-24T22:24:19.844484437Z self.send(msg) 2024-06-24T22:24:19.844490295Z File "/usr/lib/python3.10/http/client.py", line 976, in send 2024-06-24T22:24:19.844495746Z self.connect() 2024-06-24T22:24:19.844500658Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect 2024-06-24T22:24:19.844505987Z self.sock = self._new_conn() 2024-06-24T22:24:19.844517769Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 212, in _new_conn 2024-06-24T22:24:19.844522112Z raise ConnectTimeoutError( 2024-06-24T22:24:19.844531782Z urllib3.exceptions.ConnectTimeoutError: (, 'Connection to grafana timed out. (connect timeout=None)') 2024-06-24T22:24:19.844542087Z 2024-06-24T22:24:19.844546534Z The above exception was the direct cause of the following exception: 2024-06-24T22:24:19.844550485Z 2024-06-24T22:24:19.844557462Z Traceback (most recent call last): 2024-06-24T22:24:19.844561770Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send 2024-06-24T22:24:19.844566407Z resp = conn.urlopen( 2024-06-24T22:24:19.844570776Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 844, in urlopen 2024-06-24T22:24:19.844575211Z retries = retries.increment( 2024-06-24T22:24:19.844579486Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment 2024-06-24T22:24:19.844583827Z raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] 2024-06-24T22:24:19.844596281Z urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:24:19.844606751Z 2024-06-24T22:24:19.844611600Z During handling of the above exception, another exception occurred: 2024-06-24T22:24:19.844615763Z 2024-06-24T22:24:19.844620499Z Traceback (most recent call last): 2024-06-24T22:24:19.844626041Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 147, in dashboard_forever 2024-06-24T22:24:19.844630939Z self.create_dashboards() 2024-06-24T22:24:19.844635723Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 153, in create_dashboards 2024-06-24T22:24:19.844640375Z self.service.create_dashboards() 2024-06-24T22:24:19.844645917Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/service.py", line 52, in create_dashboards 2024-06-24T22:24:19.844650796Z dashboard_uids = self.grafana.get_all_dashboard_uids() 2024-06-24T22:24:19.844655982Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/backends/grafana.py", line 48, in get_all_dashboard_uids 2024-06-24T22:24:19.844660950Z response = requests.request( 2024-06-24T22:24:19.844665975Z File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request 2024-06-24T22:24:19.844670994Z return session.request(method=method, url=url, **kwargs) 2024-06-24T22:24:19.844675626Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request 2024-06-24T22:24:19.844680276Z resp = self.send(prep, **send_kwargs) 2024-06-24T22:24:19.844684987Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send 2024-06-24T22:24:19.844690159Z r = adapter.send(request, **kwargs) 2024-06-24T22:24:19.844695049Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 507, in send 2024-06-24T22:24:19.844699891Z raise ConnectTimeout(e, request=request) 2024-06-24T22:24:19.844724153Z requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:26:30.913494957Z 06/24/2024 10:26:30 PM - osm_mon.dashboarder.dashboarder - ERROR - Error creating dashboards 2024-06-24T22:26:30.913526143Z Traceback (most recent call last): 2024-06-24T22:26:30.913530473Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn 2024-06-24T22:26:30.913534139Z sock = connection.create_connection( 2024-06-24T22:26:30.913538835Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection 2024-06-24T22:26:30.913542763Z raise err 2024-06-24T22:26:30.913546339Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection 2024-06-24T22:26:30.913550677Z sock.connect(sa) 2024-06-24T22:26:30.913554308Z TimeoutError: [Errno 110] Connection timed out 2024-06-24T22:26:30.913557666Z 2024-06-24T22:26:30.913561750Z The above exception was the direct cause of the following exception: 2024-06-24T22:26:30.913565133Z 2024-06-24T22:26:30.913568715Z Traceback (most recent call last): 2024-06-24T22:26:30.913572496Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 790, in urlopen 2024-06-24T22:26:30.913576023Z response = self._make_request( 2024-06-24T22:26:30.913579511Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 496, in _make_request 2024-06-24T22:26:30.913583119Z conn.request( 2024-06-24T22:26:30.913586637Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request 2024-06-24T22:26:30.913603316Z self.endheaders() 2024-06-24T22:26:30.913606967Z File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders 2024-06-24T22:26:30.913610528Z self._send_output(message_body, encode_chunked=encode_chunked) 2024-06-24T22:26:30.913614012Z File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output 2024-06-24T22:26:30.913617413Z self.send(msg) 2024-06-24T22:26:30.913621533Z File "/usr/lib/python3.10/http/client.py", line 976, in send 2024-06-24T22:26:30.913624975Z self.connect() 2024-06-24T22:26:30.913628426Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect 2024-06-24T22:26:30.913632589Z self.sock = self._new_conn() 2024-06-24T22:26:30.913636482Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 212, in _new_conn 2024-06-24T22:26:30.913639899Z raise ConnectTimeoutError( 2024-06-24T22:26:30.913644118Z urllib3.exceptions.ConnectTimeoutError: (, 'Connection to grafana timed out. (connect timeout=None)') 2024-06-24T22:26:30.913647398Z 2024-06-24T22:26:30.913650959Z The above exception was the direct cause of the following exception: 2024-06-24T22:26:30.913654263Z 2024-06-24T22:26:30.913657694Z Traceback (most recent call last): 2024-06-24T22:26:30.913661137Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send 2024-06-24T22:26:30.913664554Z resp = conn.urlopen( 2024-06-24T22:26:30.913668090Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 844, in urlopen 2024-06-24T22:26:30.913671545Z retries = retries.increment( 2024-06-24T22:26:30.913675116Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment 2024-06-24T22:26:30.913678561Z raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] 2024-06-24T22:26:30.913685290Z urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:26:30.913688888Z 2024-06-24T22:26:30.913692420Z During handling of the above exception, another exception occurred: 2024-06-24T22:26:30.913695715Z 2024-06-24T22:26:30.913702326Z Traceback (most recent call last): 2024-06-24T22:26:30.913705841Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 147, in dashboard_forever 2024-06-24T22:26:30.913709450Z self.create_dashboards() 2024-06-24T22:26:30.913712930Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 153, in create_dashboards 2024-06-24T22:26:30.913716389Z self.service.create_dashboards() 2024-06-24T22:26:30.913719984Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/service.py", line 52, in create_dashboards 2024-06-24T22:26:30.913723465Z dashboard_uids = self.grafana.get_all_dashboard_uids() 2024-06-24T22:26:30.913727852Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/backends/grafana.py", line 48, in get_all_dashboard_uids 2024-06-24T22:26:30.913731354Z response = requests.request( 2024-06-24T22:26:30.913734984Z File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request 2024-06-24T22:26:30.913738473Z return session.request(method=method, url=url, **kwargs) 2024-06-24T22:26:30.913741915Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request 2024-06-24T22:26:30.913745447Z resp = self.send(prep, **send_kwargs) 2024-06-24T22:26:30.913753632Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send 2024-06-24T22:26:30.913757200Z r = adapter.send(request, **kwargs) 2024-06-24T22:26:30.913760717Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 507, in send 2024-06-24T22:26:30.913764185Z raise ConnectTimeout(e, request=request) 2024-06-24T22:26:30.913770126Z requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:28:41.984046352Z 06/24/2024 10:28:41 PM - osm_mon.dashboarder.dashboarder - ERROR - Error creating dashboards 2024-06-24T22:28:41.984076175Z Traceback (most recent call last): 2024-06-24T22:28:41.984079895Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn 2024-06-24T22:28:41.984082999Z sock = connection.create_connection( 2024-06-24T22:28:41.984086814Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection 2024-06-24T22:28:41.984089915Z raise err 2024-06-24T22:28:41.984092924Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection 2024-06-24T22:28:41.984096494Z sock.connect(sa) 2024-06-24T22:28:41.984099564Z TimeoutError: [Errno 110] Connection timed out 2024-06-24T22:28:41.984102336Z 2024-06-24T22:28:41.984110631Z The above exception was the direct cause of the following exception: 2024-06-24T22:28:41.984113478Z 2024-06-24T22:28:41.984116390Z Traceback (most recent call last): 2024-06-24T22:28:41.984119437Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 790, in urlopen 2024-06-24T22:28:41.984122441Z response = self._make_request( 2024-06-24T22:28:41.984125387Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 496, in _make_request 2024-06-24T22:28:41.984128339Z conn.request( 2024-06-24T22:28:41.984133821Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request 2024-06-24T22:28:41.984136880Z self.endheaders() 2024-06-24T22:28:41.984139874Z File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders 2024-06-24T22:28:41.984143077Z self._send_output(message_body, encode_chunked=encode_chunked) 2024-06-24T22:28:41.984146259Z File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output 2024-06-24T22:28:41.984149330Z self.send(msg) 2024-06-24T22:28:41.984153191Z File "/usr/lib/python3.10/http/client.py", line 976, in send 2024-06-24T22:28:41.984156214Z self.connect() 2024-06-24T22:28:41.984159220Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect 2024-06-24T22:28:41.984162757Z self.sock = self._new_conn() 2024-06-24T22:28:41.984165699Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 212, in _new_conn 2024-06-24T22:28:41.984169659Z raise ConnectTimeoutError( 2024-06-24T22:28:41.984173356Z urllib3.exceptions.ConnectTimeoutError: (, 'Connection to grafana timed out. (connect timeout=None)') 2024-06-24T22:28:41.984176144Z 2024-06-24T22:28:41.984179200Z The above exception was the direct cause of the following exception: 2024-06-24T22:28:41.984182007Z 2024-06-24T22:28:41.984185352Z Traceback (most recent call last): 2024-06-24T22:28:41.984188274Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send 2024-06-24T22:28:41.984203130Z resp = conn.urlopen( 2024-06-24T22:28:41.984206232Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 844, in urlopen 2024-06-24T22:28:41.984209179Z retries = retries.increment( 2024-06-24T22:28:41.984212070Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment 2024-06-24T22:28:41.984215153Z raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] 2024-06-24T22:28:41.984223886Z urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:28:41.984227010Z 2024-06-24T22:28:41.984229985Z During handling of the above exception, another exception occurred: 2024-06-24T22:28:41.984232722Z 2024-06-24T22:28:41.984235796Z Traceback (most recent call last): 2024-06-24T22:28:41.984245506Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 147, in dashboard_forever 2024-06-24T22:28:41.984248530Z self.create_dashboards() 2024-06-24T22:28:41.984251468Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 153, in create_dashboards 2024-06-24T22:28:41.984254460Z self.service.create_dashboards() 2024-06-24T22:28:41.984257372Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/service.py", line 52, in create_dashboards 2024-06-24T22:28:41.984260355Z dashboard_uids = self.grafana.get_all_dashboard_uids() 2024-06-24T22:28:41.984263824Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/backends/grafana.py", line 48, in get_all_dashboard_uids 2024-06-24T22:28:41.984266799Z response = requests.request( 2024-06-24T22:28:41.984270070Z File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request 2024-06-24T22:28:41.984272928Z return session.request(method=method, url=url, **kwargs) 2024-06-24T22:28:41.984275834Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request 2024-06-24T22:28:41.984278820Z resp = self.send(prep, **send_kwargs) 2024-06-24T22:28:41.984281729Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send 2024-06-24T22:28:41.984284743Z r = adapter.send(request, **kwargs) 2024-06-24T22:28:41.984287645Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 507, in send 2024-06-24T22:28:41.984290673Z raise ConnectTimeout(e, request=request) 2024-06-24T22:28:41.984319435Z requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:30:53.055712230Z 06/24/2024 10:30:53 PM - osm_mon.dashboarder.dashboarder - ERROR - Error creating dashboards 2024-06-24T22:30:53.055754009Z Traceback (most recent call last): 2024-06-24T22:30:53.055758862Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn 2024-06-24T22:30:53.055762978Z sock = connection.create_connection( 2024-06-24T22:30:53.055769105Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection 2024-06-24T22:30:53.055773317Z raise err 2024-06-24T22:30:53.055777340Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection 2024-06-24T22:30:53.055781886Z sock.connect(sa) 2024-06-24T22:30:53.055801325Z TimeoutError: [Errno 110] Connection timed out 2024-06-24T22:30:53.055805368Z 2024-06-24T22:30:53.055809732Z The above exception was the direct cause of the following exception: 2024-06-24T22:30:53.055813422Z 2024-06-24T22:30:53.055817371Z Traceback (most recent call last): 2024-06-24T22:30:53.055821294Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 790, in urlopen 2024-06-24T22:30:53.055825141Z response = self._make_request( 2024-06-24T22:30:53.055829089Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 496, in _make_request 2024-06-24T22:30:53.055832910Z conn.request( 2024-06-24T22:30:53.055836842Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request 2024-06-24T22:30:53.055840627Z self.endheaders() 2024-06-24T22:30:53.055844586Z File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders 2024-06-24T22:30:53.055853375Z self._send_output(message_body, encode_chunked=encode_chunked) 2024-06-24T22:30:53.055857532Z File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output 2024-06-24T22:30:53.055861386Z self.send(msg) 2024-06-24T22:30:53.055865916Z File "/usr/lib/python3.10/http/client.py", line 976, in send 2024-06-24T22:30:53.055869821Z self.connect() 2024-06-24T22:30:53.055874406Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect 2024-06-24T22:30:53.055878827Z self.sock = self._new_conn() 2024-06-24T22:30:53.055882664Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 212, in _new_conn 2024-06-24T22:30:53.055886689Z raise ConnectTimeoutError( 2024-06-24T22:30:53.055891258Z urllib3.exceptions.ConnectTimeoutError: (, 'Connection to grafana timed out. (connect timeout=None)') 2024-06-24T22:30:53.055894921Z 2024-06-24T22:30:53.055898818Z The above exception was the direct cause of the following exception: 2024-06-24T22:30:53.055902658Z 2024-06-24T22:30:53.055906532Z Traceback (most recent call last): 2024-06-24T22:30:53.055910366Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send 2024-06-24T22:30:53.055914448Z resp = conn.urlopen( 2024-06-24T22:30:53.055918408Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 844, in urlopen 2024-06-24T22:30:53.055922202Z retries = retries.increment( 2024-06-24T22:30:53.055926093Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment 2024-06-24T22:30:53.055929956Z raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] 2024-06-24T22:30:53.055934880Z urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:30:53.055938628Z 2024-06-24T22:30:53.055942454Z During handling of the above exception, another exception occurred: 2024-06-24T22:30:53.055946082Z 2024-06-24T22:30:53.055949871Z Traceback (most recent call last): 2024-06-24T22:30:53.055953849Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 147, in dashboard_forever 2024-06-24T22:30:53.055957620Z self.create_dashboards() 2024-06-24T22:30:53.055961632Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 153, in create_dashboards 2024-06-24T22:30:53.055965497Z self.service.create_dashboards() 2024-06-24T22:30:53.055975666Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/service.py", line 52, in create_dashboards 2024-06-24T22:30:53.055979743Z dashboard_uids = self.grafana.get_all_dashboard_uids() 2024-06-24T22:30:53.055984120Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/backends/grafana.py", line 48, in get_all_dashboard_uids 2024-06-24T22:30:53.055988118Z response = requests.request( 2024-06-24T22:30:53.055991997Z File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request 2024-06-24T22:30:53.055995922Z return session.request(method=method, url=url, **kwargs) 2024-06-24T22:30:53.055999838Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request 2024-06-24T22:30:53.056004036Z resp = self.send(prep, **send_kwargs) 2024-06-24T22:30:53.056007865Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send 2024-06-24T22:30:53.056012003Z r = adapter.send(request, **kwargs) 2024-06-24T22:30:53.056016022Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 507, in send 2024-06-24T22:30:53.056019973Z raise ConnectTimeout(e, request=request) 2024-06-24T22:30:53.056027107Z requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:33:04.129555139Z 06/24/2024 10:33:04 PM - osm_mon.dashboarder.dashboarder - ERROR - Error creating dashboards 2024-06-24T22:33:04.129599850Z Traceback (most recent call last): 2024-06-24T22:33:04.129608603Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn 2024-06-24T22:33:04.129616326Z sock = connection.create_connection( 2024-06-24T22:33:04.129625200Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection 2024-06-24T22:33:04.129632979Z raise err 2024-06-24T22:33:04.129641165Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection 2024-06-24T22:33:04.129649438Z sock.connect(sa) 2024-06-24T22:33:04.129657081Z TimeoutError: [Errno 110] Connection timed out 2024-06-24T22:33:04.129664426Z 2024-06-24T22:33:04.129672989Z The above exception was the direct cause of the following exception: 2024-06-24T22:33:04.129680304Z 2024-06-24T22:33:04.129688473Z Traceback (most recent call last): 2024-06-24T22:33:04.129696082Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 790, in urlopen 2024-06-24T22:33:04.129703670Z response = self._make_request( 2024-06-24T22:33:04.129711241Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 496, in _make_request 2024-06-24T22:33:04.129730015Z conn.request( 2024-06-24T22:33:04.129737925Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request 2024-06-24T22:33:04.129745422Z self.endheaders() 2024-06-24T22:33:04.129753048Z File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders 2024-06-24T22:33:04.129760691Z self._send_output(message_body, encode_chunked=encode_chunked) 2024-06-24T22:33:04.129768152Z File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output 2024-06-24T22:33:04.129775703Z self.send(msg) 2024-06-24T22:33:04.129783991Z File "/usr/lib/python3.10/http/client.py", line 976, in send 2024-06-24T22:33:04.129791532Z self.connect() 2024-06-24T22:33:04.129799107Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect 2024-06-24T22:33:04.129827614Z self.sock = self._new_conn() 2024-06-24T22:33:04.129835526Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 212, in _new_conn 2024-06-24T22:33:04.129843057Z raise ConnectTimeoutError( 2024-06-24T22:33:04.129852691Z urllib3.exceptions.ConnectTimeoutError: (, 'Connection to grafana timed out. (connect timeout=None)') 2024-06-24T22:33:04.129859893Z 2024-06-24T22:33:04.129867534Z The above exception was the direct cause of the following exception: 2024-06-24T22:33:04.129874766Z 2024-06-24T22:33:04.129882228Z Traceback (most recent call last): 2024-06-24T22:33:04.129889686Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send 2024-06-24T22:33:04.129897121Z resp = conn.urlopen( 2024-06-24T22:33:04.129904759Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 844, in urlopen 2024-06-24T22:33:04.129912227Z retries = retries.increment( 2024-06-24T22:33:04.129919670Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment 2024-06-24T22:33:04.129927588Z raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] 2024-06-24T22:33:04.129936032Z urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:33:04.129943350Z 2024-06-24T22:33:04.129950995Z During handling of the above exception, another exception occurred: 2024-06-24T22:33:04.129958180Z 2024-06-24T22:33:04.129965917Z Traceback (most recent call last): 2024-06-24T22:33:04.129973429Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 147, in dashboard_forever 2024-06-24T22:33:04.129980904Z self.create_dashboards() 2024-06-24T22:33:04.129988310Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 153, in create_dashboards 2024-06-24T22:33:04.129995827Z self.service.create_dashboards() 2024-06-24T22:33:04.130003303Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/service.py", line 52, in create_dashboards 2024-06-24T22:33:04.130010790Z dashboard_uids = self.grafana.get_all_dashboard_uids() 2024-06-24T22:33:04.130019504Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/backends/grafana.py", line 48, in get_all_dashboard_uids 2024-06-24T22:33:04.130027125Z response = requests.request( 2024-06-24T22:33:04.130034672Z File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request 2024-06-24T22:33:04.130042126Z return session.request(method=method, url=url, **kwargs) 2024-06-24T22:33:04.130049675Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request 2024-06-24T22:33:04.130057091Z resp = self.send(prep, **send_kwargs) 2024-06-24T22:33:04.130064589Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send 2024-06-24T22:33:04.130072017Z r = adapter.send(request, **kwargs) 2024-06-24T22:33:04.130079478Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 507, in send 2024-06-24T22:33:04.130087036Z raise ConnectTimeout(e, request=request) 2024-06-24T22:33:04.130099360Z requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:35:15.200716986Z 06/24/2024 10:35:15 PM - osm_mon.dashboarder.dashboarder - ERROR - Error creating dashboards 2024-06-24T22:35:15.200751807Z Traceback (most recent call last): 2024-06-24T22:35:15.200759412Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn 2024-06-24T22:35:15.200766402Z sock = connection.create_connection( 2024-06-24T22:35:15.200774357Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection 2024-06-24T22:35:15.200781471Z raise err 2024-06-24T22:35:15.200788532Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection 2024-06-24T22:35:15.200796109Z sock.connect(sa) 2024-06-24T22:35:15.200803152Z TimeoutError: [Errno 110] Connection timed out 2024-06-24T22:35:15.200809876Z 2024-06-24T22:35:15.200817337Z The above exception was the direct cause of the following exception: 2024-06-24T22:35:15.200824016Z 2024-06-24T22:35:15.200831023Z Traceback (most recent call last): 2024-06-24T22:35:15.200838065Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 790, in urlopen 2024-06-24T22:35:15.200845256Z response = self._make_request( 2024-06-24T22:35:15.200852193Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 496, in _make_request 2024-06-24T22:35:15.200859634Z conn.request( 2024-06-24T22:35:15.200866698Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request 2024-06-24T22:35:15.200873764Z self.endheaders() 2024-06-24T22:35:15.200880735Z File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders 2024-06-24T22:35:15.200887744Z self._send_output(message_body, encode_chunked=encode_chunked) 2024-06-24T22:35:15.200895114Z File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output 2024-06-24T22:35:15.200901966Z self.send(msg) 2024-06-24T22:35:15.200909563Z File "/usr/lib/python3.10/http/client.py", line 976, in send 2024-06-24T22:35:15.200916424Z self.connect() 2024-06-24T22:35:15.200923385Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect 2024-06-24T22:35:15.200930810Z self.sock = self._new_conn() 2024-06-24T22:35:15.200937721Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 212, in _new_conn 2024-06-24T22:35:15.200944734Z raise ConnectTimeoutError( 2024-06-24T22:35:15.200952755Z urllib3.exceptions.ConnectTimeoutError: (, 'Connection to grafana timed out. (connect timeout=None)') 2024-06-24T22:35:15.200959377Z 2024-06-24T22:35:15.200966308Z The above exception was the direct cause of the following exception: 2024-06-24T22:35:15.200973392Z 2024-06-24T22:35:15.200980643Z Traceback (most recent call last): 2024-06-24T22:35:15.200987716Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send 2024-06-24T22:35:15.200994605Z resp = conn.urlopen( 2024-06-24T22:35:15.201001636Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 844, in urlopen 2024-06-24T22:35:15.201008552Z retries = retries.increment( 2024-06-24T22:35:15.201015482Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment 2024-06-24T22:35:15.201022299Z raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] 2024-06-24T22:35:15.201034377Z urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:35:15.201056425Z 2024-06-24T22:35:15.201063836Z During handling of the above exception, another exception occurred: 2024-06-24T22:35:15.201070419Z 2024-06-24T22:35:15.201077419Z Traceback (most recent call last): 2024-06-24T22:35:15.201084384Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 147, in dashboard_forever 2024-06-24T22:35:15.201091207Z self.create_dashboards() 2024-06-24T22:35:15.201098042Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 153, in create_dashboards 2024-06-24T22:35:15.201104857Z self.service.create_dashboards() 2024-06-24T22:35:15.201111672Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/service.py", line 52, in create_dashboards 2024-06-24T22:35:15.201118430Z dashboard_uids = self.grafana.get_all_dashboard_uids() 2024-06-24T22:35:15.201126463Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/backends/grafana.py", line 48, in get_all_dashboard_uids 2024-06-24T22:35:15.201133249Z response = requests.request( 2024-06-24T22:35:15.201140224Z File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request 2024-06-24T22:35:15.201147096Z return session.request(method=method, url=url, **kwargs) 2024-06-24T22:35:15.201153861Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request 2024-06-24T22:35:15.201160634Z resp = self.send(prep, **send_kwargs) 2024-06-24T22:35:15.201167460Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send 2024-06-24T22:35:15.201174296Z r = adapter.send(request, **kwargs) 2024-06-24T22:35:15.201181061Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 507, in send 2024-06-24T22:35:15.201188057Z raise ConnectTimeout(e, request=request) 2024-06-24T22:35:15.201198765Z requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:37:26.272495928Z 06/24/2024 10:37:26 PM - osm_mon.dashboarder.dashboarder - ERROR - Error creating dashboards 2024-06-24T22:37:26.272534719Z Traceback (most recent call last): 2024-06-24T22:37:26.272541557Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn 2024-06-24T22:37:26.272547594Z sock = connection.create_connection( 2024-06-24T22:37:26.272554632Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection 2024-06-24T22:37:26.272560469Z raise err 2024-06-24T22:37:26.272566340Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection 2024-06-24T22:37:26.272579858Z sock.connect(sa) 2024-06-24T22:37:26.272585777Z TimeoutError: [Errno 110] Connection timed out 2024-06-24T22:37:26.272591124Z 2024-06-24T22:37:26.272597521Z The above exception was the direct cause of the following exception: 2024-06-24T22:37:26.272602920Z 2024-06-24T22:37:26.272608657Z Traceback (most recent call last): 2024-06-24T22:37:26.272614687Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 790, in urlopen 2024-06-24T22:37:26.272620258Z response = self._make_request( 2024-06-24T22:37:26.272625829Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 496, in _make_request 2024-06-24T22:37:26.272649160Z conn.request( 2024-06-24T22:37:26.272655292Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request 2024-06-24T22:37:26.272660838Z self.endheaders() 2024-06-24T22:37:26.272666392Z File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders 2024-06-24T22:37:26.272672238Z self._send_output(message_body, encode_chunked=encode_chunked) 2024-06-24T22:37:26.272677822Z File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output 2024-06-24T22:37:26.272683835Z self.send(msg) 2024-06-24T22:37:26.272689976Z File "/usr/lib/python3.10/http/client.py", line 976, in send 2024-06-24T22:37:26.272701315Z self.connect() 2024-06-24T22:37:26.272706891Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect 2024-06-24T22:37:26.272713024Z self.sock = self._new_conn() 2024-06-24T22:37:26.272719590Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 212, in _new_conn 2024-06-24T22:37:26.272725838Z raise ConnectTimeoutError( 2024-06-24T22:37:26.272732144Z urllib3.exceptions.ConnectTimeoutError: (, 'Connection to grafana timed out. (connect timeout=None)') 2024-06-24T22:37:26.272737463Z 2024-06-24T22:37:26.272743070Z The above exception was the direct cause of the following exception: 2024-06-24T22:37:26.272748313Z 2024-06-24T22:37:26.272753937Z Traceback (most recent call last): 2024-06-24T22:37:26.272759634Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send 2024-06-24T22:37:26.272765486Z resp = conn.urlopen( 2024-06-24T22:37:26.272771089Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 844, in urlopen 2024-06-24T22:37:26.272776675Z retries = retries.increment( 2024-06-24T22:37:26.272782156Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment 2024-06-24T22:37:26.272787593Z raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] 2024-06-24T22:37:26.272793812Z urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:37:26.272799098Z 2024-06-24T22:37:26.272804755Z During handling of the above exception, another exception occurred: 2024-06-24T22:37:26.272809953Z 2024-06-24T22:37:26.272815558Z Traceback (most recent call last): 2024-06-24T22:37:26.272821044Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 147, in dashboard_forever 2024-06-24T22:37:26.272826635Z self.create_dashboards() 2024-06-24T22:37:26.272832098Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 153, in create_dashboards 2024-06-24T22:37:26.272837609Z self.service.create_dashboards() 2024-06-24T22:37:26.272843122Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/service.py", line 52, in create_dashboards 2024-06-24T22:37:26.272848609Z dashboard_uids = self.grafana.get_all_dashboard_uids() 2024-06-24T22:37:26.272854847Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/backends/grafana.py", line 48, in get_all_dashboard_uids 2024-06-24T22:37:26.272860357Z response = requests.request( 2024-06-24T22:37:26.272865831Z File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request 2024-06-24T22:37:26.272871392Z return session.request(method=method, url=url, **kwargs) 2024-06-24T22:37:26.272886392Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request 2024-06-24T22:37:26.272892031Z resp = self.send(prep, **send_kwargs) 2024-06-24T22:37:26.272897441Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send 2024-06-24T22:37:26.272902889Z r = adapter.send(request, **kwargs) 2024-06-24T22:37:26.272908329Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 507, in send 2024-06-24T22:37:26.272913866Z raise ConnectTimeout(e, request=request) 2024-06-24T22:37:26.272940885Z requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:39:37.343076055Z 06/24/2024 10:39:37 PM - osm_mon.dashboarder.dashboarder - ERROR - Error creating dashboards 2024-06-24T22:39:37.343122997Z Traceback (most recent call last): 2024-06-24T22:39:37.343126983Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn 2024-06-24T22:39:37.343129965Z sock = connection.create_connection( 2024-06-24T22:39:37.343133678Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection 2024-06-24T22:39:37.343136798Z raise err 2024-06-24T22:39:37.343139823Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection 2024-06-24T22:39:37.343143489Z sock.connect(sa) 2024-06-24T22:39:37.343146467Z TimeoutError: [Errno 110] Connection timed out 2024-06-24T22:39:37.343149376Z 2024-06-24T22:39:37.343153200Z The above exception was the direct cause of the following exception: 2024-06-24T22:39:37.343156010Z 2024-06-24T22:39:37.343159081Z Traceback (most recent call last): 2024-06-24T22:39:37.343162123Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 790, in urlopen 2024-06-24T22:39:37.343168543Z response = self._make_request( 2024-06-24T22:39:37.343171481Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 496, in _make_request 2024-06-24T22:39:37.343174436Z conn.request( 2024-06-24T22:39:37.343177478Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request 2024-06-24T22:39:37.343180401Z self.endheaders() 2024-06-24T22:39:37.343183709Z File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders 2024-06-24T22:39:37.343186661Z self._send_output(message_body, encode_chunked=encode_chunked) 2024-06-24T22:39:37.343189671Z File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output 2024-06-24T22:39:37.343192588Z self.send(msg) 2024-06-24T22:39:37.343196025Z File "/usr/lib/python3.10/http/client.py", line 976, in send 2024-06-24T22:39:37.343198993Z self.connect() 2024-06-24T22:39:37.343202015Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect 2024-06-24T22:39:37.343205524Z self.sock = self._new_conn() 2024-06-24T22:39:37.343208508Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 212, in _new_conn 2024-06-24T22:39:37.343211474Z raise ConnectTimeoutError( 2024-06-24T22:39:37.343215043Z urllib3.exceptions.ConnectTimeoutError: (, 'Connection to grafana timed out. (connect timeout=None)') 2024-06-24T22:39:37.343217994Z 2024-06-24T22:39:37.343220967Z The above exception was the direct cause of the following exception: 2024-06-24T22:39:37.343223873Z 2024-06-24T22:39:37.343235528Z Traceback (most recent call last): 2024-06-24T22:39:37.343238915Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send 2024-06-24T22:39:37.343241784Z resp = conn.urlopen( 2024-06-24T22:39:37.343244784Z File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 844, in urlopen 2024-06-24T22:39:37.343247706Z retries = retries.increment( 2024-06-24T22:39:37.343250675Z File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment 2024-06-24T22:39:37.343253590Z raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] 2024-06-24T22:39:37.343258610Z urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)')) 2024-06-24T22:39:37.343261585Z 2024-06-24T22:39:37.343264561Z During handling of the above exception, another exception occurred: 2024-06-24T22:39:37.343267270Z 2024-06-24T22:39:37.343270238Z Traceback (most recent call last): 2024-06-24T22:39:37.343273132Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 147, in dashboard_forever 2024-06-24T22:39:37.343276141Z self.create_dashboards() 2024-06-24T22:39:37.343279282Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/dashboarder.py", line 153, in create_dashboards 2024-06-24T22:39:37.343282124Z self.service.create_dashboards() 2024-06-24T22:39:37.343284989Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/service.py", line 52, in create_dashboards 2024-06-24T22:39:37.343287875Z dashboard_uids = self.grafana.get_all_dashboard_uids() 2024-06-24T22:39:37.343291254Z File "/usr/lib/python3/dist-packages/osm_mon/dashboarder/backends/grafana.py", line 48, in get_all_dashboard_uids 2024-06-24T22:39:37.343294139Z response = requests.request( 2024-06-24T22:39:37.343297046Z File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request 2024-06-24T22:39:37.343299914Z return session.request(method=method, url=url, **kwargs) 2024-06-24T22:39:37.343305727Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request 2024-06-24T22:39:37.343308568Z resp = self.send(prep, **send_kwargs) 2024-06-24T22:39:37.343311521Z File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send 2024-06-24T22:39:37.343314569Z r = adapter.send(request, **kwargs) 2024-06-24T22:39:37.343317659Z File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 507, in send 2024-06-24T22:39:37.343320682Z raise ConnectTimeout(e, request=request) 2024-06-24T22:39:37.343326144Z requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='grafana', port=3000): Max retries exceeded with url: /api/search?tag=osm_automated (Caused by ConnectTimeoutError(, 'Connection to grafana timed out. (connect timeout=None)'))