2024-02-22T01:26:26.144741475Z Connected to MongoDB!
2024-02-22T01:26:26.144791329Z Refreshing prometheus config file for first time
2024-02-22T01:26:56.519278542Z Error in first configuration attempt!
2024-02-22T01:26:56.519304521Z No replica set members match selector "<function readable_server_selector at 0x7fe511bd4550>", Timeout: 30s, Topology Description: <TopologyDescription id: 65d6a2c22eb23bfa60def841, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.000945658999967236>]>
2024-02-22T01:27:26.548426429Z No replica set members match selector "<function readable_server_selector at 0x7fe511bd4550>", Timeout: 30s, Topology Description: <TopologyDescription id: 65d6a2c22eb23bfa60def841, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.0010433181919679555>]>
2024-02-22T01:27:26.548807228Z Detected failure while listening to prometheus jobs collection, retrying...
2024-02-22T01:28:01.575974143Z No replica set members match selector "<function readable_server_selector at 0x7fe511bd4550>", Timeout: 30s, Topology Description: <TopologyDescription id: 65d6a2c22eb23bfa60def841, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.00112187188182595>]>
2024-02-22T01:28:01.576035295Z Detected failure while listening to prometheus jobs collection, retrying...
2024-02-22T01:28:36.600051188Z No replica set members match selector "<function readable_server_selector at 0x7fe511bd4550>", Timeout: 30s, Topology Description: <TopologyDescription id: 65d6a2c22eb23bfa60def841, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.0010437684355043516>]>
2024-02-22T01:28:36.600112779Z Detected failure while listening to prometheus jobs collection, retrying...
2024-02-22T01:29:11.627260742Z No replica set members match selector "<function readable_server_selector at 0x7fe511bd4550>", Timeout: 30s, Topology Description: <TopologyDescription id: 65d6a2c22eb23bfa60def841, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.0010251258920029145>]>
2024-02-22T01:29:11.627504246Z Detected failure while listening to prometheus jobs collection, retrying...
2024-02-22T01:29:46.657761322Z No replica set members match selector "<function readable_server_selector at 0x7fe511bd4550>", Timeout: 30s, Topology Description: <TopologyDescription id: 65d6a2c22eb23bfa60def841, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.0010686088246866544>]>
2024-02-22T01:29:46.657822652Z Detected failure while listening to prometheus jobs collection, retrying...
2024-02-22T01:30:08.076137015Z Listening to changes in prometheus jobs collection
2024-02-22T01:30:09.376879031Z Change detected, updating prometheus config
2024-02-22T01:30:09.380291605Z Jobs detected : 1
2024-02-22T01:30:09.388763964Z Writing new config file to /etc/prometheus/prometheus.yml
2024-02-22T01:30:09.391470237Z New config written, updating prometheus
2024-02-22T01:30:14.414409393Z Prometheus config update successful
2024-02-22T01:30:14.416276452Z 
2024-02-22T01:25:57.312502824Z level=info ts=2024-02-22T01:25:57.312Z caller=main.go:389 msg="No time or size retention was set so using the default time retention" duration=15d
2024-02-22T01:25:57.312538208Z level=info ts=2024-02-22T01:25:57.312Z caller=main.go:443 msg="Starting Prometheus" version="(version=2.28.1, branch=HEAD, revision=b0944590a1c9a6b35dc5a696869f75f422b107a1)"
2024-02-22T01:25:57.313295229Z level=info ts=2024-02-22T01:25:57.312Z caller=main.go:448 build_context="(go=go1.16.5, user=root@2915dd495090, date=20210701-15:20:10)"
2024-02-22T01:25:57.313309902Z level=info ts=2024-02-22T01:25:57.313Z caller=main.go:449 host_details="(Linux 5.4.0-107-generic #121-Ubuntu SMP Thu Mar 24 16:04:27 UTC 2022 x86_64 prometheus-0 (none))"
2024-02-22T01:25:57.313919889Z level=info ts=2024-02-22T01:25:57.313Z caller=main.go:450 fd_limits="(soft=1048576, hard=1048576)"
2024-02-22T01:25:57.314026661Z level=info ts=2024-02-22T01:25:57.313Z caller=main.go:451 vm_limits="(soft=unlimited, hard=unlimited)"
2024-02-22T01:25:57.320725552Z level=info ts=2024-02-22T01:25:57.320Z caller=web.go:541 component=web msg="Start listening for connections" address=0.0.0.0:9090
2024-02-22T01:25:57.325307790Z level=info ts=2024-02-22T01:25:57.325Z caller=main.go:824 msg="Starting TSDB ..."
2024-02-22T01:25:57.326395727Z level=info ts=2024-02-22T01:25:57.326Z caller=tls_config.go:191 component=web msg="TLS is disabled." http2=false
2024-02-22T01:25:57.358927237Z level=info ts=2024-02-22T01:25:57.352Z caller=head.go:780 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
2024-02-22T01:25:57.358950757Z level=info ts=2024-02-22T01:25:57.352Z caller=head.go:794 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=9.713µs
2024-02-22T01:25:57.358955960Z level=info ts=2024-02-22T01:25:57.352Z caller=head.go:800 component=tsdb msg="Replaying WAL, this may take a while"
2024-02-22T01:25:57.370901792Z level=info ts=2024-02-22T01:25:57.370Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
2024-02-22T01:25:57.370916980Z level=info ts=2024-02-22T01:25:57.370Z caller=head.go:860 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=25.553µs wal_replay_duration=18.387078ms total_replay_duration=18.443972ms
2024-02-22T01:25:57.373420402Z level=info ts=2024-02-22T01:25:57.373Z caller=main.go:851 fs_type=EXT4_SUPER_MAGIC
2024-02-22T01:25:57.373437170Z level=info ts=2024-02-22T01:25:57.373Z caller=main.go:854 msg="TSDB started"
2024-02-22T01:25:57.373440896Z level=info ts=2024-02-22T01:25:57.373Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-02-22T01:25:57.374278023Z level=info ts=2024-02-22T01:25:57.374Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=815.998µs remote_storage=6.252µs web_handler=483ns query_engine=3.66µs scrape=271.202µs scrape_sd=31.205µs notify=48.25µs notify_sd=17.528µs rules=4.132µs
2024-02-22T01:25:57.374299363Z level=info ts=2024-02-22T01:25:57.374Z caller=main.go:796 msg="Server is ready to receive web requests."
2024-02-22T01:30:09.398486810Z level=info ts=2024-02-22T01:30:09.398Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-02-22T01:30:09.399664407Z level=info ts=2024-02-22T01:30:09.399Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=1.295692ms remote_storage=8.827µs web_handler=5.875µs query_engine=6.319µs scrape=145.927µs scrape_sd=47.404µs notify=40.301µs notify_sd=12.304µs rules=7.735µs
2024-02-22T04:26:24.165107412Z level=info ts=2024-02-22T04:26:24.164Z caller=compact.go:518 component=tsdb msg="write block" mint=1708565169123 maxt=1708567200000 ulid=01HQ7H12G2P80VSA3QRYT3R1YW duration=34.64392ms
2024-02-22T04:26:24.167431482Z level=info ts=2024-02-22T04:26:24.167Z caller=head.go:967 component=tsdb msg="Head GC completed" duration=1.802312ms
2024-02-22T05:00:24.162519593Z level=info ts=2024-02-22T05:00:24.162Z caller=compact.go:518 component=tsdb msg="write block" mint=1708567209123 maxt=1708574400000 ulid=01HQ7JZAP1N2FRQCMW8DEGA3HM duration=32.696976ms
2024-02-22T05:00:24.164889383Z level=info ts=2024-02-22T05:00:24.164Z caller=head.go:967 component=tsdb msg="Head GC completed" duration=1.967144ms