2024-01-30T13:29:29.079346802Z level=info ts=2024-01-30T13:29:29.077Z caller=main.go:389 msg="No time or size retention was set so using the default time retention" duration=15d
2024-01-30T13:29:29.079424904Z level=info ts=2024-01-30T13:29:29.077Z caller=main.go:443 msg="Starting Prometheus" version="(version=2.28.1, branch=HEAD, revision=b0944590a1c9a6b35dc5a696869f75f422b107a1)"
2024-01-30T13:29:29.079436863Z level=info ts=2024-01-30T13:29:29.077Z caller=main.go:448 build_context="(go=go1.16.5, user=root@2915dd495090, date=20210701-15:20:10)"
2024-01-30T13:29:29.079445463Z level=info ts=2024-01-30T13:29:29.077Z caller=main.go:449 host_details="(Linux 5.4.0-107-generic #121-Ubuntu SMP Thu Mar 24 16:04:27 UTC 2022 x86_64 prometheus-0 (none))"
2024-01-30T13:29:29.079454087Z level=info ts=2024-01-30T13:29:29.077Z caller=main.go:450 fd_limits="(soft=1048576, hard=1048576)"
2024-01-30T13:29:29.079462042Z level=info ts=2024-01-30T13:29:29.077Z caller=main.go:451 vm_limits="(soft=unlimited, hard=unlimited)"
2024-01-30T13:29:29.079733010Z level=info ts=2024-01-30T13:29:29.079Z caller=web.go:541 component=web msg="Start listening for connections" address=0.0.0.0:9090
2024-01-30T13:29:29.081713487Z level=info ts=2024-01-30T13:29:29.080Z caller=main.go:824 msg="Starting TSDB ..."
2024-01-30T13:29:29.083859006Z level=info ts=2024-01-30T13:29:29.083Z caller=tls_config.go:191 component=web msg="TLS is disabled." http2=false
2024-01-30T13:29:29.085178006Z level=info ts=2024-01-30T13:29:29.085Z caller=head.go:780 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
2024-01-30T13:29:29.085206767Z level=info ts=2024-01-30T13:29:29.085Z caller=head.go:794 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=4.905µs
2024-01-30T13:29:29.085218905Z level=info ts=2024-01-30T13:29:29.085Z caller=head.go:800 component=tsdb msg="Replaying WAL, this may take a while"
2024-01-30T13:29:29.085682058Z level=info ts=2024-01-30T13:29:29.085Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
2024-01-30T13:29:29.085706539Z level=info ts=2024-01-30T13:29:29.085Z caller=head.go:860 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=27.88µs wal_replay_duration=410.66µs total_replay_duration=457.006µs
2024-01-30T13:29:29.087307216Z level=info ts=2024-01-30T13:29:29.087Z caller=main.go:851 fs_type=EXT4_SUPER_MAGIC
2024-01-30T13:29:29.087332664Z level=info ts=2024-01-30T13:29:29.087Z caller=main.go:854 msg="TSDB started"
2024-01-30T13:29:29.087342437Z level=info ts=2024-01-30T13:29:29.087Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-01-30T13:29:29.088778624Z level=info ts=2024-01-30T13:29:29.088Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=1.291391ms remote_storage=5.232µs web_handler=383ns query_engine=3.494µs scrape=337.375µs scrape_sd=583.435µs notify=53.069µs notify_sd=10.764µs rules=3.567µs
2024-01-30T13:29:29.088793561Z level=info ts=2024-01-30T13:29:29.088Z caller=main.go:796 msg="Server is ready to receive web requests."
2024-01-30T13:35:08.258589171Z level=info ts=2024-01-30T13:35:08.258Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-01-30T13:35:08.260954635Z level=info ts=2024-01-30T13:35:08.260Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=2.53014ms remote_storage=8.184µs web_handler=7.639µs query_engine=17.814µs scrape=281.527µs scrape_sd=76.782µs notify=72.509µs notify_sd=24.451µs rules=15.199µs
2024-01-30T13:33:44.444978461Z Created MongoClient to connect to MongoDB!
2024-01-30T13:33:44.445150202Z Refreshing prometheus config file for first time
2024-01-30T13:34:14.481129050Z Error in configuration attempt! Number of tries: 1/3
2024-01-30T13:34:14.481219008Z No replica set members match selector "<function readable_server_selector at 0x7f01c7d3f4c0>", Timeout: 30s, Topology Description: <TopologyDescription id: 65b8fab855a0e44bfb8deed8, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.0018428927199893226>]>
2024-01-30T13:34:19.484060169Z Refreshing prometheus config file for first time
2024-01-30T13:34:49.534045661Z Error in configuration attempt! Number of tries: 2/3
2024-01-30T13:34:49.534131472Z No replica set members match selector "<function readable_server_selector at 0x7f01c7d3f4c0>", Timeout: 30s, Topology Description: <TopologyDescription id: 65b8fab855a0e44bfb8deed8, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.001968278016593526>]>
2024-01-30T13:34:54.539289351Z Refreshing prometheus config file for first time
2024-01-30T13:35:08.228357221Z Jobs detected : 0
2024-01-30T13:35:08.239919155Z Writing new config file to /etc/prometheus/prometheus.yml
2024-01-30T13:35:08.244122504Z alerting:
2024-01-30T13:35:08.244145407Z   alertmanagers:
2024-01-30T13:35:08.244149494Z   - static_configs:
2024-01-30T13:35:08.244152710Z     - targets: null
2024-01-30T13:35:08.244155768Z global:
2024-01-30T13:35:08.244158734Z   evaluation_interval: 15s
2024-01-30T13:35:08.244172804Z   scrape_interval: 15s
2024-01-30T13:35:08.244175958Z rule_files: null
2024-01-30T13:35:08.244179113Z scrape_configs:
2024-01-30T13:35:08.244182119Z - job_name: mon_exporter
2024-01-30T13:35:08.244185148Z   static_configs:
2024-01-30T13:35:08.244188164Z   - targets:
2024-01-30T13:35:08.244191589Z     - mon:8000
2024-01-30T13:35:08.244368343Z 
2024-01-30T13:35:08.249865656Z New config written, updating prometheus
2024-01-30T13:35:13.299504766Z Prometheus config update successful
2024-01-30T13:35:18.310119876Z Listening to changes in prometheus jobs collection