2024-01-08T21:31:03.170407563Z level=info ts=2024-01-08T21:31:03.147Z caller=main.go:389 msg="No time or size retention was set so using the default time retention" duration=15d
2024-01-08T21:31:03.170445847Z level=info ts=2024-01-08T21:31:03.148Z caller=main.go:443 msg="Starting Prometheus" version="(version=2.28.1, branch=HEAD, revision=b0944590a1c9a6b35dc5a696869f75f422b107a1)"
2024-01-08T21:31:03.170450072Z level=info ts=2024-01-08T21:31:03.148Z caller=main.go:448 build_context="(go=go1.16.5, user=root@2915dd495090, date=20210701-15:20:10)"
2024-01-08T21:31:03.170453329Z level=info ts=2024-01-08T21:31:03.148Z caller=main.go:449 host_details="(Linux 5.4.0-107-generic #121-Ubuntu SMP Thu Mar 24 16:04:27 UTC 2022 x86_64 prometheus-0 (none))"
2024-01-08T21:31:03.170456669Z level=info ts=2024-01-08T21:31:03.148Z caller=main.go:450 fd_limits="(soft=1048576, hard=1048576)"
2024-01-08T21:31:03.170459647Z level=info ts=2024-01-08T21:31:03.148Z caller=main.go:451 vm_limits="(soft=unlimited, hard=unlimited)"
2024-01-08T21:31:03.170462675Z level=info ts=2024-01-08T21:31:03.168Z caller=web.go:541 component=web msg="Start listening for connections" address=0.0.0.0:9090
2024-01-08T21:31:03.173993934Z level=info ts=2024-01-08T21:31:03.172Z caller=main.go:824 msg="Starting TSDB ..."
2024-01-08T21:31:03.186476932Z level=info ts=2024-01-08T21:31:03.183Z caller=head.go:780 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
2024-01-08T21:31:03.192790240Z level=info ts=2024-01-08T21:31:03.183Z caller=head.go:794 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=5.633µs
2024-01-08T21:31:03.192848088Z level=info ts=2024-01-08T21:31:03.183Z caller=head.go:800 component=tsdb msg="Replaying WAL, this may take a while"
2024-01-08T21:31:03.193567324Z level=info ts=2024-01-08T21:31:03.189Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
2024-01-08T21:31:03.195790487Z level=info ts=2024-01-08T21:31:03.189Z caller=head.go:860 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=20.259µs wal_replay_duration=5.732998ms total_replay_duration=5.778383ms
2024-01-08T21:31:03.195801334Z level=info ts=2024-01-08T21:31:03.189Z caller=tls_config.go:191 component=web msg="TLS is disabled." http2=false
2024-01-08T21:31:03.195805739Z level=info ts=2024-01-08T21:31:03.191Z caller=main.go:851 fs_type=EXT4_SUPER_MAGIC
2024-01-08T21:31:03.195809911Z level=info ts=2024-01-08T21:31:03.191Z caller=main.go:854 msg="TSDB started"
2024-01-08T21:31:03.195813711Z level=info ts=2024-01-08T21:31:03.191Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-01-08T21:31:03.195818289Z level=info ts=2024-01-08T21:31:03.191Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=880.748µs remote_storage=2.899µs web_handler=507ns query_engine=4.916µs scrape=462.302µs scrape_sd=28.649µs notify=44.72µs notify_sd=10.092µs rules=4.115µs
2024-01-08T21:31:03.195829524Z level=info ts=2024-01-08T21:31:03.191Z caller=main.go:796 msg="Server is ready to receive web requests."
2024-01-08T21:36:05.621465797Z level=info ts=2024-01-08T21:36:05.621Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-01-08T21:36:05.623171626Z level=info ts=2024-01-08T21:36:05.622Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=1.845089ms remote_storage=176.028µs web_handler=1.258µs query_engine=6.132µs scrape=269.923µs scrape_sd=65.332µs notify=69.421µs notify_sd=14.125µs rules=7.155µs
2024-01-08T21:36:15.685335502Z level=info ts=2024-01-08T21:36:15.684Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
2024-01-08T21:36:15.686122490Z level=info ts=2024-01-08T21:36:15.685Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=1.034802ms remote_storage=3.9µs web_handler=874ns query_engine=2.389µs scrape=177.545µs scrape_sd=50.299µs notify=38.484µs notify_sd=20.622µs rules=2.116µs
2024-01-08T21:35:20.820319422Z Created MongoClient to connect to MongoDB!
2024-01-08T21:35:20.821856543Z Refreshing prometheus config file for first time
2024-01-08T21:35:50.853761671Z Error in configuration attempt! Number of tries: 1/3
2024-01-08T21:35:50.853811009Z No replica set members match selector "<function readable_server_selector at 0x7f743eaee4c0>", Timeout: 30s, Topology Description: <TopologyDescription id: 659c6a98f7e764cbe36b6f51, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongodb-k8s', 27017) server_type: RSGhost, rtt: 0.002081256039964501>]>
2024-01-08T21:35:55.856187043Z Refreshing prometheus config file for first time
2024-01-08T21:36:05.593358458Z Jobs detected : 0
2024-01-08T21:36:05.605683980Z Writing new config file to /etc/prometheus/prometheus.yml
2024-01-08T21:36:05.607132419Z alerting:
2024-01-08T21:36:05.607145314Z   alertmanagers:
2024-01-08T21:36:05.607149028Z   - static_configs:
2024-01-08T21:36:05.607152335Z     - targets: null
2024-01-08T21:36:05.607155510Z global:
2024-01-08T21:36:05.607158866Z   evaluation_interval: 15s
2024-01-08T21:36:05.607162043Z   scrape_interval: 15s
2024-01-08T21:36:05.607165197Z rule_files: null
2024-01-08T21:36:05.607168356Z scrape_configs:
2024-01-08T21:36:05.607171590Z - job_name: mon_exporter
2024-01-08T21:36:05.607174694Z   static_configs:
2024-01-08T21:36:05.607196103Z   - targets:
2024-01-08T21:36:05.607199430Z     - mon:8000
2024-01-08T21:36:05.607202669Z 
2024-01-08T21:36:05.610292015Z New config written, updating prometheus
2024-01-08T21:36:10.646579368Z Prometheus config update successful
2024-01-08T21:36:15.652160223Z Refreshing prometheus config file for first time
2024-01-08T21:36:15.654891630Z Jobs detected : 1
2024-01-08T21:36:15.675510715Z Writing new config file to /etc/prometheus/prometheus.yml
2024-01-08T21:36:15.677885658Z alerting:
2024-01-08T21:36:15.677951649Z   alertmanagers:
2024-01-08T21:36:15.677963931Z   - static_configs:
2024-01-08T21:36:15.677974633Z     - targets: null
2024-01-08T21:36:15.677984788Z global:
2024-01-08T21:36:15.677994600Z   evaluation_interval: 15s
2024-01-08T21:36:15.678004099Z   scrape_interval: 15s
2024-01-08T21:36:15.678013901Z rule_files: null
2024-01-08T21:36:15.678023882Z scrape_configs:
2024-01-08T21:36:15.678033209Z - job_name: mon_exporter
2024-01-08T21:36:15.678044912Z   static_configs:
2024-01-08T21:36:15.678059087Z   - targets:
2024-01-08T21:36:15.678076626Z     - mon:8000
2024-01-08T21:36:15.678093016Z 
2024-01-08T21:36:15.680481483Z New config written, updating prometheus
2024-01-08T21:36:20.715122646Z Prometheus config update successful
2024-01-08T21:36:25.727005467Z Listening to changes in prometheus jobs collection