Very slow Consul server may cause locking failures #45

New issue

Open

opened 2023-11-02 19:54:13 +02:00 by max · 2 comments

max commented

2023-11-02 19:54:13 +02:00

(Migrated from git.privatevoid.net)

2023-11-02T17:50:42.762+0200 [WARN]  agent.server.kvs: Rejecting lock of key due to lock-delay: key=services/grafana/.lock

``` 2023-11-02T17:50:42.762+0200 [WARN] agent.server.kvs: Rejecting lock of key due to lock-delay: key=services/grafana/.lock ```

max commented

2024-11-23 09:41:49 +02:00

Owner

Possibly related:

consul[3137667]: Lock acquisition failed: failed to read lock: Unexpected response code: 500 (Raft leader not found in server lookup mapping)

Possibly related: ``` consul[3137667]: Lock acquisition failed: failed to read lock: Unexpected response code: 500 (Raft leader not found in server lookup mapping) ```

max commented

2024-12-04 01:33:55 +02:00

Owner

Probably the reason why tempo.service on grail dies randomly.

2024-12-03 23:01:29.364	
Deregistered service: tempo
2024-12-03 23:01:30.473	
Error running handler: signal: killed
2024-12-03 23:01:31.684	
level=info ts=2024-12-03T22:01:31.684516558Z caller=registry.go:253 tenant=single-tenant msg="collecting metrics" active_series=597
2024-12-03 23:01:32.423	
level=info ts=2024-12-03T22:01:32.423260672Z caller=flush.go:180 msg="head block cut. enqueueing flush op" userid=single-tenant block=94d35032-fa42-4dbf-ada0-9e08bb946582
2024-12-03 23:01:39.031	
level=info ts=2024-12-03T22:01:39.031788548Z caller=flush.go:256 msg="completing block" userid=single-tenant blockID=94d35032-fa42-4dbf-ada0-9e08bb946582
2024-12-03 23:01:39.135	
level=info ts=2024-12-03T22:01:39.13556989Z caller=flush.go:263 msg="block completed" userid=single-tenant blockID=94d35032-fa42-4dbf-ada0-9e08bb946582 duration=103.782102ms
2024-12-03 23:01:39.136	
level=info ts=2024-12-03T22:01:39.13638905Z caller=flush.go:312 msg="flushing block" userid=single-tenant block=94d35032-fa42-4dbf-ada0-9e08bb946582
2024-12-03 23:01:40.565	
Lock release failed: failed to release lock: Unexpected response code: 500 (No cluster leader)
2024-12-03 23:01:40.573	
level=info ts=2024-12-03T22:01:40.572257876Z caller=app.go:236 msg="=== received SIGINT/SIGTERM ===\n*** exiting"

Probably the reason why `tempo.service` on `grail` dies randomly. ``` 2024-12-03 23:01:29.364 Deregistered service: tempo 2024-12-03 23:01:30.473 Error running handler: signal: killed 2024-12-03 23:01:31.684 level=info ts=2024-12-03T22:01:31.684516558Z caller=registry.go:253 tenant=single-tenant msg="collecting metrics" active_series=597 2024-12-03 23:01:32.423 level=info ts=2024-12-03T22:01:32.423260672Z caller=flush.go:180 msg="head block cut. enqueueing flush op" userid=single-tenant block=94d35032-fa42-4dbf-ada0-9e08bb946582 2024-12-03 23:01:39.031 level=info ts=2024-12-03T22:01:39.031788548Z caller=flush.go:256 msg="completing block" userid=single-tenant blockID=94d35032-fa42-4dbf-ada0-9e08bb946582 2024-12-03 23:01:39.135 level=info ts=2024-12-03T22:01:39.13556989Z caller=flush.go:263 msg="block completed" userid=single-tenant blockID=94d35032-fa42-4dbf-ada0-9e08bb946582 duration=103.782102ms 2024-12-03 23:01:39.136 level=info ts=2024-12-03T22:01:39.13638905Z caller=flush.go:312 msg="flushing block" userid=single-tenant block=94d35032-fa42-4dbf-ada0-9e08bb946582 2024-12-03 23:01:40.565 Lock release failed: failed to release lock: Unexpected response code: 500 (No cluster leader) 2024-12-03 23:01:40.573 level=info ts=2024-12-03T22:01:40.572257876Z caller=app.go:236 msg="=== received SIGINT/SIGTERM ===\n*** exiting" ```