Replication unstable and crashing with "terminated with reason: no match of right hand value"

[amaltaro]

I saw a similar discussion in the repository, but it looks like that was because of no space left in the volume.

I am running CouchDB 3.2.2 with docker image and I have a couple of databases being replicated. Apparently for the last month or so, database replications are fairly unstable and I see many crashes like:

[error] 2024-06-21T17:27:31.189103Z [email protected] <0.12603.0> -------- gen_server <0.12603.0> terminated with reason: no match of right hand value {error,{options,{socket_options,[{nodelay,true},{keepalive,true}],einval}}} at ibrowse_http_client:send_req_1/8(line:981) <= gen_server:try_handle_call/4(line:661) <= gen_server:handle_msg/6(line:690) <= proc_lib:init_p_do_apply/3(line:249)

on the node that creates the replication documents.

This is preceded by a stacktrace and timeout replication errors, e.g.:

[error] 2024-06-21T17:27:31.184332Z [email protected] <0.14490.0> -------- Replicator, request GET to "https://***.cern.ch/couchdb/workqueue/9c2ce35b073c5f9c4cfda28a196e5cfb?revs=true&open_revs=%5B%2216
7-7ed0af9669fe0843ba85d9c3a2cba347%22%5D&latest=true" failed due to error timeout

but this timeout cannot be seen in the other node and Apache frontend either. On the other side, the request went through without any issues:

[notice] 2024-06-21T17:12:30.617564Z nonode@nohost <0.17336.7498> 2ae6533081 ***.cern.ch:5984 ip_adress undefined GET /workqueue/9c2ce35b073c5f9c4cfda28a196e5cfb?revs=true&open_revs=%5B%22167-7ed0af9669fe0843ba85d9c3a2cba347%22%5D&latest=true 200 ok 3

Does anyone understand what the problem could be and/or how I could investigate these instabilities?