redis问题与解决思路Timeout receiving bulk data from MASTER if the problem persists try

admin

Timeout receiving bulk data from MASTER if the problem persists try to set the 'repl-timeout'

  p0 Y. _' n' K! h" \问题现象：

集群状态 1主 2从，主没有开启bgsave，从开启bgsave。现象所有redis可以访问，进行操作。主不断开始bgsave 1从停止bgsave。

5 B; b$ l) M2 n8 s4 A主日志报错# Connection with slave XXXX lost.

从日志报错# Timeout receiving bulk data from MASTER... If the problem persists try to set the 'repl-timeout' parameter in redis.conf to a larger value.
- M8 N. ^4 x+ Q# X
" R" ^$ m# B. V& `+ Y
1 F0 L  W( J+ j, @: W' {
/ l3 f/ S2 I4 v问题总结：
" l" U1 a/ Q" J/ s1 r9 e- k
repl-backlog-size   repl-timeout 这2个参数 每次从和主可以同步的数据大小，如果进行同步的时候超过了这个限制，就到导致如上报错。
" c$ S2 e4 `2 e! Y! Y& X+ |
, R# h! t* D( M3 z2 c

5 M/ V! }0 D- l9 \/ X, a问题现象：

* l; P. ?3 R2 `: _' U, _8 t重启从服务器，主报错Client id=1317049445 addr=10.10.3.112:7412 fd=39 name= age=394 idle=0 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=4360 omem=76118609 events=rw cmd=psync scheduled to be closed ASAP for overcoming of output buffer limits.

注意到这么一句话：psync scheduled to be closed ASAP for overcoming of output buffer limits。看起来是psync因为超过output buffer limits将被close。

3 v2 j- t6 H! L8 ?$ U于是查看了一下client-output-buffer-limit。发现这是Redis的一个保护机制。配置格式是：

client-output-buffer-limit <class> <hard limit> <soft limit> <soft seconds>
" \1 U; c6 X4 u1 i7 ]具体参数含义如下：
- }4 q& o, {/ E2 W; Q# p, W8 [
class: 客户端种类，包括Normal，Slaves和Pub/Sub
6 `& l, _. S3 v2 H( c9 X6 w0 ZNormal: 普通的客户端。默认limit 是0，也就是不限制。
! k0 h; d; i/ @0 f+ E# rPub/Sub: 发布与订阅的客户端的。默认hard limit 32M，soft limit 8M/60s。
Slaves: 从库的复制客户端。默认hard limit 256M，soft limit 64M/60s。
hard limit: 缓冲区大小的硬性限制。
; U! G7 t$ n! P, D" hsoft limit: 缓冲去大小的软性限制。
+ D; ~$ o9 f" y6 P+ W" H. csoft seconds: 缓冲区大小达到了（超过）soft limit值的持续时间。
client-output-buffer-limit参数限制分配的缓冲区的大小，防止内存无节制的分配，Redis将会做如下自我保护：
3 A7 Y! e3 Q! h  q# U3 }  T, g
! \4 ~3 F& n1 L7 i* b- h" o( F8 sclient buffer的大小达到了soft limit并持续了soft seconds时间，将立即断开和客户端的连接
client buffer的大小达到了hard limit，server也会立即断开和客户端的连接
再看看我们从库的这个配置，其实就是默认配置：
/ i+ f& f* A+ p% s
# 客户端的输出缓冲区的限制，因为某种原因客户端从服务器读取数据的速度不够快，
. x$ R( p- H3 e8 C3 V' y4 k* p3 ?7 l& B3 e! ^
) |! D$ s; `+ V# 可用于强制断开连接（一个常见的原因是一个发布 / 订阅客户端消费消息的速度无法赶上生产它们的速度）。

# 可以三种不同客户端的方式进行设置：

# normal -> 正常客户端
" u1 D) q1 T; |: W3 U) g) W# x' v! k
# slave -> slave 和 MONITOR 客户端

7 I& s$ U8 D5 {" i& H& Y) Q# pubsub -> 至少订阅了一个 pubsub channel 或 pattern 的客户端

. W: ~: H4 s6 g/ w, y* c# 每个 client-output-buffer-limit 语法 :

. u% g1 r  t. y% `( u( k4 a# client-output-buffer-limit <class><hard limit> <soft limit> <soft seconds>
; y) T* e5 K# y' r
2 N/ f0 h8 t+ c4 i7 |0 s# 一旦达到硬限制客户端会立即断开，或者达到软限制并保持达成的指定秒数（连续）。

& f8 n/ t$ S! K- U2 K# 例如，如果硬限制为 32 兆字节和软限制为 16 兆字节 /10 秒，客户端将会立即断开
9 Z2 Y8 [( w! ]8 D! j1 O- }; b1 h
# 如果输出缓冲区的大小达到 32 兆字节，客户端达到 16 兆字节和连续超过了限制 10 秒，也将断开连接。
8 i/ S+ _( [8 n. j+ g' ^
/ p9 v1 R; q2 @, }" l( O# 默认 normal 客户端不做限制，因为他们在一个请求后未要求时（以推的方式）不接收数据，
$ l: a% N1 |# ?% a. l6 F3 O& M
. K* `4 @8 ]! l- \4 U# 只有异步客户端可能会出现请求数据的速度比它可以读取的速度快的场景。

# 把硬限制和软限制都设置为 0 来禁用该特性

+ _# n$ @2 L5 R2 c1 |2 J' oclient-output-buffer-limit normal 0 0 0

1 S# R( }2 h* \6 _client-output-buffer-limit slave 256mb 64mb 60

client-output-buffer-limit pubsub 32mb 8mb 60
0 h$ C5 L  a7 E6 r1 [' Y. A
redis的replication buffer其实就是client buffer的一种。里面存放的数据是下面三个时间内所有的master数据更新操作：

master执行rdb bgsave产生snapshot的时间
master发送rdb到slave网络传输时间
slave load rdb文件把数据恢复到内存的时间
" i7 p4 h$ {- c. i* q' D; e' T! D可以看到跟replication backlog是一模一样的！

replication buffer由client-output-buffer-limit slave设置，当这个值太小会导致主从复制链接断开:

( \4 ~' g8 k7 c; Y4 g6 s3 k当master-slave复制连接断开，server端会释放连接相关的数据结构。replication buffer中的数据也就丢失了，此时主从之间重新开始复制过程。
1 O- a) j2 E8 p' B2 ]$ w& o还有个更严重的问题，主从复制连接断开，导致主从上出现rdb bgsave和rdb重传操作无限循环。
看起来确实server(这里就是master)会因为缓冲区的大小问题主动关闭客户端(slave)链接。因为我们的数据变更量太大，超过了client-output-buffer-limit。导致主从同步连接被断开，然后slave要求psync，但是由于repl-backlog-size太小，导致psync失败，需要full sync，而full sync需要Discarding previously cached master state，重新load RDB文件到内存，而这个加载数据过程是阻塞式的。所以导致slave出现间歇式的不可用。而切换到master之后，master的整个同步操作都是fork一个子进程进行的，所以不影响父进程继续服务。所有的现象都能清清楚楚的解释上。

( V; c, I+ l) z' n/ v

更改配置 client-output-buffer-limit  client-output-buffer-limit slave 0 0 0 "  重启slave问题解决

admin

集群状态 1主 2从，主没有开启bgsave，从开启bgsave。现象所有redis可以访问，进行操作。主不断开始bgsave 1从停止bgsave。

7 Q2 q5 G2 z7 U$ r主日志报错# Connection with slave XXXX lost.
: c% X) ?6 Y5 Y" j3 E; G, l6 P+ l
从日志报错# Timeout receiving bulk data from MASTER... If the problem persists try to set the 'repl-timeout' parameter in redis.conf to a larger value.

5 f/ Q6 d6 }3 ^; i' c3 |/ ^0 I

: e  G/ U7 N7 B4 n- v2 [$ w6 r问题总结：

repl-backlog-size   repl-timeout 这2个参数 每次从和主可以同步的数据大小，如果进行同步的时候超过了这个限制，就到导致如上报错。

. ~( O7 q" N$ ]  K/ K( Y% @# r- D: u9 D
/ U8 q) x- Q- |* g! N
% o' G( E1 k" p  R问题现象：

* G2 ~' j& F/ Z4 R重启从服务器，主报错Client id=1317049445 addr=10.10.3.112:7412 fd=39 name= age=394 idle=0 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=4360 omem=76118609 events=rw cmd=psync scheduled to be closed ASAP for overcoming of output buffer limits.

: m% _; H" [' p, U注意到这么一句话：psync scheduled to be closed ASAP for overcoming of output buffer limits。看起来是psync因为超过output buffer limits将被close。
# Y( b) [; t9 _* M
; m9 s/ O9 T; k) J" _) G于是查看了一下client-output-buffer-limit。发现这是Redis的一个保护机制。配置格式是：

client-output-buffer-limit <class> <hard limit> <soft limit> <soft seconds>

admin

Answer
Redis replication timeout sets the maximum amount of time the master waits for a ping response from its slave before dropping the connection. This configuration is critical to prevent potential data loss or stale reads in case of network issues or unresponsive slaves.

The repl-timeout setting in the Redis configuration file controls this value, and it's specified in seconds. By default, it's set to 60 seconds. You can change it as follows:
5 J9 ^# {7 |, ~5 D
# In your redis.conf file
repl-timeout 120
5 v2 M9 o1 M* `& q( n. o  }$ CThis sets the replication timeout to 120 seconds. Save and exit the file, then restart the Redis server for the changes to take effect.

You can also modify this setting at runtime using the CONFIG SET command:
, y/ ~: q: B* r" K
$ ]! z8 q" ^4 s; @. o% f5 F* r# Via Redis CLI
/ K9 l" \7 k( Y  k0 R$ c* vredis-cli CONFIG SET repl-timeout 120
Be cautious when adjusting this value. Setting it too low may lead to unnecessary disconnections due to transient network issues, while setting it too high may delay the detection of genuine problems with slave nodes.

admin

Here's the output from the master's log:
>
5 o1 Q. e0 |$ Z6 q, }> [9470] 13 Sep 22:24:04.789 * Slave ask for synchronization
> [9470] 13 Sep 22:24:04.789 * Starting BGSAVE for SYNC
. k0 M9 v9 H' w% c> [9470] 13 Sep 22:24:09.454 * Background saving started by pid 18435
> [9470] 13 Sep 22:26:37.105 # Client addr=10.108.61.163:44422 fd=157 age=153
0 F' t9 B! b$ I> idle=153 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=8361
) D# V  Y3 I' s0 p/ H> oll=3975 omem=100347928 events=r cmd=sync scheduled to be closed ASAP for
" q- W1 b. `+ N8 ?- c> overcoming of output buffer limits.
% Q. r5 Y# s: _& ~0 }> [18435] 13 Sep 22:29:06.199 * DB saved on disk
/ ^, K' ~5 S2 T2 _) N9 G> [9470] 13 Sep 22:29:07.138 * Background saving terminated with success
>
> The slave's view:
> [23037] 13 Sep 21:42:07.799 * The server is now ready to accept connections
> at /tmp/redis.sock
> [23037] 13 Sep 21:42:07.799 * Connecting to MASTER...
> [23037] 13 Sep 21:42:07.799 * MASTER <-> SLAVE sync started
> [23037] 13 Sep 21:42:07.800 * Non blocking connect for SYNC fired the event.
) A$ J8 T7 m6 ?5 T7 \) I& a> [23037] 13 Sep 21:45:43.167 # Timeout receiving bulk data from MASTER...
> [23037] 13 Sep 21:45:43.167 * Connecting to MASTER...
3 |3 {9 J1 H% o7 `1 b) e9 g3 k1 ]> etc.
, i" O% X+ G3 E' _4 `0 J>
2 O: n( s4 o6 F7 i. A1 f1 U" F" `+ w> And the configuration on master:
5 g+ t' e: X7 o8 e; M# r> redis-cli -p 6380 config get client-output-buffer-limit
> 1) "client-output-buffer-limit"
; I) \& ~8 X! M+ m7 h2 |6 i> 2) "normal 0 0 0 slave 268435456 67108864 60 pubsub 33554432 8388608 60"
>
- W/ Z; S+ `* D7 z' ?> So there was only 100mb in the output buffer, but it was shut down even
9 ~- W1 J+ X0 l2 ?6 f. x4 _> though the limit was 268mb. Any ideas what might be going on? Apologies if
+ I" o. _3 R" J> this is a repost.

admin

I adopted pretty much all the recommendations, including the ssh
* d( Q) Q) q. X* y7 c0 J% ^compression idea, from you two and so:
0 d$ i7 v3 b6 K7 V* ~
root@redis3:~# redis-cli config get * | grep -A2 repl
; p+ M! v/ r. b% ^  ~4 `# Arepl-ping-slave-period
# U0 i+ l% J0 T0 q10
repl-timeout
$ T! {; r, l7 B  [5 ^" w% c1800
repl-backlog-size
1048576000
+ [( U. C9 v% r. p7 y( [repl-backlog-ttl
3600
6 h) D5 C+ {0 }- X
8 L# U$ ^+ D, [5 OI evetualy also changed this, from no to yes (although I know it may
7 S6 L& x: {# \1 }increase the bandwidth requirements):
repl-disable-tcp-nodelay
( u5 p3 {/ c: p% {- kyes
$ [# z' V2 f3 `4 R' `/ X
! ^5 u1 y# N/ [& O' eBut the result is actually worse.... by that I mean the master kicks
off the slave much faster.

9 }, m% x7 `! \9 R, i9 MMaster:
[3355] 30 May 02:56:28.875 * Slave asks for synchronization
[3355] 30 May 02:56:28.875 * Full resync requested by slave.
6 v3 m4 u* D, F# c  L[3355] 30 May 02:56:28.875 * Starting BGSAVE for SYNC
3 U1 L2 `3 Q! w1 ]8 S% c+ i0 O8 N: F% g[3355] 30 May 02:56:35.376 * Background saving started by pid 16330
2 a5 ?2 m5 A6 Z  P4 m+ ^& C9 ^' }[3355] 30 May 02:56:43.733 # Client addr=127.0.0.1:49630 fd=185 name=
* ]# M7 S1 ]" O# y9 y6 ^) u7 A& J1 y4 tage=15 idle=15 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0
) [2 y1 y- {/ O: M9 \% eobl=16372 oll=2896 omem=268753696 events=r cmd=psync scheduled to be
6 X, @" S% U5 H* f/ ^closed ASAP for overcoming of output buffer limits.
2 i/ [$ _# }5 f% W7 y1 G4 `[16330] 30 May 02:59:44.675 * DB saved on disk
[16330] 30 May 02:59:46.216 * RDB: 2397 MB of memory used by copy-on-write
[3355] 30 May 02:59:48.806 * Background saving terminated with success
...

: m# `  O1 D4 D$ [Slave:
4 ^3 s  V. N% X[19460] 30 May 02:56:27.975 * DB loaded from disk: 210.107 seconds
[19460] 30 May 02:56:27.976 * The server is now ready to accept
connections on port 6379
9 J' U$ [* Y4 z9 g) ]1 y- D[19460] 30 May 02:56:28.869 * Connecting to MASTER localhost:6280
) k8 ]! Q8 T! Q2 J& w[19460] 30 May 02:56:28.869 * MASTER <-> SLAVE sync started
1 j# {3 q! @6 |5 Z! F[19460] 30 May 02:56:28.870 * Non blocking connect for SYNC fired the event.
[19460] 30 May 02:56:28.873 * Master replied to PING, replication can
continue...
[19460] 30 May 02:56:28.875 * Partial resynchronization not possible
(no cached master)
[19460] 30 May 02:56:28.877 * Full resync from master:
2 }& Z5 U4 i" g; F" v/ Y621480e9295872416266e563939b4fd6724eb5b7:68385253408
( W& c( F% i- _...
' \) w5 z$ P/ I- K
; v# h# |: u0 x0 _( D, r% @! TSo, 3 questions please:
& ^; U2 y. b% }( J: b' c; M
" j& k8 s% f& o# X# s7 @1. Do I need to make the same and all of the configurations on the slave too?
9 o1 q  x& y; C& x) c
2. Can the "Partial resynchronization not possible (no cached master)"
be overcome somehow? I have full backups via BGSAVE of the master each
hour (yes, I stopped those while attempting the slaving stuff) so I
could use the latest one to load it masnually in the salve and then
4 Y9 l8 P' b; ~. [+ j  y6 Jattach the slave to the master - last time I tried that however, the
$ d) U. m. D9 I6 [7 x" zslave still requested a full resync....