Percona XtraDB Cluster: Failed to report last committed , -110 (Connection timed out)

I’m using Percona-XtraDB-Cluster-server-56 with Version : 5.6.39 Release : 26.25.1.el6.

with 2 node PXC + 1 percona-xtradb-cluster-garbd.

I’m facing following problem. PXC log report a a lot of warning

2018-11-19 17:33:48 19414 [Warning] WSREP: Failed to report last committed 1327742612, -110 (Connection timed out) 2018-11-19 19:07:06 19414 [Warning] WSREP: Failed to report last committed 1327963155, -110 (Connection timed out) 2018-11-19 19:26:58 19414 [Warning] WSREP: Failed to report last committed 1328000375, -110 (Connection timed out) 2018-11-19 21:11:49 19414 [Warning] WSREP: Failed to report last committed 1328209550, -110 (Connection timed out) 2018-11-19 23:44:14 19414 [Warning] WSREP: Failed to report last committed 1328381758, -110 (Connection timed out) 2018-11-20 09:31:48 19414 [Warning] WSREP: Failed to report last committed 1328978200, -110 (Connection timed out) 2018-11-20 09:43:42 19414 [Warning] WSREP: Failed to report last committed 1329040974, -110 (Connection timed out) 2018-11-20 09:46:06 19414 [Warning] WSREP: Failed to report last committed 1329052427, -110 (Connection timed out) 2018-11-20 10:20:36 19414 [Warning] WSREP: Failed to report last committed 1329227873, -110 (Connection timed out) 2018-11-20 10:28:20 19414 [Warning] WSREP: Failed to report last committed 1329265424, -110 (Connection timed out) 2018-11-20 11:03:22 19414 [Warning] WSREP: Failed to report last committed 1329453317, -110 (Connection timed out) 2018-11-20 11:23:47 19414 [Warning] WSREP: Failed to report last committed 1329559539, -110 (Connection timed out) 2018-11-20 11:25:00 19414 [Warning] WSREP: Failed to report last committed 1329564657, -110 (Connection timed out) 2018-11-20 11:33:53 19414 [Warning] WSREP: Failed to report last committed 1329604492, -110 (Connection timed out) 2018-11-20 11:57:34 19414 [Warning] WSREP: Failed to report last committed 1329715728, -110 (Connection timed out) 2018-11-20 13:42:23 19414 [Warning] WSREP: Failed to report last committed 1330147628, -110 (Connection timed out) 2018-11-20 13:43:00 19414 [Warning] WSREP: Failed to report last committed 1330149773, -110 (Connection timed out) 2018-11-20 14:27:01 19414 [Warning] WSREP: Failed to report last committed 1330379363, -110 (Connection timed out) 2018-11-20 14:30:12 19414 [Warning] WSREP: Failed to report last committed 1330391879, -110 (Connection timed out) 2018-11-20 14:32:14 19414 [Warning] WSREP: Failed to report last committed 1330401771, -110 (Connection timed out) 2018-11-20 14:34:41 19414 [Warning] WSREP: Failed to report last committed 1330411552, -110 (Connection timed out) 2018-11-20 14:35:15 19414 [Warning] WSREP: Failed to report last committed 1330413554, -110 (Connection timed out) 2018-11-20 14:36:37 19414 [Warning] WSREP: Failed to report last committed 1330419814, -110 (Connection timed out) 2018-11-20 14:37:26 19414 [Warning] WSREP: Failed to report last committed 1330422477, -110 (Connection timed out) 2018-11-20 14:41:02 19414 [Warning] WSREP: Failed to report last committed 1330437542, -110 (Connection timed out) 2018-11-20 14:50:49 19414 [Warning] WSREP: Failed to report last committed 1330484259, -110 (Connection timed out) 2018-11-20 14:57:32 19414 [Warning] WSREP: Failed to report last committed 1330517211, -110 (Connection timed out) 2018-11-20 15:01:34 19414 [Warning] WSREP: Failed to report last committed 1330536212, -110 (Connection timed out) 2018-11-20 15:43:19 19414 [Warning] WSREP: Failed to report last committed 1330769642, -110 (Connection timed out) 2018-11-20 15:50:59 19414 [Warning] WSREP: Failed to report last committed 1330808893, -110 (Connection timed out) 2018-11-20 15:59:07 19414 [Warning] WSREP: Failed to report last committed 1330848316, -110 (Connection timed out) 2018-11-20 16:08:41 19414 [Warning] WSREP: Failed to report last committed 1330891310, -110 (Connection timed out) 2018-11-20 16:10:22 19414 [Warning] WSREP: Failed to report last committed 1330898081, -110 (Connection timed out) 2018-11-20 16:13:43 19414 [Warning] WSREP: Failed to report last committed 1330913054, -110 (Connection timed out) 2018-11-20 16:19:46 19414 [Warning] WSREP: Failed to report last committed 1330940790, -110 (Connection timed out) 2018-11-20 16:23:10 19414 [Warning] WSREP: Failed to report last committed 1330954782, -110 (Connection timed out) 2018-11-20 16:23:53 19414 [Warning] WSREP: Failed to report last committed 1330957407, -110 (Connection timed out) 2018-11-20 16:26:01 19414 [Warning] WSREP: Failed to report last committed 1330967791, -110 (Connection timed out) 2018-11-20 16:29:57 19414 [Warning] WSREP: Failed to report last committed 1330984126, -110 (Connection timed out) 2018-11-20 16:32:22 19414 [Warning] WSREP: Failed to report last committed 1330994398, -110 (Connection timed out) 2018-11-20 16:40:44 19414 [Warning] WSREP: Failed to report last committed 1331034549, -110 (Connection timed out) 2018-11-20 16:49:01 19414 [Warning] WSREP: Failed to report last committed 1331068494, -110 (Connection timed out) 2018-11-20 17:09:53 19414 [Warning] WSREP: Failed to report last committed 1331161202, -110 (Connection timed out) 2018-11-20 17:10:53 19414 [Warning] WSREP: Failed to report last committed 1331164772, -110 (Connection timed out) 

It make any SQL transaction like insert and update in commit stage very slow everytime that log occur, usual take 10-11 sec to finish. Normaly that sql only take less then 1 second.

The ping between 2 node PXC is less than 0.2ms, and no drop package.

Following is my.conf. (similar on 2 node)

[client] default-character-set = utf8mb4 [mysql] default-character-set = utf8mb4 # CLIENT # port = 3306 socket = /u01/mysql-data/mysql/mysql.sock [mysqld] character-set-client-handshake = TRUE init-connect='SET NAMES utf8mb4' character-set-server = utf8mb4 collation-server = utf8mb4_unicode_ci  lower_case_table_names=1 # Disabling symbolic-links is recommended to prevent assorted security risks symbolic-links=0  # Path to Galera library wsrep_provider=/usr/lib64/libgalera_smm.so  # Cluster connection URL contains IPs of node#1, node#2 wsrep_cluster_address=gcomm://192.168.5.50,192.168.5.123,192.168.5.124 # In order for Galera to work correctly binlog format should be ROW binlog_format=ROW # MyISAM storage engine has only experimental support default_storage_engine=InnoDB # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera innodb_autoinc_lock_mode=2 # Node #6 address wsrep_node_address=192.168.5.124 # Cluster name wsrep_cluster_name=my_centos_cluster # SST method wsrep_sst_method=xtrabackup-v2 #Authentication for SST method wsrep_sst_auth="sstuser:secret" # GENERAL # user = mysql default-storage-engine = InnoDB socket = /u01/mysql-data/mysql/mysql.sock pid-file = /u01/mysql-data/mysql/mysql.pid # MyISAM # key-buffer-size = 32M myisam-recover-options = FORCE,BACKUP # SAFETY # max-allowed-packet = 16M max-connect-errors = 1000000 sql-mode = ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE # DATA STORAGE # datadir = /u01/mysql-data/mysql/ # BINARY LOGGING # log-bin = /u01/mysql-data/mysql/mysql-bin expire-logs-days = 14 sync-binlog = 1 # CACHES AND LIMITS # tmp-table-size = 32M max-heap-table-size = 32M query-cache-type = 0 query-cache-size = 0 max-connections = 1000 thread-cache-size = 200 open-files-limit = 65535 table-definition-cache = 4096 table-open-cache = 4096 # INNODB # innodb-flush-method = O_DIRECT innodb-log-files-in-group = 2 innodb-log-file-size = 512M innodb-flush-log-at-trx-commit = 1 innodb-file-per-table = 1 innodb-buffer-pool-size = 42G # LOGGING # log-error = /u01/mysql-data/log/mysql-error.log log-queries-not-using-indexes = 1 slow-query-log = 0 slow-query-log-file = /u01/mysql-data/log/mysql-slow.log tmpdir = /u01/mysql-data/tmp-dir 

Did I missing some config? How to solve that problem?