오류: semaphore wait has lasted > 600 seconds 데이터베이스 인스턴스 재부팅

생산 데이터베이스 실례가 밤에 갑자기 재개됐어요.

1. 환경:

DB version:mariadb 10.0.28 x64
OS version:centos6.6 x64
kernel:2.6.32-504.el6.x86_64
시스템sem:kernel.sem = 1000 40960001000 4096

2.error log

InnoDB: ###### Diagnostic info printed to the standard error stream
InnoDB: Warning: a long semaphore wait:
--Thread 139562561287936 has waited at row0ins.cc line 2730 for 923.00 seconds the semaphore:
X-lock (wait_ex) on RW-latch at 0x7ef7b756b4a0 '&block->lock'
a writer (thread id 139562561287936) has reserved it in mode  wait exclusive
number of readers 1, waiters flag 1, lock_word: ffffffffffffffff
Last time read locked in file row0sel.cc line 4152
Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.28/storage/xtradb/row/row0ins.cc line 2730
InnoDB: Warning: a long semaphore wait:
--Thread 139562523481856 has waited at row0ins.cc line 2730 for 921.00 seconds the semaphore:
X-lock (wait_ex) on RW-latch at 0x7ef9c7eb5500 '&block->lock'
a writer (thread id 139562523481856) has reserved it in mode  wait exclusive
number of readers 1, waiters flag 1, lock_word: ffffffffffffffff
Last time read locked in file buf0flu.cc line 1069
Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.28/storage/xtradb/btr/btr0sea.cc line 979
InnoDB: Warning: a long semaphore wait:
--Thread 139562609211136 has waited at row0ins.cc line 2730 for 911.00 seconds the semaphore:
X-lock (wait_ex) on RW-latch at 0x7ef7a6ef37e0 '&block->lock'
a writer (thread id 139562609211136) has reserved it in mode  wait exclusive
number of readers 1, waiters flag 0, lock_word: ffffffffffffffff
Last time read locked in file buf0flu.cc line 1069
Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.28/storage/xtradb/row/row0ins.cc line 2730
InnoDB: Warning: a long semaphore wait:
--Thread 139562551437056 has waited at trx0undo.ic line 171 for 907.00 seconds the semaphore:
X-lock (wait_ex) on RW-latch at 0x7ef9c5d1e4e0 '&block->lock'
a writer (thread id 139562551437056) has reserved it in mode  wait exclusive
number of readers 1, waiters flag 0, lock_word: ffffffffffffffff
Last time read locked in file buf0flu.cc line 1069
Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.28/storage/xtradb/include/trx0undo.ic line 171
InnoDB: Warning: a long semaphore wait:
--Thread 139560391223040 has waited at row0ins.cc line 2730 for 906.00 seconds the semaphore:
X-lock on RW-latch at 0x7ef9c7eb5500 '&block->lock'
a writer (thread id 139562523481856) has reserved it in mode  wait exclusive
number of readers 1, waiters flag 1, lock_word: ffffffffffffffff
Last time read locked in file buf0flu.cc line 1069
Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.28/storage/xtradb/btr/btr0sea.cc line 979

InnoDB: ###### Diagnostic info printed to the standard error stream
InnoDB: Error: semaphore wait has lasted > 600 seconds
InnoDB: We intentionally crash the server, because it appears to be hung.
2017-03-30 21:11:18 7eee7d9fe700  InnoDB: Assertion failure in thread 139562774947584 in file srv0srv.cc line 2222
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.

3. 임시 방안

매뉴얼을 볼 때 해시 색인에 적응하면 btr0sea가 발생할 수 있습니다.c 파일의 rw latch 가져오기 쟁탈SEMAPHORES 。
자세한 링크:https://dev.mysql.com/doc/refman/5.7/en/innodb-adaptive-hash.html.
임시 솔루션: set global innodb_adaptive_hash_index=0;

4. 후속 처리

mariadb에 버그를 하나 냈는데 centos6.6에 구덩이가 있는 것을 발견했다.Haswell-based Servers는centos6.6내 핵하에서 hang사를 초래할 수 있다.세부 링크
https://www.infoq.com/news/2015/05/redhat-futex
https://groups.google.com/forum/?hl=zh-Cn#!starred/codership-team/Ne6WsTWixH8
물론 후속적으로 문제가 생기면 gdbdump에서 서류를 내서 정부에 제공하여 연구를 진행하기를 바랍니다.