mongodb 集群崩溃问题

雨翔河

首页列表关于

mongodb 集群崩溃问题

2020-08-31 03:09

2020-08-28 13:44-13:45  某mongodb分片集群、路由服务是三个节点mongos进程几乎同时coredump运行崩溃。

查看系统日志发现mongos进程coredump的信息、由于操作系统默认没有打开保存coredump文件开关无法进行具体的原因分析。

```
Aug 28 01:44:54 xxx-mongodb-xxx-p-l-1 abrt-hook-ccpp: Process 24066 (mongos) of user 800 killed by SIGABRT - dumping core

Aug 28 01:44:55 xxx-mongodb-xxx-p-l-1 abrt-hook-ccpp: Failed to create core_backtrace: waitpid failed: No child processes

Aug 28 01:44:55 xxx-mongodb-xxx-p-l-1 abrt-server: Executable '/xxx/bin/mongos' doesn't belong to any package and ProcessUnpackaged is set to 'no'

Aug 28 01:44:55 xxx-mongodb-xxx-p-l-1 abrt-server: 'post-create' on '/xxx/xxx/ccpp-2020-08-28-01:44:54-24066' exited with 1

Aug 28 01:44:55 xxx-mongodb-xxx-p-l-1 abrt-server: Deleting problem directory '/xxx/xxx/ccpp-2020-08-28-01:44:54-24066 
```

分析mongos打印日志信息,在mongodb开发者社区找到对应的程序Bug、由于刷新签名密钥时客户端断开连接、mongos崩溃

```
mongos进程coredump时输出日志信息:
  
	2020-08-28T01:44:54.104-0400 I  -        [conn463501] operation was interrupted because a client disconnected

2020-08-28T01:44:54.129-0400 F  -        [conn463501] terminate() called. No exception is active 0x56215160bca1 0x56215160ba58 0x5621517161b6 0x562151
	7161f1 0x5621508644ec 0x562150aeaf09 0x562150aeb4bd 0x562150a09d10 0x562150a2e4cc 0x562150a2861f 0x562150a2b8ac 0x562150dab2f2 0x562150a2601d 0x562150
	a292d3 0x562150a276e7 0x562150a2857b 0x562150a2b8ac 0x562150dab75b 0x5621514ae494 0x7f8156c5fdd5 0x7f8156988ead</br>

----- BEGIN BACKTRACE -----
	 mongos(_ZN5mongo15printStackTraceERSo+0x41) [0x56215160bca1]
	 mongos(+0x132FA58) [0x56215160ba58]
	 mongos(_ZN10__cxxabiv111__terminateEPFvvE+0x6) [0x5621517161b6]
	 mongos(+0x143A1F1) [0x5621517161f1]
	 mongos(+0x5884EC) [0x5621508644ec]
	 mongos(+0x80EF09) [0x562150aeaf09]
	 mongos(_ZN5mongo8Strategy13clientCommandEPNS_16OperationContextERKNS_7MessageE+0x1FD) [0x562150aeb4bd]
	 mongos(_ZN5mongo23ServiceEntryPointMongos13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x3D0) [0x562150a09d10]
	 mongos(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xEC) [0x562150a2e4cc]
     libc.so.6(clone+0x6D) [0x7f8156988ead]
     -----  END BACKTRACE  -----

```

### 问题参考

[https://developer.mongodb.com/community/forums/t/mongos-service-crashed/6180](https://developer.mongodb.com/community/forums/t/mongos-service-crashed/6180)

[https://jira.mongodb.org/browse/SERVER-47553](https://jira.mongodb.org/browse/SERVER-47553)

### 临时解决方案

进程服务终止、重新拉起服务

### 根本解决方案

升级版本 v4.2.7

类型：工作标签：mongodb,集群,崩溃