排查Apache报502问题，error_log里面的表现就是一条child pid 20504 exit signal Segmentation fault (11)，apache的一个进程中断了，程序中断的很莫名其妙，因为业务逻辑是没有问题，业务上的日志也没有任何比较好的反馈，apache的日志非常简洁，没有提供什么有用的信息，如何更高效的排查这个问题呢？针对已经在运行的进程如何来调试排查，本文利用gdb来快速定位这个问题。
GDB简介
GDB是GCC的调试工具。主要提供了以下几项功能， 启动您的程序，并列出可能会影响它运行的一些信息；可以让调试的程序在你所指定的断点处停住；可以在程序停住时检查此时程序发生了什么；对程序做出相应的调整，这样您就能尝试纠正一个错误并继续发现其它错误
利用GDB命令排查问题
获取进程ID
使用ps命令获取进程号：
ps -aux | grep httpd
nobody 4254 12066 0 Nov23 ? 00:00:04 /usr/local/apache2/bin/httpd -k start
nobody 4545 12066 0 Nov23 ? 00:00:04 /usr/local/apache2/bin/httpd -k start
nobody 27191 12066 0 Nov15 ? 00:00:10 /usr/local/apache2/bin/httpd -k start
root 12066 1 0 Nov13 ? 00:00:00 /usr/local/apache2/bin/httpd -k start
attach附着进程命令
进入gdb命令：
sudo gdb
GNU gdb Fedora (6.8-37.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
attach 进程：
(gdb) attach 27191
Attaching to process 27191
进程进入trace状态
break命令设置断点
设置断点
(gdb) break 1
Breakpoint 1 at 0x2b1d95f0cf60: file /usr/src/redhat/BUILD/php-5.2.5/ext/soap/soap.c, line 1.
continue命令进程继续
让程序退出trace，继续运行
(gdb) continue
Continuing.
bt回溯命令
终端上面显示遇到received signal SIGSEGV, Segmentation fault.
Program received signal SIGSEGV, Segmentation fault.
0x00002b1d8d2174e4 in engine_unlocked_finish () from /usr/local/openssl/lib/libcrypto.so.1.0.0
回溯整个调用栈，可以发现是在libcrypto.so.1.0.0的方法engine_unlocked_finish遇到问题
(gdb) bt
#0 0x00002b1d8d2174e4 in engine_unlocked_finish () from /usr/local/openssl/lib/libcrypto.so.1.0.0
#1 0x00002b1d8d2175d1 in ENGINE_finish () from /usr/local/openssl/lib/libcrypto.so.1.0.0
#2 0x00002b1d8d22c4a5 in EVP_DigestInit_ex () from /usr/local/openssl/lib/libcrypto.so.1.0.0
#3 0x00000032eca238e4 in ssl23_connect () from /lib64/libssl.so.6
#4 0x00002b1d93bc7192 in ?? () from /usr/lib64/libcurl.so.3
#5 0x00002b1d93bc7a85 in Curl_ossl_connect () from /usr/lib64/libcurl.so.3
#6 0x00002b1d93bb7339 in Curl_http_connect () from /usr/lib64/libcurl.so.3
#7 0x00002b1d93bbe92a in Curl_protocol_connect () from /usr/lib64/libcurl.so.3
#8 0x00002b1d93bc0a4f in Curl_connect () from /usr/lib64/libcurl.so.3
#9 0x00002b1d93bcbfd1 in ?? () from /usr/lib64/libcurl.so.3
#10 0x00002b1d93bce14a in Curl_perform () from /usr/lib64/libcurl.so.3
#11 0x00002b1d9399c1fb in zif_curl_exec (ht=<value optimized out>, return_value=0xc363140, return_value_ptr=<value optimized out>, this_ptr=<value optimized out>, return_value_used=<value optimized out>) at /usr/src/redhat/BUILD/php-5.2.5/ext/curl/interface.c:1693
#12 0x00002b1d8ed34792 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fff0944ac20) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:200
#13 0x00002b1d8ed3382c in execute (op_array=0xc361b58) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:92
#14 0x00002b1d8ed34241 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fff0944afa0) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:234
#15 0x00002b1d8ed3382c in execute (op_array=0xc361318) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:92
#16 0x00002b1d8ed34241 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fff0944b400) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:234
#17 0x00002b1d8ed3382c in execute (op_array=0xc360ad8) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:92
#18 0x00002b1d8ed34241 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fff0944be90) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:234
#19 0x00002b1d8ed3382c in execute (op_array=0xc23c0c0) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:92
#20 0x00002b1d8ed34241 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fff0944c0d0) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:234
#21 0x00002b1d8ed3382c in execute (op_array=0xc230ea8) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:92
#22 0x00002b1d8ed34241 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fff0944c370) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:234
#23 0x00002b1d8ed3382c in execute (op_array=0xc21b768) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:92
#24 0x00002b1d8ed34241 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fff0944c640) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:234
#25 0x00002b1d8ed3382c in execute (op_array=0xc21b618) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:92
#26 0x00002b1d8ed34241 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fff0944c8c0) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:234
#27 0x00002b1d8ed3382c in execute (op_array=0xc21b4c8) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:92
#28 0x00002b1d8ed34241 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fff0944d1a0) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:234
#29 0x00002b1d8ed3382c in execute (op_array=0xc21f0a8) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:92
#30 0x00002b1d8ed34241 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fff0944d7a0) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:234
#31 0x00002b1d8ed3382c in execute (op_array=0xc21eca0) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:92
#32 0x00002b1d8ed34241 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fff0944dc20) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:234
#33 0x00002b1d8ed3382c in execute (op_array=0xc21eb48) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:92
#34 0x00002b1d8ed34241 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fff0944dea0) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:234
#35 0x00002b1d8ed3382c in execute (op_array=0xc21dce0) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:92
#36 0x00002b1d8ed34241 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fff0944e260) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:234
#37 0x00002b1d8ed3382c in execute (op_array=0xc214248) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend_vm_execute.h:92
#38 0x00002b1d8ed139ad in zend_execute_scripts (type=8, retval=<value optimized out>, file_count=3) at /usr/src/redhat/BUILD/php-5.2.5/Zend/zend.c:1134
#39 0x00002b1d8ecd103b in php_execute_script (primary_file=0x7fff09450730) at /usr/src/redhat/BUILD/php-5.2.5/main/main.c:2005
#40 0x00002b1d8ed93ba5 in php_handler (r=0xc263ec8) at /usr/src/redhat/BUILD/php-5.2.5/sapi/apache2handler/sapi_apache2.c:631
#41 0x00000000004402ca in ap_run_handler ()
#42 0x0000000000443502 in ap_invoke_handler ()
#43 0x0000000000470bf8 in ap_process_request ()
#44 0x000000000046de8c in ap_process_http_connection ()
#45 0x00000000004471ba in ap_run_process_connection ()
#46 0x0000000000487270 in child_main ()
#47 0x0000000000487505 in make_child ()
#48 0x0000000000487d22 in ap_mpm_run ()
#49 0x000000000042d759 in main ()
apache日志中表现 已经进程退出
tailf error_log | grep 27191
[Thu Nov 30 19:30:24 2017] [notice] child pid 27191 exit signal Segmentation fault (11)
小结
综合报错信息来看，每次都是在openssl的某个方法导致进程终止，openssl版本较低，目前做打算升级处理，解决此问题。
