Solved: HELP System Down

Added by Brian Kelly 5 months ago

Asterisk won't come up. Other components are up like the web interface and other services.... Occurred shortly after a shutdown and restart. Perhaps since I was using the web interface and that worked I didn't notice asterisk was down. Added a conference room went to test it and then noticed it wasn't working

Jun  5 22:05:00 wazo kernel: [  149.488351] netlink: 4 bytes leftover after parsing attributes in process `xivo-sysconfd'.
Jun  5 22:05:00 wazo kernel: [  149.488523] netlink: 4 bytes leftover after parsing attributes in process `xivo-sysconfd'.
Jun  5 22:05:00 wazo kernel: [  149.490360] netlink: 4 bytes leftover after parsing attributes in process `xivo-sysconfd'.
Jun  5 22:05:00 wazo kernel: [  149.490561] netlink: 4 bytes leftover after parsing attributes in process `xivo-sysconfd'.
Jun  5 22:05:01 wazo CRON[2356]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi)
Jun  5 22:05:01 wazo CRON[2357]: (root) CMD (      xivo-check-long-calls)
Jun  5 22:05:01 wazo CRON[2355]: (munin) CMD (if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi)
Jun  5 22:05:01 wazo CRON[2358]: (root) CMD (      xivo-check-dahdi-ports)
Jun  5 22:05:01 wazo kernel: [  149.814817] traps: asterisk[2268] trap divide error ip:7f3b422befb8 sp:7ffda6236380 error:0 in chan_sip.so[7f3b42295000+d2000]
Jun  5 22:05:01 wazo systemd[1]: asterisk.service: main process exited, code=dumped, status=8/FPE
Jun  5 22:05:01 wazo systemd[1]: PID 2268 read from file /var/run/asterisk/asterisk.pid does not exist or is a zombie.
Jun  5 22:05:02 wazo systemd[1]: asterisk.service never wrote its PID file. Failing.
Jun  5 22:05:02 wazo systemd[1]: Failed to start Asterisk PBX.
Jun  5 22:05:02 wazo systemd[1]: Unit asterisk.service entered failed state.

wazo1.jpg - machine status (118 KB)


Replies (9)

RE: HELP System Down - Added by Pascal Cadotte-Michaud 5 months ago

Hi Brian,

according to you log there was a division by zero that happened in the asterisk process. You probably have a core dump file in "/var/cache/asterisk" that you can use to get more information.

sp:7ffda6236380 is the address in the stack pointer at the moment of the crash which could be useful to inspect the core dump file with gdb.

You can find basic information to extract a back traces from a core dump here.

http://documentation.wazo.community/en/stable/contributors/debug_asterisk.html#debugging-asterisk-crash

If you try a "systemctl restart asterisk" does asterisk start?
What the content of /var/log/asterisk/full.log when the crash happens?

RE: HELP System Down - Added by Brian Kelly 5 months ago

Hi Pascal..... I don't see the cache file but here some other info. I made a Wazo manual backup last night but there is a lot of setup work that has been done that I would hate to have to do over.... I'd be happy to pay someone to look deeper if none of the info below is helpful. We could do SSH but I'd need info on how to add port 22 access to Iptables. SSH works from the lan but despite my router and firewall being configured properly the SSH from outside the lan doesn't work on this particular box. I've tested that it can work on others.

@

root@wazo:/# systemctl status asterisk.service
â asterisk.service - Asterisk PBX
Loaded: loaded (/lib/systemd/system/asterisk.service; enabled)
Active: failed (Result: resources) since Tue 2017-06-06 07:24:41 EDT; 39s ago
Process: 2627 ExecStartPost=/bin/bash -c for i in {1..10}; do /usr/sbin/asterisk -rx "core waitfullybooted" &>/dev/null && exit 0; sleep 1; done; exit 1 (code=exited, status=0/SUCCESS)
Process: 2623 ExecStart=/usr/sbin/asterisk -g -U asterisk (code=exited, status=0/SUCCESS)
Main PID: 2624 (code=dumped, signal=FPE)

Jun 06 07:24:39 wazo systemd1: asterisk.service: main process exited, code=dumped, status=8/FPE
Jun 06 07:24:39 wazo systemd1: PID 2624 read from file /var/run/asterisk/asterisk.pid does not exist or is a zombie.
Jun 06 07:24:41 wazo systemd1: asterisk.service never wrote its PID file. Failing.
Jun 06 07:24:41 wazo systemd1: Failed to start Asterisk PBX.
Jun 06 07:24:41 wazo systemd1: Unit asterisk.service entered failed state.
root@wazo:/# journalctl xn
-
Logs begin at Tue 2017-06-06 07:19:08 EDT, end at Tue 2017-06-06 07:27:30 EDT. --
Jun 06 07:27:15 wazo kernel: usb 2-6: USB disconnect, device number 9
Jun 06 07:27:17 wazo kernel: usb 2-6: new low-speed USB device number 10 using ohci-pci
Jun 06 07:27:17 wazo kernel: usb 2-6: New USB device found, idVendor=046d, idProduct=c05a
Jun 06 07:27:17 wazo kernel: usb 2-6: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Jun 06 07:27:17 wazo kernel: usb 2-6: Product: USB Optical Mouse
Jun 06 07:27:17 wazo kernel: usb 2-6: Manufacturer: Logitech
Jun 06 07:27:17 wazo kernel: input: Logitech USB Optical Mouse as /devices/pci0000:00/0000:00:04.0/usb2/2-6/2-6:1.0/0003:046D:C05A.0009/input/
Jun 06 07:27:17 wazo kernel: hid-generic 0003:046D:C05A.0009: input,hidraw0: USB HID v1.11 Mouse [Logitech USB Optical Mouse] on usb-0000:00:0
Jun 06 07:27:30 wazo consul612: > Caught signal: broken pipe
Jun 06 07:27:30 wazo consul[612]: > Caught signal: broken pipe

SEE ATTACHED LINK FURTHER DOWN IN THE MESSAGE FOR "full" LOG

RE: HELP System Down - Added by Brian Kelly 5 months ago

there are core files in /var/spool/asterisk but they aren't plain text files

RE: HELP System Down - Added by Pascal Cadotte-Michaud 5 months ago

Hi Brian,

sorry I gave you the wrong directory.

you can use the following line to extract a txt file from the core dump

gdb -batch -ex "bt full" -ex "thread apply all bt" asterisk core-file > bt-threads.txt

replace core-file with the filename of the core dump.

you will probably need to install some packages to be able to execute this command and get relevant information.

apt-get update
apt-get install gdb asterisk-dbg

RE: HELP System Down - Added by Sylvain Boily 5 months ago

Hello Brian,

If you need professional level 3 support from Wazo team you only need to subscribe a contract with us. It's 100 USD per hour.

Sylvain

RE: HELP System Down - Added by Brian Kelly 5 months ago

that didn't capture all the text.. I need to upload the full file... give me a minute and I'll set up a link to it

RE: HELP System Down - Added by Brian Kelly 5 months ago

Ok... working on getting that core file created

RE: HELP System Down - Added by Brian Kelly 5 months ago

File is attached...

some errors

dpkg: dependency problems prevent configuration of asterisk-dbg:
asterisk-dbg depends on asterisk (= 8:14.4.0-1~wazo9); however:
Package asterisk is not configured yet.

dpkg: error processing package asterisk-dbg (--configure):
dependency problems - leaving unconfigured
Setting up libc6-dbg:amd64 (2.19-18+deb8u9) ...
Errors were encountered while processing:
asterisk
asterisk-dbg
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@wazo:/var/spool/asterisk# gdb -batch -ex "bt full" -ex "thread apply all bt" asterisk core.2624.1496748279 > bt-threads.txt
16665 chan_sip.c: No such file or directory.

bt-threads.txt Magnifier (38.7 KB)

(1-9/9)