最近手上一台小鸡经常离线，但是过一段时间就自己恢复了，以为是主机商抽风加上工作繁忙（懒）就拖着没管。
拖到今天Jetpack Monitor提示持续宕机，访问后台提示数据库无法连接，推测是数据库挂掉了，于是SSH后台试图通过命令“service mysql restart”命令重启数据库进程。
|
1 2 3 4 |
root@AzureTokyo [11:53:32 PM] [/usr/local/mysql] -> # sudo service mysql restart Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details. |
结果重启报错，于是通过命令“systemctl status mysqld.service”进一步查看原因。
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
root@AzureTokyo [11:54:35 PM] [/usr/local] -> # systemctl status mysqld.service ● mysqld.service - LSB: start and stop MySQL Loaded: loaded (/etc/init.d/mysqld; generated) Active: failed (Result: exit-code) since Sun 2021-04-04 23:53:47 CST; 52s ago Docs: man:systemd-sysv-generator(8) Process: 16886 ExecStart=/etc/init.d/mysqld start (code=exited, status=1/FAILURE) Apr 04 23:53:45 AzureTokyo systemd[1]: mysqld.service: Succeeded. Apr 04 23:53:45 AzureTokyo systemd[1]: Stopped LSB: start and stop MySQL. Apr 04 23:53:45 AzureTokyo systemd[1]: Starting LSB: start and stop MySQL... Apr 04 23:53:45 AzureTokyo mysqld[16886]: Starting MySQL Apr 04 23:53:47 AzureTokyo mysqld[16886]: ..The server quit without updating PID file (/data/mysql/mysql.pid). ... failed! Apr 04 23:53:47 AzureTokyo systemd[1]: mysqld.service: Control process exited, code=exited, status=1/FAILURE Apr 04 23:53:47 AzureTokyo systemd[1]: mysqld.service: Failed with result 'exit-code'. Apr 04 23:53:47 AzureTokyo systemd[1]: Failed to start LSB: start and stop MySQL.g |
根据报错内容，先用命令“ps -ef|grep mysqld”查看MySQL数据库进程PID，然后用“kill -9 PID”杀掉相关进程。
|
1 2 3 4 5 6 |
root@AzureTokyo [11:55:07 PM] [/usr/local] -> # ps -ef|grep mysqld root 19586 11682 0 23:55 pts/0 00:00:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn mysqld root@AzureTokyo [11:56:28 PM] [/usr/local] -> # kill -9 11682 Killed |
完成后依然无法启动MySQL数据库进程，推测是文件满了。
使用命令“df -h”查看磁盘后，发现确实是文件满了，导致MySQL出错。
|
1 2 3 4 5 6 7 8 9 10 11 12 |
root@AzureTokyo [11:57:13 PM] [/home/q1ngyang] -> # df -h Filesystem Size Used Avail Use% Mounted on udev 948M 0 948M 0% /dev tmpfs 192M 20M 172M 11% /run /dev/sda1 30G 28G 0 100% / tmpfs 956M 0 956M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 956M 0 956M 0% /sys/fs/cgroup /dev/sda15 124M 278K 124M 1% /boot/efi /dev/sdb1 3.9G 16M 3.7G 1% /mnt/resource tmpfs 192M 0 192M 0% /run/user/1000 |
cd到根目录，使用命令“du -h –max-depth=1 ./”，查看各文件夹大小，最终发现是“/var/log/v2ray”目录下V2Ray日志文件导致磁盘被占满。
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
root@AzureTokyo [11:58:23 PM] [/data/wwwlogs] -> # cd / root@AzureTokyo [11:58:25 PM] [/] -> # du -h --max-depth=1 ./ 3.9M ./tmp 1.3G ./home 4.1G ./usr 0 ./sys 3.7M ./etc 4.0K ./srv 0 ./dev 4.0K ./media 18G ./var 36K ./mnt 18M ./root du: cannot access './proc/15462/task/15462/fd/4': No such file or directory du: cannot access './proc/15462/task/15462/fdinfo/4': No such file or directory du: cannot access './proc/15462/fd/3': No such file or directory du: cannot access './proc/15462/fdinfo/3': No such file or directory 0 ./proc 4.0K ./opt 16K ./lost+found 72M ./boot 20M ./run 2.6G ./data 28G ./ root@AzureTokyo [11:59:27 PM] [/var/log/v2ray] -> # ls -l total 18053128 -rw------- 1 root root 1880371930 Apr 4 23:59 access.log -rw------- 1 root root 16605998265 Apr 4 23:59 error.logs |
于是删除相关日志文件后，使用命令“df -h”查看磁盘容量，发现空间没有被释放。
于是使用命令“systemctl restart v2ray”重启V2Ray进程，释放磁盘空间，重启后被删除文件占用磁盘空间被释放。
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
root@AzureTokyo [12:01:05 AM] [/home/q1ngyang] -> # df -h Filesystem Size Used Avail Use% Mounted on udev 948M 0 948M 0% /dev tmpfs 192M 20M 172M 11% /run /dev/sda1 30G 28G 0 100% / tmpfs 956M 0 956M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 956M 0 956M 0% /sys/fs/cgroup /dev/sda15 124M 278K 124M 1% /boot/efi /dev/sdb1 3.9G 16M 3.7G 1% /mnt/resource tmpfs 192M 0 192M 0% /run/user/1000 root@AzureTokyo [12:01:09 AM] [/home/q1ngyang] -> # systemctl restart v2ray root@AzureTokyo [12:02:16 AM] [/home/q1ngyang] -> # df -h Filesystem Size Used Avail Use% Mounted on udev 948M 0 948M 0% /dev tmpfs 192M 20M 172M 11% /run /dev/sda1 30G 11G 18G 39% / tmpfs 956M 0 956M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 956M 0 956M 0% /sys/fs/cgroup /dev/sda15 124M 278K 124M 1% /boot/efi /dev/sdb1 3.9G 16M 3.7G 1% /mnt/resource tmpfs 192M 0 192M 0% /run/user/1000 |
然后重启MySQL服务，一切恢复正常。
|
1 2 |
root@AzureTokyo [12:02:18 AM] [/home/q1ngyang] -> # systemctl start mysql |
