近期频繁宕掉的小鸡排查发现是磁盘满了原来问题来自V2Ray日志文件

最近手上一台小鸡经常离线，但是过一段时间就自己恢复了，以为是主机商抽风加上工作繁忙（懒）就拖着没管。

拖到今天Jetpack Monitor提示持续宕机，访问后台提示数据库无法连接，推测是数据库挂掉了，于是SSH后台试图通过命令“service mysql restart”命令重启数据库进程。

root@AzureTokyo [11:53:32 PM] [/usr/local/mysql]
-> # sudo service mysql restart
Job for mysqld.service failed because the control process exited with error code.
See "systemctl status mysqld.service" and "journalctl -xe" for details.

结果重启报错，于是通过命令“systemctl status mysqld.service”进一步查看原因。

root@AzureTokyo [11:54:35 PM] [/usr/local]
-> # systemctl status mysqld.service
● mysqld.service - LSB: start and stop MySQL
   Loaded: loaded (/etc/init.d/mysqld; generated)
   Active: failed (Result: exit-code) since Sun 2021-04-04 23:53:47 CST; 52s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 16886 ExecStart=/etc/init.d/mysqld start (code=exited, status=1/FAILURE)

Apr 04 23:53:45 AzureTokyo systemd[1]: mysqld.service: Succeeded.
Apr 04 23:53:45 AzureTokyo systemd[1]: Stopped LSB: start and stop MySQL.
Apr 04 23:53:45 AzureTokyo systemd[1]: Starting LSB: start and stop MySQL...
Apr 04 23:53:45 AzureTokyo mysqld[16886]: Starting MySQL
Apr 04 23:53:47 AzureTokyo mysqld[16886]: ..The server quit without updating PID file (/data/mysql/mysql.pid). ... failed!
Apr 04 23:53:47 AzureTokyo systemd[1]: mysqld.service: Control process exited, code=exited, status=1/FAILURE
Apr 04 23:53:47 AzureTokyo systemd[1]: mysqld.service: Failed with result 'exit-code'.
Apr 04 23:53:47 AzureTokyo systemd[1]: Failed to start LSB: start and stop MySQL.g

根据报错内容，先用命令“ps -ef|grep mysqld”查看MySQL数据库进程PID，然后用“kill -9 PID”杀掉相关进程。

root@AzureTokyo [11:55:07 PM] [/usr/local]
-> # ps -ef|grep mysqld
root      19586  11682  0 23:55 pts/0    00:00:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn mysqld
root@AzureTokyo [11:56:28 PM] [/usr/local]
-> # kill -9 11682
Killed

完成后依然无法启动MySQL数据库进程，推测是文件满了。

使用命令“df -h”查看磁盘后，发现确实是文件满了，导致MySQL出错。

root@AzureTokyo [11:57:13 PM] [/home/q1ngyang]
-> # df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            948M     0  948M   0% /dev
tmpfs           192M   20M  172M  11% /run
/dev/sda1        30G   28G     0 100% /
tmpfs           956M     0  956M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           956M     0  956M   0% /sys/fs/cgroup
/dev/sda15      124M  278K  124M   1% /boot/efi
/dev/sdb1       3.9G   16M  3.7G   1% /mnt/resource
tmpfs           192M     0  192M   0% /run/user/1000

cd到根目录，使用命令“du -h --max-depth=1 ./”，查看各文件夹大小，最终发现是“/var/log/v2ray”目录下V2Ray日志文件导致磁盘被占满。

root@AzureTokyo [11:58:23 PM] [/data/wwwlogs]
-> # cd /
root@AzureTokyo [11:58:25 PM] [/]
-> # du -h --max-depth=1 ./
3.9M    ./tmp
1.3G    ./home
4.1G    ./usr
0       ./sys
3.7M    ./etc
4.0K    ./srv
0       ./dev
4.0K    ./media
18G     ./var
36K     ./mnt
18M     ./root
du: cannot access './proc/15462/task/15462/fd/4': No such file or directory
du: cannot access './proc/15462/task/15462/fdinfo/4': No such file or directory
du: cannot access './proc/15462/fd/3': No such file or directory
du: cannot access './proc/15462/fdinfo/3': No such file or directory
0       ./proc
4.0K    ./opt
16K     ./lost+found
72M     ./boot
20M     ./run
2.6G    ./data
28G     ./
root@AzureTokyo [11:59:27 PM] [/var/log/v2ray]
-> # ls -l
total 18053128
-rw------- 1 root root  1880371930 Apr  4 23:59 access.log
-rw------- 1 root root 16605998265 Apr  4 23:59 error.logs

于是删除相关日志文件后，使用命令“df -h”查看磁盘容量，发现空间没有被释放。

于是使用命令“systemctl restart v2ray”重启V2Ray进程，释放磁盘空间，重启后被删除文件占用磁盘空间被释放。

root@AzureTokyo [12:01:05 AM] [/home/q1ngyang]
-> # df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            948M     0  948M   0% /dev
tmpfs           192M   20M  172M  11% /run
/dev/sda1        30G   28G     0 100% /
tmpfs           956M     0  956M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           956M     0  956M   0% /sys/fs/cgroup
/dev/sda15      124M  278K  124M   1% /boot/efi
/dev/sdb1       3.9G   16M  3.7G   1% /mnt/resource
tmpfs           192M     0  192M   0% /run/user/1000
root@AzureTokyo [12:01:09 AM] [/home/q1ngyang]
-> # systemctl restart v2ray
root@AzureTokyo [12:02:16 AM] [/home/q1ngyang]
-> # df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            948M     0  948M   0% /dev
tmpfs           192M   20M  172M  11% /run
/dev/sda1        30G   11G   18G  39% /
tmpfs           956M     0  956M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           956M     0  956M   0% /sys/fs/cgroup
/dev/sda15      124M  278K  124M   1% /boot/efi
/dev/sdb1       3.9G   16M  3.7G   1% /mnt/resource
tmpfs           192M     0  192M   0% /run/user/1000

然后重启MySQL服务，一切恢复正常。

root@AzureTokyo [12:02:18 AM] [/home/q1ngyang]
-> # systemctl start mysql