[SLL] [resolved] fs still full after rm'ing files, killing ps that held them open

[Phil Mocek]

On Tue, Feb 19, 2008 at 02:24:50PM -0600, Jeremy C. Reed wrote:
> Maybe use lsof to see if anything else has the file still open?

Since I don't have the filename handy, I called lsof, as
superuser, for the top-level directory in the affected file
system:

phil at hostname:/var/log> sudo lsof .       
COMMAND   PID USER   FD   TYPE DEVICE SIZE NODE NAME
bash     7775 phil  cwd    DIR   58,5 8768    2 .
lsof    25700 root  cwd    DIR   58,5 8768    2 .
lsof    25701 root  cwd    DIR   58,5 8768    2 .

and then I tried:

phil at hostname:/var/log> sudo lsof /var/log

and got a long list.  Some entries included "(deleted)":

phil at hostname:/var/log> sudo lsof /var/log | grep deleted | head
httpd2-pr   619 wwwrun    2w   REG   58,5   1066897 2711 /var/log/apache2/error_log-20071107 (deleted)
httpd2-pr   619 wwwrun    6w   REG   58,5   1066897 2711 /var/log/apache2/error_log-20071107 (deleted)
httpd2-pr   619 wwwrun    8w   REG   58,5 551698432 3575 /var/log/apache2/error_log-20080218 (deleted)
httpd2-pr   619 wwwrun   12w   REG   58,5 239460352 3892 /var/log/apache2/access_log-20080218 (deleted)
httpd2-pr   619 wwwrun   13w   REG   58,5 239460352 3892 /var/log/apache2/access_log-20080218 (deleted)
httpd2-pr   623 wwwrun    2w   REG   58,5   1066897 2711 /var/log/apache2/error_log-20071107 (deleted)
httpd2-pr   623 wwwrun    6w   REG   58,5   1066897 2711 /var/log/apache2/error_log-20071107 (deleted)
httpd2-pr   623 wwwrun    8w   REG   58,5 551698432 3575 /var/log/apache2/error_log-20080218 (deleted)
httpd2-pr   623 wwwrun   12w   REG   58,5 239460352 3892 /var/log/apache2/access_log-20080218 (deleted)
httpd2-pr   623 wwwrun   13w   REG   58,5 239460352 3892 /var/log/apache2/access_log-20080218 (deleted)

There were 30 of these hanging around from various processes.  I
killed them, and things are back to normal.

Thanks for the assistance.

-- 
Phil Mocek