[SLL] limit on number of files in a directory and hashed dir vs. flat dir file access time

[Xeno Campanoli]

Adam Monsen wrote:
> Is there a limit on the number of files in a directory other than just the
> inode limit of a particular partition? I'm using the ext3 filesystem. Ubuntu
> 7.10.

Yes.  See:

something_fs.h where something is your filesystem time (like ext3, which
is most typical right now) in /usr/include/linux and there will be a
macro definition like:

#define EXT2_LINK_MAX       32000

implying that the maximum number of links to anything is 32000, which
limits a directory to 32000 files.  Now you can make that infinite by
having subdirectories, and you can also compile your own kernel to have
more links.  However typically production shops like standard kernels
(some in fact insist on "supported" kernels from chimera corporations
like RedHat) so you may find kernelmaking is not allowed for your work,
however, it probably wouldn't be hard to try.  Of course once you change
one limit, you don't always know, especially as a novice, what other
limits you'll run into next and when, so don't take the step lightly.

Anyhow, you'll probably find such a file on your system, and 32000 is
not an unlikely number you'll find.  It's what I see on both my Ubuntu
and my CentOS system I have here today.
> 
> Follow up question: is it more efficient (as far as reads are concerned) to
> "hash" files into subdirectories rather than just throw them all in a single
> directory? For instance, say I have 1 million 100 kilobyte JPEG images named
> as follows:
> 000000.jpg
> 000001.jpg
> 000002.jpg
> ...
> 999999.jpg
> 
> Would it speed up read time for a particular image if images were placed in
> directories like:
> 
> 0/0/0/0/0/1/000001.jpg and
> 0/2/3/6/1/2/023612.jpg
> 
> and so on?
> 
> 
> Here are some links I came across in my search, but if the answer I'm
> looking for is in there, I must've just skimmed right past it:
> http://en.wikipedia.org/wiki/Ext3
> http://local.google.com/answers/threadview?id=403516
> http://www.linuxquestions.org/questions/general-10/maximum-number-of-files-in-a-directory-183643/
> http://ubuntuforums.org/archive/index.php/t-487196.html
> http://www.nabble.com/Poor-Performance-WhenNumber-of-Files-%3E-1M-t4203660.html
> 


-- 
The only sustainable organizing methods focus not on scale,
but on good design of the functional unit,
not on winning battles, but on preservation.