[SLL] limit on number of files in a directory and hashed dir vs. flat dir file access time

[Xeno Campanoli]

Xeno Campanoli wrote:
> Adam Monsen wrote:
>> Is there a limit on the number of files in a directory other than just the
>> inode limit of a particular partition? I'm using the ext3 filesystem. Ubuntu
>> 7.10.
> 
> Yes.  See:
> 
> something_fs.h where something is your filesystem time (like ext3, which
> is most typical right now) in /usr/include/linux and there will be a
> macro definition like:
> 
> #define EXT2_LINK_MAX       32000
> 
> implying that the maximum number of links to anything is 32000, which
> limits a directory to 32000 files.  Now you can make that infinite by
> having subdirectories, and you can also compile your own kernel to have
> more links.  However typically production shops like standard kernels
> (some in fact insist on "supported" kernels from chimera corporations
> like RedHat) so you may find kernelmaking is not allowed for your work,
> however, it probably wouldn't be hard to try.  Of course once you change
> one limit, you don't always know, especially as a novice, what other
> limits you'll run into next and when, so don't take the step lightly.
> 
> Anyhow, you'll probably find such a file on your system, and 32000 is
> not an unlikely number you'll find.  It's what I see on both my Ubuntu
> and my CentOS system I have here today.
>> Follow up question: is it more efficient (as far as reads are concerned) to
>> "hash" files into subdirectories rather than just throw them all in a single
>> directory? For instance, say I have 1 million 100 kilobyte JPEG images named
>> as follows:
>> 000000.jpg
>> 000001.jpg
>> 000002.jpg
>> ...
>> 999999.jpg
>>
>> Would it speed up read time for a particular image if images were placed in
>> directories like:
>>
>> 0/0/0/0/0/1/000001.jpg and
>> 0/2/3/6/1/2/023612.jpg
>>
>> and so on?
>>
>>
>> Here are some links I came across in my search, but if the answer I'm
>> looking for is in there, I must've just skimmed right past it:
>> http://en.wikipedia.org/wiki/Ext3
>> http://local.google.com/answers/threadview?id=403516
>> http://www.linuxquestions.org/questions/general-10/maximum-number-of-files-in-a-directory-183643/
>> http://ubuntuforums.org/archive/index.php/t-487196.html
>> http://www.nabble.com/Poor-Performance-WhenNumber-of-Files-%3E-1M-t4203660.html
>>
> 
> 


-- 
The only sustainable organizing methods focus not on scale,
but on good design of the functional unit,
not on winning battles, but on preservation.