[SLL] limit on number of files in a directory and hashed dir vs. flat dir file access time

[Chuck Wolber]

On Wed, 31 Oct 2007, Adam Monsen wrote:

> Is there a limit on the number of files in a directory other than just 
> the inode limit of a particular partition? I'm using the ext3 
> filesystem. Ubuntu 7.10.

In old 2.4 kernels ext3 performanced inversely porportional to the number 
of files in a given directory. So in that sense there was a limit. They 
fixed that in 2.6 with a tree like data structure. "ls" will still perform 
dog slow because it tries to grok the entire dir before outputting, but 
file accesses should be fine.

As for a physical limit on the number of files in a directory, if there 
is, it would probably have something to do with how many bits used to 
represent the tree structure. That number is probably so astronomically 
large that you'll probably exceed some other filesystem limit before you 
hit it, but that's just a guess. 

Have you tried asking that question on the ext3 developers list? I don't 
know if this is the *OFFICIAL* list, but Theodore Tso posts there all the 
time:

https://www.redhat.com/mailman/listinfo/ext3-users

Also, have you considered other types of filesystems for what you're 
trying to do? IIRC ReiserFS actually has some advantages over ext3 for 
lots of small files, or maybe that was xfs...


> Follow up question: is it more efficient (as far as reads are concerned) to
> "hash" files into subdirectories rather than just throw them all in a single
> directory? For instance, say I have 1 million 100 kilobyte JPEG images named
> as follows:
> 000000.jpg
> 000001.jpg
> 000002.jpg
> ...
> 999999.jpg
> 
> Would it speed up read time for a particular image if images were placed in
> directories like:
> 
> 0/0/0/0/0/1/000001.jpg and
> 0/2/3/6/1/2/023612.jpg
> 
> and so on?

Not according to anything I've ever seen, but it's probably worth a test. 
Perhaps traversing the logical tree in a directory might add *some* 
overhead as the tree gets larger, but the 2.6 kernel really did fix a lot 
of the annoying performance problems in ext. Chopping the images into a 
directory hiearachy may make sense for other reasons though.


..Chuck..

-- 
http://www.quantumlinux.com
 Quantum Linux Laboratories, LLC.
 ACCELERATING Business with Open Technology

"Stay Hungry. Stay Foolish."
	-The Whole Earth Catalog