[SLL] limit on number of files in a directory and hashed dir vs. flat dir file access time

[Ana]

On Sat, Nov 03, 2007 at 09:36:09PM -0700, Andrew Sweger wrote:
> On Sat, 3 Nov 2007, Ana wrote:
> 
> > On Sat, Nov 03, 2007 at 05:19:04PM -0700, Andrew Sweger wrote:
> > > 
> > > Any chance your program would block on disk-io from another process?
> > > Running in single-user mode?
> > 
> > Do you think updatedb could gum things up for that long?
> 
> Certainly. You don't even have to have any significant "servers" running
> to hit disk waits (the type of hardware can have a big impact on the
> chance of hitting them too). Running several test runs without any
> significant background tasks will be best to get useful numbers
> (given/assuming a valid test algorithm that tests the hypothesis).

I disabled some stuff that had the greatest chance of interfering and
and ran the million file test three times.  The first run took a long
time but the second two were more in line with the lower file count
tests.  I wonder if the first run of the test caused the file system to
be cached which allowed the second two to run quickly?

Maybe what I need to do is boot into single user mode in order to load
as little as possible.

I think the test algorithm is probably good, but I included the code so
that others could check my logic and method.  Here's a version that's
easy to quickly read and comprehend:

/* run times(); here */
/* a hundred million times, choose a random file to open and close */
for( i = 0; i < 100000000; i++ )
{
    /* random number between 1 and 1,000,000 */
    fnumber = 1 + (int) (1000000 * (rand() / (RAND_MAX + 1.0)));
    snprintf(fname, FNAMELENGH, "files/%06x.t", fnumber);

    fd = open(fname, O_RDONLY);
    close(fd);
}
/* run times(); here */

times() returns the amount of time spent in user-space separately from
the time spent in kernel-space, so can isolate time spent in the the
open() and close() system calls from the rest of the program.

- Ana



ana at betty:/mnt/var/tmp/test$ n=0; while [ $n -lt 3 ]; do date; \
  ./testjig randopenclose 100000000 1000000 ; date; sleep 1; \
  echo ----------- ; let n++; done

Sat Nov  3 21:54:52 PDT 2007
User Ticks spent: 8791
Syst Ticks spent: 684983
Sat Nov  3 23:56:08 PDT 2007
-----------
Sat Nov  3 23:56:09 PDT 2007
User Ticks spent: 9286
Syst Ticks spent: 51732
Sun Nov  4 00:06:19 PDT 2007
-----------
Sun Nov  4 00:06:20 PDT 2007
User Ticks spent: 8354
Syst Ticks spent: 51126
Sun Nov  4 00:16:15 PDT 2007
-----------