Re: [Condor-users] tuning a file server

[Alain EMPAIN <alain.empain@xxxxxxxxx>]

Hello David & Steffen,

indeed, 3ware is a very stable solution for RAID server; it runs smootly 
 for 3 years in my lab, but a few weeks ago I met problems with condor 
/ NFS on 3ware RAID after tripling my nodes.

So I suspected not the RAID but the NFS configuration : the solution was 
simply to launch more nfs server daemons (default=4 on SuSE) and tuning 
the client side.

-------------------------
SERVER: /etc/sysconfig/nfs (SuSE)

# the kernel nfs-server supports multiple server threads
#
#AE:050727:USE_KERNEL_NFSD_NUMBER="4"
USE_KERNEL_NFSD_NUMBER="12"

--------------------------
CLIENT: /etc/fstab

10.185.2.25:/home/grid  /home/grid      nfs \
        rw,hard,nointr,tcp,vers=3,rsize=32k,wsize=32k,bg \
        0 0
----------------------------------

REM: I preferred bg to fg because I often work remotely and I do not 
want to lock the booting process if the NFS server is not responding.

Now all my nodes are 'blasting' steadily.

  Have a good day,

	Alain


David McBride wrote:
> Steffen Grunewald wrote:
>
> Hi,
>
> > some weeks ago, I had to move our file systems (/home and some data) 
> > to another server, equipped with 3ware 9000 cards, and using 8 400GB 
> > disks
> > in RAID5 with hot spare (2.3TB net capacity). To manage the disk space,
> > I had to switch to Linux kernel 2.6 - and there are indications that this
> > move caused some problems.
>
>
> 3ware tend to make excellent Linux-compatible RAID solutions, so it is 
> unlikely to be a RAID-level problem, as your basic read benchmark 
> suggested.
>
> I'd suggest also running the 'bonnie++' benchmark on your filesystem to 
> get a better idea of real-world performance (file creates/erasures, 
> small loads, etc.)
>
> > The file server has to serve 360 VMs in 180 dual-CPU machines, and if 
> > there are lots of small I/O operations, the iowait percentage goes up to
> > more than 80%! I have never seen that before (the previous server was
> > almost identical, kernel 2.4, 1.4TB each).
>
>
> Some important missing information:
>
> * How (if at all) are the fileserver contents exported? NFS?
> * What filesystem(s) are you using?
>
>  > Now I suspect that the i/o strategies introduced with 2.6 kernels are
>  > badly configured... are there any suggestions?
>
> The 2.6 IO schedulers tend to _better_ than the 2.4 one.  The default is 
> to use the 'anticipatory' scheduler which tends to be excellent for most 
> needs.
>
> Cheers,
> David

--
------------------------------------------------------------
Dr Alain EMPAIN  <alain.empain@xxxxxxxxx> <alain@xxxxxxxxxx>
      Bioinformatics, Molecular Genetics,
      Fac. Med. Vet., University of LIEGEe, Belgium
      Bd de Colonster, B43   B-4000 LIEGEe (Sart-Tilman)
WORK: +32 4 366 4159         FAX: +32 4 366 4122
HOME: rue des Martyrs,7      B- 4550 Nandrin
      +32 85 51 2341         GSM: +32 497 70 1764
-------------------------------------------------------------------------------
"I worry about my child and the Internet all the time, even though she's
too young to have logged on yet. Here's what I worry about. I worry that
10 or 15 years from now, she will come to me and say 'Daddy, where were
you when they took freedom of the press away from the Internet?'"
--Mike Godwin, Electronic Frontier Foundation
-------------------------------------------------------------------------------