Re: [HTCondor-users] Filling pool breadth-first with partitionable slots (Beyer, Christoph)

[Michael Pelletier <Michael.V.Pelletier@xxxxxxxxxxxx>]

Since youâre dealing with the entire universe at the CGSA, I suspect your data sets are much larger than mine. *chuckle* Most of our inputs have no trouble fitting in the systemsâ buffer cache and so N-1 jobs on N cores read them from there instead of the network (also I have âatimeâ turned off on the NFS mountpoints), and a lot of our systems have 10Gb since the Brocade switch per-port cost and the incremental cost of an 10Gb HP FlexLOM hasnât been prohibitively high.

 

I gather in 8.6 network bandwidth will be a manageable resource â thatâll be nice.

 

                -Michael Pelletier.

 

From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Tom Downes
Sent: Monday, October 31, 2016 12:16 PM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Filling pool breadth-first with partitionable slots (Beyer, Christoph)

 

I like depth-first owing as it is an anti-fragmentation tool, but breadth-first is, in my experience, better for input file transfers in the real-world because my storage can easily swamps the 1GbE network interface on an N-core machine run N jobs.

--

Tom Downes
Senior Scientist and Data Center Manager
Center for Gravitation, Cosmology and Astrophysics
University of Wisconsin-Milwaukee
414.229.2678