HTCondor Project List Archives


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-devel] Fwd: [condor-fw] new statistics effects on various ClassAds

You should check with Dan. He had an approach that involved memory profiling the Collector empty and then after populating it with a large number of different types of ads, probably using condor_advertise. 

Best,


matt

On 09/16/2011 06:25 PM, John (TJ) Knoeller wrote:

I don't know an easy way to calculate the in-memory or wire sizes for ads,
but I do know that the cost will be mostly the attribute labels. the stats
are nearly all int or double values.

When printed out via condor_status -long the sizes are

    Ad Type v770 v772 Level 1 v772 Level 2
    ----------- -------- ------------ -------------
    collector 4067 5042 8028
    master 1212 1753 5370
    negotiator 3625 4218 7449
    scheduler 1601 3360 9942
    slot(machine) 3691 4347 9232
    submitter* 1468 1555 1555


* there should be no change in submitter ads as a result of the new
statistics,
that bug that affected these ads has been fixed, but the fix did not
make it into 772 (yet?)

So by default slot ads grow by a bit less than 20%

On 9/16/2011 1:52 PM, Matthew Farrellee wrote:

On 09/16/2011 02:40 PM, John (TJ) Knoeller wrote:

FYI

-------- Original Message --------
Subject: [condor-fw] new statistics effects on various ClassAds
Date: Fri, 16 Sep 2011 13:20:29 -0500
From: John (TJ) Knoeller <johnkn@xxxxxxxxxxx>
To: Todd Tannenbaum <tannenba@xxxxxxxxxxx>
CC: Condor Flightworthy Team <condor-fw@xxxxxxxxxxx>



I did some measurements of how much various Ads increased as a result of
the new statistics

This table shows how many attributes were contained in each type of Ad
for v770
as well as how many more attributes are in v772 with statistics at level
1 (basic) and
at level 2 (verbose). the counts for verbose statistics include the
counts for basic.

Statistics are enabled at level 1 by default for all daemons Condor
v772.

Ad Type v770 v772 Level 1 v772 Level 2
----------- -------- ------------ -------------
collector 88 114 (+26) 187 (+ 99)
master 38 58 (+22) 139 (+103)
negotiator 95 117 (+22) 190 (+ 95)
scheduler 53 121 (+68) 265 (+212)
slot(machine) 103 133 (+30) 259 (+143)
submitter 48 67 (+22-3*) 207 (+162-3*)

Most of the increase at level 2 is from per-handler runtime statistics
in daemon core.

* the v770 submitter ad had HeldJobs, FlockedJobs and SubmitterTag these
attribs were not in 772..

-tj


The increase for slot is most concerning, then for master, maybe for
submitter.

Will you provide a table with sizes (bytes)? Dan's work in the past
suggested 2KB for slot ads, helpful in sizing CM nodes (how much
memory do you buy?).

Best,


matt



_______________________________________________
Condor-devel mailing list
Condor-devel@xxxxxxxxxxx
https://lists.cs.wisc.edu/mailman/listinfo/condor-devel