Re: [Condor-users] rooster on linux, take 2

[Dan Bradley <dan@xxxxxxxxxxxx>]

On 11/21/11 6:05 PM, Dimitri Maziuk wrote:
> I guess my original question remains unanswered: how do you tell rooster
> to wake up a node?
>
> I am getting
>
> 11/21/11 16:49:59 WARNING: Someone at 144.92.167.254 is trying to modify
> "UNHIBERNAME"
> 11/21/11 16:49:59 WARNING: Potential security problem, request refused
>
> from condor_config_val despite the
>
> ALLOW_ADMINISTRATOR = $(ALLOW_WRITE)
> ALLOW_WRITE = *.bmrb.wisc.edu

Setting configuration settings remotely requires additional 
configuration settings to allow it.  Example:

ENABLE_RUNTIME_CONFIG = true
SETTABLE_ATTRS_ADMINISTRATOR = Unhibernate

The above example allows changing configuration settings in a running 
daemon's memory (i.e. not saved permanently to disk).  I believe you 
have to run condor_reconfig after making the change to make it take effect.

Presumably, you are doing this before hibernating the machine?  Because 
after the machine goes into hibernation, you obviously can't modify its 
configuration.

> Submitting 10K jobs is not a usable debug technique. Besides, it worked
> once, but now RoosterLog again claims
>
> "Got 0 startd ads matching ROOSTER_UNHIBERNATE=Offline&&  Unhibernate"
>
> in spite of the full queue
>
> Done     Pre   Queued    Post   Ready   Un-Ready   Failed
> ===     ===      ===     ===     ===        ===      ===
> 230       0       41       4    8187          0        0
>
> and
>
> 954838.000:  Run analysis summary.  Of 44 machines,
> ...
>        4 match but are currently offline

The default Unhibernate expression should be true if 
MachineLastMatchTime is set.  MachineLastMatchTime should get set when 
the negotiator matches a job to an offline machine.

To see if the negotiator is matching jobs to offline machines, add 
D_FULLDEBUG to NEGOTIATOR_DEBUG.  You should then see the following 
message in NegotiatorLog:

"Registering attempt to match offline machine <host.name> by <user.name>."

This should result in MachineLastMatchTime getting set in the offline 
machine ad.  You should be able to look at the offline machine ad with a 
command such as this:

condor_status -l <host.name>

All rooster does is periodically query the collector to find machines 
for which Unhibernate is true.  It then uses condor_power to wake them up.

I hope that helps.

--Dan

p.s. The problem with condor_power has been fixed for 7.6.5.  We can 
give you a pre-release if you want one.