Mailing List Archives
Authenticated access
|
|
|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [HTCondor-users] Managing dynamic services
- Date: Mon, 07 Jan 2019 17:55:15 -0800
- From: Stuart Anderson <anderson@xxxxxxxxxxxxxxxx>
- Subject: Re: [HTCondor-users] Managing dynamic services
> On Nov 21, 2018, at 6:13 PM, Stuart Anderson <anderson@xxxxxxxxxxxxxxxx> wrote:
>
>> On Nov 21, 2018, at 12:52 PM, Greg Thain <gthain@xxxxxxxxxxx> wrote:
>>
>> On 11/16/18 6:57 PM, Stuart Anderson wrote:
>>> Is there a standard way for Condor to manage local privileged dynamic services on an execute machine depending on whether any currently matched jobs have requested it?
...
>>
>> This is an interesting question, and I think there's at least one way to do this with HTCondor, but I have a couple of questions to put a finer point on potential solutions
>
> The background/use case for this is Kafka. In particular, if there are Condor jobs running that want to access a popular set of messages from Kafka then a per-machine service should start to receive those data and perform some common processing for all jobs on the machine. Problems to avoid: 1) do not load down the network and Kafka brokers with thousands of extra copies of messages that will not be used; 2) Avoid the inefficiency of multiple jobs on the same execute machine requesting the same messages independent of each other.
Greg,
Any further thoughts on this?
Thanks.
--
Stuart Anderson anderson@xxxxxxxxxxxxxxxx
http://www.ligo.caltech.edu/~anderson