Re: [HTCondor-users] have people seen scalability issues with condor submission using the python bindings?

[Carl Edquist <edquist@xxxxxxxxxxx>]

Jose,

> In a sick way, it is a relief I am not the only one.

I think you mean, in a healthy way :)

Carl

On Fri, 24 Aug 2018, Jose Caballero wrote:

> Date: Fri, 24 Aug 2018 19:52:12
> From: Jose Caballero <jcaballero.hep@xxxxxxxxx>
> Reply-To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
> To: Brian Bockelman <bbockelm@xxxxxxxxxxx>
> Cc: Condor-Users Mail List <condor-users@xxxxxxxxxxx>,
>     HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
> Subject: Re: [HTCondor-users] have people seen scalability issues with condor
>     submission using the python bindings?
>
>
>
>
>
> On Aug 24, 2018, at 20:11, Brian Bockelman <bbockelm@xxxxxxxxxxx> wrote:
>
> > FWIW -
> >
> > This is the second time that I've seen someone report strange submission issues when calling schedd.submit from multiple python threads.  I think the prior reporter (maybe from a year or two ago?) also included a reproducer.
> >
> > It's not the most efficient way to roll, but I smell a bug here.  Sounds like something inside the submit implementation is releasing the GIL when it should be hanging on to it.
> >
> > Brian
>
> Hi Brian
>
> In a sick way, it is a relief I am not the only one.
> That means it is not my fault.
>
> As I mentioned, IIRC, I solved my issues with a double hack:
>
> -- my high level classes for collector and schedd are now Singletons, which reduces to 1 the number of objects trying to talk to the HTCondor resources, even for multi-threaded applications,
>
> -- I serialize the calls to condor_q / condor_history / condor_submit using Threading Lock objects.
>
> I wonder, at this point, if my little library would be helpful also for other people.
> Or, if someone wrote a better version of it, I would like to know and grab it :)
>
>    https://github.com/bnl-sdcc/libfactory/htcondorlib.py
>
> Cheers,
> Jose
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/