Mailing List ArchivesAuthenticated access |
![[Computer Systems Lab]](http://www.cs.wisc.edu/pics/csl_logo.gif){kind=logo}
|
Mailing List ArchivesAuthenticated access |
|
|
Hi Todd, This redesign look both practical and useful. (1) A small point: In the 2nd paragraph I think you mean: ... cases, being especially mindful rather than ... cases, begin especially mindful ... (2) The more scripting capabilities are placed into the language, the more often we will make mistakes. There is already a switch to get condor_submit to kick out the ClassAd(s), -verbose. It would be nice to have another one which prevents condor_submit from sending the ClassAd(s) on to SCHEDD to enable a quick check that everything
is right before pulling the “submit” trigger. Perhaps there’s already a way with –debug and the proper settings for TOOL_DEBUG . (3) If I understand correctly, your planned “…a future version of HTCondor in which the SCHEDD will have a job cluster factory that instantiates jobs on the
fly from a pattern defined in the submit file”
is a step in the direction I had tried to express of creating a “pre”condor_submit which aggregates a sequence of individual condor_submit’s to create a single group. I had been thinking of this as an intermediary process into
which you could progressively add algorithms not only for the purpose of generating grouped jobs, but also for serializing operations which otherwise have, in my experience, caused significant problems, e.g. one user running 100+ jobs all of which are issuing
condor_submit’s as fast as they can. My idea is that the majority of us are one-time or a-few-time users who are most likely going to do the one job/one condor_submit thing. And then there are people like me who are reluctant to spend the time and effort
to fix something that already works to obtain an efficiency that is certainly real but is essentially invisible to me. I will end up doing it because I see it’s important for scaling. But in my experience, serializing condor_submit’s is a significant need
which, for now, only happens when the user recognizes the need and puts in the effort to implement it. Anyway, I wanted to put in this word for something in the direction of fault tolerance in addition to the excellent ongoing work in adding functionality. Thanks and best regards, Don Don Krieger, Ph.D. Department of Neurological Surgery University of Pittsburgh > -----Original Message----- > From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf > Of John (TJ) Knoeller > Sent: Monday, March 02, 2015 3:31 PM > To: htcondor-users@xxxxxxxxxxx > Subject: Re: [HTCondor-users] change to condor_submit - user feedback > desired! (was Re: multiple condor_submit's - one cluster) > > There is now a design document here > https://docs.google.com/document/d/10UKaiu1BkirXP6qUmx9DyEOPPSNqKqmS > That describes our proposed improvements to the Queue statement for > condor_submit. > > We plan to have (most?) of this working for the 8.3.5 release. > > -tj > > On 2/6/2015 10:00 AM, Todd Tannenbaum wrote: > > On 2/6/2015 9:04 AM, Krieger, Donald N. wrote: > >> Hi Todd, > >> > >> Thanks for posting back with the answers. It's very helpful. > >> > >> My problem is that the names I'm using for the log files step through > >> a sequence of strings rather than a sequence of numbers. And I have a > >> downstream management routine that uses those names which I would > >> have to alter if I changed the naming sequence. > > > > Hi Don, > > > > Thanks for the above explanation. Indeed, $(Process) is great in your > > submit files, but only if your data files are numerically sequenced! > > > > Perhaps we could create a general solution in condor_submit that > > addresses the above and yet would still a) allow all the submits to > > happen at once into one cluster and b) not require the user to know > > how to write scripts. > > > > I listed a couple brainstorm ideas below that would be relatively easy > > to implement. Do folks think that either of the below be helpful? > > Would love to hear any feedback, alternative ideas. > > > > regards, > > Todd > > > > Some brainstorm ideas: > > > > 1. A "queue foreach <filepattern>" command. Folks could then have > > submit files that look like this: > > > > input = $(file) > > output = $(file).output > > queue foreach data/*.csv > > > > So for each file in subdir data that ends in .csv, a job would be > > submitted and $(file) would expand to the path to the file. > > > > and/or > > > > 2. A command line option to condor_submit that tells it to read stdin, > > and to do a submit for each stdin line, substituting each line from > > stdin with $(input_line). Folks could then have submit files that > > look like this: > > > > input = $(input_line) > > output = $(input_line) > > queue > > > > and invoke condor_submit via lines like: > > > > ls data/*.csv | grep foo | condor_submit -submit_per_line > > > > Comments? Other ideas? > > _______________________________________________ > > HTCondor-users mailing list > > To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx > > with a > > subject: Unsubscribe > > You can also unsubscribe by visiting > > https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users > > > > The archives can be found at: > > https://lists.cs.wisc.edu/archive/htcondor-users/ > > _______________________________________________ > HTCondor-users mailing list > To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a > subject: Unsubscribe > You can also unsubscribe by visiting > https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users > > The archives can be found at: |