Mailing List Archives

Authenticated access

 		[Computer Systems Lab]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] change to condor_submit - user feedback desired! (was Re: multiple condor_submit's - one cluster)

       Queue METHOD,ITEM from (
...
       Queue METHOD:ITEM from (
...
       Queue METHOD ITEM from (


Or, less subtly,

	... FIELDS SEPARATED BY ":" [RECORDS SEPARATED BY "X"]


On Mon, 2 Mar 2015, David Champion wrote:


Thanks for opening this.  It looks really good to me.  My only
suggestion on it (if comments are still open) concerns this notation:

	Queue METHOD,ITEM from ...

It could be useful to adjust the syntax a bit to allow specifying the
separation symbol (allowing not just whitespace-delimited lists but also
comma, colon, etc).  That could be as simple as:

	Queue METHOD,ITEM from (
		Broil, Chicken
		Fry, Fish
		Bake, Bread
		Bake, Raspberry Pie
		Spit Roast, Wild Javelina
	)

	Queue METHOD:ITEM from (
		Broil:Chicken
		Fry:Fish
		Bake:Bread
		Bake:Raspberry Pie
		Spit Roast:Wild Javelina
	)

	Queue METHOD ITEM from (
		Broil Chicken
		Fry Fish
		Bake Bread
		Bake Raspberry Pie
		Spit Roast Wild Javelina	# TOTALLY DIFFERENT, EWW
	)

... with left and right whitespace stripping for each value.


* On 02 Mar 2015, John (TJ) Knoeller wrote:

There is now a design document here
https://docs.google.com/document/d/10UKaiu1BkirXP6qUmx9DyEOPPSNqKqmS-2foztKi_zU/edit#
That describes our proposed improvements to the Queue statement for
condor_submit.

We plan to have (most?) of this working for the 8.3.5 release.

-tj

On 2/6/2015 10:00 AM, Todd Tannenbaum wrote:

On 2/6/2015 9:04 AM, Krieger, Donald N. wrote:

Hi Todd,

Thanks for posting back with the answers. It's very helpful.

My problem is that the names I'm using for the log files step through
a sequence of strings rather than a sequence of numbers. And I have a
downstream management routine that uses those names which I would
have to alter if I changed the naming sequence.


Hi Don,

Thanks for the above explanation.  Indeed, $(Process) is great in your
submit files, but only if your data files are numerically sequenced!

Perhaps we could create a general solution in condor_submit that addresses
the above and yet would still a) allow all the submits to happen at once
into one cluster and b) not require the user to know how to write scripts.

I listed a couple brainstorm ideas below that would be relatively easy to
implement.  Do folks think that either of the below be helpful? Would love
to hear any feedback, alternative ideas.

regards,
Todd

Some brainstorm ideas:

1. A "queue foreach <filepattern>" command.  Folks could then have submit
files that look like this:

   input = $(file)
   output = $(file).output
   queue foreach data/*.csv

So for each file in subdir data that ends in .csv, a job would be
submitted and $(file) would expand to the path to the file.

and/or

2. A command line option to condor_submit that tells it to read stdin, and
to do a submit for each stdin line, substituting each line from stdin with
$(input_line).  Folks could then have submit files that look like this:

  input = $(input_line)
  output = $(input_line)
  queue

and invoke condor_submit via lines like:

  ls data/*.csv | grep foo | condor_submit -submit_per_line

Comments? Other ideas?
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with
a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/


_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/


--
      David Champion â dgc@xxxxxxxxxxxx â University of Chicago
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/