Re: [HTCondor-users] Non trivial way of using DAG

[Matthew T West <m.t.west@xxxxxxxxxxxx>]

Hi All,

I would be curious about this sort of DAG too, as its a VERY common one in weather simulations. The UK's weather service is down the road from the university here in Exeter.

https://cylc.github.io/
Here is the workflow tool MetOffice built to run just this sort of thing. I imagine there is a way to do it in DAGMan (maybe) but I couldn't figure out how.

Cheers,
Matt

Matthew T. West
DevOps & HPC SysAdmin
University of Exeter, Research IT
www.exeter.ac.uk/research/researchcomputing/support/researchit
57 Laver Building, North Park Road, Exeter, EX4 4QE, United Kingdom

On 25/10/2023 21:03, Cole Bollig via HTCondor-users wrote:

CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe.

Hi Lorenzo,

This is a very interesting workflow to automate. I do have some questions regarding this:

1. What is the upper limit on the number of times this workflow can cycle? I assume it's a large number, but there must be some number of cycles you expect this work to be finished.
2. Are all N data subset computational jobs identical except for minor variations (different input/output files, arguments, etc)?
3. How important is it for you to know which iteration of the cycle is on?

-Cole Bollig

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Lorenzo Mobilia <l.mobilia@xxxxxxxxxxxxxxxx>
Sent: Wednesday, October 25, 2023 2:45 AM
To: htcondor-users@xxxxxxxxxxx <htcondor-users@xxxxxxxxxxx>
Subject: [HTCondor-users] Non trivial way of using DAG

 

Hi, 

I am finding some difficulties in using DAG. Basically I need to 

1. Take a dataset D1

2. Split it in N subdataset

3. Perform some computation in these N subdataset

4. Merge these subdataset in another dataset D2

5. Restart the process (back to point 1) now using D2

And continuing until specific characteristics have been achieved by the final dataset. The problems are:

A. I don't know a priori how many times I need to split D1

B. I don't know a priori how many times I need to perform this cycle

The solution I came up with is to build a main which controls this flow, but after some cycles it crashes. 

If anyone has some suggestions or is interested in this problem in order to have some other information, please let me know!

Hi, 

Lorenzo

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/