Re: [HTCondor-users] d/l job logs from remote submit / schedd node with py bindings?

[Gavin Price <gaprice@xxxxxxx>]

Hi Todd,

I'm not actually trying to get the output written by the job, rather I'm trying to get the logs written by HTCondor. For instance, I see these files in /condor_workdir in my htcondor-mini container:

```

[root@d91136545a5d /]# cd condor_workdir/
[root@d91136545a5d condor_workdir]# ls
__DUMMY_OUTPUT__ Âcts Âcts-foo-1.log Âcts-foo-2.log
[root@d91136545a5d condor_workdir]# ls cts/foo
cts-foo-1.err Âcts-foo-1.out Âcts-foo-2.err Âcts-foo-2.out
[root@d91136545a5d condor_workdir]#
```

In this case I want to get the *.log, *.err, and *.out files.

In my hands, retrieveÂdoesn't seem to do anything; I suspect this is because there needs to be a schedd running locally...? For my service I'm performing strictlyÂremote submission, so no htcondor daemons are running outside the htcondor-mini container:

```

In [28]: collector = htcondor2.Collector("localhost:9618")Â # exposed htcondor-mini port

In [29]: schedd_ad = collector.locate(htcondor2.DaemonTypes.Schedd)

In [30]: schedd = htcondor2.Schedd(schedd_ad)

In [31]: job_ads = schedd.history(
  ...:   constraint=f"ClusterId == 6",
  ...:   projection=[],
  ...: )

In [32]: job_ads[0]["Iwd"]
Out[32]: '/condor_workdir'

In [33]: job_ads[0]["UserLog"]
Out[33]: '/condor_workdir/cts-foo-2.log'

In [34]: job_ads[0]["Err"]
Out[34]: 'cts/foo/cts-foo-2.err'

In [35]: job_ads[0]["Out"]
Out[35]: 'cts/foo/cts-foo-2.out'

In [36]: job_ads[0]["ClusterId"]
Out[36]: 6

In [37]: job_ads[0]["ProcId"]
Out[37]: 1

In [40]: schedd.retrieve(6)

In [41]: !ls /condor_workdir/

In [42]: schedd.retrieve("6")

In [43]: !ls /condor_workdir/

```

As you can see, HTC knows where the log files are in the htcondor-mini container, but retrieve() doesn't move them to the host running the ipython / htcondor bindings process

Thanks, Gavin

On Tue, Sep 30, 2025 at 3:24âPM Todd L Miller via HTCondor-users <htcondor-users@xxxxxxxxxxx> wrote:

> This sounds like what I'm looking for but I'm not sure how I would do this
> with the python bindings - could you elaborate?

    The API is perhaps a little clunky, but the general sequence is as
follows:


import htcondor2

# Submit the job.
schedd = htcondor2.Schedd()
submit = htcondor2.Submit(...)
result = schedd.submit(submit, spool=True, count=1)

# Send the input files.
schedd.spool(result)

# Wait for the job to finish.
# (Code left as an exercise for the reader.)

# Retrieve the output files.
schedd.retrieve(result.cluster())


As usual, the job submit file specifies what files constitute the output;
I don't recall where in the process, if at all, the remaps take place,
but `retrieve()` should behave identically to the `condor_transfer_data`
command-line tool in that respect. (The default, IIRC, is to write the
output files to the directory the tool is invoked in.)

-- ToddM
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe

The archives can be found at: https://www-auth.cs.wisc.edu/lists/htcondor-users/