[HTCondor-users] Question about the curl transfer plugin

[Benoit Roland <benoit.roland@xxxxxxx>]

    Dear all,
    
    I have developed some plugin to enable HTCondor jobs to copy their
    output to storage resources requesting an access token [1].
    
    The process is working fine; the access tokens are properly
    generated and renewed when their remaining life time is about to
    expire,
    and the output of the HTCondor jobs correctly copied to some
    dedicated storage.
    
    So far, to copy the HTCondor output to the storage, I was using a
    small script included in the job.jdl:
    transfer_input_files = transfer-storage.py

This script is executed in the main executable at the very end of
    the job producing the output file:
    ./transfer-storage.py $OUTPUTFILE $STORAGE

It contains a simple davix-put command where the output to be
    transferred and the storage element can be defined.
    I am using the "-k" option to disable the check of the SSL
    credential and all is working fine.
    Alternatively, the "--capath" option specifying the CA certificates
    is also working fine.
    
    I am trying these days to use the transfer plugins available by
    default in HTCondor by including in the job.jdl:
    output_destination = helmholtz+https://xrootd.physik.uni-bonn.de:1094/cephfs/grid/helmholtz/c4p-credmon-test
    with "helmholtz" being the name of my service.
    
    When the job output "test-credential.txt" is being transferred, I
    got the following error message in the ShadowLog:
    
    DoDownload: STARTER at 2a00:139c:4:8b5:0:1:c:81 failed to send
      file(s) to <[2a00:139c:3:2e5:0:21:d2:6c]:9618>
    
    FILETRANSFER:1:non-zero exit (1) from
      /usr/libexec/condor/curl_plugin. 
    
    Error: Peer's certificate issuer has been marked as not
      trusted by the user. 
    ( URL file =
helmholtz+https://xrootd.physik.uni-bonn.de:1094/cephfs/grid/helmholtz/c4p-credmon-test/test-credential.txt
      )
    
    I tried to find in [2,3] if there is a way to define the option "-k"
    or "--capath" without modifying the curl plugin, but I didn't
    succeed so far.
    
    Can these options be specified?
    
    If not, I would slightly modify the plugin according to the need.
    
    Could I then please ask you, in which part of the code I should best
    look at?
    
    In a near future, I would also like to include some transfer plugins
    for xrootd and webdav.
    
    Thanks a lot in advance for your answer!
    
    Cheers,
    Benoit
    
    [1] https://github.com/benoitroland/C4P-HTCondor
    [2] src/condor_utils/file_transfer.cpp
    [3] src/condor_filetransfer_plugins/multifile_curl_plugin.cpp