Re: [HTCondor-users] Support for ambient credentials in built-in S3 plugin

[Tom Downes <tpdownes@xxxxxxxxx>]

Miguel:

I maintain the HTCondor solution for Google Cloud and face a similar challenge. The basic issue is that it's not typically considered aÂgood practice to store credentials in files, especially when they probably have no expiration or a very lengthy expiration. The Condor teamÂfaces a real challenge in supporting a variety of use cases but I think this is a missingÂgap.

What would be interesting to me is if the behavior of S3_SIGN_URLs = False were to make the assumption that the execute points have credentials available to them. In that case, it would implement something like the following on the execute hosts:

GCP:

curl -X GET -o file1 -H "Authorization: Bearer $(gcloud auth print-access-token)" "https://storage.googleapis.com/storage/v1/b/BUCKET/o/OBJECT?alt=media"

AWS (pattern documented here):

curl -X GET -o file1 -H "(CORRECT AWS HEADERS)"Â https://BUCKET.s3.REGION.amazonaws.com/OBJECT

I believe AWS buckets expose their region to unauthenticated users, but that should be subjected to tests.

This could be written in a way where the authorization tokens did not depend upon the presence of gcloud/aws clients, but queried the local instance metadata servers directly for an auth token/header.

In any case, this is all HTTPS and potentially all able to be done with cURL or equivalent.

Tom

On Mon, Mar 6, 2023 at 4:16âPM Miguel Garrido <miguel@xxxxxxxxx> wrote:

Thank you, I noticed that sentence as well. However, I also noticed that if I turned off pre signing then Condor would report that it is transferring files, hence my initial message to this list.

If using instance profile credentials is not something that Condor plans to support perhaps the correct behavior is for Condor to âignoreâ the transfer if there isnât a pre signed URL - since thatâs the only âsupported wayâ out of the box.

Iâll continue supporting my file transfer plugin for now.Â

On Mon, Mar 6, 2023 at 17:10 Todd L Miller <tlmiller@xxxxxxxxxxx> wrote:

> Per the documentation there is native S3 support, perhaps the curl plugin
> is used for this?

    The native S3 support requires you presigning to happen; see the
third sentence in the first paragraph of the documentation to which you
linked.

- ToddM

--

MG

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/