Re: [HTCondor-users] HTCondor file-transfer vs networked storage

[Nick LeRoy <leroy.nick@xxxxxxxxx>]

On Sat, Aug 20, 2022 at 8:44 AM Matthew T West via HTCondor-users
<htcondor-users@xxxxxxxxxxx> wrote:
>
> Hi All,
>
> When working on a single homogeneous compute cluster, are there any advantages to using HTCondor's file-transfer rather than working off shared network storage? I guess it would depend on the network and storage speeds.
>
> It's just interesting that the "always work in local scratch" mindset I am used to is seen a serious backward step performance wise:
>
> Scratch therefore only useful if your network storage or interconnects are slow or saturated ... copying bulk data to local storage / getting all users to copy to local/scratch storage is a quick way to saturate your storage infrastructure.
>
> I can find other instances of this HPC conventional wisdom and it intuitively makes sense. But I don't understand networked storage well, so I am asking the HTCondor hivemind for their thoughts.

Matt,

You need to remember that HTCondor can work in many different
environments, among these being WANs, campus-type structures, and
grids.  For these types of scenarios, file transfer is preferable, if
not required.

-Nick