Re: [Condor-devel] Avoiding redundant executables in the SPOOL

[Alan De Smet <adesmet@xxxxxxxxxxx>]

A few general points:

- It will certainly be possible to disable this functionality.
  We haven't decided if it will default on or off.

- The default configuration will something at least as secure as
  MD5 or SHA-1.  This puts the odds of an accidental collision at
  1 in 2^128.  This is such a small chance that it's worth
  treating a impossible.  

Bruce Beckles <mbb10@xxxxxxxxx> wrote:
> Question: what happens with when_to_transfer_output=ON_EXIT_OR_EVICT?

I'll need to check the code to be certain, but I'm pretty sure it
will do the Right Thing.  Output transferred back on exit or evict ends
up in a subdirectory.  I don't believe the ickpt file is
overwritten under any circumstances. 

> - I've installed a front-end that allows my users to submit
> jobs to my compute resources.  At the back end it uses Condor.
> Since my users never had direct access to Condor I submitted
> all jobs as the same user (just ensuring that the input and
> output for each job were in separate subdirectories), because
> that way the Condor submit hosts don't need to know about all
> my different users.
> 
> Now I MUST make copy_to_spool=false, or hash collisions mean
> that the wrong executable may be run (either maliciously or by
> accident).  You better make sure I know this in BIG, BRIGHT,
> GLOWING letters before I upgrade my Condor installation to a
> version of Condor with this feature.

A malicious collision is difficult, but recent attacks on MD5
have shown that it's possible if the attacker can create both the
good and evil binaries.  Research suggests SHA-1 will soon have
similar breaks.  So this is worth considering.

The people at risk are those who explicitly turn copy_to_spool on
(it's off by default), or who submit standard universe jobs
(which force copy_to_spool to TRUE).

I'll note that if you've got users submitting arbitrary
executables to a shared account, you've already got a risky
situation with very tricky configuration requirements.  If those
executables have WRITE access to the schedd, on the same machine
or over the network, it will be able modify other jobs with the
same Owner easily.  And while Condor tries really hard to track a
user job, it is possible for a job to sneak a process out.  That
rogue process can then go on to mess with later jobs that show up
under the same user.  If your machines are willing to run two
jobs at the same time, it's possible for user A and B's jobs to
run at the same time and potentially mess with each other.
Dedicated run accounts can solve the last two issues, but the
jobs no longer run as the user, and this isn't the default.  So
an admin with this environment already facing a difficult
challenge.

Out of curiosity, is anyone aware of a pool run in this way?  I'm
aware of front ends that submit all jobs as the same user, but
they run limited executables.  I'm aware of front ends that allow
arbitrary executables, but they submit the jobs as different
Owners.  Given the challenges of securing such a system, I'd be a
bit surprised to learn that someone was doing it.

That said, I agree, we don't want to have someone upgrade Condor
and introduce a security hole into their configuration.  We'll
keep this in mind when we consider making this the default.
Either way, I'll make sure that you can turn this off.  If it's
on by default, the upgrade notes will have a clear warning about
the change in behavior.

I'd been mulling letting you do their own hashing.  This would
help in this case if you wanted the linking functionality.  To
make up a syntax right now, instead of saying,
"CmdHashParts=CmdMD5", you could say
"CmdHashParts=CmdMD5,UserEmail" and add UserEmail or some other
identifying information for your user to the job.  Exactly who
can specify the hash, the user, the admin, or both (admin
overriding user), I'm still considering.

> - I use "pool accounts" for my users, i.e. there is a collection of
>    accounts that get shared out between users depending on who is using my
>    resources at the time.  User B submits an executable with the same hash
>    as user A, but which is _not_ the same executable as that submitted by
>    user A (hash collision).  User B is/was using pool account ACC01.  User
>    B leaves (temporarily or permanently), but doesn't bother to make sure
>    they've removed all their existing jobs, and user A now gets assigned
>    pool account ACC01.

We're talking about two different users who end up sharing the
same "Owner," yes?  Because if the Owner is different, there is
no risk.  If they are sharing an Owner, the bit about this being
potentially insecure above still applies.

> - Multiple submitters to the same schedd (e.g. via Condor-C,
> etc).  If a bad person can masquerade as me on ANY of those
> submitters they can submit a bad executable that will be run
> instead of the one I actually submitted.
> 
> You might say: "but in that case you've got a problem anyway".
> Yes, but previously the bad person could submit jobs and
> possibly (depending on the set-up) affect my jobs currently in
> the queue.  Now, they can DEFINITELY affect my jobs by causing
> them to run a bad executable.  That bad executable might steal
> the input data of my real jobs (which might be valuable) and
> send it to the bad person.

If you have submit access and can spoof someone else, you have
qedit and rm access (they're all just WRITE access).  Given that,
I can wreak havoc on your jobs today.

condor_qedit mbb10 CopyToSpool false
condor_qedit mbb10 Cmd /path/to/evil/binary

Our security is based on trusting your identity.  Condor isn't
set up to defend an identity against itself, nor do I think that
it's realistic.

So in summary, in that case you've got a problem anyway.

> Obviously you know that using hard links makes this filesystem
> specific.  In particular, where you get filesystems
> masquerading as "normal" UNIX-type filesystems, you may have
> the situation that the filesystem's "support" for hard links is
> actually by making duplicate copies of the file's _data_.

The assumption is that hard links work by creating a reference
counted links to the same inode.

If hard links simply don't work (lack of permission, or a file
system that doesn't support them), the system falls back to the
old behavior.  We'll have wasted some CPU time.

If hard links seem to work, but don't work as assumed, as you
note, everything will work, you'll end up wasting CPU time and
disk space.  I've not encounter a file system with such behavior,
but perhaps I'm lucky.  Is this sort of madness common?

> Also, there are situations where not all users are allowed to 
> make hard links (presumably the schedd is running as root (if it can) when 
> it does all this?).  I guess, from your previous e-mail, that here you 
> fall back to the old non-hashed behaviour?

Yes.

Pretty much any failure case will revert to the old behavior.

> One more thing: in a previous e-mail, you said that "this won't
> work in Windows".  Why not?  NTFS has supported hard links
> since at least NTFS 3.0 (Windows 2000).

It doesn't work because Alan is living in the past.  Right, we
should have Windows support (assuming NTFS).

> ...so if a user can create a hard link to the hash path then
> the ickpt file will never be unlinked.  What are the
> circumstances in which a user can do this?  

If a user can create hard links to file in the SPOOL today you
have the same problem.  Nothing changes.

> Windows (at least from Windows 2000 onward) is perfectly happy with a 
> comma (,) in the filename.  (Of course, as with spaces in filenames, users 
> need to surround the filename in double quotes.)
> 
> My MacOS 10.4.10 system was also happy with a comma in the filename.

Thanks for the info.  I haven't thought much about the escaping
algorithm, other than to establish that we need one.  The fewer
things I need to escape, the less noisy the filenames will be.
Before nailing this down I'll survey our supported platforms and
see what the largest shared set is.

-- 
Alan De Smet                              Condor Project Research
adesmet@xxxxxxxxxxx                http://www.cs.wisc.edu/condor/