HTCondor Project List Archives


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-devel] Is string quoting for custom job attributes broken in 7.4.x?

Ian,
What you observe is the crummy old-classad syntax. I've discovered this in the past while working in the code, but I didn't realize the documentation was wrong. We'll need to fix that.
The new-classad library used since 7.5.2 has C-style string literal syntax, but, unfortunately, as used in condor today, it is hidden behind the compatibility layer that mimics the old syntax for backward compatibility, so we're still stuck in the bad old days.
The only way I know of to put a newline into a string in the submit file is to use "$ENV()" to reference an environment variable that contains a newline in its value. However, the schedd will reject the job if you do this. The schedd's classad log is known to be broken when there are newlines in strings, so the schedd won't accept jobs that do this. 

The old-classad syntax is thus:

===========
String
A double quote character, followed by a list of characters terminated by a double quote character. Within the list of characters, a backslash followed by a double quote produces a literal double quote character within the string rather than terminating the string. A backslash followed by any other character produces a backslash followed by that character. In other words, the backslash is _not_ a general escape 
character.  It only has special meaning when followed by a double quote.
There is some ambiguity in this definition about whether it is possible to
produce a string that ends in a backslash, since this backslash could cause
the terminating quote character to be treated as a literal quote. The treatment 
of this case is not consistent within Condor.  In some contexts, a backslash
before the terminal quote character produces a string ending in a backslash
and in other cases, it produces a literal quote and therefore prevents the
quote from serving its purpose as a terminator of the string.
===========
I hope some day we will be able to switch to new-classad string syntax and fix the schedd classad log. Then you will be able to pack more interesting things into your job ads. Until then, tread carefully! 

--Dan

On 3/25/11 1:31 PM, Ian Chesal wrote:

I'm having some trouble wrapping my head around Condor, specifically condor_submit, string quoting for custom job attributes.

Can someone help explain the behaviour here? It seems inconsistent at best and perhaps even broken.

The docs for string attributes in 7.4 say this:

  String
  A double quote character, followed by an list of characters terminated by a double
  quote character. A backslash character inside the string causes the following character
  to be considered as part of the string, irrespective of what that character is.
  See: http://www.cs.wisc.edu/condor/manual/v7.4/4_1Condor_s_ClassAd.html#SECTION00511100000000000000

That's a vague bit of description so I was doing some testing to see what it amounts to. The following examples highlight some of the confusion, all are in the context of a custom attribute added to a job via the submit file, submitted with condor_submit.

The first bit of weird is how literal \ is treated in the string. For example:

  +submission_dir = "\\foo\bar"

  $ condor_q -l | grep submission_dir
  submission_dir = "\\foo\bar"

  $ condor_q -f "%s\n" submission_dir
  \\foo\bar

That's a bit odd based on the description given in the docs. I would have expected \\ to be a literal \ character and \b to be a bell character. This might seem okay until you realize that it removes your ability to encode a tab or newline character in to the string value of an attribute. For example:

  +submission_dir = "\\foo\tab"

I would expect that \t would be a tab character, but it isn't:

  $ condor_q -l | grep submission_dir
  submission_dir = "\\foo\tab"

  $ condor_q -f "%s\n" submission_dir
  \\foo\tab

I would have expect that \t was the tab character and \\f was a f character preceded by a \ character. But that doesn't seem to be the case.

How can I encode non-alphanumerics in string values? \n, \t, \b type stuff?

And for the final bit of weird: what if we want a " in a string? It can be done, using the \ to escape it. That seems okay until you see how Condor handles the formatting for the output:

  +submission_dir = "\\foo\"bar"

  $ condor_q -l | grep submission_dir
  submission_dir = "\\foo\"bar"

  $ condor_q -f "%s\n" submission_dir
  \\foo"bar

Interesting -- the string formatted version of that attribute's value *removed* the \ character in front of the " -- why in this case, but not in the other cases? I would have expected "\\foo\bar" in the first example to yield "\foobar" when printed with string formatting if the \ was being handled consistently for all strings. But \ appears to have special meaning depending on which character it precedes.

Can someone clear up the rules for me?

Thanks!

- Ian