<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Matt,
<div class=""><br class="">
</div>
<div class="">Three ideas come to mind:</div>
<div class=""><br class="">
</div>
<div class="">1. We've seen sites wrap the HTCondor interaction in higher-level tools that users use instead of HTCondor commands:</div>
<div class=""> - Pro: Users need to learn very little -- only enough to invoke the high-level tool correctly.</div>
<div class=""> - Con: Users don't learn how to help themselves, can only support a limited number of workflows.</div>
<div class=""><br class="">
</div>
<div class="">2. The USER_JOB_WRAPPER, run at the startd, provides a powerful hook for customizing the job startup environment. Separately, note that Singularity can be used to start jobs inside containers you curate.</div>
<div class=""> - Pro: As you execute arbitrary code, you can do a number of transformations impossible to do at the schedd side.</div>
<div class=""> - Con: As far as users are concerned, this is outright magic. They'll learn the HTCondor system more but may not be able to transfer the knowledge to other places without said magic.</div>
<div class=""><br class="">
</div>
<div class="">3. Perhaps you can just provide an easy way to generate the right environments and have users utilize HTCondor directly. That is, write a wrapper on the submit side which takes a conda environment and produces a corresponding file (which is
really a Singularity image - but no need to tell the user that) and provide explanation of how to run inside that file. This way, the conda environment is always activated in the job - pure python.</div>
<div class=""> - You still end up doing some work to write the "conda-to-singularity" script but after that the users can still see all the pieces working together.</div>
<div class=""><br class="">
</div>
<div class="">Brian<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jul 27, 2020, at 12:42 PM, West Matthew <<a href="mailto:matthew.west@aalto.fi" class="">matthew.west@aalto.fi</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div id="divtagdefaultwrapper" dir="ltr" style="caret-color: rgb(0, 0, 0); font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;" class="">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" class="">
<div style="margin-top: 0px; margin-bottom: 0px;" class="">Hi All,</div>
<br class="">
I want to stress something that I might not have made clear in previous messages.<br class="">
<br class="">
I work with a lots of folks who have little experience with running software outside of a single machine. They have a need to scale up their computing effort in order to do a quality analysis but HTC is not why they got interested in the work. This is not to
say a biologist or civil engineer cannot learn the tools to containerize their own software, but often incentives prioritizing immediate results make researchers hesitant to try new tools.<span class="Apple-converted-space"> </span><br class="">
<br class="">
- Why do I need all these other skills just to run software like it does on my laptop?<span class="Apple-converted-space"> </span><br class="">
<div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class="">
For mature projects and more experienced developers, I believe using containers is an easy sell. But I hope y'all can understand the predicament of lowering the barrier to entry to working in an HTC framework for beginners.<br class="">
<br class="">
Cheers,<br class="">
Matt<br class="">
<br class="">
</div>
</div>
<hr tabindex="-1" style="display: inline-block; width: 953.53125px;" class="">
<div id="divRplyFwdMsg" dir="ltr" class=""><font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>HTCondor-users <<a href="mailto:htcondor-users-bounces@cs.wisc.edu" class="">htcondor-users-bounces@cs.wisc.edu</a>>
on behalf of Michael Pelletier via HTCondor-users <<a href="mailto:htcondor-users@cs.wisc.edu" class="">htcondor-users@cs.wisc.edu</a>><br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Monday, July 27, 2020 1:11:19 AM<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span>HTCondor-Users Mail List<br class="">
<b class="">Cc:</b><span class="Apple-converted-space"> </span>Michael Pelletier<br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [HTCondor-users] Conda env with workstation pool</font>
<div class=""> </div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">I set up a Singularity container to auto-activate a specified environment at startup, which works pretty well for my users. You might also be able to use the
OS-native python to write a python-language wrapper that pulls in all the activation environment variables to its own environment, and then invokes the target Python script.</span></div>
<p style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""> </span></p>
<div class="">
<div style="margin: 0in 0in 0.0001pt; line-height: 17.600000381469727px; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<b class=""><span style="font-size: 10pt; line-height: 14.666666030883789px; font-family: Arial, sans-serif;" class="">Michael V Pelletier</span></b><sub class=""><span style="font-size: 10pt; line-height: 14.666666030883789px; font-family: Arial, sans-serif;" class=""></span></sub></div>
<div style="margin: 0in 0in 0.0001pt; line-height: 17.600000381469727px; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<span style="font-size: 9pt; line-height: 13.199999809265137px; font-family: Arial, sans-serif;" class="">Principal Engineer<br class="">
</span><b class=""><span lang="FR-BE" style="font-size: 9pt; line-height: 13.199999809265137px; font-family: Arial, sans-serif; color: rgb(192, 0, 0);" class=""><br class="">
</span></b><b class=""><span lang="FR-BE" style="font-size: 10pt; line-height: 14.666666030883789px; font-family: Arial, sans-serif; color: rgb(206, 17, 38);" class="">Raytheon Technologies</span></b><span style="font-size: 9pt; line-height: 13.199999809265137px; font-family: Arial, sans-serif;" class=""></span></div>
<div style="margin: 0in 0in 0.0001pt; line-height: 17.600000381469727px; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<span style="font-size: 9pt; line-height: 13.199999809265137px; font-family: Arial, sans-serif;" class="">Information Technology</span><span style="font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""></span></div>
<div style="margin: 0in 0in 0.0001pt; line-height: 17.600000381469727px; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<span style="font-size: 9pt; line-height: 13.199999809265137px; font-family: Arial, sans-serif;" class="">Digital Transormation & Innovation</span></div>
<div style="margin: 0in 0in 0.0001pt; line-height: 19.200000762939453px; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<u class=""><span style="font-size: 8pt; line-height: 12.800000190734863px; font-family: Arial, sans-serif;" class=""><span style="text-decoration: none;" class=""> </span></span></u></div>
</div>
<p style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""> </span></p>
<div class="">
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0in 0in;" class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<b class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">From:</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="Apple-converted-space"> </span>HTCondor-users <<a href="mailto:htcondor-users-bounces@cs.wisc.edu" class="">htcondor-users-bounces@cs.wisc.edu</a>><span class="Apple-converted-space"> </span><b class="">On
Behalf Of<span class="Apple-converted-space"> </span></b>West Matthew<br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Friday, July 24, 2020 7:48 AM<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span>HTCondor-Users Mail List <<a href="mailto:htcondor-users@cs.wisc.edu" class="">htcondor-users@cs.wisc.edu</a>><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>[External] Re: [HTCondor-users] Conda env with workstation pool</span></div>
</div>
</div>
<p style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
</p>
<div id="divtagdefaultwrapper" class="">
<div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-family: Calibri, sans-serif;" class="">Hi Josh,<br class="">
<br class="">
I am working from the CHTC recommendation using conda pack. I am curious if you know a way to activate the conda environment from within a python script? Having to wrap my python executable with a bash script is rather frustrating when I am trying to get away
from multi-language situation.<br class="">
<br class="">
I can create the directory and extract the contents of the tarball. I just need to activate the env.<br class="">
<br class="">
Cheers,</span></div>
<div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-family: Calibri, sans-serif;" class="">Matt</span></div>
</div>
<div align="center" style="text-align: center; margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<hr width="98%" size="2" align="center" class="">
</div>
<div id="divRplyFwdMsg" class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<b class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">From:</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="Apple-converted-space"> </span>HTCondor-users <</span><a href="mailto:htcondor-users-bounces@cs.wisc.edu" style="color: blue; text-decoration: underline;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">htcondor-users-bounces@cs.wisc.edu</span></a><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">>
on behalf of Josh Karpel via HTCondor-users <</span><a href="mailto:htcondor-users@cs.wisc.edu" style="color: blue; text-decoration: underline;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">htcondor-users@cs.wisc.edu</span></a><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">><br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Thursday, July 23, 2020 4:41:59 PM<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span>HTCondor-Users Mail List<br class="">
<b class="">Cc:</b><span class="Apple-converted-space"> </span>Josh Karpel<br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [HTCondor-users] Conda env with workstation pool</span></div>
<div class="">
<p style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
</p>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
That pretty much lines up with what we tell people to do at CHTC: <a href="https://urldefense.com/v3/__http:/chtc.cs.wisc.edu/conda-installation.shtml__;!!MvWE!WuPvPoWepMwM6AtC-xjjbqI0ch352mJpw3heCLd5ufGBsPongbywiG6AxTJ-Wn1ONGI4aA$" style="color: blue; text-decoration: underline;" class="">http://chtc.cs.wisc.edu/conda-installation.shtml</a></div>
<div class="">
<p style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
</p>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<p style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
</p>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
Josh Karpel</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<a href="mailto:karpel@wisc.edu" target="_blank" style="color: blue; text-decoration: underline;" class="">karpel@wisc.edu</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
</p>
</div>
</div>
<p style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
</p>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
On Thu, Jul 23, 2020 at 8:25 AM Michael Pelletier via HTCondor-users <<a href="mailto:htcondor-users@cs.wisc.edu" style="color: blue; text-decoration: underline;" class="">htcondor-users@cs.wisc.edu</a>> wrote:</div>
</div>
<blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0in 0in 0in 6pt; margin: 5pt 0in 5pt 4.8pt;" class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">I’ve built a Singularity definition file that installs Miniconda and creates an environment YAML file, then builds the environment, and then configures the container
so that the environment is activated automatically at the startup of the Singularity container. With Miniconda a CUDA 10.2 and 18.04 Ubuntu Singularity container file is about 850 megabytes in size.</span></div>
<p style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""> </span></p>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">Singularity doesn’t necessarily have an extra infrastructure layer, as it doesn’t require any services from the host – I bet it would be possible to input-transfer
the Singularity executable and run it on an input-transferred container.<span class="Apple-converted-space"> </span></span></div>
<p style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""> </span></p>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">Alternatively, you could build out the full virtualenv in a directory with Miniconda, and then input-transfer that whole directory and activate it when the job
starts up, which would eliminate the need for modules to be available on the exec node.<span class="Apple-converted-space"> </span></span></div>
<p style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""> </span></p>
<div class="">
<div style="margin: 0in 0in 0.0001pt; line-height: 17.600000381469727px; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<b class=""><span style="font-size: 10pt; line-height: 14.666666030883789px; font-family: Arial, sans-serif;" class="">Michael V Pelletier</span></b></div>
<div style="margin: 0in 0in 0.0001pt; line-height: 17.600000381469727px; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<span style="font-size: 9pt; line-height: 13.199999809265137px; font-family: Arial, sans-serif;" class="">Principal Engineer<br class="">
</span><b class=""><span lang="FR-BE" style="font-size: 9pt; line-height: 13.199999809265137px; font-family: Arial, sans-serif; color: rgb(192, 0, 0);" class=""><br class="">
</span></b><b class=""><span lang="FR-BE" style="font-size: 10pt; line-height: 14.666666030883789px; font-family: Arial, sans-serif; color: rgb(206, 17, 38);" class="">Raytheon Technologies</span></b></div>
<div style="margin: 0in 0in 0.0001pt; line-height: 17.600000381469727px; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<span style="font-size: 9pt; line-height: 13.199999809265137px; font-family: Arial, sans-serif;" class="">Information Technology</span></div>
<div style="margin: 0in 0in 0.0001pt; line-height: 17.600000381469727px; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<span style="font-size: 9pt; line-height: 13.199999809265137px; font-family: Arial, sans-serif;" class="">Digital Transormation & Innovation</span></div>
</div>
<p style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""> </span></p>
<div class="">
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0in 0in;" class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
<b class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">From:</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="Apple-converted-space"> </span>HTCondor-users <</span><a href="mailto:htcondor-users-bounces@cs.wisc.edu" target="_blank" style="color: blue; text-decoration: underline;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">htcondor-users-bounces@cs.wisc.edu</span></a><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">><span class="Apple-converted-space"> </span><b class="">On
Behalf Of<span class="Apple-converted-space"> </span></b>West Matthew<br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Thursday, July 23, 2020 7:03 AM<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span></span><a href="mailto:htcondor-users@cs.wisc.edu" target="_blank" style="color: blue; text-decoration: underline;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">htcondor-users@cs.wisc.edu</span></a><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>[External] [HTCondor-users] Conda env with workstation pool</span></div>
</div>
</div>
<p style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
</p>
<div id="gmail-m_2912201187930689390divtagdefaultwrapper" class="">
<div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-family: Calibri, sans-serif;" class="">I am trying to run an analysis on my local workstation pool that relies on software in a conda virtual environment. When the job runs on a remote
machine, it does not have access to the libraries in that env back on the submit machine.<br class="">
<br class="">
Given that virtual environments are common practice when running locally, I an hoping there is some means of making python libraries accessible without resorting to an extra infrastructure layer like Docker.</span></div>
<p style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-family: Calibri, sans-serif;" class=""> </span></p>
<div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-family: Calibri, sans-serif;" class="">Cheers,</span></div>
<div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-family: Calibri, sans-serif;" class="">Matt</span></div>
</div>
</div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">
_______________________________________________<br class="">
HTCondor-users mailing list<br class="">
To unsubscribe, send a message to<span class="Apple-converted-space"> </span><a href="mailto:htcondor-users-request@cs.wisc.edu" target="_blank" style="color: blue; text-decoration: underline;" class="">htcondor-users-request@cs.wisc.edu</a><span class="Apple-converted-space"> </span>with
a<br class="">
subject: Unsubscribe<br class="">
You can also unsubscribe by visiting<br class="">
<a href="https://urldefense.com/v3/__https:/lists.cs.wisc.edu/mailman/listinfo/htcondor-users__;!!MvWE!WuPvPoWepMwM6AtC-xjjbqI0ch352mJpw3heCLd5ufGBsPongbywiG6AxTJ-Wn2QpuKMtw$" target="_blank" style="color: blue; text-decoration: underline;" class="">https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users</a><br class="">
<br class="">
The archives can be found at:<br class="">
<a href="https://urldefense.com/v3/__https:/lists.cs.wisc.edu/archive/htcondor-users/__;!!MvWE!WuPvPoWepMwM6AtC-xjjbqI0ch352mJpw3heCLd5ufGBsPongbywiG6AxTJ-Wn2XQz5MjQ$" target="_blank" style="color: blue; text-decoration: underline;" class="">https://lists.cs.wisc.edu/archive/htcondor-users/</a></div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">HTCondor-users
mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">To
unsubscribe, send a message to <a href="mailto:htcondor-users-request@cs.wisc.edu" class="">
htcondor-users-request@cs.wisc.edu</a> with a</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">subject:
Unsubscribe</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">You
can also unsubscribe by visiting</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""><a href="https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users" class="">https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users</a></span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">The
archives can be found at:</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""><a href="https://lists.cs.wisc.edu/archive/htcondor-users/" class="">https://lists.cs.wisc.edu/archive/htcondor-users/</a></span></div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>