[Chtc-users] Important Annual Updates from CHTC!!

[Lauren Michael <lmichael@xxxxxxxx>]

Greetings CHTC User,

We have recently undergone some exciting changes and want you to know about them. Many will affect how you use the CHTC's resources for computing work. Here's a list of the topics we'll cover further below:

• Switch to our newer, more powerful HTCondor submit node
  
• CHTC now supports high-throughput computing (HTC) AND high-performance computing (HPC) 
  
• Make sure to update YOUR copies of CHTC Tools!
• Automatically send HTC jobs to many more computers with "Flocking" and "Glidein"
  
• Be a test case for Einstein@Home via HTCondor
  
• Updated online 'hello-world' example
• HTC support for NEW versions of Matlab and R
• Give us your feedback!
  

Accepting Users to Our New Submit Node, submit-3.chtc.wisc.edu

Our admins have been busy building and testing our newest, more-powerful submit node, submit-3.chtc.wisc.edu. While we will eventually require all users to switch to this node so that we can also enhance submit-1.chtc.wisc.edu, we are now accepting early volunteers to switch to submit-3.

Please write to chtc@xxxxxxxxxxx if you are interested in switching to the new submit node. Users of submit-3 will use their NetID's for logging in, which allows us to better monitor usage and security on CHTC compute resources and will eventually be required for all UW-Madison users of CHTC.

CHTC's New HPC Cluster

For users needing many cores working together on a one for few compute tasks (typically through MPI programming), the CHTC now offers a cluster of identical compute hardware with a shared file system and fast Infiniband networking for rapid communication between processes. This type of computing and cluster is traditionally referred to as "high-performance computing" (HPC). As for our HTC compute resources, the HPC Cluster is freely available to UW-Madison researchers, and users requiring priority access can purchase into this cluster for ownership of hardware within the cluster. Read more about the HPC Cluster on the website of CHTC's research computing partner, the Advanced Computing Infrastructure (ACI), and email chtc@xxxxxxxxxxx to discuss access to the cluster.

Send HTC Work beyond the CHTC to the Open Science Grid

Did you know that the CHTC Pool is just one of many HTCondor Pools on campus that you can send your jobs to for FREE? By default, HTC jobs submitted to CHTC will only be run on computers in the CHTC Pool (~7000 cores), but shorter jobs can and SHOULD be additionally sent to other Pools on the UW Grid (18,000 cores), and even to the nation-wide Open Science Grid (OSG) of more than 100 sites of clusters and campus grids. Users sending jobs to these additional resources can more than quadruple their overall throughput. Here's how to do it: 

We typically find that jobs less than ~4 hours (or that use self-checkpointing) can run well on the UW Grid by adding

+WantFlocking = true

to the submit file. Jobs less than ~2 hours each (or that self-checkpoint) should also be sent to the OSG by adding the following submit file line:

+WantGlidein = true

Make sure to add these to your submit files after testing a few jobs for run time, memory, and disk (as always). NOTE: OSG access is restricted to CHTC-owned submit nodes (and a few rare cases) only. See also: http://chtc.cs.wisc.edu/helloworld.shtml

Try out HTCondor's BOINC (Einstein@Home) Capability

Please contact us (chtc@xxxxxxxxxxx) if your HTC work fits the following BOINC requirements, in order to backfill millions of computers all over the world:

-static, relatively small binary code (C and Fortran code; not Matlab, Python, R, etc.)

-short or self-checkpointing jobs

-few and small files (input and output)

-very large batches

Updated CHTC 'Hello World' Guide

-Our hello-world guide now includes information on using submit file arguments and "transfer_input_files", demonstrates condor_rm, and has extra information about job testing. Check it out and let us know what you think: "Run Your First HTC Jobs" on the website, http://chtc.cs.wisc.edu/helloworld.shtml

We Now Support Matlab 2013b and R 3.0.1

If you're already familiar with our high-throughput computing tools for Matlab, Python, and R, you'll be happy to know that we have recently enabled Matlab 2013b and R-3.0.1 for our "build" tools, and within the ChtcRun package. Check out our updated guides for instructions.

We'll be sending a follow-up email to announce when the Matlab tools are fully tested and will eventually set the default version to 2013b (versus the current/past default of 2011b).

Email the CHTC (chtc@xxxxxxxxxxx) with Feedback Regarding:

-this email

-features mentioned in this email

-any other resources or services provided by CHTC

We love to hear from you on how we can improve the CHTC!

Happy Computing,

The CHTC Team