[Condor-users] New Version of Stork Data Scheduler (v2.0) Released Today

[Sivakumar Kulasekaran <sivakumar@xxxxxxxxxxx>]

The Stork team at Louisiana State University is pleased to announce that a new version of Stork Data Scheduler (Stork 2.0) is now available on the Stork Project web page (www.storkproject.org).   

Stork 2.0 comes with a novel “Throughput Estimation and Optimization Service”. This new service implements a state-of-the-art model developed by Dr. Tevfik Kosar and his team to predict the optimal number of parallel streams to achieve the best data transfer throughput. The model can generate very accurate predictions with minimal sampling overhead. In Stork 2.0, this model is implemented for GridFTP transfers and the team is planning to implement it for other data transfer protocols in future releases.  

This new service can be used for both data throughput estimation and optimization purposes. In case of “estimation”, the users can easily predict how long a data transfer will take using a single stream, what would be the optimal number of parallel streams to use for this transfer, and how long it would take using the optimal number of streams etc. In case of “optimization”, Stork scheduler will automatically determine the optimal number of parallel streams for the data transfer job submitted to it and will use this setting to perform the transfer.  

In addition to this new estimation and optimization service, Stork 2.0 release comes with support for more protocols and more platforms, remote logging capability, checkpointing, security enhancements, and many bug fixes.  

The NSF funded Stork Data Scheduler makes a distinctive contribution to distributed computing community because it focuses on planning, scheduling, monitoring and management of data movement tasks and data resources. Unlike existing approaches, Stork treats data resources and the tasks related to data access and movement as first class entities just like computational resources and compute tasks, and not simply the side effect of computation. Development of Stork is funded through NSF STCI program, and the related research efforts are funded through NSF CAREER program.  

This release of Stork 2.0 includes the following new features:

1. Estimation service which works with GridFtp protocol

2. Optimization service for GridFtp protocol

3. Implementation of Stork log that can retrieve user log from server side to client side

4. Added support for SRM transfer Modules

5. Support for more than 20 different platforms

6. Upgraded Stork to support until GCC version of 4.4

7. Upgraded Classads library to version 1.0.9

8. Reorganized stork directory structures which includes now tmp and log folders

9. Stork Usage statistical collector 

10. GSI credential temporary storage location moved to local stork installation directory from /tmp

11. Other security enhancements

12. Many bug fixes

There will be live demonstrations of Stork 2.0 at the SC10 Conference next week in New Orleans. If you are attending SC10, please stop by LSU/CCT booth (#3613) or send an email to stork-devel@xxxxxxxxxxx.