Re: [HTCondor-users] purpose of pegasus

[Keith Brown <keith6014@xxxxxxxxx>]

Mats,

Thanks for the detailed response. 

I was in the process of writing a Python API which handles job dependencies so I was doing research to make sure I wasn't reinventing the wheel. 

On Sun, Oct 27, 2013 at 1:49 PM, Mats Rynge <rynge@xxxxxxx> wrote:

On 10/27/2013 08:48 AM, Keith Brown wrote:
> I am trying to understand what is the purpose of pegasus. It seems its a
> frontend for DAGMan, is that a correct observation?

Keith,

Pegasus is using DAGMan and the rest of the HTCondor stack to execute
workflows. The value added is at a higher level. Some of the features
Pegasus provides on top of DAGMan:

Pegasus workflows are described in an abstract format, which means that
the description does not include data and software locations. These are
looked up at planning time, and this enables portability of the
workflow. You can move the same workflow around to resources such as
campus grids, supercomputers, or clouds.

Pegasus tracks all the data in the workflow, and will add data
management tasks, such as data staging and cleanup, to the executable
workflow (i.e. the DAG).

At planning time, Pegasus can apply graph transforms to the workflow,
such as cluster short tasks into a larger jobs.

Pegasus collects extra provenance data during the execution. This allows
the user to get reports at the end of the run (pegasus-statistics) and
debugging help if the workflow fails (pegasus-analyzer).

Some more information:

http://pegasus.isi.edu/wms/docs/latest/

http://pegasus.isi.edu/wms/docs/latest/execution_environments.php

http://pegasus.isi.edu/wms/docs/latest/running_workflows.php#mapping_refinement_steps

Please let me know if you would like to try out Pegasus. We can help
getting you requirements / current setup translated into a Pegasus workflow.

--
Mats Rynge
USC/ISI - Pegasus Team <http://pegasus.isi.edu>