cylc.github.io Open in urlscan Pro
2606:50c0:8002::153  Public Scan

Form analysis
0 forms found in the DOM

Text Content

C
Y
L
C

Home
Documentation
Sites
Support
Dev
News
Home
Documentation
Sites
Support
Dev
News


CYLC IS A GENERAL PURPOSE WORKFLOW ENGINE THAT ALSO ORCHESTRATES CYCLING SYSTEMS
VERY EFFICIENTLY. IT IS USED IN PRODUCTION WEATHER, CLIMATE, AND ENVIRONMENTAL
FORECASTING ON HPC, BUT IS NOT SPECIALIZED TO THOSE DOMAINS.

Cylc is an active Open Source project on GitHub.
See Cylc Publications and Reports below.


NEXT RELEASE:  8.3.X (0%)

0% 100%


Cylc is a decentralised, distributed, DAG / DCG scheduler.

Scheduler Cylc can schedule inter-dependent collections of tasks (aka workflows)
to run on platforms of your choice. Decentralised Cylc employs a decentralised
architecture where each workflow execution is fully independent. This means that
workflows are isolated for robustness and Cylc deployments scale horizontally.
Distributed Cylc workflows can be distributed across a pool of servers to
facilitate large multi-user setups. Workflows can distribute their workload
across multiple platforms. DAG Cylc can schedule Directed Acyclic Graphs of
tasks (i.e. workflows which do not cycle) similar to other tools. DCG Cylc's
unique cycling model allows it to run cyclic workflows too. This makes Cylc
especially powerful for cases where later repetitions of tasks depend on the
output of earlier instances. Tasks can repeat on different intervals within the
same workflow and complex inter-cycle dependencies can be defined between them.


CYLC DOES CYCLING PROPERLY

Cylc unwinds the cycle loop to get a dependency graph with no barrier between
cycles. This allows interleaved cycles for:

 * Fast catch-up from delays
 * Sustained high throughput off the clock
 * Seamless transitions between fast catch-up and clock-limited real-time
   scheduling






POWERFUL USER INTERFACES

Alongside a powerful command line interface, Cylc has a web-based GUI for
monitoring, controlling and analysing workflows.

 * Monitor all your workflows from one place, even in a distributed deployment.
 * Tabbed interface with multiple views catering to different use cases.
 * View workflow logs, task output, timings and more.
 * View the workflow graph live.
 * Edit task definition on-the-fly.
 * Manually intervene with workflow execution at runtime.


JUPYTER LAB INTEGRATION

Optional Jupyter Lab integration allows you to deploy the Cylc GUI and Jupyter
Lab together, providing an all in one research platform ready to be deployed to
on premise or cloud systems.


JUPYTER HUB INTEGRATION

Deploy the Cylc GUI via Jupyter Hub to unlock institution-scale multi-user
deployments with fine-grained authorisation allowing users to be granted
permissions to view and control workflows running under other user accounts.
 * Teams can collaborate on shared projects.
 * Different levels of support can be granted different levels of control of
   production systems.


WHY USE CYLC?

Production Ready
 * Cylc is used 24/7 in production weather forecasting, notorious for the size
   and complexity of its workflows.
 * Seamless transition from simulated time to real time for ease of development
   and testing.

Scaling
 * Each workflow gets a scheduler that runs as the user and relies only on the
   filesystem.
 * Schedulers can be distributed over a pool of hosts, and jobs over job
   platforms.
 * Individual workflows scale to thousands of tasks per cycle.
 * Deployments can scale infinitely.

Powerful UIs
 * A CLI that can operate on multiple workflows and tasks at once.
 * A web GUI with Jupyter extension.
 * A terminal UI with control capability.

Ease of Use
 * Cylc installs from Conda Forge or PyPI and works "out of the box" on a Linux
   host.
 * Small workflows are easy. Many features to help with large, complex
   workflows.
 * With no central scheduler or database server, Cylc has low admin overheads
   and a small security footprint.
 * Built in support for HPC resource managers like PBS and Slurm.

Multi-User Installations
 * Cylc leverages JupyterHub for authentication and to launch Cylc UI Servers.
 * Fine-grained authorization of control by other users.

Advanced Scheduling
 * Graph branching from optional task outputs.
 * Supports multiple flows in the same workflow, e.g. to re-run a past subgraph
   while the main flow continues.
 * Start a flow from anywhere in the depedency graph.
 * Cycle interleaving: this cut catch-up from delays from ~24 hours to ~30
   minutes at NIWA.

Resilience
 * Configurable task retries.
 * Triggering off of all task states.
 * Comprehensive event handling.
 * Manual job control: poll, kill, or retrigger jobs.
 * Restart from latest state even if the scheduler was killed, and follow up on
   orphaned jobs.
 * Cylc is cluster-aware: it can interact with jobs via multiple platform hosts.


PUBLICATIONS AND REPORTS

If you use Cylc please cite it in your publications and let us know if your site
can be listed on this website.

Workflow Automation for Cycling Systems: The Cylc Workflow Engine
H. Oliver et al
Computing in Science & Engineering Vol 21, Issue 4, July/Aug 2019. DOI:
10.1109/MCSE.2019.2906593
DOI: 10.1109/MCSE.2019.2906593 DOI DOI 10.1109/MCSE.2019.2906593
10.1109/MCSE.2019.2906593

Cylc: A Workflow Engine for Cycling Systems
Oliver et al., (2018).
Journal of Open Source Software, 3(27), 737. DOI: 10.21105/joss.00737



Citable DOI for Cylc code releases.


Old (2017): Cylc response to Assessment Report on Autosubmit, Cylc and ecFlow

C
Y
L
C

Home
Documentation
Sites
Support
Dev
News
Home
Documentation
Sites
Support
Dev
News