Open-source · Community-driven

Build, test, and compare workflow systems on realistic data

WfCommons is an open-source ecosystem of workflow execution instances, synthetic workflow generators, and benchmark specifications — powering research on scheduling, performance, resilience, and AI-driven workflow automation across HPC and distributed platforms.

Get started Documentation Explore the ecosystem →
$ pip install wfcommons
PyPI version Downloads License: LGPL v3
318+ Curated workflow execution instances in WfFormat
31+ Scientific applications across workflow systems
98 Research articles enabled by WfCommons
6 Interoperable tools — from schema to simulation
Why WfCommons

Workflow research needs data that reflects reality

Heterogeneous nodes (CPU/GPU), data-intensive AI pipelines, and autonomous “agentic” control loops demand datasets and benchmarks that reflect how scientific workflows actually run.

HPC + data at scale

Study scheduling, throughput, and robustness for complex workflows with thousands of tasks and diverse I/O footprints — across clusters, clouds, and supercomputers.

AI-ready workflow data

Build training and evaluation corpora for anomaly detection, runtime prediction, and resource-aware optimization by working from a consistent, validated schema.

Agentic workflows & autonomy

Test LLM/agent planners that adapt DAGs on the fly — replanning, retries, provenance-aware decisions — using realistic workflow “digital twins” derived from production traces.

Ecosystem

Interoperable building blocks, from schema to simulation

Each component is useful on its own — and stronger together, forming a complete pipeline from real executions to reproducible experiments.

Architecture docs →
Describe

WfFormat — common schema

A JSON specification for representing workflow execution instances and synthetic workflows, so tools and simulators consume data consistently.

View schema →
Collect

WfInstances — open executions

A curated collection of real workflow runs shared in WfFormat, ready to be analyzed, simulated, or used to derive generators and benchmarks.

Browse instances →
Analyze

WfChef — automated recipes

Analyzes real instances to discover recurring dependency patterns and statistical distributions, producing “recipes” that capture workflow structure and behavior.

How recipes work →
Generate

WfGen — synthetic workflows

Generates realistic synthetic workflow instances from recipes — preserving structure, runtime, and I/O distributions at any scale.

Generate workflows →
Benchmark

WfBench — executable benchmarks

Produces benchmark specifications with configurable CPU, memory, and I/O pressure for repeatable workflow experiments and fair comparisons.

Build benchmarks →
Simulate

WfSim — simulation-ready

Use real and synthetic instances directly in simulation frameworks that support WfFormat, such as WRENCH-based simulators.

Simulation support →
Quick start

From zero to a realistic workflow in five lines

The WfCommons Python package bundles the whole toolchain — instance analysis, recipes, generators, and benchmarks — behind one clean API.

  • pip install wfcommons — Python 3.9+, cross-platform
  • Analyze real execution instances in WfFormat
  • Generate realistic synthetic workflows at any scale
  • Produce executable benchmarks and simulator inputs
Read the quickstart Source on GitHub
generate_workflow.py 
from wfcommons import BlastRecipe
from wfcommons.wfgen import WorkflowGenerator

# Create a recipe modeled on real BLAST executions
recipe = BlastRecipe.from_num_tasks(500)

# Generate a realistic synthetic workflow (~500 tasks)
generator = WorkflowGenerator(recipe)
workflow = generator.build_workflow()

# Save it in WfFormat — ready for analysis or simulation
workflow.write_json('blast-workflow-500.json')
Datasets

318+ open workflow execution instances

Production workflow executions collected from multiple workflow management systems, curated and shared in WfFormat.

About WfInstances →
WfInstances Browser

Search, visualize, and simulate all 318 instances online

No cloning or setup required — the browser indexes every instance from the 6 repositories below, so you can find the right workflow before you download it.

  • Filter & sortBy tasks, files, data size, and runtime
  • Visualize DAGsInspect task graphs and export them as SVG
  • SimulateRun schedulers on a platform you define
  • DownloadGrab selected instances in WfFormat
Open the WfInstances Browser →
The WfInstances Browser: a searchable table of workflow instances with task counts, file counts, data sizes, and runtimes

Instance repositories

Pegasus Execution Instances

135instances
7applications
View on GitHub →

Nextflow Execution Instances

116instances
17applications
View on GitHub →

Snakemake Execution Instances

34instances
3applications
View on GitHub →

Makeflow Execution Instances

30instances
2applications
View on GitHub →

Hello World Execution Instances

2instances
1applications
View on GitHub →

StreamFlow Execution Instances

1instances
1applications
View on GitHub →

Run them in simulation

WfFormat instances plug directly into simulation frameworks — evaluate schedulers and systems without burning compute hours.

WRENCH WRENCH-Pegasus WRENCH-WorkQueue
Impact

Research enabled by WfCommons

WfCommons has enabled research published in 98 articles — by our own team and by researchers across the workflows community.

Research outcomes per year

98 articles, stacked by publication type

Journal · 2027
— Journal of Systems Architecture
Journal · 2026
M. Kroczek, J. Zawalska, K. Rycerz — Future Generation Computer Systems
Conference · 2026
D. Li, Z. Cheng, G. K. Nicholls, Q. Kong — Forty-third International Conference on Machine Learning
Conference · 2026
F. Barbosa, E. Damasceno, F. Queiroz, J.B. Guimaraes, D. Cordeiro, J.S. Araujo, M. Amaris — Latin America High Performance Computing Conference (CARLA)
Journal · 2026
F. Li, W.J. Tan, M. Seok, W. Cai — Mathematics
Journal · 2026
S Mohammadi, L PourKarimi, P Thamm, S Khair Un Nisa, U Leser, K Reinert — SSRN
Poster · 2026
S. Kharma, T. Wies, F. Schintke — SCA/HPC Asia 2026
Conference · 2026
T. Noack, A. Koch — 40th ACM International Conference on Supercomputing (ICS)
Journal · 2026
C. Feld, A. Calotoiu, G. Corbin, M. Geimer, M.A Hermanns, M. Knespel, B. Mohr, J.A. Reuter, M. Sander, P. Saviankou, M. Schlütter, R. Schöne, S.S. Shende, A. Visser, B. Wesarg, W.R. Williams, F. Wolf, B.J.N. Wylie, M. Zarubin — Frontiers in High Performance Computing
Journal · 2026
F. Meng, J. Zeng, W. Zhang, C. Chen — Concurrency and Computation: Practice and Experience
Journal · 2026
Q. Wu, Q. Zhou, M. Zhou, J. Feng, T. Guo, C. Chen — IEEE Transactions on Automation Science and Engineering
Conference · 2026
D. Schweisgut, A. Benoit, Y. Robert, H. Meyerhenke — 32nd European Conference on Parallel and Distributed Processing (Euro-Par)
Preprint · 2026
Z. Huang, Y.-X. Hu, F. Wu, X.-Y. Li — arXiv
Preprint · 2026
H. Dang1, T. Nguyen — arXiv
Conference · 2026
P. González-García, M.R. Sierra, J. Puente, F.J. Gil-Gala — Bioinspired Intelligent Systems: From Robotics and Computer Vision to Trustworthy Applications: 11th International Work-Conference on the Interplay Between Natural and Artificial Computation
Thesis · 2026
A.T. Ramrakhiani — MS Thesis
Thesis · 2026
J. McDonald — PhD Thesis
Journal · 2026
F. Ding, S. Zhang, Y. Yuan — IEEE Transactions on Services Computing
Conference · 2026
M. Tang, L. Guo, A. Kougkas, X-H. Sun, N.R. Tallent — IEEE International Parallel and Distributed Processing Symposium (IPDPS)
Journal · 2026
W. Chongdarakul, K. Yongcharoenchaiyasit, N. Aunsri, S. Arwatchananukul, P. Naronglerdrit — Future Generation Computer Systems
Thesis · 2026
F. Lehmann — PhD Thesis
Conference · 2026
J. Wang, Y. Gao, A. Lakhan — Advanced Intelligent Computing Technology and Applications (ICIC 2026)
Journal · 2026
D. Hari —
Book Chapter · 2026
N. Raghu, A. Kumar D. B., G. S. Shetty, R. Bhat, P. Rajashree Nambiar —
Thesis · 2026
J. Kammerer — BS Thesis
Thesis · 2026
A.T. Ramrakhiani — MS Thesis
Conference · 2026
Y. Liang, H. Qu, J. Zhao — 11th International Conference on Computer and Communication Systems (ICCCS)
Conference · 2026
K. West, V. Bountris, P. Thamm, U. Leser, Y. Elkhatib, L. Thamsen — 19th IEEE International Conference on Cloud Computing (CLOUD)
Journal · 2026
F. Meng, J. Zeng, W. Zhang, C. Chen — Concurrency and Computation: Practice and Experience
Conference · 2026
N. Damink, A. Garcia-Robledo, M. Zangiabady — Euro-Par 2026: Parallel Processing
Conference · 2026
M. Vani — International Artificial Intelligence Technology Conference (AITC)
Conference · 2026
X. Liu, Y. Dong, X. Zhang, R. Ma — IEEE International Conference on Smart Internet of Things (SmartIoT)
Preprint · 2026
X. Heng — arXiv
Journal · 2026
Y. D. Semenova, O. V. Sukhoroslov — Program Systems: Theory and Applications
Workshop · 2026
M. Khodabandehlou, J. Coleman, B. Krishnamachari, K. Chan — 6th International Workshop on the Internet of Things for Adversarial Environments (IoTAE)
Journal · 2026
J. Zhang, X. Li, L. Chen, Z. Cai, R. Ruiz, R. Buyya — IEEE Transactions on Services Computing
Conference · 2026
Q. Zhou, R. Sun, M. Zhou, Q. Wu — 2026 IEEE 22nd International Conference on Automation Science and Engineering
Preprint · 2026
— arXiv
Thesis · 2026
— PhD Thesis
Preprint · 2026
— arXiv
Workshop · 2026
— 1st International Workshop on Edge-Cloud-HPC Operational Continuum (ECHO)
Workshop · 2026
— 21st Workshop on Workflows in Support of Large-Scale Science (WORKS)
Conference · 2025
Y. Semenov, O. Sukhoroslov — Supercomputing, RuSCDays 2024
Conference · 2025
F. Lehmann, J. Bader, F. Tschirpke, N. de Mecquenem, A. Loser, S. Becker, K. E. Lewinska, L. Thamsen, U. Leser — 25th IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid)
Workshop · 2025
M. Wilhelm, T. Pionteck — 34th Heterogeneity in Computing Workshop (HCW)
Journal · 2025
L. C. R. Alvarenga, Y. Frota, D. de Oliveira, R. Coutinho — Concurrency and Computation: Practice and Experience
Conference · 2025
S. Kulagina, A. Benoit, H. Meyerhenke — 25th International Symposium on Cluster, Cloud and Internet Computing (CCGrid)
Journal · 2025
S. Wang, H. Zhang, T. Wu, Y. Zhang, W. E. Zhang, Q. Z. Sheng — IEEE Transactions on Services Computing
Workshop · 2025
J. Coleman, R. V. Agrawal, E. Hirani, B. Krishnamachari — 28th Workshop on Job Scheduling Strategies for Parallel Processing (JSSPP)
Conference · 2025
J. Coleman, B. Krishnamachari — 39th IEEE International Parallel & Distributed Processing Symposium (IPDPS)
Journal · 2025
P. Barredo, J. Puente — Natural Computing
Journal · 2025
Q. Gao, L. Han, S. Hunold, Y. Robert, F. Vivien — IEEE Transactions on Parallel and Distributed Systems
Conference · 2025
D. Schweisgut, A. Benoit, Y. Robert, H. Meyerhenke — ICPP '25: Proceedings of the 54th International Conference on Parallel Processing
Conference · 2025
H. Lee, J. Firoz, N. R. Tallent, L. Guo, M. Halappanavar — 2025 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
Workshop · 2025
M. Wang, S. Jahangiri, K. Maheshwari, C. Li — 1st Workshop on Workflows, Intelligent Scientific Data, and Optimization for Automated Management (WISDOM)
Workshop · 2025
J. McDonald, Y.C. Wong, K. Mehta, F. Suter, R. Ferreira da Silva, L. Pottier, E. Deelman, H. Casanova — 16th International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS)
Workshop · 2025
J. Chamorro, G. Twigg-Ho, J. Coleman, T. Coleman, B. Krishnamachari, M. Khodabandehlou — 20st Workshop on Workflows in Support of Large-Scale Science (WORKS)
Journal · 2025
R-G. Stan, F. Pop — IEEE Access
Conference · 2025
F. Jian, G. Shen, X. Lv, C. Guo, Y. Cui — 31th IEEE International Conference on Parallel and Distributed Systems (ICPADS)
Journal · 2025
— Integrated Computer-Aided Engineering, 32(4)
Journal · 2024
M. Fan, L. Ye, X. Zuo, X. Zhao — Expert Systems with Applications
Journal · 2024
K. Karmakar, A. Tarafdar, R. K. Das, S. Khatua — Journal of Grid Computing
Conference · 2024
P. Barredo, J. Puente — International Work-Conference on the Interplay Between Natural and Artificial Computation
Journal · 2024
J. McDonald, J. Dobbs, Y.C. Wong, R. Ferreira da Silva, H. Casanova — Future Generation Computer Systems
Conference · 2024
S Kulagina, H Meyerhenke, A Benoit — 53rd International Conference on Parallel Processing (ICPP '24)
Thesis · 2024
J. R. Coleman — PhD Thesis
Journal · 2024
Y. Su, V. Anand, J. Yu, J. Tan, A. Wierman — ACM Transactions on Modeling and Performance Evaluation of Computing Systems
Journal · 2024
B. Lin, C. Lin, X. Chen, M. Lin, G. Huang, Z. Xu — IEEE Transactions on Automation Science and Engineering
Journal · 2024
M. Fan, X. Zhao, X. Zuo, L. Ye — IEEE Transactions on Automation Science and Engineering
Conference · 2024
J. Shin, D. Arroyo, A. Tantawi, C. Wang, A. Youssef, R. Nagi — ACM Symposium on Cloud Computing (SoCC'24)
Thesis · 2024
L. F. D. Versluis —
Workshop · 2024
A. A. Da Silva, R. P. Hong Enriquez, G. Rattihalli, V. Thurimella, R. Ferreira da Silva, D. Milojicic —
Journal · 2024
E. Saeedizade, M. Ashtiani —
Journal · 2024
B. Qin, Q. Lei, X. Wang — The Journal of Supercomputing
Conference · 2023
K. Alam, B. Roy, A. Serebrenik — 30th Asia-Pacific Software Engineering Conference (APSEC)
Journal · 2023
L. Yang, L. Ye, Y. Xia, Y. Zhan — Future Generation Computer Systems
Journal · 2023
T. Coleman, H. Casanova, R. Ferreira da Silva — Future Generation Computer Systems
Thesis · 2023
T. Coleman — PhD Thesis
Journal · 2023
J. Zhang, X. Li, L. Chen, R. Ruiz — IEEE Transactions on Services Computing
Journal · 2023
Q. Zhang, Q. Wu, M. Zhou, J. Wen, S. Yao — IEEE Transactions on Automation Science and Engineering
Conference · 2023
O. Sukhoroslov — International Conference on Parallel Computing Technologies
Journal · 2023
P. Barredo, J. Puente — Natural Computing
Conference · 2023
H. Casanova, K. Berney, S. Chastel, R. Ferreira da Silva — 47th Annual Computers, Software, and Applications Conference (COMPSAC)
Conference · 2023
O. Sukhoroslov M. Gorokhovskii — Russian Supercomputing Days
Journal · 2022
A. Benoit, L. Perotin, Y. Robert, H. Sun — ACM Transactions on Parallel Computing
Conference · 2022
Z. Li, Y. Liu, L. Guo, Q. Chen, J. Cheng, W. Zheng, M. Guo — 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems
Journal · 2022
L. Xing, M. Zhang, H. Li, M. Gong, J. Yang, K. Wang — Computers & Industrial Engineering
Workshop · 2022
H. Casanova, Y. C. Wong, L. Pottier, R. Ferreira da Silva — Workshop on Job Scheduling Strategies for Parallel Processing (JSSPP)
Journal · 2022
Z. Zhang, Q-S. Hua, X. Zhang, H. Jin, X. Liao — Wireless Communications and Mobile Computing
Journal · 2022
T. Coleman, H. Casanova, L. Pottier, M. Kaushik, E. Deelman, R. Ferreira da Silva — Future Generation Computer Systems
Workshop · 2022
M. Kiamari, B. Krishnamachari — GNNet '22: Proceedings of the 1st International Workshop on Graph Neural Networking
Conference · 2022
P. Barredo, J. Puente — International Work-Conference on the Interplay Between Natural and Artificial Computation
Thesis · 2021
W. Koch — PhD Thesis
Conference · 2021
T. Coleman, H. Casanova, T. Gwartney, R. Ferreira da Silva — International Conference on Computational Science (ICCS)
Journal · 2021
E. Saeedizade, M. Ashtiani — The Journal of Supercomputing
Journal · 2021
M. Orr, O. Sinnen — Parallel Computing
Journal · 2021
S. Tuli, G. Casale, N. R. Jennings — IEEE Transactions on Parallel and Distributed Systems
Conference · 2021
T. Coleman, H. Casanova, R. Ferreira da Silva — 2021 IEEE 17th International Conference on eScience (eScience)
Cite

Cite WfCommons

Use these references in papers, reports, and benchmark artifacts.

Primary paper

Future Generation Computer Systems (FGCS), 2022.

@article{WfCommons,
  title   = {WfCommons: A Framework for Enabling Scientific Workflow
             Research and Development},
  author  = {Coleman, Taina and Casanova, Henri and Pottier, Loic and
             Kaushik, Manav and Deelman, Ewa and Ferreira da Silva, Rafael},
  journal = {Future Generation Computer Systems},
  volume  = {128},
  pages   = {16--27},
  year    = {2022},
  doi     = {10.1016/j.future.2021.09.043}
}
DOI

Core publications

The papers behind each component of the ecosystem.

H. Casanova, K. Berney, S. Chastel, R. Ferreira da Silva — 1st IEEE International Workshop on Workflows in Distributed Environments (WiDE 2023), 2023
T. Coleman, H. Casanova, K. Maheshwari, L. Pottier, S. R. Wilkinson, J. Wozniak, F. Suter, M. Shankar, R. Ferreira da Silva — 2022 IEEE/ACM International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS), 2022
T. Coleman, H. Casanova, L. Pottier, M. Kaushik, E. Deelman, and R. Ferreira da Silva — Future Generation Computer Systems, vol. 128, 2022
T. Coleman, H. Casanova, L. Pottier, M. Kaushik, E. Deelman, and R. Ferreira da Silva — 17th IEEE Conference on eScience, 2021