Open-source · Community-driven

Build, test, and compare workflow systems on realistic data

WfCommons is an open-source ecosystem of workflow execution instances, synthetic workflow generators, and benchmark specifications — powering research on scheduling, performance, resilience, and AI-driven workflow automation across HPC and distributed platforms.

Get started Documentation Explore the ecosystem →
$ pip install wfcommons
PyPI version Downloads License: LGPL v3
298+ Curated workflow execution instances in WfFormat
31+ Scientific applications across workflow systems
83 Research articles enabled by WfCommons
6 Interoperable tools — from schema to simulation
Why WfCommons

Workflow research needs data that reflects reality

Heterogeneous nodes (CPU/GPU), data-intensive AI pipelines, and autonomous “agentic” control loops demand datasets and benchmarks that reflect how scientific workflows actually run.

HPC + data at scale

Study scheduling, throughput, and robustness for complex workflows with thousands of tasks and diverse I/O footprints — across clusters, clouds, and supercomputers.

AI-ready workflow data

Build training and evaluation corpora for anomaly detection, runtime prediction, and resource-aware optimization by working from a consistent, validated schema.

Agentic workflows & autonomy

Test LLM/agent planners that adapt DAGs on the fly — replanning, retries, provenance-aware decisions — using realistic workflow “digital twins” derived from production traces.

Ecosystem

Interoperable building blocks, from schema to simulation

Each component is useful on its own — and stronger together, forming a complete pipeline from real executions to reproducible experiments.

Architecture docs →
Describe

WfFormat — common schema

A JSON specification for representing workflow execution instances and synthetic workflows, so tools and simulators consume data consistently.

View schema →
Collect

WfInstances — open executions

A curated collection of real workflow runs shared in WfFormat, ready to be analyzed, simulated, or used to derive generators and benchmarks.

Browse instances →
Analyze

WfChef — automated recipes

Analyzes real instances to discover recurring dependency patterns and statistical distributions, producing “recipes” that capture workflow structure and behavior.

How recipes work →
Generate

WfGen — synthetic workflows

Generates realistic synthetic workflow instances from recipes — preserving structure, runtime, and I/O distributions at any scale.

Generate workflows →
Benchmark

WfBench — executable benchmarks

Produces benchmark specifications with configurable CPU, memory, and I/O pressure for repeatable workflow experiments and fair comparisons.

Build benchmarks →
Simulate

WfSim — simulation-ready

Use real and synthetic instances directly in simulation frameworks that support WfFormat, such as WRENCH-based simulators.

Simulation support →
Quick start

From zero to a realistic workflow in five lines

The WfCommons Python package bundles the whole toolchain — instance analysis, recipes, generators, and benchmarks — behind one clean API.

  • pip install wfcommons — Python 3.9+, cross-platform
  • Analyze real execution instances in WfFormat
  • Generate realistic synthetic workflows at any scale
  • Produce executable benchmarks and simulator inputs
Read the quickstart Source on GitHub
generate_workflow.py 
from wfcommons import BlastRecipe
from wfcommons.wfgen import WorkflowGenerator

# Create a recipe modeled on real BLAST executions
recipe = BlastRecipe.from_num_tasks(500)

# Generate a realistic synthetic workflow (~500 tasks)
generator = WorkflowGenerator(recipe)
workflow = generator.build_workflow()

# Save it in WfFormat — ready for analysis or simulation
workflow.write_json('blast-workflow-500.json')
Datasets

298+ open workflow execution instances

Production workflow executions collected from multiple workflow management systems, curated and shared in WfFormat — browse, filter, visualize, and download them online.

Open the WfInstances browser

Pegasus Execution Instances

135instances
7applications
View collection →

Nextflow Execution Instances

96instances
17applications
View collection →

Snakemake Execution Instances

34instances
3applications
View collection →

Makeflow Execution Instances

30instances
2applications
View collection →

Hello World Execution Instances

2instances
1applications
View collection →

StreamFlow Execution Instances

1instances
1applications
View collection →

Run them in simulation

WfFormat instances plug directly into simulation frameworks — evaluate schedulers and systems without burning compute hours.

WRENCH WRENCH-Pegasus WRENCH-WorkQueue
Impact

Research enabled by WfCommons

WfCommons has enabled research published in 83 articles — by our own team and by researchers across the workflows community.

Research outcomes per year

83 articles, stacked by publication type

Conference · 2026
D. Li, Z. Cheng, G. K. Nicholls, Q. Kong — Forty-third International Conference on Machine Learning
Conference · 2026
F. Barbosa, E. Damasceno, F. Queiroz, J.B. Guimaraes, D. Cordeiro, J.S. Araujo, M. Amaris — CARLA 2026, Latin America High Performance Computing Conference
Journal · 2026
F. Li, W.J. Tan, M. Seok, W. Cai — Mathematics
Journal · 2026
S Mohammadi, L PourKarimi, P Thamm, S Khair Un Nisa, U Leser, K Reinert — SSRN
Poster · 2026
S. Kharma, T. Wies, F. Schintke — SCA/HPC Asia 2026
Conference · 2026
T. Noack, A. Koch — 40th ACM International Conference on Supercomputing, ICS ’26
Journal · 2026
C. Feld, A. Calotoiu, G. Corbin, M. Geimer, M.A Hermanns, M. Knespel, B. Mohr, J.A. Reuter, M. Sander, P. Saviankou, M. Schlütter, R. Schöne, S.S. Shende, A. Visser, B. Wesarg, W.R. Williams, F. Wolf, B.J.N. Wylie, M. Zarubin — Frontiers in High Performance Computing
Journal · 2026
F. Meng, J. Zeng, W. Zhang, C. Chen — Concurrency and Computation: Practice and Experience
Journal · 2026
Q. Wu, Q. Zhou, M. Zhou, J. Feng, T. Guo, C. Chen — IEEE Transactions on Automation Science and Engineering
Report · 2026
D. Schweisgut, A. Benoit, Y. Robert, H. Meyerhenke — arXiv
Report · 2026
Z. Huang, Y.-X. Hu, F. Wu, X.-Y. Li — arXiv
Report · 2026
H. Dang1, T. Nguyen — arXiv
Conference · 2026
P. González-García, M.R. Sierra, J. Puente, F.J. Gil-Gala — Bioinspired Intelligent Systems: From Robotics and Computer Vision to Trustworthy Applications: 11th International Work-Conference on the Interplay Between Natural and Artificial Computation
PhD Thesis · 2026
A.T. Ramrakhiani — MS Thesis
PhD Thesis · 2026
J. McDonald — PhD Thesis
Journal · 2026
F. Ding, S. Zhang, Y. Yuan — IEEE Transactions on Services Computing
Conference · 2026
M. Tang, L. Guo, A. Kougkas, X-H. Sun, N.R. Tallent — 2026 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
Journal · 2026
W. Chongdarakul, K. Yongcharoenchaiyasit, N. Aunsri, S. Arwatchananukul, P. Naronglerdrit — Future Generation Computer Systems
PhD Thesis · 2026
F. Lehmann — PhD Thesis
Conference · 2026
J. Wang, Y. Gao, A. Lakhan — Advanced Intelligent Computing Technology and Applications (ICIC 2026)
Journal · 2026
D. Hari —
Report · 2026
N. Raghu, A. Kumar D. B., G. S. Shetty, R. Bhat, P. Rajashree Nambiar —
PhD Thesis · 2026
J. Kammerer — BS Thesis
PhD Thesis · 2026
A.T. Ramrakhiani — MS Thesis
Conference · 2026
Y. Liang, H. Qu, J. Zhao — 2026 11th International Conference on Computer and Communication Systems (ICCCS)
Report · 2026
K. West, V. Bountris, P. Thamm, U. Leser, Y. Elkhatib, L. Thamsen — arXiv
Journal · 2026
F. Meng, J. Zeng, W. Zhang, C. Chen — Concurrency and Computation: Practice and Experience
Conference · 2025
Y. Semenov, O. Sukhoroslov — Supercomputing, RuSCDays 2024
Conference · 2025
F. Lehmann, J. Bader, F. Tschirpke, N. de Mecquenem, A. Loser, S. Becker, K. E. Lewinska, L. Thamsen, U. Leser — 25th IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid)
Workshop · 2025
M. Wilhelm, T. Pionteck — 34th Heterogeneity in Computing Workshop (HCW 2025)
Journal · 2025
L. C. R. Alvarenga, Y. Frota, D. de Oliveira, R. Coutinho — Concurrency and Computation: Practice and Experience
Conference · 2025
S. Kulagina, A. Benoit, H. Meyerhenke — 25th International Symposium on Cluster, Cloud and Internet Computing (CCGrid)
Journal · 2025
S. Wang, H. Zhang, T. Wu, Y. Zhang, W. E. Zhang, Q. Z. Sheng — IEEE Transactions on Services Computing
Workshop · 2025
J. Coleman, R. V. Agrawal, E. Hirani, B. Krishnamachari — 28th Workshop on Job Scheduling Strategies for Parallel Processing (JSSPP 2025)
Conference · 2025
J. Coleman, B. Krishnamachari — 39th IEEE International Parallel & Distributed Processing Symposium (IPDPS 2025)
Report · 2025
M. Kroczek, J. Zawalska, K. Rycerz —
Journal · 2025
P. Barredo, J. Puente — Natural Computing
Report · 2025
Q. Gao, L. Han, S. Hunold, Y. Robert, F. Vivien —
Conference · 2025
D. Schweisgut, A. Benoit, Y. Robert, H. Meyerhenke — ICPP '25: Proceedings of the 54th International Conference on Parallel Processing
Conference · 2025
H. Lee, J. Firoz, N. R. Tallent, L. Guo, M. Halappanavar — 2025 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
Workshop · 2025
M. Wang, S. Jahangiri, K. Maheshwari, C. Li — 54th International Conference on Parallel Processing Companion (ICPP Companion '25)
Workshop · 2025
J. McDonald, Y.C. Wong, K. Mehta, F. Suter, R. Ferreira da Silva, L. Pottier, E. Deelman, H. Casanova — Proceedings of the SC'25 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis
Workshop · 2025
J. Chamorro, G. Twigg-Ho, J. Coleman, T. Coleman, B. Krishnamachari, M. Khodabandehlou — Proceedings of the SC'25 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis
Journal · 2025
R-G. Stan, F. Pop — IEEE Access
Conference · 2025
F. Jian, G. Shen, X. Lv, C. Guo, Y. Cui — 2025 IEEE 31th International Conference on Parallel and Distributed Systems (ICPADS)
Journal · 2024
M. Fan, L. Ye, X. Zuo, X. Zhao — Expert Systems with Applications
Journal · 2024
K. Karmakar, A. Tarafdar, R. K. Das, S. Khatua — Journal of Grid Computing
Conference · 2024
P. Barredo, J. Puente — International Work-Conference on the Interplay Between Natural and Artificial Computation
Journal · 2024
J. McDonald, J. Dobbs, Y.C. Wong, R. Ferreira da Silva, H. Casanova — Future Generation Computer Systems
Conference · 2024
S Kulagina, H Meyerhenke, A Benoit — 53rd International Conference on Parallel Processing (ICPP '24)
PhD Thesis · 2024
J. R. Coleman — PhD Thesis
Journal · 2024
Y. Su, V. Anand, J. Yu, J. Tan, A. Wierman — ACM Transactions on Modeling and Performance Evaluation of Computing Systems
Journal · 2024
B. Lin, C. Lin, X. Chen, M. Lin, G. Huang, Z. Xu — IEEE Transactions on Automation Science and Engineering
Journal · 2024
M. Fan, X. Zhao, X. Zuo, L. Ye — IEEE Transactions on Automation Science and Engineering
Conference · 2024
J. Shin, D. Arroyo, A. Tantawi, C. Wang, A. Youssef, R. Nagi — ACM Symposium on Cloud Computing (SoCC'24)
PhD Thesis · 2024
L. F. D. Versluis —
Workshop · 2024
A. A. Da Silva, R. P. Hong Enriquez, G. Rattihalli, V. Thurimella, R. Ferreira da Silva, D. Milojicic —
Journal · 2024
E. Saeedizade, M. Ashtiani —
Journal · 2024
B. Qin, Q. Lei, X. Wang — The Journal of Supercomputing
Report · 2023
K. Alam, B. Roy, A. Serebrenik — arXiv preprint
Journal · 2023
L. Yang, L. Ye, Y. Xia, Y. Zhan — Future Generation Computer Systems
Journal · 2023
T. Coleman, H. Casanova, R. Ferreira da Silva — Future Generation Computer Systems
PhD Thesis · 2023
T. Coleman — PhD Thesis
Journal · 2023
J. Zhang, X. Li, L. Chen, R. Ruiz — IEEE Transactions on Services Computing
Journal · 2023
Q. Zhang, Q. Wu, M. Zhou, J. Wen, S. Yao — IEEE Transactions on Automation Science and Engineering
Conference · 2023
O. Sukhoroslov — International Conference on Parallel Computing Technologies
Journal · 2023
P. Barredo, J. Puente — Natural Computing
Workshop · 2023
H. Casanova, K. Berney, S. Chastel, R. Ferreira da Silva — 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC)
Conference · 2023
O. Sukhoroslov M. Gorokhovskii — Russian Supercomputing Days
Journal · 2022
A. Benoit, L. Perotin, Y. Robert, H. Sun — ACM Transactions on Parallel Computing
Conference · 2022
Z. Li, Y. Liu, L. Guo, Q. Chen, J. Cheng, W. Zheng, M. Guo — 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems
Journal · 2022
L. Xing, M. Zhang, H. Li, M. Gong, J. Yang, K. Wang — Computers & Industrial Engineering
Workshop · 2022
H. Casanova, Y. C. Wong, L. Pottier, R. Ferreira da Silva — Workshop on Job Scheduling Strategies for Parallel Processing (JSSPP)
Journal · 2022
Z. Zhang, Q-S. Hua, X. Zhang, H. Jin, X. Liao — Wireless Communications and Mobile Computing
Journal · 2022
T. Coleman, H. Casanova, L. Pottier, M. Kaushik, E. Deelman, R. Ferreira da Silva — Future Generation Computer Systems
Workshop · 2022
M. Kiamari, B. Krishnamachari — GNNet '22: Proceedings of the 1st International Workshop on Graph Neural Networking
Conference · 2022
P. Barredo, J. Puente — International Work-Conference on the Interplay Between Natural and Artificial Computation
PhD Thesis · 2021
W. Koch — PhD Thesis
Conference · 2021
T. Coleman, H. Casanova, T. Gwartney, R. Ferreira da Silva — International Conference on Computational Science (ICCS)
Journal · 2021
E. Saeedizade, M. Ashtiani — The Journal of Supercomputing
Journal · 2021
M. Orr, O. Sinnen — Parallel Computing
Journal · 2021
S. Tuli, G. Casale, N. R. Jennings — IEEE Transactions on Parallel and Distributed Systems
Conference · 2021
T. Coleman, H. Casanova, R. Ferreira da Silva — 2021 IEEE 17th International Conference on eScience (eScience)
Cite

Cite WfCommons

Use these references in papers, reports, and benchmark artifacts.

Primary paper

Future Generation Computer Systems (FGCS), 2022.

@article{WfCommons,
  title   = {WfCommons: A Framework for Enabling Scientific Workflow
             Research and Development},
  author  = {Coleman, Taina and Casanova, Henri and Pottier, Loic and
             Kaushik, Manav and Deelman, Ewa and Ferreira da Silva, Rafael},
  journal = {Future Generation Computer Systems},
  volume  = {128},
  pages   = {16--27},
  year    = {2022},
  doi     = {10.1016/j.future.2021.09.043}
}
DOI

Core publications

The papers behind each component of the ecosystem.

H. Casanova, K. Berney, S. Chastel, R. Ferreira da Silva — 1st IEEE International Workshop on Workflows in Distributed Environments (WiDE 2023), 2023
T. Coleman, H. Casanova, K. Maheshwari, L. Pottier, S. R. Wilkinson, J. Wozniak, F. Suter, M. Shankar, R. Ferreira da Silva — 2022 IEEE/ACM International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS), 2022
T. Coleman, H. Casanova, L. Pottier, M. Kaushik, E. Deelman, and R. Ferreira da Silva — Future Generation Computer Systems, vol. 128, 2022
T. Coleman, H. Casanova, L. Pottier, M. Kaushik, E. Deelman, and R. Ferreira da Silva — 17th IEEE Conference on eScience, 2021