Workflow research needs data that reflects reality
Heterogeneous nodes (CPU/GPU), data-intensive AI pipelines, and autonomous “agentic” control loops demand datasets and benchmarks that reflect how scientific workflows actually run.
HPC + data at scale
Study scheduling, throughput, and robustness for complex workflows with thousands of tasks and diverse I/O footprints — across clusters, clouds, and supercomputers.
AI-ready workflow data
Build training and evaluation corpora for anomaly detection, runtime prediction, and resource-aware optimization by working from a consistent, validated schema.
Agentic workflows & autonomy
Test LLM/agent planners that adapt DAGs on the fly — replanning, retries, provenance-aware decisions — using realistic workflow “digital twins” derived from production traces.
Interoperable building blocks, from schema to simulation
Each component is useful on its own — and stronger together, forming a complete pipeline from real executions to reproducible experiments.
WfFormat — common schema
A JSON specification for representing workflow execution instances and synthetic workflows, so tools and simulators consume data consistently.
View schema →WfInstances — open executions
A curated collection of real workflow runs shared in WfFormat, ready to be analyzed, simulated, or used to derive generators and benchmarks.
Browse instances →WfChef — automated recipes
Analyzes real instances to discover recurring dependency patterns and statistical distributions, producing “recipes” that capture workflow structure and behavior.
How recipes work →WfGen — synthetic workflows
Generates realistic synthetic workflow instances from recipes — preserving structure, runtime, and I/O distributions at any scale.
Generate workflows →WfBench — executable benchmarks
Produces benchmark specifications with configurable CPU, memory, and I/O pressure for repeatable workflow experiments and fair comparisons.
Build benchmarks →WfSim — simulation-ready
Use real and synthetic instances directly in simulation frameworks that support WfFormat, such as WRENCH-based simulators.
Simulation support →From zero to a realistic workflow in five lines
The WfCommons Python package bundles the whole toolchain — instance analysis, recipes, generators, and benchmarks — behind one clean API.
pip install wfcommons— Python 3.9+, cross-platform- Analyze real execution instances in WfFormat
- Generate realistic synthetic workflows at any scale
- Produce executable benchmarks and simulator inputs
from wfcommons import BlastRecipe
from wfcommons.wfgen import WorkflowGenerator
# Create a recipe modeled on real BLAST executions
recipe = BlastRecipe.from_num_tasks(500)
# Generate a realistic synthetic workflow (~500 tasks)
generator = WorkflowGenerator(recipe)
workflow = generator.build_workflow()
# Save it in WfFormat — ready for analysis or simulation
workflow.write_json('blast-workflow-500.json')268+ open workflow execution instances
Production workflow executions collected from multiple workflow management systems, curated and shared in WfFormat — browse, filter, visualize, and download them online.
Run them in simulation
WfFormat instances plug directly into simulation frameworks — evaluate schedulers and systems without burning compute hours.
Research enabled by WfCommons
WfCommons has enabled research published in 76 articles — by our own team and by researchers across the workflows community.
Research outcomes per year
76 articles, stacked by publication type
Cite WfCommons
Use these references in papers, reports, and benchmark artifacts.
Primary paper
Future Generation Computer Systems (FGCS), 2022.
@article{WfCommons,
title = {WfCommons: A Framework for Enabling Scientific Workflow
Research and Development},
author = {Coleman, Taina and Casanova, Henri and Pottier, Loic and
Kaushik, Manav and Deelman, Ewa and Ferreira da Silva, Rafael},
journal = {Future Generation Computer Systems},
volume = {128},
pages = {16--27},
year = {2022},
doi = {10.1016/j.future.2021.09.043}
}Core publications
The papers behind each component of the ecosystem.