Kedro
Kedro is an open-source Python framework that applies software engineering best practices to the development of data and machine learning pipelines.
Our verdict
Who it's for
An open-source framework for building production-ready data engineering and data science pipelines.
Who should look elsewhere
Requires familiarity with Python and software engineering concepts.
Pros
- +Promotes clean, reproducible, and maintainable code for data pipelines.
- +Standardizes project structure and encourages team collaboration.
- +Reduces time spent on 'plumbing' and allows focus on core problems.
- +Facilitates seamless transition from development to production.
- +Offers extensive integrations with popular data and ML tools.
Cons
- −Requires familiarity with Python and software engineering concepts.
- −May have a learning curve for users new to structured pipeline development.
Key features
- ✓Pipeline Visualisation (Kedro-Viz) with data lineage and execution details
- ✓Data Catalog with lightweight connectors for various file formats and systems (S3, GCP, Azure, Pandas, Spark, etc.)
- ✓Project Template for standardizing configuration, code, tests, and notebooks
- ✓Dedicated IDE support for Visual Studio Code (code navigation, autocompletion)
- ✓Pipeline Abstraction with dataset-driven workflow and automatic dependency resolution
- ✓Coding Standards (pytest, Sphinx, ruff, Python logging)
- ✓Flexible Deployment (single/distributed machine, Argo, Prefect, Kubeflow, AWS Sagemaker, Databricks)
Pricing
| Plan | Price | Billing | Notes |
|---|---|---|---|
| Open Source | — | month | Clean Code; Handles Complexity; Standardisation |
Integrations
SlackGitHub