Can I Use Spark for Small Data Pipelines? The Honest Trade-offs
Is Spark worth it for small data? The overhead of distributed computing, Polars and DuckDB as lighter options, and when Spark still makes sense.
Older articles
Page 84 of 137
Modern Data Pipeline Architecture Best Practices: A Deep Dive into Scalable ETL
Data pipeline architecture best practices: modular stages, idempotency, schema evolution, retries and monitoring for reliable ETL at scale.
Data & Analytics
10 Expert Tips for ETL Automation with Python and GitHub Actions
Ten tips for ETL with Python and GitHub Actions: scheduled workflows, secrets, caching, idempotent jobs, failure alerts and knowing when to move on.
Prefect vs Airflow Comparison 2026: Which Orchestrator Should You Choose?
Prefect vs Airflow: Pythonic dynamic workflows against the mature scheduling standard. Features, operations, total cost and which to use.
Data Engineering Consulting for ETL Migration: A Strategic Guide to Modernizing Your Pipelines
Plan an ETL migration without breaking dashboards: assessment, a migration framework, a practical checklist and tools for moving pipelines.
Databases
SurrealDB Review and Tutorial: Is This Multi-Model Database the Future of Backend?
SurrealDB reviewed with a quick tutorial: a multi-model database with documents, graphs and SurrealQL. Strengths, maturity concerns and use cases.