PySpark
Python API for Apache Spark — distributed dataframes and SQL for large-scale data processing.
data-analysis-frameworksRecently released
67
Hero Score
Popularity
88
Performance
45
Ecosystem
75
Maturity
77
Dev Experience
50
⭐ 43,607 stars⬇ 11.4M downloads/wkFirst release: May 2017Last release: Jul 2026
Async Support: NoPlugin Extensions: HighSpeed: FastDoc Quality: HighLearning Curve: Hard
Pros
- • Scales to clusters and very large datasets
- • Offers both SQL and DataFrame APIs
- • Mature ecosystem including MLlib and Structured Streaming
Cons
- • Heavy, JVM-dependent setup and runtime
- • Slower iteration cycle than local in-memory tools
- • Debugging distributed jobs is genuinely hard