Loading repository data…
LucaCanali / repository
Miscellaneous
Includes notes on using Apache Spark, with drill down on Spark for Physics, how to run TPCDS on PySpark, how to create histograms with Spark. Also tools for stress testing, measuring CPUs' performance, and I/O latency heat maps. Jupyter notebooks examples for using various DB systems.
#apache-spark#database#jupyter-notebooks#performance-analysis#performance-monitoring#performance-testing
Live repository statistics
★ 464Stars
⑂ 152Forks
◯ 2Open issues
◉ 464Watchers
README preview
Miscellaneous Projects, Tools, and Scripts
Contact: Luca.Canali@cern.ch
Performance Engineering and Apache Spark
| Folder | Description |
|---|---|
| Spark Dashboard | A tool for Apache monitoring, use to build a performance dashboard and troubleshoot Spark jobs. |
| Spark Notes | Miscellaneous tips and code snippets about Apache Spark. |
| Spark for Physics | Examples, with code and data of using Apache Spark for High Energy Physics data analysis. |
| Performance Testing | Includes: - TPCDS-PySpark, run TPCDS bemchmark at scale with PySpark and collect execution metrics- Load testing tools for CPU benchmarking, in Python and Rust- Notes on how to use various tools for performance investigations |
Data Engineering and Data Science
| Folder | Description |
|---|---|
| Kepler Analysis | A curated collection of interactive notebooks for executing Kepler's orbital analysis on Mars. |
| Deep Learning Notes | Notes and examples on Deep Learning tools and related data pipelines. |
| Pyspark_SQL_Magic_Jupyter | How to write Jupyter SQL magic functions for PySpark and Spark SQL. |
| Trino and Presto on Jupyter | Example of using Trino or Presto on a Jupyter notebook. |
| PostgreSQL and YugabyteDB on Jupyter | Example of using PostgreSQL or YugabyteDB on a Jupyter notebook. |
| Oracle_Jupyter | Examples of how to query Oracle using Jupyter/IPython notebooks. |
| Impala_SQL_Jupyter | Examples of how to run SQL on Apache Impala using Jupyter/IPython notebooks. |
| SQL_color_Mandelbrot | How to use SQL to compute and display the Mandelbrot set with colors. Examples for Oracle and PostgreSQL. |
| PLSQL_Neural_Network | An example of neural network inference using Oracle RDBMS and PL/SQL. |