Google Cloud products quickstarts:

Public Datasets

The notebooks read datasets from our public GCS bucket containing several publicly available datasets.

In this doc you can see the list of available datasets, which are located in gs://dataproc-metastore-public-binaries.
The documentation above has details about the datasets, and links to their original pages, containing their LICENSES, etc.

Antigravity Data Cloud extension

These notebooks (Spark and BigQuery) are available from your Antigravity when using the Google Cloud Data Cloud Extension:

Cloud Code VSCode extension

These notebooks are available from your VSCode IDE when using the Cloud Code extension. You can go to Notebook Templates and download each template to your environment:

Usage in Vertex AI Workbench notebooks

These notebooks are available from within the Vertex AI Workbench notebooks environment.
Navigate to JupyterLab home screen and click on Notebooks to see the list of notebooks and a button for you to download/copy them into your environment.

Usage in your local environment

1. Install gcloud cli
2. Run gcloud init to setup your default GCP configuration
3. Clone this repository by running
   git clone https://github.com/GoogleCloudPlatform/ai-ml-recipes.git
4. Install requirements by running pip install -r requirements.txt
5. For the PySpark notebooks, use one of the approaches using the Dataproc Jupyter Plugin:
   • 5.1) [PySpark Serverless Runtime on Google Cloud] Create a Runtime Template with your desired runtime config, and use it to run your PySpark notebooks.
     • Do not forget to ensure the correct network configuration
   • 5.2) [Local runtime] Use your local PySpark runtime
6. For the Bigframes notebooks, you do not need PySpark, just any kernel/environment, and the processing will leverage BigQuery in your GCP project

BigQuery Jupyter Plugin

We recommend leveraging the BigQuery Jupyter Plugin, which will be available in your local environment just by installing the dependency running pip install -r requirements.txt. This will enable you to:

• Connect your Jupyterlab notebooks from anywhere to Dataproc
• Develop in Python, SQL, Java/Scala, and R
• Manage Dataproc clusters and jobs
• Run notebooks in your favorite IDE that supports Jupyter using Dataproc as kernel
• Deploy a notebook as a recurring job
• View cloud and spark logs inside Jupyterlab
• View your BigQuery datasets schema inside Jupyterlab
• Manage your files on Google Cloud Storage (GCS)

Contributing

See the contributing instructions to get started contributing.

License

All solutions within this repository are provided under the Apache 2.0 license. Please see the LICENSE file for more detailed terms and c