From the root directory of this repository, execute:

docker-compose up

This might take a few moments the first time, as it needs to download several docker images and initialize them. For reference, a cold start on my laptop over a wired connection it takes between 30 seconds and 1 minute. Subsequent starts should be way faster. The downloaded images will use about 1Gb on your disk.

If you notice any permissions error on the logs (typically Grafana complaining a folder is not writable), this is probably due to your system running docker as root. You can fix this by setting the following env variable:

export DOCKER_COMPOSE_USER_ID=$(id -u)

or like this

sudo DOCKER_COMPOSE_USER_ID=$(id -u) docker-compose up

After a few moments, you should see the logs stabilize and stop scrolling fast. After a few seconds you should see several logs from Kafka Connect registering the connectors and showing the configuration on the logs. At that point the project is fully available.There will always be some activity, as the stack collects and store some metrics, but those appear only every few seconds.

graph TD
   subgraph "Data Analytics"
    Q[QuestDB]
  end

  subgraph "Data Science and ML"
    Q -->|SELECT FROM trades and github_events| DSN[Data Science Notebook]
    Q -->|SELECT FROM trades and github_events| FM[Forecast Model]
  end


  subgraph "Data Ingestion"
    GE[Trading Events, GitHub Events, Smart Meters, Transactions] -->|Python, NodeJS, Java, Go, Rust into topics | AK[Apache Kafka]
    AK -->|github_events topic| KC
    IO[IoT Events] -->|ILP Protocol into iot_data table| Q
    IE[Trading Events] -->|ILP Protocol into trades table| Q
    PS[Plant Sensors] -->|ILP Protocol into plant_sensors table| Q
    AK -->|trades topic| KC
    AK -->|trades topic| KSR[Kafka Schema Registry]
    AK -->|smart-meters topic| KC
    AK -->|smart-meters topic| KSR
    AK -->|transactions topic| KC
    AK -->|transactions topic| KSR
    KC[Kafka Connect] -->|into github_events table| Q[QuestDB]
    KC[Kafka Connect] -->|into trades table| Q[QuestDB]
    KC[Kafka Connect] -->|into smart_meters table| Q[QuestDB]
    KC[Kafka Connect] -->|into transactions table| Q[QuestDB]
  end

  subgraph "Real-time dashboards"
    Q -->|SELECT FROM trades, github_events, smart_meters, transactions, plant_sensors, and iot_data tables| G[Grafana]
  end



  subgraph Monitoring
    Q -->|pulls Prometheus metrics| TA[Telegraf Agent]
    AK -->|pulls MX metrics| TA
    TA -->|into monitoring tables| Q
  end

If you want to stop the components at any point, you can just ctrl+c and you can restart later running docker-compose up. For more permanent removal, please do check the Stopping all the components section.

End-to-end ingestion and visualization

Ingesting the trading real-time data

In this section we will use the Jupyter Notebook environment to ingest data using Python. To ingest the trading dataset using Golang, or Python from the command line, please go to the ingestion section of this document for details.

Navigate to http://localhost:8888/notebooks/Send-Trades-To-Kafka.ipynb.

This notebook will read the tradesMarch.csv file to read trading events, and will send the events to Apache Kafka in Avro binary format using a topic named trades. The CSV file contains real trades from different crypto currency symbols captured with the Coinbase API between March 1st and March 12th 2024. The raw data has been sampled in 30s intervals and it contains almost one million rows. By default, the script will override the original date with the current date and will wait 50ms between events before sending to Kafka, to simulate a real time stream and provide a nicer visualization. You can override those configurations by changing the constants in the script. It will keep sending data until you click stop or exit the notebook, or until the end of the file is reached.

The data you send to Kafka will be then polled by Kafka Connect and passed to QuestDB, where it will be stored into a table named trades.

Ingesting trading data directly into QuesDB, skipping Kafka

Having a message broker like Kafka in front of QuestDB makes sense, as you can replay messages in the case of any error, and also allows for interesting scenarios, like zero downtime upgrades. You can ingest data into Kafka and upgrade/restart QuestDB, and when QuestDB is available again the data from Kafka will be inserted into the database.

But, if you are fine skipping Kafka and want to ingest directly into QuestDB, you can check the http://localhost:8888/notebooks/Send-Trades-To-QuestDB-Directly.ipynb notebook.

This notebook, sends trades from several sub-processes in parallel, as this gives you higher throughput. The script uses QuestDB API to insert data directly

Visualizing the trading dataset

For now, you can navigate to the live dashboard at http://localhost:3000/d/trades-crypto-currency/trades-crypto-currency?orgId=1&refresh=250ms. User is admin and password quest.

You should see how data gets updated. The dashboard auto refreshes every 250 milliseconds. For the first few seconds some of the charts might look a bit empty, but after enough data is collected it should look better.

You can now proceed to checking Data on Kafka, unless you prefer to also ingest GitHub data. In the next sections discuss mostly the Trading dataset, but you can easily adapt all the queries and scripts to use the GitHub or any other dataset instead.

Ingesting the GitHub real-time data

In this section we will use the Jupyter Notebook environment to ingest data using Python. To ingest data using NodeJS, Golang, Rust, or JAVA, please go to the ingestion section of this document for details.

Navigate to http://localhost:8888/notebooks/Send-Github-Events-To-Kafka.ipynb.

This notebook will use the GitHub API to collect public events, will send the events to Apache Kafka using a topic named github_events, then wait 10 seconds to avoid any API rates. It will keep sending data until you click stop or exit the notebook.

Before you run the notebook, make sure you had set the GITHUB_TOKEN environment variable. Or you can just paste it where it says <YOUR_GITHUB_TOKEN>.

The data you send to Kafka will be passed to QuestDB, where it will be stored into a table named github_events. We will explore the database later.

For now, you can navigate to the live dashboard at http://localhost:3000/d/github-events-questdb/github-events-dashboard?orgId=1&refresh=5s. User is admin and password quest.

You should see how data gets updated. The dashboard auto refreshes every 5 seconds, but data is only collected every 10 seconds, so it will take ~10 seconds to see new results on the charts. For the first few minutes some of the charts might look a bit empty, but after enough data is collected it should look better.

At this point you can see you already have a running system in which you are capturing data and running it through an end-to-end data analytics platform.

Checking Data on Kafka

Apache Kafka provides a unified, high-throughput, low-latency platform for handling real-time data feeds. Data is organized into topics and store reliably for a configured retention period. Consumers can then read data from those topics immediately, or whenever is more convenient for them.

Kafka uses a binary TCP-based protocol that is optimized for efficiency and allows Kafka to turn a bursty stream of random message writes into linear writes.

There are many open source and commercial tools to add a web interface to Apache Kafka, but Apache Kafka itself doesn't has any interface other than its API. However, Kafka includes several command line tools we can use to manage our cluster and to query configuration and data.

We can, for example, use the kafka-topics tool to list and manage topics. Since we are already running a container with docker, we can attach to the container and invoke Kafka topics from itself, as in:

docker exec -ti rta_kafka_broker kafka-topics --list --bootstrap-server localhost:9092

If you run the Jupyter Notebook Send-Trades-To-Kafka.ipynb, the output of this command should include a topic named trades.

We can also consume events from that topic by running: docker exec -ti rta_kafka_broker kafka-console-consumer --bootstrap-server rta_kafka_broker:29092 --topic trades --from-beginning.

You will notice the output is very weird. When you are running any of the examples (trades, smart_meters, or transactions) that use AVRO instead of