minellius

👀 Overview

Minellius, a experimental github data analysis solution, which designed to combine data collection, persistence, integration, analysis, visualization, real-time monitoring, trend tracking, originating from a group project of COMP-3002, 2018 fall, HITsz.

Minellius aims to help users understand the history, current, trend, ups and downs of github and even the open source world, enhance the sense of the times and the cultural aesthetics of the community; adjust the path of developers' learning and self-improvement to better integrate and participate in the development of community resources and culture.

📑 Table of Contents

🔨 How It Works

Crawl data from Gharchive and Github Api, then Aggregate, store, and visualize them.

⭐️ Features

1. Data Visualization with Charts and Tables
2. Streaming Crawling and Analysis, Minimal Processing Space Requirement
3. Basic User System with Bearer JWT Auth and Github Oauth Auth
4. Secure Store and Transmission
5. Continuous Integration
6. Auto Recovery and Email Alert
7. Autogen Documents
8. I18n (Multi-language)
9. Multi-platform
10. Antd Design

👓 Example Snapshot

Current Snapshot

Period Snapshot

🔧 Installation & Usage

Client

From Binary (Recommended)

Please see release page, and download the latest version for your platform. By default, we provide windows version (portable exe) and linux version (AppImage). And because of project deadline limitation, we only provide 1280x720px client.

From Source Code (Not Recommended)

1. First, enter into client directory and install the dependencies.Sure, you need node.js environment and avaiable network (for china users).

cd client
npm install

2. Then build client for your platform.

# auto select suitable platform (recommended)
npm run electron:local
# or manually select platform
npm run electron:windows
npm run electron:linux
npm run electron:mac # no test

3. You can find output in client/release directory, then copy and use it.

From Source Code To Dev (Only to developers)

Again, install then start hot-reload dev server on http://localhost:4200 with electron. If you need chrome devtools, edit client/src/main.ts, set devTools to be true and restart it.

cd client
npm install
npm run start

Server

Important: Target server of our client has been set as our server, if you need deploy your server, you should edit it in the client source and rebuild the client as well.

With Docker-compose (Recommended)

0. Sure, you need docker and docker-compose environment.

1. First, copy docker-compose.sample.yaml to docker-compose.yaml and edit environment variables in it or export them to your current shell.

2. Copy monitor/.docker-alertd.sample.yaml to monitor/.docker-alertd.yaml. Edit alert email information in it.

3. Optional: We recommend postgresql database to deploy our service, if you use it, you can create a new database and import init tables from deploy/database/postgresql/minellius.sql. Then you can skip later database import.

4. Optional: Because of huge amount of data, our 100GB database has no more space, so we cancel its auto start function in docker mode. To crawl period data, you should manually edit crawler/Dockfile and add crawler/src/go.py to the cmd field. For new fishs to docker or needing more information, see Dockerfile Reference.

5. Then easily build and start.

docker-compose build
docker-compose up

6. Waiting minellius-server container init the database tables, you need import init insert data from deploy/database/common. If you use postgresql, you can easily import all (see step 3) .sql files, else you need manually import one by one now.

7. If you need https and http2.0, install and start a nginx web server, copy our configuration file to its vhost directory, editting to fit yourself. And if you need free https cert, visit Let's encrypt.

Manually Deploy (Not Recommended)

Important: if you insist on manually deployment, you will lose the automatic email alert, and the automatic restart of the crawler will depends all on the third-party monitoring tool you use.

0. export need enviroment variables, see in docker-compose.sample.yaml environments fields.

1. Optional: We recommend postgresql database to deploy our service, if you use it, you can create a new database and import init tables from deploy/database/postgresql/minellius.sql. Then you can skip later database import.

2. minellius-server, ensuring you have node.js installed.

cd server
npm install
npm run start:prod
# or for developement use
npm run start:dev

4. database import - waiting minellius-server container init the database tables, you need import init insert data from deploy/database/common. If you use postgresql, you can easily import all (see step 3) .sql files, else you need manually import one by one now.

5. minellius-crawler, ensuring you have basic python 3.6+ tools installed. And you need daemon tools to help you run them as services. Without docker, we recommend pm2.

cd crawler/
pip install -r requirements.txt
cd src
python current.py
# if you need period
python go.py

6. If you need https and http2.0, install and start a nginx web server, copy our configuration file to its vhost directory, editting to fit yourself. And if you need free https cert, visit Let's encrypt.

💻 Platform Support

Windows	Linux	Mac
10+(7+ possible)	3.20+ With AppImage	Need manually build, no test now

🌏 Language Support

1. English
2. 简体中文

   More languages are coming soon...

👍 Acknowledgement

Electron	Angular	NG-Zorro	Nestjs	Docker

👉 Source Code Reference (Main)

1. client
   • infra - angular + electron + ant zorro
   • components - client/src/app/**/*.component.ts
   • providers & services - client/src/app/providers/*.service.ts
   • dtos - client/src/app/dtos/*.dto.ts
   • functools - client/src/app/functools/*.functool.ts & innerCode
   • directives - client/src/app/directives/*.directive.ts
   • interfaces - client/src/app/interfaces/*.interface.ts & innerCode
   • animations - client/src/app/animations/*.animations.ts
   • style - client/src/app/**/*.scss & innerCode
   • module - client/src/app/**/*.module.ts
   • router - client/src/app/app-routing.module.ts & routeService & routerService
   • guards - client/src/app/guards/auth.guard.ts
   • current - client/src/app/components/current*.* & client/src/app/providers/current.service.ts
   • period - client/src/app/components/period*.* & client/src/app/providers/period.service.ts
   • user - client/src/app/componets/log*.* & client/src/app/providers/user.service.ts & client/src/app/components/control.component.ts
   • helper - client/src/app/components/helper
   • i18n - TranslateService & src/app/providers/custom-translate.service.ts & src/assets/i18n/*.json
   • introduction - src/app/components/{home,video}.components.*
   • img & video & geo - src/assets
   • test - **/*.spec.ts
   • bootstrap - main{.ts,.js} | index.html
   • config
   • ...
2. server
   • nodejs + nest + native python + postgresql
   • auth - server/src/auth/**/* & server/src/role/**/*
   • user & group & permission ... - server/src/role/**/*
   • data - server/src/data/**/*
   • database - server/src/database/**/* & deploy/database/* & **/*.entities.ts & TypeOrmModule & **/*.service.ts
   • config - **/*-config.interface.ts && && environments variables & ...