Project Overview

Synthetic data generation is useful when teams want to experiment, prototype, share examples, or test workflows without exposing raw sensitive datasets. However, synthetic data is often misunderstood. Generating fake-looking rows does not automatically make a dataset private, useful, or statistically realistic.

This project takes a more careful approach. It does not only generate synthetic rows; it also evaluates how closely the synthetic data preserves important properties of the original table.

The goal of this project is to demonstrate:

• A clean synthetic-data generation workflow
• Statistical and neural synthetic-data methods
• Honest quality evaluation
• Privacy-risk proxy diagnostics
• Optional train-on-synthetic, test-on-real utility evaluation
• Visual reporting
• Configurable CLI execution
• Tests and CI for reproducibility
• Clear limitations and responsible-use documentation

What This Project Does

This project can:

• Load a real tabular CSV dataset
• Generate synthetic rows using a Gaussian Copula method
• Generate synthetic rows using a lightweight VAE method
• Detect numeric and categorical columns automatically
• Preserve the original column structure in synthetic outputs
• Validate input data and configuration before generation
• Generate quality metrics in JSON format
• Generate a one-row quality_summary.csv for quick comparison
• Create visual diagnostics for distributions, PCA, correlations, and pairplots
• Produce lightweight HTML reports
• Compute privacy proxy diagnostics such as exact duplicate rate and nearest-neighbor distances
• Optionally evaluate downstream ML utility when a target column is configured
• Run automated tests and CI smoke workflows

What This Project Does Not Do

This project does not:

• Provide formal differential privacy
• Certify that synthetic data is safe to publish
• Guarantee that generated records cannot leak information
• Replace domain-specific privacy review
• Replace mature libraries such as SDV, CTGAN, or commercial privacy platforms
• Guarantee strong performance on every dataset
• Prove that a synthetic dataset is suitable for high-stakes use

A production-grade synthetic-data system would require stronger schema constraints, formal privacy evaluation, domain validation, monitoring, and security review.

Features

• Gaussian Copula generator for statistical synthetic tabular data
• Variational Autoencoder generator for neural synthetic tabular data
• Automatic schema detection for numeric and categorical columns
• Configurable VAE hyperparameters
• Configurable output directories
• Input dataframe validation
• Config validation with clear error messages
• Distribution overlap metrics
• Correlation difference metrics
• Categorical distribution similarity
• Numeric summary-statistic differences
• Boundary violation checks
• Exact duplicate-rate check
• Nearest-neighbor privacy proxy metrics
• Optional ML utility evaluation
• PCA projection chart
• Correlation heatmap
• Distribution comparison chart
• Pairplot comparison chart
• HTML report generation
• Unit test suite
• GitHub Actions CI support

Methods

The project currently supports two synthetic-data generation methods.

Gaussian Copula

The Gaussian Copula method models feature distributions and dependency patterns, then samples new rows from the fitted statistical structure.

It is often useful for small or medium-sized tabular datasets where statistical relationships are relatively stable.

Variational Autoencoder

The VAE method learns a compressed latent representation of the dataset and decodes synthetic rows from that latent space.

In this repository, the VAE is intentionally lightweight and configurable. It should be treated as a baseline neural generator, not a fully tuned production VAE.

Charts and Visual Analysis

The project automatically generates charts to make synthetic-data quality easier to inspect.

Generated charts are saved in:

outputs/<run_name>/plots/

Main charts include:

These charts are diagnostic tools. They help identify obvious quality problems, but they should not be treated as proof that synthetic data is private or production-ready.

Distribution Overlap Comparison

PCA Projection Comparison

Metric Summary

Interpretation: Higher distribution overlap and categorical similarity are better. Lower correlation difference and duplicate rate are better. On this demo dataset, Copula is the stronger generator, while the VAE is useful as a neural baseline but currently underfits the real data distribution.

How the Evaluation Works

The evaluation workflow compares real and synthetic datasets across multiple dimensions:

Real dataset
     ↓
Synthetic generator
     ↓
Synthetic dataset
     ↓
Quality metrics + visual diagnostics + optional ML utility evaluation

The evaluation includes:

• Numeric distribution similarity
• Numeric correlation preservation
• Categorical distribution similarity
• Summary-statistic differences
• Boundary validity checks
• Privacy proxy diagnostics
• Optional train-synthetic-test-real utility checks

This makes the project more useful than a generator-only demo, because it asks whether the generated data actually behaves like the original data.

Project Structure

Synthetic-Data-Artist/
│
├── .github/
│   └── workflows/
│       └── ci.yml
│
├── data/
│   ├── real_data.csv
│   └── synthetic_data_*.csv
│
├── outputs/
│   └── <run_name>/
│       ├── metrics.json
│       ├── quality_summary.csv
│       └── plots/
│           ├── distribution_overlap.png
│           ├── pca_projection.png
│           ├── correlation_heatmap.png
│           └── pairplot_comparison.png
│
├── reports/
│   └── <run_name>_report.html
│
├── synthetic_data_artist/
│   ├── main.py
│   ├── config.py
│   ├── data.py
│   ├── schema.py
│   ├── models/
│   │   ├── copula.py
│   │   └── vae.py
│   ├── evaluation/
│   │   ├── metrics/
│   │   │   ├── distribution.py
│   │   │   ├── privacy.py
│   │   │   └── utility.py
│   │   └── plots.py
│   └── reporting/
│       └── html_report.py
│
├── tests/
│   ├── test_core_contracts.py
│   ├── test_project_integrity.py
│   ├── test_enhanced_evaluation_metrics.py
│   └── test_cli_and_validation.py
│
├── config.yaml
├── requirements.txt
├── requirements-vae.txt
├── README.md
└── LICENSE

Installation

1. Clone the Repository

git clone https://github.com/AmirhosseinHonardoust/Synthetic-Data-Artist.git
cd Synthetic-Data-Artist

2. Create a Virtual Environment

On Windows CMD:

python -m venv .venv
.venv\Scripts\activate

On macOS/Linux:

python -m venv .venv
source .venv/bin/activate

3. Install

Install the project (and its dependencies) in editable mode:

pip install -e .

This also provides a synthetic-data-artist command. The Copula generator and all evaluation run on the core dependencies. The VAE generator (--method vae) additionally needs PyTorch, kept as an optional extra so the default install stays light:

pip install -e .[vae]

Prefer plain requirements files? pip install -r requirements.txt (and -r requirements-vae.txt for the VAE extra) install the same dependency sets.

Running the Generator

All commands below use python -m synthetic_data_artist.main; af