Simulating the Real World: Survey & Resources

This repository is divided into two main sections:

Our Survey Paper Collection - This section presents our survey, "Simulating the Real World: A Unified Survey of Multimodal Generative Models" (IEEE TPAMI, 2026), which systematically unify the study of 2D, video, 3D and 4D generation within a single framework.

Text2X Resources – This section continues the original Awesome-Text2X-Resources, an open collection of state-of-the-art (SOTA) and novel Text-to-X (X can be everything) methods, including papers, codes, and datasets. The goal is to track the rapid progress in this field and provide researchers with up-to-date references.

⭐ If you find this repository useful for your research or work, a star is highly appreciated!

💗 This repository is continuously updated. If you find relevant papers, blog posts, videos, or other resources that should be included, feel free to submit a pull request (PR) or open an issue. Community contributions are always welcome!

🔔 News

• 🎉 [25 Apr 2026] Our survey is accepted by IEEE TPAMI 2026!
• ✨ [16 Feb 2026] Updated our survey (Version 3, 25 pages) on arXiv.
• ✨ [13 Aug 2025] Updated our survey (Version 2, 25 pages) on arXiv.
• ✨ [6 Mar 2025] Updated our survey (Version 1) on arXiv.

Table of Contents

• Our Survey Paper Collection
  • Abstract
    • 🔥 Cite Us!
    • 💡 New to this field? Check this!
  • The Dimensional Evolution of Generative AI
  • Paradigms
    • 2D Generation
    • Video Generation
      • Algorithms
      • Applications
    • 3D Generation
      • Algorithms
      • Applications
    • 4D Generation
      • Algorithms
      • Applications
  • Datasets and Evaluations
  • Other Related Resources
    • WorldModel Benchmark
    • World Foundation Model Platform
• Text2X Resources
  • 4D Subsection
    • Accepted Papers
    • ArXiv Papers
    • Additional Info

📜 Our Survey Paper Collection

Abstract

Understanding and replicating the real world is a critical challenge in Artificial General Intelligence (AGI) research. To achieve this, many existing approaches, such as world models, aim to capture the fundamental principles governing the physical world, enabling more accurate simulations and meaningful interactions. However, current methods often treat different modalities, including 2D (images), videos, 3D, and 4D representations, as independent domains, overlooking their interdependencies. Additionally, these methods typically focus on isolated dimensions of reality without systematically integrating their connections. In this survey, we present a unified survey for multimodal generative models that investigate the progression of data dimensionality in real-world simulation. Specifically, this survey starts from 2D generation (appearance), then moves to video (appearance+dynamics) and 3D generation (appearance+geometry), and finally culminates in 4D generation that integrate all dimensions. To the best of our knowledge, this is the first attempt to systematically unify the study of 2D, video, 3D and 4D generation within a single framework. To guide future research, we provide a comprehensive review of datasets, evaluation metrics and future directions, and fostering insights for newcomers. This survey serves as a bridge to advance the study of multimodal generative models and real-world simulation within a unified framework.

⭐ Citation

If you find this paper and repo helpful for your research, please cite it below:

@article{hu2026simulating,
  title={Simulating the real world: A unified survey of multimodal generative models},
  author={Hu, Yuqi and Wang, Longguang and Liu, Xian and Chen, Ling-Hao and Guo, Yuwei and Shi, Yukai and Liu, Ce and Rao, Anyi and Wang, Zeyu and Xiong, Hui},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2026},
  publisher={IEEE}
}

🧭 Getting Started with Key Concepts

[!Note] If you are new to this field, you can find clear and concise definitions of essential technical terms and concepts, such as NeRF, 3DGS, SDS, and Diffusion Models in our Glossary.

[!TIP] Feel free to pull requests or contact us if you find any related papers that are not included here. The process to submit a pull request is as follows:

• a. Fork the project into your own repository.
• b. Add the Title, Paper link, Conference, Project/GitHub link in README.md using the following format:

[Origin] **Paper Title** [[Paper](Paper Link)] [[GitHub](GitHub Link)] [[Project Page](Project Page Link)]

• c. Submit the pull request to this branch.

The Dimensional Evolution of Generative AI

We present a unified framework connecting 2D, Video, 3D, and 4D generation through text-guided synthesis. This paradigm illustrates how higher-dimensional content is synthesized by extending foundational modalities along spatial and temporal axes. (1)2D->3D: Spatial lifting of 2D priors to achieve geometric consistency; (2)2D->Video: Temporal inflation of static features to capture motion dynamics; (3)Video->4D: Spatial reconstruction and stabilization of dynamic sequences; (4)3D->4D: Temporal animation and deformation of static geometry. This perspective underscores that higher-dimensional generation methodologies are derivatives of foundational lower-dimensional generative priors, adapted through specialized architectural extensions.

Paradigms

2D Generation

Text-to-Image Generation.

Here are some seminal papers and models.

• Imagen: [NeurIPS 2022] Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding [Paper] [Project Page]
• DALL-E: [ICML 2021] Zero-shot text-to-image generation [Paper] [GitHub]
• DALL-E 2: [arXiv 2022] Hierarchical Text-Conditional Image Generation with CLIP Latents [Paper]
• DALL-E 3: [Platform Link]
• DeepFloyd IF: [GitHub]
• Stable Diffusion: [CVPR 2022] High-Resolution Image Synthesis with Latent Diffusion Models [Paper] [GitHub]
• SDXL: [ICLR 2024 spotlight] SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis [Paper] [GitHub]
• FLUX.1: [Platform Link]

---

Video Generation

Text-to-video generation models adapt text-to-image frameworks to handle the additional dimension of dynamics in the real world. We classify these models into three categories based on different generative machine learning architectures.

Survey

• [AIRC 2023] A Survey of AI Text-to-Image and AI Text-to-Video Generators [Paper]
• [arXiv 2024] Sora as an AGI World Model? A Complete Survey on Text-to-Video Generation [Paper]

Video Algorithms

(1) VAE- and GAN-based Approaches.

VAE-based Approaches.

• SV2P: [ICLR 2018 Poster] Stochastic Variational Video Prediction [Paper] [Project Page]
• [arXiv 2021] VideoGPT: Video Generation using VQ-VAE and Transformers [Paper] [GitHub] [Project Page]
• [arXiv 2021] FitVid: Overfitting in Pixel-Level Video Prediction [Paper] [GitHub] [Project Page]

GAN-based Approaches.

• [CVPR 2018] MoCoGAN: Decomposing Motion and Content for Video Generation [Paper] [GitHub]
• [CVPR 2022] StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2 [Paper] [GitHub] [Project Page]
• DIGAN: [ICLR 2022] Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks [Paper] [GitHub] [Project Page]
• [ICCV 2023] StyleInV: A Temporal Style Modulated Inversion Network for Unconditional Video Generation [Paper] [GitHub] [Project Page]

(2) Diffusion-based Approaches.

U-Net-based Architectures.

• [NeurIPS 2022] Video Diffusion Models [Paper] [Project Page]
• [arXiv 2025] UniVerse-1: Unified Audio-Video Generation via Stitching of Experts [Paper] [GitHub] [[Projec