Awesome Knowledge Distillation of LLM Papers

A collection of papers related to knowledge distillation of large language models (LLMs). If you want to use LLMs for benefitting your own smaller models training, or use self-generated knowledge to achieve the self-improvement, just take a look at this collection.

We will update this collection every week. Welcome to star ⭐️ this repo to keep track of the updates.

❗️Legal Consideration: It's crucial to note the legal implications of utilizing LLM outputs, such as those from ChatGPT (Restrictions), Llama (License), etc. We strongly advise users to adhere to the terms of use specified by the model providers, such as the restrictions on developing competitive products, and so on.

💡 News

• 2024-2-20: 📃 We released a survey paper "A Survey on Knowledge Distillation of Large Language Models". Welcome to read and cite it. We are looking forward to your feedback and suggestions.

• Update Log

  • 2024-3-19: Add 14 papers.

Contributing to This Collection

Feel free to open an issue/PR or e-mail shawnxxh@gmail.com, minglii@umd.edu, hishentao@gmail.com and chongyangtao@gmail.com if you find any missing taxonomies or papers. We will keep updating this collection and survey.

📝 Introduction

KD of LLMs: This survey delves into knowledge distillation (KD) techniques in Large Language Models (LLMs), highlighting KD's crucial role in transferring advanced capabilities from proprietary LLMs like GPT-4 to open-source counterparts such as LLaMA and Mistral. We also explore how KD enables the compression and self-improvement of open-source LLMs by using them as teachers.

KD and Data Augmentation: Crucially, the survey navigates the intricate interplay between data augmentation (DA) and KD, illustrating how DA emerges as a powerful paradigm within the KD framework to bolster LLMs' performance. By leveraging DA to generate context-rich, skill-specific training data, KD transcends traditional boundaries, enabling open-source models to approximate the contextual adeptness, ethical alignment, and deep semantic insights characteristic of their proprietary counterparts.

Taxonomy: Our analysis is meticulously structured around three foundational pillars: algorithm, skill, and verticalization -- providing a comprehensive examination of KD mechanisms, the enhancement of specific cognitive abilities, and their practical implications across diverse fields.

KD Algorithms: For KD algorithms, we categorize it into two principal steps: "Knowledge Elicitation" focusing on eliciting knowledge from teacher LLMs, and "Distillation Algorithms" centered on injecting this knowledge into student models.

Skill Distillation: We delve into the enhancement of specific cognitive abilities, such as context following, alignment, agent, NLP task specialization, and multi-modality.

Verticalization Distillation: We explore the practical implications of KD across diverse fields, including law, medical & healthcare, finance, science, and miscellaneous domains.

Note that both Skill Distillation and Verticalization Distillation employ Knowledge Elicitation and Distillation Algorithms in KD Algorithms to achieve their KD. Thus, there are overlaps between them. However, this could also provide different perspectives for the papers.

Why KD of LLMs?

In the era of LLMs, KD of LLMs plays the following crucial roles:

Role	Description	Trend
① Advancing SLMs	Transferring advanced capabilities from proprietary LLMs to smaller SLMs, such as open source LLMs or other smaller models.	Most common
② Compression	Compressing LLMs to make them more efficient and practical.	More popular with the prosperity of open-source LLMs
③ Self-Improvement	Refining open-source LLMs' performance by leveraging their own knowledge, i.e. self-knowledge.	New trend to make open-source LLMs more competitive

📒 Table of Contents

• KD Algorithms

  • Knowledge Elicitation
    • Labeling
    • Expansion
    • Curation
    • Feature
    • Feedback
    • Self-Knowledge
  • Distillation Algorithms
    • Supervised Fine-Tuning
    • Divergence and Similarity
    • Reinforcement Learning
    • Rank Optimization

• Skill Distillation

  • Context Following
    • Instruction Following
    • Multi-turn Dialogue
    • RAG Capability
  • Alignment
    • Thinking Pattern
    • Preference
    • Value
  • Agent
    • Tool Using
    • Planning
  • NLP Task Specialization

KD Algorithms

Knowledge Elicitation

Labeling

Title	Venue	Date	Code	Data
Evidence-Focused Fact Summarization for Knowledge-Augmented Zero-Shot Question Answering	arXiv	2024-03
Aligning Large and Small Language Models via Chain-of-Thought Reasoning	EACL	2024-03	Github
Divide-or-Conquer? Which Part Should You Distill Your LLM?	arXiv	2024-02
Miko: Multimodal Intention Knowledge Distillation from Large Language Models for Social-Media Commonsense Discovery	arXiv	2024-02
KnowTuning: Knowledge-aware Fine-tuning for Large Language Models	arXiv	2024-02	Github
TinyLLM: Learning a Small Student from Multiple Large Language Models	arXiv	2024-02
Mixed Distillation Helps Smaller Language Model Better Reasoning