Awesome Chart Understanding

🔥🔥 Our survey paper has been accepted by IEEE TKDE! 🔥🔥

A curated list of recent and past chart understanding work based on our IEEE TKDE survey paper: From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models.

The repository will be continuously updated 📝. Don't forget to hit the ⭐️ and stay tuned!

If you find this resource beneficial for your research, please do not hesitate to cite the paper referenced in the Citation section. Thank you!

Table of Contents

• Tasks and Datasets
  • Chart Question Answering
  • Chart Captioning (Summarization)
  • Factual Inconsistency Detection for Chart Captioning
  • Chart Fact-checking
  • Chart Caption Factual Error Correction
  • Chart to Code
• Methods
  • Classification-based Methods
  • Generation-based Methods
  • Tool Augmentation
• Evaluation
• Analysis
• Citation

Tasks and Datasets

Chart Question Answering

Factoid Questions

• DVQA: Understanding Data Visualizations via Question Answering.

  Kushal Kafle, Brian Price, Scott Cohen, Christopher Kanan.

• FigureQA: An Annotated Figure Dataset for Visual Reasoning.

  Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler, Yoshua Bengio.

• LEAF-QA: Locate, Encode & Attend for Figure Question Answering.

  Ritwick Chaudhry, Sumit Shekhar, Utkarsh Gupta, Pranav Maneriker, Prann Bansal, Ajay Joshi.

• STL-CQA: Structure-based Transformers with Localization and Encoding for Chart Question Answering.

  Hrituraj Singh, Sumit Shekhar.

• PlotQA: Reasoning over Scientific Plots.

  Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, Pratyush Kumar.

• MapQA: A Dataset for Question Answering on Choropleth Maps.

  Shuaichen Chang, David Palzer, Jialin Li, Eric Fosler-Lussier, Ningchuan Xiao.

• ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning.

  Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, Enamul Hoque.

• SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphs.

  Shengzhi Li, Nima Tajbakhsh.

• MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning.

  Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, Dong Yu.

• MathVista: Evaluating Math Reasoning in Visual Contexts with GPT-4V, Bard, and Other Large Multimodal Models.

  Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, Jianfeng Gao.

• ChartBench: A Benchmark for Complex Visual Reasoning in Charts.

  Zhengzhuo Xu, Sinan Du, Yiyan Qi, Chengjin Xu, Chun Yuan, Jian Guo.

• Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models.

  Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, Qi Liu.

Long-form Questions

• OpenCQA: Open-ended Question Answering with Charts.

  Shankar Kantharaj, Xuan Long Do, Rixie Tiffany Leong, Jia Qing Tan, Enamul Hoque, Shafiq Joty.

Chart Captioning (Summarization)

• Neural Data-Driven Captioning of Time-Series Line Charts.

  Andrea Spreafico, Giuseppe Carenini.

• Figure Captioning with Relation Maps for Reasoning.

  Charles Chen, Ruiyi Zhang, Eunyee Koh, Sungchul Kim, Scott Cohen, Ryan Rossi.

• Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model.

  Jason Obeid, Enamul Hoque.

• What Will You Tell Me About the Chart? – Automated Description of Charts.

  Karolina Seweryn, Katarzyna Lorenc, Anna Wróblewska, Sylwia Sysko-Romańczuk.

• SciCap: Generating Captions for Scientific Figures.

  Ting-Yao Hsu, C Lee Giles, Ting-Hao Huang.

• Chart-to-Text: A Large-Scale Benchmark for Chart Summarization.

  Shankar Kantharaj, Rixie Tiffany Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, Shafiq Joty.

• LineCap: Line Charts for Data Visualization Captioning Models.

  Anita Mahinpei, Zona Kostic, Chris Tanner.

• ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summaries.

  Raian Rahman, Rizvi Hasan, Abdullah Al Farhad, Md Tahmid Rahman Laskar, Md. Hamjajul Ashmafee, Abu Raihan Mostofa Kamal. <img src='https://img.shields.io/badge