Course Syllabus
Natural language and Information Visualization are critical when humans collaborate on solving challenging data-intensive problems and/or in making complex decisions. In this course we will explore how NLP (with a focus on LLMs) and Interactive InfoVis can be synergistically combined to enable effective human-AI collaboration in solving such problems/making such decisions in engineering, science, economics....
In particular, we will focus on four types of synergies between NLP (LLMs) and InfoVis:
- Text -> InfoVis (aka Text Analytics). When the user goal is to explore/analyze large textual repositories
- InfoVis for NLP: When the user goal is either to Evaluate or Interpret the NLP model)
- InfoVis(+data) -> text (Chart Captioning and Chat QA)
- InfoVis & data & text (when both inputs and outputs can be multimodal)
There are no specific NLP/InfoVis pre-reqs for this course, but the more you already know about these fields the better ;-). Also some background in AI and HCI would help. This is a CS grad course! Please talk to me if you are unsure on whether you could successfully take this course carenini@cs.ubc.ca.
The course will be organized as a glorified reading group + a final project. Students will present research papers (or existing tutorials) (number of times depends on how many students will register). On the days a student is not presenting, they will need to write a short critical summary of the readings (typically three per lecture - 3h once a week) + a few questions on such readings..
In the first lecture, I will propose a tentative reading list but students will be allowed to propose additional complementary/supplementary readings.
A key component of the course is the final project, where you will apply what you have learned to make a (minimal/partial/potential) research contribution to an NLP+InfoVis problem you care about related to the themes of the course.
TENTATIVE COURSE SCHEDULE (classes on Wednesday 9-12 in MCLD-Floor 2-Room 2014)
| Class | Reading | Presenter | Notes/Supplemental Materials |
|---|---|---|---|
| Sept 4 |
Course Intro |
Carenini G. | |
| Sept 11 |
Intro - NLP: From Language Modeling to LLMs (you are not required to attend if you already have a strong background in NLP) |
Carenini G. |
|
| Sept 18 |
Intro: InfoVis
|
Smith R. | Visualization Analysis & Design (half day (3 hrs) version, 143 slides) halfdaycourse20: IEEE VIS 2020 Tutorial, Utah/virtual, 10/20Slides: pdf, pdf 16up, key, ppt Videos: InfoVis@UBC YouTube playlist w/ 9 separable snippets, IEEE VIS YouTube video including Q&A and VIS paper previews during breaks (3.5 hrs) |
|
Sept 25 INTERPRETABILITY INFOVIS-FOR-NLP |
(a) Benjamin Hoover, Hendrik Strobelt, Sebastian Gehrmann, , exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers" (paper) Models” ACL Demo, 2020 [video] |
Sangyun Kwon [slides] |
(Focus on Sections 1, 2.1, 3 for this class) |
| (b) Yeh, C.; Chen, Y.; Wu, A.; Chen, C.; Vi´egas, Z.; Wattenberg, M. AttentionViz: A global view of transformer attention. (paper) (video)IEEE Transactions on Visualization and Computer Graphics Vol. 30, No. 1, 262–272, 2024. |
Yuwei Yin [slides] |
||
| (c) Wang, X.; Huang, R.; Jin, Z.; Fang, T.; Qu, H. CommonsenseVIS: Visualizing and understanding commonsense reasoning capabilities of natural language models. (paper) [video ] IEEE Transactions on Visualization and Computer Graphics Vol. 30, No. 1, 273–283, 2024. |
Xiang Zhang [slides] |
||
|
INTERPRETABILITY INFOVIS-FOR-NLP |
(a) Strobelt, H.; Webson, A.; Sanh, V.; Hoover, B.; Beyer, J.; Pfister, H.; Rush, A. M. Interactive and visual prompt engineering for ad-hoc task adaptation with large language models. IEEE Transactions on Visualization and Computer Graphics Vol. 29, No. 1, https://arxiv.org/abs/2208.07852 [video] |
Peyton Rapo [slides] |
|
|
(b) Cheng et al., 2023, RELIC: Investigating Large Language Models Responses using Self-Consistency |
Amirhossein Abaskohi [slides] |
||
|
START CAPTIONING |
IEEE Transactions on Visualization and Computer Graphics ( Volume: 28, Issue: 1, January 2022) |
Ritik Vatsal [slides] |
|
|
Oct 9 CAPTIONING InfoVis+>Text
|
(a) Shankar Kantharaj, Rixie Tiffany Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, Shafiq Joty Chart-to-Text: A Large-Scale Benchmark for Chart Summarization. . |
Amirhossein Dabiriaghdam [slides] |
https://github.com/khuangaf/Awesome-Chart-Understanding?tab=readme-ov-file#chart-captioning-summarization |
|
(b) Benny Tang, Angie Boggust, Arvind Satyanarayan VisText: A Benchmark for Semantically Rich Chart Captioning. . |
Brian Diep [slides] |
||
|
(c) Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, Qi Liu Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models. .ACL 2024 |
Aisha Eldeeb [slides] |
||
|
Oct 16 UNDERSTANDING |
(a) Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi R. Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, Heng Ji. Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning.
|
Armin Talaie |
|
|
(b) Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, Dong MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning. Yu. |
Sangyun Kwon (2) | ||
|
(c) Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, Danqi Chen. CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs. |
Yuwei Yin (2 [slides]) |
||
| Oct 23 CHART UNDERSTANDING & HUMAN LLM COLLABORATION |
(a) Nikhil Singh, Andrew Head, Lucy Lu Wang, Jonathan Bragg. FigurA11y: AI Assistance for Writing Scientific Alt Text.
|
Xiang Zhang (2) [slides] |
|
|
(b) Ahmed Masry, Megh Thakkar, Aayush Bajaj, Aaryaman Kartha, Enamul Hoque, Shafiq Joty. ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild.
|
Peyton Rapo (2) [slides] |
||
|
(c) Mohammed Saidul Islam, Raian Rahman, Ahmed Masry, Md Tahmid Rahman Laskar, Mir Tafseer Nayeem, Enamul Hoque. Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMs. |
Amirhossein Abaskohi (2) [slides] |
||
|
Oct 30 TEXT ANALYTICS TEXT=>INFOVIS & LLMs in VISUAL ANALYTICS |
(a) Bridging text visualization and mining: A task-driven survey Shixia Liu, Xiting Wang, Christopher Collins, Wenwen Dou, Fangxin Ouyang, Mennatallah El-Assady, Liu Jiang, Daniel A Keim
IEEE transactions on visualization and computer graphics, 2019
|
Ritik Vatsal (2) [slides] |
An "historical" perspective on techniques and tasks for visual text analytics |
|
(b) (text analytics) VADIS: A Visual Analytics Pipeline for Dynamic Document Representation and Information Seeking |
Amirhossein Dabiriaghdam (2) [slides] |
||
|
(c) LEVA: Using Large Language Models to Enhance Visual AnalyticsDownload INFOVIS 2024 |
Brian Diep (2) [slides not submitted?] |
||
| Nov 6 |
Groups, project proposal descriptions + slides + write-up PROJECT PROPOSAL PRESENTATIONS |
||
| Nov 13 |
mid-term break: no class |
||
|
Nov 20 LLMs in VISUAL ANALYTICS |
(a) PhenoFlow: A Human-LLM Driven Visual Analytics System for INFOVIS 2024 |
Aisha Eldeeb (2) | |
|
(b) InsightLens: Discovering and Exploring Insights from Conversational Contexts in Large-Language-Model-Powered Data Analysis Luoxuan Weng, Xingbo Wang, Junyu Lu, Yingchaojie Feng, Yihan Liu, Wei Chen Arxiv Preprint, 2024 |
Armin Talaie (2) | ||
|
(c) |
|||
| Nov 27 |
PROJECT UPDATES | ||
| Dec 4 |
Project feedback on demand (my office CICSR #105 OR on Zoom) |
||
|
Dec 11 9-noon (Room ICCS 104) |
|||
|
|
|||
|
|
