Large language/multimodal models (LLMs/LMMs) store extensive pre-trained knowledge but struggle to maintain consistency with real-world updates, making it difficult to avoid catastrophic forgetting while acquiring evolving knowledge. Previous work focused on constructing textual knowledge datasets and exploring knowledge injection in LLMs, lacking exploration of multimodal evolving knowledge injection in LMMs. To address this, we propose the EVOKE benchmark to evaluate LMMs' ability to inject multimodal evolving knowledge in real-world scenarios. Meanwhile, a comprehensive evaluation of multimodal evolving knowledge injection revealed two challenges: (1) Existing knowledge injection methods perform terribly on evolving knowledge. (2) Supervised fine-tuning causes catastrophic forgetting, particularly instruction following ability is severely compromised. Additionally, we provide pathways and find that: (1) Text knowledge augmentation during the training phase improves performance, while image augmentation cannot achieve it. (2) Continual learning methods, especially Replay and MoELoRA, effectively mitigate forgetting. Our findings indicate that current knowledge injection methods have many limitations on evolving knowledge, which motivates further research on more efficient and stable knowledge injection methods.

Contribution 1: Evolving Knowledge Benchmark (EVOKE)

• We propose an automated pipeline for collecting evolving knowledge to construct EVOKE.
• A benchmark for evaluating evolving knowledge injection in real-world scenarios.

Contribution 2: Challenges of Evolving Knowledge Injection

• Extensive experiments have been conducted on evolving knowledge injection, revealing two challenges.
• Terrible performance of existing knowledge injection methods and catastrophic forgetting caused by supervised fine-tuning.

Contribution 3: Pathways of Evolving Knowledge Injection

• We provide the pathways and demonstrate that text knowledge augmentation during the training phase improves performance.
• And continual learning methods effectively mitigate forgetting.

EVOlving KnowledgE Benchmark

The EVOKE benchmark comprises 9,422 knowledge-image pairs for LMM knowledge injection, spanning 159 fine-grained types (29 New types and 130 Entity types).

Overall pipeline of the construction for EVOKE benchmark. (a) Firstly, collect original data from CNN and Wikipedia, and filter popular data. (b) Secondly, we use GPT-4o to summarize the textual content of the original data. (c) Subsequently, QA pairs are generated by GPT-4o and query images are downloaded from Google. (d) Lastly, we manually review the original knowledge image and query image. The source of heuristic query: we manually write multiple templates and randomly select one template for each piece of data.

Knowledge Injection Challenge 1: Terrible Performance

Findings 1: The non-zero performance of Vanilla.

This findings indicates that although there is no explicit exposure to query related knowledge during LMM training, model has the ability to leverage existing knowledge for reasonable inference in question answering.

Findings 2: No one knowledge injection method performs exceptionally well.

Experimental results indicate that none of the evaluated approaches demonstrate superior effectiveness in multimodal evolving knowledge injection tasks. The best-performing method, LLaVA's MM-RAG-Gloden Context, achieves only 56.13 in accuracy score, which falls short of expectations. Notably, some methods is extremely poor, for example, Qwen-VL-Chat's LoRA only has a accuracy 1.11 higher than Vanilla. These findings underscore the significant potential for advancement in the field of evolving knowledge injection.

Findings 3: MM-RAG outperforms SFT in overall performance, particularly in cross-modal retrieval.

As shown in Table1, MM-RAG performs well on the LLaVA-v1.5 and Qwen-VL-Chat without updating LLM parameters, avoiding potential side effects. Additionally, the results for Text-Only and Image-Only are similar but both are significantly lower than UniIR.

Findings 4: Internet Augmented Generation can help LMMs adapt to evolving knowledge.

By observation, Perplexity AI outperforms Gemini and approaches the Gloden Context'performance of the Qwen-VL-Chat. This highlights the internet retrieval capabilities of commercial LMMs. IAG methods independently retrieve critical information from the Internet without relying on external injection data, serving as input for model reasoning. They achieve goodish performance without side effects, motivating further research on more efficient IAG methods.

Knowledge Injection Performance on Fine-grained Types

Knowledge Injection Challenge 2: Catastrophic Forgetting

Knowledge Injection Pathway 1: Knowledge Augmentation

Knowledge Injection Pathway 2: Continual Learning

Ablation Experiment

Sequential Fine-Tuning

MM-RAG

Qualitative Examples

Previous Next

Our Team

Kailin Jiang

USTC & BIGAI

Yuntao Du

SDU

Yukai Ding

WUH & BIGAI

Yucheng Ren

USYD

Ning Jiang

NEFU

Zhi Gao

PKU & BIGAI

Zilong Zheng

BIGAI

Lei Liu

USTC

Bin Li

USTC

Qing Li

BIGAI

@article{jiang2025evoke,
  title = {When Large Multimodal Models Confront Evolving Knowledge:Challenges and Pathways},
  author = {Kailin Jiang and Yuntao Du and Yukai Ding and Yuchen Ren and Ning Jiang and Zhi Gao and Zilong Zheng and Lei Liu and Bin Li and Qing Li},
  year = {2025}
}