开云足球俱乐部大模型研究协同创新中心 - 开云足球俱乐部

3D interaction geometric pre-training for molecular relational learning

Sat, 11 Oct 2025 00:00:00 +0000

Eagle 2.5: boosting long-context post-training for frontier vision-language models

Sat, 11 Oct 2025 00:00:00 +0000

EgoExoBench: a benchmark for first-and third-person view video understanding in MLLMs

Sat, 11 Oct 2025 00:00:00 +0000

Gated integration of low-rank adaptation for continual learning of language models

Sat, 11 Oct 2025 00:00:00 +0000

LongVPO: from anchored cues to self-reasoning for long-form video preference optimization

Sat, 11 Oct 2025 00:00:00 +0000

Loquetier: a virtualized multi-LoRA framework for unified LLM fine-tuning and serving

Sat, 11 Oct 2025 00:00:00 +0000

MotionRAG: motion retrieval-augmented image-to-video generation

Sat, 11 Oct 2025 00:00:00 +0000

NeurIPS 2025 Accepted Papers Overview

Sat, 11 Oct 2025 00:00:00 +0000

NeurIPS (Annual Conference on Neural Information Processing Systems) is a top-tier conference in machine learning, alongside ICML and ICLR, recognized as one of the most challenging, highest-level, and most influential conferences in the field! NeurIPS is a CCF Class A conference and Core Conference Ranking Class A conference, with an H5 index of 278! Founded in 1987 in Canada by neural network scholars from the connectionist school, NeurIPS has grown in influence, with paper topics primarily focused on machine learning, artificial intelligence, and statistics.

The Large Model Center at Nanjing University’s School of Computer Science has 9 papers accepted to NeurIPS 2025.

01

Title: Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models

Authors: Yan-Shuo Liang, Jia-Rui Chen, Wu-Jun Li

Institution: Nanjing University

Abstract:

Thanks to the rich knowledge obtained from large-scale pre-training and subsequent fine-tuning strategies, existing large language models (LLMs) have demonstrated excellent performance across a wide range of tasks. However, when LLMs learn multiple downstream tasks sequentially, they often forget previously learned knowledge, leading to significant performance degradation on old tasks—a phenomenon known as catastrophic forgetting. Catastrophic forgetting hinders LLMs from continuously accumulating new knowledge, making it crucial to design continual learning methods that can overcome this challenge. Meanwhile, Low-Rank Adaptation (LoRA), as one of the most representative methods in parameter-efficient fine-tuning, has gained widespread attention in continual learning for LLMs. LoRA reparameterizes pre-trained weights into low-rank forms, requiring only a small number of parameters to be updated for task adaptation. Compared to full parameter updates, LoRA significantly improves fine-tuning efficiency. However, existing LoRA-based continual learning methods still have limitations. They typically expand new LoRA branches when learning new tasks while freezing old branches, thereby avoiding forgetting caused by directly modifying old parameters. During inference, these methods usually adopt simple addition to integrate new and old branches. This approach forces new and old branches to contribute equally to old tasks, which may instead cause new branches to significantly interfere with old tasks, exacerbating forgetting and reducing overall performance. To address this, we propose GainLoRA (gated integration of low-rank adaptation), a new continual learning method for LLMs. GainLoRA expands new LoRA branches for each new task and dynamically integrates new and old branches through a gating module. By imposing initialization and update constraints on the new gating module, GainLoRA significantly reduces interference from new LoRA branches on old tasks, effectively mitigating forgetting and improving the overall performance of LLMs in continual learning.

Figure 1

02

Title: StreamForest: Efficient Online Video Understanding with Persistent Event Memory

Authors: Xiangyu Zeng, Kefan Qiu, Qingyu Zhang, Xinhao Li, Jing Wang, Jiaxin Li, Ziang Yan, Kun Tian, Meng Tian, Xinhai Zhao, Yi Wang, Limin Wang

Institution: Nanjing University, Shanghai AI Laboratory, Zhejiang University, Huawei Noah’s Ark Lab, Yinwang Intelligent Technology

Abstract:

Multimodal large language models have made significant progress in video understanding in recent years. However, due to historical visual feature storage limitations and insufficient real-time spatiotemporal reasoning capabilities, their effectiveness in real-time streaming scenarios remains limited. To address these challenges, we propose StreamForest, a novel architecture specifically designed for streaming video understanding. The core of StreamForest is the Persistent Event Memory Forest, a memory mechanism that can adaptively organize video frames into multiple event-level tree structures. This process is guided by a penalty function based on temporal distance, content similarity, and merging frequency, enabling efficient long-term memory retention under limited computational resources. To enhance real-time perception, we introduce the Fine-grained Spatiotemporal Window, which captures detailed short-term visual cues to improve current scene perception. Additionally, we propose OnlineIT, an instruction tuning dataset customized for streaming video tasks. OnlineIT significantly improves MLLM performance in real-time perception and future prediction. To evaluate its generalization ability in practical applications, we introduce ODV-Bench, a new benchmark focused on real-time streaming video understanding in autonomous driving scenarios. Experimental results show that StreamForest achieves state-of-the-art performance, reaching 77.3% accuracy on StreamingBench, 60.5% on OVBench, and 55.6% on OVO-Bench. Notably, even under extreme visual token compression (limited to 1024 tokens), the model maintains 96.8% average accuracy across eight benchmarks (relative to the default 8k setting). These results highlight StreamForest’s robustness, efficiency, and versatility in streaming video understanding.

Figure 2

Figure 3

03

Title: LongVPO: From Anchored Cues to Self-Reasoning for Long-Form Video Preference Optimization

Authors: Zhenpeng Huang, Jiaqi Li, Zihan Jia, Xinhao Li, Desen Meng, Lingxue Song, Xi Chen, Liang Li, Limin Wang

Institution: Nanjing University, China Mobile Research Institute

Abstract:

Current vision-language models (VLMs) have limited performance in long video understanding: they rely on expensive and scarce long video annotations, and short-context models easily overlook intermediate content when extended to long sequences, causing performance imbalance between long and short tasks. To address this, we propose LongVPO—a two-stage direct preference optimization framework that requires no long video annotations. LongVPO first uses “anchored cues” to automatically synthesize preference data from short video clips, then achieves cross-clip alignment through “self-reasoning” on real long videos, learning complex long-range reasoning capabilities. Using only 16K synthetic data, LongVPO achieves superior performance on LVBench, LongVideoBench, MLVU, VideoMME, and other benchmarks while maintaining strong performance on short video tasks, providing a new paradigm for efficient and scalable long video understanding.

Figure 4

04

Title: Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models

Authors: Guo Chen, Zhiqi Li, Shihao Wang, Jindong Jiang, Yicheng Liu, Lidong Lu, De-An Huang, Wonmin Byeon, Matthieu Le, Tuomas Rintamaki, Tyler Poon, Max Ehrlich, Tong Lu, Limin Wang, Bryan Catanzaro, Jan Kautz, Andrew Tao, Zhiding Yu, Guilin Liu

Institution: Nanjing University, NVIDIA, Hong Kong Polytechnic University, Rutgers University

Abstract:

Eagle 2.5 is a series of frontier vision-language models (VLMs) designed for long-context multimodal understanding. Existing VLMs mainly focus on short-context tasks, with insufficient support for long video understanding and high-resolution image processing. Eagle 2.5 proposes a general training framework with two core technologies: Automatic Degradation Sampling (ADS) and Image Area Preservation (IAP), which dynamically allocate visual and text input budgets and maintain image integrity when segmenting. Additionally, the authors introduce a progressive mixed post-training strategy that gradually extends context length to improve model stability in handling diverse inputs. To support training, they construct the new Eagle-Video-110K dataset, providing story-level and clip-level dual annotations to enhance long video understanding capabilities. Experiments show that Eagle 2.5 achieves significant improvements on multiple long video and image understanding benchmarks. For example, the 8B parameter Eagle 2.5 achieves 72.4% on Video-MME with 512 frame input, approaching the performance of larger models like GPT-4o and Qwen2.5-VL-72B. The model also performs excellently on high-resolution image understanding tasks. In summary, Eagle 2.5 achieves efficient and powerful long-context multimodal understanding capabilities through innovative sampling strategies, progressive training methods, and large-scale multi-level datasets, providing a strong direction for future high-performance VLM development.

Figure 5

05

Title: VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception

Authors: Ziang Yan, Xinhao Li, Yinan He, Zhengrong Yue, Xiangyu Zeng, Yali Wang, Yu Qiao, Limin Wang, Yi Wang

Institution: Zhejiang University, Shanghai AI Laboratory, Nanjing University, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences

Abstract:

Infusing reasoning capabilities into multimodal large language models is key to achieving human-level perception and understanding. Existing methods mostly rely on the reasoning capabilities of LLMs to analyze parsed visual information, but are often limited by static perception stages. This paper proposes “Visual Test-Time Scaling,” which enhances multimodal LLM reasoning capabilities through iterative perception during inference. Under the guidance of updated text predictions, it gradually refines attention to high-confidence spatiotemporal regions, mimicking human hierarchical attention mechanisms. The training process combines reinforcement learning with spatiotemporal supervision signals for end-to-end optimization of reasoning paths. These designs allow multimodal LLMs to improve performance by increasing perception computational capacity. Extensive experiments validate the effectiveness and generalization of the iterative perception method across various tasks and benchmarks. The newly introduced Videochat-R1.5 model achieves significant improvements across more than 15 benchmarks covering video dialogue, video reasoning, and spatiotemporal perception, with an average improvement of more than 5% compared to robust baselines like Qwen2.5VL-3B and -7B.

Figure 6

06

Title: MotionRAG: Motion Retrieval-Augmented Image-to-Video Generation

Authors: Chenhui Zhu, Yilu Wu, Shuai Wang, Gangshan Wu, Limin Wang

Institution: Nanjing University

Abstract:

Thanks to the development of diffusion models, image-to-video generation technology has made significant progress. However, generating motion-realistic videos remains a formidable challenge. The core of this challenge lies in accurately modeling the complexity of motion, which requires capturing physical laws, object interactions, and domain-specific motion patterns—prior knowledge that is difficult to generalize effectively across diverse scenarios. To address this, we propose MotionRAG, a retrieval-augmented generation framework. This framework extracts and transfers motion priors from relevant reference videos through a Context-Aware Motion Adaptation (CAMA) mechanism to improve the motion realism of generated videos. The core technical innovations include: (1) Retrieval-based motion representation extraction: using video encoders and resamplers to extract semantic-level motion features from retrieved reference videos; (2) Context learning-based motion adaptation method: efficiently learning and transferring motion patterns from multiple retrieved reference videos to target scenarios through a causal Transformer architecture; (3) Attention motion injection adapter: injecting motion features into pre-trained video diffusion models to enhance motion realism. Extensive experiments demonstrate that our method achieves significant improvements across multiple scenarios and various base models, introducing only negligible computational overhead during inference. Furthermore, its modular design supports zero-shot generalization to new domains—simply updating the retrieval database without retraining any model components. This research enhances the core capabilities of video generation systems by enabling efficient retrieval and transfer of motion priors, providing a new paradigm for synthesizing videos with realistic dynamic effects.

Figure 7

07

Title: Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving

Authors: Yuchen Zhang, Hanyue Du, Chun Cao, Jingwei Xu

Institution: Nanjing University

Abstract:

Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning (PEFT) technique for adapting large language models (LLMs) to downstream tasks. Although numerous studies have explored strategies for unifying LLM training and serving, the domain of unified fine-tuning and inference for LoRA-based models remains underexplored. This paper proposes Loquetier—a virtualized multi-LoRA framework that seamlessly integrates LoRA fine-tuning and inference serving in a single runtime environment. Loquetier consists of two main components: (1) a virtualization module that isolates PEFT-based model modifications and supports deploying multiple adapters on a shared single base model; (2) an optimized computational flow with kernel designs that fuse fine-tuning and inference paths in forward propagation, enabling efficient batch processing and minimizing kernel call overhead. In extensive experiments across three task scenarios, Loquetier significantly outperforms existing baselines in both performance and flexibility: achieving 3.0× throughput of top co-serving systems in inference-only tasks, and 46.4× higher service level objective attainment rate than PEFT in unified fine-tuning and inference tasks.

Figure 8

08

Title: 3D Interaction Geometric Pre-training for Molecular Relational Learning

Authors: Namkyeong Lee, Yunhak Oh, Heewoong Noh, Gyoung S. Na, Minkai Xu, Hanchen Wang, Tianfan Fu, Chanyoung Park

Institution: KAIST, KRICT, Stanford University, Genentech, Nanjing University

Abstract:

Accurate prediction of molecular interactions is crucial in drug discovery and materials science. However, existing molecular relational learning methods are mostly limited to using 2D topological structures of molecules, ignoring 3D spatial geometric information that determines the nature of interactions—primarily because obtaining precise 3D interaction conformations is extremely expensive. To break through this bottleneck, we propose 3DMRL, an innovative 3D geometric pre-training framework. The core of this framework is that instead of relying on expensive computations to obtain true interaction conformations, it simulates how molecules contact each other in 3D space by constructing a “virtual interaction environment”—arranging multiple small molecules around a large molecule through random sampling, translation, and rotation. Based on this, we design dual pre-training tasks to guide 2D models to learn 3D geometric information in this virtual environment: one uses contrastive learning to help models understand the global geometric structure of interactions; the other uses an equivariant network to predict fine local relative geometric relationships between molecules, capturing atomic-level interaction details. Extensive experiments show that 3DMRL can significantly improve the performance of various mainstream models on molecular interaction prediction and drug-drug interaction prediction tasks, achieving up to 24.93% performance improvement across 40 tasks and demonstrating excellent generalization capabilities in out-of-distribution scenarios. This work systematically introduces 3D geometric pre-training to the field of molecular relational learning for the first time, laying a solid foundation for developing more accurate and versatile AI-assisted scientific discovery tools.

Figure 9

09

Title: EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs

Authors: Yuping He, Yifei Huang, Guo Chen, Baoqi Pei, Jilan Xu, Tong Lu, Jiangmiao Pang

Institution: Nanjing University, Shanghai AI Laboratory, University of Tokyo, Zhejiang University, Fudan University

Abstract:

Human intelligence can naturally transfer and integrate knowledge between first-person (egocentric) and third-person (exocentric) perspectives, which is crucial for learning and communication. However, although current multimodal large language models (MLLMs) have achieved significant progress in single-perspective video understanding, they still lack systematic evaluation of cross-perspective reasoning. To address this, we propose EgoExoBench—the first benchmark for evaluating MLLMs’ first-person and third-person video understanding and reasoning capabilities.

EgoExoBench is built on public datasets and contains 7300+ multiple-choice questions (MCQs) covering 11 sub-tasks, divided into three major challenges: semantic alignment, viewpoint association, and temporal reasoning. Task designs cover matching from task, action, object, to person levels, as well as cross-perspective spatial correspondence and event sequence reasoning.

The research team conducted systematic evaluation of 13 mainstream open-source and closed-source MLLMs (such as GPT-4o, Claude 3.7 Sonnet, Qwen2.5-VL, InternVL3, etc.). Results show that these models perform well on single-perspective tasks but exhibit significant performance degradation on cross-perspective tasks. For example, the best open-source model Qwen2.5-VL-72B achieves only 47% overall accuracy, while humans achieve over 90% accuracy on the same tasks. Further experiments show that chain-of-thought (CoT) prompting does not improve performance and even reduces accuracy on some tasks, indicating that cross-perspective reasoning remains a major challenge for existing models.

In summary, EgoExoBench provides a systematic and scalable evaluation framework that helps advance embodied agents and human-robot collaboration systems with human-like cross-perspective intelligence.

Figure 10

StreamForest: efficient online video understanding with persistent event memory

Sat, 11 Oct 2025 00:00:00 +0000

VideoChat-r1.5: visual test-time scaling to reinforce multimodal reasoning by iterative perception

Sat, 11 Oct 2025 00:00:00 +0000

Professor Wang Limin Receives 2025 Ant Intech Technology Award

Fri, 19 Sep 2025 00:00:00 +0000

Recently, at the 2025 Inclusion Bund Conference, the “2025 Ant Intech Award” was officially announced. 10 young scientists received the “Ant Intech Technology Award”. At the same time, 10 Chinese doctoral students from top universities worldwide received the “Ant Intech Scholarship”. Among them, Professor Wang Limin received the 2025 Ant Intech Technology Award.

The 2025 Ant Intech Award is established by Ant Group Co., Ltd., providing public welfare research funding support for outstanding young scholars and doctoral students in the field of computer science, with two core awards: the “Ant Intech Technology Award” and the “Ant Intech Scholarship”.

Figure: 2025 Ant Intech Technology Award Ceremony

Academicians and industry authorities attended the award ceremony, including Chen Chun (Academician of Chinese Academy of Engineering, Professor at Zhejiang University), Zhang Hongjiang (Foreign Academician of US National Academy of Engineering), and Zheng Weimin (Academician of Chinese Academy of Engineering, Professor at Tsinghua University). Michael I. Jordan (Member of US National Academy of Sciences, Engineering, and Arts & Sciences) and Jack Dongarra (Turing Award winner, Academician of US National Academy of Engineering, Professor at University of Tennessee) sent video messages to young scholars: “The path of research may not be smooth, but the problems you explore today will define future technologies and opportunities. Be bold in seeking truth, and your research will ultimately impact the world.”

It is understood that this year’s award recipients have demonstrated exceptional innovation capabilities in frontier areas such as Artificial General Intelligence (AGI), embodied intelligence, digital medicine, and data security, with their achievements being widely adopted by the industry. Professor Wang Limin won the award for his significant contributions to Artificial General Intelligence. The award citation: developed the first leading general video understanding large model InternVideo (with over 5 million downloads), proposed the “progressive training” method, enabling AI to understand the dynamic world in layers like humans, and empowered application scenarios such as autonomous driving.

Figure: Professor Wang Limin participating in the 2025 Ant Intech Technology Award ceremony round table forum

Correspondence as video: test-time adaption on SAM2 for reference segmentation in the wild

Tue, 12 Aug 2025 00:00:00 +0000

Divide-and-conquer for enhancing unlabeled learning, stability, and plasticity in semi-supervised continual learning

Tue, 12 Aug 2025 00:00:00 +0000

ICCV 2025 Accepted Papers

Tue, 12 Aug 2025 00:00:00 +0000

ICCV is one of the most influential top-tier conferences in computer vision. It is organized by the IEEE Computer Society and held biennially alongside CVPR and ECCV as the three flagship vision venues. ICCV covers cutting-edge topics such as image processing, object detection, 3D reconstruction, video understanding, and vision–language research, serving as a premier platform for presenting the latest advances and exchanging ideas. With its very high acceptance standards, ICCV represents the frontier trends and research hotspots of the field.

Seven papers from the Large Model Center of the Department of Computer Science and Technology, Nanjing University (NJU MCG), have been accepted to ICCV 2025.

01

Title: MobileViCLIP: An Efficient Video-Text Model for Mobile Devices

Authors: Min Yang, Zihan Jia, Zhilin Dai, Sheng Guo, Limin Wang

Affiliations: Nanjing University; Ant Group

Abstract:

Although large models have achieved strong performance on many vision tasks, efficient lightweight neural networks are receiving growing attention due to their faster inference and easier deployment on mobile devices. However, existing video models still focus on larger ViT architectures, with few attempts to build efficient video architectures. Given that many efficient CLIP models already demonstrate strong zero-shot classification and retrieval capabilities, we aim to fill the gap for video–text understanding and propose MobileViCLIP, a fast and efficient video–text model with strong zero-shot capability that can be deployed on mobile devices. Concretely, MobileViCLIP achieves performance comparable to mainstream ViT-based models on several text–video retrieval and zero-shot video classification datasets, while improving inference speed on mobile devices by tens of times. We believe focusing on efficiency for video–text models is important and valuable to the field.

02

Title: p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decay

Authors: Jun Zhang (张峻), Desen Meng (孟德森), Zhengming Zhang (张拯明), Zhenpeng Huang (黄振鹏), Tao Wu (吴涛), Limin Wang (王利民)

Affiliations: Nanjing University; China Mobile Research Institute

Abstract:

Despite the strong performance of multimodal large language models (MLLMs) on various downstream tasks, their massive training and inference costs hinder further development. A major cause is that the LLM must process an enormous number of visual tokens. We propose p-MoD, an efficient MLLM architecture that significantly reduces computational cost during both training and inference while maintaining performance. To reduce the number of visual tokens processed at each LLM Transformer layer, p-MoD introduces a Mixture-of-Depths (MoD) mechanism that processes only the most informative tokens at each layer and skips redundant ones. Integrating MoD into MLLMs is nontrivial; to address training/inference stability and limited training data, p-MoD designs Tanh-gated Weight Normalization (TanhNorm) and Symmetric Token Reweighting (STRing). Furthermore, we observe that visual token redundancy increases in deeper layers and thus propose Progressive Ratio Decay (PRD) to gradually reduce the kept-token ratio layer by layer. This key design fully unlocks MoD’s potential, markedly boosting efficiency and performance. On 15 benchmarks with LLaVA-1.5 and LLaVA-NeXT baselines, p-MoD matches or surpasses performance while using 55.6% inference TFLOPs, 53.7% KV cache, and 77.7% GPU training time.

03

Title: Scalable Image Tokenization with Index Backpropagation Quantization

Authors: Fengyuan Shi (石丰源), Zhuoyan Luo (罗卓彦), Yixiao Ge (葛艺潇), Yujiu Yang (杨余久), Ying Shan (单瀛), Limin Wang (王利民)

Affiliations: Nanjing University; Tsinghua University; Tencent

Abstract:

Existing vector quantization (VQ) methods face scalability issues, largely because codebooks updated only partially during training become unstable: as the distribution gap between inactive codes and visual features widens, codebook utilization drops and training eventually collapses. We propose Index Backpropagation Quantization (IBQ), a new VQ method that jointly optimizes all codebook embeddings and the visual encoder. By applying a straight-through estimator to the one-hot categorical distribution between encoded features and the codebook, IBQ makes all codes differentiable and maintains a latent space consistent with the visual encoder. IBQ enables scalable training of visual tokenizers and, for the first time, achieves large codebooks with high utilization at high dimension (256) and scale (2¹⁸). On ImageNet, IBQ shows strong scalability and competitive performance for both image reconstruction and autoregressive visual generation.

04

Title: Make Your Training Flexible: Towards Deployment-Efficient Video Models

Authors: Chenting Wang, Kunchang Li, Tianxiang Jiang, Xiangyu Zeng, Yi Wang, Limin Wang

Affiliations: Shanghai AI Laboratory; Shanghai Jiao Tong University; University of Science and Technology of China; Nanjing University

Abstract:

Mainstream video training typically relies on fixed spatiotemporal sampling grids that extract a fixed number of visual tokens as input, making both training and inference heavily constrained by preset sampling strategies. This rigid design hampers adaptation to varying computational budgets in downstream scenarios—especially when models trained under high compute cannot be efficiently deployed on resource-limited edge devices. We propose a new training paradigm to achieve “lossless adaptation across scenarios”: retain top performance under high compute while enabling lossless migration to low-resource environments. We first introduce Token Optimization (TO), an adaptive inference framework that dynamically samples and selects tokens according to downstream compute limits to maximize information utilization. We then develop Flux, a training-side data augmentation tool that enables flexible sampling grids with token selection, integrating seamlessly into mainstream video training frameworks to markedly enhance robustness and flexibility at near-zero extra cost. Integrated into large-scale video pretraining, FluxViT sets new SOTA under standard compute; notably, with only 1/4 tokens, FluxViT with TO still rivals the best InternVideo2 models, saving nearly 90% compute without loss.

05

Title: VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos

Authors: Jiashuo Yu, Yue Wu, Meng Chu, Zhifei Ren, Zizheng Huang, Pei Chu, Ruijie Zhang, Yinan He, Qirui Li, Songze Li, Zhenxiang Li, Zhongying Tu, Conghui He, Yu Qiao, Yali Wang, Yi Wang, Limin Wang

Affiliations: Shanghai AI Laboratory; Nanjing University; SIAT, Chinese Academy of Sciences (Shenzhen)

Abstract:

We introduce VRBench—the first long-form narrative video benchmark specifically designed to evaluate multi-step reasoning in large models—addressing limitations of existing evaluations that overlook temporal reasoning and process validity. VRBench contains 1,010 long videos (avg. length 1.6 hours), 9,468 human-annotated multi-step QA pairs, and 30,292 timestamped reasoning steps. Videos are curated through a multi-stage pipeline with expert cross-check, ensuring coherent plots and complexity. We build a human-in-the-loop framework to generate coherent chains-of-reasoning with timestamped steps across seven types (e.g., causal attribution, implicit reasoning). A multi-stage evaluation assesses models by both results and processes: beyond MCQ results, we propose an LLM-guided process score to comprehensively assess reasoning-chain quality. Experiments with 12 LLMs and 16 VLMs reveal current limitations in long-video multi-step reasoning and offer recommendations.

06

Title: Divide-and-Conquer for Enhancing Unlabeled Learning, Stability, and Plasticity in Semi-supervised Continual Learning

Authors: Yue Duan (段岳), Taicai Chen (陈泰财), Lei Qi (祁磊), Yinghuan Shi (史颖欢)

Affiliations: Nanjing University; Southeast University

Links: https://arxiv.org/abs/2508.05316, https://github.com/NJUyued/USP4SSCL

Abstract:

Semi-supervised continual learning (SSCL) aims to learn from a sequence of tasks in which only part of the data is labeled—highly practical yet challenging. The core challenge is to effectively leverage unlabeled data while balancing memory stability (avoiding forgetting) and learning plasticity (learning new knowledge). We propose USP, a divide-and-conquer collaborative framework that systematically enhances Unlabeled learning, Stability, and Plasticity via three coupled modules. For plasticity, we propose Feature Space Reservation (FSR), which uses an Equiangular Tight Frame (ETF) to reserve positions in the feature space for future classes, reducing conflicts when learning new tasks. For unlabeled learning, we design Divide-and-Conquer Pseudo-labeling (DCP), which splits unlabeled data into high- and low-confidence subsets and assigns pseudo-labels using a classifier and a more robust Nearest Class Mean (NCM), respectively, fully utilizing all data. For stability, we introduce Class-mean anchored Unlabeled Distillation (CUD), which reuses DCP’s intermediate results and anchors unlabeled data to stable class centers computed from labeled data, effectively mitigating catastrophic forgetting. Extensive experiments show that USP significantly outperforms SOTA, improving final-task accuracy by up to 5.94%.

07

Title: Correspondence as Video: Test-Time Adaption on SAM2 for Reference Segmentation in the Wild

Authors: Haoran Wang (王皓冉), Zekun Li (李泽昆), Jian Zhang (张剑), Lei Qi (祁磊), Yinghuan Shi (史颖欢)

Affiliations: Nanjing University; Southeast University

Links: https://arxiv.org/abs/2508.07759, https://github.com/wanghr64/cav-sam

Abstract:

Large vision models (e.g., SAM) often degrade on downstream segmentation tasks involving new domains or categories. Reference Segmentation, which uses an annotated reference image to guide the segmentation of a target image, is a promising direction. However, existing methods largely rely on meta-learning, requiring heavy training data and compute. We propose CAV-SAM, a new paradigm that turns the “correspondence” between the reference and target images into a “pseudo video,” enabling the latest video model SAM2 to adapt effectively through lightweight test-time tuning, completely avoiding costly meta-learning. The framework includes: (1) Diffusion-based Semantic Transition (DBST), which generates a smooth semantic transition sequence (pseudo video) from the reference to the target to handle semantic differences (same class, different instances); and (2) Test-Time Geometric Alignment (TTGA), which performs lightweight tuning of SAM2 using only the reference image and a novel enhanced cycle-consistency loss to better align geometric changes (pose, scale). Without meta-learning, CAV-SAM surpasses prior SOTA by about 5% on average across multiple datasets.

Make your training flexible: towards deployment-efficient video models

Tue, 12 Aug 2025 00:00:00 +0000

MobileViCLIP: an efficient video-text model for mobile devices

Tue, 12 Aug 2025 00:00:00 +0000

p-MoD: building mixture-of-depths MLLMs via progressive ratio decay

Tue, 12 Aug 2025 00:00:00 +0000

Scalable image tokenization with index backpropagation quantization

Tue, 12 Aug 2025 00:00:00 +0000

VRBench: a benchmark for multi-step reasoning in long narrative videos

Tue, 12 Aug 2025 00:00:00 +0000

Differentiable solver search for fast diffusion sampling

Fri, 18 Jul 2025 00:00:00 +0000

Elucidating the design space of multimodal protein language models

Fri, 18 Jul 2025 00:00:00 +0000

ICML 2025 Accepted Papers

Fri, 18 Jul 2025 00:00:00 +0000

ICML is one of the most prestigious and influential conferences in machine learning. It is among the longest-running and largest venues in the field and a CCF Class-A conference.

Four papers from the Large Model Center of the Department of Computer Science and Technology, Nanjing University (NJU MCG), have been accepted to ICML 2025.

01

Title: On the Tension between Byzantine Robustness and No-Attack Accuracy in Distributed Learning

Authors: Yi-Rui Yang, Chang-Wei Shi, Wu-Jun Li

Affiliations: Nanjing University

Link: https://cs.nju.edu.cn/lwj/paper/ICML2025_NFLinBRDL.pdf

Abstract:

Distributed machine learning leverages multiple interconnected devices (nodes) and their data to train models. As datasets and models scale up, large clusters face higher rates of software/hardware failures; in open-network scenarios such as federated learning, adversarial attacks are also more likely. Faulty or malicious nodes are called Byzantine nodes. Byzantine-robust distributed learning often uses robust aggregators to withstand such behavior. However, when no Byzantine nodes are present, the effect of robust aggregation is underexplored. This work theoretically analyzes aggregation error in the no-attack setting and proves that the worst-case aggregation error of a robust aggregator increases with the number of Byzantine nodes it is designed to tolerate—revealing an inherent tension between Byzantine robustness and no-attack accuracy. For both non-convex objectives and those satisfying the Polyak–Łojasiewicz (PL) condition, the paper establishes tight lower bounds on the convergence rate of gradient descent with robust aggregation, reflecting the same trade-off. Experiments substantiate the theory and suggest a practical recipe: use robust aggregation during most epochs to prevent crashes/restarts; near convergence, if the cluster is healthy, switch to standard averaging to further improve accuracy—accelerating training and reducing cost while preserving accuracy. Accepted as Spotlight (top 2.6% of submissions; 9.6% of accepts).

02

Title: Stochastic Layer-Wise Shuffle for Improving Vision Mamba Training

Authors: Zizheng Huang, Haoxing Chen, Jiaqi Li, Jun Lan, Huijia Zhu, Weiqiang Wang, Limin Wang

Affiliations: Nanjing University; Shanghai Institute of Advanced Innovation; China Mobile Research Institute; Shanghai AI Laboratory

Link: https://arxiv.org/abs/2408.17081

Abstract:

Vision Mamba (Vim) offers near-linear computational complexity and strong potential for high-resolution images and long videos, but training—especially at large scales—often suffers from overfitting and complicated pipelines, leaving a gap to leading ViT models on standard benchmarks. This paper proposes Stochastic Layer-Wise Shuffle (SLWS), a plug-and-play regularization method that randomly shuffles each layer’s input token sequence during training with a probability increasing linearly with depth, and restores the original order at output. SLWS encourages deep layers to learn position-invariant high-level semantics, while shallow layers remain sensitive to low-level positional cues. The induced shuffling increases task difficulty as a regularizer, mitigating overfitting. SLWS requires no architectural changes and incurs zero inference overhead. It stabilizes training of large Vim models and yields consistent gains under supervised training. With CLIP-feature–guided masked feature distillation pretraining, Vim-Huge achieves 87.6% fine-tuning accuracy on ImageNet-1K, establishing a new SOTA for Vision Mamba training.

03

Title: Elucidating the Design Space of Multimodal Protein Language Models (ICML Spotlight)

Authors: Xinyou Wang*, Cheng-Yen Hsieh*, Daiheng Zhang, Dongyu Xue, Fei Ye, Shujian Huang, Zaixiang Zheng, Quanquan Gu

Affiliations: Nanjing University; Rutgers University; ByteDance

Link: https://arxiv.org/abs/2504.11454

Abstract:

Proteins are biological macromolecules whose amino-acid sequences fold into specific 3D structures. AI-driven protein modeling and design is a key direction in AI for Science. Following the 2024 Nobel Prize in Chemistry recognizing DeepMind’s AlphaFold for solving the long-standing protein folding problem, AI methods are increasingly used in antibody design, enzyme engineering, and therapeutics. Protein sequences share structural similarity with natural language. Building on this insight, NJU’s NLP group and ByteDance Research have explored generative protein modeling, including DPLM (a general diffusion protein language model, ICML 2024) and DPLM-2 (a multimodal protein base model, ICLR 2025). This work advances that line of research. Code: https://github.com/bytedance/dplm; Project: https://bytedance.github.io/dplm/.

Multimodal Protein Language Models (PLMs) jointly model and generate protein sequences and structures. Sequences are modeled with discrete diffusion over amino-acid tokens (as in DPLM). Structures are continuous 3D coordinates that must be discretized into structure tokens for joint modeling. We identify three challenges: (1) discretizing coordinates causes information loss and harms fine-grained structural fidelity; (2) discrete structure tokens under-capture intrinsic correlations of local structure; and (3) insufficient geometric modeling hinders accurate capture of complex 3D residue interactions.

We address these by introducing a more precise generative modeling scheme tailored for protein structures to improve prediction accuracy, and by adding explicit geometric supervision via a geometric module with representation alignment to enhance geometric relational modeling. Experiments show strong gains: RMSD on folding drops from 5.52 to 2.36, comparable to ESMFold; in unconditional protein generation, sampling diversity improves by ~30% while maintaining sample quality.

04

Title: Differentiable Solver Search for Fast Diffusion Sampling

Authors: Shuai Wang, Zexian Li, Qipeng Zhang, Tianhui Song, Xubin Li, Tiezheng Ge, Bo Zheng, Limin Wang

Affiliations: Nanjing University; Alibaba

Link: https://arxiv.org/abs/2505.21114

Abstract:

Diffusion models deliver excellent generation quality but with substantial inference cost. Recent ODE-based advanced solvers target lower compute under few sampling steps, yet many are inspired by Adams-type linear multistep methods and rely solely on time-dependent Lagrange interpolation—which may be suboptimal for diffusion dynamics. This paper reveals a compact solver-design search space over time steps and solver coefficients, and proposes a differentiable solver search algorithm to discover superior solvers.

With the searched solvers, FlowMatching models SiT-XL/2 and FlowDCN-XL/2 achieve FID 2.40 and 2.35 on ImageNet 256×256 with only 10 steps; the DDPM model DiT-XL/2 reaches FID 2.33 in 10 steps. The discovered solvers substantially outperform traditional solvers (and even some distillation methods) and generalize across architectures, resolutions, and model scales.

On the tension between Byzantine robustness and no-attack accuracy in distributed learning

Fri, 18 Jul 2025 00:00:00 +0000

Stochastic layer-wise shuffle for improving Vision Mamba training

Fri, 18 Jul 2025 00:00:00 +0000

12 Papers from Nanjing University’s Large Model Center Accepted by CVPR 2025

Wed, 30 Apr 2025 00:00:00 +0000

CVPR (the IEEE/CVF Conference on Computer Vision and Pattern Recognition) is one of the world’s most influential annual academic conferences, covering cutting-edge research in computer vision, pattern recognition, and related fields. Each year it gathers top researchers, scholars, and industry professionals to discuss the latest technological advances and innovative applications. Topics range from image processing and machine learning to 3-D reconstruction and video analysis. All submissions undergo a rigorous peer-review process to ensure originality and academic value. In the 2024 Google Scholar Metrics, CVPR ranked second among all journals and conferences worldwide, just behind Nature.

The Large Model Center of the School of Computer Science at Nanjing University has had 12 papers accepted by CVPR 2025.

01

Title: UniAP: Unifying Inter- and Intra-Layer Automatic Parallelism by Mixed Integer Quadratic Programming

Authors: Hao Lin, Ke Wu, Jie Li, Jun Li, Wu-Jun Li

Affiliation: Nanjing University

Link: https://arxiv.org/abs/2307.16375

Abstract: Training large models usually demands multi-node, multi-GPU distributed setups. Even with ample hardware, 64 %–87 % of users (in our experiments) fail to obtain results because of sub-optimal hyper-parameters such as how the model and data are partitioned. Moreover, slow training is often tackled by adding GPUs while ignoring the decisive role of distributed algorithms in hardware utilization. Efficient algorithms deliver several-fold speed-ups—and cost cuts—over less efficient ones. Many existing strategies are inefficient and can even slow training as GPU count rises. We present UniAP, the first method to jointly optimize intra-layer (e.g., tensor parallelism) and inter-layer (e.g., pipeline parallelism) strategies via automatic search, together with a supporting platform. Given a model and hardware profile, UniAP automatically finds a high-performance scheme, achieving up to 3.8 × speed-up over the best prior work and up to 9 × over unoptimized baselines, while preventing the hyper-parameter mistakes that often cripple runs. UniAP has also been adapted to domestic AI accelerators. The paper was accepted as an Oral (0.7 % of submissions, 3.3 % of accepted papers) at CVPR 2025.

02

Title: Balanced Direction from Multifarious Choices: Arithmetic Meta-Learning for Domain Generalization

Authors: Xiran Wang, Jian Zhang, Lei Qi, Yinghuan Shi

Affiliation: Nanjing University; Southeast University

Link: https://arxiv.org/abs/2503.18987

Abstract: Domain generalization tackles distribution shifts between source (training) and unseen target (test) domains. First-order meta-learning based on gradient alignment finds balanced parameters across multiple sources, mitigating over-fitting. We reveal that gradient-aligned paths are not unique and that existing methods explore only one. Furthermore, they focus on directional alignment but ignore where in parameter space the model converges; ideally, the solution should lie near the centroid of each source optimum. We propose Arithmetic Meta-Learning (Arith), which introduces parameter averaging into meta-learning and designs an arithmetic-gradient optimizer that approximates the centroid while preserving gradient direction. Arith needs no extra expert networks or explicit regularizers and achieves strong generalization across benchmarks.

03

Title: Steady Progress Beats Stagnation: Mutual Aid of Foundation and Conventional Models in Mixed-Domain Semi-Supervised Medical Image Segmentation

Authors: Qinghe Ma, Jian Zhang, Zekun Li, Qian Yu, Lei Qi, Yinghuan Shi

Affiliation: Nanjing University; Southeast University

Link: https://arxiv.org/abs/2503.16997

Abstract: Large-scale pretrained vision foundation models show impressive generality, yet their rich priors can be a double-edged sword when adapted to specialized tasks. In medical-image segmentation with domain mismatch, foundation models such as MedSAM often yield over-confident but erroneous predictions, hampering leverage of unlabeled data. We introduce SynFoC, a framework that co-trains a foundation model with a from-scratch conventional model. The latter corrects high-confidence errors of the former, while the former supplies high-quality pseudo-labels early on. A Self-Mutual Confidence (SMC) module assesses pseudo-label quality and adaptively fuses them; a consensus–disagreement consistency constraint further boosts collaboration. Experiments confirm superior performance over existing approaches.

04

Title: Taste More, Taste Better: Diverse Data and Strong Model Boost Semi-Supervised Crowd Counting

Authors: Maochen Yang, Zekun Li, Jian Zhang, Lei Qi, Yinghuan Shi

Affiliation: Nanjing University; Southeast University

Link: https://arxiv.org/abs/2503.17984

Abstract: Crowd counting is vital in smart-city and public-safety applications, yet dense annotation is costly. Semi-supervised counting aims to exploit unlabeled data, but effective use remains challenging. We propose TMTB (Taste More Taste Better), advancing both data and model aspects. (1) Inpainting Augmentation uses diffusion models to regenerate image backgrounds without disturbing crowd structures, greatly enriching data diversity; unreliable regions are filtered. (2) Visual State Space Model (VSSM) serves as the backbone, capturing global context with linear complexity—ideal for extreme density, low light, or bad weather. (3) A noise-robust classification head supplies coarse-but-stable interval-count supervision, mitigating regression sensitivity to label noise. On multiple datasets, TMTB outperforms state-of-the-art methods under 5 %, 10 %, and 40 % label fractions; on JHU-Crowd++ with only 5 % labels it lowers MAE below 70 for the first time (67.0) and shows strong cross-domain generalization.

05

Title: AutoLUT: LUT-Based Image Super-Resolution with Automatic Sampling and Adaptive Residual Learning

Authors: Yuheng Xu, Shijie Yang, Xin Liu, Jie Liu, Jie Tang, Gangshan Wu

Affiliation: Nanjing University

Link: https://arxiv.org/abs/2503.01565

Abstract: The spread of high-DPI displays heightens demand for high-def images, yet edge devices struggle to host heavy SR networks, calling for efficiency. Prior LUT-based SR has scarcely mined pixel-level cues and uses fixed sampling, limiting accuracy and fine-detail capture. We introduce two plug-and-play modules: AutoSample, which learns flexible LUT sampling weights during training—adapting to pixel variations, enlarging receptive field, and incurring no inference overhead—and AdaRL, which strengthens inter-layer connections to boost fine-detail reconstruction. With similar storage, AutoLUT lifts MuLUT by ≈ 0.20 dB PSNR across five datasets; on SPF-LUT it halves storage, cuts inference time by two-thirds, and maintains fidelity.

06

Title: CATANet: Efficient Content-Aware Token Aggregation for Lightweight Image Super-Resolution

Authors: Xin Liu, Jie Liu, Jie Tang, Gangshan Wu

Affiliation: Nanjing University

Link: https://arxiv.org/abs/2503.06896

Abstract: Transformer-based SR excels on low-level vision but its quadratic complexity explodes with resolution. Existing speed-ups partition images into content-agnostic windows, curtailing long-range redundancy exploitation vital for SR. We propose CATANet, a lightweight Content-Aware Token Aggregation Network. A novel aggregation module clusters content-similar tokens across the entire image, sharing aggregation centers and updating them only during training to cut computation. We then apply intra-group self-attention for long-range interaction and inter-group cross-attention to enhance global fusion. Compared with the clustering-based SPIN, CATANet is faster at inference while gaining up to 0.33 dB PSNR.

07

Title: Tra-MoE: Learning Trajectory Prediction Model from Multiple Domains for Adaptive Policy Conditioning

Authors: Jiange Yang, Haoyi Zhu, Yating Wang, Gangshan Wu, Tong He, Limin Wang

Affiliation: Nanjing University; Shanghai AI Lab; USTC; Tongji University

Link: https://arxiv.org/pdf/2411.14519

Abstract: Data scarcity and heterogeneity challenge robot learning. Tra-MoE adopts a sparsely gated Mixture-of-Experts to learn trajectory prediction from large-scale cross-domain video without action labels, balancing parameter sharing and specialization. It fuses simulation videos rendered by different physics engines with real videos of humans, single-arm, and dual-arm robots—promising for cross-agent learning. An adaptive policy-conditioning mechanism leverages predicted trajectories to boost downstream robot control, greatly reducing needs for expensive real-robot data.

08

Title: LeviTor: 3-D Trajectory Oriented Image-to-Video Synthesis

Authors: Hanlin Wang, Hao Ouyang, Qiuyu Wang, Wen Wang, Ka Leong Cheng, Qifeng Chen, Yujun Shen, Limin Wang

Affiliation: Nanjing University; Ant Group; Zhejiang University; Hong Kong University of Science and Technology; Shanghai AI Lab

Link: https://github.com/ant-research/LeviTor

Abstract: Sketching a trajectory is an intuitive way to control motion in image-to-video synthesis, yet 2-D paths are ambiguous for out-of-plane motion. LeviTor enriches interaction by adding a depth dimension: users assign relative depth to trajectory key-points, retaining 2-D convenience while enabling 3-D control. Objects are represented by a few cluster points reflecting depth and occlusion. These, along with depth and instance maps, guide a video-diffusion generator to produce videos faithfully following 3-D trajectories. Extensive experiments demonstrate precise motion control and high realism.

09

Title: Contextual AD Narration with Interleaved Multimodal Sequence

Authors: Hanlin Wang, Zhan Tong, Kecheng Zheng, Yujun Shen, Limin Wang

Affiliation: Nanjing University; KU Leuven; Ant Group; Shanghai AI Lab

Link: https://arxiv.org/abs/2403.12922

Abstract: Audio description (AD) narrates visual content for the visually impaired. We present Uni-AD, a simple unified framework that feeds interleaved multimodal sequences—video features, text, character lists, and context—into a pretrained language model. A lightweight mapper aligns video to text space for fine-grained fusion; a character-optimization module highlights major roles in context. Coupled with context cues and a contrastive loss, Uni-AD generates fluent, context-aware narration. Experiments on multiple AD datasets confirm its superiority.

10

Title: Multiple Object Tracking as ID Prediction

Authors: Ruopeng Gao, Ji Qi, Limin Wang

Affiliation: Nanjing University; China Mobile (Jiangsu) Software Technology Co.; Shanghai AI Lab

Link: https://github.com/MCG-NJU/MOTIP

Abstract: Multi-object tracking (MOT) is traditionally decomposed into detection and association, with handcrafted algorithms maintaining trajectories and computing cost matrices—effective yet requiring extensive tuning for complex scenes. We reconceptualize MOT as context-conditioned ID prediction and propose MOTIP, an end-to-end framework that directly decodes ID labels for current detections given past trajectories. Using only appearance features, MOTIP achieves state-of-the-art results on multiple benchmarks without elaborate tricks, offering a powerful baseline for future research.

11

Title: Online Video Understanding: OVBench and VideoChat-Online

Authors: Zhenpeng Huang, Xinhao Li, Jiaqi Li, Jing Wang, Xiangyu Zeng, Cheng Liang, Tao Wu, Xi Chen, Liang Li, Limin Wang

Project Site: https://videochat-online.github.io/

Affiliation: Nanjing University; China Mobile Research Institute; Shanghai AI Lab

Abstract: Multimodal large language models have excelled at offline video understanding, but real-time scenarios (e.g., autonomous driving, HCI) pose fresh challenges. We contribute on three fronts: (1) OVBench, a comprehensive QA benchmark evaluating perception, memory, and reasoning over streaming video, spanning six task types across past, current, and future contexts (16 subtasks from diverse datasets). (2) Pyramid Memory Bank, which efficiently retains critical spatio-temporal cues. (3) An offline-to-online learning paradigm, with an alternating dialog format and the VideoChatOnline-IT instruction-tuning set for streaming data. Our resulting framework, VideoChat-Online, outperforms state-of-the-art offline and online models on common offline benchmarks and OVBench, despite lower compute cost.

12

Title: Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment

Authors: Zi’ang Yan, Zhilin Li, Yinan He, Chenting Wang, Kunchang Li, Xinhao Li, Xiangyu Zeng, Zilei Wang, Yali Wang, Yu Qiao, Limin Wang, Yi Wang

Affiliation: Shanghai AI Lab; Zhejiang University; University of Science and Technology of China; Shanghai Jiao Tong University; Shenzhen Institutes of Advanced Technology, CAS; Nanjing University

Abstract: Although multimodal LLMs excel at broad visual reasoning, they lag on fine-grained or high-precision tasks. Prior efforts either add tool-usage skills or fold specific vision tasks into the autoregressive framework, often harming overall multimodal performance. We propose Task Preference Optimization (TPO), which introduces differentiable task preferences distilled from fine-grained vision tasks to guide optimization. Learnable task tokens form dynamic links between multiple task-specific heads and the core MLLM, enabling effective use of rich labeled data. TPO supports joint multi-task training, boosting overall performance by 14.6 % versus baselines and delivering strong zero-shot generalization comparable to fully-supervised state-of-the-art models. We instantiate TPO on VideoChat and LLaVA, confirming significant gains and opening a scalable pathway to enhance MLLMs on diverse visual tasks.

Read Original

AutoLUT: LUT-based image super-resolution with automatic sampling and adaptive residual learning

Sun, 20 Apr 2025 00:00:00 +0000

Balanced direction from multifarious choices: arithmetic meta-learning for domain generalization

Sun, 20 Apr 2025 00:00:00 +0000

CATANet: efficient content-aware token aggregation for lightweight image super-resolution

Sun, 20 Apr 2025 00:00:00 +0000

Contextual AD narration with interleaved multimodal sequence

Sun, 20 Apr 2025 00:00:00 +0000

LeviTor: 3D trajectory oriented image-to-video synthesis

Sun, 20 Apr 2025 00:00:00 +0000

Multiple object tracking as id prediction

Sun, 20 Apr 2025 00:00:00 +0000

Online video understanding: a comprehensive benchmark and memory-augmented method

Sun, 20 Apr 2025 00:00:00 +0000

Steady progress beats stagnation: mutual aid of foundation and conventional models in mixed domain semi-supervised medical image segmentation

Sun, 20 Apr 2025 00:00:00 +0000

Task preference optimization: improving multimodal large language models with vision task alignment

Sun, 20 Apr 2025 00:00:00 +0000

Taste more, taste better: diverse data and strong model boost semi-supervised crowd counting

Sun, 20 Apr 2025 00:00:00 +0000

Tra-MoE: learning trajectory prediction model from multiple domains for adaptive policy conditioning

Sun, 20 Apr 2025 00:00:00 +0000

Uniap: unifying inter-and intra-layer automatic parallelism by mixed integer quadratic programming

Sun, 20 Apr 2025 00:00:00 +0000

CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding

Tue, 15 Apr 2025 00:00:00 +0000

Five Papers from Nanjing University’s School of Computer Science Large Model Innovation Center Accepted at ICLR 2025

Tue, 15 Apr 2025 00:00:00 +0000

ICLR (International Conference on Learning Representations) is one of the leading AI conferences focusing on deep learning and representation learning. Since its inception in 2013, ICLR has become a premier platform for machine learning research, particularly in deep learning, neural architectures, reinforcement learning, generative models, and NLP.

Five papers from the Large Model Innovation Center of Nanjing University’s School of Computer Science were accepted at ICLR 2025.

01

Title: TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning
Authors: Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue, Yi Wang, Yali Wang, Yu Qiao, Limin Wang
Affiliations: Nanjing University, Shanghai AI Laboratory, Chinese Academy of Sciences, etc.
Link: https://openreview.net/forum?id=nAVejJURqZ
Abstract: Most existing video multimodal large models tend to focus on irrelevant segments when understanding long videos, often leading to hallucinations. Can we enhance MLLMs’ long-video QA performance by using temporal localization as an auxiliary task to pinpoint relevant subsegments? We propose TimeSuite, which incrementally fine-tunes short-video MLLMs with time-location data to boost long-video understanding. TimeSuite includes: a simple, efficient long-video framework (VideoChat‑T); a high‑quality localization‑based instruction tuning dataset (TimePro); and a tailored instruction task (Temporal Grounded Caption). Joint tuning guides MLLMs to focus on correct segments, improving QA accuracy. First, VideoChat‑T achieves expert‑level temporal localization without external decoders while retaining strong QA generalization and zero‑shot ability. Second, integrating the expert task enhances comprehensive long‑video understanding, validating this hybrid approach. Experiments show VideoChat‑T yields 5.6% and 6.8% accuracy gains on Egoschema and VideoMME, respectively, and demonstrates superior zero‑shot localization, matching supervised expert models after fine‑tuning.

02

Title: CG-Bench: Clue‑grounded Question Answering Benchmark for Long Video Understanding
Authors: Guo Chen, Yicheng Liu, Yifei Huang, Yuping He, Baoqi Pei, Jilan Xu, Yali Wang, Tong Lu, Limin Wang
Affiliations: Nanjing University, Shanghai AI Laboratory, Fudan University, Zhejiang University
Link: https://openreview.net/forum?id=le4IoZZHy1
Abstract: We introduce CG‑Bench, a benchmark for long‑video multimodal reasoning using a “Clue‑Question‑Answer” triplet. Unlike multiple‑choice tests, models must answer correctly and accurately locate supporting video segments. CG‑Bench offers three tasks: perception (basic visual skills), reasoning (temporal & multimodal integration), and hallucination detection (reliability under ambiguity). It uses dual evaluation: white‑box IoU for localization precision and black‑box Clue Recovery Rate for context dilution. Combining multiple‑choice and open‑ended forms with human annotations and heuristic rules, CG‑Bench ensures evaluation quality. The dataset contains 1,219 long videos across 638 subcategories, totaling 12,129 QA pairs. Results show models like GPT‑4o perform well on multiple choice but drop sharply when localization is required (white‑box acc@IoU only 4.38%, open‑ended accuracy <40%). Performance varies with video length, frame sampling, and multimodal cues, highlighting challenges in precise information retrieval for long‑video reasoning.

03

Title: SPA: 3D Spatial‑Awareness Enables Effective Embodied Representation
Authors: Haoyi Zhu, Honghui Yang, Yating Wang, Jiange Yang, Limin Wang, Tong He
Affiliations: University of Science and Technology of China, Shanghai AI Laboratory, Zhejiang University, Tongji University, Nanjing University
Link: https://openreview.net/forum?id=6TLdqAZgzn
Abstract: Spatial awareness is critical for robots in complex environments, but existing methods struggle to capture 3D geometry. We propose SPA, a visual representation framework that enhances 3D spatial awareness for embodied tasks. SPA trains on a large multi‑view dataset with camera poses, depth, and semantic maps from synthetic and real robot scenes. It builds volumetric features from multi‑view input, uses mask‑based differentiable neural rendering to generate RGB, depth, and semantic maps, and applies Eikonal regularization with SDF supervision for geometric consistency. After 6,000 GPU hours, SPA outperforms baselines on 200+ tasks across real and eight simulated environments, ranking first in 30.3% of tasks.

04

Title: Proving Olympiad Inequalities by Synergizing LLMs and Symbolic Reasoning
Authors: Zenan Li, Zhaoyu Li, Wen Tang, Xian Zhang, Yuan Yao, Xujie Si, Fan Yang, Kaiyu Yang, Xiaoxing Ma
Affiliations: Nanjing University, University of Toronto, Microsoft Research Asia, Peking University, Meta
Link: https://openreview.net/forum?id=FiyS0ecSm0
Abstract: AI has advanced in competition‑level proofs, especially inequalities, which pose huge search spaces at each step. We present a neural‑symbolic system that integrates neural networks with symbolic reasoning, excelling on Olympiad‑level inequality tasks. On a standard set of 20 problems, our system solves 16 on average (versus 15 by human gold medalists), outperforming GPT and DeepSeek. This breakthrough showcases neural‑symbolic methods’ potential for complex mathematical reasoning, opening new avenues in automated theorem proving, education, and research.

05

Title: MeteoRA: Multiple‑tasks Embedded LoRA for Large Language Models
Authors: Jingwei Xu, Junyu Lai, Yunpeng Huang
Affiliations: Nanjing University
Link: https://openreview.net/pdf?id=yOOJwR15xg
Abstract: The “pretrain + finetune” paradigm underpins LLM deployment, with LoRA as a popular efficient fine‑tuning method. Yet task awareness and adapter switching remain challenging with multiple LoRA adapters. We propose MeteoRA, a scalable multi‑task LoRA architecture embedding task‑specific adapters and a routing component via a Mixture‑of‑Experts (MoE) design for adaptive adapter selection. A hybrid expert model acceleration strategy leverages PyTorch and Triton–based custom operators to avoid MoE routing loops, achieving 4× speedup. Experiments demonstrate MeteoRA’s effectiveness on composite tasks, handling up to ten serial questions per inference and showing clear routing biases, confirming adaptive switching.

View original

MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models

Tue, 15 Apr 2025 00:00:00 +0000

Proving Olympiad Inequalities by Synergizing LLMs and Symbolic Reasoning

Tue, 15 Apr 2025 00:00:00 +0000

SPA: 3D Spatial-Awareness Enables Effective Embodied Representation

Tue, 15 Apr 2025 00:00:00 +0000

TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning

Tue, 15 Apr 2025 00:00:00 +0000

Shusheng InternVideo2.5 Open-Sourced, Precisely Finding the 'Needle in a Haystack' in Tens of Thousands of Frames, with Fine-Grained Spatiotemporal Perception

Tue, 11 Feb 2025 00:00:00 +0000

Recently, the Shanghai AI Lab, in collaboration with Nanjing University and the Shenzhen Institutes of Advanced Technology, jointly open-sourced the multi-modal video model Shusheng InternVideo2.5. In the field of video understanding, the upgraded InternVideo2.5 has achieved improvements in both temporal span and fine granularity, expanding its capacity sixfold compared to the previous model. It enables a precise “needle in a haystack” search within long videos containing tens of thousands of frames, allowing AI to more accurately interpret the complex real world and infuse new quality into various applications. Previously, the Shusheng InternVideo series was applied during the live broadcast of the Paris Olympics by China Central Television, precisely pinpointing athletes’ scoring moments and corresponding slow-motion replays, significantly enhancing TV production efficiency. With enhanced long video processing capabilities, InternVideo2.5 will offer more efficient AI support for applications such as autonomous driving, security surveillance, and virtual reality.

Open source link: https://github.com/OpenGVLab/InternVideo/tree/main/InternVideo2.5
Paper link: https://arxiv.org/abs/2501.12386
Huggingface link: https://huggingface.co/OpenGVLab/InternVideo2_5_Chat_8B

Focus on Fine-Grained Spatiotemporal Understanding and Efficient Long Video Processing

Shanghai AI Lab has continuously invested in video multi-modal large model (Video MLLM) technology since 2022, successively launching and open-sourcing the general video foundation model Shusheng InternVideo, the video understanding large model Shusheng InternVideo2, and the dialogue-centric video understanding paradigm VideoChat. By leveraging its experience in video visual representation learning and multi-modal dialogue, the upgraded InternVideo2.5 focuses on fine spatiotemporal understanding through deep integration of visual perception and language comprehension, achieving breakthroughs in long video understanding.

InternVideo2.5 Capability Characteristics:

Ultra-long video processing: Accurately locate targets within tens of thousands of frames, with processing length extended from 3,000 to 10,000 frames.
Fine-grained perception: Accurately identify and locate objects, scenes, and actions while comprehending subtle spatiotemporal relationships.
Integration of multiple visual capabilities: Not only supports general video Q&A but also proficiently handles specialized tasks such as object tracking and segmentation.

Left image: Performance comparison between InternVideo2.5 and other 8-billion-parameter open models on MVBench and VideoMME; Right image: InternVideo2.5 accurately tracks and analyzes videos.

LRC Combined with Progressive Training to Overcome Bottlenecks in Long Video Modeling

For long videos and fine-grained visual tasks, traditional video multi-modal large models face significant challenges in accurately tracking target objects in ultra-long videos or recognizing subtle spatiotemporal relationships in complex scenes. For example, in “needle in a haystack” tasks, conventional methods require extensive computational resources and deliver unsatisfactory localization accuracy, thereby limiting industrial applications. To address this, Shanghai AI Lab, together with its research team, leveraged its self-developed Shusheng InternVL2.5 base model to propose Long-range Context Modeling (LRC) technology as a solution.

The Two Core Modules of Long-range Context Modeling (LRC) Technology:

Hierarchical Context Compression (HiCo): Exploits redundancy in long video visual data through layered compression. Experimental results demonstrate that with HiCo, InternVideo2.5 can accurately locate target frames within tens of thousands of frames, leading in performance among open models.
Task Preference Optimization (TPO): Transforms annotations from various fine-grained visual tasks (such as object tracking, segmentation, and temporal localization) into differentiable task preferences, thereby guiding the model’s self-learning to extend its capabilities to specialized visual applications.

Additionally, the team pre-trained InternVideo2.5 using a progressive multi-stage training strategy on over 300,000 hours of video data, ensuring robust video processing capabilities. The training corpus includes vision-language alignment data, long video sequences, and specialized visual task data, providing abundant information for comprehensive model learning. Following the progressive training scheme of Shusheng InternVL, the approach enhances fine-grained perception and temporal understanding in stages: initial basic learning for task recognition and video-language alignment; subsequent integration and training of specific task components alongside visual concept pre-training; and finally, multi-task training combined with instruction fine-tuning on mixed corpora to optimize all model components. This method achieves effective scaling from “small to large” and refinement of data from “coarse to fine”, reducing costs while enhancing performance.

View Original

The Chinese Academy of Sciences Academicians Forum on the Healthy Development and Empowerment of Large Models/AIGC Held in Nanjing

Tue, 16 Jan 2024 00:00:00 +0000

The 155th Frontier Forum of the Chinese Academy of Sciences Academicians — “The Healthy Development and Empowerment of Large Models/AIGC” was held in Nanjing from January 6 to 7, 2024. The forum was organized by the Chinese Academy of Sciences Academicians, hosted by the Academic and Publishing Work Committee and the Standing Committee of the Information Technology Science Department of the Chinese Academy of Sciences, co-organized by Nanjing University, Southeast University, and the publisher “Science in China”, with Academicians Lu Jian and Huang Ru, along with Academician Wang Jian from the Chinese Academy of Engineering, jointly serving as forum chairs.

Academician Bao Xinhai, Director of the Academic and Publishing Work Committee, attended the forum along with Zhou Dejin from the Work Bureau of the Chinese Academy of Sciences Academicians, Ren Youqun from the Ministry of Education’s Teacher Work Department, Academician Huang Ru from Southeast University, and Xu Guanghui from the Jiangsu Science and Technology Department, who delivered opening remarks.

Six academicians from the Chinese Academy of Sciences—including Bao Xinhai, Lu Jian, Huang Ru, Tan Tieniu, E Weinan, and Xu Zongben—two academicians from the Chinese Academy of Engineering, Gao Wen and Yang Shanlin, and nearly 300 experts from 87 universities, research institutes, and companies (including the Chinese Academy of Sciences, Nanjing University, Southeast University, Hong Kong University of Science and Technology, iFlytek, Huawei, Alibaba, Xiaomi, Midea, and Geely Automobile Research Institute) attended the forum, with more than half being young scientists under 45.

The forum comprised two sessions: keynote presentations and special topic reports. In the keynote session, Academician Tan Tieniu discussed trends in generative AI; Academician Gao Wen introduced the Pengcheng Brain pre-trained large model platform and open-source collaborations; Academician Yang Shanlin presented AIGC and its scientific foundations; Academician E Weinan explained the basics of deep learning; Academician Xu Zongben discussed mathematical research on large models; Professor Guo Yike, an Academician of the Royal Academy of Engineering (UK) and Vice-Chancellor of Hong Kong University of Science and Technology, addressed the intrinsic scientific issues of large models; and AI experts from iFlytek, Huawei, and Alibaba showcased applications and innovative practices of large models.

In the special topic session, experts presented reports on eight topics: “Frontier and Collaborative Innovation in the Development of Large Models/AIGC”, “Empowering Technological Development with Large Models/AIGC”, “Boosting the Real Economy with Large Models/AIGC”, “Facilitating Educational Transformation with Large Models/AIGC”, “Large Models/AIGC and Intelligent Basic Software”, “Large Models/AIGC, Computing Infrastructure, and Chip Technology”, “Safety, Controllability, Privacy Protection, and Low-cost Deployment of Large Models/AIGC”, and “Governance and Management of Large Models/AIGC”. Following the reports, experts engaged in roundtable discussions on these topics.

After two days of discussions, the experts explored key technologies and challenges in the development of large models and AI, application scenarios, industrial empowerment, and legal and ethical risks, reaching some preliminary consensus. The forum outcomes will be released in the form of briefings and special reports.

Read Original Article

Rong Gu

Tue, 01 Jan 1050 00:00:00 +0000

Rong Gu is currently a distinguished researcher in the School of Computer Science at Nanjing University, PhD supervisor, Young Yangtze River Scholar of the Ministry of Education, and DAMO Academy Young Fellow Award winner (2023). He serves as the community chair of the Fluid open source project (CNCF Sandbox project under Linux Foundation), executive committee member of ACM ChinaSys, and executive committee member of multiple CCF technical committees including Distributed Computing and Systems, Big Data Expert Committee, System Software Committee, Database Committee, and Open Source Development Committee.

Research Areas: His main research fields include cloud computing and big data systems, intelligent computing systems. Currently focusing on LLM inference and training systems, cloud-native computing systems, intelligent data management, etc. He has published over 70 papers in top-tier international academic journals and conferences including USENIX ATC, EuroSys, SIGMOD, VLDB, ICDE, KDD, WWW, INFOCOM, VLDBJ, IEEE TPDS, TON, TKDE, published 3 academic monographs, and holds 18 authorized invention patents.

Research Projects: He leads National Natural Science Foundation projects (Youth/General), sub-projects of National Key R&D Program, China Postdoctoral Science Foundation Special Grant, as well as industry collaboration projects with Huawei/Alibaba/Tencent/Ant Group/China Mobile/Sinopec. His research results have been adopted by leading enterprises and internationally renowned open source systems including Apache Spark and Alluxio, and he initiated the Fluid open source project under Cloud Native Computing Foundation.

Awards and Honors: He has received Jiangsu Province Science and Technology First Prize, Alibaba DAMO Academy Young Fellow Award (2023), IEEE TCSC Award for Excellence in Scalable Computing (Early Career, 2022, 5 recipients globally per year), CCF Distributed Computing and Systems Committee Young Innovation Pioneer (2 recipients nationally per year), IEEE HPCC Best Paper Award, CCF Big Data Conference Best Application Paper Award, Alibaba Outstanding Academic Collaboration Project Award, Huawei “Challenge Problems” Spark Award, Tencent Cloud Most Valuable Professional Award, ZTE Industry-University-Research Outstanding Collaboration Project Award, China Open Source Innovation Competition First Prize, Nanjing University May 4th Youth Medal, Jinling Young Scholar, CloudSort Track Champion in International Computer Sorting Benchmark Competition, ACM Nanjing Chapter Academic Rising Star Award, Jiangsu Computer Society Young Scientist Award/Outstanding Science and Technology Worker, China Academy of Information and Communications Technology OSCAR Peak Open Source Figure.

Large Language Model Research Group

Fri, 01 Jan 1030 00:00:00 +0000

Large Language Models: A New Era of Artificial Intelligence in Language Understanding and Generation

Large Language Models (LLMs) are one of the most groundbreaking technologies in the field of artificial intelligence in recent years. Trained on massive datasets, they are capable of understanding and generating natural language, demonstrating near-human-level performance in tasks such as text generation, translation, and question answering. This marks the dawn of a new era in artificial intelligence for language understanding and generation.

Core Technological Breakthroughs

The success of large language models is attributed to breakthroughs in the following key technologies:

In summary, large language models are profoundly transforming how we interact with machines and bringing unprecedented opportunities to various industries. With continuous technological advancements and expanding application scenarios, they will continue to drive progress in the field of artificial intelligence for language understanding and generation, creating more value for human society.

Multimodal Large Model Research Group

Sat, 01 Jan 1020 00:00:00 +0000

Multimodal Large Model Research Group is committed to promoting the research and application of multimodal large models, exploring key technologies in multimodal information fusion, interaction, and reasoning, and driving the application of multimodal large models in visual, speech, text, and other multimodal data, thereby providing technical support for the development of multimodal intelligent technologies.

Embodied Decision Large Model Research Group

Wed, 01 Jan 1017 00:00:00 +0000

Embodied Decision Large Model Research Group focuses on cutting-edge research in embodied intelligence, aiming to build a generalizable embodied agent through study in representation learning, policy learning, and hierarchical planning and execution.

Large Model Knowledge Enhancement Research Group

Sun, 01 Jan 1015 00:00:00 +0000

The LLM + Knowledge research group has been engaged in long-term research on large language model knowledge enhancement, controllable generation, and domain-specific construction.

Large Model Learning Algorithms and Platform Research Group

Mon, 01 Jan 1010 00:00:00 +0000

The Large Model Learning Algorithms and Platform Research Group focuses on the construction of systems based on large models, large-scale training/inference deployment, and the application of large models. The group conducts research to address key challenges in efficient training, deployment, and the integration of domain knowledge into large models. In terms of applications, the group has a strong focus on reasoning tasks such as Automated Theorem Proving (ATP). In undergraduate education, the group offers courses on large model development, training students to build large models from scratch.

Large Model Systems and Platforms Research Group

Mon, 01 Jan 1010 00:00:00 +0000

## Large Model Systems and Platforms: The Core Engine Driving the Scalable Application of Artificial Intelligence

With the rapid development of large model technology, efficiently training, deploying, and managing these massive models has become a critical challenge. **Large model systems and platforms** have emerged to address this need, providing the infrastructure and toolchains necessary for the development and application of large-scale artificial intelligence models. They serve as the core engine driving the scalable application of AI.

### Core Features and Capabilities

Large model systems and platforms typically offer the following core functionalities:

1. **Distributed Training**:
   - Supports distributed training for massive datasets and ultra-large models.
   - Provides efficient parallel computing and communication optimization, such as data parallelism, model parallelism, and pipeline parallelism.
   - Representative examples: Megatron-LM, DeepSpeed.

2. **Efficient Inference**:
   - Optimizes inference for large models to reduce latency and resource consumption.
   - Supports model compression, quantization, and acceleration techniques.
   - Representative examples: TensorRT, ONNX Runtime.

3. **Model Management and Deployment**:
   - Offers version control, monitoring, and updating capabilities for models.
   - Supports deployment across multiple environments, including cloud, edge, and devices.
   - Representative examples: MLflow, Kubeflow.

4. **Developer Tools and Ecosystem**:
   - Provides user-friendly APIs, SDKs, and visualization tools.
   - Builds open developer communities and ecosystems.
   - Representative examples: Hugging Face, OpenAI API.

### Representative Platforms and Systems

The following are some notable large model systems and platforms:

- **Hugging Face**: Offers a rich collection of pre-trained models and datasets, supporting model training, fine-tuning, and deployment.
- **OpenAI API**: Provides powerful interfaces for large model services, enabling tasks like text generation and code generation.
- **DeepSpeed**: Developed by Microsoft, focuses on distributed training and optimization for large-scale models.
- **Colossal-AI**: Delivers efficient solutions for parallel training and inference, supporting ultra-large models.

### Future Development Trends

The future development of large model systems and platforms will focus on the following directions:

1. **Performance Optimization**: Further improves training and inference efficiency while reducing resource consumption.
2. **Usability Enhancement**: Simplifies development processes and lowers the barrier to entry.
3. **Ecosystem Expansion**: Builds a more open and thriving developer ecosystem.
4. **Security and Trustworthiness**: Strengthens model security and explainability to ensure reliable applications.

---

**In summary, large model systems and platforms are the critical enablers for the practical application of large model technology.** With continuous technological advancements and ecosystem improvements, they will provide stronger momentum for the scalable application of artificial intelligence, driving intelligent transformation across industries.

Scientific Large Model Research Group

Tue, 01 Jan 1005 00:00:00 +0000

The Scientific Large Model Research Group is dedicated to advancing interdisciplinary research in drug development, materials innovation, and energy optimization through state-of-the-art computational simulations.

Medical Imaging Large Model Research Group

Sun, 01 Jan 1004 00:00:00 +0000

The Nanjing University Medical Imaging Large Model Research Group has been deeply engaged in the field of intelligent medical imaging, focusing on efficient training and low-cost fine-tuning of large models to explore cutting-edge applications in medical image segmentation, auxiliary diagnosis, and precision treatment. Facing the challenge of high annotation costs, the group is dedicated to methods such as sparse supervision, efficient data utilization, and pseudo-label optimization to reduce reliance on large-scale manual annotations while enhancing model generalization and robustness.

Edge Large Model System Research Group

Sat, 01 Jan 1003 00:00:00 +0000

The Edge Large Model System Research Group focuses on frontier optimization techniques for large model systems. Centered on building a high-precision, low-latency, and scalable large model service framework, our research covers operator optimization, adaptive parameter tuning, and multimodal task scheduling.

Large Language Model System Research Group

Wed, 01 Dec 1002 00:00:00 +0000

The NASA research group of Nanjing University, in collaboration with renowned institutions such as Pengcheng Laboratory and Huawei Technologies Co., Ltd., has conducted comprehensive and in-depth research on key topics including large model training/inference performance and power consumption. The research achievements have not only been published at top conferences in the field of computer architecture, but have also been successfully deployed in relevant enterprises, making positive contributions to bridging the gap between theory and practice.

Cloud Large Model System Research Group

Mon, 01 Nov 1002 00:00:00 +0000

The Cloud Large Model System Research Group is dedicated to exploring system-level performance optimization technologies for large model training, inference, and deployment in cloud environments. The team’s key research directions include: storage management optimization for cloud-based large models, efficient distribution and loading mechanisms, training process optimization strategies, and inference performance optimization technologies.

Controllable Generation Research Group

Fri, 01 Oct 1002 00:00:00 +0000

The Controlled Generation Research Group has long been engaged in research related to the generation capabilities of large language models and multimodal models. Currently, the group focuses on controllable generation techniques for large models to enhance their outputs along specific attributes. Their research centers on methods such as intervention and guidance of large models, conditional control in multimodal large models, and neuron or activation localization within large models to steer their generation.

Chen Tian