Research Paper Collection
Sensorimotor World Models: Perception for Action via Inverse Dynamics
Abstact: Let model learn: Forward dynamics: Current state + action → next state Inverse dynamics: Current state + next state → the action taken Ex: Ro
πRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models
Abstract: Lets pi_0.5 flow matching VLAs to use PPO to do RL to improve base policy. Key Idea: Its hard to find the end action probability, but if we
Kimi K3: Open Frontier Intelligence
Kimi K3's multimodal Mixture-of-Experts MoE model. Improved transformers on sequence length, network depth, model width. Used large RL for fine tuning
Learning Robot Policies from Sparse Success Signals via STL-Guided Stein Variational Policy Gradient
Abstract: Traditional RL is hard to learn signal from sparse reward tasks. This can learn from a failed trajectory by following the robustness score a
CFG: Classifier-Free Diffusion Guidance
Abstract: Jointly train conditional and unconditional diffusion model, combnie the result of conditional and unconditional score to estimate this trad
Classifier Guidance: Diffusion Models Beat GANs on Image Synthesis
Abstract: Let diffusion model generate images, let classifier provide direction at each denoising step. Input: noise + classification: "dog" Output: g
MM1 — Methods, Analysis & Insights from Multimodal LLM Pre-training
Abstract: How to train a multimodal LLM, using large amount of ablation to find best design choices. Conclusion: Image resolution: higher image res is
OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
Curated a large scale dataset. Previously, VLM recieves image + caption They made: OBELICS wants the model to learn: relations between multiple images
DataComp — In Search of the Next Generation of Multimodal Datasets
Abstract: With a set model architecture, training, and compute cost, how do we pick the best dataset? With same compute, does more data = better or hi
Scaling Laws for Generative Mixed-Modal Language Models
Abstract: Wen we put text, image, audio, code and different modalities into the same Transformer to train, do they help each other or interfere? How b
ETL: Runtime Monitoring of Perception-Based Autonomous Systems via Embedding Temporal Logic
Abstract: Image embedding space to STL. Check if robot completed: grab cup → main grabbing → place to location.
Pretrained Embeddings as a Behavior Specification Mechanism
Abstract: Use image embedding distance as STL's temporal logic, then plan with world model. STL requires robot's actual position and target state Imag
Janus — Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
Understanding: needs what Generation: needs what + where + details Janus: SigLip encoder for input image understanding and passes into LLM VQ Tokenize
Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model
Charmeleon: text: discrete tokens, each work into an ID. Image: VQ-VAE. problem: image is quantized, lose information Transfusion: text: next token pr
Chameleon: Mixed-Modal Early-Fusion Foundation Models
Abstract: convert images into same tokens as LLMs, then use same transformer to create tokens. Input: text + image combined Output: text + image combi
Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization
Abstract Reward Weighting: generate images, give each image a reward, higher reward = higher weight for flow matching loss Online Training: use online
Priming Models: Are We Overthinking One-Step Generation?
Problem: Image generation from scratch using pure distribution level matching is too difficult. They believe its the jacobian alignment: Jacobian Alig
A Gradual, Semi-Discrete Approach to Generative Network Training via Explicit Wasserstein Minimization
Goal: match $q(c, z)$ with $p(c, z)$, the generated distribuiton with real distribution within each class. We want the same probability mass in the sa
SALMONN: Towards Generic Hearing Abilities for Large Language Models
Goal: lets a LLM understand audio data. Understand person's speech, as well as surrounding audio, music, sound, and do QA on it. Problem: prior audio
LLaVA — Visual Instruction Tuning
Goal / IO: VQA with image. Idea: Visual Instruction Tuning: creating high quality data QA training data. Problem: connecting vision encoder + LLM is n
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
Main Idea: Takes a pretrained, frozen visual encoder model and LLM model, train a Q-former model in between, allowing LLM to undersatnd and answer que
Flamingo: a Visual Language Model for Few-Shot Learning
Input: Image / Video + Text + Few-shot examples Output: Text It can do visual question answering, image captioning, video understanding. Main Idea: it
Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation
Problem: Assume we want our model to have output distribution of the real dataset distribution. However, the real distribution is separated into diffe
Implicit Maximum Likelihood Estimation
Train generator $G_\theta$ Input noise, output image x. Unconditional generation at the time. Problem: Implicit generative model can't calculate likel
Back to Basics: Let Denoising Generative Models Denoise
JiT: Just image Transformers: It is a class-conditional image generator. Core Idea: Predicting what the image should look like, is different than pred
TBSM: Three-Body Scattering for Generative Modeling
Input: random noise + condition c Output: NFE=1 step image generator $x = G_\theta(z,c)$ Problems: Diffusion / FM learns noise to data denoising / vel
Visual BPE: From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
Goal: Train a multimodal LLM, given image + "Whats happening in image", it can answer: "two cats on the ground". It can do visual QA, image understand
An Image is Worth 32 Tokens for Reconstruction and Generation
Goal: Image tokenizer turns image into 2D latent grid. TiTok thinks we don't need this 2D structure, we can just make it into 1D tokens, ex with 32 to
UniTok: A Unified Tokenizer for Visual Generation and Understanding
Problem: Image understanding: for CLIP, the semantic is strong, but its continous feature, which is hard for autoregressive generation. Image generati
dMel: Speech Tokenization made Simple
Problem: Audio is continous, LLM is best with discrete tokens. We need to convert audio → tokens. Method 1: Semantic Token HuBERT + K-means. We can se
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
Goal: unlabeled images → good visual representation Contrastive self-learning: without any image label, how to train an image encoder. Representation
Reinforcement Learning for Flow-Matching Policies with Density Transport (RLDT)
Idea: Given a BC flow-matching poilcy: $\pi_\theta(a|s)$ It can complete task, but is sentitive to enviornment change. We want to improve the policy w
Temporal-Distance-JEPA: Plan-Aware Representation Learning for Latent World Model Predictive Control
不要默认latent space里的euclidean distance = “离目标还有多远” 直接从demo policy学一个还要多少步到达目标的directed distance, use for MPC 重点：预测完之后怎么判断那个未来更接近goal JEPA WM's problem:
Better Source, Better Flow: Learning Condition-Dependent Source Distribution for Flow Matching
Prior Distribution $p(\theta)$: 生成模型的起点, prior = source = base distribution. The most common version: \[z\sim\mathcal N(0,I)\]Which is standard gaussi
Differentiable Robot Rendering
Input: current joint config p = [q_1, q_2...q_J], current camera viewpoint $\theta_\pi$ Output: $I(p)$ = an RGB image of what this robot arm looks lik
Visual-RRT: Finding Paths toward Visual-Goals via Differentiable Rendering
Given goal image, find collision-free trajectory from q_start to q_final whos render image matches goal image. Find the path fully in virtual space, t
Goal-Image Conditioned Dynamic Cable Manipulation through Bayesian Inference and Multi-Objective Black-Box Optimization
Input: goal image for cable manipulation Output: how to move robot such that the cable looks like goal image Trains a forward model:\[(I_t,J_t,J_{t+1}
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
Predict future in latent representation space. Turn it from self-supervised video representation model → robot planning world model
Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models
SuSIE — Subgoal Synthesis via Image Editing 让image editing diffusion model画出下一步什么样，然后Low level policy去实现这张图 Image-editing diffusion model Input: curre
LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding
Benchmark designed such that it requires to view more frames to increase performace. Retreival: find related details in longer sequence of video foota
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
First teach model how to view an video. Then teach model what the video means. Then teach model how to use language to answer video's questions. 3 Tra
TSM: Temporal Shift Module for Efficient Video Understanding
让2D CNN看到时间信息 2D CNN + Temporal Shift = spatiotemporal CNN 3D CNN is very computationaly expensive. Conv(T, H, W) with T = Time Normal CNN:: Conv(H, W
SlowFast Networks for Video Recognition
2 Pathways: Slow pathway: Whats in the frame Fast pathway: How are objects moving Video Recognition. Input: huamn waving hand Output: action classific
Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Control (GRIF)
Connect language instruction with visual state change (start → goal) If we do language → goal image: - Instruction: "Move the pan to the left" - there
MBOLD - Model-Based Visual Planning with Self-Supervised Functional Distances
Notes coming soon.
EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models
With a pretrained VLA, using limited data, using RL to finetune the policy to make it more reliable. Given a pretrained VLA + RL-based error correctio
One-step Diffusion with Distribution Matching Distillation
Diffusion is good quality, but inference is too slow. DMD takes 100+ steps diffusion model, distill it to one forward pass generator. DMD is $x = G_\t
NP Edit: Learning an Image Editing Model without Image Editing Pairs
This is an instruction-based image editing model. Input: Input image + editing instruction Output: Edited Image Problem: for image editing model, the
MUTEX: Learning Unified Policies from Multimodal Task Specifications
Give multiple modality input to the same representation space, output unified robot policy. Multimodal information is used mainly to train a better ta
GENIMA: Generative Image as Action Models
Input: at timesetp t: \[O_t= \{I_t^{front}, I_t^{left}, I_t^{right}, I_t^{wrist} \} \]4 RGB camera views Language instruction Output: action policy ou
VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training
VIP is a fine tuned Resnet-50 encoder, when you give it current image, it outputs a latent vector $z_t$. When you give it a goal image, it outputs a l
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Speech Representation Learning: turn raw sound into numerical features that make useful information easier to extract. - what sounds were spoken Same
Deep Audio Priors Emerge From Harmonic Convolutional Networks
1. Given noisy audio track, it will output the track with less / without noise 2. Given a track with multiple insturments i.e. flute, piano, violin. I
Compositional Image Decomposition with Diffusion Models
从图片中检测compositional concepts
ControlNet: Adding Conditional Control to Text-to-Image Diffusion Models
提供一个骨架图image+文字指定prompt，然后生成一个好image Input: prompt, control image, random noise Output: edited image
InstructPix2Pix: Learning to Follow Image Editing Instructions
修图师傅：之前需要specify新的图片长什么样，现在可以直接通过语言prompt去描述怎么改这张图片 学习条件： 图片 I: 一张红色汽车 指令 c：把汽车变成蓝色，背景不变 图片 Y: 蓝汽车 Input: Instruction that tells the model How to edit