Haotian Liu

photo.jpg

I lead the Omni team at xAI.

I was a core contributor to Grok-1.5V and Grok-2. I led the vision effort of Grok-3 and Grok-3 Reasoning. I led the development of Grok Imagine v0.9 and v1, where we build image generation and video generation models from 0 to No.1 on leaderboards in 6 months.

I am generally interested in computer vision and machine learning. My recent focus is on building steerable large models. The first baby was LLaVA.

I obtained my Ph.D. from University of Wisconsin-Madison in May 2024, under the supervision of Prof. Yong Jae Lee. During my Ph.D., I was fortunate to work with Dr. Chunyuan Li at Microsoft Research. I obtained my bachler’s degree (with honor) at Zhejiang University, where I worked with Prof. Xiaogang Jin and Prof. Fei Wu.

selected publications

  1. Blog
    LLaVA-NeXT: Improved reasoning, OCR, and world knowledge
    Haotian Liu, Chunyuan Li , Yuheng Li , Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee
    Jan 2024
    Blog Code Demo
  2. llava_v15.jpg
    Improved Baselines with Visual Instruction Tuning (LLaVA-1.5)
    Haotian Liu, Chunyuan Li , Yuheng Li, and Yong Jae Lee
    CVPR, 2024
    arXiv HTML Code Demo
  3. llava2023.jpg
    Visual Instruction Tuning (LLaVA)
    Haotian Liu*, Chunyuan Li*, Qingyang Wu, and Yong Jae Lee
    NeurIPS, 2023 (Oral, top 0.5%)
    arXiv HTML Code Demo
  4. react2023.jpg
    Learning Customized Visual Models with Retrieval-Augmented Knowledge
    Haotian Liu, Kilho Son , Jianwei Yang , Ce Liu , Jianfeng Gao, Yong Jae Lee*, and Chunyuan Li*
    CVPR, 2023 (Highlight, top 2.5%)
    arXiv HTML Code
  5. gligen2023.jpg
    GLIGEN: Open-Set Grounded Text-to-Image Generation
    Yuheng Li , Haotian Liu, Qingyang Wu, Fangzhou Mu , Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee
    CVPR, 2023
    arXiv HTML Code Demo
  6. elevater2022.jpg
    ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models
    Chunyuan Li* , Haotian Liu* , Liunian Li , Pengchuan Zhang , Jyoti Aneja , Jianwei Yang, Ping Jin , Houdong Hu , Zicheng Liu , Yong Jae Lee, and Jianfeng Gao
    NeurIPS, Datasets and Benchmarks Track, 2022
    arXiv HTML Video
  7. maskpoint2022.png
    Masked Discrimination for Self-Supervised Learning on Point Clouds
    Haotian Liu, Mu Cai, and Yong Jae Lee
    ECCV, 2022
    arXiv Video Code
  8. icra2021.jpg
    YolactEdge: Real-time Instance Segmentation on the Edge
    Haotian Liu*, Rafael A Rivera Soto* , Fanyi Xiao, and Yong Jae Lee
    ICRA, 2021
    arXiv Video Code