地址

論文地址：https://github.com/deepseek-ai/Janus/blob/main/janus_pro_tech_report.pdf

時(shí)間

2025年1月27日

作者

DeepSeek團(tuán)隊(duì)

In this work, we introduce Janus-Pro, an advanced version of the previous work Janus. Specif-ically, Janus-Pro incorporates (1) an optimized training strategy, (2) expanded training data, and (3) scaling to larger model size. With these improvements, Janus-Pro achieves significant advancements in both multimodal understanding and text-to-image instruction-following capa-bilities, while also enhancing the stability of text-to-image generation. We hope this work will inspire further exploration in the field. Code and models are publicly available.

在這項(xiàng)工作中，我們推出了 Janus-Pro，這是之前工作 Janus 的高級(jí)版本,。具體而言，Janus-Pro?融合了（1）優(yōu)化的訓(xùn)練策略，（2）擴(kuò)充的訓(xùn)練數(shù)據(jù)，以及（3）對(duì)更大模型規(guī)模的支持,。憑借這些改進(jìn)，Janus-Pro 在多模態(tài)理解和文本到圖像指令遵循能力方面取得了顯著進(jìn)步，同時(shí)提升了文本到圖像生成的穩(wěn)定性,。我們希望這項(xiàng)工作能激發(fā)該領(lǐng)域的進(jìn)一步探索。代碼和模型已公開可用,。

Recent advancements in unified multimodal understanding and generation models have demonstrated significant progress [30, 40, 45, 46, 48, 50, 54, 55]. These approaches have been proven to enhance the instruction-following capabilities in visual generation tasks while re-ducing model redundancy. Most of these methods utilize the same visual encoder to process inputs for both multimodal understanding and generation tasks. Since the representations required for these two tasks differ, this often results in suboptimal performance in multimodal understanding. To address this issue, Janus [46] proposes decoupling visual encoding, which alleviates the conflict between multimodal understanding and generation tasks, achieving excellent performance in both tasks.

As a pioneering model, Janus is validated at the 1B parameter scale. However, due to the limited amount of training data and the relatively small model capacity, it exhibites certain shortcomings, such as suboptimal performance on short prompts image generation and unstable text-to-image generation quality. In this paper, we introduce Janus-Pro, an enhanced version of Janus that incorporates improvements across three dimensions: training strategies, data, and model size. The Janus-Pro series includes two model sizes: 1B and 7B, demonstrating scalability of the visual encoding decoding method.

統(tǒng)一多模態(tài)理解和生成模型的最新進(jìn)展已取得顯著成果[30, 40, 45, 46, 48, 50, 54, 55]。這些方法已被證明能夠提升視覺生成任務(wù)中的指令遵循能力，同時(shí)減少模型冗余,。大多數(shù)這些方法都使用相同的視覺編碼器來處理多模態(tài)理解和生成任務(wù)的輸入,。由于這兩個(gè)任務(wù)所需的表示不同，這往往導(dǎo)致多模態(tài)理解任務(wù)的表現(xiàn)不佳。為了解決這個(gè)問題，Janus [46] 提出了視覺編碼解耦，這緩解了多模態(tài)理解和生成任務(wù)之間的沖突，在這兩個(gè)任務(wù)中都取得了出色的表現(xiàn),。

作為開創(chuàng)性的模型，Janus 在 10 億參數(shù)規(guī)模上得到了驗(yàn)證,。然而，由于訓(xùn)練數(shù)據(jù)量有限以及模型容量相對(duì)較小，它存在一些不足之處，例如在短提示圖像生成方面的表現(xiàn)欠佳以及文本到圖像生成質(zhì)量不穩(wěn)定。在本文中，我們介紹了 Janus-Pro，這是 Janus 的增強(qiáng)版，在訓(xùn)練策略,、數(shù)據(jù)和模型規(guī)模三個(gè)維度上都有所改進(jìn),。Janus-Pro 系列包含兩種模型規(guī)模：10 億參數(shù)和 70 億參數(shù)，展示了視覺編碼解碼方法的可擴(kuò)展性。

We evaluate Janus-Pro on multiple benchmarks, and the results reveal its superior multi-modal understanding capabilities and significantly improved text-to-image instruction-following performance. Specifically, Janus-Pro-7B achieved a score of 79.2 on the multimodal understand-ing benchmark MMBench [29], surpassing state-of-the-art unified multimodal models such as Janus [46] (69.4), TokenFlow [34] (68.9) and MetaMorph [42] (75.2). Additionally, in the text-to-image instruction-following leaderboard GenEval [14], Janus-Pro-7B scores 0.80, outperforming Janus [46] (0.61), DALL-E 3 (0.67), and Stable Diffusion 3 Medium [11] (0.74).

我們?cè)诙鄠€(gè)基準(zhǔn)測(cè)試中對(duì) Janus-Pro 進(jìn)行了評(píng)估，結(jié)果表明其具有卓越的多模態(tài)理解能力和顯著提升的文本到圖像指令遵循性能,。具體而言，Janus-Pro-7B 在多模態(tài)理解基準(zhǔn) MMBench [29] 上獲得了 79.2 的分?jǐn)?shù)，超過了諸如 Janus [46]（69.4）,、TokenFlow [34]（68.9）和 MetaMorph [42]（75.2）等最先進(jìn)的統(tǒng)一多模態(tài)模型。此外，在文本到圖像指令遵循排行榜 GenEval [14] 上，Janus-Pro-7B 的得分是 0.80，優(yōu)于 Janus [46]（0.61）,、DALL-E 3（0.67）和 Stable Diffusion 3 Medium [11]（0.74）,。

This paper introduces improvements to Janus from three aspects: training strategy, data, and model size. These enhancements have led to significant advancements in both multimodal understanding and text-to-image instruction-following capabilities. However, Janus-Pro still has certain limitations. In terms of multimodal understanding, the input resolution is limited to 384 × 384, which affects its performance in fine-grained tasks such as OCR. For text-to-image generation, the low resolution, combined with reconstruction losses introduced by the vision tokenizer, results in images that, while rich in semantic content, still lack fine details. For example, small facial regions occupying limited image space may appear under-detailed. Increasing the image resolution could mitigate these issues.

本文從訓(xùn)練策略、數(shù)據(jù)和模型規(guī)模三個(gè)方面介紹了對(duì) Janus 的改進(jìn)。這些改進(jìn)在多模態(tài)理解和文本到圖像的指令遵循能力方面都取得了顯著進(jìn)展,。然而，Janus-Pro 仍存在一些局限性,。在多模態(tài)理解方面，輸入分辨率限制在 384×384，這影響了其在諸如 OCR 等細(xì)粒度任務(wù)中的表現(xiàn)。對(duì)于文本到圖像生成，低分辨率加上視覺標(biāo)記器引入的重建損失，導(dǎo)致生成的圖像雖然語義豐富，但仍缺乏細(xì)節(jié),。例如，占據(jù)圖像空間有限的小面部區(qū)域可能會(huì)顯得不夠清晰,。提高圖像分辨率可以緩解這些問題。