Choice alignment has turn out to be a vital element in enhancing the efficiency of Massive Language Fashions (LLMs), but its affect in Multimodal Massive Language Fashions (MLLMs) stays comparatively underexplored. Much like language fashions, MLLMs for picture understanding duties encounter challenges like hallucination. In MLLMs, hallucination can happen not solely by stating incorrect information but additionally by producing responses which can be inconsistent with the picture content material. A major goal of alignment for MLLMs is to encourage these fashions to align responses extra carefully with picture info. Not too long ago, a number of works have launched desire datasets for MLLMs and examined totally different alignment strategies, together with Direct Choice Optimization (DPO) and Proximal Coverage Optimization (PPO). Nonetheless, resulting from variations in datasets, base mannequin varieties, and alignment strategies, it stays unclear which particular components contribute most importantly to the reported enhancements in these works. On this paper, we independently analyze every facet of desire alignment in MLLMs. We begin by categorizing the alignment algorithms into two teams, offline (akin to DPO), and on-line (akin to online-DPO), and present that combining offline and on-line strategies can enhance the efficiency of the mannequin in sure situations. We assessment quite a lot of revealed multimodal desire datasets and talk about how the small print of their building affect mannequin efficiency. Primarily based on these insights, we introduce a novel approach of making multimodal desire information known as Bias-Pushed Hallucination Sampling (BDHS) that wants neither further annotation nor exterior fashions, and present that it might probably obtain aggressive efficiency to beforehand revealed alignment work for multimodal fashions throughout a spread of benchmarks.
- * Authors contributed equally as first authors.
- † Authors contributed equally.
