Multimodal giant language fashions (MLLMs) are more and more deployed in real-world, agentic settings the place outputs should not solely be appropriate, but additionally conform to predefined knowledge schemas. Regardless of latest progress in structured technology in textual area, there may be nonetheless no benchmark that systematically evaluates schema-grounded info extraction and reasoning over visible inputs. On this work, we conduct a complete examine of visible structural output capabilities for MLLMs with our rigorously designed SO-Bench benchmark. Masking 4 visible domains, together with UI screens, pure pictures, paperwork, and charts, SO-Bench is constructed from over 6.5K numerous JSON schemas and 1.8K curated image-schema pairs with human-verified high quality. Benchmarking experiments on open-sourced and frontier proprietary fashions reveal persistent gaps in predicting correct, schema compliant outputs, highlighting the necessity for higher multimodal structured reasoning. Past benchmarking, we additional conduct coaching experiments to largely enhance the mannequin’s structured output functionality. We plan to make the benchmark obtainable to the neighborhood.
Determine 1: Left: Overview of the multi-stage knowledge technology pipeline for SO-Bench, together with schema technology, person intent technology, and response technology levels. At every stage, proprietary frontier fashions resembling GPT-5 and Gemini-2.5-Professional act as turbines with rigorously designed prompts. Human area consultants evaluate knowledge from every stage earlier than it progresses to the following. Previous to schema technology, enter pictures and JSON schemas are embedded utilizing a CLIP mannequin for embedding search. Proper: Benchmarking outcomes amongst a number of open-source fashions and proprietary frontier fashions.
Determine 2: Overview of the multi-stage knowledge technology pipeline for SO-Bench, together with schema technology, person intent technology, and response technology levels. At every stage, proprietary frontier fashions resembling GPT-5 and Gemini-2.5-Professional act as turbines with rigorously designed prompts. Human area consultants evaluate knowledge from every stage earlier than it progresses to the following. Previous to schema technology, enter pictures and JSON schemas are embedded utilizing a CLIP mannequin for embedding search.
