Dual reasoning, broken behavior

Multimodal models supporting hybrid reasoning are expected to produce the same answer whether they “think” deeply or answer quickly. A new arXiv paper introduces PatternEval, a diagnostic benchmark of 2,415 multimodal prompts that measures not only task accuracy but also four recurring failure patterns: chain-of-thought leakage, answer repetition, logical contradiction, and performative reasoning. The results indicate that these failures appear across models from different vendors, and that the non-reasoning inference mode exhibits substantially higher failure rates, creating a systematic misalignment between the two interfaces.

PatternEval evaluates visual perception and clutter, structured image understanding, and multimodal knowledge inference. Each sample is crafted to expose a specific failure: whether the model discloses its internal chain of thought as part of the final answer, whether it loops on itself, whether two answer stages contradict each other, or whether it “performs” reasoning without reaching a conclusion. The authors stress that task accuracy alone does not capture these issues; a model can be factually correct while still emitting a raw chain of thought to the user.

According to the data, the non-reasoning inference mode shows markedly higher failure rates in all four categories. The gap is not random: when the model forgoes a reasoning budget, it tends to take shortcuts that manifest as the problematic output patterns. Consequently, a user receiving an answer from the non-reasoning inference mode experiences a qualitatively different product than one using the reasoning mode, a problem that intensifies when the same model serves both paths.

To close the gap, the team developed PatternRM, a response-level reward model that detects the four failures, and PatternRL, which imposes specific penalties for each pattern during reinforcement-learning (RL) training. Experiments on Qwen3-VL-4B and Qwen3-VL-8B demonstrate that targeted penalties reduce the misalignment between modes with only a negligible drop in task performance. In other words, the model can be taught to maintain consistent output behavior without sacrificing accuracy.

The broader contribution is a two-tool framework: PatternEval provides continual diagnostics, while PatternRL offers a corrective direction. The approach echoes the shift from correctness-only evaluation to behavior-focused testing in large language models, but here the focus is consistency across two inference settings of the same model. The paper remains a pre-print and the experiments are limited to the Qwen3-VL family; it is unclear whether the same penalties would be effective on other architectures or at different scales.

The authors do not report latency or computational cost for the corrected mode, nor how the penalties affect the tail distribution of answers. The definition of “performative reasoning” also remains somewhat vague in the current benchmark. Until additional model runs and A/B user studies are conducted, PatternRL should be viewed as a promising direction rather than a proven fix.