Yujie Lu

EAAI Journal 2026 Journal Article

Corrigendum to “Mobile localization of multiple construction hazards based on panoramic vision and spatial transformation” [Eng. Appl. Artif. Intell. 165 (2026) 113427]

• Yujie Lu
• Haisong Wang
• Wei Wei

EAAI Journal 2026 Journal Article

Mobile localization of multiple construction hazards based on panoramic vision and spatial transformation

• Yujie Lu
• Haisong Wang
• Wei Wei

Construction safety management is critical for identifying and controlling potential risks on-site. While monocular vision is commonly used for hazard recognition, its lack of depth information makes accurate hazard localization challenging. To enhance safety management efficiency, this study proposed a localization method for multiple construction hazards using panoramic mobile tracking. First, the true pose of the monocular camera's moving trajectory is recovered through panoramic spatial transformations. Second, numerous safety hazards are localized by clustering the intersection points of observation lines. A total of 1014 images of unsealed openings were collected from five construction projects for model training. The proposed method achieved high semantic segmentation accuracy (96. 38 %) and precise spatial localization (deviation <0. 8 m) of indoor unsealed openings. This research presents a monocular vision-based approach for hazard recognition and localization, offering a practical reference for intelligent construction safety inspections in large-scale spaces.

EAAI Journal 2025 Journal Article

Enhancing rotation you only look once version 8 for accurate detection of arbitrarily-oriented and multi-scale construction material in complex environment

• Yujie Lu
• Yuanjun Nong
• Dayu Zhu
• Shuo Wang

Construction material detection is integral to effective material management. While deep learning-based methods have advanced automatic detection, challenges such as arbitrarily oriented objects, similar features, large object scale variations, and noise interference still limit accuracy. This study proposes an enhanced detection method based on improved rotation you only look once version 8, incorporating three key innovations. First, the large selective kernel network employs a spatial selection mechanism to dynamically adjust the network's receptive field, capturing key features and contextual information of various materials in complex construction scenarios. This enhances object detection performance in feature-similar environments. Second, the poly kernel inception network combines non-dilated multi-scale convolutions to extract dense texture features from differently sized objects, while using contextual anchor attention to capture long-range information for small objects. These components work together to improve multi-scale object detection. Third, the rectangular self-calibration module uses horizontal and vertical pooling to model rectangular attention regions, capturing axial global context. Its shape self-calibration function adjusts these regions to better fit arbitrarily oriented construction materials, enhancing focus on objects while suppressing noise interference. Experimental results show a mean average precision of 0. 871–2. 7 % higher than the baseline model and surpassing other state-of-the-art methods—while maintaining real-time detection speed at 28. 3 frames per second. The proposed method improves material detection accuracy in complex construction environments, enabling refined material management on-site. This supports material entry and exit tracking, on-site usage monitoring, inventory management, and procurement planning, while also strengthening lean control of construction progress and costs.

NeurIPS Conference 2025 Conference Paper

MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research

• Hui Chen
• Miao Xiong
• Yujie Lu
• Wei Han
• Ailin Deng
• Yufei He
• Jiaying Wu
• Yibo Li

Recent advancements in AI agents have demonstrated their growing potential to drive and support scientific discovery. In this work, we introduce MLR-Bench, a comprehensive benchmark for evaluating AI agents on open-ended machine learning research. MLR-Bench includes three key components: (1) 201 research tasks sourced from NeurIPS, ICLR, and ICML workshops covering diverse ML topics; (2) MLR-Judge, an automated evaluation framework combining LLM-based reviewers with carefully designed review rubrics to assess research quality; and (3) MLR-Agent, a modular agent scaffold capable of completing research tasks through four stages: idea generation, proposal formulation, experimentation, and paper writing. Our framework supports both stepwise assessment across these distinct research stages, and end-to-end evaluation of the final research paper. We then use MLR-Bench to evaluate six frontier LLMs and an advanced coding agent, finding that while LLMs are effective at generating coherent ideas and well-structured papers, current coding agents frequently (e. g. , in 80\% of the cases) produce fabricated or invalidated experimental results—posing a major barrier to scientific reliability. We validate MLR-Judge through human evaluation, showing high agreement with expert reviewers, supporting its potential as a scalable tool for research evaluation. We open-source MLR-Bench to help the community benchmark, diagnose, and improve AI research agents toward trustworthy and transparent scientific discovery.

ICLR Conference 2025 Conference Paper

MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos

• Xuehai He
• Weixi Feng
• Kaizhi Zheng
• Yujie Lu
• Wanrong Zhu
• Jiachen Li
• Yue Fan
• Jianfeng Wang

Multimodal Language Language Models (MLLMs) demonstrate the emerging abilities of "world models"---interpreting and reasoning about complex real-world dynamics. To assess these abilities, we posit videos are the ideal medium, as they encapsulate rich representations of real-world dynamics and causalities. To this end, we introduce MMWorld, a new benchmark for multi-discipline, multi-faceted multimodal video understanding. MMWorld distinguishes itself from previous video understanding benchmarks with two unique advantages: (1) multi-discipline, covering various disciplines that often require domain expertise for comprehensive understanding; (2) multi-faceted reasoning, including explanation, counterfactual thinking, future prediction, etc. MMWorld consists of a human-annotated dataset to evaluate MLLMs with questions about the whole videos and a synthetic dataset to analyze MLLMs within a single modality of perception. Together, MMWorld encompasses 1,910 videos across seven broad disciplines and 69 subdisciplines, complete with 6,627 question-answer pairs and associated captions. The evaluation includes 4 proprietary and 11 open-source MLLMs, which struggle on MMWorld (e.g., GPT-4o performs the best with only 62.5% accuracy), showing large room for improvement. Further ablation studies reveal other interesting findings such as models' different skill sets from humans. We hope MMWorld can serve as an essential step towards world model evaluation in videos.

ICLR Conference 2024 Conference Paper

ImagenHub: Standardizing the evaluation of conditional image generation models

• Max Ku
• Tianle Li
• Kai Zhang 0033
• Yujie Lu
• Xingyu Fu
• Wenwen Zhuang
• Wenhu Chen

Recently, a myriad of conditional image generation and editing models have been developed to serve different downstream tasks, including text-to-image generation, text-guided image editing, subject-driven image generation, control-guided image generation, etc. However, we observe huge inconsistencies in experimental conditions: datasets, inference, and evaluation metrics -- render fair comparisons difficult. This paper proposes ImagenHub, which is a one-stop library to standardize the inference and evaluation of all the conditional image generation models. Firstly, we define seven prominent tasks and curate high-quality evaluation datasets for them. Secondly, we built a unified inference pipeline to ensure fair comparison. Thirdly, we design two human evaluation scores, i.e. Semantic Consistency and Perceptual Quality, along with comprehensive guidelines to evaluate generated images. We train expert raters to evaluate the model outputs based on the proposed metrics. Our human evaluation achieves a high inter-worker agreement of Krippendorff’s alpha on 76\% models with a value higher than 0.4. We comprehensively evaluated a total of around 30 models and observed three key takeaways: (1) the existing models’ performance is generally unsatisfying except for Text-guided Image Generation and Subject-driven Image Generation, with 74\% models achieving an overall score lower than 0.5. (2) we examined the claims from published papers and found 83\% of them hold with a few exceptions. (3) None of the existing automatic metrics has a Spearman's correlation higher than 0.2 except subject-driven image generation. Moving forward, we will continue our efforts to evaluate newly published models and update our leaderboard to keep track of the progress in conditional image generation.

ICLR Conference 2024 Conference Paper

Language Control Diffusion: Efficiently Scaling through Space, Time, and Tasks

• Edwin Zhang
• Yujie Lu
• Shinda Huang
• William Yang Wang
• Amy Zhang 0001

Training generalist agents is difficult across several axes, requiring us to deal with high-dimensional inputs (space), long horizons (time), and generalization to novel tasks. Recent advances with architectures have allowed for improved scaling along one or two of these axes, but are still computationally prohibitive to use. In this paper, we propose to address all three axes by leveraging Language to Control Diffusion models as a hierarchical planner conditioned on language (LCD). We effectively and efficiently scale diffusion models for planning in extended temporal, state, and task dimensions to tackle long horizon control problems conditioned on natural language instructions, as a step towards generalist agents. Comparing LCD with other state-of-the-art models on the CALVIN language benchmark finds that LCD outperforms other SOTA methods in multi-task success rates, whilst improving inference speed over other comparable diffusion models by 3.3x~15x. We show that LCD can successfully leverage the unique strength of diffusion models to produce coherent long range plans while addressing their weakness in generating low-level details and control.

NeurIPS Conference 2024 Conference Paper

Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2)

• Michael Saxon
• Fatima Jahara
• Mahsa Khoshnoodi
• Yujie Lu
• Aditya Sharma
• William Yang Wang

With advances in the quality of text-to-image (T2I) models has come interest in benchmarking their prompt faithfulness---the semantic coherence of generated images to the prompts they were conditioned on. A variety of T2I faithfulness metrics have been proposed, leveraging advances in cross-modal embeddings and vision-language models (VLMs). However, these metrics are not rigorously compared and benchmarked, instead presented with correlation to human Likert scores over a set of easy-to-discriminate images against seemingly weak baselines. We introduce T2IScoreScore, a curated set of semantic error graphs containing a prompt and a set of increasingly erroneous images. These allow us to rigorously judge whether a given prompt faithfulness metric can correctly order images with respect to their objective error count and significantly discriminate between different error nodes, using meta-metric scores derived from established statistical tests. Surprisingly, we find that the state-of-the-art VLM-based metrics (e. g. , TIFA, DSG, LLMScore, VIEScore) we tested fail to significantly outperform simple (and supposedly worse) feature-based metrics like CLIPScore, particularly on a hard subset of naturally-occurring T2I model errors. TS2 will enable the development of better T2I prompt faithfulness metrics through more rigorous comparison of their conformity to expected orderings and separations under objective criteria.

NeurIPS Conference 2024 Conference Paper

WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences

• Yujie Lu
• Dongfu Jiang
• Wenhu Chen
• William Yang Wang
• Yejin Choi
• Bill Yuchen Lin

Recent breakthroughs in vision-language models (VLMs) emphasize the necessity of benchmarking human preferences in real-world multimodal interactions. To address this gap, we launched WildVision-Arena (WV-Arena), an online platform that collects human preferences to evaluate VLMs. We curated WV-Bench by selecting 500 high-quality samples from 8, 000 user submissions in WV-Arena. WV-Bench uses GPT-4 as the judge to compare each VLM with Claude-3-Sonnet, achieving a Spearman correlation of 0. 94 with the WV-Arena Elo. This significantly outperforms other benchmarks like MMVet, MMMU, and MMStar. Our comprehensive analysis of 20K real-world interactions reveals important insights into the failure cases of top-performing VLMs. For example, we find that although GPT-4V surpasses many other models like Reka-Flash, Opus, and Yi-VL-Plus in simple visual recognition and reasoning tasks, it still faces challenges with subtle contextual cues, spatial reasoning, visual imagination, and expert domain knowledge. Additionally, current VLMs exhibit issues with hallucinations and safety when intentionally provoked. We are releasing our chat and feedback data to further advance research in the field of VLMs.

NeurIPS Conference 2023 Conference Paper

LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation

• Yujie Lu
• Xianjun Yang
• Xiujun Li
• Xin Eric Wang
• William Yang Wang

Existing automatic evaluation on text-to-image synthesis can only provide an image-text matching score, without considering the object-level compositionality, which results in poor correlation with human judgments. In this work, we propose LLMScore, a new framework that offers evaluation scores with multi-granularity compositionality. LLMScore leverages the large language models (LLMs) to evaluate text-to-image models. Initially, it transforms the image into image-level and object-level visual descriptions. Then an evaluation instruction is fed into the LLMs to measure the alignment between the synthesized image and the text, ultimately generating a score accompanied by a rationale. Our substantial analysis reveals the highest correlation of LLMScore with human judgments on a wide range of datasets (Attribute Binding Contrast, Concept Conjunction, MSCOCO, DrawBench, PaintSkills). Notably, our LLMScore achieves Kendall's tau correlation with human evaluations that is 58. 8% and 31. 2% higher than the commonly-used text-image matching metrics CLIP and BLIP, respectively.

ICLR Conference 2023 Conference Paper

Neuro-Symbolic Procedural Planning with Commonsense Prompting

• Yujie Lu
• Weixi Feng
• Wanrong Zhu
• Wenda Xu
• Xin Eric Wang
• Miguel P. Eckstein
• William Yang Wang

Procedural planning aims to implement complex high-level goals by decomposition into simpler low-level steps. Although procedural planning is a basic skill set for humans in daily life, it remains a challenge for large language models (LLMs) that lack a deep understanding of the cause-effect relations in procedures. Previous methods require manual exemplars to acquire procedural planning knowledge from LLMs in the zero-shot setting. However, such elicited pre-trained knowledge in LLMs induces spurious correlations between goals and steps, which impair the model generalization to unseen tasks. In contrast, this paper proposes a neuro-symbolic procedural PLANner (PLAN) that elicits procedural planning knowledge from the LLMs with commonsense-infused prompting. To mitigate spurious goal-step correlations, we use symbolic program executors on the latent procedural representations to formalize prompts from commonsense knowledge bases as a causal intervention toward the Structural Causal Model. Both automatic and human evaluations on WikiHow and RobotHow show the superiority of PLAN on procedural planning without further training or manual exemplars.

ICLR Conference 2023 Conference Paper

WikiWhy: Answering and Explaining Cause-and-Effect Questions

• Matthew Ho
• Aditya Sharma
• Justin Chang
• Michael Saxon
• Sharon Levy
• Yujie Lu
• William Yang Wang

As large language models (LLMs) grow larger and more sophisticated, assessing their "reasoning" capabilities in natural language grows more challenging. Recent question answering (QA) benchmarks that attempt to assess reasoning are often limited by a narrow scope of covered situations and subject matters. We introduce WikiWhy, a QA dataset built around a novel auxiliary task: explaining why an answer is true in natural language. WikiWhy contains over 9,000 "why" question-answer-rationale triples, grounded on Wikipedia facts across a diverse set of topics. Each rationale is a set of supporting statements connecting the question to the answer. WikiWhy serves as a benchmark for the reasoning capabilities of LLMs because it demands rigorous explicit rationales for each answer to demonstrate the acquisition of implicit commonsense knowledge, which is unlikely to be easily memorized. GPT-3 baselines achieve only 38.7% human-evaluated correctness in the end-to-end answer & explain condition, leaving significant room for future improvements.