Author name cluster

Xiaojie Wang

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

23 papers

2 author rows

AAAI Conference 2025 Conference Paper

A Systematic Exploration of Knowledge Graph Alignment with Large Language Models in Retrieval Augmented Generation

Shiyu Tian
Shuyue Xing
Xingrui Li
Yangyang Luo
Caixia Yuan
Wei Chen
Huixing Jiang
Xiaojie Wang

Retrieval Augmented Generation (RAG) with Knowledge Graphs (KGs) is an effective way to enhance Large Language Models (LLMs). Due to the natural discrepancy between structured KGs and sequential LLMs, KGs must be linearized to text before being inputted into LLMs, leading to the problem of KG Alignment with LLMs (KGA). However, recent KG+RAG methods only consider KGA as a simple step without comprehensive and in-depth explorations, leaving three essential problems unclear: (1) What are the factors and their effects in KGA? (2) How do LLMs understand KGs? (3) How to improve KG+RAG by KGA? To fill this gap, we conduct systematic explorations on KGA, where we first define the problem of KGA and subdivide it into the graph transformation phase (graph-to-graph) and the linearization phase (graph-to-text). In the graph transformation phase, we study graph features at the node, edge, and full graph levels from low to high granularity. In the linearization phase, we study factors on formats, orders, and templates from structural to token levels. We conduct substantial experiments on 15 typical LLMs and three common datasets. Our main findings include: (1) The centrality of the KG affects the final generation; formats have the greatest impact on KGA; orders are model-dependent, without an optimal order adapting for all models; the templates with special token separators are better. (2) LLMs understand KGs by a unique mechanism, different from processing natural sentences, and separators play an important role. (3) We achieved 7.3% average performance improvements on four common LLMs on the KGQA task by combining the optimal factors to enhance KGA.

PDF Details DOI

AAAI Conference 2025 Conference Paper

Concept Conductor: Orchestrating Multiple Personalized Concepts in Text-to-Image Synthesis

Zebin Yao
Fangxiang Feng
Ruifan Li
Xiaojie Wang

The customization of text-to-image models has seen significant advancements, yet generating multiple personalized concepts remains a challenging task. Current methods struggle with attribute leakage and layout confusion when handling multiple concepts, leading to reduced concept fidelity and semantic consistency. In this work, we introduce a novel training-free framework, Concept Conductor, designed to ensure visual fidelity and correct layout in multi-concept customization. Concept Conductor isolates the sampling processes of multiple customized models to prevent attribute leakage between different concepts and corrects erroneous layouts through self-attention-based spatial guidance. Additionally, we present a concept injection technique that employs shape-aware masks to specify the generation area for each concept. This technique injects the structure and appearance of personalized concepts through feature fusion in the attention layers, ensuring harmony in the final image. Extensive qualitative and quantitative experiments demonstrate that Concept Conductor can consistently generate composite images with accurate layouts while preserving the visual details of each concept. Compared to existing baselines, Concept Conductor shows significant performance improvements. Our method supports the combination of any number of concepts and maintains high fidelity even when dealing with visually similar concepts. The code and trained models will be made publicly available.

PDF Details DOI

AAAI Conference 2025 Conference Paper

Data with High and Consistent Preference Difference Are Better for Reward Model

Qi Lin
Hengtong Lu
Caixia Yuan
Xiaojie Wang
Huixing Jiang
Wei Chen

Reinforcement Learning from Human Feedback (RLHF) is a commonly used alignment method for Large Language Models (LLMs). This method relies on a reward model trained on a preference dataset to provide scalar rewards. However, the human-annotated preference data is often sparse, noisy, and costly to obtain, necessitating more efficient utilization. This paper proposes a new metric for better preference data utilization from both theoretical and empirical perspectives. Starting with the Bradley-Terry model, we compute the Mean Square Error (MSE) between the expected loss and empirical loss of the reward model. Our findings reveal that data with higher and more consistent difference result in lower MSE. We therefore propose the Preference Difference (PD), the reward difference between two samples, as a filter for preference data. Experimental results on three open-source models show that reward models trained by filtered data with PD achieve higher calibrated accuracy, as well as better RLHF alignment performance. The conclusion remains consistent when we extend the experiments and theoretical derivations to implicit reward alignment algorithms, such as Direct Preference Optimization (DPO).

PDF Details DOI

IROS Conference 2025 Conference Paper

Enhancing Tactile Sensing in Robotics Using Null-Space Diffusion Model with EIT-based Sensors

Qilin Zhang
Haofeng Chen
Xuanxuan Yang
Gang Ma
Xiaojie Wang

Robotic tactile sensors based on Electrical Impedance Tomography (EIT) have gained great attention in robotic sensing applications due to their features such as no internal wiring, "all-in-one" structure, and continuous sensing capabilities. However, the effectiveness of EIT-based tactile sensors is hampered by limited spatial resolution and artifacts in the reconstructed images. To address these challenges, various iterative optimization methods based on spatial regularizations and model-based methods have been proposed. In this study, a new EIT reconstruction method using null-space decomposition based on a diffusion model (NSDM) is proposed. Specifically, NSDM consists of a forward diffusion process that first gradually adds Gaussian noise to a clean conductivity image, followed by a backward process that learns to predict the noise that should be removed during each sampling step, utilizing a prior to ensure that the denoising process does not deviate from the correct direction. NSDM requires no training, no optimization, and only requires a pre-prepared diffusion model. Experimental results (both simulation and actual tests) demonstrate that the proposed method outperforms existing generation methods and provides higher quality reconstruction, providing a new solution for robotic tactile sensing in real scenarios.

Details

ICML Conference 2025 Conference Paper

Non-asymptotic Error Bounds in W2-Distance with Sqrt(d) Dimension Dependence and First Order Convergence for Langevin Monte Carlo beyond Log-Concavity

Bin Yang
Xiaojie Wang

Generating samples from a high dimensional probability distribution is a fundamental task with wide-ranging applications in the area of scientific computing, statistics and machine learning. This article revisits the popular Langevin Monte Carlo (LMC) sampling algorithms and provides a non-asymptotic error analysis in $\mathcal{W}_2$-distance in a non-convex setting. In particular, we prove an error bound $O(\sqrt{d} h)$, which guarantees a mixing time $ \tilde{O} (\sqrt{d} \epsilon^{-1})$ to achieve the accuracy tolerance $\epsilon$, under certain log-smooth conditions and the assumption that the target distribution satisfies a log-Sobolev inequality, as opposed to the strongly log-concave condition used in (Li et al. , 2019; 2022). This bound matches the best one in the strongly log-concave case and improves upon the best-known convergence rates in non-convex settings. To prove it, we establish a new framework of uniform-in-time convergence for discretizations of SDEs. Distinct from (Li et al. , 2019; 2022), we start from the finite-time mean-square fundamental convergence theorem, which combined with uniform-in-time moment bounds of LMC and the exponential ergodicity of SDEs in the non-convex setting gives the desired uniform-in-time convergence. Our framework also applies to the case when the gradient of the potential $U$ is non-globally Lipschitz with superlinear growth, for which modified LMC samplers are proposed and analyzed, with a non-asymptotic error bound in $\mathcal{W}_2$-distance obtained. Numerical experiments corroborate the theoretical analysis.

Details

EAAI Journal 2024 Journal Article

Automatic segmentation of curtain wall frame using a context collaboration pyramid network

Decheng Wu
Longqi Cheng
Rui Li
Pingan Yang
Xiaoyu Xu
Xiaojie Wang
Chul-Hee Lee

Accurate positioning of curtain wall frames is crucial for the automated installation of curtain wall modules. However, the current robot-based installation methods overly depend on visual guidance from operators, resulting in high costs and limiting construction efficiency. The development of deep learning has introduced an image segmentation approach that offers a new solution for the visual positioning of curtain wall frames. This paper proposes a context collaboration pyramid network to automatically segment curtain wall frames by incorporating context interaction and channel guided pyramid structure. The model adopts an “encoder-decoder” architecture with a feature interaction block strategically inserted between the encoder and decoder. Specifically, the encoder utilizes the pyramid pooling Transformer as a backbone to extract multi-level features from original RGB images. The decoder employs a channel guided pyramid convolution module to integrate multi-scale features and achieve finer prediction. Meanwhile, a context interaction fusion module between the features of adjacent levels was designed carefully to enhance the collaboration of the architecture. In addition, a benchmark dataset for the curtain wall frame segmentation task, consisting of 1547 images, was established. The dataset incorporates challenging scenarios, including strong lights, low contrast, and cluttered backgrounds. This method is evaluated on the collected dataset, and achieves an impressive accuracy of 97. 30% and an F1-Score of 88. 95%, outperforming other segmentation networks. Overall, the proposed method can extract target information accurately and efficiently and provide critical visual guidance for the robot, so as to promote the automatic installation level of the curtain wall module.

Details DOI

AAAI Conference 2023 Conference Paper

SPRING: Situated Conversation Agent Pretrained with Multimodal Questions from Incremental Layout Graph

Yuxing Long
Binyuan Hui
Fulong Ye
Yanyang Li
Zhuoxin Han
Caixia Yuan
Yongbin Li
Xiaojie Wang

Existing multimodal conversation agents have shown impressive abilities to locate absolute positions or retrieve attributes in simple scenarios, but they fail to perform well when complex relative positions and information alignments are involved, which poses a bottleneck in response quality. In this paper, we propose a Situated Conversation Agent Pretrained with Multimodal Questions from Incremental Layout Graph (SPRING) with abilities of reasoning multi-hops spatial relations and connecting them with visual attributes in crowded situated scenarios. Specifically, we design two types of Multimodal Question Answering (MQA) tasks to pretrain the agent. All QA pairs utilized during pretraining are generated from novel Increment Layout Graphs (ILG). QA pair difficulty labels automatically annotated by ILG are used to promote MQA-based Curriculum Learning. Experimental results verify the SPRING's effectiveness, showing that it significantly outperforms state-of-the-art approaches on both SIMMC 1.0 and SIMMC 2.0 datasets. We release our code and data at https://github.com/LYX0501/SPRING.

PDF Details DOI

AAAI Conference 2021 Conference Paper

Converse, Focus and Guess – Towards Multi-Document Driven Dialogue

Han Liu
Caixia Yuan
Xiaojie Wang
Yushu Yang
Huixing Jiang
Zhongyuan Wang

We propose a novel task, Multi-Document Driven Dialogue (MD3), in which an agent can guess the target document that the user is interested in by leading a dialogue. To benchmark progress, we introduce a new dataset of GuessMovie, which contains 16, 881 documents, each describing a movie, and associated 13, 434 dialogues. Further, we propose the MD3 model. Keeping guessing the target document in mind, it converses with the user conditioned on both document engagement and user feedback. In order to incorporate large-scale external documents into the dialogue, it pretrains a document representation which is sensitive to attributes it talks about an object. Then it tracks dialogue state by detecting evolvement of document belief and attribute belief, and finally optimizes dialogue policy in principle of entropy decreasing and reward increasing, which is expected to successfully guess the user’s target in a minimum number of turns. Experiments show that our method significantly outperforms several strong baseline methods and is very close to human’s performance.

PDF Details

AAAI Conference 2021 Conference Paper

MIEHDR CNN: Main Image Enhancement based Ghost-Free High Dynamic Range Imaging using Dual-Lens Systems

Xuan Dong
Xiaoyan Hu
Weixin Li
Xiaojie Wang
Yunhong Wang

We study the High Dynamic Range (HDR) imaging problem using two Low Dynamic Range (LDR) images that are shot from dual-lens systems in a single shot time with different exposures. In most of the related HDR imaging methods, the problem is usually solved by Multiple Images Merging, i. e. the final HDR image is fused from pixels of all the input LDR images. However, ghost artifacts can be hardly avoided using this strategy. Instead of directly merging the multiple LDR inputs, we use an indirect way which enhances the main image, i. e. the short exposure image IS, using the long exposure image IL serving as guidance. In detail, we propose a new model, named MIEHDR CNN model, which consists of three subnets, i. e. Soft Warp CNN, 3D Guided Denoising CNN and Fusion CNN. The Soft Warp CNN aligns IL to get the aligned result ILA using the soft exposed result of IS as reference. The 3D Guided Denoising CNN denoises the soft exposed result of IS using ILA as guidance, whose result are fed into the Fusion CNN with IS to get the HDR result. The MIEHDR CNN model is implemented by MindSpore and experimental results show that we can outperform related methods largely and avoid ghost artifacts.

PDF Details

YNIMG Journal 2021 Journal Article

The macaque brain ONPRC18 template with combined gray and white matter labelmap for multimodal neuroimaging studies of Nonhuman Primates

Alison R. Weiss
Zheng Liu
Xiaojie Wang
William A. Liguore
Christopher D. Kroenke
Jodi L. McBride

Macaques are the most common nonhuman primate (NHP) species used in neuroscience research. With the advancement of many neuroimaging techniques, new studies are beginning to apply multiple types of in vivo magnetic resonance imaging (MRI), such as structural imaging (sMRI) with T1 and T2 weighted contrasts alongside diffusion weighed (DW) imaging. In studies involving rhesus macaques, this approach can be used to better understand micro-structural changes that occur during development, in various disease states or with normative aging. However, many of the available rhesus brain atlases have been designed for only one imaging modality, making it difficult to consistently define the same brain regions across multiple imaging modalities in the same subject. To address this, we created a brain atlas from 18 adult rhesus macaques that includes co-registered templates constructed from images frequently used to characterize macroscopic brain structure (T2/SPACE and T1/MP-RAGE), and a diffusion tensor imaging (DTI) template. The DTI template was up-sampled from 1 mm isotropic resolution to resolution match to the T1 and T2-weighted images (0.5 mm isotropic), and the parameter maps were derived for FA, AD, RD and MD.The labelmap volumes delineate 57 gray matter regions of interest (ROIs; 36 cortical regions and 21 subcortical structures), as well as 74 white matter tracts. Importantly, the labelmap overlays both the structural and diffusion templates, enabling the same regions to be consistently identified across imaging modalities. A specialized condensed version of the labelmap ROIs are also included to further extend the usefulness of this tool for imaging data with lower spatial resolution, such as functional MRI (fMRI) or positron emission tomography (PET).

Details DOI

YNIMG Journal 2020 Journal Article

Anatomical and diffusion MRI brain atlases of the fetal rhesus macaque brain at 85, 110 and 135 days gestation

Zheng Liu
Xiaojie Wang
Natali Newman
Kathleen A. Grant
Colin Studholme
Christopher D. Kroenke

Recent advances in image reconstruction techniques have enabled high resolution MRI studies of fetal brain development in human subjects. Rhesus macaques (Macaca mulatta) are valuable animal models for use in studies of fetal brain development due to the similarities between this species and humans in brain development and anatomy. There is a need to develop fetal brain templates for the rhesus macaque to facilitate the characterization of the normal brain growth trajectory and departures from this trajectory in rhesus models of neurodevelopmental disorders. Here we have developed unbiased population-based anatomical T2-weighted, fractional anisotropy (FA) and apparent diffusion coefficient (ADC) templates for fetal brain from MR images scanned at 3 time points over the second and third trimesters of the 168 day gestational term. Specifically, atlas images are constructed for brains at gestational ages of 85 days (G85, N = 18, 9 females), 110 days (G110, N = 10, 7 females) and 135 days (G135, N = 16, 7 females). We utilized this atlas to perform segmentation of fetal brain MR images and fetal brain volumetric and microstructure analysis. The T2-weighted template images facilitated characterization of the growth within six fetal brain regions. The template images of diffusion tensor indices provided information related to the maturation of white matter tracts. These growth trajectories are referenced to human studies of fetal brain development. Similarities in the temporal and regional patterns of brain growth over the corresponding periods of central nervous system development are identified between the two species. Atlas images are available online as a reference for registration, reconstruction, segmentation, and for longitudinal analysis of early fetal brain growth over this unique time window.

Details DOI

AAAI Conference 2020 Conference Paper

Cycle-CNN for Colorization towards Real Monochrome-Color Camera Systems

Xuan Dong
Weixin Li
Xiaojie Wang
Yunhong Wang

Colorization in monochrome-color camera systems aims to colorize the gray image IG from the monochrome camera using the color image RC from the color camera as reference. Since monochrome cameras have better imaging quality than color cameras, the colorization can help obtain higher quality color images. Related learning based methods usually simulate the monochrome-color camera systems to generate the synthesized data for training, due to the lack of ground-truth color information of the gray image in the real data. However, the methods that are trained relying on the synthesized data may get poor results when colorizing real data, because the synthesized data may deviate from the real data. We present a new CNN model, named cycle CNN, which can directly use the real data from monochrome-color camera systems for training. In detail, we use the colorization CNN model to do the colorization twice. First, we colorize IG using RC as reference to obtain the ﬁrst-time colorization result IC. Second, we colorize the de-colored map of RC, i. e. RG, using the ﬁrst-time colorization result IC as reference to obtain the second-time colorization result R C. In this way, for the second-time colorization result R C, we use the original color map RC as ground-truth and introduce the cycle consistency loss to push R C ≈ RC. Also, for the ﬁrst-time colorization result IC, we propose a structure similarity loss to encourage the luminance maps between IG and IC to have similar structures. In addition, we introduce a spatial smoothness loss within the colorization CNN model to encourage spatial smoothness of the colorization result. Combining all these losses, we could train the colorization CNN model using the real data in the absence of the ground-truth color information of IG. Experimental results show that we can outperform related methods largely for colorizing real data.

PDF Details

IJCAI Conference 2020 Conference Paper

Multi-scale Two-way Deep Neural Network for Stock Trend Prediction

Guang Liu
Yuzhao Mao
Qi Sun
Hailong Huang
Weiguo Gao
Xuan Li
Jianping Shen
Ruifan Li

Stock Trend Prediction(STP) has drawn wide attention from various fields, especially Artificial Intelligence. Most previous studies are single-scale oriented which results in information loss from a multi-scale perspective. In fact, multi-scale behavior is vital for making intelligent investment decisions. A mature investor will thoroughly investigate the state of a stock market at various time scales. To automatically learn the multi-scale information in stock data, we propose a Multi-scale Two-way Deep Neural Network. It learns multi-scale patterns from two types of scale-information, wavelet-based and downsampling-based, by eXtreme Gradient Boosting and Recurrent Convolutional Neural Network, respectively. After combining the learned patterns from the two-way, our model achieves state-of-the-art performance on FI-2010 and CSI-2016, where the latter is our published long-range stock dataset to help future studies for STP task. Extensive experimental results on the two datasets indicate that multi-scale information can significantly improve the STP performance and our model is superior in capturing such information.

PDF Details DOI

AAAI Conference 2020 Conference Paper

Visual Dialogue State Tracking for Question Generation

Wei Pang
Xiaojie Wang

GuessWhat? ! is a visual dialogue task between a guesser and an oracle. The guesser aims to locate an object supposed by the oracle oneself in an image by asking a sequence of Yes/No questions. Asking proper questions with the progress of dialogue is vital for achieving successful ﬁnal guess. As a result, the progress of dialogue should be properly represented and tracked. Previous models for question generation pay less attention on the representation and tracking of dialogue states, and therefore are prone to asking low quality questions such as repeated questions. This paper proposes visual dialogue state tracking (VDST) based method for question generation. A visual dialogue state is deﬁned as the distribution on objects in the image as well as representations of objects. Representations of objects are updated with the change of the distribution on objects. An object-difference based attention is used to decode new question. The distribution on objects is updated by comparing the question-answer pair and objects. Experimental results on GuessWhat? ! dataset show that our model signiﬁcantly outperforms existing methods and achieves new state-of-the-art performance. It is also noticeable that our model reduces the rate of repeated questions from more than 50% to 21. 9% compared with previous stateof-the-art methods.

PDF Details

EAAI Journal 2019 Journal Article

A new metric for individual stock trend prediction

Guang Liu
Xiaojie Wang

Individual stock trend prediction is extremely valuable for investment management. Previous studies mainly focused on proposing effective approaches to make profits. However, there is an ineffectiveness in model evaluation due to the inconsistency between model’s performance and profitability. We name this inconsistency profit bias. In order to address the profit bias in model evaluation, this paper proposes a new effective metric, Mean Profit Rate (MPR). The effectiveness of metric is measured based on the correlation between the metric value and profit of the model. Experiments on five stock daily index data among four countries show that MPR outperforms the classification metrics in correlating to profit. In view of these findings, we suggest that MPR is a more effective metric than the classification metrics in stock trend prediction.

Details DOI

TIST Journal 2019 Journal Article

Deep Reinforcement Learning for Vehicular Edge Computing

Zhaolong Ning
Peiran Dong
Xiaojie Wang
Joel J. P. C. Rodrigues
Feng Xia

The development of smart vehicles brings drivers and passengers a comfortable and safe environment. Various emerging applications are promising to enrich users’ traveling experiences and daily life. However, how to execute computing-intensive applications on resource-constrained vehicles still faces huge challenges. In this article, we construct an intelligent offloading system for vehicular edge computing by leveraging deep reinforcement learning. First, both the communication and computation states are modelled by finite Markov chains. Moreover, the task scheduling and resource allocation strategy is formulated as a joint optimization problem to maximize users’ Quality of Experience (QoE). Due to its complexity, the original problem is further divided into two sub-optimization problems. A two-sided matching scheme and a deep reinforcement learning approach are developed to schedule offloading requests and allocate network resources, respectively. Performance evaluations illustrate the effectiveness and superiority of our constructed system.

Details DOI

AAAI Conference 2019 Conference Paper

Differential Networks for Visual Question Answering

Chenfei Wu
Jinlai Liu
Xiaojie Wang
Ruifan Li

The task of Visual Question Answering (VQA) has emerged in recent years for its potential applications. To address the VQA task, the model should fuse feature elements from both images and questions efficiently. Existing models fuse image feature element vi and question feature element qi directly, such as an element product viqi. Those solutions largely ignore the following two key points: 1) Whether vi and qi are in the same space. 2) How to reduce the observation noises in vi and qi. We argue that two differences between those two feature elements themselves, like (vi − vj) and (qi − qj), are more probably in the same space. And the difference operation would be beneficial to reduce observation noise. To achieve this, we first propose Differential Networks (DN), a novel plug-and-play module which enables differences between pair-wise feature elements. With the tool of DN, we then propose DN based Fusion (DF), a novel model for VQA task. We achieve state-of-the-art results on four publicly available datasets. Ablation studies also show the effectiveness of difference operations in DF model.

PDF Details

AAAI Conference 2019 Conference Paper

Learning a Deep Convolutional Network for Colorization in Monochrome-Color Dual-Lens System

Xuan Dong
Weixin Li
Xiaojie Wang
Yunhong Wang

In the monochrome-color dual-lens system, the gray image captured by the monochrome camera has better quality than the color image from the color camera, but does not have color information. To get high-quality color images, it is desired to colorize the gray image with the color image as reference. Related works usually use hand-crafted methods to search for the best-matching pixel in the reference image for each pixel in the input gray image, and copy the color of the best-matching pixel as the result. We propose a novel deep convolution network to solve the colorization problem in an end-to-end way. Based on our observation that, for each pixel in the input image, there usually exist multiple pixels in the reference image that have the correct colors, our method performs weighted average of colors of the candidate pixels in the reference image to utilize more candidate pixels with correct colors. The weight values between pixels in the input image and the reference image are obtained by learning a weight volume using deep feature representations, where an attention operation is proposed to focus on more useful candidate pixels and a 3-D regulation is performed to learn with context information. In addition, to correct wrongly colorized pixels in occlusion regions, we propose a color residue joint learning module to correct the colorization result with the input gray image as guidance. We evaluate our method on the Scene Flow, Cityscapes, Middlebury, and Sintel datasets. Experimental results show that our method largely outperforms the state-of-the-art methods.

PDF Details

YNIMG Journal 2019 Journal Article

The effects of breastfeeding versus formula-feeding on cerebral cortex maturation in infant rhesus macaques

Zheng Liu
Martha Neuringer
John W. Erdman
Matthew J. Kuchan
Lauren Renner
Emily E. Johnson
Xiaojie Wang
Christopher D. Kroenke

Breastfeeding is positively associated with several outcomes reflecting early brain development and cognitive functioning. Brain neuroimaging studies have shown that exclusively breastfed children have increased white matter and subcortical gray matter volume compared to formula-fed children. However, it is difficult to disentangle the effects of nutrition in breast milk from other confounding factors that affect brain development, particularly in studies of human subjects. Among the nutrients provided by human breast milk are the carotenoid lutein and the natural form of tocopherol, both of which are selectively deposited in brain. Lutein is the predominant carotenoid in breast milk but not in most infant formulas, whereas infant formulas are supplemented with the synthetic form of tocopherol. In this study, a non-human primate model was used to investigate the effects of breastfeeding versus formula-feeding, as well as lutein and natural RRR-α-tocopherol supplementation of infant formula, on brain maturation under controlled experimental conditions. Infant rhesus macaques (Macaca mulatta) were exclusively breastfed, or were fed infant formulas with different levels and sources of lutein and α-tocopherol. Of note, the breastfed group were mother-reared whereas the formula-fed infants were nursery-reared. Brain structural and diffusion MR images were collected, and brain T2 was measured, at two, four and six months of age. The mother-reared breastfed group was observed to differ from the formula-fed groups by possessing higher diffusion fractional anisotropy (FA) in the corpus callosum, and lower FA in the cerebral cortex at four and six months of age. Cortical regions exhibiting the largest differences include primary motor, premotor, lateral prefrontal, and inferior temporal cortices. No differences were found between the formula groups. Although this study did not identify a nutritional component of breast milk that could be provided to infant formula to facilitate brain maturation consistent with that observed in breastfed animals, our findings indicate that breastfeeding promoted maturation of the corpus callosum and cerebral cortical gray matter in the absence of several confounding factors that affect studies in human infants. However, differences in rearing experience remain as a potential contributor to brain structural differences between breastfed and formula fed infants.

Details DOI

NeurIPS Conference 2018 Conference Paper

Chain of Reasoning for Visual Question Answering

Chenfei Wu
Jinlai Liu
Xiaojie Wang
Xuan Dong

Reasoning plays an essential role in Visual Question Answering (VQA). Multi-step and dynamic reasoning is often necessary for answering complex questions. For example, a question "What is placed next to the bus on the right of the picture? " talks about a compound object "bus on the right, " which is generated by the relation. Furthermore, a new relation including this compound object is then required to infer the answer. However, previous methods support either one-step or static reasoning, without updating relations or generating compound objects. This paper proposes a novel reasoning model for addressing these problems. A chain of reasoning (CoR) is constructed for supporting multi-step and dynamic reasoning on changed relations and objects. In detail, iteratively, the relational reasoning operations form new relations between objects, and the object refining operations generate new compound objects from relations. We achieve new state-of-the-art results on four publicly available datasets. The visualization of the chain of reasoning illustrates the progress that the CoR generates new compound objects that lead to the answer of the question step by step.

PDF Details

IJCAI Conference 2018 Conference Paper

Differentiated Attentive Representation Learning for Sentence Classification

Qianrong Zhou
Xiaojie Wang
Xuan Dong

Attention-based models have shown to be effective in learning representations for sentence classification. They are typically equipped with multi-hop attention mechanism. However, existing multi-hop models still suffer from the problem of paying much attention to the most frequently noticed words, which might not be important to classify the current sentence. And there is a lack of explicitly effective way that helps the attention to be shifted out of a wrong part in the sentence. In this paper, we alleviate this problem by proposing a differentiated attentive learning model. It is composed of two branches of attention subnets and an example discriminator. An explicit signal with the loss information of the first attention subnet is passed on to the second one to drive them to learn different attentive preference. The example discriminator then selects the suitable attention subnet for sentence classification. Experimental results on real and synthetic datasets demonstrate the effectiveness of our model.

PDF Details

NeurIPS Conference 2018 Conference Paper

KDGAN: Knowledge Distillation with Generative Adversarial Networks

Xiaojie Wang
Rui Zhang
Yu Sun
Jianzhong Qi

Knowledge distillation (KD) aims to train a lightweight classifier suitable to provide accurate inference with constrained resources in multi-label learning. Instead of directly consuming feature-label pairs, the classifier is trained by a teacher, i. e. , a high-capacity model whose training may be resource-hungry. The accuracy of the classifier trained this way is usually suboptimal because it is difficult to learn the true data distribution from the teacher. An alternative method is to adversarially train the classifier against a discriminator in a two-player game akin to generative adversarial networks (GAN), which can ensure the classifier to learn the true data distribution at the equilibrium of this game. However, it may take excessively long time for such a two-player game to reach equilibrium due to high-variance gradient updates. To address these limitations, we propose a three-player game named KDGAN consisting of a classifier, a teacher, and a discriminator. The classifier and the teacher learn from each other via distillation losses and are adversarially trained against the discriminator via adversarial losses. By simultaneously optimizing the distillation and adversarial losses, the classifier will learn the true data distribution at the equilibrium. We approximate the discrete distribution learned by the classifier (or the teacher) with a concrete distribution. From the concrete distribution, we generate continuous samples to obtain low-variance gradient updates, which speed up the training. Extensive experiments using real datasets confirm the superiority of KDGAN in both accuracy and training speed.

PDF Details

IJCAI Conference 2018 Conference Paper

Show and Tell More: Topic-Oriented Multi-Sentence Image Captioning

Yuzhao Mao
Chang Zhou
Xiaojie Wang
Ruifan Li

Image captioning aims to generate textual descriptions for images. Most previous work generates a single-sentence description for each image. However, a picture is worth a thousand words. Single-sentence can hardly give a complete view of an image even by humans. In this paper, we propose a novel Topic-Oriented Multi-Sentence (\emph{TOMS}) captioning model, which can generate multiple topic-oriented sentences to describe an image. Different from object instances or attributes, topics mined by the latent Dirichlet allocation reflect hidden thematic structures in reference sentences of an image. In our model, each topic is integrated to a caption generator with a Fusion Gate Unit (FGU) to guide the generation of a sentence towards a certain topic perspective. With multiple sentences from different topics, our \emph{TOMS} provides a complete description of an image. Experimental results on both sentence and paragraph datasets demonstrate the effectiveness of our \emph{TOMS} in terms of topical consistency and descriptive completeness.

PDF Details