Mausam Mausam

NeurIPS Conference 2024 Conference Paper

AutoMix: Automatically Mixing Language Models

• Pranjal Aggarwal
• Aman Madaan
• Ankit Anand
• Srividya Pranavi Potharaju
• Swaroop Mishra
• Pei Zhou
• Aditya Gupta
• Dheeraj Rajagopal

Large language models (LLMs) are now available from cloud API providers in various sizes and configurations. While this diversity offers a broad spectrum of choices, effectively leveraging the options to optimize computational cost and performance remains challenging. In this work, we present AutoMix, an approach that strategically routes queries to larger LMs, based on the approximate correctness of outputs from a smaller LM. Central to AutoMix are two key technical contributions. First, it has a few-shot self-verification mechanism, which estimates the reliability of its own outputs without requiring extensive training. Second, given that self-verification can be noisy, it employs a POMDP based router that can effectively select an appropriately sized model, based on answer confidence. Experiments across five language models and five challenging datasets show that Automix consistently surpasses strong baselines, reducing computational cost by over 50\% for comparable performance.

PRL Workshop 2020 Workshop Paper

Symbolic Network: Generalized Neural Policies for Relational MDPs

• Sankalp Garg
• Aniket Bajpai
• Mausam Mausam

A Relational Markov Decision Process (RMDP) (Boutilier, Reiter, and Price 2001) is a first-order, predicate calculusbased representation for expressing instances of a probabilistic planning domain with a possibly unbounded number of objects. An RMDP domain has object types, relational state predicate and action symbols that are applied over objects, first order transition templates that specify probabilistic effects associated with action symbols, and a first-order reward structure. A domain instance additionally specifies a set of objects and a start state, thus defining a ground MDP with a known start state (Kolobov, Mausam, and Weld 2012)). Relational planners aim to produce a single generalized policy that can yield a ground policy for all instances of the domain, with little instance-specific computation. Domain-independent planners are representation-specific, but domain-agnostic, making them applicable to all domains expressible in the language. In this paper, we design a domain-independent relational planner. RMDP planners, in their vision, expect to scale to very large problem sizes by exploiting the first-order structures of a domain – thereby reducing the curse of dimensionality. Traditional RMDP planners attempted to find a generalized first-order value function or policy using symbolic dynamic programming (Boutilier, Reiter, and Price 2001), or by approximating them via a function over first-order basis functions (e. g. , (Guestrin et al. 2003; Sanner and Boutilier 2009)). Unfortunately, these methods met with rather limited success, for e. g. , no relational planner participated in International Probabilistic Planning Competition (IPPC)1 after 2006, even though all competition domains were relational. We believe that this lack of success may be due to the inherent limitations in the representation power of a basis function-based representation. Through this work, we wish to revive the research thread on RMDPs and explore if neural models could be effective in representing these first-order functions. We present Symbolic NetWork (S YM N ET), the first domain-independent neural relational planner that computes generalized policies for RMDPs that are expressed in the symbolic representation language of RDDL (Sanner 2010). S YM N ET outputs its generalized policy via a neural model whose all parameters are specific to a domain, but tied among all instances of that domain. So, on a new test instance, the policy can be applied out of the box using pre-trained parameters, i. e. , without any retraining on the test instance. S YM N ET is domain-independent because it converts an RDDL domain file (and instance files) completely automatically into neural architectures, without any human intervention. S YM N ET architecture uses two key ideas. First, it visualizes each state of each domain instance as a graph, where Copyright c 2020, Association for the Advancement of Artificial Intelligence (www. aaai. org). All rights reserved. 1 http: //www. icaps-conference. org/index. php/Main/ Competitions A Relational Markov Decision Process (RMDP) is a firstorder representation to express all instances of a single probabilistic planning domain with possibly unbounded number of objects. Early work in RMDPs outputs generalized (instanceindependent) first-order policies or value functions as a means to solve all instances of a domain at once. Unfortunately, this line of work met with limited success due to inherent limitations of the representation space used in such policies or value functions. Can neural models provide the missing link by easily representing more complex generalized policies, thus making them effective on all instances of a given domain? We present S YM N ET, the first neural approach for solving RMDPs that are expressed in the probabilistic planning language of RDDL. S YM N ET trains a set of shared parameters for an RDDL domain using training instances from that domain. For each instance, S YM N ET first converts it to an instance graph and then uses relational neural models to compute node embeddings. It then scores each ground action as a function over the first-order action symbols and node embeddings related to the action. Given a new test instance from the same domain, S YM N ET architecture with pre-trained parameters scores each ground action and chooses the best action. This can be accomplished in a single forward pass without any retraining on the test instance, thus implicitly representing a neural generalized policy for the whole domain. Our experiments on nine RDDL domains from IPPC demonstrate that S YM N ET policies are significantly better than random and sometimes even more effective than training a state-of-the-art deep reactive policy from scratch.

AAAI Conference 2012 Conference Paper

Dynamically Switching between Synergistic Workflows for Crowdsourcing

• Christopher Lin
• Mausam Mausam
• Daniel Weld

To ensure quality results from unreliable crowdsourced workers, task designers often construct complex workflows and aggregate worker responses from redundant runs. Frequently, they experiment with several alternative workflows to accomplish the task, and eventually deploy the one that achieves the best performance during early trials. Surprisingly, this seemingly natural design paradigm does not achieve the full potential of crowdsourcing. In particular, using a single workflow (even the best) to accomplish a task is suboptimal. We show that alternative workflows can compose synergistically to yield much higher quality output. We formalize the insight with a novel probabilistic graphical model. Based on this model, we design and implement AGENTHUNT, a POMDP-based controller that dynamically switches between these workflows to achieve higher returns on investment. Additionally, we design offline and online methods for learning model parameters. Live experiments on Amazon Mechanical Turk demonstrate the superiority of AGENTHUNT for the task of generating NLP training data, yielding up to 50% error reduction and greater net utility compared to previous methods.

IJCAI Conference 2007 Conference Paper

• Mausam Mausam
• Piergiorgio Bertoli
• Daniel S. Weld

Markov Decision Processes are a powerful framework for planning under uncertainty, but current algorithms have difficulties scaling to large problems. We present a novel probabilistic planner based on the notion of hybridizing two algorithms. In particular, we hybridize GPT, an exact MDP solver, with MBP, a planner that plans using a qualitative (non-deterministic) model of uncertainty. Whereas exact MDP solvers produce optimal solutions, qualitative planners sacrifice optimality to achieve speed and high scalability. Our hybridized planner, HybPlan, is able to obtain the best of both techniques --- speed, quality and scalability. Moreover, HybPlan has excellent anytime properties and makes effective use of available time and memory.

AAAI Conference 2006 Conference Paper

Probabilistic Temporal Planning with Uncertain Durations

• Mausam Mausam

Few temporal planners handle both concurrency and uncertain durations, but these features commonly co-occur in real-world domains. In this paper, we discuss the challenges caused by concurrent, durative actions whose durations are uncertain. We present five implemented algorithms, including ΔDUR