Darrell Whitley

AAAI Conference 2022 Conference Paper

Prune and Tune Ensembles: Low-Cost Ensemble Learning with Sparse Independent Subnetworks

• Tim Whitaker
• Darrell Whitley

Ensemble Learning is an effective method for improving generalization in machine learning. However, as state-of-the-art neural networks grow larger, the computational cost associated with training several independent networks becomes expensive. We introduce a fast, low-cost method for creating diverse ensembles of neural networks without needing to train multiple models from scratch. We do this by first training a single parent network. We then create child networks by cloning the parent and dramatically pruning the parameters of each child to create an ensemble of members with unique and diverse topologies. We then briefly train each child network for a small number of epochs, which now converge significantly faster when compared to training from scratch. We explore various ways to maximize diversity in the child networks, including the use of anti-random pruning and onecycle tuning. This diversity enables “Prune and Tune” ensembles to achieve results that are competitive with traditional ensembles at a fraction of the training cost. We benchmark our approach against state of the art low-cost ensemble methods and display marked improvement in both accuracy and uncertainty estimation on CIFAR-10 and CIFAR-100.

TCS Journal 2014 Journal Article

Exact computation of the expectation surfaces for uniform crossover along with bit-flip mutation

• Francisco Chicano
• Darrell Whitley
• Enrique Alba

Uniform crossover and bit-flip mutation are two popular operators used in genetic algorithms to generate new solutions in an iteration of the algorithm when the solutions are represented by binary strings. We use the Walsh decomposition of pseudo-Boolean functions and properties of Krawtchouk matrices to exactly compute the expected value for the fitness of a child generated by uniform crossover followed by bit-flip mutation from two parent solutions. We prove that this expectation is a polynomial in ρ, the probability of selecting the best-parent bit in the crossover, and μ, the probability of flipping a bit in the mutation. We provide efficient algorithms to compute this polynomial for Onemax and MAX-SAT problems, but the results also hold for other problems such as NK-Landscapes. We also analyze the features of the expectation surfaces.

TCS Journal 2014 Journal Article

The component model for elementary landscapes and partial neighborhoods

• Darrell Whitley
• Andrew M. Sutton
• Gabriela Ochoa
• Francisco Chicano

Local search algorithms exploit moves on an adjacency graph of the search space. An “elementary landscape” exists if the objective function f is an eigenfunction of the Laplacian of the graph induced by the neighborhood operator; this allows various statistics about the neighborhood to be computed in closed form. A new component based model makes it relatively simple to prove that certain types of landscapes are elementary. The traveling salesperson problem, weighted graph (vertex) coloring and the minimum graph bisection problem yield elementary landscapes under commonly used local search operators. The component model is then used to efficiently compute the mean objective function value over partial neighborhoods for these same problems. For a traveling salesperson problem over n cities, the 2-opt neighborhood can be decomposed into ⌊ n / 2 − 1 ⌋ partial neighborhoods. For graph coloring and the minimum graph bisection problem, partial neighborhoods can be used to focus search on those moves that are capable of producing a solution with a strictly improving objective function value.

AAAI Conference 2013 Conference Paper

Greedy or Not? Best Improving versus First Improving Stochastic Local Search for MAXSAT

• Darrell Whitley
• Adele Howe
• Doug Hains

TCS Journal 2006 Journal Article

Subthreshold-seeking local search

• Darrell Whitley
• Jonathan Rowe

Algorithms for parameter optimization display subthreshold-seeking behavior when the majority of the points that the algorithm samples have an evaluation less than some target threshold. We first analyze a simple “subthreshold-seeker” algorithm. Further theoretical analysis details conditions that allow subthreshold-seeking behavior for local search algorithms using Binary and Gray code representations. The analysis also shows that subthreshold-seeking behavior can be increased by using higher bit precision. However, higher precision also can reduce exploration. A simple modification to a bit-climber is proposed that improves its subthreshold-seeking behavior. Experiments show that this modification results in both improved search efficiency and effectiveness on common benchmark problems.

AAAI Conference 1997 Conference Paper

Representation, Search and Genetic Algorithms

• Darrell Whitley

Wolpert and Macready’ s No Free Lunch theorem proves that no search algorithm is better than any other over all possible discrete functions. The meaning of the No Free Lunch theorem has, however, been the subject of intense debate. We prove that for local neighborhood search on problems of bounded complexity, where complexity is measured in terms of number of basins of attraction in the search space a Gray coded representation is better than Binary in the sense that on average it induces fewer minima in a Hamming distance 1 search neighborhood.

AIJ Journal 1996 Journal Article

Evaluating evolutionary algorithms

• Darrell Whitley
• Soraya Rana
• John Dzubera
• Keith E. Mathias

Test functions are commonly used to evaluate the effectiveness of different search algorithms. However, the results of evaluation are as dependent on the test problems as they are on the algorithms that are the subject of comparison. Unfortunately, developing a test suite for evaluating competing search algorithms is difficult without clearly defined evaluation goals. In this paper we discuss some basic principles that can be used to develop test suites and we examine the role of test suites as they have been used to evaluate evolutionary search algorithms. Current test suites include functions that are easily solved by simple search methods such as greedy hill-climbers. Some test functions also have undesirable characteristics that are exaggerated as the dimensionality of the search space is increased. New methods are examined for constructing functions with different degrees of nonlinearity, where the interactions and the cost of evaluation scale with respect to the dimensionality of the search space.