Complexity in the case against accuracy estimation

Some authors have repeatedly pointed out that the use of the accuracy, in particular for comparing classifiers, is not adequate. The main argument concerns some assumptions of seldom validity or correctness underlying the use of this criterion. In this paper, we study the computational burden of the accuracy's replacement for building and comparing classifiers, using the framework of Inductive Logic Programming. Replacement is investigated in three ways: completion of the accuracy with an additional requirement, replacement of the accuracy with a bi-criterion recently introduced from statistical decision theory: the Receiver Operating Characteristic analysis, and replacement of the accuracy by a single criterion. We prove very hard results for most of the possible replacements. A first result shows that allowing the arbitrary multiplication of clauses appears to be totally useless. “Arbitrary” is to be taken in its broadest meaning, in particular exponential. The second point is the sudden appearance of the negative result, which is not a function of the criteria's demands. The third point is the equivalence in difficulty of all these different criteria. In contrast, the single accuracy's optimization appears to be tractable in this framework.

Authors

• Richard Nock Google Research

Keywords

No keywords are indexed for this paper.