Mode Estimation in the Space of Phylogenetic Trees with Applications to Species Tree Reconstruction

Analyzing samples of phylogenetic trees arises in many settings, including bootstrap tree sets, Bayesian posterior samples, and collections of gene trees. The Billera–Holmes–Vogtmann (BHV) tree space provides a geometric framework in which such samples can be viewed as point clouds in a common metric space. A fundamental summary in this space is the Fréchet mean, but it has a property known as stickiness: mean trees tend to lie on lower-dimensional boundaries of the space, corresponding to unresolved, non-binary trees. This behavior can be undesirable, as the mean may then fail to represent the center of interest.

In this talk, I will introduce the BHV tree space framework and discuss mode estimation as an alternative way to summarize distributions of phylogenetic trees. After motivating the use of the mode, I will present simple approaches to mode estimation and discuss their consistency and robustness properties. I will then discuss how these ideas can be applied to species tree reconstruction from conflicting gene trees. To handle larger taxon sets, I will use quartet-based aggregation, in which local modal summaries are constructed from trees restricted to sets of four taxa and then combined to reconstruct a species tree. This approach provides a scalable way to apply mode estimation to trees with many taxa and helps reduce the influence of contamination in gene tree collections, as illustrated in simulation studies.