Coupling AI and SMC-based algorithms: Inference of population structure from single genome sequencing analysis

The unprecedented availability of whole genome sequences has transformed population genetics, allowing researchers to reconstruct past demographic histories with greater resolution. Among the most widely used approaches for this purpose are methods grounded in the Sequentially Markovian Coalescent (SMC) framework, which leverage patterns of coalescence with recombination to infer fluctuations in effective population size over time. These fluctuations inform about past population trends, especially for endangered populations and extinct species.
However, despite their success, these methods often rest on simplifying assumptions. Most notably, random mating among all the individuals in a population (panmixia) assumption can lead to biased or misleading inferences when applied to populations with underlying genetic and/or geographical structure. For instance, changes in ancestral population size and population structure can lead to confounding signals in the historical coalescent rate, which presents an non identifiability problem, resulting in misleading interpretations that primarily affect the analysis in species of conservation interest.

After revising and detecting as systematic bias in SMC-based algorithms associated to population structure, we investigate the power of emerging summary statistics derived from whole genome data, proposing a novel deep learning framework that integrates the latent information contained in SMC transition matrices to distinguish between signals of population structure and true demographic change. By transforming these matrices into image-like representations, we develop a method based on deep convolutional neural networks (CNNs) in combination with perceptrons to classify demographic histories and predict key parameters of structured models, applying strategies of transfer learning. This approach enables discrimination between structured and panmictic populations and further infers specific features of the underlying demographic model. By blending the strengths of the modelling of coalescence with recombination of SMC-based approaches, deep learning, and coalescent simulations, this methodology offers a new avenue for scalable demographic inference considering structured populations.