Search PubMed⌕ Search

SEARCH · Search PubMed

Results for “genomic benchmark”

Search indexed PubMed citations on genomics, clinical trials, systematic reviews and public health. Explore titles, authors and supplied subject terms, then open the PubMed record.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 recordsLinked to original sources

A complete diploid human genome benchmark for personalized genomics.

Human genome resequencing typically involves mapping reads to a reference genome to call variants; however, this approach suffers from both technical and reference biases, leaving many duplicated and structurally polymorphic regions of the genome unmapped. Consequently, existing variant benchmarks, generated by the same methods, fail to assess these complex regions. To address this limitation, we present a telomere-to-telomere genome benchmark that achieves near-perfect accuracy (i.e. no detectable errors) across 99.4% of the complete, diploid HG002 genome. This benchmark adds 701.4 Mb of autosomal sequence and both sex chromosomes (216.8 Mb), totaling 15.3% of the genome that was absent from prior benchmarks. We also provide a diploid annotation of genes, transposable elements, segmental duplications, and satellite repeats, including 39,144 protein-coding genes across both haplotypes. To facilitate application of the benchmark, we developed tools for measuring the accuracy of sequencing reads, phased variant call sets, and genome assemblies against a diploid reference. Genome-wide analyses show that state-of-the-art de novo assembly methods resolve 2-7% more sequence and outperform variant calling accuracy by an order of magnitude, yielding just one error per 100 kb across 99.9% of the benchmark regions. Adoption of genome-based benchmarking is expected to accelerate the development of cost-effective methods for complete genome sequencing, expanding the reach of genomic medicine to the entire genome and enabling a new era of personalized genomics.

Journal Article↗

LAMBDA: A Prophage Detection Benchmark for Genomic Language Models.

Transformer-based genomic sequence models represent an emerging frontier in computational biology. Yet, their embeddings have not yet shown the same level of predictive power as natural and protein language models, indicating a gap between current implementations and theoretical promise. Existing benchmarks for DNA language models primarily focus on classifying regulatory elements in eukaryotic genomes, leaving open the fundamental question of whether these models learn sequence-level features across whole genomes. We introduce LAMBDA, a benchmark designed to rigorously evaluate genome language model embeddings through phage-bacteria sequence discrimination across four categories of increasing complexity: probing tasks, fine-tuning assessments, diagnostic tests, and genome-wide prophage detection. Our comprehensive analysis of current genomic language models provides novel insights into the importance of training data quality relative to model size, the need for domain-specific training, and the application of genomic language models for detecting prophage sequences. This benchmark represents a challenging genomic annotation task in the bacterial domain and addresses a key computational problem with direct relevance to microbiology and medicine.

DNA language model↗

LAMBDA: a prophage detection benchmark for genomic language models.

Transformer-based genomic sequence models represent an emerging frontier in computational biology. Yet, their embeddings have not yet shown the same level of predictive power as natural and protein language models, highlighting a gap between current implementations and theoretical promise. Existing benchmarks for DNA language models primarily focus on classifying regulatory elements in eukaryotic genomes, leaving open the fundamental question of whether these models learn sequence-level features across whole genomes. We introduce LAMBDA, a benchmark designed to rigorously evaluate genome language model embeddings through phage-bacteria sequence discrimination across four categories of increasing complexity: probing tasks, fine-tuning assessments, diagnostic tests, and genome-wide prophage detection. Our comprehensive analysis of current genomic language models provides insight into the importance of training data selection relative to model size, the need for domain-specific training, and the capabilities and limitations of genomic language models for detecting prophage sequences. This benchmark represents a challenging genomic annotation task in the bacterial domain and addresses a key computational problem with direct relevance to microbiology and medicine.

Prophages↗

Columba: fast approximate pattern matching with optimized search schemes.

MOTIVATION: Aligning sequencing reads to reference genomes is a fundamental task in bioinformatics. Aligners can be classified as lossy or lossless: lossy aligners prioritize speed by reporting only one or a few high-scoring alignments, whereas lossless aligners output all optimal alignments, ensuring completeness and sensitivity. RESULTS: This paper introduces Columba, a high-performance lossless aligner tailored for Illumina sequencing data. Columba processes single or paired-end reads in FASTQ format and outputs alignments in SAM format. By utilizing advanced search schemes and bit-parallel alignment techniques, Columba achieves exceptional speed. Columba is available in two variants. The first, based on the bidirectional FM-index, prioritizes speed. The second, Columba RLC, uses run-length compression using a bidirectional move structure, significantly reducing memory usage for large, repetitive datasets like pan-genomes. Benchmarks on the human genome, as well as bacterial and human pan-genome datasets, demonstrate that Columba is much faster than existing lossless aligners and even competitive with lossy tools. We integrated Columba into the OptiType HLA genotyping pipeline, where it substantially reduced computational time while maintaining accuracy. These results position Columba as a versatile, state-of-the-art tool for high-sensitivity genomic analyses. AVAILABILITY AND IMPLEMENTATION: The source code of Columba is available at https://github.com/biointec/columba under AGPL license. Scripts to reproduce the benchmarks and analyses are available at https://doi.org/10.5281/zenodo.15849246.

Software↗

Privacy-hardened and hallucination-resistant synthetic data generation with logic-solvers.

MOTIVATION: Machine-generated or synthetic data is a valuable resource for training artificial intelligence algorithms, evaluating rare workflows, and sharing data under stricter data legislations. However, current statistical and deep learning methods struggle with large data volumes, are prone to hallucinating scenarios incompatible with reality, and seldom quantify privacy meaningfully. RESULTS: Here, we introduce Genomator, a logic solving approach (SAT solving), which efficiently produces private and realistic representations of the original data. We demonstrate the method on genomic data, which arguably is the most complex and private information. We benchmark Genomator against state-of-the-art methodologies (Markov generation, Wasserstein Generative Adversarial Network and Conditional Restricted Boltzmann Machines), demonstrating a 40%-530% accuracy improvement and 57%-172% higher privacy. Genomator is also 3-100 times more efficient, making it the only tested method that scales to whole genomes. We show the universal trade-off between privacy and accuracy, and use Genomator's tuning capability to cater to all applications along the spectrum, from provable private representations of sensitive cohorts, to datasets with indistinguishable pharmacogenomic profiles. Demonstrating the production-scale generation of tuneable synthetic genomes hold great potential for balancing underrepresented populations in medical research and advancing global data exchange. AVAILABILITY AND IMPLEMENTATION: Genomator is available at https://github.com/csiro/genomator.

Algorithms↗

Predicting protein function by genomic context: quantitative evaluation and qualitative inferences.

Various new methods have been proposed to predict functional interactions between proteins based on the genomic context of their genes. The types of genomic context that they use are Type I: the fusion of genes; Type II: the conservation of gene-order or co-occurrence of genes in potential operons; and Type III: the co-occurrence of genes across genomes (phylogenetic profiles). Here we compare these types for their coverage, their correlations with various types of functional interaction, and their overlap with homology-based function assignment. We apply the methods to Mycoplasma genitalium, the standard benchmarking genome in computational and experimental genomics. Quantitatively, conservation of gene order is the technique with the highest coverage, applying to 37% of the genes. By combining gene order conservation with gene fusion (6%), the co-occurrence of genes in operons in absence of gene order conservation (8%), and the co-occurrence of genes across genomes (11%), significant context information can be obtained for 50% of the genes (the categories overlap). Qualitatively, we observe that the functional interactions between genes are stronger as the requirements for physical neighborhood on the genome are more stringent, while the fraction of potential false positives decreases. Moreover, only in cases in which gene order is conserved in a substantial fraction of the genomes, in this case six out of twenty-five, does a single type of functional interaction (physical interaction) clearly dominate (>80%). In other cases, complementary function information from homology searches, which is available for most of the genes with significant genomic context, is essential to predict the type of interaction. Using a combination of genomic context and homology searches, new functional features can be predicted for 10% of M. genitalium genes.

Bacterial Proteins↗

Operon prediction based on SVM.

The operon is a specific functional organization of genes found in bacterial genomes. Most genes within operons share common features. The support vector machine (SVM) approach is here used to predict operons at the genomic level. Four features were chosen as SVM input vectors: the intergenic distances, the number of common pathways, the number of conserved gene pairs and the mutual information of phylogenetic profiles. The analysis reveals that these common properties are indeed characteristic of the genes within operons and are different from that of non-operonic genes. Jackknife testing indicates that these input feature vectors, employed with RBF kernel SVM, achieve high accuracy. To validate the method, Escherichia coli K12 and Bacillus subtilis were taken as benchmark genomes of known operon structure, and the prediction results in both show that the SVM can detect operon genes in target genomes efficiently and offers a satisfactory balance between sensitivity and specificity.

Bacillus subtilis↗

Genetic screening of children for familial hypercholesterolaemia: the VRONI study.

BACKGROUND AND AIMS: The role of genetic testing as part of universal screening programmes for familial hypercholesterolaemia (FH) in children is not well defined. Here, a two-step approach to identify children carrying FH-causing variants was investigated. METHODS: In this study from Southern Germany, paediatricians were invited to offer FH screening to all children aged 4.8-14.9 years at routine paediatric examinations. The FH screening programme began in September 2020 in Bavaria and has involved up to 480 paediatricians. It included biochemical and genetic testing using 0.2 mL of blood taken from a fingertip. In case of low-density lipoprotein cholesterol (LDL-C) serum concentration ≥3.36 mmol/L (≥130 mg/dL), FH-causing variants were determined in the same sample with a focused panel covering most frequent variants (n = 48) and sequencing of relevant genes. RESULTS: Out of 25 431 children screened so far, 1689 children had an LDL-C ≥ 3.36 mmol/L (>130 mg/dL), which defined this concentration as the 93rd percentile. Pathogenic variants were identified by the focused panel in 157 and by next-generation sequencing in 283 children, respectively. While 17% (283/1670) of all genetically analysed children tested positive, the fraction of individuals with FH-causing variants increased across the spectrum of LDL-C serum concentrations from 4.7% (23/492) at 3.36-3.49 mmol/L (130-135 mg/dL) to 78.6% (81/103) above 5.17 mmol/L (200 mg/dL). Overall, the prevalence of FH-causing variants was high (1:90). One reason was a founder variant (n = 63) within the LDLR gene, found 40 times more frequent than European average. The analysis of recruitment data revealed significant ascertainment bias, with lower recruitment rate practices exhibiting higher prevalence. After adjustment for the bias using a generalized linear mixed model, the predicted prevalence was 1 in 163 (0.61%), which is highly consistent with large-scale genomic benchmarks as gnomAD (1:165, n = 622 057) and the UK Biobank (1:176, n = 48 741). CONCLUSIONS: The prevalence of FH determined in this study is significantly higher than previously published estimates (∼1:250), highlighting the importance of this condition for public health and supporting calls for a national paediatric screening programme, given the availability of effective treatment options. For children between 5 and 15 years, biochemical screening is an effective way to select patients for genetic testing, with sequencing of candidate genes being superior to variant screening. In summary, the VRONI study demonstrates the feasibility and efficacy of a combined biochemical and genetic screening for FH in children.

Humans↗

MIPS bacterial genomes functional annotation benchmark dataset.

MOTIVATION: Any development of new methods for automatic functional annotation of proteins according to their sequences requires high-quality data (as benchmark) as well as tedious preparatory work to generate sequence parameters required as input data for the machine learning methods. Different program settings and incompatible protocols make a comparison of the analyzed methods difficult. RESULTS: The MIPS Bacterial Functional Annotation Benchmark dataset (MIPS-BFAB) is a new, high-quality resource comprising four bacterial genomes manually annotated according to the MIPS functional catalogue (FunCat). These resources include precalculated sequence parameters, such as sequence similarity scores, InterPro domain composition and other parameters that could be used to develop and benchmark methods for functional annotation of bacterial protein sequences. These data are provided in XML format and can be used by scientists who are not necessarily experts in genome annotation. AVAILABILITY: BFAB is available at http://mips.gsf.de/proj/bfab

Bacterial Proteins↗

Genome-sequencing-based benchmarking of antimicrobial resistance, treatment outcomes and healthcare transmission events for Clostridioides difficile infection in Australian hospitals.

BACKGROUND: Clostridioides difficile infection (CDI) remains a priority for infection prevention and control in health care, particularly with the emergence of hypervirulent strains and antimicrobial resistance (AMR). AIM: To characterize the genomic epidemiology and AMR profiles of culture-confirmed CDI cases within tertiary hospitals in Australia. METHODS: A total of 155 C. difficile isolates from 142 patients with CDI diagnosed in four hospitals between 2023 and 2025 were studied. Data collected included patient demographics, severity of infection, antibiotic treatment and clinical outcomes at 8 weeks. Phenotypic susceptibility to vancomycin, fidaxomicin, metronidazole, moxifloxacin, meropenem, tetracycline and rifaximin were determined by agar dilution. Isolates underwent whole-genome sequencing (WGS) for genotyping and resistome assessment. FINDINGS: WGS differentiated 39 distinct sequence types among CDI isolates across different healthcare services. In total, 100 isolates were singletons and 55 (35% clustering rate) isolates were considered to be genomically related (difference of two or fewer single-nucleotide polymorphisms). Of these, 12 patients (8.5%) with close hospital contact formed six epidemiologically linked clusters. Phenotypic susceptibility results were obtained for 134 (86.4%) CDI isolates. There was no phenotypic resistance to vancomycin [minimum inhibitory concentration required to inhibit the growth of 90% of isolates (MIC90) 1 mg/L], metronidazole (MIC90 0.5 mg/L) or fidaxomicin (MIC90 0.5 mg/L). There was no association in the study cohort between the presence of resistance genes or reduced phenotypic susceptibility and CDI recurrence. CONCLUSION: Genomic analysis of C. difficile isolates did not identify any outbreaks or an association between the sequence type or presence of a resistance gene and clinical outcomes. High-resolution characterization and identification of antibiotic resistance, CDI clinical relapse and recent transmission offered by genome sequencing can provide important benchmarks for hospital infection control.

Antibiotic resistance↗

Developing and Benchmarking One Health Genomic Surveillance Tools for Influenza A Virus in Wastewater.

Influenza A viruses (IAV) remain a persistent One Health threat, and whole-genome sequencing from wastewater offers a promising surveillance tool. However, IAV is at low abundance in wastewater, making it difficult to sequence. We benchmarked four targeted enrichment methods suited for whole-genome sequencing including custom and off-the-shelf amplicon and probe-based methods. Our custom HA tiled-amplicon panel was sensitive, fast, and cost-effective, making it suitable for monitoring low-abundance seasonal variants of known subtypes. However, its reliance on conserved and intact primer-binding sites limited primer design to fewer subtypes. A previously published universal amplicon method targeted all IAV subtypes, but it performed poorly in wastewater due to its reliance on intact genome segments. Probe-capture methods were resilient to RNA degradation and mismatches, potentially enabling broader surveillance and detection of emerging strains. However, probes were costly, labor-intensive, and less sensitive than tiled-amplicon. When testing compatibility of sequencing methods with upstream virus concentration and extraction methods, ultrafiltration-based virus concentration outperformed large-volume direct extraction with all four sequencing methods. This set of benchmarking comparisons and custom panels provides needed information for the translation of IAV genomic sequencing into a routine component of wastewater surveillance.

Journal Article↗

Benchmarking PSI-BLAST in genome annotation.

The recognition of remote protein homologies is a major aspect of the structural and functional annotation of newly determined genomes. Here we benchmark the coverage and error rate of genome annotation using the widely used homology-searching program PSI-BLAST (position-specific iterated basic local alignment search tool). This study evaluates the one-to-many success rate for recognition, as often there are several homologues in the database and only one needs to be identified for annotating the sequence. In contrast, previous benchmarks considered one-to-one recognition in which a single query was required to find a particular target. The benchmark constructs a model genome from the full sequences of the structural classification of protein (SCOP) database and searches against a target library of remote homologous domains (<20 % identity). The structural benchmark provides a reliable list of correct and false homology assignments. PSI-BLAST successfully annotated 40 % of the domains in the model genome that had at least one homologue in the target library. This coverage is more than three times that if one-to-one recognition is evaluated (11 % coverage of domains). Although a structural benchmark was used, the results equally apply to just sequence homology searches. Accordingly, structural and sequence assignments were made to the sequences of Mycoplasma genitalium and Mycobacterium tuberculosis (see http://www.bmm.icnet. uk). The extent of missed assignments and of new superfamilies can be estimated for these genomes for both structural and functional annotations.

Algorithms↗

Benchmarking large language models for genomic knowledge with GeneTuring.

Large language models (LLMs) show promise in biomedical research, but their effectiveness for genomic inquiry remains unclear. We developed GeneTuring, a benchmark consisting of 16 genomics tasks with 1,600 curated questions, and manually evaluated 48,000 answers from ten LLM configurations, including GPT-4o (via API, ChatGPT with web access, and a custom GPT setup), GPT-3.5, Claude 3.5, Gemini Advanced, GeneGPT (both slim and full), BioGPT, and BioMedLM. A custom GPT-4o configuration integrated with NCBI APIs, developed in this study as SeqSnap, achieved the best overall performance. GPT-4o with web access and GeneGPT demonstrated complementary strengths. Our findings highlight both the promise and current limitations of LLMs in genomics, and emphasize the value of combining LLMs with domain-specific tools for robust genomic intelligence. GeneTuring offers a key resource for benchmarking and improving LLMs in biomedical research.

Benchmark↗

GPAC: benchmarking the sensitivity of genome informatics analysis to genome annotation completeness.

In view of the recent explosion in genome sequence data, and the 200 or more complete genome sequences currently available, the importance of genome-scale bioinformatics analysis is increasing rapidly. However, computational genome informatics analyses often lack a statistical assessment of their sensitivity to the completeness of the functional annotation. Therefore, a pre-analysis method to automatically validate the sensitivity of computational genome analyses with regard to genome annotation completeness is useful for this purpose. In this report we developed the Gene Prediction Accuracy Classification (GPAC) test, which provides statistical evidence of sensitivity by repeating the same analysis for five different gene groups (classified according to annotation accuracy level), and for randomly sampled gene groups, with the same number of genes as each of the five classified groups. Variability in these results is then assessed, and if the results vary significantly with different data subsets, the analysis is considered "sensitive" to annotation completeness, and careful selection of data is advised prior to the actual in silico analysis. The GPAC test has been applied to the analyses of Sakai et al., 2001, and Ohno et al., 2001, and it revealed that the analysis of Ohno et al. was more sensitive to annotation completeness. It showed that GPAC could be employed to ascertain the sensitivity of an analysis. The GPAC bendhmarking software is freely available in the latest G-language Genome Analysis Environment package, at http://www.g-language.org/.

Benchmarking↗

Response to: "best practices when benchmarking CATCH for the design of genome enrichment probes".

We clarify the design principles and evaluation choices underlying Syotti, a robust and scalable probe-design tool developed to support large, heterogeneous bacterial datasets with minimal parameter tuning. We highlight Syotti's ability to perform simultaneous large-scale designs and its effectiveness as a reliable alternative when existing tools such as CATCH are not well suited to the problem setting.

Genomics↗

Genomic biomarkers of toxicity.

The difference between measuring changes in gene expression levels in a toxicogenomics study and actually achieving reduced drug development costs is in the knowledge of which gene expression changes are predictive of toxicity. Whether the cost savings are in high-throughput screening, in discovery or in preclinical, testing, the application of genomics, proteomics or metabonomics will require a clear understanding of what these methods actually measure. Results of studies published over the past year have set valuable benchmarks for genomic studies, transforming this technology into a source of candidates for validation as predictive biomarkers of toxicity. This review will summarize some of the publications that have achieved this transformation, and the steps that are still needed for the implementation of predictive tests derived from these genomic biomarkers of toxicity.

Carrier Proteins↗

Score-based prediction of genomic islands in prokaryotic genomes using hidden Markov models.

BACKGROUND: Horizontal gene transfer (HGT) is considered a strong evolutionary force shaping the content of microbial genomes in a substantial manner. It is the difference in speed enabling the rapid adaptation to changing environmental demands that distinguishes HGT from gene genesis, duplications or mutations. For a precise characterization, algorithms are needed that identify transfer events with high reliability. Frequently, the transferred pieces of DNA have a considerable length, comprise several genes and are called genomic islands (GIs) or more specifically pathogenicity or symbiotic islands. RESULTS: We have implemented the program SIGI-HMM that predicts GIs and the putative donor of each individual alien gene. It is based on the analysis of codon usage (CU) of each individual gene of a genome under study. CU of each gene is compared against a carefully selected set of CU tables representing microbial donors or highly expressed genes. Multiple tests are used to identify putatively alien genes, to predict putative donors and to mask putatively highly expressed genes. Thus, we determine the states and emission probabilities of an inhomogeneous hidden Markov model working on gene level. For the transition probabilities, we draw upon classical test theory with the intention of integrating a sensitivity controller in a consistent manner. SIGI-HMM was written in JAVA and is publicly available. It accepts as input any file created according to the EMBL-format.It generates output in the common GFF format readable for genome browsers. Benchmark tests showed that the output of SIGI-HMM is in agreement with known findings. Its predictions were both consistent with annotated GIs and with predictions generated by different methods. CONCLUSION: SIGI-HMM is a sensitive tool for the identification of GIs in microbial genomes. It allows to interactively analyze genomes in detail and to generate or to test hypotheses about the origin of acquired genes.

Algorithms↗