We evaluated 57 large language models on the hallucination benchmark of 200 fabricated binomials. Frontier systems confined moderate or extensive hallucinations to below one percent of prompts (0-0.5%), whereas the weakest models produced them on nearly two-thirds of queries (about 65%). Counting limited answers as partial hallucinations pushes the range to roughly 0.5-92%. This variability motivates the simple scoring scheme highlighted in the example above.

To compare models on a single scale we devised a "hallucination-avoidance" score, as illustrated in the score-calculation example. Each response earns three points if the model refuses or returns NA, two points for a self-declared limited answer, one point for intermediate, and zero when it claims extensive knowledge. We then sum those points for every fictional name and divide by the number of prompts, yielding an average that ranges from 0 (pure fantasy) to 3 (perfect restraint). Thus a model that usually says "I don't know" or "limited" on fake species will hover near the high end of the scale, while one that invents rich detail will drift toward zero.

While the top performers demonstrate a remarkable ability to recognize fictional taxonomy, the models at the opposite end of the spectrum reveal interesting patterns in knowledge assertion behavior. Some models consistently claim intermediate or extensive knowledge about nonexistent species, suggesting they may be generating plausible-sounding information rather than acknowledging uncertainty.

This disparity in performance highlights the importance of careful model selection for scientific applications. Models that readily fabricate details about fictional organisms may also be prone to hallucinating information about real but lesser-known species, potentially introducing errors into scientific workflows that depend on accurate microbial information.

Our comprehensive evaluation tested multiple language models using a diverse set of query templates. Each model's responses were categorized by knowledge level, from admitting no information (NA) to claiming extensive knowledge. The following analysis presents detailed performance metrics for all tested models, allowing direct comparison of their hallucination tendencies.

To better understand model performance patterns, we analyzed quality scores across different query templates. This stratified view reveals how models perform when asked different types of questions about fictional bacterial species. Models with consistent high scores across all templates demonstrate robust detection of hallucination regardless of query type.

Each query template tests a different aspect of model knowledge, from basic taxonomic information to specific physiological characteristics. The detailed breakdowns below show how each model performed on individual templates, revealing template-specific strengths and weaknesses. This granular view helps identify which types of queries are most likely to trigger hallucinations in different models.

Web-Aligned Knowledge: Real Bacterial Names vs. Google Counts

Using the methodology described above, we evaluated how well each model's confidence levels align with actual information availability on the web. Models that show strong correlation between their knowledge claims and web presence demonstrate better calibration—they're more confident about well-documented species and appropriately uncertain about obscure ones. The following visualization highlights the models with the best knowledge-web alignment scores.

While the top-performing models demonstrate excellent calibration between their knowledge claims and web presence data, there's significant variation across the model landscape. The gap between best and worst performers reveals important insights about knowledge calibration challenges in language models.

Models with poor web alignment correlation tend to exhibit overconfidence about obscure species or inappropriate uncertainty about well-documented ones. This misalignment suggests these models may have learned spurious patterns during training or lack proper calibration mechanisms. The following visualization shows models that need the most improvement in aligning their confidence with real-world information availability.

Our comprehensive evaluation of language model knowledge calibration reveals critical insights for the deployment of AI in microbiological research. The striking differences between top-performing models and those prone to hallucination underscore the importance of proper calibration mechanisms when handling specialized scientific knowledge.

Models that excel at recognizing fictional species and align their confidence with real-world information availability demonstrate the potential for AI to serve as reliable research assistants. Conversely, models that confidently describe non-existent bacteria or fail to acknowledge well-documented species pose risks for scientific misinformation. As language models become increasingly integrated into research workflows, these calibration metrics provide essential benchmarks for assessing model reliability and guiding future improvements in training methodologies.