Our comprehensive evaluation of language model knowledge calibration reveals critical insights for the deployment of AI in microbiological research. The striking differences between top-performing models and those prone to hallucination underscore the importance of proper calibration mechanisms when handling specialized scientific knowledge.
Models that excel at recognizing fictional species and align their confidence with real-world information availability demonstrate the potential for AI to serve as reliable research assistants. Conversely, models that confidently describe non-existent bacteria or fail to acknowledge well-documented species pose risks for scientific misinformation. As language models become increasingly integrated into research workflows, these calibration metrics provide essential benchmarks for assessing model reliability and guiding future improvements in training methodologies.