Skip to content

JGalego/awesome-safety-critical-ai

BranchesTags

Repository files navigation

Awesome Safety-Critical AI

Awesome mission license languages PR Welcome last commit

👋 Welcome to Awesome Safety-Critical AI - a curated space for practitioners, researchers and engineers committed to building intelligent systems that are as reliable as they are capable.

Inspired by systems thinking and engineering rigour, this project focuses on how to move from clever prototypes to fault-tolerant, production-grade ML systems.

Whether you're working on autonomous vehicles, medical diagnosis, or airbone systems, this collection offers a blueprint for AI that can be trusted when it matters most.

AI in critical systems is not about polishing demos or chasing benchmarks. It's about anticipating chaos - and designing for it.

This isn't just another (awesome) list. It's a call to action!

AI for Safety Critical

Table of Contents

🐇 Introduction

What is a critical system?

Critical systems are systems whose failure can lead to injury 🤕, loss of life ☠️, environmental harm 🌱🚱, infrastructure damage 🏗️💥, or mission failure 🎯.

Application Industry Vertical Description Failure Impact
Patient Monitoring Healthcare Tracks vital signs Failure can delay life-saving interventions
Aircraft Navigation Aerospace / Aviation Ensures safe flight paths Errors can lead to accidents
Power Grid Control Energy Manages electricity distribution Failures can cause blackouts
Command & Control Defence Coordinates military actions Failure risks national security
Industrial Automation Control Manufacturing Oversees production processes Malfunction can cause damage or injury
Core Banking System Finance Handles transactions and account data Downtime can affect financial operations

These systems are expected to operate with exceptionally high levels of safety, reliability and availability, often under unclear and unpredictable conditions.

They're the kind of systems we rarely think about... until something goes terribly wrong 🫣

Incident Year Description Root Cause Industry Vertical References
Therac-25 Radiation Overdose 1985–1987 Radiation therapy machine gave fatal overdoses to multiple patients Race conditions and lack of safety interlocks; poor error handling Healthcare Wikipedia, Stanford
Lufthansa Flight 2904 1993 Airbus A320 crashed during landing in Warsaw due to thrust reverser failure Reversers disabled by software logic when gear compression conditions weren't met Aviation Wikipedia, Simple Flying
Ariane Flight V88 1996 Ariane 5 rocket self-destructed seconds after launch Unhandled overflow converting 64-bit float to 16-bit integer Aerospace Wikipedia, MIT
Mars Climate Orbiter 1999 NASA probe lost due to trajectory miscalculation Metric vs imperial unit mismatch between subsystems Space Exploration NASA
Patriot Missile Failure 1991 Failed interception of Scud missile during Gulf War Rounding error in floating-point time tracking caused significant drift Defence Barr Group, GAO
Knight Capital Loss 2012 Trading system triggered erratic market orders causing massive financial loss Deployment of obsolete test code; no safeguards for live operations Finance / Trading Henrico Dolfing, CNN
Toyota Unintended Acceleration 2009–10 Reports of unexpected vehicle acceleration and crashes Stack overflow and memory corruption in embedded ECU software Automotive SAE, Wikipedia
F-22 Raptor GPS Failure 2007 Multiple jets lost navigation after crossing the International Date Line Software couldn't handle date transition; triggered reboot Aerospace / Defence FlightGlobal, Wikipedia
Heartbleed Bug 2014 Security vulnerability in SSL exposed private server data Improper bounds checking in the heartbeat extension of OpenSSL Cybersecurity / IT Heartbleed, CNET
T-Mobile Sidekick Data Loss 2009 Users lost personal data during server migration Software mishandling during data center transition led to irreversible loss Telecom / Cloud Services TechCrunch, PCWorld

When the stakes are this high, conventional software engineering practices must be complemented by rigorous verification, validation and certification processes that are designed to ensure system integrity.

Critical systems don't forgive shortcuts. Only engineering rigour stands between order and disaster.

TL;DR Critical systems are built on trust - and trust is built on rock-solid engineering.

AI in Critical Systems

So, where does that leave us? Is there room for AI in critical systems?

This isn't just a theoretical question - we're already well beyond the realm of hypotheticals.

From making life-or-death decisions in the ICU to controlling UAVs, performing surveillance and threat detection, and powering self-driving cars, intelligent systems aren't just emerging in these these domains - they're already fully operational.

Use Case Brief Description Industry Domain References
Predicting ICU Length of Stay AI models use patient data to forecast ICU duration, improving resource allocation and care planning. Healthcare INFORMS, Springer
AI in Radiation Therapy Planning Optimizes dose targeting using historical patient models, improving treatment precision and safety. Healthcare Siemens Healthineers
Self-Driving Cars Powers perception, decision-making, and control systems for autonomous vehicles. Transportation Built In, Rapid Innovation
Autonomous Drone Navigation Enables drones to navigate complex terrain without GPS; supports rescue and defense operations. Aerospace / Defense MDPI, Fly Eye
AI-Based Conflict Detection in ATC Forecasts aircraft trajectories to alert controllers of potential collision risks. Aerospace / Defense Raven Aero, AviationFile
Remote Digital Towers for Airports AI interprets visual data to assist air traffic controllers in low-visibility conditions. Aerospace / Defense Airways Magazine
Predictive Maintenance in Nuclear Reactors Analyzes reactor sensor data to detect early failures, preventing major accidents. Energy Accelerant, IAEA
AI-Assisted Reactor Control Systems Supports operators by modeling physical processes and recommending safety actions in real time. Energy Uatom.org, Springer
Autonomous Navigation for Cargo Ships Enables real-time path planning to avoid obstacles and optimize maritime routes. Transportation MaritimeEd, ShipFinex
AI-Based Collision Avoidance at Sea Detects and responds to high-risk vessel situations using visual and radar data fusion. Transportation Ship Universe
AI-Driven Fraud Detection Identifies anomalous financial transactions and flags potential fraud in real time. Financial Systems Upskillist, Xenoss
AI for Compliance Monitoring Uses NLP to parse documents and logs for regulatory breaches, supporting audits and governance. Financial Systems Digital Adoption, LeewayHertz
AI in Wildfire Early Detection Processes satellite and sensor data to detect hotspots and alert emergency services. Environmental Safety NASA FireSense, PreventionWeb

Building these systems is no walk in the park. ML brings powerful capabilities, but also adds layers of complexity and risk that need to be addressed through careful engineering.

While its ability to learn patterns and make data-driven decisions is unmatched in some domains, the adoption of AI in high-stakes environments must be tempered with caution, transparency, and a sharp understanding of its limitations.

Let's briefly recap some of the most important...

1. Models can and will make mistakes

Better models may make fewer mistakes, but mistakes are generally unavoidable.

Mistakes are not a sign of poor engineering - they are an intrinsic feature of intelligence.

Working with AI means accepting this uncertainty and designing systems that can handle it gracefully.

2. Mistakes can be strange and unpredictable

AI doesn't always fail in ways that make sense to us.

It might misclassify a stop sign with a sticker as a speed limit sign or switch treatment recommendations based on the user's language.

Unlike traditional software, which follows explicit rules, AI learns from data and generalises.

Generalization allows models to make predictions beyond what they've seen so far, but it's ultimately imperfect because the real world is messy, ever-changing, and rarely fits nicely into learned patterns.

3. Model outputs are often probabilitic

Traditional software is predictable: identical inputs yield identical outputs.

In contrast, ML models, especially those involving deep learning, can break this rule and exhibit probabilistic behavior.

Their outputs are a function not only of the input features, but also of things like model architecture, learned weights, training data distribution, hyperparameters (e.g. learning rate, batch size), optimization methods, and more.

That said, inference is often deterministic. Once trained, most models are capable of producing consistent outputs for a given input, assuming fixed weights and no funky runtime randomness.

This determinism means that systematic errors and biases are reproducible - models will consistently make the same mistakes.

Moreover, models trained on similar datasets often converge to similar representations, leading to shared failure modes and blind spots. So while ML systems may appear dynamic and random, their behavior can be quite predictable.

4. Data and models can change over time

Data and models are not static things. They'll evolve continuously due to changes in the environment, user behavior, hardware, regulations and more.

Imagine you're building a supervised learning system to detect early signs of pneumonia in chest X-rays.

Over time, several factors can cause both the data and the model to evolve:

  • Data Drift: the original training data may come from a specific hospital using a particular X-ray machine. As the system is deployed in new hospitals with different imaging equipment, patient demographics, or scanning protocols, the visual characteristics and quality of the X-ray images may vary significantly. This shift in the input distribution without an accompanying change in the task can reduce the model's diagnostic accuracy. This kind of drift doesn't actually crash the model, it just makes it quietly wrong.

  • Concept Drift: clinical knowledge and medical understanding can also evolve. For instance, new variants of respiratory diseases may present differently on X-rays, or diagnostic criteria for pneumonia may be updated. The relationship between image features and the correct diagnosis changes, requiring updates to the labeling process and model retraining.

  • Model Updates: the model is periodically retrained with new data to improve diagnostic performance or reduce false positives. These updates might involve changes in architecture, training objectives, or preprocessing steps. While performance may improve on average, these changes can introduce new failure modes and even regressions in certain edge cases. Model changes must be managed and monitored carefully, with rigorous testing and rollback plans.

  • External Factors: regulatory changes or clinical guidelines may require the model to provide additional outputs, such as severity scores or explainability maps. This requires collecting new types of annotations and modifying the model's output structure.

In safety-critical domains like medical imaging, the evolution of data and models is inevitable. As such, systems must be designed with this in mind, embedding mechanisms for monitoring, validation, and traceability at every stage.

By proactively addressing data and model drift, automating model updates and defining processes for dealing with external influences, teams can ensure that AI systems remain not only accurate but also trustworthy, transparent, and robust over time.

5. Zero-error performance is expensive and often impossible

Here's an uncomfortable truth: no AI system will ever be perfect.

No matter how sophisticated your architecture, how clean your data, or how rigorous your testing - your system will eventually encounter scenarios it can't handle.

The pursuit of perfection isn't just futile; it's dangerous because it creates a false sense of security. Perfection is a mirage.

Instead of chasing the impossible, safety-critical AI demands a different mindset: assume failure and design for it.

This means embracing design principles that prioritize resilience, transparency, and human-centered control:

  • Graceful Degradation: When AI fails - and it will - what happens next? Does the system shut down safely, fall back to simpler heuristics, or alert human operators? The difference between a minor incident and a catastrophe often lies in how elegantly a system handles its own limitations.

  • Human-AI Collaboration: AI doesn't have to carry the entire burden. The most reliable critical systems often combine AI capabilities with human oversight, creating multiple layers of validation and intervention. Think of AI as a highly capable assistant, not an infallible decision-maker.

  • Monitoring and Circuit Breakers: Just as electrical systems have circuit breakers to prevent dangerous overloads, AI systems need mechanisms to detect when they're operating outside their safe boundaries. Confidence thresholds, anomaly detection, and performance monitoring aren't nice-to-haves - they're essential safety features.

  • Failure Mode Analysis: Traditional safety engineering asks what could go wrong? and designs accordingly. AI systems demand the same rigor. What happens when your model encounters adversarial inputs, when data quality degrades, or when edge cases compound in unexpected ways?

The goal isn't to eliminate failure - it's to make failure safe, detectable, and recoverable. This isn't just good engineering practice; it's an architectural requirement that separates safe systems from disasters waiting to happen.

TL;DR When failure costs lives, AI must be engineered like a scalpel, not a sledgehammer.

The Bottom Line

The challenges we've outlined aren't insurmountable obstacles; they're design constraints that demand respect, discipline, and thoughtful engineering. Each limitation - from unpredictable failures to shifting data landscapes - represents an opportunity to build more robust, transparent, and trustworthy systems.

The question isn't whether AI belongs in critical systems - it's already there, making life-and-death decisions every day. The real question is: Are we developing these systems with the rigor they deserve?

This collection exists because we believe the answer must be an emphatic yes. It's an open call to build AI systems that don't just perform well in the lab, but earn trust where it matters most.

In critical systems, good enough isn't good enough. The stakes are too high for anything less than our best engineering.

"Do you code with your loved ones in mind?"

― Emily Durie-Johnson, Strategies for Developing Safety-Critical Software in C++

🌟 Editor's Choice

🔼 Back to top

🏃 TLDR

🔼 Back to top

If you're in a hurry or just don't like reading, here's a podcast-style breakdown created with NotebookLM (courtesy of Pedro Nunes 🙏)

Safety Critical AI Podcast

📝 Articles

🔼 Back to top

✍️ Blogs / News

🔼 Back to top

📚 Books

🔼 Back to top

📜 Certifications

🔼 Back to top

🎤 Conferences

🔼 Back to top

👩‍🏫 Courses

🔼 Back to top

📙 Guidelines

🔼 Back to top

🤝 Initiatives

🔼 Back to top

🛣️ Roadmaps

📋 Reports

🔼 Back to top

📐 Standards

🔼 Back to top

Generic

  • ANSI/UL 4600 > Standard for Evaluation of Autonomous Products
  • IEEE 7009-2024 > IEEE Standard for Fail-Safe Design of Autonomous and Semi-Autonomous Systems
  • ISO/IEC 23053:2022 > Framework for Artificial Intelligence (AI) Systems Using Machine Learning (ML)
  • ISO/IEC 23894:2023 > Information technology — Artificial intelligence — Guidance on risk management
  • ISO/IEC 38507:2022 > Information technology — Governance of IT — Governance implications of the use of artificial intelligence by organizations
  • ISO/IEC 42001:2023 > Information technology — Artificial intelligence — Management system
  • ISO/IEC JTC 1/SC 42 > Artificial intelligence
  • NIST AI 100-1 > Artificial Intelligence Risk Management Framework
  • SAE G-34 > Artificial Intelligence in Aviation

Coding

🛠️ Tools

🔼 Back to top

Adversarial Attacks

Data Management

Model Evaluation

Model Fairness & Privacy

  • Dstack-TEE/dstack: TEE framework for private AI model deployment with hardware-level isolation using Intel TDX and NVIDIA Confidential Computing
  • fairlearn/fairlearn: a Python package to assess and improve fairness of ML models
  • pytorch/opacus: a library that enables training PyTorch models with differential privacy
  • tensorflow/privacy: a library for training ML models with privacy for training data
  • zama-ai/concrete-ml: a Privacy-Preserving Machine Learning (PPML) open-source set of tools built on top of Concrete by Zama

Model Intepretability

  • MAIF/shapash: user-friendly explainability and interpretability to develop reliable and transparent ML models
  • pytorch/captum: a model interpretability and understanding library for PyTorch
  • SeldonIO/alibi: a library aimed at ML model inspection and interpretation

Model Lifecycle

  • aimhubio/aim: an easy-to-use and supercharged open-source experiment tracker
  • comet-ml/opik: an open-source platform for evaluating, testing and monitoring LLM applications
  • evidentlyai/evidently: an open-source ML and LLM observability framework
  • IDSIA/sacred: a tool to help you configure, organize, log and reproduce experiments
  • mlflow/mlflow: an open-source platform for the ML lifecycle
  • wandb/wandfb: a fully-featured AI developer platform

Model Security

Model Testing & Validation

Oldies 🕰️

  • pralab/secml: Python library for the security evaluation of Machine Learning algorithms

Bleeding Edge ⚗️

Just a quick note 📌 This section includes some promising, open-source tools we're currently testing and evaluating at Critical Software. We prioritize minimal, reliable, security-first, prod-ready tools with support for local deployment. If you know better ones, feel free to reach out to one of the maintainers or open a pull request.

  • agno-agi/agno: a lightweight library for building multi-modal agents
  • Arize-ai/phoenix: an open-source AI observability platform designed for experimentation, evaluation, and troubleshooting
  • BerriAI/litellm: all LLM APIs using the OpenAI format [Bedrock, Huggingface, VertexAI, TogetherAI, Azure, OpenAI, Groq, &c.]
  • browser-use/browser-use: make websites accessible for AI agents
  • Cinnamon/kotaemon: an open-source RAG-based tool for chatting with your documents
  • ComposioHQ/composio: equip's your AI agents & LLMs with 100+ high-quality integrations via function calling
  • deepset-ai/haystack: orchestration framework to build customizable, production-ready LLM applications
  • dottxt-ai/outlines: make LLMs speak the language of every application
  • DS4SD/docling: get your documents ready for gen AI
  • eth-sri/lmql: a programming language for LLMs based on a superset of Python
  • exo-explore/exo: run your own AI cluster at home with everyday devices 📱💻 🖥️⌚
  • FlowiseAI/Flowise: drag & drop UI to build your customized LLM flow
  • groq/groq-python: the official Python library for the Groq API
  • Giskard-AI/giskard: control risks of performance, bias and security issues in AI systems
  • guidance-ai/guidance: a guidance language for controlling large language models
  • h2oai/h2o-llmstudio: a framework and no-code GUI for fine-tuning LLMs
  • hiyouga/LLaMA-Factory: unified efficient fine-tuning of 100+ LLMs and VLMs
  • instructor-ai/instructor: the most popular Python library for working with structured outputs from LLMs
  • ItzCrazyKns/Perplexica: an AI-powered search engine and open source alternative to Perplexity AI
  • keephq/keep: open-source AIOps and alert management platform
  • khoj-ai/khoj: a self-hostable AI second brain
  • langfuse/langfuse: an open source LLM engineering platform with support for LLM observability, metrics, evals, prompt management, playground, datasets
  • langgenius/dify: an open-source LLM app development platform, which combines agentic AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production
  • latitude-dev/latitude-llm: open-source prompt engineering platform to build, evaluate, and refine your prompts with AI
  • microsoft/data-formulator: transform data and create rich visualizations iteratively with AI 🪄
  • microsoft/prompty: an asset class and format for LLM prompts designed to enhance observability, understandability, and portability for developers
  • microsoft/robustlearn: a unified library for research on robust ML
  • Mintplex-Labs/anything-llm: all-in-one Desktop & Docker AI application with built-in RAG, AI agents, No-code agent builder, and more
  • ollama/ollama: get up and running with Llama 3.3, DeepSeek-R1, Phi-4, Gemma 2, and other large LMs
  • promptfoo/promptfoo: a developer-friendly local tool for testing LLM applications
  • pydantic/pydantic-ai: agent framework / shim to use Pydantic with LLMs
  • run-llama/llama_index: the leading framework for building LLM-powered agents over your data
  • ScrapeGraphAI/Scrapegraph-ai: a web scraping python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents
  • stanfordnlp/dspy: the framework for programming - not prompting - language models
  • topoteretes/cognee: reliable LLM memory for AI applications and AI agents
  • unitaryai/detoxify: trained models and code to predict toxic comments
  • unslothai/unsloth: finetune Llama 3.3, DeepSeek-R1 and reasoning LLMs 2x faster with 70% less memory! 🦥

📺 Videos

🔼 Back to top

📄 Whitepapers

🔼 Back to top

👷🏼 Working Groups

🔼 Back to top

👾 Miscellaneous

🔼 Back to top

🏁 Meta

🔼 Back to top

  • safety-critical-systems GitHub topic
  • Awesome LLM Apps: a collection of awesome LLM apps with AI Agents and RAG using OpenAI, Anthropic, Gemini and opensource models
  • Awesome Python Data Science: (probably) the best curated list of data science software in Python
  • Awesome MLOps: a curated list of awesome MLOps tools
  • Awesome Production ML: a curated list of awesome open source libraries that will help you deploy, monitor, version, scale, and secure your production machine learning
  • Awesome Prompt Hacking: an awesome list of curated resources on prompt hacking and AI safety
  • Awesome Trustworthy AI: list covering different topics in emerging research areas including but not limited to out-of-distribution generalization, adversarial examples, backdoor attack, model inversion attack, machine unlearning, &c.
  • Awesome Responsible AI: a curated list of awesome academic research, books, code of ethics, courses, data sets, frameworks, institutes, maturity models, newsletters, principles, podcasts, reports, tools, regulations and standards related to Responsible, Trustworthy, and Human-Centered AI
  • Awesome Safety Critical: a list of resources about programming practices for writing safety-critical software
  • Common Weakness Enumeration: discover AI common weaknesses such as improper validation of generative AI output
  • FDA Draft Guidance on AI: regulatory draft guidance from the US Food & Drug Association, which regulates the development and marketing of Medical Devices in the US (open for comments until April 7th 2025)

About Us

🔼 Back to top

Critical Software is a Portuguese company that specializes in safety- and mission-critical software.

Our mission is to build a better and safer world by creating safe and reliable solutions for demanding industries like Space, Energy, Banking, Defence and Medical.

We get to work every day with a variety of high-profile companies, such as Airbus, Alstom, BMW, ESA, NASA, Siemens, and Thales.

If it's true that "everything fails all the time", the stuff we do has to fail less often... or not at all.

Are you ready to begin your Critical adventure? 🚀 Check out our open roles.

What Makes Us

Contributions

🔼 Back to top

📣 We're actively looking for maintainers and contributors!

AI is a rapidly developing field and we are extremely open to contributions, whether it be in the form of issues, pull requests or discussions.

For detailed information on how to contribute, please read our guidelines.

Contributors

🔼 Back to top

contributors

Citation

🔼 Back to top

If you found this repository helpful, please consider citing it using the following:

@misc{Galego_Awesome_Safety-Critical_AI,
  author = {Galego, João and Reis Nunes, Pedro and França, Fernando and Roque, Miguel and Almeida, Tiago and Garrido, Carlos},
  title = {Awesome Safety-Critical AI},
  url = {https://github.com/JGalego/awesome-safety-critical-ai}
}

About

When the stakes are high, intelligence is only half the equation - reliability is the other ⚠️

Topics

machine-learning awesome reliability artificial-intelligence awesome-list compliance red-team ai-safety safety-critical awesome-lists explainability responsible-ai ai-privacy trustworthy-ai genai

Resources

Readme

License

MIT license

Code of conduct

Code of conduct

Contributing

Contributing
Activity

Stars

43 stars

Watchers

8 watching

Forks

14 forks
Report repository

Releases 8

release-2025-09-01 Latest
Sep 1, 2025
+ 7 releases

Packages

No packages published

Contributors 9

Languages