Captum

Open source and free to read, modify and extend, Captum is a PyTorch library for model interpretability.

It helps you work out why a neural network produced a given prediction, and it covers several modalities, including vision and text.

Researchers can add their own attribution methods to it, which is the main reason to pick it over a closed explainability product.

Captum plugs into existing PyTorch models, so you do not have to rebuild your pipeline to start inspecting it.

It also includes tools for evaluating the convergence of attribution methods, a useful check when you want to know whether an explanation can be trusted rather than just plotted.

You need to be comfortable with PyTorch, and some models may need changes to the original network before the attribution methods will run on them.

The example material is very basic and may not cover your use case, so expect to read code and experiment.

As an interpretability tool, Captum suits developers and researchers who already work in PyTorch.