Philosophy of Science and Explaining Machine Learning Models

Explanation

explain (verb)

1. To make known
2. To make plain or understandable
3. To give the reason for or cause of
4. To show the logical development or relationships of

Source: Merriam-Webster

Understanding

Understanding is an unending activity by which, in constant change and variation, we come to terms with and reconcile ourselves to reality, that is, try to be at home in the world.

Source: Hannah Arendt, Understanding and Politics (1954)

• Understanding is about finding familiar patterns and relationships in a complex world.
• Understanding can improve over time.
• Understanding is subjective and context-dependent.
• Good explanations often involve analogies to similar concepts and contrasts with different (often opposing) concepts.
• Both the similar and different concepts should be familiar to the audience.

Understanding

Software view of science

• A scientific theory is like a computer program that predicts our observations, the experimental data.
• A useful theory is a compression of the data; comprehension is compression.
• You compress things into computer programs, into concise algorithmic descriptions.
• The simpler the theory, the better you understand something.

Source: Gregory Chaitin, The Limits of Reason (2006)

Is Science Understandable?

• Science aims to explain complex phenomena using models as theories.
• Some scientific models are simple and intuitive (e.g., Newton’s laws).
• Others are complex and counterintuitive (e.g., quantum mechanics).

No one really understands quantum mechanics.

Source: Richard Feynman

Experimental Science vs Mathematics

Experimental Science

• Experimental science describes the world using guesses, known as hypotheses.

• For a hypothesis to be scientific, it must be testable / falsifiable.

• If we can design an experiment that could show the hypothesis to be false, it is scientific.

  • A hypothesis is falsified if the experimental data contradict it directly or any of its mathematically derivable consequences (simulated data).

• If we run a repeatable experiment and the hypothesis is not falsified, we say it is verified.

  • A hypothesis is verified if it is consistent with the repeated observations or if the observations can be deduced from it.

• Widely accepted and verified hypotheses are often called scientific theories or scientific laws.

• Science does not claim these laws are true, only that they have not yet been falsified.

Experimental Science vs Mathematics

Experimental Science

• If there is a finite number of hypotheses, we can eventually falsify all wrong hypotheses and find the correct one.
• In practice, hypotheses can be infinite in number and complex, making it impossible to find the correct one.
• Scientific theories evolve over time as new data emerges that falsifies old theories.
• There can be multiple competing scientific theories for the same phenomena.
• Scientific laws are compressions of experimental observations.

We never are right. We can only be sure we are wrong.

Source: Richard Feynman in The Feynman Lectures on Physics

Experimental Science vs Mathematics

Frequentist vs Bayesian Experimental Science

• In the frequentist approach to science, we reject the (null) hypothesis if the data is highly unlikely given the hypothesis.
• In the Bayesian approach, hypotheses are assigned prior probabilities based on existing belief.
• Experimental data is used to update these probabilities using Bayes’ theorem.
• Hypotheses with high posterior probabilities and good posterior predictive performance are considered more likely to be true (soft verification).
• Hypotheses with low posterior probabilities and poor posterior predictive performance are considered less likely to be true (soft falsification).
• The more informative experimental data that we collect, the more we can distinguish between competing hypotheses.

Experimental Science vs Mathematics

Frequentist vs Bayesian Experimental Science

• The hypothesis with the highest posterior probability can be selected as the best explanation of the data, but it is not guaranteed to be true.
• Bayesian methods allow us to quantify the uncertainty in our hypotheses which is useful for decision-making with complex models and/or limited data.
• Bayesian methods are more appropriate when data is limited, models are complex, or prior knowledge is available.
• Frequentist methods are often simpler and computationally less intensive.
• Both frequentist and Bayesian methods are appropriate when informative data is abundant and models are simple.

Experimental Science vs Mathematics

Mathematics

• Mathematicians compress their computational experiments into mathematical axioms.
• Axioms are irreducible (usually “obvious”) statements that cannot be proven true, but are accepted as true.
• From these axioms, mathematicians deduce theorems using logic alone. Logic itself has its own axioms which we need to accept.
• Theorems in mathematics (in contrast with scientific theories) are proven rather than verified, assuming the axioms are true.
• No experimental observations are needed.
• Mathematics is also the language by which scientists describe their hypotheses and deduce the hypotheses’ inevitable consequences to be compared with experimental data.

Experimental Science vs Mathematics

Mathematics

An example of mathematical axioms is the set of Euclid’s Axioms / Postulates in Euclidean geometry. These describe how points, lines, and shapes behave.

• Through any two distinct points, there is exactly one straight line.
• A straight line segment can be extended indefinitely.
• A circle can be drawn with any center and any radius.
• All right angles are equal.
• (Parallel postulate) Given a line and a point not on it, there is exactly one line through the point parallel to the given line.

Experimental vs Observational Science

• Experimental science involves designing and conducting experiments to test hypotheses about the natural world.
• Observational science involves collecting and analyzing data from natural settings without manipulating variables.
• Both approaches aim to understand complex phenomena, but experimental science allows for more controlled testing of causal relationships.
• Experimental science is generally more rigorous and reliable for establishing cause-and-effect relationships.
• Observational science is valuable for studying phenomena that cannot be ethically or practically manipulated, such as in ecology, astronomy, epidemiology, evolutionary biology, economics and social sciences.

Decision-making and Science

• Science is often a mean to an end: making decisions based on incomplete understanding of complex phenomena.
• Decision-making requires some level of belief or trust in the scientific models / theories used.
• But science does not directly prove hypotheses right!
• Decision-makers take the best available scientific theories that have not yet been falsified and assume/believe they are correct.
• If multiple competing theories exist, either select one of them or make decisions that are robust to the competing theories.

Biology and Complexity

• In biological systems, hypotheses are often complex and difficult to fully experimentally verify or falsify.
• Many models can fit the limited data available.
• How to make decisions responsibly in such situations?
• Scientists often rely on understandable approximations of biology in the form of mechanistic models.
• Mechanistic models can vary in complexity from simple ordinary differential equations (ODEs) to complex multi-scale simulations.
• These models are not perfect representations of reality, but they are useful tools for understanding (compressing the experimental data) and decision-making, e.g. dose selection.
• In pharmacometrics, individual mechanistic models are often combined with data-driven statistical models (random effects and residual error) to capture variability and uncertainty.

Understanding \(\to\) Trust

• When experimental data is limited, domain experts trust models they can understand.
• In quantum mechanics, understanding and intuition is limited but trust is high due to extensive experimental verification.
• Understandable models use familiar concepts and mechanisms, plausible assumptions and relationships between the variables that make sense to experts.
• Trust is essential for adoption of models in high-stakes decision-making (e.g., healthcare).
• Complex models (e.g., deep learning) can outperform simpler models but are often less understandable.
• How to explain complex models to increase trust?
• Should we fully trust complex models even if we can explain some of their behavior?
• Why should we trust the explanations themselves?

Explainability in ML

• There is no single agreed-upon definition of explainability / interpretability in ML.
• Explainability is often context-dependent and varies based on the audience and application.

Models can be loosely categorized as:

1. Intrinsically interpretable models

   • Simple models whose structure and parameters can be directly understood.
   • Examples: linear regression models with a few variables, decision trees with a few nodes, k-nearest neighbors with a small k.

2. “Black-box” models with post-hoc explainability, aka explainable models

   • Complex models that require additional techniques to explain their predictions.
   • Examples deep neural networks, ensemble methods (random forests, gradient boosting).

Types of Explanations

One may want to explain:

• The model for a specific set of inputs, aka local explainability, or
• The model’s overall behavior across all possible inputs, aka global explainability.

The explanation may focus on:

• The internal mechanics of the model, or
• Which inputs features are most influential in the model’s predictions.

The techniques used may be:

• Model-agnostic: applicable to any model type, or
• Model-specific: designed for a particular model architecture.

The field of explaining complex ML models is often called Explainable AI (XAI).

Benefits of Explaining ML Models

When explanation techniques work (they don’t always!), they can provide several benefits:

• Trust and Adoption: Users are more likely to trust and adopt models they can understand.
• Debugging and Improvement: Understanding model behavior helps identify errors and areas for improvement.
• Regulatory Compliance: Some industries require explanations for automated decisions (e.g., finance, healthcare).
• Ethical Considerations: Ensures models do not reinforce human biases or unfair practices, present in the data.
• Knowledge Discovery: Insights from models can lead to new scientific knowledge.

Spurious Correlations

• Complex models can learn spurious correlations present in the training data.
• Spurious correlations are relationships that appear in the data but lack a causal basis.
• Machine learning models may exploit these correlations to make accurate predictions.
• This is not the model’s fault! The model will use all possible associations it can find to make good predictions.
• However, relying on spurious correlations is undesirable and can lead to poor generalization to new data.
• Explanations can help identify when models rely on such spurious correlations.
• Once identified, steps can be taken to mitigate their impact (e.g., collecting more representative data, or modifying the model).

Spurious Correlations

Limitations and Dangers of Explaining ML Models

• Explanations often reveal associations in the model, not necessarily causal relationships.
• These can form the basis for causal hypotheses to be investigated but should not be taken at face value.
• Just because you can explain the inputs that influenced a prediction does not mean those inputs cause the outcome to change in reality.
• Changing the input will change the model’s prediction, but not necessarily the real-world outcome.
• Relying on explanations for decision-making without understanding the underlying data and context can lead to incorrect conclusions and decisions.

Limitations and Dangers of Explaining ML Models

• Explanations can be used to check that a model is not doing something unreasonable, e.g., relying on spurious correlations.
• However, just because a model did not do something unreasonable for many inputs does not mean it will never do so for any input, e.g. with distribution shifts or in adversarial examples.