One of my responsibilities in my job is functional safety. Roughly speaken, functional safety means that running the software does not induce any unaccaptable risk. Typical software applications which are safety relevant are avionics software, train control applications or medical devices. Today, the general approach to gain functional safety for an application is formed part of two basic things:
- to demonstrate, that the software works correctly, which means that it works as the requirements has stated, and
- that everthing is done to avoid errors rooted in human factors.
There are many safety standards in civil and military area, which prescribe methods to meet the objectives 1) and 2) in a repeatable, predictable and documented way. Many best practices of hardware and software engineering have influenced this standards.
The problem is, that there is an underlaying assumption, which is critical from my point of view: that the hardware/sofware systems are complete deterministic in the sense that any state of the system can be predicted and tested for any time point. We are familiar with this from the classical physics: if an differential equation of a system is given, we can - at least in principle - predict the system state for any time point.
But it is not hard to see that all systems getting more and more complicated. To prove functional safety requires more and more effort, and get more and more gaps. We know, that systems exist with very complicated behaviour: fractal systems or cellular automats are well-kown examples. Often, the only chance to predict their behaviour is to run the system from the beginning (the boudary conditions) up to the point of interest. And we know, that complicated interwoven structures like technical systems emerging today can behave unpredictable or "chaotic". They look like indeterministic systems (even if there is no indeterministic given in a strict mathematical sense).
And at this point, we are lost with our approach to functional safety. So what can we do ? Give up the objective to prove every function or behavior for correctness ! Huh, that would mean to allow errors, even not anticipated ones. Yes ! The future safe fault tolerant system (FSFTS)have to be designed that they can operate with errors, even unpredictable errors. Or in other words: the engineering methodology and practice have to be aligned to the objective that a system is safe not because we know every static and dynamic detail of it, but because errors does not harm the system (and if you see security issues as errors, it could be really interesting....).
Now, can we find conditions or characteristics of such systems, to find an start ? In fact, I always thought about it but I have no resilient conclusions yet. Many experiments have to be done for this. Anyway here are some basic assumptions I can share:
Assume, the state trajectory of a FSFTS is not known in detail for any time point, but it is limited in state space and constrainted by some attractors (well-known example: Lorentz attractor). In fact, a single line in state space will become a band of lines. The state trajectory is looped, but not necessarily closed in itself. Then, an error might move the system trajectory, but within this band, and the attractors ensure that the system will kept in its bounds.
In the next step, assume this sysem state trajectory will be partitioned in trajectories of several subsystems. Now if some attractors would influence some of other subsystems, we would get connected systems which would emerge in the sum functionality. In other words: I would expect that in future system design would mean to engineer attractors system state boundaries and trajectory bands. Engineering would mean not to engineer the system trajectory time point for time point, by cause and effect, but to engineer higher dynamics and (in behaviour) complex systems. I doubt that the available mathematical tools would help us here, I think we need some new basic insights in the area of complex chaotic systems. In this context it is also interesting to the progress and methods of system biology.
Well, we are far away from this, and many questions left. But I personally believe that will be the future of engineering.
