Haku

Behavioral Coverage in Black Box Testing Using Generative Models

QR-koodi

Behavioral Coverage in Black Box Testing Using Generative Models

In the era of Artificial Intelligence (AI), autonomous vehicles are increasingly taking space in everyday life. From autonomous cars to delivery drones, Cyber Physical Systems (CPS) are everywhere, in every field. More than ever, safety is an important matter, as human error is responsible for more than nine car accidents out of ten. If autonomous vehicles are to replace humans for some daily tasks to reduce accidents or for any other reason, they have to be tested carefully. With a combination of hardware and AI, interactions can be difficult to understand and reproduce in real-life conditions. Moreover, the infinite number of scenarios that can happen in real situations does not help, as it is impossible to test them all, due to both the time and the money that testing takes. A popular method used to test those CPS driven by AI is test-falsification via scenario-based validation. This consists on trying to falsify requirements to see and detect which kinds of behaviors are problematic. This master's thesis aims to satisfy output coverage criteria problem for black-box systems, criteria which are sets of mutually exclusive requirements that reflect the different possible behaviors of CPS. The aim is to find a witness, i.e. inputs that create a test that falsifies the requirement, for each single requirement of the criterion. The algorithm proposed in this thesis runs multiple searches using multiple generative models at the same time. With the help of a multi-armed bandit selector that aims to reduce the number of executions by choosing inputs that have a good chance of falsifying remaining requirements of the output criterion, the algorithm tries to find a witness in as few executions as possible. Evaluated on three different problems, the proposed algorithm, named Test suite Generation for Output Coverage (TGOC) achieves satisfying results, being consistently able to find witnesses for output coverage criteria faster, i.e. with fewer executions, than random search or sequential falsification, with more requirements covered.

Tallennettuna: