Article (Scientific journals)
Astraea: Grammar-based fairness testing
Soremekun, Ezekiel; Udeshi, Sakshi Sunil; Chattopadhyay, Sudipta
2022In IEEE Transactions on Software Engineering
Peer reviewed


Full Text
Publisher postprint (1.53 MB)

All documents in ORBilu are protected by a user license.

Send to


Keywords :
software fairness; machine learning; Software testing
Abstract :
[en] Software often produces biased outputs. In particular, machine learning (ML) based software is known to produce erroneous predictions when processing discriminatory inputs. Such unfair program behavior can be caused by societal bias. In the last few years, Amazon, Microsoft and Google have provided software services that produce unfair outputs, mostly due to societal bias (e.g. gender or race). In such events, developers are saddled with the task of conducting fairness testing. Fairness testing is challenging; developers are tasked with generating discriminatory inputs that reveal and explain biases. We propose a grammar-based fairness testing approach (called ASTRAEA) which leverages context-free grammars to generate discriminatory inputs that reveal fairness violations in software systems. Using probabilistic grammars, ASTRAEA also provides fault diagnosis by isolating the cause of observed software bias. ASTRAEA’s diagnoses facilitate the improvement of ML fairness. ASTRAEA was evaluated on 18 software systems that provide three major natural language processing (NLP) services. In our evaluation, ASTRAEA generated fairness violations at a rate of about 18%. ASTRAEA generated over 573K discriminatory test cases and found over 102K fairness violations. Furthermore, ASTRAEA improves software fairness by about 76% via model retraining, on average.
Disciplines :
Computer science
Author, co-author :
Soremekun, Ezekiel  ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal
Udeshi, Sakshi Sunil ;  Singapore University of Technology and Design, Singapore
Chattopadhyay, Sudipta;  Singapore University of Technology and Design, Singapore
 These authors have contributed equally to this work.
External co-authors :
Language :
Title :
Astraea: Grammar-based fairness testing
Publication date :
Journal title :
IEEE Transactions on Software Engineering
Publisher :
Peer reviewed :
Peer reviewed
Focus Area :
Security, Reliability and Trust
Available on ORBilu :
since 16 January 2023


Number of views
30 (1 by Unilu)
Number of downloads
23 (1 by Unilu)

Scopus citations®
Scopus citations®
without self-citations
WoS citations


Similar publications

Contact ORBilu