Investigation on Java Mutation Testing Tools

Sara Tarek ElSayed Abbas - AL-Salam University College, Hay AL-khadra’a, Baghdad, 10022, Iraq
Rohayanti Hassan - Faculty of Engineering, Universiti Teknologi Malaysia, Johor Bharu, Johor, Malaysia
Shahliza Abd Halim - Faculty of Engineering, Universiti Teknologi Malaysia, Johor Bharu, Johor, Malaysia
Shahreen Kasim - Universiti Tun Hussein Onn Malaysia, Batu Pahat, Johor, Malaysia
Rohaizan Ramlan - Universiti Tun Hussein Onn Malaysia, Batu Pahat, Johor, Malaysia

Citation Format:



Software Testing is one of the most significant phases within the software development life cycle since software bugs can be costly and traumatic. However, the traditional software testing process is not enough on its own as some undiscovered faults might still exist due to the test cases’ inability to detect all underlying faults. Amidst the various proposed techniques of test suites’ efficiency detection comes mutation testing, one of the most effective approaches as declared by many researchers. Nevertheless, there is not enough research on how well the mutation testing tools adhere to the theory of mutation or how well their mutation operators are performing the tasks they were developed for. This research paper presents an investigative study on two different mutation testing tools for Java programming language, namely PIT and µJava. The study aims to point out the weaknesses and strengths of each tool involved through performing mutation testing on four different open-source Java programs to identify the best mutation tool among them. The study aims to further identify and compare the mutation operators of each tool by calculating the mutation score. That is, the operators’ performance is evaluated with the mutation score, with the presumption that the more prominent the number of killed mutants is, the higher the mutation score, thus the more effective the mutation operator and the affiliated tool. 


Mutation testing; mutation score; PIT tool; µJava tool, JUnit.

Full Text:



P. Mudholkar, M. Mudholkar, and S. Kulkarni, “Software testing,†Proc. Int. Conf. Work. Emerg. Trends Technol., pp. 1024–1024, Feb. 2010, doi: 10.1145/1741906.1742242.

R. Black, E. van Veenendaal, and D. Graham, Foundations of software testing : ISTQB certification. 2012.

S. Sangwan, “Software Testing Techniques and Strategies,†Isha Int. J. Eng. Res. Appl., vol. 4, no. 4, pp. 99–102, 2014.

P. Ammann and J. Offutt, Introduction to Software Testing. 2007.

J. H. Andrews, L. C. Briand, and Y. Labiche, “Is mutation an appropriate tool for testing experiments?,†Proc. - 27th Int. Conf. Softw. Eng. ICSE05, pp. 402–411, 2005, doi: 10.1145/1062455.1062530.

B. Falah, M. Akour, and S. Bouriat, “RSM: Reducing mutation testing cost using random selective mutation technique,†Malaysian J. Comput. Sci., vol. 28, no. 4, pp. 338–347, 2015, doi: 10.22452/MJCS.VOL28NO4.5.

S. Hamimoune and B. Falah, “Mutation testing techniques: A comparative study,†Nov. 2016, doi: 10.1109/ICEMIS.2016.7745368.

M. Al-Hajjaji, J. Krüger, F. Benduhn, T. Leich, and G. Saake, “Efficient Mutation Testing in Configurable Systems,†in Proceedings - 2017 IEEE/ACM 2nd International Workshop on Variability and Complexity in Software Design, VACE 2017, 2017, pp. 2–8, doi: 10.1109/VACE.2017.3.

P. Jung, S. Kang, and J. Lee, “Efficient regression testing of software product lines by reducing redundant test executions,†Appl. Sci., vol. 10, no. 23, pp. 1–21, Dec. 2020, doi: 10.3390/app10238686.

M. Hafiz, “Mutation Testing Tool for Java,†2008.

M. Delamaro and J. C. Maldonado, “Proteum - A Tool for the Assessment of Test Adequacy for C Programs User’s guide,†1996.

D. Singh and B. Suri, “Mutation testing tools-An empirical study,†IET Conf. Publ., vol. 2013, no. CP646, pp. 230–239, 2013, doi: 10.1049/CP.2013.2596.

Y. Jia and M. Harman, “An analysis and survey of the development of mutation testing,†IEEE Transactions on Software Engineering, vol. 37, no. 5. pp. 649–678, 2011, doi: 10.1109/TSE.2010.62.

M. S.GeethaDevasena and M. L. Valarmathi, “Search based Software Testing Technique for Structural Test Case Generation,†Int. J. Appl. Inf. Syst., vol. 1, no. 6, pp. 20–25, 2012, doi: 10.5120/ijais12-450185.

S. Anand et al., “An orchestrated survey of methodologies for automated software test case generation,†J. Syst. Softw., vol. 86, no. 8, pp. 1978–2001, Aug. 2013, doi: 10.1016/j.jss.2013.02.061.

M. Papadakis, C. Henard, M. Harman, Y. Jia, and Y. Le Traon, “Threats to the validity of mutation-based test assessment,†in ISSTA 2016 - Proceedings of the 25th International Symposium on Software Testing and Analysis, Jul. 2016, pp. 354–365, doi: 10.1145/2931037.2931040.

R. Gopinath, I. Ahmed, M. A. Alipour, C. Jensen, and A. Groce, “Does choice of mutation tool matter?,†Softw. Qual. J., vol. 25, no. 3, pp. 871–920, 2017, doi: 10.1007/s11219-016-9317-7.

M. Delahaye and L. Du Bousquet, “Selecting a software engineering tool: Lessons learnt from mutation analysis,†in Software - Practice and Experience, Jul. 2015, vol. 45, no. 7, pp. 875–891, doi: 10.1002/spe.2312.

M. Kintis, M. Papadakis, A. Papadopoulos, E. Valvis, and N. Malevris, “Analysing and comparing the effectiveness of mutation testing tools: A manual study,†in Proceedings - 2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation, SCAM 2016, Dec. 2016, pp. 147–156, doi: 10.1109/SCAM.2016.28.

B. Venners, “Test-Driven Development,†in A Conversation with Martin Fowler, Part V, 2002, p.

S. Kim, J. A. Clark, and J. A. McDermid, “Investigating the Effectiveness of Object-Oriented Strategies with the Mutation Method,†in Mutation Testing for the New Century, Springer, Boston, MA, 2001, pp. 4–4.

C. Wohlin, M. Höst, and K. Henningsson, “Empirical research methods in software engineering,†Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 2765, pp. 7–23, 2003, doi: 10.1007/978-3-540-45143-3_2.

W. Zheng, “Automatic Software Testing Via Mining Software Data,†2011.

S. Rani, B. Suri, and S. K. Khatri, “Experimental comparison of automated mutation testing tools for Java,†Dec. 2015, doi: 10.1109/ICRITO.2015.7359265.

H. Coles, T. Laurent, C. Henard, M. Papadakis, and A. Ventresque, “PIT: A practical mutation testing tool for Java (Demo),†ISSTA 2016 - Proc. 25th Int. Symp. Softw. Test. Anal., pp. 449–452, Jul. 2016, doi: 10.1145/2931037.2948707.

H. Coles, “GitHub - pitest/pitclipse: Mutation testing for Java in Eclipse IDE. Based on PIT (Pitest).†

H. Coles, “PIT - Mutation operators.†