Professor Phil McMinn
PhD
School of Computer Science
Professor of Software Engineering
PGR lead
Head of the Testing research group
p.mcminn@sheffield.ac.uk
+44 114 222 1826
+44 114 222 1826
Regent Court (DCS)
Full contact details
Professor Phil McMinn
School of Computer Science
Regent Court (DCS)
211 Portobello
Sheffield
S1 4DP
School of Computer Science
Regent Court (DCS)
211 Portobello
Sheffield
S1 4DP
- Profile
-
Phil McMinn is a Professor of Software Engineering who specialises in software testing. He primarily works on developing automated techniques to assist software engineers in developing test suites that are effective at finding bugs and are efficient to maintain.
While he is well-known in the software testing field for his work in search-based automatic test data generation, his research has tackled a variety of problems including test flakiness, test oracles, and ensuring test quality through mutation analysis.
His work has been funded by Meta (Facebook), the EPSRC, and HEFCE.
- Research interests
-
- Software testing
- Search-based software engineering,
- Test oracles
- Test flakiness
- Mutation analysis
- Publications
-
Journal articles
- Summary of the 1st International Flaky Test Workshop (FTW 2024). ACM SIGSOFT Software Engineering Notes, 49(3), 35-36.
- Foreword to the 1st International Flaky TestsWorkshop 2024 (FTW2024). Proceedings - 2024 IEEE/ACM International Flaky Tests Workshop, FTW 2024, VI-VII.
- Empirically evaluating flaky test detection techniques combining test case rerunning and machine learning models. Empirical Software Engineering, 28. View this article in WRRO
- A Survey of the Metrics, Uses, and Subjects of Diversity-Based Techniques in Software Testing.. CoRR, abs/2311.09714.
- A Survey of Flaky Tests. ACM Transactions on Software Engineering and Methodology, 31(1), 1-74.
- Automatically identifying potential regressions in the layout of responsive web pages. Software Testing, Verification and Reliability. View this article in WRRO
- Automatic Detection and Removal of Ineffective Mutants for the Mutation Analysis of Relational Database Schemas. IEEE Transactions on Software Engineering, 45(5), 427-463. View this article in WRRO
- Random or evolutionary search for object‐oriented test suite generation?. Software Testing, Verification & Reliability. View this article in WRRO
- Effectively Incorporating Expert Knowledge in Automated Software Remodularisation. IEEE Transactions on Software Engineering. View this article in WRRO
- The Effectiveness of Test Coverage Criteria for Relational Database Schema Integrity Constraints. ACM Transactions on Software Engineering and Methodology, 25(1), 1-49. View this article in WRRO
- Design and analysis of different alternating variable searches for search-based software testing. Theoretical Computer Science, 605, 1-20. View this article in WRRO
- The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering, 41(5), 507-525.
- A Memetic Algorithm for whole test suite generation. Journal of Systems and Software, 103(C), 311-327. View this article in WRRO
- Automatic generation of valid and invalid test data for string validation routines using web searches and regular expressions. Science of Computer Programming, 97(4), 405-425.
- A Memetic Algorithm for whole test suite generation. Journal of Systems and Software. View this article in WRRO
- An orchestrated survey of methodologies for automated software test case generation. Journal of Systems and Software.
- An identification of program factors that impact crossover performance in evolutionary test input generation for the branch coverage of C programs. Information and Software Technology.
- Modelling complex biological systems using an agent-based approach.. Integr Biol (Camb), 4(1), 53-64.
- Input Domain Reduction through Irrelevant Variable Removal and its Effect on Local, Global and Hybrid Search-Based Structural Test Data Generation. IEEE Transactions on Software Engineering, 38(2), 453-477.
- Welcome Message from the Workshops Chair. 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.
- An empirical investigation into branch coverage for C programs using CUTE and AUSTIN. J SYST SOFTWARE, 83(12), 2379-2391. View this article in WRRO
- A Theoretical and Empirical Study of Search-Based Testing: Local, Global, and Hybrid Search. IEEE T SOFTWARE ENG, 36(2), 226-247.
- Empirical Evaluation of a Nesting Testability Transformation for Evolutionary Testing. ACM T SOFTW ENG METH, 18(3).
- Validation and discovery from computational biology models.. Biosystems, 93(1-2), 141-150.
- Editorial: Testing practice and research.. Softw. Test. Verification Reliab., 18, 69-70.
- Agent based modelling helps in understanding the rules by which fibroblasts support keratinocyte colony formation.. PLoS One, 3(5), e2129. View this article in WRRO
- An integrated systems biology approach to understanding the rules of keratinocyte colony formation.. J R Soc Interface, 4(17), 1077-1092.
- Editorial: Addressing industrial challenges - UKTest 2005 and beyond. SOFTW TEST VERIF REL, 16(3), 131-132.
- Development and validation of an agent-based computational model of normal human keratinocytes organisation in vitro. European Cells and Materials, 11(SUPPL.2), 28.
- Evolutionary testing using an extended Chaining Approach.. Evol Comput, 14(1), 41-64.
- Search-based software test data generation: a survey. SOFTW TEST VERIF REL, 14(2), 105-156.
- Effective automated repair of internationalization presentation failures in web applications using style similarity clustering and search‐based techniques. Software Testing, Verification and Reliability. View this article in WRRO
- Automated Visual Classification of DOM-based Presentation Failure Reports for Responsive Web Pages. Software Testing, Verification and Reliability.
Chapters
- An Empirical Comparison of EvoSuite and DSpot for Improving Developer-Written Test Suites with Respect to Mutation Score, Search-Based Software Engineering (pp. 19-34). Springer International Publishing
- Search based software engineering: Techniques, taxonomy, tutorial (pp. 1-59).
- Testability transformation - Program transformation to improve testability (pp. 320-344).
Conference proceedings papers
- Evaluating String Distance Metrics for Reducing Automatically Generated Test Suites. Proceedings of the 5th ACM/IEEE International Conference on Automation of Software Test (AST 2024)
- Batching Non-Conflicting Mutations for Efficient, Safe, Parallel Mutation Analysis in Rust. 2023 IEEE Conference on Software Testing, Verification and Validation (ICST), 16 April 2023 - 20 April 2023.
- Surveying the developer experience of flaky tests. Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice
- What do developer-repaired Flaky tests tell us about the effectiveness of automated Flaky test detection?. Proceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test
- Surveying the Developer Experience of Flaky Tests. 2022 IEEE/ACM 44th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 22 May 2022 - 24 May 2022.
- Automated Repair of Responsive Web Page Layouts. 2022 IEEE Conference on Software Testing, Verification and Validation (ICST), 4 April 2022 - 14 April 2022.
- Evaluating Features for Machine Learning Detection of Order- and Non-Order-Dependent Flaky Tests. 2022 IEEE Conference on Software Testing, Verification and Validation (ICST), 4 April 2022 - 14 April 2022.
- Diagnosability, Adequacy & Size: How Test Suites Impact Autograding. Proceedings of the Annual Hawaii International Conference on System Sciences, Vol. 2022-January (pp 861-870)
- An Empirical Study to Determine if Mutants Can Effectively Simulate Students' Programming Mistakes to Increase Tutors' Confidence in Autograding. Proceedings of the 52nd ACM Technical Symposium on Computer Science Education
- Gradeer: An Open-Source Modular Hybrid Grader. 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET), 25 May 2021 - 28 May 2021. View this article in WRRO
- The Influence of Test Suite Properties on Automated Grading of Programming Exercises. 2020 IEEE 32nd Conference on Software Engineering Education and Training (CSEE&T), 9 November 2020 - 12 November 2020.
- Hybrid Methods for Reducing Database Schema Test Suites. Proceedings of the IEEE/ACM 1st International Conference on Automation of Software Test
- STICCER: Fast and Effective Database Test Suite Reduction Through Merging of Similar Test Cases. 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), 24 October 2020 - 28 October 2020.
- Flake It 'Till You Make It. Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops
- The Influence of Test Suite Properties on Automated Grading of Programming Exercises. 2020 IEEE 32ND CONFERENCE ON SOFTWARE ENGINEERING EDUCATION AND TRAINING (CSEE&T) (pp 105-114)
- What factors make SQL test cases understandable for testers? A human study of automated test data generation techniques. 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME) (pp 437-448). Cleveland, OH, USA, 30 September 2019 - 4 October 2019. View this article in WRRO
- An empirical study on the use of defect prediction for test case prioritization. 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST). Xian, China, 22 April 2019 - 27 April 2019. View this article in WRRO
- Automatic Visual Verification of Layout Failures in Responsively Designed Web Pages. 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST), 22 April 2019 - 27 April 2019.
- Simulating Student Mistakes to Evaluate the Fairness of Automated Grading. 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET), 25 May 2019 - 31 May 2019. View this article in WRRO
- Search-based detection of deviation failures in the migration of legacy spreadsheet applications. Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis - ISSTA 2018, 16 July 2018 - 21 July 2018.
- DOMINO: Fast and Effective Test Data Generation for Relational Database Schemas. 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST) (pp 12-22), 9 April 2018 - 13 April 2018. View this article in WRRO
- Automated Repair of Internationalization Presentation Failures in Web Pages Using Style Similarity Clustering and Search-Based Techniques. Proceedings of the International Conference on Software Testing, Verification and Validation (ICST 2018) (pp 215-226), 9 April 2018 - 13 April 2018. View this article in WRRO
- Automated Repair of Mobile Friendly Problems in Web Pages. Proceedings of the 40th International Conference on Software Engineering (pp 140-150), 27 May 2018 - 3 June 2018. View this article in WRRO
- Using controlled numbers of real faults and mutants to empirically evaluate coverage-based test case prioritization. Proceedings of the 13th International Workshop on Automation of Software Test - AST '18, 28 May 2018 - 29 May 2018. View this article in WRRO
- Automated repair of layout cross browser issues using search-based techniques. Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis (pp 249-260). New York View this article in WRRO
- Evaluating CAVM: A New Search-Based Test Data Generation Tool for C (pp 143-149) View this article in WRRO
- ReDeCheck: an automatic layout failure checking tool for responsively designed web pages. Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis - ISSTA 2017, 10 July 2017 - 14 July 2017. View this article in WRRO
- XFix: an automated tool for the repair of layout cross browser issues. Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis - ISSTA 2017, 10 July 2017 - 14 July 2017. View this article in WRRO
- Disposable Testing: Avoiding Maintenance of Generated Unit Tests by Throwing Them Away. 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C), 20 May 2017 - 28 May 2017. View this article in WRRO
- Automated layout failure detection for responsive web pages without an explicit oracle. Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis - ISSTA 2017, 10 July 2017 - 14 July 2017. View this article in WRRO
- Automated Search for Good Coverage Criteria: Moving from Code Coverage to Fault Coverage Through Search-Based Software Engineering. SBST '16 Proceedings of the 9th International Workshop on Search-Based Software Testing (pp 43-44), 16 May 2016 - 17 May 2016. View this article in WRRO
- Virtual Mutation Analysis of Relational Database Schemas. AST '16 Proceedings of the 11th International Workshop on Automation of Software Test (pp 36-42), 14 May 2016 - 22 May 2016. View this article in WRRO
- Hitchhikers Need Free Vehicles! Shared Repositories for Statistical Analysis in SBST. SBST '16 Proceedings of the 9th International Workshop on Search-Based Software Testing (pp 55-56), 16 May 2016 - 17 May 2016. View this article in WRRO
- AVMf: An Open-Source Framework and Implementation of the Alternating Variable Method (pp 259-266) View this article in WRRO
- SchemaAnalyst: Search-Based Test Data Generation for Relational Database Schemas. 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2 October 2016 - 7 October 2016. View this article in WRRO
- mrstudyr: Retrospectively Studying the Effectiveness of Mutant Reduction Techniques. 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2 October 2016 - 7 October 2016. View this article in WRRO
- Automatic Detection of Potential Layout Faults Following Changes to Responsive Web Pages (N). 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), 9 November 2015 - 13 November 2015.
- Does Automated Unit Test Generation Really Help Software Testers? A Controlled Empirical Study. ACM Transactions on Software Engineering and Methodology, Vol. 24(4), 15 July 2013 - 20 July 2013. View this article in WRRO
- EXPOSE: Inferring worst-case time complexity by automatic empirical study. Proceedings of the International Conference on Software Engineering and Knowledge Engineering, SEKE, Vol. 2015-January (pp 730-731)
- Automatically Evaluating the Efficiency of Search-Based Test Data Generation for Relational Database Schemas. Proceedings of the 27th International Conference on Software Engineering and Knowledge Engineering
- Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges (T). 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), 9 November 2015 - 13 November 2015.
- Random or Genetic Algorithm Search for Object-Oriented Test Suite Generation?. Proceedings of the 2015 on Genetic and Evolutionary Computation Conference - GECCO '15, 11 July 2015 - 15 July 2015.
- Message from the chairs. 7th International Workshop on Search-Based Software Testing, SBST 2014 - Proceedings (pp iii)
- Message from the chairs. 7th International Workshop on Search-Based Software Testing, SBST 2014 - Proceedings (pp iii)
- Establishing the Source Code Disruption Caused by Automated Remodularisation Tools. 2014 IEEE International Conference on Software Maintenance and Evolution, 29 September 2014 - 3 October 2014.
- The Impact of Equivalent, Redundant and Quasi Mutants on Database Schema Mutation Analysis. 2014 14th International Conference on Quality Software, 2 October 2014 - 3 October 2014.
- A theoretical runtime and empirical analysis of different alternating variable searches for search-based testing. Genetic and Evolutionary Computation Conference (GECCO 2013) (pp 1445-1452). Amsterdam, 6 July 2013 - 10 July 2013.
- Towards the automatic identification of faulty multi-agent based simulation runs using MASTER. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 7838 LNAI (pp 143-156)
- Search-based testing of relational schema integrity constraints across multiple database management systems. Proceedings - IEEE 6th International Conference on Software Testing, Verification and Validation, ICST 2013 (pp 31-40)
- Efficient mutation analysis of relational database structure using mutant schemata and parallelisation. Proceedings - IEEE 6th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2013 (pp 63-72)
- Test Suite Generation with Memetic Algorithms. GECCO'13: PROCEEDINGS OF THE 2013 GENETIC AND EVOLUTIONARY COMPUTATION CONFERENCE (pp 1437-1444)
- Search-based propagation of regression faults in automated regression testing. Proceedings - IEEE 6th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2013 (pp 396-399)
- Evolving readable string test inputs using a natural language model to reduce human oracle cost. Proceedings - IEEE 6th International Conference on Software Testing, Verification and Validation, ICST 2013 (pp 352-361)
- Does automated white-box test generation really help software testers?. 2013 International Symposium on Software Testing and Analysis, ISSTA 2013 - Proceedings (pp 291-301) View this article in WRRO
- Supervised software modularisation. IEEE International Conference on Software Maintenance, ICSM (pp 472-481)
- Automated discovery of valid test strings from the web using dynamic regular expressions collation and natural language processing. Proceedings - International Conference on Quality Software (pp 79-88)
- Search-based test input generation for string data types using the results of web queries. Proceedings - IEEE 5th International Conference on Software Testing, Verification and Validation, ICST 2012 (pp 141-150)
- Symbolic search-based testing. 2011 26th IEEE/ACM International Conference on Automated Software Engineering, ASE 2011, Proceedings (pp 53-62) View this article in WRRO
- A multiobjective optimisation approach for the dynamic inference and refinement of agent-based model specifications. 2011 IEEE Congress of Evolutionary Computation, CEC 2011 (pp 2237-2244)
- Search-based software testing: Past, present and future. Proceedings - 4th IEEE International Conference on Software Testing, Verification, and Validation Workshops, ICSTW 2011 (pp 153-163)
- Message from the general chair. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 6956 LNCS
- Investigation of qualitative human oracle costs.. PPIG (pp 8-8)
- Optimizing for the number of tests generated in search based test data generation with an application to the oracle cost problem. ICSTW 2010 - 3rd International Conference on Software Testing, Verification, and Validation Workshops (pp 182-191)
- How does program structure impact the effectiveness of the crossover operator in evolutionary testing?. Proceedings - 2nd International Symposium on Search Based Software Engineering, SSBSE 2010 (pp 9-18)
- Using compression algorithms to support the comprehension of program traces. Proceedings of the 2010 Workshop on Dynamic Analysis, WODA 2010 (pp 8-13)
- Mutation operators for agent-based models. ICSTW 2010 - 3rd International Conference on Software Testing, Verification, and Validation Workshops (pp 151-156)
- Reducing qualitative human oracle costs associated with automatically generated test data. 1st International Workshop on Software Test Output Validation, STOV 2010, in Conjunction with the 2010 International Conference on Software Testing and Analysis, ISSTA 2010 (pp 1-4)
- Superstate identification for state machines using search-based clustering.. GECCO (pp 1381-1388)
- TAIC PART 2007 and Mutation 2007 special issue editorial. J SYST SOFTWARE, Vol. 82(11) (pp 1753-1754)
- Search-based failure discovery using testability transformations to generate pseudo-oracles. Proceedings of the 11th Annual Genetic and Evolutionary Computation Conference, GECCO-2009 (pp 1689-1696)
- Automated test data generation for coverage: Haven't we solved this problem yet?. TAIC PART 2009 - Testing: Academic and Industrial Conference - Practice and Research Techniques (pp 95-104)
- Co-testability Transformation. Dagstuhl Seminar Proceedings, Vol. 8351
- Handling dynamic data structures in search based testing. GECCO'08: Proceedings of the 10th Annual Conference on Genetic and Evolutionary Computation 2008 (pp 1759-1766)
- Co-testability Transformation. Dagstuhl Seminar Proceedings, Vol. 8051
- A theoretical & empirical znalysis of evolutionary testing and hill climbing for structural test data generation.. ISSTA (pp 73-83)
- The impact of input domain reduction on search-based test data generation. 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE 2007 (pp 155-164)
- A multi-objective approach to search-based test data generation. Proceedings of GECCO 2007: Genetic and Evolutionary Computation Conference (pp 1098-1105)
- The species per path approach to search-based test data generation. Proceedings of the 2006 International Symposium on Software Testing and Analysis, ISSTA 2006, Vol. 2006 (pp 13-23)
- Evolutionary testing of state-based programs. GECCO 2005: Genetic and Evolutionary Computation Conference, Vols 1 and 2 (pp 1013-1020)
- Hybridizing evolutionary testing with the chaining approach. GENETIC AND EVOLUTIONARY COMPUTATION GECCO 2004 , PT 2, PROCEEDINGS, Vol. 3103 (pp 1363-1374)
- The state problem for evolutionary testing. GENETIC AND EVOLUTIONARY COMPUTATION - GECCO 2003, PT II, PROCEEDINGS, Vol. 2724 (pp 2488-2498)
- Viscount: A Direct Method Call Coverage Tool for Java. Proceedings of the International Conference on Software Maintenance and Evolution (ICSME 2024)
- Private—Keep Out? Understanding How Developers Account for Code Visibility in Unit Testing. Proceedings of the International Conference on Software Maintenance and Evolution (ICSME 2024)
- PSEUDOSWEEP: A Pseudo-Tested Code Identifier. Proceedings of the International Conference on Software Maintenance and Evolution
- Exploring Pseudo-Testedness: Empirically Evaluating Extreme Mutation Testing at the Statement Level. Proceedings of the International Conference on Software Maintenance and Evolution 2024
- Do Automatic Test Generation Tools Generate Flaky Tests?. Proceedings of the International Conference on Software Engineering (ICSE) 2024
- An Investigation into the Effect of Control and Data Dependence Paths on Predicate Testability. 20th IEEE International Working Conference on Source Code Analysis and Manipulation
Preprints
- Improving the Reliability of Quantum Circuits by Evolving Heterogeneous Ensembles, arXiv.
- A Survey of the Metrics, Uses, and Subjects of Diversity-Based Techniques in Software Testing, arXiv.
- Do Automatic Test Generation Tools Generate Flaky Tests?, arXiv.
- Gradeer: An Open-Source Modular Hybrid Grader, arXiv.
- Grants
-
Current grants
- Test FLARE: Test Flakiness Automated Reproduction and Explanation, EPSRC, 10/2023 - 02/2027, £544,227, as PI
- RoarQ, RCUK, 05/2023 - 03/2026, £6,000, as PI
Previous grants
- Flake-It: Search-Based Inducement and Repair of Latent Test Flakiness, Industrial, 01/2020 - 09/2023, £40,000, as PI
- RE-PRESENT: Automatic Repair of Presentation Failures in Web Applications, EPSRC, 04/2020 - 03/2023, £34,717, as PI
- Insitute of Coding, HEFCE, 11/2017 - 03/2020, £957,000, as PI
- GReaTest: Growing Readable Software Tests, EPSRC, 03/2016 - 10/2018, £516,859, as Co-PI
- Reverse Engineering State Machine Hierarchies by Grammar Inference (REGI), EPSRC, 04/2009 - 09/2012, £315,209, as Co-PI
- Automated Discovery of Emergent Misbehaviour, EPSRC, 08/2009 - 07/2012, £241,737, as PI
- RE-COST: Reducing the Cost of Oracles for Software Testing, EPSRC, 01/2011 - 01/2014, £302,580, as PI
- Professional activities and memberships
-
- Member of the Testing research group
- SBST 2014 Chair
- ICSE 2014 Review Committee member
- ICSE 2013 Program Committee member
- ISSTA 2012 Program Committee member
- SSBSE 2011 General Chair
- Best paper winner at SSBSE 2010