Sony Developers Discuss Machine Learning in QA

Sony development leads recently presented their efforts to integrate AI and machine learning into the QA process at the CEDEC event in Yokohama, Japan. The talk, led by machine learning researchers Hiroyuki Yabe and Yutaro Miynotauchi, along with software engineer Nakahara Hiroki, focused on using real PS5 hardware to test titles more frequently and efficiently. Automated testing allowed teams to eliminate bugs earlier in the development cycle, as manual testing can only be conducted a few times per cycle. The team shared their findings on automating QA operations in Astro's Playroom, which required extensive QA testing for features like game progress integration with hardware functionality. The team had several conditions for integrating the technology, including not relying on game-specific tools and achieving a realistic cost. They developed two automated play systems: a Replay Agent and an Imitation Agent. The Replay Agent replicated exact button combinations, while the Imitation Agent reproduced human play with variance. The Imitation Agent used a machine learning model that could recreate human gameplay, allowing for testing of sections that couldn't be exactly reproduced. The team used human testers to play sections multiple times, uploading the data into the model. This enabled repeated testing of sections to ensure no bugs were overlooked. To assist the machine learning models, other AI systems like LoFTR were used to recognize scenes and help the system understand when to switch between agents. The team also introduced Class Balance to prioritize tasks with large impact, making the model more effective. The system has been refined, with benefits including improved efficiency in QA and earlier detection of bugs. While it still requires human input, the team has seen significant improvements in testing speed and accuracy. The use of machine learning in QA has integrated the process into the development cycle, allowing for more frequent testing and improving overall game quality. The development of machine learning systems in QA has made early detection and bug fixing more streamlined and effective, improving title quality and reducing bugs. Other developers can emulate this by developing their own machine learning modules, further enhancing the game development process.