Can personal development app testing be left to AI? Verification results with metronome
An example of metronome development will be used to show how much personal development app testing can be entrusted to AI. We will summarize the passing of 109 automatic tests, the discovery of screen transition defects, the analysis of 96 BPM recordings, and the final human confirmation range.
If you are developing an app by yourself, it is not easy to find enough time for testing. This is because it is necessary to implement, check operation, record results, and investigate defects by one person.
As a result of using AI in the development of the metronome app, I evaluate that In the case of individual development, AI can be the main responsibility for testing.
Not only were they able to discover defects that humans had missed, but they were also able to confirm the accuracy of the rhythm using calculations and recording analysis. However, the final confirmation of specifications and results and the decision to publish are made by humans.
The range left to AI and the range confirmed by humans
In this development, we conducted tests at the following stages:
| testing stage | Build used |
|---|---|
| local | development build |
| internal test | Verification build |
| close test | Production build |
In this article, "testing using AI" includes not only terminal operation using AI, but also verification using automated testing and analysis processing. Not everything was confirmed solely through AI screen operations or visual judgment.
We left most of the tasks to AI, such as determining the testing perspective, creating specifications in Excel format, operating the terminal, determining pass/fail, taking screenshots, and confirming the reproduction of defects.
The basic flow is as follows.
- AI identifies test perspectives and creates test specifications in Excel format
- After AI review, a human confirms the final version of the specification
- AI runs the test on the verification device and records results and screenshots
- Compare test results and evidence, and after AI review, final confirmation by humans
For evidence, use screenshots taken on the verification device during test execution. Images will not be generated or processed by generation AI.
The drawings were checked by humans with the assistance of AI. AI is in charge of execution, and humans are responsible for checking the important points of specifications, results, and appearance.
Discover defects that others missed
One of the bugs discovered by the AI was If you perform a specific operation, you could transition to a screen that should have been discarded.
I won't reveal the specific operations or screen configuration, but it's an inconsistency that allows you to return to a screen that you shouldn't be able to reach. As far as we have confirmed, there was no actual harm to users, but we determined that there was a problem with screen state management that needed to be fixed.
This issue was missed in human testing.
The AI even specified the steps to reproduce it. We make revisions based on the content, retest it using AI, and then conduct final confirmation by humans.
In past testing, we also discovered and fixed an issue where the button display for 10 to 12 beats would break lines, and a UI issue where notifications would cover input/confirm operations.
When you implement a function yourself, you tend to check it by focusing on the expected operation. Based on this result, I feel that AI can not only perform the work for me, but also take on the role of picking up problems that I missed during my own checks.
Verifying rhythm accuracy through calculation and recording
The core of a metronome is its ability to accurately mark a rhythm at a set tempo. Therefore, we checked the calculated sounding position and the actual interval between the sounds.
Check internal clock calculations
For the internal clock that generates the sound, we calculated the expected position of the sound using a calculation method different from the implementation, and compared the results. This is because the same calculation mistake will not be carried over to the expected value.
Past internal clock tests have verified the following conditions:
| Verification target | conditions |
|---|---|
| Combination of tempo and number of divisions | 30, 31, 96, 137, 299, 300 BPM x number of divisions 1 to 4. Equivalent to 10 minutes each |
| long calculations | 137 BPM sixteenth note equivalent to 30 minutes |
Within this range, we confirmed that the error from the expected sounding position was within one sample, and that there was no overlap in the expected sounding position.
"Equivalent to 10 minutes" and "equivalent to 30 minutes" are calculation verification ranges. This is not the result of measuring with an actual device that continues to produce sound for the same amount of time.
Record and check the sound coming from the speakers
In addition to internal calculations, we recorded and analyzed the actual speaker sound with a microphone for about 1 minute at 96 BPM.
We detected 88 beats from the section excluding before and after the recording, and obtained the following results.
| Item | Measurement results |
|---|---|
| Setting BPM | 96 BPM |
| Estimated BPM | 95.998 BPM |
| average beat interval | 625.011 ms (theoretical value 625 ms) |
| standard deviation of beat interval | Approximately 1.32ms |
| Beat interval range | 622〜628 ms |
Under our conditions, we were able to confirm that the average tempo was approximately 96 BPM. No long intervals indicating missing beats were detected in the analyzed interval.
However, 95.998 BPM is an analytical estimate and does not guarantee measurement accuracy up to that order of magnitude. Detection is done in 2 ms increments and includes errors in the recording and analysis system, so it cannot be said that all of the observed variations are due to the app.
Also, the output sound was measured at 96 BPM for about 1 minute. Output sound at other tempos and cumulative deviation due to long playback have not been verified.
It was particularly useful to not only be able to judge with my ears that there was no problem, but also to be able to confirm it numerically. The fact that we were able to proceed with measurement and analysis with the support of AI led to the depth of confirmation.
What I confirmed in the production build
In this production build, 109 automated tests passed. Apart from this, I am verifying the following behavior on the terminal:
- Tempo change
- Playback operation when the screen is off
- Stop from notifications
- Retain settings after reboot
We also confirmed that the audio track works even when the lights are off, and that the buffer shortage for that track is 0 at the time of acquisition. This obtained result does not guarantee that there will be no sound breaks during the entire playback period.
Although quality cannot be determined by the number of tests performed alone, by combining automatic testing, terminal operation, and analysis of output sound, we were able to check from a different perspective.
Issues that remain when entrusting
Requires preparation and maintenance of execution environment
When running multiple emulators at the same time, the PC's CPU and memory are constraints. For actual devices, the connection method and operation tool compatibility status, and for cloud terminal environments, costs should also be considered.
In addition to building the initial environment, there is also the hassle of updating tests in response to changes in screens and specifications.
People also check the expected results and drawings.
If the expected result itself is wrong, no matter how many tests you run, you will not be able to evaluate it correctly. When entrusting development and testing to the same AI, care must be taken that the same specification interpretation may be inherited.
Therefore, the final confirmation of specifications and results is performed by humans. It is also important not to let the items you were unable to perform or judge to pass the exam.
Some drawing problems and discomfort during operation are difficult to judge from just a still image. We will leave the process of touching and checking the actual machine before releasing it to the public.
Specifications and screens also serve as information to be passed on to external parties.
Specifications and screenshots passed to the AI service may reveal undisclosed functions and business content. Particular care must be taken when handling entrusted projects or user data.
We separately check whether the data will be used for learning, how long it will be saved, and who can access it, and determine whether the contract and settings to be used meet the conditions of the project. The use of dummy data and test accounts is also something you should decide upon at the time of implementation.
In personal development, you can be primarily responsible for AI.
“Is it OK to use AI alone for app testing at the sole proprietor level?”
My answer, based on my experience, is thatmost of the testing work is left to you. However, the final confirmation and decision to publish is left to the person.
At the very least, the results were better than what I was able to achieve by myself. This is because we found a bug that we had overlooked and were able to verify everything from internal calculations to the actual output sound.
This does not indicate superiority or inferiority to general engineers or QA engineers. My experience was that I was able to use AI to compensate for my lack of testing skills and the limitations of doing both development and verification by myself.
Self-employed businesses require you to wear multiple hats in a limited amount of time. AI also requires preparation and confirmation, but by entrusting it with repetitive execution, recording, and problem investigation, you can spend your time making important decisions.
Ability to expand one person's workload and advance to confirmations that were previously unreachable. This is the most important value of AI testing when developing the metronome.