In the right circumstances, using AI and ChatGPT for software testing ideas can be a lifesaver when you aren’t entirely sure where to start or need additional test ideas. In this blog, we’ll show how these tools can help test teams develop useful, actionable test ideas.
When I talk to our clients’ developers and testing specialists about how they develop ideas about what testing should cover, some themes readily emerge. One theme focuses on meeting business needs by emulating the actions and activities of a person using the software. The other is to base testing on documented requirements or acceptance criteria, presented as “the software must do this and not that.”
These seem reasonable starting points. However, there are many things beyond simple acceptance criteria that need testing. Depending on how an organization approaches testing, this might not be a problem if a team handles performance, accessibility, security testing and other non-functional requirements (NFRs). If there isn’t a team to handle NFRs, and testers have limited experience with them, how would they even know to think about them?
Another problem I see is companies assigning testers to a project with little understanding of how the software is used. Then, they brush this off, saying, “There are the requirements. Just test it.”
Story or function-level requirements rarely give testers unfamiliar with the application or business environment the information needed to do meaningful integration or regression testing and avoid potential problems.
A fair number of people are looking to ChatGPT, AI, machine learning and other trendy tools and ideas to help solve these challenges.
Can ChatGPT really help? Excellent question. I began experimenting with it to learn what it might – and might not – do for test ideas.
Using ChatGPT to Develop Test Ideas
ChatGPT is an artificial intelligence language model developed by OpenAI that uses advanced natural language processing (NLP) techniques to generate human-like responses to text prompts. Data scientists trained it on a massive dataset of text, allowing ChatGPT to learn the nuances and patterns of human language and generate natural-sounding responses.
We’ve seen how AI and ChatGPT can help with acceptance criteria for stories. Looking for ideas around testing is similar but slightly different. When test teams are unsure about how to test something, ChatGPT can help fill the gap. To make that work, the context of the problem, specifics to the situation, and the specific ask or prompt.
Let’s look at an example. A tester needs to test a signon screen. They give the context of their situation, “I am testing a user signon screen.” Then add specifics about their situation, like “requires a usersignon and password.” Then the specific ask, “What things would be good to test for?” Here’s what my request to ChatGPT looked like: If I am testing a user signon screen that requires a usersignon and password, what things would be good to test for?
The response I received from ChatGPT (version 3) looked like this:
When testing a user sign-on screen that requires a username and password, there are several important aspects to consider. Here are some key areas to focus on during your testing:
- Validation and error handling: Check if the system properly validates input fields, such as enforcing minimum and maximum length requirements, allowed characters, and format constraints. Test how the application handles incorrect or incomplete entries, and ensure appropriate error messages are displayed.
- Authentication: Verify that the sign-on process authenticates users correctly. Test various scenarios, including valid credentials, incorrect passwords, non-existent usernames, and account lockouts after multiple failed attempts.
- Password security: Assess the system’s password security measures. Verify if it enforces strong password policies (e.g., minimum length, complexity requirements) and handles password storage securely (e.g., using encryption and hashing).
- Remember me and session management: If the sign-on screen includes a “Remember me” option, test if it preserves the user session across browser sessions correctly. Verify that session timeouts and automatic logouts work as intended.
- Password recovery and reset: Evaluate the password recovery and reset functionality. Test whether the system allows users to reset their passwords securely, through mechanisms like email verification, security questions, or password reset links.
- Cross-browser and cross-device compatibility: Test the sign-on screen on various browsers (e.g., Chrome, Firefox, Safari, Edge) and devices (e.g., desktops, tablets, smartphones) to ensure consistent functionality and display across different platforms.
- Accessibility: Verify that the sign-on screen is accessible to users with disabilities. Test keyboard navigation, screen reader compatibility, and compliance with accessibility guidelines (e.g., WCAG 2.0 or 2.1).
- Usability and user experience: Evaluate the overall usability of the sign-on screen. Ensure that it provides clear instructions, an intuitive layout, and a user-friendly interface. Test for factors such as responsiveness, load times, and error feedback.
- Integration and error logging: If the sign-on screen communicates with other systems or APIs, test the integration points to ensure smooth data transfer and error handling. Monitor and verify if error logs are generated correctly for troubleshooting purposes.
- Security vulnerabilities: Conduct security testing to identify common vulnerabilities like SQL injection, cross-site scripting (XSS), cross-site request forgery (CSRF), and session hijacking. This helps ensure the sign-on screen is resilient to potential attacks.
- Performance and scalability: Test the performance of the sign-on screen under different loads, including scenarios with multiple concurrent users. Assess if the system scales well and doesn’t experience slowdowns or crashes under heavy usage.
|
Remember, this is a general guideline, and the specific requirements and context of your application might introduce additional areas for testing.
This isn’t a bad list of ideas. Most experienced testers would probably make a similar list of ideas as their starting point. One interesting thing to consider is the fifth point – password recovery and reset.
I did not mention the ability to change or recover passwords, but there is a presumption that some capability to do so should exist.
Using AI and ChatGPT in Software Testing for a Step-By-Step Process
You can also drill down into a response point and get more detailed information. We can reference the items ChatGPT generated, like point five, and ask the following: How would I test point 5 in detail?
ChatGPT returned the following response:
To test the password recovery and reset functionality in detail, you can follow these steps:
- Test password recovery options: Verify that the system provides multiple password recovery options, such as email verification, security questions, or password reset links. Test each option to ensure they work as intended.
-
- For email verification, submit a password recovery request and check if a password reset link or code is sent to the registered email address. Verify that the link or code expires after a certain time period.
- For security questions, answer the predefined questions and ensure the system validates the answers correctly before allowing password reset.
- If the system uses password reset links, test if the links are unique, expire after a certain period, and can only be used once.
- Test password reset process: Execute the password reset process and evaluate the following aspects:
-
- Validate that the password reset interface is user-friendly and intuitive, providing clear instructions on how to proceed.
- Submit a valid password reset link, code, or answer to security questions and ensure the system allows the user to set a new password.
- Verify that the password reset form enforces any password complexity requirements, such as minimum length, combination of uppercase and lowercase letters, numbers, and special characters.
- Test if the system properly handles invalid or expired password reset links, codes, or security question answers. It should display appropriate error messages and prevent unauthorized password resets.
- Test security measures: Assess the security measures implemented during the password recovery and reset process.
-
- Ensure that password reset links, codes, or security question answers are securely generated, transmitted, and stored, avoiding vulnerabilities like predictable or easily guessable values.
- Test if the system enforces appropriate account ownership verification, such as requiring the user to provide additional identifying information before allowing password reset.
- Check if the system detects and prevents abuse, such as excessive password reset attempts or brute-force attacks.
- Test edge cases and error handling: Conduct thorough testing of edge cases and error scenarios to ensure the system handles them properly.
-
- Test for scenarios where the user enters incorrect information during the password recovery process, such as providing the wrong email address, invalid security question answers, or expired password reset links. Ensure the system provides clear error messages and guides the user accordingly.
- Test concurrent password recovery/reset requests from the same account or different accounts to check if the system handles them without conflicts or security issues.
- Validate error handling and logging, ensuring that errors or exceptions during the password recovery/reset process are appropriately logged for debugging and troubleshooting purposes.
- Test usability and user experience: Evaluate the usability and user experience of the password recovery and reset process.
-
- Ensure that the password recovery/reset interface is well-designed, intuitive, and user-friendly.
- Test the responsiveness of the interface, especially when users access it from different devices or screen sizes.
- Consider conducting user acceptance testing or gathering feedback from representative users to assess the overall experience and identify any areas for improvement.
|
By following these steps, you can conduct thorough testing of the password recovery and reset functionality, ensuring that it is secure, reliable, and user-friendly.
Notice these are not test cases. Except for very general terms, there is no mention of input or expected results. Instead, they are ideas the test team can use to build out test cases, either manual or automated, when they are unfamiliar with the application or the intended business context.
Using AI and ChatGPT to Think Through Problems
AI and ChatGPT can help us think through problems we are not familiar with. It is not advanced to the point where it and similar tools will replace people any time soon. What it can do is help people work better – more efficiently. It cannot give reliable, detailed answers to these types of questions, nor will it generate code that anyone should rely on without scrutiny.
An experienced tester might easily produce a similar list of items with little or no trouble, though a novice might appreciate these ideas as a starting point. An experienced tester, likewise, can build their prompts to get very specific and potentially uncover ideas to test when they are at a mental impasse.
However, it may prove helpful when looking for test ideas in areas a tester or team is not familiar with.
Our white paper explains an effective system to ensure you always keep a human in the loop when implementing AI.
