It is time to hunt bugs! Let us talk about exploratory test

On exploratory testing projects, we will give you the freedom to explore an app/website by yourself to identify all possible bugs you can find. Fear not! We are not talking about insects or animals, a bug is a problem in a piece of software that causes unexpected results or makes the app/website crash.

Let’s tackle some common myths about this type of testing, so you can feel confident when you receive an exploratory cycle invitation.

Myth 1: There are no rules to follow in exploratory testing

WRONG! As in any other cycle, we always encourage our testers to read the complete description of the project – including the Overview and the In Scope/Out-of-Scope areas for testing. In fact, one of the main clues to become a skillful tester is your capability to read and follow instructions. Do not rush; if you miss something important in the instructions, you may waste your time!

Make sure you understand the complete description of the project before starting your hunt, and do not forget to read the Out of Scope Section carefully, as there might be parts of the website/app that you are not allowed to test.

Myth 2: It does not require previous testing knowledge

If you are a new tester, we do not want to discourage you from joining these cycles. On the contrary, we want to provide you with the necessary tools to succeed on your first hunt!

What is it that you should know before joining an exploratory project? You need to learn how to report bugs; we rely on your written skills and your capability to meticulously report issues so we can easily reproduce them afterwards by following your instructions.

A complete course about bug reporting is available at the Testerwork Training Center or Academy for you to review anytime!

Myth 3: Exploratory tests are time consuming or take longer to complete than other types of testing

When following the corresponding Overview instructions, exploratory testing should not take longer than other types of projects. In fact, by having the freedom to explore an app/software, you also have the possibility to set up your own schedule. Keep in mind that the key to success is not the quantity of errors you find, but the quality of your bug reports.

Bugs galore

Introduction: Despite extensive testing happening for software products, defects still pop and are seen by the end-users. If these were cosmetic defects, like spell errors or tiny defects, they would be tolerated. But when these are functional defects that are customer-facing, it is a cause of concern. These are the areas in which crowd testing can assist the community and the improvement of the software product by offering multiple hands to test with reasonable resources invested.

Here are a few real-world instances that I observed recently:

1. An investment arm of a banking organization incorrectly popping up spouse gender as “Male” instead of “Female”.
2. Video verification for Bank account opening KYC, failing multiple times due to browser and device compatibility issues, right at the time of getting introduced to the customer, leading to customer frustration.
3. In a stock trading application, the amount of monthly SIP being accepted is lower than the unit stock price, without any error message being prompted.
4. As a stock trading customer on a trading website, I was unable to locate the key functional requirements with ease.
5. Repeated SMS messages received for the same activity done by a customer.
6. Multiple SMS messages received with different content for the same activity done by a customer.
7. Not to mention all the spell errors noticed on the website of even well-established organizations.

All these are leading to a subject called Defect/Bug acceptance criteria, where bugs in software are acceptable to an extent by the end-user, acknowledging the fast-changing software application landscape and rapid evolution of software at the speed of a mobile button click.

 

Why are these defects occurring despite continuous testing?

Use case 1 – The integration between the Banking software and the Investment software was not tested efficiently.

Use case 2 – Browser compatibility and device compatibility were not verified to the extent required. The mobile app was given more priority than the identical website during the testing process when multiple service delivery channels were used.

Use case 3 – Field level validations were not performed, meaning an alert/error message was to be provided for the amount lesser the average stock price current market price to the investor in order to avoid such circumstances.

Use case 4 – The software was built without taking all the considerations of ease of App Learnability for the end-user.

Use case 5 – Frequent changes to the application even after UAT are performed in order to accommodate the product owner requirements.

All these and many more point towards the lack of extensive testing of the software and especially the continuous changes done to the software.

 

How can these be prevented :

Crowdsourced testing provides one of the effective solutions for preventing these defects:

1. The number of tests can be repeated across multiple browsers within a limited time frame.
2. The number of tests can be repeated across multiple devices, legacy and new, within a limited time frame.
3. Enables new pairs of hands and eyes every time the tests are run and are regressed.
4. Business-oriented testers may be a part of the crowd and hence business domain-centric tests can be executed efficiently.
5. Channel-based integration tests were accorded lower priority and hence missed out later, which could have been solved by using more testers from a crowd testing platform.
6. Extensive regression testing can be achieved by using a crowd testing platform for frequent software modifications and new feature enhancements and impact areas.

The bugs galore provide ample opportunity to crowd testers, but extensive testing prevents bugs from being rolled into production without adequate testing. Happy testing!

How to assess a bug’s risk

 

Finding bugs is an art. It’s the art of minimizing risk.

In some applications, the impact of critical bugs can result in loss of life, for example in healthcare, where an incorrect procedure on the wrong patient might mean physical death. In other applications, like banking, the existence of a critical bug might mean serious financial losses. Luckily, the bugs in most applications don’t result in such drastic losses. The riskiest impact of bugs in most software is the loss of reputation.

When you encounter a bug in software, it either stops you from achieving your goal, leaving you no option but to abandon the software and find an alternative. Or if there are non-critical bugs, the software is perceived as unprofessional, leaving the user doubting the capability of the organization behind the software to do what it is intended to do and to be trusted.

With so many options available for any one function you possibly would want to do, there is little incentive for a user to continue to struggle through errors in an attempt to complete a goal. It’s much easier to find an alternative without the bugs.
Finding bugs is thus no longer only crucial for risky software like healthcare, banking, or insurance products, but essential for any product to stand out and grow in a saturated market filled with options.

But how do you decide which bugs are critical and which are not? It’s not as easy as providing a checklist, as a spelling error in one application might be critical and in others considered trivial.

One of the most famous examples of how a spelling error resulted in an $18.5 million cost can be found in a spelling mistake in the Nasa software: on 22 July 1962, Nasa launched the spacecraft Mariner 1. A mere five minutes after lift-off, however, the mission was aborted and the spacecraft destroyed. Surely one of the most expensive failures in history, caused by a single missing hyphen in the code.
In most cases, however, a spelling error will probably have no impact on the users and most people might not even notice it. And if someone does, the biggest impact is that you might become the laughing stock of the public for a short period of time.

The key is thus to understand the impact of the error: what might be the consequence of leaving the bug in the software? To help you decide how risky the software is, ask yourself these questions before you log a bug next time:

1. Would there be a security impact? If the bug was encountered in production, would it be possible to stop someone from accessing assets they should be able to access? Or could an unauthorized person access it more easily?
2. Might there be a financial impact in either the form of a loss or a gain as a result of the bug? A simple calculation error might, for example, result in thousands of losses or gains as a result.
3. What about the reputational impact? What would people say and how would they behave if the error was left in the code?
   Would there be any impact on the ability to provide a service as a result of the bug? Does the existence of the bug stop someone from getting or providing the service customers expect?
4. Finally, is there an impact on any resources as a result of the bug? Would it cost more to do the same task? Would it take more time to achieve the same goal? Or would you need more people to do the same job as a result of the error?

When there is a substantial impact in any of these areas, it will very likely be considered a critical bug. Although the ideal is to have perfect software, the cost of fixing a trivial bug with a small impact far outweighs the cost of leaving it in the software.
Fixing, for example, a trivial bug of correcting a non-critical spelling error will result in the cost of the tester who reported the bug, the cost of the test manager reviewing the bug, the cost of the development team who needs to implement it in the code, and again the cost of another test cycle once the bug has been fixed. Compare this to the benefit and it is simply not worth the effort and cost.

So next time before you log a trivial bug, ask yourself these questions and evaluate whether the risk of leaving the bug in the software might outweigh the benefit and cost of fixing it. Not only will you get paid more for logging better bugs, but your tester rank will also go up, improving your chances of being selected for the next test cycle.