Previously, we shared about DCube’s experiences transitioning from manual testing to automated testing. In this article, we’re excited to share with you the challenges we’ve faced (from then till now) and how to overcome them.
Without further ado, let’s dive straight in!
| Caveat: The challenges are mentioned in no particular order.
Challenge 1. When and what should we automate?
“Should we automate our tests right at the start or do we wait till there’s one to two features fully implemented?”
If you’ve been asking or hearing this question, you’re not alone! This is actually one of the most common questions we’ve heard/gotten, especially from the less experienced Quality Engineer (QE).
So, when should we start automating our tests? Let’s take a look at the diagram below.
The initial overhead of test automation is very expensive. However when the manual testing effort starts to grow exponentially, that’s when the development team will see and appreciate the value of test automation.
Our topmost priority is to deliver a high quality and useful product. How do we ensure that, especially when we’re running regular but shorter sprints with the application growing larger iteratively? How do we ensure the new application version is still working properly and accurately after integrating the newly-developed features? This is why QEs should lead the test automation direction; to consider all the pros and cons, estimate the positive and negative impacts of automating certain tests on the product as a whole. Don’t just because everyone can automate something doesn’t necessarily mean that anyone should.
Having said that, test automation is not solely QE’s job; we believe that software engineers should also contribute and share the responsibility to embark on the test automation journey… which brings us to the next challenge. 😄
Challenge 2. Should QE be the one who ensures the product quality in the team?
No, the QE is not the final gatekeeper for product quality. Well~ think back about the roles in a scrum team, there’s no single role known as the QE. QEs, along with software engineers/programmers are part of the development team.
Besides, QEs are human beings too and they may overlook certain things. This is why everyone in the development team has to do their part in testing and ensure their application is in tip-top condition. Well~ If the team themselves don’t even want to ensure the quality in the product they’re making, who will… right?
One possible method is to have a visible pre-release health check dashboard that reflects the team’s readiness to release the changes to production, and this dashboard should be visible to everyone in the team. Another way is the scalable test automation strategies where all developers in the team MUST own the test stack so that they can contribute to build automated tests earlier, often, and everywhere.
Oh! Speaking of quality… No one should be blamed should anything goes wrong in the product. Even if you want to blame someone, it has to be the whole development team because of #oneTeam.
Challenge 3. Which test data strategy is appropriate for my case?
Imagine you’re about to automate some tests for a newly-implemented system, how would you go about preparing the data for your test cases?
We’ve seen methods such as generating all test data set for test automation in one round. However it’s questionable on the amount of time needed to reach the stage of fulfilling all preconditions needed for subsequent test case. Also, this method may work for the first test case but might not work for the subsequent test cases because the test data could be modified into something not useable by the first test execution.
We prefer to use the refresh data source and/or the selfish data generation patterns where we set the precondition to be part of data seeding. These two patterns are used because certain test scenarios require a unique set of data, thus requiring some setting and cleaning up of data before each test scenario is executed. Also, the test scenarios are able to run in parallel as there are no dependency on one another, which also means shorter execution time.
Imagine you bought a product with a 100-page user guide. Will you read the whole guide? But what if the product has a self-explain function that clearly tells you how to use it while operating it, will it be better? This self-explain function in the product is a form of documentation.
Documentation comes in various forms. Gone are the days when engineers write and update manual test cases, thanks to the living documentation practice found in Test-driven development (TDD) and Behavior-Driven Development (BDD) frameworks! As the developers continue to develop the features, these two sets of documents will definitely get updated.
We have these frameworks in place since two years ago and they’ve helped us to be more productive! The teams started writing both TDD and BDD and they soon realised that they don’t have to write user-acceptance testing (UAT) scripts all over again. Instead, they used the BDD as the UAT scripts and this gave the team more time to resolve any bugs before the UAT started.
Of course, to start anything new in an organisation is difficult and it requires lots of convincing. It took us a substantial amount of time to convince everyone in the team to adopt and pick up these frameworks. But now with the living documentation practice, we’ll always have the latest version of the documentation without needing to invest long hours/resources to maintain it!
Challenge 5. Are QEs required to have some infrastructure knowledge since they’ve to spin up test environments?
Not really, but it will be a bonus if they do. Setting up environments (to run tests or application under test) isn’t difficult; anyone can do their research/readings online.
As a QE, setting up environment isn’t a one-time effort kind of thing. You’ll need to know how to troubleshoot and find out the problem(s) if there’s something wrong with the automated tests. After the environment has been setup, you’d need to know how to maintain it (eg. update of web drivers, etc). Of course, all these can be googled online. But that could take some time and slow down the overall team progress. Also, having some infrastructure knowledge will definitely help the QEs in better collaboration with the software engineers because, you know, #weSpeakTheSameLanguage~
To overcome this challenge in DCube, QEs could try to leverage on the services provided by commercial cloud providers if the products are hosted on cloud. Test environments can be setup with just a few clicks. If any QE has any issues with creating new test environments or unsure of how to scale up their current test environment(s) using cloud, our friendly engineers in DCube are always there to lend a helping hand.
Challenge 6. Difficult to acquire or train skilled QE(s)
To recruit full stack QEs is a challenging task — this role not only has to be strong in technical skill-set and s/he sits at the intersection between the software engineers, designers, product managers, and customers. Thus, having the ability to articulate ideas to different groups of people is a must.
QEs must be willing to stay up-to-date with the latest technologies and trends of development to stay in touch with what the development team may be doing or moving towards. Simultaneously, a QE must also understand how to keep a holistic view of what the application in development is trying to achieve from a business perspective, and keep test cases relevant to those goals to build a world class application the team would be proud of.
QEs are, admittedly, one of the toughest technical roles to hire for. Nevertheless, the effort to find the suitable hire will be significantly rewarded.
Nobody is born a master in test automation. Everyone will make mistakes and we should learn how to reduce or avoid common pitfalls. QE faces challenges like any other startup — when the team is growing larger, we need to have better testing strategies and ensure everyone is on the same page on the importance of test automation in the scrum team for a quality product.
Setting QE processes is a journey, there’ll always be room for improvement in order to be closer to the ideal state. There will always be challenges encountered in our everyday job. Despite these, we should always put product quality as our topmost priority in mind!
Facing any test automation challenge(s)? Do share with us in the comments below because #sharingIsCaring! Or if you’re in Hive office, feel free to speak to any of us from Merlin!
- Merlin
