How Coconala saved time by automating 1,000 test items: the pursuit of an ideal QA process

From editing videos and drawing illustrations to listening to your problems, there are times when we all have minor issues at work and at home. And we want them solved by someone who knows what they’re doing. Each of us has the experience, skills, and knowledge that could help someone in need.

Coconala is a service (skill) marketplace where users can buy and sell their knowledge, skills, and experiences online.

Now that having side hustles, side businesses or working as freelance are becoming more prevalent, the number of users who buy and sell their skills has been on the rise.

As the platform’s service offering expands, how does Coconala ensure quality while accelerating development? In this interview, we spoke to Mr. Masatoshi Murakami, the Executive Officer Head of Development at Coconala; Ms. Michiko Okamiya and Mr. Yuji Suzuki, who work on QA.

– Please tell us a bit about yourselves.

Masatoshi: I’m the Executive Officer Head of Development at Coconala. It’s been three years since I joined the company. I used to be a software development researcher interested in quality, such as bug-resistant design, design testing, and from design to semi-automatic implementation. I’ve taken the lead in implementing Autify.

Michiko: I’m a QA team member of the CS group at Coconala, responsible for pre-release testing, investigating issues, performing maintenance on Autify, and improving how the QA team operates. I’ve worked at Coconala since 2013, one year after its launch. I originally worked on UI improvements and was assigned to the QA team when it was formed in 2019.

Yuji: I joined Coconala in January 2022 and am on the QA development team, which is directly under product development. I’m currently working on developing a test development process that will happen over the next six months. From my previous position at a third-party testing company, I have experience creating QA processes – from planning to executing tests, and Selenium-based test automation.

– How many QA members do you have?

Michiko: We currently have four members, but it’ll be five in October [2022].

– Please also tell us about your business.

Masatoshi: We develop and provide a service (skill) matching marketplace called Coconala. People with various skills and knowledge sign up to our platform – designers, programmers, counselor, for example – and sell skills they’ve developed through work or their experiences. Our platform allows other users to buy those skills.

Since Coconala is an online platform, sellers and buyers can be connected anytime, anywhere. Like goods, we believe we can create a marketplace where you can find any skills you need. We are operating the platform with the vision of building a world where each of us can live “our own story”.

– What challenges did you face before implementing Autify?

Masatoshi: At first, only Michiko was in charge of QA. We also didn’t have a clear QA process at the time, so it was all trial and error. However, after several projects, our QA process began to take shape, so we strengthened our QA team by adding one more member. Before we got to that point, we would frequently encounter issues in the production environment, and development had to pause each time. But things got a lot better once we had a two-person QA team.

After that, we decided to increase the number of test cases to improve quality further. However, we didn’t have a lot of know-how, and we had several projects with limited work hours. QA resources were far from enough.

Michiko: The product manager would make test items on a spreadsheet and test about 100-300 items in a couple of days. As the product grew, specification changes and new feature releases became more frequent. Before we knew it, the number of test items had increased to almost 1,000. We also decided to run regression tests, which made resources even tighter. This was when I consulted Masatoshi, the head of development at the time.

I’d tried Selenium IDE, but it could only achieve partial automation. I had the impression that it would be hard to maintain, so I couldn’t imagine putting it into operation.

– You were considering test automation so that you could test more items with fewer people, and that’s when you found Autify.

Masatoshi: At that time, I wondered whether we could automate testing without using engineering resources. That’s when someone within the company mentioned Autify. At first, I only thought it was a Web version of Selenium. However, I didn’t want to judge it without using it, so I signed up for the trial version on the same day.

Once I got an account and tried it out, my first thought was, “this might be it.” I asked Michiko and a few other members to try it, and they all responded quite positively, so we decided to consider its implementation.

Michiko: When I learned about Autify and saw how the recording played out for the first time, I was very impressed and thought, “even I could use this!” At first, I thought Autify was an overseas company. I wondered if the service was available in Japanese, so I checked the website. I was surprised to see that it’s actually a Japanese company and that the CEO is Mr. Chikazawa (who used to be involved in developing Coconala)! It had been several years, and now you’re back to help us again. Autify felt like a savior to us. I remember feeling hopeful about test automation using Autify.

– Thank you very much. That means a lot to me. – What were the deciding factors in your decision to implement Autify?

Masatoshi: The deciding factor was that, through the trial, we were able to create the test cases we needed.  We currently use it for regression testing. We use it to ensure that operations common among all features – such as member registration, login, search, and browse – are working correctly. More recently, we’ve been systematically working on ensuring our platform is compatible with framework upgrades, and we need to test at every release. Regression testing is even more critical, and I think we are using it effectively.

– When you implemented Autify, how did you prioritize which manual tests you would automate first? And how did you go about it?

Masatoshi: After creating test cases as we did before, I sorted through the ones we will do manually and those we will automate with Autify.

Michiko: I categorized items in three main groups: a user logs in and views an item, a seller lists an item, and a buyer buys an item. Each list contains items that can be automated and some that cannot. The ones that can’t be automated include things like direct carrier billing, phone consultation, and video chat. We found that the matching path can be automated for the part where the buyer buys an item, opens a talk room, and communicates with the seller.

Yuji: Originally, test cases weren’t made by the QA team but by the person who planned the feature. I think replacing manual test items with automated ones took a while because we didn’t know what type of test cases Autify could handle. After that, the QA team created test cases that were designed to be tested on Autify, and things went smoothly.

– Did you have any creative ideas when you moved forward with test automation?

Michiko: When the QA team tried to create scenarios on Autify, sometimes JS Steps and the locator setting didn’t work. These are features that we can customize by writing code, so we often sought help from Autify Support. One of the things we made sure to do when creating scenarios was to record the know-how we gained during this process and to share it with others.

In terms of using Autify, using test plans and setting the execution frequency right have been key. We have two main test plans. One is the Basic Scenario. This single test plan consists of 23 scenarios that test things like browsing, member registration, buying/selling, all of which can be completed using a single account. We run this every day as a regression test.

The other test plan is the scenario that tests deals between users. On seller and buyer accounts, the process of making a purchase, opening a talk room, sending messages, and rating each other takes place around three times. We’ve put scenarios together into one test plan and run it approximately once a week.

We also set it so that the test is run at 6-8am so by the time we arrive at the office, we get a Slack notification about the result. We thought carefully about the best time to run the test so that we could have a routine where we would investigate if a test failed.

Masatoshi: To increase our Autify proficiency, we tried using random email addresses for login. Our service doesn’t allow members to register using the same email address, so this feature enables us to test membership registration at each test execution.

Another thing we did was to use CSV data as input data. We can use the same test case with different input values using CSV data. In other words, we can run several test patterns, which is very useful. I also like that we can create Step Groups and reuse a portion of a test case, as it saves time recording scenarios. We used to only be able to insert Step Groups at the beginning of a scenario due to Autify’s functional limitations. However, thanks to quick updates, we can now insert them in the middle of scenarios, making it even more convenient.

Michiko: Step Groups have come a long way. We have quite a few scenarios that overlap, so we’ve grouped scenarios that we also use for rating users, sending messages and matching pathways, etc. Now that we can specify the execution order of scenarios in test plans, it’s even more helpful. We run a test plan from the buyer’s perspective before running it as a seller. It used to take around five hours, but now it only takes about 20 minutes, saving us a significant amount of time. Autify’s features keep improving, and it’s become even more valuable.

Yuji: They were already using Autify when I joined the team, and one of the things I really liked was that they always ran tests at night on working days. And they allocated time to do maintenance based on the results. I think this is the secret to successful test automation. Even if you make automated tests, if you don’t maintain them and they don’t work when you need them, you’d get really discouraged. It’s great that we’ve naturally avoided that.

Masatoshi: Actually, even though things are going smoothly now, it didn’t come naturally at first! Even though we did manage to make the tests, there was a period when we didn’t have the time to run or maintain them. Neglecting them would only make it more time-consuming to maintain, so we decided to set aside some time so that we could maintain the tests every so often.

Michiko: We only had two QA members, so even though we were using Autify, there was a time when it got neglected when we were too busy manually testing a new release. For a while, we kept putting off checking failed tests, but that meant that there was no point in having Autify. After discussing what to do, we decided to check regularly, even if it was only 30 minutes a day. Then Yuji joined the team and dealt with all the failed tests. Thanks to that, things are going well now.

Masatoshi: We were at a turning point at that time, so we kept the test cases we needed and got rid of those we didn’t. We examined which were suitable for end-to-end testing and which weren’t, and we reassessed the time we were spending. I think we’re past the period of things not going well; we’re moving in the right direction.

– It’s been a while since you started using Autify. What kind of changes did you see in terms of the challenges you had before?

Masatoshi: It’s still a work in progress, but more and more people within the company are interested in Autify, and people working on both the product side and the infrastructure side are asking us whether they can use Autify for their projects. This is because the need for regression testing is rising among various projects. The presence of QA is growing within the company, and I think the fact that people are counting on us is a result of the continued efforts of our team.

Michiko: We’ve also held in-house study sessions to raise awareness about what Autify can do.

– How much regression testing have you automated with Autify so far?

Yuji: Roughly 50%. If we run all scenarios on Autify, we can ensure that 50% of all features are working. There are some new features that we haven’t made test cases for, so I plan to keep improving coverage.

– So, what’s next?

Masatoshi: We will continue to work on automating tests, but we think test automation is a stepping stone. It’s not good enough if bugs are happening at the QA stage. In software development, the further along the development process is, the more rework we would need to do if a bug occurs. Finding issues early and solving them quickly is the ideal situation. What QA can do to achieve this is to accumulate know-how from past QA efforts and issues in the production environment, feed it back to the earlier development process – especially in planning, design, and implementation – and promote early bug fixes. If we can get that feedback cycle going, I think we will have an ideal situation where all we need to do at the QA stage is to run tests and ensure there are no bugs. We wouldn’t have to redo any tests. I hope to keep moving forward to achieve that.

Yuji: There are three things I would like to work on. The first is to feed back test results to the development team. The second is to optimize test cases. I want to reduce the number of test cases while getting the failure rate in the production environment close to 0%.

The third is to make test execution more efficient. Speed is the biggest advantage Autify has over Selenium. For example, you can quickly make test cases by putting Step Groups together like a puzzle. I want to think about how we can make Step Groups more versatile so that we can use them to quickly create tests for new features, not just for regression testing.

– That would allow you to constantly build regression test cases, which would increase test coverage and give you a stronger sense of security.

Masatoshi: Exactly. Having said that, some features that change frequently may not be suitable for regression testing, so we need to choose carefully.

Regarding the feedback cycle I mentioned earlier, it would be great if we could use previous issues to eliminate issues in advance. Such analysis would be easier if Autify could become a QA knowledge base, so I look forward to future updates.

– If failed test cases could be analyzed on Autify, you would be able to discuss which specifications to cover efficiently.

Masatoshi: Yes. At the moment, when a bug occurs, we identify which process caused it and manage them using a spreadsheet. If we can visualize it, we can analyze which cases are prone to bugs and discuss how we can prevent them. It would be even better if we could do all that on Autify.

Yuji: That functionality would be great. Michiko and others collect data by writing down all test results and categorizing the cause of failed tests. We’ve accumulated six months’ worth of data.

– I see. We are in the process of improving test results so that you can search and filter results. Your feedback has been very insightful. Thank you.

You mentioned earlier that you want to test faster. Autify Connect, which we recently released, allows you to run Autify tests in local environments. I hope you find it helpful.

– Do you have any advice for those planning on automating tests?

Masatoshi: In the age of digital natives, I think providing a reliable and excellent user experience isn’t a want but a need. Obviously, users will leave if a service is buggy and unstable. On the other hand, I think there are many cases where it’s hard to allocate resources for QA to achieve such stability, even if there are enough resources for feature development.

To solve this common issue many companies face, we need to create a stable QA system that requires as few work hours as possible. Autify has been very helpful for us because it allows us to automate and streamline the QA process. We look forward to further improvements to Autify’s functionality and discovering best practices by exchanging information with other businesses that use the service.

Michiko: Anyone reaching a limit with manual testing should try using an automation tool for anything that could be done automatically. Autify is especially great for non-engineers like me since its UI/UX is very intuitive and easy to understand.

Yuji: Running tests is a means to an end, so it’s important to know what you want to achieve through test automation. At our company, tests keep a record of the user story of existing features even as features are further developed and specifications change. It’s a good idea to clarify the objective before you begin.

– Finally, do you have any announcements?

Masatoshi: As of August 2022, Coconala as a service has 3.14 million users. In 2021, our company was listed on the Tokyo Stock Exchange Mothers (currently Growth Market), attracting attention from investors. The service is catching the eyes of so many people, which makes it even more important to ensure high quality. Systems and structures are in place, including an enhanced QA system and test automation. Please join us if you’re interested in QA and development for the future development of our services.

Coconala’s career page

– Thank you very much!

(Interviewer: Ryo Chikazawa, CEO, Autify Inc.)