The story of four members who led the test automation project.
Japanese B2B and SaaS startups that provide DX and back-office systems for organizations have attracted attention from overseas funds and investors. There is especially a boom in industry-specific SaaS startups. For example, ANDPAD, Inc. is a startup that aims to solve issues in the construction industry, such as workforce aging and a need for increased productivity.
ANDPAD is a cloud-based project management service, which unifies operations management to improve efficiency at construction sites. The service holds the No. 1 market share among services of its kind*. It is also registered in the NETIS (New Technology Information System) operated by the MLIT (Ministry of Land, Infrastructure, Transport, and Tourism). Currently, over 100,000 companies use ANDPAD for projects such as housing construction/renovation and commercial facility construction.
In this interview, we talked to Mr. Yoshiharu Geshi, VP of Engineering, and Ms. Hitomi Sato, QAE, about improving product quality and automating E2E tests.
* Source: Market Trends and Vendor Shares of Cloud-based Construction Management Services, Deloitte Tohmatsu MIC Research Institute Co., Ltd.
![ANDPAD Inc. Autify Success Story](../images/andpad.webp "ANDPAD Inc. is successful with testing automation platform Autify)
Please tell us about ANDPAD and what you do.
Yoshiharu: ANDPAD Inc. provides ANDPAD, a cloud service specialized in the construction industry.
The service aims to solve the construction industry’s challenges, such as the push towards DX (digital transformation). Our startup has around 400-500 employees, and we have completed the Series C funding round.
We plan to combine and develop various applications so that the service can be used more widely. We want to grow the platform into something that benefits everyone in the construction industry.
I joined ANDPAD in 2019 and oversee the entire development team, from hiring to quality assurance. I often encourage improving quality to my peers.
Hitomi: I joined ANDPAD in February 2020 and work on improving product quality.
A search for a maintenance-free, scalable tool to reduce workload
- What challenges did you face before implementing Autify?
Hitomi: We used to manually test our platform once a week before release to make sure it worked correctly. The issue was that manual testing was labor-intensive and took up everyone’s time.
We were short on QA engineers, so engineers and PMs gave us a hand with manual testing. That was an issue we wanted to solve. We hoped that automating tests would free up our time, which could then be used for development. We wanted to streamline testing so that we didn’t have to keep running the same test manually.
- Before implementing Autify, did you try other test automation services?
Yoshiharu: I tried Selenium, but it wasn’t intuitive enough to make maintenance easier. Tests made with Selenium tend to break easily, and not everyone can write code. We were spending around the same amount of time on manual testing as we did on Selenium. We would often think, “Is Selenium really helping us?”
Having experienced that, we would discuss whether there is any automation service that is maintenance-free and scalable.
Set goals and automate as a scrum project with four members
- How did you go about implementing Autify? Was there anything that you were particularly conscious of, and did you get stuck at any point?
Hitomi: First, we formed a small test automation team of four. Yoshiharu and I and a couple of members from the development team were in the team. We set up weekly meetings to decide on a target for the week and how we would achieve it. Of course, we had ongoing projects as well, so we worked on development alongside test automation.
Each week, we identified issues, clarified what we needed to do, and worked on it as a scrum team.
- Were there any benefits to have members with different roles?
Hitomi: I would ask Yoshiharu to make decisions for anything that didn’t have a contract. We had three months to try Autify, so the two development team members and I set goals of what we wanted to accomplish during the trial period. We regularly consulted Yoshiharu on what we needed to do to achieve that goal and used Autify as much as possible.
For each test case, we set monthly milestones of how much we wanted to get done and then a three-month checkpoint of whether to implement Autify based on our progress. Had I worked on it alone, we wouldn’t have been able to get this far.
- How did you decide on which test case to automate first?
Hitomi: We have many test cases, so we prioritized based on what was easiest to automate and gradually automated more and more tests. Some Test Scenarios are relatively easy to create, and some are rather challenging. We started with the easy ones.
When we began this automation project, we encouraged each other to contact the support team whenever we had a question. Our questions were answered promptly, so we didn’t get stuck.
Not using Autify was no longer an option
- What was the milestone in the third month?
Hitomi: We wanted to determine if we could keep using Autify the way we had been for the past three months. After just one month, we’d already felt that Autify is something we can’t go without. The milestone was about the number of tests we automated in three months and whether we could realistically implement it.
At first, we only automated tests that used Chrome as the test environment. But the more Test Scenarios we made, the more we wanted to try cross-browser testing. Then, we started thinking, “how convenient it would be to run tests on IE and Edge, etc?” Eventually, we found that not using Autify was no longer an option.
Yoshiharu: We initially said we’d try it for three months and decide. But we ended up subscribing after a month into the trial period.
- What was the key factor in your decision to implement Autify?
Hitomi: One major factor was that we could create automated tests by simply interacting with the screen as you usually would. With Autify, you don’t have to write code. As long as you know the Test Scenario, you don’t need to be an engineer.
And once you’ve created the test, all you have to do is hit Run. Anyone can do it, even those who aren’t a part of the test automation team. Maintenance doesn’t take much work, so anyone can use it.
Yoshiharu: In the specifications for the regression test, they’d written all the steps they planned to use Autify on. We asked ourselves, “what would we do if we didn’t have Autify?” It made us realize that Autify had already become a part of the QA process out of necessity.
- In a given test case, some parts are hard to automate, and sometimes it’s quicker to test manually. Were there any test cases that you redesigned when automating?
Hitomi: Yes, we made modifications as needed. Our original Test Scenarios were far from complete, and there was still room for improvement. As we went through our automation project, we scrutinized the Scenarios and made changes when necessary. Currently, we are working on automating tests for various products with Autify.
Automation is complete for 88% of all tests. Motivated team members were the key to success
- I heard that you automated 88% of the tests with Autify, which is pretty high. What’s the secret?
Hitomi: At first, I didn’t think about making cross-browser tests and just started with tests for Chrome. During our weekly meetings, we assigned responsibilities to each member so that we could create as many Test Scenarios as possible. Some tests don’t work depending on the browser, so we eventually started thinking about cross-browser tests.
For each test, we considered whether it’s necessary to automate it, and based on the necessity, we decided on which items to automate. By continuing those efforts, we have managed to automate 88% of our tests.
- It must have been a joint effort by Hitomi and the engineer team members to promote test automation and creating scenarios. Engineers are often busy with their daily work and don’t have the time for test automation. How did you get them to cooperate?
Yoshiharu: It wasn’t a matter of leadership. The engineers themselves were eager. There was a shared enthusiasm within the team that if we’re going to do this automation project, we might as well put our all into it.
Hitomi: Rather than trying to persuade others, engineers who wanted to use Autify came to me. I think engineers are motivated to automate because they are tired of testing manually.
Sharing knowledge with other departments via lateral connections
- Are you in charge of providing implementation support to other teams, Hitomi?
Hitomi: Yes. I hold bi-weekly meetings with other teams to discuss the best course of action. Engineers and PMs to create Test Scenarios, I check and advise. I determine which tests to start with and review tests that engineers create.
- I think QA engineers will increasingly play the role of a quality advocate rather than someone who runs tests.That seems to be what you’re doing. Have you used Autify in creative ways?
Hitomi: Initially, the automation team had four members but has since increased to eight. People who wanted to automate tests for each project have joined the team. I assign tasks, such as creating Test Scenarios, so that everyone can make progress and no one gets left behind.
Also, since there’s an extensive network of engineers within the company, I advise whom to contact to solve a particular issue. This enables the team to share knowledge while building lateral connections. Autify comes up in regular development meetings, so people also seem to exchange information in that way.
Only a sixth of time spent on testing
- What changes did you see after implementing Autify?
Hitomi: Initially, our goal was to automate end-to-end tests before release.
However, once we implemented Autify and automated more and more tests, people started asking if they could use it for other products as well. So, we began running automated tests regularly in development environments. Now we’ve expanded it to running regression tests in the development environment.
I originally hadn’t thought that far ahead, so it’s been great to see it being used more widely than expected.
- What benefits have you seen since implementing Autify?
Hitomi: More and more people are proactive about running tests, which is evidence that Autify is already quite ingrained. People are making use of test results, which has been very helpful.
Yoshiharu: I’ve seen a significant benefit to implementing Autify. It allows us to prevent failures before they happen. With manual testing, something inevitably gets missed, so there’s always a chance that a bug could fall through the cracks. Preventing that from happening has allowed us to ensure a greater level of reliability.
Also, we spend significantly less time testing than when everyone used to have multiple browsers open and clicking away. I used to spend about 30 minutes per browser, but now that’s been reduced to about 5 minutes. All that saved time can be spent on other tasks, which is excellent.
- Now that you spend less time on testing, what have you been able to do?
Yoshiharu: I have more time to develop new features. Also, there’s a greater sense of security. We get notifications of test results on Slack, and when there’s a red X mark, we get excited. It means that Autify successfully found an issue, which increases our confidence in the platform.
More people should know that test automation without coding is possible
- What are your plans for the future?
Yoshiharu: We are currently using Autify mainly for the PC version of the application, but we want to start using it on smartphone apps as well. Now that we’ve tried using it on one product, we want to apply it to the rest of our products. I hope to create a roadmap to make that happen.
Hitomi: There are still incomplete test cases, so I plan to work on them and improve their quality.
- Finally, do you have any advice for those who are planning on test automation?
Yoshiharu: As I mentioned earlier, it’s important to start with what you can do. We started test automation with four people. It’s hard to go alone, so I recommend having buddies you can work with. Also, the Customer Success team at Autify was very helpful during the implementation process. If you are considering Autify, I recommend that you take full advantage of the support available.
Hitomi: ANDPAD was the first test automation project that I worked on as a QA engineer, and Autify was the first automation tool I’ve used. It was great to have a positive experience of automating without writing code. I want people to know that even if you’ve never experienced test automation, you can do it with Autify. I’m sure there are cases where there aren’t many people who work in QA, or only one person can be allocated to the automation project. There are places and communities where you can ask for advice, so don’t hesitate to reach out.
- Do you have any announcements you’d like to make?
Yoshiharu: ANDPAD is hiring QA Engineers and Software Engineers in Test (SET). QA engineers are responsible for expanding and automating E2E tests and improving quality by getting involved at the requirements design stage. SETs have the mission to improve the development experience for engineers by systematizing automated tests. If you are interested, please apply here. You can also DM me on Twitter.
(Interviewer: Ryo Chikazawa, CEO&Co-Founder, Autify, Inc.)