“The reason for not having time to automate is because you haven’t automated yet.” Why are startups investing in test automation at an early stage?

As we continue to work remotely, shopping online has become a fundamental part of our lives. The D2C (direct to consumer) business model, which responds to the customer’s individual demand, is also booming and diversifying. E-commerce platforms such as BASE and Shopify have also improved significantly, making it easier for people to create convenient and great-looking e-commerce websites.

Then there’s_ Appify. It allows users to create official apps for online shops made with BASE and Shopify. _Appify is a subscription-based service that enables shop owners to develop and operate apps for less than 10,000 yen ($100) per month.

We interviewed Mr. Ryosuke Fukuda, the CEO of Appify Technologies, Inc., who provides Appify, about quality assurance and test automation, which is something they are currently working on.

(Appify Technologies, Inc. CEO Mr. Ryosuke Fukuda)

– First, can you tell us about your business?

Ryosuke: We provide Appify. It’s a platform that allows users who run e-commerce websites with platforms like BASE and Shopify to create official apps for their online shops. Anyone can use it to create an app easily without coding, just like building castles with Lego.

– You are using Autify at the moment. What kind of challenges were you faced with before?

Ryosuke: Three engineers run our company. Many shops use Appify, so just one minor bug will lead to a significant loss for our users. We used to run tests to guarantee that there was no problem in the API, but I wanted to improve quality by running E2E tests further.

At first, we considered implementing a testing framework called Cypress. But CI (Continuous integration) took a long time, and maintenance was labor-intensive. We didn’t have the time to think about the best practices in testing. It was around that time when I came across Autify and thought, “this might work.”

– It’s great that you have three engineers and be invested in quality already. Were you always mindful of quality assurance from the beginning?

Ryosuke: Some of the shops that use our service have over 10 million yen in monthly sales. We were always aware that a minor bug could lead to a considerable financial loss. That’s why I’ve always worked to reduce bugs as much as possible in terms of quality assurance.

We’ve always paid attention to the design and ease of use. People often say that startups shouldn’t be overly worried about the design until the beta version is released. I agree that it’s essential to get the bare minimum out first. But I believe that the right way to go is to make sure the bare minimum is good. That’s why we also worked on the design.

It’s been about six months since we released the beta version, and I’m amazed to see that the app has already generated several hundred million yen in sales. It’s not us that’s growing; it’s the shops that use our service that is growing tremendously.

From the shop owner’s point of view, 1 in 10 customers is a core fan. Official apps made with Appify should be apps that provide a great shopping experience to those core fans. That’s why it has to be easy to use at the very least, and ideally, it should work smoothly.

– How did you proceed after you decided to go with Autify?

Ryosuke: First, we automated the test for the registration process. That was the part that was most prone to bugs. The entire Appify application involves external integration and the registration flow is rather complicated. For example, when a user comes from Shopify, they’ll be taken to the Shopify registration page. We have separated the Test Scenarios into BASE registration flow, Shopify registration flow, and CAMPFIRE Community registration flow.

There were no problems, and it all went very smoothly. We were also able to automate external integration flow with Autify, such as signing into BASE. We can track the entire screen without having to add any scripts for external integration.

We have created Test Scenarios with Autify for external integrations and areas where bugs are likely to occur. However, there’s been a significant change in the registration flow from December 2020 to January 2021, and we’ve been able to continue developing while testing with Autify.

We only have a small number of staff, so we can’t make things too complicated, and it’s difficult to oversee the entire platform. Slight changes in specifications can cause unexpected bugs, but we automated tests with Autify to prevent that.

– You started with testing the registration flow. What are your plans for automation now?

Ryosuke: I’d like to automate all pages eventually, but we are expecting some changes in the service specifications in the current phase. Even if we create tests now, there might be significant changes in the next release. That’s why I don’t think it’s necessary to automate everything right now.

We don’t have much spare time, but I can try setting up a test for the registration flow. Then, I start thinking about testing something else with Autify. Little by little, you end up with a complete test. We’ll gradually automate E2E tests from stable specifications and switch to just correcting errors as they occur.

– Do you run tests before deploying?

Ryosuke: Currently, we run tests after each commit. The problem with running tests after each commit is that you end up running tests many times. I plan to change this in the future, but essentially I run a test every time a CI goes live.

– How did things go after you implemented Autify?

Ryosuke: It allowed us to make bold changes. If you can’t be sure whether there are any bugs until testing, the development team must be cautious and be mindful of the entirety of the code when making edits. Now that we can test with Autify, we know there’s a bug if the test fails. This allowed us to be fearless with making changes and check for areas that are failing tests.

For example, they say, “architecture that you have to think about is a failed architecture.” If the first thing you do when adding a feature is to think of the architecture, then the architecture itself doesn’t make sense. The point is that engineers should focus on developing features instead of architecture.

Same with testing; in an ideal world, you don’t have to think about how to design the test. With Autify, all you have to do is to think about how to develop tests within Autify’s capabilities.

If you were to program and automate everything yourself, you’d have to create the test framework, think about how to code, and then think about what to test. Autify allows us to jump right in. We can start with thinking about what to test.

We can get on with things without hesitation and being overly cautious. When we didn’t know what was broken, we had to verify everything, but now we don’t have to do that. It’s made our lives a lot easier.

– What are your plans for the future?

Ryosuke: I’m looking forward to Autify’s native app. Appify’s management screen and CS tool are web-based, and we have released multiple mobile apps as well. We haven’t been able to run E2E test on mobile apps, so I’d like to automate that, too.

Another goal is not to have a person dedicated to QA. It’s OK to have a person in charge of QA (Quality Assurance), of course, but I think it’s possible to go without.

Especially with startups, infrastructure engineers are becoming redundant. This is because Google and AWS (Amazon Web Services) are evolving and creating a level playing field. I think test automation will be like that soon.

With QA, there’s aesthetic quality and operational quality. I don’t think aesthetic QA needs to be done manually. We’ve been focusing on the aesthetics so far, but I don’t think that’s the most critical part.

QA engineers should work on things that you can’t change later, such as the API. Some companies have a QA team that exclusively works on APIs, and that team has the final say on the release. If the QA team rejects it, you can’t release features, no matter what it is. That’s how important it is. If you don’t make compromises at that level, you can fix any issue.

– Finally, do you have any advice for those planning on automating testing?

Ryosuke: If you don’t have time to automate, it’s because you haven’t automated.

The benefit of test automation compounds. And if you don’t automate, you can’t make time to automate. I think it’s wiser to invest in test automation at the get-go because testing is the most time-consuming part. That’s something that can be said about all software development.