ZOZO Technologies continues to uphold quality by increasing the frequency of testing A discussion on the power of cross-browser testing by migrating from Selenium to Autify

ZOZOTOWN is Japan’s largest fashion e-commerce site. ZOZO Technologies Inc. (hereinafter ZOZO Technologies) oversees ZOZOTOWN’s service operation and technology development.

ZOZO Technologies’ philosophy is to “change the fashion of 7 billion people with the power of technology” and engineers, designers, and analysts and other technical staff belong to the ZOZO group. Out of 400 employees, engineers makeup over 300.

COVID-19 is still hitting the apparel industry hard, with shops being forced to close temporarily or close completely. In these unprecedented times, there is a growing need for e-commerce sites, and at the same time, there is also an expectation for convenience. Engineers are working to provide better service. What kind of issues do they face?

Today, we interviewed Mr. Seiji Tsukioka (CTO Division), Mr. Hiroshi Shigenaga (BtoB SRE), and Mr. Tomoki Tamura (BtoB SRE), who are in charge of the system division at ZOZO Technologies. They talked to us about the issues faced when operating an e-commerce site and the results of introducing the test automation platform Autify.

ー What are your jobs include?

Seiji: We originally belonged to a company called Aratana Inc., but in April 2020 we merged with ZOZO Technologies. Now we oversee the system division.

While I belong to the CTO Division, I am also the team leader of the BtoB SRE (Site Reliability Engineering) team. The CTO office is mainly working on company-wide technology strategies and optimizing engineer organizations. BtoB SRE is currently in the process of organizing and optimizing infrastructure, operational systems, and testing in the effort to support our company’s e-commerce.

— I imagine there’s a growing need for a service like this. Has this been the case?

Hiroshi: I belong to the BtoB SRE division, and I am focusing on operation and quality control, including test automation.

Tomoki: I also belong to the BtoB SRE division, and my main responsibility is the operation and maintenance of the company’s e-commerce. For example, if an error occurs in our in-house e-commerce, we investigate and improve. We are also working to improve quality by creating automated tests as an in-house tool.

— Thank you. What kind of issues did you face before introducing Autify?

Seiji: Every time there was a modification in the program, Selenium had to be modified as well. Management and operation costs are quite high, such as server maintenance of Selenium’s operating environment, and there were only a few people who could do these tasks. In other words, the major issue was that the test environment was becoming more and more specialized, and I had always wanted to solve this issue.

Tomoki: I tried setting up a test environment on an in-house server, creating a scenario, and then I shared it with other members and asked them to help. However, it takes time to share, no one can keep doing it, and there isn’t enough time… In the end, I had to do it all by myself.

I was mainly running Chrome on Linux, and I was hoping to implement cross-browser automation due to the nature of the service, but I didn’t have enough time or resources.

I heard that a tool was developed to automate QA (Quality Assurance) without writing code. Can you tell us about this?

Tomoki: We used to use Selenium WebDriver, but you must learn a programming language for this. To make it easier for anyone, we made a modification so that Selenium can be run with an imported configuration file where commands were written in Excel, instead of using program language to run Selenium.

We thought that if people who don’t write code would automate, it would lead to sharing. So, I tried to make a base, but the operation hadn’t happened yet.

– You introduced Autify after this. Can you tell us about the process leading up to its introduction?

Seiji: We wanted to understand Autify, so we began by verifying whether Autify can reproduce what Selenium can do. One of the main evaluation criteria was whether Autify could solve the problems we had. We also shared with the on-site development team about this tool.

– Did Autify achieve what Selenium does?

Tomoki: I tested by recording whether Selenium’s scenario can be run in the same way with Autify. It worked smoothly without any problems so I thought this could work.

– Did Autify achieve what Selenium does?

Seiji: Regarding the specialization issue that I mentioned earlier, Mr. Tamura was the one who specialized in Selenium. I wanted to solve this issue. I wanted to make it so that even non-engineers could create and implement scenarios on the GUI (Graphical User Interface) more easily. In that respect, I think Autify could solve our issues perfectly.

– Thank you. Is the implementation at the stage where people who don’t usually write code can use it?

Seiji: We’ve only recently started actual operations, so I think it will take some more time, but in reality, things are moving towards a horizontal expansion of using Autify within the group.

It will be used not only in e-commerce sites related to FBZ (Fulfillment by ZOZO) and our in-house e-commerce support, but also in ZOZOTOWN and fashion coordination app “WEAR”. Different teams can face the same issues, so if we resolve an issue, I think it is better to reuse the solution efficiently.

– Did you have any concerns after the introduction?

Seiji: There were some concerns regarding performance during the actual introduction stage. We received some feedback from ZOZOTOWN’s team, and after working with Autify’s customer success engineer team, we were able to add a “parallel execution” option which has resolved the issue.

The test used to take 2 hours in some environments, but using parallel execution reduced the time by about 80%. It was very effective.

– I’m glad to hear that. Performance improvement is something that we are continuing to work on. Are there any ingenious ideas for operating e-commerce sites?

Seiji: There are routine processes in e-commerce websites, such as adding products to the shopping cart, registering as a member, purchasing, and canceling. We maintain quality by monitoring whether it is working properly by fixed point observation.

The ability to schedule tests with cron is something that I’m grateful for.

– Autify has an execution scheduling function. How often is it scheduled to run?

Seiji: Currently, it’s once an hour. It’s important that the test completes within the timeframe, and the key is parallel execution that I mentioned earlier.

Hiroshi: The scenarios are roughly divided into those used in the production environment and those used in the development environment, and the items to be checked are slightly different. The development environment scenario is run every hour.

In the production environment, conversions can become a problem so we can’t run it too frequently. Therefore, I try to run it once a day when the source is reflected in the production environment.

This is because there are various customizations on each e-commerce site. Depending on the customization, the way the frontend is displayed and the actions will change. To check if there is any degredation, we run an automated test, paying particular attention to whether the basic scenario works as usual.

Seiji: After a program is modified, unexpected errors can occur in unexpected places. As an extreme example, the shopping card can be affected even if something completely unrelated to the shopping cart is fixed.

If we can run a near-comprehensive scenario, it’s almost a guarantee that the e-commerce site functions properly.

– Definitely. With a large-scale service, it would be a big loss if the cart malfunctions even for a short time. Frequent testing and being able to complete it in the required time is very important. We are understanding those needs more and more and continuing to make improvements in those areas. We would appreciate any feedback and requests.

— What results have you seen after introducing Autify?

Seiji: We believe that systematization and automation are essential for greatly scaling up services. Autify has resolved problems such as specialization, maintenance of the server environment, and program modification. We believe we have adopted a test platform and a tester that can operate 24/7.

— That’s exactly what we are aiming for. It’s quite difficult for people to test 24/7.

Seiji: That’s right. It’s as if we’ve successfully hired a tester who can operate without server maintenance, and it’s highly cost-effective.

Tomoki: I wrote about this in a blog article, but I think it’s great that we can check specific OS versions, such as iOS 13, and iOS 12. In Chrome 80, there was a change in the default cookie specification. It could influence iOS 12 depending on the modification, so when we fixed this, I was able to quickly confirm with Autify whether it works normally with Safari on iOS 12.

If this happened before we introduced Autify, we would have had to prepare an actual iOS 12 iPhone and test and check manually. With Autify, we can check by just clicking a button on the dashboard so it’s much easier. I think the biggest advantage of Autify is that it can easily test multiple browsers and cross-browsers.

Hiroshi: I scheduled and ran tests properly just recently, and thanks to Autify, we were able to find a bug and it’s already been resolved. I think we are using it effectively.

Seiji: Technical support has also been very helpful. We can delegate tests that would have taken us hours, or even days.

Also, Autify responds immediately when we contact them using the chat function on the dashboard, and this has been helpful too. With conventional support systems, the response tends to be copied and pasted, but with Autify, we can be confident that we will receive appropriate advice based on what we want to achieve.

– Do you have any advice to those who are planning on test automation?

Tomoki: With Selenium IDE and Cypress, test automation is not impossible. However, there are various costs such as building an execution environment and preparing an actual machine. With Autify, there is no need to build an environment, and recording is easy, just like Selenium IDE. I think it’s a highly convenient service.

Hiroshi: I used to use Selenium as well and I remember facing a lot of issues. For example, the driver not matching when the browser version went up. With Autify, you can create scenarios easily, and there’s no need for execution or driver updates so it saves time too.

Seiji: For us, we can test automatically on 24/7 using Autify, which made our lives so much easier. For tools like this, you can’t understand how useful it is until you try it. Autify has a 2-week free trial, so I think it’s worth trying it out.

– Thank you. ZOZO Technologies has been putting effort into recruitment. I hear you’re actively hiring engineers?

Seiji: More offices are online and recruitment events are also going online. Tools such as SpatialChat and virtual SNS called Cluster are being used. Due to the coronavirus pandemic, it’s difficult to visit the office, so the office is reproduced on Cluster. We hold company information sessions, etc.

Currently, we are working on a big project of replacing the huge platform ZOZOTOWN, and we are proceeding with innovative ingenuity. I think getting involved with such a project is a rare opportunity. You can enjoy the technology, and all the team members are nice people. I’d like the company, products, and engineers to grow together.

If you are interested, please visit ZOZO Technologies’ corporate site and apply through there. Let’s work together to add value to the world.