Overview
As a leading online commercial marketplace in Sweden, Tradera hosts more than 2.5 million members offering millions of listings for products in categories like collectables, electronics, clothing and art. These listings can be purchased via auctions or fixed prices.
Tradera sees more than 4 million visits weekly to their site. Though most of their members are Swedish, Tradera is accessible to members around the world.
Operation at this scale demands agility to align with business growth and goals.
Tradera’s data warehouse was on-premise, and a decision was made to re-platform it to run on BigQuery instead. Previously having conducted a project on BigQuery, they discovered the advantages of using a scalable and serverless data warehouse.
The solution? Data warehouse modernisation.
Their goal was to migrate to the cloud, so Tradera got in touch with Devoteam Cloud Services in August of 2019 to assist in their legacy migration.
As a Google Cloud Premier partner, Devoteam has supported more than 3,000 businesses in their Google Cloud migration. The experts at Devoteam knew that an essential part of the migration process was gaining Tradera’s trust.
The first step was to undergo a week-long investigation and present their findings to Tradera through a proof-of-concept.
According to their Data Warehouse Modernisation methodology developed in tandem with Google, they first prepared a Technical Analytics Design Document, detailing how they would deliver the solution, using services such as Cloud Composer and BigQuery on Google Cloud.
They covered topics such as Backup and Restore, Migration, Transformation, Monitoring, Data Access Management and forged an agenda to develop this solution.
The plan was to implement different ETL pipelines on Google Cloud and transform the data in Big Query without sacrificing data accuracy.
Devoteam’s Huijie Wang, Senior Data Engineer, Lu Zhao, Data Engineer and Jason Quek, Lead Cloud Architect and CTO, began working on the project once the green light was given for implementation and deployment.
Their work focused on the migration of infrastructure, the data stored and maintaining the visualisation layer after the transformation.
Some of the steps included taking an inventory of data at the beginning and reconciliation of data at the end of the transformation, designing the ETL pipeline and translating MS SQL to represent the same logic in BigQuery.
Since there are no pre-existing native migration tools between the Microsoft environment and Google Cloud, Devoteam needed to develop connectors to migrate the data.
Differences in programming syntax between Microsoft SQL and BigQuery, database collation and data transfer caused some delay, but Devoteam’s experts learned and adapted to the task at hand.
The initial deadline was set for the end of January, but performing in agile sprints enabled Devoteam to present the possible delays due to changes in requirements and scope. This kept the Tradera team informed and updated on any setbacks throughout the process.
“Using the new setup on BigQuery and Cloud Composer, ETL performance was quadrupled. ETL pipelines were also easily monitored via Grafana and source controlled for increased transparency.”
Jason Quek, CTO, Devoteam Cloud Services
“We enjoyed working with the team from Devoteam, who were knowledgeable, responsive and professional during our data warehouse project together.”
Patrik Akselsson, CTO, Tradera
