Building a serverless ETL architecture in AWS for a sports media company with half a million users.
One of our clients in the UK works with major media corporations helping them understand customer behaviour and how to maximise audience monetization. They needed developers specialised in serverless technologies to support their London based team.
Much of our work involves Serverless Computing solutions. This particular project involves Extract, Transform and Load (ETL) processes, used to prepare data to be consumed by a Business Intelligence Tool.
Working with their team, we designed a Serverless ETL architecture in Amazon Web Services. Using AWS Lambda functions we connected to different providers and ingested the data into our ETL workflow, transformed it and put it into an AWS Redshift warehouse Database, to be loaded from Looker BI.
Technologies: Node.js, Python, Spark AWS Step Functions, AWS Lambda, AWS Glue, AWS Redshift, Continuous Integration
The product was launched in May 2020 and is currently being used to gather user data from 500,000 users across North America, Europe and the Middle East.
We were initially contracted for three months to assist with the launch of one project and we are currently approaching a year working together. For us, it has been a real joy working with this agency as we get to work with a knowledgeable team using modern, interesting technologies.
Working with Secret Source has been a real pleasure. They are very flexible with resources scaling up and down as our demand required. We are very happy with the work they have done for us and look forward to a long and fruitful collaboration.
As new clients are confirmed and products are expanded we hope to increase our remote team. Working with great clients that share our culture and focus on doing very high quality work using the latest technologies is what drives us and we are looking forward to the next projects we are going to do together.