Monitoring the health and performance is critical for any application and the business they are supporting. In case of an SLA breach or degradation, the application team must receive an immediate alert to investigate and resolve the issue. Our data and machine learning teams are running applications using a wide…

AWS Lambda is a serverless and event-driven computing service in Amazon’s cloud computing offerings. Lambda functions enable organizations to run application code without managing servers and can automatically scale to hundreds of thousands of executions, within account limits. Lambda functions can be deployed in two different ways; using a zip…

Amazon EMR is our go-to solution to run Apache Spark for data processing, interactive analytics, and machine learning on AWS. As a fast-growing tech company, we rely on Amazon EMR to run over 10,000 Spark applications every day to fuel our products with up-to-date data and predictions. However, over time…

We recently migrated our production data lake containing tens of terabytes of data in Amazon S3 from Apache Hive to Apache Iceberg and achieved a 90% cost saving for Amazon S3. If you are interested in the cost analysis and its reasons, you should also read our other post…

The new generation data lake table formats (Apache Hudi, Apache Iceberg, and Delta Lake) are getting more traction every day with their superior capabilities compared to Apache Hive. They enable cost-effective cloud solutions for big data analysis with ACID transactions, schema evolution, time travel, and more. Table Formats Table format technology is…

Amazon Aurora is a great choice for scalable, available, reliable, and cost-effective MySQL and PostgreSQL compatible RDBMS solutions that we heavily use at Insider. It enables us to focus more on product development while the managed service handles time-consuming tasks like hardware provisioning, database setup, patching, and backups. …

Serving machine learning models for real-time prediction is a hot topic and there are numerous solutions out there. The easiest one is using a fully managed service like Amazon Sagemaker that handles all the operational burden of the deployments and scaling for you. Other than that, there are model serving…

At Insider, we use Apache Spark as the primary data processing engine to mine our clients’ clickstream data and feed ML-ready data into our machine learning pipelines to enable personalizations. We have been using Spark since version 1.5 and always looking for ways to improve efficiency. If you are interested…

One of the great reasons for upgrading to Spark 3 and EMR 6 Spark 3.0 has been released in June 2020 and arrived at the AWS EMR service with EMR 6.1 version in September 2020. After almost a year, we managed to upgrade our Spark version to 3.1.1 and…