AWS Glue has been pioneering in the space of automating ETL processes by providing a fully managed serverless data integration service. This service is a simple and cost-effective way for customers to categorize their data, clean it, enrich it, and move it swiftly and reliably between various data stores. AWS Glue is made up of a Data Catalog (i.e a metadata store), sophisticated ETL engines with automated code generation and visual interfaces for every persona to do ETL tasks. AWS Glue customers use Apache Spark and Python engines for data integration.
Our Python customers have asked for scaling their Python workloads over large datasets. To enable these use-cases, AWS Glue added support for Ray.io (http://ray.io/) and launched AWS Glue for Ray. AWS Glue for Ray provides data engineers a distributed Pythonic data analytics platform for performing distributed data integration at scale with Ray core APIs. Using Ray's powerful abstraction of tasks/actors, we were able to horizontally scale python workloads. The simple distributed collection APIs provided by Ray dataset helped our python customers to perform ETL operations efficiently on very large datasets. Distinctly, we launch Ray clusters on ARM based platforms and using IPv6 addressing based workers. Data engineers are comfortable with Pandas and given that popularity, we integrated Modin at scale with Ray. We will cover our experiences with Ray datasets and distributed Pandas at scale. We will also talk about the innovations we did integrating with Ray's robust cluster manager and demand based autoscaler, to offer an instant-on, interactive, easy to use serverless Ray platform for our customers.
Japson Jeyasekaran is a senior software engineer in AWS Glue building serverless infrastructure platform for various data analytics engine. (edited)
Harish Sitaraman is an engineering manager at AWS. His teams own AWS Glue for Ray and the serverless platform that powers AWS Glue workloads. Prior to AWS Glue, Harish has had multiple management and technical leadership roles with AWS Networking and Juniper Networks.
Come connect with the global community of thinkers and disruptors who are building and deploying the next generation of AI and ML applications.