Churn Through Cloud Files in Parallel

People often want to run the same function over many files. However, processing files in cloud storage is often slow and expensive due to transferring cloud data in and out of AWS/GCP/Azure. In this webinar recording we’ll show how to run this “same function on many files” pattern on the cloud with Coiled, so you can run existing code faster and cheaper with minimal changes. We’ll also highlight cost optimization strategies and auto-scaling for processing terabyte-scale data in parallel for fractions of a dollar. Blog post on processing terabyte-scale NASA cloud datasets: https://medium.com/coiled-hq/processing-terabyte-scale-nasa-cloud-datasets-with-coiled-70ab552f35ec Example notebook churning through 5,000 files in parallel: https://github.com/coiled/examples/blob/9eb8cd27db1ff503de494acdca68dde89d79e674/arxiv-matplotlib.ipynb Documentation on Coiled serverless functions: https://docs.coiled.io/user_guide/usage/functions/index.html