Databricks Live Series: Optimizing Databricks for Cost & Performance: Best Practices
In this video, join Pat Ross, Solutions Engineer at Analytics8, and Databricks expert Michael Kollman as they explore essential tactics for optimizing Databricks to improve performance and manage costs. Learn how to leverage Databricks tools like Spark, Delta Lake, and Photon Engine to tailor solutions to your data needs while keeping expenses in check. Whether you're just starting with Databricks or looking to maximize an existing setup, Pat and Michael share actionable insights to help you get the most out of your Databricks environment. Timestamps and Topics: 0:00 - Introduction to Databricks Optimization 1:18 - Most important things to know when it comes to optimizing 2:49 - How to go about tackling automation 5:25 - When would you want to Increase compute size of driver node or the workers over just adding more workers? 8:44 - How could you check to see whether you’ve made the right choice? 11:23 - Are there any other quick wins and fixes for cluster optimization within the Databricks platform? 13:54 - What is the Photon Engine and how can it help you? 16:32 - How do you know when it’s right for you? 18:09 - Important Features of Delta Lake 20:53 - Are there any features we can leverage when creating a table further optimize performance? 24:50 - How to keep your tables staying performing over time 29:12 Final Takeaways: What is important when it comes to Databricks optimizations? 30:56 Closing Remarks Explore more about Databricks on our Website: https://www.analytics8.com/technologi... Additional Resources: * https://www.analytics8.com/blog/the-e... * https://www.analytics8.com/blog/guide... * https://www.analytics8.com/customer-s... Don’t forget to check out and register the rest of the events on this series! Connect with Analytics8 -Website: https://analytics8.com -Insights Blog: https://analytics.com/blog/ -Sign Up for a Data Strategy Session: https://www.analytics8.com/data-strategy-session/ -Contact Us: https://www.analytics8.com/contact-us/ -LinkedIn: https://www.linkedin.com/company/analytics8/ -Twitter: https://twitter.com/analytics8/ -Facebook: https://www.facebook.com/Analytics8 Subscribe to our monthly newsletter: https://www.analytics8.com/the-insider/