How well do LLMs build Shiny apps? How do we test that? | Karan Gathani | Data Science Lab

Karan Gathani and the Shiny team at Posit benchmark various LLMs across various measures and identify key strategies to optimize their performance. They take building excellent Shiny apps very seriously, whether you're building them line-by-line or with LLMs. LLM hype is everywhere, but what interests Karan most is moving beyond marketing claims to demonstrate how these models can be scientifically improved for specific use cases. That way, we can assess with confidence whether newly released models are genuinely better or worse. Karan is a Senior Quality Assurance Engineer at Posit, where he works on Shiny. He has worked with tech startups across the San Francisco Bay Area, helping them ship products efficiently without compromising on quality. In his spare time, he takes part in BioBlitzes around the Bay Area and contributes to this natural history blog. The Data Science Lab is a live weekly call. Register at pos.it/dslab! Discord invites go out each week on lives calls. We'd love to have you! It's is an open, messy space for learning and asking questions. Think of it like pair coding with a friend or two. Learn something new, and share what you know to help others grow. Hosting crew from Posit: Libby Heeren, Isabella Velasquez Karan Gathani's GitHub: https://github.com/karangattu Resources: Vision language models are blind: https://openaccess.thecvf.com/content/ACCV2024/papers/Rahmanzadehgervi_Vision_language_models_are_blind_ACCV_2024_paper.pdf Open Code: https://opencode.ai/ The vitals package! Writing evals for your LLM product: https://vitals.tidyverse.org/articles/writing-evals.html Inspect (for Python): https://inspect.aisi.org.uk/ Karan's blog post on Inspect: https://posit.co/blog/using-evals-in-python Built-in Skill for Shiny in Posit Assistant (using bslib!): https://github.com/posit-dev/skills/blob/main/shiny/shiny-bslib/SKILL.md Agent Skills overview: https://agentskills.io/home Shinygen: https://karangattu.github.io/shinygen/