LLMs Aren’t a PhD-Level Statistician in Your Pocket w/ George Perret- nyhackr Feb Meetup

Visit https://www.nyhackr.org to learn more and follow https://x.com/nyhackr Talk Title- LLMs Are Not a “PhD-Level Expert in Your Pocket”, at Least Not a PhD-Level Expert Statistician Talk Description- LLMs are hot right now. Bold claims about LLMs' capacity abound. Sam Altman, the CEO of OpenAI, has described the capacity of their LLM as "a team of PhD-level experts in your pocket". Dario Amodei, the CEO of Anthropic, predicted that by the present, 90% of computer code will be written by LLMs. However, little empirical evidence supports these assertions. Available benchmarking datasets are easily manipulated and do not generalize to novel tasks. In this talk, I will present pilot data that demonstrate the claims about LLMs are vastly overstated, at least within the context of statistics and statistical computing. I leverage a novel research design that enables direct comparison of LLMs and expert statisticians on a data analysis task while controlling for the effects of prompt engineering. Additionally, I apply causal inference methods to challenge the claim that LLMs have improved productivity within the knowledge economy.   Bio- I am a causal inference researcher who develops statistical tools for the social and medical sciences. Currently, I am pursuing my PhD at NYU where I research novel causal inference methods for multiple treatments and causal inference with latent confounding variables. I am also very interested in the political economy of LLMs and their applications within social science.