Talk DescriptionLarge language models are increasingly used for software development, but their generated code can sometimes incorporate malicious URLs and endpoints originating from scam websites, even in response to seemingly benign developer requests. In this talk, I will present Scam2Prompt, a scalable framework for auditing this behavior in production LLMs. Scam2Prompt starts from known malicious websites and automatically constructs realistic developer-style prompts to test whether LLMs reproduce scam infrastructure in generated code. Using this framework, we build Innoc2Scam-bench, a renewable benchmark of 1,377 prompts, and evaluate several state-of-the-art production LLMs. Our results show that malicious endpoint generation remains prevalent across recent models. Interestingly, the framework also uncovered previously unknown active scam websites, 62 of which were subsequently confirmed and added to MetaMask’s scam database. More broadly, this work highlights a security risk beyond explicit harmful prompting: LLMs may unintentionally propagate unsafe web content when assisting users with executable code.
BioZhiyang Chen is currently visiting ETH Zurich until Feb 2027, working with Prof. Zhendong Su, He is a final-year PhD candidate at the University of Toronto advised by Prof. Fan Long. His research focuses on blockchain and smart contract security, and more recently, AI security and efficient agentic systems. More broadly, he has also collaborated on research involving compilers and program analysis, as well as CUDA kernel generation. Zhiyang is also an active contributor to open-source software such as SQLite. More information about his work can be found at https://jeffchen006.github.io/.
You can also join by zoom at https://kth-se.zoom.us/j/64568739111
AgentConf
