A group of researchers managed to get both Stability AI’s Stable Diffusion and OpenAI’s DALL-E 2 text-to-image models to disregard their policies and create images of naked people, dismembered bodies, and other violent and sexual scenarios.
Their work, which they will present at the IEEE Symposium on Security and Privacy in May next year, shines a light on how easy it is to force generative AI models into disregarding their own guardrails and policies, known as “jailbreaking.” It also demonstrates how difficult it is to prevent these models from generating such content, as it’s included in the vast troves of data they’ve been trained on, says Zico Kolter, an associate professor at Carnegie Mellon University. He demonstrated a similar form of jailbreaking on ChatGPT earlier this year but was not involved in this research."
Add comment