Tweet
Growing the Community of AI Hackers with the Generative Red Team
Sven Cattell, Founder nbhd.ai & AI Village, He/Him
Austin Carson, Founder SeedAI, He/Him
Rumman Chowdhury, Founder Humane Intelligence, She/Her
45 Minutes
We’re running the largest live AI hacking event ever in the AI village this year. Anthropic, Google, HuggingFace, Meta, NVIDIA, OpenAI, and Stability, have all provided models to attack and Scale AI have built the platform. This event is orders of magnitude bigger than any previous AI red team effort. There are observers from the White House, NIST, NSF, and the EU coming to learn from hackers. We built this event to grow the community that knows how to effectively evaluate Large Language Models as it is much more than prompt injections and jailbreaks.
AI works fundamentally differently to traditional software and only forms a part of a product. Trust and Security of AI in a system thus has to work fundamentally differently to traditional software. This is especially true for generative AI systems. The core difference is AI is a stochastic component of software and is allowed to make a small amount of mistakes. This changes bug hunting, reporting, and payouts.
Come to this talk to hear about how and why we organized this, and the history of algorithmic & bias bounties that led up to the largest one ever at DEFCON 31. We’ll also give you some tips to help you in the contest.
Sven founded the AI Village in 2018 and has been running it ever since. Sven is also the founder of nbhd.ai, a startup focused on the security and integrity of datasets and the AI they build. He was previously a senior data scientist at Elastic where he built the malware model training pipeline. He has a PhD in Algebraic Topology, and a postdoc in geometric machine learning where he focused on anomaly and novelty detection.
@comathematician
Austin co-founded the Congressional Tech Staff Association, co-led the Congressional Cybersecurity Caucus and the Congressional High Tech Caucus, and is a founding fellow of the Internet Law and Policy Foundry. Currently Austin is the President and Founder of SeedAI, a nonprofit established to work with a diverse group of policymakers, academics, and private sector experts to help communities across the United States access the resources they need to engage with AI.
Rumman has built solutions in the field of applied algorithmic ethics since 2017. She is a Responsible AI Fellow at Harvard Berkman Klein’s Center for Internet and Society and the co-founder of Humane Intelligence, a nonprofit dedicated to algorithmic access and transparency. Previously, she was the Director of the ML Ethics, Transparency and Accountability team at Twitter, where she conducted their bias bounty, which was hosted at DEFCON.
REFERENCES:
We Need Bug Bounties for Bad Algorithms - Amit Elazari - https://www.vice.com/en/article/8xky...bad-algorithms
Introducing Twitter’s first algorithmic bias bounty challenge - Rumman Chowdhury & Jutta Williams - https://blog.twitter.com/engineering...unty-challenge
Sharing learnings from the first algorithmic bias bounty challenge - Kyra Yee & Irene Font Peradejordi - https://blog.twitter.com/engineering...unty-challenge
Bias Buccaneers - Rumman Chowdhury, Jutta Williams, Subho Majumdar, Scott Steinhardt, Ben Colman - https://www.biasbuccaneers.org/
An Algorithmic Framework for Bias Bounties - Ira Globus-Harris, Michael Kearns, Aaron Roth - https://arxiv.org/abs/2201.10408
Machine Learning Security Evasion Competition - Hyrum Anderson, et al. - https://mlsec.io/ https://cujo.com/announcing-the-winn...n-competition/
MITRE ATLAS - Ram Shankar Siva Kumar, et al - https://atlas.mitre.org/
The Spherical Cow of ML Security - Sven Cattell - http://aivillage.org/adversarial%20ml/spherical-cow/
The Case for a Hippocratic Oath for Connected Medical Devices: Viewpoint - Beau Woods, Andrea Coravos, and Joshua David Corman - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6444210/
Announcing OpenAI’s Bug Bounty Program - https://openai.com/blog/bug-bounty-program
Microsoft Malware Classification Challenge - Royi Ronen, Marian Radu, Corina Feuerstein, Elad Yom-Tov, Mansour Ahmadi - https://www.kaggle.com/c/malware-classification https://arxiv.org/abs/1802.10135
Sven Cattell, Founder nbhd.ai & AI Village, He/Him
Austin Carson, Founder SeedAI, He/Him
Rumman Chowdhury, Founder Humane Intelligence, She/Her
45 Minutes
We’re running the largest live AI hacking event ever in the AI village this year. Anthropic, Google, HuggingFace, Meta, NVIDIA, OpenAI, and Stability, have all provided models to attack and Scale AI have built the platform. This event is orders of magnitude bigger than any previous AI red team effort. There are observers from the White House, NIST, NSF, and the EU coming to learn from hackers. We built this event to grow the community that knows how to effectively evaluate Large Language Models as it is much more than prompt injections and jailbreaks.
AI works fundamentally differently to traditional software and only forms a part of a product. Trust and Security of AI in a system thus has to work fundamentally differently to traditional software. This is especially true for generative AI systems. The core difference is AI is a stochastic component of software and is allowed to make a small amount of mistakes. This changes bug hunting, reporting, and payouts.
Come to this talk to hear about how and why we organized this, and the history of algorithmic & bias bounties that led up to the largest one ever at DEFCON 31. We’ll also give you some tips to help you in the contest.
Sven founded the AI Village in 2018 and has been running it ever since. Sven is also the founder of nbhd.ai, a startup focused on the security and integrity of datasets and the AI they build. He was previously a senior data scientist at Elastic where he built the malware model training pipeline. He has a PhD in Algebraic Topology, and a postdoc in geometric machine learning where he focused on anomaly and novelty detection.
@comathematician
Austin co-founded the Congressional Tech Staff Association, co-led the Congressional Cybersecurity Caucus and the Congressional High Tech Caucus, and is a founding fellow of the Internet Law and Policy Foundry. Currently Austin is the President and Founder of SeedAI, a nonprofit established to work with a diverse group of policymakers, academics, and private sector experts to help communities across the United States access the resources they need to engage with AI.
Rumman has built solutions in the field of applied algorithmic ethics since 2017. She is a Responsible AI Fellow at Harvard Berkman Klein’s Center for Internet and Society and the co-founder of Humane Intelligence, a nonprofit dedicated to algorithmic access and transparency. Previously, she was the Director of the ML Ethics, Transparency and Accountability team at Twitter, where she conducted their bias bounty, which was hosted at DEFCON.
REFERENCES:
We Need Bug Bounties for Bad Algorithms - Amit Elazari - https://www.vice.com/en/article/8xky...bad-algorithms
Introducing Twitter’s first algorithmic bias bounty challenge - Rumman Chowdhury & Jutta Williams - https://blog.twitter.com/engineering...unty-challenge
Sharing learnings from the first algorithmic bias bounty challenge - Kyra Yee & Irene Font Peradejordi - https://blog.twitter.com/engineering...unty-challenge
Bias Buccaneers - Rumman Chowdhury, Jutta Williams, Subho Majumdar, Scott Steinhardt, Ben Colman - https://www.biasbuccaneers.org/
An Algorithmic Framework for Bias Bounties - Ira Globus-Harris, Michael Kearns, Aaron Roth - https://arxiv.org/abs/2201.10408
Machine Learning Security Evasion Competition - Hyrum Anderson, et al. - https://mlsec.io/ https://cujo.com/announcing-the-winn...n-competition/
MITRE ATLAS - Ram Shankar Siva Kumar, et al - https://atlas.mitre.org/
The Spherical Cow of ML Security - Sven Cattell - http://aivillage.org/adversarial%20ml/spherical-cow/
The Case for a Hippocratic Oath for Connected Medical Devices: Viewpoint - Beau Woods, Andrea Coravos, and Joshua David Corman - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6444210/
Announcing OpenAI’s Bug Bounty Program - https://openai.com/blog/bug-bounty-program
Microsoft Malware Classification Challenge - Royi Ronen, Marian Radu, Corina Feuerstein, Elad Yom-Tov, Mansour Ahmadi - https://www.kaggle.com/c/malware-classification https://arxiv.org/abs/1802.10135