DEF CON Safe Mode Platform Discussion

[Dark Tangent]

Updated v13

Lets talk possible platforms and their strengths and weaknesses.

I know we will use the forums for planning, but what to use for the event?

Currently we are exploring discord with a connection to Twitch to handle the large video stream size needed for big talks. Some people say to run a Riot server ourselves. Some want VR.

We need a solutions for:

• Thousands of people at a time watching a talk or workshop
• Thousands of people watching or listening to a DJ / VJ
• A way to be social through chat or voice with moderation capabilities to lock out trolls
• Some way to have shared experiences like watching NetFlix or a free streamed movie Mystery Science Theater 3k style.
• Allow content creators live village to manage and moderate their own spaces

Post your thoughts below and we can summarize the good and the bad.

Decision Matrix
Platforms

Name	Max Users	Self Host?	Bot?	User Privacy?	Mobile?	See in China?	Guest Access?	Notes	Meets a Need?	Url
Discord	50 watching video per channel	Yes - Lots of work	Yes		Yes	YES - Slow	Yes?	Can connect to Twitch to support essentially unlimited video streams.	Yes	discord.com
AltspaceVR	30k+	No				YES - Slow	No	Can accept video streams into rooms. 3D with laptop, 2D with phone.		altvr.com
Zoom	200? 1000?	No			Yes	YES	Yes	Video Conference - Can connect to YouTube / FB Video to steam to larger audiences.		zoom.com
ON24	10k+	No						This is what Black Hat will use, commercial provider but not a year round community		on24.com

Streaming Platforms

Name	Max Users	Self Host	Bot?	User Privacy?	Mobile	See in China?	Guest Access?	Notes	Meets a Need	Url
Twitch	1 Million	No			Yes	No	Yes	A/V streaming DJ VJ Speakers Workshops	Yes	twitch.tv
YouTube	500k+	No	No		Yes	No	Yes	Video Stream		youtube.com
Face Book	?	No	No	No	Yes	No		Video Stream		facebook.com
Vimeo			No			No				vimeo.com

Text / Chat Platforms

Name	Max Users	Self Host	Bot?	User Privacy?	Mobile	See in China?	Guest Access?	Notes	Meets a Need	Url
IRC		Yes	Yes	Yes	Yes	Yes	Yes	Text chat
Mumble	3k+	Yes	Yes	Yes	Yes			Text chat		www.mumble.com
Rocket.Chat		Yes								rocket.chat
Riot		Yes	Yes	Yes	Yes			Other communications can be bridged into Riot such as IRC, Slack, and Discord.		about.riot.im
Slack			Yes		Yes	Yes (But no Chinese)		"Where work happens"		slack.com

[badlock]

Not Zoom. As much as it's a nice platform, too many companies and governments have forbid their users from using it.

For small (1-10) users, Jitsi is good. But it doesn't scale up to hundreds.
Webex, GotoMeeting, Skype, etc. Are each good for a few dozen to a hundred people -- depending on the platform.

However, for big presentations, perhaps a hybrid solution would be best. Something like: Jitsi for the presenter, because it can be used to feed a YouTube stream (public or private). YouTube for allowing thousands of people view the presentation at the same time. And something like Discord or Slack for real-time Q&A (type your question and let the presenter answer it). Sure, this is a hack, but, well, it is a hacker conference.

[Dark Tangent]

Whatever we pick will need to support thousands in a "room" watching video at a time.

[badlock]

When you start talking about thousands or more users, you will never be able to use any of the free or commercial meeting platforms. Zoom, GoToMeeting, WebEx, Skype, etc. cannot handle the volume. The biggest issue is the outbound bandwidth. Even with multicasting, nobody (except maybe Akamai and Google) have that kind of bandwidth. If a turn server is needed (almost certainly the case), then you'll need big computation resources (i.e., Google). With small groups, turn takes up almost no resources. But with thousands of users? Bandwidth and CPU become concerns.

The best bet it to use a split stream for the presentation. E.g., Zoom, Jitsi, or whatever the user wants for presenting and have it feed into a YouTube live stream. (Think cable modem or DSL -- low bandwidth up, high bandwidth down. Low bandwidth needs for the presenter to get to the distributing service, and high bandwidth for all of the people viewing from the distribution service.) This approach also deters someone from DDoS'ing the presenter since they don't know the presenter's direct IP address.

The spit stream approach handles your first requirement: thousands of people all watching the presentation/DJ at the same time.

Your second requirement is more difficult -- getting feedback to the presenter. Discord, Slack, etc. are text based and can handle thousands of users. I like this option more than Reddit or Google Chat or YouTube comments since they refresh in real-time.

Discord can also handle audio interactions. However, unless every attendee tests their audio first and clearly enunciates, this will result in a really bad experience. I suggest limiting audio responses until all of the other bugs are worked out. Maybe have the main presentation with thousands of users, then have smaller breakout Q&A sessions with direct audio/video (GotoMeeting, Jitsi, WebEx, etc.) Or use something like a "waiting lobby" (WebEx, Skype, etc.), where people are not accepted into the room for asking questions until it is their turn to ask questions. This keeps the bandwidth limited to the people in the room, while everyone else watches the breakout session on a YouTube live stream.

Your final requirement -- having the group interact as a group -- is much more challenging. (Presenting is 1xM complexity. Q&A is NxM, where N << M. And group chat is MxM.) Big Blue Button might work, but its interface is designed for a classroom environment. (And you'd need someone with bandwidth.) Discord might work if you have small groups of about 25 each. (But I don't know how discord will perform with hundreds of groups of 25 each.)

[McGrewSecurity]

On the Music/A&E side of the event, it is worth noting that almost all of the DJs you see on most of the streaming services are just rolling the dice on getting streams muted/halted/copyright-struck, unless they individually have the licensing straight, and even then, the algorithms sometimes get them anyway. I've seen a lot of advice being passed around to quick mix, talk over streams, introduce noise, and play with key-lock off in order to get by, but it may not be something that can be relied on.

The only streaming platform I know of right now that handles licensing for music for you (and more importantly, won't kill your stream in the middle of a set :) ) is Mixcloud's new live streaming service. Clearly some bigger names do it on other services, so I guess the licensing and whitelisting can be done, but I have no idea how.

[savagejen]

Or you can just use pretzel.rocks on your platform of choice

[supersat]

Twitch seems to not muck with live streams, but will sometimes mute portions of VODs from those live streams.

As far as I can tell, right now, no one really knows how to deal with streaming DJ sets. Even the big players (e.g., Beatport) are getting their streams flagged on Twitch and YouTube.

[McGrewSecurity]

Thanks for the link, savagejen. It looks like they also have downloadable tracks (useful for DJs to load into their players) on a related project at https://www.ninety9lives.com/ . Asking the DJs to stick to royalty-free/permissive-licensed tracks might be a non-starter, though!

supersat: Thanks for the tip about Twitch. That's good to know that they're not currently hitting the live streams at least, and I hope that doesn't change.

[savagejen]

Both twitch and youtube live can support thousands connected.

[Colonel_panic]

On the VR front I think this would be a good way to achieve some of the social aspects of the convention with Defcon Safemode. We could set up a VR Movie night, have some VR parties, maybe even hold VR villages! VRChat & Bigscreen seem like the natural candidates, however like most applications in sure their security is not up to par with what is needed at DEFCON so precautions should be taken to stay safe. I would love to hear from people who would like to help me coordinate these events or suggest alternative applications to facilitate them.

[supersat]

For talks, I'd recommend both Twitch and YouTube. You can feed both with OBS, which I'd recommend using as well.

One thing you might consider is asking speakers to pre-record their talk, but be available for a live Q&A. In addition to preventing last-minute technical issues, it might help people improve the quality of their talks.

[Dark Tangent]

I watched a video from the Security Oasis virtual con, and it was interesting but looked like I was watching a cartoon, and the slide screen was too small for me to read the details. I'm not sure if VR will work for content, maybe just the social aspects?

We are also looking for viable shared experience platforms for watching movies together, like possibly the NetFlix chrome plug in or something open source for watching videos that are free.

[savagejen]

I don't know how many people it can support, but there's https://vemos.org/

[McGrewSecurity]

I agree. With VR platforms, consider accessibility issues as well: both physical and hardware-access related.

[abaranov]

I've participated in a very successful virtual conference and it used Zoom with the broadcasting add-on. So you have the speakers, the MCs and the speaker support all in a Zoom meeting. And everyone else tunes into the Youtube feed.

At the same time, everyone can be in some chat like Discord.

This, from Wild West, is long and mostly not relevant but some of it is really useful: https://www.youtube.com/watch?v=IvwM8qUrmPo

They had quite an impressive setup but they also did things like send microphones to their presenters.

Happy to answer any questions.

[badlock]

Don't get me wrong -- I like Zoom's capabilities. But the current security issues and political policies make it a no-go. For example:

- Companies and organizations that have outright banned employees from using Zoom: Google, SpaceX, Smart Communications (Philippines), the entire government of Taiwan, NASA, German Foreign Ministry, US Senate, Australian Defense Force, New Your City's Department of Education. (This isn't the entire list.) See: https://www.techrepublic.com/article...nasa-and-more/

- The Pentagon has banned hosting meetings, but employees can still participate. However, other US Gov groups have outright banned Zoom on any system that can directly, indirectly, or tangentially connect to anything in the US Gov. (E.g., if you can do your gov work from home then you can't have Zoom installed on any computer at your home.) There are per-use exceptions, but who knows how many impacted people can get the exceptions (probably not worth the effort to ask). https://www.military.com/daily-news/...-military.html

And these are just the ones that have been made public. Plenty more organizations have banned Zoom -- in part, whole, or even for non-work uses -- and have not been reported in the media.

Keep in mind, Zoom has reportedly already fixed many of the security and privacy concerns. However, fixing the bug is not the same as removing the policies and changing business restrictions. Unless the policies are changed and the restrictions are lifted, Zoom is not viable for many of the attendees.

As an aside: GotoMeeting, WebEx, Skype, and other platforms have had their own share of security and privacy issues. But only Zoom has the widespread "do not use" restriction. (I'd love to see a Defcon talk on Zoom hacking while all of the attendees use Zoom.)

[abaranov]

Just to clarify - especially to @badlock's comment below.

You don't need Zoom to watch the talks. You just need it to present. Presenters can also present using the web version. To watch, you would just need to access the stream on youtube or twitch. (Or both, maybe)

Also, the serious concerns with Zoom are to do with it not have true E2EE as per whisper protocol - I think that if you are broadcasting something to the Internet, you are pretty secure from someone downloading your encrypted stream and keys in order to decrypt your stream.

[3mul0r]

I really like Discord for the main hub since it already has most of what you are looking for. The Discord and Twitch combo ticks a lot of boxes.
Glad to see you mentioned Security Oasis virtual con which was held on AltspaceVR. That could be a interesting platform since you can also join it with a 2d desktop app.

• Thousands of people at a time watching a talk or workshop
  • perfect for twitch
• Thousands of people watching or listening to a DJ / VJ
  • If its audio only you can set up a channel in discord where only the DJ can talk. I don't believe there are any content restrictions.
• A way to be social through chat or voice with moderation capabilities to lock out trolls
  • Discord already has mod tools built to handle servers with tens of thousands of users
    • There are some limits to watch out for but they don't seem like they would be an issue https://discordia.me/en/server-limits
  • Twitch also has good mod tools https://www.engadget.com/2020-03-31-...-mod-view.html
• Some way to have shared experiences like watching NetFlix or a free streamed movie Mystery Science Theater 3k style.
  • AltspaceVR is a possibility for this. Just like the Security Oasis virtual con you can join this with a 2d desktop version if you don't have VR.
• Allow content creators live village to manage and moderate their own spaces
  • Each village can have its own Discord server and follow the defcon example

[3mul0r]

https://www.bigscreenvr.com/
Bigscreen might also be a good option for shared experiences. It has cross platform support and also has a 2d desktop version.

[savagejen]

Whatever platform you choose, I think chat is a really important feature. Even if that ends up just involving embedding an irc client in a webpage next to the stream or something.

Twitch has partner/affiliate statuses.

For affiliate you have to have 200 followers, stream 25 hours a month and get 75 viewers average for a month, so you would really need to start now if you want to hit that.

For partner status, you can reach out to twitch and simply ask for partner status. You have a big enough following that they would seriously consider partnering you.

Twitch also has an events program. Again, I would reach out.

[JRWR]

Villages in VRChat /would/ be pretty amazing, but moderating that nonsense would be pretty hard. You would want to hire a dedicated 3D Artist who knows Unity well if you wanted to go down that route.

Having all the groups and villages having their own sub channels in the master discord would work well. I suggest using a opt-in system for that (You can react to a message and it assigns roles)

For the view side, I would just stream the talks to Twitch, I suggest even sending good cameras and microphones to the speakers if you want this to come out well.

Villages would still use rooms inside discord for video+voice+chat and that would scale OK if you limited rooms to a modest amount and gave out lots of rooms per group to spread everyone out.

Edit: I've been talking with the larger discord server owners, Unless you get discord in your back pocket, and even then, you will want to split this up into several discords, 35k online is overall the best you can hope for online at once in a discord server, any more then that and its going to start dropping connections.

[3mul0r]

If each of the villages had their own discord servers it would probably keep things pretty reasonable. I also kinda doubt that we would see the same level of participation as defcon proper, but I suppose it could go either way.

It would be interesting to see an official defcon discord server and see how many join.

[Dark Tangent]

No one has talked about the Mozilla options, Riot, etc. Any experiences to share?

[sneak]

I think that a hard requirement for whatever services/platforms are used are that the TOS that is required that users agree to is compatible with the values of DEF CON in terms of communication on topics that most services usually ban (e.g. exploits, PoCs, viruses, malware). Most of them will ban you for saying the wrong things, or posting the wrong links (even in DM) due to their one-size-fits-all anti-abuse and anti-spam surveillance. Sharing links to e.g. malware reference samples to other researchers in DM may be sufficient to get your account and IP block suspended in the middle of the con, and no ready way to get back in until customer support gets to your WTF tickets after the conference is already over (VPS providers, datacenters, VPNs, and Tor exit nodes are already usually IP banned by these services, or tarpitted - they only expect this stuff to get used from "normal" business/residential end-user connections, and anything else is suspect/blocked).

For example, Discord has previously banned the r/guns subreddit's guild for, well, being about guns, and they've also wholesale banned *anti*-videogame-cheating research/development groups for necessarily sharing links to cheating software (to develop countermeasures). I think a lack of content-based censorship is a requirement. This is along the same lines of YouTube's recent famous policy gaffe about planning to ban hacking/cracking videos.

I really wish I had more suggestions for large/professional services for which ambiguous content censorship isn't baked in. There don't seem to be any that aren't explicitly catering to people who use them for illegal/violent purposes, which sucks.

[TunnelJumper]

I think it's a really good point that DefCon isn't "normal" or what these companies tend to expect from their users. We really should be cautious of using stuff like Discord. It’s just a disaster waiting to happen IMO. It doesn’t help that their business model doesn’t make a whole lot of sense. They claim to be privacy friendly but reading their privacy policy seems to indicate that they’re still scraping user data. They claim not to be selling user data or using targeted advertising but I see no other way for them to be making the money to support 56 million people every month. I’m reminded of a scene in the IT Crowd where Jen claims to be on the phone with someone but Moss points out that her phone isn’t even plugged in. In other words: I don’t trust em for shit.

[jtl]

Going to stick my head out of the sand here. I know the CCC (chaos) conference in Germany has their own livestreaming infrastructure, and if I remember right they had a Mumble server for live audio streams and some discussion during the conference. I also remember they had an IRC server to ask questions during the talks with webchat. Sure things are a little different because that's a physical conference being broadcast online, but I'm sure something similar could be done with enough resources.

I've been in Mumble servers with hundreds of concurrent users, and assuming it's moderated correctly (everything doesn't have permission to talk over eachother during a talk) it generally works well.

Regarding livestreams, that's a different topic I have less experience in, but a "split" setup where the presenter feeds their livestream into a beefy "ingress" server and that rebroadcasts to Twitch or whatever is worth thinking about.

[TunnelJumper]

>I also remember they had an IRC server to ask questions during the talks with webchat.

Yeah I'm surprised that not many people are recommending IRC. It shouldn't take a super beefy server to run and it works.

[savagejen]

They use hackint for irc and it's up all year long. If yall asked nicely, maybe they would agree to let you use the infrastructure for defcon this year. idk you would have to ask.

[TunnelJumper]

>Thousands of people at a time watching a talk or workshop

This is one of those things you might need to push onto an external service like Twitch or *shudders* YouTube. I’d avoid YT because of their views on hacking content. Y’all already streamed DCTV to Twitch yeah?


>Thousands of people watching or listening to a DJ / VJ

Again Twitch would probably be good for this. I always prefer self hosted but for the streaming stuff it doesn’t seem realistic. Basically so long as it doesn’t require an account on these services to watch I’d imagine people will be fine.


>A way to be social through chat or voice with moderation capabilities to lock out trolls

For voice chat (which would be pretty chaotic with that many people imo) DC916 has had good experience with Jitsi but I have no idea if it scales. Stress testing is needed. No idea of how moderation is on it as we haven't really needed it yet.
For text chat I’ll go along with the recommendations of others and say Riot.im is the way to go. That or plain ol’ IRC.


>Some way to have shared experiences like watching NetFlix or a free streamed movie Mystery Science Theater 3k style.

Streaming to Twitch could cause issues with this due to it being taken down as a copyrighted stream even if you license the film. I’d do it Rifftrax style and just have everyone start the movie at the same time.