Data Duplication Village at DEF CON 24!

[The Dark Tangent]

Most all the non-rainbow tables are at http://infocon.org/ The stuff that isn't there3 will be added soon.

Keep the links coming, I'll use this thread to continue gathering.

[DrCob]

Is anyone going to put this up as a torrent?

[KnightOwl]

BTW - DDV is quiet at 3AM :)

[KnightOwl]

To wrap things up in a nice little bow, here's the fun that DDV had for DC24:

We took in a total of 335 drives this year. We were able to duplicate all but 1 of them (someone dropped off an IDE drive they were scammed with through Amazon - lesson learned).

We duplicated almost twice as many InfoCon drives as the others:
Infocon - 168 dupes
Hash/Rainbow1 - 88 dupes
Hash/Rainbow2 - 85 dupes

All this was done with 16 duplicators meaning we were doing 95 concurrent copies at any given moment.
With an average of 120MB/sec for a 1 to many duplication, we were pushing about 11GB of duplication every second in that room.

We played a useful game of "good things/bad things" and have a list ready to improve for DC25 - if you have any other suggestions, let me know!

KO

[KnightOwl]

So what do you do after duplicating over 300 hard drives.... give them back, of course!

Be sure to pick up your hard drives this morning - last pickup time is from 10 to noon today (Sunday). All drives become property of DefCon after noon so don't be late!

[KnightOwl]

Wow -- DDV is going really well! We've completed well about 200 drive duplications already! If you're lucky enough to find a 6TB drive anywhere in Vegas (and believe me, people are getting creative in trying to find one), bring it in now. Today between noon and 3 is the last chance to drop them off.

For the record, the available source drives are:

6TB drive 1-1: All past hacking convention videos that DT could find, plus video collections from popular YouTube channels, and other sources.
6TB drive 1-2: GSM A5/1 hash tables plus freerainbowtables.com hash tables (1-2)
6TB drive 2-2: NTLM, LanMan, remaining freerainbowtables.com hash tables (2-2), and word lists.

By the end of con we will have pumped over 1.8 Petabytes of data through that room! Special thanks go out to DC408 (especially Dan, Jok, Carlos, Warren) and Mike for helping out this year!

[SpinachWasWeed]

In case these sound interesting (they'd be useful for me personally, I am not sure how everyone else feels about these types of resources). Any of the following stuff would be really nice to get.

http://boston.lti.cs.cmu.edu/Data/cl...ml#samplefiles - clueweb09 dataset
https://developers.google.com/freebase/ - notice says this is only even available until the end of the month

https://archive.org/download/github_201602 - github project list
https://archive.org/details/2013-10-...rnmentblackout - blacked out websites in wayback machine
https://archive.org/details/FirstTen...erNewsComments - first 10 million yc forums comments

https://archive.org/details/20140611-rdns.gz - RDNS data for all ipv4s (around 2014) - anyone ahve a more recent one?
https://archive.org/details/archivet...-downloads&and[]=subject%3A"robots.txt" - robots.txt crawls of top alexa sites
https://archive.org/details/archivet...-downloads&and[]=subject%3A"genealogy+records" and https://archive.org/details/archivet...-downloads&and[]=subject%3A"family+history" - geneology site crawls

https://archive.org/details/archiveteam-fire - 'Just in time grabs'
https://archive.org/details/httparchive - pretty big, maybe a few would be nice
https://archive.org/details/20thcenturyweb - lots of 20th century web pages
https://archive.org/details/archivet...-downloads&and[]=subject%3A"hackaday.com" - hackaday archives
https://archive.org/details/archivet...-downloads&and[]=subject%3A"lifehacker.com" lifehacker dumps
https://archive.org/details/archivet...-downloads&and[]=subject%3A"huffingtonpost.com" - huffington post archives
https://archive.org/details/archivet...-downloads&and[]=subject%3A"pastebin" - pastebin scrapes


http://opendata.cern.ch/search?p=Run...tasets&jrec=11 - some of these cern datasets would be really interesting to analyze
ftp://ftp.ensembl.org/pub/release-85/mysql/ - Genome data for hundreds of species including humans

[SpinachWasWeed]

Does this event include data beyond the security niche? general items can be extremely useful for general fun and development.

Stuff like
https://dumps.wikimedia.org/
https://musicbrainz.org/doc/MusicBra...abase/Download
https://archive.org/details/stackexchange

I have a list of these but I'm curious if they fall within the scope of this event.

[KnightOwl]

Do you need any DVD media? Are extra StarTech 1:1 standalone duplicators helpful or would you guys rather have a tower computer with clonezilla and 6(or more) open sata / power ports?

We don't have DVD clones in the plan this year. If you would like to bring 1:1 dupers in, I'm sure they could be used for Friday/Saturday daytime runs. We'll be using 1:10 and 1:5 standalone dupers to get the bulk of the work done. "We're MASS-Communicating here" (must be said with a southern accent for effective movie quote referencing). After the first few batches complete, you could even use the 1:1 dupers to get your own copies done or help out with some of the late comers.

[SporkySpork]

Do you need any DVD media? Are extra StarTech 1:1 standalone duplicators helpful or would you guys rather have a tower computer with clonezilla and 6(or more) open sata / power ports?

[KnightOwl]

... so please keep this in mind: the printed program map is not the most up-to-date...

So true! We're also expecting to have a few signs in place to help redirect people to Palace 7.

[TheCotMan]

Not that he needs validation, but yup yup, knightowl speaks the troof...

More validation is better than less. It is hard for people new to DEFCON to know who has comments that are authoritative. You being one of the CVE (Contests, Village, Events) goons in charge of CVE showing up to say they are who they claim to be, and in charge of this event helps establish their credibility.

In other news, Nikita tweeted an updated map that shows the Lawyer Meetup and "Data Duplication Village" in the space I circled in the map in the top post. The present map published on the main website now has this change reflected: https://defcon.org/images/defcon-24/dc-24-fp-final.png

if you load that latest map, and you do not see the "Tamper Evident Village" located next door to the "Social Engineer Village" on the Bally's side between Elevators and Contests & Events space, then your are probably looking at a cached copy of the old map image. Force-reload a new image to see the update.

The top-post has once again been updated to now include the new map image on the main site instead of my "ghetto" kludgey red circle ms-paint image.

I think this change to the map has happened *after* the deadline for the program work, so please keep this in mind: the printed program map is not the most up-to-date, see the online map from the main website for the most up-to-date map.

[panadero]

Not that he needs validation, but yup yup, knightowl speaks the troof...

[shortstack]

thank you!

[TheCotMan]

Just spoke with DT on this today. I'll help to run the DDV this year.

We will definitely be accepting drives on Thursday from 6pm to 7pm (as noted in the top post) at Palace 7 (thanks for the location update on that, Cot). After that, we'll continue to accept drives on Friday from 10:00am until we fill up the schedule - possibly as late as Saturday afternoon but as Cot mentions, there are no guarantees on the timing. Pickups should be about 16-24 hours after your dropoff and we'll try to give you as accurate an estimate as possible when dropping off.

Space allowing, the last drop-offs will be no later than Saturday at 5:30pm as the last drives will be run overnight.

We have some additional duplicators this year so we hope to handle all comers this year.

-KO

Thanks!

For people reading: accept answers from KnightOwl and DT on how this will work for DEF CON 24 to be authoritative while my guesses were just guesses.