infocon.org discussion

[The Dark Tangent]

I'd like to try and get more people involved with infocon.org, finding new content or ideas for how to grow or manage the collection.

I've created an automatic RSS process that will read the torrent rss subscription url https://infocon.org/infocon.org-css/infoconorg-cons.rss and post updates to a thread. It's an experiment on how automatic RSS posting could work.

Anyway, welcome to the infocon.org conversation!

[The Dark Tangent]

With the risk of You Tube taking down security content I've started planning for mirroring educational security channels on YouTube, but before I put a bunch of time into it I wanted to ask if other projects are already doing this? Is it something worthwhile for the community?

[number6]

With the risk of You Tube taking down security content I've started planning for mirroring educational security channels on YouTube, but before I put a bunch of time into it I wanted to ask if other projects are already doing this? Is it something worthwhile for the community?

If nobody else is doing this or it is unorganized, I think we should ask people on twitter, and other social networks to post links to high-risk videos they think are of greatest risk to being taken down by you tube here, or maybe in a private, member's only space, away from non-member search engines.

Then if we are going to archive stuff from youtube, the highest risk for loss items could be copied first.

Things to gather with this? The name of the account that posted the videos, and any contact information for them, so that if youtube takes down a video, we can try to reach them outside of the reach of youtube, and maybe get copies of uploaded videos direct from their creator.

Details would also be useful to help with de-duplication of videos: topic, subject, significant quotes from video, name of creator, date of first creation/upload (to help find oldest/original).

Worry about hte de-duplication later, but gather the details for each while they are available.

[deleite]

With the risk of You Tube taking down security content I've started planning for mirroring educational security channels on YouTube, but before I put a bunch of time into it I wanted to ask if other projects are already doing this? Is it something worthwhile for the community?

I think it is worthwhile for the community, I've learnt a lot of things just with Defcon's Videos. I can help with some hours.

[The Dark Tangent]

This Reddit/Data Hoarder project seems to be in action:
https://airtable.com/shrfo1SnkHU7WHP...9MUfSkK6SjVsa6

And looks like a great source of YT channels to mirror.

The problem? I'd need them to be a list of yt channels and the channels name to be useful, then I can start at the top and work my way down. I could do it but don't have the time to extract it all easier.

For example the format:
Pentester Academy TV: https://www.youtube.com/playlist?lis...i7W0E71TzPZELA

Would be really helpful if someone want to put in some time. Sound interesting deleite ?

[deleite]

This Reddit/Data Hoarder project seems to be in action:
https://airtable.com/shrfo1SnkHU7WHP...9MUfSkK6SjVsa6

And looks like a great source of YT channels to mirror.

The problem? I'd need them to be a list of yt channels and the channels name to be useful, then I can start at the top and work my way down. I could do it but don't have the time to extract it all easier.

For example the format:
Pentester Academy TV: https://www.youtube.com/playlist?lis...i7W0E71TzPZELA

Would be really helpful if someone want to put in some time. Sound interesting deleite ?

It does. Where to start?

[The Dark Tangent]

At the top? :-) Just start by going through that list and extracting the name / YouTube URL and putting it in a list. I'll then take your list and stat backing up the videos.

[deleite]

Something like this?

https://anonymousfiles.io/3NU53lrw/

[The Dark Tangent]

Thank you deleite That is a big help.

I started doing some as a test and this will save a lot of time.

One thing I noticed is any listing that has "playlist?list=" in it is not usable for my tools. I have to then navigate to the actual channel.

For example the link for Billy Ellis is:
https://www.youtube.com/playlist?lis...kKvDGlIhdUQa8A

What I really need is the actual link for his videos, which once you navigate around is:
https://www.youtube.com/user/pr0Hacks2014/videos

[deleite]

Thank you deleite That is a big help.

I started doing some as a test and this will save a lot of time.

One thing I noticed is any listing that has "playlist?list=" in it is not usable for my tools. I have to then navigate to the actual channel.

For example the link for Billy Ellis is:
https://www.youtube.com/playlist?lis...kKvDGlIhdUQa8A

What I really need is the actual link for his videos, which once you navigate around is:
https://www.youtube.com/user/pr0Hacks2014/videos

Ok, I will fix it.

[deleite]

Ok, I will fix it.

So, I managed to extract the video links from the Channel links as well as playlists links.

https://anonymousfiles.io/7fHXYWBc/
https://anonymousfiles.io/YI3xY9xO/

They add over 13k videos.

[The Dark Tangent]

We are getting closer!

There are two format options:

1. I need one link per YT channel, pointing to the "videos" link that will collect all videos from a given channel.
2. I need one link per YT playlist in a channel. This works well for organizing things, but many people don't organize by play list and you end up missing most of their videos because they just put everything in the "videos" (option 1) section.

Here is another example to help describe:

https://www.youtube.com/user/RootOfTheNull/videos
https://www.youtube.com/user/adafruit/videos
https://www.youtube.com/user/bhargavtandel/videos

[deleite]

We are getting closer!

There are two format options:

1. I need one link per YT channel, pointing to the "videos" link that will collect all videos from a given channel.
2. I need one link per YT playlist in a channel. This works well for organizing things, but many people don't organize by play list and you end up missing most of their videos because they just put everything in the "videos" (option 1) section.

Here is another example to help describe:

https://www.youtube.com/user/RootOfTheNull/videos
https://www.youtube.com/user/adafruit/videos
https://www.youtube.com/user/bhargavtandel/videos

I understand. It was simpler, but I ended up learning to use Youtube's API.

Can we communicate faster? (signal/telegram/slack?)

[deleite]

So, following format 1. Found the channel for every playlist and formatted the channels I already have with the youtube.com/channel/CHANNELLID/videos format.

You wrote examples using the username and the Channel name. A user can have more than 1 channel (I haven't found any example on the list), but the other channel may not be related to the topic.

https://anonymousfiles.io/VXLPh9ie/

We are getting closer!

There are two format options:

1. I need one link per YT channel, pointing to the "videos" link that will collect all videos from a given channel.
2. I need one link per YT playlist in a channel. This works well for organizing things, but many people don't organize by play list and you end up missing most of their videos because they just put everything in the "videos" (option 1) section.

Here is another example to help describe:

https://www.youtube.com/user/RootOfTheNull/videos
https://www.youtube.com/user/adafruit/videos
https://www.youtube.com/user/bhargavtandel/videos

[The Dark Tangent]

Thank you for that deleite , my tool will get the channel name from the links you provided so they are ultimately unnecessary.

Now I'll (slowly, one at a time) start at the top and work my way down subscribing to the channels and mirroring the content.

After that I'm not sure what the best plan is for processing.. trans-coding would be the most efficient and space saving, but will take a long time. When new videos are posted I'll have to create a process to make sure I don't spend time encoding the whole channel again but only the new videos. I think I have a solution but will need to test.

Any thoughts?

[deleite]

Thank you for that deleite , my tool will get the channel name from the links you provided so they are ultimately unnecessary.

Now I'll (slowly, one at a time) start at the top and work my way down subscribing to the channels and mirroring the content.

After that I'm not sure what the best plan is for processing.. trans-coding would be the most efficient and space saving, but will take a long time. When new videos are posted I'll have to create a process to make sure I don't spend time encoding the whole channel again but only the new videos. I think I have a solution but will need to test.

Any thoughts?

No problem.

I don't know what's your CPU/Storage capacity. If something like VideoCoin was already working, that would be a good idea. Create a distributed Youtube-like-thing.


To solve the problem regarding new content, I think you have to options:

1. Every video has a hash/ID, so for any channel you could check if you already have that ID stored. If not, download and transcode, add the ID to the completed videos.
2. Save the timestamp of the checking process of finding new videos. Every time this process runs again it will check in every channel for videos with upload timestamp newer than the 'last checked time stamp'. Would be useful to have a different timestamp per channel. That way you'll download only the newest videos.