Announcement

Collapse
No announcement yet.

infocon.org discussion

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Dark Tangent
    replied
    We had some down time a couple weeks ago, the torrent servers were running but we didn't have any traffic.

    The tracker was working, torrents showed that they were registered.. strange.

    After poking around for a while I found the issue.

    We use torrents to share our files, and because we are the source of the files we didn't enable DHT (UDP peer finding) but only TCP. If you load our torrent files they point you to our tracker, which then points you to our servers, no need for DHT. I did this to reduce the abuse / reflected amplification (AMP attack) potential of our server.

    Well it turns out, for some reason, that if DHT is disabled no one ever seems to be able to find the files and no seeding happens. Once I re-enabled DHT the packets started flowing immediately.

    Now I have to dig into the issue. Is it an issue with the tracker? Or everyone's clients just want DHT?

    Anyone have any ideas?

    Leave a comment:


  • Dark Tangent
    replied
    Good question deleite

    A quick check shows that the directory of everything is currently around 5T, before transcoding into HEVC.

    I'll first start on all the lock picking related channels.

    Leave a comment:


  • deleite
    replied
    Originally posted by Dark Tangent View Post
    Finally got the list from deleite fully subscribed to and everything downloaded. Now to decide which ones to encode and host.

    Some on the list are interesting to back up, but not really relevant to a community of hackers.

    Next up I'll add some maker / electronics hacking related channels.
    Great. I wonder how many TB you will need.

    Filter or categorizing videos of this kind is very difficult, I have been thinking how to sort the ones I'm interested about. Some are talks with no real value (except historial), some other updated tools/vulns, some other step by step tutorials and some other interesting/educational stories (like Jason Street talks).

    The challenge is how to sort all this information in a way that senior hackers, newbies and other kind of people could find what they are looking for with less effort.

    The only solution I see is having a crowd rating/categorizing system. With the 'crowd' some experts/hackers/speakers that can use their experience to sort this out a bit.

    PS: I sent you an email inviting you to a conference next year.

    Leave a comment:


  • Dark Tangent
    replied
    Finally got the list from deleite fully subscribed to and everything downloaded. Now to decide which ones to encode and host.

    Some on the list are interesting to back up, but not really relevant to a community of hackers.

    Next up I'll add some maker / electronics hacking related channels.

    Leave a comment:


  • tomigotcha
    replied
    This sounds like a great project! Feel free to let me know if I can help out too.

    Leave a comment:


  • deleite
    replied
    Ok, let me know if help is needed.

    Leave a comment:


  • Dark Tangent
    replied
    Sorry for being quiet, but I'm just slowly getting the subscriptions set up and everything mirroring. Once I'm done with that and the con is over I'll start the automation for transcoding and managing updates.
    Last edited by Dark Tangent; July 31st, 2019, 18:45.

    Leave a comment:


  • deleite
    replied
    Originally posted by Dark Tangent View Post
    Thank you for that deleite , my tool will get the channel name from the links you provided so they are ultimately unnecessary.

    Now I'll (slowly, one at a time) start at the top and work my way down subscribing to the channels and mirroring the content.

    After that I'm not sure what the best plan is for processing.. trans-coding would be the most efficient and space saving, but will take a long time. When new videos are posted I'll have to create a process to make sure I don't spend time encoding the whole channel again but only the new videos. I think I have a solution but will need to test.

    Any thoughts?
    No problem.

    I don't know what's your CPU/Storage capacity. If something like VideoCoin was already working, that would be a good idea. Create a distributed Youtube-like-thing.


    To solve the problem regarding new content, I think you have to options:

    1. Every video has a hash/ID, so for any channel you could check if you already have that ID stored. If not, download and transcode, add the ID to the completed videos.
    2. Save the timestamp of the checking process of finding new videos. Every time this process runs again it will check in every channel for videos with upload timestamp newer than the 'last checked time stamp'. Would be useful to have a different timestamp per channel. That way you'll download only the newest videos.


    Leave a comment:


  • Dark Tangent
    replied
    Thank you for that deleite , my tool will get the channel name from the links you provided so they are ultimately unnecessary.

    Now I'll (slowly, one at a time) start at the top and work my way down subscribing to the channels and mirroring the content.

    After that I'm not sure what the best plan is for processing.. trans-coding would be the most efficient and space saving, but will take a long time. When new videos are posted I'll have to create a process to make sure I don't spend time encoding the whole channel again but only the new videos. I think I have a solution but will need to test.

    Any thoughts?

    Leave a comment:


  • deleite
    replied
    So, following format 1. Found the channel for every playlist and formatted the channels I already have with the youtube.com/channel/CHANNELLID/videos format.

    You wrote examples using the username and the Channel name. A user can have more than 1 channel (I haven't found any example on the list), but the other channel may not be related to the topic.

    https://anonymousfiles.io/VXLPh9ie/


    Originally posted by Dark Tangent View Post
    We are getting closer!

    There are two format options:
    1. I need one link per YT channel, pointing to the "videos" link that will collect all videos from a given channel.
    2. I need one link per YT playlist in a channel. This works well for organizing things, but many people don't organize by play list and you end up missing most of their videos because they just put everything in the "videos" (option 1) section.
    Here is another example to help describe:

    https://www.youtube.com/user/RootOfTheNull/videos
    https://www.youtube.com/user/adafruit/videos
    https://www.youtube.com/user/bhargavtandel/videos

    Leave a comment:


  • deleite
    replied
    Originally posted by Dark Tangent View Post
    We are getting closer!

    There are two format options:
    1. I need one link per YT channel, pointing to the "videos" link that will collect all videos from a given channel.
    2. I need one link per YT playlist in a channel. This works well for organizing things, but many people don't organize by play list and you end up missing most of their videos because they just put everything in the "videos" (option 1) section.
    Here is another example to help describe:

    https://www.youtube.com/user/RootOfTheNull/videos
    https://www.youtube.com/user/adafruit/videos
    https://www.youtube.com/user/bhargavtandel/videos
    I understand. It was simpler, but I ended up learning to use Youtube's API.

    Can we communicate faster? (signal/telegram/slack?)

    Leave a comment:


  • Dark Tangent
    replied
    We are getting closer!

    There are two format options:
    1. I need one link per YT channel, pointing to the "videos" link that will collect all videos from a given channel.
    2. I need one link per YT playlist in a channel. This works well for organizing things, but many people don't organize by play list and you end up missing most of their videos because they just put everything in the "videos" (option 1) section.
    Here is another example to help describe:

    https://www.youtube.com/user/RootOfTheNull/videos
    https://www.youtube.com/user/adafruit/videos
    https://www.youtube.com/user/bhargavtandel/videos

    Leave a comment:


  • deleite
    replied
    Originally posted by deleite View Post

    Ok, I will fix it.
    So, I managed to extract the video links from the Channel links as well as playlists links.

    https://anonymousfiles.io/7fHXYWBc/
    https://anonymousfiles.io/YI3xY9xO/

    They add over 13k videos.

    Leave a comment:


  • deleite
    replied
    Originally posted by Dark Tangent View Post
    Thank you deleite That is a big help.

    I started doing some as a test and this will save a lot of time.

    One thing I noticed is any listing that has "playlist?list=" in it is not usable for my tools. I have to then navigate to the actual channel.

    For example the link for Billy Ellis is:
    https://www.youtube.com/playlist?lis...kKvDGlIhdUQa8A

    What I really need is the actual link for his videos, which once you navigate around is:
    https://www.youtube.com/user/pr0Hacks2014/videos

    Ok, I will fix it.

    Leave a comment:


  • Dark Tangent
    replied
    Thank you deleite That is a big help.

    I started doing some as a test and this will save a lot of time.

    One thing I noticed is any listing that has "playlist?list=" in it is not usable for my tools. I have to then navigate to the actual channel.

    For example the link for Billy Ellis is:
    https://www.youtube.com/playlist?lis...kKvDGlIhdUQa8A

    What I really need is the actual link for his videos, which once you navigate around is:
    https://www.youtube.com/user/pr0Hacks2014/videos


    Leave a comment:

Working...
X