Archiving and You

[Did You Know]

: OP edited to include a link to the relevant post discussing how to archive text, twitter posts and other similar sources.

Link to additional archival tools, mostly text-focused.

You there Mister Slur, do you want to be an archivist? Maybe you want a record of your Oshi's activities. Just in case. She might be a Yamanbasanji who decides that she doesn't want to leave behind any videos when she graduates, maybe she has suicidal thoughts and might just apply for Canadian Healthcare, maybe prepping for if she yabs and reveals she's either got a boyfriend or she's some whore who pays to fucks hosts with your supas before being blackmailed like a bitch. In all these cases, It's all downhill from there and the possibility of the channel being lost is high. Or maybe you're like me, someone who hates the twitch media player with a passion because it's fucking hot garbage and makes me want to run someone over every time I try and play a vod.

But I don't know shit and random googling gives recommends me suspicious download links. Fear not, actual archivists have your back with these tools. Probably. If I'm wrong someone will correct me anyway, so it's nice to have. Thanks to archivebro for actually having the setup, @thhrang for forwarding it to @Godzilla1984 and @Tatsunoko for the how to on downloading member streams.

https://github.com/yt-dlp/yt-dlp
https://github.com/Kethsar/ytarchive
https://github.com/streamlink/streamlink
https://github.com/xenova/chat-downloader
https://github.com/Instinctlol/automatic-twitch-recorder

Though there's also Jdownloader2, which I can attest to working on youtube at the very least: https://jdownloader.org/jdownloader2

How to download member streams:

1. Download yt-dlp. Put it in Users/*name* folder.
2. Get your YouTube cookings by going to YouTube.com and using something like "get cookies.txt" or "export cookies" (the latter works better for me personally).
3. Save that file in your Users/*name* folder and optionally rename it to "cookies-youtube-com.txt"
4. Start command prompt/CMD
5. Fill in : "yt-dlp.exe -f 22 --cookies youtube.com_cookies.txt -o C:\Wherever\Youwant\%(title)s.%(ext)s https://www.youtube(dot)com/watch?v=whatever"

-f 22 is the file format. 22 is normally 720p, which is more than enough for me personally. You can use "yt-dlp.exe -F https://www.youtube(dot)com/watch?v=whatever" to see a list of available file formats.
--cookies is the command that allows yt-dlp to use your cookies to be logged into YouTube.
-o is the directory where you want the file to be downloaded to.
%(title)s.%(ext)s is the format that the file is saved as.
And you add the link of the video you want to download at the end.

Edit: There's a section here for using OBS that godzilla posted on the farms but I'll just edit it in later if he doesn't post it here first.

 

[Arznbae]

"ytarchive -k -t --vp9 --monitor-channel --no-frag-files https://www.youtube.com/@CHANNELNAME 720p60/best"

This is what I use. It monitors the given channel and starts downloading whenever said channel goes live. Set it and forget it.

 

[Godzilla1984]

Spoiler: OBS LIVESTREAM RECORDING NOTES:

OBS LIVESTREAM RECORDING NOTES:
1. Open OBS.
2. Add Source.
3. Select Browser.
4. Create new. Rename it if you want. Click OK.
5. Paste URL. Change default Width and Height from 800 x 600 to 1920 x 1080. Select Control audio via OBS. Click OK.
6. Having multiple sources already made is nice to start recording quicker instead of having to set up new sources every time. A Browser>Twitter source, a Browser>Twitch source, a Browser>YouTube source covers most bases for VTubers. Just unlock and make visible then interact as normal.
7. Right click workspace > Transform > Fit to screen. You should only have to do this one after the initial setup, but sometimes OBS updates will reset the window positions, so just Fit to screen again.
8. Click Interact to launch a window to actually interact with the browser. Maximize stream window when live. Remember to set relevant stream quality. Close Interact window.
9. Default audio will record when recording, but you won't hear it. If you want to hear the audio while it is recording go to Audio Mixer > Browser > Settings > Advanced Audio Properties > Audio Monitoring > Select Monitor and Output. Audio Monitoring > Select Monitor and Output is temperamental and I need to toggle it on and off from time to time to be able to hear it.
10. Click Start Recording. End Recording when stream ends.
Bonus. If the OBS Browser window doesn't want to scroll, replace the browser source Custom CSS with:
body { background-color: #000000; }
To clear up the vanished scroll bar error that OBS Browser Source has.

Spoiler: UI Docking Layout

I am lazy.

Spoiler: Source Settings

They're all the same.

Spoiler: Recording Settings

Tailor to personal preference as each video card has different options.

Spoiler: Links

Advanced OBS settings: What they are and how to use them
Best OBS Settings for Recording - All You Need to Know!
Here are the Best OBS Settings for Recording YouTube videos
What is the Advanced Audio Properties in OBS Studio?

 

[BlueSharkTV]

Here are the commands i use on yt-dlp, why use commands and not just paste the link? sometimes it will default to webm or mkv which is not really good for some software types.
Remember to use yt-dlp -U to update the app.

This commands will download mp4 1080p, to change quality just add [height<=720] in between [ext=mp4] and +ba, replace the 720 for either 480, 360 or whatever shit quality you desire.
Use cookies from browser if it's a member stream, you need to be a member it isn't magic it will not let you download it if you're not a member, (i use edge but i'm pretty sure you can just change it to your browser's name, there's no need to make a file for cookies, you just need to enter the command every time you need them, you can delete --cookies-from-browser if you don't need it)

for livestreams (will start checking every 10 seconds if the stream started until it starts, you can change the 10 to whatever number you want (in seconds) and it will check that often to see if the stream started)

yt-dlp --live-from-start -f "bv*[ext=mp4]+ba/b[ext=mp4] / bv*+ba/b" --ignore-errors --wait-for-video 10 --cookies-from-browser edge

If you want to download a vod (use cookies from browser if it's members)

yt-dlp -f "bv*[ext=mp4]+ba[ext=m4a]/b[ext=mp4] / bv*+ba/b" --ignore-errors --cookies-from-browser edge

If you only care about the audio of a livestream you can use this one.
(you can just use yt-dlp -f 140 --ignore-errors if its a vod).

yt-dlp -f 140 --ignore-errors --live-from-start --wait-for-video 10 --cookies-from-browser edge

for clipping a vod you can use this one (you need to wait for the vod to prosses or it will give you the wrong timestamp).

yt-dlp -f "bv*[ext=mp4]+ba[ext=m4a]/b[ext=mp4] / bv*+ba/b" --ignore-errors --download-sections "*HH:MM:SS-HH:MM:SS"

 

[Aliepheese]

Does anyone has any tips of how to download Twitch VODS?

 

[Godzilla1984]

Does anyone has any tips of how to download Twitch VODS?

JDownloader 2 is temperamental with Twitch so I use WinX YouTube Downloader.

 

[agility_]

To nab videos off twitter (yeah I know) I use ffmpeg and my browser's editor function to locate the assinine .m3u address. With the url copied, I paste it to this command line

ffmpeg -i $url -c copy hanamacchiabeingmental.m4a

where [ -i] is the command for sourcing the input file be it audio or video
[ -c] tells ffmpeg to copy the output file in the specified filename and format, it doesn't have to be .m4a

Ffmpeg lets you fuck around with video and audio codecs using -vcodec copy and -acodec copy respectively but I'm not going to get into those here as I haven't run into issues with defaults. Speaking of defaults, unless you tell it otherwise your file will be placed under //home/User/Desktop/hanamacchiabeingmental.m4a
You can also find docs on how to trim and crop your video. Ffmpeg is really fast and versatile and if you put in the effort to learn its commands it'll definitely be worth it in the long run.

 

[Fucking YTs]

Does anyone has any tips of how to download Twitch VODS?

I've been using TwitchLink the last few weeks. It allows you to live download, schedule, as well as download VODs pretty much perfectly. The downside is you have to compile by source if it's not a Windows machine.

GitHub - devhotteok/TwitchLink: Twitch Stream & Video & Clip Downloader/Recorder. The best GUI utility to download/record Broadcasts/VODs/Clips.

Twitch Stream & Video & Clip Downloader/Recorder. The best GUI utility to download/record Broadcasts/VODs/Clips. - GitHub - devhotteok/TwitchLink: Twitch Stream & Video & Clip Downl...

github.com

 

[Godzilla1984]

With my recent travelling, I need to shill some mobile apps.
TubeMate is a good mobile video downloader, but as with everything else, Twitch just has to be difficult.
VodTwit answers the mobile Twitch video downloader question.

 

[Short]

My stream archive setup:

I use ytarchive, streamlink, FFmpeg, and chat downloader, mainly with custom bat files
just install chat downloader, streamlink, ytarchive, FFmpeg, and put them in the same folder together with the bat files.

ytarchive:

@echo off
set /p url_and_quality="Url and quality"
start "" ytarchive -t --vp9 --add-metadata --monitor-channel %url_and_quality%

It will ask you to put the video or channel URL and the video quality
Ex: https://www.youtube.com/watch?v= best
If no quality is input it will ask you for one after you press enter

streamlink:

@echo off
setlocal enabledelayedexpansion

@TITLE streamlink-monitor: Name

set "firstInput="
set "channelName="

:input
if not defined firstInput (
    set /p channelName="Enter twitch channel name here: "
    set "firstInput=1"
)

streamlink -o "{author}-{title}({id})-{time:%%Y%%m%%d-%%H%%M}.mp4" https://www.twitch.tv/%channelName% best --twitch-disable-hosting --twitch-disable-ads
timeout /T 30
goto input

it will ask for the Twitch channel name, for example if you want to monitor and download a Pippa stream just type pippa and press enter

chat downloader:

@echo off
set /p url="Enter the URL for chat_downloader: "
set /p output="Enter the output file name (e.g., chatlog.txt): "
chat_downloader %url% --output %output%

it will ask you to input the video or channel url after that will ask you to give the file a name

notes:
ytarchive --monitor seems to not work on youtube channels that have more than one waiting room on the latest version, try using the Pre-release version, you might need to add /live after the channel name.
When using a channel URL for a channel that has more than one waiting room chat downloader will default to the older room.
the streamlink bat will try to find the stream every 30 seconds, if you want to change the time it takes to loop just change "timeout /T 30" to whatever number you want.
If you want to archive a Twitch stream with streamlink on a different quality you can change from "best" to "720p,720p60,best" on the bat file for example.

Extra Tools:
Twitch Downloader - for twitch vods, I recommend using the GUI
ytdlp-interface - good GUI for yt-dlp
chatty - twitch chat client that has a lot of neat features, like automatically making chatlogs, shows all users connected to the channel and their individual chat logs, can be connected to Streamlink so you don't even need to open Twitch to view the stream (although there is a pretty high delay on watching the stream with Streamlink), and others.

More than happy for suggestions on how to make it better!

 

[21st Century Pipkin Man]

Edit to add an important note: After some testing, it seems that this currently works for VODs for channels that have set VODs for subscriber only, but it does not work for VODs of streams that resulted in suspensions, as those return a 403 error. If you expect or witness some real wack shit, record it locally!

Wanted to share something pretty neat I haven't seen mentioned elsewhere.

No Twitch VOD? No problem! "VodRecovery is a Python script designed to access Twitch videos and clips that are either restricted to subscribers or have been deleted."

GitHub - Shishkebaboo/VodRecovery: The purpose of this script is to obtain videos or clips that are either marked as "sub-only" or have been deleted on Twitch.

The purpose of this script is to obtain videos or clips that are either marked as "sub-only" or have been deleted on Twitch. - Shishkebaboo/VodRecovery

github.com

• I've been able to get recent broadcasts reliably with this. Older discussions on the subject suggest you can recover any VOD deleted less than two months ago, but that no longer seems to be the case, so be prepared to run this soon after the stream ends. I'd guess two weeks is the max for affiliates.
• Usually seems to take about 30 minutes from the broadcast's end for you to be able to get the VOD link
• Can also recover deleted clips and get an unmuted version of the VOD This no longer appears to work at all.
• Does not appear to always work for streamers who aren't affiliate or partner. My best guess is that if there's no clips and they delete the vod, it's actually gone.

 

[Planetary Bombardment]

Thank you to everyone who's contributed here as well as @Short specifically who's provided assistance via Matrix. I've arrived at my first members-only stream and I'm having trouble because I can't seem to get yt-dlp or ytdlp-interface to play nice. Note that I am logged into an account that has membership access to the given channel on both Firefox and Chrome. I have the Firefox extension cookies.txt and the Chrome extension Get cookies.txt LOCALLY as recommended by some cursory searching.

Attempting to use any of these arguments in either the command line or in the "custom arguments" section of the interface does not work.

--cookies C:\Users\lolmyusername\cookies.txt (as exported from either the Firefox or Chrome extension)
--cookies-from-browser firefox
--cookies-from-browser chrome

Any ideas in regards to what I'm screwing up?

 

[BlueSharkTV]

Thank you to everyone who's contributed here as well as @Short specifically who's provided assistance via Matrix. I've arrived at my first members-only stream and I'm having trouble because I can't seem to get yt-dlp or ytdlp-interface to play nice. Note that I am logged into an account that has membership access to the given channel on both Firefox and Chrome. I have the Firefox extension cookies.txt and the Chrome extension Get cookies.txt LOCALLY as recommended by some cursory searching.

Attempting to use any of these arguments in either the command line or in the "custom arguments" section of the interface does not work.

--cookies C:\Users\lolmyusername\cookies.txt (as exported from either the Firefox or Chrome extension)
--cookies-from-browser firefox
--cookies-from-browser chrome

Any ideas in regards to what I'm screwing up?

if you have a chromium based browser open you can't access cookies anymore, or download anything while it's open either (at least it seems like it may be an issue for me so it may be the same thing)

 

[Planetary Bombardment]

if you have a chromium based browser open you can't access cookies anymore, or download anything while it's open either (at least it seems like it may be an issue for me so it may be the same thing)

I don't know which of these things actually fixed the problem but here's the workflow. It is admittedly partially black magic but it's repeatable. Additional context: what I was trying to download was not an active livestream. The winning configuration for anyone in the future that's as retarded as I am:

Command line:
yt-dlp --cookies "C:\Users\lolmyusername\cookies2.txt" https://www.youtube. com/watch?v=H-QzqpNXNJA best

I've broken that URL (by adding a space) on purpose to keep the forum from thinking its a link.

Interface:
Check Custom arguments: and add --cookies "C:\Users\lolmyusername\cookies2.txt"

The trick is apparently to do everything in this order:

• Open ytdlp-interface.
• Add to custom arguments --cookies "C:\Users\lolmyusername\cookies2.txt" (or wherever/whatever you intend to save the file as).
• Navigate to the stream with Firefox, and copy the video URL to your clipboard.
• Hit the Extension (puzzle piece) button, then select the cookies.txt extension -> Current site -> file name and location at your discretion.
• Alt + Tab straight back to ytdlp-interface.
• Copy + Paste the URL into the center target box. It will show a 'failed to get info' error.
• Start download anyway.
• At no point should you close Firefox until you're done with downloading. The cookies file breaks otherwise, and you (might) have to remake it for every video.

 

[00000]

what's the best practice for archiving tweets these days? archive.ph only archives individual tweets now and can't do entire threads anymore, i'm guessing because of twitter's api changes?

 

[Porean]

Brothers, i bring you two new(?) tools;

1. I see many complaints about keeping yt-dl and yt-dlp updated, well i have recommended this before but TARTUBE is excellent for this. Besides having a GUI making it useable for those of us with merely Asperger's and not full-blown Autism, it's one click updatable and includes installs for FFMPEG, matplotlib and streamlink, and has a built in clipper via timestamps.

2. Given talk of subtitling and me recently doing some of that myself, i have tried this program; WHISPER GUI It is pretty reliant on having a Nvidia card (10xx series or up) but it transcribes videos or audio files and makes a subtitle file from it, and I've tried it on both videos and podcast with multiple speaker and the results have been great, besides some hang-ups on names and such, but even given the time you have to spend on cleaning that up it is a huge time-saver.

[EDIT;] i have here an example, attached to this post is a .rar with a generated unedited SRT file made from this clip-vid i made;

it's VERY accurate, and it even timestamps every line of dialogue. I highly recommend you check whispers-gui out.

 

[21st Century Pipkin Man]

Brothers, i bring you two new(?) tools;

1. I see many complaints about keeping yt-dl and yt-dlp updated, well i have recommended this before but TARTUBE is excellent for this. Besides having a GUI making it useable for those of us with merely Asperger's and not full-blown Autism, it's one click updatable and includes installs for FFMPEG, matplotlib and streamlink, and has a built in clipper via timestamps.

2. Given talk of subtitling and myself recently doing some of that myself, i have tried this program; WHISPER GUI It is pretty reliant on having a Nvidia card (10xx series or up) but it transcribes videos or audio files and makes a subtitle file from it, and I've tried it on both videos and podcast with multiple speaker and the results have been great, besides some hang-ups on names and such, but even given the time you have to spend on cleaning that up it is a huge time-saver.

[EDIT;] i have here an example, attached to this post is a .rar with a generated unedited SRT file made from this clip-vid i made;

it's VERY accurate, and it even timestamps every line of dialogue. I highly recommend you check whispers-gui out.

This is impressive. I ran it on a RTX2070(8gb vram), and for my test case I used a very ESL chuuba, worse than Pippa tier.
The medium model was about 60% correct, while large-v2 was more like 90%. Timings could be a little odd, but most of it was usable as is, and mostly needed something like chopping up a large subtitle block into smaller ones.
What's extra cool is, it also seemed to transcribe perfectly fine when they switched languages, although it only seemed to translate when the most of the source clip was in another language, even when the source language was forced to something else. This seems like something that would be fixed in the future, or would be fixable using parameters if running without WhisperGUI. Translations themselves seemed OK, more DeepL than Google Translate.

This is a massive upgrade over anything I've seen so far. Thank you.

 

[GOD'S STRONGEST BUILDERBEAR]

what file hosts to use for big files? I have multiple 12 gb vids

 

[Porean]

what file hosts to use for big files? I have multiple 12 gb vids

Unless you need others to easily download it, uploading to YouTube could be an option, if only you need the vids, keep em private. Unfortunately few file-hosts offer unpaid storage to the size you want.
[EDIT] Gofile would let you upload your files, however they are time limited without an account, if that is no issue that could be an option for you.

 

[21st Century Pipkin Man]

Unless you need other to easily download it, uploading to You-tube could be an option, if only you need the vids, keep em private. Unfortunately few file-hosts offer unpaid storage to the size you want.
[EDIT] Gofile would let you upload your files, however they are time limited without an account, if that is no issue that could be an option for you.

YouTube is great for squirreling away data for archival/hoarding purposes like this. However, you need to verify a phone number or fulfill some nebulous activity/upload requirement to be able to upload larger videos.

E: Additionally, and even after the first verification step, looks like you'll need to submit your actual ID or a short video to Google, or only upload a relatively small amount of videos per day. Both of these can also be bypassed by growing a channel organically, so it's basically only an issue for using a hidden channel for uploading a bunch of privated shit.