Initial commit

This commit is contained in:
thecookingsenpai 2023-12-25 13:30:31 +01:00
commit 05c4393ba1
17 changed files with 748 additions and 0 deletions

0
.env Normal file
View File

7
.gitignore vendored Normal file
View File

@ -0,0 +1,7 @@
__pycache__
*.mp3
*.mp4
*.wav
downloads/*.mp3
downloads/*.mp4
downloads/*.wav

8
.trunk/.gitignore vendored Normal file
View File

@ -0,0 +1,8 @@
*out
*logs
*actions
*notifications
*tools
plugins
user_trunk.yaml
user.yaml

View File

@ -0,0 +1,2 @@
[settings]
profile=black

View File

@ -0,0 +1,10 @@
# Autoformatter friendly markdownlint config (all formatting rules disabled)
default: true
blank_lines: false
bullet: false
html: false
indentation: false
line_length: false
spaces: false
url: false
whitespace: false

View File

@ -0,0 +1,7 @@
enable=all
source-path=SCRIPTDIR
disable=SC2154
# If you're having issues with shellcheck following source, disable the errors via:
# disable=SC1090
# disable=SC1091

View File

@ -0,0 +1,10 @@
rules:
quoted-strings:
required: only-when-needed
extra-allowed: ["{|}"]
empty-values:
forbid-in-block-mappings: true
forbid-in-flow-mappings: true
key-duplicates: {}
octal-values:
forbid-implicit-octal: true

5
.trunk/configs/ruff.toml Normal file
View File

@ -0,0 +1,5 @@
# Generic, formatter-friendly config.
select = ["B", "D3", "E", "F"]
# Never enforce `E501` (line length violations). This should be handled by formatters.
ignore = ["E501"]

43
.trunk/trunk.yaml Normal file
View File

@ -0,0 +1,43 @@
# This file controls the behavior of Trunk: https://docs.trunk.io/cli
# To learn more about the format of this file, see https://docs.trunk.io/reference/trunk-yaml
version: 0.1
cli:
version: 1.18.1
# Trunk provides extensibility via plugins. (https://docs.trunk.io/plugins)
plugins:
sources:
- id: trunk
ref: v1.4.1
uri: https://github.com/trunk-io/plugins
# Many linters and tools depend on runtimes - configure them here. (https://docs.trunk.io/runtimes)
runtimes:
enabled:
- go@1.21.0
- node@18.12.1
- python@3.10.8
# This is the section where you manage your linters. (https://docs.trunk.io/check/configuration)
lint:
enabled:
- checkov@3.1.42
- taplo@0.8.1
- trivy@0.48.1
- yamllint@1.33.0
- bandit@1.7.6
- black@23.12.0
- dotenv-linter@3.3.0
- git-diff-check
- isort@5.13.2
- markdownlint@0.38.0
- osv-scanner@1.5.0
- prettier@3.1.1
- ruff@0.1.9
- shellcheck@0.9.0
- shfmt@3.6.0
- trufflehog@3.63.5
actions:
disabled:
- trunk-check-pre-push
- trunk-fmt-pre-commit
enabled:
- trunk-announce
- trunk-upgrade-available

7
LICENSE.md Normal file
View File

@ -0,0 +1,7 @@
Copyright 2023, TheCookingSenpai
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

123
README.md Normal file
View File

@ -0,0 +1,123 @@
# TransTerm
A terminal based YouTube video and playlist downloader (mp4, mp3, wav) and transcriber with a nice TUI interface.
![transterm](https://i.imgur.com/woRKbiK.png)
PS. The transparency and retro term is proper of my KDE Plasma configuration
🌲🌲🌲🦌🌲🌲🌲🦌🌲🌲🌲
**By utilizing TransTerm, you acknowledge that you have read, comprehended, and agreed to the terms set forth in the disclaimer at the bottom of this file.**
# Table of Contents
1. [What is TransTerm?](#what-is-this)
2. [Features](#features)
3. [Installation](#installation)
- [For Mac & Linux](#installation-for-mac--linux)
- [Troubleshooting](#troubleshooting)
- [For Windows](#installation-for-windows)
4. [Usage](#usage)
5. [Manual Mode (Debug)](#manual-mode-aka-debug)
6. [Contributing](#contributing)
7. [License](#license)
8. [Disclaimer](#disclaimer)
- [Terms and Conditions](#terms-and-conditions)
- [Appropriate Use](#appropriate-use)
- [Copyright Adherence](#copyright-adherence)
- [Personal Usage](#personal-usage)
- [Third-Party Content](#third-party-content)
- [Limitation of Responsibility](#limitation-of-responsibility)
## What is this
TransTerm is an highly experimental text based graphical user interface to act on YouTube videos.
Being text based, this program runs even in the terminal.
## Features
- Download any youtube video at the highest resolution by default in mp4 format
- Is able to automatically convert the downloaded video both in mp3 or wav format
- Playlist support for the above features including automatically rename the files using the video title and the channel name
- Transcribe a ssingle downloaded video using either:
• Google Audio to Text
• Google Audio to Text + Silence detection
• Sphynx CMU (processed offline locally)
## Installation for Mac & Linux
git clone https://github.com/thecookingsenpai/transterm && cd transterm
sudo chmod +x install.sh
./install.sh
## Troubleshooting
In some cases, where python3 environment is managed by the system and not by pip3, it is advisable to install the dependencies manually as you would do with your system. If you want to force the installation anyway, first run:
pip3 install -r requirements.txt --break-system-packages
The same general solution applies to any requirements problem. PLease note that using the above command is not recommended as it could break your python installation.
## Installation for Window
git clone https://github.com/thecookingsenpai/transterm && cd transterm && pip3 install -r requirements.txt
## Usage
python3 gui.py
## Manual mode (aka debug)
You can edit the bottom part of term.py so that after the
if __name__ == "__main__":
condition you can write your own logic. Then, launch with:
python3 term.py
For convenience, a playlist example is already included (with the best playlist ever, I'd say).
## Contributing
You are welcome and free to contribute to the project. To do that, you have a few ways:
- Add some kind of functionality and create a PR
- Search the lines marked with TODO and FIXME in the code for the most urgent things
- In any case, thank you!
## License
MIT License
## Disclaimer
### Terms and Conditions:
TransTerm is a YouTube downloader and transcriber application developed for personal, non-commercial use. Users must ensure that their utilization of this tool complies with all relevant laws and regulations in their jurisdiction.
### Appropriate Use:
#### Copyright Adherence:
Users must respect the copyrights and intellectual property rights of content creators. TransTerm should not be utilized to infringe upon copyrights or distribute content without proper authorization.
#### Personal Usage:
The application is intended for personal use, allowing users to download and transcribe YouTube videos for lawful purposes such as education, research, or similar activities. Commercial use or distribution of downloaded/transcribed content without authorization is strictly prohibited.
#### Third-Party Content:
Users are responsible for confirming that they possess the legal rights to download and transcribe any third-party content from YouTube. TransTerm does not endorse or encourage the unauthorized downloading or distribution of copyrighted materials.
### Limitation of Responsibility:
The developers of TransTerm are not accountable for any misuse or illegal use of the tool. Users accept all associated risks with their use of the application and agree to comply with applicable laws and YouTube's terms of service.
**_Please Note: This disclaimer does not serve as legal advice. Users are encouraged to seek legal counsel if they have queries regarding the legality of their use of TransTerm in their specific jurisdiction._**
**By utilizing TransTerm, you acknowledge that you have read, comprehended, and agreed to the terms set forth in this disclaimer.**

1
downloads/placeholder Normal file
View File

@ -0,0 +1 @@
why are you looking at me, my only purpose is to create my parent folder

238
gui.py Normal file
View File

@ -0,0 +1,238 @@
import random
import threading
from statistics import mean
from textual import events
from textual.app import App, ComposeResult
from textual.containers import Horizontal
from textual.widgets import (
Button,
Checkbox,
Footer,
Header,
Input,
Label,
RichLog,
Select,
Sparkline,
)
import term
class TransTerm(App):
CSS_PATH = "meshterm.tcss"
lock = False
stopWatchdog = False
messageToShow = None
# INFO Composing the app
def compose(self) -> ComposeResult:
"""Create child widgets for the app."""
yield Header()
yield Footer()
yield Label(
"TransTerm - A simple terminal-based YouTube downloader and transcriber by TheCookingSenpai\n",
classes="title",
)
yield Label(
"Yellow -> idle • Red -> busy • Green -> success\n\n",
classes="status_yellow",
id="status",
)
# Inputs
yield Label("Enter a YouTube link to work with:")
yield Input(
placeholder="https://www.youtube.com/watch?v=dQw4w9WgXcQ",
id="link",
)
# Configuration
yield Horizontal(
Checkbox(
"Download a playlist",
id="downloadPlaylist",
),
Checkbox(
"Convert to mp3",
id="toMp3",
),
Checkbox(
"Convert to wav",
id="toWav",
),
Checkbox(
"Transcript to text",
id="toText",
),
)
options = [
("Google Simple", "google"),
("Google with silence detection", "google_silence"),
("Local (using Sphynx)", "local"),
]
yield Select(options, id="engine")
# Buttons
yield Button("Go", id="go")
yield Button("Exit", id="exit")
# Infos
yield Label("Video title: ", id="video_title")
yield Label("Video author: ", id="video_author")
yield Label("Video length: ", id="video_length")
yield Label("Configuration: ", id="configuration")
random.seed(73)
data = [random.expovariate(1 / 3) for _ in range(1000)]
yield Sparkline(data, summary_function=mean, id="divisor")
yield RichLog(id="main_log")
# SECTION Actions
def on_key(self, event: events.Key) -> None:
"""Handle key events."""
pass
def on_button_pressed(self, event: Button.Pressed) -> None:
# sourcery skip: extract-method, switch
"""Handle button events."""
action = str(event.button.id).lower()
if action == "exit":
try:
term.forceQuit = True
except Exception:
print("[SYSTEM] Failed to stop thread")
exit(1)
elif action == "go":
if self.lock:
self.query_one("#main_log").write("Already working!")
return
self.lock = True
status = self.query_one("#status")
status.classes = "status_red"
status.text = "Status: working..."
self.query_one("#main_log").write("Proceeding...")
self.query_one("#main_log").write("Params extraction...")
self.process = threading.Thread(name="act", target=self.act)
self.query_one("#main_log").write("Started!")
self.process.start()
def act(self):
link = self.query_one("#link")
link = link.value
if link == "":
link = "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
self.query_one("#main_log").write(f"Link: {link}")
toMp3 = self.query_one("#toMp3")
toMp3 = toMp3.value
toWav = self.query_one("#toWav")
toWav = toWav.value
toText = self.query_one("#toText")
toText = toText.value
isPlaylist = self.query_one("#downloadPlaylist")
isPlaylist = isPlaylist.value
# Branching: in playlist mode we download (and if needed convert) all the videos of the playlist
# TODO NOTE: Transcribing is intentionally disabled on playlists. Feel free to tinker but it's a risk.
# TODO: Avoid redundancy with the below branch
if isPlaylist:
self.query_one("#main_log").write("Downloading playlist...")
if toMp3 or toWav:
self.query_one("#main_log").write(
"Converting to mp3 only too (wav is not supported on playlists)...this may take a while"
)
playlist = term.printPlaylist(link)
print(playlist)
self.query_one("#main_log").write(
"Please note that the TUI might seems frozen while downloading the playlist. Check the downloads folder for the progress."
)
self.query_one("#main_log").write("...Yes, we are working on it.")
# TODO More verbosity!
d_path = term.managePlaylist(
playlist, to_download=True, to_convert=toMp3 or toWav
)
status = self.query_one("#status")
status.classes = "status_green"
self.query_one("#main_log").write("Done!")
self.query_one("#main_log").write(str(d_path))
self.lock = False
return
# Branching: in single video mode we download (and if needed convert) the video
self.query_one("#main_log").write(f"MP3: {toMp3}, WAV: {toWav}, TEXT: {toText}")
try:
infos = term.getInfo(link)
except Exception as e:
self.query_one("#main_log").write(
"ERROR: Could not retrieve informations: " + str(e) + ";"
)
return
self.query_one("#video_title").value = "Title: " + infos["title"]
self.query_one("#video_author").value = "Author: " + infos["author"]
self.query_one("#video_length").value = "Seconds: " + infos["length"]
self.query_one("#configuration").value = (
"\n" + f"MP3: {toMp3}, WAV: {toWav}, TEXT: {toText}"
)
# First we download the video
folder = term.download(link)
# Now, if the user wants to convert to mp3, we do it
if toMp3:
self.query_one("#main_log").write(
"Converting to mp3...this may take a while"
)
term.convert(folder, format="mp3")
# Now, if the user wants to convert to wav or text, we do it
if toWav and not toText:
self.query_one("#main_log").write(
"Converting to wav...this may take a while"
)
file = term.convert(folder, format="wav")
# Now, if the user wants to convert to text, we do it
if toText:
self.query_one("#main_log").write(
"Converting to text...this may take a while"
)
engine = self.query_one("#engine").value
file = term.convert(folder, format="wav")
if engine == "google_silence":
self.query_one("#main_log").write("Using Google with silence detection")
file = term.get_large_audio_transcription_on_silence(folder)
elif engine == "local":
self.query_one("#main_log").write("Using Sphynx CMU")
file = term.local_audio_transcribe(folder)
else:
self.query_one("#main_log").write("Using Google Simple")
file = term.simple_audio_transcribe(folder)
status = self.query_one("#status")
status.classes = "status_green"
self.query_one("#main_log").write("Done!")
self.query_one("#main_log").write(str(file))
self.lock = False
return True
# !SECTION Actions
def loadEnv(self):
self.env = {}
with open(".env", "r") as f:
textenv = f.readlines()
for line in textenv:
key, value = line.split("=")
self.env[key.strip()] = value.strip()
def saveEnv(self): # sourcery skip: use-join
preparedEnv = ""
for key, value in self.env.items():
preparedEnv += f"{key}={value}" + "\n"
with open(".env", "w") as f:
f.write(preparedEnv)
f.flush()
return self.env
if __name__ == "__main__":
app = TransTerm()
app.loadEnv()
app.run()

32
install.sh Normal file
View File

@ -0,0 +1,32 @@
#!/bin/bash
pip3 install -r requirements.txt
#pip3 install -r requirements.txt --break-system-packages
echo "Preparing to install PortAudio for Python"
unameOut="$(uname -s)"
case "${unameOut}" in
Linux*) machine=Linux ;;
Darwin*) machine=Mac ;;
CYGWIN*) machine=Cygwin ;;
MINGW*) machine=MinGw ;;
*) machine="UNKNOWN:${unameOut}" ;;
esac
if [[ ${machine} == "Mac" ]]; then
# check if homebrew is installed on Macos if not install Homebrew
echo "Checking Homebrew"
command -v brew >/dev/null 2>&1 || {
echo >&2 "Installing Homebrew Now"
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
}
# if homebrew is installed, update to latest
brew update-reset
# install portaudio
brew install portaudio
pip3 install pyaudio
elif [[ "$(expr substr $(uname -s) 1 5)" == "Linux" ]]; then
sudo apt-get install -y portaudio19-dev
sudo apt-get install python3-pyaudio
fi

39
meshterm.tcss Normal file
View File

@ -0,0 +1,39 @@
Button {
margin: 1 2;
}
.header {
margin: 1 0 0 2;
text-style: bold;
}
Sparkline {
width: 100%;
margin: 2;
color: red;
}
#divisor > .sparkline--max-color {
color: orange;
}
#divisor > .sparkline--min-color {
color: red;
}
.title {
color: red;
}
.status_yellow {
color: yellow
}
.status_red {
color: red
}
.status_green {
color: green
}

8
requirements.txt Normal file
View File

@ -0,0 +1,8 @@
setuptools
wheel
pydub
pyaudio
pocketsphinx
SpeechRecognition
dotenv
textual

208
term.py Normal file
View File

@ -0,0 +1,208 @@
# sourcery skip: use-fstring-for-concatenation
import contextlib
import os
import re
import shutil
import unicodedata
import speech_recognition as sr
from pydub import AudioSegment
from pydub.silence import split_on_silence
from pytube import Playlist, YouTube
forceQuit = False
r = sr.Recognizer()
# NOTE: Taken from https://stackoverflow.com/questions/295135/turn-a-string-into-a-valid-filename
def slugify(value, allow_unicode=False):
"""
Taken from https://github.com/django/django/blob/master/django/utils/text.py
Convert to ASCII if 'allow_unicode' is False. Convert spaces or repeated
dashes to single dashes. Remove characters that aren't alphanumerics,
underscores, or hyphens. Convert to lowercase. Also strip leading and
trailing whitespace, dashes, and underscores.
"""
value = str(value)
if allow_unicode:
value = unicodedata.normalize("NFKC", value)
else:
value = (
unicodedata.normalize("NFKD", value)
.encode("ascii", "ignore")
.decode("ascii")
)
value = re.sub(r"[^\w\s-]", "", value.lower())
return re.sub(r"[-\s]+", "-", value).strip("-_")
def local_audio_transcribe(path):
with sr.AudioFile(path) as source:
audio_listened = r.record(source)
# try converting it to text
r.recognize_sphinx(audio_listened, language="en-US", show_all=False)
# a function to recognize speech in the audio file
# so that we don't repeat ourselves in in other functions
def simple_audio_transcribe(path):
with sr.AudioFile(path) as source:
# listen for the data (load audio to memory)
audio_data = r.record(source)
# recognize (convert from speech to text)
text = r.recognize_google(audio_data)
print(text)
return text
def transcribe_audio(path):
# use the audio file as the audio source
with sr.AudioFile(path) as source:
audio_listened = r.record(source)
# try converting it to text
text = r.recognize_google(audio_listened)
return text
# TODO FIXME a function that splits the audio file into chunks on silence
# and applies speech recognition (mostly unused, should be fixed and tested)
def get_large_audio_transcription_on_silence(path):
"""Splitting the large audio file into chunks
and apply speech recognition on each of these chunks"""
# open the audio file using pydub
sound = AudioSegment.from_file(path)
# split audio sound where silence is 500 miliseconds or more and get chunks
chunks = split_on_silence(
sound,
# experiment with this value for your target audio file
min_silence_len=1000,
# adjust this per requirement
silence_thresh=sound.dBFS - 14,
# keep the silence for 1 second, adjustable as well
keep_silence=500,
)
folder_name = "audio-chunks"
# create a directory to store the audio chunks
if not os.path.isdir(folder_name):
os.mkdir(folder_name)
whole_text = ""
# process each chunk
for i, audio_chunk in enumerate(chunks, start=1):
# export audio chunk and save it in
# the `folder_name` directory.
chunk_filename = os.path.join(folder_name, f"chunk{i}.wav")
audio_chunk.export(chunk_filename, format="wav")
# recognize the chunk
try:
text = transcribe_audio(chunk_filename)
except sr.UnknownValueError as e:
print("Error:", e)
else:
text = f"{text.capitalize()}. "
print(chunk_filename, ":", text)
whole_text += text
# return the text for all chunks detected
return whole_text
# INFO Cleanup the leftovers
def cleanup():
with contextlib.suppress(FileNotFoundError):
os.remove("audio.wav")
with contextlib.suppress(FileNotFoundError):
os.remove("video.mp4")
with contextlib.suppress(FileNotFoundError):
os.remove("audio.mp3")
shutil.rmtree("audio-chunks")
# INFO Main youtube to mp4 downloader method
def download(link, target_filename="video"):
url = YouTube(link)
print("downloading....")
video = url.streams.get_highest_resolution()
print(video.title)
path_to_download_folder = (
str(os.path.dirname(os.path.realpath(__file__))) + "/downloads"
)
video.download(path_to_download_folder, filename=target_filename + ".mp4")
print("Downloaded! :)")
return path_to_download_folder
# INFO Print a playlist info
def printPlaylist(playlist_link):
playlist = Playlist(playlist_link)
return playlist
# INFO Download a playlist
def managePlaylist(playlist, to_download=False, to_convert=False, named=True):
print("Received playlist:", playlist)
if not to_download:
return playlist
counter = 0
path_to_download_folder = (
str(os.path.dirname(os.path.realpath(__file__))) + "/downloads"
)
for url in playlist:
counter += 1
print("Downloading video", counter, "of", len(playlist))
ytvideo = YouTube(url)
video = ytvideo.streams.get_highest_resolution()
filename = "video_" + str(counter)
# Experimental name support
if named:
filename = slugify(ytvideo.title + "_" + ytvideo.author)
print("Downloading : ", filename)
video.download(path_to_download_folder, filename=filename + ".mp4")
if to_convert:
convert(
path_to_download_folder,
format="mp3",
source_filename=filename,
target_filename=filename,
)
print("Downloaded all videos! :)")
return path_to_download_folder
# INFO Get info about the video
def getInfo(link):
print("Getting info for", link)
url = YouTube(link)
# Trying to sanitize the title
title = url.title + "_" + url.author
title = slugify(title)
return {
"title": url.title,
"author": url.author,
"length": str(url.length),
"filename": title,
}
# INFO Convert the video to audio
def convert(
path_to_download_folder,
format="wav",
source_filename="video",
target_filename="audio",
):
src = f"{path_to_download_folder}/{source_filename}.mp4"
dst = f"{path_to_download_folder}/{target_filename}.{format}"
print("Converting to audio....")
sound = AudioSegment.from_file(src, format="mp4")
sound.export(dst, format=format)
print("Converted to audio! :)")
return dst
# Testing (at the moment it is testing with a random playlist)
if __name__ == "__main__":
with contextlib.suppress(FileNotFoundError):
cleanup()
os.remove("transcription.txt")
link = "https://www.youtube.com/playlist?list=PLXaw-xRbZ6E_LHJNJiIu-fOUo4SfpjHqE"
playlist = printPlaylist(link)
managePlaylist(playlist, to_download=True, to_convert=True)