github/StreamingCommunity
1
1
Fork 0
mirror of https://github.com/Arrowar/StreamingCommunity.git synced 2025-06-07 12:05:35 +00:00
Code Issues 2 Packages Projects Releases Wiki Activity

Merge branch 'Ghost6446:main' into main

Browse Source
This commit is contained in:
Francesco Grazioso 2024-05-28 11:03:43 +02:00 committed by GitHub
commit 42ed6509bf
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
15 changed files with 211 additions and 125 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

97
README.md
View File

@ -23,6 +23,7 @@ Make sure you have the following prerequisites installed on your system:
* [python](https://www.python.org/downloads/) > 3.8
* [ffmpeg](https://www.gyan.dev/ffmpeg/builds/)
* [opnessl](https://www.openssl.org) or [pycryptodome](https://pypi.org/project/pycryptodome/)
## Installation
@ -52,38 +53,94 @@ python3 run.py
You can change some behaviors by tweaking the configuration file.
### Options (DEFAULT)
<details>
<summary><strong>DEFAULT</strong></summary>
* root_path: Path where the script will add movies and TV series folders (see [Path Examples](#Path-examples)).
- Default Value: media/streamingcommunity
* **debug**: Enables or disables debug mode.
- **Default Value**: `false`
* not_close: This option, when activated, prevents the script from closing after its initial execution, allowing it to restart automatically after completing the first run.
- Default Value: false
* **log_file**: The file where logs will be written.
- **Default Value**: `app.log`
* map_episode_name: Mapping to choose the name of all episodes of TV Shows (see [Episode Name Usage](#Episode-name-usage)).
- Default Value: %(episode_name)
- Example Value: %(tv_name) [S%(season)] [E%(episode)] %(episode_name)
* **clean_console**: Clears the console before the script runs.
- **Default Value**: `true`
* **root_path**: Path where the script will add movies and TV series folders (see [Path Examples](#Path-examples)).
- **Default Value**: `Video`
### Options (M3U8_DOWNLOAD)
* **map_episode_name**: Mapping to choose the name of all episodes of TV Shows (see [Episode Name Usage](#Episode-name-usage)).
- **Default Value**: `%(tv_name)_S%(season)E%(episode)_%(episode_name)`
* tdqm_workers: The number of workers that will cooperate to download .ts files. **A high value may slow down your PC**
- Default Value: 20
* **not_close**: When activated, prevents the script from closing after its initial execution, allowing it to restart automatically after completing the first run.
- **Default Value**: `false`
* tqdm_show_progress: Whether to show progress during downloads or not.
- Default Value: true
</details>
* create_report: When enabled, this option saves the name of the series or movie being downloaded along with the date and file size in a CSV file, providing a log of downloaded content.
- Default Value: false
<details>
<summary><strong>M3U8_DOWNLOAD</strong></summary>
* **tdqm_workers**: The number of workers that will cooperate to download .ts files. **A high value may slow down your PC**
- **Default Value**: `30`
### Options (M3U8_FILTER)
* **tqdm_show_progress**: Whether to show progress during downloads or not.
- **Default Value**: `true`
* cleanup_tmp_folder: Upon final conversion, this option ensures the removal of all unformatted audio, video tracks, and subtitles from the temporary folder, thereby maintaining cleanliness and efficiency.
- Default Value: true
* **create_report**: When enabled, saves the name of the series or movie being downloaded along with the date and file size in a CSV file, providing a log of downloaded content.
- **Default Value**: `false`
* specific_list_audio: A list of specific audio languages to download.
- Example Value: ['ara', 'baq', 'cat', 'chi', 'cze', 'dan', 'dut', 'eng', 'fil', 'fin', 'forced-ita', 'fre', 'ger', 'glg', 'gre', 'heb', 'hin', 'hun', 'ind', 'ita', 'jpn', 'kan', 'kor', 'mal', 'may', 'nob', 'nor', 'pol', 'por', 'rum', 'rus', 'spa', 'swe', 'tam', 'tel', 'tha', 'tur', 'ukr', 'vie']
</details>
<details>
<summary><strong>M3U8_FILTER</strong></summary>
* **use_codec**: Whether to use a specific codec for processing.
- **Default Value**: `false`
* **use_gpu**: Whether to use GPU acceleration.
- **Default Value**: `false`
* **default_preset**: The default preset for ffmpeg conversion.
- **Default Value**: `ultrafast`
* **check_output_conversion**: Verify if the conversion run by ffmpeg is free from corruption.
- **Default Value**: `false`
* **cleanup_tmp_folder**: Upon final conversion, ensures the removal of all unformatted audio, video tracks, and subtitles from the temporary folder, thereby maintaining cleanliness and efficiency.
- **Default Value**: `true`
* **specific_list_audio**: A list of specific audio languages to download.
- **Example Value**: `['ara', 'baq', 'cat', 'chi', 'cze', 'dan', 'dut', 'eng', 'fil', 'fin', 'forced-ita', 'fre', 'ger', 'glg', 'gre', 'heb', 'hin', 'hun', 'ind', 'ita', 'jpn', 'kan', 'kor', 'mal', 'may', 'nob', 'nor', 'pol', 'por', 'rum', 'rus', 'spa', 'swe', 'tam', 'tel', 'tha', 'tur', 'ukr', 'vie']`
* **specific_list_subtitles**: A list of specific subtitle languages to download.
- **Example Value**: `['eng']`
</details>
<details>
<summary><strong>M3U8_REQUESTS</strong></summary>
* **disable_error**: Whether to disable error messages.
- **Default Value**: `false`
* **timeout**: The timeout value for requests.
- **Default Value**: `10`
* **verify_ssl**: Whether to verify SSL certificates.
- **Default Value**: `false`
</details>
<details>
<summary><strong>M3U8_PARSER</strong></summary>
* **skip_empty_row_playlist**: Whether to skip empty rows in the playlist m3u8.
- **Default Value**: `false`
* **force_resolution**: Forces the use of a specific resolution. `-1` means no forced resolution.
- **Default Value**: `-1`
- **Example Value**: `1080`
</details>
> [!IMPORTANT]

2
Src/Api/Altadefinizione/__init__.py
View File

@ -34,5 +34,3 @@ def main_film():
url=select_title.url
)
# End
console.print("\n[red]Done")

4
Src/Api/Altadefinizione/site.py
View File

@ -7,12 +7,12 @@ import logging
# External libraries
from bs4 import BeautifulSoup
from unidecode import unidecode
# Internal utilities
from Src.Util.table import TVShowManager
from Src.Lib.Request import requests
from Src.Util.headers import get_headers
from Src.Util.console import console
from Src.Util._jsonConfig import config_manager
@ -44,7 +44,7 @@ def title_search(title_search: str) -> int:
"""
# Send request to search for titles
response = requests.get(f"https://{AD_SITE_NAME}.{AD_DOMAIN_NOW}/page/1/?story={title_search.replace(' ', '+')}&do=search&subaction=search&titleonly=3")
response = requests.get(f"https://{AD_SITE_NAME}.{AD_DOMAIN_NOW}/page/1/?story={unidecode(title_search.replace(' ', '+'))}&do=search&subaction=search&titleonly=3")
# Create soup and find table
soup = BeautifulSoup(response.text, "html.parser")

3
Src/Api/Animeunity/site.py
View File

@ -6,6 +6,7 @@ import logging
# External libraries
from bs4 import BeautifulSoup
from unidecode import unidecode
# Internal utilities
@ -160,7 +161,7 @@ def title_search(title: str) -> int:
# Prepare JSON data to be sent in the request
json_data = {
'title': title # Use the provided title for the search
'title': unidecode(title) # Use the provided title for the search
}
# Send a POST request to the API endpoint for live search

3
Src/Api/Streamingcommunity/__init__.py
View File

@ -53,6 +53,3 @@ def main_film_series():
# If no media find
else:
console.print("[red]Cant find a single element")
# End
console.print("\n[red]Done")

3
Src/Api/Streamingcommunity/site.py
View File

@ -9,6 +9,7 @@ from typing import Tuple
# External libraries
from bs4 import BeautifulSoup
from unidecode import unidecode
# Internal utilities
@ -136,7 +137,7 @@ def title_search(title_search: str, domain: str) -> int:
"""
# Send request to search for titles ( replace à to a and space to "+" )
response = requests.get(f"https://{SC_SITE_NAME}.{domain}/api/search?q={title_search.replace(' ', '+')}", headers={'user-agent': get_headers()})
response = requests.get(f"https://{SC_SITE_NAME}.{domain}/api/search?q={unidecode(title_search.replace(' ', '+'))}", headers={'user-agent': get_headers()})
# Add found titles to media search manager
for dict_title in response.json()['data']:

3
Src/Lib/E_Table/__init__.py
View File

@ -1,5 +1,4 @@
# 20.05.24
from .sql_table import SimpleDBManager, report_table, job_database
from .sql_table import SimpleDBManager, report_table
report_table: SimpleDBManager = report_table
job_database: SimpleDBManager = job_database

11
Src/Lib/E_Table/sql_table.py
View File

@ -3,8 +3,6 @@
import csv
import logging
from typing import List
# Internal utilities
from Src.Util._jsonConfig import config_manager
@ -12,7 +10,6 @@ from Src.Util._jsonConfig import config_manager
# Variable
CREATE_REPORT = config_manager.get_bool('M3U8_DOWNLOAD', 'create_report')
CREATE_JOB_DB = config_manager.get_bool('DEFAULT', 'create_job_database')
class SimpleDBManager:
@ -213,11 +210,3 @@ if CREATE_REPORT:
report_table.save_database()
else:
report_table = None
if CREATE_JOB_DB:
job_database = SimpleDBManager("Job_database.csv", ["Id", "Name", "Season_n"])
job_database.load_database()
job_database.save_database()
else:
job_database = None

135
Src/Lib/FFmpeg/capture.py
View File

@ -32,45 +32,59 @@ def capture_output(process: subprocess.Popen, description: str) -> None:
# Variable to store the length of the longest progress string
max_length = 0
for line in iter(process.stdout.readline, b''):
logging.info(f"FFMPEG: {line}")
for line in iter(process.stdout.readline, ''):
try:
line = line.strip()
if not line:
continue
# Check if termination is requested
if terminate_flag.is_set():
break
logging.info(f"FFMPEG line: {line}")
if line is not None and "size=" in str(line).strip():
# Check if termination is requested
if terminate_flag.is_set():
break
# Parse the output line to extract relevant information
data = parse_output_line(str(line).strip())
if "size=" in line:
try:
if 'q' in data:
is_end = (float(data.get('q')) == -1.0)
# Parse the output line to extract relevant information
data = parse_output_line(line)
if not is_end:
byte_size = int(re.findall(r'\d+', data.get('size'))[0]) * 1000
else:
byte_size = int(re.findall(r'\d+', data.get('Lsize'))[0]) * 1000
else:
byte_size = int(re.findall(r'\d+', data.get('size'))[0]) * 1000
if 'q' in data:
is_end = (float(data.get('q', -1.0)) == -1.0)
size_key = 'Lsize' if is_end else 'size'
byte_size = int(re.findall(r'\d+', data.get(size_key, '0'))[0]) * 1000
else:
byte_size = int(re.findall(r'\d+', data.get('size', '0'))[0]) * 1000
time_now = datetime.now().strftime('%H:%M:%S')
time_now = datetime.now().strftime('%H:%M:%S')
# Construct the progress string with formatted output information
progress_string = f"[blue][{time_now}][purple] FFmpeg [white][{description}[white]]: [white]([green]'speed': [yellow]{data.get('speed')}[white], [green]'size': [yellow]{format_size(byte_size)}[white])"
max_length = max(max_length, len(progress_string))
# Construct the progress string with formatted output information
progress_string = (f"[blue][{time_now}][purple] FFmpeg [white][{description}]: "
f"[white]([green]'speed': [yellow]{data.get('speed', 'N/A')}[white], "
f"[green]'size': [yellow]{format_size(byte_size)}[white])")
max_length = max(max_length, len(progress_string))
# Print the progress string to the console, overwriting the previous line
console.print(progress_string.ljust(max_length), end="\r")
# Print the progress string to the console, overwriting the previous line
console.print(progress_string.ljust(max_length), end="\r")
except Exception as e:
logging.error(f"Error parsing output line: {line} - {e}")
except Exception as e:
logging.error(f"Error processing line from subprocess: {e}")
except Exception as e:
logging.error(f"Error in capture_output: {e}")
finally:
terminate_process(process)
try:
terminate_process(process)
except Exception as e:
logging.error(f"Error terminating process: {e}")
def parse_output_line(line: str) -> Tuple[str, str]:
def parse_output_line(line: str) -> dict:
"""
Function to parse the output line and extract relevant information.
@ -79,22 +93,27 @@ def parse_output_line(line: str) -> Tuple[str, str]:
Returns:
dict: A dictionary containing parsed information.
Ex. {'speed': '60.0x'}
"""
data = {}
try:
# Split the line by whitespace and extract key-value pairs
parts = line.replace(" ", "").replace("= ", "=").split()
data = {}
for part in parts:
key_value = part.split('=')
# Split the line by whitespace and extract key-value pairs
parts = line.replace(" ", "").replace("= ", "=").split()
if len(key_value) == 2:
key = key_value[0]
value = key_value[1]
data[key] = value
for part in parts:
key_value = part.split('=')
return data
if len(key_value) == 2:
key = key_value[0]
value = key_value[1]
data[key] = value
return data
except Exception as e:
logging.error(f"Error parsing line: {line} - {e}")
return {}
def terminate_process(process):
@ -104,8 +123,11 @@ def terminate_process(process):
Args:
- process (subprocess.Popen): The subprocess to terminate.
"""
if process.poll() is None: # Check if the process is still running
process.kill()
try:
if process.poll() is None: # Check if the process is still running
process.kill()
except Exception as e:
logging.error(f"Failed to terminate process: {e}")
def capture_ffmpeg_real_time(ffmpeg_command: list, description: str) -> None:
@ -122,21 +144,28 @@ def capture_ffmpeg_real_time(ffmpeg_command: list, description: str) -> None:
# Clear the terminate_flag before starting a new capture
terminate_flag.clear()
# Start the ffmpeg process with subprocess.Popen
process = subprocess.Popen(ffmpeg_command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
# Start a thread to capture and print output
output_thread = threading.Thread(target=capture_output, args=(process,description))
output_thread.start()
try:
# Wait for ffmpeg process to complete
process.wait()
except KeyboardInterrupt:
print("Terminating ffmpeg process...")
except Exception as e:
logging.error(f"Error in ffmpeg process: {e}")
finally:
terminate_flag.set() # Signal the output capture thread to terminate
output_thread.join() # Wait for the output capture thread to complete
# Start the ffmpeg process with subprocess.Popen
process = subprocess.Popen(ffmpeg_command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
# Start a thread to capture and print output
output_thread = threading.Thread(target=capture_output, args=(process, description))
output_thread.start()
try:
# Wait for ffmpeg process to complete
process.wait()
except KeyboardInterrupt:
logging.error("Terminating ffmpeg process...")
except Exception as e:
logging.error(f"Error in ffmpeg process: {e}")
finally:
terminate_flag.set() # Signal the output capture thread to terminate
output_thread.join() # Wait for the output capture thread to complete
except Exception as e:
logging.error(f"Failed to start ffmpeg process: {e}")

8
Src/Lib/FFmpeg/command.py
View File

@ -19,6 +19,7 @@ except: pass
# Internal utilities
from Src.Util._jsonConfig import config_manager
from Src.Util.os import check_file_existence
from Src.Util.console import console
from .util import has_audio_stream, need_to_force_to_ts, check_ffmpeg_input
from .capture import capture_ffmpeg_real_time
@ -274,6 +275,7 @@ def join_video(video_path: str, out_path: str, vcodec: str = None, acodec: str =
"""
if not check_file_existence(video_path):
logging.error("Missing input video for ffmpeg conversion.")
sys.exit(0)
# Start command
@ -284,6 +286,7 @@ def join_video(video_path: str, out_path: str, vcodec: str = None, acodec: str =
# Add mpegts to force to detect input file as ts file
if need_to_force_to_ts(video_path):
console.log("[red]Force input file to 'mpegts'.")
ffmpeg_cmd.extend(['-f', 'mpegts'])
vcodec = "libx264"
@ -317,6 +320,7 @@ def join_video(video_path: str, out_path: str, vcodec: str = None, acodec: str =
# Check file
if CHECK_OUTPUT_CONVERSION:
console.log("[red]Check output ffmpeg")
time.sleep(0.5)
check_ffmpeg_input(out_path)
@ -337,6 +341,7 @@ def join_audios(video_path: str, audio_tracks: List[Dict[str, str]], out_path: s
"""
if not check_file_existence(video_path):
logging.error("Missing input video for ffmpeg conversion.")
sys.exit(0)
# Start command
@ -369,6 +374,7 @@ def join_audios(video_path: str, audio_tracks: List[Dict[str, str]], out_path: s
# Check file
if CHECK_OUTPUT_CONVERSION:
console.log("[red]Check output ffmpeg")
time.sleep(0.5)
check_ffmpeg_input(out_path)
@ -385,6 +391,7 @@ def join_subtitle(video_path: str, subtitles_list: List[Dict[str, str]], out_pat
"""
if not check_file_existence(video_path):
logging.error("Missing input video for ffmpeg conversion.")
sys.exit(0)
@ -426,5 +433,6 @@ def join_subtitle(video_path: str, subtitles_list: List[Dict[str, str]], out_pat
# Check file
if CHECK_OUTPUT_CONVERSION:
console.log("[red]Check output ffmpeg")
time.sleep(0.5)
check_ffmpeg_input(out_path)

14
Src/Lib/Hls/downloader.py
View File

@ -7,6 +7,10 @@ from datetime import datetime
from concurrent.futures import ThreadPoolExecutor
# External library
from unidecode import unidecode
# Internal utilities
from Src.Lib.Request.my_requests import requests
from Src.Util.headers import get_headers
@ -41,10 +45,6 @@ from .segments import M3U8_Segments
from ..E_Table import report_table
# External library
from unidecode import unidecode as transliterate
# Config
DOWNLOAD_SPECIFIC_AUDIO = config_manager.get_list('M3U8_FILTER', 'specific_list_audio')
DOWNLOAD_SPECIFIC_SUBTITLE = config_manager.get_list('M3U8_FILTER', 'specific_list_subtitles')
@ -89,7 +89,7 @@ class Downloader():
sys.exit(0)
self.output_filename = os.path.join(folder, base_name)
self.output_filename = transliterate(self.output_filename)
self.output_filename = unidecode(self.output_filename)
logging.info(f"Output filename: {self.output_filename}")
@ -460,9 +460,11 @@ class Downloader():
# Rename the output file to the desired output filename if not exist
if not os.path.exists(self.output_filename):
# Rename file converted to original set in init
os.rename(out_path, self.output_filename)
print("\n")
# Print size of the file
console.print(Panel(f"[bold green]Download completed![/bold green]\nFile size: [bold]{format_size(os.path.getsize(self.output_filename))}[/bold]", title=f"{os.path.basename(self.output_filename.replace('.mp4', ''))}", border_style="green"))
# Delete all files except the output file

1
Src/Lib/Hls/segments.py
View File

@ -302,6 +302,7 @@ class M3U8_Segments:
ascii=' #',
bar_format=f"{Colors.YELLOW}Downloading {Colors.WHITE}({add_desc}{Colors.WHITE}): {Colors.RED}{{percentage:.2f}}% {Colors.MAGENTA}{{bar}} {Colors.YELLOW}{{elapsed}} {Colors.WHITE}< {Colors.CYAN}{{remaining}}{{postfix}} {Colors.WHITE}]",
dynamic_ncols=True,
ncols=80,
mininterval=0.01
)

34
Src/Upload/update.py
View File

@ -31,29 +31,33 @@ def update():
console.print(f"[red]Error accessing GitHub API: {e}")
return
# Get start of the reposity
stargazers_count = response_reposity['stargazers_count']
# Get stargazers count from the repository
stargazers_count = response_reposity.get('stargazers_count', 0)
# Find info about latest versione deploy and the donwload count
last_version = response_releases[0]['name']
down_count = response_releases[0]['assets'][0]['download_count']
# Calculate total download count from all releases
total_download_count = sum(asset['download_count'] for release in response_releases for asset in release.get('assets', []))
# Calculate percentual of start base on download count
if down_count > 0 and stargazers_count > 0:
percentual_stars = round(stargazers_count / down_count * 100, 2)
# Get latest version name
if response_releases:
last_version = response_releases[0].get('name', 'Unknown')
else:
last_version = 'Unknown'
# Calculate percentual of stars based on download count
if total_download_count > 0 and stargazers_count > 0:
percentual_stars = round(stargazers_count / total_download_count * 100, 2)
else:
percentual_stars = 0
# Check installed version
if __version__ != last_version:
console.print(f"[red]Version: [yellow]{last_version}")
console.print(f"[red]New version available: [yellow]{last_version}")
else:
console.print(f"[red]Everything up to date")
console.print(f"[green]Everything is up to date")
print("\n")
console.print(f"[red]{repo_name} was downloaded [yellow]{down_count} [red]times, but only [yellow]{percentual_stars}% [red]of You(!!) have starred it.\n\
[cyan]Help the repository grow today, by leaving a [yellow]star [cyan]and [yellow]sharing [cyan]it to others online!")
console.print("\n")
console.print(f"[red]{repo_name} has been downloaded [yellow]{total_download_count} [red]times, but only [yellow]{percentual_stars}% [red]of users have starred it.\n\
[cyan]Help the repository grow today by leaving a [yellow]star [cyan]and [yellow]sharing [cyan]it with others online!")
time.sleep(1)
print("\n")
console.print("\n")

15
config.json
View File

@ -3,18 +3,12 @@
"debug": false,
"log_file": "app.log",
"log_to_file": true,
"show_message": false,
"clean_console": false,
"show_message": true,
"clean_console": true,
"root_path": "Video",
"map_episode_name": "%(tv_name)_S%(season)E%(episode)_%(episode_name)",
"create_job_database": false,
"not_close": false
},
"SITE": {
"streamingcommunity": "foo",
"animeunity": "to",
"altadefinizione": "food"
},
"M3U8_DOWNLOAD": {
"tdqm_workers": 30,
"tqdm_show_progress": true,
@ -39,5 +33,10 @@
"M3U8_PARSER": {
"skip_empty_row_playlist": false,
"force_resolution": -1
},
"SITE": {
"streamingcommunity": "foo",
"animeunity": "to",
"altadefinizione": "food"
}
}

1
run.py
View File

@ -18,6 +18,7 @@ from Src.Upload.update import update as git_update
from Src.Lib.FFmpeg import check_ffmpeg
from Src.Util.logger import Logger
# Internal api
from Src.Api.Streamingcommunity import main_film_series as streamingcommunity_film_serie
from Src.Api.Animeunity import main_anime as streamingcommunity_anime