Compare commits

...

15 Commits
1.1 ... main

15 changed files with 555 additions and 136 deletions

26
.pre-commit-config.yaml Normal file
View File

@ -0,0 +1,26 @@
repos:
# Using this mirror lets us use mypyc-compiled black, which is about 2x faster
- repo: https://github.com/psf/black-pre-commit-mirror
rev: 24.2.0
hooks:
- id: black
# It is recommended to specify the latest version of Python
# supported by your project here, or alternatively use
# pre-commit's default_language_version, see
# https://pre-commit.com/#top_level-default_language_version
language_version: python3.11
- repo: https://github.com/pycqa/isort
rev: 5.13.2
hooks:
- id: isort
name: isort (python)
- repo: https://github.com/astral-sh/ruff-pre-commit
# Ruff version.
rev: v0.3.0
hooks:
# Run the linter.
- id: ruff
# - id: ruff-format We don't need this because we have black and isort.

View File

@ -1,5 +1,31 @@
1.1
===
- Start with generic negromate command with entry points for other modules
- Move image generating code to utils
- Migrate setup to pep517
# Changelog
## 1.5
* Update readme and fix dependencies
* Fix typo
## 1.4
* Update dependencies and code cleaning
## 1.3
* New property for songs: has_subtitles
* Upgrade dependencies
* Add songs without ass file as pending
* Song folders must have a metadata.json file
## 1.2
* Get karaoke ass generator from negromate.web
* Add date to songs
* Convert to namespace package
* Add verbose flag
* Use configuration file for commands defaults
## 1.1
* Start with generic negromate command with entry points for other modules
* Move image generating code to utils
* Migrate setup to pep517

View File

@ -1,11 +1,58 @@
Negro Mate Songs
================
NegroMate's song database library.
[Negro Mate](https://negromate.rocks)'s song database library. Not very versatile without two another packages:
Install from source
-------------------
Instalation uses the pep517 packaging format, whichs requires pip version
19 or newer.
* [negromate.web](https://pypi.org/project/negromate.web/): Static html compiler for the web of Negro Mage
* [negromate.karaoke](https://pypi.org/project/negromate.karaoke/): Simple karaoke video browser
pip install "pip>=19"
pip install .
Provides the base `negromate` command to interact with a song repository.
negromate config: Write the configuration file
negromate songs: Update song database
negromate thumbnail: Generate cover and thumbnail for a song
Needs the following packages installed:
* imagemagick
* ffmpeg
Karaoke subtitle generation is currently broken, it worked with these
packages on older systems:
* xdotool
* aegisub
* xserver-xephyr
## Song library structure
Each song is a folder containing at least a video file, a subtitles file and a metadata file:
song_name/
├── song_name.ass
├── song_name.mp4
└── metadata.json
Video and subtitles file has to be named like the folder. Supported subtitles:
* ass (prefered)
* src
* vtt
### The metadata json file
{
"name": "Song name",
"original": "Original song, Original author",
"karaoke": true,
"date": "XXXX-XX-XX"
"author": "Author of the karaoke"
}
* `name`: used as the title of the song.
* `original`: for giving credit to original authors.
* `karaoke`: if `true`, the ass subtitles file has karaoke level subtitles (syllable level timing)
and automatik karaoke subtitles can be generated.
* `date`: approx. creation date of this version
* `author`: not required, used for givin credits to the author when it's different from the web authors.

View File

View File

@ -1,6 +1,11 @@
"""
Filesystem based database of songs.
"""
import logging
VERSION = "1.1"
VERSION = "1.5"
logger = logging.getLogger("negromate.songs")

View File

@ -1,47 +1,66 @@
"""
NegroMate command line.
"""
import argparse
import configparser
import logging
import sys
import traceback
from pathlib import Path
try:
from importlib import metadata
except ImportError: # Python <3.8
import importlib_metadata as metadata
CONFIG_FILE = "~/.negromate/config.ini"
def main():
"""
Build parser for all the commands and launch appropiate command.
Build parser for all the commands and launch appropiate command.
Each command must be a module with at least the following members:
Each command must be a module with at least the following members:
* name: String with the command name. Will be used for
argparse subcommand.
* help_text: String with the help text.
* options: Function to build the parser of the command. Takes
one parametter, the argparser parser instance for this
subcommand.
* run: Function that runs the actual command. Takes one
parametter, the Namespace with the arguments.
* name: String with the command name. Will be used for
argparse subcommand.
* help_text: String with the help text.
* initial_config: Dict for initial configuration of commands.
* options: Function to build the parser of the command. Takes
two parametters, the argparser parser instance for this
subcommand and the ConfigParser instance with all the
configuration.
* run: Function that runs the actual command. Takes two
parametters, the argparse Namespace with the arguments and
the ConfigParser with all the configuration.
Minimal module example:
Minimal module example:
# hello_world.py
name = 'hello'
help_text = 'Sample command'
# hello_world.py
name = 'hello'
help_text = 'Sample command'
initial_config = {
'who': 'World',
}
def options(parser):
parser.add_argument(
'-w', '--who', default='World'
help="Who to say hello, defaults to 'World'"
)
def options(parser, config, **kwargs):
parser.add_argument(
'-w', '--who', default=config['hello']['who'],
help="Who to say hello, defaults to '{}'".format(config['hello']['who'])
)
def run(args):
print("Hello {}".format(args.who))
def run(args, **kwargs):
print("Hello {}".format(args.who))
To add more commands to negromate register 'negromate.commands'
entry point in setup.cfg. For example:
To add more commands to negromate register 'negromate.commands'
entry point in setup.cfg. For example:
[options.entry_points]
negromate.commands =
hello = negromate.web.commands.hello_world
[options.entry_points]
negromate.commands =
hello = negromate.web.commands.hello_world
"""
commands = []
@ -50,22 +69,50 @@ def main():
args = sys.argv.copy()
sys.argv = args[:1]
entry_points = metadata.entry_points().get('negromate.commands', [])
# Load commands from entry_point
entry_points = metadata.entry_points().get("negromate.commands", [])
for entry_point in entry_points:
commands.append(entry_point.load())
try:
command = entry_point.load()
except Exception as e:
traceback.print_exc()
print(e)
continue
commands.append(command)
# Load initial configuration for commands
initial_config = {
"global": {
"song_folder": "~/negro_mate/bideoak/",
"lyrics_file": "~/negro_mate/libreto/libreto.pdf",
}
}
for command in commands:
if hasattr(command, "initial_config"):
initial_config[command.name] = command.initial_config
# Load configuration
config = configparser.ConfigParser()
config.read_dict(initial_config)
config.read(Path(CONFIG_FILE).expanduser())
# Build parser
parser = argparse.ArgumentParser()
parser.add_argument("-v", "--verbose", action="store_true", help="Display informational messages.")
parser.set_defaults(command=None)
subparsers = parser.add_subparsers()
for command in commands:
command_parser = subparsers.add_parser(command.name, help=command.help_text)
command_parser.set_defaults(command=command.name)
command.options(command_parser)
command.options(parser=command_parser, config=config)
# Run command
args = parser.parse_args(args[1:])
if args.verbose:
logging.basicConfig(level=logging.DEBUG)
if args.command is None:
parser.print_usage()
else:
for command in commands:
if args.command == command.name:
command.run(args)
command.run(args=args, config=config)

View File

@ -0,0 +1,27 @@
"""
Generate configuration file.
"""
from pathlib import Path
name = "config"
help_text = "Write the configuration"
initial_config = {
"file": "~/.negromate/config.ini",
}
def options(parser, config, **kwargs):
parser.add_argument(
"-f",
"--file",
type=Path,
default=config["config"]["file"],
help=f"Configuration file, defaults to {config['config']['file']}",
)
def run(args, config, **kwargs):
with args.file.expanduser().open("w") as f:
config.write(f)

View File

@ -1,35 +1,71 @@
import sys
"""
Load songs and generate missing files.
"""
from pathlib import Path
from ..loader import load_songs
name = 'songs'
help_text = 'Update song database'
name = "songs"
help_text = "Update song database"
initial_config = {
"generate": "yes",
"regenerate": "no",
"karaoke_template_file": "~/negro_mate/karaoke_templates/karaoke.ass",
}
def options(parser):
def options(parser, config, **kwargs):
parser.add_argument(
'song_folder', type=Path,
help="Folder with the song database.")
def run(args):
songs, pending_songs = load_songs(args.song_folder)
print(
"#######\n"
" Songs\n"
"#######"
"-s",
"--song_folder",
type=Path,
default=config["global"]["song_folder"],
help=f"Folder with the song database, defaults to {config['global']['song_folder']}",
)
parser.add_argument(
"-g",
"--generate",
action="store_const",
const="yes",
default=config["songs"]["generate"],
help=f"Generate missing files, defaults to {config['songs']['generate']}",
)
parser.add_argument(
"-r",
"--regenerate",
action="store_const",
const="yes",
default=config["songs"]["regenerate"],
help=f"Regenerate missing files, defaults to {config['songs']['regenerate']}",
)
parser.add_argument(
"-k",
"--karaoke-template",
type=Path,
default=config["songs"]["karaoke_template_file"],
help=f"Ass file with the karaoke template, defaults to {config['songs']['karaoke_template_file']}",
)
def run(args, **kwargs):
generate = args.generate == "yes"
regenerate = args.regenerate == "yes"
songs, pending_songs = load_songs(
root_folder=args.song_folder.expanduser(),
generate=generate,
regenerate=regenerate,
karaoke_template_file=args.karaoke_template.expanduser(),
)
print("#######\n Songs\n#######")
for s in songs:
print(s.name)
print(
"###############\n"
" Pending songs\n"
"###############"
)
print("###############\n Pending songs\n###############")
for s in pending_songs:
print(s.name)
total_songs = len(songs)
songs_with_karaoke = len(list(s for s in songs if s.metadata['karaoke']))
songs_with_karaoke = len(list(s for s in songs if s.metadata["karaoke"]))
percent = int(songs_with_karaoke / total_songs * 100) if total_songs else 0
print("Total songs: {}. With karaoke: {} ({}%)".format(total_songs, songs_with_karaoke, percent))
print(f"Total songs: {total_songs}. With karaoke: {songs_with_karaoke} ({percent}%)")

View File

@ -1,22 +1,24 @@
"""
Generate song images
"""
from pathlib import Path
from negromate.songs.utils import generate_cover, generate_thumbnail
name = 'thumbnail'
help_text = 'Generate cover and thumbnail for a video.'
name = "thumbnail"
help_text = "Generate cover and thumbnail for a video."
def options(parser):
parser.add_argument(
'video', help="Video of the song.", type=Path)
parser.add_argument(
'second', type=int,
help='Take snapshot at this second.')
def options(parser, **kwargs):
parser.add_argument("video", help="Video of the song.", type=Path)
parser.add_argument("second", type=int, help="Take snapshot at this second.")
def run(args):
def run(args, **kwargs):
video = args.video
cover = video.parent / 'cover.jpg'
thumbnail = video.parent / 'thumb.jpg'
cover = video.parent / "cover.jpg"
thumbnail = video.parent / "thumb.jpg"
generate_cover(video, cover, args.second)
generate_thumbnail(cover, thumbnail)

View File

@ -0,0 +1,108 @@
import os
import subprocess
import time
from contextlib import contextmanager
import ass
@contextmanager
def xephyr_env(display=":2", *args, **kwargs):
env = os.environ.copy()
xephyr = subprocess.Popen(["Xephyr", display], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
env["DISPLAY"] = display
try:
yield env
finally:
xephyr.kill()
def set_template(template_subtitles, orig_file, target_file=None):
if target_file is None:
target_file = orig_file
with open(orig_file, "r") as orig:
subtitles = ass.parse(orig)
new_events = []
for dialogue in template_subtitles.events:
new_events.append(dialogue)
for dialogue in subtitles.events:
if dialogue.effect.startswith("code"):
continue
if dialogue.effect.startswith("template"):
continue
new_events.append(dialogue)
subtitles.events = new_events
with open(target_file, "w", encoding="utf-8-sig") as target:
subtitles.dump_file(target)
def run(command, env, wait=None):
subprocess.Popen(
command,
env=env,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
if wait is not None:
time.sleep(wait)
def apply_template(subtitles, env):
run(["aegisub-3.2", subtitles], env=env, wait=2)
# Si pide confirmación para cargar video ignorar el popup
run(["xdotool", "key", "Escape"], env=env, wait=0.1)
# abrir el menú de automatización, bajar dos y darle a aplicar template
run(["xdotool", "key", "alt+u"], env=env, wait=0.1)
run(["xdotool", "key", "Down"], env=env, wait=0.1)
run(["xdotool", "key", "Down"], env=env, wait=0.1)
run(["xdotool", "key", "Return"], env=env, wait=2)
# guardar
run(["xdotool", "key", "ctrl+s"], env=env)
# cerrar programa
run(["xdotool", "key", "ctrl+q"], env=env)
def update_karaoke_songs(songs, template_file, regenerate=False):
from negromate.songs.utils import needs_change
with open(template_file, "r") as template:
template_subtitles = ass.parse(template)
with xephyr_env() as env:
for song in songs:
if song.metadata.get("karaoke"):
target = song.path / "{}.karaoke.ass".format(song.path.name)
if regenerate or needs_change(target, (song.ass, template_file)):
set_template(
template_subtitles=template_subtitles, orig_file=str(song.ass), target_file=str(target)
)
time.sleep(2)
apply_template(str(target), env)
time.sleep(2)
def generate_karaoke_ass(template_file, orig_file, target_file):
"""
Apply ass template to the subtitle file to render animations.
"""
with open(template_file, "r") as template:
template_subtitles = ass.parse(template)
with xephyr_env() as env:
set_template(
template_subtitles=template_subtitles,
orig_file=orig_file,
target_file=target_file,
)
time.sleep(2)
apply_template(target_file, env)
time.sleep(2)

View File

@ -1,19 +1,28 @@
"""
Load songs from the root folder.
"""
import json
import asstosrt
import srt
import webvtt
from .utils import needs_change, generate_cover, generate_thumbnail
from . import logger
from .karaoke_templates import generate_karaoke_ass
from .utils import generate_cover, generate_thumbnail, needs_change
class Song:
"""
Python representation of a song.
"""
def __init__(self, path, root):
self.name = path.name
self.metadata = None
self.original = None
self.author = None
self.date = None
self.path = path
self.root = root
self.video = None
@ -28,93 +37,139 @@ class Song:
self.search_media()
def search_media(self):
"""
Initialize song attributes.
"""
for entry in self.path.iterdir():
if entry.name == 'metadata.json':
with entry.open('r') as metadatafile:
if entry.name == "metadata.json":
with entry.open("r") as metadatafile:
self.metadata = json.load(metadatafile)
if 'name' in self.metadata:
self.name = self.metadata['name']
if 'original' in self.metadata:
self.original = self.metadata['original']
if 'author' in self.metadata:
self.author = self.metadata['author']
elif entry.name.endswith('mp4'):
self.name = self.metadata.get("name", self.path.name)
self.original = self.metadata.get("original", None)
self.author = self.metadata.get("author", None)
self.date = self.metadata.get("date", None)
elif entry.name.endswith("mp4"):
self.video = entry
self.video_type = 'video/mp4'
self.video_type = "video/mp4"
self.files.append(entry)
elif entry.name.endswith('webm'):
elif entry.name.endswith("webm"):
self.video = entry
self.video_type = 'video/webm'
self.video_type = "video/webm"
self.files.append(entry)
elif entry.name.endswith('ogv'):
elif entry.name.endswith("ogv"):
self.video = entry
self.video_type = 'video/ogg'
self.video_type = "video/ogg"
self.files.append(entry)
elif entry.name.endswith('vtt'):
elif entry.name.endswith("vtt"):
self.vtt = entry
elif entry.name == "{}.srt".format(self.path.name):
elif entry.name == f"{self.path.name}.srt":
self.srt = entry
self.files.append(entry)
elif entry.name == "{}.karaoke.ass".format(self.path.name):
elif entry.name == f"{self.path.name}.karaoke.ass":
self.karaoke_ass = entry
self.files.append(entry)
elif entry.name == "{}.ass".format(self.path.name):
elif entry.name == f"{self.path.name}.ass":
self.ass = entry
self.files.append(entry)
elif entry.name == 'thumb.jpg':
elif entry.name == "thumb.jpg":
self.thumbnail = entry
elif entry.name == 'cover.jpg':
elif entry.name == "cover.jpg":
self.cover = entry
elif entry.name == 'index.html':
elif entry.name == "index.html":
continue
else:
self.files.append(entry)
srt = self.path / "{}.srt".format(self.path.name)
if needs_change(srt, (self.ass,)):
self.srt = srt
with self.ass.open('r') as assfile, self.srt.open('w') as srtfile:
def generate_missing(self, regenerate=False, karaoke_template_file=None):
"""
Generate missing files, if they can be derived from another one.
"""
srt_ = self.path / f"{self.path.name}.srt"
if regenerate or needs_change(srt_, (self.ass,)):
logger.info("generating %s", str(srt_))
self.srt = srt_
with self.ass.open("r") as assfile, self.srt.open("w") as srtfile:
srtfile.write(asstosrt.convert(assfile))
self.files.append(self.srt)
vtt = self.path / "{}.vtt".format(self.path.name)
if needs_change(vtt, (self.srt,)):
vtt = self.path / f"{self.path.name}.vtt"
if regenerate or needs_change(vtt, (self.srt,)):
logger.info("generating %s", str(vtt))
self.vtt = vtt
webvtt.from_srt(str(self.srt.absolute())).save(str(self.vtt.absolute()))
cover = self.path / "cover.jpg"
if needs_change(cover, (self.video,)):
if regenerate or needs_change(cover, (self.video,)):
logger.info("generating %s", str(cover))
self.cover = cover
generate_cover(self.video, self.cover)
generate_cover(self.video, self.cover)
thumbnail = self.path / "thumb.jpg"
if needs_change(thumbnail, (self.cover,)):
if regenerate or needs_change(thumbnail, (self.cover,)):
logger.info("generating %s", str(thumbnail))
self.thumbnail = thumbnail
generate_thumbnail(self.cover, self.thumbnail)
karaoke_ass = self.path / f"{self.path.name}.karaoke.ass"
karaoke_requirements = (
self.metadata.get("karaoke", False),
regenerate or needs_change(karaoke_ass, (self.ass, karaoke_template_file)),
)
if all(karaoke_requirements):
logger.info("generating %s", str(karaoke_ass))
self.karaoke_ass = karaoke_ass
generate_karaoke_ass(str(karaoke_template_file), str(self.ass), str(karaoke_ass))
@property
def has_subtitles(self):
"""
True if the song has any type of subtitles.
"""
return self.ass or self.srt or self.vtt
@property
def publish(self):
has_subtitles = self.ass or self.srt or self.vtt
has_video = self.video
return has_video and has_subtitles
"""
True if the song can be published.
"""
return self.video and self.has_subtitles
@property
def pending(self):
"""
True if the song has a video and ass subtitles.
"""
finished = self.ass and self.video
return not finished
def load_songs(root_folder):
def load_songs(root_folder, generate=True, regenerate=False, karaoke_template_file=None):
"""
Load songs from root_folder.
If generate is True missing files will be generated.
If regenerate is True, the files will be generated again, even if they source has not changed.
karaoke_template_file can be a path to a ass file with the code to generate subtitle animations.
"""
songs = []
pending_songs = []
for entry in root_folder.iterdir():
if entry.name in ['static', 'playlist', 'home', 'todo']:
if entry.name in ["static", "playlist", "home", "todo"]:
continue
if entry.is_dir():
logger.info("building {}".format(entry.name))
if entry.is_dir() and (entry / "metadata.json").exists():
logger.info("building %s", str(entry.name))
try:
song = Song(entry, root_folder)
if generate:
song.generate_missing(regenerate, karaoke_template_file)
except Exception as e:
logger.error("Error: %s", e)
continue
if song.publish:
songs.append(song)
else:
if song.pending:
pending_songs.append(song)
songs.sort(key=lambda a: a.name)

View File

@ -1,15 +1,19 @@
"""
Helper functions.
"""
import subprocess
def needs_change(destination, dependencies):
"""
Checks if the destination file is older than its dependencies.
"""
last_dependency_change = 0
for dependency in dependencies:
if dependency is None:
return False
last_dependency_change = max(
last_dependency_change,
dependency.lstat().st_mtime
)
last_dependency_change = max(last_dependency_change, dependency.lstat().st_mtime)
if not destination.exists():
return True
@ -18,26 +22,39 @@ def needs_change(destination, dependencies):
def generate_cover(video, cover, second=2):
"""
Take a snapshot of the video to create a cover file.
"""
command = [
'ffmpeg',
'-loglevel', 'quiet',
'-i', str(video.absolute()),
'-vcodec', 'mjpeg',
'-vframes', '1',
'-an',
'-f', 'rawvideo',
'-ss', str(second),
'-y',
"ffmpeg",
"-loglevel",
"quiet",
"-i",
str(video.absolute()),
"-vcodec",
"mjpeg",
"-vframes",
"1",
"-an",
"-f",
"rawvideo",
"-ss",
str(second),
"-y",
str(cover.absolute()),
]
subprocess.check_call(command)
def generate_thumbnail(cover, thumbnail, geometry="200x200"):
"""
Generate a reduced image of the cover.
"""
command = [
'convert',
"convert",
str(cover.absolute()),
'-resize', geometry,
"-resize",
geometry,
str(thumbnail.absolute()),
]
subprocess.check_call(command)

View File

@ -1,3 +1,23 @@
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"
[tool.black]
line_length = 120
[tool.isort]
profile = "black"
lines_after_imports = 2
[tool.pylint.'MESSAGES CONTROL']
max-line-length = 120
disable = "invalid-name, unused-wildcard-import, wildcard-import"
[tool.ruff]
line-length = 120
exclude = [
"build",
]
include = ["negromate/*"]
fix = false
force-exclude = true

View File

@ -1,3 +1,4 @@
webvtt-py
ass==0.5.2
webvtt-py==0.4.6
asstosrt==0.1.6
srt==1.6.0
srt==3.5.3

View File

@ -6,10 +6,10 @@ version = attr: negromate.songs.VERSION
license-file = LICENSE.txt
author = Ales (Shagi) Zabala Alava
author_email = shagi@gisa-elkartea.org
url = http://negromate.rocks
url = https://negromate.rocks
description = NegroMate karaoke song database library
long_description_content_type = text/markdown
long_description = files: README.md, CHANGELOG.md
long_description = file: README.md, CHANGELOG.md
license = GPLv3
classifiers =
Development Status :: 5 - Production/Stable
@ -18,20 +18,22 @@ classifiers =
Topic :: Games/Entertainment
[options]
packages = find:
packages = find_namespace:
zip_safe = true
python_requires = >= 3.4
install_requires =
importlib_metadata
webvtt-py
asstosrt ==0.1.6
srt ==1.6.0
srt ==3.5.3
ass ==0.5.2
[options.entry_points]
console_scripts =
negromate = negromate.songs.commands:main
negromate.commands =
songs = negromate.songs.commands.songs
config = negromate.songs.commands.config
thumbnail = negromate.songs.commands.thumbnail
[bdist_wheel]