NegroMate
/
songs
2
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Compare commits

base: NegroMate:main
Branches Tags
NegroMate:main
NegroMate:1.5
NegroMate:1.4
NegroMate:1.1
NegroMate:1.0
..
compare: NegroMate:1.1
Branches Tags
NegroMate:main
NegroMate:1.5
NegroMate:1.4
NegroMate:1.1
NegroMate:1.0

No commits in common. "main" and "1.1" have entirely different histories.
main ... 1.1

15 changed files with 134 additions and 553 deletions
Show Stats Expand all files Collapse all files

26
.pre-commit-config.yaml
View File

@ -1,26 +0,0 @@
repos:
# Using this mirror lets us use mypyc-compiled black, which is about 2x faster
- repo: https://github.com/psf/black-pre-commit-mirror
rev: 24.2.0
hooks:
- id: black
# It is recommended to specify the latest version of Python
# supported by your project here, or alternatively use
# pre-commit's default_language_version, see
# https://pre-commit.com/#top_level-default_language_version
language_version: python3.11
- repo: https://github.com/pycqa/isort
rev: 5.13.2
hooks:
- id: isort
name: isort (python)
- repo: https://github.com/astral-sh/ruff-pre-commit
# Ruff version.
rev: v0.3.0
hooks:
# Run the linter.
- id: ruff
# - id: ruff-format We don't need this because we have black and isort.

36
CHANGELOG.md
View File

@ -1,31 +1,5 @@
# Changelog
## 1.5
* Update readme and fix dependencies
* Fix typo
## 1.4
* Update dependencies and code cleaning
## 1.3
* New property for songs: has_subtitles
* Upgrade dependencies
* Add songs without ass file as pending
* Song folders must have a metadata.json file
## 1.2
* Get karaoke ass generator from negromate.web
* Add date to songs
* Convert to namespace package
* Add verbose flag
* Use configuration file for commands defaults
## 1.1
* Start with generic negromate command with entry points for other modules
* Move image generating code to utils
* Migrate setup to pep517
1.1
===
- Start with generic negromate command with entry points for other modules
- Move image generating code to utils
- Migrate setup to pep517

61
README.md
View File

@ -1,58 +1,11 @@
Negro Mate Songs
================
[Negro Mate](https://negromate.rocks)'s song database library. Not very versatile without two another packages:
NegroMate's song database library.
* [negromate.web](https://pypi.org/project/negromate.web/): Static html compiler for the web of Negro Mage
* [negromate.karaoke](https://pypi.org/project/negromate.karaoke/): Simple karaoke video browser
Install from source
-------------------
Instalation uses the pep517 packaging format, whichs requires pip version
19 or newer.
Provides the base `negromate` command to interact with a song repository.
negromate config: Write the configuration file
negromate songs: Update song database
negromate thumbnail: Generate cover and thumbnail for a song
Needs the following packages installed:
* imagemagick
* ffmpeg
Karaoke subtitle generation is currently broken, it worked with these
packages on older systems:
* xdotool
* aegisub
* xserver-xephyr
## Song library structure
Each song is a folder containing at least a video file, a subtitles file and a metadata file:
song_name/
├── song_name.ass
├── song_name.mp4
└── metadata.json
Video and subtitles file has to be named like the folder. Supported subtitles:
* ass (prefered)
* src
* vtt
### The metadata json file
{
"name": "Song name",
"original": "Original song, Original author",
"karaoke": true,
"date": "XXXX-XX-XX"
"author": "Author of the karaoke"
}
* `name`: used as the title of the song.
* `original`: for giving credit to original authors.
* `karaoke`: if `true`, the ass subtitles file has karaoke level subtitles (syllable level timing)
and automatik karaoke subtitles can be generated.
* `date`: approx. creation date of this version
* `author`: not required, used for givin credits to the author when it's different from the web authors.
pip install "pip>=19"
pip install .

0
negromate/__init__.py Normal file
View File

7
negromate/songs/__init__.py
View File

@ -1,11 +1,6 @@
"""
Filesystem based database of songs.
"""
import logging
VERSION = "1.5"
VERSION = "1.1"
logger = logging.getLogger("negromate.songs")

107
negromate/songs/commands/__init__.py
View File

@ -1,66 +1,47 @@
"""
NegroMate command line.
"""
import argparse
import configparser
import logging
import sys
import traceback
from pathlib import Path
try:
from importlib import metadata
except ImportError: # Python <3.8
import importlib_metadata as metadata
CONFIG_FILE = "~/.negromate/config.ini"
def main():
"""
Build parser for all the commands and launch appropiate command.
Build parser for all the commands and launch appropiate command.
Each command must be a module with at least the following members:
Each command must be a module with at least the following members:
* name: String with the command name. Will be used for
argparse subcommand.
* help_text: String with the help text.
* initial_config: Dict for initial configuration of commands.
* options: Function to build the parser of the command. Takes
two parametters, the argparser parser instance for this
subcommand and the ConfigParser instance with all the
configuration.
* run: Function that runs the actual command. Takes two
parametters, the argparse Namespace with the arguments and
the ConfigParser with all the configuration.
* name: String with the command name. Will be used for
argparse subcommand.
* help_text: String with the help text.
* options: Function to build the parser of the command. Takes
one parametter, the argparser parser instance for this
subcommand.
* run: Function that runs the actual command. Takes one
parametter, the Namespace with the arguments.
Minimal module example:
Minimal module example:
# hello_world.py
name = 'hello'
help_text = 'Sample command'
initial_config = {
'who': 'World',
}
# hello_world.py
name = 'hello'
help_text = 'Sample command'
def options(parser, config, **kwargs):
parser.add_argument(
'-w', '--who', default=config['hello']['who'],
help="Who to say hello, defaults to '{}'".format(config['hello']['who'])
)
def options(parser):
parser.add_argument(
'-w', '--who', default='World'
help="Who to say hello, defaults to 'World'"
)
def run(args, **kwargs):
print("Hello {}".format(args.who))
def run(args):
print("Hello {}".format(args.who))
To add more commands to negromate register 'negromate.commands'
entry point in setup.cfg. For example:
To add more commands to negromate register 'negromate.commands'
entry point in setup.cfg. For example:
[options.entry_points]
negromate.commands =
hello = negromate.web.commands.hello_world
[options.entry_points]
negromate.commands =
hello = negromate.web.commands.hello_world
"""
commands = []
@ -69,50 +50,22 @@ def main():
args = sys.argv.copy()
sys.argv = args[:1]
# Load commands from entry_point
entry_points = metadata.entry_points().get("negromate.commands", [])
entry_points = metadata.entry_points().get('negromate.commands', [])
for entry_point in entry_points:
try:
command = entry_point.load()
except Exception as e:
traceback.print_exc()
print(e)
continue
commands.append(command)
commands.append(entry_point.load())
# Load initial configuration for commands
initial_config = {
"global": {
"song_folder": "~/negro_mate/bideoak/",
"lyrics_file": "~/negro_mate/libreto/libreto.pdf",
}
}
for command in commands:
if hasattr(command, "initial_config"):
initial_config[command.name] = command.initial_config
# Load configuration
config = configparser.ConfigParser()
config.read_dict(initial_config)
config.read(Path(CONFIG_FILE).expanduser())
# Build parser
parser = argparse.ArgumentParser()
parser.add_argument("-v", "--verbose", action="store_true", help="Display informational messages.")
parser.set_defaults(command=None)
subparsers = parser.add_subparsers()
for command in commands:
command_parser = subparsers.add_parser(command.name, help=command.help_text)
command_parser.set_defaults(command=command.name)
command.options(parser=command_parser, config=config)
command.options(command_parser)
# Run command
args = parser.parse_args(args[1:])
if args.verbose:
logging.basicConfig(level=logging.DEBUG)
if args.command is None:
parser.print_usage()
else:
for command in commands:
if args.command == command.name:
command.run(args=args, config=config)
command.run(args)

27
negromate/songs/commands/config.py
View File

@ -1,27 +0,0 @@
"""
Generate configuration file.
"""
from pathlib import Path
name = "config"
help_text = "Write the configuration"
initial_config = {
"file": "~/.negromate/config.ini",
}
def options(parser, config, **kwargs):
parser.add_argument(
"-f",
"--file",
type=Path,
default=config["config"]["file"],
help=f"Configuration file, defaults to {config['config']['file']}",
)
def run(args, config, **kwargs):
with args.file.expanduser().open("w") as f:
config.write(f)

74
negromate/songs/commands/songs.py
View File

@ -1,71 +1,35 @@
"""
Load songs and generate missing files.
"""
import sys
from pathlib import Path
from ..loader import load_songs
name = "songs"
help_text = "Update song database"
initial_config = {
"generate": "yes",
"regenerate": "no",
"karaoke_template_file": "~/negro_mate/karaoke_templates/karaoke.ass",
}
name = 'songs'
help_text = 'Update song database'
def options(parser, config, **kwargs):
def options(parser):
parser.add_argument(
"-s",
"--song_folder",
type=Path,
default=config["global"]["song_folder"],
help=f"Folder with the song database, defaults to {config['global']['song_folder']}",
)
parser.add_argument(
"-g",
"--generate",
action="store_const",
const="yes",
default=config["songs"]["generate"],
help=f"Generate missing files, defaults to {config['songs']['generate']}",
)
parser.add_argument(
"-r",
"--regenerate",
action="store_const",
const="yes",
default=config["songs"]["regenerate"],
help=f"Regenerate missing files, defaults to {config['songs']['regenerate']}",
)
parser.add_argument(
"-k",
"--karaoke-template",
type=Path,
default=config["songs"]["karaoke_template_file"],
help=f"Ass file with the karaoke template, defaults to {config['songs']['karaoke_template_file']}",
)
'song_folder', type=Path,
help="Folder with the song database.")
def run(args, **kwargs):
generate = args.generate == "yes"
regenerate = args.regenerate == "yes"
songs, pending_songs = load_songs(
root_folder=args.song_folder.expanduser(),
generate=generate,
regenerate=regenerate,
karaoke_template_file=args.karaoke_template.expanduser(),
def run(args):
songs, pending_songs = load_songs(args.song_folder)
print(
"#######\n"
" Songs\n"
"#######"
)
print("#######\n Songs\n#######")
for s in songs:
print(s.name)
print("###############\n Pending songs\n###############")
print(
"###############\n"
" Pending songs\n"
"###############"
)
for s in pending_songs:
print(s.name)
total_songs = len(songs)
songs_with_karaoke = len(list(s for s in songs if s.metadata["karaoke"]))
songs_with_karaoke = len(list(s for s in songs if s.metadata['karaoke']))
percent = int(songs_with_karaoke / total_songs * 100) if total_songs else 0
print(f"Total songs: {total_songs}. With karaoke: {songs_with_karaoke} ({percent}%)")
print("Total songs: {}. With karaoke: {} ({}%)".format(total_songs, songs_with_karaoke, percent))

24
negromate/songs/commands/thumbnail.py
View File

@ -1,24 +1,22 @@
"""
Generate song images
"""
from pathlib import Path
from negromate.songs.utils import generate_cover, generate_thumbnail
name = "thumbnail"
help_text = "Generate cover and thumbnail for a video."
name = 'thumbnail'
help_text = 'Generate cover and thumbnail for a video.'
def options(parser, **kwargs):
parser.add_argument("video", help="Video of the song.", type=Path)
parser.add_argument("second", type=int, help="Take snapshot at this second.")
def options(parser):
parser.add_argument(
'video', help="Video of the song.", type=Path)
parser.add_argument(
'second', type=int,
help='Take snapshot at this second.')
def run(args, **kwargs):
def run(args):
video = args.video
cover = video.parent / "cover.jpg"
thumbnail = video.parent / "thumb.jpg"
cover = video.parent / 'cover.jpg'
thumbnail = video.parent / 'thumb.jpg'
generate_cover(video, cover, args.second)
generate_thumbnail(cover, thumbnail)

108
negromate/songs/karaoke_templates.py
View File

@ -1,108 +0,0 @@
import os
import subprocess
import time
from contextlib import contextmanager
import ass
@contextmanager
def xephyr_env(display=":2", *args, **kwargs):
env = os.environ.copy()
xephyr = subprocess.Popen(["Xephyr", display], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
env["DISPLAY"] = display
try:
yield env
finally:
xephyr.kill()
def set_template(template_subtitles, orig_file, target_file=None):
if target_file is None:
target_file = orig_file
with open(orig_file, "r") as orig:
subtitles = ass.parse(orig)
new_events = []
for dialogue in template_subtitles.events:
new_events.append(dialogue)
for dialogue in subtitles.events:
if dialogue.effect.startswith("code"):
continue
if dialogue.effect.startswith("template"):
continue
new_events.append(dialogue)
subtitles.events = new_events
with open(target_file, "w", encoding="utf-8-sig") as target:
subtitles.dump_file(target)
def run(command, env, wait=None):
subprocess.Popen(
command,
env=env,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
if wait is not None:
time.sleep(wait)
def apply_template(subtitles, env):
run(["aegisub-3.2", subtitles], env=env, wait=2)
# Si pide confirmación para cargar video ignorar el popup
run(["xdotool", "key", "Escape"], env=env, wait=0.1)
# abrir el menú de automatización, bajar dos y darle a aplicar template
run(["xdotool", "key", "alt+u"], env=env, wait=0.1)
run(["xdotool", "key", "Down"], env=env, wait=0.1)
run(["xdotool", "key", "Down"], env=env, wait=0.1)
run(["xdotool", "key", "Return"], env=env, wait=2)
# guardar
run(["xdotool", "key", "ctrl+s"], env=env)
# cerrar programa
run(["xdotool", "key", "ctrl+q"], env=env)
def update_karaoke_songs(songs, template_file, regenerate=False):
from negromate.songs.utils import needs_change
with open(template_file, "r") as template:
template_subtitles = ass.parse(template)
with xephyr_env() as env:
for song in songs:
if song.metadata.get("karaoke"):
target = song.path / "{}.karaoke.ass".format(song.path.name)
if regenerate or needs_change(target, (song.ass, template_file)):
set_template(
template_subtitles=template_subtitles, orig_file=str(song.ass), target_file=str(target)
)
time.sleep(2)
apply_template(str(target), env)
time.sleep(2)
def generate_karaoke_ass(template_file, orig_file, target_file):
"""
Apply ass template to the subtitle file to render animations.
"""
with open(template_file, "r") as template:
template_subtitles = ass.parse(template)
with xephyr_env() as env:
set_template(
template_subtitles=template_subtitles,
orig_file=orig_file,
target_file=target_file,
)
time.sleep(2)
apply_template(target_file, env)
time.sleep(2)

133
negromate/songs/loader.py
View File

@ -1,28 +1,19 @@
"""
Load songs from the root folder.
"""
import json
import asstosrt
import srt
import webvtt
from .utils import needs_change, generate_cover, generate_thumbnail
from . import logger
from .karaoke_templates import generate_karaoke_ass
from .utils import generate_cover, generate_thumbnail, needs_change
class Song:
"""
Python representation of a song.
"""
def __init__(self, path, root):
self.name = path.name
self.metadata = None
self.original = None
self.author = None
self.date = None
self.path = path
self.root = root
self.video = None
@ -37,139 +28,93 @@ class Song:
self.search_media()
def search_media(self):
"""
Initialize song attributes.
"""
for entry in self.path.iterdir():
if entry.name == "metadata.json":
with entry.open("r") as metadatafile:
if entry.name == 'metadata.json':
with entry.open('r') as metadatafile:
self.metadata = json.load(metadatafile)
self.name = self.metadata.get("name", self.path.name)
self.original = self.metadata.get("original", None)
self.author = self.metadata.get("author", None)
self.date = self.metadata.get("date", None)
elif entry.name.endswith("mp4"):
if 'name' in self.metadata:
self.name = self.metadata['name']
if 'original' in self.metadata:
self.original = self.metadata['original']
if 'author' in self.metadata:
self.author = self.metadata['author']
elif entry.name.endswith('mp4'):
self.video = entry
self.video_type = "video/mp4"
self.video_type = 'video/mp4'
self.files.append(entry)
elif entry.name.endswith("webm"):
elif entry.name.endswith('webm'):
self.video = entry
self.video_type = "video/webm"
self.video_type = 'video/webm'
self.files.append(entry)
elif entry.name.endswith("ogv"):
elif entry.name.endswith('ogv'):
self.video = entry
self.video_type = "video/ogg"
self.video_type = 'video/ogg'
self.files.append(entry)
elif entry.name.endswith("vtt"):
elif entry.name.endswith('vtt'):
self.vtt = entry
elif entry.name == f"{self.path.name}.srt":
elif entry.name == "{}.srt".format(self.path.name):
self.srt = entry
self.files.append(entry)
elif entry.name == f"{self.path.name}.karaoke.ass":
elif entry.name == "{}.karaoke.ass".format(self.path.name):
self.karaoke_ass = entry
self.files.append(entry)
elif entry.name == f"{self.path.name}.ass":
elif entry.name == "{}.ass".format(self.path.name):
self.ass = entry
self.files.append(entry)
elif entry.name == "thumb.jpg":
elif entry.name == 'thumb.jpg':
self.thumbnail = entry
elif entry.name == "cover.jpg":
elif entry.name == 'cover.jpg':
self.cover = entry
elif entry.name == "index.html":
elif entry.name == 'index.html':
continue
else:
self.files.append(entry)
def generate_missing(self, regenerate=False, karaoke_template_file=None):
"""
Generate missing files, if they can be derived from another one.
"""
srt_ = self.path / f"{self.path.name}.srt"
if regenerate or needs_change(srt_, (self.ass,)):
logger.info("generating %s", str(srt_))
self.srt = srt_
with self.ass.open("r") as assfile, self.srt.open("w") as srtfile:
srt = self.path / "{}.srt".format(self.path.name)
if needs_change(srt, (self.ass,)):
self.srt = srt
with self.ass.open('r') as assfile, self.srt.open('w') as srtfile:
srtfile.write(asstosrt.convert(assfile))
self.files.append(self.srt)
vtt = self.path / f"{self.path.name}.vtt"
if regenerate or needs_change(vtt, (self.srt,)):
logger.info("generating %s", str(vtt))
vtt = self.path / "{}.vtt".format(self.path.name)
if needs_change(vtt, (self.srt,)):
self.vtt = vtt
webvtt.from_srt(str(self.srt.absolute())).save(str(self.vtt.absolute()))
cover = self.path / "cover.jpg"
if regenerate or needs_change(cover, (self.video,)):
logger.info("generating %s", str(cover))
if needs_change(cover, (self.video,)):
self.cover = cover
generate_cover(self.video, self.cover)
thumbnail = self.path / "thumb.jpg"
if regenerate or needs_change(thumbnail, (self.cover,)):
logger.info("generating %s", str(thumbnail))
if needs_change(thumbnail, (self.cover,)):
self.thumbnail = thumbnail
generate_thumbnail(self.cover, self.thumbnail)
karaoke_ass = self.path / f"{self.path.name}.karaoke.ass"
karaoke_requirements = (
self.metadata.get("karaoke", False),
regenerate or needs_change(karaoke_ass, (self.ass, karaoke_template_file)),
)
if all(karaoke_requirements):
logger.info("generating %s", str(karaoke_ass))
self.karaoke_ass = karaoke_ass
generate_karaoke_ass(str(karaoke_template_file), str(self.ass), str(karaoke_ass))
@property
def has_subtitles(self):
"""
True if the song has any type of subtitles.
"""
return self.ass or self.srt or self.vtt
@property
def publish(self):
"""
True if the song can be published.
"""
return self.video and self.has_subtitles
@property
def pending(self):
"""
True if the song has a video and ass subtitles.
"""
finished = self.ass and self.video
return not finished
has_subtitles = self.ass or self.srt or self.vtt
has_video = self.video
return has_video and has_subtitles
def load_songs(root_folder, generate=True, regenerate=False, karaoke_template_file=None):
"""
Load songs from root_folder.
If generate is True missing files will be generated.
If regenerate is True, the files will be generated again, even if they source has not changed.
karaoke_template_file can be a path to a ass file with the code to generate subtitle animations.
"""
def load_songs(root_folder):
songs = []
pending_songs = []
for entry in root_folder.iterdir():
if entry.name in ["static", "playlist", "home", "todo"]:
if entry.name in ['static', 'playlist', 'home', 'todo']:
continue
if entry.is_dir() and (entry / "metadata.json").exists():
logger.info("building %s", str(entry.name))
if entry.is_dir():
logger.info("building {}".format(entry.name))
try:
song = Song(entry, root_folder)
if generate:
song.generate_missing(regenerate, karaoke_template_file)
except Exception as e:
logger.error("Error: %s", e)
continue
if song.publish:
songs.append(song)
if song.pending:
else:
pending_songs.append(song)
songs.sort(key=lambda a: a.name)

47
negromate/songs/utils.py
View File

@ -1,19 +1,15 @@
"""
Helper functions.
"""
import subprocess
def needs_change(destination, dependencies):
"""
Checks if the destination file is older than its dependencies.
"""
last_dependency_change = 0
for dependency in dependencies:
if dependency is None:
return False
last_dependency_change = max(last_dependency_change, dependency.lstat().st_mtime)
last_dependency_change = max(
last_dependency_change,
dependency.lstat().st_mtime
)
if not destination.exists():
return True
@ -22,39 +18,26 @@ def needs_change(destination, dependencies):
def generate_cover(video, cover, second=2):
"""
Take a snapshot of the video to create a cover file.
"""
command = [
"ffmpeg",
"-loglevel",
"quiet",
"-i",
str(video.absolute()),
"-vcodec",
"mjpeg",
"-vframes",
"1",
"-an",
"-f",
"rawvideo",
"-ss",
str(second),
"-y",
'ffmpeg',
'-loglevel', 'quiet',
'-i', str(video.absolute()),
'-vcodec', 'mjpeg',
'-vframes', '1',
'-an',
'-f', 'rawvideo',
'-ss', str(second),
'-y',
str(cover.absolute()),
]
subprocess.check_call(command)
def generate_thumbnail(cover, thumbnail, geometry="200x200"):
"""
Generate a reduced image of the cover.
"""
command = [
"convert",
'convert',
str(cover.absolute()),
"-resize",
geometry,
'-resize', geometry,
str(thumbnail.absolute()),
]
subprocess.check_call(command)

20
pyproject.toml
View File

@ -1,23 +1,3 @@
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"
[tool.black]
line_length = 120
[tool.isort]
profile = "black"
lines_after_imports = 2
[tool.pylint.'MESSAGES CONTROL']
max-line-length = 120
disable = "invalid-name, unused-wildcard-import, wildcard-import"
[tool.ruff]
line-length = 120
exclude = [
"build",
]
include = ["negromate/*"]
fix = false
force-exclude = true

5
requirements.txt
View File

@ -1,4 +1,3 @@
ass==0.5.2
webvtt-py==0.4.6
webvtt-py
asstosrt==0.1.6
srt==3.5.3
srt==1.6.0

10
setup.cfg
View File

@ -6,10 +6,10 @@ version = attr: negromate.songs.VERSION
license-file = LICENSE.txt
author = Ales (Shagi) Zabala Alava
author_email = shagi@gisa-elkartea.org
url = https://negromate.rocks
url = http://negromate.rocks
description = NegroMate karaoke song database library
long_description_content_type = text/markdown
long_description = file: README.md, CHANGELOG.md
long_description = files: README.md, CHANGELOG.md
license = GPLv3
classifiers =
Development Status :: 5 - Production/Stable
@ -18,22 +18,20 @@ classifiers =
Topic :: Games/Entertainment
[options]
packages = find_namespace:
packages = find:
zip_safe = true
python_requires = >= 3.4
install_requires =
importlib_metadata
webvtt-py
asstosrt ==0.1.6
srt ==3.5.3
ass ==0.5.2
srt ==1.6.0
[options.entry_points]
console_scripts =
negromate = negromate.songs.commands:main
negromate.commands =
songs = negromate.songs.commands.songs
config = negromate.songs.commands.config
thumbnail = negromate.songs.commands.thumbnail
[bdist_wheel]