gronod/headphones
1
0
Fork 0
mirror of https://github.com/rembo10/headphones.git synced 2026-07-22 08:53:59 +01:00
Code Issues Packages Projects Releases Wiki Activity

mutagen, musicbrainzngs, beets lib updates

Browse Source
This commit is contained in:
Ade
2017-03-25 11:47:04 +13:00
parent f3020207c0
commit 4095bf213e
110 changed files with 17022 additions and 6397 deletions
Download Patch File Download Diff File Expand all files Collapse all files
lib/beets/LICENSE
-21
View File
@@ -1,21 +0,0 @@
The MIT License
Copyright (c) 2010-2014 Adrian Sampson
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
lib/beets/README.rst
-94
View File
@@ -1,94 +0,0 @@
.. image:: https://travis-ci.org/sampsyo/beets.svg?branch=master
:target: https://travis-ci.org/sampsyo/beets
.. image:: http://img.shields.io/coveralls/sampsyo/beets.svg
:target: https://coveralls.io/r/sampsyo/beets
.. image:: http://img.shields.io/pypi/v/beets.svg
:target: https://pypi.python.org/pypi/beets
Beets is the media library management system for obsessive-compulsive music
geeks.
The purpose of beets is to get your music collection right once and for all.
It catalogs your collection, automatically improving its metadata as it goes.
It then provides a bouquet of tools for manipulating and accessing your music.
Here's an example of beets' brainy tag corrector doing its thing::
$ beet import ~/music/ladytron
Tagging:
Ladytron - Witching Hour
(Similarity: 98.4%)
* Last One Standing -> The Last One Standing
* Beauty -> Beauty*2
* White Light Generation -> Whitelightgenerator
* All the Way -> All the Way...
Because beets is designed as a library, it can do almost anything you can
imagine for your music collection. Via `plugins`_, beets becomes a panacea:
- Fetch or calculate all the metadata you could possibly need: `album art`_,
`lyrics`_, `genres`_, `tempos`_, `ReplayGain`_ levels, or `acoustic
fingerprints`_.
- Get metadata from `MusicBrainz`_, `Discogs`_, or `Beatport`_. Or guess
metadata using songs' filenames or their acoustic fingerprints.
- `Transcode audio`_ to any format you like.
- Check your library for `duplicate tracks and albums`_ or for `albums that
are missing tracks`_.
- Clean up crufty tags left behind by other, less-awesome tools.
- Embed and extract album art from files' metadata.
- Browse your music library graphically through a Web browser and play it in any
browser that supports `HTML5 Audio`_.
- Analyze music files' metadata from the command line.
- Listen to your library with a music player that speaks the `MPD`_ protocol
and works with a staggering variety of interfaces.
If beets doesn't do what you want yet, `writing your own plugin`_ is
shockingly simple if you know a little Python.
.. _plugins: http://beets.readthedocs.org/page/plugins/
.. _MPD: http://www.musicpd.org/
.. _MusicBrainz music collection: http://musicbrainz.org/doc/Collections/
.. _writing your own plugin:
http://beets.readthedocs.org/page/dev/plugins.html
.. _HTML5 Audio:
http://www.w3.org/TR/html-markup/audio.html
.. _albums that are missing tracks:
http://beets.readthedocs.org/page/plugins/missing.html
.. _duplicate tracks and albums:
http://beets.readthedocs.org/page/plugins/duplicates.html
.. _Transcode audio:
http://beets.readthedocs.org/page/plugins/convert.html
.. _Beatport: http://www.beatport.com/
.. _Discogs: http://www.discogs.com/
.. _acoustic fingerprints:
http://beets.readthedocs.org/page/plugins/chroma.html
.. _ReplayGain: http://beets.readthedocs.org/page/plugins/replaygain.html
.. _tempos: http://beets.readthedocs.org/page/plugins/echonest.html
.. _genres: http://beets.readthedocs.org/page/plugins/lastgenre.html
.. _album art: http://beets.readthedocs.org/page/plugins/fetchart.html
.. _lyrics: http://beets.readthedocs.org/page/plugins/lyrics.html
.. _MusicBrainz: http://musicbrainz.org/
Read More
---------
Learn more about beets at `its Web site`_. Follow `@b33ts`_ on Twitter for
news and updates.
You can install beets by typing ``pip install beets``. Then check out the
`Getting Started`_ guide.
.. _its Web site: http://beets.radbox.org/
.. _Getting Started: http://beets.readthedocs.org/page/guides/main.html
.. _@b33ts: http://twitter.com/b33ts/
Authors
-------
Beets is by `Adrian Sampson`_ with a supporting cast of thousands. For help,
please contact the `mailing list`_.
.. _mailing list: https://groups.google.com/forum/#!forum/beets-users
.. _Adrian Sampson: http://homes.cs.washington.edu/~asampson/
lib/beets/__init__.py
Regular → Executable
+26 -9
View File
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2014, Adrian Sampson.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -12,17 +13,33 @@
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
# This particular version has been slightly modified to work with Headphones
# https://github.com/rembo10/headphones
__version__ = '1.3.10-headphones'
__author__ = 'Adrian Sampson <adrian@radbox.org>'
from __future__ import division, absolute_import, print_function
import os
import beets.library
from beets.util import confit
Library = beets.library.Library
# This particular version has been slightly modified to work with Headphones
# https://github.com/rembo10/headphones
__version__ = u'1.4.4-headphones'
__author__ = u'Adrian Sampson <adrian@radbox.org>'
config = confit.LazyConfig(os.path.dirname(__file__), __name__)
class IncludeLazyConfig(confit.LazyConfig):
"""A version of Confit's LazyConfig that also merges in data from
YAML files specified in an `include` setting.
"""
def read(self, user=True, defaults=True):
super(IncludeLazyConfig, self).read(user, defaults)
try:
for view in self['include']:
filename = view.as_filename()
if os.path.isfile(filename):
self.set_file(filename)
except confit.NotFoundError:
pass
# headphones
#config = IncludeLazyConfig('beets', __name__)
config = IncludeLazyConfig(os.path.dirname(__file__), __name__)
lib/beets/__main__.py
Executable
+26
View File
@@ -0,0 +1,26 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2017, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
"""The __main__ module lets you run the beets CLI interface by typing
`python -m beets`.
"""
from __future__ import division, absolute_import, print_function
import sys
from .ui import main
if __name__ == "__main__":
main(sys.argv[1:])
lib/beets/art.py
Executable
+222
View File
@@ -0,0 +1,222 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
"""High-level utilities for manipulating image files associated with
music and items' embedded album art.
"""
from __future__ import division, absolute_import, print_function
import subprocess
import platform
from tempfile import NamedTemporaryFile
import os
from beets.util import displayable_path, syspath, bytestring_path
from beets.util.artresizer import ArtResizer
from beets import mediafile
def mediafile_image(image_path, maxwidth=None):
"""Return a `mediafile.Image` object for the path.
"""
with open(syspath(image_path), 'rb') as f:
data = f.read()
return mediafile.Image(data, type=mediafile.ImageType.front)
def get_art(log, item):
# Extract the art.
try:
mf = mediafile.MediaFile(syspath(item.path))
except mediafile.UnreadableFileError as exc:
log.warning(u'Could not extract art from {0}: {1}',
displayable_path(item.path), exc)
return
return mf.art
def embed_item(log, item, imagepath, maxwidth=None, itempath=None,
compare_threshold=0, ifempty=False, as_album=False):
"""Embed an image into the item's media file.
"""
# Conditions and filters.
if compare_threshold:
if not check_art_similarity(log, item, imagepath, compare_threshold):
log.info(u'Image not similar; skipping.')
return
if ifempty and get_art(log, item):
log.info(u'media file already contained art')
return
if maxwidth and not as_album:
imagepath = resize_image(log, imagepath, maxwidth)
# Get the `Image` object from the file.
try:
log.debug(u'embedding {0}', displayable_path(imagepath))
image = mediafile_image(imagepath, maxwidth)
except IOError as exc:
log.warning(u'could not read image file: {0}', exc)
return
# Make sure the image kind is safe (some formats only support PNG
# and JPEG).
if image.mime_type not in ('image/jpeg', 'image/png'):
log.info('not embedding image of unsupported type: {}',
image.mime_type)
return
item.try_write(path=itempath, tags={'images': [image]})
def embed_album(log, album, maxwidth=None, quiet=False,
compare_threshold=0, ifempty=False):
"""Embed album art into all of the album's items.
"""
imagepath = album.artpath
if not imagepath:
log.info(u'No album art present for {0}', album)
return
if not os.path.isfile(syspath(imagepath)):
log.info(u'Album art not found at {0} for {1}',
displayable_path(imagepath), album)
return
if maxwidth:
imagepath = resize_image(log, imagepath, maxwidth)
log.info(u'Embedding album art into {0}', album)
for item in album.items():
embed_item(log, item, imagepath, maxwidth, None,
compare_threshold, ifempty, as_album=True)
def resize_image(log, imagepath, maxwidth):
"""Returns path to an image resized to maxwidth.
"""
log.debug(u'Resizing album art to {0} pixels wide', maxwidth)
imagepath = ArtResizer.shared.resize(maxwidth, syspath(imagepath))
return imagepath
def check_art_similarity(log, item, imagepath, compare_threshold):
"""A boolean indicating if an image is similar to embedded item art.
"""
with NamedTemporaryFile(delete=True) as f:
art = extract(log, f.name, item)
if art:
is_windows = platform.system() == "Windows"
# Converting images to grayscale tends to minimize the weight
# of colors in the diff score. So we first convert both images
# to grayscale and then pipe them into the `compare` command.
# On Windows, ImageMagick doesn't support the magic \\?\ prefix
# on paths, so we pass `prefix=False` to `syspath`.
convert_cmd = ['convert', syspath(imagepath, prefix=False),
syspath(art, prefix=False),
'-colorspace', 'gray', 'MIFF:-']
compare_cmd = ['compare', '-metric', 'PHASH', '-', 'null:']
log.debug(u'comparing images with pipeline {} | {}',
convert_cmd, compare_cmd)
convert_proc = subprocess.Popen(
convert_cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
close_fds=not is_windows,
)
compare_proc = subprocess.Popen(
compare_cmd,
stdin=convert_proc.stdout,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
close_fds=not is_windows,
)
# Check the convert output. We're not interested in the
# standard output; that gets piped to the next stage.
convert_proc.stdout.close()
convert_stderr = convert_proc.stderr.read()
convert_proc.stderr.close()
convert_proc.wait()
if convert_proc.returncode:
log.debug(
u'ImageMagick convert failed with status {}: {!r}',
convert_proc.returncode,
convert_stderr,
)
return
# Check the compare output.
stdout, stderr = compare_proc.communicate()
if compare_proc.returncode:
if compare_proc.returncode != 1:
log.debug(u'ImageMagick compare failed: {0}, {1}',
displayable_path(imagepath),
displayable_path(art))
return
out_str = stderr
else:
out_str = stdout
try:
phash_diff = float(out_str)
except ValueError:
log.debug(u'IM output is not a number: {0!r}', out_str)
return
log.debug(u'ImageMagick compare score: {0}', phash_diff)
return phash_diff <= compare_threshold
return True
def extract(log, outpath, item):
art = get_art(log, item)
outpath = bytestring_path(outpath)
if not art:
log.info(u'No album art present in {0}, skipping.', item)
return
# Add an extension to the filename.
ext = mediafile.image_extension(art)
if not ext:
log.warning(u'Unknown image type in {0}.',
displayable_path(item.path))
return
outpath += bytestring_path('.' + ext)
log.info(u'Extracting album art from: {0} to: {1}',
item, displayable_path(outpath))
with open(syspath(outpath), 'wb') as f:
f.write(art)
return outpath
def extract_first(log, outpath, items):
for item in items:
real_path = extract(log, outpath, item)
if real_path:
return real_path
def clear(log, lib, query):
items = lib.items(query)
log.info(u'Clearing album art from {0} items', len(items))
for item in items:
log.debug(u'Clearing art for {0}', item)
item.try_write(tags={'images': None})
lib/beets/autotag/__init__.py
Regular → Executable
+35 -7
View File
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2013, Adrian Sampson.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -14,13 +15,15 @@
"""Facilities for automatically determining files' correct metadata.
"""
import logging
from __future__ import division, absolute_import, print_function
from beets import logging
from beets import config
# Parts of external interface.
from .hooks import AlbumInfo, TrackInfo, AlbumMatch, TrackMatch # noqa
from .match import tag_item, tag_album # noqa
from .match import tag_item, tag_album, Proposal # noqa
from .match import Recommendation # noqa
# Global logger.
@@ -39,6 +42,16 @@ def apply_item_metadata(item, track_info):
item.mb_trackid = track_info.track_id
if track_info.artist_id:
item.mb_artistid = track_info.artist_id
if track_info.data_source:
item.data_source = track_info.data_source
if track_info.lyricist is not None:
item.lyricist = track_info.lyricist
if track_info.composer is not None:
item.composer = track_info.composer
if track_info.arranger is not None:
item.arranger = track_info.arranger
# At the moment, the other metadata is left intact (including album
# and track number). Perhaps these should be emptied?
@@ -47,7 +60,7 @@ def apply_metadata(album_info, mapping):
"""Set the items' metadata to match an AlbumInfo object using a
mapping from Items to TrackInfo objects.
"""
for item, track_info in mapping.iteritems():
for item, track_info in mapping.items():
# Album, artist, track count.
if track_info.artist:
item.artist = track_info.artist
@@ -90,7 +103,12 @@ def apply_metadata(album_info, mapping):
item.title = track_info.title
if config['per_disc_numbering']:
item.track = track_info.medium_index or track_info.index
# We want to let the track number be zero, but if the medium index
# is not provided we need to fall back to the overall index.
if track_info.medium_index is not None:
item.track = track_info.medium_index
else:
item.track = track_info.index
item.tracktotal = track_info.medium_total or len(album_info.tracks)
else:
item.track = track_info.index
@@ -122,7 +140,8 @@ def apply_metadata(album_info, mapping):
'language',
'country',
'albumstatus',
'albumdisambig'):
'albumdisambig',
'data_source',):
value = getattr(album_info, field)
if value is not None:
item[field] = value
@@ -132,5 +151,14 @@ def apply_metadata(album_info, mapping):
if track_info.media is not None:
item.media = track_info.media
if track_info.lyricist is not None:
item.lyricist = track_info.lyricist
if track_info.composer is not None:
item.composer = track_info.composer
if track_info.arranger is not None:
item.arranger = track_info.arranger
item.track_alt = track_info.track_alt
# Headphones seal of approval
item.comments = 'tagged by headphones/beets'
item.comments = 'tagged by headphones/beets'
lib/beets/autotag/hooks.py
Regular → Executable
+79 -38
View File
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2013, Adrian Sampson.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -13,15 +14,20 @@
# included in all copies or substantial portions of the Software.
"""Glue between metadata sources and the matching logic."""
import logging
from __future__ import division, absolute_import, print_function
from collections import namedtuple
from functools import total_ordering
import re
from beets import logging
from beets import plugins
from beets import config
from beets.util import as_string
from beets.autotag import mb
from beets.util import levenshtein
from jellyfish import levenshtein_distance
from unidecode import unidecode
import six
log = logging.getLogger('beets')
@@ -101,7 +107,7 @@ class AlbumInfo(object):
# Work around a bug in python-musicbrainz-ngs that causes some
# strings to be bytes rather than Unicode.
# https://github.com/alastair/python-musicbrainz-ngs/issues/85
def decode(self, codec='utf8'):
def decode(self, codec='utf-8'):
"""Ensure that all string attributes on this object, and the
constituent `TrackInfo` objects, are decoded to Unicode.
"""
@@ -109,7 +115,7 @@ class AlbumInfo(object):
'catalognum', 'script', 'language', 'country',
'albumstatus', 'albumdisambig', 'artist_credit', 'media']:
value = getattr(self, fld)
if isinstance(value, str):
if isinstance(value, bytes):
setattr(self, fld, value.decode(codec, 'ignore'))
if self.tracks:
@@ -134,6 +140,12 @@ class TrackInfo(object):
- ``artist_sort``: name of the track artist for sorting
- ``disctitle``: name of the individual medium (subtitle)
- ``artist_credit``: Recording-specific artist name
- ``data_source``: The original data source (MusicBrainz, Discogs, etc.)
- ``data_url``: The data source release URL.
- ``lyricist``: individual track lyricist name
- ``composer``: individual track composer name
- ``arranger`: individual track arranger name
- ``track_alt``: alternative track number (tape, vinyl, etc.)
Only ``title`` and ``track_id`` are required. The rest of the fields
may be None. The indices ``index``, ``medium``, and ``medium_index``
@@ -143,7 +155,8 @@ class TrackInfo(object):
length=None, index=None, medium=None, medium_index=None,
medium_total=None, artist_sort=None, disctitle=None,
artist_credit=None, data_source=None, data_url=None,
media=None):
media=None, lyricist=None, composer=None, arranger=None,
track_alt=None):
self.title = title
self.track_id = track_id
self.artist = artist
@@ -159,16 +172,20 @@ class TrackInfo(object):
self.artist_credit = artist_credit
self.data_source = data_source
self.data_url = data_url
self.lyricist = lyricist
self.composer = composer
self.arranger = arranger
self.track_alt = track_alt
# As above, work around a bug in python-musicbrainz-ngs.
def decode(self, codec='utf8'):
def decode(self, codec='utf-8'):
"""Ensure that all string attributes on this object are decoded
to Unicode.
"""
for fld in ['title', 'artist', 'medium', 'artist_sort', 'disctitle',
'artist_credit', 'media']:
value = getattr(self, fld)
if isinstance(value, str):
if isinstance(value, bytes):
setattr(self, fld, value.decode(codec, 'ignore'))
@@ -198,13 +215,15 @@ def _string_dist_basic(str1, str2):
transliteration/lowering to ASCII characters. Normalized by string
length.
"""
str1 = unidecode(str1)
str2 = unidecode(str2)
assert isinstance(str1, six.text_type)
assert isinstance(str2, six.text_type)
str1 = as_string(unidecode(str1))
str2 = as_string(unidecode(str2))
str1 = re.sub(r'[^a-z0-9]', '', str1.lower())
str2 = re.sub(r'[^a-z0-9]', '', str2.lower())
if not str1 and not str2:
return 0.0
return levenshtein(str1, str2) / float(max(len(str1), len(str2)))
return levenshtein_distance(str1, str2) / float(max(len(str1), len(str2)))
def string_dist(str1, str2):
@@ -281,6 +300,8 @@ class LazyClassProperty(object):
return self.value
@total_ordering
@six.python_2_unicode_compatible
class Distance(object):
"""Keeps track of multiple distance penalties. Provides a single
weighted distance for all penalties as well as a weighted distance
@@ -290,7 +311,7 @@ class Distance(object):
self._penalties = {}
@LazyClassProperty
def _weights(cls):
def _weights(cls): # noqa
"""A dictionary from keys to floating-point weights.
"""
weights_view = config['match']['distance_weights']
@@ -316,7 +337,7 @@ class Distance(object):
"""Return the maximum distance penalty (normalization factor).
"""
dist_max = 0.0
for key, penalty in self._penalties.iteritems():
for key, penalty in self._penalties.items():
dist_max += len(penalty) * self._weights[key]
return dist_max
@@ -325,7 +346,7 @@ class Distance(object):
"""Return the raw (denormalized) distance.
"""
dist_raw = 0.0
for key, penalty in self._penalties.iteritems():
for key, penalty in self._penalties.items():
dist_raw += sum(penalty) * self._weights[key]
return dist_raw
@@ -342,12 +363,21 @@ class Distance(object):
# Convert distance into a negative float we can sort items in
# ascending order (for keys, when the penalty is equal) and
# still get the items with the biggest distance first.
return sorted(list_, key=lambda (key, dist): (0 - dist, key))
return sorted(
list_,
key=lambda key_and_dist: (-key_and_dist[1], key_and_dist[0])
)
def __hash__(self):
return id(self)
def __eq__(self, other):
return self.distance == other
# Behave like a float.
def __cmp__(self, other):
return cmp(self.distance, other)
def __lt__(self, other):
return self.distance < other
def __float__(self):
return self.distance
@@ -358,6 +388,9 @@ class Distance(object):
def __rsub__(self, other):
return other - self.distance
def __str__(self):
return "{0:.2f}".format(self.distance)
# Behave like a dict.
def __getitem__(self, key):
@@ -383,9 +416,9 @@ class Distance(object):
"""
if not isinstance(dist, Distance):
raise ValueError(
'`dist` must be a Distance object, not {0}'.format(type(dist))
u'`dist` must be a Distance object, not {0}'.format(type(dist))
)
for key, penalties in dist._penalties.iteritems():
for key, penalties in dist._penalties.items():
self._penalties.setdefault(key, []).extend(penalties)
# Adding components.
@@ -407,7 +440,7 @@ class Distance(object):
"""
if not 0.0 <= dist <= 1.0:
raise ValueError(
'`dist` must be between 0.0 and 1.0, not {0}'.format(dist)
u'`dist` must be between 0.0 and 1.0, not {0}'.format(dist)
)
self._penalties.setdefault(key, []).append(dist)
@@ -516,20 +549,29 @@ def track_for_mbid(recording_id):
exc.log(log)
@plugins.notify_info_yielded(u'albuminfo_received')
def albums_for_id(album_id):
"""Get a list of albums for an ID."""
candidates = [album_for_mbid(album_id)]
candidates.extend(plugins.album_for_id(album_id))
return filter(None, candidates)
a = album_for_mbid(album_id)
if a:
yield a
for a in plugins.album_for_id(album_id):
if a:
yield a
@plugins.notify_info_yielded(u'trackinfo_received')
def tracks_for_id(track_id):
"""Get a list of tracks for an ID."""
candidates = [track_for_mbid(track_id)]
candidates.extend(plugins.track_for_id(track_id))
return filter(None, candidates)
t = track_for_mbid(track_id)
if t:
yield t
for t in plugins.track_for_id(track_id):
if t:
yield t
@plugins.notify_info_yielded(u'albuminfo_received')
def album_candidates(items, artist, album, va_likely):
"""Search for album matches. ``items`` is a list of Item objects
that make up the album. ``artist`` and ``album`` are the respective
@@ -537,43 +579,42 @@ def album_candidates(items, artist, album, va_likely):
entered by the user. ``va_likely`` is a boolean indicating whether
the album is likely to be a "various artists" release.
"""
out = []
# Base candidates if we have album and artist to match.
if artist and album:
try:
out.extend(mb.match_album(artist, album, len(items)))
for candidate in mb.match_album(artist, album, len(items)):
yield candidate
except mb.MusicBrainzAPIError as exc:
exc.log(log)
# Also add VA matches from MusicBrainz where appropriate.
if va_likely and album:
try:
out.extend(mb.match_album(None, album, len(items)))
for candidate in mb.match_album(None, album, len(items)):
yield candidate
except mb.MusicBrainzAPIError as exc:
exc.log(log)
# Candidates from plugins.
out.extend(plugins.candidates(items, artist, album, va_likely))
return out
for candidate in plugins.candidates(items, artist, album, va_likely):
yield candidate
@plugins.notify_info_yielded(u'trackinfo_received')
def item_candidates(item, artist, title):
"""Search for item matches. ``item`` is the Item to be matched.
``artist`` and ``title`` are strings and either reflect the item or
are specified by the user.
"""
out = []
# MusicBrainz candidates.
if artist and title:
try:
out.extend(mb.match_track(artist, title))
for candidate in mb.match_track(artist, title):
yield candidate
except mb.MusicBrainzAPIError as exc:
exc.log(log)
# Plugin candidates.
out.extend(plugins.item_candidates(item, artist, title))
return out
for candidate in plugins.item_candidates(item, artist, title):
yield candidate
lib/beets/autotag/match.py
Regular → Executable
+101 -72
View File
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2013, Adrian Sampson.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -15,13 +16,15 @@
"""Matches existing metadata with canonical information to identify
releases and tracks.
"""
from __future__ import division
from __future__ import division, absolute_import, print_function
import datetime
import logging
import re
from munkres import Munkres
from collections import namedtuple
from beets import logging
from beets import plugins
from beets import config
from beets.util import plurality
@@ -50,6 +53,13 @@ class Recommendation(OrderedEnum):
strong = 3
# A structure for holding a set of possible matches to choose between. This
# consists of a list of possible candidates (i.e., AlbumInfo or TrackInfo
# objects) and a recommendation value.
Proposal = namedtuple('Proposal', ('candidates', 'recommendation'))
# Primary matching functionality.
def current_metadata(items):
@@ -93,7 +103,9 @@ def assign_items(items, tracks):
costs.append(row)
# Find a minimum-cost bipartite matching.
log.debug('Computing track assignment...')
matching = Munkres().compute(costs)
log.debug('...done.')
# Produce the output matching.
mapping = dict((items[i], tracks[j]) for (i, j) in matching)
@@ -235,7 +247,7 @@ def distance(items, album_info, mapping):
# Tracks.
dist.tracks = {}
for item, track in mapping.iteritems():
for item, track in mapping.items():
dist.tracks[track] = track_distance(item, track, album_info.va)
dist.add('tracks', dist.tracks[track].distance)
@@ -258,19 +270,23 @@ def match_by_id(items):
AlbumInfo object for the corresponding album. Otherwise, returns
None.
"""
# Is there a consensus on the MB album ID?
albumids = [item.mb_albumid for item in items if item.mb_albumid]
if not albumids:
log.debug(u'No album IDs found.')
albumids = (item.mb_albumid for item in items if item.mb_albumid)
# Did any of the items have an MB album ID?
try:
first = next(albumids)
except StopIteration:
log.debug(u'No album ID found.')
return None
# Is there a consensus on the MB album ID?
for other in albumids:
if other != first:
log.debug(u'No album ID consensus.')
return None
# If all album IDs are equal, look up the album.
if bool(reduce(lambda x, y: x if x == y else (), albumids)):
albumid = albumids[0]
log.debug(u'Searching for discovered album ID: {0}'.format(albumid))
return hooks.album_for_mbid(albumid)
else:
log.debug(u'No album ID consensus.')
log.debug(u'Searching for discovered album ID: {0}', first)
return hooks.album_for_mbid(first)
def _recommendation(results):
@@ -309,10 +325,10 @@ def _recommendation(results):
keys = set(min_dist.keys())
if isinstance(results[0], hooks.AlbumMatch):
for track_dist in min_dist.tracks.values():
keys.update(track_dist.keys())
keys.update(list(track_dist.keys()))
max_rec_view = config['match']['max_rec']
for key in keys:
if key in max_rec_view.keys():
if key in list(max_rec_view.keys()):
max_rec = max_rec_view[key].as_choice({
'strong': Recommendation.strong,
'medium': Recommendation.medium,
@@ -324,17 +340,23 @@ def _recommendation(results):
return rec
def _sort_candidates(candidates):
"""Sort candidates by distance."""
return sorted(candidates, key=lambda match: match.distance)
def _add_candidate(items, results, info):
"""Given a candidate AlbumInfo object, attempt to add the candidate
to the output dictionary of AlbumMatch objects. This involves
checking the track count, ordering the items, checking for
duplicates, and calculating the distance.
"""
log.debug(u'Candidate: {0} - {1}'.format(info.artist, info.album))
log.debug(u'Candidate: {0} - {1} ({2})',
info.artist, info.album, info.album_id)
# Discard albums with zero tracks.
if not info.tracks:
log.debug('No tracks.')
log.debug(u'No tracks.')
return
# Don't duplicate.
@@ -345,7 +367,7 @@ def _add_candidate(items, results, info):
# Discard matches without required tags.
for req_tag in config['match']['required'].as_str_seq():
if getattr(info, req_tag) is None:
log.debug(u'Ignored. Missing required tag: {0}'.format(req_tag))
log.debug(u'Ignored. Missing required tag: {0}', req_tag)
return
# Find mapping between the items and the track info.
@@ -355,48 +377,52 @@ def _add_candidate(items, results, info):
dist = distance(items, info, mapping)
# Skip matches with ignored penalties.
penalties = [key for _, key in dist]
penalties = [key for key, _ in dist]
for penalty in config['match']['ignored'].as_str_seq():
if penalty in penalties:
log.debug(u'Ignored. Penalty: {0}'.format(penalty))
log.debug(u'Ignored. Penalty: {0}', penalty)
return
log.debug(u'Success. Distance: {0}'.format(dist))
log.debug(u'Success. Distance: {0}', dist)
results[info.album_id] = hooks.AlbumMatch(dist, info, mapping,
extra_items, extra_tracks)
def tag_album(items, search_artist=None, search_album=None,
search_id=None):
"""Return a tuple of a artist name, an album name, a list of
`AlbumMatch` candidates from the metadata backend, and a
`Recommendation`.
search_ids=[]):
"""Return a tuple of the current artist name, the current album
name, and a `Proposal` containing `AlbumMatch` candidates.
The artist and album are the most common values of these fields
among `items`.
The `AlbumMatch` objects are generated by searching the metadata
backends. By default, the metadata of the items is used for the
search. This can be customized by setting the parameters. The
`mapping` field of the album has the matched `items` as keys.
search. This can be customized by setting the parameters.
`search_ids` is a list of metadata backend IDs: if specified,
it will restrict the candidates to those IDs, ignoring
`search_artist` and `search album`. The `mapping` field of the
album has the matched `items` as keys.
The recommendation is calculated from the match qualitiy of the
The recommendation is calculated from the match quality of the
candidates.
"""
# Get current metadata.
likelies, consensus = current_metadata(items)
cur_artist = likelies['artist']
cur_album = likelies['album']
log.debug(u'Tagging {0} - {1}'.format(cur_artist, cur_album))
log.debug(u'Tagging {0} - {1}', cur_artist, cur_album)
# The output result (distance, AlbumInfo) tuples (keyed by MB album
# ID).
candidates = {}
# Search by explicit ID.
if search_id is not None:
log.debug(u'Searching for album ID: {0}'.format(search_id))
search_cands = hooks.albums_for_id(search_id)
if search_ids:
for search_id in search_ids:
log.debug(u'Searching for album ID: {0}', search_id)
for id_candidate in hooks.albums_for_id(search_id):
_add_candidate(items, candidates, id_candidate)
# Use existing metadata or text search.
else:
@@ -404,81 +430,84 @@ def tag_album(items, search_artist=None, search_album=None,
id_info = match_by_id(items)
if id_info:
_add_candidate(items, candidates, id_info)
rec = _recommendation(candidates.values())
log.debug(u'Album ID match recommendation is {0}'.format(str(rec)))
rec = _recommendation(list(candidates.values()))
log.debug(u'Album ID match recommendation is {0}', rec)
if candidates and not config['import']['timid']:
# If we have a very good MBID match, return immediately.
# Otherwise, this match will compete against metadata-based
# matches.
if rec == Recommendation.strong:
log.debug(u'ID match.')
return cur_artist, cur_album, candidates.values(), rec
return cur_artist, cur_album, \
Proposal(list(candidates.values()), rec)
# Search terms.
if not (search_artist and search_album):
# No explicit search terms -- use current metadata.
search_artist, search_album = cur_artist, cur_album
log.debug(u'Search terms: {0} - {1}'.format(search_artist,
search_album))
log.debug(u'Search terms: {0} - {1}', search_artist, search_album)
# Is this album likely to be a "various artist" release?
va_likely = ((not consensus['artist']) or
(search_artist.lower() in VA_ARTISTS) or
any(item.comp for item in items))
log.debug(u'Album might be VA: {0}'.format(str(va_likely)))
log.debug(u'Album might be VA: {0}', va_likely)
# Get the results from the data sources.
search_cands = hooks.album_candidates(items, search_artist,
search_album, va_likely)
log.debug(u'Evaluating {0} candidates.'.format(len(search_cands)))
for info in search_cands:
_add_candidate(items, candidates, info)
for matched_candidate in hooks.album_candidates(items,
search_artist,
search_album,
va_likely):
_add_candidate(items, candidates, matched_candidate)
log.debug(u'Evaluating {0} candidates.', len(candidates))
# Sort and get the recommendation.
candidates = sorted(candidates.itervalues())
candidates = _sort_candidates(candidates.values())
rec = _recommendation(candidates)
return cur_artist, cur_album, candidates, rec
return cur_artist, cur_album, Proposal(candidates, rec)
def tag_item(item, search_artist=None, search_title=None,
search_id=None):
"""Attempts to find metadata for a single track. Returns a
`(candidates, recommendation)` pair where `candidates` is a list of
TrackMatch objects. `search_artist` and `search_title` may be used
search_ids=[]):
"""Find metadata for a single track. Return a `Proposal` consisting
of `TrackMatch` objects.
`search_artist` and `search_title` may be used
to override the current metadata for the purposes of the MusicBrainz
title; likewise `search_id`.
title. `search_ids` may be used for restricting the search to a list
of metadata backend IDs.
"""
# Holds candidates found so far: keys are MBIDs; values are
# (distance, TrackInfo) pairs.
candidates = {}
# First, try matching by MusicBrainz ID.
trackid = search_id or item.mb_trackid
if trackid:
log.debug(u'Searching for track ID: {0}'.format(trackid))
for track_info in hooks.tracks_for_id(trackid):
dist = track_distance(item, track_info, incl_artist=True)
candidates[track_info.track_id] = \
hooks.TrackMatch(dist, track_info)
# If this is a good match, then don't keep searching.
rec = _recommendation(candidates.values())
if rec == Recommendation.strong and not config['import']['timid']:
log.debug(u'Track ID match.')
return candidates.values(), rec
trackids = search_ids or [t for t in [item.mb_trackid] if t]
if trackids:
for trackid in trackids:
log.debug(u'Searching for track ID: {0}', trackid)
for track_info in hooks.tracks_for_id(trackid):
dist = track_distance(item, track_info, incl_artist=True)
candidates[track_info.track_id] = \
hooks.TrackMatch(dist, track_info)
# If this is a good match, then don't keep searching.
rec = _recommendation(_sort_candidates(candidates.values()))
if rec == Recommendation.strong and \
not config['import']['timid']:
log.debug(u'Track ID match.')
return Proposal(_sort_candidates(candidates.values()), rec)
# If we're searching by ID, don't proceed.
if search_id is not None:
if search_ids:
if candidates:
return candidates.values(), rec
return Proposal(_sort_candidates(candidates.values()), rec)
else:
return [], Recommendation.none
return Proposal([], Recommendation.none)
# Search terms.
if not (search_artist and search_title):
search_artist, search_title = item.artist, item.title
log.debug(u'Item search terms: {0} - {1}'.format(search_artist,
search_title))
log.debug(u'Item search terms: {0} - {1}', search_artist, search_title)
# Get and evaluate candidate metadata.
for track_info in hooks.item_candidates(item, search_artist, search_title):
@@ -486,7 +515,7 @@ def tag_item(item, search_artist=None, search_title=None,
candidates[track_info.track_id] = hooks.TrackMatch(dist, track_info)
# Sort by distance and return with recommendation.
log.debug(u'Found {0} candidates.'.format(len(candidates)))
candidates = sorted(candidates.itervalues())
log.debug(u'Found {0} candidates.', len(candidates))
candidates = _sort_candidates(candidates.values())
rec = _recommendation(candidates)
return candidates, rec
return Proposal(candidates, rec)
lib/beets/autotag/mb.py
Regular → Executable
+77 -23
View File
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2013, Adrian Sampson.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -14,23 +15,29 @@
"""Searches for albums in the MusicBrainz database.
"""
import logging
from __future__ import division, absolute_import, print_function
import musicbrainzngs
import re
import traceback
from urlparse import urljoin
from six.moves.urllib.parse import urljoin
from beets import logging
import beets.autotag.hooks
import beets
from beets import util
from beets import config
import six
SEARCH_LIMIT = 5
VARIOUS_ARTISTS_ID = '89ad4ac3-39f7-470e-963a-56509c546377'
BASE_URL = 'http://musicbrainz.org/'
if util.SNI_SUPPORTED:
BASE_URL = 'https://musicbrainz.org/'
else:
BASE_URL = 'http://musicbrainz.org/'
musicbrainzngs.set_useragent('beets', beets.__version__,
'http://beets.radbox.org/')
'http://beets.io/')
class MusicBrainzAPIError(util.HumanReadableException):
@@ -39,6 +46,8 @@ class MusicBrainzAPIError(util.HumanReadableException):
"""
def __init__(self, reason, verb, query, tb=None):
self.query = query
if isinstance(reason, musicbrainzngs.WebServiceError):
reason = u'MusicBrainz not reachable'
super(MusicBrainzAPIError, self).__init__(reason, verb, tb)
def get_message(self):
@@ -49,8 +58,12 @@ class MusicBrainzAPIError(util.HumanReadableException):
log = logging.getLogger('beets')
RELEASE_INCLUDES = ['artists', 'media', 'recordings', 'release-groups',
'labels', 'artist-credits', 'aliases']
'labels', 'artist-credits', 'aliases',
'recording-level-rels', 'work-rels',
'work-level-rels', 'artist-rels']
TRACK_INCLUDES = ['artists', 'aliases']
if 'work-level-rels' in musicbrainzngs.VALID_INCLUDES['recording']:
TRACK_INCLUDES += ['work-level-rels', 'artist-rels']
def track_url(trackid):
@@ -65,7 +78,8 @@ def configure():
"""Set up the python-musicbrainz-ngs module according to settings
from the beets configuration. This should be called at startup.
"""
musicbrainzngs.set_hostname(config['musicbrainz']['host'].get(unicode))
hostname = config['musicbrainz']['host'].as_str()
musicbrainzngs.set_hostname(hostname)
musicbrainzngs.set_rate_limit(
config['musicbrainz']['ratelimit_interval'].as_number(),
config['musicbrainz']['ratelimit'].get(int),
@@ -104,7 +118,7 @@ def _flatten_artist_credit(credit):
artist_sort_parts = []
artist_credit_parts = []
for el in credit:
if isinstance(el, basestring):
if isinstance(el, six.string_types):
# Join phrase.
artist_parts.append(el)
artist_credit_parts.append(el)
@@ -157,6 +171,7 @@ def track_info(recording, index=None, medium=None, medium_index=None,
medium=medium,
medium_index=medium_index,
medium_total=medium_total,
data_source=u'MusicBrainz',
data_url=track_url(recording['id']),
)
@@ -172,6 +187,33 @@ def track_info(recording, index=None, medium=None, medium_index=None,
if recording.get('length'):
info.length = int(recording['length']) / (1000.0)
lyricist = []
composer = []
for work_relation in recording.get('work-relation-list', ()):
if work_relation['type'] != 'performance':
continue
for artist_relation in work_relation['work'].get(
'artist-relation-list', ()):
if 'type' in artist_relation:
type = artist_relation['type']
if type == 'lyricist':
lyricist.append(artist_relation['artist']['name'])
elif type == 'composer':
composer.append(artist_relation['artist']['name'])
if lyricist:
info.lyricist = u', '.join(lyricist)
if composer:
info.composer = u', '.join(composer)
arranger = []
for artist_relation in recording.get('artist-relation-list', ()):
if 'type' in artist_relation:
type = artist_relation['type']
if type == 'arranger':
arranger.append(artist_relation['artist']['name'])
if arranger:
info.arranger = u', '.join(arranger)
info.decode()
return info
@@ -210,7 +252,12 @@ def album_info(release):
for medium in release['medium-list']:
disctitle = medium.get('title')
format = medium.get('format')
for track in medium['track-list']:
all_tracks = medium['track-list']
if 'pregap' in medium:
all_tracks.insert(0, medium['pregap'])
for track in all_tracks:
# Basic information from the recording.
index += 1
ti = track_info(
@@ -222,6 +269,7 @@ def album_info(release):
)
ti.disctitle = disctitle
ti.media = format
ti.track_alt = track['number']
# Prefer track data, where present, over recording data.
if track.get('title'):
@@ -245,10 +293,12 @@ def album_info(release):
mediums=len(release['medium-list']),
artist_sort=artist_sort_name,
artist_credit=artist_credit_name,
data_source='MusicBrainz',
data_source=u'MusicBrainz',
data_url=album_url(release['id']),
)
info.va = info.artist_id == VARIOUS_ARTISTS_ID
if info.va:
info.artist = config['va_name'].as_str()
info.asin = release.get('asin')
info.releasegroup_id = release['release-group']['id']
info.country = release.get('country')
@@ -301,7 +351,7 @@ def album_info(release):
return info
def match_album(artist, album, tracks=None, limit=SEARCH_LIMIT):
def match_album(artist, album, tracks=None):
"""Searches for a single album ("release" in MusicBrainz parlance)
and returns an iterator over AlbumInfo objects. May raise a
MusicBrainzAPIError.
@@ -317,14 +367,16 @@ def match_album(artist, album, tracks=None, limit=SEARCH_LIMIT):
# Various Artists search.
criteria['arid'] = VARIOUS_ARTISTS_ID
if tracks is not None:
criteria['tracks'] = str(tracks)
criteria['tracks'] = six.text_type(tracks)
# Abort if we have no search terms.
if not any(criteria.itervalues()):
if not any(criteria.values()):
return
try:
res = musicbrainzngs.search_releases(limit=limit, **criteria)
log.debug(u'Searching for MusicBrainz releases with: {!r}', criteria)
res = musicbrainzngs.search_releases(
limit=config['musicbrainz']['searchlimit'].get(int), **criteria)
except musicbrainzngs.MusicBrainzError as exc:
raise MusicBrainzAPIError(exc, 'release search', criteria,
traceback.format_exc())
@@ -336,7 +388,7 @@ def match_album(artist, album, tracks=None, limit=SEARCH_LIMIT):
yield albuminfo
def match_track(artist, title, limit=SEARCH_LIMIT):
def match_track(artist, title):
"""Searches for a single track and returns an iterable of TrackInfo
objects. May raise a MusicBrainzAPIError.
"""
@@ -345,11 +397,12 @@ def match_track(artist, title, limit=SEARCH_LIMIT):
'recording': title.lower().strip(),
}
if not any(criteria.itervalues()):
if not any(criteria.values()):
return
try:
res = musicbrainzngs.search_recordings(limit=limit, **criteria)
res = musicbrainzngs.search_recordings(
limit=config['musicbrainz']['searchlimit'].get(int), **criteria)
except musicbrainzngs.MusicBrainzError as exc:
raise MusicBrainzAPIError(exc, 'recording search', criteria,
traceback.format_exc())
@@ -362,7 +415,7 @@ def _parse_id(s):
no ID can be found, return None.
"""
# Find the first thing that looks like a UUID/MBID.
match = re.search('[a-f0-9]{8}(-[a-f0-9]{4}){3}-[a-f0-9]{12}', s)
match = re.search(u'[a-f0-9]{8}(-[a-f0-9]{4}){3}-[a-f0-9]{12}', s)
if match:
return match.group()
@@ -372,9 +425,10 @@ def album_for_id(releaseid):
object or None if the album is not found. May raise a
MusicBrainzAPIError.
"""
log.debug(u'Requesting MusicBrainz release {}', releaseid)
albumid = _parse_id(releaseid)
if not albumid:
log.debug(u'Invalid MBID ({0}).'.format(releaseid))
log.debug(u'Invalid MBID ({0}).', releaseid)
return
try:
res = musicbrainzngs.get_release_by_id(albumid,
@@ -383,7 +437,7 @@ def album_for_id(releaseid):
log.debug(u'Album ID match failed.')
return None
except musicbrainzngs.MusicBrainzError as exc:
raise MusicBrainzAPIError(exc, 'get release by ID', albumid,
raise MusicBrainzAPIError(exc, u'get release by ID', albumid,
traceback.format_exc())
return album_info(res['release'])
@@ -394,7 +448,7 @@ def track_for_id(releaseid):
"""
trackid = _parse_id(releaseid)
if not trackid:
log.debug(u'Invalid MBID ({0}).'.format(releaseid))
log.debug(u'Invalid MBID ({0}).', releaseid)
return
try:
res = musicbrainzngs.get_recording_by_id(trackid, TRACK_INCLUDES)
@@ -402,6 +456,6 @@ def track_for_id(releaseid):
log.debug(u'Track ID match failed.')
return None
except musicbrainzngs.MusicBrainzError as exc:
raise MusicBrainzAPIError(exc, 'get recording by ID', trackid,
raise MusicBrainzAPIError(exc, u'get recording by ID', trackid,
traceback.format_exc())
return track_info(res['recording'])
lib/beets/config_default.yaml
Regular → Executable
+23 -6
View File
@@ -6,6 +6,7 @@ import:
copy: yes
move: no
link: no
hardlink: no
delete: no
resume: ask
incremental: no
@@ -22,9 +23,13 @@ import:
flat: no
group_albums: no
pretend: false
search_ids: []
duplicate_action: ask
clutter: ["Thumbs.DB", ".DS_Store"]
ignore: [".*", "*~", "System Volume Information"]
ignore: [".*", "*~", "System Volume Information", "lost+found"]
ignore_hidden: yes
replace:
'[\\/]': _
'^\.': _
@@ -41,24 +46,35 @@ max_filename_length: 0
plugins: []
pluginpath: []
threaded: yes
color: yes
timeout: 5.0
per_disc_numbering: no
verbose: no
terminal_encoding: utf8
verbose: 0
terminal_encoding:
original_date: no
id3v23: no
va_name: "Various Artists"
ui:
terminal_width: 80
length_diff_thresh: 10.0
color: yes
colors:
text_success: green
text_warning: yellow
text_error: red
text_highlight: red
text_highlight_minor: lightgray
action_default: turquoise
action: blue
list_format_item: $artist - $album - $title
list_format_album: $albumartist - $album
format_item: $artist - $album - $title
format_album: $albumartist - $album
time_format: '%Y-%m-%d %H:%M:%S'
format_raw_length: no
sort_album: albumartist+ album+
sort_item: artist+ album+ disc+ track+
sort_case_insensitive: yes
paths:
default: $albumartist/$album%aunique{}/$track $title
@@ -71,6 +87,7 @@ musicbrainz:
host: musicbrainz.org
ratelimit: 1
ratelimit_interval: 1.0
searchlimit: 5
match:
strong_rec_thresh: 0.04
lib/beets/dbcore/__init__.py
Regular → Executable
+5 -1
View File
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2014, Adrian Sampson.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -15,11 +16,14 @@
"""DBCore is an abstract database package that forms the basis for beets'
Library.
"""
from __future__ import division, absolute_import, print_function
from .db import Model, Database
from .query import Query, FieldQuery, MatchQuery, AndQuery, OrQuery
from .types import Type
from .queryparse import query_from_strings
from .queryparse import sort_from_strings
from .queryparse import parse_sorted_query
from .query import InvalidQueryError
# flake8: noqa
lib/beets/dbcore/db.py
Regular → Executable
+84 -37
View File
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2014, Adrian Sampson.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -14,6 +15,8 @@
"""The central Model and Database constructs for DBCore.
"""
from __future__ import division, absolute_import, print_function
import time
import os
from collections import defaultdict
@@ -24,14 +27,16 @@ import collections
import beets
from beets.util.functemplate import Template
from beets.util import py3_path
from beets.dbcore import types
from .query import MatchQuery, NullSort, TrueQuery
import six
class FormattedMapping(collections.Mapping):
"""A `dict`-like formatted view of a model.
The accessor `mapping[key]` returns the formated version of
The accessor `mapping[key]` returns the formatted version of
`model[key]` as a unicode string.
If `for_path` is true, all path separators in the formatted values
@@ -63,10 +68,10 @@ class FormattedMapping(collections.Mapping):
def _get_formatted(self, model, key):
value = model._type(key).format(model.get(key))
if isinstance(value, bytes):
value = value.decode('utf8', 'ignore')
value = value.decode('utf-8', 'ignore')
if self.for_path:
sep_repl = beets.config['path_sep_replace'].get(unicode)
sep_repl = beets.config['path_sep_replace'].as_str()
for sep in (os.path.sep, os.path.altsep):
if sep:
value = value.replace(sep, sep_repl)
@@ -173,9 +178,9 @@ class Model(object):
ordinary construction are bypassed.
"""
obj = cls(db)
for key, value in fixed_values.iteritems():
for key, value in fixed_values.items():
obj._values_fixed[key] = cls._type(key).from_sql(value)
for key, value in flex_values.iteritems():
for key, value in flex_values.items():
obj._values_flex[key] = cls._type(key).from_sql(value)
return obj
@@ -197,20 +202,22 @@ class Model(object):
exception is raised otherwise.
"""
if not self._db:
raise ValueError('{0} has no database'.format(type(self).__name__))
raise ValueError(
u'{0} has no database'.format(type(self).__name__)
)
if need_id and not self.id:
raise ValueError('{0} has no id'.format(type(self).__name__))
raise ValueError(u'{0} has no id'.format(type(self).__name__))
# Essential field accessors.
@classmethod
def _type(self, key):
def _type(cls, key):
"""Get the type of a field, a `Type` instance.
If the field has no explicit type, it is given the base `Type`,
which does no conversion.
"""
return self._fields.get(key) or self._types.get(key) or types.DEFAULT
return cls._fields.get(key) or cls._types.get(key) or types.DEFAULT
def __getitem__(self, key):
"""Get the value for a field. Raise a KeyError if the field is
@@ -251,23 +258,30 @@ class Model(object):
del self._values_flex[key]
self._dirty.add(key) # Mark for dropping on store.
elif key in self._getters(): # Computed.
raise KeyError('computed field {0} cannot be deleted'.format(key))
raise KeyError(u'computed field {0} cannot be deleted'.format(key))
elif key in self._fields: # Fixed.
raise KeyError('fixed field {0} cannot be deleted'.format(key))
raise KeyError(u'fixed field {0} cannot be deleted'.format(key))
else:
raise KeyError('no such field {0}'.format(key))
raise KeyError(u'no such field {0}'.format(key))
def keys(self, computed=False):
"""Get a list of available field names for this object. The
`computed` parameter controls whether computed (plugin-provided)
fields are included in the key list.
"""
base_keys = list(self._fields) + self._values_flex.keys()
base_keys = list(self._fields) + list(self._values_flex.keys())
if computed:
return base_keys + self._getters().keys()
return base_keys + list(self._getters().keys())
else:
return base_keys
@classmethod
def all_keys(cls):
"""Get a list of available keys for objects of this type.
Includes fixed and computed fields.
"""
return list(cls._fields) + list(cls._getters().keys())
# Act like a dictionary.
def update(self, values):
@@ -307,12 +321,12 @@ class Model(object):
def __getattr__(self, key):
if key.startswith('_'):
raise AttributeError('model has no attribute {0!r}'.format(key))
raise AttributeError(u'model has no attribute {0!r}'.format(key))
else:
try:
return self[key]
except KeyError:
raise AttributeError('no such field {0!r}'.format(key))
raise AttributeError(u'no such field {0!r}'.format(key))
def __setattr__(self, key, value):
if key.startswith('_'):
@@ -328,15 +342,19 @@ class Model(object):
# Database interaction (CRUD methods).
def store(self):
def store(self, fields=None):
"""Save the object's metadata into the library database.
:param fields: the fields to be stored. If not specified, all fields
will be.
"""
if fields is None:
fields = self._fields
self._check_db()
# Build assignments for query.
assignments = []
subvars = []
for key in self._fields:
for key in fields:
if key != 'id' and key in self._dirty:
self._dirty.remove(key)
assignments.append(key + '=?')
@@ -379,7 +397,7 @@ class Model(object):
"""
self._check_db()
stored_obj = self._db._get(type(self), self.id)
assert stored_obj is not None, "object {0} not in DB".format(self.id)
assert stored_obj is not None, u"object {0} not in DB".format(self.id)
self._values_fixed = {}
self._values_flex = {}
self.update(dict(stored_obj))
@@ -440,7 +458,7 @@ class Model(object):
separators will be added to the template.
"""
# Perform substitution.
if isinstance(template, basestring):
if isinstance(template, six.string_types):
template = Template(template)
return template.substitute(self.formatted(for_path),
self._template_funcs())
@@ -451,11 +469,16 @@ class Model(object):
def _parse(cls, key, string):
"""Parse a string as a value for the given key.
"""
if not isinstance(string, basestring):
raise TypeError("_parse() argument must be a string")
if not isinstance(string, six.string_types):
raise TypeError(u"_parse() argument must be a string")
return cls._type(key).parse(string)
def set_parse(self, key, string):
"""Set the object's key to a value represented by a string.
"""
self[key] = self._parse(key, string)
# Database controller and supporting interfaces.
@@ -576,6 +599,11 @@ class Results(object):
return self._row_count
def __nonzero__(self):
"""Does this result contain any objects?
"""
return self.__bool__()
def __bool__(self):
"""Does this result contain any objects?
"""
return bool(len(self))
@@ -592,10 +620,10 @@ class Results(object):
it = iter(self)
try:
for i in range(n):
it.next()
return it.next()
next(it)
return next(it)
except StopIteration:
raise IndexError('result index {0} out of range'.format(n))
raise IndexError(u'result index {0} out of range'.format(n))
def get(self):
"""Return the first matching object, or None if no objects
@@ -603,7 +631,7 @@ class Results(object):
"""
it = iter(self)
try:
return it.next()
return next(it)
except StopIteration:
return None
@@ -668,8 +696,9 @@ class Database(object):
"""The Model subclasses representing tables in this database.
"""
def __init__(self, path):
def __init__(self, path, timeout=5.0):
self.path = path
self.timeout = timeout
self._connections = {}
self._tx_stacks = defaultdict(list)
@@ -704,18 +733,36 @@ class Database(object):
if thread_id in self._connections:
return self._connections[thread_id]
else:
# Make a new connection.
conn = sqlite3.connect(
self.path,
timeout=beets.config['timeout'].as_number(),
)
# Access SELECT results like dictionaries.
conn.row_factory = sqlite3.Row
conn = self._create_connection()
self._connections[thread_id] = conn
return conn
def _create_connection(self):
"""Create a SQLite connection to the underlying database.
Makes a new connection every time. If you need to configure the
connection settings (e.g., add custom functions), override this
method.
"""
# Make a new connection. The `sqlite3` module can't use
# bytestring paths here on Python 3, so we need to
# provide a `str` using `py3_path`.
conn = sqlite3.connect(
py3_path(self.path), timeout=self.timeout
)
# Access SELECT results like dictionaries.
conn.row_factory = sqlite3.Row
return conn
def _close(self):
"""Close the all connections to the underlying SQLite database
from all threads. This does not render the database object
unusable; new connections can still be opened on demand.
"""
with self._shared_map_lock:
self._connections.clear()
@contextlib.contextmanager
def _tx_stack(self):
"""A context manager providing access to the current thread's
lib/beets/dbcore/query.py
Regular → Executable
+258 -46
View File
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2014, Adrian Sampson.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -14,10 +15,49 @@
"""The Query type hierarchy for DBCore.
"""
from __future__ import division, absolute_import, print_function
import re
from operator import attrgetter
from operator import mul
from beets import util
from datetime import datetime, timedelta
import unicodedata
from functools import reduce
import six
if not six.PY2:
buffer = memoryview # sqlite won't accept memoryview in python 2
class ParsingError(ValueError):
"""Abstract class for any unparseable user-requested album/query
specification.
"""
class InvalidQueryError(ParsingError):
"""Represent any kind of invalid query.
The query should be a unicode string or a list, which will be space-joined.
"""
def __init__(self, query, explanation):
if isinstance(query, list):
query = " ".join(query)
message = u"'{0}': {1}".format(query, explanation)
super(InvalidQueryError, self).__init__(message)
class InvalidQueryArgumentTypeError(ParsingError):
"""Represent a query argument that could not be converted as expected.
It exists to be caught in upper stack levels so a meaningful (i.e. with the
query) InvalidQueryError can be raised.
"""
def __init__(self, what, expected, detail=None):
message = u"'{0}' is not {1}".format(what, expected)
if detail:
message = u"{0}: {1}".format(message, detail)
super(InvalidQueryArgumentTypeError, self).__init__(message)
class Query(object):
@@ -25,9 +65,8 @@ class Query(object):
"""
def clause(self):
"""Generate an SQLite expression implementing the query.
Return a clause string, a sequence of substitution values for
the clause, and a Query object representing the "remainder"
Returns (clause, subvals) where clause is a valid sqlite
Return (clause, subvals) where clause is a valid sqlite
WHERE clause implementing the query and subvals is a list of
items to be substituted for ?s in the clause.
"""
@@ -39,6 +78,15 @@ class Query(object):
"""
raise NotImplementedError
def __repr__(self):
return "{0.__class__.__name__}()".format(self)
def __eq__(self, other):
return type(self) == type(other)
def __hash__(self):
return 0
class FieldQuery(Query):
"""An abstract query that searches in a specific field for a
@@ -72,6 +120,17 @@ class FieldQuery(Query):
def match(self, item):
return self.value_match(self.pattern, item.get(self.field))
def __repr__(self):
return ("{0.__class__.__name__}({0.field!r}, {0.pattern!r}, "
"{0.fast})".format(self))
def __eq__(self, other):
return super(FieldQuery, self).__eq__(other) and \
self.field == other.field and self.pattern == other.pattern
def __hash__(self):
return hash((self.field, hash(self.pattern)))
class MatchQuery(FieldQuery):
"""A query that looks for exact matches in an item field."""
@@ -86,19 +145,21 @@ class MatchQuery(FieldQuery):
class NoneQuery(FieldQuery):
def __init__(self, field, fast=True):
self.field = field
self.fast = fast
super(NoneQuery, self).__init__(field, None, fast)
def col_clause(self):
return self.field + " IS NULL", ()
@classmethod
def match(self, item):
def match(cls, item):
try:
return item[self.field] is None
return item[cls.field] is None
except KeyError:
return True
def __repr__(self):
return "{0.__class__.__name__}({0.field!r}, {0.fast})".format(self)
class StringFieldQuery(FieldQuery):
"""A FieldQuery that converts values to strings before matching
@@ -139,15 +200,31 @@ class SubstringQuery(StringFieldQuery):
class RegexpQuery(StringFieldQuery):
"""A query that matches a regular expression in a specific item
field.
Raises InvalidQueryError when the pattern is not a valid regular
expression.
"""
def __init__(self, field, pattern, fast=True):
super(RegexpQuery, self).__init__(field, pattern, fast)
pattern = self._normalize(pattern)
try:
self.pattern = re.compile(self.pattern)
except re.error as exc:
# Invalid regular expression.
raise InvalidQueryArgumentTypeError(pattern,
u"a regular expression",
format(exc))
@staticmethod
def _normalize(s):
"""Normalize a Unicode string's representation (used on both
patterns and matched values).
"""
return unicodedata.normalize('NFC', s)
@classmethod
def string_match(cls, pattern, value):
try:
res = re.search(pattern, value)
except re.error:
# Invalid regular expression.
return False
return res is not None
return pattern.search(cls._normalize(value)) is not None
class BooleanQuery(MatchQuery):
@@ -156,28 +233,26 @@ class BooleanQuery(MatchQuery):
"""
def __init__(self, field, pattern, fast=True):
super(BooleanQuery, self).__init__(field, pattern, fast)
if isinstance(pattern, basestring):
if isinstance(pattern, six.string_types):
self.pattern = util.str2bool(pattern)
self.pattern = int(self.pattern)
class BytesQuery(MatchQuery):
"""Match a raw bytes field (i.e., a path). This is a necessary hack
to work around the `sqlite3` module's desire to treat `str` and
to work around the `sqlite3` module's desire to treat `bytes` and
`unicode` equivalently in Python 2. Always use this query instead of
`MatchQuery` when matching on BLOB values.
"""
def __init__(self, field, pattern):
super(BytesQuery, self).__init__(field, pattern)
# Use a buffer representation of the pattern for SQLite
# Use a buffer/memoryview representation of the pattern for SQLite
# matching. This instructs SQLite to treat the blob as binary
# rather than encoded Unicode.
if isinstance(self.pattern, basestring):
# Implicitly coerce Unicode strings to their bytes
# equivalents.
if isinstance(self.pattern, unicode):
self.pattern = self.pattern.encode('utf8')
if isinstance(self.pattern, (six.text_type, bytes)):
if isinstance(self.pattern, six.text_type):
self.pattern = self.pattern.encode('utf-8')
self.buf_pattern = buffer(self.pattern)
elif isinstance(self.pattern, buffer):
self.buf_pattern = self.pattern
@@ -191,19 +266,26 @@ class NumericQuery(FieldQuery):
"""Matches numeric fields. A syntax using Ruby-style range ellipses
(``..``) lets users specify one- or two-sided ranges. For example,
``year:2001..`` finds music released since the turn of the century.
Raises InvalidQueryError when the pattern does not represent an int or
a float.
"""
def _convert(self, s):
"""Convert a string to a numeric type (float or int). If the
string cannot be converted, return None.
"""Convert a string to a numeric type (float or int).
Return None if `s` is empty.
Raise an InvalidQueryError if the string cannot be converted.
"""
# This is really just a bit of fun premature optimization.
if not s:
return None
try:
return int(s)
except ValueError:
try:
return float(s)
except ValueError:
return None
raise InvalidQueryArgumentTypeError(s, u"an int or a float")
def __init__(self, field, pattern, fast=True):
super(NumericQuery, self).__init__(field, pattern, fast)
@@ -224,7 +306,7 @@ class NumericQuery(FieldQuery):
if self.field not in item:
return False
value = item[self.field]
if isinstance(value, basestring):
if isinstance(value, six.string_types):
value = self._convert(value)
if self.point is not None:
@@ -248,7 +330,7 @@ class NumericQuery(FieldQuery):
elif self.rangemax is not None:
return u'{0} <= ?'.format(self.field), (self.rangemax,)
else:
return '1', ()
return u'1', ()
class CollectionQuery(Query):
@@ -273,7 +355,7 @@ class CollectionQuery(Query):
return item in self.subqueries
def clause_with_joiner(self, joiner):
"""Returns a clause created by joining together the clauses of
"""Return a clause created by joining together the clauses of
all subqueries with the string joiner (padded by spaces).
"""
clause_parts = []
@@ -288,6 +370,19 @@ class CollectionQuery(Query):
clause = (' ' + joiner + ' ').join(clause_parts)
return clause, subvals
def __repr__(self):
return "{0.__class__.__name__}({0.subqueries!r})".format(self)
def __eq__(self, other):
return super(CollectionQuery, self).__eq__(other) and \
self.subqueries == other.subqueries
def __hash__(self):
"""Since subqueries are mutable, this object should not be hashable.
However and for conveniences purposes, it can be hashed.
"""
return reduce(mul, map(hash, self.subqueries), 1)
class AnyFieldQuery(CollectionQuery):
"""A query that matches if a given FieldQuery subclass matches in
@@ -313,6 +408,17 @@ class AnyFieldQuery(CollectionQuery):
return True
return False
def __repr__(self):
return ("{0.__class__.__name__}({0.pattern!r}, {0.fields!r}, "
"{0.query_class.__name__})".format(self))
def __eq__(self, other):
return super(AnyFieldQuery, self).__eq__(other) and \
self.query_class == other.query_class
def __hash__(self):
return hash((self.pattern, tuple(self.fields), self.query_class))
class MutableCollectionQuery(CollectionQuery):
"""A collection query whose subqueries may be modified after the
@@ -343,6 +449,36 @@ class OrQuery(MutableCollectionQuery):
return any([q.match(item) for q in self.subqueries])
class NotQuery(Query):
"""A query that matches the negation of its `subquery`, as a shorcut for
performing `not(subquery)` without using regular expressions.
"""
def __init__(self, subquery):
self.subquery = subquery
def clause(self):
clause, subvals = self.subquery.clause()
if clause:
return 'not ({0})'.format(clause), subvals
else:
# If there is no clause, there is nothing to negate. All the logic
# is handled by match() for slow queries.
return clause, subvals
def match(self, item):
return not self.subquery.match(item)
def __repr__(self):
return "{0.__class__.__name__}({0.subquery!r})".format(self)
def __eq__(self, other):
return super(NotQuery, self).__eq__(other) and \
self.subquery == other.subquery
def __hash__(self):
return hash(('not', hash(self.subquery)))
class TrueQuery(Query):
"""A query that always matches."""
def clause(self):
@@ -367,13 +503,13 @@ def _to_epoch_time(date):
"""Convert a `datetime` object to an integer number of seconds since
the (local) Unix epoch.
"""
epoch = datetime.fromtimestamp(0)
delta = date - epoch
try:
if hasattr(date, 'timestamp'):
# The `timestamp` method exists on Python 3.3+.
return int(date.timestamp())
else:
epoch = datetime.fromtimestamp(0)
delta = date - epoch
return int(delta.total_seconds())
except AttributeError:
# datetime.timedelta.total_seconds() is not available on Python 2.6
return delta.seconds + delta.days * 24 * 3600
def _parse_periods(pattern):
@@ -405,7 +541,7 @@ class Period(object):
precision (a string, one of "year", "month", or "day").
"""
if precision not in Period.precisions:
raise ValueError('Invalid precision ' + str(precision))
raise ValueError(u'Invalid precision {0}'.format(precision))
self.date = date
self.precision = precision
@@ -445,7 +581,7 @@ class Period(object):
elif 'day' == precision:
return date + timedelta(days=1)
else:
raise ValueError('unhandled precision ' + str(precision))
raise ValueError(u'unhandled precision {0}'.format(precision))
class DateInterval(object):
@@ -457,7 +593,7 @@ class DateInterval(object):
def __init__(self, start, end):
if start is not None and end is not None and not start < end:
raise ValueError("start date {0} is not before end date {1}"
raise ValueError(u"start date {0} is not before end date {1}"
.format(start, end))
self.start = start
self.end = end
@@ -478,7 +614,7 @@ class DateInterval(object):
return True
def __str__(self):
return'[{0}, {1})'.format(self.start, self.end)
return '[{0}, {1})'.format(self.start, self.end)
class DateQuery(FieldQuery):
@@ -496,6 +632,8 @@ class DateQuery(FieldQuery):
self.interval = DateInterval.from_periods(start, end)
def match(self, item):
if self.field not in item:
return False
timestamp = float(item[self.field])
date = datetime.utcfromtimestamp(timestamp)
return self.interval.contains(date)
@@ -523,6 +661,33 @@ class DateQuery(FieldQuery):
return clause, subvals
class DurationQuery(NumericQuery):
"""NumericQuery that allow human-friendly (M:SS) time interval formats.
Converts the range(s) to a float value, and delegates on NumericQuery.
Raises InvalidQueryError when the pattern does not represent an int, float
or M:SS time interval.
"""
def _convert(self, s):
"""Convert a M:SS or numeric string to a float.
Return None if `s` is empty.
Raise an InvalidQueryError if the string cannot be converted.
"""
if not s:
return None
try:
return util.raw_seconds_short(s)
except ValueError:
try:
return float(s)
except ValueError:
raise InvalidQueryArgumentTypeError(
s,
u"a M:SS string or a float")
# Sorting.
class Sort(object):
@@ -547,6 +712,12 @@ class Sort(object):
"""
return False
def __hash__(self):
return 0
def __eq__(self, other):
return type(self) == type(other)
class MultipleSort(Sort):
"""Sort that encapsulates multiple sub-sorts.
@@ -606,38 +777,67 @@ class MultipleSort(Sort):
return items
def __repr__(self):
return u'MultipleSort({0})'.format(repr(self.sorts))
return 'MultipleSort({!r})'.format(self.sorts)
def __hash__(self):
return hash(tuple(self.sorts))
def __eq__(self, other):
return super(MultipleSort, self).__eq__(other) and \
self.sorts == other.sorts
class FieldSort(Sort):
"""An abstract sort criterion that orders by a specific field (of
any kind).
"""
def __init__(self, field, ascending=True):
def __init__(self, field, ascending=True, case_insensitive=True):
self.field = field
self.ascending = ascending
self.case_insensitive = case_insensitive
def sort(self, objs):
# TODO: Conversion and null-detection here. In Python 3,
# comparisons with None fail. We should also support flexible
# attributes with different types without falling over.
return sorted(objs, key=attrgetter(self.field),
reverse=not self.ascending)
def key(item):
field_val = item.get(self.field, '')
if self.case_insensitive and isinstance(field_val, six.text_type):
field_val = field_val.lower()
return field_val
return sorted(objs, key=key, reverse=not self.ascending)
def __repr__(self):
return u'<{0}: {1}{2}>'.format(
return '<{0}: {1}{2}>'.format(
type(self).__name__,
self.field,
'+' if self.ascending else '-',
)
def __hash__(self):
return hash((self.field, self.ascending))
def __eq__(self, other):
return super(FieldSort, self).__eq__(other) and \
self.field == other.field and \
self.ascending == other.ascending
class FixedFieldSort(FieldSort):
"""Sort object to sort on a fixed field.
"""
def order_clause(self):
order = "ASC" if self.ascending else "DESC"
return "{0} {1}".format(self.field, order)
if self.case_insensitive:
field = '(CASE ' \
'WHEN TYPEOF({0})="text" THEN LOWER({0}) ' \
'WHEN TYPEOF({0})="blob" THEN LOWER({0}) ' \
'ELSE {0} END)'.format(self.field)
else:
field = self.field
return "{0} {1}".format(field, order)
class SlowFieldSort(FieldSort):
@@ -650,5 +850,17 @@ class SlowFieldSort(FieldSort):
class NullSort(Sort):
"""No sorting. Leave results unsorted."""
def sort(items):
def sort(self, items):
return items
def __nonzero__(self):
return self.__bool__()
def __bool__(self):
return False
def __eq__(self, other):
return type(self) == type(other) or other is None
def __hash__(self):
return 0
lib/beets/dbcore/queryparse.py
Regular → Executable
+126 -56
View File
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2014, Adrian Sampson.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -14,13 +15,17 @@
"""Parsing of strings into DBCore queries.
"""
from __future__ import division, absolute_import, print_function
import re
import itertools
from . import query
import beets
PARSE_QUERY_PART_REGEX = re.compile(
# Non-capturing optional segment for the keyword.
r'(-|\^)?' # Negation prefixes.
r'(?:'
r'(\S+?)' # The field key.
r'(?<!\\):' # Unescaped :
@@ -34,80 +39,124 @@ PARSE_QUERY_PART_REGEX = re.compile(
def parse_query_part(part, query_classes={}, prefixes={},
default_class=query.SubstringQuery):
"""Take a query in the form of a key/value pair separated by a
colon and return a tuple of `(key, value, cls)`. `key` may be None,
indicating that any field may be matched. `cls` is a subclass of
`FieldQuery`.
"""Parse a single *query part*, which is a chunk of a complete query
string representing a single criterion.
The optional `query_classes` parameter maps field names to default
query types; `default_class` is the fallback. `prefixes` is a map
from query prefix markers and query types. Prefix-indicated queries
take precedence over type-based queries.
A query part is a string consisting of:
- A *pattern*: the value to look for.
- Optionally, a *field name* preceding the pattern, separated by a
colon. So in `foo:bar`, `foo` is the field name and `bar` is the
pattern.
- Optionally, a *query prefix* just before the pattern (and after the
optional colon) indicating the type of query that should be used. For
example, in `~foo`, `~` might be a prefix. (The set of prefixes to
look for is given in the `prefixes` parameter.)
- Optionally, a negation indicator, `-` or `^`, at the very beginning.
To determine the query class, two factors are used: prefixes and
field types. For example, the colon prefix denotes a regular
expression query and a type map might provide a special kind of
query for numeric values. If neither a prefix nor a specific query
class is available, `default_class` is used.
Both prefixes and the separating `:` character may be escaped with a
backslash to avoid their normal meaning.
For instance,
'stapler' -> (None, 'stapler', SubstringQuery)
'color:red' -> ('color', 'red', SubstringQuery)
':^Quiet' -> (None, '^Quiet', RegexpQuery)
'color::b..e' -> ('color', 'b..e', RegexpQuery)
The function returns a tuple consisting of:
- The field name: a string or None if it's not present.
- The pattern, a string.
- The query class to use, which inherits from the base
:class:`Query` type.
- A negation flag, a bool.
Prefixes may be "escaped" with a backslash to disable the keying
behavior.
The three optional parameters determine which query class is used (i.e.,
the third return value). They are:
- `query_classes`, which maps field names to query classes. These
are used when no explicit prefix is present.
- `prefixes`, which maps prefix strings to query classes.
- `default_class`, the fallback when neither the field nor a prefix
indicates a query class.
So the precedence for determining which query class to return is:
prefix, followed by field, and finally the default.
For example, assuming the `:` prefix is used for `RegexpQuery`:
- `'stapler'` -> `(None, 'stapler', SubstringQuery, False)`
- `'color:red'` -> `('color', 'red', SubstringQuery, False)`
- `':^Quiet'` -> `(None, '^Quiet', RegexpQuery, False)`, because
the `^` follows the `:`
- `'color::b..e'` -> `('color', 'b..e', RegexpQuery, False)`
- `'-color:red'` -> `('color', 'red', SubstringQuery, True)`
"""
# Apply the regular expression and extract the components.
part = part.strip()
match = PARSE_QUERY_PART_REGEX.match(part)
assert match # Regex should always match.
key = match.group(1)
term = match.group(2).replace('\:', ':')
assert match # Regex should always match
negate = bool(match.group(1))
key = match.group(2)
term = match.group(3).replace('\:', ':')
# Match the search term against the list of prefixes.
# Check whether there's a prefix in the query and use the
# corresponding query type.
for pre, query_class in prefixes.items():
if term.startswith(pre):
return key, term[len(pre):], query_class
return key, term[len(pre):], query_class, negate
# No matching prefix: use type-based or fallback/default query.
# No matching prefix, so use either the query class determined by
# the field or the default as a fallback.
query_class = query_classes.get(key, default_class)
return key, term, query_class
return key, term, query_class, negate
def construct_query_part(model_cls, prefixes, query_part):
"""Create a query from a single query component, `query_part`, for
querying instances of `model_cls`. Return a `Query` instance.
"""Parse a *query part* string and return a :class:`Query` object.
:param model_cls: The :class:`Model` class that this is a query for.
This is used to determine the appropriate query types for the
model's fields.
:param prefixes: A map from prefix strings to :class:`Query` types.
:param query_part: The string to parse.
See the documentation for `parse_query_part` for more information on
query part syntax.
"""
# Shortcut for empty query parts.
# A shortcut for empty query parts.
if not query_part:
return query.TrueQuery()
# Get the query classes for each possible field.
# Use `model_cls` to build up a map from field names to `Query`
# classes.
query_classes = {}
for k, t in itertools.chain(model_cls._fields.items(),
model_cls._types.items()):
query_classes[k] = t.query
# Parse the string.
key, pattern, query_class = \
key, pattern, query_class, negate = \
parse_query_part(query_part, query_classes, prefixes)
# No key specified.
# If there's no key (field name) specified, this is a "match
# anything" query.
if key is None:
if issubclass(query_class, query.FieldQuery):
# The query type matches a specific field, but none was
# specified. So we use a version of the query that matches
# any field.
return query.AnyFieldQuery(pattern, model_cls._search_fields,
query_class)
q = query.AnyFieldQuery(pattern, model_cls._search_fields,
query_class)
if negate:
return query.NotQuery(q)
else:
return q
else:
# Other query type.
return query_class(pattern)
# Non-field query type.
if negate:
return query.NotQuery(query_class(pattern))
else:
return query_class(pattern)
# Otherwise, this must be a `FieldQuery`. Use the field name to
# construct the query object.
key = key.lower()
return query_class(key.lower(), pattern, key in model_cls._fields)
q = query_class(key.lower(), pattern, key in model_cls._fields)
if negate:
return query.NotQuery(q)
return q
def query_from_strings(query_cls, model_cls, prefixes, query_parts):
@@ -136,13 +185,15 @@ def construct_sort_part(model_cls, part):
assert direction in ('+', '-'), "part must end with + or -"
is_ascending = direction == '+'
case_insensitive = beets.config['sort_case_insensitive'].get(bool)
if field in model_cls._sorts:
sort = model_cls._sorts[field](model_cls, is_ascending)
sort = model_cls._sorts[field](model_cls, is_ascending,
case_insensitive)
elif field in model_cls._fields:
sort = query.FixedFieldSort(field, is_ascending)
sort = query.FixedFieldSort(field, is_ascending, case_insensitive)
else:
# Flexible or computed.
sort = query.SlowFieldSort(field, is_ascending)
sort = query.SlowFieldSort(field, is_ascending, case_insensitive)
return sort
@@ -150,31 +201,50 @@ def sort_from_strings(model_cls, sort_parts):
"""Create a `Sort` from a list of sort criteria (strings).
"""
if not sort_parts:
return query.NullSort()
sort = query.NullSort()
elif len(sort_parts) == 1:
sort = construct_sort_part(model_cls, sort_parts[0])
else:
sort = query.MultipleSort()
for part in sort_parts:
sort.add_sort(construct_sort_part(model_cls, part))
return sort
return sort
def parse_sorted_query(model_cls, parts, prefixes={},
query_cls=query.AndQuery):
def parse_sorted_query(model_cls, parts, prefixes={}):
"""Given a list of strings, create the `Query` and `Sort` that they
represent.
"""
# Separate query token and sort token.
query_parts = []
sort_parts = []
for part in parts:
if part.endswith((u'+', u'-')) and u':' not in part:
sort_parts.append(part)
else:
query_parts.append(part)
# Parse each.
q = query_from_strings(
query_cls, model_cls, prefixes, query_parts
)
# Split up query in to comma-separated subqueries, each representing
# an AndQuery, which need to be joined together in one OrQuery
subquery_parts = []
for part in parts + [u',']:
if part.endswith(u','):
# Ensure we can catch "foo, bar" as well as "foo , bar"
last_subquery_part = part[:-1]
if last_subquery_part:
subquery_parts.append(last_subquery_part)
# Parse the subquery in to a single AndQuery
# TODO: Avoid needlessly wrapping AndQueries containing 1 subquery?
query_parts.append(query_from_strings(
query.AndQuery, model_cls, prefixes, subquery_parts
))
del subquery_parts[:]
else:
# Sort parts (1) end in + or -, (2) don't have a field, and
# (3) consist of more than just the + or -.
if part.endswith((u'+', u'-')) \
and u':' not in part \
and len(part) > 1:
sort_parts.append(part)
else:
subquery_parts.append(part)
# Avoid needlessly wrapping single statements in an OR
q = query.OrQuery(query_parts) if len(query_parts) > 1 else query_parts[0]
s = sort_from_strings(model_cls, sort_parts)
return q, s
lib/beets/dbcore/types.py
Regular → Executable
+17 -10
View File
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2014, Adrian Sampson.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -14,8 +15,14 @@
"""Representation of type information for DBCore model fields.
"""
from __future__ import division, absolute_import, print_function
from . import query
from beets.util import str2bool
import six
if not six.PY2:
buffer = memoryview # sqlite won't accept memoryview in python 2
# Abstract base.
@@ -34,7 +41,7 @@ class Type(object):
"""The `Query` subclass to be used when querying the field.
"""
model_type = unicode
model_type = six.text_type
"""The Python type that is used to represent the value in the model.
The model is guaranteed to return a value of this type if the field
@@ -58,9 +65,9 @@ class Type(object):
if value is None:
value = u''
if isinstance(value, bytes):
value = value.decode('utf8', 'ignore')
value = value.decode('utf-8', 'ignore')
return unicode(value)
return six.text_type(value)
def parse(self, string):
"""Parse a (possibly human-written) string and return the
@@ -93,13 +100,13 @@ class Type(object):
http://www.sqlite.org/datatype3.html
https://docs.python.org/2/library/sqlite3.html#sqlite-and-python-types
Flexible fields have the type afinity `TEXT`. This means the
`sql_value` is either a `buffer` or a `unicode` object` and the
method must handle these in addition.
Flexible fields have the type affinity `TEXT`. This means the
`sql_value` is either a `buffer`/`memoryview` or a `unicode` object`
and the method must handle these in addition.
"""
if isinstance(sql_value, buffer):
sql_value = bytes(sql_value).decode('utf8', 'ignore')
if isinstance(sql_value, unicode):
sql_value = bytes(sql_value).decode('utf-8', 'ignore')
if isinstance(sql_value, six.text_type):
return self.parse(sql_value)
else:
return self.normalize(sql_value)
@@ -191,7 +198,7 @@ class Boolean(Type):
model_type = bool
def format(self, value):
return unicode(bool(value))
return six.text_type(bool(value))
def parse(self, string):
return str2bool(string)
lib/beets/importer.py
Regular → Executable
+381 -235
View File
File diff suppressed because it is too large Load Diff
lib/beets/library.py
Regular → Executable
+453 -178
View File
File diff suppressed because it is too large Load Diff
lib/beets/logging.py
Executable
+134
View File
@@ -0,0 +1,134 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
"""A drop-in replacement for the standard-library `logging` module that
allows {}-style log formatting on Python 2 and 3.
Provides everything the "logging" module does. The only difference is
that when getLogger(name) instantiates a logger that logger uses
{}-style formatting.
"""
from __future__ import division, absolute_import, print_function
from copy import copy
from logging import * # noqa
import subprocess
import threading
import six
def logsafe(val):
"""Coerce a potentially "problematic" value so it can be formatted
in a Unicode log string.
This works around a number of pitfalls when logging objects in
Python 2:
- Logging path names, which must be byte strings, requires
conversion for output.
- Some objects, including some exceptions, will crash when you call
`unicode(v)` while `str(v)` works fine. CalledProcessError is an
example.
"""
# Already Unicode.
if isinstance(val, six.text_type):
return val
# Bytestring: needs decoding.
elif isinstance(val, bytes):
# Blindly convert with UTF-8. Eventually, it would be nice to
# (a) only do this for paths, if they can be given a distinct
# type, and (b) warn the developer if they do this for other
# bytestrings.
return val.decode('utf-8', 'replace')
# A "problem" object: needs a workaround.
elif isinstance(val, subprocess.CalledProcessError):
try:
return six.text_type(val)
except UnicodeDecodeError:
# An object with a broken __unicode__ formatter. Use __str__
# instead.
return str(val).decode('utf-8', 'replace')
# Other objects are used as-is so field access, etc., still works in
# the format string.
else:
return val
class StrFormatLogger(Logger):
"""A version of `Logger` that uses `str.format`-style formatting
instead of %-style formatting.
"""
class _LogMessage(object):
def __init__(self, msg, args, kwargs):
self.msg = msg
self.args = args
self.kwargs = kwargs
def __str__(self):
args = [logsafe(a) for a in self.args]
kwargs = dict((k, logsafe(v)) for (k, v) in self.kwargs.items())
return self.msg.format(*args, **kwargs)
def _log(self, level, msg, args, exc_info=None, extra=None, **kwargs):
"""Log msg.format(*args, **kwargs)"""
m = self._LogMessage(msg, args, kwargs)
return super(StrFormatLogger, self)._log(level, m, (), exc_info, extra)
class ThreadLocalLevelLogger(Logger):
"""A version of `Logger` whose level is thread-local instead of shared.
"""
def __init__(self, name, level=NOTSET):
self._thread_level = threading.local()
self.default_level = NOTSET
super(ThreadLocalLevelLogger, self).__init__(name, level)
@property
def level(self):
try:
return self._thread_level.level
except AttributeError:
self._thread_level.level = self.default_level
return self.level
@level.setter
def level(self, value):
self._thread_level.level = value
def set_global_level(self, level):
"""Set the level on the current thread + the default value for all
threads.
"""
self.default_level = level
self.setLevel(level)
class BeetsLogger(ThreadLocalLevelLogger, StrFormatLogger):
pass
my_manager = copy(Logger.manager)
my_manager.loggerClass = BeetsLogger
def getLogger(name=None): # noqa
if name:
return my_manager.getLogger(name)
else:
return Logger.root
lib/beets/mediafile.py
Regular → Executable
+326 -190
View File
File diff suppressed because it is too large Load Diff
lib/beets/plugins.py
+129 -64
View File
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2013, Adrian Sampson.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -14,15 +15,19 @@
"""Support for beets plugins."""
import logging
import traceback
from __future__ import division, absolute_import, print_function
import inspect
import traceback
import re
from collections import defaultdict
from functools import wraps
import beets
from beets import logging
from beets import mediafile
import six
PLUGIN_NAMESPACE = 'beetsplug'
@@ -41,6 +46,23 @@ class PluginConflictException(Exception):
"""
class PluginLogFilter(logging.Filter):
"""A logging filter that identifies the plugin that emitted a log
message.
"""
def __init__(self, plugin):
self.prefix = u'{0}: '.format(plugin.name)
def filter(self, record):
if hasattr(record.msg, 'msg') and isinstance(record.msg.msg,
six.string_types):
# A _LogMessage from our hacked-up Logging replacement.
record.msg.msg = self.prefix + record.msg.msg
elif isinstance(record.msg, six.string_types):
record.msg = self.prefix + record.msg
return True
# Managing the plugins themselves.
class BeetsPlugin(object):
@@ -51,7 +73,6 @@ class BeetsPlugin(object):
def __init__(self, name=None):
"""Perform one-time plugin setup.
"""
self.import_stages = []
self.name = name or self.__module__.split('.')[-1]
self.config = beets.config[self.name]
if not self.template_funcs:
@@ -60,6 +81,12 @@ class BeetsPlugin(object):
self.template_fields = {}
if not self.album_template_fields:
self.album_template_fields = {}
self.import_stages = []
self._log = log.getChild(self.name)
self._log.setLevel(logging.NOTSET) # Use `beets` logger level.
if not any(isinstance(f, PluginLogFilter) for f in self._log.filters):
self._log.addFilter(PluginLogFilter(self))
def commands(self):
"""Should return a list of beets.ui.Subcommand objects for
@@ -67,6 +94,46 @@ class BeetsPlugin(object):
"""
return ()
def get_import_stages(self):
"""Return a list of functions that should be called as importer
pipelines stages.
The callables are wrapped versions of the functions in
`self.import_stages`. Wrapping provides some bookkeeping for the
plugin: specifically, the logging level is adjusted to WARNING.
"""
return [self._set_log_level_and_params(logging.WARNING, import_stage)
for import_stage in self.import_stages]
def _set_log_level_and_params(self, base_log_level, func):
"""Wrap `func` to temporarily set this plugin's logger level to
`base_log_level` + config options (and restore it to its previous
value after the function returns). Also determines which params may not
be sent for backwards-compatibility.
"""
argspec = inspect.getargspec(func)
@wraps(func)
def wrapper(*args, **kwargs):
assert self._log.level == logging.NOTSET
verbosity = beets.config['verbose'].get(int)
log_level = max(logging.DEBUG, base_log_level - 10 * verbosity)
self._log.setLevel(log_level)
try:
try:
return func(*args, **kwargs)
except TypeError as exc:
if exc.args[0].startswith(func.__name__):
# caused by 'func' and not stuff internal to 'func'
kwargs = dict((arg, val) for arg, val in kwargs.items()
if arg in argspec.args)
return func(*args, **kwargs)
else:
raise
finally:
self._log.setLevel(logging.NOTSET)
return wrapper
def queries(self):
"""Should return a dict mapping prefixes to Query subclasses.
"""
@@ -123,37 +190,21 @@ class BeetsPlugin(object):
mediafile.MediaFile.add_field(name, descriptor)
library.Item._media_fields.add(name)
_raw_listeners = None
listeners = None
@classmethod
def register_listener(cls, event, func):
"""Add a function as a listener for the specified event. (An
imperative alternative to the @listen decorator.)
def register_listener(self, event, func):
"""Add a function as a listener for the specified event.
"""
if cls.listeners is None:
wrapped_func = self._set_log_level_and_params(logging.WARNING, func)
cls = self.__class__
if cls.listeners is None or cls._raw_listeners is None:
cls._raw_listeners = defaultdict(list)
cls.listeners = defaultdict(list)
cls.listeners[event].append(func)
@classmethod
def listen(cls, event):
"""Decorator that adds a function as an event handler for the
specified event (as a string). The parameters passed to function
will vary depending on what event occurred.
The function should respond to named parameters.
function(**kwargs) will trap all arguments in a dictionary.
Example:
>>> @MyPlugin.listen("imported")
>>> def importListener(**kwargs):
... pass
"""
def helper(func):
if cls.listeners is None:
cls.listeners = defaultdict(list)
cls.listeners[event].append(func)
return func
return helper
if func not in cls._raw_listeners[event]:
cls._raw_listeners[event].append(func)
cls.listeners[event].append(wrapped_func)
template_funcs = None
template_fields = None
@@ -197,14 +248,14 @@ def load_plugins(names=()):
BeetsPlugin subclasses desired.
"""
for name in names:
modname = '%s.%s' % (PLUGIN_NAMESPACE, name)
modname = '{0}.{1}'.format(PLUGIN_NAMESPACE, name)
try:
try:
namespace = __import__(modname, None, None)
except ImportError as exc:
# Again, this is hacky:
if exc.args[0].endswith(' ' + name):
log.warn(u'** plugin {0} not found'.format(name))
log.warning(u'** plugin {0} not found', name)
else:
raise
else:
@@ -214,8 +265,11 @@ def load_plugins(names=()):
_classes.add(obj)
except:
log.warn(u'** error loading plugin {0}'.format(name))
log.warn(traceback.format_exc())
log.warning(
u'** error loading plugin {}:\n{}',
name,
traceback.format_exc(),
)
_instances = {}
@@ -267,8 +321,8 @@ def types(model_cls):
if field in types and plugin_types[field] != types[field]:
raise PluginConflictException(
u'Plugin {0} defines flexible field {1} '
'which has already been defined with '
'another type.'.format(plugin.name, field)
u'which has already been defined with '
u'another type.'.format(plugin.name, field)
)
types.update(plugin_types)
return types
@@ -297,41 +351,35 @@ def album_distance(items, album_info, mapping):
def candidates(items, artist, album, va_likely):
"""Gets MusicBrainz candidates for an album from each plugin.
"""
out = []
for plugin in find_plugins():
out.extend(plugin.candidates(items, artist, album, va_likely))
return out
for candidate in plugin.candidates(items, artist, album, va_likely):
yield candidate
def item_candidates(item, artist, title):
"""Gets MusicBrainz candidates for an item from the plugins.
"""
out = []
for plugin in find_plugins():
out.extend(plugin.item_candidates(item, artist, title))
return out
for item_candidate in plugin.item_candidates(item, artist, title):
yield item_candidate
def album_for_id(album_id):
"""Get AlbumInfo objects for a given ID string.
"""
out = []
for plugin in find_plugins():
res = plugin.album_for_id(album_id)
if res:
out.append(res)
return out
album = plugin.album_for_id(album_id)
if album:
yield album
def track_for_id(track_id):
"""Get TrackInfo objects for a given ID string.
"""
out = []
for plugin in find_plugins():
res = plugin.track_for_id(track_id)
if res:
out.append(res)
return out
track = plugin.track_for_id(track_id)
if track:
yield track
def template_funcs():
@@ -349,8 +397,7 @@ def import_stages():
"""Get a list of import stage functions defined by plugins."""
stages = []
for plugin in find_plugins():
if hasattr(plugin, 'import_stages'):
stages += plugin.import_stages
stages += plugin.get_import_stages()
return stages
@@ -392,18 +439,20 @@ def event_handlers():
def send(event, **arguments):
"""Sends an event to all assigned event listeners. Event is the
name of the event to send, all other named arguments go to the
event handler(s).
"""Send an event to all assigned event listeners.
Returns a list of return values from the handlers.
`event` is the name of the event to send, all other named arguments
are passed along to the handlers.
Return a list of non-None values returned from the handlers.
"""
log.debug(u'Sending event: {0}'.format(event))
log.debug(u'Sending event: {0}', event)
results = []
for handler in event_handlers()[event]:
# Don't break legacy plugins if we want to pass more arguments
argspec = inspect.getargspec(handler).args
args = dict((k, v) for k, v in arguments.items() if k in argspec)
handler(**args)
result = handler(**arguments)
if result is not None:
results.append(result)
return results
def feat_tokens(for_artist=True):
@@ -433,3 +482,19 @@ def sanitize_choices(choices, choices_all):
if not (s in seen or seen.add(s)):
res.extend(list(others) if s == '*' else [s])
return res
def notify_info_yielded(event):
"""Makes a generator send the event 'event' every time it yields.
This decorator is supposed to decorate a generator, but any function
returning an iterable should work.
Each yielded value is passed to plugins using the 'info' parameter of
'send'.
"""
def decorator(generator):
def decorated(*args, **kwargs):
for v in generator(*args, **kwargs):
send(event, info=v)
yield v
return decorated
return decorator
lib/beets/ui/__init__.py
Regular → Executable
+462 -184
View File
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2014, Adrian Sampson.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -16,28 +17,31 @@
interface. To invoke the CLI, just call beets.ui.main(). The actual
CLI commands are implemented in the ui.commands module.
"""
from __future__ import print_function
import locale
from __future__ import division, absolute_import, print_function
import optparse
import textwrap
import sys
from difflib import SequenceMatcher
import logging
import sqlite3
import errno
import re
import struct
import traceback
import os.path
from six.moves import input
from beets import logging
from beets import library
from beets import plugins
from beets import util
from beets.util.functemplate import Template
from beets import config
from beets.util import confit
from beets.util import confit, as_string
from beets.autotag import mb
from beets.dbcore import query as db_query
import six
# On Windows platforms, use colorama to support "ANSI" terminal colors.
if sys.platform == 'win32':
@@ -56,8 +60,8 @@ log.propagate = False # Don't propagate to root handler.
PF_KEY_QUERIES = {
'comp': 'comp:true',
'singleton': 'singleton:true',
'comp': u'comp:true',
'singleton': u'singleton:true',
}
@@ -67,68 +71,149 @@ class UserError(Exception):
"""
# Utilities.
# Encoding utilities.
def _encoding():
"""Tries to guess the encoding used by the terminal."""
def _in_encoding():
"""Get the encoding to use for *inputting* strings from the console.
"""
return _stream_encoding(sys.stdin)
def _out_encoding():
"""Get the encoding to use for *outputting* strings to the console.
"""
return _stream_encoding(sys.stdout)
def _stream_encoding(stream, default='utf-8'):
"""A helper for `_in_encoding` and `_out_encoding`: get the stream's
preferred encoding, using a configured override or a default
fallback if neither is not specified.
"""
# Configured override?
encoding = config['terminal_encoding'].get()
if encoding:
return encoding
# Determine from locale settings.
try:
return locale.getdefaultlocale()[1] or 'utf8'
except ValueError:
# Invalid locale environment variable setting. To avoid
# failing entirely for no good reason, assume UTF-8.
return 'utf8'
# For testing: When sys.stdout or sys.stdin is a StringIO under the
# test harness, it doesn't have an `encoding` attribute. Just use
# UTF-8.
if not hasattr(stream, 'encoding'):
return default
# Python's guessed output stream encoding, or UTF-8 as a fallback
# (e.g., when piped to a file).
return stream.encoding or default
def decargs(arglist):
"""Given a list of command-line argument bytestrings, attempts to
decode them to Unicode strings.
decode them to Unicode strings when running under Python 2.
"""
return [s.decode(_encoding()) for s in arglist]
if six.PY2:
return [s.decode(util.arg_encoding()) for s in arglist]
else:
return arglist
def print_(*strings):
def print_(*strings, **kwargs):
"""Like print, but rather than raising an error when a character
is not in the terminal's encoding's character set, just silently
replaces it.
"""
if strings:
if isinstance(strings[0], unicode):
txt = u' '.join(strings)
else:
txt = ' '.join(strings)
else:
txt = u''
if isinstance(txt, unicode):
txt = txt.encode(_encoding(), 'replace')
print(txt)
The arguments must be Unicode strings: `unicode` on Python 2; `str` on
Python 3.
The `end` keyword argument behaves similarly to the built-in `print`
(it defaults to a newline).
"""
if not strings:
strings = [u'']
assert isinstance(strings[0], six.text_type)
txt = u' '.join(strings)
txt += kwargs.get('end', u'\n')
# Encode the string and write it to stdout.
if six.PY2:
# On Python 2, sys.stdout expects bytes.
out = txt.encode(_out_encoding(), 'replace')
sys.stdout.write(out)
else:
# On Python 3, sys.stdout expects text strings and uses the
# exception-throwing encoding error policy. To avoid throwing
# errors and use our configurable encoding override, we use the
# underlying bytes buffer instead.
if hasattr(sys.stdout, 'buffer'):
out = txt.encode(_out_encoding(), 'replace')
sys.stdout.buffer.write(out)
else:
# In our test harnesses (e.g., DummyOut), sys.stdout.buffer
# does not exist. We instead just record the text string.
sys.stdout.write(txt)
# Configuration wrappers.
def _bool_fallback(a, b):
"""Given a boolean or None, return the original value or a fallback.
"""
if a is None:
assert isinstance(b, bool)
return b
else:
assert isinstance(a, bool)
return a
def should_write(write_opt=None):
"""Decide whether a command that updates metadata should also write
tags, using the importer configuration as the default.
"""
return _bool_fallback(write_opt, config['import']['write'].get(bool))
def should_move(move_opt=None):
"""Decide whether a command that updates metadata should also move
files when they're inside the library, using the importer
configuration as the default.
Specifically, commands should move files after metadata updates only
when the importer is configured *either* to move *or* to copy files.
They should avoid moving files when the importer is configured not
to touch any filenames.
"""
return _bool_fallback(
move_opt,
config['import']['move'].get(bool) or
config['import']['copy'].get(bool)
)
# Input prompts.
def input_(prompt=None):
"""Like `raw_input`, but decodes the result to a Unicode string.
"""Like `input`, but decodes the result to a Unicode string.
Raises a UserError if stdin is not available. The prompt is sent to
stdout rather than stderr. A printed between the prompt and the
input cursor.
"""
# raw_input incorrectly sends prompts to stderr, not stdout, so we
# use print() explicitly to display prompts.
# use print_() explicitly to display prompts.
# http://bugs.python.org/issue1927
if prompt:
if isinstance(prompt, unicode):
prompt = prompt.encode(_encoding(), 'replace')
print(prompt, end=' ')
print_(prompt, end=u' ')
try:
resp = raw_input()
resp = input()
except EOFError:
raise UserError('stdin stream ended while input required')
raise UserError(u'stdin stream ended while input required')
return resp.decode(sys.stdin.encoding or 'utf8', 'ignore')
if six.PY2:
return resp.decode(_in_encoding(), 'ignore')
else:
return resp
def input_options(options, require=False, prompt=None, fallback_prompt=None,
@@ -172,7 +257,7 @@ def input_options(options, require=False, prompt=None, fallback_prompt=None,
found_letter = letter
break
else:
raise ValueError('no unambiguous lettering found')
raise ValueError(u'no unambiguous lettering found')
letters[found_letter.lower()] = option
index = option.index(found_letter)
@@ -180,7 +265,7 @@ def input_options(options, require=False, prompt=None, fallback_prompt=None,
# Mark the option's shortcut letter for display.
if not require and (
(default is None and not numrange and first) or
(isinstance(default, basestring) and
(isinstance(default, six.string_types) and
found_letter.lower() == default.lower())):
# The first option is the default; mark it.
show_letter = '[%s]' % found_letter.upper()
@@ -190,7 +275,7 @@ def input_options(options, require=False, prompt=None, fallback_prompt=None,
is_default = False
# Colorize the letter shortcut.
show_letter = colorize('turquoise' if is_default else 'blue',
show_letter = colorize('action_default' if is_default else 'action',
show_letter)
# Insert the highlighted letter back into the word.
@@ -216,11 +301,11 @@ def input_options(options, require=False, prompt=None, fallback_prompt=None,
prompt_part_lengths = []
if numrange:
if isinstance(default, int):
default_name = str(default)
default_name = colorize('turquoise', default_name)
default_name = six.text_type(default)
default_name = colorize('action_default', default_name)
tmpl = '# selection (default %s)'
prompt_parts.append(tmpl % default_name)
prompt_part_lengths.append(len(tmpl % str(default)))
prompt_part_lengths.append(len(tmpl % six.text_type(default)))
else:
prompt_parts.append('# selection')
prompt_part_lengths.append(len(prompt_parts[-1]))
@@ -255,9 +340,9 @@ def input_options(options, require=False, prompt=None, fallback_prompt=None,
# Make a fallback prompt too. This is displayed if the user enters
# something that is not recognized.
if not fallback_prompt:
fallback_prompt = 'Enter one of '
fallback_prompt = u'Enter one of '
if numrange:
fallback_prompt += '%i-%i, ' % numrange
fallback_prompt += u'%i-%i, ' % numrange
fallback_prompt += ', '.join(display_letters) + ':'
resp = input_(prompt)
@@ -296,19 +381,52 @@ def input_yn(prompt, require=False):
"yes" unless `require` is `True`, in which case there is no default.
"""
sel = input_options(
('y', 'n'), require, prompt, 'Enter Y or N:'
('y', 'n'), require, prompt, u'Enter Y or N:'
)
return sel == 'y'
return sel == u'y'
def input_select_objects(prompt, objs, rep):
"""Prompt to user to choose all, none, or some of the given objects.
Return the list of selected objects.
`prompt` is the prompt string to use for each question (it should be
phrased as an imperative verb). `rep` is a function to call on each
object to print it out when confirming objects individually.
"""
choice = input_options(
(u'y', u'n', u's'), False,
u'%s? (Yes/no/select)' % prompt)
print() # Blank line.
if choice == u'y': # Yes.
return objs
elif choice == u's': # Select.
out = []
for obj in objs:
rep(obj)
if input_yn(u'%s? (yes/no)' % prompt, True):
out.append(obj)
print() # go to a new line
return out
else: # No.
return []
# Human output formatting.
def human_bytes(size):
"""Formats size, a number of bytes, in a human-readable way."""
suffices = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB', 'HB']
for suffix in suffices:
powers = [u'', u'K', u'M', u'G', u'T', u'P', u'E', u'Z', u'Y', u'H']
unit = 'B'
for power in powers:
if size < 1024:
return "%3.1f %s" % (size, suffix)
return u"%3.1f %s%s" % (size, power, unit)
size /= 1024.0
return "big"
unit = u'iB'
return u"big"
def human_seconds(interval):
@@ -316,13 +434,13 @@ def human_seconds(interval):
interval using English words.
"""
units = [
(1, 'second'),
(60, 'minute'),
(60, 'hour'),
(24, 'day'),
(7, 'week'),
(52, 'year'),
(10, 'decade'),
(1, u'second'),
(60, u'minute'),
(60, u'hour'),
(24, u'day'),
(7, u'week'),
(52, u'year'),
(10, u'decade'),
]
for i in range(len(units) - 1):
increment, suffix = units[i]
@@ -335,7 +453,7 @@ def human_seconds(interval):
increment, suffix = units[-1]
interval /= float(increment)
return "%3.1f %ss" % (interval, suffix)
return u"%3.1f %ss" % (interval, suffix)
def human_seconds_short(interval):
@@ -346,16 +464,45 @@ def human_seconds_short(interval):
return u'%i:%02i' % (interval // 60, interval % 60)
# Colorization.
# ANSI terminal colorization code heavily inspired by pygments:
# http://dev.pocoo.org/hg/pygments-main/file/b2deea5b5030/pygments/console.py
# (pygments is by Tim Hatch, Armin Ronacher, et al.)
COLOR_ESCAPE = "\x1b["
DARK_COLORS = ["black", "darkred", "darkgreen", "brown", "darkblue",
"purple", "teal", "lightgray"]
LIGHT_COLORS = ["darkgray", "red", "green", "yellow", "blue",
"fuchsia", "turquoise", "white"]
DARK_COLORS = {
"black": 0,
"darkred": 1,
"darkgreen": 2,
"brown": 3,
"darkyellow": 3,
"darkblue": 4,
"purple": 5,
"darkmagenta": 5,
"teal": 6,
"darkcyan": 6,
"lightgray": 7
}
LIGHT_COLORS = {
"darkgray": 0,
"red": 1,
"green": 2,
"yellow": 3,
"blue": 4,
"fuchsia": 5,
"magenta": 5,
"turquoise": 6,
"cyan": 6,
"white": 7
}
RESET_COLOR = COLOR_ESCAPE + "39;49;00m"
# These abstract COLOR_NAMES are lazily mapped on to the actual color in COLORS
# as they are defined in the configuration files, see function: colorize
COLOR_NAMES = ['text_success', 'text_warning', 'text_error', 'text_highlight',
'text_highlight_minor', 'action_default', 'action']
COLORS = None
def _colorize(color, text):
"""Returns a string that prints the given text in the given color
@@ -363,34 +510,47 @@ def _colorize(color, text):
in DARK_COLORS or LIGHT_COLORS.
"""
if color in DARK_COLORS:
escape = COLOR_ESCAPE + "%im" % (DARK_COLORS.index(color) + 30)
escape = COLOR_ESCAPE + "%im" % (DARK_COLORS[color] + 30)
elif color in LIGHT_COLORS:
escape = COLOR_ESCAPE + "%i;01m" % (LIGHT_COLORS.index(color) + 30)
escape = COLOR_ESCAPE + "%i;01m" % (LIGHT_COLORS[color] + 30)
else:
raise ValueError('no such color %s', color)
raise ValueError(u'no such color %s', color)
return escape + text + RESET_COLOR
def colorize(color, text):
def colorize(color_name, text):
"""Colorize text if colored output is enabled. (Like _colorize but
conditional.)
"""
if config['color']:
if config['ui']['color']:
global COLORS
if not COLORS:
COLORS = dict((name,
config['ui']['colors'][name].as_str())
for name in COLOR_NAMES)
# In case a 3rd party plugin is still passing the actual color ('red')
# instead of the abstract color name ('text_error')
color = COLORS.get(color_name)
if not color:
log.debug(u'Invalid color_name: {0}', color_name)
color = color_name
return _colorize(color, text)
else:
return text
def _colordiff(a, b, highlight='red', minor_highlight='lightgray'):
def _colordiff(a, b, highlight='text_highlight',
minor_highlight='text_highlight_minor'):
"""Given two values, return the same pair of strings except with
their differences highlighted in the specified color. Strings are
highlighted intelligently to show differences; other values are
stringified and highlighted in their entirety.
"""
if not isinstance(a, basestring) or not isinstance(b, basestring):
if not isinstance(a, six.string_types) \
or not isinstance(b, six.string_types):
# Non-strings: use ordinary equality.
a = unicode(a)
b = unicode(b)
a = six.text_type(a)
b = six.text_type(b)
if a == b:
return a, b
else:
@@ -431,14 +591,14 @@ def _colordiff(a, b, highlight='red', minor_highlight='lightgray'):
return u''.join(a_out), u''.join(b_out)
def colordiff(a, b, highlight='red'):
def colordiff(a, b, highlight='text_highlight'):
"""Colorize differences between two values if color is enabled.
(Like _colordiff but conditional.)
"""
if config['color']:
if config['ui']['color']:
return _colordiff(a, b, highlight)
else:
return unicode(a), unicode(b)
return six.text_type(a), six.text_type(b)
def get_path_formats(subview=None):
@@ -449,7 +609,7 @@ def get_path_formats(subview=None):
subview = subview or config['paths']
for query, view in subview.items():
query = PF_KEY_QUERIES.get(query, query) # Expand common queries.
path_formats.append((query, Template(view.get(unicode))))
path_formats.append((query, Template(view.as_str())))
return path_formats
@@ -470,31 +630,6 @@ def get_replacements():
return replacements
def _pick_format(album, fmt=None):
"""Pick a format string for printing Album or Item objects,
falling back to config options and defaults.
"""
if fmt:
return fmt
if album:
return config['list_format_album'].get(unicode)
else:
return config['list_format_item'].get(unicode)
def print_obj(obj, lib, fmt=None):
"""Print an Album or Item object. If `fmt` is specified, use that
format string. Otherwise, use the configured template.
"""
album = isinstance(obj, library.Album)
fmt = _pick_format(album, fmt)
if isinstance(fmt, Template):
template = fmt
else:
template = Template(fmt)
print_(obj.evaluate_template(template))
def term_width():
"""Get the width (columns) of the terminal."""
fallback = config['ui']['terminal_width'].get(int)
@@ -542,10 +677,11 @@ def _field_diff(field, old, new):
# For strings, highlight changes. For others, colorize the whole
# thing.
if isinstance(oldval, basestring):
if isinstance(oldval, six.string_types):
oldstr, newstr = colordiff(oldval, newstr)
else:
oldstr, newstr = colorize('red', oldstr), colorize('red', newstr)
oldstr = colorize('text_error', oldstr)
newstr = colorize('text_error', newstr)
return u'{0} -> {1}'.format(oldstr, newstr)
@@ -581,18 +717,178 @@ def show_model_changes(new, old=None, fields=None, always=False):
changes.append(u' {0}: {1}'.format(
field,
colorize('red', new.formatted()[field])
colorize('text_highlight', new.formatted()[field])
))
# Print changes.
if changes or always:
print_obj(old, old._db)
print_(format(old))
if changes:
print_(u'\n'.join(changes))
return bool(changes)
def show_path_changes(path_changes):
"""Given a list of tuples (source, destination) that indicate the
path changes, log the changes as INFO-level output to the beets log.
The output is guaranteed to be unicode.
Every pair is shown on a single line if the terminal width permits it,
else it is split over two lines. E.g.,
Source -> Destination
vs.
Source
-> Destination
"""
sources, destinations = zip(*path_changes)
# Ensure unicode output
sources = list(map(util.displayable_path, sources))
destinations = list(map(util.displayable_path, destinations))
# Calculate widths for terminal split
col_width = (term_width() - len(' -> ')) // 2
max_width = len(max(sources + destinations, key=len))
if max_width > col_width:
# Print every change over two lines
for source, dest in zip(sources, destinations):
log.info(u'{0} \n -> {1}', source, dest)
else:
# Print every change on a single line, and add a header
title_pad = max_width - len('Source ') + len(' -> ')
log.info(u'Source {0} Destination', ' ' * title_pad)
for source, dest in zip(sources, destinations):
pad = max_width - len(source)
log.info(u'{0} {1} -> {2}', source, ' ' * pad, dest)
class CommonOptionsParser(optparse.OptionParser, object):
"""Offers a simple way to add common formatting options.
Options available include:
- matching albums instead of tracks: add_album_option()
- showing paths instead of items/albums: add_path_option()
- changing the format of displayed items/albums: add_format_option()
The last one can have several behaviors:
- against a special target
- with a certain format
- autodetected target with the album option
Each method is fully documented in the related method.
"""
def __init__(self, *args, **kwargs):
super(CommonOptionsParser, self).__init__(*args, **kwargs)
self._album_flags = False
# this serves both as an indicator that we offer the feature AND allows
# us to check whether it has been specified on the CLI - bypassing the
# fact that arguments may be in any order
def add_album_option(self, flags=('-a', '--album')):
"""Add a -a/--album option to match albums instead of tracks.
If used then the format option can auto-detect whether we're setting
the format for items or albums.
Sets the album property on the options extracted from the CLI.
"""
album = optparse.Option(*flags, action='store_true',
help=u'match albums instead of tracks')
self.add_option(album)
self._album_flags = set(flags)
def _set_format(self, option, opt_str, value, parser, target=None,
fmt=None, store_true=False):
"""Internal callback that sets the correct format while parsing CLI
arguments.
"""
if store_true:
setattr(parser.values, option.dest, True)
# Use the explicitly specified format, or the string from the option.
if fmt:
value = fmt
elif value:
value, = decargs([value])
else:
value = u''
parser.values.format = value
if target:
config[target._format_config_key].set(value)
else:
if self._album_flags:
if parser.values.album:
target = library.Album
else:
# the option is either missing either not parsed yet
if self._album_flags & set(parser.rargs):
target = library.Album
else:
target = library.Item
config[target._format_config_key].set(value)
else:
config[library.Item._format_config_key].set(value)
config[library.Album._format_config_key].set(value)
def add_path_option(self, flags=('-p', '--path')):
"""Add a -p/--path option to display the path instead of the default
format.
By default this affects both items and albums. If add_album_option()
is used then the target will be autodetected.
Sets the format property to u'$path' on the options extracted from the
CLI.
"""
path = optparse.Option(*flags, nargs=0, action='callback',
callback=self._set_format,
callback_kwargs={'fmt': u'$path',
'store_true': True},
help=u'print paths for matched items or albums')
self.add_option(path)
def add_format_option(self, flags=('-f', '--format'), target=None):
"""Add -f/--format option to print some LibModel instances with a
custom format.
`target` is optional and can be one of ``library.Item``, 'item',
``library.Album`` and 'album'.
Several behaviors are available:
- if `target` is given then the format is only applied to that
LibModel
- if the album option is used then the target will be autodetected
- otherwise the format is applied to both items and albums.
Sets the format property on the options extracted from the CLI.
"""
kwargs = {}
if target:
if isinstance(target, six.string_types):
target = {'item': library.Item,
'album': library.Album}[target]
kwargs['target'] = target
opt = optparse.Option(*flags, action='callback',
callback=self._set_format,
callback_kwargs=kwargs,
help=u'print with custom format')
self.add_option(opt)
def add_all_common_options(self):
"""Add album, path and format options.
"""
self.add_album_option()
self.add_path_option()
self.add_format_option()
# Subcommand parsing infrastructure.
#
# This is a fairly generic subcommand parser for optparse. It is
@@ -610,10 +906,10 @@ class Subcommand(object):
the subcommand; aliases are alternate names. parser is an
OptionParser responsible for parsing the subcommand's options.
help is a short description of the command. If no parser is
given, it defaults to a new, empty OptionParser.
given, it defaults to a new, empty CommonOptionsParser.
"""
self.name = name
self.parser = parser or optparse.OptionParser()
self.parser = parser or CommonOptionsParser()
self.aliases = aliases
self.help = help
self.hide = hide
@@ -632,11 +928,11 @@ class Subcommand(object):
@root_parser.setter
def root_parser(self, root_parser):
self._root_parser = root_parser
self.parser.prog = '{0} {1}'.format(root_parser.get_prog_name(),
self.name)
self.parser.prog = '{0} {1}'.format(
as_string(root_parser.get_prog_name()), self.name)
class SubcommandsOptionParser(optparse.OptionParser):
class SubcommandsOptionParser(CommonOptionsParser):
"""A variant of OptionParser that parses subcommands and their
arguments.
"""
@@ -648,13 +944,13 @@ class SubcommandsOptionParser(optparse.OptionParser):
"""
# A more helpful default usage.
if 'usage' not in kwargs:
kwargs['usage'] = """
kwargs['usage'] = u"""
%prog COMMAND [ARGS...]
%prog help COMMAND"""
kwargs['add_help_option'] = False
# Super constructor.
optparse.OptionParser.__init__(self, *args, **kwargs)
super(SubcommandsOptionParser, self).__init__(*args, **kwargs)
# Our root parser needs to stop on the first unrecognized argument.
self.disable_interspersed_args()
@@ -671,7 +967,7 @@ class SubcommandsOptionParser(optparse.OptionParser):
# Add the list of subcommands to the help message.
def format_help(self, formatter=None):
# Get the original help message, to which we will append.
out = optparse.OptionParser.format_help(self, formatter)
out = super(SubcommandsOptionParser, self).format_help(formatter)
if formatter is None:
formatter = self.formatter
@@ -711,7 +1007,8 @@ class SubcommandsOptionParser(optparse.OptionParser):
result.append(name)
help_width = formatter.width - help_position
help_lines = textwrap.wrap(subcommand.help, help_width)
result.append("%*s%s\n" % (indent_first, "", help_lines[0]))
help_line = help_lines[0] if help_lines else ''
result.append("%*s%s\n" % (indent_first, "", help_line))
result.extend(["%*s%s\n" % (help_position, "", line)
for line in help_lines[1:]])
formatter.dedent()
@@ -756,7 +1053,7 @@ class SubcommandsOptionParser(optparse.OptionParser):
cmdname = args.pop(0)
subcommand = self._subcommand_for_name(cmdname)
if not subcommand:
raise UserError("unknown command '{0}'".format(cmdname))
raise UserError(u"unknown command '{0}'".format(cmdname))
suboptions, subargs = subcommand.parse_args(args)
return subcommand, suboptions, subargs
@@ -765,53 +1062,24 @@ class SubcommandsOptionParser(optparse.OptionParser):
optparse.Option.ALWAYS_TYPED_ACTIONS += ('callback',)
def vararg_callback(option, opt_str, value, parser):
"""Callback for an option with variable arguments.
Manually collect arguments right of a callback-action
option (ie. with action="callback"), and add the resulting
list to the destination var.
Usage:
parser.add_option("-c", "--callback", dest="vararg_attr",
action="callback", callback=vararg_callback)
Details:
http://docs.python.org/2/library/optparse.html#callback-example-6-variable
-arguments
"""
value = [value]
def floatable(str):
try:
float(str)
return True
except ValueError:
return False
for arg in parser.rargs:
# stop on --foo like options
if arg[:2] == "--" and len(arg) > 2:
break
# stop on -a, but not on -3 or -3.0
if arg[:1] == "-" and len(arg) > 1 and not floatable(arg):
break
value.append(arg)
del parser.rargs[:len(value) - 1]
setattr(parser.values, option.dest, value)
# The main entry point and bootstrapping.
def _load_plugins(config):
"""Load the plugins specified in the configuration.
"""
paths = config['pluginpath'].get(confit.StrSeq(split=False))
paths = map(util.normpath, paths)
paths = config['pluginpath'].as_str_seq(split=False)
paths = [util.normpath(p) for p in paths]
log.debug(u'plugin paths: {0}', util.displayable_path(paths))
# On Python 3, the search paths need to be unicode.
paths = [util.py3_path(p) for p in paths]
# Extend the `beetsplug` package to include the plugin paths.
import beetsplug
beetsplug.__path__ = paths + beetsplug.__path__
# For backwards compatibility.
# For backwards compatibility, also support plugin paths that
# *contain* a `beetsplug` package.
sys.path += paths
plugins.load_plugins(config['plugins'].as_str_seq())
@@ -840,8 +1108,8 @@ def _setup(options, lib=None):
if lib is None:
lib = _open_library(config)
plugins.send("library_opened", lib=lib)
library.Item._types = plugins.types(library.Item)
library.Album._types = plugins.types(library.Album)
library.Item._types.update(plugins.types(library.Item))
library.Album._types.update(plugins.types(library.Album))
return subcommands, plugins, lib
@@ -859,28 +1127,28 @@ def _configure(options):
config.set_args(options)
# Configure the logger.
if config['verbose'].get(bool):
log.setLevel(logging.DEBUG)
if config['verbose'].get(int):
log.set_global_level(logging.DEBUG)
else:
log.setLevel(logging.INFO)
log.set_global_level(logging.INFO)
config_path = config.user_config_path()
if os.path.isfile(config_path):
log.debug(u'user configuration: {0}'.format(
util.displayable_path(config_path)))
log.debug(u'user configuration: {0}',
util.displayable_path(config_path))
else:
log.debug(u'no user configuration found at {0}'.format(
util.displayable_path(config_path)))
log.debug(u'no user configuration found at {0}',
util.displayable_path(config_path))
log.debug(u'data directory: {0}'
.format(util.displayable_path(config.config_dir())))
log.debug(u'data directory: {0}',
util.displayable_path(config.config_dir()))
return config
def _open_library(config):
"""Create a new library instance from the configuration.
"""
dbpath = config['library'].as_filename()
dbpath = util.bytestring_path(config['library'].as_filename())
try:
lib = library.Library(
dbpath,
@@ -890,14 +1158,14 @@ def _open_library(config):
)
lib.get_item(0) # Test database connection.
except (sqlite3.OperationalError, sqlite3.DatabaseError):
log.debug(traceback.format_exc())
log.debug(u'{}', traceback.format_exc())
raise UserError(u"database file {0} could not be opened".format(
util.displayable_path(dbpath)
))
log.debug(u'library database: {0}\n'
u'library directory: {1}'
.format(util.displayable_path(lib.path),
util.displayable_path(lib.directory)))
u'library directory: {1}',
util.displayable_path(lib.path),
util.displayable_path(lib.directory))
return lib
@@ -906,16 +1174,18 @@ def _raw_main(args, lib=None):
handling.
"""
parser = SubcommandsOptionParser()
parser.add_format_option(flags=('--format-item',), target=library.Item)
parser.add_format_option(flags=('--format-album',), target=library.Album)
parser.add_option('-l', '--library', dest='library',
help='library database file to use')
help=u'library database file to use')
parser.add_option('-d', '--directory', dest='directory',
help="destination music directory")
parser.add_option('-v', '--verbose', dest='verbose', action='store_true',
help='print debugging information')
help=u"destination music directory")
parser.add_option('-v', '--verbose', dest='verbose', action='count',
help=u'log more details (use twice for even more)')
parser.add_option('-c', '--config', dest='config',
help='path to configuration file')
help=u'path to configuration file')
parser.add_option('-h', '--help', dest='help', action='store_true',
help='how this help message and exit')
help=u'show this help message and exit')
parser.add_option('--version', dest='version', action='store_true',
help=optparse.SUPPRESS_HELP)
@@ -924,10 +1194,12 @@ def _raw_main(args, lib=None):
# Special case for the `config --edit` command: bypass _setup so
# that an invalid configuration does not prevent the editor from
# starting.
if subargs[0] == 'config' and ('-e' in subargs or '--edit' in subargs):
if subargs and subargs[0] == 'config' \
and ('-e' in subargs or '--edit' in subargs):
from beets.ui.commands import config_edit
return config_edit()
test_lib = bool(lib)
subcommands, plugins, lib = _setup(options, lib)
parser.add_subcommand(*subcommands)
@@ -935,6 +1207,9 @@ def _raw_main(args, lib=None):
subcommand.func(lib, suboptions, subargs)
plugins.send('cli_exit', lib=lib)
if not test_lib:
# Clean up the library unless it came from the test harness.
lib._close()
def main(args=None):
@@ -945,7 +1220,7 @@ def main(args=None):
_raw_main(args)
except UserError as exc:
message = exc.args[0] if exc.args else None
log.error(u'error: {0}'.format(message))
log.error(u'error: {0}', message)
sys.exit(1)
except util.HumanReadableException as exc:
exc.log(log)
@@ -953,11 +1228,14 @@ def main(args=None):
except library.FileOperationError as exc:
# These errors have reasonable human-readable descriptions, but
# we still want to log their tracebacks for debugging.
log.debug(traceback.format_exc())
log.error(exc)
log.debug('{}', traceback.format_exc())
log.error('{}', exc)
sys.exit(1)
except confit.ConfigError as exc:
log.error(u'configuration error: {0}'.format(exc))
log.error(u'configuration error: {0}', exc)
sys.exit(1)
except db_query.InvalidQueryError as exc:
log.error(u'invalid query: {0}', exc)
sys.exit(1)
except IOError as exc:
if exc.errno == errno.EPIPE:
@@ -967,4 +1245,4 @@ def main(args=None):
raise
except KeyboardInterrupt:
# Silently ignore ^C except in verbose mode.
log.debug(traceback.format_exc())
log.debug(u'{}', traceback.format_exc())
lib/beets/ui/commands.py
Regular → Executable
+600 -493
View File
File diff suppressed because it is too large Load Diff
lib/beets/ui/completion_base.sh
Regular → Executable
View File
lib/beets/util/__init__.py
Regular → Executable
+415 -108
View File
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2013, Adrian Sampson.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -13,21 +14,28 @@
# included in all copies or substantial portions of the Software.
"""Miscellaneous utility functions."""
from __future__ import division
from __future__ import division, absolute_import, print_function
import os
import sys
import errno
import locale
import re
import shutil
import fnmatch
from collections import defaultdict
from collections import Counter
import traceback
import subprocess
import platform
import shlex
from beets.util import hidden
import six
from unidecode import unidecode
MAX_FILENAME_LENGTH = 200
WINDOWS_MAGIC_PREFIX = u'\\\\?\\'
SNI_SUPPORTED = sys.version_info >= (2, 7, 9)
class HumanReadableException(Exception):
@@ -54,22 +62,22 @@ class HumanReadableException(Exception):
def _gerund(self):
"""Generate a (likely) gerund form of the English verb.
"""
if ' ' in self.verb:
if u' ' in self.verb:
return self.verb
gerund = self.verb[:-1] if self.verb.endswith('e') else self.verb
gerund += 'ing'
gerund = self.verb[:-1] if self.verb.endswith(u'e') else self.verb
gerund += u'ing'
return gerund
def _reasonstr(self):
"""Get the reason as a string."""
if isinstance(self.reason, unicode):
if isinstance(self.reason, six.text_type):
return self.reason
elif isinstance(self.reason, basestring): # Byte string.
return self.reason.decode('utf8', 'ignore')
elif isinstance(self.reason, bytes):
return self.reason.decode('utf-8', 'ignore')
elif hasattr(self.reason, 'strerror'): # i.e., EnvironmentError
return self.reason.strerror
else:
return u'"{0}"'.format(unicode(self.reason))
return u'"{0}"'.format(six.text_type(self.reason))
def get_message(self):
"""Create the human-readable description of the error, sans
@@ -83,7 +91,7 @@ class HumanReadableException(Exception):
"""
if self.tb:
logger.debug(self.tb)
logger.error(u'{0}: {1}'.format(self.error_kind, self.args[0]))
logger.error(u'{0}: {1}', self.error_kind, self.args[0])
class FilesystemError(HumanReadableException):
@@ -149,21 +157,22 @@ def ancestry(path):
return out
def sorted_walk(path, ignore=(), logger=None):
def sorted_walk(path, ignore=(), ignore_hidden=False, logger=None):
"""Like `os.walk`, but yields things in case-insensitive sorted,
breadth-first order. Directory and file names matching any glob
pattern in `ignore` are skipped. If `logger` is provided, then
warning messages are logged there when a directory cannot be listed.
"""
# Make sure the path isn't a Unicode string.
# Make sure the pathes aren't Unicode strings.
path = bytestring_path(path)
ignore = [bytestring_path(i) for i in ignore]
# Get all the directories and files at this level.
try:
contents = os.listdir(syspath(path))
except OSError as exc:
if logger:
logger.warn(u'could not list directory {0}: {1}'.format(
logger.warning(u'could not list directory {0}: {1}'.format(
displayable_path(path), exc.strerror
))
return
@@ -183,10 +192,11 @@ def sorted_walk(path, ignore=(), logger=None):
# Add to output as either a file or a directory.
cur = os.path.join(path, base)
if os.path.isdir(syspath(cur)):
dirs.append(base)
else:
files.append(base)
if (ignore_hidden and not hidden.is_hidden(cur)) or not ignore_hidden:
if os.path.isdir(syspath(cur)):
dirs.append(base)
else:
files.append(base)
# Sort lists (case-insensitive) and yield the current level.
dirs.sort(key=bytes.lower)
@@ -197,7 +207,7 @@ def sorted_walk(path, ignore=(), logger=None):
for base in dirs:
cur = os.path.join(path, base)
# yield from sorted_walk(...)
for res in sorted_walk(cur, ignore, logger):
for res in sorted_walk(cur, ignore, ignore_hidden, logger):
yield res
@@ -260,7 +270,9 @@ def prune_dirs(path, root=None, clutter=('.DS_Store', 'Thumbs.db')):
if not os.path.exists(directory):
# Directory gone already.
continue
if fnmatch_all(os.listdir(directory), clutter):
clutter = [bytestring_path(c) for c in clutter]
match_paths = [bytestring_path(d) for d in os.listdir(directory)]
if fnmatch_all(match_paths, clutter):
# Directory contains only clutter (or nothing).
try:
shutil.rmtree(directory)
@@ -294,6 +306,18 @@ def components(path):
return comps
def arg_encoding():
"""Get the encoding for command-line arguments (and other OS
locale-sensitive strings).
"""
try:
return locale.getdefaultlocale()[1] or 'utf-8'
except ValueError:
# Invalid locale environment variable setting. To avoid
# failing entirely for no good reason, assume UTF-8.
return 'utf-8'
def _fsencoding():
"""Get the system's filesystem encoding. On Windows, this is always
UTF-8 (not MBCS).
@@ -305,16 +329,16 @@ def _fsencoding():
# for Windows paths, so the encoding is actually immaterial so
# we can avoid dealing with this nastiness. We arbitrarily
# choose UTF-8.
encoding = 'utf8'
encoding = 'utf-8'
return encoding
def bytestring_path(path):
"""Given a path, which is either a str or a unicode, returns a str
"""Given a path, which is either a bytes or a unicode, returns a str
path (ensuring that we never deal with Unicode pathnames).
"""
# Pass through bytestrings.
if isinstance(path, str):
if isinstance(path, bytes):
return path
# On Windows, remove the magic prefix added by `syspath`. This makes
@@ -323,11 +347,14 @@ def bytestring_path(path):
if os.path.__name__ == 'ntpath' and path.startswith(WINDOWS_MAGIC_PREFIX):
path = path[len(WINDOWS_MAGIC_PREFIX):]
# Try to encode with default encodings, but fall back to UTF8.
# Try to encode with default encodings, but fall back to utf-8.
try:
return path.encode(_fsencoding())
except (UnicodeError, LookupError):
return path.encode('utf8')
return path.encode('utf-8')
PATH_SEP = bytestring_path(os.sep)
def displayable_path(path, separator=u'; '):
@@ -337,16 +364,16 @@ def displayable_path(path, separator=u'; '):
"""
if isinstance(path, (list, tuple)):
return separator.join(displayable_path(p) for p in path)
elif isinstance(path, unicode):
elif isinstance(path, six.text_type):
return path
elif not isinstance(path, str):
elif not isinstance(path, bytes):
# A non-string object: just get its unicode representation.
return unicode(path)
return six.text_type(path)
try:
return path.decode(_fsencoding(), 'ignore')
except (UnicodeError, LookupError):
return path.decode('utf8', 'ignore')
return path.decode('utf-8', 'ignore')
def syspath(path, prefix=True):
@@ -360,12 +387,12 @@ def syspath(path, prefix=True):
if os.path.__name__ != 'ntpath':
return path
if not isinstance(path, unicode):
if not isinstance(path, six.text_type):
# Beets currently represents Windows paths internally with UTF-8
# arbitrarily. But earlier versions used MBCS because it is
# reported as the FS encoding by Windows. Try both.
try:
path = path.decode('utf8')
path = path.decode('utf-8')
except UnicodeError:
# The encoding should always be MBCS, Windows' broken
# Unicode representation.
@@ -412,7 +439,7 @@ def copy(path, dest, replace=False):
path = syspath(path)
dest = syspath(dest)
if not replace and os.path.exists(dest):
raise FilesystemError('file exists', 'copy', (path, dest))
raise FilesystemError(u'file exists', 'copy', (path, dest))
try:
shutil.copyfile(path, dest)
except (OSError, IOError) as exc:
@@ -433,8 +460,7 @@ def move(path, dest, replace=False):
path = syspath(path)
dest = syspath(dest)
if os.path.exists(dest) and not replace:
raise FilesystemError('file exists', 'rename', (path, dest),
traceback.format_exc())
raise FilesystemError(u'file exists', 'rename', (path, dest))
# First, try renaming the file.
try:
@@ -452,23 +478,52 @@ def move(path, dest, replace=False):
def link(path, dest, replace=False):
"""Create a symbolic link from path to `dest`. Raises an OSError if
`dest` already exists, unless `replace` is True. Does nothing if
`path` == `dest`."""
if (samefile(path, dest)):
`path` == `dest`.
"""
if samefile(path, dest):
return
path = syspath(path)
dest = syspath(dest)
if os.path.exists(dest) and not replace:
raise FilesystemError('file exists', 'rename', (path, dest),
traceback.format_exc())
if os.path.exists(syspath(dest)) and not replace:
raise FilesystemError(u'file exists', 'rename', (path, dest))
try:
os.symlink(path, dest)
except OSError:
raise FilesystemError('Operating system does not support symbolic '
'links.', 'link', (path, dest),
os.symlink(syspath(path), syspath(dest))
except NotImplementedError:
# raised on python >= 3.2 and Windows versions before Vista
raise FilesystemError(u'OS does not support symbolic links.'
'link', (path, dest), traceback.format_exc())
except OSError as exc:
# TODO: Windows version checks can be removed for python 3
if hasattr('sys', 'getwindowsversion'):
if sys.getwindowsversion()[0] < 6: # is before Vista
exc = u'OS does not support symbolic links.'
raise FilesystemError(exc, 'link', (path, dest),
traceback.format_exc())
def hardlink(path, dest, replace=False):
"""Create a hard link from path to `dest`. Raises an OSError if
`dest` already exists, unless `replace` is True. Does nothing if
`path` == `dest`.
"""
if samefile(path, dest):
return
if os.path.exists(syspath(dest)) and not replace:
raise FilesystemError(u'file exists', 'rename', (path, dest))
try:
os.link(syspath(path), syspath(dest))
except NotImplementedError:
raise FilesystemError(u'OS does not support hard links.'
'link', (path, dest), traceback.format_exc())
except OSError as exc:
if exc.errno == errno.EXDEV:
raise FilesystemError(u'Cannot hard link across devices.'
'link', (path, dest), traceback.format_exc())
else:
raise FilesystemError(exc, 'link', (path, dest),
traceback.format_exc())
def unique_path(path):
"""Returns a version of ``path`` that does not exist on the
filesystem. Specifically, if ``path` itself already exists, then
@@ -478,7 +533,7 @@ def unique_path(path):
return path
base, ext = os.path.splitext(path)
match = re.search(r'\.(\d)+$', base)
match = re.search(br'\.(\d)+$', base)
if match:
num = int(match.group(1))
base = base[:match.start()]
@@ -486,7 +541,8 @@ def unique_path(path):
num = 0
while True:
num += 1
new_path = '%s.%i%s' % (base, num, ext)
suffix = u'.{}'.format(num).encode() + ext
new_path = base + suffix
if not os.path.exists(new_path):
return new_path
@@ -495,12 +551,12 @@ def unique_path(path):
# shares, which are sufficiently common as to cause frequent problems.
# http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247.aspx
CHAR_REPLACE = [
(re.compile(ur'[\\/]'), u'_'), # / and \ -- forbidden everywhere.
(re.compile(ur'^\.'), u'_'), # Leading dot (hidden files on Unix).
(re.compile(ur'[\x00-\x1f]'), u''), # Control characters.
(re.compile(ur'[<>:"\?\*\|]'), u'_'), # Windows "reserved characters".
(re.compile(ur'\.$'), u'_'), # Trailing dots.
(re.compile(ur'\s+$'), u''), # Trailing whitespace.
(re.compile(r'[\\/]'), u'_'), # / and \ -- forbidden everywhere.
(re.compile(r'^\.'), u'_'), # Leading dot (hidden files on Unix).
(re.compile(r'[\x00-\x1f]'), u''), # Control characters.
(re.compile(r'[<>:"\?\*\|]'), u'_'), # Windows "reserved characters".
(re.compile(r'\.$'), u'_'), # Trailing dots.
(re.compile(r'\s+$'), u''), # Trailing whitespace.
]
@@ -542,73 +598,142 @@ def truncate_path(path, length=MAX_FILENAME_LENGTH):
return os.path.join(*out)
def _legalize_stage(path, replacements, length, extension, fragment):
"""Perform a single round of path legalization steps
(sanitation/replacement, encoding from Unicode to bytes,
extension-appending, and truncation). Return the path (Unicode if
`fragment` is set, `bytes` otherwise) and whether truncation was
required.
"""
# Perform an initial sanitization including user replacements.
path = sanitize_path(path, replacements)
# Encode for the filesystem.
if not fragment:
path = bytestring_path(path)
# Preserve extension.
path += extension.lower()
# Truncate too-long components.
pre_truncate_path = path
path = truncate_path(path, length)
return path, path != pre_truncate_path
def legalize_path(path, replacements, length, extension, fragment):
"""Given a path-like Unicode string, produce a legal path. Return
the path and a flag indicating whether some replacements had to be
ignored (see below).
The legalization process (see `_legalize_stage`) consists of
applying the sanitation rules in `replacements`, encoding the string
to bytes (unless `fragment` is set), truncating components to
`length`, appending the `extension`.
This function performs up to three calls to `_legalize_stage` in
case truncation conflicts with replacements (as can happen when
truncation creates whitespace at the end of the string, for
example). The limited number of iterations iterations avoids the
possibility of an infinite loop of sanitation and truncation
operations, which could be caused by replacement rules that make the
string longer. The flag returned from this function indicates that
the path has to be truncated twice (indicating that replacements
made the string longer again after it was truncated); the
application should probably log some sort of warning.
"""
if fragment:
# Outputting Unicode.
extension = extension.decode('utf-8', 'ignore')
first_stage_path, _ = _legalize_stage(
path, replacements, length, extension, fragment
)
# Convert back to Unicode with extension removed.
first_stage_path, _ = os.path.splitext(displayable_path(first_stage_path))
# Re-sanitize following truncation (including user replacements).
second_stage_path, retruncated = _legalize_stage(
first_stage_path, replacements, length, extension, fragment
)
# If the path was once again truncated, discard user replacements
# and run through one last legalization stage.
if retruncated:
second_stage_path, _ = _legalize_stage(
first_stage_path, None, length, extension, fragment
)
return second_stage_path, retruncated
def py3_path(path):
"""Convert a bytestring path to Unicode on Python 3 only. On Python
2, return the bytestring path unchanged.
This helps deal with APIs on Python 3 that *only* accept Unicode
(i.e., `str` objects). I philosophically disagree with this
decision, because paths are sadly bytes on Unix, but that's the way
it is. So this function helps us "smuggle" the true bytes data
through APIs that took Python 3's Unicode mandate too seriously.
"""
if isinstance(path, six.text_type):
return path
assert isinstance(path, bytes)
if six.PY2:
return path
return os.fsdecode(path)
def str2bool(value):
"""Returns a boolean reflecting a human-entered string."""
if value.lower() in ('yes', '1', 'true', 't', 'y'):
return True
else:
return False
return value.lower() in (u'yes', u'1', u'true', u't', u'y')
def as_string(value):
"""Convert a value to a Unicode object for matching with a query.
None becomes the empty string. Bytestrings are silently decoded.
"""
if six.PY2:
buffer_types = buffer, memoryview # noqa: F821
else:
buffer_types = memoryview
if value is None:
return u''
elif isinstance(value, buffer):
return str(value).decode('utf8', 'ignore')
elif isinstance(value, str):
return value.decode('utf8', 'ignore')
elif isinstance(value, buffer_types):
return bytes(value).decode('utf-8', 'ignore')
elif isinstance(value, bytes):
return value.decode('utf-8', 'ignore')
else:
return unicode(value)
return six.text_type(value)
def levenshtein(s1, s2):
"""A nice DP edit distance implementation from Wikibooks:
http://en.wikibooks.org/wiki/Algorithm_implementation/Strings/
Levenshtein_distance#Python
def text_string(value, encoding='utf-8'):
"""Convert a string, which can either be bytes or unicode, to
unicode.
Text (unicode) is left untouched; bytes are decoded. This is useful
to convert from a "native string" (bytes on Python 2, str on Python
3) to a consistently unicode value.
"""
if len(s1) < len(s2):
return levenshtein(s2, s1)
if not s1:
return len(s2)
previous_row = xrange(len(s2) + 1)
for i, c1 in enumerate(s1):
current_row = [i + 1]
for j, c2 in enumerate(s2):
insertions = previous_row[j + 1] + 1
deletions = current_row[j] + 1
substitutions = previous_row[j] + (c1 != c2)
current_row.append(min(insertions, deletions, substitutions))
previous_row = current_row
return previous_row[-1]
if isinstance(value, bytes):
return value.decode(encoding)
return value
def plurality(objs):
"""Given a sequence of comparable objects, returns the object that
is most common in the set and the frequency of that object. The
"""Given a sequence of hashble objects, returns the object that
is most common in the set and the its number of appearance. The
sequence must contain at least one object.
"""
# Calculate frequencies.
freqs = defaultdict(int)
for obj in objs:
freqs[obj] += 1
if not freqs:
raise ValueError('sequence must be non-empty')
# Find object with maximum frequency.
max_freq = 0
res = None
for obj, freq in freqs.items():
if freq > max_freq:
max_freq = freq
res = obj
return res, max_freq
c = Counter(objs)
if not c:
raise ValueError(u'sequence must be non-empty')
return c.most_common(1)[0]
def cpu_count():
@@ -624,8 +749,8 @@ def cpu_count():
num = 0
elif sys.platform == 'darwin':
try:
num = int(command_output(['sysctl', '-n', 'hw.ncpu']))
except ValueError:
num = int(command_output(['/usr/sbin/sysctl', '-n', 'hw.ncpu']))
except (ValueError, OSError, subprocess.CalledProcessError):
num = 0
else:
try:
@@ -638,21 +763,40 @@ def cpu_count():
return 1
def convert_command_args(args):
"""Convert command arguments to bytestrings on Python 2 and
surrogate-escaped strings on Python 3."""
assert isinstance(args, list)
def convert(arg):
if six.PY2:
if isinstance(arg, six.text_type):
arg = arg.encode(arg_encoding())
else:
if isinstance(arg, bytes):
arg = arg.decode(arg_encoding(), 'surrogateescape')
return arg
return [convert(a) for a in args]
def command_output(cmd, shell=False):
"""Runs the command and returns its output after it has exited.
``cmd`` is a list of arguments starting with the command names. If
``shell`` is true, ``cmd`` is assumed to be a string and passed to a
``cmd`` is a list of arguments starting with the command names. The
arguments are bytes on Unix and strings on Windows.
If ``shell`` is true, ``cmd`` is assumed to be a string and passed to a
shell to execute.
If the process exits with a non-zero return code
``subprocess.CalledProcessError`` is raised. May also raise
``OSError``.
This replaces `subprocess.check_output`, which isn't available in
Python 2.6 and which can have problems if lots of output is sent to
stderr.
This replaces `subprocess.check_output` which can have problems if lots of
output is sent to stderr.
"""
cmd = convert_command_args(cmd)
proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
@@ -665,6 +809,7 @@ def command_output(cmd, shell=False):
raise subprocess.CalledProcessError(
returncode=proc.returncode,
cmd=' '.join(cmd),
output=stdout + stderr,
)
return stdout
@@ -684,3 +829,165 @@ def max_filename_length(path, limit=MAX_FILENAME_LENGTH):
return min(res[9], limit)
else:
return limit
def open_anything():
"""Return the system command that dispatches execution to the correct
program.
"""
sys_name = platform.system()
if sys_name == 'Darwin':
base_cmd = 'open'
elif sys_name == 'Windows':
base_cmd = 'start'
else: # Assume Unix
base_cmd = 'xdg-open'
return base_cmd
def editor_command():
"""Get a command for opening a text file.
Use the `EDITOR` environment variable by default. If it is not
present, fall back to `open_anything()`, the platform-specific tool
for opening files in general.
"""
editor = os.environ.get('EDITOR')
if editor:
return editor
return open_anything()
def shlex_split(s):
"""Split a Unicode or bytes string according to shell lexing rules.
Raise `ValueError` if the string is not a well-formed shell string.
This is a workaround for a bug in some versions of Python.
"""
if not six.PY2 or isinstance(s, bytes): # Shlex works fine.
return shlex.split(s)
elif isinstance(s, six.text_type):
# Work around a Python bug.
# http://bugs.python.org/issue6988
bs = s.encode('utf-8')
return [c.decode('utf-8') for c in shlex.split(bs)]
else:
raise TypeError(u'shlex_split called with non-string')
def interactive_open(targets, command):
"""Open the files in `targets` by `exec`ing a new `command`, given
as a Unicode string. (The new program takes over, and Python
execution ends: this does not fork a subprocess.)
Can raise `OSError`.
"""
assert command
# Split the command string into its arguments.
try:
args = shlex_split(command)
except ValueError: # Malformed shell tokens.
args = [command]
args.insert(0, args[0]) # for argv[0]
args += targets
return os.execlp(*args)
def _windows_long_path_name(short_path):
"""Use Windows' `GetLongPathNameW` via ctypes to get the canonical,
long path given a short filename.
"""
if not isinstance(short_path, six.text_type):
short_path = short_path.decode(_fsencoding())
import ctypes
buf = ctypes.create_unicode_buffer(260)
get_long_path_name_w = ctypes.windll.kernel32.GetLongPathNameW
return_value = get_long_path_name_w(short_path, buf, 260)
if return_value == 0 or return_value > 260:
# An error occurred
return short_path
else:
long_path = buf.value
# GetLongPathNameW does not change the case of the drive
# letter.
if len(long_path) > 1 and long_path[1] == ':':
long_path = long_path[0].upper() + long_path[1:]
return long_path
def case_sensitive(path):
"""Check whether the filesystem at the given path is case sensitive.
To work best, the path should point to a file or a directory. If the path
does not exist, assume a case sensitive file system on every platform
except Windows.
"""
# A fallback in case the path does not exist.
if not os.path.exists(syspath(path)):
# By default, the case sensitivity depends on the platform.
return platform.system() != 'Windows'
# If an upper-case version of the path exists but a lower-case
# version does not, then the filesystem must be case-sensitive.
# (Otherwise, we have more work to do.)
if not (os.path.exists(syspath(path.lower())) and
os.path.exists(syspath(path.upper()))):
return True
# Both versions of the path exist on the file system. Check whether
# they refer to different files by their inodes. Alas,
# `os.path.samefile` is only available on Unix systems on Python 2.
if platform.system() != 'Windows':
return not os.path.samefile(syspath(path.lower()),
syspath(path.upper()))
# On Windows, we check whether the canonical, long filenames for the
# files are the same.
lower = _windows_long_path_name(path.lower())
upper = _windows_long_path_name(path.upper())
return lower != upper
def raw_seconds_short(string):
"""Formats a human-readable M:SS string as a float (number of seconds).
Raises ValueError if the conversion cannot take place due to `string` not
being in the right format.
"""
match = re.match(r'^(\d+):([0-5]\d)$', string)
if not match:
raise ValueError(u'String not in M:SS format')
minutes, seconds = map(int, match.groups())
return float(minutes * 60 + seconds)
def asciify_path(path, sep_replace):
"""Decodes all unicode characters in a path into ASCII equivalents.
Substitutions are provided by the unidecode module. Path separators in the
input are preserved.
Keyword arguments:
path -- The path to be asciified.
sep_replace -- the string to be used to replace extraneous path separators.
"""
# if this platform has an os.altsep, change it to os.sep.
if os.altsep:
path = path.replace(os.altsep, os.sep)
path_components = path.split(os.sep)
for index, item in enumerate(path_components):
path_components[index] = unidecode(item).replace(os.sep, sep_replace)
if os.altsep:
path_components[index] = unidecode(item).replace(
os.altsep,
sep_replace
)
return os.sep.join(path_components)
lib/beets/util/artresizer.py
Regular → Executable
+120 -69
View File
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2014, Fabrice Laporte
# Copyright 2016, Fabrice Laporte
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -15,20 +16,26 @@
"""Abstraction layer to resize images using PIL, ImageMagick, or a
public resizing proxy if neither is available.
"""
import urllib
from __future__ import division, absolute_import, print_function
import subprocess
import os
import re
from tempfile import NamedTemporaryFile
import logging
from six.moves.urllib.parse import urlencode
from beets import logging
from beets import util
import six
# Resizing methods
PIL = 1
IMAGEMAGICK = 2
WEBPROXY = 3
PROXY_URL = 'http://images.weserv.nl/'
if util.SNI_SUPPORTED:
PROXY_URL = 'https://images.weserv.nl/'
else:
PROXY_URL = 'http://images.weserv.nl/'
log = logging.getLogger('beets')
@@ -37,9 +44,9 @@ def resize_url(url, maxwidth):
"""Return a proxied image URL that resizes the original image to
maxwidth (preserving aspect ratio).
"""
return '{0}?{1}'.format(PROXY_URL, urllib.urlencode({
return '{0}?{1}'.format(PROXY_URL, urlencode({
'url': url.replace('http://', ''),
'w': str(maxwidth),
'w': maxwidth,
}))
@@ -48,8 +55,8 @@ def temp_file_for(path):
specified path.
"""
ext = os.path.splitext(path)[1]
with NamedTemporaryFile(suffix=ext, delete=False) as f:
return f.name
with NamedTemporaryFile(suffix=util.py3_path(ext), delete=False) as f:
return util.bytestring_path(f.name)
def pil_resize(maxwidth, path_in, path_out=None):
@@ -58,9 +65,8 @@ def pil_resize(maxwidth, path_in, path_out=None):
"""
path_out = path_out or temp_file_for(path_in)
from PIL import Image
log.debug(u'artresizer: PIL resizing {0} to {1}'.format(
util.displayable_path(path_in), util.displayable_path(path_out)
))
log.debug(u'artresizer: PIL resizing {0} to {1}',
util.displayable_path(path_in), util.displayable_path(path_out))
try:
im = Image.open(util.syspath(path_in))
@@ -69,9 +75,8 @@ def pil_resize(maxwidth, path_in, path_out=None):
im.save(path_out)
return path_out
except IOError:
log.error(u"PIL cannot create thumbnail for '{0}'".format(
util.displayable_path(path_in)
))
log.error(u"PIL cannot create thumbnail for '{0}'",
util.displayable_path(path_in))
return path_in
@@ -80,9 +85,8 @@ def im_resize(maxwidth, path_in, path_out=None):
Return the output path of resized image.
"""
path_out = path_out or temp_file_for(path_in)
log.debug(u'artresizer: ImageMagick resizing {0} to {1}'.format(
util.displayable_path(path_in), util.displayable_path(path_out)
))
log.debug(u'artresizer: ImageMagick resizing {0} to {1}',
util.displayable_path(path_in), util.displayable_path(path_out))
# "-resize widthxheight>" shrinks images with dimension(s) larger
# than the corresponding width and/or height dimension(s). The >
@@ -90,13 +94,13 @@ def im_resize(maxwidth, path_in, path_out=None):
# compatibility.
try:
util.command_output([
'convert', util.syspath(path_in),
'-resize', '{0}x^>'.format(maxwidth), path_out
'convert', util.syspath(path_in, prefix=False),
'-resize', '{0}x^>'.format(maxwidth),
util.syspath(path_out, prefix=False),
])
except subprocess.CalledProcessError:
log.warn(u'artresizer: IM convert failed for {0}'.format(
util.displayable_path(path_in)
))
log.warning(u'artresizer: IM convert failed for {0}',
util.displayable_path(path_in))
return path_in
return path_out
@@ -107,34 +111,67 @@ BACKEND_FUNCS = {
}
def pil_getsize(path_in):
from PIL import Image
try:
im = Image.open(util.syspath(path_in))
return im.size
except IOError as exc:
log.error(u"PIL could not read file {}: {}",
util.displayable_path(path_in), exc)
def im_getsize(path_in):
cmd = ['identify', '-format', '%w %h',
util.syspath(path_in, prefix=False)]
try:
out = util.command_output(cmd)
except subprocess.CalledProcessError as exc:
log.warning(u'ImageMagick size query failed')
log.debug(
u'`convert` exited with (status {}) when '
u'getting size with command {}:\n{}',
exc.returncode, cmd, exc.output.strip()
)
return
try:
return tuple(map(int, out.split(b' ')))
except IndexError:
log.warning(u'Could not understand IM output: {0!r}', out)
BACKEND_GET_SIZE = {
PIL: pil_getsize,
IMAGEMAGICK: im_getsize,
}
class Shareable(type):
"""A pseudo-singleton metaclass that allows both shared and
non-shared instances. The ``MyClass.shared`` property holds a
lazily-created shared instance of ``MyClass`` while calling
``MyClass()`` to construct a new object works as usual.
"""
def __init__(cls, name, bases, dict):
super(Shareable, cls).__init__(name, bases, dict)
cls._instance = None
def __init__(self, name, bases, dict):
super(Shareable, self).__init__(name, bases, dict)
self._instance = None
@property
def shared(cls):
if cls._instance is None:
cls._instance = cls()
return cls._instance
def shared(self):
if self._instance is None:
self._instance = self()
return self._instance
class ArtResizer(object):
class ArtResizer(six.with_metaclass(Shareable, object)):
"""A singleton class that performs image resizes.
"""
__metaclass__ = Shareable
def __init__(self, method=None):
"""Create a resizer object for the given method or, if none is
specified, with an inferred method.
def __init__(self):
"""Create a resizer object with an inferred method.
"""
self.method = self._check_method(method)
log.debug(u"artresizer: method is {0}".format(self.method))
self.method = self._check_method()
log.debug(u"artresizer: method is {0}", self.method)
self.can_compare = self._can_compare()
def resize(self, maxwidth, path_in, path_out=None):
@@ -165,47 +202,61 @@ class ArtResizer(object):
"""
return self.method[0] in BACKEND_FUNCS
def get_size(self, path_in):
"""Return the size of an image file as an int couple (width, height)
in pixels.
Only available locally
"""
if self.local:
func = BACKEND_GET_SIZE[self.method[0]]
return func(path_in)
def _can_compare(self):
"""A boolean indicating whether image comparison is available"""
return self.method[0] == IMAGEMAGICK and self.method[1] > (6, 8, 7)
@staticmethod
def _check_method(method=None):
"""A tuple indicating whether current method is available and its
version. If no method is given, it returns a supported one.
"""
# Guess available method
if not method:
for m in [IMAGEMAGICK, PIL]:
_, version = ArtResizer._check_method(m)
if version:
return (m, version)
return (WEBPROXY, (0))
def _check_method():
"""Return a tuple indicating an available method and its version."""
version = get_im_version()
if version:
return IMAGEMAGICK, version
if method == IMAGEMAGICK:
version = get_pil_version()
if version:
return PIL, version
# Try invoking ImageMagick's "convert".
try:
out = util.command_output(['identify', '--version'])
return WEBPROXY, (0)
if 'imagemagick' in out.lower():
pattern = r".+ (\d+)\.(\d+)\.(\d+).*"
match = re.search(pattern, out)
if match:
return (IMAGEMAGICK,
(int(match.group(1)),
int(match.group(2)),
int(match.group(3))))
return (IMAGEMAGICK, (0))
except (subprocess.CalledProcessError, OSError):
return (IMAGEMAGICK, None)
def get_im_version():
"""Return Image Magick version or None if it is unavailable
Try invoking ImageMagick's "convert".
"""
try:
out = util.command_output(['convert', '--version'])
if method == PIL:
# Try importing PIL.
try:
__import__('PIL', fromlist=['Image'])
return (PIL, (0))
except ImportError:
return (PIL, None)
if b'imagemagick' in out.lower():
pattern = br".+ (\d+)\.(\d+)\.(\d+).*"
match = re.search(pattern, out)
if match:
return (int(match.group(1)),
int(match.group(2)),
int(match.group(3)))
return (0,)
except (subprocess.CalledProcessError, OSError) as exc:
log.debug(u'ImageMagick check `convert --version` failed: {}', exc)
return None
def get_pil_version():
"""Return Image Magick version or None if it is unavailable
Try importing PIL."""
try:
__import__('PIL', fromlist=[str('Image')])
return (0,)
except ImportError:
return None
lib/beets/util/bluelet.py
Regular → Executable
+8 -17
View File
@@ -1,3 +1,5 @@
# -*- coding: utf-8 -*-
"""Extremely simple pure-Python implementation of coroutine-style
asynchronous socket I/O. Inspired by, but inferior to, Eventlet.
Bluelet can also be thought of as a less-terrible replacement for
@@ -5,6 +7,9 @@ asyncore.
Bluelet: easy concurrency without all the messy parallelism.
"""
from __future__ import division, absolute_import, print_function
import six
import socket
import select
import sys
@@ -15,20 +20,6 @@ import time
import collections
# A little bit of "six" (Python 2/3 compatibility): cope with PEP 3109 syntax
# changes.
PY3 = sys.version_info[0] == 3
if PY3:
def _reraise(typ, exc, tb):
raise exc.with_traceback(tb)
else:
exec("""
def _reraise(typ, exc, tb):
raise typ, exc, tb
""")
# Basic events used for thread scheduling.
class Event(object):
@@ -210,7 +201,7 @@ class ThreadException(Exception):
self.exc_info = exc_info
def reraise(self):
_reraise(self.exc_info[0], self.exc_info[1], self.exc_info[2])
six.reraise(self.exc_info[0], self.exc_info[1], self.exc_info[2])
SUSPENDED = Event() # Special sentinel placeholder for suspended threads.
@@ -550,7 +541,7 @@ def spawn(coro):
and child coroutines run concurrently.
"""
if not isinstance(coro, types.GeneratorType):
raise ValueError('%s is not a coroutine' % str(coro))
raise ValueError(u'%s is not a coroutine' % coro)
return SpawnEvent(coro)
@@ -560,7 +551,7 @@ def call(coro):
returns a value using end(), then this event returns that value.
"""
if not isinstance(coro, types.GeneratorType):
raise ValueError('%s is not a coroutine' % str(coro))
raise ValueError(u'%s is not a coroutine' % coro)
return DelegationEvent(coro)
lib/beets/util/confit.py
Regular → Executable
+268 -90
View File
@@ -1,5 +1,6 @@
# This file is part of Confit.
# Copyright 2014, Adrian Sampson.
# -*- coding: utf-8 -*-
# This file is part of Confuse.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -14,19 +15,16 @@
"""Worry-free YAML configuration files.
"""
from __future__ import unicode_literals
from __future__ import division, absolute_import, print_function
import platform
import os
import pkgutil
import sys
import yaml
import types
import collections
import re
try:
from collections import OrderedDict
except ImportError:
from ordereddict import OrderedDict
from collections import OrderedDict
UNIX_DIR_VAR = 'XDG_CONFIG_HOME'
UNIX_DIR_FALLBACK = '~/.config'
@@ -40,14 +38,15 @@ ROOT_NAME = 'root'
YAML_TAB_PROBLEM = "found character '\\t' that cannot start any token"
REDACTED_TOMBSTONE = 'REDACTED'
# Utilities.
PY3 = sys.version_info[0] == 3
STRING = str if PY3 else unicode
BASESTRING = str if PY3 else basestring
NUMERIC_TYPES = (int, float) if PY3 else (int, float, long)
TYPE_TYPES = (type,) if PY3 else (type, types.ClassType)
STRING = str if PY3 else unicode # noqa: F821
BASESTRING = str if PY3 else basestring # noqa: F821
NUMERIC_TYPES = (int, float) if PY3 else (int, float, long) # noqa: F821
def iter_first(sequence):
@@ -56,10 +55,7 @@ def iter_first(sequence):
"""
it = iter(sequence)
try:
if PY3:
return next(it)
else:
return it.next()
return next(it)
except StopIteration:
raise ValueError()
@@ -96,17 +92,17 @@ class ConfigReadError(ConfigError):
self.filename = filename
self.reason = reason
message = 'file {0} could not be read'.format(filename)
message = u'file {0} could not be read'.format(filename)
if isinstance(reason, yaml.scanner.ScannerError) and \
reason.problem == YAML_TAB_PROBLEM:
# Special-case error message for tab indentation in YAML markup.
message += ': found tab character at line {0}, column {1}'.format(
message += u': found tab character at line {0}, column {1}'.format(
reason.problem_mark.line + 1,
reason.problem_mark.column + 1,
)
elif reason:
# Generic error message uses exception's message.
message += ': {0}'.format(reason)
message += u': {0}'.format(reason)
super(ConfigReadError, self).__init__(message)
@@ -120,19 +116,19 @@ class ConfigSource(dict):
def __init__(self, value, filename=None, default=False):
super(ConfigSource, self).__init__(value)
if filename is not None and not isinstance(filename, BASESTRING):
raise TypeError('filename must be a string or None')
raise TypeError(u'filename must be a string or None')
self.filename = filename
self.default = default
def __repr__(self):
return 'ConfigSource({0}, {1}, {2})'.format(
super(ConfigSource, self).__repr__(),
repr(self.filename),
repr(self.default)
return 'ConfigSource({0!r}, {1!r}, {2!r})'.format(
super(ConfigSource, self),
self.filename,
self.default,
)
@classmethod
def of(self, value):
def of(cls, value):
"""Given either a dictionary or a `ConfigSource` object, return
a `ConfigSource` object. This lets a function accept either type
of object as an argument.
@@ -142,7 +138,7 @@ class ConfigSource(dict):
elif isinstance(value, dict):
return ConfigSource(value)
else:
raise TypeError('source value must be a dict')
raise TypeError(u'source value must be a dict')
class ConfigView(object):
@@ -177,7 +173,7 @@ class ConfigView(object):
try:
return iter_first(pairs)
except ValueError:
raise NotFoundError("{0} not found".format(self.name))
raise NotFoundError(u"{0} not found".format(self.name))
def exists(self):
"""Determine whether the view has a setting in any source.
@@ -208,7 +204,31 @@ class ConfigView(object):
raise NotImplementedError
def __repr__(self):
return '<ConfigView: %s>' % self.name
return '<{}: {}>'.format(self.__class__.__name__, self.name)
def __iter__(self):
"""Iterate over the keys of a dictionary view or the *subviews*
of a list view.
"""
# Try getting the keys, if this is a dictionary view.
try:
keys = self.keys()
for key in keys:
yield key
except ConfigTypeError:
# Otherwise, try iterating over a list.
collection = self.get()
if not isinstance(collection, (list, tuple)):
raise ConfigTypeError(
u'{0} must be a dictionary or a list, not {1}'.format(
self.name, type(collection).__name__
)
)
# Yield all the indices in the list.
for index in range(len(collection)):
yield self[index]
def __getitem__(self, key):
"""Get a subview of this view."""
@@ -225,10 +245,15 @@ class ConfigView(object):
def set_args(self, namespace):
"""Overlay parsed command-line arguments, generated by a library
like argparse or optparse, onto this view's value.
like argparse or optparse, onto this view's value. ``namespace``
can be a ``dict`` or namespace object.
"""
args = {}
for key, value in namespace.__dict__.items():
if isinstance(namespace, dict):
items = namespace.items()
else:
items = namespace.__dict__.items()
for key, value in items:
if value is not None: # Avoid unset options.
args[key] = value
self.set(args)
@@ -239,14 +264,17 @@ class ConfigView(object):
# just say ``bool(view)`` or use ``view`` in a conditional.
def __str__(self):
"""Gets the value for this view as a byte string."""
return str(self.get())
"""Get the value for this view as a bytestring.
"""
if PY3:
return self.__unicode__()
else:
return bytes(self.get())
def __unicode__(self):
"""Gets the value for this view as a unicode string. (Python 2
only.)
"""Get the value for this view as a Unicode string.
"""
return unicode(self.get())
return STRING(self.get())
def __nonzero__(self):
"""Gets the value for this view as a boolean. (Python 2 only.)
@@ -276,7 +304,7 @@ class ConfigView(object):
cur_keys = dic.keys()
except AttributeError:
raise ConfigTypeError(
'{0} must be a dict, not {1}'.format(
u'{0} must be a dict, not {1}'.format(
self.name, type(dic).__name__
)
)
@@ -317,7 +345,7 @@ class ConfigView(object):
it = iter(collection)
except TypeError:
raise ConfigTypeError(
'{0} must be an iterable, not {1}'.format(
u'{0} must be an iterable, not {1}'.format(
self.name, type(collection).__name__
)
)
@@ -326,17 +354,23 @@ class ConfigView(object):
# Validation and conversion.
def flatten(self):
def flatten(self, redact=False):
"""Create a hierarchy of OrderedDicts containing the data from
this view, recursively reifying all views to get their
represented values.
If `redact` is set, then sensitive values are replaced with
the string "REDACTED".
"""
od = OrderedDict()
for key, view in self.items():
try:
od[key] = view.flatten()
except ConfigTypeError:
od[key] = view.get()
if redact and view.redact:
od[key] = REDACTED_TOMBSTONE
else:
try:
od[key] = view.flatten(redact=redact)
except ConfigTypeError:
od[key] = view.get()
return od
def get(self, template=None):
@@ -354,19 +388,60 @@ class ConfigView(object):
"""
return as_template(template).value(self, template)
# Old validation methods (deprecated).
# Shortcuts for common templates.
def as_filename(self):
"""Get the value as a path. Equivalent to `get(Filename())`.
"""
return self.get(Filename())
def as_choice(self, choices):
"""Get the value from a list of choices. Equivalent to
`get(Choice(choices))`.
"""
return self.get(Choice(choices))
def as_number(self):
"""Get the value as any number type: int or float. Equivalent to
`get(Number())`.
"""
return self.get(Number())
def as_str_seq(self):
return self.get(StrSeq())
def as_str_seq(self, split=True):
"""Get the value as a sequence of strings. Equivalent to
`get(StrSeq())`.
"""
return self.get(StrSeq(split=split))
def as_str(self):
"""Get the value as a (Unicode) string. Equivalent to
`get(unicode)` on Python 2 and `get(str)` on Python 3.
"""
return self.get(String())
# Redaction.
@property
def redact(self):
"""Whether the view contains sensitive information and should be
redacted from output.
"""
return () in self.get_redactions()
@redact.setter
def redact(self, flag):
self.set_redaction((), flag)
def set_redaction(self, path, flag):
"""Add or remove a redaction for a key path, which should be an
iterable of keys.
"""
raise NotImplementedError()
def get_redactions(self):
"""Get the set of currently-redacted sub-key-paths at this view.
"""
raise NotImplementedError()
class RootView(ConfigView):
@@ -380,6 +455,7 @@ class RootView(ConfigView):
"""
self.sources = list(sources)
self.name = ROOT_NAME
self.redactions = set()
def add(self, obj):
self.sources.append(ConfigSource.of(obj))
@@ -391,12 +467,24 @@ class RootView(ConfigView):
return ((dict(s), s) for s in self.sources)
def clear(self):
"""Remove all sources from this configuration."""
"""Remove all sources (and redactions) from this
configuration.
"""
del self.sources[:]
self.redactions.clear()
def root(self):
return self
def set_redaction(self, path, flag):
if flag:
self.redactions.add(path)
elif path in self.redactions:
self.redactions.remove(path)
def get_redactions(self):
return self.redactions
class Subview(ConfigView):
"""A subview accessed via a subscript of a parent view."""
@@ -414,11 +502,13 @@ class Subview(ConfigView):
if not isinstance(self.key, int):
self.name += '.'
if isinstance(self.key, int):
self.name += '#{0}'.format(self.key)
elif isinstance(self.key, BASESTRING):
self.name += '{0}'.format(self.key)
self.name += u'#{0}'.format(self.key)
elif isinstance(self.key, bytes):
self.name += self.key.decode('utf-8')
elif isinstance(self.key, STRING):
self.name += self.key
else:
self.name += '{0}'.format(repr(self.key))
self.name += repr(self.key)
def resolve(self):
for collection, source in self.parent.resolve():
@@ -433,7 +523,7 @@ class Subview(ConfigView):
except TypeError:
# Not subscriptable.
raise ConfigTypeError(
"{0} must be a collection, not {1}".format(
u"{0} must be a collection, not {1}".format(
self.parent.name, type(collection).__name__
)
)
@@ -448,6 +538,13 @@ class Subview(ConfigView):
def root(self):
return self.parent.root()
def set_redaction(self, path, flag):
self.parent.set_redaction((self.key,) + path, flag)
def get_redactions(self):
return (kp[1:] for kp in self.parent.get_redactions()
if kp and kp[0] == self.key)
# Config file paths, including platform-specific paths and in-package
# defaults.
@@ -536,7 +633,7 @@ class Loader(yaml.SafeLoader):
else:
raise yaml.constructor.ConstructorError(
None, None,
'expected a mapping node, but found %s' % node.id,
u'expected a mapping node, but found %s' % node.id,
node.start_mark
)
@@ -547,7 +644,7 @@ class Loader(yaml.SafeLoader):
hash(key)
except TypeError as exc:
raise yaml.constructor.ConstructorError(
'while constructing a mapping',
u'while constructing a mapping',
node.start_mark, 'found unacceptable key (%s)' % exc,
key_node.start_mark
)
@@ -595,11 +692,11 @@ class Dumper(yaml.SafeDumper):
for item_key, item_value in mapping:
node_key = self.represent_data(item_key)
node_value = self.represent_data(item_value)
if not (isinstance(node_key, yaml.ScalarNode)
and not node_key.style):
if not (isinstance(node_key, yaml.ScalarNode) and
not node_key.style):
best_style = False
if not (isinstance(node_value, yaml.ScalarNode)
and not node_value.style):
if not (isinstance(node_value, yaml.ScalarNode) and
not node_value.style):
best_style = False
value.append((node_key, node_value))
if flow_style is None:
@@ -625,9 +722,9 @@ class Dumper(yaml.SafeDumper):
"""Represent bool as 'yes' or 'no' instead of 'true' or 'false'.
"""
if data:
value = 'yes'
value = u'yes'
else:
value = 'no'
value = u'no'
return self.represent_scalar('tag:yaml.org,2002:bool', value)
def represent_none(self, data):
@@ -752,7 +849,7 @@ class Configuration(RootView):
appdir = os.environ[self._env_var]
appdir = os.path.abspath(os.path.expanduser(appdir))
if os.path.isfile(appdir):
raise ConfigError('{0} must be a directory'.format(
raise ConfigError(u'{0} must be a directory'.format(
self._env_var
))
@@ -776,7 +873,7 @@ class Configuration(RootView):
filename = os.path.abspath(filename)
self.set(ConfigSource(load_yaml(filename), filename))
def dump(self, full=True):
def dump(self, full=True, redact=False):
"""Dump the Configuration object to a YAML file.
The order of the keys is determined from the default
@@ -788,13 +885,17 @@ class Configuration(RootView):
:type filename: unicode
:param full: Dump settings that don't differ from the defaults
as well
:param redact: Remove sensitive information (views with the `redact`
flag set) from the output
"""
if full:
out_dict = self.flatten()
out_dict = self.flatten(redact=redact)
else:
# Exclude defaults when flattening.
sources = [s for s in self.sources if not s.default]
out_dict = RootView(sources).flatten()
temp_root = RootView(sources)
temp_root.redactions = self.redactions
out_dict = temp_root.flatten(redact=redact)
yaml_out = yaml.dump(out_dict, Dumper=Dumper,
default_flow_style=None, indent=4,
@@ -806,7 +907,7 @@ class Configuration(RootView):
if source.default:
default_source = source
break
if default_source:
if default_source and default_source.filename:
with open(default_source.filename, 'r') as fp:
default_data = fp.read()
yaml_out = restore_yaml_comments(yaml_out, default_data)
@@ -853,7 +954,7 @@ class LazyConfig(Configuration):
def clear(self):
"""Remove all sources from this configuration."""
del self.sources[:]
super(LazyConfig, self).clear()
self._lazy_suffix = []
self._lazy_prefix = []
@@ -870,7 +971,7 @@ should be raised when the value is missing.
class Template(object):
"""A value template for configuration fields.
The template works like a type and instructs Confit about how to
The template works like a type and instructs Confuse about how to
interpret a deserialized YAML value. This includes type conversions,
providing a default value, and validating for errors. For example, a
filepath type might expand tildes and check that the file exists.
@@ -901,7 +1002,7 @@ class Template(object):
return self.convert(value, view)
elif self.default is REQUIRED:
# Missing required value. This is an error.
raise NotFoundError("{0} not found".format(view.name))
raise NotFoundError(u"{0} not found".format(view.name))
else:
# Missing value, but not required.
return self.default
@@ -926,7 +1027,7 @@ class Template(object):
"""
exc_class = ConfigTypeError if type_error else ConfigValueError
raise exc_class(
'{0}: {1}'.format(view.name, message)
u'{0}: {1}'.format(view.name, message)
)
def __repr__(self):
@@ -947,7 +1048,7 @@ class Integer(Template):
elif isinstance(value, float):
return int(value)
else:
self.fail('must be a number', view, True)
self.fail(u'must be a number', view, True)
class Number(Template):
@@ -960,7 +1061,7 @@ class Number(Template):
return value
else:
self.fail(
'must be numeric, not {0}'.format(type(value).__name__),
u'must be numeric, not {0}'.format(type(value).__name__),
view,
True
)
@@ -1005,18 +1106,29 @@ class String(Template):
if pattern:
self.regex = re.compile(pattern)
def __repr__(self):
args = []
if self.default is not REQUIRED:
args.append(repr(self.default))
if self.pattern is not None:
args.append('pattern=' + repr(self.pattern))
return 'String({0})'.format(', '.join(args))
def convert(self, value, view):
"""Check that the value is a string and matches the pattern.
"""
if isinstance(value, BASESTRING):
if self.pattern and not self.regex.match(value):
self.fail(
"must match the pattern {0}".format(self.pattern),
u"must match the pattern {0}".format(self.pattern),
view
)
return value
else:
self.fail('must be a string', view, True)
self.fail(u'must be a string', view, True)
class Choice(Template):
@@ -1037,7 +1149,7 @@ class Choice(Template):
"""
if value not in self.choices:
self.fail(
'must be one of {0}, not {1}'.format(
u'must be one of {0}, not {1}'.format(
repr(list(self.choices)), repr(value)
),
view
@@ -1052,6 +1164,67 @@ class Choice(Template):
return 'Choice({0!r})'.format(self.choices)
class OneOf(Template):
"""A template that permits values complying to one of the given templates.
"""
def __init__(self, allowed, default=REQUIRED):
super(OneOf, self).__init__(default)
self.allowed = list(allowed)
def __repr__(self):
args = []
if self.allowed is not None:
args.append('allowed=' + repr(self.allowed))
if self.default is not REQUIRED:
args.append(repr(self.default))
return 'OneOf({0})'.format(', '.join(args))
def value(self, view, template):
self.template = template
return super(OneOf, self).value(view, template)
def convert(self, value, view):
"""Ensure that the value follows at least one template.
"""
is_mapping = isinstance(self.template, MappingTemplate)
for candidate in self.allowed:
try:
if is_mapping:
if isinstance(candidate, Filename) and \
candidate.relative_to:
next_template = candidate.template_with_relatives(
view,
self.template
)
next_template.subtemplates[view.key] = as_template(
candidate
)
else:
next_template = MappingTemplate({view.key: candidate})
return view.parent.get(next_template)[view.key]
else:
return view.get(candidate)
except ConfigTemplateError:
raise
except ConfigError:
pass
except ValueError as exc:
raise ConfigTemplateError(exc)
self.fail(
u'must be one of {0}, not {1}'.format(
repr(self.allowed), repr(value)
),
view
)
class StrSeq(Template):
"""A template for values that are lists of strings.
@@ -1070,7 +1243,7 @@ class StrSeq(Template):
def convert(self, value, view):
if isinstance(value, bytes):
value = value.decode('utf8', 'ignore')
value = value.decode('utf-8', 'ignore')
if isinstance(value, STRING):
if self.split:
@@ -1081,17 +1254,17 @@ class StrSeq(Template):
try:
value = list(value)
except TypeError:
self.fail('must be a whitespace-separated string or a list',
self.fail(u'must be a whitespace-separated string or a list',
view, True)
def convert(x):
if isinstance(x, unicode):
if isinstance(x, STRING):
return x
elif isinstance(x, BASESTRING):
return x.decode('utf8', 'ignore')
elif isinstance(x, bytes):
return x.decode('utf-8', 'ignore')
else:
self.fail('must be a list of strings', view, True)
return map(convert, value)
self.fail(u'must be a list of strings', view, True)
return list(map(convert, value))
class Filename(Template):
@@ -1107,7 +1280,7 @@ class Filename(Template):
"""
def __init__(self, default=REQUIRED, cwd=None, relative_to=None,
in_app_dir=False):
""" `relative_to` is the name of a sibling value that is
"""`relative_to` is the name of a sibling value that is
being validated at the same time.
`in_app_dir` indicates whether the path should be resolved
@@ -1140,19 +1313,19 @@ class Filename(Template):
if not isinstance(template, (collections.Mapping, MappingTemplate)):
# disallow config.get(Filename(relative_to='foo'))
raise ConfigTemplateError(
'relative_to may only be used when getting multiple values.'
u'relative_to may only be used when getting multiple values.'
)
elif self.relative_to == view.key:
raise ConfigTemplateError(
'{0} is relative to itself'.format(view.name)
u'{0} is relative to itself'.format(view.name)
)
elif self.relative_to not in view.parent.keys():
# self.relative_to is not in the config
self.fail(
(
'needs sibling value "{0}" to expand relative path'
u'needs sibling value "{0}" to expand relative path'
).format(self.relative_to),
view
)
@@ -1174,12 +1347,12 @@ class Filename(Template):
if next_relative in template.subtemplates:
# we encountered this config key previously
raise ConfigTemplateError((
'{0} and {1} are recursively relative'
u'{0} and {1} are recursively relative'
).format(view.name, self.relative_to))
else:
raise ConfigTemplateError((
'missing template for {0}, needed to expand {1}\'s' +
'relative path'
u'missing template for {0}, needed to expand {1}\'s' +
u'relative path'
).format(self.relative_to, view.name))
next_template.subtemplates[next_relative] = rel_to_template
@@ -1191,7 +1364,7 @@ class Filename(Template):
path, source = view.first()
if not isinstance(path, BASESTRING):
self.fail(
'must be a filename, not {0}'.format(type(path).__name__),
u'must be a filename, not {0}'.format(type(path).__name__),
view,
True
)
@@ -1229,7 +1402,7 @@ class TypeTemplate(Template):
def convert(self, value, view):
if not isinstance(value, self.typ):
self.fail(
'must be a {0}, not {1}'.format(
u'must be a {0}, not {1}'.format(
self.typ.__name__,
type(value).__name__,
),
@@ -1267,6 +1440,11 @@ def as_template(value):
return String()
elif isinstance(value, BASESTRING):
return String(value)
elif isinstance(value, set):
# convert to list to avoid hash related problems
return Choice(list(value))
elif isinstance(value, list):
return OneOf(value)
elif value is float:
return Number()
elif value is None:
@@ -1278,4 +1456,4 @@ def as_template(value):
elif isinstance(value, type):
return TypeTemplate(value)
else:
raise ValueError('cannot convert to template: {0!r}'.format(value))
raise ValueError(u'cannot convert to template: {0!r}'.format(value))
lib/beets/util/enumeration.py
Regular → Executable
+4 -1
View File
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2013, Adrian Sampson.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -12,6 +13,8 @@
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
from __future__ import division, absolute_import, print_function
from enum import Enum
lib/beets/util/functemplate.py
Regular → Executable
+94 -45
View File
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2013, Adrian Sampson.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -25,12 +26,15 @@ library: unknown symbols are left intact.
This is sort of like a tiny, horrible degeneration of a real templating
engine like Jinja2 or Mustache.
"""
from __future__ import print_function
from __future__ import division, absolute_import, print_function
import re
import ast
import dis
import types
import sys
import six
SYMBOL_DELIM = u'$'
FUNC_DELIM = u'%'
@@ -70,13 +74,13 @@ def ex_literal(val):
"""
if val is None:
return ast.Name('None', ast.Load())
elif isinstance(val, (int, float, long)):
elif isinstance(val, six.integer_types):
return ast.Num(val)
elif isinstance(val, bool):
return ast.Name(str(val), ast.Load())
elif isinstance(val, basestring):
return ast.Name(bytes(val), ast.Load())
elif isinstance(val, six.string_types):
return ast.Str(val)
raise TypeError('no literal for {0}'.format(type(val)))
raise TypeError(u'no literal for {0}'.format(type(val)))
def ex_varassign(name, expr):
@@ -93,7 +97,7 @@ def ex_call(func, args):
function may be an expression or the name of a function. Each
argument may be an expression or a value to be used as a literal.
"""
if isinstance(func, basestring):
if isinstance(func, six.string_types):
func = ex_rvalue(func)
args = list(args)
@@ -101,7 +105,10 @@ def ex_call(func, args):
if not isinstance(args[i], ast.expr):
args[i] = ex_literal(args[i])
return ast.Call(func, args, [], None, None)
if sys.version_info[:2] < (3, 5):
return ast.Call(func, args, [], None, None)
else:
return ast.Call(func, args, [])
def compile_func(arg_names, statements, name='_the_func', debug=False):
@@ -109,16 +116,31 @@ def compile_func(arg_names, statements, name='_the_func', debug=False):
the resulting Python function. If `debug`, then print out the
bytecode of the compiled function.
"""
func_def = ast.FunctionDef(
name,
ast.arguments(
[ast.Name(n, ast.Param()) for n in arg_names],
None, None,
[ex_literal(None) for _ in arg_names],
),
statements,
[],
)
if six.PY2:
func_def = ast.FunctionDef(
name=name.encode('utf-8'),
args=ast.arguments(
args=[ast.Name(n, ast.Param()) for n in arg_names],
vararg=None,
kwarg=None,
defaults=[ex_literal(None) for _ in arg_names],
),
body=statements,
decorator_list=[],
)
else:
func_def = ast.FunctionDef(
name=name,
args=ast.arguments(
args=[ast.arg(arg=n, annotation=None) for n in arg_names],
kwonlyargs=[],
kw_defaults=[],
defaults=[ex_literal(None) for _ in arg_names],
),
body=statements,
decorator_list=[],
)
mod = ast.Module([func_def])
ast.fix_missing_locations(mod)
@@ -132,7 +154,7 @@ def compile_func(arg_names, statements, name='_the_func', debug=False):
dis.dis(const)
the_locals = {}
exec prog in {}, the_locals
exec(prog, {}, the_locals)
return the_locals[name]
@@ -160,8 +182,12 @@ class Symbol(object):
def translate(self):
"""Compile the variable lookup."""
expr = ex_rvalue(VARIABLE_PREFIX + self.ident.encode('utf8'))
return [expr], set([self.ident.encode('utf8')]), set()
if six.PY2:
ident = self.ident.encode('utf-8')
else:
ident = self.ident
expr = ex_rvalue(VARIABLE_PREFIX + ident)
return [expr], set([ident]), set()
class Call(object):
@@ -186,15 +212,19 @@ class Call(object):
except Exception as exc:
# Function raised exception! Maybe inlining the name of
# the exception will help debug.
return u'<%s>' % unicode(exc)
return unicode(out)
return u'<%s>' % six.text_type(exc)
return six.text_type(out)
else:
return self.original
def translate(self):
"""Compile the function call."""
varnames = set()
funcnames = set([self.ident.encode('utf8')])
if six.PY2:
ident = self.ident.encode('utf-8')
else:
ident = self.ident
funcnames = set([ident])
arg_exprs = []
for arg in self.args:
@@ -209,14 +239,14 @@ class Call(object):
[ex_call(
'map',
[
ex_rvalue('unicode'),
ex_rvalue(six.text_type.__name__),
ast.List(subexprs, ast.Load()),
]
)],
))
subexpr_call = ex_call(
FUNCTION_PREFIX + self.ident.encode('utf8'),
FUNCTION_PREFIX + ident,
arg_exprs
)
return [subexpr_call], varnames, funcnames
@@ -238,11 +268,11 @@ class Expression(object):
"""
out = []
for part in self.parts:
if isinstance(part, basestring):
if isinstance(part, six.string_types):
out.append(part)
else:
out.append(part.evaluate(env))
return u''.join(map(unicode, out))
return u''.join(map(six.text_type, out))
def translate(self):
"""Compile the expression to a list of Python AST expressions, a
@@ -252,7 +282,7 @@ class Expression(object):
varnames = set()
funcnames = set()
for part in self.parts:
if isinstance(part, basestring):
if isinstance(part, six.string_types):
expressions.append(ex_literal(part))
else:
e, v, f = part.translate()
@@ -281,16 +311,24 @@ class Parser(object):
replaced with a real, accepted parsing technique (PEG, parser
generator, etc.).
"""
def __init__(self, string):
def __init__(self, string, in_argument=False):
""" Create a new parser.
:param in_arguments: boolean that indicates the parser is to be
used for parsing function arguments, ie. considering commas
(`ARG_SEP`) a special character
"""
self.string = string
self.in_argument = in_argument
self.pos = 0
self.parts = []
# Common parsing resources.
special_chars = (SYMBOL_DELIM, FUNC_DELIM, GROUP_OPEN, GROUP_CLOSE,
ARG_SEP, ESCAPE_CHAR)
special_char_re = re.compile(ur'[%s]|$' %
ESCAPE_CHAR)
special_char_re = re.compile(r'[%s]|$' %
u''.join(re.escape(c) for c in special_chars))
escapable_chars = (SYMBOL_DELIM, FUNC_DELIM, GROUP_CLOSE, ARG_SEP)
terminator_chars = (GROUP_CLOSE,)
def parse_expression(self):
"""Parse a template expression starting at ``pos``. Resulting
@@ -298,17 +336,27 @@ class Parser(object):
the ``parts`` field, a list. The ``pos`` field is updated to be
the next character after the expression.
"""
# Append comma (ARG_SEP) to the list of special characters only when
# parsing function arguments.
extra_special_chars = ()
special_char_re = self.special_char_re
if self.in_argument:
extra_special_chars = (ARG_SEP,)
special_char_re = re.compile(
r'[%s]|$' % u''.join(re.escape(c) for c in
self.special_chars + extra_special_chars))
text_parts = []
while self.pos < len(self.string):
char = self.string[self.pos]
if char not in self.special_chars:
if char not in self.special_chars + extra_special_chars:
# A non-special character. Skip to the next special
# character, treating the interstice as literal text.
next_pos = (
self.special_char_re.search(self.string[self.pos:]).start()
+ self.pos
special_char_re.search(
self.string[self.pos:]).start() + self.pos
)
text_parts.append(self.string[self.pos:next_pos])
self.pos = next_pos
@@ -318,14 +366,14 @@ class Parser(object):
# The last character can never begin a structure, so we
# just interpret it as a literal character (unless it
# terminates the expression, as with , and }).
if char not in (GROUP_CLOSE, ARG_SEP):
if char not in self.terminator_chars + extra_special_chars:
text_parts.append(char)
self.pos += 1
break
next_char = self.string[self.pos + 1]
if char == ESCAPE_CHAR and next_char in \
(SYMBOL_DELIM, FUNC_DELIM, GROUP_CLOSE, ARG_SEP):
if char == ESCAPE_CHAR and next_char in (self.escapable_chars +
extra_special_chars):
# An escaped special character ($$, $}, etc.). Note that
# ${ is not an escape sequence: this is ambiguous with
# the start of a symbol and it's not necessary (just
@@ -345,7 +393,7 @@ class Parser(object):
elif char == FUNC_DELIM:
# Parse a function call.
self.parse_call()
elif char in (GROUP_CLOSE, ARG_SEP):
elif char in self.terminator_chars + extra_special_chars:
# Template terminated.
break
elif char == GROUP_OPEN:
@@ -453,7 +501,7 @@ class Parser(object):
expressions = []
while self.pos < len(self.string):
subparser = Parser(self.string[self.pos:])
subparser = Parser(self.string[self.pos:], in_argument=True)
subparser.parse_expression()
# Extract and advance past the parsed expression.
@@ -477,7 +525,7 @@ class Parser(object):
Updates ``pos``.
"""
remainder = self.string[self.pos:]
ident = re.match(ur'\w*', remainder).group(0)
ident = re.match(r'\w*', remainder).group(0)
self.pos += len(ident)
return ident
@@ -524,6 +572,7 @@ class Template(object):
res = self.compiled(values, functions)
except: # Handle any exceptions thrown by compiled version.
res = self.interpret(values, functions)
return res
def translate(self):
@@ -532,9 +581,9 @@ class Template(object):
argnames = []
for varname in varnames:
argnames.append(VARIABLE_PREFIX.encode('utf8') + varname)
argnames.append(VARIABLE_PREFIX + varname)
for funcname in funcnames:
argnames.append(FUNCTION_PREFIX.encode('utf8') + funcname)
argnames.append(FUNCTION_PREFIX + funcname)
func = compile_func(
argnames,
@@ -559,7 +608,7 @@ if __name__ == '__main__':
import timeit
_tmpl = Template(u'foo $bar %baz{foozle $bar barzle} $bar')
_vars = {'bar': 'qux'}
_funcs = {'baz': unicode.upper}
_funcs = {'baz': six.text_type.upper}
interp_time = timeit.timeit('_tmpl.interpret(_vars, _funcs)',
'from __main__ import _tmpl, _vars, _funcs',
number=10000)
@@ -568,4 +617,4 @@ if __name__ == '__main__':
'from __main__ import _tmpl, _vars, _funcs',
number=10000)
print(comp_time)
print('Speedup:', interp_time / comp_time)
print(u'Speedup:', interp_time / comp_time)
lib/beets/util/hidden.py
Executable
+86
View File
@@ -0,0 +1,86 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
"""Simple library to work out if a file is hidden on different platforms."""
from __future__ import division, absolute_import, print_function
import os
import stat
import ctypes
import sys
import beets.util
def _is_hidden_osx(path):
"""Return whether or not a file is hidden on OS X.
This uses os.lstat to work out if a file has the "hidden" flag.
"""
file_stat = os.lstat(beets.util.syspath(path))
if hasattr(file_stat, 'st_flags') and hasattr(stat, 'UF_HIDDEN'):
return bool(file_stat.st_flags & stat.UF_HIDDEN)
else:
return False
def _is_hidden_win(path):
"""Return whether or not a file is hidden on Windows.
This uses GetFileAttributes to work out if a file has the "hidden" flag
(FILE_ATTRIBUTE_HIDDEN).
"""
# FILE_ATTRIBUTE_HIDDEN = 2 (0x2) from GetFileAttributes documentation.
hidden_mask = 2
# Retrieve the attributes for the file.
attrs = ctypes.windll.kernel32.GetFileAttributesW(beets.util.syspath(path))
# Ensure we have valid attribues and compare them against the mask.
return attrs >= 0 and attrs & hidden_mask
def _is_hidden_dot(path):
"""Return whether or not a file starts with a dot.
Files starting with a dot are seen as "hidden" files on Unix-based OSes.
"""
return os.path.basename(path).startswith(b'.')
def is_hidden(path):
"""Return whether or not a file is hidden. `path` should be a
bytestring filename.
This method works differently depending on the platform it is called on.
On OS X, it uses both the result of `is_hidden_osx` and `is_hidden_dot` to
work out if a file is hidden.
On Windows, it uses the result of `is_hidden_win` to work out if a file is
hidden.
On any other operating systems (i.e. Linux), it uses `is_hidden_dot` to
work out if a file is hidden.
"""
# Run platform specific functions depending on the platform
if sys.platform == 'darwin':
return _is_hidden_osx(path) or _is_hidden_dot(path)
elif sys.platform == 'win32':
return _is_hidden_win(path)
else:
return _is_hidden_dot(path)
__all__ = ['is_hidden']
lib/beets/util/pipeline.py
Regular → Executable
+35 -22
View File
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2013, Adrian Sampson.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -30,11 +31,13 @@ up a bottleneck stage by dividing its work among multiple threads.
To do so, pass an iterable of coroutines to the Pipeline constructor
in place of any single coroutine.
"""
from __future__ import print_function
import Queue
from __future__ import division, absolute_import, print_function
from six.moves import queue
from threading import Thread, Lock
import sys
import six
BUBBLE = '__PIPELINE_BUBBLE__'
POISON = '__PIPELINE_POISON__'
@@ -61,7 +64,17 @@ def _invalidate_queue(q, val=None, sync=True):
q.mutex.acquire()
try:
q.maxsize = 0
# Originally, we set `maxsize` to 0 here, which is supposed to mean
# an unlimited queue size. However, there is a race condition since
# Python 3.2 when this attribute is changed while another thread is
# waiting in put()/get() due to a full/empty queue.
# Setting it to 2 is still hacky because Python does not give any
# guarantee what happens if Queue methods/attributes are overwritten
# when it is already in use. However, because of our dummy _put()
# and _get() methods, it provides a workaround to let the queue appear
# to be never empty or full.
# See issue https://github.com/beetbox/beets/issues/2078
q.maxsize = 2
q._qsize = _qsize
q._put = _put
q._get = _get
@@ -73,13 +86,13 @@ def _invalidate_queue(q, val=None, sync=True):
q.mutex.release()
class CountedQueue(Queue.Queue):
class CountedQueue(queue.Queue):
"""A queue that keeps track of the number of threads that are
still feeding into it. The queue is poisoned when all threads are
finished with the queue.
"""
def __init__(self, maxsize=0):
Queue.Queue.__init__(self, maxsize)
queue.Queue.__init__(self, maxsize)
self.nthreads = 0
self.poisoned = False
@@ -246,7 +259,7 @@ class FirstPipelineThread(PipelineThread):
# Get the value from the generator.
try:
msg = self.coro.next()
msg = next(self.coro)
except StopIteration:
break
@@ -279,7 +292,7 @@ class MiddlePipelineThread(PipelineThread):
def run(self):
try:
# Prime the coroutine.
self.coro.next()
next(self.coro)
while True:
with self.abort_lock:
@@ -324,7 +337,7 @@ class LastPipelineThread(PipelineThread):
def run(self):
# Prime the coroutine.
self.coro.next()
next(self.coro)
try:
while True:
@@ -359,7 +372,7 @@ class Pipeline(object):
be at least two stages.
"""
if len(stages) < 2:
raise ValueError('pipeline must have at least two stages')
raise ValueError(u'pipeline must have at least two stages')
self.stages = []
for stage in stages:
if isinstance(stage, (list, tuple)):
@@ -409,7 +422,7 @@ class Pipeline(object):
try:
# Using a timeout allows us to receive KeyboardInterrupt
# exceptions during the join().
while threads[-1].isAlive():
while threads[-1].is_alive():
threads[-1].join(1)
except:
@@ -429,7 +442,7 @@ class Pipeline(object):
exc_info = thread.exc_info
if exc_info:
# Make the exception appear as it was raised originally.
raise exc_info[0], exc_info[1], exc_info[2]
six.reraise(exc_info[0], exc_info[1], exc_info[2])
def pull(self):
"""Yield elements from the end of the pipeline. Runs the stages
@@ -442,7 +455,7 @@ class Pipeline(object):
# "Prime" the coroutines.
for coro in coros[1:]:
coro.next()
next(coro)
# Begin the pipeline.
for out in coros[0]:
@@ -464,14 +477,14 @@ if __name__ == '__main__':
# in parallel.
def produce():
for i in range(5):
print('generating %i' % i)
print(u'generating %i' % i)
time.sleep(1)
yield i
def work():
num = yield
while True:
print('processing %i' % num)
print(u'processing %i' % num)
time.sleep(2)
num = yield num * 2
@@ -479,7 +492,7 @@ if __name__ == '__main__':
while True:
num = yield
time.sleep(1)
print('received %i' % num)
print(u'received %i' % num)
ts_start = time.time()
Pipeline([produce(), work(), consume()]).run_sequential()
@@ -488,22 +501,22 @@ if __name__ == '__main__':
ts_par = time.time()
Pipeline([produce(), (work(), work()), consume()]).run_parallel()
ts_end = time.time()
print('Sequential time:', ts_seq - ts_start)
print('Parallel time:', ts_par - ts_seq)
print('Multiply-parallel time:', ts_end - ts_par)
print(u'Sequential time:', ts_seq - ts_start)
print(u'Parallel time:', ts_par - ts_seq)
print(u'Multiply-parallel time:', ts_end - ts_par)
print()
# Test a pipeline that raises an exception.
def exc_produce():
for i in range(10):
print('generating %i' % i)
print(u'generating %i' % i)
time.sleep(1)
yield i
def exc_work():
num = yield
while True:
print('processing %i' % num)
print(u'processing %i' % num)
time.sleep(3)
if num == 3:
raise Exception()
@@ -512,6 +525,6 @@ if __name__ == '__main__':
def exc_consume():
while True:
num = yield
print('received %i' % num)
print(u'received %i' % num)
Pipeline([exc_produce(), exc_work(), exc_consume()]).run_parallel(1)
lib/beets/vfs.py
Regular → Executable
+4 -1
View File
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# This file is part of beets.
# Copyright 2013, Adrian Sampson.
# Copyright 2016, Adrian Sampson.
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -15,6 +16,8 @@
"""A simple utility for constructing filesystem-like trees from beets
libraries.
"""
from __future__ import division, absolute_import, print_function
from collections import namedtuple
from beets import util