mirror of
https://github.com/rembo10/headphones.git
synced 2026-07-20 16:03:59 +01:00
Revert "Migrate as much as possible to pip and requirements.txt"
This reverts commit 982594a4a5.
This commit is contained in:
rembo10
@@ -0,0 +1,21 @@
|
||||
The MIT License
|
||||
|
||||
Copyright (c) 2010-2014 Adrian Sampson
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in
|
||||
all copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
||||
THE SOFTWARE.
|
||||
@@ -0,0 +1,94 @@
|
||||
.. image:: https://travis-ci.org/sampsyo/beets.svg?branch=master
|
||||
:target: https://travis-ci.org/sampsyo/beets
|
||||
|
||||
.. image:: http://img.shields.io/coveralls/sampsyo/beets.svg
|
||||
:target: https://coveralls.io/r/sampsyo/beets
|
||||
|
||||
.. image:: http://img.shields.io/pypi/v/beets.svg
|
||||
:target: https://pypi.python.org/pypi/beets
|
||||
|
||||
Beets is the media library management system for obsessive-compulsive music
|
||||
geeks.
|
||||
|
||||
The purpose of beets is to get your music collection right once and for all.
|
||||
It catalogs your collection, automatically improving its metadata as it goes.
|
||||
It then provides a bouquet of tools for manipulating and accessing your music.
|
||||
|
||||
Here's an example of beets' brainy tag corrector doing its thing::
|
||||
|
||||
$ beet import ~/music/ladytron
|
||||
Tagging:
|
||||
Ladytron - Witching Hour
|
||||
(Similarity: 98.4%)
|
||||
* Last One Standing -> The Last One Standing
|
||||
* Beauty -> Beauty*2
|
||||
* White Light Generation -> Whitelightgenerator
|
||||
* All the Way -> All the Way...
|
||||
|
||||
Because beets is designed as a library, it can do almost anything you can
|
||||
imagine for your music collection. Via `plugins`_, beets becomes a panacea:
|
||||
|
||||
- Fetch or calculate all the metadata you could possibly need: `album art`_,
|
||||
`lyrics`_, `genres`_, `tempos`_, `ReplayGain`_ levels, or `acoustic
|
||||
fingerprints`_.
|
||||
- Get metadata from `MusicBrainz`_, `Discogs`_, or `Beatport`_. Or guess
|
||||
metadata using songs' filenames or their acoustic fingerprints.
|
||||
- `Transcode audio`_ to any format you like.
|
||||
- Check your library for `duplicate tracks and albums`_ or for `albums that
|
||||
are missing tracks`_.
|
||||
- Clean up crufty tags left behind by other, less-awesome tools.
|
||||
- Embed and extract album art from files' metadata.
|
||||
- Browse your music library graphically through a Web browser and play it in any
|
||||
browser that supports `HTML5 Audio`_.
|
||||
- Analyze music files' metadata from the command line.
|
||||
- Listen to your library with a music player that speaks the `MPD`_ protocol
|
||||
and works with a staggering variety of interfaces.
|
||||
|
||||
If beets doesn't do what you want yet, `writing your own plugin`_ is
|
||||
shockingly simple if you know a little Python.
|
||||
|
||||
.. _plugins: http://beets.readthedocs.org/page/plugins/
|
||||
.. _MPD: http://www.musicpd.org/
|
||||
.. _MusicBrainz music collection: http://musicbrainz.org/doc/Collections/
|
||||
.. _writing your own plugin:
|
||||
http://beets.readthedocs.org/page/dev/plugins.html
|
||||
.. _HTML5 Audio:
|
||||
http://www.w3.org/TR/html-markup/audio.html
|
||||
.. _albums that are missing tracks:
|
||||
http://beets.readthedocs.org/page/plugins/missing.html
|
||||
.. _duplicate tracks and albums:
|
||||
http://beets.readthedocs.org/page/plugins/duplicates.html
|
||||
.. _Transcode audio:
|
||||
http://beets.readthedocs.org/page/plugins/convert.html
|
||||
.. _Beatport: http://www.beatport.com/
|
||||
.. _Discogs: http://www.discogs.com/
|
||||
.. _acoustic fingerprints:
|
||||
http://beets.readthedocs.org/page/plugins/chroma.html
|
||||
.. _ReplayGain: http://beets.readthedocs.org/page/plugins/replaygain.html
|
||||
.. _tempos: http://beets.readthedocs.org/page/plugins/echonest.html
|
||||
.. _genres: http://beets.readthedocs.org/page/plugins/lastgenre.html
|
||||
.. _album art: http://beets.readthedocs.org/page/plugins/fetchart.html
|
||||
.. _lyrics: http://beets.readthedocs.org/page/plugins/lyrics.html
|
||||
.. _MusicBrainz: http://musicbrainz.org/
|
||||
|
||||
Read More
|
||||
---------
|
||||
|
||||
Learn more about beets at `its Web site`_. Follow `@b33ts`_ on Twitter for
|
||||
news and updates.
|
||||
|
||||
You can install beets by typing ``pip install beets``. Then check out the
|
||||
`Getting Started`_ guide.
|
||||
|
||||
.. _its Web site: http://beets.radbox.org/
|
||||
.. _Getting Started: http://beets.readthedocs.org/page/guides/main.html
|
||||
.. _@b33ts: http://twitter.com/b33ts/
|
||||
|
||||
Authors
|
||||
-------
|
||||
|
||||
Beets is by `Adrian Sampson`_ with a supporting cast of thousands. For help,
|
||||
please contact the `mailing list`_.
|
||||
|
||||
.. _mailing list: https://groups.google.com/forum/#!forum/beets-users
|
||||
.. _Adrian Sampson: http://homes.cs.washington.edu/~asampson/
|
||||
@@ -0,0 +1,28 @@
|
||||
# This file is part of beets.
|
||||
# Copyright 2014, Adrian Sampson.
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining
|
||||
# a copy of this software and associated documentation files (the
|
||||
# "Software"), to deal in the Software without restriction, including
|
||||
# without limitation the rights to use, copy, modify, merge, publish,
|
||||
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||
# permit persons to whom the Software is furnished to do so, subject to
|
||||
# the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
|
||||
# This particular version has been slightly modified to work with Headphones
|
||||
# https://github.com/rembo10/headphones
|
||||
|
||||
__version__ = '1.3.10-headphones'
|
||||
__author__ = 'Adrian Sampson <adrian@radbox.org>'
|
||||
|
||||
import os
|
||||
|
||||
import beets.library
|
||||
from beets.util import confit
|
||||
|
||||
Library = beets.library.Library
|
||||
|
||||
config = confit.LazyConfig(os.path.dirname(__file__), __name__)
|
||||
@@ -0,0 +1,136 @@
|
||||
# This file is part of beets.
|
||||
# Copyright 2013, Adrian Sampson.
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining
|
||||
# a copy of this software and associated documentation files (the
|
||||
# "Software"), to deal in the Software without restriction, including
|
||||
# without limitation the rights to use, copy, modify, merge, publish,
|
||||
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||
# permit persons to whom the Software is furnished to do so, subject to
|
||||
# the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
|
||||
"""Facilities for automatically determining files' correct metadata.
|
||||
"""
|
||||
import logging
|
||||
|
||||
from beets import config
|
||||
|
||||
# Parts of external interface.
|
||||
from .hooks import AlbumInfo, TrackInfo, AlbumMatch, TrackMatch # noqa
|
||||
from .match import tag_item, tag_album # noqa
|
||||
from .match import Recommendation # noqa
|
||||
|
||||
# Global logger.
|
||||
log = logging.getLogger('beets')
|
||||
|
||||
|
||||
# Additional utilities for the main interface.
|
||||
|
||||
def apply_item_metadata(item, track_info):
|
||||
"""Set an item's metadata from its matched TrackInfo object.
|
||||
"""
|
||||
item.artist = track_info.artist
|
||||
item.artist_sort = track_info.artist_sort
|
||||
item.artist_credit = track_info.artist_credit
|
||||
item.title = track_info.title
|
||||
item.mb_trackid = track_info.track_id
|
||||
if track_info.artist_id:
|
||||
item.mb_artistid = track_info.artist_id
|
||||
# At the moment, the other metadata is left intact (including album
|
||||
# and track number). Perhaps these should be emptied?
|
||||
|
||||
|
||||
def apply_metadata(album_info, mapping):
|
||||
"""Set the items' metadata to match an AlbumInfo object using a
|
||||
mapping from Items to TrackInfo objects.
|
||||
"""
|
||||
for item, track_info in mapping.iteritems():
|
||||
# Album, artist, track count.
|
||||
if track_info.artist:
|
||||
item.artist = track_info.artist
|
||||
else:
|
||||
item.artist = album_info.artist
|
||||
item.albumartist = album_info.artist
|
||||
item.album = album_info.album
|
||||
|
||||
# Artist sort and credit names.
|
||||
item.artist_sort = track_info.artist_sort or album_info.artist_sort
|
||||
item.artist_credit = (track_info.artist_credit or
|
||||
album_info.artist_credit)
|
||||
item.albumartist_sort = album_info.artist_sort
|
||||
item.albumartist_credit = album_info.artist_credit
|
||||
|
||||
# Release date.
|
||||
for prefix in '', 'original_':
|
||||
if config['original_date'] and not prefix:
|
||||
# Ignore specific release date.
|
||||
continue
|
||||
|
||||
for suffix in 'year', 'month', 'day':
|
||||
key = prefix + suffix
|
||||
value = getattr(album_info, key) or 0
|
||||
|
||||
# If we don't even have a year, apply nothing.
|
||||
if suffix == 'year' and not value:
|
||||
break
|
||||
|
||||
# Otherwise, set the fetched value (or 0 for the month
|
||||
# and day if not available).
|
||||
item[key] = value
|
||||
|
||||
# If we're using original release date for both fields,
|
||||
# also set item.year = info.original_year, etc.
|
||||
if config['original_date']:
|
||||
item[suffix] = value
|
||||
|
||||
# Title.
|
||||
item.title = track_info.title
|
||||
|
||||
if config['per_disc_numbering']:
|
||||
item.track = track_info.medium_index or track_info.index
|
||||
item.tracktotal = track_info.medium_total or len(album_info.tracks)
|
||||
else:
|
||||
item.track = track_info.index
|
||||
item.tracktotal = len(album_info.tracks)
|
||||
|
||||
# Disc and disc count.
|
||||
item.disc = track_info.medium
|
||||
item.disctotal = album_info.mediums
|
||||
|
||||
# MusicBrainz IDs.
|
||||
item.mb_trackid = track_info.track_id
|
||||
item.mb_albumid = album_info.album_id
|
||||
if track_info.artist_id:
|
||||
item.mb_artistid = track_info.artist_id
|
||||
else:
|
||||
item.mb_artistid = album_info.artist_id
|
||||
item.mb_albumartistid = album_info.artist_id
|
||||
item.mb_releasegroupid = album_info.releasegroup_id
|
||||
|
||||
# Compilation flag.
|
||||
item.comp = album_info.va
|
||||
|
||||
# Miscellaneous metadata.
|
||||
for field in ('albumtype',
|
||||
'label',
|
||||
'asin',
|
||||
'catalognum',
|
||||
'script',
|
||||
'language',
|
||||
'country',
|
||||
'albumstatus',
|
||||
'albumdisambig'):
|
||||
value = getattr(album_info, field)
|
||||
if value is not None:
|
||||
item[field] = value
|
||||
if track_info.disctitle is not None:
|
||||
item.disctitle = track_info.disctitle
|
||||
|
||||
if track_info.media is not None:
|
||||
item.media = track_info.media
|
||||
|
||||
# Headphones seal of approval
|
||||
item.comments = 'tagged by headphones/beets'
|
||||
@@ -0,0 +1,579 @@
|
||||
# This file is part of beets.
|
||||
# Copyright 2013, Adrian Sampson.
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining
|
||||
# a copy of this software and associated documentation files (the
|
||||
# "Software"), to deal in the Software without restriction, including
|
||||
# without limitation the rights to use, copy, modify, merge, publish,
|
||||
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||
# permit persons to whom the Software is furnished to do so, subject to
|
||||
# the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
|
||||
"""Glue between metadata sources and the matching logic."""
|
||||
import logging
|
||||
from collections import namedtuple
|
||||
import re
|
||||
|
||||
from beets import plugins
|
||||
from beets import config
|
||||
from beets.autotag import mb
|
||||
from beets.util import levenshtein
|
||||
from unidecode import unidecode
|
||||
|
||||
log = logging.getLogger('beets')
|
||||
|
||||
|
||||
# Classes used to represent candidate options.
|
||||
|
||||
class AlbumInfo(object):
|
||||
"""Describes a canonical release that may be used to match a release
|
||||
in the library. Consists of these data members:
|
||||
|
||||
- ``album``: the release title
|
||||
- ``album_id``: MusicBrainz ID; UUID fragment only
|
||||
- ``artist``: name of the release's primary artist
|
||||
- ``artist_id``
|
||||
- ``tracks``: list of TrackInfo objects making up the release
|
||||
- ``asin``: Amazon ASIN
|
||||
- ``albumtype``: string describing the kind of release
|
||||
- ``va``: boolean: whether the release has "various artists"
|
||||
- ``year``: release year
|
||||
- ``month``: release month
|
||||
- ``day``: release day
|
||||
- ``label``: music label responsible for the release
|
||||
- ``mediums``: the number of discs in this release
|
||||
- ``artist_sort``: name of the release's artist for sorting
|
||||
- ``releasegroup_id``: MBID for the album's release group
|
||||
- ``catalognum``: the label's catalog number for the release
|
||||
- ``script``: character set used for metadata
|
||||
- ``language``: human language of the metadata
|
||||
- ``country``: the release country
|
||||
- ``albumstatus``: MusicBrainz release status (Official, etc.)
|
||||
- ``media``: delivery mechanism (Vinyl, etc.)
|
||||
- ``albumdisambig``: MusicBrainz release disambiguation comment
|
||||
- ``artist_credit``: Release-specific artist name
|
||||
- ``data_source``: The original data source (MusicBrainz, Discogs, etc.)
|
||||
- ``data_url``: The data source release URL.
|
||||
|
||||
The fields up through ``tracks`` are required. The others are
|
||||
optional and may be None.
|
||||
"""
|
||||
def __init__(self, album, album_id, artist, artist_id, tracks, asin=None,
|
||||
albumtype=None, va=False, year=None, month=None, day=None,
|
||||
label=None, mediums=None, artist_sort=None,
|
||||
releasegroup_id=None, catalognum=None, script=None,
|
||||
language=None, country=None, albumstatus=None, media=None,
|
||||
albumdisambig=None, artist_credit=None, original_year=None,
|
||||
original_month=None, original_day=None, data_source=None,
|
||||
data_url=None):
|
||||
self.album = album
|
||||
self.album_id = album_id
|
||||
self.artist = artist
|
||||
self.artist_id = artist_id
|
||||
self.tracks = tracks
|
||||
self.asin = asin
|
||||
self.albumtype = albumtype
|
||||
self.va = va
|
||||
self.year = year
|
||||
self.month = month
|
||||
self.day = day
|
||||
self.label = label
|
||||
self.mediums = mediums
|
||||
self.artist_sort = artist_sort
|
||||
self.releasegroup_id = releasegroup_id
|
||||
self.catalognum = catalognum
|
||||
self.script = script
|
||||
self.language = language
|
||||
self.country = country
|
||||
self.albumstatus = albumstatus
|
||||
self.media = media
|
||||
self.albumdisambig = albumdisambig
|
||||
self.artist_credit = artist_credit
|
||||
self.original_year = original_year
|
||||
self.original_month = original_month
|
||||
self.original_day = original_day
|
||||
self.data_source = data_source
|
||||
self.data_url = data_url
|
||||
|
||||
# Work around a bug in python-musicbrainz-ngs that causes some
|
||||
# strings to be bytes rather than Unicode.
|
||||
# https://github.com/alastair/python-musicbrainz-ngs/issues/85
|
||||
def decode(self, codec='utf8'):
|
||||
"""Ensure that all string attributes on this object, and the
|
||||
constituent `TrackInfo` objects, are decoded to Unicode.
|
||||
"""
|
||||
for fld in ['album', 'artist', 'albumtype', 'label', 'artist_sort',
|
||||
'catalognum', 'script', 'language', 'country',
|
||||
'albumstatus', 'albumdisambig', 'artist_credit', 'media']:
|
||||
value = getattr(self, fld)
|
||||
if isinstance(value, str):
|
||||
setattr(self, fld, value.decode(codec, 'ignore'))
|
||||
|
||||
if self.tracks:
|
||||
for track in self.tracks:
|
||||
track.decode(codec)
|
||||
|
||||
|
||||
class TrackInfo(object):
|
||||
"""Describes a canonical track present on a release. Appears as part
|
||||
of an AlbumInfo's ``tracks`` list. Consists of these data members:
|
||||
|
||||
- ``title``: name of the track
|
||||
- ``track_id``: MusicBrainz ID; UUID fragment only
|
||||
- ``artist``: individual track artist name
|
||||
- ``artist_id``
|
||||
- ``length``: float: duration of the track in seconds
|
||||
- ``index``: position on the entire release
|
||||
- ``media``: delivery mechanism (Vinyl, etc.)
|
||||
- ``medium``: the disc number this track appears on in the album
|
||||
- ``medium_index``: the track's position on the disc
|
||||
- ``medium_total``: the number of tracks on the item's disc
|
||||
- ``artist_sort``: name of the track artist for sorting
|
||||
- ``disctitle``: name of the individual medium (subtitle)
|
||||
- ``artist_credit``: Recording-specific artist name
|
||||
|
||||
Only ``title`` and ``track_id`` are required. The rest of the fields
|
||||
may be None. The indices ``index``, ``medium``, and ``medium_index``
|
||||
are all 1-based.
|
||||
"""
|
||||
def __init__(self, title, track_id, artist=None, artist_id=None,
|
||||
length=None, index=None, medium=None, medium_index=None,
|
||||
medium_total=None, artist_sort=None, disctitle=None,
|
||||
artist_credit=None, data_source=None, data_url=None,
|
||||
media=None):
|
||||
self.title = title
|
||||
self.track_id = track_id
|
||||
self.artist = artist
|
||||
self.artist_id = artist_id
|
||||
self.length = length
|
||||
self.index = index
|
||||
self.media = media
|
||||
self.medium = medium
|
||||
self.medium_index = medium_index
|
||||
self.medium_total = medium_total
|
||||
self.artist_sort = artist_sort
|
||||
self.disctitle = disctitle
|
||||
self.artist_credit = artist_credit
|
||||
self.data_source = data_source
|
||||
self.data_url = data_url
|
||||
|
||||
# As above, work around a bug in python-musicbrainz-ngs.
|
||||
def decode(self, codec='utf8'):
|
||||
"""Ensure that all string attributes on this object are decoded
|
||||
to Unicode.
|
||||
"""
|
||||
for fld in ['title', 'artist', 'medium', 'artist_sort', 'disctitle',
|
||||
'artist_credit', 'media']:
|
||||
value = getattr(self, fld)
|
||||
if isinstance(value, str):
|
||||
setattr(self, fld, value.decode(codec, 'ignore'))
|
||||
|
||||
|
||||
# Candidate distance scoring.
|
||||
|
||||
# Parameters for string distance function.
|
||||
# Words that can be moved to the end of a string using a comma.
|
||||
SD_END_WORDS = ['the', 'a', 'an']
|
||||
# Reduced weights for certain portions of the string.
|
||||
SD_PATTERNS = [
|
||||
(r'^the ', 0.1),
|
||||
(r'[\[\(]?(ep|single)[\]\)]?', 0.0),
|
||||
(r'[\[\(]?(featuring|feat|ft)[\. :].+', 0.1),
|
||||
(r'\(.*?\)', 0.3),
|
||||
(r'\[.*?\]', 0.3),
|
||||
(r'(, )?(pt\.|part) .+', 0.2),
|
||||
]
|
||||
# Replacements to use before testing distance.
|
||||
SD_REPLACE = [
|
||||
(r'&', 'and'),
|
||||
]
|
||||
|
||||
|
||||
def _string_dist_basic(str1, str2):
|
||||
"""Basic edit distance between two strings, ignoring
|
||||
non-alphanumeric characters and case. Comparisons are based on a
|
||||
transliteration/lowering to ASCII characters. Normalized by string
|
||||
length.
|
||||
"""
|
||||
str1 = unidecode(str1)
|
||||
str2 = unidecode(str2)
|
||||
str1 = re.sub(r'[^a-z0-9]', '', str1.lower())
|
||||
str2 = re.sub(r'[^a-z0-9]', '', str2.lower())
|
||||
if not str1 and not str2:
|
||||
return 0.0
|
||||
return levenshtein(str1, str2) / float(max(len(str1), len(str2)))
|
||||
|
||||
|
||||
def string_dist(str1, str2):
|
||||
"""Gives an "intuitive" edit distance between two strings. This is
|
||||
an edit distance, normalized by the string length, with a number of
|
||||
tweaks that reflect intuition about text.
|
||||
"""
|
||||
if str1 is None and str2 is None:
|
||||
return 0.0
|
||||
if str1 is None or str2 is None:
|
||||
return 1.0
|
||||
|
||||
str1 = str1.lower()
|
||||
str2 = str2.lower()
|
||||
|
||||
# Don't penalize strings that move certain words to the end. For
|
||||
# example, "the something" should be considered equal to
|
||||
# "something, the".
|
||||
for word in SD_END_WORDS:
|
||||
if str1.endswith(', %s' % word):
|
||||
str1 = '%s %s' % (word, str1[:-len(word) - 2])
|
||||
if str2.endswith(', %s' % word):
|
||||
str2 = '%s %s' % (word, str2[:-len(word) - 2])
|
||||
|
||||
# Perform a couple of basic normalizing substitutions.
|
||||
for pat, repl in SD_REPLACE:
|
||||
str1 = re.sub(pat, repl, str1)
|
||||
str2 = re.sub(pat, repl, str2)
|
||||
|
||||
# Change the weight for certain string portions matched by a set
|
||||
# of regular expressions. We gradually change the strings and build
|
||||
# up penalties associated with parts of the string that were
|
||||
# deleted.
|
||||
base_dist = _string_dist_basic(str1, str2)
|
||||
penalty = 0.0
|
||||
for pat, weight in SD_PATTERNS:
|
||||
# Get strings that drop the pattern.
|
||||
case_str1 = re.sub(pat, '', str1)
|
||||
case_str2 = re.sub(pat, '', str2)
|
||||
|
||||
if case_str1 != str1 or case_str2 != str2:
|
||||
# If the pattern was present (i.e., it is deleted in the
|
||||
# the current case), recalculate the distances for the
|
||||
# modified strings.
|
||||
case_dist = _string_dist_basic(case_str1, case_str2)
|
||||
case_delta = max(0.0, base_dist - case_dist)
|
||||
if case_delta == 0.0:
|
||||
continue
|
||||
|
||||
# Shift our baseline strings down (to avoid rematching the
|
||||
# same part of the string) and add a scaled distance
|
||||
# amount to the penalties.
|
||||
str1 = case_str1
|
||||
str2 = case_str2
|
||||
base_dist = case_dist
|
||||
penalty += weight * case_delta
|
||||
|
||||
return base_dist + penalty
|
||||
|
||||
|
||||
class LazyClassProperty(object):
|
||||
"""A decorator implementing a read-only property that is *lazy* in
|
||||
the sense that the getter is only invoked once. Subsequent accesses
|
||||
through *any* instance use the cached result.
|
||||
"""
|
||||
def __init__(self, getter):
|
||||
self.getter = getter
|
||||
self.computed = False
|
||||
|
||||
def __get__(self, obj, owner):
|
||||
if not self.computed:
|
||||
self.value = self.getter(owner)
|
||||
self.computed = True
|
||||
return self.value
|
||||
|
||||
|
||||
class Distance(object):
|
||||
"""Keeps track of multiple distance penalties. Provides a single
|
||||
weighted distance for all penalties as well as a weighted distance
|
||||
for each individual penalty.
|
||||
"""
|
||||
def __init__(self):
|
||||
self._penalties = {}
|
||||
|
||||
@LazyClassProperty
|
||||
def _weights(cls):
|
||||
"""A dictionary from keys to floating-point weights.
|
||||
"""
|
||||
weights_view = config['match']['distance_weights']
|
||||
weights = {}
|
||||
for key in weights_view.keys():
|
||||
weights[key] = weights_view[key].as_number()
|
||||
return weights
|
||||
|
||||
# Access the components and their aggregates.
|
||||
|
||||
@property
|
||||
def distance(self):
|
||||
"""Return a weighted and normalized distance across all
|
||||
penalties.
|
||||
"""
|
||||
dist_max = self.max_distance
|
||||
if dist_max:
|
||||
return self.raw_distance / self.max_distance
|
||||
return 0.0
|
||||
|
||||
@property
|
||||
def max_distance(self):
|
||||
"""Return the maximum distance penalty (normalization factor).
|
||||
"""
|
||||
dist_max = 0.0
|
||||
for key, penalty in self._penalties.iteritems():
|
||||
dist_max += len(penalty) * self._weights[key]
|
||||
return dist_max
|
||||
|
||||
@property
|
||||
def raw_distance(self):
|
||||
"""Return the raw (denormalized) distance.
|
||||
"""
|
||||
dist_raw = 0.0
|
||||
for key, penalty in self._penalties.iteritems():
|
||||
dist_raw += sum(penalty) * self._weights[key]
|
||||
return dist_raw
|
||||
|
||||
def items(self):
|
||||
"""Return a list of (key, dist) pairs, with `dist` being the
|
||||
weighted distance, sorted from highest to lowest. Does not
|
||||
include penalties with a zero value.
|
||||
"""
|
||||
list_ = []
|
||||
for key in self._penalties:
|
||||
dist = self[key]
|
||||
if dist:
|
||||
list_.append((key, dist))
|
||||
# Convert distance into a negative float we can sort items in
|
||||
# ascending order (for keys, when the penalty is equal) and
|
||||
# still get the items with the biggest distance first.
|
||||
return sorted(list_, key=lambda (key, dist): (0 - dist, key))
|
||||
|
||||
# Behave like a float.
|
||||
|
||||
def __cmp__(self, other):
|
||||
return cmp(self.distance, other)
|
||||
|
||||
def __float__(self):
|
||||
return self.distance
|
||||
|
||||
def __sub__(self, other):
|
||||
return self.distance - other
|
||||
|
||||
def __rsub__(self, other):
|
||||
return other - self.distance
|
||||
|
||||
# Behave like a dict.
|
||||
|
||||
def __getitem__(self, key):
|
||||
"""Returns the weighted distance for a named penalty.
|
||||
"""
|
||||
dist = sum(self._penalties[key]) * self._weights[key]
|
||||
dist_max = self.max_distance
|
||||
if dist_max:
|
||||
return dist / dist_max
|
||||
return 0.0
|
||||
|
||||
def __iter__(self):
|
||||
return iter(self.items())
|
||||
|
||||
def __len__(self):
|
||||
return len(self.items())
|
||||
|
||||
def keys(self):
|
||||
return [key for key, _ in self.items()]
|
||||
|
||||
def update(self, dist):
|
||||
"""Adds all the distance penalties from `dist`.
|
||||
"""
|
||||
if not isinstance(dist, Distance):
|
||||
raise ValueError(
|
||||
'`dist` must be a Distance object, not {0}'.format(type(dist))
|
||||
)
|
||||
for key, penalties in dist._penalties.iteritems():
|
||||
self._penalties.setdefault(key, []).extend(penalties)
|
||||
|
||||
# Adding components.
|
||||
|
||||
def _eq(self, value1, value2):
|
||||
"""Returns True if `value1` is equal to `value2`. `value1` may
|
||||
be a compiled regular expression, in which case it will be
|
||||
matched against `value2`.
|
||||
"""
|
||||
if isinstance(value1, re._pattern_type):
|
||||
return bool(value1.match(value2))
|
||||
return value1 == value2
|
||||
|
||||
def add(self, key, dist):
|
||||
"""Adds a distance penalty. `key` must correspond with a
|
||||
configured weight setting. `dist` must be a float between 0.0
|
||||
and 1.0, and will be added to any existing distance penalties
|
||||
for the same key.
|
||||
"""
|
||||
if not 0.0 <= dist <= 1.0:
|
||||
raise ValueError(
|
||||
'`dist` must be between 0.0 and 1.0, not {0}'.format(dist)
|
||||
)
|
||||
self._penalties.setdefault(key, []).append(dist)
|
||||
|
||||
def add_equality(self, key, value, options):
|
||||
"""Adds a distance penalty of 1.0 if `value` doesn't match any
|
||||
of the values in `options`. If an option is a compiled regular
|
||||
expression, it will be considered equal if it matches against
|
||||
`value`.
|
||||
"""
|
||||
if not isinstance(options, (list, tuple)):
|
||||
options = [options]
|
||||
for opt in options:
|
||||
if self._eq(opt, value):
|
||||
dist = 0.0
|
||||
break
|
||||
else:
|
||||
dist = 1.0
|
||||
self.add(key, dist)
|
||||
|
||||
def add_expr(self, key, expr):
|
||||
"""Adds a distance penalty of 1.0 if `expr` evaluates to True,
|
||||
or 0.0.
|
||||
"""
|
||||
if expr:
|
||||
self.add(key, 1.0)
|
||||
else:
|
||||
self.add(key, 0.0)
|
||||
|
||||
def add_number(self, key, number1, number2):
|
||||
"""Adds a distance penalty of 1.0 for each number of difference
|
||||
between `number1` and `number2`, or 0.0 when there is no
|
||||
difference. Use this when there is no upper limit on the
|
||||
difference between the two numbers.
|
||||
"""
|
||||
diff = abs(number1 - number2)
|
||||
if diff:
|
||||
for i in range(diff):
|
||||
self.add(key, 1.0)
|
||||
else:
|
||||
self.add(key, 0.0)
|
||||
|
||||
def add_priority(self, key, value, options):
|
||||
"""Adds a distance penalty that corresponds to the position at
|
||||
which `value` appears in `options`. A distance penalty of 0.0
|
||||
for the first option, or 1.0 if there is no matching option. If
|
||||
an option is a compiled regular expression, it will be
|
||||
considered equal if it matches against `value`.
|
||||
"""
|
||||
if not isinstance(options, (list, tuple)):
|
||||
options = [options]
|
||||
unit = 1.0 / (len(options) or 1)
|
||||
for i, opt in enumerate(options):
|
||||
if self._eq(opt, value):
|
||||
dist = i * unit
|
||||
break
|
||||
else:
|
||||
dist = 1.0
|
||||
self.add(key, dist)
|
||||
|
||||
def add_ratio(self, key, number1, number2):
|
||||
"""Adds a distance penalty for `number1` as a ratio of `number2`.
|
||||
`number1` is bound at 0 and `number2`.
|
||||
"""
|
||||
number = float(max(min(number1, number2), 0))
|
||||
if number2:
|
||||
dist = number / number2
|
||||
else:
|
||||
dist = 0.0
|
||||
self.add(key, dist)
|
||||
|
||||
def add_string(self, key, str1, str2):
|
||||
"""Adds a distance penalty based on the edit distance between
|
||||
`str1` and `str2`.
|
||||
"""
|
||||
dist = string_dist(str1, str2)
|
||||
self.add(key, dist)
|
||||
|
||||
|
||||
# Structures that compose all the information for a candidate match.
|
||||
|
||||
AlbumMatch = namedtuple('AlbumMatch', ['distance', 'info', 'mapping',
|
||||
'extra_items', 'extra_tracks'])
|
||||
|
||||
TrackMatch = namedtuple('TrackMatch', ['distance', 'info'])
|
||||
|
||||
|
||||
# Aggregation of sources.
|
||||
|
||||
def album_for_mbid(release_id):
|
||||
"""Get an AlbumInfo object for a MusicBrainz release ID. Return None
|
||||
if the ID is not found.
|
||||
"""
|
||||
try:
|
||||
return mb.album_for_id(release_id)
|
||||
except mb.MusicBrainzAPIError as exc:
|
||||
exc.log(log)
|
||||
|
||||
|
||||
def track_for_mbid(recording_id):
|
||||
"""Get a TrackInfo object for a MusicBrainz recording ID. Return None
|
||||
if the ID is not found.
|
||||
"""
|
||||
try:
|
||||
return mb.track_for_id(recording_id)
|
||||
except mb.MusicBrainzAPIError as exc:
|
||||
exc.log(log)
|
||||
|
||||
|
||||
def albums_for_id(album_id):
|
||||
"""Get a list of albums for an ID."""
|
||||
candidates = [album_for_mbid(album_id)]
|
||||
candidates.extend(plugins.album_for_id(album_id))
|
||||
return filter(None, candidates)
|
||||
|
||||
|
||||
def tracks_for_id(track_id):
|
||||
"""Get a list of tracks for an ID."""
|
||||
candidates = [track_for_mbid(track_id)]
|
||||
candidates.extend(plugins.track_for_id(track_id))
|
||||
return filter(None, candidates)
|
||||
|
||||
|
||||
def album_candidates(items, artist, album, va_likely):
|
||||
"""Search for album matches. ``items`` is a list of Item objects
|
||||
that make up the album. ``artist`` and ``album`` are the respective
|
||||
names (strings), which may be derived from the item list or may be
|
||||
entered by the user. ``va_likely`` is a boolean indicating whether
|
||||
the album is likely to be a "various artists" release.
|
||||
"""
|
||||
out = []
|
||||
|
||||
# Base candidates if we have album and artist to match.
|
||||
if artist and album:
|
||||
try:
|
||||
out.extend(mb.match_album(artist, album, len(items)))
|
||||
except mb.MusicBrainzAPIError as exc:
|
||||
exc.log(log)
|
||||
|
||||
# Also add VA matches from MusicBrainz where appropriate.
|
||||
if va_likely and album:
|
||||
try:
|
||||
out.extend(mb.match_album(None, album, len(items)))
|
||||
except mb.MusicBrainzAPIError as exc:
|
||||
exc.log(log)
|
||||
|
||||
# Candidates from plugins.
|
||||
out.extend(plugins.candidates(items, artist, album, va_likely))
|
||||
|
||||
return out
|
||||
|
||||
|
||||
def item_candidates(item, artist, title):
|
||||
"""Search for item matches. ``item`` is the Item to be matched.
|
||||
``artist`` and ``title`` are strings and either reflect the item or
|
||||
are specified by the user.
|
||||
"""
|
||||
out = []
|
||||
|
||||
# MusicBrainz candidates.
|
||||
if artist and title:
|
||||
try:
|
||||
out.extend(mb.match_track(artist, title))
|
||||
except mb.MusicBrainzAPIError as exc:
|
||||
exc.log(log)
|
||||
|
||||
# Plugin candidates.
|
||||
out.extend(plugins.item_candidates(item, artist, title))
|
||||
|
||||
return out
|
||||
@@ -0,0 +1,492 @@
|
||||
# This file is part of beets.
|
||||
# Copyright 2013, Adrian Sampson.
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining
|
||||
# a copy of this software and associated documentation files (the
|
||||
# "Software"), to deal in the Software without restriction, including
|
||||
# without limitation the rights to use, copy, modify, merge, publish,
|
||||
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||
# permit persons to whom the Software is furnished to do so, subject to
|
||||
# the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
|
||||
"""Matches existing metadata with canonical information to identify
|
||||
releases and tracks.
|
||||
"""
|
||||
from __future__ import division
|
||||
|
||||
import datetime
|
||||
import logging
|
||||
import re
|
||||
from munkres import Munkres
|
||||
|
||||
from beets import plugins
|
||||
from beets import config
|
||||
from beets.util import plurality
|
||||
from beets.autotag import hooks
|
||||
from beets.util.enumeration import OrderedEnum
|
||||
|
||||
# Artist signals that indicate "various artists". These are used at the
|
||||
# album level to determine whether a given release is likely a VA
|
||||
# release and also on the track level to to remove the penalty for
|
||||
# differing artists.
|
||||
VA_ARTISTS = (u'', u'various artists', u'various', u'va', u'unknown')
|
||||
|
||||
# Global logger.
|
||||
log = logging.getLogger('beets')
|
||||
|
||||
|
||||
# Recommendation enumeration.
|
||||
|
||||
class Recommendation(OrderedEnum):
|
||||
"""Indicates a qualitative suggestion to the user about what should
|
||||
be done with a given match.
|
||||
"""
|
||||
none = 0
|
||||
low = 1
|
||||
medium = 2
|
||||
strong = 3
|
||||
|
||||
|
||||
# Primary matching functionality.
|
||||
|
||||
def current_metadata(items):
|
||||
"""Extract the likely current metadata for an album given a list of its
|
||||
items. Return two dictionaries:
|
||||
- The most common value for each field.
|
||||
- Whether each field's value was unanimous (values are booleans).
|
||||
"""
|
||||
assert items # Must be nonempty.
|
||||
|
||||
likelies = {}
|
||||
consensus = {}
|
||||
fields = ['artist', 'album', 'albumartist', 'year', 'disctotal',
|
||||
'mb_albumid', 'label', 'catalognum', 'country', 'media',
|
||||
'albumdisambig']
|
||||
for field in fields:
|
||||
values = [item[field] for item in items if item]
|
||||
likelies[field], freq = plurality(values)
|
||||
consensus[field] = (freq == len(values))
|
||||
|
||||
# If there's an album artist consensus, use this for the artist.
|
||||
if consensus['albumartist'] and likelies['albumartist']:
|
||||
likelies['artist'] = likelies['albumartist']
|
||||
|
||||
return likelies, consensus
|
||||
|
||||
|
||||
def assign_items(items, tracks):
|
||||
"""Given a list of Items and a list of TrackInfo objects, find the
|
||||
best mapping between them. Returns a mapping from Items to TrackInfo
|
||||
objects, a set of extra Items, and a set of extra TrackInfo
|
||||
objects. These "extra" objects occur when there is an unequal number
|
||||
of objects of the two types.
|
||||
"""
|
||||
# Construct the cost matrix.
|
||||
costs = []
|
||||
for item in items:
|
||||
row = []
|
||||
for i, track in enumerate(tracks):
|
||||
row.append(track_distance(item, track))
|
||||
costs.append(row)
|
||||
|
||||
# Find a minimum-cost bipartite matching.
|
||||
matching = Munkres().compute(costs)
|
||||
|
||||
# Produce the output matching.
|
||||
mapping = dict((items[i], tracks[j]) for (i, j) in matching)
|
||||
extra_items = list(set(items) - set(mapping.keys()))
|
||||
extra_items.sort(key=lambda i: (i.disc, i.track, i.title))
|
||||
extra_tracks = list(set(tracks) - set(mapping.values()))
|
||||
extra_tracks.sort(key=lambda t: (t.index, t.title))
|
||||
return mapping, extra_items, extra_tracks
|
||||
|
||||
|
||||
def track_index_changed(item, track_info):
|
||||
"""Returns True if the item and track info index is different. Tolerates
|
||||
per disc and per release numbering.
|
||||
"""
|
||||
return item.track not in (track_info.medium_index, track_info.index)
|
||||
|
||||
|
||||
def track_distance(item, track_info, incl_artist=False):
|
||||
"""Determines the significance of a track metadata change. Returns a
|
||||
Distance object. `incl_artist` indicates that a distance component should
|
||||
be included for the track artist (i.e., for various-artist releases).
|
||||
"""
|
||||
dist = hooks.Distance()
|
||||
|
||||
# Length.
|
||||
if track_info.length:
|
||||
diff = abs(item.length - track_info.length) - \
|
||||
config['match']['track_length_grace'].as_number()
|
||||
dist.add_ratio('track_length', diff,
|
||||
config['match']['track_length_max'].as_number())
|
||||
|
||||
# Title.
|
||||
dist.add_string('track_title', item.title, track_info.title)
|
||||
|
||||
# Artist. Only check if there is actually an artist in the track data.
|
||||
if incl_artist and track_info.artist and \
|
||||
item.artist.lower() not in VA_ARTISTS:
|
||||
dist.add_string('track_artist', item.artist, track_info.artist)
|
||||
|
||||
# Track index.
|
||||
if track_info.index and item.track:
|
||||
dist.add_expr('track_index', track_index_changed(item, track_info))
|
||||
|
||||
# Track ID.
|
||||
if item.mb_trackid:
|
||||
dist.add_expr('track_id', item.mb_trackid != track_info.track_id)
|
||||
|
||||
# Plugins.
|
||||
dist.update(plugins.track_distance(item, track_info))
|
||||
|
||||
return dist
|
||||
|
||||
|
||||
def distance(items, album_info, mapping):
|
||||
"""Determines how "significant" an album metadata change would be.
|
||||
Returns a Distance object. `album_info` is an AlbumInfo object
|
||||
reflecting the album to be compared. `items` is a sequence of all
|
||||
Item objects that will be matched (order is not important).
|
||||
`mapping` is a dictionary mapping Items to TrackInfo objects; the
|
||||
keys are a subset of `items` and the values are a subset of
|
||||
`album_info.tracks`.
|
||||
"""
|
||||
likelies, _ = current_metadata(items)
|
||||
|
||||
dist = hooks.Distance()
|
||||
|
||||
# Artist, if not various.
|
||||
if not album_info.va:
|
||||
dist.add_string('artist', likelies['artist'], album_info.artist)
|
||||
|
||||
# Album.
|
||||
dist.add_string('album', likelies['album'], album_info.album)
|
||||
|
||||
# Current or preferred media.
|
||||
if album_info.media:
|
||||
# Preferred media options.
|
||||
patterns = config['match']['preferred']['media'].as_str_seq()
|
||||
options = [re.compile(r'(\d+x)?(%s)' % pat, re.I) for pat in patterns]
|
||||
if options:
|
||||
dist.add_priority('media', album_info.media, options)
|
||||
# Current media.
|
||||
elif likelies['media']:
|
||||
dist.add_equality('media', album_info.media, likelies['media'])
|
||||
|
||||
# Mediums.
|
||||
if likelies['disctotal'] and album_info.mediums:
|
||||
dist.add_number('mediums', likelies['disctotal'], album_info.mediums)
|
||||
|
||||
# Prefer earliest release.
|
||||
if album_info.year and config['match']['preferred']['original_year']:
|
||||
# Assume 1889 (earliest first gramophone discs) if we don't know the
|
||||
# original year.
|
||||
original = album_info.original_year or 1889
|
||||
diff = abs(album_info.year - original)
|
||||
diff_max = abs(datetime.date.today().year - original)
|
||||
dist.add_ratio('year', diff, diff_max)
|
||||
# Year.
|
||||
elif likelies['year'] and album_info.year:
|
||||
if likelies['year'] in (album_info.year, album_info.original_year):
|
||||
# No penalty for matching release or original year.
|
||||
dist.add('year', 0.0)
|
||||
elif album_info.original_year:
|
||||
# Prefer matchest closest to the release year.
|
||||
diff = abs(likelies['year'] - album_info.year)
|
||||
diff_max = abs(datetime.date.today().year -
|
||||
album_info.original_year)
|
||||
dist.add_ratio('year', diff, diff_max)
|
||||
else:
|
||||
# Full penalty when there is no original year.
|
||||
dist.add('year', 1.0)
|
||||
|
||||
# Preferred countries.
|
||||
patterns = config['match']['preferred']['countries'].as_str_seq()
|
||||
options = [re.compile(pat, re.I) for pat in patterns]
|
||||
if album_info.country and options:
|
||||
dist.add_priority('country', album_info.country, options)
|
||||
# Country.
|
||||
elif likelies['country'] and album_info.country:
|
||||
dist.add_string('country', likelies['country'], album_info.country)
|
||||
|
||||
# Label.
|
||||
if likelies['label'] and album_info.label:
|
||||
dist.add_string('label', likelies['label'], album_info.label)
|
||||
|
||||
# Catalog number.
|
||||
if likelies['catalognum'] and album_info.catalognum:
|
||||
dist.add_string('catalognum', likelies['catalognum'],
|
||||
album_info.catalognum)
|
||||
|
||||
# Disambiguation.
|
||||
if likelies['albumdisambig'] and album_info.albumdisambig:
|
||||
dist.add_string('albumdisambig', likelies['albumdisambig'],
|
||||
album_info.albumdisambig)
|
||||
|
||||
# Album ID.
|
||||
if likelies['mb_albumid']:
|
||||
dist.add_equality('album_id', likelies['mb_albumid'],
|
||||
album_info.album_id)
|
||||
|
||||
# Tracks.
|
||||
dist.tracks = {}
|
||||
for item, track in mapping.iteritems():
|
||||
dist.tracks[track] = track_distance(item, track, album_info.va)
|
||||
dist.add('tracks', dist.tracks[track].distance)
|
||||
|
||||
# Missing tracks.
|
||||
for i in range(len(album_info.tracks) - len(mapping)):
|
||||
dist.add('missing_tracks', 1.0)
|
||||
|
||||
# Unmatched tracks.
|
||||
for i in range(len(items) - len(mapping)):
|
||||
dist.add('unmatched_tracks', 1.0)
|
||||
|
||||
# Plugins.
|
||||
dist.update(plugins.album_distance(items, album_info, mapping))
|
||||
|
||||
return dist
|
||||
|
||||
|
||||
def match_by_id(items):
|
||||
"""If the items are tagged with a MusicBrainz album ID, returns an
|
||||
AlbumInfo object for the corresponding album. Otherwise, returns
|
||||
None.
|
||||
"""
|
||||
# Is there a consensus on the MB album ID?
|
||||
albumids = [item.mb_albumid for item in items if item.mb_albumid]
|
||||
if not albumids:
|
||||
log.debug(u'No album IDs found.')
|
||||
return None
|
||||
|
||||
# If all album IDs are equal, look up the album.
|
||||
if bool(reduce(lambda x, y: x if x == y else (), albumids)):
|
||||
albumid = albumids[0]
|
||||
log.debug(u'Searching for discovered album ID: {0}'.format(albumid))
|
||||
return hooks.album_for_mbid(albumid)
|
||||
else:
|
||||
log.debug(u'No album ID consensus.')
|
||||
|
||||
|
||||
def _recommendation(results):
|
||||
"""Given a sorted list of AlbumMatch or TrackMatch objects, return a
|
||||
recommendation based on the results' distances.
|
||||
|
||||
If the recommendation is higher than the configured maximum for
|
||||
an applied penalty, the recommendation will be downgraded to the
|
||||
configured maximum for that penalty.
|
||||
"""
|
||||
if not results:
|
||||
# No candidates: no recommendation.
|
||||
return Recommendation.none
|
||||
|
||||
# Basic distance thresholding.
|
||||
min_dist = results[0].distance
|
||||
if min_dist < config['match']['strong_rec_thresh'].as_number():
|
||||
# Strong recommendation level.
|
||||
rec = Recommendation.strong
|
||||
elif min_dist <= config['match']['medium_rec_thresh'].as_number():
|
||||
# Medium recommendation level.
|
||||
rec = Recommendation.medium
|
||||
elif len(results) == 1:
|
||||
# Only a single candidate.
|
||||
rec = Recommendation.low
|
||||
elif results[1].distance - min_dist >= \
|
||||
config['match']['rec_gap_thresh'].as_number():
|
||||
# Gap between first two candidates is large.
|
||||
rec = Recommendation.low
|
||||
else:
|
||||
# No conclusion. Return immediately. Can't be downgraded any further.
|
||||
return Recommendation.none
|
||||
|
||||
# Downgrade to the max rec if it is lower than the current rec for an
|
||||
# applied penalty.
|
||||
keys = set(min_dist.keys())
|
||||
if isinstance(results[0], hooks.AlbumMatch):
|
||||
for track_dist in min_dist.tracks.values():
|
||||
keys.update(track_dist.keys())
|
||||
max_rec_view = config['match']['max_rec']
|
||||
for key in keys:
|
||||
if key in max_rec_view.keys():
|
||||
max_rec = max_rec_view[key].as_choice({
|
||||
'strong': Recommendation.strong,
|
||||
'medium': Recommendation.medium,
|
||||
'low': Recommendation.low,
|
||||
'none': Recommendation.none,
|
||||
})
|
||||
rec = min(rec, max_rec)
|
||||
|
||||
return rec
|
||||
|
||||
|
||||
def _add_candidate(items, results, info):
|
||||
"""Given a candidate AlbumInfo object, attempt to add the candidate
|
||||
to the output dictionary of AlbumMatch objects. This involves
|
||||
checking the track count, ordering the items, checking for
|
||||
duplicates, and calculating the distance.
|
||||
"""
|
||||
log.debug(u'Candidate: {0} - {1}'.format(info.artist, info.album))
|
||||
|
||||
# Discard albums with zero tracks.
|
||||
if not info.tracks:
|
||||
log.debug('No tracks.')
|
||||
return
|
||||
|
||||
# Don't duplicate.
|
||||
if info.album_id in results:
|
||||
log.debug(u'Duplicate.')
|
||||
return
|
||||
|
||||
# Discard matches without required tags.
|
||||
for req_tag in config['match']['required'].as_str_seq():
|
||||
if getattr(info, req_tag) is None:
|
||||
log.debug(u'Ignored. Missing required tag: {0}'.format(req_tag))
|
||||
return
|
||||
|
||||
# Find mapping between the items and the track info.
|
||||
mapping, extra_items, extra_tracks = assign_items(items, info.tracks)
|
||||
|
||||
# Get the change distance.
|
||||
dist = distance(items, info, mapping)
|
||||
|
||||
# Skip matches with ignored penalties.
|
||||
penalties = [key for _, key in dist]
|
||||
for penalty in config['match']['ignored'].as_str_seq():
|
||||
if penalty in penalties:
|
||||
log.debug(u'Ignored. Penalty: {0}'.format(penalty))
|
||||
return
|
||||
|
||||
log.debug(u'Success. Distance: {0}'.format(dist))
|
||||
results[info.album_id] = hooks.AlbumMatch(dist, info, mapping,
|
||||
extra_items, extra_tracks)
|
||||
|
||||
|
||||
def tag_album(items, search_artist=None, search_album=None,
|
||||
search_id=None):
|
||||
"""Return a tuple of a artist name, an album name, a list of
|
||||
`AlbumMatch` candidates from the metadata backend, and a
|
||||
`Recommendation`.
|
||||
|
||||
The artist and album are the most common values of these fields
|
||||
among `items`.
|
||||
|
||||
The `AlbumMatch` objects are generated by searching the metadata
|
||||
backends. By default, the metadata of the items is used for the
|
||||
search. This can be customized by setting the parameters. The
|
||||
`mapping` field of the album has the matched `items` as keys.
|
||||
|
||||
The recommendation is calculated from the match qualitiy of the
|
||||
candidates.
|
||||
"""
|
||||
# Get current metadata.
|
||||
likelies, consensus = current_metadata(items)
|
||||
cur_artist = likelies['artist']
|
||||
cur_album = likelies['album']
|
||||
log.debug(u'Tagging {0} - {1}'.format(cur_artist, cur_album))
|
||||
|
||||
# The output result (distance, AlbumInfo) tuples (keyed by MB album
|
||||
# ID).
|
||||
candidates = {}
|
||||
|
||||
# Search by explicit ID.
|
||||
if search_id is not None:
|
||||
log.debug(u'Searching for album ID: {0}'.format(search_id))
|
||||
search_cands = hooks.albums_for_id(search_id)
|
||||
|
||||
# Use existing metadata or text search.
|
||||
else:
|
||||
# Try search based on current ID.
|
||||
id_info = match_by_id(items)
|
||||
if id_info:
|
||||
_add_candidate(items, candidates, id_info)
|
||||
rec = _recommendation(candidates.values())
|
||||
log.debug(u'Album ID match recommendation is {0}'.format(str(rec)))
|
||||
if candidates and not config['import']['timid']:
|
||||
# If we have a very good MBID match, return immediately.
|
||||
# Otherwise, this match will compete against metadata-based
|
||||
# matches.
|
||||
if rec == Recommendation.strong:
|
||||
log.debug(u'ID match.')
|
||||
return cur_artist, cur_album, candidates.values(), rec
|
||||
|
||||
# Search terms.
|
||||
if not (search_artist and search_album):
|
||||
# No explicit search terms -- use current metadata.
|
||||
search_artist, search_album = cur_artist, cur_album
|
||||
log.debug(u'Search terms: {0} - {1}'.format(search_artist,
|
||||
search_album))
|
||||
|
||||
# Is this album likely to be a "various artist" release?
|
||||
va_likely = ((not consensus['artist']) or
|
||||
(search_artist.lower() in VA_ARTISTS) or
|
||||
any(item.comp for item in items))
|
||||
log.debug(u'Album might be VA: {0}'.format(str(va_likely)))
|
||||
|
||||
# Get the results from the data sources.
|
||||
search_cands = hooks.album_candidates(items, search_artist,
|
||||
search_album, va_likely)
|
||||
|
||||
log.debug(u'Evaluating {0} candidates.'.format(len(search_cands)))
|
||||
for info in search_cands:
|
||||
_add_candidate(items, candidates, info)
|
||||
|
||||
# Sort and get the recommendation.
|
||||
candidates = sorted(candidates.itervalues())
|
||||
rec = _recommendation(candidates)
|
||||
return cur_artist, cur_album, candidates, rec
|
||||
|
||||
|
||||
def tag_item(item, search_artist=None, search_title=None,
|
||||
search_id=None):
|
||||
"""Attempts to find metadata for a single track. Returns a
|
||||
`(candidates, recommendation)` pair where `candidates` is a list of
|
||||
TrackMatch objects. `search_artist` and `search_title` may be used
|
||||
to override the current metadata for the purposes of the MusicBrainz
|
||||
title; likewise `search_id`.
|
||||
"""
|
||||
# Holds candidates found so far: keys are MBIDs; values are
|
||||
# (distance, TrackInfo) pairs.
|
||||
candidates = {}
|
||||
|
||||
# First, try matching by MusicBrainz ID.
|
||||
trackid = search_id or item.mb_trackid
|
||||
if trackid:
|
||||
log.debug(u'Searching for track ID: {0}'.format(trackid))
|
||||
for track_info in hooks.tracks_for_id(trackid):
|
||||
dist = track_distance(item, track_info, incl_artist=True)
|
||||
candidates[track_info.track_id] = \
|
||||
hooks.TrackMatch(dist, track_info)
|
||||
# If this is a good match, then don't keep searching.
|
||||
rec = _recommendation(candidates.values())
|
||||
if rec == Recommendation.strong and not config['import']['timid']:
|
||||
log.debug(u'Track ID match.')
|
||||
return candidates.values(), rec
|
||||
|
||||
# If we're searching by ID, don't proceed.
|
||||
if search_id is not None:
|
||||
if candidates:
|
||||
return candidates.values(), rec
|
||||
else:
|
||||
return [], Recommendation.none
|
||||
|
||||
# Search terms.
|
||||
if not (search_artist and search_title):
|
||||
search_artist, search_title = item.artist, item.title
|
||||
log.debug(u'Item search terms: {0} - {1}'.format(search_artist,
|
||||
search_title))
|
||||
|
||||
# Get and evaluate candidate metadata.
|
||||
for track_info in hooks.item_candidates(item, search_artist, search_title):
|
||||
dist = track_distance(item, track_info, incl_artist=True)
|
||||
candidates[track_info.track_id] = hooks.TrackMatch(dist, track_info)
|
||||
|
||||
# Sort by distance and return with recommendation.
|
||||
log.debug(u'Found {0} candidates.'.format(len(candidates)))
|
||||
candidates = sorted(candidates.itervalues())
|
||||
rec = _recommendation(candidates)
|
||||
return candidates, rec
|
||||
@@ -0,0 +1,407 @@
|
||||
# This file is part of beets.
|
||||
# Copyright 2013, Adrian Sampson.
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining
|
||||
# a copy of this software and associated documentation files (the
|
||||
# "Software"), to deal in the Software without restriction, including
|
||||
# without limitation the rights to use, copy, modify, merge, publish,
|
||||
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||
# permit persons to whom the Software is furnished to do so, subject to
|
||||
# the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
|
||||
"""Searches for albums in the MusicBrainz database.
|
||||
"""
|
||||
import logging
|
||||
import musicbrainzngs
|
||||
import re
|
||||
import traceback
|
||||
from urlparse import urljoin
|
||||
|
||||
import beets.autotag.hooks
|
||||
import beets
|
||||
from beets import util
|
||||
from beets import config
|
||||
|
||||
SEARCH_LIMIT = 5
|
||||
VARIOUS_ARTISTS_ID = '89ad4ac3-39f7-470e-963a-56509c546377'
|
||||
BASE_URL = 'http://musicbrainz.org/'
|
||||
|
||||
musicbrainzngs.set_useragent('beets', beets.__version__,
|
||||
'http://beets.radbox.org/')
|
||||
|
||||
|
||||
class MusicBrainzAPIError(util.HumanReadableException):
|
||||
"""An error while talking to MusicBrainz. The `query` field is the
|
||||
parameter to the action and may have any type.
|
||||
"""
|
||||
def __init__(self, reason, verb, query, tb=None):
|
||||
self.query = query
|
||||
super(MusicBrainzAPIError, self).__init__(reason, verb, tb)
|
||||
|
||||
def get_message(self):
|
||||
return u'{0} in {1} with query {2}'.format(
|
||||
self._reasonstr(), self.verb, repr(self.query)
|
||||
)
|
||||
|
||||
log = logging.getLogger('beets')
|
||||
|
||||
RELEASE_INCLUDES = ['artists', 'media', 'recordings', 'release-groups',
|
||||
'labels', 'artist-credits', 'aliases']
|
||||
TRACK_INCLUDES = ['artists', 'aliases']
|
||||
|
||||
|
||||
def track_url(trackid):
|
||||
return urljoin(BASE_URL, 'recording/' + trackid)
|
||||
|
||||
|
||||
def album_url(albumid):
|
||||
return urljoin(BASE_URL, 'release/' + albumid)
|
||||
|
||||
|
||||
def configure():
|
||||
"""Set up the python-musicbrainz-ngs module according to settings
|
||||
from the beets configuration. This should be called at startup.
|
||||
"""
|
||||
musicbrainzngs.set_hostname(config['musicbrainz']['host'].get(unicode))
|
||||
musicbrainzngs.set_rate_limit(
|
||||
config['musicbrainz']['ratelimit_interval'].as_number(),
|
||||
config['musicbrainz']['ratelimit'].get(int),
|
||||
)
|
||||
|
||||
|
||||
def _preferred_alias(aliases):
|
||||
"""Given an list of alias structures for an artist credit, select
|
||||
and return the user's preferred alias alias or None if no matching
|
||||
alias is found.
|
||||
"""
|
||||
if not aliases:
|
||||
return
|
||||
|
||||
# Only consider aliases that have locales set.
|
||||
aliases = [a for a in aliases if 'locale' in a]
|
||||
|
||||
# Search configured locales in order.
|
||||
for locale in config['import']['languages'].as_str_seq():
|
||||
# Find matching primary aliases for this locale.
|
||||
matches = [a for a in aliases
|
||||
if a['locale'] == locale and 'primary' in a]
|
||||
# Skip to the next locale if we have no matches
|
||||
if not matches:
|
||||
continue
|
||||
|
||||
return matches[0]
|
||||
|
||||
|
||||
def _flatten_artist_credit(credit):
|
||||
"""Given a list representing an ``artist-credit`` block, flatten the
|
||||
data into a triple of joined artist name strings: canonical, sort, and
|
||||
credit.
|
||||
"""
|
||||
artist_parts = []
|
||||
artist_sort_parts = []
|
||||
artist_credit_parts = []
|
||||
for el in credit:
|
||||
if isinstance(el, basestring):
|
||||
# Join phrase.
|
||||
artist_parts.append(el)
|
||||
artist_credit_parts.append(el)
|
||||
artist_sort_parts.append(el)
|
||||
|
||||
else:
|
||||
alias = _preferred_alias(el['artist'].get('alias-list', ()))
|
||||
|
||||
# An artist.
|
||||
if alias:
|
||||
cur_artist_name = alias['alias']
|
||||
else:
|
||||
cur_artist_name = el['artist']['name']
|
||||
artist_parts.append(cur_artist_name)
|
||||
|
||||
# Artist sort name.
|
||||
if alias:
|
||||
artist_sort_parts.append(alias['sort-name'])
|
||||
elif 'sort-name' in el['artist']:
|
||||
artist_sort_parts.append(el['artist']['sort-name'])
|
||||
else:
|
||||
artist_sort_parts.append(cur_artist_name)
|
||||
|
||||
# Artist credit.
|
||||
if 'name' in el:
|
||||
artist_credit_parts.append(el['name'])
|
||||
else:
|
||||
artist_credit_parts.append(cur_artist_name)
|
||||
|
||||
return (
|
||||
''.join(artist_parts),
|
||||
''.join(artist_sort_parts),
|
||||
''.join(artist_credit_parts),
|
||||
)
|
||||
|
||||
|
||||
def track_info(recording, index=None, medium=None, medium_index=None,
|
||||
medium_total=None):
|
||||
"""Translates a MusicBrainz recording result dictionary into a beets
|
||||
``TrackInfo`` object. Three parameters are optional and are used
|
||||
only for tracks that appear on releases (non-singletons): ``index``,
|
||||
the overall track number; ``medium``, the disc number;
|
||||
``medium_index``, the track's index on its medium; ``medium_total``,
|
||||
the number of tracks on the medium. Each number is a 1-based index.
|
||||
"""
|
||||
info = beets.autotag.hooks.TrackInfo(
|
||||
recording['title'],
|
||||
recording['id'],
|
||||
index=index,
|
||||
medium=medium,
|
||||
medium_index=medium_index,
|
||||
medium_total=medium_total,
|
||||
data_url=track_url(recording['id']),
|
||||
)
|
||||
|
||||
if recording.get('artist-credit'):
|
||||
# Get the artist names.
|
||||
info.artist, info.artist_sort, info.artist_credit = \
|
||||
_flatten_artist_credit(recording['artist-credit'])
|
||||
|
||||
# Get the ID and sort name of the first artist.
|
||||
artist = recording['artist-credit'][0]['artist']
|
||||
info.artist_id = artist['id']
|
||||
|
||||
if recording.get('length'):
|
||||
info.length = int(recording['length']) / (1000.0)
|
||||
|
||||
info.decode()
|
||||
return info
|
||||
|
||||
|
||||
def _set_date_str(info, date_str, original=False):
|
||||
"""Given a (possibly partial) YYYY-MM-DD string and an AlbumInfo
|
||||
object, set the object's release date fields appropriately. If
|
||||
`original`, then set the original_year, etc., fields.
|
||||
"""
|
||||
if date_str:
|
||||
date_parts = date_str.split('-')
|
||||
for key in ('year', 'month', 'day'):
|
||||
if date_parts:
|
||||
date_part = date_parts.pop(0)
|
||||
try:
|
||||
date_num = int(date_part)
|
||||
except ValueError:
|
||||
continue
|
||||
|
||||
if original:
|
||||
key = 'original_' + key
|
||||
setattr(info, key, date_num)
|
||||
|
||||
|
||||
def album_info(release):
|
||||
"""Takes a MusicBrainz release result dictionary and returns a beets
|
||||
AlbumInfo object containing the interesting data about that release.
|
||||
"""
|
||||
# Get artist name using join phrases.
|
||||
artist_name, artist_sort_name, artist_credit_name = \
|
||||
_flatten_artist_credit(release['artist-credit'])
|
||||
|
||||
# Basic info.
|
||||
track_infos = []
|
||||
index = 0
|
||||
for medium in release['medium-list']:
|
||||
disctitle = medium.get('title')
|
||||
format = medium.get('format')
|
||||
for track in medium['track-list']:
|
||||
# Basic information from the recording.
|
||||
index += 1
|
||||
ti = track_info(
|
||||
track['recording'],
|
||||
index,
|
||||
int(medium['position']),
|
||||
int(track['position']),
|
||||
len(medium['track-list']),
|
||||
)
|
||||
ti.disctitle = disctitle
|
||||
ti.media = format
|
||||
|
||||
# Prefer track data, where present, over recording data.
|
||||
if track.get('title'):
|
||||
ti.title = track['title']
|
||||
if track.get('artist-credit'):
|
||||
# Get the artist names.
|
||||
ti.artist, ti.artist_sort, ti.artist_credit = \
|
||||
_flatten_artist_credit(track['artist-credit'])
|
||||
ti.artist_id = track['artist-credit'][0]['artist']['id']
|
||||
if track.get('length'):
|
||||
ti.length = int(track['length']) / (1000.0)
|
||||
|
||||
track_infos.append(ti)
|
||||
|
||||
info = beets.autotag.hooks.AlbumInfo(
|
||||
release['title'],
|
||||
release['id'],
|
||||
artist_name,
|
||||
release['artist-credit'][0]['artist']['id'],
|
||||
track_infos,
|
||||
mediums=len(release['medium-list']),
|
||||
artist_sort=artist_sort_name,
|
||||
artist_credit=artist_credit_name,
|
||||
data_source='MusicBrainz',
|
||||
data_url=album_url(release['id']),
|
||||
)
|
||||
info.va = info.artist_id == VARIOUS_ARTISTS_ID
|
||||
info.asin = release.get('asin')
|
||||
info.releasegroup_id = release['release-group']['id']
|
||||
info.country = release.get('country')
|
||||
info.albumstatus = release.get('status')
|
||||
|
||||
# Build up the disambiguation string from the release group and release.
|
||||
disambig = []
|
||||
if release['release-group'].get('disambiguation'):
|
||||
disambig.append(release['release-group'].get('disambiguation'))
|
||||
if release.get('disambiguation'):
|
||||
disambig.append(release.get('disambiguation'))
|
||||
info.albumdisambig = u', '.join(disambig)
|
||||
|
||||
# Release type not always populated.
|
||||
if 'type' in release['release-group']:
|
||||
reltype = release['release-group']['type']
|
||||
if reltype:
|
||||
info.albumtype = reltype.lower()
|
||||
|
||||
# Release dates.
|
||||
release_date = release.get('date')
|
||||
release_group_date = release['release-group'].get('first-release-date')
|
||||
if not release_date:
|
||||
# Fall back if release-specific date is not available.
|
||||
release_date = release_group_date
|
||||
_set_date_str(info, release_date, False)
|
||||
_set_date_str(info, release_group_date, True)
|
||||
|
||||
# Label name.
|
||||
if release.get('label-info-list'):
|
||||
label_info = release['label-info-list'][0]
|
||||
if label_info.get('label'):
|
||||
label = label_info['label']['name']
|
||||
if label != '[no label]':
|
||||
info.label = label
|
||||
info.catalognum = label_info.get('catalog-number')
|
||||
|
||||
# Text representation data.
|
||||
if release.get('text-representation'):
|
||||
rep = release['text-representation']
|
||||
info.script = rep.get('script')
|
||||
info.language = rep.get('language')
|
||||
|
||||
# Media (format).
|
||||
if release['medium-list']:
|
||||
first_medium = release['medium-list'][0]
|
||||
info.media = first_medium.get('format')
|
||||
|
||||
info.decode()
|
||||
return info
|
||||
|
||||
|
||||
def match_album(artist, album, tracks=None, limit=SEARCH_LIMIT):
|
||||
"""Searches for a single album ("release" in MusicBrainz parlance)
|
||||
and returns an iterator over AlbumInfo objects. May raise a
|
||||
MusicBrainzAPIError.
|
||||
|
||||
The query consists of an artist name, an album name, and,
|
||||
optionally, a number of tracks on the album.
|
||||
"""
|
||||
# Build search criteria.
|
||||
criteria = {'release': album.lower().strip()}
|
||||
if artist is not None:
|
||||
criteria['artist'] = artist.lower().strip()
|
||||
else:
|
||||
# Various Artists search.
|
||||
criteria['arid'] = VARIOUS_ARTISTS_ID
|
||||
if tracks is not None:
|
||||
criteria['tracks'] = str(tracks)
|
||||
|
||||
# Abort if we have no search terms.
|
||||
if not any(criteria.itervalues()):
|
||||
return
|
||||
|
||||
try:
|
||||
res = musicbrainzngs.search_releases(limit=limit, **criteria)
|
||||
except musicbrainzngs.MusicBrainzError as exc:
|
||||
raise MusicBrainzAPIError(exc, 'release search', criteria,
|
||||
traceback.format_exc())
|
||||
for release in res['release-list']:
|
||||
# The search result is missing some data (namely, the tracks),
|
||||
# so we just use the ID and fetch the rest of the information.
|
||||
albuminfo = album_for_id(release['id'])
|
||||
if albuminfo is not None:
|
||||
yield albuminfo
|
||||
|
||||
|
||||
def match_track(artist, title, limit=SEARCH_LIMIT):
|
||||
"""Searches for a single track and returns an iterable of TrackInfo
|
||||
objects. May raise a MusicBrainzAPIError.
|
||||
"""
|
||||
criteria = {
|
||||
'artist': artist.lower().strip(),
|
||||
'recording': title.lower().strip(),
|
||||
}
|
||||
|
||||
if not any(criteria.itervalues()):
|
||||
return
|
||||
|
||||
try:
|
||||
res = musicbrainzngs.search_recordings(limit=limit, **criteria)
|
||||
except musicbrainzngs.MusicBrainzError as exc:
|
||||
raise MusicBrainzAPIError(exc, 'recording search', criteria,
|
||||
traceback.format_exc())
|
||||
for recording in res['recording-list']:
|
||||
yield track_info(recording)
|
||||
|
||||
|
||||
def _parse_id(s):
|
||||
"""Search for a MusicBrainz ID in the given string and return it. If
|
||||
no ID can be found, return None.
|
||||
"""
|
||||
# Find the first thing that looks like a UUID/MBID.
|
||||
match = re.search('[a-f0-9]{8}(-[a-f0-9]{4}){3}-[a-f0-9]{12}', s)
|
||||
if match:
|
||||
return match.group()
|
||||
|
||||
|
||||
def album_for_id(releaseid):
|
||||
"""Fetches an album by its MusicBrainz ID and returns an AlbumInfo
|
||||
object or None if the album is not found. May raise a
|
||||
MusicBrainzAPIError.
|
||||
"""
|
||||
albumid = _parse_id(releaseid)
|
||||
if not albumid:
|
||||
log.debug(u'Invalid MBID ({0}).'.format(releaseid))
|
||||
return
|
||||
try:
|
||||
res = musicbrainzngs.get_release_by_id(albumid,
|
||||
RELEASE_INCLUDES)
|
||||
except musicbrainzngs.ResponseError:
|
||||
log.debug(u'Album ID match failed.')
|
||||
return None
|
||||
except musicbrainzngs.MusicBrainzError as exc:
|
||||
raise MusicBrainzAPIError(exc, 'get release by ID', albumid,
|
||||
traceback.format_exc())
|
||||
return album_info(res['release'])
|
||||
|
||||
|
||||
def track_for_id(releaseid):
|
||||
"""Fetches a track by its MusicBrainz ID. Returns a TrackInfo object
|
||||
or None if no track is found. May raise a MusicBrainzAPIError.
|
||||
"""
|
||||
trackid = _parse_id(releaseid)
|
||||
if not trackid:
|
||||
log.debug(u'Invalid MBID ({0}).'.format(releaseid))
|
||||
return
|
||||
try:
|
||||
res = musicbrainzngs.get_recording_by_id(trackid, TRACK_INCLUDES)
|
||||
except musicbrainzngs.ResponseError:
|
||||
log.debug(u'Track ID match failed.')
|
||||
return None
|
||||
except musicbrainzngs.MusicBrainzError as exc:
|
||||
raise MusicBrainzAPIError(exc, 'get recording by ID', trackid,
|
||||
traceback.format_exc())
|
||||
return track_info(res['recording'])
|
||||
@@ -0,0 +1,109 @@
|
||||
library: library.db
|
||||
directory: ~/Music
|
||||
|
||||
import:
|
||||
write: yes
|
||||
copy: yes
|
||||
move: no
|
||||
link: no
|
||||
delete: no
|
||||
resume: ask
|
||||
incremental: no
|
||||
quiet_fallback: skip
|
||||
none_rec_action: ask
|
||||
timid: no
|
||||
log:
|
||||
autotag: yes
|
||||
quiet: no
|
||||
singletons: no
|
||||
default_action: apply
|
||||
languages: []
|
||||
detail: no
|
||||
flat: no
|
||||
group_albums: no
|
||||
pretend: false
|
||||
|
||||
clutter: ["Thumbs.DB", ".DS_Store"]
|
||||
ignore: [".*", "*~", "System Volume Information"]
|
||||
replace:
|
||||
'[\\/]': _
|
||||
'^\.': _
|
||||
'[\x00-\x1f]': _
|
||||
'[<>:"\?\*\|]': _
|
||||
'\.$': _
|
||||
'\s+$': ''
|
||||
'^\s+': ''
|
||||
path_sep_replace: _
|
||||
asciify_paths: false
|
||||
art_filename: cover
|
||||
max_filename_length: 0
|
||||
|
||||
plugins: []
|
||||
pluginpath: []
|
||||
threaded: yes
|
||||
color: yes
|
||||
timeout: 5.0
|
||||
per_disc_numbering: no
|
||||
verbose: no
|
||||
terminal_encoding: utf8
|
||||
original_date: no
|
||||
id3v23: no
|
||||
|
||||
ui:
|
||||
terminal_width: 80
|
||||
length_diff_thresh: 10.0
|
||||
|
||||
list_format_item: $artist - $album - $title
|
||||
list_format_album: $albumartist - $album
|
||||
time_format: '%Y-%m-%d %H:%M:%S'
|
||||
|
||||
sort_album: albumartist+ album+
|
||||
sort_item: artist+ album+ disc+ track+
|
||||
|
||||
paths:
|
||||
default: $albumartist/$album%aunique{}/$track $title
|
||||
singleton: Non-Album/$artist/$title
|
||||
comp: Compilations/$album%aunique{}/$track $title
|
||||
|
||||
statefile: state.pickle
|
||||
|
||||
musicbrainz:
|
||||
host: musicbrainz.org
|
||||
ratelimit: 1
|
||||
ratelimit_interval: 1.0
|
||||
|
||||
match:
|
||||
strong_rec_thresh: 0.04
|
||||
medium_rec_thresh: 0.25
|
||||
rec_gap_thresh: 0.25
|
||||
max_rec:
|
||||
missing_tracks: medium
|
||||
unmatched_tracks: medium
|
||||
distance_weights:
|
||||
source: 2.0
|
||||
artist: 3.0
|
||||
album: 3.0
|
||||
media: 1.0
|
||||
mediums: 1.0
|
||||
year: 1.0
|
||||
country: 0.5
|
||||
label: 0.5
|
||||
catalognum: 0.5
|
||||
albumdisambig: 0.5
|
||||
album_id: 5.0
|
||||
tracks: 2.0
|
||||
missing_tracks: 0.9
|
||||
unmatched_tracks: 0.6
|
||||
track_title: 3.0
|
||||
track_artist: 2.0
|
||||
track_index: 1.0
|
||||
track_length: 2.0
|
||||
track_id: 5.0
|
||||
preferred:
|
||||
countries: []
|
||||
media: []
|
||||
original_year: no
|
||||
ignored: []
|
||||
required: []
|
||||
track_length_grace: 10
|
||||
track_length_max: 30
|
||||
@@ -0,0 +1,25 @@
|
||||
# This file is part of beets.
|
||||
# Copyright 2014, Adrian Sampson.
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining
|
||||
# a copy of this software and associated documentation files (the
|
||||
# "Software"), to deal in the Software without restriction, including
|
||||
# without limitation the rights to use, copy, modify, merge, publish,
|
||||
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||
# permit persons to whom the Software is furnished to do so, subject to
|
||||
# the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
|
||||
"""DBCore is an abstract database package that forms the basis for beets'
|
||||
Library.
|
||||
"""
|
||||
from .db import Model, Database
|
||||
from .query import Query, FieldQuery, MatchQuery, AndQuery, OrQuery
|
||||
from .types import Type
|
||||
from .queryparse import query_from_strings
|
||||
from .queryparse import sort_from_strings
|
||||
from .queryparse import parse_sorted_query
|
||||
|
||||
# flake8: noqa
|
||||
@@ -0,0 +1,820 @@
|
||||
# This file is part of beets.
|
||||
# Copyright 2014, Adrian Sampson.
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining
|
||||
# a copy of this software and associated documentation files (the
|
||||
# "Software"), to deal in the Software without restriction, including
|
||||
# without limitation the rights to use, copy, modify, merge, publish,
|
||||
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||
# permit persons to whom the Software is furnished to do so, subject to
|
||||
# the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
|
||||
"""The central Model and Database constructs for DBCore.
|
||||
"""
|
||||
import time
|
||||
import os
|
||||
from collections import defaultdict
|
||||
import threading
|
||||
import sqlite3
|
||||
import contextlib
|
||||
import collections
|
||||
|
||||
import beets
|
||||
from beets.util.functemplate import Template
|
||||
from beets.dbcore import types
|
||||
from .query import MatchQuery, NullSort, TrueQuery
|
||||
|
||||
|
||||
class FormattedMapping(collections.Mapping):
|
||||
"""A `dict`-like formatted view of a model.
|
||||
|
||||
The accessor `mapping[key]` returns the formated version of
|
||||
`model[key]` as a unicode string.
|
||||
|
||||
If `for_path` is true, all path separators in the formatted values
|
||||
are replaced.
|
||||
"""
|
||||
|
||||
def __init__(self, model, for_path=False):
|
||||
self.for_path = for_path
|
||||
self.model = model
|
||||
self.model_keys = model.keys(True)
|
||||
|
||||
def __getitem__(self, key):
|
||||
if key in self.model_keys:
|
||||
return self._get_formatted(self.model, key)
|
||||
else:
|
||||
raise KeyError(key)
|
||||
|
||||
def __iter__(self):
|
||||
return iter(self.model_keys)
|
||||
|
||||
def __len__(self):
|
||||
return len(self.model_keys)
|
||||
|
||||
def get(self, key, default=None):
|
||||
if default is None:
|
||||
default = self.model._type(key).format(None)
|
||||
return super(FormattedMapping, self).get(key, default)
|
||||
|
||||
def _get_formatted(self, model, key):
|
||||
value = model._type(key).format(model.get(key))
|
||||
if isinstance(value, bytes):
|
||||
value = value.decode('utf8', 'ignore')
|
||||
|
||||
if self.for_path:
|
||||
sep_repl = beets.config['path_sep_replace'].get(unicode)
|
||||
for sep in (os.path.sep, os.path.altsep):
|
||||
if sep:
|
||||
value = value.replace(sep, sep_repl)
|
||||
|
||||
return value
|
||||
|
||||
|
||||
# Abstract base for model classes.
|
||||
|
||||
class Model(object):
|
||||
"""An abstract object representing an object in the database. Model
|
||||
objects act like dictionaries (i.e., the allow subscript access like
|
||||
``obj['field']``). The same field set is available via attribute
|
||||
access as a shortcut (i.e., ``obj.field``). Three kinds of attributes are
|
||||
available:
|
||||
|
||||
* **Fixed attributes** come from a predetermined list of field
|
||||
names. These fields correspond to SQLite table columns and are
|
||||
thus fast to read, write, and query.
|
||||
* **Flexible attributes** are free-form and do not need to be listed
|
||||
ahead of time.
|
||||
* **Computed attributes** are read-only fields computed by a getter
|
||||
function provided by a plugin.
|
||||
|
||||
Access to all three field types is uniform: ``obj.field`` works the
|
||||
same regardless of whether ``field`` is fixed, flexible, or
|
||||
computed.
|
||||
|
||||
Model objects can optionally be associated with a `Library` object,
|
||||
in which case they can be loaded and stored from the database. Dirty
|
||||
flags are used to track which fields need to be stored.
|
||||
"""
|
||||
|
||||
# Abstract components (to be provided by subclasses).
|
||||
|
||||
_table = None
|
||||
"""The main SQLite table name.
|
||||
"""
|
||||
|
||||
_flex_table = None
|
||||
"""The flex field SQLite table name.
|
||||
"""
|
||||
|
||||
_fields = {}
|
||||
"""A mapping indicating available "fixed" fields on this type. The
|
||||
keys are field names and the values are `Type` objects.
|
||||
"""
|
||||
|
||||
_search_fields = ()
|
||||
"""The fields that should be queried by default by unqualified query
|
||||
terms.
|
||||
"""
|
||||
|
||||
_types = {}
|
||||
"""Optional Types for non-fixed (i.e., flexible and computed) fields.
|
||||
"""
|
||||
|
||||
_sorts = {}
|
||||
"""Optional named sort criteria. The keys are strings and the values
|
||||
are subclasses of `Sort`.
|
||||
"""
|
||||
|
||||
_always_dirty = False
|
||||
"""By default, fields only become "dirty" when their value actually
|
||||
changes. Enabling this flag marks fields as dirty even when the new
|
||||
value is the same as the old value (e.g., `o.f = o.f`).
|
||||
"""
|
||||
|
||||
@classmethod
|
||||
def _getters(cls):
|
||||
"""Return a mapping from field names to getter functions.
|
||||
"""
|
||||
# We could cache this if it becomes a performance problem to
|
||||
# gather the getter mapping every time.
|
||||
raise NotImplementedError()
|
||||
|
||||
def _template_funcs(self):
|
||||
"""Return a mapping from function names to text-transformer
|
||||
functions.
|
||||
"""
|
||||
# As above: we could consider caching this result.
|
||||
raise NotImplementedError()
|
||||
|
||||
# Basic operation.
|
||||
|
||||
def __init__(self, db=None, **values):
|
||||
"""Create a new object with an optional Database association and
|
||||
initial field values.
|
||||
"""
|
||||
self._db = db
|
||||
self._dirty = set()
|
||||
self._values_fixed = {}
|
||||
self._values_flex = {}
|
||||
|
||||
# Initial contents.
|
||||
self.update(values)
|
||||
self.clear_dirty()
|
||||
|
||||
@classmethod
|
||||
def _awaken(cls, db=None, fixed_values={}, flex_values={}):
|
||||
"""Create an object with values drawn from the database.
|
||||
|
||||
This is a performance optimization: the checks involved with
|
||||
ordinary construction are bypassed.
|
||||
"""
|
||||
obj = cls(db)
|
||||
for key, value in fixed_values.iteritems():
|
||||
obj._values_fixed[key] = cls._type(key).from_sql(value)
|
||||
for key, value in flex_values.iteritems():
|
||||
obj._values_flex[key] = cls._type(key).from_sql(value)
|
||||
return obj
|
||||
|
||||
def __repr__(self):
|
||||
return '{0}({1})'.format(
|
||||
type(self).__name__,
|
||||
', '.join('{0}={1!r}'.format(k, v) for k, v in dict(self).items()),
|
||||
)
|
||||
|
||||
def clear_dirty(self):
|
||||
"""Mark all fields as *clean* (i.e., not needing to be stored to
|
||||
the database).
|
||||
"""
|
||||
self._dirty = set()
|
||||
|
||||
def _check_db(self, need_id=True):
|
||||
"""Ensure that this object is associated with a database row: it
|
||||
has a reference to a database (`_db`) and an id. A ValueError
|
||||
exception is raised otherwise.
|
||||
"""
|
||||
if not self._db:
|
||||
raise ValueError('{0} has no database'.format(type(self).__name__))
|
||||
if need_id and not self.id:
|
||||
raise ValueError('{0} has no id'.format(type(self).__name__))
|
||||
|
||||
# Essential field accessors.
|
||||
|
||||
@classmethod
|
||||
def _type(self, key):
|
||||
"""Get the type of a field, a `Type` instance.
|
||||
|
||||
If the field has no explicit type, it is given the base `Type`,
|
||||
which does no conversion.
|
||||
"""
|
||||
return self._fields.get(key) or self._types.get(key) or types.DEFAULT
|
||||
|
||||
def __getitem__(self, key):
|
||||
"""Get the value for a field. Raise a KeyError if the field is
|
||||
not available.
|
||||
"""
|
||||
getters = self._getters()
|
||||
if key in getters: # Computed.
|
||||
return getters[key](self)
|
||||
elif key in self._fields: # Fixed.
|
||||
return self._values_fixed.get(key)
|
||||
elif key in self._values_flex: # Flexible.
|
||||
return self._values_flex[key]
|
||||
else:
|
||||
raise KeyError(key)
|
||||
|
||||
def __setitem__(self, key, value):
|
||||
"""Assign the value for a field.
|
||||
"""
|
||||
# Choose where to place the value.
|
||||
if key in self._fields:
|
||||
source = self._values_fixed
|
||||
else:
|
||||
source = self._values_flex
|
||||
|
||||
# If the field has a type, filter the value.
|
||||
value = self._type(key).normalize(value)
|
||||
|
||||
# Assign value and possibly mark as dirty.
|
||||
old_value = source.get(key)
|
||||
source[key] = value
|
||||
if self._always_dirty or old_value != value:
|
||||
self._dirty.add(key)
|
||||
|
||||
def __delitem__(self, key):
|
||||
"""Remove a flexible attribute from the model.
|
||||
"""
|
||||
if key in self._values_flex: # Flexible.
|
||||
del self._values_flex[key]
|
||||
self._dirty.add(key) # Mark for dropping on store.
|
||||
elif key in self._getters(): # Computed.
|
||||
raise KeyError('computed field {0} cannot be deleted'.format(key))
|
||||
elif key in self._fields: # Fixed.
|
||||
raise KeyError('fixed field {0} cannot be deleted'.format(key))
|
||||
else:
|
||||
raise KeyError('no such field {0}'.format(key))
|
||||
|
||||
def keys(self, computed=False):
|
||||
"""Get a list of available field names for this object. The
|
||||
`computed` parameter controls whether computed (plugin-provided)
|
||||
fields are included in the key list.
|
||||
"""
|
||||
base_keys = list(self._fields) + self._values_flex.keys()
|
||||
if computed:
|
||||
return base_keys + self._getters().keys()
|
||||
else:
|
||||
return base_keys
|
||||
|
||||
# Act like a dictionary.
|
||||
|
||||
def update(self, values):
|
||||
"""Assign all values in the given dict.
|
||||
"""
|
||||
for key, value in values.items():
|
||||
self[key] = value
|
||||
|
||||
def items(self):
|
||||
"""Iterate over (key, value) pairs that this object contains.
|
||||
Computed fields are not included.
|
||||
"""
|
||||
for key in self:
|
||||
yield key, self[key]
|
||||
|
||||
def get(self, key, default=None):
|
||||
"""Get the value for a given key or `default` if it does not
|
||||
exist.
|
||||
"""
|
||||
if key in self:
|
||||
return self[key]
|
||||
else:
|
||||
return default
|
||||
|
||||
def __contains__(self, key):
|
||||
"""Determine whether `key` is an attribute on this object.
|
||||
"""
|
||||
return key in self.keys(True)
|
||||
|
||||
def __iter__(self):
|
||||
"""Iterate over the available field names (excluding computed
|
||||
fields).
|
||||
"""
|
||||
return iter(self.keys())
|
||||
|
||||
# Convenient attribute access.
|
||||
|
||||
def __getattr__(self, key):
|
||||
if key.startswith('_'):
|
||||
raise AttributeError('model has no attribute {0!r}'.format(key))
|
||||
else:
|
||||
try:
|
||||
return self[key]
|
||||
except KeyError:
|
||||
raise AttributeError('no such field {0!r}'.format(key))
|
||||
|
||||
def __setattr__(self, key, value):
|
||||
if key.startswith('_'):
|
||||
super(Model, self).__setattr__(key, value)
|
||||
else:
|
||||
self[key] = value
|
||||
|
||||
def __delattr__(self, key):
|
||||
if key.startswith('_'):
|
||||
super(Model, self).__delattr__(key)
|
||||
else:
|
||||
del self[key]
|
||||
|
||||
# Database interaction (CRUD methods).
|
||||
|
||||
def store(self):
|
||||
"""Save the object's metadata into the library database.
|
||||
"""
|
||||
self._check_db()
|
||||
|
||||
# Build assignments for query.
|
||||
assignments = []
|
||||
subvars = []
|
||||
for key in self._fields:
|
||||
if key != 'id' and key in self._dirty:
|
||||
self._dirty.remove(key)
|
||||
assignments.append(key + '=?')
|
||||
value = self._type(key).to_sql(self[key])
|
||||
subvars.append(value)
|
||||
assignments = ','.join(assignments)
|
||||
|
||||
with self._db.transaction() as tx:
|
||||
# Main table update.
|
||||
if assignments:
|
||||
query = 'UPDATE {0} SET {1} WHERE id=?'.format(
|
||||
self._table, assignments
|
||||
)
|
||||
subvars.append(self.id)
|
||||
tx.mutate(query, subvars)
|
||||
|
||||
# Modified/added flexible attributes.
|
||||
for key, value in self._values_flex.items():
|
||||
if key in self._dirty:
|
||||
self._dirty.remove(key)
|
||||
tx.mutate(
|
||||
'INSERT INTO {0} '
|
||||
'(entity_id, key, value) '
|
||||
'VALUES (?, ?, ?);'.format(self._flex_table),
|
||||
(self.id, key, value),
|
||||
)
|
||||
|
||||
# Deleted flexible attributes.
|
||||
for key in self._dirty:
|
||||
tx.mutate(
|
||||
'DELETE FROM {0} '
|
||||
'WHERE entity_id=? AND key=?'.format(self._flex_table),
|
||||
(self.id, key)
|
||||
)
|
||||
|
||||
self.clear_dirty()
|
||||
|
||||
def load(self):
|
||||
"""Refresh the object's metadata from the library database.
|
||||
"""
|
||||
self._check_db()
|
||||
stored_obj = self._db._get(type(self), self.id)
|
||||
assert stored_obj is not None, "object {0} not in DB".format(self.id)
|
||||
self._values_fixed = {}
|
||||
self._values_flex = {}
|
||||
self.update(dict(stored_obj))
|
||||
self.clear_dirty()
|
||||
|
||||
def remove(self):
|
||||
"""Remove the object's associated rows from the database.
|
||||
"""
|
||||
self._check_db()
|
||||
with self._db.transaction() as tx:
|
||||
tx.mutate(
|
||||
'DELETE FROM {0} WHERE id=?'.format(self._table),
|
||||
(self.id,)
|
||||
)
|
||||
tx.mutate(
|
||||
'DELETE FROM {0} WHERE entity_id=?'.format(self._flex_table),
|
||||
(self.id,)
|
||||
)
|
||||
|
||||
def add(self, db=None):
|
||||
"""Add the object to the library database. This object must be
|
||||
associated with a database; you can provide one via the `db`
|
||||
parameter or use the currently associated database.
|
||||
|
||||
The object's `id` and `added` fields are set along with any
|
||||
current field values.
|
||||
"""
|
||||
if db:
|
||||
self._db = db
|
||||
self._check_db(False)
|
||||
|
||||
with self._db.transaction() as tx:
|
||||
new_id = tx.mutate(
|
||||
'INSERT INTO {0} DEFAULT VALUES'.format(self._table)
|
||||
)
|
||||
self.id = new_id
|
||||
self.added = time.time()
|
||||
|
||||
# Mark every non-null field as dirty and store.
|
||||
for key in self:
|
||||
if self[key] is not None:
|
||||
self._dirty.add(key)
|
||||
self.store()
|
||||
|
||||
# Formatting and templating.
|
||||
|
||||
_formatter = FormattedMapping
|
||||
|
||||
def formatted(self, for_path=False):
|
||||
"""Get a mapping containing all values on this object formatted
|
||||
as human-readable unicode strings.
|
||||
"""
|
||||
return self._formatter(self, for_path)
|
||||
|
||||
def evaluate_template(self, template, for_path=False):
|
||||
"""Evaluate a template (a string or a `Template` object) using
|
||||
the object's fields. If `for_path` is true, then no new path
|
||||
separators will be added to the template.
|
||||
"""
|
||||
# Perform substitution.
|
||||
if isinstance(template, basestring):
|
||||
template = Template(template)
|
||||
return template.substitute(self.formatted(for_path),
|
||||
self._template_funcs())
|
||||
|
||||
# Parsing.
|
||||
|
||||
@classmethod
|
||||
def _parse(cls, key, string):
|
||||
"""Parse a string as a value for the given key.
|
||||
"""
|
||||
if not isinstance(string, basestring):
|
||||
raise TypeError("_parse() argument must be a string")
|
||||
|
||||
return cls._type(key).parse(string)
|
||||
|
||||
|
||||
# Database controller and supporting interfaces.
|
||||
|
||||
class Results(object):
|
||||
"""An item query result set. Iterating over the collection lazily
|
||||
constructs LibModel objects that reflect database rows.
|
||||
"""
|
||||
def __init__(self, model_class, rows, db, query=None, sort=None):
|
||||
"""Create a result set that will construct objects of type
|
||||
`model_class`.
|
||||
|
||||
`model_class` is a subclass of `LibModel` that will be
|
||||
constructed. `rows` is a query result: a list of mappings. The
|
||||
new objects will be associated with the database `db`.
|
||||
|
||||
If `query` is provided, it is used as a predicate to filter the
|
||||
results for a "slow query" that cannot be evaluated by the
|
||||
database directly. If `sort` is provided, it is used to sort the
|
||||
full list of results before returning. This means it is a "slow
|
||||
sort" and all objects must be built before returning the first
|
||||
one.
|
||||
"""
|
||||
self.model_class = model_class
|
||||
self.rows = rows
|
||||
self.db = db
|
||||
self.query = query
|
||||
self.sort = sort
|
||||
|
||||
# We keep a queue of rows we haven't yet consumed for
|
||||
# materialization. We preserve the original total number of
|
||||
# rows.
|
||||
self._rows = rows
|
||||
self._row_count = len(rows)
|
||||
|
||||
# The materialized objects corresponding to rows that have been
|
||||
# consumed.
|
||||
self._objects = []
|
||||
|
||||
def _get_objects(self):
|
||||
"""Construct and generate Model objects for they query. The
|
||||
objects are returned in the order emitted from the database; no
|
||||
slow sort is applied.
|
||||
|
||||
For performance, this generator caches materialized objects to
|
||||
avoid constructing them more than once. This way, iterating over
|
||||
a `Results` object a second time should be much faster than the
|
||||
first.
|
||||
"""
|
||||
index = 0 # Position in the materialized objects.
|
||||
while index < len(self._objects) or self._rows:
|
||||
# Are there previously-materialized objects to produce?
|
||||
if index < len(self._objects):
|
||||
yield self._objects[index]
|
||||
index += 1
|
||||
|
||||
# Otherwise, we consume another row, materialize its object
|
||||
# and produce it.
|
||||
else:
|
||||
while self._rows:
|
||||
row = self._rows.pop(0)
|
||||
obj = self._make_model(row)
|
||||
# If there is a slow-query predicate, ensurer that the
|
||||
# object passes it.
|
||||
if not self.query or self.query.match(obj):
|
||||
self._objects.append(obj)
|
||||
index += 1
|
||||
yield obj
|
||||
break
|
||||
|
||||
def __iter__(self):
|
||||
"""Construct and generate Model objects for all matching
|
||||
objects, in sorted order.
|
||||
"""
|
||||
if self.sort:
|
||||
# Slow sort. Must build the full list first.
|
||||
objects = self.sort.sort(list(self._get_objects()))
|
||||
return iter(objects)
|
||||
|
||||
else:
|
||||
# Objects are pre-sorted (i.e., by the database).
|
||||
return self._get_objects()
|
||||
|
||||
def _make_model(self, row):
|
||||
# Get the flexible attributes for the object.
|
||||
with self.db.transaction() as tx:
|
||||
flex_rows = tx.query(
|
||||
'SELECT * FROM {0} WHERE entity_id=?'.format(
|
||||
self.model_class._flex_table
|
||||
),
|
||||
(row['id'],)
|
||||
)
|
||||
|
||||
cols = dict(row)
|
||||
values = dict((k, v) for (k, v) in cols.items()
|
||||
if not k[:4] == 'flex')
|
||||
flex_values = dict((row['key'], row['value']) for row in flex_rows)
|
||||
|
||||
# Construct the Python object
|
||||
obj = self.model_class._awaken(self.db, values, flex_values)
|
||||
return obj
|
||||
|
||||
def __len__(self):
|
||||
"""Get the number of matching objects.
|
||||
"""
|
||||
if not self._rows:
|
||||
# Fully materialized. Just count the objects.
|
||||
return len(self._objects)
|
||||
|
||||
elif self.query:
|
||||
# A slow query. Fall back to testing every object.
|
||||
count = 0
|
||||
for obj in self:
|
||||
count += 1
|
||||
return count
|
||||
|
||||
else:
|
||||
# A fast query. Just count the rows.
|
||||
return self._row_count
|
||||
|
||||
def __nonzero__(self):
|
||||
"""Does this result contain any objects?
|
||||
"""
|
||||
return bool(len(self))
|
||||
|
||||
def __getitem__(self, n):
|
||||
"""Get the nth item in this result set. This is inefficient: all
|
||||
items up to n are materialized and thrown away.
|
||||
"""
|
||||
if not self._rows and not self.sort:
|
||||
# Fully materialized and already in order. Just look up the
|
||||
# object.
|
||||
return self._objects[n]
|
||||
|
||||
it = iter(self)
|
||||
try:
|
||||
for i in range(n):
|
||||
it.next()
|
||||
return it.next()
|
||||
except StopIteration:
|
||||
raise IndexError('result index {0} out of range'.format(n))
|
||||
|
||||
def get(self):
|
||||
"""Return the first matching object, or None if no objects
|
||||
match.
|
||||
"""
|
||||
it = iter(self)
|
||||
try:
|
||||
return it.next()
|
||||
except StopIteration:
|
||||
return None
|
||||
|
||||
|
||||
class Transaction(object):
|
||||
"""A context manager for safe, concurrent access to the database.
|
||||
All SQL commands should be executed through a transaction.
|
||||
"""
|
||||
def __init__(self, db):
|
||||
self.db = db
|
||||
|
||||
def __enter__(self):
|
||||
"""Begin a transaction. This transaction may be created while
|
||||
another is active in a different thread.
|
||||
"""
|
||||
with self.db._tx_stack() as stack:
|
||||
first = not stack
|
||||
stack.append(self)
|
||||
if first:
|
||||
# Beginning a "root" transaction, which corresponds to an
|
||||
# SQLite transaction.
|
||||
self.db._db_lock.acquire()
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc_value, traceback):
|
||||
"""Complete a transaction. This must be the most recently
|
||||
entered but not yet exited transaction. If it is the last active
|
||||
transaction, the database updates are committed.
|
||||
"""
|
||||
with self.db._tx_stack() as stack:
|
||||
assert stack.pop() is self
|
||||
empty = not stack
|
||||
if empty:
|
||||
# Ending a "root" transaction. End the SQLite transaction.
|
||||
self.db._connection().commit()
|
||||
self.db._db_lock.release()
|
||||
|
||||
def query(self, statement, subvals=()):
|
||||
"""Execute an SQL statement with substitution values and return
|
||||
a list of rows from the database.
|
||||
"""
|
||||
cursor = self.db._connection().execute(statement, subvals)
|
||||
return cursor.fetchall()
|
||||
|
||||
def mutate(self, statement, subvals=()):
|
||||
"""Execute an SQL statement with substitution values and return
|
||||
the row ID of the last affected row.
|
||||
"""
|
||||
cursor = self.db._connection().execute(statement, subvals)
|
||||
return cursor.lastrowid
|
||||
|
||||
def script(self, statements):
|
||||
"""Execute a string containing multiple SQL statements."""
|
||||
self.db._connection().executescript(statements)
|
||||
|
||||
|
||||
class Database(object):
|
||||
"""A container for Model objects that wraps an SQLite database as
|
||||
the backend.
|
||||
"""
|
||||
_models = ()
|
||||
"""The Model subclasses representing tables in this database.
|
||||
"""
|
||||
|
||||
def __init__(self, path):
|
||||
self.path = path
|
||||
|
||||
self._connections = {}
|
||||
self._tx_stacks = defaultdict(list)
|
||||
|
||||
# A lock to protect the _connections and _tx_stacks maps, which
|
||||
# both map thread IDs to private resources.
|
||||
self._shared_map_lock = threading.Lock()
|
||||
|
||||
# A lock to protect access to the database itself. SQLite does
|
||||
# allow multiple threads to access the database at the same
|
||||
# time, but many users were experiencing crashes related to this
|
||||
# capability: where SQLite was compiled without HAVE_USLEEP, its
|
||||
# backoff algorithm in the case of contention was causing
|
||||
# whole-second sleeps (!) that would trigger its internal
|
||||
# timeout. Using this lock ensures only one SQLite transaction
|
||||
# is active at a time.
|
||||
self._db_lock = threading.Lock()
|
||||
|
||||
# Set up database schema.
|
||||
for model_cls in self._models:
|
||||
self._make_table(model_cls._table, model_cls._fields)
|
||||
self._make_attribute_table(model_cls._flex_table)
|
||||
|
||||
# Primitive access control: connections and transactions.
|
||||
|
||||
def _connection(self):
|
||||
"""Get a SQLite connection object to the underlying database.
|
||||
One connection object is created per thread.
|
||||
"""
|
||||
thread_id = threading.current_thread().ident
|
||||
with self._shared_map_lock:
|
||||
if thread_id in self._connections:
|
||||
return self._connections[thread_id]
|
||||
else:
|
||||
# Make a new connection.
|
||||
conn = sqlite3.connect(
|
||||
self.path,
|
||||
timeout=beets.config['timeout'].as_number(),
|
||||
)
|
||||
|
||||
# Access SELECT results like dictionaries.
|
||||
conn.row_factory = sqlite3.Row
|
||||
|
||||
self._connections[thread_id] = conn
|
||||
return conn
|
||||
|
||||
@contextlib.contextmanager
|
||||
def _tx_stack(self):
|
||||
"""A context manager providing access to the current thread's
|
||||
transaction stack. The context manager synchronizes access to
|
||||
the stack map. Transactions should never migrate across threads.
|
||||
"""
|
||||
thread_id = threading.current_thread().ident
|
||||
with self._shared_map_lock:
|
||||
yield self._tx_stacks[thread_id]
|
||||
|
||||
def transaction(self):
|
||||
"""Get a :class:`Transaction` object for interacting directly
|
||||
with the underlying SQLite database.
|
||||
"""
|
||||
return Transaction(self)
|
||||
|
||||
# Schema setup and migration.
|
||||
|
||||
def _make_table(self, table, fields):
|
||||
"""Set up the schema of the database. `fields` is a mapping
|
||||
from field names to `Type`s. Columns are added if necessary.
|
||||
"""
|
||||
# Get current schema.
|
||||
with self.transaction() as tx:
|
||||
rows = tx.query('PRAGMA table_info(%s)' % table)
|
||||
current_fields = set([row[1] for row in rows])
|
||||
|
||||
field_names = set(fields.keys())
|
||||
if current_fields.issuperset(field_names):
|
||||
# Table exists and has all the required columns.
|
||||
return
|
||||
|
||||
if not current_fields:
|
||||
# No table exists.
|
||||
columns = []
|
||||
for name, typ in fields.items():
|
||||
columns.append('{0} {1}'.format(name, typ.sql))
|
||||
setup_sql = 'CREATE TABLE {0} ({1});\n'.format(table,
|
||||
', '.join(columns))
|
||||
|
||||
else:
|
||||
# Table exists does not match the field set.
|
||||
setup_sql = ''
|
||||
for name, typ in fields.items():
|
||||
if name in current_fields:
|
||||
continue
|
||||
setup_sql += 'ALTER TABLE {0} ADD COLUMN {1} {2};\n'.format(
|
||||
table, name, typ.sql
|
||||
)
|
||||
|
||||
with self.transaction() as tx:
|
||||
tx.script(setup_sql)
|
||||
|
||||
def _make_attribute_table(self, flex_table):
|
||||
"""Create a table and associated index for flexible attributes
|
||||
for the given entity (if they don't exist).
|
||||
"""
|
||||
with self.transaction() as tx:
|
||||
tx.script("""
|
||||
CREATE TABLE IF NOT EXISTS {0} (
|
||||
id INTEGER PRIMARY KEY,
|
||||
entity_id INTEGER,
|
||||
key TEXT,
|
||||
value TEXT,
|
||||
UNIQUE(entity_id, key) ON CONFLICT REPLACE);
|
||||
CREATE INDEX IF NOT EXISTS {0}_by_entity
|
||||
ON {0} (entity_id);
|
||||
""".format(flex_table))
|
||||
|
||||
# Querying.
|
||||
|
||||
def _fetch(self, model_cls, query=None, sort=None):
|
||||
"""Fetch the objects of type `model_cls` matching the given
|
||||
query. The query may be given as a string, string sequence, a
|
||||
Query object, or None (to fetch everything). `sort` is an
|
||||
`Sort` object.
|
||||
"""
|
||||
query = query or TrueQuery() # A null query.
|
||||
sort = sort or NullSort() # Unsorted.
|
||||
where, subvals = query.clause()
|
||||
order_by = sort.order_clause()
|
||||
|
||||
sql = ("SELECT * FROM {0} WHERE {1} {2}").format(
|
||||
model_cls._table,
|
||||
where or '1',
|
||||
"ORDER BY {0}".format(order_by) if order_by else '',
|
||||
)
|
||||
|
||||
with self.transaction() as tx:
|
||||
rows = tx.query(sql, subvals)
|
||||
|
||||
return Results(
|
||||
model_cls, rows, self,
|
||||
None if where else query, # Slow query component.
|
||||
sort if sort.is_slow() else None, # Slow sort component.
|
||||
)
|
||||
|
||||
def _get(self, model_cls, id):
|
||||
"""Get a Model object by its id or None if the id does not
|
||||
exist.
|
||||
"""
|
||||
return self._fetch(model_cls, MatchQuery('id', id)).get()
|
||||
@@ -0,0 +1,654 @@
|
||||
# This file is part of beets.
|
||||
# Copyright 2014, Adrian Sampson.
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining
|
||||
# a copy of this software and associated documentation files (the
|
||||
# "Software"), to deal in the Software without restriction, including
|
||||
# without limitation the rights to use, copy, modify, merge, publish,
|
||||
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||
# permit persons to whom the Software is furnished to do so, subject to
|
||||
# the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
|
||||
"""The Query type hierarchy for DBCore.
|
||||
"""
|
||||
import re
|
||||
from operator import attrgetter
|
||||
from beets import util
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
|
||||
class Query(object):
|
||||
"""An abstract class representing a query into the item database.
|
||||
"""
|
||||
def clause(self):
|
||||
"""Generate an SQLite expression implementing the query.
|
||||
Return a clause string, a sequence of substitution values for
|
||||
the clause, and a Query object representing the "remainder"
|
||||
Returns (clause, subvals) where clause is a valid sqlite
|
||||
WHERE clause implementing the query and subvals is a list of
|
||||
items to be substituted for ?s in the clause.
|
||||
"""
|
||||
return None, ()
|
||||
|
||||
def match(self, item):
|
||||
"""Check whether this query matches a given Item. Can be used to
|
||||
perform queries on arbitrary sets of Items.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class FieldQuery(Query):
|
||||
"""An abstract query that searches in a specific field for a
|
||||
pattern. Subclasses must provide a `value_match` class method, which
|
||||
determines whether a certain pattern string matches a certain value
|
||||
string. Subclasses may also provide `col_clause` to implement the
|
||||
same matching functionality in SQLite.
|
||||
"""
|
||||
def __init__(self, field, pattern, fast=True):
|
||||
self.field = field
|
||||
self.pattern = pattern
|
||||
self.fast = fast
|
||||
|
||||
def col_clause(self):
|
||||
return None, ()
|
||||
|
||||
def clause(self):
|
||||
if self.fast:
|
||||
return self.col_clause()
|
||||
else:
|
||||
# Matching a flexattr. This is a slow query.
|
||||
return None, ()
|
||||
|
||||
@classmethod
|
||||
def value_match(cls, pattern, value):
|
||||
"""Determine whether the value matches the pattern. Both
|
||||
arguments are strings.
|
||||
"""
|
||||
raise NotImplementedError()
|
||||
|
||||
def match(self, item):
|
||||
return self.value_match(self.pattern, item.get(self.field))
|
||||
|
||||
|
||||
class MatchQuery(FieldQuery):
|
||||
"""A query that looks for exact matches in an item field."""
|
||||
def col_clause(self):
|
||||
return self.field + " = ?", [self.pattern]
|
||||
|
||||
@classmethod
|
||||
def value_match(cls, pattern, value):
|
||||
return pattern == value
|
||||
|
||||
|
||||
class NoneQuery(FieldQuery):
|
||||
|
||||
def __init__(self, field, fast=True):
|
||||
self.field = field
|
||||
self.fast = fast
|
||||
|
||||
def col_clause(self):
|
||||
return self.field + " IS NULL", ()
|
||||
|
||||
@classmethod
|
||||
def match(self, item):
|
||||
try:
|
||||
return item[self.field] is None
|
||||
except KeyError:
|
||||
return True
|
||||
|
||||
|
||||
class StringFieldQuery(FieldQuery):
|
||||
"""A FieldQuery that converts values to strings before matching
|
||||
them.
|
||||
"""
|
||||
@classmethod
|
||||
def value_match(cls, pattern, value):
|
||||
"""Determine whether the value matches the pattern. The value
|
||||
may have any type.
|
||||
"""
|
||||
return cls.string_match(pattern, util.as_string(value))
|
||||
|
||||
@classmethod
|
||||
def string_match(cls, pattern, value):
|
||||
"""Determine whether the value matches the pattern. Both
|
||||
arguments are strings. Subclasses implement this method.
|
||||
"""
|
||||
raise NotImplementedError()
|
||||
|
||||
|
||||
class SubstringQuery(StringFieldQuery):
|
||||
"""A query that matches a substring in a specific item field."""
|
||||
def col_clause(self):
|
||||
pattern = (self.pattern
|
||||
.replace('\\', '\\\\')
|
||||
.replace('%', '\\%')
|
||||
.replace('_', '\\_'))
|
||||
search = '%' + pattern + '%'
|
||||
clause = self.field + " like ? escape '\\'"
|
||||
subvals = [search]
|
||||
return clause, subvals
|
||||
|
||||
@classmethod
|
||||
def string_match(cls, pattern, value):
|
||||
return pattern.lower() in value.lower()
|
||||
|
||||
|
||||
class RegexpQuery(StringFieldQuery):
|
||||
"""A query that matches a regular expression in a specific item
|
||||
field.
|
||||
"""
|
||||
@classmethod
|
||||
def string_match(cls, pattern, value):
|
||||
try:
|
||||
res = re.search(pattern, value)
|
||||
except re.error:
|
||||
# Invalid regular expression.
|
||||
return False
|
||||
return res is not None
|
||||
|
||||
|
||||
class BooleanQuery(MatchQuery):
|
||||
"""Matches a boolean field. Pattern should either be a boolean or a
|
||||
string reflecting a boolean.
|
||||
"""
|
||||
def __init__(self, field, pattern, fast=True):
|
||||
super(BooleanQuery, self).__init__(field, pattern, fast)
|
||||
if isinstance(pattern, basestring):
|
||||
self.pattern = util.str2bool(pattern)
|
||||
self.pattern = int(self.pattern)
|
||||
|
||||
|
||||
class BytesQuery(MatchQuery):
|
||||
"""Match a raw bytes field (i.e., a path). This is a necessary hack
|
||||
to work around the `sqlite3` module's desire to treat `str` and
|
||||
`unicode` equivalently in Python 2. Always use this query instead of
|
||||
`MatchQuery` when matching on BLOB values.
|
||||
"""
|
||||
def __init__(self, field, pattern):
|
||||
super(BytesQuery, self).__init__(field, pattern)
|
||||
|
||||
# Use a buffer representation of the pattern for SQLite
|
||||
# matching. This instructs SQLite to treat the blob as binary
|
||||
# rather than encoded Unicode.
|
||||
if isinstance(self.pattern, basestring):
|
||||
# Implicitly coerce Unicode strings to their bytes
|
||||
# equivalents.
|
||||
if isinstance(self.pattern, unicode):
|
||||
self.pattern = self.pattern.encode('utf8')
|
||||
self.buf_pattern = buffer(self.pattern)
|
||||
elif isinstance(self.pattern, buffer):
|
||||
self.buf_pattern = self.pattern
|
||||
self.pattern = bytes(self.pattern)
|
||||
|
||||
def col_clause(self):
|
||||
return self.field + " = ?", [self.buf_pattern]
|
||||
|
||||
|
||||
class NumericQuery(FieldQuery):
|
||||
"""Matches numeric fields. A syntax using Ruby-style range ellipses
|
||||
(``..``) lets users specify one- or two-sided ranges. For example,
|
||||
``year:2001..`` finds music released since the turn of the century.
|
||||
"""
|
||||
def _convert(self, s):
|
||||
"""Convert a string to a numeric type (float or int). If the
|
||||
string cannot be converted, return None.
|
||||
"""
|
||||
# This is really just a bit of fun premature optimization.
|
||||
try:
|
||||
return int(s)
|
||||
except ValueError:
|
||||
try:
|
||||
return float(s)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
def __init__(self, field, pattern, fast=True):
|
||||
super(NumericQuery, self).__init__(field, pattern, fast)
|
||||
|
||||
parts = pattern.split('..', 1)
|
||||
if len(parts) == 1:
|
||||
# No range.
|
||||
self.point = self._convert(parts[0])
|
||||
self.rangemin = None
|
||||
self.rangemax = None
|
||||
else:
|
||||
# One- or two-sided range.
|
||||
self.point = None
|
||||
self.rangemin = self._convert(parts[0])
|
||||
self.rangemax = self._convert(parts[1])
|
||||
|
||||
def match(self, item):
|
||||
if self.field not in item:
|
||||
return False
|
||||
value = item[self.field]
|
||||
if isinstance(value, basestring):
|
||||
value = self._convert(value)
|
||||
|
||||
if self.point is not None:
|
||||
return value == self.point
|
||||
else:
|
||||
if self.rangemin is not None and value < self.rangemin:
|
||||
return False
|
||||
if self.rangemax is not None and value > self.rangemax:
|
||||
return False
|
||||
return True
|
||||
|
||||
def col_clause(self):
|
||||
if self.point is not None:
|
||||
return self.field + '=?', (self.point,)
|
||||
else:
|
||||
if self.rangemin is not None and self.rangemax is not None:
|
||||
return (u'{0} >= ? AND {0} <= ?'.format(self.field),
|
||||
(self.rangemin, self.rangemax))
|
||||
elif self.rangemin is not None:
|
||||
return u'{0} >= ?'.format(self.field), (self.rangemin,)
|
||||
elif self.rangemax is not None:
|
||||
return u'{0} <= ?'.format(self.field), (self.rangemax,)
|
||||
else:
|
||||
return '1', ()
|
||||
|
||||
|
||||
class CollectionQuery(Query):
|
||||
"""An abstract query class that aggregates other queries. Can be
|
||||
indexed like a list to access the sub-queries.
|
||||
"""
|
||||
def __init__(self, subqueries=()):
|
||||
self.subqueries = subqueries
|
||||
|
||||
# Act like a sequence.
|
||||
|
||||
def __len__(self):
|
||||
return len(self.subqueries)
|
||||
|
||||
def __getitem__(self, key):
|
||||
return self.subqueries[key]
|
||||
|
||||
def __iter__(self):
|
||||
return iter(self.subqueries)
|
||||
|
||||
def __contains__(self, item):
|
||||
return item in self.subqueries
|
||||
|
||||
def clause_with_joiner(self, joiner):
|
||||
"""Returns a clause created by joining together the clauses of
|
||||
all subqueries with the string joiner (padded by spaces).
|
||||
"""
|
||||
clause_parts = []
|
||||
subvals = []
|
||||
for subq in self.subqueries:
|
||||
subq_clause, subq_subvals = subq.clause()
|
||||
if not subq_clause:
|
||||
# Fall back to slow query.
|
||||
return None, ()
|
||||
clause_parts.append('(' + subq_clause + ')')
|
||||
subvals += subq_subvals
|
||||
clause = (' ' + joiner + ' ').join(clause_parts)
|
||||
return clause, subvals
|
||||
|
||||
|
||||
class AnyFieldQuery(CollectionQuery):
|
||||
"""A query that matches if a given FieldQuery subclass matches in
|
||||
any field. The individual field query class is provided to the
|
||||
constructor.
|
||||
"""
|
||||
def __init__(self, pattern, fields, cls):
|
||||
self.pattern = pattern
|
||||
self.fields = fields
|
||||
self.query_class = cls
|
||||
|
||||
subqueries = []
|
||||
for field in self.fields:
|
||||
subqueries.append(cls(field, pattern, True))
|
||||
super(AnyFieldQuery, self).__init__(subqueries)
|
||||
|
||||
def clause(self):
|
||||
return self.clause_with_joiner('or')
|
||||
|
||||
def match(self, item):
|
||||
for subq in self.subqueries:
|
||||
if subq.match(item):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
class MutableCollectionQuery(CollectionQuery):
|
||||
"""A collection query whose subqueries may be modified after the
|
||||
query is initialized.
|
||||
"""
|
||||
def __setitem__(self, key, value):
|
||||
self.subqueries[key] = value
|
||||
|
||||
def __delitem__(self, key):
|
||||
del self.subqueries[key]
|
||||
|
||||
|
||||
class AndQuery(MutableCollectionQuery):
|
||||
"""A conjunction of a list of other queries."""
|
||||
def clause(self):
|
||||
return self.clause_with_joiner('and')
|
||||
|
||||
def match(self, item):
|
||||
return all([q.match(item) for q in self.subqueries])
|
||||
|
||||
|
||||
class OrQuery(MutableCollectionQuery):
|
||||
"""A conjunction of a list of other queries."""
|
||||
def clause(self):
|
||||
return self.clause_with_joiner('or')
|
||||
|
||||
def match(self, item):
|
||||
return any([q.match(item) for q in self.subqueries])
|
||||
|
||||
|
||||
class TrueQuery(Query):
|
||||
"""A query that always matches."""
|
||||
def clause(self):
|
||||
return '1', ()
|
||||
|
||||
def match(self, item):
|
||||
return True
|
||||
|
||||
|
||||
class FalseQuery(Query):
|
||||
"""A query that never matches."""
|
||||
def clause(self):
|
||||
return '0', ()
|
||||
|
||||
def match(self, item):
|
||||
return False
|
||||
|
||||
|
||||
# Time/date queries.
|
||||
|
||||
def _to_epoch_time(date):
|
||||
"""Convert a `datetime` object to an integer number of seconds since
|
||||
the (local) Unix epoch.
|
||||
"""
|
||||
epoch = datetime.fromtimestamp(0)
|
||||
delta = date - epoch
|
||||
try:
|
||||
return int(delta.total_seconds())
|
||||
except AttributeError:
|
||||
# datetime.timedelta.total_seconds() is not available on Python 2.6
|
||||
return delta.seconds + delta.days * 24 * 3600
|
||||
|
||||
|
||||
def _parse_periods(pattern):
|
||||
"""Parse a string containing two dates separated by two dots (..).
|
||||
Return a pair of `Period` objects.
|
||||
"""
|
||||
parts = pattern.split('..', 1)
|
||||
if len(parts) == 1:
|
||||
instant = Period.parse(parts[0])
|
||||
return (instant, instant)
|
||||
else:
|
||||
start = Period.parse(parts[0])
|
||||
end = Period.parse(parts[1])
|
||||
return (start, end)
|
||||
|
||||
|
||||
class Period(object):
|
||||
"""A period of time given by a date, time and precision.
|
||||
|
||||
Example: 2014-01-01 10:50:30 with precision 'month' represents all
|
||||
instants of time during January 2014.
|
||||
"""
|
||||
|
||||
precisions = ('year', 'month', 'day')
|
||||
date_formats = ('%Y', '%Y-%m', '%Y-%m-%d')
|
||||
|
||||
def __init__(self, date, precision):
|
||||
"""Create a period with the given date (a `datetime` object) and
|
||||
precision (a string, one of "year", "month", or "day").
|
||||
"""
|
||||
if precision not in Period.precisions:
|
||||
raise ValueError('Invalid precision ' + str(precision))
|
||||
self.date = date
|
||||
self.precision = precision
|
||||
|
||||
@classmethod
|
||||
def parse(cls, string):
|
||||
"""Parse a date and return a `Period` object or `None` if the
|
||||
string is empty.
|
||||
"""
|
||||
if not string:
|
||||
return None
|
||||
ordinal = string.count('-')
|
||||
if ordinal >= len(cls.date_formats):
|
||||
# Too many components.
|
||||
return None
|
||||
date_format = cls.date_formats[ordinal]
|
||||
try:
|
||||
date = datetime.strptime(string, date_format)
|
||||
except ValueError:
|
||||
# Parsing failed.
|
||||
return None
|
||||
precision = cls.precisions[ordinal]
|
||||
return cls(date, precision)
|
||||
|
||||
def open_right_endpoint(self):
|
||||
"""Based on the precision, convert the period to a precise
|
||||
`datetime` for use as a right endpoint in a right-open interval.
|
||||
"""
|
||||
precision = self.precision
|
||||
date = self.date
|
||||
if 'year' == self.precision:
|
||||
return date.replace(year=date.year + 1, month=1)
|
||||
elif 'month' == precision:
|
||||
if (date.month < 12):
|
||||
return date.replace(month=date.month + 1)
|
||||
else:
|
||||
return date.replace(year=date.year + 1, month=1)
|
||||
elif 'day' == precision:
|
||||
return date + timedelta(days=1)
|
||||
else:
|
||||
raise ValueError('unhandled precision ' + str(precision))
|
||||
|
||||
|
||||
class DateInterval(object):
|
||||
"""A closed-open interval of dates.
|
||||
|
||||
A left endpoint of None means since the beginning of time.
|
||||
A right endpoint of None means towards infinity.
|
||||
"""
|
||||
|
||||
def __init__(self, start, end):
|
||||
if start is not None and end is not None and not start < end:
|
||||
raise ValueError("start date {0} is not before end date {1}"
|
||||
.format(start, end))
|
||||
self.start = start
|
||||
self.end = end
|
||||
|
||||
@classmethod
|
||||
def from_periods(cls, start, end):
|
||||
"""Create an interval with two Periods as the endpoints.
|
||||
"""
|
||||
end_date = end.open_right_endpoint() if end is not None else None
|
||||
start_date = start.date if start is not None else None
|
||||
return cls(start_date, end_date)
|
||||
|
||||
def contains(self, date):
|
||||
if self.start is not None and date < self.start:
|
||||
return False
|
||||
if self.end is not None and date >= self.end:
|
||||
return False
|
||||
return True
|
||||
|
||||
def __str__(self):
|
||||
return'[{0}, {1})'.format(self.start, self.end)
|
||||
|
||||
|
||||
class DateQuery(FieldQuery):
|
||||
"""Matches date fields stored as seconds since Unix epoch time.
|
||||
|
||||
Dates can be specified as ``year-month-day`` strings where only year
|
||||
is mandatory.
|
||||
|
||||
The value of a date field can be matched against a date interval by
|
||||
using an ellipsis interval syntax similar to that of NumericQuery.
|
||||
"""
|
||||
def __init__(self, field, pattern, fast=True):
|
||||
super(DateQuery, self).__init__(field, pattern, fast)
|
||||
start, end = _parse_periods(pattern)
|
||||
self.interval = DateInterval.from_periods(start, end)
|
||||
|
||||
def match(self, item):
|
||||
timestamp = float(item[self.field])
|
||||
date = datetime.utcfromtimestamp(timestamp)
|
||||
return self.interval.contains(date)
|
||||
|
||||
_clause_tmpl = "{0} {1} ?"
|
||||
|
||||
def col_clause(self):
|
||||
clause_parts = []
|
||||
subvals = []
|
||||
|
||||
if self.interval.start:
|
||||
clause_parts.append(self._clause_tmpl.format(self.field, ">="))
|
||||
subvals.append(_to_epoch_time(self.interval.start))
|
||||
|
||||
if self.interval.end:
|
||||
clause_parts.append(self._clause_tmpl.format(self.field, "<"))
|
||||
subvals.append(_to_epoch_time(self.interval.end))
|
||||
|
||||
if clause_parts:
|
||||
# One- or two-sided interval.
|
||||
clause = ' AND '.join(clause_parts)
|
||||
else:
|
||||
# Match any date.
|
||||
clause = '1'
|
||||
return clause, subvals
|
||||
|
||||
|
||||
# Sorting.
|
||||
|
||||
class Sort(object):
|
||||
"""An abstract class representing a sort operation for a query into
|
||||
the item database.
|
||||
"""
|
||||
|
||||
def order_clause(self):
|
||||
"""Generates a SQL fragment to be used in a ORDER BY clause, or
|
||||
None if no fragment is used (i.e., this is a slow sort).
|
||||
"""
|
||||
return None
|
||||
|
||||
def sort(self, items):
|
||||
"""Sort the list of objects and return a list.
|
||||
"""
|
||||
return sorted(items)
|
||||
|
||||
def is_slow(self):
|
||||
"""Indicate whether this query is *slow*, meaning that it cannot
|
||||
be executed in SQL and must be executed in Python.
|
||||
"""
|
||||
return False
|
||||
|
||||
|
||||
class MultipleSort(Sort):
|
||||
"""Sort that encapsulates multiple sub-sorts.
|
||||
"""
|
||||
|
||||
def __init__(self, sorts=None):
|
||||
self.sorts = sorts or []
|
||||
|
||||
def add_sort(self, sort):
|
||||
self.sorts.append(sort)
|
||||
|
||||
def _sql_sorts(self):
|
||||
"""Return the list of sub-sorts for which we can be (at least
|
||||
partially) fast.
|
||||
|
||||
A contiguous suffix of fast (SQL-capable) sub-sorts are
|
||||
executable in SQL. The remaining, even if they are fast
|
||||
independently, must be executed slowly.
|
||||
"""
|
||||
sql_sorts = []
|
||||
for sort in reversed(self.sorts):
|
||||
if not sort.order_clause() is None:
|
||||
sql_sorts.append(sort)
|
||||
else:
|
||||
break
|
||||
sql_sorts.reverse()
|
||||
return sql_sorts
|
||||
|
||||
def order_clause(self):
|
||||
order_strings = []
|
||||
for sort in self._sql_sorts():
|
||||
order = sort.order_clause()
|
||||
order_strings.append(order)
|
||||
|
||||
return ", ".join(order_strings)
|
||||
|
||||
def is_slow(self):
|
||||
for sort in self.sorts:
|
||||
if sort.is_slow():
|
||||
return True
|
||||
return False
|
||||
|
||||
def sort(self, items):
|
||||
slow_sorts = []
|
||||
switch_slow = False
|
||||
for sort in reversed(self.sorts):
|
||||
if switch_slow:
|
||||
slow_sorts.append(sort)
|
||||
elif sort.order_clause() is None:
|
||||
switch_slow = True
|
||||
slow_sorts.append(sort)
|
||||
else:
|
||||
pass
|
||||
|
||||
for sort in slow_sorts:
|
||||
items = sort.sort(items)
|
||||
return items
|
||||
|
||||
def __repr__(self):
|
||||
return u'MultipleSort({0})'.format(repr(self.sorts))
|
||||
|
||||
|
||||
class FieldSort(Sort):
|
||||
"""An abstract sort criterion that orders by a specific field (of
|
||||
any kind).
|
||||
"""
|
||||
def __init__(self, field, ascending=True):
|
||||
self.field = field
|
||||
self.ascending = ascending
|
||||
|
||||
def sort(self, objs):
|
||||
# TODO: Conversion and null-detection here. In Python 3,
|
||||
# comparisons with None fail. We should also support flexible
|
||||
# attributes with different types without falling over.
|
||||
return sorted(objs, key=attrgetter(self.field),
|
||||
reverse=not self.ascending)
|
||||
|
||||
def __repr__(self):
|
||||
return u'<{0}: {1}{2}>'.format(
|
||||
type(self).__name__,
|
||||
self.field,
|
||||
'+' if self.ascending else '-',
|
||||
)
|
||||
|
||||
|
||||
class FixedFieldSort(FieldSort):
|
||||
"""Sort object to sort on a fixed field.
|
||||
"""
|
||||
def order_clause(self):
|
||||
order = "ASC" if self.ascending else "DESC"
|
||||
return "{0} {1}".format(self.field, order)
|
||||
|
||||
|
||||
class SlowFieldSort(FieldSort):
|
||||
"""A sort criterion by some model field other than a fixed field:
|
||||
i.e., a computed or flexible field.
|
||||
"""
|
||||
def is_slow(self):
|
||||
return True
|
||||
|
||||
|
||||
class NullSort(Sort):
|
||||
"""No sorting. Leave results unsorted."""
|
||||
def sort(items):
|
||||
return items
|
||||
@@ -0,0 +1,180 @@
|
||||
# This file is part of beets.
|
||||
# Copyright 2014, Adrian Sampson.
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining
|
||||
# a copy of this software and associated documentation files (the
|
||||
# "Software"), to deal in the Software without restriction, including
|
||||
# without limitation the rights to use, copy, modify, merge, publish,
|
||||
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||
# permit persons to whom the Software is furnished to do so, subject to
|
||||
# the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
|
||||
"""Parsing of strings into DBCore queries.
|
||||
"""
|
||||
import re
|
||||
import itertools
|
||||
from . import query
|
||||
|
||||
|
||||
PARSE_QUERY_PART_REGEX = re.compile(
|
||||
# Non-capturing optional segment for the keyword.
|
||||
r'(?:'
|
||||
r'(\S+?)' # The field key.
|
||||
r'(?<!\\):' # Unescaped :
|
||||
r')?'
|
||||
|
||||
r'(.*)', # The term itself.
|
||||
|
||||
re.I # Case-insensitive.
|
||||
)
|
||||
|
||||
|
||||
def parse_query_part(part, query_classes={}, prefixes={},
|
||||
default_class=query.SubstringQuery):
|
||||
"""Take a query in the form of a key/value pair separated by a
|
||||
colon and return a tuple of `(key, value, cls)`. `key` may be None,
|
||||
indicating that any field may be matched. `cls` is a subclass of
|
||||
`FieldQuery`.
|
||||
|
||||
The optional `query_classes` parameter maps field names to default
|
||||
query types; `default_class` is the fallback. `prefixes` is a map
|
||||
from query prefix markers and query types. Prefix-indicated queries
|
||||
take precedence over type-based queries.
|
||||
|
||||
To determine the query class, two factors are used: prefixes and
|
||||
field types. For example, the colon prefix denotes a regular
|
||||
expression query and a type map might provide a special kind of
|
||||
query for numeric values. If neither a prefix nor a specific query
|
||||
class is available, `default_class` is used.
|
||||
|
||||
For instance,
|
||||
'stapler' -> (None, 'stapler', SubstringQuery)
|
||||
'color:red' -> ('color', 'red', SubstringQuery)
|
||||
':^Quiet' -> (None, '^Quiet', RegexpQuery)
|
||||
'color::b..e' -> ('color', 'b..e', RegexpQuery)
|
||||
|
||||
Prefixes may be "escaped" with a backslash to disable the keying
|
||||
behavior.
|
||||
"""
|
||||
part = part.strip()
|
||||
match = PARSE_QUERY_PART_REGEX.match(part)
|
||||
|
||||
assert match # Regex should always match.
|
||||
key = match.group(1)
|
||||
term = match.group(2).replace('\:', ':')
|
||||
|
||||
# Match the search term against the list of prefixes.
|
||||
for pre, query_class in prefixes.items():
|
||||
if term.startswith(pre):
|
||||
return key, term[len(pre):], query_class
|
||||
|
||||
# No matching prefix: use type-based or fallback/default query.
|
||||
query_class = query_classes.get(key, default_class)
|
||||
return key, term, query_class
|
||||
|
||||
|
||||
def construct_query_part(model_cls, prefixes, query_part):
|
||||
"""Create a query from a single query component, `query_part`, for
|
||||
querying instances of `model_cls`. Return a `Query` instance.
|
||||
"""
|
||||
# Shortcut for empty query parts.
|
||||
if not query_part:
|
||||
return query.TrueQuery()
|
||||
|
||||
# Get the query classes for each possible field.
|
||||
query_classes = {}
|
||||
for k, t in itertools.chain(model_cls._fields.items(),
|
||||
model_cls._types.items()):
|
||||
query_classes[k] = t.query
|
||||
|
||||
# Parse the string.
|
||||
key, pattern, query_class = \
|
||||
parse_query_part(query_part, query_classes, prefixes)
|
||||
|
||||
# No key specified.
|
||||
if key is None:
|
||||
if issubclass(query_class, query.FieldQuery):
|
||||
# The query type matches a specific field, but none was
|
||||
# specified. So we use a version of the query that matches
|
||||
# any field.
|
||||
return query.AnyFieldQuery(pattern, model_cls._search_fields,
|
||||
query_class)
|
||||
else:
|
||||
# Other query type.
|
||||
return query_class(pattern)
|
||||
|
||||
key = key.lower()
|
||||
return query_class(key.lower(), pattern, key in model_cls._fields)
|
||||
|
||||
|
||||
def query_from_strings(query_cls, model_cls, prefixes, query_parts):
|
||||
"""Creates a collection query of type `query_cls` from a list of
|
||||
strings in the format used by parse_query_part. `model_cls`
|
||||
determines how queries are constructed from strings.
|
||||
"""
|
||||
subqueries = []
|
||||
for part in query_parts:
|
||||
subqueries.append(construct_query_part(model_cls, prefixes, part))
|
||||
if not subqueries: # No terms in query.
|
||||
subqueries = [query.TrueQuery()]
|
||||
return query_cls(subqueries)
|
||||
|
||||
|
||||
def construct_sort_part(model_cls, part):
|
||||
"""Create a `Sort` from a single string criterion.
|
||||
|
||||
`model_cls` is the `Model` being queried. `part` is a single string
|
||||
ending in ``+`` or ``-`` indicating the sort.
|
||||
"""
|
||||
assert part, "part must be a field name and + or -"
|
||||
field = part[:-1]
|
||||
assert field, "field is missing"
|
||||
direction = part[-1]
|
||||
assert direction in ('+', '-'), "part must end with + or -"
|
||||
is_ascending = direction == '+'
|
||||
|
||||
if field in model_cls._sorts:
|
||||
sort = model_cls._sorts[field](model_cls, is_ascending)
|
||||
elif field in model_cls._fields:
|
||||
sort = query.FixedFieldSort(field, is_ascending)
|
||||
else:
|
||||
# Flexible or computed.
|
||||
sort = query.SlowFieldSort(field, is_ascending)
|
||||
return sort
|
||||
|
||||
|
||||
def sort_from_strings(model_cls, sort_parts):
|
||||
"""Create a `Sort` from a list of sort criteria (strings).
|
||||
"""
|
||||
if not sort_parts:
|
||||
return query.NullSort()
|
||||
else:
|
||||
sort = query.MultipleSort()
|
||||
for part in sort_parts:
|
||||
sort.add_sort(construct_sort_part(model_cls, part))
|
||||
return sort
|
||||
|
||||
|
||||
def parse_sorted_query(model_cls, parts, prefixes={},
|
||||
query_cls=query.AndQuery):
|
||||
"""Given a list of strings, create the `Query` and `Sort` that they
|
||||
represent.
|
||||
"""
|
||||
# Separate query token and sort token.
|
||||
query_parts = []
|
||||
sort_parts = []
|
||||
for part in parts:
|
||||
if part.endswith((u'+', u'-')) and u':' not in part:
|
||||
sort_parts.append(part)
|
||||
else:
|
||||
query_parts.append(part)
|
||||
|
||||
# Parse each.
|
||||
q = query_from_strings(
|
||||
query_cls, model_cls, prefixes, query_parts
|
||||
)
|
||||
s = sort_from_strings(model_cls, sort_parts)
|
||||
return q, s
|
||||
@@ -0,0 +1,208 @@
|
||||
# This file is part of beets.
|
||||
# Copyright 2014, Adrian Sampson.
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining
|
||||
# a copy of this software and associated documentation files (the
|
||||
# "Software"), to deal in the Software without restriction, including
|
||||
# without limitation the rights to use, copy, modify, merge, publish,
|
||||
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||
# permit persons to whom the Software is furnished to do so, subject to
|
||||
# the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
|
||||
"""Representation of type information for DBCore model fields.
|
||||
"""
|
||||
from . import query
|
||||
from beets.util import str2bool
|
||||
|
||||
|
||||
# Abstract base.
|
||||
|
||||
class Type(object):
|
||||
"""An object encapsulating the type of a model field. Includes
|
||||
information about how to store, query, format, and parse a given
|
||||
field.
|
||||
"""
|
||||
|
||||
sql = u'TEXT'
|
||||
"""The SQLite column type for the value.
|
||||
"""
|
||||
|
||||
query = query.SubstringQuery
|
||||
"""The `Query` subclass to be used when querying the field.
|
||||
"""
|
||||
|
||||
model_type = unicode
|
||||
"""The Python type that is used to represent the value in the model.
|
||||
|
||||
The model is guaranteed to return a value of this type if the field
|
||||
is accessed. To this end, the constructor is used by the `normalize`
|
||||
and `from_sql` methods and the `default` property.
|
||||
"""
|
||||
|
||||
@property
|
||||
def null(self):
|
||||
"""The value to be exposed when the underlying value is None.
|
||||
"""
|
||||
return self.model_type()
|
||||
|
||||
def format(self, value):
|
||||
"""Given a value of this type, produce a Unicode string
|
||||
representing the value. This is used in template evaluation.
|
||||
"""
|
||||
if value is None:
|
||||
value = self.null
|
||||
# `self.null` might be `None`
|
||||
if value is None:
|
||||
value = u''
|
||||
if isinstance(value, bytes):
|
||||
value = value.decode('utf8', 'ignore')
|
||||
|
||||
return unicode(value)
|
||||
|
||||
def parse(self, string):
|
||||
"""Parse a (possibly human-written) string and return the
|
||||
indicated value of this type.
|
||||
"""
|
||||
try:
|
||||
return self.model_type(string)
|
||||
except ValueError:
|
||||
return self.null
|
||||
|
||||
def normalize(self, value):
|
||||
"""Given a value that will be assigned into a field of this
|
||||
type, normalize the value to have the appropriate type. This
|
||||
base implementation only reinterprets `None`.
|
||||
"""
|
||||
if value is None:
|
||||
return self.null
|
||||
else:
|
||||
# TODO This should eventually be replaced by
|
||||
# `self.model_type(value)`
|
||||
return value
|
||||
|
||||
def from_sql(self, sql_value):
|
||||
"""Receives the value stored in the SQL backend and return the
|
||||
value to be stored in the model.
|
||||
|
||||
For fixed fields the type of `value` is determined by the column
|
||||
type affinity given in the `sql` property and the SQL to Python
|
||||
mapping of the database adapter. For more information see:
|
||||
http://www.sqlite.org/datatype3.html
|
||||
https://docs.python.org/2/library/sqlite3.html#sqlite-and-python-types
|
||||
|
||||
Flexible fields have the type afinity `TEXT`. This means the
|
||||
`sql_value` is either a `buffer` or a `unicode` object` and the
|
||||
method must handle these in addition.
|
||||
"""
|
||||
if isinstance(sql_value, buffer):
|
||||
sql_value = bytes(sql_value).decode('utf8', 'ignore')
|
||||
if isinstance(sql_value, unicode):
|
||||
return self.parse(sql_value)
|
||||
else:
|
||||
return self.normalize(sql_value)
|
||||
|
||||
def to_sql(self, model_value):
|
||||
"""Convert a value as stored in the model object to a value used
|
||||
by the database adapter.
|
||||
"""
|
||||
return model_value
|
||||
|
||||
|
||||
# Reusable types.
|
||||
|
||||
class Default(Type):
|
||||
null = None
|
||||
|
||||
|
||||
class Integer(Type):
|
||||
"""A basic integer type.
|
||||
"""
|
||||
sql = u'INTEGER'
|
||||
query = query.NumericQuery
|
||||
model_type = int
|
||||
|
||||
|
||||
class PaddedInt(Integer):
|
||||
"""An integer field that is formatted with a given number of digits,
|
||||
padded with zeroes.
|
||||
"""
|
||||
def __init__(self, digits):
|
||||
self.digits = digits
|
||||
|
||||
def format(self, value):
|
||||
return u'{0:0{1}d}'.format(value or 0, self.digits)
|
||||
|
||||
|
||||
class ScaledInt(Integer):
|
||||
"""An integer whose formatting operation scales the number by a
|
||||
constant and adds a suffix. Good for units with large magnitudes.
|
||||
"""
|
||||
def __init__(self, unit, suffix=u''):
|
||||
self.unit = unit
|
||||
self.suffix = suffix
|
||||
|
||||
def format(self, value):
|
||||
return u'{0}{1}'.format((value or 0) // self.unit, self.suffix)
|
||||
|
||||
|
||||
class Id(Integer):
|
||||
"""An integer used as the row id or a foreign key in a SQLite table.
|
||||
This type is nullable: None values are not translated to zero.
|
||||
"""
|
||||
null = None
|
||||
|
||||
def __init__(self, primary=True):
|
||||
if primary:
|
||||
self.sql = u'INTEGER PRIMARY KEY'
|
||||
|
||||
|
||||
class Float(Type):
|
||||
"""A basic floating-point type.
|
||||
"""
|
||||
sql = u'REAL'
|
||||
query = query.NumericQuery
|
||||
model_type = float
|
||||
|
||||
def format(self, value):
|
||||
return u'{0:.1f}'.format(value or 0.0)
|
||||
|
||||
|
||||
class NullFloat(Float):
|
||||
"""Same as `Float`, but does not normalize `None` to `0.0`.
|
||||
"""
|
||||
null = None
|
||||
|
||||
|
||||
class String(Type):
|
||||
"""A Unicode string type.
|
||||
"""
|
||||
sql = u'TEXT'
|
||||
query = query.SubstringQuery
|
||||
|
||||
|
||||
class Boolean(Type):
|
||||
"""A boolean type.
|
||||
"""
|
||||
sql = u'INTEGER'
|
||||
query = query.BooleanQuery
|
||||
model_type = bool
|
||||
|
||||
def format(self, value):
|
||||
return unicode(bool(value))
|
||||
|
||||
def parse(self, string):
|
||||
return str2bool(string)
|
||||
|
||||
|
||||
# Shared instances of common types.
|
||||
DEFAULT = Default()
|
||||
INTEGER = Integer()
|
||||
PRIMARY_ID = Id(True)
|
||||
FOREIGN_ID = Id(False)
|
||||
FLOAT = Float()
|
||||
NULL_FLOAT = NullFloat()
|
||||
STRING = String()
|
||||
BOOLEAN = Boolean()
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Executable
+435
@@ -0,0 +1,435 @@
|
||||
# This file is part of beets.
|
||||
# Copyright 2013, Adrian Sampson.
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining
|
||||
# a copy of this software and associated documentation files (the
|
||||
# "Software"), to deal in the Software without restriction, including
|
||||
# without limitation the rights to use, copy, modify, merge, publish,
|
||||
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||
# permit persons to whom the Software is furnished to do so, subject to
|
||||
# the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
|
||||
"""Support for beets plugins."""
|
||||
|
||||
import logging
|
||||
import traceback
|
||||
import inspect
|
||||
import re
|
||||
from collections import defaultdict
|
||||
|
||||
|
||||
import beets
|
||||
from beets import mediafile
|
||||
|
||||
PLUGIN_NAMESPACE = 'beetsplug'
|
||||
|
||||
# Plugins using the Last.fm API can share the same API key.
|
||||
LASTFM_KEY = '2dc3914abf35f0d9c92d97d8f8e42b43'
|
||||
|
||||
# Global logger.
|
||||
log = logging.getLogger('beets')
|
||||
|
||||
|
||||
class PluginConflictException(Exception):
|
||||
"""Indicates that the services provided by one plugin conflict with
|
||||
those of another.
|
||||
|
||||
For example two plugins may define different types for flexible fields.
|
||||
"""
|
||||
|
||||
|
||||
# Managing the plugins themselves.
|
||||
|
||||
class BeetsPlugin(object):
|
||||
"""The base class for all beets plugins. Plugins provide
|
||||
functionality by defining a subclass of BeetsPlugin and overriding
|
||||
the abstract methods defined here.
|
||||
"""
|
||||
def __init__(self, name=None):
|
||||
"""Perform one-time plugin setup.
|
||||
"""
|
||||
self.import_stages = []
|
||||
self.name = name or self.__module__.split('.')[-1]
|
||||
self.config = beets.config[self.name]
|
||||
if not self.template_funcs:
|
||||
self.template_funcs = {}
|
||||
if not self.template_fields:
|
||||
self.template_fields = {}
|
||||
if not self.album_template_fields:
|
||||
self.album_template_fields = {}
|
||||
|
||||
def commands(self):
|
||||
"""Should return a list of beets.ui.Subcommand objects for
|
||||
commands that should be added to beets' CLI.
|
||||
"""
|
||||
return ()
|
||||
|
||||
def queries(self):
|
||||
"""Should return a dict mapping prefixes to Query subclasses.
|
||||
"""
|
||||
return {}
|
||||
|
||||
def track_distance(self, item, info):
|
||||
"""Should return a Distance object to be added to the
|
||||
distance for every track comparison.
|
||||
"""
|
||||
return beets.autotag.hooks.Distance()
|
||||
|
||||
def album_distance(self, items, album_info, mapping):
|
||||
"""Should return a Distance object to be added to the
|
||||
distance for every album-level comparison.
|
||||
"""
|
||||
return beets.autotag.hooks.Distance()
|
||||
|
||||
def candidates(self, items, artist, album, va_likely):
|
||||
"""Should return a sequence of AlbumInfo objects that match the
|
||||
album whose items are provided.
|
||||
"""
|
||||
return ()
|
||||
|
||||
def item_candidates(self, item, artist, title):
|
||||
"""Should return a sequence of TrackInfo objects that match the
|
||||
item provided.
|
||||
"""
|
||||
return ()
|
||||
|
||||
def album_for_id(self, album_id):
|
||||
"""Return an AlbumInfo object or None if no matching release was
|
||||
found.
|
||||
"""
|
||||
return None
|
||||
|
||||
def track_for_id(self, track_id):
|
||||
"""Return a TrackInfo object or None if no matching release was
|
||||
found.
|
||||
"""
|
||||
return None
|
||||
|
||||
def add_media_field(self, name, descriptor):
|
||||
"""Add a field that is synchronized between media files and items.
|
||||
|
||||
When a media field is added ``item.write()`` will set the name
|
||||
property of the item's MediaFile to ``item[name]`` and save the
|
||||
changes. Similarly ``item.read()`` will set ``item[name]`` to
|
||||
the value of the name property of the media file.
|
||||
|
||||
``descriptor`` must be an instance of ``mediafile.MediaField``.
|
||||
"""
|
||||
# Defer impor to prevent circular dependency
|
||||
from beets import library
|
||||
mediafile.MediaFile.add_field(name, descriptor)
|
||||
library.Item._media_fields.add(name)
|
||||
|
||||
listeners = None
|
||||
|
||||
@classmethod
|
||||
def register_listener(cls, event, func):
|
||||
"""Add a function as a listener for the specified event. (An
|
||||
imperative alternative to the @listen decorator.)
|
||||
"""
|
||||
if cls.listeners is None:
|
||||
cls.listeners = defaultdict(list)
|
||||
cls.listeners[event].append(func)
|
||||
|
||||
@classmethod
|
||||
def listen(cls, event):
|
||||
"""Decorator that adds a function as an event handler for the
|
||||
specified event (as a string). The parameters passed to function
|
||||
will vary depending on what event occurred.
|
||||
|
||||
The function should respond to named parameters.
|
||||
function(**kwargs) will trap all arguments in a dictionary.
|
||||
Example:
|
||||
|
||||
>>> @MyPlugin.listen("imported")
|
||||
>>> def importListener(**kwargs):
|
||||
... pass
|
||||
"""
|
||||
def helper(func):
|
||||
if cls.listeners is None:
|
||||
cls.listeners = defaultdict(list)
|
||||
cls.listeners[event].append(func)
|
||||
return func
|
||||
return helper
|
||||
|
||||
template_funcs = None
|
||||
template_fields = None
|
||||
album_template_fields = None
|
||||
|
||||
@classmethod
|
||||
def template_func(cls, name):
|
||||
"""Decorator that registers a path template function. The
|
||||
function will be invoked as ``%name{}`` from path format
|
||||
strings.
|
||||
"""
|
||||
def helper(func):
|
||||
if cls.template_funcs is None:
|
||||
cls.template_funcs = {}
|
||||
cls.template_funcs[name] = func
|
||||
return func
|
||||
return helper
|
||||
|
||||
@classmethod
|
||||
def template_field(cls, name):
|
||||
"""Decorator that registers a path template field computation.
|
||||
The value will be referenced as ``$name`` from path format
|
||||
strings. The function must accept a single parameter, the Item
|
||||
being formatted.
|
||||
"""
|
||||
def helper(func):
|
||||
if cls.template_fields is None:
|
||||
cls.template_fields = {}
|
||||
cls.template_fields[name] = func
|
||||
return func
|
||||
return helper
|
||||
|
||||
|
||||
_classes = set()
|
||||
|
||||
|
||||
def load_plugins(names=()):
|
||||
"""Imports the modules for a sequence of plugin names. Each name
|
||||
must be the name of a Python module under the "beetsplug" namespace
|
||||
package in sys.path; the module indicated should contain the
|
||||
BeetsPlugin subclasses desired.
|
||||
"""
|
||||
for name in names:
|
||||
modname = '%s.%s' % (PLUGIN_NAMESPACE, name)
|
||||
try:
|
||||
try:
|
||||
namespace = __import__(modname, None, None)
|
||||
except ImportError as exc:
|
||||
# Again, this is hacky:
|
||||
if exc.args[0].endswith(' ' + name):
|
||||
log.warn(u'** plugin {0} not found'.format(name))
|
||||
else:
|
||||
raise
|
||||
else:
|
||||
for obj in getattr(namespace, name).__dict__.values():
|
||||
if isinstance(obj, type) and issubclass(obj, BeetsPlugin) \
|
||||
and obj != BeetsPlugin and obj not in _classes:
|
||||
_classes.add(obj)
|
||||
|
||||
except:
|
||||
log.warn(u'** error loading plugin {0}'.format(name))
|
||||
log.warn(traceback.format_exc())
|
||||
|
||||
|
||||
_instances = {}
|
||||
|
||||
|
||||
def find_plugins():
|
||||
"""Returns a list of BeetsPlugin subclass instances from all
|
||||
currently loaded beets plugins. Loads the default plugin set
|
||||
first.
|
||||
"""
|
||||
load_plugins()
|
||||
plugins = []
|
||||
for cls in _classes:
|
||||
# Only instantiate each plugin class once.
|
||||
if cls not in _instances:
|
||||
_instances[cls] = cls()
|
||||
plugins.append(_instances[cls])
|
||||
return plugins
|
||||
|
||||
|
||||
# Communication with plugins.
|
||||
|
||||
def commands():
|
||||
"""Returns a list of Subcommand objects from all loaded plugins.
|
||||
"""
|
||||
out = []
|
||||
for plugin in find_plugins():
|
||||
out += plugin.commands()
|
||||
return out
|
||||
|
||||
|
||||
def queries():
|
||||
"""Returns a dict mapping prefix strings to Query subclasses all loaded
|
||||
plugins.
|
||||
"""
|
||||
out = {}
|
||||
for plugin in find_plugins():
|
||||
out.update(plugin.queries())
|
||||
return out
|
||||
|
||||
|
||||
def types(model_cls):
|
||||
# Gives us `item_types` and `album_types`
|
||||
attr_name = '{0}_types'.format(model_cls.__name__.lower())
|
||||
types = {}
|
||||
for plugin in find_plugins():
|
||||
plugin_types = getattr(plugin, attr_name, {})
|
||||
for field in plugin_types:
|
||||
if field in types and plugin_types[field] != types[field]:
|
||||
raise PluginConflictException(
|
||||
u'Plugin {0} defines flexible field {1} '
|
||||
'which has already been defined with '
|
||||
'another type.'.format(plugin.name, field)
|
||||
)
|
||||
types.update(plugin_types)
|
||||
return types
|
||||
|
||||
|
||||
def track_distance(item, info):
|
||||
"""Gets the track distance calculated by all loaded plugins.
|
||||
Returns a Distance object.
|
||||
"""
|
||||
from beets.autotag.hooks import Distance
|
||||
dist = Distance()
|
||||
for plugin in find_plugins():
|
||||
dist.update(plugin.track_distance(item, info))
|
||||
return dist
|
||||
|
||||
|
||||
def album_distance(items, album_info, mapping):
|
||||
"""Returns the album distance calculated by plugins."""
|
||||
from beets.autotag.hooks import Distance
|
||||
dist = Distance()
|
||||
for plugin in find_plugins():
|
||||
dist.update(plugin.album_distance(items, album_info, mapping))
|
||||
return dist
|
||||
|
||||
|
||||
def candidates(items, artist, album, va_likely):
|
||||
"""Gets MusicBrainz candidates for an album from each plugin.
|
||||
"""
|
||||
out = []
|
||||
for plugin in find_plugins():
|
||||
out.extend(plugin.candidates(items, artist, album, va_likely))
|
||||
return out
|
||||
|
||||
|
||||
def item_candidates(item, artist, title):
|
||||
"""Gets MusicBrainz candidates for an item from the plugins.
|
||||
"""
|
||||
out = []
|
||||
for plugin in find_plugins():
|
||||
out.extend(plugin.item_candidates(item, artist, title))
|
||||
return out
|
||||
|
||||
|
||||
def album_for_id(album_id):
|
||||
"""Get AlbumInfo objects for a given ID string.
|
||||
"""
|
||||
out = []
|
||||
for plugin in find_plugins():
|
||||
res = plugin.album_for_id(album_id)
|
||||
if res:
|
||||
out.append(res)
|
||||
return out
|
||||
|
||||
|
||||
def track_for_id(track_id):
|
||||
"""Get TrackInfo objects for a given ID string.
|
||||
"""
|
||||
out = []
|
||||
for plugin in find_plugins():
|
||||
res = plugin.track_for_id(track_id)
|
||||
if res:
|
||||
out.append(res)
|
||||
return out
|
||||
|
||||
|
||||
def template_funcs():
|
||||
"""Get all the template functions declared by plugins as a
|
||||
dictionary.
|
||||
"""
|
||||
funcs = {}
|
||||
for plugin in find_plugins():
|
||||
if plugin.template_funcs:
|
||||
funcs.update(plugin.template_funcs)
|
||||
return funcs
|
||||
|
||||
|
||||
def import_stages():
|
||||
"""Get a list of import stage functions defined by plugins."""
|
||||
stages = []
|
||||
for plugin in find_plugins():
|
||||
if hasattr(plugin, 'import_stages'):
|
||||
stages += plugin.import_stages
|
||||
return stages
|
||||
|
||||
|
||||
# New-style (lazy) plugin-provided fields.
|
||||
|
||||
def item_field_getters():
|
||||
"""Get a dictionary mapping field names to unary functions that
|
||||
compute the field's value.
|
||||
"""
|
||||
funcs = {}
|
||||
for plugin in find_plugins():
|
||||
if plugin.template_fields:
|
||||
funcs.update(plugin.template_fields)
|
||||
return funcs
|
||||
|
||||
|
||||
def album_field_getters():
|
||||
"""As above, for album fields.
|
||||
"""
|
||||
funcs = {}
|
||||
for plugin in find_plugins():
|
||||
if plugin.album_template_fields:
|
||||
funcs.update(plugin.album_template_fields)
|
||||
return funcs
|
||||
|
||||
|
||||
# Event dispatch.
|
||||
|
||||
def event_handlers():
|
||||
"""Find all event handlers from plugins as a dictionary mapping
|
||||
event names to sequences of callables.
|
||||
"""
|
||||
all_handlers = defaultdict(list)
|
||||
for plugin in find_plugins():
|
||||
if plugin.listeners:
|
||||
for event, handlers in plugin.listeners.items():
|
||||
all_handlers[event] += handlers
|
||||
return all_handlers
|
||||
|
||||
|
||||
def send(event, **arguments):
|
||||
"""Sends an event to all assigned event listeners. Event is the
|
||||
name of the event to send, all other named arguments go to the
|
||||
event handler(s).
|
||||
|
||||
Returns a list of return values from the handlers.
|
||||
"""
|
||||
log.debug(u'Sending event: {0}'.format(event))
|
||||
for handler in event_handlers()[event]:
|
||||
# Don't break legacy plugins if we want to pass more arguments
|
||||
argspec = inspect.getargspec(handler).args
|
||||
args = dict((k, v) for k, v in arguments.items() if k in argspec)
|
||||
handler(**args)
|
||||
|
||||
|
||||
def feat_tokens(for_artist=True):
|
||||
"""Return a regular expression that matches phrases like "featuring"
|
||||
that separate a main artist or a song title from secondary artists.
|
||||
The `for_artist` option determines whether the regex should be
|
||||
suitable for matching artist fields (the default) or title fields.
|
||||
"""
|
||||
feat_words = ['ft', 'featuring', 'feat', 'feat.', 'ft.']
|
||||
if for_artist:
|
||||
feat_words += ['with', 'vs', 'and', 'con', '&']
|
||||
return '(?<=\s)(?:{0})(?=\s)'.format(
|
||||
'|'.join(re.escape(x) for x in feat_words)
|
||||
)
|
||||
|
||||
|
||||
def sanitize_choices(choices, choices_all):
|
||||
"""Clean up a stringlist configuration attribute: keep only choices
|
||||
elements present in choices_all, remove duplicate elements, expand '*'
|
||||
wildcard while keeping original stringlist order.
|
||||
"""
|
||||
seen = set()
|
||||
others = [x for x in choices_all if x not in choices]
|
||||
res = []
|
||||
for s in choices:
|
||||
if s in list(choices_all) + ['*']:
|
||||
if not (s in seen or seen.add(s)):
|
||||
res.extend(list(others) if s == '*' else [s])
|
||||
return res
|
||||
@@ -0,0 +1,970 @@
|
||||
# This file is part of beets.
|
||||
# Copyright 2014, Adrian Sampson.
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining
|
||||
# a copy of this software and associated documentation files (the
|
||||
# "Software"), to deal in the Software without restriction, including
|
||||
# without limitation the rights to use, copy, modify, merge, publish,
|
||||
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||
# permit persons to whom the Software is furnished to do so, subject to
|
||||
# the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
|
||||
"""This module contains all of the core logic for beets' command-line
|
||||
interface. To invoke the CLI, just call beets.ui.main(). The actual
|
||||
CLI commands are implemented in the ui.commands module.
|
||||
"""
|
||||
from __future__ import print_function
|
||||
|
||||
import locale
|
||||
import optparse
|
||||
import textwrap
|
||||
import sys
|
||||
from difflib import SequenceMatcher
|
||||
import logging
|
||||
import sqlite3
|
||||
import errno
|
||||
import re
|
||||
import struct
|
||||
import traceback
|
||||
import os.path
|
||||
|
||||
from beets import library
|
||||
from beets import plugins
|
||||
from beets import util
|
||||
from beets.util.functemplate import Template
|
||||
from beets import config
|
||||
from beets.util import confit
|
||||
from beets.autotag import mb
|
||||
|
||||
# On Windows platforms, use colorama to support "ANSI" terminal colors.
|
||||
if sys.platform == 'win32':
|
||||
try:
|
||||
import colorama
|
||||
except ImportError:
|
||||
pass
|
||||
else:
|
||||
colorama.init()
|
||||
|
||||
|
||||
log = logging.getLogger('beets')
|
||||
if not log.handlers:
|
||||
log.addHandler(logging.StreamHandler())
|
||||
log.propagate = False # Don't propagate to root handler.
|
||||
|
||||
|
||||
PF_KEY_QUERIES = {
|
||||
'comp': 'comp:true',
|
||||
'singleton': 'singleton:true',
|
||||
}
|
||||
|
||||
|
||||
class UserError(Exception):
|
||||
"""UI exception. Commands should throw this in order to display
|
||||
nonrecoverable errors to the user.
|
||||
"""
|
||||
|
||||
|
||||
# Utilities.
|
||||
|
||||
def _encoding():
|
||||
"""Tries to guess the encoding used by the terminal."""
|
||||
# Configured override?
|
||||
encoding = config['terminal_encoding'].get()
|
||||
if encoding:
|
||||
return encoding
|
||||
|
||||
# Determine from locale settings.
|
||||
try:
|
||||
return locale.getdefaultlocale()[1] or 'utf8'
|
||||
except ValueError:
|
||||
# Invalid locale environment variable setting. To avoid
|
||||
# failing entirely for no good reason, assume UTF-8.
|
||||
return 'utf8'
|
||||
|
||||
|
||||
def decargs(arglist):
|
||||
"""Given a list of command-line argument bytestrings, attempts to
|
||||
decode them to Unicode strings.
|
||||
"""
|
||||
return [s.decode(_encoding()) for s in arglist]
|
||||
|
||||
|
||||
def print_(*strings):
|
||||
"""Like print, but rather than raising an error when a character
|
||||
is not in the terminal's encoding's character set, just silently
|
||||
replaces it.
|
||||
"""
|
||||
if strings:
|
||||
if isinstance(strings[0], unicode):
|
||||
txt = u' '.join(strings)
|
||||
else:
|
||||
txt = ' '.join(strings)
|
||||
else:
|
||||
txt = u''
|
||||
if isinstance(txt, unicode):
|
||||
txt = txt.encode(_encoding(), 'replace')
|
||||
print(txt)
|
||||
|
||||
|
||||
def input_(prompt=None):
|
||||
"""Like `raw_input`, but decodes the result to a Unicode string.
|
||||
Raises a UserError if stdin is not available. The prompt is sent to
|
||||
stdout rather than stderr. A printed between the prompt and the
|
||||
input cursor.
|
||||
"""
|
||||
# raw_input incorrectly sends prompts to stderr, not stdout, so we
|
||||
# use print() explicitly to display prompts.
|
||||
# http://bugs.python.org/issue1927
|
||||
if prompt:
|
||||
if isinstance(prompt, unicode):
|
||||
prompt = prompt.encode(_encoding(), 'replace')
|
||||
print(prompt, end=' ')
|
||||
|
||||
try:
|
||||
resp = raw_input()
|
||||
except EOFError:
|
||||
raise UserError('stdin stream ended while input required')
|
||||
|
||||
return resp.decode(sys.stdin.encoding or 'utf8', 'ignore')
|
||||
|
||||
|
||||
def input_options(options, require=False, prompt=None, fallback_prompt=None,
|
||||
numrange=None, default=None, max_width=72):
|
||||
"""Prompts a user for input. The sequence of `options` defines the
|
||||
choices the user has. A single-letter shortcut is inferred for each
|
||||
option; the user's choice is returned as that single, lower-case
|
||||
letter. The options should be provided as lower-case strings unless
|
||||
a particular shortcut is desired; in that case, only that letter
|
||||
should be capitalized.
|
||||
|
||||
By default, the first option is the default. `default` can be provided to
|
||||
override this. If `require` is provided, then there is no default. The
|
||||
prompt and fallback prompt are also inferred but can be overridden.
|
||||
|
||||
If numrange is provided, it is a pair of `(high, low)` (both ints)
|
||||
indicating that, in addition to `options`, the user may enter an
|
||||
integer in that inclusive range.
|
||||
|
||||
`max_width` specifies the maximum number of columns in the
|
||||
automatically generated prompt string.
|
||||
"""
|
||||
# Assign single letters to each option. Also capitalize the options
|
||||
# to indicate the letter.
|
||||
letters = {}
|
||||
display_letters = []
|
||||
capitalized = []
|
||||
first = True
|
||||
for option in options:
|
||||
# Is a letter already capitalized?
|
||||
for letter in option:
|
||||
if letter.isalpha() and letter.upper() == letter:
|
||||
found_letter = letter
|
||||
break
|
||||
else:
|
||||
# Infer a letter.
|
||||
for letter in option:
|
||||
if not letter.isalpha():
|
||||
continue # Don't use punctuation.
|
||||
if letter not in letters:
|
||||
found_letter = letter
|
||||
break
|
||||
else:
|
||||
raise ValueError('no unambiguous lettering found')
|
||||
|
||||
letters[found_letter.lower()] = option
|
||||
index = option.index(found_letter)
|
||||
|
||||
# Mark the option's shortcut letter for display.
|
||||
if not require and (
|
||||
(default is None and not numrange and first) or
|
||||
(isinstance(default, basestring) and
|
||||
found_letter.lower() == default.lower())):
|
||||
# The first option is the default; mark it.
|
||||
show_letter = '[%s]' % found_letter.upper()
|
||||
is_default = True
|
||||
else:
|
||||
show_letter = found_letter.upper()
|
||||
is_default = False
|
||||
|
||||
# Colorize the letter shortcut.
|
||||
show_letter = colorize('turquoise' if is_default else 'blue',
|
||||
show_letter)
|
||||
|
||||
# Insert the highlighted letter back into the word.
|
||||
capitalized.append(
|
||||
option[:index] + show_letter + option[index + 1:]
|
||||
)
|
||||
display_letters.append(found_letter.upper())
|
||||
|
||||
first = False
|
||||
|
||||
# The default is just the first option if unspecified.
|
||||
if require:
|
||||
default = None
|
||||
elif default is None:
|
||||
if numrange:
|
||||
default = numrange[0]
|
||||
else:
|
||||
default = display_letters[0].lower()
|
||||
|
||||
# Make a prompt if one is not provided.
|
||||
if not prompt:
|
||||
prompt_parts = []
|
||||
prompt_part_lengths = []
|
||||
if numrange:
|
||||
if isinstance(default, int):
|
||||
default_name = str(default)
|
||||
default_name = colorize('turquoise', default_name)
|
||||
tmpl = '# selection (default %s)'
|
||||
prompt_parts.append(tmpl % default_name)
|
||||
prompt_part_lengths.append(len(tmpl % str(default)))
|
||||
else:
|
||||
prompt_parts.append('# selection')
|
||||
prompt_part_lengths.append(len(prompt_parts[-1]))
|
||||
prompt_parts += capitalized
|
||||
prompt_part_lengths += [len(s) for s in options]
|
||||
|
||||
# Wrap the query text.
|
||||
prompt = ''
|
||||
line_length = 0
|
||||
for i, (part, length) in enumerate(zip(prompt_parts,
|
||||
prompt_part_lengths)):
|
||||
# Add punctuation.
|
||||
if i == len(prompt_parts) - 1:
|
||||
part += '?'
|
||||
else:
|
||||
part += ','
|
||||
length += 1
|
||||
|
||||
# Choose either the current line or the beginning of the next.
|
||||
if line_length + length + 1 > max_width:
|
||||
prompt += '\n'
|
||||
line_length = 0
|
||||
|
||||
if line_length != 0:
|
||||
# Not the beginning of the line; need a space.
|
||||
part = ' ' + part
|
||||
length += 1
|
||||
|
||||
prompt += part
|
||||
line_length += length
|
||||
|
||||
# Make a fallback prompt too. This is displayed if the user enters
|
||||
# something that is not recognized.
|
||||
if not fallback_prompt:
|
||||
fallback_prompt = 'Enter one of '
|
||||
if numrange:
|
||||
fallback_prompt += '%i-%i, ' % numrange
|
||||
fallback_prompt += ', '.join(display_letters) + ':'
|
||||
|
||||
resp = input_(prompt)
|
||||
while True:
|
||||
resp = resp.strip().lower()
|
||||
|
||||
# Try default option.
|
||||
if default is not None and not resp:
|
||||
resp = default
|
||||
|
||||
# Try an integer input if available.
|
||||
if numrange:
|
||||
try:
|
||||
resp = int(resp)
|
||||
except ValueError:
|
||||
pass
|
||||
else:
|
||||
low, high = numrange
|
||||
if low <= resp <= high:
|
||||
return resp
|
||||
else:
|
||||
resp = None
|
||||
|
||||
# Try a normal letter input.
|
||||
if resp:
|
||||
resp = resp[0]
|
||||
if resp in letters:
|
||||
return resp
|
||||
|
||||
# Prompt for new input.
|
||||
resp = input_(fallback_prompt)
|
||||
|
||||
|
||||
def input_yn(prompt, require=False):
|
||||
"""Prompts the user for a "yes" or "no" response. The default is
|
||||
"yes" unless `require` is `True`, in which case there is no default.
|
||||
"""
|
||||
sel = input_options(
|
||||
('y', 'n'), require, prompt, 'Enter Y or N:'
|
||||
)
|
||||
return sel == 'y'
|
||||
|
||||
|
||||
def human_bytes(size):
|
||||
"""Formats size, a number of bytes, in a human-readable way."""
|
||||
suffices = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB', 'HB']
|
||||
for suffix in suffices:
|
||||
if size < 1024:
|
||||
return "%3.1f %s" % (size, suffix)
|
||||
size /= 1024.0
|
||||
return "big"
|
||||
|
||||
|
||||
def human_seconds(interval):
|
||||
"""Formats interval, a number of seconds, as a human-readable time
|
||||
interval using English words.
|
||||
"""
|
||||
units = [
|
||||
(1, 'second'),
|
||||
(60, 'minute'),
|
||||
(60, 'hour'),
|
||||
(24, 'day'),
|
||||
(7, 'week'),
|
||||
(52, 'year'),
|
||||
(10, 'decade'),
|
||||
]
|
||||
for i in range(len(units) - 1):
|
||||
increment, suffix = units[i]
|
||||
next_increment, _ = units[i + 1]
|
||||
interval /= float(increment)
|
||||
if interval < next_increment:
|
||||
break
|
||||
else:
|
||||
# Last unit.
|
||||
increment, suffix = units[-1]
|
||||
interval /= float(increment)
|
||||
|
||||
return "%3.1f %ss" % (interval, suffix)
|
||||
|
||||
|
||||
def human_seconds_short(interval):
|
||||
"""Formats a number of seconds as a short human-readable M:SS
|
||||
string.
|
||||
"""
|
||||
interval = int(interval)
|
||||
return u'%i:%02i' % (interval // 60, interval % 60)
|
||||
|
||||
|
||||
# ANSI terminal colorization code heavily inspired by pygments:
|
||||
# http://dev.pocoo.org/hg/pygments-main/file/b2deea5b5030/pygments/console.py
|
||||
# (pygments is by Tim Hatch, Armin Ronacher, et al.)
|
||||
COLOR_ESCAPE = "\x1b["
|
||||
DARK_COLORS = ["black", "darkred", "darkgreen", "brown", "darkblue",
|
||||
"purple", "teal", "lightgray"]
|
||||
LIGHT_COLORS = ["darkgray", "red", "green", "yellow", "blue",
|
||||
"fuchsia", "turquoise", "white"]
|
||||
RESET_COLOR = COLOR_ESCAPE + "39;49;00m"
|
||||
|
||||
|
||||
def _colorize(color, text):
|
||||
"""Returns a string that prints the given text in the given color
|
||||
in a terminal that is ANSI color-aware. The color must be something
|
||||
in DARK_COLORS or LIGHT_COLORS.
|
||||
"""
|
||||
if color in DARK_COLORS:
|
||||
escape = COLOR_ESCAPE + "%im" % (DARK_COLORS.index(color) + 30)
|
||||
elif color in LIGHT_COLORS:
|
||||
escape = COLOR_ESCAPE + "%i;01m" % (LIGHT_COLORS.index(color) + 30)
|
||||
else:
|
||||
raise ValueError('no such color %s', color)
|
||||
return escape + text + RESET_COLOR
|
||||
|
||||
|
||||
def colorize(color, text):
|
||||
"""Colorize text if colored output is enabled. (Like _colorize but
|
||||
conditional.)
|
||||
"""
|
||||
if config['color']:
|
||||
return _colorize(color, text)
|
||||
else:
|
||||
return text
|
||||
|
||||
|
||||
def _colordiff(a, b, highlight='red', minor_highlight='lightgray'):
|
||||
"""Given two values, return the same pair of strings except with
|
||||
their differences highlighted in the specified color. Strings are
|
||||
highlighted intelligently to show differences; other values are
|
||||
stringified and highlighted in their entirety.
|
||||
"""
|
||||
if not isinstance(a, basestring) or not isinstance(b, basestring):
|
||||
# Non-strings: use ordinary equality.
|
||||
a = unicode(a)
|
||||
b = unicode(b)
|
||||
if a == b:
|
||||
return a, b
|
||||
else:
|
||||
return colorize(highlight, a), colorize(highlight, b)
|
||||
|
||||
if isinstance(a, bytes) or isinstance(b, bytes):
|
||||
# A path field.
|
||||
a = util.displayable_path(a)
|
||||
b = util.displayable_path(b)
|
||||
|
||||
a_out = []
|
||||
b_out = []
|
||||
|
||||
matcher = SequenceMatcher(lambda x: False, a, b)
|
||||
for op, a_start, a_end, b_start, b_end in matcher.get_opcodes():
|
||||
if op == 'equal':
|
||||
# In both strings.
|
||||
a_out.append(a[a_start:a_end])
|
||||
b_out.append(b[b_start:b_end])
|
||||
elif op == 'insert':
|
||||
# Right only.
|
||||
b_out.append(colorize(highlight, b[b_start:b_end]))
|
||||
elif op == 'delete':
|
||||
# Left only.
|
||||
a_out.append(colorize(highlight, a[a_start:a_end]))
|
||||
elif op == 'replace':
|
||||
# Right and left differ. Colorise with second highlight if
|
||||
# it's just a case change.
|
||||
if a[a_start:a_end].lower() != b[b_start:b_end].lower():
|
||||
color = highlight
|
||||
else:
|
||||
color = minor_highlight
|
||||
a_out.append(colorize(color, a[a_start:a_end]))
|
||||
b_out.append(colorize(color, b[b_start:b_end]))
|
||||
else:
|
||||
assert(False)
|
||||
|
||||
return u''.join(a_out), u''.join(b_out)
|
||||
|
||||
|
||||
def colordiff(a, b, highlight='red'):
|
||||
"""Colorize differences between two values if color is enabled.
|
||||
(Like _colordiff but conditional.)
|
||||
"""
|
||||
if config['color']:
|
||||
return _colordiff(a, b, highlight)
|
||||
else:
|
||||
return unicode(a), unicode(b)
|
||||
|
||||
|
||||
def get_path_formats(subview=None):
|
||||
"""Get the configuration's path formats as a list of query/template
|
||||
pairs.
|
||||
"""
|
||||
path_formats = []
|
||||
subview = subview or config['paths']
|
||||
for query, view in subview.items():
|
||||
query = PF_KEY_QUERIES.get(query, query) # Expand common queries.
|
||||
path_formats.append((query, Template(view.get(unicode))))
|
||||
return path_formats
|
||||
|
||||
|
||||
def get_replacements():
|
||||
"""Confit validation function that reads regex/string pairs.
|
||||
"""
|
||||
replacements = []
|
||||
for pattern, repl in config['replace'].get(dict).items():
|
||||
repl = repl or ''
|
||||
try:
|
||||
replacements.append((re.compile(pattern), repl))
|
||||
except re.error:
|
||||
raise UserError(
|
||||
u'malformed regular expression in replace: {0}'.format(
|
||||
pattern
|
||||
)
|
||||
)
|
||||
return replacements
|
||||
|
||||
|
||||
def _pick_format(album, fmt=None):
|
||||
"""Pick a format string for printing Album or Item objects,
|
||||
falling back to config options and defaults.
|
||||
"""
|
||||
if fmt:
|
||||
return fmt
|
||||
if album:
|
||||
return config['list_format_album'].get(unicode)
|
||||
else:
|
||||
return config['list_format_item'].get(unicode)
|
||||
|
||||
|
||||
def print_obj(obj, lib, fmt=None):
|
||||
"""Print an Album or Item object. If `fmt` is specified, use that
|
||||
format string. Otherwise, use the configured template.
|
||||
"""
|
||||
album = isinstance(obj, library.Album)
|
||||
fmt = _pick_format(album, fmt)
|
||||
if isinstance(fmt, Template):
|
||||
template = fmt
|
||||
else:
|
||||
template = Template(fmt)
|
||||
print_(obj.evaluate_template(template))
|
||||
|
||||
|
||||
def term_width():
|
||||
"""Get the width (columns) of the terminal."""
|
||||
fallback = config['ui']['terminal_width'].get(int)
|
||||
|
||||
# The fcntl and termios modules are not available on non-Unix
|
||||
# platforms, so we fall back to a constant.
|
||||
try:
|
||||
import fcntl
|
||||
import termios
|
||||
except ImportError:
|
||||
return fallback
|
||||
|
||||
try:
|
||||
buf = fcntl.ioctl(0, termios.TIOCGWINSZ, ' ' * 4)
|
||||
except IOError:
|
||||
return fallback
|
||||
try:
|
||||
height, width = struct.unpack('hh', buf)
|
||||
except struct.error:
|
||||
return fallback
|
||||
return width
|
||||
|
||||
|
||||
FLOAT_EPSILON = 0.01
|
||||
|
||||
|
||||
def _field_diff(field, old, new):
|
||||
"""Given two Model objects, format their values for `field` and
|
||||
highlight changes among them. Return a human-readable string. If the
|
||||
value has not changed, return None instead.
|
||||
"""
|
||||
oldval = old.get(field)
|
||||
newval = new.get(field)
|
||||
|
||||
# If no change, abort.
|
||||
if isinstance(oldval, float) and isinstance(newval, float) and \
|
||||
abs(oldval - newval) < FLOAT_EPSILON:
|
||||
return None
|
||||
elif oldval == newval:
|
||||
return None
|
||||
|
||||
# Get formatted values for output.
|
||||
oldstr = old.formatted().get(field, u'')
|
||||
newstr = new.formatted().get(field, u'')
|
||||
|
||||
# For strings, highlight changes. For others, colorize the whole
|
||||
# thing.
|
||||
if isinstance(oldval, basestring):
|
||||
oldstr, newstr = colordiff(oldval, newstr)
|
||||
else:
|
||||
oldstr, newstr = colorize('red', oldstr), colorize('red', newstr)
|
||||
|
||||
return u'{0} -> {1}'.format(oldstr, newstr)
|
||||
|
||||
|
||||
def show_model_changes(new, old=None, fields=None, always=False):
|
||||
"""Given a Model object, print a list of changes from its pristine
|
||||
version stored in the database. Return a boolean indicating whether
|
||||
any changes were found.
|
||||
|
||||
`old` may be the "original" object to avoid using the pristine
|
||||
version from the database. `fields` may be a list of fields to
|
||||
restrict the detection to. `always` indicates whether the object is
|
||||
always identified, regardless of whether any changes are present.
|
||||
"""
|
||||
old = old or new._db._get(type(new), new.id)
|
||||
|
||||
# Build up lines showing changed fields.
|
||||
changes = []
|
||||
for field in old:
|
||||
# Subset of the fields. Never show mtime.
|
||||
if field == 'mtime' or (fields and field not in fields):
|
||||
continue
|
||||
|
||||
# Detect and show difference for this field.
|
||||
line = _field_diff(field, old, new)
|
||||
if line:
|
||||
changes.append(u' {0}: {1}'.format(field, line))
|
||||
|
||||
# New fields.
|
||||
for field in set(new) - set(old):
|
||||
if fields and field not in fields:
|
||||
continue
|
||||
|
||||
changes.append(u' {0}: {1}'.format(
|
||||
field,
|
||||
colorize('red', new.formatted()[field])
|
||||
))
|
||||
|
||||
# Print changes.
|
||||
if changes or always:
|
||||
print_obj(old, old._db)
|
||||
if changes:
|
||||
print_(u'\n'.join(changes))
|
||||
|
||||
return bool(changes)
|
||||
|
||||
|
||||
# Subcommand parsing infrastructure.
|
||||
#
|
||||
# This is a fairly generic subcommand parser for optparse. It is
|
||||
# maintained externally here:
|
||||
# http://gist.github.com/462717
|
||||
# There you will also find a better description of the code and a more
|
||||
# succinct example program.
|
||||
|
||||
class Subcommand(object):
|
||||
"""A subcommand of a root command-line application that may be
|
||||
invoked by a SubcommandOptionParser.
|
||||
"""
|
||||
def __init__(self, name, parser=None, help='', aliases=(), hide=False):
|
||||
"""Creates a new subcommand. name is the primary way to invoke
|
||||
the subcommand; aliases are alternate names. parser is an
|
||||
OptionParser responsible for parsing the subcommand's options.
|
||||
help is a short description of the command. If no parser is
|
||||
given, it defaults to a new, empty OptionParser.
|
||||
"""
|
||||
self.name = name
|
||||
self.parser = parser or optparse.OptionParser()
|
||||
self.aliases = aliases
|
||||
self.help = help
|
||||
self.hide = hide
|
||||
self._root_parser = None
|
||||
|
||||
def print_help(self):
|
||||
self.parser.print_help()
|
||||
|
||||
def parse_args(self, args):
|
||||
return self.parser.parse_args(args)
|
||||
|
||||
@property
|
||||
def root_parser(self):
|
||||
return self._root_parser
|
||||
|
||||
@root_parser.setter
|
||||
def root_parser(self, root_parser):
|
||||
self._root_parser = root_parser
|
||||
self.parser.prog = '{0} {1}'.format(root_parser.get_prog_name(),
|
||||
self.name)
|
||||
|
||||
|
||||
class SubcommandsOptionParser(optparse.OptionParser):
|
||||
"""A variant of OptionParser that parses subcommands and their
|
||||
arguments.
|
||||
"""
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
"""Create a new subcommand-aware option parser. All of the
|
||||
options to OptionParser.__init__ are supported in addition
|
||||
to subcommands, a sequence of Subcommand objects.
|
||||
"""
|
||||
# A more helpful default usage.
|
||||
if 'usage' not in kwargs:
|
||||
kwargs['usage'] = """
|
||||
%prog COMMAND [ARGS...]
|
||||
%prog help COMMAND"""
|
||||
kwargs['add_help_option'] = False
|
||||
|
||||
# Super constructor.
|
||||
optparse.OptionParser.__init__(self, *args, **kwargs)
|
||||
|
||||
# Our root parser needs to stop on the first unrecognized argument.
|
||||
self.disable_interspersed_args()
|
||||
|
||||
self.subcommands = []
|
||||
|
||||
def add_subcommand(self, *cmds):
|
||||
"""Adds a Subcommand object to the parser's list of commands.
|
||||
"""
|
||||
for cmd in cmds:
|
||||
cmd.root_parser = self
|
||||
self.subcommands.append(cmd)
|
||||
|
||||
# Add the list of subcommands to the help message.
|
||||
def format_help(self, formatter=None):
|
||||
# Get the original help message, to which we will append.
|
||||
out = optparse.OptionParser.format_help(self, formatter)
|
||||
if formatter is None:
|
||||
formatter = self.formatter
|
||||
|
||||
# Subcommands header.
|
||||
result = ["\n"]
|
||||
result.append(formatter.format_heading('Commands'))
|
||||
formatter.indent()
|
||||
|
||||
# Generate the display names (including aliases).
|
||||
# Also determine the help position.
|
||||
disp_names = []
|
||||
help_position = 0
|
||||
subcommands = [c for c in self.subcommands if not c.hide]
|
||||
subcommands.sort(key=lambda c: c.name)
|
||||
for subcommand in subcommands:
|
||||
name = subcommand.name
|
||||
if subcommand.aliases:
|
||||
name += ' (%s)' % ', '.join(subcommand.aliases)
|
||||
disp_names.append(name)
|
||||
|
||||
# Set the help position based on the max width.
|
||||
proposed_help_position = len(name) + formatter.current_indent + 2
|
||||
if proposed_help_position <= formatter.max_help_position:
|
||||
help_position = max(help_position, proposed_help_position)
|
||||
|
||||
# Add each subcommand to the output.
|
||||
for subcommand, name in zip(subcommands, disp_names):
|
||||
# Lifted directly from optparse.py.
|
||||
name_width = help_position - formatter.current_indent - 2
|
||||
if len(name) > name_width:
|
||||
name = "%*s%s\n" % (formatter.current_indent, "", name)
|
||||
indent_first = help_position
|
||||
else:
|
||||
name = "%*s%-*s " % (formatter.current_indent, "",
|
||||
name_width, name)
|
||||
indent_first = 0
|
||||
result.append(name)
|
||||
help_width = formatter.width - help_position
|
||||
help_lines = textwrap.wrap(subcommand.help, help_width)
|
||||
result.append("%*s%s\n" % (indent_first, "", help_lines[0]))
|
||||
result.extend(["%*s%s\n" % (help_position, "", line)
|
||||
for line in help_lines[1:]])
|
||||
formatter.dedent()
|
||||
|
||||
# Concatenate the original help message with the subcommand
|
||||
# list.
|
||||
return out + "".join(result)
|
||||
|
||||
def _subcommand_for_name(self, name):
|
||||
"""Return the subcommand in self.subcommands matching the
|
||||
given name. The name may either be the name of a subcommand or
|
||||
an alias. If no subcommand matches, returns None.
|
||||
"""
|
||||
for subcommand in self.subcommands:
|
||||
if name == subcommand.name or \
|
||||
name in subcommand.aliases:
|
||||
return subcommand
|
||||
return None
|
||||
|
||||
def parse_global_options(self, args):
|
||||
"""Parse options up to the subcommand argument. Returns a tuple
|
||||
of the options object and the remaining arguments.
|
||||
"""
|
||||
options, subargs = self.parse_args(args)
|
||||
|
||||
# Force the help command
|
||||
if options.help:
|
||||
subargs = ['help']
|
||||
elif options.version:
|
||||
subargs = ['version']
|
||||
return options, subargs
|
||||
|
||||
def parse_subcommand(self, args):
|
||||
"""Given the `args` left unused by a `parse_global_options`,
|
||||
return the invoked subcommand, the subcommand options, and the
|
||||
subcommand arguments.
|
||||
"""
|
||||
# Help is default command
|
||||
if not args:
|
||||
args = ['help']
|
||||
|
||||
cmdname = args.pop(0)
|
||||
subcommand = self._subcommand_for_name(cmdname)
|
||||
if not subcommand:
|
||||
raise UserError("unknown command '{0}'".format(cmdname))
|
||||
|
||||
suboptions, subargs = subcommand.parse_args(args)
|
||||
return subcommand, suboptions, subargs
|
||||
|
||||
|
||||
optparse.Option.ALWAYS_TYPED_ACTIONS += ('callback',)
|
||||
|
||||
|
||||
def vararg_callback(option, opt_str, value, parser):
|
||||
"""Callback for an option with variable arguments.
|
||||
Manually collect arguments right of a callback-action
|
||||
option (ie. with action="callback"), and add the resulting
|
||||
list to the destination var.
|
||||
|
||||
Usage:
|
||||
parser.add_option("-c", "--callback", dest="vararg_attr",
|
||||
action="callback", callback=vararg_callback)
|
||||
|
||||
Details:
|
||||
http://docs.python.org/2/library/optparse.html#callback-example-6-variable
|
||||
-arguments
|
||||
"""
|
||||
value = [value]
|
||||
|
||||
def floatable(str):
|
||||
try:
|
||||
float(str)
|
||||
return True
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
for arg in parser.rargs:
|
||||
# stop on --foo like options
|
||||
if arg[:2] == "--" and len(arg) > 2:
|
||||
break
|
||||
# stop on -a, but not on -3 or -3.0
|
||||
if arg[:1] == "-" and len(arg) > 1 and not floatable(arg):
|
||||
break
|
||||
value.append(arg)
|
||||
|
||||
del parser.rargs[:len(value) - 1]
|
||||
setattr(parser.values, option.dest, value)
|
||||
|
||||
|
||||
# The main entry point and bootstrapping.
|
||||
|
||||
def _load_plugins(config):
|
||||
"""Load the plugins specified in the configuration.
|
||||
"""
|
||||
paths = config['pluginpath'].get(confit.StrSeq(split=False))
|
||||
paths = map(util.normpath, paths)
|
||||
|
||||
import beetsplug
|
||||
beetsplug.__path__ = paths + beetsplug.__path__
|
||||
# For backwards compatibility.
|
||||
sys.path += paths
|
||||
|
||||
plugins.load_plugins(config['plugins'].as_str_seq())
|
||||
plugins.send("pluginload")
|
||||
return plugins
|
||||
|
||||
|
||||
def _setup(options, lib=None):
|
||||
"""Prepare and global state and updates it with command line options.
|
||||
|
||||
Returns a list of subcommands, a list of plugins, and a library instance.
|
||||
"""
|
||||
# Configure the MusicBrainz API.
|
||||
mb.configure()
|
||||
|
||||
config = _configure(options)
|
||||
|
||||
plugins = _load_plugins(config)
|
||||
|
||||
# Get the default subcommands.
|
||||
from beets.ui.commands import default_commands
|
||||
|
||||
subcommands = list(default_commands)
|
||||
subcommands.extend(plugins.commands())
|
||||
|
||||
if lib is None:
|
||||
lib = _open_library(config)
|
||||
plugins.send("library_opened", lib=lib)
|
||||
library.Item._types = plugins.types(library.Item)
|
||||
library.Album._types = plugins.types(library.Album)
|
||||
|
||||
return subcommands, plugins, lib
|
||||
|
||||
|
||||
def _configure(options):
|
||||
"""Amend the global configuration object with command line options.
|
||||
"""
|
||||
# Add any additional config files specified with --config. This
|
||||
# special handling lets specified plugins get loaded before we
|
||||
# finish parsing the command line.
|
||||
if getattr(options, 'config', None) is not None:
|
||||
config_path = options.config
|
||||
del options.config
|
||||
config.set_file(config_path)
|
||||
config.set_args(options)
|
||||
|
||||
# Configure the logger.
|
||||
if config['verbose'].get(bool):
|
||||
log.setLevel(logging.DEBUG)
|
||||
else:
|
||||
log.setLevel(logging.INFO)
|
||||
|
||||
config_path = config.user_config_path()
|
||||
if os.path.isfile(config_path):
|
||||
log.debug(u'user configuration: {0}'.format(
|
||||
util.displayable_path(config_path)))
|
||||
else:
|
||||
log.debug(u'no user configuration found at {0}'.format(
|
||||
util.displayable_path(config_path)))
|
||||
|
||||
log.debug(u'data directory: {0}'
|
||||
.format(util.displayable_path(config.config_dir())))
|
||||
return config
|
||||
|
||||
|
||||
def _open_library(config):
|
||||
"""Create a new library instance from the configuration.
|
||||
"""
|
||||
dbpath = config['library'].as_filename()
|
||||
try:
|
||||
lib = library.Library(
|
||||
dbpath,
|
||||
config['directory'].as_filename(),
|
||||
get_path_formats(),
|
||||
get_replacements(),
|
||||
)
|
||||
lib.get_item(0) # Test database connection.
|
||||
except (sqlite3.OperationalError, sqlite3.DatabaseError):
|
||||
log.debug(traceback.format_exc())
|
||||
raise UserError(u"database file {0} could not be opened".format(
|
||||
util.displayable_path(dbpath)
|
||||
))
|
||||
log.debug(u'library database: {0}\n'
|
||||
u'library directory: {1}'
|
||||
.format(util.displayable_path(lib.path),
|
||||
util.displayable_path(lib.directory)))
|
||||
return lib
|
||||
|
||||
|
||||
def _raw_main(args, lib=None):
|
||||
"""A helper function for `main` without top-level exception
|
||||
handling.
|
||||
"""
|
||||
parser = SubcommandsOptionParser()
|
||||
parser.add_option('-l', '--library', dest='library',
|
||||
help='library database file to use')
|
||||
parser.add_option('-d', '--directory', dest='directory',
|
||||
help="destination music directory")
|
||||
parser.add_option('-v', '--verbose', dest='verbose', action='store_true',
|
||||
help='print debugging information')
|
||||
parser.add_option('-c', '--config', dest='config',
|
||||
help='path to configuration file')
|
||||
parser.add_option('-h', '--help', dest='help', action='store_true',
|
||||
help='how this help message and exit')
|
||||
parser.add_option('--version', dest='version', action='store_true',
|
||||
help=optparse.SUPPRESS_HELP)
|
||||
|
||||
options, subargs = parser.parse_global_options(args)
|
||||
|
||||
# Special case for the `config --edit` command: bypass _setup so
|
||||
# that an invalid configuration does not prevent the editor from
|
||||
# starting.
|
||||
if subargs[0] == 'config' and ('-e' in subargs or '--edit' in subargs):
|
||||
from beets.ui.commands import config_edit
|
||||
return config_edit()
|
||||
|
||||
subcommands, plugins, lib = _setup(options, lib)
|
||||
parser.add_subcommand(*subcommands)
|
||||
|
||||
subcommand, suboptions, subargs = parser.parse_subcommand(subargs)
|
||||
subcommand.func(lib, suboptions, subargs)
|
||||
|
||||
plugins.send('cli_exit', lib=lib)
|
||||
|
||||
|
||||
def main(args=None):
|
||||
"""Run the main command-line interface for beets. Includes top-level
|
||||
exception handlers that print friendly error messages.
|
||||
"""
|
||||
try:
|
||||
_raw_main(args)
|
||||
except UserError as exc:
|
||||
message = exc.args[0] if exc.args else None
|
||||
log.error(u'error: {0}'.format(message))
|
||||
sys.exit(1)
|
||||
except util.HumanReadableException as exc:
|
||||
exc.log(log)
|
||||
sys.exit(1)
|
||||
except library.FileOperationError as exc:
|
||||
# These errors have reasonable human-readable descriptions, but
|
||||
# we still want to log their tracebacks for debugging.
|
||||
log.debug(traceback.format_exc())
|
||||
log.error(exc)
|
||||
sys.exit(1)
|
||||
except confit.ConfigError as exc:
|
||||
log.error(u'configuration error: {0}'.format(exc))
|
||||
sys.exit(1)
|
||||
except IOError as exc:
|
||||
if exc.errno == errno.EPIPE:
|
||||
# "Broken pipe". End silently.
|
||||
pass
|
||||
else:
|
||||
raise
|
||||
except KeyboardInterrupt:
|
||||
# Silently ignore ^C except in verbose mode.
|
||||
log.debug(traceback.format_exc())
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,162 @@
|
||||
# This file is part of beets.
|
||||
# Copyright (c) 2014, Thomas Scholtes.
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining
|
||||
# a copy of this software and associated documentation files (the
|
||||
# "Software"), to deal in the Software without restriction, including
|
||||
# without limitation the rights to use, copy, modify, merge, publish,
|
||||
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||
# permit persons to whom the Software is furnished to do so, subject to
|
||||
# the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
|
||||
|
||||
|
||||
# Completion for the `beet` command
|
||||
# =================================
|
||||
#
|
||||
# Load this script to complete beets subcommands, options, and
|
||||
# queries.
|
||||
#
|
||||
# If a beets command is found on the command line it completes filenames and
|
||||
# the subcommand's options. Otherwise it will complete global options and
|
||||
# subcommands. If the previous option on the command line expects an argument,
|
||||
# it also completes filenames or directories. Options are only
|
||||
# completed if '-' has already been typed on the command line.
|
||||
#
|
||||
# Note that completion of plugin commands only works for those plugins
|
||||
# that were enabled when running `beet completion`. It does not check
|
||||
# plugins dynamically
|
||||
#
|
||||
# Currently, only Bash 3.2 and newer is supported and the
|
||||
# `bash-completion` package is requied.
|
||||
#
|
||||
# TODO
|
||||
# ----
|
||||
#
|
||||
# * There are some issues with arguments that are quoted on the command line.
|
||||
#
|
||||
# * Complete arguments for the `--format` option by expanding field variables.
|
||||
#
|
||||
# beet ls -f "$tit[TAB]
|
||||
# beet ls -f "$title
|
||||
#
|
||||
# * Support long options with `=`, e.g. `--config=file`. Debian's bash
|
||||
# completion package can handle this.
|
||||
#
|
||||
|
||||
|
||||
# Determines the beets subcommand and dispatches the completion
|
||||
# accordingly.
|
||||
_beet_dispatch() {
|
||||
local cur prev cmd=
|
||||
|
||||
COMPREPLY=()
|
||||
_get_comp_words_by_ref -n : cur prev
|
||||
|
||||
# Look for the beets subcommand
|
||||
local arg
|
||||
for (( i=1; i < COMP_CWORD; i++ )); do
|
||||
arg="${COMP_WORDS[i]}"
|
||||
if _list_include_item "${opts___global}" $arg; then
|
||||
((i++))
|
||||
elif [[ "$arg" != -* ]]; then
|
||||
cmd="$arg"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
# Replace command shortcuts
|
||||
if [[ -n $cmd ]] && _list_include_item "$aliases" "$cmd"; then
|
||||
eval "cmd=\$alias__$cmd"
|
||||
fi
|
||||
|
||||
case $cmd in
|
||||
help)
|
||||
COMPREPLY+=( $(compgen -W "$commands" -- $cur) )
|
||||
;;
|
||||
list|remove|move|update|write|stats)
|
||||
_beet_complete_query
|
||||
;;
|
||||
"")
|
||||
_beet_complete_global
|
||||
;;
|
||||
*)
|
||||
_beet_complete
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
|
||||
# Adds option and file completion to COMPREPLY for the subcommand $cmd
|
||||
_beet_complete() {
|
||||
if [[ $cur == -* ]]; then
|
||||
local opts flags completions
|
||||
eval "opts=\$opts__$cmd"
|
||||
eval "flags=\$flags__$cmd"
|
||||
completions="${flags___common} ${opts} ${flags}"
|
||||
COMPREPLY+=( $(compgen -W "$completions" -- $cur) )
|
||||
else
|
||||
_filedir
|
||||
fi
|
||||
}
|
||||
|
||||
|
||||
# Add global options and subcommands to the completion
|
||||
_beet_complete_global() {
|
||||
case $prev in
|
||||
-h|--help)
|
||||
# Complete commands
|
||||
COMPREPLY+=( $(compgen -W "$commands" -- $cur) )
|
||||
return
|
||||
;;
|
||||
-l|--library|-c|--config)
|
||||
# Filename completion
|
||||
_filedir
|
||||
return
|
||||
;;
|
||||
-d|--directory)
|
||||
# Directory completion
|
||||
_filedir -d
|
||||
return
|
||||
;;
|
||||
esac
|
||||
|
||||
if [[ $cur == -* ]]; then
|
||||
local completions="$opts___global $flags___global"
|
||||
COMPREPLY+=( $(compgen -W "$completions" -- $cur) )
|
||||
elif [[ -n $cur ]] && _list_include_item "$aliases" "$cur"; then
|
||||
local cmd
|
||||
eval "cmd=\$alias__$cur"
|
||||
COMPREPLY+=( "$cmd" )
|
||||
else
|
||||
COMPREPLY+=( $(compgen -W "$commands" -- $cur) )
|
||||
fi
|
||||
}
|
||||
|
||||
_beet_complete_query() {
|
||||
local opts
|
||||
eval "opts=\$opts__$cmd"
|
||||
|
||||
if [[ $cur == -* ]] || _list_include_item "$opts" "$prev"; then
|
||||
_beet_complete
|
||||
elif [[ $cur != \'* && $cur != \"* &&
|
||||
$cur != *:* ]]; then
|
||||
# Do not complete quoted queries or those who already have a field
|
||||
# set.
|
||||
compopt -o nospace
|
||||
COMPREPLY+=( $(compgen -S : -W "$fields" -- $cur) )
|
||||
return 0
|
||||
fi
|
||||
}
|
||||
|
||||
# Returns true if the space separated list $1 includes $2
|
||||
_list_include_item() {
|
||||
[[ " $1 " == *[[:space:]]$2[[:space:]]* ]]
|
||||
}
|
||||
|
||||
# This is where beets dynamically adds the _beet function. This
|
||||
# function sets the variables $flags, $opts, $commands, and $aliases.
|
||||
complete -o filenames -F _beet beet
|
||||
@@ -0,0 +1,686 @@
|
||||
# This file is part of beets.
|
||||
# Copyright 2013, Adrian Sampson.
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining
|
||||
# a copy of this software and associated documentation files (the
|
||||
# "Software"), to deal in the Software without restriction, including
|
||||
# without limitation the rights to use, copy, modify, merge, publish,
|
||||
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||
# permit persons to whom the Software is furnished to do so, subject to
|
||||
# the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
|
||||
"""Miscellaneous utility functions."""
|
||||
from __future__ import division
|
||||
|
||||
import os
|
||||
import sys
|
||||
import re
|
||||
import shutil
|
||||
import fnmatch
|
||||
from collections import defaultdict
|
||||
import traceback
|
||||
import subprocess
|
||||
import platform
|
||||
|
||||
|
||||
MAX_FILENAME_LENGTH = 200
|
||||
WINDOWS_MAGIC_PREFIX = u'\\\\?\\'
|
||||
|
||||
|
||||
class HumanReadableException(Exception):
|
||||
"""An Exception that can include a human-readable error message to
|
||||
be logged without a traceback. Can preserve a traceback for
|
||||
debugging purposes as well.
|
||||
|
||||
Has at least two fields: `reason`, the underlying exception or a
|
||||
string describing the problem; and `verb`, the action being
|
||||
performed during the error.
|
||||
|
||||
If `tb` is provided, it is a string containing a traceback for the
|
||||
associated exception. (Note that this is not necessary in Python 3.x
|
||||
and should be removed when we make the transition.)
|
||||
"""
|
||||
error_kind = 'Error' # Human-readable description of error type.
|
||||
|
||||
def __init__(self, reason, verb, tb=None):
|
||||
self.reason = reason
|
||||
self.verb = verb
|
||||
self.tb = tb
|
||||
super(HumanReadableException, self).__init__(self.get_message())
|
||||
|
||||
def _gerund(self):
|
||||
"""Generate a (likely) gerund form of the English verb.
|
||||
"""
|
||||
if ' ' in self.verb:
|
||||
return self.verb
|
||||
gerund = self.verb[:-1] if self.verb.endswith('e') else self.verb
|
||||
gerund += 'ing'
|
||||
return gerund
|
||||
|
||||
def _reasonstr(self):
|
||||
"""Get the reason as a string."""
|
||||
if isinstance(self.reason, unicode):
|
||||
return self.reason
|
||||
elif isinstance(self.reason, basestring): # Byte string.
|
||||
return self.reason.decode('utf8', 'ignore')
|
||||
elif hasattr(self.reason, 'strerror'): # i.e., EnvironmentError
|
||||
return self.reason.strerror
|
||||
else:
|
||||
return u'"{0}"'.format(unicode(self.reason))
|
||||
|
||||
def get_message(self):
|
||||
"""Create the human-readable description of the error, sans
|
||||
introduction.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
def log(self, logger):
|
||||
"""Log to the provided `logger` a human-readable message as an
|
||||
error and a verbose traceback as a debug message.
|
||||
"""
|
||||
if self.tb:
|
||||
logger.debug(self.tb)
|
||||
logger.error(u'{0}: {1}'.format(self.error_kind, self.args[0]))
|
||||
|
||||
|
||||
class FilesystemError(HumanReadableException):
|
||||
"""An error that occurred while performing a filesystem manipulation
|
||||
via a function in this module. The `paths` field is a sequence of
|
||||
pathnames involved in the operation.
|
||||
"""
|
||||
def __init__(self, reason, verb, paths, tb=None):
|
||||
self.paths = paths
|
||||
super(FilesystemError, self).__init__(reason, verb, tb)
|
||||
|
||||
def get_message(self):
|
||||
# Use a nicer English phrasing for some specific verbs.
|
||||
if self.verb in ('move', 'copy', 'rename'):
|
||||
clause = u'while {0} {1} to {2}'.format(
|
||||
self._gerund(),
|
||||
displayable_path(self.paths[0]),
|
||||
displayable_path(self.paths[1])
|
||||
)
|
||||
elif self.verb in ('delete', 'write', 'create', 'read'):
|
||||
clause = u'while {0} {1}'.format(
|
||||
self._gerund(),
|
||||
displayable_path(self.paths[0])
|
||||
)
|
||||
else:
|
||||
clause = u'during {0} of paths {1}'.format(
|
||||
self.verb, u', '.join(displayable_path(p) for p in self.paths)
|
||||
)
|
||||
|
||||
return u'{0} {1}'.format(self._reasonstr(), clause)
|
||||
|
||||
|
||||
def normpath(path):
|
||||
"""Provide the canonical form of the path suitable for storing in
|
||||
the database.
|
||||
"""
|
||||
path = syspath(path, prefix=False)
|
||||
path = os.path.normpath(os.path.abspath(os.path.expanduser(path)))
|
||||
return bytestring_path(path)
|
||||
|
||||
|
||||
def ancestry(path):
|
||||
"""Return a list consisting of path's parent directory, its
|
||||
grandparent, and so on. For instance:
|
||||
|
||||
>>> ancestry('/a/b/c')
|
||||
['/', '/a', '/a/b']
|
||||
|
||||
The argument should *not* be the result of a call to `syspath`.
|
||||
"""
|
||||
out = []
|
||||
last_path = None
|
||||
while path:
|
||||
path = os.path.dirname(path)
|
||||
|
||||
if path == last_path:
|
||||
break
|
||||
last_path = path
|
||||
|
||||
if path:
|
||||
# don't yield ''
|
||||
out.insert(0, path)
|
||||
return out
|
||||
|
||||
|
||||
def sorted_walk(path, ignore=(), logger=None):
|
||||
"""Like `os.walk`, but yields things in case-insensitive sorted,
|
||||
breadth-first order. Directory and file names matching any glob
|
||||
pattern in `ignore` are skipped. If `logger` is provided, then
|
||||
warning messages are logged there when a directory cannot be listed.
|
||||
"""
|
||||
# Make sure the path isn't a Unicode string.
|
||||
path = bytestring_path(path)
|
||||
|
||||
# Get all the directories and files at this level.
|
||||
try:
|
||||
contents = os.listdir(syspath(path))
|
||||
except OSError as exc:
|
||||
if logger:
|
||||
logger.warn(u'could not list directory {0}: {1}'.format(
|
||||
displayable_path(path), exc.strerror
|
||||
))
|
||||
return
|
||||
dirs = []
|
||||
files = []
|
||||
for base in contents:
|
||||
base = bytestring_path(base)
|
||||
|
||||
# Skip ignored filenames.
|
||||
skip = False
|
||||
for pat in ignore:
|
||||
if fnmatch.fnmatch(base, pat):
|
||||
skip = True
|
||||
break
|
||||
if skip:
|
||||
continue
|
||||
|
||||
# Add to output as either a file or a directory.
|
||||
cur = os.path.join(path, base)
|
||||
if os.path.isdir(syspath(cur)):
|
||||
dirs.append(base)
|
||||
else:
|
||||
files.append(base)
|
||||
|
||||
# Sort lists (case-insensitive) and yield the current level.
|
||||
dirs.sort(key=bytes.lower)
|
||||
files.sort(key=bytes.lower)
|
||||
yield (path, dirs, files)
|
||||
|
||||
# Recurse into directories.
|
||||
for base in dirs:
|
||||
cur = os.path.join(path, base)
|
||||
# yield from sorted_walk(...)
|
||||
for res in sorted_walk(cur, ignore, logger):
|
||||
yield res
|
||||
|
||||
|
||||
def mkdirall(path):
|
||||
"""Make all the enclosing directories of path (like mkdir -p on the
|
||||
parent).
|
||||
"""
|
||||
for ancestor in ancestry(path):
|
||||
if not os.path.isdir(syspath(ancestor)):
|
||||
try:
|
||||
os.mkdir(syspath(ancestor))
|
||||
except (OSError, IOError) as exc:
|
||||
raise FilesystemError(exc, 'create', (ancestor,),
|
||||
traceback.format_exc())
|
||||
|
||||
|
||||
def fnmatch_all(names, patterns):
|
||||
"""Determine whether all strings in `names` match at least one of
|
||||
the `patterns`, which should be shell glob expressions.
|
||||
"""
|
||||
for name in names:
|
||||
matches = False
|
||||
for pattern in patterns:
|
||||
matches = fnmatch.fnmatch(name, pattern)
|
||||
if matches:
|
||||
break
|
||||
if not matches:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def prune_dirs(path, root=None, clutter=('.DS_Store', 'Thumbs.db')):
|
||||
"""If path is an empty directory, then remove it. Recursively remove
|
||||
path's ancestry up to root (which is never removed) where there are
|
||||
empty directories. If path is not contained in root, then nothing is
|
||||
removed. Glob patterns in clutter are ignored when determining
|
||||
emptiness. If root is not provided, then only path may be removed
|
||||
(i.e., no recursive removal).
|
||||
"""
|
||||
path = normpath(path)
|
||||
if root is not None:
|
||||
root = normpath(root)
|
||||
|
||||
ancestors = ancestry(path)
|
||||
if root is None:
|
||||
# Only remove the top directory.
|
||||
ancestors = []
|
||||
elif root in ancestors:
|
||||
# Only remove directories below the root.
|
||||
ancestors = ancestors[ancestors.index(root) + 1:]
|
||||
else:
|
||||
# Remove nothing.
|
||||
return
|
||||
|
||||
# Traverse upward from path.
|
||||
ancestors.append(path)
|
||||
ancestors.reverse()
|
||||
for directory in ancestors:
|
||||
directory = syspath(directory)
|
||||
if not os.path.exists(directory):
|
||||
# Directory gone already.
|
||||
continue
|
||||
if fnmatch_all(os.listdir(directory), clutter):
|
||||
# Directory contains only clutter (or nothing).
|
||||
try:
|
||||
shutil.rmtree(directory)
|
||||
except OSError:
|
||||
break
|
||||
else:
|
||||
break
|
||||
|
||||
|
||||
def components(path):
|
||||
"""Return a list of the path components in path. For instance:
|
||||
|
||||
>>> components('/a/b/c')
|
||||
['a', 'b', 'c']
|
||||
|
||||
The argument should *not* be the result of a call to `syspath`.
|
||||
"""
|
||||
comps = []
|
||||
ances = ancestry(path)
|
||||
for anc in ances:
|
||||
comp = os.path.basename(anc)
|
||||
if comp:
|
||||
comps.append(comp)
|
||||
else: # root
|
||||
comps.append(anc)
|
||||
|
||||
last = os.path.basename(path)
|
||||
if last:
|
||||
comps.append(last)
|
||||
|
||||
return comps
|
||||
|
||||
|
||||
def _fsencoding():
|
||||
"""Get the system's filesystem encoding. On Windows, this is always
|
||||
UTF-8 (not MBCS).
|
||||
"""
|
||||
encoding = sys.getfilesystemencoding() or sys.getdefaultencoding()
|
||||
if encoding == 'mbcs':
|
||||
# On Windows, a broken encoding known to Python as "MBCS" is
|
||||
# used for the filesystem. However, we only use the Unicode API
|
||||
# for Windows paths, so the encoding is actually immaterial so
|
||||
# we can avoid dealing with this nastiness. We arbitrarily
|
||||
# choose UTF-8.
|
||||
encoding = 'utf8'
|
||||
return encoding
|
||||
|
||||
|
||||
def bytestring_path(path):
|
||||
"""Given a path, which is either a str or a unicode, returns a str
|
||||
path (ensuring that we never deal with Unicode pathnames).
|
||||
"""
|
||||
# Pass through bytestrings.
|
||||
if isinstance(path, str):
|
||||
return path
|
||||
|
||||
# On Windows, remove the magic prefix added by `syspath`. This makes
|
||||
# ``bytestring_path(syspath(X)) == X``, i.e., we can safely
|
||||
# round-trip through `syspath`.
|
||||
if os.path.__name__ == 'ntpath' and path.startswith(WINDOWS_MAGIC_PREFIX):
|
||||
path = path[len(WINDOWS_MAGIC_PREFIX):]
|
||||
|
||||
# Try to encode with default encodings, but fall back to UTF8.
|
||||
try:
|
||||
return path.encode(_fsencoding())
|
||||
except (UnicodeError, LookupError):
|
||||
return path.encode('utf8')
|
||||
|
||||
|
||||
def displayable_path(path, separator=u'; '):
|
||||
"""Attempts to decode a bytestring path to a unicode object for the
|
||||
purpose of displaying it to the user. If the `path` argument is a
|
||||
list or a tuple, the elements are joined with `separator`.
|
||||
"""
|
||||
if isinstance(path, (list, tuple)):
|
||||
return separator.join(displayable_path(p) for p in path)
|
||||
elif isinstance(path, unicode):
|
||||
return path
|
||||
elif not isinstance(path, str):
|
||||
# A non-string object: just get its unicode representation.
|
||||
return unicode(path)
|
||||
|
||||
try:
|
||||
return path.decode(_fsencoding(), 'ignore')
|
||||
except (UnicodeError, LookupError):
|
||||
return path.decode('utf8', 'ignore')
|
||||
|
||||
|
||||
def syspath(path, prefix=True):
|
||||
"""Convert a path for use by the operating system. In particular,
|
||||
paths on Windows must receive a magic prefix and must be converted
|
||||
to Unicode before they are sent to the OS. To disable the magic
|
||||
prefix on Windows, set `prefix` to False---but only do this if you
|
||||
*really* know what you're doing.
|
||||
"""
|
||||
# Don't do anything if we're not on windows
|
||||
if os.path.__name__ != 'ntpath':
|
||||
return path
|
||||
|
||||
if not isinstance(path, unicode):
|
||||
# Beets currently represents Windows paths internally with UTF-8
|
||||
# arbitrarily. But earlier versions used MBCS because it is
|
||||
# reported as the FS encoding by Windows. Try both.
|
||||
try:
|
||||
path = path.decode('utf8')
|
||||
except UnicodeError:
|
||||
# The encoding should always be MBCS, Windows' broken
|
||||
# Unicode representation.
|
||||
encoding = sys.getfilesystemencoding() or sys.getdefaultencoding()
|
||||
path = path.decode(encoding, 'replace')
|
||||
|
||||
# Add the magic prefix if it isn't already there.
|
||||
# http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247.aspx
|
||||
if prefix and not path.startswith(WINDOWS_MAGIC_PREFIX):
|
||||
if path.startswith(u'\\\\'):
|
||||
# UNC path. Final path should look like \\?\UNC\...
|
||||
path = u'UNC' + path[1:]
|
||||
path = WINDOWS_MAGIC_PREFIX + path
|
||||
|
||||
return path
|
||||
|
||||
|
||||
def samefile(p1, p2):
|
||||
"""Safer equality for paths."""
|
||||
return shutil._samefile(syspath(p1), syspath(p2))
|
||||
|
||||
|
||||
def remove(path, soft=True):
|
||||
"""Remove the file. If `soft`, then no error will be raised if the
|
||||
file does not exist.
|
||||
"""
|
||||
path = syspath(path)
|
||||
if soft and not os.path.exists(path):
|
||||
return
|
||||
try:
|
||||
os.remove(path)
|
||||
except (OSError, IOError) as exc:
|
||||
raise FilesystemError(exc, 'delete', (path,), traceback.format_exc())
|
||||
|
||||
|
||||
def copy(path, dest, replace=False):
|
||||
"""Copy a plain file. Permissions are not copied. If `dest` already
|
||||
exists, raises a FilesystemError unless `replace` is True. Has no
|
||||
effect if `path` is the same as `dest`. Paths are translated to
|
||||
system paths before the syscall.
|
||||
"""
|
||||
if samefile(path, dest):
|
||||
return
|
||||
path = syspath(path)
|
||||
dest = syspath(dest)
|
||||
if not replace and os.path.exists(dest):
|
||||
raise FilesystemError('file exists', 'copy', (path, dest))
|
||||
try:
|
||||
shutil.copyfile(path, dest)
|
||||
except (OSError, IOError) as exc:
|
||||
raise FilesystemError(exc, 'copy', (path, dest),
|
||||
traceback.format_exc())
|
||||
|
||||
|
||||
def move(path, dest, replace=False):
|
||||
"""Rename a file. `dest` may not be a directory. If `dest` already
|
||||
exists, raises an OSError unless `replace` is True. Has no effect if
|
||||
`path` is the same as `dest`. If the paths are on different
|
||||
filesystems (or the rename otherwise fails), a copy is attempted
|
||||
instead, in which case metadata will *not* be preserved. Paths are
|
||||
translated to system paths.
|
||||
"""
|
||||
if samefile(path, dest):
|
||||
return
|
||||
path = syspath(path)
|
||||
dest = syspath(dest)
|
||||
if os.path.exists(dest) and not replace:
|
||||
raise FilesystemError('file exists', 'rename', (path, dest),
|
||||
traceback.format_exc())
|
||||
|
||||
# First, try renaming the file.
|
||||
try:
|
||||
os.rename(path, dest)
|
||||
except OSError:
|
||||
# Otherwise, copy and delete the original.
|
||||
try:
|
||||
shutil.copyfile(path, dest)
|
||||
os.remove(path)
|
||||
except (OSError, IOError) as exc:
|
||||
raise FilesystemError(exc, 'move', (path, dest),
|
||||
traceback.format_exc())
|
||||
|
||||
|
||||
def link(path, dest, replace=False):
|
||||
"""Create a symbolic link from path to `dest`. Raises an OSError if
|
||||
`dest` already exists, unless `replace` is True. Does nothing if
|
||||
`path` == `dest`."""
|
||||
if (samefile(path, dest)):
|
||||
return
|
||||
|
||||
path = syspath(path)
|
||||
dest = syspath(dest)
|
||||
if os.path.exists(dest) and not replace:
|
||||
raise FilesystemError('file exists', 'rename', (path, dest),
|
||||
traceback.format_exc())
|
||||
try:
|
||||
os.symlink(path, dest)
|
||||
except OSError:
|
||||
raise FilesystemError('Operating system does not support symbolic '
|
||||
'links.', 'link', (path, dest),
|
||||
traceback.format_exc())
|
||||
|
||||
|
||||
def unique_path(path):
|
||||
"""Returns a version of ``path`` that does not exist on the
|
||||
filesystem. Specifically, if ``path` itself already exists, then
|
||||
something unique is appended to the path.
|
||||
"""
|
||||
if not os.path.exists(syspath(path)):
|
||||
return path
|
||||
|
||||
base, ext = os.path.splitext(path)
|
||||
match = re.search(r'\.(\d)+$', base)
|
||||
if match:
|
||||
num = int(match.group(1))
|
||||
base = base[:match.start()]
|
||||
else:
|
||||
num = 0
|
||||
while True:
|
||||
num += 1
|
||||
new_path = '%s.%i%s' % (base, num, ext)
|
||||
if not os.path.exists(new_path):
|
||||
return new_path
|
||||
|
||||
# Note: The Windows "reserved characters" are, of course, allowed on
|
||||
# Unix. They are forbidden here because they cause problems on Samba
|
||||
# shares, which are sufficiently common as to cause frequent problems.
|
||||
# http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247.aspx
|
||||
CHAR_REPLACE = [
|
||||
(re.compile(ur'[\\/]'), u'_'), # / and \ -- forbidden everywhere.
|
||||
(re.compile(ur'^\.'), u'_'), # Leading dot (hidden files on Unix).
|
||||
(re.compile(ur'[\x00-\x1f]'), u''), # Control characters.
|
||||
(re.compile(ur'[<>:"\?\*\|]'), u'_'), # Windows "reserved characters".
|
||||
(re.compile(ur'\.$'), u'_'), # Trailing dots.
|
||||
(re.compile(ur'\s+$'), u''), # Trailing whitespace.
|
||||
]
|
||||
|
||||
|
||||
def sanitize_path(path, replacements=None):
|
||||
"""Takes a path (as a Unicode string) and makes sure that it is
|
||||
legal. Returns a new path. Only works with fragments; won't work
|
||||
reliably on Windows when a path begins with a drive letter. Path
|
||||
separators (including altsep!) should already be cleaned from the
|
||||
path components. If replacements is specified, it is used *instead*
|
||||
of the default set of replacements; it must be a list of (compiled
|
||||
regex, replacement string) pairs.
|
||||
"""
|
||||
replacements = replacements or CHAR_REPLACE
|
||||
|
||||
comps = components(path)
|
||||
if not comps:
|
||||
return ''
|
||||
for i, comp in enumerate(comps):
|
||||
for regex, repl in replacements:
|
||||
comp = regex.sub(repl, comp)
|
||||
comps[i] = comp
|
||||
return os.path.join(*comps)
|
||||
|
||||
|
||||
def truncate_path(path, length=MAX_FILENAME_LENGTH):
|
||||
"""Given a bytestring path or a Unicode path fragment, truncate the
|
||||
components to a legal length. In the last component, the extension
|
||||
is preserved.
|
||||
"""
|
||||
comps = components(path)
|
||||
|
||||
out = [c[:length] for c in comps]
|
||||
base, ext = os.path.splitext(comps[-1])
|
||||
if ext:
|
||||
# Last component has an extension.
|
||||
base = base[:length - len(ext)]
|
||||
out[-1] = base + ext
|
||||
|
||||
return os.path.join(*out)
|
||||
|
||||
|
||||
def str2bool(value):
|
||||
"""Returns a boolean reflecting a human-entered string."""
|
||||
if value.lower() in ('yes', '1', 'true', 't', 'y'):
|
||||
return True
|
||||
else:
|
||||
return False
|
||||
|
||||
|
||||
def as_string(value):
|
||||
"""Convert a value to a Unicode object for matching with a query.
|
||||
None becomes the empty string. Bytestrings are silently decoded.
|
||||
"""
|
||||
if value is None:
|
||||
return u''
|
||||
elif isinstance(value, buffer):
|
||||
return str(value).decode('utf8', 'ignore')
|
||||
elif isinstance(value, str):
|
||||
return value.decode('utf8', 'ignore')
|
||||
else:
|
||||
return unicode(value)
|
||||
|
||||
|
||||
def levenshtein(s1, s2):
|
||||
"""A nice DP edit distance implementation from Wikibooks:
|
||||
http://en.wikibooks.org/wiki/Algorithm_implementation/Strings/
|
||||
Levenshtein_distance#Python
|
||||
"""
|
||||
if len(s1) < len(s2):
|
||||
return levenshtein(s2, s1)
|
||||
if not s1:
|
||||
return len(s2)
|
||||
|
||||
previous_row = xrange(len(s2) + 1)
|
||||
for i, c1 in enumerate(s1):
|
||||
current_row = [i + 1]
|
||||
for j, c2 in enumerate(s2):
|
||||
insertions = previous_row[j + 1] + 1
|
||||
deletions = current_row[j] + 1
|
||||
substitutions = previous_row[j] + (c1 != c2)
|
||||
current_row.append(min(insertions, deletions, substitutions))
|
||||
previous_row = current_row
|
||||
|
||||
return previous_row[-1]
|
||||
|
||||
|
||||
def plurality(objs):
|
||||
"""Given a sequence of comparable objects, returns the object that
|
||||
is most common in the set and the frequency of that object. The
|
||||
sequence must contain at least one object.
|
||||
"""
|
||||
# Calculate frequencies.
|
||||
freqs = defaultdict(int)
|
||||
for obj in objs:
|
||||
freqs[obj] += 1
|
||||
|
||||
if not freqs:
|
||||
raise ValueError('sequence must be non-empty')
|
||||
|
||||
# Find object with maximum frequency.
|
||||
max_freq = 0
|
||||
res = None
|
||||
for obj, freq in freqs.items():
|
||||
if freq > max_freq:
|
||||
max_freq = freq
|
||||
res = obj
|
||||
|
||||
return res, max_freq
|
||||
|
||||
|
||||
def cpu_count():
|
||||
"""Return the number of hardware thread contexts (cores or SMT
|
||||
threads) in the system.
|
||||
"""
|
||||
# Adapted from the soundconverter project:
|
||||
# https://github.com/kassoulet/soundconverter
|
||||
if sys.platform == 'win32':
|
||||
try:
|
||||
num = int(os.environ['NUMBER_OF_PROCESSORS'])
|
||||
except (ValueError, KeyError):
|
||||
num = 0
|
||||
elif sys.platform == 'darwin':
|
||||
try:
|
||||
num = int(command_output(['sysctl', '-n', 'hw.ncpu']))
|
||||
except ValueError:
|
||||
num = 0
|
||||
else:
|
||||
try:
|
||||
num = os.sysconf('SC_NPROCESSORS_ONLN')
|
||||
except (ValueError, OSError, AttributeError):
|
||||
num = 0
|
||||
if num >= 1:
|
||||
return num
|
||||
else:
|
||||
return 1
|
||||
|
||||
|
||||
def command_output(cmd, shell=False):
|
||||
"""Runs the command and returns its output after it has exited.
|
||||
|
||||
``cmd`` is a list of arguments starting with the command names. If
|
||||
``shell`` is true, ``cmd`` is assumed to be a string and passed to a
|
||||
shell to execute.
|
||||
|
||||
If the process exits with a non-zero return code
|
||||
``subprocess.CalledProcessError`` is raised. May also raise
|
||||
``OSError``.
|
||||
|
||||
This replaces `subprocess.check_output`, which isn't available in
|
||||
Python 2.6 and which can have problems if lots of output is sent to
|
||||
stderr.
|
||||
"""
|
||||
proc = subprocess.Popen(
|
||||
cmd,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
close_fds=platform.system() != 'Windows',
|
||||
shell=shell
|
||||
)
|
||||
stdout, stderr = proc.communicate()
|
||||
if proc.returncode:
|
||||
raise subprocess.CalledProcessError(
|
||||
returncode=proc.returncode,
|
||||
cmd=' '.join(cmd),
|
||||
)
|
||||
return stdout
|
||||
|
||||
|
||||
def max_filename_length(path, limit=MAX_FILENAME_LENGTH):
|
||||
"""Attempt to determine the maximum filename length for the
|
||||
filesystem containing `path`. If the value is greater than `limit`,
|
||||
then `limit` is used instead (to prevent errors when a filesystem
|
||||
misreports its capacity). If it cannot be determined (e.g., on
|
||||
Windows), return `limit`.
|
||||
"""
|
||||
if hasattr(os, 'statvfs'):
|
||||
try:
|
||||
res = os.statvfs(path)
|
||||
except OSError:
|
||||
return limit
|
||||
return min(res[9], limit)
|
||||
else:
|
||||
return limit
|
||||
@@ -0,0 +1,211 @@
|
||||
# This file is part of beets.
|
||||
# Copyright 2014, Fabrice Laporte
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining
|
||||
# a copy of this software and associated documentation files (the
|
||||
# "Software"), to deal in the Software without restriction, including
|
||||
# without limitation the rights to use, copy, modify, merge, publish,
|
||||
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||
# permit persons to whom the Software is furnished to do so, subject to
|
||||
# the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
|
||||
"""Abstraction layer to resize images using PIL, ImageMagick, or a
|
||||
public resizing proxy if neither is available.
|
||||
"""
|
||||
import urllib
|
||||
import subprocess
|
||||
import os
|
||||
import re
|
||||
from tempfile import NamedTemporaryFile
|
||||
import logging
|
||||
from beets import util
|
||||
|
||||
# Resizing methods
|
||||
PIL = 1
|
||||
IMAGEMAGICK = 2
|
||||
WEBPROXY = 3
|
||||
|
||||
PROXY_URL = 'http://images.weserv.nl/'
|
||||
|
||||
log = logging.getLogger('beets')
|
||||
|
||||
|
||||
def resize_url(url, maxwidth):
|
||||
"""Return a proxied image URL that resizes the original image to
|
||||
maxwidth (preserving aspect ratio).
|
||||
"""
|
||||
return '{0}?{1}'.format(PROXY_URL, urllib.urlencode({
|
||||
'url': url.replace('http://', ''),
|
||||
'w': str(maxwidth),
|
||||
}))
|
||||
|
||||
|
||||
def temp_file_for(path):
|
||||
"""Return an unused filename with the same extension as the
|
||||
specified path.
|
||||
"""
|
||||
ext = os.path.splitext(path)[1]
|
||||
with NamedTemporaryFile(suffix=ext, delete=False) as f:
|
||||
return f.name
|
||||
|
||||
|
||||
def pil_resize(maxwidth, path_in, path_out=None):
|
||||
"""Resize using Python Imaging Library (PIL). Return the output path
|
||||
of resized image.
|
||||
"""
|
||||
path_out = path_out or temp_file_for(path_in)
|
||||
from PIL import Image
|
||||
log.debug(u'artresizer: PIL resizing {0} to {1}'.format(
|
||||
util.displayable_path(path_in), util.displayable_path(path_out)
|
||||
))
|
||||
|
||||
try:
|
||||
im = Image.open(util.syspath(path_in))
|
||||
size = maxwidth, maxwidth
|
||||
im.thumbnail(size, Image.ANTIALIAS)
|
||||
im.save(path_out)
|
||||
return path_out
|
||||
except IOError:
|
||||
log.error(u"PIL cannot create thumbnail for '{0}'".format(
|
||||
util.displayable_path(path_in)
|
||||
))
|
||||
return path_in
|
||||
|
||||
|
||||
def im_resize(maxwidth, path_in, path_out=None):
|
||||
"""Resize using ImageMagick's ``convert`` tool.
|
||||
Return the output path of resized image.
|
||||
"""
|
||||
path_out = path_out or temp_file_for(path_in)
|
||||
log.debug(u'artresizer: ImageMagick resizing {0} to {1}'.format(
|
||||
util.displayable_path(path_in), util.displayable_path(path_out)
|
||||
))
|
||||
|
||||
# "-resize widthxheight>" shrinks images with dimension(s) larger
|
||||
# than the corresponding width and/or height dimension(s). The >
|
||||
# "only shrink" flag is prefixed by ^ escape char for Windows
|
||||
# compatibility.
|
||||
try:
|
||||
util.command_output([
|
||||
'convert', util.syspath(path_in),
|
||||
'-resize', '{0}x^>'.format(maxwidth), path_out
|
||||
])
|
||||
except subprocess.CalledProcessError:
|
||||
log.warn(u'artresizer: IM convert failed for {0}'.format(
|
||||
util.displayable_path(path_in)
|
||||
))
|
||||
return path_in
|
||||
return path_out
|
||||
|
||||
|
||||
BACKEND_FUNCS = {
|
||||
PIL: pil_resize,
|
||||
IMAGEMAGICK: im_resize,
|
||||
}
|
||||
|
||||
|
||||
class Shareable(type):
|
||||
"""A pseudo-singleton metaclass that allows both shared and
|
||||
non-shared instances. The ``MyClass.shared`` property holds a
|
||||
lazily-created shared instance of ``MyClass`` while calling
|
||||
``MyClass()`` to construct a new object works as usual.
|
||||
"""
|
||||
def __init__(cls, name, bases, dict):
|
||||
super(Shareable, cls).__init__(name, bases, dict)
|
||||
cls._instance = None
|
||||
|
||||
@property
|
||||
def shared(cls):
|
||||
if cls._instance is None:
|
||||
cls._instance = cls()
|
||||
return cls._instance
|
||||
|
||||
|
||||
class ArtResizer(object):
|
||||
"""A singleton class that performs image resizes.
|
||||
"""
|
||||
__metaclass__ = Shareable
|
||||
|
||||
def __init__(self, method=None):
|
||||
"""Create a resizer object for the given method or, if none is
|
||||
specified, with an inferred method.
|
||||
"""
|
||||
self.method = self._check_method(method)
|
||||
log.debug(u"artresizer: method is {0}".format(self.method))
|
||||
self.can_compare = self._can_compare()
|
||||
|
||||
def resize(self, maxwidth, path_in, path_out=None):
|
||||
"""Manipulate an image file according to the method, returning a
|
||||
new path. For PIL or IMAGEMAGIC methods, resizes the image to a
|
||||
temporary file. For WEBPROXY, returns `path_in` unmodified.
|
||||
"""
|
||||
if self.local:
|
||||
func = BACKEND_FUNCS[self.method[0]]
|
||||
return func(maxwidth, path_in, path_out)
|
||||
else:
|
||||
return path_in
|
||||
|
||||
def proxy_url(self, maxwidth, url):
|
||||
"""Modifies an image URL according the method, returning a new
|
||||
URL. For WEBPROXY, a URL on the proxy server is returned.
|
||||
Otherwise, the URL is returned unmodified.
|
||||
"""
|
||||
if self.local:
|
||||
return url
|
||||
else:
|
||||
return resize_url(url, maxwidth)
|
||||
|
||||
@property
|
||||
def local(self):
|
||||
"""A boolean indicating whether the resizing method is performed
|
||||
locally (i.e., PIL or ImageMagick).
|
||||
"""
|
||||
return self.method[0] in BACKEND_FUNCS
|
||||
|
||||
def _can_compare(self):
|
||||
"""A boolean indicating whether image comparison is available"""
|
||||
|
||||
return self.method[0] == IMAGEMAGICK and self.method[1] > (6, 8, 7)
|
||||
|
||||
@staticmethod
|
||||
def _check_method(method=None):
|
||||
"""A tuple indicating whether current method is available and its
|
||||
version. If no method is given, it returns a supported one.
|
||||
"""
|
||||
# Guess available method
|
||||
if not method:
|
||||
for m in [IMAGEMAGICK, PIL]:
|
||||
_, version = ArtResizer._check_method(m)
|
||||
if version:
|
||||
return (m, version)
|
||||
return (WEBPROXY, (0))
|
||||
|
||||
if method == IMAGEMAGICK:
|
||||
|
||||
# Try invoking ImageMagick's "convert".
|
||||
try:
|
||||
out = util.command_output(['identify', '--version'])
|
||||
|
||||
if 'imagemagick' in out.lower():
|
||||
pattern = r".+ (\d+)\.(\d+)\.(\d+).*"
|
||||
match = re.search(pattern, out)
|
||||
if match:
|
||||
return (IMAGEMAGICK,
|
||||
(int(match.group(1)),
|
||||
int(match.group(2)),
|
||||
int(match.group(3))))
|
||||
return (IMAGEMAGICK, (0))
|
||||
|
||||
except (subprocess.CalledProcessError, OSError):
|
||||
return (IMAGEMAGICK, None)
|
||||
|
||||
if method == PIL:
|
||||
# Try importing PIL.
|
||||
try:
|
||||
__import__('PIL', fromlist=['Image'])
|
||||
return (PIL, (0))
|
||||
except ImportError:
|
||||
return (PIL, None)
|
||||
@@ -0,0 +1,647 @@
|
||||
"""Extremely simple pure-Python implementation of coroutine-style
|
||||
asynchronous socket I/O. Inspired by, but inferior to, Eventlet.
|
||||
Bluelet can also be thought of as a less-terrible replacement for
|
||||
asyncore.
|
||||
|
||||
Bluelet: easy concurrency without all the messy parallelism.
|
||||
"""
|
||||
import socket
|
||||
import select
|
||||
import sys
|
||||
import types
|
||||
import errno
|
||||
import traceback
|
||||
import time
|
||||
import collections
|
||||
|
||||
|
||||
# A little bit of "six" (Python 2/3 compatibility): cope with PEP 3109 syntax
|
||||
# changes.
|
||||
|
||||
PY3 = sys.version_info[0] == 3
|
||||
if PY3:
|
||||
def _reraise(typ, exc, tb):
|
||||
raise exc.with_traceback(tb)
|
||||
else:
|
||||
exec("""
|
||||
def _reraise(typ, exc, tb):
|
||||
raise typ, exc, tb
|
||||
""")
|
||||
|
||||
|
||||
# Basic events used for thread scheduling.
|
||||
|
||||
class Event(object):
|
||||
"""Just a base class identifying Bluelet events. An event is an
|
||||
object yielded from a Bluelet thread coroutine to suspend operation
|
||||
and communicate with the scheduler.
|
||||
"""
|
||||
pass
|
||||
|
||||
|
||||
class WaitableEvent(Event):
|
||||
"""A waitable event is one encapsulating an action that can be
|
||||
waited for using a select() call. That is, it's an event with an
|
||||
associated file descriptor.
|
||||
"""
|
||||
def waitables(self):
|
||||
"""Return "waitable" objects to pass to select(). Should return
|
||||
three iterables for input readiness, output readiness, and
|
||||
exceptional conditions (i.e., the three lists passed to
|
||||
select()).
|
||||
"""
|
||||
return (), (), ()
|
||||
|
||||
def fire(self):
|
||||
"""Called when an associated file descriptor becomes ready
|
||||
(i.e., is returned from a select() call).
|
||||
"""
|
||||
pass
|
||||
|
||||
|
||||
class ValueEvent(Event):
|
||||
"""An event that does nothing but return a fixed value."""
|
||||
def __init__(self, value):
|
||||
self.value = value
|
||||
|
||||
|
||||
class ExceptionEvent(Event):
|
||||
"""Raise an exception at the yield point. Used internally."""
|
||||
def __init__(self, exc_info):
|
||||
self.exc_info = exc_info
|
||||
|
||||
|
||||
class SpawnEvent(Event):
|
||||
"""Add a new coroutine thread to the scheduler."""
|
||||
def __init__(self, coro):
|
||||
self.spawned = coro
|
||||
|
||||
|
||||
class JoinEvent(Event):
|
||||
"""Suspend the thread until the specified child thread has
|
||||
completed.
|
||||
"""
|
||||
def __init__(self, child):
|
||||
self.child = child
|
||||
|
||||
|
||||
class KillEvent(Event):
|
||||
"""Unschedule a child thread."""
|
||||
def __init__(self, child):
|
||||
self.child = child
|
||||
|
||||
|
||||
class DelegationEvent(Event):
|
||||
"""Suspend execution of the current thread, start a new thread and,
|
||||
once the child thread finished, return control to the parent
|
||||
thread.
|
||||
"""
|
||||
def __init__(self, coro):
|
||||
self.spawned = coro
|
||||
|
||||
|
||||
class ReturnEvent(Event):
|
||||
"""Return a value the current thread's delegator at the point of
|
||||
delegation. Ends the current (delegate) thread.
|
||||
"""
|
||||
def __init__(self, value):
|
||||
self.value = value
|
||||
|
||||
|
||||
class SleepEvent(WaitableEvent):
|
||||
"""Suspend the thread for a given duration.
|
||||
"""
|
||||
def __init__(self, duration):
|
||||
self.wakeup_time = time.time() + duration
|
||||
|
||||
def time_left(self):
|
||||
return max(self.wakeup_time - time.time(), 0.0)
|
||||
|
||||
|
||||
class ReadEvent(WaitableEvent):
|
||||
"""Reads from a file-like object."""
|
||||
def __init__(self, fd, bufsize):
|
||||
self.fd = fd
|
||||
self.bufsize = bufsize
|
||||
|
||||
def waitables(self):
|
||||
return (self.fd,), (), ()
|
||||
|
||||
def fire(self):
|
||||
return self.fd.read(self.bufsize)
|
||||
|
||||
|
||||
class WriteEvent(WaitableEvent):
|
||||
"""Writes to a file-like object."""
|
||||
def __init__(self, fd, data):
|
||||
self.fd = fd
|
||||
self.data = data
|
||||
|
||||
def waitable(self):
|
||||
return (), (self.fd,), ()
|
||||
|
||||
def fire(self):
|
||||
self.fd.write(self.data)
|
||||
|
||||
|
||||
# Core logic for executing and scheduling threads.
|
||||
|
||||
def _event_select(events):
|
||||
"""Perform a select() over all the Events provided, returning the
|
||||
ones ready to be fired. Only WaitableEvents (including SleepEvents)
|
||||
matter here; all other events are ignored (and thus postponed).
|
||||
"""
|
||||
# Gather waitables and wakeup times.
|
||||
waitable_to_event = {}
|
||||
rlist, wlist, xlist = [], [], []
|
||||
earliest_wakeup = None
|
||||
for event in events:
|
||||
if isinstance(event, SleepEvent):
|
||||
if not earliest_wakeup:
|
||||
earliest_wakeup = event.wakeup_time
|
||||
else:
|
||||
earliest_wakeup = min(earliest_wakeup, event.wakeup_time)
|
||||
elif isinstance(event, WaitableEvent):
|
||||
r, w, x = event.waitables()
|
||||
rlist += r
|
||||
wlist += w
|
||||
xlist += x
|
||||
for waitable in r:
|
||||
waitable_to_event[('r', waitable)] = event
|
||||
for waitable in w:
|
||||
waitable_to_event[('w', waitable)] = event
|
||||
for waitable in x:
|
||||
waitable_to_event[('x', waitable)] = event
|
||||
|
||||
# If we have a any sleeping threads, determine how long to sleep.
|
||||
if earliest_wakeup:
|
||||
timeout = max(earliest_wakeup - time.time(), 0.0)
|
||||
else:
|
||||
timeout = None
|
||||
|
||||
# Perform select() if we have any waitables.
|
||||
if rlist or wlist or xlist:
|
||||
rready, wready, xready = select.select(rlist, wlist, xlist, timeout)
|
||||
else:
|
||||
rready, wready, xready = (), (), ()
|
||||
if timeout:
|
||||
time.sleep(timeout)
|
||||
|
||||
# Gather ready events corresponding to the ready waitables.
|
||||
ready_events = set()
|
||||
for ready in rready:
|
||||
ready_events.add(waitable_to_event[('r', ready)])
|
||||
for ready in wready:
|
||||
ready_events.add(waitable_to_event[('w', ready)])
|
||||
for ready in xready:
|
||||
ready_events.add(waitable_to_event[('x', ready)])
|
||||
|
||||
# Gather any finished sleeps.
|
||||
for event in events:
|
||||
if isinstance(event, SleepEvent) and event.time_left() == 0.0:
|
||||
ready_events.add(event)
|
||||
|
||||
return ready_events
|
||||
|
||||
|
||||
class ThreadException(Exception):
|
||||
def __init__(self, coro, exc_info):
|
||||
self.coro = coro
|
||||
self.exc_info = exc_info
|
||||
|
||||
def reraise(self):
|
||||
_reraise(self.exc_info[0], self.exc_info[1], self.exc_info[2])
|
||||
|
||||
|
||||
SUSPENDED = Event() # Special sentinel placeholder for suspended threads.
|
||||
|
||||
|
||||
class Delegated(Event):
|
||||
"""Placeholder indicating that a thread has delegated execution to a
|
||||
different thread.
|
||||
"""
|
||||
def __init__(self, child):
|
||||
self.child = child
|
||||
|
||||
|
||||
def run(root_coro):
|
||||
"""Schedules a coroutine, running it to completion. This
|
||||
encapsulates the Bluelet scheduler, which the root coroutine can
|
||||
add to by spawning new coroutines.
|
||||
"""
|
||||
# The "threads" dictionary keeps track of all the currently-
|
||||
# executing and suspended coroutines. It maps coroutines to their
|
||||
# currently "blocking" event. The event value may be SUSPENDED if
|
||||
# the coroutine is waiting on some other condition: namely, a
|
||||
# delegated coroutine or a joined coroutine. In this case, the
|
||||
# coroutine should *also* appear as a value in one of the below
|
||||
# dictionaries `delegators` or `joiners`.
|
||||
threads = {root_coro: ValueEvent(None)}
|
||||
|
||||
# Maps child coroutines to delegating parents.
|
||||
delegators = {}
|
||||
|
||||
# Maps child coroutines to joining (exit-waiting) parents.
|
||||
joiners = collections.defaultdict(list)
|
||||
|
||||
def complete_thread(coro, return_value):
|
||||
"""Remove a coroutine from the scheduling pool, awaking
|
||||
delegators and joiners as necessary and returning the specified
|
||||
value to any delegating parent.
|
||||
"""
|
||||
del threads[coro]
|
||||
|
||||
# Resume delegator.
|
||||
if coro in delegators:
|
||||
threads[delegators[coro]] = ValueEvent(return_value)
|
||||
del delegators[coro]
|
||||
|
||||
# Resume joiners.
|
||||
if coro in joiners:
|
||||
for parent in joiners[coro]:
|
||||
threads[parent] = ValueEvent(None)
|
||||
del joiners[coro]
|
||||
|
||||
def advance_thread(coro, value, is_exc=False):
|
||||
"""After an event is fired, run a given coroutine associated with
|
||||
it in the threads dict until it yields again. If the coroutine
|
||||
exits, then the thread is removed from the pool. If the coroutine
|
||||
raises an exception, it is reraised in a ThreadException. If
|
||||
is_exc is True, then the value must be an exc_info tuple and the
|
||||
exception is thrown into the coroutine.
|
||||
"""
|
||||
try:
|
||||
if is_exc:
|
||||
next_event = coro.throw(*value)
|
||||
else:
|
||||
next_event = coro.send(value)
|
||||
except StopIteration:
|
||||
# Thread is done.
|
||||
complete_thread(coro, None)
|
||||
except:
|
||||
# Thread raised some other exception.
|
||||
del threads[coro]
|
||||
raise ThreadException(coro, sys.exc_info())
|
||||
else:
|
||||
if isinstance(next_event, types.GeneratorType):
|
||||
# Automatically invoke sub-coroutines. (Shorthand for
|
||||
# explicit bluelet.call().)
|
||||
next_event = DelegationEvent(next_event)
|
||||
threads[coro] = next_event
|
||||
|
||||
def kill_thread(coro):
|
||||
"""Unschedule this thread and its (recursive) delegates.
|
||||
"""
|
||||
# Collect all coroutines in the delegation stack.
|
||||
coros = [coro]
|
||||
while isinstance(threads[coro], Delegated):
|
||||
coro = threads[coro].child
|
||||
coros.append(coro)
|
||||
|
||||
# Complete each coroutine from the top to the bottom of the
|
||||
# stack.
|
||||
for coro in reversed(coros):
|
||||
complete_thread(coro, None)
|
||||
|
||||
# Continue advancing threads until root thread exits.
|
||||
exit_te = None
|
||||
while threads:
|
||||
try:
|
||||
# Look for events that can be run immediately. Continue
|
||||
# running immediate events until nothing is ready.
|
||||
while True:
|
||||
have_ready = False
|
||||
for coro, event in list(threads.items()):
|
||||
if isinstance(event, SpawnEvent):
|
||||
threads[event.spawned] = ValueEvent(None) # Spawn.
|
||||
advance_thread(coro, None)
|
||||
have_ready = True
|
||||
elif isinstance(event, ValueEvent):
|
||||
advance_thread(coro, event.value)
|
||||
have_ready = True
|
||||
elif isinstance(event, ExceptionEvent):
|
||||
advance_thread(coro, event.exc_info, True)
|
||||
have_ready = True
|
||||
elif isinstance(event, DelegationEvent):
|
||||
threads[coro] = Delegated(event.spawned) # Suspend.
|
||||
threads[event.spawned] = ValueEvent(None) # Spawn.
|
||||
delegators[event.spawned] = coro
|
||||
have_ready = True
|
||||
elif isinstance(event, ReturnEvent):
|
||||
# Thread is done.
|
||||
complete_thread(coro, event.value)
|
||||
have_ready = True
|
||||
elif isinstance(event, JoinEvent):
|
||||
threads[coro] = SUSPENDED # Suspend.
|
||||
joiners[event.child].append(coro)
|
||||
have_ready = True
|
||||
elif isinstance(event, KillEvent):
|
||||
threads[coro] = ValueEvent(None)
|
||||
kill_thread(event.child)
|
||||
have_ready = True
|
||||
|
||||
# Only start the select when nothing else is ready.
|
||||
if not have_ready:
|
||||
break
|
||||
|
||||
# Wait and fire.
|
||||
event2coro = dict((v, k) for k, v in threads.items())
|
||||
for event in _event_select(threads.values()):
|
||||
# Run the IO operation, but catch socket errors.
|
||||
try:
|
||||
value = event.fire()
|
||||
except socket.error as exc:
|
||||
if isinstance(exc.args, tuple) and \
|
||||
exc.args[0] == errno.EPIPE:
|
||||
# Broken pipe. Remote host disconnected.
|
||||
pass
|
||||
else:
|
||||
traceback.print_exc()
|
||||
# Abort the coroutine.
|
||||
threads[event2coro[event]] = ReturnEvent(None)
|
||||
else:
|
||||
advance_thread(event2coro[event], value)
|
||||
|
||||
except ThreadException as te:
|
||||
# Exception raised from inside a thread.
|
||||
event = ExceptionEvent(te.exc_info)
|
||||
if te.coro in delegators:
|
||||
# The thread is a delegate. Raise exception in its
|
||||
# delegator.
|
||||
threads[delegators[te.coro]] = event
|
||||
del delegators[te.coro]
|
||||
else:
|
||||
# The thread is root-level. Raise in client code.
|
||||
exit_te = te
|
||||
break
|
||||
|
||||
except:
|
||||
# For instance, KeyboardInterrupt during select(). Raise
|
||||
# into root thread and terminate others.
|
||||
threads = {root_coro: ExceptionEvent(sys.exc_info())}
|
||||
|
||||
# If any threads still remain, kill them.
|
||||
for coro in threads:
|
||||
coro.close()
|
||||
|
||||
# If we're exiting with an exception, raise it in the client.
|
||||
if exit_te:
|
||||
exit_te.reraise()
|
||||
|
||||
|
||||
# Sockets and their associated events.
|
||||
|
||||
class SocketClosedError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class Listener(object):
|
||||
"""A socket wrapper object for listening sockets.
|
||||
"""
|
||||
def __init__(self, host, port):
|
||||
"""Create a listening socket on the given hostname and port.
|
||||
"""
|
||||
self._closed = False
|
||||
self.host = host
|
||||
self.port = port
|
||||
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
self.sock.bind((host, port))
|
||||
self.sock.listen(5)
|
||||
|
||||
def accept(self):
|
||||
"""An event that waits for a connection on the listening socket.
|
||||
When a connection is made, the event returns a Connection
|
||||
object.
|
||||
"""
|
||||
if self._closed:
|
||||
raise SocketClosedError()
|
||||
return AcceptEvent(self)
|
||||
|
||||
def close(self):
|
||||
"""Immediately close the listening socket. (Not an event.)
|
||||
"""
|
||||
self._closed = True
|
||||
self.sock.close()
|
||||
|
||||
|
||||
class Connection(object):
|
||||
"""A socket wrapper object for connected sockets.
|
||||
"""
|
||||
def __init__(self, sock, addr):
|
||||
self.sock = sock
|
||||
self.addr = addr
|
||||
self._buf = b''
|
||||
self._closed = False
|
||||
|
||||
def close(self):
|
||||
"""Close the connection."""
|
||||
self._closed = True
|
||||
self.sock.close()
|
||||
|
||||
def recv(self, size):
|
||||
"""Read at most size bytes of data from the socket."""
|
||||
if self._closed:
|
||||
raise SocketClosedError()
|
||||
|
||||
if self._buf:
|
||||
# We already have data read previously.
|
||||
out = self._buf[:size]
|
||||
self._buf = self._buf[size:]
|
||||
return ValueEvent(out)
|
||||
else:
|
||||
return ReceiveEvent(self, size)
|
||||
|
||||
def send(self, data):
|
||||
"""Sends data on the socket, returning the number of bytes
|
||||
successfully sent.
|
||||
"""
|
||||
if self._closed:
|
||||
raise SocketClosedError()
|
||||
return SendEvent(self, data)
|
||||
|
||||
def sendall(self, data):
|
||||
"""Send all of data on the socket."""
|
||||
if self._closed:
|
||||
raise SocketClosedError()
|
||||
return SendEvent(self, data, True)
|
||||
|
||||
def readline(self, terminator=b"\n", bufsize=1024):
|
||||
"""Reads a line (delimited by terminator) from the socket."""
|
||||
if self._closed:
|
||||
raise SocketClosedError()
|
||||
|
||||
while True:
|
||||
if terminator in self._buf:
|
||||
line, self._buf = self._buf.split(terminator, 1)
|
||||
line += terminator
|
||||
yield ReturnEvent(line)
|
||||
break
|
||||
data = yield ReceiveEvent(self, bufsize)
|
||||
if data:
|
||||
self._buf += data
|
||||
else:
|
||||
line = self._buf
|
||||
self._buf = b''
|
||||
yield ReturnEvent(line)
|
||||
break
|
||||
|
||||
|
||||
class AcceptEvent(WaitableEvent):
|
||||
"""An event for Listener objects (listening sockets) that suspends
|
||||
execution until the socket gets a connection.
|
||||
"""
|
||||
def __init__(self, listener):
|
||||
self.listener = listener
|
||||
|
||||
def waitables(self):
|
||||
return (self.listener.sock,), (), ()
|
||||
|
||||
def fire(self):
|
||||
sock, addr = self.listener.sock.accept()
|
||||
return Connection(sock, addr)
|
||||
|
||||
|
||||
class ReceiveEvent(WaitableEvent):
|
||||
"""An event for Connection objects (connected sockets) for
|
||||
asynchronously reading data.
|
||||
"""
|
||||
def __init__(self, conn, bufsize):
|
||||
self.conn = conn
|
||||
self.bufsize = bufsize
|
||||
|
||||
def waitables(self):
|
||||
return (self.conn.sock,), (), ()
|
||||
|
||||
def fire(self):
|
||||
return self.conn.sock.recv(self.bufsize)
|
||||
|
||||
|
||||
class SendEvent(WaitableEvent):
|
||||
"""An event for Connection objects (connected sockets) for
|
||||
asynchronously writing data.
|
||||
"""
|
||||
def __init__(self, conn, data, sendall=False):
|
||||
self.conn = conn
|
||||
self.data = data
|
||||
self.sendall = sendall
|
||||
|
||||
def waitables(self):
|
||||
return (), (self.conn.sock,), ()
|
||||
|
||||
def fire(self):
|
||||
if self.sendall:
|
||||
return self.conn.sock.sendall(self.data)
|
||||
else:
|
||||
return self.conn.sock.send(self.data)
|
||||
|
||||
|
||||
# Public interface for threads; each returns an event object that
|
||||
# can immediately be "yield"ed.
|
||||
|
||||
def null():
|
||||
"""Event: yield to the scheduler without doing anything special.
|
||||
"""
|
||||
return ValueEvent(None)
|
||||
|
||||
|
||||
def spawn(coro):
|
||||
"""Event: add another coroutine to the scheduler. Both the parent
|
||||
and child coroutines run concurrently.
|
||||
"""
|
||||
if not isinstance(coro, types.GeneratorType):
|
||||
raise ValueError('%s is not a coroutine' % str(coro))
|
||||
return SpawnEvent(coro)
|
||||
|
||||
|
||||
def call(coro):
|
||||
"""Event: delegate to another coroutine. The current coroutine
|
||||
is resumed once the sub-coroutine finishes. If the sub-coroutine
|
||||
returns a value using end(), then this event returns that value.
|
||||
"""
|
||||
if not isinstance(coro, types.GeneratorType):
|
||||
raise ValueError('%s is not a coroutine' % str(coro))
|
||||
return DelegationEvent(coro)
|
||||
|
||||
|
||||
def end(value=None):
|
||||
"""Event: ends the coroutine and returns a value to its
|
||||
delegator.
|
||||
"""
|
||||
return ReturnEvent(value)
|
||||
|
||||
|
||||
def read(fd, bufsize=None):
|
||||
"""Event: read from a file descriptor asynchronously."""
|
||||
if bufsize is None:
|
||||
# Read all.
|
||||
def reader():
|
||||
buf = []
|
||||
while True:
|
||||
data = yield read(fd, 1024)
|
||||
if not data:
|
||||
break
|
||||
buf.append(data)
|
||||
yield ReturnEvent(''.join(buf))
|
||||
return DelegationEvent(reader())
|
||||
|
||||
else:
|
||||
return ReadEvent(fd, bufsize)
|
||||
|
||||
|
||||
def write(fd, data):
|
||||
"""Event: write to a file descriptor asynchronously."""
|
||||
return WriteEvent(fd, data)
|
||||
|
||||
|
||||
def connect(host, port):
|
||||
"""Event: connect to a network address and return a Connection
|
||||
object for communicating on the socket.
|
||||
"""
|
||||
addr = (host, port)
|
||||
sock = socket.create_connection(addr)
|
||||
return ValueEvent(Connection(sock, addr))
|
||||
|
||||
|
||||
def sleep(duration):
|
||||
"""Event: suspend the thread for ``duration`` seconds.
|
||||
"""
|
||||
return SleepEvent(duration)
|
||||
|
||||
|
||||
def join(coro):
|
||||
"""Suspend the thread until another, previously `spawn`ed thread
|
||||
completes.
|
||||
"""
|
||||
return JoinEvent(coro)
|
||||
|
||||
|
||||
def kill(coro):
|
||||
"""Halt the execution of a different `spawn`ed thread.
|
||||
"""
|
||||
return KillEvent(coro)
|
||||
|
||||
|
||||
# Convenience function for running socket servers.
|
||||
|
||||
def server(host, port, func):
|
||||
"""A coroutine that runs a network server. Host and port specify the
|
||||
listening address. func should be a coroutine that takes a single
|
||||
parameter, a Connection object. The coroutine is invoked for every
|
||||
incoming connection on the listening socket.
|
||||
"""
|
||||
def handler(conn):
|
||||
try:
|
||||
yield func(conn)
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
listener = Listener(host, port)
|
||||
try:
|
||||
while True:
|
||||
conn = yield listener.accept()
|
||||
yield spawn(handler(conn))
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
finally:
|
||||
listener.close()
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,40 @@
|
||||
# This file is part of beets.
|
||||
# Copyright 2013, Adrian Sampson.
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining
|
||||
# a copy of this software and associated documentation files (the
|
||||
# "Software"), to deal in the Software without restriction, including
|
||||
# without limitation the rights to use, copy, modify, merge, publish,
|
||||
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||
# permit persons to whom the Software is furnished to do so, subject to
|
||||
# the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
|
||||
from enum import Enum
|
||||
|
||||
|
||||
class OrderedEnum(Enum):
|
||||
"""
|
||||
An Enum subclass that allows comparison of members.
|
||||
"""
|
||||
def __ge__(self, other):
|
||||
if self.__class__ is other.__class__:
|
||||
return self.value >= other.value
|
||||
return NotImplemented
|
||||
|
||||
def __gt__(self, other):
|
||||
if self.__class__ is other.__class__:
|
||||
return self.value > other.value
|
||||
return NotImplemented
|
||||
|
||||
def __le__(self, other):
|
||||
if self.__class__ is other.__class__:
|
||||
return self.value <= other.value
|
||||
return NotImplemented
|
||||
|
||||
def __lt__(self, other):
|
||||
if self.__class__ is other.__class__:
|
||||
return self.value < other.value
|
||||
return NotImplemented
|
||||
@@ -0,0 +1,571 @@
|
||||
# This file is part of beets.
|
||||
# Copyright 2013, Adrian Sampson.
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining
|
||||
# a copy of this software and associated documentation files (the
|
||||
# "Software"), to deal in the Software without restriction, including
|
||||
# without limitation the rights to use, copy, modify, merge, publish,
|
||||
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||
# permit persons to whom the Software is furnished to do so, subject to
|
||||
# the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
|
||||
"""This module implements a string formatter based on the standard PEP
|
||||
292 string.Template class extended with function calls. Variables, as
|
||||
with string.Template, are indicated with $ and functions are delimited
|
||||
with %.
|
||||
|
||||
This module assumes that everything is Unicode: the template and the
|
||||
substitution values. Bytestrings are not supported. Also, the templates
|
||||
always behave like the ``safe_substitute`` method in the standard
|
||||
library: unknown symbols are left intact.
|
||||
|
||||
This is sort of like a tiny, horrible degeneration of a real templating
|
||||
engine like Jinja2 or Mustache.
|
||||
"""
|
||||
from __future__ import print_function
|
||||
|
||||
import re
|
||||
import ast
|
||||
import dis
|
||||
import types
|
||||
|
||||
SYMBOL_DELIM = u'$'
|
||||
FUNC_DELIM = u'%'
|
||||
GROUP_OPEN = u'{'
|
||||
GROUP_CLOSE = u'}'
|
||||
ARG_SEP = u','
|
||||
ESCAPE_CHAR = u'$'
|
||||
|
||||
VARIABLE_PREFIX = '__var_'
|
||||
FUNCTION_PREFIX = '__func_'
|
||||
|
||||
|
||||
class Environment(object):
|
||||
"""Contains the values and functions to be substituted into a
|
||||
template.
|
||||
"""
|
||||
def __init__(self, values, functions):
|
||||
self.values = values
|
||||
self.functions = functions
|
||||
|
||||
|
||||
# Code generation helpers.
|
||||
|
||||
def ex_lvalue(name):
|
||||
"""A variable load expression."""
|
||||
return ast.Name(name, ast.Store())
|
||||
|
||||
|
||||
def ex_rvalue(name):
|
||||
"""A variable store expression."""
|
||||
return ast.Name(name, ast.Load())
|
||||
|
||||
|
||||
def ex_literal(val):
|
||||
"""An int, float, long, bool, string, or None literal with the given
|
||||
value.
|
||||
"""
|
||||
if val is None:
|
||||
return ast.Name('None', ast.Load())
|
||||
elif isinstance(val, (int, float, long)):
|
||||
return ast.Num(val)
|
||||
elif isinstance(val, bool):
|
||||
return ast.Name(str(val), ast.Load())
|
||||
elif isinstance(val, basestring):
|
||||
return ast.Str(val)
|
||||
raise TypeError('no literal for {0}'.format(type(val)))
|
||||
|
||||
|
||||
def ex_varassign(name, expr):
|
||||
"""Assign an expression into a single variable. The expression may
|
||||
either be an `ast.expr` object or a value to be used as a literal.
|
||||
"""
|
||||
if not isinstance(expr, ast.expr):
|
||||
expr = ex_literal(expr)
|
||||
return ast.Assign([ex_lvalue(name)], expr)
|
||||
|
||||
|
||||
def ex_call(func, args):
|
||||
"""A function-call expression with only positional parameters. The
|
||||
function may be an expression or the name of a function. Each
|
||||
argument may be an expression or a value to be used as a literal.
|
||||
"""
|
||||
if isinstance(func, basestring):
|
||||
func = ex_rvalue(func)
|
||||
|
||||
args = list(args)
|
||||
for i in range(len(args)):
|
||||
if not isinstance(args[i], ast.expr):
|
||||
args[i] = ex_literal(args[i])
|
||||
|
||||
return ast.Call(func, args, [], None, None)
|
||||
|
||||
|
||||
def compile_func(arg_names, statements, name='_the_func', debug=False):
|
||||
"""Compile a list of statements as the body of a function and return
|
||||
the resulting Python function. If `debug`, then print out the
|
||||
bytecode of the compiled function.
|
||||
"""
|
||||
func_def = ast.FunctionDef(
|
||||
name,
|
||||
ast.arguments(
|
||||
[ast.Name(n, ast.Param()) for n in arg_names],
|
||||
None, None,
|
||||
[ex_literal(None) for _ in arg_names],
|
||||
),
|
||||
statements,
|
||||
[],
|
||||
)
|
||||
mod = ast.Module([func_def])
|
||||
ast.fix_missing_locations(mod)
|
||||
|
||||
prog = compile(mod, '<generated>', 'exec')
|
||||
|
||||
# Debug: show bytecode.
|
||||
if debug:
|
||||
dis.dis(prog)
|
||||
for const in prog.co_consts:
|
||||
if isinstance(const, types.CodeType):
|
||||
dis.dis(const)
|
||||
|
||||
the_locals = {}
|
||||
exec prog in {}, the_locals
|
||||
return the_locals[name]
|
||||
|
||||
|
||||
# AST nodes for the template language.
|
||||
|
||||
class Symbol(object):
|
||||
"""A variable-substitution symbol in a template."""
|
||||
def __init__(self, ident, original):
|
||||
self.ident = ident
|
||||
self.original = original
|
||||
|
||||
def __repr__(self):
|
||||
return u'Symbol(%s)' % repr(self.ident)
|
||||
|
||||
def evaluate(self, env):
|
||||
"""Evaluate the symbol in the environment, returning a Unicode
|
||||
string.
|
||||
"""
|
||||
if self.ident in env.values:
|
||||
# Substitute for a value.
|
||||
return env.values[self.ident]
|
||||
else:
|
||||
# Keep original text.
|
||||
return self.original
|
||||
|
||||
def translate(self):
|
||||
"""Compile the variable lookup."""
|
||||
expr = ex_rvalue(VARIABLE_PREFIX + self.ident.encode('utf8'))
|
||||
return [expr], set([self.ident.encode('utf8')]), set()
|
||||
|
||||
|
||||
class Call(object):
|
||||
"""A function call in a template."""
|
||||
def __init__(self, ident, args, original):
|
||||
self.ident = ident
|
||||
self.args = args
|
||||
self.original = original
|
||||
|
||||
def __repr__(self):
|
||||
return u'Call(%s, %s, %s)' % (repr(self.ident), repr(self.args),
|
||||
repr(self.original))
|
||||
|
||||
def evaluate(self, env):
|
||||
"""Evaluate the function call in the environment, returning a
|
||||
Unicode string.
|
||||
"""
|
||||
if self.ident in env.functions:
|
||||
arg_vals = [expr.evaluate(env) for expr in self.args]
|
||||
try:
|
||||
out = env.functions[self.ident](*arg_vals)
|
||||
except Exception as exc:
|
||||
# Function raised exception! Maybe inlining the name of
|
||||
# the exception will help debug.
|
||||
return u'<%s>' % unicode(exc)
|
||||
return unicode(out)
|
||||
else:
|
||||
return self.original
|
||||
|
||||
def translate(self):
|
||||
"""Compile the function call."""
|
||||
varnames = set()
|
||||
funcnames = set([self.ident.encode('utf8')])
|
||||
|
||||
arg_exprs = []
|
||||
for arg in self.args:
|
||||
subexprs, subvars, subfuncs = arg.translate()
|
||||
varnames.update(subvars)
|
||||
funcnames.update(subfuncs)
|
||||
|
||||
# Create a subexpression that joins the result components of
|
||||
# the arguments.
|
||||
arg_exprs.append(ex_call(
|
||||
ast.Attribute(ex_literal(u''), 'join', ast.Load()),
|
||||
[ex_call(
|
||||
'map',
|
||||
[
|
||||
ex_rvalue('unicode'),
|
||||
ast.List(subexprs, ast.Load()),
|
||||
]
|
||||
)],
|
||||
))
|
||||
|
||||
subexpr_call = ex_call(
|
||||
FUNCTION_PREFIX + self.ident.encode('utf8'),
|
||||
arg_exprs
|
||||
)
|
||||
return [subexpr_call], varnames, funcnames
|
||||
|
||||
|
||||
class Expression(object):
|
||||
"""Top-level template construct: contains a list of text blobs,
|
||||
Symbols, and Calls.
|
||||
"""
|
||||
def __init__(self, parts):
|
||||
self.parts = parts
|
||||
|
||||
def __repr__(self):
|
||||
return u'Expression(%s)' % (repr(self.parts))
|
||||
|
||||
def evaluate(self, env):
|
||||
"""Evaluate the entire expression in the environment, returning
|
||||
a Unicode string.
|
||||
"""
|
||||
out = []
|
||||
for part in self.parts:
|
||||
if isinstance(part, basestring):
|
||||
out.append(part)
|
||||
else:
|
||||
out.append(part.evaluate(env))
|
||||
return u''.join(map(unicode, out))
|
||||
|
||||
def translate(self):
|
||||
"""Compile the expression to a list of Python AST expressions, a
|
||||
set of variable names used, and a set of function names.
|
||||
"""
|
||||
expressions = []
|
||||
varnames = set()
|
||||
funcnames = set()
|
||||
for part in self.parts:
|
||||
if isinstance(part, basestring):
|
||||
expressions.append(ex_literal(part))
|
||||
else:
|
||||
e, v, f = part.translate()
|
||||
expressions.extend(e)
|
||||
varnames.update(v)
|
||||
funcnames.update(f)
|
||||
return expressions, varnames, funcnames
|
||||
|
||||
|
||||
# Parser.
|
||||
|
||||
class ParseError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class Parser(object):
|
||||
"""Parses a template expression string. Instantiate the class with
|
||||
the template source and call ``parse_expression``. The ``pos`` field
|
||||
will indicate the character after the expression finished and
|
||||
``parts`` will contain a list of Unicode strings, Symbols, and Calls
|
||||
reflecting the concatenated portions of the expression.
|
||||
|
||||
This is a terrible, ad-hoc parser implementation based on a
|
||||
left-to-right scan with no lexing step to speak of; it's probably
|
||||
both inefficient and incorrect. Maybe this should eventually be
|
||||
replaced with a real, accepted parsing technique (PEG, parser
|
||||
generator, etc.).
|
||||
"""
|
||||
def __init__(self, string):
|
||||
self.string = string
|
||||
self.pos = 0
|
||||
self.parts = []
|
||||
|
||||
# Common parsing resources.
|
||||
special_chars = (SYMBOL_DELIM, FUNC_DELIM, GROUP_OPEN, GROUP_CLOSE,
|
||||
ARG_SEP, ESCAPE_CHAR)
|
||||
special_char_re = re.compile(ur'[%s]|$' %
|
||||
u''.join(re.escape(c) for c in special_chars))
|
||||
|
||||
def parse_expression(self):
|
||||
"""Parse a template expression starting at ``pos``. Resulting
|
||||
components (Unicode strings, Symbols, and Calls) are added to
|
||||
the ``parts`` field, a list. The ``pos`` field is updated to be
|
||||
the next character after the expression.
|
||||
"""
|
||||
text_parts = []
|
||||
|
||||
while self.pos < len(self.string):
|
||||
char = self.string[self.pos]
|
||||
|
||||
if char not in self.special_chars:
|
||||
# A non-special character. Skip to the next special
|
||||
# character, treating the interstice as literal text.
|
||||
next_pos = (
|
||||
self.special_char_re.search(self.string[self.pos:]).start()
|
||||
+ self.pos
|
||||
)
|
||||
text_parts.append(self.string[self.pos:next_pos])
|
||||
self.pos = next_pos
|
||||
continue
|
||||
|
||||
if self.pos == len(self.string) - 1:
|
||||
# The last character can never begin a structure, so we
|
||||
# just interpret it as a literal character (unless it
|
||||
# terminates the expression, as with , and }).
|
||||
if char not in (GROUP_CLOSE, ARG_SEP):
|
||||
text_parts.append(char)
|
||||
self.pos += 1
|
||||
break
|
||||
|
||||
next_char = self.string[self.pos + 1]
|
||||
if char == ESCAPE_CHAR and next_char in \
|
||||
(SYMBOL_DELIM, FUNC_DELIM, GROUP_CLOSE, ARG_SEP):
|
||||
# An escaped special character ($$, $}, etc.). Note that
|
||||
# ${ is not an escape sequence: this is ambiguous with
|
||||
# the start of a symbol and it's not necessary (just
|
||||
# using { suffices in all cases).
|
||||
text_parts.append(next_char)
|
||||
self.pos += 2 # Skip the next character.
|
||||
continue
|
||||
|
||||
# Shift all characters collected so far into a single string.
|
||||
if text_parts:
|
||||
self.parts.append(u''.join(text_parts))
|
||||
text_parts = []
|
||||
|
||||
if char == SYMBOL_DELIM:
|
||||
# Parse a symbol.
|
||||
self.parse_symbol()
|
||||
elif char == FUNC_DELIM:
|
||||
# Parse a function call.
|
||||
self.parse_call()
|
||||
elif char in (GROUP_CLOSE, ARG_SEP):
|
||||
# Template terminated.
|
||||
break
|
||||
elif char == GROUP_OPEN:
|
||||
# Start of a group has no meaning hear; just pass
|
||||
# through the character.
|
||||
text_parts.append(char)
|
||||
self.pos += 1
|
||||
else:
|
||||
assert False
|
||||
|
||||
# If any parsed characters remain, shift them into a string.
|
||||
if text_parts:
|
||||
self.parts.append(u''.join(text_parts))
|
||||
|
||||
def parse_symbol(self):
|
||||
"""Parse a variable reference (like ``$foo`` or ``${foo}``)
|
||||
starting at ``pos``. Possibly appends a Symbol object (or,
|
||||
failing that, text) to the ``parts`` field and updates ``pos``.
|
||||
The character at ``pos`` must, as a precondition, be ``$``.
|
||||
"""
|
||||
assert self.pos < len(self.string)
|
||||
assert self.string[self.pos] == SYMBOL_DELIM
|
||||
|
||||
if self.pos == len(self.string) - 1:
|
||||
# Last character.
|
||||
self.parts.append(SYMBOL_DELIM)
|
||||
self.pos += 1
|
||||
return
|
||||
|
||||
next_char = self.string[self.pos + 1]
|
||||
start_pos = self.pos
|
||||
self.pos += 1
|
||||
|
||||
if next_char == GROUP_OPEN:
|
||||
# A symbol like ${this}.
|
||||
self.pos += 1 # Skip opening.
|
||||
closer = self.string.find(GROUP_CLOSE, self.pos)
|
||||
if closer == -1 or closer == self.pos:
|
||||
# No closing brace found or identifier is empty.
|
||||
self.parts.append(self.string[start_pos:self.pos])
|
||||
else:
|
||||
# Closer found.
|
||||
ident = self.string[self.pos:closer]
|
||||
self.pos = closer + 1
|
||||
self.parts.append(Symbol(ident,
|
||||
self.string[start_pos:self.pos]))
|
||||
|
||||
else:
|
||||
# A bare-word symbol.
|
||||
ident = self._parse_ident()
|
||||
if ident:
|
||||
# Found a real symbol.
|
||||
self.parts.append(Symbol(ident,
|
||||
self.string[start_pos:self.pos]))
|
||||
else:
|
||||
# A standalone $.
|
||||
self.parts.append(SYMBOL_DELIM)
|
||||
|
||||
def parse_call(self):
|
||||
"""Parse a function call (like ``%foo{bar,baz}``) starting at
|
||||
``pos``. Possibly appends a Call object to ``parts`` and update
|
||||
``pos``. The character at ``pos`` must be ``%``.
|
||||
"""
|
||||
assert self.pos < len(self.string)
|
||||
assert self.string[self.pos] == FUNC_DELIM
|
||||
|
||||
start_pos = self.pos
|
||||
self.pos += 1
|
||||
|
||||
ident = self._parse_ident()
|
||||
if not ident:
|
||||
# No function name.
|
||||
self.parts.append(FUNC_DELIM)
|
||||
return
|
||||
|
||||
if self.pos >= len(self.string):
|
||||
# Identifier terminates string.
|
||||
self.parts.append(self.string[start_pos:self.pos])
|
||||
return
|
||||
|
||||
if self.string[self.pos] != GROUP_OPEN:
|
||||
# Argument list not opened.
|
||||
self.parts.append(self.string[start_pos:self.pos])
|
||||
return
|
||||
|
||||
# Skip past opening brace and try to parse an argument list.
|
||||
self.pos += 1
|
||||
args = self.parse_argument_list()
|
||||
if self.pos >= len(self.string) or \
|
||||
self.string[self.pos] != GROUP_CLOSE:
|
||||
# Arguments unclosed.
|
||||
self.parts.append(self.string[start_pos:self.pos])
|
||||
return
|
||||
|
||||
self.pos += 1 # Move past closing brace.
|
||||
self.parts.append(Call(ident, args, self.string[start_pos:self.pos]))
|
||||
|
||||
def parse_argument_list(self):
|
||||
"""Parse a list of arguments starting at ``pos``, returning a
|
||||
list of Expression objects. Does not modify ``parts``. Should
|
||||
leave ``pos`` pointing to a } character or the end of the
|
||||
string.
|
||||
"""
|
||||
# Try to parse a subexpression in a subparser.
|
||||
expressions = []
|
||||
|
||||
while self.pos < len(self.string):
|
||||
subparser = Parser(self.string[self.pos:])
|
||||
subparser.parse_expression()
|
||||
|
||||
# Extract and advance past the parsed expression.
|
||||
expressions.append(Expression(subparser.parts))
|
||||
self.pos += subparser.pos
|
||||
|
||||
if self.pos >= len(self.string) or \
|
||||
self.string[self.pos] == GROUP_CLOSE:
|
||||
# Argument list terminated by EOF or closing brace.
|
||||
break
|
||||
|
||||
# Only other way to terminate an expression is with ,.
|
||||
# Continue to the next argument.
|
||||
assert self.string[self.pos] == ARG_SEP
|
||||
self.pos += 1
|
||||
|
||||
return expressions
|
||||
|
||||
def _parse_ident(self):
|
||||
"""Parse an identifier and return it (possibly an empty string).
|
||||
Updates ``pos``.
|
||||
"""
|
||||
remainder = self.string[self.pos:]
|
||||
ident = re.match(ur'\w*', remainder).group(0)
|
||||
self.pos += len(ident)
|
||||
return ident
|
||||
|
||||
|
||||
def _parse(template):
|
||||
"""Parse a top-level template string Expression. Any extraneous text
|
||||
is considered literal text.
|
||||
"""
|
||||
parser = Parser(template)
|
||||
parser.parse_expression()
|
||||
|
||||
parts = parser.parts
|
||||
remainder = parser.string[parser.pos:]
|
||||
if remainder:
|
||||
parts.append(remainder)
|
||||
return Expression(parts)
|
||||
|
||||
|
||||
# External interface.
|
||||
|
||||
class Template(object):
|
||||
"""A string template, including text, Symbols, and Calls.
|
||||
"""
|
||||
def __init__(self, template):
|
||||
self.expr = _parse(template)
|
||||
self.original = template
|
||||
self.compiled = self.translate()
|
||||
|
||||
def __eq__(self, other):
|
||||
return self.original == other.original
|
||||
|
||||
def interpret(self, values={}, functions={}):
|
||||
"""Like `substitute`, but forces the interpreter (rather than
|
||||
the compiled version) to be used. The interpreter includes
|
||||
exception-handling code for missing variables and buggy template
|
||||
functions but is much slower.
|
||||
"""
|
||||
return self.expr.evaluate(Environment(values, functions))
|
||||
|
||||
def substitute(self, values={}, functions={}):
|
||||
"""Evaluate the template given the values and functions.
|
||||
"""
|
||||
try:
|
||||
res = self.compiled(values, functions)
|
||||
except: # Handle any exceptions thrown by compiled version.
|
||||
res = self.interpret(values, functions)
|
||||
return res
|
||||
|
||||
def translate(self):
|
||||
"""Compile the template to a Python function."""
|
||||
expressions, varnames, funcnames = self.expr.translate()
|
||||
|
||||
argnames = []
|
||||
for varname in varnames:
|
||||
argnames.append(VARIABLE_PREFIX.encode('utf8') + varname)
|
||||
for funcname in funcnames:
|
||||
argnames.append(FUNCTION_PREFIX.encode('utf8') + funcname)
|
||||
|
||||
func = compile_func(
|
||||
argnames,
|
||||
[ast.Return(ast.List(expressions, ast.Load()))],
|
||||
)
|
||||
|
||||
def wrapper_func(values={}, functions={}):
|
||||
args = {}
|
||||
for varname in varnames:
|
||||
args[VARIABLE_PREFIX + varname] = values[varname]
|
||||
for funcname in funcnames:
|
||||
args[FUNCTION_PREFIX + funcname] = functions[funcname]
|
||||
parts = func(**args)
|
||||
return u''.join(parts)
|
||||
|
||||
return wrapper_func
|
||||
|
||||
|
||||
# Performance tests.
|
||||
|
||||
if __name__ == '__main__':
|
||||
import timeit
|
||||
_tmpl = Template(u'foo $bar %baz{foozle $bar barzle} $bar')
|
||||
_vars = {'bar': 'qux'}
|
||||
_funcs = {'baz': unicode.upper}
|
||||
interp_time = timeit.timeit('_tmpl.interpret(_vars, _funcs)',
|
||||
'from __main__ import _tmpl, _vars, _funcs',
|
||||
number=10000)
|
||||
print(interp_time)
|
||||
comp_time = timeit.timeit('_tmpl.substitute(_vars, _funcs)',
|
||||
'from __main__ import _tmpl, _vars, _funcs',
|
||||
number=10000)
|
||||
print(comp_time)
|
||||
print('Speedup:', interp_time / comp_time)
|
||||
@@ -0,0 +1,517 @@
|
||||
# This file is part of beets.
|
||||
# Copyright 2013, Adrian Sampson.
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining
|
||||
# a copy of this software and associated documentation files (the
|
||||
# "Software"), to deal in the Software without restriction, including
|
||||
# without limitation the rights to use, copy, modify, merge, publish,
|
||||
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||
# permit persons to whom the Software is furnished to do so, subject to
|
||||
# the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
|
||||
"""Simple but robust implementation of generator/coroutine-based
|
||||
pipelines in Python. The pipelines may be run either sequentially
|
||||
(single-threaded) or in parallel (one thread per pipeline stage).
|
||||
|
||||
This implementation supports pipeline bubbles (indications that the
|
||||
processing for a certain item should abort). To use them, yield the
|
||||
BUBBLE constant from any stage coroutine except the last.
|
||||
|
||||
In the parallel case, the implementation transparently handles thread
|
||||
shutdown when the processing is complete and when a stage raises an
|
||||
exception. KeyboardInterrupts (^C) are also handled.
|
||||
|
||||
When running a parallel pipeline, it is also possible to use
|
||||
multiple coroutines for the same pipeline stage; this lets you speed
|
||||
up a bottleneck stage by dividing its work among multiple threads.
|
||||
To do so, pass an iterable of coroutines to the Pipeline constructor
|
||||
in place of any single coroutine.
|
||||
"""
|
||||
from __future__ import print_function
|
||||
|
||||
import Queue
|
||||
from threading import Thread, Lock
|
||||
import sys
|
||||
|
||||
BUBBLE = '__PIPELINE_BUBBLE__'
|
||||
POISON = '__PIPELINE_POISON__'
|
||||
|
||||
DEFAULT_QUEUE_SIZE = 16
|
||||
|
||||
|
||||
def _invalidate_queue(q, val=None, sync=True):
|
||||
"""Breaks a Queue such that it never blocks, always has size 1,
|
||||
and has no maximum size. get()ing from the queue returns `val`,
|
||||
which defaults to None. `sync` controls whether a lock is
|
||||
required (because it's not reentrant!).
|
||||
"""
|
||||
def _qsize(len=len):
|
||||
return 1
|
||||
|
||||
def _put(item):
|
||||
pass
|
||||
|
||||
def _get():
|
||||
return val
|
||||
|
||||
if sync:
|
||||
q.mutex.acquire()
|
||||
|
||||
try:
|
||||
q.maxsize = 0
|
||||
q._qsize = _qsize
|
||||
q._put = _put
|
||||
q._get = _get
|
||||
q.not_empty.notifyAll()
|
||||
q.not_full.notifyAll()
|
||||
|
||||
finally:
|
||||
if sync:
|
||||
q.mutex.release()
|
||||
|
||||
|
||||
class CountedQueue(Queue.Queue):
|
||||
"""A queue that keeps track of the number of threads that are
|
||||
still feeding into it. The queue is poisoned when all threads are
|
||||
finished with the queue.
|
||||
"""
|
||||
def __init__(self, maxsize=0):
|
||||
Queue.Queue.__init__(self, maxsize)
|
||||
self.nthreads = 0
|
||||
self.poisoned = False
|
||||
|
||||
def acquire(self):
|
||||
"""Indicate that a thread will start putting into this queue.
|
||||
Should not be called after the queue is already poisoned.
|
||||
"""
|
||||
with self.mutex:
|
||||
assert not self.poisoned
|
||||
assert self.nthreads >= 0
|
||||
self.nthreads += 1
|
||||
|
||||
def release(self):
|
||||
"""Indicate that a thread that was putting into this queue has
|
||||
exited. If this is the last thread using the queue, the queue
|
||||
is poisoned.
|
||||
"""
|
||||
with self.mutex:
|
||||
self.nthreads -= 1
|
||||
assert self.nthreads >= 0
|
||||
if self.nthreads == 0:
|
||||
# All threads are done adding to this queue. Poison it
|
||||
# when it becomes empty.
|
||||
self.poisoned = True
|
||||
|
||||
# Replacement _get invalidates when no items remain.
|
||||
_old_get = self._get
|
||||
|
||||
def _get():
|
||||
out = _old_get()
|
||||
if not self.queue:
|
||||
_invalidate_queue(self, POISON, False)
|
||||
return out
|
||||
|
||||
if self.queue:
|
||||
# Items remain.
|
||||
self._get = _get
|
||||
else:
|
||||
# No items. Invalidate immediately.
|
||||
_invalidate_queue(self, POISON, False)
|
||||
|
||||
|
||||
class MultiMessage(object):
|
||||
"""A message yielded by a pipeline stage encapsulating multiple
|
||||
values to be sent to the next stage.
|
||||
"""
|
||||
def __init__(self, messages):
|
||||
self.messages = messages
|
||||
|
||||
|
||||
def multiple(messages):
|
||||
"""Yield multiple([message, ..]) from a pipeline stage to send
|
||||
multiple values to the next pipeline stage.
|
||||
"""
|
||||
return MultiMessage(messages)
|
||||
|
||||
|
||||
def stage(func):
|
||||
"""Decorate a function to become a simple stage.
|
||||
|
||||
>>> @stage
|
||||
... def add(n, i):
|
||||
... return i + n
|
||||
>>> pipe = Pipeline([
|
||||
... iter([1, 2, 3]),
|
||||
... add(2),
|
||||
... ])
|
||||
>>> list(pipe.pull())
|
||||
[3, 4, 5]
|
||||
"""
|
||||
|
||||
def coro(*args):
|
||||
task = None
|
||||
while True:
|
||||
task = yield task
|
||||
task = func(*(args + (task,)))
|
||||
return coro
|
||||
|
||||
|
||||
def mutator_stage(func):
|
||||
"""Decorate a function that manipulates items in a coroutine to
|
||||
become a simple stage.
|
||||
|
||||
>>> @mutator_stage
|
||||
... def setkey(key, item):
|
||||
... item[key] = True
|
||||
>>> pipe = Pipeline([
|
||||
... iter([{'x': False}, {'a': False}]),
|
||||
... setkey('x'),
|
||||
... ])
|
||||
>>> list(pipe.pull())
|
||||
[{'x': True}, {'a': False, 'x': True}]
|
||||
"""
|
||||
|
||||
def coro(*args):
|
||||
task = None
|
||||
while True:
|
||||
task = yield task
|
||||
func(*(args + (task,)))
|
||||
return coro
|
||||
|
||||
|
||||
def _allmsgs(obj):
|
||||
"""Returns a list of all the messages encapsulated in obj. If obj
|
||||
is a MultiMessage, returns its enclosed messages. If obj is BUBBLE,
|
||||
returns an empty list. Otherwise, returns a list containing obj.
|
||||
"""
|
||||
if isinstance(obj, MultiMessage):
|
||||
return obj.messages
|
||||
elif obj == BUBBLE:
|
||||
return []
|
||||
else:
|
||||
return [obj]
|
||||
|
||||
|
||||
class PipelineThread(Thread):
|
||||
"""Abstract base class for pipeline-stage threads."""
|
||||
def __init__(self, all_threads):
|
||||
super(PipelineThread, self).__init__()
|
||||
self.abort_lock = Lock()
|
||||
self.abort_flag = False
|
||||
self.all_threads = all_threads
|
||||
self.exc_info = None
|
||||
|
||||
def abort(self):
|
||||
"""Shut down the thread at the next chance possible.
|
||||
"""
|
||||
with self.abort_lock:
|
||||
self.abort_flag = True
|
||||
|
||||
# Ensure that we are not blocking on a queue read or write.
|
||||
if hasattr(self, 'in_queue'):
|
||||
_invalidate_queue(self.in_queue, POISON)
|
||||
if hasattr(self, 'out_queue'):
|
||||
_invalidate_queue(self.out_queue, POISON)
|
||||
|
||||
def abort_all(self, exc_info):
|
||||
"""Abort all other threads in the system for an exception.
|
||||
"""
|
||||
self.exc_info = exc_info
|
||||
for thread in self.all_threads:
|
||||
thread.abort()
|
||||
|
||||
|
||||
class FirstPipelineThread(PipelineThread):
|
||||
"""The thread running the first stage in a parallel pipeline setup.
|
||||
The coroutine should just be a generator.
|
||||
"""
|
||||
def __init__(self, coro, out_queue, all_threads):
|
||||
super(FirstPipelineThread, self).__init__(all_threads)
|
||||
self.coro = coro
|
||||
self.out_queue = out_queue
|
||||
self.out_queue.acquire()
|
||||
|
||||
self.abort_lock = Lock()
|
||||
self.abort_flag = False
|
||||
|
||||
def run(self):
|
||||
try:
|
||||
while True:
|
||||
with self.abort_lock:
|
||||
if self.abort_flag:
|
||||
return
|
||||
|
||||
# Get the value from the generator.
|
||||
try:
|
||||
msg = self.coro.next()
|
||||
except StopIteration:
|
||||
break
|
||||
|
||||
# Send messages to the next stage.
|
||||
for msg in _allmsgs(msg):
|
||||
with self.abort_lock:
|
||||
if self.abort_flag:
|
||||
return
|
||||
self.out_queue.put(msg)
|
||||
|
||||
except:
|
||||
self.abort_all(sys.exc_info())
|
||||
return
|
||||
|
||||
# Generator finished; shut down the pipeline.
|
||||
self.out_queue.release()
|
||||
|
||||
|
||||
class MiddlePipelineThread(PipelineThread):
|
||||
"""A thread running any stage in the pipeline except the first or
|
||||
last.
|
||||
"""
|
||||
def __init__(self, coro, in_queue, out_queue, all_threads):
|
||||
super(MiddlePipelineThread, self).__init__(all_threads)
|
||||
self.coro = coro
|
||||
self.in_queue = in_queue
|
||||
self.out_queue = out_queue
|
||||
self.out_queue.acquire()
|
||||
|
||||
def run(self):
|
||||
try:
|
||||
# Prime the coroutine.
|
||||
self.coro.next()
|
||||
|
||||
while True:
|
||||
with self.abort_lock:
|
||||
if self.abort_flag:
|
||||
return
|
||||
|
||||
# Get the message from the previous stage.
|
||||
msg = self.in_queue.get()
|
||||
if msg is POISON:
|
||||
break
|
||||
|
||||
with self.abort_lock:
|
||||
if self.abort_flag:
|
||||
return
|
||||
|
||||
# Invoke the current stage.
|
||||
out = self.coro.send(msg)
|
||||
|
||||
# Send messages to next stage.
|
||||
for msg in _allmsgs(out):
|
||||
with self.abort_lock:
|
||||
if self.abort_flag:
|
||||
return
|
||||
self.out_queue.put(msg)
|
||||
|
||||
except:
|
||||
self.abort_all(sys.exc_info())
|
||||
return
|
||||
|
||||
# Pipeline is shutting down normally.
|
||||
self.out_queue.release()
|
||||
|
||||
|
||||
class LastPipelineThread(PipelineThread):
|
||||
"""A thread running the last stage in a pipeline. The coroutine
|
||||
should yield nothing.
|
||||
"""
|
||||
def __init__(self, coro, in_queue, all_threads):
|
||||
super(LastPipelineThread, self).__init__(all_threads)
|
||||
self.coro = coro
|
||||
self.in_queue = in_queue
|
||||
|
||||
def run(self):
|
||||
# Prime the coroutine.
|
||||
self.coro.next()
|
||||
|
||||
try:
|
||||
while True:
|
||||
with self.abort_lock:
|
||||
if self.abort_flag:
|
||||
return
|
||||
|
||||
# Get the message from the previous stage.
|
||||
msg = self.in_queue.get()
|
||||
if msg is POISON:
|
||||
break
|
||||
|
||||
with self.abort_lock:
|
||||
if self.abort_flag:
|
||||
return
|
||||
|
||||
# Send to consumer.
|
||||
self.coro.send(msg)
|
||||
|
||||
except:
|
||||
self.abort_all(sys.exc_info())
|
||||
return
|
||||
|
||||
|
||||
class Pipeline(object):
|
||||
"""Represents a staged pattern of work. Each stage in the pipeline
|
||||
is a coroutine that receives messages from the previous stage and
|
||||
yields messages to be sent to the next stage.
|
||||
"""
|
||||
def __init__(self, stages):
|
||||
"""Makes a new pipeline from a list of coroutines. There must
|
||||
be at least two stages.
|
||||
"""
|
||||
if len(stages) < 2:
|
||||
raise ValueError('pipeline must have at least two stages')
|
||||
self.stages = []
|
||||
for stage in stages:
|
||||
if isinstance(stage, (list, tuple)):
|
||||
self.stages.append(stage)
|
||||
else:
|
||||
# Default to one thread per stage.
|
||||
self.stages.append((stage,))
|
||||
|
||||
def run_sequential(self):
|
||||
"""Run the pipeline sequentially in the current thread. The
|
||||
stages are run one after the other. Only the first coroutine
|
||||
in each stage is used.
|
||||
"""
|
||||
list(self.pull())
|
||||
|
||||
def run_parallel(self, queue_size=DEFAULT_QUEUE_SIZE):
|
||||
"""Run the pipeline in parallel using one thread per stage. The
|
||||
messages between the stages are stored in queues of the given
|
||||
size.
|
||||
"""
|
||||
queue_count = len(self.stages) - 1
|
||||
queues = [CountedQueue(queue_size) for i in range(queue_count)]
|
||||
threads = []
|
||||
|
||||
# Set up first stage.
|
||||
for coro in self.stages[0]:
|
||||
threads.append(FirstPipelineThread(coro, queues[0], threads))
|
||||
|
||||
# Middle stages.
|
||||
for i in range(1, queue_count):
|
||||
for coro in self.stages[i]:
|
||||
threads.append(MiddlePipelineThread(
|
||||
coro, queues[i - 1], queues[i], threads
|
||||
))
|
||||
|
||||
# Last stage.
|
||||
for coro in self.stages[-1]:
|
||||
threads.append(
|
||||
LastPipelineThread(coro, queues[-1], threads)
|
||||
)
|
||||
|
||||
# Start threads.
|
||||
for thread in threads:
|
||||
thread.start()
|
||||
|
||||
# Wait for termination. The final thread lasts the longest.
|
||||
try:
|
||||
# Using a timeout allows us to receive KeyboardInterrupt
|
||||
# exceptions during the join().
|
||||
while threads[-1].isAlive():
|
||||
threads[-1].join(1)
|
||||
|
||||
except:
|
||||
# Stop all the threads immediately.
|
||||
for thread in threads:
|
||||
thread.abort()
|
||||
raise
|
||||
|
||||
finally:
|
||||
# Make completely sure that all the threads have finished
|
||||
# before we return. They should already be either finished,
|
||||
# in normal operation, or aborted, in case of an exception.
|
||||
for thread in threads[:-1]:
|
||||
thread.join()
|
||||
|
||||
for thread in threads:
|
||||
exc_info = thread.exc_info
|
||||
if exc_info:
|
||||
# Make the exception appear as it was raised originally.
|
||||
raise exc_info[0], exc_info[1], exc_info[2]
|
||||
|
||||
def pull(self):
|
||||
"""Yield elements from the end of the pipeline. Runs the stages
|
||||
sequentially until the last yields some messages. Each of the messages
|
||||
is then yielded by ``pulled.next()``. If the pipeline has a consumer,
|
||||
that is the last stage does not yield any messages, then pull will not
|
||||
yield any messages. Only the first coroutine in each stage is used
|
||||
"""
|
||||
coros = [stage[0] for stage in self.stages]
|
||||
|
||||
# "Prime" the coroutines.
|
||||
for coro in coros[1:]:
|
||||
coro.next()
|
||||
|
||||
# Begin the pipeline.
|
||||
for out in coros[0]:
|
||||
msgs = _allmsgs(out)
|
||||
for coro in coros[1:]:
|
||||
next_msgs = []
|
||||
for msg in msgs:
|
||||
out = coro.send(msg)
|
||||
next_msgs.extend(_allmsgs(out))
|
||||
msgs = next_msgs
|
||||
for msg in msgs:
|
||||
yield msg
|
||||
|
||||
# Smoke test.
|
||||
if __name__ == '__main__':
|
||||
import time
|
||||
|
||||
# Test a normally-terminating pipeline both in sequence and
|
||||
# in parallel.
|
||||
def produce():
|
||||
for i in range(5):
|
||||
print('generating %i' % i)
|
||||
time.sleep(1)
|
||||
yield i
|
||||
|
||||
def work():
|
||||
num = yield
|
||||
while True:
|
||||
print('processing %i' % num)
|
||||
time.sleep(2)
|
||||
num = yield num * 2
|
||||
|
||||
def consume():
|
||||
while True:
|
||||
num = yield
|
||||
time.sleep(1)
|
||||
print('received %i' % num)
|
||||
|
||||
ts_start = time.time()
|
||||
Pipeline([produce(), work(), consume()]).run_sequential()
|
||||
ts_seq = time.time()
|
||||
Pipeline([produce(), work(), consume()]).run_parallel()
|
||||
ts_par = time.time()
|
||||
Pipeline([produce(), (work(), work()), consume()]).run_parallel()
|
||||
ts_end = time.time()
|
||||
print('Sequential time:', ts_seq - ts_start)
|
||||
print('Parallel time:', ts_par - ts_seq)
|
||||
print('Multiply-parallel time:', ts_end - ts_par)
|
||||
print()
|
||||
|
||||
# Test a pipeline that raises an exception.
|
||||
def exc_produce():
|
||||
for i in range(10):
|
||||
print('generating %i' % i)
|
||||
time.sleep(1)
|
||||
yield i
|
||||
|
||||
def exc_work():
|
||||
num = yield
|
||||
while True:
|
||||
print('processing %i' % num)
|
||||
time.sleep(3)
|
||||
if num == 3:
|
||||
raise Exception()
|
||||
num = yield num * 2
|
||||
|
||||
def exc_consume():
|
||||
while True:
|
||||
num = yield
|
||||
print('received %i' % num)
|
||||
|
||||
Pipeline([exc_produce(), exc_work(), exc_consume()]).run_parallel(1)
|
||||
@@ -0,0 +1,50 @@
|
||||
# This file is part of beets.
|
||||
# Copyright 2013, Adrian Sampson.
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining
|
||||
# a copy of this software and associated documentation files (the
|
||||
# "Software"), to deal in the Software without restriction, including
|
||||
# without limitation the rights to use, copy, modify, merge, publish,
|
||||
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||
# permit persons to whom the Software is furnished to do so, subject to
|
||||
# the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
|
||||
"""A simple utility for constructing filesystem-like trees from beets
|
||||
libraries.
|
||||
"""
|
||||
from collections import namedtuple
|
||||
from beets import util
|
||||
|
||||
Node = namedtuple('Node', ['files', 'dirs'])
|
||||
|
||||
|
||||
def _insert(node, path, itemid):
|
||||
"""Insert an item into a virtual filesystem node."""
|
||||
if len(path) == 1:
|
||||
# Last component. Insert file.
|
||||
node.files[path[0]] = itemid
|
||||
else:
|
||||
# In a directory.
|
||||
dirname = path[0]
|
||||
rest = path[1:]
|
||||
if dirname not in node.dirs:
|
||||
node.dirs[dirname] = Node({}, {})
|
||||
_insert(node.dirs[dirname], rest, itemid)
|
||||
|
||||
|
||||
def libtree(lib):
|
||||
"""Generates a filesystem-like directory tree for the files
|
||||
contained in `lib`. Filesystem nodes are (files, dirs) named
|
||||
tuples in which both components are dictionaries. The first
|
||||
maps filenames to Item ids. The second maps directory names to
|
||||
child node tuples.
|
||||
"""
|
||||
root = Node({}, {})
|
||||
for item in lib.items():
|
||||
dest = item.destination(fragment=True)
|
||||
parts = util.components(dest)
|
||||
_insert(root, parts, item.id)
|
||||
return root
|
||||
Reference in New Issue
Block a user