Set macOS encoder multiprocessing method to 'fork'

default for macOS python3 is 'spawn' which is not working with the current code
Growl removed
2026-09-10 08:41:41 +01:00 · 2025-08-08 17:28:21 +12:00 · 2025-08-06 16:01:52 +12:00 · 2025-08-06 15:52:34 +12:00 · 2025-08-06 15:42:39 +12:00 · 2025-08-03 21:14:24 +12:00
505 changed files with 50775 additions and 37323 deletions
@@ -1,5 +1,31 @@
 # Changelog

+## v0.6.3
+ Released 26 May 2024
+
+Highlights:
+* Hotfix for searcher not returning results
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.6.2...v0.6.3).
+
+## v0.6.2
+ Released 26 May 2024
+
+Highlights:
+* Added soulseek support
+* Added bandcamp support
+* Changes and dependency updates to work with Python >= 3.12
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.6.1...v0.6.2).
+
+## v0.6.1
+R eleased 26 November 2023
+
+Highlights:
+* Dependency updates to work with > Python 3.11
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.6.0...v0.6.1).
+
 ## v0.6.0
 Released 13 November 2022

@@ -310,6 +310,16 @@
                                    <input type="text" name="usenet_retention" value="${config['usenet_retention']}" size="5">
                                </div>
                            </fieldset>
+                            <fieldset title="Method for downloading Bandcamp.com files.">
+                                <legend>Bandcamp</legend>
+                                <div class="row">
+                                    <label title="Path to folder where Headphones can store raw downloads from Bandcamp.com.">
+                                        Bandcamp Directory
+                                    </label>
+                                    <input type="text" name="bandcamp_dir" value="${config['bandcamp_dir']}" size="50">
+                                    <small>Full path where raw MP3s will be stored, e.g. /Users/name/Downloads/bandcamp</small>
+                                </div>
+                            </fieldset>
                        </td>
                        <td>
                            <fieldset title="Method for downloading torrent files.">
@@ -317,7 +327,7 @@
                                <input type="radio" name="torrent_downloader" id="torrent_downloader_blackhole" value="0" ${config['torrent_downloader_blackhole']}> Black Hole
                                <input type="radio" name="torrent_downloader" id="torrent_downloader_transmission" value="1" ${config['torrent_downloader_transmission']}> Transmission
                                <input type="radio" name="torrent_downloader" id="torrent_downloader_utorrent" value="2" ${config['torrent_downloader_utorrent']}> uTorrent (Beta)
-                                <input type="radio" name="torrent_downloader" id="torrent_downloader_deluge" value="3" ${config['torrent_downloader_deluge']}> Deluge (Beta)
+                                <input type="radio" name="torrent_downloader" id="torrent_downloader_deluge" value="3" ${config['torrent_downloader_deluge']}> Deluge
                                <input type="radio" name="torrent_downloader" id="torrent_downloader_qbittorrent" value="4" ${config['torrent_downloader_qbittorrent']}> QBitTorrent
                            </fieldset>
                            <fieldset id="torrent_blackhole_options">
@@ -438,6 +448,11 @@
                                    <input type="text" name="deluge_label" value="${config['deluge_label']}" size="30">
                                    <small>Labels shouldn't contain spaces (requires Label plugin)</small>
                                </div>
+                                <div class="row">
+                                    <label>Download Directory</label>
+                                    <input type="text" name="deluge_download_directory" value="${config['deluge_download_directory']}" size="30">
+                                    <small>Directory where Deluge should download to</small>
+                                </div>
                                <div class="row">
                                    <label>Move When Completed</label>
                                    <input type="text" name="deluge_done_directory" value="${config['deluge_done_directory']}" size="30">
@@ -467,7 +482,33 @@
                                    <label>Prefer</label>
                                    <input type="radio" name="prefer_torrents" id="prefer_torrents_0" value="0" ${config['prefer_torrents_0']}>NZBs
                                    <input type="radio" name="prefer_torrents" id="prefer_torrents_1" value="1" ${config['prefer_torrents_1']}>Torrents
-                                    <input type="radio" name="prefer_torrents" id="prefer_torrents_2" value="2" ${config['prefer_torrents_2']}>No Preference
+                                    <input type="radio" name="prefer_torrents" id="prefer_torrents_2" value="2" ${config['prefer_torrents_2']}>Soulseek
+                                    <input type="radio" name="prefer_torrents" id="prefer_torrents_3" value="3" ${config['prefer_torrents_3']}>No Preference
+                                </div>
+                            </fieldset>
+                        </td>
+                        <td>
+                            <fieldset>
+                                <legend>Soulseek</legend>
+                                <div class="row">
+                                    <label>Soulseek API URL</label>
+                                    <input type="text" name="soulseek_api_url" value="${config['soulseek_api_url']}" size="50">
+                                </div>
+                                <div class="row">
+                                    <label>Soulseek API KEY</label>
+                                    <input type="text" name="soulseek_api_key" value="${config['soulseek_api_key']}" size="20">
+                                </div>
+                                <div class="row">
+                                    <label title="Path to folder where Headphones can find the downloads.">
+                                        Soulseek Download Dir:
+                                    </label>
+                                    <input type="text" name="soulseek_download_dir" value="${config['soulseek_download_dir']}" size="50">
+                                </div>
+                                <div class="row">
+                                    <label title="Path to folder where Headphones can find the downloads.">
+                                        Soulseek Incomplete Download Dir:
+                                    </label>
+                                    <input type="text" name="soulseek_incomplete_download_dir" value="${config['soulseek_incomplete_download_dir']}" size="50">
                                </div>
                            </fieldset>
                        </td>
@@ -579,6 +620,19 @@
                                    </div>
                                </div>
                            </fieldset>
+                            <fieldset>
+                                <legend>Other</legend>
+                                <fieldset>
+                                    <div class="row checkbox left">
+                                        <input id="use_bandcamp" type="checkbox" class="bigcheck" name="use_bandcamp" value="1" ${config['use_bandcamp']} /><label for="use_bandcamp"><span class="option">Bandcamp</span></label>
+                                    </div>
+                                </fieldset>
+                                <fieldset>
+                                    <div class="row checkbox left">
+                                        <input id="use_soulseek" type="checkbox" class="bigcheck" name="use_soulseek" value="1" ${config['use_soulseek']} /><label for="use_soulseek"><span class="option">Soulseek</span></label>
+                                    </div>
+                                </fieldset>
+                            </fieldset>
                        </td>
                        <td>
                            <fieldset>
@@ -601,26 +655,6 @@
                                    </div>
                                </fieldset>

-                                <fieldset>
-                                    <div class="row checkbox left">
-                                        <input id="use_waffles" type="checkbox" class="bigcheck" name="use_waffles" value="1" ${config['use_waffles']} /><label for="use_waffles"><span class="option">Waffles.ch</span></label>
-                                    </div>
-                                    <div class="config">
-                                        <div class="row">
-                                            <label>UID Number</label>
-                                            <input type="text" name="waffles_uid" value="${config['waffles_uid']}" size="36">
-                                        </div>
-                                        <div class="row">
-                                            <label>Passkey</label>
-                                            <input type="password" name="waffles_passkey" value="${config['waffles_passkey']}" size="36">
-                                        </div>
-                                        <div class="row">
-                                            <label>Seed Ratio</label>
-                                            <input type="text" class="override-float" name="waffles_ratio" value="${config['waffles_ratio']}" size="10" title="Stop seeding when ratio met, 0 = unlimited. Scheduled job will remove torrent when post processed and finished seeding">
-                                        </div>
-                                    </div>
-                                </fieldset>
-
                                <fieldset>
                                    <div class="row checkbox left">
                                        <input id="use_rutracker" type="checkbox" class="bigcheck" name="use_rutracker" value="1" ${config['use_rutracker']} /><label for="use_rutracker"><span class="option">rutracker.org</span></label>
@@ -674,6 +708,10 @@
                                        <input id="use_redacted" type="checkbox" class="bigcheck" name="use_redacted" value="1" ${config['use_redacted']} /><label for="use_redacted"><span class="option">Redacted</span></label>
                                    </div>
                                    <div class="config">
+                                        <div class="row">
+                                            <label>Api Key</label>
+                                            <input type="text" name="redacted_apikey" value="${config['redacted_apikey']}" size="36">
+                                        </div>
                                        <div class="row">
                                            <label>Username</label>
                                            <input type="text" name="redacted_username" value="${config['redacted_username']}" size="36">
@@ -695,7 +733,7 @@

                                <fieldset>
                                    <div class="row checkbox left">
-                                        <input id="use_torznab" type="checkbox" class="bigcheck" name="use_torznab" value="1" ${config['use_torznab']} /><label for="use_torznab"><span class="option">Jackett / Torznab Providers</span></label>
+                                        <input id="use_torznab" type="checkbox" class="bigcheck" name="use_torznab" value="1" ${config['use_torznab']} /><label for="use_torznab"><span class="option">Torznab Providers</span></label>
                                    </div>
                                    <div id="torznab_providers">
                                        <div class="config" id="torznab1">
@@ -754,22 +792,6 @@
                                    </div>
                                </fieldset>

-                                <fieldset>
-                                    <div class="row checkbox left">
-                                        <input id="use_oldpiratebay" type="checkbox" class="bigcheck" name="use_oldpiratebay" value="1" ${config['use_oldpiratebay']} /><label for="use_oldpiratebay"><span class="option">Old Pirate Bay</span></label>
-                                    </div>
-                                    <div class="config">
-                                        <div class="row">
-                                            <label>URL</label>
-                                            <input type="text" name="oldpiratebay_url" value="${config['oldpiratebay_url']}" size="36">
-                                        </div>
-                                        <div class="row">
-                                            <label>Seed Ratio</label>
-                                            <input type="text" class="override-float" name="oldpiratebay_ratio" value="${config['oldpiratebay_ratio']}" size="10" title="Stop seeding when ratio met, 0 = unlimited. Scheduled job will remove torrent when post processed and finished seeding.">
-                                        </div>
-                                    </div>
-                                </fieldset>
-
                            </fieldset>
                        </td>
                    </tr>
@@ -970,20 +992,6 @@
                <table class="configtable" summary="Notifications">
                    <tr>
                        <td>
-                            <fieldset>
-                                <div class="row checkbox left">
-                                    <input type="checkbox" class="bigcheck" name="boxcar_enabled" id="boxcar" value="1" ${config['boxcar_enabled']} /><label for="boxcar"><span class="option">Boxcar 2</span></label>
-                                </div>
-                                <div id="boxcar_options">
-                                    <div class="row">
-                                        <label>Access Token</label><input type="text" name="boxcar_token" value="${config['boxcar_token']}" size="35">
-                                    </div>
-                                    <div class="row checkbox">
-                                        <input type="checkbox" name="boxcar_onsnatch" value="1" ${config['boxcar_onsnatch']} /><label>Notify on snatch?</label>
-                                    </div>
-                                  </div>
-                            </fieldset>
-
                            <fieldset>
                                <div class="row checkbox left">
                                    <input type="checkbox" class="bigcheck" name="email_enabled" id="email" value="1" ${config['email_enabled']} /><label for="email"><span class="option">Email</span></label>
@@ -1019,22 +1027,6 @@
                                </div>
                            </fieldset>

-                            <fieldset>
-                                <div class="row checkbox left">
-                                    <input type="checkbox" class="bigcheck" name="growl_enabled" id="growl" value="1" ${config['growl_enabled']} /><label for="growl"><span class="option">Growl</span></label>
-                                </div>
-                                <div id="growloptions">
-                                    <div class="row">
-                                        <label>Growl Host:Port</label><input type="text" name="growl_host" value="${config['growl_host']}" size="30">
-                                    </div>
-                                    <div class="row">
-                                        <label>Growl Password</label><input type="password" name="growl_password" value="${config['growl_password'] | h}" size="30">
-                                    </div>
-                                    <div class="row checkbox">
-                                        <input type="checkbox" name="growl_onsnatch" value="1" ${config['growl_onsnatch']} /><label>Notify on snatch?</label>
-                                    </div>
-                                </div>
-                            </fieldset>

                            <fieldset>
                                <div class="row checkbox left">
@@ -1123,18 +1115,9 @@

                            <fieldset>
                                <div class="row checkbox left">
-                                    <input type="checkbox" class="bigcheck" name="osx_notify_enabled" id="osx_notify" value="1" ${config['osx_notify_enabled']} /><label for="osx_notify"><span class="option">OS X</span></label>
+                                    <input type="checkbox" class="bigcheck" name="osx_notify_enabled" id="osx_notify" value="1" ${config['osx_notify_enabled']} /><label for="osx_notify"><span class="option">macOS</span></label>
                                </div>
                                <div id="osx_notify_options">
-                                    <div class="row">
-                                        <input type="text" id="osx_notify_reg" name="osx_notify_app" value="${config['osx_notify_app']}" size="50"><label>Register Notify App</label>
-                                    </div>
-                                    <div class="row">
-                                        <small>Enter the path/application name to be registered with the Notification Center, default is /Applications/Headphones</small>
-                                    </div>
-                                    <div class="row">
-                                        <input type="button" value="Register" id="osxnotifyregister"><label></label>
-                                    </div>
                                    <div class="row checkbox">
                                        <input type="checkbox" name="osx_notify_onsnatch" value="1" ${config['osx_notify_onsnatch']} /><label>Notify on snatch?</label>
                                    </div>
@@ -1418,7 +1401,7 @@
                                    libavselect = ''
                            %>
                            <div class="row">
-                                <label title="Name of encoder to use. Lame, FFmpeg and libav are available for most Linux distributions. On Ubuntu, libav replaces FFmpeg. xld is OS X-only.">
+                                <label title="Name of encoder to use. Lame, FFmpeg and libav are available for most Linux distributions. On Ubuntu, libav replaces FFmpeg. xld is macOS only.">
                                    Encoder
                                </label>
                                <select name="encoder" id="encoder">
@@ -1978,25 +1961,6 @@
                        }
                });

-                if ($("#growl").is(":checked"))
-                        {
-                            $("#growloptions").show();
-                        }
-                else
-                        {
-                            $("#growloptions").hide();
-                        }
-
-                $("#growl").click(function(){
-                        if ($("#growl").is(":checked"))
-                        {
-                            $("#growloptions").slideDown();
-                        }
-                        else
-                        {
-                            $("#growloptions").slideUp();
-                        }
-                });

                if ($("#prowl").is(":checked"))
                        {
@@ -2261,25 +2225,6 @@
                        }
                });

-                if ($("#boxcar").is(":checked"))
-                        {
-                            $("#boxcar_options").show();
-                        }
-                else
-                        {
-                            $("#boxcar_options").hide();
-                        }
-
-                $("#boxcar").click(function(){
-                        if ($("#boxcar").is(":checked"))
-                        {
-                            $("#boxcar_options").slideDown();
-                        }
-                        else
-                        {
-                            $("#boxcar_options").slideUp();
-                        }
-                });

                if ($("#subsonic").is(":checked"))
                        {
@@ -2564,8 +2509,6 @@
                initConfigCheckbox("#use_omgwtfnzbs");
                initConfigCheckbox("#use_torznab");
                initConfigCheckbox("#use_piratebay");
-                initConfigCheckbox("#use_oldpiratebay");
-                initConfigCheckbox("#use_waffles");
                initConfigCheckbox("#use_rutracker");
                initConfigCheckbox("#use_orpheus");
                initConfigCheckbox("#use_redacted");
@@ -2592,11 +2535,6 @@
                    $('#ajaxMsg').addClass('success').fadeIn().delay(3000).fadeOut();
                });

-                $('#osxnotifyregister').click(function () {
-                    var osx_notify_app = $("#osx_notify_reg").val();
-                    $.get("/osxnotifyregister", {'app': osx_notify_app}, function (data) { $('#ajaxMsg').html("<div class='msg'><span class='ui-icon ui-icon-check'></span>"+data+"</div>"); });
-                    $('#ajaxMsg').addClass('success').fadeIn().delay(3000).fadeOut()
-                })
        }

        $(document).ready(function() {
@@ -56,6 +56,8 @@
                fileid = 'torrent'
            if item['URL'].find('codeshy') != -1:
                fileid = 'nzb'
+            if item['URL'].find('bandcamp') != -1:
+                fileid = 'bandcamp'

            folder = 'Folder: ' + item['FolderName']

@@ -28,8 +28,31 @@ def getAlbumArt(albumid):

    # CAA
    logger.info("Searching for artwork at CAA")
-    artwork_path = 'https://coverartarchive.org/release-group/%s/front' % albumid
-    artwork = getartwork(artwork_path)
+    #artwork_path = 'https://coverartarchive.org/release-group/%s/front' % albumid
+    artwork_path = 'https://coverartarchive.org/release-group/%s' % albumid
+
+    data = request.request_json(artwork_path, timeout=20, whitelist_status_code=404)
+
+    image_url = None
+    if data:
+        for item in data.get("images", []):
+            try:
+                if "Front" not in item["types"]:
+                    continue
+
+                # Use desired size
+                image_url = item["image"]
+                if headphones.CONFIG.ALBUM_ART_MAX_WIDTH:
+                    if isinstance(item.get("thumbnails"), dict):
+                        image_url = item["thumbnails"].get(
+                            headphones.CONFIG.ALBUM_ART_MAX_WIDTH, image_url
+                        )
+                break
+            except KeyError:
+                pass
+
+    if image_url:
+        artwork = getartwork(image_url)
        if artwork:
            logger.info("Artwork found at CAA")
            return artwork_path, artwork
@@ -162,12 +185,14 @@ def getartwork(artwork_path):
                        "url": artwork_path,
                        "w": maxwidth
                    }
+                    headers = {"User-Agent": "Headphones"}
                    r = request.request_response(
                        url,
                        params=params,
                        timeout=20,
                        stream=True,
-                        whitelist_status_code=404
+                        whitelist_status_code=404,
+                        headers=headers
                    )
                    if r:
                        for chunk in r.iter_content(chunk_size=1024):
@@ -0,0 +1,166 @@
+#  This file is part of Headphones.
+#
+#  Headphones is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  Headphones is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with Headphones.  If not, see <http://www.gnu.org/licenses/>
+
+import headphones
+import json
+import os
+import re
+
+from headphones import logger, helpers, metadata, request
+from headphones.common import USER_AGENT
+from headphones.types import Result
+
+from mediafile import MediaFile, UnreadableFileError
+from bs4 import BeautifulSoup
+from bs4 import FeatureNotFound
+
+
+def search(album, albumlength=None, page=1, resultlist=None):
+    dic = {'...': '', ' & ': ' ', ' = ': ' ', '?': '', '$': 's', ' + ': ' ',
+           '"': '', ',': '', '*': '', '.': '', ':': ''}
+    if resultlist is None:
+        resultlist = []
+
+    cleanalbum = helpers.latinToAscii(
+        helpers.replace_all(album['AlbumTitle'], dic)
+        ).strip()
+    cleanartist = helpers.latinToAscii(
+        helpers.replace_all(album['ArtistName'], dic)
+        ).strip()
+
+    headers = {'User-Agent': USER_AGENT}
+    params = {
+        "page": page,
+        "q": cleanalbum,
+    }
+    logger.info("Looking up https://bandcamp.com/search with {}".format(
+        params))
+    content = request.request_content(
+        url='https://bandcamp.com/search',
+        params=params,
+        headers=headers
+        ).decode('utf8')
+    try:
+        soup = BeautifulSoup(content, "html5lib")
+    except FeatureNotFound:
+        soup = BeautifulSoup(content, "html.parser")
+
+    for item in soup.find_all("li", class_="searchresult"):
+        type = item.find('div', class_='itemtype').text.strip().lower()
+        if type == "album":
+            data = parse_album(item)
+
+            cleanartist_found = helpers.latinToAscii(data['artist'])
+            cleanalbum_found = helpers.latinToAscii(data['album'])
+
+            logger.debug(u"{} - {}".format(data['album'], cleanalbum_found))
+
+            logger.debug("Comparing {} to {}".format(
+                cleanalbum, cleanalbum_found))
+            if (cleanartist.lower() == cleanartist_found.lower() and
+                    cleanalbum.lower() == cleanalbum_found.lower()):
+                resultlist.append(Result(
+                    data['title'], data['size'], data['url'],
+                    'bandcamp', 'bandcamp', True))
+        else:
+            continue
+
+    if(soup.find('a', class_='next')):
+        page += 1
+        logger.debug("Calling next page ({})".format(page))
+        search(album, albumlength=albumlength,
+               page=page, resultlist=resultlist)
+
+    return resultlist
+
+
+def download(album, bestqual):
+    html = request.request_content(url=bestqual.url).decode('utf-8')
+    trackinfo = []
+    try:
+        trackinfo = json.loads(
+            re.search(r"trackinfo&quot;:(\[.*?\]),", html)
+            .group(1)
+            .replace('&quot;', '"'))
+    except ValueError as e:
+        logger.warn("Couldn't load json: {}".format(e))
+
+    directory = os.path.join(
+        headphones.CONFIG.BANDCAMP_DIR,
+        u'{} - {}'.format(
+            album['ArtistName'].replace('/', '_'),
+            album['AlbumTitle'].replace('/', '_')))
+    directory = helpers.latinToAscii(directory)
+
+    if not os.path.exists(directory):
+        try:
+            os.makedirs(directory)
+        except Exception as e:
+            logger.warn("Could not create directory ({})".format(e))
+
+    index = 1
+    for track in trackinfo:
+        filename = helpers.replace_illegal_chars(
+                    u'{:02d} - {}.mp3'.format(index, track['title']))
+        fullname = os.path.join(directory.encode('utf-8'),
+                                filename.encode('utf-8'))
+        logger.debug("Downloading to {}".format(fullname))
+
+        if 'file' in track and track['file'] != None and 'mp3-128' in track['file']:
+            content = request.request_content(track['file']['mp3-128'])
+            open(fullname, 'wb').write(content)
+            try:
+                f = MediaFile(fullname)
+                date, year = metadata._date_year(album)
+                f.update({
+                    'artist': album['ArtistName'].encode('utf-8'),
+                    'album': album['AlbumTitle'].encode('utf-8'),
+                    'title': track['title'].encode('utf-8'),
+                    'track': track['track_num'],
+                    'tracktotal': len(trackinfo),
+                    'year': year,
+                })
+                f.save()
+            except UnreadableFileError as ex:
+                logger.warn("MediaFile couldn't parse: %s (%s)",
+                            fullname,
+                            str(ex))
+
+        index += 1
+
+    return directory
+
+
+def parse_album(item):
+    album = item.find('div', class_='heading').text.strip()
+    artist = item.find('div', class_='subhead').text.strip().replace("by ", "")
+    released = item.find('div', class_='released').text.strip().replace(
+        "released ", "")
+    year = re.search(r"(\d{4})", released).group(1)
+
+    url = item.find('div', class_='heading').find('a')['href'].split("?")[0]
+
+    length = item.find('div', class_='length').text.strip()
+    tracks, minutes = length.split(",")
+    tracks = tracks.replace(" tracks", "").replace(" track", "").strip()
+    minutes = minutes.replace(" minutes", "").strip()
+    # bandcamp offers mp3 128b with should be 960KB/minute
+    size = int(minutes) * 983040
+
+    data = {"title": u'{} - {} [{}]'.format(artist, album, year),
+            "artist": artist, "album": album,
+            "url": url, "size": size}
+
+    return data
@@ -545,11 +545,13 @@ class Cache(object):
                                "url": thumb_url,
                                "w": 300
                            }
+                            headers = {"User-Agent": "Headphones"}
                            artwork_thumb = request.request_content(
                                url,
                                params=params,
                                timeout=20,
-                                whitelist_status_code=404
+                                whitelist_status_code=404,
+                                headers=headers
                            )
                        if artwork_thumb:
                            with open(thumb_path, 'wb') as f:
@@ -102,36 +102,6 @@ class Quality:

        return (anyQualities, bestQualities)

-    @staticmethod
-    def nameQuality(name):
-
-        def checkName(list, func):
-            return func([re.search(x, name, re.I) for x in list])
-
-        name = os.path.basename(name)
-
-        # if we have our exact text then assume we put it there
-        for x in Quality.qualityStrings:
-            if x == Quality.UNKNOWN:
-                continue
-
-            regex = '\W' + Quality.qualityStrings[x].replace(' ', '\W') + '\W'
-            regex_match = re.search(regex, name, re.I)
-            if regex_match:
-                return x
-
-        # TODO: fix quality checking here
-        if checkName(["mp3", "192"], any) and not checkName(["flac"], all):
-            return Quality.B192
-        elif checkName(["mp3", "256"], any) and not checkName(["flac"], all):
-            return Quality.B256
-        elif checkName(["mp3", "vbr"], any) and not checkName(["flac"], all):
-            return Quality.VBR
-        elif checkName(["mp3", "320"], any) and not checkName(["flac"], all):
-            return Quality.B320
-        else:
-            return Quality.UNKNOWN
-
    @staticmethod
    def assumeQuality(name):
        if name.lower().endswith(".mp3"):
@@ -158,13 +128,6 @@ class Quality:

        return (Quality.NONE, status)

-    @staticmethod
-    def statusFromName(name, assume=True):
-        quality = Quality.nameQuality(name)
-        if assume and quality == Quality.UNKNOWN:
-            quality = Quality.assumeQuality(name)
-        return Quality.compositeStatus(DOWNLOADED, quality)
-
    DOWNLOADED = None
    SNATCHED = None
    SNATCHED_PROPER = None
@@ -54,9 +54,6 @@ _CONFIG_DEFINITIONS = {
    'BITRATE': (int, 'General', 192),
    'BLACKHOLE': (int, 'General', 0),
    'BLACKHOLE_DIR': (path, 'General', ''),
-    'BOXCAR_ENABLED': (int, 'Boxcar', 0),
-    'BOXCAR_ONSNATCH': (int, 'Boxcar', 0),
-    'BOXCAR_TOKEN': (str, 'Boxcar', ''),
    'CACHE_DIR': (path, 'General', ''),
    'CACHE_SIZEMB': (int, 'Advanced', 32),
    'CHECK_GITHUB': (int, 'General', 1),
@@ -80,6 +77,7 @@ _CONFIG_DEFINITIONS = {
    'DELUGE_PASSWORD': (str, 'Deluge', ''),
    'DELUGE_LABEL': (str, 'Deluge', ''),
    'DELUGE_DONE_DIRECTORY': (str, 'Deluge', ''),
+    'DELUGE_DOWNLOAD_DIRECTORY': (str, 'Deluge', ''),
    'DELUGE_PAUSED': (int, 'Deluge', 0),
    'DESTINATION_DIR': (str, 'General', ''),
    'DETECT_BITRATE': (int, 'General', 0),
@@ -124,10 +122,6 @@ _CONFIG_DEFINITIONS = {
    'GIT_BRANCH': (str, 'General', 'master'),
    'GIT_PATH': (path, 'General', ''),
    'GIT_USER': (str, 'General', 'rembo10'),
-    'GROWL_ENABLED': (int, 'Growl', 0),
-    'GROWL_HOST': (str, 'Growl', ''),
-    'GROWL_ONSNATCH': (int, 'Growl', 0),
-    'GROWL_PASSWORD': (str, 'Growl', ''),
    'HEADPHONES_INDEXER': (bool_int, 'General', False),
    'HPPASS': (str, 'General', ''),
    'HPUSER': (str, 'General', ''),
@@ -196,15 +190,11 @@ _CONFIG_DEFINITIONS = {
    'OMGWTFNZBS_UID': (str, 'omgwtfnzbs', ''),
    'OPEN_MAGNET_LINKS': (int, 'General', 0),  # 0: Ignore, 1: Open, 2: Convert, 3: Embed (rtorrent)
    'MAGNET_LINKS': (int, 'General', 0),
-    'OSX_NOTIFY_APP': (str, 'OSX_Notify', '/Applications/Headphones'),
    'OSX_NOTIFY_ENABLED': (int, 'OSX_Notify', 0),
    'OSX_NOTIFY_ONSNATCH': (int, 'OSX_Notify', 0),
    'PIRATEBAY': (int, 'Piratebay', 0),
    'PIRATEBAY_PROXY_URL': (str, 'Piratebay', ''),
    'PIRATEBAY_RATIO': (str, 'Piratebay', ''),
-    'OLDPIRATEBAY': (int, 'Old Piratebay', 0),
-    'OLDPIRATEBAY_URL': (str, 'Old Piratebay', ''),
-    'OLDPIRATEBAY_RATIO': (str, 'Old Piratebay', ''),
    'PLEX_CLIENT_HOST': (str, 'Plex', ''),
    'PLEX_ENABLED': (int, 'Plex', 0),
    'PLEX_NOTIFY': (int, 'Plex', 0),
@@ -269,6 +259,11 @@ _CONFIG_DEFINITIONS = {
    'SONGKICK_ENABLED': (int, 'Songkick', 1),
    'SONGKICK_FILTER_ENABLED': (int, 'Songkick', 0),
    'SONGKICK_LOCATION': (str, 'Songkick', ''),
+    'SOULSEEK_API_URL': (str, 'Soulseek', ''),
+    'SOULSEEK_API_KEY': (str, 'Soulseek', ''),
+    'SOULSEEK_DOWNLOAD_DIR': (str, 'Soulseek', ''),
+    'SOULSEEK_INCOMPLETE_DOWNLOAD_DIR': (str, 'Soulseek', ''),
+    'SOULSEEK': (int, 'Soulseek', 0),
    'SUBSONIC_ENABLED': (int, 'Subsonic', 0),
    'SUBSONIC_HOST': (str, 'Subsonic', ''),
    'SUBSONIC_PASSWORD': (str, 'Subsonic', ''),
@@ -302,11 +297,8 @@ _CONFIG_DEFINITIONS = {
    'UTORRENT_USERNAME': (str, 'uTorrent', ''),
    'VERIFY_SSL_CERT': (bool_int, 'Advanced', 1),
    'WAIT_UNTIL_RELEASE_DATE': (int, 'General', 0),
-    'WAFFLES': (int, 'Waffles', 0),
-    'WAFFLES_PASSKEY': (str, 'Waffles', ''),
-    'WAFFLES_RATIO': (str, 'Waffles', ''),
-    'WAFFLES_UID': (str, 'Waffles', ''),
    'REDACTED': (int, 'Redacted', 0),
+    'REDACTED_APIKEY': (str, 'Redacted', ''),
    'REDACTED_USERNAME': (str, 'Redacted', ''),
    'REDACTED_PASSWORD': (str, 'Redacted', ''),
    'REDACTED_RATIO': (str, 'Redacted', ''),
@@ -317,7 +309,9 @@ _CONFIG_DEFINITIONS = {
    'XBMC_PASSWORD': (str, 'XBMC', ''),
    'XBMC_UPDATE': (int, 'XBMC', 0),
    'XBMC_USERNAME': (str, 'XBMC', ''),
-    'XLDPROFILE': (str, 'General', '')
+    'XLDPROFILE': (str, 'General', ''),
+    'BANDCAMP': (int, 'General', 0),
+    'BANDCAMP_DIR': (path, 'General', '')
 }


@@ -58,12 +58,12 @@ def _scrubber(text):
    if scrub_logs:
        try:
            # URL parameter values
-            text = re.sub('=[0-9a-zA-Z]*', '=REMOVED', text)
+            text = re.sub(r'=[0-9a-zA-Z]*', r'=REMOVED', text)
            # Local host with port
            # text = re.sub('\:\/\/.*\:', '://REMOVED:', text) # just host
-            text = re.sub('\:\/\/.*\:[0-9]*', '://REMOVED:', text)
+            text = re.sub(r'\:\/\/.*\:[0-9]*', r'://REMOVED:', text)
            # Session cookie
-            text = re.sub("_session_id'\: '.*'", "_session_id': 'REMOVED'", text)
+            text = re.sub(r"_session_id'\: '.*'", r"_session_id': 'REMOVED'", text)
            # Local Windows user path
            if text.lower().startswith('c:\\users\\'):
                k = text.split('\\')
@@ -84,7 +84,6 @@ def addTorrent(link, data=None, name=None):
        result = {}
        retid = False
        url_orpheus = ['https://orpheus.network/', 'http://orpheus.network/']
-        url_waffles = ['https://waffles.ch/', 'http://waffles.ch/']

        if link.lower().startswith('magnet:'):
            logger.debug('Deluge: Got a magnet link: %s' % _scrubber(link))
@@ -94,9 +93,6 @@ def addTorrent(link, data=None, name=None):

        elif link.lower().startswith('http://') or link.lower().startswith('https://'):
            logger.debug('Deluge: Got a URL: %s' % _scrubber(link))
-            if link.lower().startswith(tuple(url_waffles)):
-                if 'rss=' not in link:
-                    link = link + '&rss=1'
            if link.lower().startswith(tuple(url_orpheus)):
                logger.debug('Deluge: Using different User-Agent for this site')
                user_agent = 'Headphones'
@@ -128,9 +124,9 @@ def addTorrent(link, data=None, name=None):
                # Extract torrent name from .torrent
                try:
                    logger.debug('Deluge: Getting torrent name length')
-                    name_length = int(re.findall('name([0-9]*)\:.*?\:', str(torrentfile))[0])
+                    name_length = int(re.findall(r'name([0-9]*)\:.*?\:', str(torrentfile))[0])
                    logger.debug('Deluge: Getting torrent name')
-                    name = re.findall('name[0-9]*\:(.*?)\:', str(torrentfile))[0][:name_length]
+                    name = re.findall(r'name[0-9]*\:(.*?)\:', str(torrentfile))[0][:name_length]
                except Exception as e:
                    logger.debug('Deluge: Could not get torrent name, getting file name')
                    # get last part of link/path (name only)
@@ -160,9 +156,9 @@ def addTorrent(link, data=None, name=None):
                # Extract torrent name from .torrent
                try:
                    logger.debug('Deluge: Getting torrent name length')
-                    name_length = int(re.findall('name([0-9]*)\:.*?\:', str(torrentfile))[0])
+                    name_length = int(re.findall(r'name([0-9]*)\:.*?\:', str(torrentfile))[0])
                    logger.debug('Deluge: Getting torrent name')
-                    name = re.findall('name[0-9]*\:(.*?)\:', str(torrentfile))[0][:name_length]
+                    name = re.findall(r'name[0-9]*\:(.*?)\:', str(torrentfile))[0][:name_length]
                except Exception as e:
                    logger.debug('Deluge: Could not get torrent name, getting file name')
                    # get last part of link/path (name only)
@@ -466,13 +462,30 @@ def _add_torrent_url(result):

 def _add_torrent_file(result):
    logger.debug('Deluge: Adding file')
+
+    options = {}
+
+    if headphones.CONFIG.DELUGE_DOWNLOAD_DIRECTORY:
+        options['download_location'] = headphones.CONFIG.DELUGE_DOWNLOAD_DIRECTORY
+
+    if headphones.CONFIG.DELUGE_DONE_DIRECTORY or headphones.CONFIG.DOWNLOAD_TORRENT_DIR:
+        options['move_completed'] = 1
+        if headphones.CONFIG.DELUGE_DONE_DIRECTORY:
+            options['move_completed_path'] = headphones.CONFIG.DELUGE_DONE_DIRECTORY
+        else:
+            options['move_completed_path'] = headphones.CONFIG.DOWNLOAD_TORRENT_DIR
+
+    if headphones.CONFIG.DELUGE_PAUSED:
+        options['add_paused'] = headphones.CONFIG.DELUGE_PAUSED
+
    if not any(delugeweb_auth):
        _get_auth()
    try:
        # content is torrent file contents that needs to be encoded to base64
        post_data = json.dumps({"method": "core.add_torrent_file",
                                "params": [result['name'] + '.torrent',
-                                b64encode(result['content']).decode(), {}],
+                                b64encode(result['content']).decode(),
+                                options],
                                "id": 2})
        response = requests.post(delugeweb_url, data=post_data.encode('utf-8'), cookies=delugeweb_auth,
                                 verify=deluge_verify_cert, headers=headers)
@@ -566,61 +579,3 @@ def setSeedRatio(result):
        return None


-def setTorrentPath(result):
-    logger.debug('Deluge: Setting download path')
-    if not any(delugeweb_auth):
-        _get_auth()
-
-    try:
-        if headphones.CONFIG.DELUGE_DONE_DIRECTORY or headphones.CONFIG.DOWNLOAD_TORRENT_DIR:
-            post_data = json.dumps({"method": "core.set_torrent_move_completed",
-                                    "params": [result['hash'], True],
-                                    "id": 7})
-            response = requests.post(delugeweb_url, data=post_data.encode('utf-8'), cookies=delugeweb_auth,
-                verify=deluge_verify_cert, headers=headers)
-
-            if headphones.CONFIG.DELUGE_DONE_DIRECTORY:
-                move_to = headphones.CONFIG.DELUGE_DONE_DIRECTORY
-            else:
-                move_to = headphones.CONFIG.DOWNLOAD_TORRENT_DIR
-
-            if not os.path.exists(move_to):
-                logger.debug('Deluge: %s directory doesn\'t exist, let\'s create it' % move_to)
-                os.makedirs(move_to)
-            post_data = json.dumps({"method": "core.set_torrent_move_completed_path",
-                                    "params": [result['hash'], move_to],
-                                    "id": 8})
-            response = requests.post(delugeweb_url, data=post_data.encode('utf-8'), cookies=delugeweb_auth,
-                verify=deluge_verify_cert, headers=headers)
-
-            return not json.loads(response.text)['error']
-
-        return True
-    except Exception as e:
-        logger.error('Deluge: Setting torrent move-to directory failed: %s' % str(e))
-        formatted_lines = traceback.format_exc().splitlines()
-        logger.error('; '.join(formatted_lines))
-        return None
-
-
-def setTorrentPause(result):
-    logger.debug('Deluge: Pausing torrent')
-    if not any(delugeweb_auth):
-        _get_auth()
-
-    try:
-        if headphones.CONFIG.DELUGE_PAUSED:
-            post_data = json.dumps({"method": "core.pause_torrent",
-                                    "params": [[result['hash']]],
-                                    "id": 9})
-            response = requests.post(delugeweb_url, data=post_data.encode('utf-8'), cookies=delugeweb_auth,
-                verify=deluge_verify_cert, headers=headers)
-
-            return not json.loads(response.text)['error']
-
-        return True
-    except Exception as e:
-        logger.error('Deluge: Setting torrent paused failed: %s' % str(e))
-        formatted_lines = traceback.format_exc().splitlines()
-        logger.error('; '.join(formatted_lines))
-        return None
@@ -184,7 +184,7 @@ def bytes_to_mb(bytes):


 def mb_to_bytes(mb_str):
-    result = re.search('^(\d+(?:\.\d+)?)\s?(?:mb)?', mb_str, flags=re.I)
+    result = re.search(r"^(\d+(?:\.\d+)?)\s?(?:mb)?", mb_str, flags=re.I)
    if result:
        return int(float(result.group(1)) * 1048576)

@@ -253,9 +253,9 @@ def replace_all(text, dic):

 def replace_illegal_chars(string, type="file"):
    if type == "file":
-        string = re.sub('[\?"*:|<>/]', '_', string)
+        string = re.sub(r"[\?\"*:|<>/]", "_", string)
    if type == "folder":
-        string = re.sub('[:\?<>"|*]', '_', string)
+        string = re.sub(r"[:\?<>\"|*]", "_", string)
    return string


@@ -386,7 +386,7 @@ def clean_musicbrainz_name(s, return_as_string=True):


 def cleanTitle(title):
-    title = re.sub('[\.\-\/\_]', ' ', title).lower()
+    title = re.sub(r"[\.\-\/\_]", " ", title).lower()

    # Strip out extra whitespace
    title = ' '.join(title.split())
@@ -860,6 +860,8 @@ def smartMove(src, dest, delete=True):
                try:
                    os.rename(src, os.path.join(source_dir, newfile))
                    filename = newfile
+                    source_path = os.path.join(source_dir, filename)
+                    dest_path = os.path.join(dest, filename)
                except Exception as e:
                    logger.warn(f"Error renaming {src}: {e}")
                break
@@ -1050,3 +1052,10 @@ def have_pct_have_total(db_artist):
    have_pct = have_tracks / total_tracks if total_tracks else 0
    return (have_pct, total_tracks)

+
+def has_token(title, token):
+    return bool(
+            re.search(rf'(?:\W|^)+{token}(?:\W|$)+', 
+            title,
+            re.IGNORECASE | re.UNICODE)
+        )
@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 from .unittestcompat import TestCase
-from headphones.helpers import clean_name, is_valid_date, age
+from headphones.helpers import clean_name, is_valid_date, age, has_token


 class HelpersTest(TestCase):
@@ -56,3 +56,18 @@ class HelpersTest(TestCase):
        ]
        for input, expected, desc in test_cases:
            self.assertEqual(is_valid_date(input), expected, desc)
+
+    def test_has_token(self):
+        """helpers: has_token()"""
+        self.assertEqual(
+            has_token("a cat ran", "cat"), 
+            True, 
+            "return True if token is in string"
+        )
+        self.assertEqual(
+            has_token("acatran", "cat"),
+            False,
+            "return False if token is part of another word"
+        )
+
+
@@ -160,6 +160,15 @@ def encode(albumPath):

        # Use multicore if enabled
        if headphones.CONFIG.ENCODER_MULTICORE:
+
+            # Set macOS multiprocessing method
+            try:
+                if headphones.SYS_PLATFORM == "darwin":
+                    multiprocessing.set_start_method('fork')
+            except RuntimeError:
+                # Already set, ignore
+                pass
+
            if headphones.CONFIG.ENCODER_MULTICORE_COUNT == 0:
                processes = multiprocessing.cpu_count()
            else:
@@ -15,98 +15,10 @@ from headphones import logger, helpers, common, request
 from pynma import pynma
 import cherrypy
 import headphones
-import gntp.notifier
 #import oauth2 as oauth
 import twitter


-class GROWL(object):
-    """
-    Growl notifications, for OS X.
-    """
-
-    def __init__(self):
-        self.enabled = headphones.CONFIG.GROWL_ENABLED
-        self.host = headphones.CONFIG.GROWL_HOST
-        self.password = headphones.CONFIG.GROWL_PASSWORD
-
-    def conf(self, options):
-        return cherrypy.config['config'].get('Growl', options)
-
-    def notify(self, message, event):
-        if not self.enabled:
-            return
-
-        # Split host and port
-        if self.host == "":
-            host, port = "localhost", 23053
-
-        if ":" in self.host:
-            host, port = self.host.split(':', 1)
-            port = int(port)
-        else:
-            host, port = self.host, 23053
-
-        # If password is empty, assume none
-        if self.password == "":
-            password = None
-        else:
-            password = self.password
-
-        # Register notification
-        growl = gntp.notifier.GrowlNotifier(
-            applicationName='Headphones',
-            notifications=['New Event'],
-            defaultNotifications=['New Event'],
-            hostname=host,
-            port=port,
-            password=password
-        )
-
-        try:
-            growl.register()
-        except gntp.notifier.errors.NetworkError:
-            logger.warning('Growl notification failed: network error')
-            return
-        except gntp.notifier.errors.AuthError:
-            logger.warning('Growl notification failed: authentication error')
-            return
-
-        # Fix message
-        message = message.encode(headphones.SYS_ENCODING, "replace")
-
-        # Send it, including an image
-        image_file = os.path.join(str(headphones.PROG_DIR),
-                                  "data/images/headphoneslogo.png")
-
-        with open(image_file, 'rb') as f:
-            image = f.read()
-
-        try:
-            growl.notify(
-                noteType='New Event',
-                title=event,
-                description=message,
-                icon=image
-            )
-        except gntp.notifier.errors.NetworkError:
-            logger.warning('Growl notification failed: network error')
-            return
-
-        logger.info("Growl notifications sent.")
-
-    def updateLibrary(self):
-        # For uniformity reasons not removed
-        return
-
-    def test(self, host, password):
-        self.enabled = True
-        self.host = host
-        self.password = password
-
-        self.notify('ZOMG Lazors Pewpewpew!', 'Test Message')
-
-
 class PROWL(object):
    """
    Prowl notifications.
@@ -840,105 +752,12 @@ class TwitterNotifier(object):


 class OSX_NOTIFY(object):
-    def __init__(self):
+    def notify(self, title, subtitle):
        try:
-            self.objc = __import__("objc")
-            self.AppKit = __import__("AppKit")
-        except:
-            logger.warn('OS X Notification: Cannot import objc or AppKit')
-            pass
-
-    def swizzle(self, cls, SEL, func):
-        old_IMP = getattr(cls, SEL, None)
-        if old_IMP is None:
-            old_IMP = cls.instanceMethodForSelector_(SEL)
-
-        def wrapper(self, *args, **kwargs):
-            return func(self, old_IMP, *args, **kwargs)
-
-        new_IMP = self.objc.selector(
-            wrapper,
-            selector=old_IMP.selector,
-            signature=old_IMP.signature
-        )
-        self.objc.classAddMethod(cls, SEL.encode(), new_IMP)
-
-    def notify(self, title, subtitle=None, text=None, sound=True, image=None):
-
-        try:
-            self.swizzle(
-                self.objc.lookUpClass('NSBundle'),
-                'bundleIdentifier',
-                self.swizzled_bundleIdentifier
-            )
-
-            NSUserNotification = self.objc.lookUpClass('NSUserNotification')
-            NSUserNotificationCenter = self.objc.lookUpClass(
-                'NSUserNotificationCenter')
-            NSAutoreleasePool = self.objc.lookUpClass('NSAutoreleasePool')
-
-            if not NSUserNotification or not NSUserNotificationCenter:
-                return False
-
-            pool = NSAutoreleasePool.alloc().init()
-
-            notification = NSUserNotification.alloc().init()
-            notification.setTitle_(title)
-            if subtitle:
-                notification.setSubtitle_(subtitle)
-            if text:
-                notification.setInformativeText_(text)
-            if sound:
-                notification.setSoundName_(
-                    "NSUserNotificationDefaultSoundName")
-            if image:
-                source_img = self.AppKit.NSImage.alloc().\
-                    initByReferencingFile_(image)
-                notification.setContentImage_(source_img)
-                # notification.set_identityImage_(source_img)
-            notification.setHasActionButton_(False)
-
-            notification_center = NSUserNotificationCenter.\
-                defaultUserNotificationCenter()
-            notification_center.deliverNotification_(notification)
-
-            del pool
-            return True
-
+            script = f'display notification "{subtitle}" with title "{title}"'
+            subprocess.run(["osascript", "-e", script])
        except Exception as e:
-            logger.warn('Error sending OS X Notification: %s' % e)
-            return False
-
-    def swizzled_bundleIdentifier(self, original, swizzled):
-        return 'ade.headphones.osxnotify'
-
-
-class BOXCAR(object):
-    def __init__(self):
-        self.url = 'https://new.boxcar.io/api/notifications'
-
-    def notify(self, title, message, rgid=None):
-        try:
-            if rgid:
-                message += '<br></br><a href="https://musicbrainz.org/' \
-                           'release-group/%s">MusicBrainz</a>' % rgid
-
-            data = urllib.parse.urlencode({
-                'user_credentials': headphones.CONFIG.BOXCAR_TOKEN,
-                'notification[title]': title.encode('utf-8'),
-                'notification[long_message]': message.encode('utf-8'),
-                'notification[sound]': "done",
-                'notification[icon_url]': "https://raw.githubusercontent.com/rembo10/headphones/master/data/images"
-                                          "/headphoneslogo.png"
-            })
-
-            req = urllib.request.Request(self.url)
-            handle = urllib.request.urlopen(req, data)
-            handle.close()
-            return True
-
-        except urllib.error.URLError as e:
-            logger.warn('Error sending Boxcar2 Notification: %s' % e)
+            logger.warn(f"Error sending MacOS Notification: {e}")
            return False


@@ -27,7 +27,7 @@ from beets import config as beetsconfig
 from beets import logging as beetslogging
 from mediafile import MediaFile, FileTypeError, UnreadableFileError
 from beetsplug import lyrics as beetslyrics
-from headphones import notifiers, utorrent, transmission, deluge, qbittorrent
+from headphones import notifiers, utorrent, transmission, deluge, qbittorrent, soulseek
 from headphones import db, albumart, librarysync
 from headphones import logger, helpers, mb, music_encoder
 from headphones import metadata
@@ -36,18 +36,45 @@ postprocessor_lock = threading.Lock()


 def checkFolder():
-    logger.debug("Checking download folder for completed downloads (only snatched ones).")
+    logger.info("Checking download folder for completed downloads (only snatched ones).")

    with postprocessor_lock:
        myDB = db.DBConnection()
        snatched = myDB.select('SELECT * from snatched WHERE Status="Snatched"')
-
        for album in snatched:
            if album['FolderName']:
                folder_name = album['FolderName']
                single = False
-                if album['Kind'] == 'nzb':
+
+                # Soulseek, check download complete or errored
+                if album['Kind'] == 'soulseek':
+                    match = re.search(r'\{(.*?)\}(.*?)$', folder_name)    # get soulseek user from folder_name
+                    user_name = match.group(1)
+                    folder_name = match.group(2)
+                    completed, errored = soulseek.download_completed_album(user_name, folder_name)
+                    if errored:
+                        # If the album had any tracks with errors in it, the whole download is considered faulty. Status will be reset to wanted.
+                        logger.info(f"Soulseek: Album with folder '{folder_name}' had errors during download. Setting status to 'Wanted'.")
+                        myDB.action('UPDATE albums SET Status="Wanted" WHERE AlbumID=? AND Status="Snatched"', (album['AlbumID'],))
+                        myDB.action('UPDATE snatched SET status = "Unprocessed" WHERE AlbumID=?', (album['AlbumID'],))
+                        
+                        # Folder will be removed from configured complete and Incomplete directory
+                        complete_path = os.path.join(headphones.CONFIG.SOULSEEK_DOWNLOAD_DIR, folder_name)
+                        incomplete_path = os.path.join(headphones.CONFIG.SOULSEEK_INCOMPLETE_DOWNLOAD_DIR, folder_name)
+                        for path in [complete_path, incomplete_path]:
+                            try:
+                                shutil.rmtree(path)
+                            except Exception as e:
+                                pass
+                        continue
+                    elif completed:
+                        download_dir = headphones.CONFIG.SOULSEEK_DOWNLOAD_DIR
+                    else:
+                        continue
+                elif album['Kind'] == 'nzb':
                    download_dir = headphones.CONFIG.DOWNLOAD_DIR   
+                elif album['Kind'] == 'bandcamp':
+                    download_dir = headphones.CONFIG.BANDCAMP_DIR 
                else:
                    if headphones.CONFIG.DELUGE_DONE_DIRECTORY and headphones.CONFIG.TORRENT_DOWNLOADER == 3:
                        download_dir = headphones.CONFIG.DELUGE_DONE_DIRECTORY
@@ -289,7 +316,7 @@ def verify(albumid, albumpath, Kind=None, forced=False, keep_original_folder=Fal
    logger.debug('Metadata check failed. Verifying filenames...')
    for downloaded_track in downloaded_track_list:
        track_name = os.path.splitext(downloaded_track)[0]
-        split_track_name = re.sub('[\.\-\_]', ' ', track_name).lower()
+        split_track_name = re.sub(r'[\.\-\_]', r' ', track_name).lower()
        for track in tracks:

            if not track['TrackTitle']:
@@ -523,11 +550,6 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
    pushmessage = release['ArtistName'] + ' - ' + release['AlbumTitle']
    statusmessage = "Download and Postprocessing completed"

-    if headphones.CONFIG.GROWL_ENABLED:
-        logger.info("Growl request")
-        growl = notifiers.GROWL()
-        growl.notify(pushmessage, statusmessage)
-
    if headphones.CONFIG.PROWL_ENABLED:
        logger.info("Prowl request")
        prowl = notifiers.PROWL()
@@ -596,21 +618,9 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
        #twitter.notify_download(pushmessage)

    if headphones.CONFIG.OSX_NOTIFY_ENABLED:
-        from headphones import cache
-        c = cache.Cache()
-        album_art = c.get_artwork_from_cache(None, release['AlbumID'])
-        logger.info("Sending OS X notification")
-        osx_notify = notifiers.OSX_NOTIFY()
-        osx_notify.notify(release['ArtistName'],
-                          release['AlbumTitle'],
-                          statusmessage,
-                          image=album_art)
-
-    if headphones.CONFIG.BOXCAR_ENABLED:
-        logger.info("Sending Boxcar2 notification")
-        boxcar = notifiers.BOXCAR()
-        boxcar.notify('Headphones processed: ' + pushmessage,
-                      statusmessage, release['AlbumID'])
+        logger.info("Sending MacOS notification")
+        osx = notifiers.OSX_NOTIFY()
+        osx.notify(f"Headphones Processed", f"{pushmessage}\n{statusmessage}")

    if headphones.CONFIG.SUBSONIC_ENABLED:
        logger.info("Sending Subsonic update")
@@ -1168,10 +1178,15 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None, keep_orig

        if dir:
            download_dirs.append(dir)
-        if headphones.CONFIG.DOWNLOAD_DIR and not dir:
+        else:
+            if headphones.CONFIG.DOWNLOAD_DIR:
                download_dirs.append(headphones.CONFIG.DOWNLOAD_DIR)
-        if headphones.CONFIG.DOWNLOAD_TORRENT_DIR and not dir:
+            if headphones.CONFIG.SOULSEEK_DOWNLOAD_DIR:
+                download_dirs.append(headphones.CONFIG.SOULSEEK_DOWNLOAD_DIR)
+            if headphones.CONFIG.DOWNLOAD_TORRENT_DIR:
                download_dirs.append(headphones.CONFIG.DOWNLOAD_TORRENT_DIR)
+            if headphones.CONFIG.BANDCAMP:
+                download_dirs.append(headphones.CONFIG.BANDCAMP_DIR)

        # If DOWNLOAD_DIR and DOWNLOAD_TORRENT_DIR are the same, remove the duplicate to prevent us from trying to process the same folder twice.
        download_dirs = list(set(download_dirs))
@@ -1181,6 +1196,7 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None, keep_orig
        folders = []

        for download_dir in download_dirs:
+            download_dir = download_dir.encode(headphones.SYS_ENCODING, 'replace')
            if not os.path.isdir(download_dir):
                logger.warn('Directory %s does not exist. Skipping', download_dir)
                continue
@@ -1198,9 +1214,9 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None, keep_orig
                    subfolders = helpers.expand_subfolders(path_to_folder)

                    if expand_subfolders and subfolders is not None:
-                        folders.extend(subfolders)
+                        folders.extend(subfolders.decode(headphones.SYS_ENCODING, 'replace'))
                    else:
-                        folders.append(path_to_folder)
+                        folders.append(path_to_folder.decode(headphones.SYS_ENCODING, 'replace'))

    # Log number of folders
    if folders:
@@ -23,6 +23,9 @@ from headphones import logger
 import feedparser
 import headphones
 import headphones.lock
+from bs4.builder import XMLParsedAsHTMLWarning
+import warnings
+warnings.filterwarnings("ignore", category=XMLParsedAsHTMLWarning)


 # Disable SSL certificate warnings. We have our own handling
@@ -245,7 +248,9 @@ def server_message(response):

    if message:
        # Truncate message if it is too long.
-        if len(message) > 150:
-            message = message[:150] + "..."
+        if len(message) > 200:
+            if not type(message) == str:
+                message = message.decode(headphones.SYS_ENCODING, 'replace')
+            message = message[:200] + "..."

        logger.debug("Server responded with message: %s", message)
@@ -42,19 +42,22 @@ class Rutracker(object):
            'login_password': headphones.CONFIG.RUTRACKER_PASSWORD,
            'login': b'\xc2\xf5\xee\xe4'  # '%C2%F5%EE%E4'
        }
+        headers = {
+            'User-Agent' : 'Headphones'   
+        }

        logger.info("Attempting to log in to rutracker...")

        try:
-            r = self.session.post(loginpage, data=post_params, timeout=self.timeout, allow_redirects=False)
+            r = self.session.post(loginpage, data=post_params, timeout=self.timeout, allow_redirects=False, headers=headers)
            # try again
            if not self.has_bb_session_cookie(r):
                time.sleep(10)
                if headphones.CONFIG.RUTRACKER_COOKIE:
                    logger.info("Attempting to log in using predefined cookie...")
-                    r = self.session.post(loginpage, data=post_params, timeout=self.timeout, allow_redirects=False, cookies={'bb_session': headphones.CONFIG.RUTRACKER_COOKIE})
+                    r = self.session.post(loginpage, data=post_params, timeout=self.timeout, allow_redirects=False, headers=headers, cookies={'bb_session': headphones.CONFIG.RUTRACKER_COOKIE})
                else:
-                    r = self.session.post(loginpage, data=post_params, timeout=self.timeout, allow_redirects=False)
+                    r = self.session.post(loginpage, data=post_params, timeout=self.timeout, allow_redirects=False, headers=headers)
            if self.has_bb_session_cookie(r):
                self.loggedin = True
                logger.info("Successfully logged in to rutracker")
@@ -113,7 +116,10 @@ class Rutracker(object):
        Parse the search results and return valid torrent list
        """
        try:
-            headers = {'Referer': self.search_referer}
+            headers = {
+                'Referer': self.search_referer,
+                'User-Agent' : 'Headphones'   
+            }
            r = self.session.get(url=searchurl, headers=headers, timeout=self.timeout)
            soup = BeautifulSoup(r.content, 'html.parser')

@@ -183,7 +189,10 @@ class Rutracker(object):
        downloadurl = 'https://rutracker.org/forum/dl.php?t=' + torrent_id
        cookie = {'bb_dl': torrent_id}
        try:
-            headers = {'Referer': url}
+            headers = {
+                'Referer': url,
+                'User-Agent' : 'Headphones'   
+            }
            r = self.session.post(url=downloadurl, cookies=cookie, headers=headers,
                                  timeout=self.timeout)
            return r.content
@@ -0,0 +1,252 @@
+from collections import defaultdict, namedtuple
+import os
+import time
+import slskd_api
+import headphones
+from headphones import logger
+from datetime import datetime, timedelta
+
+Result = namedtuple('Result', ['title', 'size', 'user', 'provider', 'type', 'matches', 'bandwidth', 'hasFreeUploadSlot', 'queueLength', 'files', 'kind', 'url', 'folder'])
+
+def initialize_soulseek_client():
+    host = headphones.CONFIG.SOULSEEK_API_URL
+    api_key = headphones.CONFIG.SOULSEEK_API_KEY
+    return slskd_api.SlskdClient(host=host, api_key=api_key)
+
+    # Search logic, calling search and processing fucntions
+def search(artist, album, year, num_tracks, losslessOnly, allow_lossless, user_search_term):
+    client = initialize_soulseek_client()
+
+    # override search string with user provided search term if entered
+    if user_search_term:
+        artist = user_search_term
+        album = ''
+        year = ''
+    
+    # Stage 1: Search with artist, album, year, and num_tracks
+    logger.info(f"Searching Soulseek using term: {artist} {album} {year}")
+    results = execute_search(client, artist, album, year, losslessOnly, allow_lossless)
+    processed_results = process_results(results, losslessOnly, allow_lossless, num_tracks)
+    if processed_results or user_search_term or album.lower() == artist.lower():
+        return processed_results
+    
+    # Stage 2: If Stage 1 fails, search with artist, album, and num_tracks (excluding year)
+    logger.info("Soulseek search stage 1 did not meet criteria. Retrying without year...")
+    results = execute_search(client, artist, album, None, losslessOnly, allow_lossless)
+    processed_results = process_results(results, losslessOnly, allow_lossless, num_tracks)
+    if processed_results or artist == "Various Artists":
+        return processed_results
+    
+    # Stage 3: Final attempt, search only with artist and album
+    logger.info("Soulseek search stage 2 did not meet criteria. Final attempt with only artist and album.")
+    results = execute_search(client, artist, album, None, losslessOnly, allow_lossless)
+    processed_results = process_results(results, losslessOnly, allow_lossless, num_tracks, ignore_track_count=True)
+
+    return processed_results
+
+def execute_search(client, artist, album, year, losslessOnly, allow_lossless):
+    search_text = f"{artist} {album}"
+    if year:
+        search_text += f" {year}"
+
+    if losslessOnly:
+        search_text += " flac"
+    elif not allow_lossless:
+            search_text += " mp3"
+
+    # Actual search
+    search_response = client.searches.search_text(searchText=search_text, filterResponses=True)
+    search_id = search_response.get('id')
+
+    # Wait for search completion and return response
+    while not client.searches.state(id=search_id).get('isComplete'):
+        time.sleep(2)
+    
+    return client.searches.search_responses(id=search_id)
+
+# Processing the search result passed
+def process_results(results, losslessOnly, allow_lossless, num_tracks, ignore_track_count=False):
+
+    if losslessOnly:
+        valid_extensions = {'.flac'}
+    elif allow_lossless:
+        valid_extensions = {'.mp3', '.flac'}
+    else:
+        valid_extensions = {'.mp3'}
+
+    albums = defaultdict(lambda: {'files': [], 'user': None, 'hasFreeUploadSlot': None, 'queueLength': None, 'uploadSpeed': None})
+
+    # Extract info from the api response and combine files at album level
+    for result in results:
+        user = result.get('username')
+        hasFreeUploadSlot = result.get('hasFreeUploadSlot')
+        queueLength = result.get('queueLength')
+        uploadSpeed = result.get('uploadSpeed')
+
+        # Only handle .mp3 and .flac
+        for file in result.get('files', []):
+            filename = file.get('filename')
+            file_extension = os.path.splitext(filename)[1].lower()
+            if file_extension in valid_extensions:
+                #album_directory = os.path.dirname(filename)
+                album_directory = filename.rsplit('\\', 1)[0]
+                albums[album_directory]['files'].append(file)
+
+                # Update metadata only once per album_directory
+                if albums[album_directory]['user'] is None:
+                    albums[album_directory].update({
+                        'user': user,
+                        'hasFreeUploadSlot': hasFreeUploadSlot,
+                        'queueLength': queueLength,
+                        'uploadSpeed': uploadSpeed,
+                    })
+
+    # Filter albums based on num_tracks, add bunch of useful info to the compiled album
+    final_results = []
+    for directory, album_data in albums.items():
+        if ignore_track_count and len(album_data['files']) > 1 or len(album_data['files']) == num_tracks:
+            #album_title = os.path.basename(directory)
+            album_title = directory.rsplit('\\', 1)[1]
+            total_size = sum(file.get('size', 0) for file in album_data['files'])
+            final_results.append(Result(
+                title=album_title,
+                size=int(total_size),
+                user=album_data['user'],
+                provider="soulseek",
+                type="soulseek",
+                matches=True,
+                bandwidth=album_data['uploadSpeed'],
+                hasFreeUploadSlot=album_data['hasFreeUploadSlot'],
+                queueLength=album_data['queueLength'],
+                files=album_data['files'],
+                kind='soulseek',
+                url='http://' + album_data['user'] + album_title, # URL is needed in other parts of the program.
+                #folder=os.path.basename(directory)
+                folder = album_title
+            ))
+
+    return final_results
+
+
+def download(user, filelist):
+    client = initialize_soulseek_client()
+    client.transfers.enqueue(username=user, files=filelist)
+
+
+def download_completed():
+    client = initialize_soulseek_client()
+    all_downloads = client.transfers.get_all_downloads(includeRemoved=False)
+    album_completion_tracker = {}  # Tracks completion state of each album's songs
+    album_errored_tracker = {}  # Tracks albums with errored downloads
+
+    # Anything older than 24 hours will be canceled
+    cutoff_time = datetime.now() - timedelta(hours=24)
+
+    # Identify errored and completed albums
+    for download in all_downloads:
+        directories = download.get('directories', [])
+        for directory in directories:
+            album_part = directory.get('directory', '').split('\\')[-1]
+            files = directory.get('files', [])
+            for file_data in files:
+                state = file_data.get('state', '')
+                requested_at_str = file_data.get('requestedAt', '1900-01-01 00:00:00')
+                requested_at = parse_datetime(requested_at_str)
+
+                # Initialize or update album entry in trackers
+                if album_part not in album_completion_tracker:
+                    album_completion_tracker[album_part] = {'total': 0, 'completed': 0, 'errored': 0}
+                if album_part not in album_errored_tracker:
+                    album_errored_tracker[album_part] = False
+
+                album_completion_tracker[album_part]['total'] += 1
+
+                if 'Completed, Succeeded' in state:
+                    album_completion_tracker[album_part]['completed'] += 1
+                elif 'Completed, Errored' in state or requested_at < cutoff_time:
+                    album_completion_tracker[album_part]['errored'] += 1
+                    album_errored_tracker[album_part] = True  # Mark album as having errored downloads
+
+    # Identify errored albums
+    errored_albums = {album for album, errored in album_errored_tracker.items() if errored}
+
+    # Cancel downloads for errored albums
+    for download in all_downloads:
+        directories = download.get('directories', [])
+        for directory in directories:
+            album_part = directory.get('directory', '').split('\\')[-1]
+            files = directory.get('files', [])
+            for file_data in files:
+                if album_part in errored_albums:
+                    # Extract 'id' and 'username' for each file to cancel the download
+                    file_id = file_data.get('id', '')
+                    username = file_data.get('username', '')
+                    success = client.transfers.cancel_download(username, file_id)
+                    if not success:
+                        logger.debug(f"Soulseek failed to cancel download for file ID: {file_id}")
+
+    # Clear completed/canceled/errored stuff from client downloads
+    try:
+        client.transfers.remove_completed_downloads()
+    except Exception as e:
+        logger.debug(f"Soulseek failed to remove completed downloads: {e}")
+
+    # Identify completed albums
+    completed_albums = {album for album, counts in album_completion_tracker.items() if counts['total'] == counts['completed']}
+
+    # Return both completed and errored albums
+    return completed_albums, errored_albums
+
+
+def download_completed_album(username, foldername):
+    client = initialize_soulseek_client()
+    downloads = client.transfers.get_downloads(username)
+
+    # Anything older than 24 hours will be canceled
+    cutoff_time = datetime.now() - timedelta(hours=24)
+
+    total_count = 0
+    completed_count = 0
+    errored_count = 0
+    file_ids = []
+
+    # Identify errored and completed album
+    directories = downloads.get('directories', [])
+    for directory in directories:
+        album_part = directory.get('directory', '').split('\\')[-1]
+        if album_part == foldername:
+            files = directory.get('files', [])
+            for file_data in files:
+                state = file_data.get('state', '')
+                requested_at_str = file_data.get('requestedAt', '1900-01-01 00:00:00')
+                requested_at = parse_datetime(requested_at_str)
+
+                total_count += 1
+                file_id = file_data.get('id', '')
+                file_ids.append(file_id)
+
+                if 'Completed, Succeeded' in state:
+                    completed_count += 1
+                elif 'Completed, Errored' in state or requested_at < cutoff_time:
+                    errored_count += 1
+            break
+
+    completed = True if completed_count == total_count else False
+    errored = True if errored_count else False
+
+    # Cancel downloads for errored album
+    if errored:
+        for file_id in file_ids:
+            try:
+                success = client.transfers.cancel_download(username, file_id, remove=True)
+            except Exception as e:
+                logger.debug(f"Soulseek failed to cancel download for folder with file ID: {foldername} {file_id}")
+
+    return completed, errored
+
+
+def parse_datetime(datetime_string):
+    # Parse the datetime api response
+    if '.' in datetime_string:
+        datetime_string = datetime_string[:datetime_string.index('.')+7]
+    return datetime.strptime(datetime_string, '%Y-%m-%dT%H:%M:%S.%f')
@@ -427,9 +427,9 @@ class WebInterface(object):
            myDB = db.DBConnection()
            for artist in args:
                myDB.action('DELETE FROM newartists WHERE ArtistName=?',
-                            [artist.decode(headphones.SYS_ENCODING, 'replace')])
+                            [artist])
                myDB.action('UPDATE have SET Matched="Ignored" WHERE ArtistName=?',
-                            [artist.decode(headphones.SYS_ENCODING, 'replace')])
+                            [artist])
                logger.info("Artist %s removed from new artist list and set to ignored" % artist)
        raise cherrypy.HTTPRedirect("home")

@@ -1183,6 +1183,7 @@ class WebInterface(object):
            "deluge_password": headphones.CONFIG.DELUGE_PASSWORD,
            "deluge_label": headphones.CONFIG.DELUGE_LABEL,
            "deluge_done_directory": headphones.CONFIG.DELUGE_DONE_DIRECTORY,
+            "deluge_download_directory": headphones.CONFIG.DELUGE_DOWNLOAD_DIRECTORY,
            "deluge_paused": checked(headphones.CONFIG.DELUGE_PAUSED),
            "utorrent_host": headphones.CONFIG.UTORRENT_HOST,
            "utorrent_username": headphones.CONFIG.UTORRENT_USERNAME,
@@ -1197,6 +1198,8 @@ class WebInterface(object):
            "torrent_downloader_deluge": radio(headphones.CONFIG.TORRENT_DOWNLOADER, 3),
            "torrent_downloader_qbittorrent": radio(headphones.CONFIG.TORRENT_DOWNLOADER, 4),
            "download_dir": headphones.CONFIG.DOWNLOAD_DIR,
+            "soulseek_download_dir": headphones.CONFIG.SOULSEEK_DOWNLOAD_DIR,
+            "soulseek_incomplete_download_dir": headphones.CONFIG.SOULSEEK_INCOMPLETE_DOWNLOAD_DIR,
            "use_blackhole": checked(headphones.CONFIG.BLACKHOLE),
            "blackhole_dir": headphones.CONFIG.BLACKHOLE_DIR,
            "usenet_retention": headphones.CONFIG.USENET_RETENTION,
@@ -1228,13 +1231,6 @@ class WebInterface(object):
            "use_piratebay": checked(headphones.CONFIG.PIRATEBAY),
            "piratebay_proxy_url": headphones.CONFIG.PIRATEBAY_PROXY_URL,
            "piratebay_ratio": headphones.CONFIG.PIRATEBAY_RATIO,
-            "use_oldpiratebay": checked(headphones.CONFIG.OLDPIRATEBAY),
-            "oldpiratebay_url": headphones.CONFIG.OLDPIRATEBAY_URL,
-            "oldpiratebay_ratio": headphones.CONFIG.OLDPIRATEBAY_RATIO,
-            "use_waffles": checked(headphones.CONFIG.WAFFLES),
-            "waffles_uid": headphones.CONFIG.WAFFLES_UID,
-            "waffles_passkey": headphones.CONFIG.WAFFLES_PASSKEY,
-            "waffles_ratio": headphones.CONFIG.WAFFLES_RATIO,
            "use_rutracker": checked(headphones.CONFIG.RUTRACKER),
            "rutracker_user": headphones.CONFIG.RUTRACKER_USER,
            "rutracker_password": headphones.CONFIG.RUTRACKER_PASSWORD,
@@ -1246,6 +1242,7 @@ class WebInterface(object):
            "orpheus_ratio": headphones.CONFIG.ORPHEUS_RATIO,
            "orpheus_url": headphones.CONFIG.ORPHEUS_URL,
            "use_redacted": checked(headphones.CONFIG.REDACTED),
+            "redacted_apikey": headphones.CONFIG.REDACTED_APIKEY,
            "redacted_username": headphones.CONFIG.REDACTED_USERNAME,
            "redacted_password": headphones.CONFIG.REDACTED_PASSWORD,
            "redacted_ratio": headphones.CONFIG.REDACTED_RATIO,
@@ -1296,6 +1293,7 @@ class WebInterface(object):
            "prefer_torrents_0": radio(headphones.CONFIG.PREFER_TORRENTS, 0),
            "prefer_torrents_1": radio(headphones.CONFIG.PREFER_TORRENTS, 1),
            "prefer_torrents_2": radio(headphones.CONFIG.PREFER_TORRENTS, 2),
+            "prefer_torrents_3": radio(headphones.CONFIG.PREFER_TORRENTS, 3),
            "magnet_links_0": radio(headphones.CONFIG.MAGNET_LINKS, 0),
            "magnet_links_1": radio(headphones.CONFIG.MAGNET_LINKS, 1),
            "magnet_links_2": radio(headphones.CONFIG.MAGNET_LINKS, 2),
@@ -1318,10 +1316,6 @@ class WebInterface(object):
            "encoder_multicore": checked(headphones.CONFIG.ENCODER_MULTICORE),
            "encoder_multicore_count": int(headphones.CONFIG.ENCODER_MULTICORE_COUNT),
            "delete_lossless_files": checked(headphones.CONFIG.DELETE_LOSSLESS_FILES),
-            "growl_enabled": checked(headphones.CONFIG.GROWL_ENABLED),
-            "growl_onsnatch": checked(headphones.CONFIG.GROWL_ONSNATCH),
-            "growl_host": headphones.CONFIG.GROWL_HOST,
-            "growl_password": headphones.CONFIG.GROWL_PASSWORD,
            "prowl_enabled": checked(headphones.CONFIG.PROWL_ENABLED),
            "prowl_onsnatch": checked(headphones.CONFIG.PROWL_ONSNATCH),
            "prowl_keys": headphones.CONFIG.PROWL_KEYS,
@@ -1371,10 +1365,6 @@ class WebInterface(object):
            "twitter_onsnatch": checked(headphones.CONFIG.TWITTER_ONSNATCH),
            "osx_notify_enabled": checked(headphones.CONFIG.OSX_NOTIFY_ENABLED),
            "osx_notify_onsnatch": checked(headphones.CONFIG.OSX_NOTIFY_ONSNATCH),
-            "osx_notify_app": headphones.CONFIG.OSX_NOTIFY_APP,
-            "boxcar_enabled": checked(headphones.CONFIG.BOXCAR_ENABLED),
-            "boxcar_onsnatch": checked(headphones.CONFIG.BOXCAR_ONSNATCH),
-            "boxcar_token": headphones.CONFIG.BOXCAR_TOKEN,
            "mirrorlist": headphones.MIRRORLIST,
            "mirror": headphones.CONFIG.MIRROR,
            "customhost": headphones.CONFIG.CUSTOMHOST,
@@ -1413,7 +1403,12 @@ class WebInterface(object):
            "join_enabled": checked(headphones.CONFIG.JOIN_ENABLED),
            "join_onsnatch": checked(headphones.CONFIG.JOIN_ONSNATCH),
            "join_apikey": headphones.CONFIG.JOIN_APIKEY,
-            "join_deviceid": headphones.CONFIG.JOIN_DEVICEID
+            "join_deviceid": headphones.CONFIG.JOIN_DEVICEID,
+            "use_bandcamp": checked(headphones.CONFIG.BANDCAMP),
+            "bandcamp_dir": headphones.CONFIG.BANDCAMP_DIR,
+            'soulseek_api_url': headphones.CONFIG.SOULSEEK_API_URL,
+            'soulseek_api_key': headphones.CONFIG.SOULSEEK_API_KEY,
+            'use_soulseek': checked(headphones.CONFIG.SOULSEEK)
        }

        for k, v in config.items():
@@ -1458,8 +1453,7 @@ class WebInterface(object):
        checked_configs = [
            "launch_browser", "enable_https", "api_enabled", "use_blackhole", "headphones_indexer",
            "use_newznab", "newznab_enabled", "use_torznab", "torznab_enabled",
-            "use_nzbsorg", "use_omgwtfnzbs", "use_piratebay", "use_oldpiratebay",
-            "use_waffles", "use_rutracker",
+            "use_nzbsorg", "use_omgwtfnzbs", "use_piratebay", "use_rutracker",
            "use_orpheus", "use_redacted", "redacted_use_fltoken", "preferred_bitrate_allow_lossless",
            "detect_bitrate", "ignore_clean_releases", "freeze_db", "cue_split", "move_files",
            "rename_files", "rename_single_disc_ignore", "correct_metadata", "cleanup_files",
@@ -1469,20 +1463,20 @@ class WebInterface(object):
            "wait_until_release_date", "autowant_upcoming", "autowant_all",
            "autowant_manually_added", "do_not_process_unmatched", "keep_torrent_files",
            "music_encoder", "mb_ignore_age_missing",
-            "encoderlossless", "encoder_multicore", "delete_lossless_files", "growl_enabled",
-            "growl_onsnatch", "prowl_enabled",
-            "prowl_onsnatch", "xbmc_enabled", "xbmc_update", "xbmc_notify", "lms_enabled",
+            "encoderlossless", "encoder_multicore", "delete_lossless_files",
+            "prowl_enabled", "prowl_onsnatch",
+            "xbmc_enabled", "xbmc_update", "xbmc_notify", "lms_enabled",
            "plex_enabled", "plex_update", "plex_notify",
            "nma_enabled", "nma_onsnatch", "pushalot_enabled", "pushalot_onsnatch",
            "synoindex_enabled", "pushover_enabled",
            "pushover_onsnatch", "pushbullet_enabled", "pushbullet_onsnatch", "subsonic_enabled",
            "twitter_enabled", "twitter_onsnatch",
            "telegram_enabled", "telegram_onsnatch",
-            "osx_notify_enabled", "osx_notify_onsnatch", "boxcar_enabled", "boxcar_onsnatch",
+            "osx_notify_enabled", "osx_notify_onsnatch",
            "songkick_enabled", "songkick_filter_enabled",
            "mpc_enabled", "email_enabled", "email_ssl", "email_tls", "email_onsnatch",
            "customauth", "idtag", "deluge_paused",
-            "join_enabled", "join_onsnatch"
+            "join_enabled", "join_onsnatch", "use_bandcamp", "use_soulseek"
        ]
        for checked_config in checked_configs:
            if checked_config not in kwargs:
@@ -1716,20 +1710,6 @@ class WebInterface(object):
        else:
            return "Error sending tweet"

-    @cherrypy.expose
-    def osxnotifyregister(self, app):
-        cherrypy.response.headers['Cache-Control'] = "max-age=0,no-cache,no-store"
-        from osxnotify import registerapp as osxnotify
-        result, msg = osxnotify.registerapp(app)
-        if result:
-            osx_notify = notifiers.OSX_NOTIFY()
-            osx_notify.notify('Registered', result, 'Success :-)')
-            logger.info(
-                'Registered %s, to re-register a different app, delete this app first' % result)
-        else:
-            logger.warn(msg)
-        return msg
-
    @cherrypy.expose
    def testPushover(self):
        logger.info("Sending Pushover notification")
@@ -1,5 +1,10 @@
-version_info = (3, 0, 1)
-version = '3.0.1'
-release = '3.0.1'
+from pkg_resources import get_distribution, DistributionNotFound

-__version__ = release  # PEP 396
+try:
+    release = get_distribution('APScheduler').version.split('-')[0]
+except DistributionNotFound:
+    release = '3.5.0'
+
+version_info = tuple(int(x) if x.isdigit() else x for x in release.split('.'))
+version = __version__ = '.'.join(str(x) for x in version_info[:3])
+del get_distribution, DistributionNotFound
@@ -1,25 +1,33 @@
-__all__ = ('EVENT_SCHEDULER_START', 'EVENT_SCHEDULER_SHUTDOWN', 'EVENT_EXECUTOR_ADDED', 'EVENT_EXECUTOR_REMOVED',
-           'EVENT_JOBSTORE_ADDED', 'EVENT_JOBSTORE_REMOVED', 'EVENT_ALL_JOBS_REMOVED', 'EVENT_JOB_ADDED',
-           'EVENT_JOB_REMOVED', 'EVENT_JOB_MODIFIED', 'EVENT_JOB_EXECUTED', 'EVENT_JOB_ERROR', 'EVENT_JOB_MISSED',
-           'SchedulerEvent', 'JobEvent', 'JobExecutionEvent')
+__all__ = ('EVENT_SCHEDULER_STARTED', 'EVENT_SCHEDULER_SHUTDOWN', 'EVENT_SCHEDULER_PAUSED',
+           'EVENT_SCHEDULER_RESUMED', 'EVENT_EXECUTOR_ADDED', 'EVENT_EXECUTOR_REMOVED',
+           'EVENT_JOBSTORE_ADDED', 'EVENT_JOBSTORE_REMOVED', 'EVENT_ALL_JOBS_REMOVED',
+           'EVENT_JOB_ADDED', 'EVENT_JOB_REMOVED', 'EVENT_JOB_MODIFIED', 'EVENT_JOB_EXECUTED',
+           'EVENT_JOB_ERROR', 'EVENT_JOB_MISSED', 'EVENT_JOB_SUBMITTED', 'EVENT_JOB_MAX_INSTANCES',
+           'SchedulerEvent', 'JobEvent', 'JobExecutionEvent', 'JobSubmissionEvent')


-EVENT_SCHEDULER_START = 1
-EVENT_SCHEDULER_SHUTDOWN = 2
-EVENT_EXECUTOR_ADDED = 4
-EVENT_EXECUTOR_REMOVED = 8
-EVENT_JOBSTORE_ADDED = 16
-EVENT_JOBSTORE_REMOVED = 32
-EVENT_ALL_JOBS_REMOVED = 64
-EVENT_JOB_ADDED = 128
-EVENT_JOB_REMOVED = 256
-EVENT_JOB_MODIFIED = 512
-EVENT_JOB_EXECUTED = 1024
-EVENT_JOB_ERROR = 2048
-EVENT_JOB_MISSED = 4096
-EVENT_ALL = (EVENT_SCHEDULER_START | EVENT_SCHEDULER_SHUTDOWN | EVENT_JOBSTORE_ADDED | EVENT_JOBSTORE_REMOVED |
+EVENT_SCHEDULER_STARTED = EVENT_SCHEDULER_START = 2 ** 0
+EVENT_SCHEDULER_SHUTDOWN = 2 ** 1
+EVENT_SCHEDULER_PAUSED = 2 ** 2
+EVENT_SCHEDULER_RESUMED = 2 ** 3
+EVENT_EXECUTOR_ADDED = 2 ** 4
+EVENT_EXECUTOR_REMOVED = 2 ** 5
+EVENT_JOBSTORE_ADDED = 2 ** 6
+EVENT_JOBSTORE_REMOVED = 2 ** 7
+EVENT_ALL_JOBS_REMOVED = 2 ** 8
+EVENT_JOB_ADDED = 2 ** 9
+EVENT_JOB_REMOVED = 2 ** 10
+EVENT_JOB_MODIFIED = 2 ** 11
+EVENT_JOB_EXECUTED = 2 ** 12
+EVENT_JOB_ERROR = 2 ** 13
+EVENT_JOB_MISSED = 2 ** 14
+EVENT_JOB_SUBMITTED = 2 ** 15
+EVENT_JOB_MAX_INSTANCES = 2 ** 16
+EVENT_ALL = (EVENT_SCHEDULER_STARTED | EVENT_SCHEDULER_SHUTDOWN | EVENT_SCHEDULER_PAUSED |
+             EVENT_SCHEDULER_RESUMED | EVENT_EXECUTOR_ADDED | EVENT_EXECUTOR_REMOVED |
+             EVENT_JOBSTORE_ADDED | EVENT_JOBSTORE_REMOVED | EVENT_ALL_JOBS_REMOVED |
             EVENT_JOB_ADDED | EVENT_JOB_REMOVED | EVENT_JOB_MODIFIED | EVENT_JOB_EXECUTED |
-             EVENT_JOB_ERROR | EVENT_JOB_MISSED)
+             EVENT_JOB_ERROR | EVENT_JOB_MISSED | EVENT_JOB_SUBMITTED | EVENT_JOB_MAX_INSTANCES)


 class SchedulerEvent(object):
@@ -55,9 +63,21 @@ class JobEvent(SchedulerEvent):
        self.jobstore = jobstore


+class JobSubmissionEvent(JobEvent):
+    """
+    An event that concerns the submission of a job to its executor.
+
+    :ivar scheduled_run_times: a list of datetimes when the job was intended to run
+    """
+
+    def __init__(self, code, job_id, jobstore, scheduled_run_times):
+        super(JobSubmissionEvent, self).__init__(code, job_id, jobstore)
+        self.scheduled_run_times = scheduled_run_times
+
+
 class JobExecutionEvent(JobEvent):
    """
-    An event that concerns the execution of individual jobs.
+    An event that concerns the running of a job within its executor.

    :ivar scheduled_run_time: the time when the job was scheduled to be run
    :ivar retval: the return value of the successfully executed job
@@ -65,7 +85,8 @@ class JobExecutionEvent(JobEvent):
    :ivar traceback: a formatted traceback for the exception
    """

-    def __init__(self, code, job_id, jobstore, scheduled_run_time, retval=None, exception=None, traceback=None):
+    def __init__(self, code, job_id, jobstore, scheduled_run_time, retval=None, exception=None,
+                 traceback=None):
        super(JobExecutionEvent, self).__init__(code, job_id, jobstore)
        self.scheduled_run_time = scheduled_run_time
        self.retval = retval
@@ -1,28 +1,52 @@
+from __future__ import absolute_import

 import sys

 from apscheduler.executors.base import BaseExecutor, run_job
+from apscheduler.executors.base_py3 import run_coroutine_job
+from apscheduler.util import iscoroutinefunction_partial


 class AsyncIOExecutor(BaseExecutor):
    """
    Runs jobs in the default executor of the event loop.

+    If the job function is a native coroutine function, it is scheduled to be run directly in the
+    event loop as soon as possible. All other functions are run in the event loop's default
+    executor which is usually a thread pool.
+
    Plugin alias: ``asyncio``
    """

    def start(self, scheduler, alias):
        super(AsyncIOExecutor, self).start(scheduler, alias)
        self._eventloop = scheduler._eventloop
+        self._pending_futures = set()
+
+    def shutdown(self, wait=True):
+        # There is no way to honor wait=True without converting this method into a coroutine method
+        for f in self._pending_futures:
+            if not f.done():
+                f.cancel()
+
+        self._pending_futures.clear()

    def _do_submit_job(self, job, run_times):
        def callback(f):
+            self._pending_futures.discard(f)
            try:
                events = f.result()
-            except:
+            except BaseException:
                self._run_job_error(job.id, *sys.exc_info()[1:])
            else:
                self._run_job_success(job.id, events)

-        f = self._eventloop.run_in_executor(None, run_job, job, job._jobstore_alias, run_times, self._logger.name)
+        if iscoroutinefunction_partial(job.func):
+            coro = run_coroutine_job(job, job._jobstore_alias, run_times, self._logger.name)
+            f = self._eventloop.create_task(coro)
+        else:
+            f = self._eventloop.run_in_executor(None, run_job, job, job._jobstore_alias, run_times,
+                                                self._logger.name)
+
        f.add_done_callback(callback)
+        self._pending_futures.add(f)
@@ -8,13 +8,15 @@ import sys
 from pytz import utc
 import six

-from apscheduler.events import JobExecutionEvent, EVENT_JOB_MISSED, EVENT_JOB_ERROR, EVENT_JOB_EXECUTED
+from apscheduler.events import (
+    JobExecutionEvent, EVENT_JOB_MISSED, EVENT_JOB_ERROR, EVENT_JOB_EXECUTED)


 class MaxInstancesReachedError(Exception):
    def __init__(self, job):
        super(MaxInstancesReachedError, self).__init__(
-            'Job "%s" has already reached its maximum number of instances (%d)' % (job.id, job.max_instances))
+            'Job "%s" has already reached its maximum number of instances (%d)' %
+            (job.id, job.max_instances))


 class BaseExecutor(six.with_metaclass(ABCMeta, object)):
@@ -30,13 +32,14 @@ class BaseExecutor(six.with_metaclass(ABCMeta, object)):

    def start(self, scheduler, alias):
        """
-        Called by the scheduler when the scheduler is being started or when the executor is being added to an already
-        running scheduler.
+        Called by the scheduler when the scheduler is being started or when the executor is being
+        added to an already running scheduler.

-        :param apscheduler.schedulers.base.BaseScheduler scheduler: the scheduler that is starting this executor
+        :param apscheduler.schedulers.base.BaseScheduler scheduler: the scheduler that is starting
+            this executor
        :param str|unicode alias: alias of this executor as it was assigned to the scheduler
-        """

+        """
        self._scheduler = scheduler
        self._lock = scheduler._create_lock()
        self._logger = logging.getLogger('apscheduler.executors.%s' % alias)
@@ -45,7 +48,8 @@ class BaseExecutor(six.with_metaclass(ABCMeta, object)):
        """
        Shuts down this executor.

-        :param bool wait: ``True`` to wait until all submitted jobs have been executed
+        :param bool wait: ``True`` to wait until all submitted jobs
+            have been executed
        """

    def submit_job(self, job, run_times):
@@ -53,10 +57,12 @@ class BaseExecutor(six.with_metaclass(ABCMeta, object)):
        Submits job for execution.

        :param Job job: job to execute
-        :param list[datetime] run_times: list of datetimes specifying when the job should have been run
-        :raises MaxInstancesReachedError: if the maximum number of allowed instances for this job has been reached
-        """
+        :param list[datetime] run_times: list of datetimes specifying
+            when the job should have been run
+        :raises MaxInstancesReachedError: if the maximum number of
+            allowed instances for this job has been reached

+        """
        assert self._lock is not None, 'This executor has not been started yet'
        with self._lock:
            if self._instances[job.id] >= job.max_instances:
@@ -70,50 +76,71 @@ class BaseExecutor(six.with_metaclass(ABCMeta, object)):
        """Performs the actual task of scheduling `run_job` to be called."""

    def _run_job_success(self, job_id, events):
-        """Called by the executor with the list of generated events when `run_job` has been successfully called."""
+        """
+        Called by the executor with the list of generated events when :func:`run_job` has been
+        successfully called.

+        """
        with self._lock:
            self._instances[job_id] -= 1
+            if self._instances[job_id] == 0:
+                del self._instances[job_id]

        for event in events:
            self._scheduler._dispatch_event(event)

    def _run_job_error(self, job_id, exc, traceback=None):
        """Called by the executor with the exception if there is an error  calling `run_job`."""
-
        with self._lock:
            self._instances[job_id] -= 1
+            if self._instances[job_id] == 0:
+                del self._instances[job_id]

        exc_info = (exc.__class__, exc, traceback)
        self._logger.error('Error running job %s', job_id, exc_info=exc_info)


 def run_job(job, jobstore_alias, run_times, logger_name):
-    """Called by executors to run the job. Returns a list of scheduler events to be dispatched by the scheduler."""
+    """
+    Called by executors to run the job. Returns a list of scheduler events to be dispatched by the
+    scheduler.

+    """
    events = []
    logger = logging.getLogger(logger_name)
    for run_time in run_times:
-        # See if the job missed its run time window, and handle possible misfires accordingly
+        # See if the job missed its run time window, and handle
+        # possible misfires accordingly
        if job.misfire_grace_time is not None:
            difference = datetime.now(utc) - run_time
            grace_time = timedelta(seconds=job.misfire_grace_time)
            if difference > grace_time:
-                events.append(JobExecutionEvent(EVENT_JOB_MISSED, job.id, jobstore_alias, run_time))
+                events.append(JobExecutionEvent(EVENT_JOB_MISSED, job.id, jobstore_alias,
+                                                run_time))
                logger.warning('Run time of job "%s" was missed by %s', job, difference)
                continue

        logger.info('Running job "%s" (scheduled at %s)', job, run_time)
        try:
            retval = job.func(*job.args, **job.kwargs)
-        except:
+        except BaseException:
            exc, tb = sys.exc_info()[1:]
            formatted_tb = ''.join(format_tb(tb))
-            events.append(JobExecutionEvent(EVENT_JOB_ERROR, job.id, jobstore_alias, run_time, exception=exc,
-                                            traceback=formatted_tb))
+            events.append(JobExecutionEvent(EVENT_JOB_ERROR, job.id, jobstore_alias, run_time,
+                                            exception=exc, traceback=formatted_tb))
            logger.exception('Job "%s" raised an exception', job)
+
+            # This is to prevent cyclic references that would lead to memory leaks
+            if six.PY2:
+                sys.exc_clear()
+                del tb
            else:
-            events.append(JobExecutionEvent(EVENT_JOB_EXECUTED, job.id, jobstore_alias, run_time, retval=retval))
+                import traceback
+                traceback.clear_frames(tb)
+                del tb
+        else:
+            events.append(JobExecutionEvent(EVENT_JOB_EXECUTED, job.id, jobstore_alias, run_time,
+                                            retval=retval))
            logger.info('Job "%s" executed successfully', job)

    return events
@@ -0,0 +1,43 @@
+import logging
+import sys
+import traceback
+from datetime import datetime, timedelta
+from traceback import format_tb
+
+from pytz import utc
+
+from apscheduler.events import (
+    JobExecutionEvent, EVENT_JOB_MISSED, EVENT_JOB_ERROR, EVENT_JOB_EXECUTED)
+
+
+async def run_coroutine_job(job, jobstore_alias, run_times, logger_name):
+    """Coroutine version of run_job()."""
+    events = []
+    logger = logging.getLogger(logger_name)
+    for run_time in run_times:
+        # See if the job missed its run time window, and handle possible misfires accordingly
+        if job.misfire_grace_time is not None:
+            difference = datetime.now(utc) - run_time
+            grace_time = timedelta(seconds=job.misfire_grace_time)
+            if difference > grace_time:
+                events.append(JobExecutionEvent(EVENT_JOB_MISSED, job.id, jobstore_alias,
+                                                run_time))
+                logger.warning('Run time of job "%s" was missed by %s', job, difference)
+                continue
+
+        logger.info('Running job "%s" (scheduled at %s)', job, run_time)
+        try:
+            retval = await job.func(*job.args, **job.kwargs)
+        except BaseException:
+            exc, tb = sys.exc_info()[1:]
+            formatted_tb = ''.join(format_tb(tb))
+            events.append(JobExecutionEvent(EVENT_JOB_ERROR, job.id, jobstore_alias, run_time,
+                                            exception=exc, traceback=formatted_tb))
+            logger.exception('Job "%s" raised an exception', job)
+            traceback.clear_frames(tb)
+        else:
+            events.append(JobExecutionEvent(EVENT_JOB_EXECUTED, job.id, jobstore_alias, run_time,
+                                            retval=retval))
+            logger.info('Job "%s" executed successfully', job)
+
+    return events
@@ -5,7 +5,8 @@ from apscheduler.executors.base import BaseExecutor, run_job

 class DebugExecutor(BaseExecutor):
    """
-    A special executor that executes the target callable directly instead of deferring it to a thread or process.
+    A special executor that executes the target callable directly instead of deferring it to a
+    thread or process.

    Plugin alias: ``debug``
    """
@@ -13,7 +14,7 @@ class DebugExecutor(BaseExecutor):
    def _do_submit_job(self, job, run_times):
        try:
            events = run_job(job, job._jobstore_alias, run_times, self._logger.name)
-        except:
+        except BaseException:
            self._run_job_error(job.id, *sys.exc_info()[1:])
        else:
            self._run_job_success(job.id, events)
@@ -1,4 +1,4 @@
-
+from __future__ import absolute_import
 import sys

 from apscheduler.executors.base import BaseExecutor, run_job
@@ -21,9 +21,10 @@ class GeventExecutor(BaseExecutor):
        def callback(greenlet):
            try:
                events = greenlet.get()
-            except:
+            except BaseException:
                self._run_job_error(job.id, *sys.exc_info()[1:])
            else:
                self._run_job_success(job.id, events)

-        gevent.spawn(run_job, job, job._jobstore_alias, run_times, self._logger.name).link(callback)
+        gevent.spawn(run_job, job, job._jobstore_alias, run_times, self._logger.name).\
+            link(callback)
@@ -3,6 +3,11 @@ import concurrent.futures

 from apscheduler.executors.base import BaseExecutor, run_job

+try:
+    from concurrent.futures.process import BrokenProcessPool
+except ImportError:
+    BrokenProcessPool = None
+

 class BasePoolExecutor(BaseExecutor):
    @abstractmethod
@@ -19,7 +24,13 @@ class BasePoolExecutor(BaseExecutor):
            else:
                self._run_job_success(job.id, f.result())

+        try:
            f = self._pool.submit(run_job, job, job._jobstore_alias, run_times, self._logger.name)
+        except BrokenProcessPool:
+            self._logger.warning('Process pool is broken; replacing pool with a fresh instance')
+            self._pool = self._pool.__class__(self._pool._max_workers)
+            f = self._pool.submit(run_job, job, job._jobstore_alias, run_times, self._logger.name)
+
        f.add_done_callback(callback)

    def shutdown(self, wait=True):
@@ -33,10 +44,13 @@ class ThreadPoolExecutor(BasePoolExecutor):
    Plugin alias: ``threadpool``

    :param max_workers: the maximum number of spawned threads.
+    :param pool_kwargs: dict of keyword arguments to pass to the underlying
+        ThreadPoolExecutor constructor
    """

-    def __init__(self, max_workers=10):
-        pool = concurrent.futures.ThreadPoolExecutor(int(max_workers))
+    def __init__(self, max_workers=10, pool_kwargs=None):
+        pool_kwargs = pool_kwargs or {}
+        pool = concurrent.futures.ThreadPoolExecutor(int(max_workers), **pool_kwargs)
        super(ThreadPoolExecutor, self).__init__(pool)


@@ -47,8 +61,11 @@ class ProcessPoolExecutor(BasePoolExecutor):
    Plugin alias: ``processpool``

    :param max_workers: the maximum number of spawned processes.
+    :param pool_kwargs: dict of keyword arguments to pass to the underlying
+        ProcessPoolExecutor constructor
    """

-    def __init__(self, max_workers=10):
-        pool = concurrent.futures.ProcessPoolExecutor(int(max_workers))
+    def __init__(self, max_workers=10, pool_kwargs=None):
+        pool_kwargs = pool_kwargs or {}
+        pool = concurrent.futures.ProcessPoolExecutor(int(max_workers), **pool_kwargs)
        super(ProcessPoolExecutor, self).__init__(pool)
@@ -0,0 +1,54 @@
+from __future__ import absolute_import
+
+import sys
+from concurrent.futures import ThreadPoolExecutor
+
+from tornado.gen import convert_yielded
+
+from apscheduler.executors.base import BaseExecutor, run_job
+
+try:
+    from apscheduler.executors.base_py3 import run_coroutine_job
+    from apscheduler.util import iscoroutinefunction_partial
+except ImportError:
+    def iscoroutinefunction_partial(func):
+        return False
+
+
+class TornadoExecutor(BaseExecutor):
+    """
+    Runs jobs either in a thread pool or directly on the I/O loop.
+
+    If the job function is a native coroutine function, it is scheduled to be run directly in the
+    I/O loop as soon as possible. All other functions are run in a thread pool.
+
+    Plugin alias: ``tornado``
+
+    :param int max_workers: maximum number of worker threads in the thread pool
+    """
+
+    def __init__(self, max_workers=10):
+        super(TornadoExecutor, self).__init__()
+        self.executor = ThreadPoolExecutor(max_workers)
+
+    def start(self, scheduler, alias):
+        super(TornadoExecutor, self).start(scheduler, alias)
+        self._ioloop = scheduler._ioloop
+
+    def _do_submit_job(self, job, run_times):
+        def callback(f):
+            try:
+                events = f.result()
+            except BaseException:
+                self._run_job_error(job.id, *sys.exc_info()[1:])
+            else:
+                self._run_job_success(job.id, events)
+
+        if iscoroutinefunction_partial(job.func):
+            f = run_coroutine_job(job, job._jobstore_alias, run_times, self._logger.name)
+        else:
+            f = self.executor.submit(run_job, job, job._jobstore_alias, run_times,
+                                     self._logger.name)
+
+        f = convert_yielded(f)
+        f.add_done_callback(callback)
@@ -1,4 +1,4 @@
-
+from __future__ import absolute_import

 from apscheduler.executors.base import BaseExecutor, run_job

@@ -21,5 +21,5 @@ class TwistedExecutor(BaseExecutor):
            else:
                self._run_job_error(job.id, result.value, result.tb)

-        self._reactor.getThreadPool().callInThreadWithCallback(callback, run_job, job, job._jobstore_alias, run_times,
-                                                               self._logger.name)
+        self._reactor.getThreadPool().callInThreadWithCallback(
+            callback, run_job, job, job._jobstore_alias, run_times, self._logger.name)
@@ -1,11 +1,17 @@
-from collections.abc import Iterable, Mapping
+from inspect import ismethod, isclass
 from uuid import uuid4

 import six

 from apscheduler.triggers.base import BaseTrigger
-from apscheduler.util import ref_to_obj, obj_to_ref, datetime_repr, repr_escape, get_callable_name, check_callable_args, \
-    convert_to_datetime
+from apscheduler.util import (
+    ref_to_obj, obj_to_ref, datetime_repr, repr_escape, get_callable_name, check_callable_args,
+    convert_to_datetime)
+
+try:
+    from collections.abc import Iterable, Mapping
+except ImportError:
+    from collections import Iterable, Mapping


 class Job(object):
@@ -21,13 +27,20 @@ class Job(object):
    :var bool coalesce: whether to only run the job once when several run times are due
    :var trigger: the trigger object that controls the schedule of this job
    :var str executor: the name of the executor that will run this job
-    :var int misfire_grace_time: the time (in seconds) how much this job's execution is allowed to be late
-    :var int max_instances: the maximum number of concurrently executing instances allowed for this job
+    :var int misfire_grace_time: the time (in seconds) how much this job's execution is allowed to
+        be late (``None`` means "allow the job to run no matter how late it is")
+    :var int max_instances: the maximum number of concurrently executing instances allowed for this
+        job
    :var datetime.datetime next_run_time: the next scheduled run time of this job
+
+    .. note::
+        The ``misfire_grace_time`` has some non-obvious effects on job execution. See the
+        :ref:`missed-job-executions` section in the documentation for an in-depth explanation.
    """

-    __slots__ = ('_scheduler', '_jobstore_alias', 'id', 'trigger', 'executor', 'func', 'func_ref', 'args', 'kwargs',
-                 'name', 'misfire_grace_time', 'coalesce', 'max_instances', 'next_run_time')
+    __slots__ = ('_scheduler', '_jobstore_alias', 'id', 'trigger', 'executor', 'func', 'func_ref',
+                 'args', 'kwargs', 'name', 'misfire_grace_time', 'coalesce', 'max_instances',
+                 'next_run_time', '__weakref__')

    def __init__(self, scheduler, id=None, **kwargs):
        super(Job, self).__init__()
@@ -38,53 +51,69 @@ class Job(object):
    def modify(self, **changes):
        """
        Makes the given changes to this job and saves it in the associated job store.
+
        Accepted keyword arguments are the same as the variables on this class.

        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.modify_job`
-        """

+        :return Job: this job instance
+
+        """
        self._scheduler.modify_job(self.id, self._jobstore_alias, **changes)
+        return self

    def reschedule(self, trigger, **trigger_args):
        """
        Shortcut for switching the trigger on this job.

        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.reschedule_job`
-        """

+        :return Job: this job instance
+
+        """
        self._scheduler.reschedule_job(self.id, self._jobstore_alias, trigger, **trigger_args)
+        return self

    def pause(self):
        """
        Temporarily suspend the execution of this job.

        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.pause_job`
-        """

+        :return Job: this job instance
+
+        """
        self._scheduler.pause_job(self.id, self._jobstore_alias)
+        return self

    def resume(self):
        """
        Resume the schedule of this job if previously paused.

        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.resume_job`
-        """

+        :return Job: this job instance
+
+        """
        self._scheduler.resume_job(self.id, self._jobstore_alias)
+        return self

    def remove(self):
        """
        Unschedules this job and removes it from its associated job store.

        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.remove_job`
-        """

+        """
        self._scheduler.remove_job(self.id, self._jobstore_alias)

    @property
    def pending(self):
-        """Returns ``True`` if the referenced job is still waiting to be added to its designated job store."""
+        """
+        Returns ``True`` if the referenced job is still waiting to be added to its designated job
+        store.

+        """
        return self._jobstore_alias is None

    #
@@ -97,8 +126,8 @@ class Job(object):

        :type now: datetime.datetime
        :rtype: list[datetime.datetime]
-        """

+        """
        run_times = []
        next_run_time = self.next_run_time
        while next_run_time and next_run_time <= now:
@@ -108,8 +137,11 @@ class Job(object):
        return run_times

    def _modify(self, **changes):
-        """Validates the changes to the Job and makes the modifications if and only if all of them validate."""
+        """
+        Validates the changes to the Job and makes the modifications if and only if all of them
+        validate.

+        """
        approved = {}

        if 'id' in changes:
@@ -125,7 +157,7 @@ class Job(object):
            args = changes.pop('args') if 'args' in changes else self.args
            kwargs = changes.pop('kwargs') if 'kwargs' in changes else self.kwargs

-            if isinstance(func, str):
+            if isinstance(func, six.string_types):
                func_ref = func
                func = ref_to_obj(func)
            elif callable(func):
@@ -177,7 +209,8 @@ class Job(object):
        if 'trigger' in changes:
            trigger = changes.pop('trigger')
            if not isinstance(trigger, BaseTrigger):
-                raise TypeError('Expected a trigger instance, got %s instead' % trigger.__class__.__name__)
+                raise TypeError('Expected a trigger instance, got %s instead' %
+                                trigger.__class__.__name__)

            approved['trigger'] = trigger

@@ -189,10 +222,12 @@ class Job(object):

        if 'next_run_time' in changes:
            value = changes.pop('next_run_time')
-            approved['next_run_time'] = convert_to_datetime(value, self._scheduler.timezone, 'next_run_time')
+            approved['next_run_time'] = convert_to_datetime(value, self._scheduler.timezone,
+                                                            'next_run_time')

        if changes:
-            raise AttributeError('The following are not modifiable attributes of Job: %s' % ', '.join(changes))
+            raise AttributeError('The following are not modifiable attributes of Job: %s' %
+                                 ', '.join(changes))

        for key, value in six.iteritems(approved):
            setattr(self, key, value)
@@ -200,9 +235,18 @@ class Job(object):
    def __getstate__(self):
        # Don't allow this Job to be serialized if the function reference could not be determined
        if not self.func_ref:
-            raise ValueError('This Job cannot be serialized since the reference to its callable (%r) could not be '
-                             'determined. Consider giving a textual reference (module:function name) instead.' %
-                             (self.func,))
+            raise ValueError(
+                'This Job cannot be serialized since the reference to its callable (%r) could not '
+                'be determined. Consider giving a textual reference (module:function name) '
+                'instead.' % (self.func,))
+
+        # Instance methods cannot survive serialization as-is, so store the "self" argument
+        # explicitly
+        func = self.func
+        if ismethod(func) and not isclass(func.__self__) and obj_to_ref(func) == self.func_ref:
+            args = (func.__self__,) + tuple(self.args)
+        else:
+            args = self.args

        return {
            'version': 1,
@@ -210,7 +254,7 @@ class Job(object):
            'func': self.func_ref,
            'trigger': self.trigger,
            'executor': self.executor,
-            'args': self.args,
+            'args': args,
            'kwargs': self.kwargs,
            'name': self.name,
            'misfire_grace_time': self.misfire_grace_time,
@@ -221,7 +265,8 @@ class Job(object):

    def __setstate__(self, state):
        if state.get('version', 1) > 1:
-            raise ValueError('Job has version %s, but only version 1 can be handled' % state['version'])
+            raise ValueError('Job has version %s, but only version 1 can be handled' %
+                             state['version'])

        self.id = state['id']
        self.func_ref = state['func']
@@ -245,8 +290,13 @@ class Job(object):
        return '<Job (id=%s name=%s)>' % (repr_escape(self.id), repr_escape(self.name))

    def __str__(self):
-        return '%s (trigger: %s, next run at: %s)' % (repr_escape(self.name), repr_escape(str(self.trigger)),
-                                                      datetime_repr(self.next_run_time))
+        return repr_escape(self.__unicode__())

    def __unicode__(self):
-        return six.u('%s (trigger: %s, next run at: %s)') % (self.name, self.trigger, datetime_repr(self.next_run_time))
+        if hasattr(self, 'next_run_time'):
+            status = ('next run at: ' + datetime_repr(self.next_run_time) if
+                      self.next_run_time else 'paused')
+        else:
+            status = 'pending'
+
+        return u'%s (trigger: %s, %s)' % (self.name, self.trigger, status)
@@ -8,23 +8,27 @@ class JobLookupError(KeyError):
    """Raised when the job store cannot find a job for update or removal."""

    def __init__(self, job_id):
-        super(JobLookupError, self).__init__(six.u('No job by the id of %s was found') % job_id)
+        super(JobLookupError, self).__init__(u'No job by the id of %s was found' % job_id)


 class ConflictingIdError(KeyError):
    """Raised when the uniqueness of job IDs is being violated."""

    def __init__(self, job_id):
-        super(ConflictingIdError, self).__init__(six.u('Job identifier (%s) conflicts with an existing job') % job_id)
+        super(ConflictingIdError, self).__init__(
+            u'Job identifier (%s) conflicts with an existing job' % job_id)


 class TransientJobError(ValueError):
-    """Raised when an attempt to add transient (with no func_ref) job to a persistent job store is detected."""
+    """
+    Raised when an attempt to add transient (with no func_ref) job to a persistent job store is
+    detected.
+    """

    def __init__(self, job_id):
        super(TransientJobError, self).__init__(
-            six.u('Job (%s) cannot be added to this job store because a reference to the callable could not be '
-                  'determined.') % job_id)
+            u'Job (%s) cannot be added to this job store because a reference to the callable '
+            u'could not be determined.' % job_id)


 class BaseJobStore(six.with_metaclass(ABCMeta)):
@@ -36,10 +40,11 @@ class BaseJobStore(six.with_metaclass(ABCMeta)):

    def start(self, scheduler, alias):
        """
-        Called by the scheduler when the scheduler is being started or when the job store is being added to an already
-        running scheduler.
+        Called by the scheduler when the scheduler is being started or when the job store is being
+        added to an already running scheduler.

-        :param apscheduler.schedulers.base.BaseScheduler scheduler: the scheduler that is starting this job store
+        :param apscheduler.schedulers.base.BaseScheduler scheduler: the scheduler that is starting
+            this job store
        :param str|unicode alias: alias of this job store as it was assigned to the scheduler
        """

@@ -50,13 +55,22 @@ class BaseJobStore(six.with_metaclass(ABCMeta)):
    def shutdown(self):
        """Frees any resources still bound to this job store."""

+    def _fix_paused_jobs_sorting(self, jobs):
+        for i, job in enumerate(jobs):
+            if job.next_run_time is not None:
+                if i > 0:
+                    paused_jobs = jobs[:i]
+                    del jobs[:i]
+                    jobs.extend(paused_jobs)
+                break
+
    @abstractmethod
    def lookup_job(self, job_id):
        """
        Returns a specific job, or ``None`` if it isn't found..

-        The job store is responsible for setting the ``scheduler`` and ``jobstore`` attributes of the returned job to
-        point to the scheduler and itself, respectively.
+        The job store is responsible for setting the ``scheduler`` and ``jobstore`` attributes of
+        the returned job to point to the scheduler and itself, respectively.

        :param str|unicode job_id: identifier of the job
        :rtype: Job
@@ -75,7 +89,8 @@ class BaseJobStore(six.with_metaclass(ABCMeta)):
    @abstractmethod
    def get_next_run_time(self):
        """
-        Returns the earliest run time of all the jobs stored in this job store, or ``None`` if there are no active jobs.
+        Returns the earliest run time of all the jobs stored in this job store, or ``None`` if
+        there are no active jobs.

        :rtype: datetime.datetime
        """
@@ -83,11 +98,12 @@ class BaseJobStore(six.with_metaclass(ABCMeta)):
    @abstractmethod
    def get_all_jobs(self):
        """
-        Returns a list of all jobs in this job store. The returned jobs should be sorted by next run time (ascending).
-        Paused jobs (next_run_time is None) should be sorted last.
+        Returns a list of all jobs in this job store.
+        The returned jobs should be sorted by next run time (ascending).
+        Paused jobs (next_run_time == None) should be sorted last.

-        The job store is responsible for setting the ``scheduler`` and ``jobstore`` attributes of the returned jobs to
-        point to the scheduler and itself, respectively.
+        The job store is responsible for setting the ``scheduler`` and ``jobstore`` attributes of
+        the returned jobs to point to the scheduler and itself, respectively.

        :rtype: list[Job]
        """
@@ -1,4 +1,4 @@
-
+from __future__ import absolute_import

 from apscheduler.jobstores.base import BaseJobStore, JobLookupError, ConflictingIdError
 from apscheduler.util import datetime_to_utc_timestamp
@@ -13,7 +13,8 @@ class MemoryJobStore(BaseJobStore):

    def __init__(self):
        super(MemoryJobStore, self).__init__()
-        self._jobs = []  # list of (job, timestamp), sorted by next_run_time and job id (ascending)
+        # list of (job, timestamp), sorted by next_run_time and job id (ascending)
+        self._jobs = []
        self._jobs_index = {}  # id -> (job, timestamp) lookup table

    def lookup_job(self, job_id):
@@ -80,13 +81,13 @@ class MemoryJobStore(BaseJobStore):

    def _get_job_index(self, timestamp, job_id):
        """
-        Returns the index of the given job, or if it's not found, the index where the job should be inserted based on
-        the given timestamp.
+        Returns the index of the given job, or if it's not found, the index where the job should be
+        inserted based on the given timestamp.

        :type timestamp: int
        :type job_id: str
-        """

+        """
        lo, hi = 0, len(self._jobs)
        timestamp = float('inf') if timestamp is None else timestamp
        while lo < hi:
@@ -1,11 +1,12 @@
-
+from __future__ import absolute_import
+import warnings

 from apscheduler.jobstores.base import BaseJobStore, JobLookupError, ConflictingIdError
 from apscheduler.util import maybe_ref, datetime_to_utc_timestamp, utc_timestamp_to_datetime
 from apscheduler.job import Job

 try:
-    import pickle as pickle
+    import cPickle as pickle
 except ImportError:  # pragma: nocover
    import pickle

@@ -19,16 +20,18 @@ except ImportError:  # pragma: nocover

 class MongoDBJobStore(BaseJobStore):
    """
-    Stores jobs in a MongoDB database. Any leftover keyword arguments are directly passed to pymongo's `MongoClient
+    Stores jobs in a MongoDB database. Any leftover keyword arguments are directly passed to
+    pymongo's `MongoClient
    <http://api.mongodb.org/python/current/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient>`_.

    Plugin alias: ``mongodb``

    :param str database: database to store jobs in
    :param str collection: collection to store jobs in
-    :param client: a :class:`~pymongo.mongo_client.MongoClient` instance to use instead of providing connection
-                   arguments
-    :param int pickle_protocol: pickle protocol level to use (for serialization), defaults to the highest available
+    :param client: a :class:`~pymongo.mongo_client.MongoClient` instance to use instead of
+        providing connection arguments
+    :param int pickle_protocol: pickle protocol level to use (for serialization), defaults to the
+        highest available
    """

    def __init__(self, database='apscheduler', collection='jobs', client=None,
@@ -42,13 +45,22 @@ class MongoDBJobStore(BaseJobStore):
            raise ValueError('The "collection" parameter must not be empty')

        if client:
-            self.connection = maybe_ref(client)
+            self.client = maybe_ref(client)
        else:
            connect_args.setdefault('w', 1)
-            self.connection = MongoClient(**connect_args)
+            self.client = MongoClient(**connect_args)

-        self.collection = self.connection[database][collection]
-        self.collection.ensure_index('next_run_time', sparse=True)
+        self.collection = self.client[database][collection]
+
+    def start(self, scheduler, alias):
+        super(MongoDBJobStore, self).start(scheduler, alias)
+        self.collection.create_index('next_run_time', sparse=True)
+
+    @property
+    def connection(self):
+        warnings.warn('The "connection" member is deprecated -- use "client" instead',
+                      DeprecationWarning)
+        return self.client

    def lookup_job(self, job_id):
        document = self.collection.find_one(job_id, ['job_state'])
@@ -59,16 +71,19 @@ class MongoDBJobStore(BaseJobStore):
        return self._get_jobs({'next_run_time': {'$lte': timestamp}})

    def get_next_run_time(self):
-        document = self.collection.find_one({'next_run_time': {'$ne': None}}, fields=['next_run_time'],
+        document = self.collection.find_one({'next_run_time': {'$ne': None}},
+                                            projection=['next_run_time'],
                                            sort=[('next_run_time', ASCENDING)])
        return utc_timestamp_to_datetime(document['next_run_time']) if document else None

    def get_all_jobs(self):
-        return self._get_jobs({})
+        jobs = self._get_jobs({})
+        self._fix_paused_jobs_sorting(jobs)
+        return jobs

    def add_job(self, job):
        try:
-            self.collection.insert({
+            self.collection.insert_one({
                '_id': job.id,
                'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
                'job_state': Binary(pickle.dumps(job.__getstate__(), self.pickle_protocol))
@@ -81,20 +96,20 @@ class MongoDBJobStore(BaseJobStore):
            'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
            'job_state': Binary(pickle.dumps(job.__getstate__(), self.pickle_protocol))
        }
-        result = self.collection.update({'_id': job.id}, {'$set': changes})
-        if result and result['n'] == 0:
-            raise JobLookupError(id)
+        result = self.collection.update_one({'_id': job.id}, {'$set': changes})
+        if result and result.matched_count == 0:
+            raise JobLookupError(job.id)

    def remove_job(self, job_id):
-        result = self.collection.remove(job_id)
-        if result and result['n'] == 0:
+        result = self.collection.delete_one({'_id': job_id})
+        if result and result.deleted_count == 0:
            raise JobLookupError(job_id)

    def remove_all_jobs(self):
-        self.collection.remove()
+        self.collection.delete_many({})

    def shutdown(self):
-        self.connection.disconnect()
+        self.client.close()

    def _reconstitute_job(self, job_state):
        job_state = pickle.loads(job_state)
@@ -107,18 +122,20 @@ class MongoDBJobStore(BaseJobStore):
    def _get_jobs(self, conditions):
        jobs = []
        failed_job_ids = []
-        for document in self.collection.find(conditions, ['_id', 'job_state'], sort=[('next_run_time', ASCENDING)]):
+        for document in self.collection.find(conditions, ['_id', 'job_state'],
+                                             sort=[('next_run_time', ASCENDING)]):
            try:
                jobs.append(self._reconstitute_job(document['job_state']))
-            except:
-                self._logger.exception('Unable to restore job "%s" -- removing it', document['_id'])
+            except BaseException:
+                self._logger.exception('Unable to restore job "%s" -- removing it',
+                                       document['_id'])
                failed_job_ids.append(document['_id'])

        # Remove all the jobs we failed to restore
        if failed_job_ids:
-            self.collection.remove({'_id': {'$in': failed_job_ids}})
+            self.collection.delete_many({'_id': {'$in': failed_job_ids}})

        return jobs

    def __repr__(self):
-        return '<%s (client=%s)>' % (self.__class__.__name__, self.connection)
+        return '<%s (client=%s)>' % (self.__class__.__name__, self.client)
@@ -1,5 +1,7 @@
+from __future__ import absolute_import
+from datetime import datetime

-
+from pytz import utc
 import six

 from apscheduler.jobstores.base import BaseJobStore, JobLookupError, ConflictingIdError
@@ -7,26 +9,28 @@ from apscheduler.util import datetime_to_utc_timestamp, utc_timestamp_to_datetim
 from apscheduler.job import Job

 try:
-    import pickle as pickle
+    import cPickle as pickle
 except ImportError:  # pragma: nocover
    import pickle

 try:
-    from redis import StrictRedis
+    from redis import Redis
 except ImportError:  # pragma: nocover
    raise ImportError('RedisJobStore requires redis installed')


 class RedisJobStore(BaseJobStore):
    """
-    Stores jobs in a Redis database. Any leftover keyword arguments are directly passed to redis's StrictRedis.
+    Stores jobs in a Redis database. Any leftover keyword arguments are directly passed to redis's
+    :class:`~redis.StrictRedis`.

    Plugin alias: ``redis``

    :param int db: the database number to store jobs in
    :param str jobs_key: key to store jobs in
    :param str run_times_key: key to store the jobs' run times in
-    :param int pickle_protocol: pickle protocol level to use (for serialization), defaults to the highest available
+    :param int pickle_protocol: pickle protocol level to use (for serialization), defaults to the
+        highest available
    """

    def __init__(self, db=0, jobs_key='apscheduler.jobs', run_times_key='apscheduler.run_times',
@@ -43,7 +47,7 @@ class RedisJobStore(BaseJobStore):
        self.pickle_protocol = pickle_protocol
        self.jobs_key = jobs_key
        self.run_times_key = run_times_key
-        self.redis = StrictRedis(db=int(db), **connect_args)
+        self.redis = Redis(db=int(db), **connect_args)

    def lookup_job(self, job_id):
        job_state = self.redis.hget(self.jobs_key, job_id)
@@ -65,7 +69,8 @@ class RedisJobStore(BaseJobStore):
    def get_all_jobs(self):
        job_states = self.redis.hgetall(self.jobs_key)
        jobs = self._reconstitute_jobs(six.iteritems(job_states))
-        return sorted(jobs, key=lambda job: job.next_run_time)
+        paused_sort_key = datetime(9999, 12, 31, tzinfo=utc)
+        return sorted(jobs, key=lambda job: job.next_run_time or paused_sort_key)

    def add_job(self, job):
        if self.redis.hexists(self.jobs_key, job.id):
@@ -73,8 +78,12 @@ class RedisJobStore(BaseJobStore):

        with self.redis.pipeline() as pipe:
            pipe.multi()
-            pipe.hset(self.jobs_key, job.id, pickle.dumps(job.__getstate__(), self.pickle_protocol))
-            pipe.zadd(self.run_times_key, datetime_to_utc_timestamp(job.next_run_time), job.id)
+            pipe.hset(self.jobs_key, job.id, pickle.dumps(job.__getstate__(),
+                                                          self.pickle_protocol))
+            if job.next_run_time:
+                pipe.zadd(self.run_times_key,
+                          {job.id: datetime_to_utc_timestamp(job.next_run_time)})
+
            pipe.execute()

    def update_job(self, job):
@@ -82,11 +91,14 @@ class RedisJobStore(BaseJobStore):
            raise JobLookupError(job.id)

        with self.redis.pipeline() as pipe:
-            pipe.hset(self.jobs_key, job.id, pickle.dumps(job.__getstate__(), self.pickle_protocol))
+            pipe.hset(self.jobs_key, job.id, pickle.dumps(job.__getstate__(),
+                                                          self.pickle_protocol))
            if job.next_run_time:
-                pipe.zadd(self.run_times_key, datetime_to_utc_timestamp(job.next_run_time), job.id)
+                pipe.zadd(self.run_times_key,
+                          {job.id: datetime_to_utc_timestamp(job.next_run_time)})
            else:
                pipe.zrem(self.run_times_key, job.id)
+
            pipe.execute()

    def remove_job(self, job_id):
@@ -121,7 +133,7 @@ class RedisJobStore(BaseJobStore):
        for job_id, job_state in job_states:
            try:
                jobs.append(self._reconstitute_job(job_state))
-            except:
+            except BaseException:
                self._logger.exception('Unable to restore job "%s" -- removing it', job_id)
                failed_job_ids.append(job_id)

@@ -0,0 +1,155 @@
+from __future__ import absolute_import
+
+from apscheduler.jobstores.base import BaseJobStore, JobLookupError, ConflictingIdError
+from apscheduler.util import maybe_ref, datetime_to_utc_timestamp, utc_timestamp_to_datetime
+from apscheduler.job import Job
+
+try:
+    import cPickle as pickle
+except ImportError:  # pragma: nocover
+    import pickle
+
+try:
+    from rethinkdb import RethinkDB
+except ImportError:  # pragma: nocover
+    raise ImportError('RethinkDBJobStore requires rethinkdb installed')
+
+
+class RethinkDBJobStore(BaseJobStore):
+    """
+    Stores jobs in a RethinkDB database. Any leftover keyword arguments are directly passed to
+    rethinkdb's `RethinkdbClient <http://www.rethinkdb.com/api/#connect>`_.
+
+    Plugin alias: ``rethinkdb``
+
+    :param str database: database to store jobs in
+    :param str collection: collection to store jobs in
+    :param client: a :class:`rethinkdb.net.Connection` instance to use instead of providing
+        connection arguments
+    :param int pickle_protocol: pickle protocol level to use (for serialization), defaults to the
+        highest available
+    """
+
+    def __init__(self, database='apscheduler', table='jobs', client=None,
+                 pickle_protocol=pickle.HIGHEST_PROTOCOL, **connect_args):
+        super(RethinkDBJobStore, self).__init__()
+
+        if not database:
+            raise ValueError('The "database" parameter must not be empty')
+        if not table:
+            raise ValueError('The "table" parameter must not be empty')
+
+        self.database = database
+        self.table_name = table
+        self.table = None
+        self.client = client
+        self.pickle_protocol = pickle_protocol
+        self.connect_args = connect_args
+        self.r = RethinkDB()
+        self.conn = None
+
+    def start(self, scheduler, alias):
+        super(RethinkDBJobStore, self).start(scheduler, alias)
+
+        if self.client:
+            self.conn = maybe_ref(self.client)
+        else:
+            self.conn = self.r.connect(db=self.database, **self.connect_args)
+
+        if self.database not in self.r.db_list().run(self.conn):
+            self.r.db_create(self.database).run(self.conn)
+
+        if self.table_name not in self.r.table_list().run(self.conn):
+            self.r.table_create(self.table_name).run(self.conn)
+
+        if 'next_run_time' not in self.r.table(self.table_name).index_list().run(self.conn):
+            self.r.table(self.table_name).index_create('next_run_time').run(self.conn)
+
+        self.table = self.r.db(self.database).table(self.table_name)
+
+    def lookup_job(self, job_id):
+        results = list(self.table.get_all(job_id).pluck('job_state').run(self.conn))
+        return self._reconstitute_job(results[0]['job_state']) if results else None
+
+    def get_due_jobs(self, now):
+        return self._get_jobs(self.r.row['next_run_time'] <= datetime_to_utc_timestamp(now))
+
+    def get_next_run_time(self):
+        results = list(
+            self.table
+            .filter(self.r.row['next_run_time'] != None)  # noqa
+            .order_by(self.r.asc('next_run_time'))
+            .map(lambda x: x['next_run_time'])
+            .limit(1)
+            .run(self.conn)
+        )
+        return utc_timestamp_to_datetime(results[0]) if results else None
+
+    def get_all_jobs(self):
+        jobs = self._get_jobs()
+        self._fix_paused_jobs_sorting(jobs)
+        return jobs
+
+    def add_job(self, job):
+        job_dict = {
+            'id': job.id,
+            'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
+            'job_state': self.r.binary(pickle.dumps(job.__getstate__(), self.pickle_protocol))
+        }
+        results = self.table.insert(job_dict).run(self.conn)
+        if results['errors'] > 0:
+            raise ConflictingIdError(job.id)
+
+    def update_job(self, job):
+        changes = {
+            'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
+            'job_state': self.r.binary(pickle.dumps(job.__getstate__(), self.pickle_protocol))
+        }
+        results = self.table.get_all(job.id).update(changes).run(self.conn)
+        skipped = False in map(lambda x: results[x] == 0, results.keys())
+        if results['skipped'] > 0 or results['errors'] > 0 or not skipped:
+            raise JobLookupError(job.id)
+
+    def remove_job(self, job_id):
+        results = self.table.get_all(job_id).delete().run(self.conn)
+        if results['deleted'] + results['skipped'] != 1:
+            raise JobLookupError(job_id)
+
+    def remove_all_jobs(self):
+        self.table.delete().run(self.conn)
+
+    def shutdown(self):
+        self.conn.close()
+
+    def _reconstitute_job(self, job_state):
+        job_state = pickle.loads(job_state)
+        job = Job.__new__(Job)
+        job.__setstate__(job_state)
+        job._scheduler = self._scheduler
+        job._jobstore_alias = self._alias
+        return job
+
+    def _get_jobs(self, predicate=None):
+        jobs = []
+        failed_job_ids = []
+        query = (self.table.filter(self.r.row['next_run_time'] != None).filter(predicate)  # noqa
+                 if predicate else self.table)
+        query = query.order_by('next_run_time', 'id').pluck('id', 'job_state')
+
+        for document in query.run(self.conn):
+            try:
+                jobs.append(self._reconstitute_job(document['job_state']))
+            except Exception:
+                self._logger.exception('Unable to restore job "%s" -- removing it', document['id'])
+                failed_job_ids.append(document['id'])
+
+        # Remove all the jobs we failed to restore
+        if failed_job_ids:
+            self.r.expr(failed_job_ids).for_each(
+                lambda job_id: self.table.get_all(job_id).delete()).run(self.conn)
+
+        return jobs
+
+    def __repr__(self):
+        connection = self.conn
+        return '<%s (connection=%s)>' % (self.__class__.__name__, connection)
@@ -1,38 +1,47 @@
-
+from __future__ import absolute_import

 from apscheduler.jobstores.base import BaseJobStore, JobLookupError, ConflictingIdError
 from apscheduler.util import maybe_ref, datetime_to_utc_timestamp, utc_timestamp_to_datetime
 from apscheduler.job import Job

 try:
-    import pickle as pickle
+    import cPickle as pickle
 except ImportError:  # pragma: nocover
    import pickle

 try:
-    from sqlalchemy import create_engine, Table, Column, MetaData, Unicode, Float, LargeBinary, select
+    from sqlalchemy import (
+        create_engine, Table, Column, MetaData, Unicode, Float, LargeBinary, select, and_)
    from sqlalchemy.exc import IntegrityError
+    from sqlalchemy.sql.expression import null
 except ImportError:  # pragma: nocover
    raise ImportError('SQLAlchemyJobStore requires SQLAlchemy installed')


 class SQLAlchemyJobStore(BaseJobStore):
    """
-    Stores jobs in a database table using SQLAlchemy. The table will be created if it doesn't exist in the database.
+    Stores jobs in a database table using SQLAlchemy.
+    The table will be created if it doesn't exist in the database.

    Plugin alias: ``sqlalchemy``

-    :param str url: connection string (see `SQLAlchemy documentation
-                    <http://docs.sqlalchemy.org/en/latest/core/engines.html?highlight=create_engine#database-urls>`_
-                    on this)
-    :param engine: an SQLAlchemy Engine to use instead of creating a new one based on ``url``
+    :param str url: connection string (see
+        :ref:`SQLAlchemy documentation <sqlalchemy:database_urls>` on this)
+    :param engine: an SQLAlchemy :class:`~sqlalchemy.engine.Engine` to use instead of creating a
+        new one based on ``url``
    :param str tablename: name of the table to store jobs in
-    :param metadata: a :class:`~sqlalchemy.MetaData` instance to use instead of creating a new one
-    :param int pickle_protocol: pickle protocol level to use (for serialization), defaults to the highest available
+    :param metadata: a :class:`~sqlalchemy.schema.MetaData` instance to use instead of creating a
+        new one
+    :param int pickle_protocol: pickle protocol level to use (for serialization), defaults to the
+        highest available
+    :param str tableschema: name of the (existing) schema in the target database where the table
+        should be
+    :param dict engine_options: keyword arguments to :func:`~sqlalchemy.create_engine`
+        (ignored if ``engine`` is given)
    """

    def __init__(self, url=None, engine=None, tablename='apscheduler_jobs', metadata=None,
-                 pickle_protocol=pickle.HIGHEST_PROTOCOL):
+                 pickle_protocol=pickle.HIGHEST_PROTOCOL, tableschema=None, engine_options=None):
        super(SQLAlchemyJobStore, self).__init__()
        self.pickle_protocol = pickle_protocol
        metadata = maybe_ref(metadata) or MetaData()
@@ -40,23 +49,28 @@ class SQLAlchemyJobStore(BaseJobStore):
        if engine:
            self.engine = maybe_ref(engine)
        elif url:
-            self.engine = create_engine(url)
+            self.engine = create_engine(url, **(engine_options or {}))
        else:
            raise ValueError('Need either "engine" or "url" defined')

-        # 191 = max key length in MySQL for InnoDB/utf8mb4 tables, 25 = precision that translates to an 8-byte float
+        # 191 = max key length in MySQL for InnoDB/utf8mb4 tables,
+        # 25 = precision that translates to an 8-byte float
        self.jobs_t = Table(
            tablename, metadata,
-            Column('id', Unicode(191, _warn_on_bytestring=False), primary_key=True),
+            Column('id', Unicode(191), primary_key=True),
            Column('next_run_time', Float(25), index=True),
-            Column('job_state', LargeBinary, nullable=False)
+            Column('job_state', LargeBinary, nullable=False),
+            schema=tableschema
        )

+    def start(self, scheduler, alias):
+        super(SQLAlchemyJobStore, self).start(scheduler, alias)
        self.jobs_t.create(self.engine, True)

    def lookup_job(self, job_id):
-        selectable = select([self.jobs_t.c.job_state]).where(self.jobs_t.c.id == job_id)
-        job_state = self.engine.execute(selectable).scalar()
+        selectable = select(self.jobs_t.c.job_state).where(self.jobs_t.c.id == job_id)
+        with self.engine.begin() as connection:
+            job_state = connection.execute(selectable).scalar()
            return self._reconstitute_job(job_state) if job_state else None

    def get_due_jobs(self, now):
@@ -64,13 +78,17 @@ class SQLAlchemyJobStore(BaseJobStore):
        return self._get_jobs(self.jobs_t.c.next_run_time <= timestamp)

    def get_next_run_time(self):
-        selectable = select([self.jobs_t.c.next_run_time]).where(self.jobs_t.c.next_run_time != None).\
+        selectable = select(self.jobs_t.c.next_run_time).\
+            where(self.jobs_t.c.next_run_time != null()).\
            order_by(self.jobs_t.c.next_run_time).limit(1)
-        next_run_time = self.engine.execute(selectable).scalar()
+        with self.engine.begin() as connection:
+            next_run_time = connection.execute(selectable).scalar()
            return utc_timestamp_to_datetime(next_run_time)

    def get_all_jobs(self):
-        return self._get_jobs()
+        jobs = self._get_jobs()
+        self._fix_paused_jobs_sorting(jobs)
+        return jobs

    def add_job(self, job):
        insert = self.jobs_t.insert().values(**{
@@ -78,8 +96,9 @@ class SQLAlchemyJobStore(BaseJobStore):
            'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
            'job_state': pickle.dumps(job.__getstate__(), self.pickle_protocol)
        })
+        with self.engine.begin() as connection:
            try:
-            self.engine.execute(insert)
+                connection.execute(insert)
            except IntegrityError:
                raise ConflictingIdError(job.id)

@@ -88,19 +107,22 @@ class SQLAlchemyJobStore(BaseJobStore):
            'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
            'job_state': pickle.dumps(job.__getstate__(), self.pickle_protocol)
        }).where(self.jobs_t.c.id == job.id)
-        result = self.engine.execute(update)
+        with self.engine.begin() as connection:
+            result = connection.execute(update)
            if result.rowcount == 0:
-            raise JobLookupError(id)
+                raise JobLookupError(job.id)

    def remove_job(self, job_id):
        delete = self.jobs_t.delete().where(self.jobs_t.c.id == job_id)
-        result = self.engine.execute(delete)
+        with self.engine.begin() as connection:
+            result = connection.execute(delete)
            if result.rowcount == 0:
                raise JobLookupError(job_id)

    def remove_all_jobs(self):
        delete = self.jobs_t.delete()
-        self.engine.execute(delete)
+        with self.engine.begin() as connection:
+            connection.execute(delete)

    def shutdown(self):
        self.engine.dispose()
@@ -116,20 +138,22 @@ class SQLAlchemyJobStore(BaseJobStore):

    def _get_jobs(self, *conditions):
        jobs = []
-        selectable = select([self.jobs_t.c.id, self.jobs_t.c.job_state]).order_by(self.jobs_t.c.next_run_time)
-        selectable = selectable.where(*conditions) if conditions else selectable
+        selectable = select(self.jobs_t.c.id, self.jobs_t.c.job_state).\
+            order_by(self.jobs_t.c.next_run_time)
+        selectable = selectable.where(and_(*conditions)) if conditions else selectable
        failed_job_ids = set()
-        for row in self.engine.execute(selectable):
+        with self.engine.begin() as connection:
+            for row in connection.execute(selectable):
                try:
                    jobs.append(self._reconstitute_job(row.job_state))
-            except:
+                except BaseException:
                    self._logger.exception('Unable to restore job "%s" -- removing it', row.id)
                    failed_job_ids.add(row.id)

            # Remove all the jobs we failed to restore
            if failed_job_ids:
                delete = self.jobs_t.delete().where(self.jobs_t.c.id.in_(failed_job_ids))
-            self.engine.execute(delete)
+                connection.execute(delete)

        return jobs

@@ -0,0 +1,178 @@
+from __future__ import absolute_import
+
+from datetime import datetime
+
+from pytz import utc
+from kazoo.exceptions import NoNodeError, NodeExistsError
+
+from apscheduler.jobstores.base import BaseJobStore, JobLookupError, ConflictingIdError
+from apscheduler.util import maybe_ref, datetime_to_utc_timestamp, utc_timestamp_to_datetime
+from apscheduler.job import Job
+
+try:
+    import cPickle as pickle
+except ImportError:  # pragma: nocover
+    import pickle
+
+try:
+    from kazoo.client import KazooClient
+except ImportError:  # pragma: nocover
+    raise ImportError('ZooKeeperJobStore requires Kazoo installed')
+
+
+class ZooKeeperJobStore(BaseJobStore):
+    """
+    Stores jobs in a ZooKeeper tree. Any leftover keyword arguments are directly passed to
+    kazoo's `KazooClient
+    <http://kazoo.readthedocs.io/en/latest/api/client.html>`_.
+
+    Plugin alias: ``zookeeper``
+
+    :param str path: path to store jobs in
+    :param client: a :class:`~kazoo.client.KazooClient` instance to use instead of
+        providing connection arguments
+    :param int pickle_protocol: pickle protocol level to use (for serialization), defaults to the
+        highest available
+    """
+
+    def __init__(self, path='/apscheduler', client=None, close_connection_on_exit=False,
+                 pickle_protocol=pickle.HIGHEST_PROTOCOL, **connect_args):
+        super(ZooKeeperJobStore, self).__init__()
+        self.pickle_protocol = pickle_protocol
+        self.close_connection_on_exit = close_connection_on_exit
+
+        if not path:
+            raise ValueError('The "path" parameter must not be empty')
+
+        self.path = path
+
+        if client:
+            self.client = maybe_ref(client)
+        else:
+            self.client = KazooClient(**connect_args)
+        self._ensured_path = False
+
+    def _ensure_paths(self):
+        if not self._ensured_path:
+            self.client.ensure_path(self.path)
+        self._ensured_path = True
+
+    def start(self, scheduler, alias):
+        super(ZooKeeperJobStore, self).start(scheduler, alias)
+        if not self.client.connected:
+            self.client.start()
+
+    def lookup_job(self, job_id):
+        self._ensure_paths()
+        node_path = self.path + "/" + str(job_id)
+        try:
+            content, _ = self.client.get(node_path)
+            doc = pickle.loads(content)
+            job = self._reconstitute_job(doc['job_state'])
+            return job
+        except BaseException:
+            return None
+
+    def get_due_jobs(self, now):
+        timestamp = datetime_to_utc_timestamp(now)
+        jobs = [job_def['job'] for job_def in self._get_jobs()
+                if job_def['next_run_time'] is not None and job_def['next_run_time'] <= timestamp]
+        return jobs
+
+    def get_next_run_time(self):
+        next_runs = [job_def['next_run_time'] for job_def in self._get_jobs()
+                     if job_def['next_run_time'] is not None]
+        return utc_timestamp_to_datetime(min(next_runs)) if len(next_runs) > 0 else None
+
+    def get_all_jobs(self):
+        jobs = [job_def['job'] for job_def in self._get_jobs()]
+        self._fix_paused_jobs_sorting(jobs)
+        return jobs
+
+    def add_job(self, job):
+        self._ensure_paths()
+        node_path = self.path + "/" + str(job.id)
+        value = {
+            'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
+            'job_state': job.__getstate__()
+        }
+        data = pickle.dumps(value, self.pickle_protocol)
+        try:
+            self.client.create(node_path, value=data)
+        except NodeExistsError:
+            raise ConflictingIdError(job.id)
+
+    def update_job(self, job):
+        self._ensure_paths()
+        node_path = self.path + "/" + str(job.id)
+        changes = {
+            'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
+            'job_state': job.__getstate__()
+        }
+        data = pickle.dumps(changes, self.pickle_protocol)
+        try:
+            self.client.set(node_path, value=data)
+        except NoNodeError:
+            raise JobLookupError(job.id)
+
+    def remove_job(self, job_id):
+        self._ensure_paths()
+        node_path = self.path + "/" + str(job_id)
+        try:
+            self.client.delete(node_path)
+        except NoNodeError:
+            raise JobLookupError(job_id)
+
+    def remove_all_jobs(self):
+        try:
+            self.client.delete(self.path, recursive=True)
+        except NoNodeError:
+            pass
+        self._ensured_path = False
+
+    def shutdown(self):
+        if self.close_connection_on_exit:
+            self.client.stop()
+            self.client.close()
+
+    def _reconstitute_job(self, job_state):
+        job_state = job_state
+        job = Job.__new__(Job)
+        job.__setstate__(job_state)
+        job._scheduler = self._scheduler
+        job._jobstore_alias = self._alias
+        return job
+
+    def _get_jobs(self):
+        self._ensure_paths()
+        jobs = []
+        failed_job_ids = []
+        all_ids = self.client.get_children(self.path)
+        for node_name in all_ids:
+            try:
+                node_path = self.path + "/" + node_name
+                content, _ = self.client.get(node_path)
+                doc = pickle.loads(content)
+                job_def = {
+                    'job_id': node_name,
+                    'next_run_time': doc['next_run_time'] if doc['next_run_time'] else None,
+                    'job_state': doc['job_state'],
+                    'job': self._reconstitute_job(doc['job_state']),
+                    'creation_time': _.ctime
+                }
+                jobs.append(job_def)
+            except BaseException:
+                self._logger.exception('Unable to restore job "%s" -- removing it' % node_name)
+                failed_job_ids.append(node_name)
+
+        # Remove all the jobs we failed to restore
+        if failed_job_ids:
+            for failed_id in failed_job_ids:
+                self.remove_job(failed_id)
+        paused_sort_key = datetime(9999, 12, 31, tzinfo=utc)
+        return sorted(jobs, key=lambda job_def: (job_def['job'].next_run_time or paused_sort_key,
+                                                 job_def['creation_time']))
+
+    def __repr__(self):
+        self._logger.exception('<%s (client=%s)>' % (self.__class__.__name__, self.client))
+        return '<%s (client=%s)>' % (self.__class__.__name__, self.client)
@@ -1,22 +1,16 @@
-
-from functools import wraps
+from __future__ import absolute_import
+import asyncio
+from functools import wraps, partial

 from apscheduler.schedulers.base import BaseScheduler
 from apscheduler.util import maybe_ref

-try:
-    import asyncio
-except ImportError:  # pragma: nocover
-    try:
-        import trollius as asyncio
-    except ImportError:
-        raise ImportError('AsyncIOScheduler requires either Python 3.4 or the asyncio package installed')
-

 def run_in_event_loop(func):
    @wraps(func)
    def wrapper(self, *args, **kwargs):
-        self._eventloop.call_soon_threadsafe(func, self, *args, **kwargs)
+        wrapped = partial(func, self, *args, **kwargs)
+        self._eventloop.call_soon_threadsafe(wrapped)
    return wrapper


@@ -24,6 +18,8 @@ class AsyncIOScheduler(BaseScheduler):
    """
    A scheduler that runs on an asyncio (:pep:`3156`) event loop.

+    The default executor can run jobs based on native coroutines (``async def``).
+
    Extra options:

    ============== =============================================================
@@ -34,9 +30,11 @@ class AsyncIOScheduler(BaseScheduler):
    _eventloop = None
    _timeout = None

-    def start(self):
-        super(AsyncIOScheduler, self).start()
-        self.wakeup()
+    def start(self, paused=False):
+        if not self._eventloop:
+            self._eventloop = asyncio.get_event_loop()
+
+        super(AsyncIOScheduler, self).start(paused)

    @run_in_event_loop
    def shutdown(self, wait=True):
@@ -44,7 +42,7 @@ class AsyncIOScheduler(BaseScheduler):
        self._stop_timer()

    def _configure(self, config):
-        self._eventloop = maybe_ref(config.pop('event_loop', None)) or asyncio.get_event_loop()
+        self._eventloop = maybe_ref(config.pop('event_loop', None))
        super(AsyncIOScheduler, self)._configure(config)

    def _start_timer(self, wait_seconds):
@@ -1,3 +1,4 @@
+from __future__ import absolute_import

 from threading import Thread, Event

@@ -13,11 +14,12 @@ class BackgroundScheduler(BlockingScheduler):

    Extra options:

-    ========== ============================================================================================
-    ``daemon`` Set the ``daemon`` option in the background thread (defaults to ``True``,
-               see `the documentation <https://docs.python.org/3.4/library/threading.html#thread-objects>`_
+    ========== =============================================================================
+    ``daemon`` Set the ``daemon`` option in the background thread (defaults to ``True``, see
+               `the documentation
+               <https://docs.python.org/3.4/library/threading.html#thread-objects>`_
               for further details)
-    ========== ============================================================================================
+    ========== =============================================================================
    """

    _thread = None
@@ -26,14 +28,16 @@ class BackgroundScheduler(BlockingScheduler):
        self._daemon = asbool(config.pop('daemon', True))
        super(BackgroundScheduler, self)._configure(config)

-    def start(self):
-        BaseScheduler.start(self)
+    def start(self, *args, **kwargs):
+        if self._event is None or self._event.is_set():
            self._event = Event()
+
+        BaseScheduler.start(self, *args, **kwargs)
        self._thread = Thread(target=self._main_loop, name='APScheduler')
        self._thread.daemon = self._daemon
        self._thread.start()

-    def shutdown(self, wait=True):
-        super(BackgroundScheduler, self).shutdown(wait)
+    def shutdown(self, *args, **kwargs):
+        super(BackgroundScheduler, self).shutdown(*args, **kwargs)
        self._thread.join()
        del self._thread
@@ -1,21 +1,23 @@
+from __future__ import absolute_import

 from threading import Event

-from apscheduler.schedulers.base import BaseScheduler
+from apscheduler.schedulers.base import BaseScheduler, STATE_STOPPED
+from apscheduler.util import TIMEOUT_MAX


 class BlockingScheduler(BaseScheduler):
    """
-    A scheduler that runs in the foreground (:meth:`~apscheduler.schedulers.base.BaseScheduler.start` will block).
+    A scheduler that runs in the foreground
+    (:meth:`~apscheduler.schedulers.base.BaseScheduler.start` will block).
    """
-
-    MAX_WAIT_TIME = 4294967  # Maximum value accepted by Event.wait() on Windows
-
    _event = None

-    def start(self):
-        super(BlockingScheduler, self).start()
+    def start(self, *args, **kwargs):
+        if self._event is None or self._event.is_set():
            self._event = Event()
+
+        super(BlockingScheduler, self).start(*args, **kwargs)
        self._main_loop()

    def shutdown(self, wait=True):
@@ -23,10 +25,11 @@ class BlockingScheduler(BaseScheduler):
        self._event.set()

    def _main_loop(self):
-        while self.running:
-            wait_seconds = self._process_jobs()
-            self._event.wait(wait_seconds if wait_seconds is not None else self.MAX_WAIT_TIME)
+        wait_seconds = TIMEOUT_MAX
+        while self.state != STATE_STOPPED:
+            self._event.wait(wait_seconds)
            self._event.clear()
+            wait_seconds = self._process_jobs()

    def wakeup(self):
        self._event.set()
@@ -1,4 +1,4 @@
-
+from __future__ import absolute_import

 from apscheduler.schedulers.blocking import BlockingScheduler
 from apscheduler.schedulers.base import BaseScheduler
@@ -16,14 +16,14 @@ class GeventScheduler(BlockingScheduler):

    _greenlet = None

-    def start(self):
-        BaseScheduler.start(self)
+    def start(self, *args, **kwargs):
        self._event = Event()
+        BaseScheduler.start(self, *args, **kwargs)
        self._greenlet = gevent.spawn(self._main_loop)
        return self._greenlet

-    def shutdown(self, wait=True):
-        super(GeventScheduler, self).shutdown(wait)
+    def shutdown(self, *args, **kwargs):
+        super(GeventScheduler, self).shutdown(*args, **kwargs)
        self._greenlet.join()
        del self._greenlet

@@ -1,17 +1,24 @@
-
+from __future__ import absolute_import

 from apscheduler.schedulers.base import BaseScheduler

 try:
    from PyQt5.QtCore import QObject, QTimer
-except ImportError:  # pragma: nocover
+except (ImportError, RuntimeError):  # pragma: nocover
    try:
        from PyQt4.QtCore import QObject, QTimer
    except ImportError:
        try:
-            from PySide.QtCore import QObject, QTimer  # flake8: noqa
+            from PySide6.QtCore import QObject, QTimer  # noqa
        except ImportError:
-            raise ImportError('QtScheduler requires either PyQt5, PyQt4 or PySide installed')
+            try:
+                from PySide2.QtCore import QObject, QTimer  # noqa
+            except ImportError:
+                try:
+                    from PySide.QtCore import QObject, QTimer  # noqa
+                except ImportError:
+                    raise ImportError('QtScheduler requires either PyQt5, PyQt4, PySide6, PySide2 '
+                                      'or PySide installed')


 class QtScheduler(BaseScheduler):
@@ -19,18 +26,15 @@ class QtScheduler(BaseScheduler):

    _timer = None

-    def start(self):
-        super(QtScheduler, self).start()
-        self.wakeup()
-
-    def shutdown(self, wait=True):
-        super(QtScheduler, self).shutdown(wait)
+    def shutdown(self, *args, **kwargs):
+        super(QtScheduler, self).shutdown(*args, **kwargs)
        self._stop_timer()

    def _start_timer(self, wait_seconds):
        self._stop_timer()
        if wait_seconds is not None:
-            self._timer = QTimer.singleShot(wait_seconds * 1000, self._process_jobs)
+            wait_time = min(int(wait_seconds * 1000), 2147483647)
+            self._timer = QTimer.singleShot(wait_time, self._process_jobs)

    def _stop_timer(self):
        if self._timer:
@@ -1,3 +1,4 @@
+from __future__ import absolute_import

 from datetime import timedelta
 from functools import wraps
@@ -22,6 +23,8 @@ class TornadoScheduler(BaseScheduler):
    """
    A scheduler that runs on a Tornado IOLoop.

+    The default executor can run jobs based on native coroutines (``async def``).
+
    =========== ===============================================================
    ``io_loop`` Tornado IOLoop instance to use (defaults to the global IO loop)
    =========== ===============================================================
@@ -30,10 +33,6 @@ class TornadoScheduler(BaseScheduler):
    _ioloop = None
    _timeout = None

-    def start(self):
-        super(TornadoScheduler, self).start()
-        self.wakeup()
-
    @run_in_ioloop
    def shutdown(self, wait=True):
        super(TornadoScheduler, self).shutdown(wait)
@@ -53,6 +52,10 @@ class TornadoScheduler(BaseScheduler):
            self._ioloop.remove_timeout(self._timeout)
            del self._timeout

+    def _create_default_executor(self):
+        from apscheduler.executors.tornado import TornadoExecutor
+        return TornadoExecutor()
+
    @run_in_ioloop
    def wakeup(self):
        self._stop_timer()
@@ -1,3 +1,4 @@
+from __future__ import absolute_import

 from functools import wraps

@@ -35,10 +36,6 @@ class TwistedScheduler(BaseScheduler):
        self._reactor = maybe_ref(config.pop('reactor', default_reactor))
        super(TwistedScheduler, self)._configure(config)

-    def start(self):
-        super(TwistedScheduler, self).start()
-        self.wakeup()
-
    @run_in_reactor
    def shutdown(self, wait=True):
        super(TwistedScheduler, self).shutdown(wait)
@@ -1,4 +1,6 @@
 from abc import ABCMeta, abstractmethod
+from datetime import timedelta
+import random

 import six

@@ -6,11 +8,30 @@ import six
 class BaseTrigger(six.with_metaclass(ABCMeta)):
    """Abstract base class that defines the interface that every trigger must implement."""

+    __slots__ = ()
+
    @abstractmethod
    def get_next_fire_time(self, previous_fire_time, now):
        """
-        Returns the next datetime to fire on, If no such datetime can be calculated, returns ``None``.
+        Returns the next datetime to fire on, If no such datetime can be calculated, returns
+        ``None``.

        :param datetime.datetime previous_fire_time: the previous time the trigger was fired
        :param datetime.datetime now: current datetime
        """
+
+    def _apply_jitter(self, next_fire_time, jitter, now):
+        """
+        Randomize ``next_fire_time`` by adding a random value (the jitter).
+
+        :param datetime.datetime|None next_fire_time: next fire time without jitter applied. If
+            ``None``, returns ``None``.
+        :param int|None jitter: maximum number of seconds to add to ``next_fire_time``
+            (if ``None`` or ``0``, returns ``next_fire_time``)
+        :param datetime.datetime now: current datetime
+        :return datetime.datetime|None: next fire time with a jitter.
+        """
+        if next_fire_time is None or not jitter:
+            return next_fire_time
+
+        return next_fire_time + timedelta(seconds=random.uniform(0, jitter))
@@ -0,0 +1,95 @@
+from apscheduler.triggers.base import BaseTrigger
+from apscheduler.util import obj_to_ref, ref_to_obj
+
+
+class BaseCombiningTrigger(BaseTrigger):
+    __slots__ = ('triggers', 'jitter')
+
+    def __init__(self, triggers, jitter=None):
+        self.triggers = triggers
+        self.jitter = jitter
+
+    def __getstate__(self):
+        return {
+            'version': 1,
+            'triggers': [(obj_to_ref(trigger.__class__), trigger.__getstate__())
+                         for trigger in self.triggers],
+            'jitter': self.jitter
+        }
+
+    def __setstate__(self, state):
+        if state.get('version', 1) > 1:
+            raise ValueError(
+                'Got serialized data for version %s of %s, but only versions up to 1 can be '
+                'handled' % (state['version'], self.__class__.__name__))
+
+        self.jitter = state['jitter']
+        self.triggers = []
+        for clsref, state in state['triggers']:
+            cls = ref_to_obj(clsref)
+            trigger = cls.__new__(cls)
+            trigger.__setstate__(state)
+            self.triggers.append(trigger)
+
+    def __repr__(self):
+        return '<{}({}{})>'.format(self.__class__.__name__, self.triggers,
+                                   ', jitter={}'.format(self.jitter) if self.jitter else '')
+
+
+class AndTrigger(BaseCombiningTrigger):
+    """
+    Always returns the earliest next fire time that all the given triggers can agree on.
+    The trigger is considered to be finished when any of the given triggers has finished its
+    schedule.
+
+    Trigger alias: ``and``
+
+    :param list triggers: triggers to combine
+    :param int|None jitter: delay the job execution by ``jitter`` seconds at most
+    """
+
+    __slots__ = ()
+
+    def get_next_fire_time(self, previous_fire_time, now):
+        while True:
+            fire_times = [trigger.get_next_fire_time(previous_fire_time, now)
+                          for trigger in self.triggers]
+            if None in fire_times:
+                return None
+            elif min(fire_times) == max(fire_times):
+                return self._apply_jitter(fire_times[0], self.jitter, now)
+            else:
+                now = max(fire_times)
+
+    def __str__(self):
+        return 'and[{}]'.format(', '.join(str(trigger) for trigger in self.triggers))
+
+
+class OrTrigger(BaseCombiningTrigger):
+    """
+    Always returns the earliest next fire time produced by any of the given triggers.
+    The trigger is considered finished when all the given triggers have finished their schedules.
+
+    Trigger alias: ``or``
+
+    :param list triggers: triggers to combine
+    :param int|None jitter: delay the job execution by ``jitter`` seconds at most
+
+    .. note:: Triggers that depends on the previous fire time, such as the interval trigger, may
+        seem to behave strangely since they are always passed the previous fire time produced by
+        any of the given triggers.
+    """
+
+    __slots__ = ()
+
+    def get_next_fire_time(self, previous_fire_time, now):
+        fire_times = [trigger.get_next_fire_time(previous_fire_time, now)
+                      for trigger in self.triggers]
+        fire_times = [fire_time for fire_time in fire_times if fire_time is not None]
+        if fire_times:
+            return self._apply_jitter(min(fire_times), self.jitter, now)
+        else:
+            return None
+
+    def __str__(self):
+        return 'or[{}]'.format(', '.join(str(trigger) for trigger in self.triggers))
@@ -4,17 +4,20 @@ from tzlocal import get_localzone
 import six

 from apscheduler.triggers.base import BaseTrigger
-from apscheduler.triggers.cron.fields import BaseField, WeekField, DayOfMonthField, DayOfWeekField, DEFAULT_VALUES
-from apscheduler.util import datetime_ceil, convert_to_datetime, datetime_repr, astimezone
+from apscheduler.triggers.cron.fields import (
+    BaseField, MonthField, WeekField, DayOfMonthField, DayOfWeekField, DEFAULT_VALUES)
+from apscheduler.util import (
+    datetime_ceil, convert_to_datetime, datetime_repr, astimezone, localize, normalize)


 class CronTrigger(BaseTrigger):
    """
-    Triggers when current time matches all specified time constraints, similarly to how the UNIX cron scheduler works.
+    Triggers when current time matches all specified time constraints,
+    similarly to how the UNIX cron scheduler works.

    :param int|str year: 4-digit year
    :param int|str month: month (1-12)
-    :param int|str day: day of the (1-31)
+    :param int|str day: day of month (1-31)
    :param int|str week: ISO week (1-53)
    :param int|str day_of_week: number or name of weekday (0-6 or mon,tue,wed,thu,fri,sat,sun)
    :param int|str hour: hour (0-23)
@@ -22,8 +25,9 @@ class CronTrigger(BaseTrigger):
    :param int|str second: second (0-59)
    :param datetime|str start_date: earliest possible date/time to trigger on (inclusive)
    :param datetime|str end_date: latest possible date/time to trigger on (inclusive)
-    :param datetime.tzinfo|str timezone: time zone to use for the date/time calculations
-                                         (defaults to scheduler timezone)
+    :param datetime.tzinfo|str timezone: time zone to use for the date/time calculations (defaults
+        to scheduler timezone)
+    :param int|None jitter: delay the job execution by ``jitter`` seconds at most

    .. note:: The first weekday is always **monday**.
    """
@@ -31,7 +35,7 @@ class CronTrigger(BaseTrigger):
    FIELD_NAMES = ('year', 'month', 'day', 'week', 'day_of_week', 'hour', 'minute', 'second')
    FIELDS_MAP = {
        'year': BaseField,
-        'month': BaseField,
+        'month': MonthField,
        'week': WeekField,
        'day': DayOfMonthField,
        'day_of_week': DayOfWeekField,
@@ -40,15 +44,16 @@ class CronTrigger(BaseTrigger):
        'second': BaseField
    }

-    __slots__ = 'timezone', 'start_date', 'end_date', 'fields'
+    __slots__ = 'timezone', 'start_date', 'end_date', 'fields', 'jitter'

-    def __init__(self, year=None, month=None, day=None, week=None, day_of_week=None, hour=None, minute=None,
-                 second=None, start_date=None, end_date=None, timezone=None):
+    def __init__(self, year=None, month=None, day=None, week=None, day_of_week=None, hour=None,
+                 minute=None, second=None, start_date=None, end_date=None, timezone=None,
+                 jitter=None):
        if timezone:
            self.timezone = astimezone(timezone)
-        elif start_date and start_date.tzinfo:
+        elif isinstance(start_date, datetime) and start_date.tzinfo:
            self.timezone = start_date.tzinfo
-        elif end_date and end_date.tzinfo:
+        elif isinstance(end_date, datetime) and end_date.tzinfo:
            self.timezone = end_date.tzinfo
        else:
            self.timezone = get_localzone()
@@ -56,6 +61,8 @@ class CronTrigger(BaseTrigger):
        self.start_date = convert_to_datetime(start_date, self.timezone, 'start_date')
        self.end_date = convert_to_datetime(end_date, self.timezone, 'end_date')

+        self.jitter = jitter
+
        values = dict((key, value) for (key, value) in six.iteritems(locals())
                      if key in self.FIELD_NAMES and value is not None)
        self.fields = []
@@ -76,13 +83,35 @@ class CronTrigger(BaseTrigger):
            field = field_class(field_name, exprs, is_default)
            self.fields.append(field)

+    @classmethod
+    def from_crontab(cls, expr, timezone=None):
+        """
+        Create a :class:`~CronTrigger` from a standard crontab expression.
+
+        See https://en.wikipedia.org/wiki/Cron for more information on the format accepted here.
+
+        :param expr: minute, hour, day of month, month, day of week
+        :param datetime.tzinfo|str timezone: time zone to use for the date/time calculations (
+            defaults to scheduler timezone)
+        :return: a :class:`~CronTrigger` instance
+
+        """
+        values = expr.split()
+        if len(values) != 5:
+            raise ValueError('Wrong number of fields; got {}, expected 5'.format(len(values)))
+
+        return cls(minute=values[0], hour=values[1], day=values[2], month=values[3],
+                   day_of_week=values[4], timezone=timezone)
+
    def _increment_field_value(self, dateval, fieldnum):
        """
-        Increments the designated field and resets all less significant fields to their minimum values.
+        Increments the designated field and resets all less significant fields to their minimum
+        values.

        :type dateval: datetime
        :type fieldnum: int
-        :return: a tuple containing the new date, and the number of the field that was actually incremented
+        :return: a tuple containing the new date, and the number of the field that was actually
+            incremented
        :rtype: tuple
        """

@@ -115,7 +144,7 @@ class CronTrigger(BaseTrigger):
                    i += 1

        difference = datetime(**values) - dateval.replace(tzinfo=None)
-        return self.timezone.normalize(dateval + difference), fieldnum
+        return normalize(dateval + difference), fieldnum

    def _set_field_value(self, dateval, fieldnum, new_value):
        values = {}
@@ -128,12 +157,13 @@ class CronTrigger(BaseTrigger):
                else:
                    values[field.name] = new_value

-        difference = datetime(**values) - dateval.replace(tzinfo=None)
-        return self.timezone.normalize(dateval + difference)
+        return localize(datetime(**values), self.timezone)

    def get_next_fire_time(self, previous_fire_time, now):
        if previous_fire_time:
-            start_date = max(now, previous_fire_time + timedelta(microseconds=1))
+            start_date = min(now, previous_fire_time + timedelta(microseconds=1))
+            if start_date == previous_fire_time:
+                start_date += timedelta(microseconds=1)
        else:
            start_date = max(now, self.start_date) if self.start_date else now

@@ -163,7 +193,34 @@ class CronTrigger(BaseTrigger):
                return None

        if fieldnum >= 0:
-            return next_date
+            next_date = self._apply_jitter(next_date, self.jitter, now)
+            return min(next_date, self.end_date) if self.end_date else next_date
+
+    def __getstate__(self):
+        return {
+            'version': 2,
+            'timezone': self.timezone,
+            'start_date': self.start_date,
+            'end_date': self.end_date,
+            'fields': self.fields,
+            'jitter': self.jitter,
+        }
+
+    def __setstate__(self, state):
+        # This is for compatibility with APScheduler 3.0.x
+        if isinstance(state, tuple):
+            state = state[1]
+
+        if state.get('version', 1) > 2:
+            raise ValueError(
+                'Got serialized data for version %s of %s, but only versions up to 2 can be '
+                'handled' % (state['version'], self.__class__.__name__))
+
+        self.timezone = state['timezone']
+        self.start_date = state['start_date']
+        self.end_date = state['end_date']
+        self.fields = state['fields']
+        self.jitter = state.get('jitter')

    def __str__(self):
        options = ["%s='%s'" % (f.name, f) for f in self.fields if not f.is_default]
@@ -172,5 +229,11 @@ class CronTrigger(BaseTrigger):
    def __repr__(self):
        options = ["%s='%s'" % (f.name, f) for f in self.fields if not f.is_default]
        if self.start_date:
-            options.append("start_date='%s'" % datetime_repr(self.start_date))
-        return '<%s (%s)>' % (self.__class__.__name__, ', '.join(options))
+            options.append("start_date=%r" % datetime_repr(self.start_date))
+        if self.end_date:
+            options.append("end_date=%r" % datetime_repr(self.end_date))
+        if self.jitter:
+            options.append('jitter=%s' % self.jitter)
+
+        return "<%s (%s, timezone='%s')>" % (
+            self.__class__.__name__, ', '.join(options), self.timezone)
@@ -1,17 +1,16 @@
-"""
-This module contains the expressions applicable for CronTrigger's fields.
-"""
+"""This module contains the expressions applicable for CronTrigger's fields."""

 from calendar import monthrange
 import re

 from apscheduler.util import asint

-__all__ = ('AllExpression', 'RangeExpression', 'WeekdayRangeExpression', 'WeekdayPositionExpression',
-           'LastDayOfMonthExpression')
+__all__ = ('AllExpression', 'RangeExpression', 'WeekdayRangeExpression',
+           'WeekdayPositionExpression', 'LastDayOfMonthExpression')


 WEEKDAYS = ['mon', 'tue', 'wed', 'thu', 'fri', 'sat', 'sun']
+MONTHS = ['jan', 'feb', 'mar', 'apr', 'may', 'jun', 'jul', 'aug', 'sep', 'oct', 'nov', 'dec']


 class AllExpression(object):
@@ -22,6 +21,14 @@ class AllExpression(object):
        if self.step == 0:
            raise ValueError('Increment must be higher than 0')

+    def validate_range(self, field_name):
+        from apscheduler.triggers.cron.fields import MIN_VALUES, MAX_VALUES
+
+        value_range = MAX_VALUES[field_name] - MIN_VALUES[field_name]
+        if self.step and self.step > value_range:
+            raise ValueError('the step value ({}) is higher than the total range of the '
+                             'expression ({})'.format(self.step, value_range))
+
    def get_next_value(self, date, field):
        start = field.get_value(date)
        minval = field.get_min(date)
@@ -37,6 +44,9 @@ class AllExpression(object):
        if next <= maxval:
            return next

+    def __eq__(self, other):
+        return isinstance(other, self.__class__) and self.step == other.step
+
    def __str__(self):
        if self.step:
            return '*/%d' % self.step
@@ -51,7 +61,7 @@ class RangeExpression(AllExpression):
        r'(?P<first>\d+)(?:-(?P<last>\d+))?(?:/(?P<step>\d+))?$')

    def __init__(self, first, last=None, step=None):
-        AllExpression.__init__(self, step)
+        super(RangeExpression, self).__init__(step)
        first = asint(first)
        last = asint(last)
        if last is None and step is None:
@@ -61,25 +71,41 @@ class RangeExpression(AllExpression):
        self.first = first
        self.last = last

+    def validate_range(self, field_name):
+        from apscheduler.triggers.cron.fields import MIN_VALUES, MAX_VALUES
+
+        super(RangeExpression, self).validate_range(field_name)
+        if self.first < MIN_VALUES[field_name]:
+            raise ValueError('the first value ({}) is lower than the minimum value ({})'
+                             .format(self.first, MIN_VALUES[field_name]))
+        if self.last is not None and self.last > MAX_VALUES[field_name]:
+            raise ValueError('the last value ({}) is higher than the maximum value ({})'
+                             .format(self.last, MAX_VALUES[field_name]))
+        value_range = (self.last or MAX_VALUES[field_name]) - self.first
+        if self.step and self.step > value_range:
+            raise ValueError('the step value ({}) is higher than the total range of the '
+                             'expression ({})'.format(self.step, value_range))
+
    def get_next_value(self, date, field):
-        start = field.get_value(date)
+        startval = field.get_value(date)
        minval = field.get_min(date)
        maxval = field.get_max(date)

        # Apply range limits
        minval = max(minval, self.first)
-        if self.last is not None:
-            maxval = min(maxval, self.last)
-        start = max(start, minval)
+        maxval = min(maxval, self.last) if self.last is not None else maxval
+        nextval = max(minval, startval)

-        if not self.step:
-            next = start
-        else:
-            distance_to_next = (self.step - (start - minval)) % self.step
-            next = start + distance_to_next
+        # Apply the step if defined
+        if self.step:
+            distance_to_next = (self.step - (nextval - minval)) % self.step
+            nextval += distance_to_next

-        if next <= maxval:
-            return next
+        return nextval if nextval <= maxval else None
+
+    def __eq__(self, other):
+        return (isinstance(other, self.__class__) and self.first == other.first and
+                self.last == other.last)

    def __str__(self):
        if self.last != self.first and self.last is not None:
@@ -100,6 +126,37 @@ class RangeExpression(AllExpression):
        return "%s(%s)" % (self.__class__.__name__, ', '.join(args))


+class MonthRangeExpression(RangeExpression):
+    value_re = re.compile(r'(?P<first>[a-z]+)(?:-(?P<last>[a-z]+))?', re.IGNORECASE)
+
+    def __init__(self, first, last=None):
+        try:
+            first_num = MONTHS.index(first.lower()) + 1
+        except ValueError:
+            raise ValueError('Invalid month name "%s"' % first)
+
+        if last:
+            try:
+                last_num = MONTHS.index(last.lower()) + 1
+            except ValueError:
+                raise ValueError('Invalid month name "%s"' % last)
+        else:
+            last_num = None
+
+        super(MonthRangeExpression, self).__init__(first_num, last_num)
+
+    def __str__(self):
+        if self.last != self.first and self.last is not None:
+            return '%s-%s' % (MONTHS[self.first - 1], MONTHS[self.last - 1])
+        return MONTHS[self.first - 1]
+
+    def __repr__(self):
+        args = ["'%s'" % MONTHS[self.first]]
+        if self.last != self.first and self.last is not None:
+            args.append("'%s'" % MONTHS[self.last - 1])
+        return "%s(%s)" % (self.__class__.__name__, ', '.join(args))
+
+
 class WeekdayRangeExpression(RangeExpression):
    value_re = re.compile(r'(?P<first>[a-z]+)(?:-(?P<last>[a-z]+))?', re.IGNORECASE)

@@ -117,7 +174,7 @@ class WeekdayRangeExpression(RangeExpression):
        else:
            last_num = None

-        RangeExpression.__init__(self, first_num, last_num)
+        super(WeekdayRangeExpression, self).__init__(first_num, last_num)

    def __str__(self):
        if self.last != self.first and self.last is not None:
@@ -133,9 +190,11 @@ class WeekdayRangeExpression(RangeExpression):

 class WeekdayPositionExpression(AllExpression):
    options = ['1st', '2nd', '3rd', '4th', '5th', 'last']
-    value_re = re.compile(r'(?P<option_name>%s) +(?P<weekday_name>(?:\d+|\w+))' % '|'.join(options), re.IGNORECASE)
+    value_re = re.compile(r'(?P<option_name>%s) +(?P<weekday_name>(?:\d+|\w+))' %
+                          '|'.join(options), re.IGNORECASE)

    def __init__(self, option_name, weekday_name):
+        super(WeekdayPositionExpression, self).__init__(None)
        try:
            self.option_num = self.options.index(option_name.lower())
        except ValueError:
@@ -147,8 +206,7 @@ class WeekdayPositionExpression(AllExpression):
            raise ValueError('Invalid weekday name "%s"' % weekday_name)

    def get_next_value(self, date, field):
-        # Figure out the weekday of the month's first day and the number
-        # of days in that month
+        # Figure out the weekday of the month's first day and the number of days in that month
        first_day_wday, last_day = monthrange(date.year, date.month)

        # Calculate which day of the month is the first of the target weekdays
@@ -160,23 +218,28 @@ class WeekdayPositionExpression(AllExpression):
        if self.option_num < 5:
            target_day = first_hit_day + self.option_num * 7
        else:
-            target_day = first_hit_day + ((last_day - first_hit_day) / 7) * 7
+            target_day = first_hit_day + ((last_day - first_hit_day) // 7) * 7

        if target_day <= last_day and target_day >= date.day:
            return target_day

+    def __eq__(self, other):
+        return (super(WeekdayPositionExpression, self).__eq__(other) and
+                self.option_num == other.option_num and self.weekday == other.weekday)
+
    def __str__(self):
        return '%s %s' % (self.options[self.option_num], WEEKDAYS[self.weekday])

    def __repr__(self):
-        return "%s('%s', '%s')" % (self.__class__.__name__, self.options[self.option_num], WEEKDAYS[self.weekday])
+        return "%s('%s', '%s')" % (self.__class__.__name__, self.options[self.option_num],
+                                   WEEKDAYS[self.weekday])


 class LastDayOfMonthExpression(AllExpression):
    value_re = re.compile(r'last', re.IGNORECASE)

    def __init__(self):
-        pass
+        super(LastDayOfMonthExpression, self).__init__(None)

    def get_next_value(self, date, field):
        return monthrange(date.year, date.month)[1]
@@ -1,22 +1,26 @@
-"""
-Fields represent CronTrigger options which map to :class:`~datetime.datetime`
-fields.
-"""
+"""Fields represent CronTrigger options which map to :class:`~datetime.datetime` fields."""

 from calendar import monthrange
+import re
+
+import six

 from apscheduler.triggers.cron.expressions import (
-    AllExpression, RangeExpression, WeekdayPositionExpression, LastDayOfMonthExpression, WeekdayRangeExpression)
+    AllExpression, RangeExpression, WeekdayPositionExpression, LastDayOfMonthExpression,
+    WeekdayRangeExpression, MonthRangeExpression)


-__all__ = ('MIN_VALUES', 'MAX_VALUES', 'DEFAULT_VALUES', 'BaseField', 'WeekField', 'DayOfMonthField', 'DayOfWeekField')
+__all__ = ('MIN_VALUES', 'MAX_VALUES', 'DEFAULT_VALUES', 'BaseField', 'WeekField',
+           'DayOfMonthField', 'DayOfWeekField')


-MIN_VALUES = {'year': 1970, 'month': 1, 'day': 1, 'week': 1, 'day_of_week': 0, 'hour': 0, 'minute': 0, 'second': 0}
-MAX_VALUES = {'year': 2 ** 63, 'month': 12, 'day:': 31, 'week': 53, 'day_of_week': 6, 'hour': 23, 'minute': 59,
-              'second': 59}
-DEFAULT_VALUES = {'year': '*', 'month': 1, 'day': 1, 'week': '*', 'day_of_week': '*', 'hour': 0, 'minute': 0,
-                  'second': 0}
+MIN_VALUES = {'year': 1970, 'month': 1, 'day': 1, 'week': 1, 'day_of_week': 0, 'hour': 0,
+              'minute': 0, 'second': 0}
+MAX_VALUES = {'year': 9999, 'month': 12, 'day': 31, 'week': 53, 'day_of_week': 6, 'hour': 23,
+              'minute': 59, 'second': 59}
+DEFAULT_VALUES = {'year': '*', 'month': 1, 'day': 1, 'week': '*', 'day_of_week': '*', 'hour': 0,
+                  'minute': 0, 'second': 0}
+SEPARATOR = re.compile(' *, *')


 class BaseField(object):
@@ -50,23 +54,29 @@ class BaseField(object):
        self.expressions = []

        # Split a comma-separated expression list, if any
-        exprs = str(exprs).strip()
-        if ',' in exprs:
-            for expr in exprs.split(','):
+        for expr in SEPARATOR.split(str(exprs).strip()):
            self.compile_expression(expr)
-        else:
-            self.compile_expression(exprs)

    def compile_expression(self, expr):
        for compiler in self.COMPILERS:
            match = compiler.value_re.match(expr)
            if match:
                compiled_expr = compiler(**match.groupdict())
+
+                try:
+                    compiled_expr.validate_range(self.name)
+                except ValueError as e:
+                    exc = ValueError('Error validating expression {!r}: {}'.format(expr, e))
+                    six.raise_from(exc, None)
+
                self.expressions.append(compiled_expr)
                return

        raise ValueError('Unrecognized expression "%s" for field "%s"' % (expr, self.name))

+    def __eq__(self, other):
+        return isinstance(self, self.__class__) and self.expressions == other.expressions
+
    def __str__(self):
        expr_strings = (str(e) for e in self.expressions)
        return ','.join(expr_strings)
@@ -95,3 +105,7 @@ class DayOfWeekField(BaseField):

    def get_value(self, dateval):
        return dateval.weekday()
+
+
+class MonthField(BaseField):
+    COMPILERS = BaseField.COMPILERS + [MonthRangeExpression]
@@ -14,15 +14,36 @@ class DateTrigger(BaseTrigger):
    :param datetime.tzinfo|str timezone: time zone for ``run_date`` if it doesn't have one already
    """

-    __slots__ = 'timezone', 'run_date'
+    __slots__ = 'run_date'

    def __init__(self, run_date=None, timezone=None):
        timezone = astimezone(timezone) or get_localzone()
-        self.run_date = convert_to_datetime(run_date or datetime.now(), timezone, 'run_date')
+        if run_date is not None:
+            self.run_date = convert_to_datetime(run_date, timezone, 'run_date')
+        else:
+            self.run_date = datetime.now(timezone)

    def get_next_fire_time(self, previous_fire_time, now):
        return self.run_date if previous_fire_time is None else None

+    def __getstate__(self):
+        return {
+            'version': 1,
+            'run_date': self.run_date
+        }
+
+    def __setstate__(self, state):
+        # This is for compatibility with APScheduler 3.0.x
+        if isinstance(state, tuple):
+            state = state[1]
+
+        if state.get('version', 1) > 1:
+            raise ValueError(
+                'Got serialized data for version %s of %s, but only version 1 can be handled' %
+                (state['version'], self.__class__.__name__))
+
+        self.run_date = state['run_date']
+
    def __str__(self):
        return 'date[%s]' % datetime_repr(self.run_date)

@@ -4,13 +4,15 @@ from math import ceil
 from tzlocal import get_localzone

 from apscheduler.triggers.base import BaseTrigger
-from apscheduler.util import convert_to_datetime, timedelta_seconds, datetime_repr, astimezone
+from apscheduler.util import (
+    convert_to_datetime, normalize, timedelta_seconds, datetime_repr,
+    astimezone)


 class IntervalTrigger(BaseTrigger):
    """
-    Triggers on specified intervals, starting on ``start_date`` if specified, ``datetime.now()`` + interval
-    otherwise.
+    Triggers on specified intervals, starting on ``start_date`` if specified, ``datetime.now()`` +
+    interval otherwise.

    :param int weeks: number of weeks to wait
    :param int days: number of days to wait
@@ -20,12 +22,15 @@ class IntervalTrigger(BaseTrigger):
    :param datetime|str start_date: starting point for the interval calculation
    :param datetime|str end_date: latest possible date/time to trigger on
    :param datetime.tzinfo|str timezone: time zone to use for the date/time calculations
+    :param int|None jitter: delay the job execution by ``jitter`` seconds at most
    """

-    __slots__ = 'timezone', 'start_date', 'end_date', 'interval'
+    __slots__ = 'timezone', 'start_date', 'end_date', 'interval', 'interval_length', 'jitter'

-    def __init__(self, weeks=0, days=0, hours=0, minutes=0, seconds=0, start_date=None, end_date=None, timezone=None):
-        self.interval = timedelta(weeks=weeks, days=days, hours=hours, minutes=minutes, seconds=seconds)
+    def __init__(self, weeks=0, days=0, hours=0, minutes=0, seconds=0, start_date=None,
+                 end_date=None, timezone=None, jitter=None):
+        self.interval = timedelta(weeks=weeks, days=days, hours=hours, minutes=minutes,
+                                  seconds=seconds)
        self.interval_length = timedelta_seconds(self.interval)
        if self.interval_length == 0:
            self.interval = timedelta(seconds=1)
@@ -33,9 +38,9 @@ class IntervalTrigger(BaseTrigger):

        if timezone:
            self.timezone = astimezone(timezone)
-        elif start_date and start_date.tzinfo:
+        elif isinstance(start_date, datetime) and start_date.tzinfo:
            self.timezone = start_date.tzinfo
-        elif end_date and end_date.tzinfo:
+        elif isinstance(end_date, datetime) and end_date.tzinfo:
            self.timezone = end_date.tzinfo
        else:
            self.timezone = get_localzone()
@@ -44,6 +49,8 @@ class IntervalTrigger(BaseTrigger):
        self.start_date = convert_to_datetime(start_date, self.timezone, 'start_date')
        self.end_date = convert_to_datetime(end_date, self.timezone, 'end_date')

+        self.jitter = jitter
+
    def get_next_fire_time(self, previous_fire_time, now):
        if previous_fire_time:
            next_fire_time = previous_fire_time + self.interval
@@ -54,12 +61,48 @@ class IntervalTrigger(BaseTrigger):
            next_interval_num = int(ceil(timediff_seconds / self.interval_length))
            next_fire_time = self.start_date + self.interval * next_interval_num

+        if self.jitter is not None:
+            next_fire_time = self._apply_jitter(next_fire_time, self.jitter, now)
+
        if not self.end_date or next_fire_time <= self.end_date:
-            return self.timezone.normalize(next_fire_time)
+            return normalize(next_fire_time)
+
+    def __getstate__(self):
+        return {
+            'version': 2,
+            'timezone': self.timezone,
+            'start_date': self.start_date,
+            'end_date': self.end_date,
+            'interval': self.interval,
+            'jitter': self.jitter,
+        }
+
+    def __setstate__(self, state):
+        # This is for compatibility with APScheduler 3.0.x
+        if isinstance(state, tuple):
+            state = state[1]
+
+        if state.get('version', 1) > 2:
+            raise ValueError(
+                'Got serialized data for version %s of %s, but only versions up to 2 can be '
+                'handled' % (state['version'], self.__class__.__name__))
+
+        self.timezone = state['timezone']
+        self.start_date = state['start_date']
+        self.end_date = state['end_date']
+        self.interval = state['interval']
+        self.interval_length = timedelta_seconds(self.interval)
+        self.jitter = state.get('jitter')

    def __str__(self):
        return 'interval[%s]' % str(self.interval)

    def __repr__(self):
-        return "<%s (interval=%r, start_date='%s')>" % (self.__class__.__name__, self.interval,
-                                                        datetime_repr(self.start_date))
+        options = ['interval=%r' % self.interval, 'start_date=%r' % datetime_repr(self.start_date)]
+        if self.end_date:
+            options.append("end_date=%r" % datetime_repr(self.end_date))
+        if self.jitter:
+            options.append('jitter=%s' % self.jitter)
+
+        return "<%s (%s, timezone='%s')>" % (
+            self.__class__.__name__, ', '.join(options), self.timezone)
@@ -1,29 +1,36 @@
 """This module contains several handy functions primarily meant for internal use."""

+from __future__ import division

+from asyncio import iscoroutinefunction
 from datetime import date, datetime, time, timedelta, tzinfo
-from inspect import isfunction, ismethod, getargspec
 from calendar import timegm
+from functools import partial
+from inspect import isclass, ismethod
 import re
+import sys

-from pytz import timezone, utc
+from pytz import timezone, utc, FixedOffset
 import six

 try:
    from inspect import signature
 except ImportError:  # pragma: nocover
-    try:
    from funcsigs import signature
+
+try:
+    from threading import TIMEOUT_MAX
 except ImportError:
-        signature = None
+    TIMEOUT_MAX = 4294967  # Maximum value accepted by Event.wait() on Windows

 __all__ = ('asint', 'asbool', 'astimezone', 'convert_to_datetime', 'datetime_to_utc_timestamp',
-           'utc_timestamp_to_datetime', 'timedelta_seconds', 'datetime_ceil', 'get_callable_name', 'obj_to_ref',
-           'ref_to_obj', 'maybe_ref', 'repr_escape', 'check_callable_args')
+           'utc_timestamp_to_datetime', 'timedelta_seconds', 'datetime_ceil', 'get_callable_name',
+           'obj_to_ref', 'ref_to_obj', 'maybe_ref', 'repr_escape', 'check_callable_args',
+           'normalize', 'localize', 'TIMEOUT_MAX')


 class _Undefined(object):
-    def __bool__(self):
+    def __nonzero__(self):
        return False

    def __bool__(self):
@@ -32,17 +39,18 @@ class _Undefined(object):
    def __repr__(self):
        return '<undefined>'

+
 undefined = _Undefined()  #: a unique object that only signifies that no value is defined


 def asint(text):
    """
-    Safely converts a string to an integer, returning None if the string is None.
+    Safely converts a string to an integer, returning ``None`` if the string is ``None``.

    :type text: str
    :rtype: int
-    """

+    """
    if text is not None:
        return int(text)

@@ -52,8 +60,8 @@ def asbool(obj):
    Interprets an object as a boolean value.

    :rtype: bool
-    """

+    """
    if isinstance(obj, str):
        obj = obj.strip().lower()
        if obj in ('true', 'yes', 'on', 'y', 't', '1'):
@@ -69,15 +77,17 @@ def astimezone(obj):
    Interprets an object as a timezone.

    :rtype: tzinfo
-    """

+    """
    if isinstance(obj, six.string_types):
        return timezone(obj)
    if isinstance(obj, tzinfo):
-        if not hasattr(obj, 'localize') or not hasattr(obj, 'normalize'):
-            raise TypeError('Only timezones from the pytz library are supported')
-        if obj.zone == 'local':
-            raise ValueError('Unable to determine the name of the local timezone -- use an explicit timezone instead')
+        if obj.tzname(None) == 'local':
+            raise ValueError(
+                'Unable to determine the name of the local timezone -- you must explicitly '
+                'specify the name of the local timezone. Please refrain from using timezones like '
+                'EST to prevent problems with daylight saving time. Instead, use a locale based '
+                'timezone name (such as Europe/Helsinki).')
        return obj
    if obj is not None:
        raise TypeError('Expected tzinfo, got %s instead' % obj.__class__.__name__)
@@ -85,27 +95,30 @@ def astimezone(obj):

 _DATE_REGEX = re.compile(
    r'(?P<year>\d{4})-(?P<month>\d{1,2})-(?P<day>\d{1,2})'
-    r'(?: (?P<hour>\d{1,2}):(?P<minute>\d{1,2}):(?P<second>\d{1,2})'
-    r'(?:\.(?P<microsecond>\d{1,6}))?)?')
+    r'(?:[ T](?P<hour>\d{1,2}):(?P<minute>\d{1,2}):(?P<second>\d{1,2})'
+    r'(?:\.(?P<microsecond>\d{1,6}))?'
+    r'(?P<timezone>Z|[+-]\d\d:\d\d)?)?$')


 def convert_to_datetime(input, tz, arg_name):
    """
    Converts the given object to a timezone aware datetime object.
+
    If a timezone aware datetime object is passed, it is returned unmodified.
    If a native datetime object is passed, it is given the specified timezone.
    If the input is a string, it is parsed as a datetime with the given timezone.

-    Date strings are accepted in three different forms: date only (Y-m-d),
-    date with time (Y-m-d H:M:S) or with date+time with microseconds
-    (Y-m-d H:M:S.micro).
+    Date strings are accepted in three different forms: date only (Y-m-d), date with time
+    (Y-m-d H:M:S) or with date+time with microseconds (Y-m-d H:M:S.micro). Additionally you can
+    override the time zone by giving a specific offset in the format specified by ISO 8601:
+    Z (UTC), +HH:MM or -HH:MM.

    :param str|datetime input: the datetime or string to convert to a timezone aware datetime
    :param datetime.tzinfo tz: timezone to interpret ``input`` in
    :param str arg_name: the name of the argument (used in an error message)
    :rtype: datetime
-    """

+    """
    if input is None:
        return
    elif isinstance(input, datetime):
@@ -116,8 +129,17 @@ def convert_to_datetime(input, tz, arg_name):
        m = _DATE_REGEX.match(input)
        if not m:
            raise ValueError('Invalid date string')
-        values = [(k, int(v or 0)) for k, v in list(m.groupdict().items())]
-        values = dict(values)
+
+        values = m.groupdict()
+        tzname = values.pop('timezone')
+        if tzname == 'Z':
+            tz = utc
+        elif tzname:
+            hours, minutes = (int(x) for x in tzname[1:].split(':'))
+            sign = 1 if tzname[0] == '+' else -1
+            tz = FixedOffset(sign * (hours * 60 + minutes))
+
+        values = {k: int(v or 0) for k, v in values.items()}
        datetime_ = datetime(**values)
    else:
        raise TypeError('Unsupported type for %s: %s' % (arg_name, input.__class__.__name__))
@@ -125,14 +147,12 @@ def convert_to_datetime(input, tz, arg_name):
    if datetime_.tzinfo is not None:
        return datetime_
    if tz is None:
-        raise ValueError('The "tz" argument must be specified if %s has no timezone information' % arg_name)
+        raise ValueError(
+            'The "tz" argument must be specified if %s has no timezone information' % arg_name)
    if isinstance(tz, six.string_types):
        tz = timezone(tz)

-    try:
-        return tz.localize(datetime_, is_dst=None)
-    except AttributeError:
-        raise TypeError('Only pytz timezones are supported (need the localize() and normalize() methods)')
+    return localize(datetime_, tz)


 def datetime_to_utc_timestamp(timeval):
@@ -141,8 +161,8 @@ def datetime_to_utc_timestamp(timeval):

    :type timeval: datetime
    :rtype: float
-    """

+    """
    if timeval is not None:
        return timegm(timeval.utctimetuple()) + timeval.microsecond / 1000000

@@ -153,8 +173,8 @@ def utc_timestamp_to_datetime(timestamp):

    :type timestamp: float
    :rtype: datetime
-    """

+    """
    if timestamp is not None:
        return datetime.fromtimestamp(timestamp, utc)

@@ -165,8 +185,8 @@ def timedelta_seconds(delta):

    :type delta: timedelta
    :rtype: float
-    """

+    """
    return delta.days * 24 * 60 * 60 + delta.seconds + \
        delta.microseconds / 1000000.0

@@ -176,8 +196,8 @@ def datetime_ceil(dateval):
    Rounds the given datetime object upwards.

    :type dateval: datetime
-    """

+    """
    if dateval.microsecond > 0:
        return dateval + timedelta(seconds=1, microseconds=-dateval.microsecond)
    return dateval
@@ -192,8 +212,8 @@ def get_callable_name(func):
    Returns the best available display name for the given function/callable.

    :rtype: str
-    """

+    """
    # the easy case (on Python 3.3+)
    if hasattr(func, '__qualname__'):
        return func.__qualname__
@@ -201,7 +221,7 @@ def get_callable_name(func):
    # class methods, bound and unbound methods
    f_self = getattr(func, '__self__', None) or getattr(func, 'im_self', None)
    if f_self and hasattr(func, '__name__'):
-        f_class = f_self if isinstance(f_self, type) else f_self.__class__
+        f_class = f_self if isclass(f_self) else f_self.__class__
    else:
        f_class = getattr(func, 'im_class', None)

@@ -222,20 +242,35 @@ def get_callable_name(func):

 def obj_to_ref(obj):
    """
-    Returns the path to the given object.
+    Returns the path to the given callable.

    :rtype: str
+    :raises TypeError: if the given object is not callable
+    :raises ValueError: if the given object is a :class:`~functools.partial`, lambda or a nested
+        function
+
    """
+    if isinstance(obj, partial):
+        raise ValueError('Cannot create a reference to a partial()')

-    try:
-        ref = '%s:%s' % (obj.__module__, get_callable_name(obj))
-        obj2 = ref_to_obj(ref)
-        if obj != obj2:
-            raise ValueError
-    except Exception:
-        raise ValueError('Cannot determine the reference to %r' % obj)
+    name = get_callable_name(obj)
+    if '<lambda>' in name:
+        raise ValueError('Cannot create a reference to a lambda')
+    if '<locals>' in name:
+        raise ValueError('Cannot create a reference to a nested function')

-    return ref
+    if ismethod(obj):
+        if hasattr(obj, 'im_self') and obj.im_self:
+            # bound method
+            module = obj.im_self.__module__
+        elif hasattr(obj, 'im_class') and obj.im_class:
+            # unbound method
+            module = obj.im_class.__module__
+        else:
+            module = obj.__module__
+    else:
+        module = obj.__module__
+    return '%s:%s' % (module, name)


 def ref_to_obj(ref):
@@ -243,8 +278,8 @@ def ref_to_obj(ref):
    Returns the object pointed to by ``ref``.

    :type ref: str
-    """

+    """
    if not isinstance(ref, six.string_types):
        raise TypeError('References must be strings')
    if ':' not in ref:
@@ -252,12 +287,12 @@ def ref_to_obj(ref):

    modulename, rest = ref.split(':', 1)
    try:
-        obj = __import__(modulename)
+        obj = __import__(modulename, fromlist=[rest])
    except ImportError:
        raise LookupError('Error resolving reference %s: could not import module' % ref)

    try:
-        for name in modulename.split('.')[1:] + rest.split('.'):
+        for name in rest.split('.'):
            obj = getattr(obj, name)
        return obj
    except Exception:
@@ -268,8 +303,8 @@ def maybe_ref(ref):
    """
    Returns the object that the given reference points to, if it is indeed a reference.
    If it is not a reference, the object is returned as-is.
-    """

+    """
    if not isinstance(ref, str):
        return ref
    return ref_to_obj(ref)
@@ -281,7 +316,8 @@ if six.PY2:
            return string.encode('ascii', 'backslashreplace')
        return string
 else:
-    repr_escape = lambda string: string
+    def repr_escape(string):
+        return string


 def check_callable_args(func, args, kwargs):
@@ -290,21 +326,26 @@ def check_callable_args(func, args, kwargs):

    :type args: tuple
    :type kwargs: dict
-    """

+    """
    pos_kwargs_conflicts = []  # parameters that have a match in both args and kwargs
    positional_only_kwargs = []  # positional-only parameters that have a match in kwargs
    unsatisfied_args = []  # parameters in signature that don't have a match in args or kwargs
    unsatisfied_kwargs = []  # keyword-only arguments that don't have a match in kwargs
    unmatched_args = list(args)  # args that didn't match any of the parameters in the signature
-    unmatched_kwargs = list(kwargs)  # kwargs that didn't match any of the parameters in the signature
-    has_varargs = has_var_kwargs = False  # indicates if the signature defines *args and **kwargs respectively
+    # kwargs that didn't match any of the parameters in the signature
+    unmatched_kwargs = list(kwargs)
+    # indicates if the signature defines *args and **kwargs respectively
+    has_varargs = has_var_kwargs = False

-    if signature:
    try:
+        if sys.version_info >= (3, 5):
+            sig = signature(func, follow_wrapped=False)
+        else:
            sig = signature(func)
    except ValueError:
-            return  # signature() doesn't work against every kind of callable
+        # signature() doesn't work against every kind of callable
+        return

    for param in six.itervalues(sig.parameters):
        if param.kind == param.POSITIONAL_OR_KEYWORD:
@@ -333,27 +374,6 @@ def check_callable_args(func, args, kwargs):
            has_varargs = True
        elif param.kind == param.VAR_KEYWORD:
            has_var_kwargs = True
-    else:
-        if not isfunction(func) and not ismethod(func) and hasattr(func, '__call__'):
-            func = func.__call__
-
-        try:
-            argspec = getargspec(func)
-        except TypeError:
-            return  # getargspec() doesn't work certain callables
-
-        argspec_args = argspec.args if not ismethod(func) else argspec.args[1:]
-        has_varargs = bool(argspec.varargs)
-        has_var_kwargs = bool(argspec.keywords)
-        for arg, default in six.moves.zip_longest(argspec_args, argspec.defaults or (), fillvalue=undefined):
-            if arg in unmatched_kwargs and unmatched_args:
-                pos_kwargs_conflicts.append(arg)
-            elif unmatched_args:
-                del unmatched_args[0]
-            elif arg in unmatched_kwargs:
-                unmatched_kwargs.remove(arg)
-            elif default is undefined:
-                unsatisfied_args.append(arg)

    # Make sure there are no conflicts between args and kwargs
    if pos_kwargs_conflicts:
@@ -365,21 +385,46 @@ def check_callable_args(func, args, kwargs):
        raise ValueError('The following arguments cannot be given as keyword arguments: %s' %
                         ', '.join(positional_only_kwargs))

-    # Check that the number of positional arguments minus the number of matched kwargs matches the argspec
+    # Check that the number of positional arguments minus the number of matched kwargs matches the
+    # argspec
    if unsatisfied_args:
-        raise ValueError('The following arguments have not been supplied: %s' % ', '.join(unsatisfied_args))
+        raise ValueError('The following arguments have not been supplied: %s' %
+                         ', '.join(unsatisfied_args))

    # Check that all keyword-only arguments have been supplied
    if unsatisfied_kwargs:
-        raise ValueError('The following keyword-only arguments have not been supplied in kwargs: %s' %
+        raise ValueError(
+            'The following keyword-only arguments have not been supplied in kwargs: %s' %
            ', '.join(unsatisfied_kwargs))

    # Check that the callable can accept the given number of positional arguments
    if not has_varargs and unmatched_args:
-        raise ValueError('The list of positional arguments is longer than the target callable can handle '
+        raise ValueError(
+            'The list of positional arguments is longer than the target callable can handle '
            '(allowed: %d, given in args: %d)' % (len(args) - len(unmatched_args), len(args)))

    # Check that the callable can accept the given keyword arguments
    if not has_var_kwargs and unmatched_kwargs:
-        raise ValueError('The target callable does not accept the following keyword arguments: %s' %
+        raise ValueError(
+            'The target callable does not accept the following keyword arguments: %s' %
            ', '.join(unmatched_kwargs))
+
+
+def iscoroutinefunction_partial(f):
+    while isinstance(f, partial):
+        f = f.func
+
+    # The asyncio version of iscoroutinefunction includes testing for @coroutine
+    # decorations vs. the inspect version which does not.
+    return iscoroutinefunction(f)
+
+
+def normalize(dt):
+    return datetime.fromtimestamp(dt.timestamp(), dt.tzinfo)
+
+
+def localize(dt, tzinfo):
+    if hasattr(tzinfo, 'localize'):
+        return tzinfo.localize(dt)
+
+    return normalize(dt.replace(tzinfo=tzinfo))
@@ -13,28 +13,29 @@
 # included in all copies or substantial portions of the Software.


-import confuse
 from sys import stderr

-__version__ = '1.6.0'
-__author__ = 'Adrian Sampson <adrian@radbox.org>'
+import confuse
+
+__version__ = "2.0.0"
+__author__ = "Adrian Sampson <adrian@radbox.org>"


 class IncludeLazyConfig(confuse.LazyConfig):
    """A version of Confuse's LazyConfig that also merges in data from
    YAML files specified in an `include` setting.
    """
+
    def read(self, user=True, defaults=True):
        super().read(user, defaults)

        try:
-            for view in self['include']:
+            for view in self["include"]:
                self.set_file(view.as_filename())
        except confuse.NotFoundError:
            pass
        except confuse.ConfigReadError as err:
-            stderr.write("configuration `import` failed: {}"
-                         .format(err.reason))
+            stderr.write("configuration `import` failed: {}".format(err.reason))


-config = IncludeLazyConfig('beets', __name__)
+config = IncludeLazyConfig("beets", __name__)
@@ -18,6 +18,7 @@


 import sys
+
 from .ui import main

 if __name__ == "__main__":
@@ -17,21 +17,19 @@ music and items' embedded album art.
 """


-import subprocess
-import platform
-from tempfile import NamedTemporaryFile
 import os
+from tempfile import NamedTemporaryFile

-from beets.util import displayable_path, syspath, bytestring_path
-from beets.util.artresizer import ArtResizer
 import mediafile

+from beets.util import bytestring_path, displayable_path, syspath
+from beets.util.artresizer import ArtResizer
+

 def mediafile_image(image_path, maxwidth=None):
-    """Return a `mediafile.Image` object for the path.
-    """
+    """Return a `mediafile.Image` object for the path."""

-    with open(syspath(image_path), 'rb') as f:
+    with open(syspath(image_path), "rb") as f:
        data = f.read()
    return mediafile.Image(data, type=mediafile.ImageType.front)

@@ -41,170 +39,168 @@ def get_art(log, item):
    try:
        mf = mediafile.MediaFile(syspath(item.path))
    except mediafile.UnreadableFileError as exc:
-        log.warning('Could not extract art from {0}: {1}',
-                    displayable_path(item.path), exc)
+        log.warning(
+            "Could not extract art from {0}: {1}",
+            displayable_path(item.path),
+            exc,
+        )
        return

    return mf.art


-def embed_item(log, item, imagepath, maxwidth=None, itempath=None,
-               compare_threshold=0, ifempty=False, as_album=False, id3v23=None,
-               quality=0):
-    """Embed an image into the item's media file.
-    """
-    # Conditions and filters.
+def embed_item(
+    log,
+    item,
+    imagepath,
+    maxwidth=None,
+    itempath=None,
+    compare_threshold=0,
+    ifempty=False,
+    as_album=False,
+    id3v23=None,
+    quality=0,
+):
+    """Embed an image into the item's media file."""
+    # Conditions.
    if compare_threshold:
-        if not check_art_similarity(log, item, imagepath, compare_threshold):
-            log.info('Image not similar; skipping.')
+        is_similar = check_art_similarity(
+            log, item, imagepath, compare_threshold
+        )
+        if is_similar is None:
+            log.warning("Error while checking art similarity; skipping.")
            return
+        elif not is_similar:
+            log.info("Image not similar; skipping.")
+            return
+
    if ifempty and get_art(log, item):
-        log.info('media file already contained art')
+        log.info("media file already contained art")
        return
+
+    # Filters.
    if maxwidth and not as_album:
        imagepath = resize_image(log, imagepath, maxwidth, quality)

    # Get the `Image` object from the file.
    try:
-        log.debug('embedding {0}', displayable_path(imagepath))
+        log.debug("embedding {0}", displayable_path(imagepath))
        image = mediafile_image(imagepath, maxwidth)
    except OSError as exc:
-        log.warning('could not read image file: {0}', exc)
+        log.warning("could not read image file: {0}", exc)
        return

    # Make sure the image kind is safe (some formats only support PNG
    # and JPEG).
-    if image.mime_type not in ('image/jpeg', 'image/png'):
-        log.info('not embedding image of unsupported type: {}',
-                 image.mime_type)
+    if image.mime_type not in ("image/jpeg", "image/png"):
+        log.info("not embedding image of unsupported type: {}", image.mime_type)
        return

-    item.try_write(path=itempath, tags={'images': [image]}, id3v23=id3v23)
+    item.try_write(path=itempath, tags={"images": [image]}, id3v23=id3v23)


-def embed_album(log, album, maxwidth=None, quiet=False, compare_threshold=0,
-                ifempty=False, quality=0):
-    """Embed album art into all of the album's items.
-    """
+def embed_album(
+    log,
+    album,
+    maxwidth=None,
+    quiet=False,
+    compare_threshold=0,
+    ifempty=False,
+    quality=0,
+):
+    """Embed album art into all of the album's items."""
    imagepath = album.artpath
    if not imagepath:
-        log.info('No album art present for {0}', album)
+        log.info("No album art present for {0}", album)
        return
    if not os.path.isfile(syspath(imagepath)):
-        log.info('Album art not found at {0} for {1}',
-                 displayable_path(imagepath), album)
+        log.info(
+            "Album art not found at {0} for {1}",
+            displayable_path(imagepath),
+            album,
+        )
        return
    if maxwidth:
        imagepath = resize_image(log, imagepath, maxwidth, quality)

-    log.info('Embedding album art into {0}', album)
+    log.info("Embedding album art into {0}", album)

    for item in album.items():
-        embed_item(log, item, imagepath, maxwidth, None, compare_threshold,
-                   ifempty, as_album=True, quality=quality)
+        embed_item(
+            log,
+            item,
+            imagepath,
+            maxwidth,
+            None,
+            compare_threshold,
+            ifempty,
+            as_album=True,
+            quality=quality,
+        )


 def resize_image(log, imagepath, maxwidth, quality):
    """Returns path to an image resized to maxwidth and encoded with the
    specified quality level.
    """
-    log.debug('Resizing album art to {0} pixels wide and encoding at quality \
-              level {1}', maxwidth, quality)
-    imagepath = ArtResizer.shared.resize(maxwidth, syspath(imagepath),
-                                         quality=quality)
+    log.debug(
+        "Resizing album art to {0} pixels wide and encoding at quality \
+              level {1}",
+        maxwidth,
+        quality,
+    )
+    imagepath = ArtResizer.shared.resize(
+        maxwidth, syspath(imagepath), quality=quality
+    )
    return imagepath


-def check_art_similarity(log, item, imagepath, compare_threshold):
+def check_art_similarity(
+    log,
+    item,
+    imagepath,
+    compare_threshold,
+    artresizer=None,
+):
    """A boolean indicating if an image is similar to embedded item art.
+
+    If no embedded art exists, always return `True`. If the comparison fails
+    for some reason, the return value is `None`.
+
+    This must only be called if `ArtResizer.shared.can_compare` is `True`.
    """
    with NamedTemporaryFile(delete=True) as f:
        art = extract(log, f.name, item)

-        if art:
-            is_windows = platform.system() == "Windows"
-
-            # Converting images to grayscale tends to minimize the weight
-            # of colors in the diff score. So we first convert both images
-            # to grayscale and then pipe them into the `compare` command.
-            # On Windows, ImageMagick doesn't support the magic \\?\ prefix
-            # on paths, so we pass `prefix=False` to `syspath`.
-            convert_cmd = ['convert', syspath(imagepath, prefix=False),
-                           syspath(art, prefix=False),
-                           '-colorspace', 'gray', 'MIFF:-']
-            compare_cmd = ['compare', '-metric', 'PHASH', '-', 'null:']
-            log.debug('comparing images with pipeline {} | {}',
-                      convert_cmd, compare_cmd)
-            convert_proc = subprocess.Popen(
-                convert_cmd,
-                stdout=subprocess.PIPE,
-                stderr=subprocess.PIPE,
-                close_fds=not is_windows,
-            )
-            compare_proc = subprocess.Popen(
-                compare_cmd,
-                stdin=convert_proc.stdout,
-                stdout=subprocess.PIPE,
-                stderr=subprocess.PIPE,
-                close_fds=not is_windows,
-            )
-
-            # Check the convert output. We're not interested in the
-            # standard output; that gets piped to the next stage.
-            convert_proc.stdout.close()
-            convert_stderr = convert_proc.stderr.read()
-            convert_proc.stderr.close()
-            convert_proc.wait()
-            if convert_proc.returncode:
-                log.debug(
-                    'ImageMagick convert failed with status {}: {!r}',
-                    convert_proc.returncode,
-                    convert_stderr,
-                )
-                return
-
-            # Check the compare output.
-            stdout, stderr = compare_proc.communicate()
-            if compare_proc.returncode:
-                if compare_proc.returncode != 1:
-                    log.debug('ImageMagick compare failed: {0}, {1}',
-                              displayable_path(imagepath),
-                              displayable_path(art))
-                    return
-                out_str = stderr
-            else:
-                out_str = stdout
-
-            try:
-                phash_diff = float(out_str)
-            except ValueError:
-                log.debug('IM output is not a number: {0!r}', out_str)
-                return
-
-            log.debug('ImageMagick compare score: {0}', phash_diff)
-            return phash_diff <= compare_threshold
-
+        if not art:
            return True

+        if artresizer is None:
+            artresizer = ArtResizer.shared
+
+        return artresizer.compare(art, imagepath, compare_threshold)
+

 def extract(log, outpath, item):
    art = get_art(log, item)
    outpath = bytestring_path(outpath)
    if not art:
-        log.info('No album art present in {0}, skipping.', item)
+        log.info("No album art present in {0}, skipping.", item)
        return

    # Add an extension to the filename.
    ext = mediafile.image_extension(art)
    if not ext:
-        log.warning('Unknown image type in {0}.',
-                    displayable_path(item.path))
+        log.warning("Unknown image type in {0}.", displayable_path(item.path))
        return
-    outpath += bytestring_path('.' + ext)
+    outpath += bytestring_path("." + ext)

-    log.info('Extracting album art from: {0} to: {1}',
-             item, displayable_path(outpath))
-    with open(syspath(outpath), 'wb') as f:
+    log.info(
+        "Extracting album art from: {0} to: {1}",
+        item,
+        displayable_path(outpath),
+    )
+    with open(syspath(outpath), "wb") as f:
        f.write(art)
    return outpath

@@ -218,7 +214,7 @@ def extract_first(log, outpath, items):

 def clear(log, lib, query):
    items = lib.items(query)
-    log.info('Clearing album art from {0} items', len(items))
+    log.info("Clearing album art from {0} items", len(items))
    for item in items:
-        log.debug('Clearing art for {0}', item)
-        item.try_write(tags={'images': None})
+        log.debug("Clearing art for {0}", item)
+        item.try_write(tags={"images": None})
@@ -14,78 +14,91 @@

 """Facilities for automatically determining files' correct metadata.
 """
+from typing import Mapping

-
-from beets import logging
-from beets import config
+from beets import config, logging
+from beets.library import Item

 # Parts of external interface.
 from .hooks import (  # noqa
    AlbumInfo,
-    TrackInfo,
    AlbumMatch,
-    TrackMatch,
    Distance,
+    TrackInfo,
+    TrackMatch,
 )
-from .match import tag_item, tag_album, Proposal  # noqa
 from .match import Recommendation  # noqa
+from .match import Proposal, current_metadata, tag_album, tag_item  # noqa

 # Global logger.
-log = logging.getLogger('beets')
+log = logging.getLogger("beets")

 # Metadata fields that are already hardcoded, or where the tag name changes.
 SPECIAL_FIELDS = {
-    'album': (
-        'va',
-        'releasegroup_id',
-        'artist_id',
-        'album_id',
-        'mediums',
-        'tracks',
-        'year',
-        'month',
-        'day',
-        'artist',
-        'artist_credit',
-        'artist_sort',
-        'data_url'
+    "album": (
+        "va",
+        "releasegroup_id",
+        "artist_id",
+        "artists_ids",
+        "album_id",
+        "mediums",
+        "tracks",
+        "year",
+        "month",
+        "day",
+        "artist",
+        "artists",
+        "artist_credit",
+        "artists_credit",
+        "artist_sort",
+        "artists_sort",
+        "data_url",
+    ),
+    "track": (
+        "track_alt",
+        "artist_id",
+        "artists_ids",
+        "release_track_id",
+        "medium",
+        "index",
+        "medium_index",
+        "title",
+        "artist_credit",
+        "artists_credit",
+        "artist_sort",
+        "artists_sort",
+        "artist",
+        "artists",
+        "track_id",
+        "medium_total",
+        "data_url",
+        "length",
    ),
-    'track': (
-        'track_alt',
-        'artist_id',
-        'release_track_id',
-        'medium',
-        'index',
-        'medium_index',
-        'title',
-        'artist_credit',
-        'artist_sort',
-        'artist',
-        'track_id',
-        'medium_total',
-        'data_url',
-        'length'
-    )
 }


 # Additional utilities for the main interface.

-def apply_item_metadata(item, track_info):
-    """Set an item's metadata from its matched TrackInfo object.
-    """
+
+def apply_item_metadata(item: Item, track_info: TrackInfo):
+    """Set an item's metadata from its matched TrackInfo object."""
    item.artist = track_info.artist
+    item.artists = track_info.artists
    item.artist_sort = track_info.artist_sort
+    item.artists_sort = track_info.artists_sort
    item.artist_credit = track_info.artist_credit
+    item.artists_credit = track_info.artists_credit
    item.title = track_info.title
    item.mb_trackid = track_info.track_id
    item.mb_releasetrackid = track_info.release_track_id
    if track_info.artist_id:
        item.mb_artistid = track_info.artist_id
+    if track_info.artists_ids:
+        item.mb_artistids = track_info.artists_ids

    for field, value in track_info.items():
        # We only overwrite fields that are not already hardcoded.
-        if field in SPECIAL_FIELDS['track']:
+        if field in SPECIAL_FIELDS["track"]:
            continue
        if value is None:
            continue
@@ -95,45 +108,62 @@ def apply_item_metadata(item, track_info):
    # and track number). Perhaps these should be emptied?


-def apply_metadata(album_info, mapping):
+def apply_metadata(album_info: AlbumInfo, mapping: Mapping[Item, TrackInfo]):
    """Set the items' metadata to match an AlbumInfo object using a
    mapping from Items to TrackInfo objects.
    """
    for item, track_info in mapping.items():
        # Artist or artist credit.
-        if config['artist_credit']:
-            item.artist = (track_info.artist_credit or
-                           track_info.artist or
-                           album_info.artist_credit or
-                           album_info.artist)
-            item.albumartist = (album_info.artist_credit or
-                                album_info.artist)
+        if config["artist_credit"]:
+            item.artist = (
+                track_info.artist_credit
+                or track_info.artist
+                or album_info.artist_credit
+                or album_info.artist
+            )
+            item.artists = (
+                track_info.artists_credit
+                or track_info.artists
+                or album_info.artists_credit
+                or album_info.artists
+            )
+            item.albumartist = album_info.artist_credit or album_info.artist
+            item.albumartists = album_info.artists_credit or album_info.artists
        else:
-            item.artist = (track_info.artist or album_info.artist)
+            item.artist = track_info.artist or album_info.artist
+            item.artists = track_info.artists or album_info.artists
            item.albumartist = album_info.artist
+            item.albumartists = album_info.artists

        # Album.
        item.album = album_info.album

        # Artist sort and credit names.
        item.artist_sort = track_info.artist_sort or album_info.artist_sort
-        item.artist_credit = (track_info.artist_credit or
-                              album_info.artist_credit)
+        item.artists_sort = track_info.artists_sort or album_info.artists_sort
+        item.artist_credit = (
+            track_info.artist_credit or album_info.artist_credit
+        )
+        item.artists_credit = (
+            track_info.artists_credit or album_info.artists_credit
+        )
        item.albumartist_sort = album_info.artist_sort
+        item.albumartists_sort = album_info.artists_sort
        item.albumartist_credit = album_info.artist_credit
+        item.albumartists_credit = album_info.artists_credit

        # Release date.
-        for prefix in '', 'original_':
-            if config['original_date'] and not prefix:
+        for prefix in "", "original_":
+            if config["original_date"] and not prefix:
                # Ignore specific release date.
                continue

-            for suffix in 'year', 'month', 'day':
+            for suffix in "year", "month", "day":
                key = prefix + suffix
                value = getattr(album_info, key) or 0

                # If we don't even have a year, apply nothing.
-                if suffix == 'year' and not value:
+                if suffix == "year" and not value:
                    break

                # Otherwise, set the fetched value (or 0 for the month
@@ -142,13 +172,13 @@ def apply_metadata(album_info, mapping):

                # If we're using original release date for both fields,
                # also set item.year = info.original_year, etc.
-                if config['original_date']:
+                if config["original_date"]:
                    item[suffix] = value

        # Title.
        item.title = track_info.title

-        if config['per_disc_numbering']:
+        if config["per_disc_numbering"]:
            # We want to let the track number be zero, but if the medium index
            # is not provided we need to fall back to the overall index.
            if track_info.medium_index is not None:
@@ -172,7 +202,14 @@ def apply_metadata(album_info, mapping):
            item.mb_artistid = track_info.artist_id
        else:
            item.mb_artistid = album_info.artist_id
+
+        if track_info.artists_ids:
+            item.mb_artistids = track_info.artists_ids
+        else:
+            item.mb_artistids = album_info.artists_ids
+
        item.mb_albumartistid = album_info.artist_id
+        item.mb_albumartistids = album_info.artists_ids
        item.mb_releasegroupid = album_info.releasegroup_id

        # Compilation flag.
@@ -184,17 +221,17 @@ def apply_metadata(album_info, mapping):
        # Don't overwrite fields with empty values unless the
        # field is explicitly allowed to be overwritten
        for field, value in album_info.items():
-            if field in SPECIAL_FIELDS['album']:
+            if field in SPECIAL_FIELDS["album"]:
                continue
-            clobber = field in config['overwrite_null']['album'].as_str_seq()
+            clobber = field in config["overwrite_null"]["album"].as_str_seq()
            if value is None and not clobber:
                continue
            item[field] = value

        for field, value in track_info.items():
-            if field in SPECIAL_FIELDS['track']:
+            if field in SPECIAL_FIELDS["track"]:
                continue
-            clobber = field in config['overwrite_null']['track'].as_str_seq()
+            clobber = field in config["overwrite_null"]["track"].as_str_seq()
            value = getattr(track_info, field)
            if value is None and not clobber:
                continue
@@ -14,40 +14,51 @@

 """Glue between metadata sources and the matching logic."""

+from __future__ import annotations
+
+import re
 from collections import namedtuple
 from functools import total_ordering
-import re
+from typing import (
+    Any,
+    Callable,
+    Dict,
+    Iterable,
+    Iterator,
+    List,
+    Optional,
+    Tuple,
+    TypeVar,
+    Union,
+    cast,
+)

-from beets import logging
-from beets import plugins
-from beets import config
-from beets.util import as_string
-from beets.autotag import mb
 from jellyfish import levenshtein_distance
 from unidecode import unidecode

-log = logging.getLogger('beets')
+from beets import config, logging, plugins
+from beets.autotag import mb
+from beets.library import Item
+from beets.util import as_string, cached_classproperty

-# The name of the type for patterns in re changed in Python 3.7.
-try:
-    Pattern = re._pattern_type
-except AttributeError:
-    Pattern = re.Pattern
+log = logging.getLogger("beets")
+
+V = TypeVar("V")


 # Classes used to represent candidate options.
-class AttrDict(dict):
+class AttrDict(Dict[str, V]):
    """A dictionary that supports attribute ("dot") access, so `d.field`
    is equivalent to `d['field']`.
    """

-    def __getattr__(self, attr):
+    def __getattr__(self, attr: str) -> V:
        if attr in self:
-            return self.get(attr)
+            return self[attr]
        else:
            raise AttributeError

-    def __setattr__(self, key, value):
+    def __setattr__(self, key: str, value: V):
        self.__setitem__(key, value)

    def __hash__(self):
@@ -68,32 +79,73 @@ class AlbumInfo(AttrDict):
    The others are optional and may be None.
    """

-    def __init__(self, tracks, album=None, album_id=None, artist=None,
-                 artist_id=None, asin=None, albumtype=None, va=False,
-                 year=None, month=None, day=None, label=None, mediums=None,
-                 artist_sort=None, releasegroup_id=None, catalognum=None,
-                 script=None, language=None, country=None, style=None,
-                 genre=None, albumstatus=None, media=None, albumdisambig=None,
-                 releasegroupdisambig=None, artist_credit=None,
-                 original_year=None, original_month=None,
-                 original_day=None, data_source=None, data_url=None,
-                 discogs_albumid=None, discogs_labelid=None,
-                 discogs_artistid=None, **kwargs):
+    # TYPING: are all of these correct? I've assumed optional strings
+    def __init__(
+        self,
+        tracks: List[TrackInfo],
+        album: Optional[str] = None,
+        album_id: Optional[str] = None,
+        artist: Optional[str] = None,
+        artist_id: Optional[str] = None,
+        artists: Optional[List[str]] = None,
+        artists_ids: Optional[List[str]] = None,
+        asin: Optional[str] = None,
+        albumtype: Optional[str] = None,
+        albumtypes: Optional[List[str]] = None,
+        va: bool = False,
+        year: Optional[int] = None,
+        month: Optional[int] = None,
+        day: Optional[int] = None,
+        label: Optional[str] = None,
+        barcode: Optional[str] = None,
+        mediums: Optional[int] = None,
+        artist_sort: Optional[str] = None,
+        artists_sort: Optional[List[str]] = None,
+        releasegroup_id: Optional[str] = None,
+        release_group_title: Optional[str] = None,
+        catalognum: Optional[str] = None,
+        script: Optional[str] = None,
+        language: Optional[str] = None,
+        country: Optional[str] = None,
+        style: Optional[str] = None,
+        genre: Optional[str] = None,
+        albumstatus: Optional[str] = None,
+        media: Optional[str] = None,
+        albumdisambig: Optional[str] = None,
+        releasegroupdisambig: Optional[str] = None,
+        artist_credit: Optional[str] = None,
+        artists_credit: Optional[List[str]] = None,
+        original_year: Optional[int] = None,
+        original_month: Optional[int] = None,
+        original_day: Optional[int] = None,
+        data_source: Optional[str] = None,
+        data_url: Optional[str] = None,
+        discogs_albumid: Optional[str] = None,
+        discogs_labelid: Optional[str] = None,
+        discogs_artistid: Optional[str] = None,
+        **kwargs,
+    ):
        self.album = album
        self.album_id = album_id
        self.artist = artist
        self.artist_id = artist_id
+        self.artists = artists or []
+        self.artists_ids = artists_ids or []
        self.tracks = tracks
        self.asin = asin
        self.albumtype = albumtype
+        self.albumtypes = albumtypes or []
        self.va = va
        self.year = year
        self.month = month
        self.day = day
        self.label = label
+        self.barcode = barcode
        self.mediums = mediums
        self.artist_sort = artist_sort
+        self.artists_sort = artists_sort or []
        self.releasegroup_id = releasegroup_id
+        self.release_group_title = release_group_title
        self.catalognum = catalognum
        self.script = script
        self.language = language
@@ -105,6 +157,7 @@ class AlbumInfo(AttrDict):
        self.albumdisambig = albumdisambig
        self.releasegroupdisambig = releasegroupdisambig
        self.artist_credit = artist_credit
+        self.artists_credit = artists_credit or []
        self.original_year = original_year
        self.original_month = original_month
        self.original_day = original_day
@@ -115,27 +168,7 @@ class AlbumInfo(AttrDict):
        self.discogs_artistid = discogs_artistid
        self.update(kwargs)

-    # Work around a bug in python-musicbrainz-ngs that causes some
-    # strings to be bytes rather than Unicode.
-    # https://github.com/alastair/python-musicbrainz-ngs/issues/85
-    def decode(self, codec='utf-8'):
-        """Ensure that all string attributes on this object, and the
-        constituent `TrackInfo` objects, are decoded to Unicode.
-        """
-        for fld in ['album', 'artist', 'albumtype', 'label', 'artist_sort',
-                    'catalognum', 'script', 'language', 'country', 'style',
-                    'genre', 'albumstatus', 'albumdisambig',
-                    'releasegroupdisambig', 'artist_credit',
-                    'media', 'discogs_albumid', 'discogs_labelid',
-                    'discogs_artistid']:
-            value = getattr(self, fld)
-            if isinstance(value, bytes):
-                setattr(self, fld, value.decode(codec, 'ignore'))
-
-        for track in self.tracks:
-            track.decode(codec)
-
-    def copy(self):
+    def copy(self) -> AlbumInfo:
        dupe = AlbumInfo([])
        dupe.update(self)
        dupe.tracks = [track.copy() for track in self.tracks]
@@ -154,20 +187,50 @@ class TrackInfo(AttrDict):
    are all 1-based.
    """

-    def __init__(self, title=None, track_id=None, release_track_id=None,
-                 artist=None, artist_id=None, length=None, index=None,
-                 medium=None, medium_index=None, medium_total=None,
-                 artist_sort=None, disctitle=None, artist_credit=None,
-                 data_source=None, data_url=None, media=None, lyricist=None,
-                 composer=None, composer_sort=None, arranger=None,
-                 track_alt=None, work=None, mb_workid=None,
-                 work_disambig=None, bpm=None, initial_key=None, genre=None,
-                 **kwargs):
+    # TYPING: are all of these correct? I've assumed optional strings
+    def __init__(
+        self,
+        title: Optional[str] = None,
+        track_id: Optional[str] = None,
+        release_track_id: Optional[str] = None,
+        artist: Optional[str] = None,
+        artist_id: Optional[str] = None,
+        artists: Optional[List[str]] = None,
+        artists_ids: Optional[List[str]] = None,
+        length: Optional[float] = None,
+        index: Optional[int] = None,
+        medium: Optional[int] = None,
+        medium_index: Optional[int] = None,
+        medium_total: Optional[int] = None,
+        artist_sort: Optional[str] = None,
+        artists_sort: Optional[List[str]] = None,
+        disctitle: Optional[str] = None,
+        artist_credit: Optional[str] = None,
+        artists_credit: Optional[List[str]] = None,
+        data_source: Optional[str] = None,
+        data_url: Optional[str] = None,
+        media: Optional[str] = None,
+        lyricist: Optional[str] = None,
+        composer: Optional[str] = None,
+        composer_sort: Optional[str] = None,
+        arranger: Optional[str] = None,
+        track_alt: Optional[str] = None,
+        work: Optional[str] = None,
+        mb_workid: Optional[str] = None,
+        work_disambig: Optional[str] = None,
+        bpm: Optional[str] = None,
+        initial_key: Optional[str] = None,
+        genre: Optional[str] = None,
+        album: Optional[str] = None,
+        **kwargs,
+    ):
        self.title = title
        self.track_id = track_id
        self.release_track_id = release_track_id
        self.artist = artist
        self.artist_id = artist_id
+        self.artists = artists or []
+        self.artists_ids = artists_ids or []
        self.length = length
        self.index = index
        self.media = media
@@ -175,8 +238,10 @@ class TrackInfo(AttrDict):
        self.medium_index = medium_index
        self.medium_total = medium_total
        self.artist_sort = artist_sort
+        self.artists_sort = artists_sort or []
        self.disctitle = disctitle
        self.artist_credit = artist_credit
+        self.artists_credit = artists_credit or []
        self.data_source = data_source
        self.data_url = data_url
        self.lyricist = lyricist
@@ -190,20 +255,10 @@ class TrackInfo(AttrDict):
        self.bpm = bpm
        self.initial_key = initial_key
        self.genre = genre
+        self.album = album
        self.update(kwargs)

-    # As above, work around a bug in python-musicbrainz-ngs.
-    def decode(self, codec='utf-8'):
-        """Ensure that all string attributes on this object are decoded
-        to Unicode.
-        """
-        for fld in ['title', 'artist', 'medium', 'artist_sort', 'disctitle',
-                    'artist_credit', 'media']:
-            value = getattr(self, fld)
-            if isinstance(value, bytes):
-                setattr(self, fld, value.decode(codec, 'ignore'))
-
-    def copy(self):
+    def copy(self) -> TrackInfo:
        dupe = TrackInfo()
        dupe.update(self)
        return dupe
@@ -213,23 +268,23 @@ class TrackInfo(AttrDict):

 # Parameters for string distance function.
 # Words that can be moved to the end of a string using a comma.
-SD_END_WORDS = ['the', 'a', 'an']
+SD_END_WORDS = ["the", "a", "an"]
 # Reduced weights for certain portions of the string.
 SD_PATTERNS = [
-    (r'^the ', 0.1),
-    (r'[\[\(]?(ep|single)[\]\)]?', 0.0),
-    (r'[\[\(]?(featuring|feat|ft)[\. :].+', 0.1),
-    (r'\(.*?\)', 0.3),
-    (r'\[.*?\]', 0.3),
-    (r'(, )?(pt\.|part) .+', 0.2),
+    (r"^the ", 0.1),
+    (r"[\[\(]?(ep|single)[\]\)]?", 0.0),
+    (r"[\[\(]?(featuring|feat|ft)[\. :].+", 0.1),
+    (r"\(.*?\)", 0.3),
+    (r"\[.*?\]", 0.3),
+    (r"(, )?(pt\.|part) .+", 0.2),
 ]
 # Replacements to use before testing distance.
 SD_REPLACE = [
-    (r'&', 'and'),
+    (r"&", "and"),
 ]


-def _string_dist_basic(str1, str2):
+def _string_dist_basic(str1: str, str2: str) -> float:
    """Basic edit distance between two strings, ignoring
    non-alphanumeric characters and case. Comparisons are based on a
    transliteration/lowering to ASCII characters. Normalized by string
@@ -239,14 +294,14 @@ def _string_dist_basic(str1, str2):
    assert isinstance(str2, str)
    str1 = as_string(unidecode(str1))
    str2 = as_string(unidecode(str2))
-    str1 = re.sub(r'[^a-z0-9]', '', str1.lower())
-    str2 = re.sub(r'[^a-z0-9]', '', str2.lower())
+    str1 = re.sub(r"[^a-z0-9]", "", str1.lower())
+    str2 = re.sub(r"[^a-z0-9]", "", str2.lower())
    if not str1 and not str2:
        return 0.0
    return levenshtein_distance(str1, str2) / float(max(len(str1), len(str2)))


-def string_dist(str1, str2):
+def string_dist(str1: Optional[str], str2: Optional[str]) -> float:
    """Gives an "intuitive" edit distance between two strings. This is
    an edit distance, normalized by the string length, with a number of
    tweaks that reflect intuition about text.
@@ -263,10 +318,10 @@ def string_dist(str1, str2):
    # example, "the something" should be considered equal to
    # "something, the".
    for word in SD_END_WORDS:
-        if str1.endswith(', %s' % word):
-            str1 = '{} {}'.format(word, str1[:-len(word) - 2])
-        if str2.endswith(', %s' % word):
-            str2 = '{} {}'.format(word, str2[:-len(word) - 2])
+        if str1.endswith(", %s" % word):
+            str1 = "{} {}".format(word, str1[: -len(word) - 2])
+        if str2.endswith(", %s" % word):
+            str2 = "{} {}".format(word, str2[: -len(word) - 2])

    # Perform a couple of basic normalizing substitutions.
    for pat, repl in SD_REPLACE:
@@ -281,8 +336,8 @@ def string_dist(str1, str2):
    penalty = 0.0
    for pat, weight in SD_PATTERNS:
        # Get strings that drop the pattern.
-        case_str1 = re.sub(pat, '', str1)
-        case_str2 = re.sub(pat, '', str2)
+        case_str1 = re.sub(pat, "", str1)
+        case_str2 = re.sub(pat, "", str2)

        if case_str1 != str1 or case_str2 != str2:
            # If the pattern was present (i.e., it is deleted in the
@@ -304,23 +359,6 @@ def string_dist(str1, str2):
    return base_dist + penalty


-class LazyClassProperty:
-    """A decorator implementing a read-only property that is *lazy* in
-    the sense that the getter is only invoked once. Subsequent accesses
-    through *any* instance use the cached result.
-    """
-
-    def __init__(self, getter):
-        self.getter = getter
-        self.computed = False
-
-    def __get__(self, obj, owner):
-        if not self.computed:
-            self.value = self.getter(owner)
-            self.computed = True
-        return self.value
-
-
@total_ordering
 class Distance:
    """Keeps track of multiple distance penalties. Provides a single
@@ -330,12 +368,12 @@ class Distance:

    def __init__(self):
        self._penalties = {}
+        self.tracks: Dict[TrackInfo, Distance] = {}

-    @LazyClassProperty
-    def _weights(cls):  # noqa: N805
-        """A dictionary from keys to floating-point weights.
-        """
-        weights_view = config['match']['distance_weights']
+    @cached_classproperty
+    def _weights(cls) -> Dict[str, float]:  # noqa: N805
+        """A dictionary from keys to floating-point weights."""
+        weights_view = config["match"]["distance_weights"]
        weights = {}
        for key in weights_view.keys():
            weights[key] = weights_view[key].as_number()
@@ -344,7 +382,7 @@ class Distance:
    # Access the components and their aggregates.

    @property
-    def distance(self):
+    def distance(self) -> float:
        """Return a weighted and normalized distance across all
        penalties.
        """
@@ -354,24 +392,22 @@ class Distance:
        return 0.0

    @property
-    def max_distance(self):
-        """Return the maximum distance penalty (normalization factor).
-        """
+    def max_distance(self) -> float:
+        """Return the maximum distance penalty (normalization factor)."""
        dist_max = 0.0
        for key, penalty in self._penalties.items():
            dist_max += len(penalty) * self._weights[key]
        return dist_max

    @property
-    def raw_distance(self):
-        """Return the raw (denormalized) distance.
-        """
+    def raw_distance(self) -> float:
+        """Return the raw (denormalized) distance."""
        dist_raw = 0.0
        for key, penalty in self._penalties.items():
            dist_raw += sum(penalty) * self._weights[key]
        return dist_raw

-    def items(self):
+    def items(self) -> List[Tuple[str, float]]:
        """Return a list of (key, dist) pairs, with `dist` being the
        weighted distance, sorted from highest to lowest. Does not
        include penalties with a zero value.
@@ -385,87 +421,88 @@ class Distance:
        # ascending order (for keys, when the penalty is equal) and
        # still get the items with the biggest distance first.
        return sorted(
-            list_,
-            key=lambda key_and_dist: (-key_and_dist[1], key_and_dist[0])
+            list_, key=lambda key_and_dist: (-key_and_dist[1], key_and_dist[0])
        )

-    def __hash__(self):
+    def __hash__(self) -> int:
        return id(self)

-    def __eq__(self, other):
+    def __eq__(self, other) -> bool:
        return self.distance == other

    # Behave like a float.

-    def __lt__(self, other):
+    def __lt__(self, other) -> bool:
        return self.distance < other

-    def __float__(self):
+    def __float__(self) -> float:
        return self.distance

-    def __sub__(self, other):
+    def __sub__(self, other) -> float:
        return self.distance - other

-    def __rsub__(self, other):
+    def __rsub__(self, other) -> float:
        return other - self.distance

-    def __str__(self):
+    def __str__(self) -> str:
        return f"{self.distance:.2f}"

    # Behave like a dict.

-    def __getitem__(self, key):
-        """Returns the weighted distance for a named penalty.
-        """
+    def __getitem__(self, key) -> float:
+        """Returns the weighted distance for a named penalty."""
        dist = sum(self._penalties[key]) * self._weights[key]
        dist_max = self.max_distance
        if dist_max:
            return dist / dist_max
        return 0.0

-    def __iter__(self):
+    def __iter__(self) -> Iterator[Tuple[str, float]]:
        return iter(self.items())

-    def __len__(self):
+    def __len__(self) -> int:
        return len(self.items())

-    def keys(self):
+    def keys(self) -> List[str]:
        return [key for key, _ in self.items()]

-    def update(self, dist):
-        """Adds all the distance penalties from `dist`.
-        """
+    def update(self, dist: "Distance"):
+        """Adds all the distance penalties from `dist`."""
        if not isinstance(dist, Distance):
            raise ValueError(
-                '`dist` must be a Distance object, not {}'.format(type(dist))
+                "`dist` must be a Distance object, not {}".format(type(dist))
            )
        for key, penalties in dist._penalties.items():
            self._penalties.setdefault(key, []).extend(penalties)

    # Adding components.

-    def _eq(self, value1, value2):
+    def _eq(self, value1: Union[re.Pattern[str], Any], value2: Any) -> bool:
        """Returns True if `value1` is equal to `value2`. `value1` may
        be a compiled regular expression, in which case it will be
        matched against `value2`.
        """
-        if isinstance(value1, Pattern):
+        if isinstance(value1, re.Pattern):
+            value2 = cast(str, value2)
            return bool(value1.match(value2))
        return value1 == value2

-    def add(self, key, dist):
+    def add(self, key: str, dist: float):
        """Adds a distance penalty. `key` must correspond with a
        configured weight setting. `dist` must be a float between 0.0
        and 1.0, and will be added to any existing distance penalties
        for the same key.
        """
        if not 0.0 <= dist <= 1.0:
-            raise ValueError(
-                f'`dist` must be between 0.0 and 1.0, not {dist}'
-            )
+            raise ValueError(f"`dist` must be between 0.0 and 1.0, not {dist}")
        self._penalties.setdefault(key, []).append(dist)

-    def add_equality(self, key, value, options):
+    def add_equality(
+        self,
+        key: str,
+        value: Any,
+        options: Union[List[Any], Tuple[Any, ...], Any],
+    ):
        """Adds a distance penalty of 1.0 if `value` doesn't match any
        of the values in `options`. If an option is a compiled regular
        expression, it will be considered equal if it matches against
@@ -481,7 +518,7 @@ class Distance:
            dist = 1.0
        self.add(key, dist)

-    def add_expr(self, key, expr):
+    def add_expr(self, key: str, expr: bool):
        """Adds a distance penalty of 1.0 if `expr` evaluates to True,
        or 0.0.
        """
@@ -490,7 +527,7 @@ class Distance:
        else:
            self.add(key, 0.0)

-    def add_number(self, key, number1, number2):
+    def add_number(self, key: str, number1: int, number2: int):
        """Adds a distance penalty of 1.0 for each number of difference
        between `number1` and `number2`, or 0.0 when there is no
        difference. Use this when there is no upper limit on the
@@ -503,7 +540,12 @@ class Distance:
        else:
            self.add(key, 0.0)

-    def add_priority(self, key, value, options):
+    def add_priority(
+        self,
+        key: str,
+        value: Any,
+        options: Union[List[Any], Tuple[Any, ...], Any],
+    ):
        """Adds a distance penalty that corresponds to the position at
        which `value` appears in `options`. A distance penalty of 0.0
        for the first option, or 1.0 if there is no matching option. If
@@ -521,7 +563,12 @@ class Distance:
            dist = 1.0
        self.add(key, dist)

-    def add_ratio(self, key, number1, number2):
+    def add_ratio(
+        self,
+        key: str,
+        number1: Union[int, float],
+        number2: Union[int, float],
+    ):
        """Adds a distance penalty for `number1` as a ratio of `number2`.
        `number1` is bound at 0 and `number2`.
        """
@@ -532,7 +579,7 @@ class Distance:
            dist = 0.0
        self.add(key, dist)

-    def add_string(self, key, str1, str2):
+    def add_string(self, key: str, str1: Optional[str], str2: Optional[str]):
        """Adds a distance penalty based on the edit distance between
        `str1` and `str2`.
        """
@@ -542,64 +589,82 @@ class Distance:

 # Structures that compose all the information for a candidate match.

-AlbumMatch = namedtuple('AlbumMatch', ['distance', 'info', 'mapping',
-                                       'extra_items', 'extra_tracks'])
+AlbumMatch = namedtuple(
+    "AlbumMatch", ["distance", "info", "mapping", "extra_items", "extra_tracks"]
+)

-TrackMatch = namedtuple('TrackMatch', ['distance', 'info'])
+TrackMatch = namedtuple("TrackMatch", ["distance", "info"])


 # Aggregation of sources.

-def album_for_mbid(release_id):
+
+def album_for_mbid(release_id: str) -> Optional[AlbumInfo]:
    """Get an AlbumInfo object for a MusicBrainz release ID. Return None
    if the ID is not found.
    """
    try:
        album = mb.album_for_id(release_id)
        if album:
-            plugins.send('albuminfo_received', info=album)
+            plugins.send("albuminfo_received", info=album)
        return album
    except mb.MusicBrainzAPIError as exc:
        exc.log(log)
+        return None


-def track_for_mbid(recording_id):
+def track_for_mbid(recording_id: str) -> Optional[TrackInfo]:
    """Get a TrackInfo object for a MusicBrainz recording ID. Return None
    if the ID is not found.
    """
    try:
        track = mb.track_for_id(recording_id)
        if track:
-            plugins.send('trackinfo_received', info=track)
+            plugins.send("trackinfo_received", info=track)
        return track
    except mb.MusicBrainzAPIError as exc:
        exc.log(log)
+        return None


-def albums_for_id(album_id):
+def albums_for_id(album_id: str) -> Iterable[AlbumInfo]:
    """Get a list of albums for an ID."""
    a = album_for_mbid(album_id)
    if a:
        yield a
    for a in plugins.album_for_id(album_id):
        if a:
-            plugins.send('albuminfo_received', info=a)
+            plugins.send("albuminfo_received", info=a)
            yield a


-def tracks_for_id(track_id):
+def tracks_for_id(track_id: str) -> Iterable[TrackInfo]:
    """Get a list of tracks for an ID."""
    t = track_for_mbid(track_id)
    if t:
        yield t
    for t in plugins.track_for_id(track_id):
        if t:
-            plugins.send('trackinfo_received', info=t)
+            plugins.send("trackinfo_received", info=t)
            yield t


-@plugins.notify_info_yielded('albuminfo_received')
-def album_candidates(items, artist, album, va_likely, extra_tags):
+def invoke_mb(call_func: Callable, *args):
+    try:
+        return call_func(*args)
+    except mb.MusicBrainzAPIError as exc:
+        exc.log(log)
+        return ()
+
+
+@plugins.notify_info_yielded("albuminfo_received")
+def album_candidates(
+    items: List[Item],
+    artist: str,
+    album: str,
+    va_likely: bool,
+    extra_tags: Dict,
+) -> Iterable[Tuple]:
    """Search for album matches. ``items`` is a list of Item objects
    that make up the album. ``artist`` and ``album`` are the respective
    names (strings), which may be derived from the item list or may be
@@ -609,40 +674,33 @@ def album_candidates(items, artist, album, va_likely, extra_tags):
    constrain the search.
    """

+    if config["musicbrainz"]["enabled"]:
        # Base candidates if we have album and artist to match.
        if artist and album:
-        try:
-            yield from mb.match_album(artist, album, len(items),
-                                      extra_tags)
-        except mb.MusicBrainzAPIError as exc:
-            exc.log(log)
+            yield from invoke_mb(
+                mb.match_album, artist, album, len(items), extra_tags
+            )

        # Also add VA matches from MusicBrainz where appropriate.
        if va_likely and album:
-        try:
-            yield from mb.match_album(None, album, len(items),
-                                      extra_tags)
-        except mb.MusicBrainzAPIError as exc:
-            exc.log(log)
+            yield from invoke_mb(
+                mb.match_album, None, album, len(items), extra_tags
+            )

    # Candidates from plugins.
-    yield from plugins.candidates(items, artist, album, va_likely,
-                                  extra_tags)
+    yield from plugins.candidates(items, artist, album, va_likely, extra_tags)


-@plugins.notify_info_yielded('trackinfo_received')
-def item_candidates(item, artist, title):
+@plugins.notify_info_yielded("trackinfo_received")
+def item_candidates(item: Item, artist: str, title: str) -> Iterable[Tuple]:
    """Search for item matches. ``item`` is the Item to be matched.
    ``artist`` and ``title`` are strings and either reflect the item or
    are specified by the user.
    """

    # MusicBrainz candidates.
-    if artist and title:
-        try:
-            yield from mb.match_track(artist, title)
-        except mb.MusicBrainzAPIError as exc:
-            exc.log(log)
+    if config["musicbrainz"]["enabled"] and artist and title:
+        yield from invoke_mb(mb.match_track, artist, title)

    # Plugin candidates.
    yield from plugins.item_candidates(item, artist, title)
@@ -19,32 +19,53 @@ releases and tracks.

 import datetime
 import re
-from munkres import Munkres
 from collections import namedtuple
+from typing import (
+    Any,
+    Dict,
+    Iterable,
+    List,
+    Optional,
+    Sequence,
+    Tuple,
+    TypeVar,
+    Union,
+    cast,
+)

-from beets import logging
-from beets import plugins
-from beets import config
+from munkres import Munkres
+
+from beets import config, logging, plugins
+from beets.autotag import (
+    AlbumInfo,
+    AlbumMatch,
+    Distance,
+    TrackInfo,
+    TrackMatch,
+    hooks,
+)
+from beets.library import Item
 from beets.util import plurality
-from beets.autotag import hooks
 from beets.util.enumeration import OrderedEnum

 # Artist signals that indicate "various artists". These are used at the
 # album level to determine whether a given release is likely a VA
 # release and also on the track level to to remove the penalty for
 # differing artists.
-VA_ARTISTS = ('', 'various artists', 'various', 'va', 'unknown')
+VA_ARTISTS = ("", "various artists", "various", "va", "unknown")

 # Global logger.
-log = logging.getLogger('beets')
+log = logging.getLogger("beets")


 # Recommendation enumeration.

+
 class Recommendation(OrderedEnum):
    """Indicates a qualitative suggestion to the user about what should
    be done with a given match.
    """
+
    none = 0
    low = 1
    medium = 2
@@ -55,12 +76,15 @@ class Recommendation(OrderedEnum):
 # consists of a list of possible candidates (i.e., AlbumInfo or TrackInfo
 # objects) and a recommendation value.

-Proposal = namedtuple('Proposal', ('candidates', 'recommendation'))
+Proposal = namedtuple("Proposal", ("candidates", "recommendation"))


 # Primary matching functionality.

-def current_metadata(items):
+
+def current_metadata(
+    items: Iterable[Item],
+) -> Tuple[Dict[str, Any], Dict[str, Any]]:
    """Extract the likely current metadata for an album given a list of its
    items. Return two dictionaries:
     - The most common value for each field.
@@ -70,22 +94,36 @@ def current_metadata(items):

    likelies = {}
    consensus = {}
-    fields = ['artist', 'album', 'albumartist', 'year', 'disctotal',
-              'mb_albumid', 'label', 'catalognum', 'country', 'media',
-              'albumdisambig']
+    fields = [
+        "artist",
+        "album",
+        "albumartist",
+        "year",
+        "disctotal",
+        "mb_albumid",
+        "label",
+        "barcode",
+        "catalognum",
+        "country",
+        "media",
+        "albumdisambig",
+    ]
    for field in fields:
        values = [item[field] for item in items if item]
        likelies[field], freq = plurality(values)
-        consensus[field] = (freq == len(values))
+        consensus[field] = freq == len(values)

    # If there's an album artist consensus, use this for the artist.
-    if consensus['albumartist'] and likelies['albumartist']:
-        likelies['artist'] = likelies['albumartist']
+    if consensus["albumartist"] and likelies["albumartist"]:
+        likelies["artist"] = likelies["albumartist"]

    return likelies, consensus


-def assign_items(items, tracks):
+def assign_items(
+    items: Sequence[Item],
+    tracks: Sequence[TrackInfo],
+) -> Tuple[Dict[Item, TrackInfo], List[Item], List[TrackInfo]]:
    """Given a list of Items and a list of TrackInfo objects, find the
    best mapping between them. Returns a mapping from Items to TrackInfo
    objects, a set of extra Items, and a set of extra TrackInfo
@@ -93,17 +131,17 @@ def assign_items(items, tracks):
    of objects of the two types.
    """
    # Construct the cost matrix.
-    costs = []
+    costs: List[List[Distance]] = []
    for item in items:
        row = []
-        for i, track in enumerate(tracks):
+        for track in tracks:
            row.append(track_distance(item, track))
        costs.append(row)

    # Find a minimum-cost bipartite matching.
-    log.debug('Computing track assignment...')
+    log.debug("Computing track assignment...")
    matching = Munkres().compute(costs)
-    log.debug('...done.')
+    log.debug("...done.")

    # Produce the output matching.
    mapping = {items[i]: tracks[j] for (i, j) in matching}
@@ -114,14 +152,18 @@ def assign_items(items, tracks):
    return mapping, extra_items, extra_tracks


-def track_index_changed(item, track_info):
+def track_index_changed(item: Item, track_info: TrackInfo) -> bool:
    """Returns True if the item and track info index is different. Tolerates
    per disc and per release numbering.
    """
    return item.track not in (track_info.medium_index, track_info.index)


-def track_distance(item, track_info, incl_artist=False):
+def track_distance(
+    item: Item,
+    track_info: TrackInfo,
+    incl_artist: bool = False,
+) -> Distance:
    """Determines the significance of a track metadata change. Returns a
    Distance object. `incl_artist` indicates that a distance component should
    be included for the track artist (i.e., for various-artist releases).
@@ -130,26 +172,37 @@ def track_distance(item, track_info, incl_artist=False):

    # Length.
    if track_info.length:
-        diff = abs(item.length - track_info.length) - \
-            config['match']['track_length_grace'].as_number()
-        dist.add_ratio('track_length', diff,
-                       config['match']['track_length_max'].as_number())
+        item_length = cast(float, item.length)
+        track_length_grace = cast(
+            Union[float, int],
+            config["match"]["track_length_grace"].as_number(),
+        )
+        track_length_max = cast(
+            Union[float, int],
+            config["match"]["track_length_max"].as_number(),
+        )
+
+        diff = abs(item_length - track_info.length) - track_length_grace
+        dist.add_ratio("track_length", diff, track_length_max)

    # Title.
-    dist.add_string('track_title', item.title, track_info.title)
+    dist.add_string("track_title", item.title, track_info.title)

    # Artist. Only check if there is actually an artist in the track data.
-    if incl_artist and track_info.artist and \
-            item.artist.lower() not in VA_ARTISTS:
-        dist.add_string('track_artist', item.artist, track_info.artist)
+    if (
+        incl_artist
+        and track_info.artist
+        and item.artist.lower() not in VA_ARTISTS
+    ):
+        dist.add_string("track_artist", item.artist, track_info.artist)

    # Track index.
    if track_info.index and item.track:
-        dist.add_expr('track_index', track_index_changed(item, track_info))
+        dist.add_expr("track_index", track_index_changed(item, track_info))

    # Track ID.
    if item.mb_trackid:
-        dist.add_expr('track_id', item.mb_trackid != track_info.track_id)
+        dist.add_expr("track_id", item.mb_trackid != track_info.track_id)

    # Plugins.
    dist.update(plugins.track_distance(item, track_info))
@@ -157,7 +210,11 @@ def track_distance(item, track_info, incl_artist=False):
    return dist


-def distance(items, album_info, mapping):
+def distance(
+    items: Sequence[Item],
+    album_info: AlbumInfo,
+    mapping: Dict[Item, TrackInfo],
+) -> Distance:
    """Determines how "significant" an album metadata change would be.
    Returns a Distance object. `album_info` is an AlbumInfo object
    reflecting the album to be compared. `items` is a sequence of all
@@ -172,90 +229,96 @@ def distance(items, album_info, mapping):

    # Artist, if not various.
    if not album_info.va:
-        dist.add_string('artist', likelies['artist'], album_info.artist)
+        dist.add_string("artist", likelies["artist"], album_info.artist)

    # Album.
-    dist.add_string('album', likelies['album'], album_info.album)
+    dist.add_string("album", likelies["album"], album_info.album)

    # Current or preferred media.
    if album_info.media:
        # Preferred media options.
-        patterns = config['match']['preferred']['media'].as_str_seq()
-        options = [re.compile(r'(\d+x)?(%s)' % pat, re.I) for pat in patterns]
+        patterns = config["match"]["preferred"]["media"].as_str_seq()
+        patterns = cast(Sequence[str], patterns)
+        options = [re.compile(r"(\d+x)?(%s)" % pat, re.I) for pat in patterns]
        if options:
-            dist.add_priority('media', album_info.media, options)
+            dist.add_priority("media", album_info.media, options)
        # Current media.
-        elif likelies['media']:
-            dist.add_equality('media', album_info.media, likelies['media'])
+        elif likelies["media"]:
+            dist.add_equality("media", album_info.media, likelies["media"])

    # Mediums.
-    if likelies['disctotal'] and album_info.mediums:
-        dist.add_number('mediums', likelies['disctotal'], album_info.mediums)
+    if likelies["disctotal"] and album_info.mediums:
+        dist.add_number("mediums", likelies["disctotal"], album_info.mediums)

    # Prefer earliest release.
-    if album_info.year and config['match']['preferred']['original_year']:
+    if album_info.year and config["match"]["preferred"]["original_year"]:
        # Assume 1889 (earliest first gramophone discs) if we don't know the
        # original year.
        original = album_info.original_year or 1889
        diff = abs(album_info.year - original)
        diff_max = abs(datetime.date.today().year - original)
-        dist.add_ratio('year', diff, diff_max)
+        dist.add_ratio("year", diff, diff_max)
    # Year.
-    elif likelies['year'] and album_info.year:
-        if likelies['year'] in (album_info.year, album_info.original_year):
+    elif likelies["year"] and album_info.year:
+        if likelies["year"] in (album_info.year, album_info.original_year):
            # No penalty for matching release or original year.
-            dist.add('year', 0.0)
+            dist.add("year", 0.0)
        elif album_info.original_year:
            # Prefer matchest closest to the release year.
-            diff = abs(likelies['year'] - album_info.year)
-            diff_max = abs(datetime.date.today().year -
-                           album_info.original_year)
-            dist.add_ratio('year', diff, diff_max)
+            diff = abs(likelies["year"] - album_info.year)
+            diff_max = abs(
+                datetime.date.today().year - album_info.original_year
+            )
+            dist.add_ratio("year", diff, diff_max)
        else:
            # Full penalty when there is no original year.
-            dist.add('year', 1.0)
+            dist.add("year", 1.0)

    # Preferred countries.
-    patterns = config['match']['preferred']['countries'].as_str_seq()
+    patterns = config["match"]["preferred"]["countries"].as_str_seq()
+    patterns = cast(Sequence[str], patterns)
    options = [re.compile(pat, re.I) for pat in patterns]
    if album_info.country and options:
-        dist.add_priority('country', album_info.country, options)
+        dist.add_priority("country", album_info.country, options)
    # Country.
-    elif likelies['country'] and album_info.country:
-        dist.add_string('country', likelies['country'], album_info.country)
+    elif likelies["country"] and album_info.country:
+        dist.add_string("country", likelies["country"], album_info.country)

    # Label.
-    if likelies['label'] and album_info.label:
-        dist.add_string('label', likelies['label'], album_info.label)
+    if likelies["label"] and album_info.label:
+        dist.add_string("label", likelies["label"], album_info.label)

    # Catalog number.
-    if likelies['catalognum'] and album_info.catalognum:
-        dist.add_string('catalognum', likelies['catalognum'],
-                        album_info.catalognum)
+    if likelies["catalognum"] and album_info.catalognum:
+        dist.add_string(
+            "catalognum", likelies["catalognum"], album_info.catalognum
+        )

    # Disambiguation.
-    if likelies['albumdisambig'] and album_info.albumdisambig:
-        dist.add_string('albumdisambig', likelies['albumdisambig'],
-                        album_info.albumdisambig)
+    if likelies["albumdisambig"] and album_info.albumdisambig:
+        dist.add_string(
+            "albumdisambig", likelies["albumdisambig"], album_info.albumdisambig
+        )

    # Album ID.
-    if likelies['mb_albumid']:
-        dist.add_equality('album_id', likelies['mb_albumid'],
-                          album_info.album_id)
+    if likelies["mb_albumid"]:
+        dist.add_equality(
+            "album_id", likelies["mb_albumid"], album_info.album_id
+        )

    # Tracks.
    dist.tracks = {}
    for item, track in mapping.items():
        dist.tracks[track] = track_distance(item, track, album_info.va)
-        dist.add('tracks', dist.tracks[track].distance)
+        dist.add("tracks", dist.tracks[track].distance)

    # Missing tracks.
-    for i in range(len(album_info.tracks) - len(mapping)):
-        dist.add('missing_tracks', 1.0)
+    for _ in range(len(album_info.tracks) - len(mapping)):
+        dist.add("missing_tracks", 1.0)

    # Unmatched tracks.
-    for i in range(len(items) - len(mapping)):
-        dist.add('unmatched_tracks', 1.0)
+    for _ in range(len(items) - len(mapping)):
+        dist.add("unmatched_tracks", 1.0)

    # Plugins.
    dist.update(plugins.album_distance(items, album_info, mapping))
@@ -263,7 +326,7 @@ def distance(items, album_info, mapping):
    return dist


-def match_by_id(items):
+def match_by_id(items: Iterable[Item]):
    """If the items are tagged with a MusicBrainz album ID, returns an
    AlbumInfo object for the corresponding album. Otherwise, returns
    None.
@@ -274,20 +337,22 @@ def match_by_id(items):
    try:
        first = next(albumids)
    except StopIteration:
-        log.debug('No album ID found.')
+        log.debug("No album ID found.")
        return None

    # Is there a consensus on the MB album ID?
    for other in albumids:
        if other != first:
-            log.debug('No album ID consensus.')
+            log.debug("No album ID consensus.")
            return None
    # If all album IDs are equal, look up the album.
-    log.debug('Searching for discovered album ID: {0}', first)
+    log.debug("Searching for discovered album ID: {0}", first)
    return hooks.album_for_mbid(first)


-def _recommendation(results):
+def _recommendation(
+    results: Sequence[Union[AlbumMatch, TrackMatch]],
+) -> Recommendation:
    """Given a sorted list of AlbumMatch or TrackMatch objects, return a
    recommendation based on the results' distances.

@@ -301,17 +366,19 @@ def _recommendation(results):

    # Basic distance thresholding.
    min_dist = results[0].distance
-    if min_dist < config['match']['strong_rec_thresh'].as_number():
+    if min_dist < config["match"]["strong_rec_thresh"].as_number():
        # Strong recommendation level.
        rec = Recommendation.strong
-    elif min_dist <= config['match']['medium_rec_thresh'].as_number():
+    elif min_dist <= config["match"]["medium_rec_thresh"].as_number():
        # Medium recommendation level.
        rec = Recommendation.medium
    elif len(results) == 1:
        # Only a single candidate.
        rec = Recommendation.low
-    elif results[1].distance - min_dist >= \
-            config['match']['rec_gap_thresh'].as_number():
+    elif (
+        results[1].distance - min_dist
+        >= config["match"]["rec_gap_thresh"].as_number()
+    ):
        # Gap between first two candidates is large.
        rec = Recommendation.low
    else:
@@ -324,48 +391,60 @@ def _recommendation(results):
    if isinstance(results[0], hooks.AlbumMatch):
        for track_dist in min_dist.tracks.values():
            keys.update(list(track_dist.keys()))
-    max_rec_view = config['match']['max_rec']
+    max_rec_view = config["match"]["max_rec"]
    for key in keys:
        if key in list(max_rec_view.keys()):
-            max_rec = max_rec_view[key].as_choice({
-                'strong': Recommendation.strong,
-                'medium': Recommendation.medium,
-                'low': Recommendation.low,
-                'none': Recommendation.none,
-            })
+            max_rec = max_rec_view[key].as_choice(
+                {
+                    "strong": Recommendation.strong,
+                    "medium": Recommendation.medium,
+                    "low": Recommendation.low,
+                    "none": Recommendation.none,
+                }
+            )
            rec = min(rec, max_rec)

    return rec


-def _sort_candidates(candidates):
+AnyMatch = TypeVar("AnyMatch", TrackMatch, AlbumMatch)
+
+
+def _sort_candidates(candidates: Iterable[AnyMatch]) -> Sequence[AnyMatch]:
    """Sort candidates by distance."""
    return sorted(candidates, key=lambda match: match.distance)


-def _add_candidate(items, results, info):
+def _add_candidate(
+    items: Sequence[Item],
+    results: Dict[Any, AlbumMatch],
+    info: AlbumInfo,
+):
    """Given a candidate AlbumInfo object, attempt to add the candidate
    to the output dictionary of AlbumMatch objects. This involves
    checking the track count, ordering the items, checking for
    duplicates, and calculating the distance.
    """
-    log.debug('Candidate: {0} - {1} ({2})',
-              info.artist, info.album, info.album_id)
+    log.debug(
+        "Candidate: {0} - {1} ({2})", info.artist, info.album, info.album_id
+    )

    # Discard albums with zero tracks.
    if not info.tracks:
-        log.debug('No tracks.')
+        log.debug("No tracks.")
        return

-    # Don't duplicate.
-    if info.album_id in results:
-        log.debug('Duplicate.')
+    # Prevent duplicates.
+    if info.album_id and info.album_id in results:
+        log.debug("Duplicate.")
        return

    # Discard matches without required tags.
-    for req_tag in config['match']['required'].as_str_seq():
+    for req_tag in cast(
+        Sequence[str], config["match"]["required"].as_str_seq()
+    ):
        if getattr(info, req_tag) is None:
-            log.debug('Ignored. Missing required tag: {0}', req_tag)
+            log.debug("Ignored. Missing required tag: {0}", req_tag)
            return

    # Find mapping between the items and the track info.
@@ -376,18 +455,24 @@ def _add_candidate(items, results, info):

    # Skip matches with ignored penalties.
    penalties = [key for key, _ in dist]
-    for penalty in config['match']['ignored'].as_str_seq():
+    ignored = cast(Sequence[str], config["match"]["ignored"].as_str_seq())
+    for penalty in ignored:
        if penalty in penalties:
-            log.debug('Ignored. Penalty: {0}', penalty)
+            log.debug("Ignored. Penalty: {0}", penalty)
            return

-    log.debug('Success. Distance: {0}', dist)
-    results[info.album_id] = hooks.AlbumMatch(dist, info, mapping,
-                                              extra_items, extra_tracks)
+    log.debug("Success. Distance: {0}", dist)
+    results[info.album_id] = hooks.AlbumMatch(
+        dist, info, mapping, extra_items, extra_tracks
+    )


-def tag_album(items, search_artist=None, search_album=None,
-              search_ids=[]):
+def tag_album(
+    items,
+    search_artist: Optional[str] = None,
+    search_album: Optional[str] = None,
+    search_ids: List[str] = [],
+) -> Tuple[str, str, Proposal]:
    """Return a tuple of the current artist name, the current album
    name, and a `Proposal` containing `AlbumMatch` candidates.

@@ -407,20 +492,19 @@ def tag_album(items, search_artist=None, search_album=None,
    """
    # Get current metadata.
    likelies, consensus = current_metadata(items)
-    cur_artist = likelies['artist']
-    cur_album = likelies['album']
-    log.debug('Tagging {0} - {1}', cur_artist, cur_album)
+    cur_artist = cast(str, likelies["artist"])
+    cur_album = cast(str, likelies["album"])
+    log.debug("Tagging {0} - {1}", cur_artist, cur_album)

-    # The output result (distance, AlbumInfo) tuples (keyed by MB album
-    # ID).
-    candidates = {}
+    # The output result, keys are the MB album ID.
+    candidates: Dict[Any, AlbumMatch] = {}

    # Search by explicit ID.
    if search_ids:
        for search_id in search_ids:
-            log.debug('Searching for album ID: {0}', search_id)
-            for id_candidate in hooks.albums_for_id(search_id):
-                _add_candidate(items, candidates, id_candidate)
+            log.debug("Searching for album ID: {0}", search_id)
+            for album_info_for_id in hooks.albums_for_id(search_id):
+                _add_candidate(items, candidates, album_info_for_id)

    # Use existing metadata or text search.
    else:
@@ -429,51 +513,58 @@ def tag_album(items, search_artist=None, search_album=None,
        if id_info:
            _add_candidate(items, candidates, id_info)
            rec = _recommendation(list(candidates.values()))
-            log.debug('Album ID match recommendation is {0}', rec)
-            if candidates and not config['import']['timid']:
+            log.debug("Album ID match recommendation is {0}", rec)
+            if candidates and not config["import"]["timid"]:
                # If we have a very good MBID match, return immediately.
                # Otherwise, this match will compete against metadata-based
                # matches.
                if rec == Recommendation.strong:
-                    log.debug('ID match.')
-                    return cur_artist, cur_album, \
-                        Proposal(list(candidates.values()), rec)
+                    log.debug("ID match.")
+                    return (
+                        cur_artist,
+                        cur_album,
+                        Proposal(list(candidates.values()), rec),
+                    )

        # Search terms.
        if not (search_artist and search_album):
            # No explicit search terms -- use current metadata.
            search_artist, search_album = cur_artist, cur_album
-        log.debug('Search terms: {0} - {1}', search_artist, search_album)
+        log.debug("Search terms: {0} - {1}", search_artist, search_album)

        extra_tags = None
-        if config['musicbrainz']['extra_tags']:
-            tag_list = config['musicbrainz']['extra_tags'].get()
+        if config["musicbrainz"]["extra_tags"]:
+            tag_list = config["musicbrainz"]["extra_tags"].get()
            extra_tags = {k: v for (k, v) in likelies.items() if k in tag_list}
-            log.debug('Additional search terms: {0}', extra_tags)
+            log.debug("Additional search terms: {0}", extra_tags)

        # Is this album likely to be a "various artist" release?
-        va_likely = ((not consensus['artist']) or
-                     (search_artist.lower() in VA_ARTISTS) or
-                     any(item.comp for item in items))
-        log.debug('Album might be VA: {0}', va_likely)
+        va_likely = (
+            (not consensus["artist"])
+            or (search_artist.lower() in VA_ARTISTS)
+            or any(item.comp for item in items)
+        )
+        log.debug("Album might be VA: {0}", va_likely)

        # Get the results from the data sources.
-        for matched_candidate in hooks.album_candidates(items,
-                                                        search_artist,
-                                                        search_album,
-                                                        va_likely,
-                                                        extra_tags):
+        for matched_candidate in hooks.album_candidates(
+            items, search_artist, search_album, va_likely, extra_tags
+        ):
            _add_candidate(items, candidates, matched_candidate)

-    log.debug('Evaluating {0} candidates.', len(candidates))
+    log.debug("Evaluating {0} candidates.", len(candidates))
    # Sort and get the recommendation.
-    candidates = _sort_candidates(candidates.values())
-    rec = _recommendation(candidates)
-    return cur_artist, cur_album, Proposal(candidates, rec)
+    candidates_sorted = _sort_candidates(candidates.values())
+    rec = _recommendation(candidates_sorted)
+    return cur_artist, cur_album, Proposal(candidates_sorted, rec)


-def tag_item(item, search_artist=None, search_title=None,
-             search_ids=[]):
+def tag_item(
+    item,
+    search_artist: Optional[str] = None,
+    search_title: Optional[str] = None,
+    search_ids: Optional[List[str]] = None,
+) -> Proposal:
    """Find metadata for a single track. Return a `Proposal` consisting
    of `TrackMatch` objects.

@@ -485,26 +576,31 @@ def tag_item(item, search_artist=None, search_title=None,
    # Holds candidates found so far: keys are MBIDs; values are
    # (distance, TrackInfo) pairs.
    candidates = {}
+    rec: Optional[Recommendation] = None

    # First, try matching by MusicBrainz ID.
    trackids = search_ids or [t for t in [item.mb_trackid] if t]
    if trackids:
        for trackid in trackids:
-            log.debug('Searching for track ID: {0}', trackid)
+            log.debug("Searching for track ID: {0}", trackid)
            for track_info in hooks.tracks_for_id(trackid):
                dist = track_distance(item, track_info, incl_artist=True)
-                candidates[track_info.track_id] = \
-                    hooks.TrackMatch(dist, track_info)
+                candidates[track_info.track_id] = hooks.TrackMatch(
+                    dist, track_info
+                )
                # If this is a good match, then don't keep searching.
                rec = _recommendation(_sort_candidates(candidates.values()))
-                if rec == Recommendation.strong and \
-                        not config['import']['timid']:
-                    log.debug('Track ID match.')
+                if (
+                    rec == Recommendation.strong
+                    and not config["import"]["timid"]
+                ):
+                    log.debug("Track ID match.")
                    return Proposal(_sort_candidates(candidates.values()), rec)

    # If we're searching by ID, don't proceed.
    if search_ids:
        if candidates:
+            assert rec is not None
            return Proposal(_sort_candidates(candidates.values()), rec)
        else:
            return Proposal([], Recommendation.none)
@@ -512,7 +608,7 @@ def tag_item(item, search_artist=None, search_title=None,
    # Search terms.
    if not (search_artist and search_title):
        search_artist, search_title = item.artist, item.title
-    log.debug('Item search terms: {0} - {1}', search_artist, search_title)
+    log.debug("Item search terms: {0} - {1}", search_artist, search_title)

    # Get and evaluate candidate metadata.
    for track_info in hooks.item_candidates(item, search_artist, search_title):
@@ -520,7 +616,7 @@ def tag_item(item, search_artist=None, search_title=None,
        candidates[track_info.track_id] = hooks.TrackMatch(dist, track_info)

    # Sort by distance and return with recommendation.
-    log.debug('Found {0} candidates.', len(candidates))
-    candidates = _sort_candidates(candidates.values())
-    rec = _recommendation(candidates)
-    return Proposal(candidates, rec)
+    log.debug("Found {0} candidates.", len(candidates))
+    candidates_sorted = _sort_candidates(candidates.values())
+    rec = _recommendation(candidates_sorted)
+    return Proposal(candidates_sorted, rec)
@@ -1,10 +1,34 @@
+# --------------- Main ---------------
+
 library: library.db
 directory: ~/Music
+statefile: state.pickle
+
+# --------------- Plugins ---------------
+
+plugins: []
+pluginpath: []
+
+# --------------- Import ---------------
+
+clutter: ["Thumbs.DB", ".DS_Store"]
+ignore: [".*", "*~", "System Volume Information", "lost+found"]
+ignore_hidden: yes

 import:
+    # common options
    write: yes
    copy: yes
    move: no
+    timid: no
+    quiet: no
+    log:
+    # other options
+    default_action: apply
+    languages: []
+    quiet_fallback: skip
+    none_rec_action: ask
+    # rare options
    link: no
    hardlink: no
    reflink: no
@@ -13,76 +37,117 @@ import:
    incremental: no
    incremental_skip_later: no
    from_scratch: no
-    quiet_fallback: skip
-    none_rec_action: ask
-    timid: no
-    log:
    autotag: yes
-    quiet: no
    singletons: no
-    default_action: apply
-    languages: []
    detail: no
    flat: no
    group_albums: no
    pretend: false
    search_ids: []
+    duplicate_keys:
+        album: albumartist album
+        item: artist title
    duplicate_action: ask
+    duplicate_verbose_prompt: no
    bell: no
    set_fields: {}
+    ignored_alias_types: []
+    singleton_album_disambig: yes

-clutter: ["Thumbs.DB", ".DS_Store"]
-ignore: [".*", "*~", "System Volume Information", "lost+found"]
-ignore_hidden: yes
+# --------------- Paths ---------------

-replace:
-    '[\\/]': _
-    '^\.': _
-    '[\x00-\x1f]': _
-    '[<>:"\?\*\|]': _
-    '\.$': _
-    '\s+$': ''
-    '^\s+': ''
-    '^-': _
 path_sep_replace: _
 drive_sep_replace: _
 asciify_paths: false
 art_filename: cover
 max_filename_length: 0
+replace:
+    # Replace bad characters with _
+    # prohibited in many filesystem paths
+    '[<>:\?\*\|]': _
+    # double quotation mark "
+    '\"': _
+    # path separators: \ or /
+    '[\\/]': _
+    # starting and closing periods
+    '^\.': _
+    '\.$': _
+    # control characters
+    '[\x00-\x1f]': _
+    # dash at the start of a filename (causes command line ambiguity)
+    '^-': _
+    # Replace bad characters with nothing
+    # starting and closing whitespace
+    '\s+$': ''
+    '^\s+': ''

 aunique:
    keys: albumartist album
    disambiguators: albumtype year label catalognum albumdisambig releasegroupdisambig
    bracket: '[]'

-overwrite_null:
-  album: []
-  track: []
+sunique:
+    keys: artist title
+    disambiguators: year trackdisambig
+    bracket: '[]'
+
+# --------------- Tagging ---------------

-plugins: []
-pluginpath: []
-threaded: yes
-timeout: 5.0
 per_disc_numbering: no
-verbose: 0
-terminal_encoding:
 original_date: no
 artist_credit: no
 id3v23: no
 va_name: "Various Artists"
+paths:
+    default: $albumartist/$album%aunique{}/$track $title
+    singleton: Non-Album/$artist/$title
+    comp: Compilations/$album%aunique{}/$track $title
+
+# --------------- Performance ---------------
+
+threaded: yes
+timeout: 5.0
+
+# --------------- UI ---------------
+
+verbose: 0
+terminal_encoding:

 ui:
    terminal_width: 80
    length_diff_thresh: 10.0
    color: yes
    colors:
-        text_success: green
-        text_warning: yellow
-        text_error: red
-        text_highlight: red
-        text_highlight_minor: lightgray
-        action_default: turquoise
-        action: blue
+        text_success: ['bold', 'green']
+        text_warning: ['bold', 'yellow']
+        text_error: ['bold', 'red']
+        text_highlight: ['bold', 'red']
+        text_highlight_minor: ['white']
+        action_default: ['bold', 'cyan']
+        action: ['bold', 'cyan']
+        # New Colors
+        text: ['normal']
+        text_faint: ['faint']
+        import_path: ['bold', 'blue']
+        import_path_items: ['bold', 'blue']
+        added:   ['green']
+        removed: ['red']
+        changed: ['yellow']
+        added_highlight:   ['bold', 'green']
+        removed_highlight: ['bold', 'red']
+        changed_highlight: ['bold', 'yellow']
+        text_diff_added:   ['bold', 'red']
+        text_diff_removed: ['bold', 'red']
+        text_diff_changed: ['bold', 'red']
+        action_description: ['white']
+    import:
+        indentation:
+            match_header: 2
+            match_details: 2
+            match_tracklist: 5
+        layout: column
+
+# --------------- Search ---------------

 format_item: $artist - $album - $title
 format_album: $albumartist - $album
@@ -93,14 +158,13 @@ sort_album: albumartist+ album+
 sort_item: artist+ album+ disc+ track+
 sort_case_insensitive: yes

-paths:
-    default: $albumartist/$album%aunique{}/$track $title
-    singleton: Non-Album/$artist/$title
-    comp: Compilations/$album%aunique{}/$track $title
-
-statefile: state.pickle
+# --------------- Autotagger ---------------

+overwrite_null:
+  album: []
+  track: []
 musicbrainz:
+    enabled: yes
    host: musicbrainz.org
    https: no
    ratelimit: 1
@@ -108,6 +172,13 @@ musicbrainz:
    searchlimit: 5
    extra_tags: []
    genres: no
+    external_ids:
+        discogs: no
+        bandcamp: no
+        spotify: no
+        deezer: no
+        beatport: no
+        tidal: no

 match:
    strong_rec_thresh: 0.04
@@ -147,3 +218,5 @@ match:
    ignore_video_tracks: yes
    track_length_grace: 10
    track_length_max: 30
+    album_disambig_fields: data_source media year country label catalognum albumdisambig
+    singleton_disambig_fields: data_source index track_alt album
@@ -16,12 +16,20 @@
 Library.
 """

-from .db import Model, Database
-from .query import Query, FieldQuery, MatchQuery, AndQuery, OrQuery
+from .db import Database, Model, Results
+from .query import (
+    AndQuery,
+    FieldQuery,
+    InvalidQueryError,
+    MatchQuery,
+    OrQuery,
+    Query,
+)
+from .queryparse import (
+    parse_sorted_query,
+    query_from_strings,
+    sort_from_strings,
+)
 from .types import Type
-from .queryparse import query_from_strings
-from .queryparse import sort_from_strings
-from .queryparse import parse_sorted_query
-from .query import InvalidQueryError

 # flake8: noqa
@@ -12,30 +12,33 @@
 # The above copyright notice and this permission notice shall be
 # included in all copies or substantial portions of the Software.

-"""Parsing of strings into DBCore queries.
-"""
+"""Parsing of strings into DBCore queries."""

-import re
 import itertools
-from . import query
+import re
+from typing import Collection, Dict, List, Optional, Sequence, Tuple, Type
+
+from . import Model, query
+from .query import Sort

 PARSE_QUERY_PART_REGEX = re.compile(
    # Non-capturing optional segment for the keyword.
-    r'(-|\^)?'   # Negation prefixes.
-
-    r'(?:'
-    r'(\S+?)'    # The field key.
-    r'(?<!\\):'  # Unescaped :
-    r')?'
-
-    r'(.*)',         # The term itself.
-
-    re.I  # Case-insensitive.
+    r"(-|\^)?"  # Negation prefixes.
+    r"(?:"
+    r"(\S+?)"  # The field key.
+    r"(?<!\\):"  # Unescaped :
+    r")?"
+    r"(.*)",  # The term itself.
+    re.I,  # Case-insensitive.
 )


-def parse_query_part(part, query_classes={}, prefixes={},
-                     default_class=query.SubstringQuery):
+def parse_query_part(
+    part: str,
+    query_classes: Dict[str, Type[query.FieldQuery]] = {},
+    prefixes: Dict = {},
+    default_class: Type[query.SubstringQuery] = query.SubstringQuery,
+) -> Tuple[Optional[str], str, Type[query.FieldQuery], bool]:
    """Parse a single *query part*, which is a chunk of a complete query
    string representing a single criterion.

@@ -86,7 +89,7 @@ def parse_query_part(part, query_classes={}, prefixes={},
    assert match  # Regex should always match
    negate = bool(match.group(1))
    key = match.group(2)
-    term = match.group(3).replace('\\:', ':')
+    term = match.group(3).replace("\\:", ":")

    # Check whether there's a prefix in the query and use the
    # corresponding query type.
@@ -100,7 +103,11 @@ def parse_query_part(part, query_classes={}, prefixes={},
    return key, term, query_class, negate


-def construct_query_part(model_cls, prefixes, query_part):
+def construct_query_part(
+    model_cls: Type[Model],
+    prefixes: Dict,
+    query_part: str,
+) -> query.Query:
    """Parse a *query part* string and return a :class:`Query` object.

    :param model_cls: The :class:`Model` class that this is a query for.
@@ -116,40 +123,44 @@ def construct_query_part(model_cls, prefixes, query_part):
    if not query_part:
        return query.TrueQuery()

+    out_query: query.Query
+
    # Use `model_cls` to build up a map from field (or query) names to
    # `Query` classes.
-    query_classes = {}
-    for k, t in itertools.chain(model_cls._fields.items(),
-                                model_cls._types.items()):
+    query_classes: Dict[str, Type[query.FieldQuery]] = {}
+    for k, t in itertools.chain(
+        model_cls._fields.items(), model_cls._types.items()
+    ):
        query_classes[k] = t.query
    query_classes.update(model_cls._queries)  # Non-field queries.

    # Parse the string.
-    key, pattern, query_class, negate = \
-        parse_query_part(query_part, query_classes, prefixes)
+    key, pattern, query_class, negate = parse_query_part(
+        query_part, query_classes, prefixes
+    )

    # If there's no key (field name) specified, this is a "match
    # anything" query.
    if key is None:
-        if issubclass(query_class, query.FieldQuery):
        # The query type matches a specific field, but none was
        # specified. So we use a version of the query that matches
        # any field.
-            out_query = query.AnyFieldQuery(pattern, model_cls._search_fields,
-                                            query_class)
-        else:
-            # Non-field query type.
-            out_query = query_class(pattern)
+        out_query = query.AnyFieldQuery(
+            pattern, model_cls._search_fields, query_class
+        )

    # Field queries get constructed according to the name of the field
    # they are querying.
-    elif issubclass(query_class, query.FieldQuery):
-        key = key.lower()
-        out_query = query_class(key.lower(), pattern, key in model_cls._fields)
-
-    # Non-field (named) query.
    else:
-        out_query = query_class(pattern)
+        field = table = key.lower()
+        if field in model_cls.shared_db_fields:
+            # This field exists in both tables, so SQLite will encounter
+            # an OperationalError if we try to query it in a join.
+            # Using an explicit table name resolves this.
+            table = f"{model_cls._table}.{field}"
+
+        field_in_db = field in model_cls.all_db_fields
+        out_query = query_class(table, pattern, field_in_db)

    # Apply negation.
    if negate:
@@ -158,7 +169,13 @@ def construct_query_part(model_cls, prefixes, query_part):
        return out_query


-def query_from_strings(query_cls, model_cls, prefixes, query_parts):
+# TYPING ERROR
+def query_from_strings(
+    query_cls: Type[query.CollectionQuery],
+    model_cls: Type[Model],
+    prefixes: Dict,
+    query_parts: Collection[str],
+) -> query.Query:
    """Creates a collection query of type `query_cls` from a list of
    strings in the format used by parse_query_part. `model_cls`
    determines how queries are constructed from strings.
@@ -171,7 +188,11 @@ def query_from_strings(query_cls, model_cls, prefixes, query_parts):
    return query_cls(subqueries)


-def construct_sort_part(model_cls, part, case_insensitive=True):
+def construct_sort_part(
+    model_cls: Type[Model],
+    part: str,
+    case_insensitive: bool = True,
+) -> Sort:
    """Create a `Sort` from a single string criterion.

    `model_cls` is the `Model` being queried. `part` is a single string
@@ -183,12 +204,13 @@ def construct_sort_part(model_cls, part, case_insensitive=True):
    field = part[:-1]
    assert field, "field is missing"
    direction = part[-1]
-    assert direction in ('+', '-'), "part must end with + or -"
-    is_ascending = direction == '+'
+    assert direction in ("+", "-"), "part must end with + or -"
+    is_ascending = direction == "+"

    if field in model_cls._sorts:
-        sort = model_cls._sorts[field](model_cls, is_ascending,
-                                       case_insensitive)
+        sort = model_cls._sorts[field](
+            model_cls, is_ascending, case_insensitive
+        )
    elif field in model_cls._fields:
        sort = query.FixedFieldSort(field, is_ascending, case_insensitive)
    else:
@@ -197,23 +219,31 @@ def construct_sort_part(model_cls, part, case_insensitive=True):
    return sort


-def sort_from_strings(model_cls, sort_parts, case_insensitive=True):
-    """Create a `Sort` from a list of sort criteria (strings).
-    """
+def sort_from_strings(
+    model_cls: Type[Model],
+    sort_parts: Sequence[str],
+    case_insensitive: bool = True,
+) -> Sort:
+    """Create a `Sort` from a list of sort criteria (strings)."""
    if not sort_parts:
-        sort = query.NullSort()
+        return query.NullSort()
    elif len(sort_parts) == 1:
-        sort = construct_sort_part(model_cls, sort_parts[0], case_insensitive)
+        return construct_sort_part(model_cls, sort_parts[0], case_insensitive)
    else:
        sort = query.MultipleSort()
        for part in sort_parts:
-            sort.add_sort(construct_sort_part(model_cls, part,
-                                              case_insensitive))
+            sort.add_sort(
+                construct_sort_part(model_cls, part, case_insensitive)
+            )
        return sort


-def parse_sorted_query(model_cls, parts, prefixes={},
-                       case_insensitive=True):
+def parse_sorted_query(
+    model_cls: Type[Model],
+    parts: List[str],
+    prefixes: Dict = {},
+    case_insensitive: bool = True,
+) -> Tuple[query.Query, Sort]:
    """Given a list of strings, create the `Query` and `Sort` that they
    represent.
    """
@@ -224,24 +254,24 @@ def parse_sorted_query(model_cls, parts, prefixes={},
    # Split up query in to comma-separated subqueries, each representing
    # an AndQuery, which need to be joined together in one OrQuery
    subquery_parts = []
-    for part in parts + [',']:
-        if part.endswith(','):
+    for part in parts + [","]:
+        if part.endswith(","):
            # Ensure we can catch "foo, bar" as well as "foo , bar"
            last_subquery_part = part[:-1]
            if last_subquery_part:
                subquery_parts.append(last_subquery_part)
            # Parse the subquery in to a single AndQuery
            # TODO: Avoid needlessly wrapping AndQueries containing 1 subquery?
-            query_parts.append(query_from_strings(
+            query_parts.append(
+                query_from_strings(
                    query.AndQuery, model_cls, prefixes, subquery_parts
-            ))
+                )
+            )
            del subquery_parts[:]
        else:
            # Sort parts (1) end in + or -, (2) don't have a field, and
            # (3) consist of more than just the + or -.
-            if part.endswith(('+', '-')) \
-                    and ':' not in part \
-                    and len(part) > 1:
+            if part.endswith(("+", "-")) and ":" not in part and len(part) > 1:
                sort_parts.append(part)
            else:
                subquery_parts.append(part)
@@ -14,28 +14,46 @@

 """Representation of type information for DBCore model fields.
 """
+import typing
+from abc import ABC
+from typing import Any, Generic, List, TypeVar, Union, cast

-from . import query
 from beets.util import str2bool

+from .query import BooleanQuery, FieldQuery, NumericQuery, SubstringQuery

-# Abstract base.

-class Type:
+class ModelType(typing.Protocol):
+    """Protocol that specifies the required constructor for model types,
+    i.e. a function that takes any argument and attempts to parse it to the
+    given type.
+    """
+
+    def __init__(self, value: Any = None): ...
+
+
+# Generic type variables, used for the value type T and null type N (if
+# nullable, else T and N are set to the same type for the concrete subclasses
+# of Type).
+N = TypeVar("N")
+T = TypeVar("T", bound=ModelType)
+
+
+class Type(ABC, Generic[T, N]):
    """An object encapsulating the type of a model field. Includes
    information about how to store, query, format, and parse a given
    field.
    """

-    sql = 'TEXT'
+    sql: str = "TEXT"
    """The SQLite column type for the value.
    """

-    query = query.SubstringQuery
+    query: typing.Type[FieldQuery] = SubstringQuery
    """The `Query` subclass to be used when querying the field.
    """

-    model_type = str
+    model_type: typing.Type[T]
    """The Python type that is used to represent the value in the model.

    The model is guaranteed to return a value of this type if the field
@@ -44,12 +62,14 @@ class Type:
    """

    @property
-    def null(self):
-        """The value to be exposed when the underlying value is None.
-        """
-        return self.model_type()
+    def null(self) -> N:
+        """The value to be exposed when the underlying value is None."""
+        # Note that this default implementation only makes sense for T = N.
+        # It would be better to implement `null()` only in subclasses, or
+        # have a field null_type similar to `model_type` and use that here.
+        return cast(N, self.model_type())

-    def format(self, value):
+    def format(self, value: Union[N, T]) -> str:
        """Given a value of this type, produce a Unicode string
        representing the value. This is used in template evaluation.
        """
@@ -57,13 +77,13 @@ class Type:
            value = self.null
        # `self.null` might be `None`
        if value is None:
-            value = ''
-        if isinstance(value, bytes):
-            value = value.decode('utf-8', 'ignore')
-
+            return ""
+        elif isinstance(value, bytes):
+            return value.decode("utf-8", "ignore")
+        else:
            return str(value)

-    def parse(self, string):
+    def parse(self, string: str) -> Union[T, N]:
        """Parse a (possibly human-written) string and return the
        indicated value of this type.
        """
@@ -72,19 +92,23 @@ class Type:
        except ValueError:
            return self.null

-    def normalize(self, value):
+    def normalize(self, value: Any) -> Union[T, N]:
        """Given a value that will be assigned into a field of this
        type, normalize the value to have the appropriate type. This
        base implementation only reinterprets `None`.
        """
+        # TYPING ERROR
        if value is None:
            return self.null
        else:
            # TODO This should eventually be replaced by
            # `self.model_type(value)`
-            return value
+            return cast(T, value)

-    def from_sql(self, sql_value):
+    def from_sql(
+        self,
+        sql_value: Union[None, int, float, str, bytes],
+    ) -> Union[T, N]:
        """Receives the value stored in the SQL backend and return the
        value to be stored in the model.

@@ -99,13 +123,13 @@ class Type:
        and the method must handle these in addition.
        """
        if isinstance(sql_value, memoryview):
-            sql_value = bytes(sql_value).decode('utf-8', 'ignore')
+            sql_value = bytes(sql_value).decode("utf-8", "ignore")
        if isinstance(sql_value, str):
            return self.parse(sql_value)
        else:
            return self.normalize(sql_value)

-    def to_sql(self, model_value):
+    def to_sql(self, model_value: Any) -> Union[None, int, float, str, bytes]:
        """Convert a value as stored in the model object to a value used
        by the database adapter.
        """
@@ -114,18 +138,23 @@ class Type:

 # Reusable types.

-class Default(Type):
-    null = None
+
+class Default(Type[str, None]):
+    model_type = str
+
+    @property
+    def null(self):
+        return None


-class Integer(Type):
-    """A basic integer type.
-    """
-    sql = 'INTEGER'
-    query = query.NumericQuery
+class BaseInteger(Type[int, N]):
+    """A basic integer type."""
+
+    sql = "INTEGER"
+    query = NumericQuery
    model_type = int

-    def normalize(self, value):
+    def normalize(self, value: Any) -> Union[int, N]:
        try:
            return self.model_type(round(float(value)))
        except ValueError:
@@ -134,91 +163,153 @@ class Integer(Type):
            return self.null


-class PaddedInt(Integer):
+class Integer(BaseInteger[int]):
+    @property
+    def null(self) -> int:
+        return 0
+
+
+class NullInteger(BaseInteger[None]):
+    @property
+    def null(self) -> None:
+        return None
+
+
+class BasePaddedInt(BaseInteger[N]):
    """An integer field that is formatted with a given number of digits,
    padded with zeroes.
    """
-    def __init__(self, digits):
+
+    def __init__(self, digits: int):
        self.digits = digits

-    def format(self, value):
-        return '{0:0{1}d}'.format(value or 0, self.digits)
+    def format(self, value: Union[int, N]) -> str:
+        return "{0:0{1}d}".format(value or 0, self.digits)


-class NullPaddedInt(PaddedInt):
-    """Same as `PaddedInt`, but does not normalize `None` to `0.0`.
-    """
-    null = None
+class PaddedInt(BasePaddedInt[int]):
+    pass
+
+
+class NullPaddedInt(BasePaddedInt[None]):
+    """Same as `PaddedInt`, but does not normalize `None` to `0`."""
+
+    @property
+    def null(self) -> None:
+        return None


 class ScaledInt(Integer):
    """An integer whose formatting operation scales the number by a
    constant and adds a suffix. Good for units with large magnitudes.
    """
-    def __init__(self, unit, suffix=''):
+
+    def __init__(self, unit: int, suffix: str = ""):
        self.unit = unit
        self.suffix = suffix

-    def format(self, value):
-        return '{}{}'.format((value or 0) // self.unit, self.suffix)
+    def format(self, value: int) -> str:
+        return "{}{}".format((value or 0) // self.unit, self.suffix)


-class Id(Integer):
+class Id(NullInteger):
    """An integer used as the row id or a foreign key in a SQLite table.
    This type is nullable: None values are not translated to zero.
    """
-    null = None

-    def __init__(self, primary=True):
+    @property
+    def null(self) -> None:
+        return None
+
+    def __init__(self, primary: bool = True):
        if primary:
-            self.sql = 'INTEGER PRIMARY KEY'
+            self.sql = "INTEGER PRIMARY KEY"


-class Float(Type):
+class BaseFloat(Type[float, N]):
    """A basic floating-point type. The `digits` parameter specifies how
    many decimal places to use in the human-readable representation.
    """
-    sql = 'REAL'
-    query = query.NumericQuery
+
+    sql = "REAL"
+    query: typing.Type[FieldQuery[Any]] = NumericQuery
    model_type = float

-    def __init__(self, digits=1):
+    def __init__(self, digits: int = 1):
        self.digits = digits

-    def format(self, value):
-        return '{0:.{1}f}'.format(value or 0, self.digits)
+    def format(self, value: Union[float, N]) -> str:
+        return "{0:.{1}f}".format(value or 0, self.digits)


-class NullFloat(Float):
-    """Same as `Float`, but does not normalize `None` to `0.0`.
-    """
-    null = None
+class Float(BaseFloat[float]):
+    """Floating-point type that normalizes `None` to `0.0`."""
+
+    @property
+    def null(self) -> float:
+        return 0.0


-class String(Type):
-    """A Unicode string type.
-    """
-    sql = 'TEXT'
-    query = query.SubstringQuery
+class NullFloat(BaseFloat[None]):
+    """Same as `Float`, but does not normalize `None` to `0.0`."""

-    def normalize(self, value):
+    @property
+    def null(self) -> None:
+        return None
+
+
+class BaseString(Type[T, N]):
+    """A Unicode string type."""
+
+    sql = "TEXT"
+    query = SubstringQuery
+
+    def normalize(self, value: Any) -> Union[T, N]:
        if value is None:
            return self.null
        else:
            return self.model_type(value)


-class Boolean(Type):
-    """A boolean type.
+class String(BaseString[str, Any]):
+    """A Unicode string type."""
+
+    model_type = str
+
+
+class DelimitedString(BaseString[List[str], List[str]]):
+    """A list of Unicode strings, represented in-database by a single string
+    containing delimiter-separated values.
    """
-    sql = 'INTEGER'
-    query = query.BooleanQuery
+
+    model_type = list
+
+    def __init__(self, delimiter: str):
+        self.delimiter = delimiter
+
+    def format(self, value: List[str]):
+        return self.delimiter.join(value)
+
+    def parse(self, string: str):
+        if not string:
+            return []
+        return string.split(self.delimiter)
+
+    def to_sql(self, model_value: List[str]):
+        return self.delimiter.join(model_value)
+
+
+class Boolean(Type):
+    """A boolean type."""
+
+    sql = "INTEGER"
+    query = BooleanQuery
    model_type = bool

-    def format(self, value):
+    def format(self, value: bool) -> str:
        return str(bool(value))

-    def parse(self, string):
+    def parse(self, string: str) -> bool:
        return str2bool(string)


@@ -231,3 +322,7 @@ FLOAT = Float()
 NULL_FLOAT = NullFloat()
 STRING = String()
 BOOLEAN = Boolean()
+SEMICOLON_SPACE_DSV = DelimitedString(delimiter="; ")
+
+# Will set the proper null char in mediafile
+MULTI_VALUE_DSV = DelimitedString(delimiter="\\␀")
@@ -12,63 +12,50 @@
 # The above copyright notice and this permission notice shall be
 # included in all copies or substantial portions of the Software.

-"""A drop-in replacement for the standard-library `logging` module that
-allows {}-style log formatting on Python 2 and 3.
+"""A drop-in replacement for the standard-library `logging` module.

-Provides everything the "logging" module does. The only difference is
-that when getLogger(name) instantiates a logger that logger uses
-{}-style formatting.
+Provides everything the "logging" module does. In addition, beets' logger
+(as obtained by `getLogger(name)`) supports thread-local levels, and messages
+use {}-style formatting and can interpolate keywords arguments to the logging
+calls (`debug`, `info`, etc).
 """


-from copy import copy
-from logging import *  # noqa
-import subprocess
+import logging
 import threading
+from copy import copy


 def logsafe(val):
-    """Coerce a potentially "problematic" value so it can be formatted
-    in a Unicode log string.
+    """Coerce `bytes` to `str` to avoid crashes solely due to logging.

-    This works around a number of pitfalls when logging objects in
-    Python 2:
-    - Logging path names, which must be byte strings, requires
-      conversion for output.
-    - Some objects, including some exceptions, will crash when you call
-      `unicode(v)` while `str(v)` works fine. CalledProcessError is an
-      example.
+    This is particularly relevant for bytestring paths. Much of our code
+    explicitly uses `displayable_path` for them, but better be safe and prevent
+    any crashes that are solely due to log formatting.
    """
-    # Already Unicode.
-    if isinstance(val, str):
-        return val
-
-    # Bytestring: needs decoding.
-    elif isinstance(val, bytes):
+    # Bytestring: Needs decoding to be safe for substitution in format strings.
+    if isinstance(val, bytes):
        # Blindly convert with UTF-8. Eventually, it would be nice to
        # (a) only do this for paths, if they can be given a distinct
        # type, and (b) warn the developer if they do this for other
        # bytestrings.
-        return val.decode('utf-8', 'replace')
-
-    # A "problem" object: needs a workaround.
-    elif isinstance(val, subprocess.CalledProcessError):
-        try:
-            return str(val)
-        except UnicodeDecodeError:
-            # An object with a broken __unicode__ formatter. Use __str__
-            # instead.
-            return str(val).decode('utf-8', 'replace')
+        return val.decode("utf-8", "replace")

    # Other objects are used as-is so field access, etc., still works in
-    # the format string.
-    else:
+    # the format string. Relies on a working __str__ implementation.
    return val


-class StrFormatLogger(Logger):
+class StrFormatLogger(logging.Logger):
    """A version of `Logger` that uses `str.format`-style formatting
-    instead of %-style formatting.
+    instead of %-style formatting and supports keyword arguments.
+
+    We cannot easily get rid of this even in the Python 3 era: This custom
+    formatting supports substitution from `kwargs` into the message, which the
+    default `logging.Logger._log()` implementation does not.
+
+    Remark by @sampsyo: https://stackoverflow.com/a/24683360 might be a way to
+    achieve this with less code.
    """

    class _LogMessage:
@@ -82,19 +69,39 @@ class StrFormatLogger(Logger):
            kwargs = {k: logsafe(v) for (k, v) in self.kwargs.items()}
            return self.msg.format(*args, **kwargs)

-    def _log(self, level, msg, args, exc_info=None, extra=None, **kwargs):
+    def _log(
+        self,
+        level,
+        msg,
+        args,
+        exc_info=None,
+        extra=None,
+        stack_info=False,
+        **kwargs,
+    ):
        """Log msg.format(*args, **kwargs)"""
        m = self._LogMessage(msg, args, kwargs)
-        return super()._log(level, m, (), exc_info, extra)
+
+        stacklevel = kwargs.pop("stacklevel", 1)
+        stacklevel = {"stacklevel": stacklevel}
+
+        return super()._log(
+            level,
+            m,
+            (),
+            exc_info=exc_info,
+            extra=extra,
+            stack_info=stack_info,
+            **stacklevel,
+        )


-class ThreadLocalLevelLogger(Logger):
-    """A version of `Logger` whose level is thread-local instead of shared.
-    """
+class ThreadLocalLevelLogger(logging.Logger):
+    """A version of `Logger` whose level is thread-local instead of shared."""

-    def __init__(self, name, level=NOTSET):
+    def __init__(self, name, level=logging.NOTSET):
        self._thread_level = threading.local()
-        self.default_level = NOTSET
+        self.default_level = logging.NOTSET
        super().__init__(name, level)

    @property
@@ -121,12 +128,17 @@ class BeetsLogger(ThreadLocalLevelLogger, StrFormatLogger):
    pass


-my_manager = copy(Logger.manager)
+my_manager = copy(logging.Logger.manager)
 my_manager.loggerClass = BeetsLogger


+# Act like the stdlib logging module by re-exporting its namespace.
+from logging import *  # noqa
+
+
+# Override the `getLogger` to use our machinery.
 def getLogger(name=None):  # noqa
    if name:
        return my_manager.getLogger(name)
    else:
-        return Logger.root
+        return logging.Logger.root
@@ -13,14 +13,22 @@
 # included in all copies or substantial portions of the Software.


+import warnings
+
 import mediafile

-import warnings
-warnings.warn("beets.mediafile is deprecated; use mediafile instead")
+warnings.warn(
+    "beets.mediafile is deprecated; use mediafile instead",
+    # Show the location of the `import mediafile` statement as the warning's
+    # source, rather than this file, such that the offending module can be
+    # identified easily.
+    stacklevel=2,
+)

 # Import everything from the mediafile module into this module.
 for key, value in mediafile.__dict__.items():
-    if key not in ['__name__']:
+    if key not in ["__name__"]:
        globals()[key] = value

+# Cleanup namespace.
 del key, value, warnings, mediafile
@@ -15,26 +15,25 @@
 """Support for beets plugins."""


-import traceback
-import re
-import inspect
 import abc
+import inspect
+import re
+import traceback
 from collections import defaultdict
 from functools import wraps

+import mediafile

 import beets
 from beets import logging
-import mediafile

-
-PLUGIN_NAMESPACE = 'beetsplug'
+PLUGIN_NAMESPACE = "beetsplug"

 # Plugins using the Last.fm API can share the same API key.
-LASTFM_KEY = '2dc3914abf35f0d9c92d97d8f8e42b43'
+LASTFM_KEY = "2dc3914abf35f0d9c92d97d8f8e42b43"

 # Global logger.
-log = logging.getLogger('beets')
+log = logging.getLogger("beets")


 class PluginConflictException(Exception):
@@ -51,11 +50,10 @@ class PluginLogFilter(logging.Filter):
    """

    def __init__(self, plugin):
-        self.prefix = f'{plugin.name}: '
+        self.prefix = f"{plugin.name}: "

    def filter(self, record):
-        if hasattr(record.msg, 'msg') and isinstance(record.msg.msg,
-                                                     str):
+        if hasattr(record.msg, "msg") and isinstance(record.msg.msg, str):
            # A _LogMessage from our hacked-up Logging replacement.
            record.msg.msg = self.prefix + record.msg.msg
        elif isinstance(record.msg, str):
@@ -65,6 +63,7 @@ class PluginLogFilter(logging.Filter):

 # Managing the plugins themselves.

+
 class BeetsPlugin:
    """The base class for all beets plugins. Plugins provide
    functionality by defining a subclass of BeetsPlugin and overriding
@@ -72,9 +71,8 @@ class BeetsPlugin:
    """

    def __init__(self, name=None):
-        """Perform one-time plugin setup.
-        """
-        self.name = name or self.__module__.split('.')[-1]
+        """Perform one-time plugin setup."""
+        self.name = name or self.__module__.split(".")[-1]
        self.config = beets.config[self.name]
        if not self.template_funcs:
            self.template_funcs = {}
@@ -97,10 +95,11 @@ class BeetsPlugin:
        return ()

    def _set_stage_log_level(self, stages):
-        """Adjust all the stages in `stages` to WARNING logging level.
-        """
-        return [self._set_log_level_and_params(logging.WARNING, stage)
-                for stage in stages]
+        """Adjust all the stages in `stages` to WARNING logging level."""
+        return [
+            self._set_log_level_and_params(logging.WARNING, stage)
+            for stage in stages
+        ]

    def get_early_import_stages(self):
        """Return a list of functions that should be called as importer
@@ -134,12 +133,11 @@ class BeetsPlugin:
        def wrapper(*args, **kwargs):
            assert self._log.level == logging.NOTSET

-            verbosity = beets.config['verbose'].get(int)
+            verbosity = beets.config["verbose"].get(int)
            log_level = max(logging.DEBUG, base_log_level - 10 * verbosity)
            self._log.setLevel(log_level)
            if argspec.varkw is None:
-                kwargs = {k: v for k, v in kwargs.items()
-                          if k in argspec.args}
+                kwargs = {k: v for k, v in kwargs.items() if k in argspec.args}

            try:
                return func(*args, **kwargs)
@@ -149,8 +147,7 @@ class BeetsPlugin:
        return wrapper

    def queries(self):
-        """Should return a dict mapping prefixes to Query subclasses.
-        """
+        """Return a dict mapping prefixes to Query subclasses."""
        return {}

    def track_distance(self, item, info):
@@ -201,6 +198,7 @@ class BeetsPlugin:
        """
        # Defer import to prevent circular dependency
        from beets import library
+
        mediafile.MediaFile.add_field(name, descriptor)
        library.Item._media_fields.add(name)

@@ -208,8 +206,7 @@ class BeetsPlugin:
    listeners = None

    def register_listener(self, event, func):
-        """Add a function as a listener for the specified event.
-        """
+        """Add a function as a listener for the specified event."""
        wrapped_func = self._set_log_level_and_params(logging.WARNING, func)

        cls = self.__class__
@@ -230,11 +227,13 @@ class BeetsPlugin:
        function will be invoked as ``%name{}`` from path format
        strings.
        """
+
        def helper(func):
            if cls.template_funcs is None:
                cls.template_funcs = {}
            cls.template_funcs[name] = func
            return func
+
        return helper

    @classmethod
@@ -244,11 +243,13 @@ class BeetsPlugin:
        strings. The function must accept a single parameter, the Item
        being formatted.
        """
+
        def helper(func):
            if cls.template_fields is None:
                cls.template_fields = {}
            cls.template_fields[name] = func
            return func
+
        return helper


@@ -262,25 +263,29 @@ def load_plugins(names=()):
    BeetsPlugin subclasses desired.
    """
    for name in names:
-        modname = f'{PLUGIN_NAMESPACE}.{name}'
+        modname = f"{PLUGIN_NAMESPACE}.{name}"
        try:
            try:
                namespace = __import__(modname, None, None)
            except ImportError as exc:
                # Again, this is hacky:
-                if exc.args[0].endswith(' ' + name):
-                    log.warning('** plugin {0} not found', name)
+                if exc.args[0].endswith(" " + name):
+                    log.warning("** plugin {0} not found", name)
                else:
                    raise
            else:
                for obj in getattr(namespace, name).__dict__.values():
-                    if isinstance(obj, type) and issubclass(obj, BeetsPlugin) \
-                            and obj != BeetsPlugin and obj not in _classes:
+                    if (
+                        isinstance(obj, type)
+                        and issubclass(obj, BeetsPlugin)
+                        and obj != BeetsPlugin
+                        and obj not in _classes
+                    ):
                        _classes.add(obj)

        except Exception:
            log.warning(
-                '** error loading plugin {}:\n{}',
+                "** error loading plugin {}:\n{}",
                name,
                traceback.format_exc(),
            )
@@ -311,9 +316,9 @@ def find_plugins():

 # Communication with plugins.

+
 def commands():
-    """Returns a list of Subcommand objects from all loaded plugins.
-    """
+    """Returns a list of Subcommand objects from all loaded plugins."""
    out = []
    for plugin in find_plugins():
        out += plugin.commands()
@@ -332,16 +337,16 @@ def queries():

 def types(model_cls):
    # Gives us `item_types` and `album_types`
-    attr_name = f'{model_cls.__name__.lower()}_types'
+    attr_name = f"{model_cls.__name__.lower()}_types"
    types = {}
    for plugin in find_plugins():
        plugin_types = getattr(plugin, attr_name, {})
        for field in plugin_types:
            if field in types and plugin_types[field] != types[field]:
                raise PluginConflictException(
-                    'Plugin {} defines flexible field {} '
-                    'which has already been defined with '
-                    'another type.'.format(plugin.name, field)
+                    "Plugin {} defines flexible field {} "
+                    "which has already been defined with "
+                    "another type.".format(plugin.name, field)
                )
        types.update(plugin_types)
    return types
@@ -349,7 +354,7 @@ def types(model_cls):

 def named_queries(model_cls):
    # Gather `item_queries` and `album_queries` from the plugins.
-    attr_name = f'{model_cls.__name__.lower()}_queries'
+    attr_name = f"{model_cls.__name__.lower()}_queries"
    queries = {}
    for plugin in find_plugins():
        plugin_queries = getattr(plugin, attr_name, {})
@@ -362,6 +367,7 @@ def track_distance(item, info):
    Returns a Distance object.
    """
    from beets.autotag.hooks import Distance
+
    dist = Distance()
    for plugin in find_plugins():
        dist.update(plugin.track_distance(item, info))
@@ -371,6 +377,7 @@ def track_distance(item, info):
 def album_distance(items, album_info, mapping):
    """Returns the album distance calculated by plugins."""
    from beets.autotag.hooks import Distance
+
    dist = Distance()
    for plugin in find_plugins():
        dist.update(plugin.album_distance(items, album_info, mapping))
@@ -378,23 +385,21 @@ def album_distance(items, album_info, mapping):


 def candidates(items, artist, album, va_likely, extra_tags=None):
-    """Gets MusicBrainz candidates for an album from each plugin.
-    """
+    """Gets MusicBrainz candidates for an album from each plugin."""
    for plugin in find_plugins():
-        yield from plugin.candidates(items, artist, album, va_likely,
-                                     extra_tags)
+        yield from plugin.candidates(
+            items, artist, album, va_likely, extra_tags
+        )


 def item_candidates(item, artist, title):
-    """Gets MusicBrainz candidates for an item from the plugins.
-    """
+    """Gets MusicBrainz candidates for an item from the plugins."""
    for plugin in find_plugins():
        yield from plugin.item_candidates(item, artist, title)


 def album_for_id(album_id):
-    """Get AlbumInfo objects for a given ID string.
-    """
+    """Get AlbumInfo objects for a given ID string."""
    for plugin in find_plugins():
        album = plugin.album_for_id(album_id)
        if album:
@@ -402,8 +407,7 @@ def album_for_id(album_id):


 def track_for_id(track_id):
-    """Get TrackInfo objects for a given ID string.
-    """
+    """Get TrackInfo objects for a given ID string."""
    for plugin in find_plugins():
        track = plugin.track_for_id(track_id)
        if track:
@@ -439,29 +443,44 @@ def import_stages():

 # New-style (lazy) plugin-provided fields.

+
+def _check_conflicts_and_merge(plugin, plugin_funcs, funcs):
+    """Check the provided template functions for conflicts and merge into funcs.
+
+    Raises a `PluginConflictException` if a plugin defines template functions
+    for fields that another plugin has already defined template functions for.
+    """
+    if plugin_funcs:
+        if not plugin_funcs.keys().isdisjoint(funcs.keys()):
+            conflicted_fields = ", ".join(plugin_funcs.keys() & funcs.keys())
+            raise PluginConflictException(
+                f"Plugin {plugin.name} defines template functions for "
+                f"{conflicted_fields} that conflict with another plugin."
+            )
+        funcs.update(plugin_funcs)
+
+
 def item_field_getters():
    """Get a dictionary mapping field names to unary functions that
    compute the field's value.
    """
    funcs = {}
    for plugin in find_plugins():
-        if plugin.template_fields:
-            funcs.update(plugin.template_fields)
+        _check_conflicts_and_merge(plugin, plugin.template_fields, funcs)
    return funcs


 def album_field_getters():
-    """As above, for album fields.
-    """
+    """As above, for album fields."""
    funcs = {}
    for plugin in find_plugins():
-        if plugin.album_template_fields:
-            funcs.update(plugin.album_template_fields)
+        _check_conflicts_and_merge(plugin, plugin.album_template_fields, funcs)
    return funcs


 # Event dispatch.

+
 def event_handlers():
    """Find all event handlers from plugins as a dictionary mapping
    event names to sequences of callables.
@@ -482,7 +501,7 @@ def send(event, **arguments):

    Return a list of non-None values returned from the handlers.
    """
-    log.debug('Sending event: {0}', event)
+    log.debug("Sending event: {0}", event)
    results = []
    for handler in event_handlers()[event]:
        result = handler(**arguments)
@@ -497,11 +516,11 @@ def feat_tokens(for_artist=True):
    The `for_artist` option determines whether the regex should be
    suitable for matching artist fields (the default) or title fields.
    """
-    feat_words = ['ft', 'featuring', 'feat', 'feat.', 'ft.']
+    feat_words = ["ft", "featuring", "feat", "feat.", "ft."]
    if for_artist:
-        feat_words += ['with', 'vs', 'and', 'con', '&']
-    return r'(?<=\s)(?:{})(?=\s)'.format(
-        '|'.join(re.escape(x) for x in feat_words)
+        feat_words += ["with", "vs", "and", "con", "&"]
+    return r"(?<=\s)(?:{})(?=\s)".format(
+        "|".join(re.escape(x) for x in feat_words)
    )


@@ -517,7 +536,7 @@ def sanitize_choices(choices, choices_all):
        if s not in seen:
            if s in list(choices_all):
                res.append(s)
-            elif s == '*':
+            elif s == "*":
                res.extend(others)
        seen.add(s)
    return res
@@ -550,11 +569,11 @@ def sanitize_pairs(pairs, pairs_all):
                if x not in seen:
                    seen.add(x)
                    res.append(x)
-            elif k == '*':
+            elif k == "*":
                new = [o for o in others if o not in seen]
                seen.update(new)
                res.extend(new)
-            elif v == '*':
+            elif v == "*":
                new = [o for o in others if o not in seen and o[0] == k]
                seen.update(new)
                res.extend(new)
@@ -568,12 +587,15 @@ def notify_info_yielded(event):
    Each yielded value is passed to plugins using the 'info' parameter of
    'send'.
    """
+
    def decorator(generator):
        def decorated(*args, **kwargs):
            for v in generator(*args, **kwargs):
                send(event, info=v)
                yield v
+
        return decorated
+
    return decorator


@@ -583,7 +605,7 @@ def get_distance(config, data_source, info):
    """
    dist = beets.autotag.Distance()
    if info.data_source == data_source:
-        dist.add('source', config['source_weight'].as_number())
+        dist.add("source", config["source_weight"].as_number())
    return dist


@@ -620,7 +642,7 @@ def apply_item_changes(lib, item, move, pretend, write):
 class MetadataSourcePlugin(metaclass=abc.ABCMeta):
    def __init__(self):
        super().__init__()
-        self.config.add({'source_weight': 0.5})
+        self.config.add({"source_weight": 0.5})

    @abc.abstractproperty
    def id_regex(self):
@@ -643,7 +665,7 @@ class MetadataSourcePlugin(metaclass=abc.ABCMeta):
        raise NotImplementedError

    @abc.abstractmethod
-    def _search_api(self, query_type, filters, keywords=''):
+    def _search_api(self, query_type, filters, keywords=""):
        raise NotImplementedError

    @abc.abstractmethod
@@ -655,7 +677,7 @@ class MetadataSourcePlugin(metaclass=abc.ABCMeta):
        raise NotImplementedError

    @staticmethod
-    def get_artist(artists, id_key='id', name_key='name'):
+    def get_artist(artists, id_key="id", name_key="name", join_key=None):
        """Returns an artist string (all artists) and an artist_id (the main
        artist) for a list of artist object dicts.

@@ -663,6 +685,8 @@ class MetadataSourcePlugin(metaclass=abc.ABCMeta):
        and 'the') to the front and strips trailing disambiguation numbers. It
        returns a tuple containing the comma-separated string of all
        normalized artists and the ``id`` of the main/first artist.
+        Alternatively a keyword can be used to combine artists together into a
+        single string by passing the join_key argument.

        :param artists: Iterable of artist dicts or lists returned by API.
        :type artists: list[dict] or list[list]
@@ -673,39 +697,55 @@ class MetadataSourcePlugin(metaclass=abc.ABCMeta):
            to concatenate for the artist string (containing all artists).
            Defaults to 'name'.
        :type name_key: str or int
+        :param join_key: Key or index corresponding to a field containing a
+            keyword to use for combining artists into a single string, for
+            example "Feat.", "Vs.", "And" or similar. The default is None
+            which keeps the default behaviour (comma-separated).
+        :type join_key: str or int
        :return: Normalized artist string.
        :rtype: str
        """
        artist_id = None
-        artist_names = []
-        for artist in artists:
+        artist_string = ""
+        artists = list(artists)  # In case a generator was passed.
+        total = len(artists)
+        for idx, artist in enumerate(artists):
            if not artist_id:
                artist_id = artist[id_key]
            name = artist[name_key]
            # Strip disambiguation number.
-            name = re.sub(r' \(\d+\)$', '', name)
+            name = re.sub(r" \(\d+\)$", "", name)
            # Move articles to the front.
-            name = re.sub(r'^(.*?), (a|an|the)$', r'\2 \1', name, flags=re.I)
-            artist_names.append(name)
-        artist = ', '.join(artist_names).replace(' ,', ',') or None
-        return artist, artist_id
+            name = re.sub(r"^(.*?), (a|an|the)$", r"\2 \1", name, flags=re.I)
+            # Use a join keyword if requested and available.
+            if idx < (total - 1):  # Skip joining on last.
+                if join_key and artist.get(join_key, None):
+                    name += f" {artist[join_key]} "
+                else:
+                    name += ", "
+            artist_string += name

-    def _get_id(self, url_type, id_):
+        return artist_string, artist_id
+
+    @staticmethod
+    def _get_id(url_type, id_, id_regex):
        """Parse an ID from its URL if necessary.

        :param url_type: Type of URL. Either 'album' or 'track'.
        :type url_type: str
        :param id_: Album/track ID or URL.
        :type id_: str
+        :param id_regex: A dictionary containing a regular expression
+            extracting an ID from an URL (if it's not an ID already) in
+            'pattern' and the number of the match group in 'match_group'.
+        :type id_regex: dict
        :return: Album/track ID.
        :rtype: str
        """
-        self._log.debug(
-            "Searching {} for {} '{}'", self.data_source, url_type, id_
-        )
-        match = re.search(self.id_regex['pattern'].format(url_type), str(id_))
+        log.debug("Extracting {} ID from '{}'", url_type, id_)
+        match = re.search(id_regex["pattern"].format(url_type), str(id_))
        if match:
-            id_ = match.group(self.id_regex['match_group'])
+            id_ = match.group(id_regex["match_group"])
            if id_:
                return id_
        return None
@@ -726,11 +766,11 @@ class MetadataSourcePlugin(metaclass=abc.ABCMeta):
        :return: Candidate AlbumInfo objects.
        :rtype: list[beets.autotag.hooks.AlbumInfo]
        """
-        query_filters = {'album': album}
+        query_filters = {"album": album}
        if not va_likely:
-            query_filters['artist'] = artist
-        results = self._search_api(query_type='album', filters=query_filters)
-        albums = [self.album_for_id(album_id=r['id']) for r in results]
+            query_filters["artist"] = artist
+        results = self._search_api(query_type="album", filters=query_filters)
+        albums = [self.album_for_id(album_id=r["id"]) for r in results]
        return [a for a in albums if a is not None]

    def item_candidates(self, item, artist, title):
@@ -747,7 +787,7 @@ class MetadataSourcePlugin(metaclass=abc.ABCMeta):
        :rtype: list[beets.autotag.hooks.TrackInfo]
        """
        tracks = self._search_api(
-            query_type='track', keywords=title, filters={'artist': artist}
+            query_type="track", keywords=title, filters={"artist": artist}
        )
        return [self.track_for_id(track_data=track) for track in tracks]

@@ -12,24 +12,22 @@
 # The above copyright notice and this permission notice shall be
 # included in all copies or substantial portions of the Software.

-"""Get a random song or album from the library.
-"""
+"""Get a random song or album from the library."""

 import random
-from operator import attrgetter
 from itertools import groupby
+from operator import attrgetter


 def _length(obj, album):
-    """Get the duration of an item or album.
-    """
+    """Get the duration of an item or album."""
    if album:
        return sum(i.length for i in obj.items())
    else:
        return obj.length


-def _equal_chance_permutation(objs, field='albumartist', random_gen=None):
+def _equal_chance_permutation(objs, field="albumartist", random_gen=None):
    """Generate (lazily) a permutation of the objects where every group
    with equal values for `field` have an equal chance of appearing in
    any given position.
@@ -86,8 +84,9 @@ def _take_time(iter, secs, album):
    return out


-def random_objs(objs, album, number=1, time=None, equal_chance=False,
-                random_gen=None):
+def random_objs(
+    objs, album, number=1, time=None, equal_chance=False, random_gen=None
+):
    """Get a random subset of the provided `objs`.

    If `number` is provided, produce that many matches. Otherwise, if
@@ -1,5 +1,5 @@
 # This file is part of beets.
-# Copyright 2016-2019, Adrian Sampson.
+# Copyright 2024, Lars Kruse
 #
 # Permission is hereby granted, free of charge, to any person obtaining
 # a copy of this software and associated documentation files (the
@@ -12,17 +12,8 @@
 # The above copyright notice and this permission notice shall be
 # included in all copies or substantial portions of the Software.

-
-import confuse
-
-import warnings
-warnings.warn("beets.util.confit is deprecated; use confuse instead")
-
-# Import everything from the confuse module into this module.
-for key, value in confuse.__dict__.items():
-    if key not in ['__name__']:
-        globals()[key] = value
-
-
-# Cleanup namespace.
-del key, value, warnings, confuse
+"""This module contains components of beets' test environment, which
+may be of use for testing procedures of external libraries or programs.
+For example the 'TestHelper' class may be useful for creating an
+in-memory beets library filled with a few example items.
+"""
@@ -0,0 +1,328 @@
+# This file is part of beets.
+# Copyright 2016, Adrian Sampson.
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+
+"""Some common functionality for beets' test cases."""
+
+import os
+import sys
+import tempfile
+import unittest
+from contextlib import contextmanager
+
+import beets
+import beets.library
+
+# Make sure the development versions of the plugins are used
+import beetsplug
+from beets import importer, logging, util
+from beets.ui import commands
+from beets.util import syspath
+
+beetsplug.__path__ = [
+    os.path.abspath(
+        os.path.join(
+            os.path.dirname(__file__),
+            os.path.pardir,
+            os.path.pardir,
+            "beetsplug",
+        )
+    )
+]
+
+# Test resources path.
+RSRC = util.bytestring_path(
+    os.path.abspath(
+        os.path.join(
+            os.path.dirname(__file__),
+            os.path.pardir,
+            os.path.pardir,
+            "test",
+            "rsrc",
+        )
+    )
+)
+PLUGINPATH = os.path.join(RSRC.decode(), "beetsplug")
+
+# Propagate to root logger so the test runner can capture it
+log = logging.getLogger("beets")
+log.propagate = True
+log.setLevel(logging.DEBUG)
+
+# Dummy item creation.
+_item_ident = 0
+
+# OS feature test.
+HAVE_SYMLINK = sys.platform != "win32"
+HAVE_HARDLINK = sys.platform != "win32"
+
+try:
+    import reflink
+
+    HAVE_REFLINK = reflink.supported_at(tempfile.gettempdir())
+except ImportError:
+    HAVE_REFLINK = False
+
+
+def item(lib=None):
+    global _item_ident
+    _item_ident += 1
+    i = beets.library.Item(
+        title="the title",
+        artist="the artist",
+        albumartist="the album artist",
+        album="the album",
+        genre="the genre",
+        lyricist="the lyricist",
+        composer="the composer",
+        arranger="the arranger",
+        grouping="the grouping",
+        work="the work title",
+        mb_workid="the work musicbrainz id",
+        work_disambig="the work disambiguation",
+        year=1,
+        month=2,
+        day=3,
+        track=4,
+        tracktotal=5,
+        disc=6,
+        disctotal=7,
+        lyrics="the lyrics",
+        comments="the comments",
+        bpm=8,
+        comp=True,
+        path=f"somepath{_item_ident}",
+        length=60.0,
+        bitrate=128000,
+        format="FLAC",
+        mb_trackid="someID-1",
+        mb_albumid="someID-2",
+        mb_artistid="someID-3",
+        mb_albumartistid="someID-4",
+        mb_releasetrackid="someID-5",
+        album_id=None,
+        mtime=12345,
+    )
+    if lib:
+        lib.add(i)
+    return i
+
+
+def album(lib=None):
+    global _item_ident
+    _item_ident += 1
+    i = beets.library.Album(
+        artpath=None,
+        albumartist="some album artist",
+        albumartist_sort="some sort album artist",
+        albumartist_credit="some album artist credit",
+        album="the album",
+        genre="the genre",
+        year=2014,
+        month=2,
+        day=5,
+        tracktotal=0,
+        disctotal=1,
+        comp=False,
+        mb_albumid="someID-1",
+        mb_albumartistid="someID-1",
+    )
+    if lib:
+        lib.add(i)
+    return i
+
+
+# Dummy import session.
+def import_session(lib=None, loghandler=None, paths=[], query=[], cli=False):
+    cls = commands.TerminalImportSession if cli else importer.ImportSession
+    return cls(lib, loghandler, paths, query)
+
+
+class Assertions:
+    """A mixin with additional unit test assertions."""
+
+    def assertExists(self, path):  # noqa
+        assert os.path.exists(syspath(path)), f"file does not exist: {path!r}"
+
+    def assertNotExists(self, path):  # noqa
+        assert not os.path.exists(syspath(path)), f"file exists: {path!r}"
+
+    def assertIsFile(self, path):  # noqa
+        self.assertExists(path)
+        assert os.path.isfile(
+            syspath(path)
+        ), "path exists, but is not a regular file: {!r}".format(path)
+
+    def assertIsDir(self, path):  # noqa
+        self.assertExists(path)
+        assert os.path.isdir(
+            syspath(path)
+        ), "path exists, but is not a directory: {!r}".format(path)
+
+    def assert_equal_path(self, a, b):
+        """Check that two paths are equal."""
+        a_bytes, b_bytes = util.normpath(a), util.normpath(b)
+
+        assert a_bytes == b_bytes, f"{a_bytes=} != {b_bytes=}"
+
+
+# Mock I/O.
+
+
+class InputException(Exception):
+    def __init__(self, output=None):
+        self.output = output
+
+    def __str__(self):
+        msg = "Attempt to read with no input provided."
+        if self.output is not None:
+            msg += f" Output: {self.output!r}"
+        return msg
+
+
+class DummyOut:
+    encoding = "utf-8"
+
+    def __init__(self):
+        self.buf = []
+
+    def write(self, s):
+        self.buf.append(s)
+
+    def get(self):
+        return "".join(self.buf)
+
+    def flush(self):
+        self.clear()
+
+    def clear(self):
+        self.buf = []
+
+
+class DummyIn:
+    encoding = "utf-8"
+
+    def __init__(self, out=None):
+        self.buf = []
+        self.reads = 0
+        self.out = out
+
+    def add(self, s):
+        self.buf.append(s + "\n")
+
+    def close(self):
+        pass
+
+    def readline(self):
+        if not self.buf:
+            if self.out:
+                raise InputException(self.out.get())
+            else:
+                raise InputException()
+        self.reads += 1
+        return self.buf.pop(0)
+
+
+class DummyIO:
+    """Mocks input and output streams for testing UI code."""
+
+    def __init__(self):
+        self.stdout = DummyOut()
+        self.stdin = DummyIn(self.stdout)
+
+    def addinput(self, s):
+        self.stdin.add(s)
+
+    def getoutput(self):
+        res = self.stdout.get()
+        self.stdout.clear()
+        return res
+
+    def readcount(self):
+        return self.stdin.reads
+
+    def install(self):
+        sys.stdin = self.stdin
+        sys.stdout = self.stdout
+
+    def restore(self):
+        sys.stdin = sys.__stdin__
+        sys.stdout = sys.__stdout__
+
+
+# Utility.
+
+
+def touch(path):
+    open(syspath(path), "a").close()
+
+
+class Bag:
+    """An object that exposes a set of fields given as keyword
+    arguments. Any field not found in the dictionary appears to be None.
+    Used for mocking Album objects and the like.
+    """
+
+    def __init__(self, **fields):
+        self.fields = fields
+
+    def __getattr__(self, key):
+        return self.fields.get(key)
+
+
+# Platform mocking.
+
+
+@contextmanager
+def platform_windows():
+    import ntpath
+
+    old_path = os.path
+    try:
+        os.path = ntpath
+        yield
+    finally:
+        os.path = old_path
+
+
+@contextmanager
+def platform_posix():
+    import posixpath
+
+    old_path = os.path
+    try:
+        os.path = posixpath
+        yield
+    finally:
+        os.path = old_path
+
+
+@contextmanager
+def system_mock(name):
+    import platform
+
+    old_system = platform.system
+    platform.system = lambda: name
+    try:
+        yield
+    finally:
+        platform.system = old_system
+
+
+def slow_test(unused=None):
+    def _id(obj):
+        return obj
+
+    if "SKIP_SLOW_TESTS" in os.environ:
+        return unittest.skip("test is slow")
+    return _id
@@ -31,7 +31,7 @@
 # plugins dynamically
 #
 # Currently, only Bash 3.2 and newer is supported and the
-# `bash-completion` package is requied.
+# `bash-completion` package (v2.8 or newer) is required.
 #
 # TODO
 # ----
@@ -46,7 +46,30 @@
 # * Support long options with `=`, e.g. `--config=file`. Debian's bash
 #   completion package can handle this.
 #
+# Note that 'bash-completion' v2.8 is a part of Debian 10, which is part of
+# LTS until 2024-06-30.  After this date, the minimum version requirement can
+# be changed, and newer features can be used unconditionally.  See PR#5301.
+#

+if [[ ${BASH_COMPLETION_VERSINFO[0]} -ne 2 \
+   || ${BASH_COMPLETION_VERSINFO[1]} -lt 8 ]]; then
+  echo "Incompatible version of 'bash-completion'!"
+  return 1
+fi
+
+# The later code relies on 'bash-completion' version 2.12, but older versions
+# are still supported.  Here, we provide implementations of the newer functions
+# in terms of older ones, if 'bash-completion' is too old to have them.
+
+if [[ ${BASH_COMPLETION_VERSINFO[1]} -lt 12 ]]; then
+  _comp_get_words() {
+    _get_comp_words_by_ref "$@"
+  }
+
+  _comp_compgen_filedir() {
+    _filedir "$@"
+  }
+fi

 # Determines the beets subcommand and dispatches the completion
 # accordingly.
@@ -54,7 +77,7 @@ _beet_dispatch() {
  local cur prev cmd=

  COMPREPLY=()
-  _get_comp_words_by_ref -n : cur prev
+  _comp_get_words -n : cur prev

  # Look for the beets subcommand
  local arg
@@ -99,7 +122,7 @@ _beet_complete() {
    completions="${flags___common} ${opts} ${flags}"
    COMPREPLY+=( $(compgen -W "$completions"  -- $cur) )
  else
-    _filedir
+    _comp_compgen_filedir
  fi
 }

@@ -114,12 +137,12 @@ _beet_complete_global() {
      ;;
    -l|--library|-c|--config)
      # Filename completion
-      _filedir
+      _comp_compgen_filedir
      return
      ;;
    -d|--directory)
      # Directory completion
-      _filedir -d
+      _comp_compgen_filedir -d
      return
      ;;
  esac
@@ -16,23 +16,20 @@
 public resizing proxy if neither is available.
 """

-import subprocess
 import os
 import os.path
+import platform
 import re
-from tempfile import NamedTemporaryFile
+import subprocess
+from itertools import chain
 from urllib.parse import urlencode
-from beets import logging
-from beets import util

-# Resizing methods
-PIL = 1
-IMAGEMAGICK = 2
-WEBPROXY = 3
+from beets import logging, util
+from beets.util import displayable_path, get_temp_filename, syspath

-PROXY_URL = 'https://images.weserv.nl/'
+PROXY_URL = "https://images.weserv.nl/"

-log = logging.getLogger('beets')
+log = logging.getLogger("beets")


 def resize_url(url, maxwidth, quality=0):
@@ -40,39 +37,343 @@ def resize_url(url, maxwidth, quality=0):
    maxwidth (preserving aspect ratio).
    """
    params = {
-        'url': url.replace('http://', ''),
-        'w': maxwidth,
+        "url": url.replace("http://", ""),
+        "w": maxwidth,
    }

    if quality > 0:
-        params['q'] = quality
+        params["q"] = quality

-    return '{}?{}'.format(PROXY_URL, urlencode(params))
+    return "{}?{}".format(PROXY_URL, urlencode(params))


-def temp_file_for(path):
-    """Return an unused filename with the same extension as the
-    specified path.
+class LocalBackendNotAvailableError(Exception):
+    pass
+
+
+_NOT_AVAILABLE = object()
+
+
+class LocalBackend:
+    @classmethod
+    def available(cls):
+        try:
+            cls.version()
+            return True
+        except LocalBackendNotAvailableError:
+            return False
+
+
+class IMBackend(LocalBackend):
+    NAME = "ImageMagick"
+
+    # These fields are used as a cache for `version()`. `_legacy` indicates
+    # whether the modern `magick` binary is available or whether to fall back
+    # to the old-style `convert`, `identify`, etc. commands.
+    _version = None
+    _legacy = None
+
+    @classmethod
+    def version(cls):
+        """Obtain and cache ImageMagick version.
+
+        Raises `LocalBackendNotAvailableError` if not available.
        """
-    ext = os.path.splitext(path)[1]
-    with NamedTemporaryFile(suffix=util.py3_path(ext), delete=False) as f:
-        return util.bytestring_path(f.name)
+        if cls._version is None:
+            for cmd_name, legacy in (("magick", False), ("convert", True)):
+                try:
+                    out = util.command_output([cmd_name, "--version"]).stdout
+                except (subprocess.CalledProcessError, OSError) as exc:
+                    log.debug("ImageMagick version check failed: {}", exc)
+                    cls._version = _NOT_AVAILABLE
+                else:
+                    if b"imagemagick" in out.lower():
+                        pattern = rb".+ (\d+)\.(\d+)\.(\d+).*"
+                        match = re.search(pattern, out)
+                        if match:
+                            cls._version = (
+                                int(match.group(1)),
+                                int(match.group(2)),
+                                int(match.group(3)),
+                            )
+                            cls._legacy = legacy
+
+        if cls._version is _NOT_AVAILABLE:
+            raise LocalBackendNotAvailableError()
+        else:
+            return cls._version
+
+    def __init__(self):
+        """Initialize a wrapper around ImageMagick for local image operations.
+
+        Stores the ImageMagick version and legacy flag. If ImageMagick is not
+        available, raise an Exception.
+        """
+        self.version()
+
+        # Use ImageMagick's magick binary when it's available.
+        # If it's not, fall back to the older, separate convert
+        # and identify commands.
+        if self._legacy:
+            self.convert_cmd = ["convert"]
+            self.identify_cmd = ["identify"]
+            self.compare_cmd = ["compare"]
+        else:
+            self.convert_cmd = ["magick"]
+            self.identify_cmd = ["magick", "identify"]
+            self.compare_cmd = ["magick", "compare"]
+
+    def resize(
+        self, maxwidth, path_in, path_out=None, quality=0, max_filesize=0
+    ):
+        """Resize using ImageMagick.
+
+        Use the ``magick`` program or ``convert`` on older versions. Return
+        the output path of resized image.
+        """
+        if not path_out:
+            path_out = get_temp_filename(__name__, "resize_IM_", path_in)
+
+        log.debug(
+            "artresizer: ImageMagick resizing {0} to {1}",
+            displayable_path(path_in),
+            displayable_path(path_out),
+        )
+
+        # "-resize WIDTHx>" shrinks images with the width larger
+        # than the given width while maintaining the aspect ratio
+        # with regards to the height.
+        # ImageMagick already seems to default to no interlace, but we include
+        # it here for the sake of explicitness.
+        cmd = self.convert_cmd + [
+            syspath(path_in, prefix=False),
+            "-resize",
+            f"{maxwidth}x>",
+            "-interlace",
+            "none",
+        ]
+
+        if quality > 0:
+            cmd += ["-quality", f"{quality}"]
+
+        # "-define jpeg:extent=SIZEb" sets the target filesize for imagemagick
+        # to SIZE in bytes.
+        if max_filesize > 0:
+            cmd += ["-define", f"jpeg:extent={max_filesize}b"]
+
+        cmd.append(syspath(path_out, prefix=False))
+
+        try:
+            util.command_output(cmd)
+        except subprocess.CalledProcessError:
+            log.warning(
+                "artresizer: IM convert failed for {0}",
+                displayable_path(path_in),
+            )
+            return path_in
+
+        return path_out
+
+    def get_size(self, path_in):
+        cmd = self.identify_cmd + [
+            "-format",
+            "%w %h",
+            syspath(path_in, prefix=False),
+        ]
+
+        try:
+            out = util.command_output(cmd).stdout
+        except subprocess.CalledProcessError as exc:
+            log.warning("ImageMagick size query failed")
+            log.debug(
+                "`convert` exited with (status {}) when "
+                "getting size with command {}:\n{}",
+                exc.returncode,
+                cmd,
+                exc.output.strip(),
+            )
+            return None
+        try:
+            return tuple(map(int, out.split(b" ")))
+        except IndexError:
+            log.warning("Could not understand IM output: {0!r}", out)
+            return None
+
+    def deinterlace(self, path_in, path_out=None):
+        if not path_out:
+            path_out = get_temp_filename(__name__, "deinterlace_IM_", path_in)
+
+        cmd = self.convert_cmd + [
+            syspath(path_in, prefix=False),
+            "-interlace",
+            "none",
+            syspath(path_out, prefix=False),
+        ]
+
+        try:
+            util.command_output(cmd)
+            return path_out
+        except subprocess.CalledProcessError:
+            # FIXME: Should probably issue a warning?
+            return path_in
+
+    def get_format(self, filepath):
+        cmd = self.identify_cmd + ["-format", "%[magick]", syspath(filepath)]
+
+        try:
+            return util.command_output(cmd).stdout
+        except subprocess.CalledProcessError:
+            # FIXME: Should probably issue a warning?
+            return None
+
+    def convert_format(self, source, target, deinterlaced):
+        cmd = self.convert_cmd + [
+            syspath(source),
+            *(["-interlace", "none"] if deinterlaced else []),
+            syspath(target),
+        ]
+
+        try:
+            subprocess.check_call(
+                cmd, stderr=subprocess.DEVNULL, stdout=subprocess.DEVNULL
+            )
+            return target
+        except subprocess.CalledProcessError:
+            # FIXME: Should probably issue a warning?
+            return source
+
+    @property
+    def can_compare(self):
+        return self.version() > (6, 8, 7)
+
+    def compare(self, im1, im2, compare_threshold):
+        is_windows = platform.system() == "Windows"
+
+        # Converting images to grayscale tends to minimize the weight
+        # of colors in the diff score. So we first convert both images
+        # to grayscale and then pipe them into the `compare` command.
+        # On Windows, ImageMagick doesn't support the magic \\?\ prefix
+        # on paths, so we pass `prefix=False` to `syspath`.
+        convert_cmd = self.convert_cmd + [
+            syspath(im2, prefix=False),
+            syspath(im1, prefix=False),
+            "-colorspace",
+            "gray",
+            "MIFF:-",
+        ]
+        compare_cmd = self.compare_cmd + [
+            "-define",
+            "phash:colorspaces=sRGB,HCLp",
+            "-metric",
+            "PHASH",
+            "-",
+            "null:",
+        ]
+        log.debug(
+            "comparing images with pipeline {} | {}", convert_cmd, compare_cmd
+        )
+        convert_proc = subprocess.Popen(
+            convert_cmd,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+            close_fds=not is_windows,
+        )
+        compare_proc = subprocess.Popen(
+            compare_cmd,
+            stdin=convert_proc.stdout,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+            close_fds=not is_windows,
+        )
+
+        # Check the convert output. We're not interested in the
+        # standard output; that gets piped to the next stage.
+        convert_proc.stdout.close()
+        convert_stderr = convert_proc.stderr.read()
+        convert_proc.stderr.close()
+        convert_proc.wait()
+        if convert_proc.returncode:
+            log.debug(
+                "ImageMagick convert failed with status {}: {!r}",
+                convert_proc.returncode,
+                convert_stderr,
+            )
+            return None
+
+        # Check the compare output.
+        stdout, stderr = compare_proc.communicate()
+        if compare_proc.returncode:
+            if compare_proc.returncode != 1:
+                log.debug(
+                    "ImageMagick compare failed: {0}, {1}",
+                    displayable_path(im2),
+                    displayable_path(im1),
+                )
+                return None
+            out_str = stderr
+        else:
+            out_str = stdout
+
+        try:
+            phash_diff = float(out_str)
+        except ValueError:
+            log.debug("IM output is not a number: {0!r}", out_str)
+            return None
+
+        log.debug("ImageMagick compare score: {0}", phash_diff)
+        return phash_diff <= compare_threshold
+
+    @property
+    def can_write_metadata(self):
+        return True
+
+    def write_metadata(self, file, metadata):
+        assignments = list(
+            chain.from_iterable(("-set", k, v) for k, v in metadata.items())
+        )
+        command = self.convert_cmd + [file, *assignments, file]
+
+        util.command_output(command)


-def pil_resize(maxwidth, path_in, path_out=None, quality=0, max_filesize=0):
+class PILBackend(LocalBackend):
+    NAME = "PIL"
+
+    @classmethod
+    def version(cls):
+        try:
+            __import__("PIL", fromlist=["Image"])
+        except ImportError:
+            raise LocalBackendNotAvailableError()
+
+    def __init__(self):
+        """Initialize a wrapper around PIL for local image operations.
+
+        If PIL is not available, raise an Exception.
+        """
+        self.version()
+
+    def resize(
+        self, maxwidth, path_in, path_out=None, quality=0, max_filesize=0
+    ):
        """Resize using Python Imaging Library (PIL).  Return the output path
        of resized image.
        """
-    path_out = path_out or temp_file_for(path_in)
+        if not path_out:
+            path_out = get_temp_filename(__name__, "resize_PIL_", path_in)
+
        from PIL import Image

-    log.debug('artresizer: PIL resizing {0} to {1}',
-              util.displayable_path(path_in), util.displayable_path(path_out))
+        log.debug(
+            "artresizer: PIL resizing {0} to {1}",
+            displayable_path(path_in),
+            displayable_path(path_out),
+        )

        try:
-        im = Image.open(util.syspath(path_in))
+            im = Image.open(syspath(path_in))
            size = maxwidth, maxwidth
-        im.thumbnail(size, Image.ANTIALIAS)
+            im.thumbnail(size, Image.Resampling.LANCZOS)

            if quality == 0:
                # Use PIL's default quality.
@@ -80,19 +381,19 @@ def pil_resize(maxwidth, path_in, path_out=None, quality=0, max_filesize=0):

            # progressive=False only affects JPEGs and is the default,
            # but we include it here for explicitness.
-        im.save(util.py3_path(path_out), quality=quality, progressive=False)
+            im.save(os.fsdecode(path_out), quality=quality, progressive=False)

            if max_filesize > 0:
-            # If maximum filesize is set, we attempt to lower the quality of
-            # jpeg conversion by a proportional amount, up to 3 attempts
+                # If maximum filesize is set, we attempt to lower the quality
+                # of jpeg conversion by a proportional amount, up to 3 attempts
                # First, set the maximum quality to either provided, or 95
                if quality > 0:
                    lower_qual = quality
                else:
                    lower_qual = 95
                for i in range(5):
-                # 5 attempts is an abitrary choice
-                filesize = os.stat(util.syspath(path_out)).st_size
+                    # 5 attempts is an arbitrary choice
+                    filesize = os.stat(syspath(path_out)).st_size
                    log.debug("PIL Pass {0} : Output size: {1}B", i, filesize)
                    if filesize <= max_filesize:
                        return path_out
@@ -103,202 +404,106 @@ def pil_resize(maxwidth, path_in, path_out=None, quality=0, max_filesize=0):
                    if lower_qual < 10:
                        lower_qual = 10
                    # Use optimize flag to improve filesize decrease
-                im.save(util.py3_path(path_out), quality=lower_qual,
-                        optimize=True, progressive=False)
-            log.warning("PIL Failed to resize file to below {0}B",
-                        max_filesize)
+                    im.save(
+                        os.fsdecode(path_out),
+                        quality=lower_qual,
+                        optimize=True,
+                        progressive=False,
+                    )
+                log.warning(
+                    "PIL Failed to resize file to below {0}B", max_filesize
+                )
                return path_out

            else:
                return path_out
        except OSError:
-        log.error("PIL cannot create thumbnail for '{0}'",
-                  util.displayable_path(path_in))
+            log.error(
+                "PIL cannot create thumbnail for '{0}'",
+                displayable_path(path_in),
+            )
            return path_in

-
-def im_resize(maxwidth, path_in, path_out=None, quality=0, max_filesize=0):
-    """Resize using ImageMagick.
-
-    Use the ``magick`` program or ``convert`` on older versions. Return
-    the output path of resized image.
-    """
-    path_out = path_out or temp_file_for(path_in)
-    log.debug('artresizer: ImageMagick resizing {0} to {1}',
-              util.displayable_path(path_in), util.displayable_path(path_out))
-
-    # "-resize WIDTHx>" shrinks images with the width larger
-    # than the given width while maintaining the aspect ratio
-    # with regards to the height.
-    # ImageMagick already seems to default to no interlace, but we include it
-    # here for the sake of explicitness.
-    cmd = ArtResizer.shared.im_convert_cmd + [
-        util.syspath(path_in, prefix=False),
-        '-resize', f'{maxwidth}x>',
-        '-interlace', 'none',
-    ]
-
-    if quality > 0:
-        cmd += ['-quality', f'{quality}']
-
-    # "-define jpeg:extent=SIZEb" sets the target filesize for imagemagick to
-    # SIZE in bytes.
-    if max_filesize > 0:
-        cmd += ['-define', f'jpeg:extent={max_filesize}b']
-
-    cmd.append(util.syspath(path_out, prefix=False))
-
-    try:
-        util.command_output(cmd)
-    except subprocess.CalledProcessError:
-        log.warning('artresizer: IM convert failed for {0}',
-                    util.displayable_path(path_in))
-        return path_in
-
-    return path_out
-
-
-BACKEND_FUNCS = {
-    PIL: pil_resize,
-    IMAGEMAGICK: im_resize,
-}
-
-
-def pil_getsize(path_in):
+    def get_size(self, path_in):
        from PIL import Image

        try:
-        im = Image.open(util.syspath(path_in))
+            im = Image.open(syspath(path_in))
            return im.size
        except OSError as exc:
-        log.error("PIL could not read file {}: {}",
-                  util.displayable_path(path_in), exc)
-
-
-def im_getsize(path_in):
-    cmd = ArtResizer.shared.im_identify_cmd + \
-        ['-format', '%w %h', util.syspath(path_in, prefix=False)]
-
-    try:
-        out = util.command_output(cmd).stdout
-    except subprocess.CalledProcessError as exc:
-        log.warning('ImageMagick size query failed')
-        log.debug(
-            '`convert` exited with (status {}) when '
-            'getting size with command {}:\n{}',
-            exc.returncode, cmd, exc.output.strip()
+            log.error(
+                "PIL could not read file {}: {}", displayable_path(path_in), exc
            )
-        return
-    try:
-        return tuple(map(int, out.split(b' ')))
-    except IndexError:
-        log.warning('Could not understand IM output: {0!r}', out)
+            return None

+    def deinterlace(self, path_in, path_out=None):
+        if not path_out:
+            path_out = get_temp_filename(__name__, "deinterlace_PIL_", path_in)

-BACKEND_GET_SIZE = {
-    PIL: pil_getsize,
-    IMAGEMAGICK: im_getsize,
-}
-
-
-def pil_deinterlace(path_in, path_out=None):
-    path_out = path_out or temp_file_for(path_in)
        from PIL import Image

        try:
-        im = Image.open(util.syspath(path_in))
-        im.save(util.py3_path(path_out), progressive=False)
+            im = Image.open(syspath(path_in))
+            im.save(os.fsdecode(path_out), progressive=False)
            return path_out
-    except IOError:
+        except OSError:
+            # FIXME: Should probably issue a warning?
            return path_in

-
-def im_deinterlace(path_in, path_out=None):
-    path_out = path_out or temp_file_for(path_in)
-
-    cmd = ArtResizer.shared.im_convert_cmd + [
-        util.syspath(path_in, prefix=False),
-        '-interlace', 'none',
-        util.syspath(path_out, prefix=False),
-    ]
-
-    try:
-        util.command_output(cmd)
-        return path_out
-    except subprocess.CalledProcessError:
-        return path_in
-
-
-DEINTERLACE_FUNCS = {
-    PIL: pil_deinterlace,
-    IMAGEMAGICK: im_deinterlace,
-}
-
-
-def im_get_format(filepath):
-    cmd = ArtResizer.shared.im_identify_cmd + [
-        '-format', '%[magick]',
-        util.syspath(filepath)
-    ]
-
-    try:
-        return util.command_output(cmd).stdout
-    except subprocess.CalledProcessError:
-        return None
-
-
-def pil_get_format(filepath):
+    def get_format(self, filepath):
        from PIL import Image, UnidentifiedImageError

        try:
-        with Image.open(util.syspath(filepath)) as im:
+            with Image.open(syspath(filepath)) as im:
                return im.format
-    except (ValueError, TypeError, UnidentifiedImageError, FileNotFoundError):
+        except (
+            ValueError,
+            TypeError,
+            UnidentifiedImageError,
+            FileNotFoundError,
+        ):
            log.exception("failed to detect image format for {}", filepath)
            return None

-
-BACKEND_GET_FORMAT = {
-    PIL: pil_get_format,
-    IMAGEMAGICK: im_get_format,
-}
-
-
-def im_convert_format(source, target, deinterlaced):
-    cmd = ArtResizer.shared.im_convert_cmd + [
-        util.syspath(source),
-        *(["-interlace", "none"] if deinterlaced else []),
-        util.syspath(target),
-    ]
-
-    try:
-        subprocess.check_call(
-            cmd,
-            stderr=subprocess.DEVNULL,
-            stdout=subprocess.DEVNULL
-        )
-        return target
-    except subprocess.CalledProcessError:
-        return source
-
-
-def pil_convert_format(source, target, deinterlaced):
+    def convert_format(self, source, target, deinterlaced):
        from PIL import Image, UnidentifiedImageError

        try:
-        with Image.open(util.syspath(source)) as im:
-            im.save(util.py3_path(target), progressive=not deinterlaced)
+            with Image.open(syspath(source)) as im:
+                im.save(os.fsdecode(target), progressive=not deinterlaced)
                return target
-    except (ValueError, TypeError, UnidentifiedImageError, FileNotFoundError,
-            OSError):
+        except (
+            ValueError,
+            TypeError,
+            UnidentifiedImageError,
+            FileNotFoundError,
+            OSError,
+        ):
            log.exception("failed to convert image {} -> {}", source, target)
            return source

+    @property
+    def can_compare(self):
+        return False

-BACKEND_CONVERT_IMAGE_FORMAT = {
-    PIL: pil_convert_format,
-    IMAGEMAGICK: im_convert_format,
-}
+    def compare(self, im1, im2, compare_threshold):
+        # It is an error to call this when ArtResizer.can_compare is not True.
+        raise NotImplementedError()
+
+    @property
+    def can_write_metadata(self):
+        return True
+
+    def write_metadata(self, file, metadata):
+        from PIL import Image, PngImagePlugin
+
+        # FIXME: Detect and handle other file types (currently, the only user
+        # is the thumbnails plugin, which generates PNG images).
+        im = Image.open(syspath(file))
+        meta = PngImagePlugin.PngInfo()
+        for k, v in metadata.items():
+            meta.add_text(k, v, 0)
+        im.save(os.fsdecode(file), "PNG", pnginfo=meta)


 class Shareable(type):
@@ -319,28 +524,36 @@ class Shareable(type):
        return cls._instance


+BACKEND_CLASSES = [
+    IMBackend,
+    PILBackend,
+]
+
+
 class ArtResizer(metaclass=Shareable):
-    """A singleton class that performs image resizes.
-    """
+    """A singleton class that performs image resizes."""

    def __init__(self):
-        """Create a resizer object with an inferred method.
-        """
-        self.method = self._check_method()
-        log.debug("artresizer: method is {0}", self.method)
-        self.can_compare = self._can_compare()
-
-        # Use ImageMagick's magick binary when it's available. If it's
-        # not, fall back to the older, separate convert and identify
-        # commands.
-        if self.method[0] == IMAGEMAGICK:
-            self.im_legacy = self.method[2]
-            if self.im_legacy:
-                self.im_convert_cmd = ['convert']
-                self.im_identify_cmd = ['identify']
+        """Create a resizer object with an inferred method."""
+        # Check if a local backend is available, and store an instance of the
+        # backend class. Otherwise, fallback to the web proxy.
+        for backend_cls in BACKEND_CLASSES:
+            try:
+                self.local_method = backend_cls()
+                log.debug(f"artresizer: method is {self.local_method.NAME}")
+                break
+            except LocalBackendNotAvailableError:
+                continue
        else:
-                self.im_convert_cmd = ['magick']
-                self.im_identify_cmd = ['magick', 'identify']
+            log.debug("artresizer: method is WEBPROXY")
+            self.local_method = None
+
+    @property
+    def method(self):
+        if self.local:
+            return self.local_method.NAME
+        else:
+            return "WEBPROXY"

    def resize(
        self, maxwidth, path_in, path_out=None, quality=0, max_filesize=0
@@ -351,17 +564,26 @@ class ArtResizer(metaclass=Shareable):
        For WEBPROXY, returns `path_in` unmodified.
        """
        if self.local:
-            func = BACKEND_FUNCS[self.method[0]]
-            return func(maxwidth, path_in, path_out,
-                        quality=quality, max_filesize=max_filesize)
+            return self.local_method.resize(
+                maxwidth,
+                path_in,
+                path_out,
+                quality=quality,
+                max_filesize=max_filesize,
+            )
        else:
+            # Handled by `proxy_url` already.
            return path_in

    def deinterlace(self, path_in, path_out=None):
+        """Deinterlace an image.
+
+        Only available locally.
+        """
        if self.local:
-            func = DEINTERLACE_FUNCS[self.method[0]]
-            return func(path_in, path_out)
+            return self.local_method.deinterlace(path_in, path_out)
        else:
+            # FIXME: Should probably issue a warning?
            return path_in

    def proxy_url(self, maxwidth, url, quality=0):
@@ -370,6 +592,7 @@ class ArtResizer(metaclass=Shareable):
        Otherwise, the URL is returned unmodified.
        """
        if self.local:
+            # Going to be handled by `resize()`.
            return url
        else:
            return resize_url(url, maxwidth, quality)
@@ -379,7 +602,7 @@ class ArtResizer(metaclass=Shareable):
        """A boolean indicating whether the resizing method is performed
        locally (i.e., PIL or ImageMagick).
        """
-        return self.method[0] in BACKEND_FUNCS
+        return self.local_method is not None

    def get_size(self, path_in):
        """Return the size of an image file as an int couple (width, height)
@@ -388,8 +611,10 @@ class ArtResizer(metaclass=Shareable):
        Only available locally.
        """
        if self.local:
-            func = BACKEND_GET_SIZE[self.method[0]]
-            return func(path_in)
+            return self.local_method.get_size(path_in)
+        else:
+            # FIXME: Should probably issue a warning?
+            return path_in

    def get_format(self, path_in):
        """Returns the format of the image as a string.
@@ -397,8 +622,10 @@ class ArtResizer(metaclass=Shareable):
        Only available locally.
        """
        if self.local:
-            func = BACKEND_GET_FORMAT[self.method[0]]
-            return func(path_in)
+            return self.local_method.get_format(path_in)
+        else:
+            # FIXME: Should probably issue a warning?
+            return None

    def reformat(self, path_in, new_format, deinterlaced=True):
        """Converts image to desired format, updating its extension, but
@@ -407,86 +634,66 @@ class ArtResizer(metaclass=Shareable):
        Only available locally.
        """
        if not self.local:
+            # FIXME: Should probably issue a warning?
            return path_in

        new_format = new_format.lower()
        # A nonexhaustive map of image "types" to extensions overrides
        new_format = {
-            'jpeg': 'jpg',
+            "jpeg": "jpg",
        }.get(new_format, new_format)

        fname, ext = os.path.splitext(path_in)
-        path_new = fname + b'.' + new_format.encode('utf8')
-        func = BACKEND_CONVERT_IMAGE_FORMAT[self.method[0]]
+        path_new = fname + b"." + new_format.encode("utf8")

        # allows the exception to propagate, while still making sure a changed
        # file path was removed
        result_path = path_in
        try:
-            result_path = func(path_in, path_new, deinterlaced)
+            result_path = self.local_method.convert_format(
+                path_in, path_new, deinterlaced
+            )
        finally:
            if result_path != path_in:
                os.unlink(path_in)
        return result_path

-    def _can_compare(self):
+    @property
+    def can_compare(self):
        """A boolean indicating whether image comparison is available"""

-        return self.method[0] == IMAGEMAGICK and self.method[1] > (6, 8, 7)
-
-    @staticmethod
-    def _check_method():
-        """Return a tuple indicating an available method and its version.
-
-        The result has at least two elements:
-        - The method, eitehr WEBPROXY, PIL, or IMAGEMAGICK.
-        - The version.
-
-        If the method is IMAGEMAGICK, there is also a third element: a
-        bool flag indicating whether to use the `magick` binary or
-        legacy single-purpose executables (`convert`, `identify`, etc.)
-        """
-        version = get_im_version()
-        if version:
-            version, legacy = version
-            return IMAGEMAGICK, version, legacy
-
-        version = get_pil_version()
-        if version:
-            return PIL, version
-
-        return WEBPROXY, (0)
-
-
-def get_im_version():
-    """Get the ImageMagick version and legacy flag as a pair. Or return
-    None if ImageMagick is not available.
-    """
-    for cmd_name, legacy in ((['magick'], False), (['convert'], True)):
-        cmd = cmd_name + ['--version']
-
-        try:
-            out = util.command_output(cmd).stdout
-        except (subprocess.CalledProcessError, OSError) as exc:
-            log.debug('ImageMagick version check failed: {}', exc)
+        if self.local:
+            return self.local_method.can_compare
        else:
-            if b'imagemagick' in out.lower():
-                pattern = br".+ (\d+)\.(\d+)\.(\d+).*"
-                match = re.search(pattern, out)
-                if match:
-                    version = (int(match.group(1)),
-                               int(match.group(2)),
-                               int(match.group(3)))
-                    return version, legacy
+            return False

-    return None
+    def compare(self, im1, im2, compare_threshold):
+        """Return a boolean indicating whether two images are similar.

-
-def get_pil_version():
-    """Get the PIL/Pillow version, or None if it is unavailable.
+        Only available locally.
        """
-    try:
-        __import__('PIL', fromlist=['Image'])
-        return (0,)
-    except ImportError:
+        if self.local:
+            return self.local_method.compare(im1, im2, compare_threshold)
+        else:
+            # FIXME: Should probably issue a warning?
            return None
+
+    @property
+    def can_write_metadata(self):
+        """A boolean indicating whether writing image metadata is supported."""
+
+        if self.local:
+            return self.local_method.can_write_metadata
+        else:
+            return False
+
+    def write_metadata(self, file, metadata):
+        """Write key-value metadata to the image file.
+
+        Only available locally. Currently, expects the image to be a PNG file.
+        """
+        if self.local:
+            self.local_method.write_metadata(file, metadata)
+        else:
+            # FIXME: Should probably issue a warning?
+            pass
@@ -6,23 +6,24 @@ asyncore.
 Bluelet: easy concurrency without all the messy parallelism.
 """

-import socket
-import select
-import sys
-import types
-import errno
-import traceback
-import time
 import collections
-
+import errno
+import select
+import socket
+import sys
+import time
+import traceback
+import types

 # Basic events used for thread scheduling.

+
 class Event:
    """Just a base class identifying Bluelet events. An event is an
    object yielded from a Bluelet thread coroutine to suspend operation
    and communicate with the scheduler.
    """
+
    pass


@@ -31,6 +32,7 @@ class WaitableEvent(Event):
    waited for using a select() call. That is, it's an event with an
    associated file descriptor.
    """
+
    def waitables(self):
        """Return "waitable" objects to pass to select(). Should return
        three iterables for input readiness, output readiness, and
@@ -48,18 +50,21 @@ class WaitableEvent(Event):

 class ValueEvent(Event):
    """An event that does nothing but return a fixed value."""
+
    def __init__(self, value):
        self.value = value


 class ExceptionEvent(Event):
    """Raise an exception at the yield point. Used internally."""
+
    def __init__(self, exc_info):
        self.exc_info = exc_info


 class SpawnEvent(Event):
    """Add a new coroutine thread to the scheduler."""
+
    def __init__(self, coro):
        self.spawned = coro

@@ -68,12 +73,14 @@ class JoinEvent(Event):
    """Suspend the thread until the specified child thread has
    completed.
    """
+
    def __init__(self, child):
        self.child = child


 class KillEvent(Event):
    """Unschedule a child thread."""
+
    def __init__(self, child):
        self.child = child

@@ -83,6 +90,7 @@ class DelegationEvent(Event):
    once the child thread finished, return control to the parent
    thread.
    """
+
    def __init__(self, coro):
        self.spawned = coro

@@ -91,13 +99,14 @@ class ReturnEvent(Event):
    """Return a value the current thread's delegator at the point of
    delegation. Ends the current (delegate) thread.
    """
+
    def __init__(self, value):
        self.value = value


 class SleepEvent(WaitableEvent):
-    """Suspend the thread for a given duration.
-    """
+    """Suspend the thread for a given duration."""
+
    def __init__(self, duration):
        self.wakeup_time = time.time() + duration

@@ -107,6 +116,7 @@ class SleepEvent(WaitableEvent):

 class ReadEvent(WaitableEvent):
    """Reads from a file-like object."""
+
    def __init__(self, fd, bufsize):
        self.fd = fd
        self.bufsize = bufsize
@@ -120,6 +130,7 @@ class ReadEvent(WaitableEvent):

 class WriteEvent(WaitableEvent):
    """Writes to a file-like object."""
+
    def __init__(self, fd, data):
        self.fd = fd
        self.data = data
@@ -133,6 +144,7 @@ class WriteEvent(WaitableEvent):

 # Core logic for executing and scheduling threads.

+
 def _event_select(events):
    """Perform a select() over all the Events provided, returning the
    ones ready to be fired. Only WaitableEvents (including SleepEvents)
@@ -154,11 +166,11 @@ def _event_select(events):
            wlist += w
            xlist += x
            for waitable in r:
-                waitable_to_event[('r', waitable)] = event
+                waitable_to_event[("r", waitable)] = event
            for waitable in w:
-                waitable_to_event[('w', waitable)] = event
+                waitable_to_event[("w", waitable)] = event
            for waitable in x:
-                waitable_to_event[('x', waitable)] = event
+                waitable_to_event[("x", waitable)] = event

    # If we have a any sleeping threads, determine how long to sleep.
    if earliest_wakeup:
@@ -177,11 +189,11 @@ def _event_select(events):
    # Gather ready events corresponding to the ready waitables.
    ready_events = set()
    for ready in rready:
-        ready_events.add(waitable_to_event[('r', ready)])
+        ready_events.add(waitable_to_event[("r", ready)])
    for ready in wready:
-        ready_events.add(waitable_to_event[('w', ready)])
+        ready_events.add(waitable_to_event[("w", ready)])
    for ready in xready:
-        ready_events.add(waitable_to_event[('x', ready)])
+        ready_events.add(waitable_to_event[("x", ready)])

    # Gather any finished sleeps.
    for event in events:
@@ -207,6 +219,7 @@ class Delegated(Event):
    """Placeholder indicating that a thread has delegated execution to a
    different thread.
    """
+
    def __init__(self, child):
        self.child = child

@@ -277,8 +290,7 @@ def run(root_coro):
            threads[coro] = next_event

    def kill_thread(coro):
-        """Unschedule this thread and its (recursive) delegates.
-        """
+        """Unschedule this thread and its (recursive) delegates."""
        # Collect all coroutines in the delegation stack.
        coros = [coro]
        while isinstance(threads[coro], Delegated):
@@ -338,12 +350,16 @@ def run(root_coro):
                try:
                    value = event.fire()
                except OSError as exc:
-                    if isinstance(exc.args, tuple) and \
-                            exc.args[0] == errno.EPIPE:
+                    if (
+                        isinstance(exc.args, tuple)
+                        and exc.args[0] == errno.EPIPE
+                    ):
                        # Broken pipe. Remote host disconnected.
                        pass
-                    elif isinstance(exc.args, tuple) and \
-                            exc.args[0] == errno.ECONNRESET:
+                    elif (
+                        isinstance(exc.args, tuple)
+                        and exc.args[0] == errno.ECONNRESET
+                    ):
                        # Connection was reset by peer.
                        pass
                    else:
@@ -382,16 +398,16 @@ def run(root_coro):

 # Sockets and their associated events.

+
 class SocketClosedError(Exception):
    pass


 class Listener:
-    """A socket wrapper object for listening sockets.
-    """
+    """A socket wrapper object for listening sockets."""
+
    def __init__(self, host, port):
-        """Create a listening socket on the given hostname and port.
-        """
+        """Create a listening socket on the given hostname and port."""
        self._closed = False
        self.host = host
        self.port = port
@@ -410,19 +426,18 @@ class Listener:
        return AcceptEvent(self)

    def close(self):
-        """Immediately close the listening socket. (Not an event.)
-        """
+        """Immediately close the listening socket. (Not an event.)"""
        self._closed = True
        self.sock.close()


 class Connection:
-    """A socket wrapper object for connected sockets.
-    """
+    """A socket wrapper object for connected sockets."""
+
    def __init__(self, sock, addr):
        self.sock = sock
        self.addr = addr
-        self._buf = b''
+        self._buf = b""
        self._closed = False

    def close(self):
@@ -473,7 +488,7 @@ class Connection:
                self._buf += data
            else:
                line = self._buf
-                self._buf = b''
+                self._buf = b""
                yield ReturnEvent(line)
                break

@@ -482,6 +497,7 @@ class AcceptEvent(WaitableEvent):
    """An event for Listener objects (listening sockets) that suspends
    execution until the socket gets a connection.
    """
+
    def __init__(self, listener):
        self.listener = listener

@@ -497,6 +513,7 @@ class ReceiveEvent(WaitableEvent):
    """An event for Connection objects (connected sockets) for
    asynchronously reading data.
    """
+
    def __init__(self, conn, bufsize):
        self.conn = conn
        self.bufsize = bufsize
@@ -512,6 +529,7 @@ class SendEvent(WaitableEvent):
    """An event for Connection objects (connected sockets) for
    asynchronously writing data.
    """
+
    def __init__(self, conn, data, sendall=False):
        self.conn = conn
        self.data = data
@@ -530,9 +548,9 @@ class SendEvent(WaitableEvent):
 # Public interface for threads; each returns an event object that
 # can immediately be "yield"ed.

+
 def null():
-    """Event: yield to the scheduler without doing anything special.
-    """
+    """Event: yield to the scheduler without doing anything special."""
    return ValueEvent(None)


@@ -541,7 +559,7 @@ def spawn(coro):
    and child coroutines run concurrently.
    """
    if not isinstance(coro, types.GeneratorType):
-        raise ValueError('%s is not a coroutine' % coro)
+        raise ValueError("%s is not a coroutine" % coro)
    return SpawnEvent(coro)


@@ -551,7 +569,7 @@ def call(coro):
    returns a value using end(), then this event returns that value.
    """
    if not isinstance(coro, types.GeneratorType):
-        raise ValueError('%s is not a coroutine' % coro)
+        raise ValueError("%s is not a coroutine" % coro)
    return DelegationEvent(coro)


@@ -573,7 +591,8 @@ def read(fd, bufsize=None):
                if not data:
                    break
                buf.append(data)
-            yield ReturnEvent(''.join(buf))
+            yield ReturnEvent("".join(buf))
+
        return DelegationEvent(reader())

    else:
@@ -595,8 +614,7 @@ def connect(host, port):


 def sleep(duration):
-    """Event: suspend the thread for ``duration`` seconds.
-    """
+    """Event: suspend the thread for ``duration`` seconds."""
    return SleepEvent(duration)


@@ -608,19 +626,20 @@ def join(coro):


 def kill(coro):
-    """Halt the execution of a different `spawn`ed thread.
-    """
+    """Halt the execution of a different `spawn`ed thread."""
    return KillEvent(coro)


 # Convenience function for running socket servers.

+
 def server(host, port, func):
    """A coroutine that runs a network server. Host and port specify the
    listening address. func should be a coroutine that takes a single
    parameter, a Connection object. The coroutine is invoked for every
    incoming connection on the listening socket.
    """
+
    def handler(conn):
        try:
            yield func(conn)
@@ -20,6 +20,7 @@ class OrderedEnum(Enum):
    """
    An Enum subclass that allows comparison of members.
    """
+
    def __ge__(self, other):
        if self.__class__ is other.__class__:
            return self.value >= other.value
@@ -27,22 +27,21 @@ engine like Jinja2 or Mustache.
 """


-import re
 import ast
 import dis
-import types
-import sys
 import functools
+import re
+import types

-SYMBOL_DELIM = '$'
-FUNC_DELIM = '%'
-GROUP_OPEN = '{'
-GROUP_CLOSE = '}'
-ARG_SEP = ','
-ESCAPE_CHAR = '$'
+SYMBOL_DELIM = "$"
+FUNC_DELIM = "%"
+GROUP_OPEN = "{"
+GROUP_CLOSE = "}"
+ARG_SEP = ","
+ESCAPE_CHAR = "$"

-VARIABLE_PREFIX = '__var_'
-FUNCTION_PREFIX = '__func_'
+VARIABLE_PREFIX = "__var_"
+FUNCTION_PREFIX = "__func_"


 class Environment:
@@ -57,10 +56,6 @@ class Environment:

 # Code generation helpers.

-def ex_lvalue(name):
-    """A variable load expression."""
-    return ast.Name(name, ast.Store())
-

 def ex_rvalue(name):
    """A variable store expression."""
@@ -74,15 +69,6 @@ def ex_literal(val):
    return ast.Constant(val)


-def ex_varassign(name, expr):
-    """Assign an expression into a single variable. The expression may
-    either be an `ast.expr` object or a value to be used as a literal.
-    """
-    if not isinstance(expr, ast.expr):
-        expr = ex_literal(expr)
-    return ast.Assign([ex_lvalue(name)], expr)
-
-
 def ex_call(func, args):
    """A function-call expression with only positional parameters. The
    function may be an expression or the name of a function. Each
@@ -99,19 +85,18 @@ def ex_call(func, args):
    return ast.Call(func, args, [])


-def compile_func(arg_names, statements, name='_the_func', debug=False):
+def compile_func(arg_names, statements, name="_the_func", debug=False):
    """Compile a list of statements as the body of a function and return
    the resulting Python function. If `debug`, then print out the
    bytecode of the compiled function.
    """
    args_fields = {
-        'args': [ast.arg(arg=n, annotation=None) for n in arg_names],
-        'kwonlyargs': [],
-        'kw_defaults': [],
-        'defaults': [ex_literal(None) for _ in arg_names],
+        "args": [ast.arg(arg=n, annotation=None) for n in arg_names],
+        "kwonlyargs": [],
+        "kw_defaults": [],
+        "defaults": [ex_literal(None) for _ in arg_names],
    }
-    if 'posonlyargs' in ast.arguments._fields:  # Added in Python 3.8.
-        args_fields['posonlyargs'] = []
+    args_fields["posonlyargs"] = []
    args = ast.arguments(**args_fields)

    func_def = ast.FunctionDef(
@@ -123,14 +108,11 @@ def compile_func(arg_names, statements, name='_the_func', debug=False):

    # The ast.Module signature changed in 3.8 to accept a list of types to
    # ignore.
-    if sys.version_info >= (3, 8):
    mod = ast.Module([func_def], [])
-    else:
-        mod = ast.Module([func_def])

    ast.fix_missing_locations(mod)

-    prog = compile(mod, '<generated>', 'exec')
+    prog = compile(mod, "<generated>", "exec")

    # Debug: show bytecode.
    if debug:
@@ -146,6 +128,7 @@ def compile_func(arg_names, statements, name='_the_func', debug=False):

 # AST nodes for the template language.

+
 class Symbol:
    """A variable-substitution symbol in a template."""

@@ -154,7 +137,7 @@ class Symbol:
        self.original = original

    def __repr__(self):
-        return 'Symbol(%s)' % repr(self.ident)
+        return "Symbol(%s)" % repr(self.ident)

    def evaluate(self, env):
        """Evaluate the symbol in the environment, returning a Unicode
@@ -183,8 +166,9 @@ class Call:
        self.original = original

    def __repr__(self):
-        return 'Call({}, {}, {})'.format(repr(self.ident), repr(self.args),
-                                         repr(self.original))
+        return "Call({}, {}, {})".format(
+            repr(self.ident), repr(self.args), repr(self.original)
+        )

    def evaluate(self, env):
        """Evaluate the function call in the environment, returning a
@@ -197,7 +181,7 @@ class Call:
            except Exception as exc:
                # Function raised exception! Maybe inlining the name of
                # the exception will help debug.
-                return '<%s>' % str(exc)
+                return "<%s>" % str(exc)
            return str(out)
        else:
            return self.original
@@ -215,21 +199,22 @@ class Call:

            # Create a subexpression that joins the result components of
            # the arguments.
-            arg_exprs.append(ex_call(
-                ast.Attribute(ex_literal(''), 'join', ast.Load()),
-                [ex_call(
-                    'map',
+            arg_exprs.append(
+                ex_call(
+                    ast.Attribute(ex_literal(""), "join", ast.Load()),
+                    [
+                        ex_call(
+                            "map",
                            [
                                ex_rvalue(str.__name__),
                                ast.List(subexprs, ast.Load()),
-                    ]
-                )],
-            ))
-
-        subexpr_call = ex_call(
-            FUNCTION_PREFIX + self.ident,
-            arg_exprs
+                            ],
                        )
+                    ],
+                )
+            )
+
+        subexpr_call = ex_call(FUNCTION_PREFIX + self.ident, arg_exprs)
        return [subexpr_call], varnames, funcnames


@@ -242,7 +227,7 @@ class Expression:
        self.parts = parts

    def __repr__(self):
-        return 'Expression(%s)' % (repr(self.parts))
+        return "Expression(%s)" % (repr(self.parts))

    def evaluate(self, env):
        """Evaluate the entire expression in the environment, returning
@@ -254,7 +239,7 @@ class Expression:
                out.append(part)
            else:
                out.append(part.evaluate(env))
-        return ''.join(map(str, out))
+        return "".join(map(str, out))

    def translate(self):
        """Compile the expression to a list of Python AST expressions, a
@@ -276,6 +261,7 @@ class Expression:

 # Parser.

+
 class ParseError(Exception):
    pass

@@ -306,10 +292,16 @@ class Parser:
        self.parts = []

    # Common parsing resources.
-    special_chars = (SYMBOL_DELIM, FUNC_DELIM, GROUP_OPEN, GROUP_CLOSE,
-                     ESCAPE_CHAR)
-    special_char_re = re.compile(r'[%s]|\Z' %
-                                 ''.join(re.escape(c) for c in special_chars))
+    special_chars = (
+        SYMBOL_DELIM,
+        FUNC_DELIM,
+        GROUP_OPEN,
+        GROUP_CLOSE,
+        ESCAPE_CHAR,
+    )
+    special_char_re = re.compile(
+        r"[%s]|\Z" % "".join(re.escape(c) for c in special_chars)
+    )
    escapable_chars = (SYMBOL_DELIM, FUNC_DELIM, GROUP_CLOSE, ARG_SEP)
    terminator_chars = (GROUP_CLOSE,)

@@ -326,9 +318,10 @@ class Parser:
        if self.in_argument:
            extra_special_chars = (ARG_SEP,)
            special_char_re = re.compile(
-                r'[%s]|\Z' % ''.join(
-                    re.escape(c) for c in
-                    self.special_chars + extra_special_chars
+                r"[%s]|\Z"
+                % "".join(
+                    re.escape(c)
+                    for c in self.special_chars + extra_special_chars
                )
            )

@@ -341,8 +334,8 @@ class Parser:
                # A non-special character. Skip to the next special
                # character, treating the interstice as literal text.
                next_pos = (
-                    special_char_re.search(
-                        self.string[self.pos:]).start() + self.pos
+                    special_char_re.search(self.string[self.pos :]).start()
+                    + self.pos
                )
                text_parts.append(self.string[self.pos : next_pos])
                self.pos = next_pos
@@ -358,8 +351,9 @@ class Parser:
                break

            next_char = self.string[self.pos + 1]
-            if char == ESCAPE_CHAR and next_char in (self.escapable_chars +
-                                                     extra_special_chars):
+            if char == ESCAPE_CHAR and next_char in (
+                self.escapable_chars + extra_special_chars
+            ):
                # An escaped special character ($$, $}, etc.). Note that
                # ${ is not an escape sequence: this is ambiguous with
                # the start of a symbol and it's not necessary (just
@@ -370,7 +364,7 @@ class Parser:

            # Shift all characters collected so far into a single string.
            if text_parts:
-                self.parts.append(''.join(text_parts))
+                self.parts.append("".join(text_parts))
                text_parts = []

            if char == SYMBOL_DELIM:
@@ -392,7 +386,7 @@ class Parser:

        # If any parsed characters remain, shift them into a string.
        if text_parts:
-            self.parts.append(''.join(text_parts))
+            self.parts.append("".join(text_parts))

    def parse_symbol(self):
        """Parse a variable reference (like ``$foo`` or ``${foo}``)
@@ -424,16 +418,18 @@ class Parser:
                # Closer found.
                ident = self.string[self.pos : closer]
                self.pos = closer + 1
-                self.parts.append(Symbol(ident,
-                                         self.string[start_pos:self.pos]))
+                self.parts.append(
+                    Symbol(ident, self.string[start_pos : self.pos])
+                )

        else:
            # A bare-word symbol.
            ident = self._parse_ident()
            if ident:
                # Found a real symbol.
-                self.parts.append(Symbol(ident,
-                                         self.string[start_pos:self.pos]))
+                self.parts.append(
+                    Symbol(ident, self.string[start_pos : self.pos])
+                )
            else:
                # A standalone $.
                self.parts.append(SYMBOL_DELIM)
@@ -468,8 +464,7 @@ class Parser:
        # Skip past opening brace and try to parse an argument list.
        self.pos += 1
        args = self.parse_argument_list()
-        if self.pos >= len(self.string) or \
-           self.string[self.pos] != GROUP_CLOSE:
+        if self.pos >= len(self.string) or self.string[self.pos] != GROUP_CLOSE:
            # Arguments unclosed.
            self.parts.append(self.string[start_pos : self.pos])
            return
@@ -494,8 +489,10 @@ class Parser:
            expressions.append(Expression(subparser.parts))
            self.pos += subparser.pos

-            if self.pos >= len(self.string) or \
-               self.string[self.pos] == GROUP_CLOSE:
+            if (
+                self.pos >= len(self.string)
+                or self.string[self.pos] == GROUP_CLOSE
+            ):
                # Argument list terminated by EOF or closing brace.
                break

@@ -511,7 +508,7 @@ class Parser:
        Updates ``pos``.
        """
        remainder = self.string[self.pos :]
-        ident = re.match(r'\w*', remainder).group(0)
+        ident = re.match(r"\w*", remainder).group(0)
        self.pos += len(ident)
        return ident

@@ -530,26 +527,14 @@ def _parse(template):
    return Expression(parts)


-def cached(func):
-    """Like the `functools.lru_cache` decorator, but works (as a no-op)
-    on Python < 3.2.
-    """
-    if hasattr(functools, 'lru_cache'):
-        return functools.lru_cache(maxsize=128)(func)
-    else:
-        # Do nothing when lru_cache is not available.
-        return func
-
-
-@cached
+@functools.lru_cache(maxsize=128)
 def template(fmt):
    return Template(fmt)


 # External interface.
 class Template:
-    """A string template, including text, Symbols, and Calls.
-    """
+    """A string template, including text, Symbols, and Calls."""

    def __init__(self, template):
        self.expr = _parse(template)
@@ -568,8 +553,7 @@ class Template:
        return self.expr.evaluate(Environment(values, functions))

    def substitute(self, values={}, functions={}):
-        """Evaluate the template given the values and functions.
-        """
+        """Evaluate the template given the values and functions."""
        try:
            res = self.compiled(values, functions)
        except Exception:  # Handle any exceptions thrown by compiled version.
@@ -599,24 +583,29 @@ class Template:
            for funcname in funcnames:
                args[FUNCTION_PREFIX + funcname] = functions[funcname]
            parts = func(**args)
-            return ''.join(parts)
+            return "".join(parts)

        return wrapper_func


 # Performance tests.

-if __name__ == '__main__':
+if __name__ == "__main__":
    import timeit
-    _tmpl = Template('foo $bar %baz{foozle $bar barzle} $bar')
-    _vars = {'bar': 'qux'}
-    _funcs = {'baz': str.upper}
-    interp_time = timeit.timeit('_tmpl.interpret(_vars, _funcs)',
-                                'from __main__ import _tmpl, _vars, _funcs',
-                                number=10000)
+
+    _tmpl = Template("foo $bar %baz{foozle $bar barzle} $bar")
+    _vars = {"bar": "qux"}
+    _funcs = {"baz": str.upper}
+    interp_time = timeit.timeit(
+        "_tmpl.interpret(_vars, _funcs)",
+        "from __main__ import _tmpl, _vars, _funcs",
+        number=10000,
+    )
    print(interp_time)
-    comp_time = timeit.timeit('_tmpl.substitute(_vars, _funcs)',
-                              'from __main__ import _tmpl, _vars, _funcs',
-                              number=10000)
+    comp_time = timeit.timeit(
+        "_tmpl.substitute(_vars, _funcs)",
+        "from __main__ import _tmpl, _vars, _funcs",
+        number=10000,
+    )
    print(comp_time)
-    print('Speedup:', interp_time / comp_time)
+    print("Speedup:", interp_time / comp_time)
@@ -1,5 +1,6 @@
 # This file is part of beets.
 # Copyright 2016, Adrian Sampson.
+# Copyright 2024, Arav K.
 #
 # Permission is hereby granted, free of charge, to any person obtaining
 # a copy of this software and associated documentation files (the
@@ -14,71 +15,49 @@

 """Simple library to work out if a file is hidden on different platforms."""

+import ctypes
 import os
 import stat
-import ctypes
 import sys
-import beets.util
+from pathlib import Path
+from typing import Union


-def _is_hidden_osx(path):
-    """Return whether or not a file is hidden on OS X.
-
-    This uses os.lstat to work out if a file has the "hidden" flag.
+def is_hidden(path: Union[bytes, Path]) -> bool:
    """
-    file_stat = os.lstat(beets.util.syspath(path))
-
-    if hasattr(file_stat, 'st_flags') and hasattr(stat, 'UF_HIDDEN'):
-        return bool(file_stat.st_flags & stat.UF_HIDDEN)
-    else:
-        return False
-
-
-def _is_hidden_win(path):
-    """Return whether or not a file is hidden on Windows.
-
-    This uses GetFileAttributes to work out if a file has the "hidden" flag
-    (FILE_ATTRIBUTE_HIDDEN).
+    Determine whether the given path is treated as a 'hidden file' by the OS.
    """
+
+    if isinstance(path, bytes):
+        path = Path(os.fsdecode(path))
+
+    # TODO: Avoid doing a platform check on every invocation of the function.
+    # TODO: Stop supporting 'bytes' inputs once 'pathlib' is fully integrated.
+
+    if sys.platform == "win32":
+        # On Windows, we check for an FS-provided attribute.
+
        # FILE_ATTRIBUTE_HIDDEN = 2 (0x2) from GetFileAttributes documentation.
        hidden_mask = 2

        # Retrieve the attributes for the file.
-    attrs = ctypes.windll.kernel32.GetFileAttributesW(beets.util.syspath(path))
+        attrs = ctypes.windll.kernel32.GetFileAttributesW(str(path))

-    # Ensure we have valid attribues and compare them against the mask.
-    return attrs >= 0 and attrs & hidden_mask
+        # Ensure the attribute mask is valid.
+        if attrs < 0:
+            return False

+        # Check for the hidden attribute.
+        return attrs & hidden_mask

-def _is_hidden_dot(path):
-    """Return whether or not a file starts with a dot.
+    # On OS X, we check for an FS-provided attribute.
+    if sys.platform == "darwin":
+        if hasattr(os.stat_result, "st_flags") and hasattr(stat, "UF_HIDDEN"):
+            if path.lstat().st_flags & stat.UF_HIDDEN:
+                return True

-    Files starting with a dot are seen as "hidden" files on Unix-based OSes.
-    """
-    return os.path.basename(path).startswith(b'.')
+    # On all non-Windows platforms, we check for a '.'-prefixed file name.
+    if path.name.startswith("."):
+        return True

-
-def is_hidden(path):
-    """Return whether or not a file is hidden. `path` should be a
-    bytestring filename.
-
-    This method works differently depending on the platform it is called on.
-
-    On OS X, it uses both the result of `is_hidden_osx` and `is_hidden_dot` to
-    work out if a file is hidden.
-
-    On Windows, it uses the result of `is_hidden_win` to work out if a file is
-    hidden.
-
-    On any other operating systems (i.e. Linux), it uses `is_hidden_dot` to
-    work out if a file is hidden.
-    """
-    # Run platform specific functions depending on the platform
-    if sys.platform == 'darwin':
-        return _is_hidden_osx(path) or _is_hidden_dot(path)
-    elif sys.platform == 'win32':
-        return _is_hidden_win(path)
-    else:
-        return _is_hidden_dot(path)
-
-__all__ = ['is_hidden']
+    return False
@@ -0,0 +1,65 @@
+# This file is part of beets.
+# Copyright 2016, Adrian Sampson.
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+
+"""Helpers around the extraction of album/track ID's from metadata sources."""
+
+import re
+
+# Spotify IDs consist of 22 alphanumeric characters
+# (zero-left-padded base62 representation of randomly generated UUID4)
+spotify_id_regex = {
+    "pattern": r"(^|open\.spotify\.com/{}/)([0-9A-Za-z]{{22}})",
+    "match_group": 2,
+}
+
+deezer_id_regex = {
+    "pattern": r"(^|deezer\.com/)([a-z]*/)?({}/)?(\d+)",
+    "match_group": 4,
+}
+
+beatport_id_regex = {
+    "pattern": r"(^|beatport\.com/release/.+/)(\d+)$",
+    "match_group": 2,
+}
+
+# A note on Bandcamp: There is no such thing as a Bandcamp album or artist ID,
+# the URL can be used as the identifier. The Bandcamp metadata source plugin
+# works that way - https://github.com/snejus/beetcamp. Bandcamp album
+# URLs usually look like: https://nameofartist.bandcamp.com/album/nameofalbum
+
+
+def extract_discogs_id_regex(album_id):
+    """Returns the Discogs_id or None."""
+    # Discogs-IDs are simple integers. In order to avoid confusion with
+    # other metadata plugins, we only look for very specific formats of the
+    # input string:
+    # - plain integer, optionally wrapped in brackets and prefixed by an
+    #   'r', as this is how discogs displays the release ID on its webpage.
+    # - legacy url format: discogs.com/<name of release>/release/<id>
+    # - legacy url short format: discogs.com/release/<id>
+    # - current url format: discogs.com/release/<id>-<name of release>
+    # See #291, #4080 and #4085 for the discussions leading up to these
+    # patterns.
+    # Regex has been tested here https://regex101.com/r/TOu7kw/1
+
+    for pattern in [
+        r"^\[?r?(?P<id>\d+)\]?$",
+        r"discogs\.com/release/(?P<id>\d+)-?",
+        r"discogs\.com/[^/]+/release/(?P<id>\d+)",
+    ]:
+        match = re.search(pattern, album_id)
+        if match:
+            return int(match.group("id"))
+
+    return None
@@ -0,0 +1,97 @@
+# This file is part of beets.
+# Copyright 2022, J0J0 Todos.
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+
+"""Provides utilities to read, write and manipulate m3u playlist files."""
+
+import traceback
+
+from beets.util import FilesystemError, mkdirall, normpath, syspath
+
+
+class EmptyPlaylistError(Exception):
+    """Raised when a playlist file without media files is saved or loaded."""
+
+    pass
+
+
+class M3UFile:
+    """Reads and writes m3u or m3u8 playlist files."""
+
+    def __init__(self, path):
+        """``path`` is the absolute path to the playlist file.
+
+        The playlist file type, m3u or m3u8 is determined by 1) the ending
+        being m3u8 and 2) the file paths contained in the list being utf-8
+        encoded. Since the list is passed from the outside, this is currently
+        out of control of this class.
+        """
+        self.path = path
+        self.extm3u = False
+        self.media_list = []
+
+    def load(self):
+        """Reads the m3u file from disk and sets the object's attributes."""
+        pl_normpath = normpath(self.path)
+        try:
+            with open(syspath(pl_normpath), "rb") as pl_file:
+                raw_contents = pl_file.readlines()
+        except OSError as exc:
+            raise FilesystemError(
+                exc, "read", (pl_normpath,), traceback.format_exc()
+            )
+
+        self.extm3u = True if raw_contents[0].rstrip() == b"#EXTM3U" else False
+        for line in raw_contents[1:]:
+            if line.startswith(b"#"):
+                # Support for specific EXTM3U comments could be added here.
+                continue
+            self.media_list.append(normpath(line.rstrip()))
+        if not self.media_list:
+            raise EmptyPlaylistError
+
+    def set_contents(self, media_list, extm3u=True):
+        """Sets self.media_list to a list of media file paths.
+
+        Also sets additional flags, changing the final m3u-file's format.
+
+        ``media_list`` is a list of paths to media files that should be added
+        to the playlist (relative or absolute paths, that's the responsibility
+        of the caller). By default the ``extm3u`` flag is set, to ensure a
+        save-operation writes an m3u-extended playlist (comment "#EXTM3U" at
+        the top of the file).
+        """
+        self.media_list = media_list
+        self.extm3u = extm3u
+
+    def write(self):
+        """Writes the m3u file to disk.
+
+        Handles the creation of potential parent directories.
+        """
+        header = [b"#EXTM3U"] if self.extm3u else []
+        if not self.media_list:
+            raise EmptyPlaylistError
+        contents = header + self.media_list
+        pl_normpath = normpath(self.path)
+        mkdirall(pl_normpath)
+
+        try:
+            with open(syspath(pl_normpath), "wb") as pl_file:
+                for line in contents:
+                    pl_file.write(line + b"\n")
+                pl_file.write(b"\n")  # Final linefeed to prevent noeol file.
+        except OSError as exc:
+            raise FilesystemError(
+                exc, "create", (pl_normpath,), traceback.format_exc()
+            )
@@ -33,11 +33,11 @@ in place of any single coroutine.


 import queue
-from threading import Thread, Lock
 import sys
+from threading import Lock, Thread

-BUBBLE = '__PIPELINE_BUBBLE__'
-POISON = '__PIPELINE_POISON__'
+BUBBLE = "__PIPELINE_BUBBLE__"
+POISON = "__PIPELINE_POISON__"

 DEFAULT_QUEUE_SIZE = 16

@@ -48,6 +48,7 @@ def _invalidate_queue(q, val=None, sync=True):
    which defaults to None. `sync` controls whether a lock is
    required (because it's not reentrant!).
    """
+
    def _qsize(len=len):
        return 1

@@ -75,8 +76,8 @@ def _invalidate_queue(q, val=None, sync=True):
        q._qsize = _qsize
        q._put = _put
        q._get = _get
-        q.not_empty.notifyAll()
-        q.not_full.notifyAll()
+        q.not_empty.notify_all()
+        q.not_full.notify_all()

    finally:
        if sync:
@@ -168,6 +169,7 @@ def stage(func):
        while True:
            task = yield task
            task = func(*(args + (task,)))
+
    return coro


@@ -191,6 +193,7 @@ def mutator_stage(func):
        while True:
            task = yield task
            func(*(args + (task,)))
+
    return coro


@@ -218,20 +221,18 @@ class PipelineThread(Thread):
        self.exc_info = None

    def abort(self):
-        """Shut down the thread at the next chance possible.
-        """
+        """Shut down the thread at the next chance possible."""
        with self.abort_lock:
            self.abort_flag = True

            # Ensure that we are not blocking on a queue read or write.
-            if hasattr(self, 'in_queue'):
+            if hasattr(self, "in_queue"):
                _invalidate_queue(self.in_queue, POISON)
-            if hasattr(self, 'out_queue'):
+            if hasattr(self, "out_queue"):
                _invalidate_queue(self.out_queue, POISON)

    def abort_all(self, exc_info):
-        """Abort all other threads in the system for an exception.
-        """
+        """Abort all other threads in the system for an exception."""
        self.exc_info = exc_info
        for thread in self.all_threads:
            thread.abort()
@@ -373,7 +374,7 @@ class Pipeline:
        be at least two stages.
        """
        if len(stages) < 2:
-            raise ValueError('pipeline must have at least two stages')
+            raise ValueError("pipeline must have at least two stages")
        self.stages = []
        for stage in stages:
            if isinstance(stage, (list, tuple)):
@@ -405,15 +406,15 @@ class Pipeline:
        # Middle stages.
        for i in range(1, queue_count):
            for coro in self.stages[i]:
-                threads.append(MiddlePipelineThread(
+                threads.append(
+                    MiddlePipelineThread(
                        coro, queues[i - 1], queues[i], threads
-                ))
+                    )
+                )

        # Last stage.
        for coro in self.stages[-1]:
-            threads.append(
-                LastPipelineThread(coro, queues[-1], threads)
-            )
+            threads.append(LastPipelineThread(coro, queues[-1], threads))

        # Start threads.
        for thread in threads:
@@ -472,21 +473,21 @@ class Pipeline:


 # Smoke test.
-if __name__ == '__main__':
+if __name__ == "__main__":
    import time

    # Test a normally-terminating pipeline both in sequence and
    # in parallel.
    def produce():
        for i in range(5):
-            print('generating %i' % i)
+            print("generating %i" % i)
            time.sleep(1)
            yield i

    def work():
        num = yield
        while True:
-            print('processing %i' % num)
+            print("processing %i" % num)
            time.sleep(2)
            num = yield num * 2

@@ -494,7 +495,7 @@ if __name__ == '__main__':
        while True:
            num = yield
            time.sleep(1)
-            print('received %i' % num)
+            print("received %i" % num)

    ts_start = time.time()
    Pipeline([produce(), work(), consume()]).run_sequential()
@@ -503,22 +504,22 @@ if __name__ == '__main__':
    ts_par = time.time()
    Pipeline([produce(), (work(), work()), consume()]).run_parallel()
    ts_end = time.time()
-    print('Sequential time:', ts_seq - ts_start)
-    print('Parallel time:', ts_par - ts_seq)
-    print('Multiply-parallel time:', ts_end - ts_par)
+    print("Sequential time:", ts_seq - ts_start)
+    print("Parallel time:", ts_par - ts_seq)
+    print("Multiply-parallel time:", ts_end - ts_par)
    print()

    # Test a pipeline that raises an exception.
    def exc_produce():
        for i in range(10):
-            print('generating %i' % i)
+            print("generating %i" % i)
            time.sleep(1)
            yield i

    def exc_work():
        num = yield
        while True:
-            print('processing %i' % num)
+            print("processing %i" % num)
            time.sleep(3)
            if num == 3:
                raise Exception()
@@ -527,6 +528,6 @@ if __name__ == '__main__':
    def exc_consume():
        while True:
            num = yield
-            print('received %i' % num)
+            print("received %i" % num)

    Pipeline([exc_produce(), exc_work(), exc_consume()]).run_parallel(1)
@@ -17,9 +17,10 @@ libraries.
 """

 from collections import namedtuple
+
 from beets import util

-Node = namedtuple('Node', ['files', 'dirs'])
+Node = namedtuple("Node", ["files", "dirs"])


 def _insert(node, path, itemid):
@@ -17,4 +17,5 @@

 # Make this a namespace package.
 from pkgutil import extend_path
+
 __path__ = extend_path(__path__, __name__)
@@ -22,16 +22,14 @@ import json
 import os
 import subprocess
 import tempfile
-
 from distutils.spawn import find_executable
+
 import requests

-from beets import plugins
-from beets import util
-from beets import ui
+from beets import plugins, ui, util

 # We use this field to check whether AcousticBrainz info is present.
-PROBE_FIELD = 'mood_acoustic'
+PROBE_FIELD = "mood_acoustic"


 class ABSubmitError(Exception):
@@ -47,39 +45,39 @@ def call(args):
        return util.command_output(args).stdout
    except subprocess.CalledProcessError as e:
        raise ABSubmitError(
-            '{} exited with status {}'.format(args[0], e.returncode)
+            "{} exited with status {}".format(args[0], e.returncode)
        )


 class AcousticBrainzSubmitPlugin(plugins.BeetsPlugin):
-
    def __init__(self):
        super().__init__()

-        self.config.add({
-            'extractor': '',
-            'force': False,
-            'pretend': False
-        })
+        self._log.warning("This plugin is deprecated.")

-        self.extractor = self.config['extractor'].as_str()
+        self.config.add(
+            {"extractor": "", "force": False, "pretend": False, "base_url": ""}
+        )
+
+        self.extractor = self.config["extractor"].as_str()
        if self.extractor:
            self.extractor = util.normpath(self.extractor)
-            # Expicit path to extractor
+            # Explicit path to extractor
            if not os.path.isfile(self.extractor):
                raise ui.UserError(
-                    'Extractor command does not exist: {0}.'.
-                    format(self.extractor)
+                    "Extractor command does not exist: {0}.".format(
+                        self.extractor
+                    )
                )
        else:
            # Implicit path to extractor, search for it in path
-            self.extractor = 'streaming_extractor_music'
+            self.extractor = "streaming_extractor_music"
            try:
                call([self.extractor])
            except OSError:
                raise ui.UserError(
-                    'No extractor command found: please install the extractor'
-                    ' binary from https://acousticbrainz.org/download'
+                    "No extractor command found: please install the extractor"
+                    " binary from https://essentia.upf.edu/"
                )
            except ABSubmitError:
                # Extractor found, will exit with an error if not called with
@@ -92,32 +90,54 @@ class AcousticBrainzSubmitPlugin(plugins.BeetsPlugin):

        # Calculate extractor hash.
        self.extractor_sha = hashlib.sha1()
-        with open(self.extractor, 'rb') as extractor:
+        with open(self.extractor, "rb") as extractor:
            self.extractor_sha.update(extractor.read())
        self.extractor_sha = self.extractor_sha.hexdigest()

-    base_url = 'https://acousticbrainz.org/api/v1/{mbid}/low-level'
+        self.url = ""
+        base_url = self.config["base_url"].as_str()
+        if base_url:
+            if not base_url.startswith("http"):
+                raise ui.UserError(
+                    "AcousticBrainz server base URL must start "
+                    "with an HTTP scheme"
+                )
+            elif base_url[-1] != "/":
+                base_url = base_url + "/"
+            self.url = base_url + "{mbid}/low-level"

    def commands(self):
        cmd = ui.Subcommand(
-            'absubmit',
-            help='calculate and submit AcousticBrainz analysis'
+            "absubmit", help="calculate and submit AcousticBrainz analysis"
        )
        cmd.parser.add_option(
-            '-f', '--force', dest='force_refetch',
-            action='store_true', default=False,
-            help='re-download data when already present'
+            "-f",
+            "--force",
+            dest="force_refetch",
+            action="store_true",
+            default=False,
+            help="re-download data when already present",
        )
        cmd.parser.add_option(
-            '-p', '--pretend', dest='pretend_fetch',
-            action='store_true', default=False,
-            help='pretend to perform action, but show \
-only files which would be processed'
+            "-p",
+            "--pretend",
+            dest="pretend_fetch",
+            action="store_true",
+            default=False,
+            help="pretend to perform action, but show \
+only files which would be processed",
        )
        cmd.func = self.command
        return [cmd]

    def command(self, lib, opts, args):
+        if not self.url:
+            raise ui.UserError(
+                "This plugin is deprecated since AcousticBrainz no longer "
+                "accepts new submissions. See the base_url configuration "
+                "option."
+            )
+        else:
            # Get items from arguments
            items = lib.items(ui.decargs(args))
            self.opts = opts
@@ -129,28 +149,29 @@ only files which would be processed'
            self._submit_data(item, analysis)

    def _get_analysis(self, item):
-        mbid = item['mb_trackid']
+        mbid = item["mb_trackid"]

        # Avoid re-analyzing files that already have AB data.
-        if not self.opts.force_refetch and not self.config['force']:
+        if not self.opts.force_refetch and not self.config["force"]:
            if item.get(PROBE_FIELD):
                return None

        # If file has no MBID, skip it.
        if not mbid:
-            self._log.info('Not analysing {}, missing '
-                           'musicbrainz track id.', item)
+            self._log.info(
+                "Not analysing {}, missing " "musicbrainz track id.", item
+            )
            return None

-        if self.opts.pretend_fetch or self.config['pretend']:
-            self._log.info('pretend action - extract item: {}', item)
+        if self.opts.pretend_fetch or self.config["pretend"]:
+            self._log.info("pretend action - extract item: {}", item)
            return None

        # Temporary file to save extractor output to, extractor only works
        # if an output file is given. Here we use a temporary file to copy
        # the data into a python object and then remove the file from the
        # system.
-        tmp_file, filename = tempfile.mkstemp(suffix='.json')
+        tmp_file, filename = tempfile.mkstemp(suffix=".json")
        try:
            # Close the file, so the extractor can overwrite it.
            os.close(tmp_file)
@@ -158,15 +179,17 @@ only files which would be processed'
                call([self.extractor, util.syspath(item.path), filename])
            except ABSubmitError as e:
                self._log.warning(
-                    'Failed to analyse {item} for AcousticBrainz: {error}',
-                    item=item, error=e
+                    "Failed to analyse {item} for AcousticBrainz: {error}",
+                    item=item,
+                    error=e,
                )
                return None
            with open(filename) as tmp_file:
                analysis = json.load(tmp_file)
            # Add the hash to the output.
-            analysis['metadata']['version']['essentia_build_sha'] = \
-                self.extractor_sha
+            analysis["metadata"]["version"][
+                "essentia_build_sha"
+            ] = self.extractor_sha
            return analysis
        finally:
            try:
@@ -177,20 +200,28 @@ only files which would be processed'
                    raise

    def _submit_data(self, item, data):
-        mbid = item['mb_trackid']
-        headers = {'Content-Type': 'application/json'}
-        response = requests.post(self.base_url.format(mbid=mbid),
-                                 json=data, headers=headers)
+        mbid = item["mb_trackid"]
+        headers = {"Content-Type": "application/json"}
+        response = requests.post(
+            self.url.format(mbid=mbid),
+            json=data,
+            headers=headers,
+            timeout=10,
+        )
        # Test that request was successful and raise an error on failure.
        if response.status_code != 200:
            try:
-                message = response.json()['message']
+                message = response.json()["message"]
            except (ValueError, KeyError) as e:
-                message = f'unable to get error message: {e}'
+                message = f"unable to get error message: {e}"
            self._log.error(
-                'Failed to submit AcousticBrainz analysis of {item}: '
-                '{message}).', item=item, message=message
+                "Failed to submit AcousticBrainz analysis of {item}: "
+                "{message}).",
+                item=item,
+                message=message,
            )
        else:
-            self._log.debug('Successfully submitted AcousticBrainz analysis '
-                            'for {}.', item)
+            self._log.debug(
+                "Successfully submitted AcousticBrainz analysis " "for {}.",
+                item,
+            )
@@ -22,220 +22,187 @@ import requests
 from beets import plugins, ui
 from beets.dbcore import types

-ACOUSTIC_BASE = "https://acousticbrainz.org/"
 LEVELS = ["/low-level", "/high-level"]
 ABSCHEME = {
-    'highlevel': {
-        'danceability': {
-            'all': {
-                'danceable': 'danceable'
-            }
+    "highlevel": {
+        "danceability": {"all": {"danceable": "danceable"}},
+        "gender": {"value": "gender"},
+        "genre_rosamerica": {"value": "genre_rosamerica"},
+        "mood_acoustic": {"all": {"acoustic": "mood_acoustic"}},
+        "mood_aggressive": {"all": {"aggressive": "mood_aggressive"}},
+        "mood_electronic": {"all": {"electronic": "mood_electronic"}},
+        "mood_happy": {"all": {"happy": "mood_happy"}},
+        "mood_party": {"all": {"party": "mood_party"}},
+        "mood_relaxed": {"all": {"relaxed": "mood_relaxed"}},
+        "mood_sad": {"all": {"sad": "mood_sad"}},
+        "moods_mirex": {"value": "moods_mirex"},
+        "ismir04_rhythm": {"value": "rhythm"},
+        "tonal_atonal": {"all": {"tonal": "tonal"}},
+        "timbre": {"value": "timbre"},
+        "voice_instrumental": {"value": "voice_instrumental"},
    },
-        'gender': {
-            'value': 'gender'
+    "lowlevel": {"average_loudness": "average_loudness"},
+    "rhythm": {"bpm": "bpm"},
+    "tonal": {
+        "chords_changes_rate": "chords_changes_rate",
+        "chords_key": "chords_key",
+        "chords_number_rate": "chords_number_rate",
+        "chords_scale": "chords_scale",
+        "key_key": ("initial_key", 0),
+        "key_scale": ("initial_key", 1),
+        "key_strength": "key_strength",
    },
-        'genre_rosamerica': {
-            'value': 'genre_rosamerica'
-        },
-        'mood_acoustic': {
-            'all': {
-                'acoustic': 'mood_acoustic'
-            }
-        },
-        'mood_aggressive': {
-            'all': {
-                'aggressive': 'mood_aggressive'
-            }
-        },
-        'mood_electronic': {
-            'all': {
-                'electronic': 'mood_electronic'
-            }
-        },
-        'mood_happy': {
-            'all': {
-                'happy': 'mood_happy'
-            }
-        },
-        'mood_party': {
-            'all': {
-                'party': 'mood_party'
-            }
-        },
-        'mood_relaxed': {
-            'all': {
-                'relaxed': 'mood_relaxed'
-            }
-        },
-        'mood_sad': {
-            'all': {
-                'sad': 'mood_sad'
-            }
-        },
-        'moods_mirex': {
-            'value': 'moods_mirex'
-        },
-        'ismir04_rhythm': {
-            'value': 'rhythm'
-        },
-        'tonal_atonal': {
-            'all': {
-                'tonal': 'tonal'
-            }
-        },
-        'timbre': {
-            'value': 'timbre'
-        },
-        'voice_instrumental': {
-            'value': 'voice_instrumental'
-        },
-    },
-    'lowlevel': {
-        'average_loudness': 'average_loudness'
-    },
-    'rhythm': {
-        'bpm': 'bpm'
-    },
-    'tonal': {
-        'chords_changes_rate': 'chords_changes_rate',
-        'chords_key': 'chords_key',
-        'chords_number_rate': 'chords_number_rate',
-        'chords_scale': 'chords_scale',
-        'key_key': ('initial_key', 0),
-        'key_scale': ('initial_key', 1),
-        'key_strength': 'key_strength'
-
-    }
 }


 class AcousticPlugin(plugins.BeetsPlugin):
    item_types = {
-        'average_loudness': types.Float(6),
-        'chords_changes_rate': types.Float(6),
-        'chords_key': types.STRING,
-        'chords_number_rate': types.Float(6),
-        'chords_scale': types.STRING,
-        'danceable': types.Float(6),
-        'gender': types.STRING,
-        'genre_rosamerica': types.STRING,
-        'initial_key': types.STRING,
-        'key_strength': types.Float(6),
-        'mood_acoustic': types.Float(6),
-        'mood_aggressive': types.Float(6),
-        'mood_electronic': types.Float(6),
-        'mood_happy': types.Float(6),
-        'mood_party': types.Float(6),
-        'mood_relaxed': types.Float(6),
-        'mood_sad': types.Float(6),
-        'moods_mirex': types.STRING,
-        'rhythm': types.Float(6),
-        'timbre': types.STRING,
-        'tonal': types.Float(6),
-        'voice_instrumental': types.STRING,
+        "average_loudness": types.Float(6),
+        "chords_changes_rate": types.Float(6),
+        "chords_key": types.STRING,
+        "chords_number_rate": types.Float(6),
+        "chords_scale": types.STRING,
+        "danceable": types.Float(6),
+        "gender": types.STRING,
+        "genre_rosamerica": types.STRING,
+        "initial_key": types.STRING,
+        "key_strength": types.Float(6),
+        "mood_acoustic": types.Float(6),
+        "mood_aggressive": types.Float(6),
+        "mood_electronic": types.Float(6),
+        "mood_happy": types.Float(6),
+        "mood_party": types.Float(6),
+        "mood_relaxed": types.Float(6),
+        "mood_sad": types.Float(6),
+        "moods_mirex": types.STRING,
+        "rhythm": types.Float(6),
+        "timbre": types.STRING,
+        "tonal": types.Float(6),
+        "voice_instrumental": types.STRING,
    }

    def __init__(self):
        super().__init__()

-        self.config.add({
-            'auto': True,
-            'force': False,
-            'tags': []
-        })
+        self._log.warning("This plugin is deprecated.")

-        if self.config['auto']:
-            self.register_listener('import_task_files',
-                                   self.import_task_files)
+        self.config.add(
+            {"auto": True, "force": False, "tags": [], "base_url": ""}
+        )
+
+        self.base_url = self.config["base_url"].as_str()
+        if self.base_url:
+            if not self.base_url.startswith("http"):
+                raise ui.UserError(
+                    "AcousticBrainz server base URL must start "
+                    "with an HTTP scheme"
+                )
+            elif self.base_url[-1] != "/":
+                self.base_url = self.base_url + "/"
+
+        if self.config["auto"]:
+            self.register_listener("import_task_files", self.import_task_files)

    def commands(self):
-        cmd = ui.Subcommand('acousticbrainz',
-                            help="fetch metadata from AcousticBrainz")
+        cmd = ui.Subcommand(
+            "acousticbrainz", help="fetch metadata from AcousticBrainz"
+        )
        cmd.parser.add_option(
-            '-f', '--force', dest='force_refetch',
-            action='store_true', default=False,
-            help='re-download data when already present'
+            "-f",
+            "--force",
+            dest="force_refetch",
+            action="store_true",
+            default=False,
+            help="re-download data when already present",
        )

        def func(lib, opts, args):
            items = lib.items(ui.decargs(args))
-            self._fetch_info(items, ui.should_write(),
-                             opts.force_refetch or self.config['force'])
+            self._fetch_info(
+                items,
+                ui.should_write(),
+                opts.force_refetch or self.config["force"],
+            )

        cmd.func = func
        return [cmd]

    def import_task_files(self, session, task):
-        """Function is called upon beet import.
-        """
+        """Function is called upon beet import."""
        self._fetch_info(task.imported_items(), False, True)

    def _get_data(self, mbid):
+        if not self.base_url:
+            raise ui.UserError(
+                "This plugin is deprecated since AcousticBrainz has shut "
+                "down. See the base_url configuration option."
+            )
        data = {}
-        for url in _generate_urls(mbid):
-            self._log.debug('fetching URL: {}', url)
+        for url in _generate_urls(self.base_url, mbid):
+            self._log.debug("fetching URL: {}", url)

            try:
-                res = requests.get(url)
+                res = requests.get(url, timeout=10)
            except requests.RequestException as exc:
-                self._log.info('request error: {}', exc)
+                self._log.info("request error: {}", exc)
                return {}

            if res.status_code == 404:
-                self._log.info('recording ID {} not found', mbid)
+                self._log.info("recording ID {} not found", mbid)
                return {}

            try:
                data.update(res.json())
            except ValueError:
-                self._log.debug('Invalid Response: {}', res.text)
+                self._log.debug("Invalid Response: {}", res.text)
                return {}

        return data

    def _fetch_info(self, items, write, force):
-        """Fetch additional information from AcousticBrainz for the `item`s.
-        """
-        tags = self.config['tags'].as_str_seq()
+        """Fetch additional information from AcousticBrainz for the `item`s."""
+        tags = self.config["tags"].as_str_seq()
        for item in items:
            # If we're not forcing re-downloading for all tracks, check
            # whether the data is already present. We use one
            # representative field name to check for previously fetched
            # data.
            if not force:
-                mood_str = item.get('mood_acoustic', '')
+                mood_str = item.get("mood_acoustic", "")
                if mood_str:
-                    self._log.info('data already present for: {}', item)
+                    self._log.info("data already present for: {}", item)
                    continue

            # We can only fetch data for tracks with MBIDs.
            if not item.mb_trackid:
                continue

-            self._log.info('getting data for: {}', item)
+            self._log.info("getting data for: {}", item)
            data = self._get_data(item.mb_trackid)
            if data:
                for attr, val in self._map_data_to_scheme(data, ABSCHEME):
                    if not tags or attr in tags:
-                        self._log.debug('attribute {} of {} set to {}',
-                                        attr,
-                                        item,
-                                        val)
+                        self._log.debug(
+                            "attribute {} of {} set to {}", attr, item, val
+                        )
                        setattr(item, attr, val)
                    else:
-                        self._log.debug('skipping attribute {} of {}'
-                                        ' (value {}) due to config',
+                        self._log.debug(
+                            "skipping attribute {} of {}"
+                            " (value {}) due to config",
                            attr,
                            item,
-                                        val)
+                            val,
+                        )
                item.store()
                if write:
                    item.try_write()

    def _map_data_to_scheme(self, data, scheme):
-        """Given `data` as a structure of nested dictionaries, and `scheme` as a
-        structure of nested dictionaries , `yield` tuples `(attr, val)` where
-        `attr` and `val` are corresponding leaf nodes in `scheme` and `data`.
+        """Given `data` as a structure of nested dictionaries, and
+        `scheme` as a structure of nested dictionaries , `yield` tuples
+        `(attr, val)` where `attr` and `val` are corresponding leaf
+        nodes in `scheme` and `data`.

        As its name indicates, `scheme` defines how the data is structured,
        so this function tries to find leaf nodes in `data` that correspond
@@ -286,14 +253,12 @@ class AcousticPlugin(plugins.BeetsPlugin):

        # The recursive traversal.
        composites = defaultdict(list)
-        yield from self._data_to_scheme_child(data,
-                                              scheme,
-                                              composites)
+        yield from self._data_to_scheme_child(data, scheme, composites)

        # When composites has been populated, yield the composite attributes
        # by joining their parts.
        for composite_attr, value_parts in composites.items():
-            yield composite_attr, ' '.join(value_parts)
+            yield composite_attr, " ".join(value_parts)

    def _data_to_scheme_child(self, subdata, subscheme, composites):
        """The recursive business logic of :meth:`_map_data_to_scheme`:
@@ -307,28 +272,33 @@ class AcousticPlugin(plugins.BeetsPlugin):
        """
        for k, v in subscheme.items():
            if k in subdata:
-                if type(v) == dict:
-                    yield from self._data_to_scheme_child(subdata[k],
-                                                          v,
-                                                          composites)
-                elif type(v) == tuple:
+                if isinstance(v, dict):
+                    yield from self._data_to_scheme_child(
+                        subdata[k], v, composites
+                    )
+                elif isinstance(v, tuple):
                    composite_attribute, part_number = v
                    attribute_parts = composites[composite_attribute]
                    # Parts are not guaranteed to be inserted in order
                    while len(attribute_parts) <= part_number:
-                        attribute_parts.append('')
+                        attribute_parts.append("")
                    attribute_parts[part_number] = subdata[k]
                else:
                    yield v, subdata[k]
            else:
-                self._log.warning('Acousticbrainz did not provide info'
-                                  'about {}', k)
-                self._log.debug('Data {} could not be mapped to scheme {} '
-                                'because key {} was not found', subdata, v, k)
+                self._log.warning(
+                    "Acousticbrainz did not provide info " "about {}", k
+                )
+                self._log.debug(
+                    "Data {} could not be mapped to scheme {} "
+                    "because key {} was not found",
+                    subdata,
+                    v,
+                    k,
+                )


-def _generate_urls(mbid):
-    """Generates AcousticBrainz end point urls for given `mbid`.
-    """
+def _generate_urls(base_url, mbid):
+    """Generates AcousticBrainz end point urls for given `mbid`."""
    for level in LEVELS:
-        yield ACOUSTIC_BASE + mbid + level
+        yield base_url + mbid + level
@@ -0,0 +1,174 @@
+# This file is part of beets.
+# Copyright 2023, Max Rumpf.
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+
+"""Plugin to rewrite fields based on a given query."""
+
+import re
+import shlex
+from collections import defaultdict
+
+import confuse
+
+from beets.dbcore import AndQuery, query_from_strings
+from beets.dbcore.types import MULTI_VALUE_DSV
+from beets.library import Album, Item
+from beets.plugins import BeetsPlugin
+from beets.ui import UserError
+
+
+def rewriter(field, simple_rules, advanced_rules):
+    """Template field function factory.
+
+    Create a template field function that rewrites the given field
+    with the given rewriting rules.
+    ``simple_rules`` must be a list of (pattern, replacement) pairs.
+    ``advanced_rules`` must be a list of (query, replacement) pairs.
+    """
+
+    def fieldfunc(item):
+        value = item._values_fixed[field]
+        for pattern, replacement in simple_rules:
+            if pattern.match(value.lower()):
+                # Rewrite activated.
+                return replacement
+        for query, replacement in advanced_rules:
+            if query.match(item):
+                # Rewrite activated.
+                return replacement
+        # Not activated; return original value.
+        return value
+
+    return fieldfunc
+
+
+class AdvancedRewritePlugin(BeetsPlugin):
+    """Plugin to rewrite fields based on a given query."""
+
+    def __init__(self):
+        """Parse configuration and register template fields for rewriting."""
+        super().__init__()
+
+        template = confuse.Sequence(
+            confuse.OneOf(
+                [
+                    confuse.MappingValues(str),
+                    {
+                        "match": str,
+                        "replacements": confuse.MappingValues(
+                            confuse.OneOf([str, confuse.Sequence(str)]),
+                        ),
+                    },
+                ]
+            )
+        )
+
+        # Used to apply the same rewrite to the corresponding album field.
+        corresponding_album_fields = {
+            "artist": "albumartist",
+            "artists": "albumartists",
+            "artist_sort": "albumartist_sort",
+            "artists_sort": "albumartists_sort",
+        }
+
+        # Gather all the rewrite rules for each field.
+        class RulesContainer:
+            def __init__(self):
+                self.simple = []
+                self.advanced = []
+
+        rules = defaultdict(RulesContainer)
+        for rule in self.config.get(template):
+            if "match" not in rule:
+                # Simple syntax
+                if len(rule) != 1:
+                    raise UserError(
+                        "Simple rewrites must have only one rule, "
+                        "but found multiple entries. "
+                        "Did you forget to prepend a dash (-)?"
+                    )
+                key, value = next(iter(rule.items()))
+                try:
+                    fieldname, pattern = key.split(None, 1)
+                except ValueError:
+                    raise UserError(
+                        f"Invalid simple rewrite specification {key}"
+                    )
+                if fieldname not in Item._fields:
+                    raise UserError(
+                        f"invalid field name {fieldname} in rewriter"
+                    )
+                self._log.debug(
+                    f"adding simple rewrite '{pattern}' → '{value}' "
+                    f"for field {fieldname}"
+                )
+                pattern = re.compile(pattern.lower())
+                rules[fieldname].simple.append((pattern, value))
+
+                # Apply the same rewrite to the corresponding album field.
+                if fieldname in corresponding_album_fields:
+                    album_fieldname = corresponding_album_fields[fieldname]
+                    rules[album_fieldname].simple.append((pattern, value))
+            else:
+                # Advanced syntax
+                match = rule["match"]
+                replacements = rule["replacements"]
+                if len(replacements) == 0:
+                    raise UserError(
+                        "Advanced rewrites must have at least one replacement"
+                    )
+                query = query_from_strings(
+                    AndQuery,
+                    Item,
+                    prefixes={},
+                    query_parts=shlex.split(match),
+                )
+                for fieldname, replacement in replacements.items():
+                    if fieldname not in Item._fields:
+                        raise UserError(
+                            f"Invalid field name {fieldname} in rewriter"
+                        )
+                    self._log.debug(
+                        f"adding advanced rewrite to '{replacement}' "
+                        f"for field {fieldname}"
+                    )
+                    if isinstance(replacement, list):
+                        if Item._fields[fieldname] is not MULTI_VALUE_DSV:
+                            raise UserError(
+                                f"Field {fieldname} is not a multi-valued field "
+                                f"but a list was given: {', '.join(replacement)}"
+                            )
+                    elif isinstance(replacement, str):
+                        if Item._fields[fieldname] is MULTI_VALUE_DSV:
+                            replacement = [replacement]
+                    else:
+                        raise UserError(
+                            f"Invalid type of replacement {replacement} "
+                            f"for field {fieldname}"
+                        )
+
+                    rules[fieldname].advanced.append((query, replacement))
+
+                    # Apply the same rewrite to the corresponding album field.
+                    if fieldname in corresponding_album_fields:
+                        album_fieldname = corresponding_album_fields[fieldname]
+                        rules[album_fieldname].advanced.append(
+                            (query, replacement)
+                        )
+
+        # Replace each template field with the new rewriter function.
+        for fieldname, fieldrules in rules.items():
+            getter = rewriter(fieldname, fieldrules.simple, fieldrules.advanced)
+            self.template_fields[fieldname] = getter
+            if fieldname in Album._fields:
+                self.album_template_fields[fieldname] = getter
@@ -26,40 +26,42 @@ class AlbumTypesPlugin(BeetsPlugin):
    def __init__(self):
        """Init AlbumTypesPlugin."""
        super().__init__()
-        self.album_template_fields['atypes'] = self._atypes
-        self.config.add({
-            'types': [
-                ('ep', 'EP'),
-                ('single', 'Single'),
-                ('soundtrack', 'OST'),
-                ('live', 'Live'),
-                ('compilation', 'Anthology'),
-                ('remix', 'Remix')
+        self.album_template_fields["atypes"] = self._atypes
+        self.config.add(
+            {
+                "types": [
+                    ("ep", "EP"),
+                    ("single", "Single"),
+                    ("soundtrack", "OST"),
+                    ("live", "Live"),
+                    ("compilation", "Anthology"),
+                    ("remix", "Remix"),
                ],
-            'ignore_va': ['compilation'],
-            'bracket': '[]'
-        })
+                "ignore_va": ["compilation"],
+                "bracket": "[]",
+            }
+        )

    def _atypes(self, item: Album):
        """Returns a formatted string based on album's types."""
-        types = self.config['types'].as_pairs()
-        ignore_va = self.config['ignore_va'].as_str_seq()
-        bracket = self.config['bracket'].as_str()
+        types = self.config["types"].as_pairs()
+        ignore_va = self.config["ignore_va"].as_str_seq()
+        bracket = self.config["bracket"].as_str()

        # Assign a left and right bracket or leave blank if argument is empty.
        if len(bracket) == 2:
            bracket_l = bracket[0]
            bracket_r = bracket[1]
        else:
-            bracket_l = ''
-            bracket_r = ''
+            bracket_l = ""
+            bracket_r = ""

-        res = ''
-        albumtypes = item.albumtypes.split('; ')
+        res = ""
+        albumtypes = item.albumtypes
        is_va = item.mb_albumartistid == VARIOUS_ARTISTS_ID
        for type in types:
            if type[0] in albumtypes and type[1]:
                if not is_va or (type[0] not in ignore_va and is_va):
-                    res += f'{bracket_l}{type[1]}{bracket_r}'
+                    res += f"{bracket_l}{type[1]}{bracket_r}"

        return res
@@ -15,35 +15,41 @@
 """An AURA server using Flask."""


-from mimetypes import guess_type
+import os
 import re
-import os.path
-from os.path import isfile, getsize
-
-from beets.plugins import BeetsPlugin
-from beets.ui import Subcommand, _open_library
-from beets import config
-from beets.util import py3_path
-from beets.library import Item, Album
-from beets.dbcore.query import (
-    MatchQuery,
-    NotQuery,
-    RegexpQuery,
-    AndQuery,
-    FixedFieldSort,
-    SlowFieldSort,
-    MultipleSort,
-)
+import sys
+from dataclasses import dataclass
+from mimetypes import guess_type
+from typing import ClassVar, Mapping, Type

 from flask import (
    Blueprint,
    Flask,
    current_app,
-    send_file,
    make_response,
    request,
+    send_file,
 )

+if sys.version_info >= (3, 11):
+    from typing import Self
+else:
+    from typing_extensions import Self
+
+from beets import config
+from beets.dbcore.query import (
+    AndQuery,
+    FixedFieldSort,
+    MatchQuery,
+    MultipleSort,
+    NotQuery,
+    RegexpQuery,
+    SlowFieldSort,
+    SQLiteType,
+)
+from beets.library import Album, Item, LibModel, Library
+from beets.plugins import BeetsPlugin
+from beets.ui import Subcommand, _open_library

 # Constants

@@ -118,9 +124,20 @@ ARTIST_ATTR_MAP = {
 }


+@dataclass
 class AURADocument:
    """Base class for building AURA documents."""

+    model_cls: ClassVar[Type[LibModel]]
+
+    lib: Library
+    args: Mapping[str, str]
+
+    @classmethod
+    def from_app(cls) -> Self:
+        """Initialise the document using the global app and request."""
+        return cls(current_app.config["lib"], request.args)
+
    @staticmethod
    def error(status, title, detail):
        """Make a response for an error following the JSON:API spec.
@@ -136,13 +153,29 @@ class AURADocument:
        }
        return make_response(document, status)

+    @classmethod
+    def get_attribute_converter(cls, beets_attr: str) -> Type[SQLiteType]:
+        """Work out what data type an attribute should be for beets.
+
+        Args:
+            beets_attr: The name of the beets attribute, e.g. "title".
+        """
+        try:
+            # Look for field in list of Album fields
+            # and get python type of database type.
+            # See beets.library.Album and beets.dbcore.types
+            return cls.model_cls._fields[beets_attr].model_type
+        except KeyError:
+            # Fall back to string (NOTE: probably not good)
+            return str
+
    def translate_filters(self):
        """Translate filters from request arguments to a beets Query."""
        # The format of each filter key in the request parameter is:
        # filter[<attribute>]. This regex extracts <attribute>.
        pattern = re.compile(r"filter\[(?P<attribute>[a-zA-Z0-9_-]+)\]")
        queries = []
-        for key, value in request.args.items():
+        for key, value in self.args.items():
            match = pattern.match(key)
            if match:
                # Extract attribute name from key
@@ -191,10 +224,10 @@ class AURADocument:
                albums) or a list of strings (artists).
        """
        # Pages start from zero
-        page = request.args.get("page", 0, int)
+        page = self.args.get("page", 0, int)
        # Use page limit defined in config by default.
        default_limit = config["aura"]["page_limit"].get(int)
-        limit = request.args.get("limit", default_limit, int)
+        limit = self.args.get("limit", default_limit, int)
        # start = offset of first item to return
        start = page * limit
        # end = offset of last item + 1
@@ -204,10 +237,10 @@ class AURADocument:
            next_url = None
        else:
            # Not the last page so work out links.next url
-            if not request.args:
+            if not self.args:
                # No existing arguments, so current page is 0
                next_url = request.url + "?page=1"
-            elif not request.args.get("page", None):
+            elif not self.args.get("page", None):
                # No existing page argument, so add one to the end
                next_url = request.url + "&page=1"
            else:
@@ -216,7 +249,10 @@ class AURADocument:
                    f"page={page}", "page={}".format(page + 1)
                )
        # Get only the items in the page range
-        data = [self.resource_object(collection[i]) for i in range(start, end)]
+        data = [
+            self.get_resource_object(self.lib, collection[i])
+            for i in range(start, end)
+        ]
        return data, next_url

    def get_included(self, data, include_str):
@@ -250,18 +286,26 @@ class AURADocument:
            res_type = identifier["type"]
            if res_type == "track":
                track_id = int(identifier["id"])
-                track = current_app.config["lib"].get_item(track_id)
-                included.append(TrackDocument.resource_object(track))
+                track = self.lib.get_item(track_id)
+                included.append(
+                    TrackDocument.get_resource_object(self.lib, track)
+                )
            elif res_type == "album":
                album_id = int(identifier["id"])
-                album = current_app.config["lib"].get_album(album_id)
-                included.append(AlbumDocument.resource_object(album))
+                album = self.lib.get_album(album_id)
+                included.append(
+                    AlbumDocument.get_resource_object(self.lib, album)
+                )
            elif res_type == "artist":
                artist_id = identifier["id"]
-                included.append(ArtistDocument.resource_object(artist_id))
+                included.append(
+                    ArtistDocument.get_resource_object(self.lib, artist_id)
+                )
            elif res_type == "image":
                image_id = identifier["id"]
-                included.append(ImageDocument.resource_object(image_id))
+                included.append(
+                    ImageDocument.get_resource_object(self.lib, image_id)
+                )
            else:
                raise ValueError(f"Invalid resource type: {res_type}")
        return included
@@ -269,7 +313,7 @@ class AURADocument:
    def all_resources(self):
        """Build document for /tracks, /albums or /artists."""
        query = self.translate_filters()
-        sort_arg = request.args.get("sort", None)
+        sort_arg = self.args.get("sort", None)
        if sort_arg:
            sort = self.translate_sorts(sort_arg)
            # For each sort field add a query which ensures all results
@@ -292,7 +336,7 @@ class AURADocument:
        if next_url:
            document["links"] = {"next": next_url}
        # Include related resources for each element in "data"
-        include_str = request.args.get("include", None)
+        include_str = self.args.get("include", None)
        if include_str:
            document["included"] = self.get_included(data, include_str)
        return document
@@ -305,7 +349,7 @@ class AURADocument:
                resource object.
        """
        document = {"data": resource_object}
-        include_str = request.args.get("include", None)
+        include_str = self.args.get("include", None)
        if include_str:
            # [document["data"]] is because arg needs to be list
            document["included"] = self.get_included(
@@ -317,6 +361,8 @@ class AURADocument:
 class TrackDocument(AURADocument):
    """Class for building documents for /tracks endpoints."""

+    model_cls = Item
+
    attribute_map = TRACK_ATTR_MAP

    def get_collection(self, query=None, sort=None):
@@ -326,9 +372,10 @@ class TrackDocument(AURADocument):
            query: A beets Query object or a beets query string.
            sort: A beets Sort object.
        """
-        return current_app.config["lib"].items(query, sort)
+        return self.lib.items(query, sort)

-    def get_attribute_converter(self, beets_attr):
+    @classmethod
+    def get_attribute_converter(cls, beets_attr: str) -> Type[SQLiteType]:
        """Work out what data type an attribute should be for beets.

        Args:
@@ -336,20 +383,12 @@ class TrackDocument(AURADocument):
        """
        # filesize is a special field (read from disk not db?)
        if beets_attr == "filesize":
-            converter = int
-        else:
-            try:
-                # Look for field in list of Item fields
-                # and get python type of database type.
-                # See beets.library.Item and beets.dbcore.types
-                converter = Item._fields[beets_attr].model_type
-            except KeyError:
-                # Fall back to string (NOTE: probably not good)
-                converter = str
-        return converter
+            return int
+
+        return super().get_attribute_converter(beets_attr)

    @staticmethod
-    def resource_object(track):
+    def get_resource_object(lib: Library, track):
        """Construct a JSON:API resource object from a beets Item.

        Args:
@@ -387,7 +426,7 @@ class TrackDocument(AURADocument):
        Args:
            track_id: The beets id of the track (integer).
        """
-        track = current_app.config["lib"].get_item(track_id)
+        track = self.lib.get_item(track_id)
        if not track:
            return self.error(
                "404 Not Found",
@@ -396,12 +435,16 @@ class TrackDocument(AURADocument):
                    track_id
                ),
            )
-        return self.single_resource_document(self.resource_object(track))
+        return self.single_resource_document(
+            self.get_resource_object(self.lib, track)
+        )


 class AlbumDocument(AURADocument):
    """Class for building documents for /albums endpoints."""

+    model_cls = Album
+
    attribute_map = ALBUM_ATTR_MAP

    def get_collection(self, query=None, sort=None):
@@ -411,26 +454,10 @@ class AlbumDocument(AURADocument):
            query: A beets Query object or a beets query string.
            sort: A beets Sort object.
        """
-        return current_app.config["lib"].albums(query, sort)
-
-    def get_attribute_converter(self, beets_attr):
-        """Work out what data type an attribute should be for beets.
-
-        Args:
-            beets_attr: The name of the beets attribute, e.g. "title".
-        """
-        try:
-            # Look for field in list of Album fields
-            # and get python type of database type.
-            # See beets.library.Album and beets.dbcore.types
-            converter = Album._fields[beets_attr].model_type
-        except KeyError:
-            # Fall back to string (NOTE: probably not good)
-            converter = str
-        return converter
+        return self.lib.albums(query, sort)

    @staticmethod
-    def resource_object(album):
+    def get_resource_object(lib: Library, album):
        """Construct a JSON:API resource object from a beets Album.

        Args:
@@ -449,7 +476,7 @@ class AlbumDocument(AURADocument):
        # track number. Sorting is not required but it's nice.
        query = MatchQuery("album_id", album.id)
        sort = FixedFieldSort("track", ascending=True)
-        tracks = current_app.config["lib"].items(query, sort)
+        tracks = lib.items(query, sort)
        # JSON:API one-to-many relationship to tracks on the album
        relationships = {
            "tracks": {
@@ -458,7 +485,7 @@ class AlbumDocument(AURADocument):
        }
        # Add images relationship if album has associated images
        if album.artpath:
-            path = py3_path(album.artpath)
+            path = os.fsdecode(album.artpath)
            filename = path.split("/")[-1]
            image_id = f"album-{album.id}-{filename}"
            relationships["images"] = {
@@ -485,7 +512,7 @@ class AlbumDocument(AURADocument):
        Args:
            album_id: The beets id of the album (integer).
        """
-        album = current_app.config["lib"].get_album(album_id)
+        album = self.lib.get_album(album_id)
        if not album:
            return self.error(
                "404 Not Found",
@@ -494,12 +521,16 @@ class AlbumDocument(AURADocument):
                    album_id
                ),
            )
-        return self.single_resource_document(self.resource_object(album))
+        return self.single_resource_document(
+            self.get_resource_object(self.lib, album)
+        )


 class ArtistDocument(AURADocument):
    """Class for building documents for /artists endpoints."""

+    model_cls = Item
+
    attribute_map = ARTIST_ATTR_MAP

    def get_collection(self, query=None, sort=None):
@@ -510,7 +541,7 @@ class ArtistDocument(AURADocument):
            sort: A beets Sort object.
        """
        # Gets only tracks with matching artist information
-        tracks = current_app.config["lib"].items(query, sort)
+        tracks = self.lib.items(query, sort)
        collection = []
        for track in tracks:
            # Do not add duplicates
@@ -518,24 +549,8 @@ class ArtistDocument(AURADocument):
                collection.append(track.artist)
        return collection

-    def get_attribute_converter(self, beets_attr):
-        """Work out what data type an attribute should be for beets.
-
-        Args:
-            beets_attr: The name of the beets attribute, e.g. "artist".
-        """
-        try:
-            # Look for field in list of Item fields
-            # and get python type of database type.
-            # See beets.library.Item and beets.dbcore.types
-            converter = Item._fields[beets_attr].model_type
-        except KeyError:
-            # Fall back to string (NOTE: probably not good)
-            converter = str
-        return converter
-
    @staticmethod
-    def resource_object(artist_id):
+    def get_resource_object(lib: Library, artist_id):
        """Construct a JSON:API resource object for the given artist.

        Args:
@@ -543,7 +558,7 @@ class ArtistDocument(AURADocument):
        """
        # Get tracks where artist field exactly matches artist_id
        query = MatchQuery("artist", artist_id)
-        tracks = current_app.config["lib"].items(query)
+        tracks = lib.items(query)
        if not tracks:
            return None

@@ -565,7 +580,7 @@ class ArtistDocument(AURADocument):
            }
        }
        album_query = MatchQuery("albumartist", artist_id)
-        albums = current_app.config["lib"].albums(query=album_query)
+        albums = lib.albums(query=album_query)
        if len(albums) != 0:
            relationships["albums"] = {
                "data": [{"type": "album", "id": str(a.id)} for a in albums]
@@ -584,7 +599,7 @@ class ArtistDocument(AURADocument):
        Args:
            artist_id: A string which is the artist's name.
        """
-        artist_resource = self.resource_object(artist_id)
+        artist_resource = self.get_resource_object(self.lib, artist_id)
        if not artist_resource:
            return self.error(
                "404 Not Found",
@@ -608,7 +623,7 @@ def safe_filename(fn):
        return False

    # In single names, rule out Unix directory traversal names.
-    if fn in ('.', '..'):
+    if fn in (".", ".."):
        return False

    return True
@@ -617,8 +632,10 @@ def safe_filename(fn):
 class ImageDocument(AURADocument):
    """Class for building documents for /images/(id) endpoints."""

+    model_cls = Album
+
    @staticmethod
-    def get_image_path(image_id):
+    def get_image_path(lib: Library, image_id):
        """Works out the full path to the image with the given id.

        Returns None if there is no such image.
@@ -640,13 +657,13 @@ class ImageDocument(AURADocument):

        # Get the path to the directory parent's images are in
        if parent_type == "album":
-            album = current_app.config["lib"].get_album(int(parent_id))
+            album = lib.get_album(int(parent_id))
            if not album or not album.artpath:
                return None
            # Cut the filename off of artpath
            # This is in preparation for supporting images in the same
            # directory that are not tracked by beets.
-            artpath = py3_path(album.artpath)
+            artpath = os.fsdecode(album.artpath)
            dir_path = "/".join(artpath.split("/")[:-1])
        else:
            # Images for other resource types are not supported
@@ -654,13 +671,13 @@ class ImageDocument(AURADocument):

        img_path = os.path.join(dir_path, img_filename)
        # Check the image actually exists
-        if isfile(img_path):
+        if os.path.isfile(img_path):
            return img_path
        else:
            return None

    @staticmethod
-    def resource_object(image_id):
+    def get_resource_object(lib: Library, image_id):
        """Construct a JSON:API resource object for the given image.

        Args:
@@ -669,14 +686,14 @@ class ImageDocument(AURADocument):
        """
        # Could be called as a static method, so can't use
        # self.get_image_path()
-        image_path = ImageDocument.get_image_path(image_id)
+        image_path = ImageDocument.get_image_path(lib, image_id)
        if not image_path:
            return None

        attributes = {
            "role": "cover",
            "mimetype": guess_type(image_path)[0],
-            "size": getsize(image_path),
+            "size": os.path.getsize(image_path),
        }
        try:
            from PIL import Image
@@ -709,7 +726,7 @@ class ImageDocument(AURADocument):
            image_id: A string in the form
                "<parent_type>-<parent_id>-<img_filename>".
        """
-        image_resource = self.resource_object(image_id)
+        image_resource = self.get_resource_object(self.lib, image_id)
        if not image_resource:
            return self.error(
                "404 Not Found",
@@ -737,8 +754,7 @@ def server_info():
@aura_bp.route("/tracks")
 def all_tracks():
    """Respond with a list of all tracks and related information."""
-    doc = TrackDocument()
-    return doc.all_resources()
+    return TrackDocument.from_app().all_resources()


@aura_bp.route("/tracks/<int:track_id>")
@@ -748,8 +764,7 @@ def single_track(track_id):
    Args:
        track_id: The id of the track provided in the URL (integer).
    """
-    doc = TrackDocument()
-    return doc.single_resource(track_id)
+    return TrackDocument.from_app().single_resource(track_id)


@aura_bp.route("/tracks/<int:track_id>/audio")
@@ -769,8 +784,8 @@ def audio_file(track_id):
            ),
        )

-    path = py3_path(track.path)
-    if not isfile(path):
+    path = os.fsdecode(track.path)
+    if not os.path.isfile(path):
        return AURADocument.error(
            "404 Not Found",
            "No audio file for the requested track.",
@@ -821,8 +836,7 @@ def audio_file(track_id):
@aura_bp.route("/albums")
 def all_albums():
    """Respond with a list of all albums and related information."""
-    doc = AlbumDocument()
-    return doc.all_resources()
+    return AlbumDocument.from_app().all_resources()


@aura_bp.route("/albums/<int:album_id>")
@@ -832,8 +846,7 @@ def single_album(album_id):
    Args:
        album_id: The id of the album provided in the URL (integer).
    """
-    doc = AlbumDocument()
-    return doc.single_resource(album_id)
+    return AlbumDocument.from_app().single_resource(album_id)


 # Artist endpoints
@@ -843,8 +856,7 @@ def single_album(album_id):
@aura_bp.route("/artists")
 def all_artists():
    """Respond with a list of all artists and related information."""
-    doc = ArtistDocument()
-    return doc.all_resources()
+    return ArtistDocument.from_app().all_resources()


 # Using the path converter allows slashes in artist_id
@@ -856,8 +868,7 @@ def single_artist(artist_id):
        artist_id: The id of the artist provided in the URL. A string
            which is the artist's name.
    """
-    doc = ArtistDocument()
-    return doc.single_resource(artist_id)
+    return ArtistDocument.from_app().single_resource(artist_id)


 # Image endpoints
@@ -873,8 +884,7 @@ def single_image(image_id):
        image_id: The id of the image provided in the URL. A string in
            the form "<parent_type>-<parent_id>-<img_filename>".
    """
-    doc = ImageDocument()
-    return doc.single_resource(image_id)
+    return ImageDocument.from_app().single_resource(image_id)


@aura_bp.route("/images/<string:image_id>/file")
@@ -885,7 +895,7 @@ def image_file(image_id):
        image_id: The id of the image provided in the URL. A string in
            the form "<parent_type>-<parent_id>-<img_filename>".
    """
-    img_path = ImageDocument.get_image_path(image_id)
+    img_path = ImageDocument.get_image_path(current_app.config["lib"], image_id)
    if not img_path:
        return AURADocument.error(
            "404 Not Found",
@@ -0,0 +1,92 @@
+# This file is part of beets.
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+
+"""Uses Librosa to calculate the `bpm` field.
+"""
+
+
+from librosa import beat, load
+from soundfile import LibsndfileError
+
+from beets import ui, util
+from beets.plugins import BeetsPlugin
+
+
+class AutoBPMPlugin(BeetsPlugin):
+    def __init__(self):
+        super().__init__()
+        self.config.add(
+            {
+                "auto": True,
+                "overwrite": False,
+            }
+        )
+
+        if self.config["auto"].get(bool):
+            self.import_stages = [self.imported]
+
+    def commands(self):
+        cmd = ui.Subcommand(
+            "autobpm", help="detect and add bpm from audio using Librosa"
+        )
+        cmd.func = self.command
+        return [cmd]
+
+    def command(self, lib, opts, args):
+        self.calculate_bpm(lib.items(ui.decargs(args)), write=ui.should_write())
+
+    def imported(self, session, task):
+        self.calculate_bpm(task.imported_items())
+
+    def calculate_bpm(self, items, write=False):
+        overwrite = self.config["overwrite"].get(bool)
+
+        for item in items:
+            if item["bpm"]:
+                self._log.info(
+                    "found bpm {0} for {1}",
+                    item["bpm"],
+                    util.displayable_path(item.path),
+                )
+                if not overwrite:
+                    continue
+
+            try:
+                y, sr = load(util.syspath(item.path), res_type="kaiser_fast")
+            except LibsndfileError as exc:
+                self._log.error(
+                    "LibsndfileError: failed to load {0} {1}",
+                    util.displayable_path(item.path),
+                    exc,
+                )
+                continue
+            except ValueError as exc:
+                self._log.error(
+                    "ValueError: failed to load {0} {1}",
+                    util.displayable_path(item.path),
+                    exc,
+                )
+                continue
+
+            tempo, _ = beat.beat_track(y=y, sr=sr)
+            bpm = round(tempo)
+            item["bpm"] = bpm
+            self._log.info(
+                "added computed bpm {0} for {1}",
+                bpm,
+                util.displayable_path(item.path),
+            )
+
+            if write:
+                item.try_write()
+            item.store()
@@ -16,18 +16,18 @@
 """


-from subprocess import check_output, CalledProcessError, list2cmdline, STDOUT
-
-import shlex
-import os
 import errno
+import os
+import shlex
 import sys
+from subprocess import STDOUT, CalledProcessError, check_output, list2cmdline
+
 import confuse
+
+from beets import importer, ui
 from beets.plugins import BeetsPlugin
 from beets.ui import Subcommand
 from beets.util import displayable_path, par_map
-from beets import ui
-from beets import importer


 class CheckerCommandException(Exception):
@@ -52,14 +52,15 @@ class BadFiles(BeetsPlugin):
        super().__init__()
        self.verbose = False

-        self.register_listener('import_task_start',
-                               self.on_import_task_start)
-        self.register_listener('import_task_before_choice',
-                               self.on_import_task_before_choice)
+        self.register_listener("import_task_start", self.on_import_task_start)
+        self.register_listener(
+            "import_task_before_choice", self.on_import_task_before_choice
+        )

    def run_command(self, cmd):
-        self._log.debug("running command: {}",
-                        displayable_path(list2cmdline(cmd)))
+        self._log.debug(
+            "running command: {}", displayable_path(list2cmdline(cmd))
+        )
        try:
            output = check_output(cmd, stderr=STDOUT)
            errors = 0
@@ -70,7 +71,7 @@ class BadFiles(BeetsPlugin):
            status = e.returncode
        except OSError as e:
            raise CheckerCommandException(cmd, e)
-        output = output.decode(sys.getdefaultencoding(), 'replace')
+        output = output.decode(sys.getdefaultencoding(), "replace")
        return status, errors, [line for line in output.split("\n") if line]

    def check_mp3val(self, path):
@@ -88,12 +89,13 @@ class BadFiles(BeetsPlugin):
            cmd = shlex.split(command)
            cmd.append(path)
            return self.run_command(cmd)
+
        return checker

    def get_checker(self, ext):
        ext = ext.lower()
        try:
-            command = self.config['commands'].get(dict).get(ext)
+            command = self.config["commands"].get(dict).get(ext)
        except confuse.NotFoundError:
            command = None
        if command:
@@ -109,15 +111,17 @@ class BadFiles(BeetsPlugin):
        dpath = displayable_path(item.path)
        self._log.debug("checking path: {}", dpath)
        if not os.path.exists(item.path):
-            ui.print_("{}: file does not exist".format(
-                ui.colorize('text_error', dpath)))
+            ui.print_(
+                "{}: file does not exist".format(
+                    ui.colorize("text_error", dpath)
+                )
+            )

        # Run the checker against the file if one is found
-        ext = os.path.splitext(item.path)[1][1:].decode('utf8', 'ignore')
+        ext = os.path.splitext(item.path)[1][1:].decode("utf8", "ignore")
        checker = self.get_checker(ext)
        if not checker:
-            self._log.error("no checker specified in the config for {}",
-                            ext)
+            self._log.error("no checker specified in the config for {}", ext)
            return []
        path = item.path
        if not isinstance(path, str):
@@ -129,7 +133,7 @@ class BadFiles(BeetsPlugin):
                self._log.error(
                    "command not found: {} when validating file: {}",
                    e.checker,
-                    e.path
+                    e.path,
                )
            else:
                self._log.error("error invoking {}: {}", e.checker, e.msg)
@@ -139,25 +143,30 @@ class BadFiles(BeetsPlugin):

        if status > 0:
            error_lines.append(
-                "{}: checker exited with status {}"
-                .format(ui.colorize('text_error', dpath), status))
+                "{}: checker exited with status {}".format(
+                    ui.colorize("text_error", dpath), status
+                )
+            )
            for line in output:
                error_lines.append(f"  {line}")

        elif errors > 0:
            error_lines.append(
-                    "{}: checker found {} errors or warnings"
-                    .format(ui.colorize('text_warning', dpath), errors))
+                "{}: checker found {} errors or warnings".format(
+                    ui.colorize("text_warning", dpath), errors
+                )
+            )
            for line in output:
                error_lines.append(f"  {line}")
        elif self.verbose:
            error_lines.append(
-                "{}: ok".format(ui.colorize('text_success', dpath)))
+                "{}: ok".format(ui.colorize("text_success", dpath))
+            )

        return error_lines

    def on_import_task_start(self, task, session):
-        if not self.config['check_on_import'].get(False):
+        if not self.config["check_on_import"].get(False):
            return

        checks_failed = []
@@ -171,26 +180,29 @@ class BadFiles(BeetsPlugin):
            task._badfiles_checks_failed = checks_failed

    def on_import_task_before_choice(self, task, session):
-        if hasattr(task, '_badfiles_checks_failed'):
-            ui.print_('{} one or more files failed checks:'
-                      .format(ui.colorize('text_warning', 'BAD')))
+        if hasattr(task, "_badfiles_checks_failed"):
+            ui.print_(
+                "{} one or more files failed checks:".format(
+                    ui.colorize("text_warning", "BAD")
+                )
+            )
            for error in task._badfiles_checks_failed:
                for error_line in error:
                    ui.print_(error_line)

            ui.print_()
-            ui.print_('What would you like to do?')
+            ui.print_("What would you like to do?")

-            sel = ui.input_options(['aBort', 'skip', 'continue'])
+            sel = ui.input_options(["aBort", "skip", "continue"])

-            if sel == 's':
+            if sel == "s":
                return importer.action.SKIP
-            elif sel == 'c':
+            elif sel == "c":
                return None
-            elif sel == 'b':
+            elif sel == "b":
                raise importer.ImportAbort()
            else:
-                raise Exception(f'Unexpected selection: {sel}')
+                raise Exception(f"Unexpected selection: {sel}")

    def command(self, lib, opts, args):
        # Get items from arguments
@@ -204,12 +216,16 @@ class BadFiles(BeetsPlugin):
        par_map(check_and_print, items)

    def commands(self):
-        bad_command = Subcommand('bad',
-                                 help='check for corrupt or missing files')
+        bad_command = Subcommand(
+            "bad", help="check for corrupt or missing files"
+        )
        bad_command.parser.add_option(
-            '-v', '--verbose',
-            action='store_true', default=False, dest='verbose',
-            help='view results for both the bad and uncorrupted files'
+            "-v",
+            "--verbose",
+            action="store_true",
+            default=False,
+            dest="verbose",
+            help="view results for both the bad and uncorrupted files",
        )
        bad_command.func = self.command
        return [bad_command]
@@ -19,15 +19,17 @@
 """Provides a bare-ASCII matching query."""


-from beets import ui
-from beets.ui import print_, decargs
-from beets.plugins import BeetsPlugin
-from beets.dbcore.query import StringFieldQuery
 from unidecode import unidecode

+from beets import ui
+from beets.dbcore.query import StringFieldQuery
+from beets.plugins import BeetsPlugin
+from beets.ui import decargs, print_

-class BareascQuery(StringFieldQuery):
+
+class BareascQuery(StringFieldQuery[str]):
    """Compare items using bare ASCII, without accents etc."""
+
    @classmethod
    def string_match(cls, pattern, val):
        """Convert both pattern and string to plain ASCII before matching.
@@ -42,27 +44,40 @@ class BareascQuery(StringFieldQuery):
        val = unidecode(val)
        return pattern in val

+    def col_clause(self):
+        """Compare ascii version of the pattern."""
+        clause = f"unidecode({self.field})"
+        if self.pattern.islower():
+            clause = f"lower({clause})"
+
+        return rf"{clause} LIKE ? ESCAPE '\'", [f"%{unidecode(self.pattern)}%"]
+

 class BareascPlugin(BeetsPlugin):
    """Plugin to provide bare-ASCII option for beets matching."""
+
    def __init__(self):
        """Default prefix for selecting bare-ASCII matching is #."""
        super().__init__()
-        self.config.add({
-            'prefix': '#',
-        })
+        self.config.add(
+            {
+                "prefix": "#",
+            }
+        )

    def queries(self):
        """Register bare-ASCII matching."""
-        prefix = self.config['prefix'].as_str()
+        prefix = self.config["prefix"].as_str()
        return {prefix: BareascQuery}

    def commands(self):
        """Add bareasc command as unidecode version of 'list'."""
-        cmd = ui.Subcommand('bareasc',
-                            help='unidecode version of beet list command')
-        cmd.parser.usage += "\n" \
-            'Example: %prog -f \'$album: $title\' artist:beatles'
+        cmd = ui.Subcommand(
+            "bareasc", help="unidecode version of beet list command"
+        )
+        cmd.parser.usage += (
+            "\n" "Example: %prog -f '$album: $title' artist:beatles"
+        )
        cmd.parser.add_all_common_options()
        cmd.func = self.unidecode_list
        return [cmd]
@@ -19,19 +19,22 @@ import json
 import re
 from datetime import datetime, timedelta

+import confuse
 from requests_oauthlib import OAuth1Session
-from requests_oauthlib.oauth1_session import (TokenRequestDenied, TokenMissing,
-                                              VerifierMissing)
+from requests_oauthlib.oauth1_session import (
+    TokenMissing,
+    TokenRequestDenied,
+    VerifierMissing,
+)

 import beets
 import beets.ui
 from beets.autotag.hooks import AlbumInfo, TrackInfo
 from beets.plugins import BeetsPlugin, MetadataSourcePlugin, get_distance
-import confuse
-
+from beets.util.id_extractors import beatport_id_regex

 AUTH_ERRORS = (TokenRequestDenied, TokenMissing, VerifierMissing)
-USER_AGENT = f'beets/{beets.__version__} +https://beets.io/'
+USER_AGENT = f"beets/{beets.__version__} +https://beets.io/"


 class BeatportAPIError(Exception):
@@ -40,21 +43,20 @@ class BeatportAPIError(Exception):

 class BeatportObject:
    def __init__(self, data):
-        self.beatport_id = data['id']
-        self.name = str(data['name'])
-        if 'releaseDate' in data:
-            self.release_date = datetime.strptime(data['releaseDate'],
-                                                  '%Y-%m-%d')
-        if 'artists' in data:
-            self.artists = [(x['id'], str(x['name']))
-                            for x in data['artists']]
-        if 'genres' in data:
-            self.genres = [str(x['name'])
-                           for x in data['genres']]
+        self.beatport_id = data["id"]
+        self.name = str(data["name"])
+        if "releaseDate" in data:
+            self.release_date = datetime.strptime(
+                data["releaseDate"], "%Y-%m-%d"
+            )
+        if "artists" in data:
+            self.artists = [(x["id"], str(x["name"])) for x in data["artists"]]
+        if "genres" in data:
+            self.genres = [str(x["name"]) for x in data["genres"]]


 class BeatportClient:
-    _api_base = 'https://oauth-api.beatport.com'
+    _api_base = "https://oauth-api.beatport.com"

    def __init__(self, c_key, c_secret, auth_key=None, auth_secret=None):
        """Initiate the client with OAuth information.
@@ -69,11 +71,13 @@ class BeatportClient:
        :param auth_secret: OAuth1 resource owner secret
        """
        self.api = OAuth1Session(
-            client_key=c_key, client_secret=c_secret,
+            client_key=c_key,
+            client_secret=c_secret,
            resource_owner_key=auth_key,
            resource_owner_secret=auth_secret,
-            callback_uri='oob')
-        self.api.headers = {'User-Agent': USER_AGENT}
+            callback_uri="oob",
+        )
+        self.api.headers = {"User-Agent": USER_AGENT}

    def get_authorize_url(self):
        """Generate the URL for the user to authorize the application.
@@ -91,9 +95,11 @@ class BeatportClient:
        :rtype:     unicode
        """
        self.api.fetch_request_token(
-            self._make_url('/identity/1/oauth/request-token'))
+            self._make_url("/identity/1/oauth/request-token")
+        )
        return self.api.authorization_url(
-            self._make_url('/identity/1/oauth/authorize'))
+            self._make_url("/identity/1/oauth/authorize")
+        )

    def get_access_token(self, auth_data):
        """Obtain the final access token and secret for the API.
@@ -106,12 +112,14 @@ class BeatportClient:
        :rtype:             (unicode, unicode) tuple
        """
        self.api.parse_authorization_response(
-            "https://beets.io/auth?" + auth_data)
+            "https://beets.io/auth?" + auth_data
+        )
        access_data = self.api.fetch_access_token(
-            self._make_url('/identity/1/oauth/access-token'))
-        return access_data['oauth_token'], access_data['oauth_token_secret']
+            self._make_url("/identity/1/oauth/access-token")
+        )
+        return access_data["oauth_token"], access_data["oauth_token_secret"]

-    def search(self, query, release_type='release', details=True):
+    def search(self, query, release_type="release", details=True):
        """Perform a search of the Beatport catalogue.

        :param query:           Query string
@@ -126,17 +134,20 @@ class BeatportClient:
                                py:class:`BeatportRelease` or
                                :py:class:`BeatportTrack`
        """
-        response = self._get('catalog/3/search',
-                             query=query, perPage=5,
-                             facets=[f'fieldType:{release_type}'])
+        response = self._get(
+            "catalog/3/search",
+            query=query,
+            perPage=5,
+            facets=[f"fieldType:{release_type}"],
+        )
        for item in response:
-            if release_type == 'release':
+            if release_type == "release":
                if details:
-                    release = self.get_release(item['id'])
+                    release = self.get_release(item["id"])
                else:
                    release = BeatportRelease(item)
                yield release
-            elif release_type == 'track':
+            elif release_type == "track":
                yield BeatportTrack(item)

    def get_release(self, beatport_id):
@@ -146,7 +157,7 @@ class BeatportClient:
        :returns:               The matching release
        :rtype:                 :py:class:`BeatportRelease`
        """
-        response = self._get('/catalog/3/releases', id=beatport_id)
+        response = self._get("/catalog/3/releases", id=beatport_id)
        if response:
            release = BeatportRelease(response[0])
            release.tracks = self.get_release_tracks(beatport_id)
@@ -160,8 +171,9 @@ class BeatportClient:
        :returns:               Tracks in the matching release
        :rtype:                 list of :py:class:`BeatportTrack`
        """
-        response = self._get('/catalog/3/tracks', releaseId=beatport_id,
-                             perPage=100)
+        response = self._get(
+            "/catalog/3/tracks", releaseId=beatport_id, perPage=100
+        )
        return [BeatportTrack(t) for t in response]

    def get_track(self, beatport_id):
@@ -171,13 +183,13 @@ class BeatportClient:
        :returns:               The matching track
        :rtype:                 :py:class:`BeatportTrack`
        """
-        response = self._get('/catalog/3/tracks', id=beatport_id)
+        response = self._get("/catalog/3/tracks", id=beatport_id)
        return BeatportTrack(response[0])

    def _make_url(self, endpoint):
        """Get complete URL for a given API endpoint."""
-        if not endpoint.startswith('/'):
-            endpoint = '/' + endpoint
+        if not endpoint.startswith("/"):
+            endpoint = "/" + endpoint
        return self._api_base + endpoint

    def _get(self, endpoint, **kwargs):
@@ -189,13 +201,16 @@ class BeatportClient:
        try:
            response = self.api.get(self._make_url(endpoint), params=kwargs)
        except Exception as e:
-            raise BeatportAPIError("Error connecting to Beatport API: {}"
-                                   .format(e))
+            raise BeatportAPIError(
+                "Error connecting to Beatport API: {}".format(e)
+            )
        if not response:
            raise BeatportAPIError(
-                "Error {0.status_code} for '{0.request.path_url}"
-                .format(response))
-        return response.json()['results']
+                "Error {0.status_code} for '{0.request.path_url}".format(
+                    response
+                )
+            )
+        return response.json()["results"]


 class BeatportRelease(BeatportObject):
@@ -211,79 +226,83 @@ class BeatportRelease(BeatportObject):
        )

    def __repr__(self):
-        return str(self).encode('utf-8')
+        return str(self).encode("utf-8")

    def __init__(self, data):
        BeatportObject.__init__(self, data)
-        if 'catalogNumber' in data:
-            self.catalog_number = data['catalogNumber']
-        if 'label' in data:
-            self.label_name = data['label']['name']
-        if 'category' in data:
-            self.category = data['category']
-        if 'slug' in data:
+        if "catalogNumber" in data:
+            self.catalog_number = data["catalogNumber"]
+        if "label" in data:
+            self.label_name = data["label"]["name"]
+        if "category" in data:
+            self.category = data["category"]
+        if "slug" in data:
            self.url = "https://beatport.com/release/{}/{}".format(
-                data['slug'], data['id'])
-        self.genre = data.get('genre')
+                data["slug"], data["id"]
+            )
+        self.genre = data.get("genre")


 class BeatportTrack(BeatportObject):
    def __str__(self):
        artist_str = ", ".join(x[1] for x in self.artists)
-        return ("<BeatportTrack: {} - {} ({})>"
-                .format(artist_str, self.name, self.mix_name))
+        return "<BeatportTrack: {} - {} ({})>".format(
+            artist_str, self.name, self.mix_name
+        )

    def __repr__(self):
-        return str(self).encode('utf-8')
+        return str(self).encode("utf-8")

    def __init__(self, data):
        BeatportObject.__init__(self, data)
-        if 'title' in data:
-            self.title = str(data['title'])
-        if 'mixName' in data:
-            self.mix_name = str(data['mixName'])
-        self.length = timedelta(milliseconds=data.get('lengthMs', 0) or 0)
+        if "title" in data:
+            self.title = str(data["title"])
+        if "mixName" in data:
+            self.mix_name = str(data["mixName"])
+        self.length = timedelta(milliseconds=data.get("lengthMs", 0) or 0)
        if not self.length:
            try:
-                min, sec = data.get('length', '0:0').split(':')
+                min, sec = data.get("length", "0:0").split(":")
                self.length = timedelta(minutes=int(min), seconds=int(sec))
            except ValueError:
                pass
-        if 'slug' in data:
-            self.url = "https://beatport.com/track/{}/{}" \
-                .format(data['slug'], data['id'])
-        self.track_number = data.get('trackNumber')
-        self.bpm = data.get('bpm')
-        self.initial_key = str(
-            (data.get('key') or {}).get('shortName')
+        if "slug" in data:
+            self.url = "https://beatport.com/track/{}/{}".format(
+                data["slug"], data["id"]
            )
+        self.track_number = data.get("trackNumber")
+        self.bpm = data.get("bpm")
+        self.initial_key = str((data.get("key") or {}).get("shortName"))

        # Use 'subgenre' and if not present, 'genre' as a fallback.
-        if data.get('subGenres'):
-            self.genre = str(data['subGenres'][0].get('name'))
-        elif data.get('genres'):
-            self.genre = str(data['genres'][0].get('name'))
+        if data.get("subGenres"):
+            self.genre = str(data["subGenres"][0].get("name"))
+        elif data.get("genres"):
+            self.genre = str(data["genres"][0].get("name"))


 class BeatportPlugin(BeetsPlugin):
-    data_source = 'Beatport'
+    data_source = "Beatport"
+    id_regex = beatport_id_regex

    def __init__(self):
        super().__init__()
-        self.config.add({
-            'apikey': '57713c3906af6f5def151b33601389176b37b429',
-            'apisecret': 'b3fe08c93c80aefd749fe871a16cd2bb32e2b954',
-            'tokenfile': 'beatport_token.json',
-            'source_weight': 0.5,
-        })
-        self.config['apikey'].redact = True
-        self.config['apisecret'].redact = True
+        self.config.add(
+            {
+                "apikey": "57713c3906af6f5def151b33601389176b37b429",
+                "apisecret": "b3fe08c93c80aefd749fe871a16cd2bb32e2b954",
+                "tokenfile": "beatport_token.json",
+                "source_weight": 0.5,
+            }
+        )
+        self.config["apikey"].redact = True
+        self.config["apisecret"].redact = True
        self.client = None
-        self.register_listener('import_begin', self.setup)
+        self.register_listener("import_begin", self.setup)

    def setup(self, session=None):
-        c_key = self.config['apikey'].as_str()
-        c_secret = self.config['apisecret'].as_str()
+        c_key = self.config["apikey"].as_str()
+        c_secret = self.config["apisecret"].as_str()

        # Get the OAuth token from a file or log in.
        try:
@@ -293,8 +312,8 @@ class BeatportPlugin(BeetsPlugin):
            # No token yet. Generate one.
            token, secret = self.authenticate(c_key, c_secret)
        else:
-            token = tokendata['token']
-            secret = tokendata['secret']
+            token = tokendata["token"]
+            secret = tokendata["secret"]

        self.client = BeatportClient(c_key, c_secret, token, secret)

@@ -304,8 +323,8 @@ class BeatportPlugin(BeetsPlugin):
        try:
            url = auth_client.get_authorize_url()
        except AUTH_ERRORS as e:
-            self._log.debug('authentication error: {0}', e)
-            raise beets.ui.UserError('communication with Beatport failed')
+            self._log.debug("authentication error: {0}", e)
+            raise beets.ui.UserError("communication with Beatport failed")

        beets.ui.print_("To authenticate with Beatport, visit:")
        beets.ui.print_(url)
@@ -315,29 +334,26 @@ class BeatportPlugin(BeetsPlugin):
        try:
            token, secret = auth_client.get_access_token(data)
        except AUTH_ERRORS as e:
-            self._log.debug('authentication error: {0}', e)
-            raise beets.ui.UserError('Beatport token request failed')
+            self._log.debug("authentication error: {0}", e)
+            raise beets.ui.UserError("Beatport token request failed")

        # Save the token for later use.
-        self._log.debug('Beatport token {0}, secret {1}', token, secret)
-        with open(self._tokenfile(), 'w') as f:
-            json.dump({'token': token, 'secret': secret}, f)
+        self._log.debug("Beatport token {0}, secret {1}", token, secret)
+        with open(self._tokenfile(), "w") as f:
+            json.dump({"token": token, "secret": secret}, f)

        return token, secret

    def _tokenfile(self):
-        """Get the path to the JSON file for storing the OAuth token.
-        """
-        return self.config['tokenfile'].get(confuse.Filename(in_app_dir=True))
+        """Get the path to the JSON file for storing the OAuth token."""
+        return self.config["tokenfile"].get(confuse.Filename(in_app_dir=True))

    def album_distance(self, items, album_info, mapping):
        """Returns the Beatport source weight and the maximum source weight
        for albums.
        """
        return get_distance(
-            data_source=self.data_source,
-            info=album_info,
-            config=self.config
+            data_source=self.data_source, info=album_info, config=self.config
        )

    def track_distance(self, item, track_info):
@@ -345,9 +361,7 @@ class BeatportPlugin(BeetsPlugin):
        for individual tracks.
        """
        return get_distance(
-            data_source=self.data_source,
-            info=track_info,
-            config=self.config
+            data_source=self.data_source, info=track_info, config=self.config
        )

    def candidates(self, items, artist, release, va_likely, extra_tags=None):
@@ -357,34 +371,36 @@ class BeatportPlugin(BeetsPlugin):
        if va_likely:
            query = release
        else:
-            query = f'{artist} {release}'
+            query = f"{artist} {release}"
        try:
            return self._get_releases(query)
        except BeatportAPIError as e:
-            self._log.debug('API Error: {0} (query: {1})', e, query)
+            self._log.debug("API Error: {0} (query: {1})", e, query)
            return []

    def item_candidates(self, item, artist, title):
        """Returns a list of TrackInfo objects for beatport search results
        matching title and artist.
        """
-        query = f'{artist} {title}'
+        query = f"{artist} {title}"
        try:
            return self._get_tracks(query)
        except BeatportAPIError as e:
-            self._log.debug('API Error: {0} (query: {1})', e, query)
+            self._log.debug("API Error: {0} (query: {1})", e, query)
            return []

    def album_for_id(self, release_id):
        """Fetches a release by its Beatport ID and returns an AlbumInfo object
        or None if the query is not a valid ID or release is not found.
        """
-        self._log.debug('Searching for release {0}', release_id)
-        match = re.search(r'(^|beatport\.com/release/.+/)(\d+)$', release_id)
-        if not match:
-            self._log.debug('Not a valid Beatport release ID.')
+        self._log.debug("Searching for release {0}", release_id)
+
+        release_id = self._get_id("album", release_id, self.id_regex)
+        if release_id is None:
+            self._log.debug("Not a valid Beatport release ID.")
            return None
-        release = self.client.get_release(match.group(2))
+
+        release = self.client.get_release(release_id)
        if release:
            return self._get_album_info(release)
        return None
@@ -393,10 +409,10 @@ class BeatportPlugin(BeetsPlugin):
        """Fetches a track by its Beatport ID and returns a TrackInfo object
        or None if the track is not a valid Beatport ID or track is not found.
        """
-        self._log.debug('Searching for track {0}', track_id)
-        match = re.search(r'(^|beatport\.com/track/.+/)(\d+)$', track_id)
+        self._log.debug("Searching for track {0}", track_id)
+        match = re.search(r"(^|beatport\.com/track/.+/)(\d+)$", track_id)
        if not match:
-            self._log.debug('Not a valid Beatport track ID.')
+            self._log.debug("Not a valid Beatport track ID.")
            return None
        bp_track = self.client.get_track(match.group(2))
        if bp_track is not None:
@@ -404,55 +420,67 @@ class BeatportPlugin(BeetsPlugin):
        return None

    def _get_releases(self, query):
-        """Returns a list of AlbumInfo objects for a beatport search query.
-        """
+        """Returns a list of AlbumInfo objects for a beatport search query."""
        # Strip non-word characters from query. Things like "!" and "-" can
        # cause a query to return no results, even if they match the artist or
        # album title. Use `re.UNICODE` flag to avoid stripping non-english
        # word characters.
-        query = re.sub(r'\W+', ' ', query, flags=re.UNICODE)
+        query = re.sub(r"\W+", " ", query, flags=re.UNICODE)
        # Strip medium information from query, Things like "CD1" and "disk 1"
        # can also negate an otherwise positive result.
-        query = re.sub(r'\b(CD|disc)\s*\d+', '', query, flags=re.I)
-        albums = [self._get_album_info(x)
-                  for x in self.client.search(query)]
+        query = re.sub(r"\b(CD|disc)\s*\d+", "", query, flags=re.I)
+        albums = [self._get_album_info(x) for x in self.client.search(query)]
        return albums

    def _get_album_info(self, release):
-        """Returns an AlbumInfo object for a Beatport Release object.
-        """
+        """Returns an AlbumInfo object for a Beatport Release object."""
        va = len(release.artists) > 3
        artist, artist_id = self._get_artist(release.artists)
        if va:
            artist = "Various Artists"
        tracks = [self._get_track_info(x) for x in release.tracks]

-        return AlbumInfo(album=release.name, album_id=release.beatport_id,
-                         artist=artist, artist_id=artist_id, tracks=tracks,
-                         albumtype=release.category, va=va,
+        return AlbumInfo(
+            album=release.name,
+            album_id=release.beatport_id,
+            beatport_album_id=release.beatport_id,
+            artist=artist,
+            artist_id=artist_id,
+            tracks=tracks,
+            albumtype=release.category,
+            va=va,
            year=release.release_date.year,
            month=release.release_date.month,
            day=release.release_date.day,
            label=release.label_name,
-                         catalognum=release.catalog_number, media='Digital',
-                         data_source=self.data_source, data_url=release.url,
-                         genre=release.genre)
+            catalognum=release.catalog_number,
+            media="Digital",
+            data_source=self.data_source,
+            data_url=release.url,
+            genre=release.genre,
+        )

    def _get_track_info(self, track):
-        """Returns a TrackInfo object for a Beatport Track object.
-        """
+        """Returns a TrackInfo object for a Beatport Track object."""
        title = track.name
        if track.mix_name != "Original Mix":
            title += f" ({track.mix_name})"
        artist, artist_id = self._get_artist(track.artists)
        length = track.length.total_seconds()
-        return TrackInfo(title=title, track_id=track.beatport_id,
-                         artist=artist, artist_id=artist_id,
-                         length=length, index=track.track_number,
+        return TrackInfo(
+            title=title,
+            track_id=track.beatport_id,
+            artist=artist,
+            artist_id=artist_id,
+            length=length,
+            index=track.track_number,
            medium_index=track.track_number,
-                         data_source=self.data_source, data_url=track.url,
-                         bpm=track.bpm, initial_key=track.initial_key,
-                         genre=track.genre)
+            data_source=self.data_source,
+            data_url=track.url,
+            bpm=track.bpm,
+            initial_key=track.initial_key,
+            genre=track.genre,
+        )

    def _get_artist(self, artists):
        """Returns an artist string (all artists) and an artist_id (the main
@@ -463,8 +491,7 @@ class BeatportPlugin(BeetsPlugin):
        )

    def _get_tracks(self, query):
-        """Returns a list of TrackInfo objects for a Beatport query.
-        """
-        bp_tracks = self.client.search(query, release_type='track')
+        """Returns a list of TrackInfo objects for a Beatport query."""
+        bp_tracks = self.client.search(query, release_type="track")
        tracks = [self._get_track_info(x) for x in bp_tracks]
        return tracks
@@ -16,17 +16,14 @@
 """


-from beets.plugins import BeetsPlugin
-from beets import ui
-from beets import vfs
-from beets import library
-from beets.util.functemplate import Template
-from beets.autotag import match
-from beets import plugins
-from beets import importer
 import cProfile
 import timeit

+from beets import importer, library, plugins, ui, vfs
+from beets.autotag import match
+from beets.plugins import BeetsPlugin
+from beets.util.functemplate import Template
+

 def aunique_benchmark(lib, prof):
    def _build_tree():
@@ -34,74 +31,103 @@ def aunique_benchmark(lib, prof):

    # Measure path generation performance with %aunique{} included.
    lib.path_formats = [
-        (library.PF_KEY_DEFAULT,
-         Template('$albumartist/$album%aunique{}/$track $title')),
+        (
+            library.PF_KEY_DEFAULT,
+            Template("$albumartist/$album%aunique{}/$track $title"),
+        ),
    ]
    if prof:
-        cProfile.runctx('_build_tree()', {}, {'_build_tree': _build_tree},
-                        'paths.withaunique.prof')
+        cProfile.runctx(
+            "_build_tree()",
+            {},
+            {"_build_tree": _build_tree},
+            "paths.withaunique.prof",
+        )
    else:
        interval = timeit.timeit(_build_tree, number=1)
-        print('With %aunique:', interval)
+        print("With %aunique:", interval)

    # And with %aunique replaceed with a "cheap" no-op function.
    lib.path_formats = [
-        (library.PF_KEY_DEFAULT,
-         Template('$albumartist/$album%lower{}/$track $title')),
+        (
+            library.PF_KEY_DEFAULT,
+            Template("$albumartist/$album%lower{}/$track $title"),
+        ),
    ]
    if prof:
-        cProfile.runctx('_build_tree()', {}, {'_build_tree': _build_tree},
-                        'paths.withoutaunique.prof')
+        cProfile.runctx(
+            "_build_tree()",
+            {},
+            {"_build_tree": _build_tree},
+            "paths.withoutaunique.prof",
+        )
    else:
        interval = timeit.timeit(_build_tree, number=1)
-        print('Without %aunique:', interval)
+        print("Without %aunique:", interval)


 def match_benchmark(lib, prof, query=None, album_id=None):
    # If no album ID is provided, we'll match against a suitably huge
    # album.
    if not album_id:
-        album_id = '9c5c043e-bc69-4edb-81a4-1aaf9c81e6dc'
+        album_id = "9c5c043e-bc69-4edb-81a4-1aaf9c81e6dc"

    # Get an album from the library to use as the source for the match.
    items = lib.albums(query).get().items()

    # Ensure fingerprinting is invoked (if enabled).
-    plugins.send('import_task_start',
+    plugins.send(
+        "import_task_start",
        task=importer.ImportTask(None, None, items),
-                 session=importer.ImportSession(lib, None, None, None))
+        session=importer.ImportSession(lib, None, None, None),
+    )

    # Run the match.
    def _run_match():
        match.tag_album(items, search_ids=[album_id])
+
    if prof:
-        cProfile.runctx('_run_match()', {}, {'_run_match': _run_match},
-                        'match.prof')
+        cProfile.runctx(
+            "_run_match()", {}, {"_run_match": _run_match}, "match.prof"
+        )
    else:
        interval = timeit.timeit(_run_match, number=1)
-        print('match duration:', interval)
+        print("match duration:", interval)


 class BenchmarkPlugin(BeetsPlugin):
-    """A plugin for performing some simple performance benchmarks.
-    """
-    def commands(self):
-        aunique_bench_cmd = ui.Subcommand('bench_aunique',
-                                          help='benchmark for %aunique{}')
-        aunique_bench_cmd.parser.add_option('-p', '--profile',
-                                            action='store_true', default=False,
-                                            help='performance profiling')
-        aunique_bench_cmd.func = lambda lib, opts, args: \
-            aunique_benchmark(lib, opts.profile)
+    """A plugin for performing some simple performance benchmarks."""

-        match_bench_cmd = ui.Subcommand('bench_match',
-                                        help='benchmark for track matching')
-        match_bench_cmd.parser.add_option('-p', '--profile',
-                                          action='store_true', default=False,
-                                          help='performance profiling')
-        match_bench_cmd.parser.add_option('-i', '--id', default=None,
-                                          help='album ID to match against')
-        match_bench_cmd.func = lambda lib, opts, args: \
-            match_benchmark(lib, opts.profile, ui.decargs(args), opts.id)
+    def commands(self):
+        aunique_bench_cmd = ui.Subcommand(
+            "bench_aunique", help="benchmark for %aunique{}"
+        )
+        aunique_bench_cmd.parser.add_option(
+            "-p",
+            "--profile",
+            action="store_true",
+            default=False,
+            help="performance profiling",
+        )
+        aunique_bench_cmd.func = lambda lib, opts, args: aunique_benchmark(
+            lib, opts.profile
+        )
+
+        match_bench_cmd = ui.Subcommand(
+            "bench_match", help="benchmark for track matching"
+        )
+        match_bench_cmd.parser.add_option(
+            "-p",
+            "--profile",
+            action="store_true",
+            default=False,
+            help="performance profiling",
+        )
+        match_bench_cmd.parser.add_option(
+            "-i", "--id", default=None, help="album ID to match against"
+        )
+        match_bench_cmd.func = lambda lib, opts, args: match_benchmark(
+            lib, opts.profile, ui.decargs(args), opts.id
+        )

        return [aunique_bench_cmd, match_bench_cmd]
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
AdeHub	0f43e8258d	Set macOS encoder multiprocessing method to 'fork' default for macOS python3 is 'spawn' which is not working with the current code	2025-08-08 17:28:21 +12:00
AdeHub	d667cdb5c9	Growl removed	2025-08-06 16:01:52 +12:00
AdeHub	c459eb1074	Boxcar removed	2025-08-06 15:52:34 +12:00
AdeHub	b039492072	MacOS Notifications replaced unreliable method	2025-08-06 15:42:39 +12:00
AdeHub	0ac76fbebc	Blackhole folder issue fixes #3388	2025-08-03 21:14:24 +12:00
AdeHub	62757c0fcf	Merge pull request #3367 from confor/fix-deps Add imghdr and cgi	2025-06-02 08:39:31 +12:00
AdeHub	63c992a4e5	Merge pull request #3363 from hypsometric/deluge_fix Fix deluge torrent upload	2025-05-31 12:17:20 +12:00
AdeHub	63549b2471	fix weserv blocking	2025-05-31 12:05:49 +12:00
confor	3ff431da4b	add cgi (py 3.13+ support) sourced from youknowone/python-deadlib	2025-05-22 22:04:07 -04:00
confor	8ccc3cb146	add imghdr (py 3.13+ support) this package was deprecated and removed in python 3.13+. sourced from youknowone/python-deadlib	2025-05-22 21:56:10 -04:00
hypsometric	6764783aa7	Fix deluge torrent upload Fix torrent file encoding for deluge - received content is bytes - we need to convert this content into base64 - we then need to convert base64 bytes into base64 string to include in the json data binary Fixes #3362	2025-04-17 12:33:43 +02:00
AdeHub	828fc0ccda	Merge pull request #3359 from hypsometric/gazelle_improvements Gazelle improvements	2025-04-12 15:28:44 +12:00
AdeHub	68a495127e	Potential fix for #3355 Potential fix for #3355	2025-04-12 15:26:21 +12:00
Hypsometric	26a6095851	Make Gazelle torrents sorting clearer Sort by format and most seeders if `search_formats` has data. Sort only by most seeders else.	2025-03-19 17:20:46 +01:00
Hypsometric	95bd0a57ff	Refactor common Orpheus & Redacted code Orpheus & Redacted being based on Gazelle API, their code was the same (with the exception of release type filtering). Merging both will make maintenance easy and brings Redacted on par with Orpheus regarding release type filtering.	2025-03-19 17:20:46 +01:00
Hypsometric	5efbd561d6	Allow the use of apikey with GazelleAPI & Redacted	2025-03-19 17:16:55 +01:00
AdeHub	b690850019	torznab enclosure url	2024-12-07 19:58:57 +13:00
AdeHub	a09e91ff8a	cherrypy update fixes #3348	2024-12-07 19:56:21 +13:00
AdeHub	94d62430a0	Ignore bs4 warning	2024-12-07 09:55:00 +13:00
AdeHub	2d5102d29c	Remove unused directory	2024-08-24 17:02:22 +12:00
AdeHub	046d4d82b4	update Beets	2024-08-24 16:44:41 +12:00
AdeHub	a63098a919	update urllib3	2024-08-24 16:42:31 +12:00
AdeHub	38d8e13e4e	update Soupsieve	2024-08-24 16:36:55 +12:00
AdeHub	b3edfa0d87	update requests_oauthlib	2024-08-24 16:33:52 +12:00
AdeHub	aeebc67365	update requests	2024-08-24 16:25:17 +12:00
AdeHub	d6f22a2326	Update Beautiful Soup	2024-08-24 16:18:38 +12:00
AdeHub	5b1d9c0750	update mutagen	2024-08-24 16:00:16 +12:00
AdeHub	ba666f68b8	tpb tidy up	2024-08-16 12:44:17 +12:00
AdeHub	000509353a	qBitTorrent Seed Ratio fix - added missing api param	2024-08-11 07:53:58 +12:00
AdeHub	c264c75282	Removed Waffles 😧	2024-08-10 11:34:30 +12:00
AdeHub	4c15b33c58	TPB fixes - changed default to use apibay.org - removed 'Old Pirate Bay'	2024-08-10 11:20:23 +12:00
AdeHub	167b5817e7	forcePostProcess more encoding - encode all download dirs so subsequent decode will work	2024-07-23 21:11:05 +12:00
AdeHub	46c7582fb1	Merge pull request #3342 from dsm1212/issue3341 decode paths before using them to fix manual post process	2024-07-23 20:17:10 +12:00
dsm1212	7b9e32f525	fix duplicate file rename	2024-07-14 20:55:05 -04:00
AdeHub	37c4e19663	Torznab changes -Search using parent category 3000 some providers lump everything in to the parent Music Category 3000 instead of the Sub Category (e.g. 3040 lossless). search using 3000 then filter with Category (if returned) or Headphones filtering overall this should return more results across indexers -Allow Prowlarr indexers to be manually entered -Searcher fixups	2024-07-08 21:18:19 -04:00
AdeHub	e5beb5291d	Soulseek tweaks # 2 push results through filters	2024-07-08 21:18:19 -04:00
AdeHub	ebe8a60ca5	Soulseek tweaks -include soulseek results in overall search results -fixed windows paths not quite working on macOS -allow user search term -tighten search for self titled, Various Artists -postprocess by user/folder -bit more logging	2024-07-08 21:18:19 -04:00
AdeHub	040de81d3c	Torznab changes -Search using parent category 3000 some providers lump everything in to the parent Music Category 3000 instead of the Sub Category (e.g. 3040 lossless). search using 3000 then filter with Category (if returned) or Headphones filtering overall this should return more results across indexers -Allow Prowlarr indexers to be manually entered -Searcher fixups	2024-07-05 19:26:30 +12:00
Steve McAfee	1bf8c35154	decode paths before using them to fix manual post process	2024-07-03 21:22:55 -04:00
AdeHub	dae4d80490	Soulseek tweaks # 2 push results through filters	2024-06-19 20:33:40 +12:00
AdeHub	acf73368c9	Soulseek tweaks -include soulseek results in overall search results -fixed windows paths not quite working on macOS -allow user search term -tighten search for self titled, Various Artists -postprocess by user/folder -bit more logging	2024-06-16 20:01:35 +12:00
rembo10	abac9b5a15	Merge branch 'develop'	2024-05-26 21:12:22 +05:30
rembo10	ace2b4f26a	Update changelog for v0.6.3	2024-05-26 21:12:09 +05:30
rembo10	ebba115443	Fix for searcher not returning results	2024-05-26 19:58:45 +05:30
rembo10	b54218abbd	Merge branch 'develop'	2024-05-26 09:52:28 +05:30
rembo10	3e354ff128	Update changelog for v0.6.2	2024-05-26 09:52:17 +05:30
rembo10	b8168ec8eb	add slskd_api @ v0.1.5	2024-05-26 09:47:43 +05:30
rembo10	c0c636d545	Merge remote-tracking branch 'el133/slskd-python-api_support' into develop	2024-05-26 09:44:54 +05:30
rembo10	33d1d17c97	Merge remote-tracking branch 'nopoz/deluge' into develop	2024-05-26 09:42:23 +05:30
rembo10	8fd13621f5	Fix some invalid escape sequences in helpers.py	2024-05-26 09:37:05 +05:30
rembo10	56674adfea	Merge branch 'python312' into develop	2024-05-26 09:31:16 +05:30
Elmer	489c6cbe71	Basic soulseek support	2024-02-28 12:53:55 +01:00
AdeHub	6afe31bffc	bandcamp tweaks	2024-01-26 21:11:29 +13:00
rembo10	742529a92d	munkres: 1.0.6 -> 1.1.4	2024-01-18 16:12:15 +05:30
rembo10	944d066903	Fix regex warnings in postprocessor.py	2024-01-18 16:10:33 +05:30
rembo10	87819a3c74	Remove redundant bencode import from searcher.py	2024-01-18 16:09:22 +05:30
rembo10	c9fbe29c90	Fix regex warnings in deluge.py	2024-01-18 16:06:51 +05:30
rembo10	d78cb7d14e	Import mediafile from bandcamp.py	2024-01-18 16:04:42 +05:30
rembo10	e8c392824f	Fix regex warnings in pygazelle	2024-01-18 16:02:59 +05:30
rembo10	9811df2779	Slightly clean up searcher.py, fix regex warnings	2024-01-18 16:01:26 +05:30
rembo10	1a4865ed38	Remove unused functions (nameQuality, statusFromName) from common.py	2024-01-18 15:17:37 +05:30
rembo10	a06fb40f50	tzlocal: 1.1.2 -> 5.2	2024-01-18 14:59:28 +05:30
rembo10	ad6a4f570e	use pkg_resources from setuptools-69.0.3	2024-01-18 14:56:54 +05:30
rembo10	3685d32a7d	cherrypy: 18.8.0 -> 6387a2b	2024-01-18 14:55:28 +05:30
rembo10	152f5daa8c	Sync up master and develop branches	2023-11-27 13:40:34 +05:30
Mikaël Hautinandrembo10	39054a04df	Update parsing logic for updated TPB proxy html structure (#3318 ) * fix: Update parsing logic for updated TPB proxy html structure --------- Co-authored-by: rembo10 <rembo10@users.noreply.github.com>	2023-11-27 10:04:10 +05:30
kirill9617	1c4b9c10f0	Fix for 503 error from CloudFlare. (#3316 ) CloudFlare seems to block request without User-Agent header. This fix should be enough to bypass that.	2023-11-26 15:15:36 +05:30
Menno Blom	73ca787cf1	Bandcamp support (#3252 ) * Add configuration options for bandcamp * Add bandcamp config options * Add bandcamp config options * Initial crude bandcamp search and download support * Add bandcamp search support * Better utf-8 handling and tagging of the downloads * Post-process bandcamp dodwnload directory * Tweak the order of downloads (prevent querying bandcamp too often) * Show [bandcamp] link in the history page * pep8 * Use more sane loglevels * Oops. * Patch regexp to support new bandcamp page structure * Make sure the file-key exists, is not None and contains data	2023-11-26 15:13:13 +05:30
rembo10	c7bc852868	Update changelog for v0.6.1	2023-11-26 14:30:29 +05:30
rembo10	391b0cc465	Merge branch 'develop'	2023-11-26 14:27:56 +05:30
rembo10	4aaeaa704f	apscheduler: 3.0.1 -> 3.10.1	2023-06-09 07:26:41 +05:30
rembo10	4d14b028ff	update cherrypy to v18.8.0	2023-04-20 15:06:17 +05:30
doucheymcdoucherson	138d01db4a	deluge udpate	2019-06-15 00:03:03 -07:00