v0.5.10

This reverts commit 74e071b1d7.

.travis.yml

@@ -16,6 +16,5 @@ install:
  - pip install pep8
 script:
   - pep8 headphones
-  - pylint --rcfile=pylintrc headphones
   - pyflakes headphones
   - nosetests headphones

API.md

@@ -66,8 +66,10 @@ Unmark album as wanted / i.e. mark as skipped
 
 ### forceSearch
 force search for wanted albums - not launched in a separate thread so it may take a bit to complete
-### forceProcess
+### forceProcess[&dir=/path/to/folder]
 Force post process albums in download directory - also not launched in a separate thread
+### forceActiveArtistsUpdate
+force Active Artist Update - also not launched in a separate thread
 
 ### getVersion
 Returns some version information: git_path, install_type, current_version, installed_version, commits_behind
@@ -106,4 +108,4 @@ See above.
 Gives you a list of results from searcher.searchforalbum(). Basically runs a normal search, but rather than sorting them and downloading the best result, it dumps the data, which you can then pass on to download_specific_release(). Returns a list of dictionaries with params: title, size, url, provider & kind - all of these values must be passed back to download_specific_release
 
 ### download_specific_release&id=albumid&title=$title&size=$size&url=$url&provider=$provider&kind=$kind
-Allows you to manually pass a choose_specific_download release back to searcher.send_to_downloader()
+Allows you to manually pass a choose_specific_download release back to searcher.send_to_downloader()

CHANGELOG.md

@@ -1,9 +1,144 @@
 # Changelog
 
-## v0.5.2
+## v0.5.10
-Released 28 december 2014
+Released 29 January 2016
 
-Highlight:
+Highlights:
+* Added: API option to post-process single folders
+* Added: Ability to specify extension when re-encoding
+* Added: Option to stop renaming folders
+* Fixed: Utorrent torrents not being removed (#2385)
+* Fixed: Torznab to transmission
+* Fixed: Magnet folder names in history
+* Fixed: Multiple torcache fixes
+* Fixed: Updated requests & urllib3 to latest versions to fix errors with pyOpenSSL
+* Improved: Use a temporary folder during post-processing
+* Improved: Added verify_ssl_cert option
+* Improved: Fixed track matching progress
+* Improved: pylint, pep8 & pylint fixes
+* Improved: Stop JS links from scrolling to the top of the page
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.9...v0.5.10).
+
+## v0.5.9
+Released 05 September 2015
+
+Highlights:
+* Added: Providers Strike, Jackett, custom Torznabs
+* Added: Option to stop post-processing if no good match found (#2343)
+* Fixed: Blackhole -> Magnet, limit to torcache
+* Fixed: Kat 403 flac error
+* Fixed: Last.fm errors
+* Fixed: Pushover notifications
+* Improved: Rutracker logging, switched to requests lib
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.8...v0.5.9).
+
+## v0.5.8
+Released 13 July 2015
+
+Highlights:
+* Added: Option to only include official extras
+* Added: Option to wait until album release date before searching
+* Fixed: NotifyMyAndroid notifications
+* Fixed: Plex Notifications
+* Fixed: Metacritic parsing
+* Fixed: Pushbullet notifications
+* Fixed: What.cd not honoring custom search term (#2279)
+* Improved: XSS Search bug
+* Improved: Config page layout
+* Improved: Set localhost as default
+* Improved: Better single artist scanning
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.7...v0.5.8).
+
+## v0.5.7
+Released 01 July 2015
+
+Highlights:
+* Improved: Moved pushover to use requests lib
+* Improved: Plex tokens with Plex Home
+* Improved: Added getLogs & clearLogs to api
+* Improved: Cache MetaCritic scores. Added user-agent header to fix 403 errors
+* Improved: Specify whether to delete folders when force post-processing
+* Improved: Convert target bitrate to vbr preset for what.cd searching
+* Improved: Switched Pushover to requests lib
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.6...v0.5.7).
+
+## v0.5.6
+Released 08 June 2015
+
+Highlights:
+* Added: Metacritic scores
+* Added: Series support (e.g. Cafe Del Mar, Now That's What I Call Music, etc)
+* Added: Filter out clean/edited/censored releases (#2198)
+* Added: Button on the log page to toggle verbose/debug logging
+* Fixed: Connecting to SABnzbd over https with python >= 2.7.9
+* Fixed: Email Notifications with SSL
+* Fixed: Don't limit musicbrainz results to first 100
+* Fixed: nzbget url fix
+* Fixed: OSX Notifications
+* Improved: Cuesplit, allow wav, ape to be split
+* Improved: Moved the 'freeze db' option to the advanced->misc. tab
+* Improved: Moved kickass searching to json api, so it doesn't throw 404 errors anymore when there are no results
+* Improved: SSL for headphones indexer
+* Improved: Disable update dialog box if check_github is diabled
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.5...v0.5.6).
+
+## v0.5.5
+Released 04 May 2015
+
+Highlights:
+* Added: force ID3v2.3 during post processing (#2121)
+* Added: MusicBrainz authentication (#2125)
+* Added: Email notifications (addresses #1045)
+* Fixed: Kickass url updated to kickass.to (#2119)
+* Fixed: Piratebay returning 0 bytes for all files
+* Fixed: Albums stopped automatically refreshing when adding an artist
+* Fixed: Min/max sizes for target bitrate
+* Fixed: Don't filter any results if looking for a specific download
+* Fixed: Sort by size in the specific download table
+* Fixed: Deal with beets recommendation.none correctly
+* Improved: Close dialog window automatically when choosing a specific download
+* Improved: Move some repetitive log messages to debug level
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.4...v0.5.5).
+
+## v0.5.4
+Released 05 February 2015
+
+Highlights:
+* Added: backported 'Scan Artist' feature from `sarakha63/headphones`
+* Fixed: change file permissions of cache files according to settings (#2102)
+* Fixed: hide Songkick if no regional events are available
+* Fixed: only reschedule jobs if timeout changed (#2099)
+* Fixed: limit dialog height (#2096)
+* Improved: upgraded requests to 2.5.1
+* Improved: upgraded unidecode to 0.04.17
+* Improved: upgraded tzlocal to 1.1.2
+* Improved: upgraded PyTZ to 2014.10
+* Improved: upgraded mako to 1.0.1
+* Improved: upgraded mutagen to 1.27
+* Improved: upgraded beets to 1.3.10
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.3...v0.5.4).
+
+## v0.5.3
+Released 15 January 2015
+
+Highlights:
+* Added: update active artists via API (#2075)
+* Fixed: queue instance lacks `add` method (#2055)
+* Improved: better SSL error messages (#2058)
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.2...v0.5.3).
+
+## v0.5.2
+Released 28 December 2014
+
+Highlights:
 * Added: advanced option to ignore certain folders by patterns. (#2037)
 * Added: advanced option to ignore certain files by patterns (library only)
 * Added: specify optional paths to CUE splitting tools (#1938)
@@ -126,4 +261,4 @@ The full list of commits can be found [here](https://github.com/rembo10/headphon
 ## v0.1
 Released 05 August 2013
 
-The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/2156e1341405d07c5bcfbe994f6b354b32d94cda...v0.1).
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/2156e1341405d07c5bcfbe994f6b354b32d94cda...v0.1).

README.md

@@ -1,4 +1,7 @@
-#![Headphones Logo](https://github.com/rembo10/headphones/raw/master/data/images/headphoneslogo.png) Headphones
+##![Headphones Logo](https://github.com/rembo10/headphones/raw/master/data/images/headphoneslogo.png) Headphones
+
+**Master Branch:**  [![Build Status](https://travis-ci.org/rembo10/headphones.svg?branch=master)](https://travis-ci.org/rembo10/headphones)
+**Develop Branch:** [![Build Status](https://travis-ci.org/rembo10/headphones.svg?branch=develop)](https://travis-ci.org/rembo10/headphones)
 
 Headphones is an automated music downloader for NZB and Torrent, written in Python. It supports SABnzbd, NZBget, Transmission, µTorrent and Blackhole.
 
@@ -18,8 +21,8 @@ You are free to join the Headphones support community on IRC where you can ask q
 
 1. Analyze your log, you just might find the solution yourself!
 2. You read the wiki and searched existing issues, but this is not solving your problem.
-3. Post the issue with a clear title, description and the HP log and use [proper markdown syntax](https://help.github.com/articles/github-flavored-markdown) to structure your text (code/log in code blocks). 
+3. Post the issue with a clear title, description and the HP log and use [proper markdown syntax](https://help.github.com/articles/github-flavored-markdown) to structure your text (code/log in code blocks).
-4. Close your issue when it's solved! If you found the solution yourself please comment so that others benefit from it.
+4. Close your issue when it's solved! If you found the solution yourself, please comment so that others benefit from it.
 
 **Feature requests** can be reported on the GitHub issue tracker too:
 

contrib/sni_test.py

@@ -0,0 +1,145 @@
+#!/usr/bin/env python
+
+import os
+import sys
+
+# Ensure that we use the Headphones provided libraries.
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "../lib"))
+
+import urlparse
+
+
+def can_import(module):
+    """
+    Return True if a given module can be imported or not.
+    """
+
+    try:
+        __import__(module)
+    except ImportError:
+        return False
+
+    # Module can be imported
+    return True
+
+
+def check_installation():
+    """
+    Check if some core modules are available. Info is based on this topic:
+    https://github.com/rembo10/headphones/issues/2210.
+    """
+
+    if can_import("requests"):
+        import requests
+        requests_version = requests.__version__
+    else:
+        requests_version = "no"
+
+    if can_import("OpenSSL"):
+        import OpenSSL
+        openssl_version = OpenSSL.__version__
+    else:
+        openssl_version = "no"
+
+    if can_import("cryptography"):
+        import cryptography
+        cryptography_version = cryptography.__version__
+    else:
+        cryptography_version = "no"
+
+    if can_import("pyasn1"):
+        import pyasn1
+        pyasn1_version = pyasn1.__version__
+    else:
+        pyasn1_version = "no"
+
+    if can_import("ndg.httpsclient"):
+        from ndg import httpsclient
+        ndg_version = httpsclient.__date__
+    else:
+        ndg_version = "no"
+
+    # Print some system information.
+    sys.stdout.write(
+        "* Checking Python version: %s.%s.%s\n" % sys.version_info[:3])
+    sys.stdout.write("* Operating system: %s\n" % sys.platform)
+
+    sys.stdout.write(
+        "* Checking if requests can be imported: %s\n" % requests_version)
+    sys.stdout.write(
+        "* Checking if pyOpenSSL is installed: %s\n" % openssl_version)
+    sys.stdout.write(
+        "* Checking if cryptography is installed: %s\n" % cryptography_version)
+    sys.stdout.write(
+        "* Checking if pyasn1 is installed: %s\n" % pyasn1_version)
+    sys.stdout.write(
+        "* Checking if ndg.httpsclient is installed: %s\n" % ndg_version)
+
+
+def main():
+    """
+    Test if the current Headphones installation can connect to SNI-enabled
+    servers.
+    """
+
+    # Read the URL to test.
+    if len(sys.argv) == 1:
+        url = "https://sni.velox.ch/"
+    else:
+        url = sys.argv[1]
+
+    # Check if it is a HTTPS website.
+    parts = urlparse.urlparse(url)
+
+    if parts.scheme.lower() != "https":
+        sys.stderr.write(
+            "Error: provided URL does not start with https://\n")
+        return 1
+
+    # Gather information
+    check_installation()
+
+    # Do the request.
+    if not can_import("requests"):
+        sys.stderr.exit("Error: cannot continue without requests module!\n")
+        return 1
+
+    sys.stdout.write("* Performing request: %s\n" % url)
+
+    import requests
+    requests.packages.urllib3.disable_warnings()
+
+    try:
+        try:
+            response = requests.get(url)
+        except requests.exceptions.SSLError as e:
+            sys.stdout.write(
+                "- Server certificate seems invalid. I will disable "
+                "certificate check and try again. You'll see the real "
+                "exception if it fails again.\n")
+            sys.stdout.write(
+                "* Retrying request with certificate verification off.\n")
+            response = requests.get(url)
+    except Exception as e:
+        sys.stdout.write(
+            "- An error occured while performing the request. The "
+            "exception was: %s\n" % e.message)
+        sys.stdout.write(
+            "- Consult the Troubleshooting wiki (https://github.com/"
+            "rembo10/headphones/wiki/Troubleshooting) before you post an "
+            "issue!")
+        return 0
+
+    # Verify the response.
+    if response.status_code == 200:
+        sys.stdout.write("+ Got a valid response. All seems OK!\n")
+    else:
+        sys.stdout.write(
+            "- Server returned status code %s. Expected a status code 200.\n",
+            response.status_code)
+        sys.stdout.write(
+            "- However, I was able to communicate to the server!\n")
+
+# E.g. `python sni_test.py https://example.org'.
+if __name__ == "__main__":
+    sys.exit(main())

data/interfaces/default/album.html

@@ -14,15 +14,15 @@
 
 			<a id="menu_link_delete" href="deleteAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}"><i class="fa fa-trash-o"></i> Delete Album</a>
 			%if album['Status'] == 'Skipped' or album['Status'] == 'Ignored':
-			<a id="menu_link_wanted" href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=False', $(this),true)" data-success="'${album['AlbumTitle']}' added to queue"><i class="fa fa-heart"></i> Mark Album as Wanted</a>
+			<a id="menu_link_wanted" href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=False', $(this),true)" data-success="'${album['AlbumTitle']}' added to queue"><i class="fa fa-heart"></i> Mark Album as Wanted</a>
 			%elif album['Status'] == 'Wanted':
-			<a id="menu_link_check" href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=True', $(this));" data-success="Forced checking successful"><i class="fa fa-search"></i> Force Check</a>
+			<a id="menu_link_check" href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=True', $(this));" data-success="Forced checking successful"><i class="fa fa-search"></i> Force Check</a>
-			<a id="menu_link_skipped" href="#" onclick="doAjaxCall('unqueueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}', $(this),true);" data-success="'${album['AlbumTitle']}' marked as Skipped"><i class="fa fa-step-forward"></i> Mark Album as Skipped</a>
+			<a id="menu_link_skipped" href="javascript:void(0)" onclick="doAjaxCall('unqueueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}', $(this),true);" data-success="'${album['AlbumTitle']}' marked as Skipped"><i class="fa fa-step-forward"></i> Mark Album as Skipped</a>
 			%else:
-			<a id="menu_link_retry" href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=False', $(this),true);" data-success="Retrying the same version of '${album['AlbumTitle']}'"><i class="fa fa-refresh"></i> Retry Download</a>
+			<a id="menu_link_retry" href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=False', $(this),true);" data-success="Retrying the same version of '${album['AlbumTitle']}'"><i class="fa fa-refresh"></i> Retry Download</a>
-			<a id="menu_link_new" href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=True', $(this),true);" data-success="Looking for a new version of '${album['AlbumTitle']}'"><i class="fa fa-download"></i> Try New Version</a>
+			<a id="menu_link_new" href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=True', $(this),true);" data-success="Looking for a new version of '${album['AlbumTitle']}'"><i class="fa fa-download"></i> Try New Version</a>
 			%endif		
-			<a class="menu_link_edit" id="album_chooser" href="#"><i class="fa fa-pencil"></i> Choose Alternate Release</a>
+			<a class="menu_link_edit" id="album_chooser" href="javascript:void(0)"><i class="fa fa-pencil"></i> Choose Alternate Release</a>
 				<div id="dialog" title="Choose an Alternate Release" style="display:none" class="configtable">
 					<div class="links">
 						<%
@@ -30,11 +30,12 @@
 						%>
 						%if not alternate_albums:
 							<p>No alternate releases found. Try refreshing the artist (if the artist is being refreshed, please wait until it's finished)</p>
-							<h2><a id="refresh_artist" onclick="doAjaxCall('refreshArtist?ArtistID=${album['ArtistID']}', $(this)), true" href="#" data-success="'${album['ArtistName']}' is being refreshed">Refresh Artist</a></h2>
+							<h2><a id="refresh_artist" onclick="doAjaxCall('refreshArtist?ArtistID=${album['ArtistID']}', $(this)), true" href="javascript:void(0)" data-success="'${album['ArtistName']}' is being refreshed">Refresh Artist</a></h2>
 						%else:
 							%for alternate_album in alternate_albums:
 								<%
 									track_count = len(myDB.select("SELECT * from alltracks WHERE ReleaseID=?", [alternate_album['ReleaseID']]))
+									mb_link = "http://musicbrainz.org/release/" + alternate_album['ReleaseID']
 									have_track_count = len(myDB.select("SELECT * from alltracks WHERE ReleaseID=? AND Location IS NOT NULL", [alternate_album['ReleaseID']]))
 									if alternate_album['AlbumID'] == alternate_album['ReleaseID']:
 										alternate_album_name = "Headphones Default Release (" + str(alternate_album['ReleaseDate']) + ") [" + str(have_track_count) + "/" + str(track_count) + " tracks]"
@@ -42,43 +43,43 @@
 										alternate_album_name = alternate_album['AlbumTitle'] + " (" + alternate_album['ReleaseCountry'] + ", " + str(alternate_album['ReleaseDate']) + ", " + alternate_album['ReleaseFormat'] + ") [" + str(have_track_count) + "/" + str(track_count) + " tracks]"
 									
 								%>
-								<a href="#" onclick="doAjaxCall('switchAlbum?AlbumID=${album['AlbumID']}&ReleaseID=${alternate_album['ReleaseID']}', $(this), 'table');" data-success="Switched release to: ${alternate_album_name}">${alternate_album_name}</a><br>
+								<a href="javascript:void(0)" onclick="doAjaxCall('switchAlbum?AlbumID=${album['AlbumID']}&ReleaseID=${alternate_album['ReleaseID']}', $(this), 'table');" data-success="Switched release to: ${alternate_album_name}">${alternate_album_name}</a><a href="${mb_link}" target="_blank">MB</a><br>
 							%endfor
 						%endif
 					</div>
 				</div>
-			<a class="menu_link_edit" id="edit_search_term" href="#"><i class="fa fa-pencil"></i> Edit Search Term</a>
+			<a class="menu_link_edit" id="edit_search_term" href="javascript:void(0)"><i class="fa fa-pencil"></i> Edit Search Term</a>
-				<div id="dialog2" title="Enter your own search term for this album" style="display:none" class="configtable">
+			<div id="dialog2" title="Enter your own search term for this album" style="display:none" class="configtable">
-					<form action="editSearchTerm" method="GET" id="editSearchTerm">
+				<form action="editSearchTerm" method="GET" id="editSearchTerm">
-						<input type="hidden" name="AlbumID" value="${album['AlbumID']}">
+					<input type="hidden" name="AlbumID" value="${album['AlbumID']}">
-						<div class="row">
+					<div class="row">
-							<%
+						<%
-								if not album['SearchTerm']:
+							if not album['SearchTerm']:
-									search_term = ""
+								search_term = ""
-								else:
+							else:
-									search_term = album['SearchTerm']
+								search_term = album['SearchTerm']
-							%>
+						%>
-							<input type="text" value="${search_term}" name="SearchTerm" size="40" />
+						<input type="text" value="${search_term}" name="SearchTerm" size="40" />
-						</div>
+					</div>
-						<input type="button" value="Save changes" onclick="doAjaxCall('editSearchTerm',$(this),'tabs',true);return false;" data-success="Search term updated"/>
+					<input type="button" value="Save changes" onclick="doAjaxCall('editSearchTerm',$(this),'tabs',true);return false;" data-success="Search term updated"/>
-					</form>
+				</form>
-				</div>
+			</div>
-			<a class="menu_link_edit" id="choose_specific_download" href="#" onclick="getAvailableDownloads()"><i class="fa fa-search"></i> Choose Specific Download</a>
+			<a class="menu_link_edit" id="choose_specific_download" href="javascript:void(0)" onclick="getAvailableDownloads()"><i class="fa fa-search"></i> Choose Specific Download</a>
-				<div id="choose_specific_download_dialog" title="Choose a specific download for this album" style="display:none" class="configtable">
+			<div id="choose_specific_download_dialog" title="Choose a specific download for this album" style="display:none" class="configtable">
-						<table class="display" id="downloads_table">
+				<table class="display" id="downloads_table">
-							<thead>
+					<thead>
-								<tr>
+						<tr>
-									<th id="title">Title</th>
+							<th id="title">Title</th>
-									<th id="size">Size</th>
+							<th id="size">Size</th>
-									<th id="provider">Provider</th>
+							<th id="provider">Provider</th>
-									<th id="kind">Kind</th>
+							<th id="kind">Kind</th>
-									<th id="matches" title="Matches quality and/or seed settings.">Matches</th>
+							<th id="matches" title="Matches quality and/or seed settings.">Matches</th>
-								</tr>
+						</tr>
-							</thead>
+					</thead>
-							<tbody id="downloads_table_body">
+					<tbody id="downloads_table_body">
-							</tbody>
+					</tbody>
-						</table>
+				</table>
-				</div>
+			</div>
 		</div>
 	</div>
 </%def>
@@ -197,14 +198,20 @@
 		var elem = $("#albumInfo");
 		getInfo(elem,id,'album');
 	}
-	
+
 	function initThisPage() {
 		$('#album_chooser').click(function() {
-			$('#dialog').dialog({ width: "500px" });
+			$('#dialog').dialog({
+				width: 500,
+				maxHeight: 500
+			});
 			return false;
 		});
 		$('#edit_search_term').click(function() {
-			$('#dialog2').dialog({ width: "500px" });
+			$('#dialog2').dialog({
+				width: 500,
+				maxHeight: 500
+			});
 			return false;
 		});
 		$('#refresh_artist').click(function() {
@@ -213,8 +220,8 @@
 		initActions();
 		setTimeout(function(){
 			initFancybox();
-			},1000);
+		}, 1000);
-		
+
 		$('#track_table').dataTable({
 			"aaSorting": [],
 			"bFilter": false,
@@ -231,22 +238,26 @@
 			feedback.fadeOut();
 			search_results = data
 			for( var i = 0, len = data.length; i < len; i++ ) {
-    			$('#downloads_table_body').append('<tr><td id="title"><a href="#" onclick="downloadSpecificRelease('+i+')">'+data[i].title+'</a></td><td id="size"><span title='+data[i].size+'></span>'+(data[i].size / (1024*1024)).toFixed(2)+' MB</td><td id="provider">'+data[i].provider+'</td><td id="kind">'+data[i].kind+'</td><td id="matches">'+data[i].matches+'</td></tr>');
+    			$('#downloads_table_body').append('<tr><td id="title"><a href="javascript:void(0)" onclick="downloadSpecificRelease('+i+')">'+data[i].title+'</a></td><td id="size"><span title='+data[i].size+'></span>'+(data[i].size / (1024*1024)).toFixed(2)+' MB</td><td id="provider">'+data[i].provider+'</td><td id="kind">'+data[i].kind+'</td><td id="matches">'+data[i].matches+'</td></tr>');
     		}
 			$('#downloads_table').dataTable({
 				"aoColumns": [
 					null,
-					{ "sType": "title-numeric"},
+					{"sType": "title-numeric"},
 					null,
 					null,
-					null
+					{"sType": "string"}
 				],
-				"aaSorting": [],
+				"aaSorting": [[ 4, 'desc']],
 				"bFilter": false,
 				"bInfo": false,
 				"bPaginate": false,
+				"bDestroy": true
 			});
-    		$("#choose_specific_download_dialog").dialog({ width: "1000px" });
+    		$("#choose_specific_download_dialog").dialog({
+    			width: "80%",
+    			maxHeight: 500
+    		});
     		return false;
 		});
 	}
@@ -327,6 +338,21 @@
         });
     }
 
+    jQuery.extend( jQuery.fn.dataTableExt.oSort, {
+		"title-numeric-pre": function ( a ) {
+			var x = a.match(/title="*(-?[0-9\.]+)/)[1];
+			return parseFloat( x );
+		},
+
+		"title-numeric-asc": function ( a, b ) {
+			return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+		},
+
+        "title-numeric-desc": function ( a, b ) {
+            return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+        }
+    } );
+
 	$(document).ready(function()	{
 		getAlbumInfo();
 		initThisPage();

data/interfaces/default/artist.html

@@ -8,18 +8,19 @@
 <%def name="headerIncludes()">
 	<div id="subhead_container">
 		<div id="subhead_menu">
-			<a id="menu_link_refresh" onclick="doSimpleAjaxCall('refreshArtist?ArtistID=${artist['ArtistID']}')" href="#"><i class="fa fa-refresh"></i> Refresh Artist</a>
+			<a id="menu_link_refresh" onclick="doSimpleAjaxCall('refreshArtist?ArtistID=${artist['ArtistID']}')" href="javascript:void(0)"><i class="fa fa-refresh"></i> Refresh Artist</a>
 			<a id="menu_link_delete" href="deleteArtist?ArtistID=${artist['ArtistID']}"><i class="fa fa-trash-o"></i> Delete Artist</a>
+			<a id="menu_link_scan" onclick="doAjaxCall('scanArtist?ArtistID=${artist['ArtistID']}', $(this)),'table'" href="javascript:void(0)" data-success="'${artist['ArtistName']}' was scanned"><i class="fa fa-refresh"></i> Scan Artist</a>
 			%if artist['Status'] == 'Paused':
-				<a id="menu_link_resume" href="#" onclick="doAjaxCall('resumeArtist?ArtistID=${artist['ArtistID']}',$(this),true)" data-success="${artist['ArtistName']} resumed"><i class="fa fa-play"></i> Resume Artist</a>
+				<a id="menu_link_resume" href="javascript:void(0)" onclick="doAjaxCall('resumeArtist?ArtistID=${artist['ArtistID']}',$(this),true)" data-success="${artist['ArtistName']} resumed"><i class="fa fa-play"></i> Resume Artist</a>
 			%else:
-				<a id="menu_link_pauze" href="#" onclick="doAjaxCall('pauseArtist?ArtistID=${artist['ArtistID']}',$(this),true)" data-success="${artist['ArtistName']} paused"><i class="fa fa-pause"></i> Pause Artist</a>
+				<a id="menu_link_pauze" href="javascript:void(0)" onclick="doAjaxCall('pauseArtist?ArtistID=${artist['ArtistID']}',$(this),true)" data-success="${artist['ArtistName']} paused"><i class="fa fa-pause"></i> Pause Artist</a>
 			%endif
 			%if artist['IncludeExtras']:
-				<a id="menu_link_removeextra" href="#" onclick="doAjaxCall('removeExtras?ArtistID=${artist['ArtistID']}&ArtistName=${artist['ArtistName']}',$(this),'submenu&table')" data-success="Extras removed for ${artist['ArtistName']}"><i class="fa fa-minus"></i> Remove Extras</a>
+				<a id="menu_link_removeextra" href="javascript:void(0)" onclick="doAjaxCall('removeExtras?ArtistID=${artist['ArtistID']}&ArtistName=${artist['ArtistName']}',$(this),'submenu&table')" data-success="Extras removed for ${artist['ArtistName']}"><i class="fa fa-minus"></i> Remove Extras</a>
-				<a class="menu_link_edit" id="menu_link_modifyextra" href="#"><i class="fa fa-pencil"></i> Modify Extras</a>
+				<a class="menu_link_edit" id="menu_link_modifyextra" href="javascript:void(0)"><i class="fa fa-pencil"></i> Modify Extras</a>
 			%else:
-				<a id="menu_link_getextra" href="#"><i class="fa fa-plus"></i> Get Extras</a>
+				<a id="menu_link_getextra" href="javascript:void(0)"><i class="fa fa-plus"></i> Get Extras</a>
 			%endif
 				<div id="dialog" title="Choose Which Extras to Fetch" style="display:none" class="configtable">
 					<form action="getExtras" method="get" class="form">
@@ -69,6 +70,7 @@
 				<th id="albumname">Name</th>
 				<th id="reldate">Date</th>
 				<th id="type">Type</th>
+				<th id="score">Metacritic</th>
 				<th id="status">Status</th>
 				<th id="have">Have</th>
 				<th id="bitrate">Bitrate</th>
@@ -124,18 +126,19 @@
 				<td id="albumname"><a href="albumPage?AlbumID=${album['AlbumID']}">${album['AlbumTitle']}</a></td>
 				<td id="reldate">${album['ReleaseDate']}</td>
 				<td id="type">${album['Type']}</td>
+				<td id="score">${album['CriticScore']}/${album['UserScore']}</td>
 				<td id="status">${album['Status']}
 				%if album['Status'] == 'Skipped' or album['Status'] == 'Ignored':
-					[<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}',$(this),'table')" data-success="'${album['AlbumTitle']}' added to Wanted list">want</a>]
+					[<a href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}',$(this),'table')" data-success="'${album['AlbumTitle']}' added to Wanted list">want</a>]
 				%elif (album['Status'] == 'Wanted' or album['Status'] == 'Wanted Lossless'):
-                    [<a href="#" onclick="doAjaxCall('unqueueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}',$(this),'table')" data-success="'${album['AlbumTitle']}' skipped">skip</a>] [<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}', $(this),'table')" data-success="Trying to download'${album['AlbumTitle']}'" title="Search if available for download">search</a>]
+                    [<a href="javascript:void(0)" onclick="doAjaxCall('unqueueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}',$(this),'table')" data-success="'${album['AlbumTitle']}' skipped">skip</a>] [<a href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}', $(this),'table')" data-success="Trying to download'${album['AlbumTitle']}'" title="Search if available for download">search</a>]
 				%else:
-					[<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}', $(this),'table')" data-success="Retrying the same version of '${album['AlbumTitle']}'" title="Retry the same download again">retry</a>][<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=True', $(this),'table')" title="Try a new download, skipping all previously tried nzbs"  data-success="Downloading new version for '${album['AlbumTitle']}'" data-success="Looking for a new version of '${album['AlbumTitle']}'">new</a>]
+					[<a href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}', $(this),'table')" data-success="Retrying the same version of '${album['AlbumTitle']}'" title="Retry the same download again">retry</a>][<a href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=True', $(this),'table')" title="Try a new download, skipping all previously tried nzbs"  data-success="Downloading new version for '${album['AlbumTitle']}'" data-success="Looking for a new version of '${album['AlbumTitle']}'">new</a>]
 				%endif
 				%if albumformat in lossy_formats and album['Status'] == 'Skipped':
-					[<a id="wantlossless" href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&lossless=True', $(this),'table')" data-success="Lossless version of '${album['AlbumTitle']}' added to queue">want lossless</a>]
+					[<a id="wantlossless" href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&lossless=True', $(this),'table')" data-success="Lossless version of '${album['AlbumTitle']}' added to queue">want lossless</a>]
 				%elif albumformat in lossy_formats and (album['Status'] == 'Snatched' or album['Status'] == 'Downloaded'):
-					[<a id="wantlossless" href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&lossless=True', $(this),'table')" data-success="Retrying the same lossless version of '${album['AlbumTitle']}'">retry lossless</a>]
+					[<a id="wantlossless" href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&lossless=True', $(this),'table')" data-success="Retrying the same lossless version of '${album['AlbumTitle']}'">retry lossless</a>]
 				%endif
 				</td>
 				<td id="have"><span title="${percent}"><span><div class="progress-container"><div style="width:${percent}%"><div class="havetracks">${havetracks}/${totaltracks}</div></div></div></td>
@@ -189,45 +192,45 @@
 		$.getJSON("https://api.songkick.com/api/3.0/artists/mbid:${artist['ArtistID']}/calendar.json?apikey=${headphones.CONFIG.SONGKICK_APIKEY}&jsoncallback=?",
 			function(data){
 				if (data['resultsPage'].totalEntries >= 1) {
-
+                    if (${songkick_filter_enabled}) {
-                    if( ${songkick_filter_enabled} ) {
                         data.resultsPage.results.event = $.grep(data.resultsPage.results.event, function(element,index){
-                        return element.venue.metroArea.id == ${songkick_location};
+                    	    return element.venue.metroArea.id == ${songkick_location};
-                    });
+                    	});
                     }
 
-					var tourDate;
+                    if (data.resultsPage.results.event.length > 0) {
+						var tourDate;
 
-					calendarDomNode.show();
+						calendarDomNode.show();
-					$("#artistImg").addClass('on-tour');
+						$("#artistImg").addClass('on-tour');
 
-					jQuery.each( data['resultsPage'].results.event, function( i, event ) {
+						jQuery.each(data.resultsPage.results.event, function(i, event) {
-						tourDate = template;
+							tourDate = template;
-						tourDate = tourDate.replace('URI',event.uri);
+							tourDate = tourDate.replace('URI',event.uri);
-						tourDate = tourDate.replace('NAME',event.displayName);
+							tourDate = tourDate.replace('NAME',event.displayName);
-						tourDate = tourDate.replace('LOC',event.location.city);
+							tourDate = tourDate.replace('LOC',event.location.city);
 
-						calendarDomNode.append(tourDate);
+							calendarDomNode.append(tourDate);
-					});
+						});
 
-					calendarDomNode.append('<li><img src="interfaces/default/images/songkick.png" alt="concerts by songkick" class="sk-logo" /></li>');
+						calendarDomNode.append('<li><img src="interfaces/default/images/songkick.png" alt="concerts by songkick" class="sk-logo" /></li>');
-
-				$(function() {
-				  $("#artistCalendar").each(function() {
-					$("li:gt(4)", this).hide(); /* :gt() is zero-indexed */
-					$("li:nth-child(5)", this).after("<br><li class='more'><a href='#'>More...</a></li>"); /* :nth-child() is one-indexed */
-				  });
-				  $("li.more a").live("click", function() {
-					var li = $(this).parents("li:first");
-					li.parent().children().show();
-					li.remove();
-					return false;
-				  });
-				});
 
+						$(function() {
+						  $("#artistCalendar").each(function() {
+							$("li:gt(4)", this).hide(); /* :gt() is zero-indexed */
+							$("li:nth-child(5)", this).after("<br><li class='more'><a href='#'>More...</a></li>"); /* :nth-child() is one-indexed */
+						  });
+						  $("li.more").on("click", 'a', function() {
+							var li = $(this).parents("li:first");
+							li.parent().children().show();
+							li.remove();
+							return false;
+						  });
+						});
+					}
 				}
-			// data is JSON response object
+			}
-		});
+		);
 	}
 
 	var loadingMessage = false;
@@ -282,6 +285,7 @@
 				{ "sType": "date" },
 				null,
 				null,
+				null,
 				{ "sType": "title-numeric"},
 				null,
 				null

data/interfaces/default/base.html

@@ -31,14 +31,14 @@
 <body>
 	<div id="container">
 		<div id="ajaxMsg" class="ajaxMsg"></div>
-		% if not headphones.CURRENT_VERSION:
+		% if headphones.CONFIG.CHECK_GITHUB and not headphones.CURRENT_VERSION:
 			<div id="updatebar">
 				You're running an unknown version of Headphones. <a href="update">Update</a> or
-				<a href="#" onclick="$('#updatebar').slideUp('slow');">Close</a>
+				<a href="javascript:void(0)" onclick="$('#updatebar').slideUp('slow');">Close</a>
 			</div>
-		% elif headphones.CURRENT_VERSION != headphones.LATEST_VERSION and headphones.COMMITS_BEHIND > 0 and headphones.INSTALL_TYPE != 'win':
+		% elif headphones.CONFIG.CHECK_GITHUB and headphones.CURRENT_VERSION != headphones.LATEST_VERSION and headphones.COMMITS_BEHIND > 0 and headphones.INSTALL_TYPE != 'win':
 			<div id="updatebar">
-				A <a href="https://github.com/${headphones.CONFIG.GIT_USER}/headphones/compare/${headphones.CURRENT_VERSION}...${headphones.LATEST_VERSION}"> newer version</a> is available. You're ${headphones.COMMITS_BEHIND} commits behind. <a href="update">Update</a> or <a href="#" onclick="$('#updatebar').slideUp('slow');">Close</a>
+				A <a href="https://github.com/${headphones.CONFIG.GIT_USER}/headphones/compare/${headphones.CURRENT_VERSION}...${headphones.LATEST_VERSION}"> newer version</a> is available. You're ${headphones.COMMITS_BEHIND} commits behind. <a href="update">Update</a> or <a href="javascript:void(0)" onclick="$('#updatebar').slideUp('slow');">Close</a>
 			</div>
 		% endif
 
@@ -62,6 +62,7 @@
 					<select name="type" id="search_type">
 						<option value="artist">Artist</option>
 						<option value="album">Album</option>
+						<option value="series">Series</option>
 					</select>
 					<input type="submit" value="Search"/>
 				</form>
@@ -88,7 +89,7 @@
 				<small>
 				<a href="shutdown"><i class="fa fa-power-off"></i> Shutdown</a> |
 				<a href="restart"><i class="fa fa-power-off"></i> Restart</a> |
-				<a href="#" onclick="doAjaxCall('checkGithub',$(this))" data-success="Checking for update successful" data-error="Error checking for update"><i class="fa fa-refresh"></i> Check for new version</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('checkGithub',$(this))" data-success="Checking for update successful" data-error="Error checking for update"><i class="fa fa-refresh"></i> Check for new version</a>
 				</small>
 			</div>
 			<div id="version">

data/interfaces/default/css/style.css

@@ -325,7 +325,7 @@ form fieldset small.heading {
   margin-top: -15px;
 }
 form .row {
-  font-family: Helvetica, Arial;
+  font-family: Helvetica, Arial, sans-serif;
   margin-bottom: 10px;
 }
 form .row label {
@@ -334,7 +334,7 @@ form .row label {
   font-size: 12px;
   line-height: normal;
   padding-top: 7px;
-  width: 175px;
+  width: 170px;
 }
 form .row label.inline {
   margin-right: 5px;
@@ -368,7 +368,7 @@ form .row small {
   display: block;
   font-size: 9px;
   line-height: 12px;
-  margin-left: 175px;
+  margin-left: 170px;
   margin-top: 3px;
 }
 form .left label {
@@ -408,6 +408,18 @@ form .checkbox small {
 form .indent input {
   margin-left: 15px;
 }
+form .suboptions {
+  margin-left: 15px;
+}
+.option{
+  font-size: 15px;
+  font-weight: 600;
+  vertical-align: middle;
+}
+input.bigcheck[type="checkbox"] {
+  width: 16px;
+  height: 16px;
+}
 ul,
 ol {
   margin-left: 2em;
@@ -1420,6 +1432,13 @@ div#artistheader h2 a {
   line-height: 10px;
   height: 10px;
 }
+.nopad {
+  margin-bottom: 0px !important;
+  margin-top: 0px !important;
+  padding-top: 0px !important;
+  padding-bottom: 0px !important;
+  padding: 0px 0px !important;
+}
 #album_table th#albumname,
 #upcoming_table th#artistname,
 #wanted_table th#artistname {
@@ -1445,6 +1464,7 @@ div#artistheader h2 a {
 #album_table td#albumname,
 #album_table td#reldate,
 #album_table td#type,
+#album_table td#score,
 #track_table td#duration,
 #upcoming_table td#select,
 #upcoming_table td#status,
@@ -1455,6 +1475,7 @@ div#artistheader h2 a {
 }
 #album_table td#status,
 #album_table td#bitrate,
+#album_table td#score,
 #album_table td#albumformat,
 #album_table td#wantlossless {
   font-size: 13px;

data/interfaces/default/css/style.less

@@ -190,7 +190,7 @@ form {
             font-size: 12px;
             line-height: normal;
             padding-top: 7px;
-            width: 175px;
+            width: 170px;
 
             &.inline {
                 margin-right: 5px;
@@ -216,7 +216,7 @@ form {
             display: block;
             font-size: 9px;
             line-height: 12px;
-            margin-left: 175px;
+            margin-left: 170px;
             margin-top: 3px;
         }
     }

data/interfaces/default/history.html

@@ -7,12 +7,13 @@
 <%def name="headerIncludes()">
 	<div id="subhead_container">
 		<div id="subhead_menu">
-			<a id="menu_link_delete" href="#" onclick="doAjaxCall('clearhistory?type=all',$(this),'table')" data-success="All History cleared"><i class="fa fa-trash-o"></i> Clear All History</a>
+			<a id="menu_link_delete" href="javascript:void(0)" onclick="doAjaxCall('clearhistory?type=all',$(this),'table')" data-success="All History cleared"><i class="fa fa-trash-o"></i> Clear All History</a>
-			<a id="menu_link_delete" href="#" onclick="doAjaxCall('clearhistory?type=Processed',$(this),'table')" data-success="All Processed cleared"><i class="fa fa-trash-o"></i> Clear Processed</a>
+			<a id="menu_link_delete" href="javascript:void(0)" onclick="doAjaxCall('clearhistory?type=Processed',$(this),'table')" data-success="All Processed cleared"><i class="fa fa-trash-o"></i> Clear Processed</a>
-			<a id="menu_link_delete" href="#" onclick="doAjaxCall('clearhistory?type=Unprocessed',$(this),'table')" data-success="All Unprocessed cleared"><i class="fa fa-trash-o"></i> Clear Unprocessed</a>
+			<a id="menu_link_delete" href="javascript:void(0)" onclick="doAjaxCall('clearhistory?type=Unprocessed',$(this),'table')" data-success="All Unprocessed cleared"><i class="fa fa-trash-o"></i> Clear Unprocessed</a>
-			<a id="menu_link_delete" href="#" onclick="doAjaxCall('clearhistory?type=Snatched',$(this),'table')" data-success="All Snatched cleared"><i class="fa fa-trash-o"></i> Clear Snatched</a>
+			<a id="menu_link_delete" href="javascript:void(0)" onclick="doAjaxCall('clearhistory?type=Frozen',$(this),'table')" data-success="All Frozen cleared"><i class="fa fa-trash-o"></i> Clear Frozen</a>
+			<a id="menu_link_delete" href="javascript:void(0)" onclick="doAjaxCall('clearhistory?type=Snatched',$(this),'table')" data-success="All Snatched cleared"><i class="fa fa-trash-o"></i> Clear Snatched</a>
 		</div>
-	</div>	
+	</div>
 </%def>
 
 <%def name="body()">
@@ -31,32 +32,41 @@
 			</tr>
 		</thead>
 		<tbody>
-		%for item in history:
+        %for item in history:
-			<%
+        <%
-				if item['Status'] == 'Processed':
+            if item['Status'] == 'Processed':
-					grade = 'A'
+                grade = 'A'
-				elif item['Status'] == 'Snatched':
+            elif item['Status'] == 'Snatched':
-					grade = 'C'
+                grade = 'C'
-				elif item['Status'] == 'Unprocessed':
+            elif item['Status'] == 'Unprocessed':
-					grade = 'X'
+                grade = 'X'
-				else:
+            elif item['Status'] == 'Frozen':
-					grade = 'U'
+                grade = 'X'
-					
+            else:
-				fileid = 'unknown'
+                grade = 'U'
-				if item['URL'].find('nzb') != -1:
+
-					fileid = 'nzb'
+            fileid = 'unknown'
-				if item['URL'].find('torrent') != -1:
+            if item['URL'].find('nzb') != -1:
-					fileid = 'torrent'
+                fileid = 'nzb'
-				if item['URL'].find('rutracker') != -1:
+            if item['URL'].find('torrent') != -1:
-					fileid = 'torrent'
+                fileid = 'torrent'
-			%>
+            if item['URL'].find('magnet:') != -1:
+                fileid = 'torrent'
+            if item['URL'].find('rutracker') != -1:
+                fileid = 'torrent'
+            if item['URL'].find('codeshy') != -1:
+                fileid = 'nzb'
+
+            folder = 'Folder: ' + item['FolderName']
+
+        %>
 			<tr class="grade${grade}">
 				<td id="dateadded">${item['DateAdded']}</td>
 				<td id="filename">${cgi.escape(item['Title'], quote=True)} [<a href="${item['URL']}">${fileid}</a>]<a href="albumPage?AlbumID=${item['AlbumID']}">[album page]</a></td>
 				<td id="size">${helpers.bytes_to_mb(item['Size'])}</td>
-				<td id="status">${item['Status']}</td>
+				<td title="${folder}" id="status">${item['Status']}</td>
-				<td id="action">[<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${item['AlbumID']}&redirect=history', $(this),'table')"  data-success="Retrying download of '${cgi.escape(item['Title'], quote=True)}'">retry</a>][<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${item['AlbumID']}&new=True&redirect=history',$(this),'table')"  data-success="Looking for a new version of '${cgi.escape(item['Title'], quote=True)}'">new</a>]</td>
+				<td id="action">[<a href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${item['AlbumID']}&redirect=history', $(this),'table')"  data-success="Retrying download of '${cgi.escape(item['Title'], quote=True)}'">retry</a>][<a href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${item['AlbumID']}&new=True&redirect=history',$(this),'table')"  data-success="Looking for a new version of '${cgi.escape(item['Title'], quote=True)}'">new</a>]</td>
-				<td id="delete"><a href="#" onclick="doAjaxCall('clearhistory?date_added=${item['DateAdded']}&title=${cgi.escape(item['Title'], quote=True)}',$(this),'table')" data-success="${cgi.escape(item['Title'], quote=True)} cleared from history"><img src="interfaces/default/images/trashcan.png" height="18" width="18" id="trashcan" title="Clear this item from the history"></a>
+				<td id="delete"><a href="javascript:void(0)" onclick="doAjaxCall('clearhistory?date_added=${item['DateAdded']}&title=${cgi.escape(item['Title'], quote=True)}',$(this),'table')" data-success="${cgi.escape(item['Title'], quote=True)} cleared from history"><img src="interfaces/default/images/trashcan.png" height="18" width="18" id="trashcan" title="Clear this item from the history"></a>
 			</tr>
 		%endfor
 		</tbody>

data/interfaces/default/logs.html

@@ -7,6 +7,7 @@
     <div id="subhead_container">
         <div id="subhead_menu">
             <a class="menu_link_edit" href="clearLogs"><i class="fa fa-trash-o"></i> Clear log</a>
+            <a class="menu_link_edit" href="toggleVerbose"><i class="fa fa-pencil"></i> Toggle Debug Log</a>
         </div>
     </div>
 </%def>

data/interfaces/default/manage.html

@@ -6,7 +6,7 @@
 <%def name="headerIncludes()">
 	<div id="subhead_container">
 		<div id="subhead_menu">
-			<a class="menu_link_edit" id="manage_albums" href="#"><i class="fa fa-pencil"></i> Manage Albums</a>
+			<a class="menu_link_edit" id="manage_albums" href="javascript:void(0)"><i class="fa fa-pencil"></i> Manage Albums</a>
 				<div id="dialog" title="Choose Album Filter" style="display:none" class="configtable">
 					<div class="links">
 						<a href="manageAlbums?Status=Downloaded"><i class="fa fa-check fa-fw"></i> Manage Downloaded Albums</a><br>
@@ -90,7 +90,7 @@
 				%>
 				<input type="text" value="${lastfmvalue}" placeholder="Last.fm username" onfocus="if
 				(this.value==this.defaultValue) this.value='';" name="username" id="username" size="18" />
-				<a href="#" onclick="doAjaxCall('importLastFM?username=',$(this),'tabs');return false;" data-success="Last.fm username has been reset"><i class="fa fa-reply"></i> Reset username</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('importLastFM?username=',$(this),'tabs');return false;" data-success="Last.fm username has been reset"><i class="fa fa-reply"></i> Reset username</a>
 			</div>
 			</fieldset>
 			<input type="button" value="Save changes" onclick="doAjaxCall('importLastFM',$(this),'tabs',true);return false;" data-success="Last.fm artists will be imported" data-error="Fill in a last.fm username"/>
@@ -121,10 +121,10 @@
 		<fieldset>
 			<legend>Force Search</legend>
 			<div class="links">
-				<a href="#" onclick="doAjaxCall('forceSearch',$(this))" data-success="Checking for wanted albums successful" data-error="Error checking wanted albums"><i class="fa fa-search fa-fw"></i> Force Check for Wanted Albums</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('forceSearch',$(this))" data-success="Checking for wanted albums successful" data-error="Error checking wanted albums"><i class="fa fa-search fa-fw"></i> Force Check for Wanted Albums</a>
-				<a href="#" onclick="doAjaxCall('forceUpdate',$(this))" data-success="Update active artists successful" data-error="Error forcing update artists"><i class="fa fa-heart fa-fw"></i> Force Update Active Artists [Fast]</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('forceUpdate',$(this))" data-success="Update active artists successful" data-error="Error forcing update artists"><i class="fa fa-heart fa-fw"></i> Force Update Active Artists [Fast]</a>
-				<a href="#" onclick="doAjaxCall('checkGithub',$(this))" data-success="Checking for update successful" data-error="Error checking for update"><i class="fa fa-refresh fa-fw"></i> Check for Headphones Updates</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('checkGithub',$(this))" data-success="Checking for update successful" data-error="Error checking for update"><i class="fa fa-refresh fa-fw"></i> Check for Headphones Updates</a>
-				<a href="#" id="delete_empty_artists"><i class="fa fa-trash-o fa-fw"></i> Delete empty Artists</a>
+				<a href="javascript:void(0)" id="delete_empty_artists"><i class="fa fa-trash-o fa-fw"></i> Delete empty Artists</a>
 					<div id="emptyartistdialog" title="Confirm Artist Deletion" style="display:none" class="configtable">
 					    %if emptyArtists:
 						<h3>The following artists will be deleted:</h3>
@@ -137,28 +137,27 @@
 					        No empty artists found.
 					    %endif
 					</div>
-				<a href="#" onclick="doAjaxCall('forcePostProcess',$(this))" data-success="Post-Processor is being loaded" data-error="Error during Post-Processing"><i class="fa fa-wrench fa-fw"></i> Force Post-Process Albums in Download Folder</a>
+					<div id="post_process">
+					<a href="javascript:void(0)" class="btnOpenDialog"><i class="fa fa-wrench fa-fw"></i> Force Post-Process Albums in Download Folder</a>
+					</div>
 			</div>
 		</fieldset>
-		<form action="forcePostProcess" method="GET">
+		<fieldset>
-		            <fieldset>
+            <div class="row" id="post_process_alternate">
-		            <div class="row">
+                <label>Force Post-Process Albums in Alternate Folder</label>
-		                <label>Force Post-Process Albums in Alternate Folder</label>
+			    <input type="text" value="" name="dir" id="dir" size="50" />
-					    <input type="text" value="" name="dir" id="dir" size="50" /><input type="submit" />
+			    <input type="button" class="btnOpenDialog" value="Submit" />
-		            </div>
+            </div>
-		            </fieldset>
+        </fieldset>
-		      
+
-		</form>
+        <fieldset>
-		<form action="forcePostProcess" method="GET">
+            <div class="row" id="post_process_single">
-		            <fieldset>
+                <label>Post-Process Single Folder</label>
-		            <div class="row">
+			    <input type="text" value="" name="album_dir" id="album_dir" size="50" />
-		                <label>Post-Process Single Folder</label>
+			    <input type="button" class="btnOpenDialog" value="Submit" />
-					    <input type="text" value="" name="album_dir" id="album_dir" size="50" /><input type="submit" />
+            </div>
-		            </div>
+        </fieldset>
-		            </fieldset>
+
-		      
-		</form>
-		
 	</div>
 
 
@@ -167,9 +166,9 @@
 			<legend>Force Legacy</legend>
 				<p>Please note that these functions will take a significant amount of time to complete.</p>
 			<div class="links">
-				<a href="#" onclick="doAjaxCall('forceFullUpdate',$(this))" data-success="Update active artists successful" data-error="Error forcing update artists"><i class="fa fa-heart fa-fw"></i> Force Update Active Artists [Comprehensive]</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('forceFullUpdate',$(this))" data-success="Update active artists successful" data-error="Error forcing update artists"><i class="fa fa-heart fa-fw"></i> Force Update Active Artists [Comprehensive]</a>
 				<BR>
-				<a href="#" onclick="doAjaxCall('forceScan',$(this))" data-success="Library scan successful" data-error="Error forcing library scan"><i class="fa fa-refresh fa-fw"></i> Force Re-scan Library [Comprehensive]</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('forceScan',$(this))" data-success="Library scan successful" data-error="Error forcing library scan"><i class="fa fa-refresh fa-fw"></i> Force Re-scan Library [Comprehensive]</a>
 				<BR>
 				<small>*Warning: If you choose [Force Re-scan Library], any manually ignored/matched artists/albums will be reset to "unmatched".</small>
 
@@ -177,8 +176,7 @@
 		</fieldset>
 		
 	</div>
-
+	<div id="dialog-confirm"></div>
-
 
 </div>
 </%def>
@@ -187,6 +185,55 @@
 		function addScanAction() {
 			$('#autoadd').append('<input type="hidden" name="scan" value=1 />');
 		};
+
+		function fnOpenNormalDialog(id) {
+			if (id === "post_process"){
+				var url = "forcePostProcess?"
+			}
+			if (id === "post_process_alternate"){
+				var dir = $('#dir').val();
+				var url = "forcePostProcess?dir=" + dir + "&"
+			}
+			if (id === "post_process_single"){
+				var dir = $('#album_dir').val();
+				var url = "forcePostProcess?album_dir=" + dir + "&"
+			}
+			var t = $('<a data-success="Post-Processor is being loaded" data-error="Error during Post-Processing">');
+		    $("#dialog-confirm").html("Do you want to keep the original folder(s) after post-processing? If you click no, the folders still might be kept depending on your global settings");
+
+		    // Define the Dialog and its properties.
+		    $("#dialog-confirm").dialog({
+		        resizable: false,
+		        modal: true,
+		        title: "Keep original folder(s)?",
+		        height: 170,
+		        width: 400,
+		        buttons: {
+		            "Yes": function () {
+		                $(this).dialog('close');
+		                doAjaxCall(url + "keep_original_folder=True", t);
+		            },
+		                "No": function () {
+		                $(this).dialog('close');
+		                doAjaxCall(url + "keep_original_folder=False", t);
+		            }
+		        }
+		    });
+		}
+
+		$('.btnOpenDialog').click(function(e){
+			e.preventDefault();
+			var parentId = $(this).closest('div').prop('id');
+			fnOpenNormalDialog(parentId);
+		});
+
+		function callback(value) {
+		    if (value) {
+		        alert("Confirmed");
+		    } else {
+		        alert("Rejected");
+		    }
+		}
 		
 		function initThisPage() {
 			$('#manage_albums').click(function() {

data/interfaces/default/managemanual.html

@@ -43,9 +43,9 @@
 					<tr><td>Are you sure you want to reset Local Artist: ${album['ArtistName']} to unmatched?</td></tr>
 					<tr><td align="right"><BR>
 					%if album['AlbumStatus'] == "Ignored":
-						<button href="#" onclick="doAjaxCall('markManual?action=unignoreArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully reset ${album['ArtistName']} to unmatched">Reset Artist</button>
+						<button href="javascript:void(0)" onclick="doAjaxCall('markManual?action=unignoreArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully reset ${album['ArtistName']} to unmatched">Reset Artist</button>
 					%elif album['AlbumStatus'] == "Matched":
-						<button href="#" onclick="doAjaxCall('markManual?action=unmatchArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully restored ${album['ArtistName']} to unmatched">Reset Artist</button>
+						<button href="javascript:void(0)" onclick="doAjaxCall('markManual?action=unmatchArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully restored ${album['ArtistName']} to unmatched">Reset Artist</button>
 					%endif
 					</td></tr>
 					</table>
@@ -58,9 +58,9 @@
 					<tr><td>Are you sure you want to reset Local Album: ${album['AlbumTitle']} to unmatched?</td></tr>
 					<tr><td align="right"><BR>
 					%if album['AlbumStatus'] == "Ignored":
-						<button href="#" onclick="doAjaxCall('markManual?action=unignoreAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully reset ${album['AlbumTitle']} to unmatched">Reset Album</button>
+						<button href="javascript:void(0)" onclick="doAjaxCall('markManual?action=unignoreAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully reset ${album['AlbumTitle']} to unmatched">Reset Album</button>
 					%elif album['AlbumStatus'] == "Matched":
-						<button href="#" onclick="doAjaxCall('markManual?action=unmatchAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully reset ${album['AlbumTitle']} to unmatched">Reset Album</button>
+						<button href="javascript:void(0)" onclick="doAjaxCall('markManual?action=unmatchAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully reset ${album['AlbumTitle']} to unmatched">Reset Album</button>
 					%endif
 					</td></tr>
 					</table>

data/interfaces/default/manageunmatched.html

@@ -52,7 +52,7 @@
 					<table>
 					<tr><td>Are you sure you want to ignore Local Artist: ${album['ArtistName']} from future matching?</td></tr>
 					<tr><td align="right"><BR>
-					<button href="#" onclick="doAjaxCall('markUnmatched?action=ignoreArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully ignored ${album['ArtistName']} from future matching">Ignore Artist</button>
+					<button href="javascript:void(0)" onclick="doAjaxCall('markUnmatched?action=ignoreArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully ignored ${album['ArtistName']} from future matching">Ignore Artist</button>
 					</td></tr>
 					</table>
 				</div>
@@ -66,7 +66,7 @@
 						</select>
 						</td></tr>
 					<tr><td></td><td align="right"><BR>
-					<button href="#" onclick="artist_matcher(${count_albums}, '${old_artist_js}')">Match Artist</button>
+					<button href="javascript:void(0)" onclick="artist_matcher(${count_albums}, '${old_artist_js}')">Match Artist</button>
 					</td></tr>
 					</table>
 				</div>
@@ -78,7 +78,7 @@
 					<table>
 					<tr><td>Are you sure you want to ignore Local Album: ${album['AlbumTitle']} from future matching?</td></tr>
 					<tr><td align="right"><BR>
-					<button href="#" onclick="doAjaxCall('markUnmatched?action=ignoreAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully ignored ${album['AlbumTitle']} from future matching">Ignore Album</button>
+					<button href="javascript:void(0)" onclick="doAjaxCall('markUnmatched?action=ignoreAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully ignored ${album['AlbumTitle']} from future matching">Ignore Album</button>
 					</td></tr>
 					</table>
 				</div>
@@ -97,7 +97,7 @@
 					</select>
 					</td></tr>
 					<tr><td></td><td align="right"><BR>
-					<button href="#" onclick="album_matcher(${count_albums}, '${old_artist_js}', '${old_album_js}')">Match Album</button>
+					<button href="javascript:void(0)" onclick="album_matcher(${count_albums}, '${old_artist_js}', '${old_album_js}')">Match Album</button>
 					</td></tr>
 					</table>
 				</div>

data/interfaces/default/searchresults.html

@@ -17,8 +17,12 @@
                     <th id="reldate">Date</th>
                     <th id="scoresmall">Score</th>
                     <th id="mbrelid" style="display:none;"</th>
-                %else:
+                %elif type == 'artist':
                     <th id="artistname">Artist Name</th>
+                    <th id="score">Score</th>
+                %else:
+                    <th id="seriesname">Series Name</th>
+                    <th id="type">Type</th>
                     <th id="score">Score</th>
 				%endif
                 <th id="mb"></th>
@@ -40,8 +44,10 @@
 				<tr class="grade${grade}">
 				%if type == 'album':
 					<td id="albumart" style=" text-align: center; vertical-align: middle;"><div id="artistImg"><img title="${result['albumid']}" class="albumArt" height="50" width="50" onerror="tryCCA(this, '${caa_group_url}')"></div></td>
-                %else:
+                %elif type == 'artist':
 					<td id="albumart"><div id="artistImg"><img title="${result['id']}" class="albumArt" height="50" width="50"></div></td>
+                %else:
+                    <td id="albumart"></td>
 				%endif
 				%if type == 'album':
 					<td id="albumname"><a href="addReleaseById?rid=${result['albumid']}&rgid=${result['rgid']}" title="${albuminfo}">${result['title']}</a></td>
@@ -52,9 +58,14 @@
                     <td id="score"><a href="${result['albumurl']} "title="View on MusicBrainz"><div class="bar"><div class="score" style="width: ${result['score']}px">${result['score']}</div></div></a></td>
                     <td id="musicbrainz" style=" text-align: center; line-height: 0; vertical-align: middle;"><a href="${result['albumurl']}"><img src="interfaces/default/images/MusicBrainz_Album_Icon.png" title="View on MusicBrainz" height="20" width="20"></a></td>
                     <td id="mbrelid" style="display:none;">${result['albumid']}</td>
-                %else:
+                %elif type == 'artist':
                     <td id="artistname"><a href="addArtist?artistid=${result['id']}" title="${result['uniquename']}">${result['uniquename']}</a></td>
                     <td id="score"><a href="${result['url']} "title="View on MusicBrainz"><div class="bar"><div class="score" style="width: ${result['score']}px">${result['score']}</div></div></a></td>
+                    <td id="musicbrainz" style=" text-align: center; line-height: 0; vertical-align: middle;"><a href="${result['url']}"><img src="interfaces/default/images/MusicBrainz_Artist_Icon.png" title="View on MusicBrainz" height="20" width="20"></a></td>
+                %else:
+                    <td id="seriesname"><a href="addSeries?seriesid=${result['id']}" title="${result['uniquename']}">${result['uniquename']}</a></td>
+                    <td id="type">${result['type']}</td>
+                    <td id="score"><a href="${result['url']} "title="View on MusicBrainz"><div class="bar"><div class="score" style="width: ${result['score']}px">${result['score']}</div></div></a></td>
                     <td id="musicbrainz" style=" text-align: center; line-height: 0; vertical-align: middle;"><a href="${result['url']}"><img src="interfaces/default/images/MusicBrainz_Artist_Icon.png" title="View on MusicBrainz" height="20" width="20"></a></td>
 				%endif
 				</tr>

data/interfaces/default/upcoming.html

@@ -3,7 +3,7 @@
 <%def name="headerIncludes()">
 	<div id="subhead_container">
 		<div id="subhead_menu">
-			<a href="#" id="menu_link_scan" onclick="doAjaxCall('forceSearch',$(this))" data-success="Checking for wanted albums successful" data-error="Error checking wanted albums"><i class="fa fa-search"></i> Force Check</a>
+			<a href="javascript:void(0)" id="menu_link_scan" onclick="doAjaxCall('forceSearch',$(this))" data-success="Checking for wanted albums successful" data-error="Error checking wanted albums"><i class="fa fa-search"></i> Force Check</a>
 		</div>
 	</div>
 </%def>

headphones/__init__.py

@@ -16,20 +16,21 @@
 # NZBGet support added by CurlyMo <curlymoo1@gmail.com> as a part of
 # XBian - XBMC on the Raspberry Pi
 
-import os
 import sys
 import subprocess
 import threading
 import webbrowser
 import sqlite3
-import cherrypy
+import datetime
 
+import os
+import cherrypy
 from apscheduler.schedulers.background import BackgroundScheduler
 from apscheduler.triggers.interval import IntervalTrigger
-
 from headphones import versioncheck, logger
 import headphones.config
 
+
 # (append new extras to the end)
 POSSIBLE_EXTRAS = [
     "single",
@@ -93,7 +94,6 @@ UMASK = None
 
 
 def initialize(config_file):
-
     with INIT_LOCK:
 
         global CONFIG
@@ -130,11 +130,11 @@ def initialize(config_file):
 
                 if not QUIET:
                     sys.stderr.write("Unable to create the log directory. " \
-                        "Logging to screen only.\n")
+                                     "Logging to screen only.\n")
 
         # Start the logger, disable console if needed
         logger.initLogger(console=not QUIET, log_dir=CONFIG.LOG_DIR,
-            verbose=VERBOSE)
+                          verbose=VERBOSE)
 
         if not CONFIG.CACHE_DIR:
             # Put the cache dir in the data dir for now
@@ -175,7 +175,7 @@ def initialize(config_file):
                              version_lock_file, e)
 
         # Check for new versions
-        if CONFIG.CHECK_GITHUB_ON_STARTUP:
+        if CONFIG.CHECK_GITHUB and CONFIG.CHECK_GITHUB_ON_STARTUP:
             try:
                 LATEST_VERSION = versioncheck.checkGithub()
             except:
@@ -245,7 +245,6 @@ def daemonize():
 
 
 def launch_browser(host, port, root):
-
     if host == '0.0.0.0':
         host = 'localhost'
 
@@ -262,55 +261,79 @@ def launch_browser(host, port, root):
 
 def initialize_scheduler():
     """
-    Start the scheduled background tasks. Because this method can be called
+    Start the scheduled background tasks. Re-schedule if interval settings changed.
-    multiple times, the old tasks will be first removed.
     """
 
     from headphones import updater, searcher, librarysync, postprocessor, \
         torrentfinished
 
     with SCHED_LOCK:
-        # Remove all jobs first, because this method is also invoked when the
-        # settings are saved.
-        count = len(SCHED.get_jobs())
 
-        if count > 0:
+        # Check if scheduler should be started
-            logger.debug("Current number of background tasks: %d", count)
+        start_jobs = not len(SCHED.get_jobs())
-            SCHED.shutdown()
-            SCHED.remove_all_jobs()
 
         # Regular jobs
-        if CONFIG.UPDATE_DB_INTERVAL > 0:
+        minutes = CONFIG.SEARCH_INTERVAL
-            SCHED.add_job(updater.dbUpdate, trigger=IntervalTrigger(
+        schedule_job(searcher.searchforalbum, 'Search for Wanted', hours=0, minutes=minutes)
-                hours=CONFIG.UPDATE_DB_INTERVAL))
+
-        if CONFIG.SEARCH_INTERVAL > 0:
+        minutes = CONFIG.DOWNLOAD_SCAN_INTERVAL
-            SCHED.add_job(searcher.searchforalbum, trigger=IntervalTrigger(
+        schedule_job(postprocessor.checkFolder, 'Download Scan', hours=0, minutes=minutes)
-                minutes=CONFIG.SEARCH_INTERVAL))
+
-        if CONFIG.LIBRARYSCAN_INTERVAL > 0:
+        hours = CONFIG.LIBRARYSCAN_INTERVAL
-            SCHED.add_job(librarysync.libraryScan, trigger=IntervalTrigger(
+        schedule_job(librarysync.libraryScan, 'Library Scan', hours=hours, minutes=0)
-                hours=CONFIG.LIBRARYSCAN_INTERVAL))
+
-        if CONFIG.DOWNLOAD_SCAN_INTERVAL > 0:
+        hours = CONFIG.UPDATE_DB_INTERVAL
-            SCHED.add_job(postprocessor.checkFolder, trigger=IntervalTrigger(
+        schedule_job(updater.dbUpdate, 'MusicBrainz Update', hours=hours, minutes=0)
-                minutes=CONFIG.DOWNLOAD_SCAN_INTERVAL))
 
         # Update check
         if CONFIG.CHECK_GITHUB:
-            SCHED.add_job(versioncheck.checkGithub, trigger=IntervalTrigger(
+            if CONFIG.CHECK_GITHUB_INTERVAL:
-                minutes=CONFIG.CHECK_GITHUB_INTERVAL))
+                minutes = CONFIG.CHECK_GITHUB_INTERVAL
+            else:
+                minutes = 0
+            schedule_job(versioncheck.checkGithub, 'Check GitHub for updates', hours=0,
+                         minutes=minutes)
 
         # Remove Torrent + data if Post Processed and finished Seeding
-        if CONFIG.TORRENT_REMOVAL_INTERVAL > 0:
+        minutes = CONFIG.TORRENT_REMOVAL_INTERVAL
-            SCHED.add_job(torrentfinished.checkTorrentFinished,
+        schedule_job(torrentfinished.checkTorrentFinished, 'Torrent removal check', hours=0,
-                          trigger=IntervalTrigger(
+                     minutes=minutes)
-                              minutes=CONFIG.TORRENT_REMOVAL_INTERVAL))
 
         # Start scheduler
-        logger.info("(Re-)Scheduled %d background tasks", len(SCHED.get_jobs()))
+        if start_jobs and len(SCHED.get_jobs()):
-        SCHED.start()
+            try:
+                SCHED.start()
+            except Exception as e:
+                logger.info(e)
+
+                # Debug
+                # SCHED.print_jobs()
+
+
+def schedule_job(function, name, hours=0, minutes=0):
+    """
+    Start scheduled job if starting or restarting headphones.
+    Reschedule job if Interval Settings have changed.
+    Remove job if if Interval Settings changed to 0
+
+    """
+
+    job = SCHED.get_job(name)
+    if job:
+        if hours == 0 and minutes == 0:
+            SCHED.remove_job(name)
+            logger.info("Removed background task: %s", name)
+        elif job.trigger.interval != datetime.timedelta(hours=hours, minutes=minutes):
+            SCHED.reschedule_job(name, trigger=IntervalTrigger(
+                hours=hours, minutes=minutes))
+            logger.info("Re-scheduled background task: %s", name)
+    elif hours > 0 or minutes > 0:
+        SCHED.add_job(function, id=name, trigger=IntervalTrigger(
+            hours=hours, minutes=minutes))
+        logger.info("Scheduled background task: %s", name)
 
 
 def start():
-
     global started
 
     if _INITIALIZED:
@@ -325,16 +348,15 @@ def sig_handler(signum=None, frame=None):
 
 
 def dbcheck():
-
     conn = sqlite3.connect(DB_FILE)
     c = conn.cursor()
     c.execute(
-        'CREATE TABLE IF NOT EXISTS artists (ArtistID TEXT UNIQUE, ArtistName TEXT, ArtistSortName TEXT, DateAdded TEXT, Status TEXT, IncludeExtras INTEGER, LatestAlbum TEXT, ReleaseDate TEXT, AlbumID TEXT, HaveTracks INTEGER, TotalTracks INTEGER, LastUpdated TEXT, ArtworkURL TEXT, ThumbURL TEXT, Extras TEXT)')
+        'CREATE TABLE IF NOT EXISTS artists (ArtistID TEXT UNIQUE, ArtistName TEXT, ArtistSortName TEXT, DateAdded TEXT, Status TEXT, IncludeExtras INTEGER, LatestAlbum TEXT, ReleaseDate TEXT, AlbumID TEXT, HaveTracks INTEGER, TotalTracks INTEGER, LastUpdated TEXT, ArtworkURL TEXT, ThumbURL TEXT, Extras TEXT, Type TEXT, MetaCritic TEXT)')
     # ReleaseFormat here means CD,Digital,Vinyl, etc. If using the default
     # Headphones hybrid release, ReleaseID will equal AlbumID (AlbumID is
     # releasegroup id)
     c.execute(
-        'CREATE TABLE IF NOT EXISTS albums (ArtistID TEXT, ArtistName TEXT, AlbumTitle TEXT, AlbumASIN TEXT, ReleaseDate TEXT, DateAdded TEXT, AlbumID TEXT UNIQUE, Status TEXT, Type TEXT, ArtworkURL TEXT, ThumbURL TEXT, ReleaseID TEXT, ReleaseCountry TEXT, ReleaseFormat TEXT, SearchTerm TEXT)')
+        'CREATE TABLE IF NOT EXISTS albums (ArtistID TEXT, ArtistName TEXT, AlbumTitle TEXT, AlbumASIN TEXT, ReleaseDate TEXT, DateAdded TEXT, AlbumID TEXT UNIQUE, Status TEXT, Type TEXT, ArtworkURL TEXT, ThumbURL TEXT, ReleaseID TEXT, ReleaseCountry TEXT, ReleaseFormat TEXT, SearchTerm TEXT, CriticScore TEXT, UserScore TEXT)')
     # Format here means mp3, flac, etc.
     c.execute(
         'CREATE TABLE IF NOT EXISTS tracks (ArtistID TEXT, ArtistName TEXT, AlbumTitle TEXT, AlbumASIN TEXT, AlbumID TEXT, TrackTitle TEXT, TrackDuration, TrackID TEXT, TrackNumber INTEGER, Location TEXT, BitRate INTEGER, CleanName TEXT, Format TEXT, ReleaseID TEXT)')
@@ -560,12 +582,31 @@ def dbcheck():
     except sqlite3.OperationalError:
         c.execute('ALTER TABLE albums ADD COLUMN SearchTerm TEXT DEFAULT NULL')
 
+    try:
+        c.execute('SELECT CriticScore from albums')
+    except sqlite3.OperationalError:
+        c.execute('ALTER TABLE albums ADD COLUMN CriticScore TEXT DEFAULT NULL')
+
+    try:
+        c.execute('SELECT UserScore from albums')
+    except sqlite3.OperationalError:
+        c.execute('ALTER TABLE albums ADD COLUMN UserScore TEXT DEFAULT NULL')
+
+    try:
+        c.execute('SELECT Type from artists')
+    except sqlite3.OperationalError:
+        c.execute('ALTER TABLE artists ADD COLUMN Type TEXT DEFAULT NULL')
+
+    try:
+        c.execute('SELECT MetaCritic from artists')
+    except sqlite3.OperationalError:
+        c.execute('ALTER TABLE artists ADD COLUMN MetaCritic TEXT DEFAULT NULL')
+
     conn.commit()
     c.close()
 
 
 def shutdown(restart=False, update=False):
-
     cherrypy.engine.exit()
     SCHED.shutdown(wait=False)
 

headphones/albumswitcher.py

@@ -53,7 +53,6 @@ def switch(AlbumID, ReleaseID):
     c.get_artwork_from_cache(AlbumID=AlbumID)
 
     for track in newtrackdata:
-
         controlValueDict = {"TrackID": track['TrackID'],
                             "AlbumID": AlbumID}
 
@@ -79,15 +78,18 @@ def switch(AlbumID, ReleaseID):
     have_track_count = len(myDB.select(
         'SELECT * from tracks WHERE AlbumID=? AND Location IS NOT NULL', [AlbumID]))
 
-    if oldalbumdata['Status'] == 'Skipped' and ((have_track_count / float(total_track_count)) >= (headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0)):
+    if oldalbumdata['Status'] == 'Skipped' and ((have_track_count / float(total_track_count)) >= (
+        headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0)):
         myDB.action(
             'UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', AlbumID])
 
     # Update have track counts on index
     totaltracks = len(myDB.select(
-        'SELECT TrackTitle from tracks WHERE ArtistID=? AND AlbumID IN (SELECT AlbumID FROM albums WHERE Status != "Ignored")', [newalbumdata['ArtistID']]))
+        'SELECT TrackTitle from tracks WHERE ArtistID=? AND AlbumID IN (SELECT AlbumID FROM albums WHERE Status != "Ignored")',
+        [newalbumdata['ArtistID']]))
     havetracks = len(myDB.select(
-        'SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [newalbumdata['ArtistID']]))
+        'SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL',
+        [newalbumdata['ArtistID']]))
 
     controlValueDict = {"ArtistID": newalbumdata['ArtistID']}
 

headphones/api.py

@@ -13,20 +13,25 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import db, mb, importer, searcher, cache, postprocessor, versioncheck, logger
-
-import headphones
 import json
 
-cmd_list = ['getIndex', 'getArtist', 'getAlbum', 'getUpcoming', 'getWanted', 'getSimilar', 'getHistory', 'getLogs',
+from headphones import db, mb, updater, importer, searcher, cache, postprocessor, versioncheck, \
-            'findArtist', 'findAlbum', 'addArtist', 'delArtist', 'pauseArtist', 'resumeArtist', 'refreshArtist',
+    logger
-            'addAlbum', 'queueAlbum', 'unqueueAlbum', 'forceSearch', 'forceProcess', 'getVersion', 'checkGithub',
+import headphones
-            'shutdown', 'restart', 'update', 'getArtistArt', 'getAlbumArt', 'getArtistInfo', 'getAlbumInfo',
+
-            'getArtistThumb', 'getAlbumThumb', 'choose_specific_download', 'download_specific_release']
+cmd_list = ['getIndex', 'getArtist', 'getAlbum', 'getUpcoming', 'getWanted', 'getSnatched',
+            'getSimilar', 'getHistory', 'getLogs',
+            'findArtist', 'findAlbum', 'addArtist', 'delArtist', 'pauseArtist', 'resumeArtist',
+            'refreshArtist',
+            'addAlbum', 'queueAlbum', 'unqueueAlbum', 'forceSearch', 'forceProcess',
+            'forceActiveArtistsUpdate',
+            'getVersion', 'checkGithub', 'shutdown', 'restart', 'update', 'getArtistArt',
+            'getAlbumArt',
+            'getArtistInfo', 'getAlbumInfo', 'getArtistThumb', 'getAlbumThumb', 'clearLogs',
+            'choose_specific_download', 'download_specific_release']
 
 
 class Api(object):
-
     def __init__(self):
 
         self.apikey = None
@@ -165,12 +170,23 @@ class Api(object):
             "SELECT * from albums WHERE Status='Wanted'")
         return
 
+    def _getSnatched(self, **kwargs):
+        self.data = self._dic_from_query(
+            "SELECT * from albums WHERE Status='Snatched'")
+        return
+
     def _getSimilar(self, **kwargs):
         self.data = self._dic_from_query('SELECT * from lastfmcloud')
         return
 
     def _getLogs(self, **kwargs):
-        pass
+        self.data = headphones.LOG_LIST
+        return
+
+    def _clearLogs(self, **kwargs):
+        headphones.LOG_LIST = []
+        self.data = 'Cleared log'
+        return
 
     def _findArtist(self, **kwargs):
         if 'name' not in kwargs:
@@ -316,10 +332,18 @@ class Api(object):
         searcher.searchforalbum()
 
     def _forceProcess(self, **kwargs):
-        self.dir = None
+        if 'album_dir' in kwargs:
-        if 'dir' in kwargs:
+            album_dir = kwargs['album_dir']
+            dir = None
+            postprocessor.forcePostProcess(self, dir, album_dir)
+        elif 'dir' in kwargs:
             self.dir = kwargs['dir']
-        postprocessor.forcePostProcess(self.dir)
+            postprocessor.forcePostProcess(self.dir)
+        else:
+            postprocessor.forcePostProcess()
+
+    def _forceActiveArtistsUpdate(self, **kwargs):
+        updater.dbUpdate()
 
     def _getVersion(self, **kwargs):
         self.data = {
@@ -417,7 +441,6 @@ class Api(object):
         results_as_dicts = []
 
         for result in results:
-
             result_dict = {
                 'title': result[0],
                 'size': result[1],

headphones/cache.py

@@ -14,8 +14,8 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
 import os
-import headphones
 
+import headphones
 from headphones import db, helpers, logger, lastfm, request
 
 LASTFM_API_KEY = "690e1ed3bc00bc91804cd8f7fe5ed6d4"
@@ -23,28 +23,30 @@ LASTFM_API_KEY = "690e1ed3bc00bc91804cd8f7fe5ed6d4"
 
 class Cache(object):
     """
-    This class deals with getting, storing and serving up artwork (album
+    This class deals with getting, storing and serving up artwork (album art,
-    art, artist images, etc) and info/descriptions (album info, artist descrptions)
+    artist images, etc) and info/descriptions (album info, artist descrptions)
-    to and from the cache folder. This can be called from within a web interface,
+    to and from the cache folder. This can be called from within a web
-    for example, using the helper functions getInfo(id) and getArtwork(id), to utilize the cached
+    interface. For example, using the helper functions `getInfo(id)` and
-    images rather than having to retrieve them every time the page is reloaded.
+    `getArtwork(id)`, to utilize the cached images rather than having to
+    retrieve them every time the page is reloaded.
 
-    So you can call cache.getArtwork(id) which will return an absolute path
+    You can call `getArtwork(id)` which will return an absolute path to the
-    to the image file on the local machine, or if the cache directory
+    image file on the local machine, or if the cache directory doesn't exist,
-    doesn't exist, or can not be written to, it will return a url to the image.
+    or can not be written to, it will return a url to the image.
 
-    Call cache.getInfo(id) to grab the artist/album info; will return the text description
+    Call `getInfo(id)` to grab the artist/album info. This will return the
+    text description.
 
-    The basic format for art in the cache is <musicbrainzid>.<date>.<ext>
+    The basic format for art in the cache is `<musicbrainzid>.<date>.<ext>`
-    and for info it is <musicbrainzid>.<date>.txt
+    and for info it is `<musicbrainzid>.<date>.txt`
     """
 
     path_to_art_cache = os.path.join(headphones.CONFIG.CACHE_DIR, 'artwork')
 
     def __init__(self):
         self.id = None
-        self.id_type = None # 'artist' or 'album' - set automatically depending on whether ArtistID or AlbumID is passed
+        self.id_type = None  # 'artist' or 'album' - set automatically depending on whether ArtistID or AlbumID is passed
-        self.query_type = None # 'artwork','thumb' or 'info' - set automatically
+        self.query_type = None  # 'artwork','thumb' or 'info' - set automatically
 
         self.artwork_files = []
         self.thumb_files = []
@@ -180,13 +182,18 @@ class Cache(object):
         if ArtistID:
             self.id = ArtistID
             self.id_type = 'artist'
-            db_info = myDB.action('SELECT Summary, Content, LastUpdated FROM descriptions WHERE ArtistID=?', [self.id]).fetchone()
+            db_info = myDB.action(
+                'SELECT Summary, Content, LastUpdated FROM descriptions WHERE ArtistID=?',
+                [self.id]).fetchone()
         else:
             self.id = AlbumID
             self.id_type = 'album'
-            db_info = myDB.action('SELECT Summary, Content, LastUpdated FROM descriptions WHERE ReleaseGroupID=?', [self.id]).fetchone()
+            db_info = myDB.action(
+                'SELECT Summary, Content, LastUpdated FROM descriptions WHERE ReleaseGroupID=?',
+                [self.id]).fetchone()
 
-        if not db_info or not db_info['LastUpdated'] or not self._is_current(date=db_info['LastUpdated']):
+        if not db_info or not db_info['LastUpdated'] or not self._is_current(
+                date=db_info['LastUpdated']):
 
             self._update_cache()
             info_dict = {'Summary': self.info_summary, 'Content': self.info_content}
@@ -197,10 +204,10 @@ class Cache(object):
             return info_dict
 
     def get_image_links(self, ArtistID=None, AlbumID=None):
-        '''
+        """
         Here we're just going to open up the last.fm url, grab the image links and return them
         Won't save any image urls, or save the artwork in the cache. Useful for search results, etc.
-        '''
+        """
         if ArtistID:
 
             self.id_type = 'artist'
@@ -271,9 +278,10 @@ class Cache(object):
                     logger.warn('Error deleting file from the cache: %s', thumb_file)
 
     def _update_cache(self):
-        '''
+        """
         Since we call the same url for both info and artwork, we'll update both at the same time
-        '''
+        """
+
         myDB = db.DBConnection()
 
         # Since lastfm uses release ids rather than release group ids for albums, we have to do a artist + album search for albums
@@ -306,14 +314,19 @@ class Cache(object):
                 logger.debug('No artist thumbnail image found')
 
         else:
-
+            dbalbum = myDB.action(
-            dbalbum = myDB.action('SELECT ArtistName, AlbumTitle, ReleaseID FROM albums WHERE AlbumID=?', [self.id]).fetchone()
+                'SELECT ArtistName, AlbumTitle, ReleaseID FROM albums WHERE AlbumID=?',
+                [self.id]).fetchone()
             if dbalbum['ReleaseID'] != self.id:
-                data = lastfm.request_lastfm("album.getinfo", mbid=dbalbum['ReleaseID'], api_key=LASTFM_API_KEY)
+                data = lastfm.request_lastfm("album.getinfo", mbid=dbalbum['ReleaseID'],
+                                             api_key=LASTFM_API_KEY)
                 if not data:
-                    data = lastfm.request_lastfm("album.getinfo", artist=dbalbum['ArtistName'], album=dbalbum['AlbumTitle'], api_key=LASTFM_API_KEY)
+                    data = lastfm.request_lastfm("album.getinfo", artist=dbalbum['ArtistName'],
+                                                 album=dbalbum['AlbumTitle'],
+                                                 api_key=LASTFM_API_KEY)
             else:
-                data = lastfm.request_lastfm("album.getinfo", artist=dbalbum['ArtistName'], album=dbalbum['AlbumTitle'], api_key=LASTFM_API_KEY)
+                data = lastfm.request_lastfm("album.getinfo", artist=dbalbum['ArtistName'],
+                                             album=dbalbum['AlbumTitle'], api_key=LASTFM_API_KEY)
 
             if not data:
                 return
@@ -339,7 +352,8 @@ class Cache(object):
             if not thumb_url:
                 logger.debug('No album thumbnail image found')
 
-        #Save the content & summary to the database no matter what if we've opened up the url
+        # Save the content & summary to the database no matter what if we've
+        # opened up the url
         if self.id_type == 'artist':
             controlValueDict = {"ArtistID": self.id}
         else:
@@ -354,7 +368,8 @@ class Cache(object):
         # Save the image URL to the database
         if image_url:
             if self.id_type == 'artist':
-                myDB.action('UPDATE artists SET ArtworkURL=? WHERE ArtistID=?', [image_url, self.id])
+                myDB.action('UPDATE artists SET ArtworkURL=? WHERE ArtistID=?',
+                            [image_url, self.id])
             else:
                 myDB.action('UPDATE albums SET ArtworkURL=? WHERE AlbumID=?', [image_url, self.id])
 
@@ -365,7 +380,8 @@ class Cache(object):
             else:
                 myDB.action('UPDATE albums SET ThumbURL=? WHERE AlbumID=?', [thumb_url, self.id])
 
-        # Should we grab the artwork here if we're just grabbing thumbs or info?? Probably not since the files can be quite big
+        # Should we grab the artwork here if we're just grabbing thumbs or
+        # info? Probably not since the files can be quite big
         if image_url and self.query_type == 'artwork':
             artwork = request.request_content(image_url, timeout=20)
 
@@ -374,12 +390,14 @@ class Cache(object):
                 if not os.path.isdir(self.path_to_art_cache):
                     try:
                         os.makedirs(self.path_to_art_cache)
-                    except Exception as e:
+                        os.chmod(self.path_to_art_cache,
+                                 int(headphones.CONFIG.FOLDER_PERMISSIONS, 8))
+                    except OSError as e:
                         logger.error('Unable to create artwork cache dir. Error: %s', e)
                         self.artwork_errors = True
                         self.artwork_url = image_url
 
-                #Delete the old stuff
+                # Delete the old stuff
                 for artwork_file in self.artwork_files:
                     try:
                         os.remove(artwork_file)
@@ -388,17 +406,22 @@ class Cache(object):
 
                 ext = os.path.splitext(image_url)[1]
 
-                artwork_path = os.path.join(self.path_to_art_cache, self.id + '.' + helpers.today() + ext)
+                artwork_path = os.path.join(self.path_to_art_cache,
+                                            self.id + '.' + helpers.today() + ext)
                 try:
                     with open(artwork_path, 'wb') as f:
                         f.write(artwork)
-                except IOError as e:
+
+                    os.chmod(artwork_path, int(headphones.CONFIG.FILE_PERMISSIONS, 8))
+                except (OSError, IOError) as e:
                     logger.error('Unable to write to the cache dir: %s', e)
                     self.artwork_errors = True
                     self.artwork_url = image_url
 
-        # Grab the thumbnail as well if we're getting the full artwork (as long as it's missing/outdated
+        # Grab the thumbnail as well if we're getting the full artwork (as long
-        if thumb_url and self.query_type in ['thumb', 'artwork'] and not (self.thumb_files and self._is_current(self.thumb_files[0])):
+        # as it's missing/outdated.
+        if thumb_url and self.query_type in ['thumb', 'artwork'] and not (
+            self.thumb_files and self._is_current(self.thumb_files[0])):
             artwork = request.request_content(thumb_url, timeout=20)
 
             if artwork:
@@ -406,32 +429,36 @@ class Cache(object):
                 if not os.path.isdir(self.path_to_art_cache):
                     try:
                         os.makedirs(self.path_to_art_cache)
-                    except Exception as e:
+                        os.chmod(self.path_to_art_cache,
+                                 int(headphones.CONFIG.FOLDER_PERMISSIONS, 8))
+                    except OSError as e:
                         logger.error('Unable to create artwork cache dir. Error: %s' + e)
                         self.thumb_errors = True
                         self.thumb_url = thumb_url
 
-                #Delete the old stuff
+                # Delete the old stuff
                 for thumb_file in self.thumb_files:
                     try:
                         os.remove(thumb_file)
-                    except Exception as e:
+                    except OSError as e:
                         logger.error('Error deleting file from the cache: %s', thumb_file)
 
                 ext = os.path.splitext(image_url)[1]
 
-                thumb_path = os.path.join(self.path_to_art_cache, 'T_' + self.id + '.' + helpers.today() + ext)
+                thumb_path = os.path.join(self.path_to_art_cache,
+                                          'T_' + self.id + '.' + helpers.today() + ext)
                 try:
                     with open(thumb_path, 'wb') as f:
                         f.write(artwork)
-                except IOError as e:
+
+                    os.chmod(thumb_path, int(headphones.CONFIG.FILE_PERMISSIONS, 8))
+                except (OSError, IOError) as e:
                     logger.error('Unable to write to the cache dir: %s', e)
                     self.thumb_errors = True
                     self.thumb_url = image_url
 
 
 def getArtwork(ArtistID=None, AlbumID=None):
-
     c = Cache()
     artwork_path = c.get_artwork_from_cache(ArtistID, AlbumID)
 
@@ -446,7 +473,6 @@ def getArtwork(ArtistID=None, AlbumID=None):
 
 
 def getThumb(ArtistID=None, AlbumID=None):
-
     c = Cache()
     artwork_path = c.get_thumb_from_cache(ArtistID, AlbumID)
 
@@ -461,16 +487,13 @@ def getThumb(ArtistID=None, AlbumID=None):
 
 
 def getInfo(ArtistID=None, AlbumID=None):
-
     c = Cache()
-
     info_dict = c.get_info_from_cache(ArtistID, AlbumID)
 
     return info_dict
 
 
 def getImageLinks(ArtistID=None, AlbumID=None):
-
     c = Cache()
     image_links = c.get_image_links(ArtistID, AlbumID)
 

headphones/classes.py

@@ -13,9 +13,9 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-#########################################
+#######################################
-## Stolen from Sick-Beard's classes.py ##
+# Stolen from Sick-Beard's classes.py #
-#########################################
+#######################################
 
 
 import urllib
@@ -133,4 +133,5 @@ class Proper:
         self.episode = -1
 
     def __str__(self):
-        return str(self.date) + " " + self.name + " " + str(self.season) + "x" + str(self.episode) + " of " + str(self.tvdbid)
+        return str(self.date) + " " + self.name + " " + str(self.season) + "x" + str(
+            self.episode) + " of " + str(self.tvdbid)

headphones/common.py

@@ -20,15 +20,16 @@ Created on Aug 1, 2011
 '''
 import platform
 import operator
+
 import os
 import re
-
 from headphones import version
 
-#Identify Our Application
+
+# Identify Our Application
 USER_AGENT = 'Headphones/-' + version.HEADPHONES_VERSION + ' (' + platform.system() + ' ' + platform.release() + ')'
 
-### Notification Types
+# Notification Types
 NOTIFY_SNATCH = 1
 NOTIFY_DOWNLOAD = 2
 
@@ -36,26 +37,25 @@ notifyStrings = {}
 notifyStrings[NOTIFY_SNATCH] = "Started Download"
 notifyStrings[NOTIFY_DOWNLOAD] = "Download Finished"
 
-### Release statuses
+# Release statuses
-UNKNOWN = -1 # should never happen
+UNKNOWN = -1  # should never happen
-UNAIRED = 1 # releases that haven't dropped yet
+UNAIRED = 1  # releases that haven't dropped yet
-SNATCHED = 2 # qualified with quality
+SNATCHED = 2  # qualified with quality
-WANTED = 3 # releases we don't have but want to get
+WANTED = 3  # releases we don't have but want to get
-DOWNLOADED = 4 # qualified with quality
+DOWNLOADED = 4  # qualified with quality
-SKIPPED = 5 # releases we don't want
+SKIPPED = 5  # releases we don't want
-ARCHIVED = 6 # releases that you don't have locally (counts toward download completion stats)
+ARCHIVED = 6  # releases that you don't have locally (counts toward download completion stats)
-IGNORED = 7 # releases that you don't want included in your download stats
+IGNORED = 7  # releases that you don't want included in your download stats
-SNATCHED_PROPER = 9 # qualified with quality
+SNATCHED_PROPER = 9  # qualified with quality
 
 
 class Quality:
-
     NONE = 0
-    B192 = 1 << 1     # 2
+    B192 = 1 << 1  # 2
-    VBR = 1 << 2     # 4
+    VBR = 1 << 2  # 4
-    B256 = 1 << 3     # 8
+    B256 = 1 << 3  # 8
-    B320 = 1 << 4     #16
+    B320 = 1 << 4  # 16
-    FLAC = 1 << 5     #32
+    FLAC = 1 << 5  # 32
 
     # put these bits at the other end of the spectrum, far enough out that they shouldn't interfere
     UNKNOWN = 1 << 15
@@ -75,7 +75,8 @@ class Quality:
     def _getStatusStrings(status):
         toReturn = {}
         for x in Quality.qualityStrings.keys():
-            toReturn[Quality.compositeStatus(status, x)] = Quality.statusPrefixes[status] + " (" + Quality.qualityStrings[x] + ")"
+            toReturn[Quality.compositeStatus(status, x)] = Quality.statusPrefixes[status] + " (" + \
+                                                           Quality.qualityStrings[x] + ")"
         return toReturn
 
     @staticmethod
@@ -103,6 +104,9 @@ class Quality:
     @staticmethod
     def nameQuality(name):
 
+        def checkName(list, func):
+            return func([re.search(x, name, re.I) for x in list])
+
         name = os.path.basename(name)
 
         # if we have our exact text then assume we put it there
@@ -115,9 +119,7 @@ class Quality:
             if regex_match:
                 return x
 
-        checkName = lambda list, func: func([re.search(x, name, re.I) for x in list])
+        # TODO: fix quality checking here
-
-        #TODO: fix quality checking here
         if checkName(["mp3", "192"], any) and not checkName(["flac"], all):
             return Quality.B192
         elif checkName(["mp3", "256"], any) and not checkName(["flac"], all):
@@ -131,7 +133,6 @@ class Quality:
 
     @staticmethod
     def assumeQuality(name):
-
         if name.lower().endswith(".mp3"):
             return Quality.MP3
         elif name.lower().endswith(".flac"):
@@ -167,13 +168,16 @@ class Quality:
     SNATCHED = None
     SNATCHED_PROPER = None
 
+
 Quality.DOWNLOADED = [Quality.compositeStatus(DOWNLOADED, x) for x in Quality.qualityStrings.keys()]
 Quality.SNATCHED = [Quality.compositeStatus(SNATCHED, x) for x in Quality.qualityStrings.keys()]
-Quality.SNATCHED_PROPER = [Quality.compositeStatus(SNATCHED_PROPER, x) for x in Quality.qualityStrings.keys()]
+Quality.SNATCHED_PROPER = [Quality.compositeStatus(SNATCHED_PROPER, x) for x in
+                           Quality.qualityStrings.keys()]
 
 MP3 = Quality.combineQualities([Quality.B192, Quality.B256, Quality.B320, Quality.VBR], [])
 LOSSLESS = Quality.combineQualities([Quality.FLAC], [])
-ANY = Quality.combineQualities([Quality.B192, Quality.B256, Quality.B320, Quality.VBR, Quality.FLAC], [])
+ANY = Quality.combineQualities(
+    [Quality.B192, Quality.B256, Quality.B320, Quality.VBR, Quality.FLAC], [])
 
 qualityPresets = (MP3, LOSSLESS, ANY)
 qualityPresetStrings = {MP3: "MP3 (All bitrates 192+)",

headphones/config.py

@@ -1,7 +1,8 @@
-import headphones.logger
 import itertools
+
 import os
 import re
+import headphones.logger
 from configobj import ConfigObj
 
 
@@ -14,6 +15,7 @@ def bool_int(value):
             value = 0
     return int(bool(value))
 
+
 _CONFIG_DEFINITIONS = {
     'ADD_ALBUM_ART': (int, 'General', 0),
     'ADVANCEDENCODER': (str, 'General', ''),
@@ -44,16 +46,30 @@ _CONFIG_DEFINITIONS = {
     'CUE_SPLIT': (int, 'General', 1),
     'CUE_SPLIT_FLAC_PATH': (str, 'General', ''),
     'CUE_SPLIT_SHNTOOL_PATH': (str, 'General', ''),
+    'CUSTOMAUTH': (int, 'General', 0),
     'CUSTOMHOST': (str, 'General', 'localhost'),
+    'CUSTOMPASS': (str, 'General', ''),
     'CUSTOMPORT': (int, 'General', 5000),
     'CUSTOMSLEEP': (int, 'General', 1),
+    'CUSTOMUSER': (str, 'General', ''),
     'DELETE_LOSSLESS_FILES': (int, 'General', 1),
     'DESTINATION_DIR': (str, 'General', ''),
     'DETECT_BITRATE': (int, 'General', 0),
+    'DO_NOT_PROCESS_UNMATCHED': (int, 'General', 0),
     'DOWNLOAD_DIR': (str, 'General', ''),
     'DOWNLOAD_SCAN_INTERVAL': (int, 'General', 5),
     'DOWNLOAD_TORRENT_DIR': (str, 'General', ''),
     'DO_NOT_OVERRIDE_GIT_BRANCH': (int, 'General', 0),
+    'EMAIL_ENABLED': (int, 'Email', 0),
+    'EMAIL_FROM': (str, 'Email', ''),
+    'EMAIL_TO': (str, 'Email', ''),
+    'EMAIL_SMTP_SERVER': (str, 'Email', ''),
+    'EMAIL_SMTP_USER': (str, 'Email', ''),
+    'EMAIL_SMTP_PASSWORD': (str, 'Email', ''),
+    'EMAIL_SMTP_PORT': (int, 'Email', 25),
+    'EMAIL_SSL': (int, 'Email', 0),
+    'EMAIL_TLS': (int, 'Email', 0),
+    'EMAIL_ONSNATCH': (int, 'Email', 0),
     'EMBED_ALBUM_ART': (int, 'General', 0),
     'EMBED_LYRICS': (int, 'General', 0),
     'ENABLE_HTTPS': (int, 'General', 0),
@@ -68,10 +84,13 @@ _CONFIG_DEFINITIONS = {
     'ENCODER_PATH': (str, 'General', ''),
     'EXTRAS': (str, 'General', ''),
     'EXTRA_NEWZNABS': (list, 'Newznab', ''),
+    'EXTRA_TORZNABS': (list, 'Torznab', ''),
     'FILE_FORMAT': (str, 'General', 'Track Artist - Album [Year] - Title'),
     'FILE_PERMISSIONS': (str, 'General', '0644'),
+    'FILE_PERMISSIONS_ENABLED': (bool_int, 'General', True),
     'FILE_UNDERSCORES': (int, 'General', 0),
     'FOLDER_FORMAT': (str, 'General', 'Artist/Album [Year]'),
+    'FOLDER_PERMISSIONS_ENABLED': (bool_int, 'General', True),
     'FOLDER_PERMISSIONS': (str, 'General', '0755'),
     'FREEZE_DB': (int, 'General', 0),
     'GIT_BRANCH': (str, 'General', 'master'),
@@ -86,12 +105,14 @@ _CONFIG_DEFINITIONS = {
     'HPUSER': (str, 'General', ''),
     'HTTPS_CERT': (str, 'General', ''),
     'HTTPS_KEY': (str, 'General', ''),
-    'HTTP_HOST': (str, 'General', '0.0.0.0'),
+    'HTTP_HOST': (str, 'General', 'localhost'),
     'HTTP_PASSWORD': (str, 'General', ''),
     'HTTP_PORT': (int, 'General', 8181),
     'HTTP_PROXY': (int, 'General', 0),
     'HTTP_ROOT': (str, 'General', '/'),
     'HTTP_USERNAME': (str, 'General', ''),
+    'IDTAG': (int, 'Beets', 0),
+    'IGNORE_CLEAN_RELEASES': (int, 'General', 0),
     'IGNORED_WORDS': (str, 'General', ''),
     'IGNORED_FOLDERS': (list, 'Advanced', []),
     'IGNORED_FILES': (list, 'Advanced', []),
@@ -139,10 +160,11 @@ _CONFIG_DEFINITIONS = {
     'NZBSORG_HASH': (str, 'NZBsorg', ''),
     'NZBSORG_UID': (str, 'NZBsorg', ''),
     'NZB_DOWNLOADER': (int, 'General', 0),
+    'OFFICIAL_RELEASES_ONLY': (int, 'General', 0),
     'OMGWTFNZBS': (int, 'omgwtfnzbs', 0),
     'OMGWTFNZBS_APIKEY': (str, 'omgwtfnzbs', ''),
     'OMGWTFNZBS_UID': (str, 'omgwtfnzbs', ''),
-    'OPEN_MAGNET_LINKS': (int, 'General', 0),  # 0: Ignore, 1: Open, 2: Convert
+    'OPEN_MAGNET_LINKS': (int, 'General', 0),  # 0: Ignore, 1: Open, 2: Convert, 3: Embed (rtorrent)
     'MAGNET_LINKS': (int, 'General', 0),
     'OSX_NOTIFY_APP': (str, 'OSX_Notify', '/Applications/Headphones'),
     'OSX_NOTIFY_ENABLED': (int, 'OSX_Notify', 0),
@@ -160,6 +182,7 @@ _CONFIG_DEFINITIONS = {
     'PLEX_SERVER_HOST': (str, 'Plex', ''),
     'PLEX_UPDATE': (int, 'Plex', 0),
     'PLEX_USERNAME': (str, 'Plex', ''),
+    'PLEX_TOKEN': (str, 'Plex', ''),
     'PREFERRED_BITRATE': (str, 'General', ''),
     'PREFERRED_BITRATE_ALLOW_LOSSLESS': (int, 'General', 0),
     'PREFERRED_BITRATE_HIGH_BUFFER': (int, 'General', 0),
@@ -184,7 +207,10 @@ _CONFIG_DEFINITIONS = {
     'PUSHOVER_ONSNATCH': (int, 'Pushover', 0),
     'PUSHOVER_PRIORITY': (int, 'Pushover', 0),
     'RENAME_FILES': (int, 'General', 0),
+    'RENAME_UNPROCESSED': (bool_int, 'General', 1),
+    'RENAME_FROZEN': (bool_int, 'General', 1),
     'REPLACE_EXISTING_FOLDERS': (int, 'General', 0),
+    'KEEP_ORIGINAL_FOLDER': (int, 'General', 0),
     'REQUIRED_WORDS': (str, 'General', ''),
     'RUTRACKER': (int, 'Rutracker', 0),
     'RUTRACKER_PASSWORD': (str, 'Rutracker', ''),
@@ -201,6 +227,8 @@ _CONFIG_DEFINITIONS = {
     'SONGKICK_ENABLED': (int, 'Songkick', 1),
     'SONGKICK_FILTER_ENABLED': (int, 'Songkick', 0),
     'SONGKICK_LOCATION': (str, 'Songkick', ''),
+    'STRIKE': (int, 'Strike', 0),
+    'STRIKE_RATIO': (str, 'Strike', ''),
     'SUBSONIC_ENABLED': (int, 'Subsonic', 0),
     'SUBSONIC_HOST': (str, 'Subsonic', ''),
     'SUBSONIC_PASSWORD': (str, 'Subsonic', ''),
@@ -209,6 +237,10 @@ _CONFIG_DEFINITIONS = {
     'TORRENTBLACKHOLE_DIR': (str, 'General', ''),
     'TORRENT_DOWNLOADER': (int, 'General', 0),
     'TORRENT_REMOVAL_INTERVAL': (int, 'General', 720),
+    'TORZNAB': (int, 'Torznab', 0),
+    'TORZNAB_APIKEY': (str, 'Torznab', ''),
+    'TORZNAB_ENABLED': (int, 'Torznab', 1),
+    'TORZNAB_HOST': (str, 'Torznab', ''),
     'TRANSMISSION_HOST': (str, 'Transmission', ''),
     'TRANSMISSION_PASSWORD': (str, 'Transmission', ''),
     'TRANSMISSION_USERNAME': (str, 'Transmission', ''),
@@ -224,6 +256,7 @@ _CONFIG_DEFINITIONS = {
     'UTORRENT_PASSWORD': (str, 'uTorrent', ''),
     'UTORRENT_USERNAME': (str, 'uTorrent', ''),
     'VERIFY_SSL_CERT': (bool_int, 'Advanced', 1),
+    'WAIT_UNTIL_RELEASE_DATE': (int, 'General', 0),
     'WAFFLES': (int, 'Waffles', 0),
     'WAFFLES_PASSKEY': (str, 'Waffles', ''),
     'WAFFLES_RATIO': (str, 'Waffles', ''),
@@ -241,6 +274,7 @@ _CONFIG_DEFINITIONS = {
     'XLDPROFILE': (str, 'General', '')
 }
 
+
 # pylint:disable=R0902
 # it might be nice to refactor for fewer instance variables
 class Config(object):
@@ -317,7 +351,7 @@ class Config(object):
         """ Return the extra newznab tuples """
         extra_newznabs = list(
             itertools.izip(*[itertools.islice(self.EXTRA_NEWZNABS, i, None, 3)
-            for i in range(3)])
+                             for i in range(3)])
         )
         return extra_newznabs
 
@@ -332,6 +366,25 @@ class Config(object):
             extra_newznabs.append(item)
         self.EXTRA_NEWZNABS = extra_newznabs
 
+    def get_extra_torznabs(self):
+        """ Return the extra torznab tuples """
+        extra_torznabs = list(
+            itertools.izip(*[itertools.islice(self.EXTRA_TORZNABS, i, None, 3)
+                             for i in range(3)])
+        )
+        return extra_torznabs
+
+    def clear_extra_torznabs(self):
+        """ Forget about the configured extra torznabs """
+        self.EXTRA_TORZNABS = []
+
+    def add_extra_torznab(self, torznab):
+        """ Add a new extra torznab """
+        extra_torznabs = self.EXTRA_TORZNABS
+        for item in torznab:
+            extra_torznabs.append(item)
+        self.EXTRA_TORZNABS = extra_torznabs
+
     def __getattr__(self, name):
         """
         Returns something from the ini unless it is a real property

headphones/cuesplit.py

@@ -15,13 +15,13 @@
 
 # Most of this lifted from here: https://github.com/SzieberthAdam/gneposis-cdgrab
 
-import os
 import sys
-import re
 import subprocess
 import copy
 import glob
 
+import os
+import re
 import headphones
 from headphones import logger
 from mutagen.flac import FLAC
@@ -62,9 +62,7 @@ WAVE_FILE_TYPE_BY_EXTENSION = {
     '.flac': 'Free Lossless Audio Codec'
 }
 
-# TODO: Only alow flac for now
+# SHNTOOL_COMPATIBLE = ("Free Lossless Audio Codec", "Waveform Audio", "Monkey's Audio")
-#SHNTOOL_COMPATIBLE = ('Waveform Audio', 'WavPack', 'Free Lossless Audio Codec')
-SHNTOOL_COMPATIBLE = ('Free Lossless Audio Codec')
 
 # TODO: Make this better!
 # this module-level variable is bad. :(
@@ -109,10 +107,10 @@ def split_baby(split_file, split_cmd):
         env['PATH'] += os.pathsep + headphones.CONFIG.CUE_SPLIT_FLAC_PATH
 
     process = subprocess.Popen(split_cmd, startupinfo=startupinfo,
-
+                               stdin=open(os.devnull, 'rb'), stdout=subprocess.PIPE,
-    stdin=open(os.devnull, 'rb'), stdout=subprocess.PIPE,
+                               stderr=subprocess.PIPE, env=env)
-    stderr=subprocess.PIPE, env=env)
     stdout, stderr = process.communicate()
+
     if process.returncode:
         logger.error('Split failed for %s', split_file.decode(headphones.SYS_ENCODING, 'replace'))
         out = stdout if stdout else stderr
@@ -290,7 +288,7 @@ class CueFile(File):
             global line_content
             c = self.content.splitlines()
             header_dict = {}
-            #remaining_headers = CUE_HEADER
+            # remaining_headers = CUE_HEADER
             remaining_headers = copy.copy(CUE_HEADER)
             line_index = 0
             match = True
@@ -316,7 +314,8 @@ class CueFile(File):
             line_content = c[line_index]
             search_result = re.search(CUE_TRACK, line_content, re.I)
             if not search_result:
-                raise ValueError('inconsistent CUE sheet, TRACK expected at line {0}'.format(line_index + 1))
+                raise ValueError(
+                    'inconsistent CUE sheet, TRACK expected at line {0}'.format(line_index + 1))
             track_nr = int(search_result.group(1))
             line_index += 1
             next_track = False
@@ -355,7 +354,8 @@ class CueFile(File):
                         track_meta['dcpflag'] = True
                         line_index += 1
                     else:
-                        raise ValueError('unknown entry in track error, line {0}'.format(line_index + 1))
+                        raise ValueError(
+                            'unknown entry in track error, line {0}'.format(line_index + 1))
                 else:
                     next_track = True
 
@@ -373,8 +373,8 @@ class CueFile(File):
 
         if not self.content:
             try:
-                 with open(self.name, encoding="cp1252") as cue_file:
+                with open(self.name, encoding="cp1252") as cue_file:
-                     self.content = cue_file.read()
+                    self.content = cue_file.read()
             except:
                 raise ValueError('Cant encode CUE Sheet.')
 
@@ -408,9 +408,11 @@ class CueFile(File):
         for i in range(len(self.tracks)):
             if self.tracks[i]:
                 if self.tracks[i].get('artist'):
-                    content += 'track' + int_to_str(i) + 'artist' + '\t' + self.tracks[i].get('artist') + '\n'
+                    content += 'track' + int_to_str(i) + 'artist' + '\t' + self.tracks[i].get(
+                        'artist') + '\n'
                 if self.tracks[i].get('title'):
-                    content += 'track' + int_to_str(i) + 'title' + '\t' + self.tracks[i].get('title') + '\n'
+                    content += 'track' + int_to_str(i) + 'title' + '\t' + self.tracks[i].get(
+                        'title') + '\n'
         return content
 
     def htoa(self):
@@ -451,7 +453,8 @@ class MetaFile(File):
                         raise ValueError('Syntax error in album meta file')
                     if not content['tracks'][int(parsed_track.group(1))]:
                         content['tracks'][int(parsed_track.group(1))] = dict()
-                    content['tracks'][int(parsed_track.group(1))][parsed_track.group(2)] = parsed_line.group(2)
+                    content['tracks'][int(parsed_track.group(1))][
+                        parsed_track.group(2)] = parsed_line.group(2)
                 else:
                     content[parsed_line.group(1)] = parsed_line.group(2)
 
@@ -474,15 +477,16 @@ class MetaFile(File):
         if 'genre' in CUE_META.content:
             common_tags['genre'] = CUE_META.content['genre']
 
-        #freeform tags
+        # freeform tags
-        #freeform_tags['country'] = self.content['country']
+        # freeform_tags['country'] = self.content['country']
-        #freeform_tags['releasedate'] = self.content['releasedate']
+        # freeform_tags['releasedate'] = self.content['releasedate']
 
         return common_tags, freeform_tags
 
     def folders(self):
         artist = self.content['artist']
-        album = self.content['date'] + ' - ' + self.content['title'] + ' (' + self.content['label'] + ' - ' + self.content['catalog'] + ')'
+        album = self.content['date'] + ' - ' + self.content['title'] + ' (' + self.content[
+            'label'] + ' - ' + self.content['catalog'] + ')'
         return artist, album
 
     def complete(self):
@@ -537,6 +541,7 @@ class WaveFile(File):
         if self.type == 'Free Lossless Audio Codec':
             return FLAC(self.name)
 
+
 def split(albumpath):
     global CUE_META
     os.chdir(albumpath)
@@ -579,7 +584,8 @@ def split(albumpath):
         import getXldProfile
         xldprofile, xldformat, _ = getXldProfile.getXldProfile(headphones.CONFIG.XLDPROFILE)
         if not xldformat:
-            raise ValueError('Details for xld profile "%s" not found, cannot split cue' % (xldprofile))
+            raise ValueError(
+                'Details for xld profile "%s" not found, cannot split cue' % (xldprofile))
         else:
             if headphones.CONFIG.ENCODERFOLDER:
                 splitter = os.path.join(headphones.CONFIG.ENCODERFOLDER, 'xld')
@@ -592,11 +598,10 @@ def split(albumpath):
             splitter = 'shntool'
 
     if splitter == 'shntool' and not check_splitter(splitter):
-            raise ValueError('Command not found, ensure shntool with FLAC or xld (OS X) installed')
+        raise ValueError('Command not found, ensure shntool or xld installed')
 
-    # Determine if file can be split (only flac allowed for shntool)
+    # Determine if file can be split
-    if 'xld' in splitter and wave.name_ext not in WAVE_FILE_TYPE_BY_EXTENSION.keys() or \
+    if wave.name_ext not in WAVE_FILE_TYPE_BY_EXTENSION.keys():
-            wave.type not in SHNTOOL_COMPATIBLE:
         raise ValueError('Cannot split, audio file has unsupported extension')
 
     # Split with xld
@@ -640,7 +645,7 @@ def split(albumpath):
         cmd.extend(['-f'])
         cmd.extend([SPLIT_FILE_NAME])
         cmd.extend(['-o'])
-        cmd.extend(['flac'])
+        cmd.extend([wave.name_ext.lstrip('.')])
         cmd.extend([wave.name])
         split = split_baby(wave.name, cmd)
         os.remove(SPLIT_FILE_NAME)
@@ -652,9 +657,9 @@ def split(albumpath):
                 logger.info('Tagging %s...', t.name)
                 t.tag()
 
-        # rename FLAC files
+        # rename files
-        if split and CUE_META.count_tracks() == len(base_dir.tracks(ext='.flac', split=True)):
+        if split and CUE_META.count_tracks() == len(base_dir.tracks(ext=wave.name_ext, split=True)):
-            for t in base_dir.tracks(ext='.flac', split=True):
+            for t in base_dir.tracks(ext=wave.name_ext, split=True):
                 if t.name != t.filename():
                     logger.info('Renaming %s to %s...', t.name, t.filename())
                     os.rename(t.name, t.filename())

headphones/db.py

@@ -13,44 +13,41 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-#####################################
+###################################
-## Stolen from Sick-Beard's db.py  ##
+# Stolen from Sick-Beard's db.py  #
-#####################################
+###################################
 
 from __future__ import with_statement
 
-import os
 import sqlite3
 
+import os
 import headphones
-
 from headphones import logger
 
 
 def dbFilename(filename="headphones.db"):
-
     return os.path.join(headphones.DATA_DIR, filename)
 
 
 def getCacheSize():
-    #this will protect against typecasting problems produced by empty string and None settings
+    # this will protect against typecasting problems produced by empty string and None settings
     if not headphones.CONFIG.CACHE_SIZEMB:
-        #sqlite will work with this (very slowly)
+        # sqlite will work with this (very slowly)
         return 0
     return int(headphones.CONFIG.CACHE_SIZEMB)
 
 
 class DBConnection:
-
     def __init__(self, filename="headphones.db"):
 
         self.filename = filename
         self.connection = sqlite3.connect(dbFilename(filename), timeout=20)
-        #don't wait for the disk to finish writing
+        # don't wait for the disk to finish writing
         self.connection.execute("PRAGMA synchronous = OFF")
-        #journal disabled since we never do rollbacks
+        # journal disabled since we never do rollbacks
         self.connection.execute("PRAGMA journal_mode = %s" % headphones.CONFIG.JOURNAL_MODE)
-        #64mb of cache memory,probably need to make it user configurable
+        # 64mb of cache memory,probably need to make it user configurable
         self.connection.execute("PRAGMA cache_size=-%s" % (getCacheSize() * 1024))
         self.connection.row_factory = sqlite3.Row
 
@@ -92,17 +89,20 @@ class DBConnection:
 
     def upsert(self, tableName, valueDict, keyDict):
 
+        def genParams(myDict):
+            return [x + " = ?" for x in myDict.keys()]
+
         changesBefore = self.connection.total_changes
 
-        genParams = lambda myDict: [x + " = ?" for x in myDict.keys()]
+        update_query = "UPDATE " + tableName + " SET " + ", ".join(
-
+            genParams(valueDict)) + " WHERE " + " AND ".join(genParams(keyDict))
-        update_query = "UPDATE " + tableName + " SET " + ", ".join(genParams(valueDict)) + " WHERE " + " AND ".join(genParams(keyDict))
 
         self.action(update_query, valueDict.values() + keyDict.values())
 
         if self.connection.total_changes == changesBefore:
             insert_query = (
-                "INSERT INTO " + tableName + " (" + ", ".join(valueDict.keys() + keyDict.keys()) + ")" +
+                "INSERT INTO " + tableName + " (" + ", ".join(
+                    valueDict.keys() + keyDict.keys()) + ")" +
                 " VALUES (" + ", ".join(["?"] * len(valueDict.keys() + keyDict.keys())) + ")"
             )
             try:

headphones/getXldProfile.py

@@ -1,29 +1,29 @@
 import os.path
+
 import biplist
 from headphones import logger
 
 
 def getXldProfile(xldProfile):
-
     xldProfileNotFound = xldProfile
 
     expanded = os.path.expanduser('~/Library/Preferences/jp.tmkk.XLD.plist')
     if not os.path.isfile(expanded):
         logger.warn("Could not find xld preferences at: %s", expanded)
-        return(xldProfileNotFound, None, None)
+        return (xldProfileNotFound, None, None)
 
     # Get xld preferences plist
     try:
         preferences = biplist.readPlist(expanded)
     except (biplist.InvalidPlistException, biplist.NotBinaryPlistException), e:
         logger.error("Error reading xld preferences plist: %s", e)
-        return(xldProfileNotFound, None, None)
+        return (xldProfileNotFound, None, None)
 
     if not isinstance(preferences, dict):
         logger.error("Error reading xld preferences plist, not a dict: %r", preferences)
-        return(xldProfileNotFound, None, None)
+        return (xldProfileNotFound, None, None)
 
-    profiles = preferences.get('Profiles', []) # pylint:disable=E1103
+    profiles = preferences.get('Profiles', [])  # pylint:disable=E1103
 
     xldProfile = xldProfile.lower()
     for profile in profiles:
@@ -175,6 +175,6 @@ def getXldProfile(xldProfile):
             if xldFormat and not xldBitrate:
                 xldBitrate = 400
 
-            return(xldProfileForCmd, xldFormat, xldBitrate)
+            return (xldProfileForCmd, xldFormat, xldBitrate)
 
-    return(xldProfileNotFound, None, None)
+    return (xldProfileNotFound, None, None)

headphones/helpers.py

@@ -13,19 +13,19 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
-
 from operator import itemgetter
-
 import unicodedata
-import headphones
 import datetime
-import fnmatch
 import shutil
 import time
 import sys
+
+import fnmatch
 import re
 import os
+from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
+import headphones
+
 
 # Modified from https://github.com/Verrus/beets-plugin-featInTitle
 RE_FEATURING = re.compile(r"[fF]t\.|[fF]eaturing|[fF]eat\.|\b[wW]ith\b|&|vs\.")
@@ -35,7 +35,9 @@ RE_CD = re.compile(r"^(CD|dics)\s*[0-9]+$", re.I)
 
 
 def multikeysort(items, columns):
-    comparers = [((itemgetter(col[1:].strip()), -1) if col.startswith('-') else (itemgetter(col.strip()), 1)) for col in columns]
+    comparers = [
+        ((itemgetter(col[1:].strip()), -1) if col.startswith('-') else (itemgetter(col.strip()), 1))
+        for col in columns]
 
     def comparer(left, right):
         for fn, mult in comparers:
@@ -56,7 +58,6 @@ def checked(variable):
 
 
 def radio(variable, pos):
-
     if variable == pos:
         return 'Checked'
     else:
@@ -70,7 +71,7 @@ def latinToAscii(unicrap):
     xlate = {
         0xc0: 'A', 0xc1: 'A', 0xc2: 'A', 0xc3: 'A', 0xc4: 'A', 0xc5: 'A',
         0xc6: 'Ae', 0xc7: 'C',
-        0xc8: 'E', 0xc9: 'E', 0xca: 'E', 0xcb: 'E', 0x86: 'e',
+        0xc8: 'E', 0xc9: 'E', 0xca: 'E', 0xcb: 'E', 0x86: 'e', 0x39e: 'E',
         0xcc: 'I', 0xcd: 'I', 0xce: 'I', 0xcf: 'I',
         0xd0: 'Th', 0xd1: 'N',
         0xd2: 'O', 0xd3: 'O', 0xd4: 'O', 0xd5: 'O', 0xd6: 'O', 0xd8: 'O',
@@ -107,7 +108,6 @@ def latinToAscii(unicrap):
 
 
 def convert_milliseconds(ms):
-
     seconds = ms / 1000
     gmtime = time.gmtime(seconds)
     if seconds > 3600:
@@ -119,7 +119,6 @@ def convert_milliseconds(ms):
 
 
 def convert_seconds(s):
-
     gmtime = time.gmtime(s)
     if s > 3600:
         minutes = time.strftime("%H:%M:%S", gmtime)
@@ -141,7 +140,6 @@ def now():
 
 
 def get_age(date):
-
     try:
         split_date = date.split('-')
     except:
@@ -149,14 +147,13 @@ def get_age(date):
 
     try:
         days_old = int(split_date[0]) * 365 + int(split_date[1]) * 30 + int(split_date[2])
-    except IndexError:
+    except (IndexError, ValueError):
         days_old = False
 
     return days_old
 
 
 def bytes_to_mb(bytes):
-
     mb = int(bytes) / 1048576
     size = '%.1f MB' % mb
     return size
@@ -172,16 +169,16 @@ def piratesize(size):
     split = size.split(" ")
     factor = float(split[0])
     unit = split[1].upper()
-    
+
-    if unit == 'MiB':
+    if unit == 'MIB':
         size = factor * 1048576
     elif unit == 'MB':
         size = factor * 1000000
-    elif unit == 'GiB':
+    elif unit == 'GIB':
         size = factor * 1073741824
     elif unit == 'GB':
         size = factor * 1000000000
-    elif unit == 'KiB':
+    elif unit == 'KIB':
         size = factor * 1024
     elif unit == 'KB':
         size = factor * 1000
@@ -194,7 +191,6 @@ def piratesize(size):
 
 
 def replace_all(text, dic, normalize=False):
-
     if not text:
         return ''
 
@@ -221,15 +217,14 @@ def replace_illegal_chars(string, type="file"):
 
 
 def cleanName(string):
-
     pass1 = latinToAscii(string).lower()
-    out_string = re.sub('[\.\-\/\!\@\#\$\%\^\&\*\(\)\+\-\"\'\,\;\:\[\]\{\}\<\>\=\_]', '', pass1).encode('utf-8')
+    out_string = re.sub('[\.\-\/\!\@\#\$\%\^\&\*\(\)\+\-\"\'\,\;\:\[\]\{\}\<\>\=\_]', '',
+                        pass1).encode('utf-8')
 
     return out_string
 
 
 def cleanTitle(title):
-
     title = re.sub('[\.\-\/\_]', ' ', title).lower()
 
     # Strip out extra whitespace
@@ -312,16 +307,22 @@ def expand_subfolders(f):
     difference = max(path_depths) - min(path_depths)
 
     if difference > 0:
-        logger.info("Found %d media folders, but depth difference between lowest and deepest media folder is %d (expected zero). If this is a discography or a collection of albums, make sure albums are per folder.", len(media_folders), difference)
+        logger.info(
+            "Found %d media folders, but depth difference between lowest and deepest media folder is %d (expected zero). If this is a discography or a collection of albums, make sure albums are per folder.",
+            len(media_folders), difference)
 
         # While already failed, advice the user what he could try. We assume the
         # directory may contain separate CD's and maybe some extra's. The
         # structure may look like X albums at same depth, and (one or more)
         # extra folders with a higher depth.
-        extra_media_folders = [media_folder[:min(path_depths)] for media_folder in media_folders if len(media_folder) > min(path_depths)]
+        extra_media_folders = [media_folder[:min(path_depths)] for media_folder in media_folders if
-        extra_media_folders = list(set([os.path.join(*media_folder) for media_folder in extra_media_folders]))
+                               len(media_folder) > min(path_depths)]
+        extra_media_folders = list(
+            set([os.path.join(*media_folder) for media_folder in extra_media_folders]))
 
-        logger.info("Please look at the following folder(s), since they cause the depth difference: %s", extra_media_folders)
+        logger.info(
+            "Please look at the following folder(s), since they cause the depth difference: %s",
+            extra_media_folders)
         return
 
     # Convert back to paths and remove duplicates, which may be there after
@@ -368,7 +369,7 @@ def path_filter_patterns(paths, patterns, root=None):
     for path in paths[:]:
         if path_match_patterns(path, patterns):
             logger.debug("Path ignored by pattern: %s",
-                os.path.join(root or "", path))
+                         os.path.join(root or "", path))
 
             ignored += 1
             paths.remove(path)
@@ -378,11 +379,11 @@ def path_filter_patterns(paths, patterns, root=None):
 
 
 def extract_data(s):
-
     s = s.replace('_', ' ')
 
-    #headphones default format
+    # headphones default format
-    pattern = re.compile(r'(?P<name>.*?)\s\-\s(?P<album>.*?)\s[\[\(](?P<year>.*?)[\]\)]', re.VERBOSE)
+    pattern = re.compile(r'(?P<name>.*?)\s\-\s(?P<album>.*?)\s[\[\(](?P<year>.*?)[\]\)]',
+                         re.VERBOSE)
     match = pattern.match(s)
 
     if match:
@@ -391,7 +392,7 @@ def extract_data(s):
         year = match.group("year")
         return (name, album, year)
 
-    #Gonna take a guess on this one - might be enough to search on mb
+    # Gonna take a guess on this one - might be enough to search on mb
     pat = re.compile(r"(?P<name>.*?)\s*-\s*(?P<album>[^\[(-]*)")
 
     match = pat.match(s)
@@ -468,7 +469,8 @@ def extract_metadata(f):
                 old_album = new_albums[index]
                 new_albums[index] = RE_CD_ALBUM.sub("", album).strip()
 
-                logger.debug("Stripped albumd number identifier: %s -> %s", old_album, new_albums[index])
+                logger.debug("Stripped albumd number identifier: %s -> %s", old_album,
+                             new_albums[index])
 
         # Remove duplicates
         new_albums = list(set(new_albums))
@@ -498,7 +500,8 @@ def extract_metadata(f):
             return (artist, albums[0], years[0])
 
     # Not sure what to do here.
-    logger.info("Found %d artists, %d albums and %d years in metadata, so ignoring", len(artists), len(albums), len(years))
+    logger.info("Found %d artists, %d albums and %d years in metadata, so ignoring", len(artists),
+                len(albums), len(years))
     logger.debug("Artists: %s, Albums: %s, Years: %s", artists, albums, years)
 
     return (None, None, None)
@@ -519,13 +522,15 @@ def get_downloaded_track_list(albumpath):
     return downloaded_track_list
 
 
-def preserve_torrent_direcory(albumpath):
+def preserve_torrent_directory(albumpath):
     """
     Copy torrent directory to headphones-modified to keep files for seeding.
     """
     from headphones import logger
-    new_folder = os.path.join(albumpath, 'headphones-modified'.encode(headphones.SYS_ENCODING, 'replace'))
+    new_folder = os.path.join(albumpath,
-    logger.info("Copying files to 'headphones-modified' subfolder to preserve downloaded files for seeding")
+                              'headphones-modified'.encode(headphones.SYS_ENCODING, 'replace'))
+    logger.info(
+        "Copying files to 'headphones-modified' subfolder to preserve downloaded files for seeding")
     try:
         shutil.copytree(albumpath, new_folder)
         return new_folder
@@ -578,7 +583,9 @@ def cue_split(albumpath):
 
 def extract_logline(s):
     # Default log format
-    pattern = re.compile(r'(?P<timestamp>.*?)\s\-\s(?P<level>.*?)\s*\:\:\s(?P<thread>.*?)\s\:\s(?P<message>.*)', re.VERBOSE)
+    pattern = re.compile(
+        r'(?P<timestamp>.*?)\s\-\s(?P<level>.*?)\s*\:\:\s(?P<thread>.*?)\s\:\s(?P<message>.*)',
+        re.VERBOSE)
     match = pattern.match(s)
     if match:
         timestamp = match.group("timestamp")
@@ -593,7 +600,7 @@ def extract_logline(s):
 def extract_song_data(s):
     from headphones import logger
 
-    #headphones default format
+    # headphones default format
     pattern = re.compile(r'(?P<name>.*?)\s\-\s(?P<album>.*?)\s\[(?P<year>.*?)\]', re.VERBOSE)
     match = pattern.match(s)
 
@@ -605,7 +612,7 @@ def extract_song_data(s):
     else:
         logger.info("Couldn't parse %s into a valid default format", s)
 
-    #newzbin default format
+    # newzbin default format
     pattern = re.compile(r'(?P<name>.*?)\s\-\s(?P<album>.*?)\s\((?P<year>\d+?\))', re.VERBOSE)
     match = pattern.match(s)
     if match:
@@ -619,7 +626,6 @@ def extract_song_data(s):
 
 
 def smartMove(src, dest, delete=True):
-
     from headphones import logger
 
     source_dir = os.path.dirname(src)
@@ -640,7 +646,8 @@ def smartMove(src, dest, delete=True):
                     os.rename(src, os.path.join(source_dir, newfile))
                     filename = newfile
                 except Exception as e:
-                    logger.warn('Error renaming %s: %s', src.decode(headphones.SYS_ENCODING, 'replace'), e)
+                    logger.warn('Error renaming %s: %s',
+                                src.decode(headphones.SYS_ENCODING, 'replace'), e)
                 break
 
     try:
@@ -650,7 +657,9 @@ def smartMove(src, dest, delete=True):
             shutil.copy(os.path.join(source_dir, filename), os.path.join(dest, filename))
             return True
     except Exception as e:
-        logger.warn('Error moving file %s: %s', filename.decode(headphones.SYS_ENCODING, 'replace'), e)
+        logger.warn('Error moving file %s: %s', filename.decode(headphones.SYS_ENCODING, 'replace'),
+                    e)
+
 
 def walk_directory(basedir, followlinks=True):
     """
@@ -672,8 +681,8 @@ def walk_directory(basedir, followlinks=True):
                 real_path = os.path.abspath(os.readlink(path))
 
                 if real_path in traversed:
-                    logger.debug("Skipping '%s' since it is a symlink to "\
+                    logger.debug("Skipping '%s' since it is a symlink to " \
-                        "'%s', which is already visited.", path, real_path)
+                                 "'%s', which is already visited.", path, real_path)
                 else:
                     traversed.append(real_path)
 
@@ -689,8 +698,9 @@ def walk_directory(basedir, followlinks=True):
         for result in _inner(*args):
             yield result
 
+
 #########################
-#Sab renaming functions #
+# Sab renaming functions #
 #########################
 
 # TODO: Grab config values from sab to know when these options are checked. For now we'll just iterate through all combinations
@@ -739,18 +749,20 @@ def sab_sanitize_foldername(name):
     if not name:
         name = 'unknown'
 
-    #maxlen = cfg.folder_max_length()
+    # maxlen = cfg.folder_max_length()
-    #if len(name) > maxlen:
+    # if len(name) > maxlen:
     #    name = name[:maxlen]
 
     return name
 
+
 def split_string(mystring, splitvar=','):
     mylist = []
     for each_word in mystring.split(splitvar):
         mylist.append(each_word.strip())
     return mylist
 
+
 def create_https_certificates(ssl_cert, ssl_key):
     """
     Create a pair of self-signed HTTPS certificares and store in them in
@@ -768,11 +780,13 @@ def create_https_certificates(ssl_cert, ssl_key):
     # Create the CA Certificate
     cakey = createKeyPair(TYPE_RSA, 2048)
     careq = createCertRequest(cakey, CN="Certificate Authority")
-    cacert = createCertificate(careq, (careq, cakey), serial, (0, 60 * 60 * 24 * 365 * 10)) # ten years
+    cacert = createCertificate(careq, (careq, cakey), serial,
+                               (0, 60 * 60 * 24 * 365 * 10))  # ten years
 
     pkey = createKeyPair(TYPE_RSA, 2048)
     req = createCertRequest(pkey, CN="Headphones")
-    cert = createCertificate(req, (cacert, cakey), serial, (0, 60 * 60 * 24 * 365 * 10)) # ten years
+    cert = createCertificate(req, (cacert, cakey), serial,
+                             (0, 60 * 60 * 24 * 365 * 10))  # ten years
 
     # Save the key and certificate to disk
     try:

headphones/importer.py

@@ -13,39 +13,39 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import logger, helpers, db, mb, lastfm
-
-from beets.mediafile import MediaFile
-
 import time
+
+from headphones import logger, helpers, db, mb, lastfm, metacritic
+from beets.mediafile import MediaFile
 import headphones
 
 blacklisted_special_artist_names = ['[anonymous]', '[data]', '[no artist]',
-    '[traditional]', '[unknown]', 'Various Artists']
+                                    '[traditional]', '[unknown]', 'Various Artists']
 blacklisted_special_artists = ['f731ccc4-e22a-43af-a747-64213329e088',
-    '33cf029c-63b0-41a0-9855-be2a3665fb3b',
+                               '33cf029c-63b0-41a0-9855-be2a3665fb3b',
-    '314e1c25-dde7-4e4d-b2f4-0a7b9f7c56dc',
+                               '314e1c25-dde7-4e4d-b2f4-0a7b9f7c56dc',
-    'eec63d3c-3b81-4ad4-b1e4-7c147d4d2b61',
+                               'eec63d3c-3b81-4ad4-b1e4-7c147d4d2b61',
-    '9be7f096-97ec-4615-8957-8d40b5dcbc41',
+                               '9be7f096-97ec-4615-8957-8d40b5dcbc41',
-    '125ec42a-7229-4250-afc5-e057484327fe',
+                               '125ec42a-7229-4250-afc5-e057484327fe',
-    '89ad4ac3-39f7-470e-963a-56509c546377']
+                               '89ad4ac3-39f7-470e-963a-56509c546377']
 
 
 def is_exists(artistid):
     myDB = db.DBConnection()
 
     # See if the artist is already in the database
-    artistlist = myDB.select('SELECT ArtistID, ArtistName from artists WHERE ArtistID=?', [artistid])
+    artistlist = myDB.select('SELECT ArtistID, ArtistName from artists WHERE ArtistID=?',
+                             [artistid])
 
     if any(artistid in x for x in artistlist):
-        logger.info(artistlist[0][1] + u" is already in the database. Updating 'have tracks', but not artist information")
+        logger.info(artistlist[0][
+                        1] + u" is already in the database. Updating 'have tracks', but not artist information")
         return True
     else:
         return False
 
 
 def artistlist_to_mbids(artistlist, forced=False):
-
     for artist in artistlist:
 
         if not artist and artist != ' ':
@@ -77,9 +77,12 @@ def artistlist_to_mbids(artistlist, forced=False):
         myDB = db.DBConnection()
 
         if not forced:
-            bl_artist = myDB.action('SELECT * FROM blacklist WHERE ArtistID=?', [artistid]).fetchone()
+            bl_artist = myDB.action('SELECT * FROM blacklist WHERE ArtistID=?',
+                                    [artistid]).fetchone()
             if bl_artist or artistid in blacklisted_special_artists:
-                logger.info("Artist ID for '%s' is either blacklisted or Various Artists. To add artist, you must do it manually (Artist ID: %s)" % (artist, artistid))
+                logger.info(
+                    "Artist ID for '%s' is either blacklisted or Various Artists. To add artist, you must do it manually (Artist ID: %s)" % (
+                    artist, artistid))
                 continue
 
         # Add to database if it doesn't exist
@@ -88,7 +91,9 @@ def artistlist_to_mbids(artistlist, forced=False):
 
         # Just update the tracks if it does
         else:
-            havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=?', [artistid])) + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ?', [artist]))
+            havetracks = len(
+                myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=?', [artistid])) + len(
+                myDB.select('SELECT TrackTitle from have WHERE ArtistName like ?', [artist]))
             myDB.action('UPDATE artists SET HaveTracks=? WHERE ArtistID=?', [havetracks, artistid])
 
         # Delete it from the New Artists if the request came from there
@@ -111,8 +116,7 @@ def addArtistIDListToDB(artistidlist):
         addArtisttoDB(artistid)
 
 
-def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
+def addArtisttoDB(artistid, extrasonly=False, forcefull=False, type="artist"):
-
     # Putting this here to get around the circular import. We're using this to update thumbnails for artist/albums
     from headphones import cache
 
@@ -142,17 +146,24 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                         "Status": "Loading",
                         "IncludeExtras": headphones.CONFIG.INCLUDE_EXTRAS,
                         "Extras": headphones.CONFIG.EXTRAS}
+        if type == "series":
+            newValueDict['Type'] = "series"
     else:
         newValueDict = {"Status": "Loading"}
+        if dbartist["Type"] == "series":
+            type = "series"
 
     myDB.upsert("artists", newValueDict, controlValueDict)
 
-    artist = mb.getArtist(artistid, extrasonly)
+    if type == "series":
+        artist = mb.getSeries(artistid)
+    else:
+        artist = mb.getArtist(artistid, extrasonly)
 
     if artist and artist.get('artist_name') in blacklisted_special_artist_names:
         logger.warn('Cannot import blocked special purpose artist: %s' % artist.get('artist_name'))
         myDB.action('DELETE from artists WHERE ArtistID=?', [artistid])
-        #in case it's already in the db
+        # in case it's already in the db
         myDB.action('DELETE from albums WHERE ArtistID=?', [artistid])
         myDB.action('DELETE from tracks WHERE ArtistID=?', [artistid])
         return
@@ -161,7 +172,7 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
         logger.warn("Error fetching artist info. ID: " + artistid)
         if dbartist is None:
             newValueDict = {"ArtistName": "Fetch failed, try refreshing. (%s)" % (artistid),
-                    "Status": "Active"}
+                            "Status": "Active"}
         else:
             newValueDict = {"Status": "Active"}
         myDB.upsert("artists", newValueDict, controlValueDict)
@@ -184,7 +195,8 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
     # See if we need to grab extras. Artist specific extras take precedence
     # over global option. Global options are set when adding a new artist
     try:
-        db_artist = myDB.action('SELECT IncludeExtras, Extras from artists WHERE ArtistID=?', [artistid]).fetchone()
+        db_artist = myDB.action('SELECT IncludeExtras, Extras from artists WHERE ArtistID=?',
+                                [artistid]).fetchone()
         includeExtras = db_artist['IncludeExtras']
     except IndexError:
         includeExtras = False
@@ -199,9 +211,12 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
         for groups in artist['releasegroups']:
             group_list.append(groups['id'])
         if not extrasonly:
-            remove_missing_groups_from_albums = myDB.select("SELECT AlbumID FROM albums WHERE ArtistID=?", [artistid])
+            remove_missing_groups_from_albums = myDB.select(
+                "SELECT AlbumID FROM albums WHERE ArtistID=?", [artistid])
         else:
-            remove_missing_groups_from_albums = myDB.select('SELECT AlbumID FROM albums WHERE ArtistID=? AND Status="Skipped" AND Type!="Album"', [artistid])
+            remove_missing_groups_from_albums = myDB.select(
+                'SELECT AlbumID FROM albums WHERE ArtistID=? AND Status="Skipped" AND Type!="Album"',
+                [artistid])
         for items in remove_missing_groups_from_albums:
             if items['AlbumID'] not in group_list:
                 # Remove all from albums/tracks that aren't in release groups
@@ -210,12 +225,16 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                 myDB.action("DELETE FROM tracks WHERE AlbumID=?", [items['AlbumID']])
                 myDB.action("DELETE FROM alltracks WHERE AlbumID=?", [items['AlbumID']])
                 myDB.action('DELETE from releases WHERE ReleaseGroupID=?', [items['AlbumID']])
-                logger.info("[%s] Removing all references to release group %s to reflect MusicBrainz refresh" % (artist['artist_name'], items['AlbumID']))
+                logger.info(
+                    "[%s] Removing all references to release group %s to reflect MusicBrainz refresh" % (
+                    artist['artist_name'], items['AlbumID']))
                 if not extrasonly:
                     force_repackage = 1
     else:
         if not extrasonly:
-            logger.info("[%s] There was either an error pulling data from MusicBrainz or there might not be any releases for this category" % artist['artist_name'])
+            logger.info(
+                "[%s] There was either an error pulling data from MusicBrainz or there might not be any releases for this category" %
+                artist['artist_name'])
 
     # Then search for releases within releasegroups, if releases don't exist, then remove from allalbums/alltracks
     album_searches = []
@@ -225,7 +244,7 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
         today = helpers.today()
         rgid = rg['id']
         skip_log = 0
-        #Make a user configurable variable to skip update of albums with release dates older than this date (in days)
+        # Make a user configurable variable to skip update of albums with release dates older than this date (in days)
         pause_delta = headphones.CONFIG.MB_IGNORE_AGE
 
         rg_exists = myDB.action("SELECT * from albums WHERE AlbumID=?", [rg['id']]).fetchone()
@@ -240,12 +259,14 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                 new_release_group = True
 
             if new_release_group:
-                logger.info("[%s] Now adding: %s (New Release Group)" % (artist['artist_name'], rg['title']))
+                logger.info("[%s] Now adding: %s (New Release Group)" % (
+                artist['artist_name'], rg['title']))
                 new_releases = mb.get_new_releases(rgid, includeExtras)
 
             else:
                 if check_release_date is None or check_release_date == u"None":
-                    logger.info("[%s] Now updating: %s (No Release Date)" % (artist['artist_name'], rg['title']))
+                    logger.info("[%s] Now updating: %s (No Release Date)" % (
+                    artist['artist_name'], rg['title']))
                     new_releases = mb.get_new_releases(rgid, includeExtras, True)
                 else:
                     if len(check_release_date) == 10:
@@ -257,20 +278,24 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                     else:
                         release_date = today
                     if helpers.get_age(today) - helpers.get_age(release_date) < pause_delta:
-                        logger.info("[%s] Now updating: %s (Release Date <%s Days)", artist['artist_name'], rg['title'], pause_delta)
+                        logger.info("[%s] Now updating: %s (Release Date <%s Days)",
+                                    artist['artist_name'], rg['title'], pause_delta)
                         new_releases = mb.get_new_releases(rgid, includeExtras, True)
                     else:
-                        logger.info("[%s] Skipping: %s (Release Date >%s Days)", artist['artist_name'], rg['title'], pause_delta)
+                        logger.info("[%s] Skipping: %s (Release Date >%s Days)",
+                                    artist['artist_name'], rg['title'], pause_delta)
                         skip_log = 1
                         new_releases = 0
 
             if force_repackage == 1:
                 new_releases = -1
-                logger.info('[%s] Forcing repackage of %s (Release Group Removed)', artist['artist_name'], al_title)
+                logger.info('[%s] Forcing repackage of %s (Release Group Removed)',
+                            artist['artist_name'], al_title)
             else:
                 new_releases = new_releases
         else:
-            logger.info("[%s] Now adding/updating: %s (Comprehensive Force)", artist['artist_name'], rg['title'])
+            logger.info("[%s] Now adding/updating: %s (Comprehensive Force)", artist['artist_name'],
+                        rg['title'])
             new_releases = mb.get_new_releases(rgid, includeExtras, forcefull)
 
         if new_releases != 0:
@@ -284,23 +309,26 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
             # This will be used later to build a hybrid release
             fullreleaselist = []
             # Search for releases within a release group
-            find_hybrid_releases = myDB.action("SELECT * from allalbums WHERE AlbumID=?", [rg['id']])
+            find_hybrid_releases = myDB.action("SELECT * from allalbums WHERE AlbumID=?",
+                                               [rg['id']])
 
             # Build the dictionary for the fullreleaselist
             for items in find_hybrid_releases:
-                if items['ReleaseID'] != rg['id']: #don't include hybrid information, since that's what we're replacing
+                if items['ReleaseID'] != rg[
+                    'id']:  # don't include hybrid information, since that's what we're replacing
                     hybrid_release_id = items['ReleaseID']
                     newValueDict = {"ArtistID": items['ArtistID'],
-                        "ArtistName": items['ArtistName'],
+                                    "ArtistName": items['ArtistName'],
-                        "AlbumTitle": items['AlbumTitle'],
+                                    "AlbumTitle": items['AlbumTitle'],
-                        "AlbumID": items['AlbumID'],
+                                    "AlbumID": items['AlbumID'],
-                        "AlbumASIN": items['AlbumASIN'],
+                                    "AlbumASIN": items['AlbumASIN'],
-                        "ReleaseDate": items['ReleaseDate'],
+                                    "ReleaseDate": items['ReleaseDate'],
-                        "Type": items['Type'],
+                                    "Type": items['Type'],
-                        "ReleaseCountry": items['ReleaseCountry'],
+                                    "ReleaseCountry": items['ReleaseCountry'],
-                        "ReleaseFormat": items['ReleaseFormat']
+                                    "ReleaseFormat": items['ReleaseFormat']
-                    }
+                                    }
-                    find_hybrid_tracks = myDB.action("SELECT * from alltracks WHERE ReleaseID=?", [hybrid_release_id])
+                    find_hybrid_tracks = myDB.action("SELECT * from alltracks WHERE ReleaseID=?",
+                                                     [hybrid_release_id])
                     totalTracks = 1
                     hybrid_track_array = []
                     for hybrid_tracks in find_hybrid_tracks:
@@ -308,9 +336,9 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                             'number': hybrid_tracks['TrackNumber'],
                             'title': hybrid_tracks['TrackTitle'],
                             'id': hybrid_tracks['TrackID'],
-                            #'url':           hybrid_tracks['TrackURL'],
+                            # 'url':           hybrid_tracks['TrackURL'],
                             'duration': hybrid_tracks['TrackDuration']
-                            })
+                        })
                         totalTracks += 1
                     newValueDict['ReleaseID'] = hybrid_release_id
                     newValueDict['Tracks'] = hybrid_track_array
@@ -320,10 +348,12 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
             # This may end up being called with an empty fullreleaselist
             try:
                 hybridrelease = getHybridRelease(fullreleaselist)
-                logger.info('[%s] Packaging %s releases into hybrid title' % (artist['artist_name'], rg['title']))
+                logger.info('[%s] Packaging %s releases into hybrid title' % (
+                artist['artist_name'], rg['title']))
             except Exception as e:
                 errors = True
-                logger.warn('[%s] Unable to get hybrid release information for %s: %s' % (artist['artist_name'], rg['title'], e))
+                logger.warn('[%s] Unable to get hybrid release information for %s: %s' % (
+                artist['artist_name'], rg['title'], e))
                 continue
 
             # Use the ReleaseGroupID as the ReleaseID for the hybrid release to differentiate it
@@ -338,13 +368,14 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                             "AlbumASIN": hybridrelease['AlbumASIN'],
                             "ReleaseDate": hybridrelease['ReleaseDate'],
                             "Type": rg['type']
-                        }
+                            }
 
             myDB.upsert("allalbums", newValueDict, controlValueDict)
 
             for track in hybridrelease['Tracks']:
 
-                cleanname = helpers.cleanName(artist['artist_name'] + ' ' + rg['title'] + ' ' + track['title'])
+                cleanname = helpers.cleanName(
+                    artist['artist_name'] + ' ' + rg['title'] + ' ' + track['title'])
 
                 controlValueDict = {"TrackID": track['id'],
                                     "ReleaseID": rg['id']}
@@ -358,25 +389,29 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                                 "TrackDuration": track['duration'],
                                 "TrackNumber": track['number'],
                                 "CleanName": cleanname
-                            }
+                                }
 
-                match = myDB.action('SELECT Location, BitRate, Format from have WHERE CleanName=?', [cleanname]).fetchone()
+                match = myDB.action('SELECT Location, BitRate, Format from have WHERE CleanName=?',
+                                    [cleanname]).fetchone()
 
                 if not match:
-                    match = myDB.action('SELECT Location, BitRate, Format from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [artist['artist_name'], rg['title'], track['title']]).fetchone()
+                    match = myDB.action(
-                #if not match:
+                        'SELECT Location, BitRate, Format from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?',
-                    #match = myDB.action('SELECT Location, BitRate, Format from have WHERE TrackID=?', [track['id']]).fetchone()
+                        [artist['artist_name'], rg['title'], track['title']]).fetchone()
+                    # if not match:
+                    # match = myDB.action('SELECT Location, BitRate, Format from have WHERE TrackID=?', [track['id']]).fetchone()
                 if match:
                     newValueDict['Location'] = match['Location']
                     newValueDict['BitRate'] = match['BitRate']
                     newValueDict['Format'] = match['Format']
-                    #myDB.action('UPDATE have SET Matched="True" WHERE Location=?', [match['Location']])
+                    # myDB.action('UPDATE have SET Matched="True" WHERE Location=?', [match['Location']])
-                    myDB.action('UPDATE have SET Matched=? WHERE Location=?', (rg['id'], match['Location']))
+                    myDB.action('UPDATE have SET Matched=? WHERE Location=?',
+                                (rg['id'], match['Location']))
 
                 myDB.upsert("alltracks", newValueDict, controlValueDict)
 
             # Delete matched tracks from the have table
-            #myDB.action('DELETE from have WHERE Matched="True"')
+            # myDB.action('DELETE from have WHERE Matched="True"')
 
             # If there's no release in the main albums tables, add the default (hybrid)
             # If there is a release, check the ReleaseID against the AlbumID to see if they differ (user updated)
@@ -401,7 +436,7 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                             "Type": album['Type'],
                             "ReleaseCountry": album['ReleaseCountry'],
                             "ReleaseFormat": album['ReleaseFormat']
-                        }
+                            }
 
             if rg_exists:
                 newValueDict['DateAdded'] = rg_exists['DateAdded']
@@ -418,14 +453,17 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                     newValueDict['Status'] = "Wanted"
                 # Sometimes "new" albums are added to musicbrainz after their release date, so let's try to catch these
                 # The first test just makes sure we have year-month-day
-                elif helpers.get_age(album['ReleaseDate']) and helpers.get_age(today) - helpers.get_age(album['ReleaseDate']) < 21 and headphones.CONFIG.AUTOWANT_UPCOMING:
+                elif helpers.get_age(album['ReleaseDate']) and helpers.get_age(
+                        today) - helpers.get_age(
+                        album['ReleaseDate']) < 21 and headphones.CONFIG.AUTOWANT_UPCOMING:
                     newValueDict['Status'] = "Wanted"
                 else:
                     newValueDict['Status'] = "Skipped"
 
             myDB.upsert("albums", newValueDict, controlValueDict)
 
-            tracks = myDB.action('SELECT * from alltracks WHERE ReleaseID=?', [releaseid]).fetchall()
+            tracks = myDB.action('SELECT * from alltracks WHERE ReleaseID=?',
+                                 [releaseid]).fetchall()
 
             # This is used to see how many tracks you have from an album - to
             # mark it as downloaded. Default is 80%, can be set in config as
@@ -434,7 +472,7 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
 
             if total_track_count == 0:
                 logger.warning("Total track count is zero for Release ID " +
-                    "'%s', skipping.", releaseid)
+                               "'%s', skipping.", releaseid)
                 continue
 
             for track in tracks:
@@ -442,35 +480,43 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                                     "AlbumID": rg['id']}
 
                 newValueDict = {"ArtistID": track['ArtistID'],
-                            "ArtistName": track['ArtistName'],
+                                "ArtistName": track['ArtistName'],
-                            "AlbumTitle": track['AlbumTitle'],
+                                "AlbumTitle": track['AlbumTitle'],
-                            "AlbumASIN": track['AlbumASIN'],
+                                "AlbumASIN": track['AlbumASIN'],
-                            "ReleaseID": track['ReleaseID'],
+                                "ReleaseID": track['ReleaseID'],
-                            "TrackTitle": track['TrackTitle'],
+                                "TrackTitle": track['TrackTitle'],
-                            "TrackDuration": track['TrackDuration'],
+                                "TrackDuration": track['TrackDuration'],
-                            "TrackNumber": track['TrackNumber'],
+                                "TrackNumber": track['TrackNumber'],
-                            "CleanName": track['CleanName'],
+                                "CleanName": track['CleanName'],
-                            "Location": track['Location'],
+                                "Location": track['Location'],
-                            "Format": track['Format'],
+                                "Format": track['Format'],
-                            "BitRate": track['BitRate']
+                                "BitRate": track['BitRate']
-                            }
+                                }
 
                 myDB.upsert("tracks", newValueDict, controlValueDict)
 
             # Mark albums as downloaded if they have at least 80% (by default, configurable) of the album
-            have_track_count = len(myDB.select('SELECT * from tracks WHERE AlbumID=? AND Location IS NOT NULL', [rg['id']]))
+            have_track_count = len(
+                myDB.select('SELECT * from tracks WHERE AlbumID=? AND Location IS NOT NULL',
+                            [rg['id']]))
             marked_as_downloaded = False
 
             if rg_exists:
-                if rg_exists['Status'] == 'Skipped' and ((have_track_count / float(total_track_count)) >= (headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0)):
+                if rg_exists['Status'] == 'Skipped' and (
-                    myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', rg['id']])
+                    (have_track_count / float(total_track_count)) >= (
+                    headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0)):
+                    myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?',
+                                ['Downloaded', rg['id']])
                     marked_as_downloaded = True
             else:
-                if (have_track_count / float(total_track_count)) >= (headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0):
+                if (have_track_count / float(total_track_count)) >= (
-                    myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', rg['id']])
+                    headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0):
+                    myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?',
+                                ['Downloaded', rg['id']])
                     marked_as_downloaded = True
 
-            logger.info(u"[%s] Seeing if we need album art for %s" % (artist['artist_name'], rg['title']))
+            logger.info(
+                u"[%s] Seeing if we need album art for %s" % (artist['artist_name'], rg['title']))
             cache.getThumb(AlbumID=rg['id'])
 
             # Start a search for the album if it's new, hasn't been marked as
@@ -480,7 +526,8 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                 album_searches.append(rg['id'])
         else:
             if skip_log == 0:
-                logger.info(u"[%s] No new releases, so no changes made to %s" % (artist['artist_name'], rg['title']))
+                logger.info(u"[%s] No new releases, so no changes made to %s" % (
+                artist['artist_name'], rg['title']))
 
     time.sleep(3)
     finalize_update(artistid, artist['artist_name'], errors)
@@ -488,8 +535,13 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
     logger.info(u"Seeing if we need album art for: %s" % artist['artist_name'])
     cache.getThumb(ArtistID=artistid)
 
+    logger.info(u"Fetching Metacritic reviews for: %s" % artist['artist_name'])
+    metacritic.update(artistid, artist['artist_name'], artist['releasegroups'])
+
     if errors:
-        logger.info("[%s] Finished updating artist: %s but with errors, so not marking it as updated in the database" % (artist['artist_name'], artist['artist_name']))
+        logger.info(
+            "[%s] Finished updating artist: %s but with errors, so not marking it as updated in the database" % (
+            artist['artist_name'], artist['artist_name']))
     else:
         myDB.action('DELETE FROM newartists WHERE ArtistName = ?', [artist['artist_name']])
         logger.info(u"Updating complete for: %s" % artist['artist_name'])
@@ -508,10 +560,18 @@ def finalize_update(artistid, artistname, errors=False):
 
     myDB = db.DBConnection()
 
-    latestalbum = myDB.action('SELECT AlbumTitle, ReleaseDate, AlbumID from albums WHERE ArtistID=? order by ReleaseDate DESC', [artistid]).fetchone()
+    latestalbum = myDB.action(
-    totaltracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND AlbumID IN (SELECT AlbumID FROM albums WHERE Status != "Ignored")', [artistid]))
+        'SELECT AlbumTitle, ReleaseDate, AlbumID from albums WHERE ArtistID=? order by ReleaseDate DESC',
-    #havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [artistid])) + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ?', [artist['artist_name']]))
+        [artistid]).fetchone()
-    havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [artistid])) + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"', [artistname]))
+    totaltracks = len(myDB.select(
+        'SELECT TrackTitle from tracks WHERE ArtistID=? AND AlbumID IN (SELECT AlbumID FROM albums WHERE Status != "Ignored")',
+        [artistid]))
+    # havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [artistid])) + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ?', [artist['artist_name']]))
+    havetracks = len(
+        myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL',
+                    [artistid])) + len(
+        myDB.select('SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"',
+                    [artistname]))
 
     controlValueDict = {"ArtistID": artistid}
 
@@ -534,7 +594,6 @@ def finalize_update(artistid, artistname, errors=False):
 
 
 def addReleaseById(rid, rgid=None):
-
     myDB = db.DBConnection()
 
     # Create minimum info upfront if added from searchresults
@@ -553,14 +612,18 @@ def addReleaseById(rid, rgid=None):
     rgid = None
     artistid = None
     release_dict = None
-    results = myDB.select("SELECT albums.ArtistID, releases.ReleaseGroupID from releases, albums WHERE releases.ReleaseID=? and releases.ReleaseGroupID=albums.AlbumID LIMIT 1", [rid])
+    results = myDB.select(
+        "SELECT albums.ArtistID, releases.ReleaseGroupID from releases, albums WHERE releases.ReleaseID=? and releases.ReleaseGroupID=albums.AlbumID LIMIT 1",
+        [rid])
     for result in results:
         rgid = result['ReleaseGroupID']
         artistid = result['ArtistID']
-        logger.debug("Found a cached releaseid : releasegroupid relationship: " + rid + " : " + rgid)
+        logger.debug(
+            "Found a cached releaseid : releasegroupid relationship: " + rid + " : " + rgid)
     if not rgid:
-        #didn't find it in the cache, get the information from MB
+        # didn't find it in the cache, get the information from MB
-        logger.debug("Didn't find releaseID " + rid + " in the cache. Looking up its ReleaseGroupID")
+        logger.debug(
+            "Didn't find releaseID " + rid + " in the cache. Looking up its ReleaseGroupID")
         try:
             release_dict = mb.getRelease(rid)
         except Exception as e:
@@ -577,10 +640,10 @@ def addReleaseById(rid, rgid=None):
         rgid = release_dict['rgid']
         artistid = release_dict['artist_id']
 
-    #we don't want to make more calls to MB here unless we have to, could be happening quite a lot
+    # we don't want to make more calls to MB here unless we have to, could be happening quite a lot
     rg_exists = myDB.select("SELECT * from albums WHERE AlbumID=?", [rgid])
 
-    #make sure the artist exists since I don't know what happens later if it doesn't
+    # make sure the artist exists since I don't know what happens later if it doesn't
     artist_exists = myDB.select("SELECT * from artists WHERE ArtistID=?", [artistid])
 
     if not artist_exists and release_dict:
@@ -589,7 +652,8 @@ def addReleaseById(rid, rgid=None):
         else:
             sortname = release_dict['artist_name']
 
-        logger.info(u"Now manually adding: " + release_dict['artist_name'] + " - with status Paused")
+        logger.info(
+            u"Now manually adding: " + release_dict['artist_name'] + " - with status Paused")
         controlValueDict = {"ArtistID": release_dict['artist_id']}
         newValueDict = {"ArtistName": release_dict['artist_name'],
                         "ArtistSortName": sortname,
@@ -614,13 +678,14 @@ def addReleaseById(rid, rgid=None):
         myDB.upsert("artists", newValueDict, controlValueDict)
 
     elif not artist_exists and not release_dict:
-        logger.error("Artist does not exist in the database and did not get a valid response from MB. Skipping release.")
+        logger.error(
+            "Artist does not exist in the database and did not get a valid response from MB. Skipping release.")
         if status == 'Loading':
             myDB.action("DELETE FROM albums WHERE AlbumID=?", [rgid])
         return
 
-    if not rg_exists and release_dict or status == 'Loading' and release_dict:  #it should never be the case that we have an rg and not the artist
+    if not rg_exists and release_dict or status == 'Loading' and release_dict:  # it should never be the case that we have an rg and not the artist
-                                                                                #but if it is this will fail
+        # but if it is this will fail
         logger.info(u"Now adding-by-id album (" + release_dict['title'] + ") from id: " + rgid)
         controlValueDict = {"AlbumID": rgid}
         if status != 'Loading':
@@ -629,7 +694,8 @@ def addReleaseById(rid, rgid=None):
         newValueDict = {"ArtistID": release_dict['artist_id'],
                         "ReleaseID": rgid,
                         "ArtistName": release_dict['artist_name'],
-                        "AlbumTitle": release_dict['title'] if 'title' in release_dict else release_dict['rg_title'],
+                        "AlbumTitle": release_dict['title'] if 'title' in release_dict else
+                        release_dict['rg_title'],
                         "AlbumASIN": release_dict['asin'],
                         "ReleaseDate": release_dict['date'],
                         "DateAdded": helpers.today(),
@@ -640,41 +706,48 @@ def addReleaseById(rid, rgid=None):
 
         myDB.upsert("albums", newValueDict, controlValueDict)
 
-        #keep a local cache of these so that external programs that are adding releasesByID don't hammer MB
+        # keep a local cache of these so that external programs that are adding releasesByID don't hammer MB
         myDB.action('INSERT INTO releases VALUES( ?, ?)', [rid, release_dict['rgid']])
 
         for track in release_dict['tracks']:
-            cleanname = helpers.cleanName(release_dict['artist_name'] + ' ' + release_dict['rg_title'] + ' ' + track['title'])
+            cleanname = helpers.cleanName(
+                release_dict['artist_name'] + ' ' + release_dict['rg_title'] + ' ' + track['title'])
 
             controlValueDict = {"TrackID": track['id'],
                                 "AlbumID": rgid}
             newValueDict = {"ArtistID": release_dict['artist_id'],
-                        "ArtistName": release_dict['artist_name'],
+                            "ArtistName": release_dict['artist_name'],
-                        "AlbumTitle": release_dict['rg_title'],
+                            "AlbumTitle": release_dict['rg_title'],
-                        "AlbumASIN": release_dict['asin'],
+                            "AlbumASIN": release_dict['asin'],
-                        "TrackTitle": track['title'],
+                            "TrackTitle": track['title'],
-                        "TrackDuration": track['duration'],
+                            "TrackDuration": track['duration'],
-                        "TrackNumber": track['number'],
+                            "TrackNumber": track['number'],
-                        "CleanName": cleanname
+                            "CleanName": cleanname
-                        }
+                            }
 
-            match = myDB.action('SELECT Location, BitRate, Format, Matched from have WHERE CleanName=?', [cleanname]).fetchone()
+            match = myDB.action(
+                'SELECT Location, BitRate, Format, Matched from have WHERE CleanName=?',
+                [cleanname]).fetchone()
 
             if not match:
-                match = myDB.action('SELECT Location, BitRate, Format, Matched from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [release_dict['artist_name'], release_dict['rg_title'], track['title']]).fetchone()
+                match = myDB.action(
+                    'SELECT Location, BitRate, Format, Matched from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?',
+                    [release_dict['artist_name'], release_dict['rg_title'],
+                     track['title']]).fetchone()
 
-            #if not match:
+                # if not match:
-                #match = myDB.action('SELECT Location, BitRate, Format from have WHERE TrackID=?', [track['id']]).fetchone()
+                # match = myDB.action('SELECT Location, BitRate, Format from have WHERE TrackID=?', [track['id']]).fetchone()
 
             if match:
                 newValueDict['Location'] = match['Location']
                 newValueDict['BitRate'] = match['BitRate']
                 newValueDict['Format'] = match['Format']
-                #myDB.action('DELETE from have WHERE Location=?', [match['Location']])
+                # myDB.action('DELETE from have WHERE Location=?', [match['Location']])
 
                 # If the album has been scanned before adding the release it will be unmatched, update to matched
                 if match['Matched'] == 'Failed':
-                    myDB.action('UPDATE have SET Matched=? WHERE Location=?', (release_dict['rgid'], match['Location']))
+                    myDB.action('UPDATE have SET Matched=? WHERE Location=?',
+                                (release_dict['rgid'], match['Location']))
 
             myDB.upsert("tracks", newValueDict, controlValueDict)
 
@@ -693,7 +766,8 @@ def addReleaseById(rid, rgid=None):
             searcher.searchforalbum(rgid, False)
 
     elif not rg_exists and not release_dict:
-        logger.error("ReleaseGroup does not exist in the database and did not get a valid response from MB. Skipping release.")
+        logger.error(
+            "ReleaseGroup does not exist in the database and did not get a valid response from MB. Skipping release.")
         if status == 'Loading':
             myDB.action("DELETE FROM albums WHERE AlbumID=?", [rgid])
         return
@@ -801,11 +875,13 @@ def getHybridRelease(fullreleaselist):
 
     sortable_release_list.sort(key=lambda x: getSortableReleaseDate(x['releasedate']))
 
-    average_tracks = sum(x['trackscount'] for x in sortable_release_list) / float(len(sortable_release_list))
+    average_tracks = sum(x['trackscount'] for x in sortable_release_list) / float(
+        len(sortable_release_list))
     for item in sortable_release_list:
         item['trackscount_delta'] = abs(average_tracks - item['trackscount'])
 
-    a = helpers.multikeysort(sortable_release_list, ['-hasasin', 'country', 'format', 'trackscount_delta'])
+    a = helpers.multikeysort(sortable_release_list,
+                             ['-hasasin', 'country', 'format', 'trackscount_delta'])
 
     release_dict = {'ReleaseDate': sortable_release_list[0]['releasedate'],
                     'Tracks': a[0]['tracks'],

headphones/lastfm.py

@@ -14,15 +14,14 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
 import random
-import headphones
-import headphones.lock
-
-from headphones import db, logger, request
-
 from collections import defaultdict
 
-TIMEOUT = 60.0 # seconds
+import headphones
-REQUEST_LIMIT = 1.0 / 5 # seconds
+import headphones.lock
+from headphones import db, logger, request
+
+TIMEOUT = 60.0  # seconds
+REQUEST_LIMIT = 1.0 / 5  # seconds
 ENTRY_POINT = "http://ws.audioscrobbler.com/2.0/"
 API_KEY = "395e6ec6bb557382fc41fde867bce66f"
 
@@ -79,7 +78,7 @@ def getSimilar():
                 try:
                     artist_mbid = artist["mbid"]
                     artist_name = artist["name"]
-                except TypeError:
+                except KeyError:
                     continue
 
                 if not any(artist_mbid in x for x in results):
@@ -116,7 +115,7 @@ def getArtists():
         return
 
     logger.info("Fetching artists from Last.FM for username: %s", headphones.CONFIG.LASTFM_USERNAME)
-    data = request_lastfm("library.getartists", limit=10000, user=headphones.CONFIG.LASTFM_USERNAME)
+    data = request_lastfm("library.getartists", limit=1000, user=headphones.CONFIG.LASTFM_USERNAME)
 
     if data and "artists" in data:
         artistlist = []

headphones/librarysync.py

@@ -14,17 +14,17 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
 import os
+import math
+
 import headphones
-
 from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
-
 from headphones import db, logger, helpers, importer, lastfm
 
+
 # You can scan a single directory and append it to the current library by
 # specifying append=True, ArtistID and ArtistName.
 def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
-    cron=False):
+                cron=False, artistScan=False):
-
     if cron and not headphones.CONFIG.LIBRARYSCAN:
         return
 
@@ -36,11 +36,12 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
 
     # If we're appending a dir, it's coming from the post processor which is
     # already bytestring
-    if not append:
+    if not append or artistScan:
         dir = dir.encode(headphones.SYS_ENCODING)
 
     if not os.path.isdir(dir):
-        logger.warn('Cannot find directory: %s. Not scanning' % dir.decode(headphones.SYS_ENCODING, 'replace'))
+        logger.warn('Cannot find directory: %s. Not scanning' % dir.decode(headphones.SYS_ENCODING,
+                                                                           'replace'))
         return
 
     myDB = db.DBConnection()
@@ -50,13 +51,16 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
 
     if not append:
         # Clean up bad filepaths
-        tracks = myDB.select('SELECT Location from alltracks WHERE Location IS NOT NULL UNION SELECT Location from tracks WHERE Location IS NOT NULL')
+        tracks = myDB.select(
+            'SELECT Location from alltracks WHERE Location IS NOT NULL UNION SELECT Location from tracks WHERE Location IS NOT NULL')
 
         for track in tracks:
-            encoded_track_string = track['Location'].encode(headphones.SYS_ENCODING)
+            encoded_track_string = track['Location'].encode(headphones.SYS_ENCODING, 'replace')
             if not os.path.isfile(encoded_track_string):
-                myDB.action('UPDATE tracks SET Location=?, BitRate=?, Format=? WHERE Location=?', [None, None, None, track['Location']])
+                myDB.action('UPDATE tracks SET Location=?, BitRate=?, Format=? WHERE Location=?',
-                myDB.action('UPDATE alltracks SET Location=?, BitRate=?, Format=? WHERE Location=?', [None, None, None, track['Location']])
+                            [None, None, None, track['Location']])
+                myDB.action('UPDATE alltracks SET Location=?, BitRate=?, Format=? WHERE Location=?',
+                            [None, None, None, track['Location']])
 
         del_have_tracks = myDB.select('SELECT Location, Matched, ArtistName from have')
 
@@ -64,20 +68,20 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
             encoded_track_string = track['Location'].encode(headphones.SYS_ENCODING, 'replace')
             if not os.path.isfile(encoded_track_string):
                 if track['ArtistName']:
-                    #Make sure deleted files get accounted for when updating artist track counts
+                    # Make sure deleted files get accounted for when updating artist track counts
                     new_artists.append(track['ArtistName'])
                 myDB.action('DELETE FROM have WHERE Location=?', [track['Location']])
-                logger.info('File %s removed from Headphones, as it is no longer on disk' % encoded_track_string.decode(headphones.SYS_ENCODING, 'replace'))
+                logger.info(
-           ###############myDB.action('DELETE from have')
+                    'File %s removed from Headphones, as it is no longer on disk' % encoded_track_string.decode(
+                        headphones.SYS_ENCODING, 'replace'))
 
     bitrates = []
-
     song_list = []
+    latest_subdirectory = []
+
     new_song_count = 0
     file_count = 0
 
-    latest_subdirectory = []
-
     for r, d, f in helpers.walk_directory(dir):
         # Filter paths based on config. Note that these methods work directly
         # on the inputs
@@ -91,9 +95,14 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
                 latest_subdirectory.append(subdirectory)
 
                 if file_count == 0 and r.replace(dir, '') != '':
-                    logger.info("[%s] Now scanning subdirectory %s" % (dir.decode(headphones.SYS_ENCODING, 'replace'), subdirectory.decode(headphones.SYS_ENCODING, 'replace')))
+                    logger.info("[%s] Now scanning subdirectory %s" % (
-                elif latest_subdirectory[file_count] != latest_subdirectory[file_count - 1] and file_count != 0:
+                        dir.decode(headphones.SYS_ENCODING, 'replace'),
-                    logger.info("[%s] Now scanning subdirectory %s" % (dir.decode(headphones.SYS_ENCODING, 'replace'), subdirectory.decode(headphones.SYS_ENCODING, 'replace')))
+                        subdirectory.decode(headphones.SYS_ENCODING, 'replace')))
+                elif latest_subdirectory[file_count] != latest_subdirectory[
+                            file_count - 1] and file_count != 0:
+                    logger.info("[%s] Now scanning subdirectory %s" % (
+                        dir.decode(headphones.SYS_ENCODING, 'replace'),
+                        subdirectory.decode(headphones.SYS_ENCODING, 'replace')))
 
                 song = os.path.join(r, files)
 
@@ -104,10 +113,13 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
                 try:
                     f = MediaFile(song)
                 except (FileTypeError, UnreadableFileError):
-                    logger.warning("Cannot read media file '%s', skipping. It may be corrupted or not a media file.", unicode_song_path)
+                    logger.warning(
+                        "Cannot read media file '%s', skipping. It may be corrupted or not a media file.",
+                        unicode_song_path)
                     continue
                 except IOError:
-                    logger.warning("Cannnot read media file '%s', skipping. Does the file exists?", unicode_song_path)
+                    logger.warning("Cannnot read media file '%s', skipping. Does the file exists?",
+                                   unicode_song_path)
                     continue
 
                 # Grab the bitrates for the auto detect bit rate option
@@ -133,45 +145,52 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
                 controlValueDict = {'Location': unicode_song_path}
 
                 newValueDict = {'TrackID': f.mb_trackid,
-                              #'ReleaseID' : f.mb_albumid,
+                                # 'ReleaseID' : f.mb_albumid,
-                              'ArtistName': f_artist,
+                                'ArtistName': f_artist,
-                              'AlbumTitle': f.album,
+                                'AlbumTitle': f.album,
-                              'TrackNumber': f.track,
+                                'TrackNumber': f.track,
-                              'TrackLength': f.length,
+                                'TrackLength': f.length,
-                              'Genre': f.genre,
+                                'Genre': f.genre,
-                              'Date': f.date,
+                                'Date': f.date,
-                              'TrackTitle': f.title,
+                                'TrackTitle': f.title,
-                              'BitRate': f.bitrate,
+                                'BitRate': f.bitrate,
-                              'Format': f.format,
+                                'Format': f.format,
-                              'CleanName': CleanName
+                                'CleanName': CleanName
-                              }
+                                }
 
-                #song_list.append(song_dict)
+                # song_list.append(song_dict)
-                check_exist_song = myDB.action("SELECT * FROM have WHERE Location=?", [unicode_song_path]).fetchone()
+                check_exist_song = myDB.action("SELECT * FROM have WHERE Location=?",
-                #Only attempt to match songs that are new, haven't yet been matched, or metadata has changed.
+                                               [unicode_song_path]).fetchone()
+                # Only attempt to match songs that are new, haven't yet been matched, or metadata has changed.
                 if not check_exist_song:
-                    #This is a new track
+                    # This is a new track
                     if f_artist:
                         new_artists.append(f_artist)
                     myDB.upsert("have", newValueDict, controlValueDict)
                     new_song_count += 1
                 else:
-                    if check_exist_song['ArtistName'] != f_artist or check_exist_song['AlbumTitle'] != f.album or check_exist_song['TrackTitle'] != f.title:
+                    if check_exist_song['ArtistName'] != f_artist or check_exist_song[
-                        #Important track metadata has been modified, need to run matcher again
+                        'AlbumTitle'] != f.album or check_exist_song['TrackTitle'] != f.title:
+                        # Important track metadata has been modified, need to run matcher again
                         if f_artist and f_artist != check_exist_song['ArtistName']:
                             new_artists.append(f_artist)
-                        elif f_artist and f_artist == check_exist_song['ArtistName'] and check_exist_song['Matched'] != "Ignored":
+                        elif f_artist and f_artist == check_exist_song['ArtistName'] and \
+                                        check_exist_song['Matched'] != "Ignored":
                             new_artists.append(f_artist)
                         else:
                             continue
 
                         newValueDict['Matched'] = None
                         myDB.upsert("have", newValueDict, controlValueDict)
-                        myDB.action('UPDATE tracks SET Location=?, BitRate=?, Format=? WHERE Location=?', [None, None, None, unicode_song_path])
+                        myDB.action(
-                        myDB.action('UPDATE alltracks SET Location=?, BitRate=?, Format=? WHERE Location=?', [None, None, None, unicode_song_path])
+                            'UPDATE tracks SET Location=?, BitRate=?, Format=? WHERE Location=?',
+                            [None, None, None, unicode_song_path])
+                        myDB.action(
+                            'UPDATE alltracks SET Location=?, BitRate=?, Format=? WHERE Location=?',
+                            [None, None, None, unicode_song_path])
                         new_song_count += 1
                     else:
-                        #This track information hasn't changed
+                        # This track information hasn't changed
                         if f_artist and check_exist_song['Matched'] != "Ignored":
                             new_artists.append(f_artist)
 
@@ -179,9 +198,13 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
 
     # Now we start track matching
     logger.info("%s new/modified songs found and added to the database" % new_song_count)
-    song_list = myDB.action("SELECT * FROM have WHERE Matched IS NULL AND LOCATION LIKE ?", [dir.decode(headphones.SYS_ENCODING, 'replace') + "%"])
+    song_list = myDB.action("SELECT * FROM have WHERE Matched IS NULL AND LOCATION LIKE ?",
-    total_number_of_songs = myDB.action("SELECT COUNT(*) FROM have WHERE Matched IS NULL AND LOCATION LIKE ?", [dir.decode(headphones.SYS_ENCODING, 'replace') + "%"]).fetchone()[0]
+                            [dir.decode(headphones.SYS_ENCODING, 'replace') + "%"])
-    logger.info("Found " + str(total_number_of_songs) + " new/modified tracks in: '" + dir.decode(headphones.SYS_ENCODING, 'replace') + "'. Matching tracks to the appropriate releases....")
+    total_number_of_songs = \
+        myDB.action("SELECT COUNT(*) FROM have WHERE Matched IS NULL AND LOCATION LIKE ?",
+                    [dir.decode(headphones.SYS_ENCODING, 'replace') + "%"]).fetchone()[0]
+    logger.info("Found " + str(total_number_of_songs) + " new/modified tracks in: '" + dir.decode(
+        headphones.SYS_ENCODING, 'replace') + "'. Matching tracks to the appropriate releases....")
 
     # Sort the song_list by most vague (e.g. no trackid or releaseid) to most specific (both trackid & releaseid)
     # When we insert into the database, the tracks with the most specific information will overwrite the more general matches
@@ -192,6 +215,7 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
     # We'll use this to give a % completion, just because the track matching might take a while
     song_count = 0
     latest_artist = []
+    last_completion_percentage = 0
 
     for song in song_list:
 
@@ -202,27 +226,30 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
             logger.info("Now matching songs by %s" % song['ArtistName'])
 
         song_count += 1
-        completion_percentage = float(song_count) / total_number_of_songs * 100
+        completion_percentage = math.floor(float(song_count) / total_number_of_songs * 1000) / 10
 
-        if completion_percentage % 10 == 0:
+        if completion_percentage >= (last_completion_percentage + 10):
             logger.info("Track matching is " + str(completion_percentage) + "% complete")
+            last_completion_percentage = completion_percentage
 
-        #THE "MORE-SPECIFIC" CLAUSES HERE HAVE ALL BEEN REMOVED.  WHEN RUNNING A LIBRARY SCAN, THE ONLY CLAUSES THAT
+        # THE "MORE-SPECIFIC" CLAUSES HERE HAVE ALL BEEN REMOVED.  WHEN RUNNING A LIBRARY SCAN, THE ONLY CLAUSES THAT
-        #EVER GOT HIT WERE [ARTIST/ALBUM/TRACK] OR CLEANNAME.  ARTISTID & RELEASEID ARE NEVER PASSED TO THIS FUNCTION,
+        # EVER GOT HIT WERE [ARTIST/ALBUM/TRACK] OR CLEANNAME.  ARTISTID & RELEASEID ARE NEVER PASSED TO THIS FUNCTION,
-        #ARE NEVER FOUND, AND THE OTHER CLAUSES WERE NEVER HIT.  FURTHERMORE, OTHER MATCHING FUNCTIONS IN THIS PROGRAM
+        # ARE NEVER FOUND, AND THE OTHER CLAUSES WERE NEVER HIT.  FURTHERMORE, OTHER MATCHING FUNCTIONS IN THIS PROGRAM
-        #(IMPORTER.PY, MB.PY) SIMPLY DO A [ARTIST/ALBUM/TRACK] OR CLEANNAME MATCH, SO IT'S ALL CONSISTENT.
+        # (IMPORTER.PY, MB.PY) SIMPLY DO A [ARTIST/ALBUM/TRACK] OR CLEANNAME MATCH, SO IT'S ALL CONSISTENT.
 
         if song['ArtistName'] and song['AlbumTitle'] and song['TrackTitle']:
 
-            track = myDB.action('SELECT ArtistName, AlbumTitle, TrackTitle, AlbumID from tracks WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [song['ArtistName'], song['AlbumTitle'], song['TrackTitle']]).fetchone()
+            track = myDB.action(
+                'SELECT ArtistName, AlbumTitle, TrackTitle, AlbumID from tracks WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?',
+                [song['ArtistName'], song['AlbumTitle'], song['TrackTitle']]).fetchone()
             have_updated = False
             if track:
                 controlValueDict = {'ArtistName': track['ArtistName'],
-                                     'AlbumTitle': track['AlbumTitle'],
+                                    'AlbumTitle': track['AlbumTitle'],
-                                     'TrackTitle': track['TrackTitle']}
+                                    'TrackTitle': track['TrackTitle']}
                 newValueDict = {'Location': song['Location'],
-                                 'BitRate': song['BitRate'],
+                                'BitRate': song['BitRate'],
-                                 'Format': song['Format']}
+                                'Format': song['Format']}
                 myDB.upsert("tracks", newValueDict, controlValueDict)
 
                 controlValueDict2 = {'Location': song['Location']}
@@ -230,12 +257,13 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
                 myDB.upsert("have", newValueDict2, controlValueDict2)
                 have_updated = True
             else:
-                track = myDB.action('SELECT CleanName, AlbumID from tracks WHERE CleanName LIKE ?', [song['CleanName']]).fetchone()
+                track = myDB.action('SELECT CleanName, AlbumID from tracks WHERE CleanName LIKE ?',
+                                    [song['CleanName']]).fetchone()
                 if track:
                     controlValueDict = {'CleanName': track['CleanName']}
                     newValueDict = {'Location': song['Location'],
-                                     'BitRate': song['BitRate'],
+                                    'BitRate': song['BitRate'],
-                                     'Format': song['Format']}
+                                    'Format': song['Format']}
                     myDB.upsert("tracks", newValueDict, controlValueDict)
 
                     controlValueDict2 = {'Location': song['Location']}
@@ -248,26 +276,30 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
                     myDB.upsert("have", newValueDict2, controlValueDict2)
                     have_updated = True
 
-            alltrack = myDB.action('SELECT ArtistName, AlbumTitle, TrackTitle, AlbumID from alltracks WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [song['ArtistName'], song['AlbumTitle'], song['TrackTitle']]).fetchone()
+            alltrack = myDB.action(
+                'SELECT ArtistName, AlbumTitle, TrackTitle, AlbumID from alltracks WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?',
+                [song['ArtistName'], song['AlbumTitle'], song['TrackTitle']]).fetchone()
             if alltrack:
                 controlValueDict = {'ArtistName': alltrack['ArtistName'],
-                                     'AlbumTitle': alltrack['AlbumTitle'],
+                                    'AlbumTitle': alltrack['AlbumTitle'],
-                                     'TrackTitle': alltrack['TrackTitle']}
+                                    'TrackTitle': alltrack['TrackTitle']}
                 newValueDict = {'Location': song['Location'],
-                                 'BitRate': song['BitRate'],
+                                'BitRate': song['BitRate'],
-                                 'Format': song['Format']}
+                                'Format': song['Format']}
                 myDB.upsert("alltracks", newValueDict, controlValueDict)
 
                 controlValueDict2 = {'Location': song['Location']}
                 newValueDict2 = {'Matched': alltrack['AlbumID']}
                 myDB.upsert("have", newValueDict2, controlValueDict2)
             else:
-                alltrack = myDB.action('SELECT CleanName, AlbumID from alltracks WHERE CleanName LIKE ?', [song['CleanName']]).fetchone()
+                alltrack = myDB.action(
+                    'SELECT CleanName, AlbumID from alltracks WHERE CleanName LIKE ?',
+                    [song['CleanName']]).fetchone()
                 if alltrack:
                     controlValueDict = {'CleanName': alltrack['CleanName']}
                     newValueDict = {'Location': song['Location'],
-                                     'BitRate': song['BitRate'],
+                                    'BitRate': song['BitRate'],
-                                     'Format': song['Format']}
+                                    'Format': song['Format']}
                     myDB.upsert("alltracks", newValueDict, controlValueDict)
 
                     controlValueDict2 = {'Location': song['Location']}
@@ -285,86 +317,98 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
             newValueDict2 = {'Matched': "Failed"}
             myDB.upsert("have", newValueDict2, controlValueDict2)
 
-        #######myDB.action('INSERT INTO have (ArtistName, AlbumTitle, TrackNumber, TrackTitle, TrackLength, BitRate, Genre, Date, TrackID, Location, CleanName, Format) VALUES( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', [song['ArtistName'], song['AlbumTitle'], song['TrackNumber'], song['TrackTitle'], song['TrackLength'], song['BitRate'], song['Genre'], song['Date'], song['TrackID'], song['Location'], CleanName, song['Format']])
+            #######myDB.action('INSERT INTO have (ArtistName, AlbumTitle, TrackNumber, TrackTitle, TrackLength, BitRate, Genre, Date, TrackID, Location, CleanName, Format) VALUES( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', [song['ArtistName'], song['AlbumTitle'], song['TrackNumber'], song['TrackTitle'], song['TrackLength'], song['BitRate'], song['Genre'], song['Date'], song['TrackID'], song['Location'], CleanName, song['Format']])
 
-    logger.info('Completed matching tracks from directory: %s' % dir.decode(headphones.SYS_ENCODING, 'replace'))
+    logger.info('Completed matching tracks from directory: %s' % dir.decode(headphones.SYS_ENCODING,
+                                                                            'replace'))
 
-    if not append:
+    if not append or artistScan:
         logger.info('Updating scanned artist track counts')
 
         # Clean up the new artist list
         unique_artists = {}.fromkeys(new_artists).keys()
         current_artists = myDB.select('SELECT ArtistName, ArtistID from artists')
 
-        #There was a bug where artists with special characters (-,') would show up in new artists.
+        # There was a bug where artists with special characters (-,') would show up in new artists.
         artist_list = [
             x for x in unique_artists
             if helpers.cleanName(x).lower() not in [
                 helpers.cleanName(y[0]).lower()
                 for y in current_artists
+                ]
             ]
-        ]
         artists_checked = [
             x for x in unique_artists
             if helpers.cleanName(x).lower() in [
                 helpers.cleanName(y[0]).lower()
                 for y in current_artists
+                ]
             ]
-        ]
 
         # Update track counts
-
         for artist in artists_checked:
             # Have tracks are selected from tracks table and not all tracks because of duplicates
             # We update the track count upon an album switch to compliment this
             havetracks = (
-                len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistName like ? AND Location IS NOT NULL', [artist]))
+                len(myDB.select(
-                + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"', [artist]))
+                    'SELECT TrackTitle from tracks WHERE ArtistName like ? AND Location IS NOT NULL',
+                    [artist])) + len(myDB.select(
+                    'SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"',
+                    [artist]))
             )
-            #Note, some people complain about having "artist have tracks" > # of tracks total in artist official releases
+            # Note: some people complain about having "artist have tracks" > # of tracks total in artist official releases
             # (can fix by getting rid of second len statement)
             myDB.action('UPDATE artists SET HaveTracks=? WHERE ArtistName=?', [havetracks, artist])
 
         logger.info('Found %i new artists' % len(artist_list))
 
-        if len(artist_list):
+        if artist_list:
             if headphones.CONFIG.AUTO_ADD_ARTISTS:
                 logger.info('Importing %i new artists' % len(artist_list))
                 importer.artistlist_to_mbids(artist_list)
             else:
                 logger.info('To add these artists, go to Manage->Manage New Artists')
-                #myDB.action('DELETE from newartists')
+                # myDB.action('DELETE from newartists')
                 for artist in artist_list:
                     myDB.action('INSERT OR IGNORE INTO newartists VALUES (?)', [artist])
 
-        if headphones.CONFIG.DETECT_BITRATE:
+        if headphones.CONFIG.DETECT_BITRATE and bitrates:
             headphones.CONFIG.PREFERRED_BITRATE = sum(bitrates) / len(bitrates) / 1000
 
     else:
         # If we're appending a new album to the database, update the artists total track counts
         logger.info('Updating artist track counts')
 
-        havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [ArtistID])) + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"', [ArtistName]))
+        havetracks = len(
+            myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL',
+                        [ArtistID])) + len(myDB.select(
+            'SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"',
+            [ArtistName]))
         myDB.action('UPDATE artists SET HaveTracks=? WHERE ArtistID=?', [havetracks, ArtistID])
 
     if not append:
         update_album_status()
+
+    if not append and not artistScan:
         lastfm.getSimilar()
 
     logger.info('Library scan complete')
 
-#ADDED THIS SECTION TO MARK ALBUMS AS DOWNLOADED IF ARTISTS ARE ADDED EN MASSE BEFORE LIBRARY IS SCANNED
+
+# ADDED THIS SECTION TO MARK ALBUMS AS DOWNLOADED IF ARTISTS ARE ADDED EN MASSE BEFORE LIBRARY IS SCANNED
 
 
 def update_album_status(AlbumID=None):
     myDB = db.DBConnection()
     logger.info('Counting matched tracks to mark albums as skipped/downloaded')
     if AlbumID:
-        album_status_updater = myDB.action('SELECT AlbumID, AlbumTitle, Status from albums WHERE AlbumID=?', [AlbumID])
+        album_status_updater = myDB.action(
+            'SELECT AlbumID, AlbumTitle, Status from albums WHERE AlbumID=?', [AlbumID])
     else:
         album_status_updater = myDB.action('SELECT AlbumID, AlbumTitle, Status from albums')
     for album in album_status_updater:
-        track_counter = myDB.action('SELECT Location from tracks where AlbumID=?', [album['AlbumID']])
+        track_counter = myDB.action('SELECT Location from tracks where AlbumID=?',
+                                    [album['AlbumID']])
         total_tracks = 0
         have_tracks = 0
         for track in track_counter:
@@ -377,14 +421,14 @@ def update_album_status(AlbumID=None):
             album_completion = 0
             logger.info('Album %s does not have any tracks in database' % album['AlbumTitle'])
 
-        if album_completion >= headphones.CONFIG.ALBUM_COMPLETION_PCT and album['Status'] == 'Skipped':
+        if album_completion >= headphones.CONFIG.ALBUM_COMPLETION_PCT:
             new_album_status = "Downloaded"
 
         # I don't think we want to change Downloaded->Skipped.....
         # I think we can only automatically change Skipped->Downloaded when updating
         # There was a bug report where this was causing infinite downloads if the album was
         # recent, but matched to less than 80%. It would go Downloaded->Skipped->Wanted->Downloaded->Skipped->Wanted->etc....
-        #else:
+        # else:
         #    if album['Status'] == "Skipped" or album['Status'] == "Downloaded":
         #        new_album_status = "Skipped"
         #    else:

headphones/lock.py

@@ -2,11 +2,12 @@
 Locking-related classes
 """
 
-import headphones.logger
 import time
 import threading
 import Queue
 
+import headphones.logger
+
 
 class TimedLock(object):
     """
@@ -39,12 +40,12 @@ class TimedLock(object):
         sleep_amount = self.minimum_delta - delta
         if sleep_amount >= 0:
             # zero sleeps give the cpu a chance to task-switch
-            headphones.logger.info('Sleeping %s (interval)', sleep_amount)
+            headphones.logger.debug('Sleeping %s (interval)', sleep_amount)
             time.sleep(sleep_amount)
         while not self.queue.empty():
             try:
                 seconds = self.queue.get(False)
-                headphones.logger.info('Sleeping %s (queued)', seconds)
+                headphones.logger.debug('Sleeping %s (queued)', seconds)
                 time.sleep(seconds)
             except Queue.Empty:
                 continue
@@ -59,15 +60,14 @@ class TimedLock(object):
 
     def snooze(self, seconds):
         """
-        Asynchronously add time to the next request.
+        Asynchronously add time to the next request. Can be called outside
-        Can be called outside
         of the lock context, but it is possible for the next lock holder
         to not check the queue until after something adds time to it.
         """
-        # we use a queue so that we don't have to synchronize
+        # We use a queue so that we don't have to synchronize
         # across threads and with or without locks
         headphones.logger.info('Adding %s to queue', seconds)
-        self.queue.add(seconds)
+        self.queue.put(seconds)
 
 
 class FakeLock(object):

headphones/logger.py

@@ -13,24 +13,24 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import helpers
-
-from logutils.queue import QueueHandler, QueueListener
 from logging import handlers
-
 import multiprocessing
 import contextlib
-import headphones
 import threading
 import traceback
 import logging
 import errno
 import sys
+
 import os
+from headphones import helpers
+from logutils.queue import QueueHandler, QueueListener
+import headphones
+
 
 # These settings are for file logging only
 FILENAME = "headphones.log"
-MAX_SIZE = 1000000 # 1 MB
+MAX_SIZE = 1000000  # 1 MB
 MAX_FILES = 5
 
 # Headphones logger
@@ -39,6 +39,7 @@ logger = logging.getLogger("headphones")
 # Global queue for multiprocessing logging
 queue = None
 
+
 class LogListHandler(logging.Handler):
     """
     Log handler for Web UI.
@@ -71,8 +72,8 @@ def listener():
             # http://stackoverflow.com/questions/2009278 for more information.
             if e.errno == errno.EACCES:
                 logger.warning("Multiprocess logging disabled, because "
-                    "current user cannot map shared memory. You won't see any" \
+                               "current user cannot map shared memory. You won't see any" \
-                    "logging generated by the worker processed.")
+                               "logging generated by the worker processed.")
 
     # Multiprocess logging may be disabled.
     if not queue:
@@ -149,8 +150,10 @@ def initLogger(console=False, log_dir=False, verbose=False):
     if log_dir:
         filename = os.path.join(log_dir, FILENAME)
 
-        file_formatter = logging.Formatter('%(asctime)s - %(levelname)-7s :: %(threadName)s : %(message)s', '%d-%b-%Y %H:%M:%S')
+        file_formatter = logging.Formatter(
-        file_handler = handlers.RotatingFileHandler(filename, maxBytes=MAX_SIZE, backupCount=MAX_FILES)
+            '%(asctime)s - %(levelname)-7s :: %(threadName)s : %(message)s', '%d-%b-%Y %H:%M:%S')
+        file_handler = handlers.RotatingFileHandler(filename, maxBytes=MAX_SIZE,
+                                                    backupCount=MAX_FILES)
         file_handler.setLevel(logging.DEBUG)
         file_handler.setFormatter(file_formatter)
 
@@ -158,7 +161,8 @@ def initLogger(console=False, log_dir=False, verbose=False):
 
     # Setup console logger
     if console:
-        console_formatter = logging.Formatter('%(asctime)s - %(levelname)s :: %(threadName)s : %(message)s', '%d-%b-%Y %H:%M:%S')
+        console_formatter = logging.Formatter(
+            '%(asctime)s - %(levelname)s :: %(threadName)s : %(message)s', '%d-%b-%Y %H:%M:%S')
         console_handler = logging.StreamHandler()
         console_handler.setFormatter(console_formatter)
         console_handler.setLevel(logging.DEBUG)
@@ -212,11 +216,13 @@ def initHooks(global_exceptions=True, thread_exceptions=True, pass_original=True
                     raise
                 except:
                     excepthook(*sys.exc_info())
+
             self.run = new_run
 
         # Monkey patch the run() by monkey patching the __init__ method
         threading.Thread.__init__ = new_init
 
+
 # Expose logger methods
 info = logger.info
 warn = logger.warn

headphones/lyrics.py

@@ -13,18 +13,17 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import re
 import htmlentitydefs
 
+import re
 from headphones import logger, request
 
 
 def getLyrics(artist, song):
-
     params = {"artist": artist.encode('utf-8'),
-                "song": song.encode('utf-8'),
+              "song": song.encode('utf-8'),
-                "fmt": 'xml'
+              "fmt": 'xml'
-                }
+              }
 
     url = 'http://lyrics.wikia.com/api.php'
     data = request.request_minidom(url, params=params)
@@ -46,10 +45,13 @@ def getLyrics(artist, song):
         logger.warn('Error fetching lyrics from: %s' % lyricsurl)
         return
 
-    m = re.compile('''<div class='lyricbox'><div class='rtMatcher'>.*?</div>(.*?)<!--''').search(lyricspage)
+    m = re.compile('''<div class='lyricbox'><div class='rtMatcher'>.*?</div>(.*?)<!--''').search(
+        lyricspage)
 
     if not m:
-        m = re.compile('''<div class='lyricbox'><span style="padding:1em"><a href="/Category:Instrumental" title="Instrumental">''').search(lyricspage)
+        m = re.compile(
+            '''<div class='lyricbox'><span style="padding:1em"><a href="/Category:Instrumental" title="Instrumental">''').search(
+            lyricspage)
         if m:
             return u'(Instrumental)'
         else:
@@ -67,12 +69,12 @@ def convert_html_entities(s):
     if len(matches) > 0:
         hits = set(matches)
         for hit in hits:
-                name = hit[2:-1]
+            name = hit[2:-1]
-                try:
+            try:
-                        entnum = int(name)
+                entnum = int(name)
-                        s = s.replace(hit, unichr(entnum))
+                s = s.replace(hit, unichr(entnum))
-                except ValueError:
+            except ValueError:
-                        pass
+                pass
 
     matches = re.findall("&\w+;", s)
     hits = set(matches)

headphones/mb.py

@@ -31,12 +31,12 @@ except ImportError:
 
 mb_lock = headphones.lock.TimedLock(0)
 
+
 # Quick fix to add mirror switching on the fly. Need to probably return the mbhost & mbport that's
 # being used, so we can send those values to the log
 
 
 def startmb():
-
     mbuser = None
     mbpass = None
 
@@ -47,9 +47,11 @@ def startmb():
     elif headphones.CONFIG.MIRROR == "custom":
         mbhost = headphones.CONFIG.CUSTOMHOST
         mbport = int(headphones.CONFIG.CUSTOMPORT)
+        mbuser = headphones.CONFIG.CUSTOMUSER
+        mbpass = headphones.CONFIG.CUSTOMPASS
         sleepytime = int(headphones.CONFIG.CUSTOMSLEEP)
     elif headphones.CONFIG.MIRROR == "headphones":
-        mbhost = "144.76.94.239"
+        mbhost = "codeshy.com"
         mbport = 8181
         mbuser = headphones.CONFIG.HPUSER
         mbpass = headphones.CONFIG.HPPASS
@@ -64,70 +66,86 @@ def startmb():
     if sleepytime == 0:
         musicbrainzngs.set_rate_limit(False)
     else:
-        #calling it with an it ends up blocking all requests after the first
+        # calling it with an it ends up blocking all requests after the first
         musicbrainzngs.set_rate_limit(limit_or_interval=float(sleepytime))
         mb_lock.minimum_delta = sleepytime
 
     # Add headphones credentials
-    if headphones.CONFIG.MIRROR == "headphones":
+    if headphones.CONFIG.MIRROR == "headphones" or headphones.CONFIG.CUSTOMAUTH:
-        if not mbuser and mbpass:
+        if not mbuser or not mbpass:
-            logger.warn("No username or password set for VIP server")
+            logger.warn("No username or password set for MusicBrainz server")
         else:
             musicbrainzngs.hpauth(mbuser, mbpass)
 
-    logger.debug('Using the following server values: MBHost: %s, MBPort: %i, Sleep Interval: %i', mbhost, mbport, sleepytime)
+    # Let us know if we disable custom authentication
+    if not headphones.CONFIG.CUSTOMAUTH and headphones.CONFIG.MIRROR == "custom":
+        musicbrainzngs.disable_hpauth()
+
+    logger.debug('Using the following server values: MBHost: %s, MBPort: %i, Sleep Interval: %i',
+                 mbhost, mbport, sleepytime)
 
     return True
 
 
 def findArtist(name, limit=1):
-     artistlist = []
+    artistlist = []
-     artistResults = None
+    artistResults = None
 
-     chars = set('!?*-')
+    chars = set('!?*-')
-     if any((c in chars) for c in name):
+    if any((c in chars) for c in name):
-         name = '"' + name + '"'
+        name = '"' + name + '"'
 
-     criteria = {'artist': name.lower()}
+    criteria = {'artist': name.lower()}
 
-     with mb_lock:
+    with mb_lock:
-         try:
+        try:
-             artistResults = musicbrainzngs.search_artists(limit=limit, **criteria)['artist-list']
+            artistResults = musicbrainzngs.search_artists(limit=limit, **criteria)['artist-list']
-         except musicbrainzngs.WebServiceError as e:
+        except ValueError as e:
-             logger.warn('Attempt to query MusicBrainz for %s failed (%s)' % (name, str(e)))
+            if "at least one query term is required" in e.message:
-             mb_lock.snooze(5)
+                logger.error(
+                    "Tried to search without a term, or an empty one. Provided artist (probably emtpy): %s",
+                    name)
+                return False
+            else:
+                raise
+        except musicbrainzngs.WebServiceError as e:
+            logger.warn('Attempt to query MusicBrainz for %s failed (%s)' % (name, str(e)))
+            mb_lock.snooze(5)
 
-     if not artistResults:
+    if not artistResults:
-         return False
+        return False
-     for result in artistResults:
+    for result in artistResults:
-         if 'disambiguation' in result:
+        if 'disambiguation' in result:
-             uniquename = unicode(result['sort-name'] + " (" + result['disambiguation'] + ")")
+            uniquename = unicode(result['sort-name'] + " (" + result['disambiguation'] + ")")
-         else:
+        else:
-             uniquename = unicode(result['sort-name'])
+            uniquename = unicode(result['sort-name'])
-         if result['name'] != uniquename and limit == 1:
+        if result['name'] != uniquename and limit == 1:
-             logger.info('Found an artist with a disambiguation: %s - doing an album based search' % name)
+            logger.info(
-             artistdict = findArtistbyAlbum(name)
+                'Found an artist with a disambiguation: %s - doing an album based search' % name)
-             if not artistdict:
+            artistdict = findArtistbyAlbum(name)
-                 logger.info('Cannot determine the best match from an artist/album search. Using top match instead')
+            if not artistdict:
-                 artistlist.append({
+                logger.info(
-                 # Just need the artist id if the limit is 1
+                    'Cannot determine the best match from an artist/album search. Using top match instead')
-                 #    'name':             unicode(result['sort-name']),
+                artistlist.append({
-                 #    'uniquename':        uniquename,
+                    # Just need the artist id if the limit is 1
-                     'id': unicode(result['id']),
+                    #    'name':             unicode(result['sort-name']),
-                 #    'url':                 unicode("http://musicbrainz.org/artist/" + result['id']),#probably needs to be changed
+                    #    'uniquename':        uniquename,
-                 #    'score':            int(result['ext:score'])
+                    'id': unicode(result['id']),
-                     })
+                    #    'url':                 unicode("http://musicbrainz.org/artist/" + result['id']),#probably needs to be changed
-             else:
+                    #    'score':            int(result['ext:score'])
-                 artistlist.append(artistdict)
+                })
-         else:
+            else:
-             artistlist.append({
+                artistlist.append(artistdict)
-                     'name': unicode(result['sort-name']),
+        else:
-                     'uniquename': uniquename,
+            artistlist.append({
-                     'id': unicode(result['id']),
+                'name': unicode(result['sort-name']),
-                     'url': unicode("http://musicbrainz.org/artist/" + result['id']),#probably needs to be changed
+                'uniquename': uniquename,
-                     'score': int(result['ext:score'])
+                'id': unicode(result['id']),
-                     })
+                'url': unicode("http://musicbrainz.org/artist/" + result['id']),
-     return artistlist
+                # probably needs to be changed
+                'score': int(result['ext:score'])
+            })
+    return artistlist
 
 
 def findRelease(name, limit=1, artist=None):
@@ -146,8 +164,9 @@ def findRelease(name, limit=1, artist=None):
 
     with mb_lock:
         try:
-            releaseResults = musicbrainzngs.search_releases(query=name, limit=limit, artist=artist)['release-list']
+            releaseResults = musicbrainzngs.search_releases(query=name, limit=limit, artist=artist)[
-        except musicbrainzngs.WebServiceError as e: #need to update exceptions
+                'release-list']
+        except musicbrainzngs.WebServiceError as e:  # need to update exceptions
             logger.warn('Attempt to query MusicBrainz for "%s" failed: %s' % (name, str(e)))
             mb_lock.snooze(5)
 
@@ -191,28 +210,67 @@ def findRelease(name, limit=1, artist=None):
                     rg_type = secondary_type
 
         releaselist.append({
-                    'uniquename': unicode(result['artist-credit'][0]['artist']['name']),
+            'uniquename': unicode(result['artist-credit'][0]['artist']['name']),
-                    'title': unicode(title),
+            'title': unicode(title),
-                    'id': unicode(result['artist-credit'][0]['artist']['id']),
+            'id': unicode(result['artist-credit'][0]['artist']['id']),
-                    'albumid': unicode(result['id']),
+            'albumid': unicode(result['id']),
-                    'url': unicode("http://musicbrainz.org/artist/" + result['artist-credit'][0]['artist']['id']),#probably needs to be changed
+            'url': unicode(
-                    'albumurl': unicode("http://musicbrainz.org/release/" + result['id']),#probably needs to be changed
+                "http://musicbrainz.org/artist/" + result['artist-credit'][0]['artist']['id']),
-                    'score': int(result['ext:score']),
+            # probably needs to be changed
-                    'date': unicode(result['date']) if 'date' in result else '',
+            'albumurl': unicode("http://musicbrainz.org/release/" + result['id']),
-                    'country': unicode(result['country']) if 'country' in result else '',
+            # probably needs to be changed
-                    'formats': unicode(formats),
+            'score': int(result['ext:score']),
-                    'tracks': unicode(tracks),
+            'date': unicode(result['date']) if 'date' in result else '',
-                    'rgid': unicode(result['release-group']['id']),
+            'country': unicode(result['country']) if 'country' in result else '',
-                    'rgtype': unicode(rg_type)
+            'formats': unicode(formats),
-                    })
+            'tracks': unicode(tracks),
+            'rgid': unicode(result['release-group']['id']),
+            'rgtype': unicode(rg_type)
+        })
     return releaselist
 
 
+def findSeries(name, limit=1):
+    serieslist = []
+    seriesResults = None
+
+    chars = set('!?*-')
+    if any((c in chars) for c in name):
+        name = '"' + name + '"'
+
+    criteria = {'series': name.lower()}
+
+    with mb_lock:
+        try:
+            seriesResults = musicbrainzngs.search_series(limit=limit, **criteria)['series-list']
+        except musicbrainzngs.WebServiceError as e:
+            logger.warn('Attempt to query MusicBrainz for %s failed (%s)' % (name, str(e)))
+            mb_lock.snooze(5)
+
+    if not seriesResults:
+        return False
+    for result in seriesResults:
+        if 'disambiguation' in result:
+            uniquename = unicode(result['name'] + " (" + result['disambiguation'] + ")")
+        else:
+            uniquename = unicode(result['name'])
+        serieslist.append({
+            'uniquename': uniquename,
+            'name': unicode(result['name']),
+            'type': unicode(result['type']),
+            'id': unicode(result['id']),
+            'url': unicode("http://musicbrainz.org/series/" + result['id']),
+            # probably needs to be changed
+            'score': int(result['ext:score'])
+        })
+    return serieslist
+
+
 def getArtist(artistid, extrasonly=False):
     artist_dict = {}
     artist = None
     try:
-        limit = 200
+        limit = 100
         with mb_lock:
             artist = musicbrainzngs.get_artist_by_id(artistid)['artist']
         newRgs = None
@@ -227,7 +285,9 @@ def getArtist(artistid, extrasonly=False):
             newRgs = newRgs['release-group-list']
             artist['release-group-list'] += newRgs
     except musicbrainzngs.WebServiceError as e:
-        logger.warn('Attempt to retrieve artist information from MusicBrainz failed for artistid: %s (%s)' % (artistid, str(e)))
+        logger.warn(
+            'Attempt to retrieve artist information from MusicBrainz failed for artistid: %s (%s)' % (
+            artistid, str(e)))
         mb_lock.snooze(5)
     except Exception as e:
         pass
@@ -241,7 +301,7 @@ def getArtist(artistid, extrasonly=False):
 
     if not extrasonly:
         for rg in artist['release-group-list']:
-            if "secondary-type-list" in rg.keys(): #only add releases without a secondary type
+            if "secondary-type-list" in rg.keys():  # only add releases without a secondary type
                 continue
             releasegroups.append({
                 'title': unicode(rg['title']),
@@ -255,7 +315,8 @@ def getArtist(artistid, extrasonly=False):
     myDB = db.DBConnection()
 
     try:
-        db_artist = myDB.action('SELECT IncludeExtras, Extras from artists WHERE ArtistID=?', [artistid]).fetchone()
+        db_artist = myDB.action('SELECT IncludeExtras, Extras from artists WHERE ArtistID=?',
+                                [artistid]).fetchone()
         includeExtras = db_artist['IncludeExtras']
     except IndexError:
         includeExtras = False
@@ -282,7 +343,7 @@ def getArtist(artistid, extrasonly=False):
             mb_extras_list = []
 
             try:
-                limit = 200
+                limit = 100
                 newRgs = None
                 while newRgs is None or len(newRgs) >= limit:
                     with mb_lock:
@@ -291,7 +352,9 @@ def getArtist(artistid, extrasonly=False):
                     newRgs = newRgs['release-group-list']
                     mb_extras_list += newRgs
             except musicbrainzngs.WebServiceError as e:
-                logger.warn('Attempt to retrieve artist information from MusicBrainz failed for artistid: %s (%s)' % (artistid, str(e)))
+                logger.warn(
+                    'Attempt to retrieve artist information from MusicBrainz failed for artistid: %s (%s)' % (
+                    artistid, str(e)))
                 mb_lock.snooze(5)
 
             for rg in mb_extras_list:
@@ -311,6 +374,44 @@ def getArtist(artistid, extrasonly=False):
     return artist_dict
 
 
+def getSeries(seriesid):
+    series_dict = {}
+    series = None
+    try:
+        with mb_lock:
+            series = musicbrainzngs.get_series_by_id(seriesid, includes=['release-group-rels'])[
+                'series']
+    except musicbrainzngs.WebServiceError as e:
+        logger.warn(
+            'Attempt to retrieve series information from MusicBrainz failed for seriesid: %s (%s)' % (
+            seriesid, str(e)))
+        mb_lock.snooze(5)
+    except Exception as e:
+        pass
+
+    if not series:
+        return False
+
+    if 'disambiguation' in series:
+        series_dict['artist_name'] = unicode(
+            series['name'] + " (" + unicode(series['disambiguation']) + ")")
+    else:
+        series_dict['artist_name'] = unicode(series['name'])
+
+    releasegroups = []
+
+    for rg in series['release_group-relation-list']:
+        releasegroup = rg['release-group']
+        releasegroups.append({
+            'title': releasegroup['title'],
+            'date': releasegroup['first-release-date'],
+            'id': releasegroup['id'],
+            'type': rg['type']
+        })
+    series_dict['releasegroups'] = releasegroups
+    return series_dict
+
+
 def getReleaseGroup(rgid):
     """
     Returns a list of releases in a release group
@@ -322,7 +423,9 @@ def getReleaseGroup(rgid):
                 rgid, ["artists", "releases", "media", "discids", ])
             releaseGroup = releaseGroup['release-group']
     except musicbrainzngs.WebServiceError as e:
-        logger.warn('Attempt to retrieve information from MusicBrainz for release group "%s" failed (%s)' % (rgid, str(e)))
+        logger.warn(
+            'Attempt to retrieve information from MusicBrainz for release group "%s" failed (%s)' % (
+            rgid, str(e)))
         mb_lock.snooze(5)
 
     if not releaseGroup:
@@ -341,11 +444,16 @@ def getRelease(releaseid, include_artist_info=True):
     try:
         with mb_lock:
             if include_artist_info:
-                results = musicbrainzngs.get_release_by_id(releaseid, ["artists", "release-groups", "media", "recordings"]).get('release')
+                results = musicbrainzngs.get_release_by_id(releaseid,
+                                                           ["artists", "release-groups", "media",
+                                                            "recordings"]).get('release')
             else:
-                results = musicbrainzngs.get_release_by_id(releaseid, ["media", "recordings"]).get('release')
+                results = musicbrainzngs.get_release_by_id(releaseid, ["media", "recordings"]).get(
+                    'release')
     except musicbrainzngs.WebServiceError as e:
-        logger.warn('Attempt to retrieve information from MusicBrainz for release "%s" failed (%s)' % (releaseid, str(e)))
+        logger.warn(
+            'Attempt to retrieve information from MusicBrainz for release "%s" failed (%s)' % (
+            releaseid, str(e)))
         mb_lock.snooze(5)
 
     if not results:
@@ -373,7 +481,8 @@ def getRelease(releaseid, include_artist_info=True):
             try:
                 release['rg_type'] = unicode(results['release-group']['type'])
 
-                if release['rg_type'] == 'Album' and 'secondary-type-list' in results['release-group']:
+                if release['rg_type'] == 'Album' and 'secondary-type-list' in results[
+                    'release-group']:
                     secondary_type = unicode(results['release-group']['secondary-type-list'][0])
                     if secondary_type != release['rg_type']:
                         release['rg_type'] = secondary_type
@@ -393,9 +502,13 @@ def getRelease(releaseid, include_artist_info=True):
 
 
 def get_new_releases(rgid, includeExtras=False, forcefull=False):
-
     myDB = db.DBConnection()
     results = []
+
+    release_status = "official"
+    if includeExtras and not headphones.CONFIG.OFFICIAL_RELEASES_ONLY:
+        release_status = []
+
     try:
         limit = 100
         newResults = None
@@ -404,29 +517,33 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
                 newResults = musicbrainzngs.browse_releases(
                     release_group=rgid,
                     includes=['artist-credits', 'labels', 'recordings', 'release-groups', 'media'],
+                    release_status=release_status,
                     limit=limit,
                     offset=len(results))
             if 'release-list' not in newResults:
-                break #may want to raise an exception here instead ?
+                break  # may want to raise an exception here instead ?
             newResults = newResults['release-list']
             results += newResults
 
     except musicbrainzngs.WebServiceError as e:
-        logger.warn('Attempt to retrieve information from MusicBrainz for release group "%s" failed (%s)' % (rgid, str(e)))
+        logger.warn(
+            'Attempt to retrieve information from MusicBrainz for release group "%s" failed (%s)' % (
+            rgid, str(e)))
         mb_lock.snooze(5)
         return False
 
     if not results or len(results) == 0:
         return False
 
-    #Clean all references to releases in dB that are no longer referenced in musicbrainz
+    # Clean all references to releases in dB that are no longer referenced in musicbrainz
     release_list = []
     force_repackage1 = 0
     if len(results) != 0:
         for release_mark in results:
             release_list.append(unicode(release_mark['id']))
             release_title = release_mark['title']
-        remove_missing_releases = myDB.action("SELECT ReleaseID FROM allalbums WHERE AlbumID=?", [rgid])
+        remove_missing_releases = myDB.action("SELECT ReleaseID FROM allalbums WHERE AlbumID=?",
+                                              [rgid])
         if remove_missing_releases:
             for items in remove_missing_releases:
                 if items['ReleaseID'] not in release_list and items['ReleaseID'] != rgid:
@@ -435,24 +552,24 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
                     myDB.action("DELETE FROM tracks WHERE ReleaseID=?", [items['ReleaseID']])
                     myDB.action("DELETE FROM allalbums WHERE ReleaseID=?", [items['ReleaseID']])
                     myDB.action("DELETE FROM alltracks WHERE ReleaseID=?", [items['ReleaseID']])
-                    logger.info("Removing all references to release %s to reflect MusicBrainz" % items['ReleaseID'])
+                    logger.info(
+                        "Removing all references to release %s to reflect MusicBrainz" % items[
+                            'ReleaseID'])
                     force_repackage1 = 1
     else:
-        logger.info("There was either an error pulling data from MusicBrainz or there might not be any releases for this category")
+        logger.info(
+            "There was either an error pulling data from MusicBrainz or there might not be any releases for this category")
 
     num_new_releases = 0
 
     for releasedata in results:
-        #releasedata.get will return None if it doesn't have a status
-        #all official releases should have the Official status included
-        if not includeExtras and releasedata.get('status') != 'Official':
-            continue
 
         release = {}
         rel_id_check = releasedata['id']
-        album_checker = myDB.action('SELECT * from allalbums WHERE ReleaseID=?', [rel_id_check]).fetchone()
+        album_checker = myDB.action('SELECT * from allalbums WHERE ReleaseID=?',
+                                    [rel_id_check]).fetchone()
         if not album_checker or forcefull:
-            #DELETE all references to this release since we're updating it anyway.
+            # DELETE all references to this release since we're updating it anyway.
             myDB.action('DELETE from allalbums WHERE ReleaseID=?', [rel_id_check])
             myDB.action('DELETE from alltracks WHERE ReleaseID=?', [rel_id_check])
             release['AlbumTitle'] = unicode(releasedata['title'])
@@ -461,7 +578,8 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
             release['ReleaseDate'] = unicode(releasedata['date']) if 'date' in releasedata else None
             release['ReleaseID'] = releasedata['id']
             if 'release-group' not in releasedata:
-                raise Exception('No release group associated with release id ' + releasedata['id'] + ' album id' + rgid)
+                raise Exception('No release group associated with release id ' + releasedata[
+                    'id'] + ' album id' + rgid)
             release['Type'] = unicode(releasedata['release-group']['type'])
 
             if release['Type'] == 'Album' and 'secondary-type-list' in releasedata['release-group']:
@@ -469,7 +587,7 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
                 if secondary_type != release['Type']:
                     release['Type'] = secondary_type
 
-            #making the assumption that the most important artist will be first in the list
+            # making the assumption that the most important artist will be first in the list
             if 'artist-credit' in releasedata:
                 release['ArtistID'] = unicode(releasedata['artist-credit'][0]['artist']['id'])
                 release['ArtistName'] = unicode(releasedata['artist-credit-phrase'])
@@ -477,8 +595,9 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
                 logger.warn('Release ' + releasedata['id'] + ' has no Artists associated.')
                 return False
 
-            release['ReleaseCountry'] = unicode(releasedata['country']) if 'country' in releasedata else u'Unknown'
+            release['ReleaseCountry'] = unicode(
-            #assuming that the list will contain media and that the format will be consistent
+                releasedata['country']) if 'country' in releasedata else u'Unknown'
+            # assuming that the list will contain media and that the format will be consistent
             try:
                 additional_medium = ''
                 for position in releasedata['medium-list']:
@@ -490,16 +609,17 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
                     disc_number = ''
                 else:
                     disc_number = str(medium_count) + 'x'
-                packaged_medium = disc_number + releasedata['medium-list'][0]['format'] + additional_medium
+                packaged_medium = disc_number + releasedata['medium-list'][0][
+                    'format'] + additional_medium
                 release['ReleaseFormat'] = unicode(packaged_medium)
             except:
                 release['ReleaseFormat'] = u'Unknown'
 
             release['Tracks'] = getTracksFromRelease(releasedata)
 
-        # What we're doing here now is first updating the allalbums & alltracks table to the most
+            # What we're doing here now is first updating the allalbums & alltracks table to the most
-        # current info, then moving the appropriate release into the album table and its associated
+            # current info, then moving the appropriate release into the album table and its associated
-        # tracks into the tracks table
+            # tracks into the tracks table
             controlValueDict = {"ReleaseID": release['ReleaseID']}
 
             newValueDict = {"ArtistID": release['ArtistID'],
@@ -511,13 +631,14 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
                             "Type": release['Type'],
                             "ReleaseCountry": release['ReleaseCountry'],
                             "ReleaseFormat": release['ReleaseFormat']
-                        }
+                            }
 
             myDB.upsert("allalbums", newValueDict, controlValueDict)
 
             for track in release['Tracks']:
 
-                cleanname = helpers.cleanName(release['ArtistName'] + ' ' + release['AlbumTitle'] + ' ' + track['title'])
+                cleanname = helpers.cleanName(
+                    release['ArtistName'] + ' ' + release['AlbumTitle'] + ' ' + track['title'])
 
                 controlValueDict = {"TrackID": track['id'],
                                     "ReleaseID": release['ReleaseID']}
@@ -531,30 +652,37 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
                                 "TrackDuration": track['duration'],
                                 "TrackNumber": track['number'],
                                 "CleanName": cleanname
-                            }
+                                }
 
-                match = myDB.action('SELECT Location, BitRate, Format from have WHERE CleanName=?', [cleanname]).fetchone()
+                match = myDB.action('SELECT Location, BitRate, Format from have WHERE CleanName=?',
+                                    [cleanname]).fetchone()
 
                 if not match:
-                    match = myDB.action('SELECT Location, BitRate, Format from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [release['ArtistName'], release['AlbumTitle'], track['title']]).fetchone()
+                    match = myDB.action(
-                #if not match:
+                        'SELECT Location, BitRate, Format from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?',
-                    #match = myDB.action('SELECT Location, BitRate, Format from have WHERE TrackID=?', [track['id']]).fetchone()
+                        [release['ArtistName'], release['AlbumTitle'], track['title']]).fetchone()
+                    # if not match:
+                    # match = myDB.action('SELECT Location, BitRate, Format from have WHERE TrackID=?', [track['id']]).fetchone()
                 if match:
                     newValueDict['Location'] = match['Location']
                     newValueDict['BitRate'] = match['BitRate']
                     newValueDict['Format'] = match['Format']
-                    #myDB.action('UPDATE have SET Matched="True" WHERE Location=?', [match['Location']])
+                    # myDB.action('UPDATE have SET Matched="True" WHERE Location=?', [match['Location']])
-                    myDB.action('UPDATE have SET Matched=? WHERE Location=?', (release['AlbumID'], match['Location']))
+                    myDB.action('UPDATE have SET Matched=? WHERE Location=?',
+                                (release['AlbumID'], match['Location']))
 
                 myDB.upsert("alltracks", newValueDict, controlValueDict)
             num_new_releases = num_new_releases + 1
             if album_checker:
-                logger.info('[%s] Existing release %s (%s) updated' % (release['ArtistName'], release['AlbumTitle'], rel_id_check))
+                logger.info('[%s] Existing release %s (%s) updated' % (
+                release['ArtistName'], release['AlbumTitle'], rel_id_check))
             else:
-                logger.info('[%s] New release %s (%s) added' % (release['ArtistName'], release['AlbumTitle'], rel_id_check))
+                logger.info('[%s] New release %s (%s) added' % (
+                release['ArtistName'], release['AlbumTitle'], rel_id_check))
         if force_repackage1 == 1:
             num_new_releases = -1
-            logger.info('[%s] Forcing repackage of %s, since dB releases have been removed' % (release['ArtistName'], release_title))
+            logger.info('[%s] Forcing repackage of %s, since dB releases have been removed' % (
+            release['ArtistName'], release_title))
         else:
             num_new_releases = num_new_releases
 
@@ -571,23 +699,25 @@ def getTracksFromRelease(release):
             except:
                 track_title = unicode(track['recording']['title'])
             tracks.append({
-                    'number': totalTracks,
+                'number': totalTracks,
-                    'title': track_title,
+                'title': track_title,
-                    'id': unicode(track['recording']['id']),
+                'id': unicode(track['recording']['id']),
-                    'url': u"http://musicbrainz.org/track/" + track['recording']['id'],
+                'url': u"http://musicbrainz.org/track/" + track['recording']['id'],
-                    'duration': int(track['length']) if 'length' in track else 0
+                'duration': int(track['length']) if 'length' in track else 0
-                    })
+            })
             totalTracks += 1
     return tracks
 
+
 # Used when there is a disambiguation
 
 
 def findArtistbyAlbum(name):
-
     myDB = db.DBConnection()
 
-    artist = myDB.action('SELECT AlbumTitle from have WHERE ArtistName=? AND AlbumTitle IS NOT NULL ORDER BY RANDOM()', [name]).fetchone()
+    artist = myDB.action(
+        'SELECT AlbumTitle from have WHERE ArtistName=? AND AlbumTitle IS NOT NULL ORDER BY RANDOM()',
+        [name]).fetchone()
 
     if not artist:
         return False
@@ -614,21 +744,20 @@ def findArtistbyAlbum(name):
     for releaseGroup in results:
         newArtist = releaseGroup['artist-credit'][0]['artist']
         # Only need the artist ID if we're doing an artist+album lookup
-        #if 'disambiguation' in newArtist:
+        # if 'disambiguation' in newArtist:
         #    uniquename = unicode(newArtist['sort-name'] + " (" + newArtist['disambiguation'] + ")")
-        #else:
+        # else:
         #    uniquename = unicode(newArtist['sort-name'])
-        #artist_dict['name'] = unicode(newArtist['sort-name'])
+        # artist_dict['name'] = unicode(newArtist['sort-name'])
-        #artist_dict['uniquename'] = uniquename
+        # artist_dict['uniquename'] = uniquename
         artist_dict['id'] = unicode(newArtist['id'])
-        #artist_dict['url'] = u'http://musicbrainz.org/artist/' + newArtist['id']
+        # artist_dict['url'] = u'http://musicbrainz.org/artist/' + newArtist['id']
-        #artist_dict['score'] = int(releaseGroup['ext:score'])
+        # artist_dict['score'] = int(releaseGroup['ext:score'])
 
     return artist_dict
 
 
 def findAlbumID(artist=None, album=None):
-
     results = None
     chars = set('!?*-')
 
@@ -645,9 +774,11 @@ def findAlbumID(artist=None, album=None):
                 album = '"' + album + '"'
             criteria = {'release': album.lower()}
         with mb_lock:
-            results = musicbrainzngs.search_release_groups(limit=1, **criteria).get('release-group-list')
+            results = musicbrainzngs.search_release_groups(limit=1, **criteria).get(
+                'release-group-list')
     except musicbrainzngs.WebServiceError as e:
-        logger.warn('Attempt to query MusicBrainz for %s - %s failed (%s)' % (artist, album, str(e)))
+        logger.warn(
+            'Attempt to query MusicBrainz for %s - %s failed (%s)' % (artist, album, str(e)))
         mb_lock.snooze(5)
 
     if not results:

headphones/metacritic.py

@@ -0,0 +1,87 @@
+#  This file is part of Headphones.
+#
+#  Headphones is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  Headphones is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
+
+import json
+
+from headphones import db, helpers, logger, request
+
+
+def update(artistid, artist_name, release_groups):
+    """ Pretty simple and crude function to find the artist page on metacritic,
+    then parse that page to get critic & user scores for albums"""
+
+    # First let's modify the artist name to fit the metacritic convention.
+    # We could just do a search, then take the top result, but at least this will
+    # cut down on api calls. If it's ineffective then we'll switch to search
+
+    replacements = {" & ": " ", ".": ""}
+    mc_artist_name = helpers.replace_all(artist_name.lower(), replacements)
+
+    mc_artist_name = mc_artist_name.replace(" ", "-")
+
+    headers = {
+        'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2243.2 Safari/537.36'}
+
+    url = "http://www.metacritic.com/person/" + mc_artist_name + "?filter-options=music&sort_options=date&num_items=100"
+
+    res = request.request_soup(url, headers=headers)
+
+    rows = None
+
+    try:
+        table = res.find("table", class_="credits person_credits")
+        rows = table.tbody.find_all('tr')
+    except:
+        logger.info("Unable to get metacritic scores for: %s" % artist_name)
+
+    myDB = db.DBConnection()
+    artist = myDB.action('SELECT * FROM artists WHERE ArtistID=?', [artistid]).fetchone()
+
+    score_list = []
+
+    # If we couldn't get anything from MetaCritic for whatever reason,
+    # let's try to load scores from the db
+    if not rows:
+        if artist['MetaCritic']:
+            score_list = json.loads(artist['MetaCritic'])
+        else:
+            return
+
+    # If we did get scores, let's update the db with them
+    else:
+        for row in rows:
+            title = row.a.string
+            scores = row.find_all("span")
+            critic_score = scores[0].string
+            user_score = scores[1].string
+            score_dict = {'title': title, 'critic_score': critic_score, 'user_score': user_score}
+            score_list.append(score_dict)
+
+        # Save scores to the database
+        controlValueDict = {"ArtistID": artistid}
+        newValueDict = {'MetaCritic': json.dumps(score_list)}
+        myDB.upsert("artists", newValueDict, controlValueDict)
+
+    for score in score_list:
+        title = score['title']
+        # Iterate through the release groups we got passed to see if we can find
+        # a match
+        for rg in release_groups:
+            if rg['title'].lower() == title.lower():
+                critic_score = score['critic_score']
+                user_score = score['user_score']
+                controlValueDict = {"AlbumID": rg['id']}
+                newValueDict = {'CriticScore': critic_score, 'UserScore': user_score}
+                myDB.upsert("albums", newValueDict, controlValueDict)

headphones/music_encoder.py

@@ -13,16 +13,17 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import os
 import time
 import shutil
 import subprocess
-import headphones
 import multiprocessing
 
+import os
+import headphones
 from headphones import logger
 from beets.mediafile import MediaFile
 
+
 # xld
 import getXldProfile
 
@@ -32,9 +33,11 @@ def encode(albumPath):
 
     # Return if xld details not found
     if use_xld:
-        (xldProfile, xldFormat, xldBitrate) = getXldProfile.getXldProfile(headphones.CONFIG.XLDPROFILE)
+        (xldProfile, xldFormat, xldBitrate) = getXldProfile.getXldProfile(
+            headphones.CONFIG.XLDPROFILE)
         if not xldFormat:
-            logger.error('Details for xld profile \'%s\' not found, files will not be re-encoded', xldProfile)
+            logger.error('Details for xld profile \'%s\' not found, files will not be re-encoded',
+                         xldProfile)
             return None
     else:
         xldProfile = None
@@ -60,7 +63,8 @@ def encode(albumPath):
         for music in f:
             if any(music.lower().endswith('.' + x.lower()) for x in headphones.MEDIA_FORMATS):
                 if not use_xld:
-                    encoderFormat = headphones.CONFIG.ENCODEROUTPUTFORMAT.encode(headphones.SYS_ENCODING)
+                    encoderFormat = headphones.CONFIG.ENCODEROUTPUTFORMAT.encode(
+                        headphones.SYS_ENCODING)
                 else:
                     xldMusicFile = os.path.join(r, music)
                     xldInfoMusic = MediaFile(xldMusicFile)
@@ -68,9 +72,11 @@ def encode(albumPath):
 
                 if headphones.CONFIG.ENCODERLOSSLESS:
                     ext = os.path.normpath(os.path.splitext(music)[1].lstrip(".")).lower()
-                    if not use_xld and ext == 'flac' or use_xld and (ext != xldFormat and (xldInfoMusic.bitrate / 1000 > 400)):
+                    if not use_xld and ext == 'flac' or use_xld and (
+                            ext != xldFormat and (xldInfoMusic.bitrate / 1000 > 400)):
                         musicFiles.append(os.path.join(r, music))
-                        musicTemp = os.path.normpath(os.path.splitext(music)[0] + '.' + encoderFormat)
+                        musicTemp = os.path.normpath(
+                            os.path.splitext(music)[0] + '.' + encoderFormat)
                         musicTempFiles.append(os.path.join(tempDirEncode, musicTemp))
                     else:
                         logger.debug('%s is already encoded', music)
@@ -86,7 +92,7 @@ def encode(albumPath):
             encoder = os.path.join('/Applications', 'xld')
         elif headphones.CONFIG.ENCODER == 'lame':
             if headphones.SYS_PLATFORM == "win32":
-                ## NEED THE DEFAULT LAME INSTALL ON WIN!
+                # NEED THE DEFAULT LAME INSTALL ON WIN!
                 encoder = "C:/Program Files/lame/lame.exe"
             else:
                 encoder = "lame"
@@ -111,24 +117,31 @@ def encode(albumPath):
 
         if use_xld:
             if xldBitrate and (infoMusic.bitrate / 1000 <= xldBitrate):
-                logger.info('%s has bitrate <= %skb, will not be re-encoded', music.decode(headphones.SYS_ENCODING, 'replace'), xldBitrate)
+                logger.info('%s has bitrate <= %skb, will not be re-encoded',
+                            music.decode(headphones.SYS_ENCODING, 'replace'), xldBitrate)
             else:
                 encode = True
         elif headphones.CONFIG.ENCODER == 'lame':
-            if not any(music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.' + x) for x in ["mp3", "wav"]):
+            if not any(
-                logger.warn('Lame cannot encode %s format for %s, use ffmpeg', os.path.splitext(music)[1], music)
+                    music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.' + x) for x
+                    in ["mp3", "wav"]):
+                logger.warn('Lame cannot encode %s format for %s, use ffmpeg',
+                            os.path.splitext(music)[1], music)
             else:
-                if music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.mp3') and (int(infoMusic.bitrate / 1000) <= headphones.CONFIG.BITRATE):
+                if music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.mp3') and (
-                    logger.info('%s has bitrate <= %skb, will not be re-encoded', music, headphones.CONFIG.BITRATE)
+                    int(infoMusic.bitrate / 1000) <= headphones.CONFIG.BITRATE):
+                    logger.info('%s has bitrate <= %skb, will not be re-encoded', music,
+                                headphones.CONFIG.BITRATE)
                 else:
                     encode = True
         else:
             if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'ogg':
                 if music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.ogg'):
-                    logger.warn('Cannot re-encode .ogg %s', music.decode(headphones.SYS_ENCODING, 'replace'))
+                    logger.warn('Cannot re-encode .ogg %s',
+                                music.decode(headphones.SYS_ENCODING, 'replace'))
                 else:
                     encode = True
-            elif headphones.CONFIG.ENCODEROUTPUTFORMAT == 'mp3' or headphones.CONFIG.ENCODEROUTPUTFORMAT == 'm4a':
+            else:
                 if music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.' + headphones.CONFIG.ENCODEROUTPUTFORMAT) and (int(infoMusic.bitrate / 1000) <= headphones.CONFIG.BITRATE):
                     logger.info('%s has bitrate <= %skb, will not be re-encoded', music, headphones.CONFIG.BITRATE)
                 else:
@@ -155,7 +168,7 @@ def encode(albumPath):
                 processes = headphones.CONFIG.ENCODER_MULTICORE_COUNT
 
             logger.debug("Multi-core encoding enabled, spawning %d processes",
-                processes)
+                         processes)
 
         # Use multiprocessing only if it's worth the overhead. and if it is
         # enabled. If not, then use the old fashioned way.
@@ -185,7 +198,8 @@ def encode(albumPath):
         for dest in musicTempFiles:
             if not os.path.exists(dest):
                 encoder_failed = True
-                logger.error("Encoded file '%s' does not exist in the destination temp directory", dest)
+                logger.error("Encoded file '%s' does not exist in the destination temp directory",
+                             dest)
 
     # No errors, move from temp to parent
     if not encoder_failed and musicTempFiles:
@@ -211,7 +225,9 @@ def encode(albumPath):
 
     # Return with error if any encoding errors
     if encoder_failed:
-        logger.error("One or more files failed to encode. Ensure you have the latest version of %s installed.", headphones.CONFIG.ENCODER)
+        logger.error(
+            "One or more files failed to encode. Ensure you have the latest version of %s installed.",
+            headphones.CONFIG.ENCODER)
         return None
 
     time.sleep(1)
@@ -269,7 +285,8 @@ def command(encoder, musicSource, musicDest, albumPath, xldProfile):
         if not headphones.CONFIG.ADVANCEDENCODER:
             opts.extend(['-h'])
             if headphones.CONFIG.ENCODERVBRCBR == 'cbr':
-                opts.extend(['--resample', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-b', str(headphones.CONFIG.BITRATE)])
+                opts.extend(['--resample', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-b',
+                             str(headphones.CONFIG.BITRATE)])
             elif headphones.CONFIG.ENCODERVBRCBR == 'vbr':
                 opts.extend(['-v', str(headphones.CONFIG.ENCODERQUALITY)])
         else:
@@ -290,7 +307,8 @@ def command(encoder, musicSource, musicDest, albumPath, xldProfile):
             if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'm4a':
                 opts.extend(['-strict', 'experimental'])
             if headphones.CONFIG.ENCODERVBRCBR == 'cbr':
-                opts.extend(['-ar', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-ab', str(headphones.CONFIG.BITRATE) + 'k'])
+                opts.extend(['-ar', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-ab',
+                             str(headphones.CONFIG.BITRATE) + 'k'])
             elif headphones.CONFIG.ENCODERVBRCBR == 'vbr':
                 opts.extend(['-aq', str(headphones.CONFIG.ENCODERQUALITY)])
             opts.extend(['-y', '-ac', '2', '-vn'])
@@ -311,7 +329,8 @@ def command(encoder, musicSource, musicDest, albumPath, xldProfile):
             if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'm4a':
                 opts.extend(['-strict', 'experimental'])
             if headphones.CONFIG.ENCODERVBRCBR == 'cbr':
-                opts.extend(['-ar', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-ab', str(headphones.CONFIG.BITRATE) + 'k'])
+                opts.extend(['-ar', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-ab',
+                             str(headphones.CONFIG.BITRATE) + 'k'])
             elif headphones.CONFIG.ENCODERVBRCBR == 'vbr':
                 opts.extend(['-aq', str(headphones.CONFIG.ENCODERQUALITY)])
             opts.extend(['-y', '-ac', '2', '-vn'])
@@ -337,13 +356,14 @@ def command(encoder, musicSource, musicDest, albumPath, xldProfile):
     logger.debug(subprocess.list2cmdline(cmd))
 
     process = subprocess.Popen(cmd, startupinfo=startupinfo,
-        stdin=open(os.devnull, 'rb'), stdout=subprocess.PIPE,
+                               stdin=open(os.devnull, 'rb'), stdout=subprocess.PIPE,
-        stderr=subprocess.PIPE)
+                               stderr=subprocess.PIPE)
     stdout, stderr = process.communicate(headphones.CONFIG.ENCODER)
 
     # Error if return code not zero
     if process.returncode:
-        logger.error('Encoding failed for %s' % (musicSource.decode(headphones.SYS_ENCODING, 'replace')))
+        logger.error(
+            'Encoding failed for %s' % (musicSource.decode(headphones.SYS_ENCODING, 'replace')))
         out = stdout if stdout else stderr
         out = out.decode(headphones.SYS_ENCODING, 'replace')
         outlast2lines = '\n'.join(out.splitlines()[-2:])

headphones/notifiers.py

@@ -13,27 +13,27 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import logger, helpers, common, request
+from urllib import urlencode
-
+import urllib
-from xml.dom import minidom
+import subprocess
+import json
+from email.mime.text import MIMEText
+import smtplib
+import email.utils
 from httplib import HTTPSConnection
 from urlparse import parse_qsl
-from urllib import urlencode
-from pynma import pynma
-
-import base64
-import cherrypy
-import urllib
 import urllib2
-import headphones
-import os.path
-import subprocess
-import gntp.notifier
-import json
 
+import os.path
+from headphones import logger, helpers, common, request
+from pynma import pynma
+import cherrypy
+import headphones
+import gntp.notifier
 import oauth2 as oauth
 import pythontwitter as twitter
 
+
 class GROWL(object):
     """
     Growl notifications, for OS X.
@@ -90,7 +90,7 @@ class GROWL(object):
 
         # Send it, including an image
         image_file = os.path.join(str(headphones.PROG_DIR),
-            "data/images/headphoneslogo.png")
+                                  "data/images/headphoneslogo.png")
 
         with open(image_file, 'rb') as f:
             image = f.read()
@@ -109,7 +109,7 @@ class GROWL(object):
         logger.info(u"Growl notifications sent.")
 
     def updateLibrary(self):
-        #For uniformity reasons not removed
+        # For uniformity reasons not removed
         return
 
     def test(self, host, password):
@@ -146,24 +146,24 @@ class PROWL(object):
                 'priority': headphones.CONFIG.PROWL_PRIORITY}
 
         http_handler.request("POST",
-                                "/publicapi/add",
+                             "/publicapi/add",
-                                headers={'Content-type': "application/x-www-form-urlencoded"},
+                             headers={'Content-type': "application/x-www-form-urlencoded"},
-                                body=urlencode(data))
+                             body=urlencode(data))
         response = http_handler.getresponse()
         request_status = response.status
 
         if request_status == 200:
-                logger.info(u"Prowl notifications sent.")
+            logger.info(u"Prowl notifications sent.")
-                return True
+            return True
         elif request_status == 401:
-                logger.info(u"Prowl auth failed: %s" % response.reason)
+            logger.info(u"Prowl auth failed: %s" % response.reason)
-                return False
+            return False
         else:
-                logger.info(u"Prowl notification failed.")
+            logger.info(u"Prowl notification failed.")
-                return False
+            return False
 
     def updateLibrary(self):
-        #For uniformity reasons not removed
+        # For uniformity reasons not removed
         return
 
     def test(self, keys, priority):
@@ -180,7 +180,6 @@ class MPC(object):
     """
 
     def __init__(self):
-
         pass
 
     def notify(self):
@@ -213,9 +212,11 @@ class XBMC(object):
         url = host + '/jsonrpc'
 
         if self.password:
-            response = request.request_json(url, method="post", data=json.dumps(data), headers=headers, auth=(self.username, self.password))
+            response = request.request_json(url, method="post", data=json.dumps(data),
+                                            headers=headers, auth=(self.username, self.password))
         else:
-            response = request.request_json(url, method="post", data=json.dumps(data), headers=headers)
+            response = request.request_json(url, method="post", data=json.dumps(data),
+                                            headers=headers)
 
         if response:
             return response[0]['result']
@@ -239,20 +240,24 @@ class XBMC(object):
 
         header = "Headphones"
         message = "%s - %s added to your library" % (artist, album)
-        time = "3000" # in ms
+        time = "3000"  # in ms
 
         for host in hosts:
             logger.info('Sending notification command to XMBC @ ' + host)
             try:
-                version = self._sendjson(host, 'Application.GetProperties', {'properties': ['version']})['version']['major']
+                version = \
+                self._sendjson(host, 'Application.GetProperties', {'properties': ['version']})[
+                    'version']['major']
 
-                if version < 12: #Eden
+                if version < 12:  # Eden
                     notification = header + "," + message + "," + time + "," + albumartpath
-                    notifycommand = {'command': 'ExecBuiltIn', 'parameter': 'Notification(' + notification + ')'}
+                    notifycommand = {'command': 'ExecBuiltIn',
+                                     'parameter': 'Notification(' + notification + ')'}
                     request = self._sendhttp(host, notifycommand)
 
-                else: #Frodo
+                else:  # Frodo
-                    params = {'title': header, 'message': message, 'displaytime': int(time), 'image': albumartpath}
+                    params = {'title': header, 'message': message, 'displaytime': int(time),
+                              'image': albumartpath}
                     request = self._sendjson(host, 'GUI.ShowNotification', params)
 
                 if not request:
@@ -311,34 +316,34 @@ class Plex(object):
         self.client_hosts = headphones.CONFIG.PLEX_CLIENT_HOST
         self.username = headphones.CONFIG.PLEX_USERNAME
         self.password = headphones.CONFIG.PLEX_PASSWORD
+        self.token = headphones.CONFIG.PLEX_TOKEN
 
     def _sendhttp(self, host, command):
 
-        username = self.username
+        url = host + '/xbmcCmds/xbmcHttp/?' + command
-        password = self.password
 
-        url_command = urllib.urlencode(command)
+        if self.password:
-
+            response = request.request_response(url, auth=(self.username, self.password))
-        url = host + '/xbmcCmds/xbmcHttp/?' + url_command
+        else:
-
+            response = request.request_response(url)
-        req = urllib2.Request(url)
-
-        if password:
-            base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
-            req.add_header("Authorization", "Basic %s" % base64string)
-
-        logger.info('Plex url: %s' % url)
-
-        try:
-            handle = urllib2.urlopen(req)
-        except Exception as e:
-            logger.warn('Error opening Plex url: %s' % e)
-            return
-
-        response = handle.read().decode(headphones.SYS_ENCODING)
 
         return response
 
+    def _sendjson(self, host, method, params={}):
+        data = [{'id': 0, 'jsonrpc': '2.0', 'method': method, 'params': params}]
+        headers = {'Content-Type': 'application/json'}
+        url = host + '/jsonrpc'
+
+        if self.password:
+            response = request.request_json(url, method="post", data=json.dumps(data),
+                                            headers=headers, auth=(self.username, self.password))
+        else:
+            response = request.request_json(url, method="post", data=json.dumps(data),
+                                            headers=headers)
+
+        if response:
+            return response[0]['result']
+
     def update(self):
 
         # From what I read you can't update the music library on a per directory or per path basis
@@ -349,13 +354,15 @@ class Plex(object):
         for host in hosts:
             logger.info('Sending library update command to Plex Media Server@ ' + host)
             url = "%s/library/sections" % host
-            try:
+            if self.token:
-                xml_sections = minidom.parse(urllib.urlopen(url))
+                params = {'X-Plex-Token': self.token}
-            except IOError, e:
+            else:
-                logger.warn("Error while trying to contact Plex Media Server: %s" % e)
+                params = False
-                return False
+
+            r = request.request_minidom(url, params=params)
+
+            sections = r.getElementsByTagName('Directory')
 
-            sections = xml_sections.getElementsByTagName('Directory')
             if not sections:
                 logger.info(u"Plex Media Server not running on: " + host)
                 return False
@@ -363,11 +370,7 @@ class Plex(object):
             for s in sections:
                 if s.getAttribute('type') == "artist":
                     url = "%s/library/sections/%s/refresh" % (host, s.getAttribute('key'))
-                    try:
+                    request.request_response(url, params=params)
-                        urllib.urlopen(url)
-                    except Exception as e:
-                        logger.warn("Error updating library section for Plex Media Server: %s" % e)
-                        return False
 
     def notify(self, artist, album, albumartpath):
 
@@ -375,20 +378,31 @@ class Plex(object):
 
         header = "Headphones"
         message = "%s - %s added to your library" % (artist, album)
-        time = "3000" # in ms
+        time = "3000"  # in ms
 
         for host in hosts:
-            logger.info('Sending notification command to Plex Media Server @ ' + host)
+            logger.info('Sending notification command to Plex client @ ' + host)
             try:
-                notification = header + "," + message + "," + time + "," + albumartpath
+                version = \
-                notifycommand = {'command': 'ExecBuiltIn', 'parameter': 'Notification(' + notification + ')'}
+                self._sendjson(host, 'Application.GetProperties', {'properties': ['version']})[
-                request = self._sendhttp(host, notifycommand)
+                    'version']['major']
+
+                if version < 12:  # Eden
+                    notification = header + "," + message + "," + time + "," + albumartpath
+                    notifycommand = {'command': 'ExecBuiltIn',
+                                     'parameter': 'Notification(' + notification + ')'}
+                    request = self._sendhttp(host, notifycommand)
+
+                else:  # Frodo
+                    params = {'title': header, 'message': message, 'displaytime': int(time),
+                              'image': albumartpath}
+                    request = self._sendjson(host, 'GUI.ShowNotification', params)
 
                 if not request:
                     raise Exception
 
-            except:
+            except Exception:
-                logger.warn('Error sending notification request to Plex Media Server')
+                logger.error('Error sending notification request to Plex client @ ' + host)
 
 
 class NMA(object):
@@ -430,60 +444,37 @@ class NMA(object):
 
 
 class PUSHBULLET(object):
-
     def __init__(self):
         self.apikey = headphones.CONFIG.PUSHBULLET_APIKEY
         self.deviceid = headphones.CONFIG.PUSHBULLET_DEVICEID
 
-    def conf(self, options):
+    def notify(self, message, status):
-        return cherrypy.config['config'].get('PUSHBULLET', options)
-
-    def notify(self, message, event):
         if not headphones.CONFIG.PUSHBULLET_ENABLED:
             return
 
-        http_handler = HTTPSConnection("api.pushbullet.com")
+        url = "https://api.pushbullet.com/v2/pushes"
 
         data = {'type': "note",
                 'title': "Headphones",
-                'body': message.encode("utf-8")}
+                'body': message + ': ' + status}
 
-        http_handler.request("POST",
+        if self.deviceid:
-                                "/v2/pushes",
+            data['device_iden'] = self.deviceid
-                                headers={'Content-type': "application/json",
-                                            'Authorization': 'Basic %s' % base64.b64encode(headphones.CONFIG.PUSHBULLET_APIKEY + ":")},
-                                body=json.dumps(data))
-        response = http_handler.getresponse()
-        request_status = response.status
-        logger.debug(u"PushBullet response status: %r" % request_status)
-        logger.debug(u"PushBullet response headers: %r" % response.getheaders())
-        logger.debug(u"PushBullet response body: %r" % response.read())
 
-        if request_status == 200:
+        headers = {'Content-type': "application/json",
-                logger.info(u"PushBullet notifications sent.")
+                   'Authorization': 'Bearer ' + headphones.CONFIG.PUSHBULLET_APIKEY}
-                return True
+
-        elif request_status >= 400 and request_status < 500:
+        response = request.request_json(url, method="post", headers=headers, data=json.dumps(data))
-                logger.info(u"PushBullet request failed: %s" % response.reason)
+
-                return False
+        if response:
+            logger.info(u"PushBullet notifications sent.")
+            return True
         else:
-                logger.info(u"PushBullet notification failed serverside.")
+            logger.info(u"PushBullet notification failed.")
-                return False
+            return False
-
-    def updateLibrary(self):
-        #For uniformity reasons not removed
-        return
-
-    def test(self, apikey, deviceid):
-
-        self.enabled = True
-        self.apikey = apikey
-        self.deviceid = deviceid
-
-        self.notify('Main Screen Activate', 'Test Message')
 
 
 class PUSHALOT(object):
-
     def notify(self, message, event):
         if not headphones.CONFIG.PUSHALOT_ENABLED:
             return
@@ -501,9 +492,9 @@ class PUSHALOT(object):
                 'Body': message.encode("utf-8")}
 
         http_handler.request("POST",
-                                "/api/sendmessage",
+                             "/api/sendmessage",
-                                headers={'Content-type': "application/x-www-form-urlencoded"},
+                             headers={'Content-type': "application/x-www-form-urlencoded"},
-                                body=urlencode(data))
+                             body=urlencode(data))
         response = http_handler.getresponse()
         request_status = response.status
 
@@ -512,14 +503,14 @@ class PUSHALOT(object):
         logger.debug(u"Pushalot response body: %r" % response.read())
 
         if request_status == 200:
-                logger.info(u"Pushalot notifications sent.")
+            logger.info(u"Pushalot notifications sent.")
-                return True
+            return True
         elif request_status == 410:
-                logger.info(u"Pushalot auth failed: %s" % response.reason)
+            logger.info(u"Pushalot auth failed: %s" % response.reason)
-                return False
+            return False
         else:
-                logger.info(u"Pushalot notification failed.")
+            logger.info(u"Pushalot notification failed.")
-                return False
+            return False
 
 
 class Synoindex(object):
@@ -533,7 +524,8 @@ class Synoindex(object):
         path = os.path.abspath(path)
 
         if not self.util_exists():
-            logger.warn("Error sending notification: synoindex utility not found at %s" % self.util_loc)
+            logger.warn(
+                "Error sending notification: synoindex utility not found at %s" % self.util_loc)
             return
 
         if os.path.isfile(path):
@@ -541,15 +533,17 @@ class Synoindex(object):
         elif os.path.isdir(path):
             cmd_arg = '-A'
         else:
-            logger.warn("Error sending notification: Path passed to synoindex was not a file or folder.")
+            logger.warn(
+                "Error sending notification: Path passed to synoindex was not a file or folder.")
             return
 
         cmd = [self.util_loc, cmd_arg, path]
         logger.info("Calling synoindex command: %s" % str(cmd))
         try:
-            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, cwd=headphones.PROG_DIR)
+            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                                 cwd=headphones.PROG_DIR)
             out, error = p.communicate()
-            #synoindex never returns any codes other than '0', highly irritating
+            # synoindex never returns any codes other than '0', highly irritating
         except OSError, e:
             logger.warn("Error sending notification: %s" % str(e))
 
@@ -560,7 +554,6 @@ class Synoindex(object):
 
 
 class PUSHOVER(object):
-
     def __init__(self):
         self.enabled = headphones.CONFIG.PUSHOVER_ENABLED
         self.keys = headphones.CONFIG.PUSHOVER_KEYS
@@ -578,7 +571,7 @@ class PUSHOVER(object):
         if not headphones.CONFIG.PUSHOVER_ENABLED:
             return
 
-        http_handler = HTTPSConnection("api.pushover.net")
+        url = "https://api.pushover.net/1/messages.json"
 
         data = {'token': self.application_token,
                 'user': headphones.CONFIG.PUSHOVER_KEYS,
@@ -586,28 +579,19 @@ class PUSHOVER(object):
                 'message': message.encode("utf-8"),
                 'priority': headphones.CONFIG.PUSHOVER_PRIORITY}
 
-        http_handler.request("POST",
+        headers = {'Content-type': "application/x-www-form-urlencoded"}
-                                "/1/messages.json",
-                                headers={'Content-type': "application/x-www-form-urlencoded"},
-                                body=urlencode(data))
-        response = http_handler.getresponse()
-        request_status = response.status
-        logger.debug(u"Pushover response status: %r" % request_status)
-        logger.debug(u"Pushover response headers: %r" % response.getheaders())
-        logger.debug(u"Pushover response body: %r" % response.read())
 
-        if request_status == 200:
+        response = request.request_response(url, method="POST", headers=headers, data=data)
-                logger.info(u"Pushover notifications sent.")
+
-                return True
+        if response:
-        elif request_status >= 400 and request_status < 500:
+            logger.info(u"Pushover notifications sent.")
-                logger.info(u"Pushover request failed: %s" % response.reason)
+            return True
-                return False
         else:
-                logger.info(u"Pushover notification failed.")
+            logger.error(u"Pushover notification failed.")
-                return False
+            return False
 
     def updateLibrary(self):
-        #For uniformity reasons not removed
+        # For uniformity reasons not removed
         return
 
     def test(self, keys, priority):
@@ -619,7 +603,6 @@ class PUSHOVER(object):
 
 
 class TwitterNotifier(object):
-
     REQUEST_TOKEN_URL = 'https://api.twitter.com/oauth/request_token'
     ACCESS_TOKEN_URL = 'https://api.twitter.com/oauth/access_token'
     AUTHORIZATION_URL = 'https://api.twitter.com/oauth/authorize'
@@ -631,14 +614,17 @@ class TwitterNotifier(object):
 
     def notify_snatch(self, title):
         if headphones.CONFIG.TWITTER_ONSNATCH:
-            self._notifyTwitter(common.notifyStrings[common.NOTIFY_SNATCH] + ': ' + title + ' at ' + helpers.now())
+            self._notifyTwitter(
+                common.notifyStrings[common.NOTIFY_SNATCH] + ': ' + title + ' at ' + helpers.now())
 
     def notify_download(self, title):
         if headphones.CONFIG.TWITTER_ENABLED:
-            self._notifyTwitter(common.notifyStrings[common.NOTIFY_DOWNLOAD] + ': ' + title + ' at ' + helpers.now())
+            self._notifyTwitter(common.notifyStrings[
+                                    common.NOTIFY_DOWNLOAD] + ': ' + title + ' at ' + helpers.now())
 
     def test_notify(self):
-        return self._notifyTwitter("This is a test notification from Headphones at " + helpers.now(), force=True)
+        return self._notifyTwitter(
+            "This is a test notification from Headphones at " + helpers.now(), force=True)
 
     def _get_authorization(self):
 
@@ -675,7 +661,8 @@ class TwitterNotifier(object):
         logger.info('oauth_consumer: ' + str(oauth_consumer))
         oauth_client = oauth.Client(oauth_consumer, token)
         logger.info('oauth_client: ' + str(oauth_client))
-        resp, content = oauth_client.request(self.ACCESS_TOKEN_URL, method='POST', body='oauth_verifier=%s' % key)
+        resp, content = oauth_client.request(self.ACCESS_TOKEN_URL, method='POST',
+                                             body='oauth_verifier=%s' % key)
         logger.info('resp, content: ' + str(resp) + ',' + str(content))
 
         access_token = dict(parse_qsl(content))
@@ -683,7 +670,8 @@ class TwitterNotifier(object):
 
         logger.info('resp[status] = ' + str(resp['status']))
         if resp['status'] != '200':
-            logger.info('The request for a token with did not succeed: ' + str(resp['status']), logger.ERROR)
+            logger.info('The request for a token with did not succeed: ' + str(resp['status']),
+                        logger.ERROR)
             return False
         else:
             logger.info('Your Twitter Access Token key: %s' % access_token['oauth_token'])
@@ -721,28 +709,37 @@ class TwitterNotifier(object):
 
 
 class OSX_NOTIFY(object):
-
     def __init__(self):
         try:
             self.objc = __import__("objc")
+            self.AppKit = __import__("AppKit")
         except:
+            logger.warn('OS X Notification: Cannot import objc or AppKit')
             return False
 
     def swizzle(self, cls, SEL, func):
-        old_IMP = cls.instanceMethodForSelector_(SEL)
+        old_IMP = getattr(cls, SEL, None)
+        if old_IMP is None:
+            old_IMP = cls.instanceMethodForSelector_(SEL)
 
         def wrapper(self, *args, **kwargs):
             return func(self, old_IMP, *args, **kwargs)
-        new_IMP = self.objc.selector(wrapper, selector=old_IMP.selector,
-            signature=old_IMP.signature)
-        self.objc.classAddMethod(cls, SEL, new_IMP)
 
-    def notify(self, title, subtitle=None, text=None, sound=True):
+        new_IMP = self.objc.selector(
+            wrapper,
+            selector=old_IMP.selector,
+            signature=old_IMP.signature
+        )
+        self.objc.classAddMethod(cls, SEL.encode(), new_IMP)
+
+    def notify(self, title, subtitle=None, text=None, sound=True, image=None):
 
         try:
-            self.swizzle(self.objc.lookUpClass('NSBundle'),
+            self.swizzle(
-                b'bundleIdentifier',
+                self.objc.lookUpClass('NSBundle'),
-                self.swizzled_bundleIdentifier)
+                'bundleIdentifier',
+                self.swizzled_bundleIdentifier
+            )
 
             NSUserNotification = self.objc.lookUpClass('NSUserNotification')
             NSUserNotificationCenter = self.objc.lookUpClass('NSUserNotificationCenter')
@@ -761,6 +758,10 @@ class OSX_NOTIFY(object):
                 notification.setInformativeText_(text)
             if sound:
                 notification.setSoundName_("NSUserNotificationDefaultSoundName")
+            if image:
+                source_img = self.AppKit.NSImage.alloc().initByReferencingFile_(image)
+                notification.setContentImage_(source_img)
+                # notification.set_identityImage_(source_img)
             notification.setHasActionButton_(False)
 
             notification_center = NSUserNotificationCenter.defaultUserNotificationCenter()
@@ -778,7 +779,6 @@ class OSX_NOTIFY(object):
 
 
 class BOXCAR(object):
-
     def __init__(self):
         self.url = 'https://new.boxcar.io/api/notifications'
 
@@ -792,7 +792,7 @@ class BOXCAR(object):
                 'notification[title]': title.encode('utf-8'),
                 'notification[long_message]': message.encode('utf-8'),
                 'notification[sound]': "done"
-                })
+            })
 
             req = urllib2.Request(self.url)
             handle = urllib2.urlopen(req, data)
@@ -805,7 +805,6 @@ class BOXCAR(object):
 
 
 class SubSonicNotifier(object):
-
     def __init__(self):
         self.host = headphones.CONFIG.SUBSONIC_HOST
         self.username = headphones.CONFIG.SUBSONIC_USERNAME
@@ -821,4 +820,39 @@ class SubSonicNotifier(object):
 
         # Invoke request
         request.request_response(self.host + "musicFolderSettings.view?scanNow",
-            auth=(self.username, self.password))
+                                 auth=(self.username, self.password))
+
+
+class Email(object):
+    def notify(self, subject, message):
+
+        message = MIMEText(message, 'plain', "utf-8")
+        message['Subject'] = subject
+        message['From'] = email.utils.formataddr(('Headphones', headphones.CONFIG.EMAIL_FROM))
+        message['To'] = headphones.CONFIG.EMAIL_TO
+
+        try:
+            if headphones.CONFIG.EMAIL_SSL:
+                mailserver = smtplib.SMTP_SSL(headphones.CONFIG.EMAIL_SMTP_SERVER,
+                                              headphones.CONFIG.EMAIL_SMTP_PORT)
+            else:
+                mailserver = smtplib.SMTP(headphones.CONFIG.EMAIL_SMTP_SERVER,
+                                          headphones.CONFIG.EMAIL_SMTP_PORT)
+
+            if headphones.CONFIG.EMAIL_TLS:
+                mailserver.starttls()
+
+            mailserver.ehlo()
+
+            if headphones.CONFIG.EMAIL_SMTP_USER:
+                mailserver.login(headphones.CONFIG.EMAIL_SMTP_USER,
+                                 headphones.CONFIG.EMAIL_SMTP_PASSWORD)
+
+            mailserver.sendmail(headphones.CONFIG.EMAIL_FROM, headphones.CONFIG.EMAIL_TO,
+                                message.as_string())
+            mailserver.quit()
+            return True
+
+        except Exception, e:
+            logger.warn('Error sending Email: %s' % e)
+            return False

headphones/nzbget.py

@@ -19,43 +19,45 @@
 # along with Sick Beard.  If not, see <http://www.gnu.org/licenses/>.
 
 
-import httplib
-
-import headphones
-
 from base64 import standard_b64encode
+import httplib
 import xmlrpclib
 
+import headphones
 from headphones import logger
 
 
 def sendNZB(nzb):
-
     addToTop = False
-    nzbgetXMLrpc = "%(username)s:%(password)s@%(host)s/xmlrpc"
+    nzbgetXMLrpc = "%(protocol)s://%(username)s:%(password)s@%(host)s/xmlrpc"
 
-    if headphones.CONFIG.NZBGET_HOST is None:
+    if not headphones.CONFIG.NZBGET_HOST:
         logger.error(u"No NZBget host found in configuration. Please configure it.")
         return False
 
     if headphones.CONFIG.NZBGET_HOST.startswith('https://'):
-        nzbgetXMLrpc = 'https://' + nzbgetXMLrpc
+        protocol = 'https'
-        headphones.CONFIG.NZBGET_HOST.replace('https://', '', 1)
+        host = headphones.CONFIG.NZBGET_HOST.replace('https://', '', 1)
     else:
-        nzbgetXMLrpc = 'http://' + nzbgetXMLrpc
+        protocol = 'http'
-        headphones.CONFIG.NZBGET_HOST.replace('http://', '', 1)
+        host = headphones.CONFIG.NZBGET_HOST.replace('http://', '', 1)
 
-    url = nzbgetXMLrpc % {"host": headphones.CONFIG.NZBGET_HOST, "username": headphones.CONFIG.NZBGET_USERNAME, "password": headphones.CONFIG.NZBGET_PASSWORD}
+    url = nzbgetXMLrpc % {"protocol": protocol, "host": host,
+                          "username": headphones.CONFIG.NZBGET_USERNAME,
+                          "password": headphones.CONFIG.NZBGET_PASSWORD}
 
     nzbGetRPC = xmlrpclib.ServerProxy(url)
     try:
-        if nzbGetRPC.writelog("INFO", "headphones connected to drop of %s any moment now." % (nzb.name + ".nzb")):
+        if nzbGetRPC.writelog("INFO", "headphones connected to drop of %s any moment now." % (
+            nzb.name + ".nzb")):
             logger.debug(u"Successfully connected to NZBget")
         else:
-            logger.info(u"Successfully connected to NZBget, but unable to send a message" % (nzb.name + ".nzb"))
+            logger.info(u"Successfully connected to NZBget, but unable to send a message" % (
+            nzb.name + ".nzb"))
 
     except httplib.socket.error:
-        logger.error(u"Please check your NZBget host and port (if it is running). NZBget is not responding to this combination")
+        logger.error(
+            u"Please check your NZBget host and port (if it is running). NZBget is not responding to this combination")
         return False
 
     except xmlrpclib.ProtocolError, e:
@@ -82,7 +84,9 @@ def sendNZB(nzb):
         nzbget_version = int(nzbget_version_str[:nzbget_version_str.find(".")])
         if nzbget_version == 0:
             if nzbcontent64 is not None:
-                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, addToTop, nzbcontent64)
+                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb",
+                                                 headphones.CONFIG.NZBGET_CATEGORY, addToTop,
+                                                 nzbcontent64)
             else:
                 # from headphones.common.providers.generic import GenericProvider
                 # if nzb.resultType == "nzb":
@@ -95,24 +99,35 @@ def sendNZB(nzb):
                 return False
         elif nzbget_version == 12:
             if nzbcontent64 is not None:
-                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb",
+                                                 headphones.CONFIG.NZBGET_CATEGORY,
+                                                 headphones.CONFIG.NZBGET_PRIORITY, False,
                                                  nzbcontent64, False, dupekey, dupescore, "score")
             else:
-                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb",
+                                                    headphones.CONFIG.NZBGET_CATEGORY,
+                                                    headphones.CONFIG.NZBGET_PRIORITY, False,
                                                     nzb.url, False, dupekey, dupescore, "score")
         # v13+ has a new combined append method that accepts both (url and content)
         # also the return value has changed from boolean to integer
         # (Positive number representing NZBID of the queue item. 0 and negative numbers represent error codes.)
         elif nzbget_version >= 13:
-            nzbget_result = True if nzbGetRPC.append(nzb.name + ".nzb", nzbcontent64 if nzbcontent64 is not None else nzb.url,
+            nzbget_result = True if nzbGetRPC.append(nzb.name + ".nzb",
-                                                     headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False, False, dupekey, dupescore,
+                                                     nzbcontent64 if nzbcontent64 is not None else nzb.url,
+                                                     headphones.CONFIG.NZBGET_CATEGORY,
+                                                     headphones.CONFIG.NZBGET_PRIORITY, False,
+                                                     False, dupekey, dupescore,
                                                      "score") > 0 else False
         else:
             if nzbcontent64 is not None:
-                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb",
+                                                 headphones.CONFIG.NZBGET_CATEGORY,
+                                                 headphones.CONFIG.NZBGET_PRIORITY, False,
                                                  nzbcontent64)
             else:
-                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb",
+                                                    headphones.CONFIG.NZBGET_CATEGORY,
+                                                    headphones.CONFIG.NZBGET_PRIORITY, False,
                                                     nzb.url)
 
         if nzbget_result:
@@ -122,5 +137,6 @@ def sendNZB(nzb):
             logger.error(u"NZBget could not add %s to the queue" % (nzb.name + ".nzb"))
             return False
     except:
-        logger.error(u"Connect Error to NZBget: could not add %s to the queue" % (nzb.name + ".nzb"))
+        logger.error(
+            u"Connect Error to NZBget: could not add %s to the queue" % (nzb.name + ".nzb"))
         return False

headphones/request.py

@@ -13,16 +13,20 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import logger
-
 from xml.dom import minidom
-from bs4 import BeautifulSoup
+import collections
 
+import sys
+from bs4 import BeautifulSoup
 import requests
+from headphones import logger
 import feedparser
 import headphones
 import headphones.lock
-import collections
+
+
+# Disable SSL certificate warnings. We have our own handling
+requests.packages.urllib3.disable_warnings()
 
 # Dictionary with last request times, for rate limiting.
 last_requests = collections.defaultdict(int)
@@ -30,10 +34,10 @@ fake_lock = headphones.lock.FakeLock()
 
 
 def request_response(url, method="get", auto_raise=True,
-    whitelist_status_code=None, lock=fake_lock, **kwargs):
+                     whitelist_status_code=None, lock=fake_lock, **kwargs):
     """
-    Convenient wrapper for `requests.get', which will capture the exceptions and
+    Convenient wrapper for `requests.get', which will capture the exceptions
-    log them. On success, the Response object is returned. In case of a
+    and log them. On success, the Response object is returned. In case of a
     exception, None is returned.
 
     Additionally, there is support for rate limiting. To use this feature,
@@ -51,14 +55,23 @@ def request_response(url, method="get", auto_raise=True,
     # pose a security issue!
     kwargs["verify"] = bool(headphones.CONFIG.VERIFY_SSL_CERT)
 
-    # Map method to the request.XXX method. This is a simple hack, but it allows
+    #This fix is put in place for systems with broken SSL (like QNAP)
-    # requests to apply more magic per method. See lib/requests/api.py.
+    if not headphones.CONFIG.VERIFY_SSL_CERT and sys.version_info >= (2, 7, 9):
+        try:
+            import ssl
+            ssl._create_default_https_context = ssl._create_unverified_context
+        except:
+            pass
+
+    # Map method to the request.XXX method. This is a simple hack, but it
+    # allows requests to apply more magic per method. See lib/requests/api.py.
     request_method = getattr(requests, method.lower())
 
     try:
         # Request URL and wait for response
         with lock:
-            logger.debug("Requesting URL via %s method: %s", method.upper(), url)
+            logger.debug(
+                "Requesting URL via %s method: %s", method.upper(), url)
             response = request_method(url, **kwargs)
 
         # If status code != OK, then raise exception, except if the status code
@@ -76,17 +89,26 @@ def request_response(url, method="get", auto_raise=True,
             response.raise_for_status()
 
         return response
-    except requests.exceptions.SSLError:
+    except requests.exceptions.SSLError as e:
-        logger.error("Unable to connect to remote host because of a SSL " \
+        if kwargs["verify"]:
-            "error. It's likely the remote certificate is untrusted by your " \
+            logger.error(
-            "system. This check can be disabled (advanced users only).")
+                "Unable to connect to remote host because of a SSL error. "
+                "It is likely that your system cannot verify the validity"
+                "of the certificate. The remote certificate is either "
+                "self-signed, or the remote server uses SNI. See the wiki for "
+                "more information on this topic.")
+        else:
+            logger.error(
+                "SSL error raised during connection, with certificate "
+                "verification turned off: %s", e)
     except requests.ConnectionError:
         logger.error(
             "Unable to connect to remote host. Check if the remote "
             "host is up and running.")
     except requests.Timeout:
         logger.error(
-            "Request timed out. The remote host did not respond timely.")
+            "Request timed out. The remote host did not respond in a timely "
+            "manner.")
     except requests.HTTPError as e:
         if e.response is not None:
             if e.response.status_code >= 500:
@@ -97,7 +119,8 @@ def request_response(url, method="get", auto_raise=True,
                 # I don't think we will end up here, but for completeness
                 cause = "unknown"
 
-            logger.error("Request raise HTTP error with status code %d (%s).",
+            logger.error(
+                "Request raise HTTP error with status code %d (%s).",
                 e.response.status_code, cause)
 
             # Debug response
@@ -139,8 +162,8 @@ def request_json(url, **kwargs):
     Wrapper for `request_response', which will decode the response as JSON
     object and return the result, if no exceptions are raised.
 
-    As an option, a validator callback can be given, which should return True if
+    As an option, a validator callback can be given, which should return True
-    the result is valid.
+    if the result is valid.
     """
 
     validator = kwargs.pop("validator", None)
@@ -196,7 +219,8 @@ def server_message(response):
     message = None
 
     # First attempt is to 'read' the response as HTML
-    if "text/html" in response.headers.get("content-type"):
+    if response.headers.get("content-type") and \
+                    "text/html" in response.headers.get("content-type"):
         try:
             soup = BeautifulSoup(response.content, "html5lib")
         except Exception:
@@ -205,9 +229,9 @@ def server_message(response):
         # Find body and cleanup common tags to grab content, which probably
         # contains the message.
         message = soup.find("body")
+        elements = ("header", "script", "footer", "nav", "input", "textarea")
 
-        for element in ["header", "script", "footer", "nav", "input",
+        for element in elements:
-            "textarea"]:
 
             for tag in soup.find_all(element):
                 tag.replaceWith("")

headphones/rutracker.py

@@ -0,0 +1,226 @@
+#!/usr/bin/env python
+
+import urllib
+import time
+from urlparse import urlparse
+
+import re
+import requests as requests
+from bs4 import BeautifulSoup
+import headphones
+from headphones import logger
+
+
+class Rutracker(object):
+    def __init__(self):
+        self.session = requests.session()
+        self.timeout = 60
+        self.loggedin = False
+        self.maxsize = 0
+        self.search_referer = 'http://rutracker.org/forum/tracker.php'
+
+    def logged_in(self):
+        return self.loggedin
+
+    def still_logged_in(self, html):
+        if not html or "action=\"http://login.rutracker.org/forum/login.php\">" in html:
+            return False
+        else:
+            return True
+
+    def login(self):
+        """
+        Logs in user
+        """
+
+        loginpage = 'http://login.rutracker.org/forum/login.php'
+        post_params = {
+            'login_username': headphones.CONFIG.RUTRACKER_USER,
+            'login_password': headphones.CONFIG.RUTRACKER_PASSWORD,
+            'login': b'\xc2\xf5\xee\xe4'  # '%C2%F5%EE%E4'
+        }
+
+        logger.info("Attempting to log in to rutracker...")
+
+        try:
+            r = self.session.post(loginpage, data=post_params, timeout=self.timeout)
+            # try again
+            if 'bb_data' not in r.cookies.keys():
+                time.sleep(10)
+                r = self.session.post(loginpage, data=post_params, timeout=self.timeout)
+            if r.status_code != 200:
+                logger.error("rutracker login returned status code %s" % r.status_code)
+                self.loggedin = False
+            else:
+                if 'bb_data' in r.cookies.keys():
+                    self.loggedin = True
+                    logger.info("Successfully logged in to rutracker")
+                else:
+                    logger.error(
+                        "Could not login to rutracker, credentials maybe incorrect, site is down or too many attempts. Try again later")
+                    self.loggedin = False
+            return self.loggedin
+        except Exception as e:
+            logger.error("Unknown error logging in to rutracker: %s" % e)
+            self.loggedin = False
+            return self.loggedin
+
+    def searchurl(self, artist, album, year, format):
+        """
+        Return the search url
+        """
+
+        # Build search url
+        searchterm = ''
+        if artist != 'Various Artists':
+            searchterm = artist
+            searchterm = searchterm + ' '
+        searchterm = searchterm + album
+        searchterm = searchterm + ' '
+        searchterm = searchterm + year
+
+        if format == 'lossless':
+            format = '+lossless'
+            self.maxsize = 10000000000
+        elif format == 'lossless+mp3':
+            format = '+lossless||mp3||aac'
+            self.maxsize = 10000000000
+        else:
+            format = '+mp3||aac'
+            self.maxsize = 300000000
+
+        # sort by size, descending.
+        sort = '&o=7&s=2'
+
+        searchurl = "%s?nm=%s%s%s" % (self.search_referer, urllib.quote(searchterm), format, sort)
+
+        logger.info("Searching rutracker using term: %s", searchterm)
+
+        return searchurl
+
+    def search(self, searchurl):
+        """
+        Parse the search results and return valid torrent list
+        """
+
+        try:
+            headers = {'Referer': self.search_referer}
+            r = self.session.get(url=searchurl, headers=headers, timeout=self.timeout)
+
+            soup = BeautifulSoup(r.content, 'html5lib')
+
+            # Debug
+            # logger.debug (soup.prettify())
+
+            # Check if still logged in
+            if not self.still_logged_in(soup):
+                self.login()
+                r = self.session.get(url=searchurl, timeout=self.timeout)
+                soup = BeautifulSoup(r.content, 'html5lib')
+                if not self.still_logged_in(soup):
+                    logger.error("Error getting rutracker data")
+                    return None
+
+            # Process
+            rulist = []
+            i = soup.find('table', id='tor-tbl')
+            if not i:
+                logger.info("No valid results found from rutracker")
+                return None
+            minimumseeders = int(headphones.CONFIG.NUMBEROFSEEDERS) - 1
+
+            for item in zip(i.find_all(class_='hl-tags'), i.find_all(class_='dl-stub'),
+                            i.find_all(class_='seedmed')):
+                title = item[0].get_text()
+                url = item[1].get('href')
+                size_formatted = item[1].get_text()[:-2]
+                seeds = item[2].get_text()
+                size_parts = size_formatted.split()
+                size = float(size_parts[0])
+
+                if size_parts[1] == 'KB':
+                    size *= 1024
+                if size_parts[1] == 'MB':
+                    size *= 1024 ** 2
+                if size_parts[1] == 'GB':
+                    size *= 1024 ** 3
+                if size_parts[1] == 'TB':
+                    size *= 1024 ** 4
+
+                if size < self.maxsize and minimumseeders < int(seeds):
+                    logger.info('Found %s. Size: %s' % (title, size_formatted))
+                    # Torrent topic page
+                    torrent_id = dict([part.split('=') for part in urlparse(url)[4].split('&')])[
+                        't']
+                    topicurl = 'http://rutracker.org/forum/viewtopic.php?t=' + torrent_id
+                    rulist.append((title, size, topicurl, 'rutracker.org', 'torrent', True))
+                else:
+                    logger.info(
+                        "%s is larger than the maxsize or has too little seeders for this category, skipping. (Size: %i bytes, Seeders: %i)" % (
+                        title, size, int(seeds)))
+
+            if not rulist:
+                logger.info("No valid results found from rutracker")
+
+            return rulist
+
+        except Exception as e:
+            logger.error("An unknown error occurred in the rutracker parser: %s" % e)
+            return None
+
+    def get_torrent_data(self, url):
+        """
+        return the .torrent data
+        """
+
+        torrent_id = dict([part.split('=') for part in urlparse(url)[4].split('&')])['t']
+        downloadurl = 'http://dl.rutracker.org/forum/dl.php?t=' + torrent_id
+        cookie = {'bb_dl': torrent_id}
+        try:
+            headers = {'Referer': url}
+            r = self.session.post(url=downloadurl, cookies=cookie, headers=headers,
+                                  timeout=self.timeout)
+            return r.content
+        except Exception as e:
+            logger.error('Error getting torrent: %s', e)
+            return False
+
+    # TODO get this working in utorrent.py
+    def utorrent_add_file(self, data):
+
+        host = headphones.CONFIG.UTORRENT_HOST
+        if not host.startswith('http'):
+            host = 'http://' + host
+        if host.endswith('/'):
+            host = host[:-1]
+        if host.endswith('/gui'):
+            host = host[:-4]
+
+        base_url = host
+
+        url = base_url + '/gui/'
+        self.session.auth = (
+        headphones.CONFIG.UTORRENT_USERNAME, headphones.CONFIG.UTORRENT_PASSWORD)
+
+        try:
+            r = self.session.get(url + 'token.html')
+        except Exception as e:
+            logger.error('Error getting token: %s', e)
+            return
+
+        if r.status_code == 401:
+            logger.debug('Error reaching utorrent')
+            return
+
+        regex = re.search(r'.+>([^<]+)</div></html>', r.text)
+        if regex is None:
+            logger.debug('Error reading token')
+            return
+
+        self.session.params = {'token': regex.group(1)}
+        files = {'torrent_file': ("", data)}
+
+        try:
+            self.session.post(url, params={'action': 'add-file'}, files=files)
+        except Exception as e:
+            logger.exception('Error adding file to utorrent %s', e)

headphones/sab.py

@@ -13,26 +13,26 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-#####################################
+###################################
-## Stolen from Sick-Beard's sab.py ##
+# Stolen from Sick-Beard's sab.py #
-#####################################
+###################################
 
-import MultipartPostHandler
-import headphones
 import cookielib
-import urllib2
-import httplib
-import urllib
-import ast
 
+import headphones
 from headphones.common import USER_AGENT
-from headphones import logger
+from headphones import logger, helpers, request
-from headphones import helpers
 
 
-def sendNZB(nzb):
+def sab_api_call(request_type=None, params={}, **kwargs):
+    if not headphones.CONFIG.SAB_HOST.startswith('http'):
+        headphones.CONFIG.SAB_HOST = 'http://' + headphones.CONFIG.SAB_HOST
 
-    params = {}
+    if headphones.CONFIG.SAB_HOST.endswith('/'):
+        headphones.CONFIG.SAB_HOST = headphones.CONFIG.SAB_HOST[
+                                     0:len(headphones.CONFIG.SAB_HOST) - 1]
+
+    url = headphones.CONFIG.SAB_HOST + "/" + "api?"
 
     if headphones.CONFIG.SAB_USERNAME:
         params['ma_username'] = headphones.CONFIG.SAB_USERNAME
@@ -40,9 +40,24 @@ def sendNZB(nzb):
         params['ma_password'] = headphones.CONFIG.SAB_PASSWORD
     if headphones.CONFIG.SAB_APIKEY:
         params['apikey'] = headphones.CONFIG.SAB_APIKEY
-    if headphones.CONFIG.SAB_CATEGORY:
+
+    if request_type == 'send_nzb' and headphones.CONFIG.SAB_CATEGORY:
         params['cat'] = headphones.CONFIG.SAB_CATEGORY
 
+    params['output'] = 'json'
+
+    response = request.request_json(url, params=params, **kwargs)
+
+    if not response:
+        logger.error("Error connecting to SABnzbd on url: %s" % headphones.CONFIG.SAB_HOST)
+        return False
+    else:
+        logger.debug("Successfully connected to SABnzbd on url: %s" % headphones.CONFIG.SAB_HOST)
+        return response
+
+
+def sendNZB(nzb):
+    params = {}
     # if it's a normal result we just pass SAB the URL
     if nzb.resultType == "nzb":
         # for newzbin results send the ID to sab specifically
@@ -62,102 +77,42 @@ def sendNZB(nzb):
         # Sanitize the file a bit, since we can only use ascii chars with MultiPartPostHandler
         nzbdata = helpers.latinToAscii(nzb.extraInfo[0])
         params['mode'] = 'addfile'
-        multiPartParams = {"nzbfile": (helpers.latinToAscii(nzb.name) + ".nzb", nzbdata)}
+        files = {"nzbfile": (helpers.latinToAscii(nzb.name) + ".nzb", nzbdata)}
+        headers = {'User-Agent': USER_AGENT}
 
-    if not headphones.CONFIG.SAB_HOST.startswith('http'):
+    logger.info("Attempting to connect to SABnzbd on url: %s" % headphones.CONFIG.SAB_HOST)
-        headphones.CONFIG.SAB_HOST = 'http://' + headphones.CONFIG.SAB_HOST
+    if nzb.resultType == "nzb":
+        response = sab_api_call('send_nzb', params=params)
+    elif nzb.resultType == "nzbdata":
+        cookies = cookielib.CookieJar()
+        response = sab_api_call('send_nzb', params=params, method="post", files=files,
+                                cookies=cookies, headers=headers)
 
-    if headphones.CONFIG.SAB_HOST.endswith('/'):
+    if not response:
-        headphones.CONFIG.SAB_HOST = headphones.CONFIG.SAB_HOST[0:len(headphones.CONFIG.SAB_HOST) - 1]
-
-    url = headphones.CONFIG.SAB_HOST + "/" + "api?" + urllib.urlencode(params)
-
-    try:
-
-        if nzb.resultType == "nzb":
-            f = urllib.urlopen(url)
-        elif nzb.resultType == "nzbdata":
-            cookies = cookielib.CookieJar()
-            opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookies),
-                                          MultipartPostHandler.MultipartPostHandler)
-
-            req = urllib2.Request(url,
-                                  multiPartParams,
-                                  headers={'User-Agent': USER_AGENT})
-
-            f = opener.open(req)
-
-    except (EOFError, IOError) as e:
-        logger.error(u"Unable to connect to SAB with URL: %s" % url)
-        return False
-
-    except httplib.InvalidURL as e:
-        logger.error(u"Invalid SAB host, check your config. Current host: %s" % headphones.CONFIG.SAB_HOST)
-        return False
-
-    except Exception as e:
-        logger.error(u"Error: " + str(e))
-        return False
-
-    if f is None:
         logger.info(u"No data returned from SABnzbd, NZB not sent")
         return False
 
-    try:
+    if response['status']:
-        result = f.readlines()
+        logger.info(u"NZB sent to SABnzbd successfully")
-    except Exception as e:
-        logger.info(u"Error trying to get result from SAB, NZB not sent: ")
-        return False
-
-    if len(result) == 0:
-        logger.info(u"No data returned from SABnzbd, NZB not sent")
-        return False
-
-    sabText = result[0].strip()
-
-    logger.info(u"Result text from SAB: " + sabText)
-
-    if sabText == "ok":
-        logger.info(u"NZB sent to SAB successfully")
         return True
-    elif sabText == "Missing authentication":
-        logger.info(u"Incorrect username/password sent to SAB, NZB not sent")
-        return False
     else:
-        logger.info(u"Unknown failure sending NZB to sab. Return text is: " + sabText)
+        logger.error(u"Error sending NZB to SABnzbd: %s" % response['error'])
         return False
 
 
 def checkConfig():
-
     params = {'mode': 'get_config',
-               'section': 'misc'
+              'section': 'misc',
-               }
+              }
 
-    if headphones.CONFIG.SAB_USERNAME:
+    config_options = sab_api_call(params=params)
-        params['ma_username'] = headphones.CONFIG.SAB_USERNAME
-    if headphones.CONFIG.SAB_PASSWORD:
-        params['ma_password'] = headphones.CONFIG.SAB_PASSWORD
-    if headphones.CONFIG.SAB_APIKEY:
-        params['apikey'] = headphones.CONFIG.SAB_APIKEY
 
-    if not headphones.CONFIG.SAB_HOST.startswith('http'):
+    if not config_options:
-        headphones.CONFIG.SAB_HOST = 'http://' + headphones.CONFIG.SAB_HOST
+        logger.warn(
-
+            "Unable to read SABnzbd config file - cannot determine renaming options (might affect auto & forced post processing)")
-    if headphones.CONFIG.SAB_HOST.endswith('/'):
-        headphones.CONFIG.SAB_HOST = headphones.CONFIG.SAB_HOST[0:len(headphones.CONFIG.SAB_HOST) - 1]
-
-    url = headphones.CONFIG.SAB_HOST + "/" + "api?" + urllib.urlencode(params)
-
-    try:
-        f = urllib.urlopen(url).read()
-    except Exception:
-        logger.warn("Unable to read SABnzbd config file - cannot determine renaming options (might affect auto & forced post processing)")
         return (0, 0)
 
-    config_options = ast.literal_eval(f)
+    replace_spaces = config_options['config']['misc']['replace_spaces']
-
+    replace_dots = config_options['config']['misc']['replace_dots']
-    replace_spaces = config_options['misc']['replace_spaces']
-    replace_dots = config_options['misc']['replace_dots']
 
     return (replace_spaces, replace_dots)

headphones/searcher_rutracker.py

@@ -1,349 +0,0 @@
-#!/usr/bin/env python
-# coding=utf-8
-
-# Headphones rutracker.org search
-# Functions called from searcher.py
-
-from bencode import bencode as bencode, bdecode
-from urlparse import urlparse
-from bs4 import BeautifulSoup
-from tempfile import mkdtemp
-from hashlib import sha1
-
-import headphones
-import requests
-import cookielib
-import urllib2
-import urllib
-import re
-import os
-
-from headphones import db, logger
-
-
-class Rutracker():
-
-    logged_in = False
-
-    # Stores a number of login attempts to prevent recursion.
-    #login_counter = 0
-
-    def __init__(self):
-
-        self.cookiejar = cookielib.CookieJar()
-        self.opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(self.cookiejar))
-        urllib2.install_opener(self.opener)
-
-    def login(self, login, password):
-        """Implements tracker login procedure."""
-
-        self.logged_in = False
-
-        if login is None or password is None:
-            return False
-
-        #self.login_counter += 1
-
-        # No recursion wanted.
-        #if self.login_counter > 1:
-        #    return False
-
-        params = urllib.urlencode({"login_username": login,
-                                   "login_password": password,
-                                   "login": "Вход"})
-
-        try:
-            self.opener.open("http://login.rutracker.org/forum/login.php", params)
-        except Exception:
-            pass
-
-        # Check if we're logged in
-        for cookie in self.cookiejar:
-            if cookie.name == 'bb_data':
-                self.logged_in = True
-
-        return self.logged_in
-
-    def searchurl(self, artist, album, year, format):
-        """
-        Return the search url
-        """
-
-        # Build search url
-        searchterm = ''
-        if artist != 'Various Artists':
-            searchterm = artist
-            searchterm = searchterm + ' '
-        searchterm = searchterm + album
-        searchterm = searchterm + ' '
-        searchterm = searchterm + year
-
-        providerurl = "http://rutracker.org/forum/tracker.php"
-
-        if format == 'lossless':
-            format = '+lossless'
-        elif format == 'lossless+mp3':
-            format = '+lossless||mp3||aac'
-        else:
-            format = '+mp3||aac'
-
-        # sort by size, descending.
-        sort = '&o=7&s=2'
-
-        searchurl = "%s?nm=%s%s%s" % (providerurl, urllib.quote(searchterm), format, sort)
-
-        return searchurl
-
-    def search(self, searchurl, maxsize, minseeders, albumid):
-        """
-        Parse the search results and return valid torrent list
-        """
-
-        titles = []
-        urls = []
-        seeders = []
-        sizes = []
-        torrentlist = []
-        rulist = []
-
-        try:
-
-            page = self.opener.open(searchurl, timeout=60)
-            soup = BeautifulSoup(page.read())
-
-            # Debug
-            #logger.debug (soup.prettify())
-
-            # Title
-            for link in soup.find_all('a', attrs={'class': 'med tLink hl-tags bold'}):
-                title = link.get_text()
-                titles.append(title)
-
-            # Download URL
-            for link in soup.find_all('a', attrs={'class': 'small tr-dl dl-stub'}):
-                url = link.get('href')
-                urls.append(url)
-
-            # Seeders
-            for link in soup.find_all('b', attrs={'class': 'seedmed'}):
-                seeder = link.get_text()
-                seeders.append(seeder)
-
-            # Size
-            for link in soup.find_all('td', attrs={'class': 'row4 small nowrap tor-size'}):
-                size = link.u.string
-                sizes.append(size)
-
-        except:
-            pass
-
-        # Combine lists
-        torrentlist = zip(titles, urls, seeders, sizes)
-
-        # return if nothing found
-        if not torrentlist:
-            return False
-
-        # don't bother checking track counts anymore, let searcher filter instead
-        # leave code in just in case
-        check_track_count = False
-
-        if check_track_count:
-
-            # get headphones track count for album, return if not found
-            myDB = db.DBConnection()
-            tracks = myDB.select('SELECT * from tracks WHERE AlbumID=?', [albumid])
-            hptrackcount = len(tracks)
-
-            if not hptrackcount:
-                logger.info('headphones track info not found, cannot compare to torrent')
-                return False
-
-            # Return all valid entries, ignored, required words now checked in searcher.py
-
-            #unwantedlist = ['promo', 'vinyl', '[lp]', 'songbook', 'tvrip', 'hdtv', 'dvd']
-
-            formatlist = ['ape', 'flac', 'ogg', 'm4a', 'aac', 'mp3', 'wav', 'aif']
-            deluxelist = ['deluxe', 'edition', 'japanese', 'exclusive']
-
-        for torrent in torrentlist:
-
-            returntitle = torrent[0].encode('utf-8')
-            url = torrent[1]
-            seeders = torrent[2]
-            size = torrent[3]
-
-            if int(size) <= maxsize and int(seeders) >= minseeders:
-
-                #Torrent topic page
-                torrent_id = dict([part.split('=') for part in urlparse(url)[4].split('&')])['t']
-                topicurl = 'http://rutracker.org/forum/viewtopic.php?t=' + torrent_id
-
-                # add to list
-                if not check_track_count:
-                    valid = True
-                else:
-
-                    # Check torrent info
-                    self.cookiejar.set_cookie(cookielib.Cookie(version=0, name='bb_dl', value=torrent_id, port=None, port_specified=False, domain='.rutracker.org', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False))
-
-                    # Debug
-                    #for cookie in self.cookiejar:
-                    #    logger.debug ('Cookie: %s' % cookie)
-
-                    try:
-                        page = self.opener.open(url)
-                        torrent = page.read()
-                        if torrent:
-                            decoded = bdecode(torrent)
-                            metainfo = decoded['info']
-                        page.close()
-                    except Exception as e:
-                        logger.error('Error getting torrent: %s' % e)
-                        return False
-
-                    # get torrent track count and check for cue
-                    trackcount = 0
-                    cuecount = 0
-
-                    if 'files' in metainfo: # multi
-                        for pathfile in metainfo['files']:
-                            path = pathfile['path']
-                            for file in path:
-                                if any(file.lower().endswith('.' + x.lower()) for x in formatlist):
-                                    trackcount += 1
-                                if '.cue' in file:
-                                    cuecount += 1
-
-                    title = returntitle.lower()
-                    logger.debug('torrent title: %s' % title)
-                    logger.debug('headphones trackcount: %s' % hptrackcount)
-                    logger.debug('rutracker trackcount: %s' % trackcount)
-
-                    # If torrent track count less than headphones track count, and there's a cue, then attempt to get track count from log(s)
-                    # This is for the case where we have a single .flac/.wav which can be split by cue
-                    # Not great, but shouldn't be doing this too often
-                    totallogcount = 0
-                    if trackcount < hptrackcount and cuecount > 0 and cuecount < hptrackcount:
-                        page = self.opener.open(topicurl, timeout=60)
-                        soup = BeautifulSoup(page.read())
-                        findtoc = soup.find_all(text='TOC of the extracted CD')
-                        if not findtoc:
-                            findtoc = soup.find_all(text='TOC извлечённого CD')
-                        for toc in findtoc:
-                            logcount = 0
-                            for toccontent in toc.find_all_next(text=True):
-                                cut_string = toccontent.split('|')
-                                new_string = cut_string[0].lstrip().rstrip()
-                                if new_string == '1' or new_string == '01':
-                                    logcount = 1
-                                elif logcount > 0:
-                                    if new_string.isdigit():
-                                        logcount += 1
-                                    else:
-                                        break
-                            totallogcount = totallogcount + logcount
-
-                    if totallogcount > 0:
-                        trackcount = totallogcount
-                        logger.debug('rutracker logtrackcount: %s' % totallogcount)
-
-                    # If torrent track count = hp track count then return torrent,
-                    # if greater, check for deluxe/special/foreign editions
-                    # if less, then allow if it's a single track with a cue
-                    valid = False
-
-                    if trackcount == hptrackcount:
-                        valid = True
-                    elif trackcount > hptrackcount:
-                        if any(deluxe in title for deluxe in deluxelist):
-                            valid = True
-
-                # Add to list
-                if valid:
-                    rulist.append((returntitle, size, topicurl))
-                else:
-                    if topicurl:
-                        logger.info(u'<a href="%s">Torrent</a> found with %s tracks but the selected headphones release has %s tracks, skipping for rutracker.org' % (topicurl, trackcount, hptrackcount))
-            else:
-                logger.info('%s is larger than the maxsize or has too little seeders for this category, skipping. (Size: %i bytes, Seeders: %i)' % (returntitle, int(size), int(seeders)))
-
-        return rulist
-
-    def get_torrent(self, url, savelocation=None):
-
-        torrent_id = dict([part.split('=') for part in urlparse(url)[4].split('&')])['t']
-        self.cookiejar.set_cookie(cookielib.Cookie(version=0, name='bb_dl', value=torrent_id, port=None, port_specified=False, domain='.rutracker.org', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False))
-        downloadurl = 'http://dl.rutracker.org/forum/dl.php?t=' + torrent_id
-        torrent_name = torrent_id + '.torrent'
-
-        try:
-            prev = os.umask(headphones.UMASK)
-            page = self.opener.open(downloadurl)
-            torrent = page.read()
-            decoded = bdecode(torrent)
-            metainfo = decoded['info']
-            tor_hash = sha1(bencode(metainfo)).hexdigest()
-            if savelocation:
-                download_path = os.path.join(savelocation, torrent_name)
-            else:
-                tempdir = mkdtemp(suffix='_rutracker_torrents')
-                download_path = os.path.join(tempdir, torrent_name)
-
-            with open(download_path, 'wb') as f:
-                f.write(torrent)
-            os.umask(prev)
-
-            # Add file to utorrent
-            if headphones.CONFIG.TORRENT_DOWNLOADER == 2:
-                self.utorrent_add_file(download_path)
-
-        except Exception as e:
-            logger.error('Error getting torrent: %s', e)
-            return False
-
-        return download_path, tor_hash
-
-    #TODO get this working in utorrent.py
-    def utorrent_add_file(self, filename):
-
-        host = headphones.CONFIG.UTORRENT_HOST
-        if not host.startswith('http'):
-            host = 'http://' + host
-        if host.endswith('/'):
-            host = host[:-1]
-        if host.endswith('/gui'):
-            host = host[:-4]
-
-        base_url = host
-        username = headphones.CONFIG.UTORRENT_USERNAME
-        password = headphones.CONFIG.UTORRENT_PASSWORD
-
-        session = requests.Session()
-        url = base_url + '/gui/'
-        session.auth = (username, password)
-
-        try:
-            r = session.get(url + 'token.html')
-        except Exception:
-            logger.exception('Error getting token')
-            return
-
-        if r.status_code == '401':
-            logger.debug('Error reaching utorrent')
-            return
-
-        regex = re.search(r'.+>([^<]+)</div></html>', r.text)
-        if regex is None:
-            logger.debug('Error reading token')
-            return
-
-        session.params = {'token': regex.group(1)}
-
-        with open(filename, 'rb') as f:
-            try:
-                session.post(url, params={'action': 'add-file'},
-                    files={'torrent_file': f})
-            except Exception:
-                logger.exception('Error adding file to utorrent')
-                return

headphones/torrentfinished.py

@@ -13,13 +13,14 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import db, utorrent, transmission, logger
-
 import threading
+
+from headphones import db, utorrent, transmission, logger
 import headphones
 
 postprocessor_lock = threading.Lock()
 
+
 def checkTorrentFinished():
     """
     Remove Torrent + data if Post Processed and finished Seeding
@@ -41,6 +42,7 @@ def checkTorrentFinished():
                 torrent_removed = utorrent.removeTorrent(hash, True)
 
             if torrent_removed:
-                myDB.action('DELETE from snatched WHERE status = "Seed_Processed" and AlbumID=?', [albumid])
+                myDB.action('DELETE from snatched WHERE status = "Seed_Processed" and AlbumID=?',
+                            [albumid])
 
     logger.info("Checking finished torrents completed")

headphones/transmission.py

@@ -13,14 +13,15 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import logger, request
-
 import time
 import json
 import base64
 import urlparse
+
+from headphones import logger, request
 import headphones
 
+
 # This is just a simple script to send torrents to transmission. The
 # intention is to turn this into a class where we can check the state
 # of the download, set the download dir, etc.
@@ -28,12 +29,15 @@ import headphones
 #       Store torrent id so we can check up on it
 
 
-def addTorrent(link):
+def addTorrent(link, data=None):
     method = 'torrent-add'
 
-    if link.endswith('.torrent'):
+    if link.endswith('.torrent') and not link.startswith('http') or data:
-        with open(link, 'rb') as f:
+        if data:
-            metainfo = str(base64.b64encode(f.read()))
+            metainfo = str(base64.b64encode(data))
+        else:
+            with open(link, 'rb') as f:
+                metainfo = str(base64.b64encode(f.read()))
         arguments = {'metainfo': metainfo, 'download-dir': headphones.CONFIG.DOWNLOAD_TORRENT_DIR}
     else:
         arguments = {'filename': link, 'download-dir': headphones.CONFIG.DOWNLOAD_TORRENT_DIR}
@@ -93,7 +97,6 @@ def setSeedRatio(torrentid, ratio):
 
 
 def removeTorrent(torrentid, remove_data=False):
-
     method = 'torrent-get'
     arguments = {'ids': torrentid, 'fields': ['isFinished', 'name']}
 
@@ -115,7 +118,8 @@ def removeTorrent(torrentid, remove_data=False):
             response = torrentAction(method, arguments)
             return True
         else:
-            logger.info('%s has not finished seeding yet, torrent will not be removed, will try again on next run' % name)
+            logger.info(
+                '%s has not finished seeding yet, torrent will not be removed, will try again on next run' % name)
     except:
         return False
 
@@ -123,7 +127,6 @@ def removeTorrent(torrentid, remove_data=False):
 
 
 def torrentAction(method, arguments):
-
     host = headphones.CONFIG.TRANSMISSION_HOST
     username = headphones.CONFIG.TRANSMISSION_USERNAME
     password = headphones.CONFIG.TRANSMISSION_PASSWORD
@@ -149,7 +152,7 @@ def torrentAction(method, arguments):
     # Retrieve session id
     auth = (username, password) if username and password else None
     response = request.request_response(host, auth=auth,
-        whitelist_status_code=[401, 409])
+                                        whitelist_status_code=[401, 409])
 
     if response is None:
         logger.error("Error gettings Transmission session ID")
@@ -159,7 +162,7 @@ def torrentAction(method, arguments):
     if response.status_code == 401:
         if auth:
             logger.error("Username and/or password not accepted by " \
-                "Transmission")
+                         "Transmission")
         else:
             logger.error("Transmission authorization required")
 
@@ -176,7 +179,7 @@ def torrentAction(method, arguments):
     data = {'method': method, 'arguments': arguments}
 
     response = request.request_json(host, method="POST", data=json.dumps(data),
-        headers=headers, auth=auth)
+                                    headers=headers, auth=auth)
 
     print response
 

headphones/updater.py

@@ -17,10 +17,10 @@ from headphones import logger, db, importer
 
 
 def dbUpdate(forcefull=False):
-
     myDB = db.DBConnection()
 
-    active_artists = myDB.select('SELECT ArtistID, ArtistName from artists WHERE Status="Active" or Status="Loading" order by LastUpdated ASC')
+    active_artists = myDB.select(
+        'SELECT ArtistID, ArtistName from artists WHERE Status="Active" or Status="Loading" order by LastUpdated ASC')
     logger.info('Starting update for %i active artists', len(active_artists))
 
     for artist in active_artists:

headphones/utorrent.py

@@ -14,26 +14,24 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
 import urllib
+import json
+import time
+from collections import namedtuple
 import urllib2
 import urlparse
 import cookielib
-import json
+
 import re
 import os
-import time
-
 import headphones
-
 from headphones import logger
-from collections import namedtuple
 
 
 class utorrentclient(object):
-
     TOKEN_REGEX = "<div id='token' style='display:none;'>([^<>]+)</div>"
     UTSetting = namedtuple("UTSetting", ["name", "int", "str", "access"])
 
-    def __init__(self, base_url=None, username=None, password=None,):
+    def __init__(self, base_url=None, username=None, password=None, ):
 
         host = headphones.CONFIG.UTORRENT_HOST
         if not host.startswith('http'):
@@ -50,7 +48,7 @@ class utorrentclient(object):
         self.password = headphones.CONFIG.UTORRENT_PASSWORD
         self.opener = self._make_opener('uTorrent', self.base_url, self.username, self.password)
         self.token = self._get_token()
-        #TODO refresh token, when necessary
+        # TODO refresh token, when necessary
 
     def _make_opener(self, realm, base_url, username, password):
         """uTorrent API need HTTP Basic Auth and cookie support for token verify."""
@@ -73,6 +71,7 @@ class utorrentclient(object):
         except urllib2.HTTPError as err:
             logger.debug('URL: ' + str(url))
             logger.debug('Error getting Token. uTorrent responded with error: ' + str(err))
+            return
         match = re.search(utorrentclient.TOKEN_REGEX, response.read())
         return match.group(1)
 
@@ -82,7 +81,7 @@ class utorrentclient(object):
         return self._action(params)
 
     def add_url(self, url):
-        #can receive magnet or normal .torrent link
+        # can receive magnet or normal .torrent link
         params = [('action', 'add-url'), ('s', url)]
         return self._action(params)
 
@@ -147,6 +146,10 @@ class utorrentclient(object):
         return self._action(params)
 
     def _action(self, params, body=None, content_type=None):
+
+        if not self.token:
+            return
+
         url = self.base_url + '/gui/' + '?token=' + self.token + '&' + urllib.urlencode(params)
         request = urllib2.Request(url)
 
@@ -176,13 +179,15 @@ def removeTorrent(hash, remove_data=False):
     status, torrentList = uTorrentClient.list()
     torrents = torrentList['torrents']
     for torrent in torrents:
-        if torrent[0].lower() == hash:
+        if torrent[0].upper() == hash.upper():
             if torrent[21] == 'Finished':
                 logger.info('%s has finished seeding, removing torrent and data' % torrent[2])
                 uTorrentClient.remove(hash, remove_data)
                 return True
             else:
-                logger.info('%s has not finished seeding yet, torrent will not be removed, will try again on next run' % torrent[2])
+                logger.info(
+                    '%s has not finished seeding yet, torrent will not be removed, will try again on next run' %
+                    torrent[2])
                 return False
     return False
 
@@ -198,7 +203,6 @@ def setSeedRatio(hash, ratio):
 
 
 def dirTorrent(hash, cacheid=None, return_name=None):
-
     uTorrentClient = utorrentclient()
 
     if not cacheid:
@@ -215,7 +219,7 @@ def dirTorrent(hash, cacheid=None, return_name=None):
     cacheid = torrentList['torrentc']
 
     for torrent in torrents:
-        if torrent[0].upper() == hash:
+        if torrent[0].upper() == hash.upper():
             if not return_name:
                 return torrent[26], cacheid
             else:
@@ -224,18 +228,21 @@ def dirTorrent(hash, cacheid=None, return_name=None):
     return None, None
 
 
-def addTorrent(link, hash):
+def addTorrent(link):
     uTorrentClient = utorrentclient()
+    uTorrentClient.add_url(link)
+
+
+def getFolder(hash):
 
     # Get Active Directory from settings
     active_dir, completed_dir = getSettingsDirectories()
 
     if not active_dir:
-        logger.error('Could not get "Put new downloads in:" directory from uTorrent settings, please ensure it is set')
+        logger.error(
+            'Could not get "Put new downloads in:" directory from uTorrent settings, please ensure it is set')
         return None
 
-    uTorrentClient.add_url(link)
-
     # Get Torrent Folder Name
     torrent_folder, cacheid = dirTorrent(hash)
 
@@ -249,10 +256,8 @@ def addTorrent(link, hash):
 
     if torrent_folder == active_dir or not torrent_folder:
         torrent_folder, cacheid = dirTorrent(hash, cacheid, return_name=True)
-        labelTorrent(hash)
         return torrent_folder
     else:
-        labelTorrent(hash)
         if headphones.SYS_PLATFORM != "win32":
             torrent_folder = torrent_folder.replace('\\', '/')
         return os.path.basename(os.path.normpath(torrent_folder))

headphones/versioncheck.py

@@ -13,18 +13,17 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import re
-import os
 import tarfile
 import platform
-import headphones
 import subprocess
 
+import re
+import os
+import headphones
 from headphones import logger, version, request
 
 
 def runGit(args):
-
     if headphones.CONFIG.GIT_PATH:
         git_locations = ['"' + headphones.CONFIG.GIT_PATH + '"']
     else:
@@ -40,7 +39,8 @@ def runGit(args):
 
         try:
             logger.debug('Trying to execute: "' + cmd + '" with shell in ' + headphones.PROG_DIR)
-            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True, cwd=headphones.PROG_DIR)
+            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True,
+                                 cwd=headphones.PROG_DIR)
             output, err = p.communicate()
             output = output.strip()
 
@@ -62,7 +62,6 @@ def runGit(args):
 
 
 def getVersion():
-
     if version.HEADPHONES_VERSION.startswith('win32build'):
         headphones.INSTALL_TYPE = 'win'
 
@@ -92,7 +91,8 @@ def getVersion():
             branch_name = branch_name
 
             if not branch_name and headphones.CONFIG.GIT_BRANCH:
-                logger.error('Could not retrieve branch name from git. Falling back to %s' % headphones.CONFIG.GIT_BRANCH)
+                logger.error(
+                    'Could not retrieve branch name from git. Falling back to %s' % headphones.CONFIG.GIT_BRANCH)
                 branch_name = headphones.CONFIG.GIT_BRANCH
             if not branch_name:
                 logger.error('Could not retrieve branch name from git. Defaulting to master')
@@ -123,11 +123,13 @@ def checkGithub():
 
     # Get the latest version available from github
     logger.info('Retrieving latest version information from GitHub')
-    url = 'https://api.github.com/repos/%s/headphones/commits/%s' % (headphones.CONFIG.GIT_USER, headphones.CONFIG.GIT_BRANCH)
+    url = 'https://api.github.com/repos/%s/headphones/commits/%s' % (
+    headphones.CONFIG.GIT_USER, headphones.CONFIG.GIT_BRANCH)
     version = request.request_json(url, timeout=20, validator=lambda x: type(x) == dict)
 
     if version is None:
-        logger.warn('Could not get the latest version from GitHub. Are you running a local development version?')
+        logger.warn(
+            'Could not get the latest version from GitHub. Are you running a local development version?')
         return headphones.CURRENT_VERSION
 
     headphones.LATEST_VERSION = version['sha']
@@ -135,7 +137,8 @@ def checkGithub():
 
     # See how many commits behind we are
     if not headphones.CURRENT_VERSION:
-        logger.info('You are running an unknown version of Headphones. Run the updater to identify your version')
+        logger.info(
+            'You are running an unknown version of Headphones. Run the updater to identify your version')
         return headphones.LATEST_VERSION
 
     if headphones.LATEST_VERSION == headphones.CURRENT_VERSION:
@@ -143,8 +146,10 @@ def checkGithub():
         return headphones.LATEST_VERSION
 
     logger.info('Comparing currently installed version with latest GitHub version')
-    url = 'https://api.github.com/repos/%s/headphones/compare/%s...%s' % (headphones.CONFIG.GIT_USER, headphones.LATEST_VERSION, headphones.CURRENT_VERSION)
+    url = 'https://api.github.com/repos/%s/headphones/compare/%s...%s' % (
-    commits = request.request_json(url, timeout=20, whitelist_status_code=404, validator=lambda x: type(x) == dict)
+    headphones.CONFIG.GIT_USER, headphones.LATEST_VERSION, headphones.CURRENT_VERSION)
+    commits = request.request_json(url, timeout=20, whitelist_status_code=404,
+                                   validator=lambda x: type(x) == dict)
 
     if commits is None:
         logger.warn('Could not get commits behind from GitHub.')
@@ -158,7 +163,8 @@ def checkGithub():
         headphones.COMMITS_BEHIND = 0
 
     if headphones.COMMITS_BEHIND > 0:
-        logger.info('New version is available. You are %s commits behind' % headphones.COMMITS_BEHIND)
+        logger.info(
+            'New version is available. You are %s commits behind' % headphones.COMMITS_BEHIND)
     elif headphones.COMMITS_BEHIND == 0:
         logger.info('Headphones is up to date')
 
@@ -185,7 +191,8 @@ def update():
                 logger.info('Output: ' + str(output))
 
     else:
-        tar_download_url = 'https://github.com/%s/headphones/tarball/%s' % (headphones.CONFIG.GIT_USER, headphones.CONFIG.GIT_BRANCH)
+        tar_download_url = 'https://github.com/%s/headphones/tarball/%s' % (
+        headphones.CONFIG.GIT_USER, headphones.CONFIG.GIT_BRANCH)
         update_dir = os.path.join(headphones.PROG_DIR, 'update')
         version_path = os.path.join(headphones.PROG_DIR, 'version.txt')
 
@@ -214,7 +221,8 @@ def update():
         os.remove(tar_download_path)
 
         # Find update dir name
-        update_dir_contents = [x for x in os.listdir(update_dir) if os.path.isdir(os.path.join(update_dir, x))]
+        update_dir_contents = [x for x in os.listdir(update_dir) if
+                               os.path.isdir(os.path.join(update_dir, x))]
         if len(update_dir_contents) != 1:
             logger.error("Invalid update data, update failed: " + str(update_dir_contents))
             return

headphones/webstart.py

@@ -13,18 +13,17 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import os
 import sys
+
+import os
 import cherrypy
 import headphones
-
 from headphones import logger
 from headphones.webserve import WebInterface
 from headphones.helpers import create_https_certificates
 
 
 def initialize(options):
-
     # HTTPS stuff stolen from sickbeard
     enable_https = options['enable_https']
     https_cert = options['https_cert']
@@ -33,15 +32,16 @@ def initialize(options):
     if enable_https:
         # If either the HTTPS certificate or key do not exist, try to make
         # self-signed ones.
-        if not (https_cert and os.path.exists(https_cert)) or not (https_key and os.path.exists(https_key)):
+        if not (https_cert and os.path.exists(https_cert)) or not (
+            https_key and os.path.exists(https_key)):
             if not create_https_certificates(https_cert, https_key):
                 logger.warn("Unable to create certificate and key. Disabling " \
-                    "HTTPS")
+                            "HTTPS")
                 enable_https = False
 
         if not (os.path.exists(https_cert) and os.path.exists(https_key)):
             logger.warn("Disabled HTTPS because of missing certificate and " \
-                "key.")
+                        "key.")
             enable_https = False
 
     options_dict = {
@@ -63,7 +63,7 @@ def initialize(options):
         protocol = "http"
 
     logger.info("Starting Headphones web server on %s://%s:%d/", protocol,
-        options['http_host'], options['http_port'])
+                options['http_host'], options['http_port'])
     cherrypy.config.update(options_dict)
 
     conf = {
@@ -99,7 +99,8 @@ def initialize(options):
     }
 
     if options['http_password']:
-        logger.info("Web server authentication is enabled, username is '%s'", options['http_username'])
+        logger.info("Web server authentication is enabled, username is '%s'",
+                    options['http_username'])
 
         conf['/'].update({
             'tools.auth_basic.on': True,
@@ -118,7 +119,8 @@ def initialize(options):
         cherrypy.process.servers.check_port(str(options['http_host']), options['http_port'])
         cherrypy.server.start()
     except IOError:
-        sys.stderr.write('Failed to start on port: %i. Is something else running?\n' % (options['http_port']))
+        sys.stderr.write(
+            'Failed to start on port: %i. Is something else running?\n' % (options['http_port']))
         sys.exit(1)
 
     cherrypy.server.wait()

init-scripts/init-alt.freebsd

@@ -27,17 +27,17 @@ rcvar=${name}_enable
 
 load_rc_config ${name}
 
-: ${headphones_enable:="NO"}
+: "${headphones_enable:="NO"}"
-: ${headphones_user:="_sabnzbd"}
+: "${headphones_user:="_sabnzbd"}"
-: ${headphones_dir:="/usr/local/headphones"}
+: "${headphones_dir:="/usr/local/headphones"}"
-: ${headphones_chdir:="${headphones_dir}"}
+: "${headphones_chdir:="${headphones_dir}"}"
-: ${headphones_pid:="${headphones_dir}/headphones.pid"}
+: "${headphones_pid:="${headphones_dir}/headphones.pid"}"
-: ${headphones_conf:="${headphones_dir}/config.ini"}
+: "${headphones_conf:="${headphones_dir}/config.ini"}"
 
 WGET="/usr/local/bin/wget" # You need wget for this script to safely shutdown Headphones.
 if [ -e "${headphones_conf}" ]; then
-        HOST=`grep -A64 "\[General\]" "${headphones_conf}"|egrep "^http_host"|perl -wple 's/^http_host = (.*)$/$1/'`
+        HOST=$(grep -A64 "\[General\]" "${headphones_conf}"|egrep "^http_host"|perl -wple 's/^http_host = (.*)$/$1/')
-        PORT=`grep -A64 "\[General\]" "${headphones_conf}"|egrep "^http_port"|perl -wple 's/^http_port = (.*)$/$1/'`
+        PORT=$(grep -A64 "\[General\]" "${headphones_conf}"|egrep "^http_port"|perl -wple 's/^http_port = (.*)$/$1/')
 fi
 
 status_cmd="${name}_status"
@@ -53,15 +53,15 @@ if [ ! -x "${WGET}" ]; then
 fi
 
 # Ensure user is root when running this script.
-if [ `id -u` != "0" ]; then
+if [ "$(id -u)" != "0" ]; then
   echo "Oops, you should be root before running this!"
   exit 1
 fi
 
 verify_headphones_pid() {
     # Make sure the pid corresponds to the Headphones process.
-    pid=`cat ${headphones_pid} 2>/dev/null`
+    pid=$(cat "${headphones_pid}" 2>/dev/null)
-    ps -p ${pid} | grep -q "python ${headphones_dir}/Headphones.py"
+    pgrep -F "${headphones_pid}" -q "python ${headphones_dir}/Headphones.py"
     return $?
 }
 
@@ -73,10 +73,10 @@ headphones_stop() {
     fi
     echo "Stopping $name"
     verify_headphones_pid
-    ${WGET} -O - -q --user=${SBUSR} --password=${SBPWD} "http://${HOST}:${PORT}/shutdown/" >/dev/null
+    ${WGET} -O - -q --user="${SBUSR}" --password="${SBPWD}" "http://${HOST}:${PORT}/shutdown/" >/dev/null
     
     if [ -n "${pid}" ]; then
-      wait_for_pids ${pid}
+      wait_for_pids "${pid}"
       echo "Stopped $name"
     fi
 }

init-scripts/init.freebsd

@@ -28,11 +28,11 @@ rcvar=${name}_enable
 
 load_rc_config ${name}
 
-: ${headphones_enable:="NO"}
+: "${headphones_enable:="NO"}"
-: ${headphones_user:="_sabnzbd"}
+: "${headphones_user:="_sabnzbd"}"
-: ${headphones_dir:="/usr/local/headphones"}
+: "${headphones_dir:="/usr/local/headphones"}"
-: ${headphones_chdir:="${headphones_dir}"}
+: "${headphones_chdir:="${headphones_dir}"}"
-: ${headphones_pid:="${headphones_dir}/headphones.pid"}
+: "${headphones_pid:="${headphones_dir}/headphones.pid"}"
 
 status_cmd="${name}_status"
 stop_cmd="${name}_stop"
@@ -41,15 +41,15 @@ command="/usr/sbin/daemon"
 command_args="-f -p ${headphones_pid} python ${headphones_dir}/Headphones.py ${headphones_flags} --quiet --nolaunch"
 
 # Ensure user is root when running this script.
-if [ `id -u` != "0" ]; then
+if [ "$(id -u)" != "0" ]; then
   echo "Oops, you should be root before running this!"
   exit 1
 fi
 
 verify_headphones_pid() {
     # Make sure the pid corresponds to the Headphones process.
-    pid=`cat ${headphones_pid} 2>/dev/null`
+    pid=$(cat "${headphones_pid}" 2>/dev/null)
-    ps -p ${pid} | grep -q "python ${headphones_dir}/Headphones.py"
+    pgrep -F "${headphones_pid}" -q "python ${headphones_dir}/Headphones.py"
     return $?
 }
 
@@ -58,7 +58,7 @@ headphones_stop() {
     echo "Stopping $name"
     verify_headphones_pid
     if [ -n "${pid}" ]; then
-      wait_for_pids ${pid}
+      wait_for_pids "${pid}"
       echo "Stopped"
     fi
 }

init-scripts/init.osx

@@ -3,16 +3,16 @@
 <plist version="1.0">
 <dict>
   <key>Label</key>
-  <string>com.headphones.headphones</string>
+  <string>headphones</string>
   <key>ProgramArguments</key>
   <array>
+       <!-- Modify these two lines if you need to to reflect your python location and Headphones install location -->
        <string>/usr/bin/python</string>
-       <string>/Applications/Headphones/headphones.py</string>
+       <string>/Applications/Headphones/Headphones.py</string>
-       <string>--quiet</string>
-       <string>--daemon</string>
-       <string>--nolaunch</string>
   </array>
   <key>RunAtLoad</key>
   <true/>
+  <key>KeepAlive</key>
+  <true/>
 </dict>
-</plist>
+</plist>

init-scripts/init.ubuntu

@@ -32,7 +32,6 @@
 ## HP_PIDFILE=      #$PID_FILE, the location of headphones.pid, the default is /var/run/headphones/headphones.pid
 ## PYTHON_BIN=      #$DAEMON, the location of the python binary, the default is /usr/bin/python
 ## HP_OPTS=         #$EXTRA_DAEMON_OPTS, extra cli option for headphones, i.e. " --config=/home/headphones/config.ini"
-## SSD_OPTS=        #$EXTRA_SSD_OPTS, extra start-stop-daemon option like " --group=users"
 ## HP_PORT=         #$PORT_OPTS, hardcoded port for the webserver, overrides value in config.ini
 ##
 ## EXAMPLE if want to run as different user
@@ -101,9 +100,6 @@ load_settings() {
         # Extra daemon option like: HP_OPTS=" --config=/home/headphones/config.ini"
         EXTRA_DAEMON_OPTS=${HP_OPTS-}
 
-        # Extra start-stop-daemon option like START_OPTS=" --group=users"
-        EXTRA_SSD_OPTS=${SSD_OPTS-}
-
         # Hardcoded port to run on, overrides config.ini settings
         [ -n "$HP_PORT" ] && {
             PORT_OPTS=" --port=${HP_PORT} "
@@ -114,7 +110,7 @@ load_settings() {
         SETTINGS_LOADED=TRUE
     fi
 
-    [ -x $DAEMON ] || {
+    [ -x "$DAEMON" ] || {
         log_warning_msg "$DESC: Can't execute daemon, aborting. See $DAEMON";
         return 1;}
 
@@ -125,8 +121,8 @@ load_settings || exit 0
 
 is_running () {
     # returns 1 when running, else 0.
-    if [ -e $PID_FILE ]; then
+    if [ -e "$PID_FILE" ]; then
-      PID=`cat $PID_FILE`
+      PID=$(cat "$PID_FILE")
 
       RET=$?
       [ $RET -gt 1 ] && exit 1 || return $RET
@@ -136,28 +132,28 @@ is_running () {
 }
 
 handle_pid () {
-    PID_PATH=`dirname $PID_FILE`
+    PID_PATH=$(dirname "$PID_FILE")
-    [ -d $PID_PATH ] || mkdir -p $PID_PATH && chown -R $RUN_AS $PID_PATH > /dev/null || {
+    [ -d "$PID_PATH" ] || mkdir -p "$PID_PATH" && chown -R "$RUN_AS" "$PID_PATH" > /dev/null || {
         log_warning_msg "$DESC: Could not create $PID_FILE, See $SETTINGS, aborting.";
         return 1;}
 
-    if [ -e $PID_FILE ]; then
+    if [ -e "$PID_FILE" ]; then
-        PID=`cat $PID_FILE`
+        PID=$(cat "$PID_FILE")
-        if ! kill -0 $PID > /dev/null 2>&1; then
+        if ! kill -0 "$PID" > /dev/null 2>&1; then
             log_warning_msg "Removing stale $PID_FILE"
-            rm $PID_FILE
+            rm "$PID_FILE"
         fi
     fi
 }
 
 handle_datadir () {
-    [ -d $DATA_DIR ] || mkdir -p $DATA_DIR && chown -R $RUN_AS $DATA_DIR > /dev/null || {
+    [ -d "$DATA_DIR" ] || mkdir -p "$DATA_DIR" && chown -R "$RUN_AS" "$DATA_DIR" > /dev/null || {
         log_warning_msg "$DESC: Could not create $DATA_DIR, See $SETTINGS, aborting.";
         return 1;}
 }
 
 handle_updates () {
-    chown -R $RUN_AS $APP_PATH > /dev/null || {
+    chown -R "$RUN_AS" "$APP_PATH" > /dev/null || {
         log_warning_msg "$DESC: $APP_PATH not writable by $RUN_AS for web-updates";
         return 0; }
 }
@@ -168,7 +164,7 @@ start_headphones () {
     handle_updates
     if ! is_running; then
         log_daemon_msg "Starting $DESC"
-        start-stop-daemon -o -d $APP_PATH -c $RUN_AS --start $EXTRA_SSD_OPTS --pidfile $PID_FILE --exec $DAEMON -- $DAEMON_OPTS
+        start-stop-daemon -o -d "$APP_PATH" -c "$RUN_AS" --start "$EXTRA_SSD"_OPTS --pidfile "$PID_FILE" --exec "$DAEMON" -- "$DAEMON_OPTS"
         check_retval
     else
         log_success_msg "$DESC: already running (pid $PID)"
@@ -178,7 +174,7 @@ start_headphones () {
 stop_headphones () {
     if is_running; then
         log_daemon_msg "Stopping $DESC"
-        start-stop-daemon -o --stop --pidfile $PID_FILE --retry 15
+        start-stop-daemon -o --stop --pidfile "$PID_FILE" --retry 15
         check_retval
     else
         log_success_msg "$DESC: not running"

lib/beets/LICENSE

@@ -0,0 +1,21 @@
+The MIT License
+
+Copyright (c) 2010-2014 Adrian Sampson
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

lib/beets/README.rst

@@ -0,0 +1,94 @@
+.. image:: https://travis-ci.org/sampsyo/beets.svg?branch=master
+    :target: https://travis-ci.org/sampsyo/beets
+
+.. image:: http://img.shields.io/coveralls/sampsyo/beets.svg
+    :target: https://coveralls.io/r/sampsyo/beets
+
+.. image:: http://img.shields.io/pypi/v/beets.svg
+    :target: https://pypi.python.org/pypi/beets
+
+Beets is the media library management system for obsessive-compulsive music
+geeks.
+
+The purpose of beets is to get your music collection right once and for all.
+It catalogs your collection, automatically improving its metadata as it goes.
+It then provides a bouquet of tools for manipulating and accessing your music.
+
+Here's an example of beets' brainy tag corrector doing its thing::
+
+  $ beet import ~/music/ladytron
+  Tagging:
+      Ladytron - Witching Hour
+  (Similarity: 98.4%)
+   * Last One Standing      -> The Last One Standing
+   * Beauty                 -> Beauty*2
+   * White Light Generation -> Whitelightgenerator
+   * All the Way            -> All the Way...
+
+Because beets is designed as a library, it can do almost anything you can
+imagine for your music collection. Via `plugins`_, beets becomes a panacea:
+
+- Fetch or calculate all the metadata you could possibly need: `album art`_,
+  `lyrics`_, `genres`_, `tempos`_, `ReplayGain`_ levels, or `acoustic
+  fingerprints`_.
+- Get metadata from `MusicBrainz`_, `Discogs`_, or `Beatport`_. Or guess
+  metadata using songs' filenames or their acoustic fingerprints.
+- `Transcode audio`_ to any format you like.
+- Check your library for `duplicate tracks and albums`_ or for `albums that
+  are missing tracks`_.
+- Clean up crufty tags left behind by other, less-awesome tools.
+- Embed and extract album art from files' metadata.
+- Browse your music library graphically through a Web browser and play it in any
+  browser that supports `HTML5 Audio`_.
+- Analyze music files' metadata from the command line.
+- Listen to your library with a music player that speaks the `MPD`_ protocol
+  and works with a staggering variety of interfaces.
+
+If beets doesn't do what you want yet, `writing your own plugin`_ is
+shockingly simple if you know a little Python.
+
+.. _plugins: http://beets.readthedocs.org/page/plugins/
+.. _MPD: http://www.musicpd.org/
+.. _MusicBrainz music collection: http://musicbrainz.org/doc/Collections/
+.. _writing your own plugin:
+    http://beets.readthedocs.org/page/dev/plugins.html
+.. _HTML5 Audio:
+    http://www.w3.org/TR/html-markup/audio.html
+.. _albums that are missing tracks:
+    http://beets.readthedocs.org/page/plugins/missing.html
+.. _duplicate tracks and albums:
+    http://beets.readthedocs.org/page/plugins/duplicates.html
+.. _Transcode audio:
+    http://beets.readthedocs.org/page/plugins/convert.html
+.. _Beatport: http://www.beatport.com/
+.. _Discogs: http://www.discogs.com/
+.. _acoustic fingerprints:
+    http://beets.readthedocs.org/page/plugins/chroma.html
+.. _ReplayGain: http://beets.readthedocs.org/page/plugins/replaygain.html
+.. _tempos: http://beets.readthedocs.org/page/plugins/echonest.html
+.. _genres: http://beets.readthedocs.org/page/plugins/lastgenre.html
+.. _album art: http://beets.readthedocs.org/page/plugins/fetchart.html
+.. _lyrics: http://beets.readthedocs.org/page/plugins/lyrics.html
+.. _MusicBrainz: http://musicbrainz.org/
+
+Read More
+---------
+
+Learn more about beets at `its Web site`_. Follow `@b33ts`_ on Twitter for
+news and updates.
+
+You can install beets by typing ``pip install beets``. Then check out the
+`Getting Started`_ guide.
+
+.. _its Web site: http://beets.radbox.org/
+.. _Getting Started: http://beets.readthedocs.org/page/guides/main.html
+.. _@b33ts: http://twitter.com/b33ts/
+
+Authors
+-------
+
+Beets is by `Adrian Sampson`_ with a supporting cast of thousands. For help,
+please contact the `mailing list`_.
+
+.. _mailing list: https://groups.google.com/forum/#!forum/beets-users
+.. _Adrian Sampson: http://homes.cs.washington.edu/~asampson/

lib/beets/__init__.py

@@ -1,5 +1,5 @@
 # This file is part of beets.
-# Copyright 2013, Adrian Sampson.
+# Copyright 2014, Adrian Sampson.
 #
 # Permission is hereby granted, free of charge, to any person obtaining
 # a copy of this software and associated documentation files (the
@@ -12,13 +12,14 @@
 # The above copyright notice and this permission notice shall be
 # included in all copies or substantial portions of the Software.
 
-# This particular version has been slightly modified to work with headphones
+# This particular version has been slightly modified to work with Headphones
 # https://github.com/rembo10/headphones
-import os
 
-__version__ = '1.3.4'
+__version__ = '1.3.10-headphones'
 __author__ = 'Adrian Sampson <adrian@radbox.org>'
 
+import os
+
 import beets.library
 from beets.util import confit
 

lib/beets/autotag/__init__.py

@@ -14,135 +14,21 @@
 
 """Facilities for automatically determining files' correct metadata.
 """
-import os
 import logging
-import re
 
-from beets import library, mediafile, config
+from beets import config
-from beets.util import sorted_walk, ancestry, displayable_path
 
 # Parts of external interface.
-from .hooks import AlbumInfo, TrackInfo, AlbumMatch, TrackMatch
+from .hooks import AlbumInfo, TrackInfo, AlbumMatch, TrackMatch  # noqa
-from .match import tag_item, tag_album
+from .match import tag_item, tag_album  # noqa
-from .match import recommendation
+from .match import Recommendation  # noqa
 
 # Global logger.
 log = logging.getLogger('beets')
 
-# Constants for directory walker.
-MULTIDISC_MARKERS = (r'dis[ck]', r'cd')
-MULTIDISC_PAT_FMT = r'^(.*%s[\W_]*)\d'
-
 
 # Additional utilities for the main interface.
 
-def albums_in_dir(path):
-    """Recursively searches the given directory and returns an iterable
-    of (paths, items) where paths is a list of directories and items is
-    a list of Items that is probably an album. Specifically, any folder
-    containing any media files is an album.
-    """
-    collapse_pat = collapse_paths = collapse_items = None
-
-    for root, dirs, files in sorted_walk(path,
-                                         ignore=config['ignore'].as_str_seq(),
-                                         logger=log):
-        # Get a list of items in the directory.
-        items = []
-        for filename in files:
-            try:
-                i = library.Item.from_path(os.path.join(root, filename))
-            except mediafile.FileTypeError:
-                pass
-            except mediafile.UnreadableFileError:
-                log.warn(u'unreadable file: {0}'.format(
-                    displayable_path(filename))
-                )
-            else:
-                items.append(i)
-
-        # If we're currently collapsing the constituent directories in a
-        # multi-disc album, check whether we should continue collapsing
-        # and add the current directory. If so, just add the directory
-        # and move on to the next directory. If not, stop collapsing.
-        if collapse_paths:
-            if (not collapse_pat and collapse_paths[0] in ancestry(root)) or \
-                    (collapse_pat and
-                     collapse_pat.match(os.path.basename(root))):
-                # Still collapsing.
-                collapse_paths.append(root)
-                collapse_items += items
-                continue
-            else:
-                # Collapse finished. Yield the collapsed directory and
-                # proceed to process the current one.
-                if collapse_items:
-                    yield collapse_paths, collapse_items
-                collapse_pat = collapse_paths = collapse_items = None
-
-        # Check whether this directory looks like the *first* directory
-        # in a multi-disc sequence. There are two indicators: the file
-        # is named like part of a multi-disc sequence (e.g., "Title Disc
-        # 1") or it contains no items but only directories that are
-        # named in this way.
-        start_collapsing = False
-        for marker in MULTIDISC_MARKERS:
-            marker_pat = re.compile(MULTIDISC_PAT_FMT % marker, re.I)
-            match = marker_pat.match(os.path.basename(root))
-
-            # Is this directory the root of a nested multi-disc album?
-            if dirs and not items:
-                # Check whether all subdirectories have the same prefix.
-                start_collapsing = True
-                subdir_pat = None
-                for subdir in dirs:
-                    # The first directory dictates the pattern for
-                    # the remaining directories.
-                    if not subdir_pat:
-                        match = marker_pat.match(subdir)
-                        if match:
-                            subdir_pat = re.compile(r'^%s\d' %
-                                re.escape(match.group(1)), re.I)
-                        else:
-                            start_collapsing = False
-                            break
-
-                    # Subsequent directories must match the pattern.
-                    elif not subdir_pat.match(subdir):
-                        start_collapsing = False
-                        break
-
-                # If all subdirectories match, don't check other
-                # markers.
-                if start_collapsing:
-                    break
-
-            # Is this directory the first in a flattened multi-disc album?
-            elif match:
-                start_collapsing = True
-                # Set the current pattern to match directories with the same
-                # prefix as this one, followed by a digit.
-                collapse_pat = re.compile(r'^%s\d' %
-                    re.escape(match.group(1)), re.I)
-                break
-
-        # If either of the above heuristics indicated that this is the
-        # beginning of a multi-disc album, initialize the collapsed
-        # directory and item lists and check the next directory.
-        if start_collapsing:
-            # Start collapsing; continue to the next iteration.
-            collapse_paths = [root]
-            collapse_items = items
-            continue
-
-        # If it's nonempty, yield it.
-        if items:
-            yield [root], items
-
-    # Clear out any unfinished collapse.
-    if collapse_paths and collapse_items:
-        yield collapse_paths, collapse_items
-
 def apply_item_metadata(item, track_info):
     """Set an item's metadata from its matched TrackInfo object.
     """
@@ -156,6 +42,7 @@ def apply_item_metadata(item, track_info):
     # At the moment, the other metadata is left intact (including album
     # and track number). Perhaps these should be emptied?
 
+
 def apply_metadata(album_info, mapping):
     """Set the items' metadata to match an AlbumInfo object using a
     mapping from Items to TrackInfo objects.
@@ -171,8 +58,8 @@ def apply_metadata(album_info, mapping):
 
         # Artist sort and credit names.
         item.artist_sort = track_info.artist_sort or album_info.artist_sort
-        item.artist_credit = track_info.artist_credit or \
+        item.artist_credit = (track_info.artist_credit or
-                             album_info.artist_credit
+                              album_info.artist_credit)
         item.albumartist_sort = album_info.artist_sort
         item.albumartist_credit = album_info.artist_credit
 
@@ -235,7 +122,6 @@ def apply_metadata(album_info, mapping):
                       'language',
                       'country',
                       'albumstatus',
-                      'media',
                       'albumdisambig'):
             value = getattr(album_info, field)
             if value is not None:
@@ -243,5 +129,8 @@ def apply_metadata(album_info, mapping):
         if track_info.disctitle is not None:
             item.disctitle = track_info.disctitle
 
+        if track_info.media is not None:
+            item.media = track_info.media
+
         # Headphones seal of approval
-        item.comments = 'tagged by headphones/beets'
+        item.comments = 'tagged by headphones/beets'

lib/beets/autotag/hooks.py

@@ -116,6 +116,7 @@ class AlbumInfo(object):
             for track in self.tracks:
                 track.decode(codec)
 
+
 class TrackInfo(object):
     """Describes a canonical track present on a release. Appears as part
     of an AlbumInfo's ``tracks`` list. Consists of these data members:
@@ -126,6 +127,7 @@ class TrackInfo(object):
     - ``artist_id``
     - ``length``: float: duration of the track in seconds
     - ``index``: position on the entire release
+    - ``media``: delivery mechanism (Vinyl, etc.)
     - ``medium``: the disc number this track appears on in the album
     - ``medium_index``: the track's position on the disc
     - ``medium_total``: the number of tracks on the item's disc
@@ -140,13 +142,15 @@ class TrackInfo(object):
     def __init__(self, title, track_id, artist=None, artist_id=None,
                  length=None, index=None, medium=None, medium_index=None,
                  medium_total=None, artist_sort=None, disctitle=None,
-                 artist_credit=None, data_source=None, data_url=None):
+                 artist_credit=None, data_source=None, data_url=None,
+                 media=None):
         self.title = title
         self.track_id = track_id
         self.artist = artist
         self.artist_id = artist_id
         self.length = length
         self.index = index
+        self.media = media
         self.medium = medium
         self.medium_index = medium_index
         self.medium_total = medium_total
@@ -162,7 +166,7 @@ class TrackInfo(object):
         to Unicode.
         """
         for fld in ['title', 'artist', 'medium', 'artist_sort', 'disctitle',
-                    'artist_credit']:
+                    'artist_credit', 'media']:
             value = getattr(self, fld)
             if isinstance(value, str):
                 setattr(self, fld, value.decode(codec, 'ignore'))
@@ -187,6 +191,7 @@ SD_REPLACE = [
     (r'&', 'and'),
 ]
 
+
 def _string_dist_basic(str1, str2):
     """Basic edit distance between two strings, ignoring
     non-alphanumeric characters and case. Comparisons are based on a
@@ -201,13 +206,16 @@ def _string_dist_basic(str1, str2):
         return 0.0
     return levenshtein(str1, str2) / float(max(len(str1), len(str2)))
 
+
 def string_dist(str1, str2):
     """Gives an "intuitive" edit distance between two strings. This is
     an edit distance, normalized by the string length, with a number of
     tweaks that reflect intuition about text.
     """
-    if str1 is None and str2 is None: return 0.0
+    if str1 is None and str2 is None:
-    if str1 is None or str2 is None:  return 1.0
+        return 0.0
+    if str1 is None or str2 is None:
+        return 1.0
 
     str1 = str1.lower()
     str2 = str2.lower()
@@ -217,9 +225,9 @@ def string_dist(str1, str2):
     # "something, the".
     for word in SD_END_WORDS:
         if str1.endswith(', %s' % word):
-            str1 = '%s %s' % (word, str1[:-len(word)-2])
+            str1 = '%s %s' % (word, str1[:-len(word) - 2])
         if str2.endswith(', %s' % word):
-            str2 = '%s %s' % (word, str2[:-len(word)-2])
+            str2 = '%s %s' % (word, str2[:-len(word) - 2])
 
     # Perform a couple of basic normalizing substitutions.
     for pat, repl in SD_REPLACE:
@@ -256,6 +264,23 @@ def string_dist(str1, str2):
 
     return base_dist + penalty
 
+
+class LazyClassProperty(object):
+    """A decorator implementing a read-only property that is *lazy* in
+    the sense that the getter is only invoked once. Subsequent accesses
+    through *any* instance use the cached result.
+    """
+    def __init__(self, getter):
+        self.getter = getter
+        self.computed = False
+
+    def __get__(self, obj, owner):
+        if not self.computed:
+            self.value = self.getter(owner)
+            self.computed = True
+        return self.value
+
+
 class Distance(object):
     """Keeps track of multiple distance penalties. Provides a single
     weighted distance for all penalties as well as a weighted distance
@@ -264,11 +289,15 @@ class Distance(object):
     def __init__(self):
         self._penalties = {}
 
+    @LazyClassProperty
+    def _weights(cls):
+        """A dictionary from keys to floating-point weights.
+        """
         weights_view = config['match']['distance_weights']
-        self._weights = {}
+        weights = {}
         for key in weights_view.keys():
-            self._weights[key] = weights_view[key].as_number()
+            weights[key] = weights_view[key].as_number()
-
+        return weights
 
     # Access the components and their aggregates.
 
@@ -313,8 +342,7 @@ class Distance(object):
         # Convert distance into a negative float we can sort items in
         # ascending order (for keys, when the penalty is equal) and
         # still get the items with the biggest distance first.
-        return sorted(list_, key=lambda (key, dist): (0-dist, key))
+        return sorted(list_, key=lambda (key, dist): (0 - dist, key))
-
 
     # Behave like a float.
 
@@ -323,13 +351,13 @@ class Distance(object):
 
     def __float__(self):
         return self.distance
+
     def __sub__(self, other):
         return self.distance - other
 
     def __rsub__(self, other):
         return other - self.distance
 
-
     # Behave like a dict.
 
     def __getitem__(self, key):
@@ -355,11 +383,11 @@ class Distance(object):
         """
         if not isinstance(dist, Distance):
             raise ValueError(
-                    '`dist` must be a Distance object. It is: %r' % dist)
+                '`dist` must be a Distance object, not {0}'.format(type(dist))
+            )
         for key, penalties in dist._penalties.iteritems():
             self._penalties.setdefault(key, []).extend(penalties)
 
-
     # Adding components.
 
     def _eq(self, value1, value2):
@@ -379,7 +407,8 @@ class Distance(object):
         """
         if not 0.0 <= dist <= 1.0:
             raise ValueError(
-                    '`dist` must be between 0.0 and 1.0. It is: %r' % dist)
+                '`dist` must be between 0.0 and 1.0, not {0}'.format(dist)
+            )
         self._penalties.setdefault(key, []).append(dist)
 
     def add_equality(self, key, value, options):
@@ -476,6 +505,7 @@ def album_for_mbid(release_id):
     except mb.MusicBrainzAPIError as exc:
         exc.log(log)
 
+
 def track_for_mbid(recording_id):
     """Get a TrackInfo object for a MusicBrainz recording ID. Return None
     if the ID is not found.
@@ -485,18 +515,21 @@ def track_for_mbid(recording_id):
     except mb.MusicBrainzAPIError as exc:
         exc.log(log)
 
+
 def albums_for_id(album_id):
     """Get a list of albums for an ID."""
     candidates = [album_for_mbid(album_id)]
     candidates.extend(plugins.album_for_id(album_id))
     return filter(None, candidates)
 
+
 def tracks_for_id(track_id):
     """Get a list of tracks for an ID."""
     candidates = [track_for_mbid(track_id)]
     candidates.extend(plugins.track_for_id(track_id))
     return filter(None, candidates)
 
+
 def album_candidates(items, artist, album, va_likely):
     """Search for album matches. ``items`` is a list of Item objects
     that make up the album. ``artist`` and ``album`` are the respective
@@ -525,6 +558,7 @@ def album_candidates(items, artist, album, va_likely):
 
     return out
 
+
 def item_candidates(item, artist, title):
     """Search for item matches. ``item`` is the Item to be matched.
     ``artist`` and ``title`` are strings and either reflect the item or

lib/beets/autotag/match.py

@@ -25,11 +25,8 @@ from munkres import Munkres
 from beets import plugins
 from beets import config
 from beets.util import plurality
-from beets.util.enumeration import enum
 from beets.autotag import hooks
-
+from beets.util.enumeration import OrderedEnum
-# Recommendation enumeration.
-recommendation = enum('none', 'low', 'medium', 'strong', name='recommendation')
 
 # Artist signals that indicate "various artists". These are used at the
 # album level to determine whether a given release is likely a VA
@@ -41,6 +38,18 @@ VA_ARTISTS = (u'', u'various artists', u'various', u'va', u'unknown')
 log = logging.getLogger('beets')
 
 
+# Recommendation enumeration.
+
+class Recommendation(OrderedEnum):
+    """Indicates a qualitative suggestion to the user about what should
+    be done with a given match.
+    """
+    none = 0
+    low = 1
+    medium = 2
+    strong = 3
+
+
 # Primary matching functionality.
 
 def current_metadata(items):
@@ -56,10 +65,10 @@ def current_metadata(items):
     fields = ['artist', 'album', 'albumartist', 'year', 'disctotal',
               'mb_albumid', 'label', 'catalognum', 'country', 'media',
               'albumdisambig']
-    for key in fields:
+    for field in fields:
-        values = [getattr(item, key) for item in items if item]
+        values = [item[field] for item in items if item]
-        likelies[key], freq = plurality(values)
+        likelies[field], freq = plurality(values)
-        consensus[key] = (freq == len(values))
+        consensus[field] = (freq == len(values))
 
     # If there's an album artist consensus, use this for the artist.
     if consensus['albumartist'] and likelies['albumartist']:
@@ -67,6 +76,7 @@ def current_metadata(items):
 
     return likelies, consensus
 
+
 def assign_items(items, tracks):
     """Given a list of Items and a list of TrackInfo objects, find the
     best mapping between them. Returns a mapping from Items to TrackInfo
@@ -93,12 +103,14 @@ def assign_items(items, tracks):
     extra_tracks.sort(key=lambda t: (t.index, t.title))
     return mapping, extra_items, extra_tracks
 
+
 def track_index_changed(item, track_info):
     """Returns True if the item and track info index is different. Tolerates
     per disc and per release numbering.
     """
     return item.track not in (track_info.medium_index, track_info.index)
 
+
 def track_distance(item, track_info, incl_artist=False):
     """Determines the significance of a track metadata change. Returns a
     Distance object. `incl_artist` indicates that a distance component should
@@ -109,7 +121,7 @@ def track_distance(item, track_info, incl_artist=False):
     # Length.
     if track_info.length:
         diff = abs(item.length - track_info.length) - \
-               config['match']['track_length_grace'].as_number()
+            config['match']['track_length_grace'].as_number()
         dist.add_ratio('track_length', diff,
                        config['match']['track_length_max'].as_number())
 
@@ -134,6 +146,7 @@ def track_distance(item, track_info, incl_artist=False):
 
     return dist
 
+
 def distance(items, album_info, mapping):
     """Determines how "significant" an album metadata change would be.
     Returns a Distance object. `album_info` is an AlbumInfo object
@@ -239,6 +252,7 @@ def distance(items, album_info, mapping):
 
     return dist
 
+
 def match_by_id(items):
     """If the items are tagged with a MusicBrainz album ID, returns an
     AlbumInfo object for the corresponding album. Otherwise, returns
@@ -247,16 +261,17 @@ def match_by_id(items):
     # Is there a consensus on the MB album ID?
     albumids = [item.mb_albumid for item in items if item.mb_albumid]
     if not albumids:
-        log.debug('No album IDs found.')
+        log.debug(u'No album IDs found.')
         return None
 
     # If all album IDs are equal, look up the album.
-    if bool(reduce(lambda x,y: x if x==y else (), albumids)):
+    if bool(reduce(lambda x, y: x if x == y else (), albumids)):
         albumid = albumids[0]
-        log.debug('Searching for discovered album ID: ' + albumid)
+        log.debug(u'Searching for discovered album ID: {0}'.format(albumid))
         return hooks.album_for_mbid(albumid)
     else:
-        log.debug('No album ID consensus.')
+        log.debug(u'No album ID consensus.')
+
 
 def _recommendation(results):
     """Given a sorted list of AlbumMatch or TrackMatch objects, return a
@@ -268,26 +283,26 @@ def _recommendation(results):
     """
     if not results:
         # No candidates: no recommendation.
-        return recommendation.none
+        return Recommendation.none
 
     # Basic distance thresholding.
     min_dist = results[0].distance
     if min_dist < config['match']['strong_rec_thresh'].as_number():
         # Strong recommendation level.
-        rec = recommendation.strong
+        rec = Recommendation.strong
     elif min_dist <= config['match']['medium_rec_thresh'].as_number():
         # Medium recommendation level.
-        rec = recommendation.medium
+        rec = Recommendation.medium
     elif len(results) == 1:
         # Only a single candidate.
-        rec = recommendation.low
+        rec = Recommendation.low
     elif results[1].distance - min_dist >= \
             config['match']['rec_gap_thresh'].as_number():
         # Gap between first two candidates is large.
-        rec = recommendation.low
+        rec = Recommendation.low
     else:
         # No conclusion. Return immediately. Can't be downgraded any further.
-        return recommendation.none
+        return Recommendation.none
 
     # Downgrade to the max rec if it is lower than the current rec for an
     # applied penalty.
@@ -299,28 +314,40 @@ def _recommendation(results):
     for key in keys:
         if key in max_rec_view.keys():
             max_rec = max_rec_view[key].as_choice({
-                'strong': recommendation.strong,
+                'strong': Recommendation.strong,
-                'medium': recommendation.medium,
+                'medium': Recommendation.medium,
-                'low': recommendation.low,
+                'low': Recommendation.low,
-                'none': recommendation.none,
+                'none': Recommendation.none,
             })
             rec = min(rec, max_rec)
 
     return rec
 
+
 def _add_candidate(items, results, info):
     """Given a candidate AlbumInfo object, attempt to add the candidate
     to the output dictionary of AlbumMatch objects. This involves
     checking the track count, ordering the items, checking for
     duplicates, and calculating the distance.
     """
-    log.debug('Candidate: %s - %s' % (info.artist, info.album))
+    log.debug(u'Candidate: {0} - {1}'.format(info.artist, info.album))
+
+    # Discard albums with zero tracks.
+    if not info.tracks:
+        log.debug('No tracks.')
+        return
 
     # Don't duplicate.
     if info.album_id in results:
-        log.debug('Duplicate.')
+        log.debug(u'Duplicate.')
         return
 
+    # Discard matches without required tags.
+    for req_tag in config['match']['required'].as_str_seq():
+        if getattr(info, req_tag) is None:
+            log.debug(u'Ignored. Missing required tag: {0}'.format(req_tag))
+            return
+
     # Find mapping between the items and the track info.
     mapping, extra_items, extra_tracks = assign_items(items, info.tracks)
 
@@ -331,30 +358,36 @@ def _add_candidate(items, results, info):
     penalties = [key for _, key in dist]
     for penalty in config['match']['ignored'].as_str_seq():
         if penalty in penalties:
-            log.debug('Ignored. Penalty: %s' % penalty)
+            log.debug(u'Ignored. Penalty: {0}'.format(penalty))
             return
 
-    log.debug('Success. Distance: %f' % dist)
+    log.debug(u'Success. Distance: {0}'.format(dist))
     results[info.album_id] = hooks.AlbumMatch(dist, info, mapping,
                                               extra_items, extra_tracks)
 
+
 def tag_album(items, search_artist=None, search_album=None,
               search_id=None):
-    """Bundles together the functionality used to infer tags for a
+    """Return a tuple of a artist name, an album name, a list of
-    set of items comprised by an album. Returns everything relevant:
+    `AlbumMatch` candidates from the metadata backend, and a
-        - The current artist.
+    `Recommendation`.
-        - The current album.
+
-        - A list of AlbumMatch objects. The candidates are sorted by
+    The artist and album are the most common values of these fields
-          distance (i.e., best match first).
+    among `items`.
-        - A recommendation.
+
-    If search_artist and search_album or search_id are provided, then
+    The `AlbumMatch` objects are generated by searching the metadata
-    they are used as search terms in place of the current metadata.
+    backends. By default, the metadata of the items is used for the
+    search. This can be customized by setting the parameters. The
+    `mapping` field of the album has the matched `items` as keys.
+
+    The recommendation is calculated from the match qualitiy of the
+    candidates.
     """
     # Get current metadata.
     likelies, consensus = current_metadata(items)
     cur_artist = likelies['artist']
     cur_album = likelies['album']
-    log.debug('Tagging %s - %s' % (cur_artist, cur_album))
+    log.debug(u'Tagging {0} - {1}'.format(cur_artist, cur_album))
 
     # The output result (distance, AlbumInfo) tuples (keyed by MB album
     # ID).
@@ -362,7 +395,7 @@ def tag_album(items, search_artist=None, search_album=None,
 
     # Search by explicit ID.
     if search_id is not None:
-        log.debug('Searching for album ID: ' + search_id)
+        log.debug(u'Searching for album ID: {0}'.format(search_id))
         search_cands = hooks.albums_for_id(search_id)
 
     # Use existing metadata or text search.
@@ -372,32 +405,33 @@ def tag_album(items, search_artist=None, search_album=None,
         if id_info:
             _add_candidate(items, candidates, id_info)
             rec = _recommendation(candidates.values())
-            log.debug('Album ID match recommendation is ' + str(rec))
+            log.debug(u'Album ID match recommendation is {0}'.format(str(rec)))
             if candidates and not config['import']['timid']:
                 # If we have a very good MBID match, return immediately.
                 # Otherwise, this match will compete against metadata-based
                 # matches.
-                if rec == recommendation.strong:
+                if rec == Recommendation.strong:
-                    log.debug('ID match.')
+                    log.debug(u'ID match.')
                     return cur_artist, cur_album, candidates.values(), rec
 
         # Search terms.
         if not (search_artist and search_album):
             # No explicit search terms -- use current metadata.
             search_artist, search_album = cur_artist, cur_album
-        log.debug(u'Search terms: %s - %s' % (search_artist, search_album))
+        log.debug(u'Search terms: {0} - {1}'.format(search_artist,
+                                                    search_album))
 
         # Is this album likely to be a "various artist" release?
         va_likely = ((not consensus['artist']) or
-                    (search_artist.lower() in VA_ARTISTS) or
+                     (search_artist.lower() in VA_ARTISTS) or
-                    any(item.comp for item in items))
+                     any(item.comp for item in items))
-        log.debug(u'Album might be VA: %s' % str(va_likely))
+        log.debug(u'Album might be VA: {0}'.format(str(va_likely)))
 
         # Get the results from the data sources.
         search_cands = hooks.album_candidates(items, search_artist,
                                               search_album, va_likely)
 
-    log.debug(u'Evaluating %i candidates.' % len(search_cands))
+    log.debug(u'Evaluating {0} candidates.'.format(len(search_cands)))
     for info in search_cands:
         _add_candidate(items, candidates, info)
 
@@ -406,6 +440,7 @@ def tag_album(items, search_artist=None, search_album=None,
     rec = _recommendation(candidates)
     return cur_artist, cur_album, candidates, rec
 
+
 def tag_item(item, search_artist=None, search_title=None,
              search_id=None):
     """Attempts to find metadata for a single track. Returns a
@@ -421,15 +456,15 @@ def tag_item(item, search_artist=None, search_title=None,
     # First, try matching by MusicBrainz ID.
     trackid = search_id or item.mb_trackid
     if trackid:
-        log.debug('Searching for track ID: ' + trackid)
+        log.debug(u'Searching for track ID: {0}'.format(trackid))
         for track_info in hooks.tracks_for_id(trackid):
             dist = track_distance(item, track_info, incl_artist=True)
             candidates[track_info.track_id] = \
-                    hooks.TrackMatch(dist, track_info)
+                hooks.TrackMatch(dist, track_info)
             # If this is a good match, then don't keep searching.
             rec = _recommendation(candidates.values())
-            if rec == recommendation.strong and not config['import']['timid']:
+            if rec == Recommendation.strong and not config['import']['timid']:
-                log.debug('Track ID match.')
+                log.debug(u'Track ID match.')
                 return candidates.values(), rec
 
     # If we're searching by ID, don't proceed.
@@ -437,12 +472,13 @@ def tag_item(item, search_artist=None, search_title=None,
         if candidates:
             return candidates.values(), rec
         else:
-            return [], recommendation.none
+            return [], Recommendation.none
 
     # Search terms.
     if not (search_artist and search_title):
         search_artist, search_title = item.artist, item.title
-    log.debug(u'Item search terms: %s - %s' % (search_artist, search_title))
+    log.debug(u'Item search terms: {0} - {1}'.format(search_artist,
+                                                     search_title))
 
     # Get and evaluate candidate metadata.
     for track_info in hooks.item_candidates(item, search_artist, search_title):
@@ -450,7 +486,7 @@ def tag_item(item, search_artist=None, search_title=None,
         candidates[track_info.track_id] = hooks.TrackMatch(dist, track_info)
 
     # Sort by distance and return with recommendation.
-    log.debug('Found %i candidates.' % len(candidates))
+    log.debug(u'Found {0} candidates.'.format(len(candidates)))
     candidates = sorted(candidates.itervalues())
     rec = _recommendation(candidates)
     return candidates, rec

lib/beets/autotag/mb.py

@@ -32,6 +32,7 @@ BASE_URL = 'http://musicbrainz.org/'
 musicbrainzngs.set_useragent('beets', beets.__version__,
                              'http://beets.radbox.org/')
 
+
 class MusicBrainzAPIError(util.HumanReadableException):
     """An error while talking to MusicBrainz. The `query` field is the
     parameter to the action and may have any type.
@@ -41,7 +42,7 @@ class MusicBrainzAPIError(util.HumanReadableException):
         super(MusicBrainzAPIError, self).__init__(reason, verb, tb)
 
     def get_message(self):
-        return u'"{0}" in {1} with query {2}'.format(
+        return u'{0} in {1} with query {2}'.format(
             self._reasonstr(), self.verb, repr(self.query)
         )
 
@@ -51,12 +52,15 @@ RELEASE_INCLUDES = ['artists', 'media', 'recordings', 'release-groups',
                     'labels', 'artist-credits', 'aliases']
 TRACK_INCLUDES = ['artists', 'aliases']
 
+
 def track_url(trackid):
     return urljoin(BASE_URL, 'recording/' + trackid)
 
+
 def album_url(albumid):
     return urljoin(BASE_URL, 'release/' + albumid)
 
+
 def configure():
     """Set up the python-musicbrainz-ngs module according to settings
     from the beets configuration. This should be called at startup.
@@ -67,6 +71,7 @@ def configure():
         config['musicbrainz']['ratelimit'].get(int),
     )
 
+
 def _preferred_alias(aliases):
     """Given an list of alias structures for an artist credit, select
     and return the user's preferred alias alias or None if no matching
@@ -81,13 +86,15 @@ def _preferred_alias(aliases):
     # Search configured locales in order.
     for locale in config['import']['languages'].as_str_seq():
         # Find matching primary aliases for this locale.
-        matches = [a for a in aliases if a['locale'] == locale and 'primary' in a]
+        matches = [a for a in aliases
+                   if a['locale'] == locale and 'primary' in a]
         # Skip to the next locale if we have no matches
         if not matches:
             continue
 
         return matches[0]
 
+
 def _flatten_artist_credit(credit):
     """Given a list representing an ``artist-credit`` block, flatten the
     data into a triple of joined artist name strings: canonical, sort, and
@@ -133,6 +140,7 @@ def _flatten_artist_credit(credit):
         ''.join(artist_credit_parts),
     )
 
+
 def track_info(recording, index=None, medium=None, medium_index=None,
                medium_total=None):
     """Translates a MusicBrainz recording result dictionary into a beets
@@ -167,6 +175,7 @@ def track_info(recording, index=None, medium=None, medium_index=None,
     info.decode()
     return info
 
+
 def _set_date_str(info, date_str, original=False):
     """Given a (possibly partial) YYYY-MM-DD string and an AlbumInfo
     object, set the object's release date fields appropriately. If
@@ -186,6 +195,7 @@ def _set_date_str(info, date_str, original=False):
                     key = 'original_' + key
                 setattr(info, key, date_num)
 
+
 def album_info(release):
     """Takes a MusicBrainz release result dictionary and returns a beets
     AlbumInfo object containing the interesting data about that release.
@@ -199,6 +209,7 @@ def album_info(release):
     index = 0
     for medium in release['medium-list']:
         disctitle = medium.get('title')
+        format = medium.get('format')
         for track in medium['track-list']:
             # Basic information from the recording.
             index += 1
@@ -210,6 +221,7 @@ def album_info(release):
                 len(medium['track-list']),
             )
             ti.disctitle = disctitle
+            ti.media = format
 
             # Prefer track data, where present, over recording data.
             if track.get('title'):
@@ -288,6 +300,7 @@ def album_info(release):
     info.decode()
     return info
 
+
 def match_album(artist, album, tracks=None, limit=SEARCH_LIMIT):
     """Searches for a single album ("release" in MusicBrainz parlance)
     and returns an iterator over AlbumInfo objects. May raise a
@@ -297,9 +310,9 @@ def match_album(artist, album, tracks=None, limit=SEARCH_LIMIT):
     optionally, a number of tracks on the album.
     """
     # Build search criteria.
-    criteria = {'release': album.lower()}
+    criteria = {'release': album.lower().strip()}
     if artist is not None:
-        criteria['artist'] = artist.lower()
+        criteria['artist'] = artist.lower().strip()
     else:
         # Various Artists search.
         criteria['arid'] = VARIOUS_ARTISTS_ID
@@ -322,13 +335,14 @@ def match_album(artist, album, tracks=None, limit=SEARCH_LIMIT):
         if albuminfo is not None:
             yield albuminfo
 
+
 def match_track(artist, title, limit=SEARCH_LIMIT):
     """Searches for a single track and returns an iterable of TrackInfo
     objects. May raise a MusicBrainzAPIError.
     """
     criteria = {
-        'artist': artist.lower(),
+        'artist': artist.lower().strip(),
-        'recording': title.lower(),
+        'recording': title.lower().strip(),
     }
 
     if not any(criteria.itervalues()):
@@ -342,6 +356,7 @@ def match_track(artist, title, limit=SEARCH_LIMIT):
     for recording in res['recording-list']:
         yield track_info(recording)
 
+
 def _parse_id(s):
     """Search for a MusicBrainz ID in the given string and return it. If
     no ID can be found, return None.
@@ -351,38 +366,40 @@ def _parse_id(s):
     if match:
         return match.group()
 
-def album_for_id(albumid):
+
+def album_for_id(releaseid):
     """Fetches an album by its MusicBrainz ID and returns an AlbumInfo
     object or None if the album is not found. May raise a
     MusicBrainzAPIError.
     """
-    albumid = _parse_id(albumid)
+    albumid = _parse_id(releaseid)
     if not albumid:
-        log.error('Invalid MBID.')
+        log.debug(u'Invalid MBID ({0}).'.format(releaseid))
         return
     try:
         res = musicbrainzngs.get_release_by_id(albumid,
                                                RELEASE_INCLUDES)
     except musicbrainzngs.ResponseError:
-        log.debug('Album ID match failed.')
+        log.debug(u'Album ID match failed.')
         return None
     except musicbrainzngs.MusicBrainzError as exc:
         raise MusicBrainzAPIError(exc, 'get release by ID', albumid,
                                   traceback.format_exc())
     return album_info(res['release'])
 
-def track_for_id(trackid):
+
+def track_for_id(releaseid):
     """Fetches a track by its MusicBrainz ID. Returns a TrackInfo object
     or None if no track is found. May raise a MusicBrainzAPIError.
     """
-    trackid = _parse_id(trackid)
+    trackid = _parse_id(releaseid)
     if not trackid:
-        log.error('Invalid MBID.')
+        log.debug(u'Invalid MBID ({0}).'.format(releaseid))
         return
     try:
         res = musicbrainzngs.get_recording_by_id(trackid, TRACK_INCLUDES)
     except musicbrainzngs.ResponseError:
-        log.debug('Track ID match failed.')
+        log.debug(u'Track ID match failed.')
         return None
     except musicbrainzngs.MusicBrainzError as exc:
         raise MusicBrainzAPIError(exc, 'get recording by ID', trackid,

lib/beets/config_default.yaml

@@ -5,6 +5,7 @@ import:
     write: yes
     copy: yes
     move: no
+    link: no
     delete: no
     resume: ask
     incremental: no
@@ -20,6 +21,7 @@ import:
     detail: no
     flat: no
     group_albums: no
+    pretend: false
 
 clutter: ["Thumbs.DB", ".DS_Store"]
 ignore: [".*", "*~", "System Volume Information"]
@@ -32,6 +34,7 @@ replace:
     '\s+$': ''
     '^\s+': ''
 path_sep_replace: _
+asciify_paths: false
 art_filename: cover
 max_filename_length: 0
 
@@ -54,6 +57,9 @@ list_format_item: $artist - $album - $title
 list_format_album: $albumartist - $album
 time_format: '%Y-%m-%d %H:%M:%S'
 
+sort_album: albumartist+ album+
+sort_item: artist+ album+ disc+ track+
+
 paths:
     default: $albumartist/$album%aunique{}/$track $title
     singleton: Non-Album/$artist/$title
@@ -98,5 +104,6 @@ match:
         media: []
         original_year: no
     ignored: []
+    required: []
     track_length_grace: 10
     track_length_max: 30

lib/beets/dbcore/__init__.py

@@ -18,3 +18,8 @@ Library.
 from .db import Model, Database
 from .query import Query, FieldQuery, MatchQuery, AndQuery, OrQuery
 from .types import Type
+from .queryparse import query_from_strings
+from .queryparse import sort_from_strings
+from .queryparse import parse_sorted_query
+
+# flake8: noqa

lib/beets/dbcore/db.py

@@ -20,16 +20,62 @@ from collections import defaultdict
 import threading
 import sqlite3
 import contextlib
+import collections
 
 import beets
 from beets.util.functemplate import Template
-from .query import MatchQuery
+from beets.dbcore import types
+from .query import MatchQuery, NullSort, TrueQuery
 
 
+class FormattedMapping(collections.Mapping):
+    """A `dict`-like formatted view of a model.
+
+    The accessor `mapping[key]` returns the formated version of
+    `model[key]` as a unicode string.
+
+    If `for_path` is true, all path separators in the formatted values
+    are replaced.
+    """
+
+    def __init__(self, model, for_path=False):
+        self.for_path = for_path
+        self.model = model
+        self.model_keys = model.keys(True)
+
+    def __getitem__(self, key):
+        if key in self.model_keys:
+            return self._get_formatted(self.model, key)
+        else:
+            raise KeyError(key)
+
+    def __iter__(self):
+        return iter(self.model_keys)
+
+    def __len__(self):
+        return len(self.model_keys)
+
+    def get(self, key, default=None):
+        if default is None:
+            default = self.model._type(key).format(None)
+        return super(FormattedMapping, self).get(key, default)
+
+    def _get_formatted(self, model, key):
+        value = model._type(key).format(model.get(key))
+        if isinstance(value, bytes):
+            value = value.decode('utf8', 'ignore')
+
+        if self.for_path:
+            sep_repl = beets.config['path_sep_replace'].get(unicode)
+            for sep in (os.path.sep, os.path.altsep):
+                if sep:
+                    value = value.replace(sep, sep_repl)
+
+        return value
+
 
 # Abstract base for model classes.
 
-
 class Model(object):
     """An abstract object representing an object in the database. Model
     objects act like dictionaries (i.e., the allow subscript access like
@@ -66,12 +112,7 @@ class Model(object):
 
     _fields = {}
     """A mapping indicating available "fixed" fields on this type. The
-    keys are field names and the values are Type objects.
+    keys are field names and the values are `Type` objects.
-    """
-
-    _bytes_keys = ()
-    """Keys whose values should be stored as raw bytes blobs rather than
-    strings.
     """
 
     _search_fields = ()
@@ -79,6 +120,21 @@ class Model(object):
     terms.
     """
 
+    _types = {}
+    """Optional Types for non-fixed (i.e., flexible and computed) fields.
+    """
+
+    _sorts = {}
+    """Optional named sort criteria. The keys are strings and the values
+    are subclasses of `Sort`.
+    """
+
+    _always_dirty = False
+    """By default, fields only become "dirty" when their value actually
+    changes. Enabling this flag marks fields as dirty even when the new
+    value is the same as the old value (e.g., `o.f = o.f`).
+    """
+
     @classmethod
     def _getters(cls):
         """Return a mapping from field names to getter functions.
@@ -94,7 +150,6 @@ class Model(object):
         # As above: we could consider caching this result.
         raise NotImplementedError()
 
-
     # Basic operation.
 
     def __init__(self, db=None, **values):
@@ -110,6 +165,20 @@ class Model(object):
         self.update(values)
         self.clear_dirty()
 
+    @classmethod
+    def _awaken(cls, db=None, fixed_values={}, flex_values={}):
+        """Create an object with values drawn from the database.
+
+        This is a performance optimization: the checks involved with
+        ordinary construction are bypassed.
+        """
+        obj = cls(db)
+        for key, value in fixed_values.iteritems():
+            obj._values_fixed[key] = cls._type(key).from_sql(value)
+        for key, value in flex_values.iteritems():
+            obj._values_flex[key] = cls._type(key).from_sql(value)
+        return obj
+
     def __repr__(self):
         return '{0}({1})'.format(
             type(self).__name__,
@@ -132,9 +201,17 @@ class Model(object):
         if need_id and not self.id:
             raise ValueError('{0} has no id'.format(type(self).__name__))
 
-
     # Essential field accessors.
 
+    @classmethod
+    def _type(self, key):
+        """Get the type of a field, a `Type` instance.
+
+        If the field has no explicit type, it is given the base `Type`,
+        which does no conversion.
+        """
+        return self._fields.get(key) or self._types.get(key) or types.DEFAULT
+
     def __getitem__(self, key):
         """Get the value for a field. Raise a KeyError if the field is
         not available.
@@ -152,11 +229,19 @@ class Model(object):
     def __setitem__(self, key, value):
         """Assign the value for a field.
         """
-        source = self._values_fixed if key in self._fields \
+        # Choose where to place the value.
-                 else self._values_flex
+        if key in self._fields:
+            source = self._values_fixed
+        else:
+            source = self._values_flex
+
+        # If the field has a type, filter the value.
+        value = self._type(key).normalize(value)
+
+        # Assign value and possibly mark as dirty.
         old_value = source.get(key)
         source[key] = value
-        if old_value != value:
+        if self._always_dirty or old_value != value:
             self._dirty.add(key)
 
     def __delitem__(self, key):
@@ -183,7 +268,6 @@ class Model(object):
         else:
             return base_keys
 
-
     # Act like a dictionary.
 
     def update(self, values):
@@ -219,7 +303,6 @@ class Model(object):
         """
         return iter(self.keys())
 
-
     # Convenient attribute access.
 
     def __getattr__(self, key):
@@ -243,7 +326,6 @@ class Model(object):
         else:
             del self[key]
 
-
     # Database interaction (CRUD methods).
 
     def store(self):
@@ -252,19 +334,15 @@ class Model(object):
         self._check_db()
 
         # Build assignments for query.
-        assignments = ''
+        assignments = []
         subvars = []
         for key in self._fields:
             if key != 'id' and key in self._dirty:
                 self._dirty.remove(key)
-                assignments += key + '=?,'
+                assignments.append(key + '=?')
-                value = self[key]
+                value = self._type(key).to_sql(self[key])
-                # Wrap path strings in buffers so they get stored
-                # "in the raw".
-                if key in self._bytes_keys and isinstance(value, str):
-                    value = buffer(value)
                 subvars.append(value)
-        assignments = assignments[:-1]  # Knock off last ,
+        assignments = ','.join(assignments)
 
         with self._db.transaction() as tx:
             # Main table update.
@@ -302,6 +380,8 @@ class Model(object):
         self._check_db()
         stored_obj = self._db._get(type(self), self.id)
         assert stored_obj is not None, "object {0} not in DB".format(self.id)
+        self._values_fixed = {}
+        self._values_flex = {}
         self.update(dict(stored_obj))
         self.clear_dirty()
 
@@ -344,76 +424,26 @@ class Model(object):
                     self._dirty.add(key)
             self.store()
 
-
     # Formatting and templating.
 
-    @classmethod
+    _formatter = FormattedMapping
-    def _format(cls, key, value, for_path=False):
-        """Format a value as the given field for this model.
-        """
-        # Format the value as a string according to its type, if any.
-        if key in cls._fields:
-            value = cls._fields[key].format(value)
-            # Formatting must result in a string. To deal with
-            # Python2isms, implicitly convert ASCII strings.
-            assert isinstance(value, basestring), \
-                    u'field formatter must produce strings'
-            if isinstance(value, bytes):
-                value = value.decode('utf8', 'ignore')
 
-        elif not isinstance(value, unicode):
+    def formatted(self, for_path=False):
-            # Fallback formatter. Convert to unicode at all cost.
-            if value is None:
-                value = u''
-            elif isinstance(value, basestring):
-                if isinstance(value, bytes):
-                    value = value.decode('utf8', 'ignore')
-            else:
-                value = unicode(value)
-
-        if for_path:
-            sep_repl = beets.config['path_sep_replace'].get(unicode)
-            for sep in (os.path.sep, os.path.altsep):
-                if sep:
-                    value = value.replace(sep, sep_repl)
-
-        return value
-
-    def _get_formatted(self, key, for_path=False):
-        """Get a field value formatted as a string (`unicode` object)
-        for display to the user. If `for_path` is true, then the value
-        will be sanitized for inclusion in a pathname (i.e., path
-        separators will be removed from the value).
-        """
-        return self._format(key, self.get(key), for_path)
-
-    def _formatted_mapping(self, for_path=False):
         """Get a mapping containing all values on this object formatted
-        as human-readable strings.
+        as human-readable unicode strings.
         """
-        # In the future, this could be made "lazy" to avoid computing
+        return self._formatter(self, for_path)
-        # fields unnecessarily.
-        out = {}
-        for key in self.keys(True):
-            out[key] = self._get_formatted(key, for_path)
-        return out
 
     def evaluate_template(self, template, for_path=False):
         """Evaluate a template (a string or a `Template` object) using
         the object's fields. If `for_path` is true, then no new path
         separators will be added to the template.
         """
-        # Build value mapping.
-        mapping = self._formatted_mapping(for_path)
-
-        # Get template functions.
-        funcs = self._template_funcs()
-
         # Perform substitution.
         if isinstance(template, basestring):
             template = Template(template)
-        return template.substitute(mapping, funcs)
+        return template.substitute(self.formatted(for_path),
-
+                                   self._template_funcs())
 
     # Parsing.
 
@@ -424,63 +454,117 @@ class Model(object):
         if not isinstance(string, basestring):
             raise TypeError("_parse() argument must be a string")
 
-        typ = cls._fields.get(key)
+        return cls._type(key).parse(string)
-        if typ:
-            return typ.parse(string)
-        else:
-            # Fall back to unparsed string.
-            return string
-
 
 
 # Database controller and supporting interfaces.
 
-
 class Results(object):
     """An item query result set. Iterating over the collection lazily
     constructs LibModel objects that reflect database rows.
     """
-    def __init__(self, model_class, rows, db, query=None):
+    def __init__(self, model_class, rows, db, query=None, sort=None):
         """Create a result set that will construct objects of type
-        `model_class`, which should be a subclass of `LibModel`, out of
+        `model_class`.
-        the query result mapping in `rows`. The new objects are
+
-        associated with the database `db`. If `query` is provided, it is
+        `model_class` is a subclass of `LibModel` that will be
-        used as a predicate to filter the results for a "slow query" that
+        constructed. `rows` is a query result: a list of mappings. The
-        cannot be evaluated by the database directly.
+        new objects will be associated with the database `db`.
+
+        If `query` is provided, it is used as a predicate to filter the
+        results for a "slow query" that cannot be evaluated by the
+        database directly. If `sort` is provided, it is used to sort the
+        full list of results before returning. This means it is a "slow
+        sort" and all objects must be built before returning the first
+        one.
         """
         self.model_class = model_class
         self.rows = rows
         self.db = db
         self.query = query
+        self.sort = sort
+
+        # We keep a queue of rows we haven't yet consumed for
+        # materialization. We preserve the original total number of
+        # rows.
+        self._rows = rows
+        self._row_count = len(rows)
+
+        # The materialized objects corresponding to rows that have been
+        # consumed.
+        self._objects = []
+
+    def _get_objects(self):
+        """Construct and generate Model objects for they query. The
+        objects are returned in the order emitted from the database; no
+        slow sort is applied.
+
+        For performance, this generator caches materialized objects to
+        avoid constructing them more than once. This way, iterating over
+        a `Results` object a second time should be much faster than the
+        first.
+        """
+        index = 0  # Position in the materialized objects.
+        while index < len(self._objects) or self._rows:
+            # Are there previously-materialized objects to produce?
+            if index < len(self._objects):
+                yield self._objects[index]
+                index += 1
+
+            # Otherwise, we consume another row, materialize its object
+            # and produce it.
+            else:
+                while self._rows:
+                    row = self._rows.pop(0)
+                    obj = self._make_model(row)
+                    # If there is a slow-query predicate, ensurer that the
+                    # object passes it.
+                    if not self.query or self.query.match(obj):
+                        self._objects.append(obj)
+                        index += 1
+                        yield obj
+                        break
 
     def __iter__(self):
-        """Construct Python objects for all rows that pass the query
+        """Construct and generate Model objects for all matching
-        predicate.
+        objects, in sorted order.
         """
-        for row in self.rows:
+        if self.sort:
-            # Get the flexible attributes for the object.
+            # Slow sort. Must build the full list first.
-            with self.db.transaction() as tx:
+            objects = self.sort.sort(list(self._get_objects()))
-                flex_rows = tx.query(
+            return iter(objects)
-                    'SELECT * FROM {0} WHERE entity_id=?'.format(
+
-                        self.model_class._flex_table
+        else:
-                    ),
+            # Objects are pre-sorted (i.e., by the database).
-                    (row['id'],)
+            return self._get_objects()
-                )
+
-            values = dict(row)
+    def _make_model(self, row):
-            values.update(
+        # Get the flexible attributes for the object.
-                dict((row['key'], row['value']) for row in flex_rows)
+        with self.db.transaction() as tx:
+            flex_rows = tx.query(
+                'SELECT * FROM {0} WHERE entity_id=?'.format(
+                    self.model_class._flex_table
+                ),
+                (row['id'],)
             )
 
-            # Construct the Python object and yield it if it passes the
+        cols = dict(row)
-            # predicate.
+        values = dict((k, v) for (k, v) in cols.items()
-            obj = self.model_class(self.db, **values)
+                      if not k[:4] == 'flex')
-            if not self.query or self.query.match(obj):
+        flex_values = dict((row['key'], row['value']) for row in flex_rows)
-                yield obj
+
+        # Construct the Python object
+        obj = self.model_class._awaken(self.db, values, flex_values)
+        return obj
 
     def __len__(self):
         """Get the number of matching objects.
         """
-        if self.query:
+        if not self._rows:
+            # Fully materialized. Just count the objects.
+            return len(self._objects)
+
+        elif self.query:
             # A slow query. Fall back to testing every object.
             count = 0
             for obj in self:
@@ -489,7 +573,7 @@ class Results(object):
 
         else:
             # A fast query. Just count the rows.
-            return len(self.rows)
+            return self._row_count
 
     def __nonzero__(self):
         """Does this result contain any objects?
@@ -500,6 +584,11 @@ class Results(object):
         """Get the nth item in this result set. This is inefficient: all
         items up to n are materialized and thrown away.
         """
+        if not self._rows and not self.sort:
+            # Fully materialized and already in order. Just look up the
+            # object.
+            return self._objects[n]
+
         it = iter(self)
         try:
             for i in range(n):
@@ -604,7 +693,6 @@ class Database(object):
             self._make_table(model_cls._table, model_cls._fields)
             self._make_attribute_table(model_cls._flex_table)
 
-
     # Primitive access control: connections and transactions.
 
     def _connection(self):
@@ -644,7 +732,6 @@ class Database(object):
         """
         return Transaction(self)
 
-
     # Schema setup and migration.
 
     def _make_table(self, table, fields):
@@ -698,27 +785,33 @@ class Database(object):
                     ON {0} (entity_id);
                 """.format(flex_table))
 
-
     # Querying.
 
-    def _fetch(self, model_cls, query, order_by=None):
+    def _fetch(self, model_cls, query=None, sort=None):
         """Fetch the objects of type `model_cls` matching the given
         query. The query may be given as a string, string sequence, a
-        Query object, or None (to fetch everything). If provided,
+        Query object, or None (to fetch everything). `sort` is an
-        `order_by` is a SQLite ORDER BY clause for sorting.
+        `Sort` object.
         """
+        query = query or TrueQuery()  # A null query.
+        sort = sort or NullSort()  # Unsorted.
         where, subvals = query.clause()
+        order_by = sort.order_clause()
 
-        sql = "SELECT * FROM {0} WHERE {1}".format(
+        sql = ("SELECT * FROM {0} WHERE {1} {2}").format(
             model_cls._table,
             where or '1',
+            "ORDER BY {0}".format(order_by) if order_by else '',
         )
-        if order_by:
+
-            sql += " ORDER BY {0}".format(order_by)
         with self.transaction() as tx:
             rows = tx.query(sql, subvals)
 
-        return Results(model_cls, rows, self, None if where else query)
+        return Results(
+            model_cls, rows, self,
+            None if where else query,  # Slow query component.
+            sort if sort.is_slow() else None,  # Slow sort component.
+        )
 
     def _get(self, model_cls, id):
         """Get a Model object by its id or None if the id does not

lib/beets/dbcore/query.py

@@ -15,6 +15,7 @@
 """The Query type hierarchy for DBCore.
 """
 import re
+from operator import attrgetter
 from beets import util
 from datetime import datetime, timedelta
 
@@ -82,6 +83,23 @@ class MatchQuery(FieldQuery):
         return pattern == value
 
 
+class NoneQuery(FieldQuery):
+
+    def __init__(self, field, fast=True):
+        self.field = field
+        self.fast = fast
+
+    def col_clause(self):
+        return self.field + " IS NULL", ()
+
+    @classmethod
+    def match(self, item):
+        try:
+            return item[self.field] is None
+        except KeyError:
+            return True
+
+
 class StringFieldQuery(FieldQuery):
     """A FieldQuery that converts values to strings before matching
     them.
@@ -104,8 +122,11 @@ class StringFieldQuery(FieldQuery):
 class SubstringQuery(StringFieldQuery):
     """A query that matches a substring in a specific item field."""
     def col_clause(self):
-        search = '%' + (self.pattern.replace('\\','\\\\').replace('%','\\%')
+        pattern = (self.pattern
-                            .replace('_','\\_')) + '%'
+                       .replace('\\', '\\\\')
+                       .replace('%', '\\%')
+                       .replace('_', '\\_'))
+        search = '%' + pattern + '%'
         clause = self.field + " like ? escape '\\'"
         subvals = [search]
         return clause, subvals
@@ -200,7 +221,9 @@ class NumericQuery(FieldQuery):
             self.rangemax = self._convert(parts[1])
 
     def match(self, item):
-        value = getattr(item, self.field)
+        if self.field not in item:
+            return False
+        value = item[self.field]
         if isinstance(value, basestring):
             value = self._convert(value)
 
@@ -236,12 +259,16 @@ class CollectionQuery(Query):
         self.subqueries = subqueries
 
     # Act like a sequence.
+
     def __len__(self):
         return len(self.subqueries)
+
     def __getitem__(self, key):
         return self.subqueries[key]
+
     def __iter__(self):
         return iter(self.subqueries)
+
     def __contains__(self, item):
         return item in self.subqueries
 
@@ -334,10 +361,8 @@ class FalseQuery(Query):
         return False
 
 
-
 # Time/date queries.
 
-
 def _to_epoch_time(date):
     """Convert a `datetime` object to an integer number of seconds since
     the (local) Unix epoch.
@@ -393,10 +418,14 @@ class Period(object):
             return None
         ordinal = string.count('-')
         if ordinal >= len(cls.date_formats):
-            raise ValueError('date is not in one of the formats '
+            # Too many components.
-                             + ', '.join(cls.date_formats))
+            return None
         date_format = cls.date_formats[ordinal]
-        date = datetime.strptime(string, date_format)
+        try:
+            date = datetime.strptime(string, date_format)
+        except ValueError:
+            # Parsing failed.
+            return None
         precision = cls.precisions[ordinal]
         return cls(date, precision)
 
@@ -492,3 +521,134 @@ class DateQuery(FieldQuery):
             # Match any date.
             clause = '1'
         return clause, subvals
+
+
+# Sorting.
+
+class Sort(object):
+    """An abstract class representing a sort operation for a query into
+    the item database.
+    """
+
+    def order_clause(self):
+        """Generates a SQL fragment to be used in a ORDER BY clause, or
+        None if no fragment is used (i.e., this is a slow sort).
+        """
+        return None
+
+    def sort(self, items):
+        """Sort the list of objects and return a list.
+        """
+        return sorted(items)
+
+    def is_slow(self):
+        """Indicate whether this query is *slow*, meaning that it cannot
+        be executed in SQL and must be executed in Python.
+        """
+        return False
+
+
+class MultipleSort(Sort):
+    """Sort that encapsulates multiple sub-sorts.
+    """
+
+    def __init__(self, sorts=None):
+        self.sorts = sorts or []
+
+    def add_sort(self, sort):
+        self.sorts.append(sort)
+
+    def _sql_sorts(self):
+        """Return the list of sub-sorts for which we can be (at least
+        partially) fast.
+
+        A contiguous suffix of fast (SQL-capable) sub-sorts are
+        executable in SQL. The remaining, even if they are fast
+        independently, must be executed slowly.
+        """
+        sql_sorts = []
+        for sort in reversed(self.sorts):
+            if not sort.order_clause() is None:
+                sql_sorts.append(sort)
+            else:
+                break
+        sql_sorts.reverse()
+        return sql_sorts
+
+    def order_clause(self):
+        order_strings = []
+        for sort in self._sql_sorts():
+            order = sort.order_clause()
+            order_strings.append(order)
+
+        return ", ".join(order_strings)
+
+    def is_slow(self):
+        for sort in self.sorts:
+            if sort.is_slow():
+                return True
+        return False
+
+    def sort(self, items):
+        slow_sorts = []
+        switch_slow = False
+        for sort in reversed(self.sorts):
+            if switch_slow:
+                slow_sorts.append(sort)
+            elif sort.order_clause() is None:
+                switch_slow = True
+                slow_sorts.append(sort)
+            else:
+                pass
+
+        for sort in slow_sorts:
+            items = sort.sort(items)
+        return items
+
+    def __repr__(self):
+        return u'MultipleSort({0})'.format(repr(self.sorts))
+
+
+class FieldSort(Sort):
+    """An abstract sort criterion that orders by a specific field (of
+    any kind).
+    """
+    def __init__(self, field, ascending=True):
+        self.field = field
+        self.ascending = ascending
+
+    def sort(self, objs):
+        # TODO: Conversion and null-detection here. In Python 3,
+        # comparisons with None fail. We should also support flexible
+        # attributes with different types without falling over.
+        return sorted(objs, key=attrgetter(self.field),
+                      reverse=not self.ascending)
+
+    def __repr__(self):
+        return u'<{0}: {1}{2}>'.format(
+            type(self).__name__,
+            self.field,
+            '+' if self.ascending else '-',
+        )
+
+
+class FixedFieldSort(FieldSort):
+    """Sort object to sort on a fixed field.
+    """
+    def order_clause(self):
+        order = "ASC" if self.ascending else "DESC"
+        return "{0} {1}".format(self.field, order)
+
+
+class SlowFieldSort(FieldSort):
+    """A sort criterion by some model field other than a fixed field:
+    i.e., a computed or flexible field.
+    """
+    def is_slow(self):
+        return True
+
+
+class NullSort(Sort):
+    """No sorting. Leave results unsorted."""
+    def sort(items):
+        return items

lib/beets/dbcore/queryparse.py

@@ -0,0 +1,180 @@
+# This file is part of beets.
+# Copyright 2014, Adrian Sampson.
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+
+"""Parsing of strings into DBCore queries.
+"""
+import re
+import itertools
+from . import query
+
+
+PARSE_QUERY_PART_REGEX = re.compile(
+    # Non-capturing optional segment for the keyword.
+    r'(?:'
+    r'(\S+?)'    # The field key.
+    r'(?<!\\):'  # Unescaped :
+    r')?'
+
+    r'(.*)',         # The term itself.
+
+    re.I  # Case-insensitive.
+)
+
+
+def parse_query_part(part, query_classes={}, prefixes={},
+                     default_class=query.SubstringQuery):
+    """Take a query in the form of a key/value pair separated by a
+    colon and return a tuple of `(key, value, cls)`. `key` may be None,
+    indicating that any field may be matched. `cls` is a subclass of
+    `FieldQuery`.
+
+    The optional `query_classes` parameter maps field names to default
+    query types; `default_class` is the fallback. `prefixes` is a map
+    from query prefix markers and query types. Prefix-indicated queries
+    take precedence over type-based queries.
+
+    To determine the query class, two factors are used: prefixes and
+    field types. For example, the colon prefix denotes a regular
+    expression query and a type map might provide a special kind of
+    query for numeric values. If neither a prefix nor a specific query
+    class is available, `default_class` is used.
+
+    For instance,
+    'stapler' -> (None, 'stapler', SubstringQuery)
+    'color:red' -> ('color', 'red', SubstringQuery)
+    ':^Quiet' -> (None, '^Quiet', RegexpQuery)
+    'color::b..e' -> ('color', 'b..e', RegexpQuery)
+
+    Prefixes may be "escaped" with a backslash to disable the keying
+    behavior.
+    """
+    part = part.strip()
+    match = PARSE_QUERY_PART_REGEX.match(part)
+
+    assert match  # Regex should always match.
+    key = match.group(1)
+    term = match.group(2).replace('\:', ':')
+
+    # Match the search term against the list of prefixes.
+    for pre, query_class in prefixes.items():
+        if term.startswith(pre):
+            return key, term[len(pre):], query_class
+
+    # No matching prefix: use type-based or fallback/default query.
+    query_class = query_classes.get(key, default_class)
+    return key, term, query_class
+
+
+def construct_query_part(model_cls, prefixes, query_part):
+    """Create a query from a single query component, `query_part`, for
+    querying instances of `model_cls`. Return a `Query` instance.
+    """
+    # Shortcut for empty query parts.
+    if not query_part:
+        return query.TrueQuery()
+
+    # Get the query classes for each possible field.
+    query_classes = {}
+    for k, t in itertools.chain(model_cls._fields.items(),
+                                model_cls._types.items()):
+        query_classes[k] = t.query
+
+    # Parse the string.
+    key, pattern, query_class = \
+        parse_query_part(query_part, query_classes, prefixes)
+
+    # No key specified.
+    if key is None:
+        if issubclass(query_class, query.FieldQuery):
+            # The query type matches a specific field, but none was
+            # specified. So we use a version of the query that matches
+            # any field.
+            return query.AnyFieldQuery(pattern, model_cls._search_fields,
+                                       query_class)
+        else:
+            # Other query type.
+            return query_class(pattern)
+
+    key = key.lower()
+    return query_class(key.lower(), pattern, key in model_cls._fields)
+
+
+def query_from_strings(query_cls, model_cls, prefixes, query_parts):
+    """Creates a collection query of type `query_cls` from a list of
+    strings in the format used by parse_query_part. `model_cls`
+    determines how queries are constructed from strings.
+    """
+    subqueries = []
+    for part in query_parts:
+        subqueries.append(construct_query_part(model_cls, prefixes, part))
+    if not subqueries:  # No terms in query.
+        subqueries = [query.TrueQuery()]
+    return query_cls(subqueries)
+
+
+def construct_sort_part(model_cls, part):
+    """Create a `Sort` from a single string criterion.
+
+    `model_cls` is the `Model` being queried. `part` is a single string
+    ending in ``+`` or ``-`` indicating the sort.
+    """
+    assert part, "part must be a field name and + or -"
+    field = part[:-1]
+    assert field, "field is missing"
+    direction = part[-1]
+    assert direction in ('+', '-'), "part must end with + or -"
+    is_ascending = direction == '+'
+
+    if field in model_cls._sorts:
+        sort = model_cls._sorts[field](model_cls, is_ascending)
+    elif field in model_cls._fields:
+        sort = query.FixedFieldSort(field, is_ascending)
+    else:
+        # Flexible or computed.
+        sort = query.SlowFieldSort(field, is_ascending)
+    return sort
+
+
+def sort_from_strings(model_cls, sort_parts):
+    """Create a `Sort` from a list of sort criteria (strings).
+    """
+    if not sort_parts:
+        return query.NullSort()
+    else:
+        sort = query.MultipleSort()
+        for part in sort_parts:
+            sort.add_sort(construct_sort_part(model_cls, part))
+        return sort
+
+
+def parse_sorted_query(model_cls, parts, prefixes={},
+                       query_cls=query.AndQuery):
+    """Given a list of strings, create the `Query` and `Sort` that they
+    represent.
+    """
+    # Separate query token and sort token.
+    query_parts = []
+    sort_parts = []
+    for part in parts:
+        if part.endswith((u'+', u'-')) and u':' not in part:
+            sort_parts.append(part)
+        else:
+            query_parts.append(part)
+
+    # Parse each.
+    q = query_from_strings(
+        query_cls, model_cls, prefixes, query_parts
+    )
+    s = sort_from_strings(model_cls, sort_parts)
+    return q, s

lib/beets/dbcore/types.py

@@ -18,55 +18,111 @@ from . import query
 from beets.util import str2bool
 
 
-
 # Abstract base.
 
-
 class Type(object):
     """An object encapsulating the type of a model field. Includes
-    information about how to store the value in the database, query,
+    information about how to store, query, format, and parse a given
-    format, and parse a given field.
+    field.
     """
 
-    sql = None
+    sql = u'TEXT'
     """The SQLite column type for the value.
     """
 
-    query = None
+    query = query.SubstringQuery
     """The `Query` subclass to be used when querying the field.
     """
 
+    model_type = unicode
+    """The Python type that is used to represent the value in the model.
+
+    The model is guaranteed to return a value of this type if the field
+    is accessed.  To this end, the constructor is used by the `normalize`
+    and `from_sql` methods and the `default` property.
+    """
+
+    @property
+    def null(self):
+        """The value to be exposed when the underlying value is None.
+        """
+        return self.model_type()
+
     def format(self, value):
         """Given a value of this type, produce a Unicode string
         representing the value. This is used in template evaluation.
         """
-        raise NotImplementedError()
+        if value is None:
+            value = self.null
+        # `self.null` might be `None`
+        if value is None:
+            value = u''
+        if isinstance(value, bytes):
+            value = value.decode('utf8', 'ignore')
+
+        return unicode(value)
 
     def parse(self, string):
         """Parse a (possibly human-written) string and return the
         indicated value of this type.
         """
-        raise NotImplementedError()
+        try:
+            return self.model_type(string)
+        except ValueError:
+            return self.null
 
+    def normalize(self, value):
+        """Given a value that will be assigned into a field of this
+        type, normalize the value to have the appropriate type. This
+        base implementation only reinterprets `None`.
+        """
+        if value is None:
+            return self.null
+        else:
+            # TODO This should eventually be replaced by
+            # `self.model_type(value)`
+            return value
+
+    def from_sql(self, sql_value):
+        """Receives the value stored in the SQL backend and return the
+        value to be stored in the model.
+
+        For fixed fields the type of `value` is determined by the column
+        type affinity given in the `sql` property and the SQL to Python
+        mapping of the database adapter. For more information see:
+        http://www.sqlite.org/datatype3.html
+        https://docs.python.org/2/library/sqlite3.html#sqlite-and-python-types
+
+        Flexible fields have the type afinity `TEXT`. This means the
+        `sql_value` is either a `buffer` or a `unicode` object` and the
+        method must handle these in addition.
+        """
+        if isinstance(sql_value, buffer):
+            sql_value = bytes(sql_value).decode('utf8', 'ignore')
+        if isinstance(sql_value, unicode):
+            return self.parse(sql_value)
+        else:
+            return self.normalize(sql_value)
+
+    def to_sql(self, model_value):
+        """Convert a value as stored in the model object to a value used
+        by the database adapter.
+        """
+        return model_value
 
 
 # Reusable types.
 
+class Default(Type):
+    null = None
+
 
 class Integer(Type):
     """A basic integer type.
     """
     sql = u'INTEGER'
     query = query.NumericQuery
-
+    model_type = int
-    def format(self, value):
-        return unicode(value or 0)
-
-    def parse(self, string):
-        try:
-            return int(string)
-        except ValueError:
-            return 0
 
 
 class PaddedInt(Integer):
@@ -93,9 +149,14 @@ class ScaledInt(Integer):
 
 
 class Id(Integer):
-    """An integer used as the row key for a SQLite table.
+    """An integer used as the row id or a foreign key in a SQLite table.
+    This type is nullable: None values are not translated to zero.
     """
-    sql = u'INTEGER PRIMARY KEY'
+    null = None
+
+    def __init__(self, primary=True):
+        if primary:
+            self.sql = u'INTEGER PRIMARY KEY'
 
 
 class Float(Type):
@@ -103,15 +164,16 @@ class Float(Type):
     """
     sql = u'REAL'
     query = query.NumericQuery
+    model_type = float
 
     def format(self, value):
         return u'{0:.1f}'.format(value or 0.0)
 
-    def parse(self, string):
+
-        try:
+class NullFloat(Float):
-            return float(string)
+    """Same as `Float`, but does not normalize `None` to `0.0`.
-        except ValueError:
+    """
-            return 0.0
+    null = None
 
 
 class String(Type):
@@ -120,21 +182,27 @@ class String(Type):
     sql = u'TEXT'
     query = query.SubstringQuery
 
-    def format(self, value):
-        return unicode(value) if value else u''
-
-    def parse(self, string):
-        return string
-
 
 class Boolean(Type):
     """A boolean type.
     """
     sql = u'INTEGER'
     query = query.BooleanQuery
+    model_type = bool
 
     def format(self, value):
         return unicode(bool(value))
 
     def parse(self, string):
         return str2bool(string)
+
+
+# Shared instances of common types.
+DEFAULT = Default()
+INTEGER = Integer()
+PRIMARY_ID = Id(True)
+FOREIGN_ID = Id(False)
+FLOAT = Float()
+NULL_FLOAT = NullFloat()
+STRING = String()
+BOOLEAN = Boolean()

lib/beets/plugins.py

@@ -16,8 +16,11 @@
 
 import logging
 import traceback
+import inspect
+import re
 from collections import defaultdict
 
+
 import beets
 from beets import mediafile
 
@@ -30,6 +33,14 @@ LASTFM_KEY = '2dc3914abf35f0d9c92d97d8f8e42b43'
 log = logging.getLogger('beets')
 
 
+class PluginConflictException(Exception):
+    """Indicates that the services provided by one plugin conflict with
+    those of another.
+
+    For example two plugins may define different types for flexible fields.
+    """
+
+
 # Managing the plugins themselves.
 
 class BeetsPlugin(object):
@@ -40,7 +51,6 @@ class BeetsPlugin(object):
     def __init__(self, name=None):
         """Perform one-time plugin setup.
         """
-        _add_media_fields(self.item_fields())
         self.import_stages = []
         self.name = name or self.__module__.split('.')[-1]
         self.config = beets.config[self.name]
@@ -86,14 +96,6 @@ class BeetsPlugin(object):
         """
         return ()
 
-    def item_fields(self):
-        """Returns field descriptors to be added to the MediaFile class,
-        in the form of a dictionary whose keys are field names and whose
-        values are descriptor (e.g., MediaField) instances. The Library
-        database schema is not (currently) extended.
-        """
-        return {}
-
     def album_for_id(self, album_id):
         """Return an AlbumInfo object or None if no matching release was
         found.
@@ -106,6 +108,20 @@ class BeetsPlugin(object):
         """
         return None
 
+    def add_media_field(self, name, descriptor):
+        """Add a field that is synchronized between media files and items.
+
+        When a media field is added ``item.write()`` will set the name
+        property of the item's MediaFile to ``item[name]`` and save the
+        changes. Similarly ``item.read()`` will set ``item[name]`` to
+        the value of the name property of the media file.
+
+        ``descriptor`` must be an instance of ``mediafile.MediaField``.
+        """
+        # Defer impor to prevent circular dependency
+        from beets import library
+        mediafile.MediaFile.add_field(name, descriptor)
+        library.Item._media_fields.add(name)
 
     listeners = None
 
@@ -130,7 +146,7 @@ class BeetsPlugin(object):
 
             >>> @MyPlugin.listen("imported")
             >>> def importListener(**kwargs):
-            >>>     pass
+            ...     pass
         """
         def helper(func):
             if cls.listeners is None:
@@ -170,7 +186,10 @@ class BeetsPlugin(object):
             return func
         return helper
 
+
 _classes = set()
+
+
 def load_plugins(names=()):
     """Imports the modules for a sequence of plugin names. Each name
     must be the name of a Python module under the "beetsplug" namespace
@@ -185,7 +204,7 @@ def load_plugins(names=()):
             except ImportError as exc:
                 # Again, this is hacky:
                 if exc.args[0].endswith(' ' + name):
-                    log.warn('** plugin %s not found' % name)
+                    log.warn(u'** plugin {0} not found'.format(name))
                 else:
                     raise
             else:
@@ -195,10 +214,13 @@ def load_plugins(names=()):
                         _classes.add(obj)
 
         except:
-            log.warn('** error loading plugin %s' % name)
+            log.warn(u'** error loading plugin {0}'.format(name))
             log.warn(traceback.format_exc())
 
+
 _instances = {}
+
+
 def find_plugins():
     """Returns a list of BeetsPlugin subclass instances from all
     currently loaded beets plugins. Loads the default plugin set
@@ -224,6 +246,7 @@ def commands():
         out += plugin.commands()
     return out
 
+
 def queries():
     """Returns a dict mapping prefix strings to Query subclasses all loaded
     plugins.
@@ -233,6 +256,24 @@ def queries():
         out.update(plugin.queries())
     return out
 
+
+def types(model_cls):
+    # Gives us `item_types` and `album_types`
+    attr_name = '{0}_types'.format(model_cls.__name__.lower())
+    types = {}
+    for plugin in find_plugins():
+        plugin_types = getattr(plugin, attr_name, {})
+        for field in plugin_types:
+            if field in types and plugin_types[field] != types[field]:
+                raise PluginConflictException(
+                    u'Plugin {0} defines flexible field {1} '
+                    'which has already been defined with '
+                    'another type.'.format(plugin.name, field)
+                )
+        types.update(plugin_types)
+    return types
+
+
 def track_distance(item, info):
     """Gets the track distance calculated by all loaded plugins.
     Returns a Distance object.
@@ -243,6 +284,7 @@ def track_distance(item, info):
         dist.update(plugin.track_distance(item, info))
     return dist
 
+
 def album_distance(items, album_info, mapping):
     """Returns the album distance calculated by plugins."""
     from beets.autotag.hooks import Distance
@@ -251,6 +293,7 @@ def album_distance(items, album_info, mapping):
         dist.update(plugin.album_distance(items, album_info, mapping))
     return dist
 
+
 def candidates(items, artist, album, va_likely):
     """Gets MusicBrainz candidates for an album from each plugin.
     """
@@ -259,6 +302,7 @@ def candidates(items, artist, album, va_likely):
         out.extend(plugin.candidates(items, artist, album, va_likely))
     return out
 
+
 def item_candidates(item, artist, title):
     """Gets MusicBrainz candidates for an item from the plugins.
     """
@@ -267,6 +311,7 @@ def item_candidates(item, artist, title):
         out.extend(plugin.item_candidates(item, artist, title))
     return out
 
+
 def album_for_id(album_id):
     """Get AlbumInfo objects for a given ID string.
     """
@@ -277,6 +322,7 @@ def album_for_id(album_id):
             out.append(res)
     return out
 
+
 def track_for_id(track_id):
     """Get TrackInfo objects for a given ID string.
     """
@@ -287,6 +333,7 @@ def track_for_id(track_id):
             out.append(res)
     return out
 
+
 def template_funcs():
     """Get all the template functions declared by plugins as a
     dictionary.
@@ -297,12 +344,6 @@ def template_funcs():
             funcs.update(plugin.template_funcs)
     return funcs
 
-def _add_media_fields(fields):
-    """Adds a {name: descriptor} dictionary of fields to the MediaFile
-    class. Called during the plugin initialization.
-    """
-    for key, value in fields.iteritems():
-        setattr(mediafile.MediaFile, key, value)
 
 def import_stages():
     """Get a list of import stage functions defined by plugins."""
@@ -325,6 +366,7 @@ def item_field_getters():
             funcs.update(plugin.template_fields)
     return funcs
 
+
 def album_field_getters():
     """As above, for album fields.
     """
@@ -348,6 +390,7 @@ def event_handlers():
                 all_handlers[event] += handlers
     return all_handlers
 
+
 def send(event, **arguments):
     """Sends an event to all assigned event listeners. Event is the
     name of  the event to send, all other named arguments go to the
@@ -355,5 +398,38 @@ def send(event, **arguments):
 
     Returns a list of return values from the handlers.
     """
-    log.debug('Sending event: %s' % event)
+    log.debug(u'Sending event: {0}'.format(event))
-    return [handler(**arguments) for handler in event_handlers()[event]]
+    for handler in event_handlers()[event]:
+        # Don't break legacy plugins if we want to pass more arguments
+        argspec = inspect.getargspec(handler).args
+        args = dict((k, v) for k, v in arguments.items() if k in argspec)
+        handler(**args)
+
+
+def feat_tokens(for_artist=True):
+    """Return a regular expression that matches phrases like "featuring"
+    that separate a main artist or a song title from secondary artists.
+    The `for_artist` option determines whether the regex should be
+    suitable for matching artist fields (the default) or title fields.
+    """
+    feat_words = ['ft', 'featuring', 'feat', 'feat.', 'ft.']
+    if for_artist:
+        feat_words += ['with', 'vs', 'and', 'con', '&']
+    return '(?<=\s)(?:{0})(?=\s)'.format(
+        '|'.join(re.escape(x) for x in feat_words)
+    )
+
+
+def sanitize_choices(choices, choices_all):
+    """Clean up a stringlist configuration attribute: keep only choices
+    elements present in choices_all, remove duplicate elements, expand '*'
+    wildcard while keeping original stringlist order.
+    """
+    seen = set()
+    others = [x for x in choices_all if x not in choices]
+    res = []
+    for s in choices:
+        if s in list(choices_all) + ['*']:
+            if not (s in seen or seen.add(s)):
+                res.extend(list(others) if s == '*' else [s])
+    return res

lib/beets/ui/__init__.py

@@ -29,6 +29,7 @@ import errno
 import re
 import struct
 import traceback
+import os.path
 
 from beets import library
 from beets import plugins
@@ -38,9 +39,7 @@ from beets import config
 from beets.util import confit
 from beets.autotag import mb
 
-
 # On Windows platforms, use colorama to support "ANSI" terminal colors.
-
 if sys.platform == 'win32':
     try:
         import colorama
@@ -50,8 +49,10 @@ if sys.platform == 'win32':
         colorama.init()
 
 
-
+log = logging.getLogger('beets')
-# Constants.
+if not log.handlers:
+    log.addHandler(logging.StreamHandler())
+log.propagate = False  # Don't propagate to root handler.
 
 
 PF_KEY_QUERIES = {
@@ -59,19 +60,15 @@ PF_KEY_QUERIES = {
     'singleton': 'singleton:true',
 }
 
-# UI exception. Commands should throw this in order to display
+
-# nonrecoverable errors to the user.
 class UserError(Exception):
-    pass
+    """UI exception. Commands should throw this in order to display
-
+    nonrecoverable errors to the user.
-# Main logger.
+    """
-log = logging.getLogger('beets')
-
 
 
 # Utilities.
 
-
 def _encoding():
     """Tries to guess the encoding used by the terminal."""
     # Configured override?
@@ -170,7 +167,7 @@ def input_options(options, require=False, prompt=None, fallback_prompt=None,
             # Infer a letter.
             for letter in option:
                 if not letter.isalpha():
-                    continue # Don't use punctuation.
+                    continue  # Don't use punctuation.
                 if letter not in letters:
                     found_letter = letter
                     break
@@ -181,9 +178,10 @@ def input_options(options, require=False, prompt=None, fallback_prompt=None,
         index = option.index(found_letter)
 
         # Mark the option's shortcut letter for display.
-        if not require and ((default is None and not numrange and first) or
+        if not require and (
-                (isinstance(default, basestring) and
+            (default is None and not numrange and first) or
-                 found_letter.lower() == default.lower())):
+            (isinstance(default, basestring) and
+             found_letter.lower() == default.lower())):
             # The first option is the default; mark it.
             show_letter = '[%s]' % found_letter.upper()
             is_default = True
@@ -352,11 +350,13 @@ def human_seconds_short(interval):
 # http://dev.pocoo.org/hg/pygments-main/file/b2deea5b5030/pygments/console.py
 # (pygments is by Tim Hatch, Armin Ronacher, et al.)
 COLOR_ESCAPE = "\x1b["
-DARK_COLORS  = ["black", "darkred", "darkgreen", "brown", "darkblue",
+DARK_COLORS = ["black", "darkred", "darkgreen", "brown", "darkblue",
-                "purple", "teal", "lightgray"]
+               "purple", "teal", "lightgray"]
 LIGHT_COLORS = ["darkgray", "red", "green", "yellow", "blue",
                 "fuchsia", "turquoise", "white"]
 RESET_COLOR = COLOR_ESCAPE + "39;49;00m"
+
+
 def _colorize(color, text):
     """Returns a string that prints the given text in the given color
     in a terminal that is ANSI color-aware. The color must be something
@@ -441,30 +441,6 @@ def colordiff(a, b, highlight='red'):
         return unicode(a), unicode(b)
 
 
-def color_diff_suffix(a, b, highlight='red'):
-    """Colorize the differing suffix between two strings."""
-    a, b = unicode(a), unicode(b)
-    if not config['color']:
-        return a, b
-
-    # Fast path.
-    if a == b:
-        return a, b
-
-    # Find the longest common prefix.
-    first_diff = None
-    for i in range(min(len(a), len(b))):
-        if a[i] != b[i]:
-            first_diff = i
-            break
-    else:
-        first_diff = min(len(a), len(b))
-
-    # Colorize from the first difference on.
-    return a[:first_diff] + colorize(highlight, a[first_diff:]), \
-           b[:first_diff] + colorize(highlight, b[first_diff:])
-
-
 def get_path_formats(subview=None):
     """Get the configuration's path formats as a list of query/template
     pairs.
@@ -494,21 +470,6 @@ def get_replacements():
     return replacements
 
 
-def get_plugin_paths():
-    """Get the list of search paths for plugins from the config file.
-    The value for "pluginpath" may be a single string or a list of
-    strings.
-    """
-    pluginpaths = config['pluginpath'].get()
-    if isinstance(pluginpaths, basestring):
-        pluginpaths = [pluginpaths]
-    if not isinstance(pluginpaths, list):
-        raise confit.ConfigTypeError(
-            u'pluginpath must be string or a list of strings'
-        )
-    return map(util.normpath, pluginpaths)
-
-
 def _pick_format(album, fmt=None):
     """Pick a format string for printing Album or Item objects,
     falling back to config options and defaults.
@@ -558,6 +519,8 @@ def term_width():
 
 
 FLOAT_EPSILON = 0.01
+
+
 def _field_diff(field, old, new):
     """Given two Model objects, format their values for `field` and
     highlight changes among them. Return a human-readable string. If the
@@ -574,13 +537,13 @@ def _field_diff(field, old, new):
         return None
 
     # Get formatted values for output.
-    oldstr = old._get_formatted(field)
+    oldstr = old.formatted().get(field, u'')
-    newstr = new._get_formatted(field)
+    newstr = new.formatted().get(field, u'')
 
     # For strings, highlight changes. For others, colorize the whole
     # thing.
     if isinstance(oldval, basestring):
-        oldstr, newstr = colordiff(oldval, newval)
+        oldstr, newstr = colordiff(oldval, newstr)
     else:
         oldstr, newstr = colorize('red', oldstr), colorize('red', newstr)
 
@@ -613,9 +576,12 @@ def show_model_changes(new, old=None, fields=None, always=False):
 
     # New fields.
     for field in set(new) - set(old):
+        if fields and field not in fields:
+            continue
+
         changes.append(u'  {0}: {1}'.format(
             field,
-            colorize('red', new._get_formatted(field))
+            colorize('red', new.formatted()[field])
         ))
 
     # Print changes.
@@ -627,10 +593,8 @@ def show_model_changes(new, old=None, fields=None, always=False):
     return bool(changes)
 
 
-
 # Subcommand parsing infrastructure.
-
+#
-
 # This is a fairly generic subcommand parser for optparse. It is
 # maintained externally here:
 # http://gist.github.com/462717
@@ -653,46 +617,56 @@ class Subcommand(object):
         self.aliases = aliases
         self.help = help
         self.hide = hide
+        self._root_parser = None
+
+    def print_help(self):
+        self.parser.print_help()
+
+    def parse_args(self, args):
+        return self.parser.parse_args(args)
+
+    @property
+    def root_parser(self):
+        return self._root_parser
+
+    @root_parser.setter
+    def root_parser(self, root_parser):
+        self._root_parser = root_parser
+        self.parser.prog = '{0} {1}'.format(root_parser.get_prog_name(),
+                                            self.name)
+
 
 class SubcommandsOptionParser(optparse.OptionParser):
     """A variant of OptionParser that parses subcommands and their
     arguments.
     """
-    # A singleton command used to give help on other subcommands.
-    _HelpSubcommand = Subcommand('help', optparse.OptionParser(),
-        help='give detailed help on a specific sub-command',
-        aliases=('?',))
 
     def __init__(self, *args, **kwargs):
         """Create a new subcommand-aware option parser. All of the
         options to OptionParser.__init__ are supported in addition
         to subcommands, a sequence of Subcommand objects.
         """
-        # The subcommand array, with the help command included.
-        self.subcommands = list(kwargs.pop('subcommands', []))
-        self.subcommands.append(self._HelpSubcommand)
-
         # A more helpful default usage.
         if 'usage' not in kwargs:
             kwargs['usage'] = """
   %prog COMMAND [ARGS...]
   %prog help COMMAND"""
+        kwargs['add_help_option'] = False
 
         # Super constructor.
         optparse.OptionParser.__init__(self, *args, **kwargs)
 
-        # Adjust the help-visible name of each subcommand.
-        for subcommand in self.subcommands:
-            subcommand.parser.prog = '%s %s' % \
-                    (self.get_prog_name(), subcommand.name)
-
         # Our root parser needs to stop on the first unrecognized argument.
         self.disable_interspersed_args()
 
-    def add_subcommand(self, cmd):
+        self.subcommands = []
+
+    def add_subcommand(self, *cmds):
         """Adds a Subcommand object to the parser's list of commands.
         """
-        self.subcommands.append(cmd)
+        for cmd in cmds:
+            cmd.root_parser = self
+            self.subcommands.append(cmd)
 
     # Add the list of subcommands to the help message.
     def format_help(self, formatter=None):
@@ -711,6 +685,7 @@ class SubcommandsOptionParser(optparse.OptionParser):
         disp_names = []
         help_position = 0
         subcommands = [c for c in self.subcommands if not c.hide]
+        subcommands.sort(key=lambda c: c.name)
         for subcommand in subcommands:
             name = subcommand.name
             if subcommand.aliases:
@@ -756,52 +731,40 @@ class SubcommandsOptionParser(optparse.OptionParser):
                 return subcommand
         return None
 
-    def parse_args(self, a=None, v=None):
+    def parse_global_options(self, args):
-        """Like OptionParser.parse_args, but returns these four items:
+        """Parse options up to the subcommand argument. Returns a tuple
-        - options: the options passed to the root parser
+        of the options object and the remaining arguments.
-        - subcommand: the Subcommand object that was invoked
-        - suboptions: the options passed to the subcommand parser
-        - subargs: the positional arguments passed to the subcommand
         """
-        options, args = optparse.OptionParser.parse_args(self, a, v)
+        options, subargs = self.parse_args(args)
-        subcommand, suboptions, subargs = self._parse_sub(args)
-        return options, subcommand, suboptions, subargs
 
-    def _parse_sub(self, args):
+        # Force the help command
-        """Given the `args` left unused by a typical OptionParser
+        if options.help:
-        `parse_args`, return the invoked subcommand, the subcommand
+            subargs = ['help']
-        options, and the subcommand arguments.
+        elif options.version:
+            subargs = ['version']
+        return options, subargs
+
+    def parse_subcommand(self, args):
+        """Given the `args` left unused by a `parse_global_options`,
+        return the invoked subcommand, the subcommand options, and the
+        subcommand arguments.
         """
+        # Help is default command
         if not args:
-            # No command given.
+            args = ['help']
-            self.print_help()
-            self.exit()
-        else:
-            cmdname = args.pop(0)
-            subcommand = self._subcommand_for_name(cmdname)
-            if not subcommand:
-                self.error('unknown command ' + cmdname)
 
-        suboptions, subargs = subcommand.parser.parse_args(args)
+        cmdname = args.pop(0)
-
+        subcommand = self._subcommand_for_name(cmdname)
-        if subcommand is self._HelpSubcommand:
+        if not subcommand:
-            if subargs:
+            raise UserError("unknown command '{0}'".format(cmdname))
-                # particular
-                cmdname = subargs[0]
-                helpcommand = self._subcommand_for_name(cmdname)
-                if not helpcommand:
-                    self.error('no command named {0}'.format(cmdname))
-                helpcommand.parser.print_help()
-                self.exit()
-            else:
-                # general
-                self.print_help()
-                self.exit()
 
+        suboptions, subargs = subcommand.parse_args(args)
         return subcommand, suboptions, subargs
 
 
 optparse.Option.ALWAYS_TYPED_ACTIONS += ('callback',)
+
+
 def vararg_callback(option, opt_str, value, parser):
     """Callback for an option with variable arguments.
     Manually collect arguments right of a callback-action
@@ -838,53 +801,54 @@ def vararg_callback(option, opt_str, value, parser):
     setattr(parser.values, option.dest, value)
 
 
-
 # The main entry point and bootstrapping.
 
-
+def _load_plugins(config):
-def _load_plugins():
     """Load the plugins specified in the configuration.
     """
-    # Add plugin paths.
+    paths = config['pluginpath'].get(confit.StrSeq(split=False))
+    paths = map(util.normpath, paths)
+
     import beetsplug
-    beetsplug.__path__ = get_plugin_paths() + beetsplug.__path__
+    beetsplug.__path__ = paths + beetsplug.__path__
-
     # For backwards compatibility.
-    sys.path += get_plugin_paths()
+    sys.path += paths
 
-    # Load requested plugins.
     plugins.load_plugins(config['plugins'].as_str_seq())
     plugins.send("pluginload")
+    return plugins
 
 
-def _configure(args):
+def _setup(options, lib=None):
-    """Parse the command line, load configuration files (including
+    """Prepare and global state and updates it with command line options.
-    loading any indicated plugins), and return the invoked subcomand,
+
-    the subcommand options, and the subcommand arguments.
+    Returns a list of subcommands, a list of plugins, and a library instance.
     """
-    # Temporary: Migrate from 1.0-style configuration.
+    # Configure the MusicBrainz API.
-    from beets.ui import migrate
+    mb.configure()
-    migrate.automigrate()
+
+    config = _configure(options)
+
+    plugins = _load_plugins(config)
 
     # Get the default subcommands.
     from beets.ui.commands import default_commands
 
-    # Construct the root parser.
+    subcommands = list(default_commands)
-    commands = list(default_commands)
+    subcommands.extend(plugins.commands())
-    commands.append(migrate.migrate_cmd)  # Temporary.
-    parser = SubcommandsOptionParser(subcommands=commands)
-    parser.add_option('-l', '--library', dest='library',
-                      help='library database file to use')
-    parser.add_option('-d', '--directory', dest='directory',
-                      help="destination music directory")
-    parser.add_option('-v', '--verbose', dest='verbose', action='store_true',
-                      help='print debugging information')
-    parser.add_option('-c', '--config', dest='config',
-                      help='path to configuration file')
 
-    # Parse the command-line!
+    if lib is None:
-    options, args = optparse.OptionParser.parse_args(parser, args)
+        lib = _open_library(config)
+        plugins.send("library_opened", lib=lib)
+    library.Item._types = plugins.types(library.Item)
+    library.Album._types = plugins.types(library.Album)
 
+    return subcommands, plugins, lib
+
+
+def _configure(options):
+    """Amend the global configuration object with command line options.
+    """
     # Add any additional config files specified with --config. This
     # special handling lets specified plugins get loaded before we
     # finish parsing the command line.
@@ -894,22 +858,28 @@ def _configure(args):
         config.set_file(config_path)
     config.set_args(options)
 
-    # Now add the plugin commands to the parser.
+    # Configure the logger.
-    _load_plugins()
+    if config['verbose'].get(bool):
-    for cmd in plugins.commands():
+        log.setLevel(logging.DEBUG)
-        parser.add_subcommand(cmd)
+    else:
+        log.setLevel(logging.INFO)
 
-    # Parse the remainder of the command line with loaded plugins.
+    config_path = config.user_config_path()
-    return parser._parse_sub(args)
+    if os.path.isfile(config_path):
+        log.debug(u'user configuration: {0}'.format(
+            util.displayable_path(config_path)))
+    else:
+        log.debug(u'no user configuration found at {0}'.format(
+            util.displayable_path(config_path)))
+
+    log.debug(u'data directory: {0}'
+              .format(util.displayable_path(config.config_dir())))
+    return config
 
 
-def _raw_main(args):
+def _open_library(config):
-    """A helper function for `main` without top-level exception
+    """Create a new library instance from the configuration.
-    handling.
     """
-    subcommand, suboptions, subargs = _configure(args)
-
-    # Open library file.
     dbpath = config['library'].as_filename()
     try:
         lib = library.Library(
@@ -918,32 +888,52 @@ def _raw_main(args):
             get_path_formats(),
             get_replacements(),
         )
-    except sqlite3.OperationalError:
+        lib.get_item(0)  # Test database connection.
+    except (sqlite3.OperationalError, sqlite3.DatabaseError):
+        log.debug(traceback.format_exc())
         raise UserError(u"database file {0} could not be opened".format(
             util.displayable_path(dbpath)
         ))
-    plugins.send("library_opened", lib=lib)
+    log.debug(u'library database: {0}\n'
+              u'library directory: {1}'
+              .format(util.displayable_path(lib.path),
+                      util.displayable_path(lib.directory)))
+    return lib
 
-    # Configure the logger.
-    if config['verbose'].get(bool):
-        log.setLevel(logging.DEBUG)
-    else:
-        log.setLevel(logging.INFO)
-    log.debug(u'data directory: {0}\n'
-              u'library database: {1}\n'
-              u'library directory: {2}'
-        .format(
-            util.displayable_path(config.config_dir()),
-            util.displayable_path(lib.path),
-            util.displayable_path(lib.directory),
-        )
-    )
 
-    # Configure the MusicBrainz API.
+def _raw_main(args, lib=None):
-    mb.configure()
+    """A helper function for `main` without top-level exception
+    handling.
+    """
+    parser = SubcommandsOptionParser()
+    parser.add_option('-l', '--library', dest='library',
+                      help='library database file to use')
+    parser.add_option('-d', '--directory', dest='directory',
+                      help="destination music directory")
+    parser.add_option('-v', '--verbose', dest='verbose', action='store_true',
+                      help='print debugging information')
+    parser.add_option('-c', '--config', dest='config',
+                      help='path to configuration file')
+    parser.add_option('-h', '--help', dest='help', action='store_true',
+                      help='how this help message and exit')
+    parser.add_option('--version', dest='version', action='store_true',
+                      help=optparse.SUPPRESS_HELP)
 
-    # Invoke the subcommand.
+    options, subargs = parser.parse_global_options(args)
+
+    # Special case for the `config --edit` command: bypass _setup so
+    # that an invalid configuration does not prevent the editor from
+    # starting.
+    if subargs[0] == 'config' and ('-e' in subargs or '--edit' in subargs):
+        from beets.ui.commands import config_edit
+        return config_edit()
+
+    subcommands, plugins, lib = _setup(options, lib)
+    parser.add_subcommand(*subcommands)
+
+    subcommand, suboptions, subargs = parser.parse_subcommand(subargs)
     subcommand.func(lib, suboptions, subargs)
+
     plugins.send('cli_exit', lib=lib)
 
 

lib/beets/ui/migrate.py

@@ -1,401 +0,0 @@
-# This file is part of beets.
-# Copyright 2013, Adrian Sampson.
-#
-# Permission is hereby granted, free of charge, to any person obtaining
-# a copy of this software and associated documentation files (the
-# "Software"), to deal in the Software without restriction, including
-# without limitation the rights to use, copy, modify, merge, publish,
-# distribute, sublicense, and/or sell copies of the Software, and to
-# permit persons to whom the Software is furnished to do so, subject to
-# the following conditions:
-#
-# The above copyright notice and this permission notice shall be
-# included in all copies or substantial portions of the Software.
-
-"""Conversion from legacy (pre-1.1) configuration to Confit/YAML
-configuration.
-"""
-import os
-import ConfigParser
-import codecs
-import yaml
-import logging
-import time
-import itertools
-import re
-
-import beets
-from beets import util
-from beets import ui
-from beets.util import confit
-
-CONFIG_PATH_VAR = 'BEETSCONFIG'
-DEFAULT_CONFIG_FILENAME_UNIX = '.beetsconfig'
-DEFAULT_CONFIG_FILENAME_WINDOWS = 'beetsconfig.ini'
-DEFAULT_LIBRARY_FILENAME_UNIX = '.beetsmusic.blb'
-DEFAULT_LIBRARY_FILENAME_WINDOWS = 'beetsmusic.blb'
-WINDOWS_BASEDIR = os.environ.get('APPDATA') or '~'
-
-OLD_CONFIG_SUFFIX = '.old'
-PLUGIN_NAMES = {
-    'rdm': 'random',
-    'fuzzy_search': 'fuzzy',
-}
-AUTO_KEYS = ('automatic', 'autofetch', 'autoembed', 'autoscrub')
-IMPORTFEEDS_PREFIX = 'feeds_'
-CONFIG_MIGRATED_MESSAGE = u"""
-You appear to be upgrading from beets 1.0 (or earlier) to 1.1. Your
-configuration file has been migrated automatically to:
-{newconfig}
-Edit this file to configure beets. You might want to remove your
-old-style ".beetsconfig" file now. See the documentation for more
-details on the new configuration system:
-http://beets.readthedocs.org/page/reference/config.html
-""".strip()
-DB_MIGRATED_MESSAGE = u'Your database file has also been copied to:\n{newdb}'
-YAML_COMMENT = '# Automatically migrated from legacy .beetsconfig.\n\n'
-
-log = logging.getLogger('beets')
-
-# An itertools recipe.
-def grouper(n, iterable):
-    args = [iter(iterable)] * n
-    return itertools.izip_longest(*args)
-
-def _displace(fn):
-    """Move a file aside using a timestamp suffix so a new file can be
-    put in its place.
-    """
-    util.move(
-        fn,
-        u'{0}.old.{1}'.format(fn, int(time.time())),
-        True
-    )
-
-def default_paths():
-    """Produces the appropriate default config and library database
-    paths for the current system. On Unix, this is always in ~. On
-    Windows, tries ~ first and then $APPDATA for the config and library
-    files (for backwards compatibility).
-    """
-    windows = os.path.__name__ == 'ntpath'
-    if windows:
-        windata = os.environ.get('APPDATA') or '~'
-
-    # Shorthand for joining paths.
-    def exp(*vals):
-        return os.path.expanduser(os.path.join(*vals))
-
-    config = exp('~', DEFAULT_CONFIG_FILENAME_UNIX)
-    if windows and not os.path.exists(config):
-        config = exp(windata, DEFAULT_CONFIG_FILENAME_WINDOWS)
-
-    libpath = exp('~', DEFAULT_LIBRARY_FILENAME_UNIX)
-    if windows and not os.path.exists(libpath):
-        libpath = exp(windata, DEFAULT_LIBRARY_FILENAME_WINDOWS)
-
-    return config, libpath
-
-def get_config():
-    """Using the same logic as beets 1.0, locate and read the
-    .beetsconfig file. Return a ConfigParser instance or None if no
-    config is found.
-    """
-    default_config, default_libpath = default_paths()
-    if CONFIG_PATH_VAR in os.environ:
-        configpath = os.path.expanduser(os.environ[CONFIG_PATH_VAR])
-    else:
-        configpath = default_config
-
-    config = ConfigParser.SafeConfigParser()
-    if os.path.exists(util.syspath(configpath)):
-        with codecs.open(configpath, 'r', encoding='utf-8') as f:
-            config.readfp(f)
-        return config, configpath
-    else:
-        return None, configpath
-
-def flatten_config(config):
-    """Given a ConfigParser, flatten the values into a dict-of-dicts
-    representation where each section gets its own dictionary of values.
-    """
-    out = confit.OrderedDict()
-    for section in config.sections():
-        sec_dict = out[section] = confit.OrderedDict()
-        for option in config.options(section):
-            sec_dict[option] = config.get(section, option, True)
-    return out
-
-def transform_value(value):
-    """Given a string read as the value of a config option, return a
-    massaged version of that value (possibly with a different type).
-    """
-    # Booleans.
-    if value.lower() in ('false', 'no', 'off'):
-        return False
-    elif value.lower() in ('true', 'yes', 'on'):
-        return True
-
-    # Integers.
-    try:
-        return int(value)
-    except ValueError:
-        pass
-
-    # Floats.
-    try:
-        return float(value)
-    except ValueError:
-        pass
-
-    return value
-
-def transform_data(data):
-    """Given a dict-of-dicts representation of legacy config data, tweak
-    the data into a new form. This new form is suitable for dumping as
-    YAML.
-    """
-    out = confit.OrderedDict()
-
-    for section, pairs in data.items():
-        if section == 'beets':
-            # The "main" section. In the new config system, these values
-            # are in the "root": no section at all.
-            for key, value in pairs.items():
-                value = transform_value(value)
-
-                if key.startswith('import_'):
-                    # Importer config is now under an "import:" key.
-                    if 'import' not in out:
-                        out['import'] = confit.OrderedDict()
-                    out['import'][key[7:]] = value
-
-                elif key == 'plugins':
-                    # Renamed plugins.
-                    plugins = value.split()
-                    new_plugins = [PLUGIN_NAMES.get(p, p) for p in plugins]
-                    out['plugins'] = ' '.join(new_plugins)
-
-                elif key == 'replace':
-                    # YAMLy representation for character replacements.
-                    replacements = confit.OrderedDict()
-                    for pat, repl in grouper(2, value.split()):
-                        if repl == '<strip>':
-                            repl = ''
-                        replacements[pat] = repl
-                    out['replace'] = replacements
-
-                elif key == 'pluginpath':
-                    # Used to be a colon-separated string. Now a list.
-                    out['pluginpath'] = value.split(':')
-
-                else:
-                    out[key] = value
-
-        elif pairs:
-            # Other sections (plugins, etc).
-            sec_out = out[section] = confit.OrderedDict()
-            for key, value in pairs.items():
-
-                # Standardized "auto" option.
-                if key in AUTO_KEYS:
-                    key = 'auto'
-
-                # Unnecessary : hack in queries.
-                if section == 'paths':
-                    key = key.replace('_', ':')
-
-                # Changed option names for importfeeds plugin.
-                if section == 'importfeeds':
-                    if key.startswith(IMPORTFEEDS_PREFIX):
-                        key = key[len(IMPORTFEEDS_PREFIX):]
-
-                sec_out[key] = transform_value(value)
-
-    return out
-
-class Dumper(yaml.SafeDumper):
-    """A PyYAML Dumper that represents OrderedDicts as ordinary mappings
-    (in order, of course).
-    """
-    # From http://pyyaml.org/attachment/ticket/161/use_ordered_dict.py
-    def represent_mapping(self, tag, mapping, flow_style=None):
-        value = []
-        node = yaml.MappingNode(tag, value, flow_style=flow_style)
-        if self.alias_key is not None:
-            self.represented_objects[self.alias_key] = node
-        best_style = True
-        if hasattr(mapping, 'items'):
-            mapping = list(mapping.items())
-        for item_key, item_value in mapping:
-            node_key = self.represent_data(item_key)
-            node_value = self.represent_data(item_value)
-            if not (isinstance(node_key, yaml.ScalarNode) and \
-                    not node_key.style):
-                best_style = False
-            if not (isinstance(node_value, yaml.ScalarNode) and \
-                    not node_value.style):
-                best_style = False
-            value.append((node_key, node_value))
-        if flow_style is None:
-            if self.default_flow_style is not None:
-                node.flow_style = self.default_flow_style
-            else:
-                node.flow_style = best_style
-        return node
-Dumper.add_representer(confit.OrderedDict, Dumper.represent_dict)
-
-def migrate_config(replace=False):
-    """Migrate a legacy beetsconfig file to a new-style config.yaml file
-    in an appropriate place. If `replace` is enabled, then any existing
-    config.yaml will be moved aside. Otherwise, the process is aborted
-    when the file exists.
-    """
-
-    # Load legacy configuration data, if any.
-    config, configpath = get_config()
-    if not config:
-        log.debug(u'no config file found at {0}'.format(
-            util.displayable_path(configpath)
-            ))
-        return
-
-    # Get the new configuration file path and possibly move it out of
-    # the way.
-    destfn = os.path.join(beets.config.config_dir(), confit.CONFIG_FILENAME)
-    if os.path.exists(destfn):
-        if replace:
-            log.debug(u'moving old config aside: {0}'.format(
-                util.displayable_path(destfn)
-            ))
-            _displace(destfn)
-        else:
-            # File exists and we won't replace it. We're done.
-            return
-
-    log.debug(u'migrating config file {0}'.format(
-        util.displayable_path(configpath)
-    ))
-
-    # Convert the configuration to a data structure ready to be dumped
-    # as the new Confit file.
-    data = transform_data(flatten_config(config))
-
-    # Encode result as YAML.
-    yaml_out = yaml.dump(
-        data,
-        Dumper=Dumper,
-        default_flow_style=False,
-        indent=4,
-        width=1000,
-    )
-    # A ridiculous little hack to add some whitespace between "sections"
-    # in the YAML output. I hope this doesn't break any YAML syntax.
-    yaml_out = re.sub(r'(\n\w+:\n    [^-\s])', '\n\\1', yaml_out)
-    yaml_out = YAML_COMMENT + yaml_out
-
-    # Write the data to the new config destination.
-    log.debug(u'writing migrated config to {0}'.format(
-        util.displayable_path(destfn)
-    ))
-    with open(destfn, 'w') as f:
-        f.write(yaml_out)
-    return destfn
-
-def migrate_db(replace=False):
-    """Copy the beets library database file to the new location (e.g.,
-    from ~/.beetsmusic.blb to ~/.config/beets/library.db).
-    """
-    _, srcfn = default_paths()
-    destfn = beets.config['library'].as_filename()
-
-    if not os.path.exists(srcfn) or srcfn == destfn:
-        # Old DB does not exist or we're configured to point to the same
-        # database. Do nothing.
-        return
-
-    if os.path.exists(destfn):
-        if replace:
-            log.debug(u'moving old database aside: {0}'.format(
-                util.displayable_path(destfn)
-            ))
-            _displace(destfn)
-        else:
-            return
-
-    log.debug(u'copying database from {0} to {1}'.format(
-        util.displayable_path(srcfn), util.displayable_path(destfn)
-    ))
-    util.copy(srcfn, destfn)
-    return destfn
-
-def migrate_state(replace=False):
-    """Copy the beets runtime state file from the old path (i.e.,
-    ~/.beetsstate) to the new path (i.e., ~/.config/beets/state.pickle).
-    """
-    srcfn = os.path.expanduser(os.path.join('~', '.beetsstate'))
-    if not os.path.exists(srcfn):
-        return
-
-    destfn = beets.config['statefile'].as_filename()
-    if os.path.exists(destfn):
-        if replace:
-            _displace(destfn)
-        else:
-            return
-
-    log.debug(u'copying state file from {0} to {1}'.format(
-        util.displayable_path(srcfn), util.displayable_path(destfn)
-    ))
-    util.copy(srcfn, destfn)
-    return destfn
-
-
-# Automatic migration when beets starts.
-
-def automigrate():
-    """Migrate the configuration, database, and state files. If any
-    migration occurs, print out a notice with some helpful next steps.
-    """
-    config_fn = migrate_config()
-    db_fn = migrate_db()
-    migrate_state()
-
-    if config_fn:
-        ui.print_(ui.colorize('fuchsia', u'MIGRATED CONFIGURATION'))
-
-        ui.print_(CONFIG_MIGRATED_MESSAGE.format(
-            newconfig=util.displayable_path(config_fn))
-        )
-        if db_fn:
-            ui.print_(DB_MIGRATED_MESSAGE.format(
-                newdb=util.displayable_path(db_fn)
-            ))
-
-        ui.input_(ui.colorize('fuchsia', u'Press ENTER to continue:'))
-        ui.print_()
-
-
-# CLI command for explicit migration.
-
-migrate_cmd = ui.Subcommand('migrate', help='convert legacy config')
-def migrate_func(lib, opts, args):
-    """Explicit command for migrating files. Existing files in each
-    destination are moved aside.
-    """
-    config_fn = migrate_config(replace=True)
-    if config_fn:
-        log.info(u'Migrated configuration to: {0}'.format(
-            util.displayable_path(config_fn)
-        ))
-    db_fn = migrate_db(replace=True)
-    if db_fn:
-        log.info(u'Migrated library database to: {0}'.format(
-            util.displayable_path(db_fn)
-        ))
-    state_fn = migrate_state(replace=True)
-    if state_fn:
-        log.info(u'Migrated state file to: {0}'.format(
-            util.displayable_path(state_fn)
-        ))
-migrate_cmd.func = migrate_func

lib/beets/util/__init__.py

@@ -23,10 +23,13 @@ import fnmatch
 from collections import defaultdict
 import traceback
 import subprocess
+import platform
+
 
 MAX_FILENAME_LENGTH = 200
 WINDOWS_MAGIC_PREFIX = u'\\\\?\\'
 
+
 class HumanReadableException(Exception):
     """An Exception that can include a human-readable error message to
     be logged without a traceback. Can preserve a traceback for
@@ -82,6 +85,7 @@ class HumanReadableException(Exception):
             logger.debug(self.tb)
         logger.error(u'{0}: {1}'.format(self.error_kind, self.args[0]))
 
+
 class FilesystemError(HumanReadableException):
     """An error that occurred while performing a filesystem manipulation
     via a function in this module. The `paths` field is a sequence of
@@ -111,6 +115,7 @@ class FilesystemError(HumanReadableException):
 
         return u'{0} {1}'.format(self._reasonstr(), clause)
 
+
 def normpath(path):
     """Provide the canonical form of the path suitable for storing in
     the database.
@@ -119,6 +124,7 @@ def normpath(path):
     path = os.path.normpath(os.path.abspath(os.path.expanduser(path)))
     return bytestring_path(path)
 
+
 def ancestry(path):
     """Return a list consisting of path's parent directory, its
     grandparent, and so on. For instance:
@@ -137,10 +143,12 @@ def ancestry(path):
             break
         last_path = path
 
-        if path: # don't yield ''
+        if path:
+            # don't yield ''
             out.insert(0, path)
     return out
 
+
 def sorted_walk(path, ignore=(), logger=None):
     """Like `os.walk`, but yields things in case-insensitive sorted,
     breadth-first order.  Directory and file names matching any glob
@@ -192,6 +200,7 @@ def sorted_walk(path, ignore=(), logger=None):
         for res in sorted_walk(cur, ignore, logger):
             yield res
 
+
 def mkdirall(path):
     """Make all the enclosing directories of path (like mkdir -p on the
     parent).
@@ -204,6 +213,7 @@ def mkdirall(path):
                 raise FilesystemError(exc, 'create', (ancestor,),
                                       traceback.format_exc())
 
+
 def fnmatch_all(names, patterns):
     """Determine whether all strings in `names` match at least one of
     the `patterns`, which should be shell glob expressions.
@@ -218,6 +228,7 @@ def fnmatch_all(names, patterns):
             return False
     return True
 
+
 def prune_dirs(path, root=None, clutter=('.DS_Store', 'Thumbs.db')):
     """If path is an empty directory, then remove it. Recursively remove
     path's ancestry up to root (which is never removed) where there are
@@ -236,7 +247,7 @@ def prune_dirs(path, root=None, clutter=('.DS_Store', 'Thumbs.db')):
         ancestors = []
     elif root in ancestors:
         # Only remove directories below the root.
-        ancestors = ancestors[ancestors.index(root)+1:]
+        ancestors = ancestors[ancestors.index(root) + 1:]
     else:
         # Remove nothing.
         return
@@ -258,6 +269,7 @@ def prune_dirs(path, root=None, clutter=('.DS_Store', 'Thumbs.db')):
         else:
             break
 
+
 def components(path):
     """Return a list of the path components in path. For instance:
 
@@ -281,6 +293,7 @@ def components(path):
 
     return comps
 
+
 def _fsencoding():
     """Get the system's filesystem encoding. On Windows, this is always
     UTF-8 (not MBCS).
@@ -295,6 +308,7 @@ def _fsencoding():
         encoding = 'utf8'
     return encoding
 
+
 def bytestring_path(path):
     """Given a path, which is either a str or a unicode, returns a str
     path (ensuring that we never deal with Unicode pathnames).
@@ -315,6 +329,7 @@ def bytestring_path(path):
     except (UnicodeError, LookupError):
         return path.encode('utf8')
 
+
 def displayable_path(path, separator=u'; '):
     """Attempts to decode a bytestring path to a unicode object for the
     purpose of displaying it to the user. If the `path` argument is a
@@ -333,6 +348,7 @@ def displayable_path(path, separator=u'; '):
     except (UnicodeError, LookupError):
         return path.decode('utf8', 'ignore')
 
+
 def syspath(path, prefix=True):
     """Convert a path for use by the operating system. In particular,
     paths on Windows must receive a magic prefix and must be converted
@@ -356,16 +372,22 @@ def syspath(path, prefix=True):
             encoding = sys.getfilesystemencoding() or sys.getdefaultencoding()
             path = path.decode(encoding, 'replace')
 
-    # Add the magic prefix if it isn't already there
+    # Add the magic prefix if it isn't already there.
+    # http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247.aspx
     if prefix and not path.startswith(WINDOWS_MAGIC_PREFIX):
+        if path.startswith(u'\\\\'):
+            # UNC path. Final path should look like \\?\UNC\...
+            path = u'UNC' + path[1:]
         path = WINDOWS_MAGIC_PREFIX + path
 
     return path
 
+
 def samefile(p1, p2):
     """Safer equality for paths."""
     return shutil._samefile(syspath(p1), syspath(p2))
 
+
 def remove(path, soft=True):
     """Remove the file. If `soft`, then no error will be raised if the
     file does not exist.
@@ -378,6 +400,7 @@ def remove(path, soft=True):
     except (OSError, IOError) as exc:
         raise FilesystemError(exc, 'delete', (path,), traceback.format_exc())
 
+
 def copy(path, dest, replace=False):
     """Copy a plain file. Permissions are not copied. If `dest` already
     exists, raises a FilesystemError unless `replace` is True. Has no
@@ -396,6 +419,7 @@ def copy(path, dest, replace=False):
         raise FilesystemError(exc, 'copy', (path, dest),
                               traceback.format_exc())
 
+
 def move(path, dest, replace=False):
     """Rename a file. `dest` may not be a directory. If `dest` already
     exists, raises an OSError unless `replace` is True. Has no effect if
@@ -424,6 +448,27 @@ def move(path, dest, replace=False):
             raise FilesystemError(exc, 'move', (path, dest),
                                   traceback.format_exc())
 
+
+def link(path, dest, replace=False):
+    """Create a symbolic link from path to `dest`. Raises an OSError if
+    `dest` already exists, unless `replace` is True. Does nothing if
+    `path` == `dest`."""
+    if (samefile(path, dest)):
+        return
+
+    path = syspath(path)
+    dest = syspath(dest)
+    if os.path.exists(dest) and not replace:
+        raise FilesystemError('file exists', 'rename', (path, dest),
+                              traceback.format_exc())
+    try:
+        os.symlink(path, dest)
+    except OSError:
+        raise FilesystemError('Operating system does not support symbolic '
+                              'links.', 'link', (path, dest),
+                              traceback.format_exc())
+
+
 def unique_path(path):
     """Returns a version of ``path`` that does not exist on the
     filesystem. Specifically, if ``path` itself already exists, then
@@ -457,6 +502,8 @@ CHAR_REPLACE = [
     (re.compile(ur'\.$'), u'_'),  # Trailing dots.
     (re.compile(ur'\s+$'), u''),  # Trailing whitespace.
 ]
+
+
 def sanitize_path(path, replacements=None):
     """Takes a path (as a Unicode string) and makes sure that it is
     legal. Returns a new path. Only works with fragments; won't work
@@ -477,6 +524,7 @@ def sanitize_path(path, replacements=None):
         comps[i] = comp
     return os.path.join(*comps)
 
+
 def truncate_path(path, length=MAX_FILENAME_LENGTH):
     """Given a bytestring path or a Unicode path fragment, truncate the
     components to a legal length. In the last component, the extension
@@ -493,6 +541,7 @@ def truncate_path(path, length=MAX_FILENAME_LENGTH):
 
     return os.path.join(*out)
 
+
 def str2bool(value):
     """Returns a boolean reflecting a human-entered string."""
     if value.lower() in ('yes', '1', 'true', 't', 'y'):
@@ -500,6 +549,7 @@ def str2bool(value):
     else:
         return False
 
+
 def as_string(value):
     """Convert a value to a Unicode object for matching with a query.
     None becomes the empty string. Bytestrings are silently decoded.
@@ -513,6 +563,7 @@ def as_string(value):
     else:
         return unicode(value)
 
+
 def levenshtein(s1, s2):
     """A nice DP edit distance implementation from Wikibooks:
     http://en.wikibooks.org/wiki/Algorithm_implementation/Strings/
@@ -535,6 +586,7 @@ def levenshtein(s1, s2):
 
     return previous_row[-1]
 
+
 def plurality(objs):
     """Given a sequence of comparable objects, returns the object that
     is most common in the set and the frequency of that object. The
@@ -558,6 +610,7 @@ def plurality(objs):
 
     return res, max_freq
 
+
 def cpu_count():
     """Return the number of hardware thread contexts (cores or SMT
     threads) in the system.
@@ -571,7 +624,7 @@ def cpu_count():
             num = 0
     elif sys.platform == 'darwin':
         try:
-            num = int(os.popen('sysctl -n hw.ncpu').read())
+            num = int(command_output(['sysctl', '-n', 'hw.ncpu']))
         except ValueError:
             num = 0
     else:
@@ -584,23 +637,38 @@ def cpu_count():
     else:
         return 1
 
-def command_output(cmd):
+
-    """Wraps the `subprocess` module to invoke a command (given as a
+def command_output(cmd, shell=False):
-    list of arguments starting with the command name) and collect
+    """Runs the command and returns its output after it has exited.
-    stdout. The stderr stream is ignored. May raise
+
-    `subprocess.CalledProcessError` or an `OSError`.
+    ``cmd`` is a list of arguments starting with the command names.  If
+    ``shell`` is true, ``cmd`` is assumed to be a string and passed to a
+    shell to execute.
+
+    If the process exits with a non-zero return code
+    ``subprocess.CalledProcessError`` is raised. May also raise
+    ``OSError``.
 
     This replaces `subprocess.check_output`, which isn't available in
     Python 2.6 and which can have problems if lots of output is sent to
     stderr.
     """
-    with open(os.devnull, 'w') as devnull:
+    proc = subprocess.Popen(
-        proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=devnull)
+        cmd,
-        stdout, _ = proc.communicate()
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+        close_fds=platform.system() != 'Windows',
+        shell=shell
+    )
+    stdout, stderr = proc.communicate()
     if proc.returncode:
-        raise subprocess.CalledProcessError(proc.returncode, cmd)
+        raise subprocess.CalledProcessError(
+            returncode=proc.returncode,
+            cmd=' '.join(cmd),
+        )
     return stdout
 
+
 def max_filename_length(path, limit=MAX_FILENAME_LENGTH):
     """Attempt to determine the maximum filename length for the
     filesystem containing `path`. If the value is greater than `limit`,

lib/beets/util/artresizer.py

@@ -1,5 +1,5 @@
 # This file is part of beets.
-# Copyright 2013, Fabrice Laporte
+# Copyright 2014, Fabrice Laporte
 #
 # Permission is hereby granted, free of charge, to any person obtaining
 # a copy of this software and associated documentation files (the
@@ -18,6 +18,7 @@ public resizing proxy if neither is available.
 import urllib
 import subprocess
 import os
+import re
 from tempfile import NamedTemporaryFile
 import logging
 from beets import util
@@ -37,7 +38,7 @@ def resize_url(url, maxwidth):
     maxwidth (preserving aspect ratio).
     """
     return '{0}?{1}'.format(PROXY_URL, urllib.urlencode({
-        'url': url.replace('http://',''),
+        'url': url.replace('http://', ''),
         'w': str(maxwidth),
     }))
 
@@ -76,7 +77,7 @@ def pil_resize(maxwidth, path_in, path_out=None):
 
 def im_resize(maxwidth, path_in, path_out=None):
     """Resize using ImageMagick's ``convert`` tool.
-    tool. Return the output path of resized image.
+    Return the output path of resized image.
     """
     path_out = path_out or temp_file_for(path_in)
     log.debug(u'artresizer: ImageMagick resizing {0} to {1}'.format(
@@ -132,8 +133,9 @@ class ArtResizer(object):
         """Create a resizer object for the given method or, if none is
         specified, with an inferred method.
         """
-        self.method = method or self._guess_method()
+        self.method = self._check_method(method)
         log.debug(u"artresizer: method is {0}".format(self.method))
+        self.can_compare = self._can_compare()
 
     def resize(self, maxwidth, path_in, path_out=None):
         """Manipulate an image file according to the method, returning a
@@ -141,7 +143,7 @@ class ArtResizer(object):
         temporary file. For WEBPROXY, returns `path_in` unmodified.
         """
         if self.local:
-            func = BACKEND_FUNCS[self.method]
+            func = BACKEND_FUNCS[self.method[0]]
             return func(maxwidth, path_in, path_out)
         else:
             return path_in
@@ -159,30 +161,51 @@ class ArtResizer(object):
     @property
     def local(self):
         """A boolean indicating whether the resizing method is performed
-        locally (i.e., PIL or IMAGEMAGICK).
+        locally (i.e., PIL or ImageMagick).
         """
-        return self.method in BACKEND_FUNCS
+        return self.method[0] in BACKEND_FUNCS
+
+    def _can_compare(self):
+        """A boolean indicating whether image comparison is available"""
+
+        return self.method[0] == IMAGEMAGICK and self.method[1] > (6, 8, 7)
 
     @staticmethod
-    def _guess_method():
+    def _check_method(method=None):
-        """Determine which resizing method to use. Returns PIL,
+        """A tuple indicating whether current method is available and its
-        IMAGEMAGICK, or WEBPROXY depending on available dependencies.
+        version. If no method is given, it returns a supported one.
         """
-        # Try importing PIL.
+        # Guess available method
-        try:
+        if not method:
-            __import__('PIL', fromlist=['Image'])
+            for m in [IMAGEMAGICK, PIL]:
-            return PIL
+                _, version = ArtResizer._check_method(m)
-        except ImportError:
+                if version:
-            pass
+                    return (m, version)
+            return (WEBPROXY, (0))
 
-        # Try invoking ImageMagick's "convert".
+        if method == IMAGEMAGICK:
-        try:
-            out = util.command_output(['convert', '--version'])
-            if 'imagemagick' in out.lower():
-                # system32/convert.exe may be interfering
-                return IMAGEMAGICK
-        except (subprocess.CalledProcessError, OSError):
-            pass
 
-        # Fall back to Web proxy method.
+            # Try invoking ImageMagick's "convert".
-        return WEBPROXY
+            try:
+                out = util.command_output(['identify', '--version'])
+
+                if 'imagemagick' in out.lower():
+                    pattern = r".+ (\d+)\.(\d+)\.(\d+).*"
+                    match = re.search(pattern, out)
+                    if match:
+                        return (IMAGEMAGICK,
+                                (int(match.group(1)),
+                                 int(match.group(2)),
+                                 int(match.group(3))))
+                    return (IMAGEMAGICK, (0))
+
+            except (subprocess.CalledProcessError, OSError):
+                return (IMAGEMAGICK, None)
+
+        if method == PIL:
+            # Try importing PIL.
+            try:
+                __import__('PIL', fromlist=['Image'])
+                return (PIL, (0))
+            except ImportError:
+                return (PIL, None)

lib/beets/util/bluelet.py

@@ -38,6 +38,7 @@ class Event(object):
     """
     pass
 
+
 class WaitableEvent(Event):
     """A waitable event is one encapsulating an action that can be
     waited for using a select() call. That is, it's an event with an
@@ -57,21 +58,25 @@ class WaitableEvent(Event):
         """
         pass
 
+
 class ValueEvent(Event):
     """An event that does nothing but return a fixed value."""
     def __init__(self, value):
         self.value = value
 
+
 class ExceptionEvent(Event):
     """Raise an exception at the yield point. Used internally."""
     def __init__(self, exc_info):
         self.exc_info = exc_info
 
+
 class SpawnEvent(Event):
     """Add a new coroutine thread to the scheduler."""
     def __init__(self, coro):
         self.spawned = coro
 
+
 class JoinEvent(Event):
     """Suspend the thread until the specified child thread has
     completed.
@@ -79,11 +84,13 @@ class JoinEvent(Event):
     def __init__(self, child):
         self.child = child
 
+
 class KillEvent(Event):
     """Unschedule a child thread."""
     def __init__(self, child):
         self.child = child
 
+
 class DelegationEvent(Event):
     """Suspend execution of the current thread, start a new thread and,
     once the child thread finished, return control to the parent
@@ -92,6 +99,7 @@ class DelegationEvent(Event):
     def __init__(self, coro):
         self.spawned = coro
 
+
 class ReturnEvent(Event):
     """Return a value the current thread's delegator at the point of
     delegation. Ends the current (delegate) thread.
@@ -99,6 +107,7 @@ class ReturnEvent(Event):
     def __init__(self, value):
         self.value = value
 
+
 class SleepEvent(WaitableEvent):
     """Suspend the thread for a given duration.
     """
@@ -108,6 +117,7 @@ class SleepEvent(WaitableEvent):
     def time_left(self):
         return max(self.wakeup_time - time.time(), 0.0)
 
+
 class ReadEvent(WaitableEvent):
     """Reads from a file-like object."""
     def __init__(self, fd, bufsize):
@@ -120,6 +130,7 @@ class ReadEvent(WaitableEvent):
     def fire(self):
         return self.fd.read(self.bufsize)
 
+
 class WriteEvent(WaitableEvent):
     """Writes to a file-like object."""
     def __init__(self, fd, data):
@@ -192,15 +203,19 @@ def _event_select(events):
 
     return ready_events
 
+
 class ThreadException(Exception):
     def __init__(self, coro, exc_info):
         self.coro = coro
         self.exc_info = exc_info
+
     def reraise(self):
         _reraise(self.exc_info[0], self.exc_info[1], self.exc_info[2])
 
+
 SUSPENDED = Event()  # Special sentinel placeholder for suspended threads.
 
+
 class Delegated(Event):
     """Placeholder indicating that a thread has delegated execution to a
     different thread.
@@ -208,6 +223,7 @@ class Delegated(Event):
     def __init__(self, child):
         self.child = child
 
+
 def run(root_coro):
     """Schedules a coroutine, running it to completion. This
     encapsulates the Bluelet scheduler, which the root coroutine can
@@ -329,7 +345,7 @@ def run(root_coro):
                     break
 
             # Wait and fire.
-            event2coro = dict((v,k) for k,v in threads.items())
+            event2coro = dict((v, k) for k, v in threads.items())
             for event in _event_select(threads.values()):
                 # Run the IO operation, but catch socket errors.
                 try:
@@ -378,6 +394,7 @@ def run(root_coro):
 class SocketClosedError(Exception):
     pass
 
+
 class Listener(object):
     """A socket wrapper object for listening sockets.
     """
@@ -407,6 +424,7 @@ class Listener(object):
         self._closed = True
         self.sock.close()
 
+
 class Connection(object):
     """A socket wrapper object for connected sockets.
     """
@@ -468,6 +486,7 @@ class Connection(object):
                 yield ReturnEvent(line)
                 break
 
+
 class AcceptEvent(WaitableEvent):
     """An event for Listener objects (listening sockets) that suspends
     execution until the socket gets a connection.
@@ -482,6 +501,7 @@ class AcceptEvent(WaitableEvent):
         sock, addr = self.listener.sock.accept()
         return Connection(sock, addr)
 
+
 class ReceiveEvent(WaitableEvent):
     """An event for Connection objects (connected sockets) for
     asynchronously reading data.
@@ -496,6 +516,7 @@ class ReceiveEvent(WaitableEvent):
     def fire(self):
         return self.conn.sock.recv(self.bufsize)
 
+
 class SendEvent(WaitableEvent):
     """An event for Connection objects (connected sockets) for
     asynchronously writing data.
@@ -523,6 +544,7 @@ def null():
     """
     return ValueEvent(None)
 
+
 def spawn(coro):
     """Event: add another coroutine to the scheduler. Both the parent
     and child coroutines run concurrently.
@@ -531,6 +553,7 @@ def spawn(coro):
         raise ValueError('%s is not a coroutine' % str(coro))
     return SpawnEvent(coro)
 
+
 def call(coro):
     """Event: delegate to another coroutine. The current coroutine
     is resumed once the sub-coroutine finishes. If the sub-coroutine
@@ -540,12 +563,14 @@ def call(coro):
         raise ValueError('%s is not a coroutine' % str(coro))
     return DelegationEvent(coro)
 
+
 def end(value=None):
     """Event: ends the coroutine and returns a value to its
     delegator.
     """
     return ReturnEvent(value)
 
+
 def read(fd, bufsize=None):
     """Event: read from a file descriptor asynchronously."""
     if bufsize is None:
@@ -563,10 +588,12 @@ def read(fd, bufsize=None):
     else:
         return ReadEvent(fd, bufsize)
 
+
 def write(fd, data):
     """Event: write to a file descriptor asynchronously."""
     return WriteEvent(fd, data)
 
+
 def connect(host, port):
     """Event: connect to a network address and return a Connection
     object for communicating on the socket.
@@ -575,17 +602,20 @@ def connect(host, port):
     sock = socket.create_connection(addr)
     return ValueEvent(Connection(sock, addr))
 
+
 def sleep(duration):
     """Event: suspend the thread for ``duration`` seconds.
     """
     return SleepEvent(duration)
 
+
 def join(coro):
     """Suspend the thread until another, previously `spawn`ed thread
     completes.
     """
     return JoinEvent(coro)
 
+
 def kill(coro):
     """Halt the execution of a different `spawn`ed thread.
     """

lib/beets/util/confit.py

@@ -21,6 +21,8 @@ import pkgutil
 import sys
 import yaml
 import types
+import collections
+import re
 try:
     from collections import OrderedDict
 except ImportError:
@@ -47,6 +49,7 @@ BASESTRING = str if PY3 else basestring
 NUMERIC_TYPES = (int, float) if PY3 else (int, float, long)
 TYPE_TYPES = (type,) if PY3 else (type, types.ClassType)
 
+
 def iter_first(sequence):
     """Get the first element from an iterable or raise a ValueError if
     the iterator generates no values.
@@ -67,16 +70,25 @@ class ConfigError(Exception):
     """Base class for exceptions raised when querying a configuration.
     """
 
+
 class NotFoundError(ConfigError):
     """A requested value could not be found in the configuration trees.
     """
 
-class ConfigTypeError(ConfigError, TypeError):
+
+class ConfigValueError(ConfigError):
+    """The value in the configuration is illegal."""
+
+
+class ConfigTypeError(ConfigValueError):
     """The value in the configuration did not match the expected type.
     """
 
-class ConfigValueError(ConfigError, ValueError):
+
-    """The value in the configuration is illegal."""
+class ConfigTemplateError(ConfigError):
+    """Base class for exceptions raised because of an invalid template.
+    """
+
 
 class ConfigReadError(ConfigError):
     """A configuration file could not be read."""
@@ -132,6 +144,7 @@ class ConfigSource(dict):
         else:
             raise TypeError('source value must be a dict')
 
+
 class ConfigView(object):
     """A configuration "view" is a query into a program's configuration
     data. A view represents a hypothetical location in the configuration
@@ -207,6 +220,9 @@ class ConfigView(object):
         """
         self.set({key: value})
 
+    def __contains__(self, key):
+        return self[key].exists()
+
     def set_args(self, namespace):
         """Overlay parsed command-line arguments, generated by a library
         like argparse or optparse, onto this view's value.
@@ -310,98 +326,6 @@ class ConfigView(object):
 
     # Validation and conversion.
 
-    def get(self, typ=None):
-        """Returns the canonical value for the view, checked against the
-        passed-in type. If the value is not an instance of the given
-        type, a ConfigTypeError is raised. May also raise a
-        NotFoundError.
-        """
-        value, _ = self.first()
-
-        if typ is not None:
-            if not isinstance(typ, TYPE_TYPES):
-                raise TypeError('argument to get() must be a type')
-
-            if not isinstance(value, typ):
-                raise ConfigTypeError(
-                    "{0} must be of type {1}, not {2}".format(
-                        self.name, typ.__name__, type(value).__name__
-                    )
-                )
-
-        return value
-
-    def as_filename(self):
-        """Get a string as a normalized as an absolute, tilde-free path.
-
-        Relative paths are relative to the configuration directory (see
-        the `config_dir` method) if they come from a file. Otherwise,
-        they are relative to the current working directory. This helps
-        attain the expected behavior when using command-line options.
-        """
-        path, source = self.first()
-        if not isinstance(path, BASESTRING):
-            raise ConfigTypeError('{0} must be a filename, not {1}'.format(
-                self.name, type(path).__name__
-            ))
-        path = os.path.expanduser(STRING(path))
-
-        if not os.path.isabs(path) and source.filename:
-            # From defaults: relative to the app's directory.
-            path = os.path.join(self.root().config_dir(), path)
-
-        return os.path.abspath(path)
-
-    def as_choice(self, choices):
-        """Ensure that the value is among a collection of choices and
-        return it. If `choices` is a dictionary, then return the
-        corresponding value rather than the value itself (the key).
-        """
-        value = self.get()
-
-        if value not in choices:
-            raise ConfigValueError(
-                '{0} must be one of {1}, not {2}'.format(
-                    self.name, repr(list(choices)), repr(value)
-                )
-            )
-
-        if isinstance(choices, dict):
-            return choices[value]
-        else:
-            return value
-
-    def as_number(self):
-        """Ensure that a value is of numeric type."""
-        value = self.get()
-        if isinstance(value, NUMERIC_TYPES):
-            return value
-        raise ConfigTypeError(
-            '{0} must be numeric, not {1}'.format(
-                self.name, type(value).__name__
-            )
-        )
-
-    def as_str_seq(self):
-        """Get the value as a list of strings. The underlying configured
-        value can be a sequence or a single string. In the latter case,
-        the string is treated as a white-space separated list of words.
-        """
-        value = self.get()
-        if isinstance(value, bytes):
-            value = value.decode('utf8', 'ignore')
-
-        if isinstance(value, STRING):
-            return value.split()
-        else:
-            try:
-                return list(value)
-            except TypeError:
-                raise ConfigTypeError(
-                    '{0} must be a whitespace-separated string or '
-                    'a list'.format(self.name)
-                )
-
     def flatten(self):
         """Create a hierarchy of OrderedDicts containing the data from
         this view, recursively reifying all views to get their
@@ -415,6 +339,35 @@ class ConfigView(object):
                 od[key] = view.get()
         return od
 
+    def get(self, template=None):
+        """Retrieve the value for this view according to the template.
+
+        The `template` against which the values are checked can be
+        anything convertible to a `Template` using `as_template`. This
+        means you can pass in a default integer or string value, for
+        example, or a type to just check that something matches the type
+        you expect.
+
+        May raise a `ConfigValueError` (or its subclass,
+        `ConfigTypeError`) or a `NotFoundError` when the configuration
+        doesn't satisfy the template.
+        """
+        return as_template(template).value(self, template)
+
+    # Old validation methods (deprecated).
+
+    def as_filename(self):
+        return self.get(Filename())
+
+    def as_choice(self, choices):
+        return self.get(Choice(choices))
+
+    def as_number(self):
+        return self.get(Number())
+
+    def as_str_seq(self):
+        return self.get(StrSeq())
+
 
 class RootView(ConfigView):
     """The base of a view hierarchy. This view keeps track of the
@@ -518,6 +471,7 @@ def _package_path(name):
 
     return os.path.dirname(os.path.abspath(filepath))
 
+
 def config_dirs():
     """Return a platform-specific list of candidates for user
     configuration directories on the system.
@@ -606,10 +560,12 @@ class Loader(yaml.SafeLoader):
         plain = super(Loader, self).check_plain()
         return plain or self.peek() == '%'
 
+
 Loader.add_constructor('tag:yaml.org,2002:str', Loader._construct_unicode)
 Loader.add_constructor('tag:yaml.org,2002:map', Loader.construct_yaml_map)
 Loader.add_constructor('tag:yaml.org,2002:omap', Loader.construct_yaml_map)
 
+
 def load_yaml(filename):
     """Read a YAML document from a file. If the file cannot be read or
     parsed, a ConfigReadError is raised.
@@ -679,11 +635,13 @@ class Dumper(yaml.SafeDumper):
         """
         return self.represent_scalar('tag:yaml.org,2002:null', '')
 
+
 Dumper.add_representer(OrderedDict, Dumper.represent_dict)
 Dumper.add_representer(bool, Dumper.represent_bool)
 Dumper.add_representer(type(None), Dumper.represent_none)
 Dumper.add_representer(list, Dumper.represent_list)
 
+
 def restore_yaml_comments(data, default_data):
     """Scan default_data for comments (we include empty lines in our
     definition of comments) and place them before the same keys in data.
@@ -898,3 +856,426 @@ class LazyConfig(Configuration):
         del self.sources[:]
         self._lazy_suffix = []
         self._lazy_prefix = []
+
+
+# "Validated" configuration views: experimental!
+
+
+REQUIRED = object()
+"""A sentinel indicating that there is no default value and an exception
+should be raised when the value is missing.
+"""
+
+
+class Template(object):
+    """A value template for configuration fields.
+
+    The template works like a type and instructs Confit about how to
+    interpret a deserialized YAML value. This includes type conversions,
+    providing a default value, and validating for errors. For example, a
+    filepath type might expand tildes and check that the file exists.
+    """
+    def __init__(self, default=REQUIRED):
+        """Create a template with a given default value.
+
+        If `default` is the sentinel `REQUIRED` (as it is by default),
+        then an error will be raised when a value is missing. Otherwise,
+        missing values will instead return `default`.
+        """
+        self.default = default
+
+    def __call__(self, view):
+        """Invoking a template on a view gets the view's value according
+        to the template.
+        """
+        return self.value(view, self)
+
+    def value(self, view, template=None):
+        """Get the value for a `ConfigView`.
+
+        May raise a `NotFoundError` if the value is missing (and the
+        template requires it) or a `ConfigValueError` for invalid values.
+        """
+        if view.exists():
+            value, _ = view.first()
+            return self.convert(value, view)
+        elif self.default is REQUIRED:
+            # Missing required value. This is an error.
+            raise NotFoundError("{0} not found".format(view.name))
+        else:
+            # Missing value, but not required.
+            return self.default
+
+    def convert(self, value, view):
+        """Convert the YAML-deserialized value to a value of the desired
+        type.
+
+        Subclasses should override this to provide useful conversions.
+        May raise a `ConfigValueError` when the configuration is wrong.
+        """
+        # Default implementation does no conversion.
+        return value
+
+    def fail(self, message, view, type_error=False):
+        """Raise an exception indicating that a value cannot be
+        accepted.
+
+        `type_error` indicates whether the error is due to a type
+        mismatch rather than a malformed value. In this case, a more
+        specific exception is raised.
+        """
+        exc_class = ConfigTypeError if type_error else ConfigValueError
+        raise exc_class(
+            '{0}: {1}'.format(view.name, message)
+        )
+
+    def __repr__(self):
+        return '{0}({1})'.format(
+            type(self).__name__,
+            '' if self.default is REQUIRED else repr(self.default),
+        )
+
+
+class Integer(Template):
+    """An integer configuration value template.
+    """
+    def convert(self, value, view):
+        """Check that the value is an integer. Floats are rounded.
+        """
+        if isinstance(value, int):
+            return value
+        elif isinstance(value, float):
+            return int(value)
+        else:
+            self.fail('must be a number', view, True)
+
+
+class Number(Template):
+    """A numeric type: either an integer or a floating-point number.
+    """
+    def convert(self, value, view):
+        """Check that the value is an int or a float.
+        """
+        if isinstance(value, NUMERIC_TYPES):
+            return value
+        else:
+            self.fail(
+                'must be numeric, not {0}'.format(type(value).__name__),
+                view,
+                True
+            )
+
+
+class MappingTemplate(Template):
+    """A template that uses a dictionary to specify other types for the
+    values for a set of keys and produce a validated `AttrDict`.
+    """
+    def __init__(self, mapping):
+        """Create a template according to a dict (mapping). The
+        mapping's values should themselves either be Types or
+        convertible to Types.
+        """
+        subtemplates = {}
+        for key, typ in mapping.items():
+            subtemplates[key] = as_template(typ)
+        self.subtemplates = subtemplates
+
+    def value(self, view, template=None):
+        """Get a dict with the same keys as the template and values
+        validated according to the value types.
+        """
+        out = AttrDict()
+        for key, typ in self.subtemplates.items():
+            out[key] = typ.value(view[key], self)
+        return out
+
+    def __repr__(self):
+        return 'MappingTemplate({0})'.format(repr(self.subtemplates))
+
+
+class String(Template):
+    """A string configuration value template.
+    """
+    def __init__(self, default=REQUIRED, pattern=None):
+        """Create a template with the added optional `pattern` argument,
+        a regular expression string that the value should match.
+        """
+        super(String, self).__init__(default)
+        self.pattern = pattern
+        if pattern:
+            self.regex = re.compile(pattern)
+
+    def convert(self, value, view):
+        """Check that the value is a string and matches the pattern.
+        """
+        if isinstance(value, BASESTRING):
+            if self.pattern and not self.regex.match(value):
+                self.fail(
+                    "must match the pattern {0}".format(self.pattern),
+                    view
+                )
+            return value
+        else:
+            self.fail('must be a string', view, True)
+
+
+class Choice(Template):
+    """A template that permits values from a sequence of choices.
+    """
+    def __init__(self, choices):
+        """Create a template that validates any of the values from the
+        iterable `choices`.
+
+        If `choices` is a map, then the corresponding value is emitted.
+        Otherwise, the value itself is emitted.
+        """
+        self.choices = choices
+
+    def convert(self, value, view):
+        """Ensure that the value is among the choices (and remap if the
+        choices are a mapping).
+        """
+        if value not in self.choices:
+            self.fail(
+                'must be one of {0}, not {1}'.format(
+                    repr(list(self.choices)), repr(value)
+                ),
+                view
+            )
+
+        if isinstance(self.choices, collections.Mapping):
+            return self.choices[value]
+        else:
+            return value
+
+    def __repr__(self):
+        return 'Choice({0!r})'.format(self.choices)
+
+
+class StrSeq(Template):
+    """A template for values that are lists of strings.
+
+    Validates both actual YAML string lists and single strings. Strings
+    can optionally be split on whitespace.
+    """
+    def __init__(self, split=True):
+        """Create a new template.
+
+        `split` indicates whether, when the underlying value is a single
+        string, it should be split on whitespace. Otherwise, the
+        resulting value is a list containing a single string.
+        """
+        super(StrSeq, self).__init__()
+        self.split = split
+
+    def convert(self, value, view):
+        if isinstance(value, bytes):
+            value = value.decode('utf8', 'ignore')
+
+        if isinstance(value, STRING):
+            if self.split:
+                return value.split()
+            else:
+                return [value]
+
+        try:
+            value = list(value)
+        except TypeError:
+            self.fail('must be a whitespace-separated string or a list',
+                      view, True)
+
+        def convert(x):
+            if isinstance(x, unicode):
+                return x
+            elif isinstance(x, BASESTRING):
+                return x.decode('utf8', 'ignore')
+            else:
+                self.fail('must be a list of strings', view, True)
+        return map(convert, value)
+
+
+class Filename(Template):
+    """A template that validates strings as filenames.
+
+    Filenames are returned as absolute, tilde-free paths.
+
+    Relative paths are relative to the template's `cwd` argument
+    when it is specified, then the configuration directory (see
+    the `config_dir` method) if they come from a file. Otherwise,
+    they are relative to the current working directory. This helps
+    attain the expected behavior when using command-line options.
+    """
+    def __init__(self, default=REQUIRED, cwd=None, relative_to=None,
+                 in_app_dir=False):
+        """ `relative_to` is the name of a sibling value that is
+        being validated at the same time.
+
+        `in_app_dir` indicates whether the path should be resolved
+        inside the application's config directory (even when the setting
+        does not come from a file).
+        """
+        super(Filename, self).__init__(default)
+        self.cwd = cwd
+        self.relative_to = relative_to
+        self.in_app_dir = in_app_dir
+
+    def __repr__(self):
+        args = []
+
+        if self.default is not REQUIRED:
+            args.append(repr(self.default))
+
+        if self.cwd is not None:
+            args.append('cwd=' + repr(self.cwd))
+
+        if self.relative_to is not None:
+            args.append('relative_to=' + repr(self.relative_to))
+
+        if self.in_app_dir:
+            args.append('in_app_dir=True')
+
+        return 'Filename({0})'.format(', '.join(args))
+
+    def resolve_relative_to(self, view, template):
+        if not isinstance(template, (collections.Mapping, MappingTemplate)):
+            # disallow config.get(Filename(relative_to='foo'))
+            raise ConfigTemplateError(
+                'relative_to may only be used when getting multiple values.'
+            )
+
+        elif self.relative_to == view.key:
+            raise ConfigTemplateError(
+                '{0} is relative to itself'.format(view.name)
+            )
+
+        elif self.relative_to not in view.parent.keys():
+            # self.relative_to is not in the config
+            self.fail(
+                (
+                    'needs sibling value "{0}" to expand relative path'
+                ).format(self.relative_to),
+                view
+            )
+
+        old_template = {}
+        old_template.update(template.subtemplates)
+
+        # save time by skipping MappingTemplate's init loop
+        next_template = MappingTemplate({})
+        next_relative = self.relative_to
+
+        # gather all the needed templates and nothing else
+        while next_relative is not None:
+            try:
+                # pop to avoid infinite loop because of recursive
+                # relative paths
+                rel_to_template = old_template.pop(next_relative)
+            except KeyError:
+                if next_relative in template.subtemplates:
+                    # we encountered this config key previously
+                    raise ConfigTemplateError((
+                        '{0} and {1} are recursively relative'
+                    ).format(view.name, self.relative_to))
+                else:
+                    raise ConfigTemplateError((
+                        'missing template for {0}, needed to expand {1}\'s' +
+                        'relative path'
+                    ).format(self.relative_to, view.name))
+
+            next_template.subtemplates[next_relative] = rel_to_template
+            next_relative = rel_to_template.relative_to
+
+        return view.parent.get(next_template)[self.relative_to]
+
+    def value(self, view, template=None):
+        path, source = view.first()
+        if not isinstance(path, BASESTRING):
+            self.fail(
+                'must be a filename, not {0}'.format(type(path).__name__),
+                view,
+                True
+            )
+        path = os.path.expanduser(STRING(path))
+
+        if not os.path.isabs(path):
+            if self.cwd is not None:
+                # relative to the template's argument
+                path = os.path.join(self.cwd, path)
+
+            elif self.relative_to is not None:
+                path = os.path.join(
+                    self.resolve_relative_to(view, template),
+                    path,
+                )
+
+            elif source.filename or self.in_app_dir:
+                # From defaults: relative to the app's directory.
+                path = os.path.join(view.root().config_dir(), path)
+
+        return os.path.abspath(path)
+
+
+class TypeTemplate(Template):
+    """A simple template that checks that a value is an instance of a
+    desired Python type.
+    """
+    def __init__(self, typ, default=REQUIRED):
+        """Create a template that checks that the value is an instance
+        of `typ`.
+        """
+        super(TypeTemplate, self).__init__(default)
+        self.typ = typ
+
+    def convert(self, value, view):
+        if not isinstance(value, self.typ):
+            self.fail(
+                'must be a {0}, not {1}'.format(
+                    self.typ.__name__,
+                    type(value).__name__,
+                ),
+                view,
+                True
+            )
+        return value
+
+
+class AttrDict(dict):
+    """A `dict` subclass that can be accessed via attributes (dot
+    notation) for convenience.
+    """
+    def __getattr__(self, key):
+        if key in self:
+            return self[key]
+        else:
+            raise AttributeError(key)
+
+
+def as_template(value):
+    """Convert a simple "shorthand" Python value to a `Template`.
+    """
+    if isinstance(value, Template):
+        # If it's already a Template, pass it through.
+        return value
+    elif isinstance(value, collections.Mapping):
+        # Dictionaries work as templates.
+        return MappingTemplate(value)
+    elif value is int:
+        return Integer()
+    elif isinstance(value, int):
+        return Integer(value)
+    elif isinstance(value, type) and issubclass(value, BASESTRING):
+        return String()
+    elif isinstance(value, BASESTRING):
+        return String(value)
+    elif value is float:
+        return Number()
+    elif value is None:
+        return Template()
+    elif value is dict:
+        return TypeTemplate(collections.Mapping)
+    elif value is list:
+        return TypeTemplate(collections.Sequence)
+    elif isinstance(value, type):
+        return TypeTemplate(value)
+    else:
+        raise ValueError('cannot convert to template: {0!r}'.format(value))

lib/beets/util/enumeration.py

@@ -12,167 +12,29 @@
 # The above copyright notice and this permission notice shall be
 # included in all copies or substantial portions of the Software.
 
-"""A metaclass for enumerated types that really are types.
+from enum import Enum
 
-You can create enumerations with `enum(values, [name])` and they work
-how you would expect them to.
 
-    >>> from enumeration import enum
+class OrderedEnum(Enum):
-    >>> Direction = enum('north east south west', name='Direction')
-    >>> Direction.west
-    Direction.west
-    >>> Direction.west == Direction.west
-    True
-    >>> Direction.west == Direction.east
-    False
-    >>> isinstance(Direction.west, Direction)
-    True
-    >>> Direction[3]
-    Direction.west
-    >>> Direction['west']
-    Direction.west
-    >>> Direction.west.name
-    'west'
-    >>> Direction.north < Direction.west
-    True
-
-Enumerations are classes; their instances represent the possible values
-of the enumeration. Because Python classes must have names, you may
-provide a `name` parameter to `enum`; if you don't, a meaningless one
-will be chosen for you.
-"""
-import random
-
-class Enumeration(type):
-    """A metaclass whose classes are enumerations.
-
-    The `values` attribute of the class is used to populate the
-    enumeration. Values may either be a list of enumerated names or a
-    string containing a space-separated list of names. When the class
-    is created, it is instantiated for each name value in `values`.
-    Each such instance is the name of the enumerated item as the sole
-    argument.
-
-    The `Enumerated` class is a good choice for a superclass.
     """
-
+    An Enum subclass that allows comparison of members.
-    def __init__(cls, name, bases, dic):
-        super(Enumeration, cls).__init__(name, bases, dic)
-
-        if 'values' not in dic:
-            # Do nothing if no values are provided (i.e., with
-            # Enumerated itself).
-            return
-
-        # May be called with a single string, in which case we split on
-        # whitespace for convenience.
-        values = dic['values']
-        if isinstance(values, basestring):
-            values = values.split()
-
-        # Create the Enumerated instances for each value. We have to use
-        # super's __setattr__ here because we disallow setattr below.
-        super(Enumeration, cls).__setattr__('_items_dict', {})
-        super(Enumeration, cls).__setattr__('_items_list', [])
-        for value in values:
-            item = cls(value, len(cls._items_list))
-            cls._items_dict[value] = item
-            cls._items_list.append(item)
-
-    def __getattr__(cls, key):
-        try:
-            return cls._items_dict[key]
-        except KeyError:
-            raise AttributeError("enumeration '" + cls.__name__ +
-                                 "' has no item '" + key + "'")
-
-    def __setattr__(cls, key, val):
-        raise TypeError("enumerations do not support attribute assignment")
-
-    def __getitem__(cls, key):
-        if isinstance(key, int):
-            return cls._items_list[key]
-        else:
-            return getattr(cls, key)
-
-    def __len__(cls):
-        return len(cls._items_list)
-
-    def __iter__(cls):
-        return iter(cls._items_list)
-
-    def __nonzero__(cls):
-        # Ensures that __len__ doesn't get called before __init__ by
-        # pydoc.
-        return True
-
-class Enumerated(object):
-    """An item in an enumeration.
-
-    Contains instance methods inherited by enumerated objects. The
-    metaclass is preset to `Enumeration` for your convenience.
-
-    Instance attributes:
-    name -- The name of the item.
-    index -- The index of the item in its enumeration.
-
-        >>> from enumeration import Enumerated
-        >>> class Garment(Enumerated):
-        ...     values = 'hat glove belt poncho lederhosen suspenders'
-        ...     def wear(self):
-        ...         print('now wearing a ' + self.name)
-        ...
-        >>> Garment.poncho.wear()
-        now wearing a poncho
     """
+    def __ge__(self, other):
+        if self.__class__ is other.__class__:
+            return self.value >= other.value
+        return NotImplemented
 
-    __metaclass__ = Enumeration
+    def __gt__(self, other):
+        if self.__class__ is other.__class__:
+            return self.value > other.value
+        return NotImplemented
 
-    def __init__(self, name, index):
+    def __le__(self, other):
-        self.name = name
+        if self.__class__ is other.__class__:
-        self.index = index
+            return self.value <= other.value
+        return NotImplemented
 
-    def __str__(self):
+    def __lt__(self, other):
-        return type(self).__name__ + '.' + self.name
+        if self.__class__ is other.__class__:
-
+            return self.value < other.value
-    def __repr__(self):
+        return NotImplemented
-        return str(self)
-
-    def __cmp__(self, other):
-        if type(self) is type(other):
-            # Note that we're assuming that the items are direct
-            # instances of the same Enumeration (i.e., no fancy
-            # subclassing), which is probably okay.
-            return cmp(self.index, other.index)
-        else:
-            return NotImplemented
-
-def enum(*values, **kwargs):
-    """Shorthand for creating a new Enumeration class.
-
-    Call with enumeration values as a list, a space-delimited string, or
-    just an argument list. To give the class a name, pass it as the
-    `name` keyword argument. Otherwise, a name will be chosen for you.
-
-    The following are all equivalent:
-
-        enum('pinkie ring middle index thumb')
-        enum('pinkie', 'ring', 'middle', 'index', 'thumb')
-        enum(['pinkie', 'ring', 'middle', 'index', 'thumb'])
-    """
-
-    if ('name' not in kwargs) or kwargs['name'] is None:
-        # Create a probably-unique name. It doesn't really have to be
-        # unique, but getting distinct names each time helps with
-        # identification in debugging.
-        name = 'Enumeration' + hex(random.randint(0,0xfffffff))[2:].upper()
-    else:
-        name = kwargs['name']
-
-    if len(values) == 1:
-        # If there's only one value, we have a couple of alternate calling
-        # styles.
-        if isinstance(values[0], basestring) or hasattr(values[0], '__iter__'):
-            values = values[0]
-
-    return type(name, (Enumerated,), {'values': values})

lib/beets/util/functemplate.py

@@ -42,6 +42,7 @@ ESCAPE_CHAR = u'$'
 VARIABLE_PREFIX = '__var_'
 FUNCTION_PREFIX = '__func_'
 
+
 class Environment(object):
     """Contains the values and functions to be substituted into a
     template.
@@ -57,10 +58,12 @@ def ex_lvalue(name):
     """A variable load expression."""
     return ast.Name(name, ast.Store())
 
+
 def ex_rvalue(name):
     """A variable store expression."""
     return ast.Name(name, ast.Load())
 
+
 def ex_literal(val):
     """An int, float, long, bool, string, or None literal with the given
     value.
@@ -75,6 +78,7 @@ def ex_literal(val):
         return ast.Str(val)
     raise TypeError('no literal for {0}'.format(type(val)))
 
+
 def ex_varassign(name, expr):
     """Assign an expression into a single variable. The expression may
     either be an `ast.expr` object or a value to be used as a literal.
@@ -83,6 +87,7 @@ def ex_varassign(name, expr):
         expr = ex_literal(expr)
     return ast.Assign([ex_lvalue(name)], expr)
 
+
 def ex_call(func, args):
     """A function-call expression with only positional parameters. The
     function may be an expression or the name of a function. Each
@@ -98,6 +103,7 @@ def ex_call(func, args):
 
     return ast.Call(func, args, [], None, None)
 
+
 def compile_func(arg_names, statements, name='_the_func', debug=False):
     """Compile a list of statements as the body of a function and return
     the resulting Python function. If `debug`, then print out the
@@ -157,6 +163,7 @@ class Symbol(object):
         expr = ex_rvalue(VARIABLE_PREFIX + self.ident.encode('utf8'))
         return [expr], set([self.ident.encode('utf8')]), set()
 
+
 class Call(object):
     """A function call in a template."""
     def __init__(self, ident, args, original):
@@ -214,6 +221,7 @@ class Call(object):
         )
         return [subexpr_call], varnames, funcnames
 
+
 class Expression(object):
     """Top-level template construct: contains a list of text blobs,
     Symbols, and Calls.
@@ -259,6 +267,7 @@ class Expression(object):
 class ParseError(Exception):
     pass
 
+
 class Parser(object):
     """Parses a template expression string. Instantiate the class with
     the template source and call ``parse_expression``. The ``pos`` field
@@ -316,13 +325,13 @@ class Parser(object):
 
             next_char = self.string[self.pos + 1]
             if char == ESCAPE_CHAR and next_char in \
-                  (SYMBOL_DELIM, FUNC_DELIM, GROUP_CLOSE, ARG_SEP):
+                    (SYMBOL_DELIM, FUNC_DELIM, GROUP_CLOSE, ARG_SEP):
                 # An escaped special character ($$, $}, etc.). Note that
                 # ${ is not an escape sequence: this is ambiguous with
                 # the start of a symbol and it's not necessary (just
                 # using { suffices in all cases).
                 text_parts.append(next_char)
-                self.pos += 2 # Skip the next character.
+                self.pos += 2  # Skip the next character.
                 continue
 
             # Shift all characters collected so far into a single string.
@@ -372,7 +381,7 @@ class Parser(object):
 
         if next_char == GROUP_OPEN:
             # A symbol like ${this}.
-            self.pos += 1 # Skip opening.
+            self.pos += 1  # Skip opening.
             closer = self.string.find(GROUP_CLOSE, self.pos)
             if closer == -1 or closer == self.pos:
                 # No closing brace found or identifier is empty.
@@ -431,7 +440,7 @@ class Parser(object):
             self.parts.append(self.string[start_pos:self.pos])
             return
 
-        self.pos += 1 # Move past closing brace.
+        self.pos += 1  # Move past closing brace.
         self.parts.append(Call(ident, args, self.string[start_pos:self.pos]))
 
     def parse_argument_list(self):
@@ -472,6 +481,7 @@ class Parser(object):
         self.pos += len(ident)
         return ident
 
+
 def _parse(template):
     """Parse a top-level template string Expression. Any extraneous text
     is considered literal text.

lib/beets/util/pipeline.py

@@ -35,13 +35,13 @@ from __future__ import print_function
 import Queue
 from threading import Thread, Lock
 import sys
-import types
 
 BUBBLE = '__PIPELINE_BUBBLE__'
 POISON = '__PIPELINE_POISON__'
 
 DEFAULT_QUEUE_SIZE = 16
 
+
 def _invalidate_queue(q, val=None, sync=True):
     """Breaks a Queue such that it never blocks, always has size 1,
     and has no maximum size. get()ing from the queue returns `val`,
@@ -50,8 +50,10 @@ def _invalidate_queue(q, val=None, sync=True):
     """
     def _qsize(len=len):
         return 1
+
     def _put(item):
         pass
+
     def _get():
         return val
 
@@ -70,6 +72,7 @@ def _invalidate_queue(q, val=None, sync=True):
         if sync:
             q.mutex.release()
 
+
 class CountedQueue(Queue.Queue):
     """A queue that keeps track of the number of threads that are
     still feeding into it. The queue is poisoned when all threads are
@@ -104,6 +107,7 @@ class CountedQueue(Queue.Queue):
 
                 # Replacement _get invalidates when no items remain.
                 _old_get = self._get
+
                 def _get():
                     out = _old_get()
                     if not self.queue:
@@ -117,18 +121,67 @@ class CountedQueue(Queue.Queue):
                     # No items. Invalidate immediately.
                     _invalidate_queue(self, POISON, False)
 
+
 class MultiMessage(object):
     """A message yielded by a pipeline stage encapsulating multiple
     values to be sent to the next stage.
     """
     def __init__(self, messages):
         self.messages = messages
+
+
 def multiple(messages):
     """Yield multiple([message, ..]) from a pipeline stage to send
     multiple values to the next pipeline stage.
     """
     return MultiMessage(messages)
 
+
+def stage(func):
+    """Decorate a function to become a simple stage.
+
+    >>> @stage
+    ... def add(n, i):
+    ...     return i + n
+    >>> pipe = Pipeline([
+    ...     iter([1, 2, 3]),
+    ...     add(2),
+    ... ])
+    >>> list(pipe.pull())
+    [3, 4, 5]
+    """
+
+    def coro(*args):
+        task = None
+        while True:
+            task = yield task
+            task = func(*(args + (task,)))
+    return coro
+
+
+def mutator_stage(func):
+    """Decorate a function that manipulates items in a coroutine to
+    become a simple stage.
+
+    >>> @mutator_stage
+    ... def setkey(key, item):
+    ...     item[key] = True
+    >>> pipe = Pipeline([
+    ...     iter([{'x': False}, {'a': False}]),
+    ...     setkey('x'),
+    ... ])
+    >>> list(pipe.pull())
+    [{'x': True}, {'a': False, 'x': True}]
+    """
+
+    def coro(*args):
+        task = None
+        while True:
+            task = yield task
+            func(*(args + (task,)))
+    return coro
+
+
 def _allmsgs(obj):
     """Returns a list of all the messages encapsulated in obj. If obj
     is a MultiMessage, returns its enclosed messages. If obj is BUBBLE,
@@ -141,6 +194,7 @@ def _allmsgs(obj):
     else:
         return [obj]
 
+
 class PipelineThread(Thread):
     """Abstract base class for pipeline-stage threads."""
     def __init__(self, all_threads):
@@ -169,6 +223,7 @@ class PipelineThread(Thread):
         for thread in self.all_threads:
             thread.abort()
 
+
 class FirstPipelineThread(PipelineThread):
     """The thread running the first stage in a parallel pipeline setup.
     The coroutine should just be a generator.
@@ -209,6 +264,7 @@ class FirstPipelineThread(PipelineThread):
         # Generator finished; shut down the pipeline.
         self.out_queue.release()
 
+
 class MiddlePipelineThread(PipelineThread):
     """A thread running any stage in the pipeline except the first or
     last.
@@ -256,6 +312,7 @@ class MiddlePipelineThread(PipelineThread):
         # Pipeline is shutting down normally.
         self.out_queue.release()
 
+
 class LastPipelineThread(PipelineThread):
     """A thread running the last stage in a pipeline. The coroutine
     should yield nothing.
@@ -291,6 +348,7 @@ class LastPipelineThread(PipelineThread):
             self.abort_all(sys.exc_info())
             return
 
+
 class Pipeline(object):
     """Represents a staged pattern of work. Each stage in the pipeline
     is a coroutine that receives messages from the previous stage and
@@ -322,7 +380,8 @@ class Pipeline(object):
         messages between the stages are stored in queues of the given
         size.
         """
-        queues = [CountedQueue(queue_size) for i in range(len(self.stages)-1)]
+        queue_count = len(self.stages) - 1
+        queues = [CountedQueue(queue_size) for i in range(queue_count)]
         threads = []
 
         # Set up first stage.
@@ -330,10 +389,10 @@ class Pipeline(object):
             threads.append(FirstPipelineThread(coro, queues[0], threads))
 
         # Middle stages.
-        for i in range(1, len(self.stages)-1):
+        for i in range(1, queue_count):
             for coro in self.stages[i]:
                 threads.append(MiddlePipelineThread(
-                    coro, queues[i-1], queues[i], threads
+                    coro, queues[i - 1], queues[i], threads
                 ))
 
         # Last stage.
@@ -408,17 +467,20 @@ if __name__ == '__main__':
             print('generating %i' % i)
             time.sleep(1)
             yield i
+
     def work():
         num = yield
         while True:
             print('processing %i' % num)
             time.sleep(2)
-            num = yield num*2
+            num = yield num * 2
+
     def consume():
         while True:
             num = yield
             time.sleep(1)
             print('received %i' % num)
+
     ts_start = time.time()
     Pipeline([produce(), work(), consume()]).run_sequential()
     ts_seq = time.time()
@@ -437,6 +499,7 @@ if __name__ == '__main__':
             print('generating %i' % i)
             time.sleep(1)
             yield i
+
     def exc_work():
         num = yield
         while True:
@@ -445,10 +508,10 @@ if __name__ == '__main__':
             if num == 3:
                 raise Exception()
             num = yield num * 2
+
     def exc_consume():
         while True:
             num = yield
-            #if num == 4:
-            #   raise Exception()
             print('received %i' % num)
+
     Pipeline([exc_produce(), exc_work(), exc_consume()]).run_parallel(1)

lib/beets/vfs.py

@@ -20,6 +20,7 @@ from beets import util
 
 Node = namedtuple('Node', ['files', 'dirs'])
 
+
 def _insert(node, path, itemid):
     """Insert an item into a virtual filesystem node."""
     if len(path) == 1:
@@ -33,6 +34,7 @@ def _insert(node, path, itemid):
             node.dirs[dirname] = Node({}, {})
         _insert(node.dirs[dirname], rest, itemid)
 
+
 def libtree(lib):
     """Generates a filesystem-like directory tree for the files
     contained in `lib`. Filesystem nodes are (files, dirs) named

lib/beetsplug/embedart.py

@@ -16,6 +16,9 @@
 import os.path
 import logging
 import imghdr
+import subprocess
+import platform
+from tempfile import NamedTemporaryFile
 
 from beets.plugins import BeetsPlugin
 from beets import mediafile
@@ -25,6 +28,7 @@ from beets.util import syspath, normpath, displayable_path
 from beets.util.artresizer import ArtResizer
 from beets import config
 
+
 log = logging.getLogger('beets')
 
 
@@ -36,12 +40,19 @@ class EmbedCoverArtPlugin(BeetsPlugin):
         self.config.add({
             'maxwidth': 0,
             'auto': True,
+            'compare_threshold': 0,
+            'ifempty': False,
         })
-        if self.config['maxwidth'].get(int) and \
+
-                not ArtResizer.shared.local:
+        if self.config['maxwidth'].get(int) and not ArtResizer.shared.local:
             self.config['maxwidth'] = 0
             log.warn(u"embedart: ImageMagick or PIL not found; "
                      u"'maxwidth' option ignored")
+        if self.config['compare_threshold'].get(int) and not \
+                ArtResizer.shared.can_compare:
+            self.config['compare_threshold'] = 0
+            log.warn(u"embedart: ImageMagick 6.8.7 or higher not installed; "
+                     u"'compare_threshold' option ignored")
 
     def commands(self):
         # Embed command.
@@ -52,12 +63,15 @@ class EmbedCoverArtPlugin(BeetsPlugin):
             '-f', '--file', metavar='PATH', help='the image file to embed'
         )
         maxwidth = config['embedart']['maxwidth'].get(int)
+        compare_threshold = config['embedart']['compare_threshold'].get(int)
+        ifempty = config['embedart']['ifempty'].get(bool)
 
         def embed_func(lib, opts, args):
             if opts.file:
                 imagepath = normpath(opts.file)
                 for item in lib.items(decargs(args)):
-                    embed_item(item, imagepath, maxwidth)
+                    embed_item(item, imagepath, maxwidth, None,
+                               compare_threshold, ifempty)
             else:
                 for album in lib.albums(decargs(args)):
                     embed_album(album, maxwidth)
@@ -72,7 +86,8 @@ class EmbedCoverArtPlugin(BeetsPlugin):
 
         def extract_func(lib, opts, args):
             outpath = normpath(opts.outpath or 'cover')
-            extract(lib, outpath, decargs(args))
+            item = lib.items(decargs(args)).get()
+            extract(outpath, item)
         extract_cmd.func = extract_func
 
         # Clear command.
@@ -91,23 +106,43 @@ def album_imported(lib, album):
     """Automatically embed art into imported albums.
     """
     if album.artpath and config['embedart']['auto']:
-        embed_album(album, config['embedart']['maxwidth'].get(int))
+        embed_album(album, config['embedart']['maxwidth'].get(int), True)
 
 
-def embed_item(item, imagepath, maxwidth=None, itempath=None):
+def embed_item(item, imagepath, maxwidth=None, itempath=None,
+               compare_threshold=0, ifempty=False, as_album=False):
     """Embed an image into the item's media file.
     """
+    if compare_threshold:
+        if not check_art_similarity(item, imagepath, compare_threshold):
+            log.warn(u'Image not similar; skipping.')
+            return
+    if ifempty:
+        art = get_art(item)
+        if not art:
+            pass
+        else:
+            log.debug(u'embedart: media file contained art already {0}'.format(
+                displayable_path(imagepath)
+            ))
+            return
+    if maxwidth and not as_album:
+        imagepath = resize_image(imagepath, maxwidth)
+
     try:
+        log.debug(u'embedart: embedding {0}'.format(
+            displayable_path(imagepath)
+        ))
         item['images'] = [_mediafile_image(imagepath, maxwidth)]
-        item.try_write(itempath)
     except IOError as exc:
         log.error(u'embedart: could not read image file: {0}'.format(exc))
-    finally:
+    else:
-        # We don't want to store the image in the database
+        # We don't want to store the image in the database.
+        item.try_write(itempath)
         del item['images']
 
 
-def embed_album(album, maxwidth=None):
+def embed_album(album, maxwidth=None, quiet=False):
     """Embed album art into all of the album's items.
     """
     imagepath = album.artpath
@@ -115,39 +150,78 @@ def embed_album(album, maxwidth=None):
         log.info(u'No album art present: {0} - {1}'.
                  format(album.albumartist, album.album))
         return
-    if not os.path.isfile(imagepath):
+    if not os.path.isfile(syspath(imagepath)):
         log.error(u'Album art not found at {0}'
-                  .format(imagepath))
+                  .format(displayable_path(imagepath)))
         return
+    if maxwidth:
+        imagepath = resize_image(imagepath, maxwidth)
 
-    log.info(u'Embedding album art into {0.albumartist} - {0.album}.'
+    log.log(
-             .format(album))
+        logging.DEBUG if quiet else logging.INFO,
+        u'Embedding album art into {0.albumartist} - {0.album}.'.format(album),
+    )
 
     for item in album.items():
-        embed_item(item, imagepath, maxwidth)
+        embed_item(item, imagepath, maxwidth, None,
+                   config['embedart']['compare_threshold'].get(int),
+                   config['embedart']['ifempty'].get(bool), as_album=True)
+
+
+def resize_image(imagepath, maxwidth):
+    """Returns path to an image resized to maxwidth.
+    """
+    log.info(u'Resizing album art to {0} pixels wide'
+             .format(maxwidth))
+    imagepath = ArtResizer.shared.resize(maxwidth, syspath(imagepath))
+    return imagepath
+
+
+def check_art_similarity(item, imagepath, compare_threshold):
+    """A boolean indicating if an image is similar to embedded item art.
+    """
+    with NamedTemporaryFile(delete=True) as f:
+        art = extract(f.name, item)
+
+        if art:
+            # Converting images to grayscale tends to minimize the weight
+            # of colors in the diff score
+            cmd = 'convert {0} {1} -colorspace gray MIFF:- | ' \
+                  'compare -metric PHASH - null:'.format(syspath(imagepath),
+                                                         syspath(art))
+
+            proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
+                                    stderr=subprocess.PIPE,
+                                    close_fds=platform.system() != 'Windows',
+                                    shell=True)
+            stdout, stderr = proc.communicate()
+            if proc.returncode:
+                if proc.returncode != 1:
+                    log.warn(u'embedart: IM phashes compare failed for {0}, \
+                   {1}'.format(displayable_path(imagepath),
+                               displayable_path(art)))
+                    return
+                phashDiff = float(stderr)
+            else:
+                phashDiff = float(stdout)
+
+            log.info(u'embedart: compare PHASH score is {0}'.format(phashDiff))
+            if phashDiff > compare_threshold:
+                return False
+
+    return True
 
 
 def _mediafile_image(image_path, maxwidth=None):
     """Return a `mediafile.Image` object for the path.
-
-    If maxwidth is set the image is resized if necessary.
     """
-    if maxwidth:
-        image_path = ArtResizer.shared.resize(maxwidth, syspath(image_path))
 
     with open(syspath(image_path), 'rb') as f:
         data = f.read()
     return mediafile.Image(data, type=mediafile.ImageType.front)
 
 
-# 'extractart' command.
+def get_art(item):
-
-def extract(lib, outpath, query):
-    item = lib.items(query).get()
-    if not item:
-        log.error(u'No item matches query.')
-        return
-
     # Extract the art.
     try:
         mf = mediafile.MediaFile(syspath(item.path))
@@ -157,7 +231,18 @@ def extract(lib, outpath, query):
         ))
         return
 
-    art = mf.art
+    return mf.art
+
+# 'extractart' command.
+
+
+def extract(outpath, item):
+    if not item:
+        log.error(u'No item matches query.')
+        return
+
+    art = get_art(item)
+
     if not art:
         log.error(u'No album art present in {0} - {1}.'
                   .format(item.artist, item.title))
@@ -170,10 +255,11 @@ def extract(lib, outpath, query):
         return
     outpath += '.' + ext
 
-    log.info(u'Extracting album art from: {0.artist} - {0.title}\n'
+    log.info(u'Extracting album art from: {0.artist} - {0.title} '
-             u'To: {1}'.format(item, displayable_path(outpath)))
+             u'to: {1}'.format(item, displayable_path(outpath)))
     with open(syspath(outpath), 'wb') as f:
         f.write(art)
+    return outpath
 
 
 # 'clearart' command.
@@ -190,5 +276,5 @@ def clear(lib, query):
                 displayable_path(item.path), exc
             ))
             continue
-        mf.art = None
+        del mf.art
         mf.save()

lib/beetsplug/fetchart.py

@@ -1,5 +1,5 @@
 # This file is part of beets.
-# Copyright 2013, Adrian Sampson.
+# Copyright 2014, Adrian Sampson.
 #
 # Permission is hereby granted, free of charge, to any person obtaining
 # a copy of this software and associated documentation files (the
@@ -22,12 +22,18 @@ from tempfile import NamedTemporaryFile
 
 import requests
 
-from beets.plugins import BeetsPlugin
+from beets import plugins
-from beets.util.artresizer import ArtResizer
 from beets import importer
 from beets import ui
 from beets import util
 from beets import config
+from beets.util.artresizer import ArtResizer
+
+try:
+    import itunes
+    HAVE_ITUNES = True
+except ImportError:
+    HAVE_ITUNES = False
 
 IMAGE_EXTENSIONS = ['png', 'jpg', 'jpeg']
 CONTENT_TYPES = ('image/jpeg',)
@@ -73,17 +79,14 @@ CAA_URL = 'http://coverartarchive.org/release/{mbid}/front-500.jpg'
 CAA_GROUP_URL = 'http://coverartarchive.org/release-group/{mbid}/front-500.jpg'
 
 
-def caa_art(release_id):
+def caa_art(album):
-    """Return the Cover Art Archive URL given a MusicBrainz release ID.
+    """Return the Cover Art Archive and Cover Art Archive release group URLs
+    using album MusicBrainz release ID and release group ID.
     """
-    return CAA_URL.format(mbid=release_id)
+    if album.mb_albumid:
-
+        yield CAA_URL.format(mbid=album.mb_albumid)
-
+    if album.mb_releasegroupid:
-def caa_group_art(release_group_id):
+        yield CAA_GROUP_URL.format(mbid=album.mb_releasegroupid)
-    """Return the Cover Art Archive release group URL given a MusicBrainz
-    release group ID.
-    """
-    return CAA_GROUP_URL.format(mbid=release_group_id)
 
 
 # Art from Amazon.
@@ -92,10 +95,12 @@ AMAZON_URL = 'http://images.amazon.com/images/P/%s.%02i.LZZZZZZZ.jpg'
 AMAZON_INDICES = (1, 2)
 
 
-def art_for_asin(asin):
+def art_for_asin(album):
-    """Generate URLs for an Amazon ID (ASIN) string."""
+    """Generate URLs using Amazon ID (ASIN) string.
-    for index in AMAZON_INDICES:
+    """
-        yield AMAZON_URL % (asin, index)
+    if album.asin:
+        for index in AMAZON_INDICES:
+            yield AMAZON_URL % (album.asin, index)
 
 
 # AlbumArt.org scraper.
@@ -104,11 +109,14 @@ AAO_URL = 'http://www.albumart.org/index_detail.php'
 AAO_PAT = r'href\s*=\s*"([^>"]*)"[^>]*title\s*=\s*"View larger image"'
 
 
-def aao_art(asin):
+def aao_art(album):
-    """Return art URL from AlbumArt.org given an ASIN."""
+    """Return art URL from AlbumArt.org using album ASIN.
+    """
+    if not album.asin:
+        return
     # Get the page from albumart.org.
     try:
-        resp = requests_session.get(AAO_URL, params={'asin': asin})
+        resp = requests_session.get(AAO_URL, params={'asin': album.asin})
         log.debug(u'fetchart: scraped art URL: {0}'.format(resp.url))
     except requests.RequestException:
         log.debug(u'fetchart: error scraping art page')
@@ -118,7 +126,7 @@ def aao_art(asin):
     m = re.search(AAO_PAT, resp.text)
     if m:
         image_url = m.group(1)
-        return image_url
+        yield image_url
     else:
         log.debug(u'fetchart: no image found on page')
 
@@ -132,6 +140,8 @@ def google_art(album):
     """Return art URL from google.org given an album title and
     interpreter.
     """
+    if not (album.albumartist and album.album):
+        return
     search_string = (album.albumartist + ',' + album.album).encode('utf-8')
     response = requests_session.get(GOOGLE_URL, params={
         'v': '1.0',
@@ -145,14 +155,39 @@ def google_art(album):
         data = results['responseData']
         dataInfo = data['results']
         for myUrl in dataInfo:
-            return myUrl['unescapedUrl']
+            yield myUrl['unescapedUrl']
     except:
         log.debug(u'fetchart: error scraping art page')
         return
 
 
+# Art from the iTunes Store.
+
+def itunes_art(album):
+    """Return art URL from iTunes Store given an album title.
+    """
+    search_string = (album.albumartist + ' ' + album.album).encode('utf-8')
+    try:
+        # Isolate bugs in the iTunes library while searching.
+        try:
+            itunes_album = itunes.search_album(search_string)[0]
+        except Exception as exc:
+            log.debug('fetchart: iTunes search failed: {0}'.format(exc))
+            return
+
+        if itunes_album.get_artwork()['100']:
+            small_url = itunes_album.get_artwork()['100']
+            big_url = small_url.replace('100x100', '1200x1200')
+            yield big_url
+        else:
+            log.debug(u'fetchart: album has no artwork in iTunes Store')
+    except IndexError:
+        log.debug(u'fetchart: album not found in iTunes Store')
+
+
 # Art from the filesystem.
 
+
 def filename_priority(filename, cover_names):
     """Sort order for image names.
 
@@ -164,7 +199,8 @@ def filename_priority(filename, cover_names):
 
 
 def art_in_path(path, cover_names, cautious):
-    """Look for album art files in a specified directory."""
+    """Look for album art files in a specified directory.
+    """
     if not os.path.isdir(path):
         return
 
@@ -195,31 +231,27 @@ def art_in_path(path, cover_names, cautious):
 
 # Try each source in turn.
 
+SOURCES_ALL = [u'coverart', u'itunes', u'amazon', u'albumart', u'google']
 
-def _source_urls(album):
+ART_FUNCS = {
+    u'coverart': caa_art,
+    u'itunes': itunes_art,
+    u'albumart': aao_art,
+    u'amazon': art_for_asin,
+    u'google': google_art,
+}
+
+
+def _source_urls(album, sources=SOURCES_ALL):
     """Generate possible source URLs for an album's art. The URLs are
     not guaranteed to work so they each need to be attempted in turn.
     This allows the main `art_for_album` function to abort iteration
     through this sequence early to avoid the cost of scraping when not
     necessary.
     """
-    # Cover Art Archive.
+    for s in sources:
-    if album.mb_albumid:
+        urls = ART_FUNCS[s](album)
-        yield caa_art(album.mb_albumid)
+        for url in urls:
-    if album.mb_releasegroupid:
-        yield caa_group_art(album.mb_releasegroupid)
-
-    # Amazon and AlbumArt.org.
-    if album.asin:
-        for url in art_for_asin(album.asin):
-            yield url
-        url = aao_art(album.asin)
-        if url:
-            yield url
-
-    if config['fetchart']['google_search']:
-        url = google_art(album)
-        if url:
             yield url
 
 
@@ -245,7 +277,8 @@ def art_for_album(album, paths, maxwidth=None, local_only=False):
     # Web art sources.
     remote_priority = config['fetchart']['remote_priority'].get(bool)
     if not local_only and (remote_priority or not out):
-        for url in _source_urls(album):
+        for url in _source_urls(album,
+                                config['fetchart']['sources'].as_str_seq()):
             if maxwidth:
                 url = ArtResizer.shared.proxy_url(maxwidth, url)
             candidate = _fetch_image(url)
@@ -286,7 +319,7 @@ def batch_fetch_art(lib, albums, force, maxwidth=None):
                                           message))
 
 
-class FetchArtPlugin(BeetsPlugin):
+class FetchArtPlugin(plugins.BeetsPlugin):
     def __init__(self):
         super(FetchArtPlugin, self).__init__()
 
@@ -297,6 +330,7 @@ class FetchArtPlugin(BeetsPlugin):
             'cautious': False,
             'google_search': False,
             'cover_names': ['cover', 'front', 'art', 'album', 'folder'],
+            'sources': SOURCES_ALL,
         })
 
         # Holds paths to downloaded images between fetching them and
@@ -309,6 +343,12 @@ class FetchArtPlugin(BeetsPlugin):
             self.import_stages = [self.fetch_art]
             self.register_listener('import_task_files', self.assign_art)
 
+        available_sources = list(SOURCES_ALL)
+        if not HAVE_ITUNES and u'itunes' in available_sources:
+            available_sources.remove(u'itunes')
+        self.config['sources'] = plugins.sanitize_choices(
+            self.config['sources'].as_str_seq(), available_sources)
+
     # Asynchronous; after music is added to the library.
     def fetch_art(self, session, task):
         """Find art for the album being imported."""

lib/beetsplug/lyrics.py

@@ -18,25 +18,26 @@ from __future__ import print_function
 
 import re
 import logging
-import urllib
+import requests
 import json
 import unicodedata
+import urllib
 import difflib
 import itertools
+from HTMLParser import HTMLParseError
 
-from beets.plugins import BeetsPlugin
+from beets import plugins
-from beets import ui
+from beets import config, ui
-from beets import config
 
 
 # Global logger.
 
 log = logging.getLogger('beets')
 
-DIV_RE = re.compile(r'<(/?)div>?')
+DIV_RE = re.compile(r'<(/?)div>?', re.I)
 COMMENT_RE = re.compile(r'<!--.*-->', re.S)
 TAG_RE = re.compile(r'<[^>]*>')
-BREAK_RE = re.compile(r'<br\s*/?>')
+BREAK_RE = re.compile(r'\n?\s*<br([\s|/][^>]*)*>\s*\n?', re.I)
 URL_CHARACTERS = {
     u'\u2018': u"'",
     u'\u2019': u"'",
@@ -60,10 +61,14 @@ def fetch_url(url):
     is unreachable.
     """
     try:
-        return urllib.urlopen(url).read()
+        r = requests.get(url, verify=False)
-    except IOError as exc:
+    except requests.RequestException as exc:
-        log.debug(u'failed to fetch: {0} ({1})'.format(url, unicode(exc)))
+        log.debug(u'lyrics request failed: {0}'.format(exc))
-        return None
+        return
+    if r.status_code == requests.codes.ok:
+        return r.text
+    else:
+        log.debug(u'failed to fetch: {0} ({1})'.format(url, r.status_code))
 
 
 def unescape(text):
@@ -79,10 +84,20 @@ def unescape(text):
     return out
 
 
-def extract_text(html, starttag):
+def extract_text_between(html, start_marker, end_marker):
+    try:
+        _, html = html.split(start_marker, 1)
+        html, _ = html.split(end_marker, 1)
+    except ValueError:
+        return u''
+    return html
+
+
+def extract_text_in(html, starttag):
     """Extract the text from a <DIV> tag in the HTML starting with
     ``starttag``. Returns None if parsing fails.
     """
+
     # Strip off the leading text before opening tag.
     try:
         _, html = html.split(starttag, 1)
@@ -101,7 +116,6 @@ def extract_text(html, starttag):
         else:  # Opening tag.
             if level == 0:
                 parts.append(html[pos:match.start()])
-
             level += 1
 
         if level == -1:
@@ -110,26 +124,7 @@ def extract_text(html, starttag):
     else:
         print('no closing tag found!')
         return
-    lyrics = ''.join(parts)
+    return u''.join(parts)
-    return strip_cruft(lyrics)
-
-
-def strip_cruft(lyrics, wscollapse=True):
-    """Clean up HTML from an extracted lyrics string. For example, <BR>
-    tags are replaced with newlines.
-    """
-    lyrics = COMMENT_RE.sub('', lyrics)
-    lyrics = unescape(lyrics)
-    if wscollapse:
-        lyrics = re.sub(r'\s+', ' ', lyrics)  # Whitespace collapse.
-    lyrics = re.sub(r'<(script).*?</\1>(?s)', '', lyrics)  # Strip script tags.
-    lyrics = BREAK_RE.sub('\n', lyrics)  # <BR> newlines.
-    lyrics = re.sub(r'\n +', '\n', lyrics)
-    lyrics = re.sub(r' +\n', '\n', lyrics)
-    lyrics = TAG_RE.sub('', lyrics)  # Strip remaining HTML tags.
-    lyrics = lyrics.replace('\r', '\n')
-    lyrics = lyrics.strip()
-    return lyrics
 
 
 def search_pairs(item):
@@ -140,7 +135,7 @@ def search_pairs(item):
 
     In addition to the artist and title obtained from the `item` the
     method tries to strip extra information like paranthesized suffixes
-    and featured artists from the strings and add them as caniddates.
+    and featured artists from the strings and add them as candidates.
     The method also tries to split multiple titles separated with `/`.
     """
 
@@ -149,7 +144,7 @@ def search_pairs(item):
     artists = [artist]
 
     # Remove any featuring artists from the artists name
-    pattern = r"(.*?) (&|\b(and|ft|feat(uring)?\b))"
+    pattern = r"(.*?) {0}".format(plugins.feat_tokens())
     match = re.search(pattern, artist, re.IGNORECASE)
     if match:
         artists.append(match.group(1))
@@ -162,8 +157,8 @@ def search_pairs(item):
         titles.append(match.group(1))
 
     # Remove any featuring artists from the title
-    pattern = r"(.*?) \b(ft|feat(uring)?)\b"
+    pattern = r"(.*?) {0}".format(plugins.feat_tokens(for_artist=False))
-    for title in titles:
+    for title in titles[:]:
         match = re.search(pattern, title, re.IGNORECASE)
         if match:
             titles.append(match.group(1))
@@ -189,6 +184,19 @@ def _encode(s):
         s = s.encode('utf8', 'ignore')
     return urllib.quote(s)
 
+# Musixmatch
+
+MUSIXMATCH_URL_PATTERN = 'https://www.musixmatch.com/lyrics/%s/%s'
+
+
+def fetch_musixmatch(artist, title):
+    url = MUSIXMATCH_URL_PATTERN % (_lw_encode(artist.title()),
+                                    _lw_encode(title.title()))
+    html = fetch_url(url)
+    if not html:
+        return
+    lyrics = extract_text_between(html, '"lyrics_body":', '"lyrics_language":')
+    return lyrics.strip(',"').replace('\\n', '\n')
 
 # LyricsWiki.
 
@@ -212,7 +220,7 @@ def fetch_lyricswiki(artist, title):
     if not html:
         return
 
-    lyrics = extract_text(html, "<div class='lyricbox'>")
+    lyrics = extract_text_in(html, u"<div class='lyricbox'>")
     if lyrics and 'Unfortunately, we are not licensed' not in lyrics:
         return lyrics
 
@@ -238,8 +246,8 @@ def fetch_lyricscom(artist, title):
     html = fetch_url(url)
     if not html:
         return
-
+    lyrics = extract_text_between(html, '<div id="lyrics" class="SCREENONLY" '
-    lyrics = extract_text(html, '<div id="lyric_space">')
+                                  'itemprop="description">', '</div>')
     if not lyrics:
         return
     for not_found_str in LYRICSCOM_NOT_FOUND:
@@ -280,7 +288,6 @@ def is_page_candidate(urlLink, urlTitle, title, artist):
     artist = slugify(artist.lower())
     sitename = re.search(u"//([^/]+)/.*", slugify(urlLink.lower())).group(1)
     urlTitle = slugify(urlTitle.lower())
-
     # Check if URL title contains song title (exact match)
     if urlTitle.find(title) != -1:
         return True
@@ -289,41 +296,11 @@ def is_page_candidate(urlLink, urlTitle, title, artist):
     tokens = [by + '_' + artist for by in BY_TRANS] + \
              [artist, sitename, sitename.replace('www.', '')] + LYRICS_TRANS
     songTitle = re.sub(u'(%s)' % u'|'.join(tokens), u'', urlTitle)
-
+    songTitle = songTitle.strip('_|')
-    typoRatio = .8
+    typoRatio = .9
     return difflib.SequenceMatcher(None, songTitle, title).ratio() >= typoRatio
 
 
-def insert_line_feeds(text):
-    """Insert newlines before upper-case characters.
-    """
-    tokensStr = re.split("([a-z][A-Z])", text)
-    for idx in range(1, len(tokensStr), 2):
-        ltoken = list(tokensStr[idx])
-        tokensStr[idx] = ltoken[0] + '\n' + ltoken[1]
-    return ''.join(tokensStr)
-
-
-def sanitize_lyrics(text):
-    """Clean text, returning raw lyrics as output or None if it happens
-    that input text is actually not lyrics content.  Clean (x)html tags
-    in text, correct layout and syntax...
-    """
-    text = strip_cruft(text, False)
-
-    # Restore \n in input text
-    if '\n' not in text:
-        text = insert_line_feeds(text)
-
-    while text.count('\n\n') > text.count('\n') // 4:
-        # Remove first occurrence of \n for each sequence of \n
-        text = re.sub(r'\n(\n+)', '\g<1>', text)
-
-    text = re.sub(r'\n\n+', '\n\n', text)   # keep at most two \n in a row
-
-    return text
-
-
 def remove_credits(text):
     """Remove first/last line of text if it contains the word 'lyrics'
     eg 'Lyrics by songsdatabase.com'
@@ -342,13 +319,12 @@ def is_lyrics(text, artist=None):
     """Determine whether the text seems to be valid lyrics.
     """
     if not text:
-        return
+        return False
-
     badTriggersOcc = []
     nbLines = text.count('\n')
     if nbLines <= 1:
         log.debug(u"Ignoring too short lyrics '{0}'".format(text))
-        return 0
+        return False
     elif nbLines < 5:
         badTriggersOcc.append('too_short')
     else:
@@ -356,7 +332,7 @@ def is_lyrics(text, artist=None):
         # down
         text = remove_credits(text)
 
-    badTriggers = ['lyrics', 'copyright', 'property']
+    badTriggers = ['lyrics', 'copyright', 'property', 'links']
     if artist:
         badTriggersOcc += [artist]
 
@@ -366,62 +342,58 @@ def is_lyrics(text, artist=None):
 
     if badTriggersOcc:
         log.debug(u'Bad triggers detected: {0}'.format(badTriggersOcc))
-
     return len(badTriggersOcc) < 2
 
 
-def scrape_lyrics_from_url(url):
+def _scrape_strip_cruft(html, plain_text_out=False):
+    """Clean up HTML
+    """
+    html = unescape(html)
+
+    html = html.replace('\r', '\n')  # Normalize EOL.
+    html = re.sub(r' +', ' ', html)  # Whitespaces collapse.
+    html = BREAK_RE.sub('\n', html)  # <br> eats up surrounding '\n'.
+    html = re.sub(r'<(script).*?</\1>(?s)', '', html)  # Strip script tags.
+
+    if plain_text_out:  # Strip remaining HTML tags
+        html = COMMENT_RE.sub('', html)
+        html = TAG_RE.sub('', html)
+
+    html = '\n'.join([x.strip() for x in html.strip().split('\n')])
+    html = re.sub(r'\n{3,}', r'\n\n', html)
+    return html
+
+
+def _scrape_merge_paragraphs(html):
+    html = re.sub(r'</p>\s*<p(\s*[^>]*)>', '\n', html)
+    return re.sub(r'<div .*>\s*</div>', '\n', html)
+
+
+def scrape_lyrics_from_html(html):
     """Scrape lyrics from a URL. If no lyrics can be found, return None
     instead.
     """
-    from bs4 import BeautifulSoup, Comment
+    from bs4 import SoupStrainer, BeautifulSoup
-    html = fetch_url(url)
+
     if not html:
         return None
 
-    soup = BeautifulSoup(html)
+    def is_text_notcode(text):
-
+        length = len(text)
-    for tag in soup.findAll('br'):
+        return (length > 20 and
-        tag.replaceWith('\n')
+                text.count(' ') > length / 25 and
-
+                (text.find('{') == -1 or text.find(';') == -1))
-    # Remove non relevant html parts
+    html = _scrape_strip_cruft(html)
-    [s.extract() for s in soup(['head', 'script'])]
+    html = _scrape_merge_paragraphs(html)
-    comments = soup.findAll(text=lambda text: isinstance(text, Comment))
-    [s.extract() for s in comments]
 
+    # extract all long text blocks that are not code
     try:
-        for tag in soup.findAll(True):
+        soup = BeautifulSoup(html, "html.parser",
-            tag.name = 'p'          # keep tag contents
+                             parse_only=SoupStrainer(text=is_text_notcode))
-
+    except HTMLParseError:
-    except Exception, e:
+        return None
-        log.debug(u'Error {0} when replacing containing marker by p marker'
+    soup = sorted(soup.stripped_strings, key=len)[-1]
-                  .format(e, exc_info=True))
+    return soup
-
-    # Make better soup from current soup! The previous unclosed <p> sections
-    # are now closed.  Use str() rather than prettify() as it's more
-    # conservative concerning EOL
-    soup = BeautifulSoup(str(soup))
-
-    # In case lyrics are nested in no markup but <body>
-    # Insert the whole body in a <p>
-    bodyTag = soup.find('body')
-    if bodyTag:
-        pTag = soup.new_tag("p")
-        bodyTag.parent.insert(0, pTag)
-        pTag.insert(0, bodyTag)
-
-    tagTokens = []
-
-    for tag in soup.findAll('p'):
-        soup2 = BeautifulSoup(str(tag))
-        # Extract all text of <p> section.
-        tagTokens += soup2.findAll(text=True)
-
-    if tagTokens:
-        # Lyrics are expected to be the longest paragraph
-        tagTokens = sorted(tagTokens, key=len, reverse=True)
-        soup = BeautifulSoup(tagTokens[0])
-        return unescape(tagTokens[0].strip("\n\r: "))
 
 
 def fetch_google(artist, title):
@@ -443,15 +415,14 @@ def fetch_google(artist, title):
     if 'items' in data.keys():
         for item in data['items']:
             urlLink = item['link']
-            urlTitle = item['title']
+            urlTitle = item.get('title', u'')
             if not is_page_candidate(urlLink, urlTitle, title, artist):
                 continue
-            lyrics = scrape_lyrics_from_url(urlLink)
+            html = fetch_url(urlLink)
+            lyrics = scrape_lyrics_from_html(html)
             if not lyrics:
                 continue
 
-            lyrics = sanitize_lyrics(lyrics)
-
             if is_lyrics(lyrics, artist):
                 log.debug(u'got lyrics from {0}'.format(item['displayLink']))
                 return lyrics
@@ -459,8 +430,16 @@ def fetch_google(artist, title):
 
 # Plugin logic.
 
+SOURCES = ['google', 'lyricwiki', 'lyrics.com', 'musixmatch']
+SOURCE_BACKENDS = {
+    'google': fetch_google,
+    'lyricwiki': fetch_lyricswiki,
+    'lyrics.com': fetch_lyricscom,
+    'musixmatch': fetch_musixmatch,
+}
 
-class LyricsPlugin(BeetsPlugin):
+
+class LyricsPlugin(plugins.BeetsPlugin):
     def __init__(self):
         super(LyricsPlugin, self).__init__()
         self.import_stages = [self.imported]
@@ -469,12 +448,19 @@ class LyricsPlugin(BeetsPlugin):
             'google_API_key': None,
             'google_engine_ID': u'009217259823014548361:lndtuqkycfu',
             'fallback': None,
+            'force': False,
+            'sources': SOURCES,
         })
 
-        self.backends = [fetch_lyricswiki, fetch_lyricscom]
+        available_sources = list(SOURCES)
-
+        if not self.config['google_API_key'].get() and \
-        if self.config['google_API_key'].get():
+                'google' in SOURCES:
-            self.backends.insert(0, fetch_google)
+            available_sources.remove('google')
+        self.config['sources'] = plugins.sanitize_choices(
+            self.config['sources'].as_str_seq(), available_sources)
+        self.backends = []
+        for key in self.config['sources'].as_str_seq():
+            self.backends.append(SOURCE_BACKENDS[key])
 
     def commands(self):
         cmd = ui.Subcommand('lyrics', help='fetch song lyrics')
@@ -490,8 +476,10 @@ class LyricsPlugin(BeetsPlugin):
             # import_write config value.
             write = config['import']['write'].get(bool)
             for item in lib.items(ui.decargs(args)):
-                self.fetch_item_lyrics(lib, logging.INFO, item, write,
+                self.fetch_item_lyrics(
-                                       opts.force_refetch)
+                    lib, logging.INFO, item, write,
+                    opts.force_refetch or self.config['force'],
+                )
                 if opts.printlyr and item.lyrics:
                     ui.print_(item.lyrics)
 
@@ -504,7 +492,7 @@ class LyricsPlugin(BeetsPlugin):
         if self.config['auto']:
             for item in task.imported_items():
                 self.fetch_item_lyrics(session.lib, logging.DEBUG, item,
-                                       False, False)
+                                       False, self.config['force'])
 
     def fetch_item_lyrics(self, lib, loglevel, item, write, force):
         """Fetch and store lyrics for a single item. If ``write``, then the
@@ -551,8 +539,6 @@ class LyricsPlugin(BeetsPlugin):
         for backend in self.backends:
             lyrics = backend(artist, title)
             if lyrics:
-                if isinstance(lyrics, str):
-                    lyrics = lyrics.decode('utf8', 'ignore')
                 log.debug(u'got lyrics from backend: {0}'
                           .format(backend.__name__))
-                return lyrics.strip()
+                return _scrape_strip_cruft(lyrics, True)

lib/bs4/__init__.py

@@ -17,8 +17,8 @@ http://www.crummy.com/software/BeautifulSoup/bs4/doc/
 """
 
 __author__ = "Leonard Richardson (leonardr@segfault.org)"
-__version__ = "4.3.2"
+__version__ = "4.4.0"
-__copyright__ = "Copyright (c) 2004-2013 Leonard Richardson"
+__copyright__ = "Copyright (c) 2004-2015 Leonard Richardson"
 __license__ = "MIT"
 
 __all__ = ['BeautifulSoup']
@@ -45,7 +45,7 @@ from .element import (
 
 # The very first thing we do is give a useful error if someone is
 # running this code under Python 3 without converting it.
-syntax_error = u'You are trying to run the Python 2 version of Beautiful Soup under Python 3. This will not work. You need to convert the code, either by installing it (`python setup.py install`) or by running 2to3 (`2to3 -w bs4`).'
+'You are trying to run the Python 2 version of Beautiful Soup under Python 3. This will not work.'<>'You need to convert the code, either by installing it (`python setup.py install`) or by running 2to3 (`2to3 -w bs4`).'
 
 class BeautifulSoup(Tag):
     """
@@ -77,8 +77,11 @@ class BeautifulSoup(Tag):
 
     ASCII_SPACES = '\x20\x0a\x09\x0c\x0d'
 
+    NO_PARSER_SPECIFIED_WARNING = "No parser was explicitly specified, so I'm using the best available %(markup_type)s parser for this system (\"%(parser)s\"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.\n\nTo get rid of this warning, change this:\n\n BeautifulSoup([your markup])\n\nto this:\n\n BeautifulSoup([your markup], \"%(parser)s\")\n"
+
     def __init__(self, markup="", features=None, builder=None,
-                 parse_only=None, from_encoding=None, **kwargs):
+                 parse_only=None, from_encoding=None, exclude_encodings=None,
+                 **kwargs):
         """The Soup object is initialized as the 'root tag', and the
         provided markup (which can be a string or a file-like object)
         is fed into the underlying parser."""
@@ -114,9 +117,9 @@ class BeautifulSoup(Tag):
             del kwargs['isHTML']
             warnings.warn(
                 "BS4 does not respect the isHTML argument to the "
-                "BeautifulSoup constructor. You can pass in features='html' "
+                "BeautifulSoup constructor. Suggest you use "
-                "or features='xml' to get a builder capable of handling "
+                "features='lxml' for HTML and features='lxml-xml' for "
-                "one or the other.")
+                "XML.")
 
         def deprecated_argument(old_name, new_name):
             if old_name in kwargs:
@@ -140,6 +143,7 @@ class BeautifulSoup(Tag):
                 "__init__() got an unexpected keyword argument '%s'" % arg)
 
         if builder is None:
+            original_features = features
             if isinstance(features, basestring):
                 features = [features]
             if features is None or len(features) == 0:
@@ -151,6 +155,16 @@ class BeautifulSoup(Tag):
                     "requested: %s. Do you need to install a parser library?"
                     % ",".join(features))
             builder = builder_class()
+            if not (original_features == builder.NAME or
+                    original_features in builder.ALTERNATE_NAMES):
+                if builder.is_xml:
+                    markup_type = "XML"
+                else:
+                    markup_type = "HTML"
+                warnings.warn(self.NO_PARSER_SPECIFIED_WARNING % dict(
+                    parser=builder.NAME,
+                    markup_type=markup_type))
+
         self.builder = builder
         self.is_xml = builder.is_xml
         self.builder.soup = self
@@ -178,6 +192,8 @@ class BeautifulSoup(Tag):
                 # system. Just let it go.
                 pass
             if is_file:
+                if isinstance(markup, unicode):
+                    markup = markup.encode("utf8")
                 warnings.warn(
                     '"%s" looks like a filename, not markup. You should probably open this file and pass the filehandle into Beautiful Soup.' % markup)
             if markup[:5] == "http:" or markup[:6] == "https:":
@@ -185,12 +201,15 @@ class BeautifulSoup(Tag):
                 # Python 3 otherwise.
                 if ((isinstance(markup, bytes) and not b' ' in markup)
                     or (isinstance(markup, unicode) and not u' ' in markup)):
+                    if isinstance(markup, unicode):
+                        markup = markup.encode("utf8")
                     warnings.warn(
                         '"%s" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.' % markup)
 
         for (self.markup, self.original_encoding, self.declared_html_encoding,
          self.contains_replacement_characters) in (
-            self.builder.prepare_markup(markup, from_encoding)):
+             self.builder.prepare_markup(
+                 markup, from_encoding, exclude_encodings=exclude_encodings)):
             self.reset()
             try:
                 self._feed()
@@ -203,6 +222,16 @@ class BeautifulSoup(Tag):
         self.markup = None
         self.builder.soup = None
 
+    def __copy__(self):
+        return type(self)(self.encode(), builder=self.builder)
+
+    def __getstate__(self):
+        # Frequently a tree builder can't be pickled.
+        d = dict(self.__dict__)
+        if 'builder' in d and not self.builder.picklable:
+            del d['builder']
+        return d
+
     def _feed(self):
         # Convert the document to Unicode.
         self.builder.reset()
@@ -229,9 +258,7 @@ class BeautifulSoup(Tag):
 
     def new_string(self, s, subclass=NavigableString):
         """Create a new NavigableString associated with this soup."""
-        navigable = subclass(s)
+        return subclass(s)
-        navigable.setup()
-        return navigable
 
     def insert_before(self, successor):
         raise NotImplementedError("BeautifulSoup objects don't support insert_before().")
@@ -290,14 +317,49 @@ class BeautifulSoup(Tag):
     def object_was_parsed(self, o, parent=None, most_recent_element=None):
         """Add an object to the parse tree."""
         parent = parent or self.currentTag
-        most_recent_element = most_recent_element or self._most_recent_element
+        previous_element = most_recent_element or self._most_recent_element
-        o.setup(parent, most_recent_element)
+
+        next_element = previous_sibling = next_sibling = None
+        if isinstance(o, Tag):
+            next_element = o.next_element
+            next_sibling = o.next_sibling
+            previous_sibling = o.previous_sibling
+            if not previous_element:
+                previous_element = o.previous_element
+
+        o.setup(parent, previous_element, next_element, previous_sibling, next_sibling)
 
-        if most_recent_element is not None:
-            most_recent_element.next_element = o
         self._most_recent_element = o
         parent.contents.append(o)
 
+        if parent.next_sibling:
+            # This node is being inserted into an element that has
+            # already been parsed. Deal with any dangling references.
+            index = parent.contents.index(o)
+            if index == 0:
+                previous_element = parent
+                previous_sibling = None
+            else:
+                previous_element = previous_sibling = parent.contents[index-1]
+            if index == len(parent.contents)-1:
+                next_element = parent.next_sibling
+                next_sibling = None
+            else:
+                next_element = next_sibling = parent.contents[index+1]
+
+            o.previous_element = previous_element
+            if previous_element:
+                previous_element.next_element = o
+            o.next_element = next_element
+            if next_element:
+                next_element.previous_element = o
+            o.next_sibling = next_sibling
+            if next_sibling:
+                next_sibling.previous_sibling = o
+            o.previous_sibling = previous_sibling
+            if previous_sibling:
+                previous_sibling.next_sibling = o
+
     def _popToTag(self, name, nsprefix=None, inclusivePop=True):
         """Pops the tag stack up to and including the most recent
         instance of the given tag. If inclusivePop is false, pops the tag

lib/bs4/builder/__init__.py

@@ -80,9 +80,12 @@ builder_registry = TreeBuilderRegistry()
 class TreeBuilder(object):
     """Turn a document into a Beautiful Soup object tree."""
 
+    NAME = "[Unknown tree builder]"
+    ALTERNATE_NAMES = []
     features = []
 
     is_xml = False
+    picklable = False
     preserve_whitespace_tags = set()
     empty_element_tags = None # A tag will be considered an empty-element
                               # tag when and only when it has no contents.

lib/bs4/builder/_html5lib.py

@@ -2,6 +2,7 @@ __all__ = [
     'HTML5TreeBuilder',
     ]
 
+from pdb import set_trace
 import warnings
 from bs4.builder import (
     PERMISSIVE,
@@ -9,7 +10,10 @@ from bs4.builder import (
     HTML_5,
     HTMLTreeBuilder,
     )
-from bs4.element import NamespacedAttribute
+from bs4.element import (
+    NamespacedAttribute,
+    whitespace_re,
+)
 import html5lib
 from html5lib.constants import namespaces
 from bs4.element import (
@@ -22,11 +26,20 @@ from bs4.element import (
 class HTML5TreeBuilder(HTMLTreeBuilder):
     """Use html5lib to build a tree."""
 
-    features = ['html5lib', PERMISSIVE, HTML_5, HTML]
+    NAME = "html5lib"
 
-    def prepare_markup(self, markup, user_specified_encoding):
+    features = [NAME, PERMISSIVE, HTML_5, HTML]
+
+    def prepare_markup(self, markup, user_specified_encoding,
+                       document_declared_encoding=None, exclude_encodings=None):
         # Store the user-specified encoding for use later on.
         self.user_specified_encoding = user_specified_encoding
+
+        # document_declared_encoding and exclude_encodings aren't used
+        # ATM because the html5lib TreeBuilder doesn't use
+        # UnicodeDammit.
+        if exclude_encodings:
+            warnings.warn("You provided a value for exclude_encoding, but the html5lib tree builder doesn't support exclude_encoding.")
         yield (markup, None, None, False)
 
     # These methods are defined by Beautiful Soup.
@@ -101,7 +114,13 @@ class AttrList(object):
     def __iter__(self):
         return list(self.attrs.items()).__iter__()
     def __setitem__(self, name, value):
-        "set attr", name, value
+        # If this attribute is a multi-valued attribute for this element,
+        # turn its value into a list.
+        list_attr = HTML5TreeBuilder.cdata_list_attributes
+        if (name in list_attr['*']
+            or (self.element.name in list_attr
+                and name in list_attr[self.element.name])):
+            value = whitespace_re.split(value)
         self.element[name] = value
     def items(self):
         return list(self.attrs.items())
@@ -161,6 +180,12 @@ class Element(html5lib.treebuilders._base.Node):
             # immediately after the parent, if it has no children.)
             if self.element.contents:
                 most_recent_element = self.element._last_descendant(False)
+            elif self.element.next_element is not None:
+                # Something from further ahead in the parse tree is
+                # being inserted into this earlier element. This is
+                # very annoying because it means an expensive search
+                # for the last element in the tree.
+                most_recent_element = self.soup._last_descendant()
             else:
                 most_recent_element = self.element
 
@@ -172,6 +197,7 @@ class Element(html5lib.treebuilders._base.Node):
         return AttrList(self.element)
 
     def setAttributes(self, attributes):
+
         if attributes is not None and len(attributes) > 0:
 
             converted_attributes = []
@@ -218,6 +244,9 @@ class Element(html5lib.treebuilders._base.Node):
 
     def reparentChildren(self, new_parent):
         """Move all of this tag's children into another tag."""
+        # print "MOVE", self.element.contents
+        # print "FROM", self.element
+        # print "TO", new_parent.element
         element = self.element
         new_parent_element = new_parent.element
         # Determine what this tag's next_element will be once all the children
@@ -236,17 +265,28 @@ class Element(html5lib.treebuilders._base.Node):
             new_parents_last_descendant_next_element = new_parent_element.next_element
 
         to_append = element.contents
-        append_after = new_parent.element.contents
+        append_after = new_parent_element.contents
         if len(to_append) > 0:
             # Set the first child's previous_element and previous_sibling
             # to elements within the new parent
             first_child = to_append[0]
-            first_child.previous_element = new_parents_last_descendant
+            if new_parents_last_descendant:
+                first_child.previous_element = new_parents_last_descendant
+            else:
+                first_child.previous_element = new_parent_element
             first_child.previous_sibling = new_parents_last_child
+            if new_parents_last_descendant:
+                new_parents_last_descendant.next_element = first_child
+            else:
+                new_parent_element.next_element = first_child
+            if new_parents_last_child:
+                new_parents_last_child.next_sibling = first_child
 
             # Fix the last child's next_element and next_sibling
             last_child = to_append[-1]
             last_child.next_element = new_parents_last_descendant_next_element
+            if new_parents_last_descendant_next_element:
+                new_parents_last_descendant_next_element.previous_element = last_child
             last_child.next_sibling = None
 
         for child in to_append:
@@ -257,6 +297,10 @@ class Element(html5lib.treebuilders._base.Node):
         element.contents = []
         element.next_element = final_next_element
 
+        # print "DONE WITH MOVE"
+        # print "FROM", self.element
+        # print "TO", new_parent_element
+
     def cloneNode(self):
         tag = self.soup.new_tag(self.element.name, self.namespace)
         node = Element(tag, self.soup, self.namespace)
@@ -268,7 +312,7 @@ class Element(html5lib.treebuilders._base.Node):
         return self.element.contents
 
     def getNameTuple(self):
-        if self.namespace is None:
+        if self.namespace == None:
             return namespaces["html"], self.name
         else:
             return self.namespace, self.name

lib/bs4/builder/_htmlparser.py

@@ -4,10 +4,16 @@ __all__ = [
     'HTMLParserTreeBuilder',
     ]
 
-from HTMLParser import (
+from HTMLParser import HTMLParser
-    HTMLParser,
+
-    HTMLParseError,
+try:
-    )
+    from HTMLParser import HTMLParseError
+except ImportError, e:
+    # HTMLParseError is removed in Python 3.5. Since it can never be
+    # thrown in 3.5, we can just define our own class as a placeholder.
+    class HTMLParseError(Exception):
+        pass
+
 import sys
 import warnings
 
@@ -19,10 +25,10 @@ import warnings
 # At the end of this file, we monkeypatch HTMLParser so that
 # strict=True works well on Python 3.2.2.
 major, minor, release = sys.version_info[:3]
-CONSTRUCTOR_TAKES_STRICT = (
+CONSTRUCTOR_TAKES_STRICT = major == 3 and minor == 2 and release >= 3
-    major > 3
+CONSTRUCTOR_STRICT_IS_DEPRECATED = major == 3 and minor == 3
-    or (major == 3 and minor > 2)
+CONSTRUCTOR_TAKES_CONVERT_CHARREFS = major == 3 and minor >= 4
-    or (major == 3 and minor == 2 and release >= 3))
+
 
 from bs4.element import (
     CData,
@@ -63,7 +69,8 @@ class BeautifulSoupHTMLParser(HTMLParser):
 
     def handle_charref(self, name):
         # XXX workaround for a bug in HTMLParser. Remove this once
-        # it's fixed.
+        # it's fixed in all supported versions.
+        # http://bugs.python.org/issue13633
         if name.startswith('x'):
             real_name = int(name.lstrip('x'), 16)
         elif name.startswith('X'):
@@ -113,14 +120,6 @@ class BeautifulSoupHTMLParser(HTMLParser):
 
     def handle_pi(self, data):
         self.soup.endData()
-        if data.endswith("?") and data.lower().startswith("xml"):
-            # "An XHTML processing instruction using the trailing '?'
-            # will cause the '?' to be included in data." - HTMLParser
-            # docs.
-            #
-            # Strip the question mark so we don't end up with two
-            # question marks.
-            data = data[:-1]
         self.soup.handle_data(data)
         self.soup.endData(ProcessingInstruction)
 
@@ -128,15 +127,19 @@ class BeautifulSoupHTMLParser(HTMLParser):
 class HTMLParserTreeBuilder(HTMLTreeBuilder):
 
     is_xml = False
-    features = [HTML, STRICT, HTMLPARSER]
+    picklable = True
+    NAME = HTMLPARSER
+    features = [NAME, HTML, STRICT]
 
     def __init__(self, *args, **kwargs):
-        if CONSTRUCTOR_TAKES_STRICT:
+        if CONSTRUCTOR_TAKES_STRICT and not CONSTRUCTOR_STRICT_IS_DEPRECATED:
             kwargs['strict'] = False
+        if CONSTRUCTOR_TAKES_CONVERT_CHARREFS:
+            kwargs['convert_charrefs'] = False
         self.parser_args = (args, kwargs)
 
     def prepare_markup(self, markup, user_specified_encoding=None,
-                       document_declared_encoding=None):
+                       document_declared_encoding=None, exclude_encodings=None):
         """
         :return: A 4-tuple (markup, original encoding, encoding
         declared within markup, whether any characters had to be
@@ -147,7 +150,8 @@ class HTMLParserTreeBuilder(HTMLTreeBuilder):
             return
 
         try_encodings = [user_specified_encoding, document_declared_encoding]
-        dammit = UnicodeDammit(markup, try_encodings, is_html=True)
+        dammit = UnicodeDammit(markup, try_encodings, is_html=True,
+                               exclude_encodings=exclude_encodings)
         yield (dammit.markup, dammit.original_encoding,
                dammit.declared_html_encoding,
                dammit.contains_replacement_characters)

lib/bs4/builder/_lxml.py

@@ -7,7 +7,12 @@ from io import BytesIO
 from StringIO import StringIO
 import collections
 from lxml import etree
-from bs4.element import Comment, Doctype, NamespacedAttribute
+from bs4.element import (
+    Comment,
+    Doctype,
+    NamespacedAttribute,
+    ProcessingInstruction,
+)
 from bs4.builder import (
     FAST,
     HTML,
@@ -25,8 +30,11 @@ class LXMLTreeBuilderForXML(TreeBuilder):
 
     is_xml = True
 
+    NAME = "lxml-xml"
+    ALTERNATE_NAMES = ["xml"]
+
     # Well, it's permissive by XML parser standards.
-    features = [LXML, XML, FAST, PERMISSIVE]
+    features = [NAME, LXML, XML, FAST, PERMISSIVE]
 
     CHUNK_SIZE = 512
 
@@ -70,6 +78,7 @@ class LXMLTreeBuilderForXML(TreeBuilder):
             return (None, tag)
 
     def prepare_markup(self, markup, user_specified_encoding=None,
+                       exclude_encodings=None,
                        document_declared_encoding=None):
         """
         :yield: A series of 4-tuples.
@@ -95,7 +104,8 @@ class LXMLTreeBuilderForXML(TreeBuilder):
         # the document as each one in turn.
         is_html = not self.is_xml
         try_encodings = [user_specified_encoding, document_declared_encoding]
-        detector = EncodingDetector(markup, try_encodings, is_html)
+        detector = EncodingDetector(
+            markup, try_encodings, is_html, exclude_encodings)
         for encoding in detector.encodings:
             yield (detector.markup, encoding, document_declared_encoding, False)
 
@@ -189,7 +199,9 @@ class LXMLTreeBuilderForXML(TreeBuilder):
             self.nsmaps.pop()
 
     def pi(self, target, data):
-        pass
+        self.soup.endData()
+        self.soup.handle_data(target + ' ' + data)
+        self.soup.endData(ProcessingInstruction)
 
     def data(self, content):
         self.soup.handle_data(content)
@@ -212,7 +224,10 @@ class LXMLTreeBuilderForXML(TreeBuilder):
 
 class LXMLTreeBuilder(HTMLTreeBuilder, LXMLTreeBuilderForXML):
 
-    features = [LXML, HTML, FAST, PERMISSIVE]
+    NAME = LXML
+    ALTERNATE_NAMES = ["lxml-html"]
+
+    features = ALTERNATE_NAMES + [NAME, HTML, FAST, PERMISSIVE]
     is_xml = False
 
     def default_parser(self, encoding):

lib/bs4/dammit.py

@@ -3,10 +3,11 @@
 
 This library converts a bytestream to Unicode through any means
 necessary. It is heavily based on code from Mark Pilgrim's Universal
-Feed Parser. It works best on XML and XML, but it does not rewrite the
+Feed Parser. It works best on XML and HTML, but it does not rewrite the
 XML or HTML to reflect a new encoding; that's the tree builder's job.
 """
 
+from pdb import set_trace
 import codecs
 from htmlentitydefs import codepoint2name
 import re
@@ -212,8 +213,11 @@ class EncodingDetector:
 
     5. Windows-1252.
     """
-    def __init__(self, markup, override_encodings=None, is_html=False):
+    def __init__(self, markup, override_encodings=None, is_html=False,
+                 exclude_encodings=None):
         self.override_encodings = override_encodings or []
+        exclude_encodings = exclude_encodings or []
+        self.exclude_encodings = set([x.lower() for x in exclude_encodings])
         self.chardet_encoding = None
         self.is_html = is_html
         self.declared_encoding = None
@@ -224,6 +228,8 @@ class EncodingDetector:
     def _usable(self, encoding, tried):
         if encoding is not None:
             encoding = encoding.lower()
+            if encoding in self.exclude_encodings:
+                return False
             if encoding not in tried:
                 tried.add(encoding)
                 return True
@@ -266,6 +272,9 @@ class EncodingDetector:
     def strip_byte_order_mark(cls, data):
         """If a byte-order mark is present, strip it and return the encoding it implies."""
         encoding = None
+        if isinstance(data, unicode):
+            # Unicode data cannot have a byte-order mark.
+            return data, encoding
         if (len(data) >= 4) and (data[:2] == b'\xfe\xff') \
                and (data[2:4] != '\x00\x00'):
             encoding = 'utf-16be'
@@ -299,14 +308,14 @@ class EncodingDetector:
         else:
             xml_endpos = 1024
             html_endpos = max(2048, int(len(markup) * 0.05))
-
+            
         declared_encoding = None
         declared_encoding_match = xml_encoding_re.search(markup, endpos=xml_endpos)
         if not declared_encoding_match and is_html:
             declared_encoding_match = html_meta_re.search(markup, endpos=html_endpos)
         if declared_encoding_match is not None:
             declared_encoding = declared_encoding_match.groups()[0].decode(
-                'ascii')
+                'ascii', 'replace')
         if declared_encoding:
             return declared_encoding.lower()
         return None
@@ -331,13 +340,14 @@ class UnicodeDammit:
         ]
 
     def __init__(self, markup, override_encodings=[],
-                 smart_quotes_to=None, is_html=False):
+                 smart_quotes_to=None, is_html=False, exclude_encodings=[]):
         self.smart_quotes_to = smart_quotes_to
         self.tried_encodings = []
         self.contains_replacement_characters = False
         self.is_html = is_html
 
-        self.detector = EncodingDetector(markup, override_encodings, is_html)
+        self.detector = EncodingDetector(
+            markup, override_encodings, is_html, exclude_encodings)
 
         # Short-circuit if the data is in Unicode to begin with.
         if isinstance(markup, unicode) or markup == '':

lib/bs4/diagnose.py

@@ -33,12 +33,21 @@ def diagnose(data):
 
     if 'lxml' in basic_parsers:
         basic_parsers.append(["lxml", "xml"])
-        from lxml import etree
+        try:
-        print "Found lxml version %s" % ".".join(map(str,etree.LXML_VERSION))
+            from lxml import etree
+            print "Found lxml version %s" % ".".join(map(str,etree.LXML_VERSION))
+        except ImportError, e:
+            print (
+                "lxml is not installed or couldn't be imported.")
+
 
     if 'html5lib' in basic_parsers:
-        import html5lib
+        try:
-        print "Found html5lib version %s" % html5lib.__version__
+            import html5lib
+            print "Found html5lib version %s" % html5lib.__version__
+        except ImportError, e:
+            print (
+                "html5lib is not installed or couldn't be imported.")
 
     if hasattr(data, 'read'):
         data = data.read()
@@ -135,7 +144,7 @@ def rword(length=5):
 def rsentence(length=4):
     "Generate a random sentence-like string."
     return " ".join(rword(random.randint(4,9)) for i in range(length))
-
+        
 def rdoc(num_elements=1000):
     """Randomly generate an invalid HTML document."""
     tag_names = ['p', 'div', 'span', 'i', 'b', 'script', 'table']
@@ -159,7 +168,7 @@ def benchmark_parsers(num_elements=100000):
     print "Comparative parser benchmark on Beautiful Soup %s" % __version__
     data = rdoc(num_elements)
     print "Generated a large invalid HTML document (%d bytes)." % len(data)
-
+    
     for parser in ["lxml", ["lxml", "html"], "html5lib", "html.parser"]:
         success = False
         try:

lib/bs4/element.py

@@ -1,3 +1,4 @@
+from pdb import set_trace
 import collections
 import re
 import sys
@@ -185,24 +186,40 @@ class PageElement(object):
             return self.HTML_FORMATTERS.get(
                 name, HTMLAwareEntitySubstitution.substitute_xml)
 
-    def setup(self, parent=None, previous_element=None):
+    def setup(self, parent=None, previous_element=None, next_element=None,
+              previous_sibling=None, next_sibling=None):
         """Sets up the initial relations between this element and
         other elements."""
         self.parent = parent
+
         self.previous_element = previous_element
         if previous_element is not None:
             self.previous_element.next_element = self
-        self.next_element = None
+
-        self.previous_sibling = None
+        self.next_element = next_element
-        self.next_sibling = None
+        if self.next_element:
-        if self.parent is not None and self.parent.contents:
+            self.next_element.previous_element = self
-            self.previous_sibling = self.parent.contents[-1]
+
+        self.next_sibling = next_sibling
+        if self.next_sibling:
+            self.next_sibling.previous_sibling = self
+
+        if (not previous_sibling
+            and self.parent is not None and self.parent.contents):
+            previous_sibling = self.parent.contents[-1]
+
+        self.previous_sibling = previous_sibling
+        if previous_sibling:
             self.previous_sibling.next_sibling = self
 
     nextSibling = _alias("next_sibling")  # BS3
     previousSibling = _alias("previous_sibling")  # BS3
 
     def replace_with(self, replace_with):
+        if not self.parent:
+            raise ValueError(
+                "Cannot replace one element with another when the"
+                "element to be replaced is not part of a tree.")
         if replace_with is self:
             return
         if replace_with is self.parent:
@@ -216,6 +233,10 @@ class PageElement(object):
 
     def unwrap(self):
         my_parent = self.parent
+        if not self.parent:
+            raise ValueError(
+                "Cannot replace an element with its contents when that"
+                "element is not part of a tree.")
         my_index = self.parent.index(self)
         self.extract()
         for child in reversed(self.contents[:]):
@@ -240,17 +261,20 @@ class PageElement(object):
         last_child = self._last_descendant()
         next_element = last_child.next_element
 
-        if self.previous_element is not None:
+        if (self.previous_element is not None and
+            self.previous_element != next_element):
             self.previous_element.next_element = next_element
-        if next_element is not None:
+        if next_element is not None and next_element != self.previous_element:
             next_element.previous_element = self.previous_element
         self.previous_element = None
         last_child.next_element = None
 
         self.parent = None
-        if self.previous_sibling is not None:
+        if (self.previous_sibling is not None
+            and self.previous_sibling != self.next_sibling):
             self.previous_sibling.next_sibling = self.next_sibling
-        if self.next_sibling is not None:
+        if (self.next_sibling is not None
+            and self.next_sibling != self.previous_sibling):
             self.next_sibling.previous_sibling = self.previous_sibling
         self.previous_sibling = self.next_sibling = None
         return self
@@ -478,6 +502,10 @@ class PageElement(object):
     def _find_all(self, name, attrs, text, limit, generator, **kwargs):
         "Iterates over a generator looking for things that match."
 
+        if text is None and 'string' in kwargs:
+            text = kwargs['string']
+            del kwargs['string']
+
         if isinstance(name, SoupStrainer):
             strainer = name
         else:
@@ -548,17 +576,17 @@ class PageElement(object):
 
     # Methods for supporting CSS selectors.
 
-    tag_name_re = re.compile('^[a-z0-9]+$')
+    tag_name_re = re.compile('^[a-zA-Z0-9][-.a-zA-Z0-9:_]*$')
 
-    # /^(\w+)\[(\w+)([=~\|\^\$\*]?)=?"?([^\]"]*)"?\]$/
+    # /^([a-zA-Z0-9][-.a-zA-Z0-9:_]*)\[(\w+)([=~\|\^\$\*]?)=?"?([^\]"]*)"?\]$/
-    #   \---/  \---/\-------------/    \-------/
+    #   \---------------------------/  \---/\-------------/    \-------/
-    #     |      |         |               |
+    #     |                              |         |               |
-    #     |      |         |           The value
+    #     |                              |         |           The value
-    #     |      |    ~,|,^,$,* or =
+    #     |                              |    ~,|,^,$,* or =
-    #     |   Attribute
+    #     |                           Attribute
     #    Tag
     attribselect_re = re.compile(
-        r'^(?P<tag>\w+)?\[(?P<attribute>\w+)(?P<operator>[=~\|\^\$\*]?)' +
+        r'^(?P<tag>[a-zA-Z0-9][-.a-zA-Z0-9:_]*)?\[(?P<attribute>[\w-]+)(?P<operator>[=~\|\^\$\*]?)' +
         r'=?"?(?P<value>[^\]"]*)"?\]$'
         )
 
@@ -654,11 +682,17 @@ class NavigableString(unicode, PageElement):
         how to handle non-ASCII characters.
         """
         if isinstance(value, unicode):
-            return unicode.__new__(cls, value)
+            u = unicode.__new__(cls, value)
-        return unicode.__new__(cls, value, DEFAULT_OUTPUT_ENCODING)
+        else:
+            u = unicode.__new__(cls, value, DEFAULT_OUTPUT_ENCODING)
+        u.setup()
+        return u
 
     def __copy__(self):
-        return self
+        """A copy of a NavigableString has the same contents and class
+        as the original, but it is not connected to the parse tree.
+        """
+        return type(self)(self)
 
     def __getnewargs__(self):
         return (unicode(self),)
@@ -707,7 +741,7 @@ class CData(PreformattedString):
 class ProcessingInstruction(PreformattedString):
 
     PREFIX = u'<?'
-    SUFFIX = u'?>'
+    SUFFIX = u'>'
 
 class Comment(PreformattedString):
 
@@ -759,9 +793,12 @@ class Tag(PageElement):
         self.prefix = prefix
         if attrs is None:
             attrs = {}
-        elif attrs and builder.cdata_list_attributes:
+        elif attrs:
-            attrs = builder._replace_cdata_list_attribute_values(
+            if builder is not None and builder.cdata_list_attributes:
-                self.name, attrs)
+                attrs = builder._replace_cdata_list_attribute_values(
+                    self.name, attrs)
+            else:
+                attrs = dict(attrs)
         else:
             attrs = dict(attrs)
         self.attrs = attrs
@@ -778,6 +815,18 @@ class Tag(PageElement):
 
     parserClass = _alias("parser_class")  # BS3
 
+    def __copy__(self):
+        """A copy of a Tag is a new Tag, unconnected to the parse tree.
+        Its contents are a copy of the old Tag's contents.
+        """
+        clone = type(self)(None, self.builder, self.name, self.namespace,
+                           self.nsprefix, self.attrs)
+        for attr in ('can_be_empty_element', 'hidden'):
+            setattr(clone, attr, getattr(self, attr))
+        for child in self.contents:
+            clone.append(child.__copy__())
+        return clone
+
     @property
     def is_empty_element(self):
         """Is this tag an empty-element tag? (aka a self-closing tag)
@@ -971,15 +1020,25 @@ class Tag(PageElement):
         as defined in __eq__."""
         return not self == other
 
-    def __repr__(self, encoding=DEFAULT_OUTPUT_ENCODING):
+    def __repr__(self, encoding="unicode-escape"):
         """Renders this tag as a string."""
-        return self.encode(encoding)
+        if PY3K:
+            # "The return value must be a string object", i.e. Unicode
+            return self.decode()
+        else:
+            # "The return value must be a string object", i.e. a bytestring.
+            # By convention, the return value of __repr__ should also be
+            # an ASCII string.
+            return self.encode(encoding)
 
     def __unicode__(self):
         return self.decode()
 
     def __str__(self):
-        return self.encode()
+        if PY3K:
+            return self.decode()
+        else:
+            return self.encode()
 
     if PY3K:
         __str__ = __repr__ = __unicode__
@@ -1103,12 +1162,18 @@ class Tag(PageElement):
                        formatter="minimal"):
         """Renders the contents of this tag as a Unicode string.
 
+        :param indent_level: Each line of the rendering will be
+           indented this many spaces.
+
         :param eventual_encoding: The tag is destined to be
            encoded into this encoding. This method is _not_
            responsible for performing that encoding. This information
            is passed in so that it can be substituted in if the
            document contains a <META> tag that mentions the document's
            encoding.
+
+        :param formatter: The output formatter responsible for converting
+           entities to Unicode characters.
         """
         # First off, turn a string formatter into a function. This
         # will stop the lookup from happening over and over again.
@@ -1137,7 +1202,17 @@ class Tag(PageElement):
     def encode_contents(
         self, indent_level=None, encoding=DEFAULT_OUTPUT_ENCODING,
         formatter="minimal"):
-        """Renders the contents of this tag as a bytestring."""
+        """Renders the contents of this tag as a bytestring.
+
+        :param indent_level: Each line of the rendering will be
+           indented this many spaces.
+
+        :param eventual_encoding: The bytestring will be in this encoding.
+
+        :param formatter: The output formatter responsible for converting
+           entities to Unicode characters.
+        """
+
         contents = self.decode_contents(indent_level, encoding, formatter)
         return contents.encode(encoding)
 
@@ -1201,63 +1276,89 @@ class Tag(PageElement):
 
     _selector_combinators = ['>', '+', '~']
     _select_debug = False
-    def select(self, selector, _candidate_generator=None):
+    def select_one(self, selector):
         """Perform a CSS selection operation on the current element."""
-        tokens = selector.split()
+        value = self.select(selector, limit=1)
+        if value:
+            return value[0]
+        return None
+
+    def select(self, selector, _candidate_generator=None, limit=None):
+        """Perform a CSS selection operation on the current element."""
+
+        # Remove whitespace directly after the grouping operator ','
+        # then split into tokens.
+        tokens = re.sub(',[\s]*',',', selector).split()
         current_context = [self]
 
         if tokens[-1] in self._selector_combinators:
             raise ValueError(
                 'Final combinator "%s" is missing an argument.' % tokens[-1])
+
         if self._select_debug:
             print 'Running CSS selector "%s"' % selector
-        for index, token in enumerate(tokens):
+
-            if self._select_debug:
+        for index, token_group in enumerate(tokens):
-                print ' Considering token "%s"' % token
+            new_context = []
-            recursive_candidate_generator = None
+            new_context_ids = set([])
-            tag_name = None
+
+            # Grouping selectors, ie: p,a
+            grouped_tokens = token_group.split(',')
+            if '' in grouped_tokens:
+                raise ValueError('Invalid group selection syntax: %s' % token_group)
+
             if tokens[index-1] in self._selector_combinators:
                 # This token was consumed by the previous combinator. Skip it.
                 if self._select_debug:
                     print '  Token was consumed by the previous combinator.'
                 continue
-            # Each operation corresponds to a checker function, a rule
-            # for determining whether a candidate matches the
-            # selector. Candidates are generated by the active
-            # iterator.
-            checker = None
 
-            m = self.attribselect_re.match(token)
+            for token in grouped_tokens:
-            if m is not None:
+                if self._select_debug:
-                # Attribute selector
+                    print ' Considering token "%s"' % token
-                tag_name, attribute, operator, value = m.groups()
+                recursive_candidate_generator = None
-                checker = self._attribute_checker(operator, attribute, value)
+                tag_name = None
 
-            elif '#' in token:
+                # Each operation corresponds to a checker function, a rule
-                # ID selector
+                # for determining whether a candidate matches the
-                tag_name, tag_id = token.split('#', 1)
+                # selector. Candidates are generated by the active
-                def id_matches(tag):
+                # iterator.
-                    return tag.get('id', None) == tag_id
+                checker = None
-                checker = id_matches
 
-            elif '.' in token:
+                m = self.attribselect_re.match(token)
-                # Class selector
+                if m is not None:
-                tag_name, klass = token.split('.', 1)
+                    # Attribute selector
-                classes = set(klass.split('.'))
+                    tag_name, attribute, operator, value = m.groups()
-                def classes_match(candidate):
+                    checker = self._attribute_checker(operator, attribute, value)
-                    return classes.issubset(candidate.get('class', []))
-                checker = classes_match
 
-            elif ':' in token:
+                elif '#' in token:
-                # Pseudo-class
+                    # ID selector
-                tag_name, pseudo = token.split(':', 1)
+                    tag_name, tag_id = token.split('#', 1)
-                if tag_name == '':
+                    def id_matches(tag):
-                    raise ValueError(
+                        return tag.get('id', None) == tag_id
-                        "A pseudo-class must be prefixed with a tag name.")
+                    checker = id_matches
-                pseudo_attributes = re.match('([a-zA-Z\d-]+)\(([a-zA-Z\d]+)\)', pseudo)
+
-                found = []
+                elif '.' in token:
-                if pseudo_attributes is not None:
+                    # Class selector
-                    pseudo_type, pseudo_value = pseudo_attributes.groups()
+                    tag_name, klass = token.split('.', 1)
+                    classes = set(klass.split('.'))
+                    def classes_match(candidate):
+                        return classes.issubset(candidate.get('class', []))
+                    checker = classes_match
+
+                elif ':' in token:
+                    # Pseudo-class
+                    tag_name, pseudo = token.split(':', 1)
+                    if tag_name == '':
+                        raise ValueError(
+                            "A pseudo-class must be prefixed with a tag name.")
+                    pseudo_attributes = re.match('([a-zA-Z\d-]+)\(([a-zA-Z\d]+)\)', pseudo)
+                    found = []
+                    if pseudo_attributes is None:
+                        pseudo_type = pseudo
+                        pseudo_value = None
+                    else:
+                        pseudo_type, pseudo_value = pseudo_attributes.groups()
                     if pseudo_type == 'nth-of-type':
                         try:
                             pseudo_value = int(pseudo_value)
@@ -1286,109 +1387,110 @@ class Tag(PageElement):
                         raise NotImplementedError(
                             'Only the following pseudo-classes are implemented: nth-of-type.')
 
-            elif token == '*':
+                elif token == '*':
-                # Star selector -- matches everything
+                    # Star selector -- matches everything
-                pass
+                    pass
-            elif token == '>':
+                elif token == '>':
-                # Run the next token as a CSS selector against the
+                    # Run the next token as a CSS selector against the
-                # direct children of each tag in the current context.
+                    # direct children of each tag in the current context.
-                recursive_candidate_generator = lambda tag: tag.children
+                    recursive_candidate_generator = lambda tag: tag.children
-            elif token == '~':
+                elif token == '~':
-                # Run the next token as a CSS selector against the
+                    # Run the next token as a CSS selector against the
-                # siblings of each tag in the current context.
+                    # siblings of each tag in the current context.
-                recursive_candidate_generator = lambda tag: tag.next_siblings
+                    recursive_candidate_generator = lambda tag: tag.next_siblings
-            elif token == '+':
+                elif token == '+':
-                # For each tag in the current context, run the next
+                    # For each tag in the current context, run the next
-                # token as a CSS selector against the tag's next
+                    # token as a CSS selector against the tag's next
-                # sibling that's a tag.
+                    # sibling that's a tag.
-                def next_tag_sibling(tag):
+                    def next_tag_sibling(tag):
-                    yield tag.find_next_sibling(True)
+                        yield tag.find_next_sibling(True)
-                recursive_candidate_generator = next_tag_sibling
+                    recursive_candidate_generator = next_tag_sibling
 
-            elif self.tag_name_re.match(token):
+                elif self.tag_name_re.match(token):
-                # Just a tag name.
+                    # Just a tag name.
-                tag_name = token
+                    tag_name = token
-            else:
-                raise ValueError(
-                    'Unsupported or invalid CSS selector: "%s"' % token)
-
-            if recursive_candidate_generator:
-                # This happens when the selector looks like  "> foo".
-                #
-                # The generator calls select() recursively on every
-                # member of the current context, passing in a different
-                # candidate generator and a different selector.
-                #
-                # In the case of "> foo", the candidate generator is
-                # one that yields a tag's direct children (">"), and
-                # the selector is "foo".
-                next_token = tokens[index+1]
-                def recursive_select(tag):
-                    if self._select_debug:
-                        print '    Calling select("%s") recursively on %s %s' % (next_token, tag.name, tag.attrs)
-                        print '-' * 40
-                    for i in tag.select(next_token, recursive_candidate_generator):
-                        if self._select_debug:
-                            print '(Recursive select picked up candidate %s %s)' % (i.name, i.attrs)
-                        yield i
-                    if self._select_debug:
-                        print '-' * 40
-                _use_candidate_generator = recursive_select
-            elif _candidate_generator is None:
-                # By default, a tag's candidates are all of its
-                # children. If tag_name is defined, only yield tags
-                # with that name.
-                if self._select_debug:
-                    if tag_name:
-                        check = "[any]"
-                    else:
-                        check = tag_name
-                    print '   Default candidate generator, tag name="%s"' % check
-                if self._select_debug:
-                    # This is redundant with later code, but it stops
-                    # a bunch of bogus tags from cluttering up the
-                    # debug log.
-                    def default_candidate_generator(tag):
-                        for child in tag.descendants:
-                            if not isinstance(child, Tag):
-                                continue
-                            if tag_name and not child.name == tag_name:
-                                continue
-                            yield child
-                    _use_candidate_generator = default_candidate_generator
                 else:
-                    _use_candidate_generator = lambda tag: tag.descendants
+                    raise ValueError(
-            else:
+                        'Unsupported or invalid CSS selector: "%s"' % token)
-                _use_candidate_generator = _candidate_generator
+                if recursive_candidate_generator:
-
+                    # This happens when the selector looks like  "> foo".
-            new_context = []
+                    #
-            new_context_ids = set([])
+                    # The generator calls select() recursively on every
-            for tag in current_context:
+                    # member of the current context, passing in a different
-                if self._select_debug:
+                    # candidate generator and a different selector.
-                    print "    Running candidate generator on %s %s" % (
+                    #
-                        tag.name, repr(tag.attrs))
+                    # In the case of "> foo", the candidate generator is
-                for candidate in _use_candidate_generator(tag):
+                    # one that yields a tag's direct children (">"), and
-                    if not isinstance(candidate, Tag):
+                    # the selector is "foo".
-                        continue
+                    next_token = tokens[index+1]
-                    if tag_name and candidate.name != tag_name:
+                    def recursive_select(tag):
-                        continue
-                    if checker is not None:
-                        try:
-                            result = checker(candidate)
-                        except StopIteration:
-                            # The checker has decided we should no longer
-                            # run the generator.
-                            break
-                    if checker is None or result:
                         if self._select_debug:
-                            print "     SUCCESS %s %s" % (candidate.name, repr(candidate.attrs))
+                            print '    Calling select("%s") recursively on %s %s' % (next_token, tag.name, tag.attrs)
-                        if id(candidate) not in new_context_ids:
+                            print '-' * 40
-                            # If a tag matches a selector more than once,
+                        for i in tag.select(next_token, recursive_candidate_generator):
-                            # don't include it in the context more than once.
+                            if self._select_debug:
-                            new_context.append(candidate)
+                                print '(Recursive select picked up candidate %s %s)' % (i.name, i.attrs)
-                            new_context_ids.add(id(candidate))
+                            yield i
-                    elif self._select_debug:
+                        if self._select_debug:
-                        print "     FAILURE %s %s" % (candidate.name, repr(candidate.attrs))
+                            print '-' * 40
+                    _use_candidate_generator = recursive_select
+                elif _candidate_generator is None:
+                    # By default, a tag's candidates are all of its
+                    # children. If tag_name is defined, only yield tags
+                    # with that name.
+                    if self._select_debug:
+                        if tag_name:
+                            check = "[any]"
+                        else:
+                            check = tag_name
+                        print '   Default candidate generator, tag name="%s"' % check
+                    if self._select_debug:
+                        # This is redundant with later code, but it stops
+                        # a bunch of bogus tags from cluttering up the
+                        # debug log.
+                        def default_candidate_generator(tag):
+                            for child in tag.descendants:
+                                if not isinstance(child, Tag):
+                                    continue
+                                if tag_name and not child.name == tag_name:
+                                    continue
+                                yield child
+                        _use_candidate_generator = default_candidate_generator
+                    else:
+                        _use_candidate_generator = lambda tag: tag.descendants
+                else:
+                    _use_candidate_generator = _candidate_generator
+
+                count = 0
+                for tag in current_context:
+                    if self._select_debug:
+                        print "    Running candidate generator on %s %s" % (
+                            tag.name, repr(tag.attrs))
+                    for candidate in _use_candidate_generator(tag):
+                        if not isinstance(candidate, Tag):
+                            continue
+                        if tag_name and candidate.name != tag_name:
+                            continue
+                        if checker is not None:
+                            try:
+                                result = checker(candidate)
+                            except StopIteration:
+                                # The checker has decided we should no longer
+                                # run the generator.
+                                break
+                        if checker is None or result:
+                            if self._select_debug:
+                                print "     SUCCESS %s %s" % (candidate.name, repr(candidate.attrs))
+                            if id(candidate) not in new_context_ids:
+                                # If a tag matches a selector more than once,
+                                # don't include it in the context more than once.
+                                new_context.append(candidate)
+                                new_context_ids.add(id(candidate))
+                                if limit and len(new_context) >= limit:
+                                    break
+                        elif self._select_debug:
+                            print "     FAILURE %s %s" % (candidate.name, repr(candidate.attrs))
+
 
             current_context = new_context
 

lib/bs4/testing.py

@@ -1,5 +1,6 @@
 """Helper classes for tests."""
 
+import pickle
 import copy
 import functools
 import unittest
@@ -43,6 +44,16 @@ class SoupTest(unittest.TestCase):
 
         self.assertEqual(obj.decode(), self.document_for(compare_parsed_to))
 
+    def assertConnectedness(self, element):
+        """Ensure that next_element and previous_element are properly
+        set for all descendants of the given element.
+        """
+        earlier = None
+        for e in element.descendants:
+            if earlier:
+                self.assertEqual(e, earlier.next_element)
+                self.assertEqual(earlier, e.previous_element)
+            earlier = e
 
 class HTMLTreeBuilderSmokeTest(object):
 
@@ -54,6 +65,15 @@ class HTMLTreeBuilderSmokeTest(object):
     markup in these tests, there's not much room for interpretation.
     """
 
+    def test_pickle_and_unpickle_identity(self):
+        # Pickling a tree, then unpickling it, yields a tree identical
+        # to the original.
+        tree = self.soup("<a><b>foo</a>")
+        dumped = pickle.dumps(tree, 2)
+        loaded = pickle.loads(dumped)
+        self.assertEqual(loaded.__class__, BeautifulSoup)
+        self.assertEqual(loaded.decode(), tree.decode())
+
     def assertDoctypeHandled(self, doctype_fragment):
         """Assert that a given doctype string is handled correctly."""
         doctype_str, soup = self._document_with_doctype(doctype_fragment)
@@ -114,6 +134,11 @@ class HTMLTreeBuilderSmokeTest(object):
             soup.encode("utf-8").replace(b"\n", b""),
             markup.replace(b"\n", b""))
 
+    def test_processing_instruction(self):
+        markup = b"""<?PITarget PIContent?>"""
+        soup = self.soup(markup)
+        self.assertEqual(markup, soup.encode("utf8"))
+
     def test_deepcopy(self):
         """Make sure you can copy the tree builder.
 
@@ -155,6 +180,23 @@ class HTMLTreeBuilderSmokeTest(object):
     def test_nested_formatting_elements(self):
         self.assertSoupEquals("<em><em></em></em>")
 
+    def test_double_head(self):
+        html = '''<!DOCTYPE html>
+<html>
+<head>
+<title>Ordinary HEAD element test</title>
+</head>
+<script type="text/javascript">
+alert("Help!");
+</script>
+<body>
+Hello, world!
+</body>
+</html>
+'''
+        soup = self.soup(html)
+        self.assertEqual("text/javascript", soup.find('script')['type'])
+
     def test_comment(self):
         # Comments are represented as Comment objects.
         markup = "<p>foo<!--foobar-->baz</p>"
@@ -221,6 +263,14 @@ class HTMLTreeBuilderSmokeTest(object):
         soup = self.soup(markup)
         self.assertEqual(["css"], soup.div.div['class'])
 
+    def test_multivalued_attribute_on_html(self):
+        # html5lib uses a different API to set the attributes ot the
+        # <html> tag. This has caused problems with multivalued
+        # attributes.
+        markup = '<html class="a b"></html>'
+        soup = self.soup(markup)
+        self.assertEqual(["a", "b"], soup.html['class'])
+
     def test_angle_brackets_in_attribute_values_are_escaped(self):
         self.assertSoupEquals('<a b="<a>"></a>', '<a b="&lt;a&gt;"></a>')
 
@@ -253,6 +303,35 @@ class HTMLTreeBuilderSmokeTest(object):
         soup = self.soup("<html><h2>\nfoo</h2><p></p></html>")
         self.assertEqual("p", soup.h2.string.next_element.name)
         self.assertEqual("p", soup.p.name)
+        self.assertConnectedness(soup)
+
+    def test_head_tag_between_head_and_body(self):
+        "Prevent recurrence of a bug in the html5lib treebuilder."
+        content = """<html><head></head>
+  <link></link>
+  <body>foo</body>
+</html>
+"""
+        soup = self.soup(content)
+        self.assertNotEqual(None, soup.html.body)
+        self.assertConnectedness(soup)
+
+    def test_multiple_copies_of_a_tag(self):
+        "Prevent recurrence of a bug in the html5lib treebuilder."
+        content = """<!DOCTYPE html>
+<html>
+ <body>
+   <article id="a" >
+   <div><a href="1"></div>
+   <footer>
+     <a href="2"></a>
+   </footer>
+  </article>
+  </body>
+</html>
+"""
+        soup = self.soup(content)
+        self.assertConnectedness(soup.article)
 
     def test_basic_namespaces(self):
         """Parsers don't need to *understand* namespaces, but at the
@@ -463,6 +542,15 @@ class HTMLTreeBuilderSmokeTest(object):
 
 class XMLTreeBuilderSmokeTest(object):
 
+    def test_pickle_and_unpickle_identity(self):
+        # Pickling a tree, then unpickling it, yields a tree identical
+        # to the original.
+        tree = self.soup("<a><b>foo</a>")
+        dumped = pickle.dumps(tree, 2)
+        loaded = pickle.loads(dumped)
+        self.assertEqual(loaded.__class__, BeautifulSoup)
+        self.assertEqual(loaded.decode(), tree.decode())
+
     def test_docstring_generated(self):
         soup = self.soup("<root/>")
         self.assertEqual(
@@ -485,7 +573,7 @@ class XMLTreeBuilderSmokeTest(object):
   <script type="text/javascript">
   </script>
 """
-        soup = BeautifulSoup(doc, "xml")
+        soup = BeautifulSoup(doc, "lxml-xml")
         # lxml would have stripped this while parsing, but we can add
         # it later.
         soup.script.string = 'console.log("< < hey > > ");'

lib/enum/LICENSE

@@ -0,0 +1,32 @@
+Copyright (c) 2013, Ethan Furman.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+
+    Redistributions of source code must retain the above
+    copyright notice, this list of conditions and the
+    following disclaimer.
+
+    Redistributions in binary form must reproduce the above
+    copyright notice, this list of conditions and the following
+    disclaimer in the documentation and/or other materials
+    provided with the distribution.
+
+    Neither the name Ethan Furman nor the names of any
+    contributors may be used to endorse or promote products
+    derived from this software without specific prior written
+    permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.

lib/enum/README

@@ -0,0 +1,2 @@
+enum34 is the new Python stdlib enum module available in Python 3.4
+backported for previous versions of Python from 2.4 to 3.3.

lib/enum/__init__.py

@@ -0,0 +1,790 @@
+"""Python Enumerations"""
+
+import sys as _sys
+
+__all__ = ['Enum', 'IntEnum', 'unique']
+
+version = 1, 0, 4
+
+pyver = float('%s.%s' % _sys.version_info[:2])
+
+try:
+    any
+except NameError:
+    def any(iterable):
+        for element in iterable:
+            if element:
+                return True
+        return False
+
+try:
+    from collections import OrderedDict
+except ImportError:
+    OrderedDict = None
+
+try:
+    basestring
+except NameError:
+    # In Python 2 basestring is the ancestor of both str and unicode
+    # in Python 3 it's just str, but was missing in 3.1
+    basestring = str
+
+try:
+    unicode
+except NameError:
+    # In Python 3 unicode no longer exists (it's just str)
+    unicode = str
+
+class _RouteClassAttributeToGetattr(object):
+    """Route attribute access on a class to __getattr__.
+
+    This is a descriptor, used to define attributes that act differently when
+    accessed through an instance and through a class.  Instance access remains
+    normal, but access to an attribute through a class will be routed to the
+    class's __getattr__ method; this is done by raising AttributeError.
+
+    """
+    def __init__(self, fget=None):
+        self.fget = fget
+
+    def __get__(self, instance, ownerclass=None):
+        if instance is None:
+            raise AttributeError()
+        return self.fget(instance)
+
+    def __set__(self, instance, value):
+        raise AttributeError("can't set attribute")
+
+    def __delete__(self, instance):
+        raise AttributeError("can't delete attribute")
+
+
+def _is_descriptor(obj):
+    """Returns True if obj is a descriptor, False otherwise."""
+    return (
+            hasattr(obj, '__get__') or
+            hasattr(obj, '__set__') or
+            hasattr(obj, '__delete__'))
+
+
+def _is_dunder(name):
+    """Returns True if a __dunder__ name, False otherwise."""
+    return (name[:2] == name[-2:] == '__' and
+            name[2:3] != '_' and
+            name[-3:-2] != '_' and
+            len(name) > 4)
+
+
+def _is_sunder(name):
+    """Returns True if a _sunder_ name, False otherwise."""
+    return (name[0] == name[-1] == '_' and 
+            name[1:2] != '_' and
+            name[-2:-1] != '_' and
+            len(name) > 2)
+
+
+def _make_class_unpicklable(cls):
+    """Make the given class un-picklable."""
+    def _break_on_call_reduce(self, protocol=None):
+        raise TypeError('%r cannot be pickled' % self)
+    cls.__reduce_ex__ = _break_on_call_reduce
+    cls.__module__ = '<unknown>'
+
+
+class _EnumDict(dict):
+    """Track enum member order and ensure member names are not reused.
+
+    EnumMeta will use the names found in self._member_names as the
+    enumeration member names.
+
+    """
+    def __init__(self):
+        super(_EnumDict, self).__init__()
+        self._member_names = []
+
+    def __setitem__(self, key, value):
+        """Changes anything not dundered or not a descriptor.
+
+        If a descriptor is added with the same name as an enum member, the name
+        is removed from _member_names (this may leave a hole in the numerical
+        sequence of values).
+
+        If an enum member name is used twice, an error is raised; duplicate
+        values are not checked for.
+
+        Single underscore (sunder) names are reserved.
+
+        Note:   in 3.x __order__ is simply discarded as a not necessary piece
+                leftover from 2.x
+
+        """
+        if pyver >= 3.0 and key == '__order__':
+                return
+        if _is_sunder(key):
+            raise ValueError('_names_ are reserved for future Enum use')
+        elif _is_dunder(key):
+            pass
+        elif key in self._member_names:
+            # descriptor overwriting an enum?
+            raise TypeError('Attempted to reuse key: %r' % key)
+        elif not _is_descriptor(value):
+            if key in self:
+                # enum overwriting a descriptor?
+                raise TypeError('Key already defined as: %r' % self[key])
+            self._member_names.append(key)
+        super(_EnumDict, self).__setitem__(key, value)
+
+
+# Dummy value for Enum as EnumMeta explicity checks for it, but of course until
+# EnumMeta finishes running the first time the Enum class doesn't exist.  This
+# is also why there are checks in EnumMeta like `if Enum is not None`
+Enum = None
+
+
+class EnumMeta(type):
+    """Metaclass for Enum"""
+    @classmethod
+    def __prepare__(metacls, cls, bases):
+        return _EnumDict()
+
+    def __new__(metacls, cls, bases, classdict):
+        # an Enum class is final once enumeration items have been defined; it
+        # cannot be mixed with other types (int, float, etc.) if it has an
+        # inherited __new__ unless a new __new__ is defined (or the resulting
+        # class will fail).
+        if type(classdict) is dict:
+            original_dict = classdict
+            classdict = _EnumDict()
+            for k, v in original_dict.items():
+                classdict[k] = v
+
+        member_type, first_enum = metacls._get_mixins_(bases)
+        __new__, save_new, use_args = metacls._find_new_(classdict, member_type,
+                                                        first_enum)
+        # save enum items into separate mapping so they don't get baked into
+        # the new class
+        members = dict((k, classdict[k]) for k in classdict._member_names)
+        for name in classdict._member_names:
+            del classdict[name]
+
+        # py2 support for definition order
+        __order__ = classdict.get('__order__')
+        if __order__ is None:
+            if pyver < 3.0:
+                try:
+                    __order__ = [name for (name, value) in sorted(members.items(), key=lambda item: item[1])]
+                except TypeError:
+                    __order__ = [name for name in sorted(members.keys())]
+            else:
+                __order__ = classdict._member_names
+        else:
+            del classdict['__order__']
+            if pyver < 3.0:
+                __order__ = __order__.replace(',', ' ').split()
+                aliases = [name for name in members if name not in __order__]
+                __order__ += aliases
+
+        # check for illegal enum names (any others?)
+        invalid_names = set(members) & set(['mro'])
+        if invalid_names:
+            raise ValueError('Invalid enum member name(s): %s' % (
+                ', '.join(invalid_names), ))
+
+        # create our new Enum type
+        enum_class = super(EnumMeta, metacls).__new__(metacls, cls, bases, classdict)
+        enum_class._member_names_ = []               # names in random order
+        if OrderedDict is not None:
+            enum_class._member_map_ = OrderedDict()
+        else:
+            enum_class._member_map_ = {}             # name->value map
+        enum_class._member_type_ = member_type
+
+        # Reverse value->name map for hashable values.
+        enum_class._value2member_map_ = {}
+
+        # instantiate them, checking for duplicates as we go
+        # we instantiate first instead of checking for duplicates first in case
+        # a custom __new__ is doing something funky with the values -- such as
+        # auto-numbering ;)
+        if __new__ is None:
+            __new__ = enum_class.__new__
+        for member_name in __order__:
+            value = members[member_name]
+            if not isinstance(value, tuple):
+                args = (value, )
+            else:
+                args = value
+            if member_type is tuple:   # special case for tuple enums
+                args = (args, )     # wrap it one more time
+            if not use_args or not args:
+                enum_member = __new__(enum_class)
+                if not hasattr(enum_member, '_value_'):
+                    enum_member._value_ = value
+            else:
+                enum_member = __new__(enum_class, *args)
+                if not hasattr(enum_member, '_value_'):
+                    enum_member._value_ = member_type(*args)
+            value = enum_member._value_
+            enum_member._name_ = member_name
+            enum_member.__objclass__ = enum_class
+            enum_member.__init__(*args)
+            # If another member with the same value was already defined, the
+            # new member becomes an alias to the existing one.
+            for name, canonical_member in enum_class._member_map_.items():
+                if canonical_member.value == enum_member._value_:
+                    enum_member = canonical_member
+                    break
+            else:
+                # Aliases don't appear in member names (only in __members__).
+                enum_class._member_names_.append(member_name)
+            enum_class._member_map_[member_name] = enum_member
+            try:
+                # This may fail if value is not hashable. We can't add the value
+                # to the map, and by-value lookups for this value will be
+                # linear.
+                enum_class._value2member_map_[value] = enum_member
+            except TypeError:
+                pass
+
+
+        # If a custom type is mixed into the Enum, and it does not know how
+        # to pickle itself, pickle.dumps will succeed but pickle.loads will
+        # fail.  Rather than have the error show up later and possibly far
+        # from the source, sabotage the pickle protocol for this class so
+        # that pickle.dumps also fails.
+        #
+        # However, if the new class implements its own __reduce_ex__, do not
+        # sabotage -- it's on them to make sure it works correctly.  We use
+        # __reduce_ex__ instead of any of the others as it is preferred by
+        # pickle over __reduce__, and it handles all pickle protocols.
+        unpicklable = False
+        if '__reduce_ex__' not in classdict:
+            if member_type is not object:
+                methods = ('__getnewargs_ex__', '__getnewargs__',
+                        '__reduce_ex__', '__reduce__')
+                if not any(m in member_type.__dict__ for m in methods):
+                    _make_class_unpicklable(enum_class)
+                    unpicklable = True
+
+
+        # double check that repr and friends are not the mixin's or various
+        # things break (such as pickle)
+        for name in ('__repr__', '__str__', '__format__', '__reduce_ex__'):
+            class_method = getattr(enum_class, name)
+            obj_method = getattr(member_type, name, None)
+            enum_method = getattr(first_enum, name, None)
+            if name not in classdict and class_method is not enum_method:
+                if name == '__reduce_ex__' and unpicklable:
+                    continue
+                setattr(enum_class, name, enum_method)
+
+        # method resolution and int's are not playing nice
+        # Python's less than 2.6 use __cmp__
+
+        if pyver < 2.6:
+
+            if issubclass(enum_class, int):
+                setattr(enum_class, '__cmp__', getattr(int, '__cmp__'))
+
+        elif pyver < 3.0:
+
+            if issubclass(enum_class, int):
+                for method in (
+                        '__le__',
+                        '__lt__',
+                        '__gt__',
+                        '__ge__',
+                        '__eq__',
+                        '__ne__',
+                        '__hash__',
+                        ):
+                    setattr(enum_class, method, getattr(int, method))
+
+        # replace any other __new__ with our own (as long as Enum is not None,
+        # anyway) -- again, this is to support pickle
+        if Enum is not None:
+            # if the user defined their own __new__, save it before it gets
+            # clobbered in case they subclass later
+            if save_new:
+                setattr(enum_class, '__member_new__', enum_class.__dict__['__new__'])
+            setattr(enum_class, '__new__', Enum.__dict__['__new__'])
+        return enum_class
+
+    def __call__(cls, value, names=None, module=None, type=None):
+        """Either returns an existing member, or creates a new enum class.
+
+        This method is used both when an enum class is given a value to match
+        to an enumeration member (i.e. Color(3)) and for the functional API
+        (i.e. Color = Enum('Color', names='red green blue')).
+
+        When used for the functional API: `module`, if set, will be stored in
+        the new class' __module__ attribute; `type`, if set, will be mixed in
+        as the first base class.
+
+        Note: if `module` is not set this routine will attempt to discover the
+        calling module by walking the frame stack; if this is unsuccessful
+        the resulting class will not be pickleable.
+
+        """
+        if names is None:  # simple value lookup
+            return cls.__new__(cls, value)
+        # otherwise, functional API: we're creating a new Enum type
+        return cls._create_(value, names, module=module, type=type)
+
+    def __contains__(cls, member):
+        return isinstance(member, cls) and member.name in cls._member_map_
+
+    def __delattr__(cls, attr):
+        # nicer error message when someone tries to delete an attribute
+        # (see issue19025).
+        if attr in cls._member_map_:
+            raise AttributeError(
+                    "%s: cannot delete Enum member." % cls.__name__)
+        super(EnumMeta, cls).__delattr__(attr)
+
+    def __dir__(self):
+        return (['__class__', '__doc__', '__members__', '__module__'] +
+                self._member_names_)
+
+    @property
+    def __members__(cls):
+        """Returns a mapping of member name->value.
+
+        This mapping lists all enum members, including aliases. Note that this
+        is a copy of the internal mapping.
+
+        """
+        return cls._member_map_.copy()
+
+    def __getattr__(cls, name):
+        """Return the enum member matching `name`
+
+        We use __getattr__ instead of descriptors or inserting into the enum
+        class' __dict__ in order to support `name` and `value` being both
+        properties for enum members (which live in the class' __dict__) and
+        enum members themselves.
+
+        """
+        if _is_dunder(name):
+            raise AttributeError(name)
+        try:
+            return cls._member_map_[name]
+        except KeyError:
+            raise AttributeError(name)
+
+    def __getitem__(cls, name):
+        return cls._member_map_[name]
+
+    def __iter__(cls):
+        return (cls._member_map_[name] for name in cls._member_names_)
+
+    def __reversed__(cls):
+        return (cls._member_map_[name] for name in reversed(cls._member_names_))
+
+    def __len__(cls):
+        return len(cls._member_names_)
+
+    def __repr__(cls):
+        return "<enum %r>" % cls.__name__
+
+    def __setattr__(cls, name, value):
+        """Block attempts to reassign Enum members.
+
+        A simple assignment to the class namespace only changes one of the
+        several possible ways to get an Enum member from the Enum class,
+        resulting in an inconsistent Enumeration.
+
+        """
+        member_map = cls.__dict__.get('_member_map_', {})
+        if name in member_map:
+            raise AttributeError('Cannot reassign members.')
+        super(EnumMeta, cls).__setattr__(name, value)
+
+    def _create_(cls, class_name, names=None, module=None, type=None):
+        """Convenience method to create a new Enum class.
+
+        `names` can be:
+
+        * A string containing member names, separated either with spaces or
+          commas.  Values are auto-numbered from 1.
+        * An iterable of member names.  Values are auto-numbered from 1.
+        * An iterable of (member name, value) pairs.
+        * A mapping of member name -> value.
+
+        """
+        if pyver < 3.0:
+            # if class_name is unicode, attempt a conversion to ASCII
+            if isinstance(class_name, unicode):
+                try:
+                    class_name = class_name.encode('ascii')
+                except UnicodeEncodeError:
+                    raise TypeError('%r is not representable in ASCII' % class_name)
+        metacls = cls.__class__
+        if type is None:
+            bases = (cls, )
+        else:
+            bases = (type, cls)
+        classdict = metacls.__prepare__(class_name, bases)
+        __order__ = []
+
+        # special processing needed for names?
+        if isinstance(names, basestring):
+            names = names.replace(',', ' ').split()
+        if isinstance(names, (tuple, list)) and isinstance(names[0], basestring):
+            names = [(e, i+1) for (i, e) in enumerate(names)]
+
+        # Here, names is either an iterable of (name, value) or a mapping.
+        for item in names:
+            if isinstance(item, basestring):
+                member_name, member_value = item, names[item]
+            else:
+                member_name, member_value = item
+            classdict[member_name] = member_value
+            __order__.append(member_name)
+        # only set __order__ in classdict if name/value was not from a mapping
+        if not isinstance(item, basestring):
+            classdict['__order__'] = ' '.join(__order__)
+        enum_class = metacls.__new__(metacls, class_name, bases, classdict)
+
+        # TODO: replace the frame hack if a blessed way to know the calling
+        # module is ever developed
+        if module is None:
+            try:
+                module = _sys._getframe(2).f_globals['__name__']
+            except (AttributeError, ValueError):
+                pass
+        if module is None:
+            _make_class_unpicklable(enum_class)
+        else:
+            enum_class.__module__ = module
+
+        return enum_class
+
+    @staticmethod
+    def _get_mixins_(bases):
+        """Returns the type for creating enum members, and the first inherited
+        enum class.
+
+        bases: the tuple of bases that was given to __new__
+
+        """
+        if not bases or Enum is None:
+            return object, Enum
+        
+
+        # double check that we are not subclassing a class with existing
+        # enumeration members; while we're at it, see if any other data
+        # type has been mixed in so we can use the correct __new__
+        member_type = first_enum = None
+        for base in bases:
+            if  (base is not Enum and
+                    issubclass(base, Enum) and
+                    base._member_names_):
+                raise TypeError("Cannot extend enumerations")
+        # base is now the last base in bases
+        if not issubclass(base, Enum):
+            raise TypeError("new enumerations must be created as "
+                    "`ClassName([mixin_type,] enum_type)`")
+
+        # get correct mix-in type (either mix-in type of Enum subclass, or
+        # first base if last base is Enum)
+        if not issubclass(bases[0], Enum):
+            member_type = bases[0]     # first data type
+            first_enum = bases[-1]  # enum type
+        else:
+            for base in bases[0].__mro__:
+                # most common: (IntEnum, int, Enum, object)
+                # possible:    (<Enum 'AutoIntEnum'>, <Enum 'IntEnum'>,
+                #               <class 'int'>, <Enum 'Enum'>,
+                #               <class 'object'>)
+                if issubclass(base, Enum):
+                    if first_enum is None:
+                        first_enum = base
+                else:
+                    if member_type is None:
+                        member_type = base
+
+        return member_type, first_enum
+
+    if pyver < 3.0:
+        @staticmethod
+        def _find_new_(classdict, member_type, first_enum):
+            """Returns the __new__ to be used for creating the enum members.
+
+            classdict: the class dictionary given to __new__
+            member_type: the data type whose __new__ will be used by default
+            first_enum: enumeration to check for an overriding __new__
+
+            """
+            # now find the correct __new__, checking to see of one was defined
+            # by the user; also check earlier enum classes in case a __new__ was
+            # saved as __member_new__
+            __new__ = classdict.get('__new__', None)
+            if __new__:
+                return None, True, True      # __new__, save_new, use_args
+
+            N__new__ = getattr(None, '__new__')
+            O__new__ = getattr(object, '__new__')
+            if Enum is None:
+                E__new__ = N__new__
+            else:
+                E__new__ = Enum.__dict__['__new__']
+            # check all possibles for __member_new__ before falling back to
+            # __new__
+            for method in ('__member_new__', '__new__'):
+                for possible in (member_type, first_enum):
+                    try:
+                        target = possible.__dict__[method]
+                    except (AttributeError, KeyError):
+                        target = getattr(possible, method, None)
+                    if target not in [
+                            None,
+                            N__new__,
+                            O__new__,
+                            E__new__,
+                            ]:
+                        if method == '__member_new__':
+                            classdict['__new__'] = target
+                            return None, False, True
+                        if isinstance(target, staticmethod):
+                            target = target.__get__(member_type)
+                        __new__ = target
+                        break
+                if __new__ is not None:
+                    break
+            else:
+                __new__ = object.__new__
+
+            # if a non-object.__new__ is used then whatever value/tuple was
+            # assigned to the enum member name will be passed to __new__ and to the
+            # new enum member's __init__
+            if __new__ is object.__new__:
+                use_args = False
+            else:
+                use_args = True
+
+            return __new__, False, use_args
+    else:
+        @staticmethod
+        def _find_new_(classdict, member_type, first_enum):
+            """Returns the __new__ to be used for creating the enum members.
+
+            classdict: the class dictionary given to __new__
+            member_type: the data type whose __new__ will be used by default
+            first_enum: enumeration to check for an overriding __new__
+
+            """
+            # now find the correct __new__, checking to see of one was defined
+            # by the user; also check earlier enum classes in case a __new__ was
+            # saved as __member_new__
+            __new__ = classdict.get('__new__', None)
+
+            # should __new__ be saved as __member_new__ later?
+            save_new = __new__ is not None
+
+            if __new__ is None:
+                # check all possibles for __member_new__ before falling back to
+                # __new__
+                for method in ('__member_new__', '__new__'):
+                    for possible in (member_type, first_enum):
+                        target = getattr(possible, method, None)
+                        if target not in (
+                                None,
+                                None.__new__,
+                                object.__new__,
+                                Enum.__new__,
+                                ):
+                            __new__ = target
+                            break
+                    if __new__ is not None:
+                        break
+                else:
+                    __new__ = object.__new__
+
+            # if a non-object.__new__ is used then whatever value/tuple was
+            # assigned to the enum member name will be passed to __new__ and to the
+            # new enum member's __init__
+            if __new__ is object.__new__:
+                use_args = False
+            else:
+                use_args = True
+
+            return __new__, save_new, use_args
+
+
+########################################################
+# In order to support Python 2 and 3 with a single
+# codebase we have to create the Enum methods separately
+# and then use the `type(name, bases, dict)` method to
+# create the class.
+########################################################
+temp_enum_dict = {}
+temp_enum_dict['__doc__'] = "Generic enumeration.\n\n    Derive from this class to define new enumerations.\n\n"
+
+def __new__(cls, value):
+    # all enum instances are actually created during class construction
+    # without calling this method; this method is called by the metaclass'
+    # __call__ (i.e. Color(3) ), and by pickle
+    if type(value) is cls:
+        # For lookups like Color(Color.red)
+        value = value.value
+        #return value
+    # by-value search for a matching enum member
+    # see if it's in the reverse mapping (for hashable values)
+    try:
+        if value in cls._value2member_map_:
+            return cls._value2member_map_[value]
+    except TypeError:
+        # not there, now do long search -- O(n) behavior
+        for member in cls._member_map_.values():
+            if member.value == value:
+                return member
+    raise ValueError("%s is not a valid %s" % (value, cls.__name__))
+temp_enum_dict['__new__'] = __new__
+del __new__
+
+def __repr__(self):
+    return "<%s.%s: %r>" % (
+            self.__class__.__name__, self._name_, self._value_)
+temp_enum_dict['__repr__'] = __repr__
+del __repr__
+
+def __str__(self):
+    return "%s.%s" % (self.__class__.__name__, self._name_)
+temp_enum_dict['__str__'] = __str__
+del __str__
+
+def __dir__(self):
+    added_behavior = [
+            m
+            for cls in self.__class__.mro()
+            for m in cls.__dict__
+            if m[0] != '_'
+            ]
+    return (['__class__', '__doc__', '__module__', ] + added_behavior)
+temp_enum_dict['__dir__'] = __dir__
+del __dir__
+
+def __format__(self, format_spec):
+    # mixed-in Enums should use the mixed-in type's __format__, otherwise
+    # we can get strange results with the Enum name showing up instead of
+    # the value
+
+    # pure Enum branch
+    if self._member_type_ is object:
+        cls = str
+        val = str(self)
+    # mix-in branch
+    else:
+        cls = self._member_type_
+        val = self.value
+    return cls.__format__(val, format_spec)
+temp_enum_dict['__format__'] = __format__
+del __format__
+
+
+####################################
+# Python's less than 2.6 use __cmp__
+
+if pyver < 2.6:
+
+    def __cmp__(self, other):
+        if type(other) is self.__class__:
+            if self is other:
+                return 0
+            return -1
+        return NotImplemented
+        raise TypeError("unorderable types: %s() and %s()" % (self.__class__.__name__, other.__class__.__name__))
+    temp_enum_dict['__cmp__'] = __cmp__
+    del __cmp__
+
+else:
+
+    def __le__(self, other):
+        raise TypeError("unorderable types: %s() <= %s()" % (self.__class__.__name__, other.__class__.__name__))
+    temp_enum_dict['__le__'] = __le__
+    del __le__
+
+    def __lt__(self, other):
+        raise TypeError("unorderable types: %s() < %s()" % (self.__class__.__name__, other.__class__.__name__))
+    temp_enum_dict['__lt__'] = __lt__
+    del __lt__
+
+    def __ge__(self, other):
+        raise TypeError("unorderable types: %s() >= %s()" % (self.__class__.__name__, other.__class__.__name__))
+    temp_enum_dict['__ge__'] = __ge__
+    del __ge__
+
+    def __gt__(self, other):
+        raise TypeError("unorderable types: %s() > %s()" % (self.__class__.__name__, other.__class__.__name__))
+    temp_enum_dict['__gt__'] = __gt__
+    del __gt__
+    
+
+def __eq__(self, other):
+    if type(other) is self.__class__:
+        return self is other
+    return NotImplemented
+temp_enum_dict['__eq__'] = __eq__
+del __eq__
+
+def __ne__(self, other):
+    if type(other) is self.__class__:
+        return self is not other
+    return NotImplemented
+temp_enum_dict['__ne__'] = __ne__
+del __ne__
+
+def __hash__(self):
+    return hash(self._name_)
+temp_enum_dict['__hash__'] = __hash__
+del __hash__
+
+def __reduce_ex__(self, proto):
+    return self.__class__, (self._value_, )
+temp_enum_dict['__reduce_ex__'] = __reduce_ex__
+del __reduce_ex__
+
+# _RouteClassAttributeToGetattr is used to provide access to the `name`
+# and `value` properties of enum members while keeping some measure of
+# protection from modification, while still allowing for an enumeration
+# to have members named `name` and `value`.  This works because enumeration
+# members are not set directly on the enum class -- __getattr__ is
+# used to look them up.
+
+@_RouteClassAttributeToGetattr
+def name(self):
+    return self._name_
+temp_enum_dict['name'] = name
+del name
+
+@_RouteClassAttributeToGetattr
+def value(self):
+    return self._value_
+temp_enum_dict['value'] = value
+del value
+
+Enum = EnumMeta('Enum', (object, ), temp_enum_dict)
+del temp_enum_dict
+
+# Enum has now been created
+###########################
+
+class IntEnum(int, Enum):
+    """Enum where members are also (and must be) ints"""
+
+
+def unique(enumeration):
+    """Class decorator that ensures only unique members exist in an enumeration."""
+    duplicates = []
+    for name, member in enumeration.__members__.items():
+        if name != member.name:
+            duplicates.append((name, member.name))
+    if duplicates:
+        duplicate_names = ', '.join(
+                ["%s -> %s" % (alias, name) for (alias, name) in duplicates]
+                )
+        raise ValueError('duplicate names found in %r: %s' %
+                (enumeration, duplicate_names)
+                )
+    return enumeration