v0.5.10

Updated changelog for v0.5.10
Removed pylint from travis
2026-09-10 00:32:52 +01:00 · 2016-01-29 16:30:40 +00:00 · 2016-01-29 16:29:53 +00:00 · 2016-01-29 16:19:45 +00:00 · 2016-01-29 16:09:03 +00:00 · 2016-01-29 15:59:50 +00:00
107 changed files with 5891 additions and 3527 deletions
@@ -16,6 +16,5 @@ install:
 - pip install pep8
 script:
  - pep8 headphones
-  - pylint --rcfile=pylintrc headphones
  - pyflakes headphones
  - nosetests headphones
@@ -66,7 +66,7 @@ Unmark album as wanted / i.e. mark as skipped

 ### forceSearch
 force search for wanted albums - not launched in a separate thread so it may take a bit to complete
-### forceProcess
+### forceProcess[&dir=/path/to/folder]
 Force post process albums in download directory - also not launched in a separate thread
 ### forceActiveArtistsUpdate
 force Active Artist Update - also not launched in a separate thread
@@ -1,5 +1,71 @@
 # Changelog

+## v0.5.10
+Released 29 January 2016
+
+Highlights:
+* Added: API option to post-process single folders
+* Added: Ability to specify extension when re-encoding
+* Added: Option to stop renaming folders
+* Fixed: Utorrent torrents not being removed (#2385)
+* Fixed: Torznab to transmission
+* Fixed: Magnet folder names in history
+* Fixed: Multiple torcache fixes
+* Fixed: Updated requests & urllib3 to latest versions to fix errors with pyOpenSSL
+* Improved: Use a temporary folder during post-processing
+* Improved: Added verify_ssl_cert option
+* Improved: Fixed track matching progress
+* Improved: pylint, pep8 & pylint fixes
+* Improved: Stop JS links from scrolling to the top of the page
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.9...v0.5.10).
+
+## v0.5.9
+Released 05 September 2015
+
+Highlights:
+* Added: Providers Strike, Jackett, custom Torznabs
+* Added: Option to stop post-processing if no good match found (#2343)
+* Fixed: Blackhole -> Magnet, limit to torcache
+* Fixed: Kat 403 flac error
+* Fixed: Last.fm errors
+* Fixed: Pushover notifications
+* Improved: Rutracker logging, switched to requests lib
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.8...v0.5.9).
+
+## v0.5.8
+Released 13 July 2015
+
+Highlights:
+* Added: Option to only include official extras
+* Added: Option to wait until album release date before searching
+* Fixed: NotifyMyAndroid notifications
+* Fixed: Plex Notifications
+* Fixed: Metacritic parsing
+* Fixed: Pushbullet notifications
+* Fixed: What.cd not honoring custom search term (#2279)
+* Improved: XSS Search bug
+* Improved: Config page layout
+* Improved: Set localhost as default
+* Improved: Better single artist scanning
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.7...v0.5.8).
+
+## v0.5.7
+Released 01 July 2015
+
+Highlights:
+* Improved: Moved pushover to use requests lib
+* Improved: Plex tokens with Plex Home
+* Improved: Added getLogs & clearLogs to api
+* Improved: Cache MetaCritic scores. Added user-agent header to fix 403 errors
+* Improved: Specify whether to delete folders when force post-processing
+* Improved: Convert target bitrate to vbr preset for what.cd searching
+* Improved: Switched Pushover to requests lib
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.6...v0.5.7).
+
 ## v0.5.6
 Released 08 June 2015

@@ -1,4 +1,7 @@
-#![Headphones Logo](https://github.com/rembo10/headphones/raw/master/data/images/headphoneslogo.png) Headphones
+##![Headphones Logo](https://github.com/rembo10/headphones/raw/master/data/images/headphoneslogo.png) Headphones
+
+**Master Branch:**  [![Build Status](https://travis-ci.org/rembo10/headphones.svg?branch=master)](https://travis-ci.org/rembo10/headphones)
+**Develop Branch:** [![Build Status](https://travis-ci.org/rembo10/headphones.svg?branch=develop)](https://travis-ci.org/rembo10/headphones)

 Headphones is an automated music downloader for NZB and Torrent, written in Python. It supports SABnzbd, NZBget, Transmission, µTorrent and Blackhole.

@@ -19,7 +22,7 @@ You are free to join the Headphones support community on IRC where you can ask q
 1. Analyze your log, you just might find the solution yourself!
 2. You read the wiki and searched existing issues, but this is not solving your problem.
 3. Post the issue with a clear title, description and the HP log and use [proper markdown syntax](https://help.github.com/articles/github-flavored-markdown) to structure your text (code/log in code blocks).
-4. Close your issue when it's solved! If you found the solution yourself please comment so that others benefit from it.
+4. Close your issue when it's solved! If you found the solution yourself, please comment so that others benefit from it.

 **Feature requests** can be reported on the GitHub issue tracker too:

@@ -0,0 +1,145 @@
+#!/usr/bin/env python
+
+import os
+import sys
+
+# Ensure that we use the Headphones provided libraries.
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "../lib"))
+
+import urlparse
+
+
+def can_import(module):
+    """
+    Return True if a given module can be imported or not.
+    """
+
+    try:
+        __import__(module)
+    except ImportError:
+        return False
+
+    # Module can be imported
+    return True
+
+
+def check_installation():
+    """
+    Check if some core modules are available. Info is based on this topic:
+    https://github.com/rembo10/headphones/issues/2210.
+    """
+
+    if can_import("requests"):
+        import requests
+        requests_version = requests.__version__
+    else:
+        requests_version = "no"
+
+    if can_import("OpenSSL"):
+        import OpenSSL
+        openssl_version = OpenSSL.__version__
+    else:
+        openssl_version = "no"
+
+    if can_import("cryptography"):
+        import cryptography
+        cryptography_version = cryptography.__version__
+    else:
+        cryptography_version = "no"
+
+    if can_import("pyasn1"):
+        import pyasn1
+        pyasn1_version = pyasn1.__version__
+    else:
+        pyasn1_version = "no"
+
+    if can_import("ndg.httpsclient"):
+        from ndg import httpsclient
+        ndg_version = httpsclient.__date__
+    else:
+        ndg_version = "no"
+
+    # Print some system information.
+    sys.stdout.write(
+        "* Checking Python version: %s.%s.%s\n" % sys.version_info[:3])
+    sys.stdout.write("* Operating system: %s\n" % sys.platform)
+
+    sys.stdout.write(
+        "* Checking if requests can be imported: %s\n" % requests_version)
+    sys.stdout.write(
+        "* Checking if pyOpenSSL is installed: %s\n" % openssl_version)
+    sys.stdout.write(
+        "* Checking if cryptography is installed: %s\n" % cryptography_version)
+    sys.stdout.write(
+        "* Checking if pyasn1 is installed: %s\n" % pyasn1_version)
+    sys.stdout.write(
+        "* Checking if ndg.httpsclient is installed: %s\n" % ndg_version)
+
+
+def main():
+    """
+    Test if the current Headphones installation can connect to SNI-enabled
+    servers.
+    """
+
+    # Read the URL to test.
+    if len(sys.argv) == 1:
+        url = "https://sni.velox.ch/"
+    else:
+        url = sys.argv[1]
+
+    # Check if it is a HTTPS website.
+    parts = urlparse.urlparse(url)
+
+    if parts.scheme.lower() != "https":
+        sys.stderr.write(
+            "Error: provided URL does not start with https://\n")
+        return 1
+
+    # Gather information
+    check_installation()
+
+    # Do the request.
+    if not can_import("requests"):
+        sys.stderr.exit("Error: cannot continue without requests module!\n")
+        return 1
+
+    sys.stdout.write("* Performing request: %s\n" % url)
+
+    import requests
+    requests.packages.urllib3.disable_warnings()
+
+    try:
+        try:
+            response = requests.get(url)
+        except requests.exceptions.SSLError as e:
+            sys.stdout.write(
+                "- Server certificate seems invalid. I will disable "
+                "certificate check and try again. You'll see the real "
+                "exception if it fails again.\n")
+            sys.stdout.write(
+                "* Retrying request with certificate verification off.\n")
+            response = requests.get(url)
+    except Exception as e:
+        sys.stdout.write(
+            "- An error occured while performing the request. The "
+            "exception was: %s\n" % e.message)
+        sys.stdout.write(
+            "- Consult the Troubleshooting wiki (https://github.com/"
+            "rembo10/headphones/wiki/Troubleshooting) before you post an "
+            "issue!")
+        return 0
+
+    # Verify the response.
+    if response.status_code == 200:
+        sys.stdout.write("+ Got a valid response. All seems OK!\n")
+    else:
+        sys.stdout.write(
+            "- Server returned status code %s. Expected a status code 200.\n",
+            response.status_code)
+        sys.stdout.write(
+            "- However, I was able to communicate to the server!\n")
+
+# E.g. `python sni_test.py https://example.org'.
+if __name__ == "__main__":
+    sys.exit(main())
@@ -14,15 +14,15 @@

 			<a id="menu_link_delete" href="deleteAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}"><i class="fa fa-trash-o"></i> Delete Album</a>
 			%if album['Status'] == 'Skipped' or album['Status'] == 'Ignored':
-			<a id="menu_link_wanted" href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=False', $(this),true)" data-success="'${album['AlbumTitle']}' added to queue"><i class="fa fa-heart"></i> Mark Album as Wanted</a>
+			<a id="menu_link_wanted" href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=False', $(this),true)" data-success="'${album['AlbumTitle']}' added to queue"><i class="fa fa-heart"></i> Mark Album as Wanted</a>
 			%elif album['Status'] == 'Wanted':
-			<a id="menu_link_check" href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=True', $(this));" data-success="Forced checking successful"><i class="fa fa-search"></i> Force Check</a>
-			<a id="menu_link_skipped" href="#" onclick="doAjaxCall('unqueueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}', $(this),true);" data-success="'${album['AlbumTitle']}' marked as Skipped"><i class="fa fa-step-forward"></i> Mark Album as Skipped</a>
+			<a id="menu_link_check" href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=True', $(this));" data-success="Forced checking successful"><i class="fa fa-search"></i> Force Check</a>
+			<a id="menu_link_skipped" href="javascript:void(0)" onclick="doAjaxCall('unqueueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}', $(this),true);" data-success="'${album['AlbumTitle']}' marked as Skipped"><i class="fa fa-step-forward"></i> Mark Album as Skipped</a>
 			%else:
-			<a id="menu_link_retry" href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=False', $(this),true);" data-success="Retrying the same version of '${album['AlbumTitle']}'"><i class="fa fa-refresh"></i> Retry Download</a>
-			<a id="menu_link_new" href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=True', $(this),true);" data-success="Looking for a new version of '${album['AlbumTitle']}'"><i class="fa fa-download"></i> Try New Version</a>
+			<a id="menu_link_retry" href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=False', $(this),true);" data-success="Retrying the same version of '${album['AlbumTitle']}'"><i class="fa fa-refresh"></i> Retry Download</a>
+			<a id="menu_link_new" href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=True', $(this),true);" data-success="Looking for a new version of '${album['AlbumTitle']}'"><i class="fa fa-download"></i> Try New Version</a>
 			%endif		
-			<a class="menu_link_edit" id="album_chooser" href="#"><i class="fa fa-pencil"></i> Choose Alternate Release</a>
+			<a class="menu_link_edit" id="album_chooser" href="javascript:void(0)"><i class="fa fa-pencil"></i> Choose Alternate Release</a>
 				<div id="dialog" title="Choose an Alternate Release" style="display:none" class="configtable">
 					<div class="links">
 						<%
@@ -30,11 +30,12 @@
 						%>
 						%if not alternate_albums:
 							<p>No alternate releases found. Try refreshing the artist (if the artist is being refreshed, please wait until it's finished)</p>
-							<h2><a id="refresh_artist" onclick="doAjaxCall('refreshArtist?ArtistID=${album['ArtistID']}', $(this)), true" href="#" data-success="'${album['ArtistName']}' is being refreshed">Refresh Artist</a></h2>
+							<h2><a id="refresh_artist" onclick="doAjaxCall('refreshArtist?ArtistID=${album['ArtistID']}', $(this)), true" href="javascript:void(0)" data-success="'${album['ArtistName']}' is being refreshed">Refresh Artist</a></h2>
 						%else:
 							%for alternate_album in alternate_albums:
 								<%
 									track_count = len(myDB.select("SELECT * from alltracks WHERE ReleaseID=?", [alternate_album['ReleaseID']]))
+									mb_link = "http://musicbrainz.org/release/" + alternate_album['ReleaseID']
 									have_track_count = len(myDB.select("SELECT * from alltracks WHERE ReleaseID=? AND Location IS NOT NULL", [alternate_album['ReleaseID']]))
 									if alternate_album['AlbumID'] == alternate_album['ReleaseID']:
 										alternate_album_name = "Headphones Default Release (" + str(alternate_album['ReleaseDate']) + ") [" + str(have_track_count) + "/" + str(track_count) + " tracks]"
@@ -42,12 +43,12 @@
 										alternate_album_name = alternate_album['AlbumTitle'] + " (" + alternate_album['ReleaseCountry'] + ", " + str(alternate_album['ReleaseDate']) + ", " + alternate_album['ReleaseFormat'] + ") [" + str(have_track_count) + "/" + str(track_count) + " tracks]"
 									
 								%>
-								<a href="#" onclick="doAjaxCall('switchAlbum?AlbumID=${album['AlbumID']}&ReleaseID=${alternate_album['ReleaseID']}', $(this), 'table');" data-success="Switched release to: ${alternate_album_name}">${alternate_album_name}</a><br>
+								<a href="javascript:void(0)" onclick="doAjaxCall('switchAlbum?AlbumID=${album['AlbumID']}&ReleaseID=${alternate_album['ReleaseID']}', $(this), 'table');" data-success="Switched release to: ${alternate_album_name}">${alternate_album_name}</a><a href="${mb_link}" target="_blank">MB</a><br>
 							%endfor
 						%endif
 					</div>
 				</div>
-			<a class="menu_link_edit" id="edit_search_term" href="#"><i class="fa fa-pencil"></i> Edit Search Term</a>
+			<a class="menu_link_edit" id="edit_search_term" href="javascript:void(0)"><i class="fa fa-pencil"></i> Edit Search Term</a>
 			<div id="dialog2" title="Enter your own search term for this album" style="display:none" class="configtable">
 				<form action="editSearchTerm" method="GET" id="editSearchTerm">
 					<input type="hidden" name="AlbumID" value="${album['AlbumID']}">
@@ -63,7 +64,7 @@
 					<input type="button" value="Save changes" onclick="doAjaxCall('editSearchTerm',$(this),'tabs',true);return false;" data-success="Search term updated"/>
 				</form>
 			</div>
-			<a class="menu_link_edit" id="choose_specific_download" href="#" onclick="getAvailableDownloads()"><i class="fa fa-search"></i> Choose Specific Download</a>
+			<a class="menu_link_edit" id="choose_specific_download" href="javascript:void(0)" onclick="getAvailableDownloads()"><i class="fa fa-search"></i> Choose Specific Download</a>
 			<div id="choose_specific_download_dialog" title="Choose a specific download for this album" style="display:none" class="configtable">
 				<table class="display" id="downloads_table">
 					<thead>
@@ -237,7 +238,7 @@
 			feedback.fadeOut();
 			search_results = data
 			for( var i = 0, len = data.length; i < len; i++ ) {
-    			$('#downloads_table_body').append('<tr><td id="title"><a href="#" onclick="downloadSpecificRelease('+i+')">'+data[i].title+'</a></td><td id="size"><span title='+data[i].size+'></span>'+(data[i].size / (1024*1024)).toFixed(2)+' MB</td><td id="provider">'+data[i].provider+'</td><td id="kind">'+data[i].kind+'</td><td id="matches">'+data[i].matches+'</td></tr>');
+    			$('#downloads_table_body').append('<tr><td id="title"><a href="javascript:void(0)" onclick="downloadSpecificRelease('+i+')">'+data[i].title+'</a></td><td id="size"><span title='+data[i].size+'></span>'+(data[i].size / (1024*1024)).toFixed(2)+' MB</td><td id="provider">'+data[i].provider+'</td><td id="kind">'+data[i].kind+'</td><td id="matches">'+data[i].matches+'</td></tr>');
    		}
 			$('#downloads_table').dataTable({
 				"aoColumns": [
@@ -8,19 +8,19 @@
 <%def name="headerIncludes()">
 	<div id="subhead_container">
 		<div id="subhead_menu">
-			<a id="menu_link_refresh" onclick="doSimpleAjaxCall('refreshArtist?ArtistID=${artist['ArtistID']}')" href="#"><i class="fa fa-refresh"></i> Refresh Artist</a>
+			<a id="menu_link_refresh" onclick="doSimpleAjaxCall('refreshArtist?ArtistID=${artist['ArtistID']}')" href="javascript:void(0)"><i class="fa fa-refresh"></i> Refresh Artist</a>
 			<a id="menu_link_delete" href="deleteArtist?ArtistID=${artist['ArtistID']}"><i class="fa fa-trash-o"></i> Delete Artist</a>
-			<a id="menu_link_scan" onclick="doAjaxCall('scanArtist?ArtistID=${artist['ArtistID']}', $(this)),'table'" href="#" data-success="'${artist['ArtistName']}' was scanned"><i class="fa fa-refresh"></i> Scan Artist</a>
+			<a id="menu_link_scan" onclick="doAjaxCall('scanArtist?ArtistID=${artist['ArtistID']}', $(this)),'table'" href="javascript:void(0)" data-success="'${artist['ArtistName']}' was scanned"><i class="fa fa-refresh"></i> Scan Artist</a>
 			%if artist['Status'] == 'Paused':
-				<a id="menu_link_resume" href="#" onclick="doAjaxCall('resumeArtist?ArtistID=${artist['ArtistID']}',$(this),true)" data-success="${artist['ArtistName']} resumed"><i class="fa fa-play"></i> Resume Artist</a>
+				<a id="menu_link_resume" href="javascript:void(0)" onclick="doAjaxCall('resumeArtist?ArtistID=${artist['ArtistID']}',$(this),true)" data-success="${artist['ArtistName']} resumed"><i class="fa fa-play"></i> Resume Artist</a>
 			%else:
-				<a id="menu_link_pauze" href="#" onclick="doAjaxCall('pauseArtist?ArtistID=${artist['ArtistID']}',$(this),true)" data-success="${artist['ArtistName']} paused"><i class="fa fa-pause"></i> Pause Artist</a>
+				<a id="menu_link_pauze" href="javascript:void(0)" onclick="doAjaxCall('pauseArtist?ArtistID=${artist['ArtistID']}',$(this),true)" data-success="${artist['ArtistName']} paused"><i class="fa fa-pause"></i> Pause Artist</a>
 			%endif
 			%if artist['IncludeExtras']:
-				<a id="menu_link_removeextra" href="#" onclick="doAjaxCall('removeExtras?ArtistID=${artist['ArtistID']}&ArtistName=${artist['ArtistName']}',$(this),'submenu&table')" data-success="Extras removed for ${artist['ArtistName']}"><i class="fa fa-minus"></i> Remove Extras</a>
-				<a class="menu_link_edit" id="menu_link_modifyextra" href="#"><i class="fa fa-pencil"></i> Modify Extras</a>
+				<a id="menu_link_removeextra" href="javascript:void(0)" onclick="doAjaxCall('removeExtras?ArtistID=${artist['ArtistID']}&ArtistName=${artist['ArtistName']}',$(this),'submenu&table')" data-success="Extras removed for ${artist['ArtistName']}"><i class="fa fa-minus"></i> Remove Extras</a>
+				<a class="menu_link_edit" id="menu_link_modifyextra" href="javascript:void(0)"><i class="fa fa-pencil"></i> Modify Extras</a>
 			%else:
-				<a id="menu_link_getextra" href="#"><i class="fa fa-plus"></i> Get Extras</a>
+				<a id="menu_link_getextra" href="javascript:void(0)"><i class="fa fa-plus"></i> Get Extras</a>
 			%endif
 				<div id="dialog" title="Choose Which Extras to Fetch" style="display:none" class="configtable">
 					<form action="getExtras" method="get" class="form">
@@ -129,16 +129,16 @@
 				<td id="score">${album['CriticScore']}/${album['UserScore']}</td>
 				<td id="status">${album['Status']}
 				%if album['Status'] == 'Skipped' or album['Status'] == 'Ignored':
-					[<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}',$(this),'table')" data-success="'${album['AlbumTitle']}' added to Wanted list">want</a>]
+					[<a href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}',$(this),'table')" data-success="'${album['AlbumTitle']}' added to Wanted list">want</a>]
 				%elif (album['Status'] == 'Wanted' or album['Status'] == 'Wanted Lossless'):
-                    [<a href="#" onclick="doAjaxCall('unqueueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}',$(this),'table')" data-success="'${album['AlbumTitle']}' skipped">skip</a>] [<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}', $(this),'table')" data-success="Trying to download'${album['AlbumTitle']}'" title="Search if available for download">search</a>]
+                    [<a href="javascript:void(0)" onclick="doAjaxCall('unqueueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}',$(this),'table')" data-success="'${album['AlbumTitle']}' skipped">skip</a>] [<a href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}', $(this),'table')" data-success="Trying to download'${album['AlbumTitle']}'" title="Search if available for download">search</a>]
 				%else:
-					[<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}', $(this),'table')" data-success="Retrying the same version of '${album['AlbumTitle']}'" title="Retry the same download again">retry</a>][<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=True', $(this),'table')" title="Try a new download, skipping all previously tried nzbs"  data-success="Downloading new version for '${album['AlbumTitle']}'" data-success="Looking for a new version of '${album['AlbumTitle']}'">new</a>]
+					[<a href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}', $(this),'table')" data-success="Retrying the same version of '${album['AlbumTitle']}'" title="Retry the same download again">retry</a>][<a href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=True', $(this),'table')" title="Try a new download, skipping all previously tried nzbs"  data-success="Downloading new version for '${album['AlbumTitle']}'" data-success="Looking for a new version of '${album['AlbumTitle']}'">new</a>]
 				%endif
 				%if albumformat in lossy_formats and album['Status'] == 'Skipped':
-					[<a id="wantlossless" href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&lossless=True', $(this),'table')" data-success="Lossless version of '${album['AlbumTitle']}' added to queue">want lossless</a>]
+					[<a id="wantlossless" href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&lossless=True', $(this),'table')" data-success="Lossless version of '${album['AlbumTitle']}' added to queue">want lossless</a>]
 				%elif albumformat in lossy_formats and (album['Status'] == 'Snatched' or album['Status'] == 'Downloaded'):
-					[<a id="wantlossless" href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&lossless=True', $(this),'table')" data-success="Retrying the same lossless version of '${album['AlbumTitle']}'">retry lossless</a>]
+					[<a id="wantlossless" href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&lossless=True', $(this),'table')" data-success="Retrying the same lossless version of '${album['AlbumTitle']}'">retry lossless</a>]
 				%endif
 				</td>
 				<td id="have"><span title="${percent}"><span><div class="progress-container"><div style="width:${percent}%"><div class="havetracks">${havetracks}/${totaltracks}</div></div></div></td>
@@ -34,11 +34,11 @@
 		% if headphones.CONFIG.CHECK_GITHUB and not headphones.CURRENT_VERSION:
 			<div id="updatebar">
 				You're running an unknown version of Headphones. <a href="update">Update</a> or
-				<a href="#" onclick="$('#updatebar').slideUp('slow');">Close</a>
+				<a href="javascript:void(0)" onclick="$('#updatebar').slideUp('slow');">Close</a>
 			</div>
 		% elif headphones.CONFIG.CHECK_GITHUB and headphones.CURRENT_VERSION != headphones.LATEST_VERSION and headphones.COMMITS_BEHIND > 0 and headphones.INSTALL_TYPE != 'win':
 			<div id="updatebar">
-				A <a href="https://github.com/${headphones.CONFIG.GIT_USER}/headphones/compare/${headphones.CURRENT_VERSION}...${headphones.LATEST_VERSION}"> newer version</a> is available. You're ${headphones.COMMITS_BEHIND} commits behind. <a href="update">Update</a> or <a href="#" onclick="$('#updatebar').slideUp('slow');">Close</a>
+				A <a href="https://github.com/${headphones.CONFIG.GIT_USER}/headphones/compare/${headphones.CURRENT_VERSION}...${headphones.LATEST_VERSION}"> newer version</a> is available. You're ${headphones.COMMITS_BEHIND} commits behind. <a href="update">Update</a> or <a href="javascript:void(0)" onclick="$('#updatebar').slideUp('slow');">Close</a>
 			</div>
 		% endif

@@ -89,7 +89,7 @@
 				<small>
 				<a href="shutdown"><i class="fa fa-power-off"></i> Shutdown</a> |
 				<a href="restart"><i class="fa fa-power-off"></i> Restart</a> |
-				<a href="#" onclick="doAjaxCall('checkGithub',$(this))" data-success="Checking for update successful" data-error="Error checking for update"><i class="fa fa-refresh"></i> Check for new version</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('checkGithub',$(this))" data-success="Checking for update successful" data-error="Error checking for update"><i class="fa fa-refresh"></i> Check for new version</a>
 				</small>
 			</div>
 			<div id="version">
@@ -325,7 +325,7 @@ form fieldset small.heading {
  margin-top: -15px;
 }
 form .row {
-  font-family: Helvetica, Arial;
+  font-family: Helvetica, Arial, sans-serif;
  margin-bottom: 10px;
 }
 form .row label {
@@ -334,7 +334,7 @@ form .row label {
  font-size: 12px;
  line-height: normal;
  padding-top: 7px;
-  width: 175px;
+  width: 170px;
 }
 form .row label.inline {
  margin-right: 5px;
@@ -368,7 +368,7 @@ form .row small {
  display: block;
  font-size: 9px;
  line-height: 12px;
-  margin-left: 175px;
+  margin-left: 170px;
  margin-top: 3px;
 }
 form .left label {
@@ -408,6 +408,18 @@ form .checkbox small {
 form .indent input {
  margin-left: 15px;
 }
+form .suboptions {
+  margin-left: 15px;
+}
+.option{
+  font-size: 15px;
+  font-weight: 600;
+  vertical-align: middle;
+}
+input.bigcheck[type="checkbox"] {
+  width: 16px;
+  height: 16px;
+}
 ul,
 ol {
  margin-left: 2em;
@@ -1420,6 +1432,13 @@ div#artistheader h2 a {
  line-height: 10px;
  height: 10px;
 }
+.nopad {
+  margin-bottom: 0px !important;
+  margin-top: 0px !important;
+  padding-top: 0px !important;
+  padding-bottom: 0px !important;
+  padding: 0px 0px !important;
+}
 #album_table th#albumname,
 #upcoming_table th#artistname,
 #wanted_table th#artistname {
@@ -190,7 +190,7 @@ form {
            font-size: 12px;
            line-height: normal;
            padding-top: 7px;
-            width: 175px;
+            width: 170px;

            &.inline {
                margin-right: 5px;
@@ -216,7 +216,7 @@ form {
            display: block;
            font-size: 9px;
            line-height: 12px;
-            margin-left: 175px;
+            margin-left: 170px;
            margin-top: 3px;
        }
    }
@@ -7,11 +7,11 @@
 <%def name="headerIncludes()">
 	<div id="subhead_container">
 		<div id="subhead_menu">
-			<a id="menu_link_delete" href="#" onclick="doAjaxCall('clearhistory?type=all',$(this),'table')" data-success="All History cleared"><i class="fa fa-trash-o"></i> Clear All History</a>
-			<a id="menu_link_delete" href="#" onclick="doAjaxCall('clearhistory?type=Processed',$(this),'table')" data-success="All Processed cleared"><i class="fa fa-trash-o"></i> Clear Processed</a>
-			<a id="menu_link_delete" href="#" onclick="doAjaxCall('clearhistory?type=Unprocessed',$(this),'table')" data-success="All Unprocessed cleared"><i class="fa fa-trash-o"></i> Clear Unprocessed</a>
-			<a id="menu_link_delete" href="#" onclick="doAjaxCall('clearhistory?type=Frozen',$(this),'table')" data-success="All Frozen cleared"><i class="fa fa-trash-o"></i> Clear Frozen</a>
-			<a id="menu_link_delete" href="#" onclick="doAjaxCall('clearhistory?type=Snatched',$(this),'table')" data-success="All Snatched cleared"><i class="fa fa-trash-o"></i> Clear Snatched</a>
+			<a id="menu_link_delete" href="javascript:void(0)" onclick="doAjaxCall('clearhistory?type=all',$(this),'table')" data-success="All History cleared"><i class="fa fa-trash-o"></i> Clear All History</a>
+			<a id="menu_link_delete" href="javascript:void(0)" onclick="doAjaxCall('clearhistory?type=Processed',$(this),'table')" data-success="All Processed cleared"><i class="fa fa-trash-o"></i> Clear Processed</a>
+			<a id="menu_link_delete" href="javascript:void(0)" onclick="doAjaxCall('clearhistory?type=Unprocessed',$(this),'table')" data-success="All Unprocessed cleared"><i class="fa fa-trash-o"></i> Clear Unprocessed</a>
+			<a id="menu_link_delete" href="javascript:void(0)" onclick="doAjaxCall('clearhistory?type=Frozen',$(this),'table')" data-success="All Frozen cleared"><i class="fa fa-trash-o"></i> Clear Frozen</a>
+			<a id="menu_link_delete" href="javascript:void(0)" onclick="doAjaxCall('clearhistory?type=Snatched',$(this),'table')" data-success="All Snatched cleared"><i class="fa fa-trash-o"></i> Clear Snatched</a>
 		</div>
 	</div>
 </%def>
@@ -50,8 +50,12 @@
                fileid = 'nzb'
            if item['URL'].find('torrent') != -1:
                fileid = 'torrent'
+            if item['URL'].find('magnet:') != -1:
+                fileid = 'torrent'
            if item['URL'].find('rutracker') != -1:
                fileid = 'torrent'
+            if item['URL'].find('codeshy') != -1:
+                fileid = 'nzb'

            folder = 'Folder: ' + item['FolderName']

@@ -61,8 +65,8 @@
 				<td id="filename">${cgi.escape(item['Title'], quote=True)} [<a href="${item['URL']}">${fileid}</a>]<a href="albumPage?AlbumID=${item['AlbumID']}">[album page]</a></td>
 				<td id="size">${helpers.bytes_to_mb(item['Size'])}</td>
 				<td title="${folder}" id="status">${item['Status']}</td>
-				<td id="action">[<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${item['AlbumID']}&redirect=history', $(this),'table')"  data-success="Retrying download of '${cgi.escape(item['Title'], quote=True)}'">retry</a>][<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${item['AlbumID']}&new=True&redirect=history',$(this),'table')"  data-success="Looking for a new version of '${cgi.escape(item['Title'], quote=True)}'">new</a>]</td>
-				<td id="delete"><a href="#" onclick="doAjaxCall('clearhistory?date_added=${item['DateAdded']}&title=${cgi.escape(item['Title'], quote=True)}',$(this),'table')" data-success="${cgi.escape(item['Title'], quote=True)} cleared from history"><img src="interfaces/default/images/trashcan.png" height="18" width="18" id="trashcan" title="Clear this item from the history"></a>
+				<td id="action">[<a href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${item['AlbumID']}&redirect=history', $(this),'table')"  data-success="Retrying download of '${cgi.escape(item['Title'], quote=True)}'">retry</a>][<a href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${item['AlbumID']}&new=True&redirect=history',$(this),'table')"  data-success="Looking for a new version of '${cgi.escape(item['Title'], quote=True)}'">new</a>]</td>
+				<td id="delete"><a href="javascript:void(0)" onclick="doAjaxCall('clearhistory?date_added=${item['DateAdded']}&title=${cgi.escape(item['Title'], quote=True)}',$(this),'table')" data-success="${cgi.escape(item['Title'], quote=True)} cleared from history"><img src="interfaces/default/images/trashcan.png" height="18" width="18" id="trashcan" title="Clear this item from the history"></a>
 			</tr>
 		%endfor
 		</tbody>
@@ -6,7 +6,7 @@
 <%def name="headerIncludes()">
 	<div id="subhead_container">
 		<div id="subhead_menu">
-			<a class="menu_link_edit" id="manage_albums" href="#"><i class="fa fa-pencil"></i> Manage Albums</a>
+			<a class="menu_link_edit" id="manage_albums" href="javascript:void(0)"><i class="fa fa-pencil"></i> Manage Albums</a>
 				<div id="dialog" title="Choose Album Filter" style="display:none" class="configtable">
 					<div class="links">
 						<a href="manageAlbums?Status=Downloaded"><i class="fa fa-check fa-fw"></i> Manage Downloaded Albums</a><br>
@@ -90,7 +90,7 @@
 				%>
 				<input type="text" value="${lastfmvalue}" placeholder="Last.fm username" onfocus="if
 				(this.value==this.defaultValue) this.value='';" name="username" id="username" size="18" />
-				<a href="#" onclick="doAjaxCall('importLastFM?username=',$(this),'tabs');return false;" data-success="Last.fm username has been reset"><i class="fa fa-reply"></i> Reset username</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('importLastFM?username=',$(this),'tabs');return false;" data-success="Last.fm username has been reset"><i class="fa fa-reply"></i> Reset username</a>
 			</div>
 			</fieldset>
 			<input type="button" value="Save changes" onclick="doAjaxCall('importLastFM',$(this),'tabs',true);return false;" data-success="Last.fm artists will be imported" data-error="Fill in a last.fm username"/>
@@ -121,10 +121,10 @@
 		<fieldset>
 			<legend>Force Search</legend>
 			<div class="links">
-				<a href="#" onclick="doAjaxCall('forceSearch',$(this))" data-success="Checking for wanted albums successful" data-error="Error checking wanted albums"><i class="fa fa-search fa-fw"></i> Force Check for Wanted Albums</a>
-				<a href="#" onclick="doAjaxCall('forceUpdate',$(this))" data-success="Update active artists successful" data-error="Error forcing update artists"><i class="fa fa-heart fa-fw"></i> Force Update Active Artists [Fast]</a>
-				<a href="#" onclick="doAjaxCall('checkGithub',$(this))" data-success="Checking for update successful" data-error="Error checking for update"><i class="fa fa-refresh fa-fw"></i> Check for Headphones Updates</a>
-				<a href="#" id="delete_empty_artists"><i class="fa fa-trash-o fa-fw"></i> Delete empty Artists</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('forceSearch',$(this))" data-success="Checking for wanted albums successful" data-error="Error checking wanted albums"><i class="fa fa-search fa-fw"></i> Force Check for Wanted Albums</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('forceUpdate',$(this))" data-success="Update active artists successful" data-error="Error forcing update artists"><i class="fa fa-heart fa-fw"></i> Force Update Active Artists [Fast]</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('checkGithub',$(this))" data-success="Checking for update successful" data-error="Error checking for update"><i class="fa fa-refresh fa-fw"></i> Check for Headphones Updates</a>
+				<a href="javascript:void(0)" id="delete_empty_artists"><i class="fa fa-trash-o fa-fw"></i> Delete empty Artists</a>
 					<div id="emptyartistdialog" title="Confirm Artist Deletion" style="display:none" class="configtable">
 					    %if emptyArtists:
 						<h3>The following artists will be deleted:</h3>
@@ -137,28 +137,27 @@
 					        No empty artists found.
 					    %endif
 					</div>
-				<a href="#" onclick="doAjaxCall('forcePostProcess',$(this))" data-success="Post-Processor is being loaded" data-error="Error during Post-Processing"><i class="fa fa-wrench fa-fw"></i> Force Post-Process Albums in Download Folder</a>
+					<div id="post_process">
+					<a href="javascript:void(0)" class="btnOpenDialog"><i class="fa fa-wrench fa-fw"></i> Force Post-Process Albums in Download Folder</a>
+					</div>
 			</div>
 		</fieldset>
-		<form action="forcePostProcess" method="GET">
 		<fieldset>
-		            <div class="row">
+            <div class="row" id="post_process_alternate">
                <label>Force Post-Process Albums in Alternate Folder</label>
-					    <input type="text" value="" name="dir" id="dir" size="50" /><input type="submit" />
+			    <input type="text" value="" name="dir" id="dir" size="50" />
+			    <input type="button" class="btnOpenDialog" value="Submit" />
            </div>
        </fieldset>

-		</form>
-		<form action="forcePostProcess" method="GET">
        <fieldset>
-		            <div class="row">
+            <div class="row" id="post_process_single">
                <label>Post-Process Single Folder</label>
-					    <input type="text" value="" name="album_dir" id="album_dir" size="50" /><input type="submit" />
+			    <input type="text" value="" name="album_dir" id="album_dir" size="50" />
+			    <input type="button" class="btnOpenDialog" value="Submit" />
            </div>
        </fieldset>

-		</form>
-		
 	</div>


@@ -167,9 +166,9 @@
 			<legend>Force Legacy</legend>
 				<p>Please note that these functions will take a significant amount of time to complete.</p>
 			<div class="links">
-				<a href="#" onclick="doAjaxCall('forceFullUpdate',$(this))" data-success="Update active artists successful" data-error="Error forcing update artists"><i class="fa fa-heart fa-fw"></i> Force Update Active Artists [Comprehensive]</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('forceFullUpdate',$(this))" data-success="Update active artists successful" data-error="Error forcing update artists"><i class="fa fa-heart fa-fw"></i> Force Update Active Artists [Comprehensive]</a>
 				<BR>
-				<a href="#" onclick="doAjaxCall('forceScan',$(this))" data-success="Library scan successful" data-error="Error forcing library scan"><i class="fa fa-refresh fa-fw"></i> Force Re-scan Library [Comprehensive]</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('forceScan',$(this))" data-success="Library scan successful" data-error="Error forcing library scan"><i class="fa fa-refresh fa-fw"></i> Force Re-scan Library [Comprehensive]</a>
 				<BR>
 				<small>*Warning: If you choose [Force Re-scan Library], any manually ignored/matched artists/albums will be reset to "unmatched".</small>

@@ -177,8 +176,7 @@
 		</fieldset>
 		
 	</div>
-
-
+	<div id="dialog-confirm"></div>

 </div>
 </%def>
@@ -188,6 +186,55 @@
 			$('#autoadd').append('<input type="hidden" name="scan" value=1 />');
 		};

+		function fnOpenNormalDialog(id) {
+			if (id === "post_process"){
+				var url = "forcePostProcess?"
+			}
+			if (id === "post_process_alternate"){
+				var dir = $('#dir').val();
+				var url = "forcePostProcess?dir=" + dir + "&"
+			}
+			if (id === "post_process_single"){
+				var dir = $('#album_dir').val();
+				var url = "forcePostProcess?album_dir=" + dir + "&"
+			}
+			var t = $('<a data-success="Post-Processor is being loaded" data-error="Error during Post-Processing">');
+		    $("#dialog-confirm").html("Do you want to keep the original folder(s) after post-processing? If you click no, the folders still might be kept depending on your global settings");
+
+		    // Define the Dialog and its properties.
+		    $("#dialog-confirm").dialog({
+		        resizable: false,
+		        modal: true,
+		        title: "Keep original folder(s)?",
+		        height: 170,
+		        width: 400,
+		        buttons: {
+		            "Yes": function () {
+		                $(this).dialog('close');
+		                doAjaxCall(url + "keep_original_folder=True", t);
+		            },
+		                "No": function () {
+		                $(this).dialog('close');
+		                doAjaxCall(url + "keep_original_folder=False", t);
+		            }
+		        }
+		    });
+		}
+
+		$('.btnOpenDialog').click(function(e){
+			e.preventDefault();
+			var parentId = $(this).closest('div').prop('id');
+			fnOpenNormalDialog(parentId);
+		});
+
+		function callback(value) {
+		    if (value) {
+		        alert("Confirmed");
+		    } else {
+		        alert("Rejected");
+		    }
+		}
+		
 		function initThisPage() {
 			$('#manage_albums').click(function() {
 				$('#dialog').dialog();
@@ -43,9 +43,9 @@
 					<tr><td>Are you sure you want to reset Local Artist: ${album['ArtistName']} to unmatched?</td></tr>
 					<tr><td align="right"><BR>
 					%if album['AlbumStatus'] == "Ignored":
-						<button href="#" onclick="doAjaxCall('markManual?action=unignoreArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully reset ${album['ArtistName']} to unmatched">Reset Artist</button>
+						<button href="javascript:void(0)" onclick="doAjaxCall('markManual?action=unignoreArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully reset ${album['ArtistName']} to unmatched">Reset Artist</button>
 					%elif album['AlbumStatus'] == "Matched":
-						<button href="#" onclick="doAjaxCall('markManual?action=unmatchArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully restored ${album['ArtistName']} to unmatched">Reset Artist</button>
+						<button href="javascript:void(0)" onclick="doAjaxCall('markManual?action=unmatchArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully restored ${album['ArtistName']} to unmatched">Reset Artist</button>
 					%endif
 					</td></tr>
 					</table>
@@ -58,9 +58,9 @@
 					<tr><td>Are you sure you want to reset Local Album: ${album['AlbumTitle']} to unmatched?</td></tr>
 					<tr><td align="right"><BR>
 					%if album['AlbumStatus'] == "Ignored":
-						<button href="#" onclick="doAjaxCall('markManual?action=unignoreAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully reset ${album['AlbumTitle']} to unmatched">Reset Album</button>
+						<button href="javascript:void(0)" onclick="doAjaxCall('markManual?action=unignoreAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully reset ${album['AlbumTitle']} to unmatched">Reset Album</button>
 					%elif album['AlbumStatus'] == "Matched":
-						<button href="#" onclick="doAjaxCall('markManual?action=unmatchAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully reset ${album['AlbumTitle']} to unmatched">Reset Album</button>
+						<button href="javascript:void(0)" onclick="doAjaxCall('markManual?action=unmatchAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully reset ${album['AlbumTitle']} to unmatched">Reset Album</button>
 					%endif
 					</td></tr>
 					</table>
@@ -52,7 +52,7 @@
 					<table>
 					<tr><td>Are you sure you want to ignore Local Artist: ${album['ArtistName']} from future matching?</td></tr>
 					<tr><td align="right"><BR>
-					<button href="#" onclick="doAjaxCall('markUnmatched?action=ignoreArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully ignored ${album['ArtistName']} from future matching">Ignore Artist</button>
+					<button href="javascript:void(0)" onclick="doAjaxCall('markUnmatched?action=ignoreArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully ignored ${album['ArtistName']} from future matching">Ignore Artist</button>
 					</td></tr>
 					</table>
 				</div>
@@ -66,7 +66,7 @@
 						</select>
 						</td></tr>
 					<tr><td></td><td align="right"><BR>
-					<button href="#" onclick="artist_matcher(${count_albums}, '${old_artist_js}')">Match Artist</button>
+					<button href="javascript:void(0)" onclick="artist_matcher(${count_albums}, '${old_artist_js}')">Match Artist</button>
 					</td></tr>
 					</table>
 				</div>
@@ -78,7 +78,7 @@
 					<table>
 					<tr><td>Are you sure you want to ignore Local Album: ${album['AlbumTitle']} from future matching?</td></tr>
 					<tr><td align="right"><BR>
-					<button href="#" onclick="doAjaxCall('markUnmatched?action=ignoreAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully ignored ${album['AlbumTitle']} from future matching">Ignore Album</button>
+					<button href="javascript:void(0)" onclick="doAjaxCall('markUnmatched?action=ignoreAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully ignored ${album['AlbumTitle']} from future matching">Ignore Album</button>
 					</td></tr>
 					</table>
 				</div>
@@ -97,7 +97,7 @@
 					</select>
 					</td></tr>
 					<tr><td></td><td align="right"><BR>
-					<button href="#" onclick="album_matcher(${count_albums}, '${old_artist_js}', '${old_album_js}')">Match Album</button>
+					<button href="javascript:void(0)" onclick="album_matcher(${count_albums}, '${old_artist_js}', '${old_album_js}')">Match Album</button>
 					</td></tr>
 					</table>
 				</div>
@@ -3,7 +3,7 @@
 <%def name="headerIncludes()">
 	<div id="subhead_container">
 		<div id="subhead_menu">
-			<a href="#" id="menu_link_scan" onclick="doAjaxCall('forceSearch',$(this))" data-success="Checking for wanted albums successful" data-error="Error checking wanted albums"><i class="fa fa-search"></i> Force Check</a>
+			<a href="javascript:void(0)" id="menu_link_scan" onclick="doAjaxCall('forceSearch',$(this))" data-success="Checking for wanted albums successful" data-error="Error checking wanted albums"><i class="fa fa-search"></i> Force Check</a>
 		</div>
 	</div>
 </%def>
@@ -16,21 +16,21 @@
 # NZBGet support added by CurlyMo <curlymoo1@gmail.com> as a part of
 # XBian - XBMC on the Raspberry Pi

-import os
 import sys
 import subprocess
 import threading
 import webbrowser
 import sqlite3
-import cherrypy
 import datetime

+import os
+import cherrypy
 from apscheduler.schedulers.background import BackgroundScheduler
 from apscheduler.triggers.interval import IntervalTrigger
-
 from headphones import versioncheck, logger
 import headphones.config

+
 # (append new extras to the end)
 POSSIBLE_EXTRAS = [
    "single",
@@ -94,7 +94,6 @@ UMASK = None


 def initialize(config_file):
-
    with INIT_LOCK:

        global CONFIG
@@ -246,7 +245,6 @@ def daemonize():


 def launch_browser(host, port, root):
-
    if host == '0.0.0.0':
        host = 'localhost'

@@ -293,11 +291,13 @@ def initialize_scheduler():
                minutes = CONFIG.CHECK_GITHUB_INTERVAL
            else:
                minutes = 0
-            schedule_job(versioncheck.checkGithub, 'Check GitHub for updates', hours=0, minutes=minutes)
+            schedule_job(versioncheck.checkGithub, 'Check GitHub for updates', hours=0,
+                         minutes=minutes)

        # Remove Torrent + data if Post Processed and finished Seeding
        minutes = CONFIG.TORRENT_REMOVAL_INTERVAL
-        schedule_job(torrentfinished.checkTorrentFinished, 'Torrent removal check', hours=0, minutes=minutes)
+        schedule_job(torrentfinished.checkTorrentFinished, 'Torrent removal check', hours=0,
+                     minutes=minutes)

        # Start scheduler
        if start_jobs and len(SCHED.get_jobs()):
@@ -334,7 +334,6 @@ def schedule_job(function, name, hours=0, minutes=0):


 def start():
-
    global started

    if _INITIALIZED:
@@ -349,11 +348,10 @@ def sig_handler(signum=None, frame=None):


 def dbcheck():
-
    conn = sqlite3.connect(DB_FILE)
    c = conn.cursor()
    c.execute(
-        'CREATE TABLE IF NOT EXISTS artists (ArtistID TEXT UNIQUE, ArtistName TEXT, ArtistSortName TEXT, DateAdded TEXT, Status TEXT, IncludeExtras INTEGER, LatestAlbum TEXT, ReleaseDate TEXT, AlbumID TEXT, HaveTracks INTEGER, TotalTracks INTEGER, LastUpdated TEXT, ArtworkURL TEXT, ThumbURL TEXT, Extras TEXT, Type TEXT)')
+        'CREATE TABLE IF NOT EXISTS artists (ArtistID TEXT UNIQUE, ArtistName TEXT, ArtistSortName TEXT, DateAdded TEXT, Status TEXT, IncludeExtras INTEGER, LatestAlbum TEXT, ReleaseDate TEXT, AlbumID TEXT, HaveTracks INTEGER, TotalTracks INTEGER, LastUpdated TEXT, ArtworkURL TEXT, ThumbURL TEXT, Extras TEXT, Type TEXT, MetaCritic TEXT)')
    # ReleaseFormat here means CD,Digital,Vinyl, etc. If using the default
    # Headphones hybrid release, ReleaseID will equal AlbumID (AlbumID is
    # releasegroup id)
@@ -599,12 +597,16 @@ def dbcheck():
    except sqlite3.OperationalError:
        c.execute('ALTER TABLE artists ADD COLUMN Type TEXT DEFAULT NULL')

+    try:
+        c.execute('SELECT MetaCritic from artists')
+    except sqlite3.OperationalError:
+        c.execute('ALTER TABLE artists ADD COLUMN MetaCritic TEXT DEFAULT NULL')
+
    conn.commit()
    c.close()


 def shutdown(restart=False, update=False):
-
    cherrypy.engine.exit()
    SCHED.shutdown(wait=False)

@@ -53,7 +53,6 @@ def switch(AlbumID, ReleaseID):
    c.get_artwork_from_cache(AlbumID=AlbumID)

    for track in newtrackdata:
-
        controlValueDict = {"TrackID": track['TrackID'],
                            "AlbumID": AlbumID}

@@ -79,15 +78,18 @@ def switch(AlbumID, ReleaseID):
    have_track_count = len(myDB.select(
        'SELECT * from tracks WHERE AlbumID=? AND Location IS NOT NULL', [AlbumID]))

-    if oldalbumdata['Status'] == 'Skipped' and ((have_track_count / float(total_track_count)) >= (headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0)):
+    if oldalbumdata['Status'] == 'Skipped' and ((have_track_count / float(total_track_count)) >= (
+        headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0)):
        myDB.action(
            'UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', AlbumID])

    # Update have track counts on index
    totaltracks = len(myDB.select(
-        'SELECT TrackTitle from tracks WHERE ArtistID=? AND AlbumID IN (SELECT AlbumID FROM albums WHERE Status != "Ignored")', [newalbumdata['ArtistID']]))
+        'SELECT TrackTitle from tracks WHERE ArtistID=? AND AlbumID IN (SELECT AlbumID FROM albums WHERE Status != "Ignored")',
+        [newalbumdata['ArtistID']]))
    havetracks = len(myDB.select(
-        'SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [newalbumdata['ArtistID']]))
+        'SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL',
+        [newalbumdata['ArtistID']]))

    controlValueDict = {"ArtistID": newalbumdata['ArtistID']}

@@ -13,21 +13,25 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

-from headphones import db, mb, updater, importer, searcher, cache, postprocessor, versioncheck, logger
-
-import headphones
 import json

-cmd_list = ['getIndex', 'getArtist', 'getAlbum', 'getUpcoming', 'getWanted', 'getSnatched', 'getSimilar', 'getHistory', 'getLogs',
-            'findArtist', 'findAlbum', 'addArtist', 'delArtist', 'pauseArtist', 'resumeArtist', 'refreshArtist',
-            'addAlbum', 'queueAlbum', 'unqueueAlbum', 'forceSearch', 'forceProcess', 'forceActiveArtistsUpdate', 
-            'getVersion', 'checkGithub','shutdown', 'restart', 'update', 'getArtistArt', 'getAlbumArt', 
-            'getArtistInfo', 'getAlbumInfo', 'getArtistThumb', 'getAlbumThumb', 
+from headphones import db, mb, updater, importer, searcher, cache, postprocessor, versioncheck, \
+    logger
+import headphones
+
+cmd_list = ['getIndex', 'getArtist', 'getAlbum', 'getUpcoming', 'getWanted', 'getSnatched',
+            'getSimilar', 'getHistory', 'getLogs',
+            'findArtist', 'findAlbum', 'addArtist', 'delArtist', 'pauseArtist', 'resumeArtist',
+            'refreshArtist',
+            'addAlbum', 'queueAlbum', 'unqueueAlbum', 'forceSearch', 'forceProcess',
+            'forceActiveArtistsUpdate',
+            'getVersion', 'checkGithub', 'shutdown', 'restart', 'update', 'getArtistArt',
+            'getAlbumArt',
+            'getArtistInfo', 'getAlbumInfo', 'getArtistThumb', 'getAlbumThumb', 'clearLogs',
            'choose_specific_download', 'download_specific_release']


 class Api(object):
-
    def __init__(self):

        self.apikey = None
@@ -176,7 +180,13 @@ class Api(object):
        return

    def _getLogs(self, **kwargs):
-        pass
+        self.data = headphones.LOG_LIST
+        return
+
+    def _clearLogs(self, **kwargs):
+        headphones.LOG_LIST = []
+        self.data = 'Cleared log'
+        return

    def _findArtist(self, **kwargs):
        if 'name' not in kwargs:
@@ -322,10 +332,15 @@ class Api(object):
        searcher.searchforalbum()

    def _forceProcess(self, **kwargs):
-        self.dir = None
-        if 'dir' in kwargs:
+        if 'album_dir' in kwargs:
+            album_dir = kwargs['album_dir']
+            dir = None
+            postprocessor.forcePostProcess(self, dir, album_dir)
+        elif 'dir' in kwargs:
            self.dir = kwargs['dir']
            postprocessor.forcePostProcess(self.dir)
+        else:
+            postprocessor.forcePostProcess()

    def _forceActiveArtistsUpdate(self, **kwargs):
        updater.dbUpdate()
@@ -426,7 +441,6 @@ class Api(object):
        results_as_dicts = []

        for result in results:
-
            result_dict = {
                'title': result[0],
                'size': result[1],
@@ -14,8 +14,8 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

 import os
-import headphones

+import headphones
 from headphones import db, helpers, logger, lastfm, request

 LASTFM_API_KEY = "690e1ed3bc00bc91804cd8f7fe5ed6d4"
@@ -182,13 +182,18 @@ class Cache(object):
        if ArtistID:
            self.id = ArtistID
            self.id_type = 'artist'
-            db_info = myDB.action('SELECT Summary, Content, LastUpdated FROM descriptions WHERE ArtistID=?', [self.id]).fetchone()
+            db_info = myDB.action(
+                'SELECT Summary, Content, LastUpdated FROM descriptions WHERE ArtistID=?',
+                [self.id]).fetchone()
        else:
            self.id = AlbumID
            self.id_type = 'album'
-            db_info = myDB.action('SELECT Summary, Content, LastUpdated FROM descriptions WHERE ReleaseGroupID=?', [self.id]).fetchone()
+            db_info = myDB.action(
+                'SELECT Summary, Content, LastUpdated FROM descriptions WHERE ReleaseGroupID=?',
+                [self.id]).fetchone()

-        if not db_info or not db_info['LastUpdated'] or not self._is_current(date=db_info['LastUpdated']):
+        if not db_info or not db_info['LastUpdated'] or not self._is_current(
+                date=db_info['LastUpdated']):

            self._update_cache()
            info_dict = {'Summary': self.info_summary, 'Content': self.info_content}
@@ -309,13 +314,19 @@ class Cache(object):
                logger.debug('No artist thumbnail image found')

        else:
-            dbalbum = myDB.action('SELECT ArtistName, AlbumTitle, ReleaseID FROM albums WHERE AlbumID=?', [self.id]).fetchone()
+            dbalbum = myDB.action(
+                'SELECT ArtistName, AlbumTitle, ReleaseID FROM albums WHERE AlbumID=?',
+                [self.id]).fetchone()
            if dbalbum['ReleaseID'] != self.id:
-                data = lastfm.request_lastfm("album.getinfo", mbid=dbalbum['ReleaseID'], api_key=LASTFM_API_KEY)
+                data = lastfm.request_lastfm("album.getinfo", mbid=dbalbum['ReleaseID'],
+                                             api_key=LASTFM_API_KEY)
                if not data:
-                    data = lastfm.request_lastfm("album.getinfo", artist=dbalbum['ArtistName'], album=dbalbum['AlbumTitle'], api_key=LASTFM_API_KEY)
+                    data = lastfm.request_lastfm("album.getinfo", artist=dbalbum['ArtistName'],
+                                                 album=dbalbum['AlbumTitle'],
+                                                 api_key=LASTFM_API_KEY)
            else:
-                data = lastfm.request_lastfm("album.getinfo", artist=dbalbum['ArtistName'], album=dbalbum['AlbumTitle'], api_key=LASTFM_API_KEY)
+                data = lastfm.request_lastfm("album.getinfo", artist=dbalbum['ArtistName'],
+                                             album=dbalbum['AlbumTitle'], api_key=LASTFM_API_KEY)

            if not data:
                return
@@ -357,7 +368,8 @@ class Cache(object):
        # Save the image URL to the database
        if image_url:
            if self.id_type == 'artist':
-                myDB.action('UPDATE artists SET ArtworkURL=? WHERE ArtistID=?', [image_url, self.id])
+                myDB.action('UPDATE artists SET ArtworkURL=? WHERE ArtistID=?',
+                            [image_url, self.id])
            else:
                myDB.action('UPDATE albums SET ArtworkURL=? WHERE AlbumID=?', [image_url, self.id])

@@ -378,7 +390,8 @@ class Cache(object):
                if not os.path.isdir(self.path_to_art_cache):
                    try:
                        os.makedirs(self.path_to_art_cache)
-                        os.chmod(self.path_to_art_cache, int(headphones.CONFIG.FOLDER_PERMISSIONS, 8))
+                        os.chmod(self.path_to_art_cache,
+                                 int(headphones.CONFIG.FOLDER_PERMISSIONS, 8))
                    except OSError as e:
                        logger.error('Unable to create artwork cache dir. Error: %s', e)
                        self.artwork_errors = True
@@ -393,7 +406,8 @@ class Cache(object):

                ext = os.path.splitext(image_url)[1]

-                artwork_path = os.path.join(self.path_to_art_cache, self.id + '.' + helpers.today() + ext)
+                artwork_path = os.path.join(self.path_to_art_cache,
+                                            self.id + '.' + helpers.today() + ext)
                try:
                    with open(artwork_path, 'wb') as f:
                        f.write(artwork)
@@ -406,7 +420,8 @@ class Cache(object):

        # Grab the thumbnail as well if we're getting the full artwork (as long
        # as it's missing/outdated.
-        if thumb_url and self.query_type in ['thumb', 'artwork'] and not (self.thumb_files and self._is_current(self.thumb_files[0])):
+        if thumb_url and self.query_type in ['thumb', 'artwork'] and not (
+            self.thumb_files and self._is_current(self.thumb_files[0])):
            artwork = request.request_content(thumb_url, timeout=20)

            if artwork:
@@ -414,7 +429,8 @@ class Cache(object):
                if not os.path.isdir(self.path_to_art_cache):
                    try:
                        os.makedirs(self.path_to_art_cache)
-                        os.chmod(self.path_to_art_cache, int(headphones.CONFIG.FOLDER_PERMISSIONS, 8))
+                        os.chmod(self.path_to_art_cache,
+                                 int(headphones.CONFIG.FOLDER_PERMISSIONS, 8))
                    except OSError as e:
                        logger.error('Unable to create artwork cache dir. Error: %s' + e)
                        self.thumb_errors = True
@@ -429,7 +445,8 @@ class Cache(object):

                ext = os.path.splitext(image_url)[1]

-                thumb_path = os.path.join(self.path_to_art_cache, 'T_' + self.id + '.' + helpers.today() + ext)
+                thumb_path = os.path.join(self.path_to_art_cache,
+                                          'T_' + self.id + '.' + helpers.today() + ext)
                try:
                    with open(thumb_path, 'wb') as f:
                        f.write(artwork)
@@ -13,9 +13,9 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

-#########################################
-## Stolen from Sick-Beard's classes.py ##
-#########################################
+#######################################
+# Stolen from Sick-Beard's classes.py #
+#######################################


 import urllib
@@ -133,4 +133,5 @@ class Proper:
        self.episode = -1

    def __str__(self):
-        return str(self.date) + " " + self.name + " " + str(self.season) + "x" + str(self.episode) + " of " + str(self.tvdbid)
+        return str(self.date) + " " + self.name + " " + str(self.season) + "x" + str(
+            self.episode) + " of " + str(self.tvdbid)
@@ -20,15 +20,16 @@ Created on Aug 1, 2011
 '''
 import platform
 import operator
+
 import os
 import re
-
 from headphones import version

+
 # Identify Our Application
 USER_AGENT = 'Headphones/-' + version.HEADPHONES_VERSION + ' (' + platform.system() + ' ' + platform.release() + ')'

-### Notification Types
+# Notification Types
 NOTIFY_SNATCH = 1
 NOTIFY_DOWNLOAD = 2

@@ -36,7 +37,7 @@ notifyStrings = {}
 notifyStrings[NOTIFY_SNATCH] = "Started Download"
 notifyStrings[NOTIFY_DOWNLOAD] = "Download Finished"

-### Release statuses
+# Release statuses
 UNKNOWN = -1  # should never happen
 UNAIRED = 1  # releases that haven't dropped yet
 SNATCHED = 2  # qualified with quality
@@ -49,7 +50,6 @@ SNATCHED_PROPER = 9 # qualified with quality


 class Quality:
-
    NONE = 0
    B192 = 1 << 1  # 2
    VBR = 1 << 2  # 4
@@ -75,7 +75,8 @@ class Quality:
    def _getStatusStrings(status):
        toReturn = {}
        for x in Quality.qualityStrings.keys():
-            toReturn[Quality.compositeStatus(status, x)] = Quality.statusPrefixes[status] + " (" + Quality.qualityStrings[x] + ")"
+            toReturn[Quality.compositeStatus(status, x)] = Quality.statusPrefixes[status] + " (" + \
+                                                           Quality.qualityStrings[x] + ")"
        return toReturn

    @staticmethod
@@ -103,6 +104,9 @@ class Quality:
    @staticmethod
    def nameQuality(name):

+        def checkName(list, func):
+            return func([re.search(x, name, re.I) for x in list])
+
        name = os.path.basename(name)

        # if we have our exact text then assume we put it there
@@ -115,8 +119,6 @@ class Quality:
            if regex_match:
                return x

-        checkName = lambda list, func: func([re.search(x, name, re.I) for x in list])
-
        # TODO: fix quality checking here
        if checkName(["mp3", "192"], any) and not checkName(["flac"], all):
            return Quality.B192
@@ -131,7 +133,6 @@ class Quality:

    @staticmethod
    def assumeQuality(name):
-
        if name.lower().endswith(".mp3"):
            return Quality.MP3
        elif name.lower().endswith(".flac"):
@@ -167,13 +168,16 @@ class Quality:
    SNATCHED = None
    SNATCHED_PROPER = None

+
 Quality.DOWNLOADED = [Quality.compositeStatus(DOWNLOADED, x) for x in Quality.qualityStrings.keys()]
 Quality.SNATCHED = [Quality.compositeStatus(SNATCHED, x) for x in Quality.qualityStrings.keys()]
-Quality.SNATCHED_PROPER = [Quality.compositeStatus(SNATCHED_PROPER, x) for x in Quality.qualityStrings.keys()]
+Quality.SNATCHED_PROPER = [Quality.compositeStatus(SNATCHED_PROPER, x) for x in
+                           Quality.qualityStrings.keys()]

 MP3 = Quality.combineQualities([Quality.B192, Quality.B256, Quality.B320, Quality.VBR], [])
 LOSSLESS = Quality.combineQualities([Quality.FLAC], [])
-ANY = Quality.combineQualities([Quality.B192, Quality.B256, Quality.B320, Quality.VBR, Quality.FLAC], [])
+ANY = Quality.combineQualities(
+    [Quality.B192, Quality.B256, Quality.B320, Quality.VBR, Quality.FLAC], [])

 qualityPresets = (MP3, LOSSLESS, ANY)
 qualityPresetStrings = {MP3: "MP3 (All bitrates 192+)",
@@ -1,7 +1,8 @@
-import headphones.logger
 import itertools
+
 import os
 import re
+import headphones.logger
 from configobj import ConfigObj


@@ -14,6 +15,7 @@ def bool_int(value):
            value = 0
    return int(bool(value))

+
 _CONFIG_DEFINITIONS = {
    'ADD_ALBUM_ART': (int, 'General', 0),
    'ADVANCEDENCODER': (str, 'General', ''),
@@ -53,6 +55,7 @@ _CONFIG_DEFINITIONS = {
    'DELETE_LOSSLESS_FILES': (int, 'General', 1),
    'DESTINATION_DIR': (str, 'General', ''),
    'DETECT_BITRATE': (int, 'General', 0),
+    'DO_NOT_PROCESS_UNMATCHED': (int, 'General', 0),
    'DOWNLOAD_DIR': (str, 'General', ''),
    'DOWNLOAD_SCAN_INTERVAL': (int, 'General', 5),
    'DOWNLOAD_TORRENT_DIR': (str, 'General', ''),
@@ -81,10 +84,13 @@ _CONFIG_DEFINITIONS = {
    'ENCODER_PATH': (str, 'General', ''),
    'EXTRAS': (str, 'General', ''),
    'EXTRA_NEWZNABS': (list, 'Newznab', ''),
+    'EXTRA_TORZNABS': (list, 'Torznab', ''),
    'FILE_FORMAT': (str, 'General', 'Track Artist - Album [Year] - Title'),
    'FILE_PERMISSIONS': (str, 'General', '0644'),
+    'FILE_PERMISSIONS_ENABLED': (bool_int, 'General', True),
    'FILE_UNDERSCORES': (int, 'General', 0),
    'FOLDER_FORMAT': (str, 'General', 'Artist/Album [Year]'),
+    'FOLDER_PERMISSIONS_ENABLED': (bool_int, 'General', True),
    'FOLDER_PERMISSIONS': (str, 'General', '0755'),
    'FREEZE_DB': (int, 'General', 0),
    'GIT_BRANCH': (str, 'General', 'master'),
@@ -99,7 +105,7 @@ _CONFIG_DEFINITIONS = {
    'HPUSER': (str, 'General', ''),
    'HTTPS_CERT': (str, 'General', ''),
    'HTTPS_KEY': (str, 'General', ''),
-    'HTTP_HOST': (str, 'General', '0.0.0.0'),
+    'HTTP_HOST': (str, 'General', 'localhost'),
    'HTTP_PASSWORD': (str, 'General', ''),
    'HTTP_PORT': (int, 'General', 8181),
    'HTTP_PROXY': (int, 'General', 0),
@@ -154,10 +160,11 @@ _CONFIG_DEFINITIONS = {
    'NZBSORG_HASH': (str, 'NZBsorg', ''),
    'NZBSORG_UID': (str, 'NZBsorg', ''),
    'NZB_DOWNLOADER': (int, 'General', 0),
+    'OFFICIAL_RELEASES_ONLY': (int, 'General', 0),
    'OMGWTFNZBS': (int, 'omgwtfnzbs', 0),
    'OMGWTFNZBS_APIKEY': (str, 'omgwtfnzbs', ''),
    'OMGWTFNZBS_UID': (str, 'omgwtfnzbs', ''),
-    'OPEN_MAGNET_LINKS': (int, 'General', 0),  # 0: Ignore, 1: Open, 2: Convert
+    'OPEN_MAGNET_LINKS': (int, 'General', 0),  # 0: Ignore, 1: Open, 2: Convert, 3: Embed (rtorrent)
    'MAGNET_LINKS': (int, 'General', 0),
    'OSX_NOTIFY_APP': (str, 'OSX_Notify', '/Applications/Headphones'),
    'OSX_NOTIFY_ENABLED': (int, 'OSX_Notify', 0),
@@ -175,6 +182,7 @@ _CONFIG_DEFINITIONS = {
    'PLEX_SERVER_HOST': (str, 'Plex', ''),
    'PLEX_UPDATE': (int, 'Plex', 0),
    'PLEX_USERNAME': (str, 'Plex', ''),
+    'PLEX_TOKEN': (str, 'Plex', ''),
    'PREFERRED_BITRATE': (str, 'General', ''),
    'PREFERRED_BITRATE_ALLOW_LOSSLESS': (int, 'General', 0),
    'PREFERRED_BITRATE_HIGH_BUFFER': (int, 'General', 0),
@@ -199,7 +207,10 @@ _CONFIG_DEFINITIONS = {
    'PUSHOVER_ONSNATCH': (int, 'Pushover', 0),
    'PUSHOVER_PRIORITY': (int, 'Pushover', 0),
    'RENAME_FILES': (int, 'General', 0),
+    'RENAME_UNPROCESSED': (bool_int, 'General', 1),
+    'RENAME_FROZEN': (bool_int, 'General', 1),
    'REPLACE_EXISTING_FOLDERS': (int, 'General', 0),
+    'KEEP_ORIGINAL_FOLDER': (int, 'General', 0),
    'REQUIRED_WORDS': (str, 'General', ''),
    'RUTRACKER': (int, 'Rutracker', 0),
    'RUTRACKER_PASSWORD': (str, 'Rutracker', ''),
@@ -216,6 +227,8 @@ _CONFIG_DEFINITIONS = {
    'SONGKICK_ENABLED': (int, 'Songkick', 1),
    'SONGKICK_FILTER_ENABLED': (int, 'Songkick', 0),
    'SONGKICK_LOCATION': (str, 'Songkick', ''),
+    'STRIKE': (int, 'Strike', 0),
+    'STRIKE_RATIO': (str, 'Strike', ''),
    'SUBSONIC_ENABLED': (int, 'Subsonic', 0),
    'SUBSONIC_HOST': (str, 'Subsonic', ''),
    'SUBSONIC_PASSWORD': (str, 'Subsonic', ''),
@@ -224,6 +237,10 @@ _CONFIG_DEFINITIONS = {
    'TORRENTBLACKHOLE_DIR': (str, 'General', ''),
    'TORRENT_DOWNLOADER': (int, 'General', 0),
    'TORRENT_REMOVAL_INTERVAL': (int, 'General', 720),
+    'TORZNAB': (int, 'Torznab', 0),
+    'TORZNAB_APIKEY': (str, 'Torznab', ''),
+    'TORZNAB_ENABLED': (int, 'Torznab', 1),
+    'TORZNAB_HOST': (str, 'Torznab', ''),
    'TRANSMISSION_HOST': (str, 'Transmission', ''),
    'TRANSMISSION_PASSWORD': (str, 'Transmission', ''),
    'TRANSMISSION_USERNAME': (str, 'Transmission', ''),
@@ -239,6 +256,7 @@ _CONFIG_DEFINITIONS = {
    'UTORRENT_PASSWORD': (str, 'uTorrent', ''),
    'UTORRENT_USERNAME': (str, 'uTorrent', ''),
    'VERIFY_SSL_CERT': (bool_int, 'Advanced', 1),
+    'WAIT_UNTIL_RELEASE_DATE': (int, 'General', 0),
    'WAFFLES': (int, 'Waffles', 0),
    'WAFFLES_PASSKEY': (str, 'Waffles', ''),
    'WAFFLES_RATIO': (str, 'Waffles', ''),
@@ -256,6 +274,7 @@ _CONFIG_DEFINITIONS = {
    'XLDPROFILE': (str, 'General', '')
 }

+
 # pylint:disable=R0902
 # it might be nice to refactor for fewer instance variables
 class Config(object):
@@ -347,6 +366,25 @@ class Config(object):
            extra_newznabs.append(item)
        self.EXTRA_NEWZNABS = extra_newznabs

+    def get_extra_torznabs(self):
+        """ Return the extra torznab tuples """
+        extra_torznabs = list(
+            itertools.izip(*[itertools.islice(self.EXTRA_TORZNABS, i, None, 3)
+                             for i in range(3)])
+        )
+        return extra_torznabs
+
+    def clear_extra_torznabs(self):
+        """ Forget about the configured extra torznabs """
+        self.EXTRA_TORZNABS = []
+
+    def add_extra_torznab(self, torznab):
+        """ Add a new extra torznab """
+        extra_torznabs = self.EXTRA_TORZNABS
+        for item in torznab:
+            extra_torznabs.append(item)
+        self.EXTRA_TORZNABS = extra_torznabs
+
    def __getattr__(self, name):
        """
        Returns something from the ini unless it is a real property
@@ -15,13 +15,13 @@

 # Most of this lifted from here: https://github.com/SzieberthAdam/gneposis-cdgrab

-import os
 import sys
-import re
 import subprocess
 import copy
 import glob

+import os
+import re
 import headphones
 from headphones import logger
 from mutagen.flac import FLAC
@@ -314,7 +314,8 @@ class CueFile(File):
            line_content = c[line_index]
            search_result = re.search(CUE_TRACK, line_content, re.I)
            if not search_result:
-                raise ValueError('inconsistent CUE sheet, TRACK expected at line {0}'.format(line_index + 1))
+                raise ValueError(
+                    'inconsistent CUE sheet, TRACK expected at line {0}'.format(line_index + 1))
            track_nr = int(search_result.group(1))
            line_index += 1
            next_track = False
@@ -353,7 +354,8 @@ class CueFile(File):
                        track_meta['dcpflag'] = True
                        line_index += 1
                    else:
-                        raise ValueError('unknown entry in track error, line {0}'.format(line_index + 1))
+                        raise ValueError(
+                            'unknown entry in track error, line {0}'.format(line_index + 1))
                else:
                    next_track = True

@@ -406,9 +408,11 @@ class CueFile(File):
        for i in range(len(self.tracks)):
            if self.tracks[i]:
                if self.tracks[i].get('artist'):
-                    content += 'track' + int_to_str(i) + 'artist' + '\t' + self.tracks[i].get('artist') + '\n'
+                    content += 'track' + int_to_str(i) + 'artist' + '\t' + self.tracks[i].get(
+                        'artist') + '\n'
                if self.tracks[i].get('title'):
-                    content += 'track' + int_to_str(i) + 'title' + '\t' + self.tracks[i].get('title') + '\n'
+                    content += 'track' + int_to_str(i) + 'title' + '\t' + self.tracks[i].get(
+                        'title') + '\n'
        return content

    def htoa(self):
@@ -449,7 +453,8 @@ class MetaFile(File):
                        raise ValueError('Syntax error in album meta file')
                    if not content['tracks'][int(parsed_track.group(1))]:
                        content['tracks'][int(parsed_track.group(1))] = dict()
-                    content['tracks'][int(parsed_track.group(1))][parsed_track.group(2)] = parsed_line.group(2)
+                    content['tracks'][int(parsed_track.group(1))][
+                        parsed_track.group(2)] = parsed_line.group(2)
                else:
                    content[parsed_line.group(1)] = parsed_line.group(2)

@@ -480,7 +485,8 @@ class MetaFile(File):

    def folders(self):
        artist = self.content['artist']
-        album = self.content['date'] + ' - ' + self.content['title'] + ' (' + self.content['label'] + ' - ' + self.content['catalog'] + ')'
+        album = self.content['date'] + ' - ' + self.content['title'] + ' (' + self.content[
+            'label'] + ' - ' + self.content['catalog'] + ')'
        return artist, album

    def complete(self):
@@ -535,6 +541,7 @@ class WaveFile(File):
        if self.type == 'Free Lossless Audio Codec':
            return FLAC(self.name)

+
 def split(albumpath):
    global CUE_META
    os.chdir(albumpath)
@@ -577,7 +584,8 @@ def split(albumpath):
        import getXldProfile
        xldprofile, xldformat, _ = getXldProfile.getXldProfile(headphones.CONFIG.XLDPROFILE)
        if not xldformat:
-            raise ValueError('Details for xld profile "%s" not found, cannot split cue' % (xldprofile))
+            raise ValueError(
+                'Details for xld profile "%s" not found, cannot split cue' % (xldprofile))
        else:
            if headphones.CONFIG.ENCODERFOLDER:
                splitter = os.path.join(headphones.CONFIG.ENCODERFOLDER, 'xld')
@@ -13,22 +13,20 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

-#####################################
-## Stolen from Sick-Beard's db.py  ##
-#####################################
+###################################
+# Stolen from Sick-Beard's db.py  #
+###################################

 from __future__ import with_statement

-import os
 import sqlite3

+import os
 import headphones
-
 from headphones import logger


 def dbFilename(filename="headphones.db"):
-
    return os.path.join(headphones.DATA_DIR, filename)


@@ -41,7 +39,6 @@ def getCacheSize():


 class DBConnection:
-
    def __init__(self, filename="headphones.db"):

        self.filename = filename
@@ -92,17 +89,20 @@ class DBConnection:

    def upsert(self, tableName, valueDict, keyDict):

+        def genParams(myDict):
+            return [x + " = ?" for x in myDict.keys()]
+
        changesBefore = self.connection.total_changes

-        genParams = lambda myDict: [x + " = ?" for x in myDict.keys()]
-
-        update_query = "UPDATE " + tableName + " SET " + ", ".join(genParams(valueDict)) + " WHERE " + " AND ".join(genParams(keyDict))
+        update_query = "UPDATE " + tableName + " SET " + ", ".join(
+            genParams(valueDict)) + " WHERE " + " AND ".join(genParams(keyDict))

        self.action(update_query, valueDict.values() + keyDict.values())

        if self.connection.total_changes == changesBefore:
            insert_query = (
-                "INSERT INTO " + tableName + " (" + ", ".join(valueDict.keys() + keyDict.keys()) + ")" +
+                "INSERT INTO " + tableName + " (" + ", ".join(
+                    valueDict.keys() + keyDict.keys()) + ")" +
                " VALUES (" + ", ".join(["?"] * len(valueDict.keys() + keyDict.keys())) + ")"
            )
            try:
@@ -1,10 +1,10 @@
 import os.path
+
 import biplist
 from headphones import logger


 def getXldProfile(xldProfile):
-
    xldProfileNotFound = xldProfile

    expanded = os.path.expanduser('~/Library/Preferences/jp.tmkk.XLD.plist')
@@ -13,19 +13,19 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

-from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
-
 from operator import itemgetter
-
 import unicodedata
-import headphones
 import datetime
-import fnmatch
 import shutil
 import time
 import sys
+
+import fnmatch
 import re
 import os
+from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
+import headphones
+

 # Modified from https://github.com/Verrus/beets-plugin-featInTitle
 RE_FEATURING = re.compile(r"[fF]t\.|[fF]eaturing|[fF]eat\.|\b[wW]ith\b|&|vs\.")
@@ -35,7 +35,9 @@ RE_CD = re.compile(r"^(CD|dics)\s*[0-9]+$", re.I)


 def multikeysort(items, columns):
-    comparers = [((itemgetter(col[1:].strip()), -1) if col.startswith('-') else (itemgetter(col.strip()), 1)) for col in columns]
+    comparers = [
+        ((itemgetter(col[1:].strip()), -1) if col.startswith('-') else (itemgetter(col.strip()), 1))
+        for col in columns]

    def comparer(left, right):
        for fn, mult in comparers:
@@ -56,7 +58,6 @@ def checked(variable):


 def radio(variable, pos):
-
    if variable == pos:
        return 'Checked'
    else:
@@ -70,7 +71,7 @@ def latinToAscii(unicrap):
    xlate = {
        0xc0: 'A', 0xc1: 'A', 0xc2: 'A', 0xc3: 'A', 0xc4: 'A', 0xc5: 'A',
        0xc6: 'Ae', 0xc7: 'C',
-        0xc8: 'E', 0xc9: 'E', 0xca: 'E', 0xcb: 'E', 0x86: 'e',
+        0xc8: 'E', 0xc9: 'E', 0xca: 'E', 0xcb: 'E', 0x86: 'e', 0x39e: 'E',
        0xcc: 'I', 0xcd: 'I', 0xce: 'I', 0xcf: 'I',
        0xd0: 'Th', 0xd1: 'N',
        0xd2: 'O', 0xd3: 'O', 0xd4: 'O', 0xd5: 'O', 0xd6: 'O', 0xd8: 'O',
@@ -107,7 +108,6 @@ def latinToAscii(unicrap):


 def convert_milliseconds(ms):
-
    seconds = ms / 1000
    gmtime = time.gmtime(seconds)
    if seconds > 3600:
@@ -119,7 +119,6 @@ def convert_milliseconds(ms):


 def convert_seconds(s):
-
    gmtime = time.gmtime(s)
    if s > 3600:
        minutes = time.strftime("%H:%M:%S", gmtime)
@@ -141,7 +140,6 @@ def now():


 def get_age(date):
-
    try:
        split_date = date.split('-')
    except:
@@ -149,14 +147,13 @@ def get_age(date):

    try:
        days_old = int(split_date[0]) * 365 + int(split_date[1]) * 30 + int(split_date[2])
-    except IndexError:
+    except (IndexError, ValueError):
        days_old = False

    return days_old


 def bytes_to_mb(bytes):
-
    mb = int(bytes) / 1048576
    size = '%.1f MB' % mb
    return size
@@ -194,7 +191,6 @@ def piratesize(size):


 def replace_all(text, dic, normalize=False):
-
    if not text:
        return ''

@@ -221,15 +217,14 @@ def replace_illegal_chars(string, type="file"):


 def cleanName(string):
-
    pass1 = latinToAscii(string).lower()
-    out_string = re.sub('[\.\-\/\!\@\#\$\%\^\&\*\(\)\+\-\"\'\,\;\:\[\]\{\}\<\>\=\_]', '', pass1).encode('utf-8')
+    out_string = re.sub('[\.\-\/\!\@\#\$\%\^\&\*\(\)\+\-\"\'\,\;\:\[\]\{\}\<\>\=\_]', '',
+                        pass1).encode('utf-8')

    return out_string


 def cleanTitle(title):
-
    title = re.sub('[\.\-\/\_]', ' ', title).lower()

    # Strip out extra whitespace
@@ -312,16 +307,22 @@ def expand_subfolders(f):
    difference = max(path_depths) - min(path_depths)

    if difference > 0:
-        logger.info("Found %d media folders, but depth difference between lowest and deepest media folder is %d (expected zero). If this is a discography or a collection of albums, make sure albums are per folder.", len(media_folders), difference)
+        logger.info(
+            "Found %d media folders, but depth difference between lowest and deepest media folder is %d (expected zero). If this is a discography or a collection of albums, make sure albums are per folder.",
+            len(media_folders), difference)

        # While already failed, advice the user what he could try. We assume the
        # directory may contain separate CD's and maybe some extra's. The
        # structure may look like X albums at same depth, and (one or more)
        # extra folders with a higher depth.
-        extra_media_folders = [media_folder[:min(path_depths)] for media_folder in media_folders if len(media_folder) > min(path_depths)]
-        extra_media_folders = list(set([os.path.join(*media_folder) for media_folder in extra_media_folders]))
+        extra_media_folders = [media_folder[:min(path_depths)] for media_folder in media_folders if
+                               len(media_folder) > min(path_depths)]
+        extra_media_folders = list(
+            set([os.path.join(*media_folder) for media_folder in extra_media_folders]))

-        logger.info("Please look at the following folder(s), since they cause the depth difference: %s", extra_media_folders)
+        logger.info(
+            "Please look at the following folder(s), since they cause the depth difference: %s",
+            extra_media_folders)
        return

    # Convert back to paths and remove duplicates, which may be there after
@@ -378,11 +379,11 @@ def path_filter_patterns(paths, patterns, root=None):


 def extract_data(s):
-
    s = s.replace('_', ' ')

    # headphones default format
-    pattern = re.compile(r'(?P<name>.*?)\s\-\s(?P<album>.*?)\s[\[\(](?P<year>.*?)[\]\)]', re.VERBOSE)
+    pattern = re.compile(r'(?P<name>.*?)\s\-\s(?P<album>.*?)\s[\[\(](?P<year>.*?)[\]\)]',
+                         re.VERBOSE)
    match = pattern.match(s)

    if match:
@@ -468,7 +469,8 @@ def extract_metadata(f):
                old_album = new_albums[index]
                new_albums[index] = RE_CD_ALBUM.sub("", album).strip()

-                logger.debug("Stripped albumd number identifier: %s -> %s", old_album, new_albums[index])
+                logger.debug("Stripped albumd number identifier: %s -> %s", old_album,
+                             new_albums[index])

        # Remove duplicates
        new_albums = list(set(new_albums))
@@ -498,7 +500,8 @@ def extract_metadata(f):
            return (artist, albums[0], years[0])

    # Not sure what to do here.
-    logger.info("Found %d artists, %d albums and %d years in metadata, so ignoring", len(artists), len(albums), len(years))
+    logger.info("Found %d artists, %d albums and %d years in metadata, so ignoring", len(artists),
+                len(albums), len(years))
    logger.debug("Artists: %s, Albums: %s, Years: %s", artists, albums, years)

    return (None, None, None)
@@ -524,8 +527,10 @@ def preserve_torrent_directory(albumpath):
    Copy torrent directory to headphones-modified to keep files for seeding.
    """
    from headphones import logger
-    new_folder = os.path.join(albumpath, 'headphones-modified'.encode(headphones.SYS_ENCODING, 'replace'))
-    logger.info("Copying files to 'headphones-modified' subfolder to preserve downloaded files for seeding")
+    new_folder = os.path.join(albumpath,
+                              'headphones-modified'.encode(headphones.SYS_ENCODING, 'replace'))
+    logger.info(
+        "Copying files to 'headphones-modified' subfolder to preserve downloaded files for seeding")
    try:
        shutil.copytree(albumpath, new_folder)
        return new_folder
@@ -578,7 +583,9 @@ def cue_split(albumpath):

 def extract_logline(s):
    # Default log format
-    pattern = re.compile(r'(?P<timestamp>.*?)\s\-\s(?P<level>.*?)\s*\:\:\s(?P<thread>.*?)\s\:\s(?P<message>.*)', re.VERBOSE)
+    pattern = re.compile(
+        r'(?P<timestamp>.*?)\s\-\s(?P<level>.*?)\s*\:\:\s(?P<thread>.*?)\s\:\s(?P<message>.*)',
+        re.VERBOSE)
    match = pattern.match(s)
    if match:
        timestamp = match.group("timestamp")
@@ -619,7 +626,6 @@ def extract_song_data(s):


 def smartMove(src, dest, delete=True):
-
    from headphones import logger

    source_dir = os.path.dirname(src)
@@ -640,7 +646,8 @@ def smartMove(src, dest, delete=True):
                    os.rename(src, os.path.join(source_dir, newfile))
                    filename = newfile
                except Exception as e:
-                    logger.warn('Error renaming %s: %s', src.decode(headphones.SYS_ENCODING, 'replace'), e)
+                    logger.warn('Error renaming %s: %s',
+                                src.decode(headphones.SYS_ENCODING, 'replace'), e)
                break

    try:
@@ -650,7 +657,9 @@ def smartMove(src, dest, delete=True):
            shutil.copy(os.path.join(source_dir, filename), os.path.join(dest, filename))
            return True
    except Exception as e:
-        logger.warn('Error moving file %s: %s', filename.decode(headphones.SYS_ENCODING, 'replace'), e)
+        logger.warn('Error moving file %s: %s', filename.decode(headphones.SYS_ENCODING, 'replace'),
+                    e)
+

 def walk_directory(basedir, followlinks=True):
    """
@@ -689,6 +698,7 @@ def walk_directory(basedir, followlinks=True):
        for result in _inner(*args):
            yield result

+
 #########################
 # Sab renaming functions #
 #########################
@@ -745,12 +755,14 @@ def sab_sanitize_foldername(name):

    return name

+
 def split_string(mystring, splitvar=','):
    mylist = []
    for each_word in mystring.split(splitvar):
        mylist.append(each_word.strip())
    return mylist

+
 def create_https_certificates(ssl_cert, ssl_key):
    """
    Create a pair of self-signed HTTPS certificares and store in them in
@@ -768,11 +780,13 @@ def create_https_certificates(ssl_cert, ssl_key):
    # Create the CA Certificate
    cakey = createKeyPair(TYPE_RSA, 2048)
    careq = createCertRequest(cakey, CN="Certificate Authority")
-    cacert = createCertificate(careq, (careq, cakey), serial, (0, 60 * 60 * 24 * 365 * 10)) # ten years
+    cacert = createCertificate(careq, (careq, cakey), serial,
+                               (0, 60 * 60 * 24 * 365 * 10))  # ten years

    pkey = createKeyPair(TYPE_RSA, 2048)
    req = createCertRequest(pkey, CN="Headphones")
-    cert = createCertificate(req, (cacert, cakey), serial, (0, 60 * 60 * 24 * 365 * 10)) # ten years
+    cert = createCertificate(req, (cacert, cakey), serial,
+                             (0, 60 * 60 * 24 * 365 * 10))  # ten years

    # Save the key and certificate to disk
    try:
@@ -13,11 +13,10 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

-from headphones import logger, helpers, db, mb, lastfm, metacritic
-
-from beets.mediafile import MediaFile
-
 import time
+
+from headphones import logger, helpers, db, mb, lastfm, metacritic
+from beets.mediafile import MediaFile
 import headphones

 blacklisted_special_artist_names = ['[anonymous]', '[data]', '[no artist]',
@@ -35,17 +34,18 @@ def is_exists(artistid):
    myDB = db.DBConnection()

    # See if the artist is already in the database
-    artistlist = myDB.select('SELECT ArtistID, ArtistName from artists WHERE ArtistID=?', [artistid])
+    artistlist = myDB.select('SELECT ArtistID, ArtistName from artists WHERE ArtistID=?',
+                             [artistid])

    if any(artistid in x for x in artistlist):
-        logger.info(artistlist[0][1] + u" is already in the database. Updating 'have tracks', but not artist information")
+        logger.info(artistlist[0][
+                        1] + u" is already in the database. Updating 'have tracks', but not artist information")
        return True
    else:
        return False


 def artistlist_to_mbids(artistlist, forced=False):
-
    for artist in artistlist:

        if not artist and artist != ' ':
@@ -77,9 +77,12 @@ def artistlist_to_mbids(artistlist, forced=False):
        myDB = db.DBConnection()

        if not forced:
-            bl_artist = myDB.action('SELECT * FROM blacklist WHERE ArtistID=?', [artistid]).fetchone()
+            bl_artist = myDB.action('SELECT * FROM blacklist WHERE ArtistID=?',
+                                    [artistid]).fetchone()
            if bl_artist or artistid in blacklisted_special_artists:
-                logger.info("Artist ID for '%s' is either blacklisted or Various Artists. To add artist, you must do it manually (Artist ID: %s)" % (artist, artistid))
+                logger.info(
+                    "Artist ID for '%s' is either blacklisted or Various Artists. To add artist, you must do it manually (Artist ID: %s)" % (
+                    artist, artistid))
                continue

        # Add to database if it doesn't exist
@@ -88,7 +91,9 @@ def artistlist_to_mbids(artistlist, forced=False):

        # Just update the tracks if it does
        else:
-            havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=?', [artistid])) + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ?', [artist]))
+            havetracks = len(
+                myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=?', [artistid])) + len(
+                myDB.select('SELECT TrackTitle from have WHERE ArtistName like ?', [artist]))
            myDB.action('UPDATE artists SET HaveTracks=? WHERE ArtistID=?', [havetracks, artistid])

        # Delete it from the New Artists if the request came from there
@@ -112,7 +117,6 @@ def addArtistIDListToDB(artistidlist):


 def addArtisttoDB(artistid, extrasonly=False, forcefull=False, type="artist"):
-
    # Putting this here to get around the circular import. We're using this to update thumbnails for artist/albums
    from headphones import cache

@@ -191,7 +195,8 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False, type="artist"):
    # See if we need to grab extras. Artist specific extras take precedence
    # over global option. Global options are set when adding a new artist
    try:
-        db_artist = myDB.action('SELECT IncludeExtras, Extras from artists WHERE ArtistID=?', [artistid]).fetchone()
+        db_artist = myDB.action('SELECT IncludeExtras, Extras from artists WHERE ArtistID=?',
+                                [artistid]).fetchone()
        includeExtras = db_artist['IncludeExtras']
    except IndexError:
        includeExtras = False
@@ -206,9 +211,12 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False, type="artist"):
        for groups in artist['releasegroups']:
            group_list.append(groups['id'])
        if not extrasonly:
-            remove_missing_groups_from_albums = myDB.select("SELECT AlbumID FROM albums WHERE ArtistID=?", [artistid])
+            remove_missing_groups_from_albums = myDB.select(
+                "SELECT AlbumID FROM albums WHERE ArtistID=?", [artistid])
        else:
-            remove_missing_groups_from_albums = myDB.select('SELECT AlbumID FROM albums WHERE ArtistID=? AND Status="Skipped" AND Type!="Album"', [artistid])
+            remove_missing_groups_from_albums = myDB.select(
+                'SELECT AlbumID FROM albums WHERE ArtistID=? AND Status="Skipped" AND Type!="Album"',
+                [artistid])
        for items in remove_missing_groups_from_albums:
            if items['AlbumID'] not in group_list:
                # Remove all from albums/tracks that aren't in release groups
@@ -217,12 +225,16 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False, type="artist"):
                myDB.action("DELETE FROM tracks WHERE AlbumID=?", [items['AlbumID']])
                myDB.action("DELETE FROM alltracks WHERE AlbumID=?", [items['AlbumID']])
                myDB.action('DELETE from releases WHERE ReleaseGroupID=?', [items['AlbumID']])
-                logger.info("[%s] Removing all references to release group %s to reflect MusicBrainz refresh" % (artist['artist_name'], items['AlbumID']))
+                logger.info(
+                    "[%s] Removing all references to release group %s to reflect MusicBrainz refresh" % (
+                    artist['artist_name'], items['AlbumID']))
                if not extrasonly:
                    force_repackage = 1
    else:
        if not extrasonly:
-            logger.info("[%s] There was either an error pulling data from MusicBrainz or there might not be any releases for this category" % artist['artist_name'])
+            logger.info(
+                "[%s] There was either an error pulling data from MusicBrainz or there might not be any releases for this category" %
+                artist['artist_name'])

    # Then search for releases within releasegroups, if releases don't exist, then remove from allalbums/alltracks
    album_searches = []
@@ -247,12 +259,14 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False, type="artist"):
                new_release_group = True

            if new_release_group:
-                logger.info("[%s] Now adding: %s (New Release Group)" % (artist['artist_name'], rg['title']))
+                logger.info("[%s] Now adding: %s (New Release Group)" % (
+                artist['artist_name'], rg['title']))
                new_releases = mb.get_new_releases(rgid, includeExtras)

            else:
                if check_release_date is None or check_release_date == u"None":
-                    logger.info("[%s] Now updating: %s (No Release Date)" % (artist['artist_name'], rg['title']))
+                    logger.info("[%s] Now updating: %s (No Release Date)" % (
+                    artist['artist_name'], rg['title']))
                    new_releases = mb.get_new_releases(rgid, includeExtras, True)
                else:
                    if len(check_release_date) == 10:
@@ -264,20 +278,24 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False, type="artist"):
                    else:
                        release_date = today
                    if helpers.get_age(today) - helpers.get_age(release_date) < pause_delta:
-                        logger.info("[%s] Now updating: %s (Release Date <%s Days)", artist['artist_name'], rg['title'], pause_delta)
+                        logger.info("[%s] Now updating: %s (Release Date <%s Days)",
+                                    artist['artist_name'], rg['title'], pause_delta)
                        new_releases = mb.get_new_releases(rgid, includeExtras, True)
                    else:
-                        logger.info("[%s] Skipping: %s (Release Date >%s Days)", artist['artist_name'], rg['title'], pause_delta)
+                        logger.info("[%s] Skipping: %s (Release Date >%s Days)",
+                                    artist['artist_name'], rg['title'], pause_delta)
                        skip_log = 1
                        new_releases = 0

            if force_repackage == 1:
                new_releases = -1
-                logger.info('[%s] Forcing repackage of %s (Release Group Removed)', artist['artist_name'], al_title)
+                logger.info('[%s] Forcing repackage of %s (Release Group Removed)',
+                            artist['artist_name'], al_title)
            else:
                new_releases = new_releases
        else:
-            logger.info("[%s] Now adding/updating: %s (Comprehensive Force)", artist['artist_name'], rg['title'])
+            logger.info("[%s] Now adding/updating: %s (Comprehensive Force)", artist['artist_name'],
+                        rg['title'])
            new_releases = mb.get_new_releases(rgid, includeExtras, forcefull)

        if new_releases != 0:
@@ -291,11 +309,13 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False, type="artist"):
            # This will be used later to build a hybrid release
            fullreleaselist = []
            # Search for releases within a release group
-            find_hybrid_releases = myDB.action("SELECT * from allalbums WHERE AlbumID=?", [rg['id']])
+            find_hybrid_releases = myDB.action("SELECT * from allalbums WHERE AlbumID=?",
+                                               [rg['id']])

            # Build the dictionary for the fullreleaselist
            for items in find_hybrid_releases:
-                if items['ReleaseID'] != rg['id']: #don't include hybrid information, since that's what we're replacing
+                if items['ReleaseID'] != rg[
+                    'id']:  # don't include hybrid information, since that's what we're replacing
                    hybrid_release_id = items['ReleaseID']
                    newValueDict = {"ArtistID": items['ArtistID'],
                                    "ArtistName": items['ArtistName'],
@@ -307,7 +327,8 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False, type="artist"):
                                    "ReleaseCountry": items['ReleaseCountry'],
                                    "ReleaseFormat": items['ReleaseFormat']
                                    }
-                    find_hybrid_tracks = myDB.action("SELECT * from alltracks WHERE ReleaseID=?", [hybrid_release_id])
+                    find_hybrid_tracks = myDB.action("SELECT * from alltracks WHERE ReleaseID=?",
+                                                     [hybrid_release_id])
                    totalTracks = 1
                    hybrid_track_array = []
                    for hybrid_tracks in find_hybrid_tracks:
@@ -327,10 +348,12 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False, type="artist"):
            # This may end up being called with an empty fullreleaselist
            try:
                hybridrelease = getHybridRelease(fullreleaselist)
-                logger.info('[%s] Packaging %s releases into hybrid title' % (artist['artist_name'], rg['title']))
+                logger.info('[%s] Packaging %s releases into hybrid title' % (
+                artist['artist_name'], rg['title']))
            except Exception as e:
                errors = True
-                logger.warn('[%s] Unable to get hybrid release information for %s: %s' % (artist['artist_name'], rg['title'], e))
+                logger.warn('[%s] Unable to get hybrid release information for %s: %s' % (
+                artist['artist_name'], rg['title'], e))
                continue

            # Use the ReleaseGroupID as the ReleaseID for the hybrid release to differentiate it
@@ -351,7 +374,8 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False, type="artist"):

            for track in hybridrelease['Tracks']:

-                cleanname = helpers.cleanName(artist['artist_name'] + ' ' + rg['title'] + ' ' + track['title'])
+                cleanname = helpers.cleanName(
+                    artist['artist_name'] + ' ' + rg['title'] + ' ' + track['title'])

                controlValueDict = {"TrackID": track['id'],
                                    "ReleaseID": rg['id']}
@@ -367,10 +391,13 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False, type="artist"):
                                "CleanName": cleanname
                                }

-                match = myDB.action('SELECT Location, BitRate, Format from have WHERE CleanName=?', [cleanname]).fetchone()
+                match = myDB.action('SELECT Location, BitRate, Format from have WHERE CleanName=?',
+                                    [cleanname]).fetchone()

                if not match:
-                    match = myDB.action('SELECT Location, BitRate, Format from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [artist['artist_name'], rg['title'], track['title']]).fetchone()
+                    match = myDB.action(
+                        'SELECT Location, BitRate, Format from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?',
+                        [artist['artist_name'], rg['title'], track['title']]).fetchone()
                    # if not match:
                    # match = myDB.action('SELECT Location, BitRate, Format from have WHERE TrackID=?', [track['id']]).fetchone()
                if match:
@@ -378,7 +405,8 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False, type="artist"):
                    newValueDict['BitRate'] = match['BitRate']
                    newValueDict['Format'] = match['Format']
                    # myDB.action('UPDATE have SET Matched="True" WHERE Location=?', [match['Location']])
-                    myDB.action('UPDATE have SET Matched=? WHERE Location=?', (rg['id'], match['Location']))
+                    myDB.action('UPDATE have SET Matched=? WHERE Location=?',
+                                (rg['id'], match['Location']))

                myDB.upsert("alltracks", newValueDict, controlValueDict)

@@ -425,14 +453,17 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False, type="artist"):
                    newValueDict['Status'] = "Wanted"
                # Sometimes "new" albums are added to musicbrainz after their release date, so let's try to catch these
                # The first test just makes sure we have year-month-day
-                elif helpers.get_age(album['ReleaseDate']) and helpers.get_age(today) - helpers.get_age(album['ReleaseDate']) < 21 and headphones.CONFIG.AUTOWANT_UPCOMING:
+                elif helpers.get_age(album['ReleaseDate']) and helpers.get_age(
+                        today) - helpers.get_age(
+                        album['ReleaseDate']) < 21 and headphones.CONFIG.AUTOWANT_UPCOMING:
                    newValueDict['Status'] = "Wanted"
                else:
                    newValueDict['Status'] = "Skipped"

            myDB.upsert("albums", newValueDict, controlValueDict)

-            tracks = myDB.action('SELECT * from alltracks WHERE ReleaseID=?', [releaseid]).fetchall()
+            tracks = myDB.action('SELECT * from alltracks WHERE ReleaseID=?',
+                                 [releaseid]).fetchall()

            # This is used to see how many tracks you have from an album - to
            # mark it as downloaded. Default is 80%, can be set in config as
@@ -465,19 +496,27 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False, type="artist"):
                myDB.upsert("tracks", newValueDict, controlValueDict)

            # Mark albums as downloaded if they have at least 80% (by default, configurable) of the album
-            have_track_count = len(myDB.select('SELECT * from tracks WHERE AlbumID=? AND Location IS NOT NULL', [rg['id']]))
+            have_track_count = len(
+                myDB.select('SELECT * from tracks WHERE AlbumID=? AND Location IS NOT NULL',
+                            [rg['id']]))
            marked_as_downloaded = False

            if rg_exists:
-                if rg_exists['Status'] == 'Skipped' and ((have_track_count / float(total_track_count)) >= (headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0)):
-                    myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', rg['id']])
+                if rg_exists['Status'] == 'Skipped' and (
+                    (have_track_count / float(total_track_count)) >= (
+                    headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0)):
+                    myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?',
+                                ['Downloaded', rg['id']])
                    marked_as_downloaded = True
            else:
-                if (have_track_count / float(total_track_count)) >= (headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0):
-                    myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', rg['id']])
+                if (have_track_count / float(total_track_count)) >= (
+                    headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0):
+                    myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?',
+                                ['Downloaded', rg['id']])
                    marked_as_downloaded = True

-            logger.info(u"[%s] Seeing if we need album art for %s" % (artist['artist_name'], rg['title']))
+            logger.info(
+                u"[%s] Seeing if we need album art for %s" % (artist['artist_name'], rg['title']))
            cache.getThumb(AlbumID=rg['id'])

            # Start a search for the album if it's new, hasn't been marked as
@@ -487,7 +526,8 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False, type="artist"):
                album_searches.append(rg['id'])
        else:
            if skip_log == 0:
-                logger.info(u"[%s] No new releases, so no changes made to %s" % (artist['artist_name'], rg['title']))
+                logger.info(u"[%s] No new releases, so no changes made to %s" % (
+                artist['artist_name'], rg['title']))

    time.sleep(3)
    finalize_update(artistid, artist['artist_name'], errors)
@@ -496,10 +536,12 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False, type="artist"):
    cache.getThumb(ArtistID=artistid)

    logger.info(u"Fetching Metacritic reviews for: %s" % artist['artist_name'])
-    metacritic.update(artist['artist_name'], artist['releasegroups'])
+    metacritic.update(artistid, artist['artist_name'], artist['releasegroups'])

    if errors:
-        logger.info("[%s] Finished updating artist: %s but with errors, so not marking it as updated in the database" % (artist['artist_name'], artist['artist_name']))
+        logger.info(
+            "[%s] Finished updating artist: %s but with errors, so not marking it as updated in the database" % (
+            artist['artist_name'], artist['artist_name']))
    else:
        myDB.action('DELETE FROM newartists WHERE ArtistName = ?', [artist['artist_name']])
        logger.info(u"Updating complete for: %s" % artist['artist_name'])
@@ -518,10 +560,18 @@ def finalize_update(artistid, artistname, errors=False):

    myDB = db.DBConnection()

-    latestalbum = myDB.action('SELECT AlbumTitle, ReleaseDate, AlbumID from albums WHERE ArtistID=? order by ReleaseDate DESC', [artistid]).fetchone()
-    totaltracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND AlbumID IN (SELECT AlbumID FROM albums WHERE Status != "Ignored")', [artistid]))
+    latestalbum = myDB.action(
+        'SELECT AlbumTitle, ReleaseDate, AlbumID from albums WHERE ArtistID=? order by ReleaseDate DESC',
+        [artistid]).fetchone()
+    totaltracks = len(myDB.select(
+        'SELECT TrackTitle from tracks WHERE ArtistID=? AND AlbumID IN (SELECT AlbumID FROM albums WHERE Status != "Ignored")',
+        [artistid]))
    # havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [artistid])) + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ?', [artist['artist_name']]))
-    havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [artistid])) + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"', [artistname]))
+    havetracks = len(
+        myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL',
+                    [artistid])) + len(
+        myDB.select('SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"',
+                    [artistname]))

    controlValueDict = {"ArtistID": artistid}

@@ -544,7 +594,6 @@ def finalize_update(artistid, artistname, errors=False):


 def addReleaseById(rid, rgid=None):
-
    myDB = db.DBConnection()

    # Create minimum info upfront if added from searchresults
@@ -563,14 +612,18 @@ def addReleaseById(rid, rgid=None):
    rgid = None
    artistid = None
    release_dict = None
-    results = myDB.select("SELECT albums.ArtistID, releases.ReleaseGroupID from releases, albums WHERE releases.ReleaseID=? and releases.ReleaseGroupID=albums.AlbumID LIMIT 1", [rid])
+    results = myDB.select(
+        "SELECT albums.ArtistID, releases.ReleaseGroupID from releases, albums WHERE releases.ReleaseID=? and releases.ReleaseGroupID=albums.AlbumID LIMIT 1",
+        [rid])
    for result in results:
        rgid = result['ReleaseGroupID']
        artistid = result['ArtistID']
-        logger.debug("Found a cached releaseid : releasegroupid relationship: " + rid + " : " + rgid)
+        logger.debug(
+            "Found a cached releaseid : releasegroupid relationship: " + rid + " : " + rgid)
    if not rgid:
        # didn't find it in the cache, get the information from MB
-        logger.debug("Didn't find releaseID " + rid + " in the cache. Looking up its ReleaseGroupID")
+        logger.debug(
+            "Didn't find releaseID " + rid + " in the cache. Looking up its ReleaseGroupID")
        try:
            release_dict = mb.getRelease(rid)
        except Exception as e:
@@ -599,7 +652,8 @@ def addReleaseById(rid, rgid=None):
        else:
            sortname = release_dict['artist_name']

-        logger.info(u"Now manually adding: " + release_dict['artist_name'] + " - with status Paused")
+        logger.info(
+            u"Now manually adding: " + release_dict['artist_name'] + " - with status Paused")
        controlValueDict = {"ArtistID": release_dict['artist_id']}
        newValueDict = {"ArtistName": release_dict['artist_name'],
                        "ArtistSortName": sortname,
@@ -624,7 +678,8 @@ def addReleaseById(rid, rgid=None):
        myDB.upsert("artists", newValueDict, controlValueDict)

    elif not artist_exists and not release_dict:
-        logger.error("Artist does not exist in the database and did not get a valid response from MB. Skipping release.")
+        logger.error(
+            "Artist does not exist in the database and did not get a valid response from MB. Skipping release.")
        if status == 'Loading':
            myDB.action("DELETE FROM albums WHERE AlbumID=?", [rgid])
        return
@@ -639,7 +694,8 @@ def addReleaseById(rid, rgid=None):
        newValueDict = {"ArtistID": release_dict['artist_id'],
                        "ReleaseID": rgid,
                        "ArtistName": release_dict['artist_name'],
-                        "AlbumTitle": release_dict['title'] if 'title' in release_dict else release_dict['rg_title'],
+                        "AlbumTitle": release_dict['title'] if 'title' in release_dict else
+                        release_dict['rg_title'],
                        "AlbumASIN": release_dict['asin'],
                        "ReleaseDate": release_dict['date'],
                        "DateAdded": helpers.today(),
@@ -654,7 +710,8 @@ def addReleaseById(rid, rgid=None):
        myDB.action('INSERT INTO releases VALUES( ?, ?)', [rid, release_dict['rgid']])

        for track in release_dict['tracks']:
-            cleanname = helpers.cleanName(release_dict['artist_name'] + ' ' + release_dict['rg_title'] + ' ' + track['title'])
+            cleanname = helpers.cleanName(
+                release_dict['artist_name'] + ' ' + release_dict['rg_title'] + ' ' + track['title'])

            controlValueDict = {"TrackID": track['id'],
                                "AlbumID": rgid}
@@ -668,10 +725,15 @@ def addReleaseById(rid, rgid=None):
                            "CleanName": cleanname
                            }

-            match = myDB.action('SELECT Location, BitRate, Format, Matched from have WHERE CleanName=?', [cleanname]).fetchone()
+            match = myDB.action(
+                'SELECT Location, BitRate, Format, Matched from have WHERE CleanName=?',
+                [cleanname]).fetchone()

            if not match:
-                match = myDB.action('SELECT Location, BitRate, Format, Matched from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [release_dict['artist_name'], release_dict['rg_title'], track['title']]).fetchone()
+                match = myDB.action(
+                    'SELECT Location, BitRate, Format, Matched from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?',
+                    [release_dict['artist_name'], release_dict['rg_title'],
+                     track['title']]).fetchone()

                # if not match:
                # match = myDB.action('SELECT Location, BitRate, Format from have WHERE TrackID=?', [track['id']]).fetchone()
@@ -684,7 +746,8 @@ def addReleaseById(rid, rgid=None):

                # If the album has been scanned before adding the release it will be unmatched, update to matched
                if match['Matched'] == 'Failed':
-                    myDB.action('UPDATE have SET Matched=? WHERE Location=?', (release_dict['rgid'], match['Location']))
+                    myDB.action('UPDATE have SET Matched=? WHERE Location=?',
+                                (release_dict['rgid'], match['Location']))

            myDB.upsert("tracks", newValueDict, controlValueDict)

@@ -703,7 +766,8 @@ def addReleaseById(rid, rgid=None):
            searcher.searchforalbum(rgid, False)

    elif not rg_exists and not release_dict:
-        logger.error("ReleaseGroup does not exist in the database and did not get a valid response from MB. Skipping release.")
+        logger.error(
+            "ReleaseGroup does not exist in the database and did not get a valid response from MB. Skipping release.")
        if status == 'Loading':
            myDB.action("DELETE FROM albums WHERE AlbumID=?", [rgid])
        return
@@ -811,11 +875,13 @@ def getHybridRelease(fullreleaselist):

    sortable_release_list.sort(key=lambda x: getSortableReleaseDate(x['releasedate']))

-    average_tracks = sum(x['trackscount'] for x in sortable_release_list) / float(len(sortable_release_list))
+    average_tracks = sum(x['trackscount'] for x in sortable_release_list) / float(
+        len(sortable_release_list))
    for item in sortable_release_list:
        item['trackscount_delta'] = abs(average_tracks - item['trackscount'])

-    a = helpers.multikeysort(sortable_release_list, ['-hasasin', 'country', 'format', 'trackscount_delta'])
+    a = helpers.multikeysort(sortable_release_list,
+                             ['-hasasin', 'country', 'format', 'trackscount_delta'])

    release_dict = {'ReleaseDate': sortable_release_list[0]['releasedate'],
                    'Tracks': a[0]['tracks'],
@@ -14,13 +14,12 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

 import random
+from collections import defaultdict
+
 import headphones
 import headphones.lock
-
 from headphones import db, logger, request

-from collections import defaultdict
-
 TIMEOUT = 60.0  # seconds
 REQUEST_LIMIT = 1.0 / 5  # seconds
 ENTRY_POINT = "http://ws.audioscrobbler.com/2.0/"
@@ -79,7 +78,7 @@ def getSimilar():
                try:
                    artist_mbid = artist["mbid"]
                    artist_name = artist["name"]
-                except TypeError:
+                except KeyError:
                    continue

                if not any(artist_mbid in x for x in results):
@@ -116,7 +115,7 @@ def getArtists():
        return

    logger.info("Fetching artists from Last.FM for username: %s", headphones.CONFIG.LASTFM_USERNAME)
-    data = request_lastfm("library.getartists", limit=10000, user=headphones.CONFIG.LASTFM_USERNAME)
+    data = request_lastfm("library.getartists", limit=1000, user=headphones.CONFIG.LASTFM_USERNAME)

    if data and "artists" in data:
        artistlist = []
@@ -14,17 +14,17 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

 import os
+import math
+
 import headphones
-
 from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
-
 from headphones import db, logger, helpers, importer, lastfm

+
 # You can scan a single directory and append it to the current library by
 # specifying append=True, ArtistID and ArtistName.
 def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
-    cron=False):
-
+                cron=False, artistScan=False):
    if cron and not headphones.CONFIG.LIBRARYSCAN:
        return

@@ -36,11 +36,12 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,

    # If we're appending a dir, it's coming from the post processor which is
    # already bytestring
-    if not append:
+    if not append or artistScan:
        dir = dir.encode(headphones.SYS_ENCODING)

    if not os.path.isdir(dir):
-        logger.warn('Cannot find directory: %s. Not scanning' % dir.decode(headphones.SYS_ENCODING, 'replace'))
+        logger.warn('Cannot find directory: %s. Not scanning' % dir.decode(headphones.SYS_ENCODING,
+                                                                           'replace'))
        return

    myDB = db.DBConnection()
@@ -50,13 +51,16 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,

    if not append:
        # Clean up bad filepaths
-        tracks = myDB.select('SELECT Location from alltracks WHERE Location IS NOT NULL UNION SELECT Location from tracks WHERE Location IS NOT NULL')
+        tracks = myDB.select(
+            'SELECT Location from alltracks WHERE Location IS NOT NULL UNION SELECT Location from tracks WHERE Location IS NOT NULL')

        for track in tracks:
-            encoded_track_string = track['Location'].encode(headphones.SYS_ENCODING)
+            encoded_track_string = track['Location'].encode(headphones.SYS_ENCODING, 'replace')
            if not os.path.isfile(encoded_track_string):
-                myDB.action('UPDATE tracks SET Location=?, BitRate=?, Format=? WHERE Location=?', [None, None, None, track['Location']])
-                myDB.action('UPDATE alltracks SET Location=?, BitRate=?, Format=? WHERE Location=?', [None, None, None, track['Location']])
+                myDB.action('UPDATE tracks SET Location=?, BitRate=?, Format=? WHERE Location=?',
+                            [None, None, None, track['Location']])
+                myDB.action('UPDATE alltracks SET Location=?, BitRate=?, Format=? WHERE Location=?',
+                            [None, None, None, track['Location']])

        del_have_tracks = myDB.select('SELECT Location, Matched, ArtistName from have')

@@ -67,7 +71,9 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
                    # Make sure deleted files get accounted for when updating artist track counts
                    new_artists.append(track['ArtistName'])
                myDB.action('DELETE FROM have WHERE Location=?', [track['Location']])
-                logger.info('File %s removed from Headphones, as it is no longer on disk' % encoded_track_string.decode(headphones.SYS_ENCODING, 'replace'))
+                logger.info(
+                    'File %s removed from Headphones, as it is no longer on disk' % encoded_track_string.decode(
+                        headphones.SYS_ENCODING, 'replace'))

    bitrates = []
    song_list = []
@@ -89,9 +95,14 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
                latest_subdirectory.append(subdirectory)

                if file_count == 0 and r.replace(dir, '') != '':
-                    logger.info("[%s] Now scanning subdirectory %s" % (dir.decode(headphones.SYS_ENCODING, 'replace'), subdirectory.decode(headphones.SYS_ENCODING, 'replace')))
-                elif latest_subdirectory[file_count] != latest_subdirectory[file_count - 1] and file_count != 0:
-                    logger.info("[%s] Now scanning subdirectory %s" % (dir.decode(headphones.SYS_ENCODING, 'replace'), subdirectory.decode(headphones.SYS_ENCODING, 'replace')))
+                    logger.info("[%s] Now scanning subdirectory %s" % (
+                        dir.decode(headphones.SYS_ENCODING, 'replace'),
+                        subdirectory.decode(headphones.SYS_ENCODING, 'replace')))
+                elif latest_subdirectory[file_count] != latest_subdirectory[
+                            file_count - 1] and file_count != 0:
+                    logger.info("[%s] Now scanning subdirectory %s" % (
+                        dir.decode(headphones.SYS_ENCODING, 'replace'),
+                        subdirectory.decode(headphones.SYS_ENCODING, 'replace')))

                song = os.path.join(r, files)

@@ -102,10 +113,13 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
                try:
                    f = MediaFile(song)
                except (FileTypeError, UnreadableFileError):
-                    logger.warning("Cannot read media file '%s', skipping. It may be corrupted or not a media file.", unicode_song_path)
+                    logger.warning(
+                        "Cannot read media file '%s', skipping. It may be corrupted or not a media file.",
+                        unicode_song_path)
                    continue
                except IOError:
-                    logger.warning("Cannnot read media file '%s', skipping. Does the file exists?", unicode_song_path)
+                    logger.warning("Cannnot read media file '%s', skipping. Does the file exists?",
+                                   unicode_song_path)
                    continue

                # Grab the bitrates for the auto detect bit rate option
@@ -145,7 +159,8 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
                                }

                # song_list.append(song_dict)
-                check_exist_song = myDB.action("SELECT * FROM have WHERE Location=?", [unicode_song_path]).fetchone()
+                check_exist_song = myDB.action("SELECT * FROM have WHERE Location=?",
+                                               [unicode_song_path]).fetchone()
                # Only attempt to match songs that are new, haven't yet been matched, or metadata has changed.
                if not check_exist_song:
                    # This is a new track
@@ -154,19 +169,25 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
                    myDB.upsert("have", newValueDict, controlValueDict)
                    new_song_count += 1
                else:
-                    if check_exist_song['ArtistName'] != f_artist or check_exist_song['AlbumTitle'] != f.album or check_exist_song['TrackTitle'] != f.title:
+                    if check_exist_song['ArtistName'] != f_artist or check_exist_song[
+                        'AlbumTitle'] != f.album or check_exist_song['TrackTitle'] != f.title:
                        # Important track metadata has been modified, need to run matcher again
                        if f_artist and f_artist != check_exist_song['ArtistName']:
                            new_artists.append(f_artist)
-                        elif f_artist and f_artist == check_exist_song['ArtistName'] and check_exist_song['Matched'] != "Ignored":
+                        elif f_artist and f_artist == check_exist_song['ArtistName'] and \
+                                        check_exist_song['Matched'] != "Ignored":
                            new_artists.append(f_artist)
                        else:
                            continue

                        newValueDict['Matched'] = None
                        myDB.upsert("have", newValueDict, controlValueDict)
-                        myDB.action('UPDATE tracks SET Location=?, BitRate=?, Format=? WHERE Location=?', [None, None, None, unicode_song_path])
-                        myDB.action('UPDATE alltracks SET Location=?, BitRate=?, Format=? WHERE Location=?', [None, None, None, unicode_song_path])
+                        myDB.action(
+                            'UPDATE tracks SET Location=?, BitRate=?, Format=? WHERE Location=?',
+                            [None, None, None, unicode_song_path])
+                        myDB.action(
+                            'UPDATE alltracks SET Location=?, BitRate=?, Format=? WHERE Location=?',
+                            [None, None, None, unicode_song_path])
                        new_song_count += 1
                    else:
                        # This track information hasn't changed
@@ -177,9 +198,13 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,

    # Now we start track matching
    logger.info("%s new/modified songs found and added to the database" % new_song_count)
-    song_list = myDB.action("SELECT * FROM have WHERE Matched IS NULL AND LOCATION LIKE ?", [dir.decode(headphones.SYS_ENCODING, 'replace') + "%"])
-    total_number_of_songs = myDB.action("SELECT COUNT(*) FROM have WHERE Matched IS NULL AND LOCATION LIKE ?", [dir.decode(headphones.SYS_ENCODING, 'replace') + "%"]).fetchone()[0]
-    logger.info("Found " + str(total_number_of_songs) + " new/modified tracks in: '" + dir.decode(headphones.SYS_ENCODING, 'replace') + "'. Matching tracks to the appropriate releases....")
+    song_list = myDB.action("SELECT * FROM have WHERE Matched IS NULL AND LOCATION LIKE ?",
+                            [dir.decode(headphones.SYS_ENCODING, 'replace') + "%"])
+    total_number_of_songs = \
+        myDB.action("SELECT COUNT(*) FROM have WHERE Matched IS NULL AND LOCATION LIKE ?",
+                    [dir.decode(headphones.SYS_ENCODING, 'replace') + "%"]).fetchone()[0]
+    logger.info("Found " + str(total_number_of_songs) + " new/modified tracks in: '" + dir.decode(
+        headphones.SYS_ENCODING, 'replace') + "'. Matching tracks to the appropriate releases....")

    # Sort the song_list by most vague (e.g. no trackid or releaseid) to most specific (both trackid & releaseid)
    # When we insert into the database, the tracks with the most specific information will overwrite the more general matches
@@ -190,6 +215,7 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
    # We'll use this to give a % completion, just because the track matching might take a while
    song_count = 0
    latest_artist = []
+    last_completion_percentage = 0

    for song in song_list:

@@ -200,10 +226,11 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
            logger.info("Now matching songs by %s" % song['ArtistName'])

        song_count += 1
-        completion_percentage = float(song_count) / total_number_of_songs * 100
+        completion_percentage = math.floor(float(song_count) / total_number_of_songs * 1000) / 10

-        if completion_percentage % 10 == 0:
+        if completion_percentage >= (last_completion_percentage + 10):
            logger.info("Track matching is " + str(completion_percentage) + "% complete")
+            last_completion_percentage = completion_percentage

        # THE "MORE-SPECIFIC" CLAUSES HERE HAVE ALL BEEN REMOVED.  WHEN RUNNING A LIBRARY SCAN, THE ONLY CLAUSES THAT
        # EVER GOT HIT WERE [ARTIST/ALBUM/TRACK] OR CLEANNAME.  ARTISTID & RELEASEID ARE NEVER PASSED TO THIS FUNCTION,
@@ -212,7 +239,9 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,

        if song['ArtistName'] and song['AlbumTitle'] and song['TrackTitle']:

-            track = myDB.action('SELECT ArtistName, AlbumTitle, TrackTitle, AlbumID from tracks WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [song['ArtistName'], song['AlbumTitle'], song['TrackTitle']]).fetchone()
+            track = myDB.action(
+                'SELECT ArtistName, AlbumTitle, TrackTitle, AlbumID from tracks WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?',
+                [song['ArtistName'], song['AlbumTitle'], song['TrackTitle']]).fetchone()
            have_updated = False
            if track:
                controlValueDict = {'ArtistName': track['ArtistName'],
@@ -228,7 +257,8 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
                myDB.upsert("have", newValueDict2, controlValueDict2)
                have_updated = True
            else:
-                track = myDB.action('SELECT CleanName, AlbumID from tracks WHERE CleanName LIKE ?', [song['CleanName']]).fetchone()
+                track = myDB.action('SELECT CleanName, AlbumID from tracks WHERE CleanName LIKE ?',
+                                    [song['CleanName']]).fetchone()
                if track:
                    controlValueDict = {'CleanName': track['CleanName']}
                    newValueDict = {'Location': song['Location'],
@@ -246,7 +276,9 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
                    myDB.upsert("have", newValueDict2, controlValueDict2)
                    have_updated = True

-            alltrack = myDB.action('SELECT ArtistName, AlbumTitle, TrackTitle, AlbumID from alltracks WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [song['ArtistName'], song['AlbumTitle'], song['TrackTitle']]).fetchone()
+            alltrack = myDB.action(
+                'SELECT ArtistName, AlbumTitle, TrackTitle, AlbumID from alltracks WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?',
+                [song['ArtistName'], song['AlbumTitle'], song['TrackTitle']]).fetchone()
            if alltrack:
                controlValueDict = {'ArtistName': alltrack['ArtistName'],
                                    'AlbumTitle': alltrack['AlbumTitle'],
@@ -260,7 +292,9 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
                newValueDict2 = {'Matched': alltrack['AlbumID']}
                myDB.upsert("have", newValueDict2, controlValueDict2)
            else:
-                alltrack = myDB.action('SELECT CleanName, AlbumID from alltracks WHERE CleanName LIKE ?', [song['CleanName']]).fetchone()
+                alltrack = myDB.action(
+                    'SELECT CleanName, AlbumID from alltracks WHERE CleanName LIKE ?',
+                    [song['CleanName']]).fetchone()
                if alltrack:
                    controlValueDict = {'CleanName': alltrack['CleanName']}
                    newValueDict = {'Location': song['Location'],
@@ -285,9 +319,10 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,

            #######myDB.action('INSERT INTO have (ArtistName, AlbumTitle, TrackNumber, TrackTitle, TrackLength, BitRate, Genre, Date, TrackID, Location, CleanName, Format) VALUES( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', [song['ArtistName'], song['AlbumTitle'], song['TrackNumber'], song['TrackTitle'], song['TrackLength'], song['BitRate'], song['Genre'], song['Date'], song['TrackID'], song['Location'], CleanName, song['Format']])

-    logger.info('Completed matching tracks from directory: %s' % dir.decode(headphones.SYS_ENCODING, 'replace'))
+    logger.info('Completed matching tracks from directory: %s' % dir.decode(headphones.SYS_ENCODING,
+                                                                            'replace'))

-    if not append:
+    if not append or artistScan:
        logger.info('Updating scanned artist track counts')

        # Clean up the new artist list
@@ -315,8 +350,11 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
            # Have tracks are selected from tracks table and not all tracks because of duplicates
            # We update the track count upon an album switch to compliment this
            havetracks = (
-                len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistName like ? AND Location IS NOT NULL', [artist]))
-                + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"', [artist]))
+                len(myDB.select(
+                    'SELECT TrackTitle from tracks WHERE ArtistName like ? AND Location IS NOT NULL',
+                    [artist])) + len(myDB.select(
+                    'SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"',
+                    [artist]))
            )
            # Note: some people complain about having "artist have tracks" > # of tracks total in artist official releases
            # (can fix by getting rid of second len statement)
@@ -334,22 +372,29 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
                for artist in artist_list:
                    myDB.action('INSERT OR IGNORE INTO newartists VALUES (?)', [artist])

-        if headphones.CONFIG.DETECT_BITRATE:
+        if headphones.CONFIG.DETECT_BITRATE and bitrates:
            headphones.CONFIG.PREFERRED_BITRATE = sum(bitrates) / len(bitrates) / 1000

    else:
        # If we're appending a new album to the database, update the artists total track counts
        logger.info('Updating artist track counts')

-        havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [ArtistID])) + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"', [ArtistName]))
+        havetracks = len(
+            myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL',
+                        [ArtistID])) + len(myDB.select(
+            'SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"',
+            [ArtistName]))
        myDB.action('UPDATE artists SET HaveTracks=? WHERE ArtistID=?', [havetracks, ArtistID])

    if not append:
        update_album_status()
+
+    if not append and not artistScan:
        lastfm.getSimilar()

    logger.info('Library scan complete')

+
 # ADDED THIS SECTION TO MARK ALBUMS AS DOWNLOADED IF ARTISTS ARE ADDED EN MASSE BEFORE LIBRARY IS SCANNED


@@ -357,11 +402,13 @@ def update_album_status(AlbumID=None):
    myDB = db.DBConnection()
    logger.info('Counting matched tracks to mark albums as skipped/downloaded')
    if AlbumID:
-        album_status_updater = myDB.action('SELECT AlbumID, AlbumTitle, Status from albums WHERE AlbumID=?', [AlbumID])
+        album_status_updater = myDB.action(
+            'SELECT AlbumID, AlbumTitle, Status from albums WHERE AlbumID=?', [AlbumID])
    else:
        album_status_updater = myDB.action('SELECT AlbumID, AlbumTitle, Status from albums')
    for album in album_status_updater:
-        track_counter = myDB.action('SELECT Location from tracks where AlbumID=?', [album['AlbumID']])
+        track_counter = myDB.action('SELECT Location from tracks where AlbumID=?',
+                                    [album['AlbumID']])
        total_tracks = 0
        have_tracks = 0
        for track in track_counter:
@@ -374,7 +421,7 @@ def update_album_status(AlbumID=None):
            album_completion = 0
            logger.info('Album %s does not have any tracks in database' % album['AlbumTitle'])

-        if album_completion >= headphones.CONFIG.ALBUM_COMPLETION_PCT and album['Status'] == 'Skipped':
+        if album_completion >= headphones.CONFIG.ALBUM_COMPLETION_PCT:
            new_album_status = "Downloaded"

        # I don't think we want to change Downloaded->Skipped.....
@@ -2,11 +2,12 @@
 Locking-related classes
 """

-import headphones.logger
 import time
 import threading
 import Queue

+import headphones.logger
+

 class TimedLock(object):
    """
@@ -13,20 +13,20 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

-from headphones import helpers
-
-from logutils.queue import QueueHandler, QueueListener
 from logging import handlers
-
 import multiprocessing
 import contextlib
-import headphones
 import threading
 import traceback
 import logging
 import errno
 import sys
+
 import os
+from headphones import helpers
+from logutils.queue import QueueHandler, QueueListener
+import headphones
+

 # These settings are for file logging only
 FILENAME = "headphones.log"
@@ -39,6 +39,7 @@ logger = logging.getLogger("headphones")
 # Global queue for multiprocessing logging
 queue = None

+
 class LogListHandler(logging.Handler):
    """
    Log handler for Web UI.
@@ -149,8 +150,10 @@ def initLogger(console=False, log_dir=False, verbose=False):
    if log_dir:
        filename = os.path.join(log_dir, FILENAME)

-        file_formatter = logging.Formatter('%(asctime)s - %(levelname)-7s :: %(threadName)s : %(message)s', '%d-%b-%Y %H:%M:%S')
-        file_handler = handlers.RotatingFileHandler(filename, maxBytes=MAX_SIZE, backupCount=MAX_FILES)
+        file_formatter = logging.Formatter(
+            '%(asctime)s - %(levelname)-7s :: %(threadName)s : %(message)s', '%d-%b-%Y %H:%M:%S')
+        file_handler = handlers.RotatingFileHandler(filename, maxBytes=MAX_SIZE,
+                                                    backupCount=MAX_FILES)
        file_handler.setLevel(logging.DEBUG)
        file_handler.setFormatter(file_formatter)

@@ -158,7 +161,8 @@ def initLogger(console=False, log_dir=False, verbose=False):

    # Setup console logger
    if console:
-        console_formatter = logging.Formatter('%(asctime)s - %(levelname)s :: %(threadName)s : %(message)s', '%d-%b-%Y %H:%M:%S')
+        console_formatter = logging.Formatter(
+            '%(asctime)s - %(levelname)s :: %(threadName)s : %(message)s', '%d-%b-%Y %H:%M:%S')
        console_handler = logging.StreamHandler()
        console_handler.setFormatter(console_formatter)
        console_handler.setLevel(logging.DEBUG)
@@ -212,11 +216,13 @@ def initHooks(global_exceptions=True, thread_exceptions=True, pass_original=True
                    raise
                except:
                    excepthook(*sys.exc_info())
+
            self.run = new_run

        # Monkey patch the run() by monkey patching the __init__ method
        threading.Thread.__init__ = new_init

+
 # Expose logger methods
 info = logger.info
 warn = logger.warn
@@ -13,14 +13,13 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

-import re
 import htmlentitydefs

+import re
 from headphones import logger, request


 def getLyrics(artist, song):
-
    params = {"artist": artist.encode('utf-8'),
              "song": song.encode('utf-8'),
              "fmt": 'xml'
@@ -46,10 +45,13 @@ def getLyrics(artist, song):
        logger.warn('Error fetching lyrics from: %s' % lyricsurl)
        return

-    m = re.compile('''<div class='lyricbox'><div class='rtMatcher'>.*?</div>(.*?)<!--''').search(lyricspage)
+    m = re.compile('''<div class='lyricbox'><div class='rtMatcher'>.*?</div>(.*?)<!--''').search(
+        lyricspage)

    if not m:
-        m = re.compile('''<div class='lyricbox'><span style="padding:1em"><a href="/Category:Instrumental" title="Instrumental">''').search(lyricspage)
+        m = re.compile(
+            '''<div class='lyricbox'><span style="padding:1em"><a href="/Category:Instrumental" title="Instrumental">''').search(
+            lyricspage)
        if m:
            return u'(Instrumental)'
        else:
@@ -31,12 +31,12 @@ except ImportError:

 mb_lock = headphones.lock.TimedLock(0)

+
 # Quick fix to add mirror switching on the fly. Need to probably return the mbhost & mbport that's
 # being used, so we can send those values to the log


 def startmb():
-
    mbuser = None
    mbpass = None

@@ -51,7 +51,7 @@ def startmb():
        mbpass = headphones.CONFIG.CUSTOMPASS
        sleepytime = int(headphones.CONFIG.CUSTOMSLEEP)
    elif headphones.CONFIG.MIRROR == "headphones":
-        mbhost = "144.76.94.239"
+        mbhost = "codeshy.com"
        mbport = 8181
        mbuser = headphones.CONFIG.HPUSER
        mbpass = headphones.CONFIG.HPPASS
@@ -81,7 +81,8 @@ def startmb():
    if not headphones.CONFIG.CUSTOMAUTH and headphones.CONFIG.MIRROR == "custom":
        musicbrainzngs.disable_hpauth()

-    logger.debug('Using the following server values: MBHost: %s, MBPort: %i, Sleep Interval: %i', mbhost, mbport, sleepytime)
+    logger.debug('Using the following server values: MBHost: %s, MBPort: %i, Sleep Interval: %i',
+                 mbhost, mbport, sleepytime)

    return True

@@ -99,6 +100,14 @@ def findArtist(name, limit=1):
    with mb_lock:
        try:
            artistResults = musicbrainzngs.search_artists(limit=limit, **criteria)['artist-list']
+        except ValueError as e:
+            if "at least one query term is required" in e.message:
+                logger.error(
+                    "Tried to search without a term, or an empty one. Provided artist (probably emtpy): %s",
+                    name)
+                return False
+            else:
+                raise
        except musicbrainzngs.WebServiceError as e:
            logger.warn('Attempt to query MusicBrainz for %s failed (%s)' % (name, str(e)))
            mb_lock.snooze(5)
@@ -111,10 +120,12 @@ def findArtist(name, limit=1):
        else:
            uniquename = unicode(result['sort-name'])
        if result['name'] != uniquename and limit == 1:
-             logger.info('Found an artist with a disambiguation: %s - doing an album based search' % name)
+            logger.info(
+                'Found an artist with a disambiguation: %s - doing an album based search' % name)
            artistdict = findArtistbyAlbum(name)
            if not artistdict:
-                 logger.info('Cannot determine the best match from an artist/album search. Using top match instead')
+                logger.info(
+                    'Cannot determine the best match from an artist/album search. Using top match instead')
                artistlist.append({
                    # Just need the artist id if the limit is 1
                    #    'name':             unicode(result['sort-name']),
@@ -130,11 +141,13 @@ def findArtist(name, limit=1):
                'name': unicode(result['sort-name']),
                'uniquename': uniquename,
                'id': unicode(result['id']),
-                     'url': unicode("http://musicbrainz.org/artist/" + result['id']),#probably needs to be changed
+                'url': unicode("http://musicbrainz.org/artist/" + result['id']),
+                # probably needs to be changed
                'score': int(result['ext:score'])
            })
    return artistlist

+
 def findRelease(name, limit=1, artist=None):
    releaselist = []
    releaseResults = None
@@ -151,7 +164,8 @@ def findRelease(name, limit=1, artist=None):

    with mb_lock:
        try:
-            releaseResults = musicbrainzngs.search_releases(query=name, limit=limit, artist=artist)['release-list']
+            releaseResults = musicbrainzngs.search_releases(query=name, limit=limit, artist=artist)[
+                'release-list']
        except musicbrainzngs.WebServiceError as e:  # need to update exceptions
            logger.warn('Attempt to query MusicBrainz for "%s" failed: %s' % (name, str(e)))
            mb_lock.snooze(5)
@@ -200,8 +214,11 @@ def findRelease(name, limit=1, artist=None):
            'title': unicode(title),
            'id': unicode(result['artist-credit'][0]['artist']['id']),
            'albumid': unicode(result['id']),
-                    'url': unicode("http://musicbrainz.org/artist/" + result['artist-credit'][0]['artist']['id']),#probably needs to be changed
-                    'albumurl': unicode("http://musicbrainz.org/release/" + result['id']),#probably needs to be changed
+            'url': unicode(
+                "http://musicbrainz.org/artist/" + result['artist-credit'][0]['artist']['id']),
+            # probably needs to be changed
+            'albumurl': unicode("http://musicbrainz.org/release/" + result['id']),
+            # probably needs to be changed
            'score': int(result['ext:score']),
            'date': unicode(result['date']) if 'date' in result else '',
            'country': unicode(result['country']) if 'country' in result else '',
@@ -212,6 +229,7 @@ def findRelease(name, limit=1, artist=None):
        })
    return releaselist

+
 def findSeries(name, limit=1):
    serieslist = []
    seriesResults = None
@@ -241,11 +259,13 @@ def findSeries(name, limit=1):
            'name': unicode(result['name']),
            'type': unicode(result['type']),
            'id': unicode(result['id']),
-                     'url': unicode("http://musicbrainz.org/series/" + result['id']),#probably needs to be changed
+            'url': unicode("http://musicbrainz.org/series/" + result['id']),
+            # probably needs to be changed
            'score': int(result['ext:score'])
        })
    return serieslist

+
 def getArtist(artistid, extrasonly=False):
    artist_dict = {}
    artist = None
@@ -265,7 +285,9 @@ def getArtist(artistid, extrasonly=False):
            newRgs = newRgs['release-group-list']
            artist['release-group-list'] += newRgs
    except musicbrainzngs.WebServiceError as e:
-        logger.warn('Attempt to retrieve artist information from MusicBrainz failed for artistid: %s (%s)' % (artistid, str(e)))
+        logger.warn(
+            'Attempt to retrieve artist information from MusicBrainz failed for artistid: %s (%s)' % (
+            artistid, str(e)))
        mb_lock.snooze(5)
    except Exception as e:
        pass
@@ -293,7 +315,8 @@ def getArtist(artistid, extrasonly=False):
    myDB = db.DBConnection()

    try:
-        db_artist = myDB.action('SELECT IncludeExtras, Extras from artists WHERE ArtistID=?', [artistid]).fetchone()
+        db_artist = myDB.action('SELECT IncludeExtras, Extras from artists WHERE ArtistID=?',
+                                [artistid]).fetchone()
        includeExtras = db_artist['IncludeExtras']
    except IndexError:
        includeExtras = False
@@ -329,7 +352,9 @@ def getArtist(artistid, extrasonly=False):
                    newRgs = newRgs['release-group-list']
                    mb_extras_list += newRgs
            except musicbrainzngs.WebServiceError as e:
-                logger.warn('Attempt to retrieve artist information from MusicBrainz failed for artistid: %s (%s)' % (artistid, str(e)))
+                logger.warn(
+                    'Attempt to retrieve artist information from MusicBrainz failed for artistid: %s (%s)' % (
+                    artistid, str(e)))
                mb_lock.snooze(5)

            for rg in mb_extras_list:
@@ -348,14 +373,18 @@ def getArtist(artistid, extrasonly=False):
    artist_dict['releasegroups'] = releasegroups
    return artist_dict

+
 def getSeries(seriesid):
    series_dict = {}
    series = None
    try:
        with mb_lock:
-            series = musicbrainzngs.get_series_by_id(seriesid,includes=['release-group-rels'])['series']
+            series = musicbrainzngs.get_series_by_id(seriesid, includes=['release-group-rels'])[
+                'series']
    except musicbrainzngs.WebServiceError as e:
-        logger.warn('Attempt to retrieve series information from MusicBrainz failed for seriesid: %s (%s)' % (seriesid, str(e)))
+        logger.warn(
+            'Attempt to retrieve series information from MusicBrainz failed for seriesid: %s (%s)' % (
+            seriesid, str(e)))
        mb_lock.snooze(5)
    except Exception as e:
        pass
@@ -364,7 +393,8 @@ def getSeries(seriesid):
        return False

    if 'disambiguation' in series:
-        series_dict['artist_name'] = unicode(series['name'] + " (" + unicode(series['disambiguation']) + ")")
+        series_dict['artist_name'] = unicode(
+            series['name'] + " (" + unicode(series['disambiguation']) + ")")
    else:
        series_dict['artist_name'] = unicode(series['name'])

@@ -381,6 +411,7 @@ def getSeries(seriesid):
    series_dict['releasegroups'] = releasegroups
    return series_dict

+
 def getReleaseGroup(rgid):
    """
    Returns a list of releases in a release group
@@ -392,7 +423,9 @@ def getReleaseGroup(rgid):
                rgid, ["artists", "releases", "media", "discids", ])
            releaseGroup = releaseGroup['release-group']
    except musicbrainzngs.WebServiceError as e:
-        logger.warn('Attempt to retrieve information from MusicBrainz for release group "%s" failed (%s)' % (rgid, str(e)))
+        logger.warn(
+            'Attempt to retrieve information from MusicBrainz for release group "%s" failed (%s)' % (
+            rgid, str(e)))
        mb_lock.snooze(5)

    if not releaseGroup:
@@ -411,11 +444,16 @@ def getRelease(releaseid, include_artist_info=True):
    try:
        with mb_lock:
            if include_artist_info:
-                results = musicbrainzngs.get_release_by_id(releaseid, ["artists", "release-groups", "media", "recordings"]).get('release')
+                results = musicbrainzngs.get_release_by_id(releaseid,
+                                                           ["artists", "release-groups", "media",
+                                                            "recordings"]).get('release')
            else:
-                results = musicbrainzngs.get_release_by_id(releaseid, ["media", "recordings"]).get('release')
+                results = musicbrainzngs.get_release_by_id(releaseid, ["media", "recordings"]).get(
+                    'release')
    except musicbrainzngs.WebServiceError as e:
-        logger.warn('Attempt to retrieve information from MusicBrainz for release "%s" failed (%s)' % (releaseid, str(e)))
+        logger.warn(
+            'Attempt to retrieve information from MusicBrainz for release "%s" failed (%s)' % (
+            releaseid, str(e)))
        mb_lock.snooze(5)

    if not results:
@@ -443,7 +481,8 @@ def getRelease(releaseid, include_artist_info=True):
            try:
                release['rg_type'] = unicode(results['release-group']['type'])

-                if release['rg_type'] == 'Album' and 'secondary-type-list' in results['release-group']:
+                if release['rg_type'] == 'Album' and 'secondary-type-list' in results[
+                    'release-group']:
                    secondary_type = unicode(results['release-group']['secondary-type-list'][0])
                    if secondary_type != release['rg_type']:
                        release['rg_type'] = secondary_type
@@ -463,9 +502,13 @@ def getRelease(releaseid, include_artist_info=True):


 def get_new_releases(rgid, includeExtras=False, forcefull=False):
-
    myDB = db.DBConnection()
    results = []
+
+    release_status = "official"
+    if includeExtras and not headphones.CONFIG.OFFICIAL_RELEASES_ONLY:
+        release_status = []
+
    try:
        limit = 100
        newResults = None
@@ -474,6 +517,7 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
                newResults = musicbrainzngs.browse_releases(
                    release_group=rgid,
                    includes=['artist-credits', 'labels', 'recordings', 'release-groups', 'media'],
+                    release_status=release_status,
                    limit=limit,
                    offset=len(results))
            if 'release-list' not in newResults:
@@ -482,7 +526,9 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
            results += newResults

    except musicbrainzngs.WebServiceError as e:
-        logger.warn('Attempt to retrieve information from MusicBrainz for release group "%s" failed (%s)' % (rgid, str(e)))
+        logger.warn(
+            'Attempt to retrieve information from MusicBrainz for release group "%s" failed (%s)' % (
+            rgid, str(e)))
        mb_lock.snooze(5)
        return False

@@ -496,7 +542,8 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
        for release_mark in results:
            release_list.append(unicode(release_mark['id']))
            release_title = release_mark['title']
-        remove_missing_releases = myDB.action("SELECT ReleaseID FROM allalbums WHERE AlbumID=?", [rgid])
+        remove_missing_releases = myDB.action("SELECT ReleaseID FROM allalbums WHERE AlbumID=?",
+                                              [rgid])
        if remove_missing_releases:
            for items in remove_missing_releases:
                if items['ReleaseID'] not in release_list and items['ReleaseID'] != rgid:
@@ -505,22 +552,22 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
                    myDB.action("DELETE FROM tracks WHERE ReleaseID=?", [items['ReleaseID']])
                    myDB.action("DELETE FROM allalbums WHERE ReleaseID=?", [items['ReleaseID']])
                    myDB.action("DELETE FROM alltracks WHERE ReleaseID=?", [items['ReleaseID']])
-                    logger.info("Removing all references to release %s to reflect MusicBrainz" % items['ReleaseID'])
+                    logger.info(
+                        "Removing all references to release %s to reflect MusicBrainz" % items[
+                            'ReleaseID'])
                    force_repackage1 = 1
    else:
-        logger.info("There was either an error pulling data from MusicBrainz or there might not be any releases for this category")
+        logger.info(
+            "There was either an error pulling data from MusicBrainz or there might not be any releases for this category")

    num_new_releases = 0

    for releasedata in results:
-        #releasedata.get will return None if it doesn't have a status
-        #all official releases should have the Official status included
-        if not includeExtras and releasedata.get('status') != 'Official':
-            continue

        release = {}
        rel_id_check = releasedata['id']
-        album_checker = myDB.action('SELECT * from allalbums WHERE ReleaseID=?', [rel_id_check]).fetchone()
+        album_checker = myDB.action('SELECT * from allalbums WHERE ReleaseID=?',
+                                    [rel_id_check]).fetchone()
        if not album_checker or forcefull:
            # DELETE all references to this release since we're updating it anyway.
            myDB.action('DELETE from allalbums WHERE ReleaseID=?', [rel_id_check])
@@ -531,7 +578,8 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
            release['ReleaseDate'] = unicode(releasedata['date']) if 'date' in releasedata else None
            release['ReleaseID'] = releasedata['id']
            if 'release-group' not in releasedata:
-                raise Exception('No release group associated with release id ' + releasedata['id'] + ' album id' + rgid)
+                raise Exception('No release group associated with release id ' + releasedata[
+                    'id'] + ' album id' + rgid)
            release['Type'] = unicode(releasedata['release-group']['type'])

            if release['Type'] == 'Album' and 'secondary-type-list' in releasedata['release-group']:
@@ -547,7 +595,8 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
                logger.warn('Release ' + releasedata['id'] + ' has no Artists associated.')
                return False

-            release['ReleaseCountry'] = unicode(releasedata['country']) if 'country' in releasedata else u'Unknown'
+            release['ReleaseCountry'] = unicode(
+                releasedata['country']) if 'country' in releasedata else u'Unknown'
            # assuming that the list will contain media and that the format will be consistent
            try:
                additional_medium = ''
@@ -560,7 +609,8 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
                    disc_number = ''
                else:
                    disc_number = str(medium_count) + 'x'
-                packaged_medium = disc_number + releasedata['medium-list'][0]['format'] + additional_medium
+                packaged_medium = disc_number + releasedata['medium-list'][0][
+                    'format'] + additional_medium
                release['ReleaseFormat'] = unicode(packaged_medium)
            except:
                release['ReleaseFormat'] = u'Unknown'
@@ -587,7 +637,8 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):

            for track in release['Tracks']:

-                cleanname = helpers.cleanName(release['ArtistName'] + ' ' + release['AlbumTitle'] + ' ' + track['title'])
+                cleanname = helpers.cleanName(
+                    release['ArtistName'] + ' ' + release['AlbumTitle'] + ' ' + track['title'])

                controlValueDict = {"TrackID": track['id'],
                                    "ReleaseID": release['ReleaseID']}
@@ -603,10 +654,13 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
                                "CleanName": cleanname
                                }

-                match = myDB.action('SELECT Location, BitRate, Format from have WHERE CleanName=?', [cleanname]).fetchone()
+                match = myDB.action('SELECT Location, BitRate, Format from have WHERE CleanName=?',
+                                    [cleanname]).fetchone()

                if not match:
-                    match = myDB.action('SELECT Location, BitRate, Format from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [release['ArtistName'], release['AlbumTitle'], track['title']]).fetchone()
+                    match = myDB.action(
+                        'SELECT Location, BitRate, Format from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?',
+                        [release['ArtistName'], release['AlbumTitle'], track['title']]).fetchone()
                    # if not match:
                    # match = myDB.action('SELECT Location, BitRate, Format from have WHERE TrackID=?', [track['id']]).fetchone()
                if match:
@@ -614,17 +668,21 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
                    newValueDict['BitRate'] = match['BitRate']
                    newValueDict['Format'] = match['Format']
                    # myDB.action('UPDATE have SET Matched="True" WHERE Location=?', [match['Location']])
-                    myDB.action('UPDATE have SET Matched=? WHERE Location=?', (release['AlbumID'], match['Location']))
+                    myDB.action('UPDATE have SET Matched=? WHERE Location=?',
+                                (release['AlbumID'], match['Location']))

                myDB.upsert("alltracks", newValueDict, controlValueDict)
            num_new_releases = num_new_releases + 1
            if album_checker:
-                logger.info('[%s] Existing release %s (%s) updated' % (release['ArtistName'], release['AlbumTitle'], rel_id_check))
+                logger.info('[%s] Existing release %s (%s) updated' % (
+                release['ArtistName'], release['AlbumTitle'], rel_id_check))
            else:
-                logger.info('[%s] New release %s (%s) added' % (release['ArtistName'], release['AlbumTitle'], rel_id_check))
+                logger.info('[%s] New release %s (%s) added' % (
+                release['ArtistName'], release['AlbumTitle'], rel_id_check))
        if force_repackage1 == 1:
            num_new_releases = -1
-            logger.info('[%s] Forcing repackage of %s, since dB releases have been removed' % (release['ArtistName'], release_title))
+            logger.info('[%s] Forcing repackage of %s, since dB releases have been removed' % (
+            release['ArtistName'], release_title))
        else:
            num_new_releases = num_new_releases

@@ -650,14 +708,16 @@ def getTracksFromRelease(release):
            totalTracks += 1
    return tracks

+
 # Used when there is a disambiguation


 def findArtistbyAlbum(name):
-
    myDB = db.DBConnection()

-    artist = myDB.action('SELECT AlbumTitle from have WHERE ArtistName=? AND AlbumTitle IS NOT NULL ORDER BY RANDOM()', [name]).fetchone()
+    artist = myDB.action(
+        'SELECT AlbumTitle from have WHERE ArtistName=? AND AlbumTitle IS NOT NULL ORDER BY RANDOM()',
+        [name]).fetchone()

    if not artist:
        return False
@@ -698,7 +758,6 @@ def findArtistbyAlbum(name):


 def findAlbumID(artist=None, album=None):
-
    results = None
    chars = set('!?*-')

@@ -715,9 +774,11 @@ def findAlbumID(artist=None, album=None):
                album = '"' + album + '"'
            criteria = {'release': album.lower()}
        with mb_lock:
-            results = musicbrainzngs.search_release_groups(limit=1, **criteria).get('release-group-list')
+            results = musicbrainzngs.search_release_groups(limit=1, **criteria).get(
+                'release-group-list')
    except musicbrainzngs.WebServiceError as e:
-        logger.warn('Attempt to query MusicBrainz for %s - %s failed (%s)' % (artist, album, str(e)))
+        logger.warn(
+            'Attempt to query MusicBrainz for %s - %s failed (%s)' % (artist, album, str(e)))
        mb_lock.snooze(5)

    if not results:
@@ -13,12 +13,12 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

-import re
-import headphones
+import json

 from headphones import db, helpers, logger, request

-def update(artist_name,release_groups):
+
+def update(artistid, artist_name, release_groups):
    """ Pretty simple and crude function to find the artist page on metacritic,
    then parse that page to get critic & user scores for albums"""

@@ -31,28 +31,57 @@ def update(artist_name,release_groups):

    mc_artist_name = mc_artist_name.replace(" ", "-")

+    headers = {
+        'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2243.2 Safari/537.36'}
+
    url = "http://www.metacritic.com/person/" + mc_artist_name + "?filter-options=music&sort_options=date&num_items=100"

-    res = request.request_soup(url, parser='html.parser')
+    res = request.request_soup(url, headers=headers)
+
+    rows = None

    try:
-        rows = res.tbody.find_all('tr')
+        table = res.find("table", class_="credits person_credits")
+        rows = table.tbody.find_all('tr')
    except:
        logger.info("Unable to get metacritic scores for: %s" % artist_name)
-        return

    myDB = db.DBConnection()
+    artist = myDB.action('SELECT * FROM artists WHERE ArtistID=?', [artistid]).fetchone()

+    score_list = []
+
+    # If we couldn't get anything from MetaCritic for whatever reason,
+    # let's try to load scores from the db
+    if not rows:
+        if artist['MetaCritic']:
+            score_list = json.loads(artist['MetaCritic'])
+        else:
+            return
+
+    # If we did get scores, let's update the db with them
+    else:
        for row in rows:
            title = row.a.string
-        for rg in release_groups:
-            if rg['title'].lower() == title.lower():
            scores = row.find_all("span")
            critic_score = scores[0].string
            user_score = scores[1].string
+            score_dict = {'title': title, 'critic_score': critic_score, 'user_score': user_score}
+            score_list.append(score_dict)
+
+        # Save scores to the database
+        controlValueDict = {"ArtistID": artistid}
+        newValueDict = {'MetaCritic': json.dumps(score_list)}
+        myDB.upsert("artists", newValueDict, controlValueDict)
+
+    for score in score_list:
+        title = score['title']
+        # Iterate through the release groups we got passed to see if we can find
+        # a match
+        for rg in release_groups:
+            if rg['title'].lower() == title.lower():
+                critic_score = score['critic_score']
+                user_score = score['user_score']
                controlValueDict = {"AlbumID": rg['id']}
                newValueDict = {'CriticScore': critic_score, 'UserScore': user_score}
                myDB.upsert("albums", newValueDict, controlValueDict)
-
-
-
@@ -13,16 +13,17 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

-import os
 import time
 import shutil
 import subprocess
-import headphones
 import multiprocessing

+import os
+import headphones
 from headphones import logger
 from beets.mediafile import MediaFile

+
 # xld
 import getXldProfile

@@ -32,9 +33,11 @@ def encode(albumPath):

    # Return if xld details not found
    if use_xld:
-        (xldProfile, xldFormat, xldBitrate) = getXldProfile.getXldProfile(headphones.CONFIG.XLDPROFILE)
+        (xldProfile, xldFormat, xldBitrate) = getXldProfile.getXldProfile(
+            headphones.CONFIG.XLDPROFILE)
        if not xldFormat:
-            logger.error('Details for xld profile \'%s\' not found, files will not be re-encoded', xldProfile)
+            logger.error('Details for xld profile \'%s\' not found, files will not be re-encoded',
+                         xldProfile)
            return None
    else:
        xldProfile = None
@@ -60,7 +63,8 @@ def encode(albumPath):
        for music in f:
            if any(music.lower().endswith('.' + x.lower()) for x in headphones.MEDIA_FORMATS):
                if not use_xld:
-                    encoderFormat = headphones.CONFIG.ENCODEROUTPUTFORMAT.encode(headphones.SYS_ENCODING)
+                    encoderFormat = headphones.CONFIG.ENCODEROUTPUTFORMAT.encode(
+                        headphones.SYS_ENCODING)
                else:
                    xldMusicFile = os.path.join(r, music)
                    xldInfoMusic = MediaFile(xldMusicFile)
@@ -68,9 +72,11 @@ def encode(albumPath):

                if headphones.CONFIG.ENCODERLOSSLESS:
                    ext = os.path.normpath(os.path.splitext(music)[1].lstrip(".")).lower()
-                    if not use_xld and ext == 'flac' or use_xld and (ext != xldFormat and (xldInfoMusic.bitrate / 1000 > 400)):
+                    if not use_xld and ext == 'flac' or use_xld and (
+                            ext != xldFormat and (xldInfoMusic.bitrate / 1000 > 400)):
                        musicFiles.append(os.path.join(r, music))
-                        musicTemp = os.path.normpath(os.path.splitext(music)[0] + '.' + encoderFormat)
+                        musicTemp = os.path.normpath(
+                            os.path.splitext(music)[0] + '.' + encoderFormat)
                        musicTempFiles.append(os.path.join(tempDirEncode, musicTemp))
                    else:
                        logger.debug('%s is already encoded', music)
@@ -86,7 +92,7 @@ def encode(albumPath):
            encoder = os.path.join('/Applications', 'xld')
        elif headphones.CONFIG.ENCODER == 'lame':
            if headphones.SYS_PLATFORM == "win32":
-                ## NEED THE DEFAULT LAME INSTALL ON WIN!
+                # NEED THE DEFAULT LAME INSTALL ON WIN!
                encoder = "C:/Program Files/lame/lame.exe"
            else:
                encoder = "lame"
@@ -111,24 +117,31 @@ def encode(albumPath):

        if use_xld:
            if xldBitrate and (infoMusic.bitrate / 1000 <= xldBitrate):
-                logger.info('%s has bitrate <= %skb, will not be re-encoded', music.decode(headphones.SYS_ENCODING, 'replace'), xldBitrate)
+                logger.info('%s has bitrate <= %skb, will not be re-encoded',
+                            music.decode(headphones.SYS_ENCODING, 'replace'), xldBitrate)
            else:
                encode = True
        elif headphones.CONFIG.ENCODER == 'lame':
-            if not any(music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.' + x) for x in ["mp3", "wav"]):
-                logger.warn('Lame cannot encode %s format for %s, use ffmpeg', os.path.splitext(music)[1], music)
+            if not any(
+                    music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.' + x) for x
+                    in ["mp3", "wav"]):
+                logger.warn('Lame cannot encode %s format for %s, use ffmpeg',
+                            os.path.splitext(music)[1], music)
            else:
-                if music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.mp3') and (int(infoMusic.bitrate / 1000) <= headphones.CONFIG.BITRATE):
-                    logger.info('%s has bitrate <= %skb, will not be re-encoded', music, headphones.CONFIG.BITRATE)
+                if music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.mp3') and (
+                    int(infoMusic.bitrate / 1000) <= headphones.CONFIG.BITRATE):
+                    logger.info('%s has bitrate <= %skb, will not be re-encoded', music,
+                                headphones.CONFIG.BITRATE)
                else:
                    encode = True
        else:
            if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'ogg':
                if music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.ogg'):
-                    logger.warn('Cannot re-encode .ogg %s', music.decode(headphones.SYS_ENCODING, 'replace'))
+                    logger.warn('Cannot re-encode .ogg %s',
+                                music.decode(headphones.SYS_ENCODING, 'replace'))
                else:
                    encode = True
-            elif headphones.CONFIG.ENCODEROUTPUTFORMAT == 'mp3' or headphones.CONFIG.ENCODEROUTPUTFORMAT == 'm4a':
+            else:
                if music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.' + headphones.CONFIG.ENCODEROUTPUTFORMAT) and (int(infoMusic.bitrate / 1000) <= headphones.CONFIG.BITRATE):
                    logger.info('%s has bitrate <= %skb, will not be re-encoded', music, headphones.CONFIG.BITRATE)
                else:
@@ -185,7 +198,8 @@ def encode(albumPath):
        for dest in musicTempFiles:
            if not os.path.exists(dest):
                encoder_failed = True
-                logger.error("Encoded file '%s' does not exist in the destination temp directory", dest)
+                logger.error("Encoded file '%s' does not exist in the destination temp directory",
+                             dest)

    # No errors, move from temp to parent
    if not encoder_failed and musicTempFiles:
@@ -211,7 +225,9 @@ def encode(albumPath):

    # Return with error if any encoding errors
    if encoder_failed:
-        logger.error("One or more files failed to encode. Ensure you have the latest version of %s installed.", headphones.CONFIG.ENCODER)
+        logger.error(
+            "One or more files failed to encode. Ensure you have the latest version of %s installed.",
+            headphones.CONFIG.ENCODER)
        return None

    time.sleep(1)
@@ -269,7 +285,8 @@ def command(encoder, musicSource, musicDest, albumPath, xldProfile):
        if not headphones.CONFIG.ADVANCEDENCODER:
            opts.extend(['-h'])
            if headphones.CONFIG.ENCODERVBRCBR == 'cbr':
-                opts.extend(['--resample', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-b', str(headphones.CONFIG.BITRATE)])
+                opts.extend(['--resample', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-b',
+                             str(headphones.CONFIG.BITRATE)])
            elif headphones.CONFIG.ENCODERVBRCBR == 'vbr':
                opts.extend(['-v', str(headphones.CONFIG.ENCODERQUALITY)])
        else:
@@ -290,7 +307,8 @@ def command(encoder, musicSource, musicDest, albumPath, xldProfile):
            if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'm4a':
                opts.extend(['-strict', 'experimental'])
            if headphones.CONFIG.ENCODERVBRCBR == 'cbr':
-                opts.extend(['-ar', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-ab', str(headphones.CONFIG.BITRATE) + 'k'])
+                opts.extend(['-ar', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-ab',
+                             str(headphones.CONFIG.BITRATE) + 'k'])
            elif headphones.CONFIG.ENCODERVBRCBR == 'vbr':
                opts.extend(['-aq', str(headphones.CONFIG.ENCODERQUALITY)])
            opts.extend(['-y', '-ac', '2', '-vn'])
@@ -311,7 +329,8 @@ def command(encoder, musicSource, musicDest, albumPath, xldProfile):
            if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'm4a':
                opts.extend(['-strict', 'experimental'])
            if headphones.CONFIG.ENCODERVBRCBR == 'cbr':
-                opts.extend(['-ar', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-ab', str(headphones.CONFIG.BITRATE) + 'k'])
+                opts.extend(['-ar', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-ab',
+                             str(headphones.CONFIG.BITRATE) + 'k'])
            elif headphones.CONFIG.ENCODERVBRCBR == 'vbr':
                opts.extend(['-aq', str(headphones.CONFIG.ENCODERQUALITY)])
            opts.extend(['-y', '-ac', '2', '-vn'])
@@ -343,7 +362,8 @@ def command(encoder, musicSource, musicDest, albumPath, xldProfile):

    # Error if return code not zero
    if process.returncode:
-        logger.error('Encoding failed for %s' % (musicSource.decode(headphones.SYS_ENCODING, 'replace')))
+        logger.error(
+            'Encoding failed for %s' % (musicSource.decode(headphones.SYS_ENCODING, 'replace')))
        out = stdout if stdout else stderr
        out = out.decode(headphones.SYS_ENCODING, 'replace')
        outlast2lines = '\n'.join(out.splitlines()[-2:])
@@ -13,30 +13,25 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

-from headphones import logger, helpers, common, request
-
-from xml.dom import minidom
-from httplib import HTTPSConnection
-from urlparse import parse_qsl
 from urllib import urlencode
-from pynma import pynma
-
-import base64
-import cherrypy
 import urllib
-import urllib2
-import headphones
-import os.path
 import subprocess
-import gntp.notifier
 import json
-
-import oauth2 as oauth
-import pythontwitter as twitter
-
 from email.mime.text import MIMEText
 import smtplib
 import email.utils
+from httplib import HTTPSConnection
+from urlparse import parse_qsl
+import urllib2
+
+import os.path
+from headphones import logger, helpers, common, request
+from pynma import pynma
+import cherrypy
+import headphones
+import gntp.notifier
+import oauth2 as oauth
+import pythontwitter as twitter


 class GROWL(object):
@@ -185,7 +180,6 @@ class MPC(object):
    """

    def __init__(self):
-
        pass

    def notify(self):
@@ -218,9 +212,11 @@ class XBMC(object):
        url = host + '/jsonrpc'

        if self.password:
-            response = request.request_json(url, method="post", data=json.dumps(data), headers=headers, auth=(self.username, self.password))
+            response = request.request_json(url, method="post", data=json.dumps(data),
+                                            headers=headers, auth=(self.username, self.password))
        else:
-            response = request.request_json(url, method="post", data=json.dumps(data), headers=headers)
+            response = request.request_json(url, method="post", data=json.dumps(data),
+                                            headers=headers)

        if response:
            return response[0]['result']
@@ -249,15 +245,19 @@ class XBMC(object):
        for host in hosts:
            logger.info('Sending notification command to XMBC @ ' + host)
            try:
-                version = self._sendjson(host, 'Application.GetProperties', {'properties': ['version']})['version']['major']
+                version = \
+                self._sendjson(host, 'Application.GetProperties', {'properties': ['version']})[
+                    'version']['major']

                if version < 12:  # Eden
                    notification = header + "," + message + "," + time + "," + albumartpath
-                    notifycommand = {'command': 'ExecBuiltIn', 'parameter': 'Notification(' + notification + ')'}
+                    notifycommand = {'command': 'ExecBuiltIn',
+                                     'parameter': 'Notification(' + notification + ')'}
                    request = self._sendhttp(host, notifycommand)

                else:  # Frodo
-                    params = {'title': header, 'message': message, 'displaytime': int(time), 'image': albumartpath}
+                    params = {'title': header, 'message': message, 'displaytime': int(time),
+                              'image': albumartpath}
                    request = self._sendjson(host, 'GUI.ShowNotification', params)

                if not request:
@@ -316,34 +316,34 @@ class Plex(object):
        self.client_hosts = headphones.CONFIG.PLEX_CLIENT_HOST
        self.username = headphones.CONFIG.PLEX_USERNAME
        self.password = headphones.CONFIG.PLEX_PASSWORD
+        self.token = headphones.CONFIG.PLEX_TOKEN

    def _sendhttp(self, host, command):

-        username = self.username
-        password = self.password
+        url = host + '/xbmcCmds/xbmcHttp/?' + command

-        url_command = urllib.urlencode(command)
-
-        url = host + '/xbmcCmds/xbmcHttp/?' + url_command
-
-        req = urllib2.Request(url)
-
-        if password:
-            base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
-            req.add_header("Authorization", "Basic %s" % base64string)
-
-        logger.info('Plex url: %s' % url)
-
-        try:
-            handle = urllib2.urlopen(req)
-        except Exception as e:
-            logger.warn('Error opening Plex url: %s' % e)
-            return
-
-        response = handle.read().decode(headphones.SYS_ENCODING)
+        if self.password:
+            response = request.request_response(url, auth=(self.username, self.password))
+        else:
+            response = request.request_response(url)

        return response

+    def _sendjson(self, host, method, params={}):
+        data = [{'id': 0, 'jsonrpc': '2.0', 'method': method, 'params': params}]
+        headers = {'Content-Type': 'application/json'}
+        url = host + '/jsonrpc'
+
+        if self.password:
+            response = request.request_json(url, method="post", data=json.dumps(data),
+                                            headers=headers, auth=(self.username, self.password))
+        else:
+            response = request.request_json(url, method="post", data=json.dumps(data),
+                                            headers=headers)
+
+        if response:
+            return response[0]['result']
+
    def update(self):

        # From what I read you can't update the music library on a per directory or per path basis
@@ -354,13 +354,15 @@ class Plex(object):
        for host in hosts:
            logger.info('Sending library update command to Plex Media Server@ ' + host)
            url = "%s/library/sections" % host
-            try:
-                xml_sections = minidom.parse(urllib.urlopen(url))
-            except IOError, e:
-                logger.warn("Error while trying to contact Plex Media Server: %s" % e)
-                return False
+            if self.token:
+                params = {'X-Plex-Token': self.token}
+            else:
+                params = False
+
+            r = request.request_minidom(url, params=params)
+
+            sections = r.getElementsByTagName('Directory')

-            sections = xml_sections.getElementsByTagName('Directory')
            if not sections:
                logger.info(u"Plex Media Server not running on: " + host)
                return False
@@ -368,11 +370,7 @@ class Plex(object):
            for s in sections:
                if s.getAttribute('type') == "artist":
                    url = "%s/library/sections/%s/refresh" % (host, s.getAttribute('key'))
-                    try:
-                        urllib.urlopen(url)
-                    except Exception as e:
-                        logger.warn("Error updating library section for Plex Media Server: %s" % e)
-                        return False
+                    request.request_response(url, params=params)

    def notify(self, artist, album, albumartpath):

@@ -383,17 +381,28 @@ class Plex(object):
        time = "3000"  # in ms

        for host in hosts:
-            logger.info('Sending notification command to Plex Media Server @ ' + host)
+            logger.info('Sending notification command to Plex client @ ' + host)
            try:
+                version = \
+                self._sendjson(host, 'Application.GetProperties', {'properties': ['version']})[
+                    'version']['major']
+
+                if version < 12:  # Eden
                    notification = header + "," + message + "," + time + "," + albumartpath
-                notifycommand = {'command': 'ExecBuiltIn', 'parameter': 'Notification(' + notification + ')'}
+                    notifycommand = {'command': 'ExecBuiltIn',
+                                     'parameter': 'Notification(' + notification + ')'}
                    request = self._sendhttp(host, notifycommand)

+                else:  # Frodo
+                    params = {'title': header, 'message': message, 'displaytime': int(time),
+                              'image': albumartpath}
+                    request = self._sendjson(host, 'GUI.ShowNotification', params)
+
                if not request:
                    raise Exception

-            except:
-                logger.warn('Error sending notification request to Plex Media Server')
+            except Exception:
+                logger.error('Error sending notification request to Plex client @ ' + host)


 class NMA(object):
@@ -435,60 +444,37 @@ class NMA(object):


 class PUSHBULLET(object):
-
    def __init__(self):
        self.apikey = headphones.CONFIG.PUSHBULLET_APIKEY
        self.deviceid = headphones.CONFIG.PUSHBULLET_DEVICEID

-    def conf(self, options):
-        return cherrypy.config['config'].get('PUSHBULLET', options)
-
-    def notify(self, message, event):
+    def notify(self, message, status):
        if not headphones.CONFIG.PUSHBULLET_ENABLED:
            return

-        http_handler = HTTPSConnection("api.pushbullet.com")
+        url = "https://api.pushbullet.com/v2/pushes"

        data = {'type': "note",
                'title': "Headphones",
-                'body': message.encode("utf-8")}
+                'body': message + ': ' + status}
+
+        if self.deviceid:
+            data['device_iden'] = self.deviceid

-        http_handler.request("POST",
-                                "/v2/pushes",
        headers = {'Content-type': "application/json",
-                                            'Authorization': 'Basic %s' % base64.b64encode(headphones.CONFIG.PUSHBULLET_APIKEY + ":")},
-                                body=json.dumps(data))
-        response = http_handler.getresponse()
-        request_status = response.status
-        logger.debug(u"PushBullet response status: %r" % request_status)
-        logger.debug(u"PushBullet response headers: %r" % response.getheaders())
-        logger.debug(u"PushBullet response body: %r" % response.read())
+                   'Authorization': 'Bearer ' + headphones.CONFIG.PUSHBULLET_APIKEY}

-        if request_status == 200:
+        response = request.request_json(url, method="post", headers=headers, data=json.dumps(data))
+
+        if response:
            logger.info(u"PushBullet notifications sent.")
            return True
-        elif request_status >= 400 and request_status < 500:
-                logger.info(u"PushBullet request failed: %s" % response.reason)
-                return False
        else:
-                logger.info(u"PushBullet notification failed serverside.")
+            logger.info(u"PushBullet notification failed.")
            return False

-    def updateLibrary(self):
-        #For uniformity reasons not removed
-        return
-
-    def test(self, apikey, deviceid):
-
-        self.enabled = True
-        self.apikey = apikey
-        self.deviceid = deviceid
-
-        self.notify('Main Screen Activate', 'Test Message')
-

 class PUSHALOT(object):
-
    def notify(self, message, event):
        if not headphones.CONFIG.PUSHALOT_ENABLED:
            return
@@ -538,7 +524,8 @@ class Synoindex(object):
        path = os.path.abspath(path)

        if not self.util_exists():
-            logger.warn("Error sending notification: synoindex utility not found at %s" % self.util_loc)
+            logger.warn(
+                "Error sending notification: synoindex utility not found at %s" % self.util_loc)
            return

        if os.path.isfile(path):
@@ -546,13 +533,15 @@ class Synoindex(object):
        elif os.path.isdir(path):
            cmd_arg = '-A'
        else:
-            logger.warn("Error sending notification: Path passed to synoindex was not a file or folder.")
+            logger.warn(
+                "Error sending notification: Path passed to synoindex was not a file or folder.")
            return

        cmd = [self.util_loc, cmd_arg, path]
        logger.info("Calling synoindex command: %s" % str(cmd))
        try:
-            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, cwd=headphones.PROG_DIR)
+            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                                 cwd=headphones.PROG_DIR)
            out, error = p.communicate()
            # synoindex never returns any codes other than '0', highly irritating
        except OSError, e:
@@ -565,7 +554,6 @@ class Synoindex(object):


 class PUSHOVER(object):
-
    def __init__(self):
        self.enabled = headphones.CONFIG.PUSHOVER_ENABLED
        self.keys = headphones.CONFIG.PUSHOVER_KEYS
@@ -583,7 +571,7 @@ class PUSHOVER(object):
        if not headphones.CONFIG.PUSHOVER_ENABLED:
            return

-        http_handler = HTTPSConnection("api.pushover.net")
+        url = "https://api.pushover.net/1/messages.json"

        data = {'token': self.application_token,
                'user': headphones.CONFIG.PUSHOVER_KEYS,
@@ -591,24 +579,15 @@ class PUSHOVER(object):
                'message': message.encode("utf-8"),
                'priority': headphones.CONFIG.PUSHOVER_PRIORITY}

-        http_handler.request("POST",
-                                "/1/messages.json",
-                                headers={'Content-type': "application/x-www-form-urlencoded"},
-                                body=urlencode(data))
-        response = http_handler.getresponse()
-        request_status = response.status
-        logger.debug(u"Pushover response status: %r" % request_status)
-        logger.debug(u"Pushover response headers: %r" % response.getheaders())
-        logger.debug(u"Pushover response body: %r" % response.read())
+        headers = {'Content-type': "application/x-www-form-urlencoded"}

-        if request_status == 200:
+        response = request.request_response(url, method="POST", headers=headers, data=data)
+
+        if response:
            logger.info(u"Pushover notifications sent.")
            return True
-        elif request_status >= 400 and request_status < 500:
-                logger.info(u"Pushover request failed: %s" % response.reason)
-                return False
        else:
-                logger.info(u"Pushover notification failed.")
+            logger.error(u"Pushover notification failed.")
            return False

    def updateLibrary(self):
@@ -624,7 +603,6 @@ class PUSHOVER(object):


 class TwitterNotifier(object):
-
    REQUEST_TOKEN_URL = 'https://api.twitter.com/oauth/request_token'
    ACCESS_TOKEN_URL = 'https://api.twitter.com/oauth/access_token'
    AUTHORIZATION_URL = 'https://api.twitter.com/oauth/authorize'
@@ -636,14 +614,17 @@ class TwitterNotifier(object):

    def notify_snatch(self, title):
        if headphones.CONFIG.TWITTER_ONSNATCH:
-            self._notifyTwitter(common.notifyStrings[common.NOTIFY_SNATCH] + ': ' + title + ' at ' + helpers.now())
+            self._notifyTwitter(
+                common.notifyStrings[common.NOTIFY_SNATCH] + ': ' + title + ' at ' + helpers.now())

    def notify_download(self, title):
        if headphones.CONFIG.TWITTER_ENABLED:
-            self._notifyTwitter(common.notifyStrings[common.NOTIFY_DOWNLOAD] + ': ' + title + ' at ' + helpers.now())
+            self._notifyTwitter(common.notifyStrings[
+                                    common.NOTIFY_DOWNLOAD] + ': ' + title + ' at ' + helpers.now())

    def test_notify(self):
-        return self._notifyTwitter("This is a test notification from Headphones at " + helpers.now(), force=True)
+        return self._notifyTwitter(
+            "This is a test notification from Headphones at " + helpers.now(), force=True)

    def _get_authorization(self):

@@ -680,7 +661,8 @@ class TwitterNotifier(object):
        logger.info('oauth_consumer: ' + str(oauth_consumer))
        oauth_client = oauth.Client(oauth_consumer, token)
        logger.info('oauth_client: ' + str(oauth_client))
-        resp, content = oauth_client.request(self.ACCESS_TOKEN_URL, method='POST', body='oauth_verifier=%s' % key)
+        resp, content = oauth_client.request(self.ACCESS_TOKEN_URL, method='POST',
+                                             body='oauth_verifier=%s' % key)
        logger.info('resp, content: ' + str(resp) + ',' + str(content))

        access_token = dict(parse_qsl(content))
@@ -688,7 +670,8 @@ class TwitterNotifier(object):

        logger.info('resp[status] = ' + str(resp['status']))
        if resp['status'] != '200':
-            logger.info('The request for a token with did not succeed: ' + str(resp['status']), logger.ERROR)
+            logger.info('The request for a token with did not succeed: ' + str(resp['status']),
+                        logger.ERROR)
            return False
        else:
            logger.info('Your Twitter Access Token key: %s' % access_token['oauth_token'])
@@ -726,7 +709,6 @@ class TwitterNotifier(object):


 class OSX_NOTIFY(object):
-
    def __init__(self):
        try:
            self.objc = __import__("objc")
@@ -797,7 +779,6 @@ class OSX_NOTIFY(object):


 class BOXCAR(object):
-
    def __init__(self):
        self.url = 'https://new.boxcar.io/api/notifications'

@@ -824,7 +805,6 @@ class BOXCAR(object):


 class SubSonicNotifier(object):
-
    def __init__(self):
        self.host = headphones.CONFIG.SUBSONIC_HOST
        self.username = headphones.CONFIG.SUBSONIC_USERNAME
@@ -842,8 +822,8 @@ class SubSonicNotifier(object):
        request.request_response(self.host + "musicFolderSettings.view?scanNow",
                                 auth=(self.username, self.password))

-class Email(object):

+class Email(object):
    def notify(self, subject, message):

        message = MIMEText(message, 'plain', "utf-8")
@@ -852,20 +832,24 @@ class Email(object):
        message['To'] = headphones.CONFIG.EMAIL_TO

        try:
-            if (headphones.CONFIG.EMAIL_SSL):
-                mailserver = smtplib.SMTP_SSL(headphones.CONFIG.EMAIL_SMTP_SERVER, headphones.CONFIG.EMAIL_SMTP_PORT)
+            if headphones.CONFIG.EMAIL_SSL:
+                mailserver = smtplib.SMTP_SSL(headphones.CONFIG.EMAIL_SMTP_SERVER,
+                                              headphones.CONFIG.EMAIL_SMTP_PORT)
            else:
-                mailserver = smtplib.SMTP(headphones.CONFIG.EMAIL_SMTP_SERVER, headphones.CONFIG.EMAIL_SMTP_PORT)
+                mailserver = smtplib.SMTP(headphones.CONFIG.EMAIL_SMTP_SERVER,
+                                          headphones.CONFIG.EMAIL_SMTP_PORT)

-            if (headphones.CONFIG.EMAIL_TLS):
+            if headphones.CONFIG.EMAIL_TLS:
                mailserver.starttls()

            mailserver.ehlo()

            if headphones.CONFIG.EMAIL_SMTP_USER:
-                mailserver.login(headphones.CONFIG.EMAIL_SMTP_USER, headphones.CONFIG.EMAIL_SMTP_PASSWORD)
+                mailserver.login(headphones.CONFIG.EMAIL_SMTP_USER,
+                                 headphones.CONFIG.EMAIL_SMTP_PASSWORD)

-            mailserver.sendmail(headphones.CONFIG.EMAIL_FROM, headphones.CONFIG.EMAIL_TO, message.as_string())
+            mailserver.sendmail(headphones.CONFIG.EMAIL_FROM, headphones.CONFIG.EMAIL_TO,
+                                message.as_string())
            mailserver.quit()
            return True

@@ -19,18 +19,15 @@
 # along with Sick Beard.  If not, see <http://www.gnu.org/licenses/>.


-import httplib
-
-import headphones
-
 from base64 import standard_b64encode
+import httplib
 import xmlrpclib

+import headphones
 from headphones import logger


 def sendNZB(nzb):
-
    addToTop = False
    nzbgetXMLrpc = "%(protocol)s://%(username)s:%(password)s@%(host)s/xmlrpc"

@@ -45,17 +42,22 @@ def sendNZB(nzb):
        protocol = 'http'
        host = headphones.CONFIG.NZBGET_HOST.replace('http://', '', 1)

-    url = nzbgetXMLrpc % {"protocol": protocol, "host": host, "username": headphones.CONFIG.NZBGET_USERNAME, "password": headphones.CONFIG.NZBGET_PASSWORD}
+    url = nzbgetXMLrpc % {"protocol": protocol, "host": host,
+                          "username": headphones.CONFIG.NZBGET_USERNAME,
+                          "password": headphones.CONFIG.NZBGET_PASSWORD}

    nzbGetRPC = xmlrpclib.ServerProxy(url)
    try:
-        if nzbGetRPC.writelog("INFO", "headphones connected to drop of %s any moment now." % (nzb.name + ".nzb")):
+        if nzbGetRPC.writelog("INFO", "headphones connected to drop of %s any moment now." % (
+            nzb.name + ".nzb")):
            logger.debug(u"Successfully connected to NZBget")
        else:
-            logger.info(u"Successfully connected to NZBget, but unable to send a message" % (nzb.name + ".nzb"))
+            logger.info(u"Successfully connected to NZBget, but unable to send a message" % (
+            nzb.name + ".nzb"))

    except httplib.socket.error:
-        logger.error(u"Please check your NZBget host and port (if it is running). NZBget is not responding to this combination")
+        logger.error(
+            u"Please check your NZBget host and port (if it is running). NZBget is not responding to this combination")
        return False

    except xmlrpclib.ProtocolError, e:
@@ -82,7 +84,9 @@ def sendNZB(nzb):
        nzbget_version = int(nzbget_version_str[:nzbget_version_str.find(".")])
        if nzbget_version == 0:
            if nzbcontent64 is not None:
-                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, addToTop, nzbcontent64)
+                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb",
+                                                 headphones.CONFIG.NZBGET_CATEGORY, addToTop,
+                                                 nzbcontent64)
            else:
                # from headphones.common.providers.generic import GenericProvider
                # if nzb.resultType == "nzb":
@@ -95,24 +99,35 @@ def sendNZB(nzb):
                return False
        elif nzbget_version == 12:
            if nzbcontent64 is not None:
-                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb",
+                                                 headphones.CONFIG.NZBGET_CATEGORY,
+                                                 headphones.CONFIG.NZBGET_PRIORITY, False,
                                                 nzbcontent64, False, dupekey, dupescore, "score")
            else:
-                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb",
+                                                    headphones.CONFIG.NZBGET_CATEGORY,
+                                                    headphones.CONFIG.NZBGET_PRIORITY, False,
                                                    nzb.url, False, dupekey, dupescore, "score")
        # v13+ has a new combined append method that accepts both (url and content)
        # also the return value has changed from boolean to integer
        # (Positive number representing NZBID of the queue item. 0 and negative numbers represent error codes.)
        elif nzbget_version >= 13:
-            nzbget_result = True if nzbGetRPC.append(nzb.name + ".nzb", nzbcontent64 if nzbcontent64 is not None else nzb.url,
-                                                     headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False, False, dupekey, dupescore,
+            nzbget_result = True if nzbGetRPC.append(nzb.name + ".nzb",
+                                                     nzbcontent64 if nzbcontent64 is not None else nzb.url,
+                                                     headphones.CONFIG.NZBGET_CATEGORY,
+                                                     headphones.CONFIG.NZBGET_PRIORITY, False,
+                                                     False, dupekey, dupescore,
                                                     "score") > 0 else False
        else:
            if nzbcontent64 is not None:
-                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb",
+                                                 headphones.CONFIG.NZBGET_CATEGORY,
+                                                 headphones.CONFIG.NZBGET_PRIORITY, False,
                                                 nzbcontent64)
            else:
-                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb",
+                                                    headphones.CONFIG.NZBGET_CATEGORY,
+                                                    headphones.CONFIG.NZBGET_PRIORITY, False,
                                                    nzb.url)

        if nzbget_result:
@@ -122,5 +137,6 @@ def sendNZB(nzb):
            logger.error(u"NZBget could not add %s to the queue" % (nzb.name + ".nzb"))
            return False
    except:
-        logger.error(u"Connect Error to NZBget: could not add %s to the queue" % (nzb.name + ".nzb"))
+        logger.error(
+            u"Connect Error to NZBget: could not add %s to the queue" % (nzb.name + ".nzb"))
        return False
@@ -13,20 +13,20 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

-import os
-import re
 import shutil
 import uuid
-import beets
 import threading
 import itertools
-import headphones
+import tempfile

+import os
+import re
+import beets
+import headphones
 from beets import autotag
 from beets import config as beetsconfig
 from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
 from beetsplug import lyrics as beetslyrics
-
 from headphones import notifiers, utorrent, transmission
 from headphones import db, albumart, librarysync
 from headphones import logger, helpers, request, mb, music_encoder
@@ -48,19 +48,21 @@ def checkFolder():
                else:
                    download_dir = headphones.CONFIG.DOWNLOAD_TORRENT_DIR

-                album_path = os.path.join(download_dir, album['FolderName']).encode(headphones.SYS_ENCODING, 'replace')
+                album_path = os.path.join(download_dir, album['FolderName']).encode(
+                    headphones.SYS_ENCODING, 'replace')
                logger.debug("Checking if %s exists" % album_path)

                if os.path.exists(album_path):
-                    logger.info('Found "' + album['FolderName'] + '" in ' + album['Kind'] + ' download folder. Verifying....')
+                    logger.info('Found "' + album['FolderName'] + '" in ' + album[
+                        'Kind'] + ' download folder. Verifying....')
                    verify(album['AlbumID'], album_path, album['Kind'])
            else:
                logger.info("No folder name found for " + album['Title'])

    logger.debug("Checking download folder finished.")

-def verify(albumid, albumpath, Kind=None, forced=False):

+def verify(albumid, albumpath, Kind=None, forced=False, keep_original_folder=False):
    myDB = db.DBConnection()
    release = myDB.action('SELECT * from albums WHERE AlbumID=?', [albumid]).fetchone()
    tracks = myDB.select('SELECT * from tracks WHERE AlbumID=?', [albumid])
@@ -72,11 +74,14 @@ def verify(albumid, albumpath, Kind=None, forced=False):
        try:
            release_list = mb.getReleaseGroup(albumid)
        except Exception as e:
-            logger.error('Unable to get release information for manual album with rgid: %s. Error: %s', albumid, e)
+            logger.error(
+                'Unable to get release information for manual album with rgid: %s. Error: %s',
+                albumid, e)
            return

        if not release_list:
-            logger.error('Unable to get release information for manual album with rgid: %s', albumid)
+            logger.error('Unable to get release information for manual album with rgid: %s',
+                         albumid)
            return

        # Since we're just using this to create the bare minimum information to
@@ -85,7 +90,9 @@ def verify(albumid, albumpath, Kind=None, forced=False):
        release_dict = mb.getRelease(releaseid)

        if not release_dict:
-            logger.error('Unable to get release information for manual album with rgid: %s. Cannot continue', albumid)
+            logger.error(
+                'Unable to get release information for manual album with rgid: %s. Cannot continue',
+                albumid)
            return

        # Check if the artist is added to the database. In case the database is
@@ -93,7 +100,9 @@ def verify(albumid, albumpath, Kind=None, forced=False):
        # prevents new artists from appearing suddenly. In case forced is True,
        # this check is skipped, since it is assumed the user wants this.
        if headphones.CONFIG.FREEZE_DB and not forced:
-            artist = myDB.select("SELECT ArtistName, ArtistID FROM artists WHERE ArtistId=? OR ArtistName=?", [release_dict['artist_id'], release_dict['artist_name']])
+            artist = myDB.select(
+                "SELECT ArtistName, ArtistID FROM artists WHERE ArtistId=? OR ArtistName=?",
+                [release_dict['artist_id'], release_dict['artist_name']])

            if not artist:
                logger.warn("Continuing would add new artist '%s' (ID %s), " \
@@ -101,10 +110,16 @@ def verify(albumid, albumpath, Kind=None, forced=False):
                            "album with rgid: %s", release_dict['artist_name'],
                            release_dict['artist_id'], albumid)

-                myDB.action('UPDATE snatched SET status = "Frozen" WHERE status NOT LIKE "Seed%" and AlbumID=?', [albumid])
+                myDB.action(
+                    'UPDATE snatched SET status = "Frozen" WHERE status NOT LIKE "Seed%" and AlbumID=?',
+                    [albumid])
                frozen = re.search(r' \(Frozen\)(?:\[\d+\])?', albumpath)
                if not frozen:
+                    if headphones.CONFIG.RENAME_FROZEN:
                        renameUnprocessedFolder(albumpath, tag="Frozen")
+                    else:
+                        logger.warn(u"Won't rename %s to mark as 'Frozen', because it is disabled.",
+                                    albumpath.decode(headphones.SYS_ENCODING, 'replace'))
                return

        logger.info(u"Now adding/updating artist: " + release_dict['artist_name'])
@@ -120,7 +135,8 @@ def verify(albumid, albumpath, Kind=None, forced=False):
                        "DateAdded": helpers.today(),
                        "Status": "Paused"}

-        logger.info("ArtistID: " + release_dict['artist_id'] + " , ArtistName: " + release_dict['artist_name'])
+        logger.info("ArtistID: " + release_dict['artist_id'] + " , ArtistName: " + release_dict[
+            'artist_name'])

        if headphones.CONFIG.INCLUDE_EXTRAS:
            newValueDict['IncludeExtras'] = 1
@@ -147,7 +163,6 @@ def verify(albumid, albumpath, Kind=None, forced=False):
        # Delete existing tracks associated with this AlbumID since we're going to replace them and don't want any extras
        myDB.action('DELETE from tracks WHERE AlbumID=?', [albumid])
        for track in release_dict['tracks']:
-
            controlValueDict = {"TrackID": track['id'],
                                "AlbumID": albumid}

@@ -166,7 +181,8 @@ def verify(albumid, albumpath, Kind=None, forced=False):
        newValueDict = {"Status": "Paused"}

        myDB.upsert("artists", newValueDict, controlValueDict)
-        logger.info(u"Addition complete for: " + release_dict['title'] + " - " + release_dict['artist_name'])
+        logger.info(u"Addition complete for: " + release_dict['title'] + " - " + release_dict[
+            'artist_name'])

        release = myDB.action('SELECT * from albums WHERE AlbumID=?', [albumid]).fetchone()
        tracks = myDB.select('SELECT * from tracks WHERE AlbumID=?', [albumid])
@@ -182,11 +198,14 @@ def verify(albumid, albumpath, Kind=None, forced=False):
                downloaded_cuecount += 1
            # if any of the files end in *.part, we know the torrent isn't done yet. Process if forced, though
            elif files.lower().endswith(('.part', '.utpart')) and not forced:
-                logger.info("Looks like " + os.path.basename(albumpath).decode(headphones.SYS_ENCODING, 'replace') + " isn't complete yet. Will try again on the next run")
+                logger.info(
+                    "Looks like " + os.path.basename(albumpath).decode(headphones.SYS_ENCODING,
+                                                                       'replace') + " isn't complete yet. Will try again on the next run")
                return

    # Split cue
-    if headphones.CONFIG.CUE_SPLIT and downloaded_cuecount and downloaded_cuecount >= len(downloaded_track_list):
+    if headphones.CONFIG.CUE_SPLIT and downloaded_cuecount and downloaded_cuecount >= len(
+            downloaded_track_list):
        if headphones.CONFIG.KEEP_TORRENT_FILES and Kind == "torrent":
            albumpath = helpers.preserve_torrent_directory(albumpath)
        if albumpath and helpers.cue_split(albumpath):
@@ -199,7 +218,10 @@ def verify(albumid, albumpath, Kind=None, forced=False):
        try:
            f = MediaFile(downloaded_track)
        except Exception as e:
-            logger.info(u"Exception from MediaFile for: " + downloaded_track.decode(headphones.SYS_ENCODING, 'replace') + u" : " + unicode(e))
+            logger.info(
+                u"Exception from MediaFile for: " + downloaded_track.decode(headphones.SYS_ENCODING,
+                                                                            'replace') + u" : " + unicode(
+                    e))
            continue

        if not f.artist:
@@ -216,7 +238,8 @@ def verify(albumid, albumpath, Kind=None, forced=False):
        logger.debug('Matching metadata album: %s with album name: %s' % (metaalbum, dbalbum))

        if metaartist == dbartist and metaalbum == dbalbum:
-            doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list, Kind)
+            doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list, Kind,
+                             keep_original_folder)
            return

    # test #2: filenames
@@ -234,7 +257,8 @@ def verify(albumid, albumpath, Kind=None, forced=False):
            logger.debug('Checking if track title: %s is in file name: %s' % (dbtrack, filetrack))

            if dbtrack in filetrack:
-                doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list, Kind)
+                doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list, Kind,
+                                 keep_original_folder)
                return

    # test #3: number of songs and duration
@@ -266,29 +290,39 @@ def verify(albumid, albumpath, Kind=None, forced=False):
            logger.debug('Database track duration: %i' % db_track_duration)
            delta = abs(downloaded_track_duration - db_track_duration)
            if delta < 240:
-                doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list, Kind)
+                doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list, Kind,
+                                 keep_original_folder)
                return

-    logger.warn(u'Could not identify album: %s. It may not be the intended album.' % albumpath.decode(headphones.SYS_ENCODING, 'replace'))
-    myDB.action('UPDATE snatched SET status = "Unprocessed" WHERE status NOT LIKE "Seed%" and AlbumID=?', [albumid])
+    logger.warn(u'Could not identify album: %s. It may not be the intended album.',
+                albumpath.decode(headphones.SYS_ENCODING, 'replace'))
+    myDB.action(
+        'UPDATE snatched SET status = "Unprocessed" WHERE status NOT LIKE "Seed%" and AlbumID=?',
+        [albumid])
    processed = re.search(r' \(Unprocessed\)(?:\[\d+\])?', albumpath)
    if not processed:
+        if headphones.CONFIG.RENAME_UNPROCESSED:
            renameUnprocessedFolder(albumpath, tag="Unprocessed")
+        else:
+            logger.warn(u"Won't rename %s to mark as 'Unprocessed', because it is disabled.",
+                        albumpath.decode(headphones.SYS_ENCODING, 'replace'))


-def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list, Kind=None):
-
+def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list, Kind=None,
+                     keep_original_folder=False):
    logger.info('Starting post-processing for: %s - %s' % (release['ArtistName'], release['AlbumTitle']))
+    new_folder = None
    # Check to see if we're preserving the torrent dir
-    if headphones.CONFIG.KEEP_TORRENT_FILES and Kind == "torrent" and 'headphones-modified' not in albumpath:
-        new_folder = os.path.join(albumpath, 'headphones-modified'.encode(headphones.SYS_ENCODING, 'replace'))
-        logger.info("Copying files to 'headphones-modified' subfolder to preserve downloaded files for seeding")
+    if (headphones.CONFIG.KEEP_TORRENT_FILES and Kind == "torrent" and 'headphones-modified' not in albumpath) or headphones.CONFIG.KEEP_ORIGINAL_FOLDER or keep_original_folder:
+        new_folder = os.path.join(tempfile.mkdtemp(prefix="headphones_"), "headphones")
+        logger.info("Copying files to " + new_folder.decode(headphones.SYS_ENCODING, 'replace') + " subfolder to preserve downloaded files for seeding")
        try:
            shutil.copytree(albumpath, new_folder)
            # Update the album path with the new location
            albumpath = new_folder
        except Exception as e:
            logger.warn("Cannot copy/move files to temp folder: " + new_folder.decode(headphones.SYS_ENCODING, 'replace') + ". Not continuing. Error: " + str(e))
+            shutil.rmtree(new_folder)
            return

        # Need to update the downloaded track list with the new location.
@@ -316,6 +350,8 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
            return
        except IOError:
            logger.error("Unable to find media file: %s. Not continuing.")
+            if new_folder:
+                shutil.rmtree(new_folder)
            return

        # If one of the options below is set, it will access/touch/modify the
@@ -334,6 +370,8 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
                logger.error("Track file is not writable. This is required " \
                             "for some post processing steps: %s. Not continuing.",
                             downloaded_track.decode(headphones.SYS_ENCODING, "replace"))
+                if new_folder:
+                    shutil.rmtree(new_folder)
                return

    # start encoding
@@ -341,6 +379,8 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
        downloaded_track_list = music_encoder.encode(albumpath)

        if not downloaded_track_list:
+            if new_folder:
+                shutil.rmtree(new_folder)
            return

    artwork = None
@@ -372,7 +412,11 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
        addAlbumArt(artwork, albumpath, release)

    if headphones.CONFIG.CORRECT_METADATA:
-        correctMetadata(albumid, release, downloaded_track_list)
+        correctedMetadata = correctMetadata(albumid, release, downloaded_track_list)
+        if not correctedMetadata and headphones.CONFIG.DO_NOT_PROCESS_UNMATCHED:
+            if new_folder:
+                shutil.rmtree(new_folder)
+            return

    if headphones.CONFIG.EMBED_LYRICS:
        embedLyrics(downloaded_track_list)
@@ -381,7 +425,8 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
        renameFiles(albumpath, downloaded_track_list, release)

    if headphones.CONFIG.MOVE_FILES and not headphones.CONFIG.DESTINATION_DIR:
-        logger.error('No DESTINATION_DIR has been set. Set "Destination Directory" to the parent directory you want to move the files to')
+        logger.error(
+            'No DESTINATION_DIR has been set. Set "Destination Directory" to the parent directory you want to move the files to')
        albumpaths = [albumpath]
    elif headphones.CONFIG.MOVE_FILES and headphones.CONFIG.DESTINATION_DIR:
        albumpaths = moveFiles(albumpath, release, tracks)
@@ -392,15 +437,20 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,

    myDB = db.DBConnection()
    myDB.action('UPDATE albums SET status = "Downloaded" WHERE AlbumID=?', [albumid])
-    myDB.action('UPDATE snatched SET status = "Processed" WHERE Status NOT LIKE "Seed%" and AlbumID=?', [albumid])
+    myDB.action(
+        'UPDATE snatched SET status = "Processed" WHERE Status NOT LIKE "Seed%" and AlbumID=?',
+        [albumid])

    # Check if torrent has finished seeding
    if headphones.CONFIG.TORRENT_DOWNLOADER == 1 or headphones.CONFIG.TORRENT_DOWNLOADER == 2:
-        seed_snatched = myDB.action('SELECT * from snatched WHERE Status="Seed_Snatched" and AlbumID=?', [albumid]).fetchone()
+        seed_snatched = myDB.action(
+            'SELECT * from snatched WHERE Status="Seed_Snatched" and AlbumID=?',
+            [albumid]).fetchone()
        if seed_snatched:
            hash = seed_snatched['FolderName']
            torrent_removed = False
-            logger.info(u'%s - %s. Checking if torrent has finished seeding and can be removed' % (release['ArtistName'], release['AlbumTitle']))
+            logger.info(u'%s - %s. Checking if torrent has finished seeding and can be removed' % (
+            release['ArtistName'], release['AlbumTitle']))
            if headphones.CONFIG.TORRENT_DOWNLOADER == 1:
                torrent_removed = transmission.removeTorrent(hash, True)
            else:
@@ -408,15 +458,20 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,

            # Torrent removed, delete the snatched record, else update Status for scheduled job to check
            if torrent_removed:
-                myDB.action('DELETE from snatched WHERE status = "Seed_Snatched" and AlbumID=?', [albumid])
+                myDB.action('DELETE from snatched WHERE status = "Seed_Snatched" and AlbumID=?',
+                            [albumid])
            else:
-                myDB.action('UPDATE snatched SET status = "Seed_Processed" WHERE status = "Seed_Snatched" and AlbumID=?', [albumid])
+                myDB.action(
+                    'UPDATE snatched SET status = "Seed_Processed" WHERE status = "Seed_Snatched" and AlbumID=?',
+                    [albumid])

    # Update the have tracks for all created dirs:
    for albumpath in albumpaths:
-        librarysync.libraryScan(dir=albumpath, append=True, ArtistID=release['ArtistID'], ArtistName=release['ArtistName'])
+        librarysync.libraryScan(dir=albumpath, append=True, ArtistID=release['ArtistID'],
+                                ArtistName=release['ArtistName'])

-    logger.info(u'Post-processing for %s - %s complete' % (release['ArtistName'], release['AlbumTitle']))
+    logger.info(
+        u'Post-processing for %s - %s complete' % (release['ArtistName'], release['AlbumTitle']))

    pushmessage = release['ArtistName'] + ' - ' + release['AlbumTitle']
    statusmessage = "Download and Postprocessing completed"
@@ -475,7 +530,7 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
    if headphones.CONFIG.PUSHBULLET_ENABLED:
        logger.info(u"PushBullet request")
        pushbullet = notifiers.PUSHBULLET()
-        pushbullet.notify(pushmessage, "Download and Postprocessing completed")
+        pushbullet.notify(pushmessage, statusmessage)

    if headphones.CONFIG.TWITTER_ENABLED:
        logger.info(u"Sending Twitter notification")
@@ -514,6 +569,9 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
        subject = release['ArtistName'] + ' - ' + release['AlbumTitle']
        email.notify(subject, "Download and Postprocessing completed")

+    if new_folder:
+        shutil.rmtree(new_folder)
+

 def embedAlbumArt(artwork, downloaded_track_list):
    logger.info('Embedding album art')
@@ -522,7 +580,8 @@ def embedAlbumArt(artwork, downloaded_track_list):
        try:
            f = MediaFile(downloaded_track)
        except:
-            logger.error(u'Could not read %s. Not adding album art' % downloaded_track.decode(headphones.SYS_ENCODING, 'replace'))
+            logger.error(u'Could not read %s. Not adding album art' % downloaded_track.decode(
+                headphones.SYS_ENCODING, 'replace'))
            continue

        logger.debug('Adding album art to: %s' % downloaded_track)
@@ -531,7 +590,8 @@ def embedAlbumArt(artwork, downloaded_track_list):
            f.art = artwork
            f.save()
        except Exception as e:
-            logger.error(u'Error embedding album art to: %s. Error: %s' % (downloaded_track.decode(headphones.SYS_ENCODING, 'replace'), str(e)))
+            logger.error(u'Error embedding album art to: %s. Error: %s' % (
+            downloaded_track.decode(headphones.SYS_ENCODING, 'replace'), str(e)))
            continue


@@ -551,9 +611,11 @@ def addAlbumArt(artwork, albumpath, release):
              '$year': year
              }

-    album_art_name = helpers.replace_all(headphones.CONFIG.ALBUM_ART_FORMAT.strip(), values) + ".jpg"
+    album_art_name = helpers.replace_all(headphones.CONFIG.ALBUM_ART_FORMAT.strip(),
+                                         values) + ".jpg"

-    album_art_name = helpers.replace_illegal_chars(album_art_name).encode(headphones.SYS_ENCODING, 'replace')
+    album_art_name = helpers.replace_illegal_chars(album_art_name).encode(headphones.SYS_ENCODING,
+                                                                          'replace')

    if headphones.CONFIG.FILE_UNDERSCORES:
        album_art_name = album_art_name.replace(' ', '_')
@@ -579,7 +641,8 @@ def cleanupFiles(albumpath):
                try:
                    os.remove(os.path.join(r, files))
                except Exception as e:
-                    logger.error(u'Could not remove file: %s. Error: %s' % (files.decode(headphones.SYS_ENCODING, 'replace'), e))
+                    logger.error(u'Could not remove file: %s. Error: %s' % (
+                    files.decode(headphones.SYS_ENCODING, 'replace'), e))


 def renameNFO(albumpath):
@@ -588,12 +651,15 @@ def renameNFO(albumpath):
    for r, d, f in os.walk(albumpath):
        for file in f:
            if file.lower().endswith('.nfo'):
-                logger.debug('Renaming: "%s" to "%s"' % (file.decode(headphones.SYS_ENCODING, 'replace'), file.decode(headphones.SYS_ENCODING, 'replace') + '-orig'))
+                logger.debug('Renaming: "%s" to "%s"' % (
+                file.decode(headphones.SYS_ENCODING, 'replace'),
+                file.decode(headphones.SYS_ENCODING, 'replace') + '-orig'))
                try:
                    new_file_name = os.path.join(r, file)[:-3] + 'orig.nfo'
                    os.rename(os.path.join(r, file), new_file_name)
                except Exception as e:
-                    logger.error(u'Could not rename file: %s. Error: %s' % (os.path.join(r, file).decode(headphones.SYS_ENCODING, 'replace'), e))
+                    logger.error(u'Could not rename file: %s. Error: %s' % (
+                    os.path.join(r, file).decode(headphones.SYS_ENCODING, 'replace'), e))


 def moveFiles(albumpath, release, tracks):
@@ -623,7 +689,8 @@ def moveFiles(albumpath, release, tracks):

    for r, d, f in os.walk(albumpath):
        try:
-            origfolder = os.path.basename(os.path.normpath(r).decode(headphones.SYS_ENCODING, 'replace'))
+            origfolder = os.path.basename(
+                os.path.normpath(r).decode(headphones.SYS_ENCODING, 'replace'))
        except:
            origfolder = u''

@@ -665,15 +732,20 @@ def moveFiles(albumpath, release, tracks):
            files_to_move.append(os.path.join(r, files))
            if any(files.lower().endswith('.' + x.lower()) for x in headphones.LOSSY_MEDIA_FORMATS):
                lossy_media = True
-            if any(files.lower().endswith('.' + x.lower()) for x in headphones.LOSSLESS_MEDIA_FORMATS):
+            if any(files.lower().endswith('.' + x.lower()) for x in
+                   headphones.LOSSLESS_MEDIA_FORMATS):
                lossless_media = True

    # Do some sanity checking to see what directories we need to create:
    make_lossy_folder = False
    make_lossless_folder = False

-    lossy_destination_path = os.path.normpath(os.path.join(headphones.CONFIG.DESTINATION_DIR, folder)).encode(headphones.SYS_ENCODING, 'replace')
-    lossless_destination_path = os.path.normpath(os.path.join(headphones.CONFIG.LOSSLESS_DESTINATION_DIR, folder)).encode(headphones.SYS_ENCODING, 'replace')
+    lossy_destination_path = os.path.normpath(
+        os.path.join(headphones.CONFIG.DESTINATION_DIR, folder)).encode(headphones.SYS_ENCODING,
+                                                                        'replace')
+    lossless_destination_path = os.path.normpath(
+        os.path.join(headphones.CONFIG.LOSSLESS_DESTINATION_DIR, folder)).encode(
+        headphones.SYS_ENCODING, 'replace')

    # If they set a destination dir for lossless media, only create the lossy folder if there is lossy media
    if headphones.CONFIG.LOSSLESS_DESTINATION_DIR:
@@ -697,7 +769,9 @@ def moveFiles(albumpath, release, tracks):
                try:
                    shutil.rmtree(lossless_destination_path)
                except Exception as e:
-                    logger.error("Error deleting existing folder: %s. Creating duplicate folder. Error: %s" % (lossless_destination_path.decode(headphones.SYS_ENCODING, 'replace'), e))
+                    logger.error(
+                        "Error deleting existing folder: %s. Creating duplicate folder. Error: %s" % (
+                        lossless_destination_path.decode(headphones.SYS_ENCODING, 'replace'), e))
                    create_duplicate_folder = True

            if not headphones.CONFIG.REPLACE_EXISTING_FOLDERS or create_duplicate_folder:
@@ -706,7 +780,9 @@ def moveFiles(albumpath, release, tracks):
                i = 1
                while True:
                    newfolder = temp_folder + '[%i]' % i
-                    lossless_destination_path = os.path.normpath(os.path.join(headphones.CONFIG.LOSSLESS_DESTINATION_DIR, newfolder)).encode(headphones.SYS_ENCODING, 'replace')
+                    lossless_destination_path = os.path.normpath(
+                        os.path.join(headphones.CONFIG.LOSSLESS_DESTINATION_DIR, newfolder)).encode(
+                        headphones.SYS_ENCODING, 'replace')
                    if os.path.exists(lossless_destination_path):
                        i += 1
                    else:
@@ -717,7 +793,8 @@ def moveFiles(albumpath, release, tracks):
            try:
                os.makedirs(lossless_destination_path)
            except Exception as e:
-                logger.error('Could not create lossless folder for %s. (Error: %s)' % (release['AlbumTitle'], e))
+                logger.error('Could not create lossless folder for %s. (Error: %s)' % (
+                release['AlbumTitle'], e))
                if not make_lossy_folder:
                    return [albumpath]

@@ -730,7 +807,9 @@ def moveFiles(albumpath, release, tracks):
                try:
                    shutil.rmtree(lossy_destination_path)
                except Exception as e:
-                    logger.error("Error deleting existing folder: %s. Creating duplicate folder. Error: %s" % (lossy_destination_path.decode(headphones.SYS_ENCODING, 'replace'), e))
+                    logger.error(
+                        "Error deleting existing folder: %s. Creating duplicate folder. Error: %s" % (
+                        lossy_destination_path.decode(headphones.SYS_ENCODING, 'replace'), e))
                    create_duplicate_folder = True

            if not headphones.CONFIG.REPLACE_EXISTING_FOLDERS or create_duplicate_folder:
@@ -739,7 +818,9 @@ def moveFiles(albumpath, release, tracks):
                i = 1
                while True:
                    newfolder = temp_folder + '[%i]' % i
-                    lossy_destination_path = os.path.normpath(os.path.join(headphones.CONFIG.DESTINATION_DIR, newfolder)).encode(headphones.SYS_ENCODING, 'replace')
+                    lossy_destination_path = os.path.normpath(
+                        os.path.join(headphones.CONFIG.DESTINATION_DIR, newfolder)).encode(
+                        headphones.SYS_ENCODING, 'replace')
                    if os.path.exists(lossy_destination_path):
                        i += 1
                    else:
@@ -750,7 +831,8 @@ def moveFiles(albumpath, release, tracks):
            try:
                os.makedirs(lossy_destination_path)
            except Exception as e:
-                logger.error('Could not create folder for %s. Not moving: %s' % (release['AlbumTitle'], e))
+                logger.error(
+                    'Could not create folder for %s. Not moving: %s' % (release['AlbumTitle'], e))
                return [albumpath]

    logger.info('Checking which files we need to move.....')
@@ -761,26 +843,34 @@ def moveFiles(albumpath, release, tracks):

        for file_to_move in files_to_move:

-            if any(file_to_move.lower().endswith('.' + x.lower()) for x in headphones.LOSSY_MEDIA_FORMATS):
+            if any(file_to_move.lower().endswith('.' + x.lower()) for x in
+                   headphones.LOSSY_MEDIA_FORMATS):
                helpers.smartMove(file_to_move, lossy_destination_path)

-            elif any(file_to_move.lower().endswith('.' + x.lower()) for x in headphones.LOSSLESS_MEDIA_FORMATS):
+            elif any(file_to_move.lower().endswith('.' + x.lower()) for x in
+                     headphones.LOSSLESS_MEDIA_FORMATS):
                helpers.smartMove(file_to_move, lossless_destination_path)

            # If it's a non-music file, move it to both dirs
            # TODO: Move specific-to-lossless files to the lossless dir only
            else:

-                moved_to_lossy_folder = helpers.smartMove(file_to_move, lossy_destination_path, delete=False)
-                moved_to_lossless_folder = helpers.smartMove(file_to_move, lossless_destination_path, delete=False)
+                moved_to_lossy_folder = helpers.smartMove(file_to_move, lossy_destination_path,
+                                                          delete=False)
+                moved_to_lossless_folder = helpers.smartMove(file_to_move,
+                                                             lossless_destination_path,
+                                                             delete=False)

                if moved_to_lossy_folder or moved_to_lossless_folder:
                    try:
                        os.remove(file_to_move)
                    except Exception as e:
-                        logger.error("Error deleting file '" + file_to_move.decode(headphones.SYS_ENCODING, 'replace') + "' from source directory")
+                        logger.error(
+                            "Error deleting file '" + file_to_move.decode(headphones.SYS_ENCODING,
+                                                                          'replace') + "' from source directory")
                else:
-                    logger.error("Error copying '" + file_to_move.decode(headphones.SYS_ENCODING, 'replace') + "'. Not deleting from download directory")
+                    logger.error("Error copying '" + file_to_move.decode(headphones.SYS_ENCODING,
+                                                                         'replace') + "'. Not deleting from download directory")

    elif make_lossless_folder and not make_lossy_folder:

@@ -808,10 +898,16 @@ def moveFiles(albumpath, release, tracks):

            temp_f = os.path.join(temp_f, f)

+            if headphones.CONFIG.FOLDER_PERMISSIONS_ENABLED:
                try:
-                os.chmod(os.path.normpath(temp_f).encode(headphones.SYS_ENCODING, 'replace'), int(headphones.CONFIG.FOLDER_PERMISSIONS, 8))
+                    os.chmod(os.path.normpath(temp_f).encode(headphones.SYS_ENCODING, 'replace'),
+                             int(headphones.CONFIG.FOLDER_PERMISSIONS, 8))
                except Exception as e:
-                logger.error("Error trying to change permissions on folder: %s. %s", temp_f, e)
+                    logger.error("Error trying to change permissions on folder: %s. %s",
+                                 temp_f.decode(headphones.SYS_ENCODING, 'replace'), e)
+            else:
+                logger.debug("Not changing folder permissions, since it is disabled: %s",
+                             temp_f.decode(headphones.SYS_ENCODING, 'replace'))

    # If we failed to move all the files out of the directory, this will fail too
    try:
@@ -830,7 +926,6 @@ def moveFiles(albumpath, release, tracks):


 def correctMetadata(albumid, release, downloaded_track_list):
-
    logger.info('Preparing to write metadata to tracks....')
    lossy_items = []
    lossless_items = []
@@ -840,14 +935,18 @@ def correctMetadata(albumid, release, downloaded_track_list):

        try:

-            if any(downloaded_track.lower().endswith('.' + x.lower()) for x in headphones.LOSSLESS_MEDIA_FORMATS):
+            if any(downloaded_track.lower().endswith('.' + x.lower()) for x in
+                   headphones.LOSSLESS_MEDIA_FORMATS):
                lossless_items.append(beets.library.Item.from_path(downloaded_track))
-            elif any(downloaded_track.lower().endswith('.' + x.lower()) for x in headphones.LOSSY_MEDIA_FORMATS):
+            elif any(downloaded_track.lower().endswith('.' + x.lower()) for x in
+                     headphones.LOSSY_MEDIA_FORMATS):
                lossy_items.append(beets.library.Item.from_path(downloaded_track))
            else:
-                logger.warn("Skipping: %s because it is not a mutagen friendly file format", downloaded_track.decode(headphones.SYS_ENCODING, 'replace'))
+                logger.warn("Skipping: %s because it is not a mutagen friendly file format",
+                            downloaded_track.decode(headphones.SYS_ENCODING, 'replace'))
        except Exception as e:
-            logger.error("Beets couldn't create an Item from: %s - not a media file? %s", downloaded_track.decode(headphones.SYS_ENCODING, 'replace'), str(e))
+            logger.error("Beets couldn't create an Item from: %s - not a media file? %s",
+                         downloaded_track.decode(headphones.SYS_ENCODING, 'replace'), str(e))

    for items in [lossy_items, lossless_items]:

@@ -855,19 +954,25 @@ def correctMetadata(albumid, release, downloaded_track_list):
            continue

        try:
-            cur_artist, cur_album, candidates, rec = autotag.tag_album(items, search_artist=helpers.latinToAscii(release['ArtistName']), search_album=helpers.latinToAscii(release['AlbumTitle']))
+            cur_artist, cur_album, candidates, rec = autotag.tag_album(items,
+                                                                       search_artist=helpers.latinToAscii(
+                                                                           release['ArtistName']),
+                                                                       search_album=helpers.latinToAscii(
+                                                                           release['AlbumTitle']))
        except Exception as e:
            logger.error('Error getting recommendation: %s. Not writing metadata', e)
-            return
+            return False
        if str(rec) == 'Recommendation.none':
-            logger.warn('No accurate album match found for %s, %s -  not writing metadata', release['ArtistName'], release['AlbumTitle'])
-            return
+            logger.warn('No accurate album match found for %s, %s -  not writing metadata',
+                        release['ArtistName'], release['AlbumTitle'])
+            return False

        if candidates:
            dist, info, mapping, extra_items, extra_tracks = candidates[0]
        else:
-            logger.warn('No accurate album match found for %s, %s -  not writing metadata', release['ArtistName'], release['AlbumTitle'])
-            return
+            logger.warn('No accurate album match found for %s, %s -  not writing metadata',
+                        release['ArtistName'], release['AlbumTitle'])
+            return False

        logger.info('Beets recommendation for tagging items: %s' % rec)

@@ -886,9 +991,14 @@ def correctMetadata(albumid, release, downloaded_track_list):
        for item in items:
            try:
                item.write()
-                logger.info("Successfully applied metadata to: %s", item.path.decode(headphones.SYS_ENCODING, 'replace'))
+                logger.info("Successfully applied metadata to: %s",
+                            item.path.decode(headphones.SYS_ENCODING, 'replace'))
            except Exception as e:
-                logger.warn("Error writing metadata to '%s': %s", item.path.decode(headphones.SYS_ENCODING, 'replace'), str(e))
+                logger.warn("Error writing metadata to '%s': %s",
+                            item.path.decode(headphones.SYS_ENCODING, 'replace'), str(e))
+                return False
+
+        return True


 def embedLyrics(downloaded_track_list):
@@ -904,14 +1014,18 @@ def embedLyrics(downloaded_track_list):
    for downloaded_track in downloaded_track_list:

        try:
-            if any(downloaded_track.lower().endswith('.' + x.lower()) for x in headphones.LOSSLESS_MEDIA_FORMATS):
+            if any(downloaded_track.lower().endswith('.' + x.lower()) for x in
+                   headphones.LOSSLESS_MEDIA_FORMATS):
                lossless_items.append(beets.library.Item.from_path(downloaded_track))
-            elif any(downloaded_track.lower().endswith('.' + x.lower()) for x in headphones.LOSSY_MEDIA_FORMATS):
+            elif any(downloaded_track.lower().endswith('.' + x.lower()) for x in
+                     headphones.LOSSY_MEDIA_FORMATS):
                lossy_items.append(beets.library.Item.from_path(downloaded_track))
            else:
-                logger.warn("Skipping: %s because it is not a mutagen friendly file format", downloaded_track.decode(headphones.SYS_ENCODING, 'replace'))
+                logger.warn("Skipping: %s because it is not a mutagen friendly file format",
+                            downloaded_track.decode(headphones.SYS_ENCODING, 'replace'))
        except Exception as e:
-            logger.error("Beets couldn't create an Item from: %s - not a media file? %s", downloaded_track.decode(headphones.SYS_ENCODING, 'replace'), str(e))
+            logger.error("Beets couldn't create an Item from: %s - not a media file? %s",
+                         downloaded_track.decode(headphones.SYS_ENCODING, 'replace'), str(e))

    for items in [lossy_items, lossless_items]:

@@ -951,7 +1065,8 @@ def renameFiles(albumpath, downloaded_track_list, release):
        try:
            f = MediaFile(downloaded_track)
        except:
-            logger.info("MediaFile couldn't parse: %s", downloaded_track.decode(headphones.SYS_ENCODING, 'replace'))
+            logger.info("MediaFile couldn't parse: %s",
+                        downloaded_track.decode(headphones.SYS_ENCODING, 'replace'))
            continue

        if not f.disc:
@@ -1003,9 +1118,11 @@ def renameFiles(albumpath, downloaded_track_list, release):

            ext = os.path.splitext(downloaded_track)[1]

-            new_file_name = helpers.replace_all(headphones.CONFIG.FILE_FORMAT.strip(), values).replace('/', '_') + ext
+            new_file_name = helpers.replace_all(headphones.CONFIG.FILE_FORMAT.strip(),
+                                                values).replace('/', '_') + ext

-        new_file_name = helpers.replace_illegal_chars(new_file_name).encode(headphones.SYS_ENCODING, 'replace')
+        new_file_name = helpers.replace_illegal_chars(new_file_name).encode(headphones.SYS_ENCODING,
+                                                                            'replace')

        if headphones.CONFIG.FILE_UNDERSCORES:
            new_file_name = new_file_name.replace(' ', '_')
@@ -1016,29 +1133,36 @@ def renameFiles(albumpath, downloaded_track_list, release):
        new_file = os.path.join(albumpath, new_file_name)

        if downloaded_track == new_file_name:
-            logger.debug("Renaming for: " + downloaded_track.decode(headphones.SYS_ENCODING, 'replace') + " is not neccessary")
+            logger.debug("Renaming for: " + downloaded_track.decode(headphones.SYS_ENCODING,
+                                                                    'replace') + " is not neccessary")
            continue

-        logger.debug('Renaming %s ---> %s', downloaded_track.decode(headphones.SYS_ENCODING, 'replace'), new_file_name.decode(headphones.SYS_ENCODING, 'replace'))
+        logger.debug('Renaming %s ---> %s',
+                     downloaded_track.decode(headphones.SYS_ENCODING, 'replace'),
+                     new_file_name.decode(headphones.SYS_ENCODING, 'replace'))
        try:
            os.rename(downloaded_track, new_file)
        except Exception as e:
-            logger.error('Error renaming file: %s. Error: %s', downloaded_track.decode(headphones.SYS_ENCODING, 'replace'), e)
+            logger.error('Error renaming file: %s. Error: %s',
+                         downloaded_track.decode(headphones.SYS_ENCODING, 'replace'), e)
            continue


 def updateFilePermissions(albumpaths):
-
    for folder in albumpaths:
        logger.info("Updating file permissions in %s", folder)
        for r, d, f in os.walk(folder):
            for files in f:
                full_path = os.path.join(r, files)
+                if headphones.CONFIG.FILE_PERMISSIONS_ENABLED:
                    try:
                        os.chmod(full_path, int(headphones.CONFIG.FILE_PERMISSIONS, 8))
                    except:
                        logger.error("Could not change permissions for file: %s", full_path)
                        continue
+                else:
+                    logger.debug("Not changing file permissions, since it is disabled: %s",
+                                 full_path.decode(headphones.SYS_ENCODING, 'replace'))


 def renameUnprocessedFolder(path, tag):
@@ -1060,8 +1184,7 @@ def renameUnprocessedFolder(path, tag):
            return


-def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
-
+def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None, keep_original_folder=False):
    logger.info('Force checking download folder for completed downloads')

    ignored = 0
@@ -1074,9 +1197,11 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
        if dir:
            download_dirs.append(dir.encode(headphones.SYS_ENCODING, 'replace'))
        if headphones.CONFIG.DOWNLOAD_DIR and not dir:
-            download_dirs.append(headphones.CONFIG.DOWNLOAD_DIR.encode(headphones.SYS_ENCODING, 'replace'))
+            download_dirs.append(
+                headphones.CONFIG.DOWNLOAD_DIR.encode(headphones.SYS_ENCODING, 'replace'))
        if headphones.CONFIG.DOWNLOAD_TORRENT_DIR and not dir:
-            download_dirs.append(headphones.CONFIG.DOWNLOAD_TORRENT_DIR.encode(headphones.SYS_ENCODING, 'replace'))
+            download_dirs.append(
+                headphones.CONFIG.DOWNLOAD_TORRENT_DIR.encode(headphones.SYS_ENCODING, 'replace'))

        # If DOWNLOAD_DIR and DOWNLOAD_TORRENT_DIR are the same, remove the duplicate to prevent us from trying to process the same folder twice.
        download_dirs = list(set(download_dirs))
@@ -1093,7 +1218,8 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
            # Scan for subfolders
            subfolders = os.listdir(download_dir)
            ignored += helpers.path_filter_patterns(subfolders,
-                headphones.CONFIG.IGNORED_FOLDERS, root=download_dir)
+                                                    headphones.CONFIG.IGNORED_FOLDERS,
+                                                    root=download_dir)

            for folder in subfolders:
                path_to_folder = os.path.join(download_dir, folder)
@@ -1128,15 +1254,23 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
        # underscores -> dots (this might be hit or miss since it assumes all
        # spaces/underscores came from sab replacing values
        logger.debug('Attempting to find album in the snatched table')
-        snatched = myDB.action('SELECT AlbumID, Title, Kind, Status from snatched WHERE FolderName LIKE ?', [folder_basename]).fetchone()
+        snatched = myDB.action(
+            'SELECT AlbumID, Title, Kind, Status from snatched WHERE FolderName LIKE ?',
+            [folder_basename]).fetchone()

        if snatched:
-            if headphones.CONFIG.KEEP_TORRENT_FILES and snatched['Kind'] == 'torrent' and snatched['Status'] == 'Processed':
-                logger.info('%s is a torrent folder being preserved for seeding and has already been processed. Skipping.', folder_basename)
+            if headphones.CONFIG.KEEP_TORRENT_FILES and snatched['Kind'] == 'torrent' and snatched[
+                'Status'] == 'Processed':
+                logger.info(
+                    '%s is a torrent folder being preserved for seeding and has already been processed. Skipping.',
+                    folder_basename)
                continue
            else:
-                logger.info('Found a match in the database: %s. Verifying to make sure it is the correct album', snatched['Title'])
-                verify(snatched['AlbumID'], folder, snatched['Kind'])
+                logger.info(
+                    'Found a match in the database: %s. Verifying to make sure it is the correct album',
+                    snatched['Title'])
+                verify(snatched['AlbumID'], folder, snatched['Kind'],
+                       keep_original_folder=keep_original_folder)
                continue

        # Attempt 2: strip release group id from filename
@@ -1150,13 +1284,19 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):

        if rgid:
            rgid = possible_rgid
-            release = myDB.action('SELECT ArtistName, AlbumTitle, AlbumID from albums WHERE AlbumID=?', [rgid]).fetchone()
+            release = myDB.action(
+                'SELECT ArtistName, AlbumTitle, AlbumID from albums WHERE AlbumID=?',
+                [rgid]).fetchone()
            if release:
-                logger.info('Found a match in the database: %s - %s. Verifying to make sure it is the correct album', release['ArtistName'], release['AlbumTitle'])
-                verify(release['AlbumID'], folder, forced=True)
+                logger.info(
+                    'Found a match in the database: %s - %s. Verifying to make sure it is the correct album',
+                    release['ArtistName'], release['AlbumTitle'])
+                verify(release['AlbumID'], folder, forced=True,
+                       keep_original_folder=keep_original_folder)
                continue
            else:
-                logger.info('Found a (possibly) valid Musicbrainz realse group id in album folder name.')
+                logger.info(
+                    'Found a (possibly) valid Musicbrainz release group id in album folder name.')
                verify(rgid, folder, forced=True)
                continue

@@ -1169,13 +1309,18 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
            name = album = year = None

        if name and album:
-            release = myDB.action('SELECT AlbumID, ArtistName, AlbumTitle from albums WHERE ArtistName LIKE ? and AlbumTitle LIKE ?', [name, album]).fetchone()
+            release = myDB.action(
+                'SELECT AlbumID, ArtistName, AlbumTitle from albums WHERE ArtistName LIKE ? and AlbumTitle LIKE ?',
+                [name, album]).fetchone()
            if release:
-                logger.info('Found a match in the database: %s - %s. Verifying to make sure it is the correct album', release['ArtistName'], release['AlbumTitle'])
-                verify(release['AlbumID'], folder)
+                logger.info(
+                    'Found a match in the database: %s - %s. Verifying to make sure it is the correct album',
+                    release['ArtistName'], release['AlbumTitle'])
+                verify(release['AlbumID'], folder, keep_original_folder=keep_original_folder)
                continue
            else:
-                logger.info('Querying MusicBrainz for the release group id for: %s - %s', name, album)
+                logger.info('Querying MusicBrainz for the release group id for: %s - %s', name,
+                            album)
                try:
                    rgid = mb.findAlbumID(helpers.latinToAscii(name), helpers.latinToAscii(album))
                except:
@@ -1183,7 +1328,7 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
                    rgid = None

                if rgid:
-                    verify(rgid, folder)
+                    verify(rgid, folder, keep_original_folder=keep_original_folder)
                    continue
                else:
                    logger.info('No match found on MusicBrainz for: %s - %s', name, album)
@@ -1204,13 +1349,18 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
                name = album = None

        if name and album:
-            release = myDB.action('SELECT AlbumID, ArtistName, AlbumTitle from albums WHERE ArtistName LIKE ? and AlbumTitle LIKE ?', [name, album]).fetchone()
+            release = myDB.action(
+                'SELECT AlbumID, ArtistName, AlbumTitle from albums WHERE ArtistName LIKE ? and AlbumTitle LIKE ?',
+                [name, album]).fetchone()
            if release:
-                logger.info('Found a match in the database: %s - %s. Verifying to make sure it is the correct album', release['ArtistName'], release['AlbumTitle'])
-                verify(release['AlbumID'], folder)
+                logger.info(
+                    'Found a match in the database: %s - %s. Verifying to make sure it is the correct album',
+                    release['ArtistName'], release['AlbumTitle'])
+                verify(release['AlbumID'], folder, keep_original_folder=keep_original_folder)
                continue
            else:
-                logger.info('Querying MusicBrainz for the release group id for: %s - %s', name, album)
+                logger.info('Querying MusicBrainz for the release group id for: %s - %s', name,
+                            album)
                try:
                    rgid = mb.findAlbumID(helpers.latinToAscii(name), helpers.latinToAscii(album))
                except:
@@ -1218,7 +1368,7 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
                    rgid = None

                if rgid:
-                    verify(rgid, folder)
+                    verify(rgid, folder, keep_original_folder=keep_original_folder)
                    continue
                else:
                    logger.info('No match found on MusicBrainz for: %s - %s', name, album)
@@ -1228,13 +1378,18 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
        logger.debug('Attempt to extract album name by assuming it is the folder name')

        if '-' not in folder_basename:
-            release = myDB.action('SELECT AlbumID, ArtistName, AlbumTitle from albums WHERE AlbumTitle LIKE ?', [folder_basename]).fetchone()
+            release = myDB.action(
+                'SELECT AlbumID, ArtistName, AlbumTitle from albums WHERE AlbumTitle LIKE ?',
+                [folder_basename]).fetchone()
            if release:
-                logger.info('Found a match in the database: %s - %s. Verifying to make sure it is the correct album', release['ArtistName'], release['AlbumTitle'])
-                verify(release['AlbumID'], folder)
+                logger.info(
+                    'Found a match in the database: %s - %s. Verifying to make sure it is the correct album',
+                    release['ArtistName'], release['AlbumTitle'])
+                verify(release['AlbumID'], folder, keep_original_folder=keep_original_folder)
                continue
            else:
-                logger.info('Querying MusicBrainz for the release group id for: %s', folder_basename)
+                logger.info('Querying MusicBrainz for the release group id for: %s',
+                            folder_basename)
                try:
                    rgid = mb.findAlbumID(album=helpers.latinToAscii(folder_basename))
                except:
@@ -1242,7 +1397,7 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
                    rgid = None

                if rgid:
-                    verify(rgid, folder)
+                    verify(rgid, folder, keep_original_folder=keep_original_folder)
                    continue
                else:
                    logger.info('No match found on MusicBrainz for: %s - %s', name, album)
@@ -13,16 +13,20 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

-from headphones import logger
-
 from xml.dom import minidom
-from bs4 import BeautifulSoup
+import collections

+import sys
+from bs4 import BeautifulSoup
 import requests
+from headphones import logger
 import feedparser
 import headphones
 import headphones.lock
-import collections
+
+
+# Disable SSL certificate warnings. We have our own handling
+requests.packages.urllib3.disable_warnings()

 # Dictionary with last request times, for rate limiting.
 last_requests = collections.defaultdict(int)
@@ -51,6 +55,14 @@ def request_response(url, method="get", auto_raise=True,
    # pose a security issue!
    kwargs["verify"] = bool(headphones.CONFIG.VERIFY_SSL_CERT)

+    #This fix is put in place for systems with broken SSL (like QNAP)
+    if not headphones.CONFIG.VERIFY_SSL_CERT and sys.version_info >= (2, 7, 9):
+        try:
+            import ssl
+            ssl._create_default_https_context = ssl._create_unverified_context
+        except:
+            pass
+
    # Map method to the request.XXX method. This is a simple hack, but it
    # allows requests to apply more magic per method. See lib/requests/api.py.
    request_method = getattr(requests, method.lower())
@@ -95,7 +107,8 @@ def request_response(url, method="get", auto_raise=True,
            "host is up and running.")
    except requests.Timeout:
        logger.error(
-            "Request timed out. The remote host did not respond in a timely manner.")
+            "Request timed out. The remote host did not respond in a timely "
+            "manner.")
    except requests.HTTPError as e:
        if e.response is not None:
            if e.response.status_code >= 500:
@@ -206,7 +219,8 @@ def server_message(response):
    message = None

    # First attempt is to 'read' the response as HTML
-    if "text/html" in response.headers.get("content-type"):
+    if response.headers.get("content-type") and \
+                    "text/html" in response.headers.get("content-type"):
        try:
            soup = BeautifulSoup(response.content, "html5lib")
        except Exception:
@@ -0,0 +1,226 @@
+#!/usr/bin/env python
+
+import urllib
+import time
+from urlparse import urlparse
+
+import re
+import requests as requests
+from bs4 import BeautifulSoup
+import headphones
+from headphones import logger
+
+
+class Rutracker(object):
+    def __init__(self):
+        self.session = requests.session()
+        self.timeout = 60
+        self.loggedin = False
+        self.maxsize = 0
+        self.search_referer = 'http://rutracker.org/forum/tracker.php'
+
+    def logged_in(self):
+        return self.loggedin
+
+    def still_logged_in(self, html):
+        if not html or "action=\"http://login.rutracker.org/forum/login.php\">" in html:
+            return False
+        else:
+            return True
+
+    def login(self):
+        """
+        Logs in user
+        """
+
+        loginpage = 'http://login.rutracker.org/forum/login.php'
+        post_params = {
+            'login_username': headphones.CONFIG.RUTRACKER_USER,
+            'login_password': headphones.CONFIG.RUTRACKER_PASSWORD,
+            'login': b'\xc2\xf5\xee\xe4'  # '%C2%F5%EE%E4'
+        }
+
+        logger.info("Attempting to log in to rutracker...")
+
+        try:
+            r = self.session.post(loginpage, data=post_params, timeout=self.timeout)
+            # try again
+            if 'bb_data' not in r.cookies.keys():
+                time.sleep(10)
+                r = self.session.post(loginpage, data=post_params, timeout=self.timeout)
+            if r.status_code != 200:
+                logger.error("rutracker login returned status code %s" % r.status_code)
+                self.loggedin = False
+            else:
+                if 'bb_data' in r.cookies.keys():
+                    self.loggedin = True
+                    logger.info("Successfully logged in to rutracker")
+                else:
+                    logger.error(
+                        "Could not login to rutracker, credentials maybe incorrect, site is down or too many attempts. Try again later")
+                    self.loggedin = False
+            return self.loggedin
+        except Exception as e:
+            logger.error("Unknown error logging in to rutracker: %s" % e)
+            self.loggedin = False
+            return self.loggedin
+
+    def searchurl(self, artist, album, year, format):
+        """
+        Return the search url
+        """
+
+        # Build search url
+        searchterm = ''
+        if artist != 'Various Artists':
+            searchterm = artist
+            searchterm = searchterm + ' '
+        searchterm = searchterm + album
+        searchterm = searchterm + ' '
+        searchterm = searchterm + year
+
+        if format == 'lossless':
+            format = '+lossless'
+            self.maxsize = 10000000000
+        elif format == 'lossless+mp3':
+            format = '+lossless||mp3||aac'
+            self.maxsize = 10000000000
+        else:
+            format = '+mp3||aac'
+            self.maxsize = 300000000
+
+        # sort by size, descending.
+        sort = '&o=7&s=2'
+
+        searchurl = "%s?nm=%s%s%s" % (self.search_referer, urllib.quote(searchterm), format, sort)
+
+        logger.info("Searching rutracker using term: %s", searchterm)
+
+        return searchurl
+
+    def search(self, searchurl):
+        """
+        Parse the search results and return valid torrent list
+        """
+
+        try:
+            headers = {'Referer': self.search_referer}
+            r = self.session.get(url=searchurl, headers=headers, timeout=self.timeout)
+
+            soup = BeautifulSoup(r.content, 'html5lib')
+
+            # Debug
+            # logger.debug (soup.prettify())
+
+            # Check if still logged in
+            if not self.still_logged_in(soup):
+                self.login()
+                r = self.session.get(url=searchurl, timeout=self.timeout)
+                soup = BeautifulSoup(r.content, 'html5lib')
+                if not self.still_logged_in(soup):
+                    logger.error("Error getting rutracker data")
+                    return None
+
+            # Process
+            rulist = []
+            i = soup.find('table', id='tor-tbl')
+            if not i:
+                logger.info("No valid results found from rutracker")
+                return None
+            minimumseeders = int(headphones.CONFIG.NUMBEROFSEEDERS) - 1
+
+            for item in zip(i.find_all(class_='hl-tags'), i.find_all(class_='dl-stub'),
+                            i.find_all(class_='seedmed')):
+                title = item[0].get_text()
+                url = item[1].get('href')
+                size_formatted = item[1].get_text()[:-2]
+                seeds = item[2].get_text()
+                size_parts = size_formatted.split()
+                size = float(size_parts[0])
+
+                if size_parts[1] == 'KB':
+                    size *= 1024
+                if size_parts[1] == 'MB':
+                    size *= 1024 ** 2
+                if size_parts[1] == 'GB':
+                    size *= 1024 ** 3
+                if size_parts[1] == 'TB':
+                    size *= 1024 ** 4
+
+                if size < self.maxsize and minimumseeders < int(seeds):
+                    logger.info('Found %s. Size: %s' % (title, size_formatted))
+                    # Torrent topic page
+                    torrent_id = dict([part.split('=') for part in urlparse(url)[4].split('&')])[
+                        't']
+                    topicurl = 'http://rutracker.org/forum/viewtopic.php?t=' + torrent_id
+                    rulist.append((title, size, topicurl, 'rutracker.org', 'torrent', True))
+                else:
+                    logger.info(
+                        "%s is larger than the maxsize or has too little seeders for this category, skipping. (Size: %i bytes, Seeders: %i)" % (
+                        title, size, int(seeds)))
+
+            if not rulist:
+                logger.info("No valid results found from rutracker")
+
+            return rulist
+
+        except Exception as e:
+            logger.error("An unknown error occurred in the rutracker parser: %s" % e)
+            return None
+
+    def get_torrent_data(self, url):
+        """
+        return the .torrent data
+        """
+
+        torrent_id = dict([part.split('=') for part in urlparse(url)[4].split('&')])['t']
+        downloadurl = 'http://dl.rutracker.org/forum/dl.php?t=' + torrent_id
+        cookie = {'bb_dl': torrent_id}
+        try:
+            headers = {'Referer': url}
+            r = self.session.post(url=downloadurl, cookies=cookie, headers=headers,
+                                  timeout=self.timeout)
+            return r.content
+        except Exception as e:
+            logger.error('Error getting torrent: %s', e)
+            return False
+
+    # TODO get this working in utorrent.py
+    def utorrent_add_file(self, data):
+
+        host = headphones.CONFIG.UTORRENT_HOST
+        if not host.startswith('http'):
+            host = 'http://' + host
+        if host.endswith('/'):
+            host = host[:-1]
+        if host.endswith('/gui'):
+            host = host[:-4]
+
+        base_url = host
+
+        url = base_url + '/gui/'
+        self.session.auth = (
+        headphones.CONFIG.UTORRENT_USERNAME, headphones.CONFIG.UTORRENT_PASSWORD)
+
+        try:
+            r = self.session.get(url + 'token.html')
+        except Exception as e:
+            logger.error('Error getting token: %s', e)
+            return
+
+        if r.status_code == 401:
+            logger.debug('Error reaching utorrent')
+            return
+
+        regex = re.search(r'.+>([^<]+)</div></html>', r.text)
+        if regex is None:
+            logger.debug('Error reading token')
+            return
+
+        self.session.params = {'token': regex.group(1)}
+        files = {'torrent_file': ("", data)}
+
+        try:
+            self.session.post(url, params={'action': 'add-file'}, files=files)
+        except Exception as e:
+            logger.exception('Error adding file to utorrent %s', e)
@@ -13,26 +13,24 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

-#####################################
-## Stolen from Sick-Beard's sab.py ##
-#####################################
+###################################
+# Stolen from Sick-Beard's sab.py #
+###################################

-import MultipartPostHandler
-import headphones
 import cookielib
-import httplib

+import headphones
 from headphones.common import USER_AGENT
 from headphones import logger, helpers, request


 def sab_api_call(request_type=None, params={}, **kwargs):
-
    if not headphones.CONFIG.SAB_HOST.startswith('http'):
        headphones.CONFIG.SAB_HOST = 'http://' + headphones.CONFIG.SAB_HOST

    if headphones.CONFIG.SAB_HOST.endswith('/'):
-        headphones.CONFIG.SAB_HOST = headphones.CONFIG.SAB_HOST[0:len(headphones.CONFIG.SAB_HOST) - 1]
+        headphones.CONFIG.SAB_HOST = headphones.CONFIG.SAB_HOST[
+                                     0:len(headphones.CONFIG.SAB_HOST) - 1]

    url = headphones.CONFIG.SAB_HOST + "/" + "api?"

@@ -57,8 +55,8 @@ def sab_api_call(request_type=None, params={}, **kwargs):
        logger.debug("Successfully connected to SABnzbd on url: %s" % headphones.CONFIG.SAB_HOST)
        return response

-def sendNZB(nzb):

+def sendNZB(nzb):
    params = {}
    # if it's a normal result we just pass SAB the URL
    if nzb.resultType == "nzb":
@@ -87,7 +85,8 @@ def sendNZB(nzb):
        response = sab_api_call('send_nzb', params=params)
    elif nzb.resultType == "nzbdata":
        cookies = cookielib.CookieJar()
-        response = sab_api_call('send_nzb', params=params, method="post", files=files, cookies=cookies, headers=headers)
+        response = sab_api_call('send_nzb', params=params, method="post", files=files,
+                                cookies=cookies, headers=headers)

    if not response:
        logger.info(u"No data returned from SABnzbd, NZB not sent")
@@ -102,7 +101,6 @@ def sendNZB(nzb):


 def checkConfig():
-
    params = {'mode': 'get_config',
              'section': 'misc',
              }
@@ -110,7 +108,8 @@ def checkConfig():
    config_options = sab_api_call(params=params)

    if not config_options:
-        logger.warn("Unable to read SABnzbd config file - cannot determine renaming options (might affect auto & forced post processing)")
+        logger.warn(
+            "Unable to read SABnzbd config file - cannot determine renaming options (might affect auto & forced post processing)")
        return (0, 0)

    replace_spaces = config_options['config']['misc']['replace_spaces']
@@ -1,349 +0,0 @@
-#!/usr/bin/env python
-# coding=utf-8
-
-# Headphones rutracker.org search
-# Functions called from searcher.py
-
-from bencode import bencode as bencode, bdecode
-from urlparse import urlparse
-from bs4 import BeautifulSoup
-from tempfile import mkdtemp
-from hashlib import sha1
-
-import headphones
-import requests
-import cookielib
-import urllib2
-import urllib
-import re
-import os
-
-from headphones import db, logger
-
-
-class Rutracker():
-
-    logged_in = False
-
-    # Stores a number of login attempts to prevent recursion.
-    #login_counter = 0
-
-    def __init__(self):
-
-        self.cookiejar = cookielib.CookieJar()
-        self.opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(self.cookiejar))
-        urllib2.install_opener(self.opener)
-
-    def login(self, login, password):
-        """Implements tracker login procedure."""
-
-        self.logged_in = False
-
-        if login is None or password is None:
-            return False
-
-        #self.login_counter += 1
-
-        # No recursion wanted.
-        #if self.login_counter > 1:
-        #    return False
-
-        params = urllib.urlencode({"login_username": login,
-                                   "login_password": password,
-                                   "login": "Вход"})
-
-        try:
-            self.opener.open("http://login.rutracker.org/forum/login.php", params)
-        except Exception:
-            pass
-
-        # Check if we're logged in
-        for cookie in self.cookiejar:
-            if cookie.name == 'bb_data':
-                self.logged_in = True
-
-        return self.logged_in
-
-    def searchurl(self, artist, album, year, format):
-        """
-        Return the search url
-        """
-
-        # Build search url
-        searchterm = ''
-        if artist != 'Various Artists':
-            searchterm = artist
-            searchterm = searchterm + ' '
-        searchterm = searchterm + album
-        searchterm = searchterm + ' '
-        searchterm = searchterm + year
-
-        providerurl = "http://rutracker.org/forum/tracker.php"
-
-        if format == 'lossless':
-            format = '+lossless'
-        elif format == 'lossless+mp3':
-            format = '+lossless||mp3||aac'
-        else:
-            format = '+mp3||aac'
-
-        # sort by size, descending.
-        sort = '&o=7&s=2'
-
-        searchurl = "%s?nm=%s%s%s" % (providerurl, urllib.quote(searchterm), format, sort)
-
-        return searchurl
-
-    def search(self, searchurl, maxsize, minseeders, albumid):
-        """
-        Parse the search results and return valid torrent list
-        """
-
-        titles = []
-        urls = []
-        seeders = []
-        sizes = []
-        torrentlist = []
-        rulist = []
-
-        try:
-
-            page = self.opener.open(searchurl, timeout=60)
-            soup = BeautifulSoup(page.read())
-
-            # Debug
-            #logger.debug (soup.prettify())
-
-            # Title
-            for link in soup.find_all('a', attrs={'class': 'med tLink hl-tags bold'}):
-                title = link.get_text()
-                titles.append(title)
-
-            # Download URL
-            for link in soup.find_all('a', attrs={'class': 'small tr-dl dl-stub'}):
-                url = link.get('href')
-                urls.append(url)
-
-            # Seeders
-            for link in soup.find_all('b', attrs={'class': 'seedmed'}):
-                seeder = link.get_text()
-                seeders.append(seeder)
-
-            # Size
-            for link in soup.find_all('td', attrs={'class': 'row4 small nowrap tor-size'}):
-                size = link.u.string
-                sizes.append(size)
-
-        except:
-            pass
-
-        # Combine lists
-        torrentlist = zip(titles, urls, seeders, sizes)
-
-        # return if nothing found
-        if not torrentlist:
-            return False
-
-        # don't bother checking track counts anymore, let searcher filter instead
-        # leave code in just in case
-        check_track_count = False
-
-        if check_track_count:
-
-            # get headphones track count for album, return if not found
-            myDB = db.DBConnection()
-            tracks = myDB.select('SELECT * from tracks WHERE AlbumID=?', [albumid])
-            hptrackcount = len(tracks)
-
-            if not hptrackcount:
-                logger.info('headphones track info not found, cannot compare to torrent')
-                return False
-
-            # Return all valid entries, ignored, required words now checked in searcher.py
-
-            #unwantedlist = ['promo', 'vinyl', '[lp]', 'songbook', 'tvrip', 'hdtv', 'dvd']
-
-            formatlist = ['ape', 'flac', 'ogg', 'm4a', 'aac', 'mp3', 'wav', 'aif']
-            deluxelist = ['deluxe', 'edition', 'japanese', 'exclusive']
-
-        for torrent in torrentlist:
-
-            returntitle = torrent[0].encode('utf-8')
-            url = torrent[1]
-            seeders = torrent[2]
-            size = torrent[3]
-
-            if int(size) <= maxsize and int(seeders) >= minseeders:
-
-                #Torrent topic page
-                torrent_id = dict([part.split('=') for part in urlparse(url)[4].split('&')])['t']
-                topicurl = 'http://rutracker.org/forum/viewtopic.php?t=' + torrent_id
-
-                # add to list
-                if not check_track_count:
-                    valid = True
-                else:
-
-                    # Check torrent info
-                    self.cookiejar.set_cookie(cookielib.Cookie(version=0, name='bb_dl', value=torrent_id, port=None, port_specified=False, domain='.rutracker.org', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False))
-
-                    # Debug
-                    #for cookie in self.cookiejar:
-                    #    logger.debug ('Cookie: %s' % cookie)
-
-                    try:
-                        page = self.opener.open(url)
-                        torrent = page.read()
-                        if torrent:
-                            decoded = bdecode(torrent)
-                            metainfo = decoded['info']
-                        page.close()
-                    except Exception as e:
-                        logger.error('Error getting torrent: %s' % e)
-                        return False
-
-                    # get torrent track count and check for cue
-                    trackcount = 0
-                    cuecount = 0
-
-                    if 'files' in metainfo: # multi
-                        for pathfile in metainfo['files']:
-                            path = pathfile['path']
-                            for file in path:
-                                if any(file.lower().endswith('.' + x.lower()) for x in formatlist):
-                                    trackcount += 1
-                                if '.cue' in file:
-                                    cuecount += 1
-
-                    title = returntitle.lower()
-                    logger.debug('torrent title: %s' % title)
-                    logger.debug('headphones trackcount: %s' % hptrackcount)
-                    logger.debug('rutracker trackcount: %s' % trackcount)
-
-                    # If torrent track count less than headphones track count, and there's a cue, then attempt to get track count from log(s)
-                    # This is for the case where we have a single .flac/.wav which can be split by cue
-                    # Not great, but shouldn't be doing this too often
-                    totallogcount = 0
-                    if trackcount < hptrackcount and cuecount > 0 and cuecount < hptrackcount:
-                        page = self.opener.open(topicurl, timeout=60)
-                        soup = BeautifulSoup(page.read())
-                        findtoc = soup.find_all(text='TOC of the extracted CD')
-                        if not findtoc:
-                            findtoc = soup.find_all(text='TOC извлечённого CD')
-                        for toc in findtoc:
-                            logcount = 0
-                            for toccontent in toc.find_all_next(text=True):
-                                cut_string = toccontent.split('|')
-                                new_string = cut_string[0].lstrip().rstrip()
-                                if new_string == '1' or new_string == '01':
-                                    logcount = 1
-                                elif logcount > 0:
-                                    if new_string.isdigit():
-                                        logcount += 1
-                                    else:
-                                        break
-                            totallogcount = totallogcount + logcount
-
-                    if totallogcount > 0:
-                        trackcount = totallogcount
-                        logger.debug('rutracker logtrackcount: %s' % totallogcount)
-
-                    # If torrent track count = hp track count then return torrent,
-                    # if greater, check for deluxe/special/foreign editions
-                    # if less, then allow if it's a single track with a cue
-                    valid = False
-
-                    if trackcount == hptrackcount:
-                        valid = True
-                    elif trackcount > hptrackcount:
-                        if any(deluxe in title for deluxe in deluxelist):
-                            valid = True
-
-                # Add to list
-                if valid:
-                    rulist.append((returntitle, size, topicurl))
-                else:
-                    if topicurl:
-                        logger.info(u'<a href="%s">Torrent</a> found with %s tracks but the selected headphones release has %s tracks, skipping for rutracker.org' % (topicurl, trackcount, hptrackcount))
-            else:
-                logger.info('%s is larger than the maxsize or has too little seeders for this category, skipping. (Size: %i bytes, Seeders: %i)' % (returntitle, int(size), int(seeders)))
-
-        return rulist
-
-    def get_torrent(self, url, savelocation=None):
-
-        torrent_id = dict([part.split('=') for part in urlparse(url)[4].split('&')])['t']
-        self.cookiejar.set_cookie(cookielib.Cookie(version=0, name='bb_dl', value=torrent_id, port=None, port_specified=False, domain='.rutracker.org', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False))
-        downloadurl = 'http://dl.rutracker.org/forum/dl.php?t=' + torrent_id
-        torrent_name = torrent_id + '.torrent'
-
-        try:
-            prev = os.umask(headphones.UMASK)
-            page = self.opener.open(downloadurl)
-            torrent = page.read()
-            decoded = bdecode(torrent)
-            metainfo = decoded['info']
-            tor_hash = sha1(bencode(metainfo)).hexdigest()
-            if savelocation:
-                download_path = os.path.join(savelocation, torrent_name)
-            else:
-                tempdir = mkdtemp(suffix='_rutracker_torrents')
-                download_path = os.path.join(tempdir, torrent_name)
-
-            with open(download_path, 'wb') as f:
-                f.write(torrent)
-            os.umask(prev)
-
-            # Add file to utorrent
-            if headphones.CONFIG.TORRENT_DOWNLOADER == 2:
-                self.utorrent_add_file(download_path)
-
-        except Exception as e:
-            logger.error('Error getting torrent: %s', e)
-            return False
-
-        return download_path, tor_hash
-
-    #TODO get this working in utorrent.py
-    def utorrent_add_file(self, filename):
-
-        host = headphones.CONFIG.UTORRENT_HOST
-        if not host.startswith('http'):
-            host = 'http://' + host
-        if host.endswith('/'):
-            host = host[:-1]
-        if host.endswith('/gui'):
-            host = host[:-4]
-
-        base_url = host
-        username = headphones.CONFIG.UTORRENT_USERNAME
-        password = headphones.CONFIG.UTORRENT_PASSWORD
-
-        session = requests.Session()
-        url = base_url + '/gui/'
-        session.auth = (username, password)
-
-        try:
-            r = session.get(url + 'token.html')
-        except Exception:
-            logger.exception('Error getting token')
-            return
-
-        if r.status_code == '401':
-            logger.debug('Error reaching utorrent')
-            return
-
-        regex = re.search(r'.+>([^<]+)</div></html>', r.text)
-        if regex is None:
-            logger.debug('Error reading token')
-            return
-
-        session.params = {'token': regex.group(1)}
-
-        with open(filename, 'rb') as f:
-            try:
-                session.post(url, params={'action': 'add-file'},
-                    files={'torrent_file': f})
-            except Exception:
-                logger.exception('Error adding file to utorrent')
-                return
@@ -13,13 +13,14 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

-from headphones import db, utorrent, transmission, logger
-
 import threading
+
+from headphones import db, utorrent, transmission, logger
 import headphones

 postprocessor_lock = threading.Lock()

+
 def checkTorrentFinished():
    """
    Remove Torrent + data if Post Processed and finished Seeding
@@ -41,6 +42,7 @@ def checkTorrentFinished():
                torrent_removed = utorrent.removeTorrent(hash, True)

            if torrent_removed:
-                myDB.action('DELETE from snatched WHERE status = "Seed_Processed" and AlbumID=?', [albumid])
+                myDB.action('DELETE from snatched WHERE status = "Seed_Processed" and AlbumID=?',
+                            [albumid])

    logger.info("Checking finished torrents completed")
@@ -13,14 +13,15 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

-from headphones import logger, request
-
 import time
 import json
 import base64
 import urlparse
+
+from headphones import logger, request
 import headphones

+
 # This is just a simple script to send torrents to transmission. The
 # intention is to turn this into a class where we can check the state
 # of the download, set the download dir, etc.
@@ -28,10 +29,13 @@ import headphones
 #       Store torrent id so we can check up on it


-def addTorrent(link):
+def addTorrent(link, data=None):
    method = 'torrent-add'

-    if link.endswith('.torrent'):
+    if link.endswith('.torrent') and not link.startswith('http') or data:
+        if data:
+            metainfo = str(base64.b64encode(data))
+        else:
            with open(link, 'rb') as f:
                metainfo = str(base64.b64encode(f.read()))
        arguments = {'metainfo': metainfo, 'download-dir': headphones.CONFIG.DOWNLOAD_TORRENT_DIR}
@@ -93,7 +97,6 @@ def setSeedRatio(torrentid, ratio):


 def removeTorrent(torrentid, remove_data=False):
-
    method = 'torrent-get'
    arguments = {'ids': torrentid, 'fields': ['isFinished', 'name']}

@@ -115,7 +118,8 @@ def removeTorrent(torrentid, remove_data=False):
            response = torrentAction(method, arguments)
            return True
        else:
-            logger.info('%s has not finished seeding yet, torrent will not be removed, will try again on next run' % name)
+            logger.info(
+                '%s has not finished seeding yet, torrent will not be removed, will try again on next run' % name)
    except:
        return False

@@ -123,7 +127,6 @@ def removeTorrent(torrentid, remove_data=False):


 def torrentAction(method, arguments):
-
    host = headphones.CONFIG.TRANSMISSION_HOST
    username = headphones.CONFIG.TRANSMISSION_USERNAME
    password = headphones.CONFIG.TRANSMISSION_PASSWORD
@@ -17,10 +17,10 @@ from headphones import logger, db, importer


 def dbUpdate(forcefull=False):
-
    myDB = db.DBConnection()

-    active_artists = myDB.select('SELECT ArtistID, ArtistName from artists WHERE Status="Active" or Status="Loading" order by LastUpdated ASC')
+    active_artists = myDB.select(
+        'SELECT ArtistID, ArtistName from artists WHERE Status="Active" or Status="Loading" order by LastUpdated ASC')
    logger.info('Starting update for %i active artists', len(active_artists))

    for artist in active_artists:
@@ -14,22 +14,20 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

 import urllib
+import json
+import time
+from collections import namedtuple
 import urllib2
 import urlparse
 import cookielib
-import json
+
 import re
 import os
-import time
-
 import headphones
-
 from headphones import logger
-from collections import namedtuple


 class utorrentclient(object):
-
    TOKEN_REGEX = "<div id='token' style='display:none;'>([^<>]+)</div>"
    UTSetting = namedtuple("UTSetting", ["name", "int", "str", "access"])

@@ -73,6 +71,7 @@ class utorrentclient(object):
        except urllib2.HTTPError as err:
            logger.debug('URL: ' + str(url))
            logger.debug('Error getting Token. uTorrent responded with error: ' + str(err))
+            return
        match = re.search(utorrentclient.TOKEN_REGEX, response.read())
        return match.group(1)

@@ -147,6 +146,10 @@ class utorrentclient(object):
        return self._action(params)

    def _action(self, params, body=None, content_type=None):
+
+        if not self.token:
+            return
+
        url = self.base_url + '/gui/' + '?token=' + self.token + '&' + urllib.urlencode(params)
        request = urllib2.Request(url)

@@ -176,13 +179,15 @@ def removeTorrent(hash, remove_data=False):
    status, torrentList = uTorrentClient.list()
    torrents = torrentList['torrents']
    for torrent in torrents:
-        if torrent[0].lower() == hash:
+        if torrent[0].upper() == hash.upper():
            if torrent[21] == 'Finished':
                logger.info('%s has finished seeding, removing torrent and data' % torrent[2])
                uTorrentClient.remove(hash, remove_data)
                return True
            else:
-                logger.info('%s has not finished seeding yet, torrent will not be removed, will try again on next run' % torrent[2])
+                logger.info(
+                    '%s has not finished seeding yet, torrent will not be removed, will try again on next run' %
+                    torrent[2])
                return False
    return False

@@ -198,7 +203,6 @@ def setSeedRatio(hash, ratio):


 def dirTorrent(hash, cacheid=None, return_name=None):
-
    uTorrentClient = utorrentclient()

    if not cacheid:
@@ -215,7 +219,7 @@ def dirTorrent(hash, cacheid=None, return_name=None):
    cacheid = torrentList['torrentc']

    for torrent in torrents:
-        if torrent[0].upper() == hash:
+        if torrent[0].upper() == hash.upper():
            if not return_name:
                return torrent[26], cacheid
            else:
@@ -224,18 +228,21 @@ def dirTorrent(hash, cacheid=None, return_name=None):
    return None, None


-def addTorrent(link, hash):
+def addTorrent(link):
    uTorrentClient = utorrentclient()
+    uTorrentClient.add_url(link)
+
+
+def getFolder(hash):

    # Get Active Directory from settings
    active_dir, completed_dir = getSettingsDirectories()

    if not active_dir:
-        logger.error('Could not get "Put new downloads in:" directory from uTorrent settings, please ensure it is set')
+        logger.error(
+            'Could not get "Put new downloads in:" directory from uTorrent settings, please ensure it is set')
        return None

-    uTorrentClient.add_url(link)
-
    # Get Torrent Folder Name
    torrent_folder, cacheid = dirTorrent(hash)

@@ -249,10 +256,8 @@ def addTorrent(link, hash):

    if torrent_folder == active_dir or not torrent_folder:
        torrent_folder, cacheid = dirTorrent(hash, cacheid, return_name=True)
-        labelTorrent(hash)
        return torrent_folder
    else:
-        labelTorrent(hash)
        if headphones.SYS_PLATFORM != "win32":
            torrent_folder = torrent_folder.replace('\\', '/')
        return os.path.basename(os.path.normpath(torrent_folder))
@@ -13,18 +13,17 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

-import re
-import os
 import tarfile
 import platform
-import headphones
 import subprocess

+import re
+import os
+import headphones
 from headphones import logger, version, request


 def runGit(args):
-
    if headphones.CONFIG.GIT_PATH:
        git_locations = ['"' + headphones.CONFIG.GIT_PATH + '"']
    else:
@@ -40,7 +39,8 @@ def runGit(args):

        try:
            logger.debug('Trying to execute: "' + cmd + '" with shell in ' + headphones.PROG_DIR)
-            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True, cwd=headphones.PROG_DIR)
+            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True,
+                                 cwd=headphones.PROG_DIR)
            output, err = p.communicate()
            output = output.strip()

@@ -62,7 +62,6 @@ def runGit(args):


 def getVersion():
-
    if version.HEADPHONES_VERSION.startswith('win32build'):
        headphones.INSTALL_TYPE = 'win'

@@ -92,7 +91,8 @@ def getVersion():
            branch_name = branch_name

            if not branch_name and headphones.CONFIG.GIT_BRANCH:
-                logger.error('Could not retrieve branch name from git. Falling back to %s' % headphones.CONFIG.GIT_BRANCH)
+                logger.error(
+                    'Could not retrieve branch name from git. Falling back to %s' % headphones.CONFIG.GIT_BRANCH)
                branch_name = headphones.CONFIG.GIT_BRANCH
            if not branch_name:
                logger.error('Could not retrieve branch name from git. Defaulting to master')
@@ -123,11 +123,13 @@ def checkGithub():

    # Get the latest version available from github
    logger.info('Retrieving latest version information from GitHub')
-    url = 'https://api.github.com/repos/%s/headphones/commits/%s' % (headphones.CONFIG.GIT_USER, headphones.CONFIG.GIT_BRANCH)
+    url = 'https://api.github.com/repos/%s/headphones/commits/%s' % (
+    headphones.CONFIG.GIT_USER, headphones.CONFIG.GIT_BRANCH)
    version = request.request_json(url, timeout=20, validator=lambda x: type(x) == dict)

    if version is None:
-        logger.warn('Could not get the latest version from GitHub. Are you running a local development version?')
+        logger.warn(
+            'Could not get the latest version from GitHub. Are you running a local development version?')
        return headphones.CURRENT_VERSION

    headphones.LATEST_VERSION = version['sha']
@@ -135,7 +137,8 @@ def checkGithub():

    # See how many commits behind we are
    if not headphones.CURRENT_VERSION:
-        logger.info('You are running an unknown version of Headphones. Run the updater to identify your version')
+        logger.info(
+            'You are running an unknown version of Headphones. Run the updater to identify your version')
        return headphones.LATEST_VERSION

    if headphones.LATEST_VERSION == headphones.CURRENT_VERSION:
@@ -143,8 +146,10 @@ def checkGithub():
        return headphones.LATEST_VERSION

    logger.info('Comparing currently installed version with latest GitHub version')
-    url = 'https://api.github.com/repos/%s/headphones/compare/%s...%s' % (headphones.CONFIG.GIT_USER, headphones.LATEST_VERSION, headphones.CURRENT_VERSION)
-    commits = request.request_json(url, timeout=20, whitelist_status_code=404, validator=lambda x: type(x) == dict)
+    url = 'https://api.github.com/repos/%s/headphones/compare/%s...%s' % (
+    headphones.CONFIG.GIT_USER, headphones.LATEST_VERSION, headphones.CURRENT_VERSION)
+    commits = request.request_json(url, timeout=20, whitelist_status_code=404,
+                                   validator=lambda x: type(x) == dict)

    if commits is None:
        logger.warn('Could not get commits behind from GitHub.')
@@ -158,7 +163,8 @@ def checkGithub():
        headphones.COMMITS_BEHIND = 0

    if headphones.COMMITS_BEHIND > 0:
-        logger.info('New version is available. You are %s commits behind' % headphones.COMMITS_BEHIND)
+        logger.info(
+            'New version is available. You are %s commits behind' % headphones.COMMITS_BEHIND)
    elif headphones.COMMITS_BEHIND == 0:
        logger.info('Headphones is up to date')

@@ -185,7 +191,8 @@ def update():
                logger.info('Output: ' + str(output))

    else:
-        tar_download_url = 'https://github.com/%s/headphones/tarball/%s' % (headphones.CONFIG.GIT_USER, headphones.CONFIG.GIT_BRANCH)
+        tar_download_url = 'https://github.com/%s/headphones/tarball/%s' % (
+        headphones.CONFIG.GIT_USER, headphones.CONFIG.GIT_BRANCH)
        update_dir = os.path.join(headphones.PROG_DIR, 'update')
        version_path = os.path.join(headphones.PROG_DIR, 'version.txt')

@@ -214,7 +221,8 @@ def update():
        os.remove(tar_download_path)

        # Find update dir name
-        update_dir_contents = [x for x in os.listdir(update_dir) if os.path.isdir(os.path.join(update_dir, x))]
+        update_dir_contents = [x for x in os.listdir(update_dir) if
+                               os.path.isdir(os.path.join(update_dir, x))]
        if len(update_dir_contents) != 1:
            logger.error("Invalid update data, update failed: " + str(update_dir_contents))
            return
@@ -15,25 +15,25 @@

 # NZBGet support added by CurlyMo <curlymoo1@gmail.com> as a part of XBian - XBMC on the Raspberry Pi

-from headphones import logger, searcher, db, importer, mb, lastfm, librarysync, helpers, notifiers
-from headphones.helpers import checked, radio, today, cleanName
-
-from mako.lookup import TemplateLookup
-from mako import exceptions
-
 from operator import itemgetter
-
-import headphones
 import threading
-import cherrypy
-import urllib2
 import hashlib
 import random
 import urllib
 import json
 import time
+import cgi
 import sys
+import urllib2
+
 import os
+import re
+from headphones import logger, searcher, db, importer, mb, lastfm, librarysync, helpers, notifiers
+from headphones.helpers import checked, radio, today, cleanName
+from mako.lookup import TemplateLookup
+from mako import exceptions
+import headphones
+import cherrypy

 try:
    # pylint:disable=E0611
@@ -46,7 +46,6 @@ except ImportError:


 def serve_template(templatename, **kwargs):
-
    interface_dir = os.path.join(str(headphones.PROG_DIR), 'data/interfaces/')
    template_dir = os.path.join(str(interface_dir), headphones.CONFIG.INTERFACE)

@@ -60,7 +59,6 @@ def serve_template(templatename, **kwargs):


 class WebInterface(object):
-
    @cherrypy.expose
    def index(self):
        raise cherrypy.HTTPRedirect("home")
@@ -88,7 +86,8 @@ class WebInterface(object):
        if not artist:
            raise cherrypy.HTTPRedirect("home")

-        albums = myDB.select('SELECT * from albums WHERE ArtistID=? order by ReleaseDate DESC', [ArtistID])
+        albums = myDB.select('SELECT * from albums WHERE ArtistID=? order by ReleaseDate DESC',
+                             [ArtistID])

        # Serve the extras up as a dict to make things easier for new templates (append new extras to the end)
        extras_list = headphones.POSSIBLE_EXTRAS
@@ -107,7 +106,8 @@ class WebInterface(object):
                extras_dict[extra] = ""
            i += 1

-        return serve_template(templatename="artist.html", title=artist['ArtistName'], artist=artist, albums=albums, extras=extras_dict)
+        return serve_template(templatename="artist.html", title=artist['ArtistName'], artist=artist,
+                              albums=albums, extras=extras_dict)

    @cherrypy.expose
    def albumPage(self, AlbumID):
@@ -126,8 +126,10 @@ class WebInterface(object):
        if not album:
            raise cherrypy.HTTPRedirect("home")

-        tracks = myDB.select('SELECT * from tracks WHERE AlbumID=? ORDER BY CAST(TrackNumber AS INTEGER)', [AlbumID])
-        description = myDB.action('SELECT * from descriptions WHERE ReleaseGroupID=?', [AlbumID]).fetchone()
+        tracks = myDB.select(
+            'SELECT * from tracks WHERE AlbumID=? ORDER BY CAST(TrackNumber AS INTEGER)', [AlbumID])
+        description = myDB.action('SELECT * from descriptions WHERE ReleaseGroupID=?',
+                                  [AlbumID]).fetchone()

        if not album['ArtistName']:
            title = ' - '
@@ -137,7 +139,8 @@ class WebInterface(object):
            title = title + ""
        else:
            title = title + album['AlbumTitle']
-        return serve_template(templatename="album.html", title=title, album=album, tracks=tracks, description=description)
+        return serve_template(templatename="album.html", title=title, album=album, tracks=tracks,
+                              description=description)

    @cherrypy.expose
    def search(self, name, type):
@@ -149,7 +152,9 @@ class WebInterface(object):
            searchresults = mb.findRelease(name, limit=100)
        else:
            searchresults = mb.findSeries(name, limit=100)
-        return serve_template(templatename="searchresults.html", title='Search Results for: "' + name + '"', searchresults=searchresults, name=name, type=type)
+        return serve_template(templatename="searchresults.html",
+                              title='Search Results for: "' + cgi.escape(name) + '"',
+                              searchresults=searchresults, name=cgi.escape(name), type=type)

    @cherrypy.expose
    def addArtist(self, artistid):
@@ -160,7 +165,8 @@ class WebInterface(object):

    @cherrypy.expose
    def addSeries(self, seriesid):
-        thread = threading.Thread(target=importer.addArtisttoDB, args=[seriesid, False, False, "series"])
+        thread = threading.Thread(target=importer.addArtisttoDB,
+                                  args=[seriesid, False, False, "series"])
        thread.start()
        thread.join(1)
        raise cherrypy.HTTPRedirect("artistPage?ArtistID=%s" % seriesid)
@@ -198,12 +204,18 @@ class WebInterface(object):
        controlValueDict = {'ArtistID': ArtistID}
        newValueDict = {'IncludeExtras': 0}
        myDB.upsert("artists", newValueDict, controlValueDict)
-        extraalbums = myDB.select('SELECT AlbumID from albums WHERE ArtistID=? AND Status="Skipped" AND Type!="Album"', [ArtistID])
+        extraalbums = myDB.select(
+            'SELECT AlbumID from albums WHERE ArtistID=? AND Status="Skipped" AND Type!="Album"',
+            [ArtistID])
        for album in extraalbums:
-            myDB.action('DELETE from tracks WHERE ArtistID=? AND AlbumID=?', [ArtistID, album['AlbumID']])
-            myDB.action('DELETE from albums WHERE ArtistID=? AND AlbumID=?', [ArtistID, album['AlbumID']])
-            myDB.action('DELETE from allalbums WHERE ArtistID=? AND AlbumID=?', [ArtistID, album['AlbumID']])
-            myDB.action('DELETE from alltracks WHERE ArtistID=? AND AlbumID=?', [ArtistID, album['AlbumID']])
+            myDB.action('DELETE from tracks WHERE ArtistID=? AND AlbumID=?',
+                        [ArtistID, album['AlbumID']])
+            myDB.action('DELETE from albums WHERE ArtistID=? AND AlbumID=?',
+                        [ArtistID, album['AlbumID']])
+            myDB.action('DELETE from allalbums WHERE ArtistID=? AND AlbumID=?',
+                        [ArtistID, album['AlbumID']])
+            myDB.action('DELETE from alltracks WHERE ArtistID=? AND AlbumID=?',
+                        [ArtistID, album['AlbumID']])
            myDB.action('DELETE from releases WHERE ReleaseGroupID=?', [album['AlbumID']])
            from headphones import cache
            c = cache.Cache()
@@ -230,17 +242,19 @@ class WebInterface(object):
        raise cherrypy.HTTPRedirect("artistPage?ArtistID=%s" % ArtistID)

    def removeArtist(self, ArtistID):
-        logger.info(u"Deleting all traces of artist: " + ArtistID)
        myDB = db.DBConnection()
        namecheck = myDB.select('SELECT ArtistName from artists where ArtistID=?', [ArtistID])
        for name in namecheck:
            artistname = name['ArtistName']
+        logger.info(u"Deleting all traces of artist: " + artistname)
        myDB.action('DELETE from artists WHERE ArtistID=?', [ArtistID])

        from headphones import cache
        c = cache.Cache()

-        rgids = myDB.select('SELECT AlbumID FROM albums WHERE ArtistID=? UNION SELECT AlbumID FROM allalbums WHERE ArtistID=?', [ArtistID, ArtistID])
+        rgids = myDB.select(
+            'SELECT AlbumID FROM albums WHERE ArtistID=? UNION SELECT AlbumID FROM allalbums WHERE ArtistID=?',
+            [ArtistID, ArtistID])
        for rgid in rgids:
            albumid = rgid['AlbumID']
            myDB.action('DELETE from releases WHERE ReleaseGroupID=?', [albumid])
@@ -265,21 +279,84 @@ class WebInterface(object):

    @cherrypy.expose
    def scanArtist(self, ArtistID):
-        logger.info(u"Scanning artist: %s", ArtistID)
+
        myDB = db.DBConnection()
-        artistname = myDB.select('SELECT DISTINCT ArtistName FROM artists WHERE ArtistID=?', [ArtistID])
-        artistfolder = os.path.join(headphones.CONFIG.DESTINATION_DIR, artistname[0][0])
-        try:
-            threading.Thread(target=librarysync.libraryScan(dir=artistfolder)).start()
-        except Exception as e:
-            logger.error('Unable to complete the scan: %s', e)
+        artist_name = \
+        myDB.select('SELECT DISTINCT ArtistName FROM artists WHERE ArtistID=?', [ArtistID])[0][0]
+
+        logger.info(u"Scanning artist: %s", artist_name)
+
+        full_folder_format = headphones.CONFIG.FOLDER_FORMAT
+        folder_format = re.findall(r'(.*?[Aa]rtist?)\.*', full_folder_format)[0]
+
+        acceptable_formats = ["$artist", "$sortartist", "$first/$artist", "$first/$sortartist"]
+
+        if not folder_format.lower() in acceptable_formats:
+            logger.info(
+                "Can't determine the artist folder from the configured folder_format. Not scanning")
+            return
+
+        # Format the folder to match the settings
+        artist = artist_name.replace('/', '_')
+
+        if headphones.CONFIG.FILE_UNDERSCORES:
+            artist = artist.replace(' ', '_')
+
+        if artist.startswith('The '):
+            sortname = artist[4:] + ", The"
+        else:
+            sortname = artist
+
+        if sortname[0].isdigit():
+            firstchar = u'0-9'
+        else:
+            firstchar = sortname[0]
+
+        values = {'$Artist': artist,
+                  '$SortArtist': sortname,
+                  '$First': firstchar.upper(),
+                  '$artist': artist.lower(),
+                  '$sortartist': sortname.lower(),
+                  '$first': firstchar.lower(),
+                  }
+
+        folder = helpers.replace_all(folder_format.strip(), values, normalize=True)
+
+        folder = helpers.replace_illegal_chars(folder, type="folder")
+        folder = folder.replace('./', '_/').replace('/.', '/_')
+
+        if folder.endswith('.'):
+            folder = folder[:-1] + '_'
+
+        if folder.startswith('.'):
+            folder = '_' + folder[1:]
+
+        dirs = []
+        if headphones.CONFIG.MUSIC_DIR:
+            dirs.append(headphones.CONFIG.MUSIC_DIR)
+        if headphones.CONFIG.DESTINATION_DIR:
+            dirs.append(headphones.CONFIG.DESTINATION_DIR)
+        if headphones.CONFIG.LOSSLESS_DESTINATION_DIR:
+            dirs.append(headphones.CONFIG.LOSSLESS_DESTINATION_DIR)
+
+        dirs = set(dirs)
+
+        for dir in dirs:
+            artistfolder = os.path.join(dir, folder)
+            if not os.path.isdir(artistfolder):
+                logger.debug("Cannot find directory: " + artistfolder)
+                continue
+            threading.Thread(target=librarysync.libraryScan,
+                             kwargs={"dir": artistfolder, "artistScan": True, "ArtistID": ArtistID,
+                                     "ArtistName": artist_name}).start()
        raise cherrypy.HTTPRedirect("artistPage?ArtistID=%s" % ArtistID)

    @cherrypy.expose
    def deleteEmptyArtists(self):
        logger.info(u"Deleting all empty artists")
        myDB = db.DBConnection()
-        emptyArtistIDs = [row['ArtistID'] for row in myDB.select("SELECT ArtistID FROM artists WHERE LatestAlbum IS NULL")]
+        emptyArtistIDs = [row['ArtistID'] for row in
+                          myDB.select("SELECT ArtistID FROM artists WHERE LatestAlbum IS NULL")]
        for ArtistID in emptyArtistIDs:
            self.removeArtist(ArtistID)

@@ -311,8 +388,11 @@ class WebInterface(object):
            if ArtistID:
                ArtistIDT = ArtistID
            else:
-                ArtistIDT = myDB.action('SELECT ArtistID FROM albums WHERE AlbumID=?', [mbid]).fetchone()[0]
-            myDB.action('UPDATE artists SET TotalTracks=(SELECT COUNT(*) FROM tracks WHERE ArtistID = ? AND AlbumTitle IN (SELECT AlbumTitle FROM albums WHERE Status != "Ignored")) WHERE ArtistID = ?', [ArtistIDT, ArtistIDT])
+                ArtistIDT = \
+                myDB.action('SELECT ArtistID FROM albums WHERE AlbumID=?', [mbid]).fetchone()[0]
+            myDB.action(
+                'UPDATE artists SET TotalTracks=(SELECT COUNT(*) FROM tracks WHERE ArtistID = ? AND AlbumTitle IN (SELECT AlbumTitle FROM albums WHERE Status != "Ignored")) WHERE ArtistID = ?',
+                [ArtistIDT, ArtistIDT])
        if ArtistID:
            raise cherrypy.HTTPRedirect("artistPage?ArtistID=%s" % ArtistID)
        else:
@@ -325,8 +405,10 @@ class WebInterface(object):
        if action == "ignore":
            myDB = db.DBConnection()
            for artist in args:
-                myDB.action('DELETE FROM newartists WHERE ArtistName=?', [artist.decode(headphones.SYS_ENCODING, 'replace')])
-                myDB.action('UPDATE have SET Matched="Ignored" WHERE ArtistName=?', [artist.decode(headphones.SYS_ENCODING, 'replace')])
+                myDB.action('DELETE FROM newartists WHERE ArtistName=?',
+                            [artist.decode(headphones.SYS_ENCODING, 'replace')])
+                myDB.action('UPDATE have SET Matched="Ignored" WHERE ArtistName=?',
+                            [artist.decode(headphones.SYS_ENCODING, 'replace')])
                logger.info("Artist %s removed from new artist list and set to ignored" % artist)
        raise cherrypy.HTTPRedirect("home")

@@ -402,10 +484,12 @@ class WebInterface(object):
        myDB = db.DBConnection()

        myDB.action('DELETE from have WHERE Matched=?', [AlbumID])
-        album = myDB.action('SELECT ArtistID, ArtistName, AlbumTitle from albums where AlbumID=?', [AlbumID]).fetchone()
+        album = myDB.action('SELECT ArtistID, ArtistName, AlbumTitle from albums where AlbumID=?',
+                            [AlbumID]).fetchone()
        if album:
            ArtistID = album['ArtistID']
-            myDB.action('DELETE from have WHERE ArtistName=? AND AlbumTitle=?', [album['ArtistName'], album['AlbumTitle']])
+            myDB.action('DELETE from have WHERE ArtistName=? AND AlbumTitle=?',
+                        [album['ArtistName'], album['AlbumTitle']])

        myDB.action('DELETE from albums WHERE AlbumID=?', [AlbumID])
        myDB.action('DELETE from tracks WHERE AlbumID=?', [AlbumID])
@@ -445,9 +529,11 @@ class WebInterface(object):
    @cherrypy.expose
    def upcoming(self):
        myDB = db.DBConnection()
-        upcoming = myDB.select("SELECT * from albums WHERE ReleaseDate > date('now') order by ReleaseDate ASC")
+        upcoming = myDB.select(
+            "SELECT * from albums WHERE ReleaseDate > date('now') order by ReleaseDate ASC")
        wanted = myDB.select("SELECT * from albums WHERE Status='Wanted'")
-        return serve_template(templatename="upcoming.html", title="Upcoming", upcoming=upcoming, wanted=wanted)
+        return serve_template(templatename="upcoming.html", title="Upcoming", upcoming=upcoming,
+                              wanted=wanted)

    @cherrypy.expose
    def manage(self):
@@ -459,7 +545,8 @@ class WebInterface(object):
    def manageArtists(self):
        myDB = db.DBConnection()
        artists = myDB.select('SELECT * from artists order by ArtistSortName COLLATE NOCASE')
-        return serve_template(templatename="manageartists.html", title="Manage Artists", artists=artists)
+        return serve_template(templatename="manageartists.html", title="Manage Artists",
+                              artists=artists)

    @cherrypy.expose
    def manageAlbums(self, Status=None):
@@ -470,87 +557,115 @@ class WebInterface(object):
            albums = myDB.select('SELECT * from albums WHERE Status=?', [Status])
        else:
            albums = myDB.select('SELECT * from albums')
-        return serve_template(templatename="managealbums.html", title="Manage Albums", albums=albums)
+        return serve_template(templatename="managealbums.html", title="Manage Albums",
+                              albums=albums)

    @cherrypy.expose
    def manageNew(self):
        myDB = db.DBConnection()
        newartists = myDB.select('SELECT * from newartists')
-        return serve_template(templatename="managenew.html", title="Manage New Artists", newartists=newartists)
+        return serve_template(templatename="managenew.html", title="Manage New Artists",
+                              newartists=newartists)

    @cherrypy.expose
    def manageUnmatched(self):
        myDB = db.DBConnection()
        have_album_dictionary = []
        headphones_album_dictionary = []
-        have_albums = myDB.select('SELECT ArtistName, AlbumTitle, TrackTitle, CleanName from have WHERE Matched = "Failed" GROUP BY AlbumTitle ORDER BY ArtistName')
+        have_albums = myDB.select(
+            'SELECT ArtistName, AlbumTitle, TrackTitle, CleanName from have WHERE Matched = "Failed" GROUP BY AlbumTitle ORDER BY ArtistName')
        for albums in have_albums:
            # Have to skip over manually matched tracks
            if albums['ArtistName'] and albums['AlbumTitle'] and albums['TrackTitle']:
-                original_clean = helpers.cleanName(albums['ArtistName'] + " " + albums['AlbumTitle'] + " " + albums['TrackTitle'])
+                original_clean = helpers.cleanName(
+                    albums['ArtistName'] + " " + albums['AlbumTitle'] + " " + albums['TrackTitle'])
                # else:
                #     original_clean = None
                if original_clean == albums['CleanName']:
-                    have_dict = {'ArtistName': albums['ArtistName'], 'AlbumTitle': albums['AlbumTitle']}
+                    have_dict = {'ArtistName': albums['ArtistName'],
+                                 'AlbumTitle': albums['AlbumTitle']}
                    have_album_dictionary.append(have_dict)
-        headphones_albums = myDB.select('SELECT ArtistName, AlbumTitle from albums ORDER BY ArtistName')
+        headphones_albums = myDB.select(
+            'SELECT ArtistName, AlbumTitle from albums ORDER BY ArtistName')
        for albums in headphones_albums:
            if albums['ArtistName'] and albums['AlbumTitle']:
-                headphones_dict = {'ArtistName': albums['ArtistName'], 'AlbumTitle': albums['AlbumTitle']}
+                headphones_dict = {'ArtistName': albums['ArtistName'],
+                                   'AlbumTitle': albums['AlbumTitle']}
                headphones_album_dictionary.append(headphones_dict)
        # unmatchedalbums = [f for f in have_album_dictionary if f not in [x for x in headphones_album_dictionary]]

-        check = set([(cleanName(d['ArtistName']).lower(), cleanName(d['AlbumTitle']).lower()) for d in headphones_album_dictionary])
-        unmatchedalbums = [d for d in have_album_dictionary if (cleanName(d['ArtistName']).lower(), cleanName(d['AlbumTitle']).lower()) not in check]
+        check = set(
+            [(cleanName(d['ArtistName']).lower(), cleanName(d['AlbumTitle']).lower()) for d in
+             headphones_album_dictionary])
+        unmatchedalbums = [d for d in have_album_dictionary if (
+        cleanName(d['ArtistName']).lower(), cleanName(d['AlbumTitle']).lower()) not in check]

-        return serve_template(templatename="manageunmatched.html", title="Manage Unmatched Items", unmatchedalbums=unmatchedalbums)
+        return serve_template(templatename="manageunmatched.html", title="Manage Unmatched Items",
+                              unmatchedalbums=unmatchedalbums)

    @cherrypy.expose
-    def markUnmatched(self, action=None, existing_artist=None, existing_album=None, new_artist=None, new_album=None):
+    def markUnmatched(self, action=None, existing_artist=None, existing_album=None, new_artist=None,
+                      new_album=None):
        myDB = db.DBConnection()

        if action == "ignoreArtist":
            artist = existing_artist
-            myDB.action('UPDATE have SET Matched="Ignored" WHERE ArtistName=? AND Matched = "Failed"', [artist])
+            myDB.action(
+                'UPDATE have SET Matched="Ignored" WHERE ArtistName=? AND Matched = "Failed"',
+                [artist])

        elif action == "ignoreAlbum":
            artist = existing_artist
            album = existing_album
-            myDB.action('UPDATE have SET Matched="Ignored" WHERE ArtistName=? AND AlbumTitle=? AND Matched = "Failed"', (artist, album))
+            myDB.action(
+                'UPDATE have SET Matched="Ignored" WHERE ArtistName=? AND AlbumTitle=? AND Matched = "Failed"',
+                (artist, album))

        elif action == "matchArtist":
            existing_artist_clean = helpers.cleanName(existing_artist).lower()
            new_artist_clean = helpers.cleanName(new_artist).lower()
            if new_artist_clean != existing_artist_clean:
-                have_tracks = myDB.action('SELECT Matched, CleanName, Location, BitRate, Format FROM have WHERE ArtistName=?', [existing_artist])
+                have_tracks = myDB.action(
+                    'SELECT Matched, CleanName, Location, BitRate, Format FROM have WHERE ArtistName=?',
+                    [existing_artist])
                update_count = 0
                for entry in have_tracks:
                    old_clean_filename = entry['CleanName']
                    if old_clean_filename.startswith(existing_artist_clean):
-                        new_clean_filename = old_clean_filename.replace(existing_artist_clean, new_artist_clean, 1)
-                        myDB.action('UPDATE have SET CleanName=? WHERE ArtistName=? AND CleanName=?', [new_clean_filename, existing_artist, old_clean_filename])
+                        new_clean_filename = old_clean_filename.replace(existing_artist_clean,
+                                                                        new_artist_clean, 1)
+                        myDB.action(
+                            'UPDATE have SET CleanName=? WHERE ArtistName=? AND CleanName=?',
+                            [new_clean_filename, existing_artist, old_clean_filename])
                        controlValueDict = {"CleanName": new_clean_filename}
                        newValueDict = {"Location": entry['Location'],
                                        "BitRate": entry['BitRate'],
                                        "Format": entry['Format']
                                        }
                        # Attempt to match tracks with new CleanName
-                        match_alltracks = myDB.action('SELECT CleanName from alltracks WHERE CleanName=?', [new_clean_filename]).fetchone()
+                        match_alltracks = myDB.action(
+                            'SELECT CleanName from alltracks WHERE CleanName=?',
+                            [new_clean_filename]).fetchone()
                        if match_alltracks:
                            myDB.upsert("alltracks", newValueDict, controlValueDict)
-                        match_tracks = myDB.action('SELECT CleanName, AlbumID from tracks WHERE CleanName=?', [new_clean_filename]).fetchone()
+                        match_tracks = myDB.action(
+                            'SELECT CleanName, AlbumID from tracks WHERE CleanName=?',
+                            [new_clean_filename]).fetchone()
                        if match_tracks:
                            myDB.upsert("tracks", newValueDict, controlValueDict)
-                            myDB.action('UPDATE have SET Matched="Manual" WHERE CleanName=?', [new_clean_filename])
+                            myDB.action('UPDATE have SET Matched="Manual" WHERE CleanName=?',
+                                        [new_clean_filename])
                            update_count += 1
                            # This was throwing errors and I don't know why, but it seems to be working fine.
                            # else:
                            # logger.info("There was an error modifying Artist %s. This should not have happened" % existing_artist)
-                logger.info("Manual matching yielded %s new matches for Artist: %s" % (update_count, new_artist))
+                logger.info("Manual matching yielded %s new matches for Artist: %s" % (
+                update_count, new_artist))
                if update_count > 0:
                    librarysync.update_album_status()
            else:
-                logger.info("Artist %s already named appropriately; nothing to modify" % existing_artist)
+                logger.info(
+                    "Artist %s already named appropriately; nothing to modify" % existing_artist)

        elif action == "matchAlbum":
            existing_artist_clean = helpers.cleanName(existing_artist).lower()
@@ -560,83 +675,115 @@ class WebInterface(object):
            existing_clean_string = existing_artist_clean + " " + existing_album_clean
            new_clean_string = new_artist_clean + " " + new_album_clean
            if existing_clean_string != new_clean_string:
-                have_tracks = myDB.action('SELECT Matched, CleanName, Location, BitRate, Format FROM have WHERE ArtistName=? AND AlbumTitle=?', (existing_artist, existing_album))
+                have_tracks = myDB.action(
+                    'SELECT Matched, CleanName, Location, BitRate, Format FROM have WHERE ArtistName=? AND AlbumTitle=?',
+                    (existing_artist, existing_album))
                update_count = 0
                for entry in have_tracks:
                    old_clean_filename = entry['CleanName']
                    if old_clean_filename.startswith(existing_clean_string):
-                        new_clean_filename = old_clean_filename.replace(existing_clean_string, new_clean_string, 1)
-                        myDB.action('UPDATE have SET CleanName=? WHERE ArtistName=? AND AlbumTitle=? AND CleanName=?', [new_clean_filename, existing_artist, existing_album, old_clean_filename])
+                        new_clean_filename = old_clean_filename.replace(existing_clean_string,
+                                                                        new_clean_string, 1)
+                        myDB.action(
+                            'UPDATE have SET CleanName=? WHERE ArtistName=? AND AlbumTitle=? AND CleanName=?',
+                            [new_clean_filename, existing_artist, existing_album,
+                             old_clean_filename])
                        controlValueDict = {"CleanName": new_clean_filename}
                        newValueDict = {"Location": entry['Location'],
                                        "BitRate": entry['BitRate'],
                                        "Format": entry['Format']
                                        }
                        # Attempt to match tracks with new CleanName
-                        match_alltracks = myDB.action('SELECT CleanName from alltracks WHERE CleanName=?', [new_clean_filename]).fetchone()
+                        match_alltracks = myDB.action(
+                            'SELECT CleanName from alltracks WHERE CleanName=?',
+                            [new_clean_filename]).fetchone()
                        if match_alltracks:
                            myDB.upsert("alltracks", newValueDict, controlValueDict)
-                        match_tracks = myDB.action('SELECT CleanName, AlbumID from tracks WHERE CleanName=?', [new_clean_filename]).fetchone()
+                        match_tracks = myDB.action(
+                            'SELECT CleanName, AlbumID from tracks WHERE CleanName=?',
+                            [new_clean_filename]).fetchone()
                        if match_tracks:
                            myDB.upsert("tracks", newValueDict, controlValueDict)
-                            myDB.action('UPDATE have SET Matched="Manual" WHERE CleanName=?', [new_clean_filename])
+                            myDB.action('UPDATE have SET Matched="Manual" WHERE CleanName=?',
+                                        [new_clean_filename])
                            album_id = match_tracks['AlbumID']
                            update_count += 1
                            # This was throwing errors and I don't know why, but it seems to be working fine.
                            # else:
                            # logger.info("There was an error modifying Artist %s / Album %s with clean name %s" % (existing_artist, existing_album, existing_clean_string))
-                logger.info("Manual matching yielded %s new matches for Artist: %s / Album: %s" % (update_count, new_artist, new_album))
+                logger.info("Manual matching yielded %s new matches for Artist: %s / Album: %s" % (
+                update_count, new_artist, new_album))
                if update_count > 0:
                    librarysync.update_album_status(album_id)
            else:
-                logger.info("Artist %s / Album %s already named appropriately; nothing to modify" % (existing_artist, existing_album))
+                logger.info(
+                    "Artist %s / Album %s already named appropriately; nothing to modify" % (
+                    existing_artist, existing_album))

    @cherrypy.expose
    def manageManual(self):
        myDB = db.DBConnection()
        manual_albums = []
-        manualalbums = myDB.select('SELECT ArtistName, AlbumTitle, TrackTitle, CleanName, Matched from have')
+        manualalbums = myDB.select(
+            'SELECT ArtistName, AlbumTitle, TrackTitle, CleanName, Matched from have')
        for albums in manualalbums:
            if albums['ArtistName'] and albums['AlbumTitle'] and albums['TrackTitle']:
-                original_clean = helpers.cleanName(albums['ArtistName'] + " " + albums['AlbumTitle'] + " " + albums['TrackTitle'])
-                if albums['Matched'] == "Ignored" or albums['Matched'] == "Manual" or albums['CleanName'] != original_clean:
+                original_clean = helpers.cleanName(
+                    albums['ArtistName'] + " " + albums['AlbumTitle'] + " " + albums['TrackTitle'])
+                if albums['Matched'] == "Ignored" or albums['Matched'] == "Manual" or albums[
+                    'CleanName'] != original_clean:
                    if albums['Matched'] == "Ignored":
                        album_status = "Ignored"
                    elif albums['Matched'] == "Manual" or albums['CleanName'] != original_clean:
                        album_status = "Matched"
-                    manual_dict = {'ArtistName': albums['ArtistName'], 'AlbumTitle': albums['AlbumTitle'], 'AlbumStatus': album_status}
+                    manual_dict = {'ArtistName': albums['ArtistName'],
+                                   'AlbumTitle': albums['AlbumTitle'], 'AlbumStatus': album_status}
                    if manual_dict not in manual_albums:
                        manual_albums.append(manual_dict)
        manual_albums_sorted = sorted(manual_albums, key=itemgetter('ArtistName', 'AlbumTitle'))

-        return serve_template(templatename="managemanual.html", title="Manage Manual Items", manualalbums=manual_albums_sorted)
+        return serve_template(templatename="managemanual.html", title="Manage Manual Items",
+                              manualalbums=manual_albums_sorted)

    @cherrypy.expose
    def markManual(self, action=None, existing_artist=None, existing_album=None):
        myDB = db.DBConnection()
        if action == "unignoreArtist":
            artist = existing_artist
-            myDB.action('UPDATE have SET Matched="Failed" WHERE ArtistName=? AND Matched="Ignored"', [artist])
+            myDB.action('UPDATE have SET Matched="Failed" WHERE ArtistName=? AND Matched="Ignored"',
+                        [artist])
            logger.info("Artist: %s successfully restored to unmatched list" % artist)

        elif action == "unignoreAlbum":
            artist = existing_artist
            album = existing_album
-            myDB.action('UPDATE have SET Matched="Failed" WHERE ArtistName=? AND AlbumTitle=? AND Matched="Ignored"', (artist, album))
+            myDB.action(
+                'UPDATE have SET Matched="Failed" WHERE ArtistName=? AND AlbumTitle=? AND Matched="Ignored"',
+                (artist, album))
            logger.info("Album: %s successfully restored to unmatched list" % album)

        elif action == "unmatchArtist":
            artist = existing_artist
-            update_clean = myDB.select('SELECT ArtistName, AlbumTitle, TrackTitle, CleanName, Matched from have WHERE ArtistName=?', [artist])
+            update_clean = myDB.select(
+                'SELECT ArtistName, AlbumTitle, TrackTitle, CleanName, Matched from have WHERE ArtistName=?',
+                [artist])
            update_count = 0
            for tracks in update_clean:
-                original_clean = helpers.cleanName(tracks['ArtistName'] + " " + tracks['AlbumTitle'] + " " + tracks['TrackTitle']).lower()
+                original_clean = helpers.cleanName(
+                    tracks['ArtistName'] + " " + tracks['AlbumTitle'] + " " + tracks[
+                        'TrackTitle']).lower()
                album = tracks['AlbumTitle']
                track_title = tracks['TrackTitle']
                if tracks['CleanName'] != original_clean:
-                    myDB.action('UPDATE tracks SET Location=?, BitRate=?, Format=? WHERE CleanName=?', [None, None, None, tracks['CleanName']])
-                    myDB.action('UPDATE alltracks SET Location=?, BitRate=?, Format=? WHERE CleanName=?', [None, None, None, tracks['CleanName']])
-                    myDB.action('UPDATE have SET CleanName=?, Matched="Failed" WHERE ArtistName=? AND AlbumTitle=? AND TrackTitle=?', (original_clean, artist, album, track_title))
+                    myDB.action(
+                        'UPDATE tracks SET Location=?, BitRate=?, Format=? WHERE CleanName=?',
+                        [None, None, None, tracks['CleanName']])
+                    myDB.action(
+                        'UPDATE alltracks SET Location=?, BitRate=?, Format=? WHERE CleanName=?',
+                        [None, None, None, tracks['CleanName']])
+                    myDB.action(
+                        'UPDATE have SET CleanName=?, Matched="Failed" WHERE ArtistName=? AND AlbumTitle=? AND TrackTitle=?',
+                        (original_clean, artist, album, track_title))
                    update_count += 1
            if update_count > 0:
                librarysync.update_album_status()
@@ -645,18 +792,29 @@ class WebInterface(object):
        elif action == "unmatchAlbum":
            artist = existing_artist
            album = existing_album
-            update_clean = myDB.select('SELECT ArtistName, AlbumTitle, TrackTitle, CleanName, Matched from have WHERE ArtistName=? AND AlbumTitle=?', (artist, album))
+            update_clean = myDB.select(
+                'SELECT ArtistName, AlbumTitle, TrackTitle, CleanName, Matched from have WHERE ArtistName=? AND AlbumTitle=?',
+                (artist, album))
            update_count = 0
            for tracks in update_clean:
-                original_clean = helpers.cleanName(tracks['ArtistName'] + " " + tracks['AlbumTitle'] + " " + tracks['TrackTitle']).lower()
+                original_clean = helpers.cleanName(
+                    tracks['ArtistName'] + " " + tracks['AlbumTitle'] + " " + tracks[
+                        'TrackTitle']).lower()
                track_title = tracks['TrackTitle']
                if tracks['CleanName'] != original_clean:
-                    album_id_check = myDB.action('SELECT AlbumID from tracks WHERE CleanName=?', [tracks['CleanName']]).fetchone()
+                    album_id_check = myDB.action('SELECT AlbumID from tracks WHERE CleanName=?',
+                                                 [tracks['CleanName']]).fetchone()
                    if album_id_check:
                        album_id = album_id_check[0]
-                    myDB.action('UPDATE tracks SET Location=?, BitRate=?, Format=? WHERE CleanName=?', [None, None, None, tracks['CleanName']])
-                    myDB.action('UPDATE alltracks SET Location=?, BitRate=?, Format=? WHERE CleanName=?', [None, None, None, tracks['CleanName']])
-                    myDB.action('UPDATE have SET CleanName=?, Matched="Failed" WHERE ArtistName=? AND AlbumTitle=? AND TrackTitle=?', (original_clean, artist, album, track_title))
+                    myDB.action(
+                        'UPDATE tracks SET Location=?, BitRate=?, Format=? WHERE CleanName=?',
+                        [None, None, None, tracks['CleanName']])
+                    myDB.action(
+                        'UPDATE alltracks SET Location=?, BitRate=?, Format=? WHERE CleanName=?',
+                        [None, None, None, tracks['CleanName']])
+                    myDB.action(
+                        'UPDATE have SET CleanName=?, Matched="Failed" WHERE ArtistName=? AND AlbumTitle=? AND TrackTitle=?',
+                        (original_clean, artist, album, track_title))
                    update_count += 1
            if update_count > 0:
                librarysync.update_album_status(album_id)
@@ -740,9 +898,11 @@ class WebInterface(object):
        raise cherrypy.HTTPRedirect("home")

    @cherrypy.expose
-    def forcePostProcess(self, dir=None, album_dir=None):
+    def forcePostProcess(self, dir=None, album_dir=None, keep_original_folder=False):
        from headphones import postprocessor
-        threading.Thread(target=postprocessor.forcePostProcess, kwargs={'dir': dir, 'album_dir': album_dir}).start()
+        threading.Thread(target=postprocessor.forcePostProcess,
+                         kwargs={'dir': dir, 'album_dir': album_dir,
+                                 'keep_original_folder': keep_original_folder == 'True'}).start()
        raise cherrypy.HTTPRedirect("home")

    @cherrypy.expose
@@ -754,7 +914,8 @@ class WebInterface(object):
    @cherrypy.expose
    def history(self):
        myDB = db.DBConnection()
-        history = myDB.select('''SELECT * from snatched WHERE Status NOT LIKE "Seed%" order by DateAdded DESC''')
+        history = myDB.select(
+            '''SELECT AlbumID, Title, Size, URL, DateAdded, Status, Kind, ifnull(FolderName, '?') FolderName FROM snatched WHERE Status NOT LIKE "Seed%" ORDER BY DateAdded DESC''')
        return serve_template(templatename="history.html", title="History", history=history)

    @cherrypy.expose
@@ -777,7 +938,8 @@ class WebInterface(object):
        raise cherrypy.HTTPRedirect("logs")

    @cherrypy.expose
-    def getLog(self, iDisplayStart=0, iDisplayLength=100, iSortCol_0=0, sSortDir_0="desc", sSearch="", **kwargs):
+    def getLog(self, iDisplayStart=0, iDisplayLength=100, iSortCol_0=0, sSortDir_0="desc",
+               sSearch="", **kwargs):
        iDisplayStart = int(iDisplayStart)
        iDisplayLength = int(iDisplayLength)

@@ -785,7 +947,8 @@ class WebInterface(object):
        if sSearch == "":
            filtered = headphones.LOG_LIST[::]
        else:
-            filtered = [row for row in headphones.LOG_LIST for column in row if sSearch.lower() in column.lower()]
+            filtered = [row for row in headphones.LOG_LIST for column in row if
+                        sSearch.lower() in column.lower()]

        sortcolumn = 0
        if iSortCol_0 == '1':
@@ -804,7 +967,8 @@ class WebInterface(object):
        })

    @cherrypy.expose
-    def getArtists_json(self, iDisplayStart=0, iDisplayLength=100, sSearch="", iSortCol_0='0', sSortDir_0='asc', **kwargs):
+    def getArtists_json(self, iDisplayStart=0, iDisplayLength=100, sSearch="", iSortCol_0='0',
+                        sSortDir_0='asc', **kwargs):
        iDisplayStart = int(iDisplayStart)
        iDisplayLength = int(iDisplayLength)
        filtered = []
@@ -825,12 +989,15 @@ class WebInterface(object):
            filtered = myDB.select(query)
            totalcount = len(filtered)
        else:
-            query = 'SELECT * from artists WHERE ArtistSortName LIKE "%' + sSearch + '%" OR LatestAlbum LIKE "%' + sSearch + '%"' + 'ORDER BY %s COLLATE NOCASE %s' % (sortcolumn, sSortDir_0)
+            query = 'SELECT * from artists WHERE ArtistSortName LIKE "%' + sSearch + '%" OR LatestAlbum LIKE "%' + sSearch + '%"' + 'ORDER BY %s COLLATE NOCASE %s' % (
+            sortcolumn, sSortDir_0)
            filtered = myDB.select(query)
            totalcount = myDB.select('SELECT COUNT(*) from artists')[0][0]

        if sortbyhavepercent:
-            filtered.sort(key=lambda x: (float(x['HaveTracks']) / x['TotalTracks'] if x['TotalTracks'] > 0 else 0.0, x['HaveTracks'] if x['HaveTracks'] else 0.0), reverse=sSortDir_0 == "asc")
+            filtered.sort(key=lambda x: (
+            float(x['HaveTracks']) / x['TotalTracks'] if x['TotalTracks'] > 0 else 0.0,
+            x['HaveTracks'] if x['HaveTracks'] else 0.0), reverse=sSortDir_0 == "asc")

        # can't figure out how to change the datatables default sorting order when its using an ajax datasource so ill
        # just reverse it here and the first click on the "Latest Album" header will sort by descending release date
@@ -922,7 +1089,9 @@ class WebInterface(object):
                myDB.action('DELETE from snatched WHERE Status=?', [type])
        else:
            logger.info(u"Deleting '%s' from history" % title)
-            myDB.action('DELETE from snatched WHERE Status NOT LIKE "Seed%" AND Title=? AND DateAdded=?', [title, date_added])
+            myDB.action(
+                'DELETE from snatched WHERE Status NOT LIKE "Seed%" AND Title=? AND DateAdded=?',
+                [title, date_added])
        raise cherrypy.HTTPRedirect("history")

    @cherrypy.expose
@@ -943,7 +1112,8 @@ class WebInterface(object):
        logger.info('All tracks in library unmatched')
        myDB.action('UPDATE artists SET HaveTracks=NULL')
        logger.info('Reset track counts for all artists')
-        myDB.action('UPDATE albums SET Status="Skipped" WHERE Status="Skipped" OR Status="Downloaded"')
+        myDB.action(
+            'UPDATE albums SET Status="Skipped" WHERE Status="Skipped" OR Status="Downloaded"')
        logger.info('Marking all unwanted albums as Skipped')
        try:
            threading.Thread(target=librarysync.libraryScan).start()
@@ -954,7 +1124,8 @@ class WebInterface(object):
    @cherrypy.expose
    def config(self):
        interface_dir = os.path.join(headphones.PROG_DIR, 'data/interfaces/')
-        interface_list = [name for name in os.listdir(interface_dir) if os.path.isdir(os.path.join(interface_dir, name))]
+        interface_list = [name for name in os.listdir(interface_dir) if
+                          os.path.isdir(os.path.join(interface_dir, name))]

        config = {
            "http_host": headphones.CONFIG.HTTP_HOST,
@@ -1005,6 +1176,11 @@ class WebInterface(object):
            "newznab_apikey": headphones.CONFIG.NEWZNAB_APIKEY,
            "newznab_enabled": checked(headphones.CONFIG.NEWZNAB_ENABLED),
            "extra_newznabs": headphones.CONFIG.get_extra_newznabs(),
+            "use_torznab": checked(headphones.CONFIG.TORZNAB),
+            "torznab_host": headphones.CONFIG.TORZNAB_HOST,
+            "torznab_apikey": headphones.CONFIG.TORZNAB_APIKEY,
+            "torznab_enabled": checked(headphones.CONFIG.TORZNAB_ENABLED),
+            "extra_torznabs": headphones.CONFIG.get_extra_torznabs(),
            "use_nzbsorg": checked(headphones.CONFIG.NZBSORG),
            "nzbsorg_uid": headphones.CONFIG.NZBSORG_UID,
            "nzbsorg_hash": headphones.CONFIG.NZBSORG_HASH,
@@ -1041,6 +1217,8 @@ class WebInterface(object):
            "whatcd_username": headphones.CONFIG.WHATCD_USERNAME,
            "whatcd_password": headphones.CONFIG.WHATCD_PASSWORD,
            "whatcd_ratio": headphones.CONFIG.WHATCD_RATIO,
+            "use_strike": checked(headphones.CONFIG.STRIKE),
+            "strike_ratio": headphones.CONFIG.STRIKE_RATIO,
            "pref_qual_0": radio(headphones.CONFIG.PREFERRED_QUALITY, 0),
            "pref_qual_1": radio(headphones.CONFIG.PREFERRED_QUALITY, 1),
            "pref_qual_2": radio(headphones.CONFIG.PREFERRED_QUALITY, 2),
@@ -1048,7 +1226,8 @@ class WebInterface(object):
            "preferred_bitrate": headphones.CONFIG.PREFERRED_BITRATE,
            "preferred_bitrate_high": headphones.CONFIG.PREFERRED_BITRATE_HIGH_BUFFER,
            "preferred_bitrate_low": headphones.CONFIG.PREFERRED_BITRATE_LOW_BUFFER,
-            "preferred_bitrate_allow_lossless": checked(headphones.CONFIG.PREFERRED_BITRATE_ALLOW_LOSSLESS),
+            "preferred_bitrate_allow_lossless": checked(
+                headphones.CONFIG.PREFERRED_BITRATE_ALLOW_LOSSLESS),
            "detect_bitrate": checked(headphones.CONFIG.DETECT_BITRATE),
            "lossless_bitrate_from": headphones.CONFIG.LOSSLESS_BITRATE_FROM,
            "lossless_bitrate_to": headphones.CONFIG.LOSSLESS_BITRATE_TO,
@@ -1066,15 +1245,19 @@ class WebInterface(object):
            "embed_album_art": checked(headphones.CONFIG.EMBED_ALBUM_ART),
            "embed_lyrics": checked(headphones.CONFIG.EMBED_LYRICS),
            "replace_existing_folders": checked(headphones.CONFIG.REPLACE_EXISTING_FOLDERS),
+            "keep_original_folder": checked(headphones.CONFIG.KEEP_ORIGINAL_FOLDER),
            "destination_dir": headphones.CONFIG.DESTINATION_DIR,
            "lossless_destination_dir": headphones.CONFIG.LOSSLESS_DESTINATION_DIR,
            "folder_format": headphones.CONFIG.FOLDER_FORMAT,
            "file_format": headphones.CONFIG.FILE_FORMAT,
            "file_underscores": checked(headphones.CONFIG.FILE_UNDERSCORES),
            "include_extras": checked(headphones.CONFIG.INCLUDE_EXTRAS),
+            "official_releases_only": checked(headphones.CONFIG.OFFICIAL_RELEASES_ONLY),
+            "wait_until_release_date": checked(headphones.CONFIG.WAIT_UNTIL_RELEASE_DATE),
            "autowant_upcoming": checked(headphones.CONFIG.AUTOWANT_UPCOMING),
            "autowant_all": checked(headphones.CONFIG.AUTOWANT_ALL),
            "autowant_manually_added": checked(headphones.CONFIG.AUTOWANT_MANUALLY_ADDED),
+            "do_not_process_unmatched": checked(headphones.CONFIG.DO_NOT_PROCESS_UNMATCHED),
            "keep_torrent_files": checked(headphones.CONFIG.KEEP_TORRENT_FILES),
            "prefer_torrents_0": radio(headphones.CONFIG.PREFER_TORRENTS, 0),
            "prefer_torrents_1": radio(headphones.CONFIG.PREFER_TORRENTS, 1),
@@ -1082,6 +1265,7 @@ class WebInterface(object):
            "magnet_links_0": radio(headphones.CONFIG.MAGNET_LINKS, 0),
            "magnet_links_1": radio(headphones.CONFIG.MAGNET_LINKS, 1),
            "magnet_links_2": radio(headphones.CONFIG.MAGNET_LINKS, 2),
+            "magnet_links_3": radio(headphones.CONFIG.MAGNET_LINKS, 3),
            "log_dir": headphones.CONFIG.LOG_DIR,
            "cache_dir": headphones.CONFIG.CACHE_DIR,
            "interface_list": interface_list,
@@ -1120,6 +1304,7 @@ class WebInterface(object):
            "plex_client_host": headphones.CONFIG.PLEX_CLIENT_HOST,
            "plex_username": headphones.CONFIG.PLEX_USERNAME,
            "plex_password": headphones.CONFIG.PLEX_PASSWORD,
+            "plex_token": headphones.CONFIG.PLEX_TOKEN,
            "plex_update": checked(headphones.CONFIG.PLEX_UPDATE),
            "plex_notify": checked(headphones.CONFIG.PLEX_NOTIFY),
            "nma_enabled": checked(headphones.CONFIG.NMA_ENABLED),
@@ -1214,17 +1399,31 @@ class WebInterface(object):
        # Handle the variable config options. Note - keys with False values aren't getting passed

        checked_configs = [
-            "launch_browser", "enable_https", "api_enabled", "use_blackhole", "headphones_indexer", "use_newznab", "newznab_enabled",
-            "use_nzbsorg", "use_omgwtfnzbs", "use_kat", "use_piratebay", "use_oldpiratebay", "use_mininova", "use_waffles", "use_rutracker",
-            "use_whatcd", "preferred_bitrate_allow_lossless", "detect_bitrate", "ignore_clean_releases", "freeze_db", "cue_split", "move_files", "rename_files",
-            "correct_metadata", "cleanup_files", "keep_nfo", "add_album_art", "embed_album_art", "embed_lyrics", "replace_existing_folders",
-            "file_underscores", "include_extras", "autowant_upcoming", "autowant_all", "autowant_manually_added", "keep_torrent_files", "music_encoder",
-            "encoderlossless", "encoder_multicore", "delete_lossless_files", "growl_enabled", "growl_onsnatch", "prowl_enabled",
-            "prowl_onsnatch", "xbmc_enabled", "xbmc_update", "xbmc_notify", "lms_enabled", "plex_enabled", "plex_update", "plex_notify",
-            "nma_enabled", "nma_onsnatch", "pushalot_enabled", "pushalot_onsnatch", "synoindex_enabled", "pushover_enabled",
-            "pushover_onsnatch", "pushbullet_enabled", "pushbullet_onsnatch", "subsonic_enabled", "twitter_enabled", "twitter_onsnatch",
-            "osx_notify_enabled", "osx_notify_onsnatch", "boxcar_enabled", "boxcar_onsnatch", "songkick_enabled", "songkick_filter_enabled",
-            "mpc_enabled", "email_enabled", "email_ssl", "email_tls", "email_onsnatch", "customauth", "idtag"
+            "launch_browser", "enable_https", "api_enabled", "use_blackhole", "headphones_indexer",
+            "use_newznab", "newznab_enabled", "use_torznab", "torznab_enabled",
+            "use_nzbsorg", "use_omgwtfnzbs", "use_kat", "use_piratebay", "use_oldpiratebay",
+            "use_mininova", "use_waffles", "use_rutracker",
+            "use_whatcd", "use_strike", "preferred_bitrate_allow_lossless", "detect_bitrate",
+            "ignore_clean_releases", "freeze_db", "cue_split", "move_files",
+            "rename_files", "correct_metadata", "cleanup_files", "keep_nfo", "add_album_art",
+            "embed_album_art", "embed_lyrics",
+            "replace_existing_folders", "keep_original_folder", "file_underscores",
+            "include_extras", "official_releases_only",
+            "wait_until_release_date", "autowant_upcoming", "autowant_all",
+            "autowant_manually_added", "do_not_process_unmatched", "keep_torrent_files",
+            "music_encoder",
+            "encoderlossless", "encoder_multicore", "delete_lossless_files", "growl_enabled",
+            "growl_onsnatch", "prowl_enabled",
+            "prowl_onsnatch", "xbmc_enabled", "xbmc_update", "xbmc_notify", "lms_enabled",
+            "plex_enabled", "plex_update", "plex_notify",
+            "nma_enabled", "nma_onsnatch", "pushalot_enabled", "pushalot_onsnatch",
+            "synoindex_enabled", "pushover_enabled",
+            "pushover_onsnatch", "pushbullet_enabled", "pushbullet_onsnatch", "subsonic_enabled",
+            "twitter_enabled", "twitter_onsnatch",
+            "osx_notify_enabled", "osx_notify_onsnatch", "boxcar_enabled", "boxcar_onsnatch",
+            "songkick_enabled", "songkick_filter_enabled",
+            "mpc_enabled", "email_enabled", "email_ssl", "email_tls", "email_onsnatch",
+            "customauth", "idtag"
        ]
        for checked_config in checked_configs:
            if checked_config not in kwargs:
@@ -1236,6 +1435,12 @@ class WebInterface(object):
            kwargs[plain_config] = kwargs[use_config]
            del kwargs[use_config]

+        # Check if encoderoutputformat is set multiple times
+        if len(kwargs['encoderoutputformat'][-1]) > 1:
+            kwargs['encoderoutputformat'] = kwargs['encoderoutputformat'][-1]
+        else:
+            kwargs['encoderoutputformat'] = kwargs['encoderoutputformat'][0]
+
        extra_newznabs = []
        for kwarg in [x for x in kwargs if x.startswith('newznab_host')]:
            newznab_host_key = kwarg
@@ -1251,6 +1456,21 @@ class WebInterface(object):
                        del kwargs[key]
                extra_newznabs.append((newznab_host, newznab_api, newznab_enabled))

+        extra_torznabs = []
+        for kwarg in [x for x in kwargs if x.startswith('torznab_host')]:
+            torznab_host_key = kwarg
+            torznab_number = kwarg[12:]
+            if len(torznab_number):
+                torznab_api_key = 'torznab_api' + torznab_number
+                torznab_enabled_key = 'torznab_enabled' + torznab_number
+                torznab_host = kwargs.get(torznab_host_key, '')
+                torznab_api = kwargs.get(torznab_api_key, '')
+                torznab_enabled = int(kwargs.get(torznab_enabled_key, 0))
+                for key in [torznab_host_key, torznab_api_key, torznab_enabled_key]:
+                    if key in kwargs:
+                        del kwargs[key]
+                extra_torznabs.append((torznab_host, torznab_api, torznab_enabled))
+
        # Convert the extras to list then string. Coming in as 0 or 1 (append new extras to the end)
        temp_extras_list = []

@@ -1276,11 +1496,18 @@ class WebInterface(object):
                del kwargs[extra]

        headphones.CONFIG.EXTRAS = ','.join(str(n) for n in temp_extras_list)
+
        headphones.CONFIG.clear_extra_newznabs()
+        headphones.CONFIG.clear_extra_torznabs()
+
        headphones.CONFIG.process_kwargs(kwargs)
+
        for extra_newznab in extra_newznabs:
            headphones.CONFIG.add_extra_newznab(extra_newznab)

+        for extra_torznab in extra_torznabs:
+            headphones.CONFIG.add_extra_torznab(extra_torznab)
+
        # Sanity checking
        if headphones.CONFIG.SEARCH_INTERVAL and headphones.CONFIG.SEARCH_INTERVAL < 360:
            logger.info("Search interval too low. Resetting to 6 hour minimum")
@@ -1378,9 +1605,11 @@ class WebInterface(object):
            image_dict = {'artwork': image_url, 'thumbnail': thumb_url}
        elif AlbumID and (not image_dict['artwork'] or not image_dict['thumbnail']):
            if not image_dict['artwork']:
-                image_dict['artwork'] = "http://coverartarchive.org/release/%s/front-500.jpg" % AlbumID
+                image_dict[
+                    'artwork'] = "http://coverartarchive.org/release/%s/front-500.jpg" % AlbumID
            if not image_dict['thumbnail']:
-                image_dict['thumbnail'] = "http://coverartarchive.org/release/%s/front-250.jpg" % AlbumID
+                image_dict[
+                    'thumbnail'] = "http://coverartarchive.org/release/%s/front-250.jpg" % AlbumID

        return json.dumps(image_dict)

@@ -1419,11 +1648,32 @@ class WebInterface(object):
        if result:
            osx_notify = notifiers.OSX_NOTIFY()
            osx_notify.notify('Registered', result, 'Success :-)')
-            logger.info('Registered %s, to re-register a different app, delete this app first' % result)
+            logger.info(
+                'Registered %s, to re-register a different app, delete this app first' % result)
        else:
            logger.warn(msg)
        return msg

+    @cherrypy.expose
+    def testPushover(self):
+        logger.info(u"Sending Pushover notification")
+        pushover = notifiers.PUSHOVER()
+        result = pushover.notify("hooray!", "This is a test")
+        return str(result)
+
+    @cherrypy.expose
+    def testPlex(self):
+        logger.info(u"Testing plex notifications")
+        plex = notifiers.Plex()
+        plex.notify("hellooooo", "test album!", "")
+
+    @cherrypy.expose
+    def testPushbullet(self):
+        logger.info("Testing Pushbullet notifications")
+        pushbullet = notifiers.PUSHBULLET()
+        pushbullet.notify("it works!", "Test message")
+
+
 class Artwork(object):
    @cherrypy.expose
    def index(self):
@@ -1491,4 +1741,6 @@ class Artwork(object):
                return fp.read()

    thumbs = Thumbs()
+
+
 WebInterface.artwork = Artwork()
@@ -13,18 +13,17 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.

-import os
 import sys
+
+import os
 import cherrypy
 import headphones
-
 from headphones import logger
 from headphones.webserve import WebInterface
 from headphones.helpers import create_https_certificates


 def initialize(options):
-
    # HTTPS stuff stolen from sickbeard
    enable_https = options['enable_https']
    https_cert = options['https_cert']
@@ -33,7 +32,8 @@ def initialize(options):
    if enable_https:
        # If either the HTTPS certificate or key do not exist, try to make
        # self-signed ones.
-        if not (https_cert and os.path.exists(https_cert)) or not (https_key and os.path.exists(https_key)):
+        if not (https_cert and os.path.exists(https_cert)) or not (
+            https_key and os.path.exists(https_key)):
            if not create_https_certificates(https_cert, https_key):
                logger.warn("Unable to create certificate and key. Disabling " \
                            "HTTPS")
@@ -99,7 +99,8 @@ def initialize(options):
    }

    if options['http_password']:
-        logger.info("Web server authentication is enabled, username is '%s'", options['http_username'])
+        logger.info("Web server authentication is enabled, username is '%s'",
+                    options['http_username'])

        conf['/'].update({
            'tools.auth_basic.on': True,
@@ -118,7 +119,8 @@ def initialize(options):
        cherrypy.process.servers.check_port(str(options['http_host']), options['http_port'])
        cherrypy.server.start()
    except IOError:
-        sys.stderr.write('Failed to start on port: %i. Is something else running?\n' % (options['http_port']))
+        sys.stderr.write(
+            'Failed to start on port: %i. Is something else running?\n' % (options['http_port']))
        sys.exit(1)

    cherrypy.server.wait()
@@ -27,17 +27,17 @@ rcvar=${name}_enable

 load_rc_config ${name}

-: ${headphones_enable:="NO"}
-: ${headphones_user:="_sabnzbd"}
-: ${headphones_dir:="/usr/local/headphones"}
-: ${headphones_chdir:="${headphones_dir}"}
-: ${headphones_pid:="${headphones_dir}/headphones.pid"}
-: ${headphones_conf:="${headphones_dir}/config.ini"}
+: "${headphones_enable:="NO"}"
+: "${headphones_user:="_sabnzbd"}"
+: "${headphones_dir:="/usr/local/headphones"}"
+: "${headphones_chdir:="${headphones_dir}"}"
+: "${headphones_pid:="${headphones_dir}/headphones.pid"}"
+: "${headphones_conf:="${headphones_dir}/config.ini"}"

 WGET="/usr/local/bin/wget" # You need wget for this script to safely shutdown Headphones.
 if [ -e "${headphones_conf}" ]; then
-        HOST=`grep -A64 "\[General\]" "${headphones_conf}"|egrep "^http_host"|perl -wple 's/^http_host = (.*)$/$1/'`
-        PORT=`grep -A64 "\[General\]" "${headphones_conf}"|egrep "^http_port"|perl -wple 's/^http_port = (.*)$/$1/'`
+        HOST=$(grep -A64 "\[General\]" "${headphones_conf}"|egrep "^http_host"|perl -wple 's/^http_host = (.*)$/$1/')
+        PORT=$(grep -A64 "\[General\]" "${headphones_conf}"|egrep "^http_port"|perl -wple 's/^http_port = (.*)$/$1/')
 fi

 status_cmd="${name}_status"
@@ -53,15 +53,15 @@ if [ ! -x "${WGET}" ]; then
 fi

 # Ensure user is root when running this script.
-if [ `id -u` != "0" ]; then
+if [ "$(id -u)" != "0" ]; then
  echo "Oops, you should be root before running this!"
  exit 1
 fi

 verify_headphones_pid() {
    # Make sure the pid corresponds to the Headphones process.
-    pid=`cat ${headphones_pid} 2>/dev/null`
-    ps -p ${pid} | grep -q "python ${headphones_dir}/Headphones.py"
+    pid=$(cat "${headphones_pid}" 2>/dev/null)
+    pgrep -F "${headphones_pid}" -q "python ${headphones_dir}/Headphones.py"
    return $?
 }

@@ -73,10 +73,10 @@ headphones_stop() {
    fi
    echo "Stopping $name"
    verify_headphones_pid
-    ${WGET} -O - -q --user=${SBUSR} --password=${SBPWD} "http://${HOST}:${PORT}/shutdown/" >/dev/null
+    ${WGET} -O - -q --user="${SBUSR}" --password="${SBPWD}" "http://${HOST}:${PORT}/shutdown/" >/dev/null
    
    if [ -n "${pid}" ]; then
-      wait_for_pids ${pid}
+      wait_for_pids "${pid}"
      echo "Stopped $name"
    fi
 }
@@ -28,11 +28,11 @@ rcvar=${name}_enable

 load_rc_config ${name}

-: ${headphones_enable:="NO"}
-: ${headphones_user:="_sabnzbd"}
-: ${headphones_dir:="/usr/local/headphones"}
-: ${headphones_chdir:="${headphones_dir}"}
-: ${headphones_pid:="${headphones_dir}/headphones.pid"}
+: "${headphones_enable:="NO"}"
+: "${headphones_user:="_sabnzbd"}"
+: "${headphones_dir:="/usr/local/headphones"}"
+: "${headphones_chdir:="${headphones_dir}"}"
+: "${headphones_pid:="${headphones_dir}/headphones.pid"}"

 status_cmd="${name}_status"
 stop_cmd="${name}_stop"
@@ -41,15 +41,15 @@ command="/usr/sbin/daemon"
 command_args="-f -p ${headphones_pid} python ${headphones_dir}/Headphones.py ${headphones_flags} --quiet --nolaunch"

 # Ensure user is root when running this script.
-if [ `id -u` != "0" ]; then
+if [ "$(id -u)" != "0" ]; then
  echo "Oops, you should be root before running this!"
  exit 1
 fi

 verify_headphones_pid() {
    # Make sure the pid corresponds to the Headphones process.
-    pid=`cat ${headphones_pid} 2>/dev/null`
-    ps -p ${pid} | grep -q "python ${headphones_dir}/Headphones.py"
+    pid=$(cat "${headphones_pid}" 2>/dev/null)
+    pgrep -F "${headphones_pid}" -q "python ${headphones_dir}/Headphones.py"
    return $?
 }

@@ -58,7 +58,7 @@ headphones_stop() {
    echo "Stopping $name"
    verify_headphones_pid
    if [ -n "${pid}" ]; then
-      wait_for_pids ${pid}
+      wait_for_pids "${pid}"
      echo "Stopped"
    fi
 }
@@ -32,7 +32,6 @@
 ## HP_PIDFILE=      #$PID_FILE, the location of headphones.pid, the default is /var/run/headphones/headphones.pid
 ## PYTHON_BIN=      #$DAEMON, the location of the python binary, the default is /usr/bin/python
 ## HP_OPTS=         #$EXTRA_DAEMON_OPTS, extra cli option for headphones, i.e. " --config=/home/headphones/config.ini"
-## SSD_OPTS=        #$EXTRA_SSD_OPTS, extra start-stop-daemon option like " --group=users"
 ## HP_PORT=         #$PORT_OPTS, hardcoded port for the webserver, overrides value in config.ini
 ##
 ## EXAMPLE if want to run as different user
@@ -101,9 +100,6 @@ load_settings() {
        # Extra daemon option like: HP_OPTS=" --config=/home/headphones/config.ini"
        EXTRA_DAEMON_OPTS=${HP_OPTS-}

-        # Extra start-stop-daemon option like START_OPTS=" --group=users"
-        EXTRA_SSD_OPTS=${SSD_OPTS-}
-
        # Hardcoded port to run on, overrides config.ini settings
        [ -n "$HP_PORT" ] && {
            PORT_OPTS=" --port=${HP_PORT} "
@@ -114,7 +110,7 @@ load_settings() {
        SETTINGS_LOADED=TRUE
    fi

-    [ -x $DAEMON ] || {
+    [ -x "$DAEMON" ] || {
        log_warning_msg "$DESC: Can't execute daemon, aborting. See $DAEMON";
        return 1;}

@@ -125,8 +121,8 @@ load_settings || exit 0

 is_running () {
    # returns 1 when running, else 0.
-    if [ -e $PID_FILE ]; then
-      PID=`cat $PID_FILE`
+    if [ -e "$PID_FILE" ]; then
+      PID=$(cat "$PID_FILE")

      RET=$?
      [ $RET -gt 1 ] && exit 1 || return $RET
@@ -136,28 +132,28 @@ is_running () {
 }

 handle_pid () {
-    PID_PATH=`dirname $PID_FILE`
-    [ -d $PID_PATH ] || mkdir -p $PID_PATH && chown -R $RUN_AS $PID_PATH > /dev/null || {
+    PID_PATH=$(dirname "$PID_FILE")
+    [ -d "$PID_PATH" ] || mkdir -p "$PID_PATH" && chown -R "$RUN_AS" "$PID_PATH" > /dev/null || {
        log_warning_msg "$DESC: Could not create $PID_FILE, See $SETTINGS, aborting.";
        return 1;}

-    if [ -e $PID_FILE ]; then
-        PID=`cat $PID_FILE`
-        if ! kill -0 $PID > /dev/null 2>&1; then
+    if [ -e "$PID_FILE" ]; then
+        PID=$(cat "$PID_FILE")
+        if ! kill -0 "$PID" > /dev/null 2>&1; then
            log_warning_msg "Removing stale $PID_FILE"
-            rm $PID_FILE
+            rm "$PID_FILE"
        fi
    fi
 }

 handle_datadir () {
-    [ -d $DATA_DIR ] || mkdir -p $DATA_DIR && chown -R $RUN_AS $DATA_DIR > /dev/null || {
+    [ -d "$DATA_DIR" ] || mkdir -p "$DATA_DIR" && chown -R "$RUN_AS" "$DATA_DIR" > /dev/null || {
        log_warning_msg "$DESC: Could not create $DATA_DIR, See $SETTINGS, aborting.";
        return 1;}
 }

 handle_updates () {
-    chown -R $RUN_AS $APP_PATH > /dev/null || {
+    chown -R "$RUN_AS" "$APP_PATH" > /dev/null || {
        log_warning_msg "$DESC: $APP_PATH not writable by $RUN_AS for web-updates";
        return 0; }
 }
@@ -168,7 +164,7 @@ start_headphones () {
    handle_updates
    if ! is_running; then
        log_daemon_msg "Starting $DESC"
-        start-stop-daemon -o -d $APP_PATH -c $RUN_AS --start $EXTRA_SSD_OPTS --pidfile $PID_FILE --exec $DAEMON -- $DAEMON_OPTS
+        start-stop-daemon -o -d "$APP_PATH" -c "$RUN_AS" --start "$EXTRA_SSD"_OPTS --pidfile "$PID_FILE" --exec "$DAEMON" -- "$DAEMON_OPTS"
        check_retval
    else
        log_success_msg "$DESC: already running (pid $PID)"
@@ -178,7 +174,7 @@ start_headphones () {
 stop_headphones () {
    if is_running; then
        log_daemon_msg "Stopping $DESC"
-        start-stop-daemon -o --stop --pidfile $PID_FILE --retry 15
+        start-stop-daemon -o --stop --pidfile "$PID_FILE" --retry 15
        check_retval
    else
        log_success_msg "$DESC: not running"
@@ -17,8 +17,8 @@ http://www.crummy.com/software/BeautifulSoup/bs4/doc/
 """

 __author__ = "Leonard Richardson (leonardr@segfault.org)"
-__version__ = "4.3.2"
-__copyright__ = "Copyright (c) 2004-2013 Leonard Richardson"
+__version__ = "4.4.0"
+__copyright__ = "Copyright (c) 2004-2015 Leonard Richardson"
 __license__ = "MIT"

 __all__ = ['BeautifulSoup']
@@ -45,7 +45,7 @@ from .element import (

 # The very first thing we do is give a useful error if someone is
 # running this code under Python 3 without converting it.
-syntax_error = u'You are trying to run the Python 2 version of Beautiful Soup under Python 3. This will not work. You need to convert the code, either by installing it (`python setup.py install`) or by running 2to3 (`2to3 -w bs4`).'
+'You are trying to run the Python 2 version of Beautiful Soup under Python 3. This will not work.'<>'You need to convert the code, either by installing it (`python setup.py install`) or by running 2to3 (`2to3 -w bs4`).'

 class BeautifulSoup(Tag):
    """
@@ -77,8 +77,11 @@ class BeautifulSoup(Tag):

    ASCII_SPACES = '\x20\x0a\x09\x0c\x0d'

+    NO_PARSER_SPECIFIED_WARNING = "No parser was explicitly specified, so I'm using the best available %(markup_type)s parser for this system (\"%(parser)s\"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.\n\nTo get rid of this warning, change this:\n\n BeautifulSoup([your markup])\n\nto this:\n\n BeautifulSoup([your markup], \"%(parser)s\")\n"
+
    def __init__(self, markup="", features=None, builder=None,
-                 parse_only=None, from_encoding=None, **kwargs):
+                 parse_only=None, from_encoding=None, exclude_encodings=None,
+                 **kwargs):
        """The Soup object is initialized as the 'root tag', and the
        provided markup (which can be a string or a file-like object)
        is fed into the underlying parser."""
@@ -114,9 +117,9 @@ class BeautifulSoup(Tag):
            del kwargs['isHTML']
            warnings.warn(
                "BS4 does not respect the isHTML argument to the "
-                "BeautifulSoup constructor. You can pass in features='html' "
-                "or features='xml' to get a builder capable of handling "
-                "one or the other.")
+                "BeautifulSoup constructor. Suggest you use "
+                "features='lxml' for HTML and features='lxml-xml' for "
+                "XML.")

        def deprecated_argument(old_name, new_name):
            if old_name in kwargs:
@@ -140,6 +143,7 @@ class BeautifulSoup(Tag):
                "__init__() got an unexpected keyword argument '%s'" % arg)

        if builder is None:
+            original_features = features
            if isinstance(features, basestring):
                features = [features]
            if features is None or len(features) == 0:
@@ -151,6 +155,16 @@ class BeautifulSoup(Tag):
                    "requested: %s. Do you need to install a parser library?"
                    % ",".join(features))
            builder = builder_class()
+            if not (original_features == builder.NAME or
+                    original_features in builder.ALTERNATE_NAMES):
+                if builder.is_xml:
+                    markup_type = "XML"
+                else:
+                    markup_type = "HTML"
+                warnings.warn(self.NO_PARSER_SPECIFIED_WARNING % dict(
+                    parser=builder.NAME,
+                    markup_type=markup_type))
+
        self.builder = builder
        self.is_xml = builder.is_xml
        self.builder.soup = self
@@ -178,6 +192,8 @@ class BeautifulSoup(Tag):
                # system. Just let it go.
                pass
            if is_file:
+                if isinstance(markup, unicode):
+                    markup = markup.encode("utf8")
                warnings.warn(
                    '"%s" looks like a filename, not markup. You should probably open this file and pass the filehandle into Beautiful Soup.' % markup)
            if markup[:5] == "http:" or markup[:6] == "https:":
@@ -185,12 +201,15 @@ class BeautifulSoup(Tag):
                # Python 3 otherwise.
                if ((isinstance(markup, bytes) and not b' ' in markup)
                    or (isinstance(markup, unicode) and not u' ' in markup)):
+                    if isinstance(markup, unicode):
+                        markup = markup.encode("utf8")
                    warnings.warn(
                        '"%s" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.' % markup)

        for (self.markup, self.original_encoding, self.declared_html_encoding,
         self.contains_replacement_characters) in (
-            self.builder.prepare_markup(markup, from_encoding)):
+             self.builder.prepare_markup(
+                 markup, from_encoding, exclude_encodings=exclude_encodings)):
            self.reset()
            try:
                self._feed()
@@ -203,6 +222,16 @@ class BeautifulSoup(Tag):
        self.markup = None
        self.builder.soup = None

+    def __copy__(self):
+        return type(self)(self.encode(), builder=self.builder)
+
+    def __getstate__(self):
+        # Frequently a tree builder can't be pickled.
+        d = dict(self.__dict__)
+        if 'builder' in d and not self.builder.picklable:
+            del d['builder']
+        return d
+
    def _feed(self):
        # Convert the document to Unicode.
        self.builder.reset()
@@ -229,9 +258,7 @@ class BeautifulSoup(Tag):

    def new_string(self, s, subclass=NavigableString):
        """Create a new NavigableString associated with this soup."""
-        navigable = subclass(s)
-        navigable.setup()
-        return navigable
+        return subclass(s)

    def insert_before(self, successor):
        raise NotImplementedError("BeautifulSoup objects don't support insert_before().")
@@ -290,14 +317,49 @@ class BeautifulSoup(Tag):
    def object_was_parsed(self, o, parent=None, most_recent_element=None):
        """Add an object to the parse tree."""
        parent = parent or self.currentTag
-        most_recent_element = most_recent_element or self._most_recent_element
-        o.setup(parent, most_recent_element)
+        previous_element = most_recent_element or self._most_recent_element
+
+        next_element = previous_sibling = next_sibling = None
+        if isinstance(o, Tag):
+            next_element = o.next_element
+            next_sibling = o.next_sibling
+            previous_sibling = o.previous_sibling
+            if not previous_element:
+                previous_element = o.previous_element
+
+        o.setup(parent, previous_element, next_element, previous_sibling, next_sibling)

-        if most_recent_element is not None:
-            most_recent_element.next_element = o
        self._most_recent_element = o
        parent.contents.append(o)

+        if parent.next_sibling:
+            # This node is being inserted into an element that has
+            # already been parsed. Deal with any dangling references.
+            index = parent.contents.index(o)
+            if index == 0:
+                previous_element = parent
+                previous_sibling = None
+            else:
+                previous_element = previous_sibling = parent.contents[index-1]
+            if index == len(parent.contents)-1:
+                next_element = parent.next_sibling
+                next_sibling = None
+            else:
+                next_element = next_sibling = parent.contents[index+1]
+
+            o.previous_element = previous_element
+            if previous_element:
+                previous_element.next_element = o
+            o.next_element = next_element
+            if next_element:
+                next_element.previous_element = o
+            o.next_sibling = next_sibling
+            if next_sibling:
+                next_sibling.previous_sibling = o
+            o.previous_sibling = previous_sibling
+            if previous_sibling:
+                previous_sibling.next_sibling = o
+
    def _popToTag(self, name, nsprefix=None, inclusivePop=True):
        """Pops the tag stack up to and including the most recent
        instance of the given tag. If inclusivePop is false, pops the tag
@@ -80,9 +80,12 @@ builder_registry = TreeBuilderRegistry()
 class TreeBuilder(object):
    """Turn a document into a Beautiful Soup object tree."""

+    NAME = "[Unknown tree builder]"
+    ALTERNATE_NAMES = []
    features = []

    is_xml = False
+    picklable = False
    preserve_whitespace_tags = set()
    empty_element_tags = None # A tag will be considered an empty-element
                              # tag when and only when it has no contents.
@@ -2,6 +2,7 @@ __all__ = [
    'HTML5TreeBuilder',
    ]

+from pdb import set_trace
 import warnings
 from bs4.builder import (
    PERMISSIVE,
@@ -9,7 +10,10 @@ from bs4.builder import (
    HTML_5,
    HTMLTreeBuilder,
    )
-from bs4.element import NamespacedAttribute
+from bs4.element import (
+    NamespacedAttribute,
+    whitespace_re,
+)
 import html5lib
 from html5lib.constants import namespaces
 from bs4.element import (
@@ -22,11 +26,20 @@ from bs4.element import (
 class HTML5TreeBuilder(HTMLTreeBuilder):
    """Use html5lib to build a tree."""

-    features = ['html5lib', PERMISSIVE, HTML_5, HTML]
+    NAME = "html5lib"

-    def prepare_markup(self, markup, user_specified_encoding):
+    features = [NAME, PERMISSIVE, HTML_5, HTML]
+
+    def prepare_markup(self, markup, user_specified_encoding,
+                       document_declared_encoding=None, exclude_encodings=None):
        # Store the user-specified encoding for use later on.
        self.user_specified_encoding = user_specified_encoding
+
+        # document_declared_encoding and exclude_encodings aren't used
+        # ATM because the html5lib TreeBuilder doesn't use
+        # UnicodeDammit.
+        if exclude_encodings:
+            warnings.warn("You provided a value for exclude_encoding, but the html5lib tree builder doesn't support exclude_encoding.")
        yield (markup, None, None, False)

    # These methods are defined by Beautiful Soup.
@@ -101,7 +114,13 @@ class AttrList(object):
    def __iter__(self):
        return list(self.attrs.items()).__iter__()
    def __setitem__(self, name, value):
-        "set attr", name, value
+        # If this attribute is a multi-valued attribute for this element,
+        # turn its value into a list.
+        list_attr = HTML5TreeBuilder.cdata_list_attributes
+        if (name in list_attr['*']
+            or (self.element.name in list_attr
+                and name in list_attr[self.element.name])):
+            value = whitespace_re.split(value)
        self.element[name] = value
    def items(self):
        return list(self.attrs.items())
@@ -161,6 +180,12 @@ class Element(html5lib.treebuilders._base.Node):
            # immediately after the parent, if it has no children.)
            if self.element.contents:
                most_recent_element = self.element._last_descendant(False)
+            elif self.element.next_element is not None:
+                # Something from further ahead in the parse tree is
+                # being inserted into this earlier element. This is
+                # very annoying because it means an expensive search
+                # for the last element in the tree.
+                most_recent_element = self.soup._last_descendant()
            else:
                most_recent_element = self.element

@@ -172,6 +197,7 @@ class Element(html5lib.treebuilders._base.Node):
        return AttrList(self.element)

    def setAttributes(self, attributes):
+
        if attributes is not None and len(attributes) > 0:

            converted_attributes = []
@@ -218,6 +244,9 @@ class Element(html5lib.treebuilders._base.Node):

    def reparentChildren(self, new_parent):
        """Move all of this tag's children into another tag."""
+        # print "MOVE", self.element.contents
+        # print "FROM", self.element
+        # print "TO", new_parent.element
        element = self.element
        new_parent_element = new_parent.element
        # Determine what this tag's next_element will be once all the children
@@ -236,17 +265,28 @@ class Element(html5lib.treebuilders._base.Node):
            new_parents_last_descendant_next_element = new_parent_element.next_element

        to_append = element.contents
-        append_after = new_parent.element.contents
+        append_after = new_parent_element.contents
        if len(to_append) > 0:
            # Set the first child's previous_element and previous_sibling
            # to elements within the new parent
            first_child = to_append[0]
+            if new_parents_last_descendant:
                first_child.previous_element = new_parents_last_descendant
+            else:
+                first_child.previous_element = new_parent_element
            first_child.previous_sibling = new_parents_last_child
+            if new_parents_last_descendant:
+                new_parents_last_descendant.next_element = first_child
+            else:
+                new_parent_element.next_element = first_child
+            if new_parents_last_child:
+                new_parents_last_child.next_sibling = first_child

            # Fix the last child's next_element and next_sibling
            last_child = to_append[-1]
            last_child.next_element = new_parents_last_descendant_next_element
+            if new_parents_last_descendant_next_element:
+                new_parents_last_descendant_next_element.previous_element = last_child
            last_child.next_sibling = None

        for child in to_append:
@@ -257,6 +297,10 @@ class Element(html5lib.treebuilders._base.Node):
        element.contents = []
        element.next_element = final_next_element

+        # print "DONE WITH MOVE"
+        # print "FROM", self.element
+        # print "TO", new_parent_element
+
    def cloneNode(self):
        tag = self.soup.new_tag(self.element.name, self.namespace)
        node = Element(tag, self.soup, self.namespace)
@@ -268,7 +312,7 @@ class Element(html5lib.treebuilders._base.Node):
        return self.element.contents

    def getNameTuple(self):
-        if self.namespace is None:
+        if self.namespace == None:
            return namespaces["html"], self.name
        else:
            return self.namespace, self.name
@@ -4,10 +4,16 @@ __all__ = [
    'HTMLParserTreeBuilder',
    ]

-from HTMLParser import (
-    HTMLParser,
-    HTMLParseError,
-    )
+from HTMLParser import HTMLParser
+
+try:
+    from HTMLParser import HTMLParseError
+except ImportError, e:
+    # HTMLParseError is removed in Python 3.5. Since it can never be
+    # thrown in 3.5, we can just define our own class as a placeholder.
+    class HTMLParseError(Exception):
+        pass
+
 import sys
 import warnings

@@ -19,10 +25,10 @@ import warnings
 # At the end of this file, we monkeypatch HTMLParser so that
 # strict=True works well on Python 3.2.2.
 major, minor, release = sys.version_info[:3]
-CONSTRUCTOR_TAKES_STRICT = (
-    major > 3
-    or (major == 3 and minor > 2)
-    or (major == 3 and minor == 2 and release >= 3))
+CONSTRUCTOR_TAKES_STRICT = major == 3 and minor == 2 and release >= 3
+CONSTRUCTOR_STRICT_IS_DEPRECATED = major == 3 and minor == 3
+CONSTRUCTOR_TAKES_CONVERT_CHARREFS = major == 3 and minor >= 4
+

 from bs4.element import (
    CData,
@@ -63,7 +69,8 @@ class BeautifulSoupHTMLParser(HTMLParser):

    def handle_charref(self, name):
        # XXX workaround for a bug in HTMLParser. Remove this once
-        # it's fixed.
+        # it's fixed in all supported versions.
+        # http://bugs.python.org/issue13633
        if name.startswith('x'):
            real_name = int(name.lstrip('x'), 16)
        elif name.startswith('X'):
@@ -113,14 +120,6 @@ class BeautifulSoupHTMLParser(HTMLParser):

    def handle_pi(self, data):
        self.soup.endData()
-        if data.endswith("?") and data.lower().startswith("xml"):
-            # "An XHTML processing instruction using the trailing '?'
-            # will cause the '?' to be included in data." - HTMLParser
-            # docs.
-            #
-            # Strip the question mark so we don't end up with two
-            # question marks.
-            data = data[:-1]
        self.soup.handle_data(data)
        self.soup.endData(ProcessingInstruction)

@@ -128,15 +127,19 @@ class BeautifulSoupHTMLParser(HTMLParser):
 class HTMLParserTreeBuilder(HTMLTreeBuilder):

    is_xml = False
-    features = [HTML, STRICT, HTMLPARSER]
+    picklable = True
+    NAME = HTMLPARSER
+    features = [NAME, HTML, STRICT]

    def __init__(self, *args, **kwargs):
-        if CONSTRUCTOR_TAKES_STRICT:
+        if CONSTRUCTOR_TAKES_STRICT and not CONSTRUCTOR_STRICT_IS_DEPRECATED:
            kwargs['strict'] = False
+        if CONSTRUCTOR_TAKES_CONVERT_CHARREFS:
+            kwargs['convert_charrefs'] = False
        self.parser_args = (args, kwargs)

    def prepare_markup(self, markup, user_specified_encoding=None,
-                       document_declared_encoding=None):
+                       document_declared_encoding=None, exclude_encodings=None):
        """
        :return: A 4-tuple (markup, original encoding, encoding
        declared within markup, whether any characters had to be
@@ -147,7 +150,8 @@ class HTMLParserTreeBuilder(HTMLTreeBuilder):
            return

        try_encodings = [user_specified_encoding, document_declared_encoding]
-        dammit = UnicodeDammit(markup, try_encodings, is_html=True)
+        dammit = UnicodeDammit(markup, try_encodings, is_html=True,
+                               exclude_encodings=exclude_encodings)
        yield (dammit.markup, dammit.original_encoding,
               dammit.declared_html_encoding,
               dammit.contains_replacement_characters)
@@ -7,7 +7,12 @@ from io import BytesIO
 from StringIO import StringIO
 import collections
 from lxml import etree
-from bs4.element import Comment, Doctype, NamespacedAttribute
+from bs4.element import (
+    Comment,
+    Doctype,
+    NamespacedAttribute,
+    ProcessingInstruction,
+)
 from bs4.builder import (
    FAST,
    HTML,
@@ -25,8 +30,11 @@ class LXMLTreeBuilderForXML(TreeBuilder):

    is_xml = True

+    NAME = "lxml-xml"
+    ALTERNATE_NAMES = ["xml"]
+
    # Well, it's permissive by XML parser standards.
-    features = [LXML, XML, FAST, PERMISSIVE]
+    features = [NAME, LXML, XML, FAST, PERMISSIVE]

    CHUNK_SIZE = 512

@@ -70,6 +78,7 @@ class LXMLTreeBuilderForXML(TreeBuilder):
            return (None, tag)

    def prepare_markup(self, markup, user_specified_encoding=None,
+                       exclude_encodings=None,
                       document_declared_encoding=None):
        """
        :yield: A series of 4-tuples.
@@ -95,7 +104,8 @@ class LXMLTreeBuilderForXML(TreeBuilder):
        # the document as each one in turn.
        is_html = not self.is_xml
        try_encodings = [user_specified_encoding, document_declared_encoding]
-        detector = EncodingDetector(markup, try_encodings, is_html)
+        detector = EncodingDetector(
+            markup, try_encodings, is_html, exclude_encodings)
        for encoding in detector.encodings:
            yield (detector.markup, encoding, document_declared_encoding, False)

@@ -189,7 +199,9 @@ class LXMLTreeBuilderForXML(TreeBuilder):
            self.nsmaps.pop()

    def pi(self, target, data):
-        pass
+        self.soup.endData()
+        self.soup.handle_data(target + ' ' + data)
+        self.soup.endData(ProcessingInstruction)

    def data(self, content):
        self.soup.handle_data(content)
@@ -212,7 +224,10 @@ class LXMLTreeBuilderForXML(TreeBuilder):

 class LXMLTreeBuilder(HTMLTreeBuilder, LXMLTreeBuilderForXML):

-    features = [LXML, HTML, FAST, PERMISSIVE]
+    NAME = LXML
+    ALTERNATE_NAMES = ["lxml-html"]
+
+    features = ALTERNATE_NAMES + [NAME, HTML, FAST, PERMISSIVE]
    is_xml = False

    def default_parser(self, encoding):
@@ -3,10 +3,11 @@

 This library converts a bytestream to Unicode through any means
 necessary. It is heavily based on code from Mark Pilgrim's Universal
-Feed Parser. It works best on XML and XML, but it does not rewrite the
+Feed Parser. It works best on XML and HTML, but it does not rewrite the
 XML or HTML to reflect a new encoding; that's the tree builder's job.
 """

+from pdb import set_trace
 import codecs
 from htmlentitydefs import codepoint2name
 import re
@@ -212,8 +213,11 @@ class EncodingDetector:

    5. Windows-1252.
    """
-    def __init__(self, markup, override_encodings=None, is_html=False):
+    def __init__(self, markup, override_encodings=None, is_html=False,
+                 exclude_encodings=None):
        self.override_encodings = override_encodings or []
+        exclude_encodings = exclude_encodings or []
+        self.exclude_encodings = set([x.lower() for x in exclude_encodings])
        self.chardet_encoding = None
        self.is_html = is_html
        self.declared_encoding = None
@@ -224,6 +228,8 @@ class EncodingDetector:
    def _usable(self, encoding, tried):
        if encoding is not None:
            encoding = encoding.lower()
+            if encoding in self.exclude_encodings:
+                return False
            if encoding not in tried:
                tried.add(encoding)
                return True
@@ -266,6 +272,9 @@ class EncodingDetector:
    def strip_byte_order_mark(cls, data):
        """If a byte-order mark is present, strip it and return the encoding it implies."""
        encoding = None
+        if isinstance(data, unicode):
+            # Unicode data cannot have a byte-order mark.
+            return data, encoding
        if (len(data) >= 4) and (data[:2] == b'\xfe\xff') \
               and (data[2:4] != '\x00\x00'):
            encoding = 'utf-16be'
@@ -306,7 +315,7 @@ class EncodingDetector:
            declared_encoding_match = html_meta_re.search(markup, endpos=html_endpos)
        if declared_encoding_match is not None:
            declared_encoding = declared_encoding_match.groups()[0].decode(
-                'ascii')
+                'ascii', 'replace')
        if declared_encoding:
            return declared_encoding.lower()
        return None
@@ -331,13 +340,14 @@ class UnicodeDammit:
        ]

    def __init__(self, markup, override_encodings=[],
-                 smart_quotes_to=None, is_html=False):
+                 smart_quotes_to=None, is_html=False, exclude_encodings=[]):
        self.smart_quotes_to = smart_quotes_to
        self.tried_encodings = []
        self.contains_replacement_characters = False
        self.is_html = is_html

-        self.detector = EncodingDetector(markup, override_encodings, is_html)
+        self.detector = EncodingDetector(
+            markup, override_encodings, is_html, exclude_encodings)

        # Short-circuit if the data is in Unicode to begin with.
        if isinstance(markup, unicode) or markup == '':
@@ -33,12 +33,21 @@ def diagnose(data):

    if 'lxml' in basic_parsers:
        basic_parsers.append(["lxml", "xml"])
+        try:
            from lxml import etree
            print "Found lxml version %s" % ".".join(map(str,etree.LXML_VERSION))
+        except ImportError, e:
+            print (
+                "lxml is not installed or couldn't be imported.")
+

    if 'html5lib' in basic_parsers:
+        try:
            import html5lib
            print "Found html5lib version %s" % html5lib.__version__
+        except ImportError, e:
+            print (
+                "html5lib is not installed or couldn't be imported.")

    if hasattr(data, 'read'):
        data = data.read()
@@ -1,3 +1,4 @@
+from pdb import set_trace
 import collections
 import re
 import sys
@@ -185,24 +186,40 @@ class PageElement(object):
            return self.HTML_FORMATTERS.get(
                name, HTMLAwareEntitySubstitution.substitute_xml)

-    def setup(self, parent=None, previous_element=None):
+    def setup(self, parent=None, previous_element=None, next_element=None,
+              previous_sibling=None, next_sibling=None):
        """Sets up the initial relations between this element and
        other elements."""
        self.parent = parent
+
        self.previous_element = previous_element
        if previous_element is not None:
            self.previous_element.next_element = self
-        self.next_element = None
-        self.previous_sibling = None
-        self.next_sibling = None
-        if self.parent is not None and self.parent.contents:
-            self.previous_sibling = self.parent.contents[-1]
+
+        self.next_element = next_element
+        if self.next_element:
+            self.next_element.previous_element = self
+
+        self.next_sibling = next_sibling
+        if self.next_sibling:
+            self.next_sibling.previous_sibling = self
+
+        if (not previous_sibling
+            and self.parent is not None and self.parent.contents):
+            previous_sibling = self.parent.contents[-1]
+
+        self.previous_sibling = previous_sibling
+        if previous_sibling:
            self.previous_sibling.next_sibling = self

    nextSibling = _alias("next_sibling")  # BS3
    previousSibling = _alias("previous_sibling")  # BS3

    def replace_with(self, replace_with):
+        if not self.parent:
+            raise ValueError(
+                "Cannot replace one element with another when the"
+                "element to be replaced is not part of a tree.")
        if replace_with is self:
            return
        if replace_with is self.parent:
@@ -216,6 +233,10 @@ class PageElement(object):

    def unwrap(self):
        my_parent = self.parent
+        if not self.parent:
+            raise ValueError(
+                "Cannot replace an element with its contents when that"
+                "element is not part of a tree.")
        my_index = self.parent.index(self)
        self.extract()
        for child in reversed(self.contents[:]):
@@ -240,17 +261,20 @@ class PageElement(object):
        last_child = self._last_descendant()
        next_element = last_child.next_element

-        if self.previous_element is not None:
+        if (self.previous_element is not None and
+            self.previous_element != next_element):
            self.previous_element.next_element = next_element
-        if next_element is not None:
+        if next_element is not None and next_element != self.previous_element:
            next_element.previous_element = self.previous_element
        self.previous_element = None
        last_child.next_element = None

        self.parent = None
-        if self.previous_sibling is not None:
+        if (self.previous_sibling is not None
+            and self.previous_sibling != self.next_sibling):
            self.previous_sibling.next_sibling = self.next_sibling
-        if self.next_sibling is not None:
+        if (self.next_sibling is not None
+            and self.next_sibling != self.previous_sibling):
            self.next_sibling.previous_sibling = self.previous_sibling
        self.previous_sibling = self.next_sibling = None
        return self
@@ -478,6 +502,10 @@ class PageElement(object):
    def _find_all(self, name, attrs, text, limit, generator, **kwargs):
        "Iterates over a generator looking for things that match."

+        if text is None and 'string' in kwargs:
+            text = kwargs['string']
+            del kwargs['string']
+
        if isinstance(name, SoupStrainer):
            strainer = name
        else:
@@ -548,17 +576,17 @@ class PageElement(object):

    # Methods for supporting CSS selectors.

-    tag_name_re = re.compile('^[a-z0-9]+$')
+    tag_name_re = re.compile('^[a-zA-Z0-9][-.a-zA-Z0-9:_]*$')

-    # /^(\w+)\[(\w+)([=~\|\^\$\*]?)=?"?([^\]"]*)"?\]$/
-    #   \---/  \---/\-------------/    \-------/
+    # /^([a-zA-Z0-9][-.a-zA-Z0-9:_]*)\[(\w+)([=~\|\^\$\*]?)=?"?([^\]"]*)"?\]$/
+    #   \---------------------------/  \---/\-------------/    \-------/
    #     |                              |         |               |
    #     |                              |         |           The value
    #     |                              |    ~,|,^,$,* or =
    #     |                           Attribute
    #    Tag
    attribselect_re = re.compile(
-        r'^(?P<tag>\w+)?\[(?P<attribute>\w+)(?P<operator>[=~\|\^\$\*]?)' +
+        r'^(?P<tag>[a-zA-Z0-9][-.a-zA-Z0-9:_]*)?\[(?P<attribute>[\w-]+)(?P<operator>[=~\|\^\$\*]?)' +
        r'=?"?(?P<value>[^\]"]*)"?\]$'
        )

@@ -654,11 +682,17 @@ class NavigableString(unicode, PageElement):
        how to handle non-ASCII characters.
        """
        if isinstance(value, unicode):
-            return unicode.__new__(cls, value)
-        return unicode.__new__(cls, value, DEFAULT_OUTPUT_ENCODING)
+            u = unicode.__new__(cls, value)
+        else:
+            u = unicode.__new__(cls, value, DEFAULT_OUTPUT_ENCODING)
+        u.setup()
+        return u

    def __copy__(self):
-        return self
+        """A copy of a NavigableString has the same contents and class
+        as the original, but it is not connected to the parse tree.
+        """
+        return type(self)(self)

    def __getnewargs__(self):
        return (unicode(self),)
@@ -707,7 +741,7 @@ class CData(PreformattedString):
 class ProcessingInstruction(PreformattedString):

    PREFIX = u'<?'
-    SUFFIX = u'?>'
+    SUFFIX = u'>'

 class Comment(PreformattedString):

@@ -759,11 +793,14 @@ class Tag(PageElement):
        self.prefix = prefix
        if attrs is None:
            attrs = {}
-        elif attrs and builder.cdata_list_attributes:
+        elif attrs:
+            if builder is not None and builder.cdata_list_attributes:
                attrs = builder._replace_cdata_list_attribute_values(
                    self.name, attrs)
            else:
                attrs = dict(attrs)
+        else:
+            attrs = dict(attrs)
        self.attrs = attrs
        self.contents = []
        self.setup(parent, previous)
@@ -778,6 +815,18 @@ class Tag(PageElement):

    parserClass = _alias("parser_class")  # BS3

+    def __copy__(self):
+        """A copy of a Tag is a new Tag, unconnected to the parse tree.
+        Its contents are a copy of the old Tag's contents.
+        """
+        clone = type(self)(None, self.builder, self.name, self.namespace,
+                           self.nsprefix, self.attrs)
+        for attr in ('can_be_empty_element', 'hidden'):
+            setattr(clone, attr, getattr(self, attr))
+        for child in self.contents:
+            clone.append(child.__copy__())
+        return clone
+
    @property
    def is_empty_element(self):
        """Is this tag an empty-element tag? (aka a self-closing tag)
@@ -971,14 +1020,24 @@ class Tag(PageElement):
        as defined in __eq__."""
        return not self == other

-    def __repr__(self, encoding=DEFAULT_OUTPUT_ENCODING):
+    def __repr__(self, encoding="unicode-escape"):
        """Renders this tag as a string."""
+        if PY3K:
+            # "The return value must be a string object", i.e. Unicode
+            return self.decode()
+        else:
+            # "The return value must be a string object", i.e. a bytestring.
+            # By convention, the return value of __repr__ should also be
+            # an ASCII string.
            return self.encode(encoding)

    def __unicode__(self):
        return self.decode()

    def __str__(self):
+        if PY3K:
+            return self.decode()
+        else:
            return self.encode()

    if PY3K:
@@ -1103,12 +1162,18 @@ class Tag(PageElement):
                       formatter="minimal"):
        """Renders the contents of this tag as a Unicode string.

+        :param indent_level: Each line of the rendering will be
+           indented this many spaces.
+
        :param eventual_encoding: The tag is destined to be
           encoded into this encoding. This method is _not_
           responsible for performing that encoding. This information
           is passed in so that it can be substituted in if the
           document contains a <META> tag that mentions the document's
           encoding.
+
+        :param formatter: The output formatter responsible for converting
+           entities to Unicode characters.
        """
        # First off, turn a string formatter into a function. This
        # will stop the lookup from happening over and over again.
@@ -1137,7 +1202,17 @@ class Tag(PageElement):
    def encode_contents(
        self, indent_level=None, encoding=DEFAULT_OUTPUT_ENCODING,
        formatter="minimal"):
-        """Renders the contents of this tag as a bytestring."""
+        """Renders the contents of this tag as a bytestring.
+
+        :param indent_level: Each line of the rendering will be
+           indented this many spaces.
+
+        :param eventual_encoding: The bytestring will be in this encoding.
+
+        :param formatter: The output formatter responsible for converting
+           entities to Unicode characters.
+        """
+
        contents = self.decode_contents(indent_level, encoding, formatter)
        return contents.encode(encoding)

@@ -1201,26 +1276,49 @@ class Tag(PageElement):

    _selector_combinators = ['>', '+', '~']
    _select_debug = False
-    def select(self, selector, _candidate_generator=None):
+    def select_one(self, selector):
        """Perform a CSS selection operation on the current element."""
-        tokens = selector.split()
+        value = self.select(selector, limit=1)
+        if value:
+            return value[0]
+        return None
+
+    def select(self, selector, _candidate_generator=None, limit=None):
+        """Perform a CSS selection operation on the current element."""
+
+        # Remove whitespace directly after the grouping operator ','
+        # then split into tokens.
+        tokens = re.sub(',[\s]*',',', selector).split()
        current_context = [self]

        if tokens[-1] in self._selector_combinators:
            raise ValueError(
                'Final combinator "%s" is missing an argument.' % tokens[-1])
+
        if self._select_debug:
            print 'Running CSS selector "%s"' % selector
-        for index, token in enumerate(tokens):
-            if self._select_debug:
-                print ' Considering token "%s"' % token
-            recursive_candidate_generator = None
-            tag_name = None
+
+        for index, token_group in enumerate(tokens):
+            new_context = []
+            new_context_ids = set([])
+
+            # Grouping selectors, ie: p,a
+            grouped_tokens = token_group.split(',')
+            if '' in grouped_tokens:
+                raise ValueError('Invalid group selection syntax: %s' % token_group)
+
            if tokens[index-1] in self._selector_combinators:
                # This token was consumed by the previous combinator. Skip it.
                if self._select_debug:
                    print '  Token was consumed by the previous combinator.'
                continue
+
+            for token in grouped_tokens:
+                if self._select_debug:
+                    print ' Considering token "%s"' % token
+                recursive_candidate_generator = None
+                tag_name = None
+
                # Each operation corresponds to a checker function, a rule
                # for determining whether a candidate matches the
                # selector. Candidates are generated by the active
@@ -1256,7 +1354,10 @@ class Tag(PageElement):
                            "A pseudo-class must be prefixed with a tag name.")
                    pseudo_attributes = re.match('([a-zA-Z\d-]+)\(([a-zA-Z\d]+)\)', pseudo)
                    found = []
-                if pseudo_attributes is not None:
+                    if pseudo_attributes is None:
+                        pseudo_type = pseudo
+                        pseudo_value = None
+                    else:
                        pseudo_type, pseudo_value = pseudo_attributes.groups()
                    if pseudo_type == 'nth-of-type':
                        try:
@@ -1311,7 +1412,6 @@ class Tag(PageElement):
                else:
                    raise ValueError(
                        'Unsupported or invalid CSS selector: "%s"' % token)
-
                if recursive_candidate_generator:
                    # This happens when the selector looks like  "> foo".
                    #
@@ -1361,8 +1461,7 @@ class Tag(PageElement):
                else:
                    _use_candidate_generator = _candidate_generator

-            new_context = []
-            new_context_ids = set([])
+                count = 0
                for tag in current_context:
                    if self._select_debug:
                        print "    Running candidate generator on %s %s" % (
@@ -1387,9 +1486,12 @@ class Tag(PageElement):
                                # don't include it in the context more than once.
                                new_context.append(candidate)
                                new_context_ids.add(id(candidate))
+                                if limit and len(new_context) >= limit:
+                                    break
                        elif self._select_debug:
                            print "     FAILURE %s %s" % (candidate.name, repr(candidate.attrs))

+
            current_context = new_context

        if self._select_debug:
@@ -1,5 +1,6 @@
 """Helper classes for tests."""

+import pickle
 import copy
 import functools
 import unittest
@@ -43,6 +44,16 @@ class SoupTest(unittest.TestCase):

        self.assertEqual(obj.decode(), self.document_for(compare_parsed_to))

+    def assertConnectedness(self, element):
+        """Ensure that next_element and previous_element are properly
+        set for all descendants of the given element.
+        """
+        earlier = None
+        for e in element.descendants:
+            if earlier:
+                self.assertEqual(e, earlier.next_element)
+                self.assertEqual(earlier, e.previous_element)
+            earlier = e

 class HTMLTreeBuilderSmokeTest(object):

@@ -54,6 +65,15 @@ class HTMLTreeBuilderSmokeTest(object):
    markup in these tests, there's not much room for interpretation.
    """

+    def test_pickle_and_unpickle_identity(self):
+        # Pickling a tree, then unpickling it, yields a tree identical
+        # to the original.
+        tree = self.soup("<a><b>foo</a>")
+        dumped = pickle.dumps(tree, 2)
+        loaded = pickle.loads(dumped)
+        self.assertEqual(loaded.__class__, BeautifulSoup)
+        self.assertEqual(loaded.decode(), tree.decode())
+
    def assertDoctypeHandled(self, doctype_fragment):
        """Assert that a given doctype string is handled correctly."""
        doctype_str, soup = self._document_with_doctype(doctype_fragment)
@@ -114,6 +134,11 @@ class HTMLTreeBuilderSmokeTest(object):
            soup.encode("utf-8").replace(b"\n", b""),
            markup.replace(b"\n", b""))

+    def test_processing_instruction(self):
+        markup = b"""<?PITarget PIContent?>"""
+        soup = self.soup(markup)
+        self.assertEqual(markup, soup.encode("utf8"))
+
    def test_deepcopy(self):
        """Make sure you can copy the tree builder.

@@ -155,6 +180,23 @@ class HTMLTreeBuilderSmokeTest(object):
    def test_nested_formatting_elements(self):
        self.assertSoupEquals("<em><em></em></em>")

+    def test_double_head(self):
+        html = '''<!DOCTYPE html>
+<html>
+<head>
+<title>Ordinary HEAD element test</title>
+</head>
+<script type="text/javascript">
+alert("Help!");
+</script>
+<body>
+Hello, world!
+</body>
+</html>
+'''
+        soup = self.soup(html)
+        self.assertEqual("text/javascript", soup.find('script')['type'])
+
    def test_comment(self):
        # Comments are represented as Comment objects.
        markup = "<p>foo<!--foobar-->baz</p>"
@@ -221,6 +263,14 @@ class HTMLTreeBuilderSmokeTest(object):
        soup = self.soup(markup)
        self.assertEqual(["css"], soup.div.div['class'])

+    def test_multivalued_attribute_on_html(self):
+        # html5lib uses a different API to set the attributes ot the
+        # <html> tag. This has caused problems with multivalued
+        # attributes.
+        markup = '<html class="a b"></html>'
+        soup = self.soup(markup)
+        self.assertEqual(["a", "b"], soup.html['class'])
+
    def test_angle_brackets_in_attribute_values_are_escaped(self):
        self.assertSoupEquals('<a b="<a>"></a>', '<a b="&lt;a&gt;"></a>')

@@ -253,6 +303,35 @@ class HTMLTreeBuilderSmokeTest(object):
        soup = self.soup("<html><h2>\nfoo</h2><p></p></html>")
        self.assertEqual("p", soup.h2.string.next_element.name)
        self.assertEqual("p", soup.p.name)
+        self.assertConnectedness(soup)
+
+    def test_head_tag_between_head_and_body(self):
+        "Prevent recurrence of a bug in the html5lib treebuilder."
+        content = """<html><head></head>
+  <link></link>
+  <body>foo</body>
+</html>
+"""
+        soup = self.soup(content)
+        self.assertNotEqual(None, soup.html.body)
+        self.assertConnectedness(soup)
+
+    def test_multiple_copies_of_a_tag(self):
+        "Prevent recurrence of a bug in the html5lib treebuilder."
+        content = """<!DOCTYPE html>
+<html>
+ <body>
+   <article id="a" >
+   <div><a href="1"></div>
+   <footer>
+     <a href="2"></a>
+   </footer>
+  </article>
+  </body>
+</html>
+"""
+        soup = self.soup(content)
+        self.assertConnectedness(soup.article)

    def test_basic_namespaces(self):
        """Parsers don't need to *understand* namespaces, but at the
@@ -463,6 +542,15 @@ class HTMLTreeBuilderSmokeTest(object):

 class XMLTreeBuilderSmokeTest(object):

+    def test_pickle_and_unpickle_identity(self):
+        # Pickling a tree, then unpickling it, yields a tree identical
+        # to the original.
+        tree = self.soup("<a><b>foo</a>")
+        dumped = pickle.dumps(tree, 2)
+        loaded = pickle.loads(dumped)
+        self.assertEqual(loaded.__class__, BeautifulSoup)
+        self.assertEqual(loaded.decode(), tree.decode())
+
    def test_docstring_generated(self):
        soup = self.soup("<root/>")
        self.assertEqual(
@@ -485,7 +573,7 @@ class XMLTreeBuilderSmokeTest(object):
  <script type="text/javascript">
  </script>
 """
-        soup = BeautifulSoup(doc, "xml")
+        soup = BeautifulSoup(doc, "lxml-xml")
        # lxml would have stripped this while parsing, but we can add
        # it later.
        soup.script.string = 'console.log("< < hey > > ");'
@@ -20,4 +20,6 @@ from .serializer import serialize

 __all__ = ["HTMLParser", "parse", "parseFragment", "getTreeBuilder",
           "getTreeWalker", "serialize"]
-__version__ = "0.999"
+
+# this has to be at the top level, see how setup.py parses this
+__version__ = "0.999999"
@@ -1,292 +1,290 @@
 from __future__ import absolute_import, division, unicode_literals

 import string
-import gettext
-_ = gettext.gettext

 EOF = None

 E = {
    "null-character":
-        _("Null character in input stream, replaced with U+FFFD."),
+        "Null character in input stream, replaced with U+FFFD.",
    "invalid-codepoint":
-        _("Invalid codepoint in stream."),
+        "Invalid codepoint in stream.",
    "incorrectly-placed-solidus":
-        _("Solidus (/) incorrectly placed in tag."),
+        "Solidus (/) incorrectly placed in tag.",
    "incorrect-cr-newline-entity":
-        _("Incorrect CR newline entity, replaced with LF."),
+        "Incorrect CR newline entity, replaced with LF.",
    "illegal-windows-1252-entity":
-        _("Entity used with illegal number (windows-1252 reference)."),
+        "Entity used with illegal number (windows-1252 reference).",
    "cant-convert-numeric-entity":
-        _("Numeric entity couldn't be converted to character "
-          "(codepoint U+%(charAsInt)08x)."),
+        "Numeric entity couldn't be converted to character "
+        "(codepoint U+%(charAsInt)08x).",
    "illegal-codepoint-for-numeric-entity":
-        _("Numeric entity represents an illegal codepoint: "
-          "U+%(charAsInt)08x."),
+        "Numeric entity represents an illegal codepoint: "
+        "U+%(charAsInt)08x.",
    "numeric-entity-without-semicolon":
-        _("Numeric entity didn't end with ';'."),
+        "Numeric entity didn't end with ';'.",
    "expected-numeric-entity-but-got-eof":
-        _("Numeric entity expected. Got end of file instead."),
+        "Numeric entity expected. Got end of file instead.",
    "expected-numeric-entity":
-        _("Numeric entity expected but none found."),
+        "Numeric entity expected but none found.",
    "named-entity-without-semicolon":
-        _("Named entity didn't end with ';'."),
+        "Named entity didn't end with ';'.",
    "expected-named-entity":
-        _("Named entity expected. Got none."),
+        "Named entity expected. Got none.",
    "attributes-in-end-tag":
-        _("End tag contains unexpected attributes."),
+        "End tag contains unexpected attributes.",
    'self-closing-flag-on-end-tag':
-        _("End tag contains unexpected self-closing flag."),
+        "End tag contains unexpected self-closing flag.",
    "expected-tag-name-but-got-right-bracket":
-        _("Expected tag name. Got '>' instead."),
+        "Expected tag name. Got '>' instead.",
    "expected-tag-name-but-got-question-mark":
-        _("Expected tag name. Got '?' instead. (HTML doesn't "
-          "support processing instructions.)"),
+        "Expected tag name. Got '?' instead. (HTML doesn't "
+        "support processing instructions.)",
    "expected-tag-name":
-        _("Expected tag name. Got something else instead"),
+        "Expected tag name. Got something else instead",
    "expected-closing-tag-but-got-right-bracket":
-        _("Expected closing tag. Got '>' instead. Ignoring '</>'."),
+        "Expected closing tag. Got '>' instead. Ignoring '</>'.",
    "expected-closing-tag-but-got-eof":
-        _("Expected closing tag. Unexpected end of file."),
+        "Expected closing tag. Unexpected end of file.",
    "expected-closing-tag-but-got-char":
-        _("Expected closing tag. Unexpected character '%(data)s' found."),
+        "Expected closing tag. Unexpected character '%(data)s' found.",
    "eof-in-tag-name":
-        _("Unexpected end of file in the tag name."),
+        "Unexpected end of file in the tag name.",
    "expected-attribute-name-but-got-eof":
-        _("Unexpected end of file. Expected attribute name instead."),
+        "Unexpected end of file. Expected attribute name instead.",
    "eof-in-attribute-name":
-        _("Unexpected end of file in attribute name."),
+        "Unexpected end of file in attribute name.",
    "invalid-character-in-attribute-name":
-        _("Invalid character in attribute name"),
+        "Invalid character in attribute name",
    "duplicate-attribute":
-        _("Dropped duplicate attribute on tag."),
+        "Dropped duplicate attribute on tag.",
    "expected-end-of-tag-name-but-got-eof":
-        _("Unexpected end of file. Expected = or end of tag."),
+        "Unexpected end of file. Expected = or end of tag.",
    "expected-attribute-value-but-got-eof":
-        _("Unexpected end of file. Expected attribute value."),
+        "Unexpected end of file. Expected attribute value.",
    "expected-attribute-value-but-got-right-bracket":
-        _("Expected attribute value. Got '>' instead."),
+        "Expected attribute value. Got '>' instead.",
    'equals-in-unquoted-attribute-value':
-        _("Unexpected = in unquoted attribute"),
+        "Unexpected = in unquoted attribute",
    'unexpected-character-in-unquoted-attribute-value':
-        _("Unexpected character in unquoted attribute"),
+        "Unexpected character in unquoted attribute",
    "invalid-character-after-attribute-name":
-        _("Unexpected character after attribute name."),
+        "Unexpected character after attribute name.",
    "unexpected-character-after-attribute-value":
-        _("Unexpected character after attribute value."),
+        "Unexpected character after attribute value.",
    "eof-in-attribute-value-double-quote":
-        _("Unexpected end of file in attribute value (\")."),
+        "Unexpected end of file in attribute value (\").",
    "eof-in-attribute-value-single-quote":
-        _("Unexpected end of file in attribute value (')."),
+        "Unexpected end of file in attribute value (').",
    "eof-in-attribute-value-no-quotes":
-        _("Unexpected end of file in attribute value."),
+        "Unexpected end of file in attribute value.",
    "unexpected-EOF-after-solidus-in-tag":
-        _("Unexpected end of file in tag. Expected >"),
+        "Unexpected end of file in tag. Expected >",
    "unexpected-character-after-solidus-in-tag":
-        _("Unexpected character after / in tag. Expected >"),
+        "Unexpected character after / in tag. Expected >",
    "expected-dashes-or-doctype":
-        _("Expected '--' or 'DOCTYPE'. Not found."),
+        "Expected '--' or 'DOCTYPE'. Not found.",
    "unexpected-bang-after-double-dash-in-comment":
-        _("Unexpected ! after -- in comment"),
+        "Unexpected ! after -- in comment",
    "unexpected-space-after-double-dash-in-comment":
-        _("Unexpected space after -- in comment"),
+        "Unexpected space after -- in comment",
    "incorrect-comment":
-        _("Incorrect comment."),
+        "Incorrect comment.",
    "eof-in-comment":
-        _("Unexpected end of file in comment."),
+        "Unexpected end of file in comment.",
    "eof-in-comment-end-dash":
-        _("Unexpected end of file in comment (-)"),
+        "Unexpected end of file in comment (-)",
    "unexpected-dash-after-double-dash-in-comment":
-        _("Unexpected '-' after '--' found in comment."),
+        "Unexpected '-' after '--' found in comment.",
    "eof-in-comment-double-dash":
-        _("Unexpected end of file in comment (--)."),
+        "Unexpected end of file in comment (--).",
    "eof-in-comment-end-space-state":
-        _("Unexpected end of file in comment."),
+        "Unexpected end of file in comment.",
    "eof-in-comment-end-bang-state":
-        _("Unexpected end of file in comment."),
+        "Unexpected end of file in comment.",
    "unexpected-char-in-comment":
-        _("Unexpected character in comment found."),
+        "Unexpected character in comment found.",
    "need-space-after-doctype":
-        _("No space after literal string 'DOCTYPE'."),
+        "No space after literal string 'DOCTYPE'.",
    "expected-doctype-name-but-got-right-bracket":
-        _("Unexpected > character. Expected DOCTYPE name."),
+        "Unexpected > character. Expected DOCTYPE name.",
    "expected-doctype-name-but-got-eof":
-        _("Unexpected end of file. Expected DOCTYPE name."),
+        "Unexpected end of file. Expected DOCTYPE name.",
    "eof-in-doctype-name":
-        _("Unexpected end of file in DOCTYPE name."),
+        "Unexpected end of file in DOCTYPE name.",
    "eof-in-doctype":
-        _("Unexpected end of file in DOCTYPE."),
+        "Unexpected end of file in DOCTYPE.",
    "expected-space-or-right-bracket-in-doctype":
-        _("Expected space or '>'. Got '%(data)s'"),
+        "Expected space or '>'. Got '%(data)s'",
    "unexpected-end-of-doctype":
-        _("Unexpected end of DOCTYPE."),
+        "Unexpected end of DOCTYPE.",
    "unexpected-char-in-doctype":
-        _("Unexpected character in DOCTYPE."),
+        "Unexpected character in DOCTYPE.",
    "eof-in-innerhtml":
-        _("XXX innerHTML EOF"),
+        "XXX innerHTML EOF",
    "unexpected-doctype":
-        _("Unexpected DOCTYPE. Ignored."),
+        "Unexpected DOCTYPE. Ignored.",
    "non-html-root":
-        _("html needs to be the first start tag."),
+        "html needs to be the first start tag.",
    "expected-doctype-but-got-eof":
-        _("Unexpected End of file. Expected DOCTYPE."),
+        "Unexpected End of file. Expected DOCTYPE.",
    "unknown-doctype":
-        _("Erroneous DOCTYPE."),
+        "Erroneous DOCTYPE.",
    "expected-doctype-but-got-chars":
-        _("Unexpected non-space characters. Expected DOCTYPE."),
+        "Unexpected non-space characters. Expected DOCTYPE.",
    "expected-doctype-but-got-start-tag":
-        _("Unexpected start tag (%(name)s). Expected DOCTYPE."),
+        "Unexpected start tag (%(name)s). Expected DOCTYPE.",
    "expected-doctype-but-got-end-tag":
-        _("Unexpected end tag (%(name)s). Expected DOCTYPE."),
+        "Unexpected end tag (%(name)s). Expected DOCTYPE.",
    "end-tag-after-implied-root":
-        _("Unexpected end tag (%(name)s) after the (implied) root element."),
+        "Unexpected end tag (%(name)s) after the (implied) root element.",
    "expected-named-closing-tag-but-got-eof":
-        _("Unexpected end of file. Expected end tag (%(name)s)."),
+        "Unexpected end of file. Expected end tag (%(name)s).",
    "two-heads-are-not-better-than-one":
-        _("Unexpected start tag head in existing head. Ignored."),
+        "Unexpected start tag head in existing head. Ignored.",
    "unexpected-end-tag":
-        _("Unexpected end tag (%(name)s). Ignored."),
+        "Unexpected end tag (%(name)s). Ignored.",
    "unexpected-start-tag-out-of-my-head":
-        _("Unexpected start tag (%(name)s) that can be in head. Moved."),
+        "Unexpected start tag (%(name)s) that can be in head. Moved.",
    "unexpected-start-tag":
-        _("Unexpected start tag (%(name)s)."),
+        "Unexpected start tag (%(name)s).",
    "missing-end-tag":
-        _("Missing end tag (%(name)s)."),
+        "Missing end tag (%(name)s).",
    "missing-end-tags":
-        _("Missing end tags (%(name)s)."),
+        "Missing end tags (%(name)s).",
    "unexpected-start-tag-implies-end-tag":
-        _("Unexpected start tag (%(startName)s) "
-          "implies end tag (%(endName)s)."),
+        "Unexpected start tag (%(startName)s) "
+        "implies end tag (%(endName)s).",
    "unexpected-start-tag-treated-as":
-        _("Unexpected start tag (%(originalName)s). Treated as %(newName)s."),
+        "Unexpected start tag (%(originalName)s). Treated as %(newName)s.",
    "deprecated-tag":
-        _("Unexpected start tag %(name)s. Don't use it!"),
+        "Unexpected start tag %(name)s. Don't use it!",
    "unexpected-start-tag-ignored":
-        _("Unexpected start tag %(name)s. Ignored."),
+        "Unexpected start tag %(name)s. Ignored.",
    "expected-one-end-tag-but-got-another":
-        _("Unexpected end tag (%(gotName)s). "
-          "Missing end tag (%(expectedName)s)."),
+        "Unexpected end tag (%(gotName)s). "
+        "Missing end tag (%(expectedName)s).",
    "end-tag-too-early":
-        _("End tag (%(name)s) seen too early. Expected other end tag."),
+        "End tag (%(name)s) seen too early. Expected other end tag.",
    "end-tag-too-early-named":
-        _("Unexpected end tag (%(gotName)s). Expected end tag (%(expectedName)s)."),
+        "Unexpected end tag (%(gotName)s). Expected end tag (%(expectedName)s).",
    "end-tag-too-early-ignored":
-        _("End tag (%(name)s) seen too early. Ignored."),
+        "End tag (%(name)s) seen too early. Ignored.",
    "adoption-agency-1.1":
-        _("End tag (%(name)s) violates step 1, "
-          "paragraph 1 of the adoption agency algorithm."),
+        "End tag (%(name)s) violates step 1, "
+        "paragraph 1 of the adoption agency algorithm.",
    "adoption-agency-1.2":
-        _("End tag (%(name)s) violates step 1, "
-          "paragraph 2 of the adoption agency algorithm."),
+        "End tag (%(name)s) violates step 1, "
+        "paragraph 2 of the adoption agency algorithm.",
    "adoption-agency-1.3":
-        _("End tag (%(name)s) violates step 1, "
-          "paragraph 3 of the adoption agency algorithm."),
+        "End tag (%(name)s) violates step 1, "
+        "paragraph 3 of the adoption agency algorithm.",
    "adoption-agency-4.4":
-        _("End tag (%(name)s) violates step 4, "
-          "paragraph 4 of the adoption agency algorithm."),
+        "End tag (%(name)s) violates step 4, "
+        "paragraph 4 of the adoption agency algorithm.",
    "unexpected-end-tag-treated-as":
-        _("Unexpected end tag (%(originalName)s). Treated as %(newName)s."),
+        "Unexpected end tag (%(originalName)s). Treated as %(newName)s.",
    "no-end-tag":
-        _("This element (%(name)s) has no end tag."),
+        "This element (%(name)s) has no end tag.",
    "unexpected-implied-end-tag-in-table":
-        _("Unexpected implied end tag (%(name)s) in the table phase."),
+        "Unexpected implied end tag (%(name)s) in the table phase.",
    "unexpected-implied-end-tag-in-table-body":
-        _("Unexpected implied end tag (%(name)s) in the table body phase."),
+        "Unexpected implied end tag (%(name)s) in the table body phase.",
    "unexpected-char-implies-table-voodoo":
-        _("Unexpected non-space characters in "
-          "table context caused voodoo mode."),
+        "Unexpected non-space characters in "
+        "table context caused voodoo mode.",
    "unexpected-hidden-input-in-table":
-        _("Unexpected input with type hidden in table context."),
+        "Unexpected input with type hidden in table context.",
    "unexpected-form-in-table":
-        _("Unexpected form in table context."),
+        "Unexpected form in table context.",
    "unexpected-start-tag-implies-table-voodoo":
-        _("Unexpected start tag (%(name)s) in "
-          "table context caused voodoo mode."),
+        "Unexpected start tag (%(name)s) in "
+        "table context caused voodoo mode.",
    "unexpected-end-tag-implies-table-voodoo":
-        _("Unexpected end tag (%(name)s) in "
-          "table context caused voodoo mode."),
+        "Unexpected end tag (%(name)s) in "
+        "table context caused voodoo mode.",
    "unexpected-cell-in-table-body":
-        _("Unexpected table cell start tag (%(name)s) "
-          "in the table body phase."),
+        "Unexpected table cell start tag (%(name)s) "
+        "in the table body phase.",
    "unexpected-cell-end-tag":
-        _("Got table cell end tag (%(name)s) "
-          "while required end tags are missing."),
+        "Got table cell end tag (%(name)s) "
+        "while required end tags are missing.",
    "unexpected-end-tag-in-table-body":
-        _("Unexpected end tag (%(name)s) in the table body phase. Ignored."),
+        "Unexpected end tag (%(name)s) in the table body phase. Ignored.",
    "unexpected-implied-end-tag-in-table-row":
-        _("Unexpected implied end tag (%(name)s) in the table row phase."),
+        "Unexpected implied end tag (%(name)s) in the table row phase.",
    "unexpected-end-tag-in-table-row":
-        _("Unexpected end tag (%(name)s) in the table row phase. Ignored."),
+        "Unexpected end tag (%(name)s) in the table row phase. Ignored.",
    "unexpected-select-in-select":
-        _("Unexpected select start tag in the select phase "
-          "treated as select end tag."),
+        "Unexpected select start tag in the select phase "
+        "treated as select end tag.",
    "unexpected-input-in-select":
-        _("Unexpected input start tag in the select phase."),
+        "Unexpected input start tag in the select phase.",
    "unexpected-start-tag-in-select":
-        _("Unexpected start tag token (%(name)s in the select phase. "
-          "Ignored."),
+        "Unexpected start tag token (%(name)s in the select phase. "
+        "Ignored.",
    "unexpected-end-tag-in-select":
-        _("Unexpected end tag (%(name)s) in the select phase. Ignored."),
+        "Unexpected end tag (%(name)s) in the select phase. Ignored.",
    "unexpected-table-element-start-tag-in-select-in-table":
-        _("Unexpected table element start tag (%(name)s) in the select in table phase."),
+        "Unexpected table element start tag (%(name)s) in the select in table phase.",
    "unexpected-table-element-end-tag-in-select-in-table":
-        _("Unexpected table element end tag (%(name)s) in the select in table phase."),
+        "Unexpected table element end tag (%(name)s) in the select in table phase.",
    "unexpected-char-after-body":
-        _("Unexpected non-space characters in the after body phase."),
+        "Unexpected non-space characters in the after body phase.",
    "unexpected-start-tag-after-body":
-        _("Unexpected start tag token (%(name)s)"
-          " in the after body phase."),
+        "Unexpected start tag token (%(name)s)"
+        " in the after body phase.",
    "unexpected-end-tag-after-body":
-        _("Unexpected end tag token (%(name)s)"
-          " in the after body phase."),
+        "Unexpected end tag token (%(name)s)"
+        " in the after body phase.",
    "unexpected-char-in-frameset":
-        _("Unexpected characters in the frameset phase. Characters ignored."),
+        "Unexpected characters in the frameset phase. Characters ignored.",
    "unexpected-start-tag-in-frameset":
-        _("Unexpected start tag token (%(name)s)"
-          " in the frameset phase. Ignored."),
+        "Unexpected start tag token (%(name)s)"
+        " in the frameset phase. Ignored.",
    "unexpected-frameset-in-frameset-innerhtml":
-        _("Unexpected end tag token (frameset) "
-          "in the frameset phase (innerHTML)."),
+        "Unexpected end tag token (frameset) "
+        "in the frameset phase (innerHTML).",
    "unexpected-end-tag-in-frameset":
-        _("Unexpected end tag token (%(name)s)"
-          " in the frameset phase. Ignored."),
+        "Unexpected end tag token (%(name)s)"
+        " in the frameset phase. Ignored.",
    "unexpected-char-after-frameset":
-        _("Unexpected non-space characters in the "
-          "after frameset phase. Ignored."),
+        "Unexpected non-space characters in the "
+        "after frameset phase. Ignored.",
    "unexpected-start-tag-after-frameset":
-        _("Unexpected start tag (%(name)s)"
-          " in the after frameset phase. Ignored."),
+        "Unexpected start tag (%(name)s)"
+        " in the after frameset phase. Ignored.",
    "unexpected-end-tag-after-frameset":
-        _("Unexpected end tag (%(name)s)"
-          " in the after frameset phase. Ignored."),
+        "Unexpected end tag (%(name)s)"
+        " in the after frameset phase. Ignored.",
    "unexpected-end-tag-after-body-innerhtml":
-        _("Unexpected end tag after body(innerHtml)"),
+        "Unexpected end tag after body(innerHtml)",
    "expected-eof-but-got-char":
-        _("Unexpected non-space characters. Expected end of file."),
+        "Unexpected non-space characters. Expected end of file.",
    "expected-eof-but-got-start-tag":
-        _("Unexpected start tag (%(name)s)"
-          ". Expected end of file."),
+        "Unexpected start tag (%(name)s)"
+        ". Expected end of file.",
    "expected-eof-but-got-end-tag":
-        _("Unexpected end tag (%(name)s)"
-          ". Expected end of file."),
+        "Unexpected end tag (%(name)s)"
+        ". Expected end of file.",
    "eof-in-table":
-        _("Unexpected end of file. Expected table content."),
+        "Unexpected end of file. Expected table content.",
    "eof-in-select":
-        _("Unexpected end of file. Expected select content."),
+        "Unexpected end of file. Expected select content.",
    "eof-in-frameset":
-        _("Unexpected end of file. Expected frameset content."),
+        "Unexpected end of file. Expected frameset content.",
    "eof-in-script-in-script":
-        _("Unexpected end of file. Expected script content."),
+        "Unexpected end of file. Expected script content.",
    "eof-in-foreign-lands":
-        _("Unexpected end of file. Expected foreign content"),
+        "Unexpected end of file. Expected foreign content",
    "non-void-element-with-trailing-solidus":
-        _("Trailing solidus not allowed on element %(name)s"),
+        "Trailing solidus not allowed on element %(name)s",
    "unexpected-html-element-in-foreign-content":
-        _("Element %(name)s not allowed in a non-html context"),
+        "Element %(name)s not allowed in a non-html context",
    "unexpected-end-tag-before-html":
-        _("Unexpected end tag (%(name)s) before html."),
+        "Unexpected end tag (%(name)s) before html.",
    "XXX-undefined-error":
-        _("Undefined error (this sucks and should be fixed)"),
+        "Undefined error (this sucks and should be fixed)",
 }

 namespaces = {
@@ -298,7 +296,7 @@ namespaces = {
    "xmlns": "http://www.w3.org/2000/xmlns/"
 }

-scopingElements = frozenset((
+scopingElements = frozenset([
    (namespaces["html"], "applet"),
    (namespaces["html"], "caption"),
    (namespaces["html"], "html"),
@@ -316,9 +314,9 @@ scopingElements = frozenset((
    (namespaces["svg"], "foreignObject"),
    (namespaces["svg"], "desc"),
    (namespaces["svg"], "title"),
-))
+])

-formattingElements = frozenset((
+formattingElements = frozenset([
    (namespaces["html"], "a"),
    (namespaces["html"], "b"),
    (namespaces["html"], "big"),
@@ -333,9 +331,9 @@ formattingElements = frozenset((
    (namespaces["html"], "strong"),
    (namespaces["html"], "tt"),
    (namespaces["html"], "u")
-))
+])

-specialElements = frozenset((
+specialElements = frozenset([
    (namespaces["html"], "address"),
    (namespaces["html"], "applet"),
    (namespaces["html"], "area"),
@@ -416,22 +414,22 @@ specialElements = frozenset((
    (namespaces["html"], "wbr"),
    (namespaces["html"], "xmp"),
    (namespaces["svg"], "foreignObject")
-))
+])

-htmlIntegrationPointElements = frozenset((
+htmlIntegrationPointElements = frozenset([
    (namespaces["mathml"], "annotaion-xml"),
    (namespaces["svg"], "foreignObject"),
    (namespaces["svg"], "desc"),
    (namespaces["svg"], "title")
-))
+])

-mathmlTextIntegrationPointElements = frozenset((
+mathmlTextIntegrationPointElements = frozenset([
    (namespaces["mathml"], "mi"),
    (namespaces["mathml"], "mo"),
    (namespaces["mathml"], "mn"),
    (namespaces["mathml"], "ms"),
    (namespaces["mathml"], "mtext")
-))
+])

 adjustForeignAttributes = {
    "xlink:actuate": ("xlink", "actuate", namespaces["xlink"]),
@@ -451,21 +449,21 @@ adjustForeignAttributes = {
 unadjustForeignAttributes = dict([((ns, local), qname) for qname, (prefix, local, ns) in
                                  adjustForeignAttributes.items()])

-spaceCharacters = frozenset((
+spaceCharacters = frozenset([
    "\t",
    "\n",
    "\u000C",
    " ",
    "\r"
-))
+])

-tableInsertModeElements = frozenset((
+tableInsertModeElements = frozenset([
    "table",
    "tbody",
    "tfoot",
    "thead",
    "tr"
-))
+])

 asciiLowercase = frozenset(string.ascii_lowercase)
 asciiUppercase = frozenset(string.ascii_uppercase)
@@ -486,7 +484,7 @@ headingElements = (
    "h6"
 )

-voidElements = frozenset((
+voidElements = frozenset([
    "base",
    "command",
    "event-source",
@@ -502,11 +500,11 @@ voidElements = frozenset((
    "input",
    "source",
    "track"
-))
+])

-cdataElements = frozenset(('title', 'textarea'))
+cdataElements = frozenset(['title', 'textarea'])

-rcdataElements = frozenset((
+rcdataElements = frozenset([
    'style',
    'script',
    'xmp',
@@ -514,27 +512,27 @@ rcdataElements = frozenset((
    'noembed',
    'noframes',
    'noscript'
-))
+])

 booleanAttributes = {
-    "": frozenset(("irrelevant",)),
-    "style": frozenset(("scoped",)),
-    "img": frozenset(("ismap",)),
-    "audio": frozenset(("autoplay", "controls")),
-    "video": frozenset(("autoplay", "controls")),
-    "script": frozenset(("defer", "async")),
-    "details": frozenset(("open",)),
-    "datagrid": frozenset(("multiple", "disabled")),
-    "command": frozenset(("hidden", "disabled", "checked", "default")),
-    "hr": frozenset(("noshade")),
-    "menu": frozenset(("autosubmit",)),
-    "fieldset": frozenset(("disabled", "readonly")),
-    "option": frozenset(("disabled", "readonly", "selected")),
-    "optgroup": frozenset(("disabled", "readonly")),
-    "button": frozenset(("disabled", "autofocus")),
-    "input": frozenset(("disabled", "readonly", "required", "autofocus", "checked", "ismap")),
-    "select": frozenset(("disabled", "readonly", "autofocus", "multiple")),
-    "output": frozenset(("disabled", "readonly")),
+    "": frozenset(["irrelevant"]),
+    "style": frozenset(["scoped"]),
+    "img": frozenset(["ismap"]),
+    "audio": frozenset(["autoplay", "controls"]),
+    "video": frozenset(["autoplay", "controls"]),
+    "script": frozenset(["defer", "async"]),
+    "details": frozenset(["open"]),
+    "datagrid": frozenset(["multiple", "disabled"]),
+    "command": frozenset(["hidden", "disabled", "checked", "default"]),
+    "hr": frozenset(["noshade"]),
+    "menu": frozenset(["autosubmit"]),
+    "fieldset": frozenset(["disabled", "readonly"]),
+    "option": frozenset(["disabled", "readonly", "selected"]),
+    "optgroup": frozenset(["disabled", "readonly"]),
+    "button": frozenset(["disabled", "autofocus"]),
+    "input": frozenset(["disabled", "readonly", "required", "autofocus", "checked", "ismap"]),
+    "select": frozenset(["disabled", "readonly", "autofocus", "multiple"]),
+    "output": frozenset(["disabled", "readonly"]),
 }

 # entitiesWindows1252 has to be _ordered_ and needs to have an index. It
@@ -574,7 +572,7 @@ entitiesWindows1252 = (
    376     # 0x9F  0x0178  LATIN CAPITAL LETTER Y WITH DIAERESIS
 )

-xmlEntities = frozenset(('lt;', 'gt;', 'amp;', 'apos;', 'quot;'))
+xmlEntities = frozenset(['lt;', 'gt;', 'amp;', 'apos;', 'quot;'])

 entities = {
    "AElig": "\xc6",
@@ -3088,8 +3086,8 @@ tokenTypes = {
    "ParseError": 7
 }

-tagTokenTypes = frozenset((tokenTypes["StartTag"], tokenTypes["EndTag"],
-                           tokenTypes["EmptyTag"]))
+tagTokenTypes = frozenset([tokenTypes["StartTag"], tokenTypes["EndTag"],
+                           tokenTypes["EmptyTag"]])


 prefixes = dict([(v, k) for k, v in namespaces.items()])
@@ -1,8 +1,5 @@
 from __future__ import absolute_import, division, unicode_literals

-from gettext import gettext
-_ = gettext
-
 from . import _base
 from ..constants import cdataElements, rcdataElements, voidElements

@@ -23,24 +20,24 @@ class Filter(_base.Filter):
            if type in ("StartTag", "EmptyTag"):
                name = token["name"]
                if contentModelFlag != "PCDATA":
-                    raise LintError(_("StartTag not in PCDATA content model flag: %(tag)s") % {"tag": name})
+                    raise LintError("StartTag not in PCDATA content model flag: %(tag)s" % {"tag": name})
                if not isinstance(name, str):
-                    raise LintError(_("Tag name is not a string: %(tag)r") % {"tag": name})
+                    raise LintError("Tag name is not a string: %(tag)r" % {"tag": name})
                if not name:
-                    raise LintError(_("Empty tag name"))
+                    raise LintError("Empty tag name")
                if type == "StartTag" and name in voidElements:
-                    raise LintError(_("Void element reported as StartTag token: %(tag)s") % {"tag": name})
+                    raise LintError("Void element reported as StartTag token: %(tag)s" % {"tag": name})
                elif type == "EmptyTag" and name not in voidElements:
-                    raise LintError(_("Non-void element reported as EmptyTag token: %(tag)s") % {"tag": token["name"]})
+                    raise LintError("Non-void element reported as EmptyTag token: %(tag)s" % {"tag": token["name"]})
                if type == "StartTag":
                    open_elements.append(name)
                for name, value in token["data"]:
                    if not isinstance(name, str):
-                        raise LintError(_("Attribute name is not a string: %(name)r") % {"name": name})
+                        raise LintError("Attribute name is not a string: %(name)r" % {"name": name})
                    if not name:
-                        raise LintError(_("Empty attribute name"))
+                        raise LintError("Empty attribute name")
                    if not isinstance(value, str):
-                        raise LintError(_("Attribute value is not a string: %(value)r") % {"value": value})
+                        raise LintError("Attribute value is not a string: %(value)r" % {"value": value})
                if name in cdataElements:
                    contentModelFlag = "CDATA"
                elif name in rcdataElements:
@@ -51,43 +48,43 @@ class Filter(_base.Filter):
            elif type == "EndTag":
                name = token["name"]
                if not isinstance(name, str):
-                    raise LintError(_("Tag name is not a string: %(tag)r") % {"tag": name})
+                    raise LintError("Tag name is not a string: %(tag)r" % {"tag": name})
                if not name:
-                    raise LintError(_("Empty tag name"))
+                    raise LintError("Empty tag name")
                if name in voidElements:
-                    raise LintError(_("Void element reported as EndTag token: %(tag)s") % {"tag": name})
+                    raise LintError("Void element reported as EndTag token: %(tag)s" % {"tag": name})
                start_name = open_elements.pop()
                if start_name != name:
-                    raise LintError(_("EndTag (%(end)s) does not match StartTag (%(start)s)") % {"end": name, "start": start_name})
+                    raise LintError("EndTag (%(end)s) does not match StartTag (%(start)s)" % {"end": name, "start": start_name})
                contentModelFlag = "PCDATA"

            elif type == "Comment":
                if contentModelFlag != "PCDATA":
-                    raise LintError(_("Comment not in PCDATA content model flag"))
+                    raise LintError("Comment not in PCDATA content model flag")

            elif type in ("Characters", "SpaceCharacters"):
                data = token["data"]
                if not isinstance(data, str):
-                    raise LintError(_("Attribute name is not a string: %(name)r") % {"name": data})
+                    raise LintError("Attribute name is not a string: %(name)r" % {"name": data})
                if not data:
-                    raise LintError(_("%(type)s token with empty data") % {"type": type})
+                    raise LintError("%(type)s token with empty data" % {"type": type})
                if type == "SpaceCharacters":
                    data = data.strip(spaceCharacters)
                    if data:
-                        raise LintError(_("Non-space character(s) found in SpaceCharacters token: %(token)r") % {"token": data})
+                        raise LintError("Non-space character(s) found in SpaceCharacters token: %(token)r" % {"token": data})

            elif type == "Doctype":
                name = token["name"]
                if contentModelFlag != "PCDATA":
-                    raise LintError(_("Doctype not in PCDATA content model flag: %(name)s") % {"name": name})
+                    raise LintError("Doctype not in PCDATA content model flag: %(name)s" % {"name": name})
                if not isinstance(name, str):
-                    raise LintError(_("Tag name is not a string: %(tag)r") % {"tag": name})
+                    raise LintError("Tag name is not a string: %(tag)r" % {"tag": name})
                # XXX: what to do with token["data"] ?

            elif type in ("ParseError", "SerializeError"):
                pass

            else:
-                raise LintError(_("Unknown token type: %(type)s") % {"type": type})
+                raise LintError("Unknown token type: %(type)s" % {"type": type})

            yield token
@@ -18,6 +18,7 @@ from .constants import cdataElements, rcdataElements
 from .constants import tokenTypes, ReparseException, namespaces
 from .constants import htmlIntegrationPointElements, mathmlTextIntegrationPointElements
 from .constants import adjustForeignAttributes as adjustForeignAttributesMap
+from .constants import E


 def parse(doc, treebuilder="etree", encoding=None,
@@ -129,6 +130,17 @@ class HTMLParser(object):

        self.framesetOK = True

+    @property
+    def documentEncoding(self):
+        """The name of the character encoding
+        that was used to decode the input stream,
+        or :obj:`None` if that is not determined yet.
+
+        """
+        if not hasattr(self, 'tokenizer'):
+            return None
+        return self.tokenizer.stream.charEncoding[0]
+
    def isHTMLIntegrationPoint(self, element):
        if (element.name == "annotation-xml" and
                element.namespace == namespaces["mathml"]):
@@ -245,7 +257,7 @@ class HTMLParser(object):
        # XXX The idea is to make errorcode mandatory.
        self.errors.append((self.tokenizer.stream.position(), errorcode, datavars))
        if self.strict:
-            raise ParseError
+            raise ParseError(E[errorcode] % datavars)

    def normalizeToken(self, token):
        """ HTML5 specific normalizations to the token stream """
@@ -868,7 +880,7 @@ def getPhases(debug):
            self.startTagHandler = utils.MethodDispatcher([
                ("html", self.startTagHtml),
                (("base", "basefont", "bgsound", "command", "link", "meta",
-                  "noframes", "script", "style", "title"),
+                  "script", "style", "title"),
                 self.startTagProcessInHead),
                ("body", self.startTagBody),
                ("frameset", self.startTagFrameset),
@@ -1205,8 +1217,7 @@ def getPhases(debug):
            attributes["name"] = "isindex"
            self.processStartTag(impliedTagToken("input", "StartTag",
                                                 attributes=attributes,
-                                                 selfClosing=
-                                                 token["selfClosing"]))
+                                                 selfClosing=token["selfClosing"]))
            self.processEndTag(impliedTagToken("label"))
            self.processStartTag(impliedTagToken("hr", "StartTag"))
            self.processEndTag(impliedTagToken("form"))
@@ -28,7 +28,18 @@ asciiLettersBytes = frozenset([item.encode("ascii") for item in asciiLetters])
 asciiUppercaseBytes = frozenset([item.encode("ascii") for item in asciiUppercase])
 spacesAngleBrackets = spaceCharactersBytes | frozenset([b">", b"<"])

-invalid_unicode_re = re.compile("[\u0001-\u0008\u000B\u000E-\u001F\u007F-\u009F\uD800-\uDFFF\uFDD0-\uFDEF\uFFFE\uFFFF\U0001FFFE\U0001FFFF\U0002FFFE\U0002FFFF\U0003FFFE\U0003FFFF\U0004FFFE\U0004FFFF\U0005FFFE\U0005FFFF\U0006FFFE\U0006FFFF\U0007FFFE\U0007FFFF\U0008FFFE\U0008FFFF\U0009FFFE\U0009FFFF\U000AFFFE\U000AFFFF\U000BFFFE\U000BFFFF\U000CFFFE\U000CFFFF\U000DFFFE\U000DFFFF\U000EFFFE\U000EFFFF\U000FFFFE\U000FFFFF\U0010FFFE\U0010FFFF]")
+
+invalid_unicode_no_surrogate = "[\u0001-\u0008\u000B\u000E-\u001F\u007F-\u009F\uFDD0-\uFDEF\uFFFE\uFFFF\U0001FFFE\U0001FFFF\U0002FFFE\U0002FFFF\U0003FFFE\U0003FFFF\U0004FFFE\U0004FFFF\U0005FFFE\U0005FFFF\U0006FFFE\U0006FFFF\U0007FFFE\U0007FFFF\U0008FFFE\U0008FFFF\U0009FFFE\U0009FFFF\U000AFFFE\U000AFFFF\U000BFFFE\U000BFFFF\U000CFFFE\U000CFFFF\U000DFFFE\U000DFFFF\U000EFFFE\U000EFFFF\U000FFFFE\U000FFFFF\U0010FFFE\U0010FFFF]"
+
+if utils.supports_lone_surrogates:
+    # Use one extra step of indirection and create surrogates with
+    # unichr. Not using this indirection would introduce an illegal
+    # unicode literal on platforms not supporting such lone
+    # surrogates.
+    invalid_unicode_re = re.compile(invalid_unicode_no_surrogate +
+                                    eval('"\\uD800-\\uDFFF"'))
+else:
+    invalid_unicode_re = re.compile(invalid_unicode_no_surrogate)

 non_bmp_invalid_codepoints = set([0x1FFFE, 0x1FFFF, 0x2FFFE, 0x2FFFF, 0x3FFFE,
                                  0x3FFFF, 0x4FFFE, 0x4FFFF, 0x5FFFE, 0x5FFFF,
@@ -164,13 +175,18 @@ class HTMLUnicodeInputStream(object):

        """

-        # Craziness
-        if len("\U0010FFFF") == 1:
+        if not utils.supports_lone_surrogates:
+            # Such platforms will have already checked for such
+            # surrogate errors, so no need to do this checking.
+            self.reportCharacterErrors = None
+            self.replaceCharactersRegexp = None
+        elif len("\U0010FFFF") == 1:
            self.reportCharacterErrors = self.characterErrorsUCS4
-            self.replaceCharactersRegexp = re.compile("[\uD800-\uDFFF]")
+            self.replaceCharactersRegexp = re.compile(eval('"[\\uD800-\\uDFFF]"'))
        else:
            self.reportCharacterErrors = self.characterErrorsUCS2
-            self.replaceCharactersRegexp = re.compile("([\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF])")
+            self.replaceCharactersRegexp = re.compile(
+                eval('"([\\uD800-\\uDBFF](?![\\uDC00-\\uDFFF])|(?<![\\uD800-\\uDBFF])[\\uDC00-\\uDFFF])"'))

        # List of where new lines occur
        self.newLines = [0]
@@ -265,6 +281,7 @@ class HTMLUnicodeInputStream(object):
                self._bufferedCharacter = data[-1]
                data = data[:-1]

+        if self.reportCharacterErrors:
            self.reportCharacterErrors(data)

            # Replace invalid characters
@@ -2,11 +2,26 @@ from __future__ import absolute_import, division, unicode_literals

 import re
 from xml.sax.saxutils import escape, unescape
+from six.moves import urllib_parse as urlparse

 from .tokenizer import HTMLTokenizer
 from .constants import tokenTypes


+content_type_rgx = re.compile(r'''
+                               ^
+                               # Match a content type <application>/<type>
+                               (?P<content_type>[-a-zA-Z0-9.]+/[-a-zA-Z0-9.]+)
+                               # Match any character set and encoding
+                               (?:(?:;charset=(?:[-a-zA-Z0-9]+)(?:;(?:base64))?)
+                                 |(?:;(?:base64))?(?:;charset=(?:[-a-zA-Z0-9]+))?)
+                               # Assume the rest is data
+                               ,.*
+                               $
+                               ''',
+                              re.VERBOSE)
+
+
 class HTMLSanitizerMixin(object):
    """ sanitization of XHTML+MathML+SVG and of inline style attributes."""

@@ -100,8 +115,8 @@ class HTMLSanitizerMixin(object):
                      'xml:base', 'xml:lang', 'xml:space', 'xmlns', 'xmlns:xlink', 'y',
                      'y1', 'y2', 'zoomAndPan']

-    attr_val_is_uri = ['href', 'src', 'cite', 'action', 'longdesc', 'poster',
-                       'xlink:href', 'xml:base']
+    attr_val_is_uri = ['href', 'src', 'cite', 'action', 'longdesc', 'poster', 'background', 'datasrc',
+                       'dynsrc', 'lowsrc', 'ping', 'poster', 'xlink:href', 'xml:base']

    svg_attr_val_allows_ref = ['clip-path', 'color-profile', 'cursor', 'fill',
                               'filter', 'marker', 'marker-start', 'marker-mid', 'marker-end',
@@ -138,7 +153,9 @@ class HTMLSanitizerMixin(object):
    acceptable_protocols = ['ed2k', 'ftp', 'http', 'https', 'irc',
                            'mailto', 'news', 'gopher', 'nntp', 'telnet', 'webcal',
                            'xmpp', 'callto', 'feed', 'urn', 'aim', 'rsync', 'tag',
-                            'ssh', 'sftp', 'rtsp', 'afs']
+                            'ssh', 'sftp', 'rtsp', 'afs', 'data']
+
+    acceptable_content_types = ['image/png', 'image/jpeg', 'image/gif', 'image/webp', 'image/bmp', 'text/plain']

    # subclasses may define their own versions of these constants
    allowed_elements = acceptable_elements + mathml_elements + svg_elements
@@ -147,6 +164,7 @@ class HTMLSanitizerMixin(object):
    allowed_css_keywords = acceptable_css_keywords
    allowed_svg_properties = acceptable_svg_properties
    allowed_protocols = acceptable_protocols
+    allowed_content_types = acceptable_content_types

    # Sanitize the +html+, escaping all elements not in ALLOWED_ELEMENTS, and
    # stripping out all # attributes not in ALLOWED_ATTRIBUTES. Style
@@ -189,10 +207,17 @@ class HTMLSanitizerMixin(object):
                                       unescape(attrs[attr])).lower()
                # remove replacement characters from unescaped characters
                val_unescaped = val_unescaped.replace("\ufffd", "")
-                if (re.match("^[a-z0-9][-+.a-z0-9]*:", val_unescaped) and
-                    (val_unescaped.split(':')[0] not in
-                     self.allowed_protocols)):
+                uri = urlparse.urlparse(val_unescaped)
+                if uri and uri.scheme:
+                    if uri.scheme not in self.allowed_protocols:
                        del attrs[attr]
+                    if uri.scheme == 'data':
+                        m = content_type_rgx.match(uri.path)
+                        if not m:
+                            del attrs[attr]
+                        elif m.group('content_type') not in self.allowed_content_types:
+                            del attrs[attr]
+
            for attr in self.svg_attr_val_allows_ref:
                if attr in attrs:
                    attrs[attr] = re.sub(r'url\s*\(\s*[^#\s][^)]+?\)',
@@ -245,7 +270,7 @@ class HTMLSanitizerMixin(object):
            elif prop.split('-')[0].lower() in ['background', 'border', 'margin',
                                                'padding']:
                for keyword in value.split():
-                    if not keyword in self.acceptable_css_keywords and \
+                    if keyword not in self.acceptable_css_keywords and \
                            not re.match("^(#[0-9a-f]+|rgb\(\d+%?,\d*%?,?\d*%?\)?|\d{0,2}\.?\d{0,2}(cm|em|ex|in|mm|pc|pt|px|%|,|\))?)$", keyword):
                        break
                else:
@@ -1,9 +1,6 @@
 from __future__ import absolute_import, division, unicode_literals
 from six import text_type

-import gettext
-_ = gettext.gettext
-
 try:
    from functools import reduce
 except ImportError:
@@ -35,7 +32,7 @@ else:
                v = utils.surrogatePairToCodepoint(v)
            else:
                v = ord(v)
-            if not v in encode_entity_map or k.islower():
+            if v not in encode_entity_map or k.islower():
                # prefer &lt; over &LT; and similarly for &amp;, &gt;, etc.
                encode_entity_map[v] = k

@@ -208,7 +205,7 @@ class HTMLSerializer(object):
                if token["systemId"]:
                    if token["systemId"].find('"') >= 0:
                        if token["systemId"].find("'") >= 0:
-                            self.serializeError(_("System identifer contains both single and double quote characters"))
+                            self.serializeError("System identifer contains both single and double quote characters")
                        quote_char = "'"
                    else:
                        quote_char = '"'
@@ -220,7 +217,7 @@ class HTMLSerializer(object):
            elif type in ("Characters", "SpaceCharacters"):
                if type == "SpaceCharacters" or in_cdata:
                    if in_cdata and token["data"].find("</") >= 0:
-                        self.serializeError(_("Unexpected </ in CDATA"))
+                        self.serializeError("Unexpected </ in CDATA")
                    yield self.encode(token["data"])
                else:
                    yield self.encode(escape(token["data"]))
@@ -231,7 +228,7 @@ class HTMLSerializer(object):
                if name in rcdataElements and not self.escape_rcdata:
                    in_cdata = True
                elif in_cdata:
-                    self.serializeError(_("Unexpected child element of a CDATA element"))
+                    self.serializeError("Unexpected child element of a CDATA element")
                for (attr_namespace, attr_name), attr_value in token["data"].items():
                    # TODO: Add namespace support here
                    k = attr_name
@@ -279,20 +276,20 @@ class HTMLSerializer(object):
                if name in rcdataElements:
                    in_cdata = False
                elif in_cdata:
-                    self.serializeError(_("Unexpected child element of a CDATA element"))
+                    self.serializeError("Unexpected child element of a CDATA element")
                yield self.encodeStrict("</%s>" % name)

            elif type == "Comment":
                data = token["data"]
                if data.find("--") >= 0:
-                    self.serializeError(_("Comment contains --"))
+                    self.serializeError("Comment contains --")
                yield self.encodeStrict("<!--%s-->" % token["data"])

            elif type == "Entity":
                name = token["name"]
                key = name + ";"
-                if not key in entities:
-                    self.serializeError(_("Entity %s not recognized" % name))
+                if key not in entities:
+                    self.serializeError("Entity %s not recognized" % name)
                if self.resolve_entities and key not in xmlEntities:
                    data = entities[key]
                else:
@@ -158,7 +158,7 @@ def getDomBuilder(DomImplementation):
            else:
                # HACK: allow text nodes as children of the document node
                if hasattr(self.dom, '_child_node_types'):
-                    if not Node.TEXT_NODE in self.dom._child_node_types:
+                    if Node.TEXT_NODE not in self.dom._child_node_types:
                        self.dom._child_node_types = list(self.dom._child_node_types)
                        self.dom._child_node_types.append(Node.TEXT_NODE)
                self.dom.appendChild(self.dom.createTextNode(data))
@@ -10,8 +10,12 @@ returning an iterator generating tokens.

 from __future__ import absolute_import, division, unicode_literals

+__all__ = ["getTreeWalker", "pprint", "dom", "etree", "genshistream", "lxmletree",
+           "pulldom"]
+
 import sys

+from .. import constants
 from ..utils import default_etree

 treeWalkerCache = {}
@@ -55,3 +59,89 @@ def getTreeWalker(treeType, implementation=None, **kwargs):
            # XXX: NEVER cache here, caching is done in the etree submodule
            return etree.getETreeModule(implementation, **kwargs).TreeWalker
    return treeWalkerCache.get(treeType)
+
+
+def concatenateCharacterTokens(tokens):
+    pendingCharacters = []
+    for token in tokens:
+        type = token["type"]
+        if type in ("Characters", "SpaceCharacters"):
+            pendingCharacters.append(token["data"])
+        else:
+            if pendingCharacters:
+                yield {"type": "Characters", "data": "".join(pendingCharacters)}
+                pendingCharacters = []
+            yield token
+    if pendingCharacters:
+        yield {"type": "Characters", "data": "".join(pendingCharacters)}
+
+
+def pprint(walker):
+    """Pretty printer for tree walkers"""
+    output = []
+    indent = 0
+    for token in concatenateCharacterTokens(walker):
+        type = token["type"]
+        if type in ("StartTag", "EmptyTag"):
+            # tag name
+            if token["namespace"] and token["namespace"] != constants.namespaces["html"]:
+                if token["namespace"] in constants.prefixes:
+                    ns = constants.prefixes[token["namespace"]]
+                else:
+                    ns = token["namespace"]
+                name = "%s %s" % (ns, token["name"])
+            else:
+                name = token["name"]
+            output.append("%s<%s>" % (" " * indent, name))
+            indent += 2
+            # attributes (sorted for consistent ordering)
+            attrs = token["data"]
+            for (namespace, localname), value in sorted(attrs.items()):
+                if namespace:
+                    if namespace in constants.prefixes:
+                        ns = constants.prefixes[namespace]
+                    else:
+                        ns = namespace
+                    name = "%s %s" % (ns, localname)
+                else:
+                    name = localname
+                output.append("%s%s=\"%s\"" % (" " * indent, name, value))
+            # self-closing
+            if type == "EmptyTag":
+                indent -= 2
+
+        elif type == "EndTag":
+            indent -= 2
+
+        elif type == "Comment":
+            output.append("%s<!-- %s -->" % (" " * indent, token["data"]))
+
+        elif type == "Doctype":
+            if token["name"]:
+                if token["publicId"]:
+                    output.append("""%s<!DOCTYPE %s "%s" "%s">""" %
+                                  (" " * indent,
+                                   token["name"],
+                                   token["publicId"],
+                                   token["systemId"] if token["systemId"] else ""))
+                elif token["systemId"]:
+                    output.append("""%s<!DOCTYPE %s "" "%s">""" %
+                                  (" " * indent,
+                                   token["name"],
+                                   token["systemId"]))
+                else:
+                    output.append("%s<!DOCTYPE %s>" % (" " * indent,
+                                                       token["name"]))
+            else:
+                output.append("%s<!DOCTYPE >" % (" " * indent,))
+
+        elif type == "Characters":
+            output.append("%s\"%s\"" % (" " * indent, token["data"]))
+
+        elif type == "SpaceCharacters":
+            assert False, "concatenateCharacterTokens should have got rid of all Space tokens"
+
+        else:
+            raise ValueError("Unknown token type, %s" % type)
+
+    return "\n".join(output)
@@ -1,8 +1,8 @@
 from __future__ import absolute_import, division, unicode_literals
 from six import text_type, string_types

-import gettext
-_ = gettext.gettext
+__all__ = ["DOCUMENT", "DOCTYPE", "TEXT", "ELEMENT", "COMMENT", "ENTITY", "UNKNOWN",
+           "TreeWalker", "NonRecursiveTreeWalker"]

 from xml.dom import Node

@@ -58,7 +58,7 @@ class TreeWalker(object):
               "namespace": to_text(namespace),
               "data": attrs}
        if hasChildren:
-            yield self.error(_("Void element has children"))
+            yield self.error("Void element has children")

    def startTag(self, namespace, name, attrs):
        assert namespace is None or isinstance(namespace, string_types), type(namespace)
@@ -122,7 +122,7 @@ class TreeWalker(object):
        return {"type": "Entity", "name": text_type(name)}

    def unknown(self, nodeType):
-        return self.error(_("Unknown node type: ") + nodeType)
+        return self.error("Unknown node type: " + nodeType)


 class NonRecursiveTreeWalker(TreeWalker):
@@ -2,9 +2,6 @@ from __future__ import absolute_import, division, unicode_literals

 from xml.dom import Node

-import gettext
-_ = gettext.gettext
-
 from . import _base


@@ -7,12 +7,10 @@ except ImportError:
        from ordereddict import OrderedDict
    except ImportError:
        OrderedDict = dict
-import gettext
-_ = gettext.gettext

 import re

-from six import text_type
+from six import string_types

 from . import _base
 from ..utils import moduleFactoryFactory
@@ -60,7 +58,7 @@ def getETreeBuilder(ElementTreeImplementation):
                return _base.COMMENT, node.text

            else:
-                assert type(node.tag) == text_type, type(node.tag)
+                assert isinstance(node.tag, string_types), type(node.tag)
                # This is assumed to be an ordinary element
                match = tag_regexp.match(node.tag)
                if match:
@@ -4,9 +4,6 @@ from six import text_type
 from lxml import etree
 from ..treebuilders.etree import tag_regexp

-from gettext import gettext
-_ = gettext
-
 from . import _base

 from .. import ihatexml
@@ -130,7 +127,7 @@ class TreeWalker(_base.NonRecursiveTreeWalker):
    def getNodeDetails(self, node):
        if isinstance(node, tuple):  # Text node
            node, key = node
-            assert key in ("text", "tail"), _("Text nodes are text or tail, found %s") % key
+            assert key in ("text", "tail"), "Text nodes are text or tail, found %s" % key
            return _base.TEXT, ensure_str(getattr(node, key))

        elif isinstance(node, Root):
@@ -169,7 +166,7 @@ class TreeWalker(_base.NonRecursiveTreeWalker):
                    attrs, len(node) > 0 or node.text)

    def getFirstChild(self, node):
-        assert not isinstance(node, tuple), _("Text nodes have no children")
+        assert not isinstance(node, tuple), "Text nodes have no children"

        assert len(node) or node.text, "Node has no children"
        if node.text:
@@ -180,7 +177,7 @@ class TreeWalker(_base.NonRecursiveTreeWalker):
    def getNextSibling(self, node):
        if isinstance(node, tuple):  # Text node
            node, key = node
-            assert key in ("text", "tail"), _("Text nodes are text or tail, found %s") % key
+            assert key in ("text", "tail"), "Text nodes are text or tail, found %s" % key
            if key == "text":
                # XXX: we cannot use a "bool(node) and node[0] or None" construct here
                # because node[0] might evaluate to False if it has no child element
@@ -196,7 +193,7 @@ class TreeWalker(_base.NonRecursiveTreeWalker):
    def getParentNode(self, node):
        if isinstance(node, tuple):  # Text node
            node, key = node
-            assert key in ("text", "tail"), _("Text nodes are text or tail, found %s") % key
+            assert key in ("text", "tail"), "Text nodes are text or tail, found %s" % key
            if key == "text":
                return node
            # else: fallback to "normal" processing
@@ -2,6 +2,8 @@ from __future__ import absolute_import, division, unicode_literals

 from types import ModuleType

+from six import text_type
+
 try:
    import xml.etree.cElementTree as default_etree
 except ImportError:
@@ -9,7 +11,26 @@ except ImportError:


 __all__ = ["default_etree", "MethodDispatcher", "isSurrogatePair",
-           "surrogatePairToCodepoint", "moduleFactoryFactory"]
+           "surrogatePairToCodepoint", "moduleFactoryFactory",
+           "supports_lone_surrogates"]
+
+
+# Platforms not supporting lone surrogates (\uD800-\uDFFF) should be
+# caught by the below test. In general this would be any platform
+# using UTF-16 as its encoding of unicode strings, such as
+# Jython. This is because UTF-16 itself is based on the use of such
+# surrogates, and there is no mechanism to further escape such
+# escapes.
+try:
+    _x = eval('"\\uD800"')
+    if not isinstance(_x, text_type):
+        # We need this with u"" because of http://bugs.jython.org/issue2039
+        _x = eval('u"\\uD800"')
+        assert isinstance(_x, text_type)
+except:
+    supports_lone_surrogates = False
+else:
+    supports_lone_surrogates = True


 class MethodDispatcher(dict):
@@ -1,4 +1,4 @@
 #!/usr/bin/python

-from pynma import PyNMA
+from .pynma import PyNMA 

@@ -1,12 +1,20 @@
 #!/usr/bin/python

 from xml.dom.minidom import parseString
+
+try:
+        from http.client import HTTPSConnection
+except ImportError:
        from httplib import HTTPSConnection
+
+try:
+        from urllib.parse import urlencode
+except ImportError:
        from urllib import urlencode

-__version__ = "0.1"
+__version__ = "1.0"

-API_SERVER = 'nma.usk.bz'
+API_SERVER = 'www.notifymyandroid.com' 
 ADD_PATH   = '/publicapi/notify'

 USER_AGENT="PyNMA/v%s"%__version__
@@ -18,7 +26,7 @@ def uniq_preserve(seq): # Dave Kirby

 def uniq(seq):
    # Not order preserving
-    return {}.fromkeys(seq).keys()
+    return list({}.fromkeys(seq).keys())

 class PyNMA(object):
    """PyNMA(apikey=[], developerkey=None)
@@ -60,16 +68,17 @@ class PyNMA(object):
        if type(developerkey) == str and len(developerkey) == 48:
            self._developerkey = developerkey

-    def push(self, application="", event="", description="", url="", priority=0, batch_mode=False):
+    def push(self, application="", event="", description="", url="", contenttype=None, priority=0, batch_mode=False, html=False):
        """Pushes a message on the registered API keys.
 takes 5 arguments:
 - (req) application: application name [256]
 - (req) event:       event name       [1000]
 - (req) description: description      [10000]
 - (opt) url:         url              [512]
+ - (opt) contenttype: Content Type (act: None (plain text) or text/html)
 - (opt) priority:    from -2 (lowest) to 2 (highest) (def:0)
-                - (opt) batch_mode:  call API 5 by 5 (def:False)
-
+ - (opt) batch_mode:  push to all keys at once (def:False)
+ - (opt) html:        shortcut for contenttype=text/html
 Warning: using batch_mode will return error only if all API keys are bad
 cf: http://nma.usk.bz/api.php
 """
@@ -83,6 +92,9 @@ class PyNMA(object):
        if url:
            datas['url'] = url[:512]
        
+        if contenttype == "text/html" or html == True: # Currently only accepted content type
+            datas['content-type'] = "text/html"
+        
        if self._developerkey:
            datas['developerkey'] = self._developerkey

@@ -94,8 +106,7 @@ class PyNMA(object):
                res = self.callapi('POST', ADD_PATH, datas)
                results[key] = res
        else:
-            for i in range(0, len(self._apikey), 5):
-                datas['apikey'] = ",".join(self._apikey[i:i+5])
+            datas['apikey'] = ",".join(self._apikey)
            res = self.callapi('POST', ADD_PATH, datas)
            results[datas['apikey']] = res
        return results
@@ -110,7 +121,7 @@ class PyNMA(object):

        try:
            res = self._parse_reponse(resp.read())
-        except Exception, e:
+        except Exception as e:
            res = {'type':    "pynmaerror",
                   'code':    600,
                   'message': str(e)
@@ -124,12 +135,12 @@ class PyNMA(object):
        for elem in root.childNodes:
            if elem.nodeType == elem.TEXT_NODE: continue
            if elem.tagName == 'success':
-                res = dict(elem.attributes.items())
+                res = dict(list(elem.attributes.items()))
                res['message'] = ""
                res['type']    = elem.tagName
                return res
            if elem.tagName == 'error':
-                res = dict(elem.attributes.items())
+                res = dict(list(elem.attributes.items()))
                res['message'] = elem.firstChild.nodeValue
                res['type']    = elem.tagName
                return res
@@ -1,13 +0,0 @@
-Copyright 2014 Kenneth Reitz
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.
@@ -1,54 +0,0 @@
-Requests includes some vendorized python libraries to ease installation.
-
-Urllib3 License
-===============
-
-This is the MIT license: http://www.opensource.org/licenses/mit-license.php
-
-Copyright 2008-2011 Andrey Petrov and contributors (see CONTRIBUTORS.txt),
-Modifications copyright 2012 Kenneth Reitz.
-
-Permission is hereby granted, free of charge, to any person obtaining
-a copy of this software and associated documentation files (the
-"Software"), to deal in the Software without restriction, including
-without limitation the rights to use, copy, modify, merge, publish,
-distribute, sublicense, and/or sell copies of the Software, and to
-permit persons to whom the Software is furnished to do so, subject to
-the following conditions:
-
-The above copyright notice and this permission notice shall be
-included in all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
-LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
-OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
-WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
-Chardet License
-===============
-
-This library is free software; you can redistribute it and/or
-modify it under the terms of the GNU Lesser General Public
-License as published by the Free Software Foundation; either
-version 2.1 of the License, or (at your option) any later version.
-
-This library is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-Lesser General Public License for more details.
-
-You should have received a copy of the GNU Lesser General Public
-License along with this library; if not, write to the Free Software
-Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
-02110-1301  USA
-
-
-CA Bundle License
-=================
-
-This Source Code Form is subject to the terms of the Mozilla Public
-License, v. 2.0. If a copy of the MPL was not distributed with this
-file, You can obtain one at http://mozilla.org/MPL/2.0/.
@@ -1,85 +0,0 @@
-Requests: HTTP for Humans
-=========================
-
-.. image:: https://badge.fury.io/py/requests.png
-    :target: http://badge.fury.io/py/requests
-
-.. image:: https://pypip.in/d/requests/badge.png
-        :target: https://crate.io/packages/requests/
-
-
-Requests is an Apache2 Licensed HTTP library, written in Python, for human
-beings.
-
-Most existing Python modules for sending HTTP requests are extremely
-verbose and cumbersome. Python's builtin urllib2 module provides most of
-the HTTP capabilities you should need, but the api is thoroughly broken.
-It requires an enormous amount of work (even method overrides) to
-perform the simplest of tasks.
-
-Things shouldn't be this way. Not in Python.
-
-.. code-block:: python
-
-    >>> r = requests.get('https://api.github.com', auth=('user', 'pass'))
-    >>> r.status_code
-    204
-    >>> r.headers['content-type']
-    'application/json'
-    >>> r.text
-    ...
-
-See `the same code, without Requests <https://gist.github.com/973705>`_.
-
-Requests allow you to send HTTP/1.1 requests. You can add headers, form data,
-multipart files, and parameters with simple Python dictionaries, and access the
-response data in the same way. It's powered by httplib and `urllib3
-<https://github.com/shazow/urllib3>`_, but it does all the hard work and crazy
-hacks for you.
-
-
-Features
--------
-
- International Domains and URLs
- Keep-Alive & Connection Pooling
- Sessions with Cookie Persistence
- Browser-style SSL Verification
- Basic/Digest Authentication
- Elegant Key/Value Cookies
- Automatic Decompression
- Unicode Response Bodies
- Multipart File Uploads
- Connection Timeouts
- Thread-safety
- HTTP(S) proxy support
-
-
-Installation
------------
-
-To install Requests, simply:
-
-.. code-block:: bash
-
-    $ pip install requests
-
-
-Documentation
-------------
-
-Documentation is available at http://docs.python-requests.org/.
-
-
-Contribute
----------
-
-#. Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug. There is a `Contributor Friendly`_ tag for issues that should be ideal for people who are not very familiar with the codebase yet.
-#. If you feel uncomfortable or uncertain about an issue or your changes, feel free to email @sigmavirus24 and he will happily help you via email, Skype, remote pairing or whatever you are comfortable with.
-#. Fork `the repository`_ on GitHub to start making your changes to the **master** branch (or branch off of it).
-#. Write a test which shows that the bug was fixed or that the feature works as expected.
-#. Send a pull request and bug the maintainer until it gets merged and published. :) Make sure to add yourself to AUTHORS_.
-
-.. _`the repository`: http://github.com/kennethreitz/requests
-.. _AUTHORS: https://github.com/kennethreitz/requests/blob/master/AUTHORS.rst
-.. _Contributor Friendly: https://github.com/kennethreitz/requests/issues?direction=desc&labels=Contributor+Friendly&page=1&sort=updated&state=open
@@ -6,7 +6,7 @@
 #          /

 """
-requests HTTP library
+Requests HTTP library
 ~~~~~~~~~~~~~~~~~~~~~

 Requests is an HTTP library, written in Python, for human beings. Basic GET
@@ -36,17 +36,17 @@ usage:
 The other HTTP methods are supported - see `requests.api`. Full documentation
 is at <http://python-requests.org>.

-:copyright: (c) 2014 by Kenneth Reitz.
+:copyright: (c) 2015 by Kenneth Reitz.
 :license: Apache 2.0, see LICENSE for more details.

 """

 __title__ = 'requests'
-__version__ = '2.5.1'
-__build__ = 0x020501
+__version__ = '2.7.0'
+__build__ = 0x020700
 __author__ = 'Kenneth Reitz'
 __license__ = 'Apache 2.0'
-__copyright__ = 'Copyright 2014 Kenneth Reitz'
+__copyright__ = 'Copyright 2015 Kenneth Reitz'

 # Attempt to enable urllib3's SNI support, if possible
 try:
@@ -11,13 +11,14 @@ and maintain connections.
 import socket

 from .models import Response
-from .packages.urllib3 import Retry
 from .packages.urllib3.poolmanager import PoolManager, proxy_from_url
 from .packages.urllib3.response import HTTPResponse
 from .packages.urllib3.util import Timeout as TimeoutSauce
+from .packages.urllib3.util.retry import Retry
 from .compat import urlparse, basestring
 from .utils import (DEFAULT_CA_BUNDLE_PATH, get_encoding_from_headers,
-                    prepend_scheme_if_needed, get_auth_from_url, urldefragauth)
+                    prepend_scheme_if_needed, get_auth_from_url, urldefragauth,
+                    select_proxy)
 from .structures import CaseInsensitiveDict
 from .packages.urllib3.exceptions import ConnectTimeoutError
 from .packages.urllib3.exceptions import HTTPError as _HTTPError
@@ -35,6 +36,7 @@ from .auth import _basic_auth_str
 DEFAULT_POOLBLOCK = False
 DEFAULT_POOLSIZE = 10
 DEFAULT_RETRIES = 0
+DEFAULT_POOL_TIMEOUT = None


 class BaseAdapter(object):
@@ -237,8 +239,7 @@ class HTTPAdapter(BaseAdapter):
        :param url: The URL to connect to.
        :param proxies: (optional) A Requests-style dictionary of proxies used on this request.
        """
-        proxies = proxies or {}
-        proxy = proxies.get(urlparse(url.lower()).scheme)
+        proxy = select_proxy(url, proxies)

        if proxy:
            proxy = prepend_scheme_if_needed(proxy, 'http')
@@ -271,12 +272,10 @@ class HTTPAdapter(BaseAdapter):
        :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.

        :param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
-        :param proxies: A dictionary of schemes to proxy URLs.
+        :param proxies: A dictionary of schemes or schemes and hosts to proxy URLs.
        """
-        proxies = proxies or {}
+        proxy = select_proxy(request.url, proxies)
        scheme = urlparse(request.url).scheme
-        proxy = proxies.get(scheme)
-
        if proxy and scheme != 'https':
            url = urldefragauth(request.url)
        else:
@@ -309,7 +308,6 @@ class HTTPAdapter(BaseAdapter):
        :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.

        :param proxies: The url of the proxy being used for this request.
-        :param kwargs: Optional additional keyword arguments.
        """
        headers = {}
        username, password = get_auth_from_url(proxy)
@@ -326,8 +324,8 @@ class HTTPAdapter(BaseAdapter):
        :param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
        :param stream: (optional) Whether to stream the request content.
        :param timeout: (optional) How long to wait for the server to send
-            data before giving up, as a float, or a (`connect timeout, read
-            timeout <user/advanced.html#timeouts>`_) tuple.
+            data before giving up, as a float, or a :ref:`(connect timeout,
+            read timeout) <timeouts>` tuple.
        :type timeout: float or tuple
        :param verify: (optional) Whether to verify SSL certificates.
        :param cert: (optional) Any user-provided SSL certificate to be trusted.
@@ -375,7 +373,7 @@ class HTTPAdapter(BaseAdapter):
                if hasattr(conn, 'proxy_pool'):
                    conn = conn.proxy_pool

-                low_conn = conn._get_conn(timeout=timeout)
+                low_conn = conn._get_conn(timeout=DEFAULT_POOL_TIMEOUT)

                try:
                    low_conn.putrequest(request.method,
@@ -407,9 +405,6 @@ class HTTPAdapter(BaseAdapter):
                    # Then, reraise so that we can handle the actual exception.
                    low_conn.close()
                    raise
-                else:
-                    # All is well, return the connection to the pool.
-                    conn._put_conn(low_conn)

        except (ProtocolError, socket.error) as err:
            raise ConnectionError(err, request=request)
@@ -16,7 +16,6 @@ from . import sessions

 def request(method, url, **kwargs):
    """Constructs and sends a :class:`Request <Request>`.
-    Returns :class:`Response <Response>` object.

    :param method: method for the new :class:`Request` object.
    :param url: URL for the new :class:`Request` object.
@@ -28,8 +27,8 @@ def request(method, url, **kwargs):
    :param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': ('filename', fileobj)}``) for multipart encoding upload.
    :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
    :param timeout: (optional) How long to wait for the server to send data
-        before giving up, as a float, or a (`connect timeout, read timeout
-        <user/advanced.html#timeouts>`_) tuple.
+        before giving up, as a float, or a :ref:`(connect timeout, read
+        timeout) <timeouts>` tuple.
    :type timeout: float or tuple
    :param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed.
    :type allow_redirects: bool
@@ -37,6 +36,8 @@ def request(method, url, **kwargs):
    :param verify: (optional) if ``True``, the SSL cert will be verified. A CA_BUNDLE path can also be provided.
    :param stream: (optional) if ``False``, the response content will be immediately downloaded.
    :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
+    :return: :class:`Response <Response>` object
+    :rtype: requests.Response

    Usage::

@@ -54,22 +55,27 @@ def request(method, url, **kwargs):
    return response


-def get(url, **kwargs):
-    """Sends a GET request. Returns :class:`Response` object.
+def get(url, params=None, **kwargs):
+    """Sends a GET request.

    :param url: URL for the new :class:`Request` object.
+    :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
    :param \*\*kwargs: Optional arguments that ``request`` takes.
+    :return: :class:`Response <Response>` object
+    :rtype: requests.Response
    """

    kwargs.setdefault('allow_redirects', True)
-    return request('get', url, **kwargs)
+    return request('get', url, params=params, **kwargs)


 def options(url, **kwargs):
-    """Sends a OPTIONS request. Returns :class:`Response` object.
+    """Sends a OPTIONS request.

    :param url: URL for the new :class:`Request` object.
    :param \*\*kwargs: Optional arguments that ``request`` takes.
+    :return: :class:`Response <Response>` object
+    :rtype: requests.Response
    """

    kwargs.setdefault('allow_redirects', True)
@@ -77,10 +83,12 @@ def options(url, **kwargs):


 def head(url, **kwargs):
-    """Sends a HEAD request. Returns :class:`Response` object.
+    """Sends a HEAD request.

    :param url: URL for the new :class:`Request` object.
    :param \*\*kwargs: Optional arguments that ``request`` takes.
+    :return: :class:`Response <Response>` object
+    :rtype: requests.Response
    """

    kwargs.setdefault('allow_redirects', False)
@@ -88,44 +96,52 @@ def head(url, **kwargs):


 def post(url, data=None, json=None, **kwargs):
-    """Sends a POST request. Returns :class:`Response` object.
+    """Sends a POST request.

    :param url: URL for the new :class:`Request` object.
    :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
    :param json: (optional) json data to send in the body of the :class:`Request`.
    :param \*\*kwargs: Optional arguments that ``request`` takes.
+    :return: :class:`Response <Response>` object
+    :rtype: requests.Response
    """

    return request('post', url, data=data, json=json, **kwargs)


 def put(url, data=None, **kwargs):
-    """Sends a PUT request. Returns :class:`Response` object.
+    """Sends a PUT request.

    :param url: URL for the new :class:`Request` object.
    :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
    :param \*\*kwargs: Optional arguments that ``request`` takes.
+    :return: :class:`Response <Response>` object
+    :rtype: requests.Response
    """

    return request('put', url, data=data, **kwargs)


 def patch(url, data=None, **kwargs):
-    """Sends a PATCH request. Returns :class:`Response` object.
+    """Sends a PATCH request.

    :param url: URL for the new :class:`Request` object.
    :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
    :param \*\*kwargs: Optional arguments that ``request`` takes.
+    :return: :class:`Response <Response>` object
+    :rtype: requests.Response
    """

    return request('patch', url,  data=data, **kwargs)


 def delete(url, **kwargs):
-    """Sends a DELETE request. Returns :class:`Response` object.
+    """Sends a DELETE request.

    :param url: URL for the new :class:`Request` object.
    :param \*\*kwargs: Optional arguments that ``request`` takes.
+    :return: :class:`Response <Response>` object
+    :rtype: requests.Response
    """

    return request('delete', url, **kwargs)
@@ -103,7 +103,8 @@ class HTTPDigestAuth(AuthBase):
        # XXX not implemented yet
        entdig = None
        p_parsed = urlparse(url)
-        path = p_parsed.path
+        #: path is request-uri defined in RFC 2616 which should not be empty
+        path = p_parsed.path or "/"
        if p_parsed.query:
            path += '?' + p_parsed.query

@@ -124,13 +125,15 @@ class HTTPDigestAuth(AuthBase):
        s += os.urandom(8)

        cnonce = (hashlib.sha1(s).hexdigest()[:16])
-        noncebit = "%s:%s:%s:%s:%s" % (nonce, ncvalue, cnonce, qop, HA2)
        if _algorithm == 'MD5-SESS':
            HA1 = hash_utf8('%s:%s:%s' % (HA1, nonce, cnonce))

        if qop is None:
            respdig = KD(HA1, "%s:%s" % (nonce, HA2))
        elif qop == 'auth' or 'auth' in qop.split(','):
+            noncebit = "%s:%s:%s:%s:%s" % (
+                nonce, ncvalue, cnonce, 'auth', HA2
+                )
            respdig = KD(HA1, noncebit)
        else:
            # XXX handle auth-int.
@@ -176,7 +179,7 @@ class HTTPDigestAuth(AuthBase):
            # Consume content and release the original connection
            # to allow our new request to reuse the same one.
            r.content
-            r.raw.release_conn()
+            r.close()
            prep = r.request.copy()
            extract_cookies_to_jar(prep._cookies, r.request, r.raw)
            prep.prepare_cookies(prep._cookies)
@@ -21,58 +21,6 @@ is_py2 = (_ver[0] == 2)
 #: Python 3.x?
 is_py3 = (_ver[0] == 3)

-#: Python 3.0.x
-is_py30 = (is_py3 and _ver[1] == 0)
-
-#: Python 3.1.x
-is_py31 = (is_py3 and _ver[1] == 1)
-
-#: Python 3.2.x
-is_py32 = (is_py3 and _ver[1] == 2)
-
-#: Python 3.3.x
-is_py33 = (is_py3 and _ver[1] == 3)
-
-#: Python 3.4.x
-is_py34 = (is_py3 and _ver[1] == 4)
-
-#: Python 2.7.x
-is_py27 = (is_py2 and _ver[1] == 7)
-
-#: Python 2.6.x
-is_py26 = (is_py2 and _ver[1] == 6)
-
-#: Python 2.5.x
-is_py25 = (is_py2 and _ver[1] == 5)
-
-#: Python 2.4.x
-is_py24 = (is_py2 and _ver[1] == 4)   # I'm assuming this is not by choice.
-
-
-# ---------
-# Platforms
-# ---------
-
-
-# Syntax sugar.
-_ver = sys.version.lower()
-
-is_pypy = ('pypy' in _ver)
-is_jython = ('jython' in _ver)
-is_ironpython = ('iron' in _ver)
-
-# Assume CPython, if nothing else.
-is_cpython = not any((is_pypy, is_jython, is_ironpython))
-
-# Windows-based system.
-is_windows = 'win32' in str(sys.platform).lower()
-
-# Standard Linux 2+ system.
-is_linux = ('linux' in str(sys.platform).lower())
-is_osx = ('darwin' in str(sys.platform).lower())
-is_hpux = ('hpux' in str(sys.platform).lower())   # Complete guess.
-is_solaris = ('solar==' in str(sys.platform).lower())   # Complete guess.
-
 try:
    import simplejson as json
 except (ImportError, SyntaxError):
@@ -99,7 +47,6 @@ if is_py2:
    basestring = basestring
    numeric_types = (int, long, float)

-
 elif is_py3:
    from urllib.parse import urlparse, urlunparse, urljoin, urlsplit, urlencode, quote, unquote, quote_plus, unquote_plus, urldefrag
    from urllib.request import parse_http_list, getproxies, proxy_bypass
@@ -6,6 +6,7 @@ Compatibility code to be able to use `cookielib.CookieJar` with requests.
 requests.utils imports from here, so be careful with imports.
 """

+import copy
 import time
 import collections
 from .compat import cookielib, urlparse, urlunparse, Morsel
@@ -157,26 +158,28 @@ class CookieConflictError(RuntimeError):


 class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
-    """Compatibility class; is a cookielib.CookieJar, but exposes a dict interface.
+    """Compatibility class; is a cookielib.CookieJar, but exposes a dict
+    interface.

    This is the CookieJar we create by default for requests and sessions that
    don't specify one, since some clients may expect response.cookies and
    session.cookies to support dict operations.

-    Don't use the dict interface internally; it's just for compatibility with
-    with external client code. All `requests` code should work out of the box
-    with externally provided instances of CookieJar, e.g., LWPCookieJar and
-    FileCookieJar.
-
-    Caution: dictionary operations that are normally O(1) may be O(n).
+    Requests does not use the dict interface internally; it's just for
+    compatibility with external client code. All requests code should work
+    out of the box with externally provided instances of ``CookieJar``, e.g.
+    ``LWPCookieJar`` and ``FileCookieJar``.

    Unlike a regular CookieJar, this class is pickleable.
-    """

+    .. warning:: dictionary operations that are normally O(1) may be O(n).
+    """
    def get(self, name, default=None, domain=None, path=None):
        """Dict-like get() that also supports optional domain and path args in
        order to resolve naming collisions from using one cookie jar over
-        multiple domains. Caution: operation is O(n), not O(1)."""
+        multiple domains.
+
+        .. warning:: operation is O(n), not O(1)."""
        try:
            return self._find_no_duplicates(name, domain, path)
        except KeyError:
@@ -199,37 +202,38 @@ class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
        return c

    def iterkeys(self):
-        """Dict-like iterkeys() that returns an iterator of names of cookies from the jar.
-        See itervalues() and iteritems()."""
+        """Dict-like iterkeys() that returns an iterator of names of cookies
+        from the jar. See itervalues() and iteritems()."""
        for cookie in iter(self):
            yield cookie.name

    def keys(self):
-        """Dict-like keys() that returns a list of names of cookies from the jar.
-        See values() and items()."""
+        """Dict-like keys() that returns a list of names of cookies from the
+        jar. See values() and items()."""
        return list(self.iterkeys())

    def itervalues(self):
-        """Dict-like itervalues() that returns an iterator of values of cookies from the jar.
-        See iterkeys() and iteritems()."""
+        """Dict-like itervalues() that returns an iterator of values of cookies
+        from the jar. See iterkeys() and iteritems()."""
        for cookie in iter(self):
            yield cookie.value

    def values(self):
-        """Dict-like values() that returns a list of values of cookies from the jar.
-        See keys() and items()."""
+        """Dict-like values() that returns a list of values of cookies from the
+        jar. See keys() and items()."""
        return list(self.itervalues())

    def iteritems(self):
-        """Dict-like iteritems() that returns an iterator of name-value tuples from the jar.
-        See iterkeys() and itervalues()."""
+        """Dict-like iteritems() that returns an iterator of name-value tuples
+        from the jar. See iterkeys() and itervalues()."""
        for cookie in iter(self):
            yield cookie.name, cookie.value

    def items(self):
-        """Dict-like items() that returns a list of name-value tuples from the jar.
-        See keys() and values(). Allows client-code to call "dict(RequestsCookieJar)
-        and get a vanilla python dict of key value pairs."""
+        """Dict-like items() that returns a list of name-value tuples from the
+        jar. See keys() and values(). Allows client-code to call
+        ``dict(RequestsCookieJar)`` and get a vanilla python dict of key value
+        pairs."""
        return list(self.iteritems())

    def list_domains(self):
@@ -259,8 +263,9 @@ class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
        return False  # there is only one domain in jar

    def get_dict(self, domain=None, path=None):
-        """Takes as an argument an optional domain and path and returns a plain old
-        Python dict of name-value pairs of cookies that meet the requirements."""
+        """Takes as an argument an optional domain and path and returns a plain
+        old Python dict of name-value pairs of cookies that meet the
+        requirements."""
        dictionary = {}
        for cookie in iter(self):
            if (domain is None or cookie.domain == domain) and (path is None
@@ -269,21 +274,24 @@ class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
        return dictionary

    def __getitem__(self, name):
-        """Dict-like __getitem__() for compatibility with client code. Throws exception
-        if there are more than one cookie with name. In that case, use the more
-        explicit get() method instead. Caution: operation is O(n), not O(1)."""
+        """Dict-like __getitem__() for compatibility with client code. Throws
+        exception if there are more than one cookie with name. In that case,
+        use the more explicit get() method instead.
+
+        .. warning:: operation is O(n), not O(1)."""

        return self._find_no_duplicates(name)

    def __setitem__(self, name, value):
-        """Dict-like __setitem__ for compatibility with client code. Throws exception
-        if there is already a cookie of that name in the jar. In that case, use the more
-        explicit set() method instead."""
+        """Dict-like __setitem__ for compatibility with client code. Throws
+        exception if there is already a cookie of that name in the jar. In that
+        case, use the more explicit set() method instead."""

        self.set(name, value)

    def __delitem__(self, name):
-        """Deletes a cookie given a name. Wraps cookielib.CookieJar's remove_cookie_by_name()."""
+        """Deletes a cookie given a name. Wraps ``cookielib.CookieJar``'s
+        ``remove_cookie_by_name()``."""
        remove_cookie_by_name(self, name)

    def set_cookie(self, cookie, *args, **kwargs):
@@ -295,15 +303,16 @@ class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
        """Updates this jar with cookies from another CookieJar or dict-like"""
        if isinstance(other, cookielib.CookieJar):
            for cookie in other:
-                self.set_cookie(cookie)
+                self.set_cookie(copy.copy(cookie))
        else:
            super(RequestsCookieJar, self).update(other)

    def _find(self, name, domain=None, path=None):
-        """Requests uses this method internally to get cookie values. Takes as args name
-        and optional domain and path. Returns a cookie.value. If there are conflicting cookies,
-        _find arbitrarily chooses one. See _find_no_duplicates if you want an exception thrown
-        if there are conflicting cookies."""
+        """Requests uses this method internally to get cookie values. Takes as
+        args name and optional domain and path. Returns a cookie.value. If
+        there are conflicting cookies, _find arbitrarily chooses one. See
+        _find_no_duplicates if you want an exception thrown if there are
+        conflicting cookies."""
        for cookie in iter(self):
            if cookie.name == name:
                if domain is None or cookie.domain == domain:
@@ -313,10 +322,11 @@ class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
        raise KeyError('name=%r, domain=%r, path=%r' % (name, domain, path))

    def _find_no_duplicates(self, name, domain=None, path=None):
-        """__get_item__ and get call _find_no_duplicates -- never used in Requests internally.
-        Takes as args name and optional domain and path. Returns a cookie.value.
-        Throws KeyError if cookie is not found and CookieConflictError if there are
-        multiple cookies that match name and optionally domain and path."""
+        """Both ``__get_item__`` and ``get`` call this function: it's never
+        used elsewhere in Requests. Takes as args name and optional domain and
+        path. Returns a cookie.value. Throws KeyError if cookie is not found
+        and CookieConflictError if there are multiple cookies that match name
+        and optionally domain and path."""
        toReturn = None
        for cookie in iter(self):
            if cookie.name == name:
@@ -350,6 +360,21 @@ class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
        return new_cj


+def _copy_cookie_jar(jar):
+    if jar is None:
+        return None
+
+    if hasattr(jar, 'copy'):
+        # We're dealing with an instane of RequestsCookieJar
+        return jar.copy()
+    # We're dealing with a generic CookieJar instance
+    new_jar = copy.copy(jar)
+    new_jar.clear()
+    for cookie in jar:
+        new_jar.set_cookie(copy.copy(cookie))
+    return new_jar
+
+
 def create_cookie(name, value, **kwargs):
    """Make a cookie from underspecified parameters.

@@ -390,11 +415,14 @@ def morsel_to_cookie(morsel):

    expires = None
    if morsel['max-age']:
-        expires = time.time() + morsel['max-age']
+        try:
+            expires = int(time.time() + int(morsel['max-age']))
+        except ValueError:
+            raise TypeError('max-age: %s must be integer' % morsel['max-age'])
    elif morsel['expires']:
        time_template = '%a, %d-%b-%Y %H:%M:%S GMT'
-        expires = time.mktime(
-            time.strptime(morsel['expires'], time_template)) - time.timezone
+        expires = int(time.mktime(
+            time.strptime(morsel['expires'], time_template)) - time.timezone)
    return create_cookie(
        comment=morsel['comment'],
        comment_url=bool(morsel['comment']),
@@ -15,7 +15,7 @@ from .hooks import default_hooks
 from .structures import CaseInsensitiveDict

 from .auth import HTTPBasicAuth
-from .cookies import cookiejar_from_dict, get_cookie_header
+from .cookies import cookiejar_from_dict, get_cookie_header, _copy_cookie_jar
 from .packages.urllib3.fields import RequestField
 from .packages.urllib3.filepost import encode_multipart_formdata
 from .packages.urllib3.util import parse_url
@@ -30,7 +30,8 @@ from .utils import (
    iter_slices, guess_json_utf, super_len, to_native_string)
 from .compat import (
    cookielib, urlunparse, urlsplit, urlencode, str, bytes, StringIO,
-    is_py2, chardet, json, builtin_str, basestring)
+    is_py2, chardet, builtin_str, basestring)
+from .compat import json as complexjson
 from .status_codes import codes

 #: The set of HTTP status codes that indicate an automatically
@@ -42,12 +43,11 @@ REDIRECT_STATI = (
    codes.temporary_redirect, # 307
    codes.permanent_redirect, # 308
 )
+
 DEFAULT_REDIRECT_LIMIT = 30
 CONTENT_CHUNK_SIZE = 10 * 1024
 ITER_CHUNK_SIZE = 512

-json_dumps = json.dumps
-

 class RequestEncodingMixin(object):
    @property
@@ -143,13 +143,13 @@ class RequestEncodingMixin(object):
            else:
                fn = guess_filename(v) or k
                fp = v
-            if isinstance(fp, str):
-                fp = StringIO(fp)
-            if isinstance(fp, bytes):
-                fp = BytesIO(fp)

-            rf = RequestField(name=k, data=fp.read(),
-                              filename=fn, headers=fh)
+            if isinstance(fp, (str, bytes, bytearray)):
+                fdata = fp
+            else:
+                fdata = fp.read()
+
+            rf = RequestField(name=k, data=fdata, filename=fn, headers=fh)
            rf.make_multipart(content_type=ft)
            new_fields.append(rf)

@@ -206,17 +206,8 @@ class Request(RequestHooksMixin):
      <PreparedRequest [GET]>

    """
-    def __init__(self,
-        method=None,
-        url=None,
-        headers=None,
-        files=None,
-        data=None,
-        params=None,
-        auth=None,
-        cookies=None,
-        hooks=None,
-        json=None):
+    def __init__(self, method=None, url=None, headers=None, files=None,
+        data=None, params=None, auth=None, cookies=None, hooks=None, json=None):

        # Default empty dicts for dict params.
        data = [] if data is None else data
@@ -295,8 +286,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
        self.hooks = default_hooks()

    def prepare(self, method=None, url=None, headers=None, files=None,
-                data=None, params=None, auth=None, cookies=None, hooks=None,
-                json=None):
+        data=None, params=None, auth=None, cookies=None, hooks=None, json=None):
        """Prepares the entire request with the given parameters."""

        self.prepare_method(method)
@@ -305,6 +295,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
        self.prepare_cookies(cookies)
        self.prepare_body(data, files, json)
        self.prepare_auth(auth, url)
+
        # Note that prepare_auth must be last to enable authentication schemes
        # such as OAuth to work on a fully prepared request.

@@ -319,7 +310,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
        p.method = self.method
        p.url = self.url
        p.headers = self.headers.copy() if self.headers is not None else None
-        p._cookies = self._cookies.copy() if self._cookies is not None else None
+        p._cookies = _copy_cookie_jar(self._cookies)
        p.body = self.body
        p.hooks = self.hooks
        return p
@@ -356,8 +347,10 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
            raise InvalidURL(*e.args)

        if not scheme:
-            raise MissingSchema("Invalid URL {0!r}: No schema supplied. "
-                                "Perhaps you meant http://{0}?".format(url))
+            error = ("Invalid URL {0!r}: No schema supplied. Perhaps you meant http://{0}?")
+            error = error.format(to_native_string(url, 'utf8'))
+
+            raise MissingSchema(error)

        if not host:
            raise InvalidURL("Invalid URL %r: No host supplied" % url)
@@ -423,7 +416,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):

        if json is not None:
            content_type = 'application/json'
-            body = json_dumps(json)
+            body = complexjson.dumps(json)

        is_stream = all([
            hasattr(data, '__iter__'),
@@ -500,7 +493,15 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
            self.prepare_content_length(self.body)

    def prepare_cookies(self, cookies):
-        """Prepares the given HTTP cookie data."""
+        """Prepares the given HTTP cookie data.
+
+        This function eventually generates a ``Cookie`` header from the
+        given cookies using cookielib. Due to cookielib's design, the header
+        will not be regenerated if it already exists, meaning this function
+        can only be called once for the life of the
+        :class:`PreparedRequest <PreparedRequest>` object. Any subsequent calls
+        to ``prepare_cookies`` will have no actual effect, unless the "Cookie"
+        header is removed beforehand."""

        if isinstance(cookies, cookielib.CookieJar):
            self._cookies = cookies
@@ -513,6 +514,10 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):

    def prepare_hooks(self, hooks):
        """Prepares the given hooks."""
+        # hooks can be passed as None to the prepare method and to this
+        # method. To prevent iterating over None, simply use an empty list
+        # if hooks is False-y
+        hooks = hooks or []
        for event in hooks:
            self.register_hook(event, hooks[event])

@@ -523,16 +528,8 @@ class Response(object):
    """

    __attrs__ = [
-        '_content',
-        'status_code',
-        'headers',
-        'url',
-        'history',
-        'encoding',
-        'reason',
-        'cookies',
-        'elapsed',
-        'request',
+        '_content', 'status_code', 'headers', 'url', 'history',
+        'encoding', 'reason', 'cookies', 'elapsed', 'request'
    ]

    def __init__(self):
@@ -572,7 +569,11 @@ class Response(object):
        self.cookies = cookiejar_from_dict({})

        #: The amount of time elapsed between sending the request
-        #: and the arrival of the response (as a timedelta)
+        #: and the arrival of the response (as a timedelta).
+        #: This property specifically measures the time taken between sending
+        #: the first byte of the request and finishing parsing the headers. It
+        #: is therefore unaffected by consuming the response content or the
+        #: value of the ``stream`` keyword argument.
        self.elapsed = datetime.timedelta(0)

        #: The :class:`PreparedRequest <PreparedRequest>` object to which this
@@ -648,9 +649,10 @@ class Response(object):
        If decode_unicode is True, content will be decoded using the best
        available encoding based on the response.
        """
+
        def generate():
-            try:
            # Special case for urllib3.
+            if hasattr(self.raw, 'stream'):
                try:
                    for chunk in self.raw.stream(chunk_size, decode_content=True):
                        yield chunk
@@ -660,7 +662,7 @@ class Response(object):
                    raise ContentDecodingError(e)
                except ReadTimeoutError as e:
                    raise ConnectionError(e)
-            except AttributeError:
+            else:
                # Standard file-like object.
                while True:
                    chunk = self.raw.read(chunk_size)
@@ -688,6 +690,8 @@ class Response(object):
        """Iterates over the response data, one line at a time.  When
        stream=True is set on the request, this avoids reading the
        content at once into memory for large responses.
+
+        .. note:: This method is not reentrant safe.
        """

        pending = None
@@ -789,14 +793,16 @@ class Response(object):
            encoding = guess_json_utf(self.content)
            if encoding is not None:
                try:
-                    return json.loads(self.content.decode(encoding), **kwargs)
+                    return complexjson.loads(
+                        self.content.decode(encoding), **kwargs
+                    )
                except UnicodeDecodeError:
                    # Wrong UTF codec detected; usually because it's not UTF-8
                    # but some other 8-bit codec.  This is an RFC violation,
                    # and the server didn't bother to tell us what codec *was*
                    # used.
                    pass
-        return json.loads(self.text, **kwargs)
+        return complexjson.loads(self.text, **kwargs)

    @property
    def links(self):
@@ -822,10 +828,10 @@ class Response(object):
        http_error_msg = ''

        if 400 <= self.status_code < 500:
-            http_error_msg = '%s Client Error: %s' % (self.status_code, self.reason)
+            http_error_msg = '%s Client Error: %s for url: %s' % (self.status_code, self.reason, self.url)

        elif 500 <= self.status_code < 600:
-            http_error_msg = '%s Server Error: %s' % (self.status_code, self.reason)
+            http_error_msg = '%s Server Error: %s for url: %s' % (self.status_code, self.reason, self.url)

        if http_error_msg:
            raise HTTPError(http_error_msg, response=self)
@@ -836,4 +842,7 @@ class Response(object):

        *Note: Should not normally need to be called explicitly.*
        """
+        if not self._content_consumed:
+            return self.raw.close()
+
        return self.raw.release_conn()
@@ -0,0 +1,8 @@
+If you are planning to submit a pull request to requests with any changes in 
+this library do not go any further. These are independent libraries which we 
+vendor into requests. Any changes necessary to these libraries must be made in 
+them and submitted as separate pull requests to those libraries.
+
+urllib3 pull requests go here: https://github.com/shazow/urllib3
+
+chardet pull requests go here: https://github.com/chardet/chardet
@@ -55,9 +55,14 @@ def add_stderr_logger(level=logging.DEBUG):
 del NullHandler


-# Set security warning to only go off once by default.
 import warnings
-warnings.simplefilter('always', exceptions.SecurityWarning)
+# SecurityWarning's always go off by default.
+warnings.simplefilter('always', exceptions.SecurityWarning, append=True)
+# SubjectAltNameWarning's should go off once per host
+warnings.simplefilter('default', exceptions.SubjectAltNameWarning)
+# InsecurePlatformWarning's don't vary between requests, so we keep it default.
+warnings.simplefilter('default', exceptions.InsecurePlatformWarning,
+                      append=True)

 def disable_warnings(category=exceptions.HTTPWarning):
    """
@@ -14,7 +14,7 @@ try: # Python 2.7+
    from collections import OrderedDict
 except ImportError:
    from .packages.ordered_dict import OrderedDict
-from .packages.six import iterkeys, itervalues
+from .packages.six import iterkeys, itervalues, PY3


 __all__ = ['RecentlyUsedContainer', 'HTTPHeaderDict']
@@ -129,25 +129,82 @@ class HTTPHeaderDict(MutableMapping):
    'foo=bar, baz=quxx'
    >>> headers['Content-Length']
    '7'
-
-    If you want to access the raw headers with their original casing
-    for debugging purposes you can access the private ``._data`` attribute
-    which is a normal python ``dict`` that maps the case-insensitive key to a
-    list of tuples stored as (case-sensitive-original-name, value). Using the
-    structure from above as our example:
-
-    >>> headers._data
-    {'set-cookie': [('Set-Cookie', 'foo=bar'), ('set-cookie', 'baz=quxx')],
-    'content-length': [('content-length', '7')]}
    """

    def __init__(self, headers=None, **kwargs):
-        self._data = {}
-        if headers is None:
-            headers = {}
-        self.update(headers, **kwargs)
+        super(HTTPHeaderDict, self).__init__()
+        self._container = {}
+        if headers is not None:
+            if isinstance(headers, HTTPHeaderDict):
+                self._copy_from(headers)
+            else:
+                self.extend(headers)
+        if kwargs:
+            self.extend(kwargs)

-    def add(self, key, value):
+    def __setitem__(self, key, val):
+        self._container[key.lower()] = (key, val)
+        return self._container[key.lower()]
+
+    def __getitem__(self, key):
+        val = self._container[key.lower()]
+        return ', '.join(val[1:])
+
+    def __delitem__(self, key):
+        del self._container[key.lower()]
+
+    def __contains__(self, key):
+        return key.lower() in self._container
+
+    def __eq__(self, other):
+        if not isinstance(other, Mapping) and not hasattr(other, 'keys'):
+            return False
+        if not isinstance(other, type(self)):
+            other = type(self)(other)
+        return (dict((k.lower(), v) for k, v in self.itermerged()) ==
+                dict((k.lower(), v) for k, v in other.itermerged()))
+
+    def __ne__(self, other):
+        return not self.__eq__(other)
+
+    if not PY3: # Python 2
+        iterkeys = MutableMapping.iterkeys
+        itervalues = MutableMapping.itervalues
+
+    __marker = object()
+
+    def __len__(self):
+        return len(self._container)
+
+    def __iter__(self):
+        # Only provide the originally cased names
+        for vals in self._container.values():
+            yield vals[0]
+
+    def pop(self, key, default=__marker):
+        '''D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
+          If key is not found, d is returned if given, otherwise KeyError is raised.
+        '''
+        # Using the MutableMapping function directly fails due to the private marker.
+        # Using ordinary dict.pop would expose the internal structures.
+        # So let's reinvent the wheel.
+        try:
+            value = self[key]
+        except KeyError:
+            if default is self.__marker:
+                raise
+            return default
+        else:
+            del self[key]
+            return value
+
+    def discard(self, key):
+        try:
+            del self[key]
+        except KeyError:
+            pass
+
+    def add(self, key, val):
        """Adds a (name, value) pair, doesn't overwrite the value if it already
        exists.

@@ -156,43 +213,111 @@ class HTTPHeaderDict(MutableMapping):
        >>> headers['foo']
        'bar, baz'
        """
-        self._data.setdefault(key.lower(), []).append((key, value))
+        key_lower = key.lower()
+        new_vals = key, val
+        # Keep the common case aka no item present as fast as possible
+        vals = self._container.setdefault(key_lower, new_vals)
+        if new_vals is not vals:
+            # new_vals was not inserted, as there was a previous one
+            if isinstance(vals, list):
+                # If already several items got inserted, we have a list
+                vals.append(val)
+            else:
+                # vals should be a tuple then, i.e. only one item so far
+                # Need to convert the tuple to list for further extension
+                self._container[key_lower] = [vals[0], vals[1], val]
+
+    def extend(self, *args, **kwargs):
+        """Generic import function for any type of header-like object.
+        Adapted version of MutableMapping.update in order to insert items
+        with self.add instead of self.__setitem__
+        """
+        if len(args) > 1:
+            raise TypeError("extend() takes at most 1 positional "
+                            "arguments ({} given)".format(len(args)))
+        other = args[0] if len(args) >= 1 else ()
+
+        if isinstance(other, HTTPHeaderDict):
+            for key, val in other.iteritems():
+                self.add(key, val)
+        elif isinstance(other, Mapping):
+            for key in other:
+                self.add(key, other[key])
+        elif hasattr(other, "keys"):
+            for key in other.keys():
+                self.add(key, other[key])
+        else:
+            for key, value in other:
+                self.add(key, value)
+
+        for key, value in kwargs.items():
+            self.add(key, value)

    def getlist(self, key):
        """Returns a list of all the values for the named field. Returns an
        empty list if the key doesn't exist."""
-        return self[key].split(', ') if key in self else []
+        try:
+            vals = self._container[key.lower()]
+        except KeyError:
+            return []
+        else:
+            if isinstance(vals, tuple):
+                return [vals[1]]
+            else:
+                return vals[1:]

-    def copy(self):
-        h = HTTPHeaderDict()
-        for key in self._data:
-            for rawkey, value in self._data[key]:
-                h.add(rawkey, value)
-        return h
-
-    def __eq__(self, other):
-        if not isinstance(other, Mapping):
-            return False
-        other = HTTPHeaderDict(other)
-        return dict((k1, self[k1]) for k1 in self._data) == \
-                dict((k2, other[k2]) for k2 in other._data)
-
-    def __getitem__(self, key):
-        values = self._data[key.lower()]
-        return ', '.join(value[1] for value in values)
-
-    def __setitem__(self, key, value):
-        self._data[key.lower()] = [(key, value)]
-
-    def __delitem__(self, key):
-        del self._data[key.lower()]
-
-    def __len__(self):
-        return len(self._data)
-
-    def __iter__(self):
-        for headers in itervalues(self._data):
-            yield headers[0][0]
+    # Backwards compatibility for httplib
+    getheaders = getlist
+    getallmatchingheaders = getlist
+    iget = getlist

    def __repr__(self):
-        return '%s(%r)' % (self.__class__.__name__, dict(self.items()))
+        return "%s(%s)" % (type(self).__name__, dict(self.itermerged()))
+
+    def _copy_from(self, other):
+        for key in other:
+            val = other.getlist(key)
+            if isinstance(val, list):
+                # Don't need to convert tuples
+                val = list(val)
+            self._container[key.lower()] = [key] + val
+
+    def copy(self):
+        clone = type(self)()
+        clone._copy_from(self)
+        return clone
+
+    def iteritems(self):
+        """Iterate over all header lines, including duplicate ones."""
+        for key in self:
+            vals = self._container[key.lower()]
+            for val in vals[1:]:
+                yield vals[0], val
+
+    def itermerged(self):
+        """Iterate over all headers, merging duplicate ones together."""
+        for key in self:
+            val = self._container[key.lower()]
+            yield val[0], ', '.join(val[1:])
+
+    def items(self):
+        return list(self.iteritems())
+
+    @classmethod
+    def from_httplib(cls, message): # Python 2
+        """Read headers from a Python 2 httplib message object."""
+        # python2.7 does not expose a proper API for exporting multiheaders
+        # efficiently. This function re-reads raw lines from the message
+        # object and extracts the multiheaders properly.
+        headers = []
+
+        for line in message.headers:
+            if line.startswith((' ', '\t')):
+                key, value = headers[-1]
+                headers[-1] = (key, value + '\r\n' + line.rstrip())
+                continue
+
+            key, value = line.split(':', 1)
+            headers.append((key, value.strip()))
+
+        return cls(headers)
@@ -1,7 +1,7 @@
 import datetime
 import sys
 import socket
-from socket import timeout as SocketTimeout
+from socket import error as SocketError, timeout as SocketTimeout
 import warnings
 from .packages import six

@@ -36,9 +36,10 @@ except NameError:  # Python 2:


 from .exceptions import (
+    NewConnectionError,
    ConnectTimeoutError,
+    SubjectAltNameWarning,
    SystemTimeWarning,
-    SecurityWarning,
 )
 from .packages.ssl_match_hostname import match_hostname

@@ -133,11 +134,15 @@ class HTTPConnection(_HTTPConnection, object):
            conn = connection.create_connection(
                (self.host, self.port), self.timeout, **extra_kw)

-        except SocketTimeout:
+        except SocketTimeout as e:
            raise ConnectTimeoutError(
                self, "Connection to %s timed out. (connect timeout=%s)" %
                (self.host, self.timeout))

+        except SocketError as e:
+            raise NewConnectionError(
+                self, "Failed to establish a new connection: %s" % e)
+
        return conn

    def _prepare_conn(self, conn):
@@ -185,17 +190,23 @@ class VerifiedHTTPSConnection(HTTPSConnection):
    """
    cert_reqs = None
    ca_certs = None
+    ca_cert_dir = None
    ssl_version = None
    assert_fingerprint = None

    def set_cert(self, key_file=None, cert_file=None,
                 cert_reqs=None, ca_certs=None,
-                 assert_hostname=None, assert_fingerprint=None):
+                 assert_hostname=None, assert_fingerprint=None,
+                 ca_cert_dir=None):
+
+        if (ca_certs or ca_cert_dir) and cert_reqs is None:
+            cert_reqs = 'CERT_REQUIRED'

        self.key_file = key_file
        self.cert_file = cert_file
        self.cert_reqs = cert_reqs
        self.ca_certs = ca_certs
+        self.ca_cert_dir = ca_cert_dir
        self.assert_hostname = assert_hostname
        self.assert_fingerprint = assert_fingerprint

@@ -234,6 +245,7 @@ class VerifiedHTTPSConnection(HTTPSConnection):
        self.sock = ssl_wrap_socket(conn, self.key_file, self.cert_file,
                                    cert_reqs=resolved_cert_reqs,
                                    ca_certs=self.ca_certs,
+                                    ca_cert_dir=self.ca_cert_dir,
                                    server_hostname=hostname,
                                    ssl_version=resolved_ssl_version)

@@ -245,10 +257,11 @@ class VerifiedHTTPSConnection(HTTPSConnection):
            cert = self.sock.getpeercert()
            if not cert.get('subjectAltName', ()):
                warnings.warn((
-                    'Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. '
-                    'This feature is being removed by major browsers and deprecated by RFC 2818. '
-                    '(See https://github.com/shazow/urllib3/issues/497 for details.)'),
-                    SecurityWarning
+                    'Certificate for {0} has no `subjectAltName`, falling back to check for a '
+                    '`commonName` for now. This feature is being removed by major browsers and '
+                    'deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 '
+                    'for details.)'.format(hostname)),
+                    SubjectAltNameWarning
                )
            match_hostname(cert, self.assert_hostname or hostname)

@@ -260,3 +273,5 @@ if ssl:
    # Make a copy for testing.
    UnverifiedHTTPSConnection = HTTPSConnection
    HTTPSConnection = VerifiedHTTPSConnection
+else:
+    HTTPSConnection = DummyConnection
@@ -17,14 +17,17 @@ from .exceptions import (
    ClosedPoolError,
    ProtocolError,
    EmptyPoolError,
+    HeaderParsingError,
    HostChangedError,
    LocationValueError,
    MaxRetryError,
    ProxyError,
+    ConnectTimeoutError,
    ReadTimeoutError,
    SSLError,
    TimeoutError,
    InsecureRequestWarning,
+    NewConnectionError,
 )
 from .packages.ssl_match_hostname import CertificateError
 from .packages import six
@@ -38,9 +41,10 @@ from .request import RequestMethods
 from .response import HTTPResponse

 from .util.connection import is_connection_dropped
+from .util.response import assert_header_parsing
 from .util.retry import Retry
 from .util.timeout import Timeout
-from .util.url import get_host
+from .util.url import get_host, Url


 xrange = six.moves.xrange
@@ -72,6 +76,21 @@ class ConnectionPool(object):
        return '%s(host=%r, port=%r)' % (type(self).__name__,
                                         self.host, self.port)

+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.close()
+        # Return False to re-raise any potential exceptions
+        return False
+
+    def close():
+        """
+        Close all pooled connections and disable the pool.
+        """
+        pass
+
+
 # This is taken from http://hg.python.org/cpython/file/7aaba721ebc0/Lib/socket.py#l252
 _blocking_errnos = set([errno.EAGAIN, errno.EWOULDBLOCK])

@@ -105,7 +124,7 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):

    :param maxsize:
        Number of connections to save that can be reused. More than 1 is useful
-        in multithreaded situations. If ``block`` is set to false, more
+        in multithreaded situations. If ``block`` is set to False, more
        connections will be created but they will not be saved once they've
        been used.

@@ -266,6 +285,10 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
        """
        pass

+    def _prepare_proxy(self, conn):
+        # Nothing to do for HTTP connections.
+        pass
+
    def _get_timeout(self, timeout):
        """ Helper that always returns a :class:`urllib3.util.Timeout` """
        if timeout is _Default:
@@ -349,7 +372,7 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):

        # Receive the response from the server
        try:
-            try:  # Python 2.7+, use buffering of HTTP responses
+            try:  # Python 2.7, use buffering of HTTP responses
                httplib_response = conn.getresponse(buffering=True)
            except TypeError:  # Python 2.6 and older
                httplib_response = conn.getresponse()
@@ -362,8 +385,19 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
        log.debug("\"%s %s %s\" %s %s" % (method, url, http_version,
                                          httplib_response.status,
                                          httplib_response.length))
+
+        try:
+            assert_header_parsing(httplib_response.msg)
+        except HeaderParsingError as hpe:  # Platform-specific: Python 3
+            log.warning(
+                'Failed to parse headers (url=%s): %s',
+                self._absolute_url(url), hpe, exc_info=True)
+
        return httplib_response

+    def _absolute_url(self, path):
+        return Url(scheme=self.scheme, host=self.host, port=self.port, path=path).url
+
    def close(self):
        """
        Close all pooled connections and disable the pool.
@@ -510,11 +544,18 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):

        try:
            # Request a connection from the queue.
+            timeout_obj = self._get_timeout(timeout)
            conn = self._get_conn(timeout=pool_timeout)

+            conn.timeout = timeout_obj.connect_timeout
+
+            is_new_proxy_conn = self.proxy is not None and not getattr(conn, 'sock', None)
+            if is_new_proxy_conn:
+                self._prepare_proxy(conn)
+
            # Make the request on the httplib connection object.
            httplib_response = self._make_request(conn, method, url,
-                                                  timeout=timeout,
+                                                  timeout=timeout_obj,
                                                  body=body, headers=headers)

            # If we're going to release the connection in ``finally:``, then
@@ -542,26 +583,30 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
            # Close the connection. If a connection is reused on which there
            # was a Certificate error, the next request will certainly raise
            # another Certificate error.
-            if conn:
-                conn.close()
-                conn = None
+            conn = conn and conn.close()
+            release_conn = True
            raise SSLError(e)

-        except (TimeoutError, HTTPException, SocketError, ConnectionError) as e:
-            if conn:
+        except SSLError:
+            # Treat SSLError separately from BaseSSLError to preserve
+            # traceback.
+            conn = conn and conn.close()
+            release_conn = True
+            raise
+
+        except (TimeoutError, HTTPException, SocketError, ProtocolError) as e:
            # Discard the connection for these exceptions. It will be
            # be replaced during the next _get_conn() call.
-                conn.close()
-                conn = None
+            conn = conn and conn.close()
+            release_conn = True

-            stacktrace = sys.exc_info()[2]
-            if isinstance(e, SocketError) and self.proxy:
+            if isinstance(e, (SocketError, NewConnectionError)) and self.proxy:
                e = ProxyError('Cannot connect to proxy.', e)
            elif isinstance(e, (SocketError, HTTPException)):
                e = ProtocolError('Connection aborted.', e)

-            retries = retries.increment(method, url, error=e,
-                                        _pool=self, _stacktrace=stacktrace)
+            retries = retries.increment(method, url, error=e, _pool=self,
+                                        _stacktrace=sys.exc_info()[2])
            retries.sleep()

            # Keep track of the error for the retry warning.
@@ -593,6 +638,9 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
                retries = retries.increment(method, url, response=response, _pool=self)
            except MaxRetryError:
                if retries.raise_on_redirect:
+                    # Release the connection for this response, since we're not
+                    # returning it to be released manually.
+                    response.release_conn()
                    raise
                return response

@@ -629,10 +677,10 @@ class HTTPSConnectionPool(HTTPConnectionPool):
    ``assert_hostname`` and ``host`` in this order to verify connections.
    If ``assert_hostname`` is False, no verification is done.

-    The ``key_file``, ``cert_file``, ``cert_reqs``, ``ca_certs`` and
-    ``ssl_version`` are only used if :mod:`ssl` is available and are fed into
-    :meth:`urllib3.util.ssl_wrap_socket` to upgrade the connection socket
-    into an SSL socket.
+    The ``key_file``, ``cert_file``, ``cert_reqs``, ``ca_certs``,
+    ``ca_cert_dir``, and ``ssl_version`` are only used if :mod:`ssl` is
+    available and are fed into :meth:`urllib3.util.ssl_wrap_socket` to upgrade
+    the connection socket into an SSL socket.
    """

    scheme = 'https'
@@ -645,15 +693,20 @@ class HTTPSConnectionPool(HTTPConnectionPool):
                 key_file=None, cert_file=None, cert_reqs=None,
                 ca_certs=None, ssl_version=None,
                 assert_hostname=None, assert_fingerprint=None,
-                 **conn_kw):
+                 ca_cert_dir=None, **conn_kw):

        HTTPConnectionPool.__init__(self, host, port, strict, timeout, maxsize,
                                    block, headers, retries, _proxy, _proxy_headers,
                                    **conn_kw)
+
+        if ca_certs and cert_reqs is None:
+            cert_reqs = 'CERT_REQUIRED'
+
        self.key_file = key_file
        self.cert_file = cert_file
        self.cert_reqs = cert_reqs
        self.ca_certs = ca_certs
+        self.ca_cert_dir = ca_cert_dir
        self.ssl_version = ssl_version
        self.assert_hostname = assert_hostname
        self.assert_fingerprint = assert_fingerprint
@@ -669,11 +722,18 @@ class HTTPSConnectionPool(HTTPConnectionPool):
                          cert_file=self.cert_file,
                          cert_reqs=self.cert_reqs,
                          ca_certs=self.ca_certs,
+                          ca_cert_dir=self.ca_cert_dir,
                          assert_hostname=self.assert_hostname,
                          assert_fingerprint=self.assert_fingerprint)
            conn.ssl_version = self.ssl_version

-        if self.proxy is not None:
+        return conn
+
+    def _prepare_proxy(self, conn):
+        """
+        Establish tunnel connection early, because otherwise httplib
+        would improperly set Host: header to proxy's IP:port.
+        """
        # Python 2.7+
        try:
            set_tunnel = conn.set_tunnel
@@ -685,12 +745,8 @@ class HTTPSConnectionPool(HTTPConnectionPool):
        else:
            set_tunnel(self.host, self.port, self.proxy_headers)

-            # Establish tunnel connection early, because otherwise httplib
-            # would improperly set Host: header to proxy's IP:port.
        conn.connect()

-        return conn
-
    def _new_conn(self):
        """
        Return a fresh :class:`httplib.HTTPSConnection`.
@@ -700,7 +756,6 @@ class HTTPSConnectionPool(HTTPConnectionPool):
                 % (self.num_connections, self.host))

        if not self.ConnectionCls or self.ConnectionCls is DummyConnection:
-            # Platform-specific: Python without ssl
            raise SSLError("Can't connect to HTTPS URL because the SSL "
                           "module is not available.")

@@ -0,0 +1,222 @@
+import logging
+import os
+import warnings
+
+from ..exceptions import (
+    HTTPError,
+    HTTPWarning,
+    MaxRetryError,
+    ProtocolError,
+    TimeoutError,
+    SSLError
+)
+
+from ..packages.six import BytesIO
+from ..request import RequestMethods
+from ..response import HTTPResponse
+from ..util.timeout import Timeout
+from ..util.retry import Retry
+
+try:
+    from google.appengine.api import urlfetch
+except ImportError:
+    urlfetch = None
+
+
+log = logging.getLogger(__name__)
+
+
+class AppEnginePlatformWarning(HTTPWarning):
+    pass
+
+
+class AppEnginePlatformError(HTTPError):
+    pass
+
+
+class AppEngineManager(RequestMethods):
+    """
+    Connection manager for Google App Engine sandbox applications.
+
+    This manager uses the URLFetch service directly instead of using the
+    emulated httplib, and is subject to URLFetch limitations as described in
+    the App Engine documentation here:
+
+        https://cloud.google.com/appengine/docs/python/urlfetch
+
+    Notably it will raise an AppEnginePlatformError if:
+        * URLFetch is not available.
+        * If you attempt to use this on GAEv2 (Managed VMs), as full socket
+          support is available.
+        * If a request size is more than 10 megabytes.
+        * If a response size is more than 32 megabtyes.
+        * If you use an unsupported request method such as OPTIONS.
+
+    Beyond those cases, it will raise normal urllib3 errors.
+    """
+
+    def __init__(self, headers=None, retries=None, validate_certificate=True):
+        if not urlfetch:
+            raise AppEnginePlatformError(
+                "URLFetch is not available in this environment.")
+
+        if is_prod_appengine_v2():
+            raise AppEnginePlatformError(
+                "Use normal urllib3.PoolManager instead of AppEngineManager"
+                "on Managed VMs, as using URLFetch is not necessary in "
+                "this environment.")
+
+        warnings.warn(
+            "urllib3 is using URLFetch on Google App Engine sandbox instead "
+            "of sockets. To use sockets directly instead of URLFetch see "
+            "https://urllib3.readthedocs.org/en/latest/contrib.html.",
+            AppEnginePlatformWarning)
+
+        RequestMethods.__init__(self, headers)
+        self.validate_certificate = validate_certificate
+
+        self.retries = retries or Retry.DEFAULT
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        # Return False to re-raise any potential exceptions
+        return False
+
+    def urlopen(self, method, url, body=None, headers=None,
+                retries=None, redirect=True, timeout=Timeout.DEFAULT_TIMEOUT,
+                **response_kw):
+
+        retries = self._get_retries(retries, redirect)
+
+        try:
+            response = urlfetch.fetch(
+                url,
+                payload=body,
+                method=method,
+                headers=headers or {},
+                allow_truncated=False,
+                follow_redirects=(
+                    redirect and
+                    retries.redirect != 0 and
+                    retries.total),
+                deadline=self._get_absolute_timeout(timeout),
+                validate_certificate=self.validate_certificate,
+            )
+        except urlfetch.DeadlineExceededError as e:
+            raise TimeoutError(self, e)
+
+        except urlfetch.InvalidURLError as e:
+            if 'too large' in e.message:
+                raise AppEnginePlatformError(
+                    "URLFetch request too large, URLFetch only "
+                    "supports requests up to 10mb in size.", e)
+            raise ProtocolError(e)
+
+        except urlfetch.DownloadError as e:
+            if 'Too many redirects' in e.message:
+                raise MaxRetryError(self, url, reason=e)
+            raise ProtocolError(e)
+
+        except urlfetch.ResponseTooLargeError as e:
+            raise AppEnginePlatformError(
+                "URLFetch response too large, URLFetch only supports"
+                "responses up to 32mb in size.", e)
+
+        except urlfetch.SSLCertificateError as e:
+            raise SSLError(e)
+
+        except urlfetch.InvalidMethodError as e:
+            raise AppEnginePlatformError(
+                "URLFetch does not support method: %s" % method, e)
+
+        http_response = self._urlfetch_response_to_http_response(
+            response, **response_kw)
+
+        # Check for redirect response
+        if (http_response.get_redirect_location() and
+                retries.raise_on_redirect and redirect):
+            raise MaxRetryError(self, url, "too many redirects")
+
+        # Check if we should retry the HTTP response.
+        if retries.is_forced_retry(method, status_code=http_response.status):
+            retries = retries.increment(
+                method, url, response=http_response, _pool=self)
+            log.info("Forced retry: %s" % url)
+            retries.sleep()
+            return self.urlopen(
+                method, url,
+                body=body, headers=headers,
+                retries=retries, redirect=redirect,
+                timeout=timeout, **response_kw)
+
+        return http_response
+
+    def _urlfetch_response_to_http_response(self, urlfetch_resp, **response_kw):
+
+        if is_prod_appengine_v1():
+            # Production GAE handles deflate encoding automatically, but does
+            # not remove the encoding header.
+            content_encoding = urlfetch_resp.headers.get('content-encoding')
+
+            if content_encoding == 'deflate':
+                del urlfetch_resp.headers['content-encoding']
+
+        return HTTPResponse(
+            # In order for decoding to work, we must present the content as
+            # a file-like object.
+            body=BytesIO(urlfetch_resp.content),
+            headers=urlfetch_resp.headers,
+            status=urlfetch_resp.status_code,
+            **response_kw
+        )
+
+    def _get_absolute_timeout(self, timeout):
+        if timeout is Timeout.DEFAULT_TIMEOUT:
+            return 5  # 5s is the default timeout for URLFetch.
+        if isinstance(timeout, Timeout):
+            if not timeout.read is timeout.connect:
+                warnings.warn(
+                    "URLFetch does not support granular timeout settings, "
+                    "reverting to total timeout.", AppEnginePlatformWarning)
+            return timeout.total
+        return timeout
+
+    def _get_retries(self, retries, redirect):
+        if not isinstance(retries, Retry):
+            retries = Retry.from_int(
+                retries, redirect=redirect, default=self.retries)
+
+        if retries.connect or retries.read or retries.redirect:
+            warnings.warn(
+                "URLFetch only supports total retries and does not "
+                "recognize connect, read, or redirect retry parameters.",
+                AppEnginePlatformWarning)
+
+        return retries
+
+
+def is_appengine():
+    return (is_local_appengine() or
+            is_prod_appengine_v1() or
+            is_prod_appengine_v2())
+
+
+def is_appengine_sandbox():
+    return is_appengine() and not is_prod_appengine_v2()
+
+
+def is_local_appengine():
+    return ('APPENGINE_RUNTIME' in os.environ and
+            'Development/' in os.environ['SERVER_SOFTWARE'])
+
+
+def is_prod_appengine_v1():
+    return ('APPENGINE_RUNTIME' in os.environ and
+            'Google App Engine/' in os.environ['SERVER_SOFTWARE'] and
+            not is_prod_appengine_v2())
+
+
+def is_prod_appengine_v2():
+    return os.environ.get('GAE_VM', False) == 'true'
@@ -38,8 +38,6 @@ Module Variables
 ----------------

 :var DEFAULT_SSL_CIPHER_LIST: The list of supported SSL/TLS cipher suites.
-    Default: ``ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:
-    ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS``

 .. _sni: https://en.wikipedia.org/wiki/Server_Name_Indication
 .. _crime attack: https://en.wikipedia.org/wiki/CRIME_(security_exploit)
@@ -85,23 +83,16 @@ _openssl_verify = {
                       + OpenSSL.SSL.VERIFY_FAIL_IF_NO_PEER_CERT,
 }

-# A secure default.
-# Sources for more information on TLS ciphers:
-#
-# - https://wiki.mozilla.org/Security/Server_Side_TLS
-# - https://www.ssllabs.com/projects/best-practices/index.html
-# - https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
-#
-# The general intent is:
-# - Prefer cipher suites that offer perfect forward secrecy (DHE/ECDHE),
-# - prefer ECDHE over DHE for better performance,
-# - prefer any AES-GCM over any AES-CBC for better performance and security,
-# - use 3DES as fallback which is secure but slow,
-# - disable NULL authentication, MD5 MACs and DSS for security reasons.
-DEFAULT_SSL_CIPHER_LIST = "ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:" + \
-    "ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:" + \
-    "!aNULL:!MD5:!DSS"
+DEFAULT_SSL_CIPHER_LIST = util.ssl_.DEFAULT_CIPHERS

+# OpenSSL will only write 16K at a time
+SSL_WRITE_BLOCKSIZE = 16384
+
+try:
+    _ = memoryview
+    has_memoryview = True
+except NameError:
+    has_memoryview = False

 orig_util_HAS_SNI = util.HAS_SNI
 orig_connection_ssl_wrap_socket = connection.ssl_wrap_socket
@@ -191,6 +182,11 @@ class WrappedSocket(object):
                return b''
            else:
                raise
+        except OpenSSL.SSL.ZeroReturnError as e:
+            if self.connection.get_shutdown() == OpenSSL.SSL.RECEIVED_SHUTDOWN:
+                return b''
+            else:
+                raise
        except OpenSSL.SSL.WantReadError:
            rd, wd, ed = select.select(
                [self.socket], [], [], self.socket.gettimeout())
@@ -216,13 +212,21 @@ class WrappedSocket(object):
                continue

    def sendall(self, data):
-        while len(data):
-            sent = self._send_until_done(data)
-            data = data[sent:]
+        if has_memoryview and not isinstance(data, memoryview):
+            data = memoryview(data)
+
+        total_sent = 0
+        while total_sent < len(data):
+            sent = self._send_until_done(data[total_sent:total_sent+SSL_WRITE_BLOCKSIZE])
+            total_sent += sent
+
+    def shutdown(self):
+        # FIXME rethrow compatible exceptions should we ever use this
+        self.connection.shutdown()

    def close(self):
        if self._makefile_refs < 1:
-            return self.connection.shutdown()
+            return self.connection.close()
        else:
            self._makefile_refs -= 1

@@ -263,7 +267,7 @@ def _verify_callback(cnx, x509, err_no, err_depth, return_code):

 def ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=None,
                    ca_certs=None, server_hostname=None,
-                    ssl_version=None):
+                    ssl_version=None, ca_cert_dir=None):
    ctx = OpenSSL.SSL.Context(_openssl_versions[ssl_version])
    if certfile:
        keyfile = keyfile or certfile  # Match behaviour of the normal python ssl library
@@ -272,9 +276,9 @@ def ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=None,
        ctx.use_privatekey_file(keyfile)
    if cert_reqs != ssl.CERT_NONE:
        ctx.set_verify(_openssl_verify[cert_reqs], _verify_callback)
-    if ca_certs:
+    if ca_certs or ca_cert_dir:
        try:
-            ctx.load_verify_locations(ca_certs, None)
+            ctx.load_verify_locations(ca_certs, ca_cert_dir)
        except OpenSSL.SSL.Error as e:
            raise ssl.SSLError('bad ca_certs: %r' % ca_certs, e)
    else:
@@ -294,10 +298,12 @@ def ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=None,
        try:
            cnx.do_handshake()
        except OpenSSL.SSL.WantReadError:
-            select.select([sock], [], [])
+            rd, _, _ = select.select([sock], [], [], sock.gettimeout())
+            if not rd:
+                raise timeout('select timed out')
            continue
        except OpenSSL.SSL.Error as e:
-            raise ssl.SSLError('bad handshake', e)
+            raise ssl.SSLError('bad handshake: %r' % e)
        break

    return WrappedSocket(cnx, sock)
@@ -112,6 +112,9 @@ class ConnectTimeoutError(TimeoutError):
    "Raised when a socket timeout occurs while connecting to a server"
    pass

+class NewConnectionError(ConnectTimeoutError, PoolError):
+    "Raised when we fail to establish a new connection. Usually ECONNREFUSED."
+    pass

 class EmptyPoolError(PoolError):
    "Raised when a pool runs out of connections and no more are allowed."
@@ -149,6 +152,11 @@ class SecurityWarning(HTTPWarning):
    pass


+class SubjectAltNameWarning(SecurityWarning):
+    "Warned when connecting to a host with a certificate missing a SAN."
+    pass
+
+
 class InsecureRequestWarning(SecurityWarning):
    "Warned when making an unverified HTTPS request."
    pass
@@ -157,3 +165,29 @@ class InsecureRequestWarning(SecurityWarning):
 class SystemTimeWarning(SecurityWarning):
    "Warned when system time is suspected to be wrong"
    pass
+
+
+class InsecurePlatformWarning(SecurityWarning):
+    "Warned when certain SSL configuration is not available on a platform."
+    pass
+
+
+class ResponseNotChunked(ProtocolError, ValueError):
+    "Response needs to be chunked in order to read it as chunks."
+    pass
+
+
+class ProxySchemeUnknown(AssertionError, ValueError):
+    "ProxyManager does not support the supplied scheme"
+    # TODO(t-8ch): Stop inheriting from AssertionError in v2.0.
+
+    def __init__(self, scheme):
+        message = "Not supported proxy scheme %s" % scheme
+        super(ProxySchemeUnknown, self).__init__(message)
+
+
+class HeaderParsingError(HTTPError):
+    "Raised by assert_header_parsing, but we convert it to a log.warning statement."
+    def __init__(self, defects, unparsed_data):
+        message = '%s, unparsed data: %r' % (defects or 'Unknown', unparsed_data)
+        super(HeaderParsingError, self).__init__(message)
@@ -8,7 +8,7 @@ except ImportError:
 from ._collections import RecentlyUsedContainer
 from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool
 from .connectionpool import port_by_scheme
-from .exceptions import LocationValueError
+from .exceptions import LocationValueError, MaxRetryError, ProxySchemeUnknown
 from .request import RequestMethods
 from .util.url import parse_url
 from .util.retry import Retry
@@ -64,6 +64,14 @@ class PoolManager(RequestMethods):
        self.pools = RecentlyUsedContainer(num_pools,
                                           dispose_func=lambda p: p.close())

+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.clear()
+        # Return False to re-raise any potential exceptions
+        return False
+
    def _new_pool(self, scheme, host, port):
        """
        Create a new :class:`ConnectionPool` based on host, port and scheme.
@@ -167,7 +175,14 @@ class PoolManager(RequestMethods):
        if not isinstance(retries, Retry):
            retries = Retry.from_int(retries, redirect=redirect)

-        kw['retries'] = retries.increment(method, redirect_location)
+        try:
+            retries = retries.increment(method, url, response=response, _pool=conn)
+        except MaxRetryError:
+            if retries.raise_on_redirect:
+                raise
+            return response
+
+        kw['retries'] = retries
        kw['redirect'] = redirect

        log.info("Redirecting %s -> %s" % (url, redirect_location))
@@ -212,8 +227,8 @@ class ProxyManager(PoolManager):
            port = port_by_scheme.get(proxy.scheme, 80)
            proxy = proxy._replace(port=port)

-        assert proxy.scheme in ("http", "https"), \
-            'Not supported proxy scheme %s' % proxy.scheme
+        if proxy.scheme not in ("http", "https"):
+            raise ProxySchemeUnknown(proxy.scheme)

        self.proxy = proxy
        self.proxy_headers = proxy_headers or {}
@@ -71,14 +71,22 @@ class RequestMethods(object):
                                            headers=headers,
                                            **urlopen_kw)

-    def request_encode_url(self, method, url, fields=None, **urlopen_kw):
+    def request_encode_url(self, method, url, fields=None, headers=None,
+                           **urlopen_kw):
        """
        Make a request using :meth:`urlopen` with the ``fields`` encoded in
        the url. This is useful for request methods like GET, HEAD, DELETE, etc.
        """
+        if headers is None:
+            headers = self.headers
+
+        extra_kw = {'headers': headers}
+        extra_kw.update(urlopen_kw)
+
        if fields:
            url += '?' + urlencode(fields)
-        return self.urlopen(method, url, **urlopen_kw)
+
+        return self.urlopen(method, url, **extra_kw)

    def request_encode_body(self, method, url, fields=None, headers=None,
                            encode_multipart=True, multipart_boundary=None,
@@ -1,13 +1,16 @@
+from contextlib import contextmanager
 import zlib
 import io
 from socket import timeout as SocketTimeout

 from ._collections import HTTPHeaderDict
-from .exceptions import ProtocolError, DecodeError, ReadTimeoutError
-from .packages.six import string_types as basestring, binary_type
+from .exceptions import (
+    ProtocolError, DecodeError, ReadTimeoutError, ResponseNotChunked
+)
+from .packages.six import string_types as basestring, binary_type, PY3
+from .packages.six.moves import http_client as httplib
 from .connection import HTTPException, BaseSSLError
-from .util.response import is_fp_closed
-
+from .util.response import is_fp_closed, is_response_to_head


 class DeflateDecoder(object):
@@ -21,6 +24,9 @@ class DeflateDecoder(object):
        return getattr(self._obj, name)

    def decompress(self, data):
+        if not data:
+            return data
+
        if not self._first_try:
            return self._obj.decompress(data)

@@ -36,9 +42,23 @@ class DeflateDecoder(object):
                self._data = None


+class GzipDecoder(object):
+
+    def __init__(self):
+        self._obj = zlib.decompressobj(16 + zlib.MAX_WBITS)
+
+    def __getattr__(self, name):
+        return getattr(self._obj, name)
+
+    def decompress(self, data):
+        if not data:
+            return data
+        return self._obj.decompress(data)
+
+
 def _get_decoder(mode):
    if mode == 'gzip':
-        return zlib.decompressobj(16 + zlib.MAX_WBITS)
+        return GzipDecoder()

    return DeflateDecoder()

@@ -76,9 +96,10 @@ class HTTPResponse(io.IOBase):
                 strict=0, preload_content=True, decode_content=True,
                 original_response=None, pool=None, connection=None):

-        self.headers = HTTPHeaderDict()
-        if headers:
-            self.headers.update(headers)
+        if isinstance(headers, HTTPHeaderDict):
+            self.headers = headers
+        else:
+            self.headers = HTTPHeaderDict(headers)
        self.status = status
        self.version = version
        self.reason = reason
@@ -100,7 +121,17 @@ class HTTPResponse(io.IOBase):
        if hasattr(body, 'read'):
            self._fp = body

-        if preload_content and not self._body:
+        # Are we using the chunked-style of transfer encoding?
+        self.chunked = False
+        self.chunk_left = None
+        tr_enc = self.headers.get('transfer-encoding', '').lower()
+        # Don't incur the penalty of creating a list and then discarding it
+        encodings = (enc.strip() for enc in tr_enc.split(","))
+        if "chunked" in encodings:
+            self.chunked = True
+
+        # We certainly don't want to preload content when the response is chunked.
+        if not self.chunked and preload_content and not self._body:
            self._body = self.read(decode_content=decode_content)

    def get_redirect_location(self):
@@ -140,6 +171,76 @@ class HTTPResponse(io.IOBase):
        """
        return self._fp_bytes_read

+    def _init_decoder(self):
+        """
+        Set-up the _decoder attribute if necessar.
+        """
+        # Note: content-encoding value should be case-insensitive, per RFC 7230
+        # Section 3.2
+        content_encoding = self.headers.get('content-encoding', '').lower()
+        if self._decoder is None and content_encoding in self.CONTENT_DECODERS:
+            self._decoder = _get_decoder(content_encoding)
+
+    def _decode(self, data, decode_content, flush_decoder):
+        """
+        Decode the data passed in and potentially flush the decoder.
+        """
+        try:
+            if decode_content and self._decoder:
+                data = self._decoder.decompress(data)
+        except (IOError, zlib.error) as e:
+            content_encoding = self.headers.get('content-encoding', '').lower()
+            raise DecodeError(
+                "Received response with content-encoding: %s, but "
+                "failed to decode it." % content_encoding, e)
+
+        if flush_decoder and decode_content and self._decoder:
+            buf = self._decoder.decompress(binary_type())
+            data += buf + self._decoder.flush()
+
+        return data
+
+    @contextmanager
+    def _error_catcher(self):
+        """
+        Catch low-level python exceptions, instead re-raising urllib3
+        variants, so that low-level exceptions are not leaked in the
+        high-level api.
+
+        On exit, release the connection back to the pool.
+        """
+        try:
+            try:
+                yield
+
+            except SocketTimeout:
+                # FIXME: Ideally we'd like to include the url in the ReadTimeoutError but
+                # there is yet no clean way to get at it from this context.
+                raise ReadTimeoutError(self._pool, None, 'Read timed out.')
+
+            except BaseSSLError as e:
+                # FIXME: Is there a better way to differentiate between SSLErrors?
+                if 'read operation timed out' not in str(e):  # Defensive:
+                    # This shouldn't happen but just in case we're missing an edge
+                    # case, let's avoid swallowing SSL errors.
+                    raise
+
+                raise ReadTimeoutError(self._pool, None, 'Read timed out.')
+
+            except HTTPException as e:
+                # This includes IncompleteRead.
+                raise ProtocolError('Connection broken: %r' % e, e)
+        except Exception:
+            # The response may not be closed but we're not going to use it anymore
+            # so close it now to ensure that the connection is released back to the pool.
+            if self._original_response and not self._original_response.isclosed():
+                self._original_response.close()
+
+            raise
+        finally:
+            if self._original_response and self._original_response.isclosed():
+                self.release_conn()
+
    def read(self, amt=None, decode_content=None, cache_content=False):
        """
        Similar to :meth:`httplib.HTTPResponse.read`, but with two additional
@@ -161,12 +262,7 @@ class HTTPResponse(io.IOBase):
            after having ``.read()`` the file object. (Overridden if ``amt`` is
            set.)
        """
-        # Note: content-encoding value should be case-insensitive, per RFC 7230
-        # Section 3.2
-        content_encoding = self.headers.get('content-encoding', '').lower()
-        if self._decoder is None:
-            if content_encoding in self.CONTENT_DECODERS:
-                self._decoder = _get_decoder(content_encoding)
+        self._init_decoder()
        if decode_content is None:
            decode_content = self.decode_content

@@ -174,9 +270,9 @@ class HTTPResponse(io.IOBase):
            return

        flush_decoder = False
+        data = None

-        try:
-            try:
+        with self._error_catcher():
            if amt is None:
                # cStringIO doesn't like amt=None
                data = self._fp.read()
@@ -195,46 +291,16 @@ class HTTPResponse(io.IOBase):
                    self._fp.close()
                    flush_decoder = True

-            except SocketTimeout:
-                # FIXME: Ideally we'd like to include the url in the ReadTimeoutError but
-                # there is yet no clean way to get at it from this context.
-                raise ReadTimeoutError(self._pool, None, 'Read timed out.')
-
-            except BaseSSLError as e:
-                # FIXME: Is there a better way to differentiate between SSLErrors?
-                if not 'read operation timed out' in str(e):  # Defensive:
-                    # This shouldn't happen but just in case we're missing an edge
-                    # case, let's avoid swallowing SSL errors.
-                    raise
-
-                raise ReadTimeoutError(self._pool, None, 'Read timed out.')
-
-            except HTTPException as e:
-                # This includes IncompleteRead.
-                raise ProtocolError('Connection broken: %r' % e, e)
-
+        if data:
            self._fp_bytes_read += len(data)

-            try:
-                if decode_content and self._decoder:
-                    data = self._decoder.decompress(data)
-            except (IOError, zlib.error) as e:
-                raise DecodeError(
-                    "Received response with content-encoding: %s, but "
-                    "failed to decode it." % content_encoding, e)
-
-            if flush_decoder and decode_content and self._decoder:
-                buf = self._decoder.decompress(binary_type())
-                data += buf + self._decoder.flush()
+            data = self._decode(data, decode_content, flush_decoder)

            if cache_content:
                self._body = data

        return data

-        finally:
-            if self._original_response and self._original_response.isclosed():
-                self.release_conn()

    def stream(self, amt=2**16, decode_content=None):
        """
@@ -252,6 +318,10 @@ class HTTPResponse(io.IOBase):
            If True, will attempt to decode the body based on the
            'content-encoding' header.
        """
+        if self.chunked:
+            for line in self.read_chunked(amt, decode_content=decode_content):
+                yield line
+        else:
            while not is_fp_closed(self._fp):
                data = self.read(amt=amt, decode_content=decode_content)

@@ -267,14 +337,17 @@ class HTTPResponse(io.IOBase):
        Remaining parameters are passed to the HTTPResponse constructor, along
        with ``original_response=r``.
        """
+        headers = r.msg

-        headers = HTTPHeaderDict()
-        for k, v in r.getheaders():
-            headers.add(k, v)
+        if not isinstance(headers, HTTPHeaderDict):
+            if PY3: # Python 3
+                headers = HTTPHeaderDict(headers.items())
+            else: # Python 2
+                headers = HTTPHeaderDict.from_httplib(headers)

        # HTTPResponse objects in Python 3 don't have a .strict attribute
        strict = getattr(r, 'strict', 0)
-        return ResponseCls(body=r,
+        resp = ResponseCls(body=r,
                           headers=headers,
                           status=r.status,
                           version=r.version,
@@ -282,6 +355,7 @@ class HTTPResponse(io.IOBase):
                           strict=strict,
                           original_response=r,
                           **response_kw)
+        return resp

    # Backwards-compatibility methods for httplib.HTTPResponse
    def getheaders(self):
@@ -331,3 +405,81 @@ class HTTPResponse(io.IOBase):
        else:
            b[:len(temp)] = temp
            return len(temp)
+
+    def _update_chunk_length(self):
+        # First, we'll figure out length of a chunk and then
+        # we'll try to read it from socket.
+        if self.chunk_left is not None:
+            return
+        line = self._fp.fp.readline()
+        line = line.split(b';', 1)[0]
+        try:
+            self.chunk_left = int(line, 16)
+        except ValueError:
+            # Invalid chunked protocol response, abort.
+            self.close()
+            raise httplib.IncompleteRead(line)
+
+    def _handle_chunk(self, amt):
+        returned_chunk = None
+        if amt is None:
+            chunk = self._fp._safe_read(self.chunk_left)
+            returned_chunk = chunk
+            self._fp._safe_read(2)  # Toss the CRLF at the end of the chunk.
+            self.chunk_left = None
+        elif amt < self.chunk_left:
+            value = self._fp._safe_read(amt)
+            self.chunk_left = self.chunk_left - amt
+            returned_chunk = value
+        elif amt == self.chunk_left:
+            value = self._fp._safe_read(amt)
+            self._fp._safe_read(2)  # Toss the CRLF at the end of the chunk.
+            self.chunk_left = None
+            returned_chunk = value
+        else:  # amt > self.chunk_left
+            returned_chunk = self._fp._safe_read(self.chunk_left)
+            self._fp._safe_read(2)  # Toss the CRLF at the end of the chunk.
+            self.chunk_left = None
+        return returned_chunk
+
+    def read_chunked(self, amt=None, decode_content=None):
+        """
+        Similar to :meth:`HTTPResponse.read`, but with an additional
+        parameter: ``decode_content``.
+
+        :param decode_content:
+            If True, will attempt to decode the body based on the
+            'content-encoding' header.
+        """
+        self._init_decoder()
+        # FIXME: Rewrite this method and make it a class with a better structured logic.
+        if not self.chunked:
+            raise ResponseNotChunked("Response is not chunked. "
+                "Header 'transfer-encoding: chunked' is missing.")
+
+        # Don't bother reading the body of a HEAD request.
+        if self._original_response and is_response_to_head(self._original_response):
+            self._original_response.close()
+            return
+
+        with self._error_catcher():
+            while True:
+                self._update_chunk_length()
+                if self.chunk_left == 0:
+                    break
+                chunk = self._handle_chunk(amt)
+                yield self._decode(chunk, decode_content=decode_content,
+                                   flush_decoder=True)
+
+            # Chunk content ends with \r\n: discard it.
+            while True:
+                line = self._fp.fp.readline()
+                if not line:
+                    # Some sites may not end with '\r\n'.
+                    break
+                if line == b'\r\n':
+                    break
+
+            # We read everything; close the "file".
+            if self._original_response:
+                self._original_response.close()
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
rembo10	5eeab858d1	v0.5.10	2016-01-29 16:30:40 +00:00
rembo10	f64045168d	Updated changelog for v0.5.10	2016-01-29 16:29:53 +00:00
rembo10	f16d977e9b	Removed pylint from travis	2016-01-29 16:19:45 +00:00
rembo10	8274a69236	Testing build status	2016-01-29 16:09:03 +00:00
rembo10	ce9f48e3d5	Moved build status to the top	2016-01-29 15:59:50 +00:00
rembo10	69735bdb22	Merge remote-tracking branch 'hamiltont/patch-2' into develop	2016-01-29 15:43:04 +00:00
rembo10	f3e2143b4b	Revert "Add Python, Windows, OS X, and Linux to .gitignore" This reverts commit `74e071b1d7`.	2016-01-29 15:39:07 +00:00
rembo10	2110eb9855	Revert "Migrate as much as possible to pip and requirements.txt" This reverts commit `982594a4a5`.	2016-01-29 15:38:12 +00:00
rembo10	3c015990c5	Revert "Add versions, and point to rembo10 fork of musicbranizngs" This reverts commit `456b441210`.	2016-01-29 15:37:56 +00:00
Hamilton Turner	0d3026b97d	Add Travis-CI badge to Readme	2016-01-24 15:09:32 -05:00
rembo10	33c8ea7e2b	Fixed some pep8 & pyflakes errors	2015-12-21 15:28:44 -08:00
rembo10	561bed9fed	Merge conflicts	2015-12-21 15:12:43 -08:00
rembo10	7bdfa6a59d	Merge remote-tracking branch 'HammyHavoc/patch-1' into develop	2015-12-21 15:02:27 -08:00
rembo10	b887a941bf	Merge branch 'pip-test' into develop	2015-12-21 15:00:46 -08:00
Valentin Tolmer	b76f5250a8	postprocessor: use a temporary folder	2015-12-21 02:22:32 +01:00
Hammy Havoc	e372369b9e	Added a comma Sentence begins with a conjunctive, needs a comma following independent clause.	2015-12-18 14:20:38 +00:00
rembo10	aaa0ea23e2	Merge remote-tracking branch 'kooimens/patch-1' into develop	2015-12-04 17:21:26 -08:00
rembo10	f8fe2cd532	Merge remote-tracking branch 'JamieMagee/pip' into develop	2015-12-04 17:20:42 -08:00
kooimens	bb362325ad	Should do the trick Use album_dir as leading parameter. Else 'dir' and if not specify just post process	2015-12-05 00:21:53 +01:00
kooimens	56555ad550	Add API ability to postprocess single album folder	2015-12-04 23:08:52 +01:00
Jamie Magee	456b441210	Add versions, and point to rembo10 fork of musicbranizngs	2015-12-04 11:09:09 +01:00
Jamie Magee	982594a4a5	Migrate as much as possible to pip and requirements.txt	2015-12-04 11:09:09 +01:00
rembo10	eaded8dfd9	Merge conflicts from lepricon49/master	2015-12-01 12:07:18 -08:00
rembo10	c583b95ee2	Merge remote-tracking branch 'JamieMagee/gitignore' into develop	2015-12-01 11:30:01 -08:00
rembo10	cda57dd733	Fixed some merge conflicts	2015-12-01 11:29:25 -08:00
rembo10	60fbfce540	Merge remote-tracking branch 'JamieMagee/init-scripts' into develop	2015-12-01 10:13:24 -08:00
rembo10	d8af2eaf4c	Merge remote-tracking branch 'BillyNate/track-matching-patch' into develop	2015-12-01 10:12:18 -08:00
rembo10	9a571988ae	Merge remote-tracking branch 'jvs3/master' into develop	2015-12-01 10:11:04 -08:00
rembo10	b5b81a8bba	Merge remote-tracking branch 'botez/patch-1' into develop	2015-12-01 10:09:09 -08:00
rembo10	e76551b4a8	Merge remote-tracking branch 'theguardian/master' into develop	2015-12-01 10:05:51 -08:00
Jamie Magee	74e071b1d7	Add Python, Windows, OS X, and Linux to .gitignore	2015-11-27 14:10:08 +01:00
Jamie Magee	327585327f	pep8, pyflakes, and pylint suggested fixes	2015-11-27 13:55:11 +01:00
Jamie Magee	4b445ca4df	Make ShellCheck suggested changes	2015-11-27 10:48:04 +01:00
Ade	ef4a844233	rutracker fix	2015-11-14 16:02:31 +13:00
Nate	80515658b0	Fixed a bug in track matching where only progress exactly divisible by 10 was displayed Bug prevented progress from being properly displayed on albums with a track count not divisible by 10	2015-11-05 14:38:19 +01:00
jvs3	e7d7dc6b6b	Added the option to provide an extension when using 'Advanced Encoding' instead of using the extension of the format selected in 'Audio Properties'.	2015-10-22 22:21:36 +02:00
theguardian	aae0476967	check for affected python versions	2015-10-22 08:53:43 -07:00
theguardian	aa6cf97324	made requested fixes	2015-10-22 07:58:57 -07:00
Troy Olson	8506671c5c	Add fix for Ξ (capital 'Xi') for the band CHVRCHΞS I'm guessing there aren't too many other bands that this will apply to. Hex code found here: http://webdesign.about.com/od/localization/l/blhtmlcodes-gr.htm	2015-10-21 13:33:02 -07:00
theguardian	022976d2f6	added verify_ssl_cert option to fix broken ssl installs	2015-10-20 21:52:55 -07:00
Ade	18d53167d0	utorrent torrents not being removed, fix #2385	2015-10-08 21:48:39 +13:00
Ade	9519bfddd4	torznab to transmission fix	2015-10-06 19:32:55 +13:00
Ade	0c556e32fa	Strike whitelist 404	2015-09-23 20:58:10 +12:00
Ade	f9a02d9b3b	Get folder name working in history + show magnet as torrent	2015-09-18 18:16:39 +12:00
Ade	3b6c986704	More torcache fixes - reverted back to rss for kat so we can use magnet links instead of torcache link for transmission, utorrent. Also now uses the kat ’lossless’ category instead of going out to torcache to check for flac. - added torcache referer for blackhole	2015-09-14 21:31:38 +12:00
lepricon49	c78e8764a0	Add snatched email indicator prefix Prefix subject with "Snatched: " to make it more apparent that it is not the final download email notification.	2015-09-13 10:57:14 -07:00
lepricon49	7f4d10c85d	Add magnet support for KAT rTorrent via blackhole KAT torrents downloads were 100% broken. This is a temporary hack to get KAT partially working.	2015-09-13 10:54:05 -07:00
lepricon49	6b30ed0ba6	Stop JS links from scrolling to the top of the page Changed all Using href="#" to href="javascript:void(0)". Using "#" causes the browser to jump to the top of the page when the hyperlink is clicked (default empty # anchor position). Using "javascript:void(0)" will not change the browser's scroll position.	2015-09-12 13:35:48 -07:00
Ade	912d5c4d78	Fix history indent	2015-09-12 08:20:18 +12:00
Bas Stottelaar	faa4bd82ae	Add advanced option to stop renaming folders. Refs #2281 .	2015-09-10 20:37:21 +02:00
Bas Stottelaar	409f6b8dd5	At least display that it is wrong. Refs #2306 .	2015-09-10 20:26:31 +02:00
Bas Stottelaar	4e79698b4e	Add undocumented feature. Refs #2317	2015-09-10 20:23:15 +02:00
Bas Stottelaar	07b28605b5	Add config option to disable changing file permissions (advanced feature for now).	2015-09-10 20:17:27 +02:00
Bas Stottelaar	c4403cc4de	Add logging to issue #2330 .	2015-09-10 20:05:02 +02:00
Bas Stottelaar	7e0742f29b	Potential fix for #2355 .	2015-09-10 19:55:33 +02:00
Bas Stottelaar	b21aef221e	Add SSL/SNI diagnostics script.	2015-09-10 19:43:51 +02:00
Bas Stottelaar	20934f3ac8	PEP8 + ignore requests warning.	2015-09-10 19:42:25 +02:00
Bas Stottelaar	715e497789	Upgraded requests to git-master and forced urllib3 to git-master. Fixes strange errors with pyOpenSSL. See https://github.com/shazow/urllib3/issues/556 .	2015-09-10 19:42:07 +02:00
rembo10	c0d389c283	Updated changelog with right diff urls	2015-09-05 16:07:51 -07:00
rembo10	6720575bd9	v0.5.9	2015-09-05 16:01:52 -07:00
rembo10	b1ec750d1d	Updated changelog for v0.5.9	2015-09-05 16:00:55 -07:00
Ade	d9c78da997	blackhole convert magnet Seems only torcache working, comment others and add user agent	2015-09-05 11:42:35 +12:00
Ade	1fa26e763d	kat flac 403 fix	2015-09-05 00:03:37 +12:00
Ade	bb71b3d3d9	Strike provider	2015-09-04 21:09:22 +12:00
Ade	65e8e56e4a	Show artist name rather than guid when deleting	2015-09-04 18:45:32 +12:00
Ade	ee415d0e50	Fixes #2347	2015-09-04 18:32:15 +12:00
Ade	9db88a6c4f	Removing nabz broken	2015-09-03 18:13:23 +12:00
rembo10	e5151a0a19	#2343 Added option to stop post processing if no good metadata match is found	2015-09-02 12:59:17 -07:00
rembo10	570234717a	Changing the vip mirror to url to prepare for server upgrade	2015-09-02 12:11:25 -07:00
Ade	3f60d1f8de	torznab config issues	2015-09-02 21:16:22 +12:00
AdeHub	65d100c814	Merge pull request #2338 from zone117x/develop Implemented Jackett (Custom Torznab) option for Torrent Search Providers	2015-08-24 21:13:50 +12:00
unknown	61bdab8621	Implemented Jackett (Custom Torznab) option for Torrent Search Providers	2015-08-23 11:41:23 -06:00
Ade	8c94c9ecbf	last.fm fix failing if mbid not found	2015-08-21 19:23:56 +12:00
Ade	7fd895ecb5	Potential fix for #2316	2015-08-21 18:50:45 +12:00
Ade	213bc07646	Keep Original Folder fix	2015-08-07 17:43:22 +12:00
Ade	1f081c181c	rutracker 2nd login try logging in again after 10 secs if 1st attempt fails	2015-08-05 21:27:43 +12:00
Ade	db1d31900e	rutracker small fix up	2015-08-03 18:39:51 +12:00
Ade	2219c79ac3	potential kat fix	2015-08-02 13:52:41 +12:00
Ade	d2782179aa	rutracker revision - Now uses requests with more logging - Update to latest BeautifulSoup and html5lib libs	2015-08-02 12:18:25 +12:00
rembo10	d90a31afc7	Fix for #2282	2015-07-23 18:09:38 +01:00
rembo10	1c236fe0f3	Real fix for pushover	2015-07-23 11:25:06 +01:00
rembo10	24165fb5f3	v0.5.8	2015-07-13 15:22:03 -07:00
rembo10	71da960386	Updated changelog for v0.5.8	2015-07-13 15:21:44 -07:00
rembo10	a16c2d0a26	Fix for #2279 : What.cd not honoring custom search string	2015-07-13 15:09:01 -07:00
rembo10	53de44a225	typo when excluding similar artist update on an artist scan	2015-07-08 23:15:38 -07:00
rembo10	e32cff60e5	Took out some unneccessary functions if we're just scanning an artist	2015-07-08 22:26:55 -07:00
rembo10	0ebba8d17c	Fix for #2271 : Scanning an empty directory with DETECT_BITRATE turned on	2015-07-08 21:58:41 -07:00
rembo10	464db79c04	Some cleanup to the scanArtist function	2015-07-08 11:52:00 -07:00
rembo10	65019d8323	Fix for #2260 & #2129 & 2175 & #2163 : Better artist directory scanning	2015-07-08 02:41:17 -07:00
rembo10	f131d316ca	Set localhost as default	2015-07-08 00:45:33 -07:00
rembo10	e6663c96b5	Fixed some pushbullet config text	2015-07-07 23:56:17 -07:00
rembo10	5f06d11fe9	Fixed pushbullet notifications	2015-07-07 23:47:38 -07:00
rembo10	6db2836662	Moved plex notifications to requests lib for pre-v12 plex	2015-07-07 22:54:34 -07:00
rembo10	cb209f8d98	Fix for #2264 : Metacritic parsing failing with htmlparser	2015-07-06 13:05:51 -07:00
rembo10	5634e17a2e	Fix for #2155 , local variable response referenced before assignment utorrent	2015-07-06 01:28:48 -07:00
rembo10	7c990edeaf	Better fix for #2183	2015-07-06 00:34:16 -07:00
rembo10	9149f62760	Fix for #2173 : Uncaught exception in get_age	2015-07-06 00:21:26 -07:00
rembo10	872ec6b11c	Fix for #2089	2015-07-06 00:09:43 -07:00
rembo10	064a21a07c	Better fix for #2183	2015-07-06 00:02:09 -07:00
rembo10	57c3c7b2e0	Fixed songkick description to prevent spilling over on default fonts	2015-07-05 21:44:37 -07:00
rembo10	6dc75d7a44	Cleaned up the config page a little bit. Took out {option} enabled buttons and just put them in with the option. Alphabetized the notifiers. Fixed checkbox on chrome	2015-07-05 21:40:52 -07:00
rembo10	5f3b7e05c5	Fixed some appearance stuff on the config page (if fonts weren't installed)	2015-07-05 15:04:18 -07:00
rembo10	61d3d8ad00	Fix for history page showing nzbs snatched from hp indexer as unknown	2015-07-05 12:05:04 -07:00
rembo10	e36440f834	Fix for XSS bug when searching	2015-07-04 23:53:03 -07:00
rembo10	4029543d51	Added option to wait until an album's release date before searching	2015-07-04 20:43:06 -07:00
rembo10	8aee9ac91f	Fixed plex client url in config	2015-07-04 19:36:39 -07:00
rembo10	6e5e15f06f	Fixed plex notifications for version 12	2015-07-04 19:27:27 -07:00
rembo10	a72bd4cb7c	Fixed up the appearance of the config page (was kind of broken on Firefox)	2015-07-04 17:26:47 -07:00
rembo10	d062ab553c	Merge branch 'master' of https://github.com/texke/headphones into develop	2015-07-04 12:26:05 -07:00
rembo10	a41edde12a	Merge branch 'wfriesen-logtypo' into develop	2015-07-04 12:23:57 -07:00
rembo10	7608927b4f	Merge branch 'logtypo' of https://github.com/wfriesen/headphones into wfriesen-logtypo	2015-07-04 12:23:29 -07:00
rembo10	24d7647c08	Updated notifymyandroid library	2015-07-04 12:13:40 -07:00
rembo10	a14369cb7d	Added option to only include 'official' extras, rather than everything (bootlegs, promos, etc)	2015-07-04 00:47:02 -07:00
rembo10	3e81c4f3b6	Fix for #2183 . Check to make sure the response has headers before trying to parse them	2015-07-03 21:01:46 -07:00
rembo10	84a6c19677	Release v0.5.7	2015-07-01 02:18:01 -07:00
rembo10	624a1d5509	Updated changelog for 0.5.7 release	2015-07-01 02:17:51 -07:00
rembo10	8b6d35cfe2	Updated changelog to reflect metacritic changes	2015-07-01 02:08:13 -07:00
rembo10	867e502448	Cache metacritic scores in the artists table so we don't lose them if we can't connect for whatever reason	2015-07-01 02:06:27 -07:00
rembo10	74275b969e	Added a useragent to the metacritic request	2015-07-01 00:56:48 -07:00
William Friesen	275dde0630	Fixed typo in postprocessor message	2015-06-28 10:56:48 +10:00
Bas Stottelaar	46b470f30b	Merge pull request #2253 from Hellowlol/patch-1 Add getLogs and clearLogs to api	2015-06-26 08:04:05 +02:00
Hellowlol	fbfec95a8a	Add getLogs and clearLogs to api	2015-06-25 22:53:39 +02:00
rembo10	1e75a18f55	Initial commit to add plex tokens. Plex notifications need to be updated to requests, and the get request to the server might need to be cleaned up - but want to make sure it works first	2015-06-19 17:08:54 -07:00
Jens Rogier	a98849bf30	Added Musicbrainz link to "alternate release" window	2015-06-18 17:01:01 +02:00
rembo10	1b998e9d01	Updated pushover to use requests lib	2015-06-12 00:08:35 -07:00
rembo10	ebd6d3f3f9	Added option to keep folders when force post-processing under Manage. Also, added config option to keep original folder globally	2015-06-09 00:31:50 -07:00
rembo10	1204afadbb	Convert preferred bitrate to vbr preset for what.cd searching	2015-06-08 21:56:10 -07:00
rembo10	cd08b72df7	Added ability to test pushover	2015-06-08 21:55:14 -07:00