v0.5.9

Seems only torcache working, comment others and add user agent

CHANGELOG.md

@@ -1,5 +1,91 @@
 # Changelog
 
+## v0.5.9
+Released 05 September 2015
+
+Highlights:
+* Added: Providers Strike, Jackett, custom Torznabs
+* Added: Option to stop post-processing if no good match found (#2343)
+* Fixed: Blackhole -> Magnet, limit to torcache
+* Fixed: Kat 403 flac error
+* Fixed: Last.fm errors
+* Fixed: Pushover notifications
+* Improved: Rutracker logging, switched to requests lib
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.6...v0.5.7).
+
+## v0.5.8
+Released 13 July 2015
+
+Highlights:
+* Added: Option to only include official extras
+* Added: Option to wait until album release date before searching
+* Fixed: NotifyMyAndroid notifications
+* Fixed: Plex Notifications
+* Fixed: Metacritic parsing
+* Fixed: Pushbullet notifications
+* Fixed: What.cd not honoring custom search term (#2279)
+* Improved: XSS Search bug
+* Improved: Config page layout
+* Improved: Set localhost as default
+* Improved: Better single artist scanning
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.6...v0.5.7).
+
+## v0.5.7
+Released 01 July 2015
+
+Highlights:
+* Improved: Moved pushover to use requests lib
+* Improved: Plex tokens with Plex Home
+* Improved: Added getLogs & clearLogs to api
+* Improved: Cache MetaCritic scores. Added user-agent header to fix 403 errors
+* Improved: Specify whether to delete folders when force post-processing
+* Improved: Convert target bitrate to vbr preset for what.cd searching
+* Improved: Switched Pushover to requests lib
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.6...v0.5.7).
+
+## v0.5.6
+Released 08 June 2015
+
+Highlights:
+* Added: Metacritic scores
+* Added: Series support (e.g. Cafe Del Mar, Now That's What I Call Music, etc)
+* Added: Filter out clean/edited/censored releases (#2198)
+* Added: Button on the log page to toggle verbose/debug logging
+* Fixed: Connecting to SABnzbd over https with python >= 2.7.9
+* Fixed: Email Notifications with SSL
+* Fixed: Don't limit musicbrainz results to first 100
+* Fixed: nzbget url fix
+* Fixed: OSX Notifications
+* Improved: Cuesplit, allow wav, ape to be split
+* Improved: Moved the 'freeze db' option to the advanced->misc. tab
+* Improved: Moved kickass searching to json api, so it doesn't throw 404 errors anymore when there are no results
+* Improved: SSL for headphones indexer
+* Improved: Disable update dialog box if check_github is diabled
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.5...v0.5.6).
+
+## v0.5.5
+Released 04 May 2015
+
+Highlights:
+* Added: force ID3v2.3 during post processing (#2121)
+* Added: MusicBrainz authentication (#2125)
+* Added: Email notifications (addresses #1045)
+* Fixed: Kickass url updated to kickass.to (#2119)
+* Fixed: Piratebay returning 0 bytes for all files
+* Fixed: Albums stopped automatically refreshing when adding an artist
+* Fixed: Min/max sizes for target bitrate
+* Fixed: Don't filter any results if looking for a specific download
+* Fixed: Sort by size in the specific download table
+* Fixed: Deal with beets recommendation.none correctly
+* Improved: Close dialog window automatically when choosing a specific download
+* Improved: Move some repetitive log messages to debug level
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.4...v0.5.5).
+
 ## v0.5.4
 Released 05 February 2015
 
@@ -155,4 +241,4 @@ The full list of commits can be found [here](https://github.com/rembo10/headphon
 ## v0.1
 Released 05 August 2013
 
-The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/2156e1341405d07c5bcfbe994f6b354b32d94cda...v0.1).
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/2156e1341405d07c5bcfbe994f6b354b32d94cda...v0.1).

data/interfaces/default/album.html

@@ -35,6 +35,7 @@
 							%for alternate_album in alternate_albums:
 								<%
 									track_count = len(myDB.select("SELECT * from alltracks WHERE ReleaseID=?", [alternate_album['ReleaseID']]))
+									mb_link = "http://musicbrainz.org/release/" + alternate_album['ReleaseID']
 									have_track_count = len(myDB.select("SELECT * from alltracks WHERE ReleaseID=? AND Location IS NOT NULL", [alternate_album['ReleaseID']]))
 									if alternate_album['AlbumID'] == alternate_album['ReleaseID']:
 										alternate_album_name = "Headphones Default Release (" + str(alternate_album['ReleaseDate']) + ") [" + str(have_track_count) + "/" + str(track_count) + " tracks]"
@@ -42,7 +43,7 @@
 										alternate_album_name = alternate_album['AlbumTitle'] + " (" + alternate_album['ReleaseCountry'] + ", " + str(alternate_album['ReleaseDate']) + ", " + alternate_album['ReleaseFormat'] + ") [" + str(have_track_count) + "/" + str(track_count) + " tracks]"
 									
 								%>
-								<a href="#" onclick="doAjaxCall('switchAlbum?AlbumID=${album['AlbumID']}&ReleaseID=${alternate_album['ReleaseID']}', $(this), 'table');" data-success="Switched release to: ${alternate_album_name}">${alternate_album_name}</a><br>
+								<a href="#" onclick="doAjaxCall('switchAlbum?AlbumID=${album['AlbumID']}&ReleaseID=${alternate_album['ReleaseID']}', $(this), 'table');" data-success="Switched release to: ${alternate_album_name}">${alternate_album_name}</a><a href="${mb_link}" target="_blank">MB</a><br>
 							%endfor
 						%endif
 					</div>
@@ -337,6 +338,21 @@
         });
     }
 
+    jQuery.extend( jQuery.fn.dataTableExt.oSort, {
+		"title-numeric-pre": function ( a ) {
+			var x = a.match(/title="*(-?[0-9\.]+)/)[1];
+			return parseFloat( x );
+		},
+
+		"title-numeric-asc": function ( a, b ) {
+			return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+		},
+
+        "title-numeric-desc": function ( a, b ) {
+            return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+        }
+    } );
+
 	$(document).ready(function()	{
 		getAlbumInfo();
 		initThisPage();

data/interfaces/default/artist.html

@@ -70,6 +70,7 @@
 				<th id="albumname">Name</th>
 				<th id="reldate">Date</th>
 				<th id="type">Type</th>
+				<th id="score">Metacritic</th>
 				<th id="status">Status</th>
 				<th id="have">Have</th>
 				<th id="bitrate">Bitrate</th>
@@ -125,6 +126,7 @@
 				<td id="albumname"><a href="albumPage?AlbumID=${album['AlbumID']}">${album['AlbumTitle']}</a></td>
 				<td id="reldate">${album['ReleaseDate']}</td>
 				<td id="type">${album['Type']}</td>
+				<td id="score">${album['CriticScore']}/${album['UserScore']}</td>
 				<td id="status">${album['Status']}
 				%if album['Status'] == 'Skipped' or album['Status'] == 'Ignored':
 					[<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}',$(this),'table')" data-success="'${album['AlbumTitle']}' added to Wanted list">want</a>]
@@ -218,7 +220,7 @@
 							$("li:gt(4)", this).hide(); /* :gt() is zero-indexed */
 							$("li:nth-child(5)", this).after("<br><li class='more'><a href='#'>More...</a></li>"); /* :nth-child() is one-indexed */
 						  });
-						  $("li.more a").live("click", function() {
+						  $("li.more").on("click", 'a', function() {
 							var li = $(this).parents("li:first");
 							li.parent().children().show();
 							li.remove();
@@ -283,6 +285,7 @@
 				{ "sType": "date" },
 				null,
 				null,
+				null,
 				{ "sType": "title-numeric"},
 				null,
 				null

data/interfaces/default/base.html

@@ -31,12 +31,12 @@
 <body>
 	<div id="container">
 		<div id="ajaxMsg" class="ajaxMsg"></div>
-		% if not headphones.CURRENT_VERSION:
+		% if headphones.CONFIG.CHECK_GITHUB and not headphones.CURRENT_VERSION:
 			<div id="updatebar">
 				You're running an unknown version of Headphones. <a href="update">Update</a> or
 				<a href="#" onclick="$('#updatebar').slideUp('slow');">Close</a>
 			</div>
-		% elif headphones.CURRENT_VERSION != headphones.LATEST_VERSION and headphones.COMMITS_BEHIND > 0 and headphones.INSTALL_TYPE != 'win':
+		% elif headphones.CONFIG.CHECK_GITHUB and headphones.CURRENT_VERSION != headphones.LATEST_VERSION and headphones.COMMITS_BEHIND > 0 and headphones.INSTALL_TYPE != 'win':
 			<div id="updatebar">
 				A <a href="https://github.com/${headphones.CONFIG.GIT_USER}/headphones/compare/${headphones.CURRENT_VERSION}...${headphones.LATEST_VERSION}"> newer version</a> is available. You're ${headphones.COMMITS_BEHIND} commits behind. <a href="update">Update</a> or <a href="#" onclick="$('#updatebar').slideUp('slow');">Close</a>
 			</div>
@@ -62,6 +62,7 @@
 					<select name="type" id="search_type">
 						<option value="artist">Artist</option>
 						<option value="album">Album</option>
+						<option value="series">Series</option>
 					</select>
 					<input type="submit" value="Search"/>
 				</form>

data/interfaces/default/css/style.css

@@ -325,7 +325,7 @@ form fieldset small.heading {
   margin-top: -15px;
 }
 form .row {
-  font-family: Helvetica, Arial;
+  font-family: Helvetica, Arial, sans-serif;
   margin-bottom: 10px;
 }
 form .row label {
@@ -408,6 +408,18 @@ form .checkbox small {
 form .indent input {
   margin-left: 15px;
 }
+form .suboptions {
+  margin-left: 15px;
+}
+.option{
+  font-size: 15px;
+  font-weight: 600;
+  vertical-align: middle;
+}
+input.bigcheck[type="checkbox"] {
+  width: 16px;
+  height: 16px;
+}
 ul,
 ol {
   margin-left: 2em;
@@ -1420,6 +1432,13 @@ div#artistheader h2 a {
   line-height: 10px;
   height: 10px;
 }
+.nopad {
+  margin-bottom: 0px !important;
+  margin-top: 0px !important;
+  padding-top: 0px !important;
+  padding-bottom: 0px !important;
+  padding: 0px 0px !important;
+}
 #album_table th#albumname,
 #upcoming_table th#artistname,
 #wanted_table th#artistname {
@@ -1445,6 +1464,7 @@ div#artistheader h2 a {
 #album_table td#albumname,
 #album_table td#reldate,
 #album_table td#type,
+#album_table td#score,
 #track_table td#duration,
 #upcoming_table td#select,
 #upcoming_table td#status,
@@ -1455,6 +1475,7 @@ div#artistheader h2 a {
 }
 #album_table td#status,
 #album_table td#bitrate,
+#album_table td#score,
 #album_table td#albumformat,
 #album_table td#wantlossless {
   font-size: 13px;

data/interfaces/default/history.html

@@ -32,32 +32,37 @@
 			</tr>
 		</thead>
 		<tbody>
-		%for item in history:
-			<%
-				if item['Status'] == 'Processed':
-					grade = 'A'
-				elif item['Status'] == 'Snatched':
-					grade = 'C'
-				elif item['Status'] == 'Unprocessed':
-					grade = 'X'
-				elif item['Status'] == 'Frozen':
-					grade = 'X'
-				else:
-					grade = 'U'
+        %for item in history:
+        <%
+            if item['Status'] == 'Processed':
+                grade = 'A'
+            elif item['Status'] == 'Snatched':
+                grade = 'C'
+            elif item['Status'] == 'Unprocessed':
+                grade = 'X'
+            elif item['Status'] == 'Frozen':
+                grade = 'X'
+            else:
+                grade = 'U'
 
-				fileid = 'unknown'
-				if item['URL'].find('nzb') != -1:
-					fileid = 'nzb'
-				if item['URL'].find('torrent') != -1:
-					fileid = 'torrent'
-				if item['URL'].find('rutracker') != -1:
-					fileid = 'torrent'
-			%>
+            fileid = 'unknown'
+            if item['URL'].find('nzb') != -1:
+                fileid = 'nzb'
+            if item['URL'].find('torrent') != -1:
+                fileid = 'torrent'
+            if item['URL'].find('rutracker') != -1:
+                fileid = 'torrent'
+            if item['URL'].find('codeshy') != -1:
+                fileid = 'nzb'
+
+            folder = 'Folder: ' + item['FolderName']
+
+        %>
 			<tr class="grade${grade}">
 				<td id="dateadded">${item['DateAdded']}</td>
 				<td id="filename">${cgi.escape(item['Title'], quote=True)} [<a href="${item['URL']}">${fileid}</a>]<a href="albumPage?AlbumID=${item['AlbumID']}">[album page]</a></td>
 				<td id="size">${helpers.bytes_to_mb(item['Size'])}</td>
-				<td id="status">${item['Status']}</td>
+				<td title="${folder}" id="status">${item['Status']}</td>
 				<td id="action">[<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${item['AlbumID']}&redirect=history', $(this),'table')"  data-success="Retrying download of '${cgi.escape(item['Title'], quote=True)}'">retry</a>][<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${item['AlbumID']}&new=True&redirect=history',$(this),'table')"  data-success="Looking for a new version of '${cgi.escape(item['Title'], quote=True)}'">new</a>]</td>
 				<td id="delete"><a href="#" onclick="doAjaxCall('clearhistory?date_added=${item['DateAdded']}&title=${cgi.escape(item['Title'], quote=True)}',$(this),'table')" data-success="${cgi.escape(item['Title'], quote=True)} cleared from history"><img src="interfaces/default/images/trashcan.png" height="18" width="18" id="trashcan" title="Clear this item from the history"></a>
 			</tr>

data/interfaces/default/logs.html

@@ -7,6 +7,7 @@
     <div id="subhead_container">
         <div id="subhead_menu">
             <a class="menu_link_edit" href="clearLogs"><i class="fa fa-trash-o"></i> Clear log</a>
+            <a class="menu_link_edit" href="toggleVerbose"><i class="fa fa-pencil"></i> Toggle Debug Log</a>
         </div>
     </div>
 </%def>

data/interfaces/default/manage.html

@@ -137,28 +137,27 @@
 					        No empty artists found.
 					    %endif
 					</div>
-				<a href="#" onclick="doAjaxCall('forcePostProcess',$(this))" data-success="Post-Processor is being loaded" data-error="Error during Post-Processing"><i class="fa fa-wrench fa-fw"></i> Force Post-Process Albums in Download Folder</a>
+					<div id="post_process">
+					<a href="#" class="btnOpenDialog"><i class="fa fa-wrench fa-fw"></i> Force Post-Process Albums in Download Folder</a>
+					</div>
 			</div>
 		</fieldset>
-		<form action="forcePostProcess" method="GET">
-		            <fieldset>
-		            <div class="row">
-		                <label>Force Post-Process Albums in Alternate Folder</label>
-					    <input type="text" value="" name="dir" id="dir" size="50" /><input type="submit" />
-		            </div>
-		            </fieldset>
-		      
-		</form>
-		<form action="forcePostProcess" method="GET">
-		            <fieldset>
-		            <div class="row">
-		                <label>Post-Process Single Folder</label>
-					    <input type="text" value="" name="album_dir" id="album_dir" size="50" /><input type="submit" />
-		            </div>
-		            </fieldset>
-		      
-		</form>
-		
+		<fieldset>
+            <div class="row" id="post_process_alternate">
+                <label>Force Post-Process Albums in Alternate Folder</label>
+			    <input type="text" value="" name="dir" id="dir" size="50" />
+			    <input type="button" class="btnOpenDialog" value="Submit" />
+            </div>
+        </fieldset>
+
+        <fieldset>
+            <div class="row" id="post_process_single">
+                <label>Post-Process Single Folder</label>
+			    <input type="text" value="" name="album_dir" id="album_dir" size="50" />
+			    <input type="button" class="btnOpenDialog" value="Submit" />
+            </div>
+        </fieldset>
+
 	</div>
 
 
@@ -177,8 +176,7 @@
 		</fieldset>
 		
 	</div>
-
-
+	<div id="dialog-confirm"></div>
 
 </div>
 </%def>
@@ -187,6 +185,55 @@
 		function addScanAction() {
 			$('#autoadd').append('<input type="hidden" name="scan" value=1 />');
 		};
+
+		function fnOpenNormalDialog(id) {
+			if (id === "post_process"){
+				var url = "forcePostProcess?"
+			}
+			if (id === "post_process_alternate"){
+				var dir = $('#dir').val();
+				var url = "forcePostProcess?dir=" + dir + "&"
+			}
+			if (id === "post_process_single"){
+				var dir = $('#album_dir').val();
+				var url = "forcePostProcess?album_dir=" + dir + "&"
+			}
+			var t = $('<a data-success="Post-Processor is being loaded" data-error="Error during Post-Processing">');
+		    $("#dialog-confirm").html("Do you want to keep the original folder(s) after post-processing? If you click no, the folders still might be kept depending on your global settings");
+
+		    // Define the Dialog and its properties.
+		    $("#dialog-confirm").dialog({
+		        resizable: false,
+		        modal: true,
+		        title: "Keep original folder(s)?",
+		        height: 170,
+		        width: 400,
+		        buttons: {
+		            "Yes": function () {
+		                $(this).dialog('close');
+		                doAjaxCall(url + "keep_original_folder=True", t);
+		            },
+		                "No": function () {
+		                $(this).dialog('close');
+		                doAjaxCall(url + "keep_original_folder=False", t);
+		            }
+		        }
+		    });
+		}
+
+		$('.btnOpenDialog').click(function(e){
+			e.preventDefault();
+			var parentId = $(this).closest('div').prop('id');
+			fnOpenNormalDialog(parentId);
+		});
+
+		function callback(value) {
+		    if (value) {
+		        alert("Confirmed");
+		    } else {
+		        alert("Rejected");
+		    }
+		}
 		
 		function initThisPage() {
 			$('#manage_albums').click(function() {

data/interfaces/default/searchresults.html

@@ -17,8 +17,12 @@
                     <th id="reldate">Date</th>
                     <th id="scoresmall">Score</th>
                     <th id="mbrelid" style="display:none;"</th>
-                %else:
+                %elif type == 'artist':
                     <th id="artistname">Artist Name</th>
+                    <th id="score">Score</th>
+                %else:
+                    <th id="seriesname">Series Name</th>
+                    <th id="type">Type</th>
                     <th id="score">Score</th>
 				%endif
                 <th id="mb"></th>
@@ -40,8 +44,10 @@
 				<tr class="grade${grade}">
 				%if type == 'album':
 					<td id="albumart" style=" text-align: center; vertical-align: middle;"><div id="artistImg"><img title="${result['albumid']}" class="albumArt" height="50" width="50" onerror="tryCCA(this, '${caa_group_url}')"></div></td>
-                %else:
+                %elif type == 'artist':
 					<td id="albumart"><div id="artistImg"><img title="${result['id']}" class="albumArt" height="50" width="50"></div></td>
+                %else:
+                    <td id="albumart"></td>
 				%endif
 				%if type == 'album':
 					<td id="albumname"><a href="addReleaseById?rid=${result['albumid']}&rgid=${result['rgid']}" title="${albuminfo}">${result['title']}</a></td>
@@ -52,9 +58,14 @@
                     <td id="score"><a href="${result['albumurl']} "title="View on MusicBrainz"><div class="bar"><div class="score" style="width: ${result['score']}px">${result['score']}</div></div></a></td>
                     <td id="musicbrainz" style=" text-align: center; line-height: 0; vertical-align: middle;"><a href="${result['albumurl']}"><img src="interfaces/default/images/MusicBrainz_Album_Icon.png" title="View on MusicBrainz" height="20" width="20"></a></td>
                     <td id="mbrelid" style="display:none;">${result['albumid']}</td>
-                %else:
+                %elif type == 'artist':
                     <td id="artistname"><a href="addArtist?artistid=${result['id']}" title="${result['uniquename']}">${result['uniquename']}</a></td>
                     <td id="score"><a href="${result['url']} "title="View on MusicBrainz"><div class="bar"><div class="score" style="width: ${result['score']}px">${result['score']}</div></div></a></td>
+                    <td id="musicbrainz" style=" text-align: center; line-height: 0; vertical-align: middle;"><a href="${result['url']}"><img src="interfaces/default/images/MusicBrainz_Artist_Icon.png" title="View on MusicBrainz" height="20" width="20"></a></td>
+                %else:
+                    <td id="seriesname"><a href="addSeries?seriesid=${result['id']}" title="${result['uniquename']}">${result['uniquename']}</a></td>
+                    <td id="type">${result['type']}</td>
+                    <td id="score"><a href="${result['url']} "title="View on MusicBrainz"><div class="bar"><div class="score" style="width: ${result['score']}px">${result['score']}</div></div></a></td>
                     <td id="musicbrainz" style=" text-align: center; line-height: 0; vertical-align: middle;"><a href="${result['url']}"><img src="interfaces/default/images/MusicBrainz_Artist_Icon.png" title="View on MusicBrainz" height="20" width="20"></a></td>
 				%endif
 				</tr>

headphones/__init__.py

@@ -176,7 +176,7 @@ def initialize(config_file):
                              version_lock_file, e)
 
         # Check for new versions
-        if CONFIG.CHECK_GITHUB_ON_STARTUP:
+        if CONFIG.CHECK_GITHUB and CONFIG.CHECK_GITHUB_ON_STARTUP:
             try:
                 LATEST_VERSION = versioncheck.checkGithub()
             except:
@@ -288,11 +288,12 @@ def initialize_scheduler():
         schedule_job(updater.dbUpdate, 'MusicBrainz Update', hours=hours, minutes=0)
 
         #Update check
-        if CONFIG.CHECK_GITHUB_INTERVAL:
-            minutes = CONFIG.CHECK_GITHUB_INTERVAL
-        else:
-            minutes = 0
-        schedule_job(versioncheck.checkGithub, 'Check GitHub for updates', hours=0, minutes=minutes)
+        if CONFIG.CHECK_GITHUB:
+            if CONFIG.CHECK_GITHUB_INTERVAL:
+                minutes = CONFIG.CHECK_GITHUB_INTERVAL
+            else:
+                minutes = 0
+            schedule_job(versioncheck.checkGithub, 'Check GitHub for updates', hours=0, minutes=minutes)
 
         # Remove Torrent + data if Post Processed and finished Seeding
         minutes = CONFIG.TORRENT_REMOVAL_INTERVAL
@@ -352,12 +353,12 @@ def dbcheck():
     conn = sqlite3.connect(DB_FILE)
     c = conn.cursor()
     c.execute(
-        'CREATE TABLE IF NOT EXISTS artists (ArtistID TEXT UNIQUE, ArtistName TEXT, ArtistSortName TEXT, DateAdded TEXT, Status TEXT, IncludeExtras INTEGER, LatestAlbum TEXT, ReleaseDate TEXT, AlbumID TEXT, HaveTracks INTEGER, TotalTracks INTEGER, LastUpdated TEXT, ArtworkURL TEXT, ThumbURL TEXT, Extras TEXT)')
+        'CREATE TABLE IF NOT EXISTS artists (ArtistID TEXT UNIQUE, ArtistName TEXT, ArtistSortName TEXT, DateAdded TEXT, Status TEXT, IncludeExtras INTEGER, LatestAlbum TEXT, ReleaseDate TEXT, AlbumID TEXT, HaveTracks INTEGER, TotalTracks INTEGER, LastUpdated TEXT, ArtworkURL TEXT, ThumbURL TEXT, Extras TEXT, Type TEXT, MetaCritic TEXT)')
     # ReleaseFormat here means CD,Digital,Vinyl, etc. If using the default
     # Headphones hybrid release, ReleaseID will equal AlbumID (AlbumID is
     # releasegroup id)
     c.execute(
-        'CREATE TABLE IF NOT EXISTS albums (ArtistID TEXT, ArtistName TEXT, AlbumTitle TEXT, AlbumASIN TEXT, ReleaseDate TEXT, DateAdded TEXT, AlbumID TEXT UNIQUE, Status TEXT, Type TEXT, ArtworkURL TEXT, ThumbURL TEXT, ReleaseID TEXT, ReleaseCountry TEXT, ReleaseFormat TEXT, SearchTerm TEXT)')
+        'CREATE TABLE IF NOT EXISTS albums (ArtistID TEXT, ArtistName TEXT, AlbumTitle TEXT, AlbumASIN TEXT, ReleaseDate TEXT, DateAdded TEXT, AlbumID TEXT UNIQUE, Status TEXT, Type TEXT, ArtworkURL TEXT, ThumbURL TEXT, ReleaseID TEXT, ReleaseCountry TEXT, ReleaseFormat TEXT, SearchTerm TEXT, CriticScore TEXT, UserScore TEXT)')
     # Format here means mp3, flac, etc.
     c.execute(
         'CREATE TABLE IF NOT EXISTS tracks (ArtistID TEXT, ArtistName TEXT, AlbumTitle TEXT, AlbumASIN TEXT, AlbumID TEXT, TrackTitle TEXT, TrackDuration, TrackID TEXT, TrackNumber INTEGER, Location TEXT, BitRate INTEGER, CleanName TEXT, Format TEXT, ReleaseID TEXT)')
@@ -583,6 +584,26 @@ def dbcheck():
     except sqlite3.OperationalError:
         c.execute('ALTER TABLE albums ADD COLUMN SearchTerm TEXT DEFAULT NULL')
 
+    try:
+        c.execute('SELECT CriticScore from albums')
+    except sqlite3.OperationalError:
+        c.execute('ALTER TABLE albums ADD COLUMN CriticScore TEXT DEFAULT NULL')
+
+    try:
+        c.execute('SELECT UserScore from albums')
+    except sqlite3.OperationalError:
+        c.execute('ALTER TABLE albums ADD COLUMN UserScore TEXT DEFAULT NULL')
+
+    try:
+        c.execute('SELECT Type from artists')
+    except sqlite3.OperationalError:
+        c.execute('ALTER TABLE artists ADD COLUMN Type TEXT DEFAULT NULL')
+
+    try:
+        c.execute('SELECT MetaCritic from artists')
+    except sqlite3.OperationalError:
+        c.execute('ALTER TABLE artists ADD COLUMN MetaCritic TEXT DEFAULT NULL')
+
     conn.commit()
     c.close()
 

headphones/api.py

@@ -21,8 +21,8 @@ import json
 cmd_list = ['getIndex', 'getArtist', 'getAlbum', 'getUpcoming', 'getWanted', 'getSnatched', 'getSimilar', 'getHistory', 'getLogs',
             'findArtist', 'findAlbum', 'addArtist', 'delArtist', 'pauseArtist', 'resumeArtist', 'refreshArtist',
             'addAlbum', 'queueAlbum', 'unqueueAlbum', 'forceSearch', 'forceProcess', 'forceActiveArtistsUpdate', 
-            'getVersion', 'checkGithub','shutdown', 'restart', 'update', 'getArtistArt', 'getAlbumArt', 
-            'getArtistInfo', 'getAlbumInfo', 'getArtistThumb', 'getAlbumThumb', 
+            'getVersion', 'checkGithub', 'shutdown', 'restart', 'update', 'getArtistArt', 'getAlbumArt', 
+            'getArtistInfo', 'getAlbumInfo', 'getArtistThumb', 'getAlbumThumb', 'clearLogs',
             'choose_specific_download', 'download_specific_release']
 
 
@@ -176,7 +176,13 @@ class Api(object):
         return
 
     def _getLogs(self, **kwargs):
-        pass
+        self.data = headphones.LOG_LIST
+        return
+
+    def _clearLogs(self, **kwargs):
+        headphones.LOG_LIST = []
+        self.data = 'Cleared log'
+        return
 
     def _findArtist(self, **kwargs):
         if 'name' not in kwargs:

headphones/config.py

@@ -44,16 +44,30 @@ _CONFIG_DEFINITIONS = {
     'CUE_SPLIT': (int, 'General', 1),
     'CUE_SPLIT_FLAC_PATH': (str, 'General', ''),
     'CUE_SPLIT_SHNTOOL_PATH': (str, 'General', ''),
+    'CUSTOMAUTH': (int, 'General', 0),
     'CUSTOMHOST': (str, 'General', 'localhost'),
+    'CUSTOMPASS': (str, 'General', ''),
     'CUSTOMPORT': (int, 'General', 5000),
     'CUSTOMSLEEP': (int, 'General', 1),
+    'CUSTOMUSER': (str, 'General', ''),
     'DELETE_LOSSLESS_FILES': (int, 'General', 1),
     'DESTINATION_DIR': (str, 'General', ''),
     'DETECT_BITRATE': (int, 'General', 0),
+    'DO_NOT_PROCESS_UNMATCHED': (int, 'General', 0),
     'DOWNLOAD_DIR': (str, 'General', ''),
     'DOWNLOAD_SCAN_INTERVAL': (int, 'General', 5),
     'DOWNLOAD_TORRENT_DIR': (str, 'General', ''),
     'DO_NOT_OVERRIDE_GIT_BRANCH': (int, 'General', 0),
+    'EMAIL_ENABLED': (int, 'Email', 0),
+    'EMAIL_FROM': (str, 'Email', ''),
+    'EMAIL_TO': (str, 'Email', ''),
+    'EMAIL_SMTP_SERVER': (str, 'Email', ''),
+    'EMAIL_SMTP_USER': (str, 'Email', ''),
+    'EMAIL_SMTP_PASSWORD': (str, 'Email', ''),
+    'EMAIL_SMTP_PORT': (int, 'Email', 25),
+    'EMAIL_SSL': (int, 'Email', 0),
+    'EMAIL_TLS': (int, 'Email', 0),
+    'EMAIL_ONSNATCH': (int, 'Email', 0),
     'EMBED_ALBUM_ART': (int, 'General', 0),
     'EMBED_LYRICS': (int, 'General', 0),
     'ENABLE_HTTPS': (int, 'General', 0),
@@ -68,6 +82,7 @@ _CONFIG_DEFINITIONS = {
     'ENCODER_PATH': (str, 'General', ''),
     'EXTRAS': (str, 'General', ''),
     'EXTRA_NEWZNABS': (list, 'Newznab', ''),
+    'EXTRA_TORZNABS': (list, 'Torznab', ''),
     'FILE_FORMAT': (str, 'General', 'Track Artist - Album [Year] - Title'),
     'FILE_PERMISSIONS': (str, 'General', '0644'),
     'FILE_UNDERSCORES': (int, 'General', 0),
@@ -86,12 +101,14 @@ _CONFIG_DEFINITIONS = {
     'HPUSER': (str, 'General', ''),
     'HTTPS_CERT': (str, 'General', ''),
     'HTTPS_KEY': (str, 'General', ''),
-    'HTTP_HOST': (str, 'General', '0.0.0.0'),
+    'HTTP_HOST': (str, 'General', 'localhost'),
     'HTTP_PASSWORD': (str, 'General', ''),
     'HTTP_PORT': (int, 'General', 8181),
     'HTTP_PROXY': (int, 'General', 0),
     'HTTP_ROOT': (str, 'General', '/'),
     'HTTP_USERNAME': (str, 'General', ''),
+    'IDTAG': (int, 'Beets', 0),
+    'IGNORE_CLEAN_RELEASES': (int, 'General', 0),
     'IGNORED_WORDS': (str, 'General', ''),
     'IGNORED_FOLDERS': (list, 'Advanced', []),
     'IGNORED_FILES': (list, 'Advanced', []),
@@ -139,6 +156,7 @@ _CONFIG_DEFINITIONS = {
     'NZBSORG_HASH': (str, 'NZBsorg', ''),
     'NZBSORG_UID': (str, 'NZBsorg', ''),
     'NZB_DOWNLOADER': (int, 'General', 0),
+    'OFFICIAL_RELEASES_ONLY': (int, 'General', 0),
     'OMGWTFNZBS': (int, 'omgwtfnzbs', 0),
     'OMGWTFNZBS_APIKEY': (str, 'omgwtfnzbs', ''),
     'OMGWTFNZBS_UID': (str, 'omgwtfnzbs', ''),
@@ -160,6 +178,7 @@ _CONFIG_DEFINITIONS = {
     'PLEX_SERVER_HOST': (str, 'Plex', ''),
     'PLEX_UPDATE': (int, 'Plex', 0),
     'PLEX_USERNAME': (str, 'Plex', ''),
+    'PLEX_TOKEN': (str, 'Plex', ''),
     'PREFERRED_BITRATE': (str, 'General', ''),
     'PREFERRED_BITRATE_ALLOW_LOSSLESS': (int, 'General', 0),
     'PREFERRED_BITRATE_HIGH_BUFFER': (int, 'General', 0),
@@ -185,6 +204,7 @@ _CONFIG_DEFINITIONS = {
     'PUSHOVER_PRIORITY': (int, 'Pushover', 0),
     'RENAME_FILES': (int, 'General', 0),
     'REPLACE_EXISTING_FOLDERS': (int, 'General', 0),
+    'KEEP_ORIGINAL_FOLDER': (int, 'General', 0),
     'REQUIRED_WORDS': (str, 'General', ''),
     'RUTRACKER': (int, 'Rutracker', 0),
     'RUTRACKER_PASSWORD': (str, 'Rutracker', ''),
@@ -201,6 +221,8 @@ _CONFIG_DEFINITIONS = {
     'SONGKICK_ENABLED': (int, 'Songkick', 1),
     'SONGKICK_FILTER_ENABLED': (int, 'Songkick', 0),
     'SONGKICK_LOCATION': (str, 'Songkick', ''),
+    'STRIKE': (int, 'Strike', 0),
+    'STRIKE_RATIO': (str, 'Strike', ''),
     'SUBSONIC_ENABLED': (int, 'Subsonic', 0),
     'SUBSONIC_HOST': (str, 'Subsonic', ''),
     'SUBSONIC_PASSWORD': (str, 'Subsonic', ''),
@@ -209,6 +231,10 @@ _CONFIG_DEFINITIONS = {
     'TORRENTBLACKHOLE_DIR': (str, 'General', ''),
     'TORRENT_DOWNLOADER': (int, 'General', 0),
     'TORRENT_REMOVAL_INTERVAL': (int, 'General', 720),
+    'TORZNAB': (int, 'Torznab', 0),
+    'TORZNAB_APIKEY': (str, 'Torznab', ''),
+    'TORZNAB_ENABLED': (int, 'Torznab', 1),
+    'TORZNAB_HOST': (str, 'Torznab', ''),
     'TRANSMISSION_HOST': (str, 'Transmission', ''),
     'TRANSMISSION_PASSWORD': (str, 'Transmission', ''),
     'TRANSMISSION_USERNAME': (str, 'Transmission', ''),
@@ -224,6 +250,7 @@ _CONFIG_DEFINITIONS = {
     'UTORRENT_PASSWORD': (str, 'uTorrent', ''),
     'UTORRENT_USERNAME': (str, 'uTorrent', ''),
     'VERIFY_SSL_CERT': (bool_int, 'Advanced', 1),
+    'WAIT_UNTIL_RELEASE_DATE' : (int, 'General', 0),
     'WAFFLES': (int, 'Waffles', 0),
     'WAFFLES_PASSKEY': (str, 'Waffles', ''),
     'WAFFLES_RATIO': (str, 'Waffles', ''),
@@ -332,6 +359,25 @@ class Config(object):
             extra_newznabs.append(item)
         self.EXTRA_NEWZNABS = extra_newznabs
 
+    def get_extra_torznabs(self):
+        """ Return the extra torznab tuples """
+        extra_torznabs = list(
+            itertools.izip(*[itertools.islice(self.EXTRA_TORZNABS, i, None, 3)
+            for i in range(3)])
+        )
+        return extra_torznabs
+
+    def clear_extra_torznabs(self):
+        """ Forget about the configured extra torznabs """
+        self.EXTRA_TORZNABS = []
+
+    def add_extra_torznab(self, torznab):
+        """ Add a new extra torznab """
+        extra_torznabs = self.EXTRA_TORZNABS
+        for item in torznab:
+            extra_torznabs.append(item)
+        self.EXTRA_TORZNABS = extra_torznabs
+
     def __getattr__(self, name):
         """
         Returns something from the ini unless it is a real property

headphones/cuesplit.py

@@ -62,9 +62,7 @@ WAVE_FILE_TYPE_BY_EXTENSION = {
     '.flac': 'Free Lossless Audio Codec'
 }
 
-# TODO: Only alow flac for now
-#SHNTOOL_COMPATIBLE = ('Waveform Audio', 'WavPack', 'Free Lossless Audio Codec')
-SHNTOOL_COMPATIBLE = ('Free Lossless Audio Codec')
+#SHNTOOL_COMPATIBLE = ("Free Lossless Audio Codec", "Waveform Audio", "Monkey's Audio")
 
 # TODO: Make this better!
 # this module-level variable is bad. :(
@@ -109,10 +107,10 @@ def split_baby(split_file, split_cmd):
         env['PATH'] += os.pathsep + headphones.CONFIG.CUE_SPLIT_FLAC_PATH
 
     process = subprocess.Popen(split_cmd, startupinfo=startupinfo,
-
-    stdin=open(os.devnull, 'rb'), stdout=subprocess.PIPE,
-    stderr=subprocess.PIPE, env=env)
+                               stdin=open(os.devnull, 'rb'), stdout=subprocess.PIPE,
+                               stderr=subprocess.PIPE, env=env)
     stdout, stderr = process.communicate()
+
     if process.returncode:
         logger.error('Split failed for %s', split_file.decode(headphones.SYS_ENCODING, 'replace'))
         out = stdout if stdout else stderr
@@ -592,11 +590,10 @@ def split(albumpath):
             splitter = 'shntool'
 
     if splitter == 'shntool' and not check_splitter(splitter):
-            raise ValueError('Command not found, ensure shntool with FLAC or xld (OS X) installed')
+            raise ValueError('Command not found, ensure shntool or xld installed')
 
-    # Determine if file can be split (only flac allowed for shntool)
-    if 'xld' in splitter and wave.name_ext not in WAVE_FILE_TYPE_BY_EXTENSION.keys() or \
-            wave.type not in SHNTOOL_COMPATIBLE:
+    # Determine if file can be split
+    if wave.name_ext not in WAVE_FILE_TYPE_BY_EXTENSION.keys():
         raise ValueError('Cannot split, audio file has unsupported extension')
 
     # Split with xld
@@ -640,7 +637,7 @@ def split(albumpath):
         cmd.extend(['-f'])
         cmd.extend([SPLIT_FILE_NAME])
         cmd.extend(['-o'])
-        cmd.extend(['flac'])
+        cmd.extend([wave.name_ext.lstrip('.')])
         cmd.extend([wave.name])
         split = split_baby(wave.name, cmd)
         os.remove(SPLIT_FILE_NAME)
@@ -652,9 +649,9 @@ def split(albumpath):
                 logger.info('Tagging %s...', t.name)
                 t.tag()
 
-        # rename FLAC files
-        if split and CUE_META.count_tracks() == len(base_dir.tracks(ext='.flac', split=True)):
-            for t in base_dir.tracks(ext='.flac', split=True):
+        # rename files
+        if split and CUE_META.count_tracks() == len(base_dir.tracks(ext=wave.name_ext, split=True)):
+            for t in base_dir.tracks(ext=wave.name_ext, split=True):
                 if t.name != t.filename():
                     logger.info('Renaming %s to %s...', t.name, t.filename())
                     os.rename(t.name, t.filename())

headphones/helpers.py

@@ -149,7 +149,7 @@ def get_age(date):
 
     try:
         days_old = int(split_date[0]) * 365 + int(split_date[1]) * 30 + int(split_date[2])
-    except IndexError:
+    except (IndexError,ValueError):
         days_old = False
 
     return days_old
@@ -173,15 +173,15 @@ def piratesize(size):
     factor = float(split[0])
     unit = split[1].upper()
     
-    if unit == 'MiB':
+    if unit == 'MIB':
         size = factor * 1048576
     elif unit == 'MB':
         size = factor * 1000000
-    elif unit == 'GiB':
+    elif unit == 'GIB':
         size = factor * 1073741824
     elif unit == 'GB':
         size = factor * 1000000000
-    elif unit == 'KiB':
+    elif unit == 'KIB':
         size = factor * 1024
     elif unit == 'KB':
         size = factor * 1000
@@ -519,7 +519,7 @@ def get_downloaded_track_list(albumpath):
     return downloaded_track_list
 
 
-def preserve_torrent_direcory(albumpath):
+def preserve_torrent_directory(albumpath):
     """
     Copy torrent directory to headphones-modified to keep files for seeding.
     """

headphones/importer.py

@@ -13,7 +13,7 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import logger, helpers, db, mb, lastfm
+from headphones import logger, helpers, db, mb, lastfm, metacritic
 
 from beets.mediafile import MediaFile
 
@@ -111,7 +111,7 @@ def addArtistIDListToDB(artistidlist):
         addArtisttoDB(artistid)
 
 
-def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
+def addArtisttoDB(artistid, extrasonly=False, forcefull=False, type="artist"):
 
     # Putting this here to get around the circular import. We're using this to update thumbnails for artist/albums
     from headphones import cache
@@ -142,12 +142,19 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                         "Status": "Loading",
                         "IncludeExtras": headphones.CONFIG.INCLUDE_EXTRAS,
                         "Extras": headphones.CONFIG.EXTRAS}
+        if type=="series":
+            newValueDict['Type'] = "series"
     else:
         newValueDict = {"Status": "Loading"}
+        if dbartist["Type"] == "series":
+            type = "series"
 
     myDB.upsert("artists", newValueDict, controlValueDict)
 
-    artist = mb.getArtist(artistid, extrasonly)
+    if type=="series":
+        artist = mb.getSeries(artistid)
+    else:
+        artist = mb.getArtist(artistid, extrasonly)
 
     if artist and artist.get('artist_name') in blacklisted_special_artist_names:
         logger.warn('Cannot import blocked special purpose artist: %s' % artist.get('artist_name'))
@@ -488,6 +495,9 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
     logger.info(u"Seeing if we need album art for: %s" % artist['artist_name'])
     cache.getThumb(ArtistID=artistid)
 
+    logger.info(u"Fetching Metacritic reviews for: %s" % artist['artist_name'])
+    metacritic.update(artistid, artist['artist_name'], artist['releasegroups'])
+
     if errors:
         logger.info("[%s] Finished updating artist: %s but with errors, so not marking it as updated in the database" % (artist['artist_name'], artist['artist_name']))
     else:

headphones/lastfm.py

@@ -79,7 +79,7 @@ def getSimilar():
                 try:
                     artist_mbid = artist["mbid"]
                     artist_name = artist["name"]
-                except TypeError:
+                except KeyError:
                     continue
 
                 if not any(artist_mbid in x for x in results):
@@ -116,7 +116,7 @@ def getArtists():
         return
 
     logger.info("Fetching artists from Last.FM for username: %s", headphones.CONFIG.LASTFM_USERNAME)
-    data = request_lastfm("library.getartists", limit=10000, user=headphones.CONFIG.LASTFM_USERNAME)
+    data = request_lastfm("library.getartists", limit=1000, user=headphones.CONFIG.LASTFM_USERNAME)
 
     if data and "artists" in data:
         artistlist = []

headphones/librarysync.py

@@ -23,7 +23,7 @@ from headphones import db, logger, helpers, importer, lastfm
 # You can scan a single directory and append it to the current library by
 # specifying append=True, ArtistID and ArtistName.
 def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
-    cron=False):
+    cron=False, artistScan=False):
 
     if cron and not headphones.CONFIG.LIBRARYSCAN:
         return
@@ -36,7 +36,7 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
 
     # If we're appending a dir, it's coming from the post processor which is
     # already bytestring
-    if not append:
+    if not append or artistScan:
         dir = dir.encode(headphones.SYS_ENCODING)
 
     if not os.path.isdir(dir):
@@ -53,7 +53,7 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
         tracks = myDB.select('SELECT Location from alltracks WHERE Location IS NOT NULL UNION SELECT Location from tracks WHERE Location IS NOT NULL')
 
         for track in tracks:
-            encoded_track_string = track['Location'].encode(headphones.SYS_ENCODING)
+            encoded_track_string = track['Location'].encode(headphones.SYS_ENCODING, 'replace')
             if not os.path.isfile(encoded_track_string):
                 myDB.action('UPDATE tracks SET Location=?, BitRate=?, Format=? WHERE Location=?', [None, None, None, track['Location']])
                 myDB.action('UPDATE alltracks SET Location=?, BitRate=?, Format=? WHERE Location=?', [None, None, None, track['Location']])
@@ -287,7 +287,7 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
 
     logger.info('Completed matching tracks from directory: %s' % dir.decode(headphones.SYS_ENCODING, 'replace'))
 
-    if not append:
+    if not append or artistScan:
         logger.info('Updating scanned artist track counts')
 
         # Clean up the new artist list
@@ -334,7 +334,7 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
                 for artist in artist_list:
                     myDB.action('INSERT OR IGNORE INTO newartists VALUES (?)', [artist])
 
-        if headphones.CONFIG.DETECT_BITRATE:
+        if headphones.CONFIG.DETECT_BITRATE and bitrates:
             headphones.CONFIG.PREFERRED_BITRATE = sum(bitrates) / len(bitrates) / 1000
 
     else:
@@ -346,6 +346,8 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
 
     if not append:
         update_album_status()
+
+    if not append and not artistScan:
         lastfm.getSimilar()
 
     logger.info('Library scan complete')
@@ -374,7 +376,7 @@ def update_album_status(AlbumID=None):
             album_completion = 0
             logger.info('Album %s does not have any tracks in database' % album['AlbumTitle'])
 
-        if album_completion >= headphones.CONFIG.ALBUM_COMPLETION_PCT and album['Status'] == 'Skipped':
+        if album_completion >= headphones.CONFIG.ALBUM_COMPLETION_PCT:
             new_album_status = "Downloaded"
 
         # I don't think we want to change Downloaded->Skipped.....

headphones/mb.py

@@ -47,9 +47,11 @@ def startmb():
     elif headphones.CONFIG.MIRROR == "custom":
         mbhost = headphones.CONFIG.CUSTOMHOST
         mbport = int(headphones.CONFIG.CUSTOMPORT)
+        mbuser = headphones.CONFIG.CUSTOMUSER
+        mbpass = headphones.CONFIG.CUSTOMPASS
         sleepytime = int(headphones.CONFIG.CUSTOMSLEEP)
     elif headphones.CONFIG.MIRROR == "headphones":
-        mbhost = "144.76.94.239"
+        mbhost = "codeshy.com"
         mbport = 8181
         mbuser = headphones.CONFIG.HPUSER
         mbpass = headphones.CONFIG.HPPASS
@@ -69,12 +71,16 @@ def startmb():
         mb_lock.minimum_delta = sleepytime
 
     # Add headphones credentials
-    if headphones.CONFIG.MIRROR == "headphones":
-        if not mbuser and mbpass:
-            logger.warn("No username or password set for VIP server")
+    if headphones.CONFIG.MIRROR == "headphones" or headphones.CONFIG.CUSTOMAUTH:
+        if not mbuser or not mbpass:
+            logger.warn("No username or password set for MusicBrainz server")
         else:
             musicbrainzngs.hpauth(mbuser, mbpass)
 
+    # Let us know if we disable custom authentication
+    if not headphones.CONFIG.CUSTOMAUTH and headphones.CONFIG.MIRROR == "custom":
+        musicbrainzngs.disable_hpauth()
+
     logger.debug('Using the following server values: MBHost: %s, MBPort: %i, Sleep Interval: %i', mbhost, mbport, sleepytime)
 
     return True
@@ -129,7 +135,6 @@ def findArtist(name, limit=1):
                      })
      return artistlist
 
-
 def findRelease(name, limit=1, artist=None):
     releaselist = []
     releaseResults = None
@@ -207,12 +212,45 @@ def findRelease(name, limit=1, artist=None):
                     })
     return releaselist
 
+def findSeries(name, limit=1):
+     serieslist = []
+     seriesResults = None
+
+     chars = set('!?*-')
+     if any((c in chars) for c in name):
+         name = '"' + name + '"'
+
+     criteria = {'series': name.lower()}
+
+     with mb_lock:
+         try:
+             seriesResults = musicbrainzngs.search_series(limit=limit, **criteria)['series-list']
+         except musicbrainzngs.WebServiceError as e:
+             logger.warn('Attempt to query MusicBrainz for %s failed (%s)' % (name, str(e)))
+             mb_lock.snooze(5)
+
+     if not seriesResults:
+         return False
+     for result in seriesResults:
+         if 'disambiguation' in result:
+             uniquename = unicode(result['name'] + " (" + result['disambiguation'] + ")")
+         else:
+             uniquename = unicode(result['name'])
+         serieslist.append({
+                     'uniquename': uniquename,
+                     'name': unicode(result['name']),
+                     'type': unicode(result['type']),
+                     'id': unicode(result['id']),
+                     'url': unicode("http://musicbrainz.org/series/" + result['id']),#probably needs to be changed
+                     'score': int(result['ext:score'])
+                     })
+     return serieslist
 
 def getArtist(artistid, extrasonly=False):
     artist_dict = {}
     artist = None
     try:
-        limit = 200
+        limit = 100
         with mb_lock:
             artist = musicbrainzngs.get_artist_by_id(artistid)['artist']
         newRgs = None
@@ -282,7 +320,7 @@ def getArtist(artistid, extrasonly=False):
             mb_extras_list = []
 
             try:
-                limit = 200
+                limit = 100
                 newRgs = None
                 while newRgs is None or len(newRgs) >= limit:
                     with mb_lock:
@@ -310,6 +348,38 @@ def getArtist(artistid, extrasonly=False):
     artist_dict['releasegroups'] = releasegroups
     return artist_dict
 
+def getSeries(seriesid):
+    series_dict = {}
+    series = None
+    try:
+        with mb_lock:
+            series = musicbrainzngs.get_series_by_id(seriesid,includes=['release-group-rels'])['series']
+    except musicbrainzngs.WebServiceError as e:
+        logger.warn('Attempt to retrieve series information from MusicBrainz failed for seriesid: %s (%s)' % (seriesid, str(e)))
+        mb_lock.snooze(5)
+    except Exception as e:
+        pass
+
+    if not series:
+        return False
+
+    if 'disambiguation' in series:
+        series_dict['artist_name'] = unicode(series['name'] + " (" + unicode(series['disambiguation']) + ")")
+    else:
+        series_dict['artist_name'] = unicode(series['name'])
+
+    releasegroups = []
+
+    for rg in series['release_group-relation-list']:
+        releasegroup = rg['release-group']
+        releasegroups.append({
+            'title':releasegroup['title'],
+            'date':releasegroup['first-release-date'],
+            'id':releasegroup['id'],
+            'type':rg['type']
+            })
+    series_dict['releasegroups'] = releasegroups
+    return series_dict
 
 def getReleaseGroup(rgid):
     """
@@ -396,6 +466,11 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
 
     myDB = db.DBConnection()
     results = []
+
+    release_status = "official"
+    if includeExtras and not headphones.CONFIG.OFFICIAL_RELEASES_ONLY:
+        release_status = []
+
     try:
         limit = 100
         newResults = None
@@ -404,6 +479,7 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
                 newResults = musicbrainzngs.browse_releases(
                     release_group=rgid,
                     includes=['artist-credits', 'labels', 'recordings', 'release-groups', 'media'],
+                    release_status = release_status,
                     limit=limit,
                     offset=len(results))
             if 'release-list' not in newResults:
@@ -443,10 +519,6 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
     num_new_releases = 0
 
     for releasedata in results:
-        #releasedata.get will return None if it doesn't have a status
-        #all official releases should have the Official status included
-        if not includeExtras and releasedata.get('status') != 'Official':
-            continue
 
         release = {}
         rel_id_check = releasedata['id']

headphones/metacritic.py

@@ -0,0 +1,88 @@
+#  This file is part of Headphones.
+#
+#  Headphones is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  Headphones is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
+
+import re
+import json
+import headphones
+
+from headphones import db, helpers, logger, request
+from headphones.common import USER_AGENT
+
+def update(artistid, artist_name,release_groups):
+    """ Pretty simple and crude function to find the artist page on metacritic,
+    then parse that page to get critic & user scores for albums"""
+
+    # First let's modify the artist name to fit the metacritic convention.
+    # We could just do a search, then take the top result, but at least this will
+    # cut down on api calls. If it's ineffective then we'll switch to search
+
+    replacements = {" & " : " ", "." : ""}
+    mc_artist_name = helpers.replace_all(artist_name.lower(),replacements)
+
+    mc_artist_name = mc_artist_name.replace(" ","-")
+
+    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2243.2 Safari/537.36'}
+
+    url = "http://www.metacritic.com/person/" + mc_artist_name + "?filter-options=music&sort_options=date&num_items=100"
+
+    res = request.request_soup(url, headers=headers)
+
+    rows = None
+
+    try:
+        table = res.find("table", class_="credits person_credits")
+        rows = table.tbody.find_all('tr')
+    except:
+        logger.info("Unable to get metacritic scores for: %s" % artist_name)
+
+    myDB = db.DBConnection()
+    artist = myDB.action('SELECT * FROM artists WHERE ArtistID=?', [artistid]).fetchone()
+
+    score_list = []
+
+    # If we couldn't get anything from MetaCritic for whatever reason,
+    # let's try to load scores from the db
+    if not rows:
+        if artist['MetaCritic']:
+            score_list = json.loads(artist['MetaCritic'])
+        else:
+            return
+
+    # If we did get scores, let's update the db with them
+    else:
+        for row in rows:
+            title = row.a.string
+            scores = row.find_all("span")
+            critic_score = scores[0].string
+            user_score = scores[1].string
+            score_dict = {'title':title,'critic_score':critic_score,'user_score':user_score}
+            score_list.append(score_dict)
+
+        # Save scores to the database
+        controlValueDict = {"ArtistID": artistid}
+        newValueDict = {'MetaCritic':json.dumps(score_list)}
+        myDB.upsert("artists", newValueDict, controlValueDict)
+
+    for score in score_list:
+        title = score['title']
+        # Iterate through the release groups we got passed to see if we can find
+        # a match
+        for rg in release_groups:
+            if rg['title'].lower() == title.lower():
+                critic_score = score['critic_score']
+                user_score = score['user_score']
+                controlValueDict = {"AlbumID": rg['id']}
+                newValueDict = {'CriticScore':critic_score,'UserScore':user_score}
+                myDB.upsert("albums", newValueDict, controlValueDict)

headphones/notifiers.py

@@ -34,6 +34,11 @@ import json
 import oauth2 as oauth
 import pythontwitter as twitter
 
+from email.mime.text import MIMEText
+import smtplib
+import email.utils
+
+
 class GROWL(object):
     """
     Growl notifications, for OS X.
@@ -311,34 +316,32 @@ class Plex(object):
         self.client_hosts = headphones.CONFIG.PLEX_CLIENT_HOST
         self.username = headphones.CONFIG.PLEX_USERNAME
         self.password = headphones.CONFIG.PLEX_PASSWORD
+        self.token = headphones.CONFIG.PLEX_TOKEN
 
     def _sendhttp(self, host, command):
 
-        username = self.username
-        password = self.password
+        url = host + '/xbmcCmds/xbmcHttp/?' + command
 
-        url_command = urllib.urlencode(command)
-
-        url = host + '/xbmcCmds/xbmcHttp/?' + url_command
-
-        req = urllib2.Request(url)
-
-        if password:
-            base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
-            req.add_header("Authorization", "Basic %s" % base64string)
-
-        logger.info('Plex url: %s' % url)
-
-        try:
-            handle = urllib2.urlopen(req)
-        except Exception as e:
-            logger.warn('Error opening Plex url: %s' % e)
-            return
-
-        response = handle.read().decode(headphones.SYS_ENCODING)
+        if self.password:
+            response = request.request_response(url, auth=(self.username, self.password))
+        else:
+            response = request.request_response(url)
 
         return response
 
+    def _sendjson(self, host, method, params={}):
+        data = [{'id': 0, 'jsonrpc': '2.0', 'method': method, 'params': params}]
+        headers = {'Content-Type': 'application/json'}
+        url = host + '/jsonrpc'
+
+        if self.password:
+            response = request.request_json(url, method="post", data=json.dumps(data), headers=headers, auth=(self.username, self.password))
+        else:
+            response = request.request_json(url, method="post", data=json.dumps(data), headers=headers)
+
+        if response:
+            return response[0]['result']
+
     def update(self):
 
         # From what I read you can't update the music library on a per directory or per path basis
@@ -349,13 +352,15 @@ class Plex(object):
         for host in hosts:
             logger.info('Sending library update command to Plex Media Server@ ' + host)
             url = "%s/library/sections" % host
-            try:
-                xml_sections = minidom.parse(urllib.urlopen(url))
-            except IOError, e:
-                logger.warn("Error while trying to contact Plex Media Server: %s" % e)
-                return False
+            if self.token:
+                params = {'X-Plex-Token': self.token}
+            else:
+                params = False
+
+            r = request.request_minidom(url, params=params)
+
+            sections = r.getElementsByTagName('Directory')
 
-            sections = xml_sections.getElementsByTagName('Directory')
             if not sections:
                 logger.info(u"Plex Media Server not running on: " + host)
                 return False
@@ -363,11 +368,7 @@ class Plex(object):
             for s in sections:
                 if s.getAttribute('type') == "artist":
                     url = "%s/library/sections/%s/refresh" % (host, s.getAttribute('key'))
-                    try:
-                        urllib.urlopen(url)
-                    except Exception as e:
-                        logger.warn("Error updating library section for Plex Media Server: %s" % e)
-                        return False
+                    request.request_response(url, params=params)
 
     def notify(self, artist, album, albumartpath):
 
@@ -378,17 +379,24 @@ class Plex(object):
         time = "3000" # in ms
 
         for host in hosts:
-            logger.info('Sending notification command to Plex Media Server @ ' + host)
+            logger.info('Sending notification command to Plex client @ ' + host)
             try:
-                notification = header + "," + message + "," + time + "," + albumartpath
-                notifycommand = {'command': 'ExecBuiltIn', 'parameter': 'Notification(' + notification + ')'}
-                request = self._sendhttp(host, notifycommand)
+                version = self._sendjson(host, 'Application.GetProperties', {'properties': ['version']})['version']['major']
+
+                if version < 12: #Eden
+                    notification = header + "," + message + "," + time + "," + albumartpath
+                    notifycommand = {'command': 'ExecBuiltIn', 'parameter': 'Notification(' + notification + ')'}
+                    request = self._sendhttp(host, notifycommand)
+
+                else: #Frodo
+                    params = {'title': header, 'message': message, 'displaytime': int(time), 'image': albumartpath}
+                    request = self._sendjson(host, 'GUI.ShowNotification', params)
 
                 if not request:
                     raise Exception
 
-            except:
-                logger.warn('Error sending notification request to Plex Media Server')
+            except Exception:
+                logger.error('Error sending notification request to Plex client @ ' + host)
 
 
 class NMA(object):
@@ -435,52 +443,30 @@ class PUSHBULLET(object):
         self.apikey = headphones.CONFIG.PUSHBULLET_APIKEY
         self.deviceid = headphones.CONFIG.PUSHBULLET_DEVICEID
 
-    def conf(self, options):
-        return cherrypy.config['config'].get('PUSHBULLET', options)
-
-    def notify(self, message, event):
+    def notify(self, message, status):
         if not headphones.CONFIG.PUSHBULLET_ENABLED:
             return
 
-        http_handler = HTTPSConnection("api.pushbullet.com")
+        url = "https://api.pushbullet.com/v2/pushes"
 
         data = {'type': "note",
                 'title': "Headphones",
-                'body': message.encode("utf-8")}
+                'body': message + ': ' + status}
 
-        http_handler.request("POST",
-                                "/v2/pushes",
-                                headers={'Content-type': "application/json",
-                                            'Authorization': 'Basic %s' % base64.b64encode(headphones.CONFIG.PUSHBULLET_APIKEY + ":")},
-                                body=json.dumps(data))
-        response = http_handler.getresponse()
-        request_status = response.status
-        logger.debug(u"PushBullet response status: %r" % request_status)
-        logger.debug(u"PushBullet response headers: %r" % response.getheaders())
-        logger.debug(u"PushBullet response body: %r" % response.read())
+        if self.deviceid:
+            data['device_iden'] = self.deviceid
 
-        if request_status == 200:
-                logger.info(u"PushBullet notifications sent.")
-                return True
-        elif request_status >= 400 and request_status < 500:
-                logger.info(u"PushBullet request failed: %s" % response.reason)
-                return False
+        headers={'Content-type': "application/json",
+                 'Authorization': 'Bearer ' + headphones.CONFIG.PUSHBULLET_APIKEY}
+
+        response = request.request_json(url, method="post", headers=headers, data=json.dumps(data))
+
+        if response:
+            logger.info(u"PushBullet notifications sent.")
+            return True
         else:
-                logger.info(u"PushBullet notification failed serverside.")
-                return False
-
-    def updateLibrary(self):
-        #For uniformity reasons not removed
-        return
-
-    def test(self, apikey, deviceid):
-
-        self.enabled = True
-        self.apikey = apikey
-        self.deviceid = deviceid
-
-        self.notify('Main Screen Activate', 'Test Message')
-
+            logger.info(u"PushBullet notification failed.")
+            return False
 
 class PUSHALOT(object):
 
@@ -578,7 +564,7 @@ class PUSHOVER(object):
         if not headphones.CONFIG.PUSHOVER_ENABLED:
             return
 
-        http_handler = HTTPSConnection("api.pushover.net")
+        url = "https://api.pushover.net/1/messages.json"
 
         data = {'token': self.application_token,
                 'user': headphones.CONFIG.PUSHOVER_KEYS,
@@ -586,25 +572,16 @@ class PUSHOVER(object):
                 'message': message.encode("utf-8"),
                 'priority': headphones.CONFIG.PUSHOVER_PRIORITY}
 
-        http_handler.request("POST",
-                                "/1/messages.json",
-                                headers={'Content-type': "application/x-www-form-urlencoded"},
-                                body=urlencode(data))
-        response = http_handler.getresponse()
-        request_status = response.status
-        logger.debug(u"Pushover response status: %r" % request_status)
-        logger.debug(u"Pushover response headers: %r" % response.getheaders())
-        logger.debug(u"Pushover response body: %r" % response.read())
+        headers = {'Content-type': "application/x-www-form-urlencoded"}
 
-        if request_status == 200:
-                logger.info(u"Pushover notifications sent.")
-                return True
-        elif request_status >= 400 and request_status < 500:
-                logger.info(u"Pushover request failed: %s" % response.reason)
-                return False
+        response = request.request_response(url, method="POST", headers=headers, data=data)
+
+        if response:
+            logger.info(u"Pushover notifications sent.")
+            return True
         else:
-                logger.info(u"Pushover notification failed.")
-                return False
+            logger.error(u"Pushover notification failed.")
+            return False
 
     def updateLibrary(self):
         #For uniformity reasons not removed
@@ -727,23 +704,32 @@ class OSX_NOTIFY(object):
             self.objc = __import__("objc")
             self.AppKit = __import__("AppKit")
         except:
+            logger.warn('OS X Notification: Cannot import objc or AppKit')
             return False
 
     def swizzle(self, cls, SEL, func):
-        old_IMP = cls.instanceMethodForSelector_(SEL)
+        old_IMP = getattr(cls, SEL, None)
+        if old_IMP is None:
+            old_IMP = cls.instanceMethodForSelector_(SEL)
 
         def wrapper(self, *args, **kwargs):
             return func(self, old_IMP, *args, **kwargs)
-        new_IMP = self.objc.selector(wrapper, selector=old_IMP.selector,
-            signature=old_IMP.signature)
-        self.objc.classAddMethod(cls, SEL, new_IMP)
+
+        new_IMP = self.objc.selector(
+            wrapper,
+            selector=old_IMP.selector,
+            signature=old_IMP.signature
+        )
+        self.objc.classAddMethod(cls, SEL.encode(), new_IMP)
 
     def notify(self, title, subtitle=None, text=None, sound=True, image=None):
 
         try:
-            self.swizzle(self.objc.lookUpClass('NSBundle'),
-                b'bundleIdentifier',
-                self.swizzled_bundleIdentifier)
+            self.swizzle(
+                self.objc.lookUpClass('NSBundle'),
+                'bundleIdentifier',
+                self.swizzled_bundleIdentifier
+            )
 
             NSUserNotification = self.objc.lookUpClass('NSUserNotification')
             NSUserNotificationCenter = self.objc.lookUpClass('NSUserNotificationCenter')
@@ -827,3 +813,34 @@ class SubSonicNotifier(object):
         # Invoke request
         request.request_response(self.host + "musicFolderSettings.view?scanNow",
             auth=(self.username, self.password))
+
+class Email(object):
+
+    def notify(self, subject, message):
+
+        message = MIMEText(message, 'plain', "utf-8")
+        message['Subject'] = subject
+        message['From'] = email.utils.formataddr(('Headphones', headphones.CONFIG.EMAIL_FROM))
+        message['To'] = headphones.CONFIG.EMAIL_TO
+
+        try:
+            if (headphones.CONFIG.EMAIL_SSL):
+                mailserver = smtplib.SMTP_SSL(headphones.CONFIG.EMAIL_SMTP_SERVER, headphones.CONFIG.EMAIL_SMTP_PORT)
+            else:
+                mailserver = smtplib.SMTP(headphones.CONFIG.EMAIL_SMTP_SERVER, headphones.CONFIG.EMAIL_SMTP_PORT)
+
+            if (headphones.CONFIG.EMAIL_TLS):
+                mailserver.starttls()
+
+            mailserver.ehlo()
+
+            if headphones.CONFIG.EMAIL_SMTP_USER:
+                mailserver.login(headphones.CONFIG.EMAIL_SMTP_USER, headphones.CONFIG.EMAIL_SMTP_PASSWORD)
+
+            mailserver.sendmail(headphones.CONFIG.EMAIL_FROM, headphones.CONFIG.EMAIL_TO, message.as_string())
+            mailserver.quit()
+            return True
+
+        except Exception, e:
+            logger.warn('Error sending Email: %s' % e)
+            return False

headphones/nzbget.py

@@ -32,20 +32,20 @@ from headphones import logger
 def sendNZB(nzb):
 
     addToTop = False
-    nzbgetXMLrpc = "%(username)s:%(password)s@%(host)s/xmlrpc"
+    nzbgetXMLrpc = "%(protocol)s://%(username)s:%(password)s@%(host)s/xmlrpc"
 
-    if headphones.CONFIG.NZBGET_HOST is None:
+    if not headphones.CONFIG.NZBGET_HOST:
         logger.error(u"No NZBget host found in configuration. Please configure it.")
         return False
 
     if headphones.CONFIG.NZBGET_HOST.startswith('https://'):
-        nzbgetXMLrpc = 'https://' + nzbgetXMLrpc
-        headphones.CONFIG.NZBGET_HOST.replace('https://', '', 1)
+        protocol = 'https'
+        host = headphones.CONFIG.NZBGET_HOST.replace('https://', '', 1)
     else:
-        nzbgetXMLrpc = 'http://' + nzbgetXMLrpc
-        headphones.CONFIG.NZBGET_HOST.replace('http://', '', 1)
+        protocol = 'http'
+        host = headphones.CONFIG.NZBGET_HOST.replace('http://', '', 1)
 
-    url = nzbgetXMLrpc % {"host": headphones.CONFIG.NZBGET_HOST, "username": headphones.CONFIG.NZBGET_USERNAME, "password": headphones.CONFIG.NZBGET_PASSWORD}
+    url = nzbgetXMLrpc % {"protocol": protocol, "host": host, "username": headphones.CONFIG.NZBGET_USERNAME, "password": headphones.CONFIG.NZBGET_PASSWORD}
 
     nzbGetRPC = xmlrpclib.ServerProxy(url)
     try:

headphones/postprocessor.py

@@ -23,6 +23,7 @@ import itertools
 import headphones
 
 from beets import autotag
+from beets import config as beetsconfig
 from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
 from beetsplug import lyrics as beetslyrics
 
@@ -34,7 +35,7 @@ postprocessor_lock = threading.Lock()
 
 
 def checkFolder():
-    logger.info("Checking download folder for completed downloads (only snatched ones).")
+    logger.debug("Checking download folder for completed downloads (only snatched ones).")
 
     with postprocessor_lock:
         myDB = db.DBConnection()
@@ -48,7 +49,7 @@ def checkFolder():
                     download_dir = headphones.CONFIG.DOWNLOAD_TORRENT_DIR
 
                 album_path = os.path.join(download_dir, album['FolderName']).encode(headphones.SYS_ENCODING, 'replace')
-                logger.info("Checking if %s exists" % album_path)
+                logger.debug("Checking if %s exists" % album_path)
 
                 if os.path.exists(album_path):
                     logger.info('Found "' + album['FolderName'] + '" in ' + album['Kind'] + ' download folder. Verifying....')
@@ -56,9 +57,9 @@ def checkFolder():
             else:
                 logger.info("No folder name found for " + album['Title'])
 
-    logger.info("Checking download folder finished.")
+    logger.debug("Checking download folder finished.")
 
-def verify(albumid, albumpath, Kind=None, forced=False):
+def verify(albumid, albumpath, Kind=None, forced=False, keep_original_folder=False):
 
     myDB = db.DBConnection()
     release = myDB.action('SELECT * from albums WHERE AlbumID=?', [albumid]).fetchone()
@@ -187,15 +188,9 @@ def verify(albumid, albumpath, Kind=None, forced=False):
     # Split cue
     if headphones.CONFIG.CUE_SPLIT and downloaded_cuecount and downloaded_cuecount >= len(downloaded_track_list):
         if headphones.CONFIG.KEEP_TORRENT_FILES and Kind == "torrent":
-            albumpath = helpers.preserve_torrent_direcory(albumpath)
+            albumpath = helpers.preserve_torrent_directory(albumpath)
         if albumpath and helpers.cue_split(albumpath):
             downloaded_track_list = helpers.get_downloaded_track_list(albumpath)
-        else:
-            myDB.action('UPDATE snatched SET status = "Unprocessed" WHERE status NOT LIKE "Seed%" and AlbumID=?', [albumid])
-            processed = re.search(r' \(Unprocessed\)(?:\[\d+\])?', albumpath)
-            if not processed:
-                renameUnprocessedFolder(albumpath, tag="Unprocessed")
-            return
 
     # test #1: metadata - usually works
     logger.debug('Verifying metadata...')
@@ -221,7 +216,7 @@ def verify(albumid, albumpath, Kind=None, forced=False):
         logger.debug('Matching metadata album: %s with album name: %s' % (metaalbum, dbalbum))
 
         if metaartist == dbartist and metaalbum == dbalbum:
-            doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list, Kind)
+            doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list, Kind, keep_original_folder)
             return
 
     # test #2: filenames
@@ -239,7 +234,7 @@ def verify(albumid, albumpath, Kind=None, forced=False):
             logger.debug('Checking if track title: %s is in file name: %s' % (dbtrack, filetrack))
 
             if dbtrack in filetrack:
-                doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list, Kind)
+                doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list, Kind, keep_original_folder)
                 return
 
     # test #3: number of songs and duration
@@ -271,7 +266,7 @@ def verify(albumid, albumpath, Kind=None, forced=False):
             logger.debug('Database track duration: %i' % db_track_duration)
             delta = abs(downloaded_track_duration - db_track_duration)
             if delta < 240:
-                doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list, Kind)
+                doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list, Kind, keep_original_folder)
                 return
 
     logger.warn(u'Could not identify album: %s. It may not be the intended album.' % albumpath.decode(headphones.SYS_ENCODING, 'replace'))
@@ -281,11 +276,11 @@ def verify(albumid, albumpath, Kind=None, forced=False):
         renameUnprocessedFolder(albumpath, tag="Unprocessed")
 
 
-def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list, Kind=None):
+def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list, Kind=None, keep_original_folder=False):
 
     logger.info('Starting post-processing for: %s - %s' % (release['ArtistName'], release['AlbumTitle']))
     # Check to see if we're preserving the torrent dir
-    if headphones.CONFIG.KEEP_TORRENT_FILES and Kind == "torrent" and 'headphones-modified' not in albumpath:
+    if (headphones.CONFIG.KEEP_TORRENT_FILES and Kind == "torrent" and 'headphones-modified' not in albumpath) or headphones.CONFIG.KEEP_ORIGINAL_FOLDER or keep_original_folder:
         new_folder = os.path.join(albumpath, 'headphones-modified'.encode(headphones.SYS_ENCODING, 'replace'))
         logger.info("Copying files to 'headphones-modified' subfolder to preserve downloaded files for seeding")
         try:
@@ -306,7 +301,7 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
                 if any(files.lower().endswith('.' + x.lower()) for x in headphones.MEDIA_FORMATS):
                     downloaded_track_list.append(os.path.join(r, files))
 
-    # Check if files are valid media files and are writeable, before the steps
+    # Check if files are valid media files and are writable, before the steps
     # below are executed. This simplifies errors and prevents unfinished steps.
     for downloaded_track in downloaded_track_list:
         try:
@@ -336,7 +331,7 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
                     fp.seek(0)
             except IOError as e:
                 logger.debug("Write check exact error: %s", e)
-                logger.error("Track file is not writeable. This is required " \
+                logger.error("Track file is not writable. This is required " \
                     "for some post processing steps: %s. Not continuing.",
                     downloaded_track.decode(headphones.SYS_ENCODING, "replace"))
                 return
@@ -377,7 +372,9 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
         addAlbumArt(artwork, albumpath, release)
 
     if headphones.CONFIG.CORRECT_METADATA:
-        correctMetadata(albumid, release, downloaded_track_list)
+        correctedMetadata = correctMetadata(albumid, release, downloaded_track_list)
+        if not correctedMetadata and headphones.CONFIG.DO_NOT_PROCESS_UNMATCHED:
+            return
 
     if headphones.CONFIG.EMBED_LYRICS:
         embedLyrics(downloaded_track_list)
@@ -480,7 +477,7 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
     if headphones.CONFIG.PUSHBULLET_ENABLED:
         logger.info(u"PushBullet request")
         pushbullet = notifiers.PUSHBULLET()
-        pushbullet.notify(pushmessage, "Download and Postprocessing completed")
+        pushbullet.notify(pushmessage, statusmessage)
 
     if headphones.CONFIG.TWITTER_ENABLED:
         logger.info(u"Sending Twitter notification")
@@ -513,6 +510,12 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
         mpc = notifiers.MPC()
         mpc.notify()
 
+    if headphones.CONFIG.EMAIL_ENABLED:
+        logger.info(u"Sending Email notification")
+        email = notifiers.Email()
+        subject = release['ArtistName'] + ' - ' + release['AlbumTitle']
+        email.notify(subject, "Download and Postprocessing completed")
+
 
 def embedAlbumArt(artwork, downloaded_track_list):
     logger.info('Embedding album art')
@@ -594,7 +597,6 @@ def renameNFO(albumpath):
                 except Exception as e:
                     logger.error(u'Could not rename file: %s. Error: %s' % (os.path.join(r, file).decode(headphones.SYS_ENCODING, 'replace'), e))
 
-
 def moveFiles(albumpath, release, tracks):
     logger.info("Moving files: %s" % albumpath)
     try:
@@ -857,16 +859,16 @@ def correctMetadata(albumid, release, downloaded_track_list):
             cur_artist, cur_album, candidates, rec = autotag.tag_album(items, search_artist=helpers.latinToAscii(release['ArtistName']), search_album=helpers.latinToAscii(release['AlbumTitle']))
         except Exception as e:
             logger.error('Error getting recommendation: %s. Not writing metadata', e)
-            return
-        if str(rec) == 'recommendation.none':
+            return False
+        if str(rec) == 'Recommendation.none':
             logger.warn('No accurate album match found for %s, %s -  not writing metadata', release['ArtistName'], release['AlbumTitle'])
-            return
+            return False
 
         if candidates:
             dist, info, mapping, extra_items, extra_tracks = candidates[0]
         else:
             logger.warn('No accurate album match found for %s, %s -  not writing metadata', release['ArtistName'], release['AlbumTitle'])
-            return
+            return False
 
         logger.info('Beets recommendation for tagging items: %s' % rec)
 
@@ -874,13 +876,23 @@ def correctMetadata(albumid, release, downloaded_track_list):
 
         autotag.apply_metadata(info, mapping)
 
+        # Set ID3 tag version
+        if headphones.CONFIG.IDTAG:
+            beetsconfig['id3v23'] = True
+            logger.debug("Using ID3v2.3")
+        else:
+            beetsconfig['id3v23'] = False
+            logger.debug("Using ID3v2.4")
+
         for item in items:
             try:
                 item.write()
                 logger.info("Successfully applied metadata to: %s", item.path.decode(headphones.SYS_ENCODING, 'replace'))
             except Exception as e:
                 logger.warn("Error writing metadata to '%s': %s", item.path.decode(headphones.SYS_ENCODING, 'replace'), str(e))
+                return False
 
+        return True
 
 def embedLyrics(downloaded_track_list):
     logger.info('Adding lyrics')
@@ -1051,7 +1063,9 @@ def renameUnprocessedFolder(path, tag):
             return
 
 
-def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
+def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None, keep_original_folder=False):
+
+    logger.info('Force checking download folder for completed downloads')
 
     ignored = 0
 
@@ -1125,7 +1139,7 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
                 continue
             else:
                 logger.info('Found a match in the database: %s. Verifying to make sure it is the correct album', snatched['Title'])
-                verify(snatched['AlbumID'], folder, snatched['Kind'])
+                verify(snatched['AlbumID'], folder, snatched['Kind'], keep_original_folder=keep_original_folder)
                 continue
 
         # Attempt 2: strip release group id from filename
@@ -1142,10 +1156,10 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
             release = myDB.action('SELECT ArtistName, AlbumTitle, AlbumID from albums WHERE AlbumID=?', [rgid]).fetchone()
             if release:
                 logger.info('Found a match in the database: %s - %s. Verifying to make sure it is the correct album', release['ArtistName'], release['AlbumTitle'])
-                verify(release['AlbumID'], folder, forced=True)
+                verify(release['AlbumID'], folder, forced=True, keep_original_folder=keep_original_folder)
                 continue
             else:
-                logger.info('Found a (possibly) valid Musicbrainz realse group id in album folder name.')
+                logger.info('Found a (possibly) valid Musicbrainz release group id in album folder name.')
                 verify(rgid, folder, forced=True)
                 continue
 
@@ -1161,7 +1175,7 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
             release = myDB.action('SELECT AlbumID, ArtistName, AlbumTitle from albums WHERE ArtistName LIKE ? and AlbumTitle LIKE ?', [name, album]).fetchone()
             if release:
                 logger.info('Found a match in the database: %s - %s. Verifying to make sure it is the correct album', release['ArtistName'], release['AlbumTitle'])
-                verify(release['AlbumID'], folder)
+                verify(release['AlbumID'], folder, keep_original_folder=keep_original_folder)
                 continue
             else:
                 logger.info('Querying MusicBrainz for the release group id for: %s - %s', name, album)
@@ -1172,7 +1186,7 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
                     rgid = None
 
                 if rgid:
-                    verify(rgid, folder)
+                    verify(rgid, folder, keep_original_folder=keep_original_folder)
                     continue
                 else:
                     logger.info('No match found on MusicBrainz for: %s - %s', name, album)
@@ -1196,7 +1210,7 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
             release = myDB.action('SELECT AlbumID, ArtistName, AlbumTitle from albums WHERE ArtistName LIKE ? and AlbumTitle LIKE ?', [name, album]).fetchone()
             if release:
                 logger.info('Found a match in the database: %s - %s. Verifying to make sure it is the correct album', release['ArtistName'], release['AlbumTitle'])
-                verify(release['AlbumID'], folder)
+                verify(release['AlbumID'], folder, keep_original_folder=keep_original_folder)
                 continue
             else:
                 logger.info('Querying MusicBrainz for the release group id for: %s - %s', name, album)
@@ -1207,7 +1221,7 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
                     rgid = None
 
                 if rgid:
-                    verify(rgid, folder)
+                    verify(rgid, folder, keep_original_folder=keep_original_folder)
                     continue
                 else:
                     logger.info('No match found on MusicBrainz for: %s - %s', name, album)
@@ -1220,7 +1234,7 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
             release = myDB.action('SELECT AlbumID, ArtistName, AlbumTitle from albums WHERE AlbumTitle LIKE ?', [folder_basename]).fetchone()
             if release:
                 logger.info('Found a match in the database: %s - %s. Verifying to make sure it is the correct album', release['ArtistName'], release['AlbumTitle'])
-                verify(release['AlbumID'], folder)
+                verify(release['AlbumID'], folder, keep_original_folder=keep_original_folder)
                 continue
             else:
                 logger.info('Querying MusicBrainz for the release group id for: %s', folder_basename)
@@ -1231,7 +1245,7 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
                     rgid = None
 
                 if rgid:
-                    verify(rgid, folder)
+                    verify(rgid, folder, keep_original_folder=keep_original_folder)
                     continue
                 else:
                     logger.info('No match found on MusicBrainz for: %s - %s', name, album)

headphones/request.py

@@ -95,7 +95,7 @@ def request_response(url, method="get", auto_raise=True,
             "host is up and running.")
     except requests.Timeout:
         logger.error(
-            "Request timed out. The remote host did not respond timely.")
+            "Request timed out. The remote host did not respond in a timely manner.")
     except requests.HTTPError as e:
         if e.response is not None:
             if e.response.status_code >= 500:
@@ -206,7 +206,7 @@ def server_message(response):
     message = None
 
     # First attempt is to 'read' the response as HTML
-    if "text/html" in response.headers.get("content-type"):
+    if response.headers.get("content-type") and "text/html" in response.headers.get("content-type"):
         try:
             soup = BeautifulSoup(response.content, "html5lib")
         except Exception:

headphones/rutracker.py

@@ -0,0 +1,224 @@
+#!/usr/bin/env python
+
+import urllib
+import requests as requests
+from urlparse import urlparse
+from bs4 import BeautifulSoup
+
+import os
+import time
+import re
+
+import headphones
+from headphones import logger
+
+class Rutracker(object):
+
+    def __init__(self):
+        self.session = requests.session()
+        self.timeout = 60
+        self.loggedin = False
+        self.maxsize = 0
+        self.search_referer = 'http://rutracker.org/forum/tracker.php'
+
+    def logged_in(self):
+        return self.loggedin
+
+    def still_logged_in(self, html):
+        if not html or "action=\"http://login.rutracker.org/forum/login.php\">" in html:
+            return False
+        else:
+            return True
+
+    def login(self):
+        """
+        Logs in user
+        """
+
+        loginpage = 'http://login.rutracker.org/forum/login.php'
+        post_params = {
+            'login_username': headphones.CONFIG.RUTRACKER_USER,
+            'login_password': headphones.CONFIG.RUTRACKER_PASSWORD,
+            'login': b'\xc2\xf5\xee\xe4'  # '%C2%F5%EE%E4'
+        }
+
+        logger.info("Attempting to log in to rutracker...")
+
+        try:
+            r = self.session.post(loginpage, data=post_params, timeout=self.timeout)
+            # try again
+            if 'bb_data' not in r.cookies.keys():
+                time.sleep(10)
+                r = self.session.post(loginpage, data=post_params, timeout=self.timeout)
+            if r.status_code != 200:
+                logger.error("rutracker login returned status code %s" % r.status_code)
+                self.loggedin = False
+            else:
+                if 'bb_data' in r.cookies.keys():
+                    self.loggedin = True
+                    logger.info("Successfully logged in to rutracker")
+                else:
+                    logger.error("Could not login to rutracker, credentials maybe incorrect, site is down or too many attempts. Try again later")
+                    self.loggedin = False
+            return self.loggedin
+        except Exception as e:
+            logger.error("Unknown error logging in to rutracker: %s" % e)
+            self.loggedin = False
+            return self.loggedin
+
+    def searchurl(self, artist, album, year, format):
+        """
+        Return the search url
+        """
+
+        # Build search url
+        searchterm = ''
+        if artist != 'Various Artists':
+            searchterm = artist
+            searchterm = searchterm + ' '
+        searchterm = searchterm + album
+        searchterm = searchterm + ' '
+        searchterm = searchterm + year
+
+        if format == 'lossless':
+            format = '+lossless'
+            self.maxsize = 10000000000
+        elif format == 'lossless+mp3':
+            format = '+lossless||mp3||aac'
+            self.maxsize = 10000000000
+        else:
+            format = '+mp3||aac'
+            self.maxsize = 300000000
+
+        # sort by size, descending.
+        sort = '&o=7&s=2'
+
+        searchurl = "%s?nm=%s%s%s" % (self.search_referer, urllib.quote(searchterm), format, sort)
+
+        logger.info("Searching rutracker using term: %s", searchterm)
+
+        return searchurl
+
+    def search(self, searchurl):
+        """
+        Parse the search results and return valid torrent list
+        """
+
+        try:
+            headers = {'Referer': self.search_referer}
+            r = self.session.get(url=searchurl, headers=headers, timeout=self.timeout)
+
+            soup = BeautifulSoup(r.content, 'html5lib')
+
+            # Debug
+            #logger.debug (soup.prettify())
+
+            # Check if still logged in
+            if not self.still_logged_in(soup):
+                self.login()
+                r = self.session.get(url=searchurl, timeout=self.timeout)
+                soup = BeautifulSoup(r.content, 'html5lib')
+                if not self.still_logged_in(soup):
+                    logger.error("Error getting rutracker data")
+                    return None
+
+            # Process
+            rulist = []
+            i = soup.find('table', id='tor-tbl')
+            if not i:
+                logger.info("No valid results found from rutracker")
+                return None
+            minimumseeders = int(headphones.CONFIG.NUMBEROFSEEDERS) - 1
+
+            for item in zip(i.find_all(class_='hl-tags'),i.find_all(class_='dl-stub'),i.find_all(class_='seedmed')):
+                title = item[0].get_text()
+                url = item[1].get('href')
+                size_formatted = item[1].get_text()[:-2]
+                seeds = item[2].get_text()
+                size_parts = size_formatted.split()
+                size = float(size_parts[0])
+
+                if size_parts[1] == 'KB':
+                    size *= 1024
+                if size_parts[1] == 'MB':
+                    size *= 1024 ** 2
+                if size_parts[1] == 'GB':
+                    size *= 1024 ** 3
+                if size_parts[1] == 'TB':
+                    size *= 1024 ** 4
+
+                if size < self.maxsize and minimumseeders < int(seeds):
+                    logger.info('Found %s. Size: %s' % (title, size_formatted))
+                    #Torrent topic page
+                    torrent_id = dict([part.split('=') for part in urlparse(url)[4].split('&')])['t']
+                    topicurl = 'http://rutracker.org/forum/viewtopic.php?t=' + torrent_id
+                    rulist.append((title, size, topicurl, 'rutracker.org', 'torrent', True))
+                else:
+                    logger.info("%s is larger than the maxsize or has too little seeders for this category, skipping. (Size: %i bytes, Seeders: %i)" % (title, size, int(seeds)))
+
+            if not rulist:
+                logger.info("No valid results found from rutracker")
+
+            return rulist
+
+        except Exception as e:
+            logger.error("An unknown error occurred in the rutracker parser: %s" % e)
+            return None
+
+
+    def get_torrent_data(self, url):
+        """
+        return the .torrent data
+        """
+
+        torrent_id = dict([part.split('=') for part in urlparse(url)[4].split('&')])['t']
+        downloadurl = 'http://dl.rutracker.org/forum/dl.php?t=' + torrent_id
+        cookie = {'bb_dl': torrent_id}
+        try:
+            headers = {'Referer': url}
+            r = self.session.get(url=downloadurl, cookies=cookie, headers=headers, timeout=self.timeout)
+            return r.content
+        except Exception as e:
+            logger.error('Error getting torrent: %s', e)
+            return False
+
+
+    #TODO get this working in utorrent.py
+    def utorrent_add_file(self, data):
+
+        host = headphones.CONFIG.UTORRENT_HOST
+        if not host.startswith('http'):
+            host = 'http://' + host
+        if host.endswith('/'):
+            host = host[:-1]
+        if host.endswith('/gui'):
+            host = host[:-4]
+
+        base_url = host
+
+        url = base_url + '/gui/'
+        self.session.auth = (headphones.CONFIG.UTORRENT_USERNAME, headphones.CONFIG.UTORRENT_PASSWORD)
+
+        try:
+            r = self.session.get(url + 'token.html')
+        except Exception as e:
+            logger.error('Error getting token: %s', e)
+            return
+
+        if r.status_code == 401:
+            logger.debug('Error reaching utorrent')
+            return
+
+        regex = re.search(r'.+>([^<]+)</div></html>', r.text)
+        if regex is None:
+            logger.debug('Error reading token')
+            return
+
+        self.session.params = {'token': regex.group(1)}
+        files = {'torrent_file': ("", data)}
+
+        try:
+            self.session.post(url, params={'action': 'add-file'}, files=files)
+        except Exception as e:
+            logger.exception('Error adding file to utorrent %s', e)
+

headphones/sab.py

@@ -20,19 +20,21 @@
 import MultipartPostHandler
 import headphones
 import cookielib
-import urllib2
 import httplib
-import urllib
-import ast
 
 from headphones.common import USER_AGENT
-from headphones import logger
-from headphones import helpers
+from headphones import logger, helpers, request
 
 
-def sendNZB(nzb):
+def sab_api_call(request_type=None, params={}, **kwargs):
 
-    params = {}
+    if not headphones.CONFIG.SAB_HOST.startswith('http'):
+        headphones.CONFIG.SAB_HOST = 'http://' + headphones.CONFIG.SAB_HOST
+
+    if headphones.CONFIG.SAB_HOST.endswith('/'):
+        headphones.CONFIG.SAB_HOST = headphones.CONFIG.SAB_HOST[0:len(headphones.CONFIG.SAB_HOST) - 1]
+
+    url = headphones.CONFIG.SAB_HOST + "/" + "api?"
 
     if headphones.CONFIG.SAB_USERNAME:
         params['ma_username'] = headphones.CONFIG.SAB_USERNAME
@@ -40,9 +42,24 @@ def sendNZB(nzb):
         params['ma_password'] = headphones.CONFIG.SAB_PASSWORD
     if headphones.CONFIG.SAB_APIKEY:
         params['apikey'] = headphones.CONFIG.SAB_APIKEY
-    if headphones.CONFIG.SAB_CATEGORY:
+    
+    if request_type=='send_nzb' and headphones.CONFIG.SAB_CATEGORY:
         params['cat'] = headphones.CONFIG.SAB_CATEGORY
 
+    params['output']='json'
+
+    response = request.request_json(url, params=params, **kwargs)
+
+    if not response:
+        logger.error("Error connecting to SABnzbd on url: %s" % headphones.CONFIG.SAB_HOST)
+        return False
+    else:
+        logger.debug("Successfully connected to SABnzbd on url: %s" % headphones.CONFIG.SAB_HOST)
+        return response
+
+def sendNZB(nzb):
+
+    params = {}
     # if it's a normal result we just pass SAB the URL
     if nzb.resultType == "nzb":
         # for newzbin results send the ID to sab specifically
@@ -62,102 +79,41 @@ def sendNZB(nzb):
         # Sanitize the file a bit, since we can only use ascii chars with MultiPartPostHandler
         nzbdata = helpers.latinToAscii(nzb.extraInfo[0])
         params['mode'] = 'addfile'
-        multiPartParams = {"nzbfile": (helpers.latinToAscii(nzb.name) + ".nzb", nzbdata)}
+        files = {"nzbfile": (helpers.latinToAscii(nzb.name) + ".nzb", nzbdata)}
+        headers = {'User-Agent': USER_AGENT}
 
-    if not headphones.CONFIG.SAB_HOST.startswith('http'):
-        headphones.CONFIG.SAB_HOST = 'http://' + headphones.CONFIG.SAB_HOST
+    logger.info("Attempting to connect to SABnzbd on url: %s" % headphones.CONFIG.SAB_HOST)
+    if nzb.resultType == "nzb":
+        response = sab_api_call('send_nzb', params=params)
+    elif nzb.resultType == "nzbdata":
+        cookies = cookielib.CookieJar()
+        response = sab_api_call('send_nzb', params=params, method="post", files=files, cookies=cookies, headers=headers)
 
-    if headphones.CONFIG.SAB_HOST.endswith('/'):
-        headphones.CONFIG.SAB_HOST = headphones.CONFIG.SAB_HOST[0:len(headphones.CONFIG.SAB_HOST) - 1]
-
-    url = headphones.CONFIG.SAB_HOST + "/" + "api?" + urllib.urlencode(params)
-
-    try:
-
-        if nzb.resultType == "nzb":
-            f = urllib.urlopen(url)
-        elif nzb.resultType == "nzbdata":
-            cookies = cookielib.CookieJar()
-            opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookies),
-                                          MultipartPostHandler.MultipartPostHandler)
-
-            req = urllib2.Request(url,
-                                  multiPartParams,
-                                  headers={'User-Agent': USER_AGENT})
-
-            f = opener.open(req)
-
-    except (EOFError, IOError) as e:
-        logger.error(u"Unable to connect to SAB with URL: %s" % url)
-        return False
-
-    except httplib.InvalidURL as e:
-        logger.error(u"Invalid SAB host, check your config. Current host: %s" % headphones.CONFIG.SAB_HOST)
-        return False
-
-    except Exception as e:
-        logger.error(u"Error: " + str(e))
-        return False
-
-    if f is None:
+    if not response:
         logger.info(u"No data returned from SABnzbd, NZB not sent")
         return False
 
-    try:
-        result = f.readlines()
-    except Exception as e:
-        logger.info(u"Error trying to get result from SAB, NZB not sent: ")
-        return False
-
-    if len(result) == 0:
-        logger.info(u"No data returned from SABnzbd, NZB not sent")
-        return False
-
-    sabText = result[0].strip()
-
-    logger.info(u"Result text from SAB: " + sabText)
-
-    if sabText == "ok":
-        logger.info(u"NZB sent to SAB successfully")
+    if response['status']:
+        logger.info(u"NZB sent to SABnzbd successfully")
         return True
-    elif sabText == "Missing authentication":
-        logger.info(u"Incorrect username/password sent to SAB, NZB not sent")
-        return False
     else:
-        logger.info(u"Unknown failure sending NZB to sab. Return text is: " + sabText)
+        logger.error(u"Error sending NZB to SABnzbd: %s" % response['error'])
         return False
 
 
 def checkConfig():
 
     params = {'mode': 'get_config',
-               'section': 'misc'
+               'section': 'misc',
                }
 
-    if headphones.CONFIG.SAB_USERNAME:
-        params['ma_username'] = headphones.CONFIG.SAB_USERNAME
-    if headphones.CONFIG.SAB_PASSWORD:
-        params['ma_password'] = headphones.CONFIG.SAB_PASSWORD
-    if headphones.CONFIG.SAB_APIKEY:
-        params['apikey'] = headphones.CONFIG.SAB_APIKEY
-
-    if not headphones.CONFIG.SAB_HOST.startswith('http'):
-        headphones.CONFIG.SAB_HOST = 'http://' + headphones.CONFIG.SAB_HOST
-
-    if headphones.CONFIG.SAB_HOST.endswith('/'):
-        headphones.CONFIG.SAB_HOST = headphones.CONFIG.SAB_HOST[0:len(headphones.CONFIG.SAB_HOST) - 1]
-
-    url = headphones.CONFIG.SAB_HOST + "/" + "api?" + urllib.urlencode(params)
-
-    try:
-        f = urllib.urlopen(url).read()
-    except Exception:
+    config_options = sab_api_call(params=params)
+    
+    if not config_options:
         logger.warn("Unable to read SABnzbd config file - cannot determine renaming options (might affect auto & forced post processing)")
         return (0, 0)
 
-    config_options = ast.literal_eval(f)
-
-    replace_spaces = config_options['misc']['replace_spaces']
-    replace_dots = config_options['misc']['replace_dots']
+    replace_spaces = config_options['config']['misc']['replace_spaces']
+    replace_dots = config_options['config']['misc']['replace_dots']
 
     return (replace_spaces, replace_dots)

headphones/searcher.py

@@ -29,30 +29,27 @@ import string
 import shutil
 import random
 import urllib
+import datetime
 import headphones
 import subprocess
 import unicodedata
 
 from headphones.common import USER_AGENT
 from headphones import logger, db, helpers, classes, sab, nzbget, request
-from headphones import utorrent, transmission, notifiers
+from headphones import utorrent, transmission, notifiers, rutracker
 
 from bencode import bencode, bdecode
 
-import headphones.searcher_rutracker as rutrackersearch
-
 # Magnet to torrent services, for Black hole. Stolen from CouchPotato.
 TORRENT_TO_MAGNET_SERVICES = [
-    'https://zoink.it/torrent/%s.torrent',
-    'http://torrage.com/torrent/%s.torrent',
+    #'https://zoink.it/torrent/%s.torrent',
+    #'http://torrage.com/torrent/%s.torrent',
     'https://torcache.net/torrent/%s.torrent',
 ]
 
 # Persistent What.cd API object
 gazelle = None
-
-# RUtracker search object
-rutracker = rutrackersearch.Rutracker()
+ruobj = None
 
 
 def fix_url(s, charset="utf-8"):
@@ -167,6 +164,8 @@ def get_seed_ratio(provider):
         seed_ratio = headphones.CONFIG.WAFFLES_RATIO
     elif provider == 'Mininova':
         seed_ratio = headphones.CONFIG.MININOVA_RATIO
+    elif provider == 'Strike':
+    	seed_ratio = headphones.CONFIG.STRIKE_RATIO
     else:
         seed_ratio = None
 
@@ -189,10 +188,22 @@ def searchforalbum(albumid=None, new=False, losslessOnly=False,
         results = myDB.select('SELECT * from albums WHERE Status="Wanted" OR Status="Wanted Lossless"')
 
         for album in results:
+
             if not album['AlbumTitle'] or not album['ArtistName']:
                 logger.warn('Skipping release %s. No title available', album['AlbumID'])
                 continue
 
+            if headphones.CONFIG.WAIT_UNTIL_RELEASE_DATE and album['ReleaseDate']:
+                try:
+                    release_date = datetime.datetime.strptime(album['ReleaseDate'], "%Y-%m-%d")
+                except:
+                    logger.warn("No valid date for: %s. Skipping automatic search" % album['AlbumTitle'])
+                    continue
+
+                if release_date > datetime.datetime.today():
+                    logger.info("Skipping: %s. Waiting for release date of: %s" % (album['AlbumTitle'], album['ReleaseDate']))
+                    continue
+            
             new = True
 
             if album['Status'] == "Wanted Lossless":
@@ -219,13 +230,13 @@ def do_sorted_search(album, new, losslessOnly, choose_specific_download=False):
 
     NZB_PROVIDERS = (headphones.CONFIG.HEADPHONES_INDEXER or headphones.CONFIG.NEWZNAB or headphones.CONFIG.NZBSORG or headphones.CONFIG.OMGWTFNZBS)
     NZB_DOWNLOADERS = (headphones.CONFIG.SAB_HOST or headphones.CONFIG.BLACKHOLE_DIR or headphones.CONFIG.NZBGET_HOST)
-    TORRENT_PROVIDERS = (headphones.CONFIG.KAT or headphones.CONFIG.PIRATEBAY or headphones.CONFIG.OLDPIRATEBAY or headphones.CONFIG.MININOVA or headphones.CONFIG.WAFFLES or headphones.CONFIG.RUTRACKER or headphones.CONFIG.WHATCD)
+    TORRENT_PROVIDERS = (headphones.CONFIG.TORZNAB or headphones.CONFIG.KAT or headphones.CONFIG.PIRATEBAY or headphones.CONFIG.OLDPIRATEBAY or headphones.CONFIG.MININOVA or headphones.CONFIG.WAFFLES or headphones.CONFIG.RUTRACKER or headphones.CONFIG.WHATCD or headphones.CONFIG.STRIKE)
 
     results = []
     myDB = db.DBConnection()
     albumlength = myDB.select('SELECT sum(TrackDuration) from tracks WHERE AlbumID=?', [album['AlbumID']])[0][0]
 
-    if headphones.CONFIG.PREFER_TORRENTS == 0:
+    if headphones.CONFIG.PREFER_TORRENTS == 0 and not choose_specific_download:
 
         if NZB_PROVIDERS and NZB_DOWNLOADERS:
             results = searchNZB(album, new, losslessOnly, albumlength)
@@ -233,7 +244,7 @@ def do_sorted_search(album, new, losslessOnly, choose_specific_download=False):
         if not results and TORRENT_PROVIDERS:
             results = searchTorrent(album, new, losslessOnly, albumlength)
 
-    elif headphones.CONFIG.PREFER_TORRENTS == 1:
+    elif headphones.CONFIG.PREFER_TORRENTS == 1 and not choose_specific_download:
 
         if TORRENT_PROVIDERS:
             results = searchTorrent(album, new, losslessOnly, albumlength)
@@ -247,10 +258,10 @@ def do_sorted_search(album, new, losslessOnly, choose_specific_download=False):
         torrent_results = None
 
         if NZB_PROVIDERS and NZB_DOWNLOADERS:
-            nzb_results = searchNZB(album, new, losslessOnly, albumlength)
+            nzb_results = searchNZB(album, new, losslessOnly, albumlength, choose_specific_download)
 
         if TORRENT_PROVIDERS:
-            torrent_results = searchTorrent(album, new, losslessOnly, albumlength)
+            torrent_results = searchTorrent(album, new, losslessOnly, albumlength, choose_specific_download)
 
         if not nzb_results:
             nzb_results = []
@@ -306,9 +317,9 @@ def more_filtering(results, album, albumlength, new):
             targetsize = albumlength / 1000 * int(headphones.CONFIG.PREFERRED_BITRATE) * 128
             logger.info('Target size: %s' % helpers.bytes_to_mb(targetsize))
             if headphones.CONFIG.PREFERRED_BITRATE_LOW_BUFFER:
-                low_size_limit = targetsize - (targetsize * int(headphones.CONFIG.PREFERRED_BITRATE_LOW_BUFFER) / 100)
+                low_size_limit =  targetsize * int(headphones.CONFIG.PREFERRED_BITRATE_LOW_BUFFER) / 100
             if headphones.CONFIG.PREFERRED_BITRATE_HIGH_BUFFER:
-                high_size_limit = targetsize + (targetsize * int(headphones.CONFIG.PREFERRED_BITRATE_HIGH_BUFFER) / 100)
+                high_size_limit = targetsize * int(headphones.CONFIG.PREFERRED_BITRATE_HIGH_BUFFER) / 100
                 if headphones.CONFIG.PREFERRED_BITRATE_ALLOW_LOSSLESS:
                     allow_lossless = True
 
@@ -321,8 +332,10 @@ def more_filtering(results, album, albumlength, new):
         normalizedResultTitle = removeDisallowedFilenameChars(result[0])
         artistTitleCount = normalizedResultTitle.count(normalizedAlbumArtist)
 
-        if normalizedAlbumArtist in normalizedAlbumTitle and artistTitleCount < 2:
-            continue
+        # WHAT DOES THIS DO?
+        #if normalizedAlbumArtist in normalizedAlbumTitle and artistTitleCount < 2:
+        #    logger.info("Removing %s from %s" % (result[0], result[3]))
+        #    continue
 
         if low_size_limit and (int(result[1]) < low_size_limit):
             logger.info("%s from %s is too small for this album - not considering it. (Size: %s, Minsize: %s)", result[0], result[3], helpers.bytes_to_mb(result[1]), helpers.bytes_to_mb(low_size_limit))
@@ -400,7 +413,7 @@ def sort_search_results(resultlist, album, new, albumlength):
                 finallist = sorted(newlist, key=lambda title: (-title[5], title[6]))
 
                 if not len(finallist) and len(flac_list) and headphones.CONFIG.PREFERRED_BITRATE_ALLOW_LOSSLESS:
-                    logger.info("Since there were no appropriate lossy matches (and at least one lossless match, going to use lossless instead")
+                    logger.info("Since there were no appropriate lossy matches (and at least one lossless match), going to use lossless instead")
                     finallist = sorted(flac_list, key=lambda title: (title[5], int(title[1])), reverse=True)
         except Exception:
             logger.exception('Unhandled exception')
@@ -429,7 +442,7 @@ def get_year_from_release_date(release_date):
     return year
 
 
-def searchNZB(album, new=False, losslessOnly=False, albumlength=None):
+def searchNZB(album, new=False, losslessOnly=False, albumlength=None, choose_specific_download=False):
     reldate = album['ReleaseDate']
     year = get_year_from_release_date(reldate)
 
@@ -441,6 +454,8 @@ def searchNZB(album, new=False, losslessOnly=False, albumlength=None):
     # Use the provided search term if available, otherwise build a search term
     if album['SearchTerm']:
         term = album['SearchTerm']
+    elif album['Type'] == 'part of':
+        term = cleanalbum + " " + year
     else:
         # FLAC usually doesn't have a year for some reason so leave it out.
         # Various Artist albums might be listed as VA, so I'll leave that out too
@@ -481,7 +496,7 @@ def searchNZB(album, new=False, losslessOnly=False, albumlength=None):
             categories = "3030"
 
         # Request results
-        logger.info('Parsing results from Headphones Indexer')
+        logger.info('Searching Headphones Indexer with search term: %s' % term)
 
         headers = {'User-Agent': USER_AGENT}
         params = {
@@ -493,7 +508,7 @@ def searchNZB(album, new=False, losslessOnly=False, albumlength=None):
         }
 
         data = request.request_feed(
-            url="http://indexer.codeshy.com/api",
+            url="https://indexer.codeshy.com/api",
             params=params, headers=headers,
             auth=(headphones.CONFIG.HPUSER, headphones.CONFIG.HPPASS)
         )
@@ -552,7 +567,7 @@ def searchNZB(album, new=False, losslessOnly=False, albumlength=None):
                     categories = categories + ",4050"
 
             # Request results
-            logger.info('Parsing results from %s', newznab_host[0])
+            logger.info('Parsing results from %s using search term: %s' % (newznab_host[0],term))
 
             headers = {'User-Agent': USER_AGENT}
             params = {
@@ -600,9 +615,6 @@ def searchNZB(album, new=False, losslessOnly=False, albumlength=None):
             categories = "3030"
             logger.info("Album type is audiobook/spokenword. Using audiobook category")
 
-        # Request results
-        logger.info('Requesting from nzbs.org')
-
         headers = {'User-Agent': USER_AGENT}
         params = {
             "t": "search",
@@ -618,7 +630,7 @@ def searchNZB(album, new=False, losslessOnly=False, albumlength=None):
             timeout=5
         )
 
-        logger.info('Parsing results from nzbs.org')
+        logger.info('Parsing results from nzbs.org using search term: %s' % term)
         # Process feed
         if data:
             if not len(data.entries):
@@ -650,7 +662,7 @@ def searchNZB(album, new=False, losslessOnly=False, albumlength=None):
             logger.info("Album type is audiobook/spokenword. Searching all music categories")
 
         # Request results
-        logger.info('Parsing results from omgwtfnzbs')
+        logger.info('Parsing results from omgwtfnzbs using search term: %s' % term)
 
         headers = {'User-Agent': USER_AGENT}
         params = {
@@ -691,7 +703,7 @@ def searchNZB(album, new=False, losslessOnly=False, albumlength=None):
     results = [result for result in resultlist if verifyresult(result[0], artistterm, term, losslessOnly)]
 
     # Additional filtering for size etc
-    if results:
+    if results and not choose_specific_download:
         results = more_filtering(results, album, albumlength, new)
 
     return results
@@ -779,10 +791,11 @@ def send_to_downloader(data, bestqual, album):
                     # Randomize list of services
                     services = TORRENT_TO_MAGNET_SERVICES[:]
                     random.shuffle(services)
+                    headers = {'User-Agent': USER_AGENT}
 
                     for service in services:
-                        data = request.request_content(service % torrent_hash)
 
+                        data = request.request_content(service % torrent_hash, headers=headers)
                         if data and "torcache" in data:
                             if not torrent_to_file(download_path, data):
                                 return
@@ -804,15 +817,9 @@ def send_to_downloader(data, bestqual, album):
                         "to open or convert magnet links")
                     return
             else:
-                if bestqual[3] == "rutracker.org":
-                    download_path, _ = rutracker.get_torrent(bestqual[2],
-                        headphones.CONFIG.TORRENTBLACKHOLE_DIR)
 
-                    if not download_path:
-                        return
-                else:
-                    if not torrent_to_file(download_path, data):
-                        return
+                if not torrent_to_file(download_path, data):
+                    return
 
                 # Extract folder name from torrent
                 folder_name = read_torrent_name(download_path, bestqual[0])
@@ -822,13 +829,11 @@ def send_to_downloader(data, bestqual, album):
         elif headphones.CONFIG.TORRENT_DOWNLOADER == 1:
             logger.info("Sending torrent to Transmission")
 
-            # rutracker needs cookies to be set, pass the .torrent file instead of url
+            # Add torrent
             if bestqual[3] == 'rutracker.org':
-                file_or_url, torrentid = rutracker.get_torrent(bestqual[2])
+                torrentid = transmission.addTorrent('', data)
             else:
-                file_or_url = bestqual[2]
-
-            torrentid = transmission.addTorrent(file_or_url)
+                torrentid = transmission.addTorrent(bestqual[2])
 
             if not torrentid:
                 logger.error("Error sending torrent to Transmission. Are you sure it's running?")
@@ -841,13 +846,6 @@ def send_to_downloader(data, bestqual, album):
                 logger.error('Torrent folder name could not be determined')
                 return
 
-            # remove temp .torrent file created above
-            if bestqual[3] == 'rutracker.org':
-                try:
-                    shutil.rmtree(os.path.split(file_or_url)[0])
-                except Exception as e:
-                    logger.exception("Unhandled exception")
-
             # Set Seed Ratio
             seed_ratio = get_seed_ratio(bestqual[3])
             if seed_ratio is not None:
@@ -856,29 +854,29 @@ def send_to_downloader(data, bestqual, album):
         else:# if headphones.CONFIG.TORRENT_DOWNLOADER == 2:
             logger.info("Sending torrent to uTorrent")
 
-            # rutracker needs cookies to be set, pass the .torrent file instead of url
+            # Add torrent
             if bestqual[3] == 'rutracker.org':
-                file_or_url, torrentid = rutracker.get_torrent(bestqual[2])
-                folder_name, cacheid = utorrent.dirTorrent(torrentid)
-                folder_name = os.path.basename(os.path.normpath(folder_name))
-                utorrent.labelTorrent(torrentid)
+                ruobj.utorrent_add_file(data)
             else:
-                file_or_url = bestqual[2]
-                torrentid = calculate_torrent_hash(file_or_url, data)
-                folder_name = utorrent.addTorrent(file_or_url, torrentid)
+                utorrent.addTorrent(bestqual[2])
 
+            # Get hash
+            torrentid = calculate_torrent_hash(bestqual[2], data)
+            if not torrentid:
+                logger.error('Torrent id could not be determined')
+                return
+
+            # Get folder
+            folder_name = utorrent.getFolder(torrentid)
             if folder_name:
                 logger.info('Torrent folder name: %s' % folder_name)
             else:
                 logger.error('Torrent folder name could not be determined')
                 return
 
-            # remove temp .torrent file created above
-            if bestqual[3] == 'rutracker.org':
-                try:
-                    shutil.rmtree(os.path.split(file_or_url)[0])
-                except Exception as e:
-                    logger.exception("Unhandled exception")
+            # Set Label
+            if headphones.CONFIG.UTORRENT_LABEL:
+                utorrent.labelTorrent(torrentid)
 
             # Set Seed Ratio
             seed_ratio = get_seed_ratio(bestqual[3])
@@ -918,7 +916,7 @@ def send_to_downloader(data, bestqual, album):
     if headphones.CONFIG.PUSHBULLET_ENABLED and headphones.CONFIG.PUSHBULLET_ONSNATCH:
         logger.info(u"Sending PushBullet notification")
         pushbullet = notifiers.PUSHBULLET()
-        pushbullet.notify(name + " has been snatched!", "Download started")
+        pushbullet.notify(name, "Download started")
     if headphones.CONFIG.TWITTER_ENABLED and headphones.CONFIG.TWITTER_ONSNATCH:
         logger.info(u"Sending Twitter notification")
         twitter = notifiers.TwitterNotifier()
@@ -946,7 +944,11 @@ def send_to_downloader(data, bestqual, album):
         b2msg = 'From ' + provider + '<br></br>' + name
         boxcar = notifiers.BOXCAR()
         boxcar.notify('Headphones snatched: ' + title, b2msg, rgid)
-
+    if headphones.CONFIG.EMAIL_ENABLED and headphones.CONFIG.EMAIL_ONSNATCH:
+        logger.info(u"Sending Email notification")
+        email = notifiers.Email()
+        message = 'Snatched from ' + provider + '. ' + name
+        email.notify(title, message)
 
 def verifyresult(title, artistterm, term, lossless):
 
@@ -995,6 +997,13 @@ def verifyresult(title, artistterm, term, lossless):
                 logger.info("Removed '%s' from results because it doesn't contain required word: '%s'", title, each_word)
                 return False
 
+    if headphones.CONFIG.IGNORE_CLEAN_RELEASES:
+        for each_word in ['clean','edited','censored']:
+            logger.debug("Checking if '%s' is in search result: '%s'", each_word, title)
+            if each_word.lower() in title.lower() and each_word.lower() not in term.lower():
+                logger.info("Removed '%s' from results because it contains clean album word: '%s'", title, each_word)
+                return False
+
     tokens = re.split('\W', term, re.IGNORECASE | re.UNICODE)
     for token in tokens:
 
@@ -1014,14 +1023,9 @@ def verifyresult(title, artistterm, term, lossless):
     return True
 
 
-def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
+def searchTorrent(album, new=False, losslessOnly=False, albumlength=None, choose_specific_download=False):
     global gazelle  # persistent what.cd api object to reduce number of login attempts
-
-    # rutracker login
-    if headphones.CONFIG.RUTRACKER and album:
-        rulogin = rutracker.login(headphones.CONFIG.RUTRACKER_USER, headphones.CONFIG.RUTRACKER_PASSWORD)
-        if not rulogin:
-            logger.info(u'Could not login to rutracker, search results will exclude this provider')
+    global ruobj    # and rutracker
 
     albumid = album['AlbumID']
     reldate = album['ReleaseDate']
@@ -1040,7 +1044,8 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
     # of these torrent providers are just using cleanartist/cleanalbum terms
     if album['SearchTerm']:
         term = album['SearchTerm']
-
+    elif album['Type'] == 'part of':
+        term = cleanalbum + " " + year
     else:
         # FLAC usually doesn't have a year for some reason so I'll leave it out
         # Various Artist albums might be listed as VA, so I'll leave that out too
@@ -1084,6 +1089,68 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
 
         return proxy_url
 
+    if headphones.CONFIG.TORZNAB:
+        provider = "torznab"
+        torznab_hosts = []
+
+        if headphones.CONFIG.TORZNAB_HOST and headphones.CONFIG.TORZNAB_ENABLED:
+            torznab_hosts.append((headphones.CONFIG.TORZNAB_HOST, headphones.CONFIG.TORZNAB_APIKEY, headphones.CONFIG.TORZNAB_ENABLED))
+
+        for torznab_host in headphones.CONFIG.get_extra_torznabs():
+            if torznab_host[2] == '1' or torznab_host[2] == 1:
+                torznab_hosts.append(torznab_host)
+
+        if headphones.CONFIG.PREFERRED_QUALITY == 3 or losslessOnly:
+            categories = "3040"
+        elif headphones.CONFIG.PREFERRED_QUALITY == 1 or allow_lossless:
+            categories = "3040,3010"
+        else:
+            categories = "3010"
+
+        if album['Type'] == 'Other':
+            categories = "3030"
+            logger.info("Album type is audiobook/spokenword. Using audiobook category")
+
+        for torznab_host in torznab_hosts:
+
+            provider = torznab_host[0]
+
+            # Request results
+            logger.info('Parsing results from %s using search term: %s' % (torznab_host[0],term))
+
+            headers = {'User-Agent': USER_AGENT}
+            params = {
+                "t": "search",
+                "apikey": torznab_host[1],
+                "cat": categories,
+                "maxage": headphones.CONFIG.USENET_RETENTION,
+                "q": term
+            }
+
+            data = request.request_feed(
+                url=torznab_host[0] + '/api?',
+                params=params, headers=headers
+            )
+
+            # Process feed
+            if data:
+                if not len(data.entries):
+                    logger.info(u"No results found from %s for %s", torznab_host[0], term)
+                else:
+                    for item in data.entries:
+                        try:
+                            url = item.link
+                            title = item.title
+                            size = int(item.links[1]['length'])
+                            if all(word.lower() in title.lower() for word in term.split()):
+                                logger.info('Found %s. Size: %s' % (title, helpers.bytes_to_mb(size)))
+                                resultlist.append((title, size, url, provider, 'torrent', True))
+                            else:
+                                logger.info('Skipping %s, not all search term words found' % title)
+
+                        except Exception as e:
+                            logger.exception("An unknown error occurred trying to parse the feed: %s" % e)
+
     if headphones.CONFIG.KAT:
         provider = "Kick Ass Torrents"
         ka_term = term.replace("!", "")
@@ -1092,10 +1159,10 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
         if headphones.CONFIG.KAT_PROXY_URL:
             providerurl = fix_url(set_proxy(headphones.CONFIG.KAT_PROXY_URL))
         else:
-            providerurl = fix_url("https://kickass.so")
+            providerurl = fix_url("https://kickass.to")
 
         # Build URL
-        providerurl = providerurl + "/usearch/" + ka_term
+        providerurl = providerurl + "/json.php?"
 
         # Pick category for torrents
         if headphones.CONFIG.PREFERRED_QUALITY == 3 or losslessOnly:
@@ -1109,31 +1176,31 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
             maxsize = 300000000
 
         # Requesting content
-        logger.info("Searching KAT using term: %s", ka_term)
+        logger.info("Searching %s using term: %s" % (provider,ka_term))
 
         params = {
-            "categories[0]": "music",
+            "q": ka_term + "+category:music",
             "field": "seeders",
-            "sorder": "desc",
-            "rss": "1"
+            "sorder": "desc"
         }
-        data = request.request_feed(url=providerurl, params=params)
+        headers = {'User-Agent': USER_AGENT}
+        data = request.request_json(url=providerurl, params=params, headers=headers)
 
         # Process feed
         if data:
-            if not len(data.entries):
-                logger.info("No results found")
+            if not data['list']:
+                logger.info("No results found on %s using search term: %s" % (provider, ka_term))
             else:
-                for item in data.entries:
+                for item in data['list']:
                     try:
                         rightformat = True
                         title = item['title']
-                        seeders = item['torrent_seeds']
-                        url = item['links'][1]['href']
-                        size = int(item['links'][1]['length'])
+                        seeders = item['seeds']
+                        url = item['torrentLink']
+                        size = int(item['size'])
 
                         if format == "2":
-                            torrent = request.request_content(url)
+                            torrent = request.request_content(url, headers=headers)
                             if not torrent or (int(torrent.find(".mp3")) > 0 and int(torrent.find(".flac")) < 1):
                                 rightformat = False
 
@@ -1179,7 +1246,7 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
             query_items.append('bitrate:"%s"' % bitrate)
 
         # Requesting content
-        logger.info('Parsing results from Waffles')
+        logger.info('Parsing results from Waffles.fm')
 
         params = {
             "uid": headphones.CONFIG.WAFFLES_UID,
@@ -1214,45 +1281,38 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
                         logger.error(u"An error occurred while trying to parse the response from Waffles.fm: %s", e)
 
     # rutracker.org
-    if headphones.CONFIG.RUTRACKER and rulogin:
+    if headphones.CONFIG.RUTRACKER:
         provider = "rutracker.org"
 
         # Ignore if release date not specified, results too unpredictable
         if not year and not usersearchterm:
-            logger.info(u'Release date not specified, ignoring for rutracker.org')
+            logger.info(u"Release date not specified, ignoring for rutracker.org")
         else:
-
             if headphones.CONFIG.PREFERRED_QUALITY == 3 or losslessOnly:
                 format = 'lossless'
-                maxsize = 10000000000
             elif headphones.CONFIG.PREFERRED_QUALITY == 1 or allow_lossless:
                 format = 'lossless+mp3'
-                maxsize = 10000000000
             else:
                 format = 'mp3'
-                maxsize = 300000000
 
-            # build search url based on above
-            if not usersearchterm:
-                searchURL = rutracker.searchurl(artistterm, albumterm, year, format)
-            else:
-                searchURL = rutracker.searchurl(usersearchterm, ' ', ' ', format)
+            # Login
+            if not ruobj or not ruobj.logged_in():
+                ruobj = rutracker.Rutracker()
+                if not ruobj.login():
+                    ruobj = None
 
-            logger.info(u'Parsing results from <a href="%s">rutracker.org</a>' % searchURL)
+            if ruobj and ruobj.logged_in():
 
-            # parse results and get best match
-            rulist = rutracker.search(searchURL, maxsize, minimumseeders, albumid)
+                # build search url
+                if not usersearchterm:
+                    searchURL = ruobj.searchurl(artistterm, albumterm, year, format)
+                else:
+                    searchURL = ruobj.searchurl(usersearchterm, ' ', ' ', format)
 
-            # add best match to overall results list
-            if rulist:
-                for ru in rulist:
-                    title = ru[0].decode('utf-8')
-                    size = ru[1]
-                    url = ru[2]
-                    resultlist.append((title, size, url, provider, 'torrent', True))
-                    logger.info('Found %s. Size: %s' % (title, helpers.bytes_to_mb(size)))
-            else:
-                logger.info(u"No valid results found from %s" % (provider))
+                # parse results
+                rulist = ruobj.search(searchURL)
+                if rulist:
+                    resultlist.extend(rulist)
 
     if headphones.CONFIG.WHATCD:
         provider = "What.cd"
@@ -1268,6 +1328,12 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
             search_formats = [None]  # should return all
             bitrate = headphones.CONFIG.PREFERRED_BITRATE
             if bitrate:
+                if 225 <= int(bitrate) < 256:
+                    bitrate = 'V0'
+                elif 200 <= int(bitrate) < 225:
+                    bitrate = 'V1'
+                elif 175 <= int(bitrate) < 200:
+                    bitrate = 'V2'
                 for encoding_string in gazelleencoding.ALL_ENCODINGS:
                     if re.search(bitrate, encoding_string, flags=re.I):
                         bitrate_string = encoding_string
@@ -1294,7 +1360,10 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
             logger.info(u"Searching %s..." % provider)
             all_torrents = []
             for search_format in search_formats:
-                all_torrents.extend(gazelle.search_torrents(artistname=semi_clean_artist_term,
+                if usersearchterm:
+                    all_torrents.extend(gazelle.search_torrents(searchstr=usersearchterm, format=search_format, encoding=bitrate_string)['results'])
+                else:
+                    all_torrents.extend(gazelle.search_torrents(artistname=semi_clean_artist_term,
                                                               groupname=semi_clean_album_term,
                                                               format=search_format, encoding=bitrate_string)['results'])
 
@@ -1367,7 +1436,7 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
             rows = data.select('table tbody tr')
 
             if not rows:
-                logger.info("No results found")
+                logger.info("No results found from The Pirate Bay using term: %s" % tpb_term)
             else:
                 for item in rows:
                     try:
@@ -1457,6 +1526,57 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
                     except Exception as e:
                         logger.error(u"An unknown error occurred in the Old Pirate Bay parser: %s" % e)
 
+    # Strike
+    if headphones.CONFIG.STRIKE:
+        provider = "Strike"
+        s_term = term.replace("!", "")
+        providerurl = fix_url("https://getstrike.net/api/v2/torrents/search/?phrase=")
+
+        providerurl = providerurl + s_term + "&category=Music"
+
+        if headphones.CONFIG.PREFERRED_QUALITY == 3 or losslessOnly:
+            format = "2"
+            providerurl = providerurl + "&subcategory=Lossless"
+            maxsize = 10000000000
+        elif headphones.CONFIG.PREFERRED_QUALITY == 1 or allow_lossless:
+            format = "10"  # MP3 and FLAC
+            maxsize = 10000000000
+        else:
+            format = "8"  # MP3 only
+            maxsize = 300000000
+
+        logger.info("Searching %s using term: %s" % (provider, s_term))
+        data = request.request_json(url=providerurl)
+
+        if not data or not data.get('torrents'):
+            logger.info("No results found on %s using search term: %s" % (provider, s_term))
+        else:
+            for item in data['torrents']:
+                try:
+                    rightformat = True
+                    title = item['torrent_title']
+                    seeders = item['seeds']
+                    url = item['magnet_uri']
+                    size = int(item['size'])
+                    subcategory = item['sub_category']
+
+                    if format == 2:
+                        if subcategory != "Lossless":
+                            rightformat = False
+
+                    if rightformat and size < maxsize and minimumseeders < int(seeders):
+                        match = True
+                        logger.info('Found %s. Size: %s' % (title, helpers.bytes_to_mb(size)))
+                    else:
+                        match = False
+                        logger.info(
+                            '%s is larger than the maxsize, the wrong format or has too little seeders for this category, skipping. (Size: %i bytes, Seeders: %d, Format: %s)',
+                            title, size, int(seeders), rightformat)
+
+                    resultlist.append((title, size, url, provider, 'torrent', match))
+                except Exception as e:
+                    logger.exception("Unhandled exception in the Strike parser")
+
     # Mininova
     if headphones.CONFIG.MININOVA:
         provider = "Mininova"
@@ -1521,7 +1641,7 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
     results = [result for result in resultlist if verifyresult(result[0], artistterm, term, losslessOnly)]
 
     # Additional filtering for size etc
-    if results:
+    if results and not choose_specific_download:
         results = more_filtering(results, album, albumlength, new)
 
     return results
@@ -1533,12 +1653,14 @@ def preprocess(resultlist):
 
     for result in resultlist:
         if result[4] == 'torrent':
+
+            # rutracker always needs the torrent data
+            if result[3] == 'rutracker.org':
+                return ruobj.get_torrent_data(result[2]), result
+
             #Get out of here if we're using Transmission
             if headphones.CONFIG.TORRENT_DOWNLOADER == 1:  ## if not a magnet link still need the .torrent to generate hash... uTorrent support labeling
                 return True, result
-            # get outta here if rutracker
-            if result[3] == 'rutracker.org':
-                return True, result
             # Get out of here if it's a magnet link
             if result[2].lower().startswith("magnet:"):
                 return True, result
@@ -1547,7 +1669,8 @@ def preprocess(resultlist):
             headers = {}
 
             if result[3] == 'Kick Ass Torrents':
-                headers['Referer'] = 'http://kat.ph/'
+                #headers['Referer'] = 'http://kat.ph/'
+                headers['User-Agent'] = USER_AGENT
             elif result[3] == 'What.cd':
                 headers['User-Agent'] = 'Headphones'
             elif result[3] == "The Pirate Bay" or result[3] == "Old Pirate Bay":

headphones/searcher_rutracker.py

@@ -1,349 +0,0 @@
-#!/usr/bin/env python
-# coding=utf-8
-
-# Headphones rutracker.org search
-# Functions called from searcher.py
-
-from bencode import bencode as bencode, bdecode
-from urlparse import urlparse
-from bs4 import BeautifulSoup
-from tempfile import mkdtemp
-from hashlib import sha1
-
-import headphones
-import requests
-import cookielib
-import urllib2
-import urllib
-import re
-import os
-
-from headphones import db, logger
-
-
-class Rutracker():
-
-    logged_in = False
-
-    # Stores a number of login attempts to prevent recursion.
-    #login_counter = 0
-
-    def __init__(self):
-
-        self.cookiejar = cookielib.CookieJar()
-        self.opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(self.cookiejar))
-        urllib2.install_opener(self.opener)
-
-    def login(self, login, password):
-        """Implements tracker login procedure."""
-
-        self.logged_in = False
-
-        if login is None or password is None:
-            return False
-
-        #self.login_counter += 1
-
-        # No recursion wanted.
-        #if self.login_counter > 1:
-        #    return False
-
-        params = urllib.urlencode({"login_username": login,
-                                   "login_password": password,
-                                   "login": "Вход"})
-
-        try:
-            self.opener.open("http://login.rutracker.org/forum/login.php", params)
-        except Exception:
-            pass
-
-        # Check if we're logged in
-        for cookie in self.cookiejar:
-            if cookie.name == 'bb_data':
-                self.logged_in = True
-
-        return self.logged_in
-
-    def searchurl(self, artist, album, year, format):
-        """
-        Return the search url
-        """
-
-        # Build search url
-        searchterm = ''
-        if artist != 'Various Artists':
-            searchterm = artist
-            searchterm = searchterm + ' '
-        searchterm = searchterm + album
-        searchterm = searchterm + ' '
-        searchterm = searchterm + year
-
-        providerurl = "http://rutracker.org/forum/tracker.php"
-
-        if format == 'lossless':
-            format = '+lossless'
-        elif format == 'lossless+mp3':
-            format = '+lossless||mp3||aac'
-        else:
-            format = '+mp3||aac'
-
-        # sort by size, descending.
-        sort = '&o=7&s=2'
-
-        searchurl = "%s?nm=%s%s%s" % (providerurl, urllib.quote(searchterm), format, sort)
-
-        return searchurl
-
-    def search(self, searchurl, maxsize, minseeders, albumid):
-        """
-        Parse the search results and return valid torrent list
-        """
-
-        titles = []
-        urls = []
-        seeders = []
-        sizes = []
-        torrentlist = []
-        rulist = []
-
-        try:
-
-            page = self.opener.open(searchurl, timeout=60)
-            soup = BeautifulSoup(page.read())
-
-            # Debug
-            #logger.debug (soup.prettify())
-
-            # Title
-            for link in soup.find_all('a', attrs={'class': 'med tLink hl-tags bold'}):
-                title = link.get_text()
-                titles.append(title)
-
-            # Download URL
-            for link in soup.find_all('a', attrs={'class': 'small tr-dl dl-stub'}):
-                url = link.get('href')
-                urls.append(url)
-
-            # Seeders
-            for link in soup.find_all('b', attrs={'class': 'seedmed'}):
-                seeder = link.get_text()
-                seeders.append(seeder)
-
-            # Size
-            for link in soup.find_all('td', attrs={'class': 'row4 small nowrap tor-size'}):
-                size = link.u.string
-                sizes.append(size)
-
-        except:
-            pass
-
-        # Combine lists
-        torrentlist = zip(titles, urls, seeders, sizes)
-
-        # return if nothing found
-        if not torrentlist:
-            return False
-
-        # don't bother checking track counts anymore, let searcher filter instead
-        # leave code in just in case
-        check_track_count = False
-
-        if check_track_count:
-
-            # get headphones track count for album, return if not found
-            myDB = db.DBConnection()
-            tracks = myDB.select('SELECT * from tracks WHERE AlbumID=?', [albumid])
-            hptrackcount = len(tracks)
-
-            if not hptrackcount:
-                logger.info('headphones track info not found, cannot compare to torrent')
-                return False
-
-            # Return all valid entries, ignored, required words now checked in searcher.py
-
-            #unwantedlist = ['promo', 'vinyl', '[lp]', 'songbook', 'tvrip', 'hdtv', 'dvd']
-
-            formatlist = ['ape', 'flac', 'ogg', 'm4a', 'aac', 'mp3', 'wav', 'aif']
-            deluxelist = ['deluxe', 'edition', 'japanese', 'exclusive']
-
-        for torrent in torrentlist:
-
-            returntitle = torrent[0].encode('utf-8')
-            url = torrent[1]
-            seeders = torrent[2]
-            size = torrent[3]
-
-            if int(size) <= maxsize and int(seeders) >= minseeders:
-
-                #Torrent topic page
-                torrent_id = dict([part.split('=') for part in urlparse(url)[4].split('&')])['t']
-                topicurl = 'http://rutracker.org/forum/viewtopic.php?t=' + torrent_id
-
-                # add to list
-                if not check_track_count:
-                    valid = True
-                else:
-
-                    # Check torrent info
-                    self.cookiejar.set_cookie(cookielib.Cookie(version=0, name='bb_dl', value=torrent_id, port=None, port_specified=False, domain='.rutracker.org', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False))
-
-                    # Debug
-                    #for cookie in self.cookiejar:
-                    #    logger.debug ('Cookie: %s' % cookie)
-
-                    try:
-                        page = self.opener.open(url)
-                        torrent = page.read()
-                        if torrent:
-                            decoded = bdecode(torrent)
-                            metainfo = decoded['info']
-                        page.close()
-                    except Exception as e:
-                        logger.error('Error getting torrent: %s' % e)
-                        return False
-
-                    # get torrent track count and check for cue
-                    trackcount = 0
-                    cuecount = 0
-
-                    if 'files' in metainfo: # multi
-                        for pathfile in metainfo['files']:
-                            path = pathfile['path']
-                            for file in path:
-                                if any(file.lower().endswith('.' + x.lower()) for x in formatlist):
-                                    trackcount += 1
-                                if '.cue' in file:
-                                    cuecount += 1
-
-                    title = returntitle.lower()
-                    logger.debug('torrent title: %s' % title)
-                    logger.debug('headphones trackcount: %s' % hptrackcount)
-                    logger.debug('rutracker trackcount: %s' % trackcount)
-
-                    # If torrent track count less than headphones track count, and there's a cue, then attempt to get track count from log(s)
-                    # This is for the case where we have a single .flac/.wav which can be split by cue
-                    # Not great, but shouldn't be doing this too often
-                    totallogcount = 0
-                    if trackcount < hptrackcount and cuecount > 0 and cuecount < hptrackcount:
-                        page = self.opener.open(topicurl, timeout=60)
-                        soup = BeautifulSoup(page.read())
-                        findtoc = soup.find_all(text='TOC of the extracted CD')
-                        if not findtoc:
-                            findtoc = soup.find_all(text='TOC извлечённого CD')
-                        for toc in findtoc:
-                            logcount = 0
-                            for toccontent in toc.find_all_next(text=True):
-                                cut_string = toccontent.split('|')
-                                new_string = cut_string[0].lstrip().rstrip()
-                                if new_string == '1' or new_string == '01':
-                                    logcount = 1
-                                elif logcount > 0:
-                                    if new_string.isdigit():
-                                        logcount += 1
-                                    else:
-                                        break
-                            totallogcount = totallogcount + logcount
-
-                    if totallogcount > 0:
-                        trackcount = totallogcount
-                        logger.debug('rutracker logtrackcount: %s' % totallogcount)
-
-                    # If torrent track count = hp track count then return torrent,
-                    # if greater, check for deluxe/special/foreign editions
-                    # if less, then allow if it's a single track with a cue
-                    valid = False
-
-                    if trackcount == hptrackcount:
-                        valid = True
-                    elif trackcount > hptrackcount:
-                        if any(deluxe in title for deluxe in deluxelist):
-                            valid = True
-
-                # Add to list
-                if valid:
-                    rulist.append((returntitle, size, topicurl))
-                else:
-                    if topicurl:
-                        logger.info(u'<a href="%s">Torrent</a> found with %s tracks but the selected headphones release has %s tracks, skipping for rutracker.org' % (topicurl, trackcount, hptrackcount))
-            else:
-                logger.info('%s is larger than the maxsize or has too little seeders for this category, skipping. (Size: %i bytes, Seeders: %i)' % (returntitle, int(size), int(seeders)))
-
-        return rulist
-
-    def get_torrent(self, url, savelocation=None):
-
-        torrent_id = dict([part.split('=') for part in urlparse(url)[4].split('&')])['t']
-        self.cookiejar.set_cookie(cookielib.Cookie(version=0, name='bb_dl', value=torrent_id, port=None, port_specified=False, domain='.rutracker.org', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False))
-        downloadurl = 'http://dl.rutracker.org/forum/dl.php?t=' + torrent_id
-        torrent_name = torrent_id + '.torrent'
-
-        try:
-            prev = os.umask(headphones.UMASK)
-            page = self.opener.open(downloadurl)
-            torrent = page.read()
-            decoded = bdecode(torrent)
-            metainfo = decoded['info']
-            tor_hash = sha1(bencode(metainfo)).hexdigest()
-            if savelocation:
-                download_path = os.path.join(savelocation, torrent_name)
-            else:
-                tempdir = mkdtemp(suffix='_rutracker_torrents')
-                download_path = os.path.join(tempdir, torrent_name)
-
-            with open(download_path, 'wb') as f:
-                f.write(torrent)
-            os.umask(prev)
-
-            # Add file to utorrent
-            if headphones.CONFIG.TORRENT_DOWNLOADER == 2:
-                self.utorrent_add_file(download_path)
-
-        except Exception as e:
-            logger.error('Error getting torrent: %s', e)
-            return False
-
-        return download_path, tor_hash
-
-    #TODO get this working in utorrent.py
-    def utorrent_add_file(self, filename):
-
-        host = headphones.CONFIG.UTORRENT_HOST
-        if not host.startswith('http'):
-            host = 'http://' + host
-        if host.endswith('/'):
-            host = host[:-1]
-        if host.endswith('/gui'):
-            host = host[:-4]
-
-        base_url = host
-        username = headphones.CONFIG.UTORRENT_USERNAME
-        password = headphones.CONFIG.UTORRENT_PASSWORD
-
-        session = requests.Session()
-        url = base_url + '/gui/'
-        session.auth = (username, password)
-
-        try:
-            r = session.get(url + 'token.html')
-        except Exception:
-            logger.exception('Error getting token')
-            return
-
-        if r.status_code == '401':
-            logger.debug('Error reaching utorrent')
-            return
-
-        regex = re.search(r'.+>([^<]+)</div></html>', r.text)
-        if regex is None:
-            logger.debug('Error reading token')
-            return
-
-        session.params = {'token': regex.group(1)}
-
-        with open(filename, 'rb') as f:
-            try:
-                session.post(url, params={'action': 'add-file'},
-                    files={'torrent_file': f})
-            except Exception:
-                logger.exception('Error adding file to utorrent')
-                return

headphones/transmission.py

@@ -28,12 +28,15 @@ import headphones
 #       Store torrent id so we can check up on it
 
 
-def addTorrent(link):
+def addTorrent(link, data=None):
     method = 'torrent-add'
 
-    if link.endswith('.torrent'):
-        with open(link, 'rb') as f:
-            metainfo = str(base64.b64encode(f.read()))
+    if link.endswith('.torrent') or data:
+        if data:
+            metainfo = str(base64.b64encode(data))
+        else:
+            with open(link, 'rb') as f:
+                metainfo = str(base64.b64encode(f.read()))
         arguments = {'metainfo': metainfo, 'download-dir': headphones.CONFIG.DOWNLOAD_TORRENT_DIR}
     else:
         arguments = {'filename': link, 'download-dir': headphones.CONFIG.DOWNLOAD_TORRENT_DIR}

headphones/utorrent.py

@@ -73,6 +73,7 @@ class utorrentclient(object):
         except urllib2.HTTPError as err:
             logger.debug('URL: ' + str(url))
             logger.debug('Error getting Token. uTorrent responded with error: ' + str(err))
+            return
         match = re.search(utorrentclient.TOKEN_REGEX, response.read())
         return match.group(1)
 
@@ -147,6 +148,10 @@ class utorrentclient(object):
         return self._action(params)
 
     def _action(self, params, body=None, content_type=None):
+
+        if not self.token:
+            return
+
         url = self.base_url + '/gui/' + '?token=' + self.token + '&' + urllib.urlencode(params)
         request = urllib2.Request(url)
 
@@ -215,7 +220,7 @@ def dirTorrent(hash, cacheid=None, return_name=None):
     cacheid = torrentList['torrentc']
 
     for torrent in torrents:
-        if torrent[0].upper() == hash:
+        if torrent[0].upper() == hash.upper():
             if not return_name:
                 return torrent[26], cacheid
             else:
@@ -223,8 +228,12 @@ def dirTorrent(hash, cacheid=None, return_name=None):
 
     return None, None
 
+def addTorrent(link):
+    uTorrentClient = utorrentclient()
+    uTorrentClient.add_url(link)
 
-def addTorrent(link, hash):
+
+def getFolder(hash):
     uTorrentClient = utorrentclient()
 
     # Get Active Directory from settings
@@ -234,8 +243,6 @@ def addTorrent(link, hash):
         logger.error('Could not get "Put new downloads in:" directory from uTorrent settings, please ensure it is set')
         return None
 
-    uTorrentClient.add_url(link)
-
     # Get Torrent Folder Name
     torrent_folder, cacheid = dirTorrent(hash)
 
@@ -249,10 +256,8 @@ def addTorrent(link, hash):
 
     if torrent_folder == active_dir or not torrent_folder:
         torrent_folder, cacheid = dirTorrent(hash, cacheid, return_name=True)
-        labelTorrent(hash)
         return torrent_folder
     else:
-        labelTorrent(hash)
         if headphones.SYS_PLATFORM != "win32":
             torrent_folder = torrent_folder.replace('\\', '/')
         return os.path.basename(os.path.normpath(torrent_folder))

headphones/webserve.py

@@ -32,8 +32,10 @@ import random
 import urllib
 import json
 import time
+import cgi
 import sys
 import os
+import re
 
 try:
     # pylint:disable=E0611
@@ -145,9 +147,11 @@ class WebInterface(object):
             raise cherrypy.HTTPRedirect("home")
         if type == 'artist':
             searchresults = mb.findArtist(name, limit=100)
-        else:
+        elif type == 'album':
             searchresults = mb.findRelease(name, limit=100)
-        return serve_template(templatename="searchresults.html", title='Search Results for: "' + name + '"', searchresults=searchresults, name=name, type=type)
+        else:
+            searchresults = mb.findSeries(name, limit=100)
+        return serve_template(templatename="searchresults.html", title='Search Results for: "' + cgi.escape(name) + '"', searchresults=searchresults, name=cgi.escape(name), type=type)
 
     @cherrypy.expose
     def addArtist(self, artistid):
@@ -156,6 +160,13 @@ class WebInterface(object):
         thread.join(1)
         raise cherrypy.HTTPRedirect("artistPage?ArtistID=%s" % artistid)
 
+    @cherrypy.expose
+    def addSeries(self, seriesid):
+        thread = threading.Thread(target=importer.addArtisttoDB, args=[seriesid, False, False, "series"])
+        thread.start()
+        thread.join(1)
+        raise cherrypy.HTTPRedirect("artistPage?ArtistID=%s" % seriesid)
+
     @cherrypy.expose
     def getExtras(self, ArtistID, newstyle=False, **kwargs):
         # if calling this function without the newstyle, they're using the old format
@@ -221,11 +232,11 @@ class WebInterface(object):
         raise cherrypy.HTTPRedirect("artistPage?ArtistID=%s" % ArtistID)
 
     def removeArtist(self, ArtistID):
-        logger.info(u"Deleting all traces of artist: " + ArtistID)
         myDB = db.DBConnection()
         namecheck = myDB.select('SELECT ArtistName from artists where ArtistID=?', [ArtistID])
         for name in namecheck:
             artistname = name['ArtistName']
+        logger.info(u"Deleting all traces of artist: " + artistname)
         myDB.action('DELETE from artists WHERE ArtistID=?', [ArtistID])
 
         from headphones import cache
@@ -256,14 +267,72 @@ class WebInterface(object):
 
     @cherrypy.expose
     def scanArtist(self, ArtistID):
-        logger.info(u"Scanning artist: %s", ArtistID)
+
         myDB = db.DBConnection()
-        artistname = myDB.select('SELECT DISTINCT ArtistName FROM artists WHERE ArtistID=?', [ArtistID])
-        artistfolder = os.path.join(headphones.CONFIG.DESTINATION_DIR, artistname[0][0])
-        try:
-            threading.Thread(target=librarysync.libraryScan(dir=artistfolder)).start()
-        except Exception as e:
-            logger.error('Unable to complete the scan: %s', e)
+        artist_name = myDB.select('SELECT DISTINCT ArtistName FROM artists WHERE ArtistID=?', [ArtistID])[0][0]
+
+        logger.info(u"Scanning artist: %s", artist_name)
+
+        full_folder_format = headphones.CONFIG.FOLDER_FORMAT
+        folder_format = re.findall(r'(.*?[Aa]rtist?)\.*', full_folder_format)[0]
+
+        acceptable_formats = ["$artist","$sortartist","$first/$artist","$first/$sortartist"]
+
+        if not folder_format.lower() in acceptable_formats:
+            logger.info("Can't determine the artist folder from the configured folder_format. Not scanning")
+            return
+
+        # Format the folder to match the settings
+        artist = artist_name.replace('/', '_')
+
+        if headphones.CONFIG.FILE_UNDERSCORES:
+            artist = artist.replace(' ', '_')
+
+        if artist.startswith('The '):
+            sortname = artist[4:] + ", The"
+        else:
+            sortname = artist
+
+        if sortname[0].isdigit():
+            firstchar = u'0-9'
+        else:
+            firstchar = sortname[0]
+
+        values = {'$Artist': artist,
+                '$SortArtist': sortname,
+                '$First': firstchar.upper(),
+                '$artist': artist.lower(),
+                '$sortartist': sortname.lower(),
+                '$first': firstchar.lower(),
+            }
+
+        folder = helpers.replace_all(folder_format.strip(), values, normalize=True)
+
+        folder = helpers.replace_illegal_chars(folder, type="folder")
+        folder = folder.replace('./', '_/').replace('/.', '/_')
+
+        if folder.endswith('.'):
+            folder = folder[:-1] + '_'
+
+        if folder.startswith('.'):
+            folder = '_' + folder[1:]
+
+        dirs = []
+        if headphones.CONFIG.MUSIC_DIR:
+            dirs.append(headphones.CONFIG.MUSIC_DIR)
+        if headphones.CONFIG.DESTINATION_DIR:
+            dirs.append(headphones.CONFIG.DESTINATION_DIR)
+        if headphones.CONFIG.LOSSLESS_DESTINATION_DIR:
+            dirs.append(headphones.CONFIG.LOSSLESS_DESTINATION_DIR)
+
+        dirs = set(dirs)
+
+        for dir in dirs:
+            artistfolder = os.path.join(dir, folder)
+            if not os.path.isdir(artistfolder):
+                logger.debug("Cannot find directory: " + artistfolder)
+                continue
+            threading.Thread(target=librarysync.libraryScan, kwargs={"dir":artistfolder, "artistScan":True, "ArtistID":ArtistID, "ArtistName":artist_name}).start()
         raise cherrypy.HTTPRedirect("artistPage?ArtistID=%s" % ArtistID)
 
     @cherrypy.expose
@@ -374,6 +443,9 @@ class WebInterface(object):
           myDB = db.DBConnection()
           album = myDB.action('SELECT * from albums WHERE AlbumID=?', [AlbumID]).fetchone()
           searcher.send_to_downloader(data, bestqual, album)
+          return json.dumps({'result':'success'})
+        else:
+          return json.dumps({'result':'failure'})
 
     @cherrypy.expose
     def unqueueAlbum(self, AlbumID, ArtistID):
@@ -728,9 +800,9 @@ class WebInterface(object):
         raise cherrypy.HTTPRedirect("home")
 
     @cherrypy.expose
-    def forcePostProcess(self, dir=None, album_dir=None):
+    def forcePostProcess(self, dir=None, album_dir=None, keep_original_folder=False):
         from headphones import postprocessor
-        threading.Thread(target=postprocessor.forcePostProcess, kwargs={'dir': dir, 'album_dir': album_dir}).start()
+        threading.Thread(target=postprocessor.forcePostProcess, kwargs={'dir': dir, 'album_dir': album_dir, 'keep_original_folder':keep_original_folder == 'True'}).start()
         raise cherrypy.HTTPRedirect("home")
 
     @cherrypy.expose
@@ -877,6 +949,7 @@ class WebInterface(object):
         cherrypy.response.headers['Content-type'] = 'application/json'
         return json_albums
 
+    @cherrypy.expose
     def getArtistjson(self, ArtistID, **kwargs):
         myDB = db.DBConnection()
         artist = myDB.action('SELECT * FROM artists WHERE ArtistID=?', [ArtistID]).fetchone()
@@ -992,6 +1065,11 @@ class WebInterface(object):
             "newznab_apikey": headphones.CONFIG.NEWZNAB_APIKEY,
             "newznab_enabled": checked(headphones.CONFIG.NEWZNAB_ENABLED),
             "extra_newznabs": headphones.CONFIG.get_extra_newznabs(),
+            "use_torznab": checked(headphones.CONFIG.TORZNAB),
+            "torznab_host": headphones.CONFIG.TORZNAB_HOST,
+            "torznab_apikey": headphones.CONFIG.TORZNAB_APIKEY,
+            "torznab_enabled": checked(headphones.CONFIG.TORZNAB_ENABLED),
+            "extra_torznabs": headphones.CONFIG.get_extra_torznabs(),
             "use_nzbsorg": checked(headphones.CONFIG.NZBSORG),
             "nzbsorg_uid": headphones.CONFIG.NZBSORG_UID,
             "nzbsorg_hash": headphones.CONFIG.NZBSORG_HASH,
@@ -1001,6 +1079,7 @@ class WebInterface(object):
             "preferred_words": headphones.CONFIG.PREFERRED_WORDS,
             "ignored_words": headphones.CONFIG.IGNORED_WORDS,
             "required_words": headphones.CONFIG.REQUIRED_WORDS,
+            "ignore_clean_releases": checked(headphones.CONFIG.IGNORE_CLEAN_RELEASES),
             "torrentblackhole_dir": headphones.CONFIG.TORRENTBLACKHOLE_DIR,
             "download_torrent_dir": headphones.CONFIG.DOWNLOAD_TORRENT_DIR,
             "numberofseeders": headphones.CONFIG.NUMBEROFSEEDERS,
@@ -1027,6 +1106,8 @@ class WebInterface(object):
             "whatcd_username": headphones.CONFIG.WHATCD_USERNAME,
             "whatcd_password": headphones.CONFIG.WHATCD_PASSWORD,
             "whatcd_ratio": headphones.CONFIG.WHATCD_RATIO,
+            "use_strike": checked(headphones.CONFIG.STRIKE),
+            "strike_ratio": headphones.CONFIG.STRIKE_RATIO,
             "pref_qual_0": radio(headphones.CONFIG.PREFERRED_QUALITY, 0),
             "pref_qual_1": radio(headphones.CONFIG.PREFERRED_QUALITY, 1),
             "pref_qual_2": radio(headphones.CONFIG.PREFERRED_QUALITY, 2),
@@ -1052,15 +1133,19 @@ class WebInterface(object):
             "embed_album_art": checked(headphones.CONFIG.EMBED_ALBUM_ART),
             "embed_lyrics": checked(headphones.CONFIG.EMBED_LYRICS),
             "replace_existing_folders": checked(headphones.CONFIG.REPLACE_EXISTING_FOLDERS),
+            "keep_original_folder" : checked(headphones.CONFIG.KEEP_ORIGINAL_FOLDER),
             "destination_dir": headphones.CONFIG.DESTINATION_DIR,
             "lossless_destination_dir": headphones.CONFIG.LOSSLESS_DESTINATION_DIR,
             "folder_format": headphones.CONFIG.FOLDER_FORMAT,
             "file_format": headphones.CONFIG.FILE_FORMAT,
             "file_underscores": checked(headphones.CONFIG.FILE_UNDERSCORES),
             "include_extras": checked(headphones.CONFIG.INCLUDE_EXTRAS),
+            "official_releases_only": checked(headphones.CONFIG.OFFICIAL_RELEASES_ONLY),
+            "wait_until_release_date": checked(headphones.CONFIG.WAIT_UNTIL_RELEASE_DATE),
             "autowant_upcoming": checked(headphones.CONFIG.AUTOWANT_UPCOMING),
             "autowant_all": checked(headphones.CONFIG.AUTOWANT_ALL),
             "autowant_manually_added": checked(headphones.CONFIG.AUTOWANT_MANUALLY_ADDED),
+            "do_not_process_unmatched": checked(headphones.CONFIG.DO_NOT_PROCESS_UNMATCHED),
             "keep_torrent_files": checked(headphones.CONFIG.KEEP_TORRENT_FILES),
             "prefer_torrents_0": radio(headphones.CONFIG.PREFER_TORRENTS, 0),
             "prefer_torrents_1": radio(headphones.CONFIG.PREFER_TORRENTS, 1),
@@ -1106,6 +1191,7 @@ class WebInterface(object):
             "plex_client_host": headphones.CONFIG.PLEX_CLIENT_HOST,
             "plex_username": headphones.CONFIG.PLEX_USERNAME,
             "plex_password": headphones.CONFIG.PLEX_PASSWORD,
+            "plex_token": headphones.CONFIG.PLEX_TOKEN,
             "plex_update": checked(headphones.CONFIG.PLEX_UPDATE),
             "plex_notify": checked(headphones.CONFIG.PLEX_NOTIFY),
             "nma_enabled": checked(headphones.CONFIG.NMA_ENABLED),
@@ -1142,6 +1228,9 @@ class WebInterface(object):
             "customhost": headphones.CONFIG.CUSTOMHOST,
             "customport": headphones.CONFIG.CUSTOMPORT,
             "customsleep": headphones.CONFIG.CUSTOMSLEEP,
+            "customauth": checked(headphones.CONFIG.CUSTOMAUTH),
+            "customuser": headphones.CONFIG.CUSTOMUSER,
+            "custompass": headphones.CONFIG.CUSTOMPASS,
             "hpuser": headphones.CONFIG.HPUSER,
             "hppass": headphones.CONFIG.HPPASS,
             "songkick_enabled": checked(headphones.CONFIG.SONGKICK_ENABLED),
@@ -1151,7 +1240,18 @@ class WebInterface(object):
             "cache_sizemb": headphones.CONFIG.CACHE_SIZEMB,
             "file_permissions": headphones.CONFIG.FILE_PERMISSIONS,
             "folder_permissions": headphones.CONFIG.FOLDER_PERMISSIONS,
-            "mpc_enabled": checked(headphones.CONFIG.MPC_ENABLED)
+            "mpc_enabled": checked(headphones.CONFIG.MPC_ENABLED),
+            "email_enabled": checked(headphones.CONFIG.EMAIL_ENABLED),
+            "email_from": headphones.CONFIG.EMAIL_FROM,
+            "email_to": headphones.CONFIG.EMAIL_TO,
+            "email_smtp_server": headphones.CONFIG.EMAIL_SMTP_SERVER,
+            "email_smtp_user": headphones.CONFIG.EMAIL_SMTP_USER,
+            "email_smtp_password": headphones.CONFIG.EMAIL_SMTP_PASSWORD,
+            "email_smtp_port": int(headphones.CONFIG.EMAIL_SMTP_PORT),
+            "email_ssl": checked(headphones.CONFIG.EMAIL_SSL),
+            "email_tls": checked(headphones.CONFIG.EMAIL_TLS),
+            "email_onsnatch": checked(headphones.CONFIG.EMAIL_ONSNATCH),
+            "idtag": checked(headphones.CONFIG.IDTAG)
         }
 
         # Need to convert EXTRAS to a dictionary we can pass to the config:
@@ -1186,17 +1286,18 @@ class WebInterface(object):
         # Handle the variable config options. Note - keys with False values aren't getting passed
 
         checked_configs = [
-            "launch_browser", "enable_https", "api_enabled", "use_blackhole", "headphones_indexer", "use_newznab", "newznab_enabled",
+            "launch_browser", "enable_https", "api_enabled", "use_blackhole", "headphones_indexer", "use_newznab", "newznab_enabled", "use_torznab", "torznab_enabled",
             "use_nzbsorg", "use_omgwtfnzbs", "use_kat", "use_piratebay", "use_oldpiratebay", "use_mininova", "use_waffles", "use_rutracker",
-            "use_whatcd", "preferred_bitrate_allow_lossless", "detect_bitrate", "freeze_db", "cue_split", "move_files", "rename_files",
-            "correct_metadata", "cleanup_files", "keep_nfo", "add_album_art", "embed_album_art", "embed_lyrics", "replace_existing_folders",
-            "file_underscores", "include_extras", "autowant_upcoming", "autowant_all", "autowant_manually_added", "keep_torrent_files", "music_encoder",
+            "use_whatcd", "use_strike", "preferred_bitrate_allow_lossless", "detect_bitrate", "ignore_clean_releases", "freeze_db", "cue_split", "move_files",
+            "rename_files", "correct_metadata", "cleanup_files", "keep_nfo", "add_album_art", "embed_album_art", "embed_lyrics",
+            "replace_existing_folders", "keep_original_folder", "file_underscores", "include_extras", "official_releases_only",
+            "wait_until_release_date", "autowant_upcoming", "autowant_all", "autowant_manually_added", "do_not_process_unmatched", "keep_torrent_files", "music_encoder",
             "encoderlossless", "encoder_multicore", "delete_lossless_files", "growl_enabled", "growl_onsnatch", "prowl_enabled",
             "prowl_onsnatch", "xbmc_enabled", "xbmc_update", "xbmc_notify", "lms_enabled", "plex_enabled", "plex_update", "plex_notify",
             "nma_enabled", "nma_onsnatch", "pushalot_enabled", "pushalot_onsnatch", "synoindex_enabled", "pushover_enabled",
             "pushover_onsnatch", "pushbullet_enabled", "pushbullet_onsnatch", "subsonic_enabled", "twitter_enabled", "twitter_onsnatch",
             "osx_notify_enabled", "osx_notify_onsnatch", "boxcar_enabled", "boxcar_onsnatch", "songkick_enabled", "songkick_filter_enabled",
-            "mpc_enabled"
+            "mpc_enabled", "email_enabled", "email_ssl", "email_tls", "email_onsnatch", "customauth", "idtag"
         ]
         for checked_config in checked_configs:
             if checked_config not in kwargs:
@@ -1223,6 +1324,21 @@ class WebInterface(object):
                         del kwargs[key]
                 extra_newznabs.append((newznab_host, newznab_api, newznab_enabled))
 
+        extra_torznabs = []
+        for kwarg in [x for x in kwargs if x.startswith('torznab_host')]:
+            torznab_host_key = kwarg
+            torznab_number = kwarg[12:]
+            if len(torznab_number):
+                torznab_api_key = 'torznab_api' + torznab_number
+                torznab_enabled_key = 'torznab_enabled' + torznab_number
+                torznab_host = kwargs.get(torznab_host_key, '')
+                torznab_api = kwargs.get(torznab_api_key, '')
+                torznab_enabled = int(kwargs.get(torznab_enabled_key, 0))
+                for key in [torznab_host_key, torznab_api_key, torznab_enabled_key]:
+                    if key in kwargs:
+                        del kwargs[key]
+                extra_torznabs.append((torznab_host, torznab_api, torznab_enabled))
+
         # Convert the extras to list then string. Coming in as 0 or 1 (append new extras to the end)
         temp_extras_list = []
 
@@ -1248,11 +1364,18 @@ class WebInterface(object):
                 del kwargs[extra]
 
         headphones.CONFIG.EXTRAS = ','.join(str(n) for n in temp_extras_list)
+
         headphones.CONFIG.clear_extra_newznabs()
+        headphones.CONFIG.clear_extra_torznabs()
+
         headphones.CONFIG.process_kwargs(kwargs)
+
         for extra_newznab in extra_newznabs:
             headphones.CONFIG.add_extra_newznab(extra_newznab)
 
+        for extra_torznab in extra_torznabs:
+            headphones.CONFIG.add_extra_torznab(extra_torznab)
+
         # Sanity checking
         if headphones.CONFIG.SEARCH_INTERVAL and headphones.CONFIG.SEARCH_INTERVAL < 360:
             logger.info("Search interval too low. Resetting to 6 hour minimum")
@@ -1396,6 +1519,25 @@ class WebInterface(object):
             logger.warn(msg)
         return msg
 
+    @cherrypy.expose
+    def testPushover(self):
+        logger.info(u"Sending Pushover notification")
+        pushover = notifiers.PUSHOVER()
+        result = pushover.notify("hooray!", "This is a test")
+        return str(result)
+
+    @cherrypy.expose
+    def testPlex(self):
+        logger.info(u"Testing plex notifications")
+        plex = notifiers.Plex()
+        plex.notify("hellooooo", "test album!", "")
+
+    @cherrypy.expose
+    def testPushbullet(self):
+        logger.info("Testing Pushbullet notifications")
+        pushbullet = notifiers.PUSHBULLET()
+        pushbullet.notify("it works!")
+
 class Artwork(object):
     @cherrypy.expose
     def index(self):

init-scripts/init.osx

@@ -3,16 +3,16 @@
 <plist version="1.0">
 <dict>
   <key>Label</key>
-  <string>com.headphones.headphones</string>
+  <string>headphones</string>
   <key>ProgramArguments</key>
   <array>
+       <!-- Modify these two lines if you need to to reflect your python location and Headphones install location -->
        <string>/usr/bin/python</string>
-       <string>/Applications/Headphones/headphones.py</string>
-       <string>--quiet</string>
-       <string>--daemon</string>
-       <string>--nolaunch</string>
+       <string>/Applications/Headphones/Headphones.py</string>
   </array>
   <key>RunAtLoad</key>
   <true/>
+  <key>KeepAlive</key>
+  <true/>
 </dict>
-</plist>
+</plist>

lib/bs4/__init__.py

@@ -17,8 +17,8 @@ http://www.crummy.com/software/BeautifulSoup/bs4/doc/
 """
 
 __author__ = "Leonard Richardson (leonardr@segfault.org)"
-__version__ = "4.3.2"
-__copyright__ = "Copyright (c) 2004-2013 Leonard Richardson"
+__version__ = "4.4.0"
+__copyright__ = "Copyright (c) 2004-2015 Leonard Richardson"
 __license__ = "MIT"
 
 __all__ = ['BeautifulSoup']
@@ -45,7 +45,7 @@ from .element import (
 
 # The very first thing we do is give a useful error if someone is
 # running this code under Python 3 without converting it.
-syntax_error = u'You are trying to run the Python 2 version of Beautiful Soup under Python 3. This will not work. You need to convert the code, either by installing it (`python setup.py install`) or by running 2to3 (`2to3 -w bs4`).'
+'You are trying to run the Python 2 version of Beautiful Soup under Python 3. This will not work.'<>'You need to convert the code, either by installing it (`python setup.py install`) or by running 2to3 (`2to3 -w bs4`).'
 
 class BeautifulSoup(Tag):
     """
@@ -77,8 +77,11 @@ class BeautifulSoup(Tag):
 
     ASCII_SPACES = '\x20\x0a\x09\x0c\x0d'
 
+    NO_PARSER_SPECIFIED_WARNING = "No parser was explicitly specified, so I'm using the best available %(markup_type)s parser for this system (\"%(parser)s\"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.\n\nTo get rid of this warning, change this:\n\n BeautifulSoup([your markup])\n\nto this:\n\n BeautifulSoup([your markup], \"%(parser)s\")\n"
+
     def __init__(self, markup="", features=None, builder=None,
-                 parse_only=None, from_encoding=None, **kwargs):
+                 parse_only=None, from_encoding=None, exclude_encodings=None,
+                 **kwargs):
         """The Soup object is initialized as the 'root tag', and the
         provided markup (which can be a string or a file-like object)
         is fed into the underlying parser."""
@@ -114,9 +117,9 @@ class BeautifulSoup(Tag):
             del kwargs['isHTML']
             warnings.warn(
                 "BS4 does not respect the isHTML argument to the "
-                "BeautifulSoup constructor. You can pass in features='html' "
-                "or features='xml' to get a builder capable of handling "
-                "one or the other.")
+                "BeautifulSoup constructor. Suggest you use "
+                "features='lxml' for HTML and features='lxml-xml' for "
+                "XML.")
 
         def deprecated_argument(old_name, new_name):
             if old_name in kwargs:
@@ -140,6 +143,7 @@ class BeautifulSoup(Tag):
                 "__init__() got an unexpected keyword argument '%s'" % arg)
 
         if builder is None:
+            original_features = features
             if isinstance(features, basestring):
                 features = [features]
             if features is None or len(features) == 0:
@@ -151,6 +155,16 @@ class BeautifulSoup(Tag):
                     "requested: %s. Do you need to install a parser library?"
                     % ",".join(features))
             builder = builder_class()
+            if not (original_features == builder.NAME or
+                    original_features in builder.ALTERNATE_NAMES):
+                if builder.is_xml:
+                    markup_type = "XML"
+                else:
+                    markup_type = "HTML"
+                warnings.warn(self.NO_PARSER_SPECIFIED_WARNING % dict(
+                    parser=builder.NAME,
+                    markup_type=markup_type))
+
         self.builder = builder
         self.is_xml = builder.is_xml
         self.builder.soup = self
@@ -178,6 +192,8 @@ class BeautifulSoup(Tag):
                 # system. Just let it go.
                 pass
             if is_file:
+                if isinstance(markup, unicode):
+                    markup = markup.encode("utf8")
                 warnings.warn(
                     '"%s" looks like a filename, not markup. You should probably open this file and pass the filehandle into Beautiful Soup.' % markup)
             if markup[:5] == "http:" or markup[:6] == "https:":
@@ -185,12 +201,15 @@ class BeautifulSoup(Tag):
                 # Python 3 otherwise.
                 if ((isinstance(markup, bytes) and not b' ' in markup)
                     or (isinstance(markup, unicode) and not u' ' in markup)):
+                    if isinstance(markup, unicode):
+                        markup = markup.encode("utf8")
                     warnings.warn(
                         '"%s" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.' % markup)
 
         for (self.markup, self.original_encoding, self.declared_html_encoding,
          self.contains_replacement_characters) in (
-            self.builder.prepare_markup(markup, from_encoding)):
+             self.builder.prepare_markup(
+                 markup, from_encoding, exclude_encodings=exclude_encodings)):
             self.reset()
             try:
                 self._feed()
@@ -203,6 +222,16 @@ class BeautifulSoup(Tag):
         self.markup = None
         self.builder.soup = None
 
+    def __copy__(self):
+        return type(self)(self.encode(), builder=self.builder)
+
+    def __getstate__(self):
+        # Frequently a tree builder can't be pickled.
+        d = dict(self.__dict__)
+        if 'builder' in d and not self.builder.picklable:
+            del d['builder']
+        return d
+
     def _feed(self):
         # Convert the document to Unicode.
         self.builder.reset()
@@ -229,9 +258,7 @@ class BeautifulSoup(Tag):
 
     def new_string(self, s, subclass=NavigableString):
         """Create a new NavigableString associated with this soup."""
-        navigable = subclass(s)
-        navigable.setup()
-        return navigable
+        return subclass(s)
 
     def insert_before(self, successor):
         raise NotImplementedError("BeautifulSoup objects don't support insert_before().")
@@ -290,14 +317,49 @@ class BeautifulSoup(Tag):
     def object_was_parsed(self, o, parent=None, most_recent_element=None):
         """Add an object to the parse tree."""
         parent = parent or self.currentTag
-        most_recent_element = most_recent_element or self._most_recent_element
-        o.setup(parent, most_recent_element)
+        previous_element = most_recent_element or self._most_recent_element
+
+        next_element = previous_sibling = next_sibling = None
+        if isinstance(o, Tag):
+            next_element = o.next_element
+            next_sibling = o.next_sibling
+            previous_sibling = o.previous_sibling
+            if not previous_element:
+                previous_element = o.previous_element
+
+        o.setup(parent, previous_element, next_element, previous_sibling, next_sibling)
 
-        if most_recent_element is not None:
-            most_recent_element.next_element = o
         self._most_recent_element = o
         parent.contents.append(o)
 
+        if parent.next_sibling:
+            # This node is being inserted into an element that has
+            # already been parsed. Deal with any dangling references.
+            index = parent.contents.index(o)
+            if index == 0:
+                previous_element = parent
+                previous_sibling = None
+            else:
+                previous_element = previous_sibling = parent.contents[index-1]
+            if index == len(parent.contents)-1:
+                next_element = parent.next_sibling
+                next_sibling = None
+            else:
+                next_element = next_sibling = parent.contents[index+1]
+
+            o.previous_element = previous_element
+            if previous_element:
+                previous_element.next_element = o
+            o.next_element = next_element
+            if next_element:
+                next_element.previous_element = o
+            o.next_sibling = next_sibling
+            if next_sibling:
+                next_sibling.previous_sibling = o
+            o.previous_sibling = previous_sibling
+            if previous_sibling:
+                previous_sibling.next_sibling = o
+
     def _popToTag(self, name, nsprefix=None, inclusivePop=True):
         """Pops the tag stack up to and including the most recent
         instance of the given tag. If inclusivePop is false, pops the tag

lib/bs4/builder/__init__.py

@@ -80,9 +80,12 @@ builder_registry = TreeBuilderRegistry()
 class TreeBuilder(object):
     """Turn a document into a Beautiful Soup object tree."""
 
+    NAME = "[Unknown tree builder]"
+    ALTERNATE_NAMES = []
     features = []
 
     is_xml = False
+    picklable = False
     preserve_whitespace_tags = set()
     empty_element_tags = None # A tag will be considered an empty-element
                               # tag when and only when it has no contents.

lib/bs4/builder/_html5lib.py

@@ -2,6 +2,7 @@ __all__ = [
     'HTML5TreeBuilder',
     ]
 
+from pdb import set_trace
 import warnings
 from bs4.builder import (
     PERMISSIVE,
@@ -9,7 +10,10 @@ from bs4.builder import (
     HTML_5,
     HTMLTreeBuilder,
     )
-from bs4.element import NamespacedAttribute
+from bs4.element import (
+    NamespacedAttribute,
+    whitespace_re,
+)
 import html5lib
 from html5lib.constants import namespaces
 from bs4.element import (
@@ -22,11 +26,20 @@ from bs4.element import (
 class HTML5TreeBuilder(HTMLTreeBuilder):
     """Use html5lib to build a tree."""
 
-    features = ['html5lib', PERMISSIVE, HTML_5, HTML]
+    NAME = "html5lib"
 
-    def prepare_markup(self, markup, user_specified_encoding):
+    features = [NAME, PERMISSIVE, HTML_5, HTML]
+
+    def prepare_markup(self, markup, user_specified_encoding,
+                       document_declared_encoding=None, exclude_encodings=None):
         # Store the user-specified encoding for use later on.
         self.user_specified_encoding = user_specified_encoding
+
+        # document_declared_encoding and exclude_encodings aren't used
+        # ATM because the html5lib TreeBuilder doesn't use
+        # UnicodeDammit.
+        if exclude_encodings:
+            warnings.warn("You provided a value for exclude_encoding, but the html5lib tree builder doesn't support exclude_encoding.")
         yield (markup, None, None, False)
 
     # These methods are defined by Beautiful Soup.
@@ -101,7 +114,13 @@ class AttrList(object):
     def __iter__(self):
         return list(self.attrs.items()).__iter__()
     def __setitem__(self, name, value):
-        "set attr", name, value
+        # If this attribute is a multi-valued attribute for this element,
+        # turn its value into a list.
+        list_attr = HTML5TreeBuilder.cdata_list_attributes
+        if (name in list_attr['*']
+            or (self.element.name in list_attr
+                and name in list_attr[self.element.name])):
+            value = whitespace_re.split(value)
         self.element[name] = value
     def items(self):
         return list(self.attrs.items())
@@ -161,6 +180,12 @@ class Element(html5lib.treebuilders._base.Node):
             # immediately after the parent, if it has no children.)
             if self.element.contents:
                 most_recent_element = self.element._last_descendant(False)
+            elif self.element.next_element is not None:
+                # Something from further ahead in the parse tree is
+                # being inserted into this earlier element. This is
+                # very annoying because it means an expensive search
+                # for the last element in the tree.
+                most_recent_element = self.soup._last_descendant()
             else:
                 most_recent_element = self.element
 
@@ -172,6 +197,7 @@ class Element(html5lib.treebuilders._base.Node):
         return AttrList(self.element)
 
     def setAttributes(self, attributes):
+
         if attributes is not None and len(attributes) > 0:
 
             converted_attributes = []
@@ -218,6 +244,9 @@ class Element(html5lib.treebuilders._base.Node):
 
     def reparentChildren(self, new_parent):
         """Move all of this tag's children into another tag."""
+        # print "MOVE", self.element.contents
+        # print "FROM", self.element
+        # print "TO", new_parent.element
         element = self.element
         new_parent_element = new_parent.element
         # Determine what this tag's next_element will be once all the children
@@ -236,17 +265,28 @@ class Element(html5lib.treebuilders._base.Node):
             new_parents_last_descendant_next_element = new_parent_element.next_element
 
         to_append = element.contents
-        append_after = new_parent.element.contents
+        append_after = new_parent_element.contents
         if len(to_append) > 0:
             # Set the first child's previous_element and previous_sibling
             # to elements within the new parent
             first_child = to_append[0]
-            first_child.previous_element = new_parents_last_descendant
+            if new_parents_last_descendant:
+                first_child.previous_element = new_parents_last_descendant
+            else:
+                first_child.previous_element = new_parent_element
             first_child.previous_sibling = new_parents_last_child
+            if new_parents_last_descendant:
+                new_parents_last_descendant.next_element = first_child
+            else:
+                new_parent_element.next_element = first_child
+            if new_parents_last_child:
+                new_parents_last_child.next_sibling = first_child
 
             # Fix the last child's next_element and next_sibling
             last_child = to_append[-1]
             last_child.next_element = new_parents_last_descendant_next_element
+            if new_parents_last_descendant_next_element:
+                new_parents_last_descendant_next_element.previous_element = last_child
             last_child.next_sibling = None
 
         for child in to_append:
@@ -257,6 +297,10 @@ class Element(html5lib.treebuilders._base.Node):
         element.contents = []
         element.next_element = final_next_element
 
+        # print "DONE WITH MOVE"
+        # print "FROM", self.element
+        # print "TO", new_parent_element
+
     def cloneNode(self):
         tag = self.soup.new_tag(self.element.name, self.namespace)
         node = Element(tag, self.soup, self.namespace)
@@ -268,7 +312,7 @@ class Element(html5lib.treebuilders._base.Node):
         return self.element.contents
 
     def getNameTuple(self):
-        if self.namespace is None:
+        if self.namespace == None:
             return namespaces["html"], self.name
         else:
             return self.namespace, self.name

lib/bs4/builder/_htmlparser.py

@@ -4,10 +4,16 @@ __all__ = [
     'HTMLParserTreeBuilder',
     ]
 
-from HTMLParser import (
-    HTMLParser,
-    HTMLParseError,
-    )
+from HTMLParser import HTMLParser
+
+try:
+    from HTMLParser import HTMLParseError
+except ImportError, e:
+    # HTMLParseError is removed in Python 3.5. Since it can never be
+    # thrown in 3.5, we can just define our own class as a placeholder.
+    class HTMLParseError(Exception):
+        pass
+
 import sys
 import warnings
 
@@ -19,10 +25,10 @@ import warnings
 # At the end of this file, we monkeypatch HTMLParser so that
 # strict=True works well on Python 3.2.2.
 major, minor, release = sys.version_info[:3]
-CONSTRUCTOR_TAKES_STRICT = (
-    major > 3
-    or (major == 3 and minor > 2)
-    or (major == 3 and minor == 2 and release >= 3))
+CONSTRUCTOR_TAKES_STRICT = major == 3 and minor == 2 and release >= 3
+CONSTRUCTOR_STRICT_IS_DEPRECATED = major == 3 and minor == 3
+CONSTRUCTOR_TAKES_CONVERT_CHARREFS = major == 3 and minor >= 4
+
 
 from bs4.element import (
     CData,
@@ -63,7 +69,8 @@ class BeautifulSoupHTMLParser(HTMLParser):
 
     def handle_charref(self, name):
         # XXX workaround for a bug in HTMLParser. Remove this once
-        # it's fixed.
+        # it's fixed in all supported versions.
+        # http://bugs.python.org/issue13633
         if name.startswith('x'):
             real_name = int(name.lstrip('x'), 16)
         elif name.startswith('X'):
@@ -113,14 +120,6 @@ class BeautifulSoupHTMLParser(HTMLParser):
 
     def handle_pi(self, data):
         self.soup.endData()
-        if data.endswith("?") and data.lower().startswith("xml"):
-            # "An XHTML processing instruction using the trailing '?'
-            # will cause the '?' to be included in data." - HTMLParser
-            # docs.
-            #
-            # Strip the question mark so we don't end up with two
-            # question marks.
-            data = data[:-1]
         self.soup.handle_data(data)
         self.soup.endData(ProcessingInstruction)
 
@@ -128,15 +127,19 @@ class BeautifulSoupHTMLParser(HTMLParser):
 class HTMLParserTreeBuilder(HTMLTreeBuilder):
 
     is_xml = False
-    features = [HTML, STRICT, HTMLPARSER]
+    picklable = True
+    NAME = HTMLPARSER
+    features = [NAME, HTML, STRICT]
 
     def __init__(self, *args, **kwargs):
-        if CONSTRUCTOR_TAKES_STRICT:
+        if CONSTRUCTOR_TAKES_STRICT and not CONSTRUCTOR_STRICT_IS_DEPRECATED:
             kwargs['strict'] = False
+        if CONSTRUCTOR_TAKES_CONVERT_CHARREFS:
+            kwargs['convert_charrefs'] = False
         self.parser_args = (args, kwargs)
 
     def prepare_markup(self, markup, user_specified_encoding=None,
-                       document_declared_encoding=None):
+                       document_declared_encoding=None, exclude_encodings=None):
         """
         :return: A 4-tuple (markup, original encoding, encoding
         declared within markup, whether any characters had to be
@@ -147,7 +150,8 @@ class HTMLParserTreeBuilder(HTMLTreeBuilder):
             return
 
         try_encodings = [user_specified_encoding, document_declared_encoding]
-        dammit = UnicodeDammit(markup, try_encodings, is_html=True)
+        dammit = UnicodeDammit(markup, try_encodings, is_html=True,
+                               exclude_encodings=exclude_encodings)
         yield (dammit.markup, dammit.original_encoding,
                dammit.declared_html_encoding,
                dammit.contains_replacement_characters)

lib/bs4/builder/_lxml.py

@@ -7,7 +7,12 @@ from io import BytesIO
 from StringIO import StringIO
 import collections
 from lxml import etree
-from bs4.element import Comment, Doctype, NamespacedAttribute
+from bs4.element import (
+    Comment,
+    Doctype,
+    NamespacedAttribute,
+    ProcessingInstruction,
+)
 from bs4.builder import (
     FAST,
     HTML,
@@ -25,8 +30,11 @@ class LXMLTreeBuilderForXML(TreeBuilder):
 
     is_xml = True
 
+    NAME = "lxml-xml"
+    ALTERNATE_NAMES = ["xml"]
+
     # Well, it's permissive by XML parser standards.
-    features = [LXML, XML, FAST, PERMISSIVE]
+    features = [NAME, LXML, XML, FAST, PERMISSIVE]
 
     CHUNK_SIZE = 512
 
@@ -70,6 +78,7 @@ class LXMLTreeBuilderForXML(TreeBuilder):
             return (None, tag)
 
     def prepare_markup(self, markup, user_specified_encoding=None,
+                       exclude_encodings=None,
                        document_declared_encoding=None):
         """
         :yield: A series of 4-tuples.
@@ -95,7 +104,8 @@ class LXMLTreeBuilderForXML(TreeBuilder):
         # the document as each one in turn.
         is_html = not self.is_xml
         try_encodings = [user_specified_encoding, document_declared_encoding]
-        detector = EncodingDetector(markup, try_encodings, is_html)
+        detector = EncodingDetector(
+            markup, try_encodings, is_html, exclude_encodings)
         for encoding in detector.encodings:
             yield (detector.markup, encoding, document_declared_encoding, False)
 
@@ -189,7 +199,9 @@ class LXMLTreeBuilderForXML(TreeBuilder):
             self.nsmaps.pop()
 
     def pi(self, target, data):
-        pass
+        self.soup.endData()
+        self.soup.handle_data(target + ' ' + data)
+        self.soup.endData(ProcessingInstruction)
 
     def data(self, content):
         self.soup.handle_data(content)
@@ -212,7 +224,10 @@ class LXMLTreeBuilderForXML(TreeBuilder):
 
 class LXMLTreeBuilder(HTMLTreeBuilder, LXMLTreeBuilderForXML):
 
-    features = [LXML, HTML, FAST, PERMISSIVE]
+    NAME = LXML
+    ALTERNATE_NAMES = ["lxml-html"]
+
+    features = ALTERNATE_NAMES + [NAME, HTML, FAST, PERMISSIVE]
     is_xml = False
 
     def default_parser(self, encoding):

lib/bs4/dammit.py

@@ -3,10 +3,11 @@
 
 This library converts a bytestream to Unicode through any means
 necessary. It is heavily based on code from Mark Pilgrim's Universal
-Feed Parser. It works best on XML and XML, but it does not rewrite the
+Feed Parser. It works best on XML and HTML, but it does not rewrite the
 XML or HTML to reflect a new encoding; that's the tree builder's job.
 """
 
+from pdb import set_trace
 import codecs
 from htmlentitydefs import codepoint2name
 import re
@@ -212,8 +213,11 @@ class EncodingDetector:
 
     5. Windows-1252.
     """
-    def __init__(self, markup, override_encodings=None, is_html=False):
+    def __init__(self, markup, override_encodings=None, is_html=False,
+                 exclude_encodings=None):
         self.override_encodings = override_encodings or []
+        exclude_encodings = exclude_encodings or []
+        self.exclude_encodings = set([x.lower() for x in exclude_encodings])
         self.chardet_encoding = None
         self.is_html = is_html
         self.declared_encoding = None
@@ -224,6 +228,8 @@ class EncodingDetector:
     def _usable(self, encoding, tried):
         if encoding is not None:
             encoding = encoding.lower()
+            if encoding in self.exclude_encodings:
+                return False
             if encoding not in tried:
                 tried.add(encoding)
                 return True
@@ -266,6 +272,9 @@ class EncodingDetector:
     def strip_byte_order_mark(cls, data):
         """If a byte-order mark is present, strip it and return the encoding it implies."""
         encoding = None
+        if isinstance(data, unicode):
+            # Unicode data cannot have a byte-order mark.
+            return data, encoding
         if (len(data) >= 4) and (data[:2] == b'\xfe\xff') \
                and (data[2:4] != '\x00\x00'):
             encoding = 'utf-16be'
@@ -299,14 +308,14 @@ class EncodingDetector:
         else:
             xml_endpos = 1024
             html_endpos = max(2048, int(len(markup) * 0.05))
-
+            
         declared_encoding = None
         declared_encoding_match = xml_encoding_re.search(markup, endpos=xml_endpos)
         if not declared_encoding_match and is_html:
             declared_encoding_match = html_meta_re.search(markup, endpos=html_endpos)
         if declared_encoding_match is not None:
             declared_encoding = declared_encoding_match.groups()[0].decode(
-                'ascii')
+                'ascii', 'replace')
         if declared_encoding:
             return declared_encoding.lower()
         return None
@@ -331,13 +340,14 @@ class UnicodeDammit:
         ]
 
     def __init__(self, markup, override_encodings=[],
-                 smart_quotes_to=None, is_html=False):
+                 smart_quotes_to=None, is_html=False, exclude_encodings=[]):
         self.smart_quotes_to = smart_quotes_to
         self.tried_encodings = []
         self.contains_replacement_characters = False
         self.is_html = is_html
 
-        self.detector = EncodingDetector(markup, override_encodings, is_html)
+        self.detector = EncodingDetector(
+            markup, override_encodings, is_html, exclude_encodings)
 
         # Short-circuit if the data is in Unicode to begin with.
         if isinstance(markup, unicode) or markup == '':

lib/bs4/diagnose.py

@@ -33,12 +33,21 @@ def diagnose(data):
 
     if 'lxml' in basic_parsers:
         basic_parsers.append(["lxml", "xml"])
-        from lxml import etree
-        print "Found lxml version %s" % ".".join(map(str,etree.LXML_VERSION))
+        try:
+            from lxml import etree
+            print "Found lxml version %s" % ".".join(map(str,etree.LXML_VERSION))
+        except ImportError, e:
+            print (
+                "lxml is not installed or couldn't be imported.")
+
 
     if 'html5lib' in basic_parsers:
-        import html5lib
-        print "Found html5lib version %s" % html5lib.__version__
+        try:
+            import html5lib
+            print "Found html5lib version %s" % html5lib.__version__
+        except ImportError, e:
+            print (
+                "html5lib is not installed or couldn't be imported.")
 
     if hasattr(data, 'read'):
         data = data.read()
@@ -135,7 +144,7 @@ def rword(length=5):
 def rsentence(length=4):
     "Generate a random sentence-like string."
     return " ".join(rword(random.randint(4,9)) for i in range(length))
-
+        
 def rdoc(num_elements=1000):
     """Randomly generate an invalid HTML document."""
     tag_names = ['p', 'div', 'span', 'i', 'b', 'script', 'table']
@@ -159,7 +168,7 @@ def benchmark_parsers(num_elements=100000):
     print "Comparative parser benchmark on Beautiful Soup %s" % __version__
     data = rdoc(num_elements)
     print "Generated a large invalid HTML document (%d bytes)." % len(data)
-
+    
     for parser in ["lxml", ["lxml", "html"], "html5lib", "html.parser"]:
         success = False
         try:

lib/bs4/element.py

@@ -1,3 +1,4 @@
+from pdb import set_trace
 import collections
 import re
 import sys
@@ -185,24 +186,40 @@ class PageElement(object):
             return self.HTML_FORMATTERS.get(
                 name, HTMLAwareEntitySubstitution.substitute_xml)
 
-    def setup(self, parent=None, previous_element=None):
+    def setup(self, parent=None, previous_element=None, next_element=None,
+              previous_sibling=None, next_sibling=None):
         """Sets up the initial relations between this element and
         other elements."""
         self.parent = parent
+
         self.previous_element = previous_element
         if previous_element is not None:
             self.previous_element.next_element = self
-        self.next_element = None
-        self.previous_sibling = None
-        self.next_sibling = None
-        if self.parent is not None and self.parent.contents:
-            self.previous_sibling = self.parent.contents[-1]
+
+        self.next_element = next_element
+        if self.next_element:
+            self.next_element.previous_element = self
+
+        self.next_sibling = next_sibling
+        if self.next_sibling:
+            self.next_sibling.previous_sibling = self
+
+        if (not previous_sibling
+            and self.parent is not None and self.parent.contents):
+            previous_sibling = self.parent.contents[-1]
+
+        self.previous_sibling = previous_sibling
+        if previous_sibling:
             self.previous_sibling.next_sibling = self
 
     nextSibling = _alias("next_sibling")  # BS3
     previousSibling = _alias("previous_sibling")  # BS3
 
     def replace_with(self, replace_with):
+        if not self.parent:
+            raise ValueError(
+                "Cannot replace one element with another when the"
+                "element to be replaced is not part of a tree.")
         if replace_with is self:
             return
         if replace_with is self.parent:
@@ -216,6 +233,10 @@ class PageElement(object):
 
     def unwrap(self):
         my_parent = self.parent
+        if not self.parent:
+            raise ValueError(
+                "Cannot replace an element with its contents when that"
+                "element is not part of a tree.")
         my_index = self.parent.index(self)
         self.extract()
         for child in reversed(self.contents[:]):
@@ -240,17 +261,20 @@ class PageElement(object):
         last_child = self._last_descendant()
         next_element = last_child.next_element
 
-        if self.previous_element is not None:
+        if (self.previous_element is not None and
+            self.previous_element != next_element):
             self.previous_element.next_element = next_element
-        if next_element is not None:
+        if next_element is not None and next_element != self.previous_element:
             next_element.previous_element = self.previous_element
         self.previous_element = None
         last_child.next_element = None
 
         self.parent = None
-        if self.previous_sibling is not None:
+        if (self.previous_sibling is not None
+            and self.previous_sibling != self.next_sibling):
             self.previous_sibling.next_sibling = self.next_sibling
-        if self.next_sibling is not None:
+        if (self.next_sibling is not None
+            and self.next_sibling != self.previous_sibling):
             self.next_sibling.previous_sibling = self.previous_sibling
         self.previous_sibling = self.next_sibling = None
         return self
@@ -478,6 +502,10 @@ class PageElement(object):
     def _find_all(self, name, attrs, text, limit, generator, **kwargs):
         "Iterates over a generator looking for things that match."
 
+        if text is None and 'string' in kwargs:
+            text = kwargs['string']
+            del kwargs['string']
+
         if isinstance(name, SoupStrainer):
             strainer = name
         else:
@@ -548,17 +576,17 @@ class PageElement(object):
 
     # Methods for supporting CSS selectors.
 
-    tag_name_re = re.compile('^[a-z0-9]+$')
+    tag_name_re = re.compile('^[a-zA-Z0-9][-.a-zA-Z0-9:_]*$')
 
-    # /^(\w+)\[(\w+)([=~\|\^\$\*]?)=?"?([^\]"]*)"?\]$/
-    #   \---/  \---/\-------------/    \-------/
-    #     |      |         |               |
-    #     |      |         |           The value
-    #     |      |    ~,|,^,$,* or =
-    #     |   Attribute
+    # /^([a-zA-Z0-9][-.a-zA-Z0-9:_]*)\[(\w+)([=~\|\^\$\*]?)=?"?([^\]"]*)"?\]$/
+    #   \---------------------------/  \---/\-------------/    \-------/
+    #     |                              |         |               |
+    #     |                              |         |           The value
+    #     |                              |    ~,|,^,$,* or =
+    #     |                           Attribute
     #    Tag
     attribselect_re = re.compile(
-        r'^(?P<tag>\w+)?\[(?P<attribute>\w+)(?P<operator>[=~\|\^\$\*]?)' +
+        r'^(?P<tag>[a-zA-Z0-9][-.a-zA-Z0-9:_]*)?\[(?P<attribute>[\w-]+)(?P<operator>[=~\|\^\$\*]?)' +
         r'=?"?(?P<value>[^\]"]*)"?\]$'
         )
 
@@ -654,11 +682,17 @@ class NavigableString(unicode, PageElement):
         how to handle non-ASCII characters.
         """
         if isinstance(value, unicode):
-            return unicode.__new__(cls, value)
-        return unicode.__new__(cls, value, DEFAULT_OUTPUT_ENCODING)
+            u = unicode.__new__(cls, value)
+        else:
+            u = unicode.__new__(cls, value, DEFAULT_OUTPUT_ENCODING)
+        u.setup()
+        return u
 
     def __copy__(self):
-        return self
+        """A copy of a NavigableString has the same contents and class
+        as the original, but it is not connected to the parse tree.
+        """
+        return type(self)(self)
 
     def __getnewargs__(self):
         return (unicode(self),)
@@ -707,7 +741,7 @@ class CData(PreformattedString):
 class ProcessingInstruction(PreformattedString):
 
     PREFIX = u'<?'
-    SUFFIX = u'?>'
+    SUFFIX = u'>'
 
 class Comment(PreformattedString):
 
@@ -759,9 +793,12 @@ class Tag(PageElement):
         self.prefix = prefix
         if attrs is None:
             attrs = {}
-        elif attrs and builder.cdata_list_attributes:
-            attrs = builder._replace_cdata_list_attribute_values(
-                self.name, attrs)
+        elif attrs:
+            if builder is not None and builder.cdata_list_attributes:
+                attrs = builder._replace_cdata_list_attribute_values(
+                    self.name, attrs)
+            else:
+                attrs = dict(attrs)
         else:
             attrs = dict(attrs)
         self.attrs = attrs
@@ -778,6 +815,18 @@ class Tag(PageElement):
 
     parserClass = _alias("parser_class")  # BS3
 
+    def __copy__(self):
+        """A copy of a Tag is a new Tag, unconnected to the parse tree.
+        Its contents are a copy of the old Tag's contents.
+        """
+        clone = type(self)(None, self.builder, self.name, self.namespace,
+                           self.nsprefix, self.attrs)
+        for attr in ('can_be_empty_element', 'hidden'):
+            setattr(clone, attr, getattr(self, attr))
+        for child in self.contents:
+            clone.append(child.__copy__())
+        return clone
+
     @property
     def is_empty_element(self):
         """Is this tag an empty-element tag? (aka a self-closing tag)
@@ -971,15 +1020,25 @@ class Tag(PageElement):
         as defined in __eq__."""
         return not self == other
 
-    def __repr__(self, encoding=DEFAULT_OUTPUT_ENCODING):
+    def __repr__(self, encoding="unicode-escape"):
         """Renders this tag as a string."""
-        return self.encode(encoding)
+        if PY3K:
+            # "The return value must be a string object", i.e. Unicode
+            return self.decode()
+        else:
+            # "The return value must be a string object", i.e. a bytestring.
+            # By convention, the return value of __repr__ should also be
+            # an ASCII string.
+            return self.encode(encoding)
 
     def __unicode__(self):
         return self.decode()
 
     def __str__(self):
-        return self.encode()
+        if PY3K:
+            return self.decode()
+        else:
+            return self.encode()
 
     if PY3K:
         __str__ = __repr__ = __unicode__
@@ -1103,12 +1162,18 @@ class Tag(PageElement):
                        formatter="minimal"):
         """Renders the contents of this tag as a Unicode string.
 
+        :param indent_level: Each line of the rendering will be
+           indented this many spaces.
+
         :param eventual_encoding: The tag is destined to be
            encoded into this encoding. This method is _not_
            responsible for performing that encoding. This information
            is passed in so that it can be substituted in if the
            document contains a <META> tag that mentions the document's
            encoding.
+
+        :param formatter: The output formatter responsible for converting
+           entities to Unicode characters.
         """
         # First off, turn a string formatter into a function. This
         # will stop the lookup from happening over and over again.
@@ -1137,7 +1202,17 @@ class Tag(PageElement):
     def encode_contents(
         self, indent_level=None, encoding=DEFAULT_OUTPUT_ENCODING,
         formatter="minimal"):
-        """Renders the contents of this tag as a bytestring."""
+        """Renders the contents of this tag as a bytestring.
+
+        :param indent_level: Each line of the rendering will be
+           indented this many spaces.
+
+        :param eventual_encoding: The bytestring will be in this encoding.
+
+        :param formatter: The output formatter responsible for converting
+           entities to Unicode characters.
+        """
+
         contents = self.decode_contents(indent_level, encoding, formatter)
         return contents.encode(encoding)
 
@@ -1201,63 +1276,89 @@ class Tag(PageElement):
 
     _selector_combinators = ['>', '+', '~']
     _select_debug = False
-    def select(self, selector, _candidate_generator=None):
+    def select_one(self, selector):
         """Perform a CSS selection operation on the current element."""
-        tokens = selector.split()
+        value = self.select(selector, limit=1)
+        if value:
+            return value[0]
+        return None
+
+    def select(self, selector, _candidate_generator=None, limit=None):
+        """Perform a CSS selection operation on the current element."""
+
+        # Remove whitespace directly after the grouping operator ','
+        # then split into tokens.
+        tokens = re.sub(',[\s]*',',', selector).split()
         current_context = [self]
 
         if tokens[-1] in self._selector_combinators:
             raise ValueError(
                 'Final combinator "%s" is missing an argument.' % tokens[-1])
+
         if self._select_debug:
             print 'Running CSS selector "%s"' % selector
-        for index, token in enumerate(tokens):
-            if self._select_debug:
-                print ' Considering token "%s"' % token
-            recursive_candidate_generator = None
-            tag_name = None
+
+        for index, token_group in enumerate(tokens):
+            new_context = []
+            new_context_ids = set([])
+
+            # Grouping selectors, ie: p,a
+            grouped_tokens = token_group.split(',')
+            if '' in grouped_tokens:
+                raise ValueError('Invalid group selection syntax: %s' % token_group)
+
             if tokens[index-1] in self._selector_combinators:
                 # This token was consumed by the previous combinator. Skip it.
                 if self._select_debug:
                     print '  Token was consumed by the previous combinator.'
                 continue
-            # Each operation corresponds to a checker function, a rule
-            # for determining whether a candidate matches the
-            # selector. Candidates are generated by the active
-            # iterator.
-            checker = None
 
-            m = self.attribselect_re.match(token)
-            if m is not None:
-                # Attribute selector
-                tag_name, attribute, operator, value = m.groups()
-                checker = self._attribute_checker(operator, attribute, value)
+            for token in grouped_tokens:
+                if self._select_debug:
+                    print ' Considering token "%s"' % token
+                recursive_candidate_generator = None
+                tag_name = None
 
-            elif '#' in token:
-                # ID selector
-                tag_name, tag_id = token.split('#', 1)
-                def id_matches(tag):
-                    return tag.get('id', None) == tag_id
-                checker = id_matches
+                # Each operation corresponds to a checker function, a rule
+                # for determining whether a candidate matches the
+                # selector. Candidates are generated by the active
+                # iterator.
+                checker = None
 
-            elif '.' in token:
-                # Class selector
-                tag_name, klass = token.split('.', 1)
-                classes = set(klass.split('.'))
-                def classes_match(candidate):
-                    return classes.issubset(candidate.get('class', []))
-                checker = classes_match
+                m = self.attribselect_re.match(token)
+                if m is not None:
+                    # Attribute selector
+                    tag_name, attribute, operator, value = m.groups()
+                    checker = self._attribute_checker(operator, attribute, value)
 
-            elif ':' in token:
-                # Pseudo-class
-                tag_name, pseudo = token.split(':', 1)
-                if tag_name == '':
-                    raise ValueError(
-                        "A pseudo-class must be prefixed with a tag name.")
-                pseudo_attributes = re.match('([a-zA-Z\d-]+)\(([a-zA-Z\d]+)\)', pseudo)
-                found = []
-                if pseudo_attributes is not None:
-                    pseudo_type, pseudo_value = pseudo_attributes.groups()
+                elif '#' in token:
+                    # ID selector
+                    tag_name, tag_id = token.split('#', 1)
+                    def id_matches(tag):
+                        return tag.get('id', None) == tag_id
+                    checker = id_matches
+
+                elif '.' in token:
+                    # Class selector
+                    tag_name, klass = token.split('.', 1)
+                    classes = set(klass.split('.'))
+                    def classes_match(candidate):
+                        return classes.issubset(candidate.get('class', []))
+                    checker = classes_match
+
+                elif ':' in token:
+                    # Pseudo-class
+                    tag_name, pseudo = token.split(':', 1)
+                    if tag_name == '':
+                        raise ValueError(
+                            "A pseudo-class must be prefixed with a tag name.")
+                    pseudo_attributes = re.match('([a-zA-Z\d-]+)\(([a-zA-Z\d]+)\)', pseudo)
+                    found = []
+                    if pseudo_attributes is None:
+                        pseudo_type = pseudo
+                        pseudo_value = None
+                    else:
+                        pseudo_type, pseudo_value = pseudo_attributes.groups()
                     if pseudo_type == 'nth-of-type':
                         try:
                             pseudo_value = int(pseudo_value)
@@ -1286,109 +1387,110 @@ class Tag(PageElement):
                         raise NotImplementedError(
                             'Only the following pseudo-classes are implemented: nth-of-type.')
 
-            elif token == '*':
-                # Star selector -- matches everything
-                pass
-            elif token == '>':
-                # Run the next token as a CSS selector against the
-                # direct children of each tag in the current context.
-                recursive_candidate_generator = lambda tag: tag.children
-            elif token == '~':
-                # Run the next token as a CSS selector against the
-                # siblings of each tag in the current context.
-                recursive_candidate_generator = lambda tag: tag.next_siblings
-            elif token == '+':
-                # For each tag in the current context, run the next
-                # token as a CSS selector against the tag's next
-                # sibling that's a tag.
-                def next_tag_sibling(tag):
-                    yield tag.find_next_sibling(True)
-                recursive_candidate_generator = next_tag_sibling
+                elif token == '*':
+                    # Star selector -- matches everything
+                    pass
+                elif token == '>':
+                    # Run the next token as a CSS selector against the
+                    # direct children of each tag in the current context.
+                    recursive_candidate_generator = lambda tag: tag.children
+                elif token == '~':
+                    # Run the next token as a CSS selector against the
+                    # siblings of each tag in the current context.
+                    recursive_candidate_generator = lambda tag: tag.next_siblings
+                elif token == '+':
+                    # For each tag in the current context, run the next
+                    # token as a CSS selector against the tag's next
+                    # sibling that's a tag.
+                    def next_tag_sibling(tag):
+                        yield tag.find_next_sibling(True)
+                    recursive_candidate_generator = next_tag_sibling
 
-            elif self.tag_name_re.match(token):
-                # Just a tag name.
-                tag_name = token
-            else:
-                raise ValueError(
-                    'Unsupported or invalid CSS selector: "%s"' % token)
-
-            if recursive_candidate_generator:
-                # This happens when the selector looks like  "> foo".
-                #
-                # The generator calls select() recursively on every
-                # member of the current context, passing in a different
-                # candidate generator and a different selector.
-                #
-                # In the case of "> foo", the candidate generator is
-                # one that yields a tag's direct children (">"), and
-                # the selector is "foo".
-                next_token = tokens[index+1]
-                def recursive_select(tag):
-                    if self._select_debug:
-                        print '    Calling select("%s") recursively on %s %s' % (next_token, tag.name, tag.attrs)
-                        print '-' * 40
-                    for i in tag.select(next_token, recursive_candidate_generator):
-                        if self._select_debug:
-                            print '(Recursive select picked up candidate %s %s)' % (i.name, i.attrs)
-                        yield i
-                    if self._select_debug:
-                        print '-' * 40
-                _use_candidate_generator = recursive_select
-            elif _candidate_generator is None:
-                # By default, a tag's candidates are all of its
-                # children. If tag_name is defined, only yield tags
-                # with that name.
-                if self._select_debug:
-                    if tag_name:
-                        check = "[any]"
-                    else:
-                        check = tag_name
-                    print '   Default candidate generator, tag name="%s"' % check
-                if self._select_debug:
-                    # This is redundant with later code, but it stops
-                    # a bunch of bogus tags from cluttering up the
-                    # debug log.
-                    def default_candidate_generator(tag):
-                        for child in tag.descendants:
-                            if not isinstance(child, Tag):
-                                continue
-                            if tag_name and not child.name == tag_name:
-                                continue
-                            yield child
-                    _use_candidate_generator = default_candidate_generator
+                elif self.tag_name_re.match(token):
+                    # Just a tag name.
+                    tag_name = token
                 else:
-                    _use_candidate_generator = lambda tag: tag.descendants
-            else:
-                _use_candidate_generator = _candidate_generator
-
-            new_context = []
-            new_context_ids = set([])
-            for tag in current_context:
-                if self._select_debug:
-                    print "    Running candidate generator on %s %s" % (
-                        tag.name, repr(tag.attrs))
-                for candidate in _use_candidate_generator(tag):
-                    if not isinstance(candidate, Tag):
-                        continue
-                    if tag_name and candidate.name != tag_name:
-                        continue
-                    if checker is not None:
-                        try:
-                            result = checker(candidate)
-                        except StopIteration:
-                            # The checker has decided we should no longer
-                            # run the generator.
-                            break
-                    if checker is None or result:
+                    raise ValueError(
+                        'Unsupported or invalid CSS selector: "%s"' % token)
+                if recursive_candidate_generator:
+                    # This happens when the selector looks like  "> foo".
+                    #
+                    # The generator calls select() recursively on every
+                    # member of the current context, passing in a different
+                    # candidate generator and a different selector.
+                    #
+                    # In the case of "> foo", the candidate generator is
+                    # one that yields a tag's direct children (">"), and
+                    # the selector is "foo".
+                    next_token = tokens[index+1]
+                    def recursive_select(tag):
                         if self._select_debug:
-                            print "     SUCCESS %s %s" % (candidate.name, repr(candidate.attrs))
-                        if id(candidate) not in new_context_ids:
-                            # If a tag matches a selector more than once,
-                            # don't include it in the context more than once.
-                            new_context.append(candidate)
-                            new_context_ids.add(id(candidate))
-                    elif self._select_debug:
-                        print "     FAILURE %s %s" % (candidate.name, repr(candidate.attrs))
+                            print '    Calling select("%s") recursively on %s %s' % (next_token, tag.name, tag.attrs)
+                            print '-' * 40
+                        for i in tag.select(next_token, recursive_candidate_generator):
+                            if self._select_debug:
+                                print '(Recursive select picked up candidate %s %s)' % (i.name, i.attrs)
+                            yield i
+                        if self._select_debug:
+                            print '-' * 40
+                    _use_candidate_generator = recursive_select
+                elif _candidate_generator is None:
+                    # By default, a tag's candidates are all of its
+                    # children. If tag_name is defined, only yield tags
+                    # with that name.
+                    if self._select_debug:
+                        if tag_name:
+                            check = "[any]"
+                        else:
+                            check = tag_name
+                        print '   Default candidate generator, tag name="%s"' % check
+                    if self._select_debug:
+                        # This is redundant with later code, but it stops
+                        # a bunch of bogus tags from cluttering up the
+                        # debug log.
+                        def default_candidate_generator(tag):
+                            for child in tag.descendants:
+                                if not isinstance(child, Tag):
+                                    continue
+                                if tag_name and not child.name == tag_name:
+                                    continue
+                                yield child
+                        _use_candidate_generator = default_candidate_generator
+                    else:
+                        _use_candidate_generator = lambda tag: tag.descendants
+                else:
+                    _use_candidate_generator = _candidate_generator
+
+                count = 0
+                for tag in current_context:
+                    if self._select_debug:
+                        print "    Running candidate generator on %s %s" % (
+                            tag.name, repr(tag.attrs))
+                    for candidate in _use_candidate_generator(tag):
+                        if not isinstance(candidate, Tag):
+                            continue
+                        if tag_name and candidate.name != tag_name:
+                            continue
+                        if checker is not None:
+                            try:
+                                result = checker(candidate)
+                            except StopIteration:
+                                # The checker has decided we should no longer
+                                # run the generator.
+                                break
+                        if checker is None or result:
+                            if self._select_debug:
+                                print "     SUCCESS %s %s" % (candidate.name, repr(candidate.attrs))
+                            if id(candidate) not in new_context_ids:
+                                # If a tag matches a selector more than once,
+                                # don't include it in the context more than once.
+                                new_context.append(candidate)
+                                new_context_ids.add(id(candidate))
+                                if limit and len(new_context) >= limit:
+                                    break
+                        elif self._select_debug:
+                            print "     FAILURE %s %s" % (candidate.name, repr(candidate.attrs))
+
 
             current_context = new_context
 

lib/bs4/testing.py

@@ -1,5 +1,6 @@
 """Helper classes for tests."""
 
+import pickle
 import copy
 import functools
 import unittest
@@ -43,6 +44,16 @@ class SoupTest(unittest.TestCase):
 
         self.assertEqual(obj.decode(), self.document_for(compare_parsed_to))
 
+    def assertConnectedness(self, element):
+        """Ensure that next_element and previous_element are properly
+        set for all descendants of the given element.
+        """
+        earlier = None
+        for e in element.descendants:
+            if earlier:
+                self.assertEqual(e, earlier.next_element)
+                self.assertEqual(earlier, e.previous_element)
+            earlier = e
 
 class HTMLTreeBuilderSmokeTest(object):
 
@@ -54,6 +65,15 @@ class HTMLTreeBuilderSmokeTest(object):
     markup in these tests, there's not much room for interpretation.
     """
 
+    def test_pickle_and_unpickle_identity(self):
+        # Pickling a tree, then unpickling it, yields a tree identical
+        # to the original.
+        tree = self.soup("<a><b>foo</a>")
+        dumped = pickle.dumps(tree, 2)
+        loaded = pickle.loads(dumped)
+        self.assertEqual(loaded.__class__, BeautifulSoup)
+        self.assertEqual(loaded.decode(), tree.decode())
+
     def assertDoctypeHandled(self, doctype_fragment):
         """Assert that a given doctype string is handled correctly."""
         doctype_str, soup = self._document_with_doctype(doctype_fragment)
@@ -114,6 +134,11 @@ class HTMLTreeBuilderSmokeTest(object):
             soup.encode("utf-8").replace(b"\n", b""),
             markup.replace(b"\n", b""))
 
+    def test_processing_instruction(self):
+        markup = b"""<?PITarget PIContent?>"""
+        soup = self.soup(markup)
+        self.assertEqual(markup, soup.encode("utf8"))
+
     def test_deepcopy(self):
         """Make sure you can copy the tree builder.
 
@@ -155,6 +180,23 @@ class HTMLTreeBuilderSmokeTest(object):
     def test_nested_formatting_elements(self):
         self.assertSoupEquals("<em><em></em></em>")
 
+    def test_double_head(self):
+        html = '''<!DOCTYPE html>
+<html>
+<head>
+<title>Ordinary HEAD element test</title>
+</head>
+<script type="text/javascript">
+alert("Help!");
+</script>
+<body>
+Hello, world!
+</body>
+</html>
+'''
+        soup = self.soup(html)
+        self.assertEqual("text/javascript", soup.find('script')['type'])
+
     def test_comment(self):
         # Comments are represented as Comment objects.
         markup = "<p>foo<!--foobar-->baz</p>"
@@ -221,6 +263,14 @@ class HTMLTreeBuilderSmokeTest(object):
         soup = self.soup(markup)
         self.assertEqual(["css"], soup.div.div['class'])
 
+    def test_multivalued_attribute_on_html(self):
+        # html5lib uses a different API to set the attributes ot the
+        # <html> tag. This has caused problems with multivalued
+        # attributes.
+        markup = '<html class="a b"></html>'
+        soup = self.soup(markup)
+        self.assertEqual(["a", "b"], soup.html['class'])
+
     def test_angle_brackets_in_attribute_values_are_escaped(self):
         self.assertSoupEquals('<a b="<a>"></a>', '<a b="&lt;a&gt;"></a>')
 
@@ -253,6 +303,35 @@ class HTMLTreeBuilderSmokeTest(object):
         soup = self.soup("<html><h2>\nfoo</h2><p></p></html>")
         self.assertEqual("p", soup.h2.string.next_element.name)
         self.assertEqual("p", soup.p.name)
+        self.assertConnectedness(soup)
+
+    def test_head_tag_between_head_and_body(self):
+        "Prevent recurrence of a bug in the html5lib treebuilder."
+        content = """<html><head></head>
+  <link></link>
+  <body>foo</body>
+</html>
+"""
+        soup = self.soup(content)
+        self.assertNotEqual(None, soup.html.body)
+        self.assertConnectedness(soup)
+
+    def test_multiple_copies_of_a_tag(self):
+        "Prevent recurrence of a bug in the html5lib treebuilder."
+        content = """<!DOCTYPE html>
+<html>
+ <body>
+   <article id="a" >
+   <div><a href="1"></div>
+   <footer>
+     <a href="2"></a>
+   </footer>
+  </article>
+  </body>
+</html>
+"""
+        soup = self.soup(content)
+        self.assertConnectedness(soup.article)
 
     def test_basic_namespaces(self):
         """Parsers don't need to *understand* namespaces, but at the
@@ -463,6 +542,15 @@ class HTMLTreeBuilderSmokeTest(object):
 
 class XMLTreeBuilderSmokeTest(object):
 
+    def test_pickle_and_unpickle_identity(self):
+        # Pickling a tree, then unpickling it, yields a tree identical
+        # to the original.
+        tree = self.soup("<a><b>foo</a>")
+        dumped = pickle.dumps(tree, 2)
+        loaded = pickle.loads(dumped)
+        self.assertEqual(loaded.__class__, BeautifulSoup)
+        self.assertEqual(loaded.decode(), tree.decode())
+
     def test_docstring_generated(self):
         soup = self.soup("<root/>")
         self.assertEqual(
@@ -485,7 +573,7 @@ class XMLTreeBuilderSmokeTest(object):
   <script type="text/javascript">
   </script>
 """
-        soup = BeautifulSoup(doc, "xml")
+        soup = BeautifulSoup(doc, "lxml-xml")
         # lxml would have stripped this while parsing, but we can add
         # it later.
         soup.script.string = 'console.log("< < hey > > ");'

lib/html5lib/__init__.py

@@ -20,4 +20,6 @@ from .serializer import serialize
 
 __all__ = ["HTMLParser", "parse", "parseFragment", "getTreeBuilder",
            "getTreeWalker", "serialize"]
-__version__ = "0.999"
+
+# this has to be at the top level, see how setup.py parses this
+__version__ = "0.999999"

lib/html5lib/constants.py

@@ -1,292 +1,290 @@
 from __future__ import absolute_import, division, unicode_literals
 
 import string
-import gettext
-_ = gettext.gettext
 
 EOF = None
 
 E = {
     "null-character":
-        _("Null character in input stream, replaced with U+FFFD."),
+        "Null character in input stream, replaced with U+FFFD.",
     "invalid-codepoint":
-        _("Invalid codepoint in stream."),
+        "Invalid codepoint in stream.",
     "incorrectly-placed-solidus":
-        _("Solidus (/) incorrectly placed in tag."),
+        "Solidus (/) incorrectly placed in tag.",
     "incorrect-cr-newline-entity":
-        _("Incorrect CR newline entity, replaced with LF."),
+        "Incorrect CR newline entity, replaced with LF.",
     "illegal-windows-1252-entity":
-        _("Entity used with illegal number (windows-1252 reference)."),
+        "Entity used with illegal number (windows-1252 reference).",
     "cant-convert-numeric-entity":
-        _("Numeric entity couldn't be converted to character "
-          "(codepoint U+%(charAsInt)08x)."),
+        "Numeric entity couldn't be converted to character "
+        "(codepoint U+%(charAsInt)08x).",
     "illegal-codepoint-for-numeric-entity":
-        _("Numeric entity represents an illegal codepoint: "
-          "U+%(charAsInt)08x."),
+        "Numeric entity represents an illegal codepoint: "
+        "U+%(charAsInt)08x.",
     "numeric-entity-without-semicolon":
-        _("Numeric entity didn't end with ';'."),
+        "Numeric entity didn't end with ';'.",
     "expected-numeric-entity-but-got-eof":
-        _("Numeric entity expected. Got end of file instead."),
+        "Numeric entity expected. Got end of file instead.",
     "expected-numeric-entity":
-        _("Numeric entity expected but none found."),
+        "Numeric entity expected but none found.",
     "named-entity-without-semicolon":
-        _("Named entity didn't end with ';'."),
+        "Named entity didn't end with ';'.",
     "expected-named-entity":
-        _("Named entity expected. Got none."),
+        "Named entity expected. Got none.",
     "attributes-in-end-tag":
-        _("End tag contains unexpected attributes."),
+        "End tag contains unexpected attributes.",
     'self-closing-flag-on-end-tag':
-        _("End tag contains unexpected self-closing flag."),
+        "End tag contains unexpected self-closing flag.",
     "expected-tag-name-but-got-right-bracket":
-        _("Expected tag name. Got '>' instead."),
+        "Expected tag name. Got '>' instead.",
     "expected-tag-name-but-got-question-mark":
-        _("Expected tag name. Got '?' instead. (HTML doesn't "
-          "support processing instructions.)"),
+        "Expected tag name. Got '?' instead. (HTML doesn't "
+        "support processing instructions.)",
     "expected-tag-name":
-        _("Expected tag name. Got something else instead"),
+        "Expected tag name. Got something else instead",
     "expected-closing-tag-but-got-right-bracket":
-        _("Expected closing tag. Got '>' instead. Ignoring '</>'."),
+        "Expected closing tag. Got '>' instead. Ignoring '</>'.",
     "expected-closing-tag-but-got-eof":
-        _("Expected closing tag. Unexpected end of file."),
+        "Expected closing tag. Unexpected end of file.",
     "expected-closing-tag-but-got-char":
-        _("Expected closing tag. Unexpected character '%(data)s' found."),
+        "Expected closing tag. Unexpected character '%(data)s' found.",
     "eof-in-tag-name":
-        _("Unexpected end of file in the tag name."),
+        "Unexpected end of file in the tag name.",
     "expected-attribute-name-but-got-eof":
-        _("Unexpected end of file. Expected attribute name instead."),
+        "Unexpected end of file. Expected attribute name instead.",
     "eof-in-attribute-name":
-        _("Unexpected end of file in attribute name."),
+        "Unexpected end of file in attribute name.",
     "invalid-character-in-attribute-name":
-        _("Invalid character in attribute name"),
+        "Invalid character in attribute name",
     "duplicate-attribute":
-        _("Dropped duplicate attribute on tag."),
+        "Dropped duplicate attribute on tag.",
     "expected-end-of-tag-name-but-got-eof":
-        _("Unexpected end of file. Expected = or end of tag."),
+        "Unexpected end of file. Expected = or end of tag.",
     "expected-attribute-value-but-got-eof":
-        _("Unexpected end of file. Expected attribute value."),
+        "Unexpected end of file. Expected attribute value.",
     "expected-attribute-value-but-got-right-bracket":
-        _("Expected attribute value. Got '>' instead."),
+        "Expected attribute value. Got '>' instead.",
     'equals-in-unquoted-attribute-value':
-        _("Unexpected = in unquoted attribute"),
+        "Unexpected = in unquoted attribute",
     'unexpected-character-in-unquoted-attribute-value':
-        _("Unexpected character in unquoted attribute"),
+        "Unexpected character in unquoted attribute",
     "invalid-character-after-attribute-name":
-        _("Unexpected character after attribute name."),
+        "Unexpected character after attribute name.",
     "unexpected-character-after-attribute-value":
-        _("Unexpected character after attribute value."),
+        "Unexpected character after attribute value.",
     "eof-in-attribute-value-double-quote":
-        _("Unexpected end of file in attribute value (\")."),
+        "Unexpected end of file in attribute value (\").",
     "eof-in-attribute-value-single-quote":
-        _("Unexpected end of file in attribute value (')."),
+        "Unexpected end of file in attribute value (').",
     "eof-in-attribute-value-no-quotes":
-        _("Unexpected end of file in attribute value."),
+        "Unexpected end of file in attribute value.",
     "unexpected-EOF-after-solidus-in-tag":
-        _("Unexpected end of file in tag. Expected >"),
+        "Unexpected end of file in tag. Expected >",
     "unexpected-character-after-solidus-in-tag":
-        _("Unexpected character after / in tag. Expected >"),
+        "Unexpected character after / in tag. Expected >",
     "expected-dashes-or-doctype":
-        _("Expected '--' or 'DOCTYPE'. Not found."),
+        "Expected '--' or 'DOCTYPE'. Not found.",
     "unexpected-bang-after-double-dash-in-comment":
-        _("Unexpected ! after -- in comment"),
+        "Unexpected ! after -- in comment",
     "unexpected-space-after-double-dash-in-comment":
-        _("Unexpected space after -- in comment"),
+        "Unexpected space after -- in comment",
     "incorrect-comment":
-        _("Incorrect comment."),
+        "Incorrect comment.",
     "eof-in-comment":
-        _("Unexpected end of file in comment."),
+        "Unexpected end of file in comment.",
     "eof-in-comment-end-dash":
-        _("Unexpected end of file in comment (-)"),
+        "Unexpected end of file in comment (-)",
     "unexpected-dash-after-double-dash-in-comment":
-        _("Unexpected '-' after '--' found in comment."),
+        "Unexpected '-' after '--' found in comment.",
     "eof-in-comment-double-dash":
-        _("Unexpected end of file in comment (--)."),
+        "Unexpected end of file in comment (--).",
     "eof-in-comment-end-space-state":
-        _("Unexpected end of file in comment."),
+        "Unexpected end of file in comment.",
     "eof-in-comment-end-bang-state":
-        _("Unexpected end of file in comment."),
+        "Unexpected end of file in comment.",
     "unexpected-char-in-comment":
-        _("Unexpected character in comment found."),
+        "Unexpected character in comment found.",
     "need-space-after-doctype":
-        _("No space after literal string 'DOCTYPE'."),
+        "No space after literal string 'DOCTYPE'.",
     "expected-doctype-name-but-got-right-bracket":
-        _("Unexpected > character. Expected DOCTYPE name."),
+        "Unexpected > character. Expected DOCTYPE name.",
     "expected-doctype-name-but-got-eof":
-        _("Unexpected end of file. Expected DOCTYPE name."),
+        "Unexpected end of file. Expected DOCTYPE name.",
     "eof-in-doctype-name":
-        _("Unexpected end of file in DOCTYPE name."),
+        "Unexpected end of file in DOCTYPE name.",
     "eof-in-doctype":
-        _("Unexpected end of file in DOCTYPE."),
+        "Unexpected end of file in DOCTYPE.",
     "expected-space-or-right-bracket-in-doctype":
-        _("Expected space or '>'. Got '%(data)s'"),
+        "Expected space or '>'. Got '%(data)s'",
     "unexpected-end-of-doctype":
-        _("Unexpected end of DOCTYPE."),
+        "Unexpected end of DOCTYPE.",
     "unexpected-char-in-doctype":
-        _("Unexpected character in DOCTYPE."),
+        "Unexpected character in DOCTYPE.",
     "eof-in-innerhtml":
-        _("XXX innerHTML EOF"),
+        "XXX innerHTML EOF",
     "unexpected-doctype":
-        _("Unexpected DOCTYPE. Ignored."),
+        "Unexpected DOCTYPE. Ignored.",
     "non-html-root":
-        _("html needs to be the first start tag."),
+        "html needs to be the first start tag.",
     "expected-doctype-but-got-eof":
-        _("Unexpected End of file. Expected DOCTYPE."),
+        "Unexpected End of file. Expected DOCTYPE.",
     "unknown-doctype":
-        _("Erroneous DOCTYPE."),
+        "Erroneous DOCTYPE.",
     "expected-doctype-but-got-chars":
-        _("Unexpected non-space characters. Expected DOCTYPE."),
+        "Unexpected non-space characters. Expected DOCTYPE.",
     "expected-doctype-but-got-start-tag":
-        _("Unexpected start tag (%(name)s). Expected DOCTYPE."),
+        "Unexpected start tag (%(name)s). Expected DOCTYPE.",
     "expected-doctype-but-got-end-tag":
-        _("Unexpected end tag (%(name)s). Expected DOCTYPE."),
+        "Unexpected end tag (%(name)s). Expected DOCTYPE.",
     "end-tag-after-implied-root":
-        _("Unexpected end tag (%(name)s) after the (implied) root element."),
+        "Unexpected end tag (%(name)s) after the (implied) root element.",
     "expected-named-closing-tag-but-got-eof":
-        _("Unexpected end of file. Expected end tag (%(name)s)."),
+        "Unexpected end of file. Expected end tag (%(name)s).",
     "two-heads-are-not-better-than-one":
-        _("Unexpected start tag head in existing head. Ignored."),
+        "Unexpected start tag head in existing head. Ignored.",
     "unexpected-end-tag":
-        _("Unexpected end tag (%(name)s). Ignored."),
+        "Unexpected end tag (%(name)s). Ignored.",
     "unexpected-start-tag-out-of-my-head":
-        _("Unexpected start tag (%(name)s) that can be in head. Moved."),
+        "Unexpected start tag (%(name)s) that can be in head. Moved.",
     "unexpected-start-tag":
-        _("Unexpected start tag (%(name)s)."),
+        "Unexpected start tag (%(name)s).",
     "missing-end-tag":
-        _("Missing end tag (%(name)s)."),
+        "Missing end tag (%(name)s).",
     "missing-end-tags":
-        _("Missing end tags (%(name)s)."),
+        "Missing end tags (%(name)s).",
     "unexpected-start-tag-implies-end-tag":
-        _("Unexpected start tag (%(startName)s) "
-          "implies end tag (%(endName)s)."),
+        "Unexpected start tag (%(startName)s) "
+        "implies end tag (%(endName)s).",
     "unexpected-start-tag-treated-as":
-        _("Unexpected start tag (%(originalName)s). Treated as %(newName)s."),
+        "Unexpected start tag (%(originalName)s). Treated as %(newName)s.",
     "deprecated-tag":
-        _("Unexpected start tag %(name)s. Don't use it!"),
+        "Unexpected start tag %(name)s. Don't use it!",
     "unexpected-start-tag-ignored":
-        _("Unexpected start tag %(name)s. Ignored."),
+        "Unexpected start tag %(name)s. Ignored.",
     "expected-one-end-tag-but-got-another":
-        _("Unexpected end tag (%(gotName)s). "
-          "Missing end tag (%(expectedName)s)."),
+        "Unexpected end tag (%(gotName)s). "
+        "Missing end tag (%(expectedName)s).",
     "end-tag-too-early":
-        _("End tag (%(name)s) seen too early. Expected other end tag."),
+        "End tag (%(name)s) seen too early. Expected other end tag.",
     "end-tag-too-early-named":
-        _("Unexpected end tag (%(gotName)s). Expected end tag (%(expectedName)s)."),
+        "Unexpected end tag (%(gotName)s). Expected end tag (%(expectedName)s).",
     "end-tag-too-early-ignored":
-        _("End tag (%(name)s) seen too early. Ignored."),
+        "End tag (%(name)s) seen too early. Ignored.",
     "adoption-agency-1.1":
-        _("End tag (%(name)s) violates step 1, "
-          "paragraph 1 of the adoption agency algorithm."),
+        "End tag (%(name)s) violates step 1, "
+        "paragraph 1 of the adoption agency algorithm.",
     "adoption-agency-1.2":
-        _("End tag (%(name)s) violates step 1, "
-          "paragraph 2 of the adoption agency algorithm."),
+        "End tag (%(name)s) violates step 1, "
+        "paragraph 2 of the adoption agency algorithm.",
     "adoption-agency-1.3":
-        _("End tag (%(name)s) violates step 1, "
-          "paragraph 3 of the adoption agency algorithm."),
+        "End tag (%(name)s) violates step 1, "
+        "paragraph 3 of the adoption agency algorithm.",
     "adoption-agency-4.4":
-        _("End tag (%(name)s) violates step 4, "
-          "paragraph 4 of the adoption agency algorithm."),
+        "End tag (%(name)s) violates step 4, "
+        "paragraph 4 of the adoption agency algorithm.",
     "unexpected-end-tag-treated-as":
-        _("Unexpected end tag (%(originalName)s). Treated as %(newName)s."),
+        "Unexpected end tag (%(originalName)s). Treated as %(newName)s.",
     "no-end-tag":
-        _("This element (%(name)s) has no end tag."),
+        "This element (%(name)s) has no end tag.",
     "unexpected-implied-end-tag-in-table":
-        _("Unexpected implied end tag (%(name)s) in the table phase."),
+        "Unexpected implied end tag (%(name)s) in the table phase.",
     "unexpected-implied-end-tag-in-table-body":
-        _("Unexpected implied end tag (%(name)s) in the table body phase."),
+        "Unexpected implied end tag (%(name)s) in the table body phase.",
     "unexpected-char-implies-table-voodoo":
-        _("Unexpected non-space characters in "
-          "table context caused voodoo mode."),
+        "Unexpected non-space characters in "
+        "table context caused voodoo mode.",
     "unexpected-hidden-input-in-table":
-        _("Unexpected input with type hidden in table context."),
+        "Unexpected input with type hidden in table context.",
     "unexpected-form-in-table":
-        _("Unexpected form in table context."),
+        "Unexpected form in table context.",
     "unexpected-start-tag-implies-table-voodoo":
-        _("Unexpected start tag (%(name)s) in "
-          "table context caused voodoo mode."),
+        "Unexpected start tag (%(name)s) in "
+        "table context caused voodoo mode.",
     "unexpected-end-tag-implies-table-voodoo":
-        _("Unexpected end tag (%(name)s) in "
-          "table context caused voodoo mode."),
+        "Unexpected end tag (%(name)s) in "
+        "table context caused voodoo mode.",
     "unexpected-cell-in-table-body":
-        _("Unexpected table cell start tag (%(name)s) "
-          "in the table body phase."),
+        "Unexpected table cell start tag (%(name)s) "
+        "in the table body phase.",
     "unexpected-cell-end-tag":
-        _("Got table cell end tag (%(name)s) "
-          "while required end tags are missing."),
+        "Got table cell end tag (%(name)s) "
+        "while required end tags are missing.",
     "unexpected-end-tag-in-table-body":
-        _("Unexpected end tag (%(name)s) in the table body phase. Ignored."),
+        "Unexpected end tag (%(name)s) in the table body phase. Ignored.",
     "unexpected-implied-end-tag-in-table-row":
-        _("Unexpected implied end tag (%(name)s) in the table row phase."),
+        "Unexpected implied end tag (%(name)s) in the table row phase.",
     "unexpected-end-tag-in-table-row":
-        _("Unexpected end tag (%(name)s) in the table row phase. Ignored."),
+        "Unexpected end tag (%(name)s) in the table row phase. Ignored.",
     "unexpected-select-in-select":
-        _("Unexpected select start tag in the select phase "
-          "treated as select end tag."),
+        "Unexpected select start tag in the select phase "
+        "treated as select end tag.",
     "unexpected-input-in-select":
-        _("Unexpected input start tag in the select phase."),
+        "Unexpected input start tag in the select phase.",
     "unexpected-start-tag-in-select":
-        _("Unexpected start tag token (%(name)s in the select phase. "
-          "Ignored."),
+        "Unexpected start tag token (%(name)s in the select phase. "
+        "Ignored.",
     "unexpected-end-tag-in-select":
-        _("Unexpected end tag (%(name)s) in the select phase. Ignored."),
+        "Unexpected end tag (%(name)s) in the select phase. Ignored.",
     "unexpected-table-element-start-tag-in-select-in-table":
-        _("Unexpected table element start tag (%(name)s) in the select in table phase."),
+        "Unexpected table element start tag (%(name)s) in the select in table phase.",
     "unexpected-table-element-end-tag-in-select-in-table":
-        _("Unexpected table element end tag (%(name)s) in the select in table phase."),
+        "Unexpected table element end tag (%(name)s) in the select in table phase.",
     "unexpected-char-after-body":
-        _("Unexpected non-space characters in the after body phase."),
+        "Unexpected non-space characters in the after body phase.",
     "unexpected-start-tag-after-body":
-        _("Unexpected start tag token (%(name)s)"
-          " in the after body phase."),
+        "Unexpected start tag token (%(name)s)"
+        " in the after body phase.",
     "unexpected-end-tag-after-body":
-        _("Unexpected end tag token (%(name)s)"
-          " in the after body phase."),
+        "Unexpected end tag token (%(name)s)"
+        " in the after body phase.",
     "unexpected-char-in-frameset":
-        _("Unexpected characters in the frameset phase. Characters ignored."),
+        "Unexpected characters in the frameset phase. Characters ignored.",
     "unexpected-start-tag-in-frameset":
-        _("Unexpected start tag token (%(name)s)"
-          " in the frameset phase. Ignored."),
+        "Unexpected start tag token (%(name)s)"
+        " in the frameset phase. Ignored.",
     "unexpected-frameset-in-frameset-innerhtml":
-        _("Unexpected end tag token (frameset) "
-          "in the frameset phase (innerHTML)."),
+        "Unexpected end tag token (frameset) "
+        "in the frameset phase (innerHTML).",
     "unexpected-end-tag-in-frameset":
-        _("Unexpected end tag token (%(name)s)"
-          " in the frameset phase. Ignored."),
+        "Unexpected end tag token (%(name)s)"
+        " in the frameset phase. Ignored.",
     "unexpected-char-after-frameset":
-        _("Unexpected non-space characters in the "
-          "after frameset phase. Ignored."),
+        "Unexpected non-space characters in the "
+        "after frameset phase. Ignored.",
     "unexpected-start-tag-after-frameset":
-        _("Unexpected start tag (%(name)s)"
-          " in the after frameset phase. Ignored."),
+        "Unexpected start tag (%(name)s)"
+        " in the after frameset phase. Ignored.",
     "unexpected-end-tag-after-frameset":
-        _("Unexpected end tag (%(name)s)"
-          " in the after frameset phase. Ignored."),
+        "Unexpected end tag (%(name)s)"
+        " in the after frameset phase. Ignored.",
     "unexpected-end-tag-after-body-innerhtml":
-        _("Unexpected end tag after body(innerHtml)"),
+        "Unexpected end tag after body(innerHtml)",
     "expected-eof-but-got-char":
-        _("Unexpected non-space characters. Expected end of file."),
+        "Unexpected non-space characters. Expected end of file.",
     "expected-eof-but-got-start-tag":
-        _("Unexpected start tag (%(name)s)"
-          ". Expected end of file."),
+        "Unexpected start tag (%(name)s)"
+        ". Expected end of file.",
     "expected-eof-but-got-end-tag":
-        _("Unexpected end tag (%(name)s)"
-          ". Expected end of file."),
+        "Unexpected end tag (%(name)s)"
+        ". Expected end of file.",
     "eof-in-table":
-        _("Unexpected end of file. Expected table content."),
+        "Unexpected end of file. Expected table content.",
     "eof-in-select":
-        _("Unexpected end of file. Expected select content."),
+        "Unexpected end of file. Expected select content.",
     "eof-in-frameset":
-        _("Unexpected end of file. Expected frameset content."),
+        "Unexpected end of file. Expected frameset content.",
     "eof-in-script-in-script":
-        _("Unexpected end of file. Expected script content."),
+        "Unexpected end of file. Expected script content.",
     "eof-in-foreign-lands":
-        _("Unexpected end of file. Expected foreign content"),
+        "Unexpected end of file. Expected foreign content",
     "non-void-element-with-trailing-solidus":
-        _("Trailing solidus not allowed on element %(name)s"),
+        "Trailing solidus not allowed on element %(name)s",
     "unexpected-html-element-in-foreign-content":
-        _("Element %(name)s not allowed in a non-html context"),
+        "Element %(name)s not allowed in a non-html context",
     "unexpected-end-tag-before-html":
-        _("Unexpected end tag (%(name)s) before html."),
+        "Unexpected end tag (%(name)s) before html.",
     "XXX-undefined-error":
-        _("Undefined error (this sucks and should be fixed)"),
+        "Undefined error (this sucks and should be fixed)",
 }
 
 namespaces = {
@@ -298,7 +296,7 @@ namespaces = {
     "xmlns": "http://www.w3.org/2000/xmlns/"
 }
 
-scopingElements = frozenset((
+scopingElements = frozenset([
     (namespaces["html"], "applet"),
     (namespaces["html"], "caption"),
     (namespaces["html"], "html"),
@@ -316,9 +314,9 @@ scopingElements = frozenset((
     (namespaces["svg"], "foreignObject"),
     (namespaces["svg"], "desc"),
     (namespaces["svg"], "title"),
-))
+])
 
-formattingElements = frozenset((
+formattingElements = frozenset([
     (namespaces["html"], "a"),
     (namespaces["html"], "b"),
     (namespaces["html"], "big"),
@@ -333,9 +331,9 @@ formattingElements = frozenset((
     (namespaces["html"], "strong"),
     (namespaces["html"], "tt"),
     (namespaces["html"], "u")
-))
+])
 
-specialElements = frozenset((
+specialElements = frozenset([
     (namespaces["html"], "address"),
     (namespaces["html"], "applet"),
     (namespaces["html"], "area"),
@@ -416,22 +414,22 @@ specialElements = frozenset((
     (namespaces["html"], "wbr"),
     (namespaces["html"], "xmp"),
     (namespaces["svg"], "foreignObject")
-))
+])
 
-htmlIntegrationPointElements = frozenset((
+htmlIntegrationPointElements = frozenset([
     (namespaces["mathml"], "annotaion-xml"),
     (namespaces["svg"], "foreignObject"),
     (namespaces["svg"], "desc"),
     (namespaces["svg"], "title")
-))
+])
 
-mathmlTextIntegrationPointElements = frozenset((
+mathmlTextIntegrationPointElements = frozenset([
     (namespaces["mathml"], "mi"),
     (namespaces["mathml"], "mo"),
     (namespaces["mathml"], "mn"),
     (namespaces["mathml"], "ms"),
     (namespaces["mathml"], "mtext")
-))
+])
 
 adjustForeignAttributes = {
     "xlink:actuate": ("xlink", "actuate", namespaces["xlink"]),
@@ -451,21 +449,21 @@ adjustForeignAttributes = {
 unadjustForeignAttributes = dict([((ns, local), qname) for qname, (prefix, local, ns) in
                                   adjustForeignAttributes.items()])
 
-spaceCharacters = frozenset((
+spaceCharacters = frozenset([
     "\t",
     "\n",
     "\u000C",
     " ",
     "\r"
-))
+])
 
-tableInsertModeElements = frozenset((
+tableInsertModeElements = frozenset([
     "table",
     "tbody",
     "tfoot",
     "thead",
     "tr"
-))
+])
 
 asciiLowercase = frozenset(string.ascii_lowercase)
 asciiUppercase = frozenset(string.ascii_uppercase)
@@ -486,7 +484,7 @@ headingElements = (
     "h6"
 )
 
-voidElements = frozenset((
+voidElements = frozenset([
     "base",
     "command",
     "event-source",
@@ -502,11 +500,11 @@ voidElements = frozenset((
     "input",
     "source",
     "track"
-))
+])
 
-cdataElements = frozenset(('title', 'textarea'))
+cdataElements = frozenset(['title', 'textarea'])
 
-rcdataElements = frozenset((
+rcdataElements = frozenset([
     'style',
     'script',
     'xmp',
@@ -514,27 +512,27 @@ rcdataElements = frozenset((
     'noembed',
     'noframes',
     'noscript'
-))
+])
 
 booleanAttributes = {
-    "": frozenset(("irrelevant",)),
-    "style": frozenset(("scoped",)),
-    "img": frozenset(("ismap",)),
-    "audio": frozenset(("autoplay", "controls")),
-    "video": frozenset(("autoplay", "controls")),
-    "script": frozenset(("defer", "async")),
-    "details": frozenset(("open",)),
-    "datagrid": frozenset(("multiple", "disabled")),
-    "command": frozenset(("hidden", "disabled", "checked", "default")),
-    "hr": frozenset(("noshade")),
-    "menu": frozenset(("autosubmit",)),
-    "fieldset": frozenset(("disabled", "readonly")),
-    "option": frozenset(("disabled", "readonly", "selected")),
-    "optgroup": frozenset(("disabled", "readonly")),
-    "button": frozenset(("disabled", "autofocus")),
-    "input": frozenset(("disabled", "readonly", "required", "autofocus", "checked", "ismap")),
-    "select": frozenset(("disabled", "readonly", "autofocus", "multiple")),
-    "output": frozenset(("disabled", "readonly")),
+    "": frozenset(["irrelevant"]),
+    "style": frozenset(["scoped"]),
+    "img": frozenset(["ismap"]),
+    "audio": frozenset(["autoplay", "controls"]),
+    "video": frozenset(["autoplay", "controls"]),
+    "script": frozenset(["defer", "async"]),
+    "details": frozenset(["open"]),
+    "datagrid": frozenset(["multiple", "disabled"]),
+    "command": frozenset(["hidden", "disabled", "checked", "default"]),
+    "hr": frozenset(["noshade"]),
+    "menu": frozenset(["autosubmit"]),
+    "fieldset": frozenset(["disabled", "readonly"]),
+    "option": frozenset(["disabled", "readonly", "selected"]),
+    "optgroup": frozenset(["disabled", "readonly"]),
+    "button": frozenset(["disabled", "autofocus"]),
+    "input": frozenset(["disabled", "readonly", "required", "autofocus", "checked", "ismap"]),
+    "select": frozenset(["disabled", "readonly", "autofocus", "multiple"]),
+    "output": frozenset(["disabled", "readonly"]),
 }
 
 # entitiesWindows1252 has to be _ordered_ and needs to have an index. It
@@ -574,7 +572,7 @@ entitiesWindows1252 = (
     376     # 0x9F  0x0178  LATIN CAPITAL LETTER Y WITH DIAERESIS
 )
 
-xmlEntities = frozenset(('lt;', 'gt;', 'amp;', 'apos;', 'quot;'))
+xmlEntities = frozenset(['lt;', 'gt;', 'amp;', 'apos;', 'quot;'])
 
 entities = {
     "AElig": "\xc6",
@@ -3088,8 +3086,8 @@ tokenTypes = {
     "ParseError": 7
 }
 
-tagTokenTypes = frozenset((tokenTypes["StartTag"], tokenTypes["EndTag"],
-                           tokenTypes["EmptyTag"]))
+tagTokenTypes = frozenset([tokenTypes["StartTag"], tokenTypes["EndTag"],
+                           tokenTypes["EmptyTag"]])
 
 
 prefixes = dict([(v, k) for k, v in namespaces.items()])

lib/html5lib/filters/lint.py

@@ -1,8 +1,5 @@
 from __future__ import absolute_import, division, unicode_literals
 
-from gettext import gettext
-_ = gettext
-
 from . import _base
 from ..constants import cdataElements, rcdataElements, voidElements
 
@@ -23,24 +20,24 @@ class Filter(_base.Filter):
             if type in ("StartTag", "EmptyTag"):
                 name = token["name"]
                 if contentModelFlag != "PCDATA":
-                    raise LintError(_("StartTag not in PCDATA content model flag: %(tag)s") % {"tag": name})
+                    raise LintError("StartTag not in PCDATA content model flag: %(tag)s" % {"tag": name})
                 if not isinstance(name, str):
-                    raise LintError(_("Tag name is not a string: %(tag)r") % {"tag": name})
+                    raise LintError("Tag name is not a string: %(tag)r" % {"tag": name})
                 if not name:
-                    raise LintError(_("Empty tag name"))
+                    raise LintError("Empty tag name")
                 if type == "StartTag" and name in voidElements:
-                    raise LintError(_("Void element reported as StartTag token: %(tag)s") % {"tag": name})
+                    raise LintError("Void element reported as StartTag token: %(tag)s" % {"tag": name})
                 elif type == "EmptyTag" and name not in voidElements:
-                    raise LintError(_("Non-void element reported as EmptyTag token: %(tag)s") % {"tag": token["name"]})
+                    raise LintError("Non-void element reported as EmptyTag token: %(tag)s" % {"tag": token["name"]})
                 if type == "StartTag":
                     open_elements.append(name)
                 for name, value in token["data"]:
                     if not isinstance(name, str):
-                        raise LintError(_("Attribute name is not a string: %(name)r") % {"name": name})
+                        raise LintError("Attribute name is not a string: %(name)r" % {"name": name})
                     if not name:
-                        raise LintError(_("Empty attribute name"))
+                        raise LintError("Empty attribute name")
                     if not isinstance(value, str):
-                        raise LintError(_("Attribute value is not a string: %(value)r") % {"value": value})
+                        raise LintError("Attribute value is not a string: %(value)r" % {"value": value})
                 if name in cdataElements:
                     contentModelFlag = "CDATA"
                 elif name in rcdataElements:
@@ -51,43 +48,43 @@ class Filter(_base.Filter):
             elif type == "EndTag":
                 name = token["name"]
                 if not isinstance(name, str):
-                    raise LintError(_("Tag name is not a string: %(tag)r") % {"tag": name})
+                    raise LintError("Tag name is not a string: %(tag)r" % {"tag": name})
                 if not name:
-                    raise LintError(_("Empty tag name"))
+                    raise LintError("Empty tag name")
                 if name in voidElements:
-                    raise LintError(_("Void element reported as EndTag token: %(tag)s") % {"tag": name})
+                    raise LintError("Void element reported as EndTag token: %(tag)s" % {"tag": name})
                 start_name = open_elements.pop()
                 if start_name != name:
-                    raise LintError(_("EndTag (%(end)s) does not match StartTag (%(start)s)") % {"end": name, "start": start_name})
+                    raise LintError("EndTag (%(end)s) does not match StartTag (%(start)s)" % {"end": name, "start": start_name})
                 contentModelFlag = "PCDATA"
 
             elif type == "Comment":
                 if contentModelFlag != "PCDATA":
-                    raise LintError(_("Comment not in PCDATA content model flag"))
+                    raise LintError("Comment not in PCDATA content model flag")
 
             elif type in ("Characters", "SpaceCharacters"):
                 data = token["data"]
                 if not isinstance(data, str):
-                    raise LintError(_("Attribute name is not a string: %(name)r") % {"name": data})
+                    raise LintError("Attribute name is not a string: %(name)r" % {"name": data})
                 if not data:
-                    raise LintError(_("%(type)s token with empty data") % {"type": type})
+                    raise LintError("%(type)s token with empty data" % {"type": type})
                 if type == "SpaceCharacters":
                     data = data.strip(spaceCharacters)
                     if data:
-                        raise LintError(_("Non-space character(s) found in SpaceCharacters token: %(token)r") % {"token": data})
+                        raise LintError("Non-space character(s) found in SpaceCharacters token: %(token)r" % {"token": data})
 
             elif type == "Doctype":
                 name = token["name"]
                 if contentModelFlag != "PCDATA":
-                    raise LintError(_("Doctype not in PCDATA content model flag: %(name)s") % {"name": name})
+                    raise LintError("Doctype not in PCDATA content model flag: %(name)s" % {"name": name})
                 if not isinstance(name, str):
-                    raise LintError(_("Tag name is not a string: %(tag)r") % {"tag": name})
+                    raise LintError("Tag name is not a string: %(tag)r" % {"tag": name})
                 # XXX: what to do with token["data"] ?
 
             elif type in ("ParseError", "SerializeError"):
                 pass
 
             else:
-                raise LintError(_("Unknown token type: %(type)s") % {"type": type})
+                raise LintError("Unknown token type: %(type)s" % {"type": type})
 
             yield token

lib/html5lib/html5parser.py

@@ -18,6 +18,7 @@ from .constants import cdataElements, rcdataElements
 from .constants import tokenTypes, ReparseException, namespaces
 from .constants import htmlIntegrationPointElements, mathmlTextIntegrationPointElements
 from .constants import adjustForeignAttributes as adjustForeignAttributesMap
+from .constants import E
 
 
 def parse(doc, treebuilder="etree", encoding=None,
@@ -129,6 +130,17 @@ class HTMLParser(object):
 
         self.framesetOK = True
 
+    @property
+    def documentEncoding(self):
+        """The name of the character encoding
+        that was used to decode the input stream,
+        or :obj:`None` if that is not determined yet.
+
+        """
+        if not hasattr(self, 'tokenizer'):
+            return None
+        return self.tokenizer.stream.charEncoding[0]
+
     def isHTMLIntegrationPoint(self, element):
         if (element.name == "annotation-xml" and
                 element.namespace == namespaces["mathml"]):
@@ -245,7 +257,7 @@ class HTMLParser(object):
         # XXX The idea is to make errorcode mandatory.
         self.errors.append((self.tokenizer.stream.position(), errorcode, datavars))
         if self.strict:
-            raise ParseError
+            raise ParseError(E[errorcode] % datavars)
 
     def normalizeToken(self, token):
         """ HTML5 specific normalizations to the token stream """
@@ -868,7 +880,7 @@ def getPhases(debug):
             self.startTagHandler = utils.MethodDispatcher([
                 ("html", self.startTagHtml),
                 (("base", "basefont", "bgsound", "command", "link", "meta",
-                  "noframes", "script", "style", "title"),
+                  "script", "style", "title"),
                  self.startTagProcessInHead),
                 ("body", self.startTagBody),
                 ("frameset", self.startTagFrameset),
@@ -1205,8 +1217,7 @@ def getPhases(debug):
             attributes["name"] = "isindex"
             self.processStartTag(impliedTagToken("input", "StartTag",
                                                  attributes=attributes,
-                                                 selfClosing=
-                                                 token["selfClosing"]))
+                                                 selfClosing=token["selfClosing"]))
             self.processEndTag(impliedTagToken("label"))
             self.processStartTag(impliedTagToken("hr", "StartTag"))
             self.processEndTag(impliedTagToken("form"))

lib/html5lib/inputstream.py

@@ -28,7 +28,18 @@ asciiLettersBytes = frozenset([item.encode("ascii") for item in asciiLetters])
 asciiUppercaseBytes = frozenset([item.encode("ascii") for item in asciiUppercase])
 spacesAngleBrackets = spaceCharactersBytes | frozenset([b">", b"<"])
 
-invalid_unicode_re = re.compile("[\u0001-\u0008\u000B\u000E-\u001F\u007F-\u009F\uD800-\uDFFF\uFDD0-\uFDEF\uFFFE\uFFFF\U0001FFFE\U0001FFFF\U0002FFFE\U0002FFFF\U0003FFFE\U0003FFFF\U0004FFFE\U0004FFFF\U0005FFFE\U0005FFFF\U0006FFFE\U0006FFFF\U0007FFFE\U0007FFFF\U0008FFFE\U0008FFFF\U0009FFFE\U0009FFFF\U000AFFFE\U000AFFFF\U000BFFFE\U000BFFFF\U000CFFFE\U000CFFFF\U000DFFFE\U000DFFFF\U000EFFFE\U000EFFFF\U000FFFFE\U000FFFFF\U0010FFFE\U0010FFFF]")
+
+invalid_unicode_no_surrogate = "[\u0001-\u0008\u000B\u000E-\u001F\u007F-\u009F\uFDD0-\uFDEF\uFFFE\uFFFF\U0001FFFE\U0001FFFF\U0002FFFE\U0002FFFF\U0003FFFE\U0003FFFF\U0004FFFE\U0004FFFF\U0005FFFE\U0005FFFF\U0006FFFE\U0006FFFF\U0007FFFE\U0007FFFF\U0008FFFE\U0008FFFF\U0009FFFE\U0009FFFF\U000AFFFE\U000AFFFF\U000BFFFE\U000BFFFF\U000CFFFE\U000CFFFF\U000DFFFE\U000DFFFF\U000EFFFE\U000EFFFF\U000FFFFE\U000FFFFF\U0010FFFE\U0010FFFF]"
+
+if utils.supports_lone_surrogates:
+    # Use one extra step of indirection and create surrogates with
+    # unichr. Not using this indirection would introduce an illegal
+    # unicode literal on platforms not supporting such lone
+    # surrogates.
+    invalid_unicode_re = re.compile(invalid_unicode_no_surrogate +
+                                    eval('"\\uD800-\\uDFFF"'))
+else:
+    invalid_unicode_re = re.compile(invalid_unicode_no_surrogate)
 
 non_bmp_invalid_codepoints = set([0x1FFFE, 0x1FFFF, 0x2FFFE, 0x2FFFF, 0x3FFFE,
                                   0x3FFFF, 0x4FFFE, 0x4FFFF, 0x5FFFE, 0x5FFFF,
@@ -164,13 +175,18 @@ class HTMLUnicodeInputStream(object):
 
         """
 
-        # Craziness
-        if len("\U0010FFFF") == 1:
+        if not utils.supports_lone_surrogates:
+            # Such platforms will have already checked for such
+            # surrogate errors, so no need to do this checking.
+            self.reportCharacterErrors = None
+            self.replaceCharactersRegexp = None
+        elif len("\U0010FFFF") == 1:
             self.reportCharacterErrors = self.characterErrorsUCS4
-            self.replaceCharactersRegexp = re.compile("[\uD800-\uDFFF]")
+            self.replaceCharactersRegexp = re.compile(eval('"[\\uD800-\\uDFFF]"'))
         else:
             self.reportCharacterErrors = self.characterErrorsUCS2
-            self.replaceCharactersRegexp = re.compile("([\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF])")
+            self.replaceCharactersRegexp = re.compile(
+                eval('"([\\uD800-\\uDBFF](?![\\uDC00-\\uDFFF])|(?<![\\uD800-\\uDBFF])[\\uDC00-\\uDFFF])"'))
 
         # List of where new lines occur
         self.newLines = [0]
@@ -265,11 +281,12 @@ class HTMLUnicodeInputStream(object):
                 self._bufferedCharacter = data[-1]
                 data = data[:-1]
 
-        self.reportCharacterErrors(data)
+        if self.reportCharacterErrors:
+            self.reportCharacterErrors(data)
 
-        # Replace invalid characters
-        # Note U+0000 is dealt with in the tokenizer
-        data = self.replaceCharactersRegexp.sub("\ufffd", data)
+            # Replace invalid characters
+            # Note U+0000 is dealt with in the tokenizer
+            data = self.replaceCharactersRegexp.sub("\ufffd", data)
 
         data = data.replace("\r\n", "\n")
         data = data.replace("\r", "\n")

lib/html5lib/sanitizer.py

@@ -2,11 +2,26 @@ from __future__ import absolute_import, division, unicode_literals
 
 import re
 from xml.sax.saxutils import escape, unescape
+from six.moves import urllib_parse as urlparse
 
 from .tokenizer import HTMLTokenizer
 from .constants import tokenTypes
 
 
+content_type_rgx = re.compile(r'''
+                               ^
+                               # Match a content type <application>/<type>
+                               (?P<content_type>[-a-zA-Z0-9.]+/[-a-zA-Z0-9.]+)
+                               # Match any character set and encoding
+                               (?:(?:;charset=(?:[-a-zA-Z0-9]+)(?:;(?:base64))?)
+                                 |(?:;(?:base64))?(?:;charset=(?:[-a-zA-Z0-9]+))?)
+                               # Assume the rest is data
+                               ,.*
+                               $
+                               ''',
+                              re.VERBOSE)
+
+
 class HTMLSanitizerMixin(object):
     """ sanitization of XHTML+MathML+SVG and of inline style attributes."""
 
@@ -100,8 +115,8 @@ class HTMLSanitizerMixin(object):
                       'xml:base', 'xml:lang', 'xml:space', 'xmlns', 'xmlns:xlink', 'y',
                       'y1', 'y2', 'zoomAndPan']
 
-    attr_val_is_uri = ['href', 'src', 'cite', 'action', 'longdesc', 'poster',
-                       'xlink:href', 'xml:base']
+    attr_val_is_uri = ['href', 'src', 'cite', 'action', 'longdesc', 'poster', 'background', 'datasrc',
+                       'dynsrc', 'lowsrc', 'ping', 'poster', 'xlink:href', 'xml:base']
 
     svg_attr_val_allows_ref = ['clip-path', 'color-profile', 'cursor', 'fill',
                                'filter', 'marker', 'marker-start', 'marker-mid', 'marker-end',
@@ -138,7 +153,9 @@ class HTMLSanitizerMixin(object):
     acceptable_protocols = ['ed2k', 'ftp', 'http', 'https', 'irc',
                             'mailto', 'news', 'gopher', 'nntp', 'telnet', 'webcal',
                             'xmpp', 'callto', 'feed', 'urn', 'aim', 'rsync', 'tag',
-                            'ssh', 'sftp', 'rtsp', 'afs']
+                            'ssh', 'sftp', 'rtsp', 'afs', 'data']
+
+    acceptable_content_types = ['image/png', 'image/jpeg', 'image/gif', 'image/webp', 'image/bmp', 'text/plain']
 
     # subclasses may define their own versions of these constants
     allowed_elements = acceptable_elements + mathml_elements + svg_elements
@@ -147,6 +164,7 @@ class HTMLSanitizerMixin(object):
     allowed_css_keywords = acceptable_css_keywords
     allowed_svg_properties = acceptable_svg_properties
     allowed_protocols = acceptable_protocols
+    allowed_content_types = acceptable_content_types
 
     # Sanitize the +html+, escaping all elements not in ALLOWED_ELEMENTS, and
     # stripping out all # attributes not in ALLOWED_ATTRIBUTES. Style
@@ -189,10 +207,17 @@ class HTMLSanitizerMixin(object):
                                        unescape(attrs[attr])).lower()
                 # remove replacement characters from unescaped characters
                 val_unescaped = val_unescaped.replace("\ufffd", "")
-                if (re.match("^[a-z0-9][-+.a-z0-9]*:", val_unescaped) and
-                    (val_unescaped.split(':')[0] not in
-                     self.allowed_protocols)):
-                    del attrs[attr]
+                uri = urlparse.urlparse(val_unescaped)
+                if uri and uri.scheme:
+                    if uri.scheme not in self.allowed_protocols:
+                        del attrs[attr]
+                    if uri.scheme == 'data':
+                        m = content_type_rgx.match(uri.path)
+                        if not m:
+                            del attrs[attr]
+                        elif m.group('content_type') not in self.allowed_content_types:
+                            del attrs[attr]
+
             for attr in self.svg_attr_val_allows_ref:
                 if attr in attrs:
                     attrs[attr] = re.sub(r'url\s*\(\s*[^#\s][^)]+?\)',
@@ -245,7 +270,7 @@ class HTMLSanitizerMixin(object):
             elif prop.split('-')[0].lower() in ['background', 'border', 'margin',
                                                 'padding']:
                 for keyword in value.split():
-                    if not keyword in self.acceptable_css_keywords and \
+                    if keyword not in self.acceptable_css_keywords and \
                             not re.match("^(#[0-9a-f]+|rgb\(\d+%?,\d*%?,?\d*%?\)?|\d{0,2}\.?\d{0,2}(cm|em|ex|in|mm|pc|pt|px|%|,|\))?)$", keyword):
                         break
                 else:

lib/html5lib/serializer/htmlserializer.py

@@ -1,9 +1,6 @@
 from __future__ import absolute_import, division, unicode_literals
 from six import text_type
 
-import gettext
-_ = gettext.gettext
-
 try:
     from functools import reduce
 except ImportError:
@@ -35,7 +32,7 @@ else:
                 v = utils.surrogatePairToCodepoint(v)
             else:
                 v = ord(v)
-            if not v in encode_entity_map or k.islower():
+            if v not in encode_entity_map or k.islower():
                 # prefer < over < and similarly for &, >, etc.
                 encode_entity_map[v] = k
 
@@ -208,7 +205,7 @@ class HTMLSerializer(object):
                 if token["systemId"]:
                     if token["systemId"].find('"') >= 0:
                         if token["systemId"].find("'") >= 0:
-                            self.serializeError(_("System identifer contains both single and double quote characters"))
+                            self.serializeError("System identifer contains both single and double quote characters")
                         quote_char = "'"
                     else:
                         quote_char = '"'
@@ -220,7 +217,7 @@ class HTMLSerializer(object):
             elif type in ("Characters", "SpaceCharacters"):
                 if type == "SpaceCharacters" or in_cdata:
                     if in_cdata and token["data"].find("</") >= 0:
-                        self.serializeError(_("Unexpected </ in CDATA"))
+                        self.serializeError("Unexpected </ in CDATA")
                     yield self.encode(token["data"])
                 else:
                     yield self.encode(escape(token["data"]))
@@ -231,7 +228,7 @@ class HTMLSerializer(object):
                 if name in rcdataElements and not self.escape_rcdata:
                     in_cdata = True
                 elif in_cdata:
-                    self.serializeError(_("Unexpected child element of a CDATA element"))
+                    self.serializeError("Unexpected child element of a CDATA element")
                 for (attr_namespace, attr_name), attr_value in token["data"].items():
                     # TODO: Add namespace support here
                     k = attr_name
@@ -279,20 +276,20 @@ class HTMLSerializer(object):
                 if name in rcdataElements:
                     in_cdata = False
                 elif in_cdata:
-                    self.serializeError(_("Unexpected child element of a CDATA element"))
+                    self.serializeError("Unexpected child element of a CDATA element")
                 yield self.encodeStrict("</%s>" % name)
 
             elif type == "Comment":
                 data = token["data"]
                 if data.find("--") >= 0:
-                    self.serializeError(_("Comment contains --"))
+                    self.serializeError("Comment contains --")
                 yield self.encodeStrict("<!--%s-->" % token["data"])
 
             elif type == "Entity":
                 name = token["name"]
                 key = name + ";"
-                if not key in entities:
-                    self.serializeError(_("Entity %s not recognized" % name))
+                if key not in entities:
+                    self.serializeError("Entity %s not recognized" % name)
                 if self.resolve_entities and key not in xmlEntities:
                     data = entities[key]
                 else:

lib/html5lib/treebuilders/dom.py

@@ -158,7 +158,7 @@ def getDomBuilder(DomImplementation):
             else:
                 # HACK: allow text nodes as children of the document node
                 if hasattr(self.dom, '_child_node_types'):
-                    if not Node.TEXT_NODE in self.dom._child_node_types:
+                    if Node.TEXT_NODE not in self.dom._child_node_types:
                         self.dom._child_node_types = list(self.dom._child_node_types)
                         self.dom._child_node_types.append(Node.TEXT_NODE)
                 self.dom.appendChild(self.dom.createTextNode(data))

lib/html5lib/treewalkers/__init__.py

@@ -10,8 +10,12 @@ returning an iterator generating tokens.
 
 from __future__ import absolute_import, division, unicode_literals
 
+__all__ = ["getTreeWalker", "pprint", "dom", "etree", "genshistream", "lxmletree",
+           "pulldom"]
+
 import sys
 
+from .. import constants
 from ..utils import default_etree
 
 treeWalkerCache = {}
@@ -55,3 +59,89 @@ def getTreeWalker(treeType, implementation=None, **kwargs):
             # XXX: NEVER cache here, caching is done in the etree submodule
             return etree.getETreeModule(implementation, **kwargs).TreeWalker
     return treeWalkerCache.get(treeType)
+
+
+def concatenateCharacterTokens(tokens):
+    pendingCharacters = []
+    for token in tokens:
+        type = token["type"]
+        if type in ("Characters", "SpaceCharacters"):
+            pendingCharacters.append(token["data"])
+        else:
+            if pendingCharacters:
+                yield {"type": "Characters", "data": "".join(pendingCharacters)}
+                pendingCharacters = []
+            yield token
+    if pendingCharacters:
+        yield {"type": "Characters", "data": "".join(pendingCharacters)}
+
+
+def pprint(walker):
+    """Pretty printer for tree walkers"""
+    output = []
+    indent = 0
+    for token in concatenateCharacterTokens(walker):
+        type = token["type"]
+        if type in ("StartTag", "EmptyTag"):
+            # tag name
+            if token["namespace"] and token["namespace"] != constants.namespaces["html"]:
+                if token["namespace"] in constants.prefixes:
+                    ns = constants.prefixes[token["namespace"]]
+                else:
+                    ns = token["namespace"]
+                name = "%s %s" % (ns, token["name"])
+            else:
+                name = token["name"]
+            output.append("%s<%s>" % (" " * indent, name))
+            indent += 2
+            # attributes (sorted for consistent ordering)
+            attrs = token["data"]
+            for (namespace, localname), value in sorted(attrs.items()):
+                if namespace:
+                    if namespace in constants.prefixes:
+                        ns = constants.prefixes[namespace]
+                    else:
+                        ns = namespace
+                    name = "%s %s" % (ns, localname)
+                else:
+                    name = localname
+                output.append("%s%s=\"%s\"" % (" " * indent, name, value))
+            # self-closing
+            if type == "EmptyTag":
+                indent -= 2
+
+        elif type == "EndTag":
+            indent -= 2
+
+        elif type == "Comment":
+            output.append("%s<!-- %s -->" % (" " * indent, token["data"]))
+
+        elif type == "Doctype":
+            if token["name"]:
+                if token["publicId"]:
+                    output.append("""%s<!DOCTYPE %s "%s" "%s">""" %
+                                  (" " * indent,
+                                   token["name"],
+                                   token["publicId"],
+                                   token["systemId"] if token["systemId"] else ""))
+                elif token["systemId"]:
+                    output.append("""%s<!DOCTYPE %s "" "%s">""" %
+                                  (" " * indent,
+                                   token["name"],
+                                   token["systemId"]))
+                else:
+                    output.append("%s<!DOCTYPE %s>" % (" " * indent,
+                                                       token["name"]))
+            else:
+                output.append("%s<!DOCTYPE >" % (" " * indent,))
+
+        elif type == "Characters":
+            output.append("%s\"%s\"" % (" " * indent, token["data"]))
+
+        elif type == "SpaceCharacters":
+            assert False, "concatenateCharacterTokens should have got rid of all Space tokens"
+
+        else:
+            raise ValueError("Unknown token type, %s" % type)
+
+    return "\n".join(output)

lib/html5lib/treewalkers/_base.py

@@ -1,8 +1,8 @@
 from __future__ import absolute_import, division, unicode_literals
 from six import text_type, string_types
 
-import gettext
-_ = gettext.gettext
+__all__ = ["DOCUMENT", "DOCTYPE", "TEXT", "ELEMENT", "COMMENT", "ENTITY", "UNKNOWN",
+           "TreeWalker", "NonRecursiveTreeWalker"]
 
 from xml.dom import Node
 
@@ -58,7 +58,7 @@ class TreeWalker(object):
                "namespace": to_text(namespace),
                "data": attrs}
         if hasChildren:
-            yield self.error(_("Void element has children"))
+            yield self.error("Void element has children")
 
     def startTag(self, namespace, name, attrs):
         assert namespace is None or isinstance(namespace, string_types), type(namespace)
@@ -122,7 +122,7 @@ class TreeWalker(object):
         return {"type": "Entity", "name": text_type(name)}
 
     def unknown(self, nodeType):
-        return self.error(_("Unknown node type: ") + nodeType)
+        return self.error("Unknown node type: " + nodeType)
 
 
 class NonRecursiveTreeWalker(TreeWalker):

lib/html5lib/treewalkers/dom.py

@@ -2,9 +2,6 @@ from __future__ import absolute_import, division, unicode_literals
 
 from xml.dom import Node
 
-import gettext
-_ = gettext.gettext
-
 from . import _base
 
 

lib/html5lib/treewalkers/etree.py

@@ -7,12 +7,10 @@ except ImportError:
         from ordereddict import OrderedDict
     except ImportError:
         OrderedDict = dict
-import gettext
-_ = gettext.gettext
 
 import re
 
-from six import text_type
+from six import string_types
 
 from . import _base
 from ..utils import moduleFactoryFactory
@@ -60,7 +58,7 @@ def getETreeBuilder(ElementTreeImplementation):
                 return _base.COMMENT, node.text
 
             else:
-                assert type(node.tag) == text_type, type(node.tag)
+                assert isinstance(node.tag, string_types), type(node.tag)
                 # This is assumed to be an ordinary element
                 match = tag_regexp.match(node.tag)
                 if match:

lib/html5lib/treewalkers/lxmletree.py

@@ -4,9 +4,6 @@ from six import text_type
 from lxml import etree
 from ..treebuilders.etree import tag_regexp
 
-from gettext import gettext
-_ = gettext
-
 from . import _base
 
 from .. import ihatexml
@@ -130,7 +127,7 @@ class TreeWalker(_base.NonRecursiveTreeWalker):
     def getNodeDetails(self, node):
         if isinstance(node, tuple):  # Text node
             node, key = node
-            assert key in ("text", "tail"), _("Text nodes are text or tail, found %s") % key
+            assert key in ("text", "tail"), "Text nodes are text or tail, found %s" % key
             return _base.TEXT, ensure_str(getattr(node, key))
 
         elif isinstance(node, Root):
@@ -169,7 +166,7 @@ class TreeWalker(_base.NonRecursiveTreeWalker):
                     attrs, len(node) > 0 or node.text)
 
     def getFirstChild(self, node):
-        assert not isinstance(node, tuple), _("Text nodes have no children")
+        assert not isinstance(node, tuple), "Text nodes have no children"
 
         assert len(node) or node.text, "Node has no children"
         if node.text:
@@ -180,7 +177,7 @@ class TreeWalker(_base.NonRecursiveTreeWalker):
     def getNextSibling(self, node):
         if isinstance(node, tuple):  # Text node
             node, key = node
-            assert key in ("text", "tail"), _("Text nodes are text or tail, found %s") % key
+            assert key in ("text", "tail"), "Text nodes are text or tail, found %s" % key
             if key == "text":
                 # XXX: we cannot use a "bool(node) and node[0] or None" construct here
                 # because node[0] might evaluate to False if it has no child element
@@ -196,7 +193,7 @@ class TreeWalker(_base.NonRecursiveTreeWalker):
     def getParentNode(self, node):
         if isinstance(node, tuple):  # Text node
             node, key = node
-            assert key in ("text", "tail"), _("Text nodes are text or tail, found %s") % key
+            assert key in ("text", "tail"), "Text nodes are text or tail, found %s" % key
             if key == "text":
                 return node
             # else: fallback to "normal" processing

lib/html5lib/utils.py

@@ -2,6 +2,8 @@ from __future__ import absolute_import, division, unicode_literals
 
 from types import ModuleType
 
+from six import text_type
+
 try:
     import xml.etree.cElementTree as default_etree
 except ImportError:
@@ -9,7 +11,26 @@ except ImportError:
 
 
 __all__ = ["default_etree", "MethodDispatcher", "isSurrogatePair",
-           "surrogatePairToCodepoint", "moduleFactoryFactory"]
+           "surrogatePairToCodepoint", "moduleFactoryFactory",
+           "supports_lone_surrogates"]
+
+
+# Platforms not supporting lone surrogates (\uD800-\uDFFF) should be
+# caught by the below test. In general this would be any platform
+# using UTF-16 as its encoding of unicode strings, such as
+# Jython. This is because UTF-16 itself is based on the use of such
+# surrogates, and there is no mechanism to further escape such
+# escapes.
+try:
+    _x = eval('"\\uD800"')
+    if not isinstance(_x, text_type):
+        # We need this with u"" because of http://bugs.jython.org/issue2039
+        _x = eval('u"\\uD800"')
+        assert isinstance(_x, text_type)
+except:
+    supports_lone_surrogates = False
+else:
+    supports_lone_surrogates = True
 
 
 class MethodDispatcher(dict):

lib/musicbrainzngs/__init__.py

@@ -1 +1,2 @@
 from musicbrainzngs.musicbrainz import *
+from musicbrainzngs.caa import *

lib/musicbrainzngs/caa.py

@@ -0,0 +1,177 @@
+# This file is part of the musicbrainzngs library
+# Copyright (C) Alastair Porter, Wieland Hoffmann, and others
+# This file is distributed under a BSD-2-Clause type license.
+# See the COPYING file for more information.
+
+__all__ = [
+    'set_caa_hostname', 'get_image_list', 'get_release_group_image_list',
+    'get_release_group_image_front', 'get_image_front', 'get_image_back',
+    'get_image'
+    ]
+
+import json
+
+from musicbrainzngs import compat
+from musicbrainzngs import musicbrainz
+
+hostname = "coverartarchive.org"
+
+
+def set_caa_hostname(new_hostname):
+    """Set the base hostname for Cover Art Archive requests.
+    Defaults to 'coverartarchive.org'."""
+    global hostname
+    hostname = new_hostname
+
+
+def _caa_request(mbid, imageid=None, size=None, entitytype="release"):
+    """ Make a CAA request.
+
+    :param imageid: ``front``, ``back`` or a number from the listing obtained
+                    with :meth:`get_image_list`.
+    :type imageid: str
+
+    :param size: 250, 500
+    :type size: str or None
+
+    :param entitytype: ``release`` or ``release-group``
+    :type entitytype: str
+    """
+    # Construct the full URL for the request, including hostname and
+    # query string.
+    path = [entitytype, mbid]
+    if imageid and size:
+        path.append("%s-%s" % (imageid, size))
+    elif imageid:
+        path.append(imageid)
+    url = compat.urlunparse((
+        'http',
+        hostname,
+        '/%s' % '/'.join(path),
+        '',
+        '',
+        ''
+    ))
+    musicbrainz._log.debug("GET request for %s" % (url, ))
+
+    # Set up HTTP request handler and URL opener.
+    httpHandler = compat.HTTPHandler(debuglevel=0)
+    handlers = [httpHandler]
+
+    opener = compat.build_opener(*handlers)
+
+    # Make request.
+    req = musicbrainz._MusicbrainzHttpRequest("GET", url, None)
+    # Useragent isn't needed for CAA, but we'll add it if it exists
+    if musicbrainz._useragent != "":
+        req.add_header('User-Agent', musicbrainz._useragent)
+        musicbrainz._log.debug("requesting with UA %s" % musicbrainz._useragent)
+
+    resp = musicbrainz._safe_read(opener, req, None)
+
+    # TODO: The content type declared by the CAA for JSON files is
+    # 'applicaiton/octet-stream'. This is not useful to detect whether the
+    # content is JSON, so default to decoding JSON if no imageid was supplied.
+    # http://tickets.musicbrainz.org/browse/CAA-75
+    if imageid:
+        # If we asked for an image, return the image
+        return resp
+    else:
+        # Otherwise it's json
+        return json.loads(resp)
+
+
+def get_image_list(releaseid):
+    """Get the list of cover art associated with a release.
+
+    The return value is the deserialized response of the `JSON listing
+    <http://musicbrainz.org/doc/Cover_Art_Archive/API#.2Frelease.2F.7Bmbid.7D.2F>`_
+    returned by the Cover Art Archive API.
+
+    If an error occurs then a musicbrainz.ResponseError will
+    be raised with one of the following HTTP codes:
+
+    * 400: `Releaseid` is not a valid UUID
+    * 404: No release exists with an MBID of `releaseid`
+    * 503: Ratelimit exceeded
+    """
+    return _caa_request(releaseid)
+
+
+def get_release_group_image_list(releasegroupid):
+    """Get the list of cover art associated with a release group.
+
+    The return value is the deserialized response of the `JSON listing
+    <http://musicbrainz.org/doc/Cover_Art_Archive/API#.2Frelease-group.2F.7Bmbid.7D.2F>`_
+    returned by the Cover Art Archive API.
+
+    If an error occurs then a musicbrainz.ResponseError will
+    be raised with one of the following HTTP codes:
+
+    * 400: `Releaseid` is not a valid UUID
+    * 404: No release exists with an MBID of `releaseid`
+    * 503: Ratelimit exceeded
+    """
+    return _caa_request(releasegroupid, entitytype="release-group")
+
+
+def get_release_group_image_front(releasegroupid, size=None):
+    """Download the front cover art for a release group.
+    The `size` argument and the possible error conditions are the same as for
+    :meth:`get_image`.
+    """
+    return get_image(releasegroupid, "front", size=size,
+                              entitytype="release-group")
+
+
+def get_image_front(releaseid, size=None):
+    """Download the front cover art for a release.
+    The `size` argument and the possible error conditions are the same as for
+    :meth:`get_image`.
+    """
+    return get_image(releaseid, "front", size=size)
+
+
+def get_image_back(releaseid, size=None):
+    """Download the back cover art for a release.
+    The `size` argument and the possible error conditions are the same as for
+    :meth:`get_image`.
+    """
+    return get_image(releaseid, "back", size=size)
+
+
+def get_image(mbid, coverid, size=None, entitytype="release"):
+    """Download cover art for a release. The coverart file to download
+    is specified by the `coverid` argument.
+
+    If `size` is not specified, download the largest copy present, which can be
+    very large.
+
+    If an error occurs then a musicbrainz.ResponseError will be raised with one
+    of the following HTTP codes:
+
+    * 400: `Releaseid` is not a valid UUID or `coverid` is invalid
+    * 404: No release exists with an MBID of `releaseid`
+    * 503: Ratelimit exceeded
+
+    :param coverid: ``front``, ``back`` or a number from the listing obtained with
+                    :meth:`get_image_list`
+    :type coverid: int or str
+
+    :param size: 250, 500 or None. If it is None, the largest available picture
+                 will be downloaded. If the image originally uploaded to the
+                 Cover Art Archive was smaller than the requested size, only
+                 the original image will be returned.
+    :type size: str or None
+
+    :param entitytype: The type of entity for which to download the cover art.
+                       This is either ``release`` or ``release-group``.
+    :type entitytype: str
+    :return: The binary image data
+    :type: str
+    """
+    if isinstance(coverid, int):
+        coverid = "%d" % (coverid, )
+    if isinstance(size, int):
+        size = "%d" % (size, )
+    return _caa_request(mbid, coverid, size=size, entitytype=entitytype)

lib/musicbrainzngs/mbxml.py

@@ -36,6 +36,22 @@ NS_MAP = {"http://musicbrainz.org/ns/mmd-2.0#": "ws2",
           "http://musicbrainz.org/ns/ext#-2.0": "ext"}
 _log = logging.getLogger("musicbrainzngs")
 
+def get_error_message(error):
+    """ Given an error XML message from the webservice containing
+    <error><text>x</text><text>y</text></error>, return a list
+    of [x, y]"""
+    try:
+        tree = util.bytes_to_elementtree(error)
+        root = tree.getroot()
+        errors = []
+        if root.tag == "error":
+            for ch in root:
+                if ch.tag == "text":
+                    errors.append(ch.text)
+        return errors
+    except ET.ParseError:
+        return None
+
 def make_artist_credit(artists):
     names = []
     for artist in artists:
@@ -123,6 +139,7 @@ def parse_message(message):
                       "place": parse_place,
                       "release": parse_release,
                       "release-group": parse_release_group,
+                      "series": parse_series,
                       "recording": parse_recording,
                       "work": parse_work,
                       "url": parse_url,
@@ -138,6 +155,7 @@ def parse_message(message):
                       "place-list": parse_place_list,
                       "release-list": parse_release_list,
                       "release-group-list": parse_release_group_list,
+                      "series-list": parse_series_list,
                       "recording-list": parse_recording_list,
                       "work-list": parse_work_list,
                       "url-list": parse_url_list,
@@ -297,7 +315,7 @@ def parse_relation_list(rl):
 def parse_relation(relation):
     result = {}
     attribs = ["type", "type-id"]
-    elements = ["target", "direction", "begin", "end", "ended"]
+    elements = ["target", "direction", "begin", "end", "ended", "ordering-key"]
     inner_els = {"area": parse_area,
                  "artist": parse_artist,
                  "label": parse_label,
@@ -305,6 +323,7 @@ def parse_relation(relation):
                  "recording": parse_recording,
                  "release": parse_release,
                  "release-group": parse_release_group,
+                 "series": parse_series,
                  "attribute-list": parse_element_list,
                  "work": parse_work,
                  "target": parse_relation_target
@@ -324,6 +343,8 @@ def parse_release(release):
                  "label-info-list": parse_label_info_list,
                  "medium-list": parse_medium_list,
                  "release-group": parse_release_group,
+                 "tag-list": parse_tag_list,
+                 "user-tag-list": parse_tag_list,
                  "relation-list": parse_relation_list,
                  "annotation": parse_annotation,
                  "cover-art-archive": parse_caa,
@@ -408,6 +429,22 @@ def parse_recording(recording):
 
     return result
 
+def parse_series_list(sl):
+    return [parse_series(s) for s in sl]
+
+def parse_series(series):
+    result = {}
+    attribs = ["id", "type", "ext:score"]
+    elements = ["name", "disambiguation"]
+    inner_els = {"alias-list": parse_alias_list,
+                 "relation-list": parse_relation_list,
+                 "annotation": parse_annotation}
+
+    result.update(parse_attributes(attribs, series))
+    result.update(parse_elements(elements, inner_els, series))
+
+    return result
+
 def parse_external_id_list(pl):
     return [parse_attributes(["id"], p)["id"] for p in pl]
 
@@ -427,13 +464,28 @@ def parse_work(work):
                  "alias-list": parse_alias_list,
                  "iswc-list": parse_element_list,
                  "relation-list": parse_relation_list,
-                 "annotation": parse_response_message}
+                 "annotation": parse_response_message,
+                 "attribute-list": parse_work_attribute_list
+    }
 
     result.update(parse_attributes(attribs, work))
     result.update(parse_elements(elements, inner_els, work))
 
     return result
 
+def parse_work_attribute_list(wal):
+    return [parse_work_attribute(wa) for wa in wal]
+
+def parse_work_attribute(wa):
+    result = {}
+    attribs = ["type"]
+
+    result.update(parse_attributes(attribs, wa))
+    result["attribute"] = wa.text
+
+    return result
+
+
 def parse_url_list(ul):
     return [parse_url(u) for u in ul]
 
@@ -617,45 +669,40 @@ def make_barcode_request(release2barcode):
 
     return ET.tostring(root, "utf-8")
 
-def make_tag_request(artist2tags, recording2tags):
+def make_tag_request(**kwargs):
     NS = "http://musicbrainz.org/ns/mmd-2.0#"
     root = ET.Element("{%s}metadata" % NS)
-    rec_list = ET.SubElement(root, "{%s}recording-list" % NS)
-    for rec, tags in recording2tags.items():
-        rec_xml = ET.SubElement(rec_list, "{%s}recording" % NS)
-        rec_xml.set("{%s}id" % NS, rec)
-        taglist = ET.SubElement(rec_xml, "{%s}user-tag-list" % NS)
-        for tag in tags:
-            usertag_xml = ET.SubElement(taglist, "{%s}user-tag" % NS)
-            name_xml = ET.SubElement(usertag_xml, "{%s}name" % NS)
-            name_xml.text = tag
-    art_list = ET.SubElement(root, "{%s}artist-list" % NS)
-    for art, tags in artist2tags.items():
-        art_xml = ET.SubElement(art_list, "{%s}artist" % NS)
-        art_xml.set("{%s}id" % NS, art)
-        taglist = ET.SubElement(art_xml, "{%s}user-tag-list" % NS)
-        for tag in tags:
-            usertag_xml = ET.SubElement(taglist, "{%s}user-tag" % NS)
-            name_xml = ET.SubElement(usertag_xml, "{%s}name" % NS)
-            name_xml.text = tag
+    for entity_type in ['artist', 'label', 'place', 'recording', 'release', 'release_group', 'work']:
+        entity_tags = kwargs.pop(entity_type + '_tags', None)
+        if entity_tags is not None:
+            e_list = ET.SubElement(root, "{%s}%s-list" % (NS, entity_type.replace('_', '-')))
+            for e, tags in entity_tags.items():
+                e_xml = ET.SubElement(e_list, "{%s}%s" % (NS, entity_type.replace('_', '-')))
+                e_xml.set("{%s}id" % NS, e)
+                taglist = ET.SubElement(e_xml, "{%s}user-tag-list" % NS)
+                for tag in tags:
+                    usertag_xml = ET.SubElement(taglist, "{%s}user-tag" % NS)
+                    name_xml = ET.SubElement(usertag_xml, "{%s}name" % NS)
+                    name_xml.text = tag
+    if kwargs.keys():
+        raise TypeError("make_tag_request() got an unexpected keyword argument '%s'" % kwargs.popitem()[0])
 
     return ET.tostring(root, "utf-8")
 
-def make_rating_request(artist2rating, recording2rating):
+def make_rating_request(**kwargs):
     NS = "http://musicbrainz.org/ns/mmd-2.0#"
     root = ET.Element("{%s}metadata" % NS)
-    rec_list = ET.SubElement(root, "{%s}recording-list" % NS)
-    for rec, rating in recording2rating.items():
-        rec_xml = ET.SubElement(rec_list, "{%s}recording" % NS)
-        rec_xml.set("{%s}id" % NS, rec)
-        rating_xml = ET.SubElement(rec_xml, "{%s}user-rating" % NS)
-        rating_xml.text = str(rating)
-    art_list = ET.SubElement(root, "{%s}artist-list" % NS)
-    for art, rating in artist2rating.items():
-        art_xml = ET.SubElement(art_list, "{%s}artist" % NS)
-        art_xml.set("{%s}id" % NS, art)
-        rating_xml = ET.SubElement(art_xml, "{%s}user-rating" % NS)
-        rating_xml.text = str(rating)
+    for entity_type in ['artist', 'label', 'recording', 'release_group', 'work']:
+        entity_ratings = kwargs.pop(entity_type + '_ratings', None)
+        if entity_ratings is not None:
+            e_list = ET.SubElement(root, "{%s}%s-list" % (NS, entity_type.replace('_', '-')))
+            for e, rating in entity_ratings.items():
+                e_xml = ET.SubElement(e_list, "{%s}%s" % (NS, entity_type.replace('_', '-')))
+                e_xml.set("{%s}id" % NS, e)
+                rating_xml = ET.SubElement(e_xml, "{%s}user-rating" % NS)
+                rating_xml.text = str(rating)
+    if kwargs.keys():
+        raise TypeError("make_rating_request() got an unexpected keyword argument '%s'" % kwargs.popitem()[0])
 
     return ET.tostring(root, "utf-8")
 

lib/musicbrainzngs/musicbrainz.py

@@ -11,25 +11,25 @@ import socket
 import hashlib
 import locale
 import sys
-import base64
+import json
 import xml.etree.ElementTree as etree
 from xml.parsers import expat
-from warnings import warn, simplefilter
+from warnings import warn
 
 from musicbrainzngs import mbxml
 from musicbrainzngs import util
 from musicbrainzngs import compat
 
+import base64
+
 _version = "0.6devMODIFIED"
 _log = logging.getLogger("musicbrainzngs")
 
-# turn on DeprecationWarnings below
-simplefilter(action="once", category=DeprecationWarning)
-
+LUCENE_SPECIAL = r'([+\-&|!(){}\[\]\^"~*?:\\\/])'
 
 # Constants for validation.
 
-RELATABLE_TYPES = ['area', 'artist', 'label', 'place', 'recording', 'release', 'release-group', 'url', 'work']
+RELATABLE_TYPES = ['area', 'artist', 'label', 'place', 'recording', 'release', 'release-group', 'series', 'url', 'work']
 RELATION_INCLUDES = [entity + '-rels' for entity in RELATABLE_TYPES]
 TAG_INCLUDES = ["tags", "user-tags"]
 RATING_INCLUDES = ["ratings", "user-ratings"]
@@ -43,6 +43,9 @@ VALID_INCLUDES = {
     ] + RELATION_INCLUDES + TAG_INCLUDES + RATING_INCLUDES,
     'annotation': [
 
+    ],
+    'instrument': [
+
     ],
     'label': [
         "releases", # Subqueries
@@ -59,20 +62,23 @@ VALID_INCLUDES = {
         "artists", "labels", "recordings", "release-groups", "media",
         "artist-credits", "discids", "puids", "isrcs",
         "recording-level-rels", "work-level-rels", "annotation", "aliases"
-    ] + RELATION_INCLUDES,
+    ] + TAG_INCLUDES + RELATION_INCLUDES,
     'release-group': [
         "artists", "releases", "discids", "media",
         "artist-credits", "annotation", "aliases"
     ] + TAG_INCLUDES + RATING_INCLUDES + RELATION_INCLUDES,
+    'series': [
+        "annotation", "aliases"
+    ] + RELATION_INCLUDES,
     'work': [
         "artists", # Subqueries
         "aliases", "annotation"
     ] + TAG_INCLUDES + RATING_INCLUDES + RELATION_INCLUDES,
     'url': RELATION_INCLUDES,
-    'discid': [
+    'discid': [ # Discid should be the same as release
         "artists", "labels", "recordings", "release-groups", "media",
         "artist-credits", "discids", "puids", "isrcs",
-        "recording-level-rels", "work-level-rels"
+        "recording-level-rels", "work-level-rels", "annotation", "aliases"
     ] + RELATION_INCLUDES,
     'isrc': ["artists", "releases", "puids", "isrcs"],
     'iswc': ["artists"],
@@ -93,7 +99,8 @@ VALID_RELEASE_TYPES = [
     "nat",
     "album", "single", "ep", "broadcast", "other", # primary types
     "compilation", "soundtrack", "spokenword", "interview", "audiobook",
-    "live", "remix", "dj-mix", "mixtape/street", "demo", # secondary types
+    "live", "remix", "dj-mix", "mixtape/street", # secondary types
+    "demo", # headphones
 ]
 #: These can be used to filter whenever releases or release-groups are involved
 VALID_RELEASE_STATUSES = ["official", "promotion", "bootleg", "pseudo-release"]
@@ -101,6 +108,10 @@ VALID_SEARCH_FIELDS = {
     'annotation': [
         'entity', 'name', 'text', 'type'
     ],
+    'area': [
+        'aid', 'area', 'alias', 'begin', 'comment', 'end', 'ended',
+        'iso', 'iso1', 'iso2', 'iso3', 'type'
+    ],
     'artist': [
         'arid', 'artist', 'artistaccent', 'alias', 'begin', 'comment',
         'country', 'end', 'ended', 'gender', 'ipi', 'sortname', 'tag', 'type',
@@ -133,12 +144,20 @@ VALID_SEARCH_FIELDS = {
         'rgid', 'script', 'secondarytype', 'status', 'tag', 'tracks',
         'tracksmedium', 'type'
     ],
+    'series': [
+        'alias', 'comment', 'sid', 'series', 'type'
+    ],
     'work': [
         'alias', 'arid', 'artist', 'comment', 'iswc', 'lang', 'tag',
         'type', 'wid', 'work', 'workaccent'
     ],
 }
 
+# Constants
+class AUTH_YES: pass
+class AUTH_NO: pass
+class AUTH_IFSET: pass
+
 
 # Exceptions.
 
@@ -271,6 +290,7 @@ user = password = ""
 hostname = "musicbrainz.org"
 _client = ""
 _useragent = ""
+mb_auth = False
 
 def auth(u, p):
 	"""Set the username and password to be used in subsequent queries to
@@ -279,14 +299,21 @@ def auth(u, p):
 	global user, password
 	user = u
 	password = p
-
+	
 def hpauth(u, p):
     """Set the username and password to be used in subsequent queries to
     the MusicBrainz XML API that require authentication.
     """
-    global hpuser, hppassword
+    global hpuser, hppassword, mb_auth
     hpuser = u
     hppassword = p
+    mb_auth = True
+
+def disable_hpauth():
+    """Disable the authentication for MusicBrainz XML API
+    """
+    global mb_auth
+    mb_auth = False
 
 def set_useragent(app, version, contact=None):
     """Set the User-Agent to be used for requests to the MusicBrainz webservice.
@@ -551,28 +578,33 @@ def set_format(fmt="xml"):
     """Sets the format that should be returned by the Web Service.
     The server currently supports `xml` and `json`.
 
-    When you set the format to anything different from the default,
-    you need to provide your own parser with :func:`set_parser`.
+    This method will set a default parser for the specified format,
+    but you can modify it with :func:`set_parser`.
 
     .. warning:: The json format used by the server is different from
         the json format returned by the `musicbrainzngs` internal parser
-        when using the `xml` format!
+        when using the `xml` format! This format may change at any time.
     """
     global ws_format
-    if fmt not in ["xml", "json"]:
-        raise ValueError("invalid format: %s" % fmt)
-    else:
+    if fmt == "xml":
         ws_format = fmt
+        set_parser() # set to default
+    elif fmt == "json":
+        ws_format = fmt
+        warn("The json format is non-official and may change at any time")
+        set_parser(json.loads)
+    else:
+        raise ValueError("invalid format: %s" % fmt)
 
 
 @_rate_limit
-def _mb_request(path, method='GET', auth_required=False, client_required=False,
-                args=None, data=None, body=None):
+def _mb_request(path, method='GET', auth_required=AUTH_NO,
+                client_required=False, args=None, data=None, body=None):
     """Makes a request for the specified `path` (endpoint) on /ws/2 on
     the globally-specified hostname. Parses the responses and returns
     the resulting object.  `auth_required` and `client_required` control
-    whether exceptions should be raised if the client and
-    username/password are left unspecified, respectively.
+    whether exceptions should be raised if the username/password and
+    client are left unspecified, respectively.
     """
     global parser_fun
 
@@ -618,11 +650,19 @@ def _mb_request(path, method='GET', auth_required=False, client_required=False,
     handlers = [httpHandler]
 
     # Add credentials if required.
-    if auth_required:
+    add_auth = False
+    if auth_required == AUTH_YES:
         _log.debug("Auth required for %s" % url)
         if not user:
             raise UsageError("authorization required; "
                              "use auth(user, pass) first")
+        add_auth = True
+
+    if auth_required == AUTH_IFSET and user:
+        _log.debug("Using auth for %s because user and pass is set" % url)
+        add_auth = True
+
+    if add_auth:
         passwordMgr = _RedirectPasswordMgr()
         authHandler = _DigestAuthHandler(passwordMgr)
         authHandler.add_password("musicbrainz.org", (), user, password)
@@ -633,9 +673,10 @@ def _mb_request(path, method='GET', auth_required=False, client_required=False,
     # Make request.
     req = _MusicbrainzHttpRequest(method, url, data)
     req.add_header('User-Agent', _useragent)
+    
 
     # Add headphones credentials
-    if hostname == '144.76.94.239:8181':
+    if mb_auth:
         base64string = base64.encodestring('%s:%s' % (hpuser, hppassword)).replace('\n', '')
         req.add_header("Authorization", "Basic %s" % base64string)
 
@@ -650,16 +691,19 @@ def _mb_request(path, method='GET', auth_required=False, client_required=False,
 
     return parser_fun(resp)
 
-def _is_auth_required(entity, includes):
-	""" Some calls require authentication. This returns
-	True if a call does, False otherwise
-	"""
-	if "user-tags" in includes or "user-ratings" in includes:
-		return True
-	elif entity.startswith("collection"):
-		return True
-	else:
-		return False
+def _get_auth_type(entity, id, includes):
+    """ Some calls require authentication. This returns
+    True if a call does, False otherwise
+    """
+    if "user-tags" in includes or "user-ratings" in includes:
+        return AUTH_YES
+    elif entity.startswith("collection"):
+        if not id:
+            return AUTH_YES
+        else:
+            return AUTH_IFSET
+    else:
+        return AUTH_NO
 
 def _do_mb_query(entity, id, includes=[], params={}):
 	"""Make a single GET call to the MusicBrainz XML API. `entity` is a
@@ -673,7 +717,7 @@ def _do_mb_query(entity, id, includes=[], params={}):
 	if not isinstance(includes, list):
 		includes = [includes]
 	_check_includes(entity, includes)
-	auth_required = _is_auth_required(entity, includes)
+	auth_required = _get_auth_type(entity, id, includes)
 	args = dict(params)
 	if len(includes) > 0:
 		inc = " ".join(includes)
@@ -696,8 +740,8 @@ def _do_mb_search(entity, query='', fields={},
 	if query:
 		clean_query = util._unicode(query)
 		if fields:
-			clean_query = re.sub(r'([+\-&|!(){}\[\]\^"~*?:\\])',
-					r'\\\1', clean_query)
+			clean_query = re.sub(LUCENE_SPECIAL, r'\\\1',
+					     clean_query)
 			if strict:
 				query_parts.append('"%s"' % clean_query)
 			else:
@@ -713,11 +757,11 @@ def _do_mb_search(entity, query='', fields={},
 		elif key == "puid":
 			warn("PUID support was removed from server\n"
 			     "the 'puid' field is ignored",
-			     DeprecationWarning, stacklevel=2)
+			     Warning, stacklevel=2)
 
 		# Escape Lucene's special characters.
 		value = util._unicode(value)
-		value = re.sub(r'([+\-&|!(){}\[\]\^"~*?:\\\/])', r'\\\1', value)
+		value = re.sub(LUCENE_SPECIAL, r'\\\1', value)
 		if value:
 			if strict:
 				query_parts.append('%s:"%s"' % (key, value))
@@ -744,18 +788,18 @@ def _do_mb_search(entity, query='', fields={},
 def _do_mb_delete(path):
 	"""Send a DELETE request for the specified object.
 	"""
-	return _mb_request(path, 'DELETE', True, True)
+	return _mb_request(path, 'DELETE', AUTH_YES, True)
 
 def _do_mb_put(path):
 	"""Send a PUT request for the specified object.
 	"""
-	return _mb_request(path, 'PUT', True, True)
+	return _mb_request(path, 'PUT', AUTH_YES, True)
 
 def _do_mb_post(path, body):
 	"""Perform a single POST call for an endpoint with a specified
 	request body.
 	"""
-	return _mb_request(path, 'POST', True, True, body=body)
+	return _mb_request(path, 'POST', AUTH_YES, True, body=body)
 
 
 # The main interface!
@@ -780,6 +824,15 @@ def get_artist_by_id(id, includes=[], release_status=[], release_type=[]):
                                            release_status, release_type)
     return _do_mb_query("artist", id, includes, params)
 
+@_docstring('instrument')
+def get_instrument_by_id(id, includes=[], release_status=[], release_type=[]):
+    """Get the instrument with the MusicBrainz `id` as a dict with an 'artist' key.
+
+    *Available includes*: {includes}"""
+    params = _check_filter_and_make_params("instrument", includes,
+                                           release_status, release_type)
+    return _do_mb_query("instrument", id, includes, params)
+
 @_docstring('label')
 def get_label_by_id(id, includes=[], release_status=[], release_type=[]):
     """Get the label with the MusicBrainz `id` as a dict with a 'label' key.
@@ -828,6 +881,13 @@ def get_release_group_by_id(id, includes=[],
                                            release_status, release_type)
     return _do_mb_query("release-group", id, includes, params)
 
+@_docstring('series')
+def get_series_by_id(id, includes=[]):
+    """Get the series with the MusicBrainz `id` as a dict with a 'series' key.
+
+    *Available includes*: {includes}"""
+    return _do_mb_query("series", id, includes)
+
 @_docstring('work')
 def get_work_by_id(id, includes=[]):
     """Get the work with the MusicBrainz `id` as a dict with a 'work' key.
@@ -852,6 +912,13 @@ def search_annotations(query='', limit=None, offset=None, strict=False, **fields
     *Available search fields*: {fields}"""
     return _do_mb_search('annotation', query, fields, limit, offset, strict)
 
+@_docstring('area')
+def search_areas(query='', limit=None, offset=None, strict=False, **fields):
+    """Search for areas and return a dict with an 'area-list' key.
+
+    *Available search fields*: {fields}"""
+    return _do_mb_search('area', query, fields, limit, offset, strict)
+
 @_docstring('artist')
 def search_artists(query='', limit=None, offset=None, strict=False, **fields):
     """Search for artists and return a dict with an 'artist-list' key.
@@ -890,6 +957,13 @@ def search_release_groups(query='', limit=None, offset=None,
     *Available search fields*: {fields}"""
     return _do_mb_search('release-group', query, fields, limit, offset, strict)
 
+@_docstring('series')
+def search_series(query='', limit=None, offset=None, strict=False, **fields):
+    """Search for series and return a dict with a 'series-list' key.
+
+    *Available search fields*: {fields}"""
+    return _do_mb_search('series', query, fields, limit, offset, strict)
+
 @_docstring('work')
 def search_works(query='', limit=None, offset=None, strict=False, **fields):
     """Search for works and return a dict with a 'work-list' key.
@@ -899,18 +973,25 @@ def search_works(query='', limit=None, offset=None, strict=False, **fields):
 
 
 # Lists of entities
-@_docstring('release')
-def get_releases_by_discid(id, includes=[], toc=None, cdstubs=True):
-    """Search for releases with a :musicbrainz:`Disc ID`.
+@_docstring('discid')
+def get_releases_by_discid(id, includes=[], toc=None, cdstubs=True, media_format=None):
+    """Search for releases with a :musicbrainz:`Disc ID` or table of contents.
 
     When a `toc` is provided and no release with the disc ID is found,
     a fuzzy search by the toc is done.
     The `toc` should have to same format as :attr:`discid.Disc.toc_string`.
+    When a `toc` is provided, the format of the discid itself is not
+    checked server-side, so any value may be passed if searching by only
+    `toc` is desired.
 
     If no toc matches in musicbrainz but a :musicbrainz:`CD Stub` does,
     the CD Stub will be returned. Prevent this from happening by
     passing `cdstubs=False`.
 
+    By default only results that match a format that allows discids
+    (e.g. CD) are included. To include all media formats, pass
+    `media_format='all'`.
+
     The result is a dict with either a 'disc' , a 'cdstub' key
     or a 'release-list' (fuzzy match with TOC).
     A 'disc' has a 'release-list' and a 'cdstub' key has direct 'artist'
@@ -923,6 +1004,8 @@ def get_releases_by_discid(id, includes=[], toc=None, cdstubs=True):
         params["toc"] = toc
     if not cdstubs:
         params["cdstubs"] = "no"
+    if media_format:
+        params["media-format"] = media_format
     return _do_mb_query("discid", id, includes, params)
 
 @_docstring('recording')
@@ -932,7 +1015,7 @@ def get_recordings_by_echoprint(echoprint, includes=[], release_status=[],
     (not available on server)"""
     warn("Echoprints were never introduced\n"
          "and will not be found (404)",
-         DeprecationWarning, stacklevel=2)
+         Warning, stacklevel=2)
     raise ResponseError(cause=compat.HTTPError(
                                             None, 404, "Not Found", None, None))
 
@@ -943,7 +1026,7 @@ def get_recordings_by_puid(puid, includes=[], release_status=[],
     (not available on server)"""
     warn("PUID support was removed from the server\n"
          "and no PUIDs will be found (404)",
-         DeprecationWarning, stacklevel=2)
+         Warning, stacklevel=2)
     raise ResponseError(cause=compat.HTTPError(
                                             None, 404, "Not Found", None, None))
 
@@ -1108,7 +1191,7 @@ def submit_puids(recording_puids):
     """
     warn("PUID support was dropped at the server\n"
          "nothing will be submitted",
-         DeprecationWarning, stacklevel=2)
+         Warning, stacklevel=2)
     return {'message': {'text': 'OK'}}
 
 def submit_echoprints(recording_echoprints):
@@ -1117,7 +1200,7 @@ def submit_echoprints(recording_echoprints):
     """
     warn("Echoprints were never introduced\n"
          "nothing will be submitted",
-         DeprecationWarning, stacklevel=2)
+         Warning, stacklevel=2)
     return {'message': {'text': 'OK'}}
 
 def submit_isrcs(recording_isrcs):
@@ -1131,20 +1214,29 @@ def submit_isrcs(recording_isrcs):
     query = mbxml.make_isrc_request(rec2isrcs)
     return _do_mb_post("recording", query)
 
-def submit_tags(artist_tags={}, recording_tags={}):
+def submit_tags(**kwargs):
     """Submit user tags.
-    Artist or recording parameters are of the form:
+    Takes parameters named e.g. 'artist_tags', 'recording_tags', etc.,
+    and of the form:
     {entity_id1: [tag1, ...], ...}
+
+    The user's tags for each entity will be set to that list, adding or
+    removing tags as necessary. Submitting an empty list for an entity
+    will remove all tags for that entity by the user.
     """
-    query = mbxml.make_tag_request(artist_tags, recording_tags)
+    query = mbxml.make_tag_request(**kwargs)
     return _do_mb_post("tag", query)
 
-def submit_ratings(artist_ratings={}, recording_ratings={}):
-    """ Submit user ratings.
-    Artist or recording parameters are of the form:
+def submit_ratings(**kwargs):
+    """Submit user ratings.
+    Takes parameters named e.g. 'artist_ratings', 'recording_ratings', etc.,
+    and of the form:
     {entity_id1: rating, ...}
+
+    Ratings are numbers from 0-100, at intervals of 20 (20 per 'star').
+    Submitting a rating of 0 will remove the user's rating.
     """
-    query = mbxml.make_rating_request(artist_ratings, recording_ratings)
+    query = mbxml.make_rating_request(**kwargs)
     return _do_mb_post("rating", query)
 
 def add_releases_to_collection(collection, releases=[]):

lib/pynma/__init__.py

@@ -1,4 +1,4 @@
 #!/usr/bin/python
 
-from pynma import PyNMA
+from .pynma import PyNMA 
 

lib/pynma/pynma.py

@@ -1,12 +1,20 @@
 #!/usr/bin/python
 
 from xml.dom.minidom import parseString
-from httplib import HTTPSConnection
-from urllib import urlencode
 
-__version__ = "0.1"
+try:
+        from http.client import HTTPSConnection
+except ImportError:
+        from httplib import HTTPSConnection
 
-API_SERVER = 'nma.usk.bz'
+try:
+        from urllib.parse import urlencode
+except ImportError:
+        from urllib import urlencode
+
+__version__ = "1.0"
+
+API_SERVER = 'www.notifymyandroid.com' 
 ADD_PATH   = '/publicapi/notify'
 
 USER_AGENT="PyNMA/v%s"%__version__
@@ -18,14 +26,14 @@ def uniq_preserve(seq): # Dave Kirby
 
 def uniq(seq):
     # Not order preserving
-    return {}.fromkeys(seq).keys()
+    return list({}.fromkeys(seq).keys())
 
 class PyNMA(object):
     """PyNMA(apikey=[], developerkey=None)
-        takes 2 optional arguments:
-            - (opt) apykey:      might me a string containing 1 key or an array of keys
-            - (opt) developerkey: where you can store your developer key
-    """
+takes 2 optional arguments:
+ - (opt) apykey:      might me a string containing 1 key or an array of keys
+ - (opt) developerkey: where you can store your developer key
+"""
 
     def __init__(self, apikey=[], developerkey=None):
         self._developerkey = None
@@ -60,19 +68,20 @@ class PyNMA(object):
         if type(developerkey) == str and len(developerkey) == 48:
             self._developerkey = developerkey
 
-    def push(self, application="", event="", description="", url="", priority=0, batch_mode=False):
+    def push(self, application="", event="", description="", url="", contenttype=None, priority=0, batch_mode=False, html=False):
         """Pushes a message on the registered API keys.
-            takes 5 arguments:
-                - (req) application: application name [256]
-                - (req) event:       event name       [1000]
-                - (req) description: description      [10000]
-                - (opt) url:         url              [512]
-                - (opt) priority:    from -2 (lowest) to 2 (highest) (def:0)
-                - (opt) batch_mode:  call API 5 by 5 (def:False)
-
-            Warning: using batch_mode will return error only if all API keys are bad
-            cf: http://nma.usk.bz/api.php
-        """
+takes 5 arguments:
+ - (req) application: application name [256]
+ - (req) event:       event name       [1000]
+ - (req) description: description      [10000]
+ - (opt) url:         url              [512]
+ - (opt) contenttype: Content Type (act: None (plain text) or text/html)
+ - (opt) priority:    from -2 (lowest) to 2 (highest) (def:0)
+ - (opt) batch_mode:  push to all keys at once (def:False)
+ - (opt) html:        shortcut for contenttype=text/html
+Warning: using batch_mode will return error only if all API keys are bad
+ cf: http://nma.usk.bz/api.php
+"""
         datas = {
             'application': application[:256].encode('utf8'),
             'event':       event[:1024].encode('utf8'),
@@ -82,7 +91,10 @@ class PyNMA(object):
 
         if url:
             datas['url'] = url[:512]
-
+        
+        if contenttype == "text/html" or html == True: # Currently only accepted content type
+            datas['content-type'] = "text/html"
+        
         if self._developerkey:
             datas['developerkey'] = self._developerkey
 
@@ -94,12 +106,11 @@ class PyNMA(object):
                 res = self.callapi('POST', ADD_PATH, datas)
                 results[key] = res
         else:
-            for i in range(0, len(self._apikey), 5):
-                datas['apikey'] = ",".join(self._apikey[i:i+5])
-                res = self.callapi('POST', ADD_PATH, datas)
-                results[datas['apikey']] = res
+            datas['apikey'] = ",".join(self._apikey)
+            res = self.callapi('POST', ADD_PATH, datas)
+            results[datas['apikey']] = res
         return results
-
+        
     def callapi(self, method, path, args):
         headers = { 'User-Agent': USER_AGENT }
         if method == "POST":
@@ -110,13 +121,13 @@ class PyNMA(object):
 
         try:
             res = self._parse_reponse(resp.read())
-        except Exception, e:
+        except Exception as e:
             res = {'type':    "pynmaerror",
                    'code':    600,
                    'message': str(e)
                    }
             pass
-
+        
         return res
 
     def _parse_reponse(self, response):
@@ -124,14 +135,14 @@ class PyNMA(object):
         for elem in root.childNodes:
             if elem.nodeType == elem.TEXT_NODE: continue
             if elem.tagName == 'success':
-                res = dict(elem.attributes.items())
+                res = dict(list(elem.attributes.items()))
                 res['message'] = ""
                 res['type']    = elem.tagName
                 return res
             if elem.tagName == 'error':
-                res = dict(elem.attributes.items())
+                res = dict(list(elem.attributes.items()))
                 res['message'] = elem.firstChild.nodeValue
                 res['type']    = elem.tagName
                 return res
-
-
+                                        
+