Release v0.5.5

* origin/master:
  Release v0.5.4
  Added API call getSnatched
  Potential fix for #2055

API.md

@@ -68,6 +68,8 @@ Unmark album as wanted / i.e. mark as skipped
 force search for wanted albums - not launched in a separate thread so it may take a bit to complete
 ### forceProcess
 Force post process albums in download directory - also not launched in a separate thread
+### forceActiveArtistsUpdate
+force Active Artist Update - also not launched in a separate thread
 
 ### getVersion
 Returns some version information: git_path, install_type, current_version, installed_version, commits_behind

CHANGELOG.md

@@ -1,9 +1,57 @@
 # Changelog
 
-## v0.5.2
-Released 28 december 2014
+## v0.5.5
+Released 04 May 2015
 
-Highlight:
+Highlights:
+* Added: force ID3v2.3 during post processing (#2121)
+* Added: MusicBrainz authentication (#2125)
+* Added: Email notifications (addresses #1045)
+* Fixed: Kickass url updated to kickass.to (#2119)
+* Fixed: Piratebay returning 0 bytes for all files
+* Fixed: Albums stopped automatically refreshing when adding an artist
+* Fixed: Min/max sizes for target bitrate
+* Fixed: Don't filter any results if looking for a specific download
+* Fixed: Sort by size in the specific download table
+* Fixed: Deal with beets recommendation.none correctly
+* Improved: Close dialog window automatically when choosing a specific download
+* Improved: Move some repetitive log messages to debug level
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.4...v0.5.5).
+
+## v0.5.4
+Released 05 February 2015
+
+Highlights:
+* Added: backported 'Scan Artist' feature from `sarakha63/headphones`
+* Fixed: change file permissions of cache files according to settings (#2102)
+* Fixed: hide Songkick if no regional events are available
+* Fixed: only reschedule jobs if timeout changed (#2099)
+* Fixed: limit dialog height (#2096)
+* Improved: upgraded requests to 2.5.1
+* Improved: upgraded unidecode to 0.04.17
+* Improved: upgraded tzlocal to 1.1.2
+* Improved: upgraded PyTZ to 2014.10
+* Improved: upgraded mako to 1.0.1
+* Improved: upgraded mutagen to 1.27
+* Improved: upgraded beets to 1.3.10
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.3...v0.5.4).
+
+## v0.5.3
+Released 15 January 2015
+
+Highlights:
+* Added: update active artists via API (#2075)
+* Fixed: queue instance lacks `add` method (#2055)
+* Improved: better SSL error messages (#2058)
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.2...v0.5.3).
+
+## v0.5.2
+Released 28 December 2014
+
+Highlights:
 * Added: advanced option to ignore certain folders by patterns. (#2037)
 * Added: advanced option to ignore certain files by patterns (library only)
 * Added: specify optional paths to CUE splitting tools (#1938)
@@ -126,4 +174,4 @@ The full list of commits can be found [here](https://github.com/rembo10/headphon
 ## v0.1
 Released 05 August 2013
 
-The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/2156e1341405d07c5bcfbe994f6b354b32d94cda...v0.1).
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/2156e1341405d07c5bcfbe994f6b354b32d94cda...v0.1).

data/interfaces/default/album.html

@@ -48,37 +48,37 @@
 					</div>
 				</div>
 			<a class="menu_link_edit" id="edit_search_term" href="#"><i class="fa fa-pencil"></i> Edit Search Term</a>
-				<div id="dialog2" title="Enter your own search term for this album" style="display:none" class="configtable">
-					<form action="editSearchTerm" method="GET" id="editSearchTerm">
-						<input type="hidden" name="AlbumID" value="${album['AlbumID']}">
-						<div class="row">
-							<%
-								if not album['SearchTerm']:
-									search_term = ""
-								else:
-									search_term = album['SearchTerm']
-							%>
-							<input type="text" value="${search_term}" name="SearchTerm" size="40" />
-						</div>
-						<input type="button" value="Save changes" onclick="doAjaxCall('editSearchTerm',$(this),'tabs',true);return false;" data-success="Search term updated"/>
-					</form>
-				</div>
+			<div id="dialog2" title="Enter your own search term for this album" style="display:none" class="configtable">
+				<form action="editSearchTerm" method="GET" id="editSearchTerm">
+					<input type="hidden" name="AlbumID" value="${album['AlbumID']}">
+					<div class="row">
+						<%
+							if not album['SearchTerm']:
+								search_term = ""
+							else:
+								search_term = album['SearchTerm']
+						%>
+						<input type="text" value="${search_term}" name="SearchTerm" size="40" />
+					</div>
+					<input type="button" value="Save changes" onclick="doAjaxCall('editSearchTerm',$(this),'tabs',true);return false;" data-success="Search term updated"/>
+				</form>
+			</div>
 			<a class="menu_link_edit" id="choose_specific_download" href="#" onclick="getAvailableDownloads()"><i class="fa fa-search"></i> Choose Specific Download</a>
-				<div id="choose_specific_download_dialog" title="Choose a specific download for this album" style="display:none" class="configtable">
-						<table class="display" id="downloads_table">
-							<thead>
-								<tr>
-									<th id="title">Title</th>
-									<th id="size">Size</th>
-									<th id="provider">Provider</th>
-									<th id="kind">Kind</th>
-									<th id="matches" title="Matches quality and/or seed settings.">Matches</th>
-								</tr>
-							</thead>
-							<tbody id="downloads_table_body">
-							</tbody>
-						</table>
-				</div>
+			<div id="choose_specific_download_dialog" title="Choose a specific download for this album" style="display:none" class="configtable">
+				<table class="display" id="downloads_table">
+					<thead>
+						<tr>
+							<th id="title">Title</th>
+							<th id="size">Size</th>
+							<th id="provider">Provider</th>
+							<th id="kind">Kind</th>
+							<th id="matches" title="Matches quality and/or seed settings.">Matches</th>
+						</tr>
+					</thead>
+					<tbody id="downloads_table_body">
+					</tbody>
+				</table>
+			</div>
 		</div>
 	</div>
 </%def>
@@ -197,14 +197,20 @@
 		var elem = $("#albumInfo");
 		getInfo(elem,id,'album');
 	}
-	
+
 	function initThisPage() {
 		$('#album_chooser').click(function() {
-			$('#dialog').dialog({ width: "500px" });
+			$('#dialog').dialog({
+				width: 500,
+				maxHeight: 500
+			});
 			return false;
 		});
 		$('#edit_search_term').click(function() {
-			$('#dialog2').dialog({ width: "500px" });
+			$('#dialog2').dialog({
+				width: 500,
+				maxHeight: 500
+			});
 			return false;
 		});
 		$('#refresh_artist').click(function() {
@@ -213,8 +219,8 @@
 		initActions();
 		setTimeout(function(){
 			initFancybox();
-			},1000);
-		
+		}, 1000);
+
 		$('#track_table').dataTable({
 			"aaSorting": [],
 			"bFilter": false,
@@ -236,17 +242,21 @@
 			$('#downloads_table').dataTable({
 				"aoColumns": [
 					null,
-					{ "sType": "title-numeric"},
+					{"sType": "title-numeric"},
 					null,
 					null,
-					null
+					{"sType": "string"}
 				],
-				"aaSorting": [],
+				"aaSorting": [[ 4, 'desc']],
 				"bFilter": false,
 				"bInfo": false,
 				"bPaginate": false,
+				"bDestroy": true
 			});
-    		$("#choose_specific_download_dialog").dialog({ width: "1000px" });
+    		$("#choose_specific_download_dialog").dialog({
+    			width: "80%",
+    			maxHeight: 500
+    		});
     		return false;
 		});
 	}
@@ -327,6 +337,21 @@
         });
     }
 
+    jQuery.extend( jQuery.fn.dataTableExt.oSort, {
+		"title-numeric-pre": function ( a ) {
+			var x = a.match(/title="*(-?[0-9\.]+)/)[1];
+			return parseFloat( x );
+		},
+
+		"title-numeric-asc": function ( a, b ) {
+			return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+		},
+
+        "title-numeric-desc": function ( a, b ) {
+            return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+        }
+    } );
+
 	$(document).ready(function()	{
 		getAlbumInfo();
 		initThisPage();

data/interfaces/default/artist.html

@@ -10,6 +10,7 @@
 		<div id="subhead_menu">
 			<a id="menu_link_refresh" onclick="doSimpleAjaxCall('refreshArtist?ArtistID=${artist['ArtistID']}')" href="#"><i class="fa fa-refresh"></i> Refresh Artist</a>
 			<a id="menu_link_delete" href="deleteArtist?ArtistID=${artist['ArtistID']}"><i class="fa fa-trash-o"></i> Delete Artist</a>
+			<a id="menu_link_scan" onclick="doAjaxCall('scanArtist?ArtistID=${artist['ArtistID']}', $(this)),'table'" href="#" data-success="'${artist['ArtistName']}' was scanned"><i class="fa fa-refresh"></i> Scan Artist</a>
 			%if artist['Status'] == 'Paused':
 				<a id="menu_link_resume" href="#" onclick="doAjaxCall('resumeArtist?ArtistID=${artist['ArtistID']}',$(this),true)" data-success="${artist['ArtistName']} resumed"><i class="fa fa-play"></i> Resume Artist</a>
 			%else:
@@ -189,45 +190,45 @@
 		$.getJSON("https://api.songkick.com/api/3.0/artists/mbid:${artist['ArtistID']}/calendar.json?apikey=${headphones.CONFIG.SONGKICK_APIKEY}&jsoncallback=?",
 			function(data){
 				if (data['resultsPage'].totalEntries >= 1) {
-
-                    if( ${songkick_filter_enabled} ) {
+                    if (${songkick_filter_enabled}) {
                         data.resultsPage.results.event = $.grep(data.resultsPage.results.event, function(element,index){
-                        return element.venue.metroArea.id == ${songkick_location};
-                    });
+                    	    return element.venue.metroArea.id == ${songkick_location};
+                    	});
                     }
 
-					var tourDate;
+                    if (data.resultsPage.results.event.length > 0) {
+						var tourDate;
 
-					calendarDomNode.show();
-					$("#artistImg").addClass('on-tour');
+						calendarDomNode.show();
+						$("#artistImg").addClass('on-tour');
 
-					jQuery.each( data['resultsPage'].results.event, function( i, event ) {
-						tourDate = template;
-						tourDate = tourDate.replace('URI',event.uri);
-						tourDate = tourDate.replace('NAME',event.displayName);
-						tourDate = tourDate.replace('LOC',event.location.city);
+						jQuery.each(data.resultsPage.results.event, function(i, event) {
+							tourDate = template;
+							tourDate = tourDate.replace('URI',event.uri);
+							tourDate = tourDate.replace('NAME',event.displayName);
+							tourDate = tourDate.replace('LOC',event.location.city);
 
-						calendarDomNode.append(tourDate);
-					});
+							calendarDomNode.append(tourDate);
+						});
 
-					calendarDomNode.append('<li><img src="interfaces/default/images/songkick.png" alt="concerts by songkick" class="sk-logo" /></li>');
-
-				$(function() {
-				  $("#artistCalendar").each(function() {
-					$("li:gt(4)", this).hide(); /* :gt() is zero-indexed */
-					$("li:nth-child(5)", this).after("<br><li class='more'><a href='#'>More...</a></li>"); /* :nth-child() is one-indexed */
-				  });
-				  $("li.more a").live("click", function() {
-					var li = $(this).parents("li:first");
-					li.parent().children().show();
-					li.remove();
-					return false;
-				  });
-				});
+						calendarDomNode.append('<li><img src="interfaces/default/images/songkick.png" alt="concerts by songkick" class="sk-logo" /></li>');
 
+						$(function() {
+						  $("#artistCalendar").each(function() {
+							$("li:gt(4)", this).hide(); /* :gt() is zero-indexed */
+							$("li:nth-child(5)", this).after("<br><li class='more'><a href='#'>More...</a></li>"); /* :nth-child() is one-indexed */
+						  });
+						  $("li.more a").live("click", function() {
+							var li = $(this).parents("li:first");
+							li.parent().children().show();
+							li.remove();
+							return false;
+						  });
+						});
+					}
 				}
-			// data is JSON response object
-		});
+			}
+		);
 	}
 
 	var loadingMessage = false;

data/interfaces/default/config.html

@@ -138,7 +138,7 @@
                     </tr>
                     <tr>
                         <div class="configmessage">
-                            <i class="fa fa-info-circle"></i> Web Interface changes require a restart to take effect. Saving settings will restart intervals.
+                            <i class="fa fa-info-circle"></i> Web Interface changes require a restart to take effect. Saving settings will restart intervals if changed.
                         </div>
                     </tr>
                 </table>
@@ -786,6 +786,10 @@
                                         Embed lyrics
                                         <input type="checkbox" name="embed_lyrics" value="1" ${config['embed_lyrics']}>
                                     </label>
+                                    <label title="Use ID3v2.3 instead of ID3v2.4">
+                                        Tag using ID3v2.3
+                                        <input type="checkbox" name="idtag" value="1" ${config['idtag']}>
+                                    </label>
                                 </div>
                                 <div class="row">
                                     <label>
@@ -1104,6 +1108,39 @@
                                 </div>
                             </fieldset>
 
+                           <fieldset>
+                                <h3>Email</h3>
+                                <div class="row checkbox">
+                                    <input type="checkbox" name="email_enabled" id="email" value="1" ${config['email_enabled']} /><label>Enable Email Notifications</label>
+                                </div>
+                                <div id="email_options">
+                                    <div class="row">
+                                        <label>From</label><input type="text" name="email_from" value="${config['email_from']}" size="254">
+                                    </div>
+                                    <div class="row">
+                                        <label>To</label><input type="text" name="email_to" value="${config['email_to']}" size="254">
+                                    </div>
+                                    <div class="row">
+                                        <label>SMTP Server</label><input type="text" name="email_smtp_server" value="${config['email_smtp_server']}" size="254">
+                                    </div>
+                                    <div class="row">
+                                        <label>SMTP User</label><input type="text" name="email_smtp_user" value="${config['email_smtp_user']}" size="254">
+                                    </div>
+                                    <div class="row">
+                                        <label>SMTP Password</label><input type="password" name="email_smtp_password" value="${config['email_smtp_password']}" size="50">
+                                    </div>
+                                    <div class="row checkbox">
+                                        <input type="text" class="override-float" name="email_smtp_port" value="${config['email_smtp_port']}" size="4"><label>SMTP Port</label>
+                                    </div>
+                                    <div class="row checkbox">
+                                        <input type="checkbox" name="email_tls" value="1" ${config['email_tls']} /><label>TLS</label>
+                                    </div>
+                                    <div class="row checkbox">
+                                        <input type="checkbox" name="email_onsnatch" value="1" ${config['email_onsnatch']} /><label>Notify on snatch?</label>
+                                    </div>
+                                </div>
+                            </fieldset>
+
                             <fieldset>
                             </td>
                     </tr>
@@ -1416,6 +1453,17 @@
                                     <div class="row">
                                         <label>Port</label><input type="text" name="customport" value="${config['customport']}" size="8">
                                     </div>
+                                    <div class="row checkbox">
+                                        <input type="checkbox" name="customauth" id="customauth" value="1" ${config['customauth']} /><label>Requires Authentication</label>
+                                    </div>
+                                    <div id="customauth_options">
+                                        <div class="row">
+                                            <label>Username</label><input type="text" class="customuser" name="customuser" value="${config['customuser']}" size="20">
+                                        </div>
+                                        <div class="row">
+                                            <label>Password</label><input type="password" class="custompass" name="custompass" value="${config['custompass']}" size="15"><br>
+                                        </div>
+                                    </div>
                                     <div class="row">
                                         <label>Sleep Interval</label><input type="text" name="customsleep" value="${config['customsleep']}" size="4">
                                     </div>
@@ -1526,6 +1574,25 @@
                     $('#api_key').val(data);
             });
         });
+                if ($("#customauth").is(":checked"))
+                        {
+                            $("#customauth_options").show();
+                        }
+                else
+                        {
+                            $("#customauth_options").hide();
+                        }
+
+                $("#customauth").click(function(){
+                        if ($("#customauth").is(":checked"))
+                        {
+                            $("#customauth_options").slideDown();
+                        }
+                        else
+                        {
+                            $("#customauth_options").slideUp();
+                        }
+                });
                 if ($("#enable_https").is(":checked"))
                         {
                             $("#https_options").show();
@@ -1866,6 +1933,26 @@
                         }
                 });
 
+               if ($("#email").is(":checked"))
+                        {
+                            $("#email_options").show();
+                        }
+                else
+                        {
+                            $("#email_options").hide();
+                        }
+
+                $("#email").click(function(){
+                        if ($("#email").is(":checked"))
+                        {
+                            $("#email_options").slideDown();
+                        }
+                        else
+                        {
+                            $("#email_options").slideUp();
+                        }
+                });
+
                 if ($("#songkick").is(":checked"))
                         {
                             $("#songkickoptions").show();
@@ -2074,6 +2161,7 @@
                 initConfigCheckbox("#use_whatcd");
                 initConfigCheckbox("#api_enabled");
                 initConfigCheckbox("#enable_https");
+                initConfigCheckbox("#customauth");
 
 
                 $('#twitterStep1').click(function () {

data/interfaces/default/history.html

@@ -10,9 +10,10 @@
 			<a id="menu_link_delete" href="#" onclick="doAjaxCall('clearhistory?type=all',$(this),'table')" data-success="All History cleared"><i class="fa fa-trash-o"></i> Clear All History</a>
 			<a id="menu_link_delete" href="#" onclick="doAjaxCall('clearhistory?type=Processed',$(this),'table')" data-success="All Processed cleared"><i class="fa fa-trash-o"></i> Clear Processed</a>
 			<a id="menu_link_delete" href="#" onclick="doAjaxCall('clearhistory?type=Unprocessed',$(this),'table')" data-success="All Unprocessed cleared"><i class="fa fa-trash-o"></i> Clear Unprocessed</a>
+			<a id="menu_link_delete" href="#" onclick="doAjaxCall('clearhistory?type=Frozen',$(this),'table')" data-success="All Frozen cleared"><i class="fa fa-trash-o"></i> Clear Frozen</a>
 			<a id="menu_link_delete" href="#" onclick="doAjaxCall('clearhistory?type=Snatched',$(this),'table')" data-success="All Snatched cleared"><i class="fa fa-trash-o"></i> Clear Snatched</a>
 		</div>
-	</div>	
+	</div>
 </%def>
 
 <%def name="body()">
@@ -39,9 +40,11 @@
 					grade = 'C'
 				elif item['Status'] == 'Unprocessed':
 					grade = 'X'
+				elif item['Status'] == 'Frozen':
+					grade = 'X'
 				else:
 					grade = 'U'
-					
+
 				fileid = 'unknown'
 				if item['URL'].find('nzb') != -1:
 					fileid = 'nzb'

headphones/__init__.py

@@ -23,6 +23,7 @@ import threading
 import webbrowser
 import sqlite3
 import cherrypy
+import datetime
 
 from apscheduler.schedulers.background import BackgroundScheduler
 from apscheduler.triggers.interval import IntervalTrigger
@@ -262,51 +263,73 @@ def launch_browser(host, port, root):
 
 def initialize_scheduler():
     """
-    Start the scheduled background tasks. Because this method can be called
-    multiple times, the old tasks will be first removed.
+    Start the scheduled background tasks. Re-schedule if interval settings changed.
     """
 
     from headphones import updater, searcher, librarysync, postprocessor, \
         torrentfinished
 
     with SCHED_LOCK:
-        # Remove all jobs first, because this method is also invoked when the
-        # settings are saved.
-        count = len(SCHED.get_jobs())
 
-        if count > 0:
-            logger.debug("Current number of background tasks: %d", count)
-            SCHED.shutdown()
-            SCHED.remove_all_jobs()
+        # Check if scheduler should be started
+        start_jobs = not len(SCHED.get_jobs())
 
         # Regular jobs
-        if CONFIG.UPDATE_DB_INTERVAL > 0:
-            SCHED.add_job(updater.dbUpdate, trigger=IntervalTrigger(
-                hours=CONFIG.UPDATE_DB_INTERVAL))
-        if CONFIG.SEARCH_INTERVAL > 0:
-            SCHED.add_job(searcher.searchforalbum, trigger=IntervalTrigger(
-                minutes=CONFIG.SEARCH_INTERVAL))
-        if CONFIG.LIBRARYSCAN_INTERVAL > 0:
-            SCHED.add_job(librarysync.libraryScan, trigger=IntervalTrigger(
-                hours=CONFIG.LIBRARYSCAN_INTERVAL))
-        if CONFIG.DOWNLOAD_SCAN_INTERVAL > 0:
-            SCHED.add_job(postprocessor.checkFolder, trigger=IntervalTrigger(
-                minutes=CONFIG.DOWNLOAD_SCAN_INTERVAL))
+        minutes = CONFIG.SEARCH_INTERVAL
+        schedule_job(searcher.searchforalbum, 'Search for Wanted', hours=0, minutes=minutes)
 
-        # Update check
-        if CONFIG.CHECK_GITHUB:
-            SCHED.add_job(versioncheck.checkGithub, trigger=IntervalTrigger(
-                minutes=CONFIG.CHECK_GITHUB_INTERVAL))
+        minutes = CONFIG.DOWNLOAD_SCAN_INTERVAL
+        schedule_job(postprocessor.checkFolder, 'Download Scan', hours=0, minutes=minutes)
+
+        hours = CONFIG.LIBRARYSCAN_INTERVAL
+        schedule_job(librarysync.libraryScan, 'Library Scan', hours=hours, minutes=0)
+
+        hours = CONFIG.UPDATE_DB_INTERVAL
+        schedule_job(updater.dbUpdate, 'MusicBrainz Update', hours=hours, minutes=0)
+
+        #Update check
+        if CONFIG.CHECK_GITHUB_INTERVAL:
+            minutes = CONFIG.CHECK_GITHUB_INTERVAL
+        else:
+            minutes = 0
+        schedule_job(versioncheck.checkGithub, 'Check GitHub for updates', hours=0, minutes=minutes)
 
         # Remove Torrent + data if Post Processed and finished Seeding
-        if CONFIG.TORRENT_REMOVAL_INTERVAL > 0:
-            SCHED.add_job(torrentfinished.checkTorrentFinished,
-                          trigger=IntervalTrigger(
-                              minutes=CONFIG.TORRENT_REMOVAL_INTERVAL))
+        minutes = CONFIG.TORRENT_REMOVAL_INTERVAL
+        schedule_job(torrentfinished.checkTorrentFinished, 'Torrent removal check', hours=0, minutes=minutes)
 
         # Start scheduler
-        logger.info("(Re-)Scheduled %d background tasks", len(SCHED.get_jobs()))
-        SCHED.start()
+        if start_jobs and len(SCHED.get_jobs()):
+            try:
+                SCHED.start()
+            except Exception as e:
+                logger.info(e)
+
+        # Debug
+        #SCHED.print_jobs()
+
+
+def schedule_job(function, name, hours=0, minutes=0):
+    """
+    Start scheduled job if starting or restarting headphones.
+    Reschedule job if Interval Settings have changed.
+    Remove job if if Interval Settings changed to 0
+
+    """
+
+    job = SCHED.get_job(name)
+    if job:
+        if hours == 0 and minutes == 0:
+            SCHED.remove_job(name)
+            logger.info("Removed background task: %s", name)
+        elif job.trigger.interval != datetime.timedelta(hours=hours, minutes=minutes):
+            SCHED.reschedule_job(name, trigger=IntervalTrigger(
+                hours=hours, minutes=minutes))
+            logger.info("Re-scheduled background task: %s", name)
+    elif hours > 0 or minutes > 0:
+        SCHED.add_job(function, id=name, trigger=IntervalTrigger(
+            hours=hours, minutes=minutes))
+        logger.info("Scheduled background task: %s", name)
 
 
 def start():

headphones/api.py

@@ -13,16 +13,17 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import db, mb, importer, searcher, cache, postprocessor, versioncheck, logger
+from headphones import db, mb, updater, importer, searcher, cache, postprocessor, versioncheck, logger
 
 import headphones
 import json
 
-cmd_list = ['getIndex', 'getArtist', 'getAlbum', 'getUpcoming', 'getWanted', 'getSimilar', 'getHistory', 'getLogs',
+cmd_list = ['getIndex', 'getArtist', 'getAlbum', 'getUpcoming', 'getWanted', 'getSnatched', 'getSimilar', 'getHistory', 'getLogs',
             'findArtist', 'findAlbum', 'addArtist', 'delArtist', 'pauseArtist', 'resumeArtist', 'refreshArtist',
-            'addAlbum', 'queueAlbum', 'unqueueAlbum', 'forceSearch', 'forceProcess', 'getVersion', 'checkGithub',
-            'shutdown', 'restart', 'update', 'getArtistArt', 'getAlbumArt', 'getArtistInfo', 'getAlbumInfo',
-            'getArtistThumb', 'getAlbumThumb', 'choose_specific_download', 'download_specific_release']
+            'addAlbum', 'queueAlbum', 'unqueueAlbum', 'forceSearch', 'forceProcess', 'forceActiveArtistsUpdate', 
+            'getVersion', 'checkGithub','shutdown', 'restart', 'update', 'getArtistArt', 'getAlbumArt', 
+            'getArtistInfo', 'getAlbumInfo', 'getArtistThumb', 'getAlbumThumb', 
+            'choose_specific_download', 'download_specific_release']
 
 
 class Api(object):
@@ -165,6 +166,11 @@ class Api(object):
             "SELECT * from albums WHERE Status='Wanted'")
         return
 
+    def _getSnatched(self, **kwargs):
+        self.data = self._dic_from_query(
+            "SELECT * from albums WHERE Status='Snatched'")
+        return
+        
     def _getSimilar(self, **kwargs):
         self.data = self._dic_from_query('SELECT * from lastfmcloud')
         return
@@ -321,6 +327,9 @@ class Api(object):
             self.dir = kwargs['dir']
         postprocessor.forcePostProcess(self.dir)
 
+    def _forceActiveArtistsUpdate(self, **kwargs):
+        updater.dbUpdate()
+
     def _getVersion(self, **kwargs):
         self.data = {
             'git_path': headphones.CONFIG.GIT_PATH,

headphones/cache.py

@@ -23,20 +23,22 @@ LASTFM_API_KEY = "690e1ed3bc00bc91804cd8f7fe5ed6d4"
 
 class Cache(object):
     """
-    This class deals with getting, storing and serving up artwork (album
-    art, artist images, etc) and info/descriptions (album info, artist descrptions)
-    to and from the cache folder. This can be called from within a web interface,
-    for example, using the helper functions getInfo(id) and getArtwork(id), to utilize the cached
-    images rather than having to retrieve them every time the page is reloaded.
+    This class deals with getting, storing and serving up artwork (album art,
+    artist images, etc) and info/descriptions (album info, artist descrptions)
+    to and from the cache folder. This can be called from within a web
+    interface. For example, using the helper functions `getInfo(id)` and
+    `getArtwork(id)`, to utilize the cached images rather than having to
+    retrieve them every time the page is reloaded.
 
-    So you can call cache.getArtwork(id) which will return an absolute path
-    to the image file on the local machine, or if the cache directory
-    doesn't exist, or can not be written to, it will return a url to the image.
+    You can call `getArtwork(id)` which will return an absolute path to the
+    image file on the local machine, or if the cache directory doesn't exist,
+    or can not be written to, it will return a url to the image.
 
-    Call cache.getInfo(id) to grab the artist/album info; will return the text description
+    Call `getInfo(id)` to grab the artist/album info. This will return the
+    text description.
 
-    The basic format for art in the cache is <musicbrainzid>.<date>.<ext>
-    and for info it is <musicbrainzid>.<date>.txt
+    The basic format for art in the cache is `<musicbrainzid>.<date>.<ext>`
+    and for info it is `<musicbrainzid>.<date>.txt`
     """
 
     path_to_art_cache = os.path.join(headphones.CONFIG.CACHE_DIR, 'artwork')
@@ -197,10 +199,10 @@ class Cache(object):
             return info_dict
 
     def get_image_links(self, ArtistID=None, AlbumID=None):
-        '''
+        """
         Here we're just going to open up the last.fm url, grab the image links and return them
         Won't save any image urls, or save the artwork in the cache. Useful for search results, etc.
-        '''
+        """
         if ArtistID:
 
             self.id_type = 'artist'
@@ -271,9 +273,10 @@ class Cache(object):
                     logger.warn('Error deleting file from the cache: %s', thumb_file)
 
     def _update_cache(self):
-        '''
+        """
         Since we call the same url for both info and artwork, we'll update both at the same time
-        '''
+        """
+
         myDB = db.DBConnection()
 
         # Since lastfm uses release ids rather than release group ids for albums, we have to do a artist + album search for albums
@@ -306,7 +309,6 @@ class Cache(object):
                 logger.debug('No artist thumbnail image found')
 
         else:
-
             dbalbum = myDB.action('SELECT ArtistName, AlbumTitle, ReleaseID FROM albums WHERE AlbumID=?', [self.id]).fetchone()
             if dbalbum['ReleaseID'] != self.id:
                 data = lastfm.request_lastfm("album.getinfo", mbid=dbalbum['ReleaseID'], api_key=LASTFM_API_KEY)
@@ -339,7 +341,8 @@ class Cache(object):
             if not thumb_url:
                 logger.debug('No album thumbnail image found')
 
-        #Save the content & summary to the database no matter what if we've opened up the url
+        # Save the content & summary to the database no matter what if we've
+        # opened up the url
         if self.id_type == 'artist':
             controlValueDict = {"ArtistID": self.id}
         else:
@@ -365,7 +368,8 @@ class Cache(object):
             else:
                 myDB.action('UPDATE albums SET ThumbURL=? WHERE AlbumID=?', [thumb_url, self.id])
 
-        # Should we grab the artwork here if we're just grabbing thumbs or info?? Probably not since the files can be quite big
+        # Should we grab the artwork here if we're just grabbing thumbs or
+        # info? Probably not since the files can be quite big
         if image_url and self.query_type == 'artwork':
             artwork = request.request_content(image_url, timeout=20)
 
@@ -374,12 +378,13 @@ class Cache(object):
                 if not os.path.isdir(self.path_to_art_cache):
                     try:
                         os.makedirs(self.path_to_art_cache)
-                    except Exception as e:
+                        os.chmod(self.path_to_art_cache, int(headphones.CONFIG.FOLDER_PERMISSIONS, 8))
+                    except OSError as e:
                         logger.error('Unable to create artwork cache dir. Error: %s', e)
                         self.artwork_errors = True
                         self.artwork_url = image_url
 
-                #Delete the old stuff
+                # Delete the old stuff
                 for artwork_file in self.artwork_files:
                     try:
                         os.remove(artwork_file)
@@ -392,12 +397,15 @@ class Cache(object):
                 try:
                     with open(artwork_path, 'wb') as f:
                         f.write(artwork)
-                except IOError as e:
+
+                    os.chmod(artwork_path, int(headphones.CONFIG.FILE_PERMISSIONS, 8))
+                except (OSError, IOError) as e:
                     logger.error('Unable to write to the cache dir: %s', e)
                     self.artwork_errors = True
                     self.artwork_url = image_url
 
-        # Grab the thumbnail as well if we're getting the full artwork (as long as it's missing/outdated
+        # Grab the thumbnail as well if we're getting the full artwork (as long
+        # as it's missing/outdated.
         if thumb_url and self.query_type in ['thumb', 'artwork'] and not (self.thumb_files and self._is_current(self.thumb_files[0])):
             artwork = request.request_content(thumb_url, timeout=20)
 
@@ -406,16 +414,17 @@ class Cache(object):
                 if not os.path.isdir(self.path_to_art_cache):
                     try:
                         os.makedirs(self.path_to_art_cache)
-                    except Exception as e:
+                        os.chmod(self.path_to_art_cache, int(headphones.CONFIG.FOLDER_PERMISSIONS, 8))
+                    except OSError as e:
                         logger.error('Unable to create artwork cache dir. Error: %s' + e)
                         self.thumb_errors = True
                         self.thumb_url = thumb_url
 
-                #Delete the old stuff
+                # Delete the old stuff
                 for thumb_file in self.thumb_files:
                     try:
                         os.remove(thumb_file)
-                    except Exception as e:
+                    except OSError as e:
                         logger.error('Error deleting file from the cache: %s', thumb_file)
 
                 ext = os.path.splitext(image_url)[1]
@@ -424,14 +433,15 @@ class Cache(object):
                 try:
                     with open(thumb_path, 'wb') as f:
                         f.write(artwork)
-                except IOError as e:
+
+                    os.chmod(thumb_path, int(headphones.CONFIG.FILE_PERMISSIONS, 8))
+                except (OSError, IOError) as e:
                     logger.error('Unable to write to the cache dir: %s', e)
                     self.thumb_errors = True
                     self.thumb_url = image_url
 
 
 def getArtwork(ArtistID=None, AlbumID=None):
-
     c = Cache()
     artwork_path = c.get_artwork_from_cache(ArtistID, AlbumID)
 
@@ -446,7 +456,6 @@ def getArtwork(ArtistID=None, AlbumID=None):
 
 
 def getThumb(ArtistID=None, AlbumID=None):
-
     c = Cache()
     artwork_path = c.get_thumb_from_cache(ArtistID, AlbumID)
 
@@ -461,16 +470,13 @@ def getThumb(ArtistID=None, AlbumID=None):
 
 
 def getInfo(ArtistID=None, AlbumID=None):
-
     c = Cache()
-
     info_dict = c.get_info_from_cache(ArtistID, AlbumID)
 
     return info_dict
 
 
 def getImageLinks(ArtistID=None, AlbumID=None):
-
     c = Cache()
     image_links = c.get_image_links(ArtistID, AlbumID)
 

headphones/config.py

@@ -44,9 +44,12 @@ _CONFIG_DEFINITIONS = {
     'CUE_SPLIT': (int, 'General', 1),
     'CUE_SPLIT_FLAC_PATH': (str, 'General', ''),
     'CUE_SPLIT_SHNTOOL_PATH': (str, 'General', ''),
+    'CUSTOMAUTH': (int, 'General', 0),
     'CUSTOMHOST': (str, 'General', 'localhost'),
+    'CUSTOMPASS': (str, 'General', ''),
     'CUSTOMPORT': (int, 'General', 5000),
     'CUSTOMSLEEP': (int, 'General', 1),
+    'CUSTOMUSER': (str, 'General', ''),
     'DELETE_LOSSLESS_FILES': (int, 'General', 1),
     'DESTINATION_DIR': (str, 'General', ''),
     'DETECT_BITRATE': (int, 'General', 0),
@@ -54,6 +57,15 @@ _CONFIG_DEFINITIONS = {
     'DOWNLOAD_SCAN_INTERVAL': (int, 'General', 5),
     'DOWNLOAD_TORRENT_DIR': (str, 'General', ''),
     'DO_NOT_OVERRIDE_GIT_BRANCH': (int, 'General', 0),
+    'EMAIL_ENABLED': (int, 'Email', 0),
+    'EMAIL_FROM': (str, 'Email', ''),
+    'EMAIL_TO': (str, 'Email', ''),
+    'EMAIL_SMTP_SERVER': (str, 'Email', ''),
+    'EMAIL_SMTP_USER': (str, 'Email', ''),
+    'EMAIL_SMTP_PASSWORD': (str, 'Email', ''),
+    'EMAIL_SMTP_PORT': (int, 'Email', 25),
+    'EMAIL_TLS': (int, 'Email', 0),
+    'EMAIL_ONSNATCH': (int, 'Email', 0),
     'EMBED_ALBUM_ART': (int, 'General', 0),
     'EMBED_LYRICS': (int, 'General', 0),
     'ENABLE_HTTPS': (int, 'General', 0),
@@ -92,6 +104,7 @@ _CONFIG_DEFINITIONS = {
     'HTTP_PROXY': (int, 'General', 0),
     'HTTP_ROOT': (str, 'General', '/'),
     'HTTP_USERNAME': (str, 'General', ''),
+    'IDTAG': (int, 'Beets', 0),
     'IGNORED_WORDS': (str, 'General', ''),
     'IGNORED_FOLDERS': (list, 'Advanced', []),
     'IGNORED_FILES': (list, 'Advanced', []),

headphones/helpers.py

@@ -173,15 +173,15 @@ def piratesize(size):
     factor = float(split[0])
     unit = split[1].upper()
     
-    if unit == 'MiB':
+    if unit == 'MIB':
         size = factor * 1048576
     elif unit == 'MB':
         size = factor * 1000000
-    elif unit == 'GiB':
+    elif unit == 'GIB':
         size = factor * 1073741824
     elif unit == 'GB':
         size = factor * 1000000000
-    elif unit == 'KiB':
+    elif unit == 'KIB':
         size = factor * 1024
     elif unit == 'KB':
         size = factor * 1000

headphones/librarysync.py

@@ -64,20 +64,18 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
             encoded_track_string = track['Location'].encode(headphones.SYS_ENCODING, 'replace')
             if not os.path.isfile(encoded_track_string):
                 if track['ArtistName']:
-                    #Make sure deleted files get accounted for when updating artist track counts
+                    # Make sure deleted files get accounted for when updating artist track counts
                     new_artists.append(track['ArtistName'])
                 myDB.action('DELETE FROM have WHERE Location=?', [track['Location']])
                 logger.info('File %s removed from Headphones, as it is no longer on disk' % encoded_track_string.decode(headphones.SYS_ENCODING, 'replace'))
-           ###############myDB.action('DELETE from have')
 
     bitrates = []
-
     song_list = []
+    latest_subdirectory = []
+
     new_song_count = 0
     file_count = 0
 
-    latest_subdirectory = []
-
     for r, d, f in helpers.walk_directory(dir):
         # Filter paths based on config. Note that these methods work directly
         # on the inputs
@@ -313,7 +311,6 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
         ]
 
         # Update track counts
-
         for artist in artists_checked:
             # Have tracks are selected from tracks table and not all tracks because of duplicates
             # We update the track count upon an album switch to compliment this
@@ -321,13 +318,13 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
                 len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistName like ? AND Location IS NOT NULL', [artist]))
                 + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"', [artist]))
             )
-            #Note, some people complain about having "artist have tracks" > # of tracks total in artist official releases
+            # Note: some people complain about having "artist have tracks" > # of tracks total in artist official releases
             # (can fix by getting rid of second len statement)
             myDB.action('UPDATE artists SET HaveTracks=? WHERE ArtistName=?', [havetracks, artist])
 
         logger.info('Found %i new artists' % len(artist_list))
 
-        if len(artist_list):
+        if artist_list:
             if headphones.CONFIG.AUTO_ADD_ARTISTS:
                 logger.info('Importing %i new artists' % len(artist_list))
                 importer.artistlist_to_mbids(artist_list)

headphones/lock.py

@@ -39,12 +39,12 @@ class TimedLock(object):
         sleep_amount = self.minimum_delta - delta
         if sleep_amount >= 0:
             # zero sleeps give the cpu a chance to task-switch
-            headphones.logger.info('Sleeping %s (interval)', sleep_amount)
+            headphones.logger.debug('Sleeping %s (interval)', sleep_amount)
             time.sleep(sleep_amount)
         while not self.queue.empty():
             try:
                 seconds = self.queue.get(False)
-                headphones.logger.info('Sleeping %s (queued)', seconds)
+                headphones.logger.debug('Sleeping %s (queued)', seconds)
                 time.sleep(seconds)
             except Queue.Empty:
                 continue
@@ -59,15 +59,14 @@ class TimedLock(object):
 
     def snooze(self, seconds):
         """
-        Asynchronously add time to the next request.
-        Can be called outside
+        Asynchronously add time to the next request. Can be called outside
         of the lock context, but it is possible for the next lock holder
         to not check the queue until after something adds time to it.
         """
-        # we use a queue so that we don't have to synchronize
+        # We use a queue so that we don't have to synchronize
         # across threads and with or without locks
         headphones.logger.info('Adding %s to queue', seconds)
-        self.queue.add(seconds)
+        self.queue.put(seconds)
 
 
 class FakeLock(object):

headphones/mb.py

@@ -47,6 +47,8 @@ def startmb():
     elif headphones.CONFIG.MIRROR == "custom":
         mbhost = headphones.CONFIG.CUSTOMHOST
         mbport = int(headphones.CONFIG.CUSTOMPORT)
+        mbuser = headphones.CONFIG.CUSTOMUSER
+        mbpass = headphones.CONFIG.CUSTOMPASS
         sleepytime = int(headphones.CONFIG.CUSTOMSLEEP)
     elif headphones.CONFIG.MIRROR == "headphones":
         mbhost = "144.76.94.239"
@@ -69,12 +71,16 @@ def startmb():
         mb_lock.minimum_delta = sleepytime
 
     # Add headphones credentials
-    if headphones.CONFIG.MIRROR == "headphones":
-        if not mbuser and mbpass:
-            logger.warn("No username or password set for VIP server")
+    if headphones.CONFIG.MIRROR == "headphones" or headphones.CONFIG.CUSTOMAUTH:
+        if not mbuser or not mbpass:
+            logger.warn("No username or password set for MusicBrainz server")
         else:
             musicbrainzngs.hpauth(mbuser, mbpass)
 
+    # Let us know if we disable custom authentication
+    if not headphones.CONFIG.CUSTOMAUTH and headphones.CONFIG.MIRROR == "custom":
+        musicbrainzngs.disable_hpauth()
+
     logger.debug('Using the following server values: MBHost: %s, MBPort: %i, Sleep Interval: %i', mbhost, mbport, sleepytime)
 
     return True

headphones/notifiers.py

@@ -34,6 +34,11 @@ import json
 import oauth2 as oauth
 import pythontwitter as twitter
 
+from email.mime.text import MIMEText
+import smtplib
+import email.utils
+
+
 class GROWL(object):
     """
     Growl notifications, for OS X.
@@ -725,6 +730,7 @@ class OSX_NOTIFY(object):
     def __init__(self):
         try:
             self.objc = __import__("objc")
+            self.AppKit = __import__("AppKit")
         except:
             return False
 
@@ -737,7 +743,7 @@ class OSX_NOTIFY(object):
             signature=old_IMP.signature)
         self.objc.classAddMethod(cls, SEL, new_IMP)
 
-    def notify(self, title, subtitle=None, text=None, sound=True):
+    def notify(self, title, subtitle=None, text=None, sound=True, image=None):
 
         try:
             self.swizzle(self.objc.lookUpClass('NSBundle'),
@@ -761,6 +767,10 @@ class OSX_NOTIFY(object):
                 notification.setInformativeText_(text)
             if sound:
                 notification.setSoundName_("NSUserNotificationDefaultSoundName")
+            if image:
+                source_img = self.AppKit.NSImage.alloc().initByReferencingFile_(image)
+                notification.setContentImage_(source_img)
+                #notification.set_identityImage_(source_img)
             notification.setHasActionButton_(False)
 
             notification_center = NSUserNotificationCenter.defaultUserNotificationCenter()
@@ -822,3 +832,31 @@ class SubSonicNotifier(object):
         # Invoke request
         request.request_response(self.host + "musicFolderSettings.view?scanNow",
             auth=(self.username, self.password))
+
+class Email(object):
+
+    def notify(self, subject, message):
+
+        message = MIMEText(message, 'plain', "utf-8")
+        message['Subject'] = subject
+        message['From'] = email.utils.formataddr(('Headphones', headphones.CONFIG.EMAIL_FROM))
+        message['To'] = headphones.CONFIG.EMAIL_TO
+
+        try:
+            mailserver = smtplib.SMTP(headphones.CONFIG.EMAIL_SMTP_SERVER, headphones.CONFIG.EMAIL_SMTP_PORT)
+
+            if (headphones.CONFIG.EMAIL_TLS):
+                mailserver.starttls()
+
+            mailserver.ehlo()
+
+            if headphones.CONFIG.EMAIL_SMTP_USER:
+                mailserver.login(headphones.CONFIG.EMAIL_SMTP_USER, headphones.CONFIG.EMAIL_SMTP_PASSWORD)
+
+            mailserver.sendmail(headphones.CONFIG.EMAIL_FROM, headphones.CONFIG.EMAIL_TO, message.as_string())
+            mailserver.quit()
+            return True
+
+        except Exception, e:
+            logger.warn('Error sending Email: %s' % e)
+            return False

headphones/postprocessor.py

@@ -19,9 +19,11 @@ import shutil
 import uuid
 import beets
 import threading
+import itertools
 import headphones
 
 from beets import autotag
+from beets import config as beetsconfig
 from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
 from beetsplug import lyrics as beetslyrics
 
@@ -33,7 +35,7 @@ postprocessor_lock = threading.Lock()
 
 
 def checkFolder():
-    logger.info("Checking download folder for completed downloads")
+    logger.debug("Checking download folder for completed downloads (only snatched ones).")
 
     with postprocessor_lock:
         myDB = db.DBConnection()
@@ -55,7 +57,7 @@ def checkFolder():
             else:
                 logger.info("No folder name found for " + album['Title'])
 
-    logger.info("Checking download folder finished")
+    logger.debug("Checking download folder finished.")
 
 def verify(albumid, albumpath, Kind=None, forced=False):
 
@@ -98,6 +100,11 @@ def verify(albumid, albumpath, Kind=None, forced=False):
                     "but database is frozen. Will skip postprocessing for " \
                     "album with rgid: %s", release_dict['artist_name'],
                     release_dict['artist_id'], albumid)
+
+                myDB.action('UPDATE snatched SET status = "Frozen" WHERE status NOT LIKE "Seed%" and AlbumID=?', [albumid])
+                frozen = re.search(r' \(Frozen\)(?:\[\d+\])?', albumpath)
+                if not frozen:
+                    renameUnprocessedFolder(albumpath, tag="Frozen")
                 return
 
         logger.info(u"Now adding/updating artist: " + release_dict['artist_name'])
@@ -188,7 +195,7 @@ def verify(albumid, albumpath, Kind=None, forced=False):
             myDB.action('UPDATE snatched SET status = "Unprocessed" WHERE status NOT LIKE "Seed%" and AlbumID=?', [albumid])
             processed = re.search(r' \(Unprocessed\)(?:\[\d+\])?', albumpath)
             if not processed:
-                renameUnprocessedFolder(albumpath)
+                renameUnprocessedFolder(albumpath, tag="Unprocessed")
             return
 
     # test #1: metadata - usually works
@@ -272,9 +279,7 @@ def verify(albumid, albumpath, Kind=None, forced=False):
     myDB.action('UPDATE snatched SET status = "Unprocessed" WHERE status NOT LIKE "Seed%" and AlbumID=?', [albumid])
     processed = re.search(r' \(Unprocessed\)(?:\[\d+\])?', albumpath)
     if not processed:
-        renameUnprocessedFolder(albumpath)
-    else:
-        logger.info(u"Already marked as unprocessed: " + albumpath.decode(headphones.SYS_ENCODING, 'replace'))
+        renameUnprocessedFolder(albumpath, tag="Unprocessed")
 
 
 def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list, Kind=None):
@@ -484,11 +489,15 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
         twitter.notify_download(pushmessage)
 
     if headphones.CONFIG.OSX_NOTIFY_ENABLED:
+        from headphones import cache
+        c = cache.Cache()
+        album_art = c.get_artwork_from_cache(None, release['AlbumID'])
         logger.info(u"Sending OS X notification")
         osx_notify = notifiers.OSX_NOTIFY()
         osx_notify.notify(release['ArtistName'],
                           release['AlbumTitle'],
-                          statusmessage)
+                          statusmessage,
+                          image=album_art)
 
     if headphones.CONFIG.BOXCAR_ENABLED:
         logger.info(u"Sending Boxcar2 notification")
@@ -505,6 +514,12 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
         mpc = notifiers.MPC()
         mpc.notify()
 
+    if headphones.CONFIG.EMAIL_ENABLED:
+        logger.info(u"Sending Email notification")
+        email = notifiers.Email()
+        subject = release['ArtistName'] + ' - ' + release['AlbumTitle']
+        email.notify(subject, "Download and Postprocessing completed")
+
 
 def embedAlbumArt(artwork, downloaded_track_list):
     logger.info('Embedding album art')
@@ -850,7 +865,7 @@ def correctMetadata(albumid, release, downloaded_track_list):
         except Exception as e:
             logger.error('Error getting recommendation: %s. Not writing metadata', e)
             return
-        if str(rec) == 'recommendation.none':
+        if str(rec) == 'Recommendation.none':
             logger.warn('No accurate album match found for %s, %s -  not writing metadata', release['ArtistName'], release['AlbumTitle'])
             return
 
@@ -865,6 +880,14 @@ def correctMetadata(albumid, release, downloaded_track_list):
         # TODO: Handle extra_items & extra_tracks
 
         autotag.apply_metadata(info, mapping)
+        
+        # Set ID3 tag version
+        if headphones.CONFIG.IDTAG:
+            beetsconfig['id3v23'] = True
+            logger.debug("Using ID3v2.3")
+        else:
+            beetsconfig['id3v23'] = False
+            logger.debug("Using ID3v2.4")
 
         for item in items:
             try:
@@ -1024,25 +1047,29 @@ def updateFilePermissions(albumpaths):
                     continue
 
 
-def renameUnprocessedFolder(albumpath):
+def renameUnprocessedFolder(path, tag):
+    """
+    Rename a unprocessed folder to a new unique name to indicate a certain
+    status.
+    """
 
-    i = 0
-    while True:
+    for i in itertools.count():
         if i == 0:
-            new_folder_name = albumpath + ' (Unprocessed)'
+            new_path = "%s (%s)" % (path, tag)
         else:
-            new_folder_name = albumpath + ' (Unprocessed)[%i]' % i
+            new_path = "%s (%s[%d])" % (path, tag, i)
 
-        if os.path.exists(new_folder_name):
+        if os.path.exists(new_path):
             i += 1
-
         else:
-            os.rename(albumpath, new_folder_name)
+            os.rename(path, new_path)
             return
 
 
 def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
 
+    logger.info('Force checking download folder for completed downloads')
+	
     ignored = 0
 
     if album_dir:

headphones/request.py

@@ -30,10 +30,10 @@ fake_lock = headphones.lock.FakeLock()
 
 
 def request_response(url, method="get", auto_raise=True,
-    whitelist_status_code=None, lock=fake_lock, **kwargs):
+                     whitelist_status_code=None, lock=fake_lock, **kwargs):
     """
-    Convenient wrapper for `requests.get', which will capture the exceptions and
-    log them. On success, the Response object is returned. In case of a
+    Convenient wrapper for `requests.get', which will capture the exceptions
+    and log them. On success, the Response object is returned. In case of a
     exception, None is returned.
 
     Additionally, there is support for rate limiting. To use this feature,
@@ -51,14 +51,15 @@ def request_response(url, method="get", auto_raise=True,
     # pose a security issue!
     kwargs["verify"] = bool(headphones.CONFIG.VERIFY_SSL_CERT)
 
-    # Map method to the request.XXX method. This is a simple hack, but it allows
-    # requests to apply more magic per method. See lib/requests/api.py.
+    # Map method to the request.XXX method. This is a simple hack, but it
+    # allows requests to apply more magic per method. See lib/requests/api.py.
     request_method = getattr(requests, method.lower())
 
     try:
         # Request URL and wait for response
         with lock:
-            logger.debug("Requesting URL via %s method: %s", method.upper(), url)
+            logger.debug(
+                "Requesting URL via %s method: %s", method.upper(), url)
             response = request_method(url, **kwargs)
 
         # If status code != OK, then raise exception, except if the status code
@@ -76,10 +77,18 @@ def request_response(url, method="get", auto_raise=True,
             response.raise_for_status()
 
         return response
-    except requests.exceptions.SSLError:
-        logger.error("Unable to connect to remote host because of a SSL " \
-            "error. It's likely the remote certificate is untrusted by your " \
-            "system. This check can be disabled (advanced users only).")
+    except requests.exceptions.SSLError as e:
+        if kwargs["verify"]:
+            logger.error(
+                "Unable to connect to remote host because of a SSL error. "
+                "It is likely that your system cannot verify the validity"
+                "of the certificate. The remote certificate is either "
+                "self-signed, or the remote server uses SNI. See the wiki for "
+                "more information on this topic.")
+        else:
+            logger.error(
+                "SSL error raised during connection, with certificate "
+                "verification turned off: %s", e)
     except requests.ConnectionError:
         logger.error(
             "Unable to connect to remote host. Check if the remote "
@@ -97,7 +106,8 @@ def request_response(url, method="get", auto_raise=True,
                 # I don't think we will end up here, but for completeness
                 cause = "unknown"
 
-            logger.error("Request raise HTTP error with status code %d (%s).",
+            logger.error(
+                "Request raise HTTP error with status code %d (%s).",
                 e.response.status_code, cause)
 
             # Debug response
@@ -139,8 +149,8 @@ def request_json(url, **kwargs):
     Wrapper for `request_response', which will decode the response as JSON
     object and return the result, if no exceptions are raised.
 
-    As an option, a validator callback can be given, which should return True if
-    the result is valid.
+    As an option, a validator callback can be given, which should return True
+    if the result is valid.
     """
 
     validator = kwargs.pop("validator", None)
@@ -205,9 +215,9 @@ def server_message(response):
         # Find body and cleanup common tags to grab content, which probably
         # contains the message.
         message = soup.find("body")
+        elements = ("header", "script", "footer", "nav", "input", "textarea")
 
-        for element in ["header", "script", "footer", "nav", "input",
-            "textarea"]:
+        for element in elements:
 
             for tag in soup.find_all(element):
                 tag.replaceWith("")

headphones/searcher.py

@@ -225,7 +225,7 @@ def do_sorted_search(album, new, losslessOnly, choose_specific_download=False):
     myDB = db.DBConnection()
     albumlength = myDB.select('SELECT sum(TrackDuration) from tracks WHERE AlbumID=?', [album['AlbumID']])[0][0]
 
-    if headphones.CONFIG.PREFER_TORRENTS == 0:
+    if headphones.CONFIG.PREFER_TORRENTS == 0 and not choose_specific_download:
 
         if NZB_PROVIDERS and NZB_DOWNLOADERS:
             results = searchNZB(album, new, losslessOnly, albumlength)
@@ -233,7 +233,7 @@ def do_sorted_search(album, new, losslessOnly, choose_specific_download=False):
         if not results and TORRENT_PROVIDERS:
             results = searchTorrent(album, new, losslessOnly, albumlength)
 
-    elif headphones.CONFIG.PREFER_TORRENTS == 1:
+    elif headphones.CONFIG.PREFER_TORRENTS == 1 and not choose_specific_download:
 
         if TORRENT_PROVIDERS:
             results = searchTorrent(album, new, losslessOnly, albumlength)
@@ -247,10 +247,10 @@ def do_sorted_search(album, new, losslessOnly, choose_specific_download=False):
         torrent_results = None
 
         if NZB_PROVIDERS and NZB_DOWNLOADERS:
-            nzb_results = searchNZB(album, new, losslessOnly, albumlength)
+            nzb_results = searchNZB(album, new, losslessOnly, albumlength, choose_specific_download)
 
         if TORRENT_PROVIDERS:
-            torrent_results = searchTorrent(album, new, losslessOnly, albumlength)
+            torrent_results = searchTorrent(album, new, losslessOnly, albumlength, choose_specific_download)
 
         if not nzb_results:
             nzb_results = []
@@ -306,9 +306,9 @@ def more_filtering(results, album, albumlength, new):
             targetsize = albumlength / 1000 * int(headphones.CONFIG.PREFERRED_BITRATE) * 128
             logger.info('Target size: %s' % helpers.bytes_to_mb(targetsize))
             if headphones.CONFIG.PREFERRED_BITRATE_LOW_BUFFER:
-                low_size_limit = targetsize - (targetsize * int(headphones.CONFIG.PREFERRED_BITRATE_LOW_BUFFER) / 100)
+                low_size_limit =  targetsize * int(headphones.CONFIG.PREFERRED_BITRATE_LOW_BUFFER) / 100
             if headphones.CONFIG.PREFERRED_BITRATE_HIGH_BUFFER:
-                high_size_limit = targetsize + (targetsize * int(headphones.CONFIG.PREFERRED_BITRATE_HIGH_BUFFER) / 100)
+                high_size_limit = targetsize * int(headphones.CONFIG.PREFERRED_BITRATE_HIGH_BUFFER) / 100
                 if headphones.CONFIG.PREFERRED_BITRATE_ALLOW_LOSSLESS:
                     allow_lossless = True
 
@@ -429,7 +429,7 @@ def get_year_from_release_date(release_date):
     return year
 
 
-def searchNZB(album, new=False, losslessOnly=False, albumlength=None):
+def searchNZB(album, new=False, losslessOnly=False, albumlength=None, choose_specific_download=False):
     reldate = album['ReleaseDate']
     year = get_year_from_release_date(reldate)
 
@@ -691,7 +691,7 @@ def searchNZB(album, new=False, losslessOnly=False, albumlength=None):
     results = [result for result in resultlist if verifyresult(result[0], artistterm, term, losslessOnly)]
 
     # Additional filtering for size etc
-    if results:
+    if results and not choose_specific_download:
         results = more_filtering(results, album, albumlength, new)
 
     return results
@@ -932,15 +932,25 @@ def send_to_downloader(data, bestqual, album):
         pushalot = notifiers.PUSHALOT()
         pushalot.notify(name, "Download started")
     if headphones.CONFIG.OSX_NOTIFY_ENABLED and headphones.CONFIG.OSX_NOTIFY_ONSNATCH:
+        from headphones import cache
+        c = cache.Cache()
+        album_art = c.get_artwork_from_cache(None, rgid)
         logger.info(u"Sending OS X notification")
         osx_notify = notifiers.OSX_NOTIFY()
-        osx_notify.notify(artist, albumname, 'Snatched: ' + provider + '. ' + name)
+        osx_notify.notify(artist,
+                          albumname,
+                          'Snatched: ' + provider + '. ' + name,
+                          image=album_art)
     if headphones.CONFIG.BOXCAR_ENABLED and headphones.CONFIG.BOXCAR_ONSNATCH:
         logger.info(u"Sending Boxcar2 notification")
         b2msg = 'From ' + provider + '<br></br>' + name
         boxcar = notifiers.BOXCAR()
         boxcar.notify('Headphones snatched: ' + title, b2msg, rgid)
-
+    if headphones.CONFIG.EMAIL_ENABLED and headphones.CONFIG.EMAIL_ONSNATCH:
+        logger.info(u"Sending Email notification")
+        email = notifiers.Email()
+        message = 'Snatched from ' + provider + '. ' + name
+        email.notify(title, message)
 
 def verifyresult(title, artistterm, term, lossless):
 
@@ -1008,7 +1018,7 @@ def verifyresult(title, artistterm, term, lossless):
     return True
 
 
-def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
+def searchTorrent(album, new=False, losslessOnly=False, albumlength=None, choose_specific_download=False):
     global gazelle  # persistent what.cd api object to reduce number of login attempts
 
     # rutracker login
@@ -1086,7 +1096,7 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
         if headphones.CONFIG.KAT_PROXY_URL:
             providerurl = fix_url(set_proxy(headphones.CONFIG.KAT_PROXY_URL))
         else:
-            providerurl = fix_url("https://kickass.so")
+            providerurl = fix_url("https://kickass.to")
 
         # Build URL
         providerurl = providerurl + "/usearch/" + ka_term
@@ -1515,7 +1525,7 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
     results = [result for result in resultlist if verifyresult(result[0], artistterm, term, losslessOnly)]
 
     # Additional filtering for size etc
-    if results:
+    if results and not choose_specific_download:
         results = more_filtering(results, album, albumlength, new)
 
     return results

headphones/webserve.py

@@ -254,6 +254,18 @@ class WebInterface(object):
         self.removeArtist(ArtistID)
         raise cherrypy.HTTPRedirect("home")
 
+    @cherrypy.expose
+    def scanArtist(self, ArtistID):
+        logger.info(u"Scanning artist: %s", ArtistID)
+        myDB = db.DBConnection()
+        artistname = myDB.select('SELECT DISTINCT ArtistName FROM artists WHERE ArtistID=?', [ArtistID])
+        artistfolder = os.path.join(headphones.CONFIG.DESTINATION_DIR, artistname[0][0])
+        try:
+            threading.Thread(target=librarysync.libraryScan(dir=artistfolder)).start()
+        except Exception as e:
+            logger.error('Unable to complete the scan: %s', e)
+        raise cherrypy.HTTPRedirect("artistPage?ArtistID=%s" % ArtistID)
+
     @cherrypy.expose
     def deleteEmptyArtists(self):
         logger.info(u"Deleting all empty artists")
@@ -362,6 +374,9 @@ class WebInterface(object):
           myDB = db.DBConnection()
           album = myDB.action('SELECT * from albums WHERE AlbumID=?', [AlbumID]).fetchone()
           searcher.send_to_downloader(data, bestqual, album)
+          return json.dumps({'result':'success'})
+        else:
+          return json.dumps({'result':'failure'})
 
     @cherrypy.expose
     def unqueueAlbum(self, AlbumID, ArtistID):
@@ -865,6 +880,7 @@ class WebInterface(object):
         cherrypy.response.headers['Content-type'] = 'application/json'
         return json_albums
 
+    @cherrypy.expose
     def getArtistjson(self, ArtistID, **kwargs):
         myDB = db.DBConnection()
         artist = myDB.action('SELECT * FROM artists WHERE ArtistID=?', [ArtistID]).fetchone()
@@ -1130,6 +1146,9 @@ class WebInterface(object):
             "customhost": headphones.CONFIG.CUSTOMHOST,
             "customport": headphones.CONFIG.CUSTOMPORT,
             "customsleep": headphones.CONFIG.CUSTOMSLEEP,
+            "customauth": checked(headphones.CONFIG.CUSTOMAUTH),
+            "customuser": headphones.CONFIG.CUSTOMUSER,
+            "custompass": headphones.CONFIG.CUSTOMPASS,
             "hpuser": headphones.CONFIG.HPUSER,
             "hppass": headphones.CONFIG.HPPASS,
             "songkick_enabled": checked(headphones.CONFIG.SONGKICK_ENABLED),
@@ -1139,7 +1158,17 @@ class WebInterface(object):
             "cache_sizemb": headphones.CONFIG.CACHE_SIZEMB,
             "file_permissions": headphones.CONFIG.FILE_PERMISSIONS,
             "folder_permissions": headphones.CONFIG.FOLDER_PERMISSIONS,
-            "mpc_enabled": checked(headphones.CONFIG.MPC_ENABLED)
+            "mpc_enabled": checked(headphones.CONFIG.MPC_ENABLED),
+            "email_enabled": checked(headphones.CONFIG.EMAIL_ENABLED),
+            "email_from": headphones.CONFIG.EMAIL_FROM,
+            "email_to": headphones.CONFIG.EMAIL_TO,
+            "email_smtp_server": headphones.CONFIG.EMAIL_SMTP_SERVER,
+            "email_smtp_user": headphones.CONFIG.EMAIL_SMTP_USER,
+            "email_smtp_password": headphones.CONFIG.EMAIL_SMTP_PASSWORD,
+            "email_smtp_port": int(headphones.CONFIG.EMAIL_SMTP_PORT),
+            "email_tls": checked(headphones.CONFIG.EMAIL_TLS),
+            "email_onsnatch": checked(headphones.CONFIG.EMAIL_ONSNATCH),
+            "idtag": checked(headphones.CONFIG.IDTAG)
         }
 
         # Need to convert EXTRAS to a dictionary we can pass to the config:
@@ -1184,7 +1213,7 @@ class WebInterface(object):
             "nma_enabled", "nma_onsnatch", "pushalot_enabled", "pushalot_onsnatch", "synoindex_enabled", "pushover_enabled",
             "pushover_onsnatch", "pushbullet_enabled", "pushbullet_onsnatch", "subsonic_enabled", "twitter_enabled", "twitter_onsnatch",
             "osx_notify_enabled", "osx_notify_onsnatch", "boxcar_enabled", "boxcar_onsnatch", "songkick_enabled", "songkick_filter_enabled",
-            "mpc_enabled"
+            "mpc_enabled", "email_enabled", "email_tls", "email_onsnatch", "customauth", "idtag"
         ]
         for checked_config in checked_configs:
             if checked_config not in kwargs:

lib/beets/LICENSE

@@ -0,0 +1,21 @@
+The MIT License
+
+Copyright (c) 2010-2014 Adrian Sampson
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

lib/beets/README.rst

@@ -0,0 +1,94 @@
+.. image:: https://travis-ci.org/sampsyo/beets.svg?branch=master
+    :target: https://travis-ci.org/sampsyo/beets
+
+.. image:: http://img.shields.io/coveralls/sampsyo/beets.svg
+    :target: https://coveralls.io/r/sampsyo/beets
+
+.. image:: http://img.shields.io/pypi/v/beets.svg
+    :target: https://pypi.python.org/pypi/beets
+
+Beets is the media library management system for obsessive-compulsive music
+geeks.
+
+The purpose of beets is to get your music collection right once and for all.
+It catalogs your collection, automatically improving its metadata as it goes.
+It then provides a bouquet of tools for manipulating and accessing your music.
+
+Here's an example of beets' brainy tag corrector doing its thing::
+
+  $ beet import ~/music/ladytron
+  Tagging:
+      Ladytron - Witching Hour
+  (Similarity: 98.4%)
+   * Last One Standing      -> The Last One Standing
+   * Beauty                 -> Beauty*2
+   * White Light Generation -> Whitelightgenerator
+   * All the Way            -> All the Way...
+
+Because beets is designed as a library, it can do almost anything you can
+imagine for your music collection. Via `plugins`_, beets becomes a panacea:
+
+- Fetch or calculate all the metadata you could possibly need: `album art`_,
+  `lyrics`_, `genres`_, `tempos`_, `ReplayGain`_ levels, or `acoustic
+  fingerprints`_.
+- Get metadata from `MusicBrainz`_, `Discogs`_, or `Beatport`_. Or guess
+  metadata using songs' filenames or their acoustic fingerprints.
+- `Transcode audio`_ to any format you like.
+- Check your library for `duplicate tracks and albums`_ or for `albums that
+  are missing tracks`_.
+- Clean up crufty tags left behind by other, less-awesome tools.
+- Embed and extract album art from files' metadata.
+- Browse your music library graphically through a Web browser and play it in any
+  browser that supports `HTML5 Audio`_.
+- Analyze music files' metadata from the command line.
+- Listen to your library with a music player that speaks the `MPD`_ protocol
+  and works with a staggering variety of interfaces.
+
+If beets doesn't do what you want yet, `writing your own plugin`_ is
+shockingly simple if you know a little Python.
+
+.. _plugins: http://beets.readthedocs.org/page/plugins/
+.. _MPD: http://www.musicpd.org/
+.. _MusicBrainz music collection: http://musicbrainz.org/doc/Collections/
+.. _writing your own plugin:
+    http://beets.readthedocs.org/page/dev/plugins.html
+.. _HTML5 Audio:
+    http://www.w3.org/TR/html-markup/audio.html
+.. _albums that are missing tracks:
+    http://beets.readthedocs.org/page/plugins/missing.html
+.. _duplicate tracks and albums:
+    http://beets.readthedocs.org/page/plugins/duplicates.html
+.. _Transcode audio:
+    http://beets.readthedocs.org/page/plugins/convert.html
+.. _Beatport: http://www.beatport.com/
+.. _Discogs: http://www.discogs.com/
+.. _acoustic fingerprints:
+    http://beets.readthedocs.org/page/plugins/chroma.html
+.. _ReplayGain: http://beets.readthedocs.org/page/plugins/replaygain.html
+.. _tempos: http://beets.readthedocs.org/page/plugins/echonest.html
+.. _genres: http://beets.readthedocs.org/page/plugins/lastgenre.html
+.. _album art: http://beets.readthedocs.org/page/plugins/fetchart.html
+.. _lyrics: http://beets.readthedocs.org/page/plugins/lyrics.html
+.. _MusicBrainz: http://musicbrainz.org/
+
+Read More
+---------
+
+Learn more about beets at `its Web site`_. Follow `@b33ts`_ on Twitter for
+news and updates.
+
+You can install beets by typing ``pip install beets``. Then check out the
+`Getting Started`_ guide.
+
+.. _its Web site: http://beets.radbox.org/
+.. _Getting Started: http://beets.readthedocs.org/page/guides/main.html
+.. _@b33ts: http://twitter.com/b33ts/
+
+Authors
+-------
+
+Beets is by `Adrian Sampson`_ with a supporting cast of thousands. For help,
+please contact the `mailing list`_.
+
+.. _mailing list: https://groups.google.com/forum/#!forum/beets-users
+.. _Adrian Sampson: http://homes.cs.washington.edu/~asampson/

lib/beets/__init__.py

@@ -1,5 +1,5 @@
 # This file is part of beets.
-# Copyright 2013, Adrian Sampson.
+# Copyright 2014, Adrian Sampson.
 #
 # Permission is hereby granted, free of charge, to any person obtaining
 # a copy of this software and associated documentation files (the
@@ -12,13 +12,14 @@
 # The above copyright notice and this permission notice shall be
 # included in all copies or substantial portions of the Software.
 
-# This particular version has been slightly modified to work with headphones
+# This particular version has been slightly modified to work with Headphones
 # https://github.com/rembo10/headphones
-import os
 
-__version__ = '1.3.4'
+__version__ = '1.3.10-headphones'
 __author__ = 'Adrian Sampson <adrian@radbox.org>'
 
+import os
+
 import beets.library
 from beets.util import confit
 

lib/beets/autotag/__init__.py

@@ -14,135 +14,21 @@
 
 """Facilities for automatically determining files' correct metadata.
 """
-import os
 import logging
-import re
 
-from beets import library, mediafile, config
-from beets.util import sorted_walk, ancestry, displayable_path
+from beets import config
 
 # Parts of external interface.
-from .hooks import AlbumInfo, TrackInfo, AlbumMatch, TrackMatch
-from .match import tag_item, tag_album
-from .match import recommendation
+from .hooks import AlbumInfo, TrackInfo, AlbumMatch, TrackMatch  # noqa
+from .match import tag_item, tag_album  # noqa
+from .match import Recommendation  # noqa
 
 # Global logger.
 log = logging.getLogger('beets')
 
-# Constants for directory walker.
-MULTIDISC_MARKERS = (r'dis[ck]', r'cd')
-MULTIDISC_PAT_FMT = r'^(.*%s[\W_]*)\d'
-
 
 # Additional utilities for the main interface.
 
-def albums_in_dir(path):
-    """Recursively searches the given directory and returns an iterable
-    of (paths, items) where paths is a list of directories and items is
-    a list of Items that is probably an album. Specifically, any folder
-    containing any media files is an album.
-    """
-    collapse_pat = collapse_paths = collapse_items = None
-
-    for root, dirs, files in sorted_walk(path,
-                                         ignore=config['ignore'].as_str_seq(),
-                                         logger=log):
-        # Get a list of items in the directory.
-        items = []
-        for filename in files:
-            try:
-                i = library.Item.from_path(os.path.join(root, filename))
-            except mediafile.FileTypeError:
-                pass
-            except mediafile.UnreadableFileError:
-                log.warn(u'unreadable file: {0}'.format(
-                    displayable_path(filename))
-                )
-            else:
-                items.append(i)
-
-        # If we're currently collapsing the constituent directories in a
-        # multi-disc album, check whether we should continue collapsing
-        # and add the current directory. If so, just add the directory
-        # and move on to the next directory. If not, stop collapsing.
-        if collapse_paths:
-            if (not collapse_pat and collapse_paths[0] in ancestry(root)) or \
-                    (collapse_pat and
-                     collapse_pat.match(os.path.basename(root))):
-                # Still collapsing.
-                collapse_paths.append(root)
-                collapse_items += items
-                continue
-            else:
-                # Collapse finished. Yield the collapsed directory and
-                # proceed to process the current one.
-                if collapse_items:
-                    yield collapse_paths, collapse_items
-                collapse_pat = collapse_paths = collapse_items = None
-
-        # Check whether this directory looks like the *first* directory
-        # in a multi-disc sequence. There are two indicators: the file
-        # is named like part of a multi-disc sequence (e.g., "Title Disc
-        # 1") or it contains no items but only directories that are
-        # named in this way.
-        start_collapsing = False
-        for marker in MULTIDISC_MARKERS:
-            marker_pat = re.compile(MULTIDISC_PAT_FMT % marker, re.I)
-            match = marker_pat.match(os.path.basename(root))
-
-            # Is this directory the root of a nested multi-disc album?
-            if dirs and not items:
-                # Check whether all subdirectories have the same prefix.
-                start_collapsing = True
-                subdir_pat = None
-                for subdir in dirs:
-                    # The first directory dictates the pattern for
-                    # the remaining directories.
-                    if not subdir_pat:
-                        match = marker_pat.match(subdir)
-                        if match:
-                            subdir_pat = re.compile(r'^%s\d' %
-                                re.escape(match.group(1)), re.I)
-                        else:
-                            start_collapsing = False
-                            break
-
-                    # Subsequent directories must match the pattern.
-                    elif not subdir_pat.match(subdir):
-                        start_collapsing = False
-                        break
-
-                # If all subdirectories match, don't check other
-                # markers.
-                if start_collapsing:
-                    break
-
-            # Is this directory the first in a flattened multi-disc album?
-            elif match:
-                start_collapsing = True
-                # Set the current pattern to match directories with the same
-                # prefix as this one, followed by a digit.
-                collapse_pat = re.compile(r'^%s\d' %
-                    re.escape(match.group(1)), re.I)
-                break
-
-        # If either of the above heuristics indicated that this is the
-        # beginning of a multi-disc album, initialize the collapsed
-        # directory and item lists and check the next directory.
-        if start_collapsing:
-            # Start collapsing; continue to the next iteration.
-            collapse_paths = [root]
-            collapse_items = items
-            continue
-
-        # If it's nonempty, yield it.
-        if items:
-            yield [root], items
-
-    # Clear out any unfinished collapse.
-    if collapse_paths and collapse_items:
-        yield collapse_paths, collapse_items
-
 def apply_item_metadata(item, track_info):
     """Set an item's metadata from its matched TrackInfo object.
     """
@@ -156,6 +42,7 @@ def apply_item_metadata(item, track_info):
     # At the moment, the other metadata is left intact (including album
     # and track number). Perhaps these should be emptied?
 
+
 def apply_metadata(album_info, mapping):
     """Set the items' metadata to match an AlbumInfo object using a
     mapping from Items to TrackInfo objects.
@@ -171,8 +58,8 @@ def apply_metadata(album_info, mapping):
 
         # Artist sort and credit names.
         item.artist_sort = track_info.artist_sort or album_info.artist_sort
-        item.artist_credit = track_info.artist_credit or \
-                             album_info.artist_credit
+        item.artist_credit = (track_info.artist_credit or
+                              album_info.artist_credit)
         item.albumartist_sort = album_info.artist_sort
         item.albumartist_credit = album_info.artist_credit
 
@@ -235,7 +122,6 @@ def apply_metadata(album_info, mapping):
                       'language',
                       'country',
                       'albumstatus',
-                      'media',
                       'albumdisambig'):
             value = getattr(album_info, field)
             if value is not None:
@@ -243,5 +129,8 @@ def apply_metadata(album_info, mapping):
         if track_info.disctitle is not None:
             item.disctitle = track_info.disctitle
 
+        if track_info.media is not None:
+            item.media = track_info.media
+
         # Headphones seal of approval
-        item.comments = 'tagged by headphones/beets'
+        item.comments = 'tagged by headphones/beets'

lib/beets/autotag/hooks.py

@@ -116,6 +116,7 @@ class AlbumInfo(object):
             for track in self.tracks:
                 track.decode(codec)
 
+
 class TrackInfo(object):
     """Describes a canonical track present on a release. Appears as part
     of an AlbumInfo's ``tracks`` list. Consists of these data members:
@@ -126,6 +127,7 @@ class TrackInfo(object):
     - ``artist_id``
     - ``length``: float: duration of the track in seconds
     - ``index``: position on the entire release
+    - ``media``: delivery mechanism (Vinyl, etc.)
     - ``medium``: the disc number this track appears on in the album
     - ``medium_index``: the track's position on the disc
     - ``medium_total``: the number of tracks on the item's disc
@@ -140,13 +142,15 @@ class TrackInfo(object):
     def __init__(self, title, track_id, artist=None, artist_id=None,
                  length=None, index=None, medium=None, medium_index=None,
                  medium_total=None, artist_sort=None, disctitle=None,
-                 artist_credit=None, data_source=None, data_url=None):
+                 artist_credit=None, data_source=None, data_url=None,
+                 media=None):
         self.title = title
         self.track_id = track_id
         self.artist = artist
         self.artist_id = artist_id
         self.length = length
         self.index = index
+        self.media = media
         self.medium = medium
         self.medium_index = medium_index
         self.medium_total = medium_total
@@ -162,7 +166,7 @@ class TrackInfo(object):
         to Unicode.
         """
         for fld in ['title', 'artist', 'medium', 'artist_sort', 'disctitle',
-                    'artist_credit']:
+                    'artist_credit', 'media']:
             value = getattr(self, fld)
             if isinstance(value, str):
                 setattr(self, fld, value.decode(codec, 'ignore'))
@@ -187,6 +191,7 @@ SD_REPLACE = [
     (r'&', 'and'),
 ]
 
+
 def _string_dist_basic(str1, str2):
     """Basic edit distance between two strings, ignoring
     non-alphanumeric characters and case. Comparisons are based on a
@@ -201,13 +206,16 @@ def _string_dist_basic(str1, str2):
         return 0.0
     return levenshtein(str1, str2) / float(max(len(str1), len(str2)))
 
+
 def string_dist(str1, str2):
     """Gives an "intuitive" edit distance between two strings. This is
     an edit distance, normalized by the string length, with a number of
     tweaks that reflect intuition about text.
     """
-    if str1 is None and str2 is None: return 0.0
-    if str1 is None or str2 is None:  return 1.0
+    if str1 is None and str2 is None:
+        return 0.0
+    if str1 is None or str2 is None:
+        return 1.0
 
     str1 = str1.lower()
     str2 = str2.lower()
@@ -217,9 +225,9 @@ def string_dist(str1, str2):
     # "something, the".
     for word in SD_END_WORDS:
         if str1.endswith(', %s' % word):
-            str1 = '%s %s' % (word, str1[:-len(word)-2])
+            str1 = '%s %s' % (word, str1[:-len(word) - 2])
         if str2.endswith(', %s' % word):
-            str2 = '%s %s' % (word, str2[:-len(word)-2])
+            str2 = '%s %s' % (word, str2[:-len(word) - 2])
 
     # Perform a couple of basic normalizing substitutions.
     for pat, repl in SD_REPLACE:
@@ -256,6 +264,23 @@ def string_dist(str1, str2):
 
     return base_dist + penalty
 
+
+class LazyClassProperty(object):
+    """A decorator implementing a read-only property that is *lazy* in
+    the sense that the getter is only invoked once. Subsequent accesses
+    through *any* instance use the cached result.
+    """
+    def __init__(self, getter):
+        self.getter = getter
+        self.computed = False
+
+    def __get__(self, obj, owner):
+        if not self.computed:
+            self.value = self.getter(owner)
+            self.computed = True
+        return self.value
+
+
 class Distance(object):
     """Keeps track of multiple distance penalties. Provides a single
     weighted distance for all penalties as well as a weighted distance
@@ -264,11 +289,15 @@ class Distance(object):
     def __init__(self):
         self._penalties = {}
 
+    @LazyClassProperty
+    def _weights(cls):
+        """A dictionary from keys to floating-point weights.
+        """
         weights_view = config['match']['distance_weights']
-        self._weights = {}
+        weights = {}
         for key in weights_view.keys():
-            self._weights[key] = weights_view[key].as_number()
-
+            weights[key] = weights_view[key].as_number()
+        return weights
 
     # Access the components and their aggregates.
 
@@ -313,8 +342,7 @@ class Distance(object):
         # Convert distance into a negative float we can sort items in
         # ascending order (for keys, when the penalty is equal) and
         # still get the items with the biggest distance first.
-        return sorted(list_, key=lambda (key, dist): (0-dist, key))
-
+        return sorted(list_, key=lambda (key, dist): (0 - dist, key))
 
     # Behave like a float.
 
@@ -323,13 +351,13 @@ class Distance(object):
 
     def __float__(self):
         return self.distance
+
     def __sub__(self, other):
         return self.distance - other
 
     def __rsub__(self, other):
         return other - self.distance
 
-
     # Behave like a dict.
 
     def __getitem__(self, key):
@@ -355,11 +383,11 @@ class Distance(object):
         """
         if not isinstance(dist, Distance):
             raise ValueError(
-                    '`dist` must be a Distance object. It is: %r' % dist)
+                '`dist` must be a Distance object, not {0}'.format(type(dist))
+            )
         for key, penalties in dist._penalties.iteritems():
             self._penalties.setdefault(key, []).extend(penalties)
 
-
     # Adding components.
 
     def _eq(self, value1, value2):
@@ -379,7 +407,8 @@ class Distance(object):
         """
         if not 0.0 <= dist <= 1.0:
             raise ValueError(
-                    '`dist` must be between 0.0 and 1.0. It is: %r' % dist)
+                '`dist` must be between 0.0 and 1.0, not {0}'.format(dist)
+            )
         self._penalties.setdefault(key, []).append(dist)
 
     def add_equality(self, key, value, options):
@@ -476,6 +505,7 @@ def album_for_mbid(release_id):
     except mb.MusicBrainzAPIError as exc:
         exc.log(log)
 
+
 def track_for_mbid(recording_id):
     """Get a TrackInfo object for a MusicBrainz recording ID. Return None
     if the ID is not found.
@@ -485,18 +515,21 @@ def track_for_mbid(recording_id):
     except mb.MusicBrainzAPIError as exc:
         exc.log(log)
 
+
 def albums_for_id(album_id):
     """Get a list of albums for an ID."""
     candidates = [album_for_mbid(album_id)]
     candidates.extend(plugins.album_for_id(album_id))
     return filter(None, candidates)
 
+
 def tracks_for_id(track_id):
     """Get a list of tracks for an ID."""
     candidates = [track_for_mbid(track_id)]
     candidates.extend(plugins.track_for_id(track_id))
     return filter(None, candidates)
 
+
 def album_candidates(items, artist, album, va_likely):
     """Search for album matches. ``items`` is a list of Item objects
     that make up the album. ``artist`` and ``album`` are the respective
@@ -525,6 +558,7 @@ def album_candidates(items, artist, album, va_likely):
 
     return out
 
+
 def item_candidates(item, artist, title):
     """Search for item matches. ``item`` is the Item to be matched.
     ``artist`` and ``title`` are strings and either reflect the item or

lib/beets/autotag/match.py

@@ -25,11 +25,8 @@ from munkres import Munkres
 from beets import plugins
 from beets import config
 from beets.util import plurality
-from beets.util.enumeration import enum
 from beets.autotag import hooks
-
-# Recommendation enumeration.
-recommendation = enum('none', 'low', 'medium', 'strong', name='recommendation')
+from beets.util.enumeration import OrderedEnum
 
 # Artist signals that indicate "various artists". These are used at the
 # album level to determine whether a given release is likely a VA
@@ -41,6 +38,18 @@ VA_ARTISTS = (u'', u'various artists', u'various', u'va', u'unknown')
 log = logging.getLogger('beets')
 
 
+# Recommendation enumeration.
+
+class Recommendation(OrderedEnum):
+    """Indicates a qualitative suggestion to the user about what should
+    be done with a given match.
+    """
+    none = 0
+    low = 1
+    medium = 2
+    strong = 3
+
+
 # Primary matching functionality.
 
 def current_metadata(items):
@@ -56,10 +65,10 @@ def current_metadata(items):
     fields = ['artist', 'album', 'albumartist', 'year', 'disctotal',
               'mb_albumid', 'label', 'catalognum', 'country', 'media',
               'albumdisambig']
-    for key in fields:
-        values = [getattr(item, key) for item in items if item]
-        likelies[key], freq = plurality(values)
-        consensus[key] = (freq == len(values))
+    for field in fields:
+        values = [item[field] for item in items if item]
+        likelies[field], freq = plurality(values)
+        consensus[field] = (freq == len(values))
 
     # If there's an album artist consensus, use this for the artist.
     if consensus['albumartist'] and likelies['albumartist']:
@@ -67,6 +76,7 @@ def current_metadata(items):
 
     return likelies, consensus
 
+
 def assign_items(items, tracks):
     """Given a list of Items and a list of TrackInfo objects, find the
     best mapping between them. Returns a mapping from Items to TrackInfo
@@ -93,12 +103,14 @@ def assign_items(items, tracks):
     extra_tracks.sort(key=lambda t: (t.index, t.title))
     return mapping, extra_items, extra_tracks
 
+
 def track_index_changed(item, track_info):
     """Returns True if the item and track info index is different. Tolerates
     per disc and per release numbering.
     """
     return item.track not in (track_info.medium_index, track_info.index)
 
+
 def track_distance(item, track_info, incl_artist=False):
     """Determines the significance of a track metadata change. Returns a
     Distance object. `incl_artist` indicates that a distance component should
@@ -109,7 +121,7 @@ def track_distance(item, track_info, incl_artist=False):
     # Length.
     if track_info.length:
         diff = abs(item.length - track_info.length) - \
-               config['match']['track_length_grace'].as_number()
+            config['match']['track_length_grace'].as_number()
         dist.add_ratio('track_length', diff,
                        config['match']['track_length_max'].as_number())
 
@@ -134,6 +146,7 @@ def track_distance(item, track_info, incl_artist=False):
 
     return dist
 
+
 def distance(items, album_info, mapping):
     """Determines how "significant" an album metadata change would be.
     Returns a Distance object. `album_info` is an AlbumInfo object
@@ -239,6 +252,7 @@ def distance(items, album_info, mapping):
 
     return dist
 
+
 def match_by_id(items):
     """If the items are tagged with a MusicBrainz album ID, returns an
     AlbumInfo object for the corresponding album. Otherwise, returns
@@ -247,16 +261,17 @@ def match_by_id(items):
     # Is there a consensus on the MB album ID?
     albumids = [item.mb_albumid for item in items if item.mb_albumid]
     if not albumids:
-        log.debug('No album IDs found.')
+        log.debug(u'No album IDs found.')
         return None
 
     # If all album IDs are equal, look up the album.
-    if bool(reduce(lambda x,y: x if x==y else (), albumids)):
+    if bool(reduce(lambda x, y: x if x == y else (), albumids)):
         albumid = albumids[0]
-        log.debug('Searching for discovered album ID: ' + albumid)
+        log.debug(u'Searching for discovered album ID: {0}'.format(albumid))
         return hooks.album_for_mbid(albumid)
     else:
-        log.debug('No album ID consensus.')
+        log.debug(u'No album ID consensus.')
+
 
 def _recommendation(results):
     """Given a sorted list of AlbumMatch or TrackMatch objects, return a
@@ -268,26 +283,26 @@ def _recommendation(results):
     """
     if not results:
         # No candidates: no recommendation.
-        return recommendation.none
+        return Recommendation.none
 
     # Basic distance thresholding.
     min_dist = results[0].distance
     if min_dist < config['match']['strong_rec_thresh'].as_number():
         # Strong recommendation level.
-        rec = recommendation.strong
+        rec = Recommendation.strong
     elif min_dist <= config['match']['medium_rec_thresh'].as_number():
         # Medium recommendation level.
-        rec = recommendation.medium
+        rec = Recommendation.medium
     elif len(results) == 1:
         # Only a single candidate.
-        rec = recommendation.low
+        rec = Recommendation.low
     elif results[1].distance - min_dist >= \
             config['match']['rec_gap_thresh'].as_number():
         # Gap between first two candidates is large.
-        rec = recommendation.low
+        rec = Recommendation.low
     else:
         # No conclusion. Return immediately. Can't be downgraded any further.
-        return recommendation.none
+        return Recommendation.none
 
     # Downgrade to the max rec if it is lower than the current rec for an
     # applied penalty.
@@ -299,28 +314,40 @@ def _recommendation(results):
     for key in keys:
         if key in max_rec_view.keys():
             max_rec = max_rec_view[key].as_choice({
-                'strong': recommendation.strong,
-                'medium': recommendation.medium,
-                'low': recommendation.low,
-                'none': recommendation.none,
+                'strong': Recommendation.strong,
+                'medium': Recommendation.medium,
+                'low': Recommendation.low,
+                'none': Recommendation.none,
             })
             rec = min(rec, max_rec)
 
     return rec
 
+
 def _add_candidate(items, results, info):
     """Given a candidate AlbumInfo object, attempt to add the candidate
     to the output dictionary of AlbumMatch objects. This involves
     checking the track count, ordering the items, checking for
     duplicates, and calculating the distance.
     """
-    log.debug('Candidate: %s - %s' % (info.artist, info.album))
+    log.debug(u'Candidate: {0} - {1}'.format(info.artist, info.album))
+
+    # Discard albums with zero tracks.
+    if not info.tracks:
+        log.debug('No tracks.')
+        return
 
     # Don't duplicate.
     if info.album_id in results:
-        log.debug('Duplicate.')
+        log.debug(u'Duplicate.')
         return
 
+    # Discard matches without required tags.
+    for req_tag in config['match']['required'].as_str_seq():
+        if getattr(info, req_tag) is None:
+            log.debug(u'Ignored. Missing required tag: {0}'.format(req_tag))
+            return
+
     # Find mapping between the items and the track info.
     mapping, extra_items, extra_tracks = assign_items(items, info.tracks)
 
@@ -331,30 +358,36 @@ def _add_candidate(items, results, info):
     penalties = [key for _, key in dist]
     for penalty in config['match']['ignored'].as_str_seq():
         if penalty in penalties:
-            log.debug('Ignored. Penalty: %s' % penalty)
+            log.debug(u'Ignored. Penalty: {0}'.format(penalty))
             return
 
-    log.debug('Success. Distance: %f' % dist)
+    log.debug(u'Success. Distance: {0}'.format(dist))
     results[info.album_id] = hooks.AlbumMatch(dist, info, mapping,
                                               extra_items, extra_tracks)
 
+
 def tag_album(items, search_artist=None, search_album=None,
               search_id=None):
-    """Bundles together the functionality used to infer tags for a
-    set of items comprised by an album. Returns everything relevant:
-        - The current artist.
-        - The current album.
-        - A list of AlbumMatch objects. The candidates are sorted by
-          distance (i.e., best match first).
-        - A recommendation.
-    If search_artist and search_album or search_id are provided, then
-    they are used as search terms in place of the current metadata.
+    """Return a tuple of a artist name, an album name, a list of
+    `AlbumMatch` candidates from the metadata backend, and a
+    `Recommendation`.
+
+    The artist and album are the most common values of these fields
+    among `items`.
+
+    The `AlbumMatch` objects are generated by searching the metadata
+    backends. By default, the metadata of the items is used for the
+    search. This can be customized by setting the parameters. The
+    `mapping` field of the album has the matched `items` as keys.
+
+    The recommendation is calculated from the match qualitiy of the
+    candidates.
     """
     # Get current metadata.
     likelies, consensus = current_metadata(items)
     cur_artist = likelies['artist']
     cur_album = likelies['album']
-    log.debug('Tagging %s - %s' % (cur_artist, cur_album))
+    log.debug(u'Tagging {0} - {1}'.format(cur_artist, cur_album))
 
     # The output result (distance, AlbumInfo) tuples (keyed by MB album
     # ID).
@@ -362,7 +395,7 @@ def tag_album(items, search_artist=None, search_album=None,
 
     # Search by explicit ID.
     if search_id is not None:
-        log.debug('Searching for album ID: ' + search_id)
+        log.debug(u'Searching for album ID: {0}'.format(search_id))
         search_cands = hooks.albums_for_id(search_id)
 
     # Use existing metadata or text search.
@@ -372,32 +405,33 @@ def tag_album(items, search_artist=None, search_album=None,
         if id_info:
             _add_candidate(items, candidates, id_info)
             rec = _recommendation(candidates.values())
-            log.debug('Album ID match recommendation is ' + str(rec))
+            log.debug(u'Album ID match recommendation is {0}'.format(str(rec)))
             if candidates and not config['import']['timid']:
                 # If we have a very good MBID match, return immediately.
                 # Otherwise, this match will compete against metadata-based
                 # matches.
-                if rec == recommendation.strong:
-                    log.debug('ID match.')
+                if rec == Recommendation.strong:
+                    log.debug(u'ID match.')
                     return cur_artist, cur_album, candidates.values(), rec
 
         # Search terms.
         if not (search_artist and search_album):
             # No explicit search terms -- use current metadata.
             search_artist, search_album = cur_artist, cur_album
-        log.debug(u'Search terms: %s - %s' % (search_artist, search_album))
+        log.debug(u'Search terms: {0} - {1}'.format(search_artist,
+                                                    search_album))
 
         # Is this album likely to be a "various artist" release?
         va_likely = ((not consensus['artist']) or
-                    (search_artist.lower() in VA_ARTISTS) or
-                    any(item.comp for item in items))
-        log.debug(u'Album might be VA: %s' % str(va_likely))
+                     (search_artist.lower() in VA_ARTISTS) or
+                     any(item.comp for item in items))
+        log.debug(u'Album might be VA: {0}'.format(str(va_likely)))
 
         # Get the results from the data sources.
         search_cands = hooks.album_candidates(items, search_artist,
                                               search_album, va_likely)
 
-    log.debug(u'Evaluating %i candidates.' % len(search_cands))
+    log.debug(u'Evaluating {0} candidates.'.format(len(search_cands)))
     for info in search_cands:
         _add_candidate(items, candidates, info)
 
@@ -406,6 +440,7 @@ def tag_album(items, search_artist=None, search_album=None,
     rec = _recommendation(candidates)
     return cur_artist, cur_album, candidates, rec
 
+
 def tag_item(item, search_artist=None, search_title=None,
              search_id=None):
     """Attempts to find metadata for a single track. Returns a
@@ -421,15 +456,15 @@ def tag_item(item, search_artist=None, search_title=None,
     # First, try matching by MusicBrainz ID.
     trackid = search_id or item.mb_trackid
     if trackid:
-        log.debug('Searching for track ID: ' + trackid)
+        log.debug(u'Searching for track ID: {0}'.format(trackid))
         for track_info in hooks.tracks_for_id(trackid):
             dist = track_distance(item, track_info, incl_artist=True)
             candidates[track_info.track_id] = \
-                    hooks.TrackMatch(dist, track_info)
+                hooks.TrackMatch(dist, track_info)
             # If this is a good match, then don't keep searching.
             rec = _recommendation(candidates.values())
-            if rec == recommendation.strong and not config['import']['timid']:
-                log.debug('Track ID match.')
+            if rec == Recommendation.strong and not config['import']['timid']:
+                log.debug(u'Track ID match.')
                 return candidates.values(), rec
 
     # If we're searching by ID, don't proceed.
@@ -437,12 +472,13 @@ def tag_item(item, search_artist=None, search_title=None,
         if candidates:
             return candidates.values(), rec
         else:
-            return [], recommendation.none
+            return [], Recommendation.none
 
     # Search terms.
     if not (search_artist and search_title):
         search_artist, search_title = item.artist, item.title
-    log.debug(u'Item search terms: %s - %s' % (search_artist, search_title))
+    log.debug(u'Item search terms: {0} - {1}'.format(search_artist,
+                                                     search_title))
 
     # Get and evaluate candidate metadata.
     for track_info in hooks.item_candidates(item, search_artist, search_title):
@@ -450,7 +486,7 @@ def tag_item(item, search_artist=None, search_title=None,
         candidates[track_info.track_id] = hooks.TrackMatch(dist, track_info)
 
     # Sort by distance and return with recommendation.
-    log.debug('Found %i candidates.' % len(candidates))
+    log.debug(u'Found {0} candidates.'.format(len(candidates)))
     candidates = sorted(candidates.itervalues())
     rec = _recommendation(candidates)
     return candidates, rec

lib/beets/autotag/mb.py

@@ -32,6 +32,7 @@ BASE_URL = 'http://musicbrainz.org/'
 musicbrainzngs.set_useragent('beets', beets.__version__,
                              'http://beets.radbox.org/')
 
+
 class MusicBrainzAPIError(util.HumanReadableException):
     """An error while talking to MusicBrainz. The `query` field is the
     parameter to the action and may have any type.
@@ -41,7 +42,7 @@ class MusicBrainzAPIError(util.HumanReadableException):
         super(MusicBrainzAPIError, self).__init__(reason, verb, tb)
 
     def get_message(self):
-        return u'"{0}" in {1} with query {2}'.format(
+        return u'{0} in {1} with query {2}'.format(
             self._reasonstr(), self.verb, repr(self.query)
         )
 
@@ -51,12 +52,15 @@ RELEASE_INCLUDES = ['artists', 'media', 'recordings', 'release-groups',
                     'labels', 'artist-credits', 'aliases']
 TRACK_INCLUDES = ['artists', 'aliases']
 
+
 def track_url(trackid):
     return urljoin(BASE_URL, 'recording/' + trackid)
 
+
 def album_url(albumid):
     return urljoin(BASE_URL, 'release/' + albumid)
 
+
 def configure():
     """Set up the python-musicbrainz-ngs module according to settings
     from the beets configuration. This should be called at startup.
@@ -67,6 +71,7 @@ def configure():
         config['musicbrainz']['ratelimit'].get(int),
     )
 
+
 def _preferred_alias(aliases):
     """Given an list of alias structures for an artist credit, select
     and return the user's preferred alias alias or None if no matching
@@ -81,13 +86,15 @@ def _preferred_alias(aliases):
     # Search configured locales in order.
     for locale in config['import']['languages'].as_str_seq():
         # Find matching primary aliases for this locale.
-        matches = [a for a in aliases if a['locale'] == locale and 'primary' in a]
+        matches = [a for a in aliases
+                   if a['locale'] == locale and 'primary' in a]
         # Skip to the next locale if we have no matches
         if not matches:
             continue
 
         return matches[0]
 
+
 def _flatten_artist_credit(credit):
     """Given a list representing an ``artist-credit`` block, flatten the
     data into a triple of joined artist name strings: canonical, sort, and
@@ -133,6 +140,7 @@ def _flatten_artist_credit(credit):
         ''.join(artist_credit_parts),
     )
 
+
 def track_info(recording, index=None, medium=None, medium_index=None,
                medium_total=None):
     """Translates a MusicBrainz recording result dictionary into a beets
@@ -167,6 +175,7 @@ def track_info(recording, index=None, medium=None, medium_index=None,
     info.decode()
     return info
 
+
 def _set_date_str(info, date_str, original=False):
     """Given a (possibly partial) YYYY-MM-DD string and an AlbumInfo
     object, set the object's release date fields appropriately. If
@@ -186,6 +195,7 @@ def _set_date_str(info, date_str, original=False):
                     key = 'original_' + key
                 setattr(info, key, date_num)
 
+
 def album_info(release):
     """Takes a MusicBrainz release result dictionary and returns a beets
     AlbumInfo object containing the interesting data about that release.
@@ -199,6 +209,7 @@ def album_info(release):
     index = 0
     for medium in release['medium-list']:
         disctitle = medium.get('title')
+        format = medium.get('format')
         for track in medium['track-list']:
             # Basic information from the recording.
             index += 1
@@ -210,6 +221,7 @@ def album_info(release):
                 len(medium['track-list']),
             )
             ti.disctitle = disctitle
+            ti.media = format
 
             # Prefer track data, where present, over recording data.
             if track.get('title'):
@@ -288,6 +300,7 @@ def album_info(release):
     info.decode()
     return info
 
+
 def match_album(artist, album, tracks=None, limit=SEARCH_LIMIT):
     """Searches for a single album ("release" in MusicBrainz parlance)
     and returns an iterator over AlbumInfo objects. May raise a
@@ -297,9 +310,9 @@ def match_album(artist, album, tracks=None, limit=SEARCH_LIMIT):
     optionally, a number of tracks on the album.
     """
     # Build search criteria.
-    criteria = {'release': album.lower()}
+    criteria = {'release': album.lower().strip()}
     if artist is not None:
-        criteria['artist'] = artist.lower()
+        criteria['artist'] = artist.lower().strip()
     else:
         # Various Artists search.
         criteria['arid'] = VARIOUS_ARTISTS_ID
@@ -322,13 +335,14 @@ def match_album(artist, album, tracks=None, limit=SEARCH_LIMIT):
         if albuminfo is not None:
             yield albuminfo
 
+
 def match_track(artist, title, limit=SEARCH_LIMIT):
     """Searches for a single track and returns an iterable of TrackInfo
     objects. May raise a MusicBrainzAPIError.
     """
     criteria = {
-        'artist': artist.lower(),
-        'recording': title.lower(),
+        'artist': artist.lower().strip(),
+        'recording': title.lower().strip(),
     }
 
     if not any(criteria.itervalues()):
@@ -342,6 +356,7 @@ def match_track(artist, title, limit=SEARCH_LIMIT):
     for recording in res['recording-list']:
         yield track_info(recording)
 
+
 def _parse_id(s):
     """Search for a MusicBrainz ID in the given string and return it. If
     no ID can be found, return None.
@@ -351,38 +366,40 @@ def _parse_id(s):
     if match:
         return match.group()
 
-def album_for_id(albumid):
+
+def album_for_id(releaseid):
     """Fetches an album by its MusicBrainz ID and returns an AlbumInfo
     object or None if the album is not found. May raise a
     MusicBrainzAPIError.
     """
-    albumid = _parse_id(albumid)
+    albumid = _parse_id(releaseid)
     if not albumid:
-        log.error('Invalid MBID.')
+        log.debug(u'Invalid MBID ({0}).'.format(releaseid))
         return
     try:
         res = musicbrainzngs.get_release_by_id(albumid,
                                                RELEASE_INCLUDES)
     except musicbrainzngs.ResponseError:
-        log.debug('Album ID match failed.')
+        log.debug(u'Album ID match failed.')
         return None
     except musicbrainzngs.MusicBrainzError as exc:
         raise MusicBrainzAPIError(exc, 'get release by ID', albumid,
                                   traceback.format_exc())
     return album_info(res['release'])
 
-def track_for_id(trackid):
+
+def track_for_id(releaseid):
     """Fetches a track by its MusicBrainz ID. Returns a TrackInfo object
     or None if no track is found. May raise a MusicBrainzAPIError.
     """
-    trackid = _parse_id(trackid)
+    trackid = _parse_id(releaseid)
     if not trackid:
-        log.error('Invalid MBID.')
+        log.debug(u'Invalid MBID ({0}).'.format(releaseid))
         return
     try:
         res = musicbrainzngs.get_recording_by_id(trackid, TRACK_INCLUDES)
     except musicbrainzngs.ResponseError:
-        log.debug('Track ID match failed.')
+        log.debug(u'Track ID match failed.')
         return None
     except musicbrainzngs.MusicBrainzError as exc:
         raise MusicBrainzAPIError(exc, 'get recording by ID', trackid,

lib/beets/config_default.yaml

@@ -5,6 +5,7 @@ import:
     write: yes
     copy: yes
     move: no
+    link: no
     delete: no
     resume: ask
     incremental: no
@@ -20,6 +21,7 @@ import:
     detail: no
     flat: no
     group_albums: no
+    pretend: false
 
 clutter: ["Thumbs.DB", ".DS_Store"]
 ignore: [".*", "*~", "System Volume Information"]
@@ -32,6 +34,7 @@ replace:
     '\s+$': ''
     '^\s+': ''
 path_sep_replace: _
+asciify_paths: false
 art_filename: cover
 max_filename_length: 0
 
@@ -54,6 +57,9 @@ list_format_item: $artist - $album - $title
 list_format_album: $albumartist - $album
 time_format: '%Y-%m-%d %H:%M:%S'
 
+sort_album: albumartist+ album+
+sort_item: artist+ album+ disc+ track+
+
 paths:
     default: $albumartist/$album%aunique{}/$track $title
     singleton: Non-Album/$artist/$title
@@ -98,5 +104,6 @@ match:
         media: []
         original_year: no
     ignored: []
+    required: []
     track_length_grace: 10
     track_length_max: 30

lib/beets/dbcore/__init__.py

@@ -18,3 +18,8 @@ Library.
 from .db import Model, Database
 from .query import Query, FieldQuery, MatchQuery, AndQuery, OrQuery
 from .types import Type
+from .queryparse import query_from_strings
+from .queryparse import sort_from_strings
+from .queryparse import parse_sorted_query
+
+# flake8: noqa

lib/beets/dbcore/db.py

@@ -20,16 +20,62 @@ from collections import defaultdict
 import threading
 import sqlite3
 import contextlib
+import collections
 
 import beets
 from beets.util.functemplate import Template
-from .query import MatchQuery
+from beets.dbcore import types
+from .query import MatchQuery, NullSort, TrueQuery
 
 
+class FormattedMapping(collections.Mapping):
+    """A `dict`-like formatted view of a model.
+
+    The accessor `mapping[key]` returns the formated version of
+    `model[key]` as a unicode string.
+
+    If `for_path` is true, all path separators in the formatted values
+    are replaced.
+    """
+
+    def __init__(self, model, for_path=False):
+        self.for_path = for_path
+        self.model = model
+        self.model_keys = model.keys(True)
+
+    def __getitem__(self, key):
+        if key in self.model_keys:
+            return self._get_formatted(self.model, key)
+        else:
+            raise KeyError(key)
+
+    def __iter__(self):
+        return iter(self.model_keys)
+
+    def __len__(self):
+        return len(self.model_keys)
+
+    def get(self, key, default=None):
+        if default is None:
+            default = self.model._type(key).format(None)
+        return super(FormattedMapping, self).get(key, default)
+
+    def _get_formatted(self, model, key):
+        value = model._type(key).format(model.get(key))
+        if isinstance(value, bytes):
+            value = value.decode('utf8', 'ignore')
+
+        if self.for_path:
+            sep_repl = beets.config['path_sep_replace'].get(unicode)
+            for sep in (os.path.sep, os.path.altsep):
+                if sep:
+                    value = value.replace(sep, sep_repl)
+
+        return value
+
 
 # Abstract base for model classes.
 
-
 class Model(object):
     """An abstract object representing an object in the database. Model
     objects act like dictionaries (i.e., the allow subscript access like
@@ -66,12 +112,7 @@ class Model(object):
 
     _fields = {}
     """A mapping indicating available "fixed" fields on this type. The
-    keys are field names and the values are Type objects.
-    """
-
-    _bytes_keys = ()
-    """Keys whose values should be stored as raw bytes blobs rather than
-    strings.
+    keys are field names and the values are `Type` objects.
     """
 
     _search_fields = ()
@@ -79,6 +120,21 @@ class Model(object):
     terms.
     """
 
+    _types = {}
+    """Optional Types for non-fixed (i.e., flexible and computed) fields.
+    """
+
+    _sorts = {}
+    """Optional named sort criteria. The keys are strings and the values
+    are subclasses of `Sort`.
+    """
+
+    _always_dirty = False
+    """By default, fields only become "dirty" when their value actually
+    changes. Enabling this flag marks fields as dirty even when the new
+    value is the same as the old value (e.g., `o.f = o.f`).
+    """
+
     @classmethod
     def _getters(cls):
         """Return a mapping from field names to getter functions.
@@ -94,7 +150,6 @@ class Model(object):
         # As above: we could consider caching this result.
         raise NotImplementedError()
 
-
     # Basic operation.
 
     def __init__(self, db=None, **values):
@@ -110,6 +165,20 @@ class Model(object):
         self.update(values)
         self.clear_dirty()
 
+    @classmethod
+    def _awaken(cls, db=None, fixed_values={}, flex_values={}):
+        """Create an object with values drawn from the database.
+
+        This is a performance optimization: the checks involved with
+        ordinary construction are bypassed.
+        """
+        obj = cls(db)
+        for key, value in fixed_values.iteritems():
+            obj._values_fixed[key] = cls._type(key).from_sql(value)
+        for key, value in flex_values.iteritems():
+            obj._values_flex[key] = cls._type(key).from_sql(value)
+        return obj
+
     def __repr__(self):
         return '{0}({1})'.format(
             type(self).__name__,
@@ -132,9 +201,17 @@ class Model(object):
         if need_id and not self.id:
             raise ValueError('{0} has no id'.format(type(self).__name__))
 
-
     # Essential field accessors.
 
+    @classmethod
+    def _type(self, key):
+        """Get the type of a field, a `Type` instance.
+
+        If the field has no explicit type, it is given the base `Type`,
+        which does no conversion.
+        """
+        return self._fields.get(key) or self._types.get(key) or types.DEFAULT
+
     def __getitem__(self, key):
         """Get the value for a field. Raise a KeyError if the field is
         not available.
@@ -152,11 +229,19 @@ class Model(object):
     def __setitem__(self, key, value):
         """Assign the value for a field.
         """
-        source = self._values_fixed if key in self._fields \
-                 else self._values_flex
+        # Choose where to place the value.
+        if key in self._fields:
+            source = self._values_fixed
+        else:
+            source = self._values_flex
+
+        # If the field has a type, filter the value.
+        value = self._type(key).normalize(value)
+
+        # Assign value and possibly mark as dirty.
         old_value = source.get(key)
         source[key] = value
-        if old_value != value:
+        if self._always_dirty or old_value != value:
             self._dirty.add(key)
 
     def __delitem__(self, key):
@@ -183,7 +268,6 @@ class Model(object):
         else:
             return base_keys
 
-
     # Act like a dictionary.
 
     def update(self, values):
@@ -219,7 +303,6 @@ class Model(object):
         """
         return iter(self.keys())
 
-
     # Convenient attribute access.
 
     def __getattr__(self, key):
@@ -243,7 +326,6 @@ class Model(object):
         else:
             del self[key]
 
-
     # Database interaction (CRUD methods).
 
     def store(self):
@@ -252,19 +334,15 @@ class Model(object):
         self._check_db()
 
         # Build assignments for query.
-        assignments = ''
+        assignments = []
         subvars = []
         for key in self._fields:
             if key != 'id' and key in self._dirty:
                 self._dirty.remove(key)
-                assignments += key + '=?,'
-                value = self[key]
-                # Wrap path strings in buffers so they get stored
-                # "in the raw".
-                if key in self._bytes_keys and isinstance(value, str):
-                    value = buffer(value)
+                assignments.append(key + '=?')
+                value = self._type(key).to_sql(self[key])
                 subvars.append(value)
-        assignments = assignments[:-1]  # Knock off last ,
+        assignments = ','.join(assignments)
 
         with self._db.transaction() as tx:
             # Main table update.
@@ -302,6 +380,8 @@ class Model(object):
         self._check_db()
         stored_obj = self._db._get(type(self), self.id)
         assert stored_obj is not None, "object {0} not in DB".format(self.id)
+        self._values_fixed = {}
+        self._values_flex = {}
         self.update(dict(stored_obj))
         self.clear_dirty()
 
@@ -344,76 +424,26 @@ class Model(object):
                     self._dirty.add(key)
             self.store()
 
-
     # Formatting and templating.
 
-    @classmethod
-    def _format(cls, key, value, for_path=False):
-        """Format a value as the given field for this model.
-        """
-        # Format the value as a string according to its type, if any.
-        if key in cls._fields:
-            value = cls._fields[key].format(value)
-            # Formatting must result in a string. To deal with
-            # Python2isms, implicitly convert ASCII strings.
-            assert isinstance(value, basestring), \
-                    u'field formatter must produce strings'
-            if isinstance(value, bytes):
-                value = value.decode('utf8', 'ignore')
+    _formatter = FormattedMapping
 
-        elif not isinstance(value, unicode):
-            # Fallback formatter. Convert to unicode at all cost.
-            if value is None:
-                value = u''
-            elif isinstance(value, basestring):
-                if isinstance(value, bytes):
-                    value = value.decode('utf8', 'ignore')
-            else:
-                value = unicode(value)
-
-        if for_path:
-            sep_repl = beets.config['path_sep_replace'].get(unicode)
-            for sep in (os.path.sep, os.path.altsep):
-                if sep:
-                    value = value.replace(sep, sep_repl)
-
-        return value
-
-    def _get_formatted(self, key, for_path=False):
-        """Get a field value formatted as a string (`unicode` object)
-        for display to the user. If `for_path` is true, then the value
-        will be sanitized for inclusion in a pathname (i.e., path
-        separators will be removed from the value).
-        """
-        return self._format(key, self.get(key), for_path)
-
-    def _formatted_mapping(self, for_path=False):
+    def formatted(self, for_path=False):
         """Get a mapping containing all values on this object formatted
-        as human-readable strings.
+        as human-readable unicode strings.
         """
-        # In the future, this could be made "lazy" to avoid computing
-        # fields unnecessarily.
-        out = {}
-        for key in self.keys(True):
-            out[key] = self._get_formatted(key, for_path)
-        return out
+        return self._formatter(self, for_path)
 
     def evaluate_template(self, template, for_path=False):
         """Evaluate a template (a string or a `Template` object) using
         the object's fields. If `for_path` is true, then no new path
         separators will be added to the template.
         """
-        # Build value mapping.
-        mapping = self._formatted_mapping(for_path)
-
-        # Get template functions.
-        funcs = self._template_funcs()
-
         # Perform substitution.
         if isinstance(template, basestring):
             template = Template(template)
-        return template.substitute(mapping, funcs)
-
+        return template.substitute(self.formatted(for_path),
+                                   self._template_funcs())
 
     # Parsing.
 
@@ -424,63 +454,117 @@ class Model(object):
         if not isinstance(string, basestring):
             raise TypeError("_parse() argument must be a string")
 
-        typ = cls._fields.get(key)
-        if typ:
-            return typ.parse(string)
-        else:
-            # Fall back to unparsed string.
-            return string
-
+        return cls._type(key).parse(string)
 
 
 # Database controller and supporting interfaces.
 
-
 class Results(object):
     """An item query result set. Iterating over the collection lazily
     constructs LibModel objects that reflect database rows.
     """
-    def __init__(self, model_class, rows, db, query=None):
+    def __init__(self, model_class, rows, db, query=None, sort=None):
         """Create a result set that will construct objects of type
-        `model_class`, which should be a subclass of `LibModel`, out of
-        the query result mapping in `rows`. The new objects are
-        associated with the database `db`. If `query` is provided, it is
-        used as a predicate to filter the results for a "slow query" that
-        cannot be evaluated by the database directly.
+        `model_class`.
+
+        `model_class` is a subclass of `LibModel` that will be
+        constructed. `rows` is a query result: a list of mappings. The
+        new objects will be associated with the database `db`.
+
+        If `query` is provided, it is used as a predicate to filter the
+        results for a "slow query" that cannot be evaluated by the
+        database directly. If `sort` is provided, it is used to sort the
+        full list of results before returning. This means it is a "slow
+        sort" and all objects must be built before returning the first
+        one.
         """
         self.model_class = model_class
         self.rows = rows
         self.db = db
         self.query = query
+        self.sort = sort
+
+        # We keep a queue of rows we haven't yet consumed for
+        # materialization. We preserve the original total number of
+        # rows.
+        self._rows = rows
+        self._row_count = len(rows)
+
+        # The materialized objects corresponding to rows that have been
+        # consumed.
+        self._objects = []
+
+    def _get_objects(self):
+        """Construct and generate Model objects for they query. The
+        objects are returned in the order emitted from the database; no
+        slow sort is applied.
+
+        For performance, this generator caches materialized objects to
+        avoid constructing them more than once. This way, iterating over
+        a `Results` object a second time should be much faster than the
+        first.
+        """
+        index = 0  # Position in the materialized objects.
+        while index < len(self._objects) or self._rows:
+            # Are there previously-materialized objects to produce?
+            if index < len(self._objects):
+                yield self._objects[index]
+                index += 1
+
+            # Otherwise, we consume another row, materialize its object
+            # and produce it.
+            else:
+                while self._rows:
+                    row = self._rows.pop(0)
+                    obj = self._make_model(row)
+                    # If there is a slow-query predicate, ensurer that the
+                    # object passes it.
+                    if not self.query or self.query.match(obj):
+                        self._objects.append(obj)
+                        index += 1
+                        yield obj
+                        break
 
     def __iter__(self):
-        """Construct Python objects for all rows that pass the query
-        predicate.
+        """Construct and generate Model objects for all matching
+        objects, in sorted order.
         """
-        for row in self.rows:
-            # Get the flexible attributes for the object.
-            with self.db.transaction() as tx:
-                flex_rows = tx.query(
-                    'SELECT * FROM {0} WHERE entity_id=?'.format(
-                        self.model_class._flex_table
-                    ),
-                    (row['id'],)
-                )
-            values = dict(row)
-            values.update(
-                dict((row['key'], row['value']) for row in flex_rows)
+        if self.sort:
+            # Slow sort. Must build the full list first.
+            objects = self.sort.sort(list(self._get_objects()))
+            return iter(objects)
+
+        else:
+            # Objects are pre-sorted (i.e., by the database).
+            return self._get_objects()
+
+    def _make_model(self, row):
+        # Get the flexible attributes for the object.
+        with self.db.transaction() as tx:
+            flex_rows = tx.query(
+                'SELECT * FROM {0} WHERE entity_id=?'.format(
+                    self.model_class._flex_table
+                ),
+                (row['id'],)
             )
 
-            # Construct the Python object and yield it if it passes the
-            # predicate.
-            obj = self.model_class(self.db, **values)
-            if not self.query or self.query.match(obj):
-                yield obj
+        cols = dict(row)
+        values = dict((k, v) for (k, v) in cols.items()
+                      if not k[:4] == 'flex')
+        flex_values = dict((row['key'], row['value']) for row in flex_rows)
+
+        # Construct the Python object
+        obj = self.model_class._awaken(self.db, values, flex_values)
+        return obj
 
     def __len__(self):
         """Get the number of matching objects.
         """
-        if self.query:
+        if not self._rows:
+            # Fully materialized. Just count the objects.
+            return len(self._objects)
+
+        elif self.query:
             # A slow query. Fall back to testing every object.
             count = 0
             for obj in self:
@@ -489,7 +573,7 @@ class Results(object):
 
         else:
             # A fast query. Just count the rows.
-            return len(self.rows)
+            return self._row_count
 
     def __nonzero__(self):
         """Does this result contain any objects?
@@ -500,6 +584,11 @@ class Results(object):
         """Get the nth item in this result set. This is inefficient: all
         items up to n are materialized and thrown away.
         """
+        if not self._rows and not self.sort:
+            # Fully materialized and already in order. Just look up the
+            # object.
+            return self._objects[n]
+
         it = iter(self)
         try:
             for i in range(n):
@@ -604,7 +693,6 @@ class Database(object):
             self._make_table(model_cls._table, model_cls._fields)
             self._make_attribute_table(model_cls._flex_table)
 
-
     # Primitive access control: connections and transactions.
 
     def _connection(self):
@@ -644,7 +732,6 @@ class Database(object):
         """
         return Transaction(self)
 
-
     # Schema setup and migration.
 
     def _make_table(self, table, fields):
@@ -698,27 +785,33 @@ class Database(object):
                     ON {0} (entity_id);
                 """.format(flex_table))
 
-
     # Querying.
 
-    def _fetch(self, model_cls, query, order_by=None):
+    def _fetch(self, model_cls, query=None, sort=None):
         """Fetch the objects of type `model_cls` matching the given
         query. The query may be given as a string, string sequence, a
-        Query object, or None (to fetch everything). If provided,
-        `order_by` is a SQLite ORDER BY clause for sorting.
+        Query object, or None (to fetch everything). `sort` is an
+        `Sort` object.
         """
+        query = query or TrueQuery()  # A null query.
+        sort = sort or NullSort()  # Unsorted.
         where, subvals = query.clause()
+        order_by = sort.order_clause()
 
-        sql = "SELECT * FROM {0} WHERE {1}".format(
+        sql = ("SELECT * FROM {0} WHERE {1} {2}").format(
             model_cls._table,
             where or '1',
+            "ORDER BY {0}".format(order_by) if order_by else '',
         )
-        if order_by:
-            sql += " ORDER BY {0}".format(order_by)
+
         with self.transaction() as tx:
             rows = tx.query(sql, subvals)
 
-        return Results(model_cls, rows, self, None if where else query)
+        return Results(
+            model_cls, rows, self,
+            None if where else query,  # Slow query component.
+            sort if sort.is_slow() else None,  # Slow sort component.
+        )
 
     def _get(self, model_cls, id):
         """Get a Model object by its id or None if the id does not

lib/beets/dbcore/query.py

@@ -15,6 +15,7 @@
 """The Query type hierarchy for DBCore.
 """
 import re
+from operator import attrgetter
 from beets import util
 from datetime import datetime, timedelta
 
@@ -82,6 +83,23 @@ class MatchQuery(FieldQuery):
         return pattern == value
 
 
+class NoneQuery(FieldQuery):
+
+    def __init__(self, field, fast=True):
+        self.field = field
+        self.fast = fast
+
+    def col_clause(self):
+        return self.field + " IS NULL", ()
+
+    @classmethod
+    def match(self, item):
+        try:
+            return item[self.field] is None
+        except KeyError:
+            return True
+
+
 class StringFieldQuery(FieldQuery):
     """A FieldQuery that converts values to strings before matching
     them.
@@ -104,8 +122,11 @@ class StringFieldQuery(FieldQuery):
 class SubstringQuery(StringFieldQuery):
     """A query that matches a substring in a specific item field."""
     def col_clause(self):
-        search = '%' + (self.pattern.replace('\\','\\\\').replace('%','\\%')
-                            .replace('_','\\_')) + '%'
+        pattern = (self.pattern
+                       .replace('\\', '\\\\')
+                       .replace('%', '\\%')
+                       .replace('_', '\\_'))
+        search = '%' + pattern + '%'
         clause = self.field + " like ? escape '\\'"
         subvals = [search]
         return clause, subvals
@@ -200,7 +221,9 @@ class NumericQuery(FieldQuery):
             self.rangemax = self._convert(parts[1])
 
     def match(self, item):
-        value = getattr(item, self.field)
+        if self.field not in item:
+            return False
+        value = item[self.field]
         if isinstance(value, basestring):
             value = self._convert(value)
 
@@ -236,12 +259,16 @@ class CollectionQuery(Query):
         self.subqueries = subqueries
 
     # Act like a sequence.
+
     def __len__(self):
         return len(self.subqueries)
+
     def __getitem__(self, key):
         return self.subqueries[key]
+
     def __iter__(self):
         return iter(self.subqueries)
+
     def __contains__(self, item):
         return item in self.subqueries
 
@@ -334,10 +361,8 @@ class FalseQuery(Query):
         return False
 
 
-
 # Time/date queries.
 
-
 def _to_epoch_time(date):
     """Convert a `datetime` object to an integer number of seconds since
     the (local) Unix epoch.
@@ -393,10 +418,14 @@ class Period(object):
             return None
         ordinal = string.count('-')
         if ordinal >= len(cls.date_formats):
-            raise ValueError('date is not in one of the formats '
-                             + ', '.join(cls.date_formats))
+            # Too many components.
+            return None
         date_format = cls.date_formats[ordinal]
-        date = datetime.strptime(string, date_format)
+        try:
+            date = datetime.strptime(string, date_format)
+        except ValueError:
+            # Parsing failed.
+            return None
         precision = cls.precisions[ordinal]
         return cls(date, precision)
 
@@ -492,3 +521,134 @@ class DateQuery(FieldQuery):
             # Match any date.
             clause = '1'
         return clause, subvals
+
+
+# Sorting.
+
+class Sort(object):
+    """An abstract class representing a sort operation for a query into
+    the item database.
+    """
+
+    def order_clause(self):
+        """Generates a SQL fragment to be used in a ORDER BY clause, or
+        None if no fragment is used (i.e., this is a slow sort).
+        """
+        return None
+
+    def sort(self, items):
+        """Sort the list of objects and return a list.
+        """
+        return sorted(items)
+
+    def is_slow(self):
+        """Indicate whether this query is *slow*, meaning that it cannot
+        be executed in SQL and must be executed in Python.
+        """
+        return False
+
+
+class MultipleSort(Sort):
+    """Sort that encapsulates multiple sub-sorts.
+    """
+
+    def __init__(self, sorts=None):
+        self.sorts = sorts or []
+
+    def add_sort(self, sort):
+        self.sorts.append(sort)
+
+    def _sql_sorts(self):
+        """Return the list of sub-sorts for which we can be (at least
+        partially) fast.
+
+        A contiguous suffix of fast (SQL-capable) sub-sorts are
+        executable in SQL. The remaining, even if they are fast
+        independently, must be executed slowly.
+        """
+        sql_sorts = []
+        for sort in reversed(self.sorts):
+            if not sort.order_clause() is None:
+                sql_sorts.append(sort)
+            else:
+                break
+        sql_sorts.reverse()
+        return sql_sorts
+
+    def order_clause(self):
+        order_strings = []
+        for sort in self._sql_sorts():
+            order = sort.order_clause()
+            order_strings.append(order)
+
+        return ", ".join(order_strings)
+
+    def is_slow(self):
+        for sort in self.sorts:
+            if sort.is_slow():
+                return True
+        return False
+
+    def sort(self, items):
+        slow_sorts = []
+        switch_slow = False
+        for sort in reversed(self.sorts):
+            if switch_slow:
+                slow_sorts.append(sort)
+            elif sort.order_clause() is None:
+                switch_slow = True
+                slow_sorts.append(sort)
+            else:
+                pass
+
+        for sort in slow_sorts:
+            items = sort.sort(items)
+        return items
+
+    def __repr__(self):
+        return u'MultipleSort({0})'.format(repr(self.sorts))
+
+
+class FieldSort(Sort):
+    """An abstract sort criterion that orders by a specific field (of
+    any kind).
+    """
+    def __init__(self, field, ascending=True):
+        self.field = field
+        self.ascending = ascending
+
+    def sort(self, objs):
+        # TODO: Conversion and null-detection here. In Python 3,
+        # comparisons with None fail. We should also support flexible
+        # attributes with different types without falling over.
+        return sorted(objs, key=attrgetter(self.field),
+                      reverse=not self.ascending)
+
+    def __repr__(self):
+        return u'<{0}: {1}{2}>'.format(
+            type(self).__name__,
+            self.field,
+            '+' if self.ascending else '-',
+        )
+
+
+class FixedFieldSort(FieldSort):
+    """Sort object to sort on a fixed field.
+    """
+    def order_clause(self):
+        order = "ASC" if self.ascending else "DESC"
+        return "{0} {1}".format(self.field, order)
+
+
+class SlowFieldSort(FieldSort):
+    """A sort criterion by some model field other than a fixed field:
+    i.e., a computed or flexible field.
+    """
+    def is_slow(self):
+        return True
+
+
+class NullSort(Sort):
+    """No sorting. Leave results unsorted."""
+    def sort(items):
+        return items

lib/beets/dbcore/queryparse.py

@@ -0,0 +1,180 @@
+# This file is part of beets.
+# Copyright 2014, Adrian Sampson.
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+
+"""Parsing of strings into DBCore queries.
+"""
+import re
+import itertools
+from . import query
+
+
+PARSE_QUERY_PART_REGEX = re.compile(
+    # Non-capturing optional segment for the keyword.
+    r'(?:'
+    r'(\S+?)'    # The field key.
+    r'(?<!\\):'  # Unescaped :
+    r')?'
+
+    r'(.*)',         # The term itself.
+
+    re.I  # Case-insensitive.
+)
+
+
+def parse_query_part(part, query_classes={}, prefixes={},
+                     default_class=query.SubstringQuery):
+    """Take a query in the form of a key/value pair separated by a
+    colon and return a tuple of `(key, value, cls)`. `key` may be None,
+    indicating that any field may be matched. `cls` is a subclass of
+    `FieldQuery`.
+
+    The optional `query_classes` parameter maps field names to default
+    query types; `default_class` is the fallback. `prefixes` is a map
+    from query prefix markers and query types. Prefix-indicated queries
+    take precedence over type-based queries.
+
+    To determine the query class, two factors are used: prefixes and
+    field types. For example, the colon prefix denotes a regular
+    expression query and a type map might provide a special kind of
+    query for numeric values. If neither a prefix nor a specific query
+    class is available, `default_class` is used.
+
+    For instance,
+    'stapler' -> (None, 'stapler', SubstringQuery)
+    'color:red' -> ('color', 'red', SubstringQuery)
+    ':^Quiet' -> (None, '^Quiet', RegexpQuery)
+    'color::b..e' -> ('color', 'b..e', RegexpQuery)
+
+    Prefixes may be "escaped" with a backslash to disable the keying
+    behavior.
+    """
+    part = part.strip()
+    match = PARSE_QUERY_PART_REGEX.match(part)
+
+    assert match  # Regex should always match.
+    key = match.group(1)
+    term = match.group(2).replace('\:', ':')
+
+    # Match the search term against the list of prefixes.
+    for pre, query_class in prefixes.items():
+        if term.startswith(pre):
+            return key, term[len(pre):], query_class
+
+    # No matching prefix: use type-based or fallback/default query.
+    query_class = query_classes.get(key, default_class)
+    return key, term, query_class
+
+
+def construct_query_part(model_cls, prefixes, query_part):
+    """Create a query from a single query component, `query_part`, for
+    querying instances of `model_cls`. Return a `Query` instance.
+    """
+    # Shortcut for empty query parts.
+    if not query_part:
+        return query.TrueQuery()
+
+    # Get the query classes for each possible field.
+    query_classes = {}
+    for k, t in itertools.chain(model_cls._fields.items(),
+                                model_cls._types.items()):
+        query_classes[k] = t.query
+
+    # Parse the string.
+    key, pattern, query_class = \
+        parse_query_part(query_part, query_classes, prefixes)
+
+    # No key specified.
+    if key is None:
+        if issubclass(query_class, query.FieldQuery):
+            # The query type matches a specific field, but none was
+            # specified. So we use a version of the query that matches
+            # any field.
+            return query.AnyFieldQuery(pattern, model_cls._search_fields,
+                                       query_class)
+        else:
+            # Other query type.
+            return query_class(pattern)
+
+    key = key.lower()
+    return query_class(key.lower(), pattern, key in model_cls._fields)
+
+
+def query_from_strings(query_cls, model_cls, prefixes, query_parts):
+    """Creates a collection query of type `query_cls` from a list of
+    strings in the format used by parse_query_part. `model_cls`
+    determines how queries are constructed from strings.
+    """
+    subqueries = []
+    for part in query_parts:
+        subqueries.append(construct_query_part(model_cls, prefixes, part))
+    if not subqueries:  # No terms in query.
+        subqueries = [query.TrueQuery()]
+    return query_cls(subqueries)
+
+
+def construct_sort_part(model_cls, part):
+    """Create a `Sort` from a single string criterion.
+
+    `model_cls` is the `Model` being queried. `part` is a single string
+    ending in ``+`` or ``-`` indicating the sort.
+    """
+    assert part, "part must be a field name and + or -"
+    field = part[:-1]
+    assert field, "field is missing"
+    direction = part[-1]
+    assert direction in ('+', '-'), "part must end with + or -"
+    is_ascending = direction == '+'
+
+    if field in model_cls._sorts:
+        sort = model_cls._sorts[field](model_cls, is_ascending)
+    elif field in model_cls._fields:
+        sort = query.FixedFieldSort(field, is_ascending)
+    else:
+        # Flexible or computed.
+        sort = query.SlowFieldSort(field, is_ascending)
+    return sort
+
+
+def sort_from_strings(model_cls, sort_parts):
+    """Create a `Sort` from a list of sort criteria (strings).
+    """
+    if not sort_parts:
+        return query.NullSort()
+    else:
+        sort = query.MultipleSort()
+        for part in sort_parts:
+            sort.add_sort(construct_sort_part(model_cls, part))
+        return sort
+
+
+def parse_sorted_query(model_cls, parts, prefixes={},
+                       query_cls=query.AndQuery):
+    """Given a list of strings, create the `Query` and `Sort` that they
+    represent.
+    """
+    # Separate query token and sort token.
+    query_parts = []
+    sort_parts = []
+    for part in parts:
+        if part.endswith((u'+', u'-')) and u':' not in part:
+            sort_parts.append(part)
+        else:
+            query_parts.append(part)
+
+    # Parse each.
+    q = query_from_strings(
+        query_cls, model_cls, prefixes, query_parts
+    )
+    s = sort_from_strings(model_cls, sort_parts)
+    return q, s

lib/beets/dbcore/types.py

@@ -18,55 +18,111 @@ from . import query
 from beets.util import str2bool
 
 
-
 # Abstract base.
 
-
 class Type(object):
     """An object encapsulating the type of a model field. Includes
-    information about how to store the value in the database, query,
-    format, and parse a given field.
+    information about how to store, query, format, and parse a given
+    field.
     """
 
-    sql = None
+    sql = u'TEXT'
     """The SQLite column type for the value.
     """
 
-    query = None
+    query = query.SubstringQuery
     """The `Query` subclass to be used when querying the field.
     """
 
+    model_type = unicode
+    """The Python type that is used to represent the value in the model.
+
+    The model is guaranteed to return a value of this type if the field
+    is accessed.  To this end, the constructor is used by the `normalize`
+    and `from_sql` methods and the `default` property.
+    """
+
+    @property
+    def null(self):
+        """The value to be exposed when the underlying value is None.
+        """
+        return self.model_type()
+
     def format(self, value):
         """Given a value of this type, produce a Unicode string
         representing the value. This is used in template evaluation.
         """
-        raise NotImplementedError()
+        if value is None:
+            value = self.null
+        # `self.null` might be `None`
+        if value is None:
+            value = u''
+        if isinstance(value, bytes):
+            value = value.decode('utf8', 'ignore')
+
+        return unicode(value)
 
     def parse(self, string):
         """Parse a (possibly human-written) string and return the
         indicated value of this type.
         """
-        raise NotImplementedError()
+        try:
+            return self.model_type(string)
+        except ValueError:
+            return self.null
 
+    def normalize(self, value):
+        """Given a value that will be assigned into a field of this
+        type, normalize the value to have the appropriate type. This
+        base implementation only reinterprets `None`.
+        """
+        if value is None:
+            return self.null
+        else:
+            # TODO This should eventually be replaced by
+            # `self.model_type(value)`
+            return value
+
+    def from_sql(self, sql_value):
+        """Receives the value stored in the SQL backend and return the
+        value to be stored in the model.
+
+        For fixed fields the type of `value` is determined by the column
+        type affinity given in the `sql` property and the SQL to Python
+        mapping of the database adapter. For more information see:
+        http://www.sqlite.org/datatype3.html
+        https://docs.python.org/2/library/sqlite3.html#sqlite-and-python-types
+
+        Flexible fields have the type afinity `TEXT`. This means the
+        `sql_value` is either a `buffer` or a `unicode` object` and the
+        method must handle these in addition.
+        """
+        if isinstance(sql_value, buffer):
+            sql_value = bytes(sql_value).decode('utf8', 'ignore')
+        if isinstance(sql_value, unicode):
+            return self.parse(sql_value)
+        else:
+            return self.normalize(sql_value)
+
+    def to_sql(self, model_value):
+        """Convert a value as stored in the model object to a value used
+        by the database adapter.
+        """
+        return model_value
 
 
 # Reusable types.
 
+class Default(Type):
+    null = None
+
 
 class Integer(Type):
     """A basic integer type.
     """
     sql = u'INTEGER'
     query = query.NumericQuery
-
-    def format(self, value):
-        return unicode(value or 0)
-
-    def parse(self, string):
-        try:
-            return int(string)
-        except ValueError:
-            return 0
+    model_type = int
 
 
 class PaddedInt(Integer):
@@ -93,9 +149,14 @@ class ScaledInt(Integer):
 
 
 class Id(Integer):
-    """An integer used as the row key for a SQLite table.
+    """An integer used as the row id or a foreign key in a SQLite table.
+    This type is nullable: None values are not translated to zero.
     """
-    sql = u'INTEGER PRIMARY KEY'
+    null = None
+
+    def __init__(self, primary=True):
+        if primary:
+            self.sql = u'INTEGER PRIMARY KEY'
 
 
 class Float(Type):
@@ -103,15 +164,16 @@ class Float(Type):
     """
     sql = u'REAL'
     query = query.NumericQuery
+    model_type = float
 
     def format(self, value):
         return u'{0:.1f}'.format(value or 0.0)
 
-    def parse(self, string):
-        try:
-            return float(string)
-        except ValueError:
-            return 0.0
+
+class NullFloat(Float):
+    """Same as `Float`, but does not normalize `None` to `0.0`.
+    """
+    null = None
 
 
 class String(Type):
@@ -120,21 +182,27 @@ class String(Type):
     sql = u'TEXT'
     query = query.SubstringQuery
 
-    def format(self, value):
-        return unicode(value) if value else u''
-
-    def parse(self, string):
-        return string
-
 
 class Boolean(Type):
     """A boolean type.
     """
     sql = u'INTEGER'
     query = query.BooleanQuery
+    model_type = bool
 
     def format(self, value):
         return unicode(bool(value))
 
     def parse(self, string):
         return str2bool(string)
+
+
+# Shared instances of common types.
+DEFAULT = Default()
+INTEGER = Integer()
+PRIMARY_ID = Id(True)
+FOREIGN_ID = Id(False)
+FLOAT = Float()
+NULL_FLOAT = NullFloat()
+STRING = String()
+BOOLEAN = Boolean()

lib/beets/plugins.py

@@ -16,8 +16,11 @@
 
 import logging
 import traceback
+import inspect
+import re
 from collections import defaultdict
 
+
 import beets
 from beets import mediafile
 
@@ -30,6 +33,14 @@ LASTFM_KEY = '2dc3914abf35f0d9c92d97d8f8e42b43'
 log = logging.getLogger('beets')
 
 
+class PluginConflictException(Exception):
+    """Indicates that the services provided by one plugin conflict with
+    those of another.
+
+    For example two plugins may define different types for flexible fields.
+    """
+
+
 # Managing the plugins themselves.
 
 class BeetsPlugin(object):
@@ -40,7 +51,6 @@ class BeetsPlugin(object):
     def __init__(self, name=None):
         """Perform one-time plugin setup.
         """
-        _add_media_fields(self.item_fields())
         self.import_stages = []
         self.name = name or self.__module__.split('.')[-1]
         self.config = beets.config[self.name]
@@ -86,14 +96,6 @@ class BeetsPlugin(object):
         """
         return ()
 
-    def item_fields(self):
-        """Returns field descriptors to be added to the MediaFile class,
-        in the form of a dictionary whose keys are field names and whose
-        values are descriptor (e.g., MediaField) instances. The Library
-        database schema is not (currently) extended.
-        """
-        return {}
-
     def album_for_id(self, album_id):
         """Return an AlbumInfo object or None if no matching release was
         found.
@@ -106,6 +108,20 @@ class BeetsPlugin(object):
         """
         return None
 
+    def add_media_field(self, name, descriptor):
+        """Add a field that is synchronized between media files and items.
+
+        When a media field is added ``item.write()`` will set the name
+        property of the item's MediaFile to ``item[name]`` and save the
+        changes. Similarly ``item.read()`` will set ``item[name]`` to
+        the value of the name property of the media file.
+
+        ``descriptor`` must be an instance of ``mediafile.MediaField``.
+        """
+        # Defer impor to prevent circular dependency
+        from beets import library
+        mediafile.MediaFile.add_field(name, descriptor)
+        library.Item._media_fields.add(name)
 
     listeners = None
 
@@ -130,7 +146,7 @@ class BeetsPlugin(object):
 
             >>> @MyPlugin.listen("imported")
             >>> def importListener(**kwargs):
-            >>>     pass
+            ...     pass
         """
         def helper(func):
             if cls.listeners is None:
@@ -170,7 +186,10 @@ class BeetsPlugin(object):
             return func
         return helper
 
+
 _classes = set()
+
+
 def load_plugins(names=()):
     """Imports the modules for a sequence of plugin names. Each name
     must be the name of a Python module under the "beetsplug" namespace
@@ -185,7 +204,7 @@ def load_plugins(names=()):
             except ImportError as exc:
                 # Again, this is hacky:
                 if exc.args[0].endswith(' ' + name):
-                    log.warn('** plugin %s not found' % name)
+                    log.warn(u'** plugin {0} not found'.format(name))
                 else:
                     raise
             else:
@@ -195,10 +214,13 @@ def load_plugins(names=()):
                         _classes.add(obj)
 
         except:
-            log.warn('** error loading plugin %s' % name)
+            log.warn(u'** error loading plugin {0}'.format(name))
             log.warn(traceback.format_exc())
 
+
 _instances = {}
+
+
 def find_plugins():
     """Returns a list of BeetsPlugin subclass instances from all
     currently loaded beets plugins. Loads the default plugin set
@@ -224,6 +246,7 @@ def commands():
         out += plugin.commands()
     return out
 
+
 def queries():
     """Returns a dict mapping prefix strings to Query subclasses all loaded
     plugins.
@@ -233,6 +256,24 @@ def queries():
         out.update(plugin.queries())
     return out
 
+
+def types(model_cls):
+    # Gives us `item_types` and `album_types`
+    attr_name = '{0}_types'.format(model_cls.__name__.lower())
+    types = {}
+    for plugin in find_plugins():
+        plugin_types = getattr(plugin, attr_name, {})
+        for field in plugin_types:
+            if field in types and plugin_types[field] != types[field]:
+                raise PluginConflictException(
+                    u'Plugin {0} defines flexible field {1} '
+                    'which has already been defined with '
+                    'another type.'.format(plugin.name, field)
+                )
+        types.update(plugin_types)
+    return types
+
+
 def track_distance(item, info):
     """Gets the track distance calculated by all loaded plugins.
     Returns a Distance object.
@@ -243,6 +284,7 @@ def track_distance(item, info):
         dist.update(plugin.track_distance(item, info))
     return dist
 
+
 def album_distance(items, album_info, mapping):
     """Returns the album distance calculated by plugins."""
     from beets.autotag.hooks import Distance
@@ -251,6 +293,7 @@ def album_distance(items, album_info, mapping):
         dist.update(plugin.album_distance(items, album_info, mapping))
     return dist
 
+
 def candidates(items, artist, album, va_likely):
     """Gets MusicBrainz candidates for an album from each plugin.
     """
@@ -259,6 +302,7 @@ def candidates(items, artist, album, va_likely):
         out.extend(plugin.candidates(items, artist, album, va_likely))
     return out
 
+
 def item_candidates(item, artist, title):
     """Gets MusicBrainz candidates for an item from the plugins.
     """
@@ -267,6 +311,7 @@ def item_candidates(item, artist, title):
         out.extend(plugin.item_candidates(item, artist, title))
     return out
 
+
 def album_for_id(album_id):
     """Get AlbumInfo objects for a given ID string.
     """
@@ -277,6 +322,7 @@ def album_for_id(album_id):
             out.append(res)
     return out
 
+
 def track_for_id(track_id):
     """Get TrackInfo objects for a given ID string.
     """
@@ -287,6 +333,7 @@ def track_for_id(track_id):
             out.append(res)
     return out
 
+
 def template_funcs():
     """Get all the template functions declared by plugins as a
     dictionary.
@@ -297,12 +344,6 @@ def template_funcs():
             funcs.update(plugin.template_funcs)
     return funcs
 
-def _add_media_fields(fields):
-    """Adds a {name: descriptor} dictionary of fields to the MediaFile
-    class. Called during the plugin initialization.
-    """
-    for key, value in fields.iteritems():
-        setattr(mediafile.MediaFile, key, value)
 
 def import_stages():
     """Get a list of import stage functions defined by plugins."""
@@ -325,6 +366,7 @@ def item_field_getters():
             funcs.update(plugin.template_fields)
     return funcs
 
+
 def album_field_getters():
     """As above, for album fields.
     """
@@ -348,6 +390,7 @@ def event_handlers():
                 all_handlers[event] += handlers
     return all_handlers
 
+
 def send(event, **arguments):
     """Sends an event to all assigned event listeners. Event is the
     name of  the event to send, all other named arguments go to the
@@ -355,5 +398,38 @@ def send(event, **arguments):
 
     Returns a list of return values from the handlers.
     """
-    log.debug('Sending event: %s' % event)
-    return [handler(**arguments) for handler in event_handlers()[event]]
+    log.debug(u'Sending event: {0}'.format(event))
+    for handler in event_handlers()[event]:
+        # Don't break legacy plugins if we want to pass more arguments
+        argspec = inspect.getargspec(handler).args
+        args = dict((k, v) for k, v in arguments.items() if k in argspec)
+        handler(**args)
+
+
+def feat_tokens(for_artist=True):
+    """Return a regular expression that matches phrases like "featuring"
+    that separate a main artist or a song title from secondary artists.
+    The `for_artist` option determines whether the regex should be
+    suitable for matching artist fields (the default) or title fields.
+    """
+    feat_words = ['ft', 'featuring', 'feat', 'feat.', 'ft.']
+    if for_artist:
+        feat_words += ['with', 'vs', 'and', 'con', '&']
+    return '(?<=\s)(?:{0})(?=\s)'.format(
+        '|'.join(re.escape(x) for x in feat_words)
+    )
+
+
+def sanitize_choices(choices, choices_all):
+    """Clean up a stringlist configuration attribute: keep only choices
+    elements present in choices_all, remove duplicate elements, expand '*'
+    wildcard while keeping original stringlist order.
+    """
+    seen = set()
+    others = [x for x in choices_all if x not in choices]
+    res = []
+    for s in choices:
+        if s in list(choices_all) + ['*']:
+            if not (s in seen or seen.add(s)):
+                res.extend(list(others) if s == '*' else [s])
+    return res

lib/beets/ui/__init__.py

@@ -29,6 +29,7 @@ import errno
 import re
 import struct
 import traceback
+import os.path
 
 from beets import library
 from beets import plugins
@@ -38,9 +39,7 @@ from beets import config
 from beets.util import confit
 from beets.autotag import mb
 
-
 # On Windows platforms, use colorama to support "ANSI" terminal colors.
-
 if sys.platform == 'win32':
     try:
         import colorama
@@ -50,8 +49,10 @@ if sys.platform == 'win32':
         colorama.init()
 
 
-
-# Constants.
+log = logging.getLogger('beets')
+if not log.handlers:
+    log.addHandler(logging.StreamHandler())
+log.propagate = False  # Don't propagate to root handler.
 
 
 PF_KEY_QUERIES = {
@@ -59,19 +60,15 @@ PF_KEY_QUERIES = {
     'singleton': 'singleton:true',
 }
 
-# UI exception. Commands should throw this in order to display
-# nonrecoverable errors to the user.
+
 class UserError(Exception):
-    pass
-
-# Main logger.
-log = logging.getLogger('beets')
-
+    """UI exception. Commands should throw this in order to display
+    nonrecoverable errors to the user.
+    """
 
 
 # Utilities.
 
-
 def _encoding():
     """Tries to guess the encoding used by the terminal."""
     # Configured override?
@@ -170,7 +167,7 @@ def input_options(options, require=False, prompt=None, fallback_prompt=None,
             # Infer a letter.
             for letter in option:
                 if not letter.isalpha():
-                    continue # Don't use punctuation.
+                    continue  # Don't use punctuation.
                 if letter not in letters:
                     found_letter = letter
                     break
@@ -181,9 +178,10 @@ def input_options(options, require=False, prompt=None, fallback_prompt=None,
         index = option.index(found_letter)
 
         # Mark the option's shortcut letter for display.
-        if not require and ((default is None and not numrange and first) or
-                (isinstance(default, basestring) and
-                 found_letter.lower() == default.lower())):
+        if not require and (
+            (default is None and not numrange and first) or
+            (isinstance(default, basestring) and
+             found_letter.lower() == default.lower())):
             # The first option is the default; mark it.
             show_letter = '[%s]' % found_letter.upper()
             is_default = True
@@ -352,11 +350,13 @@ def human_seconds_short(interval):
 # http://dev.pocoo.org/hg/pygments-main/file/b2deea5b5030/pygments/console.py
 # (pygments is by Tim Hatch, Armin Ronacher, et al.)
 COLOR_ESCAPE = "\x1b["
-DARK_COLORS  = ["black", "darkred", "darkgreen", "brown", "darkblue",
-                "purple", "teal", "lightgray"]
+DARK_COLORS = ["black", "darkred", "darkgreen", "brown", "darkblue",
+               "purple", "teal", "lightgray"]
 LIGHT_COLORS = ["darkgray", "red", "green", "yellow", "blue",
                 "fuchsia", "turquoise", "white"]
 RESET_COLOR = COLOR_ESCAPE + "39;49;00m"
+
+
 def _colorize(color, text):
     """Returns a string that prints the given text in the given color
     in a terminal that is ANSI color-aware. The color must be something
@@ -441,30 +441,6 @@ def colordiff(a, b, highlight='red'):
         return unicode(a), unicode(b)
 
 
-def color_diff_suffix(a, b, highlight='red'):
-    """Colorize the differing suffix between two strings."""
-    a, b = unicode(a), unicode(b)
-    if not config['color']:
-        return a, b
-
-    # Fast path.
-    if a == b:
-        return a, b
-
-    # Find the longest common prefix.
-    first_diff = None
-    for i in range(min(len(a), len(b))):
-        if a[i] != b[i]:
-            first_diff = i
-            break
-    else:
-        first_diff = min(len(a), len(b))
-
-    # Colorize from the first difference on.
-    return a[:first_diff] + colorize(highlight, a[first_diff:]), \
-           b[:first_diff] + colorize(highlight, b[first_diff:])
-
-
 def get_path_formats(subview=None):
     """Get the configuration's path formats as a list of query/template
     pairs.
@@ -494,21 +470,6 @@ def get_replacements():
     return replacements
 
 
-def get_plugin_paths():
-    """Get the list of search paths for plugins from the config file.
-    The value for "pluginpath" may be a single string or a list of
-    strings.
-    """
-    pluginpaths = config['pluginpath'].get()
-    if isinstance(pluginpaths, basestring):
-        pluginpaths = [pluginpaths]
-    if not isinstance(pluginpaths, list):
-        raise confit.ConfigTypeError(
-            u'pluginpath must be string or a list of strings'
-        )
-    return map(util.normpath, pluginpaths)
-
-
 def _pick_format(album, fmt=None):
     """Pick a format string for printing Album or Item objects,
     falling back to config options and defaults.
@@ -558,6 +519,8 @@ def term_width():
 
 
 FLOAT_EPSILON = 0.01
+
+
 def _field_diff(field, old, new):
     """Given two Model objects, format their values for `field` and
     highlight changes among them. Return a human-readable string. If the
@@ -574,13 +537,13 @@ def _field_diff(field, old, new):
         return None
 
     # Get formatted values for output.
-    oldstr = old._get_formatted(field)
-    newstr = new._get_formatted(field)
+    oldstr = old.formatted().get(field, u'')
+    newstr = new.formatted().get(field, u'')
 
     # For strings, highlight changes. For others, colorize the whole
     # thing.
     if isinstance(oldval, basestring):
-        oldstr, newstr = colordiff(oldval, newval)
+        oldstr, newstr = colordiff(oldval, newstr)
     else:
         oldstr, newstr = colorize('red', oldstr), colorize('red', newstr)
 
@@ -613,9 +576,12 @@ def show_model_changes(new, old=None, fields=None, always=False):
 
     # New fields.
     for field in set(new) - set(old):
+        if fields and field not in fields:
+            continue
+
         changes.append(u'  {0}: {1}'.format(
             field,
-            colorize('red', new._get_formatted(field))
+            colorize('red', new.formatted()[field])
         ))
 
     # Print changes.
@@ -627,10 +593,8 @@ def show_model_changes(new, old=None, fields=None, always=False):
     return bool(changes)
 
 
-
 # Subcommand parsing infrastructure.
-
-
+#
 # This is a fairly generic subcommand parser for optparse. It is
 # maintained externally here:
 # http://gist.github.com/462717
@@ -653,46 +617,56 @@ class Subcommand(object):
         self.aliases = aliases
         self.help = help
         self.hide = hide
+        self._root_parser = None
+
+    def print_help(self):
+        self.parser.print_help()
+
+    def parse_args(self, args):
+        return self.parser.parse_args(args)
+
+    @property
+    def root_parser(self):
+        return self._root_parser
+
+    @root_parser.setter
+    def root_parser(self, root_parser):
+        self._root_parser = root_parser
+        self.parser.prog = '{0} {1}'.format(root_parser.get_prog_name(),
+                                            self.name)
+
 
 class SubcommandsOptionParser(optparse.OptionParser):
     """A variant of OptionParser that parses subcommands and their
     arguments.
     """
-    # A singleton command used to give help on other subcommands.
-    _HelpSubcommand = Subcommand('help', optparse.OptionParser(),
-        help='give detailed help on a specific sub-command',
-        aliases=('?',))
 
     def __init__(self, *args, **kwargs):
         """Create a new subcommand-aware option parser. All of the
         options to OptionParser.__init__ are supported in addition
         to subcommands, a sequence of Subcommand objects.
         """
-        # The subcommand array, with the help command included.
-        self.subcommands = list(kwargs.pop('subcommands', []))
-        self.subcommands.append(self._HelpSubcommand)
-
         # A more helpful default usage.
         if 'usage' not in kwargs:
             kwargs['usage'] = """
   %prog COMMAND [ARGS...]
   %prog help COMMAND"""
+        kwargs['add_help_option'] = False
 
         # Super constructor.
         optparse.OptionParser.__init__(self, *args, **kwargs)
 
-        # Adjust the help-visible name of each subcommand.
-        for subcommand in self.subcommands:
-            subcommand.parser.prog = '%s %s' % \
-                    (self.get_prog_name(), subcommand.name)
-
         # Our root parser needs to stop on the first unrecognized argument.
         self.disable_interspersed_args()
 
-    def add_subcommand(self, cmd):
+        self.subcommands = []
+
+    def add_subcommand(self, *cmds):
         """Adds a Subcommand object to the parser's list of commands.
         """
-        self.subcommands.append(cmd)
+        for cmd in cmds:
+            cmd.root_parser = self
+            self.subcommands.append(cmd)
 
     # Add the list of subcommands to the help message.
     def format_help(self, formatter=None):
@@ -711,6 +685,7 @@ class SubcommandsOptionParser(optparse.OptionParser):
         disp_names = []
         help_position = 0
         subcommands = [c for c in self.subcommands if not c.hide]
+        subcommands.sort(key=lambda c: c.name)
         for subcommand in subcommands:
             name = subcommand.name
             if subcommand.aliases:
@@ -756,52 +731,40 @@ class SubcommandsOptionParser(optparse.OptionParser):
                 return subcommand
         return None
 
-    def parse_args(self, a=None, v=None):
-        """Like OptionParser.parse_args, but returns these four items:
-        - options: the options passed to the root parser
-        - subcommand: the Subcommand object that was invoked
-        - suboptions: the options passed to the subcommand parser
-        - subargs: the positional arguments passed to the subcommand
+    def parse_global_options(self, args):
+        """Parse options up to the subcommand argument. Returns a tuple
+        of the options object and the remaining arguments.
         """
-        options, args = optparse.OptionParser.parse_args(self, a, v)
-        subcommand, suboptions, subargs = self._parse_sub(args)
-        return options, subcommand, suboptions, subargs
+        options, subargs = self.parse_args(args)
 
-    def _parse_sub(self, args):
-        """Given the `args` left unused by a typical OptionParser
-        `parse_args`, return the invoked subcommand, the subcommand
-        options, and the subcommand arguments.
+        # Force the help command
+        if options.help:
+            subargs = ['help']
+        elif options.version:
+            subargs = ['version']
+        return options, subargs
+
+    def parse_subcommand(self, args):
+        """Given the `args` left unused by a `parse_global_options`,
+        return the invoked subcommand, the subcommand options, and the
+        subcommand arguments.
         """
+        # Help is default command
         if not args:
-            # No command given.
-            self.print_help()
-            self.exit()
-        else:
-            cmdname = args.pop(0)
-            subcommand = self._subcommand_for_name(cmdname)
-            if not subcommand:
-                self.error('unknown command ' + cmdname)
+            args = ['help']
 
-        suboptions, subargs = subcommand.parser.parse_args(args)
-
-        if subcommand is self._HelpSubcommand:
-            if subargs:
-                # particular
-                cmdname = subargs[0]
-                helpcommand = self._subcommand_for_name(cmdname)
-                if not helpcommand:
-                    self.error('no command named {0}'.format(cmdname))
-                helpcommand.parser.print_help()
-                self.exit()
-            else:
-                # general
-                self.print_help()
-                self.exit()
+        cmdname = args.pop(0)
+        subcommand = self._subcommand_for_name(cmdname)
+        if not subcommand:
+            raise UserError("unknown command '{0}'".format(cmdname))
 
+        suboptions, subargs = subcommand.parse_args(args)
         return subcommand, suboptions, subargs
 
 
 optparse.Option.ALWAYS_TYPED_ACTIONS += ('callback',)
+
+
 def vararg_callback(option, opt_str, value, parser):
     """Callback for an option with variable arguments.
     Manually collect arguments right of a callback-action
@@ -838,53 +801,54 @@ def vararg_callback(option, opt_str, value, parser):
     setattr(parser.values, option.dest, value)
 
 
-
 # The main entry point and bootstrapping.
 
-
-def _load_plugins():
+def _load_plugins(config):
     """Load the plugins specified in the configuration.
     """
-    # Add plugin paths.
+    paths = config['pluginpath'].get(confit.StrSeq(split=False))
+    paths = map(util.normpath, paths)
+
     import beetsplug
-    beetsplug.__path__ = get_plugin_paths() + beetsplug.__path__
-
+    beetsplug.__path__ = paths + beetsplug.__path__
     # For backwards compatibility.
-    sys.path += get_plugin_paths()
+    sys.path += paths
 
-    # Load requested plugins.
     plugins.load_plugins(config['plugins'].as_str_seq())
     plugins.send("pluginload")
+    return plugins
 
 
-def _configure(args):
-    """Parse the command line, load configuration files (including
-    loading any indicated plugins), and return the invoked subcomand,
-    the subcommand options, and the subcommand arguments.
+def _setup(options, lib=None):
+    """Prepare and global state and updates it with command line options.
+
+    Returns a list of subcommands, a list of plugins, and a library instance.
     """
-    # Temporary: Migrate from 1.0-style configuration.
-    from beets.ui import migrate
-    migrate.automigrate()
+    # Configure the MusicBrainz API.
+    mb.configure()
+
+    config = _configure(options)
+
+    plugins = _load_plugins(config)
 
     # Get the default subcommands.
     from beets.ui.commands import default_commands
 
-    # Construct the root parser.
-    commands = list(default_commands)
-    commands.append(migrate.migrate_cmd)  # Temporary.
-    parser = SubcommandsOptionParser(subcommands=commands)
-    parser.add_option('-l', '--library', dest='library',
-                      help='library database file to use')
-    parser.add_option('-d', '--directory', dest='directory',
-                      help="destination music directory")
-    parser.add_option('-v', '--verbose', dest='verbose', action='store_true',
-                      help='print debugging information')
-    parser.add_option('-c', '--config', dest='config',
-                      help='path to configuration file')
+    subcommands = list(default_commands)
+    subcommands.extend(plugins.commands())
 
-    # Parse the command-line!
-    options, args = optparse.OptionParser.parse_args(parser, args)
+    if lib is None:
+        lib = _open_library(config)
+        plugins.send("library_opened", lib=lib)
+    library.Item._types = plugins.types(library.Item)
+    library.Album._types = plugins.types(library.Album)
 
+    return subcommands, plugins, lib
+
+
+def _configure(options):
+    """Amend the global configuration object with command line options.
+    """
     # Add any additional config files specified with --config. This
     # special handling lets specified plugins get loaded before we
     # finish parsing the command line.
@@ -894,22 +858,28 @@ def _configure(args):
         config.set_file(config_path)
     config.set_args(options)
 
-    # Now add the plugin commands to the parser.
-    _load_plugins()
-    for cmd in plugins.commands():
-        parser.add_subcommand(cmd)
+    # Configure the logger.
+    if config['verbose'].get(bool):
+        log.setLevel(logging.DEBUG)
+    else:
+        log.setLevel(logging.INFO)
 
-    # Parse the remainder of the command line with loaded plugins.
-    return parser._parse_sub(args)
+    config_path = config.user_config_path()
+    if os.path.isfile(config_path):
+        log.debug(u'user configuration: {0}'.format(
+            util.displayable_path(config_path)))
+    else:
+        log.debug(u'no user configuration found at {0}'.format(
+            util.displayable_path(config_path)))
+
+    log.debug(u'data directory: {0}'
+              .format(util.displayable_path(config.config_dir())))
+    return config
 
 
-def _raw_main(args):
-    """A helper function for `main` without top-level exception
-    handling.
+def _open_library(config):
+    """Create a new library instance from the configuration.
     """
-    subcommand, suboptions, subargs = _configure(args)
-
-    # Open library file.
     dbpath = config['library'].as_filename()
     try:
         lib = library.Library(
@@ -918,32 +888,52 @@ def _raw_main(args):
             get_path_formats(),
             get_replacements(),
         )
-    except sqlite3.OperationalError:
+        lib.get_item(0)  # Test database connection.
+    except (sqlite3.OperationalError, sqlite3.DatabaseError):
+        log.debug(traceback.format_exc())
         raise UserError(u"database file {0} could not be opened".format(
             util.displayable_path(dbpath)
         ))
-    plugins.send("library_opened", lib=lib)
+    log.debug(u'library database: {0}\n'
+              u'library directory: {1}'
+              .format(util.displayable_path(lib.path),
+                      util.displayable_path(lib.directory)))
+    return lib
 
-    # Configure the logger.
-    if config['verbose'].get(bool):
-        log.setLevel(logging.DEBUG)
-    else:
-        log.setLevel(logging.INFO)
-    log.debug(u'data directory: {0}\n'
-              u'library database: {1}\n'
-              u'library directory: {2}'
-        .format(
-            util.displayable_path(config.config_dir()),
-            util.displayable_path(lib.path),
-            util.displayable_path(lib.directory),
-        )
-    )
 
-    # Configure the MusicBrainz API.
-    mb.configure()
+def _raw_main(args, lib=None):
+    """A helper function for `main` without top-level exception
+    handling.
+    """
+    parser = SubcommandsOptionParser()
+    parser.add_option('-l', '--library', dest='library',
+                      help='library database file to use')
+    parser.add_option('-d', '--directory', dest='directory',
+                      help="destination music directory")
+    parser.add_option('-v', '--verbose', dest='verbose', action='store_true',
+                      help='print debugging information')
+    parser.add_option('-c', '--config', dest='config',
+                      help='path to configuration file')
+    parser.add_option('-h', '--help', dest='help', action='store_true',
+                      help='how this help message and exit')
+    parser.add_option('--version', dest='version', action='store_true',
+                      help=optparse.SUPPRESS_HELP)
 
-    # Invoke the subcommand.
+    options, subargs = parser.parse_global_options(args)
+
+    # Special case for the `config --edit` command: bypass _setup so
+    # that an invalid configuration does not prevent the editor from
+    # starting.
+    if subargs[0] == 'config' and ('-e' in subargs or '--edit' in subargs):
+        from beets.ui.commands import config_edit
+        return config_edit()
+
+    subcommands, plugins, lib = _setup(options, lib)
+    parser.add_subcommand(*subcommands)
+
+    subcommand, suboptions, subargs = parser.parse_subcommand(subargs)
     subcommand.func(lib, suboptions, subargs)
+
     plugins.send('cli_exit', lib=lib)
 
 

lib/beets/ui/migrate.py

@@ -1,401 +0,0 @@
-# This file is part of beets.
-# Copyright 2013, Adrian Sampson.
-#
-# Permission is hereby granted, free of charge, to any person obtaining
-# a copy of this software and associated documentation files (the
-# "Software"), to deal in the Software without restriction, including
-# without limitation the rights to use, copy, modify, merge, publish,
-# distribute, sublicense, and/or sell copies of the Software, and to
-# permit persons to whom the Software is furnished to do so, subject to
-# the following conditions:
-#
-# The above copyright notice and this permission notice shall be
-# included in all copies or substantial portions of the Software.
-
-"""Conversion from legacy (pre-1.1) configuration to Confit/YAML
-configuration.
-"""
-import os
-import ConfigParser
-import codecs
-import yaml
-import logging
-import time
-import itertools
-import re
-
-import beets
-from beets import util
-from beets import ui
-from beets.util import confit
-
-CONFIG_PATH_VAR = 'BEETSCONFIG'
-DEFAULT_CONFIG_FILENAME_UNIX = '.beetsconfig'
-DEFAULT_CONFIG_FILENAME_WINDOWS = 'beetsconfig.ini'
-DEFAULT_LIBRARY_FILENAME_UNIX = '.beetsmusic.blb'
-DEFAULT_LIBRARY_FILENAME_WINDOWS = 'beetsmusic.blb'
-WINDOWS_BASEDIR = os.environ.get('APPDATA') or '~'
-
-OLD_CONFIG_SUFFIX = '.old'
-PLUGIN_NAMES = {
-    'rdm': 'random',
-    'fuzzy_search': 'fuzzy',
-}
-AUTO_KEYS = ('automatic', 'autofetch', 'autoembed', 'autoscrub')
-IMPORTFEEDS_PREFIX = 'feeds_'
-CONFIG_MIGRATED_MESSAGE = u"""
-You appear to be upgrading from beets 1.0 (or earlier) to 1.1. Your
-configuration file has been migrated automatically to:
-{newconfig}
-Edit this file to configure beets. You might want to remove your
-old-style ".beetsconfig" file now. See the documentation for more
-details on the new configuration system:
-http://beets.readthedocs.org/page/reference/config.html
-""".strip()
-DB_MIGRATED_MESSAGE = u'Your database file has also been copied to:\n{newdb}'
-YAML_COMMENT = '# Automatically migrated from legacy .beetsconfig.\n\n'
-
-log = logging.getLogger('beets')
-
-# An itertools recipe.
-def grouper(n, iterable):
-    args = [iter(iterable)] * n
-    return itertools.izip_longest(*args)
-
-def _displace(fn):
-    """Move a file aside using a timestamp suffix so a new file can be
-    put in its place.
-    """
-    util.move(
-        fn,
-        u'{0}.old.{1}'.format(fn, int(time.time())),
-        True
-    )
-
-def default_paths():
-    """Produces the appropriate default config and library database
-    paths for the current system. On Unix, this is always in ~. On
-    Windows, tries ~ first and then $APPDATA for the config and library
-    files (for backwards compatibility).
-    """
-    windows = os.path.__name__ == 'ntpath'
-    if windows:
-        windata = os.environ.get('APPDATA') or '~'
-
-    # Shorthand for joining paths.
-    def exp(*vals):
-        return os.path.expanduser(os.path.join(*vals))
-
-    config = exp('~', DEFAULT_CONFIG_FILENAME_UNIX)
-    if windows and not os.path.exists(config):
-        config = exp(windata, DEFAULT_CONFIG_FILENAME_WINDOWS)
-
-    libpath = exp('~', DEFAULT_LIBRARY_FILENAME_UNIX)
-    if windows and not os.path.exists(libpath):
-        libpath = exp(windata, DEFAULT_LIBRARY_FILENAME_WINDOWS)
-
-    return config, libpath
-
-def get_config():
-    """Using the same logic as beets 1.0, locate and read the
-    .beetsconfig file. Return a ConfigParser instance or None if no
-    config is found.
-    """
-    default_config, default_libpath = default_paths()
-    if CONFIG_PATH_VAR in os.environ:
-        configpath = os.path.expanduser(os.environ[CONFIG_PATH_VAR])
-    else:
-        configpath = default_config
-
-    config = ConfigParser.SafeConfigParser()
-    if os.path.exists(util.syspath(configpath)):
-        with codecs.open(configpath, 'r', encoding='utf-8') as f:
-            config.readfp(f)
-        return config, configpath
-    else:
-        return None, configpath
-
-def flatten_config(config):
-    """Given a ConfigParser, flatten the values into a dict-of-dicts
-    representation where each section gets its own dictionary of values.
-    """
-    out = confit.OrderedDict()
-    for section in config.sections():
-        sec_dict = out[section] = confit.OrderedDict()
-        for option in config.options(section):
-            sec_dict[option] = config.get(section, option, True)
-    return out
-
-def transform_value(value):
-    """Given a string read as the value of a config option, return a
-    massaged version of that value (possibly with a different type).
-    """
-    # Booleans.
-    if value.lower() in ('false', 'no', 'off'):
-        return False
-    elif value.lower() in ('true', 'yes', 'on'):
-        return True
-
-    # Integers.
-    try:
-        return int(value)
-    except ValueError:
-        pass
-
-    # Floats.
-    try:
-        return float(value)
-    except ValueError:
-        pass
-
-    return value
-
-def transform_data(data):
-    """Given a dict-of-dicts representation of legacy config data, tweak
-    the data into a new form. This new form is suitable for dumping as
-    YAML.
-    """
-    out = confit.OrderedDict()
-
-    for section, pairs in data.items():
-        if section == 'beets':
-            # The "main" section. In the new config system, these values
-            # are in the "root": no section at all.
-            for key, value in pairs.items():
-                value = transform_value(value)
-
-                if key.startswith('import_'):
-                    # Importer config is now under an "import:" key.
-                    if 'import' not in out:
-                        out['import'] = confit.OrderedDict()
-                    out['import'][key[7:]] = value
-
-                elif key == 'plugins':
-                    # Renamed plugins.
-                    plugins = value.split()
-                    new_plugins = [PLUGIN_NAMES.get(p, p) for p in plugins]
-                    out['plugins'] = ' '.join(new_plugins)
-
-                elif key == 'replace':
-                    # YAMLy representation for character replacements.
-                    replacements = confit.OrderedDict()
-                    for pat, repl in grouper(2, value.split()):
-                        if repl == '<strip>':
-                            repl = ''
-                        replacements[pat] = repl
-                    out['replace'] = replacements
-
-                elif key == 'pluginpath':
-                    # Used to be a colon-separated string. Now a list.
-                    out['pluginpath'] = value.split(':')
-
-                else:
-                    out[key] = value
-
-        elif pairs:
-            # Other sections (plugins, etc).
-            sec_out = out[section] = confit.OrderedDict()
-            for key, value in pairs.items():
-
-                # Standardized "auto" option.
-                if key in AUTO_KEYS:
-                    key = 'auto'
-
-                # Unnecessary : hack in queries.
-                if section == 'paths':
-                    key = key.replace('_', ':')
-
-                # Changed option names for importfeeds plugin.
-                if section == 'importfeeds':
-                    if key.startswith(IMPORTFEEDS_PREFIX):
-                        key = key[len(IMPORTFEEDS_PREFIX):]
-
-                sec_out[key] = transform_value(value)
-
-    return out
-
-class Dumper(yaml.SafeDumper):
-    """A PyYAML Dumper that represents OrderedDicts as ordinary mappings
-    (in order, of course).
-    """
-    # From http://pyyaml.org/attachment/ticket/161/use_ordered_dict.py
-    def represent_mapping(self, tag, mapping, flow_style=None):
-        value = []
-        node = yaml.MappingNode(tag, value, flow_style=flow_style)
-        if self.alias_key is not None:
-            self.represented_objects[self.alias_key] = node
-        best_style = True
-        if hasattr(mapping, 'items'):
-            mapping = list(mapping.items())
-        for item_key, item_value in mapping:
-            node_key = self.represent_data(item_key)
-            node_value = self.represent_data(item_value)
-            if not (isinstance(node_key, yaml.ScalarNode) and \
-                    not node_key.style):
-                best_style = False
-            if not (isinstance(node_value, yaml.ScalarNode) and \
-                    not node_value.style):
-                best_style = False
-            value.append((node_key, node_value))
-        if flow_style is None:
-            if self.default_flow_style is not None:
-                node.flow_style = self.default_flow_style
-            else:
-                node.flow_style = best_style
-        return node
-Dumper.add_representer(confit.OrderedDict, Dumper.represent_dict)
-
-def migrate_config(replace=False):
-    """Migrate a legacy beetsconfig file to a new-style config.yaml file
-    in an appropriate place. If `replace` is enabled, then any existing
-    config.yaml will be moved aside. Otherwise, the process is aborted
-    when the file exists.
-    """
-
-    # Load legacy configuration data, if any.
-    config, configpath = get_config()
-    if not config:
-        log.debug(u'no config file found at {0}'.format(
-            util.displayable_path(configpath)
-            ))
-        return
-
-    # Get the new configuration file path and possibly move it out of
-    # the way.
-    destfn = os.path.join(beets.config.config_dir(), confit.CONFIG_FILENAME)
-    if os.path.exists(destfn):
-        if replace:
-            log.debug(u'moving old config aside: {0}'.format(
-                util.displayable_path(destfn)
-            ))
-            _displace(destfn)
-        else:
-            # File exists and we won't replace it. We're done.
-            return
-
-    log.debug(u'migrating config file {0}'.format(
-        util.displayable_path(configpath)
-    ))
-
-    # Convert the configuration to a data structure ready to be dumped
-    # as the new Confit file.
-    data = transform_data(flatten_config(config))
-
-    # Encode result as YAML.
-    yaml_out = yaml.dump(
-        data,
-        Dumper=Dumper,
-        default_flow_style=False,
-        indent=4,
-        width=1000,
-    )
-    # A ridiculous little hack to add some whitespace between "sections"
-    # in the YAML output. I hope this doesn't break any YAML syntax.
-    yaml_out = re.sub(r'(\n\w+:\n    [^-\s])', '\n\\1', yaml_out)
-    yaml_out = YAML_COMMENT + yaml_out
-
-    # Write the data to the new config destination.
-    log.debug(u'writing migrated config to {0}'.format(
-        util.displayable_path(destfn)
-    ))
-    with open(destfn, 'w') as f:
-        f.write(yaml_out)
-    return destfn
-
-def migrate_db(replace=False):
-    """Copy the beets library database file to the new location (e.g.,
-    from ~/.beetsmusic.blb to ~/.config/beets/library.db).
-    """
-    _, srcfn = default_paths()
-    destfn = beets.config['library'].as_filename()
-
-    if not os.path.exists(srcfn) or srcfn == destfn:
-        # Old DB does not exist or we're configured to point to the same
-        # database. Do nothing.
-        return
-
-    if os.path.exists(destfn):
-        if replace:
-            log.debug(u'moving old database aside: {0}'.format(
-                util.displayable_path(destfn)
-            ))
-            _displace(destfn)
-        else:
-            return
-
-    log.debug(u'copying database from {0} to {1}'.format(
-        util.displayable_path(srcfn), util.displayable_path(destfn)
-    ))
-    util.copy(srcfn, destfn)
-    return destfn
-
-def migrate_state(replace=False):
-    """Copy the beets runtime state file from the old path (i.e.,
-    ~/.beetsstate) to the new path (i.e., ~/.config/beets/state.pickle).
-    """
-    srcfn = os.path.expanduser(os.path.join('~', '.beetsstate'))
-    if not os.path.exists(srcfn):
-        return
-
-    destfn = beets.config['statefile'].as_filename()
-    if os.path.exists(destfn):
-        if replace:
-            _displace(destfn)
-        else:
-            return
-
-    log.debug(u'copying state file from {0} to {1}'.format(
-        util.displayable_path(srcfn), util.displayable_path(destfn)
-    ))
-    util.copy(srcfn, destfn)
-    return destfn
-
-
-# Automatic migration when beets starts.
-
-def automigrate():
-    """Migrate the configuration, database, and state files. If any
-    migration occurs, print out a notice with some helpful next steps.
-    """
-    config_fn = migrate_config()
-    db_fn = migrate_db()
-    migrate_state()
-
-    if config_fn:
-        ui.print_(ui.colorize('fuchsia', u'MIGRATED CONFIGURATION'))
-
-        ui.print_(CONFIG_MIGRATED_MESSAGE.format(
-            newconfig=util.displayable_path(config_fn))
-        )
-        if db_fn:
-            ui.print_(DB_MIGRATED_MESSAGE.format(
-                newdb=util.displayable_path(db_fn)
-            ))
-
-        ui.input_(ui.colorize('fuchsia', u'Press ENTER to continue:'))
-        ui.print_()
-
-
-# CLI command for explicit migration.
-
-migrate_cmd = ui.Subcommand('migrate', help='convert legacy config')
-def migrate_func(lib, opts, args):
-    """Explicit command for migrating files. Existing files in each
-    destination are moved aside.
-    """
-    config_fn = migrate_config(replace=True)
-    if config_fn:
-        log.info(u'Migrated configuration to: {0}'.format(
-            util.displayable_path(config_fn)
-        ))
-    db_fn = migrate_db(replace=True)
-    if db_fn:
-        log.info(u'Migrated library database to: {0}'.format(
-            util.displayable_path(db_fn)
-        ))
-    state_fn = migrate_state(replace=True)
-    if state_fn:
-        log.info(u'Migrated state file to: {0}'.format(
-            util.displayable_path(state_fn)
-        ))
-migrate_cmd.func = migrate_func

lib/beets/util/__init__.py

@@ -23,10 +23,13 @@ import fnmatch
 from collections import defaultdict
 import traceback
 import subprocess
+import platform
+
 
 MAX_FILENAME_LENGTH = 200
 WINDOWS_MAGIC_PREFIX = u'\\\\?\\'
 
+
 class HumanReadableException(Exception):
     """An Exception that can include a human-readable error message to
     be logged without a traceback. Can preserve a traceback for
@@ -82,6 +85,7 @@ class HumanReadableException(Exception):
             logger.debug(self.tb)
         logger.error(u'{0}: {1}'.format(self.error_kind, self.args[0]))
 
+
 class FilesystemError(HumanReadableException):
     """An error that occurred while performing a filesystem manipulation
     via a function in this module. The `paths` field is a sequence of
@@ -111,6 +115,7 @@ class FilesystemError(HumanReadableException):
 
         return u'{0} {1}'.format(self._reasonstr(), clause)
 
+
 def normpath(path):
     """Provide the canonical form of the path suitable for storing in
     the database.
@@ -119,6 +124,7 @@ def normpath(path):
     path = os.path.normpath(os.path.abspath(os.path.expanduser(path)))
     return bytestring_path(path)
 
+
 def ancestry(path):
     """Return a list consisting of path's parent directory, its
     grandparent, and so on. For instance:
@@ -137,10 +143,12 @@ def ancestry(path):
             break
         last_path = path
 
-        if path: # don't yield ''
+        if path:
+            # don't yield ''
             out.insert(0, path)
     return out
 
+
 def sorted_walk(path, ignore=(), logger=None):
     """Like `os.walk`, but yields things in case-insensitive sorted,
     breadth-first order.  Directory and file names matching any glob
@@ -192,6 +200,7 @@ def sorted_walk(path, ignore=(), logger=None):
         for res in sorted_walk(cur, ignore, logger):
             yield res
 
+
 def mkdirall(path):
     """Make all the enclosing directories of path (like mkdir -p on the
     parent).
@@ -204,6 +213,7 @@ def mkdirall(path):
                 raise FilesystemError(exc, 'create', (ancestor,),
                                       traceback.format_exc())
 
+
 def fnmatch_all(names, patterns):
     """Determine whether all strings in `names` match at least one of
     the `patterns`, which should be shell glob expressions.
@@ -218,6 +228,7 @@ def fnmatch_all(names, patterns):
             return False
     return True
 
+
 def prune_dirs(path, root=None, clutter=('.DS_Store', 'Thumbs.db')):
     """If path is an empty directory, then remove it. Recursively remove
     path's ancestry up to root (which is never removed) where there are
@@ -236,7 +247,7 @@ def prune_dirs(path, root=None, clutter=('.DS_Store', 'Thumbs.db')):
         ancestors = []
     elif root in ancestors:
         # Only remove directories below the root.
-        ancestors = ancestors[ancestors.index(root)+1:]
+        ancestors = ancestors[ancestors.index(root) + 1:]
     else:
         # Remove nothing.
         return
@@ -258,6 +269,7 @@ def prune_dirs(path, root=None, clutter=('.DS_Store', 'Thumbs.db')):
         else:
             break
 
+
 def components(path):
     """Return a list of the path components in path. For instance:
 
@@ -281,6 +293,7 @@ def components(path):
 
     return comps
 
+
 def _fsencoding():
     """Get the system's filesystem encoding. On Windows, this is always
     UTF-8 (not MBCS).
@@ -295,6 +308,7 @@ def _fsencoding():
         encoding = 'utf8'
     return encoding
 
+
 def bytestring_path(path):
     """Given a path, which is either a str or a unicode, returns a str
     path (ensuring that we never deal with Unicode pathnames).
@@ -315,6 +329,7 @@ def bytestring_path(path):
     except (UnicodeError, LookupError):
         return path.encode('utf8')
 
+
 def displayable_path(path, separator=u'; '):
     """Attempts to decode a bytestring path to a unicode object for the
     purpose of displaying it to the user. If the `path` argument is a
@@ -333,6 +348,7 @@ def displayable_path(path, separator=u'; '):
     except (UnicodeError, LookupError):
         return path.decode('utf8', 'ignore')
 
+
 def syspath(path, prefix=True):
     """Convert a path for use by the operating system. In particular,
     paths on Windows must receive a magic prefix and must be converted
@@ -356,16 +372,22 @@ def syspath(path, prefix=True):
             encoding = sys.getfilesystemencoding() or sys.getdefaultencoding()
             path = path.decode(encoding, 'replace')
 
-    # Add the magic prefix if it isn't already there
+    # Add the magic prefix if it isn't already there.
+    # http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247.aspx
     if prefix and not path.startswith(WINDOWS_MAGIC_PREFIX):
+        if path.startswith(u'\\\\'):
+            # UNC path. Final path should look like \\?\UNC\...
+            path = u'UNC' + path[1:]
         path = WINDOWS_MAGIC_PREFIX + path
 
     return path
 
+
 def samefile(p1, p2):
     """Safer equality for paths."""
     return shutil._samefile(syspath(p1), syspath(p2))
 
+
 def remove(path, soft=True):
     """Remove the file. If `soft`, then no error will be raised if the
     file does not exist.
@@ -378,6 +400,7 @@ def remove(path, soft=True):
     except (OSError, IOError) as exc:
         raise FilesystemError(exc, 'delete', (path,), traceback.format_exc())
 
+
 def copy(path, dest, replace=False):
     """Copy a plain file. Permissions are not copied. If `dest` already
     exists, raises a FilesystemError unless `replace` is True. Has no
@@ -396,6 +419,7 @@ def copy(path, dest, replace=False):
         raise FilesystemError(exc, 'copy', (path, dest),
                               traceback.format_exc())
 
+
 def move(path, dest, replace=False):
     """Rename a file. `dest` may not be a directory. If `dest` already
     exists, raises an OSError unless `replace` is True. Has no effect if
@@ -424,6 +448,27 @@ def move(path, dest, replace=False):
             raise FilesystemError(exc, 'move', (path, dest),
                                   traceback.format_exc())
 
+
+def link(path, dest, replace=False):
+    """Create a symbolic link from path to `dest`. Raises an OSError if
+    `dest` already exists, unless `replace` is True. Does nothing if
+    `path` == `dest`."""
+    if (samefile(path, dest)):
+        return
+
+    path = syspath(path)
+    dest = syspath(dest)
+    if os.path.exists(dest) and not replace:
+        raise FilesystemError('file exists', 'rename', (path, dest),
+                              traceback.format_exc())
+    try:
+        os.symlink(path, dest)
+    except OSError:
+        raise FilesystemError('Operating system does not support symbolic '
+                              'links.', 'link', (path, dest),
+                              traceback.format_exc())
+
+
 def unique_path(path):
     """Returns a version of ``path`` that does not exist on the
     filesystem. Specifically, if ``path` itself already exists, then
@@ -457,6 +502,8 @@ CHAR_REPLACE = [
     (re.compile(ur'\.$'), u'_'),  # Trailing dots.
     (re.compile(ur'\s+$'), u''),  # Trailing whitespace.
 ]
+
+
 def sanitize_path(path, replacements=None):
     """Takes a path (as a Unicode string) and makes sure that it is
     legal. Returns a new path. Only works with fragments; won't work
@@ -477,6 +524,7 @@ def sanitize_path(path, replacements=None):
         comps[i] = comp
     return os.path.join(*comps)
 
+
 def truncate_path(path, length=MAX_FILENAME_LENGTH):
     """Given a bytestring path or a Unicode path fragment, truncate the
     components to a legal length. In the last component, the extension
@@ -493,6 +541,7 @@ def truncate_path(path, length=MAX_FILENAME_LENGTH):
 
     return os.path.join(*out)
 
+
 def str2bool(value):
     """Returns a boolean reflecting a human-entered string."""
     if value.lower() in ('yes', '1', 'true', 't', 'y'):
@@ -500,6 +549,7 @@ def str2bool(value):
     else:
         return False
 
+
 def as_string(value):
     """Convert a value to a Unicode object for matching with a query.
     None becomes the empty string. Bytestrings are silently decoded.
@@ -513,6 +563,7 @@ def as_string(value):
     else:
         return unicode(value)
 
+
 def levenshtein(s1, s2):
     """A nice DP edit distance implementation from Wikibooks:
     http://en.wikibooks.org/wiki/Algorithm_implementation/Strings/
@@ -535,6 +586,7 @@ def levenshtein(s1, s2):
 
     return previous_row[-1]
 
+
 def plurality(objs):
     """Given a sequence of comparable objects, returns the object that
     is most common in the set and the frequency of that object. The
@@ -558,6 +610,7 @@ def plurality(objs):
 
     return res, max_freq
 
+
 def cpu_count():
     """Return the number of hardware thread contexts (cores or SMT
     threads) in the system.
@@ -571,7 +624,7 @@ def cpu_count():
             num = 0
     elif sys.platform == 'darwin':
         try:
-            num = int(os.popen('sysctl -n hw.ncpu').read())
+            num = int(command_output(['sysctl', '-n', 'hw.ncpu']))
         except ValueError:
             num = 0
     else:
@@ -584,23 +637,38 @@ def cpu_count():
     else:
         return 1
 
-def command_output(cmd):
-    """Wraps the `subprocess` module to invoke a command (given as a
-    list of arguments starting with the command name) and collect
-    stdout. The stderr stream is ignored. May raise
-    `subprocess.CalledProcessError` or an `OSError`.
+
+def command_output(cmd, shell=False):
+    """Runs the command and returns its output after it has exited.
+
+    ``cmd`` is a list of arguments starting with the command names.  If
+    ``shell`` is true, ``cmd`` is assumed to be a string and passed to a
+    shell to execute.
+
+    If the process exits with a non-zero return code
+    ``subprocess.CalledProcessError`` is raised. May also raise
+    ``OSError``.
 
     This replaces `subprocess.check_output`, which isn't available in
     Python 2.6 and which can have problems if lots of output is sent to
     stderr.
     """
-    with open(os.devnull, 'w') as devnull:
-        proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=devnull)
-        stdout, _ = proc.communicate()
+    proc = subprocess.Popen(
+        cmd,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+        close_fds=platform.system() != 'Windows',
+        shell=shell
+    )
+    stdout, stderr = proc.communicate()
     if proc.returncode:
-        raise subprocess.CalledProcessError(proc.returncode, cmd)
+        raise subprocess.CalledProcessError(
+            returncode=proc.returncode,
+            cmd=' '.join(cmd),
+        )
     return stdout
 
+
 def max_filename_length(path, limit=MAX_FILENAME_LENGTH):
     """Attempt to determine the maximum filename length for the
     filesystem containing `path`. If the value is greater than `limit`,

lib/beets/util/artresizer.py

@@ -1,5 +1,5 @@
 # This file is part of beets.
-# Copyright 2013, Fabrice Laporte
+# Copyright 2014, Fabrice Laporte
 #
 # Permission is hereby granted, free of charge, to any person obtaining
 # a copy of this software and associated documentation files (the
@@ -18,6 +18,7 @@ public resizing proxy if neither is available.
 import urllib
 import subprocess
 import os
+import re
 from tempfile import NamedTemporaryFile
 import logging
 from beets import util
@@ -37,7 +38,7 @@ def resize_url(url, maxwidth):
     maxwidth (preserving aspect ratio).
     """
     return '{0}?{1}'.format(PROXY_URL, urllib.urlencode({
-        'url': url.replace('http://',''),
+        'url': url.replace('http://', ''),
         'w': str(maxwidth),
     }))
 
@@ -76,7 +77,7 @@ def pil_resize(maxwidth, path_in, path_out=None):
 
 def im_resize(maxwidth, path_in, path_out=None):
     """Resize using ImageMagick's ``convert`` tool.
-    tool. Return the output path of resized image.
+    Return the output path of resized image.
     """
     path_out = path_out or temp_file_for(path_in)
     log.debug(u'artresizer: ImageMagick resizing {0} to {1}'.format(
@@ -132,8 +133,9 @@ class ArtResizer(object):
         """Create a resizer object for the given method or, if none is
         specified, with an inferred method.
         """
-        self.method = method or self._guess_method()
+        self.method = self._check_method(method)
         log.debug(u"artresizer: method is {0}".format(self.method))
+        self.can_compare = self._can_compare()
 
     def resize(self, maxwidth, path_in, path_out=None):
         """Manipulate an image file according to the method, returning a
@@ -141,7 +143,7 @@ class ArtResizer(object):
         temporary file. For WEBPROXY, returns `path_in` unmodified.
         """
         if self.local:
-            func = BACKEND_FUNCS[self.method]
+            func = BACKEND_FUNCS[self.method[0]]
             return func(maxwidth, path_in, path_out)
         else:
             return path_in
@@ -159,30 +161,51 @@ class ArtResizer(object):
     @property
     def local(self):
         """A boolean indicating whether the resizing method is performed
-        locally (i.e., PIL or IMAGEMAGICK).
+        locally (i.e., PIL or ImageMagick).
         """
-        return self.method in BACKEND_FUNCS
+        return self.method[0] in BACKEND_FUNCS
+
+    def _can_compare(self):
+        """A boolean indicating whether image comparison is available"""
+
+        return self.method[0] == IMAGEMAGICK and self.method[1] > (6, 8, 7)
 
     @staticmethod
-    def _guess_method():
-        """Determine which resizing method to use. Returns PIL,
-        IMAGEMAGICK, or WEBPROXY depending on available dependencies.
+    def _check_method(method=None):
+        """A tuple indicating whether current method is available and its
+        version. If no method is given, it returns a supported one.
         """
-        # Try importing PIL.
-        try:
-            __import__('PIL', fromlist=['Image'])
-            return PIL
-        except ImportError:
-            pass
+        # Guess available method
+        if not method:
+            for m in [IMAGEMAGICK, PIL]:
+                _, version = ArtResizer._check_method(m)
+                if version:
+                    return (m, version)
+            return (WEBPROXY, (0))
 
-        # Try invoking ImageMagick's "convert".
-        try:
-            out = util.command_output(['convert', '--version'])
-            if 'imagemagick' in out.lower():
-                # system32/convert.exe may be interfering
-                return IMAGEMAGICK
-        except (subprocess.CalledProcessError, OSError):
-            pass
+        if method == IMAGEMAGICK:
 
-        # Fall back to Web proxy method.
-        return WEBPROXY
+            # Try invoking ImageMagick's "convert".
+            try:
+                out = util.command_output(['identify', '--version'])
+
+                if 'imagemagick' in out.lower():
+                    pattern = r".+ (\d+)\.(\d+)\.(\d+).*"
+                    match = re.search(pattern, out)
+                    if match:
+                        return (IMAGEMAGICK,
+                                (int(match.group(1)),
+                                 int(match.group(2)),
+                                 int(match.group(3))))
+                    return (IMAGEMAGICK, (0))
+
+            except (subprocess.CalledProcessError, OSError):
+                return (IMAGEMAGICK, None)
+
+        if method == PIL:
+            # Try importing PIL.
+            try:
+                __import__('PIL', fromlist=['Image'])
+                return (PIL, (0))
+            except ImportError:
+                return (PIL, None)

lib/beets/util/bluelet.py

@@ -38,6 +38,7 @@ class Event(object):
     """
     pass
 
+
 class WaitableEvent(Event):
     """A waitable event is one encapsulating an action that can be
     waited for using a select() call. That is, it's an event with an
@@ -57,21 +58,25 @@ class WaitableEvent(Event):
         """
         pass
 
+
 class ValueEvent(Event):
     """An event that does nothing but return a fixed value."""
     def __init__(self, value):
         self.value = value
 
+
 class ExceptionEvent(Event):
     """Raise an exception at the yield point. Used internally."""
     def __init__(self, exc_info):
         self.exc_info = exc_info
 
+
 class SpawnEvent(Event):
     """Add a new coroutine thread to the scheduler."""
     def __init__(self, coro):
         self.spawned = coro
 
+
 class JoinEvent(Event):
     """Suspend the thread until the specified child thread has
     completed.
@@ -79,11 +84,13 @@ class JoinEvent(Event):
     def __init__(self, child):
         self.child = child
 
+
 class KillEvent(Event):
     """Unschedule a child thread."""
     def __init__(self, child):
         self.child = child
 
+
 class DelegationEvent(Event):
     """Suspend execution of the current thread, start a new thread and,
     once the child thread finished, return control to the parent
@@ -92,6 +99,7 @@ class DelegationEvent(Event):
     def __init__(self, coro):
         self.spawned = coro
 
+
 class ReturnEvent(Event):
     """Return a value the current thread's delegator at the point of
     delegation. Ends the current (delegate) thread.
@@ -99,6 +107,7 @@ class ReturnEvent(Event):
     def __init__(self, value):
         self.value = value
 
+
 class SleepEvent(WaitableEvent):
     """Suspend the thread for a given duration.
     """
@@ -108,6 +117,7 @@ class SleepEvent(WaitableEvent):
     def time_left(self):
         return max(self.wakeup_time - time.time(), 0.0)
 
+
 class ReadEvent(WaitableEvent):
     """Reads from a file-like object."""
     def __init__(self, fd, bufsize):
@@ -120,6 +130,7 @@ class ReadEvent(WaitableEvent):
     def fire(self):
         return self.fd.read(self.bufsize)
 
+
 class WriteEvent(WaitableEvent):
     """Writes to a file-like object."""
     def __init__(self, fd, data):
@@ -192,15 +203,19 @@ def _event_select(events):
 
     return ready_events
 
+
 class ThreadException(Exception):
     def __init__(self, coro, exc_info):
         self.coro = coro
         self.exc_info = exc_info
+
     def reraise(self):
         _reraise(self.exc_info[0], self.exc_info[1], self.exc_info[2])
 
+
 SUSPENDED = Event()  # Special sentinel placeholder for suspended threads.
 
+
 class Delegated(Event):
     """Placeholder indicating that a thread has delegated execution to a
     different thread.
@@ -208,6 +223,7 @@ class Delegated(Event):
     def __init__(self, child):
         self.child = child
 
+
 def run(root_coro):
     """Schedules a coroutine, running it to completion. This
     encapsulates the Bluelet scheduler, which the root coroutine can
@@ -329,7 +345,7 @@ def run(root_coro):
                     break
 
             # Wait and fire.
-            event2coro = dict((v,k) for k,v in threads.items())
+            event2coro = dict((v, k) for k, v in threads.items())
             for event in _event_select(threads.values()):
                 # Run the IO operation, but catch socket errors.
                 try:
@@ -378,6 +394,7 @@ def run(root_coro):
 class SocketClosedError(Exception):
     pass
 
+
 class Listener(object):
     """A socket wrapper object for listening sockets.
     """
@@ -407,6 +424,7 @@ class Listener(object):
         self._closed = True
         self.sock.close()
 
+
 class Connection(object):
     """A socket wrapper object for connected sockets.
     """
@@ -468,6 +486,7 @@ class Connection(object):
                 yield ReturnEvent(line)
                 break
 
+
 class AcceptEvent(WaitableEvent):
     """An event for Listener objects (listening sockets) that suspends
     execution until the socket gets a connection.
@@ -482,6 +501,7 @@ class AcceptEvent(WaitableEvent):
         sock, addr = self.listener.sock.accept()
         return Connection(sock, addr)
 
+
 class ReceiveEvent(WaitableEvent):
     """An event for Connection objects (connected sockets) for
     asynchronously reading data.
@@ -496,6 +516,7 @@ class ReceiveEvent(WaitableEvent):
     def fire(self):
         return self.conn.sock.recv(self.bufsize)
 
+
 class SendEvent(WaitableEvent):
     """An event for Connection objects (connected sockets) for
     asynchronously writing data.
@@ -523,6 +544,7 @@ def null():
     """
     return ValueEvent(None)
 
+
 def spawn(coro):
     """Event: add another coroutine to the scheduler. Both the parent
     and child coroutines run concurrently.
@@ -531,6 +553,7 @@ def spawn(coro):
         raise ValueError('%s is not a coroutine' % str(coro))
     return SpawnEvent(coro)
 
+
 def call(coro):
     """Event: delegate to another coroutine. The current coroutine
     is resumed once the sub-coroutine finishes. If the sub-coroutine
@@ -540,12 +563,14 @@ def call(coro):
         raise ValueError('%s is not a coroutine' % str(coro))
     return DelegationEvent(coro)
 
+
 def end(value=None):
     """Event: ends the coroutine and returns a value to its
     delegator.
     """
     return ReturnEvent(value)
 
+
 def read(fd, bufsize=None):
     """Event: read from a file descriptor asynchronously."""
     if bufsize is None:
@@ -563,10 +588,12 @@ def read(fd, bufsize=None):
     else:
         return ReadEvent(fd, bufsize)
 
+
 def write(fd, data):
     """Event: write to a file descriptor asynchronously."""
     return WriteEvent(fd, data)
 
+
 def connect(host, port):
     """Event: connect to a network address and return a Connection
     object for communicating on the socket.
@@ -575,17 +602,20 @@ def connect(host, port):
     sock = socket.create_connection(addr)
     return ValueEvent(Connection(sock, addr))
 
+
 def sleep(duration):
     """Event: suspend the thread for ``duration`` seconds.
     """
     return SleepEvent(duration)
 
+
 def join(coro):
     """Suspend the thread until another, previously `spawn`ed thread
     completes.
     """
     return JoinEvent(coro)
 
+
 def kill(coro):
     """Halt the execution of a different `spawn`ed thread.
     """

lib/beets/util/confit.py

@@ -21,6 +21,8 @@ import pkgutil
 import sys
 import yaml
 import types
+import collections
+import re
 try:
     from collections import OrderedDict
 except ImportError:
@@ -47,6 +49,7 @@ BASESTRING = str if PY3 else basestring
 NUMERIC_TYPES = (int, float) if PY3 else (int, float, long)
 TYPE_TYPES = (type,) if PY3 else (type, types.ClassType)
 
+
 def iter_first(sequence):
     """Get the first element from an iterable or raise a ValueError if
     the iterator generates no values.
@@ -67,16 +70,25 @@ class ConfigError(Exception):
     """Base class for exceptions raised when querying a configuration.
     """
 
+
 class NotFoundError(ConfigError):
     """A requested value could not be found in the configuration trees.
     """
 
-class ConfigTypeError(ConfigError, TypeError):
+
+class ConfigValueError(ConfigError):
+    """The value in the configuration is illegal."""
+
+
+class ConfigTypeError(ConfigValueError):
     """The value in the configuration did not match the expected type.
     """
 
-class ConfigValueError(ConfigError, ValueError):
-    """The value in the configuration is illegal."""
+
+class ConfigTemplateError(ConfigError):
+    """Base class for exceptions raised because of an invalid template.
+    """
+
 
 class ConfigReadError(ConfigError):
     """A configuration file could not be read."""
@@ -132,6 +144,7 @@ class ConfigSource(dict):
         else:
             raise TypeError('source value must be a dict')
 
+
 class ConfigView(object):
     """A configuration "view" is a query into a program's configuration
     data. A view represents a hypothetical location in the configuration
@@ -207,6 +220,9 @@ class ConfigView(object):
         """
         self.set({key: value})
 
+    def __contains__(self, key):
+        return self[key].exists()
+
     def set_args(self, namespace):
         """Overlay parsed command-line arguments, generated by a library
         like argparse or optparse, onto this view's value.
@@ -310,98 +326,6 @@ class ConfigView(object):
 
     # Validation and conversion.
 
-    def get(self, typ=None):
-        """Returns the canonical value for the view, checked against the
-        passed-in type. If the value is not an instance of the given
-        type, a ConfigTypeError is raised. May also raise a
-        NotFoundError.
-        """
-        value, _ = self.first()
-
-        if typ is not None:
-            if not isinstance(typ, TYPE_TYPES):
-                raise TypeError('argument to get() must be a type')
-
-            if not isinstance(value, typ):
-                raise ConfigTypeError(
-                    "{0} must be of type {1}, not {2}".format(
-                        self.name, typ.__name__, type(value).__name__
-                    )
-                )
-
-        return value
-
-    def as_filename(self):
-        """Get a string as a normalized as an absolute, tilde-free path.
-
-        Relative paths are relative to the configuration directory (see
-        the `config_dir` method) if they come from a file. Otherwise,
-        they are relative to the current working directory. This helps
-        attain the expected behavior when using command-line options.
-        """
-        path, source = self.first()
-        if not isinstance(path, BASESTRING):
-            raise ConfigTypeError('{0} must be a filename, not {1}'.format(
-                self.name, type(path).__name__
-            ))
-        path = os.path.expanduser(STRING(path))
-
-        if not os.path.isabs(path) and source.filename:
-            # From defaults: relative to the app's directory.
-            path = os.path.join(self.root().config_dir(), path)
-
-        return os.path.abspath(path)
-
-    def as_choice(self, choices):
-        """Ensure that the value is among a collection of choices and
-        return it. If `choices` is a dictionary, then return the
-        corresponding value rather than the value itself (the key).
-        """
-        value = self.get()
-
-        if value not in choices:
-            raise ConfigValueError(
-                '{0} must be one of {1}, not {2}'.format(
-                    self.name, repr(list(choices)), repr(value)
-                )
-            )
-
-        if isinstance(choices, dict):
-            return choices[value]
-        else:
-            return value
-
-    def as_number(self):
-        """Ensure that a value is of numeric type."""
-        value = self.get()
-        if isinstance(value, NUMERIC_TYPES):
-            return value
-        raise ConfigTypeError(
-            '{0} must be numeric, not {1}'.format(
-                self.name, type(value).__name__
-            )
-        )
-
-    def as_str_seq(self):
-        """Get the value as a list of strings. The underlying configured
-        value can be a sequence or a single string. In the latter case,
-        the string is treated as a white-space separated list of words.
-        """
-        value = self.get()
-        if isinstance(value, bytes):
-            value = value.decode('utf8', 'ignore')
-
-        if isinstance(value, STRING):
-            return value.split()
-        else:
-            try:
-                return list(value)
-            except TypeError:
-                raise ConfigTypeError(
-                    '{0} must be a whitespace-separated string or '
-                    'a list'.format(self.name)
-                )
-
     def flatten(self):
         """Create a hierarchy of OrderedDicts containing the data from
         this view, recursively reifying all views to get their
@@ -415,6 +339,35 @@ class ConfigView(object):
                 od[key] = view.get()
         return od
 
+    def get(self, template=None):
+        """Retrieve the value for this view according to the template.
+
+        The `template` against which the values are checked can be
+        anything convertible to a `Template` using `as_template`. This
+        means you can pass in a default integer or string value, for
+        example, or a type to just check that something matches the type
+        you expect.
+
+        May raise a `ConfigValueError` (or its subclass,
+        `ConfigTypeError`) or a `NotFoundError` when the configuration
+        doesn't satisfy the template.
+        """
+        return as_template(template).value(self, template)
+
+    # Old validation methods (deprecated).
+
+    def as_filename(self):
+        return self.get(Filename())
+
+    def as_choice(self, choices):
+        return self.get(Choice(choices))
+
+    def as_number(self):
+        return self.get(Number())
+
+    def as_str_seq(self):
+        return self.get(StrSeq())
+
 
 class RootView(ConfigView):
     """The base of a view hierarchy. This view keeps track of the
@@ -518,6 +471,7 @@ def _package_path(name):
 
     return os.path.dirname(os.path.abspath(filepath))
 
+
 def config_dirs():
     """Return a platform-specific list of candidates for user
     configuration directories on the system.
@@ -606,10 +560,12 @@ class Loader(yaml.SafeLoader):
         plain = super(Loader, self).check_plain()
         return plain or self.peek() == '%'
 
+
 Loader.add_constructor('tag:yaml.org,2002:str', Loader._construct_unicode)
 Loader.add_constructor('tag:yaml.org,2002:map', Loader.construct_yaml_map)
 Loader.add_constructor('tag:yaml.org,2002:omap', Loader.construct_yaml_map)
 
+
 def load_yaml(filename):
     """Read a YAML document from a file. If the file cannot be read or
     parsed, a ConfigReadError is raised.
@@ -679,11 +635,13 @@ class Dumper(yaml.SafeDumper):
         """
         return self.represent_scalar('tag:yaml.org,2002:null', '')
 
+
 Dumper.add_representer(OrderedDict, Dumper.represent_dict)
 Dumper.add_representer(bool, Dumper.represent_bool)
 Dumper.add_representer(type(None), Dumper.represent_none)
 Dumper.add_representer(list, Dumper.represent_list)
 
+
 def restore_yaml_comments(data, default_data):
     """Scan default_data for comments (we include empty lines in our
     definition of comments) and place them before the same keys in data.
@@ -898,3 +856,426 @@ class LazyConfig(Configuration):
         del self.sources[:]
         self._lazy_suffix = []
         self._lazy_prefix = []
+
+
+# "Validated" configuration views: experimental!
+
+
+REQUIRED = object()
+"""A sentinel indicating that there is no default value and an exception
+should be raised when the value is missing.
+"""
+
+
+class Template(object):
+    """A value template for configuration fields.
+
+    The template works like a type and instructs Confit about how to
+    interpret a deserialized YAML value. This includes type conversions,
+    providing a default value, and validating for errors. For example, a
+    filepath type might expand tildes and check that the file exists.
+    """
+    def __init__(self, default=REQUIRED):
+        """Create a template with a given default value.
+
+        If `default` is the sentinel `REQUIRED` (as it is by default),
+        then an error will be raised when a value is missing. Otherwise,
+        missing values will instead return `default`.
+        """
+        self.default = default
+
+    def __call__(self, view):
+        """Invoking a template on a view gets the view's value according
+        to the template.
+        """
+        return self.value(view, self)
+
+    def value(self, view, template=None):
+        """Get the value for a `ConfigView`.
+
+        May raise a `NotFoundError` if the value is missing (and the
+        template requires it) or a `ConfigValueError` for invalid values.
+        """
+        if view.exists():
+            value, _ = view.first()
+            return self.convert(value, view)
+        elif self.default is REQUIRED:
+            # Missing required value. This is an error.
+            raise NotFoundError("{0} not found".format(view.name))
+        else:
+            # Missing value, but not required.
+            return self.default
+
+    def convert(self, value, view):
+        """Convert the YAML-deserialized value to a value of the desired
+        type.
+
+        Subclasses should override this to provide useful conversions.
+        May raise a `ConfigValueError` when the configuration is wrong.
+        """
+        # Default implementation does no conversion.
+        return value
+
+    def fail(self, message, view, type_error=False):
+        """Raise an exception indicating that a value cannot be
+        accepted.
+
+        `type_error` indicates whether the error is due to a type
+        mismatch rather than a malformed value. In this case, a more
+        specific exception is raised.
+        """
+        exc_class = ConfigTypeError if type_error else ConfigValueError
+        raise exc_class(
+            '{0}: {1}'.format(view.name, message)
+        )
+
+    def __repr__(self):
+        return '{0}({1})'.format(
+            type(self).__name__,
+            '' if self.default is REQUIRED else repr(self.default),
+        )
+
+
+class Integer(Template):
+    """An integer configuration value template.
+    """
+    def convert(self, value, view):
+        """Check that the value is an integer. Floats are rounded.
+        """
+        if isinstance(value, int):
+            return value
+        elif isinstance(value, float):
+            return int(value)
+        else:
+            self.fail('must be a number', view, True)
+
+
+class Number(Template):
+    """A numeric type: either an integer or a floating-point number.
+    """
+    def convert(self, value, view):
+        """Check that the value is an int or a float.
+        """
+        if isinstance(value, NUMERIC_TYPES):
+            return value
+        else:
+            self.fail(
+                'must be numeric, not {0}'.format(type(value).__name__),
+                view,
+                True
+            )
+
+
+class MappingTemplate(Template):
+    """A template that uses a dictionary to specify other types for the
+    values for a set of keys and produce a validated `AttrDict`.
+    """
+    def __init__(self, mapping):
+        """Create a template according to a dict (mapping). The
+        mapping's values should themselves either be Types or
+        convertible to Types.
+        """
+        subtemplates = {}
+        for key, typ in mapping.items():
+            subtemplates[key] = as_template(typ)
+        self.subtemplates = subtemplates
+
+    def value(self, view, template=None):
+        """Get a dict with the same keys as the template and values
+        validated according to the value types.
+        """
+        out = AttrDict()
+        for key, typ in self.subtemplates.items():
+            out[key] = typ.value(view[key], self)
+        return out
+
+    def __repr__(self):
+        return 'MappingTemplate({0})'.format(repr(self.subtemplates))
+
+
+class String(Template):
+    """A string configuration value template.
+    """
+    def __init__(self, default=REQUIRED, pattern=None):
+        """Create a template with the added optional `pattern` argument,
+        a regular expression string that the value should match.
+        """
+        super(String, self).__init__(default)
+        self.pattern = pattern
+        if pattern:
+            self.regex = re.compile(pattern)
+
+    def convert(self, value, view):
+        """Check that the value is a string and matches the pattern.
+        """
+        if isinstance(value, BASESTRING):
+            if self.pattern and not self.regex.match(value):
+                self.fail(
+                    "must match the pattern {0}".format(self.pattern),
+                    view
+                )
+            return value
+        else:
+            self.fail('must be a string', view, True)
+
+
+class Choice(Template):
+    """A template that permits values from a sequence of choices.
+    """
+    def __init__(self, choices):
+        """Create a template that validates any of the values from the
+        iterable `choices`.
+
+        If `choices` is a map, then the corresponding value is emitted.
+        Otherwise, the value itself is emitted.
+        """
+        self.choices = choices
+
+    def convert(self, value, view):
+        """Ensure that the value is among the choices (and remap if the
+        choices are a mapping).
+        """
+        if value not in self.choices:
+            self.fail(
+                'must be one of {0}, not {1}'.format(
+                    repr(list(self.choices)), repr(value)
+                ),
+                view
+            )
+
+        if isinstance(self.choices, collections.Mapping):
+            return self.choices[value]
+        else:
+            return value
+
+    def __repr__(self):
+        return 'Choice({0!r})'.format(self.choices)
+
+
+class StrSeq(Template):
+    """A template for values that are lists of strings.
+
+    Validates both actual YAML string lists and single strings. Strings
+    can optionally be split on whitespace.
+    """
+    def __init__(self, split=True):
+        """Create a new template.
+
+        `split` indicates whether, when the underlying value is a single
+        string, it should be split on whitespace. Otherwise, the
+        resulting value is a list containing a single string.
+        """
+        super(StrSeq, self).__init__()
+        self.split = split
+
+    def convert(self, value, view):
+        if isinstance(value, bytes):
+            value = value.decode('utf8', 'ignore')
+
+        if isinstance(value, STRING):
+            if self.split:
+                return value.split()
+            else:
+                return [value]
+
+        try:
+            value = list(value)
+        except TypeError:
+            self.fail('must be a whitespace-separated string or a list',
+                      view, True)
+
+        def convert(x):
+            if isinstance(x, unicode):
+                return x
+            elif isinstance(x, BASESTRING):
+                return x.decode('utf8', 'ignore')
+            else:
+                self.fail('must be a list of strings', view, True)
+        return map(convert, value)
+
+
+class Filename(Template):
+    """A template that validates strings as filenames.
+
+    Filenames are returned as absolute, tilde-free paths.
+
+    Relative paths are relative to the template's `cwd` argument
+    when it is specified, then the configuration directory (see
+    the `config_dir` method) if they come from a file. Otherwise,
+    they are relative to the current working directory. This helps
+    attain the expected behavior when using command-line options.
+    """
+    def __init__(self, default=REQUIRED, cwd=None, relative_to=None,
+                 in_app_dir=False):
+        """ `relative_to` is the name of a sibling value that is
+        being validated at the same time.
+
+        `in_app_dir` indicates whether the path should be resolved
+        inside the application's config directory (even when the setting
+        does not come from a file).
+        """
+        super(Filename, self).__init__(default)
+        self.cwd = cwd
+        self.relative_to = relative_to
+        self.in_app_dir = in_app_dir
+
+    def __repr__(self):
+        args = []
+
+        if self.default is not REQUIRED:
+            args.append(repr(self.default))
+
+        if self.cwd is not None:
+            args.append('cwd=' + repr(self.cwd))
+
+        if self.relative_to is not None:
+            args.append('relative_to=' + repr(self.relative_to))
+
+        if self.in_app_dir:
+            args.append('in_app_dir=True')
+
+        return 'Filename({0})'.format(', '.join(args))
+
+    def resolve_relative_to(self, view, template):
+        if not isinstance(template, (collections.Mapping, MappingTemplate)):
+            # disallow config.get(Filename(relative_to='foo'))
+            raise ConfigTemplateError(
+                'relative_to may only be used when getting multiple values.'
+            )
+
+        elif self.relative_to == view.key:
+            raise ConfigTemplateError(
+                '{0} is relative to itself'.format(view.name)
+            )
+
+        elif self.relative_to not in view.parent.keys():
+            # self.relative_to is not in the config
+            self.fail(
+                (
+                    'needs sibling value "{0}" to expand relative path'
+                ).format(self.relative_to),
+                view
+            )
+
+        old_template = {}
+        old_template.update(template.subtemplates)
+
+        # save time by skipping MappingTemplate's init loop
+        next_template = MappingTemplate({})
+        next_relative = self.relative_to
+
+        # gather all the needed templates and nothing else
+        while next_relative is not None:
+            try:
+                # pop to avoid infinite loop because of recursive
+                # relative paths
+                rel_to_template = old_template.pop(next_relative)
+            except KeyError:
+                if next_relative in template.subtemplates:
+                    # we encountered this config key previously
+                    raise ConfigTemplateError((
+                        '{0} and {1} are recursively relative'
+                    ).format(view.name, self.relative_to))
+                else:
+                    raise ConfigTemplateError((
+                        'missing template for {0}, needed to expand {1}\'s' +
+                        'relative path'
+                    ).format(self.relative_to, view.name))
+
+            next_template.subtemplates[next_relative] = rel_to_template
+            next_relative = rel_to_template.relative_to
+
+        return view.parent.get(next_template)[self.relative_to]
+
+    def value(self, view, template=None):
+        path, source = view.first()
+        if not isinstance(path, BASESTRING):
+            self.fail(
+                'must be a filename, not {0}'.format(type(path).__name__),
+                view,
+                True
+            )
+        path = os.path.expanduser(STRING(path))
+
+        if not os.path.isabs(path):
+            if self.cwd is not None:
+                # relative to the template's argument
+                path = os.path.join(self.cwd, path)
+
+            elif self.relative_to is not None:
+                path = os.path.join(
+                    self.resolve_relative_to(view, template),
+                    path,
+                )
+
+            elif source.filename or self.in_app_dir:
+                # From defaults: relative to the app's directory.
+                path = os.path.join(view.root().config_dir(), path)
+
+        return os.path.abspath(path)
+
+
+class TypeTemplate(Template):
+    """A simple template that checks that a value is an instance of a
+    desired Python type.
+    """
+    def __init__(self, typ, default=REQUIRED):
+        """Create a template that checks that the value is an instance
+        of `typ`.
+        """
+        super(TypeTemplate, self).__init__(default)
+        self.typ = typ
+
+    def convert(self, value, view):
+        if not isinstance(value, self.typ):
+            self.fail(
+                'must be a {0}, not {1}'.format(
+                    self.typ.__name__,
+                    type(value).__name__,
+                ),
+                view,
+                True
+            )
+        return value
+
+
+class AttrDict(dict):
+    """A `dict` subclass that can be accessed via attributes (dot
+    notation) for convenience.
+    """
+    def __getattr__(self, key):
+        if key in self:
+            return self[key]
+        else:
+            raise AttributeError(key)
+
+
+def as_template(value):
+    """Convert a simple "shorthand" Python value to a `Template`.
+    """
+    if isinstance(value, Template):
+        # If it's already a Template, pass it through.
+        return value
+    elif isinstance(value, collections.Mapping):
+        # Dictionaries work as templates.
+        return MappingTemplate(value)
+    elif value is int:
+        return Integer()
+    elif isinstance(value, int):
+        return Integer(value)
+    elif isinstance(value, type) and issubclass(value, BASESTRING):
+        return String()
+    elif isinstance(value, BASESTRING):
+        return String(value)
+    elif value is float:
+        return Number()
+    elif value is None:
+        return Template()
+    elif value is dict:
+        return TypeTemplate(collections.Mapping)
+    elif value is list:
+        return TypeTemplate(collections.Sequence)
+    elif isinstance(value, type):
+        return TypeTemplate(value)
+    else:
+        raise ValueError('cannot convert to template: {0!r}'.format(value))

lib/beets/util/enumeration.py

@@ -12,167 +12,29 @@
 # The above copyright notice and this permission notice shall be
 # included in all copies or substantial portions of the Software.
 
-"""A metaclass for enumerated types that really are types.
+from enum import Enum
 
-You can create enumerations with `enum(values, [name])` and they work
-how you would expect them to.
 
-    >>> from enumeration import enum
-    >>> Direction = enum('north east south west', name='Direction')
-    >>> Direction.west
-    Direction.west
-    >>> Direction.west == Direction.west
-    True
-    >>> Direction.west == Direction.east
-    False
-    >>> isinstance(Direction.west, Direction)
-    True
-    >>> Direction[3]
-    Direction.west
-    >>> Direction['west']
-    Direction.west
-    >>> Direction.west.name
-    'west'
-    >>> Direction.north < Direction.west
-    True
-
-Enumerations are classes; their instances represent the possible values
-of the enumeration. Because Python classes must have names, you may
-provide a `name` parameter to `enum`; if you don't, a meaningless one
-will be chosen for you.
-"""
-import random
-
-class Enumeration(type):
-    """A metaclass whose classes are enumerations.
-
-    The `values` attribute of the class is used to populate the
-    enumeration. Values may either be a list of enumerated names or a
-    string containing a space-separated list of names. When the class
-    is created, it is instantiated for each name value in `values`.
-    Each such instance is the name of the enumerated item as the sole
-    argument.
-
-    The `Enumerated` class is a good choice for a superclass.
+class OrderedEnum(Enum):
     """
-
-    def __init__(cls, name, bases, dic):
-        super(Enumeration, cls).__init__(name, bases, dic)
-
-        if 'values' not in dic:
-            # Do nothing if no values are provided (i.e., with
-            # Enumerated itself).
-            return
-
-        # May be called with a single string, in which case we split on
-        # whitespace for convenience.
-        values = dic['values']
-        if isinstance(values, basestring):
-            values = values.split()
-
-        # Create the Enumerated instances for each value. We have to use
-        # super's __setattr__ here because we disallow setattr below.
-        super(Enumeration, cls).__setattr__('_items_dict', {})
-        super(Enumeration, cls).__setattr__('_items_list', [])
-        for value in values:
-            item = cls(value, len(cls._items_list))
-            cls._items_dict[value] = item
-            cls._items_list.append(item)
-
-    def __getattr__(cls, key):
-        try:
-            return cls._items_dict[key]
-        except KeyError:
-            raise AttributeError("enumeration '" + cls.__name__ +
-                                 "' has no item '" + key + "'")
-
-    def __setattr__(cls, key, val):
-        raise TypeError("enumerations do not support attribute assignment")
-
-    def __getitem__(cls, key):
-        if isinstance(key, int):
-            return cls._items_list[key]
-        else:
-            return getattr(cls, key)
-
-    def __len__(cls):
-        return len(cls._items_list)
-
-    def __iter__(cls):
-        return iter(cls._items_list)
-
-    def __nonzero__(cls):
-        # Ensures that __len__ doesn't get called before __init__ by
-        # pydoc.
-        return True
-
-class Enumerated(object):
-    """An item in an enumeration.
-
-    Contains instance methods inherited by enumerated objects. The
-    metaclass is preset to `Enumeration` for your convenience.
-
-    Instance attributes:
-    name -- The name of the item.
-    index -- The index of the item in its enumeration.
-
-        >>> from enumeration import Enumerated
-        >>> class Garment(Enumerated):
-        ...     values = 'hat glove belt poncho lederhosen suspenders'
-        ...     def wear(self):
-        ...         print('now wearing a ' + self.name)
-        ...
-        >>> Garment.poncho.wear()
-        now wearing a poncho
+    An Enum subclass that allows comparison of members.
     """
+    def __ge__(self, other):
+        if self.__class__ is other.__class__:
+            return self.value >= other.value
+        return NotImplemented
 
-    __metaclass__ = Enumeration
+    def __gt__(self, other):
+        if self.__class__ is other.__class__:
+            return self.value > other.value
+        return NotImplemented
 
-    def __init__(self, name, index):
-        self.name = name
-        self.index = index
+    def __le__(self, other):
+        if self.__class__ is other.__class__:
+            return self.value <= other.value
+        return NotImplemented
 
-    def __str__(self):
-        return type(self).__name__ + '.' + self.name
-
-    def __repr__(self):
-        return str(self)
-
-    def __cmp__(self, other):
-        if type(self) is type(other):
-            # Note that we're assuming that the items are direct
-            # instances of the same Enumeration (i.e., no fancy
-            # subclassing), which is probably okay.
-            return cmp(self.index, other.index)
-        else:
-            return NotImplemented
-
-def enum(*values, **kwargs):
-    """Shorthand for creating a new Enumeration class.
-
-    Call with enumeration values as a list, a space-delimited string, or
-    just an argument list. To give the class a name, pass it as the
-    `name` keyword argument. Otherwise, a name will be chosen for you.
-
-    The following are all equivalent:
-
-        enum('pinkie ring middle index thumb')
-        enum('pinkie', 'ring', 'middle', 'index', 'thumb')
-        enum(['pinkie', 'ring', 'middle', 'index', 'thumb'])
-    """
-
-    if ('name' not in kwargs) or kwargs['name'] is None:
-        # Create a probably-unique name. It doesn't really have to be
-        # unique, but getting distinct names each time helps with
-        # identification in debugging.
-        name = 'Enumeration' + hex(random.randint(0,0xfffffff))[2:].upper()
-    else:
-        name = kwargs['name']
-
-    if len(values) == 1:
-        # If there's only one value, we have a couple of alternate calling
-        # styles.
-        if isinstance(values[0], basestring) or hasattr(values[0], '__iter__'):
-            values = values[0]
-
-    return type(name, (Enumerated,), {'values': values})
+    def __lt__(self, other):
+        if self.__class__ is other.__class__:
+            return self.value < other.value
+        return NotImplemented

lib/beets/util/functemplate.py

@@ -42,6 +42,7 @@ ESCAPE_CHAR = u'$'
 VARIABLE_PREFIX = '__var_'
 FUNCTION_PREFIX = '__func_'
 
+
 class Environment(object):
     """Contains the values and functions to be substituted into a
     template.
@@ -57,10 +58,12 @@ def ex_lvalue(name):
     """A variable load expression."""
     return ast.Name(name, ast.Store())
 
+
 def ex_rvalue(name):
     """A variable store expression."""
     return ast.Name(name, ast.Load())
 
+
 def ex_literal(val):
     """An int, float, long, bool, string, or None literal with the given
     value.
@@ -75,6 +78,7 @@ def ex_literal(val):
         return ast.Str(val)
     raise TypeError('no literal for {0}'.format(type(val)))
 
+
 def ex_varassign(name, expr):
     """Assign an expression into a single variable. The expression may
     either be an `ast.expr` object or a value to be used as a literal.
@@ -83,6 +87,7 @@ def ex_varassign(name, expr):
         expr = ex_literal(expr)
     return ast.Assign([ex_lvalue(name)], expr)
 
+
 def ex_call(func, args):
     """A function-call expression with only positional parameters. The
     function may be an expression or the name of a function. Each
@@ -98,6 +103,7 @@ def ex_call(func, args):
 
     return ast.Call(func, args, [], None, None)
 
+
 def compile_func(arg_names, statements, name='_the_func', debug=False):
     """Compile a list of statements as the body of a function and return
     the resulting Python function. If `debug`, then print out the
@@ -157,6 +163,7 @@ class Symbol(object):
         expr = ex_rvalue(VARIABLE_PREFIX + self.ident.encode('utf8'))
         return [expr], set([self.ident.encode('utf8')]), set()
 
+
 class Call(object):
     """A function call in a template."""
     def __init__(self, ident, args, original):
@@ -214,6 +221,7 @@ class Call(object):
         )
         return [subexpr_call], varnames, funcnames
 
+
 class Expression(object):
     """Top-level template construct: contains a list of text blobs,
     Symbols, and Calls.
@@ -259,6 +267,7 @@ class Expression(object):
 class ParseError(Exception):
     pass
 
+
 class Parser(object):
     """Parses a template expression string. Instantiate the class with
     the template source and call ``parse_expression``. The ``pos`` field
@@ -316,13 +325,13 @@ class Parser(object):
 
             next_char = self.string[self.pos + 1]
             if char == ESCAPE_CHAR and next_char in \
-                  (SYMBOL_DELIM, FUNC_DELIM, GROUP_CLOSE, ARG_SEP):
+                    (SYMBOL_DELIM, FUNC_DELIM, GROUP_CLOSE, ARG_SEP):
                 # An escaped special character ($$, $}, etc.). Note that
                 # ${ is not an escape sequence: this is ambiguous with
                 # the start of a symbol and it's not necessary (just
                 # using { suffices in all cases).
                 text_parts.append(next_char)
-                self.pos += 2 # Skip the next character.
+                self.pos += 2  # Skip the next character.
                 continue
 
             # Shift all characters collected so far into a single string.
@@ -372,7 +381,7 @@ class Parser(object):
 
         if next_char == GROUP_OPEN:
             # A symbol like ${this}.
-            self.pos += 1 # Skip opening.
+            self.pos += 1  # Skip opening.
             closer = self.string.find(GROUP_CLOSE, self.pos)
             if closer == -1 or closer == self.pos:
                 # No closing brace found or identifier is empty.
@@ -431,7 +440,7 @@ class Parser(object):
             self.parts.append(self.string[start_pos:self.pos])
             return
 
-        self.pos += 1 # Move past closing brace.
+        self.pos += 1  # Move past closing brace.
         self.parts.append(Call(ident, args, self.string[start_pos:self.pos]))
 
     def parse_argument_list(self):
@@ -472,6 +481,7 @@ class Parser(object):
         self.pos += len(ident)
         return ident
 
+
 def _parse(template):
     """Parse a top-level template string Expression. Any extraneous text
     is considered literal text.

lib/beets/util/pipeline.py

@@ -35,13 +35,13 @@ from __future__ import print_function
 import Queue
 from threading import Thread, Lock
 import sys
-import types
 
 BUBBLE = '__PIPELINE_BUBBLE__'
 POISON = '__PIPELINE_POISON__'
 
 DEFAULT_QUEUE_SIZE = 16
 
+
 def _invalidate_queue(q, val=None, sync=True):
     """Breaks a Queue such that it never blocks, always has size 1,
     and has no maximum size. get()ing from the queue returns `val`,
@@ -50,8 +50,10 @@ def _invalidate_queue(q, val=None, sync=True):
     """
     def _qsize(len=len):
         return 1
+
     def _put(item):
         pass
+
     def _get():
         return val
 
@@ -70,6 +72,7 @@ def _invalidate_queue(q, val=None, sync=True):
         if sync:
             q.mutex.release()
 
+
 class CountedQueue(Queue.Queue):
     """A queue that keeps track of the number of threads that are
     still feeding into it. The queue is poisoned when all threads are
@@ -104,6 +107,7 @@ class CountedQueue(Queue.Queue):
 
                 # Replacement _get invalidates when no items remain.
                 _old_get = self._get
+
                 def _get():
                     out = _old_get()
                     if not self.queue:
@@ -117,18 +121,67 @@ class CountedQueue(Queue.Queue):
                     # No items. Invalidate immediately.
                     _invalidate_queue(self, POISON, False)
 
+
 class MultiMessage(object):
     """A message yielded by a pipeline stage encapsulating multiple
     values to be sent to the next stage.
     """
     def __init__(self, messages):
         self.messages = messages
+
+
 def multiple(messages):
     """Yield multiple([message, ..]) from a pipeline stage to send
     multiple values to the next pipeline stage.
     """
     return MultiMessage(messages)
 
+
+def stage(func):
+    """Decorate a function to become a simple stage.
+
+    >>> @stage
+    ... def add(n, i):
+    ...     return i + n
+    >>> pipe = Pipeline([
+    ...     iter([1, 2, 3]),
+    ...     add(2),
+    ... ])
+    >>> list(pipe.pull())
+    [3, 4, 5]
+    """
+
+    def coro(*args):
+        task = None
+        while True:
+            task = yield task
+            task = func(*(args + (task,)))
+    return coro
+
+
+def mutator_stage(func):
+    """Decorate a function that manipulates items in a coroutine to
+    become a simple stage.
+
+    >>> @mutator_stage
+    ... def setkey(key, item):
+    ...     item[key] = True
+    >>> pipe = Pipeline([
+    ...     iter([{'x': False}, {'a': False}]),
+    ...     setkey('x'),
+    ... ])
+    >>> list(pipe.pull())
+    [{'x': True}, {'a': False, 'x': True}]
+    """
+
+    def coro(*args):
+        task = None
+        while True:
+            task = yield task
+            func(*(args + (task,)))
+    return coro
+
+
 def _allmsgs(obj):
     """Returns a list of all the messages encapsulated in obj. If obj
     is a MultiMessage, returns its enclosed messages. If obj is BUBBLE,
@@ -141,6 +194,7 @@ def _allmsgs(obj):
     else:
         return [obj]
 
+
 class PipelineThread(Thread):
     """Abstract base class for pipeline-stage threads."""
     def __init__(self, all_threads):
@@ -169,6 +223,7 @@ class PipelineThread(Thread):
         for thread in self.all_threads:
             thread.abort()
 
+
 class FirstPipelineThread(PipelineThread):
     """The thread running the first stage in a parallel pipeline setup.
     The coroutine should just be a generator.
@@ -209,6 +264,7 @@ class FirstPipelineThread(PipelineThread):
         # Generator finished; shut down the pipeline.
         self.out_queue.release()
 
+
 class MiddlePipelineThread(PipelineThread):
     """A thread running any stage in the pipeline except the first or
     last.
@@ -256,6 +312,7 @@ class MiddlePipelineThread(PipelineThread):
         # Pipeline is shutting down normally.
         self.out_queue.release()
 
+
 class LastPipelineThread(PipelineThread):
     """A thread running the last stage in a pipeline. The coroutine
     should yield nothing.
@@ -291,6 +348,7 @@ class LastPipelineThread(PipelineThread):
             self.abort_all(sys.exc_info())
             return
 
+
 class Pipeline(object):
     """Represents a staged pattern of work. Each stage in the pipeline
     is a coroutine that receives messages from the previous stage and
@@ -322,7 +380,8 @@ class Pipeline(object):
         messages between the stages are stored in queues of the given
         size.
         """
-        queues = [CountedQueue(queue_size) for i in range(len(self.stages)-1)]
+        queue_count = len(self.stages) - 1
+        queues = [CountedQueue(queue_size) for i in range(queue_count)]
         threads = []
 
         # Set up first stage.
@@ -330,10 +389,10 @@ class Pipeline(object):
             threads.append(FirstPipelineThread(coro, queues[0], threads))
 
         # Middle stages.
-        for i in range(1, len(self.stages)-1):
+        for i in range(1, queue_count):
             for coro in self.stages[i]:
                 threads.append(MiddlePipelineThread(
-                    coro, queues[i-1], queues[i], threads
+                    coro, queues[i - 1], queues[i], threads
                 ))
 
         # Last stage.
@@ -408,17 +467,20 @@ if __name__ == '__main__':
             print('generating %i' % i)
             time.sleep(1)
             yield i
+
     def work():
         num = yield
         while True:
             print('processing %i' % num)
             time.sleep(2)
-            num = yield num*2
+            num = yield num * 2
+
     def consume():
         while True:
             num = yield
             time.sleep(1)
             print('received %i' % num)
+
     ts_start = time.time()
     Pipeline([produce(), work(), consume()]).run_sequential()
     ts_seq = time.time()
@@ -437,6 +499,7 @@ if __name__ == '__main__':
             print('generating %i' % i)
             time.sleep(1)
             yield i
+
     def exc_work():
         num = yield
         while True:
@@ -445,10 +508,10 @@ if __name__ == '__main__':
             if num == 3:
                 raise Exception()
             num = yield num * 2
+
     def exc_consume():
         while True:
             num = yield
-            #if num == 4:
-            #   raise Exception()
             print('received %i' % num)
+
     Pipeline([exc_produce(), exc_work(), exc_consume()]).run_parallel(1)

lib/beets/vfs.py

@@ -20,6 +20,7 @@ from beets import util
 
 Node = namedtuple('Node', ['files', 'dirs'])
 
+
 def _insert(node, path, itemid):
     """Insert an item into a virtual filesystem node."""
     if len(path) == 1:
@@ -33,6 +34,7 @@ def _insert(node, path, itemid):
             node.dirs[dirname] = Node({}, {})
         _insert(node.dirs[dirname], rest, itemid)
 
+
 def libtree(lib):
     """Generates a filesystem-like directory tree for the files
     contained in `lib`. Filesystem nodes are (files, dirs) named

lib/beetsplug/embedart.py

@@ -16,6 +16,9 @@
 import os.path
 import logging
 import imghdr
+import subprocess
+import platform
+from tempfile import NamedTemporaryFile
 
 from beets.plugins import BeetsPlugin
 from beets import mediafile
@@ -25,6 +28,7 @@ from beets.util import syspath, normpath, displayable_path
 from beets.util.artresizer import ArtResizer
 from beets import config
 
+
 log = logging.getLogger('beets')
 
 
@@ -36,12 +40,19 @@ class EmbedCoverArtPlugin(BeetsPlugin):
         self.config.add({
             'maxwidth': 0,
             'auto': True,
+            'compare_threshold': 0,
+            'ifempty': False,
         })
-        if self.config['maxwidth'].get(int) and \
-                not ArtResizer.shared.local:
+
+        if self.config['maxwidth'].get(int) and not ArtResizer.shared.local:
             self.config['maxwidth'] = 0
             log.warn(u"embedart: ImageMagick or PIL not found; "
                      u"'maxwidth' option ignored")
+        if self.config['compare_threshold'].get(int) and not \
+                ArtResizer.shared.can_compare:
+            self.config['compare_threshold'] = 0
+            log.warn(u"embedart: ImageMagick 6.8.7 or higher not installed; "
+                     u"'compare_threshold' option ignored")
 
     def commands(self):
         # Embed command.
@@ -52,12 +63,15 @@ class EmbedCoverArtPlugin(BeetsPlugin):
             '-f', '--file', metavar='PATH', help='the image file to embed'
         )
         maxwidth = config['embedart']['maxwidth'].get(int)
+        compare_threshold = config['embedart']['compare_threshold'].get(int)
+        ifempty = config['embedart']['ifempty'].get(bool)
 
         def embed_func(lib, opts, args):
             if opts.file:
                 imagepath = normpath(opts.file)
                 for item in lib.items(decargs(args)):
-                    embed_item(item, imagepath, maxwidth)
+                    embed_item(item, imagepath, maxwidth, None,
+                               compare_threshold, ifempty)
             else:
                 for album in lib.albums(decargs(args)):
                     embed_album(album, maxwidth)
@@ -72,7 +86,8 @@ class EmbedCoverArtPlugin(BeetsPlugin):
 
         def extract_func(lib, opts, args):
             outpath = normpath(opts.outpath or 'cover')
-            extract(lib, outpath, decargs(args))
+            item = lib.items(decargs(args)).get()
+            extract(outpath, item)
         extract_cmd.func = extract_func
 
         # Clear command.
@@ -91,23 +106,43 @@ def album_imported(lib, album):
     """Automatically embed art into imported albums.
     """
     if album.artpath and config['embedart']['auto']:
-        embed_album(album, config['embedart']['maxwidth'].get(int))
+        embed_album(album, config['embedart']['maxwidth'].get(int), True)
 
 
-def embed_item(item, imagepath, maxwidth=None, itempath=None):
+def embed_item(item, imagepath, maxwidth=None, itempath=None,
+               compare_threshold=0, ifempty=False, as_album=False):
     """Embed an image into the item's media file.
     """
+    if compare_threshold:
+        if not check_art_similarity(item, imagepath, compare_threshold):
+            log.warn(u'Image not similar; skipping.')
+            return
+    if ifempty:
+        art = get_art(item)
+        if not art:
+            pass
+        else:
+            log.debug(u'embedart: media file contained art already {0}'.format(
+                displayable_path(imagepath)
+            ))
+            return
+    if maxwidth and not as_album:
+        imagepath = resize_image(imagepath, maxwidth)
+
     try:
+        log.debug(u'embedart: embedding {0}'.format(
+            displayable_path(imagepath)
+        ))
         item['images'] = [_mediafile_image(imagepath, maxwidth)]
-        item.try_write(itempath)
     except IOError as exc:
         log.error(u'embedart: could not read image file: {0}'.format(exc))
-    finally:
-        # We don't want to store the image in the database
+    else:
+        # We don't want to store the image in the database.
+        item.try_write(itempath)
         del item['images']
 
 
-def embed_album(album, maxwidth=None):
+def embed_album(album, maxwidth=None, quiet=False):
     """Embed album art into all of the album's items.
     """
     imagepath = album.artpath
@@ -115,39 +150,78 @@ def embed_album(album, maxwidth=None):
         log.info(u'No album art present: {0} - {1}'.
                  format(album.albumartist, album.album))
         return
-    if not os.path.isfile(imagepath):
+    if not os.path.isfile(syspath(imagepath)):
         log.error(u'Album art not found at {0}'
-                  .format(imagepath))
+                  .format(displayable_path(imagepath)))
         return
+    if maxwidth:
+        imagepath = resize_image(imagepath, maxwidth)
 
-    log.info(u'Embedding album art into {0.albumartist} - {0.album}.'
-             .format(album))
+    log.log(
+        logging.DEBUG if quiet else logging.INFO,
+        u'Embedding album art into {0.albumartist} - {0.album}.'.format(album),
+    )
 
     for item in album.items():
-        embed_item(item, imagepath, maxwidth)
+        embed_item(item, imagepath, maxwidth, None,
+                   config['embedart']['compare_threshold'].get(int),
+                   config['embedart']['ifempty'].get(bool), as_album=True)
+
+
+def resize_image(imagepath, maxwidth):
+    """Returns path to an image resized to maxwidth.
+    """
+    log.info(u'Resizing album art to {0} pixels wide'
+             .format(maxwidth))
+    imagepath = ArtResizer.shared.resize(maxwidth, syspath(imagepath))
+    return imagepath
+
+
+def check_art_similarity(item, imagepath, compare_threshold):
+    """A boolean indicating if an image is similar to embedded item art.
+    """
+    with NamedTemporaryFile(delete=True) as f:
+        art = extract(f.name, item)
+
+        if art:
+            # Converting images to grayscale tends to minimize the weight
+            # of colors in the diff score
+            cmd = 'convert {0} {1} -colorspace gray MIFF:- | ' \
+                  'compare -metric PHASH - null:'.format(syspath(imagepath),
+                                                         syspath(art))
+
+            proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
+                                    stderr=subprocess.PIPE,
+                                    close_fds=platform.system() != 'Windows',
+                                    shell=True)
+            stdout, stderr = proc.communicate()
+            if proc.returncode:
+                if proc.returncode != 1:
+                    log.warn(u'embedart: IM phashes compare failed for {0}, \
+                   {1}'.format(displayable_path(imagepath),
+                               displayable_path(art)))
+                    return
+                phashDiff = float(stderr)
+            else:
+                phashDiff = float(stdout)
+
+            log.info(u'embedart: compare PHASH score is {0}'.format(phashDiff))
+            if phashDiff > compare_threshold:
+                return False
+
+    return True
 
 
 def _mediafile_image(image_path, maxwidth=None):
     """Return a `mediafile.Image` object for the path.
-
-    If maxwidth is set the image is resized if necessary.
     """
-    if maxwidth:
-        image_path = ArtResizer.shared.resize(maxwidth, syspath(image_path))
 
     with open(syspath(image_path), 'rb') as f:
         data = f.read()
     return mediafile.Image(data, type=mediafile.ImageType.front)
 
 
-# 'extractart' command.
-
-def extract(lib, outpath, query):
-    item = lib.items(query).get()
-    if not item:
-        log.error(u'No item matches query.')
-        return
-
+def get_art(item):
     # Extract the art.
     try:
         mf = mediafile.MediaFile(syspath(item.path))
@@ -157,7 +231,18 @@ def extract(lib, outpath, query):
         ))
         return
 
-    art = mf.art
+    return mf.art
+
+# 'extractart' command.
+
+
+def extract(outpath, item):
+    if not item:
+        log.error(u'No item matches query.')
+        return
+
+    art = get_art(item)
+
     if not art:
         log.error(u'No album art present in {0} - {1}.'
                   .format(item.artist, item.title))
@@ -170,10 +255,11 @@ def extract(lib, outpath, query):
         return
     outpath += '.' + ext
 
-    log.info(u'Extracting album art from: {0.artist} - {0.title}\n'
-             u'To: {1}'.format(item, displayable_path(outpath)))
+    log.info(u'Extracting album art from: {0.artist} - {0.title} '
+             u'to: {1}'.format(item, displayable_path(outpath)))
     with open(syspath(outpath), 'wb') as f:
         f.write(art)
+    return outpath
 
 
 # 'clearart' command.
@@ -190,5 +276,5 @@ def clear(lib, query):
                 displayable_path(item.path), exc
             ))
             continue
-        mf.art = None
+        del mf.art
         mf.save()

lib/beetsplug/fetchart.py

@@ -1,5 +1,5 @@
 # This file is part of beets.
-# Copyright 2013, Adrian Sampson.
+# Copyright 2014, Adrian Sampson.
 #
 # Permission is hereby granted, free of charge, to any person obtaining
 # a copy of this software and associated documentation files (the
@@ -22,12 +22,18 @@ from tempfile import NamedTemporaryFile
 
 import requests
 
-from beets.plugins import BeetsPlugin
-from beets.util.artresizer import ArtResizer
+from beets import plugins
 from beets import importer
 from beets import ui
 from beets import util
 from beets import config
+from beets.util.artresizer import ArtResizer
+
+try:
+    import itunes
+    HAVE_ITUNES = True
+except ImportError:
+    HAVE_ITUNES = False
 
 IMAGE_EXTENSIONS = ['png', 'jpg', 'jpeg']
 CONTENT_TYPES = ('image/jpeg',)
@@ -73,17 +79,14 @@ CAA_URL = 'http://coverartarchive.org/release/{mbid}/front-500.jpg'
 CAA_GROUP_URL = 'http://coverartarchive.org/release-group/{mbid}/front-500.jpg'
 
 
-def caa_art(release_id):
-    """Return the Cover Art Archive URL given a MusicBrainz release ID.
+def caa_art(album):
+    """Return the Cover Art Archive and Cover Art Archive release group URLs
+    using album MusicBrainz release ID and release group ID.
     """
-    return CAA_URL.format(mbid=release_id)
-
-
-def caa_group_art(release_group_id):
-    """Return the Cover Art Archive release group URL given a MusicBrainz
-    release group ID.
-    """
-    return CAA_GROUP_URL.format(mbid=release_group_id)
+    if album.mb_albumid:
+        yield CAA_URL.format(mbid=album.mb_albumid)
+    if album.mb_releasegroupid:
+        yield CAA_GROUP_URL.format(mbid=album.mb_releasegroupid)
 
 
 # Art from Amazon.
@@ -92,10 +95,12 @@ AMAZON_URL = 'http://images.amazon.com/images/P/%s.%02i.LZZZZZZZ.jpg'
 AMAZON_INDICES = (1, 2)
 
 
-def art_for_asin(asin):
-    """Generate URLs for an Amazon ID (ASIN) string."""
-    for index in AMAZON_INDICES:
-        yield AMAZON_URL % (asin, index)
+def art_for_asin(album):
+    """Generate URLs using Amazon ID (ASIN) string.
+    """
+    if album.asin:
+        for index in AMAZON_INDICES:
+            yield AMAZON_URL % (album.asin, index)
 
 
 # AlbumArt.org scraper.
@@ -104,11 +109,14 @@ AAO_URL = 'http://www.albumart.org/index_detail.php'
 AAO_PAT = r'href\s*=\s*"([^>"]*)"[^>]*title\s*=\s*"View larger image"'
 
 
-def aao_art(asin):
-    """Return art URL from AlbumArt.org given an ASIN."""
+def aao_art(album):
+    """Return art URL from AlbumArt.org using album ASIN.
+    """
+    if not album.asin:
+        return
     # Get the page from albumart.org.
     try:
-        resp = requests_session.get(AAO_URL, params={'asin': asin})
+        resp = requests_session.get(AAO_URL, params={'asin': album.asin})
         log.debug(u'fetchart: scraped art URL: {0}'.format(resp.url))
     except requests.RequestException:
         log.debug(u'fetchart: error scraping art page')
@@ -118,7 +126,7 @@ def aao_art(asin):
     m = re.search(AAO_PAT, resp.text)
     if m:
         image_url = m.group(1)
-        return image_url
+        yield image_url
     else:
         log.debug(u'fetchart: no image found on page')
 
@@ -132,6 +140,8 @@ def google_art(album):
     """Return art URL from google.org given an album title and
     interpreter.
     """
+    if not (album.albumartist and album.album):
+        return
     search_string = (album.albumartist + ',' + album.album).encode('utf-8')
     response = requests_session.get(GOOGLE_URL, params={
         'v': '1.0',
@@ -145,14 +155,39 @@ def google_art(album):
         data = results['responseData']
         dataInfo = data['results']
         for myUrl in dataInfo:
-            return myUrl['unescapedUrl']
+            yield myUrl['unescapedUrl']
     except:
         log.debug(u'fetchart: error scraping art page')
         return
 
 
+# Art from the iTunes Store.
+
+def itunes_art(album):
+    """Return art URL from iTunes Store given an album title.
+    """
+    search_string = (album.albumartist + ' ' + album.album).encode('utf-8')
+    try:
+        # Isolate bugs in the iTunes library while searching.
+        try:
+            itunes_album = itunes.search_album(search_string)[0]
+        except Exception as exc:
+            log.debug('fetchart: iTunes search failed: {0}'.format(exc))
+            return
+
+        if itunes_album.get_artwork()['100']:
+            small_url = itunes_album.get_artwork()['100']
+            big_url = small_url.replace('100x100', '1200x1200')
+            yield big_url
+        else:
+            log.debug(u'fetchart: album has no artwork in iTunes Store')
+    except IndexError:
+        log.debug(u'fetchart: album not found in iTunes Store')
+
+
 # Art from the filesystem.
 
+
 def filename_priority(filename, cover_names):
     """Sort order for image names.
 
@@ -164,7 +199,8 @@ def filename_priority(filename, cover_names):
 
 
 def art_in_path(path, cover_names, cautious):
-    """Look for album art files in a specified directory."""
+    """Look for album art files in a specified directory.
+    """
     if not os.path.isdir(path):
         return
 
@@ -195,31 +231,27 @@ def art_in_path(path, cover_names, cautious):
 
 # Try each source in turn.
 
+SOURCES_ALL = [u'coverart', u'itunes', u'amazon', u'albumart', u'google']
 
-def _source_urls(album):
+ART_FUNCS = {
+    u'coverart': caa_art,
+    u'itunes': itunes_art,
+    u'albumart': aao_art,
+    u'amazon': art_for_asin,
+    u'google': google_art,
+}
+
+
+def _source_urls(album, sources=SOURCES_ALL):
     """Generate possible source URLs for an album's art. The URLs are
     not guaranteed to work so they each need to be attempted in turn.
     This allows the main `art_for_album` function to abort iteration
     through this sequence early to avoid the cost of scraping when not
     necessary.
     """
-    # Cover Art Archive.
-    if album.mb_albumid:
-        yield caa_art(album.mb_albumid)
-    if album.mb_releasegroupid:
-        yield caa_group_art(album.mb_releasegroupid)
-
-    # Amazon and AlbumArt.org.
-    if album.asin:
-        for url in art_for_asin(album.asin):
-            yield url
-        url = aao_art(album.asin)
-        if url:
-            yield url
-
-    if config['fetchart']['google_search']:
-        url = google_art(album)
-        if url:
+    for s in sources:
+        urls = ART_FUNCS[s](album)
+        for url in urls:
             yield url
 
 
@@ -245,7 +277,8 @@ def art_for_album(album, paths, maxwidth=None, local_only=False):
     # Web art sources.
     remote_priority = config['fetchart']['remote_priority'].get(bool)
     if not local_only and (remote_priority or not out):
-        for url in _source_urls(album):
+        for url in _source_urls(album,
+                                config['fetchart']['sources'].as_str_seq()):
             if maxwidth:
                 url = ArtResizer.shared.proxy_url(maxwidth, url)
             candidate = _fetch_image(url)
@@ -286,7 +319,7 @@ def batch_fetch_art(lib, albums, force, maxwidth=None):
                                           message))
 
 
-class FetchArtPlugin(BeetsPlugin):
+class FetchArtPlugin(plugins.BeetsPlugin):
     def __init__(self):
         super(FetchArtPlugin, self).__init__()
 
@@ -297,6 +330,7 @@ class FetchArtPlugin(BeetsPlugin):
             'cautious': False,
             'google_search': False,
             'cover_names': ['cover', 'front', 'art', 'album', 'folder'],
+            'sources': SOURCES_ALL,
         })
 
         # Holds paths to downloaded images between fetching them and
@@ -309,6 +343,12 @@ class FetchArtPlugin(BeetsPlugin):
             self.import_stages = [self.fetch_art]
             self.register_listener('import_task_files', self.assign_art)
 
+        available_sources = list(SOURCES_ALL)
+        if not HAVE_ITUNES and u'itunes' in available_sources:
+            available_sources.remove(u'itunes')
+        self.config['sources'] = plugins.sanitize_choices(
+            self.config['sources'].as_str_seq(), available_sources)
+
     # Asynchronous; after music is added to the library.
     def fetch_art(self, session, task):
         """Find art for the album being imported."""

lib/beetsplug/lyrics.py

@@ -18,25 +18,26 @@ from __future__ import print_function
 
 import re
 import logging
-import urllib
+import requests
 import json
 import unicodedata
+import urllib
 import difflib
 import itertools
+from HTMLParser import HTMLParseError
 
-from beets.plugins import BeetsPlugin
-from beets import ui
-from beets import config
+from beets import plugins
+from beets import config, ui
 
 
 # Global logger.
 
 log = logging.getLogger('beets')
 
-DIV_RE = re.compile(r'<(/?)div>?')
+DIV_RE = re.compile(r'<(/?)div>?', re.I)
 COMMENT_RE = re.compile(r'<!--.*-->', re.S)
 TAG_RE = re.compile(r'<[^>]*>')
-BREAK_RE = re.compile(r'<br\s*/?>')
+BREAK_RE = re.compile(r'\n?\s*<br([\s|/][^>]*)*>\s*\n?', re.I)
 URL_CHARACTERS = {
     u'\u2018': u"'",
     u'\u2019': u"'",
@@ -60,10 +61,14 @@ def fetch_url(url):
     is unreachable.
     """
     try:
-        return urllib.urlopen(url).read()
-    except IOError as exc:
-        log.debug(u'failed to fetch: {0} ({1})'.format(url, unicode(exc)))
-        return None
+        r = requests.get(url, verify=False)
+    except requests.RequestException as exc:
+        log.debug(u'lyrics request failed: {0}'.format(exc))
+        return
+    if r.status_code == requests.codes.ok:
+        return r.text
+    else:
+        log.debug(u'failed to fetch: {0} ({1})'.format(url, r.status_code))
 
 
 def unescape(text):
@@ -79,10 +84,20 @@ def unescape(text):
     return out
 
 
-def extract_text(html, starttag):
+def extract_text_between(html, start_marker, end_marker):
+    try:
+        _, html = html.split(start_marker, 1)
+        html, _ = html.split(end_marker, 1)
+    except ValueError:
+        return u''
+    return html
+
+
+def extract_text_in(html, starttag):
     """Extract the text from a <DIV> tag in the HTML starting with
     ``starttag``. Returns None if parsing fails.
     """
+
     # Strip off the leading text before opening tag.
     try:
         _, html = html.split(starttag, 1)
@@ -101,7 +116,6 @@ def extract_text(html, starttag):
         else:  # Opening tag.
             if level == 0:
                 parts.append(html[pos:match.start()])
-
             level += 1
 
         if level == -1:
@@ -110,26 +124,7 @@ def extract_text(html, starttag):
     else:
         print('no closing tag found!')
         return
-    lyrics = ''.join(parts)
-    return strip_cruft(lyrics)
-
-
-def strip_cruft(lyrics, wscollapse=True):
-    """Clean up HTML from an extracted lyrics string. For example, <BR>
-    tags are replaced with newlines.
-    """
-    lyrics = COMMENT_RE.sub('', lyrics)
-    lyrics = unescape(lyrics)
-    if wscollapse:
-        lyrics = re.sub(r'\s+', ' ', lyrics)  # Whitespace collapse.
-    lyrics = re.sub(r'<(script).*?</\1>(?s)', '', lyrics)  # Strip script tags.
-    lyrics = BREAK_RE.sub('\n', lyrics)  # <BR> newlines.
-    lyrics = re.sub(r'\n +', '\n', lyrics)
-    lyrics = re.sub(r' +\n', '\n', lyrics)
-    lyrics = TAG_RE.sub('', lyrics)  # Strip remaining HTML tags.
-    lyrics = lyrics.replace('\r', '\n')
-    lyrics = lyrics.strip()
-    return lyrics
+    return u''.join(parts)
 
 
 def search_pairs(item):
@@ -140,7 +135,7 @@ def search_pairs(item):
 
     In addition to the artist and title obtained from the `item` the
     method tries to strip extra information like paranthesized suffixes
-    and featured artists from the strings and add them as caniddates.
+    and featured artists from the strings and add them as candidates.
     The method also tries to split multiple titles separated with `/`.
     """
 
@@ -149,7 +144,7 @@ def search_pairs(item):
     artists = [artist]
 
     # Remove any featuring artists from the artists name
-    pattern = r"(.*?) (&|\b(and|ft|feat(uring)?\b))"
+    pattern = r"(.*?) {0}".format(plugins.feat_tokens())
     match = re.search(pattern, artist, re.IGNORECASE)
     if match:
         artists.append(match.group(1))
@@ -162,8 +157,8 @@ def search_pairs(item):
         titles.append(match.group(1))
 
     # Remove any featuring artists from the title
-    pattern = r"(.*?) \b(ft|feat(uring)?)\b"
-    for title in titles:
+    pattern = r"(.*?) {0}".format(plugins.feat_tokens(for_artist=False))
+    for title in titles[:]:
         match = re.search(pattern, title, re.IGNORECASE)
         if match:
             titles.append(match.group(1))
@@ -189,6 +184,19 @@ def _encode(s):
         s = s.encode('utf8', 'ignore')
     return urllib.quote(s)
 
+# Musixmatch
+
+MUSIXMATCH_URL_PATTERN = 'https://www.musixmatch.com/lyrics/%s/%s'
+
+
+def fetch_musixmatch(artist, title):
+    url = MUSIXMATCH_URL_PATTERN % (_lw_encode(artist.title()),
+                                    _lw_encode(title.title()))
+    html = fetch_url(url)
+    if not html:
+        return
+    lyrics = extract_text_between(html, '"lyrics_body":', '"lyrics_language":')
+    return lyrics.strip(',"').replace('\\n', '\n')
 
 # LyricsWiki.
 
@@ -212,7 +220,7 @@ def fetch_lyricswiki(artist, title):
     if not html:
         return
 
-    lyrics = extract_text(html, "<div class='lyricbox'>")
+    lyrics = extract_text_in(html, u"<div class='lyricbox'>")
     if lyrics and 'Unfortunately, we are not licensed' not in lyrics:
         return lyrics
 
@@ -238,8 +246,8 @@ def fetch_lyricscom(artist, title):
     html = fetch_url(url)
     if not html:
         return
-
-    lyrics = extract_text(html, '<div id="lyric_space">')
+    lyrics = extract_text_between(html, '<div id="lyrics" class="SCREENONLY" '
+                                  'itemprop="description">', '</div>')
     if not lyrics:
         return
     for not_found_str in LYRICSCOM_NOT_FOUND:
@@ -280,7 +288,6 @@ def is_page_candidate(urlLink, urlTitle, title, artist):
     artist = slugify(artist.lower())
     sitename = re.search(u"//([^/]+)/.*", slugify(urlLink.lower())).group(1)
     urlTitle = slugify(urlTitle.lower())
-
     # Check if URL title contains song title (exact match)
     if urlTitle.find(title) != -1:
         return True
@@ -289,41 +296,11 @@ def is_page_candidate(urlLink, urlTitle, title, artist):
     tokens = [by + '_' + artist for by in BY_TRANS] + \
              [artist, sitename, sitename.replace('www.', '')] + LYRICS_TRANS
     songTitle = re.sub(u'(%s)' % u'|'.join(tokens), u'', urlTitle)
-
-    typoRatio = .8
+    songTitle = songTitle.strip('_|')
+    typoRatio = .9
     return difflib.SequenceMatcher(None, songTitle, title).ratio() >= typoRatio
 
 
-def insert_line_feeds(text):
-    """Insert newlines before upper-case characters.
-    """
-    tokensStr = re.split("([a-z][A-Z])", text)
-    for idx in range(1, len(tokensStr), 2):
-        ltoken = list(tokensStr[idx])
-        tokensStr[idx] = ltoken[0] + '\n' + ltoken[1]
-    return ''.join(tokensStr)
-
-
-def sanitize_lyrics(text):
-    """Clean text, returning raw lyrics as output or None if it happens
-    that input text is actually not lyrics content.  Clean (x)html tags
-    in text, correct layout and syntax...
-    """
-    text = strip_cruft(text, False)
-
-    # Restore \n in input text
-    if '\n' not in text:
-        text = insert_line_feeds(text)
-
-    while text.count('\n\n') > text.count('\n') // 4:
-        # Remove first occurrence of \n for each sequence of \n
-        text = re.sub(r'\n(\n+)', '\g<1>', text)
-
-    text = re.sub(r'\n\n+', '\n\n', text)   # keep at most two \n in a row
-
-    return text
-
-
 def remove_credits(text):
     """Remove first/last line of text if it contains the word 'lyrics'
     eg 'Lyrics by songsdatabase.com'
@@ -342,13 +319,12 @@ def is_lyrics(text, artist=None):
     """Determine whether the text seems to be valid lyrics.
     """
     if not text:
-        return
-
+        return False
     badTriggersOcc = []
     nbLines = text.count('\n')
     if nbLines <= 1:
         log.debug(u"Ignoring too short lyrics '{0}'".format(text))
-        return 0
+        return False
     elif nbLines < 5:
         badTriggersOcc.append('too_short')
     else:
@@ -356,7 +332,7 @@ def is_lyrics(text, artist=None):
         # down
         text = remove_credits(text)
 
-    badTriggers = ['lyrics', 'copyright', 'property']
+    badTriggers = ['lyrics', 'copyright', 'property', 'links']
     if artist:
         badTriggersOcc += [artist]
 
@@ -366,62 +342,58 @@ def is_lyrics(text, artist=None):
 
     if badTriggersOcc:
         log.debug(u'Bad triggers detected: {0}'.format(badTriggersOcc))
-
     return len(badTriggersOcc) < 2
 
 
-def scrape_lyrics_from_url(url):
+def _scrape_strip_cruft(html, plain_text_out=False):
+    """Clean up HTML
+    """
+    html = unescape(html)
+
+    html = html.replace('\r', '\n')  # Normalize EOL.
+    html = re.sub(r' +', ' ', html)  # Whitespaces collapse.
+    html = BREAK_RE.sub('\n', html)  # <br> eats up surrounding '\n'.
+    html = re.sub(r'<(script).*?</\1>(?s)', '', html)  # Strip script tags.
+
+    if plain_text_out:  # Strip remaining HTML tags
+        html = COMMENT_RE.sub('', html)
+        html = TAG_RE.sub('', html)
+
+    html = '\n'.join([x.strip() for x in html.strip().split('\n')])
+    html = re.sub(r'\n{3,}', r'\n\n', html)
+    return html
+
+
+def _scrape_merge_paragraphs(html):
+    html = re.sub(r'</p>\s*<p(\s*[^>]*)>', '\n', html)
+    return re.sub(r'<div .*>\s*</div>', '\n', html)
+
+
+def scrape_lyrics_from_html(html):
     """Scrape lyrics from a URL. If no lyrics can be found, return None
     instead.
     """
-    from bs4 import BeautifulSoup, Comment
-    html = fetch_url(url)
+    from bs4 import SoupStrainer, BeautifulSoup
+
     if not html:
         return None
 
-    soup = BeautifulSoup(html)
-
-    for tag in soup.findAll('br'):
-        tag.replaceWith('\n')
-
-    # Remove non relevant html parts
-    [s.extract() for s in soup(['head', 'script'])]
-    comments = soup.findAll(text=lambda text: isinstance(text, Comment))
-    [s.extract() for s in comments]
+    def is_text_notcode(text):
+        length = len(text)
+        return (length > 20 and
+                text.count(' ') > length / 25 and
+                (text.find('{') == -1 or text.find(';') == -1))
+    html = _scrape_strip_cruft(html)
+    html = _scrape_merge_paragraphs(html)
 
+    # extract all long text blocks that are not code
     try:
-        for tag in soup.findAll(True):
-            tag.name = 'p'          # keep tag contents
-
-    except Exception, e:
-        log.debug(u'Error {0} when replacing containing marker by p marker'
-                  .format(e, exc_info=True))
-
-    # Make better soup from current soup! The previous unclosed <p> sections
-    # are now closed.  Use str() rather than prettify() as it's more
-    # conservative concerning EOL
-    soup = BeautifulSoup(str(soup))
-
-    # In case lyrics are nested in no markup but <body>
-    # Insert the whole body in a <p>
-    bodyTag = soup.find('body')
-    if bodyTag:
-        pTag = soup.new_tag("p")
-        bodyTag.parent.insert(0, pTag)
-        pTag.insert(0, bodyTag)
-
-    tagTokens = []
-
-    for tag in soup.findAll('p'):
-        soup2 = BeautifulSoup(str(tag))
-        # Extract all text of <p> section.
-        tagTokens += soup2.findAll(text=True)
-
-    if tagTokens:
-        # Lyrics are expected to be the longest paragraph
-        tagTokens = sorted(tagTokens, key=len, reverse=True)
-        soup = BeautifulSoup(tagTokens[0])
-        return unescape(tagTokens[0].strip("\n\r: "))
+        soup = BeautifulSoup(html, "html.parser",
+                             parse_only=SoupStrainer(text=is_text_notcode))
+    except HTMLParseError:
+        return None
+    soup = sorted(soup.stripped_strings, key=len)[-1]
+    return soup
 
 
 def fetch_google(artist, title):
@@ -443,15 +415,14 @@ def fetch_google(artist, title):
     if 'items' in data.keys():
         for item in data['items']:
             urlLink = item['link']
-            urlTitle = item['title']
+            urlTitle = item.get('title', u'')
             if not is_page_candidate(urlLink, urlTitle, title, artist):
                 continue
-            lyrics = scrape_lyrics_from_url(urlLink)
+            html = fetch_url(urlLink)
+            lyrics = scrape_lyrics_from_html(html)
             if not lyrics:
                 continue
 
-            lyrics = sanitize_lyrics(lyrics)
-
             if is_lyrics(lyrics, artist):
                 log.debug(u'got lyrics from {0}'.format(item['displayLink']))
                 return lyrics
@@ -459,8 +430,16 @@ def fetch_google(artist, title):
 
 # Plugin logic.
 
+SOURCES = ['google', 'lyricwiki', 'lyrics.com', 'musixmatch']
+SOURCE_BACKENDS = {
+    'google': fetch_google,
+    'lyricwiki': fetch_lyricswiki,
+    'lyrics.com': fetch_lyricscom,
+    'musixmatch': fetch_musixmatch,
+}
 
-class LyricsPlugin(BeetsPlugin):
+
+class LyricsPlugin(plugins.BeetsPlugin):
     def __init__(self):
         super(LyricsPlugin, self).__init__()
         self.import_stages = [self.imported]
@@ -469,12 +448,19 @@ class LyricsPlugin(BeetsPlugin):
             'google_API_key': None,
             'google_engine_ID': u'009217259823014548361:lndtuqkycfu',
             'fallback': None,
+            'force': False,
+            'sources': SOURCES,
         })
 
-        self.backends = [fetch_lyricswiki, fetch_lyricscom]
-
-        if self.config['google_API_key'].get():
-            self.backends.insert(0, fetch_google)
+        available_sources = list(SOURCES)
+        if not self.config['google_API_key'].get() and \
+                'google' in SOURCES:
+            available_sources.remove('google')
+        self.config['sources'] = plugins.sanitize_choices(
+            self.config['sources'].as_str_seq(), available_sources)
+        self.backends = []
+        for key in self.config['sources'].as_str_seq():
+            self.backends.append(SOURCE_BACKENDS[key])
 
     def commands(self):
         cmd = ui.Subcommand('lyrics', help='fetch song lyrics')
@@ -490,8 +476,10 @@ class LyricsPlugin(BeetsPlugin):
             # import_write config value.
             write = config['import']['write'].get(bool)
             for item in lib.items(ui.decargs(args)):
-                self.fetch_item_lyrics(lib, logging.INFO, item, write,
-                                       opts.force_refetch)
+                self.fetch_item_lyrics(
+                    lib, logging.INFO, item, write,
+                    opts.force_refetch or self.config['force'],
+                )
                 if opts.printlyr and item.lyrics:
                     ui.print_(item.lyrics)
 
@@ -504,7 +492,7 @@ class LyricsPlugin(BeetsPlugin):
         if self.config['auto']:
             for item in task.imported_items():
                 self.fetch_item_lyrics(session.lib, logging.DEBUG, item,
-                                       False, False)
+                                       False, self.config['force'])
 
     def fetch_item_lyrics(self, lib, loglevel, item, write, force):
         """Fetch and store lyrics for a single item. If ``write``, then the
@@ -551,8 +539,6 @@ class LyricsPlugin(BeetsPlugin):
         for backend in self.backends:
             lyrics = backend(artist, title)
             if lyrics:
-                if isinstance(lyrics, str):
-                    lyrics = lyrics.decode('utf8', 'ignore')
                 log.debug(u'got lyrics from backend: {0}'
                           .format(backend.__name__))
-                return lyrics.strip()
+                return _scrape_strip_cruft(lyrics, True)

lib/enum/LICENSE

@@ -0,0 +1,32 @@
+Copyright (c) 2013, Ethan Furman.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+
+    Redistributions of source code must retain the above
+    copyright notice, this list of conditions and the
+    following disclaimer.
+
+    Redistributions in binary form must reproduce the above
+    copyright notice, this list of conditions and the following
+    disclaimer in the documentation and/or other materials
+    provided with the distribution.
+
+    Neither the name Ethan Furman nor the names of any
+    contributors may be used to endorse or promote products
+    derived from this software without specific prior written
+    permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.

lib/enum/README

@@ -0,0 +1,2 @@
+enum34 is the new Python stdlib enum module available in Python 3.4
+backported for previous versions of Python from 2.4 to 3.3.

lib/enum/__init__.py

@@ -0,0 +1,790 @@
+"""Python Enumerations"""
+
+import sys as _sys
+
+__all__ = ['Enum', 'IntEnum', 'unique']
+
+version = 1, 0, 4
+
+pyver = float('%s.%s' % _sys.version_info[:2])
+
+try:
+    any
+except NameError:
+    def any(iterable):
+        for element in iterable:
+            if element:
+                return True
+        return False
+
+try:
+    from collections import OrderedDict
+except ImportError:
+    OrderedDict = None
+
+try:
+    basestring
+except NameError:
+    # In Python 2 basestring is the ancestor of both str and unicode
+    # in Python 3 it's just str, but was missing in 3.1
+    basestring = str
+
+try:
+    unicode
+except NameError:
+    # In Python 3 unicode no longer exists (it's just str)
+    unicode = str
+
+class _RouteClassAttributeToGetattr(object):
+    """Route attribute access on a class to __getattr__.
+
+    This is a descriptor, used to define attributes that act differently when
+    accessed through an instance and through a class.  Instance access remains
+    normal, but access to an attribute through a class will be routed to the
+    class's __getattr__ method; this is done by raising AttributeError.
+
+    """
+    def __init__(self, fget=None):
+        self.fget = fget
+
+    def __get__(self, instance, ownerclass=None):
+        if instance is None:
+            raise AttributeError()
+        return self.fget(instance)
+
+    def __set__(self, instance, value):
+        raise AttributeError("can't set attribute")
+
+    def __delete__(self, instance):
+        raise AttributeError("can't delete attribute")
+
+
+def _is_descriptor(obj):
+    """Returns True if obj is a descriptor, False otherwise."""
+    return (
+            hasattr(obj, '__get__') or
+            hasattr(obj, '__set__') or
+            hasattr(obj, '__delete__'))
+
+
+def _is_dunder(name):
+    """Returns True if a __dunder__ name, False otherwise."""
+    return (name[:2] == name[-2:] == '__' and
+            name[2:3] != '_' and
+            name[-3:-2] != '_' and
+            len(name) > 4)
+
+
+def _is_sunder(name):
+    """Returns True if a _sunder_ name, False otherwise."""
+    return (name[0] == name[-1] == '_' and 
+            name[1:2] != '_' and
+            name[-2:-1] != '_' and
+            len(name) > 2)
+
+
+def _make_class_unpicklable(cls):
+    """Make the given class un-picklable."""
+    def _break_on_call_reduce(self, protocol=None):
+        raise TypeError('%r cannot be pickled' % self)
+    cls.__reduce_ex__ = _break_on_call_reduce
+    cls.__module__ = '<unknown>'
+
+
+class _EnumDict(dict):
+    """Track enum member order and ensure member names are not reused.
+
+    EnumMeta will use the names found in self._member_names as the
+    enumeration member names.
+
+    """
+    def __init__(self):
+        super(_EnumDict, self).__init__()
+        self._member_names = []
+
+    def __setitem__(self, key, value):
+        """Changes anything not dundered or not a descriptor.
+
+        If a descriptor is added with the same name as an enum member, the name
+        is removed from _member_names (this may leave a hole in the numerical
+        sequence of values).
+
+        If an enum member name is used twice, an error is raised; duplicate
+        values are not checked for.
+
+        Single underscore (sunder) names are reserved.
+
+        Note:   in 3.x __order__ is simply discarded as a not necessary piece
+                leftover from 2.x
+
+        """
+        if pyver >= 3.0 and key == '__order__':
+                return
+        if _is_sunder(key):
+            raise ValueError('_names_ are reserved for future Enum use')
+        elif _is_dunder(key):
+            pass
+        elif key in self._member_names:
+            # descriptor overwriting an enum?
+            raise TypeError('Attempted to reuse key: %r' % key)
+        elif not _is_descriptor(value):
+            if key in self:
+                # enum overwriting a descriptor?
+                raise TypeError('Key already defined as: %r' % self[key])
+            self._member_names.append(key)
+        super(_EnumDict, self).__setitem__(key, value)
+
+
+# Dummy value for Enum as EnumMeta explicity checks for it, but of course until
+# EnumMeta finishes running the first time the Enum class doesn't exist.  This
+# is also why there are checks in EnumMeta like `if Enum is not None`
+Enum = None
+
+
+class EnumMeta(type):
+    """Metaclass for Enum"""
+    @classmethod
+    def __prepare__(metacls, cls, bases):
+        return _EnumDict()
+
+    def __new__(metacls, cls, bases, classdict):
+        # an Enum class is final once enumeration items have been defined; it
+        # cannot be mixed with other types (int, float, etc.) if it has an
+        # inherited __new__ unless a new __new__ is defined (or the resulting
+        # class will fail).
+        if type(classdict) is dict:
+            original_dict = classdict
+            classdict = _EnumDict()
+            for k, v in original_dict.items():
+                classdict[k] = v
+
+        member_type, first_enum = metacls._get_mixins_(bases)
+        __new__, save_new, use_args = metacls._find_new_(classdict, member_type,
+                                                        first_enum)
+        # save enum items into separate mapping so they don't get baked into
+        # the new class
+        members = dict((k, classdict[k]) for k in classdict._member_names)
+        for name in classdict._member_names:
+            del classdict[name]
+
+        # py2 support for definition order
+        __order__ = classdict.get('__order__')
+        if __order__ is None:
+            if pyver < 3.0:
+                try:
+                    __order__ = [name for (name, value) in sorted(members.items(), key=lambda item: item[1])]
+                except TypeError:
+                    __order__ = [name for name in sorted(members.keys())]
+            else:
+                __order__ = classdict._member_names
+        else:
+            del classdict['__order__']
+            if pyver < 3.0:
+                __order__ = __order__.replace(',', ' ').split()
+                aliases = [name for name in members if name not in __order__]
+                __order__ += aliases
+
+        # check for illegal enum names (any others?)
+        invalid_names = set(members) & set(['mro'])
+        if invalid_names:
+            raise ValueError('Invalid enum member name(s): %s' % (
+                ', '.join(invalid_names), ))
+
+        # create our new Enum type
+        enum_class = super(EnumMeta, metacls).__new__(metacls, cls, bases, classdict)
+        enum_class._member_names_ = []               # names in random order
+        if OrderedDict is not None:
+            enum_class._member_map_ = OrderedDict()
+        else:
+            enum_class._member_map_ = {}             # name->value map
+        enum_class._member_type_ = member_type
+
+        # Reverse value->name map for hashable values.
+        enum_class._value2member_map_ = {}
+
+        # instantiate them, checking for duplicates as we go
+        # we instantiate first instead of checking for duplicates first in case
+        # a custom __new__ is doing something funky with the values -- such as
+        # auto-numbering ;)
+        if __new__ is None:
+            __new__ = enum_class.__new__
+        for member_name in __order__:
+            value = members[member_name]
+            if not isinstance(value, tuple):
+                args = (value, )
+            else:
+                args = value
+            if member_type is tuple:   # special case for tuple enums
+                args = (args, )     # wrap it one more time
+            if not use_args or not args:
+                enum_member = __new__(enum_class)
+                if not hasattr(enum_member, '_value_'):
+                    enum_member._value_ = value
+            else:
+                enum_member = __new__(enum_class, *args)
+                if not hasattr(enum_member, '_value_'):
+                    enum_member._value_ = member_type(*args)
+            value = enum_member._value_
+            enum_member._name_ = member_name
+            enum_member.__objclass__ = enum_class
+            enum_member.__init__(*args)
+            # If another member with the same value was already defined, the
+            # new member becomes an alias to the existing one.
+            for name, canonical_member in enum_class._member_map_.items():
+                if canonical_member.value == enum_member._value_:
+                    enum_member = canonical_member
+                    break
+            else:
+                # Aliases don't appear in member names (only in __members__).
+                enum_class._member_names_.append(member_name)
+            enum_class._member_map_[member_name] = enum_member
+            try:
+                # This may fail if value is not hashable. We can't add the value
+                # to the map, and by-value lookups for this value will be
+                # linear.
+                enum_class._value2member_map_[value] = enum_member
+            except TypeError:
+                pass
+
+
+        # If a custom type is mixed into the Enum, and it does not know how
+        # to pickle itself, pickle.dumps will succeed but pickle.loads will
+        # fail.  Rather than have the error show up later and possibly far
+        # from the source, sabotage the pickle protocol for this class so
+        # that pickle.dumps also fails.
+        #
+        # However, if the new class implements its own __reduce_ex__, do not
+        # sabotage -- it's on them to make sure it works correctly.  We use
+        # __reduce_ex__ instead of any of the others as it is preferred by
+        # pickle over __reduce__, and it handles all pickle protocols.
+        unpicklable = False
+        if '__reduce_ex__' not in classdict:
+            if member_type is not object:
+                methods = ('__getnewargs_ex__', '__getnewargs__',
+                        '__reduce_ex__', '__reduce__')
+                if not any(m in member_type.__dict__ for m in methods):
+                    _make_class_unpicklable(enum_class)
+                    unpicklable = True
+
+
+        # double check that repr and friends are not the mixin's or various
+        # things break (such as pickle)
+        for name in ('__repr__', '__str__', '__format__', '__reduce_ex__'):
+            class_method = getattr(enum_class, name)
+            obj_method = getattr(member_type, name, None)
+            enum_method = getattr(first_enum, name, None)
+            if name not in classdict and class_method is not enum_method:
+                if name == '__reduce_ex__' and unpicklable:
+                    continue
+                setattr(enum_class, name, enum_method)
+
+        # method resolution and int's are not playing nice
+        # Python's less than 2.6 use __cmp__
+
+        if pyver < 2.6:
+
+            if issubclass(enum_class, int):
+                setattr(enum_class, '__cmp__', getattr(int, '__cmp__'))
+
+        elif pyver < 3.0:
+
+            if issubclass(enum_class, int):
+                for method in (
+                        '__le__',
+                        '__lt__',
+                        '__gt__',
+                        '__ge__',
+                        '__eq__',
+                        '__ne__',
+                        '__hash__',
+                        ):
+                    setattr(enum_class, method, getattr(int, method))
+
+        # replace any other __new__ with our own (as long as Enum is not None,
+        # anyway) -- again, this is to support pickle
+        if Enum is not None:
+            # if the user defined their own __new__, save it before it gets
+            # clobbered in case they subclass later
+            if save_new:
+                setattr(enum_class, '__member_new__', enum_class.__dict__['__new__'])
+            setattr(enum_class, '__new__', Enum.__dict__['__new__'])
+        return enum_class
+
+    def __call__(cls, value, names=None, module=None, type=None):
+        """Either returns an existing member, or creates a new enum class.
+
+        This method is used both when an enum class is given a value to match
+        to an enumeration member (i.e. Color(3)) and for the functional API
+        (i.e. Color = Enum('Color', names='red green blue')).
+
+        When used for the functional API: `module`, if set, will be stored in
+        the new class' __module__ attribute; `type`, if set, will be mixed in
+        as the first base class.
+
+        Note: if `module` is not set this routine will attempt to discover the
+        calling module by walking the frame stack; if this is unsuccessful
+        the resulting class will not be pickleable.
+
+        """
+        if names is None:  # simple value lookup
+            return cls.__new__(cls, value)
+        # otherwise, functional API: we're creating a new Enum type
+        return cls._create_(value, names, module=module, type=type)
+
+    def __contains__(cls, member):
+        return isinstance(member, cls) and member.name in cls._member_map_
+
+    def __delattr__(cls, attr):
+        # nicer error message when someone tries to delete an attribute
+        # (see issue19025).
+        if attr in cls._member_map_:
+            raise AttributeError(
+                    "%s: cannot delete Enum member." % cls.__name__)
+        super(EnumMeta, cls).__delattr__(attr)
+
+    def __dir__(self):
+        return (['__class__', '__doc__', '__members__', '__module__'] +
+                self._member_names_)
+
+    @property
+    def __members__(cls):
+        """Returns a mapping of member name->value.
+
+        This mapping lists all enum members, including aliases. Note that this
+        is a copy of the internal mapping.
+
+        """
+        return cls._member_map_.copy()
+
+    def __getattr__(cls, name):
+        """Return the enum member matching `name`
+
+        We use __getattr__ instead of descriptors or inserting into the enum
+        class' __dict__ in order to support `name` and `value` being both
+        properties for enum members (which live in the class' __dict__) and
+        enum members themselves.
+
+        """
+        if _is_dunder(name):
+            raise AttributeError(name)
+        try:
+            return cls._member_map_[name]
+        except KeyError:
+            raise AttributeError(name)
+
+    def __getitem__(cls, name):
+        return cls._member_map_[name]
+
+    def __iter__(cls):
+        return (cls._member_map_[name] for name in cls._member_names_)
+
+    def __reversed__(cls):
+        return (cls._member_map_[name] for name in reversed(cls._member_names_))
+
+    def __len__(cls):
+        return len(cls._member_names_)
+
+    def __repr__(cls):
+        return "<enum %r>" % cls.__name__
+
+    def __setattr__(cls, name, value):
+        """Block attempts to reassign Enum members.
+
+        A simple assignment to the class namespace only changes one of the
+        several possible ways to get an Enum member from the Enum class,
+        resulting in an inconsistent Enumeration.
+
+        """
+        member_map = cls.__dict__.get('_member_map_', {})
+        if name in member_map:
+            raise AttributeError('Cannot reassign members.')
+        super(EnumMeta, cls).__setattr__(name, value)
+
+    def _create_(cls, class_name, names=None, module=None, type=None):
+        """Convenience method to create a new Enum class.
+
+        `names` can be:
+
+        * A string containing member names, separated either with spaces or
+          commas.  Values are auto-numbered from 1.
+        * An iterable of member names.  Values are auto-numbered from 1.
+        * An iterable of (member name, value) pairs.
+        * A mapping of member name -> value.
+
+        """
+        if pyver < 3.0:
+            # if class_name is unicode, attempt a conversion to ASCII
+            if isinstance(class_name, unicode):
+                try:
+                    class_name = class_name.encode('ascii')
+                except UnicodeEncodeError:
+                    raise TypeError('%r is not representable in ASCII' % class_name)
+        metacls = cls.__class__
+        if type is None:
+            bases = (cls, )
+        else:
+            bases = (type, cls)
+        classdict = metacls.__prepare__(class_name, bases)
+        __order__ = []
+
+        # special processing needed for names?
+        if isinstance(names, basestring):
+            names = names.replace(',', ' ').split()
+        if isinstance(names, (tuple, list)) and isinstance(names[0], basestring):
+            names = [(e, i+1) for (i, e) in enumerate(names)]
+
+        # Here, names is either an iterable of (name, value) or a mapping.
+        for item in names:
+            if isinstance(item, basestring):
+                member_name, member_value = item, names[item]
+            else:
+                member_name, member_value = item
+            classdict[member_name] = member_value
+            __order__.append(member_name)
+        # only set __order__ in classdict if name/value was not from a mapping
+        if not isinstance(item, basestring):
+            classdict['__order__'] = ' '.join(__order__)
+        enum_class = metacls.__new__(metacls, class_name, bases, classdict)
+
+        # TODO: replace the frame hack if a blessed way to know the calling
+        # module is ever developed
+        if module is None:
+            try:
+                module = _sys._getframe(2).f_globals['__name__']
+            except (AttributeError, ValueError):
+                pass
+        if module is None:
+            _make_class_unpicklable(enum_class)
+        else:
+            enum_class.__module__ = module
+
+        return enum_class
+
+    @staticmethod
+    def _get_mixins_(bases):
+        """Returns the type for creating enum members, and the first inherited
+        enum class.
+
+        bases: the tuple of bases that was given to __new__
+
+        """
+        if not bases or Enum is None:
+            return object, Enum
+        
+
+        # double check that we are not subclassing a class with existing
+        # enumeration members; while we're at it, see if any other data
+        # type has been mixed in so we can use the correct __new__
+        member_type = first_enum = None
+        for base in bases:
+            if  (base is not Enum and
+                    issubclass(base, Enum) and
+                    base._member_names_):
+                raise TypeError("Cannot extend enumerations")
+        # base is now the last base in bases
+        if not issubclass(base, Enum):
+            raise TypeError("new enumerations must be created as "
+                    "`ClassName([mixin_type,] enum_type)`")
+
+        # get correct mix-in type (either mix-in type of Enum subclass, or
+        # first base if last base is Enum)
+        if not issubclass(bases[0], Enum):
+            member_type = bases[0]     # first data type
+            first_enum = bases[-1]  # enum type
+        else:
+            for base in bases[0].__mro__:
+                # most common: (IntEnum, int, Enum, object)
+                # possible:    (<Enum 'AutoIntEnum'>, <Enum 'IntEnum'>,
+                #               <class 'int'>, <Enum 'Enum'>,
+                #               <class 'object'>)
+                if issubclass(base, Enum):
+                    if first_enum is None:
+                        first_enum = base
+                else:
+                    if member_type is None:
+                        member_type = base
+
+        return member_type, first_enum
+
+    if pyver < 3.0:
+        @staticmethod
+        def _find_new_(classdict, member_type, first_enum):
+            """Returns the __new__ to be used for creating the enum members.
+
+            classdict: the class dictionary given to __new__
+            member_type: the data type whose __new__ will be used by default
+            first_enum: enumeration to check for an overriding __new__
+
+            """
+            # now find the correct __new__, checking to see of one was defined
+            # by the user; also check earlier enum classes in case a __new__ was
+            # saved as __member_new__
+            __new__ = classdict.get('__new__', None)
+            if __new__:
+                return None, True, True      # __new__, save_new, use_args
+
+            N__new__ = getattr(None, '__new__')
+            O__new__ = getattr(object, '__new__')
+            if Enum is None:
+                E__new__ = N__new__
+            else:
+                E__new__ = Enum.__dict__['__new__']
+            # check all possibles for __member_new__ before falling back to
+            # __new__
+            for method in ('__member_new__', '__new__'):
+                for possible in (member_type, first_enum):
+                    try:
+                        target = possible.__dict__[method]
+                    except (AttributeError, KeyError):
+                        target = getattr(possible, method, None)
+                    if target not in [
+                            None,
+                            N__new__,
+                            O__new__,
+                            E__new__,
+                            ]:
+                        if method == '__member_new__':
+                            classdict['__new__'] = target
+                            return None, False, True
+                        if isinstance(target, staticmethod):
+                            target = target.__get__(member_type)
+                        __new__ = target
+                        break
+                if __new__ is not None:
+                    break
+            else:
+                __new__ = object.__new__
+
+            # if a non-object.__new__ is used then whatever value/tuple was
+            # assigned to the enum member name will be passed to __new__ and to the
+            # new enum member's __init__
+            if __new__ is object.__new__:
+                use_args = False
+            else:
+                use_args = True
+
+            return __new__, False, use_args
+    else:
+        @staticmethod
+        def _find_new_(classdict, member_type, first_enum):
+            """Returns the __new__ to be used for creating the enum members.
+
+            classdict: the class dictionary given to __new__
+            member_type: the data type whose __new__ will be used by default
+            first_enum: enumeration to check for an overriding __new__
+
+            """
+            # now find the correct __new__, checking to see of one was defined
+            # by the user; also check earlier enum classes in case a __new__ was
+            # saved as __member_new__
+            __new__ = classdict.get('__new__', None)
+
+            # should __new__ be saved as __member_new__ later?
+            save_new = __new__ is not None
+
+            if __new__ is None:
+                # check all possibles for __member_new__ before falling back to
+                # __new__
+                for method in ('__member_new__', '__new__'):
+                    for possible in (member_type, first_enum):
+                        target = getattr(possible, method, None)
+                        if target not in (
+                                None,
+                                None.__new__,
+                                object.__new__,
+                                Enum.__new__,
+                                ):
+                            __new__ = target
+                            break
+                    if __new__ is not None:
+                        break
+                else:
+                    __new__ = object.__new__
+
+            # if a non-object.__new__ is used then whatever value/tuple was
+            # assigned to the enum member name will be passed to __new__ and to the
+            # new enum member's __init__
+            if __new__ is object.__new__:
+                use_args = False
+            else:
+                use_args = True
+
+            return __new__, save_new, use_args
+
+
+########################################################
+# In order to support Python 2 and 3 with a single
+# codebase we have to create the Enum methods separately
+# and then use the `type(name, bases, dict)` method to
+# create the class.
+########################################################
+temp_enum_dict = {}
+temp_enum_dict['__doc__'] = "Generic enumeration.\n\n    Derive from this class to define new enumerations.\n\n"
+
+def __new__(cls, value):
+    # all enum instances are actually created during class construction
+    # without calling this method; this method is called by the metaclass'
+    # __call__ (i.e. Color(3) ), and by pickle
+    if type(value) is cls:
+        # For lookups like Color(Color.red)
+        value = value.value
+        #return value
+    # by-value search for a matching enum member
+    # see if it's in the reverse mapping (for hashable values)
+    try:
+        if value in cls._value2member_map_:
+            return cls._value2member_map_[value]
+    except TypeError:
+        # not there, now do long search -- O(n) behavior
+        for member in cls._member_map_.values():
+            if member.value == value:
+                return member
+    raise ValueError("%s is not a valid %s" % (value, cls.__name__))
+temp_enum_dict['__new__'] = __new__
+del __new__
+
+def __repr__(self):
+    return "<%s.%s: %r>" % (
+            self.__class__.__name__, self._name_, self._value_)
+temp_enum_dict['__repr__'] = __repr__
+del __repr__
+
+def __str__(self):
+    return "%s.%s" % (self.__class__.__name__, self._name_)
+temp_enum_dict['__str__'] = __str__
+del __str__
+
+def __dir__(self):
+    added_behavior = [
+            m
+            for cls in self.__class__.mro()
+            for m in cls.__dict__
+            if m[0] != '_'
+            ]
+    return (['__class__', '__doc__', '__module__', ] + added_behavior)
+temp_enum_dict['__dir__'] = __dir__
+del __dir__
+
+def __format__(self, format_spec):
+    # mixed-in Enums should use the mixed-in type's __format__, otherwise
+    # we can get strange results with the Enum name showing up instead of
+    # the value
+
+    # pure Enum branch
+    if self._member_type_ is object:
+        cls = str
+        val = str(self)
+    # mix-in branch
+    else:
+        cls = self._member_type_
+        val = self.value
+    return cls.__format__(val, format_spec)
+temp_enum_dict['__format__'] = __format__
+del __format__
+
+
+####################################
+# Python's less than 2.6 use __cmp__
+
+if pyver < 2.6:
+
+    def __cmp__(self, other):
+        if type(other) is self.__class__:
+            if self is other:
+                return 0
+            return -1
+        return NotImplemented
+        raise TypeError("unorderable types: %s() and %s()" % (self.__class__.__name__, other.__class__.__name__))
+    temp_enum_dict['__cmp__'] = __cmp__
+    del __cmp__
+
+else:
+
+    def __le__(self, other):
+        raise TypeError("unorderable types: %s() <= %s()" % (self.__class__.__name__, other.__class__.__name__))
+    temp_enum_dict['__le__'] = __le__
+    del __le__
+
+    def __lt__(self, other):
+        raise TypeError("unorderable types: %s() < %s()" % (self.__class__.__name__, other.__class__.__name__))
+    temp_enum_dict['__lt__'] = __lt__
+    del __lt__
+
+    def __ge__(self, other):
+        raise TypeError("unorderable types: %s() >= %s()" % (self.__class__.__name__, other.__class__.__name__))
+    temp_enum_dict['__ge__'] = __ge__
+    del __ge__
+
+    def __gt__(self, other):
+        raise TypeError("unorderable types: %s() > %s()" % (self.__class__.__name__, other.__class__.__name__))
+    temp_enum_dict['__gt__'] = __gt__
+    del __gt__
+    
+
+def __eq__(self, other):
+    if type(other) is self.__class__:
+        return self is other
+    return NotImplemented
+temp_enum_dict['__eq__'] = __eq__
+del __eq__
+
+def __ne__(self, other):
+    if type(other) is self.__class__:
+        return self is not other
+    return NotImplemented
+temp_enum_dict['__ne__'] = __ne__
+del __ne__
+
+def __hash__(self):
+    return hash(self._name_)
+temp_enum_dict['__hash__'] = __hash__
+del __hash__
+
+def __reduce_ex__(self, proto):
+    return self.__class__, (self._value_, )
+temp_enum_dict['__reduce_ex__'] = __reduce_ex__
+del __reduce_ex__
+
+# _RouteClassAttributeToGetattr is used to provide access to the `name`
+# and `value` properties of enum members while keeping some measure of
+# protection from modification, while still allowing for an enumeration
+# to have members named `name` and `value`.  This works because enumeration
+# members are not set directly on the enum class -- __getattr__ is
+# used to look them up.
+
+@_RouteClassAttributeToGetattr
+def name(self):
+    return self._name_
+temp_enum_dict['name'] = name
+del name
+
+@_RouteClassAttributeToGetattr
+def value(self):
+    return self._value_
+temp_enum_dict['value'] = value
+del value
+
+Enum = EnumMeta('Enum', (object, ), temp_enum_dict)
+del temp_enum_dict
+
+# Enum has now been created
+###########################
+
+class IntEnum(int, Enum):
+    """Enum where members are also (and must be) ints"""
+
+
+def unique(enumeration):
+    """Class decorator that ensures only unique members exist in an enumeration."""
+    duplicates = []
+    for name, member in enumeration.__members__.items():
+        if name != member.name:
+            duplicates.append((name, member.name))
+    if duplicates:
+        duplicate_names = ', '.join(
+                ["%s -> %s" % (alias, name) for (alias, name) in duplicates]
+                )
+        raise ValueError('duplicate names found in %r: %s' %
+                (enumeration, duplicate_names)
+                )
+    return enumeration

lib/enum/doc/enum.rst

@@ -0,0 +1,725 @@
+``enum`` --- support for enumerations
+========================================
+
+.. :synopsis: enumerations are sets of symbolic names bound to unique, constant
+  values.
+.. :moduleauthor:: Ethan Furman <ethan@stoneleaf.us>
+.. :sectionauthor:: Barry Warsaw <barry@python.org>,
+.. :sectionauthor:: Eli Bendersky <eliben@gmail.com>,
+.. :sectionauthor:: Ethan Furman <ethan@stoneleaf.us>
+
+----------------
+
+An enumeration is a set of symbolic names (members) bound to unique, constant
+values.  Within an enumeration, the members can be compared by identity, and
+the enumeration itself can be iterated over.
+
+
+Module Contents
+---------------
+
+This module defines two enumeration classes that can be used to define unique
+sets of names and values: ``Enum`` and ``IntEnum``.  It also defines
+one decorator, ``unique``.
+
+``Enum``
+
+Base class for creating enumerated constants.  See section `Functional API`_
+for an alternate construction syntax.
+
+``IntEnum``
+
+Base class for creating enumerated constants that are also subclasses of ``int``.
+
+``unique``
+
+Enum class decorator that ensures only one name is bound to any one value.
+
+
+Creating an Enum
+----------------
+
+Enumerations are created using the ``class`` syntax, which makes them
+easy to read and write.  An alternative creation method is described in
+`Functional API`_.  To define an enumeration, subclass ``Enum`` as
+follows::
+
+    >>> from enum import Enum
+    >>> class Color(Enum):
+    ...     red = 1
+    ...     green = 2
+    ...     blue = 3
+
+Note: Nomenclature
+
+  - The class ``Color`` is an *enumeration* (or *enum*)
+  - The attributes ``Color.red``, ``Color.green``, etc., are
+    *enumeration members* (or *enum members*).
+  - The enum members have *names* and *values* (the name of
+    ``Color.red`` is ``red``, the value of ``Color.blue`` is
+    ``3``, etc.)
+    
+Note:
+
+    Even though we use the ``class`` syntax to create Enums, Enums
+    are not normal Python classes.  See `How are Enums different?`_ for
+    more details.
+
+Enumeration members have human readable string representations::
+
+    >>> print(Color.red)
+    Color.red
+
+...while their ``repr`` has more information::
+
+    >>> print(repr(Color.red))
+    <Color.red: 1>
+
+The *type* of an enumeration member is the enumeration it belongs to::
+
+    >>> type(Color.red)
+    <enum 'Color'>
+    >>> isinstance(Color.green, Color)
+    True
+    >>>
+
+Enum members also have a property that contains just their item name::
+
+    >>> print(Color.red.name)
+    red
+
+Enumerations support iteration.  In Python 3.x definition order is used; in
+Python 2.x the definition order is not available, but class attribute
+``__order__`` is supported;  otherwise, value order is used::
+
+    >>> class Shake(Enum):
+    ...   __order__ = 'vanilla chocolate cookies mint'  # only needed in 2.x
+    ...   vanilla = 7
+    ...   chocolate = 4
+    ...   cookies = 9
+    ...   mint = 3
+    ...
+    >>> for shake in Shake:
+    ...   print(shake)
+    ...
+    Shake.vanilla
+    Shake.chocolate
+    Shake.cookies
+    Shake.mint
+
+The ``__order__`` attribute is always removed, and in 3.x it is also ignored
+(order is definition order); however, in the stdlib version it will be ignored
+but not removed.
+
+Enumeration members are hashable, so they can be used in dictionaries and sets::
+
+    >>> apples = {}
+    >>> apples[Color.red] = 'red delicious'
+    >>> apples[Color.green] = 'granny smith'
+    >>> apples == {Color.red: 'red delicious', Color.green: 'granny smith'}
+    True
+
+
+Programmatic access to enumeration members and their attributes
+---------------------------------------------------------------
+
+Sometimes it's useful to access members in enumerations programmatically (i.e.
+situations where ``Color.red`` won't do because the exact color is not known
+at program-writing time).  ``Enum`` allows such access::
+
+    >>> Color(1)
+    <Color.red: 1>
+    >>> Color(3)
+    <Color.blue: 3>
+
+If you want to access enum members by *name*, use item access::
+
+    >>> Color['red']
+    <Color.red: 1>
+    >>> Color['green']
+    <Color.green: 2>
+
+If have an enum member and need its ``name`` or ``value``::
+
+    >>> member = Color.red
+    >>> member.name
+    'red'
+    >>> member.value
+    1
+
+
+Duplicating enum members and values
+-----------------------------------
+
+Having two enum members (or any other attribute) with the same name is invalid;
+in Python 3.x this would raise an error, but in Python 2.x the second member
+simply overwrites the first::
+
+    >>> # python 2.x
+    >>> class Shape(Enum):
+    ...   square = 2
+    ...   square = 3
+    ...
+    >>> Shape.square
+    <Shape.square: 3>
+
+    >>> # python 3.x
+    >>> class Shape(Enum):
+    ...   square = 2
+    ...   square = 3
+    Traceback (most recent call last):
+    ...
+    TypeError: Attempted to reuse key: 'square'
+
+However, two enum members are allowed to have the same value.  Given two members
+A and B with the same value (and A defined first), B is an alias to A.  By-value
+lookup of the value of A and B will return A.  By-name lookup of B will also
+return A::
+
+    >>> class Shape(Enum):
+    ...   __order__ = 'square diamond circle alias_for_square'  # only needed in 2.x
+    ...   square = 2
+    ...   diamond = 1
+    ...   circle = 3
+    ...   alias_for_square = 2
+    ...
+    >>> Shape.square
+    <Shape.square: 2>
+    >>> Shape.alias_for_square
+    <Shape.square: 2>
+    >>> Shape(2)
+    <Shape.square: 2>
+
+
+Allowing aliases is not always desirable.  ``unique`` can be used to ensure
+that none exist in a particular enumeration::
+
+    >>> from enum import unique
+    >>> @unique
+    ... class Mistake(Enum):
+    ...   __order__ = 'one two three four'  # only needed in 2.x
+    ...   one = 1
+    ...   two = 2
+    ...   three = 3
+    ...   four = 3
+    Traceback (most recent call last):
+    ...
+    ValueError: duplicate names found in <enum 'Mistake'>: four -> three
+
+Iterating over the members of an enum does not provide the aliases::
+
+    >>> list(Shape)
+    [<Shape.square: 2>, <Shape.diamond: 1>, <Shape.circle: 3>]
+
+The special attribute ``__members__`` is a dictionary mapping names to members.
+It includes all names defined in the enumeration, including the aliases::
+
+    >>> for name, member in sorted(Shape.__members__.items()):
+    ...   name, member
+    ...
+    ('alias_for_square', <Shape.square: 2>)
+    ('circle', <Shape.circle: 3>)
+    ('diamond', <Shape.diamond: 1>)
+    ('square', <Shape.square: 2>)
+
+The ``__members__`` attribute can be used for detailed programmatic access to
+the enumeration members.  For example, finding all the aliases::
+
+    >>> [name for name, member in Shape.__members__.items() if member.name != name]
+    ['alias_for_square']
+
+Comparisons
+-----------
+
+Enumeration members are compared by identity::
+
+    >>> Color.red is Color.red
+    True
+    >>> Color.red is Color.blue
+    False
+    >>> Color.red is not Color.blue
+    True
+
+Ordered comparisons between enumeration values are *not* supported.  Enum
+members are not integers (but see `IntEnum`_ below)::
+
+    >>> Color.red < Color.blue
+    Traceback (most recent call last):
+      File "<stdin>", line 1, in <module>
+    TypeError: unorderable types: Color() < Color()
+
+.. warning::
+
+    In Python 2 *everything* is ordered, even though the ordering may not
+    make sense.  If you want your enumerations to have a sensible ordering
+    check out the `OrderedEnum`_ recipe below.
+
+
+Equality comparisons are defined though::
+
+    >>> Color.blue == Color.red
+    False
+    >>> Color.blue != Color.red
+    True
+    >>> Color.blue == Color.blue
+    True
+
+Comparisons against non-enumeration values will always compare not equal
+(again, ``IntEnum`` was explicitly designed to behave differently, see
+below)::
+
+    >>> Color.blue == 2
+    False
+
+
+Allowed members and attributes of enumerations
+----------------------------------------------
+
+The examples above use integers for enumeration values.  Using integers is
+short and handy (and provided by default by the `Functional API`_), but not
+strictly enforced.  In the vast majority of use-cases, one doesn't care what
+the actual value of an enumeration is.  But if the value *is* important,
+enumerations can have arbitrary values.
+
+Enumerations are Python classes, and can have methods and special methods as
+usual.  If we have this enumeration::
+
+    >>> class Mood(Enum):
+    ...   funky = 1
+    ...   happy = 3
+    ... 
+    ...   def describe(self):
+    ...     # self is the member here
+    ...     return self.name, self.value
+    ... 
+    ...   def __str__(self):
+    ...     return 'my custom str! {0}'.format(self.value)
+    ... 
+    ...   @classmethod
+    ...   def favorite_mood(cls):
+    ...     # cls here is the enumeration
+    ...     return cls.happy
+
+Then::
+
+    >>> Mood.favorite_mood()
+    <Mood.happy: 3>
+    >>> Mood.happy.describe()
+    ('happy', 3)
+    >>> str(Mood.funky)
+    'my custom str! 1'
+
+The rules for what is allowed are as follows: _sunder_ names (starting and
+ending with a single underscore) are reserved by enum and cannot be used;
+all other attributes defined within an enumeration will become members of this
+enumeration, with the exception of *__dunder__* names and descriptors (methods
+are also descriptors).
+
+Note:
+
+    If your enumeration defines ``__new__`` and/or ``__init__`` then
+    whatever value(s) were given to the enum member will be passed into
+    those methods.  See `Planet`_ for an example.
+
+
+Restricted subclassing of enumerations
+--------------------------------------
+
+Subclassing an enumeration is allowed only if the enumeration does not define
+any members.  So this is forbidden::
+
+    >>> class MoreColor(Color):
+    ...   pink = 17
+    Traceback (most recent call last):
+    ...
+    TypeError: Cannot extend enumerations
+
+But this is allowed::
+
+    >>> class Foo(Enum):
+    ...   def some_behavior(self):
+    ...     pass
+    ...
+    >>> class Bar(Foo):
+    ...   happy = 1
+    ...   sad = 2
+    ...
+
+Allowing subclassing of enums that define members would lead to a violation of
+some important invariants of types and instances.  On the other hand, it makes
+sense to allow sharing some common behavior between a group of enumerations.
+(See `OrderedEnum`_ for an example.)
+
+
+Pickling
+--------
+
+Enumerations can be pickled and unpickled::
+
+    >>> from enum.test_enum import Fruit
+    >>> from pickle import dumps, loads
+    >>> Fruit.tomato is loads(dumps(Fruit.tomato, 2))
+    True
+
+The usual restrictions for pickling apply: picklable enums must be defined in
+the top level of a module, since unpickling requires them to be importable
+from that module.
+
+Note:
+
+    With pickle protocol version 4 (introduced in Python 3.4) it is possible
+    to easily pickle enums nested in other classes.
+
+
+
+Functional API
+--------------
+
+The ``Enum`` class is callable, providing the following functional API::
+
+    >>> Animal = Enum('Animal', 'ant bee cat dog')
+    >>> Animal
+    <enum 'Animal'>
+    >>> Animal.ant
+    <Animal.ant: 1>
+    >>> Animal.ant.value
+    1
+    >>> list(Animal)
+    [<Animal.ant: 1>, <Animal.bee: 2>, <Animal.cat: 3>, <Animal.dog: 4>]
+
+The semantics of this API resemble ``namedtuple``. The first argument
+of the call to ``Enum`` is the name of the enumeration. 
+
+The second argument is the *source* of enumeration member names.  It can be a
+whitespace-separated string of names, a sequence of names, a sequence of
+2-tuples with key/value pairs, or a mapping (e.g. dictionary) of names to
+values.  The last two options enable assigning arbitrary values to
+enumerations; the others auto-assign increasing integers starting with 1.  A
+new class derived from ``Enum`` is returned.  In other words, the above
+assignment to ``Animal`` is equivalent to::
+
+    >>> class Animals(Enum):
+    ...   ant = 1
+    ...   bee = 2
+    ...   cat = 3
+    ...   dog = 4
+
+Pickling enums created with the functional API can be tricky as frame stack
+implementation details are used to try and figure out which module the
+enumeration is being created in (e.g. it will fail if you use a utility
+function in separate module, and also may not work on IronPython or Jython).
+The solution is to specify the module name explicitly as follows::
+
+    >>> Animals = Enum('Animals', 'ant bee cat dog', module=__name__)
+
+Derived Enumerations
+--------------------
+
+IntEnum
+^^^^^^^
+
+A variation of ``Enum`` is provided which is also a subclass of
+``int``.  Members of an ``IntEnum`` can be compared to integers;
+by extension, integer enumerations of different types can also be compared
+to each other::
+
+    >>> from enum import IntEnum
+    >>> class Shape(IntEnum):
+    ...   circle = 1
+    ...   square = 2
+    ...
+    >>> class Request(IntEnum):
+    ...   post = 1
+    ...   get = 2
+    ...
+    >>> Shape == 1
+    False
+    >>> Shape.circle == 1
+    True
+    >>> Shape.circle == Request.post
+    True
+
+However, they still can't be compared to standard ``Enum`` enumerations::
+
+    >>> class Shape(IntEnum):
+    ...   circle = 1
+    ...   square = 2
+    ...
+    >>> class Color(Enum):
+    ...   red = 1
+    ...   green = 2
+    ...
+    >>> Shape.circle == Color.red
+    False
+
+``IntEnum`` values behave like integers in other ways you'd expect::
+
+    >>> int(Shape.circle)
+    1
+    >>> ['a', 'b', 'c'][Shape.circle]
+    'b'
+    >>> [i for i in range(Shape.square)]
+    [0, 1]
+
+For the vast majority of code, ``Enum`` is strongly recommended,
+since ``IntEnum`` breaks some semantic promises of an enumeration (by
+being comparable to integers, and thus by transitivity to other
+unrelated enumerations).  It should be used only in special cases where
+there's no other choice; for example, when integer constants are
+replaced with enumerations and backwards compatibility is required with code
+that still expects integers.
+
+
+Others
+^^^^^^
+
+While ``IntEnum`` is part of the ``enum`` module, it would be very
+simple to implement independently::
+
+    class IntEnum(int, Enum):
+        pass
+
+This demonstrates how similar derived enumerations can be defined; for example
+a ``StrEnum`` that mixes in ``str`` instead of ``int``.
+
+Some rules:
+
+1. When subclassing ``Enum``, mix-in types must appear before
+   ``Enum`` itself in the sequence of bases, as in the ``IntEnum``
+   example above.
+2. While ``Enum`` can have members of any type, once you mix in an
+   additional type, all the members must have values of that type, e.g.
+   ``int`` above.  This restriction does not apply to mix-ins which only
+   add methods and don't specify another data type such as ``int`` or
+   ``str``.
+3. When another data type is mixed in, the ``value`` attribute is *not the
+   same* as the enum member itself, although it is equivalant and will compare
+   equal.
+4. %-style formatting:  ``%s`` and ``%r`` call ``Enum``'s ``__str__`` and
+   ``__repr__`` respectively; other codes (such as ``%i`` or ``%h`` for
+   IntEnum) treat the enum member as its mixed-in type.
+
+   Note: Prior to Python 3.4 there is a bug in ``str``'s %-formatting: ``int``
+   subclasses are printed as strings and not numbers when the ``%d``, ``%i``,
+   or ``%u`` codes are used.
+5. ``str.__format__`` (or ``format``) will use the mixed-in
+   type's ``__format__``.  If the ``Enum``'s ``str`` or
+   ``repr`` is desired use the ``!s`` or ``!r`` ``str`` format codes.
+
+
+Decorators
+----------
+
+unique
+^^^^^^
+
+A ``class`` decorator specifically for enumerations.  It searches an
+enumeration's ``__members__`` gathering any aliases it finds; if any are
+found ``ValueError`` is raised with the details::
+
+    >>> @unique
+    ... class NoDupes(Enum):
+    ...    first = 'one'
+    ...    second = 'two'
+    ...    third = 'two'
+    Traceback (most recent call last):
+    ...
+    ValueError: duplicate names found in <enum 'NoDupes'>: third -> second
+
+
+Interesting examples
+--------------------
+
+While ``Enum`` and ``IntEnum`` are expected to cover the majority of
+use-cases, they cannot cover them all.  Here are recipes for some different
+types of enumerations that can be used directly, or as examples for creating
+one's own.
+
+
+AutoNumber
+^^^^^^^^^^
+
+Avoids having to specify the value for each enumeration member::
+
+    >>> class AutoNumber(Enum):
+    ...     def __new__(cls):
+    ...         value = len(cls.__members__) + 1
+    ...         obj = object.__new__(cls)
+    ...         obj._value_ = value
+    ...         return obj
+    ...
+    >>> class Color(AutoNumber):
+    ...     __order__ = "red green blue"  # only needed in 2.x
+    ...     red = ()
+    ...     green = ()
+    ...     blue = ()
+    ...
+    >>> Color.green.value == 2
+    True
+
+Note:
+
+    The `__new__` method, if defined, is used during creation of the Enum
+    members; it is then replaced by Enum's `__new__` which is used after
+    class creation for lookup of existing members.  Due to the way Enums are
+    supposed to behave, there is no way to customize Enum's `__new__`.
+
+
+UniqueEnum
+^^^^^^^^^^
+
+Raises an error if a duplicate member name is found instead of creating an
+alias::
+
+    >>> class UniqueEnum(Enum):
+    ...     def __init__(self, *args):
+    ...         cls = self.__class__
+    ...         if any(self.value == e.value for e in cls):
+    ...             a = self.name
+    ...             e = cls(self.value).name
+    ...             raise ValueError(
+    ...                     "aliases not allowed in UniqueEnum:  %r --> %r"
+    ...                     % (a, e))
+    ... 
+    >>> class Color(UniqueEnum):
+    ...     red = 1
+    ...     green = 2
+    ...     blue = 3
+    ...     grene = 2
+    Traceback (most recent call last):
+    ...
+    ValueError: aliases not allowed in UniqueEnum:  'grene' --> 'green'
+    
+
+OrderedEnum
+^^^^^^^^^^^
+
+An ordered enumeration that is not based on ``IntEnum`` and so maintains
+the normal ``Enum`` invariants (such as not being comparable to other
+enumerations)::
+
+    >>> class OrderedEnum(Enum):
+    ...     def __ge__(self, other):
+    ...         if self.__class__ is other.__class__:
+    ...             return self._value_ >= other._value_
+    ...         return NotImplemented
+    ...     def __gt__(self, other):
+    ...         if self.__class__ is other.__class__:
+    ...             return self._value_ > other._value_
+    ...         return NotImplemented
+    ...     def __le__(self, other):
+    ...         if self.__class__ is other.__class__:
+    ...             return self._value_ <= other._value_
+    ...         return NotImplemented
+    ...     def __lt__(self, other):
+    ...         if self.__class__ is other.__class__:
+    ...             return self._value_ < other._value_
+    ...         return NotImplemented
+    ...
+    >>> class Grade(OrderedEnum):
+    ...     __ordered__ = 'A B C D F'
+    ...     A = 5
+    ...     B = 4
+    ...     C = 3
+    ...     D = 2
+    ...     F = 1
+    ...
+    >>> Grade.C < Grade.A
+    True
+
+
+Planet
+^^^^^^
+
+If ``__new__`` or ``__init__`` is defined the value of the enum member
+will be passed to those methods::
+
+    >>> class Planet(Enum):
+    ...     MERCURY = (3.303e+23, 2.4397e6)
+    ...     VENUS   = (4.869e+24, 6.0518e6)
+    ...     EARTH   = (5.976e+24, 6.37814e6)
+    ...     MARS    = (6.421e+23, 3.3972e6)
+    ...     JUPITER = (1.9e+27,   7.1492e7)
+    ...     SATURN  = (5.688e+26, 6.0268e7)
+    ...     URANUS  = (8.686e+25, 2.5559e7)
+    ...     NEPTUNE = (1.024e+26, 2.4746e7)
+    ...     def __init__(self, mass, radius):
+    ...         self.mass = mass       # in kilograms
+    ...         self.radius = radius   # in meters
+    ...     @property
+    ...     def surface_gravity(self):
+    ...         # universal gravitational constant  (m3 kg-1 s-2)
+    ...         G = 6.67300E-11
+    ...         return G * self.mass / (self.radius * self.radius)
+    ... 
+    >>> Planet.EARTH.value
+    (5.976e+24, 6378140.0)
+    >>> Planet.EARTH.surface_gravity
+    9.802652743337129
+
+
+How are Enums different?
+------------------------
+
+Enums have a custom metaclass that affects many aspects of both derived Enum
+classes and their instances (members).
+
+
+Enum Classes
+^^^^^^^^^^^^
+
+The ``EnumMeta`` metaclass is responsible for providing the
+``__contains__``, ``__dir__``, ``__iter__`` and other methods that
+allow one to do things with an ``Enum`` class that fail on a typical
+class, such as ``list(Color)`` or ``some_var in Color``.  ``EnumMeta`` is
+responsible for ensuring that various other methods on the final ``Enum``
+class are correct (such as ``__new__``, ``__getnewargs__``,
+``__str__`` and ``__repr__``)
+
+
+Enum Members (aka instances)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The most interesting thing about Enum members is that they are singletons.
+``EnumMeta`` creates them all while it is creating the ``Enum``
+class itself, and then puts a custom ``__new__`` in place to ensure
+that no new ones are ever instantiated by returning only the existing
+member instances.
+
+
+Finer Points
+^^^^^^^^^^^^
+
+Enum members are instances of an Enum class, and even though they are
+accessible as ``EnumClass.member``, they are not accessible directly from
+the member::
+
+    >>> Color.red
+    <Color.red: 1>
+    >>> Color.red.blue
+    Traceback (most recent call last):
+    ...
+    AttributeError: 'Color' object has no attribute 'blue'
+
+Likewise, ``__members__`` is only available on the class.
+
+In Python 3.x ``__members__`` is always an ``OrderedDict``, with the order being
+the definition order.  In Python 2.7 ``__members__`` is an ``OrderedDict`` if
+``__order__`` was specified, and a plain ``dict`` otherwise.  In all other Python
+2.x versions ``__members__`` is a plain ``dict`` even if ``__order__`` was specified
+as the ``OrderedDict`` type didn't exist yet.
+
+If you give your ``Enum`` subclass extra methods, like the `Planet`_
+class above, those methods will show up in a `dir` of the member,
+but not of the class::
+
+    >>> dir(Planet)
+    ['EARTH', 'JUPITER', 'MARS', 'MERCURY', 'NEPTUNE', 'SATURN', 'URANUS',
+    'VENUS', '__class__', '__doc__', '__members__', '__module__']
+    >>> dir(Planet.EARTH)
+    ['__class__', '__doc__', '__module__', 'name', 'surface_gravity', 'value']
+
+A ``__new__`` method will only be used for the creation of the
+``Enum`` members -- after that it is replaced.  This means if you wish to
+change how ``Enum`` members are looked up you either have to write a
+helper function or a ``classmethod``.

lib/enum/enum.py

@@ -0,0 +1,790 @@
+"""Python Enumerations"""
+
+import sys as _sys
+
+__all__ = ['Enum', 'IntEnum', 'unique']
+
+version = 1, 0, 4
+
+pyver = float('%s.%s' % _sys.version_info[:2])
+
+try:
+    any
+except NameError:
+    def any(iterable):
+        for element in iterable:
+            if element:
+                return True
+        return False
+
+try:
+    from collections import OrderedDict
+except ImportError:
+    OrderedDict = None
+
+try:
+    basestring
+except NameError:
+    # In Python 2 basestring is the ancestor of both str and unicode
+    # in Python 3 it's just str, but was missing in 3.1
+    basestring = str
+
+try:
+    unicode
+except NameError:
+    # In Python 3 unicode no longer exists (it's just str)
+    unicode = str
+
+class _RouteClassAttributeToGetattr(object):
+    """Route attribute access on a class to __getattr__.
+
+    This is a descriptor, used to define attributes that act differently when
+    accessed through an instance and through a class.  Instance access remains
+    normal, but access to an attribute through a class will be routed to the
+    class's __getattr__ method; this is done by raising AttributeError.
+
+    """
+    def __init__(self, fget=None):
+        self.fget = fget
+
+    def __get__(self, instance, ownerclass=None):
+        if instance is None:
+            raise AttributeError()
+        return self.fget(instance)
+
+    def __set__(self, instance, value):
+        raise AttributeError("can't set attribute")
+
+    def __delete__(self, instance):
+        raise AttributeError("can't delete attribute")
+
+
+def _is_descriptor(obj):
+    """Returns True if obj is a descriptor, False otherwise."""
+    return (
+            hasattr(obj, '__get__') or
+            hasattr(obj, '__set__') or
+            hasattr(obj, '__delete__'))
+
+
+def _is_dunder(name):
+    """Returns True if a __dunder__ name, False otherwise."""
+    return (name[:2] == name[-2:] == '__' and
+            name[2:3] != '_' and
+            name[-3:-2] != '_' and
+            len(name) > 4)
+
+
+def _is_sunder(name):
+    """Returns True if a _sunder_ name, False otherwise."""
+    return (name[0] == name[-1] == '_' and 
+            name[1:2] != '_' and
+            name[-2:-1] != '_' and
+            len(name) > 2)
+
+
+def _make_class_unpicklable(cls):
+    """Make the given class un-picklable."""
+    def _break_on_call_reduce(self, protocol=None):
+        raise TypeError('%r cannot be pickled' % self)
+    cls.__reduce_ex__ = _break_on_call_reduce
+    cls.__module__ = '<unknown>'
+
+
+class _EnumDict(dict):
+    """Track enum member order and ensure member names are not reused.
+
+    EnumMeta will use the names found in self._member_names as the
+    enumeration member names.
+
+    """
+    def __init__(self):
+        super(_EnumDict, self).__init__()
+        self._member_names = []
+
+    def __setitem__(self, key, value):
+        """Changes anything not dundered or not a descriptor.
+
+        If a descriptor is added with the same name as an enum member, the name
+        is removed from _member_names (this may leave a hole in the numerical
+        sequence of values).
+
+        If an enum member name is used twice, an error is raised; duplicate
+        values are not checked for.
+
+        Single underscore (sunder) names are reserved.
+
+        Note:   in 3.x __order__ is simply discarded as a not necessary piece
+                leftover from 2.x
+
+        """
+        if pyver >= 3.0 and key == '__order__':
+                return
+        if _is_sunder(key):
+            raise ValueError('_names_ are reserved for future Enum use')
+        elif _is_dunder(key):
+            pass
+        elif key in self._member_names:
+            # descriptor overwriting an enum?
+            raise TypeError('Attempted to reuse key: %r' % key)
+        elif not _is_descriptor(value):
+            if key in self:
+                # enum overwriting a descriptor?
+                raise TypeError('Key already defined as: %r' % self[key])
+            self._member_names.append(key)
+        super(_EnumDict, self).__setitem__(key, value)
+
+
+# Dummy value for Enum as EnumMeta explicity checks for it, but of course until
+# EnumMeta finishes running the first time the Enum class doesn't exist.  This
+# is also why there are checks in EnumMeta like `if Enum is not None`
+Enum = None
+
+
+class EnumMeta(type):
+    """Metaclass for Enum"""
+    @classmethod
+    def __prepare__(metacls, cls, bases):
+        return _EnumDict()
+
+    def __new__(metacls, cls, bases, classdict):
+        # an Enum class is final once enumeration items have been defined; it
+        # cannot be mixed with other types (int, float, etc.) if it has an
+        # inherited __new__ unless a new __new__ is defined (or the resulting
+        # class will fail).
+        if type(classdict) is dict:
+            original_dict = classdict
+            classdict = _EnumDict()
+            for k, v in original_dict.items():
+                classdict[k] = v
+
+        member_type, first_enum = metacls._get_mixins_(bases)
+        __new__, save_new, use_args = metacls._find_new_(classdict, member_type,
+                                                        first_enum)
+        # save enum items into separate mapping so they don't get baked into
+        # the new class
+        members = dict((k, classdict[k]) for k in classdict._member_names)
+        for name in classdict._member_names:
+            del classdict[name]
+
+        # py2 support for definition order
+        __order__ = classdict.get('__order__')
+        if __order__ is None:
+            if pyver < 3.0:
+                try:
+                    __order__ = [name for (name, value) in sorted(members.items(), key=lambda item: item[1])]
+                except TypeError:
+                    __order__ = [name for name in sorted(members.keys())]
+            else:
+                __order__ = classdict._member_names
+        else:
+            del classdict['__order__']
+            if pyver < 3.0:
+                __order__ = __order__.replace(',', ' ').split()
+                aliases = [name for name in members if name not in __order__]
+                __order__ += aliases
+
+        # check for illegal enum names (any others?)
+        invalid_names = set(members) & set(['mro'])
+        if invalid_names:
+            raise ValueError('Invalid enum member name(s): %s' % (
+                ', '.join(invalid_names), ))
+
+        # create our new Enum type
+        enum_class = super(EnumMeta, metacls).__new__(metacls, cls, bases, classdict)
+        enum_class._member_names_ = []               # names in random order
+        if OrderedDict is not None:
+            enum_class._member_map_ = OrderedDict()
+        else:
+            enum_class._member_map_ = {}             # name->value map
+        enum_class._member_type_ = member_type
+
+        # Reverse value->name map for hashable values.
+        enum_class._value2member_map_ = {}
+
+        # instantiate them, checking for duplicates as we go
+        # we instantiate first instead of checking for duplicates first in case
+        # a custom __new__ is doing something funky with the values -- such as
+        # auto-numbering ;)
+        if __new__ is None:
+            __new__ = enum_class.__new__
+        for member_name in __order__:
+            value = members[member_name]
+            if not isinstance(value, tuple):
+                args = (value, )
+            else:
+                args = value
+            if member_type is tuple:   # special case for tuple enums
+                args = (args, )     # wrap it one more time
+            if not use_args or not args:
+                enum_member = __new__(enum_class)
+                if not hasattr(enum_member, '_value_'):
+                    enum_member._value_ = value
+            else:
+                enum_member = __new__(enum_class, *args)
+                if not hasattr(enum_member, '_value_'):
+                    enum_member._value_ = member_type(*args)
+            value = enum_member._value_
+            enum_member._name_ = member_name
+            enum_member.__objclass__ = enum_class
+            enum_member.__init__(*args)
+            # If another member with the same value was already defined, the
+            # new member becomes an alias to the existing one.
+            for name, canonical_member in enum_class._member_map_.items():
+                if canonical_member.value == enum_member._value_:
+                    enum_member = canonical_member
+                    break
+            else:
+                # Aliases don't appear in member names (only in __members__).
+                enum_class._member_names_.append(member_name)
+            enum_class._member_map_[member_name] = enum_member
+            try:
+                # This may fail if value is not hashable. We can't add the value
+                # to the map, and by-value lookups for this value will be
+                # linear.
+                enum_class._value2member_map_[value] = enum_member
+            except TypeError:
+                pass
+
+
+        # If a custom type is mixed into the Enum, and it does not know how
+        # to pickle itself, pickle.dumps will succeed but pickle.loads will
+        # fail.  Rather than have the error show up later and possibly far
+        # from the source, sabotage the pickle protocol for this class so
+        # that pickle.dumps also fails.
+        #
+        # However, if the new class implements its own __reduce_ex__, do not
+        # sabotage -- it's on them to make sure it works correctly.  We use
+        # __reduce_ex__ instead of any of the others as it is preferred by
+        # pickle over __reduce__, and it handles all pickle protocols.
+        unpicklable = False
+        if '__reduce_ex__' not in classdict:
+            if member_type is not object:
+                methods = ('__getnewargs_ex__', '__getnewargs__',
+                        '__reduce_ex__', '__reduce__')
+                if not any(m in member_type.__dict__ for m in methods):
+                    _make_class_unpicklable(enum_class)
+                    unpicklable = True
+
+
+        # double check that repr and friends are not the mixin's or various
+        # things break (such as pickle)
+        for name in ('__repr__', '__str__', '__format__', '__reduce_ex__'):
+            class_method = getattr(enum_class, name)
+            obj_method = getattr(member_type, name, None)
+            enum_method = getattr(first_enum, name, None)
+            if name not in classdict and class_method is not enum_method:
+                if name == '__reduce_ex__' and unpicklable:
+                    continue
+                setattr(enum_class, name, enum_method)
+
+        # method resolution and int's are not playing nice
+        # Python's less than 2.6 use __cmp__
+
+        if pyver < 2.6:
+
+            if issubclass(enum_class, int):
+                setattr(enum_class, '__cmp__', getattr(int, '__cmp__'))
+
+        elif pyver < 3.0:
+
+            if issubclass(enum_class, int):
+                for method in (
+                        '__le__',
+                        '__lt__',
+                        '__gt__',
+                        '__ge__',
+                        '__eq__',
+                        '__ne__',
+                        '__hash__',
+                        ):
+                    setattr(enum_class, method, getattr(int, method))
+
+        # replace any other __new__ with our own (as long as Enum is not None,
+        # anyway) -- again, this is to support pickle
+        if Enum is not None:
+            # if the user defined their own __new__, save it before it gets
+            # clobbered in case they subclass later
+            if save_new:
+                setattr(enum_class, '__member_new__', enum_class.__dict__['__new__'])
+            setattr(enum_class, '__new__', Enum.__dict__['__new__'])
+        return enum_class
+
+    def __call__(cls, value, names=None, module=None, type=None):
+        """Either returns an existing member, or creates a new enum class.
+
+        This method is used both when an enum class is given a value to match
+        to an enumeration member (i.e. Color(3)) and for the functional API
+        (i.e. Color = Enum('Color', names='red green blue')).
+
+        When used for the functional API: `module`, if set, will be stored in
+        the new class' __module__ attribute; `type`, if set, will be mixed in
+        as the first base class.
+
+        Note: if `module` is not set this routine will attempt to discover the
+        calling module by walking the frame stack; if this is unsuccessful
+        the resulting class will not be pickleable.
+
+        """
+        if names is None:  # simple value lookup
+            return cls.__new__(cls, value)
+        # otherwise, functional API: we're creating a new Enum type
+        return cls._create_(value, names, module=module, type=type)
+
+    def __contains__(cls, member):
+        return isinstance(member, cls) and member.name in cls._member_map_
+
+    def __delattr__(cls, attr):
+        # nicer error message when someone tries to delete an attribute
+        # (see issue19025).
+        if attr in cls._member_map_:
+            raise AttributeError(
+                    "%s: cannot delete Enum member." % cls.__name__)
+        super(EnumMeta, cls).__delattr__(attr)
+
+    def __dir__(self):
+        return (['__class__', '__doc__', '__members__', '__module__'] +
+                self._member_names_)
+
+    @property
+    def __members__(cls):
+        """Returns a mapping of member name->value.
+
+        This mapping lists all enum members, including aliases. Note that this
+        is a copy of the internal mapping.
+
+        """
+        return cls._member_map_.copy()
+
+    def __getattr__(cls, name):
+        """Return the enum member matching `name`
+
+        We use __getattr__ instead of descriptors or inserting into the enum
+        class' __dict__ in order to support `name` and `value` being both
+        properties for enum members (which live in the class' __dict__) and
+        enum members themselves.
+
+        """
+        if _is_dunder(name):
+            raise AttributeError(name)
+        try:
+            return cls._member_map_[name]
+        except KeyError:
+            raise AttributeError(name)
+
+    def __getitem__(cls, name):
+        return cls._member_map_[name]
+
+    def __iter__(cls):
+        return (cls._member_map_[name] for name in cls._member_names_)
+
+    def __reversed__(cls):
+        return (cls._member_map_[name] for name in reversed(cls._member_names_))
+
+    def __len__(cls):
+        return len(cls._member_names_)
+
+    def __repr__(cls):
+        return "<enum %r>" % cls.__name__
+
+    def __setattr__(cls, name, value):
+        """Block attempts to reassign Enum members.
+
+        A simple assignment to the class namespace only changes one of the
+        several possible ways to get an Enum member from the Enum class,
+        resulting in an inconsistent Enumeration.
+
+        """
+        member_map = cls.__dict__.get('_member_map_', {})
+        if name in member_map:
+            raise AttributeError('Cannot reassign members.')
+        super(EnumMeta, cls).__setattr__(name, value)
+
+    def _create_(cls, class_name, names=None, module=None, type=None):
+        """Convenience method to create a new Enum class.
+
+        `names` can be:
+
+        * A string containing member names, separated either with spaces or
+          commas.  Values are auto-numbered from 1.
+        * An iterable of member names.  Values are auto-numbered from 1.
+        * An iterable of (member name, value) pairs.
+        * A mapping of member name -> value.
+
+        """
+        if pyver < 3.0:
+            # if class_name is unicode, attempt a conversion to ASCII
+            if isinstance(class_name, unicode):
+                try:
+                    class_name = class_name.encode('ascii')
+                except UnicodeEncodeError:
+                    raise TypeError('%r is not representable in ASCII' % class_name)
+        metacls = cls.__class__
+        if type is None:
+            bases = (cls, )
+        else:
+            bases = (type, cls)
+        classdict = metacls.__prepare__(class_name, bases)
+        __order__ = []
+
+        # special processing needed for names?
+        if isinstance(names, basestring):
+            names = names.replace(',', ' ').split()
+        if isinstance(names, (tuple, list)) and isinstance(names[0], basestring):
+            names = [(e, i+1) for (i, e) in enumerate(names)]
+
+        # Here, names is either an iterable of (name, value) or a mapping.
+        for item in names:
+            if isinstance(item, basestring):
+                member_name, member_value = item, names[item]
+            else:
+                member_name, member_value = item
+            classdict[member_name] = member_value
+            __order__.append(member_name)
+        # only set __order__ in classdict if name/value was not from a mapping
+        if not isinstance(item, basestring):
+            classdict['__order__'] = ' '.join(__order__)
+        enum_class = metacls.__new__(metacls, class_name, bases, classdict)
+
+        # TODO: replace the frame hack if a blessed way to know the calling
+        # module is ever developed
+        if module is None:
+            try:
+                module = _sys._getframe(2).f_globals['__name__']
+            except (AttributeError, ValueError):
+                pass
+        if module is None:
+            _make_class_unpicklable(enum_class)
+        else:
+            enum_class.__module__ = module
+
+        return enum_class
+
+    @staticmethod
+    def _get_mixins_(bases):
+        """Returns the type for creating enum members, and the first inherited
+        enum class.
+
+        bases: the tuple of bases that was given to __new__
+
+        """
+        if not bases or Enum is None:
+            return object, Enum
+        
+
+        # double check that we are not subclassing a class with existing
+        # enumeration members; while we're at it, see if any other data
+        # type has been mixed in so we can use the correct __new__
+        member_type = first_enum = None
+        for base in bases:
+            if  (base is not Enum and
+                    issubclass(base, Enum) and
+                    base._member_names_):
+                raise TypeError("Cannot extend enumerations")
+        # base is now the last base in bases
+        if not issubclass(base, Enum):
+            raise TypeError("new enumerations must be created as "
+                    "`ClassName([mixin_type,] enum_type)`")
+
+        # get correct mix-in type (either mix-in type of Enum subclass, or
+        # first base if last base is Enum)
+        if not issubclass(bases[0], Enum):
+            member_type = bases[0]     # first data type
+            first_enum = bases[-1]  # enum type
+        else:
+            for base in bases[0].__mro__:
+                # most common: (IntEnum, int, Enum, object)
+                # possible:    (<Enum 'AutoIntEnum'>, <Enum 'IntEnum'>,
+                #               <class 'int'>, <Enum 'Enum'>,
+                #               <class 'object'>)
+                if issubclass(base, Enum):
+                    if first_enum is None:
+                        first_enum = base
+                else:
+                    if member_type is None:
+                        member_type = base
+
+        return member_type, first_enum
+
+    if pyver < 3.0:
+        @staticmethod
+        def _find_new_(classdict, member_type, first_enum):
+            """Returns the __new__ to be used for creating the enum members.
+
+            classdict: the class dictionary given to __new__
+            member_type: the data type whose __new__ will be used by default
+            first_enum: enumeration to check for an overriding __new__
+
+            """
+            # now find the correct __new__, checking to see of one was defined
+            # by the user; also check earlier enum classes in case a __new__ was
+            # saved as __member_new__
+            __new__ = classdict.get('__new__', None)
+            if __new__:
+                return None, True, True      # __new__, save_new, use_args
+
+            N__new__ = getattr(None, '__new__')
+            O__new__ = getattr(object, '__new__')
+            if Enum is None:
+                E__new__ = N__new__
+            else:
+                E__new__ = Enum.__dict__['__new__']
+            # check all possibles for __member_new__ before falling back to
+            # __new__
+            for method in ('__member_new__', '__new__'):
+                for possible in (member_type, first_enum):
+                    try:
+                        target = possible.__dict__[method]
+                    except (AttributeError, KeyError):
+                        target = getattr(possible, method, None)
+                    if target not in [
+                            None,
+                            N__new__,
+                            O__new__,
+                            E__new__,
+                            ]:
+                        if method == '__member_new__':
+                            classdict['__new__'] = target
+                            return None, False, True
+                        if isinstance(target, staticmethod):
+                            target = target.__get__(member_type)
+                        __new__ = target
+                        break
+                if __new__ is not None:
+                    break
+            else:
+                __new__ = object.__new__
+
+            # if a non-object.__new__ is used then whatever value/tuple was
+            # assigned to the enum member name will be passed to __new__ and to the
+            # new enum member's __init__
+            if __new__ is object.__new__:
+                use_args = False
+            else:
+                use_args = True
+
+            return __new__, False, use_args
+    else:
+        @staticmethod
+        def _find_new_(classdict, member_type, first_enum):
+            """Returns the __new__ to be used for creating the enum members.
+
+            classdict: the class dictionary given to __new__
+            member_type: the data type whose __new__ will be used by default
+            first_enum: enumeration to check for an overriding __new__
+
+            """
+            # now find the correct __new__, checking to see of one was defined
+            # by the user; also check earlier enum classes in case a __new__ was
+            # saved as __member_new__
+            __new__ = classdict.get('__new__', None)
+
+            # should __new__ be saved as __member_new__ later?
+            save_new = __new__ is not None
+
+            if __new__ is None:
+                # check all possibles for __member_new__ before falling back to
+                # __new__
+                for method in ('__member_new__', '__new__'):
+                    for possible in (member_type, first_enum):
+                        target = getattr(possible, method, None)
+                        if target not in (
+                                None,
+                                None.__new__,
+                                object.__new__,
+                                Enum.__new__,
+                                ):
+                            __new__ = target
+                            break
+                    if __new__ is not None:
+                        break
+                else:
+                    __new__ = object.__new__
+
+            # if a non-object.__new__ is used then whatever value/tuple was
+            # assigned to the enum member name will be passed to __new__ and to the
+            # new enum member's __init__
+            if __new__ is object.__new__:
+                use_args = False
+            else:
+                use_args = True
+
+            return __new__, save_new, use_args
+
+
+########################################################
+# In order to support Python 2 and 3 with a single
+# codebase we have to create the Enum methods separately
+# and then use the `type(name, bases, dict)` method to
+# create the class.
+########################################################
+temp_enum_dict = {}
+temp_enum_dict['__doc__'] = "Generic enumeration.\n\n    Derive from this class to define new enumerations.\n\n"
+
+def __new__(cls, value):
+    # all enum instances are actually created during class construction
+    # without calling this method; this method is called by the metaclass'
+    # __call__ (i.e. Color(3) ), and by pickle
+    if type(value) is cls:
+        # For lookups like Color(Color.red)
+        value = value.value
+        #return value
+    # by-value search for a matching enum member
+    # see if it's in the reverse mapping (for hashable values)
+    try:
+        if value in cls._value2member_map_:
+            return cls._value2member_map_[value]
+    except TypeError:
+        # not there, now do long search -- O(n) behavior
+        for member in cls._member_map_.values():
+            if member.value == value:
+                return member
+    raise ValueError("%s is not a valid %s" % (value, cls.__name__))
+temp_enum_dict['__new__'] = __new__
+del __new__
+
+def __repr__(self):
+    return "<%s.%s: %r>" % (
+            self.__class__.__name__, self._name_, self._value_)
+temp_enum_dict['__repr__'] = __repr__
+del __repr__
+
+def __str__(self):
+    return "%s.%s" % (self.__class__.__name__, self._name_)
+temp_enum_dict['__str__'] = __str__
+del __str__
+
+def __dir__(self):
+    added_behavior = [
+            m
+            for cls in self.__class__.mro()
+            for m in cls.__dict__
+            if m[0] != '_'
+            ]
+    return (['__class__', '__doc__', '__module__', ] + added_behavior)
+temp_enum_dict['__dir__'] = __dir__
+del __dir__
+
+def __format__(self, format_spec):
+    # mixed-in Enums should use the mixed-in type's __format__, otherwise
+    # we can get strange results with the Enum name showing up instead of
+    # the value
+
+    # pure Enum branch
+    if self._member_type_ is object:
+        cls = str
+        val = str(self)
+    # mix-in branch
+    else:
+        cls = self._member_type_
+        val = self.value
+    return cls.__format__(val, format_spec)
+temp_enum_dict['__format__'] = __format__
+del __format__
+
+
+####################################
+# Python's less than 2.6 use __cmp__
+
+if pyver < 2.6:
+
+    def __cmp__(self, other):
+        if type(other) is self.__class__:
+            if self is other:
+                return 0
+            return -1
+        return NotImplemented
+        raise TypeError("unorderable types: %s() and %s()" % (self.__class__.__name__, other.__class__.__name__))
+    temp_enum_dict['__cmp__'] = __cmp__
+    del __cmp__
+
+else:
+
+    def __le__(self, other):
+        raise TypeError("unorderable types: %s() <= %s()" % (self.__class__.__name__, other.__class__.__name__))
+    temp_enum_dict['__le__'] = __le__
+    del __le__
+
+    def __lt__(self, other):
+        raise TypeError("unorderable types: %s() < %s()" % (self.__class__.__name__, other.__class__.__name__))
+    temp_enum_dict['__lt__'] = __lt__
+    del __lt__
+
+    def __ge__(self, other):
+        raise TypeError("unorderable types: %s() >= %s()" % (self.__class__.__name__, other.__class__.__name__))
+    temp_enum_dict['__ge__'] = __ge__
+    del __ge__
+
+    def __gt__(self, other):
+        raise TypeError("unorderable types: %s() > %s()" % (self.__class__.__name__, other.__class__.__name__))
+    temp_enum_dict['__gt__'] = __gt__
+    del __gt__
+    
+
+def __eq__(self, other):
+    if type(other) is self.__class__:
+        return self is other
+    return NotImplemented
+temp_enum_dict['__eq__'] = __eq__
+del __eq__
+
+def __ne__(self, other):
+    if type(other) is self.__class__:
+        return self is not other
+    return NotImplemented
+temp_enum_dict['__ne__'] = __ne__
+del __ne__
+
+def __hash__(self):
+    return hash(self._name_)
+temp_enum_dict['__hash__'] = __hash__
+del __hash__
+
+def __reduce_ex__(self, proto):
+    return self.__class__, (self._value_, )
+temp_enum_dict['__reduce_ex__'] = __reduce_ex__
+del __reduce_ex__
+
+# _RouteClassAttributeToGetattr is used to provide access to the `name`
+# and `value` properties of enum members while keeping some measure of
+# protection from modification, while still allowing for an enumeration
+# to have members named `name` and `value`.  This works because enumeration
+# members are not set directly on the enum class -- __getattr__ is
+# used to look them up.
+
+@_RouteClassAttributeToGetattr
+def name(self):
+    return self._name_
+temp_enum_dict['name'] = name
+del name
+
+@_RouteClassAttributeToGetattr
+def value(self):
+    return self._value_
+temp_enum_dict['value'] = value
+del value
+
+Enum = EnumMeta('Enum', (object, ), temp_enum_dict)
+del temp_enum_dict
+
+# Enum has now been created
+###########################
+
+class IntEnum(int, Enum):
+    """Enum where members are also (and must be) ints"""
+
+
+def unique(enumeration):
+    """Class decorator that ensures only unique members exist in an enumeration."""
+    duplicates = []
+    for name, member in enumeration.__members__.items():
+        if name != member.name:
+            duplicates.append((name, member.name))
+    if duplicates:
+        duplicate_names = ', '.join(
+                ["%s -> %s" % (alias, name) for (alias, name) in duplicates]
+                )
+        raise ValueError('duplicate names found in %r: %s' %
+                (enumeration, duplicate_names)
+                )
+    return enumeration

lib/mako/LICENSE

@@ -0,0 +1,20 @@
+This is the MIT license: http://www.opensource.org/licenses/mit-license.php
+
+Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>.
+Mako is a trademark of Michael Bayer.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this
+software and associated documentation files (the "Software"), to deal in the Software
+without restriction, including without limitation the rights to use, copy, modify, merge,
+publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons
+to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or
+substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
+INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE
+FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.

lib/mako/README.rst

@@ -0,0 +1,52 @@
+=========================
+Mako Templates for Python
+=========================
+
+Mako is a template library written in Python. It provides a familiar, non-XML 
+syntax which compiles into Python modules for maximum performance. Mako's 
+syntax and API borrows from the best ideas of many others, including Django
+templates, Cheetah, Myghty, and Genshi. Conceptually, Mako is an embedded 
+Python (i.e. Python Server Page) language, which refines the familiar ideas
+of componentized layout and inheritance to produce one of the most 
+straightforward and flexible models available, while also maintaining close 
+ties to Python calling and scoping semantics.
+
+Nutshell
+========
+
+::
+
+    <%inherit file="base.html"/>
+    <%
+        rows = [[v for v in range(0,10)] for row in range(0,10)]
+    %>
+    <table>
+        % for row in rows:
+            ${makerow(row)}
+        % endfor
+    </table>
+
+    <%def name="makerow(row)">
+        <tr>
+        % for name in row:
+            <td>${name}</td>\
+        % endfor
+        </tr>
+    </%def>
+
+Philosophy
+===========
+
+Python is a great scripting language. Don't reinvent the wheel...your templates can handle it !
+
+Documentation
+==============
+
+See documentation for Mako at http://www.makotemplates.org/docs/
+
+License
+========
+
+Mako is licensed under an MIT-style license (see LICENSE).
+Other incorporated projects may be licensed under different licenses.
+All licenses allow for non-commercial and commercial use.

lib/mako/__init__.py

@@ -1,9 +1,8 @@
 # mako/__init__.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
 
 
-__version__ = '0.9.1'
-
+__version__ = '1.0.1'

lib/mako/_ast_util.py

@@ -1,5 +1,5 @@
 # mako/_ast_util.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
@@ -34,42 +34,42 @@ from _ast import *
 from mako.compat import arg_stringname
 
 BOOLOP_SYMBOLS = {
-    And:        'and',
-    Or:         'or'
+    And: 'and',
+    Or: 'or'
 }
 
 BINOP_SYMBOLS = {
-    Add:        '+',
-    Sub:        '-',
-    Mult:       '*',
-    Div:        '/',
-    FloorDiv:   '//',
-    Mod:        '%',
-    LShift:     '<<',
-    RShift:     '>>',
-    BitOr:      '|',
-    BitAnd:     '&',
-    BitXor:     '^'
+    Add: '+',
+    Sub: '-',
+    Mult: '*',
+    Div: '/',
+    FloorDiv: '//',
+    Mod: '%',
+    LShift: '<<',
+    RShift: '>>',
+    BitOr: '|',
+    BitAnd: '&',
+    BitXor: '^'
 }
 
 CMPOP_SYMBOLS = {
-    Eq:         '==',
-    Gt:         '>',
-    GtE:        '>=',
-    In:         'in',
-    Is:         'is',
-    IsNot:      'is not',
-    Lt:         '<',
-    LtE:        '<=',
-    NotEq:      '!=',
-    NotIn:      'not in'
+    Eq: '==',
+    Gt: '>',
+    GtE: '>=',
+    In: 'in',
+    Is: 'is',
+    IsNot: 'is not',
+    Lt: '<',
+    LtE: '<=',
+    NotEq: '!=',
+    NotIn: 'not in'
 }
 
 UNARYOP_SYMBOLS = {
-    Invert:     '~',
-    Not:        'not',
-    UAdd:       '+',
-    USub:       '-'
+    Invert: '~',
+    Not: 'not',
+    UAdd: '+',
+    USub: '-'
 }
 
 ALL_SYMBOLS = {}
@@ -215,8 +215,8 @@ def get_compile_mode(node):
     if not isinstance(node, mod):
         raise TypeError('expected mod node, got %r' % node.__class__.__name__)
     return {
-        Expression:     'eval',
-        Interactive:    'single'
+        Expression: 'eval',
+        Interactive: 'single'
     }.get(node.__class__, 'expr')
 
 

lib/mako/ast.py

@@ -1,5 +1,5 @@
 # mako/ast.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
@@ -8,7 +8,6 @@
 code, as well as generating Python from AST nodes"""
 
 from mako import exceptions, pyparser, compat
-from mako.compat import arg_stringname
 import re
 
 class PythonCode(object):
@@ -107,43 +106,70 @@ class FunctionDecl(object):
         f.visit(expr)
         if not hasattr(self, 'funcname'):
             raise exceptions.CompileException(
-                              "Code '%s' is not a function declaration" % code,
-                              **exception_kwargs)
+                            "Code '%s' is not a function declaration" % code,
+                            **exception_kwargs)
         if not allow_kwargs and self.kwargs:
             raise exceptions.CompileException(
                                 "'**%s' keyword argument not allowed here" %
-                                self.argnames[-1], **exception_kwargs)
+                                self.kwargnames[-1], **exception_kwargs)
 
-    def get_argument_expressions(self, include_defaults=True):
-        """return the argument declarations of this FunctionDecl as a printable
-        list."""
+    def get_argument_expressions(self, as_call=False):
+        """Return the argument declarations of this FunctionDecl as a printable
+        list.
+
+        By default the return value is appropriate for writing in a ``def``;
+        set `as_call` to true to build arguments to be passed to the function
+        instead (assuming locals with the same names as the arguments exist).
+        """
 
         namedecls = []
-        defaults = [d for d in self.defaults]
-        kwargs = self.kwargs
-        varargs = self.varargs
-        argnames = [f for f in self.argnames]
-        argnames.reverse()
-        for arg in argnames:
-            default = None
-            if kwargs:
-                arg = "**" + arg_stringname(arg)
-                kwargs = False
-            elif varargs:
-                arg = "*" + arg_stringname(arg)
-                varargs = False
+
+        # Build in reverse order, since defaults and slurpy args come last
+        argnames = self.argnames[::-1]
+        kwargnames = self.kwargnames[::-1]
+        defaults = self.defaults[::-1]
+        kwdefaults = self.kwdefaults[::-1]
+
+        # Named arguments
+        if self.kwargs:
+            namedecls.append("**" + kwargnames.pop(0))
+
+        for name in kwargnames:
+            # Keyword-only arguments must always be used by name, so even if
+            # this is a call, print out `foo=foo`
+            if as_call:
+                namedecls.append("%s=%s" % (name, name))
+            elif kwdefaults:
+                default = kwdefaults.pop(0)
+                if default is None:
+                    # The AST always gives kwargs a default, since you can do
+                    # `def foo(*, a=1, b, c=3)`
+                    namedecls.append(name)
+                else:
+                    namedecls.append("%s=%s" % (
+                        name, pyparser.ExpressionGenerator(default).value()))
             else:
-                default = len(defaults) and defaults.pop() or None
-            if include_defaults and default:
-                namedecls.insert(0, "%s=%s" %
-                            (arg,
-                            pyparser.ExpressionGenerator(default).value()
-                            )
-                        )
+                namedecls.append(name)
+
+        # Positional arguments
+        if self.varargs:
+            namedecls.append("*" + argnames.pop(0))
+
+        for name in argnames:
+            if as_call or not defaults:
+                namedecls.append(name)
             else:
-                namedecls.insert(0, arg)
+                default = defaults.pop(0)
+                namedecls.append("%s=%s" % (
+                    name, pyparser.ExpressionGenerator(default).value()))
+
+        namedecls.reverse()
         return namedecls
 
+    @property
+    def allargnames(self):
+        return tuple(self.argnames) + tuple(self.kwargnames)
+
 class FunctionArgs(FunctionDecl):
     """the argument portion of a function declaration"""
 

lib/mako/cache.py

@@ -1,5 +1,5 @@
 # mako/cache.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
@@ -88,9 +88,10 @@ class Cache(object):
         if not self.template.cache_enabled:
             return creation_function()
 
-        return self.impl.get_or_create(key,
-                        creation_function,
-                        **self._get_cache_kw(kw, context))
+        return self.impl.get_or_create(
+            key,
+            creation_function,
+            **self._get_cache_kw(kw, context))
 
     def set(self, key, value, **kw):
         """Place a value in the cache.
@@ -178,6 +179,7 @@ class Cache(object):
             tmpl_kw.setdefault('context', context)
         return tmpl_kw
 
+
 class CacheImpl(object):
     """Provide a cache implementation for use by :class:`.Cache`."""
 

lib/mako/cmd.py

@@ -0,0 +1,62 @@
+# mako/cmd.py
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
+#
+# This module is part of Mako and is released under
+# the MIT License: http://www.opensource.org/licenses/mit-license.php
+from argparse import ArgumentParser
+from os.path import isfile, dirname
+import sys
+from mako.template import Template
+from mako.lookup import TemplateLookup
+from mako import exceptions
+
+def varsplit(var):
+    if "=" not in var:
+        return (var, "")
+    return var.split("=", 1)
+
+def _exit():
+    sys.stderr.write(exceptions.text_error_template().render())
+    sys.exit(1)
+
+def cmdline(argv=None):
+
+    parser = ArgumentParser("usage: %prog [FILENAME]")
+    parser.add_argument("--var", default=[], action="append",
+                  help="variable (can be used multiple times, use name=value)")
+    parser.add_argument("--template-dir", default=[], action="append",
+                  help="Directory to use for template lookup (multiple "
+                    "directories may be provided). If not given then if the "
+                    "template is read from stdin, the value defaults to be "
+                    "the current directory, otherwise it defaults to be the "
+                    "parent directory of the file provided.")
+    parser.add_argument('input', nargs='?', default='-')
+
+    options = parser.parse_args(argv)
+    if options.input == '-':
+        lookup_dirs = options.template_dir or ["."]
+        lookup = TemplateLookup(lookup_dirs)
+        try:
+            template = Template(sys.stdin.read(), lookup=lookup)
+        except:
+            _exit()
+    else:
+        filename = options.input
+        if not isfile(filename):
+            raise SystemExit("error: can't find %s" % filename)
+        lookup_dirs = options.template_dir or [dirname(filename)]
+        lookup = TemplateLookup(lookup_dirs)
+        try:
+            template = Template(filename=filename, lookup=lookup)
+        except:
+            _exit()
+
+    kw = dict([varsplit(var) for var in options.var])
+    try:
+        print(template.render(**kw))
+    except:
+        _exit()
+
+
+if __name__ == "__main__":
+    cmdline()

lib/mako/codegen.py

@@ -1,5 +1,5 @@
 # mako/codegen.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
@@ -14,7 +14,7 @@ from mako import util, ast, parsetree, filters, exceptions
 from mako import compat
 
 
-MAGIC_NUMBER = 9
+MAGIC_NUMBER = 10
 
 # names which are hardwired into the
 # template and are not accessed via the
@@ -99,7 +99,6 @@ class _GenerateRenderMethod(object):
     """
     def __init__(self, printer, compiler, node):
         self.printer = printer
-        self.last_source_line = -1
         self.compiler = compiler
         self.node = node
         self.identifier_stack = [None]
@@ -146,6 +145,26 @@ class _GenerateRenderMethod(object):
             for node in defs:
                 _GenerateRenderMethod(printer, compiler, node)
 
+        if not self.in_def:
+            self.write_metadata_struct()
+
+    def write_metadata_struct(self):
+        self.printer.source_map[self.printer.lineno] = \
+                    max(self.printer.source_map)
+        struct = {
+            "filename": self.compiler.filename,
+            "uri": self.compiler.uri,
+            "source_encoding": self.compiler.source_encoding,
+            "line_map": self.printer.source_map,
+        }
+        self.printer.writelines(
+            '"""',
+            '__M_BEGIN_METADATA',
+            compat.json.dumps(struct),
+            '__M_END_METADATA\n'
+            '"""'
+        )
+
     @property
     def identifiers(self):
         return self.identifier_stack[-1]
@@ -187,7 +206,7 @@ class _GenerateRenderMethod(object):
 
         # module-level names, python code
         if self.compiler.generate_magic_comment and \
-            self.compiler.source_encoding:
+                self.compiler.source_encoding:
             self.printer.writeline("# -*- coding:%s -*-" %
                                     self.compiler.source_encoding)
 
@@ -232,7 +251,7 @@ class _GenerateRenderMethod(object):
                             [n.name for n in
                             main_identifiers.topleveldefs.values()]
                         )
-        self.printer.write("\n\n")
+        self.printer.write_blanks(2)
 
         if len(module_code):
             self.write_module_code(module_code)
@@ -255,8 +274,9 @@ class _GenerateRenderMethod(object):
             decorator = node.decorator
             if decorator:
                 self.printer.writeline(
-                                 "@runtime._decorate_toplevel(%s)" % decorator)
+                                "@runtime._decorate_toplevel(%s)" % decorator)
 
+        self.printer.start_source(node.lineno)
         self.printer.writelines(
             "def %s(%s):" % (name, ','.join(args)),
                 # push new frame, assign current frame to __M_caller
@@ -267,7 +287,7 @@ class _GenerateRenderMethod(object):
             self.printer.writeline("context._push_buffer()")
 
         self.identifier_stack.append(
-                                   self.compiler.identifiers.branch(self.node))
+                                self.compiler.identifiers.branch(self.node))
         if (not self.in_def or self.node.is_block) and '**pageargs' in args:
             self.identifier_stack[-1].argument_declared.add('pageargs')
 
@@ -288,7 +308,7 @@ class _GenerateRenderMethod(object):
 
         self.write_def_finish(self.node, buffered, filtered, cached)
         self.printer.writeline(None)
-        self.printer.write("\n\n")
+        self.printer.write_blanks(2)
         if cached:
             self.write_cache_decorator(
                                 node, name,
@@ -299,7 +319,7 @@ class _GenerateRenderMethod(object):
         """write module-level template code, i.e. that which
         is enclosed in <%! %> tags in the template."""
         for n in module_code:
-            self.write_source_comment(n)
+            self.printer.start_source(n.lineno)
             self.printer.write_indented_block(n.text)
 
     def write_inherit(self, node):
@@ -309,9 +329,9 @@ class _GenerateRenderMethod(object):
             "def _mako_inherit(template, context):",
                 "_mako_generate_namespaces(context)",
                 "return runtime._inherit_from(context, %s, _template_uri)" %
-                 (node.parsed_attributes['file']),
+                (node.parsed_attributes['file']),
                 None
-            )
+        )
 
     def write_namespaces(self, namespaces):
         """write the module-level namespace-generating callable."""
@@ -323,14 +343,14 @@ class _GenerateRenderMethod(object):
                     "_mako_generate_namespaces(context)",
                 "return context.namespaces[(__name__, name)]",
             None, None
-            )
+        )
         self.printer.writeline("def _mako_generate_namespaces(context):")
 
 
         for node in namespaces.values():
             if 'import' in node.attributes:
                 self.compiler.has_ns_imports = True
-            self.write_source_comment(node)
+            self.printer.start_source(node.lineno)
             if len(node.nodes):
                 self.printer.writeline("def make_namespace():")
                 export = []
@@ -369,9 +389,9 @@ class _GenerateRenderMethod(object):
                                 " templateuri=%s, callables=%s, "
                                 " calling_uri=_template_uri)" %
                                 (
-                                   node.name,
-                                   node.parsed_attributes.get('file', 'None'),
-                                   callable_name,
+                                    node.name,
+                                    node.parsed_attributes.get('file', 'None'),
+                                    callable_name,
                                 )
                             )
             elif 'module' in node.parsed_attributes:
@@ -381,9 +401,10 @@ class _GenerateRenderMethod(object):
                                 " callables=%s, calling_uri=_template_uri,"
                                 " module=%s)" %
                                 (
-                                   node.name,
-                                   callable_name,
-                                   node.parsed_attributes.get('module', 'None')
+                                    node.name,
+                                    callable_name,
+                                    node.parsed_attributes.get(
+                                                'module', 'None')
                                 )
                             )
             else:
@@ -400,8 +421,8 @@ class _GenerateRenderMethod(object):
                 self.printer.writeline("context['self'].%s = ns" % (node.name))
 
             self.printer.writeline(
-                   "context.namespaces[(__name__, %s)] = ns" % repr(node.name))
-            self.printer.write("\n")
+                "context.namespaces[(__name__, %s)] = ns" % repr(node.name))
+            self.printer.write_blanks(1)
         if not len(namespaces):
             self.printer.writeline("pass")
         self.printer.writeline(None)
@@ -466,7 +487,7 @@ class _GenerateRenderMethod(object):
             for ident, ns in self.compiler.namespaces.items():
                 if 'import' in ns.attributes:
                     self.printer.writeline(
-                            "_mako_get_namespace(context, %r)."\
+                            "_mako_get_namespace(context, %r)."
                                     "_populate(_import_ns, %r)" %
                             (
                                 ident,
@@ -532,18 +553,11 @@ class _GenerateRenderMethod(object):
 
         self.printer.writeline("__M_writer = context.writer()")
 
-    def write_source_comment(self, node):
-        """write a source comment containing the line number of the
-        corresponding template line."""
-        if self.last_source_line != node.lineno:
-            self.printer.writeline("# SOURCE LINE %d" % node.lineno)
-            self.last_source_line = node.lineno
-
     def write_def_decl(self, node, identifiers):
         """write a locally-available callable referencing a top-level def"""
         funcname = node.funcname
         namedecls = node.get_argument_expressions()
-        nameargs = node.get_argument_expressions(include_defaults=False)
+        nameargs = node.get_argument_expressions(as_call=True)
 
         if not self.in_def and (
                                 len(self.identifiers.locally_assigned) > 0 or
@@ -553,7 +567,7 @@ class _GenerateRenderMethod(object):
             nameargs.insert(0, 'context')
         self.printer.writeline("def %s(%s):" % (funcname, ",".join(namedecls)))
         self.printer.writeline(
-                       "return render_%s(%s)" % (funcname, ",".join(nameargs)))
+                    "return render_%s(%s)" % (funcname, ",".join(nameargs)))
         self.printer.writeline(None)
 
     def write_inline_def(self, node, identifiers, nested):
@@ -564,9 +578,9 @@ class _GenerateRenderMethod(object):
         decorator = node.decorator
         if decorator:
             self.printer.writeline(
-                          "@runtime._decorate_inline(context, %s)" % decorator)
+                        "@runtime._decorate_inline(context, %s)" % decorator)
         self.printer.writeline(
-                          "def %s(%s):" % (node.funcname, ",".join(namedecls)))
+                        "def %s(%s):" % (node.funcname, ",".join(namedecls)))
         filtered = len(node.filter_args.args) > 0
         buffered = eval(node.attributes.get('buffered', 'False'))
         cached = eval(node.attributes.get('cached', 'False'))
@@ -574,11 +588,11 @@ class _GenerateRenderMethod(object):
             # push new frame, assign current frame to __M_caller
             "__M_caller = context.caller_stack._push_frame()",
             "try:"
-            )
+        )
         if buffered or filtered or cached:
             self.printer.writelines(
                 "context._push_buffer()",
-                )
+            )
 
         identifiers = identifiers.branch(node, nested=nested)
 
@@ -627,8 +641,8 @@ class _GenerateRenderMethod(object):
                 )
             else:
                 self.printer.writelines(
-                   "finally:",
-                       "__M_buf, __M_writer = context._pop_buffer_and_writer()"
+                    "finally:",
+                    "__M_buf, __M_writer = context._pop_buffer_and_writer()"
                 )
 
             if callstack:
@@ -684,7 +698,7 @@ class _GenerateRenderMethod(object):
 
         # form "arg1, arg2, arg3=arg3, arg4=arg4", etc.
         pass_args = [
-                        '=' in a and "%s=%s" % ((a.split('=')[0],)*2) or a
+                        "%s=%s" % ((a.split('=')[0],) * 2) if '=' in a else a
                         for a in args
                     ]
 
@@ -696,11 +710,11 @@ class _GenerateRenderMethod(object):
         if buffered:
             s = "context.get('local')."\
                 "cache._ctx_get_or_create("\
-                "%s, lambda:__M_%s(%s),  context, %s__M_defname=%r)" % \
-                            (cachekey, name, ','.join(pass_args),
-                            ''.join(["%s=%s, " % (k, v)
-                            for k, v in cache_args.items()]),
-                            name
+                "%s, lambda:__M_%s(%s),  context, %s__M_defname=%r)" % (
+                                cachekey, name, ','.join(pass_args),
+                                ''.join(["%s=%s, " % (k, v)
+                                for k, v in cache_args.items()]),
+                                name
                             )
             # apply buffer_filters
             s = self.create_filter_callable(self.compiler.buffer_filters, s,
@@ -709,12 +723,13 @@ class _GenerateRenderMethod(object):
         else:
             self.printer.writelines(
                     "__M_writer(context.get('local')."
-                    "cache._ctx_get_or_create("\
+                    "cache._ctx_get_or_create("
                     "%s, lambda:__M_%s(%s), context, %s__M_defname=%r))" %
-                    (cachekey, name, ','.join(pass_args),
-                    ''.join(["%s=%s, " % (k, v)
+                    (
+                        cachekey, name, ','.join(pass_args),
+                        ''.join(["%s=%s, " % (k, v)
                         for k, v in cache_args.items()]),
-                    name,
+                        name,
                     ),
                     "return ''",
                 None
@@ -745,18 +760,17 @@ class _GenerateRenderMethod(object):
                 continue
             m = re.match(r'(.+?)(\(.*\))', e)
             if m:
-                (ident, fargs) = m.group(1,2)
+                ident, fargs = m.group(1, 2)
                 f = locate_encode(ident)
                 e = f + fargs
             else:
-                x = e
                 e = locate_encode(e)
                 assert e is not None
             target = "%s(%s)" % (e, target)
         return target
 
     def visitExpression(self, node):
-        self.write_source_comment(node)
+        self.printer.start_source(node.lineno)
         if len(node.escapes) or \
                 (
                     self.compiler.pagetag is not None and
@@ -778,7 +792,7 @@ class _GenerateRenderMethod(object):
                 self.printer.writeline("loop = __M_loop._exit()")
                 self.printer.writeline(None)
         else:
-            self.write_source_comment(node)
+            self.printer.start_source(node.lineno)
             if self.compiler.enable_loop and node.keyword == 'for':
                 text = mangle_mako_loop(node, self.printer)
             else:
@@ -800,7 +814,7 @@ class _GenerateRenderMethod(object):
                 self.printer.writeline("pass")
 
     def visitText(self, node):
-        self.write_source_comment(node)
+        self.printer.start_source(node.lineno)
         self.printer.writeline("__M_writer(%s)" % repr(node.content))
 
     def visitTextTag(self, node):
@@ -822,11 +836,11 @@ class _GenerateRenderMethod(object):
                                 "__M_buf.getvalue()",
                                 False),
                 None
-                )
+            )
 
     def visitCode(self, node):
         if not node.ismodule:
-            self.write_source_comment(node)
+            self.printer.start_source(node.lineno)
             self.printer.write_indented_block(node.text)
 
             if not self.in_def and len(self.identifiers.locally_assigned) > 0:
@@ -835,20 +849,20 @@ class _GenerateRenderMethod(object):
                 # which is used for def calls within the same template,
                 # to simulate "enclosing scope"
                 self.printer.writeline(
-                      '__M_locals_builtin_stored = __M_locals_builtin()')
+                    '__M_locals_builtin_stored = __M_locals_builtin()')
                 self.printer.writeline(
-                      '__M_locals.update(__M_dict_builtin([(__M_key,'
-                      ' __M_locals_builtin_stored[__M_key]) for __M_key in'
-                      ' [%s] if __M_key in __M_locals_builtin_stored]))' %
-                      ','.join([repr(x) for x in node.declared_identifiers()]))
+                    '__M_locals.update(__M_dict_builtin([(__M_key,'
+                    ' __M_locals_builtin_stored[__M_key]) for __M_key in'
+                    ' [%s] if __M_key in __M_locals_builtin_stored]))' %
+                    ','.join([repr(x) for x in node.declared_identifiers()]))
 
     def visitIncludeTag(self, node):
-        self.write_source_comment(node)
+        self.printer.start_source(node.lineno)
         args = node.attributes.get('args')
         if args:
             self.printer.writeline(
-                      "runtime._include_file(context, %s, _template_uri, %s)" %
-                      (node.parsed_attributes['file'], args))
+                    "runtime._include_file(context, %s, _template_uri, %s)" %
+                    (node.parsed_attributes['file'], args))
         else:
             self.printer.writeline(
                         "runtime._include_file(context, %s, _template_uri)" %
@@ -864,7 +878,7 @@ class _GenerateRenderMethod(object):
         if node.is_anonymous:
             self.printer.writeline("%s()" % node.funcname)
         else:
-            nameargs = node.get_argument_expressions(include_defaults=False)
+            nameargs = node.get_argument_expressions(as_call=True)
             nameargs += ['**pageargs']
             self.printer.writeline("if 'parent' not in context._data or "
                                   "not hasattr(context._data['parent'], '%s'):"
@@ -941,9 +955,9 @@ class _GenerateRenderMethod(object):
             # push on caller for nested call
             "context.caller_stack.nextcaller = "
                 "runtime.Namespace('caller', context, "
-                                  "callables=ccall(__M_caller))",
+                                "callables=ccall(__M_caller))",
             "try:")
-        self.write_source_comment(node)
+        self.printer.start_source(node.lineno)
         self.printer.writelines(
                 "__M_writer(%s)" % self.create_filter_callable(
                                                     [], node.expression, True),
@@ -966,9 +980,9 @@ class _Identifiers(object):
                 # things that have already been declared
                 # in an enclosing namespace (i.e. names we can just use)
                 self.declared = set(parent.declared).\
-                         union([c.name for c in parent.closuredefs.values()]).\
-                         union(parent.locally_declared).\
-                         union(parent.argument_declared)
+                        union([c.name for c in parent.closuredefs.values()]).\
+                        union(parent.locally_declared).\
+                        union(parent.argument_declared)
 
                 # if these identifiers correspond to a "nested"
                 # scope, it means whatever the parent identifiers
@@ -1012,7 +1026,7 @@ class _Identifiers(object):
             node.accept_visitor(self)
 
         illegal_names = self.compiler.reserved_names.intersection(
-                                                         self.locally_declared)
+                                                        self.locally_declared)
         if illegal_names:
             raise exceptions.NameConflictError(
                 "Reserved words declared in template: %s" %
@@ -1047,7 +1061,7 @@ class _Identifiers(object):
 
         for ident in node.undeclared_identifiers():
             if ident != 'context' and\
-                       ident not in self.declared.union(self.locally_declared):
+                    ident not in self.declared.union(self.locally_declared):
                 self.undeclared.add(ident)
         for ident in node.declared_identifiers():
             self.locally_declared.add(ident)
@@ -1067,7 +1081,7 @@ class _Identifiers(object):
         if not node.ismodule:
             self.check_declared(node)
             self.locally_assigned = self.locally_assigned.union(
-                                                   node.declared_identifiers())
+                                                node.declared_identifiers())
 
     def visitNamespaceTag(self, node):
         # only traverse into the sub-elements of a
@@ -1095,8 +1109,8 @@ class _Identifiers(object):
             self._check_name_exists(self.closuredefs, node)
 
         for ident in node.undeclared_identifiers():
-            if ident != 'context' and\
-                       ident not in self.declared.union(self.locally_declared):
+            if ident != 'context' and \
+                    ident not in self.declared.union(self.locally_declared):
                 self.undeclared.add(ident)
 
         # visit defs only one level deep
@@ -1108,8 +1122,7 @@ class _Identifiers(object):
                 n.accept_visitor(self)
 
     def visitBlockTag(self, node):
-        if node is not self.node and \
-            not node.is_anonymous:
+        if node is not self.node and not node.is_anonymous:
 
             if isinstance(self.node, parsetree.DefTag):
                 raise exceptions.CompileException(
@@ -1123,7 +1136,7 @@ class _Identifiers(object):
 
         for ident in node.undeclared_identifiers():
             if ident != 'context' and \
-                       ident not in self.declared.union(self.locally_declared):
+                    ident not in self.declared.union(self.locally_declared):
                 self.undeclared.add(ident)
 
         if not node.is_anonymous:
@@ -1139,7 +1152,7 @@ class _Identifiers(object):
     def visitTextTag(self, node):
         for ident in node.undeclared_identifiers():
             if ident != 'context' and \
-                ident not in self.declared.union(self.locally_declared):
+                    ident not in self.declared.union(self.locally_declared):
                 self.undeclared.add(ident)
 
     def visitIncludeTag(self, node):
@@ -1156,8 +1169,9 @@ class _Identifiers(object):
     def visitCallTag(self, node):
         if node is self.node:
             for ident in node.undeclared_identifiers():
-                if ident != 'context' and\
-                       ident not in self.declared.union(self.locally_declared):
+                if ident != 'context' and \
+                        ident not in self.declared.union(
+                                                self.locally_declared):
                     self.undeclared.add(ident)
             for ident in node.declared_identifiers():
                 self.argument_declared.add(ident)
@@ -1165,15 +1179,16 @@ class _Identifiers(object):
                 n.accept_visitor(self)
         else:
             for ident in node.undeclared_identifiers():
-                if ident != 'context' and\
-                       ident not in self.declared.union(self.locally_declared):
+                if ident != 'context' and \
+                        ident not in self.declared.union(
+                                                self.locally_declared):
                     self.undeclared.add(ident)
 
 
 _FOR_LOOP = re.compile(
         r'^for\s+((?:\(?)\s*[A-Za-z_][A-Za-z_0-9]*'
         r'(?:\s*,\s*(?:[A-Za-z_][A-Za-z0-9_]*),??)*\s*(?:\)?))\s+in\s+(.*):'
-    )
+)
 
 def mangle_mako_loop(node, printer):
     """converts a for loop into a context manager wrapped around a for loop
@@ -1189,7 +1204,7 @@ def mangle_mako_loop(node, printer):
                     'loop = __M_loop._enter(%s)' % match.group(2),
                     'try:'
                     #'with __M_loop(%s) as loop:' % match.group(2)
-                )
+            )
             text = 'for %s in loop:' % match.group(1)
         else:
             raise SyntaxError("Couldn't apply loop context: %s" % node.text)

lib/mako/compat.py

@@ -3,8 +3,8 @@ import time
 
 py3k = sys.version_info >= (3, 0)
 py33 = sys.version_info >= (3, 3)
+py2k = sys.version_info < (3,)
 py26 = sys.version_info >= (2, 6)
-py25 = sys.version_info >= (2, 5)
 jython = sys.platform.startswith('java')
 win32 = sys.platform.startswith('win')
 pypy = hasattr(sys, 'pypy_version_info')
@@ -23,6 +23,9 @@ if py3k:
     def u(s):
         return s
 
+    def b(s):
+        return s.encode("latin-1")
+
     def octal(lit):
         return eval("0o" + lit)
 
@@ -44,6 +47,9 @@ else:
     def u(s):
         return unicode(s, "utf-8")
 
+    def b(s):
+        return s
+
     def octal(lit):
         return eval("0" + lit)
 
@@ -62,6 +68,18 @@ else:
             fp.close()
 
 
+if py3k:
+    def reraise(tp, value, tb=None, cause=None):
+        if cause is not None:
+            value.__cause__ = cause
+        if value.__traceback__ is not tb:
+            raise value.with_traceback(tb)
+        raise value
+else:
+    exec("def reraise(tp, value, tb=None, cause=None):\n"
+            "    raise tp, value, tb\n")
+
+
 def exception_as():
     return sys.exc_info()[1]
 
@@ -93,23 +111,12 @@ except:
             return func(*(args + fargs), **newkeywords)
         return newfunc
 
-if not py25:
-    def all(iterable):
-        for i in iterable:
-            if not i:
-                return False
-        return True
 
-    def exception_name(exc):
-        try:
-            return exc.__class__.__name__
-        except AttributeError:
-            return exc.__name__
-else:
-    all = all
+all = all
+import json
 
-    def exception_name(exc):
-        return exc.__class__.__name__
+def exception_name(exc):
+    return exc.__class__.__name__
 
 try:
     from inspect import CO_VARKEYWORDS, CO_VARARGS

lib/mako/exceptions.py

@@ -1,5 +1,5 @@
 # mako/exceptions.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
@@ -8,7 +8,6 @@
 
 import traceback
 import sys
-import re
 from mako import util, compat
 
 class MakoException(Exception):
@@ -28,7 +27,7 @@ class CompileException(MakoException):
     def __init__(self, message, source, lineno, pos, filename):
         MakoException.__init__(self,
                               message + _format_filepos(lineno, pos, filename))
-        self.lineno =lineno
+        self.lineno = lineno
         self.pos = pos
         self.filename = filename
         self.source = source
@@ -37,7 +36,7 @@ class SyntaxException(MakoException):
     def __init__(self, message, source, lineno, pos, filename):
         MakoException.__init__(self,
                               message + _format_filepos(lineno, pos, filename))
-        self.lineno =lineno
+        self.lineno = lineno
         self.pos = pos
         self.filename = filename
         self.source = source
@@ -77,7 +76,6 @@ class RichTraceback(object):
         self.records = self._init(traceback)
 
         if isinstance(self.error, (CompileException, SyntaxException)):
-            import mako.template
             self.source = self.error.source
             self.lineno = self.error.lineno
             self._has_source = True
@@ -167,19 +165,19 @@ class RichTraceback(object):
                                             None, None, None, None))
                     continue
 
-                template_ln = module_ln = 1
-                line_map = {}
-                for line in module_source.split("\n"):
-                    match = re.match(r'\s*# SOURCE LINE (\d+)', line)
-                    if match:
-                        template_ln = int(match.group(1))
-                    module_ln += 1
-                    line_map[module_ln] = template_ln
+                template_ln = 1
+
+                source_map = mako.template.ModuleInfo.\
+                                get_module_source_metadata(
+                                    module_source, full_line_map=True)
+                line_map = source_map['full_line_map']
+
                 template_lines = [line for line in
                                     template_source.split("\n")]
                 mods[filename] = (line_map, template_lines)
 
-            template_ln = line_map[lineno]
+            template_ln = line_map[lineno - 1]
+
             if template_ln <= len(template_lines):
                 template_line = template_lines[template_ln - 1]
             else:
@@ -188,7 +186,7 @@ class RichTraceback(object):
                                 line, template_filename, template_ln,
                                 template_line, template_source))
         if not self.source:
-            for l in range(len(new_trcback)-1, 0, -1):
+            for l in range(len(new_trcback) - 1, 0, -1):
                 if new_trcback[l][5]:
                     self.source = new_trcback[l][7]
                     self.lineno = new_trcback[l][5]
@@ -260,10 +258,11 @@ def html_error_template():
     filenames, line numbers and code for that of the originating source
     template, as applicable.
 
-    The template's default ``encoding_errors`` value is ``'htmlentityreplace'``. The
-    template has two options. With the ``full`` option disabled, only a section of
-    an HTML document is returned. With the ``css`` option disabled, the default
-    stylesheet won't be included.
+    The template's default ``encoding_errors`` value is
+    ``'htmlentityreplace'``. The template has two options. With the
+    ``full`` option disabled, only a section of an HTML document is
+    returned. With the ``css`` option disabled, the default stylesheet
+    won't be included.
 
     """
     import mako.template

lib/mako/ext/autohandler.py

@@ -1,5 +1,5 @@
 # ext/autohandler.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
@@ -48,7 +48,7 @@ def autohandler(template, context, name='autohandler'):
         if len(tokens) == 1:
             break
         tokens[-2:] = [name]
-
+ 
     if not lookup.filesystem_checks:
         return lookup._uri_cache.setdefault(
                             (autohandler, _template_uri, name), None)
@@ -62,4 +62,4 @@ def _file_exists(lookup, path):
             return True
     else:
         return False
-
+ 

lib/mako/ext/babelplugin.py

@@ -1,14 +1,36 @@
 # ext/babelplugin.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
 
 """gettext message extraction via Babel: http://babel.edgewall.org/"""
 from babel.messages.extract import extract_python
-from mako.compat import StringIO
-from mako import compat
-from mako import lexer, parsetree
+from mako.ext.extract import MessageExtractor
+
+
+class BabelMakoExtractor(MessageExtractor):
+    def __init__(self, keywords, comment_tags, options):
+        self.keywords = keywords
+        self.options = options
+        self.config = {
+                'comment-tags': u' '.join(comment_tags),
+                'encoding': options.get('input_encoding',
+                    options.get('encoding', None)),
+            }
+        super(BabelMakoExtractor, self).__init__()
+
+    def __call__(self, fileobj):
+        return self.process_file(fileobj)
+
+    def process_python(self, code, code_lineno, translator_strings):
+        comment_tags = self.config['comment-tags']
+        for lineno, funcname, messages, python_translator_comments \
+                in extract_python(code,
+                        self.keywords, comment_tags, self.options):
+            yield (code_lineno + (lineno - 1), funcname, messages,
+                   translator_strings + python_translator_comments)
+
 
 def extract(fileobj, keywords, comment_tags, options):
     """Extract messages from Mako templates.
@@ -22,105 +44,6 @@ def extract(fileobj, keywords, comment_tags, options):
     :return: an iterator over ``(lineno, funcname, message, comments)`` tuples
     :rtype: ``iterator``
     """
-    encoding = options.get('input_encoding', options.get('encoding', None))
-
-    template_node = lexer.Lexer(fileobj.read(),
-                                input_encoding=encoding).parse()
-    for extracted in extract_nodes(template_node.get_children(),
-                                   keywords, comment_tags, options):
-        yield extracted
-
-def extract_nodes(nodes, keywords, comment_tags, options):
-    """Extract messages from Mako's lexer node objects
-
-    :param nodes: an iterable of Mako parsetree.Node objects to extract from
-    :param keywords: a list of keywords (i.e. function names) that should be
-                     recognized as translation functions
-    :param comment_tags: a list of translator tags to search for and include
-                         in the results
-    :param options: a dictionary of additional options (optional)
-    :return: an iterator over ``(lineno, funcname, message, comments)`` tuples
-    :rtype: ``iterator``
-    """
-    translator_comments = []
-    in_translator_comments = False
-
-    for node in nodes:
-        child_nodes = None
-        if in_translator_comments and isinstance(node, parsetree.Text) and \
-                not node.content.strip():
-            # Ignore whitespace within translator comments
-            continue
-
-        if isinstance(node, parsetree.Comment):
-            value = node.text.strip()
-            if in_translator_comments:
-                translator_comments.extend(_split_comment(node.lineno, value))
-                continue
-            for comment_tag in comment_tags:
-                if value.startswith(comment_tag):
-                    in_translator_comments = True
-                    translator_comments.extend(_split_comment(node.lineno,
-                                                              value))
-            continue
-
-        if isinstance(node, parsetree.DefTag):
-            code = node.function_decl.code
-            child_nodes = node.nodes
-        elif isinstance(node, parsetree.BlockTag):
-            code = node.body_decl.code
-            child_nodes = node.nodes
-        elif isinstance(node, parsetree.CallTag):
-            code = node.code.code
-            child_nodes = node.nodes
-        elif isinstance(node, parsetree.PageTag):
-            code = node.body_decl.code
-        elif isinstance(node, parsetree.CallNamespaceTag):
-            code = node.expression
-            child_nodes = node.nodes
-        elif isinstance(node, parsetree.ControlLine):
-            if node.isend:
-                in_translator_comments = False
-                continue
-            code = node.text
-        elif isinstance(node, parsetree.Code):
-            in_translator_comments = False
-            code = node.code.code
-        elif isinstance(node, parsetree.Expression):
-            code = node.code.code
-        else:
-            continue
-
-        # Comments don't apply unless they immediately preceed the message
-        if translator_comments and \
-                translator_comments[-1][0] < node.lineno - 1:
-            translator_comments = []
-
-        translator_strings = [comment[1] for comment in translator_comments]
-
-        if isinstance(code, compat.text_type):
-            code = code.encode('ascii', 'backslashreplace')
-
-        used_translator_comments = False
-        code = compat.byte_buffer(code)
-        for lineno, funcname, messages, python_translator_comments \
-                in extract_python(code, keywords, comment_tags, options):
-            yield (node.lineno + (lineno - 1), funcname, messages,
-                   translator_strings + python_translator_comments)
-            used_translator_comments = True
-
-        if used_translator_comments:
-            translator_comments = []
-        in_translator_comments = False
-
-        if child_nodes:
-            for extracted in extract_nodes(child_nodes, keywords, comment_tags,
-                                           options):
-                yield extracted
-
-
-def _split_comment(lineno, comment):
-    """Return the multiline comment at lineno split into a list of comment line
-    numbers and the accompanying comment line"""
-    return [(lineno + index, line) for index, line in
-            enumerate(comment.splitlines())]
+    extractor = BabelMakoExtractor(keywords, comment_tags, options)
+    for message in extractor(fileobj):
+        yield message

lib/mako/ext/beaker_cache.py

@@ -4,7 +4,16 @@ from mako import exceptions
 
 from mako.cache import CacheImpl
 
+try:
+    from beaker import cache as beaker_cache
+except:
+    has_beaker = False
+else:
+    has_beaker = True
+
 _beaker_cache = None
+
+
 class BeakerCacheImpl(CacheImpl):
     """A :class:`.CacheImpl` provided for the Beaker caching system.
 
@@ -15,15 +24,11 @@ class BeakerCacheImpl(CacheImpl):
     """
 
     def __init__(self, cache):
+        if not has_beaker:
+            raise exceptions.RuntimeException(
+                "Can't initialize Beaker plugin; Beaker is not installed.")
         global _beaker_cache
         if _beaker_cache is None:
-            try:
-                from beaker import cache as beaker_cache
-            except ImportError:
-                raise exceptions.RuntimeException(
-                            "the Beaker package is required to use cache "
-                            "functionality.")
-
             if 'manager' in cache.template.cache_args:
                 _beaker_cache = cache.template.cache_args['manager']
             else:
@@ -48,7 +53,7 @@ class BeakerCacheImpl(CacheImpl):
             cache = _beaker_cache.get_cache_region(self.cache.id, region, **kw)
         else:
             cache = _beaker_cache.get_cache(self.cache.id, **kw)
-        cache_args = {'starttime':self.cache.starttime}
+        cache_args = {'starttime': self.cache.starttime}
         if expiretime:
             cache_args['expiretime'] = expiretime
         return cache, cache_args

lib/mako/ext/extract.py

@@ -0,0 +1,101 @@
+import re
+from mako import compat
+from mako import lexer
+from mako import parsetree
+
+
+class MessageExtractor(object):
+    def process_file(self, fileobj):
+        template_node = lexer.Lexer(
+            fileobj.read(),
+            input_encoding=self.config['encoding']).parse()
+        for extracted in self.extract_nodes(template_node.get_children()):
+            yield extracted
+
+    def extract_nodes(self, nodes):
+        translator_comments = []
+        in_translator_comments = False
+        comment_tags = list(
+            filter(None, re.split(r'\s+', self.config['comment-tags'])))
+
+        for node in nodes:
+            child_nodes = None
+            if in_translator_comments and \
+                    isinstance(node, parsetree.Text) and \
+                    not node.content.strip():
+                # Ignore whitespace within translator comments
+                continue
+
+            if isinstance(node, parsetree.Comment):
+                value = node.text.strip()
+                if in_translator_comments:
+                    translator_comments.extend(
+                        self._split_comment(node.lineno, value))
+                    continue
+                for comment_tag in comment_tags:
+                    if value.startswith(comment_tag):
+                        in_translator_comments = True
+                        translator_comments.extend(
+                            self._split_comment(node.lineno, value))
+                continue
+
+            if isinstance(node, parsetree.DefTag):
+                code = node.function_decl.code
+                child_nodes = node.nodes
+            elif isinstance(node, parsetree.BlockTag):
+                code = node.body_decl.code
+                child_nodes = node.nodes
+            elif isinstance(node, parsetree.CallTag):
+                code = node.code.code
+                child_nodes = node.nodes
+            elif isinstance(node, parsetree.PageTag):
+                code = node.body_decl.code
+            elif isinstance(node, parsetree.CallNamespaceTag):
+                code = node.expression
+                child_nodes = node.nodes
+            elif isinstance(node, parsetree.ControlLine):
+                if node.isend:
+                    in_translator_comments = False
+                    continue
+                code = node.text
+            elif isinstance(node, parsetree.Code):
+                in_translator_comments = False
+                code = node.code.code
+            elif isinstance(node, parsetree.Expression):
+                code = node.code.code
+            else:
+                continue
+
+            # Comments don't apply unless they immediately preceed the message
+            if translator_comments and \
+                    translator_comments[-1][0] < node.lineno - 1:
+                translator_comments = []
+
+            translator_strings = [
+                comment[1] for comment in translator_comments]
+
+            if isinstance(code, compat.text_type):
+                code = code.encode('ascii', 'backslashreplace')
+
+            used_translator_comments = False
+            code = compat.byte_buffer(code)
+
+            for message in self.process_python(
+                    code, node.lineno, translator_strings):
+                yield message
+                used_translator_comments = True
+
+            if used_translator_comments:
+                translator_comments = []
+            in_translator_comments = False
+
+            if child_nodes:
+                for extracted in self.extract_nodes(child_nodes):
+                    yield extracted
+
+    @staticmethod
+    def _split_comment(lineno, comment):
+        """Return the multiline comment at lineno split into a list of
+        comment line numbers and the accompanying comment line"""
+        return [(lineno + index, line) for index, line in
+                enumerate(comment.splitlines())]

lib/mako/ext/linguaplugin.py

@@ -0,0 +1,38 @@
+import io
+from lingua.extractors import Extractor
+from lingua.extractors import Message
+from lingua.extractors import get_extractor
+from mako.ext.extract import MessageExtractor
+from mako import compat
+
+
+class LinguaMakoExtractor(Extractor, MessageExtractor):
+    '''Mako templates'''
+    extensions = ['.mako']
+    default_config = {
+        'encoding': 'utf-8',
+        'comment-tags': '',
+    }
+
+    def __call__(self, filename, options, fileobj=None):
+        self.options = options
+        self.filename = filename
+        self.python_extractor = get_extractor('x.py')
+        if fileobj is None:
+            fileobj = open(filename, 'rb')
+        return self.process_file(fileobj)
+
+    def process_python(self, code, code_lineno, translator_strings):
+        source = code.getvalue().strip()
+        if source.endswith(compat.b(':')):
+            source += compat.b(' pass')
+            code = io.BytesIO(source)
+        for msg in self.python_extractor(
+                self.filename, self.options, code, code_lineno):
+            if translator_strings:
+                msg = Message(msg.msgctxt, msg.msgid, msg.msgid_plural,
+                              msg.flags,
+                              compat.u(' ').join(
+                                  translator_strings + [msg.comment]),
+                              msg.tcomment, msg.location)
+            yield msg

lib/mako/ext/preprocessors.py

@@ -1,20 +1,20 @@
 # ext/preprocessors.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
 
-"""preprocessing functions, used with the 'preprocessor'
+"""preprocessing functions, used with the 'preprocessor' 
 argument on Template, TemplateLookup"""
 
 import re
 
 def convert_comments(text):
     """preprocess old style comments.
-
+ 
     example:
-
+ 
     from mako.ext.preprocessors import convert_comments
-    t = Template(..., preprocessor=preprocess_comments)"""
+    t = Template(..., preprocessor=convert_comments)"""
     return re.sub(r'(?<=\n)\s*#[^#]', "##", text)
 

lib/mako/ext/pygmentplugin.py

@@ -1,5 +1,5 @@
 # ext/pygmentplugin.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php

lib/mako/ext/turbogears.py

@@ -1,5 +1,5 @@
 # ext/turbogears.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php

lib/mako/filters.py

@@ -1,5 +1,5 @@
 # mako/filters.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
@@ -48,6 +48,10 @@ def url_escape(string):
     string = string.encode("utf8")
     return quote_plus(string)
 
+def legacy_url_escape(string):
+    # convert into a list of octets
+    return quote_plus(string)
+
 def url_unescape(string):
     text = unquote_plus(string)
     if not is_ascii_str(text):
@@ -64,7 +68,7 @@ class Decode(object):
             if isinstance(x, compat.text_type):
                 return x
             elif not isinstance(x, compat.binary_type):
-                return compat.text_type(str(x), encoding=key)
+                return decode(str(x))
             else:
                 return compat.text_type(x, encoding=key)
         return decode
@@ -193,4 +197,5 @@ if compat.py3k:
 
 NON_UNICODE_ESCAPES = DEFAULT_ESCAPES.copy()
 NON_UNICODE_ESCAPES['h'] = 'filters.legacy_html_escape'
+NON_UNICODE_ESCAPES['u'] = 'filters.legacy_url_escape'
 

lib/mako/lexer.py

@@ -1,5 +1,5 @@
 # mako/lexer.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
@@ -44,10 +44,10 @@ class Lexer(object):
 
     @property
     def exception_kwargs(self):
-        return {'source':self.text,
-                'lineno':self.matched_lineno,
-                'pos':self.matched_charpos,
-                'filename':self.filename}
+        return {'source': self.text,
+                'lineno': self.matched_lineno,
+                'pos': self.matched_charpos,
+                'filename': self.filename}
 
     def match(self, regexp, flags=None):
         """compile the given regexp, cache the reg, and call match_reg()."""
@@ -83,8 +83,8 @@ class Lexer(object):
             self.matched_lineno = self.lineno
             lines = re.findall(r"\n", self.text[mp:self.match_position])
             cp = mp - 1
-            while (cp >= 0 and cp<self.textlength and self.text[cp] != '\n'):
-                cp -=1
+            while (cp >= 0 and cp < self.textlength and self.text[cp] != '\n'):
+                cp -= 1
             self.matched_charpos = mp - cp
             self.lineno += len(lines)
             #print "MATCHED:", match.group(0), "LINE START:",
@@ -111,8 +111,8 @@ class Lexer(object):
                     brace_level -= 1
                     continue
                 return \
-                    self.text[startpos:\
-                              self.match_position-len(match.group(1))],\
+                    self.text[startpos:
+                              self.match_position - len(match.group(1))],\
                     match.group(1)
             match = self.match(r"(.*?)(?=\"|\'|#|%s)" % text_re, re.S)
             if match:
@@ -162,9 +162,9 @@ class Lexer(object):
             elif self.control_line and \
                     not self.control_line[-1].is_ternary(node.keyword):
                 raise exceptions.SyntaxException(
-                          "Keyword '%s' not a legal ternary for keyword '%s'" %
-                          (node.keyword, self.control_line[-1].keyword),
-                          **self.exception_kwargs)
+                        "Keyword '%s' not a legal ternary for keyword '%s'" %
+                        (node.keyword, self.control_line[-1].keyword),
+                        **self.exception_kwargs)
 
     _coding_re = re.compile(r'#.*coding[:=]\s*([-\w.]+).*\r?\n')
 
@@ -201,10 +201,10 @@ class Lexer(object):
                 text = text.decode(parsed_encoding)
             except UnicodeDecodeError:
                 raise exceptions.CompileException(
-                           "Unicode decode operation of encoding '%s' failed" %
-                           parsed_encoding,
-                           text.decode('utf-8', 'ignore'),
-                           0, 0, filename)
+                        "Unicode decode operation of encoding '%s' failed" %
+                        parsed_encoding,
+                        text.decode('utf-8', 'ignore'),
+                        0, 0, filename)
 
         return parsed_encoding, text
 
@@ -254,11 +254,11 @@ class Lexer(object):
                                                 **self.exception_kwargs)
         if len(self.control_line):
             raise exceptions.SyntaxException(
-                                      "Unterminated control keyword: '%s'" %
-                                      self.control_line[-1].keyword,
-                                      self.text,
-                                      self.control_line[-1].lineno,
-                                      self.control_line[-1].pos, self.filename)
+                                    "Unterminated control keyword: '%s'" %
+                                    self.control_line[-1].keyword,
+                                    self.text,
+                                    self.control_line[-1].lineno,
+                                    self.control_line[-1].pos, self.filename)
         return self.template
 
     def match_tag_start(self):
@@ -311,14 +311,14 @@ class Lexer(object):
         if match:
             if not len(self.tag):
                 raise exceptions.SyntaxException(
-                                   "Closing tag without opening tag: </%%%s>" %
-                                    match.group(1),
-                                    **self.exception_kwargs)
+                                "Closing tag without opening tag: </%%%s>" %
+                                match.group(1),
+                                **self.exception_kwargs)
             elif self.tag[-1].keyword != match.group(1):
                 raise exceptions.SyntaxException(
-                             "Closing tag </%%%s> does not match tag: <%%%s>" %
-                             (match.group(1), self.tag[-1].keyword),
-                             **self.exception_kwargs)
+                            "Closing tag </%%%s> does not match tag: <%%%s>" %
+                            (match.group(1), self.tag[-1].keyword),
+                            **self.exception_kwargs)
             self.tag.pop()
             return True
         else:
@@ -370,9 +370,9 @@ class Lexer(object):
             # compiler.parse() not complain about indentation
             text = adjust_whitespace(text) + "\n"
             self.append_node(
-                            parsetree.Code,
-                            text,
-                            match.group(1)=='!', lineno=line, pos=pos)
+                        parsetree.Code,
+                        text,
+                        match.group(1) == '!', lineno=line, pos=pos)
             return True
         else:
             return False
@@ -397,8 +397,8 @@ class Lexer(object):
 
     def match_control_line(self):
         match = self.match(
-                      r"(?<=^)[\t ]*(%(?!%)|##)[\t ]*((?:(?:\\r?\n)|[^\r\n])*)"
-                      r"(?:\r?\n|\Z)", re.M)
+                    r"(?<=^)[\t ]*(%(?!%)|##)[\t ]*((?:(?:\\r?\n)|[^\r\n])*)"
+                    r"(?:\r?\n|\Z)", re.M)
         if match:
             operator = match.group(1)
             text = match.group(2)

lib/mako/lookup.py

@@ -1,5 +1,5 @@
 # mako/lookup.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php

lib/mako/parsetree.py

@@ -1,5 +1,5 @@
 # mako/parsetree.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
@@ -437,7 +437,7 @@ class DefTag(Tag):
         return self.function_decl.get_argument_expressions(**kw)
 
     def declared_identifiers(self):
-        return self.function_decl.argnames
+        return self.function_decl.allargnames
 
     def undeclared_identifiers(self):
         res = []
@@ -451,7 +451,7 @@ class DefTag(Tag):
         ).union(
             self.expression_undeclared_identifiers
         ).difference(
-            self.function_decl.argnames
+            self.function_decl.allargnames
         )
 
 class BlockTag(Tag):
@@ -502,7 +502,7 @@ class BlockTag(Tag):
         return self.body_decl.get_argument_expressions(**kw)
 
     def declared_identifiers(self):
-        return self.body_decl.argnames
+        return self.body_decl.allargnames
 
     def undeclared_identifiers(self):
         return (self.filter_args.\
@@ -524,7 +524,7 @@ class CallTag(Tag):
                                             **self.exception_kwargs)
 
     def declared_identifiers(self):
-        return self.code.declared_identifiers.union(self.body_decl.argnames)
+        return self.code.declared_identifiers.union(self.body_decl.allargnames)
 
     def undeclared_identifiers(self):
         return self.code.undeclared_identifiers.\
@@ -554,7 +554,7 @@ class CallNamespaceTag(Tag):
                                     **self.exception_kwargs)
 
     def declared_identifiers(self):
-        return self.code.declared_identifiers.union(self.body_decl.argnames)
+        return self.code.declared_identifiers.union(self.body_decl.allargnames)
 
     def undeclared_identifiers(self):
         return self.code.undeclared_identifiers.\
@@ -589,6 +589,6 @@ class PageTag(Tag):
                                 **self.exception_kwargs)
 
     def declared_identifiers(self):
-        return self.body_decl.argnames
+        return self.body_decl.allargnames
 
 

lib/mako/pygen.py

@@ -1,5 +1,5 @@
 # mako/pygen.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
@@ -26,6 +26,9 @@ class PythonPrinter(object):
         # the stream we are writing to
         self.stream = stream
 
+        # current line number
+        self.lineno = 1
+
         # a list of lines that represents a buffered "block" of code,
         # which can be later printed relative to an indent level
         self.line_buffer = []
@@ -34,8 +37,20 @@ class PythonPrinter(object):
 
         self._reset_multi_line_flags()
 
-    def write(self, text):
-        self.stream.write(text)
+        # mapping of generated python lines to template
+        # source lines
+        self.source_map = {}
+
+    def _update_lineno(self, num):
+        self.lineno += num
+
+    def start_source(self, lineno):
+        if self.lineno not in self.source_map:
+            self.source_map[self.lineno] = lineno
+
+    def write_blanks(self, num):
+        self.stream.write("\n" * num)
+        self._update_lineno(num)
 
     def write_indented_block(self, block):
         """print a line or lines of python which already contain indentation.
@@ -45,6 +60,7 @@ class PythonPrinter(object):
         self.in_indent_lines = False
         for l in re.split(r'\r?\n', block):
             self.line_buffer.append(l)
+            self._update_lineno(1)
 
     def writelines(self, *lines):
         """print a series of lines of python."""
@@ -80,7 +96,7 @@ class PythonPrinter(object):
             ):
 
             if self.indent > 0:
-                self.indent -=1
+                self.indent -= 1
                 # if the indent_detail stack is empty, the user
                 # probably put extra closures - the resulting
                 # module wont compile.
@@ -94,6 +110,7 @@ class PythonPrinter(object):
 
         # write the line
         self.stream.write(self._indent_line(line) + "\n")
+        self._update_lineno(len(line.split("\n")))
 
         # see if this line should increase the indentation level.
         # note that a line can both decrase (before printing) and
@@ -108,7 +125,7 @@ class PythonPrinter(object):
             if match:
                 # its a "compound" keyword, so we will check for "unindentors"
                 indentor = match.group(1)
-                self.indent +=1
+                self.indent += 1
                 self.indent_detail.append(indentor)
             else:
                 indentor = None
@@ -265,7 +282,7 @@ def adjust_whitespace(text):
 
         return start_state
 
-    def _indent_line(line, stripspace = ''):
+    def _indent_line(line, stripspace=''):
         return re.sub(r"^%s" % stripspace, '', line)
 
     lines = []

lib/mako/pyparser.py

@@ -1,5 +1,5 @@
 # mako/pyparser.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
@@ -11,7 +11,7 @@ module is used.
 """
 
 from mako import exceptions, util, compat
-from mako.compat import StringIO, arg_stringname
+from mako.compat import arg_stringname
 import operator
 
 if compat.py3k:
@@ -29,29 +29,16 @@ else:
     # the "id" attribute on a function node
     arg_id = operator.attrgetter('id')
 
-
-try:
-    import _ast
-    util.restore__ast(_ast)
-    from mako import _ast_util
-except ImportError:
-    _ast = None
-    from compiler import parse as compiler_parse
-    from compiler import visitor
-
+import _ast
+util.restore__ast(_ast)
+from mako import _ast_util
 
 
 def parse(code, mode='exec', **exception_kwargs):
     """Parse an expression into AST"""
 
-
     try:
-        if _ast:
-            return _ast_util.parse(code, '<unknown>', mode)
-        else:
-            if isinstance(code, compat.text_type):
-                code = code.encode('ascii', 'backslashreplace')
-            return compiler_parse(code, mode)
+        return _ast_util.parse(code, '<unknown>', mode)
     except Exception:
         raise exceptions.SyntaxException(
                     "(%s) %s (%r)" % (
@@ -61,512 +48,185 @@ def parse(code, mode='exec', **exception_kwargs):
                     ), **exception_kwargs)
 
 
-if _ast:
-    class FindIdentifiers(_ast_util.NodeVisitor):
+class FindIdentifiers(_ast_util.NodeVisitor):
 
-        def __init__(self, listener, **exception_kwargs):
-            self.in_function = False
-            self.in_assign_targets = False
-            self.local_ident_stack = set()
-            self.listener = listener
-            self.exception_kwargs = exception_kwargs
+    def __init__(self, listener, **exception_kwargs):
+        self.in_function = False
+        self.in_assign_targets = False
+        self.local_ident_stack = set()
+        self.listener = listener
+        self.exception_kwargs = exception_kwargs
 
-        def _add_declared(self, name):
-            if not self.in_function:
-                self.listener.declared_identifiers.add(name)
-            else:
-                self.local_ident_stack.add(name)
+    def _add_declared(self, name):
+        if not self.in_function:
+            self.listener.declared_identifiers.add(name)
+        else:
+            self.local_ident_stack.add(name)
 
-        def visit_ClassDef(self, node):
-            self._add_declared(node.name)
+    def visit_ClassDef(self, node):
+        self._add_declared(node.name)
 
-        def visit_Assign(self, node):
+    def visit_Assign(self, node):
 
-            # flip around the visiting of Assign so the expression gets
-            # evaluated first, in the case of a clause like "x=x+5" (x
-            # is undeclared)
+        # flip around the visiting of Assign so the expression gets
+        # evaluated first, in the case of a clause like "x=x+5" (x
+        # is undeclared)
 
-            self.visit(node.value)
-            in_a = self.in_assign_targets
-            self.in_assign_targets = True
-            for n in node.targets:
-                self.visit(n)
-            self.in_assign_targets = in_a
+        self.visit(node.value)
+        in_a = self.in_assign_targets
+        self.in_assign_targets = True
+        for n in node.targets:
+            self.visit(n)
+        self.in_assign_targets = in_a
 
-        if compat.py3k:
+    if compat.py3k:
 
-            # ExceptHandler is in Python 2, but this block only works in
-            # Python 3 (and is required there)
+        # ExceptHandler is in Python 2, but this block only works in
+        # Python 3 (and is required there)
 
-            def visit_ExceptHandler(self, node):
-                if node.name is not None:
-                    self._add_declared(node.name)
-                if node.type is not None:
-                    self.listener.undeclared_identifiers.add(node.type.id)
-                for statement in node.body:
-                    self.visit(statement)
-
-        def visit_Lambda(self, node, *args):
-            self._visit_function(node, True)
-
-        def visit_FunctionDef(self, node):
-            self._add_declared(node.name)
-            self._visit_function(node, False)
-
-        def _expand_tuples(self, args):
-            for arg in args:
-                if isinstance(arg, _ast.Tuple):
-                    for n in arg.elts:
-                        yield n
-                else:
-                    yield arg
-
-        def _visit_function(self, node, islambda):
-
-            # push function state onto stack.  dont log any more
-            # identifiers as "declared" until outside of the function,
-            # but keep logging identifiers as "undeclared". track
-            # argument names in each function header so they arent
-            # counted as "undeclared"
-
-            inf = self.in_function
-            self.in_function = True
-
-            local_ident_stack = self.local_ident_stack
-            self.local_ident_stack = local_ident_stack.union([
-                arg_id(arg) for arg in self._expand_tuples(node.args.args)
-            ])
-            if islambda:
-                self.visit(node.body)
-            else:
-                for n in node.body:
-                    self.visit(n)
-            self.in_function = inf
-            self.local_ident_stack = local_ident_stack
-
-        def visit_For(self, node):
-
-            # flip around visit
-
-            self.visit(node.iter)
-            self.visit(node.target)
+        def visit_ExceptHandler(self, node):
+            if node.name is not None:
+                self._add_declared(node.name)
+            if node.type is not None:
+                self.visit(node.type)
             for statement in node.body:
                 self.visit(statement)
-            for statement in node.orelse:
-                self.visit(statement)
 
-        def visit_Name(self, node):
-            if isinstance(node.ctx, _ast.Store):
-                # this is eqiuvalent to visit_AssName in
-                # compiler
-                self._add_declared(node.id)
-            elif node.id not in reserved and node.id \
-                not in self.listener.declared_identifiers and node.id \
-                not in self.local_ident_stack:
-                self.listener.undeclared_identifiers.add(node.id)
+    def visit_Lambda(self, node, *args):
+        self._visit_function(node, True)
 
-        def visit_Import(self, node):
-            for name in node.names:
-                if name.asname is not None:
-                    self._add_declared(name.asname)
-                else:
-                    self._add_declared(name.name.split('.')[0])
+    def visit_FunctionDef(self, node):
+        self._add_declared(node.name)
+        self._visit_function(node, False)
 
-        def visit_ImportFrom(self, node):
-            for name in node.names:
-                if name.asname is not None:
-                    self._add_declared(name.asname)
-                else:
-                    if name.name == '*':
-                        raise exceptions.CompileException(
-                          "'import *' is not supported, since all identifier "
-                          "names must be explicitly declared.  Please use the "
-                          "form 'from <modulename> import <name1>, <name2>, "
-                          "...' instead.", **self.exception_kwargs)
-                    self._add_declared(name.name)
-
-
-    class FindTuple(_ast_util.NodeVisitor):
-
-        def __init__(self, listener, code_factory, **exception_kwargs):
-            self.listener = listener
-            self.exception_kwargs = exception_kwargs
-            self.code_factory = code_factory
-
-        def visit_Tuple(self, node):
-            for n in node.elts:
-                p = self.code_factory(n, **self.exception_kwargs)
-                self.listener.codeargs.append(p)
-                self.listener.args.append(ExpressionGenerator(n).value())
-                self.listener.declared_identifiers = \
-                    self.listener.declared_identifiers.union(
-                                                    p.declared_identifiers)
-                self.listener.undeclared_identifiers = \
-                    self.listener.undeclared_identifiers.union(
-                                                    p.undeclared_identifiers)
-
-
-    class ParseFunc(_ast_util.NodeVisitor):
-
-        def __init__(self, listener, **exception_kwargs):
-            self.listener = listener
-            self.exception_kwargs = exception_kwargs
-
-        def visit_FunctionDef(self, node):
-            self.listener.funcname = node.name
-            argnames = [arg_id(arg) for arg in node.args.args]
-            if node.args.vararg:
-                argnames.append(arg_stringname(node.args.vararg))
-            if node.args.kwarg:
-                argnames.append(arg_stringname(node.args.kwarg))
-            self.listener.argnames = argnames
-            self.listener.defaults = node.args.defaults  # ast
-            self.listener.varargs = node.args.vararg
-            self.listener.kwargs = node.args.kwarg
-
-    class ExpressionGenerator(object):
-
-        def __init__(self, astnode):
-            self.generator = _ast_util.SourceGenerator(' ' * 4)
-            self.generator.visit(astnode)
-
-        def value(self):
-            return ''.join(self.generator.result)
-else:
-    class FindIdentifiers(object):
-
-        def __init__(self, listener, **exception_kwargs):
-            self.in_function = False
-            self.local_ident_stack = set()
-            self.listener = listener
-            self.exception_kwargs = exception_kwargs
-
-        def _add_declared(self, name):
-            if not self.in_function:
-                self.listener.declared_identifiers.add(name)
+    def _expand_tuples(self, args):
+        for arg in args:
+            if isinstance(arg, _ast.Tuple):
+                for n in arg.elts:
+                    yield n
             else:
-                self.local_ident_stack.add(name)
+                yield arg
 
-        def visitClass(self, node, *args):
-            self._add_declared(node.name)
+    def _visit_function(self, node, islambda):
 
-        def visitAssName(self, node, *args):
-            self._add_declared(node.name)
+        # push function state onto stack.  dont log any more
+        # identifiers as "declared" until outside of the function,
+        # but keep logging identifiers as "undeclared". track
+        # argument names in each function header so they arent
+        # counted as "undeclared"
 
-        def visitAssign(self, node, *args):
+        inf = self.in_function
+        self.in_function = True
 
-            # flip around the visiting of Assign so the expression gets
-            # evaluated first, in the case of a clause like "x=x+5" (x
-            # is undeclared)
-            self.visit(node.expr, *args)
-            for n in node.nodes:
-                self.visit(n, *args)
+        local_ident_stack = self.local_ident_stack
+        self.local_ident_stack = local_ident_stack.union([
+            arg_id(arg) for arg in self._expand_tuples(node.args.args)
+        ])
+        if islambda:
+            self.visit(node.body)
+        else:
+            for n in node.body:
+                self.visit(n)
+        self.in_function = inf
+        self.local_ident_stack = local_ident_stack
 
-        def visitLambda(self, node, *args):
-            self._visit_function(node, args)
+    def visit_For(self, node):
 
-        def visitFunction(self, node, *args):
-            self._add_declared(node.name)
-            self._visit_function(node, args)
+        # flip around visit
 
-        def _expand_tuples(self, args):
-            for arg in args:
-                if isinstance(arg, tuple):
-                    for n in arg:
-                        yield n
-                else:
-                    yield arg
+        self.visit(node.iter)
+        self.visit(node.target)
+        for statement in node.body:
+            self.visit(statement)
+        for statement in node.orelse:
+            self.visit(statement)
 
-        def _visit_function(self, node, args):
-
-            # push function state onto stack.  dont log any more
-            # identifiers as "declared" until outside of the function,
-            # but keep logging identifiers as "undeclared". track
-            # argument names in each function header so they arent
-            # counted as "undeclared"
-
-            inf = self.in_function
-            self.in_function = True
-
-            local_ident_stack = self.local_ident_stack
-            self.local_ident_stack = local_ident_stack.union([
-                arg for arg in self._expand_tuples(node.argnames)
-            ])
-
-            for n in node.getChildNodes():
-                self.visit(n, *args)
-            self.in_function = inf
-            self.local_ident_stack = local_ident_stack
-
-        def visitFor(self, node, *args):
-
-            # flip around visit
-
-            self.visit(node.list, *args)
-            self.visit(node.assign, *args)
-            self.visit(node.body, *args)
-
-        def visitName(self, node, *args):
-            if node.name not in reserved and node.name \
-                not in self.listener.declared_identifiers and node.name \
+    def visit_Name(self, node):
+        if isinstance(node.ctx, _ast.Store):
+            # this is eqiuvalent to visit_AssName in
+            # compiler
+            self._add_declared(node.id)
+        elif node.id not in reserved and node.id \
+            not in self.listener.declared_identifiers and node.id \
                 not in self.local_ident_stack:
-                self.listener.undeclared_identifiers.add(node.name)
+            self.listener.undeclared_identifiers.add(node.id)
 
-        def visitImport(self, node, *args):
-            for mod, alias in node.names:
-                if alias is not None:
-                    self._add_declared(alias)
-                else:
-                    self._add_declared(mod.split('.')[0])
+    def visit_Import(self, node):
+        for name in node.names:
+            if name.asname is not None:
+                self._add_declared(name.asname)
+            else:
+                self._add_declared(name.name.split('.')[0])
 
-        def visitFrom(self, node, *args):
-            for mod, alias in node.names:
-                if alias is not None:
-                    self._add_declared(alias)
-                else:
-                    if mod == '*':
-                        raise exceptions.CompileException(
+    def visit_ImportFrom(self, node):
+        for name in node.names:
+            if name.asname is not None:
+                self._add_declared(name.asname)
+            else:
+                if name.name == '*':
+                    raise exceptions.CompileException(
                         "'import *' is not supported, since all identifier "
                         "names must be explicitly declared.  Please use the "
                         "form 'from <modulename> import <name1>, <name2>, "
                         "...' instead.", **self.exception_kwargs)
-                    self._add_declared(mod)
-
-        def visit(self, expr):
-            visitor.walk(expr, self)  # , walker=walker())
+                self._add_declared(name.name)
 
 
-    class FindTuple(object):
+class FindTuple(_ast_util.NodeVisitor):
 
-        def __init__(self, listener, code_factory, **exception_kwargs):
-            self.listener = listener
-            self.exception_kwargs = exception_kwargs
-            self.code_factory = code_factory
+    def __init__(self, listener, code_factory, **exception_kwargs):
+        self.listener = listener
+        self.exception_kwargs = exception_kwargs
+        self.code_factory = code_factory
 
-        def visitTuple(self, node, *args):
-            for n in node.nodes:
-                p = self.code_factory(n, **self.exception_kwargs)
-                self.listener.codeargs.append(p)
-                self.listener.args.append(ExpressionGenerator(n).value())
-                self.listener.declared_identifiers = \
-                    self.listener.declared_identifiers.union(
-                                                      p.declared_identifiers)
-                self.listener.undeclared_identifiers = \
-                    self.listener.undeclared_identifiers.union(
-                                                      p.undeclared_identifiers)
-
-        def visit(self, expr):
-            visitor.walk(expr, self)  # , walker=walker())
+    def visit_Tuple(self, node):
+        for n in node.elts:
+            p = self.code_factory(n, **self.exception_kwargs)
+            self.listener.codeargs.append(p)
+            self.listener.args.append(ExpressionGenerator(n).value())
+            self.listener.declared_identifiers = \
+                self.listener.declared_identifiers.union(
+                                                p.declared_identifiers)
+            self.listener.undeclared_identifiers = \
+                self.listener.undeclared_identifiers.union(
+                                                p.undeclared_identifiers)
 
 
-    class ParseFunc(object):
+class ParseFunc(_ast_util.NodeVisitor):
 
-        def __init__(self, listener, **exception_kwargs):
-            self.listener = listener
-            self.exception_kwargs = exception_kwargs
+    def __init__(self, listener, **exception_kwargs):
+        self.listener = listener
+        self.exception_kwargs = exception_kwargs
 
-        def visitFunction(self, node, *args):
-            self.listener.funcname = node.name
-            self.listener.argnames = node.argnames
-            self.listener.defaults = node.defaults
-            self.listener.varargs = node.varargs
-            self.listener.kwargs = node.kwargs
+    def visit_FunctionDef(self, node):
+        self.listener.funcname = node.name
 
-        def visit(self, expr):
-            visitor.walk(expr, self)
+        argnames = [arg_id(arg) for arg in node.args.args]
+        if node.args.vararg:
+            argnames.append(arg_stringname(node.args.vararg))
 
+        if compat.py2k:
+            # kw-only args don't exist in Python 2
+            kwargnames = []
+        else:
+            kwargnames = [arg_id(arg) for arg in node.args.kwonlyargs]
+        if node.args.kwarg:
+            kwargnames.append(arg_stringname(node.args.kwarg))
+        self.listener.argnames = argnames
+        self.listener.defaults = node.args.defaults  # ast
+        self.listener.kwargnames = kwargnames
+        if compat.py2k:
+            self.listener.kwdefaults = []
+        else:
+            self.listener.kwdefaults = node.args.kw_defaults
+        self.listener.varargs = node.args.vararg
+        self.listener.kwargs = node.args.kwarg
 
-    class ExpressionGenerator(object):
+class ExpressionGenerator(object):
 
-        """given an AST node, generates an equivalent literal Python
-        expression."""
+    def __init__(self, astnode):
+        self.generator = _ast_util.SourceGenerator(' ' * 4)
+        self.generator.visit(astnode)
 
-        def __init__(self, astnode):
-            self.buf = StringIO()
-            visitor.walk(astnode, self)  # , walker=walker())
-
-        def value(self):
-            return self.buf.getvalue()
-
-        def operator(self, op, node, *args):
-            self.buf.write('(')
-            self.visit(node.left, *args)
-            self.buf.write(' %s ' % op)
-            self.visit(node.right, *args)
-            self.buf.write(')')
-
-        def booleanop(self, op, node, *args):
-            self.visit(node.nodes[0])
-            for n in node.nodes[1:]:
-                self.buf.write(' ' + op + ' ')
-                self.visit(n, *args)
-
-        def visitConst(self, node, *args):
-            self.buf.write(repr(node.value))
-
-        def visitAssName(self, node, *args):
-
-            # TODO: figure out OP_ASSIGN, other OP_s
-
-            self.buf.write(node.name)
-
-        def visitName(self, node, *args):
-            self.buf.write(node.name)
-
-        def visitMul(self, node, *args):
-            self.operator('*', node, *args)
-
-        def visitAnd(self, node, *args):
-            self.booleanop('and', node, *args)
-
-        def visitOr(self, node, *args):
-            self.booleanop('or', node, *args)
-
-        def visitBitand(self, node, *args):
-            self.booleanop('&', node, *args)
-
-        def visitBitor(self, node, *args):
-            self.booleanop('|', node, *args)
-
-        def visitBitxor(self, node, *args):
-            self.booleanop('^', node, *args)
-
-        def visitAdd(self, node, *args):
-            self.operator('+', node, *args)
-
-        def visitGetattr(self, node, *args):
-            self.visit(node.expr, *args)
-            self.buf.write('.%s' % node.attrname)
-
-        def visitSub(self, node, *args):
-            self.operator('-', node, *args)
-
-        def visitNot(self, node, *args):
-            self.buf.write('not ')
-            self.visit(node.expr)
-
-        def visitDiv(self, node, *args):
-            self.operator('/', node, *args)
-
-        def visitFloorDiv(self, node, *args):
-            self.operator('//', node, *args)
-
-        def visitSubscript(self, node, *args):
-            self.visit(node.expr)
-            self.buf.write('[')
-            [self.visit(x) for x in node.subs]
-            self.buf.write(']')
-
-        def visitUnarySub(self, node, *args):
-            self.buf.write('-')
-            self.visit(node.expr)
-
-        def visitUnaryAdd(self, node, *args):
-            self.buf.write('-')
-            self.visit(node.expr)
-
-        def visitSlice(self, node, *args):
-            self.visit(node.expr)
-            self.buf.write('[')
-            if node.lower is not None:
-                self.visit(node.lower)
-            self.buf.write(':')
-            if node.upper is not None:
-                self.visit(node.upper)
-            self.buf.write(']')
-
-        def visitDict(self, node):
-            self.buf.write('{')
-            c = node.getChildren()
-            for i in range(0, len(c), 2):
-                self.visit(c[i])
-                self.buf.write(': ')
-                self.visit(c[i + 1])
-                if i < len(c) - 2:
-                    self.buf.write(', ')
-            self.buf.write('}')
-
-        def visitTuple(self, node):
-            self.buf.write('(')
-            c = node.getChildren()
-            for i in range(0, len(c)):
-                self.visit(c[i])
-                if i < len(c) - 1:
-                    self.buf.write(', ')
-            self.buf.write(')')
-
-        def visitList(self, node):
-            self.buf.write('[')
-            c = node.getChildren()
-            for i in range(0, len(c)):
-                self.visit(c[i])
-                if i < len(c) - 1:
-                    self.buf.write(', ')
-            self.buf.write(']')
-
-        def visitListComp(self, node):
-            self.buf.write('[')
-            self.visit(node.expr)
-            self.buf.write(' ')
-            for n in node.quals:
-                self.visit(n)
-            self.buf.write(']')
-
-        def visitListCompFor(self, node):
-            self.buf.write(' for ')
-            self.visit(node.assign)
-            self.buf.write(' in ')
-            self.visit(node.list)
-            for n in node.ifs:
-                self.visit(n)
-
-        def visitListCompIf(self, node):
-            self.buf.write(' if ')
-            self.visit(node.test)
-
-        def visitCompare(self, node):
-            self.visit(node.expr)
-            for tup in node.ops:
-                self.buf.write(tup[0])
-                self.visit(tup[1])
-
-        def visitCallFunc(self, node, *args):
-            self.visit(node.node)
-            self.buf.write('(')
-            if len(node.args):
-                self.visit(node.args[0])
-                for a in node.args[1:]:
-                    self.buf.write(', ')
-                    self.visit(a)
-            self.buf.write(')')
-
-        def visitLambda(self, node, *args):
-            self.buf.write('lambda ')
-
-            argnames = list(node.argnames)
-
-            kw = arg = None
-            if node.kwargs > 0:
-                kw = argnames.pop(-1)
-            if node.varargs > 0:
-                arg = argnames.pop(-1)
-
-            if arg:
-                argnames.append("*%s" % arg)
-            if kw:
-                argnames.append("**%s" % kw)
-
-            self.buf.write(", ".join(argnames))
-
-            self.buf.write(': ')
-            self.visit(node.code)
-
-
-    class walker(visitor.ASTVisitor):
-
-        def dispatch(self, node, *args):
-            print('Node:', str(node))
-
-            # print "dir:", dir(node)
-
-            return visitor.ASTVisitor.dispatch(self, node, *args)
+    def value(self):
+        return ''.join(self.generator.result)

lib/mako/runtime.py

@@ -1,5 +1,5 @@
 # mako/runtime.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
@@ -9,9 +9,7 @@ Namespace, and various helper functions."""
 
 from mako import exceptions, util, compat
 from mako.compat import compat_builtins
-import inspect
 import sys
-import collections
 
 
 class Context(object):
@@ -132,9 +130,7 @@ class Context(object):
     def get(self, key, default=None):
         """Return a value from this :class:`.Context`."""
 
-        return self._data.get(key,
-                compat_builtins.__dict__.get(key, default)
-                )
+        return self._data.get(key, compat_builtins.__dict__.get(key, default))
 
     def write(self, string):
         """Write a string to this :class:`.Context` object's
@@ -474,8 +470,8 @@ class Namespace(object):
     def get_template(self, uri):
         """Return a :class:`.Template` from the given ``uri``.
 
-        The ``uri`` resolution is relative to the ``uri`` of this :class:`.Namespace`
-        object's :class:`.Template`.
+        The ``uri`` resolution is relative to the ``uri`` of this
+        :class:`.Namespace` object's :class:`.Template`.
 
         """
         return _lookup_template(self.context, uri, self._templateuri)
@@ -673,7 +669,7 @@ def supports_caller(func):
 
     """
 
-    def wrap_stackframe(context,  *args, **kwargs):
+    def wrap_stackframe(context, *args, **kwargs):
         context.caller_stack._push_frame()
         try:
             return func(context, *args, **kwargs)
@@ -691,8 +687,8 @@ def capture(context, callable_, *args, **kwargs):
 
     if not compat.callable(callable_):
         raise exceptions.RuntimeException(
-                           "capture() function expects a callable as "
-                           "its argument (i.e. capture(func, *args, **kwargs))"
+                        "capture() function expects a callable as "
+                        "its argument (i.e. capture(func, *args, **kwargs))"
                         )
     context._push_buffer()
     try:
@@ -853,7 +849,6 @@ def _exec_template(callable_, context, args=None, kwargs=None):
     template = context._with_template
     if template is not None and \
             (template.format_exceptions or template.error_handler):
-        error = None
         try:
             callable_(context, *args, **kwargs)
         except Exception:
@@ -868,11 +863,12 @@ def _render_error(template, context, error):
     if template.error_handler:
         result = template.error_handler(context, error)
         if not result:
-            raise error
+            compat.reraise(*sys.exc_info())
     else:
         error_template = exceptions.html_error_template()
         if context._outputting_as_unicode:
-            context._buffer_stack[:] = [util.FastEncodingBuffer(as_unicode=True)]
+            context._buffer_stack[:] = [
+                                    util.FastEncodingBuffer(as_unicode=True)]
         else:
             context._buffer_stack[:] = [util.FastEncodingBuffer(
                                             error_template.output_encoding,

lib/mako/template.py

@@ -1,5 +1,5 @@
 # mako/template.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
@@ -596,6 +596,25 @@ class ModuleInfo(object):
         if module_filename:
             self._modules[module_filename] = self
 
+    @classmethod
+    def get_module_source_metadata(cls, module_source, full_line_map=False):
+        source_map = re.search(
+                        r"__M_BEGIN_METADATA(.+?)__M_END_METADATA",
+                        module_source, re.S).group(1)
+        source_map = compat.json.loads(source_map)
+        source_map['line_map'] = dict((int(k), int(v))
+                                    for k, v in source_map['line_map'].items())
+        if full_line_map:
+            f_line_map = source_map['full_line_map'] = []
+            line_map = source_map['line_map']
+
+            curr_templ_line = 1
+            for mod_line in range(1, max(line_map)):
+                if mod_line in line_map:
+                    curr_templ_line = line_map[mod_line]
+                f_line_map.append(curr_templ_line)
+        return source_map
+
     @property
     def code(self):
         if self.module_source is not None:

lib/mako/util.py

@@ -1,5 +1,5 @@
 # mako/util.py
-# Copyright (C) 2006-2013 the Mako authors and contributors <see AUTHORS file>
+# Copyright (C) 2006-2015 the Mako authors and contributors <see AUTHORS file>
 #
 # This module is part of Mako and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
@@ -11,15 +11,10 @@ import os
 from mako import compat
 import operator
 
-def function_named(fn, name):
-    """Return a function with a given __name__.
-
-    Will assign to __name__ and return the original function if possible on
-    the Python implementation, otherwise a new function will be constructed.
-
-    """
-    fn.__name__ = name
-    return fn
+def update_wrapper(decorated, fn):
+    decorated.__wrapped__ = fn
+    decorated.__name__ = fn.__name__
+    return decorated
 
 
 class PluginLoader(object):

lib/musicbrainzngs/musicbrainz.py

@@ -271,6 +271,7 @@ user = password = ""
 hostname = "musicbrainz.org"
 _client = ""
 _useragent = ""
+mb_auth = False
 
 def auth(u, p):
 	"""Set the username and password to be used in subsequent queries to
@@ -284,9 +285,16 @@ def hpauth(u, p):
     """Set the username and password to be used in subsequent queries to
     the MusicBrainz XML API that require authentication.
     """
-    global hpuser, hppassword
+    global hpuser, hppassword, mb_auth
     hpuser = u
     hppassword = p
+    mb_auth = True
+
+def disable_hpauth():
+    """Disable the authentication for MusicBrainz XML API
+    """
+    global mb_auth
+    mb_auth = False
 
 def set_useragent(app, version, contact=None):
     """Set the User-Agent to be used for requests to the MusicBrainz webservice.
@@ -635,7 +643,7 @@ def _mb_request(path, method='GET', auth_required=False, client_required=False,
     req.add_header('User-Agent', _useragent)
 
     # Add headphones credentials
-    if hostname == '144.76.94.239:8181':
+    if mb_auth:
         base64string = base64.encodestring('%s:%s' % (hpuser, hppassword)).replace('\n', '')
         req.add_header("Authorization", "Basic %s" % base64string)
 

lib/mutagen/README.rst

@@ -0,0 +1,58 @@
+Mutagen
+=======
+
+Mutagen is a Python module to handle audio metadata. It supports ASF, FLAC, 
+M4A, Monkey's Audio, MP3, Musepack, Ogg Opus, Ogg FLAC, Ogg Speex, Ogg 
+Theora, Ogg Vorbis, True Audio, WavPack, OptimFROG, and AIFF audio files. 
+All versions of ID3v2 are supported, and all standard ID3v2.4 frames are 
+parsed. It can read Xing headers to accurately calculate the bitrate and 
+length of MP3s. ID3 and APEv2 tags can be edited regardless of audio 
+format. It can also manipulate Ogg streams on an individual packet/page 
+level.
+
+Mutagen works on Python 2.6, 2.7, 3.3, 3.4 (CPython and PyPy) and has no 
+dependencies outside the Python standard library.
+
+
+Installing
+----------
+
+ $ ./setup.py build
+ $ su -c "./setup.py install"
+
+
+Documentation
+-------------
+
+The primary documentation for Mutagen is the doc strings found in
+the source code and the sphinx documentation in the docs/ directory.
+
+To build the docs (needs sphinx):
+
+ $ ./setup.py build_sphinx
+
+The tools/ directory contains several useful examples.
+
+The docs are also hosted on readthedocs.org:
+
+ http://mutagen.readthedocs.org
+
+
+Testing the Module
+------------------
+
+To test Mutagen's MP3 reading support, run
+ $ tools/mutagen-pony <your top-level MP3 directory here>
+Mutagen will try to load all of them, and report any errors.
+
+To look at the tags in files, run
+ $ tools/mutagen-inspect filename ...
+
+To run our test suite,
+ $ ./setup.py test
+
+
+Compatibility/Bugs
+------------------
+
+See docs/bugs.rst

lib/mutagen/__init__.py

@@ -1,4 +1,5 @@
-# mutagen aims to be an all purpose media tagging library
+# -*- coding: utf-8 -*-
+
 # Copyright (C) 2005  Michael Urman
 #
 # This program is free software; you can redistribute it and/or modify
@@ -6,7 +7,7 @@
 # published by the Free Software Foundation.
 
 
-"""Mutagen aims to be an all purpose tagging library.
+"""Mutagen aims to be an all purpose multimedia tagging library.
 
 ::
 
@@ -19,245 +20,22 @@ depending on tag or format. They may also be entirely different objects
 for certain keys, again depending on format.
 """
 
-version = (1, 22)
+from mutagen._util import MutagenError
+from mutagen._file import FileType, StreamInfo, File
+from mutagen._tags import Metadata
+
+version = (1, 27)
 """Version tuple."""
 
 version_string = ".".join(map(str, version))
 """Version string."""
 
+MutagenError
 
-import warnings
+FileType
 
-import mutagen._util
+StreamInfo
 
+File
 
-class Metadata(object):
-    """An abstract dict-like object.
-
-    Metadata is the base class for many of the tag objects in Mutagen.
-    """
-
-    def __init__(self, *args, **kwargs):
-        if args or kwargs:
-            self.load(*args, **kwargs)
-
-    def load(self, *args, **kwargs):
-        raise NotImplementedError
-
-    def save(self, filename=None):
-        """Save changes to a file."""
-
-        raise NotImplementedError
-
-    def delete(self, filename=None):
-        """Remove tags from a file."""
-
-        raise NotImplementedError
-
-
-class FileType(mutagen._util.DictMixin):
-    """An abstract object wrapping tags and audio stream information.
-
-    Attributes:
-
-    * info -- stream information (length, bitrate, sample rate)
-    * tags -- metadata tags, if any
-
-    Each file format has different potential tags and stream
-    information.
-
-    FileTypes implement an interface very similar to Metadata; the
-    dict interface, save, load, and delete calls on a FileType call
-    the appropriate methods on its tag data.
-    """
-
-    info = None
-    tags = None
-    filename = None
-    _mimes = ["application/octet-stream"]
-
-    def __init__(self, filename=None, *args, **kwargs):
-        if filename is None:
-            warnings.warn("FileType constructor requires a filename",
-                          DeprecationWarning)
-        else:
-            self.load(filename, *args, **kwargs)
-
-    def load(self, filename, *args, **kwargs):
-        raise NotImplementedError
-
-    def __getitem__(self, key):
-        """Look up a metadata tag key.
-
-        If the file has no tags at all, a KeyError is raised.
-        """
-
-        if self.tags is None:
-            raise KeyError(key)
-        else:
-            return self.tags[key]
-
-    def __setitem__(self, key, value):
-        """Set a metadata tag.
-
-        If the file has no tags, an appropriate format is added (but
-        not written until save is called).
-        """
-
-        if self.tags is None:
-            self.add_tags()
-        self.tags[key] = value
-
-    def __delitem__(self, key):
-        """Delete a metadata tag key.
-
-        If the file has no tags at all, a KeyError is raised.
-        """
-
-        if self.tags is None:
-            raise KeyError(key)
-        else:
-            del(self.tags[key])
-
-    def keys(self):
-        """Return a list of keys in the metadata tag.
-
-        If the file has no tags at all, an empty list is returned.
-        """
-
-        if self.tags is None:
-            return []
-        else:
-            return self.tags.keys()
-
-    def delete(self, filename=None):
-        """Remove tags from a file."""
-
-        if self.tags is not None:
-            if filename is None:
-                filename = self.filename
-            else:
-                warnings.warn(
-                    "delete(filename=...) is deprecated, reload the file",
-                    DeprecationWarning)
-            return self.tags.delete(filename)
-
-    def save(self, filename=None, **kwargs):
-        """Save metadata tags."""
-
-        if filename is None:
-            filename = self.filename
-        else:
-            warnings.warn(
-                "save(filename=...) is deprecated, reload the file",
-                DeprecationWarning)
-        if self.tags is not None:
-            return self.tags.save(filename, **kwargs)
-        else:
-            raise ValueError("no tags in file")
-
-    def pprint(self):
-        """Print stream information and comment key=value pairs."""
-
-        stream = "%s (%s)" % (self.info.pprint(), self.mime[0])
-        try:
-            tags = self.tags.pprint()
-        except AttributeError:
-            return stream
-        else:
-            return stream + ((tags and "\n" + tags) or "")
-
-    def add_tags(self):
-        """Adds new tags to the file.
-
-        Raises if tags already exist.
-        """
-
-        raise NotImplementedError
-
-    @property
-    def mime(self):
-        """A list of mime types"""
-
-        mimes = []
-        for Kind in type(self).__mro__:
-            for mime in getattr(Kind, '_mimes', []):
-                if mime not in mimes:
-                    mimes.append(mime)
-        return mimes
-
-    @staticmethod
-    def score(filename, fileobj, header):
-        raise NotImplementedError
-
-
-def File(filename, options=None, easy=False):
-    """Guess the type of the file and try to open it.
-
-    The file type is decided by several things, such as the first 128
-    bytes (which usually contains a file type identifier), the
-    filename extension, and the presence of existing tags.
-
-    If no appropriate type could be found, None is returned.
-
-    :param options: Sequence of :class:`FileType` implementations, defaults to
-                    all included ones.
-
-    :param easy: If the easy wrappers should be returnd if available.
-                 For example :class:`EasyMP3 <mp3.EasyMP3>` instead
-                 of :class:`MP3 <mp3.MP3>`.
-    """
-
-    if options is None:
-        from mutagen.asf import ASF
-        from mutagen.apev2 import APEv2File
-        from mutagen.flac import FLAC
-        if easy:
-            from mutagen.easyid3 import EasyID3FileType as ID3FileType
-        else:
-            from mutagen.id3 import ID3FileType
-        if easy:
-            from mutagen.mp3 import EasyMP3 as MP3
-        else:
-            from mutagen.mp3 import MP3
-        from mutagen.oggflac import OggFLAC
-        from mutagen.oggspeex import OggSpeex
-        from mutagen.oggtheora import OggTheora
-        from mutagen.oggvorbis import OggVorbis
-        from mutagen.oggopus import OggOpus
-        if easy:
-            from mutagen.trueaudio import EasyTrueAudio as TrueAudio
-        else:
-            from mutagen.trueaudio import TrueAudio
-        from mutagen.wavpack import WavPack
-        if easy:
-            from mutagen.easymp4 import EasyMP4 as MP4
-        else:
-            from mutagen.mp4 import MP4
-        from mutagen.musepack import Musepack
-        from mutagen.monkeysaudio import MonkeysAudio
-        from mutagen.optimfrog import OptimFROG
-        options = [MP3, TrueAudio, OggTheora, OggSpeex, OggVorbis, OggFLAC,
-                   FLAC, APEv2File, MP4, ID3FileType, WavPack, Musepack,
-                   MonkeysAudio, OptimFROG, ASF, OggOpus]
-
-    if not options:
-        return None
-
-    fileobj = open(filename, "rb")
-    try:
-        header = fileobj.read(128)
-        # Sort by name after score. Otherwise import order affects
-        # Kind sort order, which affects treatment of things with
-        # equals scores.
-        results = [(Kind.score(filename, fileobj, header), Kind.__name__)
-                   for Kind in options]
-    finally:
-        fileobj.close()
-    results = zip(results, options)
-    results.sort()
-    (score, name), Kind = results[-1]
-    if score > 0:
-        return Kind(filename)
-    else:
-        return None
+Metadata

lib/mutagen/_compat.py

@@ -0,0 +1,84 @@
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2013  Christoph Reiter
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of version 2 of the GNU General Public License as
+# published by the Free Software Foundation.
+
+import sys
+
+
+PY2 = sys.version_info[0] == 2
+PY3 = not PY2
+
+if PY2:
+    from StringIO import StringIO
+    BytesIO = StringIO
+    from cStringIO import StringIO as cBytesIO
+
+    long_ = long
+    integer_types = (int, long)
+    string_types = (str, unicode)
+    text_type = unicode
+
+    xrange = xrange
+    cmp = cmp
+    chr_ = chr
+
+    def endswith(text, end):
+        return text.endswith(end)
+
+    iteritems = lambda d: d.iteritems()
+    itervalues = lambda d: d.itervalues()
+    iterkeys = lambda d: d.iterkeys()
+
+    iterbytes = lambda b: iter(b)
+
+    exec("def reraise(tp, value, tb):\n raise tp, value, tb")
+
+    def swap_to_string(cls):
+        if "__str__" in cls.__dict__:
+            cls.__unicode__ = cls.__str__
+
+        if "__bytes__" in cls.__dict__:
+            cls.__str__ = cls.__bytes__
+
+        return cls
+
+elif PY3:
+    from io import StringIO
+    StringIO = StringIO
+    from io import BytesIO
+    cBytesIO = BytesIO
+
+    long_ = int
+    integer_types = (int,)
+    string_types = (str,)
+    text_type = str
+
+    xrange = range
+    cmp = lambda a, b: (a > b) - (a < b)
+    chr_ = lambda x: bytes([x])
+
+    def endswith(text, end):
+        # usefull for paths which can be both, str and bytes
+        if isinstance(text, str):
+            if not isinstance(end, str):
+                end = end.decode("ascii")
+        else:
+            if not isinstance(end, bytes):
+                end = end.encode("ascii")
+        return text.endswith(end)
+
+    iteritems = lambda d: iter(d.items())
+    itervalues = lambda d: iter(d.values())
+    iterkeys = lambda d: iter(d.keys())
+
+    iterbytes = lambda b: (bytes([v]) for v in b)
+
+    def reraise(tp, value, tb):
+        raise tp(value).with_traceback(tb)
+
+    def swap_to_string(cls):
+        return cls

lib/mutagen/_constants.py

@@ -1,3 +1,5 @@
+# -*- coding: utf-8 -*-
+
 """Constants used by Mutagen."""
 
 GENRES = [

lib/mutagen/_file.py

@@ -0,0 +1,237 @@
+# Copyright (C) 2005  Michael Urman
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of version 2 of the GNU General Public License as
+# published by the Free Software Foundation.
+
+import warnings
+
+from mutagen._util import DictMixin
+
+
+class FileType(DictMixin):
+    """An abstract object wrapping tags and audio stream information.
+
+    Attributes:
+
+    * info -- stream information (length, bitrate, sample rate)
+    * tags -- metadata tags, if any
+
+    Each file format has different potential tags and stream
+    information.
+
+    FileTypes implement an interface very similar to Metadata; the
+    dict interface, save, load, and delete calls on a FileType call
+    the appropriate methods on its tag data.
+    """
+
+    __module__ = "mutagen"
+
+    info = None
+    tags = None
+    filename = None
+    _mimes = ["application/octet-stream"]
+
+    def __init__(self, filename=None, *args, **kwargs):
+        if filename is None:
+            warnings.warn("FileType constructor requires a filename",
+                          DeprecationWarning)
+        else:
+            self.load(filename, *args, **kwargs)
+
+    def load(self, filename, *args, **kwargs):
+        raise NotImplementedError
+
+    def __getitem__(self, key):
+        """Look up a metadata tag key.
+
+        If the file has no tags at all, a KeyError is raised.
+        """
+
+        if self.tags is None:
+            raise KeyError(key)
+        else:
+            return self.tags[key]
+
+    def __setitem__(self, key, value):
+        """Set a metadata tag.
+
+        If the file has no tags, an appropriate format is added (but
+        not written until save is called).
+        """
+
+        if self.tags is None:
+            self.add_tags()
+        self.tags[key] = value
+
+    def __delitem__(self, key):
+        """Delete a metadata tag key.
+
+        If the file has no tags at all, a KeyError is raised.
+        """
+
+        if self.tags is None:
+            raise KeyError(key)
+        else:
+            del(self.tags[key])
+
+    def keys(self):
+        """Return a list of keys in the metadata tag.
+
+        If the file has no tags at all, an empty list is returned.
+        """
+
+        if self.tags is None:
+            return []
+        else:
+            return self.tags.keys()
+
+    def delete(self, filename=None):
+        """Remove tags from a file."""
+
+        if self.tags is not None:
+            if filename is None:
+                filename = self.filename
+            else:
+                warnings.warn(
+                    "delete(filename=...) is deprecated, reload the file",
+                    DeprecationWarning)
+            return self.tags.delete(filename)
+
+    def save(self, filename=None, **kwargs):
+        """Save metadata tags."""
+
+        if filename is None:
+            filename = self.filename
+        else:
+            warnings.warn(
+                "save(filename=...) is deprecated, reload the file",
+                DeprecationWarning)
+        if self.tags is not None:
+            return self.tags.save(filename, **kwargs)
+        else:
+            raise ValueError("no tags in file")
+
+    def pprint(self):
+        """Print stream information and comment key=value pairs."""
+
+        stream = "%s (%s)" % (self.info.pprint(), self.mime[0])
+        try:
+            tags = self.tags.pprint()
+        except AttributeError:
+            return stream
+        else:
+            return stream + ((tags and "\n" + tags) or "")
+
+    def add_tags(self):
+        """Adds new tags to the file.
+
+        Raises if tags already exist.
+        """
+
+        raise NotImplementedError
+
+    @property
+    def mime(self):
+        """A list of mime types"""
+
+        mimes = []
+        for Kind in type(self).__mro__:
+            for mime in getattr(Kind, '_mimes', []):
+                if mime not in mimes:
+                    mimes.append(mime)
+        return mimes
+
+    @staticmethod
+    def score(filename, fileobj, header):
+        raise NotImplementedError
+
+
+class StreamInfo(object):
+    """Abstract stream information object.
+
+    Provides attributes for length, bitrate, sample rate etc.
+
+    See the implementations for details.
+    """
+
+    __module__ = "mutagen"
+
+    def pprint(self):
+        """Print stream information"""
+
+        raise NotImplementedError
+
+
+def File(filename, options=None, easy=False):
+    """Guess the type of the file and try to open it.
+
+    The file type is decided by several things, such as the first 128
+    bytes (which usually contains a file type identifier), the
+    filename extension, and the presence of existing tags.
+
+    If no appropriate type could be found, None is returned.
+
+    :param options: Sequence of :class:`FileType` implementations, defaults to
+                    all included ones.
+
+    :param easy: If the easy wrappers should be returnd if available.
+                 For example :class:`EasyMP3 <mp3.EasyMP3>` instead
+                 of :class:`MP3 <mp3.MP3>`.
+    """
+
+    if options is None:
+        from mutagen.asf import ASF
+        from mutagen.apev2 import APEv2File
+        from mutagen.flac import FLAC
+        if easy:
+            from mutagen.easyid3 import EasyID3FileType as ID3FileType
+        else:
+            from mutagen.id3 import ID3FileType
+        if easy:
+            from mutagen.mp3 import EasyMP3 as MP3
+        else:
+            from mutagen.mp3 import MP3
+        from mutagen.oggflac import OggFLAC
+        from mutagen.oggspeex import OggSpeex
+        from mutagen.oggtheora import OggTheora
+        from mutagen.oggvorbis import OggVorbis
+        from mutagen.oggopus import OggOpus
+        if easy:
+            from mutagen.trueaudio import EasyTrueAudio as TrueAudio
+        else:
+            from mutagen.trueaudio import TrueAudio
+        from mutagen.wavpack import WavPack
+        if easy:
+            from mutagen.easymp4 import EasyMP4 as MP4
+        else:
+            from mutagen.mp4 import MP4
+        from mutagen.musepack import Musepack
+        from mutagen.monkeysaudio import MonkeysAudio
+        from mutagen.optimfrog import OptimFROG
+        from mutagen.aiff import AIFF
+        from mutagen.aac import AAC
+        options = [MP3, TrueAudio, OggTheora, OggSpeex, OggVorbis, OggFLAC,
+                   FLAC, AIFF, APEv2File, MP4, ID3FileType, WavPack,
+                   Musepack, MonkeysAudio, OptimFROG, ASF, OggOpus, AAC]
+
+    if not options:
+        return None
+
+    fileobj = open(filename, "rb")
+    try:
+        header = fileobj.read(128)
+        # Sort by name after score. Otherwise import order affects
+        # Kind sort order, which affects treatment of things with
+        # equals scores.
+        results = [(Kind.score(filename, fileobj, header), Kind.__name__)
+                   for Kind in options]
+    finally:
+        fileobj.close()
+    results = list(zip(results, options))
+    results.sort()
+    (score, name), Kind = results[-1]
+    if score > 0:
+        return Kind(filename)
+    else:
+        return None

lib/mutagen/_tags.py

@@ -0,0 +1,31 @@
+# Copyright (C) 2005  Michael Urman
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of version 2 of the GNU General Public License as
+# published by the Free Software Foundation.
+
+
+class Metadata(object):
+    """An abstract dict-like object.
+
+    Metadata is the base class for many of the tag objects in Mutagen.
+    """
+
+    __module__ = "mutagen"
+
+    def __init__(self, *args, **kwargs):
+        if args or kwargs:
+            self.load(*args, **kwargs)
+
+    def load(self, *args, **kwargs):
+        raise NotImplementedError
+
+    def save(self, filename=None):
+        """Save changes to a file."""
+
+        raise NotImplementedError
+
+    def delete(self, filename=None):
+        """Remove tags from a file."""
+
+        raise NotImplementedError

lib/mutagen/_util.py

@@ -1,4 +1,6 @@
-# Copyright 2006 Joe Wreschnig
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2006  Joe Wreschnig
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 2 as
@@ -11,10 +13,76 @@ intended for internal use in Mutagen only.
 """
 
 import struct
+import codecs
 
 from fnmatch import fnmatchcase
 
+from ._compat import chr_, text_type, PY2, iteritems, iterbytes, \
+    integer_types, xrange
 
+
+class MutagenError(Exception):
+    """Base class for all custom exceptions in mutagen
+
+    .. versionadded:: 1.25
+    """
+
+
+def total_ordering(cls):
+    assert "__eq__" in cls.__dict__
+    assert "__lt__" in cls.__dict__
+
+    cls.__le__ = lambda self, other: self == other or self < other
+    cls.__gt__ = lambda self, other: not (self == other or self < other)
+    cls.__ge__ = lambda self, other: not self < other
+    cls.__ne__ = lambda self, other: not self.__eq__(other)
+
+    return cls
+
+
+def hashable(cls):
+    """Makes sure the class is hashable.
+
+    Needs a working __eq__ and __hash__ and will add a __ne__.
+    """
+
+    # py2
+    assert "__hash__" in cls.__dict__
+    # py3
+    assert cls.__dict__["__hash__"] is not None
+    assert "__eq__" in cls.__dict__
+
+    cls.__ne__ = lambda self, other: not self.__eq__(other)
+
+    return cls
+
+
+def enum(cls):
+    assert cls.__bases__ == (object,)
+
+    d = dict(cls.__dict__)
+    new_type = type(cls.__name__, (int,), d)
+    new_type.__module__ = cls.__module__
+
+    map_ = {}
+    for key, value in iteritems(d):
+        if key.upper() == key and isinstance(value, integer_types):
+            value_instance = new_type(value)
+            setattr(new_type, key, value_instance)
+            map_[value] = key
+
+    def repr_(self):
+        if self in map_:
+            return "%s.%s" % (type(self).__name__, map_[self])
+        else:
+            return "%s(%s)" % (type(self).__name__, self)
+
+    setattr(new_type, "__repr__", repr_)
+
+    return new_type
+
+
+@total_ordering
 class DictMixin(object):
     """Implement the dict API using keys() and __*item__ methods.
 
@@ -33,27 +101,37 @@ class DictMixin(object):
     def __iter__(self):
         return iter(self.keys())
 
-    def has_key(self, key):
+    def __has_key(self, key):
         try:
             self[key]
         except KeyError:
             return False
         else:
             return True
-    __contains__ = has_key
 
-    iterkeys = lambda self: iter(self.keys())
+    if PY2:
+        has_key = __has_key
+
+    __contains__ = __has_key
+
+    if PY2:
+        iterkeys = lambda self: iter(self.keys())
 
     def values(self):
-        return map(self.__getitem__, self.keys())
-    itervalues = lambda self: iter(self.values())
+        return [self[k] for k in self.keys()]
+
+    if PY2:
+        itervalues = lambda self: iter(self.values())
 
     def items(self):
-        return zip(self.keys(), self.values())
-    iteritems = lambda s: iter(s.items())
+        return list(zip(self.keys(), self.values()))
+
+    if PY2:
+        iteritems = lambda s: iter(s.items())
 
     def clear(self):
-        map(self.__delitem__, self.keys())
+        for key in list(self.keys()):
+            self.__delitem__(key)
 
     def pop(self, key, *args):
         if len(args) > 1:
@@ -69,11 +147,11 @@ class DictMixin(object):
         return value
 
     def popitem(self):
-        try:
-            key = self.keys()[0]
-            return key, self.pop(key)
-        except IndexError:
+        for key in self.keys():
+            break
+        else:
             raise KeyError("dictionary is empty")
+        return key, self.pop(key)
 
     def update(self, other=None, **kwargs):
         if other is None:
@@ -81,7 +159,8 @@ class DictMixin(object):
             other = {}
 
         try:
-            map(self.__setitem__, other.keys(), other.values())
+            for key, value in other.items():
+                self.__setitem__(key, value)
         except AttributeError:
             for key, value in other:
                 self[key] = value
@@ -102,11 +181,11 @@ class DictMixin(object):
     def __repr__(self):
         return repr(dict(self.items()))
 
-    def __cmp__(self, other):
-        if other is None:
-            return 1
-        else:
-            return cmp(dict(self.items()), other)
+    def __eq__(self, other):
+        return dict(self.items()) == other
+
+    def __lt__(self, other):
+        return dict(self.items()) < other
 
     __hash__ = object.__hash__
 
@@ -132,56 +211,63 @@ class DictProxy(DictMixin):
         return self.__dict.keys()
 
 
+def _fill_cdata(cls):
+    """Add struct pack/unpack functions"""
+
+    funcs = {}
+    for key, name in [("b", "char"), ("h", "short"),
+                      ("i", "int"), ("q", "longlong")]:
+        for echar, esuffix in [("<", "le"), (">", "be")]:
+            esuffix = "_" + esuffix
+            for unsigned in [True, False]:
+                s = struct.Struct(echar + (key.upper() if unsigned else key))
+                get_wrapper = lambda f: lambda *a, **k: f(*a, **k)[0]
+                unpack = get_wrapper(s.unpack)
+                unpack_from = get_wrapper(s.unpack_from)
+
+                def get_unpack_from(s):
+                    def unpack_from(data, offset=0):
+                        return s.unpack_from(data, offset)[0], offset + s.size
+                    return unpack_from
+
+                unpack_from = get_unpack_from(s)
+                pack = s.pack
+
+                prefix = "u" if unsigned else ""
+                if s.size == 1:
+                    esuffix = ""
+                bits = str(s.size * 8)
+                funcs["%s%s%s" % (prefix, name, esuffix)] = unpack
+                funcs["%sint%s%s" % (prefix, bits, esuffix)] = unpack
+                funcs["%s%s%s_from" % (prefix, name, esuffix)] = unpack_from
+                funcs["%sint%s%s_from" % (prefix, bits, esuffix)] = unpack_from
+                funcs["to_%s%s%s" % (prefix, name, esuffix)] = pack
+                funcs["to_%sint%s%s" % (prefix, bits, esuffix)] = pack
+
+    for key, func in iteritems(funcs):
+        setattr(cls, key, staticmethod(func))
+
+
 class cdata(object):
-    """C character buffer to Python numeric type conversions."""
+    """C character buffer to Python numeric type conversions.
+
+    For each size/sign/endianness:
+    uint32_le(data)/to_uint32_le(num)/uint32_le_from(data, offset=0)
+    """
 
     from struct import error
     error = error
 
-    short_le = staticmethod(lambda data: struct.unpack('<h', data)[0])
-    ushort_le = staticmethod(lambda data: struct.unpack('<H', data)[0])
-
-    short_be = staticmethod(lambda data: struct.unpack('>h', data)[0])
-    ushort_be = staticmethod(lambda data: struct.unpack('>H', data)[0])
-
-    int_le = staticmethod(lambda data: struct.unpack('<i', data)[0])
-    uint_le = staticmethod(lambda data: struct.unpack('<I', data)[0])
-
-    int_be = staticmethod(lambda data: struct.unpack('>i', data)[0])
-    uint_be = staticmethod(lambda data: struct.unpack('>I', data)[0])
-
-    longlong_le = staticmethod(lambda data: struct.unpack('<q', data)[0])
-    ulonglong_le = staticmethod(lambda data: struct.unpack('<Q', data)[0])
-
-    longlong_be = staticmethod(lambda data: struct.unpack('>q', data)[0])
-    ulonglong_be = staticmethod(lambda data: struct.unpack('>Q', data)[0])
-
-    to_short_le = staticmethod(lambda data: struct.pack('<h', data))
-    to_ushort_le = staticmethod(lambda data: struct.pack('<H', data))
-
-    to_short_be = staticmethod(lambda data: struct.pack('>h', data))
-    to_ushort_be = staticmethod(lambda data: struct.pack('>H', data))
-
-    to_int_le = staticmethod(lambda data: struct.pack('<i', data))
-    to_uint_le = staticmethod(lambda data: struct.pack('<I', data))
-
-    to_int_be = staticmethod(lambda data: struct.pack('>i', data))
-    to_uint_be = staticmethod(lambda data: struct.pack('>I', data))
-
-    to_longlong_le = staticmethod(lambda data: struct.pack('<q', data))
-    to_ulonglong_le = staticmethod(lambda data: struct.pack('<Q', data))
-
-    to_longlong_be = staticmethod(lambda data: struct.pack('>q', data))
-    to_ulonglong_be = staticmethod(lambda data: struct.pack('>Q', data))
-
-    bitswap = ''.join([chr(sum([((val >> i) & 1) << (7-i) for i in range(8)]))
-                       for val in range(256)])
-    del(i)
-    del(val)
+    bitswap = b''.join(
+        chr_(sum(((val >> i) & 1) << (7 - i) for i in range(8)))
+        for val in range(256))
 
     test_bit = staticmethod(lambda value, n: bool((value >> n) & 1))
 
 
+_fill_cdata(cdata)
+
+
 def lock(fileobj):
     """Lock a file object 'safely'.
 
@@ -223,7 +309,7 @@ def unlock(fileobj):
     fcntl.lockf(fileobj, fcntl.LOCK_UN)
 
 
-def insert_bytes(fobj, size, offset, BUFFER_SIZE=2**16):
+def insert_bytes(fobj, size, offset, BUFFER_SIZE=2 ** 16):
     """Insert size bytes of empty space starting at offset.
 
     fobj must be an open file object, open rb+ or
@@ -237,16 +323,16 @@ def insert_bytes(fobj, size, offset, BUFFER_SIZE=2**16):
     fobj.seek(0, 2)
     filesize = fobj.tell()
     movesize = filesize - offset
-    fobj.write('\x00' * size)
+    fobj.write(b'\x00' * size)
     fobj.flush()
     try:
         try:
             import mmap
-            map = mmap.mmap(fobj.fileno(), filesize + size)
+            file_map = mmap.mmap(fobj.fileno(), filesize + size)
             try:
-                map.move(offset + size, offset, movesize)
+                file_map.move(offset + size, offset, movesize)
             finally:
-                map.close()
+                file_map.close()
         except (ValueError, EnvironmentError, ImportError):
             # handle broken mmap scenarios
             locked = lock(fobj)
@@ -258,7 +344,7 @@ def insert_bytes(fobj, size, offset, BUFFER_SIZE=2**16):
             # the file out several megs.
             while padsize:
                 addsize = min(BUFFER_SIZE, padsize)
-                fobj.write("\x00" * addsize)
+                fobj.write(b"\x00" * addsize)
                 padsize -= addsize
 
             fobj.seek(filesize, 0)
@@ -285,7 +371,7 @@ def insert_bytes(fobj, size, offset, BUFFER_SIZE=2**16):
             unlock(fobj)
 
 
-def delete_bytes(fobj, size, offset, BUFFER_SIZE=2**16):
+def delete_bytes(fobj, size, offset, BUFFER_SIZE=2 ** 16):
     """Delete size bytes of empty space starting at offset.
 
     fobj must be an open file object, open rb+ or
@@ -305,11 +391,11 @@ def delete_bytes(fobj, size, offset, BUFFER_SIZE=2**16):
             fobj.flush()
             try:
                 import mmap
-                map = mmap.mmap(fobj.fileno(), filesize)
+                file_map = mmap.mmap(fobj.fileno(), filesize)
                 try:
-                    map.move(offset, offset + size, movesize)
+                    file_map.move(offset, offset + size, movesize)
                 finally:
-                    map.close()
+                    file_map.close()
             except (ValueError, EnvironmentError, ImportError):
                 # handle broken mmap scenarios
                 locked = lock(fobj)
@@ -328,22 +414,190 @@ def delete_bytes(fobj, size, offset, BUFFER_SIZE=2**16):
             unlock(fobj)
 
 
-def utf8(data):
-    """Convert a basestring to a valid UTF-8 str."""
-
-    if isinstance(data, str):
-        return data.decode("utf-8", "replace").encode("utf-8")
-    elif isinstance(data, unicode):
-        return data.encode("utf-8")
-    else:
-        raise TypeError("only unicode/str types can be converted to UTF-8")
-
-
 def dict_match(d, key, default=None):
-    try:
+    """Like __getitem__ but works as if the keys() are all filename patterns.
+    Returns the value of any dict key that matches the passed key.
+    """
+
+    if key in d and "[" not in key:
         return d[key]
-    except KeyError:
-        for pattern, value in d.iteritems():
+    else:
+        for pattern, value in iteritems(d):
             if fnmatchcase(key, pattern):
                 return value
     return default
+
+
+def decode_terminated(data, encoding, strict=True):
+    """Returns the decoded data until the first NULL terminator
+    and all data after it.
+
+    In case the data can't be decoded raises UnicodeError.
+    In case the encoding is not found raises LookupError.
+    In case the data isn't null terminated (even if it is encoded correctly)
+    raises ValueError except if strict is False, then the decoded string
+    will be returned anyway.
+    """
+
+    codec_info = codecs.lookup(encoding)
+
+    # normalize encoding name so we can compare by name
+    encoding = codec_info.name
+
+    # fast path
+    if encoding in ("utf-8", "iso8859-1"):
+        index = data.find(b"\x00")
+        if index == -1:
+            # make sure we raise UnicodeError first, like in the slow path
+            res = data.decode(encoding), b""
+            if strict:
+                raise ValueError("not null terminated")
+            else:
+                return res
+        return data[:index].decode(encoding), data[index + 1:]
+
+    # slow path
+    decoder = codec_info.incrementaldecoder()
+    r = []
+    for i, b in enumerate(iterbytes(data)):
+        c = decoder.decode(b)
+        if c == u"\x00":
+            return u"".join(r), data[i + 1:]
+        r.append(c)
+    else:
+        # make sure the decoder is finished
+        r.append(decoder.decode(b"", True))
+        if strict:
+            raise ValueError("not null terminated")
+        return u"".join(r), b""
+
+
+def split_escape(string, sep, maxsplit=None, escape_char="\\"):
+    """Like unicode/str/bytes.split but allows for the separator to be escaped
+
+    If passed unicode/str/bytes will only return list of unicode/str/bytes.
+    """
+
+    assert len(sep) == 1
+    assert len(escape_char) == 1
+
+    if isinstance(string, bytes):
+        if isinstance(escape_char, text_type):
+            escape_char = escape_char.encode("ascii")
+        iter_ = iterbytes
+    else:
+        iter_ = iter
+
+    if maxsplit is None:
+        maxsplit = len(string)
+
+    empty = string[:0]
+    result = []
+    current = empty
+    escaped = False
+    for char in iter_(string):
+        if escaped:
+            if char != escape_char and char != sep:
+                current += escape_char
+            current += char
+            escaped = False
+        else:
+            if char == escape_char:
+                escaped = True
+            elif char == sep and len(result) < maxsplit:
+                result.append(current)
+                current = empty
+            else:
+                current += char
+    result.append(current)
+    return result
+
+
+class BitReaderError(Exception):
+    pass
+
+
+class BitReader(object):
+
+    def __init__(self, fileobj):
+        self._fileobj = fileobj
+        self._buffer = 0
+        self._bits = 0
+        self._pos = fileobj.tell()
+
+    def bits(self, count):
+        """Reads `count` bits and returns an uint, MSB read first.
+
+        May raise BitReaderError if not enough data could be read or
+        IOError by the underlying file object.
+        """
+
+        if count < 0:
+            raise ValueError
+
+        if count > self._bits:
+            n_bytes = (count - self._bits + 7) // 8
+            data = self._fileobj.read(n_bytes)
+            if len(data) != n_bytes:
+                raise BitReaderError("not enough data")
+            for b in bytearray(data):
+                self._buffer = (self._buffer << 8) | b
+            self._bits += n_bytes * 8
+
+        self._bits -= count
+        value = self._buffer >> self._bits
+        self._buffer &= (1 << self._bits) - 1
+        assert self._bits < 8
+        return value
+
+    def bytes(self, count):
+        """Returns a bytearray of length `count`. Works unaligned."""
+
+        if count < 0:
+            raise ValueError
+
+        # fast path
+        if self._bits == 0:
+            data = self._fileobj.read(count)
+            if len(data) != count:
+                raise BitReaderError("not enough data")
+            return data
+
+        return bytes(bytearray(self.bits(8) for _ in xrange(count)))
+
+    def skip(self, count):
+        """Skip `count` bits.
+
+        Might raise BitReaderError if there wasn't enough data to skip,
+        but might also fail on the next bits() instead.
+        """
+
+        if count < 0:
+            raise ValueError
+
+        if count <= self._bits:
+            self.bits(count)
+        else:
+            count -= self.align()
+            n_bytes = count // 8
+            self._fileobj.seek(n_bytes, 1)
+            count -= n_bytes * 8
+            self.bits(count)
+
+    def get_position(self):
+        """Returns the amount of bits read or skipped so far"""
+
+        return (self._fileobj.tell() - self._pos) * 8 - self._bits
+
+    def align(self):
+        """Align to the next byte, returns the amount of bits skipped"""
+
+        bits = self._bits
+        self._buffer = 0
+        self._bits = 0
+        return bits
+
+    def is_aligned(self):
+        """If we are currently aligned to bytes and nothing is buffered"""
+
+        return self._bits == 0

lib/mutagen/_vorbis.py

@@ -1,5 +1,7 @@
-# Vorbis comment support for Mutagen
-# Copyright 2005-2006 Joe Wreschnig
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2005-2006  Joe Wreschnig
+#                    2013  Christoph Reiter
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of version 2 of the GNU General Public License as
@@ -16,9 +18,8 @@ The specification is at http://www.xiph.org/vorbis/doc/v-comment.html.
 
 import sys
 
-from cStringIO import StringIO
-
 import mutagen
+from ._compat import reraise, BytesIO, text_type, xrange, PY3, PY2
 from mutagen._util import DictMixin, cdata
 
 
@@ -27,13 +28,20 @@ def is_valid_key(key):
 
     Valid Vorbis comment keys are printable ASCII between 0x20 (space)
     and 0x7D ('}'), excluding '='.
+
+    Takes str/unicode in Python 2, unicode in Python 3
     """
+
+    if PY3 and isinstance(key, bytes):
+        raise TypeError("needs to be str not bytes")
+
     for c in key:
         if c < " " or c > "}" or c == "=":
             return False
     else:
         return bool(key)
 
+
 istag = is_valid_key
 
 
@@ -60,7 +68,8 @@ class VComment(mutagen.Metadata, list):
     file-like object, not a filename.
 
     Attributes:
-    vendor -- the stream 'vendor' (i.e. writer); default 'Mutagen'
+
+    * vendor -- the stream 'vendor' (i.e. writer); default 'Mutagen'
     """
 
     vendor = u"Mutagen " + mutagen.version_string
@@ -70,25 +79,26 @@ class VComment(mutagen.Metadata, list):
         # override just load and get equivalent magic for the
         # constructor.
         if data is not None:
-            if isinstance(data, str):
-                data = StringIO(data)
+            if isinstance(data, bytes):
+                data = BytesIO(data)
             elif not hasattr(data, 'read'):
-                raise TypeError("VComment requires string data or a file-like")
+                raise TypeError("VComment requires bytes or a file-like")
             self.load(data, *args, **kwargs)
 
     def load(self, fileobj, errors='replace', framing=True):
         """Parse a Vorbis comment from a file-like object.
 
         Keyword arguments:
-        errors:
-          'strict', 'replace', or 'ignore'. This affects Unicode decoding
-          and how other malformed content is interpreted.
-        framing -- if true, fail if a framing bit is not present
+
+        * errors:
+            'strict', 'replace', or 'ignore'. This affects Unicode decoding
+            and how other malformed content is interpreted.
+        * framing -- if true, fail if a framing bit is not present
 
         Framing bits are required by the Vorbis comment specification,
         but are not used in FLAC Vorbis comment blocks.
-
         """
+
         try:
             vendor_length = cdata.uint_le(fileobj.read(4))
             self.vendor = fileobj.read(vendor_length).decode('utf-8', errors)
@@ -101,21 +111,25 @@ class VComment(mutagen.Metadata, list):
                     raise error("cannot read %d bytes, too large" % length)
                 try:
                     tag, value = string.split('=', 1)
-                except ValueError, err:
+                except ValueError as err:
                     if errors == "ignore":
                         continue
                     elif errors == "replace":
                         tag, value = u"unknown%d" % i, string
                     else:
-                        raise VorbisEncodingError, err, sys.exc_info()[2]
+                        reraise(VorbisEncodingError, err, sys.exc_info()[2])
                 try:
                     tag = tag.encode('ascii', errors)
                 except UnicodeEncodeError:
                     raise VorbisEncodingError("invalid tag name %r" % tag)
                 else:
+                    # string keys in py3k
+                    if PY3:
+                        tag = tag.decode("ascii")
                     if is_valid_key(tag):
                         self.append((tag, value))
-            if framing and not ord(fileobj.read(1)) & 0x01:
+
+            if framing and not bytearray(fileobj.read(1))[0] & 0x01:
                 raise VorbisUnsetFrameError("framing bit was unset")
         except (cdata.error, TypeError):
             raise error("file is not a valid Vorbis comment")
@@ -126,9 +140,14 @@ class VComment(mutagen.Metadata, list):
         Check to make sure every key used is a valid Vorbis key, and
         that every value used is a valid Unicode or UTF-8 string. If
         any invalid keys or values are found, a ValueError is raised.
+
+        In Python 3 all keys and values have to be a string.
         """
 
-        if not isinstance(self.vendor, unicode):
+        if not isinstance(self.vendor, text_type):
+            if PY3:
+                raise ValueError("vendor needs to be str")
+
             try:
                 self.vendor.decode('utf-8')
             except UnicodeDecodeError:
@@ -138,19 +157,25 @@ class VComment(mutagen.Metadata, list):
             try:
                 if not is_valid_key(key):
                     raise ValueError
-            except:
+            except TypeError:
                 raise ValueError("%r is not a valid key" % key)
-            if not isinstance(value, unicode):
+
+            if not isinstance(value, text_type):
+                if PY3:
+                    raise ValueError("%r needs to be str" % key)
+
                 try:
-                    value.encode("utf-8")
+                    value.decode("utf-8")
                 except:
                     raise ValueError("%r is not a valid value" % value)
-        else:
-            return True
+
+        return True
 
     def clear(self):
         """Clear all keys from the comment."""
-        del(self[:])
+
+        for i in list(self):
+            self.remove(i)
 
     def write(self, framing=True):
         """Return a string representation of the data.
@@ -159,25 +184,41 @@ class VComment(mutagen.Metadata, list):
         invalid data may raise a ValueError.
 
         Keyword arguments:
-        framing -- if true, append a framing bit (see load)
+
+        * framing -- if true, append a framing bit (see load)
         """
 
         self.validate()
 
-        f = StringIO()
-        f.write(cdata.to_uint_le(len(self.vendor.encode('utf-8'))))
-        f.write(self.vendor.encode('utf-8'))
+        def _encode(value):
+            if not isinstance(value, bytes):
+                return value.encode('utf-8')
+            return value
+
+        f = BytesIO()
+        vendor = _encode(self.vendor)
+        f.write(cdata.to_uint_le(len(vendor)))
+        f.write(vendor)
         f.write(cdata.to_uint_le(len(self)))
         for tag, value in self:
-            comment = "%s=%s" % (tag, value.encode('utf-8'))
+            tag = _encode(tag)
+            value = _encode(value)
+            comment = tag + b"=" + value
             f.write(cdata.to_uint_le(len(comment)))
             f.write(comment)
         if framing:
-            f.write("\x01")
+            f.write(b"\x01")
         return f.getvalue()
 
     def pprint(self):
-        return "\n".join(["%s=%s" % (k.lower(), v) for k, v in self])
+
+        def _decode(value):
+            if not isinstance(value, text_type):
+                return value.decode('utf-8', 'replace')
+            return value
+
+        tags = [u"%s=%s" % (_decode(k), _decode(v)) for k, v in self]
+        return u"\n".join(tags)
 
 
 class VCommentDict(VComment, DictMixin):
@@ -199,9 +240,17 @@ class VCommentDict(VComment, DictMixin):
 
         This is a copy, so comment['title'].append('a title') will not
         work.
-
         """
-        key = key.lower().encode('ascii')
+
+        # PY3 only
+        if isinstance(key, slice):
+            return VComment.__getitem__(self, key)
+
+        if not is_valid_key(key):
+            raise ValueError
+
+        key = key.lower()
+
         values = [value for (k, value) in self if k.lower() == key]
         if not values:
             raise KeyError(key)
@@ -210,16 +259,29 @@ class VCommentDict(VComment, DictMixin):
 
     def __delitem__(self, key):
         """Delete all values associated with the key."""
-        key = key.lower().encode('ascii')
-        to_delete = filter(lambda x: x[0].lower() == key, self)
+
+        # PY3 only
+        if isinstance(key, slice):
+            return VComment.__delitem__(self, key)
+
+        if not is_valid_key(key):
+            raise ValueError
+
+        key = key.lower()
+        to_delete = [x for x in self if x[0].lower() == key]
         if not to_delete:
             raise KeyError(key)
         else:
-            map(self.remove, to_delete)
+            for item in to_delete:
+                self.remove(item)
 
     def __contains__(self, key):
         """Return true if the key has any values."""
-        key = key.lower().encode('ascii')
+
+        if not is_valid_key(key):
+            raise ValueError
+
+        key = key.lower()
         for k, value in self:
             if k.lower() == key:
                 return True
@@ -232,23 +294,34 @@ class VCommentDict(VComment, DictMixin):
         Setting a value overwrites all old ones. The value may be a
         list of Unicode or UTF-8 strings, or a single Unicode or UTF-8
         string.
-
         """
 
-        key = key.encode('ascii')
+        # PY3 only
+        if isinstance(key, slice):
+            return VComment.__setitem__(self, key, values)
+
+        if not is_valid_key(key):
+            raise ValueError
+
         if not isinstance(values, list):
             values = [values]
         try:
             del(self[key])
         except KeyError:
             pass
+
+        if PY2:
+            key = key.encode('ascii')
+
         for value in values:
             self.append((key, value))
 
     def keys(self):
         """Return all keys in the comment."""
-        return self and list(set([k.lower() for k, v in self]))
+
+        return list(set([k.lower() for k, v in self]))
 
     def as_dict(self):
         """Return a copy of the comment data in a real dict."""
+
         return dict([(key, self[key]) for key in self.keys()])

lib/mutagen/aac.py

@@ -0,0 +1,407 @@
+# -*- coding: utf-8 -*-
+# Copyright (C) 2014 Christoph Reiter
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of version 2 of the GNU General Public License as
+# published by the Free Software Foundation.
+
+"""
+* ADTS - Audio Data Transport Stream
+* ADIF - Audio Data Interchange Format
+* See ISO/IEC 13818-7 / 14496-03
+"""
+
+from mutagen import StreamInfo
+from mutagen._file import FileType
+from mutagen._util import BitReader, BitReaderError, MutagenError
+from mutagen._compat import endswith, xrange
+
+
+_FREQS = [
+    96000, 88200, 64000, 48000,
+    44100, 32000, 24000, 22050,
+    16000, 12000, 11025, 8000,
+    7350,
+]
+
+
+class _ADTSStream(object):
+    """Represents a series of frames belonging to the same stream"""
+
+    parsed_frames = 0
+    """Number of successfully parsed frames"""
+
+    offset = 0
+    """offset in bytes at which the stream starts (the first sync word)"""
+
+    @classmethod
+    def find_stream(cls, fileobj, max_bytes):
+        """Returns a possibly valid _ADTSStream or None.
+
+        Args:
+            max_bytes (int): maximum bytes to read
+        """
+
+        r = BitReader(fileobj)
+        stream = cls(r)
+        if stream.sync(max_bytes):
+            stream.offset = (r.get_position() - 12) // 8
+            return stream
+
+    def sync(self, max_bytes):
+        """Find the next sync.
+        Returns True if found."""
+
+        # at least 2 bytes for the sync
+        max_bytes = max(max_bytes, 2)
+
+        r = self._r
+        r.align()
+        while max_bytes > 0:
+            try:
+                b = r.bytes(1)
+                if b == b"\xff":
+                    if r.bits(4) == 0xf:
+                        return True
+                    r.align()
+                    max_bytes -= 2
+                else:
+                    max_bytes -= 1
+            except BitReaderError:
+                return False
+        return False
+
+    def __init__(self, r):
+        """Use _ADTSStream.find_stream to create a stream"""
+
+        self._fixed_header_key = None
+        self._r = r
+        self.offset = -1
+        self.parsed_frames = 0
+
+        self._samples = 0
+        self._payload = 0
+        self._start = r.get_position() / 8
+        self._last = self._start
+
+    @property
+    def bitrate(self):
+        """Bitrate of the raw aac blocks, excluding framing/crc"""
+
+        assert self.parsed_frames, "no frame parsed yet"
+
+        if self._samples == 0:
+            return 0
+
+        return (8 * self._payload * self.frequency) // self._samples
+
+    @property
+    def samples(self):
+        """samples so far"""
+
+        assert self.parsed_frames, "no frame parsed yet"
+
+        return self._samples
+
+    @property
+    def size(self):
+        """bytes read in the stream so far (including framing)"""
+
+        assert self.parsed_frames, "no frame parsed yet"
+
+        return self._last - self._start
+
+    @property
+    def channels(self):
+        """0 means unknown"""
+
+        assert self.parsed_frames, "no frame parsed yet"
+
+        b_index = self._fixed_header_key[6]
+        if b_index == 7:
+            return 8
+        elif b_index > 7:
+            return 0
+        else:
+            return b_index
+
+    @property
+    def frequency(self):
+        """0 means unknown"""
+
+        assert self.parsed_frames, "no frame parsed yet"
+
+        f_index = self._fixed_header_key[4]
+        try:
+            return _FREQS[f_index]
+        except IndexError:
+            return 0
+
+    def parse_frame(self):
+        """True if parsing was successful.
+        Fails either because the frame wasn't valid or the stream ended.
+        """
+
+        try:
+            return self._parse_frame()
+        except BitReaderError:
+            return False
+
+    def _parse_frame(self):
+        r = self._r
+        # start == position of sync word
+        start = r.get_position() - 12
+
+        # adts_fixed_header
+        id_ = r.bits(1)
+        layer = r.bits(2)
+        protection_absent = r.bits(1)
+
+        profile = r.bits(2)
+        sampling_frequency_index = r.bits(4)
+        private_bit = r.bits(1)
+        # TODO: if 0 we could parse program_config_element()
+        channel_configuration = r.bits(3)
+        original_copy = r.bits(1)
+        home = r.bits(1)
+
+        # the fixed header has to be the same for every frame in the stream
+        fixed_header_key = (
+            id_, layer, protection_absent, profile, sampling_frequency_index,
+            private_bit, channel_configuration, original_copy, home,
+        )
+
+        if self._fixed_header_key is None:
+            self._fixed_header_key = fixed_header_key
+        else:
+            if self._fixed_header_key != fixed_header_key:
+                return False
+
+        # adts_variable_header
+        r.skip(2)  # copyright_identification_bit/start
+        frame_length = r.bits(13)
+        r.skip(11)  # adts_buffer_fullness
+        nordbif = r.bits(2)
+        # adts_variable_header end
+
+        crc_overhead = 0
+        if not protection_absent:
+            crc_overhead += (nordbif + 1) * 16
+            if nordbif != 0:
+                crc_overhead *= 2
+
+        left = (frame_length * 8) - (r.get_position() - start)
+        if left < 0:
+            return False
+        r.skip(left)
+        assert r.is_aligned()
+
+        self._payload += (left - crc_overhead) / 8
+        self._samples += (nordbif + 1) * 1024
+        self._last = r.get_position() / 8
+
+        self.parsed_frames += 1
+        return True
+
+
+class ProgramConfigElement(object):
+
+    element_instance_tag = None
+    object_type = None
+    sampling_frequency_index = None
+    channels = None
+
+    def __init__(self, r):
+        """Reads the program_config_element()
+
+        Raises BitReaderError
+        """
+
+        self.element_instance_tag = r.bits(4)
+        self.object_type = r.bits(2)
+        self.sampling_frequency_index = r.bits(4)
+        num_front_channel_elements = r.bits(4)
+        num_side_channel_elements = r.bits(4)
+        num_back_channel_elements = r.bits(4)
+        num_lfe_channel_elements = r.bits(2)
+        num_assoc_data_elements = r.bits(3)
+        num_valid_cc_elements = r.bits(4)
+
+        mono_mixdown_present = r.bits(1)
+        if mono_mixdown_present == 1:
+            r.skip(4)
+        stereo_mixdown_present = r.bits(1)
+        if stereo_mixdown_present == 1:
+            r.skip(4)
+        matrix_mixdown_idx_present = r.bits(1)
+        if matrix_mixdown_idx_present == 1:
+            r.skip(3)
+
+        elms = num_front_channel_elements + num_side_channel_elements + \
+            num_back_channel_elements
+        channels = 0
+        for i in xrange(elms):
+            channels += 1
+            element_is_cpe = r.bits(1)
+            if element_is_cpe:
+                channels += 1
+            r.skip(4)
+        channels += num_lfe_channel_elements
+        self.channels = channels
+
+        r.skip(4 * num_lfe_channel_elements)
+        r.skip(4 * num_assoc_data_elements)
+        r.skip(5 * num_valid_cc_elements)
+        r.align()
+        comment_field_bytes = r.bits(8)
+        r.skip(8 * comment_field_bytes)
+
+
+class AACError(MutagenError):
+    pass
+
+
+class AACInfo(StreamInfo):
+    """AAC stream information.
+
+    Attributes:
+
+    * channels -- number of audio channels
+    * length -- file length in seconds, as a float
+    * sample_rate -- audio sampling rate in Hz
+    * bitrate -- audio bitrate, in bits per second
+
+    The length of the stream is just a guess and might not be correct.
+    """
+
+    channels = 0
+    length = 0
+    sample_rate = 0
+    bitrate = 0
+
+    def __init__(self, fileobj):
+        # skip id3v2 header
+        start_offset = 0
+        header = fileobj.read(10)
+        from mutagen.id3 import BitPaddedInt
+        if header.startswith(b"ID3"):
+            size = BitPaddedInt(header[6:])
+            start_offset = size + 10
+
+        fileobj.seek(start_offset)
+        adif = fileobj.read(4)
+        if adif == b"ADIF":
+            self._parse_adif(fileobj)
+            self._type = "ADIF"
+        else:
+            self._parse_adts(fileobj, start_offset)
+            self._type = "ADTS"
+
+    def _parse_adif(self, fileobj):
+        r = BitReader(fileobj)
+        try:
+            copyright_id_present = r.bits(1)
+            if copyright_id_present:
+                r.skip(72)  # copyright_id
+            r.skip(1 + 1)  # original_copy, home
+            bitstream_type = r.bits(1)
+            self.bitrate = r.bits(23)
+            npce = r.bits(4)
+            if bitstream_type == 0:
+                r.skip(20)  # adif_buffer_fullness
+
+            pce = ProgramConfigElement(r)
+            try:
+                self.sample_rate = _FREQS[pce.sampling_frequency_index]
+            except IndexError:
+                pass
+            self.channels = pce.channels
+
+            # other pces..
+            for i in xrange(npce):
+                ProgramConfigElement(r)
+            r.align()
+        except BitReaderError as e:
+            raise AACError(e)
+
+        # use bitrate + data size to guess length
+        start = fileobj.tell()
+        fileobj.seek(0, 2)
+        length = fileobj.tell() - start
+        if self.bitrate != 0:
+            self.length = (8.0 * length) / self.bitrate
+
+    def _parse_adts(self, fileobj, start_offset):
+        max_initial_read = 512
+        max_resync_read = 10
+        max_sync_tries = 10
+
+        frames_max = 100
+        frames_needed = 3
+
+        # Try up to X times to find a sync word and read up to Y frames.
+        # If more than Z frames are valid we assume a valid stream
+        offset = start_offset
+        for i in xrange(max_sync_tries):
+            fileobj.seek(offset)
+            s = _ADTSStream.find_stream(fileobj, max_initial_read)
+            if s is None:
+                raise AACError("sync not found")
+            # start right after the last found offset
+            offset += s.offset + 1
+
+            for i in xrange(frames_max):
+                if not s.parse_frame():
+                    break
+                if not s.sync(max_resync_read):
+                    break
+
+            if s.parsed_frames >= frames_needed:
+                break
+        else:
+            raise AACError(
+                "no valid stream found (only %d frames)" % s.parsed_frames)
+
+        self.sample_rate = s.frequency
+        self.channels = s.channels
+        self.bitrate = s.bitrate
+
+        # size from stream start to end of file
+        fileobj.seek(0, 2)
+        stream_size = fileobj.tell() - (offset + s.offset)
+        # approx
+        self.length = float(s.samples * stream_size) / (s.size * s.frequency)
+
+    def pprint(self):
+        return "AAC (%s), %d Hz, %.2f seconds, %d channel(s), %d bps" % (
+            self._type, self.sample_rate, self.length, self.channels,
+            self.bitrate)
+
+
+class AAC(FileType):
+    """Load ADTS or ADIF streams containing AAC.
+
+    Tagging is not supported.
+    Use the ID3/APEv2 classes directly instead.
+    """
+
+    _mimes = ["audio/x-aac"]
+
+    def load(self, filename):
+        self.filename = filename
+        with open(filename, "rb") as h:
+            self.info = AACInfo(h)
+
+    @staticmethod
+    def score(filename, fileobj, header):
+        filename = filename.lower()
+        s = endswith(filename, ".aac") or endswith(filename, ".adts") or \
+            endswith(filename, ".adif")
+        s += b"ADIF" in header
+        return s
+
+
+Open = AAC
+error = AACError
+
+__all__ = ["AAC", "Open"]

lib/mutagen/aiff.py

@@ -0,0 +1,362 @@
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2014  Evan Purkhiser
+#               2014  Ben Ockmore
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of version 2 of the GNU General Public License as
+# published by the Free Software Foundation.
+
+"""AIFF audio stream information and tags."""
+
+# NOTE from Ben Ockmore - according to the Py3k migration guidelines, AIFF
+# chunk keys should be unicode in Py3k, and unicode or bytes in Py2k (ASCII).
+# To make this easier, chunk keys should be stored internally as unicode.
+
+import struct
+from struct import pack
+
+from ._compat import endswith, text_type, PY3
+from mutagen import StreamInfo, FileType
+
+from mutagen.id3 import ID3
+from mutagen.id3._util import error as ID3Error
+from mutagen._util import insert_bytes, delete_bytes, MutagenError
+
+__all__ = ["AIFF", "Open", "delete"]
+
+
+class error(MutagenError, RuntimeError):
+    pass
+
+
+class InvalidChunk(error, IOError):
+    pass
+
+
+# based on stdlib's aifc
+_HUGE_VAL = 1.79769313486231e+308
+
+
+def is_valid_chunk_id(id):
+    if not isinstance(id, text_type):
+        if PY3:
+            raise TypeError("AIFF chunk must be unicode")
+
+        try:
+            id = id.decode('ascii')
+        except UnicodeDecodeError:
+            return False
+
+    return ((len(id) <= 4) and (min(id) >= u' ') and
+            (max(id) <= u'~'))
+
+
+def read_float(data):  # 10 bytes
+    expon, himant, lomant = struct.unpack('>hLL', data)
+    sign = 1
+    if expon < 0:
+        sign = -1
+        expon = expon + 0x8000
+    if expon == himant == lomant == 0:
+        f = 0.0
+    elif expon == 0x7FFF:
+        f = _HUGE_VAL
+    else:
+        expon = expon - 16383
+        f = (himant * 0x100000000 + lomant) * pow(2.0, expon - 63)
+    return sign * f
+
+
+class IFFChunk(object):
+    """Representation of a single IFF chunk"""
+
+    # Chunk headers are 8 bytes long (4 for ID and 4 for the size)
+    HEADER_SIZE = 8
+
+    def __init__(self, fileobj, parent_chunk=None):
+        self.__fileobj = fileobj
+        self.parent_chunk = parent_chunk
+        self.offset = fileobj.tell()
+
+        header = fileobj.read(self.HEADER_SIZE)
+        if len(header) < self.HEADER_SIZE:
+            raise InvalidChunk()
+
+        self.id, self.data_size = struct.unpack('>4si', header)
+
+        if not isinstance(self.id, text_type):
+            self.id = self.id.decode('ascii')
+
+        if not is_valid_chunk_id(self.id):
+            raise InvalidChunk()
+
+        self.size = self.HEADER_SIZE + self.data_size
+        self.data_offset = fileobj.tell()
+        self.data = None
+
+    def read(self):
+        """Read the chunks data"""
+        self.__fileobj.seek(self.data_offset)
+        self.data = self.__fileobj.read(self.data_size)
+
+    def delete(self):
+        """Removes the chunk from the file"""
+        delete_bytes(self.__fileobj, self.size, self.offset)
+        if self.parent_chunk is not None:
+            self.parent_chunk.resize(self.parent_chunk.data_size - self.size)
+
+    def resize(self, data_size):
+        """Update the size of the chunk"""
+        self.__fileobj.seek(self.offset + 4)
+        self.__fileobj.write(pack('>I', data_size))
+        if self.parent_chunk is not None:
+            size_diff = self.data_size - data_size
+            self.parent_chunk.resize(self.parent_chunk.data_size - size_diff)
+        self.data_size = data_size
+        self.size = data_size + self.HEADER_SIZE
+
+
+class IFFFile(object):
+    """Representation of a IFF file"""
+
+    def __init__(self, fileobj):
+        self.__fileobj = fileobj
+        self.__chunks = {}
+
+        # AIFF Files always start with the FORM chunk which contains a 4 byte
+        # ID before the start of other chunks
+        fileobj.seek(0)
+        self.__chunks[u'FORM'] = IFFChunk(fileobj)
+
+        # Skip past the 4 byte FORM id
+        fileobj.seek(IFFChunk.HEADER_SIZE + 4)
+
+        # Where the next chunk can be located. We need to keep track of this
+        # since the size indicated in the FORM header may not match up with the
+        # offset determined from the size of the last chunk in the file
+        self.__next_offset = fileobj.tell()
+
+        # Load all of the chunks
+        while True:
+            try:
+                chunk = IFFChunk(fileobj, self[u'FORM'])
+            except InvalidChunk:
+                break
+            self.__chunks[chunk.id.strip()] = chunk
+
+            # Calculate the location of the next chunk,
+            # considering the pad byte
+            self.__next_offset = chunk.offset + chunk.size
+            self.__next_offset += self.__next_offset % 2
+            fileobj.seek(self.__next_offset)
+
+    def __contains__(self, id_):
+        """Check if the IFF file contains a specific chunk"""
+
+        if not isinstance(id_, text_type):
+            id_ = id_.decode('ascii')
+
+        if not is_valid_chunk_id(id_):
+            raise KeyError("AIFF key must be four ASCII characters.")
+
+        return id_ in self.__chunks
+
+    def __getitem__(self, id_):
+        """Get a chunk from the IFF file"""
+
+        if not isinstance(id_, text_type):
+            id_ = id_.decode('ascii')
+
+        if not is_valid_chunk_id(id_):
+            raise KeyError("AIFF key must be four ASCII characters.")
+
+        try:
+            return self.__chunks[id_]
+        except KeyError:
+            raise KeyError(
+                "%r has no %r chunk" % (self.__fileobj.name, id_))
+
+    def __delitem__(self, id_):
+        """Remove a chunk from the IFF file"""
+
+        if not isinstance(id_, text_type):
+            id_ = id_.decode('ascii')
+
+        if not is_valid_chunk_id(id_):
+            raise KeyError("AIFF key must be four ASCII characters.")
+
+        self.__chunks.pop(id_).delete()
+
+    def insert_chunk(self, id_):
+        """Insert a new chunk at the end of the IFF file"""
+
+        if not isinstance(id_, text_type):
+            id_ = id_.decode('ascii')
+
+        if not is_valid_chunk_id(id_):
+            raise KeyError("AIFF key must be four ASCII characters.")
+
+        self.__fileobj.seek(self.__next_offset)
+        self.__fileobj.write(pack('>4si', id_.ljust(4).encode('ascii'), 0))
+        self.__fileobj.seek(self.__next_offset)
+        chunk = IFFChunk(self.__fileobj, self[u'FORM'])
+        self[u'FORM'].resize(self[u'FORM'].data_size + chunk.size)
+
+        self.__chunks[id_] = chunk
+        self.__next_offset = chunk.offset + chunk.size
+
+
+class AIFFInfo(StreamInfo):
+    """AIFF audio stream information.
+
+    Information is parsed from the COMM chunk of the AIFF file
+
+    Useful attributes:
+
+    * length -- audio length, in seconds
+    * bitrate -- audio bitrate, in bits per second
+    * channels -- The number of audio channels
+    * sample_rate -- audio sample rate, in Hz
+    * sample_size -- The audio sample size
+    """
+
+    length = 0
+    bitrate = 0
+    channels = 0
+    sample_rate = 0
+
+    def __init__(self, fileobj):
+        iff = IFFFile(fileobj)
+        try:
+            common_chunk = iff[u'COMM']
+        except KeyError as e:
+            raise error(str(e))
+
+        common_chunk.read()
+
+        info = struct.unpack('>hLh10s', common_chunk.data[:18])
+        channels, frame_count, sample_size, sample_rate = info
+
+        self.sample_rate = int(read_float(sample_rate))
+        self.sample_size = sample_size
+        self.channels = channels
+        self.bitrate = channels * sample_size * self.sample_rate
+        self.length = frame_count / float(self.sample_rate)
+
+    def pprint(self):
+        return "%d channel AIFF @ %d bps, %s Hz, %.2f seconds" % (
+            self.channels, self.bitrate, self.sample_rate, self.length)
+
+
+class _IFFID3(ID3):
+    """A AIFF file with ID3v2 tags"""
+
+    def _load_header(self):
+        try:
+            self._fileobj.seek(IFFFile(self._fileobj)[u'ID3'].data_offset)
+        except (InvalidChunk, KeyError):
+            raise ID3Error()
+        super(_IFFID3, self)._load_header()
+
+    def save(self, filename=None, v2_version=4, v23_sep='/'):
+        """Save ID3v2 data to the AIFF file"""
+
+        framedata = self._prepare_framedata(v2_version, v23_sep)
+        framesize = len(framedata)
+
+        if filename is None:
+            filename = self.filename
+
+        # Unlike the parent ID3.save method, we won't save to a blank file
+        # since we would have to construct a empty AIFF file
+        fileobj = open(filename, 'rb+')
+        iff_file = IFFFile(fileobj)
+
+        try:
+            if u'ID3' not in iff_file:
+                iff_file.insert_chunk(u'ID3')
+
+            chunk = iff_file[u'ID3']
+            fileobj.seek(chunk.data_offset)
+
+            header = fileobj.read(10)
+            header = self._prepare_id3_header(header, framesize, v2_version)
+            header, new_size, _ = header
+
+            data = header + framedata + (b'\x00' * (new_size - framesize))
+
+            # Include ID3 header size in 'new_size' calculation
+            new_size += 10
+
+            # Expand the chunk if necessary, including pad byte
+            if new_size > chunk.size:
+                insert_at = chunk.offset + chunk.size
+                insert_size = new_size - chunk.size + new_size % 2
+                insert_bytes(fileobj, insert_size, insert_at)
+                chunk.resize(new_size)
+
+            fileobj.seek(chunk.data_offset)
+            fileobj.write(data)
+        finally:
+            fileobj.close()
+
+    def delete(self, filename=None):
+        """Completely removes the ID3 chunk from the AIFF file"""
+
+        if filename is None:
+            filename = self.filename
+        delete(filename)
+        self.clear()
+
+
+def delete(filename):
+    """Completely removes the ID3 chunk from the AIFF file"""
+
+    with open(filename, "rb+") as file_:
+        try:
+            del IFFFile(file_)[u'ID3']
+        except KeyError:
+            pass
+
+
+class AIFF(FileType):
+    """An AIFF audio file.
+
+    :ivar info: :class:`AIFFInfo`
+    :ivar tags: :class:`ID3`
+    """
+
+    _mimes = ["audio/aiff", "audio/x-aiff"]
+
+    @staticmethod
+    def score(filename, fileobj, header):
+        filename = filename.lower()
+
+        return (header.startswith(b"FORM") * 2 + endswith(filename, b".aif") +
+                endswith(filename, b".aiff") + endswith(filename, b".aifc"))
+
+    def add_tags(self):
+        """Add an empty ID3 tag to the file."""
+        if self.tags is None:
+            self.tags = _IFFID3()
+        else:
+            raise error("an ID3 tag already exists")
+
+    def load(self, filename, **kwargs):
+        """Load stream and tag information from a file."""
+        self.filename = filename
+
+        try:
+            self.tags = _IFFID3(filename, **kwargs)
+        except ID3Error:
+            self.tags = None
+
+        try:
+            fileobj = open(filename, "rb")
+            self.info = AIFFInfo(fileobj)
+        finally:
+            fileobj.close()
+
+
+Open = AIFF

lib/mutagen/apev2.py

@@ -1,6 +1,6 @@
-# An APEv2 tag reader
-#
-# Copyright 2005 Joe Wreschnig
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2005  Joe Wreschnig
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 2 as
@@ -30,16 +30,31 @@ http://wiki.hydrogenaudio.org/index.php?title=APEv2_specification.
 
 __all__ = ["APEv2", "APEv2File", "Open", "delete"]
 
+import sys
 import struct
-from cStringIO import StringIO
+from collections import MutableSequence
 
-from mutagen import Metadata, FileType
-from mutagen._util import DictMixin, cdata, utf8, delete_bytes
+from ._compat import (cBytesIO, PY3, text_type, PY2, reraise, swap_to_string,
+                      xrange)
+from mutagen import Metadata, FileType, StreamInfo
+from mutagen._util import (DictMixin, cdata, delete_bytes, total_ordering,
+                           MutagenError)
 
 
 def is_valid_apev2_key(key):
-    return (2 <= len(key) <= 255 and min(key) >= ' ' and max(key) <= '~' and
-            key not in ["OggS", "TAG", "ID3", "MP+"])
+    if not isinstance(key, text_type):
+        if PY3:
+            raise TypeError("APEv2 key must be str")
+
+        try:
+            key = key.decode('ascii')
+        except UnicodeDecodeError:
+            return False
+
+    # PY26 - Change to set literal syntax (since set is faster than list here)
+    return ((2 <= len(key) <= 255) and (min(key) >= u' ') and
+            (max(key) <= u'~') and
+            (key not in [u"OggS", u"TAG", u"ID3", u"MP+"]))
 
 # There are three different kinds of APE tag values.
 # "0: Item contains text information coded in UTF-8
@@ -48,12 +63,12 @@ def is_valid_apev2_key(key):
 #  3: reserved"
 TEXT, BINARY, EXTERNAL = range(3)
 
-HAS_HEADER = 1L << 31
-HAS_NO_FOOTER = 1L << 30
-IS_HEADER = 1L << 29
+HAS_HEADER = 1 << 31
+HAS_NO_FOOTER = 1 << 30
+IS_HEADER = 1 << 29
 
 
-class error(IOError):
+class error(IOError, MutagenError):
     pass
 
 
@@ -89,9 +104,17 @@ class _APEv2Data(object):
 
     def __init__(self, fileobj):
         self.__find_metadata(fileobj)
-        self.metadata = max(self.header, self.footer)
+
+        if self.header is None:
+            self.metadata = self.footer
+        elif self.footer is None:
+            self.metadata = self.header
+        else:
+            self.metadata = max(self.header, self.footer)
+
         if self.metadata is None:
             return
+
         self.__fill_missing(fileobj)
         self.__fix_brokenness(fileobj)
         if self.data is not None:
@@ -107,7 +130,7 @@ class _APEv2Data(object):
         except IOError:
             fileobj.seek(0, 2)
             return
-        if fileobj.read(8) == "APETAGEX":
+        if fileobj.read(8) == b"APETAGEX":
             fileobj.seek(-8, 1)
             self.footer = self.metadata = fileobj.tell()
             return
@@ -115,10 +138,10 @@ class _APEv2Data(object):
         # Check for an APEv2 tag followed by an ID3v1 tag at the end.
         try:
             fileobj.seek(-128, 2)
-            if fileobj.read(3) == "TAG":
+            if fileobj.read(3) == b"TAG":
 
                 fileobj.seek(-35, 1)  # "TAG" + header length
-                if fileobj.read(8) == "APETAGEX":
+                if fileobj.read(8) == b"APETAGEX":
                     fileobj.seek(-8, 1)
                     self.footer = fileobj.tell()
                     return
@@ -127,7 +150,7 @@ class _APEv2Data(object):
                 # (http://www.id3.org/lyrics3200.html)
                 # (header length - "APETAGEX") - "LYRICS200"
                 fileobj.seek(15, 1)
-                if fileobj.read(9) == 'LYRICS200':
+                if fileobj.read(9) == b'LYRICS200':
                     fileobj.seek(-15, 1)  # "LYRICS200" + size tag
                     try:
                         offset = int(fileobj.read(6))
@@ -135,7 +158,7 @@ class _APEv2Data(object):
                         raise IOError
 
                     fileobj.seek(-32 - offset - 6, 1)
-                    if fileobj.read(8) == "APETAGEX":
+                    if fileobj.read(8) == b"APETAGEX":
                         fileobj.seek(-8, 1)
                         self.footer = fileobj.tell()
                         return
@@ -145,7 +168,7 @@ class _APEv2Data(object):
 
         # Check for a tag at the start.
         fileobj.seek(0, 0)
-        if fileobj.read(8) == "APETAGEX":
+        if fileobj.read(8) == b"APETAGEX":
             self.is_at_start = True
             self.header = 0
 
@@ -162,7 +185,7 @@ class _APEv2Data(object):
             # offset + the size, which includes the footer.
             self.end = self.data + self.size
             fileobj.seek(self.end - 32, 0)
-            if fileobj.read(8) == "APETAGEX":
+            if fileobj.read(8) == b"APETAGEX":
                 self.footer = self.end - 32
         elif self.footer is not None:
             self.end = self.footer + 32
@@ -194,7 +217,7 @@ class _APEv2Data(object):
             except IOError:
                 break
             else:
-                if fileobj.read(8) == "APETAGEX":
+                if fileobj.read(8) == b"APETAGEX":
                     fileobj.seek(-8, 1)
                     start = fileobj.tell()
                 else:
@@ -202,18 +225,12 @@ class _APEv2Data(object):
         self.start = start
 
 
-class APEv2(DictMixin, Metadata):
-    """A file with an APEv2 tag.
-
-    ID3v1 tags are silently ignored and overwritten.
-    """
-
-    filename = None
+class _CIDictProxy(DictMixin):
 
     def __init__(self, *args, **kwargs):
         self.__casemap = {}
         self.__dict = {}
-        super(APEv2, self).__init__(*args, **kwargs)
+        super(_CIDictProxy, self).__init__(*args, **kwargs)
         # Internally all names are stored as lowercase, but the case
         # they were set with is remembered and used when saving.  This
         # is roughly in line with the standard, which says that keys
@@ -221,14 +238,40 @@ class APEv2(DictMixin, Metadata):
         # not allowed, and recommends case-insensitive
         # implementations.
 
+    def __getitem__(self, key):
+        return self.__dict[key.lower()]
+
+    def __setitem__(self, key, value):
+        lower = key.lower()
+        self.__casemap[lower] = key
+        self.__dict[lower] = value
+
+    def __delitem__(self, key):
+        lower = key.lower()
+        del(self.__casemap[lower])
+        del(self.__dict[lower])
+
+    def keys(self):
+        return [self.__casemap.get(key, key) for key in self.__dict.keys()]
+
+
+class APEv2(_CIDictProxy, Metadata):
+    """A file with an APEv2 tag.
+
+    ID3v1 tags are silently ignored and overwritten.
+    """
+
+    filename = None
+
     def pprint(self):
         """Return tag key=value pairs in a human-readable format."""
-        items = self.items()
-        items.sort()
-        return "\n".join(["%s=%s" % (k, v.pprint()) for k, v in items])
+
+        items = sorted(self.items())
+        return u"\n".join(u"%s=%s" % (k, v.pprint()) for k, v in items)
 
     def load(self, filename):
         """Load tags from a filename."""
+
         self.filename = filename
         fileobj = open(filename, "rb")
         try:
@@ -237,15 +280,14 @@ class APEv2(DictMixin, Metadata):
             fileobj.close()
         if data.tag:
             self.clear()
-            self.__casemap.clear()
             self.__parse_tag(data.tag, data.items)
         else:
             raise APENoHeaderError("No APE tag found")
 
     def __parse_tag(self, tag, count):
-        fileobj = StringIO(tag)
+        fileobj = cBytesIO(tag)
 
-        for i in range(count):
+        for i in xrange(count):
             size_data = fileobj.read(4)
             # someone writes wrong item counts
             if not size_data:
@@ -259,25 +301,37 @@ class APEv2(DictMixin, Metadata):
             if kind == 3:
                 raise APEBadItemError("value type must be 0, 1, or 2")
             key = value = fileobj.read(1)
-            while key[-1:] != '\x00' and value:
+            while key[-1:] != b'\x00' and value:
                 value = fileobj.read(1)
                 key += value
-            if key[-1:] == "\x00":
+            if key[-1:] == b"\x00":
                 key = key[:-1]
+            if PY3:
+                try:
+                    key = key.decode("ascii")
+                except UnicodeError as err:
+                    reraise(APEBadItemError, err, sys.exc_info()[2])
             value = fileobj.read(size)
-            self[key] = APEValue(value, kind)
+
+            value = _get_value_type(kind)._new(value)
+
+            self[key] = value
 
     def __getitem__(self, key):
         if not is_valid_apev2_key(key):
             raise KeyError("%r is not a valid APEv2 key" % key)
-        key = key.encode('ascii')
-        return self.__dict[key.lower()]
+        if PY2:
+            key = key.encode('ascii')
+
+        return super(APEv2, self).__getitem__(key)
 
     def __delitem__(self, key):
         if not is_valid_apev2_key(key):
             raise KeyError("%r is not a valid APEv2 key" % key)
-        key = key.encode('ascii')
-        del(self.__dict[key.lower()])
+        if PY2:
+            key = key.encode('ascii')
+
+        super(APEv2, self).__delitem__(key)
 
     def __setitem__(self, key, value):
         """'Magic' value setter.
@@ -288,6 +342,9 @@ class APEv2(DictMixin, Metadata):
         as a list of string/Unicode values.  If you pass in a string
         that is not valid UTF-8, it assumes it is a binary value.
 
+        Python 3: all bytes will be assumed to be a byte value, even
+        if they are valid utf-8.
+
         If you need to force a specific type of value (e.g. binary
         data that also happens to be valid UTF-8, or an external
         reference), use the APEValue factory and set the value to the
@@ -299,30 +356,40 @@ class APEv2(DictMixin, Metadata):
 
         if not is_valid_apev2_key(key):
             raise KeyError("%r is not a valid APEv2 key" % key)
-        key = key.encode('ascii')
+
+        if PY2:
+            key = key.encode('ascii')
 
         if not isinstance(value, _APEValue):
             # let's guess at the content if we're not already a value...
-            if isinstance(value, unicode):
+            if isinstance(value, text_type):
                 # unicode? we've got to be text.
-                value = APEValue(utf8(value), TEXT)
+                value = APEValue(value, TEXT)
             elif isinstance(value, list):
+                items = []
+                for v in value:
+                    if not isinstance(v, text_type):
+                        if PY3:
+                            raise TypeError("item in list not str")
+                        v = v.decode("utf-8")
+                    items.append(v)
+
                 # list? text.
-                value = APEValue("\0".join(map(utf8, value)), TEXT)
+                value = APEValue(u"\0".join(items), TEXT)
             else:
-                try:
-                    value.decode("utf-8")
-                except UnicodeError:
-                    # invalid UTF8 text, probably binary
+                if PY3:
                     value = APEValue(value, BINARY)
                 else:
-                    # valid UTF8, probably text
-                    value = APEValue(value, TEXT)
-        self.__casemap[key.lower()] = key
-        self.__dict[key.lower()] = value
+                    try:
+                        value.decode("utf-8")
+                    except UnicodeError:
+                        # invalid UTF8 text, probably binary
+                        value = APEValue(value, BINARY)
+                    else:
+                        # valid UTF8, probably text
+                        value = APEValue(value, TEXT)
 
-    def keys(self):
-        return [self.__casemap.get(key, key) for key in self.__dict.keys()]
+        super(APEv2, self).__setitem__(key, value)
 
     def save(self, filename=None):
         """Save changes to a file.
@@ -348,33 +415,49 @@ class APEv2(DictMixin, Metadata):
             fileobj.truncate()
         fileobj.seek(0, 2)
 
+        tags = []
+        for key, value in self.items():
+            # Packed format for an item:
+            # 4B: Value length
+            # 4B: Value type
+            # Key name
+            # 1B: Null
+            # Key value
+            value_data = value._write()
+            if not isinstance(key, bytes):
+                key = key.encode("utf-8")
+            tag_data = bytearray()
+            tag_data += struct.pack("<2I", len(value_data), value.kind << 1)
+            tag_data += key + b"\0" + value_data
+            tags.append(bytes(tag_data))
+
         # "APE tags items should be sorted ascending by size... This is
         # not a MUST, but STRONGLY recommended. Actually the items should
         # be sorted by importance/byte, but this is not feasible."
-        tags = [v._internal(k) for k, v in self.items()]
-        tags.sort(lambda a, b: cmp(len(a), len(b)))
+        tags.sort(key=len)
         num_tags = len(tags)
-        tags = "".join(tags)
+        tags = b"".join(tags)
 
-        header = "APETAGEX%s%s" % (
-            # version, tag size, item count, flags
-            struct.pack("<4I", 2000, len(tags) + 32, num_tags,
-                        HAS_HEADER | IS_HEADER),
-            "\0" * 8)
+        header = bytearray(b"APETAGEX")
+        # version, tag size, item count, flags
+        header += struct.pack("<4I", 2000, len(tags) + 32, num_tags,
+                              HAS_HEADER | IS_HEADER)
+        header += b"\0" * 8
         fileobj.write(header)
 
         fileobj.write(tags)
 
-        footer = "APETAGEX%s%s" % (
-            # version, tag size, item count, flags
-            struct.pack("<4I", 2000, len(tags) + 32, num_tags,
-                        HAS_HEADER),
-            "\0" * 8)
+        footer = bytearray(b"APETAGEX")
+        footer += struct.pack("<4I", 2000, len(tags) + 32, num_tags,
+                              HAS_HEADER)
+        footer += b"\0" * 8
+
         fileobj.write(footer)
         fileobj.close()
 
     def delete(self, filename=None):
         """Remove tags from a file."""
+
         filename = filename or self.filename
         fileobj = open(filename, "r+b")
         try:
@@ -391,105 +474,212 @@ Open = APEv2
 
 def delete(filename):
     """Remove tags from a file."""
+
     try:
         APEv2(filename).delete()
     except APENoHeaderError:
         pass
 
 
+def _get_value_type(kind):
+    """Returns a _APEValue subclass or raises ValueError"""
+
+    if kind == TEXT:
+        return APETextValue
+    elif kind == BINARY:
+        return APEBinaryValue
+    elif kind == EXTERNAL:
+        return APEExtValue
+    raise ValueError("unknown kind %r" % kind)
+
+
 def APEValue(value, kind):
     """APEv2 tag value factory.
 
     Use this if you need to specify the value's type manually.  Binary
     and text data are automatically detected by APEv2.__setitem__.
     """
-    if kind == TEXT:
-        return APETextValue(value, kind)
-    elif kind == BINARY:
-        return APEBinaryValue(value, kind)
-    elif kind == EXTERNAL:
-        return APEExtValue(value, kind)
-    else:
+
+    try:
+        type_ = _get_value_type(kind)
+    except ValueError:
         raise ValueError("kind must be TEXT, BINARY, or EXTERNAL")
+    else:
+        return type_(value)
 
 
 class _APEValue(object):
-    def __init__(self, value, kind):
-        self.kind = kind
-        self.value = value
 
-    def __len__(self):
-        return len(self.value)
+    kind = None
+    value = None
 
-    def __str__(self):
-        return self.value
+    def __init__(self, value, kind=None):
+        # kind kwarg is for backwards compat
+        if kind is not None and kind != self.kind:
+            raise ValueError
+        self.value = self._validate(value)
 
-    # Packed format for an item:
-    # 4B: Value length
-    # 4B: Value type
-    # Key name
-    # 1B: Null
-    # Key value
-    def _internal(self, key):
-        return "%s%s\0%s" % (
-            struct.pack("<2I", len(self.value), self.kind << 1),
-            key, self.value)
+    @classmethod
+    def _new(cls, data):
+        instance = cls.__new__(cls)
+        instance._parse(data)
+        return instance
+
+    def _parse(self, data):
+        """Sets value or raises APEBadItemError"""
+
+        raise NotImplementedError
+
+    def _write(self):
+        """Returns bytes"""
+
+        raise NotImplementedError
+
+    def _validate(self, value):
+        """Returns validated value or raises TypeError/ValueErrr"""
+
+        raise NotImplementedError
 
     def __repr__(self):
         return "%s(%r, %d)" % (type(self).__name__, self.value, self.kind)
 
 
-class APETextValue(_APEValue):
+@swap_to_string
+@total_ordering
+class _APEUtf8Value(_APEValue):
+
+    def _parse(self, data):
+        try:
+            self.value = data.decode("utf-8")
+        except UnicodeDecodeError as e:
+            reraise(APEBadItemError, e, sys.exc_info()[2])
+
+    def _validate(self, value):
+        if not isinstance(value, text_type):
+            if PY3:
+                raise TypeError("value not str")
+            else:
+                value = value.decode("utf-8")
+        return value
+
+    def _write(self):
+        return self.value.encode("utf-8")
+
+    def __len__(self):
+        return len(self.value)
+
+    def __bytes__(self):
+        return self._write()
+
+    def __eq__(self, other):
+        return self.value == other
+
+    def __lt__(self, other):
+        return self.value < other
+
+    def __str__(self):
+        return self.value
+
+
+class APETextValue(_APEUtf8Value, MutableSequence):
     """An APEv2 text value.
 
     Text values are Unicode/UTF-8 strings. They can be accessed like
-    strings (with a null seperating the values), or arrays of strings."""
+    strings (with a null separating the values), or arrays of strings.
+    """
 
-    def __unicode__(self):
-        return unicode(str(self), "utf-8")
+    kind = TEXT
 
     def __iter__(self):
         """Iterate over the strings of the value (not the characters)"""
-        return iter(unicode(self).split("\0"))
+
+        return iter(self.value.split(u"\0"))
 
     def __getitem__(self, index):
-        return unicode(self).split("\0")[index]
+        return self.value.split(u"\0")[index]
 
     def __len__(self):
-        return self.value.count("\0") + 1
-
-    def __cmp__(self, other):
-        return cmp(unicode(self), other)
-
-    __hash__ = _APEValue.__hash__
+        return self.value.count(u"\0") + 1
 
     def __setitem__(self, index, value):
+        if not isinstance(value, text_type):
+            if PY3:
+                raise TypeError("value not str")
+            else:
+                value = value.decode("utf-8")
+
         values = list(self)
-        values[index] = value.encode("utf-8")
-        self.value = "\0".join(values).encode("utf-8")
+        values[index] = value
+        self.value = u"\0".join(values)
+
+    def insert(self, index, value):
+        if not isinstance(value, text_type):
+            if PY3:
+                raise TypeError("value not str")
+            else:
+                value = value.decode("utf-8")
+
+        values = list(self)
+        values.insert(index, value)
+        self.value = u"\0".join(values)
+
+    def __delitem__(self, index):
+        values = list(self)
+        del values[index]
+        self.value = u"\0".join(values)
 
     def pprint(self):
-        return " / ".join(self)
+        return u" / ".join(self)
 
 
+@swap_to_string
+@total_ordering
 class APEBinaryValue(_APEValue):
     """An APEv2 binary value."""
 
+    kind = BINARY
+
+    def _parse(self, data):
+        self.value = data
+
+    def _write(self):
+        return self.value
+
+    def _validate(self, value):
+        if not isinstance(value, bytes):
+            raise TypeError("value not bytes")
+        return bytes(value)
+
+    def __len__(self):
+        return len(self.value)
+
+    def __bytes__(self):
+        return self._write()
+
+    def __eq__(self, other):
+        return self.value == other
+
+    def __lt__(self, other):
+        return self.value < other
+
     def pprint(self):
-        return "[%d bytes]" % len(self)
+        return u"[%d bytes]" % len(self)
 
 
-class APEExtValue(_APEValue):
+class APEExtValue(_APEUtf8Value):
     """An APEv2 external value.
 
     External values are usually URI or IRI strings.
     """
+
+    kind = EXTERNAL
+
     def pprint(self):
-        return "[External] %s" % unicode(self)
+        return u"[External] %s" % self.value
 
 
 class APEv2File(FileType):
-    class _Info(object):
+    class _Info(StreamInfo):
         length = 0
         bitrate = 0
 
@@ -498,14 +688,14 @@ class APEv2File(FileType):
 
         @staticmethod
         def pprint():
-            return "Unknown format with APEv2 tag."
+            return u"Unknown format with APEv2 tag."
 
     def load(self, filename):
         self.filename = filename
         self.info = self._Info(open(filename, "rb"))
         try:
             self.tags = APEv2(filename)
-        except error:
+        except APENoHeaderError:
             self.tags = None
 
     def add_tags(self):
@@ -521,5 +711,4 @@ class APEv2File(FileType):
         except IOError:
             fileobj.seek(0)
         footer = fileobj.read()
-        filename = filename.lower()
-        return (("APETAGEX" in footer) - header.startswith("ID3"))
+        return ((b"APETAGEX" in footer) - header.startswith(b"ID3"))

lib/mutagen/asf.py

@@ -1,5 +1,8 @@
-# Copyright 2006-2007 Lukas Lalinsky
-# Copyright 2005-2006 Joe Wreschnig
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2005-2006  Joe Wreschnig
+# Copyright (C) 2006-2007  Lukas Lalinsky
+
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 2 as
@@ -9,12 +12,16 @@
 
 __all__ = ["ASF", "Open"]
 
+import sys
 import struct
-from mutagen import FileType, Metadata
-from mutagen._util import insert_bytes, delete_bytes, DictMixin
+from mutagen import FileType, Metadata, StreamInfo
+from mutagen._util import (insert_bytes, delete_bytes, DictMixin,
+                           total_ordering, MutagenError)
+from ._compat import swap_to_string, text_type, PY2, string_types, reraise, \
+    xrange, long_, PY3
 
 
-class error(IOError):
+class error(IOError, MutagenError):
     pass
 
 
@@ -26,7 +33,7 @@ class ASFHeaderError(error):
     pass
 
 
-class ASFInfo(object):
+class ASFInfo(StreamInfo):
     """ASF stream information."""
 
     def __init__(self):
@@ -45,7 +52,7 @@ class ASFTags(list, DictMixin, Metadata):
     """Dictionary containing ASF attributes."""
 
     def pprint(self):
-        return "\n".join(["%s=%s" % (k, v) for k, v in self])
+        return "\n".join("%s=%s" % (k, v) for k, v in self)
 
     def __getitem__(self, key):
         """A list of values for the key.
@@ -54,6 +61,11 @@ class ASFTags(list, DictMixin, Metadata):
         work.
 
         """
+
+        # PY3 only
+        if isinstance(key, slice):
+            return list.__getitem__(self, key)
+
         values = [value for (k, value) in self if k == key]
         if not values:
             raise KeyError(key)
@@ -62,11 +74,17 @@ class ASFTags(list, DictMixin, Metadata):
 
     def __delitem__(self, key):
         """Delete all values associated with the key."""
-        to_delete = filter(lambda x: x[0] == key, self)
+
+        # PY3 only
+        if isinstance(key, slice):
+            return list.__delitem__(self, key)
+
+        to_delete = [x for x in self if x[0] == key]
         if not to_delete:
             raise KeyError(key)
         else:
-            map(self.remove, to_delete)
+            for k in to_delete:
+                self.remove(k)
 
     def __contains__(self, key):
         """Return true if the key has any values."""
@@ -84,29 +102,41 @@ class ASFTags(list, DictMixin, Metadata):
         string.
 
         """
+
+        # PY3 only
+        if isinstance(key, slice):
+            return list.__setitem__(self, key, values)
+
         if not isinstance(values, list):
             values = [values]
-        try:
-            del(self[key])
-        except KeyError:
-            pass
+
+        to_append = []
         for value in values:
-            if key in _standard_attribute_names:
-                value = unicode(value)
-            elif not isinstance(value, ASFBaseAttribute):
-                if isinstance(value, basestring):
+            if not isinstance(value, ASFBaseAttribute):
+                if isinstance(value, string_types):
                     value = ASFUnicodeAttribute(value)
+                elif PY3 and isinstance(value, bytes):
+                    value = ASFByteArrayAttribute(value)
                 elif isinstance(value, bool):
                     value = ASFBoolAttribute(value)
                 elif isinstance(value, int):
                     value = ASFDWordAttribute(value)
-                elif isinstance(value, long):
+                elif isinstance(value, long_):
                     value = ASFQWordAttribute(value)
-            self.append((key, value))
+                else:
+                    raise TypeError("Invalid type %r" % type(value))
+            to_append.append((key, value))
+
+        try:
+            del(self[key])
+        except KeyError:
+            pass
+
+        self.extend(to_append)
 
     def keys(self):
         """Return all keys in the comment."""
-        return self and set(zip(*self)[0])
+        return self and set(next(iter(zip(*self))))
 
     def as_dict(self):
         """Return a copy of the comment data in a real dict."""
@@ -127,7 +157,19 @@ class ASFBaseAttribute(object):
         if data:
             self.value = self.parse(data, **kwargs)
         else:
-            self.value = value
+            if value is None:
+                # we used to support not passing any args and instead assign
+                # them later, keep that working..
+                self.value = None
+            else:
+                self.value = self._validate(value)
+
+    def _validate(self, value):
+        """Raises TypeError or ValueError in case the user supplied value
+        isn't valid.
+        """
+
+        return value
 
     def data_size(self):
         raise NotImplementedError
@@ -142,13 +184,13 @@ class ASFBaseAttribute(object):
         return name
 
     def render(self, name):
-        name = name.encode("utf-16-le") + "\x00\x00"
+        name = name.encode("utf-16-le") + b"\x00\x00"
         data = self._render()
         return (struct.pack("<H", len(name)) + name +
                 struct.pack("<HH", self.TYPE, len(data)) + data)
 
     def render_m(self, name):
-        name = name.encode("utf-16-le") + "\x00\x00"
+        name = name.encode("utf-16-le") + b"\x00\x00"
         if self.TYPE == 2:
             data = self._render(dword=False)
         else:
@@ -157,59 +199,96 @@ class ASFBaseAttribute(object):
                             self.TYPE, len(data)) + name + data)
 
     def render_ml(self, name):
-        name = name.encode("utf-16-le") + "\x00\x00"
+        name = name.encode("utf-16-le") + b"\x00\x00"
         if self.TYPE == 2:
             data = self._render(dword=False)
         else:
             data = self._render()
+
         return (struct.pack("<HHHHI", self.language or 0, self.stream or 0,
                             len(name), self.TYPE, len(data)) + name + data)
 
 
+@swap_to_string
+@total_ordering
 class ASFUnicodeAttribute(ASFBaseAttribute):
     """Unicode string attribute."""
     TYPE = 0x0000
 
     def parse(self, data):
-        return data.decode("utf-16-le").strip("\x00")
+        try:
+            return data.decode("utf-16-le").strip("\x00")
+        except UnicodeDecodeError as e:
+            reraise(ASFError, e, sys.exc_info()[2])
+
+    def _validate(self, value):
+        if not isinstance(value, text_type):
+            if PY2:
+                return value.decode("utf-8")
+            else:
+                raise TypeError("%r not str" % value)
+        return value
 
     def _render(self):
-        return self.value.encode("utf-16-le") + "\x00\x00"
+        return self.value.encode("utf-16-le") + b"\x00\x00"
 
     def data_size(self):
         return len(self._render())
 
+    def __bytes__(self):
+        return self.value.encode("utf-16-le")
+
     def __str__(self):
         return self.value
 
-    def __cmp__(self, other):
-        return cmp(unicode(self), other)
+    def __eq__(self, other):
+        return text_type(self) == other
+
+    def __lt__(self, other):
+        return text_type(self) < other
 
     __hash__ = ASFBaseAttribute.__hash__
 
 
+@swap_to_string
+@total_ordering
 class ASFByteArrayAttribute(ASFBaseAttribute):
     """Byte array attribute."""
     TYPE = 0x0001
 
     def parse(self, data):
+        assert isinstance(data, bytes)
         return data
 
     def _render(self):
+        assert isinstance(self.value, bytes)
         return self.value
 
+    def _validate(self, value):
+        if not isinstance(value, bytes):
+            raise TypeError("must be bytes/str: %r" % value)
+        return value
+
     def data_size(self):
         return len(self.value)
 
-    def __str__(self):
-        return "[binary data (%s bytes)]" % len(self.value)
+    def __bytes__(self):
+        return self.value
 
-    def __cmp__(self, other):
-        return cmp(str(self), other)
+    def __str__(self):
+        return "[binary data (%d bytes)]" % len(self.value)
+
+    def __eq__(self, other):
+        return self.value == other
+
+    def __lt__(self, other):
+        return self.value < other
 
     __hash__ = ASFBaseAttribute.__hash__
 
 
+@swap_to_string
+@total_ordering
 class ASFBoolAttribute(ASFBaseAttribute):
     """Bool attribute."""
     TYPE = 0x0002
@@ -222,25 +301,36 @@ class ASFBoolAttribute(ASFBaseAttribute):
 
     def _render(self, dword=True):
         if dword:
-            return struct.pack("<I", int(self.value))
+            return struct.pack("<I", bool(self.value))
         else:
-            return struct.pack("<H", int(self.value))
+            return struct.pack("<H", bool(self.value))
+
+    def _validate(self, value):
+        return bool(value)
 
     def data_size(self):
         return 4
 
     def __bool__(self):
-        return self.value
+        return bool(self.value)
+
+    def __bytes__(self):
+        return text_type(self.value).encode('utf-8')
 
     def __str__(self):
-        return str(self.value)
+        return text_type(self.value)
 
-    def __cmp__(self, other):
-        return cmp(bool(self), other)
+    def __eq__(self, other):
+        return bool(self.value) == other
+
+    def __lt__(self, other):
+        return bool(self.value) < other
 
     __hash__ = ASFBaseAttribute.__hash__
 
 
+@swap_to_string
+@total_ordering
 class ASFDWordAttribute(ASFBaseAttribute):
     """DWORD attribute."""
     TYPE = 0x0003
@@ -251,21 +341,35 @@ class ASFDWordAttribute(ASFBaseAttribute):
     def _render(self):
         return struct.pack("<L", self.value)
 
+    def _validate(self, value):
+        value = int(value)
+        if not 0 <= value <= 2 ** 32 - 1:
+            raise ValueError("Out of range")
+        return value
+
     def data_size(self):
         return 4
 
     def __int__(self):
         return self.value
 
-    def __str__(self):
-        return str(self.value)
+    def __bytes__(self):
+        return text_type(self.value).encode('utf-8')
 
-    def __cmp__(self, other):
-        return cmp(int(self), other)
+    def __str__(self):
+        return text_type(self.value)
+
+    def __eq__(self, other):
+        return int(self.value) == other
+
+    def __lt__(self, other):
+        return int(self.value) < other
 
     __hash__ = ASFBaseAttribute.__hash__
 
 
+@swap_to_string
+@total_ordering
 class ASFQWordAttribute(ASFBaseAttribute):
     """QWORD attribute."""
     TYPE = 0x0004
@@ -276,21 +380,35 @@ class ASFQWordAttribute(ASFBaseAttribute):
     def _render(self):
         return struct.pack("<Q", self.value)
 
+    def _validate(self, value):
+        value = int(value)
+        if not 0 <= value <= 2 ** 64 - 1:
+            raise ValueError("Out of range")
+        return value
+
     def data_size(self):
         return 8
 
     def __int__(self):
         return self.value
 
-    def __str__(self):
-        return str(self.value)
+    def __bytes__(self):
+        return text_type(self.value).encode('utf-8')
 
-    def __cmp__(self, other):
-        return cmp(int(self), other)
+    def __str__(self):
+        return text_type(self.value)
+
+    def __eq__(self, other):
+        return int(self.value) == other
+
+    def __lt__(self, other):
+        return int(self.value) < other
 
     __hash__ = ASFBaseAttribute.__hash__
 
 
+@swap_to_string
+@total_ordering
 class ASFWordAttribute(ASFBaseAttribute):
     """WORD attribute."""
     TYPE = 0x0005
@@ -301,39 +419,66 @@ class ASFWordAttribute(ASFBaseAttribute):
     def _render(self):
         return struct.pack("<H", self.value)
 
+    def _validate(self, value):
+        value = int(value)
+        if not 0 <= value <= 2 ** 16 - 1:
+            raise ValueError("Out of range")
+        return value
+
     def data_size(self):
         return 2
 
     def __int__(self):
         return self.value
 
-    def __str__(self):
-        return str(self.value)
+    def __bytes__(self):
+        return text_type(self.value).encode('utf-8')
 
-    def __cmp__(self, other):
-        return cmp(int(self), other)
+    def __str__(self):
+        return text_type(self.value)
+
+    def __eq__(self, other):
+        return int(self.value) == other
+
+    def __lt__(self, other):
+        return int(self.value) < other
 
     __hash__ = ASFBaseAttribute.__hash__
 
 
+@swap_to_string
+@total_ordering
 class ASFGUIDAttribute(ASFBaseAttribute):
     """GUID attribute."""
     TYPE = 0x0006
 
     def parse(self, data):
+        assert isinstance(data, bytes)
         return data
 
     def _render(self):
+        assert isinstance(self.value, bytes)
         return self.value
 
+    def _validate(self, value):
+        if not isinstance(value, bytes):
+            raise TypeError("must be bytes/str: %r" % value)
+        return value
+
     def data_size(self):
         return len(self.value)
 
-    def __str__(self):
+    def __bytes__(self):
         return self.value
 
-    def __cmp__(self, other):
-        return cmp(str(self), other)
+    def __str__(self):
+        return repr(self.value)
+
+    def __eq__(self, other):
+        return self.value == other
+
+    def __lt__(self, other):
+        return self.value < other
 
     __hash__ = ASFBaseAttribute.__hash__
 
@@ -348,10 +493,12 @@ GUID = ASFGUIDAttribute.TYPE
 
 
 def ASFValue(value, kind, **kwargs):
-    for t, c in _attribute_types.items():
-        if kind == t:
-            return c(value=value, **kwargs)
-    raise ValueError("Unknown value type")
+    try:
+        attr_type = _attribute_types[kind]
+    except KeyError:
+        raise ValueError("Unknown value type %r" % kind)
+    else:
+        return attr_type(value=value, **kwargs)
 
 
 _attribute_types = {
@@ -365,15 +512,6 @@ _attribute_types = {
 }
 
 
-_standard_attribute_names = [
-    "Title",
-    "Author",
-    "Copyright",
-    "Description",
-    "Rating"
-]
-
-
 class BaseObject(object):
     """Base ASF object."""
     GUID = None
@@ -389,17 +527,26 @@ class BaseObject(object):
 class UnknownObject(BaseObject):
     """Unknown ASF object."""
     def __init__(self, guid):
+        assert isinstance(guid, bytes)
         self.GUID = guid
 
 
 class HeaderObject(object):
     """ASF header."""
-    GUID = "\x30\x26\xB2\x75\x8E\x66\xCF\x11\xA6\xD9\x00\xAA\x00\x62\xCE\x6C"
+    GUID = b"\x30\x26\xB2\x75\x8E\x66\xCF\x11\xA6\xD9\x00\xAA\x00\x62\xCE\x6C"
 
 
 class ContentDescriptionObject(BaseObject):
     """Content description."""
-    GUID = "\x33\x26\xB2\x75\x8E\x66\xCF\x11\xA6\xD9\x00\xAA\x00\x62\xCE\x6C"
+    GUID = b"\x33\x26\xB2\x75\x8E\x66\xCF\x11\xA6\xD9\x00\xAA\x00\x62\xCE\x6C"
+
+    NAMES = [
+        u"Title",
+        u"Author",
+        u"Copyright",
+        u"Description",
+        u"Rating",
+    ]
 
     def parse(self, asf, data, fileobj, size):
         super(ContentDescriptionObject, self).parse(asf, data, fileobj, size)
@@ -410,36 +557,32 @@ class ContentDescriptionObject(BaseObject):
         for length in lengths:
             end = pos + length
             if length > 0:
-                texts.append(data[pos:end].decode("utf-16-le").strip("\x00"))
+                texts.append(data[pos:end].decode("utf-16-le").strip(u"\x00"))
             else:
                 texts.append(None)
             pos = end
-        title, author, copyright, desc, rating = texts
-        for key, value in dict(
-            Title=title,
-            Author=author,
-            Copyright=copyright,
-            Description=desc,
-            Rating=rating
-        ).items():
+
+        for key, value in zip(self.NAMES, texts):
             if value is not None:
-                asf.tags[key] = value
+                value = ASFUnicodeAttribute(value=value)
+                asf._tags.setdefault(self.GUID, []).append((key, value))
 
     def render(self, asf):
         def render_text(name):
-            value = asf.tags.get(name, [])
-            if value:
-                return value[0].encode("utf-16-le") + "\x00\x00"
+            value = asf.to_content_description.get(name)
+            if value is not None:
+                return text_type(value).encode("utf-16-le") + b"\x00\x00"
             else:
-                return ""
-        texts = map(render_text, _standard_attribute_names)
-        data = struct.pack("<HHHHH", *map(len, texts)) + "".join(texts)
+                return b""
+
+        texts = [render_text(x) for x in self.NAMES]
+        data = struct.pack("<HHHHH", *map(len, texts)) + b"".join(texts)
         return self.GUID + struct.pack("<Q", 24 + len(data)) + data
 
 
 class ExtendedContentDescriptionObject(BaseObject):
     """Extended content description."""
-    GUID = "\x40\xA4\xD0\xD2\x07\xE3\xD2\x11\x97\xF0\x00\xA0\xC9\x5E\xA8\x50"
+    GUID = b"\x40\xA4\xD0\xD2\x07\xE3\xD2\x11\x97\xF0\x00\xA0\xC9\x5E\xA8\x50"
 
     def parse(self, asf, data, fileobj, size):
         super(ExtendedContentDescriptionObject, self).parse(
@@ -447,38 +590,39 @@ class ExtendedContentDescriptionObject(BaseObject):
         asf.extended_content_description_obj = self
         num_attributes, = struct.unpack("<H", data[0:2])
         pos = 2
-        for i in range(num_attributes):
-            name_length, = struct.unpack("<H", data[pos:pos+2])
+        for i in xrange(num_attributes):
+            name_length, = struct.unpack("<H", data[pos:pos + 2])
             pos += 2
-            name = data[pos:pos+name_length].decode("utf-16-le").strip("\x00")
+            name = data[pos:pos + name_length]
+            name = name.decode("utf-16-le").strip("\x00")
             pos += name_length
-            value_type, value_length = struct.unpack("<HH", data[pos:pos+4])
+            value_type, value_length = struct.unpack("<HH", data[pos:pos + 4])
             pos += 4
-            value = data[pos:pos+value_length]
+            value = data[pos:pos + value_length]
             pos += value_length
             attr = _attribute_types[value_type](data=value)
-            asf.tags.append((name, attr))
+            asf._tags.setdefault(self.GUID, []).append((name, attr))
 
     def render(self, asf):
         attrs = asf.to_extended_content_description.items()
-        data = "".join([attr.render(name) for (name, attr) in attrs])
+        data = b"".join(attr.render(name) for (name, attr) in attrs)
         data = struct.pack("<QH", 26 + len(data), len(attrs)) + data
         return self.GUID + data
 
 
 class FilePropertiesObject(BaseObject):
     """File properties."""
-    GUID = "\xA1\xDC\xAB\x8C\x47\xA9\xCF\x11\x8E\xE4\x00\xC0\x0C\x20\x53\x65"
+    GUID = b"\xA1\xDC\xAB\x8C\x47\xA9\xCF\x11\x8E\xE4\x00\xC0\x0C\x20\x53\x65"
 
     def parse(self, asf, data, fileobj, size):
         super(FilePropertiesObject, self).parse(asf, data, fileobj, size)
         length, _, preroll = struct.unpack("<QQQ", data[40:64])
-        asf.info.length = length / 10000000.0 - preroll / 1000.0
+        asf.info.length = (length / 10000000.0) - (preroll / 1000.0)
 
 
 class StreamPropertiesObject(BaseObject):
     """Stream properties."""
-    GUID = "\x91\x07\xDC\xB7\xB7\xA9\xCF\x11\x8E\xE6\x00\xC0\x0C\x20\x53\x65"
+    GUID = b"\x91\x07\xDC\xB7\xB7\xA9\xCF\x11\x8E\xE6\x00\xC0\x0C\x20\x53\x65"
 
     def parse(self, asf, data, fileobj, size):
         super(StreamPropertiesObject, self).parse(asf, data, fileobj, size)
@@ -490,7 +634,7 @@ class StreamPropertiesObject(BaseObject):
 
 class HeaderExtensionObject(BaseObject):
     """Header extension."""
-    GUID = "\xb5\x03\xbf_.\xa9\xcf\x11\x8e\xe3\x00\xc0\x0c Se"
+    GUID = b"\xb5\x03\xbf_.\xa9\xcf\x11\x8e\xe3\x00\xc0\x0c Se"
 
     def parse(self, asf, data, fileobj, size):
         super(HeaderExtensionObject, self).parse(asf, data, fileobj, size)
@@ -499,79 +643,83 @@ class HeaderExtensionObject(BaseObject):
         datapos = 0
         self.objects = []
         while datapos < datasize:
-            guid, size = struct.unpack("<16sQ", data[22+datapos:22+datapos+24])
+            guid, size = struct.unpack(
+                "<16sQ", data[22 + datapos:22 + datapos + 24])
             if guid in _object_types:
                 obj = _object_types[guid]()
             else:
                 obj = UnknownObject(guid)
-            obj.parse(asf, data[22+datapos+24:22+datapos+size], fileobj, size)
+            obj.parse(asf, data[22 + datapos + 24:22 + datapos + size],
+                      fileobj, size)
             self.objects.append(obj)
             datapos += size
 
     def render(self, asf):
-        data = "".join([obj.render(asf) for obj in self.objects])
+        data = b"".join(obj.render(asf) for obj in self.objects)
         return (self.GUID + struct.pack("<Q", 24 + 16 + 6 + len(data)) +
-                "\x11\xD2\xD3\xAB\xBA\xA9\xcf\x11" +
-                "\x8E\xE6\x00\xC0\x0C\x20\x53\x65" +
-                "\x06\x00" + struct.pack("<I", len(data)) + data)
+                b"\x11\xD2\xD3\xAB\xBA\xA9\xcf\x11" +
+                b"\x8E\xE6\x00\xC0\x0C\x20\x53\x65" +
+                b"\x06\x00" + struct.pack("<I", len(data)) + data)
 
 
 class MetadataObject(BaseObject):
     """Metadata description."""
-    GUID = "\xea\xcb\xf8\xc5\xaf[wH\x84g\xaa\x8cD\xfaL\xca"
+    GUID = b"\xea\xcb\xf8\xc5\xaf[wH\x84g\xaa\x8cD\xfaL\xca"
 
     def parse(self, asf, data, fileobj, size):
         super(MetadataObject, self).parse(asf, data, fileobj, size)
         asf.metadata_obj = self
         num_attributes, = struct.unpack("<H", data[0:2])
         pos = 2
-        for i in range(num_attributes):
+        for i in xrange(num_attributes):
             (reserved, stream, name_length, value_type,
-             value_length) = struct.unpack("<HHHHI", data[pos:pos+12])
+             value_length) = struct.unpack("<HHHHI", data[pos:pos + 12])
             pos += 12
-            name = data[pos:pos+name_length].decode("utf-16-le").strip("\x00")
+            name = data[pos:pos + name_length]
+            name = name.decode("utf-16-le").strip("\x00")
             pos += name_length
-            value = data[pos:pos+value_length]
+            value = data[pos:pos + value_length]
             pos += value_length
             args = {'data': value, 'stream': stream}
             if value_type == 2:
                 args['dword'] = False
             attr = _attribute_types[value_type](**args)
-            asf.tags.append((name, attr))
+            asf._tags.setdefault(self.GUID, []).append((name, attr))
 
     def render(self, asf):
         attrs = asf.to_metadata.items()
-        data = "".join([attr.render_m(name) for (name, attr) in attrs])
+        data = b"".join([attr.render_m(name) for (name, attr) in attrs])
         return (self.GUID + struct.pack("<QH", 26 + len(data), len(attrs)) +
                 data)
 
 
 class MetadataLibraryObject(BaseObject):
     """Metadata library description."""
-    GUID = "\x94\x1c#D\x98\x94\xd1I\xa1A\x1d\x13NEpT"
+    GUID = b"\x94\x1c#D\x98\x94\xd1I\xa1A\x1d\x13NEpT"
 
     def parse(self, asf, data, fileobj, size):
         super(MetadataLibraryObject, self).parse(asf, data, fileobj, size)
         asf.metadata_library_obj = self
         num_attributes, = struct.unpack("<H", data[0:2])
         pos = 2
-        for i in range(num_attributes):
+        for i in xrange(num_attributes):
             (language, stream, name_length, value_type,
-             value_length) = struct.unpack("<HHHHI", data[pos:pos+12])
+             value_length) = struct.unpack("<HHHHI", data[pos:pos + 12])
             pos += 12
-            name = data[pos:pos+name_length].decode("utf-16-le").strip("\x00")
+            name = data[pos:pos + name_length]
+            name = name.decode("utf-16-le").strip("\x00")
             pos += name_length
-            value = data[pos:pos+value_length]
+            value = data[pos:pos + value_length]
             pos += value_length
             args = {'data': value, 'language': language, 'stream': stream}
             if value_type == 2:
                 args['dword'] = False
             attr = _attribute_types[value_type](**args)
-            asf.tags.append((name, attr))
+            asf._tags.setdefault(self.GUID, []).append((name, attr))
 
     def render(self, asf):
         attrs = asf.to_metadata_library
-        data = "".join([attr.render_ml(name) for (name, attr) in attrs])
+        data = b"".join([attr.render_ml(name) for (name, attr) in attrs])
         return (self.GUID + struct.pack("<QH", 26 + len(data), len(attrs)) +
                 data)
 
@@ -595,8 +743,7 @@ class ASF(FileType):
 
     def load(self, filename):
         self.filename = filename
-        fileobj = open(filename, "rb")
-        try:
+        with open(filename, "rb") as fileobj:
             self.size = 0
             self.size1 = 0
             self.size2 = 0
@@ -606,27 +753,34 @@ class ASF(FileType):
             self.info = ASFInfo()
             self.tags = ASFTags()
             self.__read_file(fileobj)
-        finally:
-            fileobj.close()
 
     def save(self):
         # Move attributes to the right objects
+        self.to_content_description = {}
         self.to_extended_content_description = {}
         self.to_metadata = {}
         self.to_metadata_library = []
         for name, value in self.tags:
-            if name in _standard_attribute_names:
-                continue
             library_only = (value.data_size() > 0xFFFF or value.TYPE == GUID)
-            if (value.language is None and value.stream is None and
-                    name not in self.to_extended_content_description and
-                    not library_only):
-                self.to_extended_content_description[name] = value
-            elif (value.language is None and value.stream is not None and
-                  name not in self.to_metadata and not library_only):
-                self.to_metadata[name] = value
-            else:
+            can_cont_desc = value.TYPE == UNICODE
+
+            if library_only or value.language is not None:
                 self.to_metadata_library.append((name, value))
+            elif value.stream is not None:
+                if name not in self.to_metadata:
+                    self.to_metadata[name] = value
+                else:
+                    self.to_metadata_library.append((name, value))
+            elif name in ContentDescriptionObject.NAMES:
+                if name not in self.to_content_description and can_cont_desc:
+                    self.to_content_description[name] = value
+                else:
+                    self.to_metadata_library.append((name, value))
+            else:
+                if name not in self.to_extended_content_description:
+                    self.to_extended_content_description[name] = value
+                else:
+                    self.to_metadata_library.append((name, value))
 
         # Add missing objects
         if not self.content_description_obj:
@@ -651,13 +805,12 @@ class ASF(FileType):
             self.header_extension_obj.objects.append(self.metadata_library_obj)
 
         # Render the header
-        data = "".join([obj.render(self) for obj in self.objects])
+        data = b"".join([obj.render(self) for obj in self.objects])
         data = (HeaderObject.GUID +
                 struct.pack("<QL", len(data) + 30, len(self.objects)) +
-                "\x01\x02" + data)
+                b"\x01\x02" + data)
 
-        fileobj = open(self.filename, "rb+")
-        try:
+        with open(self.filename, "rb+") as fileobj:
             size = len(data)
             if size > self.size:
                 insert_bytes(fileobj, size - self.size, self.size)
@@ -665,8 +818,6 @@ class ASF(FileType):
                 delete_bytes(fileobj, self.size - size, 0)
             fileobj.seek(0)
             fileobj.write(data)
-        finally:
-            fileobj.close()
 
         self.size = size
         self.num_objects = len(self.objects)
@@ -684,9 +835,16 @@ class ASF(FileType):
 
         self.size, self.num_objects = struct.unpack("<QL", header[16:28])
         self.objects = []
-        for i in range(self.num_objects):
+        self._tags = {}
+        for i in xrange(self.num_objects):
             self.__read_object(fileobj)
 
+        for guid in [ContentDescriptionObject.GUID,
+                ExtendedContentDescriptionObject.GUID, MetadataObject.GUID,
+                MetadataLibraryObject.GUID]:
+            self.tags.extend(self._tags.pop(guid, []))
+        assert not self._tags
+
     def __read_object(self, fileobj):
         guid, size = struct.unpack("<16sQ", fileobj.read(24))
         if guid in _object_types:

lib/mutagen/easyid3.py

@@ -1,5 +1,6 @@
-# Simpler (but far more limited) API for ID3 editing
-# Copyright 2006 Joe Wreschnig
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2006  Joe Wreschnig
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of version 2 of the GNU General Public License as
@@ -13,6 +14,7 @@ more like Vorbis or APEv2 tags.
 
 import mutagen.id3
 
+from ._compat import iteritems, text_type, PY2
 from mutagen import Metadata
 from mutagen._util import DictMixin, dict_match
 from mutagen.id3 import ID3, error, delete, ID3FileType
@@ -154,6 +156,8 @@ class EasyID3(DictMixin, Metadata):
                 for v in value:
                     if v and max(v) > u'\x7f':
                         enc = 3
+                        break
+
                 id3.add(mutagen.id3.TXXX(encoding=enc, text=value, desc=desc))
             else:
                 frame.text = value
@@ -171,8 +175,10 @@ class EasyID3(DictMixin, Metadata):
     load = property(lambda s: s.__id3.load,
                     lambda s, v: setattr(s.__id3, 'load', v))
 
-    save = property(lambda s: s.__id3.save,
-                    lambda s, v: setattr(s.__id3, 'save', v))
+    def save(self, *args, **kwargs):
+        # ignore v2_version until we support 2.3 here
+        kwargs.pop("v2_version", None)
+        self.__id3.save(*args, **kwargs)
 
     delete = property(lambda s: s.__id3.delete,
                       lambda s, v: setattr(s.__id3, 'delete', v))
@@ -193,8 +199,12 @@ class EasyID3(DictMixin, Metadata):
 
     def __setitem__(self, key, value):
         key = key.lower()
-        if isinstance(value, basestring):
-            value = [value]
+        if PY2:
+            if isinstance(value, basestring):
+                value = [value]
+        else:
+            if isinstance(value, text_type):
+                value = [value]
         func = dict_match(self.Set, key, self.SetFallback)
         if func is not None:
             return func(self.__id3, key, value)
@@ -263,6 +273,18 @@ def date_delete(id3, key):
     del(id3["TDRC"])
 
 
+def original_date_get(id3, key):
+    return [stamp.text for stamp in id3["TDOR"].text]
+
+
+def original_date_set(id3, key, value):
+    id3.add(mutagen.id3.TDOR(encoding=3, text=value))
+
+
+def original_date_delete(id3, key):
+    del(id3["TDOR"])
+
+
 def performer_get(id3, key):
     people = []
     wanted_role = key.split(":", 1)[1]
@@ -433,7 +455,7 @@ def peakgain_list(id3, key):
         keys.append("replaygain_%s_peak" % frame.desc)
     return keys
 
-for frameid, key in {
+for frameid, key in iteritems({
     "TALB": "album",
     "TBPM": "bpm",
     "TCMP": "compilation",  # iTunes extension
@@ -461,18 +483,20 @@ for frameid, key in {
     "TSOT": "titlesort",
     "TSRC": "isrc",
     "TSST": "discsubtitle",
-}.iteritems():
+    "TLAN": "language",
+}):
     EasyID3.RegisterTextKey(key, frameid)
 
 EasyID3.RegisterKey("genre", genre_get, genre_set, genre_delete)
 EasyID3.RegisterKey("date", date_get, date_set, date_delete)
+EasyID3.RegisterKey("originaldate", original_date_get, original_date_set,
+                    original_date_delete)
 EasyID3.RegisterKey(
     "performer:*", performer_get, performer_set, performer_delete,
     performer_list)
 EasyID3.RegisterKey("musicbrainz_trackid", musicbrainz_trackid_get,
                     musicbrainz_trackid_set, musicbrainz_trackid_delete)
 EasyID3.RegisterKey("website", website_get, website_set, website_delete)
-EasyID3.RegisterKey("website", website_get, website_set, website_delete)
 EasyID3.RegisterKey(
     "replaygain_*_gain", gain_get, gain_set, gain_delete, peakgain_list)
 EasyID3.RegisterKey("replaygain_*_peak", peak_get, peak_set, peak_delete)
@@ -481,7 +505,7 @@ EasyID3.RegisterKey("replaygain_*_peak", peak_get, peak_set, peak_delete)
 # http://musicbrainz.org/docs/specs/metadata_tags.html
 # http://bugs.musicbrainz.org/ticket/1383
 # http://musicbrainz.org/doc/MusicBrainzTag
-for desc, key in {
+for desc, key in iteritems({
     u"MusicBrainz Artist Id": "musicbrainz_artistid",
     u"MusicBrainz Album Id": "musicbrainz_albumid",
     u"MusicBrainz Album Artist Id": "musicbrainz_albumartistid",
@@ -495,7 +519,13 @@ for desc, key in {
     u"ASIN": "asin",
     u"ALBUMARTISTSORT": "albumartistsort",
     u"BARCODE": "barcode",
-}.iteritems():
+    u"CATALOGNUMBER": "catalognumber",
+    u"MusicBrainz Release Track Id": "musicbrainz_releasetrackid",
+    u"MusicBrainz Release Group Id": "musicbrainz_releasegroupid",
+    u"MusicBrainz Work Id": "musicbrainz_workid",
+    u"Acoustid Fingerprint": "acoustid_fingerprint",
+    u"Acoustid Id": "acoustid_id",
+}):
     EasyID3.RegisterTXXXKey(key, desc)
 
 

lib/mutagen/easymp4.py

@@ -1,12 +1,16 @@
-# Copyright 2009 Joe Wreschnig
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2009  Joe Wreschnig
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of version 2 of the GNU General Public License as
 # published by the Free Software Foundation.
 
 from mutagen import Metadata
-from mutagen._util import DictMixin, dict_match, utf8
+from mutagen._util import DictMixin, dict_match
 from mutagen.mp4 import MP4, MP4Tags, error, delete
+from ._compat import PY2, text_type, PY3
+
 
 __all__ = ["EasyMP4Tags", "EasyMP4", "delete", "error"]
 
@@ -91,16 +95,16 @@ class EasyMP4Tags(DictMixin, Metadata):
         cls.RegisterKey(key, getter, setter, deleter)
 
     @classmethod
-    def RegisterIntKey(cls, key, atomid, min_value=0, max_value=2**16-1):
+    def RegisterIntKey(cls, key, atomid, min_value=0, max_value=(2 ** 16) - 1):
         """Register a scalar integer key.
         """
 
         def getter(tags, key):
-            return map(unicode, tags[atomid])
+            return list(map(text_type, tags[atomid]))
 
         def setter(tags, key, value):
             clamp = lambda x: int(min(max(min_value, x), max_value))
-            tags[atomid] = map(clamp, map(int, value))
+            tags[atomid] = [clamp(v) for v in map(int, value)]
 
         def deleter(tags, key):
             del(tags[atomid])
@@ -108,14 +112,15 @@ class EasyMP4Tags(DictMixin, Metadata):
         cls.RegisterKey(key, getter, setter, deleter)
 
     @classmethod
-    def RegisterIntPairKey(cls, key, atomid, min_value=0, max_value=2**16-1):
+    def RegisterIntPairKey(cls, key, atomid, min_value=0,
+                           max_value=(2 ** 16) - 1):
         def getter(tags, key):
             ret = []
             for (track, total) in tags[atomid]:
                 if total:
                     ret.append(u"%d/%d" % (track, total))
                 else:
-                    ret.append(unicode(track))
+                    ret.append(text_type(track))
             return ret
 
         def setter(tags, key, value):
@@ -148,13 +153,20 @@ class EasyMP4Tags(DictMixin, Metadata):
             EasyMP4Tags.RegisterFreeformKey(
                 "musicbrainz_artistid", "MusicBrainz Artist Id")
         """
-        atomid = "----:%s:%s" % (mean, name)
+        atomid = "----:" + mean + ":" + name
 
         def getter(tags, key):
             return [s.decode("utf-8", "replace") for s in tags[atomid]]
 
         def setter(tags, key, value):
-            tags[atomid] = map(utf8, value)
+            encoded = []
+            for v in value:
+                if not isinstance(v, text_type):
+                    if PY3:
+                        raise TypeError("%r not str" % v)
+                    v = v.decode("utf-8")
+                encoded.append(v.encode("utf-8"))
+            tags[atomid] = encoded
 
         def deleter(tags, key):
             del(tags[atomid])
@@ -171,8 +183,14 @@ class EasyMP4Tags(DictMixin, Metadata):
 
     def __setitem__(self, key, value):
         key = key.lower()
-        if isinstance(value, basestring):
-            value = [value]
+
+        if PY2:
+            if isinstance(value, basestring):
+                value = [value]
+        else:
+            if isinstance(value, text_type):
+                value = [value]
+
         func = dict_match(self.Set, key)
         if func is not None:
             return func(self.__mp4, key, value)

lib/mutagen/flac.py

@@ -1,5 +1,6 @@
-# FLAC comment support for Mutagen
-# Copyright 2005 Joe Wreschnig
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2005  Joe Wreschnig
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of version 2 of the GNU General Public License as
@@ -22,17 +23,16 @@ http://flac.sourceforge.net/format.html
 __all__ = ["FLAC", "Open", "delete"]
 
 import struct
-from cStringIO import StringIO
-from _vorbis import VCommentDict
-from mutagen import FileType
-from mutagen._util import insert_bytes
+from ._vorbis import VCommentDict
+import mutagen
+
+from ._compat import cBytesIO, endswith, chr_
+from mutagen._util import insert_bytes, MutagenError
 from mutagen.id3 import BitPaddedInt
-import sys
-if sys.version_info >= (2, 6):
-    from functools import reduce
+from functools import reduce
 
 
-class error(IOError):
+class error(IOError, MutagenError):
     pass
 
 
@@ -44,10 +44,10 @@ class FLACVorbisError(ValueError, error):
     pass
 
 
-def to_int_be(string):
+def to_int_be(data):
     """Convert an arbitrarily-long string to a long using big-endian
     byte order."""
-    return reduce(lambda a, b: (a << 8) + ord(b), string, 0L)
+    return reduce(lambda a, b: (a << 8) + b, bytearray(data), 0)
 
 
 class StrictFileObject(object):
@@ -89,8 +89,8 @@ class MetadataBlock(object):
         The metadata header should not be included."""
         if data is not None:
             if not isinstance(data, StrictFileObject):
-                if isinstance(data, str):
-                    data = StringIO(data)
+                if isinstance(data, bytes):
+                    data = cBytesIO(data)
                 elif not hasattr(data, 'read'):
                     raise TypeError(
                         "StreamInfo requires string data or a file-like")
@@ -110,30 +110,33 @@ class MetadataBlock(object):
         codes = [[block.code, block.write()] for block in blocks]
         codes[-1][0] |= 128
         for code, datum in codes:
-            byte = chr(code)
-            if len(datum) > 2**24:
+            byte = chr_(code)
+            if len(datum) > 2 ** 24:
                 raise error("block is too long to write")
             length = struct.pack(">I", len(datum))[-3:]
             data.append(byte + length + datum)
-        return "".join(data)
+        return b"".join(data)
 
     @staticmethod
     def group_padding(blocks):
         """Consolidate FLAC padding metadata blocks.
 
         The overall size of the rendered blocks does not change, so
-        this adds several bytes of padding for each merged block."""
-        paddings = filter(lambda x: isinstance(x, Padding), blocks)
-        map(blocks.remove, paddings)
+        this adds several bytes of padding for each merged block.
+        """
+
+        paddings = [b for b in blocks if isinstance(b, Padding)]
+        for p in paddings:
+            blocks.remove(p)
         # total padding size is the sum of padding sizes plus 4 bytes
         # per removed header.
-        size = sum([padding.length for padding in paddings])
+        size = sum(padding.length for padding in paddings)
         padding = Padding()
         padding.length = size + 4 * (len(paddings) - 1)
         blocks.append(padding)
 
 
-class StreamInfo(MetadataBlock):
+class StreamInfo(MetadataBlock, mutagen.StreamInfo):
     """FLAC stream information.
 
     This contains information about the audio data in the FLAC file.
@@ -188,13 +191,13 @@ class StreamInfo(MetadataBlock):
         bps_tail = bps_total >> 36
         bps_head = (sample_channels_bps & 1) << 4
         self.bits_per_sample = int(bps_head + bps_tail + 1)
-        self.total_samples = bps_total & 0xFFFFFFFFFL
+        self.total_samples = bps_total & 0xFFFFFFFFF
         self.length = self.total_samples / float(self.sample_rate)
 
         self.md5_signature = to_int_be(data.read(16))
 
     def write(self):
-        f = StringIO()
+        f = cBytesIO()
         f.write(struct.pack(">I", self.min_blocksize)[-2:])
         f.write(struct.pack(">I", self.max_blocksize)[-2:])
         f.write(struct.pack(">I", self.min_framesize)[-3:])
@@ -206,18 +209,18 @@ class StreamInfo(MetadataBlock):
         byte = (self.sample_rate & 0xF) << 4
         byte += ((self.channels - 1) & 7) << 1
         byte += ((self.bits_per_sample - 1) >> 4) & 1
-        f.write(chr(byte))
+        f.write(chr_(byte))
         # 4 bits of bps, 4 of sample count
         byte = ((self.bits_per_sample - 1) & 0xF) << 4
         byte += (self.total_samples >> 32) & 0xF
-        f.write(chr(byte))
+        f.write(chr_(byte))
         # last 32 of sample count
-        f.write(struct.pack(">I", self.total_samples & 0xFFFFFFFFL))
+        f.write(struct.pack(">I", self.total_samples & 0xFFFFFFFF))
         # MD5 signature
         sig = self.md5_signature
         f.write(struct.pack(
-            ">4I", (sig >> 96) & 0xFFFFFFFFL, (sig >> 64) & 0xFFFFFFFFL,
-            (sig >> 32) & 0xFFFFFFFFL, sig & 0xFFFFFFFFL))
+            ">4I", (sig >> 96) & 0xFFFFFFFF, (sig >> 64) & 0xFFFFFFFF,
+            (sig >> 32) & 0xFFFFFFFF, sig & 0xFFFFFFFF))
         return f.getvalue()
 
     def pprint(self):
@@ -284,7 +287,7 @@ class SeekTable(MetadataBlock):
             sp = data.tryread(self.__SEEKPOINT_SIZE)
 
     def write(self):
-        f = StringIO()
+        f = cBytesIO()
         for seekpoint in self.seekpoints:
             packed = struct.pack(
                 self.__SEEKPOINT_FORMAT,
@@ -378,10 +381,10 @@ class CueSheetTrack(object):
     __hash__ = object.__hash__
 
     def __repr__(self):
-        return ("<%s number=%r, offset=%d, isrc=%r, type=%r, "
-                "pre_emphasis=%r, indexes=%r)>") % (
-                    type(self).__name__, self.track_number, self.start_offset,
-                    self.isrc, self.type, self.pre_emphasis, self.indexes)
+        return (("<%s number=%r, offset=%d, isrc=%r, type=%r, "
+                "pre_emphasis=%r, indexes=%r)>") %
+                (type(self).__name__, self.track_number, self.start_offset,
+                 self.isrc, self.type, self.pre_emphasis, self.indexes))
 
 
 class CueSheet(MetadataBlock):
@@ -409,7 +412,7 @@ class CueSheet(MetadataBlock):
 
     code = 5
 
-    media_catalog_number = ''
+    media_catalog_number = b''
     lead_in_samples = 88200
     compact_disc = True
 
@@ -432,7 +435,7 @@ class CueSheet(MetadataBlock):
         header = data.read(self.__CUESHEET_SIZE)
         media_catalog_number, lead_in_samples, flags, num_tracks = \
             struct.unpack(self.__CUESHEET_FORMAT, header)
-        self.media_catalog_number = media_catalog_number.rstrip('\0')
+        self.media_catalog_number = media_catalog_number.rstrip(b'\0')
         self.lead_in_samples = lead_in_samples
         self.compact_disc = bool(flags & 0x80)
         self.tracks = []
@@ -440,7 +443,7 @@ class CueSheet(MetadataBlock):
             track = data.read(self.__CUESHEET_TRACK_SIZE)
             start_offset, track_number, isrc_padded, flags, num_indexes = \
                 struct.unpack(self.__CUESHEET_TRACK_FORMAT, track)
-            isrc = isrc_padded.rstrip('\0')
+            isrc = isrc_padded.rstrip(b'\0')
             type_ = (flags & 0x80) >> 7
             pre_emphasis = bool(flags & 0x40)
             val = CueSheetTrack(
@@ -454,7 +457,7 @@ class CueSheet(MetadataBlock):
             self.tracks.append(val)
 
     def write(self):
-        f = StringIO()
+        f = cBytesIO()
         flags = 0
         if self.compact_disc:
             flags |= 0x80
@@ -480,10 +483,10 @@ class CueSheet(MetadataBlock):
         return f.getvalue()
 
     def __repr__(self):
-        return ("<%s media_catalog_number=%r, lead_in=%r, compact_disc=%r, "
-                "tracks=%r>") % (
-                    type(self).__name__, self.media_catalog_number,
-                    self.lead_in_samples, self.compact_disc, self.tracks)
+        return (("<%s media_catalog_number=%r, lead_in=%r, compact_disc=%r, "
+                 "tracks=%r>") %
+                (type(self).__name__, self.media_catalog_number,
+                 self.lead_in_samples, self.compact_disc, self.tracks))
 
 
 class Picture(MetadataBlock):
@@ -513,7 +516,7 @@ class Picture(MetadataBlock):
         self.height = 0
         self.depth = 0
         self.colors = 0
-        self.data = ''
+        self.data = b''
         super(Picture, self).__init__(data)
 
     def __eq__(self, other):
@@ -541,7 +544,7 @@ class Picture(MetadataBlock):
         self.data = data.read(length)
 
     def write(self):
-        f = StringIO()
+        f = cBytesIO()
         mime = self.mime.encode('UTF-8')
         f.write(struct.pack('>2I', self.type, len(mime)))
         f.write(mime)
@@ -569,7 +572,7 @@ class Padding(MetadataBlock):
 
     code = 1
 
-    def __init__(self, data=""):
+    def __init__(self, data=b""):
         super(Padding, self).__init__(data)
 
     def load(self, data):
@@ -577,7 +580,7 @@ class Padding(MetadataBlock):
 
     def write(self):
         try:
-            return "\x00" * self.length
+            return b"\x00" * self.length
         # On some 64 bit platforms this won't generate a MemoryError
         # or OverflowError since you might have enough RAM, but it
         # still generates a ValueError. On other 64 bit platforms,
@@ -596,7 +599,7 @@ class Padding(MetadataBlock):
         return "<%s (%d bytes)>" % (type(self).__name__, self.length)
 
 
-class FLAC(FileType):
+class FLAC(mutagen.FileType):
     """A FLAC audio file.
 
     Attributes:
@@ -615,9 +618,9 @@ class FLAC(FileType):
     """Known metadata block types, indexed by ID."""
 
     @staticmethod
-    def score(filename, fileobj, header):
-        return (header.startswith("fLaC") +
-                filename.lower().endswith(".flac") * 3)
+    def score(filename, fileobj, header_data):
+        return (header_data.startswith(b"fLaC") +
+                endswith(filename.lower(), ".flac") * 3)
 
     def __read_metadata_block(self, fileobj):
         byte = ord(fileobj.read(1))
@@ -721,13 +724,15 @@ class FLAC(FileType):
 
     def clear_pictures(self):
         """Delete all pictures from the file."""
-        self.metadata_blocks = filter(lambda b: b.code != Picture.code,
-                                      self.metadata_blocks)
+
+        blocks = [b for b in self.metadata_blocks if b.code != Picture.code]
+        self.metadata_blocks = blocks
 
     @property
     def pictures(self):
         """List of embedded pictures"""
-        return filter(lambda b: b.code == Picture.code, self.metadata_blocks)
+
+        return [b for b in self.metadata_blocks if b.code == Picture.code]
 
     def save(self, filename=None, deleteid3=False):
         """Save metadata blocks to a file.
@@ -742,7 +747,7 @@ class FLAC(FileType):
         try:
             # Ensure we've got padding at the end, and only at the end.
             # If adding makes it too large, we'll scale it down later.
-            self.metadata_blocks.append(Padding('\x00' * 1020))
+            self.metadata_blocks.append(Padding(b'\x00' * 1020))
             MetadataBlock.group_padding(self.metadata_blocks)
 
             header = self.__check_header(f)
@@ -776,7 +781,7 @@ class FLAC(FileType):
                 insert_bytes(f, diff, header)
 
             f.seek(header - 4)
-            f.write("fLaC" + data)
+            f.write(b"fLaC" + data)
 
             # Delete ID3v1
             if deleteid3:
@@ -785,7 +790,7 @@ class FLAC(FileType):
                 except IOError:
                     pass
                 else:
-                    if f.read(3) == "TAG":
+                    if f.read(3) == b"TAG":
                         f.seek(-128, 2)
                         f.truncate()
         finally:
@@ -812,12 +817,12 @@ class FLAC(FileType):
     def __check_header(self, fileobj):
         size = 4
         header = fileobj.read(4)
-        if header != "fLaC":
+        if header != b"fLaC":
             size = None
-            if header[:3] == "ID3":
+            if header[:3] == b"ID3":
                 size = 14 + BitPaddedInt(fileobj.read(6)[2:])
                 fileobj.seek(size - 4)
-                if fileobj.read(4) != "fLaC":
+                if fileobj.read(4) != b"fLaC":
                     size = None
         if size is None:
             raise FLACNoHeaderError(

lib/mutagen/id3/__init__.py

@@ -1,4 +1,5 @@
-# id3 support for mutagen
+# -*- coding: utf-8 -*-
+
 # Copyright (C) 2005  Michael Urman
 #               2006  Lukas Lalinsky
 #               2013  Christoph Reiter
@@ -32,15 +33,30 @@ interested in the :class:`ID3` class to start with.
 __all__ = ['ID3', 'ID3FileType', 'Frames', 'Open', 'delete']
 
 import struct
+import errno
 
 from struct import unpack, pack, error as StructError
 
 import mutagen
-from mutagen._util import insert_bytes, delete_bytes, DictProxy
+from mutagen._util import insert_bytes, delete_bytes, DictProxy, enum
+from .._compat import chr_, PY3
 
-from mutagen._id3util import *
-from mutagen._id3frames import *
-from mutagen._id3specs import *
+from ._util import *
+from ._frames import *
+from ._specs import *
+
+
+@enum
+class ID3v1SaveOptions(object):
+
+    REMOVE = 0
+    """ID3v1 tags will be removed"""
+
+    UPDATE = 1
+    """ID3v1 tags will be updated but not added"""
+
+    CREATE = 2
+    """ID3v1 tags will be created and/or updated"""
 
 
 class ID3(DictProxy, mutagen.Metadata):
@@ -53,8 +69,11 @@ class ID3(DictProxy, mutagen.Metadata):
     * size -- the total size of the ID3 tag, including the header
     """
 
+    __module__ = "mutagen.id3"
+
     PEDANTIC = True
     version = (2, 4, 0)
+    """ID3 tag version as a tuple (of the loaded file)"""
 
     filename = None
     size = 0
@@ -73,15 +92,17 @@ class ID3(DictProxy, mutagen.Metadata):
         super(ID3, self).__init__(*args, **kwargs)
 
     def __fullread(self, size):
+        """ Read a certain number of bytes from the source file. """
+
         try:
             if size < 0:
                 raise ValueError('Requested bytes (%s) less than zero' % size)
             if size > self.__filesize:
                 raise EOFError('Requested %#x of %#x (%s)' % (
-                    long(size), long(self.__filesize), self.filename))
+                    int(size), int(self.__filesize), self.filename))
         except AttributeError:
             pass
-        data = self.__fileobj.read(size)
+        data = self._fileobj.read(size)
         if len(data) != size:
             raise EOFError
         self.__readbytes += size
@@ -107,37 +128,31 @@ class ID3(DictProxy, mutagen.Metadata):
             mutagen.id3.ID3(filename, known_frames=my_frames)
         """
 
-        if not v2_version in (3, 4):
+        if v2_version not in (3, 4):
             raise ValueError("Only 3 and 4 possible for v2_version")
 
         from os.path import getsize
 
         self.filename = filename
         self.__known_frames = known_frames
-        self.__fileobj = open(filename, 'rb')
+        self._fileobj = open(filename, 'rb')
         self.__filesize = getsize(filename)
         try:
             try:
-                self.__load_header()
+                self._load_header()
             except EOFError:
                 self.size = 0
                 raise ID3NoHeaderError("%s: too small (%d bytes)" % (
                     filename, self.__filesize))
-            except (ID3NoHeaderError, ID3UnsupportedVersionError), err:
+            except (ID3NoHeaderError, ID3UnsupportedVersionError):
                 self.size = 0
-                import sys
-                stack = sys.exc_info()[2]
-                try:
-                    self.__fileobj.seek(-128, 2)
-                except EnvironmentError:
-                    raise err, None, stack
-                else:
-                    frames = ParseID3v1(self.__fileobj.read(128))
-                    if frames is not None:
-                        self.version = self._V11
-                        map(self.add, frames.values())
-                    else:
-                        raise err, None, stack
+                frames, offset = _find_id3v1(self._fileobj)
+                if frames is None:
+                    raise
+
+                self.version = self._V11
+                for v in frames.values():
+                    self.add(v)
             else:
                 frames = self.__known_frames
                 if frames is None:
@@ -151,10 +166,10 @@ class ID3(DictProxy, mutagen.Metadata):
                         self.add(frame)
                     else:
                         self.unknown_frames.append(frame)
-                self.__unknown_version = self.version
+                self.__unknown_version = self.version[:2]
         finally:
-            self.__fileobj.close()
-            del self.__fileobj
+            self._fileobj.close()
+            del self._fileobj
             del self.__filesize
             if translate:
                 if v2_version == 3:
@@ -188,8 +203,9 @@ class ID3(DictProxy, mutagen.Metadata):
             del(self[key])
         else:
             key = key + ":"
-            for k in filter(lambda s: s.startswith(key), self.keys()):
-                del(self[k])
+            for k in list(self.keys()):
+                if k.startswith(key):
+                    del(self[k])
 
     def setall(self, key, values):
         """Delete frames of the given type and add frames in 'values'."""
@@ -209,8 +225,7 @@ class ID3(DictProxy, mutagen.Metadata):
 
             ``POPM=user@example.org=3 128/255``
         """
-        frames = list(map(Frame.pprint, self.values()))
-        frames.sort()
+        frames = sorted(Frame.pprint(s) for s in self.values())
         return "\n".join(frames)
 
     def loaded_frame(self, tag):
@@ -227,7 +242,7 @@ class ID3(DictProxy, mutagen.Metadata):
         """Add a frame to the tag."""
         return self.loaded_frame(frame)
 
-    def __load_header(self):
+    def _load_header(self):
         fn = self.filename
         data = self.__fullread(10)
         id3, vmaj, vrev, flags, size = unpack('>3sBBB4s', data)
@@ -235,24 +250,25 @@ class ID3(DictProxy, mutagen.Metadata):
         self.size = BitPaddedInt(size) + 10
         self.version = (2, vmaj, vrev)
 
-        if id3 != 'ID3':
-            raise ID3NoHeaderError("'%s' doesn't start with an ID3 tag" % fn)
+        if id3 != b'ID3':
+            raise ID3NoHeaderError("%r doesn't start with an ID3 tag" % fn)
         if vmaj not in [2, 3, 4]:
-            raise ID3UnsupportedVersionError("'%s' ID3v2.%d not supported"
+            raise ID3UnsupportedVersionError("%r ID3v2.%d not supported"
                                              % (fn, vmaj))
 
         if self.PEDANTIC:
             if not BitPaddedInt.has_valid_padding(size):
                 raise ValueError("Header size not synchsafe")
 
-            if self._V24 <= self.version and (flags & 0x0f):
-                raise ValueError("'%s' has invalid flags %#02x" % (fn, flags))
-            elif self._V23 <= self.version < self._V24 and (flags & 0x1f):
-                raise ValueError("'%s' has invalid flags %#02x" % (fn, flags))
+            if (self._V24 <= self.version) and (flags & 0x0f):
+                raise ValueError("%r has invalid flags %#02x" % (fn, flags))
+            elif (self._V23 <= self.version < self._V24) and (flags & 0x1f):
+                raise ValueError("%r has invalid flags %#02x" % (fn, flags))
 
         if self.f_extended:
             extsize = self.__fullread(4)
-            if extsize in Frames:
+            frame_id = extsize.decode("ascii", "replace") if PY3 else extsize
+            if frame_id in Frames:
                 # Some tagger sets the extended header flag but
                 # doesn't write an extended header; in this case, the
                 # ID3 data follows immediately. Since no extended
@@ -263,7 +279,7 @@ class ID3(DictProxy, mutagen.Metadata):
                 # http://code.google.com/p/quodlibet/issues/detail?id=126
                 self.__flags ^= 0x40
                 self.__extsize = 0
-                self.__fileobj.seek(-4, 1)
+                self._fileobj.seek(-4, 1)
                 self.__readbytes -= 4
             elif self.version >= self._V24:
                 # "Where the 'Extended header size' is the size of the whole
@@ -279,49 +295,13 @@ class ID3(DictProxy, mutagen.Metadata):
             if self.__extsize:
                 self.__extdata = self.__fullread(self.__extsize)
             else:
-                self.__extdata = ""
+                self.__extdata = b""
 
-    def __determine_bpi(self, data, frames, EMPTY="\x00" * 10):
+    def __determine_bpi(self, data, frames):
         if self.version < self._V24:
             return int
-        # have to special case whether to use bitpaddedints here
-        # spec says to use them, but iTunes has it wrong
 
-        # count number of tags found as BitPaddedInt and how far past
-        o = 0
-        asbpi = 0
-        while o < len(data) - 10:
-            part = data[o:o + 10]
-            if part == EMPTY:
-                bpioff = -((len(data) - o) % 10)
-                break
-            name, size, flags = unpack('>4sLH', part)
-            size = BitPaddedInt(size)
-            o += 10 + size
-            if name in frames:
-                asbpi += 1
-        else:
-            bpioff = o - len(data)
-
-        # count number of tags found as int and how far past
-        o = 0
-        asint = 0
-        while o < len(data) - 10:
-            part = data[o:o + 10]
-            if part == EMPTY:
-                intoff = -((len(data) - o) % 10)
-                break
-            name, size, flags = unpack('>4sLH', part)
-            o += 10 + size
-            if name in frames:
-                asint += 1
-        else:
-            intoff = o - len(data)
-
-        # if more tags as int, or equal and bpi is past and int is not
-        if asint > asbpi or (asint == asbpi and (bpioff >= 1 and intoff <= 1)):
-            return int
-        return BitPaddedInt
+        return _determine_bpi(data, frames)
 
     def __read_frames(self, data, frames):
         if self.version < self._V24 and self.f_unsynch:
@@ -338,14 +318,27 @@ class ID3(DictProxy, mutagen.Metadata):
                     name, size, flags = unpack('>4sLH', header)
                 except struct.error:
                     return  # not enough header
-                if name.strip('\x00') == '':
+                if name.strip(b'\x00') == b'':
                     return
+
                 size = bpi(size)
-                framedata = data[10:10+size]
-                data = data[10+size:]
+                framedata = data[10:10 + size]
+                data = data[10 + size:]
                 if size == 0:
                     continue  # drop empty frames
+
+                if PY3:
+                    try:
+                        name = name.decode('ascii')
+                    except UnicodeDecodeError:
+                        continue
+
                 try:
+                    # someone writes 2.3 frames with 2.2 names
+                    if name[-1] == "\x00":
+                        tag = Frames_2_2[name[:-1]]
+                        name = tag.__base__.__name__
+
                     tag = frames[name]
                 except KeyError:
                     if is_valid_frame_id(name):
@@ -365,13 +358,21 @@ class ID3(DictProxy, mutagen.Metadata):
                     name, size = unpack('>3s3s', header)
                 except struct.error:
                     return  # not enough header
-                size, = struct.unpack('>L', '\x00'+size)
-                if name.strip('\x00') == '':
+                size, = struct.unpack('>L', b'\x00' + size)
+                if name.strip(b'\x00') == b'':
                     return
-                framedata = data[6:6+size]
-                data = data[6+size:]
+
+                framedata = data[6:6 + size]
+                data = data[6 + size:]
                 if size == 0:
                     continue  # drop empty frames
+
+                if PY3:
+                    try:
+                        name = name.decode('ascii')
+                    except UnicodeDecodeError:
+                        continue
+
                 try:
                     tag = frames[name]
                 except KeyError:
@@ -393,29 +394,9 @@ class ID3(DictProxy, mutagen.Metadata):
     f_experimental = property(lambda s: bool(s.__flags & 0x20))
     f_footer = property(lambda s: bool(s.__flags & 0x10))
 
-    #f_crc = property(lambda s: bool(s.__extflags & 0x8000))
-
-    def save(self, filename=None, v1=1, v2_version=4, v23_sep='/'):
-        """Save changes to a file.
-
-        If no filename is given, the one most recently loaded is used.
-
-        Keyword arguments:
-        v1 -- if 0, ID3v1 tags will be removed
-              if 1, ID3v1 tags will be updated but not added
-              if 2, ID3v1 tags will be created and/or updated
-        v2 -- version of ID3v2 tags (3 or 4).
-
-        By default Mutagen saves ID3v2.4 tags. If you want to save ID3v2.3
-        tags, you must call method update_to_v23 before saving the file.
-
-        v23_sep -- the separator used to join multiple text values
-                   if v2_version == 3. Defaults to '/' but if it's None
-                   will be the ID3v2v2.4 null separator.
-
-        The lack of a way to update only an ID3v1 tag is intentional.
-        """
+    # f_crc = property(lambda s: bool(s.__extflags & 0x8000))
 
+    def _prepare_framedata(self, v2_version, v23_sep):
         if v2_version == 3:
             version = self._V23
         elif v2_version == 4:
@@ -425,38 +406,83 @@ class ID3(DictProxy, mutagen.Metadata):
 
         # Sort frames by 'importance'
         order = ["TIT2", "TPE1", "TRCK", "TALB", "TPOS", "TDRC", "TCON"]
-        order = dict(zip(order, range(len(order))))
+        order = dict((b, a) for a, b in enumerate(order))
         last = len(order)
-        frames = self.items()
-        frames.sort(lambda a, b: cmp(order.get(a[0][:4], last),
-                                     order.get(b[0][:4], last)))
+        frames = sorted(self.items(),
+                        key=lambda a: (order.get(a[0][:4], last), a[0]))
 
         framedata = [self.__save_frame(frame, version=version, v23_sep=v23_sep)
                      for (key, frame) in frames]
 
         # only write unknown frames if they were loaded from the version
         # we are saving with or upgraded to it
-        if self.__unknown_version == version:
-            framedata.extend([data for data in self.unknown_frames
-                              if len(data) > 10])
+        if self.__unknown_version == version[:2]:
+            framedata.extend(data for data in self.unknown_frames
+                             if len(data) > 10)
+
+        return b''.join(framedata)
+
+    def _prepare_id3_header(self, original_header, framesize, v2_version):
+        try:
+            id3, vmaj, vrev, flags, insize = \
+                unpack('>3sBBB4s', original_header)
+        except struct.error:
+            id3, insize = b'', 0
+        insize = BitPaddedInt(insize)
+        if id3 != b'ID3':
+            insize = -10
+
+        if insize >= framesize:
+            outsize = insize
+        else:
+            outsize = (framesize + 1023) & ~0x3FF
+
+        framesize = BitPaddedInt.to_str(outsize, width=4)
+        header = pack('>3sBBB4s', b'ID3', v2_version, 0, 0, framesize)
+
+        return (header, outsize, insize)
+
+    def save(self, filename=None, v1=1, v2_version=4, v23_sep='/'):
+        """Save changes to a file.
+
+        Args:
+            filename:
+                Filename to save the tag to. If no filename is given,
+                the one most recently loaded is used.
+            v1 (ID3v1SaveOptions):
+                if 0, ID3v1 tags will be removed.
+                if 1, ID3v1 tags will be updated but not added.
+                if 2, ID3v1 tags will be created and/or updated
+            v2 (int):
+                version of ID3v2 tags (3 or 4).
+            v23_sep (str):
+                the separator used to join multiple text values
+                if v2_version == 3. Defaults to '/' but if it's None
+                will be the ID3v2v2.4 null separator.
+
+        By default Mutagen saves ID3v2.4 tags. If you want to save ID3v2.3
+        tags, you must call method update_to_v23 before saving the file.
+
+        The lack of a way to update only an ID3v1 tag is intentional.
+        """
+
+        framedata = self._prepare_framedata(v2_version, v23_sep)
+        framesize = len(framedata)
 
         if not framedata:
             try:
                 self.delete(filename)
-            except EnvironmentError, err:
+            except EnvironmentError as err:
                 from errno import ENOENT
                 if err.errno != ENOENT:
                     raise
             return
 
-        framedata = ''.join(framedata)
-        framesize = len(framedata)
-
         if filename is None:
             filename = self.filename
         try:
             f = open(filename, 'rb+')
-        except IOError, err:
+        except IOError as err:
             from errno import ENOENT
             if err.errno != ENOENT:
                 raise
@@ -464,61 +490,33 @@ class ID3(DictProxy, mutagen.Metadata):
             f = open(filename, 'rb+')
         try:
             idata = f.read(10)
-            try:
-                id3, vmaj, vrev, flags, insize = unpack('>3sBBB4s', idata)
-            except struct.error:
-                id3, insize = '', 0
-            insize = BitPaddedInt(insize)
-            if id3 != 'ID3':
-                insize = -10
 
-            if insize >= framesize:
-                outsize = insize
-            else:
-                outsize = (framesize + 1023) & ~0x3FF
-            framedata += '\x00' * (outsize - framesize)
+            header = self._prepare_id3_header(idata, framesize, v2_version)
+            header, outsize, insize = header
 
-            framesize = BitPaddedInt.to_str(outsize, width=4)
-            flags = 0
-            header = pack('>3sBBB4s', 'ID3', v2_version, 0, flags, framesize)
-            data = header + framedata
+            data = header + framedata + (b'\x00' * (outsize - framesize))
 
             if (insize < outsize):
-                insert_bytes(f, outsize-insize, insize+10)
+                insert_bytes(f, outsize - insize, insize + 10)
             f.seek(0)
             f.write(data)
 
-            try:
-                f.seek(-128, 2)
-            except IOError, err:
-                # If the file is too small, that's OK - it just means
-                # we're certain it doesn't have a v1 tag.
-                from errno import EINVAL
-                if err.errno != EINVAL:
-                    # If we failed to see for some other reason, bail out.
-                    raise
-                # Since we're sure this isn't a v1 tag, don't read it.
-                f.seek(0, 2)
-
-            data = f.read(128)
-            try:
-                idx = data.index("TAG")
-            except ValueError:
-                offset = 0
-                has_v1 = False
-            else:
-                offset = idx - len(data)
-                has_v1 = True
-
-            f.seek(offset, 2)
-            if v1 == 1 and has_v1 or v1 == 2:
-                f.write(MakeID3v1(self))
-            else:
-                f.truncate()
+            self.__save_v1(f, v1)
 
         finally:
             f.close()
 
+    def __save_v1(self, f, v1):
+        tag, offset = _find_id3v1(f)
+        has_v1 = tag is not None
+
+        f.seek(offset, 2)
+        if v1 == ID3v1SaveOptions.UPDATE and has_v1 or \
+                v1 == ID3v1SaveOptions.CREATE:
+            f.write(MakeID3v1(self))
+        else:
+            f.truncate()
+
     def delete(self, filename=None, delete_v1=True, delete_v2=True):
         """Remove tags from a file.
 
@@ -538,7 +536,7 @@ class ID3(DictProxy, mutagen.Metadata):
         flags = 0
         if self.PEDANTIC and isinstance(frame, TextFrame):
             if len(str(frame)) == 0:
-                return ''
+                return b''
 
         if version == self._V23:
             framev23 = frame._get_v23_frame(sep=v23_sep)
@@ -551,8 +549,8 @@ class ID3(DictProxy, mutagen.Metadata):
             # Disabled as this causes iTunes and other programs
             # to fail to find these frames, which usually includes
             # e.g. APIC.
-            #framedata = BitPaddedInt.to_str(usize) + framedata.encode('zlib')
-            #flags |= Frame.FLAG24_COMPRESS | Frame.FLAG24_DATALEN
+            # framedata = BitPaddedInt.to_str(usize) + framedata.encode('zlib')
+            # flags |= Frame.FLAG24_COMPRESS | Frame.FLAG24_DATALEN
             pass
 
         if version == self._V24:
@@ -563,7 +561,16 @@ class ID3(DictProxy, mutagen.Metadata):
             raise ValueError
 
         datasize = BitPaddedInt.to_str(len(framedata), width=4, bits=bits)
-        header = pack('>4s4sH', name or type(frame).__name__, datasize, flags)
+
+        if name is not None:
+            assert isinstance(name, bytes)
+            frame_name = name
+        else:
+            frame_name = type(frame).__name__
+            if PY3:
+                frame_name = frame_name.encode("ascii")
+
+        header = pack('>4s4sH', frame_name, datasize, flags)
         return header + framedata
 
     def __update_common(self):
@@ -573,20 +580,19 @@ class ID3(DictProxy, mutagen.Metadata):
             # Get rid of "(xx)Foobr" format.
             self["TCON"].genres = self["TCON"].genres
 
-        if self.version < self._V23:
-            # ID3v2.2 PIC frames are slightly different.
-            pics = self.getall("APIC")
-            mimes = {"PNG": "image/png", "JPG": "image/jpeg"}
-            self.delall("APIC")
-            for pic in pics:
+        # ID3v2.2 LNK frames are just way too different to upgrade.
+        for frame in self.getall("LINK"):
+            if len(frame.frameid) != 4:
+                del self[frame.HashKey]
+
+        mimes = {"PNG": "image/png", "JPG": "image/jpeg"}
+        for pic in self.getall("APIC"):
+            if pic.mime in mimes:
                 newpic = APIC(
-                    encoding=pic.encoding, mime=mimes.get(pic.mime, pic.mime),
+                    encoding=pic.encoding, mime=mimes[pic.mime],
                     type=pic.type, desc=pic.desc, data=pic.data)
                 self.add(newpic)
 
-            # ID3v2.2 LNK frames are just way too different to upgrade.
-            self.delall("LINK")
-
     def update_to_v24(self):
         """Convert older tags into an ID3v2.4 tag.
 
@@ -597,7 +603,7 @@ class ID3(DictProxy, mutagen.Metadata):
 
         self.__update_common()
 
-        if self.__unknown_version == (2, 3, 0):
+        if self.__unknown_version == (2, 3):
             # convert unknown 2.3 frames (flags/size) to 2.4
             converted = []
             for frame in self.unknown_frames:
@@ -606,19 +612,23 @@ class ID3(DictProxy, mutagen.Metadata):
                     frame = BinaryFrame.fromData(self, flags, frame[10:])
                 except (struct.error, error):
                     continue
+
                 converted.append(self.__save_frame(frame, name=name))
             self.unknown_frames[:] = converted
-            self.__unknown_version = (2, 4, 0)
+            self.__unknown_version = (2, 4)
 
         # TDAT, TYER, and TIME have been turned into TDRC.
         try:
-            if str(self.get("TYER", "")).strip("\x00"):
-                date = str(self.pop("TYER"))
-                if str(self.get("TDAT", "")).strip("\x00"):
-                    dat = str(self.pop("TDAT"))
+            date = text_type(self.get("TYER", ""))
+            if date.strip(u"\x00"):
+                self.pop("TYER")
+                dat = text_type(self.get("TDAT", ""))
+                if dat.strip("\x00"):
+                    self.pop("TDAT")
                     date = "%s-%s-%s" % (date, dat[2:], dat[:2])
-                    if str(self.get("TIME", "")).strip("\x00"):
-                        time = str(self.pop("TIME"))
+                    time = text_type(self.get("TIME", ""))
+                    if time.strip("\x00"):
+                        self.pop("TIME")
                         date += "T%s:%s:00" % (time[:2], time[2:])
                 if "TDRC" not in self:
                     self.add(TDRC(encoding=0, text=date))
@@ -723,14 +733,10 @@ def delete(filename, delete_v1=True, delete_v2=True):
     f = open(filename, 'rb+')
 
     if delete_v1:
-        try:
-            f.seek(-128, 2)
-        except IOError:
-            pass
-        else:
-            if f.read(3) == "TAG":
-                f.seek(-128, 2)
-                f.truncate()
+        tag, offset = _find_id3v1(f)
+        if tag is not None:
+            f.seek(offset, 2)
+            f.truncate()
 
     # technically an insize=0 tag is invalid, but we delete it anyway
     # (primarily because we used to write it)
@@ -740,9 +746,9 @@ def delete(filename, delete_v1=True, delete_v2=True):
         try:
             id3, vmaj, vrev, flags, insize = unpack('>3sBBB4s', idata)
         except struct.error:
-            id3, insize = '', -1
+            id3, insize = b'', -1
         insize = BitPaddedInt(insize)
-        if id3 == 'ID3' and insize >= 0:
+        if id3 == b'ID3' and insize >= 0:
             delete_bytes(f, insize + 10, 0)
 
 
@@ -750,15 +756,116 @@ def delete(filename, delete_v1=True, delete_v2=True):
 Open = ID3
 
 
-# ID3v1.1 support.
-def ParseID3v1(string):
-    """Parse an ID3v1 tag, returning a list of ID3v2.4 frames."""
+def _determine_bpi(data, frames, EMPTY=b"\x00" * 10):
+    """Takes id3v2.4 frame data and determines if ints or bitpaddedints
+    should be used for parsing. Needed because iTunes used to write
+    normal ints for frame sizes.
+    """
+
+    # count number of tags found as BitPaddedInt and how far past
+    o = 0
+    asbpi = 0
+    while o < len(data) - 10:
+        part = data[o:o + 10]
+        if part == EMPTY:
+            bpioff = -((len(data) - o) % 10)
+            break
+        name, size, flags = unpack('>4sLH', part)
+        size = BitPaddedInt(size)
+        o += 10 + size
+        if PY3:
+            try:
+                name = name.decode("ascii")
+            except UnicodeDecodeError:
+                continue
+        if name in frames:
+            asbpi += 1
+    else:
+        bpioff = o - len(data)
+
+    # count number of tags found as int and how far past
+    o = 0
+    asint = 0
+    while o < len(data) - 10:
+        part = data[o:o + 10]
+        if part == EMPTY:
+            intoff = -((len(data) - o) % 10)
+            break
+        name, size, flags = unpack('>4sLH', part)
+        o += 10 + size
+        if PY3:
+            try:
+                name = name.decode("ascii")
+            except UnicodeDecodeError:
+                continue
+        if name in frames:
+            asint += 1
+    else:
+        intoff = o - len(data)
+
+    # if more tags as int, or equal and bpi is past and int is not
+    if asint > asbpi or (asint == asbpi and (bpioff >= 1 and intoff <= 1)):
+        return int
+    return BitPaddedInt
+
+
+def _find_id3v1(fileobj):
+    """Returns a tuple of (id3tag, offset_to_end) or (None, 0)
+
+    offset mainly because we used to write too short tags in some cases and
+    we need the offset to delete them.
+    """
+
+    # id3v1 is always at the end (after apev2)
+
+    extra_read = b"APETAGEX".index(b"TAG")
 
     try:
-        string = string[string.index("TAG"):]
+        fileobj.seek(-128 - extra_read, 2)
+    except IOError as e:
+        if e.errno == errno.EINVAL:
+            # If the file is too small, might be ok since we wrote too small
+            # tags at some point. let's see how the parsing goes..
+            fileobj.seek(0, 0)
+        else:
+            raise
+
+    data = fileobj.read(128 + extra_read)
+    try:
+        idx = data.index(b"TAG")
+    except ValueError:
+        return (None, 0)
+    else:
+        # FIXME: make use of the apev2 parser here
+        # if TAG is part of APETAGEX assume this is an APEv2 tag
+        try:
+            ape_idx = data.index(b"APETAGEX")
+        except ValueError:
+            pass
+        else:
+            if idx == ape_idx + extra_read:
+                return (None, 0)
+
+        tag = ParseID3v1(data[idx:])
+        if tag is None:
+            return (None, 0)
+
+        offset = idx - len(data)
+        return (tag, offset)
+
+
+# ID3v1.1 support.
+def ParseID3v1(data):
+    """Parse an ID3v1 tag, returning a list of ID3v2.4 frames.
+
+    Returns a {frame_name: frame} dict or None.
+    """
+
+    try:
+        data = data[data.index(b"TAG"):]
     except ValueError:
         return None
-    if 128 < len(string) or len(string) < 124:
+    if 128 < len(data) or len(data) < 124:
         return None
 
     # Issue #69 - Previous versions of Mutagen, when encountering
@@ -766,19 +873,19 @@ def ParseID3v1(string):
     # wrote only the characters available - e.g. "1" or "" - into the
     # year field. To parse those, reduce the size of the year field.
     # Amazingly, "0s" works as a struct format string.
-    unpack_fmt = "3s30s30s30s%ds29sBB" % (len(string) - 124)
+    unpack_fmt = "3s30s30s30s%ds29sBB" % (len(data) - 124)
 
     try:
         tag, title, artist, album, year, comment, track, genre = unpack(
-            unpack_fmt, string)
+            unpack_fmt, data)
     except StructError:
         return None
 
-    if tag != "TAG":
+    if tag != b"TAG":
         return None
 
-    def fix(string):
-        return string.split("\x00")[0].strip().decode('latin1')
+    def fix(data):
+        return data.split(b"\x00")[0].strip().decode('latin1')
 
     title, artist, album, year, comment = map(
         fix, [title, artist, album, year, comment])
@@ -797,7 +904,7 @@ def ParseID3v1(string):
             encoding=0, lang="eng", desc="ID3v1 Comment", text=comment)
     # Don't read a track number if it looks like the comment was
     # padded with spaces instead of nulls (thanks, WinAmp).
-    if track and (track != 32 or string[-3] == '\x00'):
+    if track and ((track != 32) or (data[-3] == b'\x00'[0])):
         frames["TRCK"] = TRCK(encoding=0, text=str(track))
     if genre != 255:
         frames["TCON"] = TCON(encoding=0, text=str(genre))
@@ -814,22 +921,22 @@ def MakeID3v1(id3):
         if v2id in id3:
             text = id3[v2id].text[0].encode('latin1', 'replace')[:30]
         else:
-            text = ""
-        v1[name] = text + ("\x00" * (30 - len(text)))
+            text = b""
+        v1[name] = text + (b"\x00" * (30 - len(text)))
 
     if "COMM" in id3:
         cmnt = id3["COMM"].text[0].encode('latin1', 'replace')[:28]
     else:
-        cmnt = ""
-    v1["comment"] = cmnt + ("\x00" * (29 - len(cmnt)))
+        cmnt = b""
+    v1["comment"] = cmnt + (b"\x00" * (29 - len(cmnt)))
 
     if "TRCK" in id3:
         try:
-            v1["track"] = chr(+id3["TRCK"])
+            v1["track"] = chr_(+id3["TRCK"])
         except ValueError:
-            v1["track"] = "\x00"
+            v1["track"] = b"\x00"
     else:
-        v1["track"] = "\x00"
+        v1["track"] = b"\x00"
 
     if "TCON" in id3:
         try:
@@ -838,20 +945,28 @@ def MakeID3v1(id3):
             pass
         else:
             if genre in TCON.GENRES:
-                v1["genre"] = chr(TCON.GENRES.index(genre))
+                v1["genre"] = chr_(TCON.GENRES.index(genre))
     if "genre" not in v1:
-        v1["genre"] = "\xff"
+        v1["genre"] = b"\xff"
 
     if "TDRC" in id3:
-        year = str(id3["TDRC"])
+        year = text_type(id3["TDRC"]).encode('ascii')
     elif "TYER" in id3:
-        year = str(id3["TYER"])
+        year = text_type(id3["TYER"]).encode('ascii')
     else:
-        year = ""
-    v1["year"] = (year + "\x00\x00\x00\x00")[:4]
+        year = b""
+    v1["year"] = (year + b"\x00\x00\x00\x00")[:4]
 
-    return ("TAG%(title)s%(artist)s%(album)s%(year)s%(comment)s"
-            "%(track)s%(genre)s") % v1
+    return (
+        b"TAG" +
+        v1["title"] +
+        v1["artist"] +
+        v1["album"] +
+        v1["year"] +
+        v1["comment"] +
+        v1["track"] +
+        v1["genre"]
+    )
 
 
 class ID3FileType(mutagen.FileType):
@@ -859,7 +974,7 @@ class ID3FileType(mutagen.FileType):
 
     ID3 = ID3
 
-    class _Info(object):
+    class _Info(mutagen.StreamInfo):
         length = 0
 
         def __init__(self, fileobj, offset):
@@ -870,8 +985,8 @@ class ID3FileType(mutagen.FileType):
             return "Unknown format with ID3 tag"
 
     @staticmethod
-    def score(filename, fileobj, header):
-        return header.startswith("ID3")
+    def score(filename, fileobj, header_data):
+        return header_data.startswith(b"ID3")
 
     def add_tags(self, ID3=None):
         """Add an empty ID3 tag to the file.