Release v0.5.2

* master:
  Log exact write check error to log (debug)

.travis.yml

@@ -8,9 +8,10 @@ language: python
 python:
   - "2.6"
   - "2.7"
+# pylint 1.4 does not run under python 2.6
 install:
  - pip install pyOpenSSL
- - pip install pylint
+ - pip install pylint==1.3.1
  - pip install pyflakes
  - pip install pep8
 script:

CHANGELOG.md

@@ -1,5 +1,21 @@
 # Changelog
 
+## v0.5.2
+Released 28 december 2014
+
+Highlight:
+* Added: advanced option to ignore certain folders by patterns. (#2037)
+* Added: advanced option to ignore certain files by patterns (library only)
+* Added: specify optional paths to CUE splitting tools (#1938)
+* Added: experimental support for OPB (provide details yourself!)
+* Fixed: magnet to torrent conversion (#1926)
+* Fixed: new KAT URL (#2043)
+* Fixed: LMS notifications (#1564)
+* Improved: notify user of SSL-related warnings, instead of silently failing.
+* Improved: show all search results for 'Choose Specific Release'
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.1...v0.5.2).
+
 ## v0.5.1
 Released 13 November 2014
 

CONTRIBUTING.md

@@ -1,26 +1,28 @@
 # Contributing to Headphones
 
-## For users
+## Issues
 In case you read this because you are posting an issue, please take a minute and conside the things below. The issue tracker is not a support forum. It is primarily intended to submit bugs, improvements or feature requests. However, we are glad to help you, and make sure the problem is not caused by Headphones, but don't expect step-by-step answers.
 
-* Use the search function. Chances are that your problem is already discussed.
+* Use the search function. Chances are that your problem is already discussed. Do not append to (closed) issues if your problem does not fit the discussion.
 * Visit the [Troubleshooting](../../wiki/TroubleShooting) wiki first.
 * Use [proper formatting](https://help.github.com/articles/github-flavored-markdown/). Paste your logs in code blocks.
 * Close your issue if you resolved it.
 
-## For developers
+## Pull Requests
 If you think you can contribute code to the Headphones repository, do not hesitate to submit a pull request.
 
 ### Branches
-All pull requests should be based on the `develop` branch. When you want to develop a new feature, clone the repository with `git clone origin/develop -b FEATURE_NAME`. Use meaningful commit messages.
+All pull requests should be based on the `develop` branch, to minimize cross merges. When you want to develop a new feature, clone the repository with `git clone origin/develop -b FEATURE_NAME`. Use meaningful commit messages.
 
-### Code compatibility
+### Python Code
+
+#### Compatibility
 The code should work with Python 2.6 and 2.7. Note that Headphones runs on different platforms, including Network Attached Storage devices such as Synology.
 
 Re-use existing code. Do not hesitate to add logging in your code. You can the logger module `headphones.logger.*` for this. Web requests are invoked via `headphones.request.*` and derived ones. Use these methods to automatically add proper and meaningful error handling.
 
-### Code conventions
+#### Code conventions
-Altough Headphones did not adapt a code convention in the past, we try to follow the [PEP8](http://legacy.python.org/dev/peps/pep-0008/) conventions for future code. A short summary to remind you (copied from http://wiki.ros.org/PyStyleGuide):
+Although Headphones did not adapt a code convention in the past, we try to follow the [PEP8](http://legacy.python.org/dev/peps/pep-0008/) conventions for future code. A short summary to remind you (copied from http://wiki.ros.org/PyStyleGuide):
 
  * 4 space indentation
  * 80 characters per line
@@ -32,4 +34,19 @@ Altough Headphones did not adapt a code convention in the past, we try to follow
  * `self.__really_private_field`
  * `_global`
 
-Document your code!
+#### Documentation
+Document your code. Use docstrings See [PEP-257](https://www.python.org/dev/peps/pep-0257/) for more information.
+
+#### Continuous Integration
+Headphones has a configuration file for [travis-ci](https://travis-ci.org/). You can add your forked repo to Travis to have it check your code against PEP8, PyLint, and PyFlakes for you. Your pull request will show a green check mark or a red cross on each tested commit, depending on if linting passes.
+
+### HTML/Template code
+
+#### Compatibility
+HTML5 compatible browsers are targetted. There is no specific mobile version of Headphones yet.
+
+#### Conventions
+* 4 space indentation
+* `methodName`
+* `variableName`
+* `ClassName`

Headphones.py

@@ -89,6 +89,10 @@ def main():
     if args.quiet:
         headphones.QUIET = True
 
+    # Do an intial setup of the logger.
+    logger.initLogger(console=not headphones.QUIET, log_dir=False,
+        verbose=headphones.VERBOSE)
+
     if args.daemon:
         if sys.platform == 'win32':
             sys.stderr.write(
@@ -103,8 +107,8 @@ def main():
         # If the pidfile already exists, headphones may still be running, so
         # exit
         if os.path.exists(headphones.PIDFILE):
-            sys.exit(
+            raise SystemExit("PID file '%s' already exists. Exiting." %
-                "PID file '" + headphones.PIDFILE + "' already exists. Exiting.")
+                headphones.PIDFILE)
 
         # The pidfile is only useful in daemon mode, make sure we can write the
         # file properly
@@ -115,11 +119,10 @@ def main():
                 with open(headphones.PIDFILE, 'w') as fp:
                     fp.write("pid\n")
             except IOError as e:
-                raise SystemExit(
+                raise SystemExit("Unable to write PID file: %s", e)
-                    "Unable to write PID file: %s [%d]", e.strerror, e.errno)
         else:
-            logger.warn(
+            logger.warn("Not running in daemon mode. PID file creation " \
-                "Not running in daemon mode. PID file creation disabled.")
+                "disabled.")
 
     # Determine which data directory and config file to use
     if args.datadir:
@@ -188,13 +191,14 @@ def main():
     }
     webstart.initialize(web_config)
 
+    # Start the background threads
+    headphones.start()
+
+    # Open webbrowser
     if headphones.CONFIG.LAUNCH_BROWSER and not args.nolaunch:
         headphones.launch_browser(headphones.CONFIG.HTTP_HOST, http_port,
                                   headphones.CONFIG.HTTP_ROOT)
 
-    # Start the background threads
-    headphones.start()
-
     # Wait endlessy for a signal to happen
     while True:
         if not headphones.SIGNAL:

data/interfaces/default/album.html

@@ -57,13 +57,12 @@
 									search_term = ""
 								else:
 									search_term = album['SearchTerm']
-								
 							%>
 							<input type="text" value="${search_term}" name="SearchTerm" size="40" />
 						</div>
 						<input type="button" value="Save changes" onclick="doAjaxCall('editSearchTerm',$(this),'tabs',true);return false;" data-success="Search term updated"/>
 					</form>
-				</div>	
+				</div>
 			<a class="menu_link_edit" id="choose_specific_download" href="#" onclick="getAvailableDownloads()"><i class="fa fa-search"></i> Choose Specific Download</a>
 				<div id="choose_specific_download_dialog" title="Choose a specific download for this album" style="display:none" class="configtable">
 						<table class="display" id="downloads_table">
@@ -73,19 +72,20 @@
 									<th id="size">Size</th>
 									<th id="provider">Provider</th>
 									<th id="kind">Kind</th>
+									<th id="matches" title="Matches quality and/or seed settings.">Matches</th>
 								</tr>
 							</thead>
 							<tbody id="downloads_table_body">
 							</tbody>
 						</table>
-				</div>	
+				</div>
 		</div>
-	</div>	
+	</div>
 </%def>
 
 <%def name="body()">
 	<div class="table_wrapper">
-		<div id="albumheader" class="clearfix">	
+		<div id="albumheader" class="clearfix">
 			<div id="albumImg">
 				<img height="200" alt="" class="albumArt" src="artwork/album/${album['AlbumID']}">
 			</div>
@@ -231,13 +231,14 @@
 			feedback.fadeOut();
 			search_results = data
 			for( var i = 0, len = data.length; i < len; i++ ) {
-    			$('#downloads_table_body').append('<tr><td id="title"><a href="#" onclick="downloadSpecificRelease('+i+')">'+data[i].title+'</a></td><td id="size"><span title='+data[i].size+'></span>'+(data[i].size / (1024*1024)).toFixed(2)+' MB</td><td id="provider">'+data[i].provider+'</td><td id="kind">'+data[i].kind+'</td></tr>');
+    			$('#downloads_table_body').append('<tr><td id="title"><a href="#" onclick="downloadSpecificRelease('+i+')">'+data[i].title+'</a></td><td id="size"><span title='+data[i].size+'></span>'+(data[i].size / (1024*1024)).toFixed(2)+' MB</td><td id="provider">'+data[i].provider+'</td><td id="kind">'+data[i].kind+'</td><td id="matches">'+data[i].matches+'</td></tr>');
     		}
 			$('#downloads_table').dataTable({
 				"aoColumns": [
 					null,
 					{ "sType": "title-numeric"},
 					null,
+					null,
 					null
 				],
 				"aaSorting": [],

data/interfaces/default/base.html

@@ -52,7 +52,7 @@
 				<li><a href="extras">extras</a></li>
 				<li><a href="manage">manage</a></li>
 				<li><a href="history">history</a></li>
-				<li><a href="logs" class="log">logs</a></li>
+				<li><a href="logs">logs</a></li>
 				<li><a href="config" class="config"><i class="fa fa-gear fa-lg"></i></a></li>
 			</ul>
 			<div id="searchbar">
@@ -139,4 +139,4 @@
 
 <%def name="javascriptIncludes()"></%def>
 <%def name="headIncludes()"></%def>
-<%def name="headerIncludes()"></%def>
+<%def name="headerIncludes()"></%def>

data/interfaces/default/config.html

@@ -533,7 +533,24 @@
                                         </div>
                                         <div class="row">
                                             <label>Seed Ratio</label>
-                                            <input type="text" class="override-float" name="piratebay_ratio" value="${config['piratebay_ratio']}" size="10" title="Stop seeding when ratio met, 0 = unlimited. Scheduled job will remove torrent when post processed and finished seeding">
+                                            <input type="text" class="override-float" name="piratebay_ratio" value="${config['piratebay_ratio']}" size="10" title="Stop seeding when ratio met, 0 = unlimited. Scheduled job will remove torrent when post processed and finished seeding.">
+                                        </div>
+                                    </div>
+                                </fieldset>
+
+                                <fieldset>
+                                    <legend>Old Pirate Bay</legend>
+                                    <div class="row checkbox">
+                                        <input id="use_oldpiratebay" type="checkbox" name="use_oldpiratebay" value="1" ${config['use_oldpiratebay']} /><label>Use Old Pirate Bay</label>
+                                    </div>
+                                    <div class="config">
+                                        <div class="row">
+                                            <label>URL</label>
+                                            <input type="text" name="oldpiratebay_url" value="${config['oldpiratebay_url']}" size="36">
+                                        </div>
+                                        <div class="row">
+                                            <label>Seed Ratio</label>
+                                            <input type="text" class="override-float" name="oldpiratebay_ratio" value="${config['oldpiratebay_ratio']}" size="10" title="Stop seeding when ratio met, 0 = unlimited. Scheduled job will remove torrent when post processed and finished seeding.">
                                         </div>
                                     </div>
                                 </fieldset>
@@ -2050,6 +2067,7 @@
                 initConfigCheckbox("#use_omgwtfnzbs");
                 initConfigCheckbox("#use_kat");
                 initConfigCheckbox("#use_piratebay");
+                initConfigCheckbox("#use_oldpiratebay");
                 initConfigCheckbox("#use_mininova");
                 initConfigCheckbox("#use_waffles");
                 initConfigCheckbox("#use_rutracker");

data/interfaces/default/managemanual.html

@@ -17,7 +17,7 @@
 <%def name="body()">
 	<div class="table_wrapper">
 	<div id="manageheader" class="title">
-		<h1 class="clearfix"><i class="fa fa-music"></i> Manage Manually Changed Albums</h1>
+		<h1 class="clearfix"><i class="fa fa-music"></i> Manage Manually Matched Albums</h1>
 	</div>
 
 	<table class="display" id="artist_table">
@@ -37,30 +37,30 @@
 				old_album_clean = album['AlbumTitle'].replace('&','%26').replace('+', '%2B').replace("'","%27")
 			%>
 			<td id="artist">${album['ArtistName']}<BR>
-				<button id="restore_artist${count_albums}" onClick="restore_Artist(this.id)">(<-) Restore Artist</button>
+				<button id="reset_artist${count_albums}" onClick="reset_Artist(this.id)">(<-) Reset Artist</button>
-				<div id="restore_artist_dialog${count_albums}" title="Restore Artist" style="display:none">
+				<div id="reset_artist_dialog${count_albums}" title="Reset Artist" style="display:none">
 					<table>
-					<tr><td>Are you sure you want to restore Local Artist: ${album['ArtistName']} to unmatched?</td></tr>
+					<tr><td>Are you sure you want to reset Local Artist: ${album['ArtistName']} to unmatched?</td></tr>
 					<tr><td align="right"><BR>
 					%if album['AlbumStatus'] == "Ignored":
-						<button href="#" onclick="doAjaxCall('markManual?action=unignoreArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully restored ${album['ArtistName']} to unmatched">Restore Artist</button>
+						<button href="#" onclick="doAjaxCall('markManual?action=unignoreArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully reset ${album['ArtistName']} to unmatched">Reset Artist</button>
 					%elif album['AlbumStatus'] == "Matched":
-						<button href="#" onclick="doAjaxCall('markManual?action=unmatchArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully restored ${album['ArtistName']} to unmatched">Restore Artist</button>
+						<button href="#" onclick="doAjaxCall('markManual?action=unmatchArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully restored ${album['ArtistName']} to unmatched">Reset Artist</button>
 					%endif
 					</td></tr>
 					</table>
 				</div>
 			</td>
 			<td id="album">${album['AlbumTitle']}<BR>
-				<button id="restore_album${count_albums}" onClick="restore_Album(this.id)">(<-) Restore Album</button>
+				<button id="reset_album${count_albums}" onClick="reset_Album(this.id)">(<-) Reset Album</button>
-				<div id="restore_album_dialog${count_albums}" title="Restore Album" style="display:none">
+				<div id="reset_album_dialog${count_albums}" title="Reset Album" style="display:none">
 					<table>
-					<tr><td>Are you sure you want to restore Local Album: ${album['AlbumTitle']} to unmatched?</td></tr>
+					<tr><td>Are you sure you want to reset Local Album: ${album['AlbumTitle']} to unmatched?</td></tr>
 					<tr><td align="right"><BR>
 					%if album['AlbumStatus'] == "Ignored":
-						<button href="#" onclick="doAjaxCall('markManual?action=unignoreAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully restored ${album['AlbumTitle']} to unmatched">Restore Album</button>
+						<button href="#" onclick="doAjaxCall('markManual?action=unignoreAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully reset ${album['AlbumTitle']} to unmatched">Reset Album</button>
 					%elif album['AlbumStatus'] == "Matched":
-						<button href="#" onclick="doAjaxCall('markManual?action=unmatchAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully restored ${album['AlbumTitle']} to unmatched">Restore Album</button>
+						<button href="#" onclick="doAjaxCall('markManual?action=unmatchAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully reset ${album['AlbumTitle']} to unmatched">Reset Album</button>
 					%endif
 					</td></tr>
 					</table>
@@ -107,15 +107,15 @@
 		initActions();
 	});
 
-	function restore_Artist(clicked_id) {
+	function reset_Artist(clicked_id) {
-		n=clicked_id.replace("restore_artist","");
+		n=clicked_id.replace("reset_artist","");
-		$("#restore_artist_dialog"+n).dialog();
+		$("#reset_artist_dialog"+n).dialog();
 		return false;
 	}
 
-	function restore_Album(clicked_id) {
+	function reset_Album(clicked_id) {
-		n=clicked_id.replace("restore_album","");
+		n=clicked_id.replace("reset_album","");
-		$("#restore_album_dialog"+n).dialog();
+		$("#reset_album_dialog"+n).dialog();
 		return false;
 	}
 

headphones/__init__.py

@@ -126,12 +126,15 @@ def initialize(config_file):
             try:
                 os.makedirs(CONFIG.LOG_DIR)
             except OSError:
-                if VERBOSE:
+                CONFIG.LOG_DIR = None
-                    sys.stderr.write(
+
-                        'Unable to create the log directory. Logging to screen only.\n')
+                if not QUIET:
+                    sys.stderr.write("Unable to create the log directory. " \
+                        "Logging to screen only.\n")
 
         # Start the logger, disable console if needed
-        logger.initLogger(console=not QUIET, verbose=VERBOSE)
+        logger.initLogger(console=not QUIET, log_dir=CONFIG.LOG_DIR,
+            verbose=VERBOSE)
 
         if not CONFIG.CACHE_DIR:
             # Put the cache dir in the data dir for now
@@ -169,7 +172,7 @@ def initialize(config_file):
                     fp.write(CURRENT_VERSION)
             except IOError as e:
                 logger.error("Unable to write current version to file '%s': %s",
-                    version_lock_file, e)
+                             version_lock_file, e)
 
         # Check for new versions
         if CONFIG.CHECK_GITHUB_ON_STARTUP:
@@ -267,7 +270,8 @@ def initialize_scheduler():
         torrentfinished
 
     with SCHED_LOCK:
-        # Remove all jobs
+        # Remove all jobs first, because this method is also invoked when the
+        # settings are saved.
         count = len(SCHED.get_jobs())
 
         if count > 0:
@@ -297,11 +301,11 @@ def initialize_scheduler():
         # Remove Torrent + data if Post Processed and finished Seeding
         if CONFIG.TORRENT_REMOVAL_INTERVAL > 0:
             SCHED.add_job(torrentfinished.checkTorrentFinished,
-                trigger=IntervalTrigger(
+                          trigger=IntervalTrigger(
-                    minutes=CONFIG.TORRENT_REMOVAL_INTERVAL))
+                              minutes=CONFIG.TORRENT_REMOVAL_INTERVAL))
 
         # Start scheduler
-        logger.info("(Re-)Scheduling background tasks")
+        logger.info("(Re-)Scheduled %d background tasks", len(SCHED.get_jobs()))
         SCHED.start()
 
 

headphones/config.py

@@ -42,6 +42,8 @@ _CONFIG_DEFINITIONS = {
     'CONFIG_VERSION': (str, 'General', '0'),
     'CORRECT_METADATA': (int, 'General', 0),
     'CUE_SPLIT': (int, 'General', 1),
+    'CUE_SPLIT_FLAC_PATH': (str, 'General', ''),
+    'CUE_SPLIT_SHNTOOL_PATH': (str, 'General', ''),
     'CUSTOMHOST': (str, 'General', 'localhost'),
     'CUSTOMPORT': (int, 'General', 5000),
     'CUSTOMSLEEP': (int, 'General', 1),
@@ -91,6 +93,8 @@ _CONFIG_DEFINITIONS = {
     'HTTP_ROOT': (str, 'General', '/'),
     'HTTP_USERNAME': (str, 'General', ''),
     'IGNORED_WORDS': (str, 'General', ''),
+    'IGNORED_FOLDERS': (list, 'Advanced', []),
+    'IGNORED_FILES': (list, 'Advanced', []),
     'INCLUDE_EXTRAS': (int, 'General', 0),
     'INTERFACE': (str, 'General', 'default'),
     'JOURNAL_MODE': (str, 'Advanced', 'wal'),
@@ -146,6 +150,9 @@ _CONFIG_DEFINITIONS = {
     'PIRATEBAY': (int, 'Piratebay', 0),
     'PIRATEBAY_PROXY_URL': (str, 'Piratebay', ''),
     'PIRATEBAY_RATIO': (str, 'Piratebay', ''),
+    'OLDPIRATEBAY': (int, 'Old Piratebay', 0),
+    'OLDPIRATEBAY_URL': (str, 'Old Piratebay', ''),
+    'OLDPIRATEBAY_RATIO': (str, 'Old Piratebay', ''),
     'PLEX_CLIENT_HOST': (str, 'Plex', ''),
     'PLEX_ENABLED': (int, 'Plex', 0),
     'PLEX_NOTIFY': (int, 'Plex', 0),

headphones/cuesplit.py

@@ -72,11 +72,13 @@ CUE_META = None
 
 
 def check_splitter(command):
-    '''Check xld or shntools installed'''
+    '''Check xld or shntool installed'''
     try:
         env = os.environ.copy()
         if 'xld' in command:
             env['PATH'] += os.pathsep + '/Applications'
+        elif headphones.CONFIG.CUE_SPLIT_FLAC_PATH:
+            command = os.path.join(headphones.CONFIG.CUE_SPLIT_SHNTOOL_PATH, 'shntool')
         devnull = open(os.devnull)
         subprocess.Popen([command], stdout=devnull, stderr=devnull, env=env).communicate()
     except OSError as e:
@@ -103,6 +105,8 @@ def split_baby(split_file, split_cmd):
     env = os.environ.copy()
     if 'xld' in split_cmd:
         env['PATH'] += os.pathsep + '/Applications'
+    elif headphones.CONFIG.CUE_SPLIT_FLAC_PATH:
+        env['PATH'] += os.pathsep + headphones.CONFIG.CUE_SPLIT_FLAC_PATH
 
     process = subprocess.Popen(split_cmd, startupinfo=startupinfo,
 
@@ -424,7 +428,7 @@ class CueFile(File):
             elif t >= 1:
                 t_index = self.tracks[t]['index']
                 content += t_index[1]
-                if (t < len(self.tracks) - 1):
+                if t < (len(self.tracks) - 1):
                     content += '\n'
         return content
 
@@ -546,8 +550,8 @@ def split(albumpath):
     for _cue in base_dir.filter('CueFile'):
         for _wave in base_dir.filter('WaveFile'):
             if _cue.header['file'] == _wave.name:
-                logger.info('CUE Sheet found: {0}'.format(_cue.name))
+                logger.info('CUE Sheet found: %s', _cue.name)
-                logger.info('Music file found: {0}'.format(_wave.name))
+                logger.info('Music file found: %s', _wave.name)
                 cue = _cue
                 wave = _wave
     # if no perfect match found then try without extensions
@@ -556,9 +560,9 @@ def split(albumpath):
         for _cue in base_dir.filter('CueFile'):
             for _wave in base_dir.filter('WaveFile'):
                 if ''.join(os.path.splitext(_cue.header['file'])[:-1]) == _wave.name_name:
-                    logger.info('Possible CUE Sheet found: {0}'.format(_cue.name))
+                    logger.info('Possible CUE Sheet found: %s', _cue.name)
-                    logger.info('CUE Sheet refers music file: {0}'.format(_cue.header['file']))
+                    logger.info('CUE Sheet refers music file: %s', _cue.header['file'])
-                    logger.info('Possible Music file found: {0}'.format(_wave.name))
+                    logger.info('Possible Music file found: %s', _wave.name)
                     cue = _cue
                     wave = _wave
                     cue.header['file'] = wave.name
@@ -588,9 +592,9 @@ def split(albumpath):
             splitter = 'shntool'
 
     if splitter == 'shntool' and not check_splitter(splitter):
-            raise ValueError('Command not found, ensure shntools with FLAC or xld (OS X) installed')
+            raise ValueError('Command not found, ensure shntool with FLAC or xld (OS X) installed')
 
-    # Determine if file can be split (only flac allowed for shntools)
+    # Determine if file can be split (only flac allowed for shntool)
     if 'xld' in splitter and wave.name_ext not in WAVE_FILE_TYPE_BY_EXTENSION.keys() or \
             wave.type not in SHNTOOL_COMPATIBLE:
         raise ValueError('Cannot split, audio file has unsupported extension')
@@ -627,7 +631,11 @@ def split(albumpath):
         with open(SPLIT_FILE_NAME, mode='w') as split_file:
             split_file.write(cue.breakpoints())
 
-        cmd = ['shntool']
+        if headphones.CONFIG.CUE_SPLIT_SHNTOOL_PATH:
+            cmd = [os.path.join(headphones.CONFIG.CUE_SPLIT_SHNTOOL_PATH, 'shntool')]
+        else:
+            cmd = ['shntool']
+
         cmd.extend(['split'])
         cmd.extend(['-f'])
         cmd.extend([SPLIT_FILE_NAME])
@@ -641,14 +649,14 @@ def split(albumpath):
         # tag FLAC files
         if split and CUE_META.count_tracks() == len(base_dir.tracks(ext='.flac', split=True)):
             for t in base_dir.tracks(ext='.flac', split=True):
-                logger.info('Tagging {0}...'.format(t.name))
+                logger.info('Tagging %s...', t.name)
                 t.tag()
 
         # rename FLAC files
         if split and CUE_META.count_tracks() == len(base_dir.tracks(ext='.flac', split=True)):
             for t in base_dir.tracks(ext='.flac', split=True):
                 if t.name != t.filename():
-                    logger.info('Renaming {0} to {1}...'.format(t.name, t.filename()))
+                    logger.info('Renaming %s to %s...', t.name, t.filename())
                     os.rename(t.name, t.filename())
 
         os.remove(ALBUM_META_FILE_NAME)

headphones/db.py

@@ -96,11 +96,16 @@ class DBConnection:
 
         genParams = lambda myDict: [x + " = ?" for x in myDict.keys()]
 
-        query = "UPDATE " + tableName + " SET " + ", ".join(genParams(valueDict)) + " WHERE " + " AND ".join(genParams(keyDict))
+        update_query = "UPDATE " + tableName + " SET " + ", ".join(genParams(valueDict)) + " WHERE " + " AND ".join(genParams(keyDict))
 
-        self.action(query, valueDict.values() + keyDict.values())
+        self.action(update_query, valueDict.values() + keyDict.values())
 
         if self.connection.total_changes == changesBefore:
-            query = "INSERT INTO " + tableName + " (" + ", ".join(valueDict.keys() + keyDict.keys()) + ")" + \
+            insert_query = (
-                        " VALUES (" + ", ".join(["?"] * len(valueDict.keys() + keyDict.keys())) + ")"
+                "INSERT INTO " + tableName + " (" + ", ".join(valueDict.keys() + keyDict.keys()) + ")" +
-            self.action(query, valueDict.values() + keyDict.values())
+                " VALUES (" + ", ".join(["?"] * len(valueDict.keys() + keyDict.keys())) + ")"
+            )
+            try:
+                self.action(insert_query, valueDict.values() + keyDict.values())
+            except sqlite3.IntegrityError:
+                logger.info('Queries failed: %s and %s', update_query, insert_query)

headphones/getXldProfile.py

@@ -1,32 +1,31 @@
 import os.path
-import plistlib
+import biplist
-import sys
-import xml.parsers.expat as expat
 from headphones import logger
 
 
 def getXldProfile(xldProfile):
+
     xldProfileNotFound = xldProfile
-    expandedPath = os.path.expanduser('~/Library/Preferences/jp.tmkk.XLD.plist')
+
-    try:
+    expanded = os.path.expanduser('~/Library/Preferences/jp.tmkk.XLD.plist')
-        preferences = plistlib.Plist.fromFile(expandedPath)
+    if not os.path.isfile(expanded):
-    except (expat.ExpatError):
+        logger.warn("Could not find xld preferences at: %s", expanded)
-        os.system("/usr/bin/plutil -convert xml1 %s" % expandedPath)
-        try:
-            preferences = plistlib.Plist.fromFile(expandedPath)
-        except (ImportError):
-            os.system("/usr/bin/plutil -convert binary1 %s" % expandedPath)
-            logger.info('The plist at "%s" has a date in it, and therefore is not useable.', expandedPath)
-            return(xldProfileNotFound, None, None)
-    except (ImportError):
-        logger.info('The plist at "%s" has a date in it, and therefore is not useable.', expandedPath)
-    except:
-        logger.info('Unexpected error: %s', sys.exc_info()[0])
         return(xldProfileNotFound, None, None)
 
-    xldProfile = xldProfile.lower()
+    # Get xld preferences plist
-    profiles = preferences.get('Profiles')
+    try:
+        preferences = biplist.readPlist(expanded)
+    except (biplist.InvalidPlistException, biplist.NotBinaryPlistException), e:
+        logger.error("Error reading xld preferences plist: %s", e)
+        return(xldProfileNotFound, None, None)
 
+    if not isinstance(preferences, dict):
+        logger.error("Error reading xld preferences plist, not a dict: %r", preferences)
+        return(xldProfileNotFound, None, None)
+
+    profiles = preferences.get('Profiles', []) # pylint:disable=E1103
+
+    xldProfile = xldProfile.lower()
     for profile in profiles:
 
         profilename = profile.get('XLDProfileManager_ProfileName')

headphones/helpers.py

@@ -13,17 +13,19 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import os
+from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
-import re
-import time
-import shutil
-import datetime
-import headphones
-import unicodedata
-import sys
 
 from operator import itemgetter
-from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
+
+import unicodedata
+import headphones
+import datetime
+import fnmatch
+import shutil
+import time
+import sys
+import re
+import os
 
 # Modified from https://github.com/Verrus/beets-plugin-featInTitle
 RE_FEATURING = re.compile(r"[fF]t\.|[fF]eaturing|[fF]eat\.|\b[wW]ith\b|&|vs\.")
@@ -65,7 +67,8 @@ def latinToAscii(unicrap):
     """
     From couch potato
     """
-    xlate = {0xc0: 'A', 0xc1: 'A', 0xc2: 'A', 0xc3: 'A', 0xc4: 'A', 0xc5: 'A',
+    xlate = {
+        0xc0: 'A', 0xc1: 'A', 0xc2: 'A', 0xc3: 'A', 0xc4: 'A', 0xc5: 'A',
         0xc6: 'Ae', 0xc7: 'C',
         0xc8: 'E', 0xc9: 'E', 0xca: 'E', 0xcb: 'E', 0x86: 'e',
         0xcc: 'I', 0xcd: 'I', 0xce: 'I', 0xcf: 'I',
@@ -90,7 +93,7 @@ def latinToAscii(unicrap):
         0xb9: '{^1}', 0xba: '{^o}', 0xbb: '>>',
         0xbc: '{1/4}', 0xbd: '{1/2}', 0xbe: '{3/4}', 0xbf: '?',
         0xd7: '*', 0xf7: '/'
-        }
+    }
 
     r = ''
     for i in unicrap:
@@ -168,13 +171,20 @@ def mb_to_bytes(mb_str):
 def piratesize(size):
     split = size.split(" ")
     factor = float(split[0])
-    unit = split[1]
+    unit = split[1].upper()
+    
     if unit == 'MiB':
         size = factor * 1048576
+    elif unit == 'MB':
+        size = factor * 1000000
     elif unit == 'GiB':
         size = factor * 1073741824
+    elif unit == 'GB':
+        size = factor * 1000000000
     elif unit == 'KiB':
         size = factor * 1024
+    elif unit == 'KB':
+        size = factor * 1000
     elif unit == "B":
         size = factor
     else:
@@ -329,6 +339,44 @@ def expand_subfolders(f):
     return media_folders
 
 
+def path_match_patterns(path, patterns):
+    """
+    Check if a path matches one or more patterns. The whole path will be
+    matched be matched against the patterns.
+    """
+
+    for pattern in patterns:
+        if fnmatch.fnmatch(path, pattern):
+            return True
+
+    # No match
+    return False
+
+
+def path_filter_patterns(paths, patterns, root=None):
+    """
+    Scan for ignored paths based on glob patterns. Note that the whole path
+    will be matched, therefore paths should only contain the relative paths.
+
+    The root is optional, and only used for producing meaningful debug info.
+    """
+
+    from headphones import logger
+
+    ignored = 0
+
+    for path in paths[:]:
+        if path_match_patterns(path, patterns):
+            logger.debug("Path ignored by pattern: %s",
+                os.path.join(root or "", path))
+
+            ignored += 1
+            paths.remove(path)
+
+    # Return number of ignored paths
+    return ignored
+
+
 def extract_data(s):
 
     s = s.replace('_', ' ')
@@ -473,8 +521,8 @@ def get_downloaded_track_list(albumpath):
 
 def preserve_torrent_direcory(albumpath):
     """
-     Copy torrent directory to headphones-modified to keep files for seeding.
+    Copy torrent directory to headphones-modified to keep files for seeding.
-     """
+    """
     from headphones import logger
     new_folder = os.path.join(albumpath, 'headphones-modified'.encode(headphones.SYS_ENCODING, 'replace'))
     logger.info("Copying files to 'headphones-modified' subfolder to preserve downloaded files for seeding")

headphones/importer.py

@@ -48,7 +48,7 @@ def artistlist_to_mbids(artistlist, forced=False):
 
     for artist in artistlist:
 
-        if not artist and not (artist == ' '):
+        if not artist and artist != ' ':
             continue
 
         # If adding artists through Manage New Artists, they're coming through as non-unicode (utf-8?)
@@ -466,7 +466,7 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                     myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', rg['id']])
                     marked_as_downloaded = True
             else:
-                if ((have_track_count / float(total_track_count)) >= (headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0)):
+                if (have_track_count / float(total_track_count)) >= (headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0):
                     myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', rg['id']])
                     marked_as_downloaded = True
 
@@ -600,6 +600,17 @@ def addReleaseById(rid, rgid=None):
             newValueDict['IncludeExtras'] = 1
             newValueDict['Extras'] = headphones.CONFIG.EXTRAS
 
+        if 'title' in release_dict:
+            newValueDict['LatestAlbum'] = release_dict['title']
+        elif 'rg_title' in release_dict:
+            newValueDict['LatestAlbum'] = release_dict['rg_title']
+
+        if 'date' in release_dict:
+            newValueDict['ReleaseDate'] = release_dict['date']
+
+        if rgid:
+            newValueDict['AlbumID'] = rgid
+
         myDB.upsert("artists", newValueDict, controlValueDict)
 
     elif not artist_exists and not release_dict:

headphones/lastfm.py

@@ -13,10 +13,9 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import time
 import random
-import threading
 import headphones
+import headphones.lock
 
 from headphones import db, logger, request
 
@@ -28,7 +27,7 @@ ENTRY_POINT = "http://ws.audioscrobbler.com/2.0/"
 API_KEY = "395e6ec6bb557382fc41fde867bce66f"
 
 # Required for API request limit
-lock = threading.Lock()
+lastfm_lock = headphones.lock.TimedLock(REQUEST_LIMIT)
 
 
 def request_lastfm(method, **kwargs):
@@ -49,8 +48,7 @@ def request_lastfm(method, **kwargs):
     logger.debug("Calling Last.FM method: %s", method)
     logger.debug("Last.FM call parameters: %s", kwargs)
 
-    data = request.request_json(ENTRY_POINT, timeout=TIMEOUT, params=kwargs,
+    data = request.request_json(ENTRY_POINT, timeout=TIMEOUT, params=kwargs, lock=lastfm_lock)
-        rate_limit=(lock, REQUEST_LIMIT))
 
     # Parse response and check for errors.
     if not data:
@@ -73,7 +71,6 @@ def getSimilar():
 
     for result in results[:12]:
         data = request_lastfm("artist.getsimilar", mbid=result["ArtistId"])
-        time.sleep(10)
 
         if data and "similarartists" in data:
             artists = data["similarartists"]["artist"]

headphones/librarysync.py

@@ -20,10 +20,10 @@ from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
 
 from headphones import db, logger, helpers, importer, lastfm
 
-# You can scan a single directory and append it to the current library by specifying append=True, ArtistID & ArtistName
+# You can scan a single directory and append it to the current library by
-
+# specifying append=True, ArtistID and ArtistName.
-
+def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
-def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=False):
+    cron=False):
 
     if cron and not headphones.CONFIG.LIBRARYSCAN:
         return
@@ -79,20 +79,17 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
     latest_subdirectory = []
 
     for r, d, f in helpers.walk_directory(dir):
-        # Need to abuse slicing to get a copy of the list, doing it directly
+        # Filter paths based on config. Note that these methods work directly
-        # will skip the element after a deleted one using a list comprehension
+        # on the inputs
-        # will not work correctly for nested subdirectories (os.walk keeps its
+        helpers.path_filter_patterns(d, headphones.CONFIG.IGNORED_FOLDERS, r)
-        # original list)
+        helpers.path_filter_patterns(f, headphones.CONFIG.IGNORED_FILES, r)
-        for directory in d[:]:
-            if directory.startswith("."):
-                d.remove(directory)
 
         for files in f:
             # MEDIA_FORMATS = music file extensions, e.g. mp3, flac, etc
             if any(files.lower().endswith('.' + x.lower()) for x in headphones.MEDIA_FORMATS):
-
                 subdirectory = r.replace(dir, '')
                 latest_subdirectory.append(subdirectory)
+
                 if file_count == 0 and r.replace(dir, '') != '':
                     logger.info("[%s] Now scanning subdirectory %s" % (dir.decode(headphones.SYS_ENCODING, 'replace'), subdirectory.decode(headphones.SYS_ENCODING, 'replace')))
                 elif latest_subdirectory[file_count] != latest_subdirectory[file_count - 1] and file_count != 0:
@@ -353,6 +350,7 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
     if not append:
         update_album_status()
         lastfm.getSimilar()
+
     logger.info('Library scan complete')
 
 #ADDED THIS SECTION TO MARK ALBUMS AS DOWNLOADED IF ARTISTS ARE ADDED EN MASSE BEFORE LIBRARY IS SCANNED

headphones/lock.py

@@ -0,0 +1,88 @@
+"""
+Locking-related classes
+"""
+
+import headphones.logger
+import time
+import threading
+import Queue
+
+
+class TimedLock(object):
+    """
+    Enforce request rate limit if applicable. This uses the lock so there
+    is synchronized access to the API. When N threads enter this method, the
+    first will pass trough, since there there was no last request recorded.
+    The last request time will be set. Then, the second thread will unlock,
+    and see that the last request was X seconds ago. It will sleep
+    (request_limit - X) seconds, and then continue. Then the third one will
+    unblock, and so on. After all threads finish, the total time will at
+    least be (N * request_limit) seconds. If some request takes longer than
+    request_limit seconds, the next unblocked thread will wait less.
+    """
+
+    def __init__(self, minimum_delta=0):
+        """
+        Set up the lock
+        """
+        self.lock = threading.Lock()
+        self.last_used = 0
+        self.minimum_delta = minimum_delta
+        self.queue = Queue.Queue()
+
+    def __enter__(self):
+        """
+        Called when with lock: is invoked
+        """
+        self.lock.acquire()
+        delta = time.time() - self.last_used
+        sleep_amount = self.minimum_delta - delta
+        if sleep_amount >= 0:
+            # zero sleeps give the cpu a chance to task-switch
+            headphones.logger.info('Sleeping %s (interval)', sleep_amount)
+            time.sleep(sleep_amount)
+        while not self.queue.empty():
+            try:
+                seconds = self.queue.get(False)
+                headphones.logger.info('Sleeping %s (queued)', seconds)
+                time.sleep(seconds)
+            except Queue.Empty:
+                continue
+            self.queue.task_done()
+
+    def __exit__(self, type, value, traceback):
+        """
+        Called when exiting the with block.
+        """
+        self.last_used = time.time()
+        self.lock.release()
+
+    def snooze(self, seconds):
+        """
+        Asynchronously add time to the next request.
+        Can be called outside
+        of the lock context, but it is possible for the next lock holder
+        to not check the queue until after something adds time to it.
+        """
+        # we use a queue so that we don't have to synchronize
+        # across threads and with or without locks
+        headphones.logger.info('Adding %s to queue', seconds)
+        self.queue.add(seconds)
+
+
+class FakeLock(object):
+    """
+    If no locking or request throttling is needed, use this
+    """
+
+    def __enter__(self):
+        """
+        Do nothing on enter
+        """
+        pass
+
+    def __exit__(self, type, value, traceback):
+        """
+        Do nothing on exit
+        """
+        pass

headphones/logger.py

@@ -39,7 +39,6 @@ logger = logging.getLogger("headphones")
 # Global queue for multiprocessing logging
 queue = None
 
-
 class LogListHandler(logging.Handler):
     """
     Log handler for Web UI.
@@ -112,7 +111,7 @@ def initMultiprocessing():
     threading.current_thread().name = multiprocessing.current_process().name
 
 
-def initLogger(console=False, verbose=False):
+def initLogger(console=False, log_dir=False, verbose=False):
     """
     Setup logging for Headphones. It uses the logger instance with the name
     'headphones'. Three log handlers are added:
@@ -121,7 +120,8 @@ def initLogger(console=False, verbose=False):
     * LogListHandler: for Web UI
     * StreamHandler: for console (if console)
 
-    Console logging is only enabled if console is set to True.
+    Console logging is only enabled if console is set to True. This method can
+    be invoked multiple times, during different stages of Headphones.
     """
 
     # Close and remove old handlers. This is required to reinit the loggers
@@ -139,22 +139,23 @@ def initLogger(console=False, verbose=False):
     logger.propagate = False
     logger.setLevel(logging.DEBUG if verbose else logging.INFO)
 
-    # Setup file logger
-    filename = os.path.join(headphones.CONFIG.LOG_DIR, FILENAME)
-
-    file_formatter = logging.Formatter('%(asctime)s - %(levelname)-7s :: %(threadName)s : %(message)s', '%d-%b-%Y %H:%M:%S')
-    file_handler = handlers.RotatingFileHandler(filename, maxBytes=MAX_SIZE, backupCount=MAX_FILES)
-    file_handler.setLevel(logging.DEBUG)
-    file_handler.setFormatter(file_formatter)
-
-    logger.addHandler(file_handler)
-
     # Add list logger
     loglist_handler = LogListHandler()
     loglist_handler.setLevel(logging.DEBUG)
 
     logger.addHandler(loglist_handler)
 
+    # Setup file logger
+    if log_dir:
+        filename = os.path.join(log_dir, FILENAME)
+
+        file_formatter = logging.Formatter('%(asctime)s - %(levelname)-7s :: %(threadName)s : %(message)s', '%d-%b-%Y %H:%M:%S')
+        file_handler = handlers.RotatingFileHandler(filename, maxBytes=MAX_SIZE, backupCount=MAX_FILES)
+        file_handler.setLevel(logging.DEBUG)
+        file_handler.setFormatter(file_formatter)
+
+        logger.addHandler(file_handler)
+
     # Setup console logger
     if console:
         console_formatter = logging.Formatter('%(asctime)s - %(levelname)s :: %(threadName)s : %(message)s', '%d-%b-%Y %H:%M:%S')

headphones/mb.py

@@ -16,10 +16,9 @@
 
 from headphones import logger, db, helpers
 
-import time
-import threading
 import headphones
 import musicbrainzngs
+import headphones.lock
 
 try:
     # pylint:disable=E0611
@@ -30,7 +29,7 @@ except ImportError:
     # Python 2.6.x fallback, from libs
     from ordereddict import OrderedDict
 
-mb_lock = threading.Lock()
+mb_lock = headphones.lock.TimedLock(0)
 
 # Quick fix to add mirror switching on the fly. Need to probably return the mbhost & mbport that's
 # being used, so we can send those values to the log
@@ -60,11 +59,14 @@ def startmb():
 
     musicbrainzngs.set_useragent("headphones", "0.0", "https://github.com/rembo10/headphones")
     musicbrainzngs.set_hostname(mbhost + ":" + str(mbport))
+
+    # Their rate limiting should be redundant to our lock
     if sleepytime == 0:
         musicbrainzngs.set_rate_limit(False)
     else:
         #calling it with an it ends up blocking all requests after the first
         musicbrainzngs.set_rate_limit(limit_or_interval=float(sleepytime))
+        mb_lock.minimum_delta = sleepytime
 
     # Add headphones credentials
     if headphones.CONFIG.MIRROR == "headphones":
@@ -79,335 +81,315 @@ def startmb():
 
 
 def findArtist(name, limit=1):
+     artistlist = []
+     artistResults = None
 
-    with mb_lock:
+     chars = set('!?*-')
-        artistlist = []
+     if any((c in chars) for c in name):
-        artistResults = None
+         name = '"' + name + '"'
 
-        chars = set('!?*-')
+     criteria = {'artist': name.lower()}
-        if any((c in chars) for c in name):
-            name = '"' + name + '"'
 
-        criteria = {'artist': name.lower()}
+     with mb_lock:
+         try:
+             artistResults = musicbrainzngs.search_artists(limit=limit, **criteria)['artist-list']
+         except musicbrainzngs.WebServiceError as e:
+             logger.warn('Attempt to query MusicBrainz for %s failed (%s)' % (name, str(e)))
+             mb_lock.snooze(5)
 
-        try:
+     if not artistResults:
-            artistResults = musicbrainzngs.search_artists(limit=limit, **criteria)['artist-list']
+         return False
-        except musicbrainzngs.WebServiceError as e:
+     for result in artistResults:
-            logger.warn('Attempt to query MusicBrainz for %s failed (%s)' % (name, str(e)))
+         if 'disambiguation' in result:
-            time.sleep(5)
+             uniquename = unicode(result['sort-name'] + " (" + result['disambiguation'] + ")")
-
+         else:
-        if not artistResults:
+             uniquename = unicode(result['sort-name'])
-            return False
+         if result['name'] != uniquename and limit == 1:
-        for result in artistResults:
+             logger.info('Found an artist with a disambiguation: %s - doing an album based search' % name)
-            if 'disambiguation' in result:
+             artistdict = findArtistbyAlbum(name)
-                uniquename = unicode(result['sort-name'] + " (" + result['disambiguation'] + ")")
+             if not artistdict:
-            else:
+                 logger.info('Cannot determine the best match from an artist/album search. Using top match instead')
-                uniquename = unicode(result['sort-name'])
+                 artistlist.append({
-            if result['name'] != uniquename and limit == 1:
+                 # Just need the artist id if the limit is 1
-                logger.info('Found an artist with a disambiguation: %s - doing an album based search' % name)
+                 #    'name':             unicode(result['sort-name']),
-                artistdict = findArtistbyAlbum(name)
+                 #    'uniquename':        uniquename,
-                if not artistdict:
+                     'id': unicode(result['id']),
-                    logger.info('Cannot determine the best match from an artist/album search. Using top match instead')
+                 #    'url':                 unicode("http://musicbrainz.org/artist/" + result['id']),#probably needs to be changed
-                    artistlist.append({
+                 #    'score':            int(result['ext:score'])
-                    # Just need the artist id if the limit is 1
+                     })
-                    #    'name':             unicode(result['sort-name']),
+             else:
-                    #    'uniquename':        uniquename,
+                 artistlist.append(artistdict)
-                        'id': unicode(result['id']),
+         else:
-                    #    'url':                 unicode("http://musicbrainz.org/artist/" + result['id']),#probably needs to be changed
+             artistlist.append({
-                    #    'score':            int(result['ext:score'])
+                     'name': unicode(result['sort-name']),
-                        })
+                     'uniquename': uniquename,
-                else:
+                     'id': unicode(result['id']),
-                    artistlist.append(artistdict)
+                     'url': unicode("http://musicbrainz.org/artist/" + result['id']),#probably needs to be changed
-            else:
+                     'score': int(result['ext:score'])
-                artistlist.append({
+                     })
-                        'name': unicode(result['sort-name']),
+     return artistlist
-                        'uniquename': uniquename,
-                        'id': unicode(result['id']),
-                        'url': unicode("http://musicbrainz.org/artist/" + result['id']),#probably needs to be changed
-                        'score': int(result['ext:score'])
-                        })
-        return artistlist
 
 
 def findRelease(name, limit=1, artist=None):
+    releaselist = []
+    releaseResults = None
+
+    # additional artist search
+    if not artist and ':' in name:
+        name, artist = name.rsplit(":", 1)
+
+    chars = set('!?*-')
+    if any((c in chars) for c in name):
+        name = '"' + name + '"'
+    if artist and any((c in chars) for c in artist):
+        artist = '"' + artist + '"'
 
     with mb_lock:
-        releaselist = []
-        releaseResults = None
-
-        # additional artist search
-        if not artist and ':' in name:
-            name, artist = name.rsplit(":", 1)
-
-        chars = set('!?*-')
-        if any((c in chars) for c in name):
-            name = '"' + name + '"'
-        if artist and any((c in chars) for c in artist):
-            artist = '"' + artist + '"'
-
         try:
             releaseResults = musicbrainzngs.search_releases(query=name, limit=limit, artist=artist)['release-list']
         except musicbrainzngs.WebServiceError as e: #need to update exceptions
             logger.warn('Attempt to query MusicBrainz for "%s" failed: %s' % (name, str(e)))
-            time.sleep(5)
+            mb_lock.snooze(5)
 
-        if not releaseResults:
+    if not releaseResults:
-            return False
+        return False
 
-        for result in releaseResults:
+    for result in releaseResults:
 
-            title = result['title']
+        title = result['title']
-            if 'disambiguation' in result:
+        if 'disambiguation' in result:
-                title += ' (' + result['disambiguation'] + ')'
+            title += ' (' + result['disambiguation'] + ')'
 
-            # Get formats and track counts
+        # Get formats and track counts
-            format_dict = OrderedDict()
+        format_dict = OrderedDict()
-            formats = ''
+        formats = ''
-            tracks = ''
+        tracks = ''
-            if 'medium-list' in result:
+        if 'medium-list' in result:
-                for medium in result['medium-list']:
+            for medium in result['medium-list']:
-                    if 'format' in medium:
+                if 'format' in medium:
-                        format = medium['format']
+                    format = medium['format']
-                        if format not in format_dict:
+                    if format not in format_dict:
-                            format_dict[format] = 0
+                        format_dict[format] = 0
-                        format_dict[format] += 1
+                    format_dict[format] += 1
-                    if 'track-count' in medium:
+                if 'track-count' in medium:
-                        if tracks:
+                    if tracks:
-                            tracks += ' + '
+                        tracks += ' + '
-                        tracks += str(medium['track-count'])
+                    tracks += str(medium['track-count'])
-                for format, count in format_dict.items():
+            for format, count in format_dict.items():
-                    if formats:
+                if formats:
-                        formats += ' + '
+                    formats += ' + '
-                    if count > 1:
+                if count > 1:
-                        formats += str(count) + 'x'
+                    formats += str(count) + 'x'
-                    formats += format
+                formats += format
 
-            rg_type = ''
+        rg_type = ''
-            if 'type' in result['release-group']:
+        if 'type' in result['release-group']:
-                rg_type = result['release-group']['type']
+            rg_type = result['release-group']['type']
-                if rg_type == 'Album' and 'secondary-type-list' in result['release-group']:
+            if rg_type == 'Album' and 'secondary-type-list' in result['release-group']:
-                    secondary_type = result['release-group']['secondary-type-list'][0]
+                secondary_type = result['release-group']['secondary-type-list'][0]
-                    if secondary_type != rg_type:
+                if secondary_type != rg_type:
-                        rg_type = secondary_type
+                    rg_type = secondary_type
 
-            releaselist.append({
+        releaselist.append({
-                        'uniquename': unicode(result['artist-credit'][0]['artist']['name']),
+                    'uniquename': unicode(result['artist-credit'][0]['artist']['name']),
-                        'title': unicode(title),
+                    'title': unicode(title),
-                        'id': unicode(result['artist-credit'][0]['artist']['id']),
+                    'id': unicode(result['artist-credit'][0]['artist']['id']),
-                        'albumid': unicode(result['id']),
+                    'albumid': unicode(result['id']),
-                        'url': unicode("http://musicbrainz.org/artist/" + result['artist-credit'][0]['artist']['id']),#probably needs to be changed
+                    'url': unicode("http://musicbrainz.org/artist/" + result['artist-credit'][0]['artist']['id']),#probably needs to be changed
-                        'albumurl': unicode("http://musicbrainz.org/release/" + result['id']),#probably needs to be changed
+                    'albumurl': unicode("http://musicbrainz.org/release/" + result['id']),#probably needs to be changed
-                        'score': int(result['ext:score']),
+                    'score': int(result['ext:score']),
-                        'date': unicode(result['date']) if 'date' in result else '',
+                    'date': unicode(result['date']) if 'date' in result else '',
-                        'country': unicode(result['country']) if 'country' in result else '',
+                    'country': unicode(result['country']) if 'country' in result else '',
-                        'formats': unicode(formats),
+                    'formats': unicode(formats),
-                        'tracks': unicode(tracks),
+                    'tracks': unicode(tracks),
-                        'rgid': unicode(result['release-group']['id']),
+                    'rgid': unicode(result['release-group']['id']),
-                        'rgtype': unicode(rg_type)
+                    'rgtype': unicode(rg_type)
-                        })
+                    })
-        return releaselist
+    return releaselist
 
 
 def getArtist(artistid, extrasonly=False):
-
+    artist_dict = {}
-    with mb_lock:
+    artist = None
-        artist_dict = {}
+    try:
-
+        limit = 200
-        artist = None
+        with mb_lock:
-
-        try:
-            limit = 200
             artist = musicbrainzngs.get_artist_by_id(artistid)['artist']
-            newRgs = None
+        newRgs = None
-            artist['release-group-list'] = []
+        artist['release-group-list'] = []
-            while newRgs is None or len(newRgs) >= limit:
+        while newRgs is None or len(newRgs) >= limit:
-                newRgs = musicbrainzngs.browse_release_groups(artistid, release_type="album", offset=len(artist['release-group-list']), limit=limit)['release-group-list']
+            with mb_lock:
-                artist['release-group-list'] += newRgs
+                newRgs = musicbrainzngs.browse_release_groups(
-        except musicbrainzngs.WebServiceError as e:
+                    artistid,
-            logger.warn('Attempt to retrieve artist information from MusicBrainz failed for artistid: %s (%s)' % (artistid, str(e)))
+                    release_type="album",
-            time.sleep(5)
+                    offset=len(artist['release-group-list']),
-        except Exception as e:
+                    limit=limit)
-            pass
+            newRgs = newRgs['release-group-list']
+            artist['release-group-list'] += newRgs
+    except musicbrainzngs.WebServiceError as e:
+        logger.warn('Attempt to retrieve artist information from MusicBrainz failed for artistid: %s (%s)' % (artistid, str(e)))
+        mb_lock.snooze(5)
+    except Exception as e:
+        pass
 
-        if not artist:
+    if not artist:
-            return False
+        return False
 
-        #if 'disambiguation' in artist:
+    artist_dict['artist_name'] = unicode(artist['name'])
-        #    uniquename = unicode(artist['sort-name'] + " (" + artist['disambiguation'] + ")")
-        #else:
-        #    uniquename = unicode(artist['sort-name'])
 
-        artist_dict['artist_name'] = unicode(artist['name'])
+    releasegroups = []
 
-        # Not using the following values anywhere yet so we don't need to grab them.
+    if not extrasonly:
-        # Was causing an exception to be raised if they didn't exist.
+        for rg in artist['release-group-list']:
-        #
+            if "secondary-type-list" in rg.keys(): #only add releases without a secondary type
-        #artist_dict['artist_sortname'] = unicode(artist['sort-name'])
+                continue
-        #artist_dict['artist_uniquename'] = uniquename
+            releasegroups.append({
-        #artist_dict['artist_type'] = unicode(artist['type'])
+                'title': unicode(rg['title']),
+                'id': unicode(rg['id']),
+                'url': u"http://musicbrainz.org/release-group/" + rg['id'],
+                'type': unicode(rg['type'])
+            })
 
-        #artist_dict['artist_begindate'] = None
+    # See if we need to grab extras. Artist specific extras take precedence over global option
-        #artist_dict['artist_enddate'] = None
+    # Global options are set when adding a new artist
-        #if 'life-span' in artist:
+    myDB = db.DBConnection()
-        #    if 'begin' in artist['life-span']:
-        #        artist_dict['artist_begindate'] = unicode(artist['life-span']['begin'])
-        #    if 'end' in artist['life-span']:
-        #        artist_dict['artist_enddate'] = unicode(artist['life-span']['end'])
 
-        releasegroups = []
+    try:
+        db_artist = myDB.action('SELECT IncludeExtras, Extras from artists WHERE ArtistID=?', [artistid]).fetchone()
+        includeExtras = db_artist['IncludeExtras']
+    except IndexError:
+        includeExtras = False
+
+    if includeExtras:
+
+        # Need to convert extras string from something like '2,5.6' to ['ep','live','remix'] (append new extras to end)
+        if db_artist['Extras']:
+            extras = map(int, db_artist['Extras'].split(','))
+        else:
+            extras = []
+        extras_list = headphones.POSSIBLE_EXTRAS
+
+        includes = []
+
+        i = 1
+        for extra in extras_list:
+            if i in extras:
+                includes.append(extra)
+            i += 1
+
+        for include in includes:
+
+            mb_extras_list = []
+
+            try:
+                limit = 200
+                newRgs = None
+                while newRgs is None or len(newRgs) >= limit:
+                    with mb_lock:
+                        newRgs = musicbrainzngs.browse_release_groups(
+                            artistid, release_type=include, offset=len(mb_extras_list), limit=limit)
+                    newRgs = newRgs['release-group-list']
+                    mb_extras_list += newRgs
+            except musicbrainzngs.WebServiceError as e:
+                logger.warn('Attempt to retrieve artist information from MusicBrainz failed for artistid: %s (%s)' % (artistid, str(e)))
+                mb_lock.snooze(5)
+
+            for rg in mb_extras_list:
+                rg_type = rg['type']
+                if rg_type == 'Album' and 'secondary-type-list' in rg:
+                    secondary_type = rg['secondary-type-list'][0]
+                    if secondary_type != rg_type:
+                        rg_type = secondary_type
 
-        if not extrasonly:
-            for rg in artist['release-group-list']:
-                if "secondary-type-list" in rg.keys(): #only add releases without a secondary type
-                    continue
                 releasegroups.append({
-                            'title': unicode(rg['title']),
+                    'title': unicode(rg['title']),
-                            'id': unicode(rg['id']),
+                    'id': unicode(rg['id']),
-                            'url': u"http://musicbrainz.org/release-group/" + rg['id'],
+                    'url': u"http://musicbrainz.org/release-group/" + rg['id'],
-                            'type': unicode(rg['type'])
+                    'type': unicode(rg_type)
-                    })
+                })
-
+    artist_dict['releasegroups'] = releasegroups
-        # See if we need to grab extras. Artist specific extras take precedence over global option
+    return artist_dict
-        # Global options are set when adding a new artist
-        myDB = db.DBConnection()
-
-        try:
-            db_artist = myDB.action('SELECT IncludeExtras, Extras from artists WHERE ArtistID=?', [artistid]).fetchone()
-            includeExtras = db_artist['IncludeExtras']
-        except IndexError:
-            includeExtras = False
-
-        if includeExtras:
-
-            # Need to convert extras string from something like '2,5.6' to ['ep','live','remix'] (append new extras to end)
-            if db_artist['Extras']:
-                extras = map(int, db_artist['Extras'].split(','))
-            else:
-                extras = []
-            extras_list = headphones.POSSIBLE_EXTRAS
-
-            includes = []
-
-            i = 1
-            for extra in extras_list:
-                if i in extras:
-                    includes.append(extra)
-                i += 1
-
-            for include in includes:
-
-                mb_extras_list = []
-
-                try:
-                    limit = 200
-                    newRgs = None
-                    while newRgs is None or len(newRgs) >= limit:
-                        newRgs = musicbrainzngs.browse_release_groups(artistid, release_type=include, offset=len(mb_extras_list), limit=limit)['release-group-list']
-                        mb_extras_list += newRgs
-                except musicbrainzngs.WebServiceError as e:
-                    logger.warn('Attempt to retrieve artist information from MusicBrainz failed for artistid: %s (%s)' % (artistid, str(e)))
-                    time.sleep(5)
-
-                for rg in mb_extras_list:
-
-                    rg_type = rg['type']
-                    if rg_type == 'Album' and 'secondary-type-list' in rg:
-                        secondary_type = rg['secondary-type-list'][0]
-                        if secondary_type != rg_type:
-                            rg_type = secondary_type
-
-                    releasegroups.append({
-                            'title': unicode(rg['title']),
-                            'id': unicode(rg['id']),
-                            'url': u"http://musicbrainz.org/release-group/" + rg['id'],
-                            'type': unicode(rg_type)
-                        })
-
-        artist_dict['releasegroups'] = releasegroups
-
-        return artist_dict
 
 
 def getReleaseGroup(rgid):
     """
     Returns a list of releases in a release group
     """
-    with mb_lock:
+    releaseGroup = None
+    try:
+        with mb_lock:
+            releaseGroup = musicbrainzngs.get_release_group_by_id(
+                rgid, ["artists", "releases", "media", "discids", ])
+            releaseGroup = releaseGroup['release-group']
+    except musicbrainzngs.WebServiceError as e:
+        logger.warn('Attempt to retrieve information from MusicBrainz for release group "%s" failed (%s)' % (rgid, str(e)))
+        mb_lock.snooze(5)
 
-        releaseGroup = None
+    if not releaseGroup:
-
+        return False
-        try:
+    else:
-            releaseGroup = musicbrainzngs.get_release_group_by_id(rgid, ["artists", "releases", "media", "discids", ])['release-group']
+        return releaseGroup['release-list']
-        except musicbrainzngs.WebServiceError as e:
-            logger.warn('Attempt to retrieve information from MusicBrainz for release group "%s" failed (%s)' % (rgid, str(e)))
-            time.sleep(5)
-
-        if not releaseGroup:
-            return False
-        else:
-            return releaseGroup['release-list']
 
 
 def getRelease(releaseid, include_artist_info=True):
     """
     Deep release search to get track info
     """
-    with mb_lock:
+    release = {}
+    results = None
 
-        release = {}
+    try:
-        results = None
+        with mb_lock:
-
-        try:
             if include_artist_info:
                 results = musicbrainzngs.get_release_by_id(releaseid, ["artists", "release-groups", "media", "recordings"]).get('release')
             else:
                 results = musicbrainzngs.get_release_by_id(releaseid, ["media", "recordings"]).get('release')
-        except musicbrainzngs.WebServiceError as e:
+    except musicbrainzngs.WebServiceError as e:
-            logger.warn('Attempt to retrieve information from MusicBrainz for release "%s" failed (%s)' % (releaseid, str(e)))
+        logger.warn('Attempt to retrieve information from MusicBrainz for release "%s" failed (%s)' % (releaseid, str(e)))
-            time.sleep(5)
+        mb_lock.snooze(5)
 
-        if not results:
+    if not results:
-            return False
+        return False
 
-        release['title'] = unicode(results['title'])
+    release['title'] = unicode(results['title'])
-        release['id'] = unicode(results['id'])
+    release['id'] = unicode(results['id'])
-        release['asin'] = unicode(results['asin']) if 'asin' in results else None
+    release['asin'] = unicode(results['asin']) if 'asin' in results else None
-        release['date'] = unicode(results['date']) if 'date' in results else None
+    release['date'] = unicode(results['date']) if 'date' in results else None
-        try:
+    try:
-            release['format'] = unicode(results['medium-list'][0]['format'])
+        release['format'] = unicode(results['medium-list'][0]['format'])
-        except:
+    except:
-            release['format'] = u'Unknown'
+        release['format'] = u'Unknown'
 
-        try:
+    try:
-            release['country'] = unicode(results['country'])
+        release['country'] = unicode(results['country'])
-        except:
+    except:
-            release['country'] = u'Unknown'
+        release['country'] = u'Unknown'
 
-        if include_artist_info:
+    if include_artist_info:
 
-            if 'release-group' in results:
+        if 'release-group' in results:
-                release['rgid'] = unicode(results['release-group']['id'])
+            release['rgid'] = unicode(results['release-group']['id'])
-                release['rg_title'] = unicode(results['release-group']['title'])
+            release['rg_title'] = unicode(results['release-group']['title'])
-                try:
+            try:
-                    release['rg_type'] = unicode(results['release-group']['type'])
+                release['rg_type'] = unicode(results['release-group']['type'])
 
-                    if release['rg_type'] == 'Album' and 'secondary-type-list' in results['release-group']:
+                if release['rg_type'] == 'Album' and 'secondary-type-list' in results['release-group']:
-                        secondary_type = unicode(results['release-group']['secondary-type-list'][0])
+                    secondary_type = unicode(results['release-group']['secondary-type-list'][0])
-                        if secondary_type != release['rg_type']:
+                    if secondary_type != release['rg_type']:
-                            release['rg_type'] = secondary_type
+                        release['rg_type'] = secondary_type
 
-                except KeyError:
+            except KeyError:
-                    release['rg_type'] = u'Unknown'
+                release['rg_type'] = u'Unknown'
 
-            else:
+        else:
-                logger.warn("Release " + releaseid + "had no ReleaseGroup associated")
+            logger.warn("Release " + releaseid + "had no ReleaseGroup associated")
 
-            release['artist_name'] = unicode(results['artist-credit'][0]['artist']['name'])
+        release['artist_name'] = unicode(results['artist-credit'][0]['artist']['name'])
-            release['artist_id'] = unicode(results['artist-credit'][0]['artist']['id'])
+        release['artist_id'] = unicode(results['artist-credit'][0]['artist']['id'])
 
-        release['tracks'] = getTracksFromRelease(results)
+    release['tracks'] = getTracksFromRelease(results)
 
-        return release
+    return release
 
 
 def get_new_releases(rgid, includeExtras=False, forcefull=False):
@@ -418,7 +400,12 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
         limit = 100
         newResults = None
         while newResults is None or len(newResults) >= limit:
-            newResults = musicbrainzngs.browse_releases(release_group=rgid, includes=['artist-credits', 'labels', 'recordings', 'release-groups', 'media'], limit=limit, offset=len(results))
+            with mb_lock:
+                newResults = musicbrainzngs.browse_releases(
+                    release_group=rgid,
+                    includes=['artist-credits', 'labels', 'recordings', 'release-groups', 'media'],
+                    limit=limit,
+                    offset=len(results))
             if 'release-list' not in newResults:
                 break #may want to raise an exception here instead ?
             newResults = newResults['release-list']
@@ -426,7 +413,7 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
 
     except musicbrainzngs.WebServiceError as e:
         logger.warn('Attempt to retrieve information from MusicBrainz for release group "%s" failed (%s)' % (rgid, str(e)))
-        time.sleep(5)
+        mb_lock.snooze(5)
         return False
 
     if not results or len(results) == 0:
@@ -565,11 +552,11 @@ def get_new_releases(rgid, includeExtras=False, forcefull=False):
                 logger.info('[%s] Existing release %s (%s) updated' % (release['ArtistName'], release['AlbumTitle'], rel_id_check))
             else:
                 logger.info('[%s] New release %s (%s) added' % (release['ArtistName'], release['AlbumTitle'], rel_id_check))
-    if force_repackage1 == 1:
+        if force_repackage1 == 1:
-        num_new_releases = -1
+            num_new_releases = -1
-        logger.info('[%s] Forcing repackage of %s, since dB releases have been removed' % (release['ArtistName'], release_title))
+            logger.info('[%s] Forcing repackage of %s, since dB releases have been removed' % (release['ArtistName'], release_title))
-    else:
+        else:
-        num_new_releases = num_new_releases
+            num_new_releases = num_new_releases
 
     return num_new_releases
 
@@ -614,10 +601,11 @@ def findArtistbyAlbum(name):
     results = None
 
     try:
-        results = musicbrainzngs.search_release_groups(term).get('release-group-list')
+        with mb_lock:
+            results = musicbrainzngs.search_release_groups(term).get('release-group-list')
     except musicbrainzngs.WebServiceError as e:
         logger.warn('Attempt to query MusicBrainz for %s failed (%s)' % (name, str(e)))
-        time.sleep(5)
+        mb_lock.snooze(5)
 
     if not results:
         return False
@@ -656,11 +644,11 @@ def findAlbumID(artist=None, album=None):
             if any((c in chars) for c in album):
                 album = '"' + album + '"'
             criteria = {'release': album.lower()}
-
+        with mb_lock:
-        results = musicbrainzngs.search_release_groups(limit=1, **criteria).get('release-group-list')
+            results = musicbrainzngs.search_release_groups(limit=1, **criteria).get('release-group-list')
     except musicbrainzngs.WebServiceError as e:
         logger.warn('Attempt to query MusicBrainz for %s - %s failed (%s)' % (artist, album, str(e)))
-        time.sleep(5)
+        mb_lock.snooze(5)
 
     if not results:
         return False

headphones/music_encoder.py

@@ -14,12 +14,12 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
 import os
-import headphones
-import shutil
 import time
+import shutil
+import subprocess
+import headphones
 import multiprocessing
 
-import subprocess
 from headphones import logger
 from beets.mediafile import MediaFile
 
@@ -36,6 +36,8 @@ def encode(albumPath):
         if not xldFormat:
             logger.error('Details for xld profile \'%s\' not found, files will not be re-encoded', xldProfile)
             return None
+    else:
+        xldProfile = None
 
     tempDirEncode = os.path.join(albumPath, "temp")
     musicFiles = []
@@ -64,7 +66,7 @@ def encode(albumPath):
                     xldInfoMusic = MediaFile(xldMusicFile)
                     encoderFormat = xldFormat
 
-                if (headphones.CONFIG.ENCODERLOSSLESS):
+                if headphones.CONFIG.ENCODERLOSSLESS:
                     ext = os.path.normpath(os.path.splitext(music)[1].lstrip(".")).lower()
                     if not use_xld and ext == 'flac' or use_xld and (ext != xldFormat and (xldInfoMusic.bitrate / 1000 > 400)):
                         musicFiles.append(os.path.join(r, music))
@@ -116,7 +118,7 @@ def encode(albumPath):
             if not any(music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.' + x) for x in ["mp3", "wav"]):
                 logger.warn('Lame cannot encode %s format for %s, use ffmpeg', os.path.splitext(music)[1], music)
             else:
-                if (music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.mp3') and (int(infoMusic.bitrate / 1000) <= headphones.CONFIG.BITRATE)):
+                if music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.mp3') and (int(infoMusic.bitrate / 1000) <= headphones.CONFIG.BITRATE):
                     logger.info('%s has bitrate <= %skb, will not be re-encoded', music, headphones.CONFIG.BITRATE)
                 else:
                     encode = True
@@ -126,14 +128,14 @@ def encode(albumPath):
                     logger.warn('Cannot re-encode .ogg %s', music.decode(headphones.SYS_ENCODING, 'replace'))
                 else:
                     encode = True
-            elif (headphones.CONFIG.ENCODEROUTPUTFORMAT == 'mp3' or headphones.CONFIG.ENCODEROUTPUTFORMAT == 'm4a'):
+            elif headphones.CONFIG.ENCODEROUTPUTFORMAT == 'mp3' or headphones.CONFIG.ENCODEROUTPUTFORMAT == 'm4a':
-                if (music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.' + headphones.CONFIG.ENCODEROUTPUTFORMAT) and (int(infoMusic.bitrate / 1000) <= headphones.CONFIG.BITRATE)):
+                if music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.' + headphones.CONFIG.ENCODEROUTPUTFORMAT) and (int(infoMusic.bitrate / 1000) <= headphones.CONFIG.BITRATE):
                     logger.info('%s has bitrate <= %skb, will not be re-encoded', music, headphones.CONFIG.BITRATE)
                 else:
                     encode = True
         # encode
         if encode:
-            job = (encoder, music, musicTempFiles[i], albumPath)
+            job = (encoder, music, musicTempFiles[i], albumPath, xldProfile)
             jobs.append(job)
         else:
             musicFiles[i] = None
@@ -242,22 +244,16 @@ def command_map(args):
         return False
 
 
-def command(encoder, musicSource, musicDest, albumPath):
+def command(encoder, musicSource, musicDest, albumPath, xldProfile):
     """
     Encode a given music file with a certain encoder. Returns True on success,
     or False otherwise.
     """
-    use_xld = headphones.CONFIG.ENCODER == 'xld'
 
     startMusicTime = time.time()
     cmd = []
 
-    # Return if xld details not found
+    if xldProfile:
-    if use_xld:
-        (xldProfile, xldFormat, xldBitrate) = getXldProfile.getXldProfile(headphones.CONFIG.XLDPROFILE)
-        if not xldFormat:
-            logger.error('Details for xld profile \'%s\' not found, files will not be re-encoded', xldProfile)
-            return None
         xldDestDir = os.path.split(musicDest)[0]
         cmd = [encoder]
         cmd.extend([musicSource])

headphones/notifiers.py

@@ -290,7 +290,7 @@ class LMS(object):
             return response['result']
         except:
             logger.warn('LMS returned error: %s' % response['error'])
-            return
+            return response['error']
 
     def update(self):
 
@@ -300,7 +300,7 @@ class LMS(object):
             logger.info('Sending library rescan command to LMS @ ' + host)
             request = self._sendjson(host)
 
-            if not request:
+            if request:
                 logger.warn('Error sending rescan request to LMS')
 
 
@@ -444,16 +444,15 @@ class PUSHBULLET(object):
 
         http_handler = HTTPSConnection("api.pushbullet.com")
 
-        data = {'device_iden': headphones.CONFIG.PUSHBULLET_DEVICEID,
+        data = {'type': "note",
-                'type': "note",
                 'title': "Headphones",
                 'body': message.encode("utf-8")}
 
         http_handler.request("POST",
-                                "/api/pushes",
+                                "/v2/pushes",
-                                headers={'Content-type': "application/x-www-form-urlencoded",
+                                headers={'Content-type': "application/json",
                                             'Authorization': 'Basic %s' % base64.b64encode(headphones.CONFIG.PUSHBULLET_APIKEY + ":")},
-                                body=urlencode(data))
+                                body=json.dumps(data))
         response = http_handler.getresponse()
         request_status = response.status
         logger.debug(u"PushBullet response status: %r" % request_status)

headphones/nzbget.py

@@ -59,7 +59,7 @@ def sendNZB(nzb):
         return False
 
     except xmlrpclib.ProtocolError, e:
-        if (e.errmsg == "Unauthorized"):
+        if e.errmsg == "Unauthorized":
             logger.error(u"NZBget password is incorrect.")
         else:
             logger.error(u"Protocol Error: " + e.errmsg)

headphones/postprocessor.py

@@ -33,16 +33,14 @@ postprocessor_lock = threading.Lock()
 
 
 def checkFolder():
+    logger.info("Checking download folder for completed downloads")
 
     with postprocessor_lock:
-
         myDB = db.DBConnection()
         snatched = myDB.select('SELECT * from snatched WHERE Status="Snatched"')
 
         for album in snatched:
-
             if album['FolderName']:
-
                 if album['Kind'] == 'nzb':
                     download_dir = headphones.CONFIG.DOWNLOAD_DIR
                 else:
@@ -50,13 +48,14 @@ def checkFolder():
 
                 album_path = os.path.join(download_dir, album['FolderName']).encode(headphones.SYS_ENCODING, 'replace')
                 logger.info("Checking if %s exists" % album_path)
+
                 if os.path.exists(album_path):
                     logger.info('Found "' + album['FolderName'] + '" in ' + album['Kind'] + ' download folder. Verifying....')
                     verify(album['AlbumID'], album_path, album['Kind'])
-
             else:
                 logger.info("No folder name found for " + album['Title'])
 
+    logger.info("Checking download folder finished")
 
 def verify(albumid, albumpath, Kind=None, forced=False):
 
@@ -329,9 +328,10 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
            headphones.CONFIG.MOVE_FILES:
 
             try:
-                with open(downloaded_track, "a+b"):
+                with open(downloaded_track, "a+b") as fp:
-                    pass
+                    fp.seek(0)
-            except IOError:
+            except IOError as e:
+                logger.debug("Write check exact error: %s", e)
                 logger.error("Track file is not writeable. This is required " \
                     "for some post processing steps: %s. Not continuing.",
                     downloaded_track.decode(headphones.SYS_ENCODING, "replace"))
@@ -1043,11 +1043,13 @@ def renameUnprocessedFolder(albumpath):
 
 def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
 
+    ignored = 0
+
     if album_dir:
         folders = [album_dir.encode(headphones.SYS_ENCODING, 'replace')]
-
     else:
         download_dirs = []
+
         if dir:
             download_dirs.append(dir.encode(headphones.SYS_ENCODING, 'replace'))
         if headphones.CONFIG.DOWNLOAD_DIR and not dir:
@@ -1066,7 +1068,13 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
             if not os.path.isdir(download_dir):
                 logger.warn('Directory %s does not exist. Skipping', download_dir)
                 continue
-            for folder in os.listdir(download_dir):
+
+            # Scan for subfolders
+            subfolders = os.listdir(download_dir)
+            ignored += helpers.path_filter_patterns(subfolders,
+                headphones.CONFIG.IGNORED_FOLDERS, root=download_dir)
+
+            for folder in subfolders:
                 path_to_folder = os.path.join(download_dir, folder)
 
                 if os.path.isdir(path_to_folder):
@@ -1079,8 +1087,9 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
 
     # Log number of folders
     if folders:
-        logger.info('Found %i folders to process.', len(folders))
         logger.debug('Expanded post processing folders: %s', folders)
+        logger.info('Found %d folders to process (%d ignored).',
+            len(folders), ignored)
     else:
         logger.info('Found no folders to process. Aborting.')
         return
@@ -1221,4 +1230,4 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
         logger.info("Couldn't parse '%s' into any valid format. If adding " \
             "albums from another source, they must be in an 'Artist - Album " \
             "[Year]' format, or end with the musicbrainz release group id.",
-            folder_basename)
+            folder_basename)

headphones/request.py

@@ -18,18 +18,19 @@ from headphones import logger
 from xml.dom import minidom
 from bs4 import BeautifulSoup
 
-import time
 import requests
 import feedparser
 import headphones
+import headphones.lock
 import collections
 
 # Dictionary with last request times, for rate limiting.
 last_requests = collections.defaultdict(int)
+fake_lock = headphones.lock.FakeLock()
 
 
 def request_response(url, method="get", auto_raise=True,
-    whitelist_status_code=None, rate_limit=None, **kwargs):
+    whitelist_status_code=None, lock=fake_lock, **kwargs):
     """
     Convenient wrapper for `requests.get', which will capture the exceptions and
     log them. On success, the Response object is returned. In case of a
@@ -54,36 +55,11 @@ def request_response(url, method="get", auto_raise=True,
     # requests to apply more magic per method. See lib/requests/api.py.
     request_method = getattr(requests, method.lower())
 
-    # Enfore request rate limit if applicable. This uses the lock so there
-    # is synchronized access to the API. When N threads enter this method, the
-    # first will pass trough, since there there was no last request recorded.
-    # The last request time will be set. Then, the second thread will unlock,
-    # and see that the last request was X seconds ago. It will sleep
-    # (request_limit - X) seconds, and then continue. Then the third one will
-    # unblock, and so on. After all threads finished, the total time will at
-    # least be (N * request_limit) seconds. If some request takes longer than
-    # request_limit seconds, the next unblocked thread will wait less.
-    if rate_limit:
-        lock, request_limit = rate_limit
-
-        with lock:
-            delta = time.time() - last_requests[lock]
-            limit = int(1.0 / request_limit)
-
-            if delta < request_limit:
-                logger.debug("Sleeping %.2f seconds for request, limit " \
-                    "is %d req/sec.", request_limit - delta, limit)
-
-                # Sleep the remaining time
-                time.sleep(request_limit - delta)
-
-            # Update last request time.
-            last_requests[lock] = time.time()
-
     try:
         # Request URL and wait for response
-        logger.debug("Requesting URL via %s method: %s", method.upper(), url)
+        with lock:
-        response = request_method(url, **kwargs)
+            logger.debug("Requesting URL via %s method: %s", method.upper(), url)
+            response = request_method(url, **kwargs)
 
         # If status code != OK, then raise exception, except if the status code
         # is white listed.
@@ -92,19 +68,25 @@ def request_response(url, method="get", auto_raise=True,
                 try:
                     response.raise_for_status()
                 except:
-                    logger.debug("Response status code %d is not white " \
+                    logger.debug(
-                         "listed, raised exception", response.status_code)
+                        "Response status code %d is not white "
+                        "listed, raised exception", response.status_code)
                     raise
         elif auto_raise:
             response.raise_for_status()
 
         return response
+    except requests.exceptions.SSLError:
+        logger.error("Unable to connect to remote host because of a SSL " \
+            "error. It's likely the remote certificate is untrusted by your " \
+            "system. This check can be disabled (advanced users only).")
     except requests.ConnectionError:
-        logger.error("Unable to connect to remote host. Check if the remote " \
+        logger.error(
+            "Unable to connect to remote host. Check if the remote "
             "host is up and running.")
     except requests.Timeout:
-        logger.error("Request timed out. The remote host did not respeond " \
+        logger.error(
-            "timely.")
+            "Request timed out. The remote host did not respond timely.")
     except requests.HTTPError as e:
         if e.response is not None:
             if e.response.status_code >= 500:

headphones/searcher.py

@@ -28,6 +28,7 @@ import re
 import string
 import shutil
 import random
+import urllib
 import headphones
 import subprocess
 import unicodedata
@@ -128,7 +129,8 @@ def read_torrent_name(torrent_file, default_name=None):
 
 def calculate_torrent_hash(link, data=None):
     """
-    Calculate the torrent hash from a magnet link or data.
+    Calculate the torrent hash from a magnet link or data. Raises a ValueError
+    when it cannot create a torrent hash given the input data.
     """
 
     if link.startswith("magnet:"):
@@ -142,7 +144,7 @@ def calculate_torrent_hash(link, data=None):
         raise ValueError("Cannot calculate torrent hash without magnet link " \
             "or data")
 
-    return torrent_hash
+    return torrent_hash.upper()
 
 
 def get_seed_ratio(provider):
@@ -159,6 +161,8 @@ def get_seed_ratio(provider):
         seed_ratio = headphones.CONFIG.WHATCD_RATIO
     elif provider == 'The Pirate Bay':
         seed_ratio = headphones.CONFIG.PIRATEBAY_RATIO
+    elif provider == 'Old Pirate Bay':
+        seed_ratio = headphones.CONFIG.OLDPIRATEBAY_RATIO
     elif provider == 'Waffles.fm':
         seed_ratio = headphones.CONFIG.WAFFLES_RATIO
     elif provider == 'Mininova':
@@ -175,16 +179,18 @@ def get_seed_ratio(provider):
     return seed_ratio
 
 
-def searchforalbum(albumid=None, new=False, losslessOnly=False, choose_specific_download=False):
+def searchforalbum(albumid=None, new=False, losslessOnly=False,
+    choose_specific_download=False):
+
+    logger.info('Searching for wanted albums')
     myDB = db.DBConnection()
 
     if not albumid:
         results = myDB.select('SELECT * from albums WHERE Status="Wanted" OR Status="Wanted Lossless"')
 
         for album in results:
-
             if not album['AlbumTitle'] or not album['ArtistName']:
-                logger.warn('Skipping release %s. No title available' % album['AlbumID'])
+                logger.warn('Skipping release %s. No title available', album['AlbumID'])
                 continue
 
             new = True
@@ -196,7 +202,6 @@ def searchforalbum(albumid=None, new=False, losslessOnly=False, choose_specific_
             do_sorted_search(album, new, losslessOnly)
 
     elif albumid and choose_specific_download:
-
         album = myDB.action('SELECT * from albums WHERE AlbumID=?', [albumid]).fetchone()
         logger.info('Searching for "%s - %s"' % (album['ArtistName'], album['AlbumTitle']))
         results = do_sorted_search(album, new, losslessOnly, choose_specific_download=True)
@@ -207,14 +212,14 @@ def searchforalbum(albumid=None, new=False, losslessOnly=False, choose_specific_
         logger.info('Searching for "%s - %s" since it was marked as wanted' % (album['ArtistName'], album['AlbumTitle']))
         do_sorted_search(album, new, losslessOnly)
 
-    logger.info('Search for Wanted albums complete')
+    logger.info('Search for wanted albums complete')
 
 
 def do_sorted_search(album, new, losslessOnly, choose_specific_download=False):
 
     NZB_PROVIDERS = (headphones.CONFIG.HEADPHONES_INDEXER or headphones.CONFIG.NEWZNAB or headphones.CONFIG.NZBSORG or headphones.CONFIG.OMGWTFNZBS)
     NZB_DOWNLOADERS = (headphones.CONFIG.SAB_HOST or headphones.CONFIG.BLACKHOLE_DIR or headphones.CONFIG.NZBGET_HOST)
-    TORRENT_PROVIDERS = (headphones.CONFIG.KAT or headphones.CONFIG.PIRATEBAY or headphones.CONFIG.MININOVA or headphones.CONFIG.WAFFLES or headphones.CONFIG.RUTRACKER or headphones.CONFIG.WHATCD)
+    TORRENT_PROVIDERS = (headphones.CONFIG.KAT or headphones.CONFIG.PIRATEBAY or headphones.CONFIG.OLDPIRATEBAY or headphones.CONFIG.MININOVA or headphones.CONFIG.WAFFLES or headphones.CONFIG.RUTRACKER or headphones.CONFIG.WHATCD)
 
     results = []
     myDB = db.DBConnection()
@@ -258,6 +263,10 @@ def do_sorted_search(album, new, losslessOnly, choose_specific_download=False):
     if choose_specific_download:
         return results
 
+    # Filter all results that do not comply
+    results = [ result for result in results if result[5] ]
+
+    # Sort the remaining results
     sorted_search_results = sort_search_results(results, album, new, albumlength)
 
     if not sorted_search_results:
@@ -500,9 +509,8 @@ def searchNZB(album, new=False, losslessOnly=False, albumlength=None):
                         title = item.title
                         size = int(item.links[1]['length'])
 
-                        resultlist.append((title, size, url, provider, 'nzb'))
+                        resultlist.append((title, size, url, provider, 'nzb', True))
                         logger.info('Found %s. Size: %s' % (title, helpers.bytes_to_mb(size)))
-
                     except Exception as e:
                         logger.error(u"An unknown error occurred trying to parse the feed: %s" % e)
 
@@ -572,7 +580,7 @@ def searchNZB(album, new=False, losslessOnly=False, albumlength=None):
                             size = int(item.links[1]['length'])
                             if all(word.lower() in title.lower() for word in term.split()):
                                 logger.info('Found %s. Size: %s' % (title, helpers.bytes_to_mb(size)))
-                                resultlist.append((title, size, url, provider, 'nzb'))
+                                resultlist.append((title, size, url, provider, 'nzb', True))
                             else:
                                 logger.info('Skipping %s, not all search term words found' % title)
 
@@ -593,7 +601,7 @@ def searchNZB(album, new=False, losslessOnly=False, albumlength=None):
             logger.info("Album type is audiobook/spokenword. Using audiobook category")
 
         # Request results
-        logger.info('Parsing results from nzbs.org')
+        logger.info('Requesting from nzbs.org')
 
         headers = {'User-Agent': USER_AGENT}
         params = {
@@ -606,9 +614,11 @@ def searchNZB(album, new=False, losslessOnly=False, albumlength=None):
 
         data = request.request_feed(
             url='http://beta.nzbs.org/api',
-            params=params, headers=headers
+            params=params, headers=headers,
+            timeout=5
         )
 
+        logger.info('Parsing results from nzbs.org')
         # Process feed
         if data:
             if not len(data.entries):
@@ -620,7 +630,7 @@ def searchNZB(album, new=False, losslessOnly=False, albumlength=None):
                         title = item.title
                         size = int(item.links[1]['length'])
 
-                        resultlist.append((title, size, url, provider, 'nzb'))
+                        resultlist.append((title, size, url, provider, 'nzb', True))
                         logger.info('Found %s. Size: %s' % (title, helpers.bytes_to_mb(size)))
                     except Exception as e:
                         logger.exception("Unhandled exception while parsing feed")
@@ -667,7 +677,7 @@ def searchNZB(album, new=False, losslessOnly=False, albumlength=None):
                         title = item['release']
                         size = int(item['sizebytes'])
 
-                        resultlist.append((title, size, url, provider, 'nzb'))
+                        resultlist.append((title, size, url, provider, 'nzb', True))
                         logger.info('Found %s. Size: %s', title, helpers.bytes_to_mb(size))
                     except Exception as e:
                         logger.exception("Unhandled exception")
@@ -1002,7 +1012,6 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
     global gazelle  # persistent what.cd api object to reduce number of login attempts
 
     # rutracker login
-
     if headphones.CONFIG.RUTRACKER and album:
         rulogin = rutracker.login(headphones.CONFIG.RUTRACKER_USER, headphones.CONFIG.RUTRACKER_PASSWORD)
         if not rulogin:
@@ -1077,7 +1086,7 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
         if headphones.CONFIG.KAT_PROXY_URL:
             providerurl = fix_url(set_proxy(headphones.CONFIG.KAT_PROXY_URL))
         else:
-            providerurl = fix_url("https://kickass.to")
+            providerurl = fix_url("https://kickass.so")
 
         # Build URL
         providerurl = providerurl + "/usearch/" + ka_term
@@ -1123,10 +1132,13 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
                                 rightformat = False
 
                         if rightformat and size < maxsize and minimumseeders < int(seeders):
-                            resultlist.append((title, size, url, provider, 'torrent'))
+                            match = True
                             logger.info('Found %s. Size: %s' % (title, helpers.bytes_to_mb(size)))
                         else:
+                            match = False
                             logger.info('%s is larger than the maxsize, the wrong format or has too little seeders for this category, skipping. (Size: %i bytes, Seeders: %d, Format: %s)', title, size, int(seeders), rightformat)
+
+                        resultlist.append((title, size, url, provider, 'torrent', match))
                     except Exception as e:
                         logger.exception("Unhandled exception in the KAT parser")
 
@@ -1190,14 +1202,13 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
                         desc_match = re.search(r"Size: (\d+)<", item.description)
                         size = int(desc_match.group(1))
                         url = item.link
-                        resultlist.append((title, size, url, provider, 'torrent'))
+                        resultlist.append((title, size, url, provider, 'torrent', True))
                         logger.info('Found %s. Size: %s', title, helpers.bytes_to_mb(size))
                     except Exception as e:
                         logger.error(u"An error occurred while trying to parse the response from Waffles.fm: %s", e)
 
     # rutracker.org
     if headphones.CONFIG.RUTRACKER and rulogin:
-
         provider = "rutracker.org"
 
         # Ignore if release date not specified, results too unpredictable
@@ -1232,7 +1243,7 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
                     title = ru[0].decode('utf-8')
                     size = ru[1]
                     url = ru[2]
-                    resultlist.append((title, size, url, provider, 'torrent'))
+                    resultlist.append((title, size, url, provider, 'torrent', True))
                     logger.info('Found %s. Size: %s' % (title, helpers.bytes_to_mb(size)))
             else:
                 logger.info(u"No valid results found from %s" % (provider))
@@ -1312,7 +1323,7 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
                                    torrent.size,
                                    gazelle.generate_torrent_link(torrent.id),
                                    provider,
-                                   'torrent'))
+                                   'torrent', True))
 
     # Pirate Bay
     if headphones.CONFIG.PIRATEBAY:
@@ -1342,7 +1353,7 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
         # Request content
         logger.info("Searching The Pirate Bay using term: %s", tpb_term)
 
-        headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2049.0 Safari/537.36'}
+        headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2243.2 Safari/537.36'}
         data = request.request_soup(url=providerurl + category, headers=headers)
 
         # Process content
@@ -1378,13 +1389,69 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
                         size = helpers.piratesize(formatted_size)
 
                         if size < maxsize and minimumseeders < seeds and url is not None:
-                            resultlist.append((title, size, url, provider, "torrent"))
+                            match = True
                             logger.info('Found %s. Size: %s' % (title, formatted_size))
                         else:
+                            match = False
                             logger.info('%s is larger than the maxsize or has too little seeders for this category, skipping. (Size: %i bytes, Seeders: %i)' % (title, size, int(seeds)))
+
+                        resultlist.append((title, size, url, provider, "torrent", match))
                     except Exception as e:
                         logger.error(u"An unknown error occurred in the Pirate Bay parser: %s" % e)
 
+    # Old Pirate Bay Compatible
+    if headphones.CONFIG.OLDPIRATEBAY:
+        provider = "Old Pirate Bay"
+        tpb_term = term.replace("!", "")
+
+        # Pick category for torrents
+        if headphones.CONFIG.PREFERRED_QUALITY == 3 or losslessOnly:
+            maxsize = 10000000000
+        elif headphones.CONFIG.PREFERRED_QUALITY == 1 or allow_lossless:
+            maxsize = 10000000000
+        else:
+            maxsize = 300000000
+
+        # Requesting content
+        logger.info("Parsing results from Old Pirate Bay")
+
+        headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2243.2 Safari/537.36'}
+        provider_url = fix_url(headphones.CONFIG.OLDPIRATEBAY_URL) + \
+            "/search.php?" + urllib.urlencode({"q": tpb_term, "iht": 6})
+
+        data = request.request_soup(url=provider_url, headers=headers)
+
+        # Process content
+        if data:
+            rows = data.select('table tbody tr')
+
+            if not rows:
+                logger.info("No results found")
+            else:
+                for item in rows:
+                    try:
+                        links = item.select("td.title-row a")
+
+                        rightformat = True
+                        title = links[1].text
+                        seeds = int(item.select("td.seeders-row")[0].text)
+                        url = links[0]["href"] # Magnet link. The actual download link is not based on the URL
+
+                        formatted_size = item.select("td.size-row")[0].text
+                        size = helpers.piratesize(formatted_size)
+
+                        if size < maxsize and minimumseeders < seeds and url is not None:
+                            match = True
+                            logger.info('Found %s. Size: %s' % (title, formatted_size))
+                        else:
+                            match = False
+                            logger.info('%s is larger than the maxsize or has too little seeders for this category, skipping. (Size: %i bytes, Seeders: %i)' % (title, size, int(seeds)))
+
+                        resultlist.append((title, size, url, provider, "torrent", match))
+                    except Exception as e:
+                        logger.error(u"An unknown error occurred in the Old Pirate Bay parser: %s" % e)
+
+    # Mininova
     if headphones.CONFIG.MININOVA:
         provider = "Mininova"
         providerurl = fix_url("http://www.mininova.org/rss/" + term + "/5")
@@ -1430,12 +1497,15 @@ def searchTorrent(album, new=False, losslessOnly=False, albumlength=None):
                             torrent = request.request_content(url)
                             if not torrent or (int(torrent.find(".mp3")) > 0 and int(torrent.find(".flac")) < 1):
                                 rightformat = False
+
                         if rightformat and size < maxsize and minimumseeders < seeds:
-                            resultlist.append((title, size, url, provider, 'torrent'))
+                            match = True
                             logger.info('Found %s. Size: %s' % (title, helpers.bytes_to_mb(size)))
                         else:
+                            match = False
                             logger.info('%s is larger than the maxsize, the wrong format or has too little seeders for this category, skipping. (Size: %i bytes, Seeders: %i, Format: %s)' % (title, size, int(seeds), rightformat))
 
+                        resultlist.append((title, size, url, provider, 'torrent', match))
                     except Exception as e:
                         logger.exception("Unhandled exception in Mininova Parser")
 
@@ -1474,8 +1544,8 @@ def preprocess(resultlist):
                 headers['Referer'] = 'http://kat.ph/'
             elif result[3] == 'What.cd':
                 headers['User-Agent'] = 'Headphones'
-            elif result[3] == "The Pirate Bay":
+            elif result[3] == "The Pirate Bay" or result[3] == "Old Pirate Bay":
-                headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2049.0 Safari/537.36'
+                headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2243.2 Safari/537.36'
             return request.request_content(url=result[2], headers=headers), result
 
         else:

headphones/torrentfinished.py

@@ -13,21 +13,21 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
+from headphones import db, utorrent, transmission, logger
+
 import threading
 import headphones
-from headphones import db, utorrent, transmission, logger
 
 postprocessor_lock = threading.Lock()
 
-# Remove Torrent + data if Post Processed and finished Seeding
-
-
 def checkTorrentFinished():
+    """
+    Remove Torrent + data if Post Processed and finished Seeding
+    """
 
     logger.info("Checking if any torrents have finished seeding and can be removed")
 
     with postprocessor_lock:
-
         myDB = db.DBConnection()
         results = myDB.select('SELECT * from snatched WHERE Status="Seed_Processed"')
 
@@ -42,3 +42,5 @@ def checkTorrentFinished():
 
             if torrent_removed:
                 myDB.action('DELETE from snatched WHERE status = "Seed_Processed" and AlbumID=?', [albumid])
+
+    logger.info("Checking finished torrents completed")

headphones/transmission.py

@@ -184,4 +184,4 @@ def torrentAction(method, arguments):
         logger.error("Error sending torrent to Transmission")
         return
 
-    return response
+    return response

headphones/updater.py

@@ -20,11 +20,10 @@ def dbUpdate(forcefull=False):
 
     myDB = db.DBConnection()
 
-    activeartists = myDB.select('SELECT ArtistID, ArtistName from artists WHERE Status="Active" or Status="Loading" order by LastUpdated ASC')
+    active_artists = myDB.select('SELECT ArtistID, ArtistName from artists WHERE Status="Active" or Status="Loading" order by LastUpdated ASC')
-    logger.info('Starting update for %i active artists' % len(activeartists))
+    logger.info('Starting update for %i active artists', len(active_artists))
-
-    for artist in activeartists:
 
+    for artist in active_artists:
         artistid = artist[0]
         importer.addArtisttoDB(artistid=artistid, extrasonly=False, forcefull=forcefull)
 

headphones/utorrent.py

@@ -215,7 +215,7 @@ def dirTorrent(hash, cacheid=None, return_name=None):
     cacheid = torrentList['torrentc']
 
     for torrent in torrents:
-        if (torrent[0].lower() == hash):
+        if torrent[0].upper() == hash:
             if not return_name:
                 return torrent[26], cacheid
             else:

headphones/webserve.py

@@ -23,13 +23,17 @@ from mako import exceptions
 
 from operator import itemgetter
 
-import os
+import headphones
-import sys
+import threading
+import cherrypy
+import urllib2
+import hashlib
+import random
+import urllib
 import json
 import time
-import cherrypy
+import sys
-import threading
+import os
-import headphones
 
 try:
     # pylint:disable=E0611
@@ -57,36 +61,35 @@ def serve_template(templatename, **kwargs):
 
 class WebInterface(object):
 
+    @cherrypy.expose
     def index(self):
         raise cherrypy.HTTPRedirect("home")
-    index.exposed = True
 
+    @cherrypy.expose
     def home(self):
         myDB = db.DBConnection()
         artists = myDB.select('SELECT * from artists order by ArtistSortName COLLATE NOCASE')
         return serve_template(templatename="index.html", title="Home", artists=artists)
-    home.exposed = True
 
+    @cherrypy.expose
     def artistPage(self, ArtistID):
         myDB = db.DBConnection()
         artist = myDB.action('SELECT * FROM artists WHERE ArtistID=?', [ArtistID]).fetchone()
-        albums = myDB.select('SELECT * from albums WHERE ArtistID=? order by ReleaseDate DESC', [ArtistID])
 
         # Don't redirect to the artist page until it has the bare minimum info inserted
         # Redirect to the home page if we still can't get it after 5 seconds
         retry = 0
 
-        while retry < 5:
+        while not artist and retry < 5:
-            if not artist:
+            time.sleep(1)
-                time.sleep(1)
+            artist = myDB.action('SELECT * FROM artists WHERE ArtistID=?', [ArtistID]).fetchone()
-                artist = myDB.action('SELECT * FROM artists WHERE ArtistID=?', [ArtistID]).fetchone()
+            retry += 1
-                retry += 1
-            else:
-                break
 
         if not artist:
             raise cherrypy.HTTPRedirect("home")
 
+        albums = myDB.select('SELECT * from albums WHERE ArtistID=? order by ReleaseDate DESC', [ArtistID])
+
         # Serve the extras up as a dict to make things easier for new templates (append new extras to the end)
         extras_list = headphones.POSSIBLE_EXTRAS
         if artist['Extras']:
@@ -105,13 +108,11 @@ class WebInterface(object):
             i += 1
 
         return serve_template(templatename="artist.html", title=artist['ArtistName'], artist=artist, albums=albums, extras=extras_dict)
-    artistPage.exposed = True
 
+    @cherrypy.expose
     def albumPage(self, AlbumID):
         myDB = db.DBConnection()
         album = myDB.action('SELECT * from albums WHERE AlbumID=?', [AlbumID]).fetchone()
-        tracks = myDB.select('SELECT * from tracks WHERE AlbumID=? ORDER BY CAST(TrackNumber AS INTEGER)', [AlbumID])
-        description = myDB.action('SELECT * from descriptions WHERE ReleaseGroupID=?', [AlbumID]).fetchone()
 
         retry = 0
         while retry < 5:
@@ -125,6 +126,9 @@ class WebInterface(object):
         if not album:
             raise cherrypy.HTTPRedirect("home")
 
+        tracks = myDB.select('SELECT * from tracks WHERE AlbumID=? ORDER BY CAST(TrackNumber AS INTEGER)', [AlbumID])
+        description = myDB.action('SELECT * from descriptions WHERE ReleaseGroupID=?', [AlbumID]).fetchone()
+
         if not album['ArtistName']:
             title = ' - '
         else:
@@ -134,8 +138,8 @@ class WebInterface(object):
         else:
             title = title + album['AlbumTitle']
         return serve_template(templatename="album.html", title=title, album=album, tracks=tracks, description=description)
-    albumPage.exposed = True
 
+    @cherrypy.expose
     def search(self, name, type):
         if len(name) == 0:
             raise cherrypy.HTTPRedirect("home")
@@ -144,13 +148,15 @@ class WebInterface(object):
         else:
             searchresults = mb.findRelease(name, limit=100)
         return serve_template(templatename="searchresults.html", title='Search Results for: "' + name + '"', searchresults=searchresults, name=name, type=type)
-    search.exposed = True
 
+    @cherrypy.expose
     def addArtist(self, artistid):
-        threading.Thread(target=importer.addArtisttoDB, args=[artistid]).start()
+        thread = threading.Thread(target=importer.addArtisttoDB, args=[artistid])
+        thread.start()
+        thread.join(1)
         raise cherrypy.HTTPRedirect("artistPage?ArtistID=%s" % artistid)
-    addArtist.exposed = True
 
+    @cherrypy.expose
     def getExtras(self, ArtistID, newstyle=False, **kwargs):
         # if calling this function without the newstyle, they're using the old format
         # which doesn't separate extras, so we'll grab all of them
@@ -172,10 +178,12 @@ class WebInterface(object):
         newValueDict = {'IncludeExtras': 1,
                         'Extras': extras}
         myDB.upsert("artists", newValueDict, controlValueDict)
-        threading.Thread(target=importer.addArtisttoDB, args=[ArtistID, True, False]).start()
+        thread = threading.Thread(target=importer.addArtisttoDB, args=[ArtistID, True, False])
+        thread.start()
+        thread.join(1)
         raise cherrypy.HTTPRedirect("artistPage?ArtistID=%s" % ArtistID)
-    getExtras.exposed = True
 
+    @cherrypy.expose
     def removeExtras(self, ArtistID, ArtistName):
         myDB = db.DBConnection()
         controlValueDict = {'ArtistID': ArtistID}
@@ -193,8 +201,8 @@ class WebInterface(object):
             c.remove_from_cache(AlbumID=album['AlbumID'])
         importer.finalize_update(ArtistID, ArtistName)
         raise cherrypy.HTTPRedirect("artistPage?ArtistID=%s" % ArtistID)
-    removeExtras.exposed = True
 
+    @cherrypy.expose
     def pauseArtist(self, ArtistID):
         logger.info(u"Pausing artist: " + ArtistID)
         myDB = db.DBConnection()
@@ -202,8 +210,8 @@ class WebInterface(object):
         newValueDict = {'Status': 'Paused'}
         myDB.upsert("artists", newValueDict, controlValueDict)
         raise cherrypy.HTTPRedirect("artistPage?ArtistID=%s" % ArtistID)
-    pauseArtist.exposed = True
 
+    @cherrypy.expose
     def resumeArtist(self, ArtistID):
         logger.info(u"Resuming artist: " + ArtistID)
         myDB = db.DBConnection()
@@ -211,7 +219,6 @@ class WebInterface(object):
         newValueDict = {'Status': 'Active'}
         myDB.upsert("artists", newValueDict, controlValueDict)
         raise cherrypy.HTTPRedirect("artistPage?ArtistID=%s" % ArtistID)
-    resumeArtist.exposed = True
 
     def removeArtist(self, ArtistID):
         logger.info(u"Deleting all traces of artist: " + ArtistID)
@@ -242,24 +249,27 @@ class WebInterface(object):
         myDB.action('DELETE from descriptions WHERE ArtistID=?', [ArtistID])
         myDB.action('INSERT OR REPLACE into blacklist VALUES (?)', [ArtistID])
 
+    @cherrypy.expose
     def deleteArtist(self, ArtistID):
         self.removeArtist(ArtistID)
         raise cherrypy.HTTPRedirect("home")
-    deleteArtist.exposed = True
 
+    @cherrypy.expose
     def deleteEmptyArtists(self):
         logger.info(u"Deleting all empty artists")
         myDB = db.DBConnection()
         emptyArtistIDs = [row['ArtistID'] for row in myDB.select("SELECT ArtistID FROM artists WHERE LatestAlbum IS NULL")]
         for ArtistID in emptyArtistIDs:
             self.removeArtist(ArtistID)
-    deleteEmptyArtists.exposed = True
 
+    @cherrypy.expose
     def refreshArtist(self, ArtistID):
-        threading.Thread(target=importer.addArtisttoDB, args=[ArtistID, False, True]).start()
+        thread = threading.Thread(target=importer.addArtisttoDB, args=[ArtistID, False, True])
+        thread.start()
+        thread.join(1)
         raise cherrypy.HTTPRedirect("artistPage?ArtistID=%s" % ArtistID)
-    refreshArtist.exposed = True
 
+    @cherrypy.expose
     def markAlbums(self, ArtistID=None, action=None, **args):
         myDB = db.DBConnection()
         if action == 'WantedNew' or action == 'WantedLossless':
@@ -286,8 +296,8 @@ class WebInterface(object):
             raise cherrypy.HTTPRedirect("artistPage?ArtistID=%s" % ArtistID)
         else:
             raise cherrypy.HTTPRedirect("upcoming")
-    markAlbums.exposed = True
 
+    @cherrypy.expose
     def addArtists(self, action=None, **args):
         if action == "add":
             threading.Thread(target=importer.artistlist_to_mbids, args=[args, True]).start()
@@ -298,8 +308,8 @@ class WebInterface(object):
                 myDB.action('UPDATE have SET Matched="Ignored" WHERE ArtistName=?', [artist.decode(headphones.SYS_ENCODING, 'replace')])
                 logger.info("Artist %s removed from new artist list and set to ignored" % artist)
         raise cherrypy.HTTPRedirect("home")
-    addArtists.exposed = True
 
+    @cherrypy.expose
     def queueAlbum(self, AlbumID, ArtistID=None, new=False, redirect=None, lossless=False):
         logger.info(u"Marking album: " + AlbumID + " as wanted...")
         myDB = db.DBConnection()
@@ -312,41 +322,34 @@ class WebInterface(object):
         myDB.upsert("albums", newValueDict, controlValueDict)
         searcher.searchforalbum(AlbumID, new)
         if ArtistID:
-            raise cherrypy.HTTPRedirect("artistPage?ArtistID=%s" % ArtistID)
+            redirect = "artistPage?ArtistID=%s" % ArtistID
-        else:
+        raise cherrypy.HTTPRedirect(redirect)
-            raise cherrypy.HTTPRedirect(redirect)
-    queueAlbum.exposed = True
 
+    @cherrypy.expose
     def choose_specific_download(self, AlbumID):
         results = searcher.searchforalbum(AlbumID, choose_specific_download=True)
 
         results_as_dicts = []
 
         for result in results:
-
             result_dict = {
                 'title': result[0],
                 'size': result[1],
                 'url': result[2],
                 'provider': result[3],
-                'kind': result[4]
+                'kind': result[4],
+                'matches': result[5]
             }
             results_as_dicts.append(result_dict)
-
         s = json.dumps(results_as_dicts)
         cherrypy.response.headers['Content-type'] = 'application/json'
         return s
 
-    choose_specific_download.exposed = True
+    @cherrypy.expose
-
     def download_specific_release(self, AlbumID, title, size, url, provider, kind, **kwargs):
-
         # Handle situations where the torrent url contains arguments that are parsed
         if kwargs:
-            import urllib
-            import urllib2
             url = urllib2.quote(url, safe=":?/=&") + '&' + urllib.urlencode(kwargs)
-
         try:
             result = [(title, int(size), url, provider, kind)]
         except ValueError:
@@ -360,8 +363,7 @@ class WebInterface(object):
           album = myDB.action('SELECT * from albums WHERE AlbumID=?', [AlbumID]).fetchone()
           searcher.send_to_downloader(data, bestqual, album)
 
-    download_specific_release.exposed = True
+    @cherrypy.expose
-
     def unqueueAlbum(self, AlbumID, ArtistID):
         logger.info(u"Marking album: " + AlbumID + "as skipped...")
         myDB = db.DBConnection()
@@ -369,15 +371,16 @@ class WebInterface(object):
         newValueDict = {'Status': 'Skipped'}
         myDB.upsert("albums", newValueDict, controlValueDict)
         raise cherrypy.HTTPRedirect("artistPage?ArtistID=%s" % ArtistID)
-    unqueueAlbum.exposed = True
 
+    @cherrypy.expose
     def deleteAlbum(self, AlbumID, ArtistID=None):
         logger.info(u"Deleting all traces of album: " + AlbumID)
         myDB = db.DBConnection()
 
         myDB.action('DELETE from have WHERE Matched=?', [AlbumID])
-        album = myDB.action('SELECT ArtistName, AlbumTitle from albums where AlbumID=?', [AlbumID]).fetchone()
+        album = myDB.action('SELECT ArtistID, ArtistName, AlbumTitle from albums where AlbumID=?', [AlbumID]).fetchone()
         if album:
+            ArtistID = album['ArtistID']
             myDB.action('DELETE from have WHERE ArtistName=? AND AlbumTitle=?', [album['ArtistName'], album['AlbumTitle']])
 
         myDB.action('DELETE from albums WHERE AlbumID=?', [AlbumID])
@@ -395,17 +398,18 @@ class WebInterface(object):
             raise cherrypy.HTTPRedirect("artistPage?ArtistID=%s" % ArtistID)
         else:
             raise cherrypy.HTTPRedirect("home")
-    deleteAlbum.exposed = True
 
+    @cherrypy.expose
     def switchAlbum(self, AlbumID, ReleaseID):
-        '''
+        """
-        Take the values from allalbums/alltracks (based on the ReleaseID) and swap it into the album & track tables
+        Take the values from allalbums/alltracks (based on the ReleaseID) and
-        '''
+        swap it into the album & track tables
+        """
         from headphones import albumswitcher
         albumswitcher.switch(AlbumID, ReleaseID)
         raise cherrypy.HTTPRedirect("albumPage?AlbumID=%s" % AlbumID)
-    switchAlbum.exposed = True
 
+    @cherrypy.expose
     def editSearchTerm(self, AlbumID, SearchTerm):
         logger.info(u"Updating search term for albumid: " + AlbumID)
         myDB = db.DBConnection()
@@ -413,27 +417,27 @@ class WebInterface(object):
         newValueDict = {'SearchTerm': SearchTerm}
         myDB.upsert("albums", newValueDict, controlValueDict)
         raise cherrypy.HTTPRedirect("albumPage?AlbumID=%s" % AlbumID)
-    editSearchTerm.exposed = True
 
+    @cherrypy.expose
     def upcoming(self):
         myDB = db.DBConnection()
         upcoming = myDB.select("SELECT * from albums WHERE ReleaseDate > date('now') order by ReleaseDate ASC")
         wanted = myDB.select("SELECT * from albums WHERE Status='Wanted'")
         return serve_template(templatename="upcoming.html", title="Upcoming", upcoming=upcoming, wanted=wanted)
-    upcoming.exposed = True
 
+    @cherrypy.expose
     def manage(self):
         myDB = db.DBConnection()
         emptyArtists = myDB.select("SELECT * FROM artists WHERE LatestAlbum IS NULL")
         return serve_template(templatename="manage.html", title="Manage", emptyArtists=emptyArtists)
-    manage.exposed = True
 
+    @cherrypy.expose
     def manageArtists(self):
         myDB = db.DBConnection()
         artists = myDB.select('SELECT * from artists order by ArtistSortName COLLATE NOCASE')
         return serve_template(templatename="manageartists.html", title="Manage Artists", artists=artists)
-    manageArtists.exposed = True
 
+    @cherrypy.expose
     def manageAlbums(self, Status=None):
         myDB = db.DBConnection()
         if Status == "Upcoming":
@@ -443,14 +447,14 @@ class WebInterface(object):
         else:
             albums = myDB.select('SELECT * from albums')
         return serve_template(templatename="managealbums.html", title="Manage Albums", albums=albums)
-    manageAlbums.exposed = True
 
+    @cherrypy.expose
     def manageNew(self):
         myDB = db.DBConnection()
         newartists = myDB.select('SELECT * from newartists')
         return serve_template(templatename="managenew.html", title="Manage New Artists", newartists=newartists)
-    manageNew.exposed = True
 
+    @cherrypy.expose
     def manageUnmatched(self):
         myDB = db.DBConnection()
         have_album_dictionary = []
@@ -476,8 +480,8 @@ class WebInterface(object):
         unmatchedalbums = [d for d in have_album_dictionary if (cleanName(d['ArtistName']).lower(), cleanName(d['AlbumTitle']).lower()) not in check]
 
         return serve_template(templatename="manageunmatched.html", title="Manage Unmatched Items", unmatchedalbums=unmatchedalbums)
-    manageUnmatched.exposed = True
 
+    @cherrypy.expose
     def markUnmatched(self, action=None, existing_artist=None, existing_album=None, new_artist=None, new_album=None):
         myDB = db.DBConnection()
 
@@ -563,8 +567,7 @@ class WebInterface(object):
             else:
                 logger.info("Artist %s / Album %s already named appropriately; nothing to modify" % (existing_artist, existing_album))
 
-    markUnmatched.exposed = True
+    @cherrypy.expose
-
     def manageManual(self):
         myDB = db.DBConnection()
         manual_albums = []
@@ -583,8 +586,8 @@ class WebInterface(object):
         manual_albums_sorted = sorted(manual_albums, key=itemgetter('ArtistName', 'AlbumTitle'))
 
         return serve_template(templatename="managemanual.html", title="Manage Manual Items", manualalbums=manual_albums_sorted)
-    manageManual.exposed = True
 
+    @cherrypy.expose
     def markManual(self, action=None, existing_artist=None, existing_album=None):
         myDB = db.DBConnection()
         if action == "unignoreArtist":
@@ -635,8 +638,7 @@ class WebInterface(object):
                 librarysync.update_album_status(album_id)
             logger.info("Album: %s successfully restored to unmatched list" % album)
 
-    markManual.exposed = True
+    @cherrypy.expose
-
     def markArtists(self, action=None, **args):
         myDB = db.DBConnection()
         artistsToAdd = []
@@ -657,28 +659,29 @@ class WebInterface(object):
             logger.debug("Refreshing artists: %s" % artistsToAdd)
             threading.Thread(target=importer.addArtistIDListToDB, args=[artistsToAdd]).start()
         raise cherrypy.HTTPRedirect("home")
-    markArtists.exposed = True
 
+    @cherrypy.expose
     def importLastFM(self, username):
         headphones.CONFIG.LASTFM_USERNAME = username
         headphones.CONFIG.write()
         threading.Thread(target=lastfm.getArtists).start()
         raise cherrypy.HTTPRedirect("home")
-    importLastFM.exposed = True
 
+    @cherrypy.expose
     def importLastFMTag(self, tag, limit):
         threading.Thread(target=lastfm.getTagTopArtists, args=(tag, limit)).start()
         raise cherrypy.HTTPRedirect("home")
-    importLastFMTag.exposed = True
 
+    @cherrypy.expose
     def importItunes(self, path):
         headphones.CONFIG.PATH_TO_XML = path
         headphones.CONFIG.write()
-        threading.Thread(target=importer.itunesImport, args=[path]).start()
+        thread = threading.Thread(target=importer.itunesImport, args=[path])
-        time.sleep(10)
+        thread.start()
+        thread.join(10)
         raise cherrypy.HTTPRedirect("home")
-    importItunes.exposed = True
 
+    @cherrypy.expose
     def musicScan(self, path, scan=0, redirect=None, autoadd=0, libraryscan=0):
         headphones.CONFIG.LIBRARYSCAN = libraryscan
         headphones.CONFIG.AUTO_ADD_ARTISTS = autoadd
@@ -693,64 +696,64 @@ class WebInterface(object):
             raise cherrypy.HTTPRedirect(redirect)
         else:
             raise cherrypy.HTTPRedirect("home")
-    musicScan.exposed = True
 
+    @cherrypy.expose
     def forceUpdate(self):
         from headphones import updater
         threading.Thread(target=updater.dbUpdate, args=[False]).start()
         raise cherrypy.HTTPRedirect("home")
-    forceUpdate.exposed = True
 
+    @cherrypy.expose
     def forceFullUpdate(self):
         from headphones import updater
         threading.Thread(target=updater.dbUpdate, args=[True]).start()
         raise cherrypy.HTTPRedirect("home")
-    forceFullUpdate.exposed = True
 
+    @cherrypy.expose
     def forceSearch(self):
         from headphones import searcher
         threading.Thread(target=searcher.searchforalbum).start()
         raise cherrypy.HTTPRedirect("home")
-    forceSearch.exposed = True
 
+    @cherrypy.expose
     def forcePostProcess(self, dir=None, album_dir=None):
         from headphones import postprocessor
         threading.Thread(target=postprocessor.forcePostProcess, kwargs={'dir': dir, 'album_dir': album_dir}).start()
         raise cherrypy.HTTPRedirect("home")
-    forcePostProcess.exposed = True
 
+    @cherrypy.expose
     def checkGithub(self):
         from headphones import versioncheck
         versioncheck.checkGithub()
         raise cherrypy.HTTPRedirect("home")
-    checkGithub.exposed = True
 
+    @cherrypy.expose
     def history(self):
         myDB = db.DBConnection()
         history = myDB.select('''SELECT * from snatched WHERE Status NOT LIKE "Seed%" order by DateAdded DESC''')
         return serve_template(templatename="history.html", title="History", history=history)
-    history.exposed = True
 
+    @cherrypy.expose
     def logs(self):
         return serve_template(templatename="logs.html", title="Log", lineList=headphones.LOG_LIST)
-    logs.exposed = True
 
+    @cherrypy.expose
     def clearLogs(self):
         headphones.LOG_LIST = []
         logger.info("Web logs cleared")
         raise cherrypy.HTTPRedirect("logs")
-    clearLogs.exposed = True
 
+    @cherrypy.expose
     def toggleVerbose(self):
         headphones.VERBOSE = not headphones.VERBOSE
-        logger.initLogger(not headphones.QUIET, headphones.VERBOSE)
+        logger.initLogger(console=not headphones.QUIET,
+            log_dir=headphones.CONFIG.LOG_DIR, verbose=headphones.VERBOSE)
         logger.info("Verbose toggled, set to %s", headphones.VERBOSE)
         logger.debug("If you read this message, debug logging is available")
         raise cherrypy.HTTPRedirect("logs")
-    toggleVerbose.exposed = True
 
+    @cherrypy.expose
     def getLog(self, iDisplayStart=0, iDisplayLength=100, iSortCol_0=0, sSortDir_0="desc", sSearch="", **kwargs):
-
         iDisplayStart = int(iDisplayStart)
         iDisplayLength = int(iDisplayLength)
 
@@ -775,8 +778,8 @@ class WebInterface(object):
             'iTotalRecords': len(headphones.LOG_LIST),
             'aaData': rows,
         })
-    getLog.exposed = True
 
+    @cherrypy.expose
     def getArtists_json(self, iDisplayStart=0, iDisplayLength=100, sSearch="", iSortCol_0='0', sSortDir_0='asc', **kwargs):
         iDisplayStart = int(iDisplayStart)
         iDisplayLength = int(iDisplayLength)
@@ -847,8 +850,8 @@ class WebInterface(object):
         s = json.dumps(dict)
         cherrypy.response.headers['Content-type'] = 'application/json'
         return s
-    getArtists_json.exposed = True
 
+    @cherrypy.expose
     def getAlbumsByArtist_json(self, artist=None):
         myDB = db.DBConnection()
         album_json = {}
@@ -861,7 +864,6 @@ class WebInterface(object):
 
         cherrypy.response.headers['Content-type'] = 'application/json'
         return json_albums
-    getAlbumsByArtist_json.exposed = True
 
     def getArtistjson(self, ArtistID, **kwargs):
         myDB = db.DBConnection()
@@ -871,8 +873,8 @@ class WebInterface(object):
                                     'Status': artist['Status']
                                  })
         return artist_json
-    getArtistjson.exposed = True
 
+    @cherrypy.expose
     def getAlbumjson(self, AlbumID, **kwargs):
         myDB = db.DBConnection()
         album = myDB.action('SELECT * from albums WHERE AlbumID=?', [AlbumID]).fetchone()
@@ -882,8 +884,8 @@ class WebInterface(object):
                                    'Status': album['Status']
         })
         return album_json
-    getAlbumjson.exposed = True
 
+    @cherrypy.expose
     def clearhistory(self, type=None, date_added=None, title=None):
         myDB = db.DBConnection()
         if type:
@@ -897,19 +899,14 @@ class WebInterface(object):
             logger.info(u"Deleting '%s' from history" % title)
             myDB.action('DELETE from snatched WHERE Status NOT LIKE "Seed%" AND Title=? AND DateAdded=?', [title, date_added])
         raise cherrypy.HTTPRedirect("history")
-    clearhistory.exposed = True
 
+    @cherrypy.expose
     def generateAPI(self):
-
-        import hashlib
-        import random
-
         apikey = hashlib.sha224(str(random.getrandbits(256))).hexdigest()[0:32]
         logger.info("New API generated")
         return apikey
 
-    generateAPI.exposed = True
+    @cherrypy.expose
-
     def forceScan(self, keepmatched=None):
         myDB = db.DBConnection()
         #########################################
@@ -928,10 +925,9 @@ class WebInterface(object):
         except Exception as e:
             logger.error('Unable to complete the scan: %s' % e)
         raise cherrypy.HTTPRedirect("home")
-    forceScan.exposed = True
 
+    @cherrypy.expose
     def config(self):
-
         interface_dir = os.path.join(headphones.PROG_DIR, 'data/interfaces/')
         interface_list = [name for name in os.listdir(interface_dir) if os.path.isdir(os.path.join(interface_dir, name))]
 
@@ -1002,6 +998,9 @@ class WebInterface(object):
             "use_piratebay": checked(headphones.CONFIG.PIRATEBAY),
             "piratebay_proxy_url": headphones.CONFIG.PIRATEBAY_PROXY_URL,
             "piratebay_ratio": headphones.CONFIG.PIRATEBAY_RATIO,
+            "use_oldpiratebay": checked(headphones.CONFIG.OLDPIRATEBAY),
+            "oldpiratebay_url": headphones.CONFIG.OLDPIRATEBAY_URL,
+            "oldpiratebay_ratio": headphones.CONFIG.OLDPIRATEBAY_RATIO,
             "use_mininova": checked(headphones.CONFIG.MININOVA),
             "mininova_ratio": headphones.CONFIG.MININOVA_RATIO,
             "use_waffles": checked(headphones.CONFIG.WAFFLES),
@@ -1029,6 +1028,8 @@ class WebInterface(object):
             "lossless_bitrate_to": headphones.CONFIG.LOSSLESS_BITRATE_TO,
             "freeze_db": checked(headphones.CONFIG.FREEZE_DB),
             "cue_split": checked(headphones.CONFIG.CUE_SPLIT),
+            "cue_split_flac_path": headphones.CONFIG.CUE_SPLIT_FLAC_PATH,
+            "cue_split_shntool_path": headphones.CONFIG.CUE_SPLIT_SHNTOOL_PATH,
             "move_files": checked(headphones.CONFIG.MOVE_FILES),
             "rename_files": checked(headphones.CONFIG.RENAME_FILES),
             "correct_metadata": checked(headphones.CONFIG.CORRECT_METADATA),
@@ -1167,17 +1168,17 @@ class WebInterface(object):
         config["extras"] = extras_dict
 
         return serve_template(templatename="config.html", title="Settings", config=config)
-    config.exposed = True
 
+    @cherrypy.expose
     def configUpdate(self, **kwargs):
         # Handle the variable config options. Note - keys with False values aren't getting passed
 
         checked_configs = [
             "launch_browser", "enable_https", "api_enabled", "use_blackhole", "headphones_indexer", "use_newznab", "newznab_enabled",
-            "use_nzbsorg", "use_omgwtfnzbs", "use_kat", "use_piratebay", "use_mininova", "use_waffles", "use_rutracker", "use_whatcd",
+            "use_nzbsorg", "use_omgwtfnzbs", "use_kat", "use_piratebay", "use_oldpiratebay", "use_mininova", "use_waffles", "use_rutracker",
-            "preferred_bitrate_allow_lossless", "detect_bitrate", "freeze_db", "cue_split", "move_files", "rename_files", "correct_metadata",
+            "use_whatcd", "preferred_bitrate_allow_lossless", "detect_bitrate", "freeze_db", "cue_split", "move_files", "rename_files",
-            "cleanup_files", "keep_nfo", "add_album_art", "embed_album_art", "embed_lyrics", "replace_existing_folders", "file_underscores",
+            "correct_metadata", "cleanup_files", "keep_nfo", "add_album_art", "embed_album_art", "embed_lyrics", "replace_existing_folders",
-            "include_extras", "autowant_upcoming", "autowant_all", "autowant_manually_added", "keep_torrent_files", "music_encoder",
+            "file_underscores", "include_extras", "autowant_upcoming", "autowant_all", "autowant_manually_added", "keep_torrent_files", "music_encoder",
             "encoderlossless", "encoder_multicore", "delete_lossless_files", "growl_enabled", "growl_onsnatch", "prowl_enabled",
             "prowl_onsnatch", "xbmc_enabled", "xbmc_update", "xbmc_notify", "lms_enabled", "plex_enabled", "plex_update", "plex_notify",
             "nma_enabled", "nma_onsnatch", "pushalot_enabled", "pushalot_onsnatch", "synoindex_enabled", "pushover_enabled",
@@ -1256,61 +1257,54 @@ class WebInterface(object):
 
         raise cherrypy.HTTPRedirect("config")
 
-    configUpdate.exposed = True
+    @cherrypy.expose
-
     def do_state_change(self, signal, title, timer):
         headphones.SIGNAL = signal
         message = title + '...'
         return serve_template(templatename="shutdown.html", title=title,
                               message=message, timer=timer)
 
+    @cherrypy.expose
     def shutdown(self):
         return self.do_state_change('shutdown', 'Shutting Down', 15)
-    shutdown.exposed = True
 
+    @cherrypy.expose
     def restart(self):
         return self.do_state_change('restart', 'Restarting', 30)
-    restart.exposed = True
 
+    @cherrypy.expose
     def update(self):
         return self.do_state_change('update', 'Updating', 120)
-    update.exposed = True
 
+    @cherrypy.expose
     def extras(self):
         myDB = db.DBConnection()
         cloudlist = myDB.select('SELECT * from lastfmcloud')
         return serve_template(templatename="extras.html", title="Extras", cloudlist=cloudlist)
-    extras.exposed = True
 
+    @cherrypy.expose
     def addReleaseById(self, rid, rgid=None):
         threading.Thread(target=importer.addReleaseById, args=[rid, rgid]).start()
         if rgid:
             raise cherrypy.HTTPRedirect("albumPage?AlbumID=%s" % rgid)
         else:
             raise cherrypy.HTTPRedirect("home")
-    addReleaseById.exposed = True
 
+    @cherrypy.expose
     def updateCloud(self):
-
         lastfm.getSimilar()
         raise cherrypy.HTTPRedirect("extras")
 
-    updateCloud.exposed = True
+    @cherrypy.expose
-
     def api(self, *args, **kwargs):
-
         from headphones.api import Api
 
         a = Api()
-
         a.checkParams(*args, **kwargs)
 
-        data = a.fetchData()
+        return a.fetchData()
-
-        return data
-
-    api.exposed = True
 
+    @cherrypy.expose
     def getInfo(self, ArtistID=None, AlbumID=None):
 
         from headphones import cache
@@ -1318,25 +1312,22 @@ class WebInterface(object):
 
         return json.dumps(info_dict)
 
-    getInfo.exposed = True
+    @cherrypy.expose
-
     def getArtwork(self, ArtistID=None, AlbumID=None):
 
         from headphones import cache
         return cache.getArtwork(ArtistID, AlbumID)
 
-    getArtwork.exposed = True
+    @cherrypy.expose
-
     def getThumb(self, ArtistID=None, AlbumID=None):
 
         from headphones import cache
         return cache.getThumb(ArtistID, AlbumID)
 
-    getThumb.exposed = True
+    # If you just want to get the last.fm image links for an album, make sure
-
+    # to pass a releaseid and not a releasegroupid
-    # If you just want to get the last.fm image links for an album, make sure to pass a releaseid and not a releasegroupid
+    @cherrypy.expose
     def getImageLinks(self, ArtistID=None, AlbumID=None):
-
         from headphones import cache
         image_dict = cache.getImageLinks(ArtistID, AlbumID)
 
@@ -1353,14 +1344,13 @@ class WebInterface(object):
 
         return json.dumps(image_dict)
 
-    getImageLinks.exposed = True
+    @cherrypy.expose
-
     def twitterStep1(self):
         cherrypy.response.headers['Cache-Control'] = "max-age=0,no-cache,no-store"
         tweet = notifiers.TwitterNotifier()
         return tweet._get_authorization()
-    twitterStep1.exposed = True
 
+    @cherrypy.expose
     def twitterStep2(self, key):
         cherrypy.response.headers['Cache-Control'] = "max-age=0,no-cache,no-store"
         tweet = notifiers.TwitterNotifier()
@@ -1370,8 +1360,8 @@ class WebInterface(object):
             return "Key verification successful"
         else:
             return "Unable to verify key"
-    twitterStep2.exposed = True
 
+    @cherrypy.expose
     def testTwitter(self):
         cherrypy.response.headers['Cache-Control'] = "max-age=0,no-cache,no-store"
         tweet = notifiers.TwitterNotifier()
@@ -1380,8 +1370,8 @@ class WebInterface(object):
             return "Tweet successful, check your twitter to make sure it worked"
         else:
             return "Error sending tweet"
-    testTwitter.exposed = True
 
+    @cherrypy.expose
     def osxnotifyregister(self, app):
         cherrypy.response.headers['Cache-Control'] = "max-age=0,no-cache,no-store"
         from osxnotify import registerapp as osxnotify
@@ -1393,14 +1383,13 @@ class WebInterface(object):
         else:
             logger.warn(msg)
         return msg
-    osxnotifyregister.exposed = True
-
 
 class Artwork(object):
+    @cherrypy.expose
     def index(self):
         return "Artwork"
-    index.exposed = True
 
+    @cherrypy.expose
     def default(self, ArtistOrAlbum="", ID=None):
         from headphones import cache
         ArtistID = None
@@ -1425,16 +1414,15 @@ class Artwork(object):
             cherrypy.response.headers['Content-type'] = 'image/' + fileext
             cherrypy.response.headers['Cache-Control'] = 'max-age=31556926'
 
-        path = os.path.normpath(path)
+        with open(os.path.normpath(path), "rb") as fp:
-        f = open(path, 'rb')
+            return fp.read()
-        return f.read()
-    default.exposed = True
 
     class Thumbs(object):
+        @cherrypy.expose
         def index(self):
             return "Here be thumbs"
-        index.exposed = True
 
+        @cherrypy.expose
         def default(self, ArtistOrAlbum="", ID=None):
             from headphones import cache
             ArtistID = None
@@ -1459,12 +1447,8 @@ class Artwork(object):
                 cherrypy.response.headers['Content-type'] = 'image/' + fileext
                 cherrypy.response.headers['Cache-Control'] = 'max-age=31556926'
 
-            path = os.path.normpath(path)
+            with open(os.path.normpath(path), "rb") as fp:
-            f = open(path, 'rb')
+                return fp.read()
-            return f.read()
-        default.exposed = True
 
     thumbs = Thumbs()
-
-
 WebInterface.artwork = Artwork()

lib/biplist/__init__.py

@@ -0,0 +1,803 @@
+"""biplist -- a library for reading and writing binary property list files.
+
+Binary Property List (plist) files provide a faster and smaller serialization
+format for property lists on OS X. This is a library for generating binary
+plists which can be read by OS X, iOS, or other clients.
+
+The API models the plistlib API, and will call through to plistlib when
+XML serialization or deserialization is required.
+
+To generate plists with UID values, wrap the values with the Uid object. The
+value must be an int.
+
+To generate plists with NSData/CFData values, wrap the values with the
+Data object. The value must be a string.
+
+Date values can only be datetime.datetime objects.
+
+The exceptions InvalidPlistException and NotBinaryPlistException may be 
+thrown to indicate that the data cannot be serialized or deserialized as
+a binary plist.
+
+Plist generation example:
+    
+    from biplist import *
+    from datetime import datetime
+    plist = {'aKey':'aValue',
+             '0':1.322,
+             'now':datetime.now(),
+             'list':[1,2,3],
+             'tuple':('a','b','c')
+             }
+    try:
+        writePlist(plist, "example.plist")
+    except (InvalidPlistException, NotBinaryPlistException), e:
+        print "Something bad happened:", e
+
+Plist parsing example:
+
+    from biplist import *
+    try:
+        plist = readPlist("example.plist")
+        print plist
+    except (InvalidPlistException, NotBinaryPlistException), e:
+        print "Not a plist:", e
+"""
+
+import sys
+from collections import namedtuple
+import datetime
+import io
+import math
+import plistlib
+from struct import pack, unpack
+from struct import error as struct_error
+import sys
+import time
+
+try:
+    unicode
+    unicodeEmpty = r''
+except NameError:
+    unicode = str
+    unicodeEmpty = ''
+try:
+    long
+except NameError:
+    long = int
+try:
+    {}.iteritems
+    iteritems = lambda x: x.iteritems()
+except AttributeError:
+    iteritems = lambda x: x.items()
+
+__all__ = [
+    'Uid', 'Data', 'readPlist', 'writePlist', 'readPlistFromString',
+    'writePlistToString', 'InvalidPlistException', 'NotBinaryPlistException'
+]
+
+# Apple uses Jan 1, 2001 as a base for all plist date/times.
+apple_reference_date = datetime.datetime.utcfromtimestamp(978307200)
+
+class Uid(int):
+    """Wrapper around integers for representing UID values. This
+       is used in keyed archiving."""
+    def __repr__(self):
+        return "Uid(%d)" % self
+
+class Data(bytes):
+    """Wrapper around str types for representing Data values."""
+    pass
+
+class InvalidPlistException(Exception):
+    """Raised when the plist is incorrectly formatted."""
+    pass
+
+class NotBinaryPlistException(Exception):
+    """Raised when a binary plist was expected but not encountered."""
+    pass
+
+def readPlist(pathOrFile):
+    """Raises NotBinaryPlistException, InvalidPlistException"""
+    didOpen = False
+    result = None
+    if isinstance(pathOrFile, (bytes, unicode)):
+        pathOrFile = open(pathOrFile, 'rb')
+        didOpen = True
+    try:
+        reader = PlistReader(pathOrFile)
+        result = reader.parse()
+    except NotBinaryPlistException as e:
+        try:
+            pathOrFile.seek(0)
+            result = None
+            if hasattr(plistlib, 'loads'):
+                contents = None
+                if isinstance(pathOrFile, (bytes, unicode)):
+                    with open(pathOrFile, 'rb') as f:
+                        contents = f.read()
+                else:
+                    contents = pathOrFile.read()
+                result = plistlib.loads(contents)
+            else:
+                result = plistlib.readPlist(pathOrFile)
+            result = wrapDataObject(result, for_binary=True)
+        except Exception as e:
+            raise InvalidPlistException(e)
+    finally:
+        if didOpen:
+            pathOrFile.close()
+    return result
+
+def wrapDataObject(o, for_binary=False):
+    if isinstance(o, Data) and not for_binary:
+        v = sys.version_info
+        if not (v[0] >= 3 and v[1] >= 4):
+            o = plistlib.Data(o)
+    elif isinstance(o, (bytes, plistlib.Data)) and for_binary:
+        if hasattr(o, 'data'):
+            o = Data(o.data)
+    elif isinstance(o, tuple):
+        o = wrapDataObject(list(o), for_binary)
+        o = tuple(o)
+    elif isinstance(o, list):
+        for i in range(len(o)):
+            o[i] = wrapDataObject(o[i], for_binary)
+    elif isinstance(o, dict):
+        for k in o:
+            o[k] = wrapDataObject(o[k], for_binary)
+    return o
+
+def writePlist(rootObject, pathOrFile, binary=True):
+    if not binary:
+        rootObject = wrapDataObject(rootObject, binary)
+        if hasattr(plistlib, "dump"):
+            if isinstance(pathOrFile, (bytes, unicode)):
+                with open(pathOrFile, 'wb') as f:
+                    return plistlib.dump(rootObject, f)
+            else:
+                return plistlib.dump(rootObject, pathOrFile)
+        else:
+            return plistlib.writePlist(rootObject, pathOrFile)
+    else:
+        didOpen = False
+        if isinstance(pathOrFile, (bytes, unicode)):
+            pathOrFile = open(pathOrFile, 'wb')
+            didOpen = True
+        writer = PlistWriter(pathOrFile)
+        result = writer.writeRoot(rootObject)
+        if didOpen:
+            pathOrFile.close()
+        return result
+
+def readPlistFromString(data):
+    return readPlist(io.BytesIO(data))
+
+def writePlistToString(rootObject, binary=True):
+    if not binary:
+        rootObject = wrapDataObject(rootObject, binary)
+        if hasattr(plistlib, "dumps"):
+            return plistlib.dumps(rootObject)
+        elif hasattr(plistlib, "writePlistToBytes"):
+            return plistlib.writePlistToBytes(rootObject)
+        else:
+            return plistlib.writePlistToString(rootObject)
+    else:
+        ioObject = io.BytesIO()
+        writer = PlistWriter(ioObject)
+        writer.writeRoot(rootObject)
+        return ioObject.getvalue()
+
+def is_stream_binary_plist(stream):
+    stream.seek(0)
+    header = stream.read(7)
+    if header == b'bplist0':
+        return True
+    else:
+        return False
+
+PlistTrailer = namedtuple('PlistTrailer', 'offsetSize, objectRefSize, offsetCount, topLevelObjectNumber, offsetTableOffset')
+PlistByteCounts = namedtuple('PlistByteCounts', 'nullBytes, boolBytes, intBytes, realBytes, dateBytes, dataBytes, stringBytes, uidBytes, arrayBytes, setBytes, dictBytes')
+
+class PlistReader(object):
+    file = None
+    contents = ''
+    offsets = None
+    trailer = None
+    currentOffset = 0
+    
+    def __init__(self, fileOrStream):
+        """Raises NotBinaryPlistException."""
+        self.reset()
+        self.file = fileOrStream
+    
+    def parse(self):
+        return self.readRoot()
+    
+    def reset(self):
+        self.trailer = None
+        self.contents = ''
+        self.offsets = []
+        self.currentOffset = 0
+    
+    def readRoot(self):
+        result = None
+        self.reset()
+        # Get the header, make sure it's a valid file.
+        if not is_stream_binary_plist(self.file):
+            raise NotBinaryPlistException()
+        self.file.seek(0)
+        self.contents = self.file.read()
+        if len(self.contents) < 32:
+            raise InvalidPlistException("File is too short.")
+        trailerContents = self.contents[-32:]
+        try:
+            self.trailer = PlistTrailer._make(unpack("!xxxxxxBBQQQ", trailerContents))
+            offset_size = self.trailer.offsetSize * self.trailer.offsetCount
+            offset = self.trailer.offsetTableOffset
+            offset_contents = self.contents[offset:offset+offset_size]
+            offset_i = 0
+            while offset_i < self.trailer.offsetCount:
+                begin = self.trailer.offsetSize*offset_i
+                tmp_contents = offset_contents[begin:begin+self.trailer.offsetSize]
+                tmp_sized = self.getSizedInteger(tmp_contents, self.trailer.offsetSize)
+                self.offsets.append(tmp_sized)
+                offset_i += 1
+            self.setCurrentOffsetToObjectNumber(self.trailer.topLevelObjectNumber)
+            result = self.readObject()
+        except TypeError as e:
+            raise InvalidPlistException(e)
+        return result
+    
+    def setCurrentOffsetToObjectNumber(self, objectNumber):
+        self.currentOffset = self.offsets[objectNumber]
+    
+    def readObject(self):
+        result = None
+        tmp_byte = self.contents[self.currentOffset:self.currentOffset+1]
+        marker_byte = unpack("!B", tmp_byte)[0]
+        format = (marker_byte >> 4) & 0x0f
+        extra = marker_byte & 0x0f
+        self.currentOffset += 1
+        
+        def proc_extra(extra):
+            if extra == 0b1111:
+                #self.currentOffset += 1
+                extra = self.readObject()
+            return extra
+        
+        # bool, null, or fill byte
+        if format == 0b0000:
+            if extra == 0b0000:
+                result = None
+            elif extra == 0b1000:
+                result = False
+            elif extra == 0b1001:
+                result = True
+            elif extra == 0b1111:
+                pass # fill byte
+            else:
+                raise InvalidPlistException("Invalid object found at offset: %d" % (self.currentOffset - 1))
+        # int
+        elif format == 0b0001:
+            extra = proc_extra(extra)
+            result = self.readInteger(pow(2, extra))
+        # real
+        elif format == 0b0010:
+            extra = proc_extra(extra)
+            result = self.readReal(extra)
+        # date
+        elif format == 0b0011 and extra == 0b0011:
+            result = self.readDate()
+        # data
+        elif format == 0b0100:
+            extra = proc_extra(extra)
+            result = self.readData(extra)
+        # ascii string
+        elif format == 0b0101:
+            extra = proc_extra(extra)
+            result = self.readAsciiString(extra)
+        # Unicode string
+        elif format == 0b0110:
+            extra = proc_extra(extra)
+            result = self.readUnicode(extra)
+        # uid
+        elif format == 0b1000:
+            result = self.readUid(extra)
+        # array
+        elif format == 0b1010:
+            extra = proc_extra(extra)
+            result = self.readArray(extra)
+        # set
+        elif format == 0b1100:
+            extra = proc_extra(extra)
+            result = set(self.readArray(extra))
+        # dict
+        elif format == 0b1101:
+            extra = proc_extra(extra)
+            result = self.readDict(extra)
+        else:    
+            raise InvalidPlistException("Invalid object found: {format: %s, extra: %s}" % (bin(format), bin(extra)))
+        return result
+    
+    def readInteger(self, byteSize):
+        result = 0
+        original_offset = self.currentOffset
+        data = self.contents[self.currentOffset:self.currentOffset + byteSize]
+        result = self.getSizedInteger(data, byteSize, as_number=True)
+        self.currentOffset = original_offset + byteSize
+        return result
+    
+    def readReal(self, length):
+        result = 0.0
+        to_read = pow(2, length)
+        data = self.contents[self.currentOffset:self.currentOffset+to_read]
+        if length == 2: # 4 bytes
+            result = unpack('>f', data)[0]
+        elif length == 3: # 8 bytes
+            result = unpack('>d', data)[0]
+        else:
+            raise InvalidPlistException("Unknown real of length %d bytes" % to_read)
+        return result
+    
+    def readRefs(self, count):    
+        refs = []
+        i = 0
+        while i < count:
+            fragment = self.contents[self.currentOffset:self.currentOffset+self.trailer.objectRefSize]
+            ref = self.getSizedInteger(fragment, len(fragment))
+            refs.append(ref)
+            self.currentOffset += self.trailer.objectRefSize
+            i += 1
+        return refs
+    
+    def readArray(self, count):
+        result = []
+        values = self.readRefs(count)
+        i = 0
+        while i < len(values):
+            self.setCurrentOffsetToObjectNumber(values[i])
+            value = self.readObject()
+            result.append(value)
+            i += 1
+        return result
+    
+    def readDict(self, count):
+        result = {}
+        keys = self.readRefs(count)
+        values = self.readRefs(count)
+        i = 0
+        while i < len(keys):
+            self.setCurrentOffsetToObjectNumber(keys[i])
+            key = self.readObject()
+            self.setCurrentOffsetToObjectNumber(values[i])
+            value = self.readObject()
+            result[key] = value
+            i += 1
+        return result
+    
+    def readAsciiString(self, length):
+        result = unpack("!%ds" % length, self.contents[self.currentOffset:self.currentOffset+length])[0]
+        self.currentOffset += length
+        return result
+    
+    def readUnicode(self, length):
+        actual_length = length*2
+        data = self.contents[self.currentOffset:self.currentOffset+actual_length]
+        # unpack not needed?!! data = unpack(">%ds" % (actual_length), data)[0]
+        self.currentOffset += actual_length
+        return data.decode('utf_16_be')
+    
+    def readDate(self):
+        result = unpack(">d", self.contents[self.currentOffset:self.currentOffset+8])[0]
+        # Use timedelta to workaround time_t size limitation on 32-bit python.
+        result = datetime.timedelta(seconds=result) + apple_reference_date
+        self.currentOffset += 8
+        return result
+    
+    def readData(self, length):
+        result = self.contents[self.currentOffset:self.currentOffset+length]
+        self.currentOffset += length
+        return Data(result)
+    
+    def readUid(self, length):
+        return Uid(self.readInteger(length+1))
+    
+    def getSizedInteger(self, data, byteSize, as_number=False):
+        """Numbers of 8 bytes are signed integers when they refer to numbers, but unsigned otherwise."""
+        result = 0
+        # 1, 2, and 4 byte integers are unsigned
+        if byteSize == 1:
+            result = unpack('>B', data)[0]
+        elif byteSize == 2:
+            result = unpack('>H', data)[0]
+        elif byteSize == 4:
+            result = unpack('>L', data)[0]
+        elif byteSize == 8:
+            if as_number:
+                result = unpack('>q', data)[0]
+            else:
+                result = unpack('>Q', data)[0]
+        elif byteSize <= 16:
+            # Handle odd-sized or integers larger than 8 bytes
+            # Don't naively go over 16 bytes, in order to prevent infinite loops.
+            result = 0
+            if hasattr(int, 'from_bytes'):
+                result = int.from_bytes(data, 'big')
+            else:
+                for byte in data:
+                    result = (result << 8) | unpack('>B', byte)[0]
+        else:
+            raise InvalidPlistException("Encountered integer longer than 16 bytes.")
+        return result
+
+class HashableWrapper(object):
+    def __init__(self, value):
+        self.value = value
+    def __repr__(self):
+        return "<HashableWrapper: %s>" % [self.value]
+
+class BoolWrapper(object):
+    def __init__(self, value):
+        self.value = value
+    def __repr__(self):
+        return "<BoolWrapper: %s>" % self.value
+
+class FloatWrapper(object):
+    _instances = {}
+    def __new__(klass, value):
+        # Ensure FloatWrapper(x) for a given float x is always the same object
+        wrapper = klass._instances.get(value)
+        if wrapper is None:
+            wrapper = object.__new__(klass)
+            wrapper.value = value
+            klass._instances[value] = wrapper
+        return wrapper
+    def __repr__(self):
+        return "<FloatWrapper: %s>" % self.value
+
+class PlistWriter(object):
+    header = b'bplist00bybiplist1.0'
+    file = None
+    byteCounts = None
+    trailer = None
+    computedUniques = None
+    writtenReferences = None
+    referencePositions = None
+    wrappedTrue = None
+    wrappedFalse = None
+    
+    def __init__(self, file):
+        self.reset()
+        self.file = file
+        self.wrappedTrue = BoolWrapper(True)
+        self.wrappedFalse = BoolWrapper(False)
+
+    def reset(self):
+        self.byteCounts = PlistByteCounts(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)
+        self.trailer = PlistTrailer(0, 0, 0, 0, 0)
+        
+        # A set of all the uniques which have been computed.
+        self.computedUniques = set()
+        # A list of all the uniques which have been written.
+        self.writtenReferences = {}
+        # A dict of the positions of the written uniques.
+        self.referencePositions = {}
+        
+    def positionOfObjectReference(self, obj):
+        """If the given object has been written already, return its
+           position in the offset table. Otherwise, return None."""
+        return self.writtenReferences.get(obj)
+        
+    def writeRoot(self, root):
+        """
+        Strategy is:
+        - write header
+        - wrap root object so everything is hashable
+        - compute size of objects which will be written
+          - need to do this in order to know how large the object refs
+            will be in the list/dict/set reference lists
+        - write objects
+          - keep objects in writtenReferences
+          - keep positions of object references in referencePositions
+          - write object references with the length computed previously
+        - computer object reference length
+        - write object reference positions
+        - write trailer
+        """
+        output = self.header
+        wrapped_root = self.wrapRoot(root)
+        should_reference_root = True#not isinstance(wrapped_root, HashableWrapper)
+        self.computeOffsets(wrapped_root, asReference=should_reference_root, isRoot=True)
+        self.trailer = self.trailer._replace(**{'objectRefSize':self.intSize(len(self.computedUniques))})
+        (_, output) = self.writeObjectReference(wrapped_root, output)
+        output = self.writeObject(wrapped_root, output, setReferencePosition=True)
+        
+        # output size at this point is an upper bound on how big the
+        # object reference offsets need to be.
+        self.trailer = self.trailer._replace(**{
+            'offsetSize':self.intSize(len(output)),
+            'offsetCount':len(self.computedUniques),
+            'offsetTableOffset':len(output),
+            'topLevelObjectNumber':0
+            })
+        
+        output = self.writeOffsetTable(output)
+        output += pack('!xxxxxxBBQQQ', *self.trailer)
+        self.file.write(output)
+
+    def wrapRoot(self, root):
+        if isinstance(root, bool):
+            if root is True:
+                return self.wrappedTrue
+            else:
+                return self.wrappedFalse
+        elif isinstance(root, float):
+            return FloatWrapper(root)
+        elif isinstance(root, set):
+            n = set()
+            for value in root:
+                n.add(self.wrapRoot(value))
+            return HashableWrapper(n)
+        elif isinstance(root, dict):
+            n = {}
+            for key, value in iteritems(root):
+                n[self.wrapRoot(key)] = self.wrapRoot(value)
+            return HashableWrapper(n)
+        elif isinstance(root, list):
+            n = []
+            for value in root:
+                n.append(self.wrapRoot(value))
+            return HashableWrapper(n)
+        elif isinstance(root, tuple):
+            n = tuple([self.wrapRoot(value) for value in root])
+            return HashableWrapper(n)
+        else:
+            return root
+
+    def incrementByteCount(self, field, incr=1):
+        self.byteCounts = self.byteCounts._replace(**{field:self.byteCounts.__getattribute__(field) + incr})
+
+    def computeOffsets(self, obj, asReference=False, isRoot=False):
+        def check_key(key):
+            if key is None:
+                raise InvalidPlistException('Dictionary keys cannot be null in plists.')
+            elif isinstance(key, Data):
+                raise InvalidPlistException('Data cannot be dictionary keys in plists.')
+            elif not isinstance(key, (bytes, unicode)):
+                raise InvalidPlistException('Keys must be strings.')
+        
+        def proc_size(size):
+            if size > 0b1110:
+                size += self.intSize(size)
+            return size
+        # If this should be a reference, then we keep a record of it in the
+        # uniques table.
+        if asReference:
+            if obj in self.computedUniques:
+                return
+            else:
+                self.computedUniques.add(obj)
+        
+        if obj is None:
+            self.incrementByteCount('nullBytes')
+        elif isinstance(obj, BoolWrapper):
+            self.incrementByteCount('boolBytes')
+        elif isinstance(obj, Uid):
+            size = self.intSize(obj)
+            self.incrementByteCount('uidBytes', incr=1+size)
+        elif isinstance(obj, (int, long)):
+            size = self.intSize(obj)
+            self.incrementByteCount('intBytes', incr=1+size)
+        elif isinstance(obj, FloatWrapper):
+            size = self.realSize(obj)
+            self.incrementByteCount('realBytes', incr=1+size)
+        elif isinstance(obj, datetime.datetime):    
+            self.incrementByteCount('dateBytes', incr=2)
+        elif isinstance(obj, Data):
+            size = proc_size(len(obj))
+            self.incrementByteCount('dataBytes', incr=1+size)
+        elif isinstance(obj, (unicode, bytes)):
+            size = proc_size(len(obj))
+            self.incrementByteCount('stringBytes', incr=1+size)
+        elif isinstance(obj, HashableWrapper):
+            obj = obj.value
+            if isinstance(obj, set):
+                size = proc_size(len(obj))
+                self.incrementByteCount('setBytes', incr=1+size)
+                for value in obj:
+                    self.computeOffsets(value, asReference=True)
+            elif isinstance(obj, (list, tuple)):
+                size = proc_size(len(obj))
+                self.incrementByteCount('arrayBytes', incr=1+size)
+                for value in obj:
+                    asRef = True
+                    self.computeOffsets(value, asReference=True)
+            elif isinstance(obj, dict):
+                size = proc_size(len(obj))
+                self.incrementByteCount('dictBytes', incr=1+size)
+                for key, value in iteritems(obj):
+                    check_key(key)
+                    self.computeOffsets(key, asReference=True)
+                    self.computeOffsets(value, asReference=True)
+        else:
+            raise InvalidPlistException("Unknown object type.")
+
+    def writeObjectReference(self, obj, output):
+        """Tries to write an object reference, adding it to the references
+           table. Does not write the actual object bytes or set the reference
+           position. Returns a tuple of whether the object was a new reference
+           (True if it was, False if it already was in the reference table)
+           and the new output.
+        """
+        position = self.positionOfObjectReference(obj)
+        if position is None:
+            self.writtenReferences[obj] = len(self.writtenReferences)
+            output += self.binaryInt(len(self.writtenReferences) - 1, byteSize=self.trailer.objectRefSize)
+            return (True, output)
+        else:
+            output += self.binaryInt(position, byteSize=self.trailer.objectRefSize)
+            return (False, output)
+
+    def writeObject(self, obj, output, setReferencePosition=False):
+        """Serializes the given object to the output. Returns output.
+           If setReferencePosition is True, will set the position the
+           object was written.
+        """
+        def proc_variable_length(format, length):
+            result = b''
+            if length > 0b1110:
+                result += pack('!B', (format << 4) | 0b1111)
+                result = self.writeObject(length, result)
+            else:
+                result += pack('!B', (format << 4) | length)
+            return result
+        
+        if isinstance(obj, (str, unicode)) and obj == unicodeEmpty:
+            # The Apple Plist decoder can't decode a zero length Unicode string.
+            obj = b''
+       
+        if setReferencePosition:
+            self.referencePositions[obj] = len(output)
+        
+        if obj is None:
+            output += pack('!B', 0b00000000)
+        elif isinstance(obj, BoolWrapper):
+            if obj.value is False:
+                output += pack('!B', 0b00001000)
+            else:
+                output += pack('!B', 0b00001001)
+        elif isinstance(obj, Uid):
+            size = self.intSize(obj)
+            output += pack('!B', (0b1000 << 4) | size - 1)
+            output += self.binaryInt(obj)
+        elif isinstance(obj, (int, long)):
+            byteSize = self.intSize(obj)
+            root = math.log(byteSize, 2)
+            output += pack('!B', (0b0001 << 4) | int(root))
+            output += self.binaryInt(obj, as_number=True)
+        elif isinstance(obj, FloatWrapper):
+            # just use doubles
+            output += pack('!B', (0b0010 << 4) | 3)
+            output += self.binaryReal(obj)
+        elif isinstance(obj, datetime.datetime):
+            timestamp = (obj - apple_reference_date).total_seconds()
+            output += pack('!B', 0b00110011)
+            output += pack('!d', float(timestamp))
+        elif isinstance(obj, Data):
+            output += proc_variable_length(0b0100, len(obj))
+            output += obj
+        elif isinstance(obj, unicode):
+            byteData = obj.encode('utf_16_be')
+            output += proc_variable_length(0b0110, len(byteData)//2)
+            output += byteData
+        elif isinstance(obj, bytes):
+            output += proc_variable_length(0b0101, len(obj))
+            output += obj
+        elif isinstance(obj, HashableWrapper):
+            obj = obj.value
+            if isinstance(obj, (set, list, tuple)):
+                if isinstance(obj, set):
+                    output += proc_variable_length(0b1100, len(obj))
+                else:
+                    output += proc_variable_length(0b1010, len(obj))
+            
+                objectsToWrite = []
+                for objRef in obj:
+                    (isNew, output) = self.writeObjectReference(objRef, output)
+                    if isNew:
+                        objectsToWrite.append(objRef)
+                for objRef in objectsToWrite:
+                    output = self.writeObject(objRef, output, setReferencePosition=True)
+            elif isinstance(obj, dict):
+                output += proc_variable_length(0b1101, len(obj))
+                keys = []
+                values = []
+                objectsToWrite = []
+                for key, value in iteritems(obj):
+                    keys.append(key)
+                    values.append(value)
+                for key in keys:
+                    (isNew, output) = self.writeObjectReference(key, output)
+                    if isNew:
+                        objectsToWrite.append(key)
+                for value in values:
+                    (isNew, output) = self.writeObjectReference(value, output)
+                    if isNew:
+                        objectsToWrite.append(value)
+                for objRef in objectsToWrite:
+                    output = self.writeObject(objRef, output, setReferencePosition=True)
+        return output
+    
+    def writeOffsetTable(self, output):
+        """Writes all of the object reference offsets."""
+        all_positions = []
+        writtenReferences = list(self.writtenReferences.items())
+        writtenReferences.sort(key=lambda x: x[1])
+        for obj,order in writtenReferences:
+            # Porting note: Elsewhere we deliberately replace empty unicdoe strings
+            # with empty binary strings, but the empty unicode string
+            # goes into writtenReferences.  This isn't an issue in Py2
+            # because u'' and b'' have the same hash; but it is in
+            # Py3, where they don't.
+            if bytes != str and obj == unicodeEmpty:
+                obj = b''
+            position = self.referencePositions.get(obj)
+            if position is None:
+                raise InvalidPlistException("Error while writing offsets table. Object not found. %s" % obj)
+            output += self.binaryInt(position, self.trailer.offsetSize)
+            all_positions.append(position)
+        return output
+    
+    def binaryReal(self, obj):
+        # just use doubles
+        result = pack('>d', obj.value)
+        return result
+    
+    def binaryInt(self, obj, byteSize=None, as_number=False):
+        result = b''
+        if byteSize is None:
+            byteSize = self.intSize(obj)
+        if byteSize == 1:
+            result += pack('>B', obj)
+        elif byteSize == 2:
+            result += pack('>H', obj)
+        elif byteSize == 4:
+            result += pack('>L', obj)
+        elif byteSize == 8:
+            if as_number:
+                result += pack('>q', obj)
+            else:
+                result += pack('>Q', obj)
+        elif byteSize <= 16:
+            try:
+                result = pack('>Q', 0) + pack('>Q', obj)
+            except struct_error as e:
+                raise InvalidPlistException("Unable to pack integer %d: %s" % (obj, e))
+        else:
+            raise InvalidPlistException("Core Foundation can't handle integers with size greater than 16 bytes.")
+        return result
+    
+    def intSize(self, obj):
+        """Returns the number of bytes necessary to store the given integer."""
+        # SIGNED
+        if obj < 0: # Signed integer, always 8 bytes
+            return 8
+        # UNSIGNED
+        elif obj <= 0xFF: # 1 byte
+            return 1
+        elif obj <= 0xFFFF: # 2 bytes
+            return 2
+        elif obj <= 0xFFFFFFFF: # 4 bytes
+            return 4
+        # SIGNED
+        # 0x7FFFFFFFFFFFFFFF is the max.
+        elif obj <= 0x7FFFFFFFFFFFFFFF: # 8 bytes signed
+            return 8
+        elif obj <= 0xffffffffffffffff: # 8 bytes unsigned
+            return 16
+        else:
+            raise InvalidPlistException("Core Foundation can't handle integers with size greater than 8 bytes.")
+    
+    def realSize(self, obj):
+        return 8

lib/requests/__init__.py

@@ -13,7 +13,7 @@ Requests is an HTTP library, written in Python, for human beings. Basic GET
 usage:
 
    >>> import requests
-   >>> r = requests.get('http://python.org')
+   >>> r = requests.get('https://www.python.org')
    >>> r.status_code
    200
    >>> 'Python is a programming language' in r.content
@@ -22,7 +22,7 @@ usage:
 ... or POST:
 
    >>> payload = dict(key1='value1', key2='value2')
-   >>> r = requests.post("http://httpbin.org/post", data=payload)
+   >>> r = requests.post('http://httpbin.org/post', data=payload)
    >>> print(r.text)
    {
      ...
@@ -42,8 +42,8 @@ is at <http://python-requests.org>.
 """
 
 __title__ = 'requests'
-__version__ = '2.4.1'
+__version__ = '2.5.0'
-__build__ = 0x020401
+__build__ = 0x020500
 __author__ = 'Kenneth Reitz'
 __license__ = 'Apache 2.0'
 __copyright__ = 'Copyright 2014 Kenneth Reitz'

lib/requests/adapters.py

@@ -15,9 +15,9 @@ from .packages.urllib3 import Retry
 from .packages.urllib3.poolmanager import PoolManager, proxy_from_url
 from .packages.urllib3.response import HTTPResponse
 from .packages.urllib3.util import Timeout as TimeoutSauce
-from .compat import urlparse, basestring, urldefrag
+from .compat import urlparse, basestring
 from .utils import (DEFAULT_CA_BUNDLE_PATH, get_encoding_from_headers,
-                    prepend_scheme_if_needed, get_auth_from_url)
+                    prepend_scheme_if_needed, get_auth_from_url, urldefragauth)
 from .structures import CaseInsensitiveDict
 from .packages.urllib3.exceptions import ConnectTimeoutError
 from .packages.urllib3.exceptions import HTTPError as _HTTPError
@@ -26,9 +26,10 @@ from .packages.urllib3.exceptions import ProxyError as _ProxyError
 from .packages.urllib3.exceptions import ProtocolError
 from .packages.urllib3.exceptions import ReadTimeoutError
 from .packages.urllib3.exceptions import SSLError as _SSLError
+from .packages.urllib3.exceptions import ResponseError
 from .cookies import extract_cookies_to_jar
 from .exceptions import (ConnectionError, ConnectTimeout, ReadTimeout, SSLError,
-                         ProxyError)
+                         ProxyError, RetryError)
 from .auth import _basic_auth_str
 
 DEFAULT_POOLBLOCK = False
@@ -60,8 +61,12 @@ class HTTPAdapter(BaseAdapter):
     :param pool_connections: The number of urllib3 connection pools to cache.
     :param pool_maxsize: The maximum number of connections to save in the pool.
     :param int max_retries: The maximum number of retries each connection
-        should attempt. Note, this applies only to failed connections and
+        should attempt. Note, this applies only to failed DNS lookups, socket
-        timeouts, never to requests where the server returns a response.
+        connections and connection timeouts, never to requests where data has
+        made it to the server. By default, Requests does not retry failed
+        connections. If you need granular control over the conditions under
+        which we retry a request, import urllib3's ``Retry`` class and pass
+        that instead.
     :param pool_block: Whether the connection pool should block for connections.
 
     Usage::
@@ -77,7 +82,10 @@ class HTTPAdapter(BaseAdapter):
     def __init__(self, pool_connections=DEFAULT_POOLSIZE,
                  pool_maxsize=DEFAULT_POOLSIZE, max_retries=DEFAULT_RETRIES,
                  pool_block=DEFAULT_POOLBLOCK):
-        self.max_retries = max_retries
+        if max_retries == DEFAULT_RETRIES:
+            self.max_retries = Retry(0, read=False)
+        else:
+            self.max_retries = Retry.from_int(max_retries)
         self.config = {}
         self.proxy_manager = {}
 
@@ -123,7 +131,7 @@ class HTTPAdapter(BaseAdapter):
         self._pool_block = block
 
         self.poolmanager = PoolManager(num_pools=connections, maxsize=maxsize,
-                                       block=block, **pool_kwargs)
+                                       block=block, strict=True, **pool_kwargs)
 
     def proxy_manager_for(self, proxy, **proxy_kwargs):
         """Return urllib3 ProxyManager for the given proxy.
@@ -270,7 +278,7 @@ class HTTPAdapter(BaseAdapter):
         proxy = proxies.get(scheme)
 
         if proxy and scheme != 'https':
-            url, _ = urldefrag(request.url)
+            url = urldefragauth(request.url)
         else:
             url = request.path_url
 
@@ -358,7 +366,7 @@ class HTTPAdapter(BaseAdapter):
                     assert_same_host=False,
                     preload_content=False,
                     decode_content=False,
-                    retries=Retry(self.max_retries, read=False),
+                    retries=self.max_retries,
                     timeout=timeout
                 )
 
@@ -410,6 +418,9 @@ class HTTPAdapter(BaseAdapter):
             if isinstance(e.reason, ConnectTimeoutError):
                 raise ConnectTimeout(e, request=request)
 
+            if isinstance(e.reason, ResponseError):
+                raise RetryError(e, request=request)
+
             raise ConnectionError(e, request=request)
 
         except _ProxyError as e:

lib/requests/api.py

@@ -22,6 +22,7 @@ def request(method, url, **kwargs):
     :param url: URL for the new :class:`Request` object.
     :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
     :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
+    :param json: (optional) json data to send in the body of the :class:`Request`.
     :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
     :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
     :param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': ('filename', fileobj)}``) for multipart encoding upload.
@@ -45,7 +46,12 @@ def request(method, url, **kwargs):
     """
 
     session = sessions.Session()
-    return session.request(method=method, url=url, **kwargs)
+    response = session.request(method=method, url=url, **kwargs)
+    # By explicitly closing the session, we avoid leaving sockets open which
+    # can trigger a ResourceWarning in some cases, and look like a memory leak
+    # in others.
+    session.close()
+    return response
 
 
 def get(url, **kwargs):
@@ -81,15 +87,16 @@ def head(url, **kwargs):
     return request('head', url, **kwargs)
 
 
-def post(url, data=None, **kwargs):
+def post(url, data=None, json=None, **kwargs):
     """Sends a POST request. Returns :class:`Response` object.
 
     :param url: URL for the new :class:`Request` object.
     :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
+    :param json: (optional) json data to send in the body of the :class:`Request`.
     :param \*\*kwargs: Optional arguments that ``request`` takes.
     """
 
-    return request('post', url, data=data, **kwargs)
+    return request('post', url, data=data, json=json, **kwargs)
 
 
 def put(url, data=None, **kwargs):

lib/requests/auth.py

@@ -17,6 +17,7 @@ from base64 import b64encode
 from .compat import urlparse, str
 from .cookies import extract_cookies_to_jar
 from .utils import parse_dict_header, to_native_string
+from .status_codes import codes
 
 CONTENT_TYPE_FORM_URLENCODED = 'application/x-www-form-urlencoded'
 CONTENT_TYPE_MULTI_PART = 'multipart/form-data'
@@ -150,6 +151,11 @@ class HTTPDigestAuth(AuthBase):
 
         return 'Digest %s' % (base)
 
+    def handle_redirect(self, r, **kwargs):
+        """Reset num_401_calls counter on redirects."""
+        if r.is_redirect:
+            setattr(self, 'num_401_calls', 1)
+
     def handle_401(self, r, **kwargs):
         """Takes the given response and tries digest-auth, if needed."""
 
@@ -182,7 +188,7 @@ class HTTPDigestAuth(AuthBase):
 
             return _r
 
-        setattr(self, 'num_401_calls', 1)
+        setattr(self, 'num_401_calls', num_401_calls + 1)
         return r
 
     def __call__(self, r):
@@ -192,6 +198,11 @@ class HTTPDigestAuth(AuthBase):
         try:
             self.pos = r.body.tell()
         except AttributeError:
-            pass
+            # In the case of HTTPDigestAuth being reused and the body of
+            # the previous request was a file-like object, pos has the
+            # file position of the previous body. Ensure it's set to
+            # None.
+            self.pos = None
         r.register_hook('response', self.handle_401)
+        r.register_hook('response', self.handle_redirect)
         return r

lib/requests/exceptions.py

@@ -89,3 +89,11 @@ class ChunkedEncodingError(RequestException):
 
 class ContentDecodingError(RequestException, BaseHTTPError):
     """Failed to decode response content"""
+
+
+class StreamConsumedError(RequestException, TypeError):
+    """The content for this response was already consumed"""
+
+
+class RetryError(RequestException):
+    """Custom retries logic failed"""

lib/requests/models.py

@@ -22,8 +22,9 @@ from .packages.urllib3.util import parse_url
 from .packages.urllib3.exceptions import (
     DecodeError, ReadTimeoutError, ProtocolError)
 from .exceptions import (
-    HTTPError, RequestException, MissingSchema, InvalidURL,
+    HTTPError, RequestException, MissingSchema, InvalidURL, 
-    ChunkedEncodingError, ContentDecodingError, ConnectionError)
+    ChunkedEncodingError, ContentDecodingError, ConnectionError, 
+    StreamConsumedError)
 from .utils import (
     guess_filename, get_auth_from_url, requote_uri,
     stream_decode_response_unicode, to_key_val_list, parse_header_links,
@@ -46,6 +47,8 @@ DEFAULT_REDIRECT_LIMIT = 30
 CONTENT_CHUNK_SIZE = 10 * 1024
 ITER_CHUNK_SIZE = 512
 
+json_dumps = json.dumps
+
 
 class RequestEncodingMixin(object):
     @property
@@ -189,7 +192,8 @@ class Request(RequestHooksMixin):
     :param url: URL to send.
     :param headers: dictionary of headers to send.
     :param files: dictionary of {filename: fileobject} files to multipart upload.
-    :param data: the body to attach the request. If a dictionary is provided, form-encoding will take place.
+    :param data: the body to attach to the request. If a dictionary is provided, form-encoding will take place.
+    :param json: json for the body to attach to the request (if data is not specified).
     :param params: dictionary of URL parameters to append to the URL.
     :param auth: Auth handler or (user, pass) tuple.
     :param cookies: dictionary or CookieJar of cookies to attach to this request.
@@ -212,7 +216,8 @@ class Request(RequestHooksMixin):
         params=None,
         auth=None,
         cookies=None,
-        hooks=None):
+        hooks=None,
+        json=None):
 
         # Default empty dicts for dict params.
         data = [] if data is None else data
@@ -230,6 +235,7 @@ class Request(RequestHooksMixin):
         self.headers = headers
         self.files = files
         self.data = data
+        self.json = json
         self.params = params
         self.auth = auth
         self.cookies = cookies
@@ -246,6 +252,7 @@ class Request(RequestHooksMixin):
             headers=self.headers,
             files=self.files,
             data=self.data,
+            json=self.json,
             params=self.params,
             auth=self.auth,
             cookies=self.cookies,
@@ -289,14 +296,15 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
         self.hooks = default_hooks()
 
     def prepare(self, method=None, url=None, headers=None, files=None,
-                data=None, params=None, auth=None, cookies=None, hooks=None):
+                data=None, params=None, auth=None, cookies=None, hooks=None,
+                json=None):
         """Prepares the entire request with the given parameters."""
 
         self.prepare_method(method)
         self.prepare_url(url, params)
         self.prepare_headers(headers)
         self.prepare_cookies(cookies)
-        self.prepare_body(data, files)
+        self.prepare_body(data, files, json)
         self.prepare_auth(auth, url)
         # Note that prepare_auth must be last to enable authentication schemes
         # such as OAuth to work on a fully prepared request.
@@ -326,15 +334,18 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
     def prepare_url(self, url, params):
         """Prepares the given HTTP URL."""
         #: Accept objects that have string representations.
-        try:
+        #: We're unable to blindy call unicode/str functions
-            url = unicode(url)
+        #: as this will include the bytestring indicator (b'')
-        except NameError:
+        #: on python 3.x.
-            # We're on Python 3.
+        #: https://github.com/kennethreitz/requests/pull/2238
-            url = str(url)
+        if isinstance(url, bytes):
-        except UnicodeDecodeError:
+            url = url.decode('utf8')
-            pass
+        else:
+            url = unicode(url) if is_py2 else str(url)
 
-        # Don't do any URL preparation for oddball schemes
+        # Don't do any URL preparation for non-HTTP schemes like `mailto`,
+        # `data` etc to work around exceptions from `url_parse`, which
+        # handles RFC 3986 only.
         if ':' in url and not url.lower().startswith('http'):
             self.url = url
             return
@@ -397,7 +408,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
         else:
             self.headers = CaseInsensitiveDict()
 
-    def prepare_body(self, data, files):
+    def prepare_body(self, data, files, json=None):
         """Prepares the given HTTP body data."""
 
         # Check if file, fo, generator, iterator.
@@ -408,6 +419,10 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
         content_type = None
         length = None
 
+        if json is not None:
+            content_type = 'application/json'
+            body = json_dumps(json)
+
         is_stream = all([
             hasattr(data, '__iter__'),
             not isinstance(data, (basestring, list, tuple, dict))
@@ -433,7 +448,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
             if files:
                 (body, content_type) = self._encode_files(files, data)
             else:
-                if data:
+                if data and json is None:
                     body = self._encode_params(data)
                     if isinstance(data, basestring) or hasattr(data, 'read'):
                         content_type = None
@@ -443,7 +458,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
             self.prepare_content_length(body)
 
             # Add content-type if it wasn't explicitly provided.
-            if (content_type) and (not 'content-type' in self.headers):
+            if content_type and ('content-type' not in self.headers):
                 self.headers['Content-Type'] = content_type
 
         self.body = body
@@ -457,7 +472,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
             l = super_len(body)
             if l:
                 self.headers['Content-Length'] = builtin_str(l)
-        elif self.method not in ('GET', 'HEAD'):
+        elif (self.method not in ('GET', 'HEAD')) and (self.headers.get('Content-Length') is None):
             self.headers['Content-Length'] = '0'
 
     def prepare_auth(self, auth, url=''):
@@ -653,6 +668,8 @@ class Response(object):
 
             self._content_consumed = True
 
+        if self._content_consumed and isinstance(self._content, bool):
+            raise StreamConsumedError()
         # simulate reading small chunks of the content
         reused_chunks = iter_slices(self._content, chunk_size)
 
@@ -665,7 +682,7 @@ class Response(object):
 
         return chunks
 
-    def iter_lines(self, chunk_size=ITER_CHUNK_SIZE, decode_unicode=None):
+    def iter_lines(self, chunk_size=ITER_CHUNK_SIZE, decode_unicode=None, delimiter=None):
         """Iterates over the response data, one line at a time.  When
         stream=True is set on the request, this avoids reading the
         content at once into memory for large responses.
@@ -677,7 +694,11 @@ class Response(object):
 
             if pending is not None:
                 chunk = pending + chunk
-            lines = chunk.splitlines()
+
+            if delimiter:
+                lines = chunk.split(delimiter)
+            else:
+                lines = chunk.splitlines()
 
             if lines and lines[-1] and chunk and lines[-1][-1] == chunk[-1]:
                 pending = lines.pop()

lib/requests/packages/chardet/__init__.py

@@ -15,7 +15,7 @@
 # 02110-1301  USA
 ######################### END LICENSE BLOCK #########################
 
-__version__ = "2.2.1"
+__version__ = "2.3.0"
 from sys import version_info
 
 

lib/requests/packages/chardet/chardetect.py

@@ -12,34 +12,68 @@ Example::
 If no paths are provided, it takes its input from stdin.
 
 """
-from io import open
-from sys import argv, stdin
 
+from __future__ import absolute_import, print_function, unicode_literals
+
+import argparse
+import sys
+from io import open
+
+from chardet import __version__
 from chardet.universaldetector import UniversalDetector
 
 
-def description_of(file, name='stdin'):
+def description_of(lines, name='stdin'):
-    """Return a string describing the probable encoding of a file."""
+    """
+    Return a string describing the probable encoding of a file or
+    list of strings.
+
+    :param lines: The lines to get the encoding of.
+    :type lines: Iterable of bytes
+    :param name: Name of file or collection of lines
+    :type name: str
+    """
     u = UniversalDetector()
-    for line in file:
+    for line in lines:
         u.feed(line)
     u.close()
     result = u.result
     if result['encoding']:
-        return '%s: %s with confidence %s' % (name,
+        return '{0}: {1} with confidence {2}'.format(name, result['encoding'],
-                                              result['encoding'],
+                                                     result['confidence'])
-                                              result['confidence'])
     else:
-        return '%s: no result' % name
+        return '{0}: no result'.format(name)
 
 
-def main():
+def main(argv=None):
-    if len(argv) <= 1:
+    '''
-        print(description_of(stdin))
+    Handles command line arguments and gets things started.
-    else:
+
-        for path in argv[1:]:
+    :param argv: List of arguments, as if specified on the command-line.
-            with open(path, 'rb') as f:
+                 If None, ``sys.argv[1:]`` is used instead.
-                print(description_of(f, path))
+    :type argv: list of str
+    '''
+    # Get command line arguments
+    parser = argparse.ArgumentParser(
+        description="Takes one or more file paths and reports their detected \
+                     encodings",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+        conflict_handler='resolve')
+    parser.add_argument('input',
+                        help='File whose encoding we would like to determine.',
+                        type=argparse.FileType('rb'), nargs='*',
+                        default=[sys.stdin])
+    parser.add_argument('--version', action='version',
+                        version='%(prog)s {0}'.format(__version__))
+    args = parser.parse_args(argv)
+
+    for f in args.input:
+        if f.isatty():
+            print("You are running chardetect interactively. Press " +
+                  "CTRL-D twice at the start of a blank line to signal the " +
+                  "end of your input. If you want help, run chardetect " +
+                  "--help\n", file=sys.stderr)
+        print(description_of(f, f.name))
 
 
 if __name__ == '__main__':

lib/requests/packages/chardet/jpcntx.py

@@ -177,6 +177,12 @@ class JapaneseContextAnalysis:
         return -1, 1
 
 class SJISContextAnalysis(JapaneseContextAnalysis):
+    def __init__(self):
+        self.charset_name = "SHIFT_JIS"
+
+    def get_charset_name(self):
+        return self.charset_name
+
     def get_order(self, aBuf):
         if not aBuf:
             return -1, 1
@@ -184,6 +190,8 @@ class SJISContextAnalysis(JapaneseContextAnalysis):
         first_char = wrap_ord(aBuf[0])
         if ((0x81 <= first_char <= 0x9F) or (0xE0 <= first_char <= 0xFC)):
             charLen = 2
+            if (first_char == 0x87) or (0xFA <= first_char <= 0xFC):
+                self.charset_name = "CP932"
         else:
             charLen = 1
 

lib/requests/packages/chardet/latin1prober.py

@@ -129,11 +129,11 @@ class Latin1Prober(CharSetProber):
         if total < 0.01:
             confidence = 0.0
         else:
-            confidence = ((self._mFreqCounter[3] / total)
+            confidence = ((self._mFreqCounter[3] - self._mFreqCounter[1] * 20.0)
-                          - (self._mFreqCounter[1] * 20.0 / total))
+                          / total)
         if confidence < 0.0:
             confidence = 0.0
         # lower the confidence of latin1 so that other more accurate
         # detector can take priority.
-        confidence = confidence * 0.5
+        confidence = confidence * 0.73
         return confidence

lib/requests/packages/chardet/mbcssm.py

@@ -353,7 +353,7 @@ SJIS_cls = (
     2,2,2,2,2,2,2,2,  # 68 - 6f
     2,2,2,2,2,2,2,2,  # 70 - 77
     2,2,2,2,2,2,2,1,  # 78 - 7f
-    3,3,3,3,3,3,3,3,  # 80 - 87
+    3,3,3,3,3,2,2,3,  # 80 - 87
     3,3,3,3,3,3,3,3,  # 88 - 8f
     3,3,3,3,3,3,3,3,  # 90 - 97
     3,3,3,3,3,3,3,3,  # 98 - 9f
@@ -369,9 +369,8 @@ SJIS_cls = (
     2,2,2,2,2,2,2,2,  # d8 - df
     3,3,3,3,3,3,3,3,  # e0 - e7
     3,3,3,3,3,4,4,4,  # e8 - ef
-    4,4,4,4,4,4,4,4,  # f0 - f7
+    3,3,3,3,3,3,3,3,  # f0 - f7
-    4,4,4,4,4,0,0,0   # f8 - ff
+    3,3,3,3,3,0,0,0)  # f8 - ff
-)
 
 
 SJIS_st = (
@@ -571,5 +570,3 @@ UTF8SMModel = {'classTable': UTF8_cls,
                'stateTable': UTF8_st,
                'charLenTable': UTF8CharLenTable,
                'name': 'UTF-8'}
-
-# flake8: noqa

lib/requests/packages/chardet/sjisprober.py

@@ -47,7 +47,7 @@ class SJISProber(MultiByteCharSetProber):
         self._mContextAnalyzer.reset()
 
     def get_charset_name(self):
-        return "SHIFT_JIS"
+        return self._mContextAnalyzer.get_charset_name()
 
     def feed(self, aBuf):
         aLen = len(aBuf)

lib/requests/packages/chardet/universaldetector.py

@@ -71,9 +71,9 @@ class UniversalDetector:
 
         if not self._mGotData:
             # If the data starts with BOM, we know it is UTF
-            if aBuf[:3] == codecs.BOM:
+            if aBuf[:3] == codecs.BOM_UTF8:
                 # EF BB BF  UTF-8 with BOM
-                self.result = {'encoding': "UTF-8", 'confidence': 1.0}
+                self.result = {'encoding': "UTF-8-SIG", 'confidence': 1.0}
             elif aBuf[:4] == codecs.BOM_UTF32_LE:
                 # FF FE 00 00  UTF-32, little-endian BOM
                 self.result = {'encoding': "UTF-32LE", 'confidence': 1.0}

lib/requests/packages/urllib3/__init__.py

@@ -57,7 +57,7 @@ del NullHandler
 
 # Set security warning to only go off once by default.
 import warnings
-warnings.simplefilter('module', exceptions.SecurityWarning)
+warnings.simplefilter('always', exceptions.SecurityWarning)
 
 def disable_warnings(category=exceptions.HTTPWarning):
     """

lib/requests/packages/urllib3/_collections.py

@@ -14,7 +14,7 @@ try: # Python 2.7+
     from collections import OrderedDict
 except ImportError:
     from .packages.ordered_dict import OrderedDict
-from .packages.six import itervalues
+from .packages.six import iterkeys, itervalues
 
 
 __all__ = ['RecentlyUsedContainer', 'HTTPHeaderDict']
@@ -85,8 +85,7 @@ class RecentlyUsedContainer(MutableMapping):
     def clear(self):
         with self.lock:
             # Copy pointers to all values, then wipe the mapping
-            # under Python 2, this copies the list of values twice :-|
+            values = list(itervalues(self._container))
-            values = list(self._container.values())
             self._container.clear()
 
         if self.dispose_func:
@@ -95,7 +94,7 @@ class RecentlyUsedContainer(MutableMapping):
 
     def keys(self):
         with self.lock:
-            return self._container.keys()
+            return list(iterkeys(self._container))
 
 
 class HTTPHeaderDict(MutableMapping):

lib/requests/packages/urllib3/connection.py

@@ -3,6 +3,7 @@ import sys
 import socket
 from socket import timeout as SocketTimeout
 import warnings
+from .packages import six
 
 try:  # Python 3
     from http.client import HTTPConnection as _HTTPConnection, HTTPException
@@ -26,12 +27,20 @@ except (ImportError, AttributeError):  # Platform-specific: No SSL.
         pass
 
 
+try:  # Python 3:
+    # Not a no-op, we're adding this to the namespace so it can be imported.
+    ConnectionError = ConnectionError
+except NameError:  # Python 2:
+    class ConnectionError(Exception):
+        pass
+
+
 from .exceptions import (
     ConnectTimeoutError,
     SystemTimeWarning,
+    SecurityWarning,
 )
 from .packages.ssl_match_hostname import match_hostname
-from .packages import six
 
 from .util.ssl_ import (
     resolve_cert_reqs,
@@ -40,8 +49,8 @@ from .util.ssl_ import (
     assert_fingerprint,
 )
 
-from .util import connection
 
+from .util import connection
 
 port_by_scheme = {
     'http': 80,
@@ -233,8 +242,15 @@ class VerifiedHTTPSConnection(HTTPSConnection):
                                self.assert_fingerprint)
         elif resolved_cert_reqs != ssl.CERT_NONE \
                 and self.assert_hostname is not False:
-            match_hostname(self.sock.getpeercert(),
+            cert = self.sock.getpeercert()
-                           self.assert_hostname or hostname)
+            if not cert.get('subjectAltName', ()):
+                warnings.warn((
+                    'Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. '
+                    'This feature is being removed by major browsers and deprecated by RFC 2818. '
+                    '(See https://github.com/shazow/urllib3/issues/497 for details.)'),
+                    SecurityWarning
+                )
+            match_hostname(cert, self.assert_hostname or hostname)
 
         self.is_verified = (resolved_cert_reqs == ssl.CERT_REQUIRED
                             or self.assert_fingerprint is not None)

lib/requests/packages/urllib3/connectionpool.py

@@ -32,7 +32,7 @@ from .connection import (
     port_by_scheme,
     DummyConnection,
     HTTPConnection, HTTPSConnection, VerifiedHTTPSConnection,
-    HTTPException, BaseSSLError,
+    HTTPException, BaseSSLError, ConnectionError
 )
 from .request import RequestMethods
 from .response import HTTPResponse
@@ -278,6 +278,23 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
             # can be removed later
             return Timeout.from_float(timeout)
 
+    def _raise_timeout(self, err, url, timeout_value):
+        """Is the error actually a timeout? Will raise a ReadTimeout or pass"""
+
+        if isinstance(err, SocketTimeout):
+            raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
+
+        # See the above comment about EAGAIN in Python 3. In Python 2 we have
+        # to specifically catch it and throw the timeout error
+        if hasattr(err, 'errno') and err.errno in _blocking_errnos:
+            raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
+
+        # Catch possible read timeouts thrown as SSL errors. If not the
+        # case, rethrow the original. We need to do this because of:
+        # http://bugs.python.org/issue10272
+        if 'timed out' in str(err) or 'did not complete (read)' in str(err):  # Python 2.6
+            raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
+
     def _make_request(self, conn, method, url, timeout=_Default,
                       **httplib_request_kw):
         """
@@ -301,7 +318,12 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
         conn.timeout = timeout_obj.connect_timeout
 
         # Trigger any extra validation we need to do.
-        self._validate_conn(conn)
+        try:
+            self._validate_conn(conn)
+        except (SocketTimeout, BaseSSLError) as e:
+            # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout.
+            self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
+            raise
 
         # conn.request() calls httplib.*.request, not the method in
         # urllib3.request. It also calls makefile (recv) on the socket.
@@ -331,28 +353,8 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
                 httplib_response = conn.getresponse(buffering=True)
             except TypeError:  # Python 2.6 and older
                 httplib_response = conn.getresponse()
-        except SocketTimeout:
+        except (SocketTimeout, BaseSSLError, SocketError) as e:
-            raise ReadTimeoutError(
+            self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
-                self, url, "Read timed out. (read timeout=%s)" % read_timeout)
-
-        except BaseSSLError as e:
-            # Catch possible read timeouts thrown as SSL errors. If not the
-            # case, rethrow the original. We need to do this because of:
-            # http://bugs.python.org/issue10272
-            if 'timed out' in str(e) or \
-               'did not complete (read)' in str(e):  # Python 2.6
-                raise ReadTimeoutError(
-                        self, url, "Read timed out. (read timeout=%s)" % read_timeout)
-
-            raise
-
-        except SocketError as e:  # Platform-specific: Python 2
-            # See the above comment about EAGAIN in Python 3. In Python 2 we
-            # have to specifically catch it and throw the timeout error
-            if e.errno in _blocking_errnos:
-                raise ReadTimeoutError(
-                    self, url, "Read timed out. (read timeout=%s)" % read_timeout)
-
             raise
 
         # AppEngine doesn't have a version attr.
@@ -537,12 +539,15 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
             raise EmptyPoolError(self, "No pool connections are available.")
 
         except (BaseSSLError, CertificateError) as e:
-            # Release connection unconditionally because there is no way to
+            # Close the connection. If a connection is reused on which there
-            # close it externally in case of exception.
+            # was a Certificate error, the next request will certainly raise
-            release_conn = True
+            # another Certificate error.
+            if conn:
+                conn.close()
+                conn = None
             raise SSLError(e)
 
-        except (TimeoutError, HTTPException, SocketError) as e:
+        except (TimeoutError, HTTPException, SocketError, ConnectionError) as e:
             if conn:
                 # Discard the connection for these exceptions. It will be
                 # be replaced during the next _get_conn() call.
@@ -725,8 +730,7 @@ class HTTPSConnectionPool(HTTPConnectionPool):
             warnings.warn((
                 'Unverified HTTPS request is being made. '
                 'Adding certificate verification is strongly advised. See: '
-                'https://urllib3.readthedocs.org/en/latest/security.html '
+                'https://urllib3.readthedocs.org/en/latest/security.html'),
-                '(This warning will only appear once by default.)'),
                 InsecureRequestWarning)
 
 

lib/requests/packages/urllib3/contrib/pyopenssl.py

@@ -29,7 +29,7 @@ Now you can use :mod:`urllib3` as you normally would, and it will support SNI
 when the required modules are installed.
 
 Activating this module also has the positive side effect of disabling SSL/TLS
-encryption in Python 2 (see `CRIME attack`_).
+compression in Python 2 (see `CRIME attack`_).
 
 If you want to configure the default list of supported cipher suites, you can
 set the ``urllib3.contrib.pyopenssl.DEFAULT_SSL_CIPHER_LIST`` variable.
@@ -70,9 +70,14 @@ HAS_SNI = SUBJ_ALT_NAME_SUPPORT
 # Map from urllib3 to PyOpenSSL compatible parameter-values.
 _openssl_versions = {
     ssl.PROTOCOL_SSLv23: OpenSSL.SSL.SSLv23_METHOD,
-    ssl.PROTOCOL_SSLv3: OpenSSL.SSL.SSLv3_METHOD,
     ssl.PROTOCOL_TLSv1: OpenSSL.SSL.TLSv1_METHOD,
 }
+
+try:
+    _openssl_versions.update({ssl.PROTOCOL_SSLv3: OpenSSL.SSL.SSLv3_METHOD})
+except AttributeError:
+    pass
+
 _openssl_verify = {
     ssl.CERT_NONE: OpenSSL.SSL.VERIFY_NONE,
     ssl.CERT_OPTIONAL: OpenSSL.SSL.VERIFY_PEER,
@@ -199,8 +204,21 @@ class WrappedSocket(object):
     def settimeout(self, timeout):
         return self.socket.settimeout(timeout)
 
+    def _send_until_done(self, data):
+        while True:
+            try:
+                return self.connection.send(data)
+            except OpenSSL.SSL.WantWriteError:
+                _, wlist, _ = select.select([], [self.socket], [],
+                                            self.socket.gettimeout())
+                if not wlist:
+                    raise timeout()
+                continue
+
     def sendall(self, data):
-        return self.connection.sendall(data)
+        while len(data):
+            sent = self._send_until_done(data)
+            data = data[sent:]
 
     def close(self):
         if self._makefile_refs < 1:
@@ -248,6 +266,7 @@ def ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=None,
                     ssl_version=None):
     ctx = OpenSSL.SSL.Context(_openssl_versions[ssl_version])
     if certfile:
+        keyfile = keyfile or certfile  # Match behaviour of the normal python ssl library
         ctx.use_certificate_file(certfile)
     if keyfile:
         ctx.use_privatekey_file(keyfile)

lib/requests/packages/urllib3/exceptions.py

@@ -72,11 +72,8 @@ class MaxRetryError(RequestError):
     def __init__(self, pool, url, reason=None):
         self.reason = reason
 
-        message = "Max retries exceeded with url: %s" % url
+        message = "Max retries exceeded with url: %s (Caused by %r)" % (
-        if reason:
+            url, reason)
-            message += " (Caused by %r)" % reason
-        else:
-            message += " (Caused by redirect)"
 
         RequestError.__init__(self, pool, url, message)
 
@@ -141,6 +138,12 @@ class LocationParseError(LocationValueError):
         self.location = location
 
 
+class ResponseError(HTTPError):
+    "Used as a container for an error reason supplied in a MaxRetryError."
+    GENERIC_ERROR = 'too many error responses'
+    SPECIFIC_ERROR = 'too many {status_code} error responses'
+
+
 class SecurityWarning(HTTPWarning):
     "Warned when perfoming security reducing actions"
     pass

lib/requests/packages/urllib3/request.py

@@ -118,18 +118,24 @@ class RequestMethods(object):
         which is used to compose the body of the request. The random boundary
         string can be explicitly set with the ``multipart_boundary`` parameter.
         """
-        if encode_multipart:
-            body, content_type = encode_multipart_formdata(
-                fields or {}, boundary=multipart_boundary)
-        else:
-            body, content_type = (urlencode(fields or {}),
-                                  'application/x-www-form-urlencoded')
-
         if headers is None:
             headers = self.headers
 
-        headers_ = {'Content-Type': content_type}
+        extra_kw = {'headers': {}}
-        headers_.update(headers)
 
-        return self.urlopen(method, url, body=body, headers=headers_,
+        if fields:
-                            **urlopen_kw)
+            if 'body' in urlopen_kw:
+                raise TypeError('request got values for both \'fields\' and \'body\', can only specify one.')
+
+            if encode_multipart:
+                body, content_type = encode_multipart_formdata(fields, boundary=multipart_boundary)
+            else:
+                body, content_type = urlencode(fields), 'application/x-www-form-urlencoded'
+
+            extra_kw['body'] = body
+            extra_kw['headers'] = {'Content-Type': content_type}
+
+        extra_kw['headers'].update(headers)
+        extra_kw.update(urlopen_kw)
+
+        return self.urlopen(method, url, **extra_kw)

lib/requests/packages/urllib3/util/retry.py

@@ -2,10 +2,11 @@ import time
 import logging
 
 from ..exceptions import (
-    ProtocolError,
     ConnectTimeoutError,
-    ReadTimeoutError,
     MaxRetryError,
+    ProtocolError,
+    ReadTimeoutError,
+    ResponseError,
 )
 from ..packages import six
 
@@ -36,7 +37,6 @@ class Retry(object):
     Errors will be wrapped in :class:`~urllib3.exceptions.MaxRetryError` unless
     retries are disabled, in which case the causing exception will be raised.
 
-
     :param int total:
         Total number of retries to allow. Takes precedence over other counts.
 
@@ -184,8 +184,8 @@ class Retry(object):
         return isinstance(err, ConnectTimeoutError)
 
     def _is_read_error(self, err):
-        """ Errors that occur after the request has been started, so we can't
+        """ Errors that occur after the request has been started, so we should
-        assume that the server did not process any of it.
+        assume that the server began processing it.
         """
         return isinstance(err, (ReadTimeoutError, ProtocolError))
 
@@ -198,8 +198,7 @@ class Retry(object):
         return self.status_forcelist and status_code in self.status_forcelist
 
     def is_exhausted(self):
-        """ Are we out of retries?
+        """ Are we out of retries? """
-        """
         retry_counts = (self.total, self.connect, self.read, self.redirect)
         retry_counts = list(filter(None, retry_counts))
         if not retry_counts:
@@ -230,6 +229,7 @@ class Retry(object):
         connect = self.connect
         read = self.read
         redirect = self.redirect
+        cause = 'unknown'
 
         if error and self._is_connection_error(error):
             # Connect retry?
@@ -251,10 +251,16 @@ class Retry(object):
             # Redirect retry?
             if redirect is not None:
                 redirect -= 1
+            cause = 'too many redirects'
 
         else:
-            # FIXME: Nothing changed, scenario doesn't make sense.
+            # Incrementing because of a server error like a 500 in
+            # status_forcelist and a the given method is in the whitelist
             _observed_errors += 1
+            cause = ResponseError.GENERIC_ERROR
+            if response and response.status:
+                cause = ResponseError.SPECIFIC_ERROR.format(
+                    status_code=response.status)
 
         new_retry = self.new(
             total=total,
@@ -262,7 +268,7 @@ class Retry(object):
             _observed_errors=_observed_errors)
 
         if new_retry.is_exhausted():
-            raise MaxRetryError(_pool, url, error)
+            raise MaxRetryError(_pool, url, error or ResponseError(cause))
 
         log.debug("Incremented Retry for (url='%s'): %r" % (url, new_retry))
 

lib/requests/packages/urllib3/util/ssl_.py

@@ -4,18 +4,84 @@ from hashlib import md5, sha1
 from ..exceptions import SSLError
 
 
-try:  # Test for SSL features
+SSLContext = None
-    SSLContext = None
+HAS_SNI = False
-    HAS_SNI = False
+create_default_context = None
 
-    import ssl
+import errno
+import ssl
+
+try:  # Test for SSL features
     from ssl import wrap_socket, CERT_NONE, PROTOCOL_SSLv23
-    from ssl import SSLContext  # Modern SSL?
     from ssl import HAS_SNI  # Has SNI?
 except ImportError:
     pass
 
 
+try:
+    from ssl import OP_NO_SSLv2, OP_NO_SSLv3, OP_NO_COMPRESSION
+except ImportError:
+    OP_NO_SSLv2, OP_NO_SSLv3 = 0x1000000, 0x2000000
+    OP_NO_COMPRESSION = 0x20000
+
+try:
+    from ssl import _DEFAULT_CIPHERS
+except ImportError:
+    _DEFAULT_CIPHERS = (
+        'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:'
+        'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:ECDH+RC4:'
+        'DH+RC4:RSA+RC4:!aNULL:!eNULL:!MD5'
+    )
+
+try:
+    from ssl import SSLContext  # Modern SSL?
+except ImportError:
+    import sys
+
+    class SSLContext(object):  # Platform-specific: Python 2 & 3.1
+        supports_set_ciphers = sys.version_info >= (2, 7)
+
+        def __init__(self, protocol_version):
+            self.protocol = protocol_version
+            # Use default values from a real SSLContext
+            self.check_hostname = False
+            self.verify_mode = ssl.CERT_NONE
+            self.ca_certs = None
+            self.options = 0
+            self.certfile = None
+            self.keyfile = None
+            self.ciphers = None
+
+        def load_cert_chain(self, certfile, keyfile):
+            self.certfile = certfile
+            self.keyfile = keyfile
+
+        def load_verify_locations(self, location):
+            self.ca_certs = location
+
+        def set_ciphers(self, cipher_suite):
+            if not self.supports_set_ciphers:
+                raise TypeError(
+                    'Your version of Python does not support setting '
+                    'a custom cipher suite. Please upgrade to Python '
+                    '2.7, 3.2, or later if you need this functionality.'
+                )
+            self.ciphers = cipher_suite
+
+        def wrap_socket(self, socket, server_hostname=None):
+            kwargs = {
+                'keyfile': self.keyfile,
+                'certfile': self.certfile,
+                'ca_certs': self.ca_certs,
+                'cert_reqs': self.verify_mode,
+                'ssl_version': self.protocol,
+            }
+            if self.supports_set_ciphers:  # Platform-specific: Python 2.7+
+                return wrap_socket(socket, ciphers=self.ciphers, **kwargs)
+            else:  # Platform-specific: Python 2.6
+                return wrap_socket(socket, **kwargs)
+
+
 def assert_fingerprint(cert, fingerprint):
     """
     Checks if given fingerprint matches the supplied certificate.
@@ -91,42 +157,98 @@ def resolve_ssl_version(candidate):
     return candidate
 
 
-if SSLContext is not None:  # Python 3.2+
+def create_urllib3_context(ssl_version=None, cert_reqs=ssl.CERT_REQUIRED,
-    def ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=None,
+                           options=None, ciphers=None):
-                        ca_certs=None, server_hostname=None,
+    """All arguments have the same meaning as ``ssl_wrap_socket``.
-                        ssl_version=None):
-        """
-        All arguments except `server_hostname` have the same meaning as for
-        :func:`ssl.wrap_socket`
 
-        :param server_hostname:
+    By default, this function does a lot of the same work that
-            Hostname of the expected certificate
+    ``ssl.create_default_context`` does on Python 3.4+. It:
-        """
-        context = SSLContext(ssl_version)
-        context.verify_mode = cert_reqs
 
-        # Disable TLS compression to migitate CRIME attack (issue #309)
+    - Disables SSLv2, SSLv3, and compression
-        OP_NO_COMPRESSION = 0x20000
+    - Sets a restricted set of server ciphers
-        context.options |= OP_NO_COMPRESSION
 
-        if ca_certs:
+    If you wish to enable SSLv3, you can do::
-            try:
+
-                context.load_verify_locations(ca_certs)
+        from urllib3.util import ssl_
-            # Py32 raises IOError
+        context = ssl_.create_urllib3_context()
-            # Py33 raises FileNotFoundError
+        context.options &= ~ssl_.OP_NO_SSLv3
-            except Exception as e:  # Reraise as SSLError
+
+    You can do the same to enable compression (substituting ``COMPRESSION``
+    for ``SSLv3`` in the last line above).
+
+    :param ssl_version:
+        The desired protocol version to use. This will default to
+        PROTOCOL_SSLv23 which will negotiate the highest protocol that both
+        the server and your installation of OpenSSL support.
+    :param cert_reqs:
+        Whether to require the certificate verification. This defaults to
+        ``ssl.CERT_REQUIRED``.
+    :param options:
+        Specific OpenSSL options. These default to ``ssl.OP_NO_SSLv2``,
+        ``ssl.OP_NO_SSLv3``, ``ssl.OP_NO_COMPRESSION``.
+    :param ciphers:
+        Which cipher suites to allow the server to select.
+    :returns:
+        Constructed SSLContext object with specified options
+    :rtype: SSLContext
+    """
+    context = SSLContext(ssl_version or ssl.PROTOCOL_SSLv23)
+
+    if options is None:
+        options = 0
+        # SSLv2 is easily broken and is considered harmful and dangerous
+        options |= OP_NO_SSLv2
+        # SSLv3 has several problems and is now dangerous
+        options |= OP_NO_SSLv3
+        # Disable compression to prevent CRIME attacks for OpenSSL 1.0+
+        # (issue #309)
+        options |= OP_NO_COMPRESSION
+
+    context.options |= options
+
+    if getattr(context, 'supports_set_ciphers', True):  # Platform-specific: Python 2.6
+        context.set_ciphers(ciphers or _DEFAULT_CIPHERS)
+
+    context.verify_mode = cert_reqs
+    if getattr(context, 'check_hostname', None) is not None:  # Platform-specific: Python 3.2
+        context.check_hostname = (context.verify_mode == ssl.CERT_REQUIRED)
+    return context
+
+
+def ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=None,
+                    ca_certs=None, server_hostname=None,
+                    ssl_version=None, ciphers=None, ssl_context=None):
+    """
+    All arguments except for server_hostname and ssl_context have the same
+    meaning as they do when using :func:`ssl.wrap_socket`.
+
+    :param server_hostname:
+        When SNI is supported, the expected hostname of the certificate
+    :param ssl_context:
+        A pre-made :class:`SSLContext` object. If none is provided, one will
+        be created using :func:`create_urllib3_context`.
+    :param ciphers:
+        A string of ciphers we wish the client to support. This is not
+        supported on Python 2.6 as the ssl module does not support it.
+    """
+    context = ssl_context
+    if context is None:
+        context = create_urllib3_context(ssl_version, cert_reqs,
+                                         ciphers=ciphers)
+
+    if ca_certs:
+        try:
+            context.load_verify_locations(ca_certs)
+        except IOError as e:  # Platform-specific: Python 2.6, 2.7, 3.2
+            raise SSLError(e)
+        # Py33 raises FileNotFoundError which subclasses OSError
+        # These are not equivalent unless we check the errno attribute
+        except OSError as e:  # Platform-specific: Python 3.3 and beyond
+            if e.errno == errno.ENOENT:
                 raise SSLError(e)
-        if certfile:
+            raise
-            # FIXME: This block needs a test.
+    if certfile:
-            context.load_cert_chain(certfile, keyfile)
+        context.load_cert_chain(certfile, keyfile)
-        if HAS_SNI:  # Platform-specific: OpenSSL with enabled SNI
+    if HAS_SNI:  # Platform-specific: OpenSSL with enabled SNI
-            return context.wrap_socket(sock, server_hostname=server_hostname)
+        return context.wrap_socket(sock, server_hostname=server_hostname)
-        return context.wrap_socket(sock)
+    return context.wrap_socket(sock)
-
-else:  # Python 3.1 and earlier
-    def ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=None,
-                        ca_certs=None, server_hostname=None,
-                        ssl_version=None):
-        return wrap_socket(sock, keyfile=keyfile, certfile=certfile,
-                           ca_certs=ca_certs, cert_reqs=cert_reqs,
-                           ssl_version=ssl_version)

lib/requests/packages/urllib3/util/url.py

@@ -40,6 +40,48 @@ class Url(namedtuple('Url', url_attrs)):
             return '%s:%d' % (self.host, self.port)
         return self.host
 
+    @property
+    def url(self):
+        """
+        Convert self into a url
+
+        This function should more or less round-trip with :func:`.parse_url`. The
+        returned url may not be exactly the same as the url inputted to
+        :func:`.parse_url`, but it should be equivalent by the RFC (e.g., urls
+        with a blank port will have : removed).
+
+        Example: ::
+
+            >>> U = parse_url('http://google.com/mail/')
+            >>> U.url
+            'http://google.com/mail/'
+            >>> Url('http', 'username:password', 'host.com', 80,
+            ... '/path', 'query', 'fragment').url
+            'http://username:password@host.com:80/path?query#fragment'
+        """
+        scheme, auth, host, port, path, query, fragment = self
+        url = ''
+
+        # We use "is not None" we want things to happen with empty strings (or 0 port)
+        if scheme is not None:
+            url += scheme + '://'
+        if auth is not None:
+            url += auth + '@'
+        if host is not None:
+            url += host
+        if port is not None:
+            url += ':' + str(port)
+        if path is not None:
+            url += path
+        if query is not None:
+            url += '?' + query
+        if fragment is not None:
+            url += '#' + fragment
+
+        return url
+
+    def __str__(self):
+        return self.url
 
 def split_first(s, delims):
     """
@@ -84,7 +126,7 @@ def parse_url(url):
     Example::
 
         >>> parse_url('http://google.com/mail/')
-        Url(scheme='http', host='google.com', port=None, path='/', ...)
+        Url(scheme='http', host='google.com', port=None, path='/mail/', ...)
         >>> parse_url('google.com:80')
         Url(scheme=None, host='google.com', port=80, path=None, ...)
         >>> parse_url('/foo?bar')
@@ -162,7 +204,6 @@ def parse_url(url):
 
     return Url(scheme, auth, host, port, path, query, fragment)
 
-
 def get_host(url):
     """
     Deprecated. Use :func:`.parse_url` instead.

lib/requests/sessions.py

@@ -13,7 +13,7 @@ from collections import Mapping
 from datetime import datetime
 
 from .auth import _basic_auth_str
-from .compat import cookielib, OrderedDict, urljoin, urlparse, builtin_str
+from .compat import cookielib, OrderedDict, urljoin, urlparse
 from .cookies import (
     cookiejar_from_dict, extract_cookies_to_jar, RequestsCookieJar, merge_cookies)
 from .models import Request, PreparedRequest, DEFAULT_REDIRECT_LIMIT
@@ -21,6 +21,7 @@ from .hooks import default_hooks, dispatch_hook
 from .utils import to_key_val_list, default_headers, to_native_string
 from .exceptions import (
     TooManyRedirects, InvalidSchema, ChunkedEncodingError, ContentDecodingError)
+from .packages.urllib3._collections import RecentlyUsedContainer
 from .structures import CaseInsensitiveDict
 
 from .adapters import HTTPAdapter
@@ -35,6 +36,8 @@ from .status_codes import codes
 # formerly defined here, reexposed here for backward compatibility
 from .models import REDIRECT_STATI
 
+REDIRECT_CACHE_SIZE = 1000
+
 
 def merge_setting(request_setting, session_setting, dict_class=OrderedDict):
     """
@@ -128,7 +131,7 @@ class SessionRedirectMixin(object):
             # Facilitate relative 'location' headers, as allowed by RFC 7231.
             # (e.g. '/path/to/resource' instead of 'http://domain.tld/path/to/resource')
             # Compliant with RFC3986, we percent encode the url.
-            if not urlparse(url).netloc:
+            if not parsed.netloc:
                 url = urljoin(resp.url, requote_uri(url))
             else:
                 url = requote_uri(url)
@@ -271,9 +274,10 @@ class Session(SessionRedirectMixin):
     """
 
     __attrs__ = [
-        'headers', 'cookies', 'auth', 'timeout', 'proxies', 'hooks',
+        'headers', 'cookies', 'auth', 'proxies', 'hooks', 'params', 'verify',
-        'params', 'verify', 'cert', 'prefetch', 'adapters', 'stream',
+        'cert', 'prefetch', 'adapters', 'stream', 'trust_env',
-        'trust_env', 'max_redirects', 'redirect_cache']
+        'max_redirects',
+    ]
 
     def __init__(self):
 
@@ -326,7 +330,8 @@ class Session(SessionRedirectMixin):
         self.mount('https://', HTTPAdapter())
         self.mount('http://', HTTPAdapter())
 
-        self.redirect_cache = {}
+        # Only store 1000 redirects to prevent using infinite memory
+        self.redirect_cache = RecentlyUsedContainer(REDIRECT_CACHE_SIZE)
 
     def __enter__(self):
         return self
@@ -365,6 +370,7 @@ class Session(SessionRedirectMixin):
             url=request.url,
             files=request.files,
             data=request.data,
+            json=request.json,
             headers=merge_setting(request.headers, self.headers, dict_class=CaseInsensitiveDict),
             params=merge_setting(request.params, self.params),
             auth=merge_setting(auth, self.auth),
@@ -386,7 +392,8 @@ class Session(SessionRedirectMixin):
         hooks=None,
         stream=None,
         verify=None,
-        cert=None):
+        cert=None,
+        json=None):
         """Constructs a :class:`Request <Request>`, prepares it and sends it.
         Returns :class:`Response <Response>` object.
 
@@ -396,6 +403,8 @@ class Session(SessionRedirectMixin):
             string for the :class:`Request`.
         :param data: (optional) Dictionary or bytes to send in the body of the
             :class:`Request`.
+        :param json: (optional) json to send in the body of the
+            :class:`Request`.
         :param headers: (optional) Dictionary of HTTP Headers to send with the
             :class:`Request`.
         :param cookies: (optional) Dict or CookieJar object to send with the
@@ -420,7 +429,7 @@ class Session(SessionRedirectMixin):
             If Tuple, ('cert', 'key') pair.
         """
 
-        method = builtin_str(method)
+        method = to_native_string(method)
 
         # Create the Request.
         req = Request(
@@ -429,6 +438,7 @@ class Session(SessionRedirectMixin):
             headers = headers,
             files = files,
             data = data or {},
+            json = json,
             params = params or {},
             auth = auth,
             cookies = cookies,
@@ -482,15 +492,16 @@ class Session(SessionRedirectMixin):
         kwargs.setdefault('allow_redirects', False)
         return self.request('HEAD', url, **kwargs)
 
-    def post(self, url, data=None, **kwargs):
+    def post(self, url, data=None, json=None, **kwargs):
         """Sends a POST request. Returns :class:`Response` object.
 
         :param url: URL for the new :class:`Request` object.
         :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
+        :param json: (optional) json to send in the body of the :class:`Request`.
         :param \*\*kwargs: Optional arguments that ``request`` takes.
         """
 
-        return self.request('POST', url, data=data, **kwargs)
+        return self.request('POST', url, data=data, json=json, **kwargs)
 
     def put(self, url, data=None, **kwargs):
         """Sends a PUT request. Returns :class:`Response` object.
@@ -535,8 +546,13 @@ class Session(SessionRedirectMixin):
         if not isinstance(request, PreparedRequest):
             raise ValueError('You can only send PreparedRequests.')
 
+        checked_urls = set()
         while request.url in self.redirect_cache:
-            request.url = self.redirect_cache.get(request.url)
+            checked_urls.add(request.url)
+            new_url = self.redirect_cache.get(request.url)
+            if new_url in checked_urls:
+                break
+            request.url = new_url
 
         # Set up variables needed for resolve_redirects and dispatching of hooks
         allow_redirects = kwargs.pop('allow_redirects', True)
@@ -646,12 +662,19 @@ class Session(SessionRedirectMixin):
             self.adapters[key] = self.adapters.pop(key)
 
     def __getstate__(self):
-        return dict((attr, getattr(self, attr, None)) for attr in self.__attrs__)
+        state = dict((attr, getattr(self, attr, None)) for attr in self.__attrs__)
+        state['redirect_cache'] = dict(self.redirect_cache)
+        return state
 
     def __setstate__(self, state):
+        redirect_cache = state.pop('redirect_cache', {})
         for attr, value in state.items():
             setattr(self, attr, value)
 
+        self.redirect_cache = RecentlyUsedContainer(REDIRECT_CACHE_SIZE)
+        for redirect, to in redirect_cache.items():
+            self.redirect_cache[redirect] = to
+
 
 def session():
     """Returns a :class:`Session` for context-management."""

lib/requests/utils.py

@@ -19,6 +19,7 @@ import re
 import sys
 import socket
 import struct
+import warnings
 
 from . import __version__
 from . import certs
@@ -287,6 +288,11 @@ def get_encodings_from_content(content):
 
     :param content: bytestring to extract encodings from.
     """
+    warnings.warn((
+        'In requests 3.0, get_encodings_from_content will be removed. For '
+        'more information, please see the discussion on issue #2266. (This'
+        ' warning should only appear once.)'),
+        DeprecationWarning)
 
     charset_re = re.compile(r'<meta.*?charset=["\']*(.+?)["\'>]', flags=re.I)
     pragma_re = re.compile(r'<meta.*?content=["\']*;?charset=(.+?)["\'>]', flags=re.I)
@@ -351,12 +357,14 @@ def get_unicode_from_response(r):
     Tried:
 
     1. charset from content-type
-
+    2. fall back and replace all unicode characters
-    2. every encodings from ``<meta ... charset=XXX>``
-
-    3. fall back and replace all unicode characters
 
     """
+    warnings.warn((
+        'In requests 3.0, get_unicode_from_response will be removed. For '
+        'more information, please see the discussion on issue #2266. (This'
+        ' warning should only appear once.)'),
+        DeprecationWarning)
 
     tried_encodings = []
 
@@ -570,7 +578,7 @@ def parse_header_links(value):
 
     replace_chars = " '\""
 
-    for val in value.split(","):
+    for val in re.split(", *<", value):
         try:
             url, params = val.split(";", 1)
         except ValueError:
@@ -672,3 +680,18 @@ def to_native_string(string, encoding='ascii'):
             out = string.decode(encoding)
 
     return out
+
+
+def urldefragauth(url):
+    """
+    Given a url remove the fragment and the authentication part
+    """
+    scheme, netloc, path, params, query, fragment = urlparse(url)
+
+    # see func:`prepend_scheme_if_needed`
+    if not netloc:
+        netloc, path = path, netloc
+
+    netloc = netloc.rsplit('@', 1)[-1]
+
+    return urlunparse((scheme, netloc, path, params, query, ''))

pylintrc

@@ -38,9 +38,6 @@ load-plugins=
 # --enable=similarities". If you want to run only the classes checker, but have
 # no Warning level messages displayed, use"--disable=all --enable=classes
 # --disable=W"
-#C0303 whitespace between the end of a line and the newline.
-#C0325 a single item in parentheses follows an if, for, or other keyword
-#C0326 wrong number of spaces is used around an operator, bracket or block opener
 #I0011 an inline option disables a pylint message or a messages category
 #R0801 a set of similar lines has been detected among multiple file
 #W0142 a function or method is called using *args or **kwargs to dispatch argument
@@ -49,7 +46,7 @@ load-plugins=
 # C0330(bad-continuation)
 # E1205(logging-too-many-args)
 
-disable=C0303,C0325,C0326,I0011,R0801,W0142,C0103,C0111,C0301,C0302,C0304,C0321,C1001,E0101,E0203,E0602,E1101,E1123,R0201,R0401,R0911,R0912,R0914,R0915,R0923,W0102,W0109,W0120,W0141,W0201,W0212,W0231,W0232,W0233,W0301,W0311,W0401,W0403,W0404,W0511,W0601,W0602,W0603,W0611,W0612,W0613,W0621,W0622,W0633,W0702,W0703,W1401,W1201,C0330
+disable=I0011,R0801,W0142,C0103,C0111,C0301,C0302,C0304,C0321,C1001,E0101,E0203,E0602,E1101,E1123,R0201,R0401,R0911,R0912,R0914,R0915,R0923,W0102,W0109,W0120,W0141,W0201,W0212,W0231,W0232,W0233,W0301,W0311,W0401,W0403,W0404,W0511,W0601,W0602,W0603,W0611,W0612,W0613,W0621,W0622,W0633,W0702,W0703,W1401,W1201,C0330
 
 [REPORTS]